Demystifying GPU Sizing for LLMs

Méoc

 The Context

Large Open Source (OSS) language models are springing up everywhere – Mistrals, Llama, surprise releases like Kimi K2 and, this month, OpenAI OSS models. Each of them promises open‑source autonomy and fine‑tuned intelligence, but there’s a very practical question lurking in the background: how much hardware does it take to run them? The answer isn’t just about having “some GPUs”; it hinges on understanding the interplay between model size, memory bandwidth and inference throughput. One of my hobby projects is a GPU infrastructure estimator that addresses exactly this question. Initially built as an unwieldy spreadsheet, it has evolved into a simple web tool designed to answer a surprisingly common question: “Can I run the latest open‑source models on‑prem for tens or hundreds of thousands of euros?”.

What the tool does

The App interface

At its core, the estimator calculates three things: memory requirements, compute throughput, and latency. To make the experience approachable, it includes a small library of popular models like Mistral 7B, Llama 2 7B, and even the experimental Kimi K2 and OpenAI OSS recently released models. Each model entry stores basic facts – number of parameters, “active” parameters (for mixture‑of‑experts models), precision in bytes per parameter, hidden dimension and number of layers[2]. We can also freely enter your own LLM config. On the hardware side, the tool ships with a catalogue of GPUs ranging from the modest A10 to H100 NVL cards, complete with their compute performance, VRAM size, memory bandwidth and approximate cost. Users can tweak all of these values or enter their own hardware specifications.
Since it’s a hobby app, usual disclaimers apply and it’s important to study the LLM provider recommendation as a first insight for deployment decision.

Why sizing is non‑trivial

The memory footprint of a model is not just its weights. During inference a transformer maintains a key‑value (KV) cache for each token in the context. Basically, the whole LLM is loaded in memory to speed up access. The tool follows a straightforward formula: the weight memory in gibibytes (vram_model_gib) is calculated as [P × 10^9 × B / 1024^3] , where P is the total parameters (in billions) and B is the number of bytes per parameter. The KV cache per token (kv_cache_per_token_gib) is computed as 2 * B * H * L / 1024^3. The factor 2 represents the K and V matrices; H is the hidden dimension and L is the number of layers. Total KV cache footprint is this per‑token value multiplied by the batch size (queries per second) and the sequence length (input tokens + output tokens). Finally, a small fragmentation factor (≈5 %) is applied to get the total VRAM requirement.

Compute sizing is equally important. For each GPU model the estimator uses either 8‑bit TOPS or 16‑bit TFLOPS depending on the chosen precision. It then calculates how many tokens per second a single GPU can generate based on the active parameter count: tokens_per_s_per_gpu_compute = floor((TF × 10¹²) / (Pa × 10⁹ × 2)). The tool compares the GPU count required to satisfy memory, throughput and latency constraints and recommends the maximum of these numbers. Multiply by the per‑GPU price to get an estimated capital expenditure. All the formulas are listed in Github.

GPU list


To see how these formulas work in practice, consider Mistral 7B. It has ~7 billion parameters with FP16 precision (2 bytes per parameter), a hidden size of 4,096 and 32 layers. Suppose we target 150 queries per second, each with 250 input tokens and 150 generated tokens, and pick an NVIDIA H100 PCIe card (80 GB VRAM, ~989 TFLOPS, 2 TB/s bandwidth and a price around €27,300). Clearly, a single H100 is enough; the estimator will show the memory, compute and latency bottlenecks, along with the estimated cost. But you could also run this same model with less powerful hardware using quantization. It's basically reducing the number of bytes used to store the parameters. Thanks to this app, you can see the result by simply changing the precision field. Then, you can find quantized version on Hugging Face, Sloth or create your own.

Building this tool forced me to dig deeper into transformer internals. Before I wrote a single line of TypeScript, I spent days playing with formulas in an Excel spreadsheet and reading about KV caches, activation recomputation and bandwidth bottlenecks. I wanted to maintain a balance: capture enough detail to make the sizing credible, but not drown users in the ocean of possible configurations. Some simplifications were deliberate—for example, the app assumes 90 % of a GPU’s VRAM is usable for the KV cache and ignores routing delays. It also focuses solely on inference; it does not model redundancy, networking or containerisation overheads. These trade‑offs keep the interface clean and allow architects to make quick, high‑level decisions.

The first version of the estimator was a bit of a monster spreadsheet. It worked, but sharing it was cumbersome. Converting it into a web application using React, TypeScript, and Vite brought immediate benefits: pre‑defined models and GPUs live in a constants file, calculations run in a single function, and users can export their configuration to CSV. Most of the heavy lifting was done by Google Lab Studio code‑generation service, which scaffolded the UI in minutes. My main task afterwards was double checking formulas using my original Excel calculations (Yes, AI makes mistakes to size itself :)).

Takeaways: cost isn’t trivial, but it’s achievable

Experimenting with the estimator has been eye‑opening. Running cutting‑edge models offline like Mistral 24B or Magistral start around 30K while Kimi K2 with his huge 1TB go quickly near 500K€ according to the Kimi K2 creator’s number of GPU’s recommendation.
The surprise comes with the recent release of Open AI open source models which start close to 120K€ (1x H100) for their 120B parameters. It’s actually really cheap for a LLM of this size. The trick comes from the number of bytes needed for the parameters. Reducing this part might affect accuracy so the next few days should tell us if this openAI OSS model is up to current expectations.

Overall, yet the total price tag, while not pocket change, is far from impossible for many companies.


most viewed articles

From Chat to Action: The New Gen AI Revolution

Méoc
Image
Artificial intelligence has already impressed us with its ability to generate text, answer questions, translate, code, and summarize. But all of that… remained passive. Today, AI is taking a new leap. It moves into action. It no longer just talks to you. It can do: publish an article, send an email, schedule a meeting, even run an entire blog. And that changes everything. Real-world examples of AI in action When I talk about generative AI that acts, I am not talking about just producing text or images. I'm talking about executing concrete, autonomous, targeted tasks. Here are a few examples from the field: 1. Fully automated blogging (personal case) An AI agent can: Generate a blog post from an idea. Publish it on a platform like Blogger. Create a cover image or visual summary. Share it automatically on social media. Update a tracking sheet or project management board. This isn’t a dream. I already have it running, and I’ll share the how-t...
READ MORE

GenAI App Architecture Explained (Part 1: The Big Picture)

Méoc
Image
 The first time you interact with an app like ChatGPT, all you see is the chat box. It feels magical — you type, it answers. But for anyone in Ops or engineering, one question immediately comes up:  what’s really happening behind the scenes? At the heart of every GenAI app lies an LLM (Large Language Model), but the LLM alone is just one piece of the puzzle. Around it sits an entire stack: orchestration layers, data pipelines, embeddings, vector databases, plugins, caches, observability tools, and guardrails. Together, they transform a “highly clever text generator” into a production-grade GenAI application. In this post, we’ll start with the big picture: the main components and flows that make an app like ChatGPT work. In the next parts, we’ll drill down into each layer (RAG, Ops monitoring, validation, etc.), highlighting both how they work and what usually breaks in production. Picture of what it feels like the first time we interact with a LLM: Which remind me of this famo...
READ MORE

How to Pick the Best LLM for Programming Cost vs Capability in 2025

Méoc
Image
In 2025, dozens of language models can now write, debug, or refactor code. But which ones are actually good — and which give the best value for money? Let's try to make a quick recap for your future programming session. Disclaimer: this is a moving market and each day can see tremendous changes so this is a picture in time and a methodology example for your own review  🧱 How We Calculate Token Costs The 80 / 20 blended formula Most IDE copilots send a large prompt (project context, previous messages, code files) and receive a short answer (patch, explanation). Field logs from Cline, GitHub Copilot, Cursor and Sourcegraph Cody show that roughly 80 % of tokens are input and 20 % are output . To give a single price you can reason about, we compute: Blended Cost ($/M) = 0.80 × Price_in  +  0.20 × Price_out Example : Claude 4 Opus costs $15 /M input and $75 /M output → 0.80×15 + 0.20×...
READ MORE

GenAI App Architecture Explained (Part 2: Completing the Big Picture)

Méoc
Image
 In the first article , we outlined the high-level architecture of a modern GenAI application: the orchestrator, the embeddings model, the vector database, and the external actions or tools. Today, we complete that picture by adding the supporting components that make the system resilient, observable, and reliable . The often-overlooked building blocks In the diagram above, these gray blocks are the hidden backbone of any production-grade GenAI app. Let's focus on the 3 remaining ones. LLM Cache This is where previously generated model responses are stored so they can be reused later. The goal: reduce latency and avoid unnecessary calls to expensive models. Typical tools: Redis, SQLite, GPTCache. Caching is crucial for both cost control and responsiveness . When a user asks the same or a very similar question, the system can serve a cached response instead of re-querying the model. But caching in the LLM context is not as straightforward as caching a static API response. Queries ...
READ MORE

Can Natural Language Really Replace Code? The Revolution Is Already Underway

Méoc
Image
Natural language processing  has been floating around the tech world for yearsbut what about Natural Language programming?  What about programming whole application directly in Natural language? The tools are here, the agents are live, and people who’ve never written a line of code are already building automations, querying data, and triggering real-world actions … just by describing what they want. Before we get ahead of ourselves, let’s unpack what this term actually means—and why it’s already changing the way we create. Natural language programming isn’t one thing — it splits into two complementary approaches. Depending on the goal, you might want to generate code from a plain-English prompt, or trigger actions directly without code ever surfacing. These two paths are reshaping what “programming” even means. Text-to-code Tools like Codex, Copilot, Cline, or Windsurf translate prose into Python, JavaScript, and more. Text-to-action Autonomous agents interpret i...
READ MORE

Internet Is Nasty? Really? How?

Méoc
Image
Many years ago I connected a server directly to the Internet and watched the incoming traffic. Within minutes, connection attempts targeted every open port on the machine. Most were automated bots scanning the entire address space, looking for weak spots to steal data or take control of the host. That was nasty — and it happened twenty years ago. Fast-forward to today. I wanted to show my kids that the Internet is more than YouTube and Netflix; it is also full of tireless robots poking anything left unprotected. Normally I would drop an idea like this because of the effort involved, but I also wanted to try a new “code vibing” tool called Windsurf. Code vibing? Think of it as ultra-fast, AI-assisted development: you describe what you want, and the tool either codes it for you or constantly boosts your productivity. Here’s a video showing how I changed a small part of the interface by simply… asking for it. Building a Live Attack Dashboard in 6 Hours with AI To create my app, I ...
READ MORE

GenAI App Architecture Explained (Part 3: The Hardware stack)

Méoc
Image
 In  Part 1  and  Part 2  of this series, we explored the high-level components of a modern GenAI application, from the user interface to the RAG pipeline. Now, let's get down to the metal. What type of resources do we need to make it all run? What Resources Do We Need? Like any application, LLMs rely on fundamental hardware resources to function: compute, memory, storage, and network. And sometimes, they need a  lot  of them. Compute LLM architectures, particularly Transformers, are heavy consumers of compute and memory. This is inherent to their design, which must process massive amounts of calculations to predict subsequent tokens. Think about it: each new token generated is the result of a complex calculation. All tokens from the prompt are turned into mathematical representations (vectors) to understand the intricate relationships between them. Based on this analysis, the model performs billions of calculations to predict the most likely next toke...
READ MORE