Picture someone standing in a computer shop, phone in hand, staring at a wall of graphics cards. They do not care about frame rates. They want to run AI on their own machine, privately, no monthly fee, no sending their notes to a server somewhere. They have read that they need enough VRAM. But how much? One forum post says 8 GB is plenty. Another says you need 24 GB or do not bother. A third throws around words like quantisation and KV cache. The numbers do not line up, the sales assistant shrugs, and the confusion wins.
Here is the good news. There is one number that decides almost everything, and the math behind it is simple. Once you see it, the whole wall of cards sorts itself into neat tiers.
VRAM is the number that decides everything
When you run a language model on your own machine, the model has to live in memory while it works. On a graphics card, that memory is the VRAM. The whole model needs to fit. If it fits, the card runs it fast. If it does not fit, things get slow or stop working.
Everything else, the number of cores, the clock speed, the brand, matters for how quickly words come out once the model is loaded. But VRAM decides whether you can load it at all. That is why it is the single most important spec for local AI, and why it deserves your attention before anything else.
The simple math behind the numbers
A model is a big pile of numbers called parameters, and we count them in billions. A 7B model has seven billion. A 70B model has seventy billion. To know how much VRAM you need, you multiply the parameter count by how many bytes each parameter takes up. That second part is set by the quantisation, which is just how tightly each number is packed.
Here is the whole rule in three lines:
- 4 bit (Q4): about half a GB of VRAM per billion parameters, plus a little overhead
- 8 bit (Q8): about 1 GB per billion parameters
- 16 bit (full): about 2 GB per billion parameters
Most people run models in Q4, because it is the popular balance of small size and good quality. So in Q4 the math is easy. Take the billions of parameters, halve it, and that is roughly your VRAM in GB.
That gives you a clean set of real numbers. A 7B to 8B model needs roughly 5 GB. A 13B to 14B model needs roughly 9 GB to 10 GB. A 32B model needs roughly 20 GB. A 70B model needs roughly 40 GB. Then add a little on top for the context, the KV cache, which grows with how long your prompt and reply are. A short chat needs very little. A long document needs more.
Match the numbers to real graphics cards
Now the fun part. Once you know the model math, you can look at a card, read its VRAM, and know what it will comfortably run. Comfortable means the model fits with room left over for context, so you are not living right on the edge.
| GPU VRAM | Example cards | Largest comfortable model in Q4 |
|---|---|---|
| 8 GB | RTX 3050, RTX 4060 | 7B to 8B, the entry point |
| 12 GB | RTX 3060 12 GB, RTX 4070 | up to 13B to 14B with room to spare |
| 16 GB | RTX 4060 Ti 16 GB, RTX 4080 | 14B comfortably, larger with tighter context |
| 24 GB | RTX 3090, RTX 4090 | 32B, the serious single card tier |
| 48 GB or more | two 24 GB cards, or a pro card | 70B in Q4 |
Read that table once and the shop wall stops being scary. An 8 GB card is the entry point and runs the popular 7B to 8B models, which are genuinely useful for chat, writing and coding help. A 12 GB card steps you up to 13B to 14B. A 16 GB card gives 14B breathing room. A 24 GB card is where a single card gets serious and runs a 32B model well. And 70B, the big one, needs 48 GB or more, which usually means two 24 GB cards or a professional card.
Quality, quants, and the trade you get to make
Q4 is the default for a reason, but it is not the only choice. Higher quants like Q5, Q6 and Q8 store each number with more bits, so they use more memory in exchange for slightly better quality. If you have VRAM to spare after your model fits, spending it on a higher quant is a nice upgrade.
So the trade is yours to make. Tight on memory? Q4 keeps the model small. Room left over? Step up a quant for a small quality bump. This is why two people with the same card can run the same model at different sizes. They are choosing different points on the quality curve.
What happens when it does not fit
Say you try a model that is a bit too big for your card. It usually still runs. Most local AI tools will offload the part that does not fit into your system RAM and lean on the CPU for it. It works, but it crawls, because system memory and the CPU cannot feed data as fast as the graphics card. The difference is not small. A model that flies fully inside VRAM can slow to a frustrating trickle once it spills over.
That is the whole reason the fitting rule matters. Getting the model fully into VRAM is the difference between a snappy local assistant and one you give up on. When you are close to the line, dropping to a smaller quant to squeeze the model fully into VRAM is almost always faster than letting it spill.
One note for Mac owners. Apple Silicon uses unified memory, a single shared pool instead of separate VRAM. The same fitting logic applies to that pool, just remember the system is already using some of it, so leave headroom.
Try it before you buy it
You do not have to do this math in your head. This post has a live VRAM calculator embedded right below. Plug in a model size and a quant, and it shows you the fit in plain GB. Then head into our tool and check the specific models you actually want to run against the exact card you are eyeing, so you buy once and buy right.
Start with the popular sizes. Look at [Llama 3.1 8B](/ai-models/llama-3-1-8b) and [Mistral 7B](/ai-models/mistral-7b) if you are shopping around an 8 GB card, and step up to [Qwen3 8B, 14B and 32B](/ai-models/qwen3-8b-14b-32b) as you climb the VRAM tiers. The number that had you stuck at the shop wall turns out to be the one that makes the whole decision easy.
