The message that started it was simple. A friend sent me a screenshot of Llama 3.1 70B answering a tricky question, running fully offline on his own desktop. No API bill. No data leaving the room. Just a private model that felt close to the big cloud assistants. I wanted that. So I did what most people do. I opened a terminal, typed the command to pull a 70B model, and waited.
Then reality arrived.
The download alone was tens of gigabytes. My graphics card had 12 GB of memory. The model needed far more than that. It started, sort of, then crawled along at a speed that made me want to close the laptop. That was the day I learned that running a 70B model at home is less about wanting it and more about doing the memory math first. So let me save you the painful trial and error and walk you through exactly what it takes.
What a 70B model actually is
The 70B in the name means about 70 billion parameters. Think of parameters as the numbers the model learned during training. To run the model, your computer has to load those numbers into fast memory and keep them there. The whole question of whether your machine can run it comes down to one thing. Can it hold all those numbers, plus a little working room, in memory at once.
That is why the first number you should care about is not speed. It is memory. If the weights do not fit, nothing else matters much.
How precision changes everything
Here is the part that surprises people. A 70B model does not have one fixed size. Its size depends on precision, which is how many bits you use to store each parameter. This is called quantisation.
At full precision, called FP16, each parameter takes about 2 bytes. Multiply that by 70 billion and you get roughly 140 GB just for the weights. That is data center territory, not a home desk.
Drop to 8 bit, often written Q8, and each parameter takes about 1 byte. Now the weights are around 70 GB. Better, but still a heavy load.
Drop again to 4 bit, for example the popular Q4_K_M format, and each parameter takes roughly half a byte plus a little overhead. In practice the weights land around 40 GB to 43 GB. This is the sweet spot most people use at home, because it keeps most of the quality while shrinking the model to something a strong machine can actually hold.
The memory you really need
Weights are only part of the story. On top of them you need room for the context, the running memory of your conversation or the document you fed in. This is called the KV cache, and it grows as the conversation gets longer. Budget several extra GB for it, and more if you plan to work with long documents.
So the real number you plan around is weights plus context, not weights alone. Here is how the three common precisions compare.
| Precision | Approx weights size | Memory you realistically need | Typical way to run it |
|---|---|---|---|
| FP16 | about 140 GB | 150 GB or more with context | Multiple data center GPUs, server hardware |
| Q8 | about 70 GB | roughly 80 GB with context | Three or four 24 GB GPUs, or a large unified memory machine |
| Q4 (Q4_K_M) | about 40 GB to 43 GB | roughly 48 GB with context | Two 24 GB GPUs, one 48 GB card, or a 64 GB Mac |
What this means for real hardware
Now let us turn those numbers into actual machines people own.
A single 24 GB card, like an RTX 3090 or RTX 4090, cannot hold a 70B model in Q4 fully in its own memory. It comes close on paper but falls short once context is added. It will still run by offloading some layers to your system RAM, which works, but it gets much slower. Think usable for patient tinkering, not snappy chat.
Two 24 GB cards, giving you 48 GB total, is the common home sweet spot. That is enough to hold Q4 comfortably and run at a genuinely usable speed. A single 48 GB professional card does the same job in one slot.
Q8 at around 70 GB is a bigger ask. Once you add context you are looking at roughly 80 GB of memory, which means three or four 24 GB cards or a large unified memory machine.
Apple Silicon deserves its own mention, because it changed the game for home users. Macs share one pool of unified memory between the processor and the graphics side. That means a Mac with 64 GB or more of unified memory can run a 70B model in Q4, and 128 GB is very comfortable. For a lot of people this is the simplest single box path to a private 70B assistant.
CPU only, or heavy offloading to RAM, does technically work. I have watched it happen. But it is slow, often a few tokens per second or less, which feels like waiting for each word to arrive. Fine for a one off question, painful for real work.
The quality trade off
You might wonder if squeezing the model down to 4 bit makes it dumb. The honest answer is that lower bits do reduce quality a little. But the drop from FP16 to a good Q4 build is smaller than most people expect. For a 70B model, Q4 keeps most of the reasoning and knowledge while fitting into memory you can actually buy. That balance is exactly why Q4 is the default choice for running these models at home.
If you have the memory to spare, Q8 gets you a touch closer to full quality. If you do not, Q4 is not a compromise you will regret.
So, can your machine do it
Here is the short version. If you have two 24 GB GPUs, a 48 GB card, or a Mac with 64 GB or more of unified memory, a 70B model in Q4 is well within reach and will feel usable. One 24 GB card can do it slowly with offloading. Anything smaller means CPU speeds and a lot of patience.
The cleanest way to know for sure is to try your own numbers. This post has a live VRAM calculator embedded right below it. Punch in your precision and your context length and see the memory it needs. Then check specific models against your exact GPU in our tool, starting with Llama 3.1 70B, Llama 3.3 70B, and the easy way to run them with Ollama.
Your private 70B assistant might be closer than you think. It just starts with getting the memory math right.
