Best local AI models for NVIDIA RTX 4060 Ti 8GB
8 GB GDDR6. At a 4k context, 123 of the 233 models in our catalog with verified parameter counts fit fully, up to Mochi 1 at 10B parameters.
Check your own machine against every model →The largest models that fit fully
The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.
| Model | Parameters | Best quant that fits | Memory used at 4k |
|---|---|---|---|
| Mochi 1 | 10B | Q4_K_M | 7.3 GB |
| Gemma 2 9B | 9B | Q4_K_M | 8 GB |
| Nemotron Nano 4B / 9B | 9B | Q5_K_M | 7.7 GB |
| GLM-4 9B / GLM-4.5-Air | 9B | Q5_K_M | 7.7 GB |
| Yi-Coder 1.5B / 9B | 9B | Q5_K_M | 7.7 GB |
| GLM-4-9B-Chat / CodeGeeX4 | 9B | Q5_K_M | 7.7 GB |
| GLM-4V-9B / GLM-4.1V-Thinking | 9B | Q5_K_M | 7.7 GB |
| Chroma | 8.9B | Q5_K_M | 7.6 GB |
| Llama 3.1 8B | 8B | Q5_K_M | 7.4 GB |
| Granite 3.3 2B / 8B | 8B | Q6_K | 7.9 GB |
| Ministral 3B / 8B | 8B | Q6_K | 7.9 GB |
| InternLM 3 8B | 8B | Q6_K | 7.9 GB |
| OpenCoder 1.5B / 8B | 8B | Q6_K | 7.9 GB |
| Seed-Coder 8B | 8B | Q6_K | 7.9 GB |
| MiniCPM-V 2.6 / MiniCPM-o 2.6 | 8B | Q6_K | 7.9 GB |
| Idefics 3 8B | 8B | Q6_K | 7.9 GB |
| Fuyu-8B | 8B | Q6_K | 7.9 GB |
| Emu3 | 8B | Q6_K | 7.9 GB |
| Stable Diffusion 3.5 Large / Turbo | 8B | Q6_K | 7.9 GB |
| EXAONE 3.5 2.4B / 7.8B | 7.8B | Q6_K | 7.7 GB |
| Mistral 7B | 7B | Q6_K | 7.4 GB |
| Qwen2.5 0.5B / 1.5B / 3B / 7B | 7B | Q6_K | 6.9 GB |
| OLMo 2 1B / 7B | 7B | Q6_K | 6.9 GB |
| Falcon 3 1B / 3B / 7B | 7B | Q6_K | 6.9 GB |
| Command R7B | 7B | Q6_K | 6.9 GB |
| OpenHermes 2.5 | 7B | Q6_K | 6.9 GB |
| Zephyr 7B Beta | 7B | Q6_K | 6.9 GB |
| OpenChat 3.5 | 7B | Q6_K | 6.9 GB |
| Starling LM 7B | 7B | Q6_K | 6.9 GB |
| Codestral Mamba 7B | 7B | Q6_K | 6.9 GB |
Close, but only with CPU offload
These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.
| Model | Parameters | Memory at Q4_K_M | System RAM at 4k |
|---|---|---|---|
| Open-Sora 2.0 | 11B | 8.1 GB needed | 10.1 GB |
| FLUX.1 dev | 12B | 14.4 GB needed | 16.4 GB |
| Gemma 3 12B | 12B | 8.8 GB needed | 10.8 GB |
| Gemma 4 12B | 12B | 8.8 GB needed | 10.8 GB |
| Mistral NeMo 12B | 12B | 8.8 GB needed | 10.8 GB |
| Pixtral 12B | 12B | 8.8 GB needed | 10.8 GB |
| FLUX.1 schnell | 12B | 8.8 GB needed | 10.8 GB |
| FLUX.1 Kontext dev | 12B | 8.8 GB needed | 10.8 GB |
| FLUX.1 Krea dev | 12B | 8.8 GB needed | 10.8 GB |
| Vicuna 13B | 13B | 9.5 GB needed | 11.5 GB |
How to read this
The NVIDIA RTX 4060 Ti features 8 GB of GDDR6 memory. This onboard memory determines the maximum size of the artificial intelligence models you can run entirely on your graphics card. When a model fits completely within this 8 GB limit, it runs at maximum speed because the GPU can access the data directly without waiting for the slower system RAM.
The quantization column shows the optimal compression level for each model. Quantization reduces the precision of model weights to save space. For example, the Q4_K_M quant allows Mochi 1 to use 7.3 GB of memory. The Q6_K quant allows Granite 3.3 8B to fit within 7.9 GB of memory. Higher quantizations preserve more model intelligence but require more memory.
If a model exceeds the 8 GB limit, you must offload some layers to your system RAM. This process requires a system with at least 32 GB of system RAM. For example, running Gemma 3 12B at Q4_K_M requires 8.8 GB of graphics memory and 10.8 GB of system RAM. Offloading allows you to run larger models like Vicuna 13B, but it significantly reduces processing speed.
Running models at their maximum capacity leaves very little room for context. The listed memory usage figures assume a standard 4k context window. If you increase the context window to process longer documents, the memory usage will grow. This extra memory demand can push a fitting model over the 8 GB limit and force slow CPU offloading.
For pure GPU execution, you can run Llama 3.1 8B at Q5_K_M using 7.4 GB of memory. You can also run Mistral 7B at Q6_K using 7.4 GB of memory. If you choose to offload to system RAM, you can run FLUX.1 dev at FP8 which needs 14.4 GB of graphics memory and 16.4 GB of system RAM.