Best local AI models for NVIDIA RTX 3060 Ti

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX 3060 Ti graphics card features 8 GB of GDDR6 memory. This dedicated memory determines the size of the artificial intelligence models you can run locally. To run a model entirely on your graphics hardware, the model files and the active workspace must fit within this 8 GB limit. If a model exceeds this limit, your system must use slower system memory.

Quantization is a method that compresses model files to save space. The quant column shows the best balance of size and quality for this card. For example, Mochi 1 is a 10B model that fits at the Q4_K_M quantization level because it uses 7.3 GB of memory. Gemma 2 9B fits at the Q4_K_M level using exactly 8 GB of memory. Models like Llama 3.1 8B fit at the Q5_K_M level using 7.4 GB of memory.

Smaller models can use higher precision quantization levels. Granite 3.3 8B, Ministral 8B, InternLM 3 8B, and OpenCoder 8B fit at the Q6_K level using 7.9 GB of memory. Seed-Coder 8B, MiniCPM-V 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large also run at the Q6_K level using 7.9 GB of memory. EXAONE 3.5 7.8B uses 7.7 GB at the Q6_K level. Mistral 7B uses 7.4 GB at the Q6_K level.

Other popular options run comfortably within the memory limit. Qwen2.5 7B, OLMo 2 7B, Falcon 3 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, and Codestral Mamba 7B all fit at the Q6_K level using 6.9 GB of memory. Nemotron Nano 9B, GLM-4 9B, Yi-Coder 9B, GLM-4-9B-Chat, and GLM-4V-9B fit at the Q5_K_M level using 7.7 GB of memory. Chroma uses 7.6 GB at the Q5_K_M level.

You can run larger models by offloading parts of the workload to your system memory. This requires a system with at least 32 GB of system RAM. Offloading allows you to run Gemma 3 12B, Gemma 4 12B, Mistral NeMo 12B, Pixtral 12B, FLUX.1 schnell, FLUX.1 Kontext dev, and FLUX.1 Krea dev at Q4_K_M. These models need 8.8 GB of graphics memory and 10.8 GB of system RAM. Open-Sora 2.0 needs 8.1 GB at Q4_K_M and 10.1 GB of system RAM.

Other offload options include FLUX.1 dev which needs 14.4 GB at FP8 and 16.4 GB of system RAM. Vicuna 13B needs 9.5 GB at Q4_K_M and 11.5 GB of system RAM. Offloading makes these larger models accessible but it reduces processing speed significantly. You must also consider that these memory calculations assume a standard 4k context window. Increasing the context window length will require more memory and might cause the model to exceed your available hardware limits.