Best local AI models for NVIDIA RTX 2060 12GB

The largest models that fit fully

The 30 largest of the 147 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX 2060 12GB graphics card features 12 GB of GDDR6 video memory. This hardware memory limit determines which local AI models you can run entirely on your GPU. To run a model smoothly without slowdowns, the model files and the active context data must fit inside this 12 GB limit.

The best quant column shows the optimal quantization level for each model on this hardware. Quantization compresses model weights to save space. A Q4_K_M quant uses four bits per weight and offers a great balance of speed and intelligence. A Q5_K_M quant uses five bits for better accuracy, while Q6_K and Q8_0 quants use six and eight bits to deliver maximum precision when memory allows.

For models that fit completely in video memory, you can run DeepSeek-Coder-V2 16B or Kimi-VL A3B at Q4_K_M which use 11.7 GB. You can also run Apriel-1.5-15B-Thinker or StarCoder2 15B at Q4_K_M using 11 GB. The Qwen2.5 14B model fits at Q4_K_M using 11.6 GB. Models like Phi-4, Phi-4-reasoning / -plus, Wan 2.2 T2I, Wan 2.1 (1.3B / 14B), and SkyReels V2 fit at Q5_K_M using 11.9 GB. Vicuna 13B, HunyuanVideo, HunyuanVideo-Avatar, LTX-Video / LTX-2, and FramePack fit at Q5_K_M using 11.1 GB.

Smaller models can run at higher precision levels. Gemma 3 12B, Gemma 4 12B, Mistral NeMo 12B, Pixtral 12B, FLUX.1 schnell, FLUX.1 Kontext dev, and FLUX.1 Krea dev fit at Q6_K using 11.8 GB. Open-Sora 2.0 fits at Q6_K using 10.8 GB, while Mochi 1 fits at Q6_K using 9.8 GB. Gemma 2 9B fits at Q6_K using 10.3 GB. Nemotron Nano 4B / 9B, GLM-4 9B / GLM-4.5-Air, Yi-Coder 1.5B / 9B, and GLM-4-9B-Chat / CodeGeeX4 fit at Q8_0 using 11.4 GB.

When a model exceeds the 12 GB video memory limit, you must offload parts of it to your system RAM. This CPU offload process requires a system with 32 GB of system RAM. Offloading allows you to run larger models, but it reduces generation speed because system RAM is much slower than GDDR6 video memory.

With CPU offload, you can run FLUX.1 dev which needs 14.4 GB at FP8 / optimized and uses 16.4 GB of system RAM. Ling-Coder-Lite needs 12.3 GB at Q4_K_M and uses 14.3 GB of system RAM. HunyuanImage 2.1 / 3.0 needs 12.4 GB at Q4_K_M and uses 14.4 GB of system RAM. CogVLM2 needs 13.9 GB at Q4_K_M and uses 15.9 GB of system RAM. Qwen-Image and Qwen-Image-Edit need 14.6 GB at Q4_K_M and use 16.6 GB of system RAM. Larger options like gpt-oss-20b and Reka Flash 3 need 15.4 GB at Q4_K_M and use 17.4 GB of system RAM. Solar Pro and Codestral 22B need 16.1 GB at Q4_K_M and use 18.1 GB of system RAM.

All memory calculations assume a standard 4k context window. If you increase the context window to process longer documents or chat histories, the model will require significantly more memory. Running close to the 12 GB limit with a large context can cause out of memory errors or force your system into slow CPU offloading.