Best local AI models for NVIDIA GTX 1060 6GB

The largest models that fit fully

The 30 largest of the 114 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA GTX 1060 6GB graphics card features 6 GB of GDDR5 video memory. This physical limit determines which artificial intelligence models can run entirely on your graphics hardware. When a model fits completely within this video memory, you get the fastest generation speeds. If a model exceeds this limit, you must use alternative execution strategies.

The quantization column shows the compression level applied to each model. Quantization reduces the size of model weights to save memory. For this graphics card, the best quantization level for eight billion parameter models is Q4_K_M, which uses 5.9 GB of video memory. For seven billion parameter models, you can use the higher quality Q5_K_M quantization, which uses 6 GB of video memory.

Several models fit entirely within the video memory of your card. Granite 3.3 8B, Ministral 8B, InternLM 3 8B, OpenCoder 8B, Seed-Coder 8B, MiniCPM-V 2.6, MiniCPM-o 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large all run at Q4_K_M quantization using 5.9 GB. EXAONE 3.5 7.8B and Mistral 7B run at Q4_K_M using 5.7 GB. Magicoder-S-DS 6.7B runs at Q5_K_M using 5.7 GB.

Other models also fit fully inside the video memory at Q5_K_M quantization using 6 GB. These models include Qwen2.5 7B, OLMo 2 7B, Falcon 3 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, Codestral Mamba 7B, CodeGemma 7B, aiXcoder-7B, Nxcode, CodeQwen 1.5 7B, Janus-Pro 7B, Ruyi-Mini-7B, Qwen2-Audio 7B, Qwen2.5-Omni 7B, and YuE.

When a model is too large for the video memory, you can offload parts of it to your system RAM. This requires a system with 32 GB of system RAM. Offloading allows you to run larger models, but it significantly reduces processing speed because system RAM is slower than video memory. For example, Llama 3.1 8B needs 6.4 GB at Q4_K_M and requires 8.4 GB of system RAM. Gemma 2 9B needs 8 GB at Q4_K_M and requires 10 GB of system RAM.

Other offload options include Chroma, which needs 6.5 GB at Q4_K_M and 8.5 GB of system RAM. Nemotron Nano 9B, GLM-4 9B, Yi-Coder 9B, GLM-4-9B-Chat, CodeGeeX4, GLM-4V-9B, and GLM-4.1V-Thinking all need 6.6 GB at Q4_K_M and 8.6 GB of system RAM. Mochi 1 needs 7.3 GB at Q4_K_M and 9.3 GB of system RAM. Open-Sora 2.0 needs 8.1 GB at Q4_K_M and 10.1 GB of system RAM.

Memory calculations assume a standard context window of four thousand tokens. Running models with longer context windows increases memory usage. If you generate very long responses or input large documents, the model might exceed the video memory limit and force system RAM offloading even if the base model fits.