Best local AI models for NVIDIA GTX 1660

The largest models that fit fully

The 30 largest of the 114 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA GTX 1660 graphics card features 6 GB of GDDR5 video memory. This memory limit determines which local AI models can run directly on your hardware. To fit models within this space, we use quantized versions. The quantization column shows the compression level used to reduce model size while keeping accuracy high. A Q4_K_M quant represents a four bit quantization level. A Q5_K_M quant represents a five bit quantization level.

For complete on board execution, models must fit within the 6 GB limit. The largest fully fitting models include Granite 8B, Ministral 8B, InternLM 3 8B, OpenCoder 8B, Seed-Coder 8B, MiniCPM-V 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large. These models run at the Q4_K_M quant and use 5.9 GB of video memory. EXAONE 3.5 7.8B and Mistral 7B also fit completely using 5.7 GB of video memory at the Q4_K_M quant.

Several 7B models can run entirely in video memory at a higher Q5_K_M quant using exactly 6 GB of space. This group includes Qwen2.5 7B, OLMo 2 7B, Falcon 3 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, Codestral Mamba 7B, CodeGemma 7B, aiXcoder-7B, Nxcode, Janus-Pro 7B, Ruyi-Mini-7B, Qwen2-Audio 7B, Qwen2.5-Omni 7B, and YuE. Magicoder-S-DS 6.7B also fits at Q5_K_M using 5.7 GB of video memory.

When a model exceeds the 6 GB video memory limit, you can use CPU offload. This technique splits the model between your graphics card and system RAM. CPU offloading requires a system with 32 GB of system RAM. Offloading allows you to run larger models, but it costs performance. Your processing speed will decrease because system RAM is much slower than the GDDR5 video memory on your graphics card.

Several popular models require CPU offloading on this hardware. Llama 3.1 8B needs 6.4 GB at Q4_K_M and uses 8.4 GB of system RAM. Chroma needs 6.5 GB at Q4_K_M and uses 8.5 GB of system RAM. Gemma 2 9B needs 8 GB at Q4_K_M and uses 10 GB of system RAM. Nemotron Nano 9B, GLM-4 9B, Yi-Coder 9B, GLM-4-9B-Chat, and GLM-4V-9B all need 6.6 GB at Q4_K_M and use 8.6 GB of system RAM. Mochi 1 needs 7.3 GB at Q4_K_M and uses 9.3 GB of system RAM. Open-Sora 2.0 needs 8.1 GB at Q4_K_M and uses 10.1 GB of system RAM.

All memory calculations are based on a standard 4k context window. Running a model with a larger context window requires more memory for the active conversation history. If you increase the context length beyond 4k tokens, the model will exceed the 6 GB video memory limit much faster. This will force the system to use CPU offload even for models that normally fit entirely on the graphics card.