Best local AI models for NVIDIA RTX 2060

The largest models that fit fully

The 30 largest of the 114 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX 2060 graphics card features 6 GB of GDDR6 memory. This onboard memory is the most critical factor when running artificial intelligence models locally. To achieve fast processing speeds, the entire active model should ideally fit inside this video memory. If a model exceeds this capacity, your system must use slower system memory to handle the extra data.

The quantization column indicates the compression level applied to each model. Quantization reduces the precision of model weights to save space. For this hardware, the Q4_K_M and Q5_K_M formats offer the best balance of size and intelligence. A Q5_K_M quant uses slightly more memory than a Q4_K_M quant but preserves more original model accuracy.

Several highly capable models fit entirely within the 6 GB limit of your card. You can run Granite 3.3 8B, Ministral 8B, InternLM 3 8B, OpenCoder 8B, Seed-Coder 8B, MiniCPM-V 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large at the Q4_K_M quantization level using 5.9 GB of video memory. The EXAONE 3.5 7.8B model and Mistral 7B model also fit comfortably at Q4_K_M while using 5.7 GB of memory.

You can also run many 7B models at the higher quality Q5_K_M quantization level. Models like Qwen2.5 7B, OLMo 2 7B, Falcon 3 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, Codestral Mamba 7B, CodeGemma 7B, aiXcoder-7B, Nxcode, Janus-Pro 7B, Ruyi-Mini-7B, Qwen2-Audio 7B, Qwen2.5-Omni 7B, and YuE utilize exactly 6 GB of memory. Magicoder-S-DS 6.7B requires slightly less space at 5.7 GB for the same Q5_K_M quantization.

When a model is too large for the graphics card, you can offload parts of it to your system RAM. This approach requires a system with 32 GB of RAM. For example, Llama 3.1 8B requires 6.4 GB of video memory at Q4_K_M and needs 8.4 GB of system RAM. Similarly, Gemma 2 9B requires 8 GB of video memory and 10 GB of system RAM. Other options like Nemotron Nano 9B, GLM-4 9B, Yi-Coder 9B, GLM-4-9B-Chat, and GLM-4V-9B require 6.6 GB of video memory and 8.6 GB of system RAM.

Larger models like Mochi 1 require 7.3 GB of video memory and 9.3 GB of system RAM. Open-Sora 2.0 requires 8.1 GB of video memory and 10.1 GB of system RAM. Chroma requires 6.5 GB of video memory and 8.5 GB of system RAM. While offloading enables you to run these larger models, it significantly reduces processing speed because system RAM is much slower than GDDR6 video memory.

All listed memory measurements are calculated using a standard 4k context window. If you increase the context window to process longer documents or chat histories, the model will require significantly more memory. Running close to the 6 GB limit of your RTX 2060 means that expanding the context window beyond 4k tokens may cause out of memory errors or force system RAM offloading.