Best local AI models for NVIDIA RTX 2060 MAX-Q

The largest models that fit fully

The 30 largest of the 114 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX 2060 MAX-Q is a mobile graphics card equipped with 6 GB of GDDR6 memory. This dedicated video memory is the primary constraint when running local artificial intelligence models. To achieve fast processing speeds, the entire active model should ideally fit within this physical memory space. If a model exceeds this limit, your system must rely on slower memory management methods.

The quantization column indicates the compression level applied to each model. Quantization reduces the precision of model weights to save space. For this hardware, the Q4_K_M and Q5_K_M formats represent the best balance of size and accuracy. For example, Granite 3.3 8B, Ministral 8B, and InternLM 3 8B fit within 5.9 GB of video memory using the Q4_K_M quantization. Similarly, Qwen2.5 7B, OLMo 2 7B, and Falcon 3 7B utilize the Q5_K_M quantization to fit exactly within 6 GB of video memory.

Other models like OpenCoder 8B, Seed-Coder 8B, MiniCPM-V 2.6, and Idefics 3 8B also run fully on the graphics card at Q4_K_M using 5.9 GB of memory. You can also run Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large at Q4_K_M within 5.9 GB. Models like EXAONE 3.5 7.8B and Mistral 7B require 5.7 GB at Q4_K_M. For Q5_K_M quantization, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, and OpenChat 3.5 utilize 6 GB of memory. Magicoder-S-DS 6.7B fits within 5.7 GB using Q5_K_M.

When a model is too large for the graphics card, you can offload parts of it to your system RAM. This process requires a system with at least 32 GB of system RAM. Offloading allows you to run larger models but reduces processing speed significantly. For instance, Llama 3.1 8B requires 6.4 GB at Q4_K_M and needs 8.4 GB of system RAM. Gemma 2 9B requires 8 GB at Q4_K_M and needs 10 GB of system RAM. Nemotron Nano 9B and GLM-4 9B both require 6.6 GB at Q4_K_M and need 8.6 GB of system RAM.

Other offload options include Mochi 1 which requires 7.3 GB at Q4_K_M and needs 9.3 GB of system RAM. Open-Sora 2.0 requires 8.1 GB at Q4_K_M and needs 10.1 GB of system RAM. When running any of these models, remember the 4k context caveat. The memory figures listed here are calculated using a standard context window of 4000 tokens. Processing longer documents or maintaining longer conversations will increase memory consumption beyond these baseline figures.