Best local AI models for NVIDIA RTX 2060 Laptop

The largest models that fit fully

The 30 largest of the 114 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX 2060 Laptop graphics card features 6 GB GDDR6 memory. This dedicated video memory determines which artificial intelligence models can run entirely on your graphics hardware. Running models completely inside this memory ensures the fastest response speeds. If a model exceeds this capacity, it will require system memory offloading which slows down performance.

The quantization column indicates the compression level applied to each model. For example, the Q4_K_M and Q5_K_M formats compress model weights to save space while keeping high accuracy. On this hardware, 8B models like Granite 3.3 8B, Ministral 8B, InternLM 3 8B, OpenCoder 8B, Seed-Coder 8B, MiniCPM-V 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large fit at Q4_K_M quantization using 5.9 GB of memory. EXAONE 3.5 7.8B and Mistral 7B fit at Q4_K_M using 5.7 GB of memory.

Several 7B models run at the higher quality Q5_K_M quantization using exactly 6 GB of memory. These models include Qwen2.5 7B, OLMo 2 7B, Falcon 3 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, Codestral Mamba 7B, CodeGemma 7B, aiXcoder-7B, Nxcode, Janus-Pro 7B, Ruyi-Mini-7B, Qwen2-Audio 7B, Qwen2.5-Omni 7B, and YuE. Magicoder-S-DS 6.7B fits at Q5_K_M using 5.7 GB of memory.

When a model is too large for the 6 GB video memory, you can offload parts of it to your system RAM. This process requires a system with 32 GB RAM. For instance, Llama 3.1 8B needs 6.4 GB at Q4_K_M and uses 8.4 GB of system RAM. Chroma needs 6.5 GB at Q4_K_M and uses 8.5 GB of system RAM. Gemma 2 9B needs 8 GB at Q4_K_M and uses 10 GB of system RAM. Nemotron Nano 9B, GLM-4 9B, Yi-Coder 9B, GLM-4-9B-Chat, and GLM-4V-9B all need 6.6 GB at Q4_K_M and use 8.6 GB of system RAM.

Larger models also run via offloading on a 32 GB system. Mochi 1 needs 7.3 GB at Q4_K_M and uses 9.3 GB of system RAM. Open-Sora 2.0 needs 8.1 GB at Q4_K_M and uses 10.1 GB of system RAM. Remember that memory usage calculations assume a standard 4k context window. If you increase the context window to process longer documents, the memory requirements will rise and may exceed your hardware limits.