Best local AI models for NVIDIA RTX 2060 SUPER

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX 2060 SUPER features 8 GB of GDDR6 video memory. This dedicated memory determines the maximum size of the AI models you can run locally. To run a model entirely on your graphics hardware, the model files and its working memory must fit within this 8 GB limit. If a model exceeds this capacity, your system must use system memory, which slows down processing speeds.

Quantization is a method used to compress AI models so they fit into smaller memory spaces. The quant column shows the best compression level for each model. For example, Gemma 2 9B fits in 8 GB of video memory at the Q4_K_M quantization level. Smaller models like Mistral 7B can run at the higher quality Q6_K quantization level while using only 7.4 GB of video memory.

When a model is too large for the 8 GB video memory, you can offload parts of it to your system RAM. This approach requires a system with at least 32 GB of system RAM. For example, FLUX.1 dev requires 14.4 GB of video memory at FP8 or optimized settings, which means it needs 16.4 GB of system RAM to run. Similarly, Gemma 3 12B needs 8.8 GB of video memory at Q4_K_M and requires 10.8 GB of system RAM.

Offloading models to system RAM comes with a performance cost. System RAM is much slower than the GDDR6 memory on your graphics card. While offloading allows you to run larger models like Vicuna 13B or Pixtral 12B, the generation speed will be significantly slower than running models entirely on the graphics card. For the fastest generation speeds, choose models that fit completely within the 8 GB limit.

You must also consider the memory required for context. The memory figures listed are calculated using a standard 4k context window. If you increase the context window to process longer documents or longer conversations, the model will require more video memory. Running a model close to the 8 GB limit, such as Granite 3.3 8B at Q6_K using 7.9 GB, leaves very little room for expanding the context window.