Best local AI models for NVIDIA RTX 2080 SUPER

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX 2080 SUPER features 8 GB of GDDR6 onboard memory. This memory pool determines which local AI models you can run entirely on your graphics hardware. To run a model smoothly at native speeds, the model files and the active context must fit inside this physical limit. If a model exceeds your available hardware memory, performance drops because the system must transfer data back and forth.

The quantization column shows the compression level applied to each model. Quantization reduces the size of model weights to save memory. For this hardware, Q4_K_M represents a four bit quantization that balances size and quality. Q5_K_M and Q6_K offer higher precision but require more memory. For example, Mochi 1 is a 10B model that fits at Q4_K_M using 7.3 GB of memory. Gemma 2 9B fits at Q4_K_M using exactly 8 GB of memory.

Many 9B and 8B models fit well on this card using higher precision quants. Nemotron Nano 9B, GLM-4 9B, Yi-Coder 9B, GLM-4-9B-Chat, and GLM-4V-9B all fit at Q5_K_M using 7.7 GB of memory. Chroma is an 8.9B model that fits at Q5_K_M using 7.6 GB of memory. Llama 3.1 8B fits at Q5_K_M using 7.4 GB of memory. Granite 3.3 8B, Ministral 8B, InternLM 3 8B, OpenCoder 8B, Seed-Coder 8B, MiniCPM-V 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large fit at Q6_K using 7.9 GB of memory.

Smaller models leave more room for active processing. EXAONE 3.5 7.8B fits at Q6_K using 7.7 GB of memory. Mistral 7B fits at Q6_K using 7.4 GB of memory. Qwen2.5 7B, OLMo 2 7B, Falcon 3 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, and Codestral Mamba 7B all fit at Q6_K using 6.9 GB of memory. These models run fast because they leave a buffer inside the 8 GB limit.

When a model is too large for the graphics card, you can offload layers to your system RAM. This offload process allows you to run larger models but slows down generation speeds. For these cases, we assume a system with 32 GB of system RAM. Open-Sora 2.0 is an 11B model that needs 8.1 GB at Q4_K_M and requires 10.1 GB of system RAM. FLUX.1 dev is a 12B model that needs 14.4 GB at FP8 and requires 16.4 GB of system RAM.

Other 12B models can also run using system memory. Gemma 3 12B, Gemma 4 12B, Mistral NeMo 12B, Pixtral 12B, FLUX.1 schnell, FLUX.1 Kontext dev, and FLUX.1 Krea dev need 8.8 GB at Q4_K_M and require 10.8 GB of system RAM. Vicuna 13B is a larger 13B model that needs 9.5 GB at Q4_K_M and requires 11.5 GB of system RAM.

Keep in mind the 4k context caveat when planning your setup. The memory figures listed here are calculated using a standard context window of four thousand tokens. If you increase the context length to process longer documents or chat histories, the memory usage will grow. Running close to the 8 GB limit with a large context can cause the model to spill over into system RAM and slow down.