Best local AI models for NVIDIA RTX 2070 SUPER

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX 2070 SUPER graphics card features 8 GB of GDDR6 dedicated video memory. This memory capacity determines the size of the artificial intelligence models you can run locally. When running local models, the entire model weights should ideally fit inside this video memory to ensure fast processing speeds. If a model exceeds this limit, your system must use alternative execution methods.

The quantization column indicates the compression level applied to each model. Quantization reduces the precision of model weights to save space. For example, the Q4_K_M quant represents a four bit quantization level. The Q5_K_M quant uses five bits, while the Q6_K quant uses six bits. Higher quantization levels like Q6_K preserve more original model quality but require more video memory. Lower quantization levels like Q4_K_M allow larger models to fit within your 8 GB limit.

With 8 GB of video memory, you can run several capable models entirely on your graphics card. Mochi 1 at 10B fits using the Q4_K_M quant which uses 7.3 GB of memory. Gemma 2 9B fits at Q4_K_M using exactly 8 GB of memory. Nemotron Nano 9B, GLM-4 9B, Yi-Coder 9B, GLM-4-9B-Chat, and GLM-4V-9B all fit at Q5_K_M using 7.7 GB of memory. Chroma 8.9B fits at Q5_K_M using 7.6 GB of memory. Llama 3.1 8B fits at Q5_K_M using 7.4 GB of memory.

Other models fit comfortably using the higher quality Q6_K quantization. Granite 3.3 8B, Ministral 8B, InternLM 3 8B, OpenCoder 8B, Seed-Coder 8B, MiniCPM-V 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large all use 7.9 GB of memory at Q6_K. EXAONE 3.5 7.8B uses 7.7 GB of memory at Q6_K. Mistral 7B uses 7.4 GB of memory at Q6_K. Qwen2.5 7B, OLMo 2 7B, Falcon 3 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, and Codestral Mamba 7B all use 6.9 GB of memory at Q6_K.

When a model is too large for your video memory, you can use CPU offloading. This process splits the model between your graphics card and your system RAM. CPU offloading allows you to run larger models but reduces processing speed significantly. For these cases, we assume your system has 32 GB of system RAM. Open-Sora 2.0 at 11B needs 8.1 GB of video memory at Q4_K_M and 10.1 GB of system RAM. FLUX.1 dev at 12B needs 14.4 GB of video memory at FP8 and 16.4 GB of system RAM.

Other offload options include Gemma 3 12B, Gemma 4 12B, Mistral NeMo 12B, Pixtral 12B, FLUX.1 schnell, FLUX.1 Kontext dev, and FLUX.1 Krea dev. These 12B models all need 8.8 GB of video memory at Q4_K_M and 10.8 GB of system RAM. Vicuna 13B needs 9.5 GB of video memory at Q4_K_M and 11.5 GB of system RAM. You must also consider the context window limit. These memory calculations are based on a standard 4k context window. Running longer text contexts will increase memory usage and may cause out of memory errors.