Best local AI models for NVIDIA RTX 2080

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX 2080 graphics card features 8 GB of GDDR6 video memory. This dedicated memory determines the maximum size of the artificial intelligence models you can run locally. To run a model entirely on your graphics hardware, the model files and the active context data must fit within this 8 GB limit. If a model exceeds this capacity, your system must use alternative execution methods.

Quantization is a compression technique that reduces model size. The quant column shows the best format to balance file size and output quality. For example, the Q4_K_M quant uses four bit quantization to fit larger models into smaller memory footprints. The Q5_K_M and Q6_K quants provide higher precision but require more memory space. Using these optimized formats allows the RTX 2080 to run advanced architectures that would otherwise be too large.

Many capable models fit completely within the 8 GB video memory of the RTX 2080. The Mochi 1 10B model fits at the Q4_K_M quant while using 7.3 GB of memory. Gemma 2 9B utilizes exactly 8 GB at the Q4_K_M quant. Several 9B models like Nemotron Nano 4B / 9B, GLM-4 9B / GLM-4.5-Air, Yi-Coder 1.5B / 9B, GLM-4-9B-Chat / CodeGeeX4, and GLM-4V-9B / GLM-4.1V-Thinking fit at the Q5_K_M quant using 7.7 GB of memory. Chroma 8.9B fits at the Q5_K_M quant using 7.6 GB. Llama 3.1 8B fits at the Q5_K_M quant using 7.4 GB.

Other models fit within the limit using the higher quality Q6_K quant. Granite 3.3 2B / 8B, Ministral 3B / 8B, InternLM 3 8B, OpenCoder 1.5B / 8B, Seed-Coder 8B, MiniCPM-V 2.6 / MiniCPM-o 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large / Turbo all use 7.9 GB of memory. EXAONE 3.5 2.4B / 7.8B uses 7.7 GB at the Q6_K quant. Mistral 7B uses 7.4 GB. Qwen2.5 0.5B / 1.5B / 3B / 7B, OLMo 2 1B / 7B, Falcon 3 1B / 3B / 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, and Codestral Mamba 7B all use 6.9 GB of memory at the Q6_K quant.

When a model is too large for the 8 GB video memory, you can offload parts of it to your system RAM. This offloading process allows you to run larger models but slows down processing speeds. Assuming you have 32 GB of system RAM, you can run Open-Sora 2.0 11B which needs 8.1 GB at Q4_K_M and 10.1 GB of system RAM. FLUX.1 dev 12B needs 14.4 GB at FP8 / optimized and 16.4 GB of system RAM. Vicuna 13B needs 9.5 GB at Q4_K_M and 11.5 GB of system RAM.

Other 12B models can also run using system RAM offloading. Gemma 3 12B, Gemma 4 12B, Mistral NeMo 12B, Pixtral 12B, FLUX.1 schnell, FLUX.1 Kontext dev, and FLUX.1 Krea dev all require 8.8 GB at the Q4_K_M quant and 10.8 GB of system RAM. You must also consider the 4k context caveat. The listed memory usage figures are calculated using a standard 4k context window. If you increase the context window to process longer texts, the memory usage will rise and may exceed your available video memory.