Best local AI models for NVIDIA RTX 2080 MAX-Q

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX 2080 MAX-Q is a mobile graphics card equipped with 8 GB of GDDR6 memory. This dedicated video memory determines the size of the artificial intelligence models you can run locally. To load a model entirely on your graphics hardware for fast generation speeds, the model files and runtime memory must fit within this 8 GB limit.

The quantization column shows the compression level used to shrink these models. Quantization formats like Q4_K_M, Q5_K_M, and Q6_K reduce the precision of model weights to save space. For example, Mochi 1 is a 10B model that fits into 7.3 GB of video memory using a Q4_K_M quantization. Gemma 2 9B fits exactly at the limit using 8 GB of video memory with the same Q4_K_M quantization.

Models in the 8B and 9B range can run at higher precision levels. Llama 3.1 8B uses 7.4 GB of video memory at Q5_K_M quantization. Granite 3.3 8B, Ministral 8B, and InternLM 3 8B can utilize a higher quality Q6_K quantization which uses 7.9 GB of video memory. Popular 7B models like Mistral 7B and Qwen2.5 7B also run comfortably at Q6_K quantization using 7.4 GB and 6.9 GB of video memory respectively.

When a model exceeds your 8 GB of video memory, you must offload parts of it to your system RAM. This offloading process allows you to run larger models but slows down generation speeds significantly. For these setups, we assume a system with 32 GB of system RAM. Under this configuration, Gemma 3 12B and Mistral NeMo 12B require 8.8 GB at Q4_K_M quantization and need 10.8 GB of system RAM to function.

Other large models require similar offloading strategies. FLUX.1 dev requires 14.4 GB at FP8 or optimized settings and needs 16.4 GB of system RAM. Vicuna 13B requires 9.5 GB at Q4_K_M quantization and needs 11.5 GB of system RAM. Running these models will work but you will experience much lower tokens per second compared to models that fit entirely on your graphics card.

You must also consider the context window when planning your memory usage. The memory figures listed here are calculated using a standard 4k context window. If you increase the context length to process longer documents or chat histories, the memory requirement will grow. This extra memory usage might push a model that normally fits on your card into system RAM offloading.