Best local AI models for NVIDIA RTX 2070 SUPER MAX-Q

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX 2070 SUPER MAX-Q is a mobile graphics card equipped with 8 GB of GDDR6 VRAM. This memory capacity determines the size of the artificial intelligence models you can run locally. To run a model entirely on your graphics hardware, the model files and its working memory must fit within this 8 GB limit. Running models locally ensures your data remains private and does not require an active internet connection.

Model sizes are measured in billions of parameters. To fit these models into your VRAM, developers use quantization to compress the files. The quant column shows the specific compression level used for each model. For example, Gemma 2 9B fits at the Q4_K_M quantization level which uses 8 GB of VRAM. Smaller models like Mistral 7B can run at a higher quality Q6_K quantization level while using only 7.4 GB of VRAM.

Many capable models fit completely inside your 8 GB VRAM limit. You can run Llama 3.1 8B at Q5_K_M quantization using 7.4 GB of VRAM. You can also run Granite 3.3 2B / 8B, Ministral 3B / 8B, and InternLM 3 8B at Q6_K quantization using 7.9 GB of VRAM. For vision tasks, MiniCPM-V 2.6 / MiniCPM-o 2.6 fits at Q6_K quantization using 7.9 GB of VRAM. Image generation models like Stable Diffusion 3.5 Large / Turbo also run at Q6_K quantization using 7.9 GB of VRAM.

If a model is too large for your 8 GB VRAM, you can use CPU offloading. This process splits the model between your graphics card and your system memory. CPU offloading allows you to run larger models but it reduces your processing speed. For this setup, we assume your computer has 32 GB of system RAM. Under these conditions, you can run Gemma 3 12B or Mistral NeMo 12B at Q4_K_M quantization which needs 8.8 GB of VRAM and 10.8 GB of system RAM.

Other large models can also run using CPU offloading. FLUX.1 dev needs 14.4 GB at FP8 / optimized quantization and uses 16.4 GB of system RAM. You can run FLUX.1 schnell at Q4_K_M quantization which needs 8.8 GB of VRAM and 10.8 GB of system RAM. When using any of these models, remember that active memory usage increases as you type. The memory figures listed here are measured at a standard 4k context window, so longer conversations will require more memory.