Best local AI models for NVIDIA RTX 2070

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX 2070 graphics card features 8 GB of GDDR6 video memory. This onboard memory determines which artificial intelligence models you can run entirely on your hardware. When a model fits completely within this VRAM limit, you get the fastest possible generation speeds. If a model exceeds this limit, your system must transfer data between your graphics card and system memory.

To fit larger models into the 8 GB limit, developers use quantization. The quant column shows the compression level applied to each model. For example, the Mochi 1 10B model fits at a Q4_K_M quantization which uses 7.3 GB of VRAM. Gemma 2 9B fits at Q4_K_M using exactly 8 GB of VRAM. Other models like Llama 3.1 8B can run at a higher quality Q5_K_M quantization using 7.4 GB of VRAM.

Many popular models can run at the high quality Q6_K quantization on this hardware. Granite 3.3 8B, Ministral 8B, InternLM 3 8B, and OpenCoder 8B all use 7.9 GB of VRAM at this level. You can also run vision models like MiniCPM-V 2.6 and image generators like Stable Diffusion 3.5 Large at Q6_K using 7.9 GB of VRAM. Standard 7B models like Mistral 7B use 7.4 GB of VRAM at Q6_K, while Qwen2.5 7B and Falcon 3 7B use 6.9 GB of VRAM at Q6_K.

When a model requires more than 8 GB of memory, you must use CPU offloading. This process splits the model weights between your graphics card and system RAM. For this setup, we assume your computer has 32 GB of system RAM. Offloading allows you to run larger models, but it significantly reduces processing speed because system RAM is much slower than GDDR6 video memory.

Several larger models are accessible through CPU offloading. Gemma 3 12B, Gemma 4 12B, Mistral NeMo 12B, and Pixtral 12B require 8.8 GB of memory at Q4_K_M, which needs 10.8 GB of system RAM. Image generators like FLUX.1 schnell also require 8.8 GB at Q4_K_M with 10.8 GB of system RAM. The FLUX.1 dev model requires 14.4 GB at FP8 or optimized settings, which needs 16.4 GB of system RAM. Vicuna 13B requires 9.5 GB at Q4_K_M, which needs 11.5 GB of system RAM.

VRAM usage calculations assume a standard 4k context window. As you input longer prompts or generate longer responses, the context window expands and consumes additional video memory. If you run a model that sits close to the 8 GB limit, a long conversation might exceed your VRAM and trigger slow system RAM offloading automatically.