Best local AI models for NVIDIA RTX 3070

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX 3070 graphics card features 8 GB of GDDR6 video memory. This onboard memory determines the size of the artificial intelligence models you can run locally. To run a model entirely on your graphics hardware, the model files and the active workspace must fit within this 8 GB limit. Keeping the entire model on your video memory ensures the fastest possible processing speeds.

The quantization column shows the compression level used to shrink these models. Quantization reduces the precision of model weights to save space. For example, a Q4_K_M quant uses a four bit format, while a Q6_K quant uses a six bit format. These formats allow larger models to fit into your video memory. A Q4_K_M quant is highly compressed, while a Q6_K quant offers better output quality at the cost of more memory usage.

Several large models can run entirely on your card using specific compression levels. Mochi 1 is a 10B model that fits at Q4_K_M quant using 7.3 GB of memory. Gemma 2 9B fits at Q4_K_M quant using exactly 8 GB of memory. Other models like Nemotron Nano 9B, GLM-4 9B, GLM-4.5-Air, Yi-Coder 9B, GLM-4-9B-Chat, CodeGeeX4, GLM-4V-9B, and GLM-4.1V-Thinking fit at Q5_K_M quant using 7.7 GB of memory. Chroma is an 8.9B model that fits at Q5_K_M quant using 7.6 GB of memory. Llama 3.1 8B fits at Q5_K_M quant using 7.4 GB of memory.

You can also run 8B models using a higher quality Q6_K quant which uses 7.9 GB of memory. This group includes Granite 3.3 8B, Ministral 8B, InternLM 3 8B, OpenCoder 8B, Seed-Coder 8B, MiniCPM-V 2.6, MiniCPM-o 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large or Turbo. EXAONE 3.5 7.8B fits at Q6_K quant using 7.7 GB of memory. Mistral 7B fits at Q6_K quant using 7.4 GB of memory. Models like Qwen2.5 7B, OLMo 2 7B, Falcon 3 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, and Codestral Mamba 7B fit at Q6_K quant using 6.9 GB of memory.

If a model exceeds your 8 GB video memory, you must offload parts of it to your system RAM. This process requires a system with 32 GB of system RAM. Offloading allows you to run larger models, but it slows down processing speeds significantly because system RAM is much slower than GDDR6 video memory. For example, Open-Sora 2.0 is an 11B model that needs 8.1 GB at Q4_K_M quant and requires 10.1 GB of system RAM. FLUX.1 dev is a 12B model that needs 14.4 GB at FP8 or optimized settings and requires 16.4 GB of system RAM.

Other offload options include Gemma 3 12B, Gemma 4 12B, Mistral NeMo 12B, Pixtral 12B, FLUX.1 schnell, FLUX.1 Kontext dev, and FLUX.1 Krea dev. These 12B models need 8.8 GB at Q4_K_M quant and require 10.8 GB of system RAM. Vicuna 13B is a 13B model that needs 9.5 GB at Q4_K_M quant and requires 11.5 GB of system RAM. When planning your memory usage, remember that these calculations assume a standard 4k context window. Increasing the context window length will require more memory and may force you to use higher compression or more system RAM offloading.