Best local AI models for NVIDIA GTX 970

The largest models that fit fully

The 30 largest of the 81 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA GTX 970 graphics card features 4 GB of GDDR5 memory. This memory size determines which local AI models can run entirely on your hardware. To fit within this limit, models must use quantization. The quant column shows the compression level used to reduce model size. A Q4_K_M or Q6_K quant reduces the memory footprint so the model can load without exceeding your video memory.

Several high quality models fit directly into the 4 GB memory of the GTX 970. For image generation, Lumina-Next or Lumina-Image 2.0 at 5B parameters runs at the Q4_K_M quant using 3.7 GB of memory. CogVideoX 2B or 5B also fits at 5B parameters using the Q4_K_M quant with 3.7 GB used. For vision tasks, DeepSeek-VL2 at 4.5B parameters fits at Q5_K_M using 3.8 GB of memory, while Phi-3.5-vision at 4.2B parameters uses 3.6 GB at the Q5_K_M quant.

Text generation models also fit well. Qwen3 4B, Gemma 3 4B, Gemma 4 E4B, MiniCPM 3 4B, and Danube 3 4B all run at the Q6_K quant using 3.9 GB of memory. Phi-4-mini-instruct and Phi-3.5 Mini at 3.8B parameters use 3.7 GB of memory at the Q6_K quant. For audio tasks, Fish Speech 1.5 or OpenAudio S1 at 4B parameters fits at Q6_K using 3.9 GB of memory. You can also run Orpheus TTS or Higgs Audio v2 at 3B parameters using the Q8_0 quant with 3.8 GB used.

When a model is too large for the 4 GB video memory, you can use CPU offload. This method shares the workload between your graphics card and your system RAM. We assume your system has 32 GB of system RAM for these setups. Offloading allows you to run larger models, but it costs performance. The processing speed drops because data must travel between the system RAM and the graphics card memory.

Using CPU offload, you can run Mistral 7B at the Q4_K_M quant. It needs 5.7 GB of memory and uses 7.7 GB of system RAM. Qwen2.5 0.5B / 1.5B / 3B / 7B at 7B parameters needs 5.1 GB at Q4_K_M and uses 7.1 GB of system RAM. OLMo 2 1B / 7B, Falcon 3 1B / 3B / 7B, Command R7B, and OpenHermes 2.5 also run at 7B parameters using 5.1 GB at Q4_K_M with 7.1 GB of system RAM. For image generation, Stable Diffusion XL at 3.417B parameters needs 4.1 GB at FP8 or optimized settings and uses 6.1 GB of system RAM.

You must consider the 4k context caveat when running these models. The memory numbers listed only cover the model itself at startup. As you type prompts and the model generates responses, the context memory grows. Running a model close to the 4 GB limit of the GTX 970 leaves very little room for long conversations. Keeping your context length short prevents the system from running out of memory during generation.