Best local AI models for NVIDIA RTX 5060 Ti 8GB

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX 5060 Ti graphics card comes equipped with 8 GB of GDDR7 memory. This dedicated video memory determines the maximum size of the artificial intelligence models you can run locally. To fit a model entirely within this 8 GB limit, the model parameters must be compressed. This compression process is called quantization, which reduces the precision of the model weights to save space.

The quantization column shows the best format to balance speed and accuracy on this hardware. For example, Gemma 2 9B fits at the Q4_K_M quantization level and uses exactly 8 GB of video memory. Models like Llama 3.1 8B fit at the Q5_K_M quantization level while using 7.4 GB of memory. Smaller models like Mistral 7B can run at the higher quality Q6_K quantization level, which uses 7.4 GB of memory.

Many other models run fully on the graphics card. You can run Mochi 1 at Q4_K_M using 7.3 GB of memory. Nemotron Nano 4B / 9B, GLM-4 9B / GLM-4.5-Air, Yi-Coder 1.5B / 9B, GLM-4-9B-Chat / CodeGeeX4, and GLM-4V-9B / GLM-4.1V-Thinking all fit at Q5_K_M using 7.7 GB. Chroma fits at Q5_K_M using 7.6 GB. Granite 3.3 2B / 8B, Ministral 3B / 8B, InternLM 3 8B, OpenCoder 1.5B / 8B, Seed-Coder 8B, MiniCPM-V 2.6 / MiniCPM-o 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large / Turbo all run at Q6_K using 7.9 GB.

You can also run EXAONE 3.5 2.4B / 7.8B at Q6_K using 7.7 GB. Several 7B models run at Q6_K using 6.9 GB of video memory. These include Qwen2.5 0.5B / 1.5B / 3B / 7B, OLMo 2 1B / 7B, Falcon 3 1B / 3B / 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, and Codestral Mamba 7B.

When a model is too large for the 8 GB video memory, you must use CPU offloading. This process splits the model between your graphics card and your system RAM. Offloading allows you to run larger models but reduces your processing speed. For these cases, we assume your computer has 32 GB of system RAM.

With CPU offloading, you can run FLUX.1 dev at FP8 / optimized, which needs 14.4 GB of video memory and 16.4 GB of system RAM. You can run Vicuna 13B at Q4_K_M, which needs 9.5 GB of video memory and 11.5 GB of system RAM. Open-Sora 2.0 needs 8.1 GB of video memory and 10.1 GB of system RAM at Q4_K_M. Gemma 3 12B, Gemma 4 12B, Mistral NeMo 12B, Pixtral 12B, FLUX.1 schnell, FLUX.1 Kontext dev, and FLUX.1 Krea dev all need 8.8 GB of video memory and 10.8 GB of system RAM at Q4_K_M.

Memory usage calculations assume a standard 4k context window. If you increase the context window to process longer texts, the model will require more memory. This extra memory demand might force you to use a lower quantization level or rely more on CPU offloading.