Best local AI models for AMD RX 5600 XT

The largest models that fit fully

The 30 largest of the 114 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The AMD Radeon RX 5600 XT graphics card features 6 GB of GDDR6 video memory. This memory size determines which artificial intelligence models can run entirely on your hardware. For local execution, the model files must fit inside this video memory to run at full speed. When a model exceeds this limit, your system must use slower system memory to process the remaining data.

To fit larger models into the 6 GB limit, we use quantized versions. The quant column shows the specific compression level used for each model. For example, the Q4_K_M and Q5_K_M quants compress the model weights to four or five bits. This compression reduces the memory footprint while keeping most of the model accuracy. A Q4_K_M quant of the Granite 3.3 8B model uses 5.9 GB of video memory, which fits safely inside your hardware limit.

Several 8B and 7B models can run entirely on your video card. The Ministral 8B, InternLM 3 8B, OpenCoder 8B, Seed-Coder 8B, MiniCPM-V 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large models fit at Q4_K_M quant using 5.9 GB of memory. The EXAONE 3.5 7.8B and Mistral 7B models fit at Q4_K_M quant using 5.7 GB of memory. The Magicoder-S-DS 6.7B model also fits at Q5_K_M quant using 5.7 GB of memory.

Other models can run at the Q5_K_M quant using exactly 6 GB of video memory. These models include Qwen2.5 7B, OLMo 2 7B, Falcon 3 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, Codestral Mamba 7B, CodeGemma 7B, aiXcoder-7B, Nxcode, CodeQwen 1.5 7B, Janus-Pro 7B, Ruyi-Mini-7B, Qwen2-Audio 7B, Qwen2.5-Omni 7B, and YuE. Running these models at 6 GB leaves no extra video memory for other tasks.

If you want to run larger models, you must use CPU offload. This method splits the model between your video memory and your system RAM. For these cases, we assume your computer has 32 GB of system RAM. For example, Llama 3.1 8B needs 6.4 GB at Q4_K_M quant and requires 8.4 GB of system RAM. Chroma needs 6.5 GB at Q4_K_M quant and requires 8.5 GB of system RAM. Gemma 2 9B needs 8 GB at Q4_K_M quant and requires 10 GB of system RAM.

Other offload options include Nemotron Nano 9B, GLM-4 9B, Yi-Coder 9B, GLM-4-9B-Chat, and GLM-4V-9B. These models need 6.6 GB at Q4_K_M quant and require 8.6 GB of system RAM. Mochi 1 needs 7.3 GB at Q4_K_M quant and requires 9.3 GB of system RAM. Open-Sora 2.0 needs 8.1 GB at Q4_K_M quant and requires 10.1 GB of system RAM. Offloading models slows down the generation speed because system RAM is slower than video memory.

All memory calculations in this guide use a standard 4k context window. The context window is the amount of text the model can read and write at one time. If you increase the context window beyond 4k, the model will require more memory. This extra memory usage can cause a model that fits at 4k context to exceed your 6 GB video memory limit.