Best local AI models for AMD RX 570X

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The AMD Radeon RX 570X graphics card features 8 GB of GDDR5 onboard memory. This dedicated memory determines the size of the artificial intelligence models you can run directly on the hardware. To run a model entirely on the graphics card, the model files and the active memory space must fit within this 8 GB limit. Keeping the entire model inside the graphics memory ensures the fastest possible processing speeds.

The quantization column shows the specific compression level used for each model. Quantization reduces the size of a model so it fits into smaller memory spaces. For example, Gemma 2 9B fits in 8 GB of memory when using the Q4_K_M quantization. Other models like Llama 3.1 8B fit in 7.4 GB of memory when using the Q5_K_M quantization. Smaller models like Mistral 7B can use a higher quality Q6_K quantization and consume 7.4 GB of memory.

If a model is too large for the 8 GB graphics memory, you must use CPU offload. This technique splits the model between your graphics card and your system RAM. We assume a standard system RAM size of 32 GB for these scenarios. For example, running the 12B Mistral NeMo requires 8.8 GB of graphics memory at Q4_K_M quantization and needs an additional 10.8 GB of system RAM. CPU offload allows you to run larger models like Vicuna 13B, but it significantly slows down generation speeds because system RAM is much slower than GDDR5 graphics memory.

When selecting your model, you must also consider the context window. The memory figures listed here are calculated using a baseline 4k context window. If you increase the context window to process longer documents or longer conversations, the model will require more memory. This extra memory usage might push a model that normally fits in 8 GB over the limit, forcing your system to use slower CPU offloading.

Many capable models fit completely within the 8 GB limit of your card. You can run Granite 3.3 8B, Ministral 8B, and InternLM 3 8B at Q6_K quantization using 7.9 GB of memory. Image generation models like Stable Diffusion 3.5 Large also fit within 7.9 GB of memory at Q6_K quantization. For even lighter workloads, models like Qwen2.5 7B and Falcon 3 7B run at Q6_K quantization using only 6.9 GB of memory.