Best local AI models for AMD RX 6700 XT

The largest models that fit fully

The 30 largest of the 147 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The AMD RX 6700 XT graphics card features 12 GB of GDDR6 onboard memory. This memory size determines which artificial intelligence models can run entirely on your hardware. For local execution, the model weights must fit inside this video memory to ensure fast processing speeds. If a model exceeds this limit, your system must use alternative memory management strategies.

The quantization column shows the optimal compression level for each model. Quantization reduces the precision of model weights to save space. A Q4_K_M quant represents a medium four bit quantization that balances size and accuracy. A Q5_K_M quant uses five bits, while Q6_K and Q8_0 quants offer higher precision at the cost of larger memory footprints. Choosing the right quant allows you to run larger architectures within your hardware limits.

Several high performance models fit completely within the 12 GB boundary of your card. The DeepSeek-Coder-V2 16B and Kimi-VL A3B models run at Q4_K_M quality using 11.7 GB of video memory. The Apriel-1.5-15B-Thinker and StarCoder2 15B models fit at Q4_K_M quality using 11 GB. For higher precision, you can run the Mistral NeMo 12B or FLUX.1 schnell at Q6_K quality using 11.8 GB of memory.

When a model is too large for the graphics card, you can use CPU offload. This process splits the workload between your graphics card and your system RAM. For example, running FLUX.1 dev at FP8 requires 14.4 GB of memory, which uses 16.4 GB of system RAM alongside your GPU. Larger models like Codestral 22B require 16.1 GB at Q4_K_M quality, which utilizes 18.1 GB of system RAM.

CPU offload allows you to run advanced models like Solar Pro or CogVLM2, but it comes with a performance cost. Transferring data between system RAM and video memory is much slower than running everything on the graphics card. Your generation speeds will drop significantly when offloading. You should use offloading only when model size is more important than speed.

Memory calculations are typically based on a standard 4k context window. As your conversation grows longer, the context window consumes additional video memory. Running a model very close to the 12 GB limit of your AMD RX 6700 XT may cause out of memory errors during long sessions. You should select a slightly smaller model or a lower quantization level if you need to process very long texts.