Best local AI models for AMD RX 6800M

The largest models that fit fully

The 30 largest of the 147 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The AMD Radeon RX 6800M is a mobile graphics processor equipped with 12 GB of GDDR6 memory. This physical memory limit determines which artificial intelligence models can run entirely on your hardware. For local execution, the model weights must fit inside this video memory space to ensure fast processing speeds. If a model exceeds this limit, the system must transfer data between your graphics card and system memory.

The quantization column indicates the compression level applied to each model. Quantization reduces the precision of model weights to save space. A Q4_K_M quant represents a four bit quantization level which offers a balanced mix of size and accuracy. Higher quants like Q5_K_M, Q6_K, and Q8_0 provide better output quality but require more memory. For example, the 14B parameter Phi-4 model fits within 11.9 GB of memory when using a Q5_K_M quant.

Models like DeepSeek-Coder-V2 16B and Kimi-VL A3B utilize 11.7 GB of memory at Q4_K_M. This leaves very little remaining space in your 12 GB video memory. When running these large models, you must consider the context window. Standard calculations assume a base 4k context window. Expanding your context window beyond this limit requires additional memory for the key value cache, which can cause out of memory errors on a 12 GB card.

When a model is too large for the 12 GB video memory, you can use CPU offloading. This technique splits the workload between your graphics card and your system RAM. For CPU offloading, we assume your computer has 32 GB of system RAM. Offloading allows you to run larger models, but it significantly reduces generation speed because system RAM is much slower than GDDR6 video memory.

Using CPU offloading, you can run the 12B parameter FLUX.1 dev model which needs 14.4 GB of space at FP8 and uses 16.4 GB of system RAM. You can also run the 22B parameter Codestral 22B model. This model requires 16.1 GB of space at Q4_K_M and utilizes 18.1 GB of system RAM. Other offload options include the 20B parameter Qwen-Image model and the 21B parameter Reka Flash 3 model.