Best local AI models for AMD RX 6800S

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The AMD Radeon RX 6800S is a mobile graphics card equipped with 8 GB of GDDR6 memory. This dedicated video memory determines the size of the artificial intelligence models you can run locally. To run a model entirely on the graphics hardware, the model files and its working memory must fit within this 8 GB limit. Keeping the model fully inside the video memory ensures the fastest processing speeds.

The quantization column shows the compression level used to fit these models into your hardware. Raw models are often too large for consumer graphics cards. Quantization reduces the precision of the model weights to save space. A Q4_K_M quant represents a four bit compression that balances size and quality. A Q6_K quant uses six bits which provides higher accuracy but requires more memory.

Several high quality models fit directly into the 8 GB video memory of the RX 6800S. The Mochi 1 10B model fits at a Q4_K_M quantization using 7.3 GB of memory. Gemma 2 9B fits at Q4_K_M using exactly 8 GB. You can also run Nemotron Nano 9B, GLM-4 9B, Yi-Coder 9B, GLM-4-9B-Chat, and GLM-4V-9B at Q5_K_M quantization using 7.7 GB of memory. Chroma 8.9B fits at Q5_K_M using 7.6 GB. Llama 3.1 8B fits at Q5_K_M using 7.4 GB.

Other models can run at the higher Q6_K quantization level. Granite 3.3 8B, Ministral 8B, InternLM 3 8B, OpenCoder 8B, Seed-Coder 8B, MiniCPM-V 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large use 7.9 GB of memory. EXAONE 3.5 7.8B uses 7.7 GB. Mistral 7B uses 7.4 GB. Qwen2.5 7B, OLMo 2 7B, Falcon 3 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, and Codestral Mamba 7B all use 6.9 GB of memory at Q6_K.

When a model is too large for the 8 GB video memory you can use CPU offload. This process splits the model between your graphics card and your system RAM. CPU offload allows you to run larger models but it slows down the processing speed significantly. For these cases we assume your computer has 32 GB of system RAM to handle the overflow.

Using CPU offload you can run Open-Sora 2.0 11B which needs 8.1 GB at Q4_K_M and 10.1 GB of system RAM. FLUX.1 dev 12B needs 14.4 GB at FP8 and 16.4 GB of system RAM. Gemma 3 12B, Gemma 4 12B, Mistral NeMo 12B, Pixtral 12B, FLUX.1 schnell, FLUX.1 Kontext dev, and FLUX.1 Krea dev need 8.8 GB at Q4_K_M and 10.8 GB of system RAM. Vicuna 13B needs 9.5 GB at Q4_K_M and 11.5 GB of system RAM.

You must also consider the context limit of your models. The memory numbers listed here are calculated using a standard 4k context window. If you increase the context window to process longer documents the memory usage will rise. Running larger context windows might require you to use a lower quantization level or force the system to use CPU offload.