Best local AI models for AMD RX 6700S

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The AMD Radeon RX 6700S is a mobile graphics card equipped with 8 GB of GDDR6 memory. This dedicated video memory determines the maximum size of the artificial intelligence models you can run entirely on the hardware. When a model fits completely within this limit, it runs at maximum speed because the GPU can access the model weights directly through its high speed memory bus.

To fit larger models into the 8 GB limit, developers use quantization. The quant column shows the compression level applied to each model. For example, a Q4_K_M quant uses approximately four bits per parameter, while a Q6_K quant uses approximately six bits per parameter. Higher quantization levels like Q6_K preserve more original model accuracy but require more memory. Lower levels like Q4_K_M reduce the memory footprint to let larger models fit.

For models that fit entirely on the GPU, you can run Mochi 1 10B at Q4_K_M using 7.3 GB of memory. Gemma 2 9B fits at Q4_K_M using exactly 8 GB of memory. Several 9B models fit at the Q5_K_M quantization level using 7.7 GB of memory, including Nemotron Nano 4B / 9B, GLM-4 9B / GLM-4.5-Air, Yi-Coder 1.5B / 9B, GLM-4-9B-Chat / CodeGeeX4, and GLM-4V-9B / GLM-4.1V-Thinking. Chroma 8.9B fits at Q5_K_M using 7.6 GB of memory.

You can also run 8B models at the Q6_K quantization level using 7.9 GB of memory. This group includes Granite 3.3 2B / 8B, Ministral 3B / 8B, InternLM 3 8B, OpenCoder 1.5B / 8B, Seed-Coder 8B, MiniCPM-V 2.6 / MiniCPM-o 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large / Turbo. Llama 3.1 8B fits at Q5_K_M using 7.4 GB of memory. EXAONE 3.5 2.4B / 7.8B fits at Q6_K using 7.7 GB of memory. Popular 7B models like Mistral 7B fit at Q6_K using 7.4 GB of memory, while Qwen2.5 0.5B / 1.5B / 3B / 7B, OLMo 2 1B / 7B, Falcon 3 1B / 3B / 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, and Codestral Mamba 7B fit at Q6_K using 6.9 GB of memory.

When a model exceeds the 8 GB video memory limit, you must offload the remaining data to your system RAM. This process requires a system with 32 GB of system RAM. Offloading allows you to run larger models, but it costs significant processing speed because system RAM is much slower than video memory. For example, FLUX.1 dev at FP8 requires 14.4 GB of memory and needs 16.4 GB of system RAM. Open-Sora 2.0 at Q4_K_M requires 8.1 GB of memory and needs 10.1 GB of system RAM.

Other offload options include Gemma 3 12B, Gemma 4 12B, Mistral NeMo 12B, Pixtral 12B, FLUX.1 schnell, FLUX.1 Kontext dev, and FLUX.1 Krea dev. These models require 8.8 GB of memory at Q4_K_M and need 10.8 GB of system RAM. Vicuna 13B at Q4_K_M requires 9.5 GB of memory and needs 11.5 GB of system RAM.

All memory calculations are based on a standard 4k context window. The context window is the amount of text the model can remember during a conversation. If you increase the context window beyond 4k tokens, the memory usage will increase. This extra memory usage might force a model that normally fits on the GPU to require CPU offloading instead.