Best local AI models for AMD Pro 5700

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The AMD Radeon Pro 5700 workstation graphics card features 8 GB of GDDR6 dedicated video memory. This onboard VRAM determines the maximum size of the artificial intelligence models you can run entirely on the hardware. When a model fits completely within this 8 GB limit, execution is fast because the GPU can access the model weights directly at high bandwidth.

To fit larger models into the VRAM, developers use quantization. The quant column indicates the compression level applied to the model weights. For example, the Q4_K_M quant uses approximately four bits per parameter, while the Q6_K quant uses six bits. Higher quantization levels like Q6_K preserve more original model accuracy but require more memory. Lower quantization levels like Q4_K_M allow larger models to fit into your available hardware space.

With 8 GB of VRAM, you can run several highly capable models locally. The Mochi 1 10B model fits at the Q4_K_M quant using 7.3 GB of memory. Gemma 2 9B fits at Q4_K_M using exactly 8 GB. Several 9B models fit at the Q5_K_M quant using 7.7 GB, including Nemotron Nano 4B / 9B, GLM-4 9B / GLM-4.5-Air, Yi-Coder 1.5B / 9B, GLM-4-9B-Chat / CodeGeeX4, and GLM-4V-9B / GLM-4.1V-Thinking. Chroma 8.9B fits at Q5_K_M using 7.6 GB. Llama 3.1 8B fits at Q5_K_M using 7.4 GB.

Other 8B models can run at the higher quality Q6_K quant using 7.9 GB of VRAM. These include Granite 3.3 2B / 8B, Ministral 3B / 8B, InternLM 3 8B, OpenCoder 1.5B / 8B, Seed-Coder 8B, MiniCPM-V 2.6 / MiniCPM-o 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large / Turbo. EXAONE 3.5 2.4B / 7.8B fits at Q6_K using 7.7 GB. Mistral 7B fits at Q6_K using 7.4 GB. Many 7B models fit at Q6_K using 6.9 GB, including Qwen2.5 0.5B / 1.5B / 3B / 7B, OLMo 2 1B / 7B, Falcon 3 1B / 3B / 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, and Codestral Mamba 7B.

When a model is too large for the 8 GB VRAM, you can offload parts of it to your system RAM. This requires a system with at least 32 GB of system RAM. Offloading allows you to run larger models, but it costs performance because system RAM is much slower than GDDR6 VRAM. For example, Gemma 3 12B, Gemma 4 12B, Mistral NeMo 12B, Pixtral 12B, FLUX.1 schnell, FLUX.1 Kontext dev, and FLUX.1 Krea dev need 8.8 GB at Q4_K_M and 10.8 GB of system RAM. Open-Sora 2.0 11B needs 8.1 GB at Q4_K_M and 10.1 GB of system RAM. FLUX.1 dev 12B needs 14.4 GB at FP8 / optimized and 16.4 GB of system RAM. Vicuna 13B needs 9.5 GB at Q4_K_M and 11.5 GB of system RAM.

You must also consider the context window size when calculating memory usage. The memory figures listed here assume a standard 4k context window. If you increase the context window to process longer documents or chat histories, the key value cache will grow. This extra data requires more VRAM, which might force you to use a lower quantization level or offload more layers to system RAM to prevent out of memory errors.