Best local AI models for AMD FirePro W9000

The largest models that fit fully

The 30 largest of the 114 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The AMD FirePro W9000 is an enterprise workstation graphics card equipped with 6 GB of GDDR5 memory. This onboard memory capacity determines the maximum size of the artificial intelligence models you can run entirely on the hardware. When a model fits completely within this 6 GB limit, the graphics processor handles all calculations. This local execution ensures the fastest possible generation speeds without relying on external cloud servers.

To fit modern models into this memory footprint, you must use quantized versions. Quantization is a compression technique that reduces the precision of model weights. The quant column indicates the specific level of compression applied to the model. For example, the Q4_K_M quant represents a medium four bit quantization, while the Q5_K_M quant represents a five bit quantization. Higher quantization levels preserve more model intelligence but require more memory.

Several high quality models fit entirely within the 6 GB memory of the card. You can run the 8B versions of Granite 3.3, Ministral, InternLM 3, OpenCoder, Seed-Coder, MiniCPM-V 2.6, Idefics 3, Fuyu-8B, Emu3, and Stable Diffusion 3.5 at the Q4_K_M quant, which uses 5.9 GB of memory. The 7.8B version of EXAONE 3.5 and the 7B version of Mistral also fit at Q4_K_M, consuming 5.7 GB of memory. Magicoder-S-DS 6.7B fits at Q5_K_M while using 5.7 GB of memory.

Other 7B models can run at the Q5_K_M quant by utilizing exactly 6 GB of memory. These models include Qwen2.5, OLMo 2, Falcon 3, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, Codestral Mamba 7B, CodeGemma, aiXcoder-7B, Nxcode, Janus-Pro, Ruyi-Mini-7B, Qwen2-Audio, Qwen2.5-Omni, and YuE. Running these models at 6 GB leaves no spare room in the onboard memory.

If a model exceeds the 6 GB onboard memory, you can offload the remaining parts to your system RAM. This setup assumes you have 32 GB of system RAM available. For example, Llama 3.1 8B requires 6.4 GB at Q4_K_M, which uses 8.4 GB of system RAM for offloading. Chroma requires 6.5 GB at Q4_K_M and uses 8.5 GB of system RAM. Gemma 2 9B requires 8 GB at Q4_K_M and uses 10 GB of system RAM. Offloading allows you to run larger models like Mochi 1 or Open-Sora 2.0, but it reduces processing speed because system RAM is slower than GDDR5 memory.

Other offload options include Nemotron Nano 9B, GLM-4 9B, Yi-Coder 9B, GLM-4-9B-Chat, and GLM-4V-9B. These models require 6.6 GB at Q4_K_M and use 8.6 GB of system RAM. All memory calculations for these models are based on a standard 4k context window. If you increase the context window to process longer documents, the memory requirements will rise, which may force you to use lower quants or offload more data to system RAM.