Best local AI models for AMD PRO W7500

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The AMD Radeon PRO W7500 workstation graphics card features 8 GB of GDDR6 dedicated video memory. This memory capacity determines the size of the artificial intelligence models you can run locally. To run a model entirely on the graphics card, the model files and the active working memory must fit within this 8 GB limit.

The quantization column shows the compression level used to fit these models into memory. Quantization reduces the precision of model weights to save space. For example, the Q4_K_M quantization allows the Mochi 1 10B model to run using 7.3 GB of video memory. The Gemma 2 9B model fits exactly at the limit using 8 GB of video memory with Q4_K_M quantization.

Models like Llama 3.1 8B fit comfortably with a higher quality Q5_K_M quantization using 7.4 GB of video memory. Smaller models like Mistral 7B and Qwen2.5 7B can run at Q6_K quantization. These Q6_K quants use 7.4 GB and 6.9 GB of video memory respectively. Higher quantization levels preserve more of the original model accuracy.

When a model is too large for the 8 GB video memory, you must use CPU offloading. This process splits the model between your graphics card and your system RAM. A system with 32 GB of system RAM can run larger models by accepting a performance cost. Offloading causes slower processing speeds because data must travel between the system RAM and the graphics card.

For example, running the FLUX.1 dev 12B model at FP8 requires 14.4 GB of memory, which uses all your video memory and 16.4 GB of system RAM. Similarly, the Mistral NeMo 12B model at Q4_K_M requires 8.8 GB of memory, which uses 10.8 GB of system RAM. This offloading allows you to run advanced models at the cost of generation speed.

All memory calculations assume a standard 4k context window. The context window is the amount of text the model can remember during a conversation. If you increase the context window beyond 4k tokens, the model will require significantly more video memory. This extra memory usage can cause a model that normally fits to exceed the 8 GB limit.