Best local AI models for AMD Pro W6600M

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The AMD Radeon Pro W6600M is a professional mobile workstation graphics card equipped with 8 GB of GDDR6 dedicated video memory. This onboard memory pool determines the maximum size of the artificial intelligence models you can run entirely on the hardware. Keeping a model fully inside the graphics memory ensures the fastest processing speeds for text generation and image creation tasks.

To fit larger models into the 8 GB limit you must use quantized versions. Quantization is a compression technique that reduces the precision of model weights. The quant column shows the optimal format for each model. For example a Q4_K_M quant uses a four bit format while a Q6_K quant uses a six bit format. Higher quantization levels preserve more of the original model accuracy but require more memory space.

Several highly capable models fit completely within the graphics memory of this card. You can run Gemma 2 9B at the Q4_K_M quantization level which uses exactly 8 GB of memory. The Llama 3.1 8B model fits comfortably at the Q5_K_M quantization level using 7.4 GB. For image generation Mochi 1 10B can run locally using 7.3 GB of memory at the Q4_K_M quantization level.

If a model exceeds the 8 GB graphics memory limit you can offload parts of it to your system RAM. This process requires a system with at least 32 GB of system RAM. Offloading allows you to run larger models like Mistral NeMo 12B which needs 8.8 GB of memory at Q4_K_M and 10.8 GB of system RAM. You can also run FLUX.1 dev at FP8 which needs 14.4 GB of memory and 16.4 GB of system RAM. Offloading makes these models run much slower because system RAM is slower than graphics memory.

When planning your local deployments you must also consider the context window. The memory figures listed here are calculated using a standard four thousand token context window. If you increase the context window to process longer documents the model will require significantly more memory. This extra memory demand may force you to use a lower quantization level or offload more layers to system RAM.