Best local AI models for NVIDIA Quadro P6000

The largest models that fit fully

The 30 largest of the 173 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA Quadro P6000 is equipped with 24 GB of GDDR5X frame buffer memory. This dedicated onboard memory determines the maximum size of the artificial intelligence models you can run entirely on the graphics card. Keeping the entire model inside this video memory ensures the fastest possible processing speeds during local inference.

The quant column indicates the quantization level of each model. Quantization is a compression method that reduces the precision of model weights to save space. For example, the Q4_K_M quant represents a four bit medium compression level. This allows larger models like Qwen3 32B, Qwen3.5 32B, Aya Expanse 32B, Granite 4.0 32B, and Qwen2.5-Coder 32B to fit into 23.4 GB of video memory.

As model sizes decrease, you can run higher quality quantizations. The Q5_K_M quant fits models like Gemma 3 27B, Wan 2.2 27B, and Gemma 4 26B-A4B using around 23 GB of memory. The Q6_K quant allows Mistral Small 3.2 24B, Solar Pro 22B, and Codestral 22B to run at 23.6 GB and 21.6 GB. The highest fidelity Q8_0 quant is available for models like HunyuanImage 3.0 17B, DeepSeek-Coder-V2 16B, and Qwen2.5 14.7B.

When a model exceeds the 24 GB video memory limit, you must offload parts of it to your system RAM. This offloading process requires a system with at least 32 GB of system RAM. For instance, running the Yi 1.5 34B or Granite Code 34B model at Q4_K_M requires 24.9 GB of model space and 26.9 GB of system RAM. Offloading allows you to run larger architectures, but it introduces a significant performance cost because system RAM is much slower than GDDR5X.

You must also account for the context window when planning your memory usage. The memory figures listed here are calculated using a baseline 4k context window. If you increase the context length to process longer documents or chat histories, the key value cache will consume additional video memory. This extra memory demand can push a model that fits at 4k context over the 24 GB limit, forcing unexpected CPU offloading.