Best local AI models for NVIDIA Quadro M6000

The largest models that fit fully

The 30 largest of the 147 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA Quadro M6000 is equipped with 12 GB GDDR5 memory. This onboard memory capacity determines the maximum size of the artificial intelligence models you can run locally. To fit a model entirely within this hardware limit, the model files must be compressed. This compression process is called quantization, which reduces the precision of the model weights to save space.

The quantization column indicates the optimal balance between model size and output quality. For example, the DeepSeek-Coder-V2 16B model fits within 11.7 GB of memory when using the Q4_K_M quantization level. Similarly, the Kimi-VL A3B model uses 11.7 GB at Q4_K_M. Other models like the Apriel-1.5-15B-Thinker and StarCoder2 15B require 11 GB of memory at the same Q4_K_M level. When a model fits completely inside the 12 GB GDDR5 memory, you get the fastest possible processing speeds.

For slightly smaller models, you can use higher precision quantizations. The Phi-4 model and the Phi-4-reasoning / -plus model are 14B parameters and use 11.9 GB of memory at the Q5_K_M level. The Wan 2.2 T2I and Wan 2.1 (1.3B / 14B) models also use 11.9 GB at Q5_K_M. If you select a 12B model like Gemma 3 12B, Gemma 4 12B, Mistral NeMo 12B, or Pixtral 12B, you can use the higher quality Q6_K quantization which uses 11.8 GB of memory.

You can run larger models by offloading a portion of the workload to your system RAM. This approach requires a system with at least 32 GB of system RAM. For example, the FLUX.1 dev model requires 14.4 GB at FP8 / optimized, which uses 16.4 GB of system RAM. The Solar Pro 22B and Codestral 22B models need 16.1 GB at Q4_K_M, which uses 18.1 GB of system RAM. Offloading allows you to run these larger models, but it significantly reduces the generation speed because system RAM is much slower than GDDR5 memory.

When running these models, you must monitor your context window usage. The memory figures listed are calculated using a standard 4k context window. If you increase the context length to process longer documents or conversations, the memory usage will rise. This extra memory demand can exceed your 12 GB limit and cause the system to slow down or fail.