Best local AI models for NVIDIA Quadro M6000 24GB

The largest models that fit fully

The 30 largest of the 173 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA Quadro M6000 24GB graphics card features 24 GB of GDDR5 memory. This onboard memory determines the size of the artificial intelligence models you can run locally. To run a model entirely on the graphics card, the model files and the active memory space must fit within this 24 GB limit. Keeping the model on the card ensures the fastest processing speeds.

The quantization column shows the compression level used for each model. Quantization reduces the size of a model so it fits into smaller memory spaces. For example, the Qwen3 32B, Qwen3.5 32B, Aya Expanse 32B, Granite 4.0 32B, and Qwen2.5-Coder 32B models fit on this card using the Q4_K_M quantization, which uses 23.4 GB of memory. Other models like the Qwen3-30B-A3B and Qwen3-Coder 30B-A3B use 22 GB with the same Q4_K_M quantization.

Higher precision quantizations are available for slightly smaller models. The Gemma 3 27B, Gemma 3 27B vision, and Wan 2.2 / 2.5 models run at the Q5_K_M quantization level using 23 GB of memory. The Gemma 4 26B-A4B and Gemma 4 models fit at Q5_K_M using 22.2 GB. The Aria model fits at Q5_K_M using 21.3 GB. Mistral Small 3.2, Magistral Small, and Devstral Small 1.1 run at the Q6_K quantization level using 23.6 GB of memory. Solar Pro and Codestral 22B run at Q6_K using 21.6 GB.

You can also run models at the high Q8_0 quantization level. The HunyuanImage 2.1 / 3.0 models use 21.6 GB of memory at Q8_0. Ling-Coder-Lite uses 21.4 GB at Q8_0. DeepSeek-Coder-V2 16B and Kimi-VL A3B use 20.4 GB at Q8_0. Apriel-1.5-15B-Thinker and StarCoder2 15B use 19.1 GB at Q8_0. Qwen2.5 14B uses 19.5 GB at Q8_0. Other models like gpt-oss-20b, Reka Flash 3, Qwen-Image, Qwen-Image-Edit, and CogVLM2 fit within the 24 GB limit using Q6_K quantization.

When a model is too large for the 24 GB graphics memory, you can offload parts of it to your system RAM. This offloading process allows you to run larger models but slows down the processing speed. For example, running DeepSeek-Coder 33B or WizardCoder 33B requires 24.2 GB at Q4_K_M and 26.2 GB of system RAM. Running Yi 1.5 34B, Granite Code 34B, LLaVA 1.5 / 1.6 34B, or Ovis 2 requires 24.9 GB at Q4_K_M and 26.9 GB of system RAM.

Other large offload options include OTel 2.0 LLM 31B IT, which requires 27.5 GB at Q4_K_M and 29.5 GB of system RAM. Qwen3.6-35B-A3B and Command R 35B require 25.6 GB at Q4_K_M and 27.6 GB of system RAM. Seed-OSS 36B requires 26.4 GB at Q4_K_M and 28.4 GB of system RAM. All memory calculations assume a standard 4k context window. Running longer context windows will require more memory and may force you to use lower quantizations.