Best local AI models for NVIDIA Quadro M4000

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA Quadro M4000 is a workstation graphics card equipped with 8 GB of GDDR5 memory. When running AI models locally, this onboard memory size is the main limiting factor. The entire active model must fit inside this memory space to run at maximum speed. If a model exceeds this limit, the system must use alternative execution methods.

The quantization column shows the compression level used to shrink the model size. Quantization formats like Q4_K_M, Q5_K_M, and Q6_K reduce the precision of model weights. This allows larger models to fit into the 8 GB memory space of the Quadro M4000. Higher quantization numbers like Q6_K preserve more model accuracy but require more memory space than Q4_K_M.

For models that fit entirely within the 8 GB limit, several options exist. The largest fitting model is Mochi 1 at 10B using the Q4_K_M quant which uses 7.3 GB. Gemma 2 9B fits at Q4_K_M using 8 GB. Several 9B models like Nemotron Nano 4B / 9B, GLM-4 9B / GLM-4.5-Air, Yi-Coder 1.5B / 9B, GLM-4-9B-Chat / CodeGeeX4, and GLM-4V-9B / GLM-4.1V-Thinking fit at Q5_K_M using 7.7 GB. Chroma 8.9B fits at Q5_K_M using 7.6 GB. Llama 3.1 8B fits at Q5_K_M using 7.4 GB.

Other 8B models fit at the higher Q6_K quant using 7.9 GB of memory. These include Granite 3.3 2B / 8B, Ministral 3B / 8B, InternLM 3 8B, OpenCoder 1.5B / 8B, Seed-Coder 8B, MiniCPM-V 2.6 / MiniCPM-o 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large / Turbo. EXAONE 3.5 2.4B / 7.8B fits at Q6_K using 7.7 GB. Mistral 7B fits at Q6_K using 7.4 GB. Several 7B models like Qwen2.5 0.5B / 1.5B / 3B / 7B, OLMo 2 1B / 7B, Falcon 3 1B / 3B / 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, and Codestral Mamba 7B fit at Q6_K using 6.9 GB.

When a model is too large for the onboard memory, you must use CPU offloading. This process splits the model between the graphics card and your system RAM. Offloading allows you to run larger models but it costs significant processing speed. For these cases, we assume a system with 32 GB of system RAM. Open-Sora 2.0 at 11B needs 8.1 GB at Q4_K_M and 10.1 GB of system RAM. FLUX.1 dev at 12B needs 14.4 GB at FP8 / optimized and 16.4 GB of system RAM.

Other 12B models like Gemma 3 12B, Gemma 4 12B, Mistral NeMo 12B, Pixtral 12B, FLUX.1 schnell, FLUX.1 Kontext dev, and FLUX.1 Krea dev need 8.8 GB at Q4_K_M and 10.8 GB of system RAM. Vicuna 13B needs 9.5 GB at Q4_K_M and 11.5 GB of system RAM. Users must also remember the 4k context caveat. The memory figures listed are calculated using a standard 4k context window. Increasing the context window size will require more memory and may force a model to exceed the 8 GB limit.