Best local AI models for NVIDIA Quadro M2000

The largest models that fit fully

The 30 largest of the 81 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA Quadro M2000 is equipped with 4 GB of GDDR5 memory. This dedicated video memory determines the maximum size of the artificial intelligence models you can run entirely on the graphics card. To fit within this 4 GB limit, models must be compressed using quantization. The quantization column shows the specific compression level required to run each model. A lower quantization level like Q4_K_M reduces the model size further but slightly lowers output quality, while a higher level like Q8_0 preserves more original precision.

For models that fit completely inside the 4 GB video memory, execution is fast because the graphics processor has direct access to the weights. The largest fitting models include Lumina-Next or Lumina-Image 2.0 at 5B parameters using the Q4_K_M quant which consumes 3.7 GB of memory. Similarly, CogVideoX 2B or 5B fits at 5B parameters using the Q4_K_M quant with 3.7 GB used. DeepSeek-VL2 at 4.5B parameters fits using the Q5_K_M quant which uses 3.8 GB of video memory.

Other capable models can run entirely on the card with slightly higher quantization levels. Qwen3 4B, Gemma 3 4B, Gemma 4 E4B, MiniCPM 3 4B, Danube 3 4B, and Fish Speech 1.5 or OpenAudio S1 all run at 4B parameters using the Q6_K quant and consume 3.9 GB of memory. The Phi-4-mini-instruct and Phi-3.5 Mini models at 3.8B parameters use the Q6_K quant and require 3.7 GB. For audio and image tasks, SDXL Turbo at 3.5B parameters fits using the Q6_K quant with 3.4 GB used, while Stable Diffusion 3.5 Medium at 2.5B parameters fits using the Q8_0 quant with 3.2 GB used.

When a model is too large for the 4 GB video memory, you can use CPU offload. This technique splits the model weights between your graphics card and your system memory. We assume your computer has 32 GB of system RAM for these scenarios. Offloading allows you to run larger models, but it costs performance because transferring data between the system RAM and the graphics card is much slower than using dedicated GDDR5 memory directly.

Several popular models require CPU offload to run on this hardware. For example, Mistral 7B needs 5.7 GB of memory at the Q4_K_M quant and requires 7.7 GB of system RAM. Qwen2.5 0.5B or 1.5B or 3B or 7B at the 7B size needs 5.1 GB at the Q4_K_M quant and requires 7.1 GB of system RAM. The same memory and system RAM requirements apply to OLMo 2 1B or 7B at 7B, Falcon 3 1B or 3B or 7B at 7B, Command R7B, and OpenHermes 2.5. For image generation, Stable Diffusion XL at 3.417B parameters needs 4.1 GB at FP8 or optimized settings and requires 6.1 GB of system RAM.

You must also consider the context window when running these models. The memory numbers listed here are calculated using a basic 4k context window. If you increase the context window to process longer documents or larger chat histories, the memory usage will grow. This extra memory demand can exceed the 4 GB limit of your card and force the system to slow down or fail to generate a response.