Best local AI models for NVIDIA Quadro M5000

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA Quadro M5000 is equipped with 8 GB of GDDR5 graphics memory. This hardware memory size determines the maximum size of the artificial intelligence models you can run entirely on the graphics card. Keeping a model inside the onboard memory ensures the fastest possible processing speeds. If a model exceeds this limit, your system must use slower alternative pathways to process the data.

To fit larger models into the 8 GB limit, developers use quantization. The quant column shows the specific compression level applied to each model. For example, the Mochi 1 10B model fits within 7.3 GB of memory when compressed to the Q4_K_M quantization level. Similarly, Llama 3.1 8B fits within 7.4 GB using the Q5_K_M quantization level. Higher quantizations like Q6_K allow models like Mistral 7B to run at 7.4 GB with excellent precision.

When a model requires more than 8 GB of memory, you must offload some processing to your system RAM. This offloading process allows you to run larger models but reduces processing speed. For instance, running the FLUX.1 dev 12B model at FP8 requires 14.4 GB of memory, which uses 16.4 GB of system RAM alongside your graphics card. Other models like Gemma 3 12B and Mistral NeMo 12B require 8.8 GB of memory at Q4_K_M quantization, which utilizes 10.8 GB of system RAM.

Other offload options include the Vicuna 13B model, which requires 9.5 GB at the Q4_K_M quantization level and utilizes 11.5 GB of system RAM. The Open-Sora 2.0 11B model requires 8.1 GB at Q4_K_M and uses 10.1 GB of system RAM. These options expand your capabilities if you have at least 32 GB of system RAM installed in your computer.

You must also consider the memory required for active conversations. The listed memory usage figures are calculated using a standard 4k context window. If you increase the context window to remember longer conversations, the system will require more memory. This extra memory demand may force you to use smaller models or lower quantization levels to prevent your system from running out of graphics memory.