Best local AI models for NVIDIA Quadro RTX 5000

The largest models that fit fully

The 30 largest of the 155 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA Quadro RTX 5000 is a professional workstation graphics card equipped with 16 GB GDDR6 VRAM. This dedicated memory determines the maximum size of the artificial intelligence models you can run entirely on the hardware. To run a model smoothly at native speeds, the entire model weights and the active context window must fit within this 16 GB limit.

Quantization is a compression method that reduces model size with minimal loss in quality. The best quant column indicates the highest precision level that fits comfortably inside your VRAM. For example, you can run the 21B gpt-oss-20b or Reka Flash 3 at a Q4_K_M quantization using 15.4 GB of memory. Smaller models like the 15B StarCoder2 3B / 7B / 15B or the 14B Phi-4 can run at a higher Q6_K precision using 14.8 GB and 13.8 GB respectively. Highly optimized models like Gemma 3 12B and Mistral NeMo 12B can even run at Q8_0 precision using 15.3 GB of VRAM.

When a model size exceeds your 16 GB VRAM, you can offload part of the workload to your system RAM. This process requires at least 32 GB of system memory to function. For instance, running the 22B Solar Pro or Codestral 22B at Q4_K_M requires 16.1 GB of space and 18.1 GB of system RAM. Larger models like Gemma 3 27B require 19.8 GB of space and 21.8 GB of system RAM. Offloading allows you to run these larger models, but it significantly reduces processing speed because system RAM is much slower than GDDR6 VRAM.

Memory calculations in our catalog assume a standard 4k context window. The context window is the amount of text the model can read and write at one time. If you increase the context window beyond 4k, the model will require more memory. This extra memory usage might force you to use a lower quantization level or offload layers to your system RAM to avoid running out of VRAM.

The NVIDIA Quadro RTX 5000 also supports vision and image generation models. You can run Qwen-Image at Q4_K_M using 14.6 GB of VRAM. For video creation, HunyuanVideo and LTX-Video / LTX-2 run at Q6_K using 12.8 GB of VRAM. Popular image models like FLUX.1 dev can run using 14.4 GB of VRAM with an FP8 optimized quantization.