Best local AI models for NVIDIA Quadro RTX 4000 MAX-Q

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA Quadro RTX 4000 MAX-Q is a professional mobile graphics card equipped with 8 GB of GDDR6 video memory. This dedicated memory size determines which artificial intelligence models you can run entirely on the hardware. To run a model smoothly at native speed, the active weights and working memory must fit within this 8 GB limit. If a model exceeds this boundary, the system must use alternative execution methods.

The quantization column indicates the compression level applied to each model. Quantization reduces the precision of model weights to save space. For example, a Q4_K_M quant uses approximately four bits per weight, while a Q6_K quant uses six bits. Higher quantization levels like Q6_K preserve more of the original model accuracy but require more memory. Lower quants like Q4_K_M allow larger models to fit inside your 8 GB limit at a small cost to output quality.

Models like Gemma 2 9B and Mochi 1 10B can fit directly into your video memory when using a Q4_K_M quantization. This setup uses up to 8 GB of your graphics memory. Other models like Llama 3.1 8B and Granite 3.3 8B can run at a higher Q5_K_M or Q6_K quantization. These configurations use between 7.4 GB and 7.9 GB of video memory. Running these models entirely on your graphics card ensures the fastest possible processing speeds.

When a model is too large for the 8 GB video memory, you can use CPU offloading. This process splits the model weights between your graphics card and your system RAM. For example, running Mistral NeMo 12B at Q4_K_M requires 8.8 GB of space, which uses all your video memory and 10.8 GB of system RAM. Running FLUX.1 dev at FP8 requires 14.4 GB of space, which uses your video memory and 16.4 GB of system RAM. CPU offloading allows you to run these larger models, but it significantly reduces processing speed.

You must also consider the memory cost of context length. The listed memory usage figures assume a standard 4k context window. If you increase the context window to process longer documents, the system requires more memory for the active conversation history. This extra memory demand can push a model that normally fits your 8 GB video memory into CPU offloading. Keeping your context window at 4k or lower helps maintain maximum speed on your hardware.