Best local AI models for NVIDIA Quadro RTX 4000

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA Quadro RTX 4000 is equipped with 8 GB GDDR6 of dedicated video memory. This memory size determines which artificial intelligence models can run directly on your hardware. To run a model entirely on the graphics card, the model files and active memory must fit within this 8 GB limit. If a model exceeds this capacity, it will fail to load or require system memory offloading.

The quantization column indicates the compression level applied to each model. Quantization reduces the precision of model weights to save memory. For example, the Q4_K_M quant represents a four bit quantization level. The Q5_K_M and Q6_K quants offer higher precision but require more memory. Using these optimized formats allows you to run larger models like Gemma 2 9B at Q4_K_M using 8 GB or Mistral 7B at Q6_K using 7.4 GB.

Running models with a high context window of 4k tokens or more increases memory consumption. The listed memory usage figures represent the base model requirements. When you generate long responses or input large documents, the memory usage will rise. You must keep a buffer of free video memory to prevent out of memory errors during long conversations.

When a model size exceeds the 8 GB limit of your graphics card, you can offload layers to your system RAM. This process requires a system with at least 32 GB system RAM. Offloading allows you to run larger models such as FLUX.1 dev at FP8 which needs 14.4 GB of memory and 16.4 GB of system RAM. You can also run Mistral NeMo 12B at Q4_K_M which needs 8.8 GB of memory and 10.8 GB of system RAM.

Offloading comes with a performance cost. Transferring data between your system RAM and the graphics card over the PCIe bus is much slower than using dedicated GDDR6 memory. Models like Vicuna 13B require 9.5 GB at Q4_K_M and 11.5 GB of system RAM, which will result in slower generation speeds compared to fully local models like Llama 3.1 8B which runs entirely in video memory using 7.4 GB.