Best local AI models for NVIDIA Quadro K600

The largest models that fit fully

The 28 largest of the 28 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA Quadro K600 is an entry level professional graphics card equipped with 1 GB DDR3 of onboard video memory. This hardware memory size represents the absolute limit for storing model weights directly on the graphics card. To run artificial intelligence models locally on this hardware, you must select small architectures or use aggressive quantization to reduce the memory footprint of the weights.

The quantization column indicates the specific numerical format used to compress the model. For example, a Q4_K_M quant uses approximately four bits per weight to compress the Tango 2 1.4B model down to 1 GB of video memory. Higher precision formats like FP16 preserve more original quality but require much smaller models like the StyleTTS 2 0.15B model which uses 0.4 GB of video memory.

When a model size exceeds the physical 1 GB limit of the graphics card, you must use CPU offload. This technique splits the workload between your graphics card and your system memory. To use CPU offload, your computer should have 32 GB system RAM. For example, running Stable Diffusion 1.5 requires 1.3 GB of video memory at FP8 or optimized settings, which forces 3.3 GB of data into your system RAM.

Offloading weights to system RAM carries a significant performance cost. System RAM and DDR3 video memory transfer data much slower than modern graphics memory. While offloading allows you to run larger architectures like the StableLM 2 1.6B model or Sana 1.6B model, the generation speed will be much slower because of the data transfer bottleneck between the processor and system memory.

You must also consider the context window when running text models. The listed memory usage figures only account for the static model weights. Running a model with a 4k context window requires additional video memory to store the active conversation history. This extra memory overhead can easily exceed the 1 GB limit of the card, so you may need to reduce the context length to prevent out of memory errors.