Best local AI models for NVIDIA Quadro K620

The largest models that fit fully

The 30 largest of the 56 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA Quadro K620 is an entry level workstation graphics card equipped with 2 GB of DDR3 video memory. This VRAM capacity determines the maximum size of the artificial intelligence models you can run entirely on the hardware. When a model fits completely within this 2 GB limit, the GPU processes tokens and generation tasks at its maximum possible speed.

To fit larger models into this limited space, you must use quantization. The quant column indicates the specific compression level required to run a model on this hardware. For example, the Allegro 2.8B model fits within 2 GB of VRAM when compressed to the Q4_K_M format. Smaller models like Moondream 2 can run at a higher quality Q6_K quantization while using 1.9 GB of VRAM. TinyLlama 1.1B can run at the high quality Q8_0 quantization level while using only 1.4 GB of VRAM.

If you want to run models that exceed the 2 GB VRAM limit, you must use CPU offloading. This technique splits the model layers between your graphics card and your system memory. For this setup, we assume your computer has 32 GB of system RAM. Offloading allows you to run larger options such as the SmolLM3 3B model or the MusicGen small medium large 3.3B model. While offloading enables these larger models to function, it introduces a significant performance cost because system RAM is much slower than video memory.

When using CPU offloading, the system RAM requirements scale with the model size. The Kandinsky 3.1 model requires 2.2 GB of VRAM at Q4_K_M quantization and needs an additional 4.2 GB of system RAM to function. Running Stable Diffusion XL requires 4.1 GB at FP8 or optimized settings along with 6.1 GB of system RAM. SDXL Turbo needs 2.6 GB of VRAM at Q4_K_M and 4.6 GB of system RAM.

You must also consider the memory cost of context windows. The memory figures listed for these models assume a standard 4k context window. If you increase the context length to process longer documents or conversations, the memory usage will rise. This extra memory demand can exceed the 2 GB VRAM limit of the Quadro K620 and force the system to slow down.