Best local AI models for NVIDIA Quadro P620

The largest models that fit fully

The 30 largest of the 56 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA Quadro P620 is an entry level professional graphics card equipped with 2 GB of GDDR5 video memory. This onboard memory size is the primary constraint when running local artificial intelligence models. To execute a model entirely on the GPU for maximum speed, the entire model weight file and its active working memory must fit within this 2 GB limit.

The quantization column indicates the compression level applied to each model. Quantization reduces the precision of model weights to save space. For example, a Q4_K_M quantization represents a four bit medium compression level, which allows larger models like Allegro 2.8B or Open-Sora Plan 2.7B to fit into 2 GB of VRAM. Higher quality quantizations like Q6_K or Q8_0 preserve more original model accuracy but require more memory per parameter, limiting you to smaller models like Moondream 2 1.9B or TinyLlama 1.1B.

When a model size exceeds the 2 GB physical limit of your GPU, you must use CPU offloading. This process splits the model weights between your video memory and your system RAM. For instance, running the 3B parameter SmolLM3 or Replit Code v1.5 requires 2.2 GB of space at Q4_K_M quantization. Since this exceeds the onboard memory, the system offloads the remaining data to your system RAM, requiring 4.2 GB of system RAM alongside your GPU resources.

Offloading comes with a significant performance cost. System RAM is much slower than the GDDR5 memory on your graphics card. While offloading allows you to run larger architectures like MusicGen at 3.3B or Stable Diffusion XL at 3.417B, the processing speed will drop noticeably because data must constantly travel over the system bus between the CPU and GPU.

You must also account for the context window memory cost. Running a text model at a standard 4k context window length requires extra memory to store the active conversation history. This context overhead is not included in the base model file sizes listed. If you use the maximum context window, you may need to choose a smaller base model like Qwen3 1.7B or SmolLM2 1.7B to avoid running out of video memory.