Best local AI models for NVIDIA P106-090

The largest models that fit fully

The 30 largest of the 76 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA P106-090 is a budget graphics card equipped with 3 GB of GDDR5 memory. This memory size determines the maximum size of the local AI models you can run directly on the hardware. Because the card has 3 GB of onboard memory, any model and its active data must fit within this limit to run at full speed. If a model exceeds this capacity, it will fail to load or run extremely slowly.

To fit capable models into 3 GB of memory, we use quantized versions. The quantization column shows the specific compression level used for each model. For example, Q4_K_M represents a four bit quantization that balances model accuracy and memory usage. Higher quantizations like Q6_K or Q8_0 offer better quality but require more memory space. A model like Qwen3 4B fits in 2.9 GB using the Q4_K_M quant, while the smaller SmolLM3 3B can use a higher quality Q6_K quant and take up 3 GB.

You can run larger models by offloading parts of the workload to your system RAM. This process requires a computer with at least 32 GB of system RAM. Offloading allows you to run models that exceed the 3 GB limit of the graphics card, but it comes with a speed penalty. The system RAM and CPU must handle the extra data, which is much slower than the dedicated GDDR5 memory on your graphics card.

For example, Mistral 7B requires 5.7 GB of memory at Q4_K_M quantization, which uses 7.7 GB of system RAM during offloading. Similarly, Phi-4-multimodal is a 5.6B model that needs 4.1 GB at Q4_K_M and uses 6.1 GB of system RAM. Image generators like Stable Diffusion XL need 4.1 GB at FP8 or optimized settings, which requires 6.1 GB of system RAM to function with this card.

When running models close to the 3 GB limit, you must consider the context window. The memory figures listed are for the model itself at a basic 4k context window. Generating long responses or processing large documents increases memory usage. If you push the context window beyond 4k tokens, the model will require more memory and may exceed the 3 GB capacity of your card.