Best local AI models for NVIDIA T400

The largest models that fit fully

The 30 largest of the 56 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA T400 is an entry level graphics card equipped with 2 GB GDDR6 memory. This memory size is the absolute limit for running local AI models entirely on the GPU. When a model runs inside this VRAM limit, execution is fast because the GPU can access the model weights directly. If a model exceeds this capacity, it cannot load completely into the graphics memory.

To fit models onto this hardware, you must use quantized versions. The quant column indicates the compression level applied to the model weights. For example, the Allegro 2.8B model fits within 2 GB used at a Q4_K_M quantization. Smaller models like Moondream 2 at 1.9B can run at a higher quality Q6_K quantization while using 1.9 GB. TinyLlama 1.1B can run at Q8_0 quantization using only 1.4 GB.

Running models at their limit leaves very little room for context. The 4k context caveat means that as you type longer prompts or receive longer answers, the active memory usage increases. If your model uses 1.9 GB of VRAM out of your 2 GB limit, a long conversation will quickly exceed the remaining space and cause a crash or slow down.

You can run larger models by using CPU offload if your computer has at least 32 GB system RAM. Offloading means the system splits the model between the fast GPU memory and the slower system RAM. For example, SmolLM3 3B needs 2.2 GB at Q4_K_M on the GPU and 4.2 GB on the system RAM. Stable Diffusion XL needs 4.1 GB at FP8 or optimized settings on the GPU and 6.1 GB on the system RAM.

Offloading allows you to run models like SDXL Turbo at 3.5B or MusicGen small/medium/large at 3.3B. The cost of this flexibility is speed. Transferring data between the system RAM and the GPU is much slower than keeping everything inside the 2 GB GDDR6 memory. For the fastest response times, choose models that fit completely within the local graphics memory.