Best local AI models for NVIDIA GT 640

The largest models that fit fully

The 30 largest of the 56 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA GT 640 is an entry level graphics card equipped with 2 GB of DDR3 memory. This hardware configuration places strict limits on the size of the artificial intelligence models you can run locally. To load a model entirely onto this GPU, the total memory footprint of the model must remain under the 2 GB physical VRAM limit.

The quantization column indicates the compression level applied to each model. Quantization reduces the precision of model weights to save space. For example, the Allegro 2.8B model fits within 2 GB of VRAM when using the Q4_K_M quantization. Other models like SmolLM2 1.7B can run at a higher quality Q6_K quantization while using 1.7 GB of VRAM. Smaller models like TinyLlama 1.1B can run at the high quality Q8_0 quantization because they only require 1.4 GB of VRAM.

When a model exceeds the 2 GB VRAM limit, you must use CPU offload. This technique splits the model layers between your GPU memory and your system RAM. Assuming your computer has 32 GB of system RAM, you can run larger models like the SmolLM3 3B or the Kandinsky 3.1. These 3B models require 2.2 GB of VRAM at Q4_K_M quantization and need an additional 4.2 GB of system RAM to function.

Using CPU offload comes with a significant performance cost. System RAM is much slower than video memory. When layers are offloaded to system RAM, the processing speed drops because data transfer takes longer. For instance, running Stable Diffusion XL at FP8 requires 4.1 GB of VRAM and 6.1 GB of system RAM, which will result in slow generation times on this hardware.

You must also consider the memory cost of context length. Running text models with a standard 4k context window requires extra memory to store the active conversation history. This active memory is called the KV cache. If you use a model close to the 2 GB limit, like the LFM2 2.6B at Q4_K_M using 1.9 GB of VRAM, a long conversation will quickly exceed your remaining memory and cause an out of memory error.