Best local AI models for NVIDIA GTX 680

The largest models that fit fully

The 30 largest of the 56 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA GTX 680 graphics card has 2 GB of GDDR5 memory. This onboard memory determines which artificial intelligence models can run directly on the hardware. To run a model completely on the graphics card, the model files and active memory must fit entirely within this 2 GB limit. If a model exceeds this limit, it cannot run on the graphics card alone.

Quantization is a method that compresses model files to save space. The quant column shows the best compression format that fits your hardware. For example, the Allegro 2.8B model fits in 2 GB of memory using the Q4_K_M quant. The SmolVLM 256M / 500M / 2B model fits in 2 GB of memory using the Q6_K quant. The TinyLlama 1.1B model fits in 1.4 GB of memory using the Q8_0 quant.

Larger models require more memory than the graphics card has. You can run these models by offloading some data to your system RAM. This process requires a system with 32 GB of system RAM. Offloading allows you to run larger models, but it costs performance because system RAM is much slower than graphics card memory.

For example, the SmolLM3 3B model needs 2.2 GB of graphics memory at the Q4_K_M quant and 4.2 GB of system RAM. The Stable Diffusion XL model needs 4.1 GB of graphics memory at the FP8 / optimized quant and 6.1 GB of system RAM. The SDXL Turbo 3.5B model needs 2.6 GB of graphics memory at the Q4_K_M quant and 4.6 GB of system RAM.

There is an important caveat regarding context length. Running models with a standard 4k context window increases memory usage. If you generate long responses or process large prompts, the model might exceed the 2 GB limit of your graphics card. You may need to reduce the context size to prevent out of memory errors.