Best local AI models for NVIDIA GTX 1080

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA GTX 1080 features 8 GB of GDDR5X onboard memory. This memory size determines which artificial intelligence models can run directly on your graphics card. When a model fits entirely within this 8 GB limit, it executes with the fastest possible processing speeds. If a model exceeds this limit, you must use system memory to run it, which decreases performance.

The quantization column shows the compression level applied to each model. Quantization reduces the size of a model so it can fit into your graphics card memory. For example, Gemma 2 9B fits into 8 GB of memory when compressed to the Q4_K_M quantization. Other models like Llama 3.1 8B require 7.4 GB of memory at the Q5_K_M quantization, while Mistral 7B uses 7.4 GB at the Q6_K quantization.

Models with higher parameter counts can still run by offloading parts of their workload to your system memory. This process requires a computer with at least 32 GB of system RAM. For example, running FLUX.1 dev at FP8 requires 14.4 GB of memory, which uses your graphics card and 16.4 GB of system RAM. Similarly, Gemma 3 12B requires 8.8 GB at Q4_K_M and uses 10.8 GB of system RAM.

Offloading models to system RAM comes with a performance cost. Because system RAM is much slower than the GDDR5X memory on your graphics card, generation speeds will drop significantly. Models like Vicuna 13B require 9.5 GB at Q4_K_M and use 11.5 GB of system RAM, which will result in slower response times compared to models that fit entirely on the card.

You must also consider the memory required for context. The memory usage figures listed for these models are calculated using a standard 4k context window. If you increase the context window to process longer documents or chat histories, the model will require more memory. This extra memory usage might push a model over the 8 GB limit and force it to offload to system RAM.