Best local AI models for NVIDIA GTX 1080 MAX-Q

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA GTX 1080 MAX-Q is a mobile graphics card equipped with 8 GB of GDDR5X frame buffer memory. This dedicated video memory determines the size of the artificial intelligence models you can run locally. To run a model entirely on your graphics hardware, the model files and the active context data must fit within this 8 GB limit. If a model exceeds this capacity, your system must transfer data to system memory, which slows down execution speed.

The quantization column shows the compression level used to fit these models into your hardware memory. Quantization reduces the precision of model weights to save space. For example, a Q4_K_M quantization uses a four bit format to compress larger models like Mochi 1 10B down to 7.3 GB of used memory. A higher quantization level like Q6_K preserves more original model quality but requires more memory, as seen with Mistral 7B using 7.4 GB of memory at Q6_K.

Several high quality models fit completely within your 8 GB video memory. You can run Gemma 2 9B at the Q4_K_M quantization level using exactly 8 GB of memory. Other models like Nemotron Nano 9B, GLM-4 9B, Yi-Coder 9B, and GLM-4V-9B fit comfortably at the Q5_K_M quantization level using 7.7 GB of memory. Llama 3.1 8B fits at Q5_K_M using 7.4 GB of memory. Popular models like Qwen2.5 7B, Falcon 3 7B, and Command R7B run at Q6_K quantization using 6.9 GB of memory.

When a model is too large for your 8 GB video memory, you can use CPU offload if your computer has at least 32 GB of system RAM. This technique splits the model between your graphics card and your system memory. For instance, FLUX.1 dev requires 14.4 GB at FP8 or optimized settings, which uses 16.4 GB of system RAM. Other models like Gemma 3 12B, Mistral NeMo 12B, and FLUX.1 schnell require 8.8 GB of memory at Q4_K_M and use 10.8 GB of system RAM.

CPU offloading allows you to run larger architectures but it comes with a performance cost. Transferring data between your system RAM and your graphics card is much slower than running models entirely on GDDR5X memory. You will experience a lower generation speed when running offloaded models like Vicuna 13B, which needs 9.5 GB at Q4_K_M and uses 11.5 GB of system RAM.

All listed memory usage figures assume a standard 4k context window. The context window is the amount of text the model can read and remember at one time. If you increase the context window beyond 4k tokens, the model will require significantly more memory. This extra memory demand can push a model over your 8 GB limit and force your system into slow CPU offloading.