Best local AI models for NVIDIA GTX 1070 MAX-Q

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA GTX 1070 MAX-Q is a mobile graphics card equipped with 8 GB of GDDR5 video memory. This dedicated memory pool determines which artificial intelligence models you can run entirely on your graphics hardware. Running a model completely within your video memory ensures the fastest possible processing speeds for text generation and image creation.

The quantization column indicates the compression level applied to each model. Quantization reduces the size of a model so it fits into your hardware memory. For example, the Mochi 1 10B model fits into 7.3 GB of video memory using the Q4_K_M quantization. Other models like Gemma 2 9B require exactly 8 GB of video memory at the Q4_K_M quantization level.

Many models can run at higher quality levels with less compression on this hardware. The Llama 3.1 8B model fits comfortably using 7.4 GB of video memory at the Q5_K_M quantization. Popular models like Mistral 7B require 7.4 GB of video memory at the Q6_K quantization. You can also run Qwen2.5 7B, OLMo 2 7B, and Falcon 3 7B using 6.9 GB of video memory at the Q6_K quantization.

When a model exceeds your 8 GB of video memory, you must offload some processing to your system memory. This offloading process requires a system with at least 32 GB of system RAM. Offloading allows you to run larger models but it reduces your processing speed significantly because system RAM is much slower than GDDR5 video memory.

For example, FLUX.1 dev requires 14.4 GB of memory at the FP8 or optimized level, which uses 16.4 GB of system RAM. Mistral NeMo 12B and Pixtral 12B both require 8.8 GB of memory at the Q4_K_M quantization, which uses 10.8 GB of system RAM. Vicuna 13B requires 9.5 GB of memory at the Q4_K_M quantization, which uses 11.5 GB of system RAM.

You must also consider the memory cost of context length. The memory figures listed here assume a standard 4k context window. If you increase the context window to process longer documents or chat histories, the model will require more video memory. This extra memory usage might force you to use a lower quantization level or offload processing to your system RAM.