Best local AI models for NVIDIA RTX 4070 Laptop

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX 4070 Laptop graphics card features 8 GB of GDDR6 memory. This dedicated video memory determines which artificial intelligence models you can run entirely on your hardware. When a model fits completely within this limit, it runs at maximum speed because the graphics processor accesses the data directly.

The quantization column shows the best compression level for each model. Quantization reduces the size of a model so it uses less memory. For example, Gemma 2 9B fits in 8 GB of memory when using the Q4_K_M quantization. Other models like Llama 3.1 8B require 7.4 GB of memory at the Q5_K_M quantization level. Smaller models like Mistral 7B and Qwen2.5 7B can run at the higher quality Q6_K quantization while using 7.4 GB and 6.9 GB of memory.

If a model exceeds the 8 GB limit, you must use CPU offloading. This process splits the model between your graphics card and your system memory. You need a system with 32 GB of system RAM to run these larger models. For instance, FLUX.1 dev requires 14.4 GB of memory at FP8 or optimized settings, which uses 16.4 GB of system RAM. Gemma 3 12B and Mistral NeMo 12B require 8.8 GB of memory at Q4_K_M, which uses 10.8 GB of system RAM.

CPU offloading allows you to run larger options like Vicuna 13B, which needs 9.5 GB of memory at Q4_K_M and uses 11.5 GB of system RAM. However, offloading comes with a performance cost. Moving data between your system RAM and your graphics card is much slower than keeping everything in the dedicated video memory. Your generation speeds will drop significantly when you offload layers.

You must also consider the memory cost of context length. The memory figures listed are calculated using a standard 4k context window. If you increase the context window to process longer documents or chat histories, the model will require more memory. This extra memory usage might force you to use a lower quantization level or rely on CPU offloading to avoid running out of video memory.