Best local AI models for NVIDIA RTX A4000H

The largest models that fit fully

The 30 largest of the 155 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX A4000H graphics card features 16 GB of GDDR6 memory. This dedicated video memory determines the maximum size of the artificial intelligence models you can run locally. To load a model entirely on the graphics card, the model files and the active memory must fit within this 16 GB limit. Running models fully in video memory ensures the fastest processing speeds.

The quantization column shows the compression level used to fit these models into memory. Quantization reduces the precision of model weights to save space. For example, Q4_K_M represents a four bit quantization that balances size and quality. Higher quantizations like Q6_K or Q8_0 offer better accuracy but require more memory. The gpt-oss-20b and Reka Flash 3 models utilize Q4_K_M to fit their 21B parameters into 15.4 GB of video memory.

For models that exceed 16 GB, you can use CPU offloading if your computer has at least 32 GB of system RAM. This technique splits the model between the graphics card and your system memory. Solar Pro and Codestral 22B require 16.1 GB of memory at Q4_K_M, which needs 18.1 GB of system RAM. Larger models like Gemma 3 27B require 19.8 GB at Q4_K_M, which needs 21.8 GB of system RAM.

CPU offloading allows you to run larger models like Mistral Small 3.2, Magistral Small, and Devstral Small 1.1. These 24B models need 17.6 GB at Q4_K_M and require 19.6 GB of system RAM. Aria needs 18.3 GB at Q4_K_M and requires 20.3 GB of system RAM. While offloading enables these larger models to run, it reduces processing speed because system RAM is slower than GDDR6 video memory.

You can run several highly capable models entirely within the 16 GB video memory limit. DeepSeek-Coder-V2 16B and Kimi-VL A3B fit into 15.7 GB using Q6_K. Qwen2.5 14B fits into 15.3 GB using Q6_K. Image and video models like FLUX.1 dev fit into 14.4 GB using FP8 or optimized settings. Gemma 3 12B, Gemma 4 12B, Mistral NeMo 12B, and Pixtral 12B fit into 15.3 GB using Q8_0.

Memory calculations must also account for the context window. The listed memory figures assume a standard 4k context window. If you increase the context window to process longer documents or chat histories, the memory usage will rise. This extra memory requirement may force you to use a lower quantization level or switch to CPU offloading to prevent out of memory errors.