Best local AI models for NVIDIA RTX 4000 Ada Generation

The largest models that fit fully

The 30 largest of the 166 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX 4000 Ada Generation workstation graphics card features 20 GB GDDR6 of dedicated video memory. This onboard memory capacity determines the size of the artificial intelligence models you can run entirely on the graphics hardware. Keeping the entire model inside the video memory ensures the fastest possible processing speeds for your local inference tasks.

The quantization column indicates the compression level applied to each model. Quantization reduces the precision of model weights to save memory. For this hardware, larger models like Gemma 3 27B, Gemma 4 26B-A4B, and Wan 2.2 use the Q4_K_M quantization to fit within 19.8 GB or 19 GB of video memory. Medium models like Solar Pro 22B and Codestral 22B run at Q5_K_M quantization using 18.7 GB. Smaller models like Qwen2.5 14B and Phi-4 can run at the higher quality Q8_0 quantization using 19.5 GB and 17.8 GB respectively.

When a model size exceeds the 20 GB limit of your graphics card, you can use CPU offload. This technique splits the model layers between your video memory and your system memory. For example, running Qwen3 32B or Aya Expanse 32B requires 23.4 GB of memory at Q4_K_M quantization. This setup requires 25.4 GB of system RAM to function. Similarly, DeepSeek-Coder 33B requires 24.2 GB at Q4_K_M quantization and needs 26.2 GB of system RAM.

CPU offload comes with a significant performance cost. Transferring data between the system RAM and the graphics card over the system bus is much slower than reading directly from the onboard GDDR6 memory. While offloading allows you to run larger models like OTel 2.0 LLM 31B IT, your generation speed will drop noticeably compared to models that fit entirely on the graphics card.

You must also consider the memory cost of the context window. The memory figures listed for these models assume a standard 4k context window. If you increase the context length to process longer documents or chat histories, the system will require additional video memory to store the active attention cache. This extra memory usage can push a model that normally fits on the card into CPU offload territory.