Best local AI models for NVIDIA RTX 4000 SFF Ada Generation

The largest models that fit fully

The 30 largest of the 166 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX 4000 SFF Ada Generation is a low profile workstation graphics card. It features 20 GB of GDDR6 dedicated video memory. This memory size determines which artificial intelligence models you can run entirely on the graphics hardware. Keeping the model files inside this 20 GB limit ensures fast processing speeds and low latency.

The quantization column indicates the compression level of each model. Quantization reduces the precision of model weights to save space. For example, Gemma 3 27B, Gemma 3 27B vision, and Wan 2.2 / 2.5 can run at the Q4_K_M quantization level using 19.8 GB of video memory. Gemma 4 26B-A4B and Gemma 4 all sizes also run at Q4_K_M using 19 GB of video memory. Aria fits at Q4_K_M using 18.3 GB of video memory.

Smaller models can run at higher precision levels. Apriel-1.5-15B-Thinker and StarCoder2 15B fit at the Q8_0 quantization level using 19.1 GB of video memory. Qwen2.5 14B fits at Q8_0 using 19.5 GB of video memory. Phi-3 Medium, Phi-4, Phi-4-reasoning / -plus, Wan 2.2 T2I, Wan 2.1 14B, and SkyReels V2 all fit at Q8_0 using 17.8 GB of video memory. Vicuna 13B fits at Q8_0 using 16.5 GB of video memory.

When a model is too large for the 20 GB video memory, you must offload parts of it to the system RAM. This offload process allows you to run larger models but slows down the generation speed significantly. For these setups, we assume a system with 32 GB of system RAM. Qwen3 32B, Qwen3.5 dense variants, Aya Expanse 32B, Granite 4.0 Small/Tiny, and Qwen2.5-Coder 32B need 23.4 GB of memory at Q4_K_M, which requires 25.4 GB of system RAM.

Other offload options include Qwen3-30B-A3B and Qwen3-Coder 30B-A3B, which need 22 GB at Q4_K_M and require 24 GB of system RAM. OTel 2.0 LLM 31B IT needs 27.5 GB at Q4_K_M and requires 29.5 GB of system RAM. DeepSeek-Coder 33B and WizardCoder 33B need 24.2 GB at Q4_K_M and require 26.2 GB of system RAM.

You must also consider the context window size when loading these models. The memory usage figures listed here are calculated using a baseline 4k context window. If you increase the context window to process longer documents, the system will require additional video memory. This extra memory requirement might force you to use a lower quantization level or offload more layers to system RAM.