Best local AI models for NVIDIA RTX 4090

The largest models that fit fully

The 30 largest of the 173 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX 4090 carries 24 GB of GDDR6X video memory, and that number, more than the chip itself, decides which language models it can hold. A model only runs well when its weights and its working context both sit inside that memory.

Of the 233 models in our catalog with verified parameter counts, 173 fit this card fully at a 4k context. The largest is Qwen3 8B / 14B / 32B at 32B parameters, which fits at Q4_K_M using 23.4 GB. Below it sit models like Qwen2.5-Coder 0.5B to 32B at 32B, which fits at Q4_K_M in 23.4 GB, with room to spare for a longer context.

The quant column matters as much as the parameter count. A quant is a compressed copy of the model: Q4_K_M is the practical floor most people run, Q5_K_M and Q6_K trade a little more memory for measurably better output, Q8_0 is close to lossless, and FP16 is the uncompressed original. The table lists the best quant that fully fits this card for each model, so a model shown at Q4_K_M is at its limit here, while one shown at Q8_0 or FP16 has headroom you can spend on a longer context instead.

The close but does not fit list holds models whose weights spill into system memory. OTel 2.0 LLM 31B IT at 32.1B needs 27.5 GB of video memory at Q4_K_M, so on this card part of it runs from system RAM (the figures assume 32 GB of it). That works, and it is how many people run models one size above their card, but every token then waits on system memory bandwidth, so expect output several times slower than a model that fits fully.

Every figure on this page comes from the same calculator the verdict pages use: file sizes are calibrated against published GGUF releases, the context cost is computed from each model's own attention shape, and the fit is measured at a 4k context. Doubling the context roughly doubles the context cost, so a model listed here at its limit will not hold a 16k conversation on this card. Numbers you can check beat adjectives, so nothing on this page says fast or smooth; it says fits, spills, or does not fit.

If you want this table computed against your own machine instead of this card alone, upload your DxDiag on the AI models page and every model gets a verdict for your exact memory, processor and system RAM, including the quant table and a context length slider. The card is only part of the answer; free memory at run time, background load and the runtime you choose all move the practical limit a little.