Best local AI models for NVIDIA L4

The largest models that fit fully

The 30 largest of the 173 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA L4 graphics card features 24 GB of GDDR6 memory. This dedicated memory size determines which artificial intelligence models can run entirely on the hardware. When a model fits completely within this 24 GB limit, it runs at maximum speed because the processor has direct and fast access to all parameters.

The quantization column shows the best compression format for each model. Quantization reduces the size of model weights to save memory. For example, the Q4_K_M format allows 32B models like Qwen3, Qwen3.5 dense variants, Aya Expanse, Granite 4.0, and Qwen2.5-Coder to fit inside 23.4 GB of memory. The Q5_K_M format fits the 27B Gemma 3, Gemma 3 vision, and Wan 2.2 or 2.5 models inside 23 GB. The Q6_K format fits the 24B Mistral Small 3.2, Magistral Small, and Devstral Small 1.1 inside 23.6 GB. The Q8_0 format fits the 17B HunyuanImage 2.1 or 3.0 inside 21.6 GB.

Models that require more than 24 GB of memory cannot fit entirely on the NVIDIA L4. These models require CPU offloading to run on a system with 32 GB of system RAM. Offloading means some model parts are stored in system memory instead of graphics memory. This process allows you to run larger models, but it costs significant processing speed because system RAM is much slower than GDDR6 memory.

Examples of offloaded models include the OTel 2.0 LLM 31B IT which needs 27.5 GB of memory at Q4_K_M and uses 29.5 GB of system RAM. The DeepSeek-Coder 33B and WizardCoder 33B models need 24.2 GB at Q4_K_M and use 26.2 GB of system RAM. The Yi 1.5 34B, Granite Code 34B, LLaVA 1.5 or 1.6 34B, and Ovis 2 models need 24.9 GB at Q4_K_M and use 26.9 GB of system RAM. The Qwen3.6-35B-A3B and Command R 35B models need 25.6 GB at Q4_K_M and use 27.6 GB of system RAM. The Seed-OSS 36B needs 26.4 GB at Q4_K_M and uses 28.4 GB of system RAM.

All listed memory figures are calculated using a standard context window of 4k tokens. If you increase the context window to process longer documents, the model will require more memory. This extra memory usage can cause a model that normally fits on the card to overflow into system RAM, which will decrease performance.