Best local AI models for NVIDIA L40
48 GB GDDR6. At a 4k context, 186 of the 233 models in our catalog with verified parameter counts fit fully, up to Jamba 1.5 Mini / Large at 52B parameters.
Check your own machine against every model →The largest models that fit fully
The 30 largest of the 186 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.
| Model | Parameters | Best quant that fits | Memory used at 4k |
|---|---|---|---|
| Jamba 1.5 Mini / Large | 52B | Q5_K_M | 44.3 GB |
| Llama 3.1 Nemotron 51B | 51B | Q5_K_M | 43.5 GB |
| Mixtral 8x7B | 47B | Q6_K | 46.2 GB |
| Seed-OSS 36B | 36B | Q8_0 | 45.8 GB |
| Qwen3.6-35B-A3B | 35B | Q8_0 | 44.5 GB |
| Command R (35B) | 35B | Q8_0 | 44.5 GB |
| Yi 1.5 9B / 34B | 34B | Q8_0 | 43.2 GB |
| Granite Code 3B to 34B | 34B | Q8_0 | 43.2 GB |
| LLaVA 1.5 / 1.6 (7B to 34B) | 34B | Q8_0 | 43.2 GB |
| Ovis 2 | 34B | Q8_0 | 43.2 GB |
| DeepSeek-Coder 1.3B / 6.7B / 33B | 33B | Q8_0 | 42 GB |
| WizardCoder 33B | 33B | Q8_0 | 42 GB |
| OTel 2.0 LLM 31B IT | 32.1B | Q8_0 | 44.9 GB |
| Qwen3 8B / 14B / 32B | 32B | Q8_0 | 40.7 GB |
| Qwen3.5 (dense variants) | 32B | Q8_0 | 40.7 GB |
| Aya Expanse 8B / 32B | 32B | Q8_0 | 40.7 GB |
| Granite 4.0 Small/Tiny | 32B | Q8_0 | 40.7 GB |
| Qwen2.5-Coder 0.5B to 32B | 32B | Q8_0 | 40.7 GB |
| Qwen3-30B-A3B | 30B | Q8_0 | 38.2 GB |
| Qwen3-Coder 30B-A3B | 30B | Q8_0 | 38.2 GB |
| Gemma 3 27B | 27B | Q8_0 | 34.3 GB |
| Gemma 3 4B/12B/27B (vision) | 27B | Q8_0 | 34.3 GB |
| Wan 2.2 / 2.5 | 27B | Q8_0 | 34.3 GB |
| Gemma 4 26B-A4B | 26B | Q8_0 | 33.1 GB |
| Gemma 4 (all sizes) | 26B | Q8_0 | 33.1 GB |
| Aria | 25B | Q8_0 | 31.8 GB |
| Mistral Small 3.2 | 24B | Q8_0 | 30.5 GB |
| Magistral Small | 24B | Q8_0 | 30.5 GB |
| Devstral Small 1.1 | 24B | Q8_0 | 30.5 GB |
| Solar Pro | 22B | Q8_0 | 28 GB |
How to read this
The NVIDIA L40 graphics card features 48 GB of GDDR6 memory. This dedicated memory determines the maximum size of the artificial intelligence models you can run locally. To run a model at high speeds, the entire model and its active working data must fit completely inside this 48 GB space.
The quantization column indicates the compression level used to shrink the model files. Quantization reduces the precision of model weights to save space. A Q8_0 quant represents eight bit quantization, which preserves excellent output quality. For larger models, a Q5_K_M or Q6_K quant is used to fit the model within the hardware limits.
When a model exceeds the 48 GB memory of the NVIDIA L40, some parts must be offloaded to the system RAM. This system has 32 GB of system RAM. However, offloading data to system RAM is not required for any of the listed configurations. Every model listed here fits entirely within the graphics card memory to ensure maximum processing speed.
The largest models that fit the memory include Jamba 1.5 Mini / Large at 52B parameters using a Q5_K_M quant which consumes 44.3 GB. Llama 3.1 Nemotron 51B fits at Q5_K_M using 43.5 GB. Mixtral 8x7B fits at Q6_K using 46.2 GB. Seed-OSS 36B, Qwen3.6-35B-A3B, and Command R (35B) all run at Q8_0 quantization using under 46 GB of memory.
Other high performance options include Yi 1.5 9B / 34B, Granite Code 3B to 34B, LLaVA 1.5 / 1.6 (7B to 34B), and Ovis 2. These 34B models use 43.2 GB with a Q8_0 quant. You can also run DeepSeek-Coder 1.3B / 6.7B / 33B and WizardCoder 33B at Q8_0 using 42 GB. Qwen3 8B / 14B / 32B and Qwen2.5-Coder 0.5B to 32B fit at Q8_0 using 40.7 GB.
When planning your deployments, remember that these memory figures are calculated using a standard 4k context window. If you increase the context window to process longer documents, the system will require additional memory for the context cache. You must leave some of the 48 GB memory free to accommodate this extra data.