Best local AI models for NVIDIA P102-100
5 GB GDDR5X. At a 4k context, 85 of the 233 models in our catalog with verified parameter counts fit fully, up to Magicoder-S-DS 6.7B at 6.7B parameters.
Check your own machine against every model →The largest models that fit fully
The 30 largest of the 85 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.
| Model | Parameters | Best quant that fits | Memory used at 4k |
|---|---|---|---|
| Magicoder-S-DS 6.7B | 6.7B | Q4_K_M | 4.9 GB |
| Phi-4-multimodal | 5.6B | Q5_K_M | 4.8 GB |
| Lumina-Next / Lumina-Image 2.0 | 5B | Q6_K | 4.9 GB |
| CogVideoX 2B / 5B | 5B | Q6_K | 4.9 GB |
| DeepSeek-VL2 | 4.5B | Q6_K | 4.4 GB |
| DeepFloyd IF | 4.3B | Q6_K | 4.2 GB |
| Phi-3.5-vision | 4.2B | Q6_K | 4.1 GB |
| Qwen3 4B | 4B | Q6_K | 3.9 GB |
| Gemma 3 4B | 4B | Q6_K | 3.9 GB |
| Gemma 4 E4B | 4B | Q6_K | 3.9 GB |
| MiniCPM 3 4B | 4B | Q6_K | 3.9 GB |
| Danube 3 4B | 4B | Q6_K | 3.9 GB |
| Fish Speech 1.5 / OpenAudio S1 | 4B | Q6_K | 3.9 GB |
| Phi-3 Mini | 3.8B | Q5_K_M | 4.8 GB |
| Phi-4-mini-instruct | 3.8B | Q8_0 | 4.8 GB |
| Phi-3.5 Mini | 3.8B | Q8_0 | 4.8 GB |
| OmniGen / OmniGen2 | 3.8B | Q8_0 | 4.8 GB |
| SD Cascade (Würstchen v3) | 3.6B | Q8_0 | 4.6 GB |
| SDXL Turbo | 3.5B | Q8_0 | 4.5 GB |
| SDXL Lightning | 3.5B | Q8_0 | 4.5 GB |
| ACE-Step | 3.5B | Q8_0 | 4.5 GB |
| Stable Diffusion XL | 3.417B | FP8 / optimized | 4.1 GB |
| MusicGen small/medium/large | 3.3B | Q8_0 | 4.2 GB |
| SmolLM3 3B | 3B | Q8_0 | 3.8 GB |
| Replit Code v1.5 3B | 3B | Q8_0 | 3.8 GB |
| Kandinsky 3.1 | 3B | Q8_0 | 3.8 GB |
| Voxtral Mini / Small | 3B | Q8_0 | 3.8 GB |
| Orpheus TTS | 3B | Q8_0 | 3.8 GB |
| Higgs Audio v2 | 3B | Q8_0 | 3.8 GB |
| Allegro | 2.8B | Q8_0 | 3.6 GB |
Close, but only with CPU offload
These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.
| Model | Parameters | Memory at Q4_K_M | System RAM at 4k |
|---|---|---|---|
| Mistral 7B | 7B | 5.7 GB needed | 7.7 GB |
| Qwen2.5 0.5B / 1.5B / 3B / 7B | 7B | 5.1 GB needed | 7.1 GB |
| OLMo 2 1B / 7B | 7B | 5.1 GB needed | 7.1 GB |
| Falcon 3 1B / 3B / 7B | 7B | 5.1 GB needed | 7.1 GB |
| Command R7B | 7B | 5.1 GB needed | 7.1 GB |
| OpenHermes 2.5 | 7B | 5.1 GB needed | 7.1 GB |
| Zephyr 7B Beta | 7B | 5.1 GB needed | 7.1 GB |
| OpenChat 3.5 | 7B | 5.1 GB needed | 7.1 GB |
| Starling LM 7B | 7B | 5.1 GB needed | 7.1 GB |
| Codestral Mamba 7B | 7B | 5.1 GB needed | 7.1 GB |
How to read this
The NVIDIA P102-100 is a specialized graphics card equipped with 5 GB of GDDR5X memory. This memory size is the absolute limit for loading local artificial intelligence models directly onto the hardware. To run a model entirely on the graphics card, the combined size of the model weights and the active context window must not exceed this 5 GB boundary.
The quantization column shows the best compression format that fits within the hardware limits. Quantization reduces the precision of model weights to save space. For example, Magicoder-S-DS 6.7B fits at Q4_K_M using 4.9 GB of memory. Phi-4-multimodal fits at Q5_K_M using 4.8 GB of memory. Lumina-Next, Lumina-Image 2.0, CogVideoX 2B / 5B, DeepSeek-VL2, DeepFloyd IF, and Phi-3.5-vision all fit at Q6_K quantization. Qwen3 4B, Gemma 3 4B, Gemma 4 E4B, MiniCPM 3 4B, Danube 3 4B, and Fish Speech 1.5 / OpenAudio S1 also use Q6_K quantization and require 3.9 GB of memory.
Models with smaller parameter counts can run at higher precision levels like Q8_0 or optimized FP8. Phi-3 Mini, Phi-4-mini-instruct, Phi-3.5 Mini, and OmniGen / OmniGen2 fit at Q8_0 or Q5_K_M using 4.8 GB of memory. SD Cascade (Würstchen v3), SDXL Turbo, SDXL Lightning, and ACE-Step fit at Q8_0 using 4.5 GB to 4.6 GB of memory. Stable Diffusion XL fits at FP8 / optimized using 4.1 GB of memory. MusicGen small/medium/large fits at Q8_0 using 4.2 GB of memory. SmolLM3 3B, Replit Code v1.5 3B, Kandinsky 3.1, Voxtral Mini / Small, Orpheus TTS, and Higgs Audio v2 all fit at Q8_0 using 3.8 GB of memory. Allegro fits at Q8_0 using 3.6 GB of memory.
When a model is too large for the 5 GB graphics memory, you must use CPU offload. This process splits the model layers between your graphics card and your system RAM. CPU offload allows you to run larger 7B models but it reduces processing speed because system RAM is much slower than GDDR5X memory. This setup assumes you have at least 32 GB of system RAM available.
Several 7B models can run using CPU offload. Mistral 7B requires 5.7 GB of graphics memory at Q4_K_M quantization and needs 7.7 GB of system RAM. Qwen2.5 0.5B / 1.5B / 3B / 7B, OLMo 2 1B / 7B, Falcon 3 1B / 3B / 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, and Codestral Mamba 7B all require 5.1 GB of graphics memory at Q4_K_M quantization and need 7.1 GB of system RAM.
You must also consider the 4k context window caveat. The memory figures listed are calculated using a standard 4k context window. If you increase the context window to process longer documents or chat histories, the memory usage will grow. This extra memory demand can easily push a model past the 5 GB limit of the card and cause a system slowdown.