Best local AI models for NVIDIA TITAN V CEO Edition

The largest models that fit fully

The 30 largest of the 183 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA TITAN V CEO Edition features 32 GB of HBM2 memory. This high speed VRAM acts as the primary storage space for running local artificial intelligence models. To run a model entirely on the graphics card, the model size and its working memory must fit within this 32 GB limit.

The quantization column shows the compression level used to fit these models. Quantization reduces the precision of model weights to save space. For this hardware, a Q5_K_M quant represents the best balance of size and accuracy for models like Seed-OSS 36B at 30.7 GB used, Qwen3.6-35B-A3B at 29.8 GB used, and Command R (35B) at 29.8 GB used. Other models like Yi 1.5 9B / 34B, Granite Code 3B to 34B, LLaVA 1.5 / 1.6 (7B to 34B), and Ovis 2 use 29 GB at this same Q5_K_M level.

Slightly smaller models can run at higher precision levels. DeepSeek-Coder 1.3B / 6.7B / 33B and WizardCoder 33B fit at Q5_K_M using 28.1 GB. OTel 2.0 LLM 31B IT uses 31.4 GB at Q5_K_M. You can run Qwen3 8B / 14B / 32B, Qwen3.5 (dense variants), Aya Expanse 8B / 32B, Granite 4.0 Small/Tiny, and Qwen2.5-Coder 0.5B to 32B at a higher Q6_K quant using 31.5 GB. Qwen3-30B-A3B and Qwen3-Coder 30B-A3B use 29.5 GB at Q6_K. Gemma 3 27B, Gemma 3 4B/12B/27B (vision), and Wan 2.2 / 2.5 use 26.6 GB at Q6_K. Gemma 4 26B-A4B and Gemma 4 (all sizes) use 25.6 GB at Q6_K.

For maximum precision, some models can run at the Q8_0 quant level. Aria uses 31.8 GB at Q8_0. Mistral Small 3.2, Magistral Small, and Devstral Small 1.1 use 30.5 GB at Q8_0. Solar Pro and Codestral 22B use 28 GB at Q8_0. The gpt-oss-20b and Reka Flash 3 models use 26.7 GB at Q8_0.

If a model is too large for the 32 GB VRAM, you must offload parts of it to your system CPU and system RAM. Offloading allows you to run larger models but slows down processing speed significantly. For example, Mixtral 8x7B needs 34.4 GB at Q4_K_M and requires 36.4 GB of system RAM. Llama 3.1 Nemotron 51B needs 37.3 GB at Q4_K_M and requires 39.3 GB of system RAM. Jamba 1.5 Mini / Large needs 38.1 GB at Q4_K_M and requires 40.1 GB of system RAM.

All memory calculations assume a standard 4k context window. If you increase the context window to process longer texts, the model will require more VRAM. You may need to choose a lower quantization level to keep the model within the 32 GB HBM2 limit when using longer context lengths.