Best local AI models for NVIDIA GTX TITAN

The largest models that fit fully

The 30 largest of the 114 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA GTX TITAN features 6 GB of GDDR5 memory. This onboard memory size dictates the maximum size of the artificial intelligence models you can run entirely on the graphics card. To run a model without slowdowns, the model files and the active memory must fit within this 6 GB limit. If a model exceeds this capacity, your system must use alternative execution methods.

The quantization column indicates the compression level used to shrink these models. Quantization reduces the precision of model weights to save space. For example, the Q4_K_M quant represents a medium four bit quantization. The Q5_K_M quant represents a five bit quantization. Using these quants allows you to fit larger models like the 8B or 7B variants into the limited memory of your graphics card.

Several 8B models can fit entirely within the graphics card memory at the Q4_K_M quantization level. Granite 3.3 8B, Ministral 8B, InternLM 3 8B, OpenCoder 8B, and Seed-Coder 8B each use 5.9 GB of memory. Vision and multimodal models like MiniCPM-V 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large also run at Q4_K_M using 5.9 GB of memory. EXAONE 3.5 7.8B and Mistral 7B fit at Q4_K_M while using 5.7 GB of memory.

Other models can run at the higher Q5_K_M quantization level. Qwen2.5 7B, OLMo 2 7B, Falcon 3 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, and Starling LM 7B all utilize exactly 6 GB of memory. Coding models like Codestral Mamba 7B, CodeGemma 7B, aiXcoder-7B, and Nxcode 7B also fit at Q5_K_M using 6 GB of memory. Multimodal options including Janus-Pro 7B, Ruyi-Mini-7B, Qwen2-Audio 7B, Qwen2.5-Omni 7B, and YuE also use 6 GB of memory at Q5_K_M. Magicoder-S-DS 6.7B fits at Q5_K_M using 5.7 GB of memory.

When a model is too large for the graphics card, you must use CPU offloading. This process splits the model between your graphics card and your system RAM. Assuming you have 32 GB of system RAM, you can run larger models with a performance penalty. Llama 3.1 8B requires 6.4 GB at Q4_K_M and needs 8.4 GB of system RAM. Chroma 8.9B needs 6.5 GB at Q4_K_M and 8.5 GB of system RAM. Gemma 2 9B requires 8 GB at Q4_K_M and 10 GB of system RAM. Nemotron Nano 9B, GLM-4 9B, Yi-Coder 9B, GLM-4-9B-Chat, and GLM-4V-9B all require 6.6 GB at Q4_K_M and 8.6 GB of system RAM. Mochi 1 10B needs 7.3 GB at Q4_K_M and 9.3 GB of system RAM. Open-Sora 2.0 11B needs 8.1 GB at Q4_K_M and 10.1 GB of system RAM.

Running models close to the memory limit introduces a context limit caveat. The memory figures listed only cover the model weights at a standard 4k context window. If you increase the context window to process longer documents, the memory usage will rise. This extra memory demand can exceed your 6 GB limit and force the system to slow down.