Best local AI models for NVIDIA CMP 30HX

The largest models that fit fully

The 30 largest of the 114 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA CMP 30HX is a dedicated mining card equipped with 6 GB of GDDR6 memory. This memory size is the absolute limit for running local AI models directly on the graphics hardware. To fit models within this physical boundary, you must use quantized versions. Quantization reduces the precision of model weights to save space. The quant column indicates the highest quality level that fits your hardware without exceeding the memory capacity.

For models up to 8B parameters, a Q4_K_M quantization is the standard target. This format fits models like Granite 3.3 8B, Ministral 8B, InternLM 3 8B, OpenCoder 8B, and Seed-Coder 8B into 5.9 GB of memory. Vision and multimodal models like MiniCPM-V 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large also run at this level using 5.9 GB of VRAM. EXAONE 3.5 7.8B and Mistral 7B fit comfortably at Q4_K_M using 5.7 GB of memory.

When a model is slightly smaller, you can use a higher quality Q5_K_M quantization. This applies to 7B models which utilize exactly 6 GB of memory. Models in this category include Qwen2.5 7B, OLMo 2 7B, Falcon 3 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, and Starling LM 7B. Specialized tools like Codestral Mamba 7B, CodeGemma 7B, aiXcoder-7B, Nxcode, Janus-Pro 7B, Ruyi-Mini-7B, Qwen2-Audio 7B, Qwen2.5-Omni 7B, and YuE also run at Q5_K_M using 6 GB. Magicoder-S-DS 6.7B fits at Q5_K_M using 5.7 GB.

If you want to run larger models, you must use CPU offloading. This process splits the model between your graphics card and your system RAM. Offloading allows you to run Llama 3.1 8B, which needs 6.4 GB at Q4_K_M and requires 8.4 GB of system RAM. Chroma 8.9B needs 6.5 GB at Q4_K_M and 8.5 GB of system RAM. Gemma 2 9B requires 8 GB at Q4_K_M and 10 GB of system RAM. Offloading makes larger models run, but it reduces processing speed because system RAM is much slower than GDDR6 memory.

Other models that require CPU offloading include Nemotron Nano 9B, GLM-4 9B, Yi-Coder 9B, GLM-4-9B-Chat, and GLM-4V-9B. These models need 6.6 GB at Q4_K_M and 8.6 GB of system RAM. Mochi 1 10B needs 7.3 GB at Q4_K_M and 9.3 GB of system RAM. Open-Sora 2.0 11B needs 8.1 GB at Q4_K_M and 10.1 GB of system RAM. You must also remember the 4k context caveat. Running these models at context windows larger than 4000 tokens will require extra memory, which will force more data into your system RAM and slow down generation.