Best local AI models for NVIDIA CMP 40HX

The largest models that fit fully

The 30 largest of the 123 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA CMP 40HX is a dedicated mining card that repurposes well for local artificial intelligence. It features 8 GB of GDDR6 memory. This onboard memory size determines the maximum size of the neural network you can run entirely on the graphics hardware. Running models completely inside the video memory ensures the fastest possible processing speeds.

To fit larger models into the 8 GB limit you must use quantized versions. Quantization reduces the precision of model weights to save space. The best quantization column shows the optimal compromise between model accuracy and memory usage. For example Mochi 1 is a 10B model that fits using the Q4_K_M quantization which consumes 7.3 GB of memory. Gemma 2 9B fits at Q4_K_M using exactly 8 GB of memory.

Several 9B models fit comfortably using the Q5_K_M quantization which requires 7.7 GB of memory. These include Nemotron Nano 4B / 9B, GLM-4 9B / GLM-4.5-Air, Yi-Coder 1.5B / 9B, GLM-4-9B-Chat / CodeGeeX4, and GLM-4V-9B / GLM-4.1V-Thinking. Chroma is an 8.9B model that fits at Q5_K_M using 7.6 GB of memory. Llama 3.1 8B also fits at Q5_K_M using 7.4 GB of memory.

Many 8B models can run at a higher quality Q6_K quantization which uses 7.9 GB of memory. This group includes Granite 3.3 2B / 8B, Ministral 3B / 8B, InternLM 3 8B, OpenCoder 1.5B / 8B, Seed-Coder 8B, MiniCPM-V 2.6 / MiniCPM-o 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large / Turbo. EXAONE 3.5 2.4B / 7.8B uses 7.7 GB at Q6_K. Mistral 7B uses 7.4 GB at Q6_K. Other 7B models like Qwen2.5 0.5B / 1.5B / 3B / 7B, OLMo 2 1B / 7B, Falcon 3 1B / 3B / 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, and Codestral Mamba 7B use 6.9 GB at Q6_K.

When a model exceeds the 8 GB video memory you must offload layers to the system RAM. This offload process allows you to run larger models but it reduces generation speed. We assume a system with 32 GB of system RAM. Open-Sora 2.0 is an 11B model requiring 8.1 GB at Q4_K_M and 10.1 GB of system RAM. FLUX.1 dev is a 12B model requiring 14.4 GB at FP8 / optimized and 16.4 GB of system RAM.

Other 12B models require 8.8 GB at Q4_K_M and 10.8 GB of system RAM. These are Gemma 3 12B, Gemma 4 12B, Mistral NeMo 12B, Pixtral 12B, FLUX.1 schnell, FLUX.1 Kontext dev, and FLUX.1 Krea dev. Vicuna 13B requires 9.5 GB at Q4_K_M and 11.5 GB of system RAM. Note that memory calculations assume a standard 4k context window. Expanding the context window beyond 4k tokens will require more memory and may force you to use lower quantizations.