Best local AI models for NVIDIA RTX A6000

The largest models that fit fully

The 30 largest of the 186 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

How to read this

The NVIDIA RTX A6000 graphics card features 48 GB of GDDR6 memory. This large frame buffer allows you to run highly capable artificial intelligence models entirely on local hardware. When choosing a model, the total memory usage must remain under this 48 GB limit to ensure fast processing speeds.

The best quantization column shows the highest quality version of each model that fits within the hardware limits. Quantization reduces the size of model weights to save memory. A Q8_0 quantization level represents an eight bit format which preserves excellent output precision. For larger model architectures, a Q6_K or Q5_K_M quantization is used to fit the model parameters into the available space.

Models like Jamba 1.5 Mini / Large at 52B parameters fit on this card using a Q5_K_M quantization which consumes 44.3 GB of memory. The Llama 3.1 Nemotron 51B model also runs efficiently at Q5_K_M quantization using 43.5 GB. Mixtral 8x7B fits at Q6_K quantization using 46.2 GB of memory.

Many powerful models fit at the maximum Q8_0 quantization level. These include Seed-OSS 36B using 45.8 GB, Qwen3.6-35B-A3B using 44.5 GB, and Command R (35B) using 44.5 GB. You can also run Yi 1.5 9B / 34B, Granite Code 3B to 34B, LLaVA 1.5 / 1.6 (7B to 34B), and Ovis 2 at Q8_0 quantization with each using 43.2 GB of memory.

Other excellent options at Q8_0 quantization include DeepSeek-Coder 1.3B / 6.7B / 33B using 42 GB and WizardCoder 33B using 42 GB. OTel 2.0 LLM 31B IT uses 44.9 GB. Qwen3 8B / 14B / 32B, Qwen3.5 (dense variants), Aya Expanse 8B / 32B, Granite 4.0 Small/Tiny, and Qwen2.5-Coder 0.5B to 32B all use 40.7 GB. Gemma 3 27B and Wan 2.2 / 2.5 use 34.3 GB.

Running models entirely on the graphics card avoids CPU offload. If a model exceeds the onboard memory, parts of the model must load into system RAM. CPU offloading dramatically slows down generation speeds because system memory buses are much slower than GDDR6 graphics memory.

All memory calculations are based on a standard 4k context window. If you increase the context window to process longer documents, the active memory usage will grow. You must leave a safety margin in the 48 GB memory pool to accommodate this extra context data during active generation.