Best local AI models for NVIDIA RTX A2000

The largest models that fit fully

The 30 largest of the 114 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX A2000 is a compact workstation graphics card equipped with 6 GB of GDDR6 memory. This onboard memory size dictates the maximum size of the artificial intelligence models you can run entirely on the hardware. To run a model locally without slowdowns, the model weights and active memory must fit within this 6 GB frame.

The quantization column indicates the compression level applied to each model. Quantization reduces the precision of model weights to save space. For example, the Q4_K_M quant represents a four bit medium quantization. This allows larger models like the Granite 3.3 8B, Ministral 8B, or InternLM 3 8B to fit into 5.9 GB of video memory. Other models like Qwen2.5 7B or Falcon 3 7B can run at a higher Q5_K_M quantization using exactly 6 GB of video memory.

When a model exceeds the local video memory, you must use CPU offloading. This process splits the workload between your graphics card and your system RAM. For instance, running Llama 3.1 8B at Q4_K_M requires 6.4 GB of memory, which uses your 6 GB of video memory and 8.4 GB of system RAM. Offloading allows you to run larger models like Gemma 2 9B or Mochi 1 10B, but it significantly reduces processing speed because system RAM is much slower than GDDR6 memory.

The memory calculations for these models assume a standard four thousand token context window. As your conversation or input text grows, the model requires more memory to track the history. Running close to the 6 GB limit on models like the Zephyr 7B Beta or Command R7B means that long conversations might exceed your video memory and cause performance slowdowns.

For specialized tasks, the RTX A2000 can handle diverse architectures within its memory limits. You can run coding assistants like Magicoder-S-DS 6.7B using 5.7 GB of memory at Q5_K_M quantization. Vision models like MiniCPM-V 2.6 and image generators like Stable Diffusion 3.5 Large also fit into 5.9 GB of video memory at Q4_K_M quantization.