Best local AI models for NVIDIA RTX A2000 12GB

The largest models that fit fully

The 30 largest of the 147 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX A2000 graphics card features 12 GB of GDDR6 memory. This dedicated memory size determines which artificial intelligence models can run entirely on your hardware. For local execution, the model weights must fit inside this video memory to ensure fast processing speeds. If a model exceeds this limit, your system must use slower system memory which reduces performance.

To fit larger models into the 12 GB limit, developers use quantization to compress the files. The best quant column indicates the highest quality compression level that still fits within your video memory. For example, the DeepSeek-Coder-V2 16B model fits at the Q4_K_M quantization level using 11.7 GB of memory. Similarly, the Phi-4 model fits at the Q5_K_M quantization level using 11.9 GB of memory, while the Gemma 4 12B model can run at the higher quality Q6_K quantization level using 11.8 GB of memory.

When running these models, you must consider the memory required for context. The listed memory usage figures represent the base model weights. Running a model with a standard 4k context window requires additional memory space. If you use the maximum available video memory for the model weights, you may experience out of memory errors when processing long conversations or large documents.

If you want to run larger models, you can use CPU offloading. This technique splits the model between your graphics card and your system memory. To use this method, your computer needs sufficient system RAM. For example, running the FLUX.1 dev model requires 14.4 GB of video memory at FP8 quality, which means you must offload some data to your system RAM, requiring at least 16.4 GB of system RAM to function.

Other large models also rely on CPU offloading to run on this hardware configuration. The Solar Pro model requires 16.1 GB of memory at the Q4_K_M quantization level and needs 18.1 GB of system RAM. The Codestral 22B model also requires 16.1 GB of memory at the Q4_K_M quantization level and needs 18.1 GB of system RAM. Using CPU offloading allows you to run these advanced systems, but the processing speed will be slower than running models entirely on the graphics card.