Best local AI models for NVIDIA RTX A3000 Laptop

The largest models that fit fully

The 30 largest of the 114 models that fit; every smaller model in the catalog fits too. Best quant means the highest quality compression whose weights and 4k context both sit inside the memory.

Close, but only with CPU offload

These need more than the card holds at their smallest practical quant, so part of the model runs from system memory (figures assume 32 GB of it). They work, several times slower.

How to read this

The NVIDIA RTX A3000 Laptop GPU features 6 GB of GDDR6 memory. This dedicated video memory determines the maximum size of the artificial intelligence models you can run entirely on your graphics hardware. To run a model smoothly without performance drops, the model files and the active context data must fit within this 6 GB limit.

The quantization column indicates the compression level applied to each model. Quantization reduces the precision of model weights to save memory. For example, the Q4_K_M quant represents a four bit medium quantization that balances file size and intelligence. The Q5_K_M quant uses five bits to deliver higher accuracy but requires more memory space.

Several high quality models fit completely inside your 6 GB video memory. You can run Granite 3.3 8B, Ministral 8B, InternLM 3 8B, OpenCoder 8B, Seed-Coder 8B, MiniCPM-V 2.6, Idefics 3 8B, Fuyu-8B, Emu3, and Stable Diffusion 3.5 Large at the Q4_K_M quantization level using 5.9 GB of memory. EXAONE 3.5 7.8B and Mistral 7B also fit comfortably at Q4_K_M using 5.7 GB of memory.

You can also run many 7B models at the sharper Q5_K_M quantization level which uses exactly 6 GB of memory. This includes Qwen2.5 7B, OLMo 2 7B, Falcon 3 7B, Command R7B, OpenHermes 2.5, Zephyr 7B Beta, OpenChat 3.5, Starling LM 7B, Codestral Mamba 7B, CodeGemma 7B, aiXcoder-7B, Nxcode, Janus-Pro 7B, Ruyi-Mini-7B, Qwen2-Audio 7B, Qwen2.5-Omni 7B, and YuE. Magicoder-S-DS 6.7B fits at Q5_K_M using 5.7 GB of memory.

If you want to run larger models, you must offload some layers to your system RAM. Assuming your laptop has 32 GB of system RAM, you can run Llama 3.1 8B at Q4_K_M which needs 6.4 GB of memory and 8.4 GB of system RAM. You can also run Chroma, Nemotron Nano 9B, GLM-4 9B, Yi-Coder 9B, GLM-4-9B-Chat, GLM-4V-9B, Mochi 1, and Open-Sora 2.0. Offloading layers to system RAM allows you to run these larger models but it reduces your generation speed significantly.

Keep in mind that these memory calculations assume a standard 4k context window. If you increase the context window to process longer documents or chat histories, the memory requirement will grow. Running close to your 6 GB limit means that large context inputs might exceed your video memory and slow down your system.