Guide · Local AI
How much memory do you need for local AI?
Start with the weights, then add headroom
A useful lower-bound calculation is parameter count × bits per weight ÷ 8. An idealized 8-billion-parameter model at 4 bits uses 4GB for weights; 32B uses 16GB; 70B uses 35GB. These are arithmetic examples, not measured requirements. Real formats include scales, metadata and tensors stored at different precisions. GB is decimal; GiB is binary, so compare the same units.
Check the exact artifact size and runtime documentation next. Reserve additional memory for the KV cache, temporary buffers, image encoders and any draft model. Longer context and more simultaneous requests can change a setup that fits into one that runs out of memory. Training adds gradients and optimizer state; inference estimates do not size a training machine.
Hugging Face explains quantization and its backend requirements. A label such as “4-bit” describes storage precision, not universal hardware compatibility or a fixed quality level.
MoE: active parameters are not the download size
Qwen3-30B-A3B lists 30.5B total parameters but 3.3B active per token. Budget resident weights using the full model artifact, not just the active count. Expert offloading is a separate runtime choice with its own memory and speed tradeoffs. See the Qwen model card.
Use memory tiers to narrow the shortlist
- 8–12GB: consider smaller quantized artifacts first. Check the RTX 3060 12GB against your actual requirement.
- 16GB: more room for weights or context, with options such as the RTX 5060 Ti 16GB. It is not a guarantee for every 30B model.
- 20–24GB: compare the 3090, 4090 and 7900 XTX, checking software support separately.
- 32GB: the 5090 adds capacity; decide whether your workload needs it.
- 48–96GB: workstation options include the A6000 and RTX PRO 6000 Blackwell. Capacity alone does not establish throughput or compatibility.
Shared memory and multiple GPUs
On a Mac, the operating system and applications share unified memory with the model. Installed memory is not all available to inference. Two GPUs also do not automatically become one larger card: the runtime must split the workload, and communication, cache placement and per-device limits matter.
A practical acceptance check
- Record the model revision, artifact, runtime version and backend.
- Set the context and concurrency you will actually use.
- Run a representative long prompt and generation; check peak memory and whether layers spill to CPU.
- Measure prompt processing and generation separately. If testing before purchase is impossible, find a reproducible report matching that configuration and buy with suitable return terms.
FAQ
Is 24GB enough for local AI?
For some workloads, yes. Check the exact model artifact, context, runtime and concurrency. There is no universal parameter-count cutoff for a 24GB card.
Sources and review
Hardware advice reviewed October 4, 2026. These are documentation-based recommendations; we have not benchmarked these configurations.
- Ollama hardware and operating-system support
- Ollama context and memory guidance
- MLX LM model support and memory controls
- llama.cpp multi-GPU configuration
Compare GPU prices · Estimate memory needs · Compare Mac and GPU builds