Estimate VRAM first, pick hardware second — the reverse order buys the wrong thing
When planning local LLM hardware, the instinctive question is "which GPU should I buy." That question comes too early. The card is only a carrier for VRAM — what actually decides the design is three capacity numbers:
Compute these three numbers first, then pick hardware. Doing it in reverse is how teams end up with the wrong purchase.
Weights are the easiest item to calculate and the one most often forgotten:
weight VRAM ≈ parameters × bytes per parameter × 1.1 (quantization metadata + alignment overhead)Bytes per parameter come from precision: FP16/BF16 is 2 bytes, INT8 is 1 byte, INT4 is 0.5 bytes. The 1.1 multiplier leaves room for scale factors and group metadata.
| Model size | FP16 / BF16 | INT8 | INT4 |
|---|---|---|---|
| 1.5B – 3B | ~3.3GB – 6.6GB | ~1.7GB – 3.3GB | ~0.9GB – 1.7GB |
| 7B – 8B | ~15.4GB – 17.6GB | ~7.7GB – 8.8GB | ~3.9GB – 4.4GB |
| 13B – 14B | ~28.6GB – 30.8GB | ~14.3GB – 15.4GB | ~7.2GB – 7.7GB |
| 30B – 32B | ~66GB – 70.4GB | ~33GB – 35.2GB | ~16.5GB – 17.6GB |
| 70B – 72B | ~154GB – 158.4GB | ~77GB – 79.2GB | ~38.5GB – 39.6GB |
This table explains an apparent contradiction: "a 7B model is lightweight" and "a 7B model won't run" are both true. The first statement is about weights; the second is about weights plus a 32K context, several concurrent requests, and the inference framework itself.
Weights are a fixed cost. KV cache is a variable cost that grows linearly with context length:
KV cache ≈ 2 (K and V) × layers × attention heads × head dim × context length × bytes per element × concurrencyFor mainstream dense architectures this means going from a 4K to a 32K context multiplies KV cache by eight. Add concurrency and VRAM demand can double or more. This is why "let's just try a larger context" is often where VRAM incidents begin.
Three techniques stack well and all reduce KV cache pressure:
Three cost buckets are routinely underestimated in real deployments:
The safe rule is to hold target usage at or below 85% of physical VRAM. A plan that fills a 24GB card to 23GB often survives a benchmark and OOMs under realistic concurrency.
Applying the estimates to procurement gives three tiers. Prices and specific models shift with the market, so only capacity targets are listed here.
| Tier | Typical use | VRAM target | RAM target | Model scale it supports |
|---|---|---|---|---|
| Proof of concept | Internal testing, prompt debugging, single evaluator | 16–24GB | 32–64GB | 7B–8B at INT4 / INT8, short context |
| Department rollout | One department in daily use, tens of users | 48–80GB (or 2×24GB) | 128GB and up | 13B–32B at INT4/INT8, moderate context |
| Production | Cross-department service, high availability and audit needs | 80GB × 2 and up | 256GB and up | 70B at INT4, or multiple 32B instances |
Selection works backwards: define the model scale and context length the business can accept → compute VRAM from the formulas above → then choose the card. Choosing the card first and figuring out what it can run afterwards is backwards.
For the broader private deployment roadmap — data boundaries, network isolation, acceptance criteria — see How to Choose a Private AI Deployment Plan.
| Approach | How it works | What it costs you |
|---|---|---|
| Quantization | Compress weights to INT8 / INT4, and quantize the KV cache when needed | Accuracy loss, most visible on long-context and multi-step reasoning |
| Multi-GPU sharding | Split the model across cards by layer, or use tensor parallelism | Inter-GPU communication overhead; sensitive to interconnect bandwidth, higher latency |
| CPU offload | Keep the layers that do not fit in system memory and swap them in on demand | PCIe bandwidth is far below VRAM bandwidth — throughput can drop by an order of magnitude |
Recommended order: quantize first, add cards second, treat offload as a last resort. Offload exists to let a small model boot; it is not a production throughput strategy. PCIe transfers between system memory and VRAM typically run in the 10–30GB/s range, one to two orders of magnitude below the hundreds of GB/s available inside VRAM.
For serving patterns — service endpoints, concurrency control, authentication — see AI Agent Integration Architecture.
Parameter count only sets the weight cost. Real requirements add KV cache, framework overhead, and activations — in long-context scenarios the last two can be the same order of magnitude as the weights. Compute weights first, then add headroom.
Quantization loss is task-dependent. On simple Q&A, classification, or summarization the difference is often negligible; on multi-step reasoning, code generation, and long tool chains it degrades noticeably. Run one comparison pass on your own task set before cutting hardware.
Cross-card communication is a real cost. Layer sharding is bandwidth-sensitive, and a PCIe-connected multi-GPU setup is often less reliable and simpler-feeling alternatives lose to — a single card with large VRAM. Judge by "is this request bound by compute or by data movement," not by card count.
| # | Check | Pass criterion |
|---|---|---|
| 1 | Target model and precision are fixed | Named model and quantization level, no "we'll figure it out later" |
| 2 | Weight footprint is computed | Parameters × bytes × 1.1 written down as a number |
| 3 | KV cache sized against real context length | Calculated at the target context ceiling, not the default |
| 4 | Concurrency is confirmed | Peak concurrent users ÷ average handling time = required concurrent instances |
| 5 | VRAM headroom is preserved | Peak usage ≤ 85% of physical VRAM |
| 6 | RAM and disk planned for peak | Includes model switching, quantization conversion, and temp files |
| 7 | A real task-set comparison was run | Pre- and post-quantization results compared on your own tasks |
Item 4 is the one most often skipped, yet it is what decides whether you need another card. Concurrency is not user count, it is simultaneous request count. Ten people taking turns asking questions and ten people asking at once have completely different VRAM requirements.
Local LLM hardware planning does not need to be complicated, but it does need a derivation chain: business scale → model scale → precision → weight VRAM → add KV cache → add runtime headroom → choose the card. Skip any link and you get either "it does not fit" or "it fits but collapses under load."
Estimate before you purchase, and validate on real tasks before you go live. Once those two are done, most of the risk in hardware selection is gone; what remains is model and price comparison.