When data cannot leave your network, don't buy GPUs first — separate "it runs" from "it runs under load", and the hardware budget can differ by half
| Dimension | Architecture A: Fully on-prem | Architecture B: Hybrid | Architecture C: Internal API gateway |
|---|---|---|---|
| Data boundary | Everything stays inside: model and inference on your own servers | Sensitive data local, general Q&A via external API | Data stays internal; models can be self-hosted or external |
| Hardware cost | High: GPU servers, rack space, ops | Medium: small GPU or CPU fallback | Low to medium: gateway host plus on-demand compute |
| Quality ceiling | Limited by the local model's capability | Sensitive tasks limited locally, general tasks not | Depends on the models behind it |
| Time to launch | 2–6 weeks including procurement | 1–2 weeks | 1–3 weeks |
| Best for | Finance, healthcare, legal — strict compliance | Most internal enterprise assistants | A single entry point for many departments, with audit and quotas |
In one line: strict compliance → A, most companies → B, many departments with one entry point → C. The most common mistake is a blanket "everything on-prem" policy — pushing non-sensitive general Q&A onto a small local model, which ends up both expensive and unused.
People usually ask "how many GPUs for a 70B model?" But what actually drives hardware is peak concurrency. One GPU handles a single conversation smoothly; put ten people on it and they queue, because VRAM must hold both the weights and the KV cache of every active session.
Translate business needs into a model tier before jumping to "bigger is better":
| Tier | Typical use | Quantized VRAM reference | Hardware reference |
|---|---|---|---|
| 7B–9B | Classification, extraction, summarization, simple Q&A | ~5–8 GB | A single 16GB consumer GPU |
| 14B–32B | Enterprise assistant, support Q&A, document Q&A | ~10–24 GB | A single 24GB GPU to start |
| 70B class | Complex reasoning, long-document analysis | ~40–48 GB | Two professional cards or one 80GB card |
Note: these are single-stream weight footprints. As concurrency rises, the KV cache takes additional VRAM — especially with long contexts.
Concurrency is not "how many employees you have" — it is how many people are waiting for a reply in the same minute. A practical approach:
Example: a 200-person department with 20 peak online users yields roughly 3–4 concurrent requests; with a 1.5x buffer you size for 6 concurrent streams — not 200.
Model weights, inference service, vector store and logs all live inside the network, so there is physically no outbound data path. This is the common choice for finance, healthcare and legal scenarios.
Key points:
The trade-off is explicit: the quality ceiling is locked to the local model, procurement takes time, and someone must own operations.
Route requests by sensitivity: customer data, contracts and financial records go to the local model, while general knowledge Q&A and copy polishing go to an external API. You hold the compliance line without buying the most expensive GPUs for general capability.
Risk: one routing mistake means data leaving the network. Before launch, run a "sensitive data mis-release" test with synthetic samples.
Whether the backend is a local model or an external API, every request passes through an internal gateway. The gateway owns four jobs: authentication, quotas, audit, routing.
The gateway only pays off at scale: overhead for a single team, essential once three to five departments are on board.
Symptom: single-stream tests pass, then the service crashes minutes after users arrive via IM.
✅ Fix: budget VRAM for the KV cache separately, load-test at target concurrency before choosing card count, and cap the maximum context length per request so one long conversation cannot exhaust VRAM.
Symptom: procurement sized for "70B needs two cards", then 20 simultaneous users queue for tens of seconds.
✅ Fix: estimate peak concurrency first, then total VRAM = weights + concurrency × per-stream KV cache. For high concurrency, several smaller cards are often more cost-effective than one large card.
Symptom: a new model version performs worse, but the old weights were overwritten and only a backup file remains.
✅ Fix: store model files in versioned directories and reference the version in config rather than a path; keep the previous weights so rollback is a one-line change; run a fixed regression question set before every switch.
Symptom: as the knowledge base grows, inference slows down and is eventually killed by the OS.
✅ Fix: deploy the vector store separately, cap the inference service's memory with headroom, and run index rebuilds during off-peak hours.
Symptom: you suspect someone asked about sensitive content, but logs only show request counts, not attribution.
✅ Fix: log user identity, timestamp, model, latency and a redacted request summary at the gateway; store audit logs separately with a retention policy so they are not rotated away with application logs.
Symptom: it works well at launch, then nobody maintains it and both the model and the knowledge base go stale within six months.
✅ Fix: write "quarterly model evaluation, monthly knowledge-base refresh, monthly log spot-check" into the operations plan at launch, with a named owner. On-prem deployments without operations are usually abandoned within six months.
| # | Item | Pass criteria |
|---|---|---|
| 1 | Data classification | Tiers defined, and every tier has an approved model path |
| 2 | Concurrency load test | Passes at peak concurrency × 1.5 with no OOM and no timeout pile-up |
| 3 | Context cap | Maximum request length enforced, with a truncation strategy for long conversations |
| 4 | Model versioning | Versions traceable; rollback requires only a config change |
| 5 | Auth and quotas | Per-person or per-department keys, quota caps, instant revocation on offboarding |
| 6 | Audit logs | Can answer "who, when, which model, what was asked (redacted)" |
| 7 | Sensitive-data mis-release test | Synthetic sample run shows no sensitive content reaching external channels |
| 8 | Operations plan | Frequency and owner defined for model evaluation, knowledge-base refresh and log spot-checks |
The hard part of private deployment is not "getting the model to run" — it is getting concurrency, compliance and operations right at the same time. Buy too much hardware and you waste budget; buy too little and the business stops using it. Get the data boundary wrong and it becomes a compliance incident. Skip the operations plan and you end up with idle machines within six months.
If you are evaluating a private deployment, start with two artifacts: a data classification table and a peak-concurrency estimate. Once those two exist, the architecture choice and hardware budget are mostly determined.