HPE LLM Inference Estimator
Sizing estimate: the HPE server and GPUs your models and users need, with each model running within one server. Does not cover a model spread over several servers, or NVL rack-scale systems.
A single-page tool for estimating what LLM inference needs. Pick a model, the number of concurrent users, a workload and a speed goal, and it recommends the node estimated to meet the goal with the least GPU memory in total. Add up to four models and it sizes them together on one kind of board:
- Memory: model weights, KV cache, activations and overhead, per GPU and in total, in GB or GiB.
- Speed: estimated tokens per second per user and time to first reply while new prompts keep arriving, each with a likely range measured against published benchmarks, including thinking time for models that reason before answering.
- Capacity: how many GPUs of each board one node takes, and how many users it can handle: what fits in memory, and what also reaches the speed goal on the estimate.
- Server (optional, beta): the CPUs, system memory and local storage around those GPUs, following NVIDIA's recommendations for certified inference servers.
It covers 40+ popular open models (Qwen, Gemma, Llama, DeepSeek, gpt-oss, GLM, Kimi, Mistral and others) on NVIDIA L4, L40S, H200 NVL, HGX H200, HGX B200, HGX B300, RTX PRO 4500/6000 Blackwell, and AMD MI350P and MI355X.
Single node
Every result is one server of up to 8 GPUs. If one node of a board can't serve every user, its row says how many it does serve. For production, plan at least one more node (or replica) than this, so the service keeps running if one fails or is being updated.
GPU counts round up to what can be bought. H200 NVL servers take 1, 2, 4 or 8 cards, but an NVLink bridge joins at most 4 of them, so one copy of a model spans at most 4 and 8 cards run two copies. Pipeline across bridge groups in Advanced settings (off by default) lets one copy span both groups as two pipeline stages, tensor-parallel within each and handing activations between them over PCIe (vLLM's --tensor-parallel-size 4 --pipeline-parallel-size 2): that fits a model or a KV cache that one group can't hold, at about one group's speed per user, since each stage still reads all of its weights for every token; two copies of 4 are still preferred when they fit. Pipelined figures are modeled, not yet fitted to a benchmark. The HGX H200, HGX B200 and HGX B300 are sold only as 8-GPU systems, with all 8 GPUs joined through NVSwitch. So is the AMD MI355X, but with no switch: each GPU has one Infinity Fabric link to each of the other 7, so a copy split across 2 or 4 of them uses only the links among them and pays more per layer than on an HGX. RTX PRO 4500 servers hold up to 5 cards (the limit on HPE servers).
A copy is split only where the engine can split it: vLLM divides a layer's attention heads evenly over the tensor-parallel GPUs, so the query heads must divide by the split, and the KV heads must too (or the split by them). Phi-4's 10 KV heads run on 1 or 2 GPUs, and Qwen2.5 7B (28 heads) and GLM-4.7 Flash (20) on up to 4.
Servers view
The answer opens by server. Servers in scope picks the HPE servers to consider (all by default; the customer's standard builds, say). The page sizes the model on every configuration of those servers — each server with each GPU it takes, at the count and wiring HPE supports — and shows:
- the recommendation: the configuration the GPU view would suggest — not split over PCIe, then the least GPU memory, not pipelined, fewer GPUs per copy, faster — then the most room and the fewest rack units, with its build (chassis, GPUs, bridges, CPUs, memory, storage, power), figures and how the model is laid out on the cards;
- room to grow: on the recommendation, what the same server serves with its open bays filled;
- every server in scope: the configurations that hold the model, with the smallest server that also works tagged Smaller server and bigger ones with more room tagged More room; for the rest, the most GPU memory they carry against what the model needs;
- one configuration in detail, folded away under its name until opened with the arrow: how its cards are wired, every layout the wiring allows (TP within an NVLink group or NVSwitch; TP within groups × PP across them, when Pipeline across bridge groups is on; on PCIe cards, TP over PCIe or PP over PCIe) at each number of copies, and its build.
GPUs per copy, when fixed in Advanced settings, and Pipeline across bridge groups limit the layouts here as they do in the GPU view.
Several models are sized together, as the GPU view does: each model runs as copies of its own, the copies are packed into servers of one kind with one kind of GPU, and spare GPUs run extra copies of the models furthest from their goals. Each model must fit on one server. The recommendation follows the GPU view's order, plus one rule it never needs: on the same GPU, fewer servers win even with more cards. The card gives each server's build and every model's figures and layout; the configuration detail is for one model only.
Pipelined layouts are modeled, not yet fitted to a benchmark, and say so. The server catalog is the JSON block id="catalog-hpe" in index.html; edit it when HPE revises its QuickSpecs. URL keys: vw=gpu opens the GPU view, hs lists the servers in scope (dot-separated ids), cf the configuration in detail, mm the System memory setting.
HPE servers
In the GPU view each board is sized as in one HPE server that takes it, at that server's limits, and says so: the server the Servers view recommends for that board whenever the figures are the same there, else the first server that sells it; the Servers view compares every server, and Servers in scope narrows it (URL key hs). The list is every current HPE ProLiant and HPE Compute XD server that takes one of these GPUs:
| HPE server (US QuickSpecs version) | Boards (most cards per server) | Notes |
|---|
| ProLiant Compute DL380a Gen12 · 4U · 2 × Xeon 6 (v20) | L4, L40S, RTX PRO 6000 (8), H200 NVL | H200 NVL in two 4-way NVLink groups (S4A91C, supported only here, and not in the 10-card layout). Holds up to 10 double-wide / 16 single-wide; a node is sized at 8 |
| ProLiant Compute DL380 Gen12 · 2U · 2 × Xeon 6 (v20) | L4 (8), L40S (3), RTX PRO 4500 (5), RTX PRO 6000 (3), H200 NVL (2) | H200 NVL and RTX PRO 6000 power-capped at 450 W, 27 °C; NVLink bridges not supported, so a split copy runs over PCIe |
| ProLiant DL385 Gen11 · 2U · 2 × EPYC 9004/9005 (v49) | L4 (8), L40S (4), RTX PRO 4500 (4), RTX PRO 6000 (4), H200 NVL (4), MI350P (4) | H200 NVL with the 2-way S4A90C bridge only: a copy spans 2, or 4 as TP 2 × 2 stages. MI350P needs 450 W power capping; HPE targets shipping from December 2026 |
| ProLiant Compute DL345 Gen12 · 2U · 1 × EPYC 9005 (v13) | L4 (6), L40S (4), RTX PRO 4500 (4), RTX PRO 6000 (2) | L4: 6 in the rear of the standard chassis, 4 in the front of the GPU chassis |
| ProLiant Compute DL340 Gen12 · 2U · 1 × Xeon 6 (v16) | L4 (4), L40S (4) | front GPU chassis; the 4x kit needs a Rich I/O 6xx1P |
| ProLiant Compute ML350 Gen12 · 4U tower · 2 × Xeon 6 (v18) | L4 (4), L40S (2), RTX PRO 4500 (4) | external GPU fan kit for 30 °C |
| ProLiant ML110 Gen11 · tower · 1 × Xeon (v36) | L4 (2) | CPU limited to 150 W; 25 °C |
| ProLiant Compute DL320 Gen12 · 1U · 1 × Xeon 6 (v21) | L4 (4), L40S (2) | GPU server needs a Rich I/O 6xx1P; above 270 W, the closed-loop liquid cooling fan kit |
| ProLiant Compute DL325 Gen12 · 1U · 1 × EPYC 9005 (v13) | L4 (4) | front of the GPU server |
| ProLiant Compute DL360 Gen12 · 1U · 2 × Xeon 6 (v20) | L4 (3) | low-profile, rear slots |
| ProLiant DL365 Gen11 · 1U · 2 × EPYC 9004/9005 (v46) | L4 (2), RTX PRO 4500 (2) | GPU CTO model |
| ProLiant DL145 Gen11 · 2U edge · 1 × EPYC 8004 (v23) | L4 (3), L40S (1), RTX PRO 4500 (1) | |
| ProLiant Compute XD685 · 5U DLC / 6U air · 2 × EPYC 9005 (v16) | HGX H200, HGX B200, HGX B300, MI355X | whole 8-GPU systems; H200 air-cooled in 6U or liquid in 5U; B200, B300 and MI355X liquid-cooled only, in 5U |
|
On Auto each board sits in the first server above that lists it. That matches the board's own figures, except the MI350P, which is now sized at the DL385 Gen11's 4 cards. Gen11 servers with a Gen12 successor that takes the same GPUs are left out. HPE also lists the RTX PRO 4500 and 6000 for the EL2000 EL240 Gen12 and Edge Server e930t without a card count, so those are named beside the picker but not sized. Storage, Superdome, Compute Scale-up and EL8000 systems are out of scope.
Cooling is set per configuration, since the GPU decides it on some servers. Every PCIe server here is air-cooled. The XD685 runs the HGX H200 air-cooled in 6U or with direct liquid cooling in 5U, and the HGX B200, B300 and MI355X only with direct liquid cooling, in 5U; the XD690 runs the HGX B300 air-cooled in 10U. Where both exist, the page sizes and counts the air-cooled build, which needs no facility water. Each build has a Cooling row, flagged when the configuration is liquid-cooled only (it needs facility water and a coolant distribution unit), naming any server that runs the same GPUs air-cooled; the server tables show each configuration's chassis and cooling, and rack units, and so the tie-break on the fewest, follow the configuration (cooling in the catalog).
Sources, all US: HPE's NVIDIA Accelerators for HPE QuickSpecs (version 83, 8 September 2026) for which servers take which GPU and bridge (the XD690 and the Gen13 servers, which it doesn't list, from their own QuickSpecs), and each server's US QuickSpecs (version in the table; link, version and date in the catalog's quickspecs field) for card counts and constraints. The sizer is for the US; confirm a build against the current US QuickSpecs.
Separately, one copy of a model is split across 1, 2, 4 or 8 GPUs. Serving engines need the split to divide the model's attention heads evenly, so this is a software rule that applies to every board. On a 5-card RTX PRO 4500 server, one copy uses at most 4 cards and the fifth can run another copy. When the GPUs bought are more than one copy needs, as with a small model on an 8-GPU HGX system, they run more copies rather than one copy split wider, unless only the wider split reaches the speed goal: copies leave room for far more users, and it is how small models are served in production.
Speed under load: each concurrent user is taken to send a new request as soon as the last is answered, so new prompts keep arriving while replies stream. Reading them takes a share of the engine's time, in steps shared with everyone's next token, so speed per user is the average over prompt reads and the time between them, and the first token includes the average wait for other users' prompts queued ahead (each user has one request at a time, so a single user never waits). Output, all users is the replies finished each second times their length, which is less than users × speed per user, since each request streams only after its wait and its prompt read. Speed on this setup shows the request rate, how busy reading prompts keeps the engine, and the speed between prompt reads. Long new prompts (RAG, long documents) cost the most; chat barely notices.
Prompt caching is modelled the way vLLM and SGLang run by default: the repeated part of each prompt (a shared system prompt, earlier turns) is reused, so only the new part is read before the first reply, and the shared part is stored once per GPU. That presumes a warm cache, so every recommendation says so, with how much of each prompt it reads (512 of each 2,048-token Chat prompt, by default). Prompts not yet in the cache take longer: turn caching off in Advanced settings to size for them. With 16 Chat users on Llama 3.1 8B, that changes the suggestion from an RTX PRO 4500 to an L40S.
Several workloads: one model often serves more than one kind of request at once, such as 16 chat users alongside 10 RAG users. Add another workload gives each its own number of concurrent users and its own prompt and reply lengths, up to four in all. They share the same copies of the model: memory adds up across them, and because one decode step serves the whole batch, every user streams at the same speed. Each workload has its own first-token time: the headline shows the slowest, and the speed goal is checked against it. Every workload's prompts join one queue, so each first token is the same average wait for prompts ahead plus the time to read that workload's own prompt. Workloads that need a different model are a second model (below).
Several models: Add another model sizes up to four models together, each with its own users, workloads, precision and speed goal. Tabs above the controls switch between them; overhead, prompt caching and Server apply to all. Each model runs as copies of its own on GPUs of its own, sized as it would be alone but before rounding up to a server size. The copies of every model are then packed into servers of one kind of board, the widest first, so every server but the last is full; a server that isn't full shrinks to the smallest size the board comes in, and its spare GPUs run extra copies of the models closest to their goals. Each model then uses the fewest GPUs per copy that still meet its goal, and still gets at most one node. The suggestion is the board where every model meets its goal with the least GPU memory in all. A Models panel shows how each model runs and the GPUs of every server; with Server set to NVIDIA guidance, each server is sized for the models on it, its storage holding each one's weights once.
Thinking models: the Thinking switch under Reply length sets how many tokens a model thinks before answering. Users see the thinking says which wait the speed goal is judged on. On (the default), the UI streams the thinking, so it's the first token. Off, users wait for the answer, so the tile becomes Answer starts and the goal's limit applies to that: within 15 s for Fast, 30 s for Smooth chat and 60 s for Readable, limits of the estimator's own, since a first reply in 1 to 5 s is out of reach once thousands of thinking tokens come first.
RAG pipelines: the estimate covers the LLM only. RAG also needs an embedding model and usually a reranker, which aren't sized. Small ones (under ~1B parameters) add a few GB and can share a GPU or run on CPU at low traffic; a 4–8B reranker, or heavy traffic, needs its own GPU.
Not sure how many concurrent users to plan for? Estimate from daily traffic turns people, requests per day and hours of use into a concurrent-user count.
Server, by NVIDIA's guidance (beta)
This part is in beta: its rules and figures are still being worked on and may change.
Set Server to NVIDIA guidance in Advanced settings to size the whole server, following the inference server table in NVIDIA's Certified Systems Configuration Guide. GPU counts then round up to 2, 4 or 8 per server, split evenly across two CPU sockets (2 or 4 for the RTX PRO 4500). That can change which board is suggested: 56 chat users on Llama 3.1 8B move from 1 × RTX PRO 6000 to 2 × L40S. A Server panel under the answer gives what goes around those GPUs:
- CPUs: two sockets with PCIe Gen5 and a base clock of 2.0 GHz or more. Cores follow the AI worker nodes HPE builds for HPE Private Cloud AI: 32 physical cores per GPU in a 2- or 4-GPU server and 16 per GPU in an 8-GPU one (2 × 32-core CPUs for 2 GPUs, 2 × 64-core for 4 or 8), and 16 per GPU at any count for the L4 and RTX PRO 4500. NVIDIA's own floor is 6 per GPU. In the Servers view the build starts with one CPU, so the GPUs, memory and serving process share one NUMA node, and goes to two only when it must: the server takes two processors only (DL380a Gen12, XD685), a GPU sits in a slot wired to CPU 2 (from each server's riser tables in its US QuickSpecs), or the cores won't fit in the largest CPU the server takes. With two, the GPUs split evenly across the sockets. A build whose cores exceed what the server's CPUs offer is flagged.
- Recommended minimum system memory: twice the GPU memory, as NVIDIA recommends. In the GPU view, how it's made up from memory modules is left to the vendor. In the Servers view the build makes it from the server's own RDIMMs. Memory is costly, so by default it does not fill every channel: it takes the smallest modules the server takes, as few as reach the minimum, one per channel, so that much memory lands on as many channels as it can. A larger module is used only when the smallest can't reach the minimum at one per channel, and two per channel only when one can't. System memory in Advanced settings (URL key
mm) trades cost against memory speed: Least memory (mm=least) takes the fewest GB that reach the minimum, putting two DIMMs on some channels where that is cheaper, which slows the memory (2 × RTX PRO 6000 in a DL380 Gen12: 384 GB instead of 512; 8 × H200 NVL in a DL380a Gen12: 2,304 GB instead of 4,096); One per channel is the default above; Every channel (mm=all) fills every channel, as NVIDIA's guide asks, for the full bandwidth. Each CPU holds the same DIMMs, a count HPE's population rules allow. The row says how many channels that fills and, when not all, flags it with roughly the share of the CPU's memory bandwidth it leaves; fill more channels if the CPU offloads KV cache or does heavy work of its own. Each server's channels, DIMMs per channel and module sizes come from its US QuickSpecs (the catalog's mem field). The DIMM counts per CPU (pop) come from the server's table in HPE's memory population rules for its processor family (rules). Xeon 6 servers take 1, 2, 4, 8, 12 or 16 DIMMs per CPU, with no 6, so a 6-channel build there becomes 8.
- Slots and switches: each GPU in an x16 slot of its own PCIe generation, behind PCIe Gen5 switches when there are 8.
NVIDIA's guide also covers network cards, remote management and TPM; the page leaves those to the conversation with the customer. The HGX H200, HGX B200 and HGX B300 come as whole systems from their manufacturer, so their CPUs, memory and storage are left to its spec sheet. The AMD boards are left out while Server is set to NVIDIA guidance, since the guide is written for NVIDIA GPUs; deleting guide:false from a board's entry in GPUS brings it back, sized by the same rules as a stand-in (the MI355X, a whole 8-GPU system, is left to its spec sheet like the HGX systems). These are planning minimums, not a certification: a server is NVIDIA-Certified only if it is on NVIDIA's list. With Server left on GPUs only, the GPUs are sized as if the option were not there.
Use it
Open index.html in a browser. It is one self-contained file with no build step and no server.
Run it in a container
The image is nginx plus that one file, about 20 MB, running as a non-root user on port 8080.
docker compose up --build
Then open http://localhost:8080. The same image works anywhere a container runs:
docker build -t llm-inference-estimator .
docker run --rm -p 8080:8080 --read-only --tmpfs /tmp llm-inference-estimator
A prebuilt image for amd64 and arm64 is published to GHCR by the Container workflow on every merge to main (:latest and :sha-<commit>) and on every v* tag (:1.2.3, :1.2):
docker run --rm -p 8080:8080 --read-only --tmpfs /tmp ghcr.io/nvme101010/llm-inference-estimator:latest
/healthz answers 200 ok for health checks. The page loads its two fonts from Google Fonts; without internet access it falls back to system fonts and everything else works offline.
Kubernetes
deploy/k8s holds a Deployment (2 replicas, non-root, read-only root filesystem, no capabilities, readiness and liveness probes on /healthz, 10m CPU and 16 Mi memory requested), a Service on port 80, a PodDisruptionBudget, and an optional Ingress. Apply it from the repository or straight from GitHub:
kubectl apply -k "https://github.com/nvme101010/LLM-Inference-Estimator//deploy/k8s?ref=v1.11.10"
Then port-forward to try it (kubectl port-forward svc/llm-inference-estimator 8080:80) or set the host in ingress.yaml and add it to the kustomization. The image runs two nginx workers regardless of node size, so the 32 Mi limit holds everywhere.
Share a sizing
- Download summary saves a one-page report of the recommendation, the inputs, every board compared and the assumptions. Open it and choose Print to save it as a PDF.
- Copy link appears when the page is served from the web, for example GitHub Pages. The address holds every setting, so the link opens the same scenario. With several models, model 1's settings keep their keys, models 2 to 4 use the same keys with the model's number appended (
m2, u2, …), and e says which model's tab is open; a one-model link is as it always was.
Both record the estimator's version (v= in the link). A link made with an older version opens with a note listing what has changed in the figures since, and the image tag to run to see them as they were (ghcr.io/nvme101010/llm-inference-estimator:<version>). Links made before versions were recorded are treated as 1.0.3.
When a release changes what the figures come to, bump VERSION in index.html, add a line to CHANGES saying what changed, and tag the release with the same version.
Accuracy
Model and board data as of October 2026. These are planning estimates, not benchmarks. Memory figures follow each model's published architecture, and board memory is the size on the box, with 90% treated as usable (vLLM's default limit).
Speed and first-token times come from a roofline model: each board's datasheet memory bandwidth and dense compute, times the shares of them serving reaches, plus vLLM's fixed costs per step and per request. Those shares and costs are fitted to 295 published benchmark points (NVIDIA NIM and SemiAnalysis InferenceX: Llama 3.1 8B, Llama 3.3 70B and gpt-oss-120b on H200, B200 and L40S), with a further factor for AMD's software stack fitted to MI355X runs. The figures are averages over steady traffic at the server: they leave out the network round trip, and bursts of traffic make the wait for a first token longer.
Measured error. Every figure here comes from benchmarks the fit did not see: each group of benchmarks was predicted by a fit made without it.
- Speed per user lands at 1.02 × measured (80% of points between 0.80 and 1.30, 77% within ±25%). The page shows that spread as the likely range, and judges the speed goal at its low end, so a setup it passes over-promised by more than 25% on 3 of 295 benchmark points.
- First token with nothing queued lands at 1.03 × measured (80% between 0.65 and 1.49), and the goal is judged at the slow end of that range. The wait for other users' prompts under load is modelled; no benchmark here tests it.
- AMD: with the ROCm factor fitted on one model's MI355X runs, the other model's land at 0.99 × measured (80% between 0.85 and 1.25).
- Memory: where the page says a load won't fit, the engine held requests back for want of cache space in 18 of 20 cases.
Boards with no benchmark of their own (L4, the RTX PRO cards, H200 NVL, B300, MI350P) rest on the same fitted shares applied to their datasheets. The page marks their speed and first-token figures extrapolated, in the headline and the board table, and the summary stars them. Real results vary with the serving engine and its settings: before committing to hardware, confirm with a load test at your request rate (for example vllm bench serve against vLLM or SGLang).
Method, the fitted constants, sources, caveats and every point: validation. After a change to the model, refit with node validation/fit.mjs --apply and recheck with node validation/compare.mjs --write. node validation/sweep.mjs lists which sizings changed since the last release.
