The GPU sizing equation

Phase 0 · Fix the language

Count the same thing on both sides

A gigabyte is 10⁹ bytes; a gibibyte is 2³⁰ bytes. An 80 GB nameplate converts to 74.51 GiB arithmetically, but packaging, firmware reservations, ECC mode and driver reporting mean runtime-visible capacity must be queried rather than inferred from the label. Network links are commonly quoted in gigabits/s while memory links use gigabytes/s. Thus 400 Gb/s is 50 GB/s raw line rate, not 400 GB/s, before overhead.

The minus one matters because the first token ends TTFT. More important: an average ITL cannot establish a p99 E2E objective. Say whether “tokens/s” means per request, replica, GPU or fleet. The vLLM metrics reference separates queue, prefill, generation and inter-token components.

Arrival rate also does not equal concurrency. Under stable steady state, Little’s Law gives average in-flight requests:

It does not size a burst buffer or prove a tail SLA. Preserve the joint trace of arrival gaps, input length, output length, prefix identity and priority. Correlations—long prompts that also generate long answers—are where averages fail.

Phase 1 · Reconstruct the model

Parameters are a consequence of architecture

Start from the exact checkpoint revision, not a family nickname. Record hidden width H, layers L, query heads H_q, KV heads H_kv, head dimension D, vocabulary, tied embeddings, intermediate width, expert count and experts selected per token. Usually D = H / H_q, but modern architectures can split positional and non-positional dimensions.

Llama 3.1 70B is a clean dense example: 80 layers, width 8,192, 64 query heads, 8 KV heads and head dimension 128. That 8:1 ratio is grouped-query attention (GQA). Multi-head attention has one KV head per query head; multi-query attention has one KV head shared by all query heads. See the original MQA and GQA papers.

Weight memory

This is a payload floor, not allocated memory. Quantized checkpoints also carry scales, zero points, group metadata, packing alignment and sometimes higher-precision embeddings or output heads. A 4-bit format is not automatically 0.5 bytes per parameter end to end. Sum actual tensor byte sizes, inspect the quantization config, then measure memory after load.

MoE: two parameter counts, two constraints

A mixture-of-experts model needs total parameters for residency but only a routed subset for each token’s expert MLP work. Mixtral 8×7B’s config declares eight local experts and selects two per token; the official model card reports 46.7B total and 12.9B active parameters. BF16 weight payload is 93.4 GB = 86.99 GiB, even though per-token linear FLOPs resemble a smaller dense model.

Do not turn “active parameters” directly into decode bytes. Across a batch, the union of selected experts can approach all experts; hot experts may be reused while cold experts create uneven traffic. Capacity padding, routing imbalance, shared experts and all-to-all exchange add work not represented by P_active.

Phase 2 · Account for state

The KV cache is architecture × retained tokens

For conventional MHA, GQA or MQA with uniform full attention, logical cache bytes are:

B is live sequences, S retained tokens per sequence, two represents key and value, and b_kv is bytes per cached element. This excludes page rounding, allocator metadata, alignment and temporary attention workspace.

Sliding, hybrid and paged caches

For a sliding-window layer, S_layer = min(S, W); full-attention layers retain the full sequence. Hybrid models can have recurrent or state-space layers with fixed per-sequence state. Runtime managers may group layers into common page shapes and pad to the largest group. “Context window × bytes/token” can overstate an ideal sliding cache yet understate real padded allocation.

Paged allocation maps logical blocks to non-contiguous physical blocks; it does not make KV free. The PagedAttention paper reports low waste for its evaluated allocator, while current runtimes add prefix caching, offload and hybrid policies with their own metadata and eviction behavior.

MLA is not “GQA with fewer heads”

Multi-head latent attention stores a compressed joint KV latent plus positional key state, with exact contents dependent on the implementation. The public DeepSeek-V3 config describes width 7,168; 61 layers; 128 attention heads; non-positional Q/K head dimension 128; RoPE head dimension 64; value head dimension 128; and KV latent rank 512. Its first three layers are dense; later MoE layers expose 256 routed experts, select eight, and add one shared expert. A common absorbed MLA representation has this BF16 floor:

Prefix caching changes compute demand, not the existence of cached bytes. Report token hit rate, eviction rate and prefix-length distribution; request hit rate alone is misleading.

Phase 3 · Bound the work

Prefill and decode live on different rooflines

For a dense decoder-only transformer, a useful first-order linear-layer cost is 2 × active parameters × tokens, counting a multiply-add as two FLOPs. A naive full-attention accounting adds an approximately quadratic term:

The second term covers QK scores and applying attention probabilities to V; it is not a universal model-FLOPs estimator. Causal kernels avoid work on masked positions, FlashAttention changes memory traffic, and sliding/hybrid attention, MoE routing, embeddings, norms and logits add or remove work. For Llama 3.1 70B at 8,192 tokens, this deliberately conservative arithmetic gives a 1,156.71 TFLOP linear term and 175.92 TFLOP attention term—13.2% of the simplified total.

Decode processes one new token per live sequence, reads retained KV and traverses the model. The right bound is a roofline, not a fixed efficiency percentage:

At batch one, a dense decode step may be close to streaming weight bytes once, so BW / weight bytes is a useful upper bound on per-request tokens/s. At batch B, a fused matrix multiply can reuse a weight tile across sequences. In the ideal weight-only limit, aggregate tokens/s grows toward B × BW / weight bytes, until KV reads, compute, scheduling or communication bind. “Every output token loads all weights” without a batch qualifier is wrong.

The original Roofline model frames this as FLOPs per byte. Batching moves decode rightward by amortizing weight traffic; long-context KV reads can pull it back toward bandwidth pressure. Effective compute and bandwidth must be measured for the kernel, dtype, batch and topology—brochure peaks are ceilings.

From service demand to replicas

Divide demand by measured sustainable throughput only after filtering benchmark points that meet the latency SLO. The fastest saturated run is irrelevant if p99 TTFT is 12 seconds and the target is 800 ms. Tie headroom to failure domains, uncertainty and rolling updates—not a universal constant.

Phase 4 · Map onto hardware

Parallelism changes both bytes and wires

Tensor parallelism is attractive inside a high-bandwidth NVLink/NVSwitch island. It also inserts collectives into the critical path. KV sharding is not guaranteed to scale with TP: if KV heads divide across ranks it often shards cleanly; with MQA or TP larger than the KV-head count, implementations may replicate heads or form replication groups. Decode-context parallelism can shard cache along sequence, at extra communication cost.

Pipeline parallelism places layers on stages, splitting layer-owned weights and KV but adding traversal and bubbles. Data parallelism provides independent replicas and is usually the cleanest throughput scale once one replica fits. Expert parallelism distributes expert weights but leaves attention, shared experts and dense components replicated or sharded by another axis; all-to-all traffic makes topology part of the model. The Megatron-LM paper derives intra-layer TP.

Form factor is topology

An H100 SXM module in an HGX baseboard is not interchangeable with an H100 PCIe card. NVIDIA lists up to 3.35 TB/s HBM and 900 GB/s total NVLink bandwidth for H100 SXM, with different power and interconnect characteristics for PCIe, on its H100 specification page. These are vendor peak conventions, not application throughput.

Never compare aggregate bidirectional NVLink to one-way NIC payload bandwidth. For inter-node TP or EP, measure collective bus bandwidth and tail latency across exact rail count, oversubscription, GPU-NIC affinity and message sizes. Nominal link rate is not NCCL algorithmic bandwidth.

Phase 5 · Give the runtime room

The engine owns memory the model sheet cannot see

Chunked prefill caps tokens admitted to a prefill iteration, limiting activation spikes and reducing how badly a long prompt blocks decode. It can improve ITL fairness while increasing a long prompt’s TTFT. Continuous batching packs work dynamically; the batch is a changing token budget, not a static request count. CUDA graphs reduce launch overhead but reserve capture buffers. Paged caches reduce fragmentation but block rounding and copy-on-write remain.

vLLM and SGLang are policies, not interchangeable labels

This article checked the release pages for vLLM v0.30.0 and SGLang v0.5.21, plus each project’s latest documentation, on 04 October 2026. “Latest” docs can move; pin the package or image digest and archive resolved arguments with every result.

Prefix caching helps only when prefixes repeat and remain resident. Paged allocation helps when block policy matches workload. Graphs help captured shapes and increase reserved memory. For both engines, record attention backend, quantization kernels, KV dtype, page size, maximum length, scheduler limits, speculative decoding, prefix policy, parallel axes and graph shapes.

Phase 6 · Prove the envelope

Benchmark the service you intend to operate

An analytic sheet should output constraints and uncertainty, not “predicted latency.” Use it to choose candidate topologies. Then run the same model revision, tokenizer, quantization, engine build and GPU firmware that production will use.

1. Freeze identity.Checkpoint commit and tensors; engine/container digest; driver, CUDA, kernels, NCCL; GPU SKU, form factor, clocks and topology.
2. Replay distributions.Joint input/output lengths, arrivals, prefix reuse, priorities, cancellations and multi-turn think time. Include cold and warm cache.
3. Sweep admission.Offered RPS, running requests, token budget, cache fraction, chunk size and parallel layout. Hold seed and dataset constant.
4. Observe every stage.Queue, TTFT, ITL/TPOT and E2E at p50/p90/p95/p99; input/output tok/s; KV occupancy, evictions, preemptions; HBM, SM, network and power.
5. Find sustainable capacity.Highest load meeting every SLO without an unbounded queue, OOM, thermal drift or error-rate regression during a long soak.
6. Exercise failure.Rolling update, one replica unavailable, cache cold start, burst, long-prompt outliers and network degradation.

Report confidence intervals and run-to-run variance. Separate measurements from interpolation; label multi-node extrapolation. The vLLM benchmark suite provides harnesses, but not your traffic distribution or acceptance criteria.

A final sizing checklist

The useful GPU count is the smallest tested topology whose worst relevant constraint still clears the service objective. Everything before that sentence is disciplined elimination. Everything after it is operations.