LLM Decode Roofline

The physical ceiling on decode throughput. Generating one token re-reads the whole model from memory, so tokens/s is set by memory bandwidth, not FLOPS. Open-source companion to LeCompute.

Model (params read / token)

Precision

GPU

GB read / token

ms / token (floor)

tokens / s (floor)

GPU	TB/s	ms/tok	tok/s

The math

At batch 1, decode is memory-bound: ms/token = bytes_read_per_token / memory_bandwidth, and tokens/s = 1000 / (ms/token). Bytes read per token ≈ the weights (params × bits ÷ 8) plus the KV cache. Compute never binds here — that's why an H100 and a B200 with the same Tensor Cores differ in decode tok/s almost exactly by their bandwidth ratio (3.35 vs 8 TB/s).

It's a floor. Real serving is slower (batching, overhead); nothing makes it faster than this.
MoE models read only their active experts per token — pick the active-param preset, not the total size.
Prefill is the opposite (compute-bound) — this floor is for decode, which dominates real serving time.

Will the model even fit? VRAM calculator. Full mechanism: why LLMs are memory-bound.