AGIWORK / USEFUL TOOLS

LLM decode speed
calculator.

Explore how model size, memory and context affect token generation. Compare scenarios before benchmarking on your own hardware.

MODEL & HARDWAREMEMORY ESTIMATERUNS IN YOUR BROWSER

INTERACTIVE ESTIMATE

Start with a scenario.

Select an example, then adjust the values to match your model and hardware. Results update as you type.

Weights MoE model

Weights size, GB (quantized) Shared part W^shared, GB

Total experts, E Active, k

KV cache

Layers KV heads (GQA) Head dim

KV precision Context L, tokens

For MLA-style caches, use an equivalent cache size only when it matches your model and runtime; implementations can differ.

Memory

VRAM / unified, GB/s Capacity, GB

RAM, GB/s Capacity, GB

SSD, GB/s (random read) SSD capacity is not modelled

Greedy placement: KV cache, then shared weights, then experts fill VRAM / unified memory, RAM and SSD in that order.

Mode and corrections

η, share of rated BW t₀, ms per step

Batch B S, tokens per pass

How to interpret this estimate

This is a simplified memory-bandwidth model of the decode phase. It assumes sequential memory reads, a fixed per-step overhead, even expert selection, and greedy placement of KV cache, shared weights and experts across memory tiers. It does not model prompt processing, GPU compute limits, actual disk capacity, operating-system memory use, contention, latency or runtime-specific optimisations. Speculative decoding uses the accepted tokens per pass supplied above. Measured throughput can differ substantially; verify on your own hardware and workload.

Sizes use decimal GB (10⁹ bytes). Bandwidth and memory capacities are approximate inputs, not hardware guarantees. Batch throughput assumes similar-length requests; the expert estimate reflects the expected fraction touched across a batch.

Calculation: per-request tokens/s = accepted tokens per pass ÷ (fixed overhead + sum of read size ÷ effective bandwidth for each tier). Aggregate throughput = batch size × per-request throughput. KV bytes per request = 2 × layers × KV heads × head dimension × bytes per element × context length.

Mixture of experts: the expected fraction of routed experts read for a batch is ρ = 1 − (1 − k/E)^B, where E is the number of experts, k is the number active per request and B is the batch size. This assumes independent, uniformly distributed expert selection. Each tier’s estimated read is its KV cache and shared weights, plus ρ times its routed expert weights.

LLM decode speedcalculator.

Start with a scenario.

How to interpret this estimate

LLM decode speed
calculator.