The 9 LLM Inference Metrics That Tell You If AI Is Fast

Posted on by
Contents show
LLM inference

A high token throughput does not necessarily mean a fast AI application. The first token may take too long to arrive, or the stream may stall midway under concurrent load, and none of that shows up in a single throughput number. The nine key LLM inference metrics cover that gap, and knowing which one to read is the difference between fixing a problem and guessing at it: they split one request into a first-token wait and a streaming phase, then describe how a server behaves across many requests and how many of them actually meet your targets.

This guide covers each of the nine LLM inference metrics with its formula and the practical question it answers, the distribution statistics that turn single-request numbers into service-level truth, and the definition traps that make benchmark comparisons misleading if you do not check them.

Reading an LLM inference request timeline

Four values define nearly every latency metric, and laying them on a timeline makes the whole family of metrics intuitive.

The request leaves the client at t_send, the first output token arrives at t_first, the last token arrives at t_last, and the reply contains N_out tokens. Those timestamps cut the request into two segments: before the first token, the request may wait in a queue and then the server runs prefill, processing the prompt to produce the first output token. After the first token, the server runs decode, emitting tokens one step at a time.

Every LLM inference metric below measures one of those segments or the whole span, and every throughput metric measures the server across a window of many such requests. The two phases of LLM inference matter separately because they have different bottlenecks: prefill is compute-bound and grows with prompt length, while decode is memory-bandwidth-bound and grows with output length, which is why modern serving systems disaggregate them onto different hardware.

1. Time to first token (TTFT)

TTFT is the delay between sending the request and receiving the first output token.

Its formula is simply the first-token arrival minus the send time. It includes network travel, any wait in the server queue, and the full prompt-processing step, so it grows with server load and with prompt length. Long-context applications, agents that carry whole documents in their prompts, and retrieval systems that inject large retrieved blocks all push TTFT up before a single output token exists.

Interactive chat applications watch this LLM inference number most closely, because it sets how long the user stares at a blank screen before anything responds. A streaming interface that shows the first word quickly feels responsive even if the full reply takes a while, which is why TTFT is usually the metric to optimize first for user-facing products.

2. End-to-end latency (E2E)

E2E latency is the total LLM inference wait: from sending the request to receiving the last token.

It is the sum of the first-token wait and the streaming time, and it is the number a user experiences as how long the whole answer took. For non-streaming applications, an internal API that returns a complete JSON response, E2E is the only latency that matters, because there is no visible stream.

E2E inherits every problem in both phases: a slow queue, a long prefill, or a sluggish decode each push it up, so it is the metric to watch for total user experience but the hardest one to debug, because it aggregates three different causes.

3. Token generation time

Token generation time is the duration of the LLM inference streaming segment: how long the server takes to deliver every token after the first.

It is the difference between the last and first token timestamps, which makes it exactly E2E minus TTFT. It grows with the number of output tokens, so a longer reply takes proportionally longer to stream, which is why judging stream speed on this metric alone is misleading: a slow 1,000-token reply and a fast 100-token reply can post the same generation time.

Dividing it out is what the next metric does, and that division is where stream quality actually lives.

4. Time per output token (TPOT)

TPOT is the average time between LLM inference output tokens after the first one: the token generation time divided by N_out minus 1, because the first token belongs to TTFT. It is the smoothness of the stream, the pace at which text appears once it starts appearing, and for a reply with a single token it is undefined.

The definitional trap matters when comparing benchmarks. Some tools include the first-token wait in the average, some treat ITL as another name for TPOT, and the resulting numbers differ in ways that look like performance differences. Confirm the definition before comparing results across tools, because a faster-looking TPOT can simply be a different formula.

5. Inter-token latency (ITL)

ITL is the gap between two consecutive LLM inference output tokens, and unlike TPOT it keeps every individual gap as its own measurement rather than averaging them away.

For a single request, the mean of its ITL values equals its TPOT. Across many requests, the two averages weight data differently: mean ITL counts every gap once, so a few long requests dominate it, while mean TPOT counts every request once, giving short and long requests equal say. Neither is wrong; they answer different questions.

The practical value of ITL is stall detection. A stream that produces a token every 30 milliseconds and then freezes for two seconds has a fine average TPOT and a terrible worst-case ITL, and users experience the freeze, not the average. If your application shows typing indicators or streams into a UI, the tail of the ITL distribution is what your users feel.

6. Mean, median, and tail percentiles

Latency varies from request to request, so one number per request becomes a distribution across many, and three statistics describe that distribution.

The mean is the average of all values and follows outliers. The median, P50, is the value half of requests fall at or below, reflecting the typical user. P99 is the value 99 percent of requests fall at or below, showing the tail, with P90 and P95 as softer versions of the same idea.

Two rules make these numbers honest. Apply all three separately to TTFT, TPOT, and E2E, because a service with excellent median latency and terrible P99 feels broken to one user in a hundred, which is a real complaint rate at scale. And compute percentiles from the raw samples of all servers combined, because an average of per-server P99 values is not the P99 of the combined traffic, a mistake that flatters multi-server deployments.

7. Requests per second (RPS)

RPS counts the LLM inference requests the server completes per second of wall-clock time: completed requests divided by the measurement window.

It measures capacity, and it ignores how much work each request carries, which is the trap. A server posting 50 requests per second on short prompts and short replies is doing far less work than one posting 5 requests per second on document-sized contexts, so RPS is only comparable across runs with the same mix of prompt and output lengths. Changing the workload changes the number, and the change is not a performance change.

8. Tokens per second (TPS)

TPS counts the tokens the server processes per second across all active LLM inference requests, and it has an output form and an input form that must be stated separately.

Output TPS sums the generated tokens across requests and divides by the window; input TPS does the same for prompt tokens. The distinction matters because a workload with long prompts and short replies can post a high input TPS while producing few output tokens, and quoting the higher number without the label is how capacity claims mislead. Prefill-heavy serving infrastructure, the kind optimized for document ingestion, lives on high input TPS; chat serving lives on output TPS.

TPS is the capacity metric to pair with RPS: RPS says how many users the server can serve, TPS says how much work it is doing while serving them, and both are needed to reason about scaling.

9. Goodput

Goodput is the rate of LLM inference requests successfully completed within your service-level objectives, and it is the metric that turns the rest of the family into a business number.

An SLO is an LLM inference latency limit, such as a ceiling on TTFT or TPOT, and a request counts as good only when it satisfies every limit. Counting the good requests in a window and dividing by its length gives goodput, which never exceeds RPS, and the gap between them is the rate of requests that missed at least one target.

The term has a second meaning in the research literature. The DistServe paper, which introduced prefill-decode disaggregation into modern serving, uses goodput for a capacity: the maximum request rate a GPU sustains within its latency constraints. The two definitions agree in spirit, both count requests that meet quality bars, but the serving-benchmark version measures a run and the paper version measures a ceiling.

Goodput is the metric that justifies spending on infrastructure, because throughput without quality is inventory you cannot sell: a server completing 100 requests per second where 30 exceed the latency SLO delivers 70 good requests, and the economics of the deployment should be judged on the 70.

How to use the metrics together

The nine LLM inference numbers form a diagnostic sequence rather than a leaderboard, and the order matters when something feels slow.

Start with E2E to confirm there is an LLM inference problem. Split it into TTFT and token generation time to find which phase owns the delay: a high TTFT points at the queue and prefill, a long generation time points at decode. Check TPOT and ITL to distinguish a uniformly slow stream from one that stalls, and look at the P99 alongside the median to learn whether the problem affects everyone or the unlucky tail. Then move to RPS and TPS to see whether load is the cause, and finish with goodput to express the damage in units your business can act on.

The same sequence works for LLM inference capacity planning in reverse: set SLOs first, measure goodput at increasing load, and the load where goodput starts falling below RPS is the capacity your deployment can honestly claim.

For readers building on the serving systems these metrics measure, the routing layer that decides which model handles which request is examined in our AI gateway comparison, the model pricing that makes goodput an economic number is covered in our Gemini 4 Argon cost breakdown, and the fast desktop-work model whose perceived speed depends on TTFT is analyzed in our GPT-6.1 Terra piece.

The LLM inference prefill and decode split behind the numbers

The reason these metrics split so cleanly into two families is the architecture of LLM inference itself, and understanding it explains why single optimizations often fail.

In LLM inference, prefill processes the entire prompt in parallel and produces the first token; it is compute-bound, meaning it saturates the GPU's matrix units, and its duration scales with prompt length. Decode generates one token per step, each depending on the last; it is memory-bandwidth-bound, meaning the bottleneck is reading the model's weights and cached context rather than computing, and its pace barely changes with prompt length but dominates long outputs.

Because the phases of LLM inference have different bottlenecks, batching them together forces compromises: a decode-heavy request in the same batch as a prefill-heavy one slows both, which is the insight behind the disaggregated serving that DistServe introduced and that production systems have adopted. Prefill runs on compute-optimized hardware, decode on memory-bandwidth-optimized hardware, and the metrics let operators size each pool against the phase it serves.

For anyone tuning an LLM inference deployment, the practical metric map is: TTFT problems live in prefill and the queue, TPOT and ITL problems live in decode, and TPS input figures measure the prefill side while output TPS measures decode. The metrics are not just reporting; they are the instrumentation that tells you which hardware pool to grow.

Common LLM inference measurement mistakes

The LLM inference metrics themselves are simple; the measurement practices around them are where accuracy dies, and four mistakes account for most misleading numbers.

The first is averaging away the tail. Reporting a mean latency without a P99 hides the worst user experience, and at scale the tail is where complaints and cancellations concentrate. Every latency number should ship with its distribution, and the median-to-P99 ratio is the quickest health check a deployment has.

The second is mixing server populations. Percentiles computed per server and then averaged are not the percentiles of the combined traffic, because the tail of a combined distribution is driven by whichever server had its bad moments during the window. Raw samples from all servers, pooled before percentile math, is the correct method.

The third is unlabeled throughput. A TPS number without its input-or-output designation, or an RPS compared across different workload mixes, reads as performance when it is really a change in the test. The fix is cultural: every quoted number carries its definition, and any benchmark comparison starts by confirming both sides measure the same thing.

The fourth is ignoring the SLO when reporting throughput. A deployment posting strong RPS with a third of requests over the latency ceiling is delivering less service than a slower deployment that stays inside its SLO, and goodput is the metric that exposes the difference. Teams that track RPS alone optimize for the wrong number, and the correction costs nothing but a change in what the dashboard shows.

Worked example: every LLM inference metric on one request

A concrete LLM inference request makes every formula tangible, and one worked example covers the entire family.

Suppose a request leaves the client at t_send, the first token arrives 400 milliseconds later, the last of 300 output tokens arrives 4.4 seconds after the send, and the reply streams smoothly. The metrics compute as follows: TTFT is 400 milliseconds, the blank-screen wait. E2E is 4.4 seconds, the whole wait. Token generation time is 4.0 seconds, the streaming segment. TPOT is 4.0 seconds divided by 299 gaps, roughly 13.4 milliseconds between tokens, or about 75 tokens per second of stream speed.

The percentiles need many requests, so suppose the deployment logs 10,000 requests with a median TTFT of 380 milliseconds and a P99 TTFT of 2.1 seconds. The typical user sees a first token in under half a second; one user in a hundred waits two seconds staring at a blank screen, and that gap is where the next optimization project should point, likely at queue behavior or prefill capacity on the busiest servers.

On the server side, suppose the window is one minute and the deployment completed 1,200 requests producing 900,000 output tokens. RPS is 20, output TPS is 15,000, and if the SLO requires TTFT under one second and TPOT under 50 milliseconds, goodput counts only the requests meeting both. If 1,100 of the 1,200 qualify, goodput is 18.3 requests per second, and the 1.7 requests per second gap is the SLO miss rate, which is the number that justifies the next capacity purchase or tuning pass.

The example shows the LLM inference division of labor: TTFT and TPOT describe one request, percentiles describe the fleet, RPS and TPS describe the window, and goodput converts all of it into the service level the business actually sells.

Questions readers are asking

What is the difference between TPOT and ITL?

TPOT is the average gap between output tokens for one request; ITL is each individual gap measured separately. They match for a single request but weight long and short requests differently across many, and ITL is where stream stalls show up.

Which metric should a chat application optimize first?

TTFT, because it sets how long the user stares at a blank screen. A fast first token makes a streaming reply feel responsive even when the full answer takes longer.

Why is goodput lower than throughput?

Goodput counts only requests that meet every latency SLO, so the gap between goodput and RPS is the rate of requests that finished but finished too slowly to count as service.

Are TPOT numbers comparable across benchmarking tools?

Not without checking definitions, because some tools include the first-token wait in the average, some use ITL and TPOT interchangeably, and the differences look like performance differences.

Sources
[fluentform id="2"]