"Sub-100ms AI" is an incomplete performance claim. It could mean 80 milliseconds to classify one image, 80 milliseconds until the first token appears, or 80 milliseconds measured inside a model server while the user waits much longer. Those are different systems and different user experiences.
A defensible real-time inference target names a specific journey, a latency metric, a percentile, and a production load. This guide shows how to define those terms, run a benchmark that survives scrutiny, and optimize the part of the stack that is actually slow.
Quick answer: sub-100ms can be realistic for some small models, local tasks, cached results, and time to first token on carefully configured systems. It is not a universal threshold for AI, and it does not mean a full generative response arrives in 100 milliseconds.
Real-time is a product requirement, not one number
A camera pipeline, autocomplete feature, speech interface, and long-form assistant do not need the same response shape. A classifier returns one result. An autoregressive language model returns a sequence, usually streamed token by token. Comparing their raw "inference time" hides more than it reveals.
| Workload | What the user or system waits for | Useful measurement |
|---|---|---|
| Image or tabular classification | A complete prediction | End-to-end latency at the required percentile |
| Autocomplete | A short suggestion before it becomes irrelevant | End-to-end latency, cancellation rate, quality |
| Streaming language model | First visible output, then a steady stream | TTFT, inter-token latency, total latency |
| Voice interaction | End of speech through first audible response | Turn latency across speech, model, and audio stages |
| Batch processing | Completion of the whole job | Throughput, cost, deadline success |
Write the service-level objective in one sentence. For example: "At the expected weekday load, at least 99 percent of attempted chat requests should be admitted and complete successfully; among successful requests, p95 TTFT must stay within the product target while output quality remains above the approved threshold." Report rejected, failed, and timed-out requests separately so admission control cannot make the latency chart look better by dropping work. The target values belong to the product team. They should come from user research, upstream deadlines, or a system safety analysis, not a generic blog post.
Safety-critical control systems need domain-specific engineering, validation, and standards. A language-model benchmark is not evidence that an automotive, medical, or industrial system is safe.
The latency metrics you need to name
Time to first token
Time to first token (TTFT) is the interval from sending a request until the client receives the first non-empty generated token. NVIDIA's current LLM benchmarking guide notes that TTFT generally includes network latency, queuing, and prefill. Longer input usually increases prefill work because the model must process the prompt before decoding can begin.
TTFT is the closest model-serving metric to "when does something appear?" It still may not equal the product's perceived latency. A web application can add authentication, retrieval, tool calls, moderation, rendering, and its own network hop before or after the model server.
Reasoning models need one more distinction. NVIDIA's AIPerf metrics reference defines time to first output (TTFO) as the wait for the first non-reasoning output token, separately from TTFT, which may include an internal reasoning token. Name the metric that matches what the user can actually see.
Inter-token latency and time per output token
Inter-token latency (ITL) measures the gaps between generated tokens. Time per output token (TPOT) is often calculated over the decode period after the first token. Tools do not always use identical definitions, so record the tool and version. A low TTFT followed by a slow stream still feels slow.
For a response with more than one output token, NVIDIA AIPerf defines average ITL as:
ITL = (end_to_end_latency - TTFT) / (output_tokens - 1)
That average can hide pauses. If a smooth stream matters, use a tool that exposes delivery gaps and name its definition. SGLang reports per-token ITL percentiles; AIPerf reports inter-chunk latency, where one streamed chunk can contain more than one token.
A published result makes the difference concrete. In NVIDIA's NIM 1.8 performance table, Llama 3.3 70B on two H100 80 GB GPUs with FP8 and tensor parallelism 2 recorded 47.77 ms TTFT and 18.91 ms ITL at concurrency 1 for 500 input and 2,000 output tokens. Applying the definition above gives an estimated full-response latency of about 37.85 seconds:
47.77 ms + (1,999 x 18.91 ms) = 37,848.86 ms
This is not a prediction for another deployment. It is an illustration of why a first-token number below 100 ms can coexist with a response that takes tens of seconds to finish.
End-to-end latency
End-to-end latency ends when the complete response reaches the client. For a simple request, you can model it as:
client latency = app work + network + queue + prefill + decode + post-processing
For an agent or retrieval system, add every model call, search, tool, retry, and approval step on the critical path. For parallel branches, measure their coordination overhead and the slowest branch instead of summing them as sequential work.
Throughput, concurrency, and goodput
Throughput reports completed requests or tokens over time. Concurrency is the number of requests in flight. Higher concurrency can improve hardware utilization and total throughput while making each user wait longer. In serving tools such as vLLM and AIPerf, goodput is the completed request rate that meets the configured latency SLOs. Task quality is a separate gate and should be reported beside goodput.
This is why a maximum tokens-per-second result and a low-latency result may describe different operating points. Publish the load used for each one.
Why AI latency benchmarks disagree
Two benchmark charts are comparable only when their workload and measurement boundaries match. Check at least these variables:
- Model and revision: architecture, parameter count, tokenizer, context configuration, and any adapters.
- Precision and quantization: the exact format, calibration method, kernels, and quality change against the chosen task.
- Hardware: accelerator, count, memory, interconnect, CPU, and whether resources are shared.
- Software: serving engine, version, drivers, runtime, scheduler, and enabled optimizations.
- Input and output lengths: averages alone are weak. Report a distribution or fixed buckets.
- Load shape: closed-loop concurrency is not the same as requests arriving at a controlled rate.
- Streaming: a non-streaming endpoint cannot provide meaningful client-side TTFT.
- Cache state: prefix cache hits, response cache hits, warm weights, and cold starts must be identified.
- Measurement boundary: in-process kernel time, server time, and remote client time answer different questions.
- Percentile and duration: p50 can look healthy while p99 is unusable. Short runs may miss queue buildup and autoscaling events.
MLPerf Inference addresses this by defining distinct scenarios and required quality checks. Its official inference rules include preprocessing and post-processing in a benchmark run and apply scenario-specific latency requirements. Even without reproducing MLPerf, the same discipline applies: define the scenario before publishing the number.
A benchmark plan you can reproduce
1. Define the user-visible boundary
Instrument the client, application, and model server with correlated request IDs and monotonic timestamps. Decide whether the test begins at a UI action, at your application gateway, or at the model endpoint. For product decisions, prefer the widest boundary you control.
2. Build a representative workload
Sample real, consented, and sanitized traffic when possible. Otherwise create a synthetic set that matches expected input lengths, output lengths, languages, media sizes, tool use, and cancellation behavior. Keep a separate quality evaluation. A faster system that returns worse answers has not necessarily improved.
3. Freeze the configuration
Record the model identifier and revision, serving image or commit, hardware, precision, tensor or pipeline parallel settings, maximum context, sampling settings, batch policy, cache settings, region, and benchmark client version. Save the configuration beside the raw results. Remove credentials, redact sensitive inputs and outputs, and set access and retention limits for the benchmark artifacts.
4. Separate warm and cold tests
Run enough warmup requests to initialize kernels and caches before the steady-state test. Measure cold starts separately if autoscaling or scale-to-zero is part of the architecture. Mixing the two into one unexplained average makes the result hard to use.
5. Sweep arrival rate or concurrency
Begin below the expected load, then increase it until latency rises sharply, errors appear, or the quality constraint fails. This reveals the saturation point. Test the expected peak and a failure scenario, not only concurrency one.
Current tools expose the necessary metrics. The stable vLLM benchmark CLI reports TTFT, TPOT, ITL, throughput, and percentiles. SGLang's official serving benchmark guide covers controlled rates, concurrency limits, streaming, and JSONL output. NVIDIA's current NIM guide documents an AIPerf workflow for compatible endpoints. Pick one client, pin its release or commit, and keep the sanitized raw output.
6. Report distributions and quality
At minimum, report attempted, admitted, successful, rejected, failed, and timed-out request counts; request rate; input and output token distributions; and p50, p95, and p99 TTFT, ITL or TPOT, and end-to-end latency. Add throughput, accelerator utilization, cost per attempted and successful request, and the quality score that applies to the task.
| Field | What to record |
|---|---|
| Test identity | Date, code revision, model revision, region |
| Workload | Dataset version, request count, arrival pattern, length distributions |
| System | Hardware, server version, precision, parallelism, cache and batch settings |
| Latency | p50, p95, p99 TTFT, ITL or TPOT, and end-to-end latency |
| Capacity | Accepted request rate, token throughput, utilization, errors |
| Quality and cost | Evaluation result, rejected and failed cases, cost per attempted and successful request |
Optimize the bottleneck you measured
Network and application overhead
Place the service near users or near the data source, reuse connections, stream responses, and remove serial application calls that are not required. Measure from a realistic client location. Running the load generator on the model host isolates the server, but cannot predict internet latency.
Queue time and admission control
When offered load exceeds capacity, requests wait even if the model itself is fast. Add timeouts, queue limits, backpressure, and admission control. Scale before saturation if the cost model allows it. A dashboard should separate queue time from compute time so the team does not try to fix overload by rewriting prompts.
Prefill
Reduce unnecessary input, avoid repeatedly sending static text, and test prefix caching when requests share a long prefix. Retrieval should return the smallest authoritative context that still passes quality checks. Chunked prefill or separating prefill and decode may help some workloads, but each changes scheduling behavior and must be measured under the intended load.
Decode
Use a smaller suitable model, a faster precision format, optimized kernels, or better hardware when decode is the limit. Cap output length only when the product can tolerate truncation. Speculative decoding can reduce token-generation latency, but its benefit depends on acceptance rate, draft cost, model, and load. vLLM's own speculative decoding results vary by dataset and configuration, evidence against copying a headline speedup.
Batching
Batching often raises total throughput, but waiting to form a batch consumes latency budget. NVIDIA Triton's batcher documentation recommends measuring the default behavior, then increasing batch size or queue delay only while latency remains within budget. Continuous or inflight batching can reuse slots as requests finish, which matters most when generated sequence lengths vary.
When a sub-100ms claim is credible
A sub-100ms figure is credible only when the author states all of the following:
- which operation completed, such as classification, embedding, first token, or full response;
- where the timer started and stopped;
- the model revision, hardware, software, precision, and number of devices;
- input and output lengths plus cache state;
- the load, sample count, run duration, and percentile;
- the quality threshold and whether the configuration passed it.
Without that context, "7 ms on an H100" and "80 ms real-time AI" are marketing fragments, not architecture guidance. A first-token measurement cannot be compared with a full voice turn. A batch-one result cannot establish capacity for hundreds of simultaneous users. A cached response cannot establish model latency.
On-device inference can remove a network round trip and keep some data local, but its speed depends on the exact model, device, thermal state, runtime, and workload. Our on-device AI guide explains how to separate local capability from cloud fallback and what to test before choosing the architecture.
Four latency decisions to resolve before shipping
Treat 100 ms as a product target, not a human limit
The figure is sometimes used as a user-interface rule of thumb, but it is not a universal boundary for every action or modality. Expectations change with feedback, task, device, and response shape. Measure the actual user journey and give immediate interface feedback when the underlying work takes longer.
TTFT and token rate describe different waits
For streamed chat, both matter. TTFT controls the initial wait; ITL or per-user token rate controls how the response unfolds. Total system throughput determines capacity and cost. Report all three at the expected load.
Quantization needs target-hardware evidence
Results depend on hardware support, kernels, memory pressure, batch shape, and conversion method. Quantization may improve speed or capacity, but it can also change quality. Benchmark the quantized artifact on the target hardware and rerun the task evaluation.
Compare edge and cloud end to end
Edge removes some network delay, while a cloud accelerator may execute the model much faster than a client device. Privacy, availability, update strategy, power, and cost also matter. Compare complete architectures with the same task and quality threshold.
