Measure four different latency signals
A single latency number hides where users are waiting. Streaming chat can feel fast with a low TTFT even when the full completion takes longer, while batch extraction may care only about end-to-end duration.
- Time to first token: request start to first streamed token.
- Generation throughput: output tokens divided by generation time.
- End-to-end duration: request start to completed response.
- Reliability: successful responses divided by total attempts.
Control the benchmark workload
Use a fixed prompt set that represents the intended workload. Record model ID, provider route, region, request parameters, input tokens and requested output length.
- Run warm-up requests before recording measurements.
- Randomize provider order to reduce time-of-day bias.
- Repeat tests during more than one traffic window.
- Keep streaming and non-streaming results separate.
Use distributions, not a leaderboard screenshot
Report median, p90 and p95 values with sample size. A route with the fastest median but frequent extreme delays may provide a worse production experience than a slightly slower but more stable route.
{
"sample_size": 100,
"ttft_ms": { "median": 640, "p95": 1840 },
"output_tokens_per_second": { "median": 42.1 },
"end_to_end_ms": { "median": 5120, "p95": 9080 },
"success_rate": 0.98
}The values above are an illustrative reporting format, not a live CometAPI benchmark result.
Publish methodology with every report
A benchmark is useful only when a reader can understand what was measured and reproduce the approach. Include limitations and explain whether the provider route was pinned or automatically selected.