AI agent token costs grow when each step repeatedly processes instructions, conversation history, tool results, and intermediate state.
Reduce token volume with run-level budgets, tool-result filtering, context compaction, retry limits, and controlled reasoning. Use prompt caching for stable repeated input, but optimize the agent loop before switching to a cheaper model.
The most useful production metric is cost per successful task, measured across the complete run—not price per request or the context size of the final call.
This guide focuses specifically on multi-step AI agents. It explains how repeated context compounds across a run, how to identify the largest source of waste, and which controls to implement first.
Introduction
A chatbot may make one model request per user message. An AI agent can make 10, 20, or more calls before completing one task.
Reviewed for clarity, source attribution and current API terminology.
Each step may resend instructions, conversation history, tool results, and intermediate state. Retries, reasoning, and subagents add more usage, so a short final answer can still consume a large number of tokens.
As usage scales, these costs become harder to predict and can quickly reduce product margins. Lowering them requires optimizing the full agent loop—not simply switching to a cheaper model.
This article concentrates on Agent-specific costs. For a broader guide covering prompt caching, exact response caching, semantic caching, model routing, and general API cost management, see How to Reduce AI API Costs.
Why Do AI Agent Token Costs Compound?
In a multi-step agent, the cost of one task is the sum of every model call—not only the final response.
The main sources of Agent Token usage are:
Cost source
What causes it
First control to test
Repeated instructions
System prompts, tool schemas, policies, examples
Stabilize the reusable prefix
Growing history
Earlier turns are resent at each step
Compact or selectively retrieve state
Tool results
Search pages, files, logs, and database records
Filter before adding them to context
Intermediate output
Plans, status messages, and verbose tool decisions
Use compact structured outputs
Reasoning tokens
High reasoning effort on routine steps
Match effort to task complexity
Retries
Invalid output, timeouts, tool errors, and rate limits
Classify failures and cap retries
Subagents
Workers duplicate context, tools, and analysis
Send each worker a narrow context slice
There are two distinct ways to reduce the bill:
Process fewer tokens through filtering, compaction, output limits, and loop controls.
Reduce the effective price of necessary tokens through prompt caching or model selection.
Key distinction: Prompt caching lowers the cost of repeated input. Context compaction reduces the repeated input itself.
How Can a 12-Step Agent Process 147,000 Tokens?
Consider a hypothetical support agent with:
A 4,000-token stable prefix
1,500 new tokens added after each step
The complete accumulated history resent on every request
The final call contains only 20,500 input tokens, but the full run processes 147,000 cumulative input tokens.
Now apply two controls:
Cache the stable 4,000-token prefix after the first call.
Compact the history after step six into a 2,500-token state summary.
Scenario
Uncached input
Cached input
Total processed input
Change
Full history on every step
147,000
0
147,000
Baseline
Stable prefix cached
103,000
44,000
147,000
Same volume, cheaper mix
Cache plus compaction
64,000
44,000
108,000
26.5% fewer processed tokens
This is a planning calculation, not a provider benchmark.
It assumes that every request includes the complete accumulated history. Agents that selectively construct state, summarize old messages, or retrieve only relevant information may follow a different cost curve.
Cost-growth rule: Measure cumulative input across the full run. The final context size does not represent the total number of tokens processed.
Which Metrics Reveal Agent Token Waste?
Do not begin by changing models. First identify where the workflow is spending tokens without improving the result.
Record these fields for every Agent step:
Field
Why it matters
run_id, step_id, parent_step_id
Reconstructs the Agent and subagent tree
Rendered input tokens
Shows how context grows between calls
Cached and uncached input
Separates reuse from new context
Output and reasoning tokens
Identifies expensive generation steps
Tool result size and retained tokens
Shows how much raw evidence enters later prompts
Retry reason and attempt number
Identifies repeated failures
Compaction tokens before and after
Measures actual context reduction
Worker ID and returned tokens
Reveals duplicated subagent work
Accepted, rejected, or escalated result
Connects cost to task quality
The primary metric should be:
cost per successful task
= total workflow cost
/ accepted tasks
A cheaper run is not an improvement when it causes more failed tasks, repeated tools, or human correction.
Four Agent-specific metrics help locate the problem.
A high value indicates that earlier context has been processed repeatedly.
Tool Retention Ratio
tool retention ratio
= tool-result tokens retained in context
/ tokens originally returned by tools
A high ratio can indicate that the Agent is carrying too much raw evidence between steps.
Retry Tax
retry tax
= retry and repair cost
/ total workflow cost
Reasoning Share
reasoning share
= reasoning-token cost
/ total model cost
Measure each workload separately. Research, coding, browser, and customer-support agents should not share one global baseline.
Six Ways to Reduce AI Agent Token Costs
1. Set a Budget for the Complete Run
A per-request output limit does not control a multi-step Agent.
Set run-level limits for:
Total model steps
Cumulative input and output
Tool calls and tool-result size
Retries by failure type
Subagents
Total elapsed time or estimated cost
The following provider-neutral Python example evaluates the run before each model call:
from dataclasses import dataclass
from enum import Enum
class Action(str, Enum):
CONTINUE = "continue"
COMPACT = "compact"
STOP = "stop"
@dataclass(frozen=True)
class Budget:
max_steps: int = 12
max_input_tokens: int = 120_000
max_output_tokens: int = 18_000
compact_at: float = 0.80
@dataclass
class Usage:
steps: int = 0
input_tokens: int = 0
output_tokens: int = 0
def evaluate_budget(usage: Usage, budget: Budget) -> Action:
if (
usage.steps >= budget.max_steps
or usage.input_tokens >= budget.max_input_tokens
or usage.output_tokens >= budget.max_output_tokens
):
return Action.STOP
input_ratio = usage.input_tokens / budget.max_input_tokens
if input_ratio >= budget.compact_at:
return Action.COMPACT
return Action.CONTINUE
Run the check before every model request and update Usage from provider-reported token data.
At 80% of the input budget, compact state or narrow the next tool query. At 100%, stop with a structured reason.
Common mistake: Limiting each response while allowing unlimited steps, tools, and retries.
2. Filter Tool Results Before They Enter the Transcript
Return only the evidence required for the Agent’s next decision.
Do not append an entire:
Web page
Log file
Repository tree
Database response
Terminal session
API payload
when the next step needs only a few fields.
A search tool might return:
{
"source_id": "search_17",
"title": "Relevant page title",
"url": "https://example.com/page",
"relevant_passage": "A short evidence block"
}
Store the full artifact outside the prompt and retrieve a narrower section later.
Tool-filtering rule: Return the fields needed for the next decision—not every field that might become useful later.
Common mistake: Truncating the first 1,000 characters of a JSON payload. This can break the structure or remove the records the Agent actually needs.
Parse the payload first, select fields structurally, limit arrays, and then serialize valid JSON.
3. Compact Operational State, Not Just Conversation Text
Compaction should preserve the information required to continue the task while removing history that no longer affects the next action.
A useful compacted state contains:
User goal and success criteria
Decisions already made
Verified facts and source IDs
Files or records changed
Failed approaches
Open questions
The next action
Safety and output constraints
It should not retell the full conversation.
OpenAI documents compaction for long-running Responses API interactions. Anthropic provides context-management controls for clearing or summarizing older content. These implementations differ, so verify the current provider fields before integration.
Compaction rule: Preserve decisions and unresolved work. Remove narration and evidence that can be retrieved again.
Common mistake: Dropping source IDs, changed filenames, rejected approaches, or unresolved constraints.
After adding compaction, measure whether the Agent repeats searches or tool calls. A shorter prompt is not cheaper if the Agent must rebuild lost state.
4. Keep the Reusable Prefix Stable
Agent prompts often contain large reusable blocks:
System instructions
Tool schemas
Safety policies
Output formats
Shared reference material
Repository or product instructions
Place these stable elements before request-specific data:
1. System instructions
2. Policies and constraints
3. Tool definitions
4. Stable examples
5. Shared reference material
6. Request-specific data
Avoid placing timestamps, request IDs, session data, or frequently changing values near the beginning.
Caching is most useful when the prefix is long, stable, and reused. It may not save money for short sessions or frequently changing prompts.
Common mistake: Optimizing for cache-hit rate without measuring cache-write, read, or storage cost.
For a broader comparison of provider prompt caching, exact response caching, and semantic caching, see How to Reduce AI API Costs.
5. Prevent Retries From Replaying the Same Context
A retry is another Agent step, often with the same large prompt.
Do not repeat a failed request without changing the cause of failure.
Failure
Better response
Invalid structured output
Return the validation error and retry once
Tool timeout
Retry an idempotent operation once, then stop or use a fallback
Context overflow
Compact state or retrieve less evidence
Repeated tool call
Deduplicate using an operation hash
Rate limit
Back off or use a tested fallback route
Low-confidence result
Request missing information or escalate
Use idempotency keys for side-effecting operations such as payments, emails, deployments, and database writes.
Common mistake: Retrying a rate-limited model several times while resending the entire Agent context on every attempt.
Track retry tax by failure type so the team can fix the largest loop first.
6. Limit Reasoning and Subagents to Steps That Need Them
Not every Agent step requires deep reasoning.
Extraction, formatting, classification, validation, and routine tool selection can often use lower reasoning effort and compact structured output.
Reserve higher reasoning effort for tasks such as:
Complex planning
Difficult coding
Multi-document synthesis
Ambiguous decisions
Recovery from failed execution
Reasoning rule: Use the lowest reasoning effort that preserves the accepted-task rate.
Subagents also require a clear boundary. Give each worker:
A narrow task
A task-specific context slice
A tool allowlist
A token budget
A compact output schema
The root Agent usually needs findings, evidence IDs, confidence, and unresolved issues—not the worker’s full transcript.
Subagent rule: Parallelize independent work, not duplicated context.
Common mistake: Sending the complete root-Agent history to every worker before assigning a narrow task.
Which Optimization Should You Apply First?
Use Agent telemetry to choose the first intervention.
The thresholds below are investigation triggers, not universal standards.
Observed signal
Start here
Context amplification is high
Compact history and selectively retrieve state
Tool output dominates the prompt
Filter fields and store full artifacts externally
Retry tax is high
Fix validation, timeouts, and repeated tool calls
Reasoning share is high
Lower effort on routine steps
Subagents repeat the same evidence
Narrow worker scopes and context slices
Cached input remains low
Stabilize the reusable prefix
Costs remain high after loop cleanup
Compare lower-cost model routes
A safe implementation sequence is:
Measure cumulative input, tool retention, retries, and reasoning.
Add hard limits for steps, tools, retries, and total tokens.
Filter large tool results.
Compact older state at a measured threshold.
Stabilize the reusable prompt prefix.
Compare model routes only after the Agent loop is clean.
Change one major variable at a time and replay the same evaluation set.
Compare:
Task acceptance rate
Cost per successful task
Cumulative input
Tool-call count
Retry tax
Reasoning share
p50 and p95 latency
Human-review time
Roll back changes that save tokens by reducing task quality or removing necessary evidence.
Test Agent Workflows With CometAPI
Before running a multi-model evaluation, use the CometAPI pricing page and cost estimation guide to estimate input, output, cached-token, and reasoning costs.
Then use the model catalog to identify eligible routes and the Quickstart to configure an OpenAI-compatible client.
For production fallback, follow the CometAPI model fallback guide to switch routes without repeating completed tool calls or discarding validated state.
Unified access simplifies model comparison and fallback integration. Token budgets, compaction, validation, tool filtering, retry limits, and acceptance criteria still belong at the application level.
FAQ
Why do AI agents use more tokens than chatbots?
Agents make multiple model calls and may resend previous messages, tool results, instructions, and intermediate state at every step. This causes earlier context to be processed repeatedly.
Does prompt caching reduce context-window usage?
No. Prompt caching can reduce the effective price or latency of repeated input, but cached tokens still form part of the processed context. Use compaction, filtering, or selective retrieval to reduce prompt size.
When should an AI agent compact its context?
Compact before context growth begins to affect cost, latency, or available output space. Verify that the compacted state preserves decisions, evidence IDs, changed files, open questions, and safety constraints.
Do subagents reduce token costs?
Not automatically. They can reduce elapsed time or improve coverage for independent work, but duplicated context and overlapping analysis often increase total Token usage.
What is the best metric for AI agent cost optimization?
Use cost per successful task as the primary metric. Diagnose it with cumulative input, context amplification, tool retention, retry tax, reasoning share, latency, and human-review time.
