How to Reduce AI Agent Token Costs in Production

TL;DR

AI agent token costs grow when each step repeatedly processes instructions, conversation history, tool results, and intermediate state.

Reduce token volume with run-level budgets, tool-result filtering, context compaction, retry limits, and controlled reasoning. Use prompt caching for stable repeated input, but optimize the agent loop before switching to a cheaper model.

The most useful production metric is cost per successful task, measured across the complete run—not price per request or the context size of the final call.

This guide focuses specifically on multi-step AI agents. It explains how repeated context compounds across a run, how to identify the largest source of waste, and which controls to implement first.

Introduction

A chatbot may make one model request per user message. An AI agent can make 10, 20, or more calls before completing one task.

Each step may resend instructions, conversation history, tool results, and intermediate state. Retries, reasoning, and subagents add more usage, so a short final answer can still consume a large number of tokens.

As usage scales, these costs become harder to predict and can quickly reduce product margins. Lowering them requires optimizing the full agent loop—not simply switching to a cheaper model.

This article concentrates on Agent-specific costs. For a broader guide covering prompt caching, exact response caching, semantic caching, model routing, and general API cost management, see How to Reduce AI API Costs.

Why Do AI Agent Token Costs Compound?

In a multi-step agent, the cost of one task is the sum of every model call—not only the final response.

The main sources of Agent Token usage are:

There are two distinct ways to reduce the bill:

1. Process fewer tokens through filtering, compaction, output limits, and loop controls.
2. Reduce the effective price of necessary tokens through prompt caching or model selection.

Key distinction: Prompt caching lowers the cost of repeated input. Context compaction reduces the repeated input itself.

How Can a 12-Step Agent Process 147,000 Tokens?

Consider a hypothetical support agent with:

• A 4,000-token stable prefix
• 1,500 new tokens added after each step
• The complete accumulated history resent on every request
• 12 total model calls

The input at step n is:

Input at step n = 4,000 + 1,500 × (n - 1)

The cumulative input across 12 calls is:

Total input
= 4,000 × 12 + 1,500 × (0 + 1 + ... + 11)
= 48,000 + 99,000
= 147,000 input tokens

The final call contains only 20,500 input tokens, but the full run processes 147,000 cumulative input tokens.

Now apply two controls:

1. Cache the stable 4,000-token prefix after the first call.
2. Compact the history after step six into a 2,500-token state summary.

This is a planning calculation, not a provider benchmark.

It assumes that every request includes the complete accumulated history. Agents that selectively construct state, summarize old messages, or retrieve only relevant information may follow a different cost curve.

Cost-growth rule: Measure cumulative input across the full run. The final context size does not represent the total number of tokens processed.

Which Metrics Reveal Agent Token Waste?

Do not begin by changing models. First identify where the workflow is spending tokens without improving the result.

Record these fields for every Agent step:

The primary metric should be:

cost per successful task
= total workflow cost
/ accepted tasks

A cheaper run is not an improvement when it causes more failed tasks, repeated tools, or human correction.

Four Agent-specific metrics help locate the problem.

Context Amplification

context amplification
= cumulative input tokens
/ final-step input tokens

A high value indicates that earlier context has been processed repeatedly.

Tool Retention Ratio

tool retention ratio
= tool-result tokens retained in context
/ tokens originally returned by tools

A high ratio can indicate that the Agent is carrying too much raw evidence between steps.

Retry Tax

retry tax
= retry and repair cost
/ total workflow cost

Reasoning Share

reasoning share
= reasoning-token cost
/ total model cost

Measure each workload separately. Research, coding, browser, and customer-support agents should not share one global baseline.

Six Ways to Reduce AI Agent Token Costs

1. Set a Budget for the Complete Run

A per-request output limit does not control a multi-step Agent.

Set run-level limits for:

• Total model steps
• Cumulative input and output
• Tool calls and tool-result size
• Retries by failure type
• Subagents
• Total elapsed time or estimated cost

The following provider-neutral Python example evaluates the run before each model call:

from dataclasses import dataclass
from enum import Enum


class Action(str, Enum):
    CONTINUE = "continue"
    COMPACT = "compact"
    STOP = "stop"


@dataclass(frozen=True)
class Budget:
    max_steps: int = 12
    max_input_tokens: int = 120_000
    max_output_tokens: int = 18_000
    compact_at: float = 0.80


@dataclass
class Usage:
    steps: int = 0
    input_tokens: int = 0
    output_tokens: int = 0


def evaluate_budget(usage: Usage, budget: Budget) -> Action:
    if (
        usage.steps >= budget.max_steps
        or usage.input_tokens >= budget.max_input_tokens
        or usage.output_tokens >= budget.max_output_tokens
    ):
        return Action.STOP

    input_ratio = usage.input_tokens / budget.max_input_tokens

    if input_ratio >= budget.compact_at:
        return Action.COMPACT

    return Action.CONTINUE

Run the check before every model request and update Usage from provider-reported token data.

At 80% of the input budget, compact state or narrow the next tool query. At 100%, stop with a structured reason.

Common mistake: Limiting each response while allowing unlimited steps, tools, and retries.

2. Filter Tool Results Before They Enter the Transcript

Return only the evidence required for the Agent’s next decision.

Do not append an entire:

• Web page
• Log file
• Repository tree
• Database response
• Terminal session
• API payload

when the next step needs only a few fields.

A search tool might return:

{
  "source_id": "search_17",
  "title": "Relevant page title",
  "url": "https://example.com/page",
  "relevant_passage": "A short evidence block"
}

Store the full artifact outside the prompt and retrieve a narrower section later.

Tool-filtering rule: Return the fields needed for the next decision—not every field that might become useful later.

Common mistake: Truncating the first 1,000 characters of a JSON payload. This can break the structure or remove the records the Agent actually needs.

Parse the payload first, select fields structurally, limit arrays, and then serialize valid JSON.

3. Compact Operational State, Not Just Conversation Text

Compaction should preserve the information required to continue the task while removing history that no longer affects the next action.

A useful compacted state contains:

• User goal and success criteria
• Decisions already made
• Verified facts and source IDs
• Files or records changed
• Failed approaches
• Open questions
• The next action
• Safety and output constraints

It should not retell the full conversation.

OpenAI documents compaction for long-running Responses API interactions. Anthropic provides context-management controls for clearing or summarizing older content. These implementations differ, so verify the current provider fields before integration.

Compaction rule: Preserve decisions and unresolved work. Remove narration and evidence that can be retrieved again.

Common mistake: Dropping source IDs, changed filenames, rejected approaches, or unresolved constraints.

After adding compaction, measure whether the Agent repeats searches or tool calls. A shorter prompt is not cheaper if the Agent must rebuild lost state.

4. Keep the Reusable Prefix Stable

Agent prompts often contain large reusable blocks:

• System instructions
• Tool schemas
• Safety policies
• Output formats
• Shared reference material
• Repository or product instructions

Place these stable elements before request-specific data:

1. System instructions
2. Policies and constraints
3. Tool definitions
4. Stable examples
5. Shared reference material
6. Request-specific data

Avoid placing timestamps, request IDs, session data, or frequently changing values near the beginning.

Caching is most useful when the prefix is long, stable, and reused. It may not save money for short sessions or frequently changing prompts.

Common mistake: Optimizing for cache-hit rate without measuring cache-write, read, or storage cost.

For a broader comparison of provider prompt caching, exact response caching, and semantic caching, see How to Reduce AI API Costs.

5. Prevent Retries From Replaying the Same Context

A retry is another Agent step, often with the same large prompt.

Do not repeat a failed request without changing the cause of failure.

Use idempotency keys for side-effecting operations such as payments, emails, deployments, and database writes.

Common mistake: Retrying a rate-limited model several times while resending the entire Agent context on every attempt.

Track retry tax by failure type so the team can fix the largest loop first.

6. Limit Reasoning and Subagents to Steps That Need Them

Not every Agent step requires deep reasoning.

Extraction, formatting, classification, validation, and routine tool selection can often use lower reasoning effort and compact structured output.

Reserve higher reasoning effort for tasks such as:

• Complex planning
• Difficult coding
• Multi-document synthesis
• Ambiguous decisions
• Recovery from failed execution

Reasoning rule: Use the lowest reasoning effort that preserves the accepted-task rate.

Subagents also require a clear boundary. Give each worker:

• A narrow task
• A task-specific context slice
• A tool allowlist
• A token budget
• A compact output schema

The root Agent usually needs findings, evidence IDs, confidence, and unresolved issues—not the worker’s full transcript.

Subagent rule: Parallelize independent work, not duplicated context.

Common mistake: Sending the complete root-Agent history to every worker before assigning a narrow task.

Which Optimization Should You Apply First?

Use Agent telemetry to choose the first intervention.

The thresholds below are investigation triggers, not universal standards.

A safe implementation sequence is:

1. Measure cumulative input, tool retention, retries, and reasoning.
2. Add hard limits for steps, tools, retries, and total tokens.
3. Filter large tool results.
4. Compact older state at a measured threshold.
5. Stabilize the reusable prompt prefix.
6. Compare model routes only after the Agent loop is clean.

Change one major variable at a time and replay the same evaluation set.

Compare:

• Task acceptance rate
• Cost per successful task
• Cumulative input
• Tool-call count
• Retry tax
• Reasoning share
• p50 and p95 latency
• Human-review time

Roll back changes that save tokens by reducing task quality or removing necessary evidence.

Test Agent Workflows With CometAPI

Before running a multi-model evaluation, use the CometAPI pricing page and cost estimation guide to estimate input, output, cached-token, and reasoning costs.

Then use the model catalog to identify eligible routes and the Quickstart to configure an OpenAI-compatible client.

For production fallback, follow the CometAPI model fallback guide to switch routes without repeating completed tool calls or discarding validated state.

Unified access simplifies model comparison and fallback integration. Token budgets, compaction, validation, tool filtering, retry limits, and acceptance criteria still belong at the application level.

FAQ

Why do AI agents use more tokens than chatbots?

Agents make multiple model calls and may resend previous messages, tool results, instructions, and intermediate state at every step. This causes earlier context to be processed repeatedly.

Does prompt caching reduce context-window usage?

No. Prompt caching can reduce the effective price or latency of repeated input, but cached tokens still form part of the processed context. Use compaction, filtering, or selective retrieval to reduce prompt size.

When should an AI agent compact its context?

Compact before context growth begins to affect cost, latency, or available output space. Verify that the compacted state preserves decisions, evidence IDs, changed files, open questions, and safety constraints.

Do subagents reduce token costs?

Not automatically. They can reduce elapsed time or improve coverage for independent work, but duplicated context and overlapping analysis often increase total Token usage.

What is the best metric for AI agent cost optimization?

Use cost per successful task as the primary metric. Diagnose it with cumulative input, context amplification, tool retention, retry tax, reasoning share, latency, and human-review time.