Why prompt caching changes the math
A blended “$X per million input tokens” estimate is one number. What a sustained agentic workload actually pays is usually mostly cache reads, billed at a fraction of that rate — here is why, and how to account for it.
What caching is, and why it exists
Modern LLM APIs let a provider avoid reprocessing input tokens it has already seen recently — a system prompt, a long document, or the earlier turns of a multi-step agentic conversation. Instead of billing those repeated tokens at the full input rate every single time, the provider bills them at a steep discount, commonly a fifth to a tenth of the fresh-input rate, because serving them from a cache costs the provider far less compute than processing them fresh. Different providers call this different things (prompt caching, context caching, cache reads and writes) and structure the discount slightly differently, but the shape is the same: some fraction of input is fresh, some fraction is a cache hit, and the two are billed at different rates.
Why this fraction is usually large for agents, not small
A one-shot question-and-answer call has little to cache -- most of its input is new every time. An agentic loop is the opposite case: a coding assistant, a tool-using agent, or a long chat session typically resends the entire conversation and tool context on every turn, appending only a small amount of new content each time. Nearly all of that resent context is identical to what was just sent a moment ago, so it is eligible for the cache discount on every subsequent turn. The cached share tends to grow as a session runs longer, not shrink -- the accumulated history is what gets resent, and it only gets longer.
This is not a small effect. Publicly reported figures from real, sustained agentic-coding sessions describe cache reads making up well over 90% of total input tokens across a session. At that share, the cached rate -- not the fresh-input rate most pricing pages headline -- is the number that actually determines most of the bill.
What this means for estimating cost
Estimating cost from total input tokens times one blended rate silently picks a rate to apply to the whole total, and that rate is usually either the fresh rate (overstating the bill, often badly, for an agentic workload) or a rate assumed from someone else's cache share (which may not match yours). The fix is not a better blended number -- it is not blending at all: track fresh and cached input tokens as separate figures, and price each at its own rate. That is the whole reason the LLM API Pricing Calculator asks for them separately rather than asking for one input-token total.
This page describes a general, widely documented mechanism common to current LLM APIs. It is not a claim about any specific provider's current cache discount, cache duration, or pricing -- check the provider's own pricing page for those.
Common questions
What is prompt/context caching in an LLM API?
A way for a provider to avoid reprocessing input tokens it has already seen recently — a system prompt, a long document, or the earlier turns of a multi-step agentic conversation. Instead of billing those repeated tokens at the full input rate every time, the provider bills them at a steep discount, because serving them from a cache costs the provider far less compute than processing them fresh.
Why do cached tokens dominate an agentic workload's bill?
An agentic loop — a coding assistant, a tool-using agent, a long chat session — typically resends the entire conversation and tool context on every turn, appending only a small amount of new content each time. Nearly all of that resent context is identical to what was just sent a moment ago, so it is eligible for the cache discount. Publicly reported real-world figures put cache reads at well over 90% of total input tokens for sustained agentic sessions, which makes the cached rate, not the fresh rate, the number that actually determines the bill.
Why does a single blended per-token rate get this wrong?
Because it assumes every input token costs the same, when in an agentic workload most of them are billed at a fraction of the fresh-input rate. Estimating cost from total tokens times one blended rate — often the fresh-input rate, since that is the headline number on a pricing page — overstates the bill for exactly the workloads it is most often applied to.