The short version
There is no other setting in the LLM stack that returns 90% of a line item for an afternoon's work. There is also no other setting that fails this silently — no error, no warning, just a bill that quietly does not go down.
This page covers what caching is, what every vendor charges for it, why it breaks, and the one change that took a real production system from a 7% hit rate to 84%.
Prompt caching stores the model's internal state after it has read a chunk of your prompt, so the next request that starts with the same chunk can skip re-reading it. You are not caching the answer. You are caching the model's understanding of the question so far.
OpenAI's own cookbook puts the mechanism plainly: caching "applies specifically to the key and value projections inside the attention layers", and a cache hit means "skipping prefill compute". That is why the discount is on input tokens only, and why the output is completely unaffected — caching never changes what the model says.
It is not a response cache. Two different questions that share a long system prompt both benefit. Two identical questions asked twice are a different mechanism entirely — that is gateway-level response caching, covered further down, and it is not the same product.
Because of how transformers work. A token's internal representation depends on every token before it and none of the tokens after it. So the model's state after reading your first 5,000 tokens is reusable — but only if those 5,000 tokens are byte-for-byte identical. Change token 3, and every representation from token 3 onward is different.
Fireworks states the consequence with no ambiguity: "even a single-token difference can invalidate the cache from that token onward."
OpenAI caches in increments of 128 tokens. Mistral works in 64-token blocks, and states that prompts under 64 tokens never get a cache hit at all. Anthropic works at the content-block level. This is why a very short prefix is not merely inefficient to cache — it is uncacheable.
This is the detail almost nobody knows. OpenAI routes requests "to a machine based on a hash of the initial prefix… typically the first 256 tokens". Fireworks is blunter: "prompt caching only works within 1 replica."
Which means a cache is not a global object. If your traffic scatters across machines, your hit rate collapses even when the prompts are identical. OpenAI's fix is prompt_cache_key; Fireworks uses a session-affinity header. Both do the same job: keep the same prefix landing on the same machine.
| Provider | Cache read | Write fee | Storage fee | Minimum prefix | TTL | Automatic? |
|---|---|---|---|---|---|---|
| Anthropic | 0.1× (90% off) | 1.25× (5 min) / 2× (1 hr) | None | 512–4,096 by model | 5 min or 1 hr | Opt-in or automatic |
| OpenAI (GPT-5.5) | "up to 90% off" — $0.50 vs $5.00 on GPT-5.5 | None | None | 1,024 tokens | 5–10 min idle, up to 1 hr; or 24 hr on request | Yes |
| Google Gemini (implicit) | Discounted, no guarantee | None | None | 4,096 (Gemini 3.x) | Short, not controllable | Yes |
| Google Gemini (explicit) | Discounted, guaranteed | Input rate | Per token-hour × TTL | 4,096 | You set it; defaults to 1 hr | No — you manage it |
| DeepSeek | 0.02× — $0.0028 vs $0.14 | None | None | Prefix units | Hours to days (disk-based) | Yes, on by default |
| Groq | 0.5× (50% off) | None | None | — | 2 hours without use | Yes |
| Mistral | 0.1× | None | None | 64 tokens | — | Yes |
| Fireworks | 0.5× default | None | None | — | Minutes to hours | Yes |
Two things stand out. DeepSeek's discount is in a different league — a cache read costs about 2% of a miss, not 10%, because it caches to disk rather than holding GPU memory. And Anthropic is the only major vendor that charges you to write the cache, which changes the maths for low-traffic apps.
Anthropic charges 1.25× the input rate to write a cache with a five-minute TTL, or 2× for one hour, and 0.1× to read it. Its own published break-even: caching pays off "after just one cache read for the 5-minute duration… or after two cache reads for the 1-hour duration".
The write fee sounds like a catch and mostly is not. If a prefix is read even twice you are ahead. Where it bites is a low-traffic app whose cache expires between users — then you pay 1.25× forever and read it back never. That is a real failure mode and it looks exactly like caching working.
Anthropic's minimum cacheable prefix varies by model: 512 tokens on Fable 5, 1,024 on Opus 4.8 and the Sonnets, 2,048 on Opus 4.7, and 4,096 on Haiku 4.5. A 2,000-token system prompt caches on Opus 4.8 and does absolutely nothing on Haiku — and Anthropic returns no error either way. Full detail is in our Claude pricing breakdown.
OpenAI enables caching automatically for any prompt of 1,024 tokens or more, with no write fee and no configuration. It reports the result at usage.prompt_tokens_details.cached_tokens, and its documentation claims caching can "reduce latency by up to 80% and input token costs by up to 90%".
Because OpenAI routes on a hash of your first 256 tokens, traffic with identical prefixes can still land on different machines and miss. The prompt_cache_key parameter pins related requests together — OpenAI describes it as a database shard key.
The published effect: "one of our coding customers saw an improved hit rate from 60% to 87% when they started using prompt_cache_key." That is a 27-point gain from one field.
One constraint to respect: keep each unique prefix-plus-key combination under 15 requests per minute, or you overflow the cache and it starts evicting.
OpenAI's FAQ is explicit: "Do cached prompts contribute to TPM rate limits? Yes, as caching does not affect rate limits." Anthropic and Groq take the opposite position — cached reads do not consume their token-per-minute budget. If you are rate-limited rather than cost-limited, that difference matters more than the discount.
OpenAI's own benchmarks show 40–80% better cache utilisation on the Responses API, because the raw chain-of-thought tokens persist between calls instead of being dropped. If you use reasoning models through Chat Completions, you are throwing away cache hits on every turn.
Google runs two systems. Implicit caching is automatic on Gemini 2.5 and newer with, in Google's own words, "no cost saving guarantee". Explicit caching requires you to create a CachedContent object and reference it, and Google says it exists for cases "where you want to guarantee cost savings, but with some added developer work".
The catch is on the explicit path: Google bills storage by TTL duration multiplied by cached token count. Neither Anthropic nor OpenAI charges rent on a cache. Google does, and the default TTL is one hour.
Google's own advice for lifting the implicit hit rate is worth quoting because it is the whole discipline in two lines: "Try putting large and common contents at the beginning of your prompt" and "try to send requests with similar prefix in a short amount of time."
Gemini 3.5 Flash and Gemini 3.1 Pro Preview both require 4,096 tokens to cache; Gemini 2.5 needs 2,048. That is four times OpenAI's threshold. A modest system prompt that caches happily on GPT will not cache on Gemini at all.
DeepSeek caches to disk rather than GPU memory, which changes the economics completely: a cache hit costs $0.0028 per 1M input tokens against $0.14 for a miss — roughly a 98% discount, not 90%. Cache lifetime is "a few hours to a few days" rather than minutes. It is on by default with no code change.
DeepSeek is honest about the limits: "the cache system works on a best-effort basis and does not guarantee a 100% cache hit rate."
Groq gives 50% off cached input, expires caches after two hours without use, and — usefully — states that cached tokens do not count toward rate limits. Mistral gives a full 90% off but works in 64-token blocks, so short prompts never hit. Fireworks defaults to 50% off and requires a session-affinity header to route you back to the replica holding your cache.
The most useful public caching case study we found comes from a security-tooling team running a large agent on Claude. It is worth reading in detail because the numbers are weekly and the cause is identified.
| Week | Cache hit rate | What changed |
|---|---|---|
| 2 February | 4.2% | Baseline |
| 9 February | 7.6% | — |
| 16 February | 73.7% | Moved runtime context out of the system prompt |
| 16 March | 84.3% | Breakpoint tuning |
| 23 March | 85.0% | — |
Overall result: cost down 59%, and 70% lower in the final ten days measured. Nearly 10 billion tokens served from cache.
They called it the relocation trick. Their system prompt contained working memory — runtime state, current context, per-user variables — which changed on every request. Because it sat near the top of the prompt, it invalidated everything after it.
They moved it out of the prefix and into a message at the tail of the conversation. Hit rate went from 7% to 74% in a single deployment.
The detail that should end every argument about whether caching is worth the effort: the team recorded one task consuming 67.5 million input tokens across 1,225 steps at a 91.8% cache rate. A near-identical task at a 3.2% cache rate cost roughly 60 times more.
Same work. Same tokens. Sixty times the bill, decided entirely by whether the prefix stayed still.
OpenAI publishes the list, and it generalises across every vendor.
Per-user template variables. If your system prompt renders the user's name or plan tier, every user gets a different prefix and none of them share a cache. Render stable placeholders instead, so the prefix is byte-identical across all users, all threads, all days.
Provider caches do not travel. A request served by Anthropic's direct API and a follow-up served through Bedrock will not share a cache, even with identical prompts. If your gateway load-balances across providers, it is silently destroying your hit rate — which is exactly why OpenRouter added sticky provider routing.
It is the single most common cache killer, and it is entirely self-inflicted. Fireworks: "Even a one-second difference in the timestamp will invalidate the entire cache."
A developer on Hacker News described the moment of realisation better than any documentation could: "It was a real facepalm moment when I realised we were busting the cache on every request by including date time near the top of the main prompt." Moving it took them from roughly 30–50% cached tokens to 50–70%.
The fixes, in order of preference: freeze the datetime once per run and use date only, no clock time; move it to the tail of the conversation; or, if it exists purely for debugging, put it in request metadata, which OpenAI explicitly recommends.
An AI gateway can cache too, but it caches responses, not prefixes — and that is a materially different product with different failure modes.
| Provider prefix cache | Gateway response cache | |
|---|---|---|
| What it stores | The model's internal KV state | The full response body |
| Fires when | The prompt starts the same | The request is identical |
| Different question, same system prompt | Hit | Miss |
| Does temperature matter? | No — sampling happens after prefill | Yes — the key includes the whole body |
| Saves | 90% of input cost | 100% of the call |
Cloudflare's AI Gateway is the clearest example: caching is free on all plans, but the key is a hash of the provider, endpoint, model, auth header and the entire request body. In their words, "any difference in the body — including messages, tools, or model parameters — will result in a separate cache entry". They also state plainly that semantic caching is not shipped: "we plan on adding semantic search for caching in the future."
The two mechanisms compose. Use provider prefix caching for the 90% discount on every call, and a gateway response cache for the small set of requests that genuinely repeat. Our gateway guide covers which gateways offer what.
A semantic cache matches on meaning rather than bytes, so "what's your refund policy" hits the entry stored for "how do I get a refund". The published upside is real and the published downside is under-reported.
The most-cited case study analysed 100,000 production queries and found the traffic split into 18% exact duplicates, 47% semantically similar, and 35% genuinely novel. An exact-match cache can only ever catch that first 18%. Adding semantic matching took the team from an 18% hit rate to 67%, cut the monthly bill from $47,000 to $12,700, and dropped average latency from 850ms to 300ms — at a 0.8% false-positive rate.
That is one company, and it is the number every semantic-cache vendor now quotes. Independent write-ups put realistic production hit rates at 20–45% for mixed traffic, not the 90%+ sometimes marketed. Plan for the low end.
And a false positive in a semantic cache is not a slow response. It is a wrong response, served confidently. Portkey ships a default similarity threshold of 0.95 and restricts semantic caching to requests under 8,191 tokens and four messages. Tune the threshold per use case — loose for an FAQ, tight for anything transactional, because the cost of a wrong answer is not symmetric.
prompt_cache_key on OpenAI, or the equivalent session key elsewhere. One customer went from a 60% to an 87% hit rate on this alone.allowed_tools instead.cached_tokens on OpenAI and Mistral, prompt_cache_hit_tokens on DeepSeek, cache_read_input_tokens on Anthropic. If you are not looking at this number weekly, you do not know whether caching is on.cached_tokens.Prompt caching stores the model's internal key-value tensors after it reads a chunk of your prompt, so a later request starting with the same chunk skips the prefill compute. You pay a fraction of the input price — 0.1× on Anthropic, "up to 90% off" on OpenAI. It never changes the model's output.
On input tokens, up to 90% with most major vendors and about 98% on DeepSeek. In one documented production case, raising the hit rate from 7% to 84% cut total LLM cost by 59%.
No — only on the prefix, and only on an exact match. A single differing token invalidates the cache from that token onward, because a transformer's representation of a token depends on everything before it.
No. Sampling parameters act after the prefill stage, so temperature, top_p and top_k do not affect a provider's prefix cache. They do affect a gateway's response cache, which keys on the whole request body.
Most likely a dynamic value near the top of your prompt — a timestamp, a user name, a session ID. Or your prefix is below the minimum (1,024 tokens on OpenAI, 4,096 on Gemini and Haiku 4.5). Neither condition returns an error.
It depends on the vendor. OpenAI says yes: "caching does not affect rate limits." Anthropic and Groq say no — cached reads do not consume your token-per-minute budget, which effectively multiplies your throughput.
OpenAI evicts after 5–10 minutes of inactivity, up to an hour off-peak, with a 24-hour retention option. Anthropic offers 5 minutes or 1 hour, priced differently. Groq expires after 2 hours unused. DeepSeek's disk cache lasts hours to days.
Yes — 1.25× the input rate for a five-minute TTL, or 2× for an hour. It pays for itself after one read at the short TTL. OpenAI, Google and DeepSeek charge nothing to write.
Implicit caching is automatic on Gemini 2.5 and newer, but Google explicitly offers "no cost saving guarantee". Explicit caching means creating a CachedContent object yourself; Google guarantees the saving but bills storage by TTL duration multiplied by token count.
It is an OpenAI parameter that pins related requests to the same machine, because OpenAI routes on a hash of your first 256 tokens. OpenAI reports one customer moving from a 60% to an 87% hit rate by adding it. Keep each prefix-plus-key combination under 15 requests per minute.
Sometimes. One production case moved from an 18% hit rate to 67% and cut its bill 73%. But independent analyses put realistic hit rates at 20–45% for mixed traffic, and a false positive serves a wrong answer rather than a slow one. Tune the threshold per use case.
No. Caching skips recomputation of the prefill stage; generation still happens normally and is still affected by temperature. Fireworks states it directly: "prompt caching doesn't alter the result generated by the model."
No. A cache written on Anthropic's direct API is not readable through Bedrock, and a gateway that load-balances across providers will destroy your hit rate. Pin the provider if caching matters.
On raw discount, DeepSeek — a cache read costs about 2% of a miss, versus 10% elsewhere, because it caches to disk. On flexibility, Anthropic, because you control the TTL and the breakpoints. On simplicity, OpenAI, because it is automatic and free to write.
Prompt caching is the highest-return change available in an LLM stack, and almost nobody has it properly configured. The discount is not the hard part — every vendor offers it, most enable it automatically. The hard part is keeping your prefix byte-identical, because a single timestamp destroys the whole thing and nothing tells you it happened.
So the work is not "turn on caching". The work is: put the static content first, get every changing value out of the prefix, pin the provider, and then look at cached_tokens every week. That is the entire discipline, and one team that did exactly this went from a 7% hit rate to 84% and cut their bill by more than half.
If you run agents, this is not optional. Their bill is the cached prefix, re-read on every turn, which is why our breakdown of agent costs ends in the same place this one does.
prompt_cache_key result, Responses API cache utilisation, the cache-miss checklist.Copyright © 2026 Best AI Gateways
Independent ranking. We may be rewarded for recommending the service we rate best and sending users to it — that reward pays for the research behind this comparison and never buys a ranking position, at no extra cost to you. Information is provided “as is” without warranty. See our Disclaimer & Terms.