AI API gateways in 2026: how they work, what they really cost, and when to skip them

Most teams arrive at AI gateways the same way. You ship something on GPT. Six months later half your traffic wants Claude, your finance lead wants the bill cut, and a Gemini model just turned out to be four times cheaper for the one job that eats 60% of your tokens. Now you have three vendor accounts, three SDKs, three invoices and no single number for what AI costs you.

That is the problem a gateway is built for. Whether it is the right fix for you depends on numbers most gateway landing pages leave out — so this piece leads with those.

What is an AI API gateway?

An AI API gateway is a service that exposes one API endpoint for many AI models. You send a request in OpenAI's format to the gateway's base URL; the gateway forwards it to whichever provider hosts the model you asked for, and returns the answer in the same format. You keep one API key and one balance instead of one per vendor.

The category goes by several names — LLM gateway, AI gateway, LLM router, unified LLM API, OpenAI-compatible proxy. They all describe the same layer.

What "OpenAI-compatible" actually means

It means the service implements OpenAI's HTTP contract: a POST to /v1/chat/completions, a JSON body with model and messages, a bearer token in the Authorization header, and server-sent events when you pass stream: true. OpenRouter, for example, documents that it implements the OpenAI spec for both /completions and /chat/completions.

The practical consequence: you keep the OpenAI SDK you already have. You change base_url, you change the key, and you change the model string. That is usually the entire migration.

Gateway vs. SDK vs. cloud model platform

An SDK like LangChain abstracts models inside your process — it does not sit on the network path, cannot fail over for you at the infrastructure level, and cannot enforce a spend cap across your whole company.

A cloud model platform (AWS Bedrock, Azure AI Foundry, Google Vertex) is a first-party catalogue: strong compliance story, billing through a cloud account you already have, but a narrower model list and, in several cases, a price premium. Anthropic's own pricing documentation notes that regional endpoints on Bedrock and Vertex carry a 10% premium over global endpoints.

A gateway sits on the wire between your app and every provider, and that position is what lets it route, retry, cache, meter and cap.

How an AI API gateway works

The request path, step by step

1. Your app posts an OpenAI-shaped request to the gateway.
2. The gateway authenticates your key and checks your budget and rate limits.
3. It looks for a cache hit. If it finds one, it answers without touching a provider.
4. It picks a provider for the requested model — by price, by latency, or by an explicit pin you set.
5. It forwards the call, and on a 429 or 5xx it retries against a fallback.
6. It streams the response back and records tokens, cost and latency against your key.

What changes in your code

Almost nothing. Two lines and a model string, in most stacks. The interesting change is what you stop building: retry logic, provider-specific error handling, a cost dashboard, per-team keys, and the small internal library everyone writes to paper over three SDKs.

Key facts and numbers

What a gateway actually solves

Five things, in roughly the order teams discover they need them.

One key, one balance, many models

The obvious one, and the one people underrate until they try to add a fourth provider. Every new vendor is an account, a payment method, a set of keys to rotate, a rate-limit regime and an invoice. A gateway collapses that into one of each.

Failover when a provider has a bad day

Providers rate-limit and providers go down. If your app's only path to a model is a direct call to one vendor, a 429 is an outage for your users. A gateway can retry the same request against a second provider — or a second model — without your code knowing. This is the single feature that most often turns a gateway from "nice" into "load-bearing".

Cost control that finance can actually see

Per-key spend caps, per-team budgets, per-feature cost attribution. Doing this yourself means instrumenting every call site and reconciling three invoices by hand.

Observability you don't have to build

Token counts, latency percentiles, error rates and cost, sliced by key and model, without adding a tracing layer to your own code.

Model routing by task

Not every request needs your best model. The published price spreads are large enough that this matters: Claude Haiku 4.5 at $1 / $5 per 1M tokens against Claude Opus 4.8 at $5 / $25 is a 5× difference. On OpenAI's list, gpt-5.4-nano at $0.20 / $1.25 against gpt-5.6-sol at $5 / $30 is roughly 25× on input and 24× on output. Route classification and extraction to the small model, keep the big one for reasoning, and the arithmetic does the rest.

A caveat worth stating plainly: we have not found any credible, independently published figure for "average savings from model routing". Vendors quote them freely; none of them show their working. The price ratios above are real because they are arithmetic on published list prices. Treat any percentage beyond that as marketing.

The four kinds of AI gateway

They are not interchangeable, and picking the wrong category is a more expensive mistake than picking the wrong vendor within a category.

One trap hides in that last row. Portkey meters logs, not requests — so a team doing 500,000 calls a month on a 10,000-log plan keeps routing fine and quietly loses visibility into 98% of its traffic. Cloudflare's free tier caps you at 100,000 logs in total, across all gateways, forever. Read the unit before you read the price.

How gateways make money (and where your bill goes strange)

There are two fees, they are unrelated, and almost every "cheapest gateway" comparison confuses them.

Fee one: the token markup

A percentage added to what the model provider charges. Requesty is refreshingly blunt about it: a model that costs $10 per 1M tokens from OpenAI costs $10.50 through Requesty — 5%. OpenRouter, by contrast, states it applies no markup on inference and passes provider pricing straight through.

Fee two: the fee on money you load

This is the one that surprises people. OpenRouter charges 5.5% on card credit purchases, with an $0.80 minimum. Do the arithmetic: top up $10 and the minimum bites, so your effective fee is 8%. Top up $5 and it is 16%. The headline "0% markup" is true and the fee is still real. Cloudflare's Unified Billing charges 5% on credits the same way — $100 of credit costs you $105.

Neither is a scandal. Both are simply a different place to put the same margin, and the only way to compare gateways honestly is to compute your landed cost per million tokens, top-up fee included, at the top-up size you actually use.

Fee three, sometimes: bring your own key

Some gateways let you plug in your own OpenAI or Anthropic key and just use them as a router. OpenRouter allows this free for the first 1M BYOK requests per month, then charges 5% of what the same call would have cost on their platform. It is a good deal for large teams and a slightly odd one for small ones.

The awkward question: how can a reseller be cheaper than the list price?

Some gateways sell frontier models below what the model's own vendor charges. Our own Fable 5 comparison shows a 40% spread between the cheapest gateway and Anthropic's list price. CometAPI markets a "permanent 20% discount" across its catalogue. Meanwhile at least one reseller sits above list.

The honest answer is that below-list pricing has three plausible explanations, and you cannot tell from the outside which one applies:

• Committed-capacity arbitrage. Large buyers negotiate volume rates and resell part of that capacity. This is normal, legal and boring.
• Deliberate subsidy. A gateway buying market share sells below cost, the way every marketplace has since the invention of the marketplace. Real today, not guaranteed to be real next year.
• The model is not quite the model. A quantized weight, a "turbo" endpoint, a different serving stack. This is the one that should worry you, and it is discussed openly in developer communities — there is an open pull request in Alibaba's qwen-code repository literally titled "Avoid quantized models on OpenRouter".

So do not take a cheap price on faith, and do not dismiss it either. Test it. Send the same 50 prompts to the gateway and to the vendor's own API, diff the outputs, and compare latency. If a gateway lets you pin a specific upstream provider, pin it. Half an hour of work settles a question no marketing page will.

What models actually cost in 2026

Every number below is from the vendor's own public pricing page, checked in July 2026, per 1M tokens, standard tier.

Why the per-token price is not the bill

Anthropic's documentation states that its newer models — Opus 4.7 and later, Fable 5, Mythos 5, Sonnet 5 — use an updated tokenizer that produces roughly 30% more tokens for the same text. A price cut of 20% paired with a 30% token increase is a price rise. This is the kind of thing a gateway's cost dashboard shows you and a spreadsheet of list prices does not.

Two more line items hide in the same place: Anthropic charges a 1.1× multiplier for pinned US data residency, and OpenAI applies a 10% uplift on regional processing endpoints for models released from March 2026 onward. Compliance has a price, and it is not zero.

Real ways to cut the bill, with real numbers

Prompt caching — up to 90% off input

All three major vendors now price a cache read at one tenth of the base input rate. Anthropic charges 0.1× on cache reads, with a write costing 1.25× (5-minute TTL) or 2× (1-hour TTL). OpenAI prices cached input at 10% of base across the GPT-5.4, 5.5 and 5.6 families. Google does the same for Gemini, plus $1.00 per 1M tokens per hour of cache storage.

The break-even is quick: with Anthropic's 5-minute cache, a single subsequent read already pays for the write. If your system prompt is long and stable — and if you have a RAG app, it is — this is the first thing to turn on, before you even think about switching providers.

Batch API — 50% off, everywhere

Anthropic, OpenAI and Google all discount asynchronous batch processing by 50%. If a workload can tolerate a delay — evaluations, backfills, classification of yesterday's tickets, document enrichment — it should not be running synchronously. Anthropic further confirms that the batch discount stacks with prompt caching.

Semantic caching — and the honest hit rate

An exact-match cache only fires when two requests are byte-identical, and in the production dataset above that was true of just 18% of redundant calls. A semantic cache matches on meaning instead. In that same case study, the team took their hit rate from 18% to 67%, cut the monthly LLM bill from $47,000 to $12,700 (−73%) and average latency from 850ms to 300ms, at a 0.8% false-positive rate.

That is one company, and it is the number every semantic-cache vendor now quotes. Independent write-ups put realistic production hit rates at 20–45% for mixed traffic, not the 90%+ that sometimes gets marketed. Plan for the low end, and tune your similarity threshold per use case — the same study used 0.94 for FAQ, 0.88 for product search and 0.97 for transactional queries, because a false positive on a transaction is a bug and a false positive on an FAQ is a shrug.

How much latency does a gateway add?

The honest summary: a well-tuned gateway adds single-digit milliseconds. Against an LLM call that takes 800ms to several seconds, that is noise. A badly provisioned one can add tens of milliseconds or fall over under load — Bifrost's own benchmark shows LiteLLM dropping to an 88.8% success rate at 500 RPS on a 2-vCPU box, while LiteLLM's own documentation reports 8ms P95 at 1,000 RPS across four properly sized instances. Both can be true. Neither is neutral.

If latency is genuinely your constraint, benchmark the gateway on your own traffic. Nobody else's numbers are about your workload.

The risks nobody puts on the pricing page

Model substitution and quantization

This is the big one. When you ask an aggregator for a model, you are trusting it to route you to the real thing at full precision. Developers have reported cases where a model served through an aggregator performed noticeably worse than the same model called directly — suspicion falling on quantized weights or a "turbo" serving endpoint chosen for throughput. There is an open pull request in Alibaba's qwen-code project recommending developers set provider quantization filters to fp8 or better when routing through OpenRouter.

Mitigation: pin the upstream provider explicitly if the gateway supports it, run your own evals after any routing change, and treat "same model, cheaper" as a hypothesis rather than a fact until your evals agree.

Prompt logging and data retention

Read the policy, not the marketing. OpenRouter states it does not log prompts or completions by default — and separately offers a 1% usage discount if you opt in to letting it log them. That is an unusually explicit data-for-money trade, and an honest one. Other gateways are far less clear. If you send anything regulated, you want a written zero-data-retention commitment, not a paragraph on a landing page.

Supply-chain risk in self-hosted gateways

Self-hosting removes the vendor and adds a different problem. On 24 March 2026, two LiteLLM releases (1.82.7 and 1.82.8) were published to PyPI containing a backdoor; they were live for roughly 40 minutes. The compromise came in through an unpinned CI dependency that exfiltrated the project's publish token, and the payload harvested credentials and attempted lateral movement. LiteLLM has on the order of 95 million monthly downloads, and 1.82.6 was the last clean release.

This is not an argument against LiteLLM — the project disclosed it clearly and quickly, which is more than most do. It is an argument for pinning versions, verifying hashes and treating your gateway as production infrastructure, because that is exactly what it is.

The abstraction itself can be deprecated

As of July 2026, Cloudflare's own OpenAI-compatible "Unified API" is marked Deprecated in its documentation navigation. The compatibility shim that a gateway sells you is itself software someone else maintains, and it can be withdrawn.

Pooled rate limits and lock-in

On aggregators you may share upstream quota with other customers, and heavy neighbours can cost you throughput at exactly the wrong moment. And while the OpenAI-compatible surface makes leaving easy, the gateway's own config — routing rules, fallback chains, cache policies, virtual keys — usually does not travel. Keep it in version control, not in a web console.

Should you use a gateway at all?

Managed or self-hosted?

The line is drawn by arithmetic, not ideology. A managed gateway typically costs you somewhere between 0% and 5.5% of spend, depending on where the vendor puts its fee. Running LiteLLM or Bifrost yourself costs nothing in licence and something real in infrastructure and attention — Postgres, optional Redis, compute, monitoring, upgrades and a person who gets paged.

Which means: self-host when a few percent of your monthly token spend exceeds the fully loaded cost of running the proxy. Below that line you are paying an engineer to save less than the fee. Above it, the maths flips, and it flips hard at scale.

A middle path that suits a lot of teams: use a managed gateway with your own provider keys (BYOK), so you get the routing, caching and dashboards while your tokens are still billed at your negotiated vendor rate.

How to pick one: a 7-point checklist

1. Compute the landed price. Token rate plus top-up fee, at the top-up size you actually use. A 5.5% fee with an $0.80 floor is 16% on a $5 top-up.
2. Check what is metered. Requests, tokens, or logs? Log-metered plans go quiet, not down — which is worse.
3. Demand model provenance. Can you pin the upstream provider? Does the gateway state its quantization policy? If it will not say, assume the cheapest serving path.
4. Read the retention policy. Are prompts logged by default? Is there a written ZDR option? Is there a discount for letting them keep your data — and do you want it?
5. Test failover for real. Point it at a dead model and see whether it retries or throws.
6. Look for an exit. Pay-as-you-go over subscriptions. No expiring credits — note that OpenRouter reserves the right to expire unused credits after a year.
7. Run your evals. Same prompts, gateway versus direct API, before you move production traffic. Nothing else on this list matters if the outputs are worse.

How we rank gateways on this site

We score each gateway on roughly forty criteria. Three we will name: real landed price per 1M tokens (fees included, not headline rate), model coverage and how fast new models appear, and reliability — uptime, a public status page, and whether credits lock you in. The rest we keep undisclosed, deliberately: a fully published rubric is a rubric that gets gamed, and the moment it is gamed it stops being useful to you.

The ranking itself, with the landed price per 1M tokens for every gateway we track, lives on our comparison page and in the model price table. Both are updated as prices move.

Independent ranking. We may be rewarded for recommending the service we rate best and sending users to it. That reward pays for the research behind this comparison and never buys a ranking position, at no extra cost to you. Prices are set by each provider and change — always verify in the provider's own dashboard before you build.

How to migrate in ten minutes

If you are already on the OpenAI SDK, the whole change is a base URL and a key.

from openai import OpenAI

client = OpenAI(
    base_url="https://your-gateway.example/v1",   # was: api.openai.com
    api_key=os.environ["GATEWAY_API_KEY"],        # one key, one balance
)

resp = client.chat.completions.create(
    model="claude-fable-5",                       # or gpt-5.5, or gemini-3.1-pro
    messages=[{"role": "user", "content": "Summarise this ticket."}],
    stream=True,
)

Then, before you route real traffic: run your evaluation set through both the gateway and the vendor's direct API, compare outputs and p95 latency, and only cut over if they match. Keep a direct-API key as your fallback. Pin exact model slugs rather than aliases, because aliases move.

Common mistakes

• Comparing headline rates and ignoring top-up fees. The cheapest token rate is regularly not the cheapest bill.
• Assuming "gateway" means "cheaper". At least one reseller in our own Fable 5 comparison charges 10% above Anthropic's list price. Look before you switch.
• Skipping caching because the gateway feels like the optimisation. Caching is 90% off input. No gateway margin comes close.
• Trusting an alias. Pin exact model versions, or you will wake up to a silent swap.
• Not testing failover. An untested fallback is a fallback that does not exist.
• Leaving routing config in a web console. Put it in version control with everything else that can take your app down.

FAQ

What is an AI API gateway in one sentence?

It is one OpenAI-compatible endpoint that routes your requests to many model providers, so you keep a single API key, a single balance and a single invoice instead of one of each per vendor.

Do AI gateways mark up your tokens?

Some do and some do not, and this is the most misunderstood question in the category. Requesty states a 5% markup on token cost. OpenRouter states no markup on inference — but charges 5.5% on card credit purchases with an $0.80 minimum. Compare landed cost, not headline rate.

What is the cheapest AI gateway?

There is no single answer, because the cheapest option depends on how you top up and which models you use. A 0%-markup gateway with a 5.5% credit fee beats a 5%-markup gateway if you top up in large amounts, and loses to it if you top up $5 at a time. Self-hosting is cheapest of all at high volume, once your saved fees exceed your infrastructure cost.

Can one API key call GPT, Claude and Gemini?

Yes. That is the core function of an aggregator gateway: one key and one base URL, with the model chosen per request by changing a single string.

Does a gateway make my app slower?

Marginally. Published figures for well-provisioned gateways run from roughly 1ms to 8ms of added overhead, against LLM calls that take hundreds of milliseconds to several seconds. Under-provisioned gateways are a different story — benchmark on your own traffic.

Are gateways safe with sensitive data?

It depends entirely on the gateway's retention policy. Some log nothing by default; others log by default. If you handle regulated data, require a written zero-data-retention commitment and check for SOC 2, GDPR or HIPAA coverage — and remember that a gateway is an additional data processor in your compliance story.

How does model fallback work?

When the primary provider returns a 429 or a 5xx, the gateway retries the request against a configured alternative — a different host for the same model, or a different model entirely. Configure the chain explicitly and test it, because defaults vary and an untested fallback tends not to work.

Why would a gateway be cheaper than the model's own vendor?

Three plausible reasons: it resells negotiated volume capacity, it is subsidising growth, or it is serving a quantized or otherwise non-identical version of the model. The first two are fine. The third is not. Run your evals and pin the upstream provider where you can.

Can a gateway be more expensive than going direct?

Yes, and it happens. In our own Fable 5 price comparison, one gateway charges $11 / $55 per 1M tokens against Anthropic's $10 / $50 list — about 10% above list. "Reseller" does not imply "discount".

Should I self-host LiteLLM instead?

Self-host when a few percent of your monthly spend is larger than the cost of running and maintaining the proxy — database, compute, monitoring and someone on call. Below that threshold the managed fee is cheaper than the engineering time. Whichever you choose, pin your versions: two LiteLLM releases were briefly backdoored on PyPI in March 2026.

What is BYOK, and is it worth it?

Bring Your Own Key means the gateway routes your calls but bills land on your own vendor account, so you keep any negotiated rate. OpenRouter allows the first 1M BYOK requests per month free, then charges 5% of what the equivalent call would have cost on their platform. It suits teams with existing vendor agreements.

Do prepaid gateway credits expire?

Sometimes. OpenRouter's FAQ states it reserves the right to expire unused credits after one year, and its refund window is 24 hours with platform fees non-refundable. Read the credit terms before you load a large balance.

What is prompt caching and how much does it actually save?

Caching stores a stable prefix — typically your system prompt and retrieved context — so you are not charged full price to send it again. Anthropic, OpenAI and Google all price cache reads at one tenth of base input, a 90% discount, with Anthropic charging 1.25× or 2× to write the cache depending on TTL. For any RAG or agent workload it is the single biggest lever available.

Is semantic caching worth turning on?

Often, but calibrate your expectations. One published production case moved from an 18% hit rate to 67% and cut its LLM bill 73%; independent analyses put realistic hit rates for mixed traffic at 20–45%. Tune similarity thresholds per use case, and never let a semantic cache serve transactional queries loosely.

Do I need a gateway if I only use one model?

No. If you call one model from one vendor and have no plans to change, a gateway is an extra hop, an extra bill and an extra data processor for no benefit. Add one when you add your second provider — or when you first get paged because your only provider rate-limited you.

The bottom line

A gateway is infrastructure, not a discount. It buys you optionality — the ability to switch models, survive an outage, cap a budget and see where the money goes, without rewriting your app each time. That is worth a few percent to most teams, and worth building yourself to a few.

What it does not buy you is a free lunch. The largest savings available in 2026 are published by the model vendors themselves: 90% off cached input, 50% off batch, and a 5× to 24× price spread between a vendor's flagship and its small model. A gateway makes those easier to reach. It does not invent them, and no gateway's margin is bigger than any one of them.

So the order of operations is: turn on caching, move what you can to batch, route the boring work to a small model — and then choose a gateway to make all three easy to manage. Do it the other way round and you will have optimised the smallest number on the page.

Sources

• OpenRouter — FAQ and pricing (fees, BYOK, logging discount, credit expiry). Retrieved July 2026.
• Anthropic — Claude Platform pricing (list prices, cache multipliers, batch discount, tokenizer change, residency multipliers). Retrieved July 2026.
• OpenAI — API pricing (list prices, cached input, batch/flex, priority tier, regional uplift). Retrieved July 2026.
• Google — Gemini API pricing (list prices, context caching, batch discount). Retrieved July 2026.
• Cloudflare — AI Gateway pricing and limits (free tier, log caps, 5% Unified Billing fee, Unified API deprecation). Updated May 2026.
• Requesty — pricing page (5% token markup, free tier, EU data residency). Retrieved July 2026.
• LiteLLM — published benchmarks; security update on the March 2026 PyPI supply-chain compromise.
• Bifrost (Maxim AI) — published Bifrost vs. LiteLLM benchmarks. Vendor-run; attributed as such.
• Menlo Ventures — The State of Generative AI in the Enterprise (enterprise LLM API spend and provider share), 2025.
• Dataiku / Harris Poll — survey of 600 enterprise CIOs on multi-model strategy, April 2026.
• VentureBeat — production case study on semantic caching (query mix, hit rates, cost reduction), 2026.
• QwenLM qwen-code — open pull request on avoiding quantized models when routing through an aggregator.
• Best AI Gateways — our own Fable 5 gateway price comparison, July 2026.