The Model Gateway Is the Control Plane Your LLM Stack Needs

Model providers are not five-nines infrastructure. Anthropic's 90-day uptime sat near 98% as of April 2026, and OpenAI's global outage in mid-2025 ran more than fifteen hours ¹. A team that calls one provider directly has built a single point of failure into the middle of its stack, and its only recovery option is a human noticing the error pages and editing an environment variable ¹.

The gateway is the piece that sits between your application and every model provider. Your code talks to one HTTP endpoint with one API key, and the gateway handles the rest: routing decisions, cached responses, retries with fallback, rate limits, and cost accounting ². Model spend hit $12.5 billion in 2025, up from $3.5 billion in late 2024, and 72% of organizations planned to grow that number through 2026 ³. Most of that budget leaks through the same three holes: routing every request to a frontier model regardless of difficulty, paying full token price for responses already computed, and depending on a single provider that takes the whole app down when it degrades ³.

The gateway is a control plane, not a load balancer

An API proxy forwards requests and handles authentication. A gateway adds a token-level control plane above that forwarding layer: it makes routing decisions based on cost and latency, matches prompts that are similar but not identical, and recovers from provider failures before any error reaches the caller ³. The distinction is where the intelligence lives.

This matters because production LLM workloads are not uniform. The same application sends classification calls, summarization tasks, and multi-step reasoning chains, and without a router all of it goes to the same frontier model at frontier prices ³. The model market has bifurcated: Claude Haiku 4.5 at $0.80 per million input tokens is 18 times cheaper than Claude Opus 4.7 at $15 per million, and GPT-4o-mini at $0.15 is 100 times cheaper than o3 at $15 ⁴. Most production traffic is simple enough for the cheap tier to handle cleanly; the slice that genuinely needs Opus-priced reasoning is small ⁴. Routing is the mechanism that puts each request where it belongs.

Roughly 37% of enterprises now run five or more models in production ⁵. Direct integration with each provider means maintaining separate API formats, auth flows, and error handling per model, and the engineering overhead scales linearly with every model you add ⁶. A gateway collapses all of that to one integration.

Three jobs the gateway owns

Routing. Before any provider call happens, the gateway intercepts each outbound request and applies a routing policy ³. Four strategies dominate production. Cost-based routing assigns each model a cost score and picks the cheapest option above a capability floor you define. Latency-based routing tracks rolling p95 latency per model and shifts traffic toward the fastest provider. Complexity-based routing estimates task difficulty and sends simple requests to small models and hard reasoning to frontier tiers; hybrid systems built on it report 37-46% reductions in LLM usage ³. Load-balanced routing spreads traffic across multiple API keys to stay under per-key rate limits ³.

The cascade pattern is the one that ships most often: try the cheapest model first, and only escalate when the cheap call fails a quality signal ⁴. The hard part is defining the signal. A self-reported confidence field is cheap but unreliable because models calibrate their own uncertainty poorly. Schema validation is fast and reliable but only works for structured-output tasks. An LLM-as-judge catches semantic failures the other two miss but adds latency and a small token cost ⁴. The trap is a bad signal that produces false failures, because then you pay for both the cheap call and the expensive escalation. Track net savings, not the headline routing rate, or a 70% cheap-tier route with a 50% false-failure rate is worse than just running Sonnet for everything ⁴.

Failover. Fallback and cascade solve different problems. Cascade is about cost: escalate on a quality failure. Fallback is about availability: when the primary provider is down or rate-limited, try the next one ⁴. A fallback chain defines a priority-ordered list of providers and the triggers that move traffic between them, and a well-built chain fails over on provider error, rate-limit exhaustion, timeout, or a latency threshold ⁵⁷. The caller never changes code; it sees a single response regardless of which model served it, and the gateway records which one did for cost attribution ⁵.

The failure mode most teams miss is cascading retry amplification. When a provider slows down instead of timing out, requests pile up waiting, and retries spawn duplicate in-flight requests against an already-struggling provider, which eventually overwhelms the healthy ones too ³. The fix is a circuit breaker that stops sending new requests to a degraded provider immediately rather than after N timeouts, then reopens after a health check confirms recovery ³. LiteLLM, Kong, and Portkey all implement circuit breakers, but the defaults are too permissive for high traffic. Set a short timeout, three-to-five seconds for interactive requests, trip the circuit after three consecutive failures, and require a successful health probe before reopening ³.

Budgets. Provider APIs enforce rate limits, not budget guardrails. Without a gateway there is no mechanism to reject an override before it hits the invoice, attribute spend to the team or feature that generated it, or kill a runaway prompt loop before it drains a monthly budget in an hour ³. This becomes a safety mechanism, not just a cost tool, once agents are in the mix: an autonomous agent with no token budget is a runaway-cost risk, and a session-scoped virtual key with a hard limit is the standard way to cap it ⁵. Hierarchical budgets layer organization, team, user, application, and feature ceilings so one tenant's spike cannot starve another's quota ⁵.

Caching is the biggest single lever

Routing gets the attention, but caching cuts deeper. Semantic cache hits return in under 5 milliseconds against 2-5 seconds for live inference, and production deployments typically hit 30-40% cache hit rates ³. Each hit eliminates the full token cost of that call. Three caching layers stack independently. Exact-match caching stores full prompt-response pairs and reaches 70-80% hit rates on templated, narrow request distributions. Semantic caching uses vector similarity so that "what is your return policy?" and "can I return this item?" map to the same slot when their embeddings fall within a tunable cosine threshold, typically 0.90 to 0.98 ³. Provider-level prompt caching runs inside the provider on repeated token prefixes; Anthropic's prefix caching delivers up to 90% cost reduction and 85% latency reduction on prompts with frequently reused prefixes ³.

The three levers compound. Combining prompt caching, semantic caching, and model routing produces a 47-80% total cost reduction, and on heavily repeated workloads the savings run higher ³⁵. The right caching investment depends on your request distribution. Inspect a month of prompt logs and plot the top repeated semantic clusters: if more than half of traffic falls into a couple dozen intent buckets, semantic caching pays for itself in the first week; if the distribution is long-tailed, concentrate on prompt caching for the system prefix and accept weak semantic-cache performance until volume scales ³.

Choosing a gateway

The toolset has converged on the same core features, so the decision is spend level and operational context, not a feature checklist ⁷. This gateway layer sits in front of the running models themselves; if you are serving those models on Kubernetes, the inference-aware routing we covered in our piece on Kubernetes for LLM inference handles placement, and the gateway handles provider choice and failover on top of it. LiteLLM is the de facto standard for self-hosted gateways in 2026: it covers 100-plus providers behind an OpenAI-compatible interface and ships six routing strategies, but its throughput ceiling becomes a real risk above roughly 2,000 requests per second, where it has historically consumed significant memory and cascaded into timeouts ³⁷. The broad guidance: start with LiteLLM under about $10,000 a month in spend, consider Cloudflare AI Gateway for geographically distributed users because it deploys to more than 300 edge nodes with zero operational overhead, and move to Portkey or Kong above $50,000 a month when audit trails, team-level budget enforcement, and SLA-backed support justify themselves ³.

There is a hidden inflection point at about $2,000 a month in token spend: at that level a managed gateway fee is cheaper than the engineering time required to operate a self-hosted instance ³. Run that calculation before defaulting to open source.

One caution applies to every choice: the LiteLLM supply chain attack in March 2026 planted malware in versions 1.82.7 and 1.82.8 that harvested SSH keys and credentials, so staying current on gateway patches is as important as choosing the right one ⁷.

Doing it inside a real pipeline

We route model traffic through a gateway layer in our own delivery pipeline, for the same reasons any team running agents against cloud providers should. We keep a local model for free-text subsystems that do not need the strongest reasoning, and we send structured and tool-calling work to cloud providers that enforce output constraints reliably. A gateway makes that split cheap to maintain: the classification rules that send a request to the local model or a cloud model live in one config file, not scattered as model-picking conditionals across services. When a cloud provider's latency drifts upward or a rate limit trips, the fallback path is already configured, the difference between a graceful shift and a pager at 3 a.m.

We hit the shared failure mode when the routing signal first over-applied the cheapest route. The net result was more escalation calls than the routing saved, exactly the cascade trap in the research: a 70% cheap-tier target with a false-failure-heavy signal costs more than a sensible default ⁴. The fix was to tighten the quality signal and measure net saving rather than the headline cheap-tier percentage. The lesson is the one any review gate applies to infrastructure: the architecture is sound, and the economics only hold when you measure the right number.

The decision we made that you have to make too

The gateway category crossed from optional tooling to production infrastructure, not because the market announced it but because providers kept going down ⁷. The decision is no longer whether to run multi-provider routing. Either you have a routing layer or you have a single point of failure ⁷. Teams that move now pay the integration cost once, on their own schedule. Teams that wait pay it during the next outage, plus the downtime ⁷.

Start smaller than it sounds. A LiteLLM proxy deploys in under an hour and gives you routing, caching, and failover behind the OpenAI-compatible interface your code probably already speaks ³⁴. Add semantic caching once you have a month of prompt logs to understand the request distribution. Wire up a fallback chain before users hit the product. Instrument four metrics from day one: token cost per request by model, cache hit rate by request type, provider error rates and latency at p95 and p99, and fallback activation rate, with a working rule that more than 10% of requests hitting secondary providers means the primary is struggling and you should investigate before users notice ³.

The ceiling is high and the entry point is low. A gateway is the control plane your model traffic has been missing, and the cost of adding one now is a fraction of the cost of the next provider outage ³.