What is an LLM Gateway? (vs Router & Proxy)
What is an LLM gateway? An LLM gateway is a proxy layer between your application and language-model providers. It exposes one API for routing, failover, billing, and usage tracking across many models, so application code does not talk to each vendor separately.
Teams usually meet this problem after the first production integration: one SDK for OpenAI, another key for a second vendor, a third dashboard for spend. An LLM gateway exists to collapse that surface into a single contract.
How is an LLM gateway different from a router, a proxy, and an inference provider?
| Layer | What it does | What it does not do |
|---|---|---|
| Proxy | Forwards HTTP to an upstream URL. | Usually no multi-model catalog, billing, or failover policy. |
| Router | Chooses which model or channel should handle a request. | May not own keys, credits, or a product API for developers. |
| LLM gateway | Combines a stable developer API with routing, auth, metering, and operations. | It is not the GPU cluster that runs the model. |
| Inference provider | Trains or hosts models and returns completions. | Does not unify other vendors’ APIs for you. |
In practice a product like ZBStream is a gateway that includes routing: one OpenAI-compatible endpoint, one API key, and policies for which upstream channel to use when a model is requested.
Why do teams add an LLM gateway?
- One integration. Application code keeps using Chat Completions (or Messages) while models change behind the gateway.
- Availability. If one upstream is rate-limited or down, another channel can serve the same model id when you have configured failover.
- Cost and audit. Tokens, cache hits, and credits live in one ledger instead of N vendor invoices.
- Access control. Keys can be scoped to an application and rotated without touching every client.
What does a typical request path look like?
- Your app sends
Authorization: Bearer <gateway-key>to the gateway base URL. - The gateway authenticates the key and resolves the requested
modelid. - A routing policy picks an upstream channel (provider account + model name).
- The upstream returns tokens; the gateway records usage and returns an OpenAI-shaped response.
Details for ZBStream are in the first-call docs and model routing guide.
When is a gateway the wrong tool?
If you only ever call one vendor, with one key, and you already have billing alerts there, a gateway adds a hop you may not need. If you must pin a request to a specific GPU region or a private VPC endpoint that the gateway cannot reach, talk to the inference provider directly. Gateways help when you have more than one model, vendor, or environment.
FAQ
Is an LLM gateway the same as an API gateway?
No. A generic API gateway (Kong, AWS API Gateway) terminates HTTP, auth, and rate limits for any service. An LLM gateway additionally understands model ids, token usage, and provider-specific error handling.
Does ZBStream replace OpenAI?
No. ZBStream is not an inference provider. It is a gateway in front of models you already want to call, including OpenAI-compatible routes operated by other vendors.
Can I keep using the OpenAI SDK?
Yes. Point base_url at the gateway and use a gateway API key. See Unified LLM API.