Overview

The AI Gateway is a single Go binary that sits between your applications and upstream model providers. It exposes one OpenAI-compatible API for chat, streaming, embeddings, and images, and handles routing, authentication, plugins, and observability in one place — including a built-in operations dashboard, so there's nothing else to deploy to see what it's doing.

For a deeper technical view, continue to Architecture and Request lifecycle.

When to use it

• You call more than one LLM provider and want a single OpenAI-compatible endpoint instead of a client integration per vendor.
• You want failover, load balancing, or cost-aware routing configured in YAML — not rewritten into every service that calls a model.
• You need centralized auth, per-key budgets, and rate limiting in front of every provider, with a scoped Admin API to issue and revoke keys.
• You want one place to see what every request cost, how long it took, and which provider served it — the embedded dashboard, Prometheus metrics, or OpenTelemetry traces, without standing up a separate service.
• You're wiring MCP tool servers into an agentic loop and want the same guardrails and budget checks enforced on every turn, not just the first.

It matters less if you call exactly one provider directly and have no near-term plan for routing, guardrails, or centralized observability.

Core capabilities

Capability	What it gives you	Learn more
Routing	8 strategies (single, fallback, load balance, least latency, cost-optimized, content-based, A/B test, conditional). targets is an allowlist — a provider you've configured only routes traffic if it's listed there.	Routing
Providers	30 providers and 2,500+ models behind one API, with a capability matrix for which OpenAI parameters each one supports.	Providers
Plugins & guardrails	6 built-in plugins — word filter, max-token, response cache, rate limit, budget, request logger — running at before_request, after_request, and on_error stages.	Plugins
Admin API & key management	Scoped API keys (read_only / admin), dashboard sessions, an audit trail, and config history with rollback.	Admin API
Embedded dashboard	An operations console compiled into the binary and served from the gateway's own root — no separate image or origin to deploy.	Dashboard guide
Observability	OpenTelemetry tracing, Prometheus metrics, and request logs carrying duration_ms, ttft_ms, and cost_usd.	Monitoring
MCP	Connects to MCP tool servers over stdio or Streamable HTTP and drives the agentic tool_calls loop itself.	MCP guide

How it works

1. Your app sends an OpenAI-format request to the gateway. /v1/* requires a bearer token unless you've explicitly disabled it.
2. Guardrail, auth, and other before_request plugins run, then the routing strategy picks a target order.
3. The gateway calls the selected provider, retrying and failing over to the next target per your configuration.
4. after_request plugins run, the response is returned to your app, and the request is recorded to metrics, traces, and the request log.

Health and readiness

The gateway exposes three endpoints for orchestrators and monitoring, each answering a different question:

• /livez — is the process alive.
• /readyz — can the gateway actually route traffic; it returns 503 with reason no routable targets when none of your configured targets can serve a request.
• /health — a deeper diagnostic with per-provider status and circuit-breaker state.

Learn by depth

• Architecture — components, the embedded dashboard, and system boundaries.
• Request lifecycle — each processing stage and the fallback path.
• Providers, Routing, Plugins — configure what the gateway routes to and how.
• Guides — the dashboard, MCP, auth, and other operational tasks.