Ferro Labs AI Gateway
Ferro Labs AI Gateway is an open-source, self-hosted AI gateway: a single Go binary that sits in front of all your LLM traffic, speaks an OpenAI-compatible API, and routes requests across 30 providers and 2,500+ models — with safety and cost policies enforced by built-in plugins, production observability, and a React dashboard embedded in the binary, served at the gateway's own root. No client code changes required.
Why put a gateway in front at all? Because you're calling one LLM provider directly: when they go down, your product goes down. When they raise prices, you scramble. When you want to try a new model, you rewrite integration code. When you need observability, you build it yourself.
Built for production from the start: sub-millisecond p99 routing overhead (benchmarks), retries and circuit breakers under every routing strategy, per-target concurrency limits, and /livez + /readyz probes for orchestrators.
Install in one command
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
docker run -d -p 8080:8080 \
-e OPENAI_API_KEY \
-e ANTHROPIC_API_KEY \
-e MASTER_KEY=$(openssl rand -hex 24) \
ghcr.io/ferro-labs/ai-gateway:latest
Send requests to http://localhost:8080/v1/chat/completions exactly as you would to OpenAI, authenticating with the MASTER_KEY you set (or a scoped key you issue from it). Open http://localhost:8080/ in a browser for the built-in dashboard. New here? The quickstart walks through ferrogw init, which scaffolds a config and a key for you.
/v1/* requires a key by defaultEvery inference route needs a bearer token unless you explicitly set ALLOW_UNAUTHENTICATED_PROXY=true. See Authentication.
Key capabilities
| Capability | Details |
|---|---|
| 30 AI providers | OpenAI, Anthropic, Gemini, Vertex AI, Mistral, Groq, Cohere, DeepSeek, Together, Perplexity, Fireworks, AI21, Azure OpenAI, Azure AI Foundry, xAI, Ollama, Ollama Cloud, Replicate, AWS Bedrock, Hugging Face, Cerebras, NVIDIA NIM, Cloudflare Workers AI, Databricks, Novita, Qwen, Moonshot, SambaNova, DeepInfra, OpenRouter — see the matrix → |
| 8 routing strategies | Single, fallback, weighted load balancing, least-latency, cost-optimized, conditional, content-based, and A/B test — routing guide → |
| 6 built-in plugins | Word filter, max-token, rate limit, budget, response cache, request logger — plus 5 Ferro Labs Managed security plugins. Plugin catalogue → |
| Unified request pipeline | Chat, streaming, embeddings, and image generation share one routing pipeline — retry, circuit breaking, per-target concurrency, timeouts, metrics, and request logging are true of every surface. |
| Native endpoints | Chat completions, embeddings, images, rerank, moderations, audio (STT/TTS), files & batches, and /v1/responses — governed and priced, not blindly proxied. API reference → |
| MCP tool-calling | Model Context Protocol servers over HTTP or stdio (run any npx/uvx server as a managed subprocess), with per-turn guardrails and budgeting on agentic loops. MCP guide → |
| Embedded dashboard | Overview, keys, request logs, analytics, config history, playground, audit, and tracing — compiled into the binary, served at the root. Dashboard tour → |
| Observability | Prometheus /metrics (scoped), OpenTelemetry tracing, structured JSON logs with 32-hex trace IDs, and /livez + /readyz probes. Observability → |
| Built in Go | Ships as a single static binary; sub-millisecond p99 routing overhead. See benchmarks → |
The dashboard now ships embedded in the OSS binary, all four core surfaces share one routing pipeline (with retry honoured under every strategy), and rerank / moderations / audio / responses became first-class routed endpoints. Some breaking changes ship with it — read the changelog before upgrading.
Where to next
Get started
- Overview — when and why to use the gateway
- Quickstart — Docker,
ferrogw init, your first request - Architecture — components and the unified request pipeline
- Configuration — the full config reference
- Concepts — routing, plugins, observability, MCP
Core reference
- Providers — all 30 providers, supported endpoints, and how to enable them
- Routing — the 8 strategies, with pool-vs-named behaviour and examples
- Plugins — the 6 built-in plugins, config, and pipeline stages
- Dashboard — the embedded console
- API reference — endpoints, streaming, admin, and errors
Operate
- Authentication — master key, scoped keys, and sessions
- Observability & Monitoring
- Server settings — every environment variable
- Troubleshooting
Need multi-tenancy, hosted infrastructure, semantic caching, and enterprise security plugins on top of the OSS engine? Ferro Labs Managed is in early access — join the waitlist →.