Frequently Asked Questions

Setup & Compatibility

Do I need to change my SDK or application code?

No. The gateway uses the OpenAI wire format. Change base_url and pass a bearer token as api_key:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="fgw_...")

Unlike many proxies, the gateway does not ignore api_key — every /v1/* route requires a bearer token by default (MASTER_KEY or an issued fgw_... key). A placeholder value like api_key="any" gets a 401. Auth can be disabled for local development only with ALLOW_UNAUTHENTICATED_PROXY=true; it is refused when GATEWAY_ENV=production. See Authentication.

Is there a web UI?

Yes. A dashboard is embedded directly in the gateway binary and served at the root path (/) — there is no separate container, image, or GATEWAY_BASE_URL to configure. It covers API key management, request logs, config history, provider health, analytics, and an in-browser Playground that authenticates with the same session token the admin API uses. See the Dashboard guide.

Which model names should I use?

Use the model IDs native to each provider — claude-3-5-sonnet-20241022 for Anthropic, gemini-1.5-pro for Gemini, gpt-4o for OpenAI, and so on. You can also define model aliases to decouple your application from a specific provider's model names, or use targets[].models to declare a model the gateway can't otherwise discover.

Can I run it locally?

Yes. Use Docker or build from source:

# Docker
docker run -p 8080:8080 -e OPENAI_API_KEY=sk-... ghcr.io/ferro-labs/ai-gateway:latest

# From source
git clone https://github.com/ferro-labs/ai-gateway && cd ai-gateway && make run

The image ghcr.io/ferro-labs/ai-gateway is a single multi-platform manifest — there are no separate -amd64/-arm64 tags to choose between.

Does it support streaming?

Yes. Set stream: true in your request body. The gateway streams the response using Server-Sent Events, identical to the OpenAI streaming format, across chat completions and every provider that supports it.

What Go version is required?

Go 1.25 or later to build from source. The gateway binary is a single static binary; pre-built binaries and a multi-platform Docker image (ghcr.io/ferro-labs/ai-gateway) are available on the GitHub releases page.

Providers

How do I enable a provider?

Set the provider's required environment variable(s) before starting the gateway — a provider is auto-registered only when its credential env var is present:

export ANTHROPIC_API_KEY=sk-ant-...
export GROQ_API_KEY=gsk_...

Registering a provider isn't the whole story: it also needs a targets[].virtual_key entry naming it in your config, or requests to it are never routed. See Provider configuration.

Can I use multiple providers at the same time?

Yes. Set credentials for as many providers as you want, then list each as a targets[] entry with a routing strategy. targets is an allowlist on every routed surface — a registered provider that isn't listed in targets[] returns 404 model_not_found for its models. See Routing for fallback, load-balance, and cost-optimized strategies.

Can I use Ollama or other self-hosted models?

Yes. Set OLLAMA_HOST=http://localhost:11434 (Ollama's server root — no API key needed) and, optionally, FERRO_OLLAMA_MODELS=llama3.2,mistral to narrow which models /v1/models advertises. The older OLLAMA_MODELS variable is deprecated and scheduled for removal — it's Ollama's own models-directory variable, and the gateway drops a path-shaped value with a warning rather than misreading it as a model list.

For any other OpenAI-compatible server, point the matching provider's <PROVIDER>_BASE_URL override at it (write the API root verbatim, version segment included).

Can I force a specific provider for one request?

Only on the pass-through proxy. X-Provider resolves which provider a /v1/* request that the gateway doesn't route natively (/v1/files, /v1/batches, and similar) gets forwarded to:

curl http://localhost:8080/v1/files \
  -H "Authorization: Bearer $MASTER_KEY" \
  -H "X-Provider: openai"

On routed surfaces — chat completions, streaming, embeddings, images — X-Provider has no effect. Those always resolve through your configured strategy and targets allowlist; a provider not in targets[] is refused regardless of any header.

Routing

What happens if a provider is down?

Depends on the strategy. Pool modes — fallback, loadbalance, least-latency, cost-optimized, ab-test — advance to the next target in the pool after a failover-safe failure — the provider was unreachable, timed out, returned 408/429/5xx, or is circuit-open or saturated; any other 4xx (400, 401, 403, 404, 422, …) is returned to the caller instead. Named modes — single, conditional, content-based — stay inside what was named: single reports its one target's failure, and a conditional or content-based rule walks its target_keys chain on the same failover-safe failures without ever reaching a target it did not name. Every mode skips a target whose circuit breaker is open, or that is parked after a 429, among the candidates it offers; if every target for a model is open, the request gets 503. targets[].retry is honored under every mode, so set attempts: 1 to keep single-attempt behavior even in a pool.

How does cost-optimized routing work?

The gateway ships with an embedded catalog of 2,500+ models with per-million-token input/output pricing. For each request under mode: cost-optimized, it estimates cost per configured target and routes to the cheapest one that's still routable. unpriced_strategy (fallback | skip | allow) controls what happens when a target's model has no catalog price.

What's the difference between least-latency and fallback?

fallback is reactive — it only moves to the next target after a failure. least-latency is proactive — it continuously tracks P50 latency from successful requests and prefers the fastest available target, even when every target is healthy.

Can I route different request types to different providers?

Yes, with the conditional strategy — but only on a closed set of keys, validated at config load: model (exact match) or model_prefix (prefix match). Arbitrary request headers or custom metadata aren't supported condition keys; a conditions[] entry naming anything else fails ferrogw validate before the gateway starts. For matching on prompt content instead of the model name, use content-based. See Conditional routing.

Plugins & Safety

What happens when I run out of budget?

The budget plugin returns 402 Payment Required with error type insufficient_quota once committed spend reaches the configured limit — not 429. It's a read-only soft-cap check (no reservation), evaluated at before_request, and must also be listed at after_request with byte-identical config so it can record spend. See Budget plugin.

Do plugins affect latency?

Guardrail plugins like word-filter and max-token add minimal overhead — pattern matching, not a network call. response-cache can dramatically reduce latency on cache hits by returning a response without calling any provider.

Will plugins block my requests in production?

Yes, when a plugin issues a Reject verdict — the request gets a 4xx, 429, or 402 response depending on the plugin. Separately, a plugin error (the plugin itself breaking) fails closed with a 500 for guardrail, auth, rate-limit, and transform plugins, but fails open for logging and metrics plugins, so a broken logger doesn't take down traffic. Review request logs at /admin/logs before enabling a new guardrail in production.

Can I write custom plugins?

Yes. Implement the plugin.Plugin interface in Go and register a factory with plugin.RegisterFactory in an init() function, then add a blank import in cmd/ferrogw/main.go. Plugins are configured globally under top-level plugins: — there's no per-route plugin field. See Plugins overview.

MCP (Model Context Protocol)

What is MCP?

Model Context Protocol is an open standard for connecting AI models to external tools and data sources. The gateway implements MCP as a client: when mcp_servers[] entries are configured, it injects their tools into chat completion requests and runs the agentic tool-calling loop internally. See the MCP guide.

Do my clients need to implement the tool loop?

No. Your client sends a normal POST /v1/chat/completions and gets back a final text response; every intermediate tool call happens inside the gateway, bounded by max_call_depth (default 5) and capped at 64 tool calls per turn.

Does MCP only work over HTTP?

No — each mcp_servers[] entry sets exactly one of url (Streamable HTTP, the 2025-11-25 transport revision) or command (+args, a stdio subprocess). A stdio server is launched at gateway startup and kept for the gateway's lifetime. It does not inherit the gateway's environment — only PATH/HOME/LANG/TMPDIR plus whatever you list explicitly in its env: block, so credentials like OPENAI_API_KEY or MASTER_KEY never reach it implicitly:

mcp_servers:
  - name: filesystem
    command: npx
    args: ["-y", "@modelcontextprotocol/server-filesystem", "/data"]
    env:
      SOME_TOKEN: ${SOME_TOKEN}
    required: false

Setting required: true gates GET /readyz on that server's initialize handshake — a required server that's down takes the whole instance out of rotation. Guardrail and budget plugins still run on every turn of the agentic loop, not just the initial request.

Operations

How do I know which providers are active?

GET /health reports per-provider status, model counts, and circuit-breaker state and is unauthenticated. GET /v1/models lists available models, but — like every other /v1/* route — it requires a bearer token unless ALLOW_UNAUTHENTICATED_PROXY=true is set.

What's the difference between /health, /livez, and /readyz?

All three are unauthenticated. /livez just confirms the process is up. /readyz answers whether the gateway can serve traffic: 200 ready when at least one configured target is routable, 503 not_ready (reason no routable targets) when none are — this is the one to point a load balancer or Kubernetes readiness probe at. It also folds in any MCP server marked required: true. /health is the deep diagnostic: per-provider status, model counts, and circuit state.

Is there an admin API?

Yes, under /admin/* — API key management, request logs, audit trail, config history and rollback, plugin catalog. Bootstrap it with MASTER_KEY (generated via ferrogw init), then issue per-operator fgw_-prefixed keys via POST /admin/keys. Omitting scopes on creation defaults to read_only. See Authentication and the interactive API reference.

Can I use PostgreSQL instead of SQLite?

Yes, per store. Set REQUEST_LOG_STORE_BACKEND=postgres and REQUEST_LOG_STORE_DSN=postgres://... for request logs. For the admin API key/session/audit store, set API_KEY_STORE_BACKEND=postgres and API_KEY_STORE_DSN=postgres://.... Config history follows the same pattern with CONFIG_STORE_BACKEND/CONFIG_STORE_DSN. All three default to in-memory, which doesn't survive a restart. See Server settings.

Is it production-ready?

Yes. Auth is on by default, per-IP rate limiting is on by default (20 rps / burst 40), and the gateway ships circuit breakers, retries with exponential backoff, Prometheus metrics, structured request logging, and an audit trail. Setting GATEWAY_ENV=production adds startup checks that refuse to boot with ALLOW_UNAUTHENTICATED_PROXY=true or a wildcard CORS_ORIGINS, and warns on an in-memory key store. See Monitoring.