Skip to main content

Architecture

This page describes how a single gateway instance is actually built: one routing pipeline shared by every surface, the plugin stages that wrap it, the embedded dashboard and admin control plane, the MCP subsystem, and the state that does (and doesn't) survive a restart or scale across replicas.

High-level architecture​

The dashboard is not a separate service — it's a static SPA bundle embedded in the same binary and served from the gateway's root path, behind the same bearer-auth chain as the admin API.

Request path​

Core components​

Unified routing pipeline (routeTargets)​

This is the defining piece of the current architecture. Chat, streaming chat, embeddings, and image generation all route through the same function, routeTargets in gateway_pipeline.go — not four parallel implementations. The strategy contributes target order and nothing else; everything that happens once a target is chosen lives in the pipeline, once, for all four surfaces:

  • per-target retry (targets[].retry — honoured under every routing mode, not only fallback)
  • the per-target circuit breaker (one breaker per virtual_key, shared across all four surfaces — a target that only ever fails /v1/embeddings can still trip the breaker that stops it serving chat)
  • the per-target concurrency limiter (targets[].concurrency — max_concurrency + queue_size; overflow returns 429)
  • error classification, latency recording, metrics, and request logging

Plugin stages run outside this walk on purpose: a retry that re-ran a budget or guardrail plugin would bill or check the same request multiple times for one call. The pass-through /v1/* proxy and the priced /v1/responses surface both run through the same admission, plugin, and routing lifecycle as the four routed surfaces — they aren't a separate, ungoverned code path.

API compatibility layer​

Accepts OpenAI-compatible request/response shapes so application code stays stable as backend models and providers change. Requests to endpoints the gateway doesn't natively model (most non-chat OpenAI routes) fall through to the pass-through proxy, which still runs the governed lifecycle above.

Routing strategy​

Selects and orders candidate targets — nothing more. All 8 modes implement one method, SelectTargets: single, fallback, loadbalance, conditional, content-based, least-latency, cost-optimized, and ab-test. See Routing for per-strategy config and examples.

Plugin stages​

Global middleware (before_request, after_request, on_error) wrapping the pipeline — guardrails, rate limiting, budget enforcement, response caching, and request logging. See Plugins for stage semantics and failure policy.

MCP subsystem​

External tool servers for agentic tool-calling, wired from mcp_servers[] in config. Each entry is exactly one transport: url for Streamable HTTP, or command/args for a stdio subprocess (which inherits no gateway environment — only PATH/HOME/LANG/TMPDIR plus its own env block). Guardrails and budget checks re-run on every agentic loop turn, not just the first. See MCP.

Embedded dashboard + admin control plane​

The operations dashboard is a React SPA embedded in the OSS binary and served from the gateway's root path — there's no standalone dashboard container or second origin. The admin control plane behind it (API keys, dashboard sessions, config history/rollback, request logs, audit trail) uses the same bearer-auth chain the data plane does, so the in-browser Playground can call /v1/* with the operator's own session.

Observability​

Emits Prometheus metrics at /metrics and, when configured, OpenTelemetry traces. Health is split three ways:

EndpointAnswersAuth
/livezIs the process up?none
/readyzCan it serve traffic? 503 no routable targets when zero targets are routablenone
/healthDeep diagnostic — per-provider status and circuit statenone

Failover: pool vs named modes​

A circuit breaker doesn't produce failover by itself — a deployment with no breaker configured still fails over. The two do different jobs: the breaker makes failover cheap (skips a dead target's connection timeout), the routing mode makes it happen (decides whether to try a sibling at all).

ModesOn a target failure
Poolfallback, loadbalance, least-latency, cost-optimized, ab-testthe walk advances to the next candidate after a failover-safe failure; any other 4xx is returned
Namedsingle, conditional, content-basedthe walk stays inside what was named: single stops; a rule walks its target_keys chain on the same failover-safe failures and stops at its end

A pool mode picks its head target for a reason that's about the pool — spread load, take the cheapest, take the fastest — from targets declared interchangeable, so handing the request to a sibling is what was configured. A named mode picks its head because something named that target specifically; serving from a target the rule did not name would demote the rule to a suggestion, so a rule's target_keys chain is a hard boundary.

Every mode skips a target whose circuit is open, or that is parked after a 429, among the candidates it offers — a pool's siblings, or a rule's chain. single and a rule with one target offer no other candidate, so an open circuit there is refused with 503. When no other target serves the model, the walk still attempts the open one (so a model that plainly exists never 404s), the breaker refuses it, and the caller gets 503 upstream_unavailable instead.

Scaling boundaries​

Several pieces of gateway state are per-process and do not share across replicas unless noted:

  • Circuit breaker state, the least-latency tracker's p50 samples, and the rate-limit plugin's token buckets all reset with the process and are not shared across instances.
  • The response-cache and budget plugins are in-memory by default; budget spend does not survive a restart and is not a durable billing ledger.
  • The per-IP HTTP rate limiter (RATE_LIMIT_RPS) is also per-process.

What does persist and can be shared, when backed by sqlite or postgres (API_KEY_STORE_BACKEND, CONFIG_STORE_BACKEND, REQUEST_LOG_STORE_BACKEND): API keys, dashboard sessions, config history, the audit trail, and request logs. Point every replica at the same backend DSN to share those across a fleet; the in-memory default (used when these are unset) keeps each replica's copy independent and loses it on restart.

Deployment recommendations​

  • Start with one gateway instance; ferrogw init scaffolds a config from whichever provider credentials are already in the environment.
  • Set GATEWAY_ENV=production before exposing an instance publicly — it refuses to start with ALLOW_UNAUTHENTICATED_PROXY=true or a wildcard CORS_ORIGINS, and warns on the in-memory key store.
  • Move to multiple replicas behind a load balancer for HA; point API_KEY_STORE_BACKEND/CONFIG_STORE_BACKEND/REQUEST_LOG_STORE_BACKEND at a shared postgres DSN so keys, config history, and logs agree across instances (per-process state above still won't share).
  • Configure a circuit_breaker on any target that can fail, and prefer a pool routing mode (fallback/loadbalance) over a named one where interchangeable providers exist — that's what makes failover automatic rather than a manual failure to notice.
  • Use /metrics and /admin/logs to tune retry, concurrency, and routing weights over time.