Skip to main content

Configuration

As of v1.5.x

This page documents the config schema shipped by v1.5.2. Keys introduced on the v1.5 line — targets[].timeout, strategy.sticky, strategy.failover_on_status_codes, conditions[].target_keys — are rejected by a v1.4 binary's strict decoder. apiVersion is advisory — an unrecognized value is kept and only logged as a warning, so a newer config file still starts on an older binary as long as it uses only keys that binary knows.

The gateway loads configuration from a YAML or JSON file at the path set by GATEWAY_CONFIG.

export GATEWAY_CONFIG=./config.yaml
./ferrogw

Supported extensions: .yaml, .yml, .json. Decoding is strict — an unknown key is rejected (with its name and line number) rather than silently ignored, and the same strict decoder validates PUT/POST /admin/config, so a config pushed through the admin API can't disagree with a file on what's valid. Loading only decodes; a separate validation pass (ValidateConfig, also run by ferrogw validate) checks the values — an unroutable target_key, a negative weight, a duplicate multi-stage plugin — and a failure on either exits the process (os.Exit(1)).

Top-level fields​

KeyTypeDefaultDescription
apiVersionstringv1Advisory schema version. Never causes a load failure.
max_request_bytesint6410485760 (10 MiB)Body-size cap for /v1/* and admin write endpoints. Over the limit = HTTP 413. Batch/file uploads are exempt.
request_timeoutGo duration stringunset (no deadline)Bounds one non-streaming request end to end — plugin stages, provider call, and every retry/fallback attempt. Streaming requests are exempt, except an MCP agentic loop, which is delivered as a single chunk and is treated like any other non-streaming request.
strategyobjectmode singleRouting configuration — see Strategy.
targetsarray— (at least 1 required)Provider targets — see Targets.
batch_targetstringunset (surface returns 501)virtual_key of the target that serves /v1/files* and /v1/batches*. Must name a configured target whose provider supports batch pass-through (openai, azure-openai, groq, novita, qwen).
responses_targetstringunset (id sub-routes return 501)virtual_key of the target that serves the stateful /v1/responses/{id} sub-routes (retrieve/delete/cancel/input_items). POST /v1/responses (create) still routes by model regardless.
aliasesmap—Friendly model name → concrete model id, resolved before routing and plugins.
pluginsarray—Plugin middleware entries — see Plugins.
mcp_serversarray—External MCP tool servers — see MCP servers.
compatibilityobjecton_unsupported_param: warnSee Compatibility.
observabilityobjectNoOp (tracing off)See Observability.

Strategy​

The top-level strategy block controls how requests are routed.

strategy:
  mode: fallback  # single | fallback | loadbalance | conditional | least-latency | cost-optimized | content-based | ab-test
ModeFamilyDescription
singleNamedRoute every request to targets[0] only.
fallbackPoolTry targets in declared order; advance to the next after a failover-safe failure.
loadbalancePoolWeighted random distribution across targets (targets[].weight).
conditionalNamedMatch a request field (model, model_prefix, user, stream, has_tools, or a metadata header entry) to a target or an ordered target_keys chain; first match wins.
least-latencyPoolRoute to the compatible target with the lowest observed p50 time to first byte for the upstream model; samples expire and one request in ten explores a runner-up.
cost-optimizedPoolEstimate input plus output cost from the model catalog and pick the cheapest compatible target; equal-cost targets draw by weight.
content-basedNamedRoute on user-message content by substring or regex; first match wins.
ab-testPoolSplit traffic across labeled, weighted variants for comparison testing.

The mode families matter for failure handling: a pool mode (fallback, loadbalance, least-latency, cost-optimized, ab-test) advances the request pipeline to the next candidate target when one fails in a failover-safe way — a transport error, an attempt that timed out waiting on the target, 408, 429, 5xx, an open circuit, or saturation; any other 4xx is returned to the client, and the request's own cancellation or deadline stops routing. A named mode (single, conditional, content-based) stays inside what was named: single reports its one target's failure, and a rule walks its target_keys chain on the same failover-safe failures and stops at its end — it never reaches a target the rule did not name. strategy.failover_on_status_codes adds upstream statuses to the failover-safe set (never 400, 401, 403, 404 or 422).

Two rules hold under every mode, named or pool:

  • targets[].retry is honored regardless of mode — it re-asks the same target the number of configured times. Whether a different target is asked afterward is the mode's decision alone.
  • A target whose circuit breaker is open, or that is parked after answering 429 (for its Retry-After, a minute at most), is skipped among the candidates the mode offers. When every candidate is open or parked, the request is still attempted and answered 503 — that's a different signal from 404 model_not_found ("nothing serves this model" vs. "everything that could serve it is currently down"). A rule with one target offers no other candidate, so an open circuit there is 503.
  • targets[].timeout bounds one attempt against a target inside request_timeout; a timed-out attempt is failover-safe.
  • Every routed response carries X-Gateway-Provider, X-Gateway-Target, X-Gateway-Model and X-Gateway-Attempts.

Conditional rules​

strategy:
  mode: conditional
  conditions:
    - key: model
      value: gpt-4o
      target_key: openai
    - key: model_prefix
      value: claude-
      target_key: anthropic
targets:
  - virtual_key: openai   # targets[0] doubles as the no-match fallback
  - virtual_key: anthropic

key is one of model (exact match), model_prefix (prefix match), user (the request's user field), stream and has_tools ("true" / "false"), or metadata with field naming one entry of the X-Gateway-Metadata request header — a closed set validated at load; anything else is a config error, not a silent no-op. value is what key is matched against. A rule routes to target_key (one target) or target_keys (an ordered chain); exactly one is set, every entry must name a configured targets[].virtual_key, and none may repeat.

Rules are evaluated in order; the first match wins and the request stays inside the matched rule's chain — walking it on failover-safe failures and never reaching a target outside it. A request for a model the matched chain doesn't serve is 404 model_not_found, even when another configured target does serve it; a one-target rule whose target's circuit is open is 503. Write another rule, or add a chain member, rather than relying on failover. Unmatched requests fall to targets[0]. See Conditional.

Content-based routing​

strategy:
  mode: content-based
  content_conditions:
    - type: prompt_regex
      value: "(?i)\\b(code|function|class|implement)\\b"
      target_key: deepseek
    - type: prompt_contains
      value: "translate"
      target_key: gemini
    - type: prompt_not_contains
      value: "confidential"
      target_key: openai
targets:
  - virtual_key: openai   # no-match fallback
  - virtual_key: deepseek
  - virtual_key: gemini

Three condition types, evaluated over user-role messages only (system and assistant content is never inspected): prompt_contains (case-insensitive substring), prompt_not_contains (true when no user message contains the value — matches broadly, so ordering matters), and prompt_regex (Go regexp, compiled at startup — an invalid pattern is a startup error). First match wins and the request commits to that target; unmatched requests fall to targets[0].

content-based is a named mode: a rule may name a target_keys chain, which is walked on failover-safe failures, and the request never reaches a target the rule did not name. On non-chat surfaces, which carry no messages, the request takes the first target that can serve it.

Sticky hashing​

strategy:
  mode: loadbalance          # or ab-test
  sticky:
    on: user                 # the only supported key
    ttl: 1h                  # optional; a pin lasts at most one window

Under loadbalance and ab-test, sticky pins each request to the same target — or variant — for the same user field, so a conversation keeps its provider prompt cache and a session does not flip variants. It is a stateless hash: no shared state, the same answer on every replica, a random draw for a request without user. Refused under any other mode.

A/B test routing​

strategy:
  mode: ab-test
  ab_variants:
    - target_key: openai
      weight: 70
      label: control
    - target_key: anthropic
      weight: 30
      label: challenger

Weights are relative (weight / sum(weights)); a zero weight drains a variant to no traffic, a negative weight or an all-zero set is a load error. The draw is over eligible variants only — a variant whose provider doesn't serve the requested model never wins that draw. Each variant's label travels as ferro.routing.ab_variant_label on the request's observability events (delivered to registered exporters and custom providers — not a request-logger column or a span attribute). ab-test is a pool mode: a failover-safe failure at the drawn variant advances to the next configured target, while any other 4xx is returned to the client; the label stays the drawn variant's either way.

Targets​

Targets are provider references. Each virtual_key must name a registered provider (registration comes from setting that provider's env vars — see Provider configuration). targets is an allowlist on every routed surface: a provider can be fully registered via its env vars and still serve nothing if it isn't listed here — a model no configured target owns answers 404 model_not_found.

targets:
  - virtual_key: openai
    weight: 1.0           # relative share under loadbalance only; ignored elsewhere
    retry:
      attempts: 3
      on_status_codes: [429, 502, 503]   # omit for the default policy: transport errors + 408/429/5xx
      initial_backoff_ms: 100            # base for full-jitter exponential backoff
    circuit_breaker:
      failure_threshold: 5    # consecutive failures before opening (default 5)
      success_threshold: 1    # half-open successes needed to close (default 1)
      max_half_threshold: 1   # concurrent probes allowed while half-open (default 1)
      timeout: 30s             # time in open state before a half-open probe (default 30s)
    concurrency:
      max_concurrency: 32     # simultaneous in-flight requests to this target (1..10000)
      queue_size: 500         # requests allowed to wait for a slot; overflow => HTTP 429

  - virtual_key: gemini
    models:                   # models this target serves, declared by the operator —
      - gemini-2.5-flash      # ADDITIVE to the model catalog and live discovery
FieldTypeNotes
virtual_keystring, requiredNames a registered provider.
weightfloat64Relative share under loadbalance, and the tie-break among equal-cost targets under cost-optimized. 0 drains the target. Negative (any mode that reads it) or all-zero under loadbalance is a load error.
timeoutdurationBound on one attempt against this target ("8s"), inside request_timeout. A unary attempt is bounded through its response; a streaming attempt only until the provider answers. A timed-out attempt is failover-safe. Must be a positive Go duration.
models[]stringOperator-declared model ids this target serves, in addition to whatever the catalog and live discovery already report. Exact ids only — wildcards are rejected at load. Purely additive: it never hides models the target already serves, and declaring one the catalog already knows is a harmless no-op. Advertised on /v1/models.
model_mapmapPer-target translation of a name clients use into this target's upstream model id (smart: gpt-4o-mini). The visible name routes to this target and is listed in /v1/models; the upstream call and pricing use the mapped id; the response carries the visible name. Per target, unlike the global aliases. See Routing.
retryobjectattempts (int, 1 = no retry), on_status_codes ([]int), initial_backoff_ms (int, default 100). Applies under every routing mode — it re-asks this one target; it does not by itself try a different target.
circuit_breakerobjectOne breaker per virtual_key, shared across chat, streaming, embeddings, and image generation to that target — a target that fails only on one surface still opens the shared breaker and stops serving all four.
concurrencyobjectmax_concurrency (1..10000) and queue_size. A streaming request holds its slot until the stream ends, not just until headers arrive. Overflow past the queue is 429.

A 429, a client disconnect, a caller-supplied deadline, an unsupported-parameter rejection, and a concurrency shed do not count toward opening the circuit breaker. A redirect, a 5xx, a connection failure, and the gateway's own request_timeout or stream idle bound elapsing all do.

Declared models (targets[].models)​

Reach for models when a target serves a model none of the automatic sources can see: an id newer than the catalog, a preview or regional name, a self-hosted deployment behind <PROVIDER>_BASE_URL, or a provider with no /models endpoint to enumerate. It's provider-agnostic — every target has the field — and strictly additive, so it can never accidentally unroute a model the target already serves.

Batch and Responses backends​

batch_target: openai        # serves /v1/files* and /v1/batches* pass-through
responses_target: openai    # serves the stateful /v1/responses/{id} sub-routes

Both endpoints carry no model — a batch references an opaque, provider-scoped id, so a single named target serves the whole surface with native ids and zero gateway state. batch_target's provider must support batch pass-through (openai, azure-openai, groq, novita, qwen); responses_target is typically openai or xai. Omitting either leaves the corresponding surface returning 501 — POST /v1/responses (create) still routes normally by model even with responses_target unset.

Model aliases​

Aliases resolve before routing and before plugins run. They let you use short names and swap the backing model without changing client code.

aliases:
  fast: gpt-4o-mini
  smart: claude-sonnet-4-6
  cheap: gemini-2.5-flash
  code: deepseek-coder

No empty names or values, no self-reference, and no chained aliases (an alias pointing at another alias is rejected at load).

Plugins​

Each entry specifies name, type (a label only — the plugin's own reported type decides fail-open vs. fail-closed, not this field), stage, enabled, and config. Order within a stage is execution order.

plugins:
  - name: word-filter
    type: guardrail
    stage: before_request
    enabled: true
    config:
      blocked_words: ["password", "secret"]
      case_sensitive: false

  - name: max-token
    type: guardrail
    stage: before_request
    enabled: true
    config:
      max_tokens: 4096
      max_messages: 50
      max_input_length: 0   # 0 = no limit

  - name: rate-limit
    type: guardrail
    stage: before_request
    enabled: true
    config:
      requests_per_second: 100
      burst: 100
      key_rpm: 60      # optional per-API-key limit
      user_rpm: 30     # optional per-user limit (keyed on Request.User)
Multi-stage plugins need byte-identical config

response-cache, budget, and request-logger each implement more than one stage — response-cache checks the cache before the provider call and stores after it, budget checks spend before and records after, request-logger writes a row at every stage it's listed at. Every entry for the same plugin must carry identical config (checked as JSON-encoded equality of the whole config block), or the gateway refuses to start. The same store_id/name alone is not enough — any other key differing between entries is still two disagreeing configs.

plugins:
  # response-cache: before_request serves a cache hit (skips only the
  # provider call — every other before_request plugin still runs); after_request stores it.
  - name: response-cache
    type: transform
    stage: before_request
    enabled: true
    config:
      max_age: 300
      max_entries: 1000
  - name: response-cache
    type: transform
    stage: after_request
    enabled: true
    config:
      max_age: 300
      max_entries: 1000

  # request-logger: register at all three stages it implements, including
  # on_error — a failed request never reaches after_request, so without the
  # on_error entry a failure produces no terminal row at all.
  - name: request-logger
    type: logging
    stage: before_request
    enabled: true
    config:
      level: info
      persist: true
  - name: request-logger
    type: logging
    stage: after_request
    enabled: true
    config:
      level: info
      persist: true
  - name: request-logger
    type: logging
    stage: on_error
    enabled: true
    config:
      level: info
      persist: true

  # budget: before_request checks accumulated spend against the limit;
  # after_request records what the completed request cost.
  - name: budget
    type: guardrail
    stage: before_request
    enabled: true
    config:
      store_id: default            # instances sharing store_id share spend counters
      spend_limit_usd: 10.0        # 0 = unlimited
      input_per_m_tokens: 3.0      # USD per 1M prompt tokens
      output_per_m_tokens: 15.0    # USD per 1M completion tokens
      cache_read_per_m_tokens: 0.30   # optional: USD per 1M cached-read prompt tokens
      cache_write_per_m_tokens: 3.75  # optional: USD per 1M cache-write tokens
      max_keys: 10000               # max tracked API keys; evicts lowest-spend key at cap
  - name: budget
    type: guardrail
    stage: after_request
    enabled: true
    config:
      store_id: default
      spend_limit_usd: 10.0
      input_per_m_tokens: 3.0
      output_per_m_tokens: 15.0
      cache_read_per_m_tokens: 0.30
      cache_write_per_m_tokens: 3.75
      max_keys: 10000

An exhausted budget rejects the request with 402 insufficient_quota, not a 429 — explicitly so SDKs don't retry a request that will never succeed. PromptTokens from the provider is inclusive of CacheReadTokens, so setting cache_read_per_m_tokens bills the cached subset at that rate and the remainder at input_per_m_tokens; leaving it unset bills the whole prompt at input_per_m_tokens as before.

Plugin config string values support ${VAR} — see ${VAR} resolution below.

See Plugins for all 6 built-in OSS plugins (word-filter, max-token, response-cache, request-logger, rate-limit, budget) and their full option sets. The 5 additional guardrails (pii-redact, secret-scan, prompt-shield, schema-guard, regex-guard) are Ferro Labs Managed-only — see Enterprise plugins.

MCP servers​

Configure external MCP tool servers for agentic tool-calling. When mcp_servers is configured, the gateway injects the discovered tools into a chat completion request only when the request itself carries no tools of its own — a caller-supplied tools array passes through untouched and MCP sits out entirely for that request.

Each entry sets exactly one of url (Streamable HTTP transport) or command (+ args, stdio subprocess transport).

mcp_servers:
  # Streamable HTTP transport
  - name: filesystem
    url: "http://localhost:3001/mcp"
    timeout_seconds: 10     # per tool call, both transports; default 30
    max_call_depth: 3       # agentic loop turn cap; min positive value across servers wins, default 5

  - name: database
    url: "https://mcp-db.internal/mcp"
    headers:
      Authorization: "Bearer ${MCP_DB_TOKEN}"
    allowed_tools:           # empty = all discovered tools exposed
      - query_readonly
      - list_tables
    timeout_seconds: 15
    max_call_depth: 5
    required: false          # true gates /readyz on this server's initialize handshake

  # stdio transport — the gateway launches and owns the subprocess.
  # It does NOT inherit the gateway's environment: only PATH/HOME/LANG/TMPDIR
  # (when set) plus the explicit `env` map below reach the child process.
  - name: brave-search
    command: npx
    args:
      - -y
      - "@modelcontextprotocol/server-brave-search"
    env:
      BRAVE_API_KEY: "${BRAVE_API_KEY}"
    max_call_depth: 3
FieldTypeDefaultNotes
namestring, required—Unique; used in logs, metrics, and the /readyz body.
urlstring—Streamable HTTP endpoint. Set exactly one of url / command.
command / argsstring / []string—stdio subprocess launched at gateway startup, kept for the gateway's lifetime.
headersmap[string]string{}HTTP transport only. ${VAR} supported.
envmap[string]string{}stdio transport only — the sole credential channel to the subprocess, since it inherits no gateway env. ${VAR} supported.
allowed_tools[]stringallRestricts which discovered tools are exposed to the LLM.
timeout_secondsint30Per-tool-call timeout.
max_call_depthint5Agentic loop depth bound; the minimum positive value across all configured servers applies.
requiredboolfalseWhen true, this server's readiness gates GET /readyz (503, reason "required mcp server unavailable" when unready). Unready means the initialize handshake hasn't completed. Death after a successful handshake is detected for stdio servers only — an HTTP server that goes unreachable post-handshake keeps reporting ready.

Guardrails and the budget plugin run on every turn of the agentic loop, not just the initial request. See MCP integration for the full tool-execution flow.

Compatibility​

Controls how the gateway treats an OpenAI-shaped request parameter that the routed provider can't express (see GET /v1/capabilities for the full matrix).

compatibility:
  on_unsupported_param: warn   # warn | drop | reject
ValueBehavior
warn (default)Forward the parameter anyway and log a warning.
dropRemove the parameter from the upstream request and log.
rejectFail the request with HTTP 400 naming the parameter.

warn and drop differ only for providers reached over an OpenAI-compatible request body — there, warn genuinely forwards the parameter. A provider with a native wire format (Anthropic, Bedrock, Gemini, Cohere, AI21, Replicate) builds a payload with nowhere to put an unsupported parameter, so warn and drop send byte-identical upstream requests there and both log "dropping." Use reject when a caller needs to know a parameter wasn't honored rather than silently dropped.

Observability​

OpenTelemetry tracing. Omit the section entirely (or leave endpoint empty with no OTEL_EXPORTER_OTLP_* env set) and the gateway runs a zero-allocation NoOp provider at no cost.

observability:
  tracing:
    enabled: true              # tri-state: omit = infer from endpoint, false = hard off, true = force on
    endpoint: ""                # URL or bare host:port; blank falls back to OTEL_EXPORTER_OTLP_* env
    protocol: grpc              # grpc | http/protobuf
    service_name: ferrogw
    sample_ratio: 1.0           # head sampler 0.0-1.0, wrapped in ParentBased
    privacy_level: metadata     # none | metadata | full
    shutdown_grace: 10s         # per OTel shutdown stage; total shutdown can take up to 2x this
    headers:                    # OTLP export headers, e.g. vendor API keys
      dd-api-key: "${DATADOG_API_KEY}"

  exporters:                    # plugin observability exporters — none ship in this repo
    - name: langsmith
      enabled: true
      config:
        api_key: "${LANGSMITH_API_KEY}"

The gateway itself reads only two OTEL_* environment variables — OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_TRACES_ENDPOINT — and either one takes precedence over observability.tracing.endpoint when set. Setting either turns tracing on. OTEL_TRACES_SAMPLER has no effect; the sampler is config-only (sample_ratio, ParentBased) — an inbound request that already carries a sampled traceparent is followed regardless of sample_ratio.

exporters[] entries reference exporter plugins registered via observability.RegisterExporter in a plugin's init(); they ship separately in the ai-gateway-plugins repo. An unrecognized name logs a warning and is skipped — it isn't fatal.

${VAR} resolution​

Config string values in plugins[].config, mcp_servers[].headers, mcp_servers[].env, observability.tracing.headers, and observability.exporters[].config support environment variable references.

  • Only the braced form is a reference. ${NAME} (matching [A-Za-z_][A-Za-z0-9_]*) is expanded; a bare $ is literal data — $100 and pa$$w0rd survive byte-for-byte.
  • An undefined variable is a hard error naming every missing variable — never a silent empty-string substitution.
  • Resolution happens at component construction, not at config load. The loaded Config keeps the literal ${VAR} text, so a secret never reaches the config-history store, GET /admin/config, or a rollback snapshot — and a config pushed through the admin/GitOps API (which never passes through LoadConfig) is expanded identically.
  • It does not apply to core routing fields (virtual_key, mode, model ids, aliases) — only the free-form/credential-bearing maps listed above.

Complete example​

Copy config.example.yaml from the repository for a fully annotated example covering all 30 providers, both routing families, all 6 OSS plugins with correct multi-stage registration, MCP (both transports), compatibility, and observability.