Skip to main content

Fallback routing strategy

Fallback tries targets in the order you declare them, advancing to the next target when the current one fails in a way another provider can fix. It optimizes for availability: a request that failed because a provider was unreachable, timed out or overloaded is answered by the next-best provider instead of being reported as a failure — not for spreading load (loadbalance), speed (least-latency), or cost (cost-optimized). Set strategy.mode: fallback to use it.

In plain words

List your providers in the order you trust them. If the first is unreachable, times out or is overloaded, the same request goes to the next one and the client never sees the failure. If the first says the request is wrong — bad key, unknown model, invalid input — the client gets that answer, because another provider cannot fix it.

What happens to a request​

With targets: [openai, anthropic] and retry.attempts: 3 on OpenAI:

OpenAI…The gateway…The client gets
answers 200returns itOpenAI's answer
returns 503 three timesretries twice with backoff, then asks AnthropicAnthropic's answer
returns 429 with Retry-After: 2waits 2 s and retries; moves to Anthropic once the attempts are spentan answer from whichever target succeeded
never sends response headersgives up when its targets[].timeout (or the provider transport's timeout) expires and asks AnthropicAnthropic's answer
says the prompt exceeds its context windowasks Anthropic, whose model may have a larger windowAnthropic's answer
returned 429 a moment agois skipped for its Retry-After; Anthropic is asked directlyAnthropic's answer
returns 401 (revoked key)stops — neither a retry nor a sibling can fix a bad key401 from OpenAI
— the client disconnects firststops routingnothing; the request was cancelled

Behaviour​

fallback is a pool mode — the only mode whose entire purpose is advancing past a failed target in the order you declared. Fallback.SelectTargets (internal/strategies/fallback.go) returns every targets[].virtual_key in declared order, built once at construction; there is no per-request selection logic, no model filtering, and targets[].weight is ignored under this mode. The gateway pipeline (routeTargets, gateway_pipeline.go) walks that order: when the head target fails in a failover-safe way — a transport failure, an attempt that timed out waiting on the target, 408, 429, 5xx, a provider's own context-length overflow, any status in strategy.failover_on_status_codes, an open circuit, a park after a 429, or a full concurrency queue — and its retry budget, where one applies, is spent, the walk advances to the next declared target rather than reporting the failure back to the caller. Any other 4xx is a verdict on the request itself and is returned unchanged; the request's own cancellation or deadline stops the walk.

targets[].retry governs how many times one target is re-asked before the pipeline moves on — it applies under every routing mode, not only fallback. Fallback holds no retry policy of its own; advancing to the next target is fallback's behaviour alone. An open-circuit or surface-incapable target is skipped over before the pipeline commits to it, under every mode. SelectTargets never returns nil (declaring at least one target is required to build the strategy), but if every candidate's circuit is open the pipeline still attempts one and answers 503 rather than 404.

Config keys​

KeyTypeDefaultDescription
strategy.modestring—Set to fallback.
targets[].timeoutdurationnoneBound on one attempt against this target, inside request_timeout. Unary attempts are bounded through the response; streaming attempts only until the provider answers. A timed-out attempt is failover-safe.
targets[].retry.attemptsint1 (no retry) — omitted or <= 0 normalizes to 1Maximum attempts against this one target before the pipeline advances to the next. Read per target, honored under every strategy mode.
targets[].retry.on_status_codes[]inttransport errors + 408, 429, and 5xxRestricts retries to these HTTP status codes. Any other 4xx is treated as a deterministic client error and is never retried.
targets[].retry.initial_backoff_msint100Base for exponential backoff with full jitter: the wait before attempt N is drawn uniformly from [0, initial_backoff_ms * 2^(N-1)). An upstream Retry-After header, when present, wins over the computed wait.

Minimal working YAML​

strategy:
  mode: fallback

targets:
  - virtual_key: openai
    retry:
      attempts: 3
      on_status_codes: [429, 502, 503]
      initial_backoff_ms: 100
  - virtual_key: anthropic

One name across providers​

Providers use different model IDs, so give the name your clients send a translation on each target:

strategy:
  mode: fallback

targets:
  - virtual_key: openai
    model_map:
      smart: gpt-4o
    retry:
      attempts: 3
  - virtual_key: anthropic
    model_map:
      smart: claude-sonnet-4-6

A request for smart is served by OpenAI as gpt-4o and, on failover, by Anthropic as claude-sonnet-4-6; the response says "model": "smart" either way. See One model name, different upstream IDs.

When to use​

  • A primary provider plus one or more backups, where getting an answer from someone matters more than getting it from the preferred provider every time.
  • Declared priority matters: you want traffic to favor a specific provider (cheapest, fastest, contractually preferred) and spill to the next one only when the preferred provider fails.
  • You want failover without hand-tuning weights (loadbalance) or relying on measured latency (least-latency).

Gotchas​

  • Fallback holds no retry policy of its own. targets[].retry.* is the pipeline's config, resolved per target and applied identically under every routing mode. What's specific to fallback is only the decision to advance to the next target once that target's retry budget is spent.
  • Declared order is priority order — there are no weights. targets[].weight is read only by loadbalance; under fallback, the first entry in targets[] is always tried first.
  • Not every failure fails over. Any other 4xx from a target — 400, 401, 403, 404, 422, and the rest — is returned to the client; another provider cannot fix a malformed request or a revoked key. An attempt that timed out waiting on the target, an open circuit and a full concurrency queue advance to the next target at once, without spending a retry; a transport failure, 408, 429 and 5xx advance once the retry budget is spent. The client's own cancellation or deadline stops routing entirely — nothing is retried or advanced on the caller's behalf.
  • A Retry-After hint longer than 30 seconds abandons the target instead of waiting on it. The gateway won't hold a request open past that cap; it advances to the next target immediately and logs that the target was abandoned.
  • Only deterministic-retry statuses are retried by default. With no on_status_codes set, retries apply to transport failures plus 408, 429, and 5xx. Every other 4xx (e.g. 400, 401, 404) is treated as a client error that a retry against the same target cannot fix.
  • An open-circuit target is skipped before commitment, same as every other mode. If every declared target's circuit is open, the pipeline still attempts one rather than reporting a routing-level 404 — the caller sees 503.