Skip to main content

Cost-optimized routing strategy

Cost-optimized routes each request to the cheapest model-compatible target, ranked by estimated input plus output cost from the built-in model catalog. It optimizes for spend — not failover order (fallback), distribution (loadbalance), or latency (least-latency). Set strategy.mode: cost-optimized to use it.

In plain words

Each request goes to the cheapest provider that serves the model, using the built-in price list, a rough guess at how many tokens the prompt is, and the completion budget the request asked for. If the cheapest one is unreachable, times out or is overloaded, the next-cheapest takes over; two providers that cost the same share the traffic by weight. A provider with no known price is used last (the default), skipped, or treated as free — your choice.

What happens to a request​

Two providers serve the same small model under one name, with catalog prices at the time of writing:

strategy:
  mode: cost-optimized

targets:
  - virtual_key: groq
    model_map:
      llama-8b: llama-3.1-8b-instant                            # $0.05 per 1M input tokens
  - virtual_key: together
    model_map:
      llama-8b: meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo     # $0.18 per 1M input tokens
Request for llama-8bThe gateway…The client gets
normalasks Groq, the cheaper oneGroq's answer, "model": "llama-8b"
Groq returns 429retries per Groq's retry, then asks TogetherTogether's answer, still "model": "llama-8b"
Groq returns 400stops400
a third target with no catalog price is addedunder the default unpriced_strategy: fallback it is tried only when no priced target is compatible—

Behaviour​

cost-optimized is a pool mode: after a failover-safe failure (a transport failure, an attempt that timed out waiting on the target, 408, 429, 5xx, an open circuit, or a full concurrency queue) the pipeline advances to the next candidate in the ranked order rather than reporting the failure back to the caller; any other 4xx is returned to the client. targets[].retry still governs how many times any one target is retried before the pipeline moves on.

On each request, CostOptimized.SelectTargets (internal/strategies/costoptimized.go) filters targets[] down to those whose provider is registered and whose SupportsModel(req.Model) returns true, then estimates the prompt at roughly 4 characters per token and the completion at the request's max_tokens / max_completion_tokens (or 256 tokens when it sets none) — a routing heuristic, not a billing figure — and prices each compatible target's upstream model — the model_map translation when the target has one, otherwise the requested model — through the model catalog at the rate for that model's mode: input plus output for chat, per token for embeddings, per image, per minute or character for audio. Candidates are ranked by ascending estimated cost — the cheapest target leads — and a run of equal-cost candidates leads with one drawn by targets[].weight (equally when no weight is set), since declaration order is not a contract. strategy.unpriced_strategy decides which cataloged-but-unpriced candidates are allowed to rank at all (see below). Candidates the catalog doesn't recognize as that model (ModelFound: false) never rank, regardless of unpriced_strategy.

Config keys​

KeyTypeDefaultDescription
strategy.modestring—Set to cost-optimized.
strategy.unpriced_strategystringfallbackfallback | skip | allow. Governs how compatible targets the catalog knows about but has no price for are treated during ranking.
targets[].weightfloat640Breaks ties between equal-cost candidates; unset or zero everywhere means an equal draw. A negative weight is rejected at load.

unpriced_strategy in detail​

ValueRankingWhen nothing ranks
fallback (default)Only priced candidates rank, cheapest first.The compatible candidates lead the returned order in declared order (not targets[0], the first compatible one) — no error.
skipOnly priced candidates rank, cheapest first.Returns an error wrapping core.ErrNoCapableProvider — the only strategy mode that can return a non-nil error from SelectTargets.
allowEvery model-compatible candidate ranks, priced or not. A missing price sorts as $0 — the cheapest possible value — so an unpriced target wins the draw over any priced one.— (there is always at least one ranked candidate if any target is compatible).

ferrogw validate accepts only an empty value (the default), fallback, skip, or allow for unpriced_strategy — anything else fails at load time, so an invalid value never reaches a live request.

Minimal working YAML​

strategy:
  mode: cost-optimized
  unpriced_strategy: fallback

targets:
  - virtual_key: deepseek
  - virtual_key: openai
  - virtual_key: anthropic

When to use​

  • Multiple providers serve the same (or equivalent) model and the catalog carries pricing for them — let cost decide instead of hand-picking a provider.
  • You want spend minimized automatically as new, cheaper models or providers are added to targets[], without a config change.
  • Combined with retry on each target, so a failed cheapest-provider attempt falls through to the next-cheapest rather than erroring.

Gotchas​

  • Estimated, not billed. The prompt is counted at ~4 characters per token and the completion at the request's own ceiling or 256 tokens, so the score is a comparison of list prices for a typical request, not a billing-accurate number. Before v1.5.2 only input price counted, so a target cheap to read and expensive to write could win a request with a large completion budget; it no longer does.
  • The same order on every surface. An embeddings, image or audio request is priced at the catalog's rate for that model's mode, so those models no longer tie at zero and fall to declared order.
  • allow makes unpriced look free. Under unpriced_strategy: allow, any compatible target the catalog has no price for is treated as $0 and wins the draw over every priced target, every time. This is useful for a self-hosted/local target you always want preferred, but a surprise for one you didn't intend to be prioritized.
  • skip narrows what /v1/models advertises. Under unpriced_strategy: skip, targets with no catalog price are excluded from ranking entirely; if a request's model has no priced target, the request errors rather than falling through. targets[].models entries are unpriced by construction (they're operator-declared, not catalog data), so they're excluded here too.
  • Default (fallback) only reaches unpriced targets as a last resort. A self-hosted or otherwise-unpriced target under the default strategy is used only when no compatible target in targets[] has any catalog price at all — not merely when it's cheaper.
  • This is a pool mode. The pipeline advances past an open-circuit target, or one that failed in a failover-safe way, to the next one in the ranked order; any other 4xx (400, 401, 403, 404, 422, …) is returned to the client. cost-optimized gets failover "for free" as a side effect of the mode family, not because of anything cost-specific.
  • Only model-compatible targets are eligible. A target whose provider isn't registered, or that doesn't serve the requested model, is filtered out before pricing is even looked up.
  • Ranking is per-request. Because it depends on the requested model and that request's estimated prompt size, the effective order can differ from one request to the next through the same config — it is not a static, once-computed target list.