Observability

The gateway ships with four observability layers: Prometheus metrics, OpenTelemetry tracing, structured log output, and a deep health endpoint.

Prometheus metrics

Metrics are exposed at GET /metrics in the standard Prometheus text format. Scrape this endpoint with your Prometheus server. All metric names use the gateway_ prefix.

Available metrics

Metric	Type	Labels	Description
gateway_requests_total	Counter	provider, model, status	Total requests processed (status: success|error|rejected)
gateway_request_duration_seconds	Histogram	provider, model	End-to-end request latency
gateway_tokens_input_total	Counter	provider, model	Prompt tokens sent to providers
gateway_tokens_output_total	Counter	provider, model	Completion tokens received from providers
gateway_provider_errors_total	Counter	provider, error_type	Provider errors by type (provider_error|circuit_open|timeout)
gateway_circuit_breaker_state	Gauge	provider	Circuit breaker state (0=closed, 1=open, 2=half-open)
gateway_rate_limit_rejections_total	Counter	key_type	Requests rejected by rate limiting (key_type: ip|api_key|plugin)
gateway_request_cost_usd_total	Counter	provider, model	Estimated cumulative request cost in USD (public pricing tables)
gateway_server_connections_current	Gauge	state	Current inbound HTTP connections by state
gateway_server_connection_transitions_total	Counter	state	Inbound HTTP connection state transitions
gateway_hook_events_dropped_total	Counter	subject	Hook dispatches dropped because the worker queue was full
gateway_catalog_loads_total	Counter	source, result	Model-catalog load attempts (source: remote|fallback; result: success|error)

Example Prometheus scrape config

scrape_configs:
  - job_name: ferrogw
    static_configs:
      - targets: ["localhost:8080"]
    metrics_path: /metrics
    scrape_interval: 15s

Useful PromQL queries

# Request rate by provider
rate(gateway_requests_total[5m])

# P99 request latency
histogram_quantile(0.99, rate(gateway_request_duration_seconds_bucket[5m]))

# Error rate percentage (failed requests over all requests)
sum(rate(gateway_requests_total{status="error"}[5m]))
  / sum(rate(gateway_requests_total[5m])) * 100

# Token throughput per minute (input + output)
(rate(gateway_tokens_input_total[1m]) + rate(gateway_tokens_output_total[1m])) * 60

# Estimated spend rate (USD/hour) by model
rate(gateway_request_cost_usd_total[5m]) * 3600

# Open circuit breakers
gateway_circuit_breaker_state == 1

OpenTelemetry tracing

Added in v1.1.0. The gateway emits OpenTelemetry traces over OTLP. Tracing is opt-in — until you configure an OTLP endpoint or an exporter, the gateway uses a zero-allocation no-op tracer with no overhead on the hot path.

Each request opens a gateway.request root span stamped with gen_ai.* and ferro.* attributes (model, token counts, estimated cost, routing decision). Plugin executions and MCP tool calls emit child spans, and outbound provider calls are instrumented with otelhttp so the W3C traceparent is propagated upstream.

Unified trace ID

The OTel trace_id, the structured-log trace_id, and the X-Request-ID response header are all the same value. Copy a trace ID from a log line and look it up directly in your tracing backend.

Configuration

Configure tracing under the observability block in your gateway config:

observability:
  tracing:
    enabled: true
    endpoint: ""             # host:port; blank falls back to OTEL_EXPORTER_OTLP_ENDPOINT
    protocol: grpc           # grpc | http/protobuf  (https:// endpoint ⇒ TLS, else insecure)
    service_name: ferrogw
    sample_ratio: 1.0        # head sampler, 0.0–1.0
    privacy_level: metadata  # none | metadata (default) | full
    shutdown_grace: 10s      # max drain time for in-flight exports on shutdown
  exporters:                 # plugin exporters receiving completed/failed events
    - name: langsmith
      enabled: false
      config: {}

Standard OpenTelemetry environment variables take precedence over the config file:

Variable	Purpose
OTEL_EXPORTER_OTLP_ENDPOINT	OTLP collector endpoint; enables tracing when set
OTEL_EXPORTER_OTLP_HEADERS	Headers sent to the collector (e.g. auth tokens)
OTEL_TRACES_SAMPLER / OTEL_TRACES_SAMPLER_ARG	Head-sampler overrides

When both a config value and the matching OTEL_* variable are present, the environment variable wins.

Export headers

To attach static metadata or backend auth tokens to every OTLP export, set the in-config observability.tracing.headers map. Each value supports ${ENV_VAR} (and $ENV_VAR) interpolation that is resolved at exporter-build time — so secrets live in the environment, never literally in the in-memory config. A header whose value resolves to empty (e.g. it references an unset variable) is dropped and a warning is logged.

observability:
  tracing:
    enabled: true
    endpoint: api.honeycomb.io:443
    protocol: grpc
    headers:
      x-honeycomb-team: ${HONEYCOMB_API_KEY}   # resolved from the environment
      x-ferro-env: production                  # literal value, passed through

The standard OTEL_EXPORTER_OTLP_HEADERS environment variable is also honored and takes precedence.

Span attributes

Each span is stamped with two attribute groups. Group A follows the OpenTelemetry GenAI semantic conventions; Group B is the Ferro ferro.* extension namespace. Every build advertises its attribute schema version via ferro.schema.version (currently 1.0.0-draft), so exporters can branch on schema migrations.

Group A — gen_ai.* (OpenTelemetry GenAI conventions)

Attribute	Meaning
gen_ai.system	Provider system (e.g. openai, anthropic)
gen_ai.operation.name	Operation (e.g. chat)
gen_ai.request.model	Requested model ID
gen_ai.response.model	Model that actually served the response
gen_ai.request.max_tokens	Requested max output tokens
gen_ai.request.temperature	Sampling temperature
gen_ai.request.top_p	Nucleus-sampling top-p
gen_ai.request.is_stream	Whether streaming was requested
gen_ai.usage.input_tokens	Prompt tokens
gen_ai.usage.output_tokens	Completion tokens
gen_ai.usage.reasoning_tokens	Reasoning tokens (reasoning models)
gen_ai.response.finish_reasons	Finish reasons

Group B — ferro.* (Ferro extension)

Attribute	Meaning
ferro.schema.version	Attribute schema version (1.0.0-draft)
ferro.gateway.trace_id	Unified trace ID (same value as the log trace_id / X-Request-ID)
ferro.gateway.version	Gateway build version
ferro.routing.strategy	Routing strategy used (fallback, loadbalance, …)
ferro.routing.target_key	Selected target / virtual key
ferro.routing.attempt	Attempt number within the strategy
ferro.routing.ab_variant_label	A/B variant label, when applicable
ferro.cost.usd	Estimated total request cost in USD
ferro.cost.input_usd / ferro.cost.output_usd	Estimated input / output cost
ferro.cost.cache_read_usd / ferro.cost.cache_write_usd	Estimated prompt-cache read / write cost
ferro.cost.reasoning_usd	Estimated reasoning-token cost
ferro.cost.model_found	Whether the model was found in the catalog for pricing
ferro.cache.hit	Whether a response-cache hit served the request
ferro.cache.kind	Cache kind
ferro.plugin.name / ferro.plugin.kind	Plugin identity (child spans)
ferro.plugin.stage	Plugin stage (before_request, after_request, on_error)
ferro.plugin.outcome / ferro.plugin.reason	Plugin outcome and reason
ferro.mcp.server / ferro.mcp.tool	MCP server and tool name (child spans)
ferro.mcp.depth / ferro.mcp.latency_ms	MCP call depth and latency
ferro.stream.time_to_first_token_ms	Streaming time-to-first-token
ferro.stream.time_to_last_token_ms	Streaming time-to-last-token
ferro.circuit_breaker.state / ferro.circuit_breaker.opened	Circuit-breaker state and whether it opened during the request
ferro.request.api_key_id / ferro.request.tenant_id	Request API key ID and tenant ID
ferro.error.upstream_status	Upstream HTTP status on failure
ferro.error.retry_count	Number of retries performed

Privacy levels

privacy_level controls how much request content reaches your tracing backend:

Level	Behavior
none	No prompt/response content; error messages omitted
metadata	Default. Metadata only; error messages redacted (email, JWT, AWS keys)
full	Raw error text included — use only in trusted environments

Exporters

Beyond raw OTLP, the gateway exposes an exporter event seam: registered exporters receive gateway.request.completed / gateway.request.failed events for bridging to backends like LangSmith, Langfuse, or Datadog. No exporters ship in the core gateway — they live in the separate ai-gateway-plugins repo. List enabled exporters under observability.exporters; unknown or failed exporters are logged and skipped (non-fatal).

Structured JSON logs

The gateway writes structured JSON to stdout. Each log line includes:

Field	Description
time	ISO 8601 timestamp
level	debug, info, warn, error
trace_id	Per-request UUID for log correlation
msg	Log message
provider	Provider name (on request/response lines)
model	Model ID
latency_ms	Provider round-trip latency in milliseconds
status	HTTP status code
tokens_prompt	Prompt token count
tokens_completion	Completion token count

Example log line:

{
  "time": "2026-03-11T10:23:45Z",
  "level": "info",
  "trace_id": "a3f9b1c2-d4e5-4678-8901-abcdef012345",
  "msg": "request complete",
  "provider": "anthropic",
  "model": "claude-3-5-sonnet-20241022",
  "latency_ms": 412,
  "status": 200,
  "tokens_prompt": 312,
  "tokens_completion": 87
}

Log level

Set the log level with LOG_LEVEL (default: info). Use LOG_FORMAT=text for human-readable output during development.

export LOG_LEVEL=debug
export LOG_FORMAT=text

Health endpoint

GET /health returns a deep health check with per-provider availability and latency:

{
  "status": "ok",
  "providers": {
    "openai": { "healthy": true, "latency_ms": 245 },
    "anthropic": { "healthy": true, "latency_ms": 312 },
    "groq": { "healthy": false, "error": "connection refused" }
  }
}

Returns 200 OK if at least one provider is healthy, 503 Service Unavailable if all providers are down.

For monitoring in production, see Monitoring.