Streaming (SSE) contract

The gateway streams chat completions using Server-Sent Events (SSE). The wire format is OpenAI-compatible: each event is a data: line carrying one JSON chunk, and a normal stream ends with a literal data: [DONE] sentinel.

Enabling streaming

Set stream: true in the body of a POST /v1/chat/completions request. The gateway responds with:

• Content-Type: text/event-stream
• Cache-Control: no-cache
• Connection: keep-alive

Each chunk is emitted as a line of the form data: <json> followed by a blank line. Use a client that does not buffer the response (for curl, pass -N).

curl -N https://gateway.example.com/v1/chat/completions \
  -H "Authorization: Bearer $FERRO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "stream": true,
    "messages": [
      { "role": "user", "content": "Write a haiku about iron." }
    ]
  }'

Normal event stream

Token chunks use the OpenAI chat.completion.chunk object shape. The gateway fills in object (chat.completion.chunk) and created if the upstream provider omits them. After the final content chunk, the gateway writes the terminal data: [DONE] marker and closes the connection.

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1718600000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1718600000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":"Cold"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1718600000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":" iron"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1718600000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1718600000,"model":"gpt-4o-mini","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":4,"total_tokens":16}}

data: [DONE]

The blank line after each data: event is part of the SSE framing — do not strip it before your parser sees the event boundary.

Terminal usage frame

The gateway asks the upstream provider for a usage-carrying terminal chunk on every OpenAI-compatible stream — it sets stream_options.include_usage: true unless the client's own request already configured stream_options. That final chunk carries choices: [] (never null; a usage-only frame still serializes an empty array so a client that unconditionally indexes choices[0] does not break) and a populated usage object.

The gateway needs real token counts for metering, cost accounting, and the budget plugin regardless of what the client asked for, so it always requests the frame from the provider. What reaches the client is separate: if the original request set stream_options.include_usage: false, the gateway still collects the real usage internally but strips the usage field from the copy of the chunk it forwards, so an explicit client opt-out is honored without losing gateway-side accounting.

Mid-stream errors

The response headers — including the X-Gateway-Provider, X-Gateway-Target, X-Gateway-Model and X-Gateway-Attempts attribution headers — are sent before the first chunk, because the routing walk has finished choosing by then. If the upstream provider fails after the stream has started, the gateway cannot reuse a normal HTTP status code (headers are already sent). Instead it emits a single error event as a data: line and then closes the stream. No [DONE] sentinel follows an error event. The failure is also recorded on the request's span, counted in Prometheus metrics, and written to the request log via the on_error plugin stage — a mid-stream failure is no longer a silent truncation on any of those surfaces.

The error chunk has this exact JSON shape:

{
  "error": {
    "message": "<upstream error text>",
    "type": "stream_error",
    "code": "stream_error"
  }
}

On the wire:

data: {"error":{"message":"upstream provider returned 502","type":"stream_error","code":"stream_error"}}

Idle and write timeouts

The gateway guards a stalled stream with two independent deadlines:

Guard	Value	Behavior
Idle timeout	2 minutes	Resets on every chunk. If no new chunk arrives within the window, the gateway emits a stream_timeout error event and closes the stream.
Per-write deadline	15 seconds	Applied to each individual flush to the client. If a single write cannot complete in time, the stream is terminated.

When the idle timer fires, the gateway emits this exact event before closing:

{
  "error": {
    "message": "stream timed out waiting for next chunk",
    "type": "timeout_error",
    "code": "stream_timeout"
  }
}

Note that an idle timeout uses type: "timeout_error" with code: "stream_timeout", whereas an upstream provider failure uses type: "stream_error" with code: "stream_error". Both are delivered as error events; neither is followed by [DONE].

Distinguishing token chunks from error events

Every SSE event is a data: line. Clients should classify each event as follows:

1. If the payload is the literal [DONE], the stream completed normally — stop reading.
2. Otherwise parse the payload as JSON, then:
   • A top-level error object means a failure event. Read error.code (stream_error or stream_timeout) and error.message, surface it, and stop.
   • A payload with object: "chat.completion.chunk" and a non-empty choices array is a normal token chunk. Append choices[].delta.content.
   • A payload with object: "chat.completion.chunk" and an empty choices array is the terminal usage frame — read usage and otherwise ignore it; it carries no content to append.

The presence of a top-level error key is the reliable discriminator: normal and usage chunks never carry one, and error events never carry choices.

MCP-augmented streaming

Whether MCP tool servers change the streaming behavior of a request depends on whether the request itself carries a tools array:

• The request sends its own tools. MCP does not participate at all — the gateway never advertises its MCP tools alongside a caller-supplied set, because a model could then return a mix of gateway-owned and caller-owned tool calls that neither side alone can execute. Such a request passes straight through the ordinary streaming path described above, untouched, whatever MCP servers are configured.
• The request sends no tools, and the gateway has MCP tools ready to advertise. The gateway injects its MCP tool definitions and the request is diverted: the full agentic tool-call loop (call the tool, feed the result back, re-ask the model) runs to completion before anything is sent to the client. Every intermediate turn is forced to non-streaming so its response can be inspected for tool_calls. Once the loop produces a final answer with no more pending tool calls, that answer is wrapped into a single SSE chunk followed by data: [DONE] — the client still receives stream: true semantics (SSE framing, one [DONE] terminator), but gets one content chunk rather than an incremental token stream.

Gating is on tools actually being advertised (registered and past their MCP handshake), not merely on mcp_servers being configured — a server that failed to initialize does not silently collapse every tool-less stream on the gateway into a buffered response.

Every turn of the loop runs the request's guardrail, rate-limit, and budget plugins — not only the turn the caller made. A tool result returned by an external MCP server is content the caller never wrote, so it is inspected exactly as the original prompt was before being sent to the model again. This means a loop can be rejected mid-way through, most commonly by the budget plugin once accumulated per-turn spend crosses its cap.

A mid-loop rejection is not an SSE error event: because the loop runs entirely before any bytes reach the client, a rejection happens before the gateway ever opens the SSE response, so the caller gets an ordinary (non-streaming) JSON error — for a budget rejection, 402 insufficient_quota — instead of a stream_error frame.

Notes

• The gateway proxies 30 LLM providers behind this single OpenAI-compatible streaming contract, so client code targets one format regardless of the upstream provider.
• Always send and parse SSE incrementally. Treat the connection close after [DONE] (or after an error event) as the authoritative end of the stream.