Streaming

Set stream to true to receive the answer as server-sent events while it's written, in OpenAI or Anthropic format, with token usage at the end.

Set "stream": true and the answer arrives as server-sent events while the model writes it. SDKs read the stream for you.

stream = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Count to five."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

What the stream contains

On /v1/chat/completions, each event is a chat.completion.chunk, and the stream ends with data: [DONE]. The last chunk before it carries usage and an empty choices array. The gateway always asks the provider for usage, so this chunk arrives even if you didn't set stream_options.include_usage. Check that choices isn't empty before reading from it, as the Python example does.

On /v1/messages, the stream is Anthropic's event sequence; see Anthropic Messages.

When a provider fails

Failover happens before the first byte. If the first provider errors or times out, the gateway tries the next one, and you see a normal stream from whichever answered. Once a stream has started, it can't move to another provider. If the provider breaks off mid-stream, the stream simply ends without a final [DONE]. Check for it, and retry if it's missing.

If your client disconnects, the gateway still reads the provider's answer to the end, and the request is billed for the tokens the provider produced.

Timeouts

Each attempt has two minutes for the provider to finish the whole answer, stream or not. Very long answers from slow models can reach that limit; lower max_tokens or stream to a model with faster output.

Checked against the gateway on 31 October 2026. Something wrong or missing? Tell us.