Why LLM APIs fail, and failover

LLM APIs fail more often than most web APIs: providers time out, return 429 rate limits and 5xx overload errors, especially at busy times. Failover sends the same request to another provider or model when that happens, while errors caused by the request itself are returned straight away.

Start free Read the docs

Asked forauto (balanced)

1. anthropic / claude-haiku-5-5503 · 240 ms
2. openai / gpt-6-luna200 · 1.4 s

Your code receives one normal answer. x-digitspace-provider: openai

Errors, and what to do with them

Error	Usual cause	Retry elsewhere?
Timeout, connection reset	Provider overloaded or network trouble	Yes
429	Rate limit or quota reached on that provider	Yes
500, 502, 503, 529	Provider error or overload	Yes
400, 401, 403, 404, 422	The request, key or model name is wrong	No: fix the request

An engraving of a wall-mounted telephone transmitter from a 1906 telephone handbook. — From a 1906 handbook on telephone exchanges.
The practical telephone handbook and guide to the telephonic exchange, 1906. Internet Archive Book Images, no known copyright restrictions, via Wikimedia Commons.

Doing failover well

Track health: Count recent successes and failures per provider and model, and try healthy ones first.
Cap attempts: Two or three tries is enough; more only makes a slow failure slower.
Fail fast on bad requests: Another provider won't fix a malformed request.
Record every attempt: So a provider that keeps failing is visible before it becomes an incident.

Questions

A telephone exchange room around 1900, with a long wooden switchboard, operator stools and a desk. — General Telephone Company of the Northwest, exchange room, about 1900, public domain, cropped, via Wikimedia Commons.

Isn't a simple retry loop enough?

Retrying the same provider during an outage usually fails again. Moving to a different provider or model is what keeps the request alive.

Does failover change the answer?

Moving to another provider of the same model gives the same model's answer. Moving to another model, as "auto" may, can change wording but stays within your quality level.

Live in a minute.

Create a key, change the base URL, send a request. No card needed with your own provider keys.

Start free