Handle failures, retry exceptions
2026-05-30 · #systems #craft
Any system that runs at scale has to be as good at failing as it is at succeeding. Past a certain volume, failure stops being an emergency and becomes a category of ordinary output: it gets its own path, its own metrics, its own handling, the same way success does. If you haven’t designed that path, the system has designed it for you, and you will not like what it chose.
The core of the system I build runs off a priority queue. Events arrive, workers pull the most urgent first, process them, acknowledge. I have spent a lot of time on its failure paths, and the worst recurring bug I meet there isn’t a failure we handle badly. It’s a failure we file under the wrong heading.
The two ways to fail
Pull the failures apart and there are two kinds that want opposite things.
Some failures are the environment misbehaving. A network blip, a spot instance reclaimed mid-task, a node that vanished in a deploy. The event was fine; the world flinched. The correct response is to try again, usually somewhere else, usually a moment later, and it goes through. Call these exceptional failures. Retry is the whole point of them.
Other failures are the event being wrong. The shape is off, the data is bad, a field violates an invariant nothing downstream will tolerate. Trying again changes nothing, because nothing about the event will be different next time. The correct response is a verdict: reject it, set it aside, hand it to a human. Call these standard failures. Retry is precisely the wrong tool.
Almost every queue-based system sorts its failures into these two buckets, explicitly or not, and almost all of the time it sorts them correctly. The entire design leans on that sorting being right.
When a standard failure dresses as exceptional
The trouble is the event that belongs in the second bucket and lands in the first.
It’s a genuinely broken event, a standard failure by rights, but it doesn’t fail like one. It never comes back as a clean rejection you can route aside. It exhausts memory, trips a pathological code path, wedges a parser, and takes the worker down with it. From the queue’s side, a worker that died holding an event is indistinguishable from a worker that got reclaimed mid-task. So the event goes down the exceptional path. It gets retried. And retry is the one response that cannot possibly help, because the thing that is wrong travels with the event.
Retry is a bet that the next attempt lands in a different world. A poison event is the part of the world that travels with the message.
People call this a poison pill, and the clean version - an event that deterministically kills whatever touches it - is the easy case. The one that actually ruins your week is grey: it doesn’t fail everywhere, and the damage it does is set less by the event than by where it lands. Drop it into a part of the pipeline that is already a bottleneck - a stage gated by a handful of licences, or work so expensive that a worker can only run a few at a time - and a small number of poison events can choke thousands of good ones, even while events keep processing between the failures. The thing that killed the worker is partly the event and partly the place, which is exactly the ambiguity that keeps it filed under exceptional, where it gets retried, and retried, and retried.
The pill that walks
I have watched this happen. It is far worse than one misclassified event, because the misclassification compounds.
The worker dies before it can acknowledge, so the event survives and the worker doesn’t. The orchestrator notices a dead worker and does its job: a fresh one comes up. The queue notices an unacknowledged event and does its job: it hands that event to the fresh worker. The two most reliable, most boring pieces of the system now form a machine whose only product is dead servers.
event ─> worker A x crash
└─> worker B x crash
└─> worker C x crash
restarts: A' B' C' ...
queue: _ . : | || growing
Priority neither saves you nor damns you. The high-priority poison event comes back to the front and takes down a worker, but the visibility timeouts still tick and the rest of the fleet keeps processing other events perfectly well between the crashes. That continued success is the trap. If the poison jammed everything, you’d find it in minutes; instead the work flows on around each gap, and the only trace is a server that quietly dropped out and came back.
One pill is a nuisance - a server flaps, restarts, you might not notice. The danger is that it doesn’t stop, and rarely on just two hosts: it walks the fleet, and how fast it hurts depends entirely on where it landed. In a constrained stage, losing even a couple of workers is a large slice of capacity, so the backlog builds fast and the incident is obvious. In a roomy stage the spare capacity absorbs the losses, the backlog creeps, and the thing can run for hours before anyone names it. The less it hurts per minute, the longer it hides. Recovery time only sharpens this: if a dead worker takes two minutes to come back, that’s two minutes of a scarce slot doing nothing, and a poison event that trades a few seconds of crash for two minutes of restart is winning every time it fires.
And the backlog is the last thing to look alarming. Queue depth lags; for a while it climbs slowly while the dashboards stay green - servers up and busy, restarting promptly when they fall over, health checks passing in the gaps. The self-healing is working perfectly. It’s just that the thing it’s healing is the delivery mechanism for the outage, handing the pill a fresh victim on every restart. The system isn’t recovering. It’s reloading.
Why nobody finds the event
Here is the part I have watched cost people hours: the root cause is one specific event, and they cannot find it.
It’s never where you’d look. The offending event is almost never sitting at the head of the queue - by the time a worker dies it has already been checked out, in flight, hidden inside a visibility timeout, waiting to reappear on whichever host is unlucky next. Around it the fleet is busy succeeding, thousands of events a second going through clean, so the crashes scatter across hosts with no obvious thread joining them.
And no single signal convicts. Message age looks promising until you remember that a healthy event retrying against a slow downstream is old too. Crash rate looks promising until you remember that services go flaky for a hundred reasons that have nothing to do with any one event. What you actually need is the conjunction: an aging, redelivering event whose presence on a host keeps coinciding with that host dying. Neither half is enough alone.
The witness, meanwhile, is dead. The worker that could name the event is the one that crashed, and it was holding dozens of events at once, so its death implicates all of them and names none. The only thing that works is recorded from outside and recorded before the fact: something that durably notes “this worker is now processing this event” the moment work starts, watches which workers die, and intersects the in-flight sets to find the event common to every crash. More than one pill at once muddies the intersection. It is real engineering, it earns nothing on an ordinary day, and almost nobody builds it until the first night they’ve done the intersection by hand.
What actually helps
None of it is clever. It’s the boring disciplines, applied to the failure path with the seriousness you’d give the success path.
- Classify on how it failed, not how often. The retry count tells you almost nothing - a good event against a flaky downstream will retry past any threshold you’d care to pick. The manner of the failure tells you something. A clean, caught rejection is a standard failure announcing itself honestly; a worker death is a standard failure that’s been misfiled, or an exceptional one, and at the moment of the crash you cannot yet tell which. The count is a red herring. The failure mode is the signal.
- Quarantine, and respect the backstop you already have. A maximum-receive count feeding a dead-letter queue is the workhorse, and it’s easy to look down on. Don’t: it is the one mechanism that fires automatically when the worker dies before it can acknowledge, which is the exact case that hurts. Set it well above the honest retry count of your flakiest downstream, so you don’t quarantine good events for the crime of being slow. Pulling an event the moment it’s implicated in repeated deaths is better, but it needs the identification machinery above - which is why most teams never build it and lean on max-receive instead. Lean on it deliberately.
- Shrink the blast radius, then assume it fails anyway. Bound what a single event can cost: memory ceilings on the unit of work, defensive parsing, timeouts, catching the panic instead of riding it down. Every failure you convert from a worker death into a clean rejection is one that goes down the right path and names itself in the log. But you will not get to zero - the out-of-memory killer doesn’t read your try/catch, and some events take the worker down however carefully you wrapped them. The goal was never a worker that can’t die. It’s a worker that dies rarely and cheaply, in a system that expects it to.
A system at scale fails constantly, and most of that failing is fine - it’s the world flinching, caught by a retry path built for exactly that. The same path is the perfect hiding place for the failure that doesn’t belong on it. A standard failure in exceptional clothing gets handled with care, retried with patience, and walked through your fleet, all by machinery doing precisely what you asked of it.
So the discipline is to keep the two apart. The retry path is for exceptions, and exceptions should be exceptional: rare, environmental, gone on the next attempt. Everything else is a failure to be handled - given a verdict, set aside, named in a log - not retried in the hope the world has changed. The rest is unglamorous and worth it: making it hard for a standard failure to pass as an exceptional one for long, and keeping the machinery to name the offending event the moment it tries.
Failure is always an option - retry when the world failed, and hunt the event down when it didn’t.