Remember the first time you pushed a “tiny” change to a bot in production? Mine was a one-line tweak to a retry timeout. Felt harmless. Two hours later the queue was backed up, the webhook handler was timing out, and the logs were a wall of the same error repeated ten thousand times. Nothing was broken. Everything was broken. That’s the part nobody warns you about when you start wiring LLM calls to tool calls to databases to third-party APIs.
There’s a short treatise floating around dev circles called How Complex Systems Fail — a piece about the nature of failure, how failure gets evaluated, how it gets pinned on a proximate cause, and what that means for safety. It came out of patient safety work, not software. But if you build bots for a living, it reads like someone watched your last incident review over your shoulder.
Cascades, not crashes
Complex systems mostly don’t fail the way we imagine them failing. We picture a single dramatic break — a server dies, a key expires, the model endpoint goes down. What actually happens is a cascade. One small degradation nudges another component into a state it was never tested in, which nudges a third, and by the time your alerting fires you’re three layers away from the thing that actually started it.
Bot architectures are unusually good at producing cascades, because we chain everything:
- A retriever returns slightly worse results, so the model asks for more tool calls.
- More tool calls mean more concurrent API requests, so you hit a rate limit.
- Rate limits trigger retries, which multiply the load that caused them.
- Retries fill your queue, so latency climbs, so timeouts fire upstream.
Not one of those steps is a bug. Each is a component doing exactly what you told it to. The failure lives in the interaction, which is why postmortems that end at “we found the root cause” usually haven’t found much. Proximate cause is a comfortable story. It rarely explains why the same class of incident shows up again next quarter.
Every new feature ships a new failure path
One perspective on that treatise stuck with me: new feature development adds new failure paths to a system you thought you understood. Increased entropy, plus a team that hasn’t been trained on the new shape of things. That’s the real cost of shipping fast, and it never shows up in the estimate.
Think about what happens when you add a single tool to an agent. You haven’t added one path — you’ve added every combination of that tool with every other tool, in every order the model might choose, with every partial failure mode each one can produce. The state space grows in a way your test suite does not. And the person on call at 3am has a mental model of the system from two sprints ago.
This is why I’ve stopped treating “the bot works” as a meaningful statement. Working is a snapshot. What matters is whether the thing degrades gracefully when one leg gets kicked out.
Maintenance and production are at cross purposes
Here’s the tension the treatise names better than I could: maintenance and production pull against each other. If you shut everything down every time you have to change a light bulb, you’re never going to get anything done. So you don’t shut things down. You patch around live traffic, defer the migration, leave the deprecated SDK in place because it still works.
Then the security side of the ledger comes due. Many updates carry patches for security vulnerabilities, and digital threats move fast — updates are frequently aimed at specific new attack vectors. Skipping them isn’t neutral. It’s an accumulating position against your own uptime. Going into 2026, proactive maintenance and cybersecurity are the whole ballgame for anyone running bots that touch real user data or real money.
The uncomfortable resolution is that you can’t pick one side. You need a way to change light bulbs without killing the lights.
What I actually do about it
None of this is solved by being smarter. It’s solved by design choices that assume you’ll be wrong:
- Bulkheads over retries. Isolate each external dependency so its bad day stays its own. Circuit breakers beat exponential backoff when the thing you’re retrying is the thing you’re overloading.
- Budget your calls. Hard caps on tool calls, tokens, and concurrency per request. An agent with no ceiling will find one for you.
- Patch on a schedule, not on a scare. Dependency updates as routine work, pinned versions, and a staging path that lets you apply them without a full stop-the-world event.
- Practice the failure. Kill a dependency in staging and watch what your bot does. Bad answer beats hung request.
- Write down the current shape. Entropy plus untrained operators is the actual outage. Update the architecture doc when you ship the tool.
Researchers have been working on predicting and preparing for the failure of complex systems for years, and much of that work is openly licensed and free to read. Worth borrowing from, because bot builders are late arrivals to a problem that aviation, medicine, and power grids have been chewing on for decades. Our systems are younger. They fail the same way.
🕒 Published: