What if the biggest constraint on your next bot build isn’t the context window or the token bill, but a letter from a central banker to the G20?
That sounds like a stretch until you read what just happened. Andrew Bailey, Bank of England governor and chair of the Financial Stability Board, sent a letter to G20 leaders flagging frontier AI models as a threat to the financial system. His framing, as reported, is the part that stuck with me: AI could alter the speed, scale and economics of cyber risk. Not “AI is scary.” Not “AI will take jobs.” Speed, scale, and economics. Three variables that anyone who has shipped an agent to production already thinks about every day.
Why that phrasing matters to builders
Most of the AI risk talk aimed at developers is about model behavior. Does it hallucinate? Does it refuse the right things? Can you red-team the prompt? Useful questions, but they’re all local. Bailey’s concern is structural. If a capability gets cheap enough and fast enough, the math of an attack changes, and so does the math of a cascading failure.
Translate that to the stack you and I work in. An agent that can read a codebase, write a script, call an API, and retry on failure is a productivity story on Monday and an automation-of-attack-surface story on Tuesday. Same architecture. The difference is intent and access. That’s uncomfortable, because access is exactly what we spend our time expanding. Give the bot the database. Give it the ticketing system. Give it a shell. Every tutorial nudges in that direction, including plenty of mine.
Concentration risk is an architecture problem
The financial stability angle also touches something builders rarely think of as a risk at all: how few places our intelligence actually comes from. Most production bots today route to a small number of frontier model providers. If you sketch the dependency graph of the AI apps you use, it collapses into a handful of nodes fast.
Regulators look at that and see the same shape they see in clearing houses and cloud providers, a single point whose bad day becomes everyone’s bad day. Developers look at it and see convenience. Both readings are correct, which is why this warning is likely to turn into actual requirements rather than a passing news cycle.
What I’d change in a build today
None of this requires a compliance department. It requires the kind of engineering discipline we’d apply to any dependency we didn’t fully control:
- Treat the model as an untrusted component. Validate its outputs the way you’d validate a form submission from the open internet. Structured output schemas, allowlists for tool calls, hard limits on what a single agent turn can touch.
- Scope tool access per task, not per agent. A summarizer does not need write permissions. An agent with a shell should have a short-lived, narrow one. Blast radius is a design decision, not an accident.
- Design a fallback path. A second provider, a smaller local model, or a degraded rules-based mode. If your product becomes a blank screen when one API returns 503, you’ve built a single point of failure and called it a feature.
- Log the reasoning trail. Which model version, which prompt, which tools were invoked, what came back. When something goes wrong at speed, the only thing that helps is a record you can replay.
- Rate-limit your own agents. Loops are cheap to write and expensive to run. The same guardrail that protects your budget also caps how much damage a compromised agent can do per minute.
The uncomfortable middle ground
There’s a reflex in developer circles to read warnings like this as regulatory overreach dressed up as concern, an excuse to slow down open models and lock capability behind licensing. That reflex isn’t baseless, and the reporting around Bailey’s letter has already drawn plenty of eye-rolling from the crypto and open-source crowd.
But the specific claim on the table is narrow and, honestly, hard to argue with. Cheaper and faster automation changes attack economics. We know this because we’re the ones making automation cheaper and faster. Pretending otherwise means arguing our tools don’t work.
The useful response isn’t to defend AI as a category or to concede the point and wait for rules. It’s to build systems where the answer to “what happens if this goes wrong quickly” is already written down. Financial regulators are asking that question about the system as a whole. We should be asking it about every agent we deploy.
The bots we ship this year will be far more capable than the ones we shipped last year. Capability without containment is just exposure with better marketing. That’s not a reason to slow down. It’s a reason to spend a bit more of your build time on the boring parts, the permissions, the fallbacks, the logs, before someone else writes those requirements for you.
🕒 Published: