Remember when the worst thing your automation could do was spam a Slack channel at 3 a.m.? You’d wake up to forty duplicate notifications, mutter something unkind about your own cron job, add a dedupe check, and get on with your day. The failure was embarrassing. It was not expensive.
That era is over. Reuters, Ars Technica, Computerworld, and Newser all reported on the same story this week: Mark Zuckerberg’s plan to replace Meta staff with AI agents did not go the way the slide deck promised. Ars Technica’s framing is the part that should interest anyone who builds bots for a living. The agents made what were described as “large-scale, disruptive actions.” Meta is reportedly backtracking.
Note what that phrase does and does not say. It doesn’t say the models were dumb. It doesn’t say they hallucinated a fact or wrote clumsy code. It says they acted, at scale, and the acting caused damage. Those are two very different failure modes, and our tooling is wildly better at catching the first one.
Capability and permission are separate problems
Every agent I’ve shipped has taught me the same lesson in a slightly more expensive way. The hard part was never getting the model to do the thing. The hard part was bounding what happens when it does the thing at the wrong time, to the wrong resource, a thousand times in a row.
We evaluate agents almost entirely on capability. Can it resolve the ticket? Can it pass the test suite? Can it refactor the module? Those benchmarks answer “is this smart enough to be useful.” They say nothing about “what is the maximum damage this can do between now and when a human notices.”
That second number is blast radius, and almost nobody puts it in a readme. It’s not in your eval use. It’s not in the demo video. It shows up for the first time in an incident review, which is exactly the wrong place to discover a design parameter.
Why scale makes small errors structural
A human employee who misunderstands a task makes one mistake and usually feels weird about it halfway through. That hesitation is a rate limiter we never had to build. An agent fleet has no such instinct. Give the same flawed reasoning to a hundred parallel workers with write access and the mistake stops being a mistake and becomes a migration.
This is the part of the Meta story that generalizes past Meta. You don’t need a frontier lab budget to reproduce it. Any of us can wire an agent to a production API, hand it a loop, and walk away. The pattern that broke at Meta’s scale breaks at yours too. It’s just quieter.
What I actually build in now
My checklist has gotten less exciting and much more boring over the last year, which I take as a good sign:
- Separate the read agent from the write agent. Analysis and action should not share a credential. Most agent work is reading anyway, and reads are cheap to be wrong about.
- Cap actions per run, not just tokens per run. Token budgets protect your bill. Action budgets protect your data. An agent allowed twelve writes per session cannot cause a large-scale disruptive anything.
- Make every mutation reversible or reviewable, never neither. Branches instead of commits to main. Drafts instead of sends. Soft deletes instead of deletes. If reversal isn’t possible, a human approves.
- Log intent, not just output. When something goes sideways, “what did it think it was doing” is the only question that matters, and you cannot answer it retroactively.
- Test the fleet, not the agent. One agent behaving correctly tells you very little about fifty agents sharing state. Run the swarm in staging and watch what they do to each other.
The org chart is not an API spec
There’s a deeper assumption worth questioning here, and it’s the one I think caused the most trouble. “Replace workers with agents” treats a job as a well-defined function with typed inputs and outputs. Real jobs are mostly negotiation, context, and knowing which rule to quietly ignore this week. That tacit layer is what keeps a company from executing its own stated processes literally enough to hurt itself.
Agents execute stated processes literally. That’s the entire value proposition, and also the entire risk. When you swap out the humans, you’re not just removing labor, you’re removing the judgment that was silently correcting your specs the whole time.
A better goal than replacement
The useful version of this technology, in my experience, is narrow, permissioned, and observable. An agent that owns one workflow end to end, with a hard action cap and a reversible output, is genuinely great. It’s also unglamorous, which is probably why it doesn’t make headlines.
Meta ran the ambitious experiment in public and the reporting suggests it cost them. The rest of us get the findings for free. My read: build the smallest agent that solves a real problem, then spend most of your engineering effort on the walls around it. Capability is getting handed to us for free every few months. Containment is still ours to write.
🕒 Published: