AI in healthcare was pitched as a cost-cutting story. The Blue Cross Blue Shield Association’s analysis says hospital AI tools used during claims submission added $942 million in healthcare spending over two years. Both of those things are being said out loud, at the same time, by serious people.
I build bots for a living. That number doesn’t surprise me at all, and if you’ve ever shipped an automation that worked a little too well, it probably doesn’t surprise you either.
What the insurers are actually alleging
The claim, as reported by TechCrunch and examined by The New York Times, is not that hospital AI is hallucinating nonsense or committing fraud. It’s narrower and more interesting: AI is inflating billed complexity without a matching change in the care delivered. Same patient, same treatment, richer paperwork. The New York Times frames it as an escalation of a longstanding feud between hospitals and insurers, with both sides now bringing software to a fight they used to have with humans.
That framing matters. This isn’t a story about a model behaving badly. It’s a story about two models behaving exactly as instructed, pointed at each other.
The objective function is doing its job
Here’s what I keep thinking about as a builder. If a hospital deploys a system whose measurable goal is “maximize appropriate reimbursement,” the system will find every defensible path to a higher number. Not because it’s scheming, but because that’s the gradient you handed it. Coding guidelines are long, ambiguous, and full of legitimate judgment calls. An automated system that reads charts and suggests the most supportable code is, technically, doing the job.
Insurers, meanwhile, are running their own automation on the review side, looking for patterns that justify a denial or a downgrade. So you get an arms race where each side’s software gets better at reading the other’s output.
Anyone who has built adversarial systems knows how this goes. Both sides burn compute. Both sides add latency. Neither side’s core value changes. The cost of the fight is real and, in this case, the figure attached to it is $942 million over two years.
Three lessons I’d take into any bot project
You don’t need to work in healthcare for this to be relevant. The structural pattern shows up anywhere automation touches a negotiation.
- Optimizing a proxy metric optimizes the proxy, not the goal. “Billed complexity” was always a stand-in for “how sick was this patient and how much work did we do.” Automate against the proxy and the proxy drifts from reality faster than any human process would have allowed.
- Speed changes the nature of a system, not just its throughput. A coder reviewing 30 charts a day produces a slow, self-limiting signal. A system reviewing 30,000 produces a structural shift. Same logic, different outcome, because volume is part of the design.
- Adversarial automation rarely settles. If your counterparty can also deploy software, expect your gains to be met and matched. Budget for the escalation, because the other side is budgeting for it too.
Where I think the engineering conversation goes wrong
This is my read, not something in the reporting: a lot of us in the bot-building community treat “the model was accurate” as the end of the ethical review. Did it extract the right entities from the chart? Did it pick a code a human auditor would defend? Ship it.
But accuracy at the record level tells you nothing about the aggregate effect. A system can make defensible calls a million times and still move hundreds of millions of dollars in a direction nobody signed off on. That’s a systems question, and it needs systems-level instrumentation: dashboards that track distribution shift in your outputs over time, alerts when your model’s aggregate behavior diverges from the historical baseline, and a human who owns the question “is the overall pattern here still what we intended?”
Most of the pipelines I’ve reviewed have excellent per-request logging and nothing resembling that. We monitor for errors. We don’t monitor for drift in the thing we’re collectively producing.
The part that should bother builders
Hospitals will say their AI is finally capturing revenue they were always owed, and that decades of underpayment made these tools necessary. Insurers will say the numbers prove otherwise. I’m not in a position to adjudicate that, and the reporting doesn’t settle it either.
What I can say is that the technical community shipped both sides of this. The same skills, the same model families, the same retrieval patterns, sold to opposing parties, each promised an edge. That’s a fine business. It’s just not the efficiency story the industry has been telling.
If you’re building in a space where two parties have opposing interests and both can buy your product, you’re not automating a workflow. You’re supplying an arms race. Worth knowing which one you signed up for.
đź•’ Published: