AI was supposed to cut waste out of healthcare. Blue Cross Blue Shield says hospital use of AI tools added $942 million in costs over two years. Both of those statements come from people who believe them completely, which is what makes this story interesting to anyone who builds automated systems for a living.
I spend most of my time writing bots that talk to other software. Sometimes that software talks back. And the thing I’ve learned, usually the hard way, is that an automated system optimizing hard against another automated system does not produce efficiency. It produces escalation. Healthcare billing just became the highest-stakes public demo of that principle we’ve ever had.
Two agents, opposing objective functions
Strip the healthcare specifics away and look at the architecture. On one side, hospitals run AI documentation and coding tools whose job is to make sure every billable detail of a patient encounter shows up in the claim. On the other side, insurers run AI that reviews claims and decides what gets paid. Each system is doing exactly what it was built to do. Each system’s success metric is the other system’s failure metric.
Payers are already reporting that AI-driven documentation and coding is pushing billing amounts up. That’s not a bug in the hospital’s tool. That’s the tool working. If you build a model to find every justifiable code in a chart, and the previous process was a tired human missing some of them, your output goes up. The dollar figure is the product feature.
Now add the response. The insurer’s model gets more aggressive at flagging. The hospital’s model gets more thorough at documenting. Neither side stops, because stopping means losing money to a counterparty that didn’t stop. There’s no equilibrium where both sides relax.
Why this pattern shows up everywhere
If you’ve built bots in any adversarial environment, this will feel familiar:
- Ad tech. Bid optimizers versus fraud detection, both burning compute, neither lowering the cost of advertising.
- Search. SEO automation versus ranking algorithms, an arms race that has run for two decades and mostly produced more content nobody reads.
- Trading. Latency competition where the winning move is spending more on infrastructure than the other guy.
- Scraping. Crawlers versus bot detection, an endless loop of fingerprints and countermeasures.
In every case the automation is real and the gains are real for whoever deploys it first. The system-level result is more spend, more complexity, and a cost that lands on someone who never chose to participate. In healthcare, that someone is the patient holding the bill.
What builders should take from this
I don’t think the lesson is “don’t automate billing.” The lesson is that your objective function is the entire design. When you point a model at a number and tell it to move that number, you need to know who is on the other side of the transaction and what they’re going to do about it.
Three questions I now ask before shipping anything that negotiates, classifies, or scores on behalf of one party against another:
Is there a counterparty with their own automation?
If yes, model the response. Your projected savings assume the other side stays static. They won’t. Budget for the second and third move, not just the first.
Does the metric I’m optimizing include the cost of the fight?
A coding tool that increases reimbursement 8 percent looks great until you account for appeals, denials, resubmissions, and the staff time that goes with them. A denial engine that catches more questionable claims looks great until you count the disputes it generates. Both sides can show a positive ROI on their own dashboard while the total cost of the system climbs.
Who absorbs the friction?
This is the uncomfortable one. In most adversarial automation, the party paying for the escalation isn’t at the table. Ad fraud costs get passed to advertisers. Billing disputes get passed to premiums and deductibles. If your system’s externalities land on a third party, you’ve built something that works for your client and not much else.
The unglamorous alternative
What would actually reduce cost here is boring and hard: shared standards, agreed-upon coding definitions, automation that both sides trust because both sides can inspect it. Cooperative protocols instead of competing black boxes. That’s a coordination problem, not a modeling problem, and no amount of GPU time solves it.
Which is why I expect the arms race to keep running. The individual incentive to deploy a better model is overwhelming, and the collective incentive to stand down is nobody’s quarterly goal. The longstanding feud between hospitals and insurers existed before either side had AI. All the technology did was let both of them fight faster.
If you’re building agents that act on someone’s behalf in a contested transaction, you’re in this story now. Write the counterparty into your design docs. Their bot is reading your output, and it’s got a budget too.
🕒 Published: