Every stage magician’s cabinet has two sides. The audience side is lacquered, lit, and beautiful. The back side is plywood, gaffer tape, and a person crouched in the dark holding a mirror steady. Nobody buys a ticket to see the plywood. But if you sell the cabinet as medical equipment, the plywood becomes the whole story.
That’s the thought I keep returning to as the Elizabeth Holmes news cycle spins up again. A24 has a documentary called “You Can See Everything,” with interviews conducted by Nathan Fielder, arriving this fall. ABC News describes it as an intimate look at Holmes before her imprisonment. There’s also talk of a possible move to a halfway house in 2027, though the reporting doesn’t give specifics. And the UVA Darden School of Business has put out a new case study on Theranos, framed around the risks that come with building something new.
I build bots for a living. Chat agents, retrieval pipelines, tool-calling systems that reach into real APIs and do real things. So when a story about a demo that outran its engineering comes back around, I don’t read it as true crime. I read it as a code review.
The demo is not the system
Here’s what I see constantly in bot-building circles, including in my own work. You wire up an agent, you give it three tools, you run it against a handful of prompts you chose yourself, and it performs beautifully. You record a screen capture. You post it. People are impressed. Somewhere in that pipeline there’s a hardcoded fallback, a mocked endpoint, a retry loop that quietly swallows failures, and a prompt you tuned specifically against the examples in your video.
None of that is fraud. Mocking a service while you build the layer above it is normal engineering. The gap opens when the demo becomes the claim. When the screen capture starts functioning as evidence of a capability you haven’t actually measured.
The Darden case is pitched around the risks of entrepreneurship, and that framing is useful for anyone shipping agents right now. The risk isn’t ambition. The risk is a feedback loop where the only thing you’re measuring is how the thing looks when you drive it.
What “you can see everything” means in a bot
I find the documentary title a little pointed, honestly. “You Can See Everything” is exactly what we tell users about our agents, and it’s almost never true. A modern bot stack is layers of opacity:
- A model whose weights you didn’t train and can’t inspect
- A prompt that’s probably been edited forty times with no version history
- A retrieval step pulling from an index you haven’t audited in weeks
- Tool calls firing against services that may be degraded, rate-limited, or returning stale data
- A response that reads fluent and confident whether or not any of the above worked
That last one is the dangerous part. A broken database throws an error. A broken agent writes you a paragraph. Fluency is not a health check, and a bot that sounds certain while operating on garbage is the software equivalent of a result printed on nice letterhead.
The instrumentation I actually ship
So what do you do about it? Boring things, mostly. I log every tool call with inputs, outputs, and latency, and I keep those logs long enough to be useful in an argument. I write evaluation sets I didn’t personally curate, including the ugly inputs real users send. I make the agent surface when retrieval returned nothing instead of letting the model paper over the hole. I track refusal and fallback rates, because a rising fallback rate is the earliest signal that something upstream broke.
And I separate two numbers in my head that are easy to blur together: how often the bot produces an answer, and how often that answer is correct. A system can hit 99% on the first and be worthless on the second.
Why this story keeps landing
Theranos ran on blood tests, which is about as high-stakes as outputs get. Most of us are shipping customer support agents and internal research assistants, nowhere near that consequence level. But the structural lesson scales down fine. If the only people who know what’s behind your demo are you and the plywood, you’ve built a show, not a product.
I’ll watch the documentary when it drops. Fielder’s whole body of work is about the distance between performance and reality, which makes him a strange and probably perfect choice for this subject. In the meantime, I’d suggest the cheaper exercise: open your own agent’s trace logs and read the last fifty real conversations. Not the ones you demoed. The real ones.
That’s the back side of the cabinet. Most of us have never looked.
🕒 Published: