Two things are true at the same time right now. The three biggest model labs in the world — OpenAI, Anthropic, and Google — have reportedly been in talks about AI safety for weeks, according to OpenAI global policy chief Chris Lehane. And in that same stretch, two viral conversations about AI safety went sideways in a way that made it genuinely hard to tell fact from fiction, as TechCrunch put it this week.
So: the most sophisticated safety coordination we’ve seen, running in parallel with a public discourse that can’t reliably establish what happened. I build bots for a living. That combination is the part that actually worries me.
The verification problem is a builder problem
When ABC News and CBS News both ran segments in the same week about models showing unexpected behavior, the interesting detail wasn’t the incidents themselves. It was how little any of us watching could do with the information. The clips exist. The concern is real. The specifics are thin enough that you can’t act on them.
That’s not a media criticism. It’s a structural issue, and if you ship anything with a model inside it, you’ve felt it. Something weird happens in production. A tool call fires that shouldn’t have. Output drifts in a direction you didn’t design for. You go looking for prior art — has anyone else seen this? — and what you find is a mix of screenshots, secondhand accounts, and a thread where someone confidently explains a mechanism they clearly inferred from the outside.
TechCrunch’s framing stuck with me: one of those viral cases involved Andrew Yang, the former presidential candidate. Not a random poster. Someone with a platform, a reputation, and every incentive to be accurate. And it still ended up in the pile of things that are hard to verify.
What I actually do about it
I’ve stopped treating external reports as evidence and started treating them as hypotheses. That shift changed how I build.
- Log the inputs, not just the outputs. When your bot does something strange, the output is the least useful artifact. You need the full prompt, the system message version, the tool schema, the retrieved context, and the model version string. Without those, you have an anecdote.
- Version your prompts like code. If you can’t diff what changed between the run that worked and the run that didn’t, you’re guessing. Prompts in a config file with a commit history cost almost nothing and save entire afternoons.
- Write a regression test the moment you see odd behavior. Even a bad test. Even one that’s flaky. A failing test is a fact. A screenshot is a story.
- Pin model versions in production. “Latest” is a fine default for prototyping and a terrible one for anything users touch. Unexpected behavior is a lot easier to reason about when the model underneath didn’t silently change.
- Keep a private incident log. Date, what you expected, what happened, what you changed. Mine has maybe forty entries and it’s the single most useful document in my toolkit, because it’s the only source of AI behavior claims I can fully verify.
Why the lab talks matter more than the discourse
Experts have been warning about the pace of advancement and the risks that come with it, and those warnings are reasonable. But warnings don’t give me anything to implement. Coordination between labs might.
The thing I’m hoping comes out of weeks of conversation between OpenAI, Anthropic, and Google isn’t a joint statement. It’s shared vocabulary and shared reporting formats. Right now every lab describes model failures in its own terms, which means a behavior documented against one model is nearly impossible to check against another. If those three agreed on how to name and report a class of unexpected behavior, the rest of us could finally compare notes instead of trading screenshots.
That’s an unglamorous ask. It’s also the one that would change my week-to-week work more than any safety framework published as a PDF.
Sit with the discomfort
The honest position, if you build with these models, is that you know less than the confident posts suggest and more than the dismissive ones do. Something real is happening — two major networks covered it, three major labs are talking about it, and experts are flagging it. And the public record is still too noisy to tell you which specific claims to trust.
My advice is to stop trying to resolve that tension from the outside. Build the instrumentation that lets you generate your own facts. Log aggressively, pin versions, test the weird stuff, and treat your own incident history as the primary source it actually is.
Everyone else’s screenshots are someone else’s problem. Your logs are yours.
🕒 Published: