What would you actually build if you believed an AI could brute-force its way through a problem that stumped humans for a century? Because that’s the question sitting on my desk right now, and I don’t think the answer is “a better chatbot.”
Here’s what happened: GPT-6 Astra, an AI system built by Amazon, cracked a 108-year-old WWI German radio cipher in 2026. The decoded message revealed British warship movements. And the part that matters most to anyone who builds things for a living — the result was verified against historical naval logs.
That last sentence is the whole story. Let me explain why.
Verification is the feature, not the footnote
I’ve lost count of how many bot projects I’ve watched collapse because nobody built a way to check the output. You wire up a model, it produces something that looks plausible, everyone nods, and three weeks later you discover half the results were confident nonsense.
Cipher work is unusually honest about this. A decryption either produces coherent German text that matches a known historical record, or it doesn’t. There’s no partial credit, no “well, it captured the vibe.” The naval logs functioned as ground truth, and that’s the only reason this result is interesting instead of just another impressive-sounding claim.
So when I look at this as a builder, I don’t see a cryptography story. I see a template:
- A problem with a clear, checkable answer
- An external source of truth that wasn’t part of the model’s reasoning process
- A result that either matches or fails, with no room for interpretation
Most tasks we hand to bots have none of these properties. That’s a design problem, not a model problem.
What this changes about how I scope projects
I used to sort candidate bot tasks into “the model can probably do this” and “the model probably can’t.” After this, I’m sorting them differently: can I tell whether the output is correct?
If yes, I’ll throw a capable model at it and let it grind. If no, I need to either build a verification layer first or pick a different problem. The ordering matters. Verification isn’t the thing you bolt on after the demo works — it’s the thing that determines whether the demo means anything.
For the kind of systems we talk about on this site, that usually means:
- Pick problems with checkable outputs. Code that compiles and passes tests. Data transforms you can validate against a schema. Parsing tasks where you have a known-good sample set.
- Keep the checker independent. If the same model both produces and grades the answer, you’ve built a mirror, not a test. The naval logs weren’t generated by the AI.
- Log the failures loudly. A pipeline that quietly drops the attempts that didn’t verify is lying to you about its success rate.
The unglamorous part nobody puts in the headline
Nothing in the facts here tells us how much human work went into framing the problem, sourcing the ciphertext, or locating the right logs to check against. I’d bet real money it was substantial, because it always is. Somebody had to know which naval records to pull. Somebody had to decide that a match against those records counted as proof.
That’s the work that doesn’t trend. It’s also where most of the engineering lives. If you’re building a bot and you find yourself spending eighty percent of your time on data plumbing and validation rather than prompts, you’re probably doing it right.
A small exercise, if you want one
Take something in your own stack that you’ve been treating as unverifiable. A summarization step, a classification pass, a code generation task. Then ask: what would the naval logs be here? What external record could I compare against that I’m not currently comparing against?
Sometimes there isn’t one, and that’s a real answer — it means you’re operating on trust and should size your risk accordingly. But often there’s something sitting right there. Historical outputs you never diffed. A second data source you assumed agreed with the first. A test suite you could run on generated code but don’t.
The century-old cipher is a good headline. The habit worth stealing is smaller and less exciting: decide how you’d know you were wrong, and build that part first.
I’ll be putting together a walkthrough on independent verification layers for generation pipelines soon. If you’ve got a case where you solved this cleanly, send it over — I’d rather show real patterns than invented ones.
— Sam
đź•’ Published: