Google’s Gemini answered every question three hikers asked about Mount Shasta. Those same three hikers ended up lost in the dark waiting on a rescue crew in Siskiyou County, California.
Both things are true, and the gap between them is where I spend most of my working life. I build bots. I ship them. And this story is the cleanest example I’ve seen this year of a failure mode that has nothing to do with model quality and everything to do with how we wire these systems together.
According to the sheriff’s office, the men told a deputy at the end of the rescue that they had relied heavily on Gemini for route information and for what to pack. The food and water guidance they got was not adequate. They descended in the dark, lost the route, and needed help. The Siskiyou County Sheriff’s Office posted a warning on Facebook telling other hikers not to repeat the mistake, and to check with local authorities and the Forest Service instead.
The failure wasn’t hallucination, it was missing plumbing
It’s tempting to file this under “AI made stuff up.” I don’t think that’s the interesting part. The interesting part is that a general-purpose assistant answered a question that required live, local, authoritative data, and nothing in the interaction signaled that it was operating outside its competence.
Any of us who have built a retrieval layer know exactly what was missing here. A trip-planning bot that deserves to exist would need:
- A live weather source, pulled at query time, for the specific elevation band
- Current trail and permit status from the managing agency
- Sunset time for the date of the hike, in that location
- Water calculations tied to exertion level, temperature, and total hours on trail
- A hard citation trail back to whichever ranger district actually governs the route
None of that is exotic. Every item on that list is an API call or a scraped official page. The reason a general assistant doesn’t do it is that it wasn’t built to be a mountaineering planner. It was built to answer things. And answering things is what it did.
Confidence is a design choice
Here’s what I keep coming back to as a builder. Text output has a uniform texture. A model’s answer about pasta water and a model’s answer about alpine water look identical on screen. Same tone, same structure, same implied authority. The user has no signal to grade the two differently.
That’s not the model’s fault. That’s an interface problem, and interfaces are our department.
If you’re building anything where a wrong answer has physical consequences, you owe your users a visible confidence boundary. Some patterns that work:
Route the query, don’t just answer it
Classify incoming questions before generating. If a request touches safety, medical, legal, or navigational territory, hand it to a path that requires grounding sources or refuses outright. A classifier plus a refusal path is maybe forty lines of code. It’s the highest-value forty lines in the whole app.
Cite or decline
Make sourcing a hard requirement, not a nice-to-have. If retrieval returns nothing authoritative, the bot says so and names who to ask. In this case the correct output was short: contact the Forest Service and the local ranger district for current conditions. That answer is less satisfying than a packing list. It’s also the one that keeps people off the mountain after dark.
Surface what you didn’t check
I’ve started adding an explicit “not verified” block to outputs in high-stakes tools. It lists what the system could not confirm. Users read it. It reframes the whole response from verdict to draft.
What this means if you ship bots
The sheriff’s warning was aimed at hikers, but the actionable version is aimed at us. Users are going to treat any confident paragraph as a plan. They did it here, on a mountain, with real consequences. They’ll do it with your medication tracker, your legal doc summarizer, your logistics assistant.
You cannot fix that by adding a disclaimer in the footer. You fix it structurally: narrow the domain, ground the answers, gate the risky paths, and design refusals that are genuinely useful instead of dead ends. A bot that says “I can’t tell you how much water to carry, but here’s the ranger district phone number and today’s forecast” is doing more work than one that confidently produces a list.
Three people got off Mount Shasta safely, which is the outcome that matters. What we get to take from it is a spec. Build for the moment your system is wrong, because that’s the moment your architecture is actually being tested.
đź•’ Published: