Three hikers went up Mount Shasta. Rescuers brought them down.
According to the Siskiyou County Sheriff’s Office, the trio used Google’s Gemini to plan their expedition. The AI told them to pack less food and water than the climb actually demanded. They ended up stranded overnight in a steep canyon after going off course. The sheriff’s office followed up with a public plea: don’t plan trips this way. Talk to local authorities and the Forest Service instead.
I build bots for a living. When I read this, my first reaction wasn’t “AI bad.” It was “somebody shipped a planning assistant with no grounding layer and no refusal path.” That’s a design problem, and it’s one most of us reading this site have already built into something.
The failure wasn’t hallucination, it was confidence without sourcing
A model asked “how much water should I bring up Mount Shasta” will answer. It will always answer. It has read enough hiking content to produce a plausible number with a plausible justification attached. What it cannot do is check the current snow conditions, the route closure status, the group’s fitness, or the fact that these were novice climbers.
The output looked identical either way. That’s the part builders should sit with. A wrong water estimate and a right water estimate arrive in the same tone, the same formatting, the same reassuring cadence. There’s no visual difference between a number pulled from a Forest Service advisory and a number averaged out of a thousand blog posts about easier mountains.
Users read tone as a signal of reliability. We know it isn’t one. We build interfaces that pretend otherwise anyway.
What a grounded version of this would look like
If you were building a trip-planning bot today, the architecture writes itself once you accept the model shouldn’t be the source of truth:
- Retrieval before generation. Route conditions, permit requirements, and weather come from live sources. The model formats and explains; it doesn’t originate the facts.
- Hard refusal categories. Some questions shouldn’t get an answer at all. Water and calorie requirements for a technical alpine ascent belong in that bucket. The correct output is a pointer to the ranger station, not a number.
- Source attribution per claim. Not a citation dump at the bottom. Every specific quantity tagged with where it came from, so an unsourced guess looks visibly different from a sourced fact.
- Confidence that degrades visibly. If the retrieval layer returned nothing for Mount Shasta in September, the response should look thinner, not smoother.
- Escalation defaults. Any query touching physical safety ends with a human authority to contact, every time, no exceptions for brevity.
None of that is hard to build. It’s a retrieval pipeline, a classifier on intent, and some discipline about what your bot is allowed to say. The reason general assistants don’t work this way is that refusing feels like failing, and product teams optimize for helpfulness metrics that treat every answered question as a win.
The category we keep getting wrong
I’ve been sorting bot use cases into two piles lately. In the first pile, being wrong costs you time. Bad code suggestion, awkward email draft, mediocre summary. You notice, you fix it, you move on. Most of what we build lives here, and models are genuinely good at it.
In the second pile, being wrong costs something you can’t get back. Dosage. Structural load. How much water to carry above the treeline. The feedback loop is delayed and the correction window closes before you know you needed it. These three hikers found out they were under-supplied at the exact moment that information became useless to them.
Chat interfaces flatten both piles into the same box. Same text field, same friendly response, same absence of friction. A user who has had fifty good experiences asking about recipes and Python errors has no reason to treat the fifty-first question differently, even though it’s a completely different risk class.
That’s on us as builders, not on the hikers. They used a tool the way the tool invites you to use it.
What I’d take back to your codebase
Go look at whatever assistant you’re shipping and ask a plain question: what’s the worst thing a user could do based on a confident wrong answer from this thing? If you can name it, you have a refusal path to write and a retrieval source to wire in. If you can’t name it, you probably haven’t looked hard enough at how people actually use what you built.
The sheriff’s office gave a solid piece of engineering advice without meaning to. Route the question to the system that actually knows. For a mountain, that’s the Forest Service. For your bot, it’s whatever authoritative source you’ve been too busy to integrate.
🕒 Published: