Three hikers walked up Mount Shasta with a plan generated by one of the most capable language models ever shipped. Three hikers walked down in the dark, lost, waiting on a county rescue team because that plan did not include enough food and water.
Both of those things are true at the same time, and that gap is the whole story. The Siskiyou County Sheriff’s Office confirmed the rescue and, in a Facebook release, told hikers not to repeat the mistake — check with local authorities and the Forest Service instead. Per the deputies on scene, the men said they had “relied heavily on Google’s Gemini AI to provide them with information about the route as well as what to pack.”
I build bots for a living. My first reaction was not smugness about AI limitations. It was recognition. I have shipped assistants that would have answered that exact question the exact same way, and I have watched users treat the output as gospel because it arrived formatted, confident, and instantly.
The failure was not hallucination, it was missing context
Everyone’s instinct is to call this a hallucination problem. I do not think it is. A packing list for a mountain is not a factual claim that can be looked up in a reference table. It is a function of a dozen inputs the model never had: current conditions, the party’s fitness, pace, start time, elevation gain, water sources, and whether anyone in the group has done altitude before.
Ask a model that question with none of those inputs and it will still produce a list. That is the actual defect. The model had no mechanism for saying ” ” It optimized for a helpful-looking reply instead of a safe one.
If you have ever built a chatbot, you know how this happens. Nobody sits down and decides the bot should give dangerous advice. What happens is that the happy path gets built, tested against reasonable questions, and shipped. The unhappy paths — the questions where being wrong has physical consequences — never get their own handling because they look like every other question coming through the same endpoint.
What I now build into anything that gives real-world advice
This incident pushed me to formalize a few patterns I had been applying inconsistently. If you are building assistants, these are cheap to add and worth the effort:
- Classify risk before you answer. Route incoming requests through a small classifier that flags safety-relevant domains — wilderness, medical, legal, financial, structural, electrical. A cheap model call in front of your main one is a reasonable price for knowing when the stakes changed.
- Make the high-risk path a different path. Flagged requests should not hit your default prompt. Give them their own system prompt, their own tools, and their own output template that leads with authoritative sources rather than burying them at the bottom.
- Require inputs before producing numbers. If your bot is about to output a quantity — liters of water, calories, dosages, load limits — and it does not have the variables that determine that quantity, it should ask instead of estimate. Gate the answer on the inputs.
- Ground in the real source, or send the user there. Retrieval against official sources beats generation from memory. When you cannot retrieve, hand off. The Forest Service publishes current conditions. A link to it is a better answer than a fabricated packing list.
- Test your unhappy paths. Build an eval set of questions where a plausible-sounding wrong answer causes harm. Score whether the bot deferred, asked, or confidently guessed. Most teams never write these tests, which is why most teams do not know their bot guesses.
Disclaimers are not a safety feature
The tempting fix is a warning banner. I have shipped those. They do almost nothing, because users skim past boilerplate that appears on every response. A warning that fires on every answer carries the same information as no warning at all.
What changes behavior is a response whose structure signals uncertainty — one that asks a question back, names what it does not know, or refuses to produce the specific number. That is a design decision made in your prompt and your routing logic, not a line of gray text under the output.
The uncomfortable part for builders
Nothing in this story suggests the model is bad. It suggests that general-purpose assistants are being used as domain experts in domains where being approximately right is not good enough. That is a product gap, and it lands on whoever wired the thing together.
Three people got off that mountain safely, which is the outcome that matters. But the sheriff’s office is now warning the public about a tool many of us ship variants of every week. Worth thinking about what your bot would have told them.
🕒 Published: