One woman. Two states. Zero visits to North Dakota. Yet somehow, AI decided she was guilty.
A Tennessee woman recently found herself arrested for crimes committed in a state she claims she’s never set foot in, all thanks to facial recognition technology that pointed police straight to her door. The Fargo police chief has since apologized for the mistakes that led to her arrest, but the damage was done—and the questions this raises for those of us building AI systems are impossible to ignore.
As someone who builds bots for a living, I spend my days thinking about how AI can make our lives easier. But this case is a stark reminder that when we hand over critical decisions to algorithms, we’d better be damn sure they’re getting it right.
The False Match That Changed Everything
The details are chilling in their simplicity. Law enforcement ran facial recognition software. The system flagged a match. Officers made an arrest. A grandmother ended up in jail for fraud she didn’t commit, in a place she’d never been.
This isn’t a sci-fi dystopia—it’s happening right now, with technology that’s already deployed across police departments nationwide. And if you’re building AI systems, especially ones that touch people’s lives in meaningful ways, this should keep you up at night.
Why Facial Recognition Gets It Wrong
Here’s what most people don’t understand about facial recognition: it’s not magic, and it’s definitely not foolproof. These systems are trained on datasets, and those datasets have biases baked in from day one.
The technology works by converting faces into mathematical representations and comparing them against databases. But lighting conditions, camera angles, image quality, and yes—the demographic makeup of training data—all affect accuracy. Studies have repeatedly shown that facial recognition systems perform worse on women and people of color, with error rates that should make any responsible developer pause before deployment.
When I’m building a chatbot that recommends products, a wrong answer means someone buys the wrong widget. Annoying, but fixable. When facial recognition makes a mistake, someone loses their freedom.
The Human Factor We Keep Forgetting
But here’s the thing that really gets me: this wasn’t just an AI failure. It was a human failure too.
Somewhere in this process, people looked at a facial recognition match and treated it as gospel truth rather than what it actually is—a probabilistic suggestion that requires verification. The system said “possible match,” and humans heard “definitely guilty.”
This is the danger zone for anyone building AI tools. We create systems that output confidence scores, probability percentages, and ranked results. But users often treat these outputs as binary yes/no answers. They see 78% confidence and think “close enough,” without understanding what that number actually means or what could go wrong in the remaining 22%.
What Bot Builders Need to Learn From This
If you’re building AI systems—whether that’s facial recognition, content moderation bots, or automated decision-making tools—this case offers some hard lessons:
First, your confidence scores matter. Don’t just slap a percentage on an output and call it a day. Think carefully about what that number means and how users will interpret it. Better yet, force users to understand the limitations before they can act on your system’s recommendations.
Second, build in friction for high-stakes decisions. When the consequences are serious—and they don’t get much more serious than arrest and imprisonment—your system should make it harder, not easier, to act on AI recommendations alone. Require human verification. Demand multiple sources of evidence. Make users work for it.
Third, test your systems on edge cases and adversarial examples. That means diverse datasets, real-world conditions, and scenarios where your AI might fail. If you’re only testing on clean, well-lit photos of people staring directly at cameras, you’re not ready for deployment.
Moving Forward Without Moving Fast and Breaking Things
The tech industry loves to “move fast and break things,” but we can’t afford that attitude when we’re building systems that affect people’s lives. A broken shopping cart is one thing. A broken justice system is another entirely.
This Tennessee woman’s ordeal should be a wake-up call. Not to abandon AI—the technology has real value when used responsibly—but to approach it with the humility and caution it deserves.
As bot builders, we have a responsibility to understand not just what our systems can do, but what they shouldn’t do. Sometimes the most important code we write is the code that says “stop—this decision needs a human.”
Because at the end of the day, when AI gets it wrong, it’s real people who pay the price. And no algorithm should have that much power without serious guardrails in place.
đź•’ Published: