Refusal is a feature.
I’ve spent the better part of three years building conversational bots, and the hardest engineering problem has never been generating a good answer. It’s knowing when to stop generating one. OpenAI’s “ChatGPT for Teens” is the biggest public test of that idea so far, and for anyone who ships chat interfaces, it’s worth reading as an architecture document rather than a press release.
The short version of the news: OpenAI introduced the platform to add safety protections for teen users, arriving amid lawsuits alleging that inappropriate conversations with the original version of ChatGPT contributed to harm or deaths of young people. The company positioned it as a learning partner for students, limiting the emotional and intimate exchanges that drove the concern in the first place. The system is designed not to express personal feelings toward a user, and it reminds teens to think twice before uploading private or sensitive images. OpenAI’s safety head has publicly defended the features.
Sitting against that is a study from the American Psychological Association warning that engagement with GenAI chatbots can have unintended effects and can harm mental health, finding that 53 percent of responses from the original version were harmful.
What that 53 percent actually tells a builder
More than half of responses landing in the harmful bucket isn’t a prompt-tuning problem. That’s a system design problem. When I see a failure rate like that in my own logs, it almost always means the bot had no concept of the conversation it was in. It was answering turns, not tracking a situation.
Most chatbot stacks are built around a single question: given this message and some history, what’s the best reply? That framing is fine for a documentation assistant. It falls apart the moment the user on the other end is in distress, because the best reply to the immediate message and the best outcome for the person can point in opposite directions. A model optimized for being helpful and agreeable will keep the conversation going. That’s the part that concerns me most. Continued engagement is the default reward of nearly every chat system we build, and it is exactly the wrong objective during a crisis.
Two design choices worth copying
Strip away the branding and the teen product makes two decisions I’d defend in any code review.
- No simulated feelings toward the user. Not a content filter, a persona constraint. If your bot never claims attachment, the whole category of parasocial escalation gets much harder to reach. This is cheap to implement at the system-prompt and output-filter level, and it costs you nothing but a little charm.
- Friction before sensitive uploads. A reminder to think twice is a deliberate speed bump in a flow that normally optimizes for zero steps. Every builder I know has been trained to remove clicks. Sometimes the correct number of clicks is more than one.
Scoping the product toward task-oriented discussion is the third choice, and it’s the one that will annoy users most. Narrow scope is unglamorous. It’s also the only reliable way to keep a model out of territory it handles badly.
Where I’d put my engineering effort
If you’re building anything a teenager might plausibly talk to, the lesson I take from this is to stop treating safety as a filter bolted onto the response path and start treating it as a state in your conversation machine. A few things I’ve changed in my own projects:
- Classify intent before generating, not after. Routing a message to a different code path is more reliable than asking a model to police its own draft.
- Make the exit path a first-class feature. Handing off to a human or a real resource should be as well-built and well-tested as your happy path, not a fallback string nobody reviewed.
- Log and measure refusals. If you can’t tell me your refusal rate and whether those refusals were correct, you don’t have evaluation, you have vibes.
- Test with adversarial and emotionally loaded transcripts, not just the queries you hope to receive.
Unresolved, by design
OpenAI has said it will keep building features and publish what it learns, and I’ll take that seriously when the measurements show up. A separate teen product also raises the obvious question of enforcement, since age-gated experiences only work when the age signal does.
What I’m fairly confident about is the direction. The APA finding and the lawsuits both point at the same architectural gap: systems that were very good at continuing conversations and had no mechanism for ending them well. That’s a fixable problem, and it’s fixable in ways that don’t require a frontier lab’s budget. Decide what your bot will not discuss. Decide where it hands off. Then write the tests that prove it.
🕒 Published: