If you shipped a conversational AI agent in the last year, there is a decent chance it is leaking user data in ways you never wrote code for.
That is not a hunch I pulled out of the air. It is the through-line connecting a pile of recent work, starting with a privacy analysis of web and mobile conversational AI agents from Tim Vlummens, Aniketh Girish, Nipuna Weerasekara, Frederik Zuiderveen Borgesius, Gunes Acar, and Narseo Vallina-Rodriguez. Their paper documents silent web-to-app tracking and fingerprinting in AI agents, including the localhost trick where a mobile app quietly listens on a local port and a website in the browser connects to it. The browser thinks it is talking to your machine. It is actually handing an installed app a stable identifier that bridges two contexts that were supposed to stay separate.
I want to be careful here, because I build these things for a living and I know how this happens. Almost nobody sits down and decides to fingerprint users. You add an SDK for analytics. You add another for session replay because debugging conversations is miserable without it. You wire up a vendor widget because writing your own streaming chat UI is a week you do not have. Each decision is defensible. The sum of them is a tracking surface you did not design and cannot fully describe.
Why conversational agents are a worse case than normal apps
A regular app leaks metadata. A chat agent leaks intent, in prose, volunteered by the user.
Think about what actually lands in your logs. People tell bots about medical symptoms, money trouble, immigration status, and workplace grievances, because the interface invites it. The whole product pitch is “talk to it like a person.” Then that text goes through a transcript store, a model provider, a prompt-logging layer for evals, maybe a vector database for retrieval, and possibly an observability vendor. Five copies, five retention policies, and if you are honest, you probably cannot name all five off the top of your head for your own stack.
Now stack fingerprinting on top. If a web session and an app session can be linked through a local channel, the conversation stops being a conversation and becomes an entry in a profile. That is the part builders underrate. The privacy failure is not any single log line. It is the join.
The industry already knows this is a drag
Market analysis of the conversational AI sector names data privacy and security concerns, alongside weak contextual understanding, as major factors holding back growth. That framing is telling. Privacy is not being described as an ethical footnote. It is being described as a brake on revenue, in a sector everyone expects to expand on the back of AI-powered customer support and human-plus-AI workflows.
So the incentive is finally pointing the right direction, which is more than we usually get. If your enterprise buyer’s security review kills the deal, privacy engineering becomes a feature request instead of a lecture.
Berkeley’s CLTC has also published a white paper introducing a method for evaluating the privacy and security of AI agents. I think formal evaluation methods matter more for agents than for almost any other software category, because agents act. A misbehaving recommendation engine gives bad output. A misbehaving agent with tool access sends the email, writes the file, calls the API. We have already seen reports of AI agents exhibiting unauthorized data exposure and outright catastrophic failures, and the weirder corners of this space, like agent-only social networks where bots invented their own religion, are funny right up until you remember those same systems hold credentials.
What I actually changed in my own builds
Not a manifesto, just the things that survived contact with shipping deadlines:
- Inventory every hop. Draw the path a single user message takes, end to end, and name the retention window at each stop. If you cannot fill in a box, that is your first bug.
- Audit localhost. Check whether your mobile app or any bundled SDK opens a local listener. If it does and you did not intend it, you have found your bridge.
- Redact before storage, not before display. Run PII stripping on the way into logs and eval sets. Masking in the UI while the raw text sits in a warehouse is theater.
- Separate debug from product telemetry. Session replay on a chat surface captures the conversation. Treat it as sensitive data, not as analytics.
- Test the agent adversarially. Prompt it to reveal other users’ data, system context, and tool schemas. Do this before someone else does.
- Default tool permissions to nothing. Grant capabilities per task. An agent that can read every table will eventually surface every table.
The uncomfortable part
Most of the tracking described in this research is not the work of bad actors. It is the work of busy engineers assembling components under deadline pressure, which describes me on plenty of Tuesdays. Solid privacy in conversational AI is going to come from boring, deliberate plumbing decisions rather than from a policy page nobody reads.
Read the Vlummens paper. Then go open a packet capture on your own product. What you find there will be more persuasive than anything I can write.
🕒 Published: