My scraper broke.
Not dramatically. No stack trace, no 429, no captcha wall. It just started returning something that didn’t look like a search results page anymore. I’d built that thing three years ago, patched it maybe a dozen times, and it had always survived because the shape of Google was stable. Ten blue links. Predictable DOM. Annoying but parseable.
Then May 2026 happened, and the shape changed.
What actually shifted
On 19 May 2026, Google announced the biggest changes to Search in 25 years. AI Mode became the global default. The search box got rebuilt for conversational prompts rather than keyword strings. Traditional rankings stopped mattering in the way they used to, because the output isn’t a ranked list anymore — it’s a generated answer.
Google’s own framing is that this is “the next chapter of Google Search: intelligent AI that puts the world’s information to work for you.” Which is marketing language, sure, but it’s also an accurate description of what changed architecturally. Search became an assistant. And assistants don’t have a tenth result.
For anyone building bots, that’s not a cosmetic update. That’s a change to one of the most reliable data interfaces on the internet.
Why this hits bot builders harder than it hits marketers
Marketers have been loud about this, and fairly so — shifting strategies, rethinking what optimization even means when there’s no ranking to optimize for. But I think the quieter disruption lands on people like us, the ones who wired Google into things.
Think about how many bot architectures quietly assumed a search engine underneath:
- RAG pipelines that used search as the retrieval layer for fresh information
- Research agents that issued queries and scored the result list
- Monitoring bots that watched position changes as a proxy for visibility
- Fact-checking flows that compared multiple sources from a single query
- Anything that treated “what does Google return for X” as ground truth
Every one of those patterns depends on getting back a set of candidates you can reason over. A generated answer is a single collapsed opinion. You can’t rerank it. You can’t diversify it. You can’t check it against its own siblings, because it doesn’t have any.
We used to build a layer of judgment on top of Google’s list. Now Google ships judgment as the product, and our layer sits awkwardly on top of someone else’s conclusion.
The composability problem
This is the part I keep circling back to. Good bot architecture is about composable pieces — retrieval here, ranking there, synthesis at the end, each one swappable and testable. You can unit test a retriever. You can measure recall.
Conversational AI search fuses all three stages into one opaque call. When output quality drops, you can’t tell whether retrieval got worse or synthesis got worse, because you only see the exit. Debugging becomes vibes.
And the pace of change doesn’t help. There was a broad core update in May 2026 and a spam update in June 2026. Meanwhile Google Ads stopped allowing new smart campaigns as of 3 August, with existing ones affected too. Different product surfaces, same underlying pattern: the interfaces you built against are moving, and they’re moving faster than your integration tests.
How I’m adjusting
Nothing exotic. Mostly boring engineering discipline applied earlier than I’d have bothered before.
Treat search as a pluggable provider. One interface, multiple backends. If your code has Google-specific parsing logic scattered through it, that logic is now a liability. Pull it behind a boundary.
Own your index for anything that matters. If a specific corpus is central to what your bot does, crawl and embed it yourself. Slower to build, dramatically more stable. You control the retrieval, so you can actually measure it.
Stop assuming a ranked list exists. Design retrieval interfaces that accept either a list or a synthesized answer, and handle both. Awkward, but it means the next shift doesn’t break you.
Log provenance obsessively. When answers come pre-synthesized, your only defense against silent drift is a record of what came back and when.
Weird isn’t the same as wrong
I want to be fair here. Conversational search is genuinely more useful for a lot of human questions. Asking a messy, half-formed thing and getting a real answer beats guessing at keywords. Most people, most of the time, will prefer this.
The friction is that our tools inherited assumptions from an era that just ended. Google didn’t break anything intentionally. It stopped being a search engine, and a lot of us had quietly built on the assumption that it always would be.
That’s a reasonable thing to be caught out by. It’s also a good reminder that any dependency you don’t control is a dependency that can change shape on you — and the fix isn’t outrage, it’s architecture that expects it.
My scraper’s retired now. The interface it replaced is better designed. So, fine. Thanks, I guess.
🕒 Published: