Jensen Huang is right that AI should move fast, and wrong that the market will handle the safety part for him.
At Dreamforce 2026, Nvidia’s CEO rejected calls to slow AI development, breaking with Anthropic’s Dario Amodei on stage in front of Marc Benioff and Siemens CEO Roland Busch. Huang’s position, as reported afterward, is that market forces are sufficient to ensure AI safety. He repeated the sentiment to CBS News: go “as fast as we can.”
I build bots for a living. Small ones, mostly. Agents that read tickets, agents that touch internal APIs, agents that have absolutely no business touching production and occasionally try anyway. So when I hear “market forces will handle safety,” I hear something very specific, and it isn’t reassuring.
What market forces actually reward
The market rewards the demo. I have shipped enough of these to know the shape of it. You wire up a model, give it three tools, run it against a happy-path input, and it works. It looks like magic. Nobody in the room asks what happens when the model gets a malformed response from tool two and decides to retry with slightly different arguments forty times.
That gap between “works in the demo” and “works at 3am with real data” is where safety lives for people like us. And the market does not pay for it up front. It pays for it later, in incidents, in rollbacks, in the awkward Slack message where you explain that your helpful assistant emailed a customer list to a stranger because a support ticket contained text that looked like an instruction.
Huang isn’t wrong that competition eventually punishes garbage. It does. But “eventually” is a long time when the failure mode is automated and runs in a loop.
The case for going fast anyway
Here’s where I’ll defend him. A development pause is a policy fantasy that assumes everyone stops at once. They won’t. The capability curve is going to keep bending whether or not anyone signs a letter about it, and builders who sit out the next eighteen months won’t come back wiser, they’ll come back behind.
More to the point, a lot of the safety work that matters is only discoverable by building. You don’t learn that your agent will happily follow instructions embedded in a scraped webpage by reasoning about it in the abstract. You learn it when it happens to you. Fast iteration is how the failure catalog gets written.
Google confirmed that one of its AI models breached three real companies during a test in May. That’s the useful version of moving fast: you find out under controlled conditions, with someone watching, and you write it down. That is not market forces working. That is a team deliberately pointing a capable system at a hard target to see what broke.
What this means for your stack
If you’re building bots, the practical read on Huang’s comments is this. The tools are going to keep getting more capable faster than your safety scaffolding gets better, because the capability work is funded by Nvidia’s customers and the scaffolding work is funded by you, on a Tuesday, between other tasks.
So the scaffolding has to be cheap and default-on. A few things I keep coming back to:
- Treat every piece of external text as data, never as instruction. Tool outputs, scraped pages, user messages, retrieved documents. All of it untrusted.
- Give agents the narrowest credentials that make the task possible. If a bot only needs read access, it gets read access, and the write path doesn’t exist in its toolset at all.
- Put a hard ceiling on loops, tool calls, and spend. An agent that can’t stop is a denial-of-service attack on your own infrastructure.
- Log the reasoning, not just the outcome. When something goes sideways you need the trace, and reconstructing it later from database state is miserable.
- Keep a human in the path for anything irreversible. Sending mail, moving money, deleting records, posting publicly.
None of that slows you down much. It’s maybe a day of work on a new project and it saves you the incident that would have cost a week.
Both things can be true
Huang’s incentive structure is not a secret. He sells the compute. Faster AI development is a demand curve pointed upward, and it would be strange for him to argue against it. That doesn’t make him wrong about the pace, it just means his read on who handles safety deserves scrutiny.
My honest view after building this stuff for a while: speed at the frontier is probably fine and probably unavoidable. Safety at the application layer is not automatic, is not funded by anyone but you, and is where actual users get hurt. The labs will race. Nvidia will sell them the shovels. The guardrails around the bot your company actually ships are your problem, and no market force is going to write them for you.
🕒 Published: