Zero percent is a bold number.
Nvidia CEO Jensen Huang told CBS News that “2030 is not going to be the end of the world. There is 0% chance that’s going to be the end of the world.” He called the extinction talk “doomsday narratives,” added that “scaring people is unnecessary. It is irresponsible,” and pointed to safe product deployment as central to his company’s success.
He was answering Evan Hubinger, formerly Alignment Science Organization Lead at Anthropic, who put the odds of AI destroying the world in the next decade at over 10%.
So we have 0% versus 10%+. Two numbers, neither one derived from anything you could reproduce in a notebook. I build bots for a living, and that gap is the part I keep staring at.
Nobody ships with a zero in the error column
Here is what bothers me about “0%,” and it has nothing to do with whether I think the world ends in 2030. I don’t. It’s that engineers don’t talk like that about systems they’ve built.
Every bot I have ever put in front of real users has surprised me. Not in a cinematic way. In the boring way: a retrieval step returns a stale document and the agent confidently answers from it. A tool call retries three times because a timeout got swallowed, and now there are three Jira tickets instead of one. A prompt injection buried in a scraped page convinces the agent that its actual instructions were different. None of that is superintelligence. All of it is the same category of problem — a system doing something its builder did not predict.
If I can’t hit 0% on “my support bot will not email a customer the wrong invoice,” I have a hard time treating 0% as a serious estimate for anything larger. That’s not doomerism. It’s just what shipping teaches you.
The part of Huang’s answer that’s actually useful
Strip out the headline number and there’s a real claim underneath: safe deployment is tied to commercial success. That one I’ll sign. It’s the version of safety that shows up in my repos rather than in conference panels.
Concretely, for anyone building agents right now:
- Scope tool permissions like you scope database credentials. An agent with write access to everything is a single bad retrieval away from a bad day. Separate read tools from write tools, and gate the write tools behind explicit approval.
- Treat retrieved content as untrusted input. A web page, a PDF, a user upload — none of it is instructions. If your system prompt and your retrieved context share the same channel with the same authority, you have built an injection target.
- Log the whole trajectory, not the final answer. When something goes wrong you need the tool calls, the arguments, and the intermediate reasoning. Post-hoc debugging of a single output string is guesswork.
- Write evals before features. A small set of cases that must not regress beats a long document about principles. Twenty adversarial prompts you actually run are worth more than a policy nobody tests.
- Put a human in the loop where the action is irreversible. Sending money, deleting records, contacting customers. Reversibility is the cleanest line I know for deciding what needs approval.
None of that depends on resolving the 0-versus-10 argument. It’s the same work either way, which is sort of the point.
Percentages are a bad format for this conversation
The trouble with trading probabilities is that it flattens very different questions into one number. “Will a model autonomously cause catastrophe” and “will a poorly supervised agent pipeline cause real harm at scale” are not the same risk, and they don’t have the same mitigations. A single percentage forces you to pick a team instead of picking a failure mode.
Builders tend to be more comfortable with failure modes. We name them, reproduce them, write a test, move on. That habit doesn’t scale perfectly to civilization-level questions, but it beats vibes-based forecasting, and it produces artifacts other people can check.
What I’d want to hear instead
From the people selling the compute and the people warning about it alike: show the method. What does the safe deployment process look like internally? What gets blocked, by whom, and on what evidence? Which evals gate a release? That’s checkable. A percentage isn’t.
Huang is right that scaring people for its own sake is a bad way to run a public conversation. I’d add that reassuring people with an absolute is also a bad way, because it gives everyone downstream permission to stop checking their own systems. The engineers I trust most are the ones who tell me the number is small and they’re still watching it.
Build like your agent will do something you didn’t plan for. Mine always does.
🕒 Published: