When was the last time you checked how many tokens your bot burns per watt? If the answer is never, you’re in good company, and that’s exactly the gap NVIDIA is poking at with its latest argument about AI factories.
The company’s pitch comes down to three words: productive, durable, fungible. NVIDIA says the AI factories returning the most on their build cost are the ones that earn more per megawatt, stay useful for longer, and can run a wide range of workloads instead of one narrow thing. It reads like infrastructure marketing, and partly it is. But as someone who spends most of the week wiring up agents, retrieval pipelines, and inference endpoints, I think the framing is more useful to builders than it first appears.
Productive means your unit economics, not someone’s benchmark
NVIDIA’s headline claim is about throughput per megawatt. The Vera Rubin NVL72, according to the company, delivers 30x or more throughput per megawatt than the GB300 NVL72, and up to 45x lower cost per million tokens on DeepSeek V4 Pro.
Those are vendor figures on a vendor-selected workload, so treat them as a direction rather than a promise. What matters for anyone shipping bots is the unit being measured. Not teraflops. Not GPU count. Cost per million tokens, which is the same number that lands on your API invoice at the end of the month.
That shift in metric has a direct consequence for how you design. If you’ve ever built a multi-step agent, you know token consumption does not grow politely. A single user question can fan out into planning calls, tool calls, retries, and a summarization pass. Ten tokens of user input can become fifty thousand tokens of internal reasoning. When the cost per million tokens on the serving side drops by a large factor, designs that were previously too expensive to justify, like deeper reasoning loops, multiple candidate generations, or self-checking passes, start to pencil out.
So the practical read is this: architectural choices you ruled out eighteen months ago on cost grounds deserve a second look. The economics underneath them move.
Durable is the part nobody expected
The accepted wisdom in this space was that AI hardware ages like milk. NVIDIA pushes back with two data points. The A100, which shipped in 2020, remains in commercial service. And CoreWeave has extended bookings through 2029.
The broader signal is in the accounting. Every major operator has extended the depreciation schedule on its servers, which is a quiet but meaningful admission that these machines earn revenue for longer than originally modeled. Depreciation schedules are not where companies go to be optimistic. Finance teams stretch them when the asset keeps paying.
For builders, this changes the risk calculation on committing to a stack. If older silicon stays commercially viable for years, then the model you fine-tuned on last generation’s hardware does not become orphaned the moment new systems ship. It gets pushed down the stack to cheaper capacity. That’s a reason to build with portability in mind rather than chasing whatever is newest.
Fungible is the word doing the most work
Of the three, fungible is the one I’d underline. NVIDIA’s claim is broader workload support, meaning the same infrastructure handles training, fine-tuning, inference, and whatever the next shape of AI work turns out to be.
Anyone running a real bot in production already knows why this matters. A working system is rarely one model doing one job. It’s an embedding model for retrieval, a small fast model for routing and classification, a larger model for the hard reasoning, maybe a reranker, maybe a vision model for document parsing. Each of those has a different compute profile. Infrastructure that only does one of them well forces you into fragmented capacity and awkward handoffs.
This also explains why the depreciation story and the fungibility story are the same story. Hardware stays useful for years precisely because it can be reassigned. Training capacity becomes inference capacity. Frontier serving becomes batch processing. The asset outlives any single job it was bought for.
What I’d take back to the keyboard
Strip out the vendor framing and three practical habits fall out:
- Instrument tokens per request, per feature, and per user session. You cannot reason about cost you do not measure.
- Write your serving layer behind an abstraction so you can move between model sizes and hardware generations without a rewrite.
- Stop assuming expensive designs stay expensive. Re-benchmark the ideas you shelved.
Whether NVIDIA’s specific multiples hold up in your workload is an empirical question, and the honest answer is you’ll have to test it. But the mental model, that returns come from productivity, durability, and flexibility rather than raw horsepower, is a better lens than the spec-sheet comparisons most of us default to. The interesting number was never how many GPUs you have. It’s how much useful work each watt produces, and for how many years it keeps producing it.
🕒 Published: