One number that should stop every bot builder cold
$0. That is what a GPU costs you when it is sitting idle — but the moment you fire up a training run or spin up an inference endpoint, the meter starts, and right now it does not stop until it has lapped your entire salary budget. That is not a hypothetical. Bryan Catanzaro, Vice President of Applied Deep Learning at Nvidia, said it plainly: “The cost of compute is far beyond the costs of the employees.” At one of the most powerful AI companies on the planet, the machines already cost more than the people running them.
As someone who spends most of my week wiring up bots, tuning prompts, and watching token counts tick up in real time, that quote hit differently than most executive soundbites. I have felt this creeping up for a while — the AWS bill that quietly doubled, the OpenAI invoice that arrived looking like a mortgage payment — but hearing it confirmed at the Nvidia level makes it official. We are in a new cost regime, and most teams building on top of AI have not fully reckoned with what that means.
Why compute got so expensive so fast
The short answer is that modern AI models are hungry in a way that older software simply was not. A traditional web app serves a request by looking something up and returning a value. An LLM-powered bot generates every single token on the fly, burning GPU cycles with each one. Scale that across thousands of users, add retrieval-augmented generation pipelines, throw in embedding calls and re-ranking steps, and you have a system that is doing an enormous amount of floating-point arithmetic just to answer one question.
At Nvidia, where teams are training and iterating on frontier models, the compute bill is not just inference — it is the full stack: experimentation, fine-tuning, evaluation, and production serving. When Catanzaro says compute costs exceed employee costs, he is describing a team of highly paid engineers whose combined salaries still cannot keep up with what the cluster charges per hour. That is a striking inversion of how software economics have worked for the past thirty years.
What this means if you are building bots right now
For those of us building at a smaller scale, the dynamic is different but the pressure is real. Here is what I am seeing in practice:
- Token waste is money waste. Bloated system prompts, redundant context, and lazy chunking strategies are not just sloppy — they are expensive. Every unnecessary token in your prompt is a direct line item on your bill.
- Model selection matters more than ever. Reaching for the largest, most capable model by default is a habit worth breaking. A well-prompted smaller model often handles classification, routing, and simple Q&A tasks at a fraction of the cost.
- Caching is underused. Semantic caching — storing and reusing responses for near-duplicate queries — can cut inference costs significantly on high-traffic bots. Most teams I talk to are not doing this yet.
- Async and batching are your friends. Not every bot interaction needs a sub-second response. Shifting non-urgent workloads to batched, off-peak processing is one of the fastest ways to reduce spend without touching your architecture.
The deeper strategic question
Catanzaro’s comment was not a warning — it was a description of where things already are. But it raises a question that every team building AI products needs to answer honestly: are you tracking compute cost per outcome, or just compute cost in aggregate?
There is a big difference between knowing your monthly GPU spend and knowing what it costs to successfully resolve one customer support ticket, generate one qualified lead, or complete one automated workflow. The teams that will build sustainable AI products are the ones treating compute as a unit-economics problem, not a line item to be managed at the end of the quarter.
Human labor has always been optimized this way. We measure output per employee, cost per hire, revenue per headcount. Now that compute has crossed the threshold where it rivals — and in some cases exceeds — human labor costs, it deserves the same scrutiny. Build your bots like the GPU bill is a salary. Because at Nvidia, it already is.
Where this goes from here
Hardware gets cheaper over time. Inference efficiency is improving fast. New model architectures are doing more with less. The current cost crunch is real, but it is not permanent. What is permanent is the discipline of building efficiently — and the teams that develop that muscle now will be in a much stronger position when costs do come down and the real scaling begins.
For now, treat every API call like it costs something. Because it does.
🕒 Published: