At Chiplet Summit 2026, Dr. Nasrullah framed AI’s power problem in terms nobody in the room could wave off: a 10GW challenge, met with four design techniques, starting with node mixing and low-power approaches. Not a keynote about how amazing the future will be. A list. Four things you can actually do.
That framing stuck with me, because it’s the same shape as the advice I give people building bots. You don’t fix a slow agent by believing harder in it. You fix it by naming the four things eating your latency and killing them one at a time.
Why a bot builder should care about packaging
I write orchestration code. I wire up model calls, retries, tool schemas, vector lookups. Silicon packaging feels like someone else’s department. But every architectural decision I make gets priced by hardware I don’t control, and lately that price is denominated in watts.
If you run anything at scale — a support bot fielding thousands of concurrent sessions, a retrieval pipeline re-embedding a corpus nightly — you’ve already felt this. Inference cost isn’t abstract. It shows up as rate limits, as cold starts, as the reason your finance lead asks whether the agent really needs to call the big model on every turn.
So when research says chiplet co-design meaningfully cuts both energy and design cost for AI accelerators, that’s not a semiconductor story. That’s a story about what my stack will cost to run in two years.
Co-design is the part I recognize
The word doing the work here is co-design. Not “better chiplets.” Designing the chiplet arrangement and the neural network accelerator together, as one problem.
The Fengshui work on chiplet ecosystem and bespoke accelerator co-design is a clean example. For datacenter mixture-of-experts and dense LLM serving, it reports prefill energy reductions of up to 16.8%, and up to 28.7% on the energy-delay metric, achieved through operator-level heterogeneity and expert parallelism.
Read those two mechanisms again if you build agents. Operator-level heterogeneity means different operations get matched to hardware suited to them rather than everything queuing for one uniform compute block. Expert parallelism means an MoE model’s experts get spread across silicon in a way that respects how the model actually routes tokens.
That’s a familiar move. It’s what happens when you stop treating your pipeline as one undifferentiated blob of “AI” and start matching each stage to the right tool:
- Cheap classifier up front to decide whether the expensive model is needed at all
- Small model for extraction and formatting, large model reserved for reasoning
- Cached embeddings instead of recomputing the same vectors nightly
- Batching where latency budget allows, streaming where it doesn’t
Same principle, different altitude. Shape the system around the workload instead of forcing the workload through a generic system. Hardware people are now doing at the package level what good bot architecture does at the orchestration level, and they’re getting double-digit energy wins for it.
The design cost half matters more than people think
Energy savings get the headlines. The cost reduction is the part that changes who gets to play.
The University of Michigan research and related work on optimizing chiplet ecosystems point at a world where building a purpose-specific accelerator stops being a billion-dollar bet. Public programs are pushing the same direction — the U.S. CHIPS National Advanced Packaging Manufacturing Program explicitly covers chiplet ecosystems and co-design. Cadence is building platform tooling for teams designing with chiplets.
Put those together and you get an ecosystem, not a monopoly. Mix-and-match components. Shared tooling. Lower barrier to a bespoke design.
For those of us further up the stack, that means more variety in what we can rent. Not one accelerator family that everything must be squeezed onto, but accelerators shaped for retrieval, for sparse routing, for long-context prefill. The Fengshui numbers being specific to MoE and dense LLM serving is the tell. Specificity is the point.
What I’d actually do with this
Nothing, today. I’m not buying chiplets. But I am changing how I write things down.
Start logging energy-adjacent numbers alongside your accuracy metrics. Tokens per request, prefill versus decode split, how many calls hit the biggest model, how much of your context window is dead weight on every turn. Most agent codebases I look at have zero visibility into this. If hardware is about to get better at serving specific workload shapes, you want to know your workload’s shape.
Also: stop treating model choice as a one-time decision. The reason co-design works is that it refuses to freeze one layer while optimizing another. Your bot architecture deserves the same treatment. Prompt, model, retrieval strategy, and batching policy are one system, and tuning them in isolation leaves the biggest wins on the table.
Dr. Nasrullah’s four techniques were aimed at gigawatts. Mine are aimed at a bot that answers faster and costs less. Different scale, same discipline — measure the thing, name the waste, design the pieces to fit each other.
đź•’ Published: