Remember when the smart speaker on your kitchen counter was supposed to change everything? You’d say a wake word, it would ship your voice to a datacenter, and roughly a second later it would tell you the weather or mishear you entirely and start playing jazz. That was the shape of consumer AI for years: a microphone with a very long extension cord.
Andrew Feldman, CEO of Cerebras, thinks we’re closing in on something different. TechCrunch reports he believes we’re nearing a future where everyone has a personal AI assistant at home, and that these assistants will run locally. He predicts they’ll become as commonplace as household appliances, running partly on local devices and partly in the cloud.
That last clause is the part I keep rereading. Not “local instead of cloud.” Partly both. For anyone who builds bots for a living, that’s not a prediction about consumer gadgets. That’s a prediction about your architecture diagram.
Hybrid is the hard part
Running a model locally is no longer exotic. Running a local model and a remote model as one coherent assistant, where the user never thinks about which one answered, is still genuinely difficult. I’ve built enough of these to know where the seams show.
The routing decision is the whole ballgame. Something has to decide, per request, whether the on-device model can handle this or whether it needs to call out. Get it wrong in one direction and you burn tokens and latency on “what time is it.” Get it wrong in the other and your assistant confidently fumbles a question that needed a bigger brain.
Most teams I’ve seen start with a naive classifier and a prompt-length heuristic. It works until it doesn’t. The patterns that hold up tend to be more boring and more layered:
- Capability tiers, not model tiers. Route on what the task needs — tool calls, long context, multi-step reasoning — rather than on a vague notion of “hard” versus “easy.”
- Local-first with escalation. Let the small model attempt the task, then promote to the remote model when confidence drops or a required tool isn’t available on-device.
- Shared state across both paths. Conversation memory, user preferences, and tool results need to live somewhere both models can read, or the handoff feels like talking to two different entities.
- Graceful degradation when the network dies. The home assistant that becomes a brick during an outage will not be treated like an appliance. Appliances work.
The appliance comparison is doing real work
Feldman’s framing of these assistants as household appliances is worth taking literally, because appliances have properties software people are bad at. A dishwasher doesn’t need an account. It doesn’t push a firmware update that changes the buttons. It doesn’t stop working because a subscription lapsed or a vendor sunset the product line. It sits there for a decade and does its job.
That’s a brutally high bar for anything built on current AI stacks. Our tooling assumes we can update the model weekly, patch the prompt on Friday, and watch the metrics. An appliance-grade assistant means versioned behavior, offline capability, and interfaces that don’t drift under the user’s hands. If you’re building bots now, that’s a mindset shift more than a tech shift.
What this means for builders this year
I’m not going to pretend to know the timeline. The reporting doesn’t give one, and Feldman’s comments stop at “nearing,” which could mean a lot of things coming from a chipmaker with an obvious interest in fast inference. Take the prediction as a direction, not a date.
But the direction has practical consequences for anyone writing bot code today. Start treating the model as a swappable backend rather than a hardcoded dependency. If your agent logic calls one specific API with one specific schema, you’re building something that can’t move on-device later. An abstraction layer over inference, with a clean tool-calling contract underneath, costs you a day now and saves you a rewrite.
Pay attention to latency budgets too. Cloud-only assistants trained users to accept a beat of delay. Local inference removes that beat, and once people feel the difference, the slow path starts feeling broken. Instrument your round trips now so you know which parts of your pipeline are actually the problem. In my experience it’s rarely the model — it’s the three sequential tool calls you never bothered to parallelize.
And get comfortable with small models. Not as a compromise, but as a design constraint that forces clarity. A 3B model with good tools and tight prompts beats a huge model with sloppy orchestration more often than anyone wants to admit.
The quiet version of the future
If Feldman is right, the win condition isn’t an assistant that does everything. It’s an assistant nobody thinks about. No wake-word anxiety, no latency tell, no awareness of where the computation happened. Just a thing in your house that works.
Which is a strange goal for an industry that loves demos. The best outcome here is software so unremarkable you’d put it next to the microwave and forget it’s there. That’s the target I’m building toward, and the architecture decisions that get you there start now, not when the hardware ships.
🕒 Published: