The dystopian take on AI interviewers is the wrong one. An AI asking you follow-up questions about your code is a fairer judge than the half-distracted engineer who skimmed your resume in the elevator and decided within ninety seconds whether you were “a fit.” I’ve sat on both sides of that table. The human version was never the gold standard we’re now mourning.
HackerRank’s AI interviewer, Chakra, has conducted more than 500,000 interviews, with Snowflake, Snorkel, and Capgemini among its early testers. That’s not a pilot program. That’s a deployed system operating at a scale most of us will never touch with anything we build.
What the system actually evaluates
The part that matters to me as someone who builds bots for a living isn’t the hiring angle. It’s the evaluation design. Chakra doesn’t just score the answer. It assesses critical thinking, judgment, and what HackerRank calls “AI fluency” — how well a candidate works alongside AI assistance rather than around it.
The interviews resemble real-world tasks: code repositories, AI assistance available, follow-up questions probing the candidate’s problem-solving approach. That last piece is the architecturally interesting one. A single-shot grader is easy. You compare output to a rubric and emit a number. An interviewer that asks follow-ups has to:
- Maintain state across a conversation that branches unpredictably
- Decide which thread is worth pulling on and which is noise
- Generate questions that discriminate between a candidate who understood the solution and one who got lucky
- Produce a defensible, quantitative assessment at the end of an unstructured exchange
Anyone who has tried to build a conversational agent that does more than retrieve and respond knows how much of that is hard. The follow-up question is where most agent designs fall apart. It’s easy to write a prompt that asks “can you explain your approach?” It’s much harder to build something that reads a candidate’s reasoning, spots the gap, and targets it.
Why process beats output
The shift from grading what you solved to grading how you solved it is the right call, and not only for hiring. It’s the same problem we have with evaluating agents.
If you’ve built anything agentic, you’ve hit this wall. Your bot produces the correct final answer, so your test passes. But it took eleven tool calls when three would do, it hallucinated a function name and recovered by accident, and it never checked whether the file it was editing existed. The output was right. The process was a mess. Output-only evaluation hides all of it.
Chakra’s approach — watch the work, ask about the decisions, score the reasoning — is a direct answer to that. Which means the design is worth studying even if you never hire a single engineer. It’s a template for building evaluators that look at trajectories instead of endpoints.
The AI fluency question is the honest one
Of all the things being measured here, AI fluency is the one that strikes me as overdue. The work has changed. I spend a meaningful chunk of my day deciding what to hand to a model, how to frame it, when to reject what comes back, and when to just write the thing myself. That’s a skill. It’s a learnable one. And until recently there was no credible way to assess it in an interview, because most interview formats banned AI assistance outright and then hired people into jobs where AI assistance is everywhere.
Giving a candidate a repository and an AI assistant and watching what they do is closer to the actual job than any whiteboard exercise I’ve ever administered. Not perfect. Closer.
What I’d watch for
None of this makes me uncritical. Evaluation systems encode the values of whoever built them, and a model scoring “judgment” is scoring a pattern it learned from somewhere. I’d want to know how those scores are validated against real job performance, how disagreement between the AI’s assessment and a human reviewer’s gets resolved, and what happens to a strong candidate whose reasoning style simply doesn’t match the pattern.
Those are the same questions I ask about any evaluator I build. Where’s the ground truth? What’s the failure mode? Who checks the checker?
But the complaint I keep hearing — that it’s dehumanizing to be interviewed by software — assumes the alternative was humane. Half a million interviews in, the more useful conversation isn’t whether this should exist. It’s what the design teaches the rest of us about building systems that judge work rather than just checking answers. That’s a problem I’d like solved, and not just for hiring.
🕒 Published: