AI made the polished resume and rehearsed answer cheap to produce. It exposed a more fundamental problem: many companies never defined what good looks like for the role.
The AI interview process most companies run today still relies on signals that were easy to grade: coding exercises, take-home projects, polished narratives. AI can now produce those signals on demand.
Allowing AI in an interview is the easy decision. The harder one is whether your AI interview process can tell someone who knows how to use AI from someone who knows how to do the job.
One Anthropic Manager Changed the Interview. The Evaluation Problem Remains.
As reported by The Next Web, Felix Rieseberg, an engineering manager at Anthropic, wrote in late October that he has dropped the coding interview from his team’s hiring. In his view, a small React challenge no longer makes sense because Claude is now better at that kind of task. He described small coding challenges as having been a weak but useful predictor of success on his teams.
For now, Rieseberg scans resumes and portfolios for small prototypes that show taste, empathy for users, and principled thinking.
This is one manager’s decision for his team, not an announced Anthropic-wide policy. Rieseberg also said he has not yet worked out how to interview people who build well with AI.
That admission is the most useful part of the story.
Permitting AI in an interview is one thing. Judging work produced with it is another. Only the first has an easy answer.
The Policy and the Behavior Have Diverged
Karat’s Engineering Interview Trends 2026 survey covered 400 engineering leaders across the US, India, and China.
Karat sells interview services and doesn’t state when it fielded the survey, so treat the figures as a directional signal rather than a precise measurement. Even so, the contradiction is hard to ignore.
The policy and the behavior have diverged. Banning AI doesn’t restore the signal in your AI interview process if candidates are using it anyway. It leaves the interviewer guessing.
The Resume Is a Weaker Signal. The Interview May Be, Too.
At Hoola Hoop, we spend a lot of time helping companies qualify CTO and CPO candidates. AI has made a polished resume a weaker signal of capability. Experience, scope, and career progression still matter, but the presentation itself tells us less than it used to.
A recruiting firm leader recently told me that cutting through this noise has become the hardest part of his job. He has also seen companies hire people who leaned on AI during the interview, only to discover that they weren’t a fit once they were in the role.
That’s one person’s experience, not a trend line. But it points to two distinct failures.
Upstream, a polished resume and a polished answer are no longer sufficient evidence of capability.
Downstream, someone who performs well with AI in the room may struggle to operate without it, or to supervise it once it is doing real work.
Any AI interview process needs the same remedy for both problems: test judgment under conditions that resemble the job.
Define Good Before Your AI Interview Process Starts
Jayesh Govindarajan, who leads the roughly 700-person AI engineering organization behind Salesforce’s Agentforce, described how he screens product managers on Aakash Gupta’s Product Growth podcast.
He looks for agency, a habit of teaching an agent to do a task after doing it by hand a few times, healthy skepticism about outputs, craftsmanship, and the ability to set the bar.
That last one is the most transferable.
A candidate who can define what good looks like can also help create the evals and test cases that determine whether an agent is doing good work.
The same principle applies to the interviewer.
Too many hiring loops measure what is easy to grade because nobody has defined what the role actually requires. AI has made that shortcut harder to defend.
Before the first interview, the hiring leader should be able to answer:
Write it down. Share it with the candidates.
A written definition gives both sides a clearer basis for evaluating the role and makes the whole AI interview process easier to run. It also exposes quickly whether the hiring leader has done the work.
Three Rounds That Test Judgment in an AI Interview Process
Our view is that the ability to define, evaluate, and take responsibility for work is becoming more valuable as AI takes on more of the execution.
Critical thinking is part of that, but it’s too generic to be useful on its own. The interview needs to make it observable.
These three rounds provide a practical starting point for an AI interview process.
Round 1: Judgment Under Ambiguity
Give the candidate a messy problem with incomplete information.
Look for how they frame the problem, identify constraints, challenge assumptions, and decide what matters. What are they optimizing for? Which information do they need? And what would change their mind?
Don’t reward a polished answer. Reward the quality of the thinking that produced it.
Round 2: Working With Agents
Let the candidate use AI. Then ask them to justify their decisions.
What did the agent get wrong? Which parts did they change, and why? What would they not delegate? How do they know the result is good enough?
The ability to generate an answer is becoming less distinctive. The ability to recognize a bad answer, understand its consequences, and correct it is not.
Round 3: A Live Conversation About Decisions
Finish with a live conversation, with the hiring leader in the room, about something the candidate built, led, or changed.
Go beyond the outcome. Ask what they got wrong, what they would do differently, and how they knew when to change direction.
Then change the facts. Cut the budget in half, remove a key person from the team, or add a skeptical stakeholder, and ask how the decision would change. A rehearsed story or an AI-prepared narrative holds up until the facts move. Real understanding adapts.
You’re looking for depth of understanding, ownership, and the ability to reason in real time. Not just what they produced, but how they think about the work.
This round also tests the hiring leader. It only works if the leader knows what good looks like, because someone has to recognize a strong adaptation when they hear one.
What This Looks Like for VP and C-Level Roles
For more senior searches, we encourage companies to use a case study. In practice it combines the second and third rounds above, and it is the heart of the AI interview process we recommend for VP and C-level hires.
The case is a real executive-level problem. The candidate’s job is to tell us how they would go about solving it. It isn’t a consulting engagement, and we aren’t looking for a free solution. What matters is the approach: how they frame the problem, what they would investigate first, what they would decide, and what they would leave open. Depending on the role, one case can test technical depth, leadership judgment, and critical thinking at the same time.
We expect candidates to use AI to help them, and they typically share their thinking in writing. Then they come back for a one-hour working session with a group from the company, where the approach gets challenged in depth.
What the Working Session Reveals
That session is where the signal is. Anyone can bring a polished document. Defending each part of it live is harder, and it quickly becomes clear how deep a candidate can go in a particular area and where the limits of their experience are. Those limits don’t necessarily disqualify anyone, but it’s far better to find them before the offer than after the start date.
Because the session is a group setting, it also tests culture fit. You see how the candidate takes challenge from several people at once, how they handle the edge of their own knowledge, and whether they collaborate or get defensive. That is hard to learn from a resume or a one-on-one interview, and it matters most in roles where the person will shape the team around them.
It also addresses the take-home problem. Karat reports that the hiring signal from take-home projects degrades fastest under AI. Here the written work is only the starting point, because the candidate has to explain and defend it live.
Boards Hiring a CTO Face the Same Problem
The same problem exists one level up.
Boards and CEOs hiring a CTO or CPO often lean on pedigree, a polished narrative, and references. Those are useful signals, but they don’t tell you whether someone can lead an organization through the changes AI is bringing.
A board that hasn’t defined what good looks like for the role faces the same problem as an engineering manager reading flawless resumes.
The hiring process itself is revealing. If a company can’t articulate what it needs from its next technology leader, it probably hasn’t aligned on what the technology organization needs to become.
This is where our executive search and job description work at Hoola Hoop tends to start: before anyone is interviewed.
It’s also why the AI interview process is worth examining in a product and engineering assessment. How a team hires usually tells you a good deal about how it ships.
Two Questions to Take Into Your Next Hiring Loop
Before your next round of interviews, start with two questions about the role and your own track record:
- What leadership qualities, role-specific hard skills, and level of critical thinking do we expect from someone in this role?
- What have we learned from past hires who turned out to be a poor fit or a great one, and which attributes made the difference?
If those questions are difficult to answer, the problem isn’t AI.
It’s the bar you’re using to hire.
If you’re rethinking how you hire or what your next CTO, CPO, or VP needs to deliver, Hoola Hoop’s Product and Engineering Assessment can help establish that baseline.
Rewriting the Bar for a CTO, CPO, or VP Hire?
Book a 30-minute conversation to talk through what good looks like for your next senior hire.
