Participants loved using the AI mock interview coach, but was it actually getting them ready for job interviews?
Authors: Maya Welch and Ankita Ghoshal
This summer, The Bike Shop, an AI research lab at MIT and the University of Chicago, ran an AI for Social Impact Fellowship in partnership with Anthropic and NextLadder Ventures. The fellowship is built on a different idea of what AI can be: augment what people do rather than replace it. Over a 10-week sprint, the fellows partnered with frontline organizations to build an AI product that raises the ceiling of impact.
Partner
The Center for Employment Opportunities (CEO) is the largest reentry employment organization in the United States, helping formerly incarcerated people find and keep jobs across more than 30 cities. After years or decades away from the job market, the job interview is one of the highest-stakes moments in a participant’s reentry journey. CEO's expert job coaches prepare participants for it, but they can't be available around the clock for all the practice rounds that someone might need. So last year, CEO launched “ACE,” a phone-based AI coach for mock interviews. A participant calls a number, answers real interview questions out loud, and receives immediate feedback.
Problem
Participants, it turned out, really liked talking to ACE. But a participant giving positive feedback on a tool and the tool actually better preparing them for a job interview are two very different things. The only way for CEO staff to check how the AI coach was behaving was to read 8- to 20-page transcripts by hand. Reading these transcripts one-by-one made it next to impossible to identify consistent failure modes of the AI coach.
But there were plenty of ways the AI coach could get things wrong and do real harm. If a participant recently home from prison overshares their conviction history and the AI coach praises the answer, they may repeat it in a real interview and not get the job. For the population CEO serves, confidence in the product’s behavior is imperative.
Solution
Evaluations (‘evals’) are a systematic, repeatable way to measure what an AI actually does against a clear definition of what it should do. In our case, that meant a second AI, an "LLM-as-judge," scoring every coaching call against a rubric for good coaching.
Building that rubric began with sitting with CEO's job coaches and reading hundreds of transcripts by hand. This produced a list of 39 criteria. But not everything coaches cared about could make it into our rubric. For example, with a dimension like “compassion for nerves,” coaches can read the energy of someone sitting in front of them. But our AI eval was reading a transcript, which doesn’t contain that kind of information.
We eventually pared the initial exhaustive list down to 10 rubric dimensions. Some are about longform conversation: "Conversational flow" catches a coach that interrupts repeatedly. Some are to protect the vulnerable population: the PII judge checks whether the coach reminds a participant not to share sensitive details.
Then, we asked twelve job coaches to grade 54 call transcripts on each of our final dimensions of coaching quality. Each dimension was labeled pass or fail. We trained graders on the rubric, gave them simple quizzes to ensure they understood the definitions, and built a user-friendly data labeling platform that let them collect evidence before delivering their verdicts.
But, we were in for a surprise: when we measured how much the job coaches agreed when grading the same transcript on the same dimensions, agreement turned out to be worse than chance on several dimensions.
That wasn't bad or inattentive labeling. When we clustered the disagreements, it turned out that the coaches fell into distinct camps: strict graders passing only excellent AI coach behavior while lenient graders tended to fail only the egregious cases of the AI coach getting it wrong.
Once we looked at the agreement of coaches in, say, the strict camp, agreement doubled! This pattern revealed that the coaches were disagreeing about where the bar for a certain behavior lies. So we stopped trying to get the coaches to agree and rewrote the rubric so agreement was possible. We split up rubric dimensions that were doing multiple jobs, and set each bar at a place a grader could consistently score against.
We then built an LLM judge for each rubric dimension. Each judge was an AI model prompted to score transcripts for a specific quality mimicking as closely as possible how CEO’s job coaches had graded the transcripts. To get the judges to behave like the human coaches, we split our set of labeled transcripts into three parts. We used a handful of examples in the prompt to teach the AI what “good” and “bad” behavior looked like for that specific rubric item. We then used a development (“dev”) data set to test how well our judge was doing at matching the human labels and iterated the prompt until we couldn’t get any further improvement. Importantly, we kept one set of transcripts that we didn’t use during the improvement step so we could get a final verdict on how well the judge was performing on data it had never seen before.
We now had nine LLM judges that scored every real call in production and fed a live dashboard we could monitor in real-time.
Impact
When we scored all 825 historical calls from the pilot, we saw three clear failure patterns: ACE was not succeeding at creating a true dialogue with the participant, participants rarely practiced an answer again after getting feedback from ACE, and ACE had a tendency to overpraise weak participant answers.
We designed a new version of the coach to address those weaknesses through a more rigorous practice loop. Now when a participant gives a weak answer, the coach frames the retry as the default rather than a choice, and nudges them to try again if they decline.
Our eval suite of LLM judges allowed us to systematically test the new version of the coach with CEO participants. We ran a 50/50 A/B test on the live phone line for 150 calls: Arm A was identical to the original coach. Arm B was the redesigned coach. Both were scored by the same nine LLM judges.
This time we were in for a positive surprise: The new voice coach we had built (“Arm B”) improved on all dimensions we targeted.
- The new AI coach got participants to practice! Now 96% of participants tried giving an improved interview answer after getting feedback on their first attempt from the AI coach, compared to only 46% with the original coach.
- The new AI coach did better at limiting the load, speaking in shorter turns and not overwhelming the participant with feedback. Now 98% of the time when the AI voice coach gave feedback it was successfully limiting the load, compared to 44% of the time with the original coach.
- The new AI coach was able to scaffold: It helped participants build answers by drawing on their own experience 68% of the time, compared to 10% of the time before.
- The new coach was able to make it a dialogue 60% of the time. It asked participants to reflect on their own answers before jumping into giving feedback. Before that happened exactly – never!
Insight
When we first interviewed CEO's coaches and leadership, the main feedback we heard was that the coach felt "too nice." We could have solved that by asking the coach for harsher feedback and it might have worked to solve that one issue. But, that approach becomes a game of whack-a-mole. Every prompt change can create a new "biggest issue" for the next prompt to address, with no systematic data on how behavior is changing between versions.
We asked the CEO team to describe the ideal coach rather than evaluate the one they had. That is how "makes it a dialogue" became a key dimension. The AI coach was never prompted to ask reflective questions, but coaches told us when a participant seems checked out, or isn’t really understanding the feedback, they may ask the participant to switch roles and think about what an employer wants to hear. For our AI coach, that translated to the coach asking ‘what number out of 10 would you score that response?’ or ‘what do you think an employer would think of that answer?’ Getting participants to reflect on their own answers went against what people expect from an AI agent, and turned out to be essential in making the experience productive and engaging.
The success of a "squishy" eval suite comes down to dimensions that are measurable and that a human can actually label. Splitting dimensions that were doing multiple jobs took careful reading of where labelers diverged. Our first attempt at the over-praise problem was an "honest but kind" dimension, but labelers couldn't score it. What do you do with feedback that is kind but not really accurate? We split it into "feedback is correct" and "kind delivery," then dropped "kind delivery" entirely because the underlying models default to kindness, and there weren't enough real failure cases to train a judge against.
"Feedback is correct" ran into its own problem: how good is good enough to pass? We split it again into a high bar, where only nuanced feedback matched to the participant's skill level passes, and a low bar, where only outlandish over-praise for a weak answer fails. Our final feedback judge was the low bar. Clear excellence was too rare in the corpus to train judges on, and for this population, it is more useful to CEO to pinpoint truly bad coaching than to grade for perfection.
What It Takes to Make This Work and Keep It Running
Transparently, this project took two dedicated fellows working full time for 10 weeks, hours of frontline staff time to label and re-label transcripts, and a heap of Claude tokens. Going forward the token cost is small and predictable: roughly $0.10–0.20 per scored call, plus ~$20 a month for server hosting.
To keep it running, coaches will need to label a sample of production calls on a regular cadence. To apply these judges to the same AI coach working with a different population, transcripts will need to be labeled again, this time to confirm the judges still agree with humans in the new context. New dimensions of quality (we suggest adding an "annoyingness" judge to make sure Ace doesn't get too pushy) will need to undergo the same rigorous process: refining precise definitions of pass/fail, then iterating the judge prompt against human labels until the verdicts match.
This is real ongoing work, and it’s worth it: First, the process gets easier each time as the org builds its build-measure-iterate muscle. Second, an eval system is less work than the alternative of reading every transcript as users scale. It’s also less dangerous than what might really happen if only some get read, whenever someone has time. Third, and most importantly, a well-calibrated eval suite gives you real levers to protect users and ensure they interact with a useful product.
In the non-profit ecosystem, where budget constraints are real and problems are being solved from a place of care, there is a particular responsibility to make sure what we build and deploy really works, and that vulnerable users are protected. These constraints are why good measurement matters. An un-evaled product spends scarce budget on something that may not work, and the people who bear that cost are the ones least able to absorb it.
What Comes Next
For CEO, this eval suite is not the finish line. It is infrastructure for continuously improving the coach and for eventually answering the more important question: whether better AI coaching helps participants enter interviews more prepared, more confident, and better able to tell their own stories.
Additional Reading
If you enjoyed this content, follow along as we dive into the nitty gritty details of all the projects from the summer fellowship in our blog series. The first blog explores the value of eval forward product development, and can be found here.
We'd love to hear your thoughts! Reach out to Maya (mayaw888@mit.edu) and Ankita (aghoshal@mit.edu) with any questions or comments.
