Evidano home
Published in the American Journal of EvaluationResponsible AI

How Good Is an AI Interviewer? A Framework for Judging AI-Mediated Interviews in Evaluation

Evidano10 min read
Ali Safarnejad, Senior Evaluation Specialist

The opinions expressed here are the authors’ own and do not necessarily reflect the positions, strategies, or opinions of the organizations they work for.

Evaluators are starting to use large language models not just to analyse interviews but to conduct them. Early pilots, from adaptive survey agents to interview bots used in after-action reviews, show that an AI can hold a fluent conversation. But fluency is not the standard an evaluation interview is judged by. Sound evaluative judgement depends on interview data that are accurate, neutral, aligned with the questions the evaluation set out to ask, and probed far enough to be credible. Until now, evaluators have had no systematic way to check whether an AI interviewer meets that standard.

That gap is the focus of “Assessing AI-Mediated Interviewing Quality: A Theory-Grounded Comparison of Models for Evaluation Data Collection,” published in the American Journal of Evaluation in 2026 by Ali Safarnejad of UNICEF and Hippolyte Lefebvre of University College Dublin. The authors translate evaluation, qualitative interviewing and validity theory into a framework of seven quality dimensions and eight assessment measures, then use it to test six generative AI models, five prompted-only and one fine-tuned, in two realistic evaluation interview scenarios.

We spoke with Dr. Ali Safarnejad, Senior Evaluation Specialist, to learn more about what the comparison showed, where the models fell short, and what evaluators should put in place before using AI to collect interview data.

About the article

Research question

How can the methodological quality of AI-mediated interviewing be assessed in evaluation contexts, and to what extent do leading LLM-based interviewers meet these quality expectations across realistic evaluation scenarios?

Key finding

Most models reliably caught incomplete and irrelevant answers and probed them, and the fine-tuned model matched the human benchmark on 11 of 16 measurements, nearly 70% and almost double the next-best model. But performance was highly context-dependent: no model kept question accuracy in both scenarios, and all of them struggled with neutrality and with clarification probing. The authors conclude that AI interviewers can collect evaluation data only with significant human oversight.

Why this research matters

Evaluation interviewing has long faced quality threats that AI might ease: interviewers who behave inconsistently despite training, protocols applied differently across sites, the cost of skilled interviewers who can probe well, and the difficulty of supervising teams spread across many locations. An AI interviewer promises consistency and scale. The authors observe, however, that the studies published so far mostly assess technical capabilities such as fluency, turn-taking and adaptation, rather than whether the resulting data meet the methodological standards evaluation depends on.

This study shifts the question from “can AI hold an interview?” to “is the interview good enough to base evaluative judgements on?” It grounds every measure in established theory, from Guba and Lincoln and Patton to Messick and Scriven, so that an evaluation team can judge an AI interviewer against the same expectations it would apply to a human one, and explain that judgement to commissioners. Ali Safarnejad explained why the authors took this approach: “AI models evolve so rapidly that a simple leaderboard comparison is outdated the moment it is published. More importantly, conversational fluency is not methodological rigor. By anchoring our assessment in foundational evaluation and validity theory, we wanted to move the conversation from whether a bot sounds human to whether the data it collects can actually support credible evaluative judgment. A theory-grounded framework gives evaluators a durable benchmark that applies whether they are testing today’s LLMs or the next generation of voice-assisted avatar agents.”

Key findings from the article

Three findings stand out because they show both what AI interviewers can already do and where evaluators need to stay alert.

  • Overt problems were caught; subtle ones mostly were not. Across both scenarios, incomplete answers typically triggered completion probes and irrelevant answers triggered steering probes. Subtler problems were another matter. In the second scenario the simulated answers contained a fabricated phrase (“match boxing”), an ambiguous term (“Pacificity”) and inconsistent figures on girls and boys pursuing STEM fields: only one model probed the fabricated phrase, two probed the ambiguous term, and only Gemini 2.5 Pro picked up the numerical inconsistency.
  • Fine-tuning helped procedure, not judgement. A Gemini 2.5 Flash-Lite model fine-tuned on 20 annotated, simulated interview transcripts, built from the structure of UNICEF evaluation interview guides, matched the benchmark on 11 of 16 measurements across the two scenarios, compared with 7 for the next-best model and 7 fewer than its own un-tuned base model. Its gains were strongest in the scenario that resembled its training data, and it still fell short on neutrality and on detecting quantitative inconsistencies, which the authors read as evidence that fine-tuning improves in-domain procedural compliance but does not instil evaluative judgement.
  • Neutrality and question fidelity depended on context. In the first scenario, every model compromised neutrality by folding optional follow-up prompts from the interview guide into its first asking of a question, narrowing the answers a respondent might give. With a strictly neutral guide in the second scenario, four of six models stayed neutral, but two still introduced leading elements of their own. No model kept question accuracy in both scenarios; the most common failures were turning closed questions into open ones, repeating questions already answered, and merging separate questions.

Taken together, the results show that conversational skill and methodological rigour are not the same thing. The models balanced depth and coverage very differently, with additional probes ranging from 11% to 50% of all questions asked, and the authors stress that a model that performs well in one interview context cannot be assumed to perform well in another. Ali Safarnejad reflected on the results: “Two things stood out sharply for me. First, how clearly fine-tuning separated procedural compliance from actual evaluative judgment: training a lightweight model on just 20 curated transcripts nearly doubled its benchmark matches and made it much better at pacing and formatting, and yet it still failed to spot basic quantitative contradictions when the interview context shifted. Second, the models universally struggled with neutrality. Even with explicit instructions, even after fine-tuning on strictly neutral examples, the models repeatedly folded optional follow-up examples into their initial questions or introduced subtle framing of their own. The insight here is that the current LLMs are wired to be helpful conversational partners, which often works directly against the methodological restraint required of a neutral interviewer.”

Methods and research approach

The study used a convergent mixed-methods design, developed in phases from early experiments in 2024 and a 2025 pilot of an AI interview agent in an after-action review of UNICEF’s emergency response to a typhoon in Viet Nam, through to data collection in late 2025. The authors drew seven quality dimensions from the literature: construct validity, responsiveness, neutrality, accuracy of representation, comprehensiveness, depth and procedural integrity. They operationalised these into eight measures: six rated as binary Yes/No judgements (question accuracy, flow adaptation, question neutrality, response accuracy assurance, formatting and singular questions) and two calculated from the transcripts (an effective quota utilisation index and the rate of additional probing).

Six models were tested: Gemini 2.5 Flash-Lite, Gemini 2.5 Pro, GPT-5, Claude Sonnet 4.5, DeepSeek V3.1 and the fine-tuned Gemini model, all with identical prompt instructions and interview guides, and under standard conversational configurations (temperature of 1.0 and Top-P of 0.95 for the Gemini models, and default web-portal settings for the others) to support natural conversational flow rather than being artificially constrained for deterministic accuracy. Each model conducted one text-based interview in each of two fictive scenarios, written from scratch so that no model could draw on memorised evaluation reports. Scenario 1 was a constrained qualitative interview with more guide questions than fit a 23-question limit, eight “bait” responses and optional steering prompts; Scenario 2 was a standard intramethod interview of 13 questions and probes, including numerical, ranking and Likert items. The author played the respondent throughout, answering from a fixed script, and idealised benchmark transcripts were scored with the same measures as a human reference point.

The authors are candid about the study’s limits. All qualitative measures were rated by a single appraiser, and binary ratings can hide gradations in performance. Only six models were assessed, with several open-weight and smaller models (including three Meta Llama variants) excluded during piloting due to poor instruction-following or context-window limits that could not handle the cumulative token demands of multi-turn state management. Each model conducted one interview per scenario, the interviews were text-based only, and decoding settings such as temperature were not varied, so some reasoning failures may partly reflect the conversational settings used. In addition, the supervised fine-tuning dataset, while effective for behavioral alignment via parameter-efficient LoRA training, was intentionally modest (20 multi-turn transcripts averaging 46 turns each) and focused on qualitative structures, which bounded its generalization to quantitative items.

Broader implications

For evaluation offices, the study replaces a yes-or-no debate about AI interviewers with a question of fit. The authors see a role for AI interviewing in large or geographically dispersed evaluations in resource-limited settings, where consistent application of instructions can reduce interviewer variance; in rapid evaluations where timeliness is paramount; and where the main concern is catching incomplete or off-topic answers. They are equally clear about the oversight this requires: a post-interview review of transcripts for leading questions, human verification of any quantitative responses, and checks that closed questions stayed closed and open questions stayed open. Ali Safarnejad was clear about where evaluators must stay in control: “Evaluators cannot treat AI interviewing as a ‘set-and-forget’ automation pipeline. Our findings point to three non-negotiable oversight mechanisms. First, you need post-interview transcript audits specifically checking for neutrality violations, because models frequently slip leading assumptions into their questions even when the interview guide is neutral. Second, you must verify question-type fidelity, as models routinely turn closed-ended or ranking questions into open-ended discussions, which undermines the logic of mixed-methods aggregation. And third, any quantitative or numerical data collected during an interview requires strict human validation, as all but one reasoning-specialized model failed to catch basic mathematical inconsistencies.”

The findings also carry a lesson for how interview guides are written. Because the models could not reliably repair a poorly ordered guide, and sometimes introduced leading elements even when the guide was neutral, the authors recommend that guides intended for AI administration be strictly neutral, free of leading examples, and logically sequenced from the outset. Asked for advice, Ali Safarnejad said: “Start by treating the AI as a junior data collector that follows structure well but lacks contextual judgment, and match it to the right evaluation problem. AI interviewers are genuinely valuable for rapid assessments, emergency after-action reviews, or large, multisite evaluations where human interviewer inconsistency is a major threat and the primary goal is capturing broad qualitative input and probing incomplete answers. Before deploying, invest heavily in your interview guide: make it strictly neutral, strip out optional leading examples that the model might deploy prematurely, and sequence your questions logically (e.g. knowledge before experience, general before specific) because you cannot rely on the model to fix a poorly structured guide on the fly.”

It is equally important to be clear about what the study does not show. It does not suggest that AI interviewers can run evaluations without people, and it does not establish how the same models would perform with real respondents, in other languages, or with later model versions. Its contribution is a replicable way of judging when an AI interviewer is good enough for a given evaluation, and what oversight it needs.

About Ali Safarnejad

Ali Safarnejad, Senior Evaluation Specialist

Ali Safarnejad, PhD, is currently a Senior Evaluation Specialist (Data Science) with the Asian Development Bank, based in Manila. At the time of the research on the AI interviewers, he was an Evaluation Specialist at UNICEF. Dr. Safarnejad leads evaluations across humanitarian and development contexts, using technology and advanced analytics in development programme evaluations and emergency response after-action reviews, with a strong interest in improving the efficiency and quality of evaluative evidence through thoughtful use of AI and machine learning.

About Hippolyte Lefebvre

Hippolyte Lefebvre, Assistant Professor in Management Information Systems at University College Dublin

Dr. Hippolyte Lefebvre is an Assistant Professor in Management Information Systems at University College Dublin. His research, which focuses on improving Data and AI management in organizations, is featured in outlets such as the European Journal of Information Systems, Electronic Markets and Harvard Business Review. Hippolyte lectures about Data and AI Management at UCD Michael Smurfit Graduate Business School. Before joining academia, Hippolyte worked as a Monitoring and Data Analyst for the United Nations, supporting 11 countries across Asia and Africa.

About the authors

  • Ali Safarnejad (Senior Evaluation Specialist (Data Science), ADB; formerly Multi-Country Evaluation Specialist, UNICEF)
  • Hippolyte Lefebvre (School of Business, University College Dublin, Ireland)

Read the article

Read the full research article, “Assessing AI-Mediated Interviewing Quality: A Theory-Grounded Comparison of Models for Evaluation Data Collection,” published in the American Journal of Evaluation in 2026.

Full citation

Safarnejad, A., & Lefebvre, H. (2026). Assessing AI-mediated interviewing quality: A theory-grounded comparison of models for evaluation data collection. American Journal of Evaluation. doi.org/10.1177/10982140261476953

View the publication

We are grateful to Dr. Safarnejad for sharing this perspective on the study and for this contribution to the responsible, transparent use of AI in evaluation.

Research, accuracy and responsible AI

How Evidano's output has been evaluated, how its models were built, how your data is handled, and how to disclose or cite AI use in your own work.

  • Resource

    Human vs AI: validated accuracy benchmarks

    Four head-to-head comparisons against expert human analysis — 371 interview transcripts with Arizona State and Penn State, 298 evaluation reports, 224 open-ended survey responses with UT Austin and Wichita State, and a 160-response manual-coding review.

  • Resource

    How Evidano’s models were trained

    Model provenance for research-ethics review: a provenance-tiered curated corpus, and customer uploads, prompts and outputs excluded from training entirely.

  • Resource

    Data security for qualitative research

    Encryption in transit and at rest, customer-controlled deletion, regional storage, and SOC 2 Type 2, GDPR, HIPAA-ready and CCPA/CPRA compliance.

  • Resource

    How to cite and disclose Evidano

    Citation formats in APA, MLA, Chicago, Harvard and Vancouver, methods-section wording per workflow, and an AI-policy review of 41 journals across 8 disciplines.