Research teams have tested Evidano’s AI-powered qualitative analysis against their own human coding, and published the results.
In a peer-reviewed paper, Arizona State University and Penn State researchers used Evidano to analyze 371 transcripts.
"The benefit of using Evidano was its capacity to generate explanations and justifications for coding decisions. This provided transparency that strengthened our human-in-the-loop process"
Use This AI Approach for:
Interview analysis • Transcript coding • Qualitative research • Thematic Analysis • Grounded Theory
The UN Evaluation Group used Evidano to analyze 298 evaluation reports using a realist evaluation approach.
"The AI tool achieved an accuracy of 92 per cent for coding the evaluation reports ... [with] a total of 132 CMO configurations were identified"
Use This AI Approach for:
Literature reviews • Meta-evaluations • Systematic reviews • Report synthesis • Document screening • Evidence mapping
In a peer-reviewed paper, University of Texas at Austin and Wichita State researchers used Evidano to check their own thematic analysis of 224 open-ended survey responses.
"...were supported by the AI-assisted analysis with only minor differences in wording"
Use This Approach For:
Survey open-ends • Analytic triangulation • Second-coder checks • Blind-spot and bias review
UNICEF compared human and Evidano coding of 160 management responses.
"Overall, the Evidano platform did a very good job in identifying key themes in the data and also in summarizing these key themes in prose"
"provided a wide range of examples to illustrate the themes, pulling directly from recommendations and actions"
Use This Approach For:
Survey open-ends • Social media comment analysis • Feedback coding • Management response analysis • Thematic analysis at scale
The wider benchmark
Evidano’s results sit inside a wider benchmark: 34 published papers (2024–2026) that measured AI-generated codes and themes against human coders, covering general-purpose models and the AI features of other QDA packages as well as Evidano. The chart averages one headline result per study for each tool. The method and every study, with its source, are under the chart, and each study is summarised on the tool comparison.
Average of 92.0% coding accuracy in Brouek et al., κ = .959 inter-rater reliability in Firetto et al., and 8 of 9 human-coded themes (88.9%) supported in Abner & Mee.
Sample: 5,133 evaluation units · Sources: Brouek, Mansour & Goh (2024), Firetto, Murphy, Yan & Tang (2025), Abner & Mee (2026)
Average of six studies: weighted κ = .755 for DeepSeek V3 in Bennis & Mouaffaq, mean code-tuned κ = .529 in Matveyenko et al., κ = .53 in Camporese et al., κ = .68 in Mizumoto & Teng, mean κ = .69 in Na & Feng, and 15 of 20 human themes (75%) identified in Yu et al.
Sample: ≈288,718 evaluation units · Sources: Bennis & Mouaffaq (2025), Matveyenko et al. (2026), Camporese, Massacci & Gong (2026), Mizumoto & Teng (2025), Na & Feng (2025), Yu et al. (2026)
Average of one headline result from each of 24 studies, from κ = .10 across all excerpts in Wang et al. to all 7 human themes matched in Bang et al. Sixteen of the 24 results are kappa values; the rest are F1, percent agreement, or the share of human codes or themes matched.
Sample: ≈312,794 evaluation units · Sources: Zhang et al. (2025), Bennis & Mouaffaq (2025), Dunivin (2025), Cui et al. (2026), Hill et al. (2026), Matveyenko et al. (2026), Ayik, Gu, Zan, Kim & Kim (2026), Jeong et al. (2025), DeRegnaucourt et al. (2026), Liu et al. (2025), Wen et al. (2026), von der Heyde et al. (2025), Turner et al. (2026), Wang et al. (2025), Bang et al. (2026), Yang & Ma (2025), Qiao et al. (2025), Castellanos et al. (2025), Koh et al. (2025), Ren et al. (2026), Habib et al. (2026), López-Pérez et al. (2025), Camporese, Massacci & Gong (2026), Na & Feng (2025)
Average of 64.0% segment correspondence in Schueller et al., 50.0% exact-theme correspondence in Ayik et al., κ = .38 (the best of the reported prompts and overlap thresholds) in Boettinger et al., and F1 = .80 in a pilot by Červeňová & Demkanin.
Sample: 1,789 evaluation units · Sources: Schueller, Trettin & Huber (2026), Ayik, Gu, Zan, Kim & Kim (2026), Boettinger, Bürgel & Bartsch (2026), Červeňová & Demkanin (2025)
Average of ten studies: κ = .420 in Zhang et al., weighted κ = .780 in Bennis & Mouaffaq, κ = .350 in Hill et al., F1 = .831 under structured prompting in AlGhamdi, mean κ = .71 in Misgav et al., mean κ = .43 in Koh et al., mean F1 = .63 in Ren et al., κ = .25 in Habib et al., κ > .76 in López-Pérez et al., and κ = .30 in Camporese et al.
Sample: 7,835 evaluation units · Sources: Zhang et al. (2025), Bennis & Mouaffaq (2025), Hill et al. (2026), AlGhamdi (2026), Misgav et al. (2026), Koh et al. (2025), Ren et al. (2026), Habib et al. (2026), López-Pérez et al. (2025), Camporese, Massacci & Gong (2026)
Ayik et al. reported one exact match among six validated human themes.
Sample: 6 evaluation units · Source: Ayik, Gu, Zan, Kim & Kim (2026)
No qualifying quantitative benchmark. NVivo has limited AI capabilities.
This comparison summarises every quantitative study we could find that evaluated AI-generated qualitative codes, themes or classifications against human analysis. For each tool or model family, one headline result was taken from each relevant study and converted to a percentage; where a paper evaluated several models from the same family, those scores were averaged first. The displayed figure is the simple average across a tool’s studies. Evidano’s 92.3%, for example, averages the 92% coding accuracy reported in the UNFPA–UNICEF meta-synthesis, the κ = .959 reported by Firetto et al. and the 8 of 9 human themes (88.9%) supported in Abner & Mee.
A study qualifies when it was published in 2024–2026 in a peer-reviewed journal or proceedings, or as a United Nations evaluation, and compares a named tool or model with human coding. Where a paper compared prompts or settings, the result for its final or best-performing prompted configuration was used, never a fine-tuned model or a subset that leaves out the AI’s misses; where it reported results only per code or category, those were averaged. Preprints, and studies that measured only embedding similarity, preference ratings or run-to-run consistency, are not included.
The studies used different measures, including Cohen’s kappa, F1 score, percent agreement, the share of human codes or themes matched, and agreement with manually coded segments; kappa and F1 values are shown as percentages for readability. Benchmark sample sizes sum the evaluation units each study reported (reports × codes, discussion turns, utterances, coded segments, excerpts, themes, responses, transcripts), counting the same source data once per tool.
All 34 studies in the comparison
Published studies that ran Evidano alongside their own manual coding and reported how the two compared.
Two researchers coded 11 interviews by hand with over 90% agreement, then ran the anonymised transcripts through Evidano and compared the results for triangulation and validation.
209 survey participants, a 97% response rate
Educational Psychology 46(1): 27–51 (Routledge / Taylor & Francis), open access
A peer-reviewed 2026 Education Sciences study used Evidano as a thought partner after three rounds of human coding, surfacing a gender pattern the team had missed.
22 courses’ open-ended student evaluation comments analyzed
Education Sciences (MDPI), 16(3), 448, open access
Published in Frontiers in Education (2026): encrypted data, human-verified themes, and a transparent AI coding workflow across English and Arabic interviews.
9 Teachers interviewed
Frontiers in Education, 11:1690181 (Frontiers Media, 2026), open access
Full write-ups of these studies, interviews with the researchers behind them, and the wording their authors published when citing Evidano.
Peer-reviewed studies, UN evaluations and university research where teams used Evidano for AI-assisted qualitative analysis — with the methods, the numbers and the published source for each.
Interviews with researchers at leading universities and institutes about how they actually used Evidano in work they went on to publish.
Verbatim excerpts from 20 published studies describing what Evidano did in their analysis — as a thought partner, a second coder, for initial coding, or for synthesis.
Practical guides to qualitative methods — thematic analysis, grounded theory, evidence synthesis and more — alongside the wider Evidano article library.
The only convincing benchmark is your data, your codebook, and your judgment. The free plan is free forever — run a transcript through and audit the output.
