A randomized preprint compared Evidano and NVivo on the same qualitative task

The study asked whether AI assistance could reduce the time needed for thematic analysis without reducing report quality. Five participants used Evidano and five used NVivo. All had graduate-level qualitative-methods training, prior NVivo experience, and no prior experience with Evidano or another AI-assisted qualitative tool.

Participants were not told that the study was comparing the two platforms. Each person analyzed 12 interview transcripts about parents’ views of childhood influenza vaccination, organized the findings through the COM-B behavior-change framework, selected supporting quotations, and produced a report.

Participants assigned to Evidano received a standardized one-hour training session. They uploaded all 12 transcripts, reviewed the initial AI-generated themes and evidence, reorganized the material through COM-B, selected quotations, and wrote the final analysis.

The comparison group imported and coded the same data in NVivo, developed themes, mapped them to COM-B, selected quotations, and prepared the same deliverables.

All participants logged their time. The final reports were anonymized and reformatted so evaluators could not infer which tool had been used.

Five qualitative researchers with at least ten years of thematic-analysis experience each reviewed five randomly assigned Evidano-versus-NVivo pairs. Across the 25 comparisons, they rated thematic comprehensiveness, analytical depth, supporting evidence, theoretical integration, and overall quality.

The evaluators also received the original transcripts and an earlier published analysis of the dataset as a reference point. None had prior Evidano experience.

Mean analysis time was 7.38 hours in the Evidano group and 27.71 hours in the NVivo group, a difference of 20.33 hours. The reported Welch test was statistically significant, and the ranges did not overlap: the slowest Evidano participant still finished in less than half the time of the fastest NVivo participant.

The expert panel preferred Evidano-assisted reports in 19 of 25 overall comparisons. The largest reported advantage was comprehensiveness, where the Evidano report was preferred in 21 of 25 comparisons. Preferences were narrower and not statistically significant for supporting-evidence quality and theoretical integration.

The qualitative feedback suggests the AI-assisted reports were better at surfacing a broad set of themes, socioeconomic constraints, and interactions between COM-B components. NVivo sometimes retained advantages in close evidence selection and theory integration.

The study goes beyond a testimonial. It uses random assignment, the same source material, time logs, blind comparisons, multiple evaluators, and explicit scoring dimensions. It also identifies where AI assistance appeared stronger and where the advantage was uncertain.

The workflow supports a practical interpretation: Evidano can accelerate first-pass pattern detection and free time for review and interpretation, but conceptual integration and judgment still depend on the researcher.

This is a preprint and has not been peer reviewed. The sample included only ten analysts and one 12-transcript health-behavior dataset. All analysts were graduate students rather than senior qualitative researchers. The one-hour tool training differed from participants’ prior NVivo experience, and the COM-B framework may favor structured rather than deeply inductive analysis.

Quality ratings in qualitative research remain subjective even when evaluators are blinded. The supporting-evidence and theoretical-integration comparisons did not reach statistical significance. The results are best read as promising experimental evidence, not a universal claim that one platform outperforms another.