Evidano home
Published in the Nagoya Journal of Medical ScienceResponsible AINagoya University Hospital, Japan

What ChatGPT Gets Right and Misses in Qualitative Analysis

Evidano7 min read
Takeshi Kondo, Hospital Lecturer at Nagoya University Hospital

Imagine a medical education researcher reading an interview transcript from a supervising physician. The task is not just to summarize the words, but to decide which patterns matter, how they relate to the clinical setting, and what larger explanation might make sense of them. That shift from description to interpretation is where qualitative analysis becomes difficult.

A 2024 study in the Nagoya Journal of Medical Science asks what happens when part of that work is handed to ChatGPT. Takeshi Kondo and colleagues gave GPT-4 selected interview material from a previously published medical education study, instructed it to follow the six steps of thematic analysis, and compared the result with the earlier human-led analysis. The model was clear and often persuasive, but it also missed context, failed to develop theory, and allowed information from the prompt to leak into its findings.

This makes the paper less a contest between people and machines than a close look at their different strengths. We spoke with Takeshi Kondo, a medical education researcher at Nagoya University Hospital, to learn more about where AI-assisted thematic analysis is convincing, where it becomes brittle, and why context still matters.

About the article

Research question

Can a general-purpose large language model conduct an inductive thematic analysis comparable with a human analysis, and what differences emerge when the two are examined closely? The study assesses both the themes produced and the quality of the analytic process.

Key finding

GPT-4 produced a coherent, traceable analysis that three expert assessors rated above the fixed human benchmark on several dimensions. Yet the integrated comparison exposed major limits: prompt information entered the findings, the analysis stayed close to explicit content, and it did not independently build the contextual or theoretical explanations that made the human analysis deeper.

Why this research matters

Qualitative research is valuable because meaning often depends on more than literal wording. A researcher may know how a hospital works, why a question was asked, what was left unsaid, or which theory helps explain a pattern. A system that can organize text quickly but does not reliably reconstruct that surrounding world may still be useful, but its role needs to be carefully bounded.

Kondo and colleagues make that distinction visible. ChatGPT was strong at extracting salient points and showing a step-by-step chain from interview text to codes and themes. That traceability can make an analysis easier to inspect, but it does not guarantee that the interpretation is sound: one prompt-derived idea entered the final report, and none of the three knowledgeable assessors initially noticed.

For researchers using interview data, the practical question is therefore how machine-supported organization should be combined with human judgment. Kondo reflected on the broader significance of the study: “The question is not whether AI can produce themes. It can, and quickly. AI tools now keep a detailed record of each analytic step and take on much of the laborious work of integrating data. This may free researchers to think more about the abstract and theoretical side of the analysis. But the interpretation is still the researcher’s responsibility, because what participants’ words mean depends on the setting, the relationships and the theory a researcher brings to the data.” The paper’s answer is cautious rather than dismissive: AI can contribute, but researchers remain responsible for deciding whether an interpretation is warranted by the data and context.

Key findings from the article

Three contributions stand out when the numerical ratings are read alongside the assessors’ comments.

  • ChatGPT scored strongly on traceability, credibility and procedural clarity. The human analysis was fixed at 4 on a seven-point reference scale. All three assessors rated ChatGPT above that benchmark for confirmability, credibility, dependability and consistency; all three gave dependability the maximum score of 7, while credibility scores were 6, 6 and 7. The assessors repeatedly praised the explicit links between interview text, codes, themes and the research question.
  • The apparent strength was uneven once analytic depth was examined. Depth scores were 2, 3 and 7, a striking spread. ChatGPT often summarized the interview material clearly, but the human analysis moved further into context-sensitive meanings, including the mismatch between a generic rubric and the clinical setting and an interpretation involving cognitive load.
  • Prompt contamination created a subtle failure that expert review did not catch immediately. The final ChatGPT report said the localized assessment tools better captured the hospital’s specific context, even though that idea came from background supplied in the prompt rather than the interview excerpt. The authors treated this as contamination by directions in the prompt and warned that plausible, polished output can make such errors harder to spot.

Taken together, the findings suggest that a fluent analytic workflow can be useful and deceptively reassuring at the same time. Asked what surprised him most, Kondo answered: “The prompt contamination. A sentence from the background I had written into the prompt appeared in ChatGPT’s findings as if it had come from the interviews. None of the assessors noticed it, including me as an author of the original study. It was only noticed when a researcher who had not been involved with the original data worked through the analysis.”

Methods and research approach

The study used a convergent mixed-methods design. The researchers returned to publicly available interview material from an earlier study on locally adapted assessment rubrics for medical residents. GPT-4 was asked to analyze selected material using Braun and Clarke’s six-step thematic analysis sequence: familiarization, initial coding, searching for themes, reviewing themes, defining and naming themes, and producing a report. The recorded ChatGPT workflow took 45 minutes.

Three assessors who knew the original study closely, two authors and one peer reviewer, evaluated the AI analysis on six seven-point dimensions: depth, confirmability, credibility, transferability, dependability and consistency. The human analysis was fixed at 4 as a reference point. The assessors also wrote free-text comments, which the researchers integrated with the ratings. The study is small, uses a purpose-built scale, and analyzed only part of the original interview material. It was also conducted with GPT-4 in spring 2023, so it should be read as a methodological case study rather than a benchmark of current large language models.

Broader implications

For researchers, the paper suggests a division of labor rather than simple substitution. AI may help surface salient passages, produce initial codes, organize an inductive pass and document how one analytic step led to another. Human researchers remain essential when interpretation depends on social context, theoretical framing, reflexivity, non-verbal knowledge, or the judgment to recognize when a plausible output has been shaped by the prompt rather than the data.

The authors also point toward a more interactive future. They distinguish deduction, induction and abduction, with abduction involving repeated movement between data and theory, and suggest that deeper analysis may require dialogue in which humans supply context and theoretical possibilities. Kondo reflected on responsible practice and the next research step: “Reading the output line by line against the data was not enough. AI can amplify the researcher’s own assumptions, and this kind of bias is very hard to see in one’s own work. In our study, it took someone from a different position to see it. Building discussion among people with different perspectives into the analysis may become more important, not less. Models have changed a great deal since 2023, so these questions should be tested again with current models.” The paper does not show that such a workflow solves the problem, but it gives future studies a concrete direction to test.

About Takeshi Kondo

Takeshi Kondo, Hospital Lecturer at Nagoya University Hospital

Takeshi Kondo, MD, MHPE, PhD, is a Hospital Lecturer at the Center for Postgraduate Clinical Training and Career Development, Nagoya University Hospital, Japan. He is a general physician and medical education researcher. He received a Master of Health Professions Education from Maastricht University and a PhD from Nagoya University, and is currently a PhD candidate at the School of Health Professions Education, Maastricht University. His research focuses on workplace-based assessment, postgraduate medical education and the use of AI in medical education. He serves on the editorial board of Medical Education (Japan).

About the authors

  • Takeshi Kondo (Department of General Medicine/Family & Community Medicine and Center for Medical Education, Nagoya University Graduate School of Medicine, Japan)
  • Junichiro Miyachi (Center for Medical Education, Nagoya University Graduate School of Medicine; Academic and Research Centre, Hokkaido Centre for Family Medicine, Japan)
  • Anders Jönsson (Faculty of Education, Kristianstad University, Sweden)
  • Hiroshi Nishigori (Center for Medical Education, Nagoya University Graduate School of Medicine, Japan)

Read the article

Read the full research article, “A mixed-methods study comparing human-led and ChatGPT-driven qualitative analysis in medical education research,” published in the Nagoya Journal of Medical Science in 2024.

Full citation

Kondo, T., Miyachi, J., Jönsson, A., & Nishigori, H. (2024). A mixed-methods study comparing human-led and ChatGPT-driven qualitative analysis in medical education research. Nagoya Journal of Medical Science, 86(4), 620–644. doi.org/10.18999/nagjms.86.4.620

View the publication

We are grateful to Dr. Kondo for sharing his perspective on this study and for his contribution to the careful, transparent use of AI in qualitative research.

Research, accuracy and responsible AI

How Evidano's output has been evaluated, how its models were built, how your data is handled, and how to disclose or cite AI use in your own work.

  • Resource

    Human vs AI: validated accuracy benchmarks

    Four head-to-head comparisons against expert human analysis — 371 interview transcripts with Arizona State and Penn State, 298 evaluation reports, 224 open-ended survey responses with UT Austin and Wichita State, and a 160-response manual-coding review.

  • Resource

    How Evidano’s models were trained

    Model provenance for research-ethics review: a provenance-tiered curated corpus, and customer uploads, prompts and outputs excluded from training entirely.

  • Resource

    Data security for qualitative research

    Encryption in transit and at rest, customer-controlled deletion, regional storage, and SOC 2 Type 2, GDPR, HIPAA-ready and CCPA/CPRA compliance.

  • Resource

    How to cite and disclose Evidano

    Citation formats in APA, MLA, Chicago, Harvard and Vancouver, methods-section wording per workflow, and an AI-policy review of 41 journals across 8 disciplines.