When ChatGPT Joins the Coding Process: What Changes in Thematic Analysis?

That tension sits at the centre of Mike Perkins and Jasper Roe’s article, “The use of Generative AI in qualitative analysis: Inductive thematic analysis with ChatGPT.” The authors built a case study around parallel human and AI-supported analyses of academic-publisher policies on generative AI. Their aim was to see where GPT-4 could add speed or perspective, where it introduced risk, and how its outputs could be reconciled with qualitative judgement.

We spoke with Mike Perkins, Associate Professor and Head of the Centre for Research & Innovation at British University Vietnam, to learn more about what AI-assisted thematic analysis can contribute, where it breaks down, and why researcher judgement remains central.

Qualitative analysis depends on interpretation. Researchers do not simply sort sentences into boxes; they decide what distinctions matter, how evidence fits together, and whether a pattern is meaningful in context. Generative AI changes the practical conditions of that work because a model can rapidly process large amounts of text and suggest structures that a researcher may not have considered. But speed creates a methodological problem if an output looks plausible while being ungrounded, inconsistent, or insensitive to context.

Perkins and Roe’s case study is useful because it makes that trade-off concrete. The authors report that ChatGPT sometimes generated quotations or data that were not present in the source material, which meant every useful-looking output still had to be checked against the original dataset. The paper therefore treats AI as a source of additional analytical possibilities, not as a substitute for evidentiary discipline. Perkins reflected on the broader significance of the study: “AI can help researchers consider interpretations they might otherwise overlook, but a plausible theme still needs to earn its place in the analysis. The value comes from returning to the data, questioning the suggestion, and deciding whether it helps us understand what is actually there.” That distinction helps readers separate productivity gains from claims about analytical validity.

The same issue applies to interviews, open-ended surveys, policy documents, and other qualitative material: AI can propose patterns quickly, but researchers remain responsible for deciding whether those patterns are supported and appropriately interpreted.

The paper is a methodological case study, not a benchmark of coding accuracy. Its evidence comes from the documented workflow, the policy dataset, and the practical advantages and failures observed across the two analytical routes.

• The study used a deliberately independent, dual-track analysis. The researchers began with 107 publishing-related entities. Manual checking found 36 with AI policies, eight of them duplicates or replicas, leaving 28 unique policies. Each researcher kept a separate dataset and withheld discussion until the individual analyses were complete, allowing the two routes to be compared before synthesis.
• ChatGPT generated themes quickly and at greater granularity, while the manual analysis captured more context and nuance. The AI-supported route processed the material rapidly and produced more candidate themes. Manual scrutiny was better at contextual interpretation and subtle distinctions. The final thematic structure therefore required comparison, debate, and researcher judgement rather than accepting either output on its own.
• Human verification was indispensable because the model hallucinated and was not reliably replicable. ChatGPT sometimes produced quotations or data absent from the dataset, requiring continuous source checking. Repeating the same prompts would not be expected to reproduce the same outputs because the model is stochastic, and changes to GPT-4 further complicate reproducibility over time.

Taken together, the findings support a “co-pilot” model in which AI can widen the set of patterns a researcher considers, while humans remain responsible for grounding, interpretation, and synthesis. Perkins reflected on the result: “What stood out was how much analytical work happened when we brought the two analyses together. The differences gave Jasper and me something useful to debate, and resolving those differences required us to return to the source material and explain why particular interpretations were convincing.”

The case study grew out of a broader project on academic-publisher policies for generative AI. Perkins and Roe first assembled publisher and journal policies through web searching and manual review, removed suspected predatory sources and repeated policies, and arrived at 28 unique documents. They then used inductive thematic analysis following Braun and Clarke’s framework. “Inductive” here means that themes were developed from the dataset rather than imposed from a pre-existing coding scheme.

The analysis was conducted at the semantic level from a realist perspective. In practical terms, the researchers focused on the explicit content and framing of the policies rather than trying to infer hidden or latent meanings. One researcher followed a conventional workflow, including systematic coding in MAXQDA and development of themes from those codes. The AI-supported workflow involved structuring the data for ChatGPT, iteratively rewriting prompts, inspecting the themes it produced, checking those themes against specific examples in the source material, and rerunning analyses at different times. Only after the separate analyses were complete did the researchers compare areas of agreement and divergence and negotiate the final thematic set.

The method also defines its limits. Because the policies were relatively explicit, the authors caution that the approach may not transfer cleanly to datasets where meaning depends on irony, sarcasm, humour, or other latent interpretation. Prompt development took time, outputs changed, and hallucinations demanded source verification. The article reports no numerical agreement or accuracy score between the human and AI analyses, so it does not establish that AI analysis is equivalent to human analysis.

For researchers, the practical implication is less “automate thematic analysis” than “redesign the workflow.” AI can help with data structuring, candidate-theme generation, comparison, and analytical prompting, but these steps need explicit validation. A defensible workflow should preserve source access, make human checking routine, and document how prompts and outputs informed decisions.

For institutions and methods trainers, the paper raises a reproducibility problem unlike ordinary software use. A model can change between analyses, and identical prompts can yield different results. Versioning, transparency about AI use, and careful reporting of researcher judgement therefore matter, while efficiency claims must account for prompt tuning, verification, and disagreement resolution.

The article demonstrates that generative AI can contribute productively to a qualitative workflow, but it does not show that AI can independently deliver a trustworthy thematic analysis across all kinds of data. The most promising direction is therefore one of disciplined augmentation, with the researcher retaining responsibility for meaning and evidence. Perkins’s guidance for practitioners adds an important final layer: “Check every proposed quotation against the original data, retain a record of the prompts and outputs that informed important decisions, and explain how you evaluated the AI’s contribution. This connects directly with my current work on AI disclosure: readers need to understand where AI influenced the research and what the researchers did to verify it.”