Evidano home
Published in the Journal of Computer Assisted LearningResponsible AIKorea Institute for Curriculum and Evaluation

When AI Becomes a Second Coder: What Classroom Dialogue Reveals About Human Judgment

Evidano8 min read
Eunhye Shin, Research Fellow at the Korea Institute for Curriculum and Evaluation

A single science lesson can produce hundreds of short exchanges: a question, a correction, a half-formed explanation, a joke, a remark about what to write next. For a teacher-researcher, those fragments can be analytically rich and painfully slow to code. Meaning can depend on what happened seconds earlier, what students were doing, and whether a sentence was literal, sarcastic, procedural, or reflective.

That is the problem at the center of Eunhye Shin’s 2025 study of human–ChatGPT co-coding in science education. Rather than asking whether a large language model could replace a qualitative researcher, Shin tested a more practical question: could one researcher use ChatGPT as a coding partner while retaining control over interpretation? The study reanalysed 1,287 student utterances from middle school science classes and compared human–human with human–ChatGPT agreement, then examined what happened when the researcher and model disagreed.

We spoke with Eunhye Shin, Research Fellow at the Korea Institute for Curriculum and Evaluation and the study’s author, to learn more about how AI-assisted coding can support solo qualitative researchers without removing the need for contextual judgment.

About the article

Research question

How does collaborating with ChatGPT affect codebook development and qualitative coding when a single researcher analyses classroom dialogue, and what can disagreements between human and AI reveal about the analytic process?

Key finding

ChatGPT did not match human coding reliability overall, but it contributed a more fine-grained codebook and sometimes forced the researcher to reconsider her own assumptions. Its clearest weakness was context-sensitive interpretation, especially when intent, sarcasm, or preceding turns mattered.

Why this research matters

The difficult part of qualitative coding is deciding what a piece of language is doing. In classroom dialogue, “Positive and negative” can be an observation or a response depending on what came before it. A direction to “compare the phenomena” can look procedural in isolation but function as a reflection on the group’s approach. The study tests AI assistance in precisely this messy setting.

For solo researchers, including teachers studying their own classrooms, a second human coder may be difficult to recruit. Shin’s workflow shows a possible middle ground: use AI to generate alternative classifications and rationales, then make the researcher responsible for reconciliation. That matters methodologically because the value of the model may lie less in automatic correctness than in creating a structured disagreement that exposes hidden assumptions. Shin reflected on the broader significance of this approach: “At the time I conducted this study, I was a science teacher, and it was difficult to find a colleague who could work with me as a qualitative co-coder. Having a competent colleague who can independently review and challenge my analysis can be a major source of rigor and momentum in research. In this sense, AI provided a valuable opportunity to have another perspective examine my coding decisions and raise questions I might not have considered on my own. The value was not that the AI was always correct, but that its alternative interpretations helped me verify and reconsider my analysis.”

The paper is also a useful caution against treating agreement metrics as the whole story. Human–ChatGPT agreement was moderate, with a Cohen’s kappa of 0.560, compared with 0.646 between human coders. Yet one of the study’s most interesting moments came when the AI’s rationale persuaded the researcher to change her own code. In other cases, the researcher rejected the model because classroom context carried meaning the model had missed.

Key findings from the article

Three contributions stand out because they show both what AI can add to qualitative analysis and where human interpretation remains indispensable.

  • ChatGPT helped produce a more granular codebook. When prompted with the full dataset, ChatGPT proposed between 14 and 16 codes, often splitting broad human categories into more specific functions. Shin used these suggestions to revise a prior seven-code scheme into nine codes, including separate categories for Observation and Experiment Conduction. Those distinctions later made hands-on experimental talk easier to separate from conceptual question-and-response exchanges.
  • Agreement was moderate overall, with large differences by code. Across the full 1,287 utterances, researcher–ChatGPT agreement was κ = 0.560, while researcher–human agreement on a 150-utterance sample was κ = 0.646. ChatGPT aligned relatively well on Question (κ = 0.709), Explanation (κ = 0.690), and Experiment Conduction (κ = 0.645), but agreement fell sharply for Meta-cognition (κ = 0.271). Notably, Experiment Conduction was one category where human–ChatGPT agreement exceeded human–human agreement, suggesting that clearly defined procedural language may be comparatively tractable for the model.
  • Disagreements became a source of reflection, not just error correction. The researcher asked ChatGPT to justify mismatched codes and then made the final decision. When the model classified a scientifically incorrect causal statement as an Explanation rather than a Hypothesis/Inference, its rationale exposed that the researcher had been partly judging scientific correctness instead of applying the codebook’s functional definition. The final code was changed to Explanation. Elsewhere, the researcher overruled the model when sarcasm, prior turns, or classroom activity changed the meaning of an utterance.

Taken together, the results suggest that the most productive human–AI relationship in this study was adversarial in a constructive sense: the model supplied an alternative reading, while the researcher retained responsibility for context and interpretation. Shin reflected on the result: “One of the most useful disagreements occurred when ChatGPT and I coded a student utterance differently because it contained scientifically incorrect content. By questioning ChatGPT about its coding rationale, I realized that, as a science teacher, I had unconsciously allowed the scientific correctness of the student’s statement to influence my coding. I was surprised to realize that I had momentarily forgotten that I was acting as a researcher analyzing my students’ discourse, rather than as their science teacher evaluating their scientific understanding. The experience reminded me to focus on the qualitative features of the interaction rather than the correctness of its scientific content.”

Methods and research approach

Shin used an explanatory mixed-method design built around real classroom data from second-year middle school static-electricity lessons she taught herself. Twenty-eight students used collaborative Google Docs worksheets and 21 used paper worksheets. About 10 hours of audio were recorded across 12 small groups, yielding 1,287 clear student utterances translated into English. Her teacher-researcher role provided rich context but also created a risk of subjective interpretation.

The AI workflow had two stages. First, ChatGPT-4o was asked to develop a codebook inductively from all 1,287 utterances. Second, Shin and ChatGPT independently applied the revised codebook to the complete dataset. Prompt development was iterative: a zero-shot test produced 54% agreement with the researcher, a few-shot version reached 62%, and clearer definitions and examples raised agreement to 91% on a test set. Feeding 130 utterances at once produced end-of-list coding failures, so the final workflow used batches of 100 utterances with 30 examples in the prompt.

Reliability was assessed with a confusion matrix and Cohen’s kappa, while an additional human coder independently coded 150 randomly selected utterances. Shin reconciled every human–AI disagreement after reviewing context and asking ChatGPT for its rationale. The final codes were then analysed with ordered network analysis, which maps how coded events follow one another. The study’s limits include data from Shin’s own classes, a limited analysed period, incomplete temporal context in AI coding, and nondeterministic model outputs that constrain reproducibility.

Broader implications

The study does not demonstrate that ChatGPT can replace an experienced qualitative coder. In most categories, human–human agreement was higher, and the model struggled when meaning depended on speaker intent, sarcasm, prior conversational turns, or tacit knowledge of the classroom. Those are not peripheral details in qualitative research. They are often where interpretation lives.

What the paper does demonstrate is a workable form of researcher-controlled co-coding. The AI can propose categories, apply a defined codebook, and explain its classifications. The human can treat those outputs as prompts for scrutiny rather than final answers. For researchers without a coding team, this may introduce useful challenge into a solitary analytic process, provided confidentiality, context, model variability, and domain expertise are handled carefully.

The next research question is therefore not simply whether AI–human agreement can be pushed higher. It is how to design co-coding procedures that make disagreement informative while preserving accountability with the researcher. Shin reflected on what researchers should do next: “One important consideration is how rapidly LLMs are evolving. I conducted my study using GPT-4o because I needed to continue the analysis with the model available at the time. However, even before my paper was formally accepted, the performance of newer LLMs had improved substantially, and I found myself wondering whether I should conduct the study again using a newer model. This experience made me realize that researchers using LLMs need to continually reconsider whether the model they are using remains appropriate for their research. Given the rapid pace of change, it is important to rigorously examine and test which model is most suitable for the specific research purpose, dataset, and coding framework.”

About Eunhye Shin

Eunhye Shin, Research Fellow at the Korea Institute for Curriculum and Evaluation

Eunhye Shin is a Research Fellow at the Korea Institute for Curriculum and Evaluation in South Korea. She previously served as a public school science teacher for over ten years. Her recent research spans science education, learning analytics, classroom dialogue, ordered network analysis, and AI-assisted qualitative coding, with a focus on understanding how students learn through interaction and technology-supported activities.

About the authors

  • Eunhye Shin (Korea Institute for Curriculum and Evaluation, South Korea)

Read the article

Read the full research article, “Co-Coding Classroom Dialogue: A Single Researcher Case Study of ChatGPT-Assisted Analysis in Science Education,” published in the Journal of Computer Assisted Learning in 2025.

Full citation

Shin, E. (2025). Co-coding classroom dialogue: A single researcher case study of ChatGPT-assisted analysis in science education. Journal of Computer Assisted Learning, 41(4), e70089. doi.org/10.1111/jcal.70089

View the publication

We are grateful to Eunhye Shin for sharing her perspective on this study and for her contribution to the careful, transparent use of AI in qualitative research.

Research, accuracy and responsible AI

How Evidano's output has been evaluated, how its models were built, how your data is handled, and how to disclose or cite AI use in your own work.

  • Resource

    Human vs AI: validated accuracy benchmarks

    Four head-to-head comparisons against expert human analysis — 371 interview transcripts with Arizona State and Penn State, 298 evaluation reports, 224 open-ended survey responses with UT Austin and Wichita State, and a 160-response manual-coding review.

  • Resource

    How Evidano’s models were trained

    Model provenance for research-ethics review: a provenance-tiered curated corpus, and customer uploads, prompts and outputs excluded from training entirely.

  • Resource

    Data security for qualitative research

    Encryption in transit and at rest, customer-controlled deletion, regional storage, and SOC 2 Type 2, GDPR, HIPAA-ready and CCPA/CPRA compliance.

  • Resource

    How to cite and disclose Evidano

    Citation formats in APA, MLA, Chicago, Harvard and Vancouver, methods-section wording per workflow, and an AI-policy review of 41 journals across 8 disciplines.