What Happens When LLMs Code Cognitive Presence in Online Discussions?

That tension sits at the center of “Transforming online learning research: Leveraging GPT large language models for automated content analysis of cognitive presence,” by Daniela Castellanos-Reyes, Larisa Olesova, and Ayesha Sadaf. The study asks whether GPT models can classify “cognitive presence,” how learners construct and confirm meaning through reflection and discussion, using four phases: triggering event, exploration, integration, and resolution.

Daniela Castellanos-Reyes, Assistant Professor of Learning, Design, and Technology at North Carolina State University, shares her perspective on what automated content analysis can and cannot do when an LLM interprets student discourse.

Discussion boards are one of online education’s richest sources of evidence about how students think together. The Practical Inquiry Model distinguishes different reasoning moves, from recognizing a problem through exploring possibilities to integrating ideas and applying a solution. Yet the framework is costly to use at scale because content analysis requires training, repeated coding, negotiation between coders, and close reading.

The appeal of an LLM is therefore not simply speed. The more consequential question is whether it can make theoretically grounded analysis more accessible while preserving the distinctions that make the framework useful. Castellanos-Reyes reflected on that balance: “LLM-assisted content analysis has the potential to make theoretically grounded analysis of online discussions more accessible to researchers and instructors. To realize that potential, however, we need clear coding guidelines and careful validation, because lower costs and faster coding are valuable only when we also understand the quality and limitations of the classifications.” The results make that question concrete because the model was much better at some cognitive phases than others.

Three results show both the promise and the limits of automated coding.

• Moderate reliability was possible, but the best LLM still fell well short of trained human agreement. The two researchers reached a weighted Cohen’s kappa of .84. Against their coding, the best model, fine-tuned GPT with a one-shot prompt, reached .59. Default models were lower at .39 for one-shot and .36 for few-shot prompts, while the fine-tuned few-shot model reached .52.
• Performance depended heavily on the kind of reasoning being recognized. Integration was clearest: agreement with researchers exceeded 75 percent across all configurations and reached 87.5 percent for the best fine-tuned one-shot model. Resolution was much harder, with 0 percent agreement for both default models and 10 percent and 16.7 percent for the fine-tuned models. The best model also mislabeled 48 exploration paragraphs as integration, illustrating a difficult boundary between those phases. Because resolution examples were scarce, these percentages should be interpreted cautiously.
• In this study, a simpler prompt performed better, while computational savings were substantial. The one-shot prompt used definitions and one example per phase rather than the fuller indicators in the few-shot prompt, yet performed better in both default and fine-tuned models. Human coding, including written rationales, took 5 hours 48 minutes and was estimated at $311. Fine-tuned model training took under eight minutes, automated coding under three minutes, and reported training plus coding costs stayed below $2 per configuration at the API prices used in the study. These figures do not represent the full cost of preparing and validating an automated analysis workflow.

Taken together, the findings suggest that more prompt detail is not automatically better and that efficiency cannot be separated from classification quality. Castellanos-Reyes reflected on the result: “I was surprised that the simpler prompt performed better in our study, which showed us that adding more detail does not necessarily improve performance. For me, this finding reinforced the importance of testing prompts against carefully coded data rather than assuming that a more elaborate prompt will be more reliable.”

The study used three week-long discussion threads from a 16-week online graduate course in instructional design. The case-based discussions were structured around the four Practical Inquiry Model phases. After anonymization, 180 posts were segmented into 293 paragraphs, averaging 98 words. Forty-five paragraphs formed the training sample and the remaining 248 formed the testing sample.

The first two authors, both experienced with the Practical Inquiry Model, coded training data through repeated rounds of discussion and reliability checking. They adapted an established codebook into imperative, model-facing instructions, tested one-shot and few-shot prompts, and fine-tuned a GPT Turbo model through the OpenAI API. Default and fine-tuned configurations were evaluated against human coding using weighted Cohen’s kappa, percent agreement, confusion matrices, time, and cost.

The study is deliberately bounded. Its data came from one case-based course whose prompts were designed around the same framework being coded. Triggering-event and resolution examples were relatively scarce, formatting was lost during preprocessing, and the cost comparison excluded the time required to learn the programming and data-literacy skills needed for API automation. The authors also note that changing LLM versions complicate replication.

The paper’s most useful contribution may be methodological rather than technological. It shows that evaluating AI-assisted coding requires more than one agreement score. Researchers need to ask where disagreement occurs, whether categories are conceptually fuzzy, how prompt design reshapes a codebook, and what human expertise remains necessary. Here, difficulty distinguishing exploration from integration echoed a long-running conceptual problem in the framework itself.

The authors explicitly resist treating LLM-assisted content analysis (LACA) as a replacement for manual coding. They position it as a complementary tool, especially for instructors or researchers new to the Practical Inquiry Model, while human judgment remains important for context, nuance, validation, and unexpected patterns. The work also points toward moving from retrospective classification to tools that could help instructors notice discussion patterns while a course is underway.

For similar workflows, researchers need to decide what the model should do, how its performance should be evaluated, and where human judgment is necessary. Castellanos-Reyes reflected on that boundary: “Researchers should be cautious about assuming that a model will perform equally well in another course or with a different student population. For that reason, I would like future studies to test these approaches across learning contexts and model versions, examining performance within each coding category as well as overall agreement.”