Evidano home
Published in BMC Medical Research MethodologyResponsible AIWitten/Herdecke University

Making LLM Use Visible in Qualitative Research: From a Scoping Review of 75 Studies to the COREQ+LLM Checklist

Evidano10 min read
Leonard Fehring, gastroenterologist and digital health researcher at Witten/Herdecke University

A journal reviewer opens a qualitative interview study and reads that “ChatGPT was used to support thematic analysis.” Which model was it, and which version? Was it used through a web interface or an API, with what settings and with what prompts? Who checked its output, and how? In many published studies, the reviewer would find no answer to most of these questions. The established reporting checklist for interview and focus group studies, the Consolidated Criteria for Reporting Qualitative Research (COREQ), was written in 2007 and contains a single item on software. It says nothing about model selection, prompting, configuration or validation.

A team at Witten/Herdecke University in Germany has set out to close that gap in two linked pieces of work. In “The use and methodological reporting of large language models in qualitative research: a scoping review,” published in BMC Medical Research Methodology in 2026, Christian Kempny, Julian Frings, Paul Rust, Sven Meister and Leonard Fehring map how large language models (LLMs) are used in qualitative research and how that use is reported. Their findings directly informed “Reporting of Qualitative Research Using Large Language Models (COREQ+LLM): A Delphi-based Extension of the COREQ Reporting Guideline,” a preprint on medRxiv by Leonard Fehring, Julian Frings, Paul Rust, Christian Kempny, Petra Thürmann, Emre Tamerler and Sven Meister, which sets out a consensus-based checklist of 31 reporting items for qualitative research that uses LLMs.

We spoke with Dr. Leonard Fehring, of Health Care Informatics in the Faculty of Health at Witten/Herdecke University, to learn more about what the review revealed, how the expert panel reached consensus, and what the checklist asks of researchers.

About the work

Research question

The scoping review asked two questions: how are LLMs currently being applied in qualitative research, and what reporting practices do qualitative studies that use LLMs follow? The COREQ+LLM study then asked which items researchers should report when they use LLMs in qualitative research, building on COREQ’s established structure.

Key finding

Across 75 included studies, LLMs were used at almost every stage of qualitative work, most often for coding assistance and theme identification, but technical reporting was highly inconsistent: three quarters of studies reported no parameter settings at all, and about half did not say how the model was deployed. An international Delphi process then agreed a final checklist of 31 items in seven categories. The authors are clear about the limits: the review covered English-language, peer-reviewed articles only, and Delphi consensus reflects collective expert judgement rather than empirical validation of the checklist.

Why this research matters

LLMs differ from earlier qualitative data analysis software in ways that matter for rigour. As the COREQ+LLM authors explain, their outputs are probabilistic, so the same prompt can produce different answers; they can reproduce biases in their training data; and they can confidently produce statements that are wrong. Without explicit reporting, the authors warn, studies may appear human-led while in fact relying substantially on algorithmic synthesis. Existing AI reporting guidelines such as TRIPOD-LLM, CONSORT-AI and SPIRIT-AI focus on prediction models and clinical trials, not on interpretive qualitative work.

This work matters because it builds the guidance from evidence rather than opinion. The scoping review documents, study by study, what is and is not being reported, and the Delphi process then turns those gaps into concrete reporting items agreed by a large international panel. The result gives authors, reviewers and editors a shared vocabulary for describing how humans and LLMs worked together in a study. Leonard Fehring explained why the team took this on: “Our group works at the intersection of clinical medicine and digital health. A substantial part of our research is qualitative, focusing on understanding how digital innovations (i.e. AI) can be successfully integrated into clinical practice.

“Given our expertise in both the technical aspects of AI and qualitative research methodologies, we were excited about the new opportunities that AI could offer for qualitative research. At the same time, we recognized that these opportunities come with new challenges, methodological pitfalls, and potential risks that need to be carefully examined and transparently documented.

“Considering the absence of any suitable reporting standards we saw a responsibility to take the initiative and develop a guideline that brings together perspectives from both fields. Our ambition was to create a checklist that would be methodologically sound, practically useful, and broadly accepted by experts in artificial intelligence and qualitative research.”

Key findings from the work

Three findings stand out because they show both how fast LLMs have entered qualitative research and how little of that use is currently visible to readers.

  • LLMs are already used across the qualitative workflow, and mostly through one provider. The search identified 5,049 records, of which 4,201 remained after removing duplicates, and 75 studies met the inclusion criteria. OpenAI’s GPT models appeared in 93% of them. Coding assistance (43 studies) and theme identification (41) were the most common uses, and thematic analysis (38) and content analysis (12) were the most common methods. In 45 studies, LLM output was compared with human coders.
  • Technical reporting is the weakest link. Only 13 studies reported temperature settings, 12 documented context length and 4 provided top_p values. About half of the studies (45%, 34) did not specify whether the model was used through an API, a web interface or a local installation, and 75% (56) reported no parameter settings at all. While 61% provided complete or partial prompts, 13% gave no prompting details. Reported agreement between LLM and human coders ranged from 36% to 99%, which the authors link to task complexity, prompt quality and the rigour of validation. Nearly all studies discussed ethical considerations (95%) and incorporated human verification of AI outputs (97%).
  • A large expert panel agreed 31 reporting items. The Delphi study invited 1,455 experts from 67 countries; 141 valid responses were analysed in the first round, and 115 participants completed the second (a follow-up response rate of 81.6%). Starting from 29 candidate items drawn from the review, the panel added items suggested in free-text comments, rejected others (for example, detailed reporting of costs and resources), and settled the remaining questions at a virtual consensus conference. The final checklist groups 31 items under seven headings: research team and reflexivity; study design and theoretical framework; model and access; prompting; context use and session management; data analysis and findings; and ethics and data handling.

Taken together, the two papers describe a field that has adopted LLMs quickly but documents them unevenly, and a practical response that asks researchers to make visible how, when and why an LLM was used. Leonard Fehring reflected on the findings: “I was quite surprised by how little information was being reported on the use of LLMs in scientific publications. This is a serious concern, particularly when it comes to reproducibility and transparency. If researchers do not clearly document how and for what purposes LLMs have been used, it becomes difficult to assess the robustness and reliability of their findings. Against this background, it is hardly surprising that there is still considerable skepticism about the use of LLMs in qualitative research.”

Methods and research approach

The scoping review followed the PRISMA-ScR guidelines and the Joanna Briggs Institute framework, under a pre-specified protocol. The team searched PubMed, CINAHL, PsycINFO, Business Source Premier and Scopus for peer-reviewed empirical studies published in English between January 2020 and May 2025 that used at least one LLM in a substantive stage of qualitative research, beyond manuscript drafting. Screeners worked independently in Rayyan, and data were extracted with a standardised template covering model type and version, qualitative method, purpose, parameter settings, deployment, prompting, ethical considerations and human–AI interaction.

The COREQ+LLM guideline followed the EQUATOR Network’s guideline development framework and was registered as a guideline under development. Two online Delphi rounds ran between August and October 2025. Participants rated each candidate item on a five-point scale, with inclusion defined in advance as more than 75% of respondents rating it 4 or 5, and could suggest new items or wording in free text, which the team analysed thematically. Items still undecided after the second round went to a virtual consensus conference on 16 October 2025, attended by nine experts, who voted anonymously. The panel also agreed that the reporting principles apply to other natural-language processing models, not only generative LLMs, with authors able to mark items as not applicable.

The authors acknowledge the limits of both studies. The review was restricted to English-language, peer-reviewed journal articles, so innovations first reported in other languages, preprints or grey literature may be missing, and no formal quality appraisal was carried out. The Delphi panel was skewed towards researchers from Europe and North America with qualitative backgrounds, perspectives from the Global South may be underrepresented, and the consensus conference was small. Finally, consensus is not the same as empirical validation: future studies should test whether using COREQ+LLM actually improves reporting quality and rigour.

Broader implications

For researchers, the most practical implication is a set of details to record while the work is being done, not reconstructed afterwards: the model and version, how it was accessed, the settings used, the full prompts, how sessions and context were managed, how outputs were validated against human judgement, and how the LLM’s contribution was integrated into the interpretation. For reviewers and editors, the checklist offers a consistent way to assess whether LLM-based tools were applied responsibly.

The work also points to gaps in the evidence base. The authors call for comparative studies of different models, prompting strategies and validation approaches, for more research in non-English and multilingual contexts, and for attention to the field’s heavy reliance on proprietary models. They also note that reporting guidelines themselves are debated in qualitative research, and that standardised reporting should not oversimplify interpretive, context-dependent and reflexive work. Asked for his advice, Leonard Fehring said: “LLMs are powerful tools that can genuinely facilitate our everyday work as researchers. However, we do not necessarily need to rely on the most advanced commercial models for every task. There is a growing range of open-source alternatives that can be run locally on our own computers. This offers considerable advantages, particularly when working with sensitive data, as it gives researchers greater control over where their data is processed and how it is handled. Ultimately, the key is to choose the model that is appropriate for the specific task, rather than automatically opting for the most known one.”

It is equally important to be clear about what this work does not do. The scoping review maps how LLMs are used and reported; it does not evaluate how well they perform. The COREQ+LLM checklist sets out what to report, not which tools or workflows to use, and its value in practice has yet to be tested. What the work offers is a clear, evidence-based starting point for making LLM use in qualitative research transparent and open to scrutiny.

Read the COREQ+LLM preprint at doi.org/10.64898/2026.10.01.26364479. The checklist and further information are at coreq-llm.github.io.

About Leonard Fehring

Leonard Fehring, gastroenterologist and digital health researcher at Witten/Herdecke University

Priv.-Doz. Dr. med. Leonard Fehring is a gastroenterologist and digital health researcher at Witten/Herdecke University’s Faculty of Health, where he is affiliated with Health Care Informatics. He is the corresponding author of the COREQ+LLM reporting guideline and of the accompanying scoping review on how large language models are used and reported in qualitative research.

About Christian Kempny

Christian Kempny, researcher at Witten/Herdecke University

Christian Kempny is a researcher at the Chair of Health Services Research at Witten/Herdecke University. His work focuses on diversity in healthcare, qualitative methodology, and the use of large language models in research. He co-developed the COREQ+LLM reporting guideline and is first author of the Springer textbook “KI in der qualitativen Forschung” (AI in Qualitative Research). He also teaches research methods and works as an organizational development consultant.

About the authors

  • Christian Kempny (Health Services Research, Faculty of Health, Witten/Herdecke University, Germany)
  • Julian Frings (Health Care Informatics, Faculty of Health, Witten/Herdecke University, Germany)
  • Paul Rust (Health Care Informatics, Faculty of Health, Witten/Herdecke University, Germany)
  • Sven Meister (Health Care Informatics, Witten/Herdecke University; Department Healthcare, Fraunhofer Institute for Software and Systems Engineering ISST, Dortmund, Germany)
  • Leonard Fehring (Health Care Informatics, Faculty of Health, Witten/Herdecke University; Department of Gastroenterology, Helios University Hospital Wuppertal, Germany)
  • Petra Thürmann (Clinical Pharmacology, Witten/Herdecke University; co-author of the COREQ+LLM preprint)
  • Emre Tamerler (Health Care Informatics, Witten/Herdecke University; co-author of the COREQ+LLM preprint)

Read the article

Read the scoping review, “The use and methodological reporting of large language models in qualitative research: a scoping review,” published in BMC Medical Research Methodology in 2026.

Full citation

Kempny, C., Frings, J., Rust, P., Meister, S., & Fehring, L. (2026). The use and methodological reporting of large language models in qualitative research: A scoping review. BMC Medical Research Methodology, 26, 137. doi.org/10.1186/s12874-026-02913-1

View the publication

We are grateful to Dr. Fehring for sharing his perspective on this work and for his contribution to the responsible, transparent use of AI in qualitative research.

Research, accuracy and responsible AI

How Evidano's output has been evaluated, how its models were built, how your data is handled, and how to disclose or cite AI use in your own work.

  • Resource

    Human vs AI: validated accuracy benchmarks

    Four head-to-head comparisons against expert human analysis — 371 interview transcripts with Arizona State and Penn State, 298 evaluation reports, 224 open-ended survey responses with UT Austin and Wichita State, and a 160-response manual-coding review.

  • Resource

    How Evidano’s models were trained

    Model provenance for research-ethics review: a provenance-tiered curated corpus, and customer uploads, prompts and outputs excluded from training entirely.

  • Resource

    Data security for qualitative research

    Encryption in transit and at rest, customer-controlled deletion, regional storage, and SOC 2 Type 2, GDPR, HIPAA-ready and CCPA/CPRA compliance.

  • Resource

    How to cite and disclose Evidano

    Citation formats in APA, MLA, Chicago, Harvard and Vancouver, methods-section wording per workflow, and an AI-policy review of 41 journals across 8 disciplines.