Evidano home
PhD in Information ScienceResponsible AIUniversity of Brasília (UnB)

Where LLMs Help in Systematic Reviews

Evidano9 min read
Marília Augusta de Freitas, University of Brasília

Featured researcher: this paper is featured independently and does not evaluate Evidano.

Systematic reviews depend on explicit protocols, predefined eligibility criteria and documented procedures for searching, selecting, appraising and synthesising studies. That rigour is what makes their evidence trustworthy, but it also places a heavy operational load on reviewers, and the accelerating growth of the scientific literature keeps adding to it. Since generative models spread from late 2022, large language models (LLMs) such as GPT, Claude, Gemini, Llama and Mistral have been used to support or automate stages of the review process, especially reference screening and data extraction. Studies report productivity gains, yet doubts about reliability, reproducibility and performance persist, and the evidence is scattered across studies of individual models, stages and domains. For a review team deciding whether, and how, to bring an LLM into its workflow, a consolidated view has been hard to find.

That gap is the starting point for “Large Language Models em revisões sistemáticas e síntese de evidências: um mapeamento bibliométrico TEMAC das aplicações, do desempenho e dos níveis de automação” (Large Language Models in Systematic Reviews and Evidence Synthesis: A TEMAC Bibliometric Mapping of Applications, Performance and Automation Levels), a full paper by Marília Augusta de Freitas and Ari Melo Mariano of the University of Brasília (UnB), published in 2026 in the proceedings of the XXVI National Meeting of Research and Graduate Studies in Information Science (ENANCIB), in its Data, Information and Technology working group. Combining the Consolidated Meta-Analytic Approach Theory (TEMAC) with PRISMA 2020 reporting guidance, the authors mapped 106 studies published between 2023 and 2026 and synthesised the 95 available in full text, asking which review stages LLMs support, how well they perform against human reviewers and what levels of automation they reach.

Dr Marília Augusta de Freitas, a librarian at the University of Brasília (UnB), discusses what the findings mean for review teams.

About the article

Research question

The paper asks what empirical evidence exists on the use of LLMs to support or automate stages of systematic reviews. The authors break this into seven questions: which review stages are supported, which models are used, which metrics evaluate them, how they perform relative to human reviewers, what benefits are reported, what limitations are identified and what levels of automation are achieved.

Key finding

Most applications involved title and abstract screening or data extraction. In screening, models often retrieved a high proportion of eligible studies, while the proportion of eligible studies among those they flagged varied. The authors see a role for LLMs in preliminary screening and as an additional reviewer under human supervision.

Why this research matters

Bringing LLMs into systematic reviews raises questions beyond performance, because it redistributes work between researchers and automated systems. The paper places this in a longer argument about automation and intellectual work. Braverman (1974) and Zuboff (2019) emphasise the risks of transferring human skills to technical systems, while Davenport and Kirby (2016) propose augmentation, in which technology takes on repetitive, large-scale tasks and leaves professionals the work that requires interpretation, judgement and responsibility. The authors adopt the augmentation perspective to ask whether the empirical evidence points towards substitution, complementarity or augmentation of reviewers’ work, and which governance mechanisms condition each level of automation.

The study makes that debate concrete by consolidating dispersed evidence, and it brings an information-science lens to it. Selecting, indexing, condensing and representing content are classic operations of the field, and LLMs now act on them. The pattern of high recall with more variable precision restates, on a new technological base, the familiar tension between exhaustiveness and relevance. And the predominance of human-in-the-loop workflows reaffirms the role of the information professional as a qualified mediator who formulates prompts, sets criteria and validates decisions. Marília Augusta de Freitas explained why the team undertook the mapping: “We wanted to know what happened when reviewers used these models to make everyday decisions. In information science, we ask which relevant records a search or a selection process might leave out. That question matters just as much when a model helps screen the results.”

Key findings from the article

Three findings stand out because they show both where LLMs already help and where their limits lie.

  • The research base is young, fast-growing and concentrated on two tasks. Records grew from 15 in 2023 to 127 in 2025 in the Web of Science, and from 57 to 198 in Scopus. Across the 95 full-text studies, applications concentrated on data extraction and title and abstract screening, followed by complete workflows, risk-of-bias assessment and full-text screening. The GPT family appeared in 80 studies, followed by Claude (22), Gemini (18), Llama (14) and Mistral (7), while classic machine learning and tools such as ASReview, Rayyan and RobotReviewer served as comparative baselines. Individual studies could evaluate more than one model family.
  • Performance depends on the task: high recall, more variable precision. The studies relied mainly on recall or sensitivity, precision, F1, work saved (WSS), accuracy and agreement with humans (kappa). In title and abstract screening they predominantly reported high recall with more variable precision, which favours using models as a first filter while keeping human review. In data extraction, performance was more consistent for structured fields and more sensitive to the complexity of the variables, the length of the documents, the wording of the prompts and the need for quantitative reasoning.
  • Human oversight is the norm, and governance means more than a human in the loop. Automation levels were predominantly assistive, with a human in the loop; full automation appeared only in restricted tasks with audit mechanisms. Recurring limitations included hallucination, complex inclusion criteria, dependence on training data, low reproducibility and bias. The authors argue that governance therefore requires recording prompts, identifying model versions, documenting parameters, preserving inputs and outputs, auditing, and defining who is responsible for decisions.

From these findings the authors build an integrative model with four interdependent dimensions: tasks (the review stages supported), technology (the models and architectures used), evaluation (metrics, validation designs and performance patterns) and governance (benefits, risks and degrees of autonomy). The main result, they write, is not a demonstration of unrestricted feasibility but the finding that reliable use of LLMs depends on balancing performance, transparency, supervision and accountability. Marília Augusta de Freitas reflected on the results: “What stood out to me was how much the answer depended on the task. A model might retrieve most of the relevant abstracts and still pass on many that do not meet the criteria. For a review team, those are two different questions: what might we miss, and how much do we still need to read?”

Methods and research approach

The study is a literature review with a mixed approach, in which bibliometric mapping and qualitative synthesis play complementary roles. The bibliometric dimension followed TEMAC (Mariano and Rocha, 2017), with stages for preparing the search, interrelating the data and detailing the field through co-citation and bibliographic coupling. Selection and reporting followed the PRISMA 2020 flow, and the maps were built in VOSviewer. The authors searched Scopus and the Web of Science Core Collection for articles and conference papers from 2023 to 2026, combining four conceptual blocks (type of evidence synthesis, review stages and tools, AI techniques, and performance terms) and excluding purely bibliometric studies, umbrella reviews and risk-prediction studies.

The searches yielded 744 records, 463 from Scopus and 281 from Web of Science. After deduplication, 535 records entered screening. Claude supported the preliminary thematic classification. Following specialist assessment, the researchers re-examined all 130 uncertain records and advanced ten of them to full-text assessment. In total, 178 records reached eligibility assessment and 106 studies entered the bibliometric corpus. Full texts were available for the qualitative synthesis of 95 studies. The team recorded screening decisions in auditable spreadsheets and monitored agreement between human assessors using kappa. The researchers retained responsibility for final inclusion.

Co-citation analysis identified three converging schools in the field’s intellectual base: reporting and conduct standards for systematic reviews; text mining and machine learning for screening; and the generative turn of LLMs. Bibliographic coupling identified three active research fronts: selection and screening, data extraction and synthesis, and complete workflows that chain several review stages. The authors note two limitations: the recent time window, and the qualitative interpretation of the maps, which they mitigated by considering the network structure together with the content of representative studies.

Broader implications

For research teams, the most practical message is about roles. The evidence supports using LLMs as a first filter in screening and as a second reviewer, with human validation, rather than as a replacement for reviewers. It also implies documenting AI use as carefully as any other part of the protocol: prompts, model versions, parameters, inputs and outputs, and who is responsible for each decision. The extraction-and-synthesis front includes work on qualitative data, such as an evaluation of generative AI for qualitative data extraction in community-based fisheries management literature (Spillias et al., 2025), so the same questions of validation and accountability reach qualitative evidence synthesis too. Marília Augusta de Freitas was clear about where reviewers need to stay in control: “In our review, we went back to the 130 records classified as uncertain. Ten went forward for full-text assessment. Deciding whether a study meets the eligibility criteria is a judgement the researchers need to be able to explain.”

For information science, the authors see gains on four complementary fronts: the organisation and representation of information, its retrieval and evaluation, mediation, and information governance and ethics. Limitations such as hallucination, bias and low reproducibility turn into requirements for prompt records, version identification and assigned responsibility, which are curation and provenance practices the field already knows well. The integrative model offers a shared vocabulary of tasks, technology, evaluation and governance for investigating, teaching and regulating the use of LLMs in evidence synthesis. Asked for her advice, Marília Augusta de Freitas said: “Start with one task and a set of records your team has already assessed. Look at the disagreements, especially the relevant studies the model missed. Keep the prompt, model version and outputs so another reviewer can examine how you reached a decision.”

It is equally important to be clear about what the study does not show. It maps and synthesises published evaluations rather than testing a tool, and it does not pool results in a quantitative meta-analysis, which the authors list among next steps alongside performance by task and mechanisms for auditing, traceability and algorithmic accountability. Its corpus comes from two databases over a short, fast-moving period. Nor does it argue against LLMs: the studies report large reductions in workload and time. What it shows is that, for now, reliable use depends on the conditions around the model, including supervision, transparency and accountability, as much as on the model itself.

About Marília Augusta de Freitas

Marília Augusta de Freitas, University of Brasília

Marília Augusta de Freitas is a librarian at the University of Brasília (UnB), where she works in institutional evaluation within the Decanato de Planejamento, Orçamento e Avaliação Institucional (DPO). She holds a PhD in Information Science from UnB and conducts postdoctoral research at the University of São Paulo’s School of Communications and Arts (ECA-USP). Her research interests include bibliometrics, the societal impact of research and the use of large language models in evidence synthesis.

About Ari Melo Mariano

Ari Melo Mariano, University of Brasília

Ari Melo Mariano is a professor and researcher in production engineering at the University of Brasília (UnB), where he founded and directs the DataLab. He holds a PhD in administration and has conducted postdoctoral research in data science, scientific methodology and quantitative methods. His research interests include artificial intelligence, bibliometrics and text mining. He created the TEMAC platform for bibliometric mapping.

About the authors

  • Marília Augusta de Freitas (Universidade de Brasília, UnB, Brazil)
  • Ari Melo Mariano (Universidade de Brasília, UnB, Brazil)

Read the article

Read the full paper, “Large Language Models em revisões sistemáticas e síntese de evidências: um mapeamento bibliométrico TEMAC das aplicações, do desempenho e dos níveis de automação,” published in the proceedings of ENANCIB in 2026.

Full citation

Freitas, M. A. de, & Mariano, A. M. (2026). Large language models em revisões sistemáticas e síntese de evidências: Um mapeamento bibliométrico TEMAC das aplicações, do desempenho e dos níveis de automação. In Anais do XXVI Encontro Nacional de Pesquisa e Pós-Graduação em Ciência da Informação (ENANCIB 2026) (pp. 1–17). ANCIB. enancib.emnuvens.com.br/enancib/pt_BR/article/view/2491

View the publication

We are grateful to Dr de Freitas for sharing her perspective on this study and for her contribution to the responsible, transparent use of AI in research and evidence synthesis.

Research, accuracy and responsible AI

How Evidano's output has been evaluated, how its models were built, how your data is handled, and how to disclose or cite AI use in your own work.

  • Resource

    Human vs AI: validated accuracy benchmarks

    Four head-to-head comparisons against expert human analysis — 371 interview transcripts with Arizona State and Penn State, 298 evaluation reports, 224 open-ended survey responses with UT Austin and Wichita State, and a 160-response manual-coding review.

  • Resource

    How Evidano’s models were trained

    Model provenance for research-ethics review: a provenance-tiered curated corpus, and customer uploads, prompts and outputs excluded from training entirely.

  • Resource

    Data security for qualitative research

    Encryption in transit and at rest, customer-controlled deletion, regional storage, and SOC 2 Type 2, GDPR, HIPAA-ready and CCPA/CPRA compliance.

  • Resource

    How to cite and disclose Evidano

    Citation formats in APA, MLA, Chicago, Harvard and Vancouver, methods-section wording per workflow, and an AI-policy review of 41 journals across 8 disciplines.