Research Spotlight: Using AI to Strengthen Costa Rica’s Evaluation Agenda

That challenge is explored in the 2026 book “Casos de estudio sobre el proceso OCDE – Costa Rica: una guía académica para docentes y estudiantes de Administración Pública,” by Esteban O. Mora-Martínez and published by the Center for Research and Training in Public Administration at the University of Costa Rica. A dedicated chapter reviews evaluations from Costa Rica’s National Evaluation Agenda and asks what they collectively reveal about the design, coordination, results, and durability of public interventions.

The chapter uses AILYZE.com, now known as Evidano, to conduct a structured analysis of ten evaluation reports across employment, productivity, education, health, culture, environment, and international cooperation. Esteban O. Mora-Martínez served as evaluator, while Evidano analyzed the reports using the study’s evaluation framework.

Individual evaluations can explain what happened within a single program. A cross-program meta-evaluation can reveal patterns that are harder to see in isolation, such as repeated coordination gaps, similar weaknesses in monitoring, or a common dependence on short-term political and financial conditions.

For public institutions, the value of evaluation therefore depends on more than completing a report. Evaluation evidence must inform redesign, resource allocation, coordination, and long-term learning. The chapter frames findings and recommendations as a strategic opportunity to improve institutional decision-making, rather than as an administrative formality.

Dr. Mora-Martínez reflected on the broader significance of the chapter: “The most important implication of this meta-evaluation is that the quality of public policy depends not only on program design and implementation, but also on the quality of the evaluations used to guide decisions. Across sectors, we found recurring challenges related to target population definition, institutional coordination, monitoring systems, and sustainability. For public institutions and policymakers, this highlights the need to strengthen evaluation as a mechanism for continuous learning, strategic adaptation, and evidence-informed governance, rather than treating it as a compliance exercise.”

Four recurring findings organize the chapter’s synthesis of evaluations conducted between 2016 and 2021.

• Relevance was present, but unevenly specified. Most interventions responded to recognizable national needs, yet several did not define beneficiaries precisely or clearly connect broad goals with operational objectives. Sensitivity to equity, economic conditions, and social differences also varied across programs.
• Coherence was constrained by limited coordination. Weak internal articulation and insufficient collaboration across institutions produced duplication, coverage gaps, and missed opportunities for complementary action. Some initiatives aligned well with national priorities, but connections to other policies and actors were often limited.
• Effectiveness was easier to show through outputs than through lasting change. Programs frequently reported coverage, participation, or training results, but had greater difficulty demonstrating sustained qualitative effects such as decent employment, social mobility, institutional transformation, or structural change.
• Sustainability and adaptability remained fragile. Long-term benefits were threatened by limited resources, technical capacity, formal institutional arrangements, and monitoring systems. Without updated diagnostics and agile feedback, programs also struggled to adapt to changing political, social, or environmental conditions.

Taken together, these findings point to a consistent lesson: clearer theories of change, stronger coordination, formal continuity mechanisms, and richer measures of impact are needed if evaluations are to support durable improvements in people’s lives.

Looking back on the findings, Dr. Mora-Martínez said: “What surprised me most was how consistently similar weaknesses appeared across programs operating in very different policy areas. While many interventions were clearly relevant and often achieved important quantitative outputs, the evidence repeatedly pointed to gaps in coordination, unclear theories of change, limited measurement of long-term outcomes, and weak continuity mechanisms. This suggests that some of the most important challenges are systemic rather than sector-specific.”

The study used an exploratory, AI-assisted meta-evaluation. It systematically reviewed evaluation reports and synthesis documents for ten interventions: Empléate; PRONAMYPE; the Community Intelligent Centers project; cultural and creative micro, small, and medium enterprises; the national strategy on chronic non-communicable diseases and obesity; the National Forestry Development Plan; human papillomavirus vaccination; the Construyendo Puentes y Sinergias student-retention strategy; non-reimbursable international cooperation in biodiversity and climate change; and the DESCUBRE program.

Each report was examined through four internationally recognized criteria: relevance, internal and external coherence, effectiveness, and sustainability. The guiding questions asked whether interventions responded to beneficiary and institutional priorities, adapted to changing circumstances, complemented other programs and legal frameworks, achieved intended results, and could sustain those results over time.

The source material consisted of the original evaluation reports and their syntheses. The analysis extracted supporting quotations, findings, and recommendations, then compared them across sectors. Because this was a meta-evaluation, it assessed the evidence and reasoning contained in existing evaluations; it did not conduct a new impact evaluation of the programs themselves.

Evidano supported the application of the same evaluative questions and criteria across a heterogeneous set of reports. This made it possible to organize evidence program by program, identify recurring patterns, and produce an evaluative synthesis for each study, presented in the book’s annexes.

The book describes the process as an iterative analytical model and reports that it reached conclusions similar to earlier work using traditional qualitative methods. The division of roles remained explicit: Mora-Martínez acted as evaluator, while Evidano conducted the analysis. In this workflow, AI-assisted synthesis helped manage and compare the documentary evidence, while evaluative interpretation remained grounded in the study’s criteria and the evaluator’s judgment.

This use case shows where AI-assisted qualitative analysis can add practical value in public-sector evaluation: applying a consistent framework across many documents, surfacing cross-cutting issues, and accelerating the movement from dispersed reports to a comparative evidence base.

Dr. Mora-Martínez also noted how AI tools can support careful evaluation workflows when used responsibly: “Evidano helped apply the same analytical framework consistently across a diverse set of evaluation reports, making it easier to organize evidence, identify recurring patterns, and compare findings across sectors. At the same time, evaluators should remember that AI does not replace professional judgment. The value of AI-assisted analysis depends on having clear evaluation criteria, transparent methods, high-quality source material, and careful human interpretation of the results. AI can strengthen rigor and efficiency, but responsibility for the conclusions must remain with the evaluator.”

For policymakers and program designers, the chapter recommends defining a clear theory of change that links the public problem, causal logic, target population, activities, and intended outcomes. Better specification at the design stage improves relevance and makes later evaluation more meaningful.

For evaluation units, robust and longitudinal monitoring systems are essential. Quantitative indicators should be combined with qualitative and impact-oriented evidence on social, environmental, and economic change. Feedback mechanisms should also be designed early enough to support timely adaptation.

For institutions, sustainability requires more than temporary political support. Programs need stable legal and organizational foundations, adequate financing, technical capacity, and coordination arrangements that can continue across changes of government.

For communities and civil society, broader participation can improve both legitimacy and substance. The chapter particularly emphasizes the inclusion of traditionally marginalized groups and other key actors so that evaluation and policy design reflect a wider range of needs and experiences.

Dr. Mora-Martínez offered this takeaway for policymakers, evaluators, and public institutions: “My main message is that evaluation should be understood as a tool for institutional learning and improvement. Producing an evaluation report is not enough. The greatest public value emerges when findings are used to strengthen program design, improve coordination, build better monitoring systems, and ensure long-term sustainability. When evaluation becomes part of an ongoing learning cycle, it can contribute meaningfully to more effective and resilient public policies.”

The central takeaway is that evaluation should function as a system of continuous learning. When findings are connected to redesign, coordination, monitoring, and institutional commitment, they can help public organizations respond more effectively to complex and changing conditions.