Evidano home
Published in Perspectives on TerrorismResponsible AIPolarization & Extremism Research & Innovation Lab (PERIL), American University

Can a Chatbot Draft an M&E Plan? What One Test in Violence Prevention Shows, and Where Evaluators Stay Essential

Evidano7 min read
Susan Szmania, non-resident fellow at American University’s PERIL

A community organisation running a small violence-prevention programme is asked by its funder for a monitoring and evaluation plan. It has no evaluator on staff, and evaluation typically costs between five and fifteen percent of a programme budget, a price many community-based programmes cannot afford. Toolkits exist, but many are lengthy handbooks better suited to academic audiences than to practitioners in the field. A freely available AI assistant could produce a draft plan in seconds. The harder question is whether that draft would help or mislead.

That question is at the heart of “AI as Decision Support in P/CVE Evaluation: Opportunities, Limits, and Risks,” a research note by Susan Szmania published in Perspectives on Terrorism in June 2026. Rather than discussing AI in the abstract, the note runs a short, transparent demonstration: it asks Microsoft Copilot to build an M&E plan for a real bystander awareness programme and then compares the output with the programme’s publicly available external evaluation, published by the US Department of Homeland Security.

We spoke with Susan Szmania, non-resident fellow at American University’s Polarization & Extremism Research & Innovation Lab (PERIL), to learn more about what the comparison revealed and why an “evaluator-in-the-loop” remains essential.

About the article

Research question

The note asks how AI tools might reduce evaluation barriers for practitioners in preventing and countering violent extremism (P/CVE), and what their opportunities, limits and risks are when used as decision support in early-stage M&E planning. It explicitly does not propose AI as a comprehensive or prescriptive solution, but as a starting point for innovation.

Key finding

From a single, relatively basic prompt, Copilot generated a standard eight-part M&E plan in seconds, and several of its recommendations matched issues the real external evaluation later identified. But the plan lacked considerable context and required substantial expertise to interpret and implement. The note concludes that AI may “jump-start” discussion among stakeholders, but that improvements in evaluation practice depend on professional judgement, relationship management and adaptive decision-making that AI tools cannot replicate.

Why this research matters

Evaluating community-based violence prevention programmes is challenging, particularly in terrorism prevention and P/CVE, a field with a relatively short history of programme evaluation. Despite efforts to strengthen monitoring and evaluation capacity through conferences, indicator banks and at least seventeen dedicated evaluation toolkits between 2014 and 2023, translating guidance into practice remains difficult. This challenge is especially true for smaller programmes facing what the note calls an “age of austerity and instability” as funding shifts from early prevention towards hard security measures such as physical gates or bulletproof windows. Yet demonstrating the impact of early prevention efforts remains important, and generative AI is attractive precisely because it promises fast, low-cost support where evaluation expertise is scarce.

What makes this research note useful is that it grounds the debate in a real programme and a real benchmark. Instead of asking whether AI can evaluate, it asks a narrower, practical question: how close does an AI-drafted plan come to what experienced external evaluators actually did, and what is missing? Susan Szmania explained why she chose this approach: “I wanted to explore how AI tools could help overcome practical evaluation challenges, both for funders and practitioners, and to see how those results measured up to current evaluation approaches.”

Key findings from the article

The comparison highlights three lessons for evaluators and programme teams considering AI support in M&E planning.

  • AI can surface sound design choices early. Copilot recommended pre- and post-tests for the awareness training, the same adjustment the external evaluators made to the programme after implementers initially relied on post-training satisfaction surveys. It also anticipated the absence of baseline data and the need to designate a “responsible party” for data collection, both challenges the actual evaluation identified.
  • Plausible is not the same as appropriate. The note encountered no hallucinations in this demonstration, but the responses, while generally plausible, lacked significant context and were not necessarily correct or feasible. Some indicators were P/CVE-specific, such as the percentage increase in knowledge of warning signs, while others were generic counts, and the external evaluation, unlike the AI draft, explained why its indicators were selected.
  • The hardest evaluation questions are relational, not technical. Many important challenges concern community backlash, staff turnover or shifting priorities, and feasibility trade-offs. The real programme’s youth component could not be completed “due to the logistics of receiving parental consent,” a judgement call that sits with people, not tools.

Taken together, the demonstration positions AI as an early starting point for deliberation that may lower the initial barrier to M&E planning, provided it is coupled with sustained human oversight. A second prompt asked Copilot to identify potential biases in its own plan, which offered a starting point, but, as the note stresses, identifying bias does not equate to mitigating it. Susan Szmania reflected on the exercise: “AI tools identify trends and patterns, but human practitioners need to confront ethical challenges and biases in context-appropriate, culturally-relevant ways.”

Methods and research approach

The demonstration used Microsoft Copilot (GPT-4-turbo with internet access), chosen for its ease of use and accessibility to non-technical users. A single prompt simulated a typical practitioner request: “Can you assist with developing an M&E plan for a bystander awareness programme aimed at increasing bystanders’ awareness of indicators of mass violence risk?” The output included a programme overview, a logic model, output, outcome and impact indicators, data collection methods and risk mitigation, and the full query and response are reproduced in the note’s appendices.

The benchmark was the published process and outcome evaluation of the Palm Beach County Sheriff’s Office programme, funded under the US Department of Homeland Security’s Targeted Violence and Terrorism Prevention grant programme. The analysis relied only on publicly available information, and no programmatic data or personally identifiable information was entered into the AI tool. The note builds on earlier applied work by van der Vet and Malkki, who tested fifty prompts to see how Copilot could support P/CVE practitioners.

The author is clear about the limits of the design. It was not a controlled experiment, it used one programme and one tool, and the example was intended to simulate an early-stage funding application rather than the prompt a professional evaluator would use. The author also acknowledges former colleagues at the Department of Homeland Security’s Center for Prevention Programs and Partnerships (CP3), who tested these concepts during spring 2025.

Broader implications

For practitioners and funders, the note suggests a realistic role for AI: drafting rapid M&E plans during proposal phases or alongside donor templates, so that design gaps such as missing baselines or unclear data ownership are discussed from the outset rather than discovered late. Susan Szmania emphasized the collaborative potential of AI tools, especially in grant-funded projects: “AI can assist practitioners and donors align priorities and anticipate challenges that may arise during the programme lifecycle.”

The note also calls for specialised training that enables practitioners to critically analyse, rather than just use, AI tools, for documentation protocols that disclose AI use, and for clear ethical guidelines on how P/CVE programmes engage AI, especially around personal data. Asked for her advice, she said: “It’s important to test out these tools and see how they work in practice. Rather than seeing AI as an endpoint, these tools may offer a starting point in evaluation design.”

It is equally important to be clear about what the note does not show. It does not test AI on programme data, on qualitative analysis or on more intensive interventions such as disengagement, rehabilitation or reintegration, which may require specialised indicators or more complex designs that publicly available materials do not capture. Its contribution is a grounded, cautious account of AI as decision support, not a claim that AI can replace professional evaluation.

About Susan Szmania

Susan Szmania, non-resident fellow at American University’s PERIL

Susan Szmania is a non-resident fellow at American University’s Polarization & Extremism Research & Innovation Lab (PERIL), currently based in Nicosia, Cyprus. She was most recently Associate Director for Research and Content Development at the Center for Prevention Programs and Partnerships (CP3) at the US Department of Homeland Security, and has held government and academic positions on terrorism prevention in the United States, Europe and the Middle East. She holds a PhD in Communication Studies from the University of Texas at Austin.

About the authors

  • Susan Szmania (Non-resident fellow, Polarization & Extremism Research & Innovation Lab (PERIL), American University)

Read the article

Read the full research note, “AI as Decision Support in P/CVE Evaluation: Opportunities, Limits, and Risks,” published in Perspectives on Terrorism in 2026.

Full citation

Szmania, S. (2026). AI as decision support in P/CVE evaluation: Opportunities, limits, and risks. Perspectives on Terrorism, 20(2), 112–138. doi.org/10.19165/SZVJ6748

View the publication

We are grateful to Susan Szmania for sharing her perspective on this research note and for her contribution to the responsible, transparent use of AI in evaluation.

Research, accuracy and responsible AI

How Evidano's output has been evaluated, how its models were built, how your data is handled, and how to disclose or cite AI use in your own work.

  • Resource

    Human vs AI: validated accuracy benchmarks

    Four head-to-head comparisons against expert human analysis — 371 interview transcripts with Arizona State and Penn State, 298 evaluation reports, 224 open-ended survey responses with UT Austin and Wichita State, and a 160-response manual-coding review.

  • Resource

    How Evidano’s models were trained

    Model provenance for research-ethics review: a provenance-tiered curated corpus, and customer uploads, prompts and outputs excluded from training entirely.

  • Resource

    Data security for qualitative research

    Encryption in transit and at rest, customer-controlled deletion, regional storage, and SOC 2 Type 2, GDPR, HIPAA-ready and CCPA/CPRA compliance.

  • Resource

    How to cite and disclose Evidano

    Citation formats in APA, MLA, Chicago, Harvard and Vancouver, methods-section wording per workflow, and an AI-policy review of 41 journals across 8 disciplines.