Fast, defensible insight into assessment design problems: that’s the payoff. A 2026 multimethod study (Donough et al., PLOS ONE, published 9 July 2026) combined nine in-depth interviews and a document review of 70 moderation reports and found persistent misalignment between intended learning outcomes and exam questions, with an overemphasis on lower-order cognitive items. If you run assessment reviews, quality assurance, or curriculum design, this post shows how AI-enabled qualitative analysis can turn the study’s raw transcripts and moderation reports into prioritized, auditable improvements, faster and with reproducible coding. Source: PLOS ONE.
Key Takeaways
Evidano is an AI-powered qualitative data analysis platform that ingests transcripts and moderation reports to produce prioritized, auditable remediation plans.
AI-enabled qualitative analysis can convert the Donough et al. study’s raw materials (nine interviews and 70 moderation reports) into reproducible, scalable audits in days rather than weeks.
- Donough et al. (PLOS ONE, 9 July 2026) combined nine interviews (May–Nov 2021) and 70 moderation reports (2015–2019) to triangulate gaps between intended outcomes and exam items.
- Seventy moderation reports flagged inconsistent question distribution, low cognitive-level weighting, unclear mark allocation, and grammar/format problems.
- Practical constraints identified include deadlines, large classes, limited training, and non-native English authorship, which repeatedly degraded item quality.
- Teams can operationalize moderation feedback by turning reports into dashboards, quantifying recurring faults, and prioritizing high-impact papers for redesign.
Findings Snapshot: study composition and implications
Findings Snapshot: The study combined educator interviews and moderator reports to reveal recurring, actionable issues across modules and years.
Findings Snapshot
| Date / Metric | Value | Source | Implication |
|---|---|---|---|
| Published | 9 July 2026 | PLOS ONE | Recent peer-reviewed evidence |
| Interviews (Study 1) | 9 interviews to thematic sufficiency (from 22 eligible educators); May–Nov 2021 | Donough et al. | Rich, in-depth educator perspectives |
| Document review (Study 2) | 70 moderation reports analysed (22 internal, 48 external) covering 2015–2019 | Donough et al. | Macro-level QA evidence of inconsistency |
| Key patterns | Overemphasis on lower-order Bloom levels; inconsistent mark allocation; language/grammar issues; time constraints | Triangulated findings | Actionable targets for redesign and training |
| Ethics / data | Raw transcripts not public (POPIA constraints); data access via university ethics committee | Donough et al. | Sensitive data, requires secure handling |
What the study found (plain English)
The study found consistent gaps between intended assessment design and enacted papers.
- Seventy moderation reports (2015–2019) flagged inconsistent question distribution, low cognitive-level weighting, unclear mark allocation, and grammar/format problems.
- Educators reported familiarity with Bloom’s taxonomy and constructive alignment but cited time pressure, limited training, and language barriers as reasons why higher-order items are under-produced.
- Triangulation showed perception not equal to practice: educators thought papers were appropriately pitched, moderators documented otherwise.
- Practical constraints (deadlines, large classes, non-native English authorship) repeatedly degraded item quality.
So what for researchers, QA leads, and educators
For assessment designers & nurse educators
Assessment designers and nurse educators should prioritize a cognitive-level audit and fix the highest-impact papers first. Prioritize a cognitive-level audit (map items to NQF/Bloom levels) and fix the highest-impact papers first (e.g., final-year, high-stakes modules).
Use moderation feedback as structured data rather than ad-hoc comments: classify and quantify recurring issues such as unclear stems or missing marks.
For programme leads & quality assurance
Programme leads and quality assurance teams should convert scattered moderation reports into dashboards to target training and policy updates. Turn scattered moderation reports into a dashboard of recurring faults (frequency, module, year level) to target training and policy updates.
Protect assessment data according to POPIA and institutional ethics: use encrypted, access-controlled tools when sharing transcripts or reports.
For qualitative researchers and evaluators
Qualitative researchers and evaluators should triangulate interviews with document analysis and quantify theme prevalence to move from thematic claims to operational recommendations. Triangulation is essential: combine interviews (n=9 saturation point here) with document analysis (n=70 reports) and quantify theme prevalence to move from thematic claims to operational recommendations.
Document inter-rater reliability for coding and retain audit trails to defend findings during accreditation or external moderation.
Do more, faster with Evidano
Overview
Evidano helps teams scale and standardize qualitative analysis of moderation reports and transcripts.
Problem: scattered transcripts and moderation reports
The problem is dispersed source material that is hard to search and aggregate; the solution is bulk ingest and a single searchable corpus. Solution: Bulk ingest and index documents (transcripts, moderation reports, module guides) into Evidano for a single searchable corpus.
Problem: manual coding is slow and inconsistent
The problem is slow, inconsistent manual coding; the solution is AI-assisted thematic coding with hierarchical visualizations and exportable codebooks. Solution: AI-assisted thematic coding with hierarchical code → subcode visualizations and exportable codebooks so you can standardize across raters and retain an audit trail.
Problem: need to quantify misalignment (how often, where)
The problem is knowing frequency and location of misalignment; the solution is frequency and cross-segment analysis that counts occurrences by module and year. Solution: Frequency and cross-segment analysis that counts occurrences (e.g., 'lower-order cognitive' tags by module/year) and compares segments (first vs fourth year).
Problem: language & accessibility issues
The problem is multilingual, inconsistent language in items; the solution is integrated transcription, translation, and custom dictionaries. Solution: Integrated transcription and translation with custom dictionaries to standardize clinical/discipline terms; PII redaction for compliance with POPIA and other privacy laws.
Problem: follow-up data required from students or staff
The problem is scaling structured follow-up interviews; the solution is deployable AI avatar interviewers and pipeline reintegration. Solution: Deploy AI avatar interviewers to collect structured follow-up interviews at scale, then feed responses back into the same analysis pipeline.
Security & trust
Evidano uses encrypted storage and does not expose your data to third-party training. Start operationalizing moderation feedback today at Evidano.
Quick start: 7-step week-one workflow
Follow this 7-step workflow to convert the study’s approach into an operational QA sprint in the first week.
- 1) Collect documents: gather moderation reports, exam papers, mark schemes, and 6–12 educator interview transcripts in a secure folder.
- 2) Upload to Evidano: enable PII redaction and set a custom nursing dictionary for terms and verbs.
- 3) Run automated transcription and translation if interviews are multilingual.
- 4) Apply an initial codebook (constructive alignment, Bloom levels, grammar/format issues) and let AI pre-code the corpus.
- 5) Review AI-suggested codes, reconcile differences, and lock the codebook with an exported audit trail.
- 6) Generate frequency and cross-segment reports (for example, percentage of items per Bloom level by NQF year) and co-occurrence visualizations to find hotspots.
- 7) Produce a one-page action memo for module leads listing the top three modules to rework, the top five recurring item flaws, and recommended training topics.
FAQ: qualitative analysis of assessment design
The following frequently asked questions and answers are drawn from the study’s methods and findings.
FAQ: qualitative analysis of assessment design
What did Donough et al. find about exam cognitive levels?
+
The study found an overemphasis on lower-order Bloom levels and inconsistent mark allocation across moderation reports. The study’s triangulation of interviews and 70 moderation reports (2015–2019) documented recurring low cognitive-level weighting and unclear mark allocation.
How many interviews and reports were analysed?
+
The study analysed nine in-depth educator interviews and 70 moderation reports. The nine interviews reached thematic sufficiency (from 22 eligible educators, May–Nov 2021) and the document review covered 70 moderation reports from 2015–2019.
Why are higher-order items underproduced?
+
Higher-order items were underproduced because of practical constraints including time pressure, limited training, language barriers, deadlines, and large class sizes. Educators stated familiarity with Bloom’s taxonomy but reported these constraints as barriers to producing higher-order items.
How can moderation reports be used to prioritise fixes?
+
Moderation reports can be converted into dashboards, coded for recurring faults, quantified by frequency and module/year, and used to prioritise high-impact papers for redesign. The post recommends turning scattered moderation feedback into structured, countable data to target training and policy updates.
How should sensitive transcripts be handled for research and QA?
+
Sensitive transcripts must be protected in encrypted, access-controlled tools with PII redaction to meet POPIA and institutional ethics requirements. The study notes raw transcripts are not public and data access may require university ethics committee approval.
Conclusion & next steps
Donough et al. (PLOS ONE, 9 July 2026) demonstrate that assessment design problems are detectable and actionable when interviews and moderation reports are triangulated and quantified.
The bottleneck is scale: parsing dozens of reports and transcripts by hand consumes time and introduces inconsistency; operationalising the study’s methods speeds reproducible audits and produces prioritized remediation lists.
If you need reproducible audits of exam cognitive levels, prioritised remediation lists, and secure handling of sensitive transcripts, Try Evidano for free. Original study: PLOS ONE.
Topics
- qualitative analysis of assessment design
- assessment design
- moderation reports
- Bloom's taxonomy
- Evidano
Keep reading
- Commentary on NewsAI qualitative analysis: sports officiating programsHow AI-enabled qualitative analysis reveals psychosocial impacts of NTIS HPOP (2021). Read study facts from PLOS ONE (Aug 5, 2026) and learn practical steps with Evidano.
- Commentary on NewsQualitative Analysis of SLP Job ReadinessTurn PLOS One (Jul 21, 2026) SLP focus-group findings into reproducible thematic evidence with AI transcription, translation, coding, and secure audit trails.
- Commentary on NewsAI-enabled Qualitative Research: Cultural IdentityReproducible AI-enabled qualitative workflow for cultural curricula: extract themes from a 47-teacher PLOS One study (Jul 21, 2026) and run a two-week pilot.