Evidano is an AI-powered qualitative data analysis platform that helps researchers analyze interviews, open-ended surveys, and documents. According to the PLOS One paper by Çelebi and Penczynski published August 4, 2026, transcribing researcher codebooks into LLM prompts reached 82–88% agreement with human annotators when tested across three experimental-economics classification tasks. The PLOS One study reports that the authors ran approximately 703, 253 API classification calls between 24 March and 6 April 2026 and evaluated 1, 494 promise utterances, 493 level-k messages (L I), and 851 level-k messages (L II). This post refracts those results through the lens of AI-enabled qualitative research and shows how practical teams can apply the study's recommendations.
Key Takeaways
According to the PLOS One article (Çelebi & Penczynski, published August 4, 2026), faithfully transcribing human codebooks into LLM prompts produced 82–88% agreement with human-agreed labels across three classification tasks. The PLOS One paper also reports that the experiments used about 703, 253 API calls run between 24 March and 6 April 2026 and evaluated 1, 494 promise utterances, 493 L I messages, and 851 L II messages.
- 1) The PLOS One authors found that verbatim codebooks plus worked examples yield 82–88% model–human agreement in August 2026 tests.
- 2) The PLOS One experiments (24 March–6 April 2026) ran ~703, 253 classification requests and used paired comparisons to control run-to-run LLM noise.
- 3) The PLOS One paper concludes: "The practical recommendation is to pair a frontier model with a well-defined codebook, " a quote attributed to Çelebi and Penczynski (PLOS One, Aug 4, 2026).
- 4) The PLOS One results show that content (instructions, context, examples) matters far more than formatting, framing, or high-effort reasoning for well-specified prompts.
What Happened and how it was measured
Answer: The PLOS One study evaluated whether human codebooks can be used directly as LLM prompts, and measured agreement with human annotators across three experimental tasks.
According to the PLOS One paper (Çelebi & Penczynski, published August 4, 2026), the authors converted three experimental-economics codebooks into prompt variants by filtering, separating, compressing, and assembling codebook components, then ran roughly 703, 253 API classification calls between 24 March and 6 April 2026 to test 1, 494 promise utterances, 493 L I messages, and 851 L II messages.
According to the PLOS One paper, accuracy was defined as the proportion of items where the model prediction matched labels on which two independent human annotators agreed, and the authors used paired McNemar tests with p < 0.001 and 95% Wilson confidence intervals to control for run-to-run variability.
Findings snapshot
| Date | Metric | Value | Implication |
|---|---|---|---|
| 24 Mar–6 Apr 2026 | API calls | 703, 253 | Large factorial design; results robust to paired testing, per PLOS One |
| Aug 4, 2026 | Model–human agreement (verbatim codebook + examples) | 82–88% | Verbatim codebooks with examples produced high agreement, per PLOS One |
| Dataset counts (reported in PLOS One) | Promise / L I / L II items | 1, 494 / 493 / 851 | Task difficulty and example counts varied by task, influencing gains from examples |
| Experiment date reported | Significance threshold | p < 0.001 | Authors used McNemar paired tests to isolate treatment effects |
Why content beats surface prompt engineering
Answer: The PLOS One paper shows that the informational detail in a prompt drives accuracy more than formatting, framing, or reasoning settings.
According to the PLOS One study (Çelebi & Penczynski, Aug 4, 2026), the four prompt components were classification instructions, experiment context, theory context, and worked examples; the authors found that detailed classification instructions and examples were the dominant accuracy drivers, while formatting and role personas had negligible effects for frontier models.
According to the PLOS One paper, for recognition-heavy tasks (promise classification) model choice explained most variance, while for learning-heavy tasks (strategic reasoning) the level of instruction detail mattered most; the authors note that larger models made better use of additional detail and were robust to surface variations.
Implications for qualitative researchers and UX teams
How should researchers adapt codebooks for LLM classification?
Answer: Researchers should transcribe existing human codebooks verbatim, include worked examples, and prioritize informational detail over formatting tweaks.
According to the PLOS One paper (Çelebi & Penczynski, Aug 4, 2026), faithful transcription plus examples produced 82–88% agreement and made surface variations largely irrelevant for frontier models.
According to the PLOS One paper, researchers working in more variable domains should treat robustness to formatting and phrasing as a diagnostic: if small paraphrases change outputs, the prompt is likely underspecified and needs richer instructions or examples.
What model choices matter for reproducibility?
Answer: Frontier models give higher accuracy and more replicable outputs; smaller models are more sensitive to surface variations.
According to the PLOS One study, GPT 5.4 and Qwen 3.5 397B made better use of detailed content and were robust to formatting, while smaller models like Qwen 9B were more brittle and sometimes harmed by extra detail.
According to the PLOS One paper, adjacent GPT generations show high inter-model agreement, and the authors write that a move from one GPT generation to the next is comparable to replacing one human annotator with another.
How Evidano helps implement the PLOS One codebook workflow
Problem: Manual codebook conversion and scale
Answer: Researchers spend hours converting codebooks into machine prompts and running thousands of API calls, which is error-prone and costly.
The PLOS One paper reports a large experimental scale (about 703, 253 calls) that highlights the operational work required to test prompt variants at scale (Çelebi & Penczynski, Aug 4, 2026).
Solution: Automate the pipeline with Evidano features
Answer: Evidano automates codebook ingestion, prompt assembly, and batch classification while retaining researcher control over instruction detail.
Evidano is an AI-powered qualitative data analysis platform that helps researchers analyze interviews, open-ended surveys, and documents; Evidano's transcription and prompt-assembly workflows let teams import a codebook, retain the verbatim category definitions and worked examples, and run repeatable classification jobs.
Evidano supports end-to-end steps relevant to the PLOS One pipeline: importing codebooks and documents, extracting and preserving classification instructions and examples, and running large-scale classification experiments with paired comparison controls. See the Evidano features page for applicable capabilities.
Security and governance during LLM classification
Answer: Use platforms that encrypt data and prevent third-party model training on private inputs.
Evidano provides enterprise controls and encryption; see Evidano's data security page for policies that align with research governance and reproducibility needs.
FAQ: codebook prompting for LLMs
Can I use my existing human codebook as an LLM prompt without rewriting it?
Answer: Yes, if the codebook includes clear category definitions and worked examples; PLOS One found verbatim transcription works well for capable models.
According to the PLOS One paper (Çelebi & Penczynski, Aug 4, 2026), a minimal filtering step to remove lab-only instructions combined with verbatim category definitions and examples reached 82–88% agreement with human labels.
Do formatting and role personas substantially improve classification accuracy?
Answer: No for well-specified prompts on frontier models; formatting and personas matter mainly when prompts are underspecified.
According to the PLOS One study, formatting and framing had negligible effects on verbatim prompts for GPT 5.4, and surface effects only mattered inconsistently for smaller models.
Can chain-of-thought or high-effort reasoning replace a detailed codebook?
Answer: No, reasoning does not substitute for missing domain content and often adds little on well-specified prompts.
According to the PLOS One paper, reasoning settings did not reliably improve performance and sometimes hurt results when classification instructions were sparse (Çelebi & Penczynski, Aug 4, 2026).
How do I know if my prompt is well-specified?
Answer: A well-specified prompt yields robust classifications under paraphrase and simple surface changes.
According to the PLOS One authors, robustness to formatting, framing, or phrasing is the practical signal that a prompt has reached information saturation for the model being used (Çelebi & Penczynski, Aug 4, 2026).
What should I do if my domain codebook is much sparser than in the PLOS One experiments?
Answer: Invest time in enriching the codebook with inclusion/exclusion rules and worked examples before focusing on surface prompt engineering.
According to the PLOS One paper, adding category definitions, boundary cases, and examples is more effective than iterative formatting or persona tuning for learning-heavy tasks.
Conclusion & Next Steps
Answer: For reliable LLM classification, prioritize detailed codebook content and use a capable model; PLOS One demonstrates that verbatim codebooks plus examples deliver 82–88% agreement (Çelebi & Penczynski, Aug 4, 2026).
The PLOS One authors write: "The practical recommendation is to pair a frontier model with a well-defined codebook, " which we interpret as a call to focus researcher effort on instruction quality rather than formatting tricks.
If you want to operationalize the PLOS One pipeline, Evidano automates codebook ingestion, prompt assembly, and large-batch classification with built-in governance; learn more on the Evidano features page.
Try the approach yourself: Try Evidano for free.