Better LLM Coding: Codebook Prompting for LLMs

According to the PLOS One article (Çelebi & Penczynski, published August 4, 2026), faithfully transcribing human codebooks into LLM prompts produced 82–88% agreement with human-agreed labels across three classification tasks. The PLOS One paper also reports that the experiments used about 703, 253 API calls run between 24 March and 6 April 2026 and evaluated 1, 494 promise utterances, 493 L I messages, and 851 L II messages.

• 1) The PLOS One authors found that verbatim codebooks plus worked examples yield 82–88% model–human agreement in August 2026 tests.
• 2) The PLOS One experiments (24 March–6 April 2026) ran ~703, 253 classification requests and used paired comparisons to control run-to-run LLM noise.
• 3) The PLOS One paper concludes: "The practical recommendation is to pair a frontier model with a well-defined codebook, " a quote attributed to Çelebi and Penczynski (PLOS One, Aug 4, 2026).
• 4) The PLOS One results show that content (instructions, context, examples) matters far more than formatting, framing, or high-effort reasoning for well-specified prompts.

Answer: The PLOS One study evaluated whether human codebooks can be used directly as LLM prompts, and measured agreement with human annotators across three experimental tasks.

According to the PLOS One paper (Çelebi & Penczynski, published August 4, 2026), the authors converted three experimental-economics codebooks into prompt variants by filtering, separating, compressing, and assembling codebook components, then ran roughly 703, 253 API classification calls between 24 March and 6 April 2026 to test 1, 494 promise utterances, 493 L I messages, and 851 L II messages.

According to the PLOS One paper, accuracy was defined as the proportion of items where the model prediction matched labels on which two independent human annotators agreed, and the authors used paired McNemar tests with p < 0.001 and 95% Wilson confidence intervals to control for run-to-run variability.

Answer: The PLOS One paper shows that the informational detail in a prompt drives accuracy more than formatting, framing, or reasoning settings.

According to the PLOS One study (Çelebi & Penczynski, Aug 4, 2026), the four prompt components were classification instructions, experiment context, theory context, and worked examples; the authors found that detailed classification instructions and examples were the dominant accuracy drivers, while formatting and role personas had negligible effects for frontier models.

According to the PLOS One paper, for recognition-heavy tasks (promise classification) model choice explained most variance, while for learning-heavy tasks (strategic reasoning) the level of instruction detail mattered most; the authors note that larger models made better use of additional detail and were robust to surface variations.

Answer: For reliable LLM classification, prioritize detailed codebook content and use a capable model; PLOS One demonstrates that verbatim codebooks plus examples deliver 82–88% agreement (Çelebi & Penczynski, Aug 4, 2026).

The PLOS One authors write: "The practical recommendation is to pair a frontier model with a well-defined codebook, " which we interpret as a call to focus researcher effort on instruction quality rather than formatting tricks.

If you want to operationalize the PLOS One pipeline, Evidano automates codebook ingestion, prompt assembly, and large-batch classification with built-in governance; learn more on the Evidano features page.

Try the approach yourself: Try Evidano for free.