Codebook Prompts: LLM Classification Best Practice

According to the PLOS One study published August 4, 2026, faithfully transcribing a human codebook into an LLM prompt (including category definitions and worked examples) produced 82–88% agreement with human-agreed labels on three experimental classification tasks (PLOS One).

• The PLOS One study reports that 703, 253 classification requests were submitted between March 24 and April 6, 2026, with 702, 809 successful calls and a 0.06% error rate.
• The PLOS One study measured 1, 494 items for the promise task, 493 items for level-k task I, and 851 items for level-k task II, giving task-level sample sizes for replication.
• The PLOS One study finds that model choice explains most variation on recognition-heavy tasks, while the level of informational detail in the prompt drives performance on learning-heavy tasks.
• The PLOS One authors conclude: "preparing instructions as one would for human annotators, with detailed context, category definitions, and examples, " is the recommended approach (Çelebi & Penczynski, PLOS One, August 4, 2026).

Answer: The PLOS One team converted three published human codebooks into prompt variants and tested them across four LLMs and six experimental manipulations to measure classification accuracy.

According to the PLOS One study published August 4, 2026, the researchers translated each codebook into prompt components (experiment context, theory context, classification instructions, worked examples) and ran roughly 703, 253 API calls between March 24 and April 6, 2026 across GPT 5.4, GPT 5.4-nano, Qwen 3.5 397B, and Qwen 3.5 9B (PLOS One).

According to the PLOS One study, ground truth was defined by items where two human annotators agreed, and accuracy was the proportion of items where the model matched that agreed label; the study reports 95% Wilson confidence intervals for all accuracy estimates.

Answer: Invest time improving the codebook (definitions, boundary cases, worked examples) and pair it with a capable model rather than iterating on formatting and persona prompts.

According to the PLOS One study published August 4, 2026, detailed classification instructions and worked examples drive accuracy on learning-heavy tasks, while model choice dominates recognition-heavy tasks such as promise detection.

According to the PLOS One study, surface choices like markdown formatting, role persona framing, or minor phrasing changes had negligible effects when the codebook content was complete and verbatim.

According to the PLOS One study, "The practical recommendation is to pair a frontier model with a well-defined codebook, " which means teams should prioritize clear category definitions and examples over superficial prompt engineering (Çelebi & Penczynski, PLOS One, August 4, 2026).

Answer: Follow the PLOS One recommendation: transcribe your human codebook verbatim, include worked examples, and pair it with a capable model for robust LLM classification.

According to the PLOS One study published August 4, 2026, this approach produced 82–88% agreement with human-agreed labels across three tasks and 703, 253 total classification requests during their experimental window.

If your team needs a platform to ingest codebooks, run prompt variants, and produce paired analyses and visualizations, Evidano automates those steps and preserves data security.

Get started and Try Evidano for free.