Faster Insights: qualitative analysis of AI health content

Evidano is an AI-powered qualitative data analysis platform that helps researchers reproduce and scale the PLOS ONE checks on AI health content, from ingestion through readability and quality scoring to stakeholder reports.

The PLOS ONE study (June 4, 2026) found that 18 queries produced 54 AI responses that were all above a 6th-grade reading level (p < 0.001), with Perplexity scoring highest on quality metrics and ChatGPT producing the clearest text.

• The PLOS ONE corpus: 18 queries produced 54 responses (18 × 3 models), derived from a Google Trends snapshot on March 1, 2026.
• Readability risk: all model outputs were significantly more complex than a 6th-grade reading level (p < 0.001) according to multiple indices.
• Quality vs readability: Perplexity led on JAMA, DISCERN, GQS and EQIP scores, while ChatGPT led on linguistic clarity, demonstrating a readability‑vs‑reliability trade-off.
• Reproducibility: two independent raters produced inter-rater ICCs between 0.761 and 0.999, indicating good to excellent agreement.

The PLOS ONE study selected 18 high-relevance English keywords about myofascial pain syndrome using Google Trends as of March 1, 2026, ran each query in clean sessions against three free LLM versions and saved the first response for analysis.

The PLOS ONE authors measured readability with six indices (FRES, FKGL, GFOG, CLI, ARI, SMOG) and measured quality and reliability with GQS, EQIP, Modified DISCERN and JAMA scores; two independent raters scored reliability and quality and the study reported Kruskal–Wallis, Mann–Whitney U and Wilcoxon tests plus ICC for rater agreement.

• Corpus: 54 responses (18 × 3 models).
• Readability: All models produced outputs significantly more complex than 6th-grade (p < 0.001).
• Quality vs readability: Perplexity > ChatGPT on quality metrics; ChatGPT > Perplexity on readability.
• Inter-rater reliability: ICCs ranged from 0.761 to 0.999 (good to excellent).

This seven-step checklist reproduces the PLOS ONE study and extends it to larger corpora and languages.

Step 1: Pull keywords via Google Trends and freeze the list, the PLOS ONE paper used a March 1, 2026 snapshot.

Step 2: Capture first-response outputs from each model in clean sessions and document the model versions (GPT-5.2, Gemini 3 Flash, Perplexity Sonar-4).

Step 3: Import responses into Evidano as documents or CSV files.

Step 4: Run an automated readability suite (FRES, FKGL, GFOG, CLI, ARI, SMOG) and export summary tables.

Step 5: Apply reliability and quality codes (DISCERN, JAMA, GQS, EQIP) as structured annotations inside Evidano and compute ICC between raters.

Step 6: Perform thematic coding and co-occurrence analysis to map misinformation risk areas, for example treatment advice without citation.

Step 7: Produce a stakeholder brief with recommended clinician-in-the-loop edits and plain-language rewrites, and publish the reproducible dataset and methods as the PLOS ONE authors did.

The next steps are to replicate the PLOS ONE checks (June 4, 2026) or scale them to larger corpora and languages using Evidano's pipeline from ingestion to stakeholder-ready reports.

• Import the open Figshare dataset or your own chatbot logs and run the seven-step workflow above.
• Use Evidano to operationalize clinician-in-the-loop validation and produce reproducible reports that meet audit needs.
• Security note: Evidano encrypts customer data and does not use it to train third-party models, which is important for sensitive health research.

Ready to move from anecdote to evidence? Try Evidano for free or import the PLOS ONE Figshare dump to reproduce the study in hours, not weeks.