Evidano is an AI-powered qualitative data analysis platform that converts transcripts, tags judgment signals, compares segments, and produces stakeholder-ready visuals. Per recent research (published 17 Jun 2026), users often perceive mental health chatbots as "judgmental" even when messages are identical to human-delivered text. This post shows how to run a targeted qualitative analysis of mental health chatbots to surface where perceived judgment emerges, measure its frequency across segments, and iterate conversational prompts. You’ll get a reproducible workflow and concrete coding checks you can run in Evidano to convert transcripts, tag judgment signals, compare segments (age, risk level), and produce stakeholder-ready visuals fast.
Key Takeaways
Labeling identical therapy-style messages as coming from a chatbot increases perceived judgment and can reduce trust compared with the same messages labeled human. Implement a reproducible qualitative audit to find judgment triggers, compare segments, and iterate prompts to reduce perceived judgment. Use transcription, codebooks, cross-segment comparisons, and safe routing to turn findings into product changes and measurable trust improvements.
- A study led by Ryan Raimi reported on 17 Jun 2026 and using roughly 2, 000 participants found that labeling alone made identical responses seem more judgmental.
- Run a 7-step workflow (collect, transcribe, auto-extract candidate phrases, build a codebook, cross-segment analysis, surface edits, implement routing) to audit and reduce perceived judgment.
- Measure changes by comparing theme frequency across label condition, age, prior therapy experience, and risk scores, and prioritize safety triage for higher-risk users.
Fast take & source
Researchers led by Ryan Raimi at the University of Texas at Dallas found that when nearly 2, 000 participants were told identical therapy-style messages came from a chatbot, the participants rated those messages as more judgmental than when labeled human. Full story: TechTarget (17 Jun 2026).
- Why it matters: perceived judgment reduces trust and may increase isolation for users seeking help.
- Key empirical anchor: approximately 2, 000 participants in an MIS Quarterly study; labeling alone shifted perceptions.
- Policy context: only 52.1% of adults with mental illness received treatment in 2024 (NAMI), so chatbots are a promising access channel, but design matters.
Findings snapshot
| Metric | Value | Source | Implication |
|---|---|---|---|
| Study sample | ≈2, 000 participants | MIS Quarterly / Raimi et al. (reported 17 Jun 2026) | Sufficient scale to detect labeling effects on perception |
| Label effect | Identical messages perceived as more judgmental when labeled 'chatbot' | MIS Quarterly | Design and labeling changes can reduce harm without changing content |
| US treatment gap (2024) | 52.1% adults with mental illness received treatment | NAMI | Access gap motivates safe chatbot deployment |
| Stigma indicator | 84% believe 'mental illness' carries stigma (2025) | American Psychological Association | High stigma increases sensitivity to perceived judgment |
What happened (plain English)
Raimi's team found that presenting identical messages as if authored by a chatbot led participants to rate those messages as more judgmental and less validating. The researchers presented participants with the same therapist and client message pairs but varied whether the responder was described as a human therapist or a chatbot, and labels changed perception despite identical wording.
- Qualitative probing revealed users expect humans to draw on lived context and validate feelings, machines lack that lived-experience framing.
- Standard empathic phrases like "I know how you feel" ring hollow when uttered by a machine; participants preferred transparent, evidence-based language.
- Raimi recommends prompt engineering changes, for example explicit transparency such as "I don't have personal experience, but here are findings from others, " and risk-based routing for higher-risk users.
So what for researchers & UX teams
UX researchers
UX researchers should run targeted qualitative analysis to locate 'judgment triggers' in chat transcripts. Run targeted qualitative analysis to locate 'judgment triggers', phrases, or response styles that spike negative ratings in chat transcripts.
Segment reactions by age, prior therapy experience, and self-reported risk; labeling effects often vary by cohort.
Use A/B labeling tests in labs and remote diaries to validate whether design changes reduce perceived judgment.
Product & PMs
Product and PM teams should treat labeling as a feature and expose provenance and model limits rather than hiding them. Treat labeling as a feature: expose provenance and model limits rather than hiding them.
Implement guardrails: automatic escalation for high-risk language and clearer disclaimers that avoid false empathy.
Measure success not only by engagement but by trust metrics such as felt validation and willingness to seek follow-up care.
Clinical & policy teams
Clinical and policy teams should require safety triage so chatbots handle only low-risk interactions and refer others to clinicians. Require safety triage: chatbots should handle only low-risk interactions and refer others to clinicians.
Mandate logging of worked examples, that is what responses reduced distress, for auditability and ongoing improvement.
Prioritize informed consent and clear statements about the chatbot's limitations.
Do more, faster with Evidano
Problem: messy, multilingual transcripts
Use Evidano transcription and translation with custom dictionaries to produce clean, research-grade transcripts from audio or chat logs before coding.
Problem: finding 'judgment' signals across thousands of messages
Use Evidano thematic and frequency analysis to flag candidate phrases, and run co-occurrence networks to see what language clusters with negative ratings.
Problem: comparing segments reliably
Use Evidano cross-segment analysis to export side-by-side theme frequency by age, risk level, and labeling condition so you can demonstrate differential effects to stakeholders.
Problem: inconsistent coding
Import a codebook into Evidano, apply AI-assisted coding, review disagreement cases, then produce hierarchical code to subcode visuals for executive reports.
Security & compliance
Evidano encrypts data end-to-end and does not share your data to train third-party models, which is critical when handling sensitive mental health material.
Checklist: 7-step workflow to audit perceived judgment
This checklist provides a 7-step workflow to audit perceived judgment in conversational agents and chat transcripts.
Step 1: Collect a stratified sample of chat transcripts and any human versus chat labeled A/B messages (n ≈ 500–2, 000 per condition).
Step 2: Transcribe and normalize text with Evidano, apply a custom therapy dictionary, and redact PII.
Step 3: Run an automated pass to extract candidate 'judgment' phrases via keyword and sentiment frequency.
Step 4: Build a codebook (judgment, validation, transparency, referral) and apply AI-assisted coding; review edge cases manually.
Step 5: Cross-segment analysis: compare theme frequency by label (chatbot vs human), age, prior therapy experience, and risk score.
Step 6: Surface top co-occurring phrases and sample quotes for design teams; run prompt edits in a sandbox and re-test.
Step 7: Implement safe routing and revised phrasing; monitor trust metrics and drop in perceived judgment over time.
FAQ: qualitative analysis of mental health chatbots
What counts as a 'judgment' signal?
A 'judgment' signal is language that corrects, minimizes, or offers canned empathy that feels inauthentic. Common markers include corrective language such as "You shouldn't...", minimizing phrases like "It's not a big deal", and canned empathy that assumes experience; tag both phrase-level and contextual cues.
How do I compare segments reliably?
Use stratified samples and normalized rates to compare segments reliably. Use stratified samples and Evidano cross-segment frequency analysis to produce normalized rates (for example per 1, 000 messages) and confidence intervals for comparison.
Is this research ethical for sensitive users?
Yes, auditing chat transcripts can be ethical when run under consent and secure protocols. Run the research with consent, PII redaction, secure storage, and clinical handoffs for high-risk cases; this guidance is research-focused and non-diagnostic.
Wrapping up & next steps
Raimi's findings reported on 17 Jun 2026 show labeling and phrasing shape whether chatbots build or erode trust. If your team is deploying conversational agents for mental health, run a focused qualitative audit now, tag judgment triggers, compare segments, and iterate prompts with measurable objectives.
- Try the 7-step workflow above in a pilot using Evidano to get rapid transcripts, thematic and cross-segment analysis, and exportable visuals.
- If you want a ready-to-run template, Try Evidano for free and adapt the codebook to your clinical safety protocols.