Researchers and UX teams face two recurring barriers when studying how people use generative AI for search: articulation (users can't describe problems) and discoverability (users don't know what AI can do). In NN/g’s Aug 22, 2025 article, Kate Moran shows real user tests where AI both helped and fell short, e.g., one participant spent nearly 15 minutes failing to find a fix until Gemini suggested keywords and steps (source: www.nngroup.com/articles/ai-information-seeking-keyword-foraging/). This post explains how to run a practical qualitative analysis of AI information-seeking to surface where prompts, UI, and training data break down, and how to close those gaps with Evidano (www.evidano.com). Read on to get a compact workflow, quick metrics to track, and exact Evidano features to map problems to product fixes.
Fast take + source
In brief: NN/g (Aug 22, 2025) documents two consistent failure modes for AI-assisted search, users can’t name what they need (articulation) and they don’t discover AI affordances (discoverability). The article includes a concrete user session where a participant spent ~15 minutes before an AI chat produced usable keywords and action steps (www.nngroup.com/articles/ai-information-seeking-keyword-foraging/).
- Primary insight: Generative AI reduces simple 'keyword foraging' but does not eliminate higher-order articulation problems.
- Practical implication: Qualitative analysis must code for prompt detail, AI follow-ups, information scent, and whether the AI supplies actionable links.
- Link: Read the original NN/g article at www.nngroup.com/articles/ai-information-seeking-keyword-foraging/.
Findings snapshot
| Date | Metric / Observation | Value / Example | Source / Note |
|---|---|---|---|
| 2025-08-22 | Article publish date | Aug 22, 2025 | NN/g |
| User testing | Typical failure mode | Articulation + discoverability barriers | Participant quotes in NN/g article |
| Task time | Example session length before success | ≈ 15 minutes (participant nearly gave up) | NN/g user session |
| AI behaviors | Where AI helped | Provided keywords, step-by-step guidance | Gemini example in article |
| AI behaviors | Where AI failed | Did not ask clarifying follow-ups or provide direct links | NN/g analysis |
What happened: key mechanics
NN/g observed that generative AI accepts verbose, natural-language prompts and can rescue users who lack domain vocabulary (the 'keyword-foraging' problem). However, success depends on two things the user often lacks: (1) experience in crafting prompts and (2) awareness of AI features (e.g., follow-up Q&A, ask-for-links). In the plumbing vignette, an imprecise prompt produced a long, mixed-response; only after iterative prompting did the user obtain useful search terms and steps.
- Keyword foraging: Users run preliminary searches to discover the right terms; AI reduces but doesn’t remove this need.
- Articulation barrier: When users can’t describe their situation, AI should ask clarifying questions but often does not.
- Discoverability barrier: Users don’t know which AI behaviors or follow-up prompts are possible, so they underuse capabilities.
How to run a qualitative analysis of AI information-seeking
Step 1; Capture sessions and prompts
Collect screen recordings, full chat transcripts, and any pre-chat search queries. Include explicit metadata: participant experience with AI, device, and task context.
Why: NN/g shows that what users type (and what they omit) is diagnostic of both articulation and discoverability failure modes.
Step 2; Code for four core phenomena
Codebook suggestions: articulation (missing details), prompt iterations, AI follow-up behavior (asked question vs. monologue), and actionability (links, videos, product options).
Why: These categories map directly to the usability gaps NN/g identifies.
Step 3; Cross-segment analysis
Compare novice vs. experienced AI users, mobile vs. desktop, and task types (repair, shopping, health info). Look for where AI supplies keywords vs. where it supplies links or clarifying Qs.
Why: NN/g shows the same model (e.g., Gemini) can succeed for some tasks and fail for others (segmentation reveals patterns.
Step 4) Measure time-to-usable-answer and abandonment
Track time to first actionable recommendation (link, keyword, step) and instances where participants consider giving up (abandonment markers).
Why: In the NN/g example, nearly 15 minutes of effort preceded a usable result: that delay is measurable and targetable.
Implications for researchers and product teams
Translate codes into product decisions: prioritize clarifying question flows, surface quick-action links (videos, purchases), and provide adaptive prompts for novices. The NN/g analysis implies that simply improving model accuracy isn't enough, the UI must guide users to reveal critical context and must make AI affordances discoverable.
- Design: Add affordances that prompt users to give context (location, expertise level, constraints).
- Metrics: Track prompt iterations, clarifying-question rate, time-to-first-link, and successful completion.
- Policy/ethics: For sensitive domains (e.g., health), flag that guidance is research-oriented and non-diagnostic; route to specialists when necessary.
Do more, faster with Evidano
Ingest labeled transcripts and prompts
Evidano ingests chat transcripts, search logs, and screen-recording transcripts (with PII redaction). This centralizes prompts and AI responses so you can code for articulation and discoverability at scale.
Relevant features: transcription with custom dictionary and PII redaction; upload transcripts and search logs (www.evidano.com).
Automate thematic + cross-segment analysis
Run thematic, frequency, and cross-segment analyses to see which user cohorts fail to get clarifying questions or actionable links. Use hierarchical codes → subcodes to map 'articulation' → 'missing location' vs 'missing expertise level'.
Relevant features: thematic, content, frequency, and cross-segment analyses.
Surface UI fixes as data-driven playbooks
Turn coded patterns into product plays (e.g., add a 2-question clarifier before long-form prompts for novice users). Export findings with clickable quotes and co-occurrence networks to align stakeholders quickly.
Relevant features: AI chat over your corpus, visualizations (co-occurrence network, hierarchical code trees).
Secure, research-grade controls
Keep participant data encrypted and private; Evidano’s models are proprietary and data is never used to train third-party models, a critical assurance when handling interview transcripts that include sensitive context.
Quick checklist: pilot in 2 weeks
Use this run-book to reproduce NN/g-style findings and move to fixes quickly.
- Week 0: Gather 10–20 recorded sessions (search logs + chat transcripts).
- Week 1: Import into Evidano, run auto-theme extraction for articulation and discoverability codes.
- Week 2: Produce segment comparisons, time-to-action metrics, and a one-page stakeholder brief with recommended UI changes.
- Deliverable: prioritized fixes (clarifying Qs, tippable links, novice prompt templates).
FAQ: qualitative analysis of AI information-seeking
Q: How do I ensure transcripts capture prompts accurately?
A: Use Evidano transcription with a custom dictionary (product names, domain terms) and enforce session metadata capture (device, user AI experience).
Q: How do I compare segments (novice vs expert)?
A: Run cross-segment frequency and co-occurrence analyses in Evidano to see which themes (e.g., missing location) correlate with longer time-to-answer or abandonment.
Q: Is this safe for sensitive topics?
A: Treat outputs as research-oriented, non-diagnostic. For domains like health, include an ethics note and escalate to subject-matter experts when needed.
Conclusion & next steps
NN/g’s Aug 22, 2025 piece highlights a clear, actionable research problem: AI helps with simple keyword gaps but often fails where users can't articulate their situation or don't know what AI can do. A focused qualitative analysis will reveal whether failures are model, prompt, or UI-driven, and it gives product teams a direct path to fixes.
- Next move: Run a 2-week pilot that captures prompts, AI replies, and timestamps; import into Evidano and produce a stakeholder brief with prioritized UI plays.
- Start your pilot: learn how at www.evidano.com and bring your transcripts, search logs, and session metadata to see where AI succeeds, and where it needs a better interface.
Keep reading
- Commentary on NewsTwo Definitions: Climate Change Acceptance for UndergradsHow a PLoS One Delphi study (Aug 25, 2026) defined climate change acceptance for undergraduate science students, and how AI-enabled qualitative analysis applies it.
- Commentary on NewsResearcher-in-the-loop: AI-enabled UX researchHow the researcher-in-the-loop model governs AI-enabled UX research. Learn practical governance, stats from the August 2026 piece, and how Evidano supports this workflow.
- Commentary on NewsResearcher-in-the-Loop: Governance for AI UX ResearchGovern AI in qualitative UX research with the researcher-in-the-loop model from Jennifer L. Bowie (Aug 25, 2026): practical rules, risks, and tool mappings.