Review and Feedback Mining: analysing text you did not collect

The strength of this material is that nobody asked for it. Reviews and tickets are written for the writer's own purposes — warning others, obtaining a refund, venting — which means they surface issues no research instrument thought to ask about, in the customer's own vocabulary.

The cost is selection. People write reviews at the extremes; they open tickets when something breaks; they fill in the free-text box when they have a grievance. The corpus is not a sample of customers and cannot be treated as one, which rules out the most tempting claim: "37% of our customers are unhappy with onboarding" is not supportable from a review corpus, whatever the arithmetic says.

What the corpus does support is a claim about the issue space: what kinds of problems exist, how customers describe them, what vocabulary they use, and how the composition of complaints changes over time. Those are real findings and they are enough to be useful.

The most rigorous treatment of using text corpora in consumer research is Humphreys and Wang's Automated Text Analysis for Consumer Research in the Journal of Consumer Research, which sets out when automated approaches are defensible and what validation they require. It is the paper to read before building any pipeline.

Netzer and colleagues' Mine Your Own Business: Market-Structure Surveillance Through Text Mining demonstrates that competitive market structure can be inferred from forum text — the strongest published demonstration of what found text can support beyond complaint counting.

An earlier applied treatment of forum analysis is available as Using text mining to analyze user forums, and for the cultural dimension of online discussion Kozinets's The Field Behind the Screen remains the reference — analysing an online community as a community, not just as a text source.

Sentiment classification is the default first move and the weakest part of most pipelines. It is worth being clear about its failure modes rather than abandoning it.

• Sarcasm and irony invert it. "Brilliant, another update that logs me out" scores positive on most classifiers.
• Negation and scope are handled unevenly. "Not what I would call fast" is routinely misread.
• Domain terms carry the wrong polarity. "Sick", "insane", "ridiculous" are positive in some product categories and negative in others.
• Mixed documents get averaged. A review praising the product and condemning delivery is not neutral; it contains two findings, and the average destroys both.
• A score is not a reason. Knowing sentiment fell tells you nothing actionable. The category composition does.

Sentiment is useful as a triage signal — sort the negative material to the top of the reading queue — and it is not a finding. Reporting a sentiment score as the headline of a feedback analysis is the equivalent of reporting a satisfaction average from a usability test.

A software company analysed 14,000 support tickets and 3,200 app-store reviews spanning six months either side of a major release.

The corpus definition stated plainly that it excluded the roughly 60% of users who had never contacted support, and that app-store reviewers skewed to one and five stars.

Inductive coding on 400 documents produced eleven categories, one of which — "cannot find where X moved to" — used customers' own framing rather than the internal category "navigation". That distinction mattered: the internal taxonomy would have merged it with genuine navigation defects, and the fix is different.

Automated classification across the full corpus was validated against 300 hand-coded documents, with agreement reported at 0.81 and the disagreements characterised: the classifier over-assigned the billing category to any document containing a currency symbol.

The composition analysis was the finding. The "cannot find where X moved to" category ran at 3% of tickets before the release and 22% for five weeks after, then decayed to 6% — where it stayed, well above baseline. The residual 6% was concentrated in accounts created more than two years earlier.

Close reading of that residual gave the actionable detail: those users were navigating from muscle memory to a location that no longer existed and had no redirect. The aggregate said where to look; the individual tickets said what to do. Neither would have been enough alone, and no claim was made about how many customers were affected — only about what share of the people who complained were complaining about this.

• Treating the corpus as a sample of customers. The single most consequential error, and it produces confident numbers about a population that was never surveyed.
• Sentiment score as the finding. It is triage, not a result.
• No validation of automated coding. An unchecked classifier is an assumption at scale.
• Importing the internal taxonomy. Loses the customer vocabulary that is the data's distinctive value.
• Ignoring fake and incentivised content. Especially in public review corpora.
• Aggregate without close reading. Counts locate the problem; the text explains it.
• Comparing across platforms without adjustment. App-store reviews, support tickets and survey free text have different populations and different norms, and pooling them merges three biases.

The reporting requirements are specific: the corpus definition with dates, sources and exclusions; an explicit statement of who is absent; whether coding was inductive or applied, and how any automated step was validated; results framed as composition of the corpus rather than prevalence in the population; and time-series comparisons made against a defined baseline.

A feedback analysis that reports percentages without saying "of complaints" is the one to send back.

Selection bias is structural and cannot be corrected, only stated. No weighting scheme recovers the silent majority from a corpus of self-selected writers.

The data is also thin per document. A review gives you an outcome and a complaint, rarely the context, the alternative considered or the sequence of events — which is why review mining identifies where to investigate rather than concluding an investigation.

Platform changes break comparability: a review prompt redesign, a support-form change, or a moderation policy shift will move the composition without anything changing in the product. And public review corpora carry terms-of-service constraints on collection and storage that need checking before a pipeline is built.

Call or chat log analysis is the same operation on a private corpus, with better context and worse coverage.

Social listening extends it to public conversation not addressed to the company.

Voice-of-customer interviews supply the context reviews lack, and are the right follow-up once mining has located the issue.

Summative content analysis is the academic counterpart — counting occurrences, then interpreting the context of use.

This is the methodology with the largest corpus and the strongest case for tooling: 17,000 documents cannot be read, and inductive coding on a sample followed by consistent application at scale is exactly the shape of work a qualitative platform handles.

The two things worth insisting on are that the coding scheme comes from the data rather than from a stored taxonomy, and that a hand-coded validation sample is checked against the automated output and reported. Evidano supports review and feedback mining as a named methodology and handles spreadsheet and free-text corpora at this scale. Defining the corpus honestly — and saying who is missing from it — is the researcher's responsibility, and it is what determines whether the analysis is evidence or a confident-looking artefact of who happened to complain.