Sentiment analysis has an appealing pitch. You've got four hundred open text responses that nobody has time to read, and a machine will read them all and hand you a number. Seventy two percent positive. It sounds like the problem of unreadable feedback has been solved by arithmetic.
What's actually happened is that you've converted your least structured, most informative data into your most structured, least informative data. The whole value of an open answer is that the customer chose their own words and told you something specific. A sentiment score takes that and throws away every part of it except the mood.
Consider two responses. "Lovely staff, but the app crashed twice while I was trying to check in and I nearly gave up." And: "Fine, I suppose. Nothing wrong with it." The first is broadly positive in tone, contains a specific bug, and describes a customer who almost failed to complete a transaction. The second is neutral and contains nothing at all. A sentiment model will likely rate the first higher than the second, and in one narrow sense that's correct. But you'd want to act on the first one this afternoon and the second one never, and the score puts them the wrong way round for the purpose you actually have.
The mood was never the interesting part
Here's the thing that gets lost. You mostly already know the mood. Your star ratings tell you the mood. If you're running any kind of structured question alongside your open box, you have a perfectly serviceable measure of how people feel, and it's more direct than inferring it from prose.
What you don't know is the content: what happened, which part of the experience, and what would have changed it. That's exactly the information a sentiment score discards. So the metric duplicates something you had and destroys something you didn't, which is a strange trade to make voluntarily.
There's a version of this failure that shows up in dashboards everywhere. A sentiment trend line, sitting next to a rating trend line, moving in near lockstep, because both are measuring the same underlying thing through different instruments. Two charts, one fact. It looks like a richer picture and it's the same picture drawn twice.
Where the models genuinely struggle
Even taken on its own terms, sentiment scoring is unreliable in the specific situations small businesses generate most.
Sarcasm and dry understatement break it. "Brilliant, waited forty minutes" is not positive. British and Irish feedback in particular is full of this, and models tend to read "not bad at all" as tepid when it's genuine praise.
Mixed responses get flattened. Most useful feedback is mixed, because a customer who liked one thing and disliked another is telling you something more precise than someone who liked everything. Scoring that as sixty percent positive is a description of the average of two facts, and the average of two facts is not a fact.
Domain language misleads it. In a dental practice, "the injection hurt a bit" is a normal description of a routine appointment, not a complaint. In a restaurant, "the sauce was sharp" might be a compliment. Sentiment models don't know your trade.
And negation with distance in the sentence trips them up more than you'd expect. "I wouldn't say the room was clean" is a complaint that reads, superficially, like a mild statement about cleanliness.
None of this means the technology is bad. It means a single number derived from it carries error bars nobody displays, and then that number gets a decimal point and a target attached to it. False precision in feedback metrics covers what happens next, and it's the same story as NPS: a real measurement gets treated as more exact than it is, and decisions get made on movements that are inside the noise. Why NPS is probably lying to you is the longer version of that argument.
What to want instead
The useful thing an AI can do with four hundred open responses isn't to score them. It's to tell you what they say.
Themes, with counts and examples. "Eleven people mentioned the wait at the counter, mostly at lunchtime, here are four of them in their own words." That's the output that does the job the sentiment score was pretending to do. It preserves the specifics, it points you at something you can fix, and it lets you go and read the original responses when you want to. The count is the metric and the quotes are the evidence, and neither of them requires you to trust a model's read on tone.
Summaries have their own failure modes, and you should know them. They smooth over the unusual single response that mattered, and they can make a business sound tidier than it is. So the summary is a way into the responses rather than a replacement for reading them. AI customer feedback analysis goes through what the current tools do well and where they're still weak.
This is how Qria handles the same problem. It reads across the structured feedback you collect and the public reviews you're getting on Google, Yelp and the rest, and produces a plain language summary of what customers keep saying rather than a mood percentage. On the Pro plan you can ask it questions directly, which in practice is where the value is: "what did people complain about at the Leeds site last month" gets you an answer built out of actual responses, with the responses still there to check.
When a score is fine
Two cases where I'd happily use one.
If your volume is genuinely large, tens of thousands of responses across many locations, an aggregate tone measure is a reasonable way to decide which location or which month to go and read properly. As a triage instrument pointing at where to look, it's fine. The mistake is treating it as the finding rather than the pointer.
And if you're tracking one specific change over a short period with everything else held constant, a tone measure will move if the change was significant. Coarse, but not meaningless.
What I'd avoid is putting a sentiment percentage on a wall, giving someone a target for it, and reporting it monthly to people who won't read the underlying responses. That's the point where the number stops describing your customers and starts being the thing you manage instead of them. For the wider question of which metrics to keep at all, NPS, CSAT and CES compared is a reasonable place to start.
