1. Start with a small thing
Everyone selling on Amazon knows reviews deserve a close reading, but when it is your turn to scroll through a few hundred of them, your brain goes numb around number twenty, the first twenty you still note down which one mentions the coating and which one the power supply, after that you are basically skimming, and in the end your judgment rests on impression.
AI tools are everywhere now, you feed the reviews in, run them once, and the output looks presentable, but most people drop the tool after one use, because when they run it again the next day the issue categories come back different, same batch of reviews, two runs, two stories, when the tool cannot agree with itself, who would dare make a decision on its output.
Over the past few months I have run 30 reports across 20 categories, more than eleven thousand reviews in total, all of it processed by tools, and I have stepped in most of the potholes along the way, so today, drawing on these 30 reports, let me lay out what a tool can actually do for you, what it cannot, and where the real threshold sits.
First let me explain why results differ from run to run, because this is what blocks most people.
A large language model is not a database, every time it reads the same batch of reviews it reorganizes the answer from scratch, model versions keep updating, and the category framework is something it improvises on the spot, this run it decides battery issues and slow charging belong in one bucket, next run it decides they belong in two, when your conclusions shift, the model did not compute anything wrong, it rebuilt the foundation under your feet, the definition of issue categories itself.
Most people use the tool like this, export the reviews, toss them to an AI, ask it to summarize the main problems, the AI hands back a polished paragraph that covers everything, you screenshot it and call it done, the next time you ask again and it hands back another polished paragraph, both sound right and the two contradict each other, the problem is you treated it as a fortune teller, when it should be working an assembly line, a line worker needs a fixed station and fixed motions, not improvisation.
So a tool only produces results once you nail down the foundation, everything below is about how to nail it down.
2. Three things a tool can do for you
First, the manual labor.
Reading a few hundred reviews one by one without missing any, no need to argue this one, a person tires out at twenty, a machine reads twenty thousand without panting, and numbers computed by a program never come out shaky or miscopied, anyone who has organized a spreadsheet by hand knows, one percentage copied wrong and the whole analysis behind it is wasted.
There is one step inside this labor layer that people skip, cleaning, the reviews that come back are not directly usable, there are duplicates, image-only reviews, obvious incentivized and ad reviews, and entries where the star rating does not match the text, skip this and your denominator is dirty, every one of our reports opens with the sampling basis, the air fryer report for example states 438 text-bearing reviews, that number is counted by the program, not estimated, and it reconciles end to end.
One boundary has to be stated too, our sampling leans toward recent reviews and negative ones, so the complaint density in a report is naturally higher than the platform overall, every percentage in the report uses this sample as its base and describes the structure inside the negative reviews, align on this before reading any number, otherwise you will read 15% as a platform-wide negative rate.
Second, the measurement stick.
An AI left to improvise cannot produce stable results, let it invent its own categories and every run hands you a different frame, last week it made seven buckets, this week nine, you have no way to line up two reports side by side.
The fix is to freeze the labels, every report uses the same stick, that is the only way the complaint structure of the feeder report and the air fryer report can be compared, data only accumulates when the ruler stays constant, otherwise every report talks to itself.
Inside the measurement stick the deadliest part is the denominator, the same batch of negative reviews can be measured as a share of pain-point mentions, of negative-review counts, or of total reviews, three denominators, three results, each defensible on its own, mixing them is self-deception, our current rule is that the pain-group summary uses pain-point mentions as its base while the physical-cause table uses negative-review counts, both are labeled in the report, so the reader knows which ground each percentage stands on.
Third, structure.
Which component concentrates the complaints, which months they suddenly spike, what the positives praise, what the top return reason is, put these in a table and you see them at a glance, a human scrolling through a few hundred reviews cannot see structure, scroll long enough and everything dissolves into impression.
The most typical example, in the air fryer report Cosori sits at 4.8 stars on the platform and Chefman at 4.4, a gap of 0.4, but take the 438 reviews apart and the negative rates are 15.1% versus 30.5%, a factor of two, this information will never appear on the listing page, the score is cumulative and diluted by positives, structure is what happened recently.
Monthly trends push this even harder, Chefman's single-month negative rate was 29.4% in August, spiked to 61.5% in September, fell back to 35.7% in October, word of mouth doubled and half-collapsed inside a single month, none of that movement shows in a 4.4 average, if you only watch the star rating, by the time something feels wrong the container may already be on the water.
Structure has one more play, hanging a second data line next to the reviews, the returns, in the air fryer report Cosori's return-mention rate is 7.17% and the top stated reason, at 21.1%, is exactly the coating material mismatch, which lines up with those PTFE complaints in the reviews, Chefman's return-mention rate is 9.09% and its top reason, at 55.6%, is the unit stopping working, matching its own premature-failure grouping, when two independent data chains point at the same spot you can finally land a conclusion, and at an assumed thousand units a month the estimated monthly loss is $7,169 for one and $7,266 for the other, nearly ninety thousand dollars a year, which a revised listing page might have partly plugged.
3. Three category-level findings from stacking 30 reports
One report shows you a product, thirty reports stacked show you a category, none of the findings below are visible in any single report, they only surface once the measurement stick stays constant.
First, ratings are barbell-shaped.
Lay the rating distributions of every product across 30 reports side by side and the median share of 4-star reviews comes in under 10%, while 5-star plus 3-star together make up about 67%, buyers simply do not score on five levels, they use three, 5 stars when delighted, 3 stars when reserved, 1 to 2 stars when something broke, the 4-star tier is nearly empty.
This directly changes how you read a competitor's rating, a 4.4-star product almost certainly has few genuine moderates in its reviews, it is full of devotion and rage, the platform average flattens two different kinds of pain into one decimal, we call it the false smooth, only by pulling out the distribution do you see what is buried under 4.4.
It also reaches into your review operations, if the 4-star tier is basically unused, then the five stars harvested by request-a-review emails deserve suspicion, when buyers score spontaneously there is almost no middle state between satisfied and unhappy, and blending solicited five stars with earned five stars makes the average dirtier, not a proven law, but across 30 reports the shape is too consistent to ignore.
Second, categories differ enormously in negative rate.
Weighted by review counts, 20 categories line up into a table, automatic litter boxes highest at 47.5%, Bluetooth speakers lowest at 11.8%, with translation earbuds at 39.3%, smart feeders at 37.3% and power banks at 30.7% in between, no single report can tell you whether a category is hard or forgiving overall, and for product selection this table adds a dimension the sales ranking never shows.
The table also corrects instincts, take humidifiers, everyone assumes Q4 seasonal demand with moderate competition, yet the category's negative rate is 20.2%, rank 14 of 20, not dangerous but nowhere near safe, one in every five reviews is a negative, and peak-season traffic amplifies every one of them, while Bluetooth speakers, which sound like a red ocean, hold the lowest negative rate on the site, whether a category is easy has little to do with feel.
Third, the same dataset tells two stories at two levels.
The air fryer report is the cleanest case, at the group level Cosori's top pain is the long tail and Chefman's is premature failure, two entirely different directions for the improvement budget, but at the granular level the most frequent item for both products is the same sentence, the plastic clip on the drawer broke, 7.5% for one and 26% for the other.
The group level tells you which direction the money goes, the granular level tells you which part to fix, both are true and mixing them makes them fight, this trap is invisible to the naked eye, it only surfaces when the classification frame is frozen and every category must be traceable to a buyer's own words, and where the two levels disagree often turns out to be where the truth lives.
4. The one thing a tool cannot do
Judgment.
Which problem is the main cause, where the improvement should start, a tool cannot make this call, a person has to.
Take the feeder report, across 397 reviews, dispensing accuracy turns out to be the best-scoring dimension for both products, which says the mechanical act of dispensing kibble carries a low bar, and the problems all pile up at the connectivity layer, the top pain group for both products is connectivity, at 40% and 48% of their negatives, and inside it, server outages that block dispensing account for 15 mentions and skipped feedings during network outages for 10, the tool can lay these numbers out spotless, but it cannot conclude that the dispensing mechanism needs no touch and the money belongs in connectivity fail-safes, that sentence weighs mechanics, networking and cost on the same scale, and what is being weighed is the business, the tool only sees words.
Another example from the air fryer report, the tool lists coating as one of Cosori's pain points, 3 mentions, but the buyer who pulled the basket out and said the surface feels like PTFE while the listing promises ceramic, chased one layer deeper the issue leaves the quality-control frame entirely and lands on the advertising-promise layer, revising copy and revising a coating cost completely different amounts, that judgment is made by a person, the tool only saw a word.
To be fair though, the tool does not make the judgment, it walks the judgment to your feet, every report ends with a sensitivity estimate, remove the negatives tied to the top pain point and re-average the rest, Cosori's negative rate would fall from 15.1% to 4.8%, Chefman's from 30.5% to 5.3%, meaning the bulk of both products' complaints indeed sits in one place and the ceiling of fixing it is right there on the table, whether it is worth a new mold, you do that math with the ceiling in front of you, the tool lays out the ledger, you sign it.
The tool saves the labor, the decision is still yours, I have said this line through 30 reports now.
5. The real threshold sits in two steps
The question I get asked most is whether someone could build this themselves, the answer is yes, but the real threshold is not code, it is these two steps.
Acquisition.
Review data has been locked down year by year, everyone in this business has tasted what bot defenses can do to a scraper, this water runs deep and I will not detail how we run ours, only that at choicesages the reviews arrive on their own now, a user submits an ASIN and the rest runs unattended, reviews come in by themselves and turn into a report by themselves, and the effort is worth it, because for any third party the review section is almost the only public large sample available, return reasons sit in a private backend, customer-service records sit in inboxes, the complaints about someone else's product in your category are simply invisible, reviews are the only first-hand pain-point data reachable across brands.
Classification.
This step looks easy and is in fact where the whole pipeline breaks most often, and the two traps point in opposite directions.
One is merging different problems into one pile, a buyer who says kibble gets stuck and one who says nothing comes out are both feeding issues to a tool, merge them and the conclusion skews, the other is splitting one problem into pieces, a buyer who says the feeder is offline, cannot reach the app, shows no connection, is describing the same hole three ways, leave them unmerged and each share is diluted, in the feeder report, had the three phrasings of the connectivity cluster been scored separately none would have cracked the top three, merged they hold forty percent, the difference between the top pain point and a footnote is one classification decision.
After stepping in both traps I set one rule, label names may not be invented, they must be phrases that actually appear in buyer reviews, every category that comes out of the grouping must be traceable back to a buyer's original sentence, a category that cannot be traced is one the tool invented, delete it and regroup.
One more thing people overlook, what to do with 3-star reviews, they are full of gripes yet never reach negative territory, mixing them in dilutes the top pain point's share, throwing them away wastes them, our approach is keyword tagging as reference only, never entering any conclusion, and this boundary is written into every report's data notes, which layer feeds conclusions and which is reference only, in black and white.
Clear these two steps and the tool's output finally holds, fail them and every bit of labor you saved gets handed back.
6. What the whole thing looks like now
The current shape is simple, submit an ASIN, the system pulls reviews, runs the clustering and produces the report, the report carries the negative structure, the pain-point breakdown, the dimension scores, the return-reason ranking and the monthly trend, from submission to finished report nobody has to babysit anything.
Each table has its own job, the pain-point breakdown shows what buyers are attacking and at what share, the dimension scores split the product into facets so the weak facet shows at a glance, the return-reason ranking lays out what return shipping is actually paying for, and the monthly trend watches the direction of word of mouth, fitted together these tables put a competitor's fundamentals on one page.
All reports are public, they sit on the choicesages public report pages, 30 of them now across 20 categories, humidifiers, electric toothbrushes, shavers, earbuds, power banks, feeders, litter boxes, air fryers and the rest, every number reconciles against the review sections, take any one and check it against Amazon yourself.
One suggestion on usage, do not read only your own report, put it next to the one covering the category's best seller, and compare the two pain-point tables, complaints that appear in yours but not theirs are your unique problems, fixing those turns into selling points, complaints that appear in both are the category's floor problems, the whole industry aches together and the winner is whoever fixes them first, reading tables is enough, no technical background needed.
7. In the end
What a tool saves, in the end, is the labor, the judgment stays yours, but turning a few hundred reviews into a table you can actually read does save most of an evening, and it matters that the work does not tire you out, tired people skip steps, machines do not.
Spend the saved hours on competitor listings and improvement directions, that beats scrolling reviews one by one every time.
Tell me in the comments which category the next report should take apart.---
Delivery checklist (internal)
- Title width: "What Can an Amazon Review Analysis Tool Actually Do? A Look Back After 30 Reports" = 76 chars (H1 can be shortened at publish: "What an Amazon Review Analysis Tool Can Actually Do" = 52 β use "What Amazon Review Analysis Tools Can Actually Do for You" = 58)
- Word count: see verification below
- Data anchors verified against live reports 2026-10-30: air fryer b0c33chg99 (438, 4.8/4.4, 15.1%/30.5%, monthly 29.4/61.5/35.7, returns 7.17%/$7,169 & 9.09%/$7,266, PTFE 3 mentions, return reason #1 coating mismatch 21.1%), feeder b0b5zjy9mt (397, dispensing best dimension both, connectivity 40%/48%, server outages 15, skipped feedings 10), category rates from stats set (30 reports / 20 categories / 11,685 reviews; humidifier 20.2% rank 14; charging banks 30.7%)
- Feeder count uses 397 (total_reviews basis), not the legacy 403
- related_report_slugs: b0c33chg99,b0b5zjy9mt,b077z2qcs5