Large-Scale Text Analytics

At what volume should I switch from manual coding to text analytics?

Roughly 5,000 items is a working boundary, and often higher where the items are rich. Below it, human or AI-assisted reading of every item produces better analysis, because the interpretive work is the value and a classifier's error rate buys nothing. Above it, reading everything stops being feasible and the discipline shifts to validation. The boundary is not just about volume: an interpretive question about what a theme means belongs with thematic analysis at any scale, and a large corpus changes only how you sample for that reading.

Why is overall classification accuracy misleading?

Because corpora are imbalanced. A classifier that labels everything as the largest class can score 70% overall while detecting nothing else, and it is the small classes that usually carry the risk. Report per class instead: precision, meaning of the items assigned to a class, how many genuinely belong, which governs whether your examples are trustworthy; and recall, meaning of the items that genuinely belong, how many were found, which governs whether your prevalence figure is an undercount. They trade off, and where a class matters, tuning toward recall and accepting false positives is usually the right trade.

What makes a gold set good enough?

Five things. Sample randomly from the deduplicated corpus for overall accuracy, and over-sample rare classes, because a random 500 from a corpus where a class occurs at 0.5% contains two or three examples and measures nothing. Double-code a portion independently, measure agreement and adjudicate disagreements, because where humans agree poorly no classifier can do better. Hold it out from anything used to build or tune the classifier. Size it for the claim, since an accuracy estimate on 50 items has an interval of roughly plus or minus fourteen points. And document the labelling instructions, which are the operational definition of every category.

Are topic model outputs themes?

No. They are groupings of co-occurring words produced by a statistical procedure, and they require human naming before they mean anything. Some are meaningless artefacts of boilerplate, a single event or one prolific author, and the right handling is to discard them rather than force an interpretation. Check stability by running with different initialisations and topic counts and keeping only what persists. The number of topics is a choice made on interpretability by a human reading examples, not a result. Always show the top terms and several representative items with identifiers alongside any name.

What is the biggest risk in text analytics?

Silence about something that mattered. A percentage that is five points wrong causes a mild misjudgement. A serious issue appearing forty times in sixty thousand items that was never surfaced causes real harm, and it is invisible in every quality metric the pipeline reports. Automated classification is optimised on aggregate performance, so it performs worst on the classes with fewest examples, which are frequently the ones with the highest consequence: safety, discrimination, regulatory breach, severe harm.

How do I find rare but important content?

Treat it as a search problem, not a prevalence problem, and run it as a separate track. Build a deliberately over-inclusive keyword and pattern list with domain input, accept a high false positive rate, and read everything it returns, because recall is the objective and precision is irrelevant here. Review low-confidence and unmatched items. Where the organisation knows of specific incidents that should be in the corpus, test whether the pipeline finds them, which is a direct measure of the blind spot. And read a few hundred random items regardless, to see what the categories do not capture.

Does a large text corpus tell me what customers think?

It tells you what the people who wrote said. Reviews cluster at the extremes because moderate experiences do not motivate writing. Complaint corpora over-represent the articulate and the persistent. Consultation responses over-represent organised interests. Incentivised reviews differ systematically from unincentivised ones. A text corpus is not a sample of a population, so where the question is about a population, no corpus size answers it. State the selection mechanism in a sentence that travels with every proportion.

How do duplicates and bot text affect the analysis?

They inflate everything silently. Run three passes before analysis: exact duplicates, near duplicates differing only in a name or a number, and templates or automated text with a fixed structure and variable slots. Coordinated campaigns are not always deleted, because their existence can be the finding, but they must be separated and analysed as their own object, otherwise every proportion measures the campaign as if it were opinion. Log counts at each pass and report corpus size before and after.

Can I use one accuracy figure for a multilingual corpus?

No. Per-language accuracy differs, often substantially, so a figure measured in one language does not describe another. Machine translation before classification introduces systematic rather than random error, concentrated in idiom, sarcasm, emotional intensity and culturally specific reference, which is exactly the content that carries meaning. Validate per language with per-language gold sets, count mixed-language items separately, and where a language's volume is too small to validate, exclude it from derived numbers rather than absorbing it silently.

What must I disclose when classification was machine-performed?

What was machine-classified, against what frame, validated how, on a gold set of what size, at what measured per-class accuracy, what human verification was performed, and what was not validated. This belongs with the results rather than in an appendix, because a reader who knows a figure came from a classifier with 0.68 recall on that class can use it correctly, and one who does not will treat it as a count. Also disclose the classes that were not reportable, since the list of what could not be measured is part of the result.