The most repeated advice about building sentiment analysis data is to collect more of it. Scrape every review, pull two years of posts, dump the lot into a fine-tuning set. Volume feels like progress because it's the easiest thing to measure.
It's also the cheapest part of the job, and the part least likely to decide whether your classifier works. What decides it is whether the text answers the question you asked, whether two people reading the same sentence would write the same label, and whether the rows are allowed to be in the training set at all. None of those can be retrofitted after training.
A team can spend a quarter on model selection and then discover that its gold labels disagreed with each other a third of the time. That isn't a modelling problem. It's a data-quality problem that was visible on day one in a spreadsheet nobody opened.
Treat label disagreement, representation and privacy as first-step design decisions. Here's the sequence I'd run, in the order I'd run it.
Choose Sources That Match Your Question
A source can match your topic and still produce the wrong training data. A brand team that exports every post mentioning its name may end up with a corpus dominated by shared links, not opinions. The classifier can perform well on that data while failing to answer how customers feel about the brand.
Before choosing a collection method, settle three decisions: Which domain matters, such as product reviews, support tickets or public posts? Do you need polarity or a graded scale? How many examples can people realistically label this month? Those answers determine which source fits the task and how much annotation it will require.
Source types and what they can and cannot answer
| Source | Volume reference | Label type | Main risk | Best use |
|---|---|---|---|---|
| SemEval Twitter 2013 benchmark | 9,497 training messages | Crowd labels, three classes | Short, informal, heavily imbalanced | Baselines and evaluation |
| SemEval Twitter 2017 benchmark | 50,333 training tweets | Five-level ordinal labels | Neutral is about 44.9% of training data | Fine-grained scoring and comparison |
| Your own product reviews | Hundreds to thousands per product line | Star ratings as weak labels | Rating does not always equal sentiment | Monitoring customer opinion on your products |
| Support tickets and messages | Thousands per month at a mid-sized desk | Usually unlabelled | Personal data and private content | Service quality and specific complaints |
| News, forums and comment threads | Unbounded, set by crawl budget | Usually unlabelled | Longer, more neutral tone | Brand and issue monitoring |
Label granularity affects both the task and the annotation budget. The 2017 SemEval Twitter benchmark shifted the task from binary polarity to five ordinal levels. Its principal English training set contained 50,333 tweets, including 22,591 neutral tweets. After years of shared-task work, fine-grained Twitter sentiment remained far from solved. Neutrality and distinctions between nearby sentiment levels still make labels hard to apply consistently. Use a graded scheme when stakeholders need those distinctions, and plan for the added annotation effort. For complaint triage, three classes may answer the question.
Benchmarks provide a familiar label scheme, but short, informal posts may not resemble reviews or support tickets. Content from your own product or service is closer to the target, though star ratings are weak labels: check whether they agree with the review text. Support messages can capture detailed opinions, while also containing private information. Treat that privacy exposure as a source-selection constraint, not a cleanup task for later.
Write the question in one sentence
For a delivery company investigating customer dissatisfaction with shipping, recent support tickets mentioning delivery are a better fit than general brand tweets. Label the complaint as lateness, damage or an outdated tracking page. A public benchmark can then help distinguish a difficult ticket corpus from one that differs in domain and language.
A rule that saves rework: choose one primary source and one benchmark for comparison. Strong benchmark results paired with weak results on your own source point to a mismatch between the benchmark and target data. Adding more rows from the same source will not resolve that difference.
When the primary source is your own feedback channels, define what information each message needs to retain. This customer feedback collection workflow covers keeping channel, timestamp and rating attached to each message. Conversation logs need a separate unit of analysis: a session may contain a customer's question, an agent's reply and the resolution. Use AI training chat transcript categories to decide which turns express sentiment and which are procedural, then label only the units that match your question.
Collect Dynamic Pages Responsibly
Most review pages look dynamic and aren't. Load a single URL with a plain HTTP client and compare the returned HTML against what a browser shows. If the review text is already in the body, you've just avoided a rendering pipeline, a proxy bill and a whole class of selector breakage.
That check is the first thing I'd hand a new engineer on this kind of collection work.

Decide whether the page needs a browser at all
- Test one URL with a plain request. Note whether review text, ratings and dates appear in the raw HTML, and save that response. You'll want to re-check it after the site ships a redesign.
- Hunt for the JSON endpoint. Open the network tab, filter to fetch and XHR, then scroll. If a call returns reviews as structured data, call that endpoint directly. It's faster, lighter and easier to paginate than rendering.
- Render only when there's no alternative. With a headless browser, wait for the review selector rather than a fixed delay, then scroll until two consecutive scrolls add nothing new. Fixed sleeps are why pipelines fail at 3am on a page that loaded a second slower.
- Persist the raw response first. Write URL, timestamp, page number and raw body to storage before parsing. When a selector changes, you re-parse instead of re-crawling.
- Stop at a wall. A login prompt, an unresolvable consent banner or a CAPTCHA that keeps returning means stop. Look for a licensed export or a public dataset instead.
Minimum viable extraction: wait on
[data-review]with Playwright, pull the text with$eval, and append one JSON line per review, so a crash costs you a page rather than the run.
Tie each proxy to one country when the page is market-specific. Language and currency follow the exit node, and mixing markets through one pool produces a corpus where a translated complaint sits beside the original with no way to tell them apart. A per-market proxy keeps your language metadata honest, which matters later when you report scores per language.
Choosing a fetch method by page type
| Page type | Fetch method | Proxy need | Common failure and fix |
|---|---|---|---|
| Static HTML page | Plain HTTP client | Low | Rate limiting, so slow down and cache responses |
| Page that loads reviews from a JSON endpoint | Call the endpoint directly | Low | Expired tokens, so refresh headers and cookies |
| JavaScript-rendered reviews | Headless browser | Medium, per market | Empty DOM, so wait for the review selector |
| Infinite-scroll feed | Headless browser with a scroll loop | Medium | Duplicate items, so deduplicate on item ID |
| Login or consent wall | Stop collection | Not applicable | Account bans, so request a licensed export instead |
A collection log you can hand to a reviewer is worth more than a faster crawler, and this guide to web scraping for research covers the documentation habits that make a crawl defensible.
Label Data Without Guessing
A model cannot outrun a label scheme nobody wrote down. Write the categories first, then guidelines with worked examples for the items annotators will fight over: sarcasm, mixed reviews ("fast shipping, broken product"), rhetorical questions, and praise buried inside a complaint. Those arguments are quality work. They are not a cleanup pass you schedule after training.
Measure agreement before you trust a label
Build the set in stages. Define the ontology and guidelines. Sample across sources, languages, time periods and product categories so one market or dialect does not own the labels. Collect multiple independent labels, measure agreement with Krippendorff's alpha or Cohen's kappa, adjudicate the ambiguous cases, then freeze a time- or source-separated test set.
Human agreement on three-class sentiment is often only moderate. An AI labeler tends to match human consensus even less often. Apparent model errors frequently sit on items people already split on. What you are buying is a documented ceiling.
A worked example with 300 messages
Three annotators label 300 support messages as negative, neutral or positive. Forty items disagree, nearly all on whether an unresolved complaint counts as negative or neutral. Two things happen next. The guideline gets an explicit rule with examples: a complaint stating a problem with no emotional language is negative, not neutral. Those 40 items get relabelled by someone who did not see the original labels. Then you measure agreement again.
If the second pass still splits on the same slice, the ontology is the problem, not the annotators. I would rather drop a class than keep one nobody can apply the same way twice.
Disagreement is a measurement, not a mess to clean up. If trained annotators can't agree on a meaningful share of items, a model scoring well against majority labels is already at the ceiling the labels allow.
Labelling strategies compared
| Strategy | Best for | Quality control | Main risk |
|---|---|---|---|
| Manual expert annotation | Small, high-stakes sets with domain jargon | Dual annotation and adjudication | Slow and costly to scale |
| Crowd annotation | Large sets of general-language text | Gold questions, worker filters, agreement threshold | Low-quality or rushed annotators |
| Weak supervision | Large unlabelled sets with noisy rules | Accuracy check on each labelling rule | Rules encode their author's bias |
| Heuristic labels (lexicons, star ratings) | Fast baselines | Spot-check a random sample against humans | Star rating is not the same as sentiment |
| LLM-assisted first pass | Scaling a first labelling round | Human review of the disagreement slice | Systematic errors hidden behind confident output |
LLM-assisted labelling is a common first pass because many developers already use AI tools in daily work. The failure mode is a labeler that sounds equally confident on the sarcastic items and on the clear ones. Review the disagreement slice by hand, and draw that review from every language and product category in the sample. A fluent majority dialect will otherwise hide the misses.
Keep annotator IDs, disagreement flags, confidence scores and the original text in every row. Document sampling, annotator backgrounds and conflict resolution next to the data. Eight months later those fields are how you tell a text problem from a labeler who never saw that dialect. If one language or product category supplies most of the agreed labels, the ceiling you measured only covers that slice. Limit who can open rows that contain names, emails or order details. An audit trail that leaks customer text is a quality record and a privacy failure at the same time.
How to Clean, Balance and Split the Corpus
Run this as a build order on a real folder of files rather than a list you skim. Each step exists because skipping it inflates a score somewhere downstream.

The build order
- Hash and dedupe exact copies. Normalise case and whitespace, hash the text, drop repeats. Retweets, syndicated reviews and copied forum posts inflate the majority class and leak across splits.
- Strip the page, keep the text. Remove HTML, navigation, cookie banners and URLs, but write the result into a new column so the raw text survives.
- Detect language per row. Tag code-mixed items as their own group instead of dropping them. Dropping is how a model ships with a blind spot nobody wrote down.
- Normalise Unicode and whitespace, keep intensity. Emojis and stretched words like "soooo" are signal.
- Keep negation words. "Not", "no" and "never" stay in the cleaned text. Stopword removal is a topic-modelling habit and it flips meaning here.
- Drop short rows and log the count per source. Losing 40% of one channel changes what the model learns, and you want that visible.
- Balance the training split only. Try class weights first, then undersample the majority class. Oversample rare classes after the split, never before.
- Split by source and time. A random shuffle puts near-duplicates from the same product on both sides.
- Freeze the test set. Hash it, store it, stop touching it.
Two lines that prevent the most common leak:
df = df.drop_duplicates(subset=['text_norm'])before any splitting, then a stratified split on the label column and a group split on source so no source appears on both sides.
Splitting by language and time
Multilingual corpora need one more cut. An evaluation that selected 80 high-quality datasets from 342 collected, covering 6,164,942 text samples, 27 languages and 11 models, found that top-performing models were not statistically different overall, even though one model could be competitive across languages (multilingual sentiment evaluation). Small aggregate gaps don't justify deployment. Stratify by language and class, keep translated copies on one side of the split, and report macro-F1 with confidence intervals per language. In that benchmark, XLM-RoBERTa reached 0.60 accuracy in Arabic against 0.66 in English, while classical methods sat around 0.44 to 0.50 in the same settings.
If you'd rather watch the split logic before writing it, this walkthrough covers the workflow end to end.
Watch the video on YouTubeStore Data So You Can Audit and Relabel It
Relabelling is normal. Guidelines change, annotators improve, and a year later someone asks why a sarcastic review was scored positive. Storage that can't answer that question turns a two-hour investigation into a re-collection project.
Keep three tiers, each versioned by content hash and date: raw (exact response plus headers), cleaned (post-normalisation, rebuildable from raw), and labelled (text, labels, annotator IDs, agreement). The cleaned tier must always be reproducible. If it isn't, you've lost the ability to test whether a cleaning change helped or hurt.
Pick the format by the question you'll ask later
JSON Lines suits text with nested metadata: one record per line, append-friendly, streams without loading the file. CSV works for a simple id, text and label table, with the usual caveats about encoding and embedded newlines. Parquet handles large corpora with typed columns and compression, which is what you want when an analyst scans the whole set in SQL.
The minimum schema, with the field each question needs later:
- id, source, url so you can slice by channel and check whether the model favours one.
- collected_at so drift becomes a measurement instead of an argument.
- language, split so evaluation stays per-language and the test set stays frozen.
- text_raw, text_clean so a parser bug is fixable without re-crawling.
- label, annotator_ids, agreement_score so every label carries provenance.
- consent_flag so a deletion request can be honoured across every version.
Write a short dataset card beside each version listing collection dates, sources, labelling method, agreement statistics and known gaps. It's the artefact reviewers ask for first and the one teams skip. This database schema design reference is worth reading before you settle column types and indexes for the tables sitting around the corpus.
Measure Quality and Augment Carefully
A strong overall score can hide a model that misses neutral sentiment. In the SemEval-2013 Twitter corpus, neutral examples are a useful reminder to inspect performance by class rather than trust a single headline number.
Why accuracy lies on skewed data
Report macro-F1 as the primary metric, with per-class precision and recall beside it. Read the confusion matrix before accepting an apparent improvement. On skewed sentiment data, an accuracy gain may reflect better predictions on the majority class while performance on neutral examples remains weak.
Metrics to report and what each one hides
| Metric | What it shows | Use it when | Pitfall |
|---|---|---|---|
| Accuracy | Share of items labelled correctly | Classes are roughly balanced | Hides poor performance on rare neutral items |
| Macro-F1 | Average F1 across classes, equal weight each | Default for imbalanced sentiment data | Noisy on very small classes, so report counts |
| Per-class precision and recall | Which label the model misses or over-predicts | Diagnosing neutral or sarcasm errors | Many numbers to read, so compare side by side |
| Confusion matrix | Which labels get mixed up | Checking positive versus neutral confusion | Needs reading by class, not just in total |
| Agreement with consensus labels | How far the model matches human consensus | Judging whether the reference labels are reliable | Mixes model error with label ambiguity |
Augmentation that survives the test set
Paraphrasing and back-translation can add examples for rare classes, but they can also change the sentiment. A mildly negative review may return as neutral, and a label copied automatically will conceal the drift. Mark every augmented row so error analysis can separate it from original data. Keep augmented examples out of validation and test, then manually relabel a sample of 50 and compare the results.
Use a controlled comparison: train once on the original data and once on the original plus augmented data. Keep the change only if macro-F1 improves on the untouched test set across at least three random seeds. A gain from one seed on a small corpus may be noise, not a dependable improvement.
Check per-language before you ship
A pooled score can hide weak results in low-resource languages. Per-language scores expose where the dataset or labels are not sufficient, while class-level results show whether neutral examples are being missed. Neutral is often the hardest class in Twitter sentiment data, so check its precision and recall separately rather than assuming a strong overall score means it is handled well.
A study linked through this low-resource sentiment benchmark study is relevant to low-resource evaluation. Use evidence that directly supports the language and task you are measuring, and avoid treating results from one benchmark as a prediction for another.
A single global score is a marketing number. The engineering number is per language, per class, with the sample count next to it.
Keep code-mixed text as its own slice, and do not let a translated copy fall on the opposite side of a split from its original. Otherwise, evaluation can reward familiarity with duplicated content instead of measuring how the model handles new examples.
Check Privacy and Representation Before You Train
Volume doesn't correct sampling bias. A million posts from a platform whose users skew young and urban tells you about that platform's users, not about your customers. A dataset twice the size with the same skew is twice as confident about the wrong population.
This is a go or no-go gate, not cleanup work after the model trains.

The pre-training checklist
- Strip identifiers you don't need. Names, emails, phone numbers, account handles. Hash the IDs you must keep so rows can still be grouped by user without revealing who that user is.
- Record your legal basis. Confirm it for your region, write it into the dataset card, and if a vendor processes personal data for you, get a signable Data Processing Agreement in place before collection. Scrappey offers a signable DPA for customers processing personal data through the API.
- Set retention and deletion. One person's records must be removable from raw, cleaned and labelled tiers, not just from the training table.
- Read each platform's terms. Scraping and reuse rules differ, and a licensed export is often cheaper once you price the risk.
- Estimate who is represented. Compare the share of posts by language, platform, region and user type against the population you want to describe.
- Audit recall by group. A model can be accurate overall and still miss one language's complaints consistently.
- Report ranges, not one percentage. Track shifts after platform policy changes, because a rule change on the platform looks exactly like a change in public opinion.
A 2025 systematic review in public health found that social-media-only analyses can lack generalizability and underrepresent vulnerable groups, listing demographic bias, platform-policy volatility, privacy concerns and limited interpretability as standing challenges, and the benchmark evidence it reviews puts average accuracy for Twitter sentiment systems at 61%, with the best at 71% (2025 public health systematic review). The operational question isn't "what is the sentiment". It's whose sentiment, at what error rate, measured over which period.
Start with a pilot, not a crawl
Label 500 items from a single source, run every check above, and publish the dataset card before scaling collection. A pilot that fails a check costs a week. A failed collection at scale costs the quarter and, if it involved personal data you can't cleanly delete, rather more than that.
Scrappey is designed for accessing publicly available web data and powering automation workflows, which is the right shape for this stage of the work: public pages, documented provenance, and a decision made deliberately about what stays out.
If you'd rather spend your week on label guidelines than on proxy rotation, Scrappey handles the collection layer: real-browser automation for JavaScript-heavy review pages, per-market residential proxies so language and currency stay consistent, and structured output you can write straight into your raw tier. E-commerce review collection, with ratings and review text intact, is one of the workflows it's built around. Pull one source end to end on the free trial at Scrappey before you scale, and see whether your labels or your collection is really the bottleneck.



