Many NLP projects succeed or fail on their data rather than their model. A well-built text dataset matches the real task, is labeled consistently and is split so that evaluation reflects real-world performance. Building one takes planning, but the process is similar whether you are classifying support messages, tagging named entities or detecting topics.
Step 1: Define the task precisely
Write down what the input is, what the output should be and how the result will be used. "Classify customer emails" is vague; "assign each incoming email one of six routing categories, or 'other'" is precise. A precise definition tells you what text to collect, which labels you need and how to measure success.
Step 2: Collect text responsibly
Collect text that resembles what the system will see in practice: same language, length, style and channel. Before collecting, consider:
- Rights and terms of use: websites and platforms have terms that may restrict scraping or reuse; openly licensed corpora state their conditions.
- Personal data: text often contains names, email addresses and other personal information. Data protection rules may apply, and you may need to remove or mask such details.
- Consent and purpose: internal data collected for one purpose may not automatically be usable for another; check your organization's policies.
Record where every document came from so you can trace and, if necessary, remove it later.
Step 3: Clean and normalize
Convert everything to one encoding, usually UTF-8, normalize Unicode, strip markup and boilerplate, and remove empty or broken records. Keep the raw version alongside the cleaned version so that you can change cleaning rules later without collecting again.
Step 4: Deduplicate
Remove exact duplicates and consider near-duplicates, such as the same template message with different names. Duplicates make the dataset look larger than it is and can leak between training and test sets.
Step 5: Write labeling guidelines
Labeling guidelines are the most important document in the project. Good guidelines include:
- a definition of every label in plain language;
- several positive and negative examples for each label;
- rules for ambiguous cases and for text that fits no label;
- whether a document can have one label or several.
Expect to revise the guidelines after the first batch of labeling, when real edge cases appear.
Step 6: Label and measure agreement
Have at least two people label the same sample independently, then compare results. Agreement measures such as Cohen's kappa summarize how consistently labels are applied beyond chance. Low agreement usually means the guidelines or label set need work, not that annotators are careless. Resolve disagreements and update the guidelines before labeling at scale.
Step 7: Check balance and coverage
Count examples per label. Rare labels may need targeted collection. Also check coverage of important variations: different lengths, writing styles, dialects or product lines. A dataset that only covers the easy cases will produce a model that fails on the hard ones.
Step 8: Split the data carefully
Divide the data into training, validation and test sets. Make sure related items stay together: messages from the same conversation, documents from the same author or near-duplicates should not appear on both sides of a split. When data changes over time, a time-based split, with the newest data held out for testing, often gives a more realistic estimate.
Step 9: Document the dataset
Write a short datasheet describing the source, collection dates, cleaning steps, label definitions, agreement results, known gaps and intended uses. Documentation helps others use the data correctly and helps you reproduce your own results.
Key takeaways
- Start from a precise task definition and collect text that matches it.
- Respect terms of use and personal data rules, and record sources.
- Invest in labeling guidelines and measure agreement early.
- Split data so related items do not leak between training and test sets.