Raw text is messy. It contains inconsistent capitalization, stray markup, different encodings of the same character and many forms of the same word. Preprocessing turns that raw text into a consistent form that a model or algorithm can work with. The right steps depend on the task: what helps a keyword search can harm a sentiment classifier or a modern neural model.
Below are ten common steps, roughly in the order they are usually applied, with notes on when to use them and when to leave them out.
1. Fix the encoding
Make sure all text is decoded into a single, known encoding, normally UTF-8, before anything else. Text decoded with the wrong encoding produces garbled characters, sometimes called mojibake, that no later step can repair reliably. If your sources vary, detect or record the encoding at the point of collection.
2. Normalize Unicode
The same visible character can be stored in different ways. An accented letter such as "é" can be one code point or a plain "e" followed by a combining accent. Unicode normalization forms such as NFC and NFKC map these variants to a consistent representation. NFKC also folds compatibility characters, such as full-width letters, which is useful for search but can remove distinctions that matter in some languages.
3. Remove markup and boilerplate
Web pages and documents contain HTML tags, navigation menus, cookie notices and footers. Stripping them leaves the content that matters. Be careful not to remove structure you need, such as paragraph boundaries or headings, which can be useful features.
4. Tokenize
Tokenization splits text into units such as words, punctuation marks or subword pieces. Simple whitespace splitting fails on punctuation, contractions and languages that do not separate words with spaces, such as Chinese or Japanese. Use a tokenizer designed for your language and, for neural models, the exact tokenizer the model was trained with.
5. Lowercase (sometimes)
Lowercasing merges "Apple" and "apple", which reduces vocabulary size for count-based methods. It also destroys information: proper nouns, acronyms and sentence starts become indistinguishable. Named-entity recognition and many neural models perform better with original casing, so lowercasing is a choice, not a default.
6. Handle punctuation, numbers and special tokens
Decide what to do with punctuation, digits, URLs, email addresses and emoji. Options include keeping them, removing them or replacing them with placeholder tokens such as <URL> or <NUM>. Emoji and exclamation marks can carry sentiment, so removing them may hurt a sentiment task.
7. Remove stop words (with care)
Stop words are very frequent words such as "the", "is" and "of". Removing them shrinks bag-of-words representations and can sharpen keyword extraction. For tasks that depend on meaning, however, small words matter: removing "not" changes "not good" into "good". Contextual neural models usually work better with stop words left in.
8. Stem or lemmatize
Stemming chops words to a crude root with rules, so "connected" and "connection" may both become "connect". Lemmatization uses vocabulary and grammar to return a dictionary form, so "better" can become "good". Both reduce sparsity for classical methods. Lemmatization is more accurate but slower and language-specific. Neural models with subword tokenizers rarely need either.
9. Deduplicate
Datasets often contain exact and near-duplicate documents, especially when scraped from the web. Duplicates inflate some patterns and can leak between training and test sets, making evaluation look better than it really is. Hashing normalized text finds exact duplicates; techniques such as MinHash help find near-duplicates.
10. Detect language and filter
If a model is meant for one language, filter out text in other languages using a language identification step. For multilingual projects, record the detected language as metadata so you can evaluate performance per language.
Choosing the right pipeline
| Approach | Typical preprocessing |
|---|---|
| Keyword search, bag-of-words, TF-IDF | Normalize, tokenize, lowercase, remove stop words, stem or lemmatize |
| Classical classifiers | Similar, but test whether stop-word removal and stemming actually help |
| Pretrained neural models | Fix encoding, light cleaning, then the model's own tokenizer; little else |
Key takeaways
- Encoding and Unicode normalization come first and prevent hard-to-find errors.
- Every step removes information; apply it only if your task benefits.
- Always use the same preprocessing at training time and at prediction time.