Before a computer can analyze text, it needs to break it into pieces. Those pieces are called tokens, and the process is called tokenization. It sounds simple, but the choice of tokenizer affects vocabulary size, how unknown words are handled, how long sequences become and even how well a model performs in different languages.
What a token is
A token is the basic unit a program processes. Depending on the method, a token can be a word, part of a word, a single character or even a byte. Each distinct token is mapped to a number (an ID), and models work with those numbers rather than with the characters directly.
Word-level tokenization
The most intuitive approach splits text into words and punctuation. "The cats sat." becomes The, cats, sat, .. Word tokenization is easy to understand and works well for classical methods such as bag-of-words.
It has two main problems. First, the vocabulary becomes very large, because every inflection, compound and spelling variant is a separate token. Second, any word not seen during training is unknown, often replaced by a special out-of-vocabulary token, which loses its meaning entirely. Languages written without spaces between words also need specialized segmentation.
Character-level tokenization
At the other extreme, every character is a token. The vocabulary is small and no word is ever unknown. The cost is sequence length: a sentence becomes many more tokens, and the model must learn to build meaning from individual letters, which requires more computation and data.
Subword tokenization
Most modern language models use subword tokenization, a middle ground. Frequent words stay whole, while rare words are split into smaller, reusable pieces. For example, "unhappiness" might be split into un, happi and ness. The vocabulary stays manageable, and new words can still be represented from known pieces.
Common subword algorithms include:
- Byte Pair Encoding (BPE): starts from characters (or bytes) and repeatedly merges the most frequent adjacent pair into a new token until the vocabulary reaches a chosen size.
- WordPiece: similar in spirit, but chooses merges based on how much they improve the likelihood of the training data.
- Unigram language model: starts with a large candidate vocabulary and removes pieces that contribute least, keeping a probabilistic segmentation.
Tools such as SentencePiece implement BPE and unigram models directly on raw text, treating spaces as ordinary symbols, which helps with languages that do not use spaces.
Byte-level tokenization
Some tokenizers work on the bytes of UTF-8 encoded text rather than on characters. Because every possible input is a sequence of bytes, nothing is ever out of vocabulary, including emoji and rare scripts. Byte-level BPE combines this guarantee with subword merges.
Why token counts matter
Many models have a maximum input length measured in tokens, and processing cost often scales with the number of tokens. The same sentence can produce different token counts with different tokenizers. Languages and scripts that were less common in the tokenizer's training data are often split into more, smaller pieces, so the same content may use more tokens in those languages.
Practical rules
- Use the exact tokenizer that a pretrained model was trained with; mixing tokenizers breaks the model's input.
- Do not lowercase or strip accents before a model's tokenizer unless the model expects it.
- When estimating length limits, count tokens with the real tokenizer instead of counting words.
- For classical methods, a good word tokenizer for your language is often sufficient.
Key takeaways
- Tokens are the units a model actually sees; they can be words, subwords, characters or bytes.
- Subword methods such as BPE and WordPiece balance vocabulary size and coverage.
- Token counts vary by tokenizer and language, which affects limits and cost.