On this page
Tokenization is the process that converts raw text into discrete units called tokens that large language models can process computationally. It solves the problem that neural networks cannot work directly with text, they need numbers. Modern LLMs use Byte Pair Encoding, an algorithm that iteratively merges frequent character sequences to build a vocabulary balancing sequence length against coverage.
- Tokenization converts text into numbered tokens because neural networks perform matrix operations on numbers, not text.
- Character-level tokenization produces very long sequences; word-level tokenization fails on rare words and misspellings.
- Byte Pair Encoding starts with 256 byte tokens and iteratively merges the most frequent pairs until reaching a target vocabulary size.
- GPT-2 uses a vocabulary of 50,257 tokens, while GPT-4 uses approximately 100,000 and newer models use approximately 200,000.
- Tokenization boundaries can split related concepts across multiple tokens, potentially fragmenting reasoning and affecting how models process language.
Large language models do not read text the way humans do. They perform matrix multiplication on vectors of numbers, which means text must first be converted into a numerical form the model can process. That conversion is tokenization: the breaking of raw text into discrete units called tokens, each mapped to a number.
Understanding tokenization matters because it is not invisible. The choice of tokenization algorithm and vocabulary size shapes what sequences the model sees, how efficiently it processes them, and where its reasoning can fragment. A misspelled word, a mathematical symbol, or a word in another language hits the tokenizer differently than common English prose, and the model's behavior downstream reflects that.
What tokenization is in large language models
Tokenization is the process that breaks text into discrete units, tokens, that a language model can process. Each token is assigned a unique integer identifier. When you input text to a model, the tokenizer converts it to a sequence of these integers before the model's neural network sees it.
Think of tokens as the atoms of language for the model. They are not always words. A token might be a single character, a common word, a subword fragment, or even a byte. The tokenizer's job is to split the input text into these units consistently, using a fixed vocabulary that was built during the model's training phase. That vocabulary is finite, GPT-2 has 50,257 tokens, for example, which means every possible input text must be representable as some sequence of tokens from that fixed set.
Why language models need tokenization
Neural networks, including language models, perform computation through matrix multiplication and addition. They operate on numbers, not text. Raw text is a sequence of characters, which are symbols, not numbers. Tokenization bridges that gap by converting symbols into numbers that the network can process.
Without tokenization, the model would have no way to ingest text at all. You cannot multiply a matrix by the letter 'a'. Tokenization solves this by assigning each token a unique integer, so the text becomes a sequence of integers that can be fed into the model's embedding layer, which maps each integer to a learned vector representation. From that point forward, the model works entirely with numbers.
How tokenization works: a concrete example
Suppose a tokenizer has a vocabulary of 256 base tokens (one per byte) plus a set of learned subword tokens. You input the text "hello world". The tokenizer scans the text and maps each character or subword to its token ID from the vocabulary. If "hello" is token 15,234 and "world" is token 8,892 in the vocabulary, the text becomes the sequence [15234, 8892]. The model then receives these integers, not the original text.
If a word is not in the vocabulary as a whole, the tokenizer breaks it into smaller pieces. This is why tokenization matters: the model sees fragmented input, and that fragmentation can affect how it reasons about the word. The exact sequence of tokens, their boundaries and granularity, depends entirely on the tokenization algorithm and the vocabulary it produced.
Early tokenization approaches and their limitations
Before modern algorithms, researchers tried two simple strategies: splitting on characters or splitting on spaces. Both have severe drawbacks that make them impractical for large language models.
Character-level tokenization
The simplest tokenization is to treat every character as a token. The text "hello world" becomes seven tokens: 'h', 'e', 'l', 'l', 'o', ' ', 'w', 'o', 'r', 'l', 'd'. A model processing this sequence must attend to ten separate positions where a human reader sees two words. For a long document, the sequence becomes enormous. Models have limits on how long a sequence they can process (the context window), so character-level tokenization wastes that window on representing text that could be compressed. The model also has to learn relationships between characters that humans already know, that 'e' and 'l' next to each other often form part of words. This is inefficient.
Word-level tokenization
A natural next step is to split on spaces: each word is one token. "hello world" becomes two tokens. This is much more efficient for common words. But it breaks down immediately for anything outside the training vocabulary. If the model was trained on a fixed set of 10,000 words and encounters a proper noun, a technical term, or a misspelling, it has no token for it. The tokenizer must either drop the word, replace it with an unknown token, or fall back to characters, all of which lose information. Additionally, related words like "running", "runs", and "run" are treated as completely separate tokens even though they share meaning. A model has to learn their relationships from scratch rather than exploiting their structural similarity.
Byte Pair Encoding: the dominant modern approach
Byte Pair Encoding (BPE) was originally invented as a data compression algorithm in 1994, over two decades before it was adapted for language models. It solves the problems of both character-level and word-level tokenization by building a vocabulary of subword units learned from the training data itself. BPE is used in large language models including GPT-2, GPT-3, and GPT-4.
How Byte Pair Encoding builds a vocabulary
BPE starts with a base vocabulary of 256 tokens representing individual byte values. It then iteratively merges the most frequent pairs of adjacent tokens in the training data until reaching a target vocabulary size. Here is how the process works in practice.
Suppose the training corpus contains the words "hello", "help", and "world" repeated many times. The initial tokenization is character-level: h-e-l-l-o, h-e-l-p, w-o-r-l-d. The algorithm counts all adjacent pairs: (h, e) appears 2 times, (e, l) appears 4 times, (l, l) appears 1 time, (l, o) appears 1 time, (e, p) appears 1 time, (w, o) appears 1 time, (o, r) appears 1 time, (r, l) appears 1 time, (l, d) appears 1 time. The most frequent pair is (e, l) with 4 occurrences. BPE merges all instances of (e, l) into a new token, say "el". Now the corpus becomes: h-el-l-o, h-el-p, w-o-r-l-d. The algorithm repeats: it counts pairs again in the updated corpus, finds the most frequent, and merges them. This continues until the vocabulary reaches the target size. Each merge operation adds one new token to the vocabulary.
The result is a vocabulary where common sequences are single tokens and rare sequences are broken into pieces. "hello" might be a single token if it appears frequently enough, while an obscure word is split into subwords.
Why BPE balances efficiency and coverage
BPE creates a tradeoff between sequence length and vocabulary size. A small vocabulary forces the tokenizer to break most words into many subword pieces, creating long sequences. A large vocabulary means fewer merges, so words stay as larger units and sequences are shorter. But a very large vocabulary requires the model to learn embeddings and parameters for millions of tokens, which increases model size and memory use.
Modern LLMs choose a middle ground. GPT-2 uses a vocabulary of 50,257 tokens, which produces moderate sequence lengths for typical English text. GPT-4 and GPT-3.5-Turbo use approximately 100,000 tokens, and newer models like GPT-4o and the GPT-5 family use approximately 200,000 tokens. These sizes keep sequences manageable while covering a wide range of text, including code, multiple languages, and special characters. The general range for modern LLMs is 32,000 to 100,000 tokens, though newer models push toward the higher end.
Other tokenization algorithms in use
BPE is dominant, but alternatives exist. WordPiece merges pairs based on maximizing the likelihood of the training data, choosing merges that most increase the probability of the corpus. It was originally developed for speech recognition and is used in BERT and other encoder-only models. SentencePiece is a language-independent tokenization framework that can use either BPE or a variant called Unigram LM; it is used in models like LLaMA and PaLM. These alternatives make different tradeoffs, WordPiece favors likelihood over frequency, and SentencePiece handles multiple languages more uniformly, but they solve the same core problem: converting text to a manageable vocabulary of subword units. For most purposes, the differences between these algorithms matter less than understanding that tokenization itself is a learned, vocabulary-dependent process.
How tokenization affects model behavior
Tokenization is not a neutral preprocessing step. The boundaries and granularity of tokens shape what the model sees and how it can reason.
Token boundaries and reasoning
When a concept is split across multiple tokens, the model must learn to reason about it using separate representations. Suppose a rare mathematical operator is split into two tokens. The model has to learn that these two tokens, when adjacent, represent a single operator. This is learnable, but it adds cognitive burden. The model must dedicate capacity to bridging that gap. For common, unsplit tokens, the concept is atomic, a single learned representation, and the model can reason about it more directly.
This becomes acute with misspellings, code, and non-English text. A typo like "recieve" instead of "receive" may tokenize very differently, potentially confusing the model. Code with unusual operators or identifiers can fragment across many tokens. Languages written in scripts the tokenizer was not trained on may be split into characters or rare subwords. In all these cases, the model's reasoning is fragmented compared to how it processes common English words.
Vocabulary size and model capacity
Larger vocabularies require more parameters in the model. The embedding layer, which maps each token ID to a learned vector, must have one row per token. If the vocabulary is 50,000 tokens and embeddings are 768-dimensional, that is 38.4 million parameters just for embeddings. Larger vocabularies increase computational costs. Newer models with larger vocabularies may provide better coverage for rare words, code, and non-English text. This is why tokenizer vocabulary size has grown as models have become larger and more capable.
Tokenization and the limits of language model understanding
Tokenization reveals something fundamental about how language models work: they do not process meaning directly. They process sequences of learned numerical representations of subword units. This is powerful, it allows them to generalize across languages, code, and unseen words. But it also has limits.
A model cannot reason about a concept that is tokenized as fragments any better than a human could reason about a sentence where every fifth word is hidden. The model learns to work around it, but the fragmentation is real. This is why models sometimes struggle with spelling-sensitive tasks, with rare technical terms, or with languages far from their training distribution. The tokenizer has already decided how to break the input before the model sees it, and that decision cannot be undone.
Moreover, tokenization is deterministic and fixed. Once a model is trained with a particular tokenizer, changing the tokenizer is not straightforward, it would require retraining or extensive fine-tuning. This means early tokenization choices have long-term consequences. Understanding tokenization is understanding a fundamental constraint on how the model can represent and reason about language.
Do this automatically
A human manually tokenizing text must decide boundaries for each subword unit, verify vocabulary assignments, and repeat this process for every new document, work that scales linearly with corpus size and introduces inconsistency across large datasets. Software systems can apply tokenization algorithms consistently across millions of tokens in seconds, eliminating the manual overhead while maintaining uniform encoding standards. IMBeacon.ai runs this kind of analysis end to end and cites every figure back to its original source, across as many subjects as you need, with Mimir attaching a citation to every figure. It will not make the decision for you. It gathers and cites the evidence; the judgment stays yours, and deliberately so. Every plan includes it, including the free one.
Start your first report and run it on something you already know well.
This article is general information and research commentary. It is not professional advice, and it does not account for your particular circumstances.
