Tokenizer
Learn about language model tokenization.
Large language models process text using tokens, which are common sequences of characters found in a set of text. These models learn the statistical relationships between tokens and use them to predict the next token in a sequence.
Use the tool below to understand how a piece of text might be tokenized by a language model. Each colored block is a single token — see the breakdown, the token IDs, and the total counts in real time.