Tokenizer
Learn about language model tokenization.
Large language models process text using tokens, which are common sequences of characters found in a set of text. These models learn the statistical relationships between tokens and use them to predict the next token in a sequence.
Paste some text below to see how a language model breaks it apart. Each colored block is one token, and you can switch to the token IDs or watch the counts update as you type.