← Concept IndexDEFINITION WHY IT MATTERS COMMONLY CONFUSED WITH
Tokens and tokenization
Also called: subword units, token count
Models read and write in tokens — chunks of text roughly the size of a short word or word-piece — not whole words or letters. Text is split into tokens before the model ever sees it.
Tokens are the unit of both cost and the context window. They also explain quirks like a model struggling to count letters in a word: it never saw the letters, only the token.
Words. A token is often part of a word; long or rare words become several tokens.