LANDMARK · 6 MIN · UNDERSTAND
Tokens
After this landmark, you can explain what a token is and predict which tasks a model will be oddly bad at because of it.
Before a model sees your text, the text is chopped into tokens: common chunks of characters drawn from a fixed vocabulary. A short familiar word is usually one token; a longer or unusual one is several; a space is often carried along with the word that follows it. As a rough English guide, a token averages about four characters, so a page of prose is around 500 tokens. Everything downstream is counted in tokens — the context limit, the price, the speed. More importantly, the model never sees the letters inside a token. It sees an ID for the whole chunk. That single fact explains a family of failures that otherwise look like stupidity: miscounting letters in a word, struggling to spell backwards, mangling rare names, and being worse in languages whose scripts are split into many more tokens per word.
This is the canonical concept. It stays the same across learner lenses so personalization never changes the underlying facts.
◌ EVERYDAY LENS
What this looks like for you
If a model gets a letter-counting or wordplay question wrong, it isn’t being careless — it can’t see inside the chunks it reads. Ask it to write the word out letter by letter first, and it will often get the answer right, because now the letters are separate tokens.
MAKE A DECISION
A colleague reports that an assistant insists “strawberry” contains two Rs, even after being corrected. What is the best explanation?
CARRY THISPaste a paragraph of your own work into a tokeniser and note the token count against the word count. Repeat with the same content in another language you use.