LANDMARK · 7 MIN · UNDERSTAND
How multimodal models work
After this landmark, you can explain how a model handles a picture or a recording, and predict the categories of task it will be unreliable at.
A multimodal model works by turning other media into the same kind of sequence it already handles. An image is cut into patches, each patch encoded into a vector, and those vectors join the text tokens in one shared space — so attention can relate “what does this chart show?” to the region of the picture that answers it. Audio is handled similarly, through spectrogram-like representations. Generation runs the other way and often uses different machinery: image and video generation is usually diffusion, which starts from noise and removes it step by step towards something matching the prompt. The important consequence is resolution. An image is compressed into a limited number of patch tokens, so fine detail — small text, thin lines, precise positions, dense table rules — is genuinely not fully present. That is why a model can describe a photograph beautifully and misread the small print in it, and why it will state the misreading with the same confidence.
This is the canonical concept. It stays the same across learner lenses so personalization never changes the underlying facts.
◌ EVERYDAY LENS
What this looks like for you
Great for “what is this plant”, “describe this photo”, “summarise this whiteboard”. Not to be trusted for exact figures in a screenshot, a dosage on a label, or a number in a receipt — crop and zoom before asking, and check anything precise against the original.
MAKE A DECISION
A finance workflow uploads photographed receipts and asks a multimodal model to extract the total. What is the necessary safeguard?
CARRY THISPhotograph a document with small print and ask an assistant to read a specific figure from it. Check the answer against the original.