LANDMARK · 6 MIN · UNDERSTAND
Seeing inside a model
After this landmark, you can explain why a model is a black box, what interpretability research is trying to do about it, and why a model's own explanation of itself is not evidence.
When a model produces an answer, that answer is the result of billions of numbers multiplying together in ways no one wrote by hand and no one can read at a glance. Nothing inside labels itself “this is the part that decided.” That's what people mean by calling a model a black box: we can see what goes in and what comes out, but the middle is opaque. Interpretability is the field trying to change that — to find, inside all those numbers, the recognisable patterns (“features”) a model uses and the pathways it strings together to get from input to output, so that its behaviour can eventually be inspected rather than only observed. Early results are striking: researchers have located internal features that correspond to specific concepts and even nudged behaviour by turning them up or down. But this is a research frontier, not a feature of the apps you use. The practical consequence for now is a discipline about explanations: when you ask a model “why did you say that?”, its answer is another prediction of a plausible-sounding reason, generated fresh — not a readout of what actually happened inside. It can sound authoritative and be entirely invented.
This is the canonical concept. It stays the same across learner lenses so personalization never changes the underlying facts.
◌ EVERYDAY LENS
What this looks like for you
Treat a model's explanation of itself as a story, not a confession. “I said that because the sources showed X” might be true, or might be a plausible sentence it produced on the spot. If the why matters, check it against something outside the model.
MAKE A DECISION
An automated tool declines someone's application and, when asked, the model writes a paragraph explaining “why”. A reviewer wants to use that paragraph as the official reason for the decision. What's the problem?
CARRY THISAsk an AI tool to explain why it gave a specific answer, then try to verify one concrete claim in that explanation against an outside source. Note whether the explanation held up — and whether you could even check it.