What Is a Feature?

Part 1 of a short series on mechanistic interpretabilityThe term was coined by Chris Olah in the Circuits thread on Distill.pub (2020–2021), part of an effort to reverse engineer neural networks just like one might reverse engineer a computer program. Saphra and Wiegreffe (2024) later surveyed how the term is actually used and proposed a narrow technical definition: research that uncovers the causal mechanisms explaining a network's transformation from input to output at the level of intermediate representations., told through small interactive widgets.

At any given point while processing text, a language model has an activation vector: a long list of numbers describing its current internal state. Certain patterns within those numbers consistently correspond to concepts like formality, gender, or whether the text is in English. Mechanistic interpretability calls one such pattern a feature: not necessarily a single neuron or a single number, but a direction in that space, one particular way of nudging thousands of numbers together. If we can identify these directions reliably, we can start to understand what a model actually represents internally, and even change its behavior by pushing along them.

Try it below. Click a word and watch a single feature light up: whether it's English. Alongside it is a more abstract, but still toy, version of the same idea, two directions this time, formality and warmth, where dragging the dot continuously changes how strongly each one fires.

Real models, however, don't operate in a two-dimensional space. Their internal representation, called the residual stream, has thousands of dimensions instead of twoElhage et al. (2021), "A Mathematical Framework for Transformer Circuits," describes the residual stream as a shared communication channel that every layer reads from and writes to.. But the geometric intuition from the toy above carries over: a feature is still a direction, and how strongly a given input "has" that feature is just how much its activation vector points along it.

This is intentionally a loose definition; different interpretability papers formalize "feature" in slightly different ways. Olah et al.'s original framing treated neurons as the fundamental unit of a neural network, the atoms that circuits are built out ofOlah et al. (2020), "Zoom In: An Introduction to Circuits," proposes that features are the basic building blocks connected by circuits into a network's algorithm.. At first glance, you might imagine each feature lives inside its own neuron. Unfortunately, real networks aren't that straightforward: a single neuron will often respond to several unrelated features at once, a phenomenon known as superpositionElhage et al. (2022), "Toy Models of Superposition," shows that networks can represent more features than they have dimensions by tolerating some interference between them., which is a fancy term for there being more features than a model has room for.

The reason this framing matters is very practical. If a feature is a direction in space, you can do arithmetic with it: add it, subtract it, amplify it, and watch the model's behavior shift accordingly. That is exactly what activation steering, for instance, does in practice. Once you identify the direction, you can push on it.

Each bar above is one coordinate of a twenty-dimensional activation vector. Dragging one moves the activation vector along that one axis of the space, exactly like the arrows in the widget earlier, just with eighteen more axes lying around. Real language models work the same way: they just have thousands of coordinates instead of twenty. Only two of these particular ones happen to mean something on their own; drag them and watch a sample sentence shift language and register. The other eighteen stand in for the many dimensions that exist in a real model but aren't relevant to this example: some would encode entirely different concepts, some would only mean something in combination with a few others, and quite a few would resist a clean explanation at all.