What Is a Feature?

Part 1 of a short series on mechanistic interpretabilityThe term was coined by Chris Olah in the Circuits thread on Distill.pub (2020–2021), part of an effort to reverse engineer neural networks just like one might reverse engineer a computer program. Saphra and Wiegreffe (2024) later surveyed how the term is actually used and proposed a narrow technical definition: research that uncovers the causal mechanisms explaining a network's transformation from input to output at the level of intermediate representations., told through small interactive widgets.

At any point while processing text, a language model has an activation vector: a long list of numbers describing its current internal state. Certain patterns within those numbers consistently correspond to concepts like refusal, gender, or whether the text is in English. Mechanistic interpretability calls one such pattern a feature, which is not necessarily a single number or a single neuron, but a direction in activation space, one particular way of nudging thousands of numbers together. If we can identify these directions, we can start to understand what a model actually represents internally, and attempt to change its behavior by pushing along them.

Try it below. Click a word and watch a single feature light up: whether it's English. Alongside it is a direction whose projection predicts whether the model will refuse a requestArditi et al. (2024) find this direction by taking the difference in mean activations between prompts a model complies with and prompts it refuses. Ablating the direction from every layer makes the model comply with almost any request; adding it back makes the model refuse even harmless ones.. Click one of the six requests below and watch its dashed line show how far along that direction it falls.

Real models, however, don't operate in a two-dimensional space. Their internal representation, called the residual stream, has thousands of dimensions instead of twoElhage et al. (2021), "A Mathematical Framework for Transformer Circuits," describes the residual stream as a shared communication channel that every layer reads from and writes to.. But the geometric intuition from the toy examples above carries over: a feature is still a direction, and how strongly a given input "has" that feature is how much its activation vector points along it.

This is intentionally a loose definition; different interpretability papers formalize "feature" in slightly different ways. Olah et al.'s original framing treated neurons as the fundamental unit of a neural network, the atoms that circuits are built out ofOlah et al. (2020), "Zoom In: An Introduction to Circuits," proposes that neurons are the basic building blocks connected by circuits into a network's algorithm.. At first glance, you might imagine each feature lives inside its own neuron. Unfortunately, real networks aren't that straightforward: a single neuron will often respond to several unrelated features at once, a phenomenon known as superpositionElhage et al. (2022), "Toy Models of Superposition," shows that networks can represent more features than they have dimensions by tolerating some interference between them., which is a fancy term for there being more features than a model has room for.

The reason this framing matters is very practical. If a feature is a direction in space, you can do arithmetic with it: add it, subtract it, amplify it, and watch the model's behavior shift accordingly. That is exactly what activation steering, for instance, does in practice. Once you identify the direction, you can push on it.

Each bar above is one coordinate of a twenty-dimensional activation vector. Dragging one moves the activation vector along that one axis of the space, exactly like the arrows in the widget earlier, just with eighteen more axes lying around. Real language models work the same way, but with thousands of coordinates instead of twenty. Two small groups of three bars each happen to combine into something meaningful here; drag all three bars of a group together and watch a sample sentence shift language and register. Move just one of the three by itself and the meter barely twitches, since none of these coordinates alone is the feature, only the right combination of them is. The other fourteen represent the many dimensions that exist in a real model but aren't relevant to this example: some would encode entirely different concepts, some would combine into features of their own, and quite a few would would not have a clean explanation at all.