Interpretability is a teaching problem — mechanistic insight only matters if humans can act on it
Mechanistic interpretability has made striking progress: sparse autoencoders recover monosemantic features from polysemantic neurons, circuit analysis traces specific behaviors to attention head compositions, and tools like Neuronpedia let you explore these features interactively. But there's a gap between finding a circuit and knowing what to do about it. Interpretability becomes useful only when the insight can be acted on — by engineers debugging a model, by auditors assessing risk, by users calibrating their trust. That makes interpretability fundamentally a teaching problem: how do you translate mechanistic insight into actionable understanding?
The SAE literature illustrates both the promise and the gap. Anthropic's 2024 work on Claude 3 Sonnet extracted millions of features, including safety-relevant ones like deception and power-seeking. The automated interpretation pipeline — LLM labels generated from max-activating examples, scored by how well the label predicts activations on new examples — is impressive. But what does an engineer do when the SAE reports a "deception feature" activated? The feature is evidence, not a diagnosis. Acting on it requires understanding the circuit that drives it, the inputs that trigger it, and the intervention points that could change it.
The interface between insight and action
This is where gradient attribution methods and circuit analysis meet. Attribution tells you which input tokens influenced which output logits — it's the "why did the model say this" explanation. Circuit analysis tells you which internal components implemented that computation — the "which neurons did the work" explanation. Both are necessary for actionability. Attribution without circuits is explanation without mechanism; circuits without attribution are mechanism without context.
The Jacobian lens and global workspace theories extend this: they're attempts to trace how information flows through the model, not just which components were active. But the output of these analyses is still a mechanistic description — activation patterns, attention weights, feature directions. The teaching problem is translating that description into something a human can use: "the model attended to this token because of this circuit, which means you should X to fix it."
What actionable interpretability looks like
For interpretability to be actionable, it needs to answer three questions: (1) What behavior is this explaining? — the specific output or decision the user cares about; (2) What mechanism produced it? — the circuit, feature, or attention pattern that drove the behavior; (3) What intervention changes it? — the fix: retrain with different data, patch the circuit, adjust the prompt, add a guardrail.
The Tracr compiler demonstrates one direction: build a synthetic transformer from a known program, then verify that interpretability tools recover that program. This is ground-truth evaluation — you know the right answer, so you can score the interpretability method. But Tracr's programs are toy examples. The real test is whether interpretability helps you debug a model that's failing in production, where you don't know the right answer and need the tool to tell you what's broken.
The teaching gap in AI safety
This is also the gap in AI safety work. Mechanistic interpretability is sold as a safety tool — if we can see what the model is doing, we can catch deception before it's deployed. But catching it requires recognizing the pattern as dangerous, which requires a human who understands both the mechanism and the threat model. That's a teaching problem: how do you train engineers and auditors to recognize dangerous circuits the way a mechanic recognizes a failing part?
One concession: not all interpretability needs to be actionable immediately. Basic science — understanding how models represent information, how circuits compose, what universality means — is valuable even without immediate application. But if interpretability is going to justify its cost in safety-critical deployments, it needs to cross the gap from insight to action. And that crossing is a teaching and interface design problem, not a mechanistic one.