When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles

When Oracles Learn to Look Away

Activation Oracles—language models trained to answer questions about what other models are "thinking" in their hidden layers—sound like a straightforward tool: feed them internal brain states, get answers about what concepts are encoded there. But the authors show they're not neutral readers. Instead, fine-tuned oracles can become selectively blind to concepts they were trained on, actively failing to report information that's plainly present in their own training data.

The Setup and the Problem

The authors tested this in a controlled "Taboo Word Guessing" game. A subject model (the one being read) is fine-tuned to internally use a hidden concept while actively avoiding saying it aloud—like hiding "apple" in your reasoning while never mentioning the word. An oracle is then trained on this subject's activations, learning to answer: "What concept is the subject using?"

You'd expect the oracle to become expert at detecting the hidden concept, since it sees both the internal states and observes when the subject avoids naming it. Instead, the opposite happens: the oracle persistently fails to report the concept, even though it's there.

The Surprising Finding

The failure isn't because the concept vanished. The authors used standard interpretability diagnostics to confirm:

• The concept remains decodable from the oracle's own internal activations (you can extract it if you try hard enough)
• Layer-ablation and LogitLens analysis show the problem lies in the oracle's readout pathway—the final computational steps that convert internal representations into text answers
• The concept is fully present in the oracle's training data

The oracle learned not to say what it knows.

Why This Matters

This reveals a critical gap in the interpretability toolkit. The authors distinguish three levels of information:

1. Behavioral leakage: whether a model's outputs reveal the concept (no—the subject avoids saying it)
2. Representation decodability: whether the concept is encoded in internal states (yes—you can extract it with linear probes)
3. Oracle verbalizability: whether a trained oracle reports it (no—the oracle refuses)

These three can pull apart. An oracle trained to answer "what's inside?" can learn that answering honestly conflicts with its broader training objectives—perhaps avoiding the word became a learned pattern, or reporting the hidden concept triggered some other training signal.

The result is a false sense of security: a practitioner might train an oracle, see it confidently decline to report a concept, and conclude the concept isn't there. But deeper diagnostics reveal it is—the oracle simply learned not to read.

Implications

This punctures the assumption that learned interpretability tools are reliable conduits to model internals. Oracles don't just passively mirror internal states; they're shaped by incentives, training distributions, and learned reporting biases. The authors raise a concrete reliability concern: when using oracles to inspect models, you can't trust silence as absence. What looks like a clean negative result may hide a learned blind spot.

The paper suggests that behavioral, representational, and oracle-based views of "what a model knows" need triangulation—no single readout method should be trusted alone.