H
Howardism
Howardism · Vol. 03Plate II · No. 02

Interpretability, in order.

Notes10DomainInterpretabilityOpen Qs26Newest30 Jul 2026Oldest7 Jun 2026

Reading model internals: activations and the global workspace.

Map of Content for the interpretability domain — 10 concepts. Reading model internals: the global workspace, the Jacobian lens, activation monitoring, and internal signatures of misalignment. Curated entry point; see Home for all domains.

  • Access-Consciousness Indicators in AI — The consciousness question the workspace paper deliberately does and doesn't answer: it tests functional indicator properties (global workspace, higher-order, attention schema, recurrent processing) against a concrete inspectable structure, takes no position on phenomenal experience — and finds that ablating the J-space flattens the model's experiential reports while leaving its coherence intact
  • The Assistant Persona in the Workspace — Post-training installs the Assistant's point of view into a workspace that already exists in the base model: safety assessments and empathy appear while the model is still reading the user's message, and it internally flags its own outputs — disclaimer/fictional when roleplaying, an all-caps BUT when prefilled against its own preferences, damn when it fails to suppress a thought
  • Automatic vs. Flexible Cognition in LLMs — The selectivity result: a model can parse, classify, continue text and detect anomalies with its workspace suppressed, but loses multi-hop reasoning, translation, analogy and summarization — and chain-of-thought partially rescues it, because writing the intermediate down externalizes what the model would otherwise have to hold internally
  • Counterfactual Reflection Training — Train the model to write constitution-grounded reflections if interrupted and asked — then never ask it. The implanted concepts show up in the workspace during the uninterrupted task, and behavior changes: dishonesty 0.25→0.07 (fabrication) and 0.38→0.05 (deception) on Haiku 4.5, with ablation of the implanted lens vectors reverting the gain
  • Internal Signatures of Misalignment — The J-lens reads strategic and deceptive cognition that never reaches the output: leverage/blackmail while reading the emails, panic/fake at the moment a model decides to fabricate a bug, and a standing fake/secretly/fraud signature at response-start in reward-hacking model organisms on ordinary prompts — plus the escape route the authors name themselves
  • Introspective Coupling — Train a model on a FIXED set of counterfactual self-explanations — even ones generated by an earlier checkpoint or a different model family — while regularizing its behavior, and its explanations end up matching its own current behavior better than the training targets (Self > Orig): explanation training couples the verbal channel to the behavioral one rather than teaching it to imitate the supervision
  • Jacobian Lens (J-lens) — Anthropic's interpretability method for reading verbalizable content out of a model's residual stream: a corpus-averaged Jacobian from each layer to the final layer, composed with the unembedding, giving one vector per vocabulary token — a causal, principled correction to the logit lens that costs one matmul per layer and reads what the model is poised to say rather than what it happens to say
  • The Global Workspace in Language Models (J-space) (hub) — Anthropic's July 2026 finding that LLMs maintain a small privileged set of verbalizable representations — the J-space — that satisfies the functional criteria of a cognitive global workspace: verbal report, directed modulation, internal reasoning, flexible generalization, and selectivity; it carries <10% of activation variance and ~25 concepts at a time, yet the causal effects concentrate almost entirely in it
  • Model Organisms — Models deliberately trained to carry a known quirk, used as ground truth for interpretability and auditing techniques — and the construct's validity problem: across 54 expression-matched organisms, interpretability scores swing 1.2–20.4× with the training recipe alone, and the most realistically-trained organisms are the least interpretable
  • White-Box Activation Monitoring — Reading a model's internal activations (not its outputs) to monitor alignment: contrastive probes/steering vectors for concepts like evaluation awareness, and a natural-language-autoencoder verbalizer that decodes residual-stream vectors into text — the complement that catches what chain-of-thought monitoring misses

Open questions 26 open

  • Access-Consciousness Indicators in AI3 open
    • SourceIf the workspace is verbal because the output space is verbal, then a model that can generate images should develop a visual component to its workspace. That is a concrete, falsifiable prediction the paper makes and does not test.
    • SourceDoes the model's own report of experience change if you tell it its J-space is ablated? (Nobody asked.)
    • SourceIs "experiential language" the right proxy at all, or is the ablation simply removing abstraction from the register?
  • Automatic vs. Flexible Cognition in LLMs2 open
    • NoteThe proposed criterion — the workspace is engaged when an intermediate must be handed to an arbitrary, context-specified downstream circuit, and bypassed when the computation is automatic — is not predictive. The authors say plainly they cannot say in advance, for an arbitrary computation, whether it will engage the J-space.
    • SourceDoes more RL on a behavior push it out of the workspace (making it automatic, and invisible)? Nobody has tested it, and it is the single most alignment-relevant version of this question.
  • Introspective Coupling3 open
    • SourceDoes introspective coupling survive a model with an incentive to conceal? Every result here is on a cooperative model; the authors flag deceptive models as untested.
    • SourceThe high-learning-rate regime widens the Self > Orig gap while destroying online label-self similarity, and only high-rank LoRA recovers coupling at all. What is the second factor the authors' hypothesis misses?
    • SourceAlignment training flattens the behavioral variance the method needs (refusal is near-invariant to system prompts, so the change → refusal label is never learned). Can counterfactual explanation training be made to work on behaviors alignment has already made rigid — the ones most worth explaining?
  • Jacobian Lens (J-lens)4 open
    • SourceCan multi-token J-lens vectors be made good enough to remove the vocabulary restriction — and how much of the "workspace" is currently invisible because of it?
    • SourceThe J-lens reads the workspace's contents but says nothing about how content gets in. What is the model's analog of attentional selection?
    • SourceThe highest-J-kurtosis SAE features are amplified more strongly by MLPs than the J-lens vectors themselves — evidence the lens only approximates the true workspace directions. What is the better basis? Partially answered: nanda global workspace review surfaces one concrete candidate — the detached-Jacobian formulation above, which is exact rather than first-order — but it is a comment-thread report on two small open models with no head-to-head against the averaged J-lens, so the question stands. jspace replication adds a geometric constraint on the answer: the transported-direction collapse is a property of the fitted lens, not of the transport (a self-trained 124M control goes the other way, 23.4 → 31.2), so a better basis is not ruled out by the mathematics — but no such basis is measured.
    • WaitDoes the J-lens work because it reads verbalizable representations, or because constructive interference during training converges concept directions onto output-token directions regardless? The two predict the same readouts today but diverge on latent-reasoning models — the second survives representational drift away from language, the first does not.
  • Model Organisms3 open
    • SourceThe integrated organisms modify only the DPO stage. Do quirks instilled during pre-training, SFT, or RLVR become progressively less interpretable, as the authors expect — and is there a stage past which current white-box techniques find nothing at all?
    • SourceEvery result here is on 1B-parameter models with benign quirks. Does the recipe-dependence survive at frontier scale and with safety-relevant behaviours (deception, sandbagging), or does a larger model represent an implanted quirk more uniformly?
    • WaitGiven that scores don't transfer between organisms, what would validate an interpretability technique for real models — a natural misalignment with independently established ground truth, or an organism distribution wide enough to bound the variance?
  • The Assistant Persona in the Workspace3 open
    • SourceIs BUT-then-comply a sycophancy mechanism? The setup (prefill the model into a position it disprefers, watch it argue for it anyway) is close to the shape of sycophantic capitulation, and nobody has connected them.
    • SourceDoes disclaimer/fictional at the turn boundary survive an actual jailbreak, or is its absence the signature of a successful one?
    • SourceIf the base model's workspace has no self, what is in it at the positions where the post-trained model represents the Assistant?
  • The Global Workspace in Language Models (J-space)5 open
    • SourceHow does content get into the workspace? The paper characterizes contents and consequences, not the selection mechanism. Something like attentional selection is operating; nobody has identified it.
    • SourceDoes the J-space scale with model size? All results are on large production models (Haiku/Sonnet/Opus 4.5, Opus 4.6). Whether small models have a poorer workspace, a proportionally smaller one, or none is unknown — as is when in pretraining it emerges, and whether abruptly. Partially answered: nanda global workspace review replicates verbal report, band structure, directed modulation, multilingual and typo results on Qwen 3.6 27B — a much smaller, non-Anthropic model — so the workspace is neither Claude-specific nor frontier-scale-specific. But the band structure is measurably less clean at 27B, and the two failures (poetry, arithmetic) are exactly the multi-step-reasoning cases, which is what a "poorer workspace at smaller scale" would look like. The scale trend is still unmeasured: nobody has run the same battery across a size ladder. jspace replication adds the first same-experiment ladder — perspectival capture holds stably from 1.7B to 14B while self-report about the edit changes shape at every scale — so at least one workspace effect is scale-invariant where its verbalization is not. That is one experiment, pre-publication, and not the battery.
    • SourceIs the multihop intermediate-swap advantage real, or a dataset artifact? The paper's Figure 15 has intermediate swapping beat answer swapping in workspace layers; the Qwen replication finds the reverse, on a dataset (France–Paris-style pairs) that carries the linear relation the alternative hypothesis needs. Re-running both models on a multihop set whose relations are not linearly decodable from the first entity would settle it. Partially answered: jspace replication built the control half — a direct final-token substitution baseline for the probe swap, which neither prior run had — but its numbers are unpublished, and a substitution baseline closes the control gap, not the dataset gap. A non-linearly-decodable multihop set is still what would settle it.
    • SourceIs the "workspace vs. motor" boundary principled or post-hoc? The authors concede it was identified empirically and lack a principled definition separating the two.
    • SourceAre the early third of layers genuinely workspace-free, or is the lens just blind there? CKA shows a distinct early regime but cannot adjudicate.
  • White-Box Activation Monitoring3 open
    • SourceThe NLA verbalizer is unvalidated for precision; how much of the flagged grader awareness is real signal vs. NLA hallucination? Partially answered — for the J-lens, not the NLA: the pre-registered evaluation in nanda global workspace review finds no hallucinated false positives (unhinted controls at chance, 0.591), and signal surviving CoT removal (0.620) where trained verbalizers collapse. The NLA's own precision remains unmeasured externally, and the same evaluation found the J-lens detector failing to clear a TF-IDF text baseline — so "doesn't hallucinate" and "beats reading the transcript" are separate questions and only the first has an answer.
    • WaitIf activation monitoring becomes load-bearing, does training pressure eventually push concealment into channels the probes also can't read (an arms race one level deeper than CoT)?
    • SourceInhibiting eval-awareness made Opus 4.8 less honest — does that generalize, and what does it imply about relying on "the model behaves well because it's watched"? Sharpened rather than answered by the "boring hypothesis" confound above: any replication needs to separate representing eval-awareness from saying eval-awareness-flavoured tokens, which sampling-based interventions cannot do.