Interpretability After the Transformer: We Changed the Architecture and Kept the Microscope

Moonshot published Kimi K3's weights on 26 July. 2.8 trillion parameters, 16 of 896 experts firing on any given token, a million-token context, and an attention mechanism that for three out of every four layers isn't attention at all. Anyone can download it. The whole thing is sitting on Hugging Face right now, about 1.4 terabytes of it.

So I went looking for what you would use to actually read it, and the honest answer turned out to depend on which layer you happen to be standing in.

That surprised me more than it should have. I have spent the last two years writing about the architectures that replaced the transformer's expensive parts, and at no point did I stop to ask what those replacements did to the tools we use to look inside. It turns out the answer is not the one I expected, and not the one the doom-flavoured version of this post would have told you.

TL;DR

Mechanistic interpretability was built by pointing instruments at a softmax transformer. Linear-recurrent models replaced the static activation vector with an evolving state, diffusion removed the causal token order, and sparse routing turned out to encode geometry rather than meaning. Each of those can be rebuilt, and in the diffusion case a 28.6x transparency gap looks like it collapses to 1.1x once the intermediate states can be shown to be interpretable. The cost isn't blindness. It's that legibility stopped being a property you inherit from the architecture and became a research programme you fund, once per architecture.

What the Microscope Assumed

The lineage is short enough to state in a sentence. Induction heads in 2021, sparse autoencoders and the monosemanticity work in 2023, circuit tracing and attribution graphs by 2025. Five years, one substrate.

That substrate came with four assumptions baked in so deep that nobody wrote them down, because for five years there was nothing to contrast them against.

There is an attention map. A matrix of token-to-token weights you can render as a heatmap and stare at. Half the field's intuitions were built by staring at those heatmaps.

There is a static activation vector at each position. This is the one SAEs depend on. A sparse autoencoder decomposes a fixed-size vector into an overcomplete dictionary of features. It needs a vector to decompose.

There is a fixed causal order. Token 40 was produced after token 39 and cannot retroactively change. This makes "what did the model do before it said that" a well-formed question, which is a bigger gift than it sounds.

Every parameter is live on every token. Dense models have no routing, so there is no separate decision about which weights participate.

Meanwhile the frontier tooling keeps getting better at exactly this substrate. Anthropic open-sourced its circuit-tracing library with a Neuronpedia frontend, and the June 2026 Circuits Update introduces turn-averaged SAEs, which reconstruct the averaged residual stream of a whole conversational turn instead of every token, so a human reads far fewer active features per turn. It is genuinely excellent work. The venue is called transformer-circuits.pub.

The shape of what happened next looks like this.

One shared root, five branches, and a different instrument at the end of each one.

The Concession: It Mostly Transfers

I need to hand you the strongest argument against this post before I make it, because I nearly wrote the wrong post.

In 2024, someone asked the question directly. Does Transformer Interpretability Transfer to RNNs? answered largely yes. Then Towards Universality went further, using SAEs to isolate features from both Transformers and Mambas and finding that most features are similar across the two. Not analogous in spirit. Similar. They also partially reverse-engineered induction circuits in Mamba and found them structurally analogous to the transformer versions.

The paper does name one structural difference, and it is my favourite kind of detail because it is so specific:

"The information of one token is written into the SSM state in its next position."

— Wang et al., Towards Universality, on what they call the Off-by-One motif

That is the whole architectural difference expressed as an indexing quirk.

So the universality hypothesis has real evidence behind it. Different architectures, trained on the same data for the same objective, converge on mechanistically similar solutions. If you came here expecting me to tell you the black box got blacker, that paper is sitting there and it disagrees.

What follows is narrower. Not "the tools broke." Four specific affordances, each one traded away for a specific speedup, and each one replaceable at a cost that nobody put in the changelog.

The Roadmap, and Where the State Moved

The whole argument fits in a table. The rest of the post walks its rows.

Start with the recurrent row, because it is the cleanest case of the object changing rather than disappearing.

A linear-recurrent layer carries a state forward and updates it at each step, roughly:

$S_t=A_t\odot S_{t-1}+v_t{k}_t^{\top }$

The gate $A_t$ controls what decays out of the state. Kimi Delta Attention's contribution, inherited from the Kimi Linear work, is to make that gate act on each feature channel independently rather than as one scalar per head, so different key dimensions forget at different rates.

Now look at what a sparse autoencoder wants. It wants a fixed-size vector at a position, which it factors into a dictionary. But $S_t$ isn't a snapshot at a position. It is an accumulator whose contents depend on everything that came before and whose meaning is distributed across time. Decomposing it as though it were an activation vector answers the wrong question.

This is WriteSAE's explicit motivation, published this May: decompose the recurrent state itself, not the activations around it.

The rest of the SSM interpretability literature reads like a cottage industry, which is the tell. MambaLRP adapts layer-wise relevance propagation to selective state spaces. Activation subspace bottlenecks handle steering. Each one is a real contribution and each one exists because the general tool didn't fit.

And the geometry underneath may be genuinely different. An SSRN preprint from Roy and colleagues trained 114 TopK sparse autoencoders across Mamba-130M and 370M against GPT-2 124M and 355M baselines, varying depth, expansion factor, and seed. They report that Mamba's activations spread over a substantially higher-dimensional manifold than GPT-2's, with higher SAE reconstruction error at matched expansion factors. It has not been peer reviewed, so hold it loosely.

I want to be careful here, because this is an SSRN preprint rather than a peer-reviewed result, and because at first glance it looks like it contradicts Towards Universality.

It doesn't. Features can rhyme across architectures while the geometry they live in differs and the dictionary fits worse. One paper is measuring what the features mean, the other is measuring how they are packed. Both can be right, and taken together they say something more useful than either alone: the concepts survive the port, the coordinate system doesn't.

Where the Causal Order Dissolved, and Someone Rebuilt It Anyway

Diffusion is where I expected to write the bleakest section, and it turned into the most encouraging one.

The mechanical problem is real and it has no autoregressive analogue. In a diffusion language model, every token prediction in the canvas can change at every denoising step. There is no settled prefix. "What did the model do before it produced that token" stops being a question with an answer, because before has stopped meaning anything durable.

The DeepMind interpretability team put it plainly in June: this gives the model "the power to implement complicated distributed algorithms during the denoising process."

Their paper, How Transparent is DiffusionGemma?, is worth reading in full. It names three phenomena that simply have no counterpart in an autoregressive model. Non-chronological reasoning, where tokens settle out of temporal order. Token and sequence smearing, where representations bleed across adjacent positions. Intermediate-context reasoning, where later steps reference earlier denoising states rather than the emitted text.

Step through one token's life to see why this breaks the usual framing.

(Illustrative walkthrough of the pattern the paper documents, not a trace from a specific model run.)

So far this is the pessimistic story. Then the paper turns.

The DeepMind team measured how bad it actually was, using opaque serial depth as the proxy. Their own summary of the result:

"The opaque serial depth for DiffusionGemma is much larger (28.6X) the corresponding Gemma model. But if we were able to show the intermediates were interpretable, this would drop to 1.1X."

— Engels, McDougall, Chughtai, Kramár, Rajamanoharan et al., How Transparent Is DiffusionGemma (and why it matters)

Read that conditional carefully, because it is doing a lot of work. The 1.1x is not a measurement of how legible DiffusionGemma is today. It is what the gap collapses to once you can establish that the intermediate states are interpretable, which is the entire job. The headline finding is that the job looks doable: they report monitorability is similar between Gemma and DiffusionGemma.

Still, a 28.6x gap is the kind of number you cite when you want to argue an architecture is fundamentally unauditable, and most of it turns out to be recoverable rather than intrinsic. The reason the authors care is stated plainly in the same post: "CoT monitoring is a load-bearing aspect of many safety cases, but future models may perform more of their reasoning in latent spaces."

DLM-Scope, from February, adds a genuinely strange wrinkle. Inserting a sparse autoencoder into an autoregressive model normally costs you loss. In early diffusion layers, inserting one can reduce masked-token cross-entropy, a regime the authors describe as absent or much weaker in autoregressive baselines. Their SAE features also support diffusion-time interventions that, in their words, often outperform LLM steering.

Diffusion is arguably the success story here. Which is exactly why it makes the argument sharper rather than weaker: the affordance was destroyed, a replacement was built, it worked, and building it was a research programme rather than a library upgrade.

Where the Signal Was Never There

The MoE section is different from the others, because nothing was lost. The thing people thought they had was never there to begin with.

The folk model, stated as fairly as I can manage. A sparse mixture-of-experts model routes each token to a handful of specialists out of a large pool. Specialists specialise. So routing hands you a free semantic decomposition of the model: expert 412 is the Python expert, expert 87 handles French, and if you want to know what the model is doing you can watch which experts light up.

I believed a soft version of this for years. It is wrong, and the reason is embarrassingly simple.

A router is a linear projection followed by a top-$k$:

$g_t=\mathrm{TopK}\left(\mathrm{softmax}(W_r{h}_t)\right)$

Which means if two tokens have similar hidden states, $W_r{h}_i\approx W_r{h}_j$, and they route to similar experts. By construction. Regardless of what they mean.

Wang, Hayou and Nalisnick at Johns Hopkins made this precise in April, and their phrasing is the one to remember:

"specialization is therefore an emergent property of the representation space, not of the routing architecture itself"

— The Myth of Expert Specialization in MoEs

The empirical shadow of that claim is the number I keep coming back to. Take two models. Give them the same question. Measure how much their expert selections overlap.

Roughly 60%. Which is about the same overlap you get when they are solving entirely different problems.

If routing were semantic specialisation, that number would be much higher, because the same problem should recruit the same competence. It doesn't, because routing follows geometry and geometry is model-specific. A separate May paper on geometric coupling in sparse MoE reaches the same place independently.

Now the concession, because this one has real counter-evidence. SAE features can linearly predict expert selection, which yields faithful natural-language explanations of what a given expert responds to. And rank-1 experts under cosine-similarity routing can be monosemantic by construction, if you build them that way deliberately.

So the claim isn't that MoE routing is unreadable. It is that routing is not automatically semantic, and the free lunch a lot of us assumed we were eating was imagined. You can recover interpretability here. You just have to do the work, which is the theme of this entire post.

Kimi K3 activates 16 experts out of 896.

The Microscope Is Also Under Attack

I would be telling you a tidy story if I stopped here, and the story isn't tidy, because the instrument itself is having a bad two years independent of any architecture change.

Start with the result that should unsettle anyone who has published an auto-interpretability score. Heap, Lawson, Farnik and Aitchison applied SAEs to transformers whose parameters were sampled from a Gaussian rather than trained on text, and found that random and trained transformers produce similarly interpretable SAE latents under a standard open-source auto-interp pipeline. Their own recommendation is to treat common SAE metrics as useful but insufficient proxies, and to run randomized baselines as a matter of routine.

Then there is the testbed problem. The Model Organism Lottery, from Szablewski, Konar-Steenberg, Fornasiere, Menon and Heimersheim, trained 54 model organisms on OLMo2-1B and gemma-3-1b-it bases across seven training techniques. Interpretability varied strongly with training objective, target behaviour, architecture, and data-generation pipeline. Substantial variance remained even after controlling for how strongly the target behaviour was expressed.

The inversion is the part worth sitting with: the more realistic integrated training often produced less interpretable organisms than the standard post-hoc methods. Which means the benchmarks that look most like real training are the ones our methods do worst on.

Add seed dependence producing unstable features, and steering comparisons where SAEs don't cleanly beat simple baselines, and you get the mood of the field right now. One practitioner's summary is that the sparse-autoencoder bubble popped and the tools are still promising, and that reads about right to me.

This matters for the architecture story because the two problems compound. A method whose validity is actively contested is now also being ported to substrates it was never designed for, and the porting work is being evaluated with metrics that we know can't distinguish a trained model from a random one.

Kimi K3: Three Problems in One Downloadable File

Which brings me back to the model I opened with, and why its timing is what made me write this.

The technical report is out now. Kimi K3 runs 2.8 trillion total parameters with 16 of 896 routed experts active per token, roughly 50 billion doing work on any given forward pass, under a scheme Moonshot calls Stable LatentMoE. A million-token context. Native vision through a MoonViT-V2 encoder trained from scratch with next-token prediction rather than contrastive pretraining.

The number that matters for our purposes is the layer mix. KDA and Gated MLA are interleaved at a 3:1 ratio. Three linear-attention layers for every one full-attention layer.

Sit with what that means. Roughly a quarter of this model's layers have an attention map you can look at. The other three quarters carry recurrent state.

The payoff is why they did it: KV-cache cut by up to 75%, up to 6x decoding throughput at million-token context, and what Moonshot puts at roughly 2.5x better overall scaling efficiency than Kimi K2. That last number is the company's own, measured by the company, and it describes how well compute converts into capability rather than promising you a 2.5x cheaper inference bill. Take it as a direction of travel and not a benchmark.

So if you want to understand what Kimi K3 is doing, you need one class of method for the KDA layers, a different one for the MLA layers, and a third for the router. Nothing in the current toolkit covers the whole model. The 25% that has a readable attention map is the most legible part, and it is a minority of the network.

The largest open-weight model ever released is downloadable by anyone and legible in quarter-measures. That's a different failure of openness than the licensing problem I wrote about in April, and it's one that no license can fix.

Where This Leaves You

Every affordance in this post was sold off deliberately, and the trades were good ones. Quadratic attention gave us a readable map and cost $O(n^2)$; we took the deal. Dense parameters gave us "every weight participates" and cost throughput; we took that one too. Nobody was tricked and nothing was hidden.

The interpretability bill just arrives later than the latency win, and it arrives itemised.

Check which layers you can actually read before you promise anyone an explanation. If you are deploying a hybrid model, the debugging story did not come along with the weights. A 3:1 split means most of your network is in the part your tools handle worst.

Stop reading routing traces as semantics. Whatever expert 412 is doing, "Python" is not it. Treat routing as a compute-allocation decision, which is what it is, and get your interpretability from methods that were built to provide it.

Run randomized baselines on any interpretability metric you report. If your pipeline scores a Gaussian-initialised model about as well as your real one, the number is not measuring what you think.

Budget for the port. The DiffusionGemma result is the good news in this post: a 28.6x transparency gap closed to 1.1x. It closed because a well-resourced team went and did it. If you adopt an architecture because it is fast, the interpretability work for that architecture is a line item, not a freebie.

And the clock has already run out. Article 50 of the EU AI Act became applicable on 2 August, with fines of up to €15 million or 3% of worldwide turnover behind it, while the Digital Omnibus pushed the Annex III high-risk duties out to December 2027. So explainability became a legal obligation in Europe exactly one week after the largest open-weight model ever released turned out to be readable in one layer out of four. I don't think anyone planned that sequencing, which is somehow worse than if they had.

The microscope still works. We just spent three years redesigning the specimen, and nobody wrote it down in the changelog.

---

Sources

• How Transparent is DiffusionGemma? — Engels, McDougall, Chughtai, Kramár, Rajamanoharan, Wu, Conmy et al. (DeepMind), June 2026. The 28.6x to 1.1x result, and the source for non-chronological reasoning, smearing, and intermediate-context reasoning.
• How Transparent Is DiffusionGemma (and why it matters) — the authors' own plain-language writeup of the same work, and the source of the quotes used above.
• Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures — Wang, Ge, Shu, Tang, Zhou, He, Qiu. Most SAE features similar across Transformers and Mambas; the Off-by-One motif.
• Does Transformer Interpretability Transfer to RNNs? — the 2024 baseline answer, and the historical anchor for the concession.
• The Myth of Expert Specialization in MoEs — Wang, Hayou, Nalisnick (Johns Hopkins), April 2026. Routing reflects representation geometry; the ~60% overlap figure.
• Routers Learn the Geometry of Their Experts — independent corroboration of the geometric-coupling finding.
• WriteSAE: Sparse Autoencoders for Recurrent State — Jack Young, May 2026. Why the recurrent state, not the activation vector, is the object to decompose.
• DLM-Scope: Mechanistic Interpretability of Diffusion Language Models via Sparse Autoencoders — first SAE framework for diffusion LMs; the early-layer loss reduction and steering results.
• Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformers — Heap, Lawson, Farnik, Aitchison. SAEs on randomly initialised Pythia models score about as well as on trained ones; the authors call for routine randomized baselines.
• The Model Organism Lottery — Szablewski, Konar-Steenberg, Fornasiere, Menon, Heimersheim, July 2026. 54 model organisms, seven training techniques; casts "substantial doubt on the validity of current MOs as interpretability proxies."
• MambaLRP: Explaining Selective State Space Sequence Models and Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation — the per-architecture instruments.
• Superposition and Polysemanticity in Mamba's Selective State Space — Roy, Roy, Singh, Misra. SSRN preprint, not peer reviewed; the higher-dimensional-manifold claim.
• Kimi Linear and the Kimi K3 technical blog — KDA's channel-wise gating and the 3:1 layer mix.
• Circuits Updates, June 2026 and Anthropic's open-source circuit tracing — where the frontier tooling is going.
• EU AI Act Article 50 transparency rules — the 2 August 2026 applicability date.