
The Neuron Basis: Goodfire's Curt Tigges on Agentic Interp, Monosemanticity & Reverse Engineering
- 1h 56m
- Transcript
Summary
Written by a model from the transcript; not yet reviewed.
This episode explores the Goodfire paper "Neurons as an Interpretable Basis," challenging the widespread assumption that neural network internals are too messy to study directly. The guest discusses how modern SwiGLU mixture-of-experts models like Qwen-3.6 35B can have up to 90% of their neurons ablated without losing core performance, and how frontier LLM agents can be used to scale the characterization of individual neurons. The discussion covers the mechanics of gated MLPs, positive and negative activations, and the prospects for moving from individual neurons toward ambitious reverse engineering and model alignment.
- Models can maintain core performance after ablating nearly ninety percent of their neurons. 2:48
- Frontier models can be used to iteratively generate and refine hypotheses about what individual neurons do. 42:30
- Facts are stored not within single neurons, but within the exponential number of connections between them. 1:21:41
- Mixture-of-experts models show high specialization, with individual experts focusing on narrow topics like math or law. 1:25:47
Chapters
- 0:00Podcast Introduction and Host Overview
The hosts introduce the episode, the show, and the focus on neuron interpretability.
- 1:31Model Architecture Deep Dive
Detailed explanation of gated MLPs, up and gate projections, and Swiglu activation behavior.
- 2:21Sparsity Ablation Strategies
Comparison of mean ablation, top‑K, and percentile‑based neuron removal techniques.
- 3:32Neuron Scientist Agent Workflow
Description of the agent that generates hypotheses about neuron functions and validates them.
- 6:11Monosemanticity vs Polysemic Neurons
Discussion of how neurons can encode multiple related or unrelated concepts and the challenges of labeling.
- 6:50Closing Reflections on Interpretability
Hosts reflect on the significance of the findings and future directions for interpretability research.
- 42:30Neuron Scientist Validation Process
How hypotheses are tested by a separate model that predicts neuron activations and patches them back in.
- 46:40Performance Impact of Ablation
Analysis of how removing neurons affects reasoning, confidence, and benchmark scores.
- 51:09Interpretability Metrics and KL Recovery
Explanation of KL divergence recovery, weighting, and alternative evaluation methods.
- 1:00:00Neuronal Signatures of Topics
Discussion of how specific neuron subsets activate for different topics, revealing subcircuits.
- 1:02:16Noun vs Verb Neuron Functions
Exploration of neurons acting as verbs—sending signals that influence downstream behavior rather than just detecting features.
- 1:05:08Ambitious Reverse Engineering Roadmap
Outline of steps needed to move from neuron-level understanding to higher-level behavior explanations.
- 1:22:38Factual Recall and Neuron Connectivity
Examination of how facts are stored in connections rather than single neurons and the impact of sparsity on recall.
- 1:26:53Expert Neuron Specialization
Analysis of how expert modules are highly localized to specific topics, challenging previous assumptions of diffuse expertise.
Highlights
- 7:17
Challenging Neural Interp Dogmainsight
The guest explains the core argument that individual neurons in neural networks are far more interpretable than previously believed by the field.
- 12:37
Model Architecture Detailsdata
The host and guest break down the Qwen model architecture, which features 40 layers, 256 experts per layer, and 512 neurons per expert.
- 15:17
Gated MLP Mechanismstrategy
The guest explains how gated MLPs use separate up and gate projections to act as logical conjunctions and switching functions.
- 27:43
Sparsity and Mean Ablationstrategy
The guest details how mean ablation is used when testing sparsity, noting that the average activation for bipolar neurons is close to zero.
- 37:16
Performance Boost from Ablationdata
The host notes that removing 80 to 90 percent of neurons actually produces a slight performance bump on certain ARC-AGI puzzles.
- 41:43
The Neuron Scientist Agentstrategy
The guest describes the automated agent used to cluster neuron activations, form hypotheses, and run tools to explain individual neurons.
- 1:21:41
Where Facts Are Storedinsight
The guest explains that facts do not live inside individual neurons, but rather in the high-dimensional connections between multiple neurons.
Quotes
“individual neurons are in fact much more directly interpretable than previously believed, that most neurons in fact have one main function”
Unnamed speaker · 0:43 “In short, Kurt shows that you can ablate roughly 90% of the individual neurons, keeping only those with the strongest activations, with surprisingly minimal performance degradation.”
Unnamed speaker · 2:48 “We attribute their success, as all else, to divine benevolence.”
Unnamed speaker · 2:01 “We have a graph of that in our paper where like the auto interp stuff is the lowest.”
Unnamed speaker · 46:40 “We recover 74% of the KL divergence loss, which I'd like to understand a little bit better.”
Unnamed speaker · 50:39 “And the interesting thing about this also, from a mechanistic perspective, is that we can see the signatures of these concepts in the weights themselves,”
Unnamed speaker · 1:00:00 “So first of all, I think the shift from noun to verb can be seen in a couple of different ways.”
Unnamed speaker · 1:02:15 “There's a lot of reasoning specific ones that are just really interesting in this way.”
Unnamed speaker · 1:04:46 “I think there's more than just neurons to reverse engineering. We need to understand attention and other things in the model.”
Unnamed speaker · 1:05:08
Sponsored segments
- Anthropic 22:59–24:16
- Outsystems 24:21–26:21
- Tasklet 39:01–40:44
- A16Z 1:54:14–1:55:02
About this episode
Goodfire technical staff member Curt Tigges discusses research revealing that individual MLP neurons in modern language models are far more interpretable and functionally bounded than previously thought. Using an automated agent to test hypotheses across architectures like Qwen3.6-35B-A3B, Tigges explains how modern gated MLPs act as legible conjunctions and control signals capable of extensive dynamic sparsity without lasting benchmark loss. By showing that many internal representations can be directly enumerated, this work offers an alternative to complex external featurizers and provides practical avenues for behavior monitoring and persona steering. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/the-neuron-basis-goodfire-s-curt-tigges-on-agentic-interp-monosemanticity-reverse-engineering/ Sponsors: Claude: Claude is the AI collaborator built for problem solvers, extending your thinking across coding, modeling, and strategic workflows. Get started and discover Claude Pro at https://claude.ai/tcr OutSystems: OutSystems is the leading agentic systems platform that enables enterprises to engineer, orchestrate, and govern AI applications on a single unified platform. Learn more and see how it works at https://outsystems.com/tcr Tasklet: Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits CHAPTERS: (00:00) About the Episode (05:55) Rethinking individual neuron interpretability (12:25) Understanding gated MLPs (22:57) Sparsity and ablation experiments (Part 1) (23:03) Sponsors: Claude | OutSystems (26:20) Sparsity and ablation experiments (Part 2) (Part 1) (39:07) Sponsor: Tasklet (40:44) Sparsity and ablation experiments (Part 2) (Part 2) (41:30) The neuron scientist agent (55:10) Monosemanticity versus circuit mechanics (01:09:37) Reverse engineering for alignment (01:20:24) Factual recall and experts (01:28:41) Designing interpretable architectures (01:36:16) Restoring reasoning performance (01:45:43) Future directions and wrap-up (01:51:34) Episode Outro (01:54:15) Outro PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk
Transcript
16,919 words · 3 speakers
Topics
- MLP
- Relu
- Swigglu
- Silu
- residual stream
- MedQA
- Gemma
- Gemma four
- Red Glue
- Kimi K3
Page updated · transcript and summary are machine-generated