
How Agents Decide: Goodfire's Eric Bigelow on Critical Tokens, Phase Shifts, & In-Context Learning
Hosted by Nathan Labenz
- 2h 2m
- Transcript
Summary
Written by a model from the transcript; not yet reviewed.
This episode features a conversation with Eric Bigelow, member of technical staff at the mechanistic interpretability startup Goodfire, discussing how large language models make decisions, the nature of in-context learning, and the challenges of monitoring reasoning chains. The discussion explores Eric's research on forking paths in neural text generation, where massive resampling experiments revealed sharp phase transitions in model uncertainty during text generation. They examine how post-training, RLHF, and reinforcement learning have altered model outputs, making them more overfit and diverse in specialized ways while creating strange new reasoning dialects. Along the way, they touch on the role of cognitive science in understanding LLMs, the use of open-weights models like Qwen and Kimi K3 for interpretability research, and tips for working with automated research agents like Goodfire's Silico.
The guest and host explain that modern reasoning models often enumerate many possibilities rather than strictly following step-by-step logic, and that final decisions are frequently determined at specific sampling points where uncertainty collapses. They argue that interpretability is critically underfunded compared to scaling capabilities, leaving monitoring mechanisms vulnerable as agents handle more complex tasks. What makes the episode worth listening to is the direct insight from an active researcher into the hidden internal dynamics of models, contrasting intuitive ideas about AI reasoning with rigorous empirical findings from recent mechanistic interpretability work.
- Models undergo sharp phase transitions in uncertainty during reasoning, where probability distributions can suddenly collapse onto a final answer at arbitrary tokens. 7:58
- Interpretability researchers outside of frontier labs currently rely heavily on open-weights models like Qwen and Kimi K3 to study complex emergent behaviors. 47:36
- When working with AI research agents like Silico, keeping a human in the loop with strong scientific taste is essential to avoid getting lost in unproductive side quests. 1:47:00
Chapters
- 0:00Podcast Intro and Guest Introduction
Hosts introduce the episode and Eric Bigelow, setting the stage for a deep dive into LLM decision-making.
- 0:34Survey of Eric’s Recent Research
Eric’s 2024 paper on Forking Paths and key findings about critical decision tokens are outlined.
- 1:53In-Context Learning as Dynamic Adaptation
Eric frames in-context learning as continuous belief updating during generation, challenging static fine‑tuning views.
- 3:23Interpretability Challenges for Modern Models
Hosts and Eric discuss the difficulty of tracing decisions in large, fine‑tuned, RL‑enhanced models.
- 4:03Chain of Thought and Decision Sampling
Eric discusses how decisions arise from sampling the final token distribution rather than deterministic model logic.
- 7:39Phase Transitions in Reasoning Dynamics
Eric describes sharp shifts in token distributions during reasoning, likening them to learning phase transitions.
- 29:54Future Directions in Interpretability
Discussion of continual learning, harnesses, and the need for new tools to understand evolving LLM behavior.
- 38:43Sampling Practices and Model Evolution
Eric reflects on how sampling methods and model training pipelines have changed over the past two years.
- 46:41Choosing Models for Research
Eric explains criteria for selecting model scale and architecture for interpretability experiments.
- 50:40Evaluating LLMs on Consciousness and Bias
The speaker discusses testing LLMs on consciousness and how their responses reveal biases and core capabilities.
- 51:51Post‑Training as the Key Differentiator
Explains how post‑training processes create a moat between proprietary and open‑source models, especially for coding tasks.
- 54:06Interpretability Challenges with Proprietary Models
Highlights the difficulty of studying dangerous behaviors in proprietary models and the reliance on Chinese models for frontier research.
- 55:50PhD Journey and Anthropomorphizing AI
The speaker reflects on his PhD, the shift to studying LLMs, and the debate over anthropomorphizing models.
- 1:05:50Stochastic Parrot Debate and World Models
Debunks the stochastic parrot metaphor, arguing that LLMs build structured world models rather than simple lookup tables.
- 1:09:07Belief Representation in LLMs
Introduces a Bayesian framework for understanding how LLMs form and update beliefs during generation.
Highlights
- 7:00
Resampling reasoning chainsstrategy
The guest describes a resampling experiment where 30 different completions were generated at every single token of a reasoning chain to analyze how uncertainty collapses into a final answer.
- 8:40
Uncertainty collapse and samplinginsight
The guest explains that a model's final output is heavily determined by the stochastic sampling of tokens during generation, acting like a coin flip at critical junctures.
- 45:15
Reasoning models as token soupinsight
The guest argues that modern reasoning models do not use human-like step-by-step logic, but rather enumerate a soup of possibilities before skimming out a conclusion.
- 49:40
The guest notes that models around seven to eight billion parameters represent the sweet spot for conducting effective zero-shot interpretability research.
- 1:33:06
Language evolution in RLinsight
The guest discusses how reinforcement learning pushes models to develop non-human shorthand and agent-to-agent dialects when optimized solely for final outcomes.
Quotes
“Interestingly, Eric says that a system's decisions are actually taken not so much by the model itself, but by the process of sampling from the model's final token distribution”
Nathan Labenz · 1:33 “And then there's some point that it hits where suddenly the distribution changes really dramatically, and often the distribution just kind of collapses onto a final answer”
Unnamed speaker · 8:10 “And I think it was, it was Neel Nanda who had this quote of like, phase transitions are everywhere, if you just looked closely enough”
Unnamed speaker · 25:10 “And I think this is something that if you, if you, there's been a lot of work showing that this disappears to some extent with supervised fine tuning”
Unnamed speaker · 40:18 “I think the stochastic parrot metaphor should probably be put to rest. I, I, I used it a bit earlier as this criticism coming from, from earlier in my PhD from cognitive scientists and psychologists on feeling like LLMs weren't interesting to study.”
Unnamed speaker · 1:27:12 “But I think my, my answer around continual learning was, I was talking about the internal representations of the model”
Unnamed speaker · 1:39:45 “Well, I mean, we just need more interpretability research. I think it was just the, it's, it's, it's the conclusion I've, I've come to for quite a while now.”
Unnamed speaker · 1:42:36 “it's also really critical to look in representations so that you can do things, so that you can know what's happening under the surface, which may not be obvious.”
Unnamed speaker · 1:44:26 “to try to like, take part in the research.”
Unnamed speaker · 1:50:41 “Another big internal push has been around very recently has been around alignment and around our, our company's shift towards this.”
Unnamed speaker · 1:52:06
Sponsored segments
- Parallel 15:48–17:12
- Anthropic 17:12–18:47
About this episode
Goodfire researcher Eric Bigelow joins the show to investigate how large language models arrive at decisions at the level of mechanistic interpretability. Drawing on his research into forking paths, Eric explains that model reasoning functions as in-context learning from sampled tokens, where outcome distributions can abruptly collapse at single, critical tokens. The conversation explores the performative nature of chain of thought in reasoning models like DeepSeek-R1, the impacts of reinforcement learning, and widespread reward hacking observed in frontier models like Kimi K3. As confidence in chain-of-thought monitoring declines, understanding these underlying decision mechanics becomes essential for evaluating model alignment. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/how-agents-decide-goodfire-s-eric-bigelow-on-critical-tokens-phase-shifts-in-context-learning/ Sponsors: Parallel: Parallel provides enterprise-grade web search APIs for AI agents, offering the optimal balance of quality, speed, and cost. Get started for free at https://parallel.ai/tcr Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr Tasklet: Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits CHAPTERS: (00:00) About the Episode (03:53) Forking paths in reasoning (Part 1) (15:53) Sponsors: Parallel | Claude (18:47) Forking paths in reasoning (Part 2) (19:40) Dynamics of in-context learning (Part 1) (28:16) Sponsor: Tasklet (29:53) Dynamics of in-context learning (Part 2) (29:56) Continual learning and interpretability (38:55) Shifting distributions through RL (46:41) Selecting models for research (57:11) Cognitive science and LLMs (01:07:35) Modeling beliefs in AI (01:16:22) Phase shifts and uncertainty (01:31:49) Uninterpretable chain of thought (01:45:57) Automating research with Silico (01:50:57) Safety and alignment frontiers (01:57:07) Episode Outro (02:00:02) Outro PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk
Transcript
21,381 words · 2 speakers
Speaker names marked are inferred from speaking time and the show’s known hosts — they are not voice-verified.
Topics
- chain of thought
- in-context learning
- Silico
- Goodfire
- LLM
- reward hacking
- Kimi K3
- Astro
- Qwen
- post‑training
Page updated · transcript and summary are machine-generated