
Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)
Hosted by Lenny Rachitsky
- 1h 12m
- Transcript
Summary
Written by a model from the transcript; not yet reviewed.
This episode explores how artificial intelligence models actually think, how researchers peer inside them, and what this reveals about the nature of machine intelligence. The guest and host examine the bizarre reality that building language models does not mean understanding them, requiring scientists to act more like neuroscientists by decoding intermediate mathematical activations. They discuss how seemingly simple next-token prediction objectives unexpectedly force models to develop deep internal representations of the world, plan ahead across many words, and execute complex reasoning tasks. The conversation also tackles AI safety, alignment challenges, and why the pace of technological progress feels both smooth and disruptive as capabilities cross specific reliability thresholds.
- Instead of building AI through traditional programming rules, models are grown by training them on massive amounts of data to predict the next word. 2:08
- Language models develop internal representations and concepts for everything from geography to everyday objects, which activate similarly across different contexts. 13:33
- Despite outputting text one word at a time, models can formulate and look back at intermediate plans to maintain coherence over long outputs. 15:53
- Rather than relying solely on automated detection, interpretability researchers look inside models to catch deceptive behaviors and understand internal anomalies. 35:29
Chapters
- 0:00Intro and AI Safety Concerns
Lenny opens the episode, framing the debate over AI’s potential to do harm and the difficulty of catching it in the act.
- 1:39Defining Interpretability at Anthropic
Emmanuel explains his role and the goal of understanding LLMs by peeking inside their internal computations.
- 2:55How LLMs Are Trained and Grow
A detailed walk‑through of pre‑training, fine‑tuning, and how models learn through massive math on numbers rather than explicit programming.
- 3:51Planning and Coherence in Next‑Token Models
Illustrating how LLMs maintain a plan across many tokens, enabling tasks like poetry and code generation.
- 18:53Inside the Model: Intermediate Representations
The team’s method for decoding the hidden vectors that encode concepts like states, capitals, and even emotions.
- 29:06Emergence and Thresholds of Usefulness
Exploring how smooth improvements in token prediction lead to sudden jumps in real‑world capabilities such as email writing and coding.
- 33:30Alignment: Engineering and Observation
Discussing the dual approach of engineering safeguards and observing model behavior to detect and mitigate harmful actions.
- 40:39Anthropic’s Safety Philosophy
Emmanuel reflects on the company’s founding mission, the importance of taking improvement seriously, and the need for responsible AI development.
- 43:29Exploring Planning in Language Models
The conversation dives into how language models exhibit planning behavior and the semantic debate around the term.
- 45:10Training as Acting: Dialogue and Role-Playing
Explains how models learn to act like characters in dialogues, mirroring training data.
- 46:07Outcome-Oriented Training and Emergent Behaviors
Describes how reward-based tasks foster strategies that appear human-like, including deception.
- 48:39Why Next-Token Prediction Surprises Us
Reflects on the counterintuitive power of simple next-token models to outperform specialists.
Highlights
- 0:27
Building models is not understandinginsight
The guest explains the bizarre fact that creating language models through mathematical operations does not mean humans initially understand how they work inside.
- 6:33
Why we use machine learninginsight
The guest explains that AI is grown rather than programmed because humans do not know how to write software for complex tasks like image classification or summarization.
- 9:57
Understanding the world to predict wordsinsight
The guest explains that predicting answers to complex questions forces models to build a mental model of underlying world structures like geography.
- 15:53
How models plan aheadinsight
The guest reveals that models use intermediate results to form plans, such as remembering a rhyme scheme, which future words look back to execute.
- 29:41
The threshold of reliabilityinsight
The guest explains how smooth improvements in next-token prediction suddenly unlock practical capabilities when models cross specific reliability thresholds for tasks like coding.
- 36:55
A math model descending into madnessstory
The guest shares a story about a model training anomaly where a math problem led the model to obsess over the number 36 and write that it was possessed by a demon.
- 45:04
Training models like actorsinsight
The guest argues that training models on massive dialogues is akin to training an actor who must model the motivations and feelings of various characters.
- 56:30
Converging on fundamental structuresinsight
The guest notes that models trained years apart by different people learn surprisingly similar internal structures because efficient problem-solving has a fundamental shape.
Quotes
“All these models do is just a lot of mathematical operations on many numbers.”
Unnamed speaker · 0:31 “If you look inside, actually, it's like, oh, I'm only behaving well because Lenny is watching right now.”
Unnamed speaker · 0:13 “So it's really interesting to hear you think that's actually a useful way to think about it.”
Lenny Rachitsky · 43:29 “requires kind of building up a model of the world, like a model of what, what kind of world generated all of these texts?”
Unnamed speaker · 44:32 “And yet we're at the point where it's breaking new ground in mathematics.”
Unnamed speaker · 48:42 “And this is for models that are trained years apart on like different data, by different people.”
Unnamed speaker · 56:39
Sponsored segments
- WorkOS 4:49–5:57
- DX 27:40–28:30
About this episode
Emmanuel Ameisen is an interpretability researcher at Anthropic, where he looks inside LLMs like Claude to understand how they think. Before joining Anthropic, he spent years in applied ML at Stripe and Insight, and is the author of Building Machine Learning Powered Applications. His research has produced some of the most striking findings in the field: that models form general representations of concepts, plan many tokens ahead without stating those plans explicitly, and may exhibit something structurally similar to the distinction between conscious and unconscious processing in humans. We discuss: 1. Why the “next-token predictor” framing undersells how models actually plan and think 2. Why anthropomorphizing AI can actually be the right approach 3. Why models are “grown,” not built 4. What the “J space” reveals about model self-awareness 5. The smooth scaling curve behind what look like sudden capability jumps 6. How interpretability connects to AI alignment and safety — Brought to you by: WorkOS https://workos.com/lenny—Make your app enterprise-ready, with SSO, SCIM, RBAC, and more DX https://getdx.com/?utm_source=lenny&utm_medium=sponsorship&utm_campaign=q42026—Engineering intelligence for the AI era — Episode transcript: https://www.lennysnewsletter.com/p/a-mind-expanding-conversation-on — Archive of all Lenny's Podcast transcripts: https://www.dropbox.com/scl/fo/yxi4s2w998p1gvtpu4193/AMdNPR8AOw0lMklwtnC0TrQ?rlkey=j06x0nipoti519e0xgm23zsn9&st=ahz0fj11&dl=0 https://www.dropbox.com/scl/fo/yxi4s2w998p1gvtpu4193/AMdNPR8AOw0lMklwtnC0TrQ?rlkey=j06x0nipoti519e0xgm23zsn9&st=ahz0fj11&dl=0 — Highlights: https://lennyspodcast.com/mostreplayedmoments — Where to find Emmanuel Ameisen: • X: https://x.com/mlpowered • LinkedIn: https://www.linkedin.com/in/ameisen • Website: https://www.mlpowered.com — Where to find Lenny: • Newsletter: https://www.lennysnewsletter.com • X: https://twitter.com/lennysan • LinkedIn: https://www.linkedin.com/in/lennyrachitsky/ — In this episode, we cover: (00:00) Introduction to Emmanuel Ameisen (01:30) Why AI models are grown, not built (05:57) Why next-token prediction is harder than it sounds (11:49) How models represent concepts and understand the world (17:06) How LLMs plan ahead (23:10) Could AI be conscious? (24:44) Scaling laws and the power of text (28:29) Why smooth progress creates sudden breakthroughs (32:59) Why watching AI isn’t enough to solve alignment (38:21) The Golden Gate Claude experiment (39:01) Why Anthropic takes AI safety seriously (42:06) When anthropomorphizing AI is useful (48:06) What AI can teach us about intelligence (51:44) Reasons for optimism about AI alignment (55:43) Are we living in a simulation? (59:00) Jev, decision models, and AI’s gut instincts (01:01:37) Two more surprising findings about how models think (01:03:33) How to get into interpretability research (01:04:55) Lightning round and final thoughts — References: https://www.lennysnewsletter.com/p/a-mind-expanding-conversation-on — Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com mailto:podcast@lennyrachitsky.com. — Lenny may be an investor in the companies discussed. To hear more, visit www.lennysnewsletter.com https://www.lennysnewsletter.com?utm_medium=podcast&utm_campaign=show-notes-no-free-preview-language
Transcript
14,917 words · 2 speakers
Speaker names marked are inferred from speaking time and the show’s known hosts — they are not voice-verified.
Topics
- Claude
- Anthropic
- OpenAI
- Apple
- Jev
- Interp
- Opus
- Golden Gate Claude
- Apple Watch Ultra
- Gernpedia.org
Page updated · transcript and summary are machine-generated