Skip to content

Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again

Training Data

53m
Transcript

Original podcast

Where to listen

Summary

Written by a model from the transcript; not yet reviewed.

Rich Sutton, pioneer of reinforcement learning, and his former student and co-founder Khurram Javed join the podcast to discuss the limitations of current large language models, the principles of their new venture Oak Lab, and why continual learning is essential for true artificial intelligence. The guests argue that static foundation models that freeze their weights after training are fundamentally limited by human bottlenecks, and that systems must instead learn continuously from their own direct experience in an infinitely complex world. They explain how the famous Bitter Lesson points away from hand-crafted human knowledge and synthetic data toward algorithms that scale with computation over time. Throughout the conversation, the participants debate the role of supervised learning, the feasibility of synthetic data generation, and the path toward self-maintaining architectures that can form their own abstractions and plan effectively.

  • Prevent catastrophic forgetting during continual learning by assigning separate, meta-learned step sizes to every individual weight in a network. 40:16
  • Recognize that synthetic data generation remains bottlenecked by human expertise because people must still decide what constitutes good or bad data. 12:31
  • Build intelligent systems that continue learning from their own sensory experience after deployment rather than freezing their weights after pre-training. 23:35
  • Focus research on algorithms that scale with computation and raw experience rather than relying on hand-coded human knowledge or static datasets. 9:13

Chapters

  1. 0:00
    Podcast Introduction and Host Setup

    The episode opens with a brief host introduction and a quick overview of the guests and topic.

  2. 0:59
    Rich Sutton’s Early Career and Motivation

    Rich Sutton explains his decision to focus on reinforcement learning in 2003 during an AI winter and his personal health struggles that motivated his research.

  3. 1:53
    The Bitter Lesson and Its Foundations

    Discussion of Sutton’s seminal essay, the historical context, and the core idea that algorithms that scale with computation outperform hand‑crafted knowledge.

  4. 11:03
    Synthetic Data vs. Real Experience

    Exploration of the limits of synthetic data generation, the big‑world hypothesis, and why agents need to learn from their own experience rather than curated datasets.

  5. 51:02
    Exploring a Single Design for Multiple Minds

    The speakers discuss whether a single AI design can encompass all learning tasks or if multiple specialized systems are required.

  6. 51:36
    The Limits of a Single System’s Learning Capacity

    They explain that one system cannot learn infinitely many things, highlighting the need for multiple systems learning from diverse experiences.

  7. 52:41
    Commitment to Super Alignment and Productivity

    They emphasize the importance of being super aligned to work productively and scale progress together.

Highlights

  • 2:31

    Conviction During An AI Winterstory

    Rich Sutton explains how he maintained his research focus on reinforcement learning despite battling cancer during an AI winter in 2003.

  • 9:03

    The Essence of The Bitter Lessoninsight

    Rich Sutton defines the core message of his famous essay: researchers must focus on learning methods that scale with computation rather than injecting human knowledge.

  • 10:02

    LLMs as a Mixed Exampleinsight

    The guests evaluate large language models as both a positive example of scaling computation and a negative example limited by finite internet data.

  • 11:28

    Rejecting Synthetic Data Generationinsight

    The guests argue that generating synthetic data to scale models is fundamentally flawed because it creates a microscopic, human-bottlenecked simulation of a complex world.

  • 23:35

    The Problem with Frozen Weightsinsight

    The speakers point out that modern language models are not continual learners because their weights completely stop changing once deployment begins.

  • 40:09

    Fixing Catastrophic Forgettingstrategy

    Rich Sutton outlines the technical solution to catastrophic forgetting: meta-learning separate step sizes for every weight and using generative units.

  • 46:43

    Energy Efficiency Projectionsdata

    Khurram Javed and Rich Sutton discuss how two orders of magnitude of Moore's Law improvements over a decade could enable complex models to run on low power.

Quotes

  • “All learning is continual. We always act and we learn. That's just the normal way of thinking.”
    Unnamed speaker · 0:30
  • “If everything goes right, we implement the architecture. We can have genuine continual learning and we can form abstractions so that we can do planning and reasoning and so on.”
    Unnamed speaker · 49:20
  • “It's an important part, you know, it's like 20% or a quarter of intelligence.”
    Unnamed speaker · 50:52
  • “The many different, there'll be many minds.”
    Unnamed speaker · 51:24
  • “So a single system would never be able to get to a point where it can learn everything.”
    Unnamed speaker · 51:55
  • “We want to be super aligned.”
    Unnamed speaker · 52:40

About this episode

Rich Sutton, who helped pioneer reinforcement learning and wrote the seminal AI essay The Bitter Lesson, has now cofounded Oak Lab with his former student Khurram Javed. Their goal: to build agents that continuously learn from their own experience rather than from us. Rich doesn't think he holds a radical view: "I'm not weird. The field is weird." He says all learning is continual, and the field is the one that needed a new name for it. Rich and Khurram argue synthetic data is "a big mistake." Their "big world hypothesis" is that the world is massively more complex than any agent or simulator, so approximations have to be updated continuously rather than frozen at deployment. Rich calls LLMs an unanticipated scientific breakthrough, but says they represent roughly a quarter of intelligence. He says catastrophic forgetting is "totally curable" with the ideas behind their continual backprop algorithm. Khurram explains why the frontier labs can't follow: they sit in a local minimum where a new paradigm gets worse before it gets better. Their target, five to ten years out, is a trillion-parameter mind that keeps learning, stays coherent, and runs on 20 watts. Hosted by Sonya Huang and Alfred Lin, Sequoia Capital 00:00 Introduction 02:10 An AI winter, a cancer diagnosis, and the move to Alberta 07:07 Writing "The Bitter Lesson," and what people get wrong 09:53 Are LLMs a positive or a negative example of it? 11:03 Synthetic data is "just a big mistake," and the Big World Hypothesis 18:01 AlphaGo, human priors, and why prior knowledge and learning should be friends 22:37 "Their weights never change": do LLM assistants actually learn? 26:09 Babies, squirrels, and why no animal learns by supervised learning 32:02 Rockets, imagination, and where paradigm shifts come from 36:42 The Alberta Plan and its 12 steps 38:53 Catastrophic forgetting and the cure 43:43 Oak's biggest ambition: a self-maintaining mind 47:56 Why the big labs are stuck in a local minimum 49:13 If everything goes right: LLMs, many minds, and hiring

Transcript

0:00
53:43

9,419 words · 4 speakers

Topics

  • synthetic data
  • The Bitter Lesson
  • continual learning
  • human knowledge
  • reinforcement learning
  • AlphaGo
  • continual backprop
  • University of Alberta
  • step size optimization
  • ImageNet

Page updated · transcript and summary are machine-generated