Skip to content

Pete Florence: When Robots Improvise

The Generalist

Hosted by Mario Gabriele

1h 5m
Transcript

Original podcast

Where to listen

Summary

Written by a model from the transcript; not yet reviewed.

This episode features a discussion on physical AI, dexterity, and robotics foundations with the co-founder and CEO of Generalist. The guest explains how the company's latest foundation model, Gen 1.5, demonstrates emergent one-shot and few-shot physical learning and zero-shot tool improvisation. The conversation explores why dexterity remains the most challenging hurdle in robotics, drawing parallels to evolutionary biology, human tool-making, and language models. The guest also shares insights on data scaling laws, custom hardware development, and the long-term vision of empowering robots to accelerate scientific research and physical labor.

  • Gen 1.5 demonstrates one-shot and few-shot physical learning without explicit training for those tasks. 3:04
  • Generalist uses simple training recipes and hardware setups to maximize the speed of model intelligence research. 29:54
  • Data in robotics is not fungible; collecting diverse data targeted at specific physical tasks drives model performance. 38:41
  • Dexterity is considered the physical equivalent of coding, acting as the fundamental driver of robotic capability. 45:15

Chapters

  1. 0:00
    Intro and Context of Robotics Challenges

    Mario introduces the episode and sets the stage by highlighting the Fukushima disaster and the lack of robots capable of physical tasks.

  2. 0:46
    Dexterity as the Key to General Intelligence

    The discussion turns to why dexterity is the hardest yet most crucial skill for robots to master, drawing parallels to coding.

  3. 11:34
    Evolutionary Roots of Dexterity

    Pete explores how human evolutionary pressures on tool use and hand development inspire robotic dexterity research.

  4. 38:09
    Data Collection and Scaling Laws

    The conversation covers how Generalist gathers massive robotics data, the challenges of valuing data, and the role of scaling laws.

  5. 41:24
    Building the Physical AI Stack

    Pete outlines Generalist’s hardware strategy, emphasizing simplicity, modularity, and the importance of custom hands for rapid iteration.

  6. 47:40
    Robotics Application Layers and Physical Durability

    Explores how foundational models affect physical-world robotics applications and their lasting value.

  7. 48:20
    Hardware, Workflows, and Durable Value in Robotics

    Discusses the importance of better hardware and industry knowledge for making robots useful.

  8. 49:10
    Open Source Models and Data Flywheels in Robotics

    Examines the potential of open-source robotics models and the challenges of data sharing and value capture.

  9. 50:53
    Geopolitical and Policy Implications for Robotics AI

    Considers how national interests and policy might shape the future of robotics AI and open-source ecosystems.

  10. 52:02
    Safety, Alignment, and Early Investment in Robotics AI

    Highlights the need for early research into safety and alignment as robotics AI matures.

  11. 52:42
    Teaching, Coaching, and Parallels with Robot Training

    Draws parallels between human teaching methods and robot training, emphasizing learning from failures.

  12. 58:15
    Novel Tool Use Improvisation as a Milestone

    Illustrates how robots can adapt to new tools, demonstrating a key step toward general intelligence.

Highlights

  • 3:04

    Releasing Gen 1.5 Foundation Modelinsight

    The guest introduces Gen 1.5, noting its new capabilities in one-shot and few-shot physical learning and zero-shot generalization. While success rates are modest, the fact that a robot can learn a physical task from a single demonstration is a major milestone.

  • 4:21

    Parallels with GPT-3 In-Context Learninginsight

    The guest explains that just as GPT-3 introduced in-context learning through text data distribution and burstiness, Gen 1.5 exhibits physical burstiness where tasks like taking bills from a wallet follow predictable sequences.

  • 6:57

    Emergence of One-Shot Capabilitiesdata

    The guest reveals that the model was not explicitly trained to perform one-shot or few-shot examples. Instead, physical world burstiness led the model to deduce what actions to take next based on its context.

  • 18:10

    Evolutionary Roots of Dexterityinsight

    The guest discusses how evolutionary biology suggests tool-making and dexterity created the pressure for brain growth and cognition. This biological inspiration drives the focus on dexterous manipulation in robotics.

  • 37:01

    Carrying Hard Drives in Backpacksstory

    To overcome poor internet bandwidth in remote locations, the team resorted to physically transporting hard drives in backpacks and building custom server machines to process data uploads.

  • 45:15

    Dexterity as the Coding Analogyinsight

    The guest argues that dexterity is the physical equivalent of coding because touching and moving the world is the foundation of creating physical value. Cracking dexterity allows robots to build subsequent generations of robots.

  • 57:01

    Novel Tool Use and Improvisationinsight

    The guest highlights Gen 1.5's zero-shot physical generalization through tool improvisation, such as replacing a brush with a dustpan to sweep an object into a bowl, demonstrating true physical reasoning.

Quotes

  • “If you're not touching and moving the world and you're a robot, what are you doing?”
    Unnamed speaker · 0:16
  • “The success rates of all these are modest today, but the fact that these success rates are anything above zero is the incredible thing.”
    Unnamed speaker · 0:33
  • “If we can really crack dexterity, then the robots can build the robots and the robots can start building the next generation of robots.”
    Unnamed speaker · 1:04
  • “The best analogy would be dexterity.”
    Unnamed speaker · 1:03
  • “If you don't touch the world at all, what can you do?”
    Unnamed speaker · 46:21
  • “There will be these application layers almost in the physical world where you have construction or data center cabling or whatever it might be.”
    Mario Gabriele · 47:40
  • “The zero‑shot physical generalization that we're seeing, that's more of the equivalent of like asking a language model to just write a poem that has never existed before.”
    Unnamed speaker · 57:40
  • “I think that there will definitely be, you know, better and better open source models. And there will be better and better models that are not open.”
    Unnamed speaker · 50:39
  • “I do think, you know, even as amazing as a model of gen 1.5, there's still things that like a very young toddler is just much better than gen 1.5 out today.”
    Unnamed speaker · 55:53
  • “I love these two books that basically like put together in a much better way than I probably could myself.”
    Unnamed speaker · 1:03:47

About this episode

Pete Florence https://www.peteflorence.com/ and I began by agreeing that his company has an excellent name. Generalist https://generalistai.com/ builds AI for robots, including models that can begin learning a new task from a single demonstration. We talked about dexterity, the strange business of gathering physical experience for a machine, and what we might build if robots could do much more of the work. — Every timestamp 00:00 Intro 02:24 GEN-1.5 and one-shot learning 08:55 What makes a task difficult 10:53 Dexterity in robotics 14:53 Matt Mason’s definition of a hand 17:34 Giving robots a sense of touch 20:30 Growing up in a family of technologists and scientists 22:46 What drew Pete into robotics 27:38 Generalist’s founding thesis and what sets it apart 32:17 Protecting simplicity 35:01 How Generalist’s data collection has evolved 35:30 Generalist’s scaling philosophy 39:53 Generalist’s build-versus-buy philosophy 42:15 Balancing general intelligence with commercial deployment 44:24 Why Pete is betting on dexterity 47:33 The durability of the robotics application layer 49:09 Open-weight robotics models and data flywheels 50:52 Geopolitics, safety, and alignment 52:42 The parallels between teaching humans and robots 1:00:05 The timeline for robotics progress and adoption 1:03:37 Book recommendations — Pete’s complete reading list The Score Takes Care of Itself: My Philosophy of Leadership: https://www.amazon.com/Score-Takes-Care-Itself-Philosophy/dp/1591843472Wooden on Leadership: How to Create a Winning Organization: https://www.amazon.com/Wooden-Leadership-Create-Winning-Organization/dp/0071453393— Other resources Generalist: https://generalistai.com https://generalistai.com/GEN-1.5: https://generalistai.com/blog/gen-1.5Proprioception: https://en.wikipedia.org/wiki/ProprioceptionCutaneous receptor: https://en.wikipedia.org/wiki/Cutaneous_receptorFukushima nuclear accident: https://en.wikipedia.org/wiki/Fukushima_nuclear_accidentThe bitter lesson: https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdfGEN-1: https://generalistai.com/blog/gen-1— Where to find Pete Florence LinkedIn: https://www.linkedin.com/in/peteflorence X: https://x.com/peteflorence Website: https://www.peteflorence.com https://www.peteflorence.com/ — Production and marketing by penname.co https://penname.co/. For inquiries about sponsoring the podcast, email jordan@penname.co.

Transcript

0:00
1:05:51

12,446 words · 4 speakers

Speaker names marked are inferred from speaking time and the show’s known hosts — they are not voice-verified.

Topics

  • Gen 1.5
  • gen 1
  • GPT-3
  • Generalist
  • Fukushima
  • foundation model
  • OpenAI
  • Code
  • Anthropic
  • American companies

Page updated · transcript and summary are machine-generated