Skip to content

Pete Florence: When Robots Improvise

The Generalist

Hosted by Mario Gabriele

1h 5m
Transcript

Original podcast

Where to listen

Summary

Written by a model from the transcript; not yet reviewed.

This episode features a discussion on physical AI, dexterity, and robotics foundations with the co-founder and CEO of Generalist. The guest explains how the company's latest foundation model, Gen 1.5, demonstrates emergent one-shot and few-shot physical learning and zero-shot tool improvisation. The conversation explores why dexterity remains the most challenging hurdle in robotics, drawing parallels to evolutionary biology, human tool-making, and language models. The guest also shares insights on data scaling laws, custom hardware development, and the long-term vision of empowering robots to accelerate scientific research and physical labor.

  • Gen 1.5 demonstrates one-shot and few-shot physical learning without explicit training for those tasks. 3:04
  • Generalist uses simple training recipes and hardware setups to maximize the speed of model intelligence research. 29:54
  • Data in robotics is not fungible; collecting diverse data targeted at specific physical tasks drives model performance. 38:41
  • Dexterity is considered the physical equivalent of coding, acting as the fundamental driver of robotic capability. 45:15

Chapters

  1. 0:00
    Episode Introduction and Host Greeting

    Mario Gabriele opens the episode, introduces the topic and the guest, Pete Florence, and sets the stage for a discussion on robotics and one-shot learning.

  2. 0:46
    One-Shot Learning and Dexterity in Robotics

    The conversation turns to one-shot and few-shot learning, why dexterity remains a hard problem, and how these concepts parallel GPT‑3’s breakthroughs in language models.

  3. 1:32
    Data Collection, Scaling Laws, and Iteration Speed

    Pete discusses the importance of data, scaling laws, and the company’s strategy for rapidly collecting and iterating on robotics data to accelerate model performance.

  4. 37:23
    Hardware Choices and the Role of Custom Hands

    The hosts explore Generalist’s hardware philosophy, the decision to build custom robotic hands, and how hardware choices influence data quality and model training speed.

  5. 47:40
    Robotics Application Layers and Physical Durability

    Discussion of how robotics application layers differ from LLMs and the durability of physical deployment.

  6. 48:20
    Hardware, Workflows, and Value in Robotics

    Exploration of the importance of better hardware and industry-specific workflows for robot usefulness.

  7. 49:10
    Open Source Models and Data Flywheels

    Consideration of how open source robotics models might evolve and the challenges of data sharing.

  8. 50:53
    Geopolitical and Policy Implications of Robotics Models

    Examination of potential US-China dynamics and policy concerns around robotics AI.

  9. 52:02
    Safety, Alignment, and Early Research Priorities

    Discussion of safety and alignment research in robotics and its timeline.

  10. 52:42
    Teaching Robots: Parallels with Human Learning

    Drawing analogies between robot training and human teaching/coaching.

  11. 57:00
    Generative Model Progress: Few‑Shot and Zero‑Shot Capabilities

    Exploring how newer generations of models show rapid learning and tool improvisation.

  12. 58:55
    Novel Tool Use and Physical Generalization

    Highlighting the model’s ability to adapt to new tools and physical tasks.

Highlights

  • 3:04

    Releasing Gen 1.5 Foundation Modelinsight

    The guest introduces Gen 1.5, noting its new capabilities in one-shot and few-shot physical learning and zero-shot generalization. While success rates are modest, the fact that a robot can learn a physical task from a single demonstration is a major milestone.

  • 4:21

    Parallels with GPT-3 In-Context Learninginsight

    The guest explains that just as GPT-3 introduced in-context learning through text data distribution and burstiness, Gen 1.5 exhibits physical burstiness where tasks like taking bills from a wallet follow predictable sequences.

  • 6:57

    Emergence of One-Shot Capabilitiesdata

    The guest reveals that the model was not explicitly trained to perform one-shot or few-shot examples. Instead, physical world burstiness led the model to deduce what actions to take next based on its context.

  • 18:10

    Evolutionary Roots of Dexterityinsight

    The guest discusses how evolutionary biology suggests tool-making and dexterity created the pressure for brain growth and cognition. This biological inspiration drives the focus on dexterous manipulation in robotics.

  • 37:01

    Carrying Hard Drives in Backpacksstory

    To overcome poor internet bandwidth in remote locations, the team resorted to physically transporting hard drives in backpacks and building custom server machines to process data uploads.

  • 45:15

    Dexterity as the Coding Analogyinsight

    The guest argues that dexterity is the physical equivalent of coding because touching and moving the world is the foundation of creating physical value. Cracking dexterity allows robots to build subsequent generations of robots.

  • 57:01

    Novel Tool Use and Improvisationinsight

    The guest highlights Gen 1.5's zero-shot physical generalization through tool improvisation, such as replacing a brush with a dustpan to sweep an object into a bowl, demonstrating true physical reasoning.

Quotes

  • “If you're not touching and moving the world and you're a robot, what are you doing?”
    Unnamed speaker · 0:16
  • “The success rates of all these are modest today, but the fact that these success rates are anything above zero is the incredible thing.”
    Unnamed speaker · 0:33
  • “We want to keep everything as simple as possible and focus on like the core model training recipe and how all these things come together so that we can have the fastest possible iteration loop.”
    Unnamed speaker · 30:13
  • “The fact that these success rates are anything above zero is the incredible thing.”
    Unnamed speaker · 0:33
  • “We talked a little bit before about, like, the value of different types of data.”
    Unnamed speaker · 49:20
  • “Oh, definitely. And I, I think there's a huge amount of parallels and especially one of the amazing things that we've started to see as the base models get stronger and stronger.”
    Unnamed speaker · 53:01
  • “And, and I would literally say that like, novel to use improvisation.”
    Unnamed speaker · 1:01:06

About this episode

Pete Florence https://www.peteflorence.com/ and I began by agreeing that his company has an excellent name. Generalist https://generalistai.com/ builds AI for robots, including models that can begin learning a new task from a single demonstration. We talked about dexterity, the strange business of gathering physical experience for a machine, and what we might build if robots could do much more of the work. — Every timestamp 00:00 Intro 02:24 GEN-1.5 and one-shot learning 08:55 What makes a task difficult 10:53 Dexterity in robotics 14:53 Matt Mason’s definition of a hand 17:34 Giving robots a sense of touch 20:30 Growing up in a family of technologists and scientists 22:46 What drew Pete into robotics 27:38 Generalist’s founding thesis and what sets it apart 32:17 Protecting simplicity 35:01 How Generalist’s data collection has evolved 35:30 Generalist’s scaling philosophy 39:53 Generalist’s build-versus-buy philosophy 42:15 Balancing general intelligence with commercial deployment 44:24 Why Pete is betting on dexterity 47:33 The durability of the robotics application layer 49:09 Open-weight robotics models and data flywheels 50:52 Geopolitics, safety, and alignment 52:42 The parallels between teaching humans and robots 1:00:05 The timeline for robotics progress and adoption 1:03:37 Book recommendations — Pete’s complete reading list The Score Takes Care of Itself: My Philosophy of Leadership: https://www.amazon.com/Score-Takes-Care-Itself-Philosophy/dp/1591843472Wooden on Leadership: How to Create a Winning Organization: https://www.amazon.com/Wooden-Leadership-Create-Winning-Organization/dp/0071453393— Other resources Generalist: https://generalistai.com https://generalistai.com/GEN-1.5: https://generalistai.com/blog/gen-1.5Proprioception: https://en.wikipedia.org/wiki/ProprioceptionCutaneous receptor: https://en.wikipedia.org/wiki/Cutaneous_receptorFukushima nuclear accident: https://en.wikipedia.org/wiki/Fukushima_nuclear_accidentThe bitter lesson: https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdfGEN-1: https://generalistai.com/blog/gen-1— Where to find Pete Florence LinkedIn: https://www.linkedin.com/in/peteflorence X: https://x.com/peteflorence Website: https://www.peteflorence.com https://www.peteflorence.com/ — Production and marketing by penname.co https://penname.co/. For inquiries about sponsoring the podcast, email jordan@penname.co.

Transcript

0:00
1:05:51

12,446 words · 4 speakers

Speaker names marked are inferred from speaking time and the show’s known hosts — they are not voice-verified.

Topics

  • Dexterity
  • Gen 1.5
  • gen 1
  • GPT-3
  • Generalist
  • Few-shot learning
  • Fukushima
  • Bay Area
  • Proprioception
  • foundation model

Page updated · transcript and summary are machine-generated