Skip to content

Physical AI at NVIDIA

Software Engineering Daily

56m
Transcript

Original podcast

Where to listen

Summary

Written by a model from the transcript; not yet reviewed.

This episode explores the frontier of physical AI and robotics, focusing on how world models are bringing intelligence into the physical world. The discussion covers the architectural differences between digital AI and physical AI, including how multi-modal world models process sensory data, video, and audio to predict future states and plan actions. The conversation highlights the two-tower architecture combining reasoning and action, the heavy data pipelines required for training, and the simulation infrastructure needed for reinforcement learning. Additionally, the episode addresses safety, real-world deployment on edge devices like Nvidia's Jetson platform, and how software developers can start building physical AI projects.

  • Software developers can start building physical AI projects using open-source models, simulators, and affordable hardware setups. 46:43
  • World models use a two-tower architecture where a low-frequency reasoning brain plans tasks and a high-frequency action tower controls motor currents. 12:28
  • Training physical AI requires a heterogeneous computing architecture spanning data preprocessing, transformer training, and physics simulation. 24:07
  • World models evaluate multiple counterfactual futures at inference time to select the safest and most optimal action. 36:32

Chapters

  1. 0:00
    Introduction to Physical AI and NVIDIA Cosmos

    Matt Merrill introduces the episode and Amit Goel explains NVIDIA’s focus on world models and the Cosmos family for physical AI.

  2. 2:58
    Defining Physical AI vs Digital AI

    Amit clarifies that physical AI deals with atoms, sensors, and 3D dynamics, contrasting it with LLMs and coding agents.

  3. 6:05
    World Models: Inputs, Outputs, and Planning

    The conversation explains how world models ingest multimodal observations and predict future states, enabling planning and action in robots and cars.

  4. 10:00
    Cosmos Three Edge: Deploying on Jetson Devices

    The episode details how the 4‑billion‑parameter Cosmos Three Edge model runs in real time on passport‑sized Jetson hardware, enabling on‑device reasoning.

  5. 16:37
    Two‑Tower Architecture and Latent Space Sharing

    Amit describes the reasoning (planning) and action towers that share a latent space, allowing high‑bandwidth communication between the two brains.

  6. 22:37
    Data Pipelines and Training Infrastructure

    The discussion covers the need for new data pipelines, the Curator system, and heterogeneous compute stacks for training, simulating, and deploying world models.

  7. 32:51
    Safety, Guardrails, and Hallucination Mitigation

    Amit outlines strategies to prevent out‑of‑distribution hallucinations, including conservative defaults, simulation testing, and planning‑stage guardrails.

  8. 45:28
    Getting Started: From Simulation to Real Robots

    Matt and Amit explain how developers can begin with simulation, acquire affordable robotic arms, and deploy trained policies onto Jetson devices.

  9. 50:33
    Future Outlook: Ubiquitous, Multi‑Task Robots

    Amit predicts that within five years robots will be multifunctional, ubiquitous, and easier to program thanks to foundation models.

  10. 51:56
    Demonstrating Tasks to Robots

    Explains how showing a robot a task once or twice can enable it to learn and perform the task automatically.

  11. 52:50
    Future of Multifunction Robots

    Discusses the vision of robots that can be instructed simply by showing them what to do.

  12. 53:46
    Getting Started with Robotics via Simulation

    Recommends starting with simulation to learn robotics concepts before moving to hardware.

Highlights

  • 3:09

    Working With Atoms Versus Bitsinsight

    The guest explains that the primary difference between digital AI and physical AI is dealing with atoms, sensors, spatial information, and physical dynamics.

  • 12:28

    Two-Tower Architecture for Roboticsstrategy

    The guest breaks down the 100-millisecond pipeline from sensor capture to motor command, utilizing synchronized data and a two-tower model for reasoning and action generation.

  • 24:07

    Heterogeneous Compute Requirementsinsight

    Training physical AI requires three distinct computer environments for data processing, reinforcement learning via physics simulation, and edge deployment.

  • 36:32

    Evaluating Counterfactual Futuresinsight

    World models perform counterfactual reasoning at evaluation time, predicting ten different futures and choosing the safest and most objective-aligned outcome.

  • 43:21

    Cosmos Three Edge Modeldata

    Nvidia developed a 4 billion parameter version of their Cosmos model that runs in real-time on a passport-sized Jetson Thor computer inside robots.

  • 46:43

    Getting Started with Physical AIstrategy

    A developer can get started in physical AI with a thousand-dollar budget using an existing computer, a simulator, and a 3D-printed robotic arm.

Quotes

  • “physical AI essentially is a model that can take as inputs the information of the world around you and predict what's going to happen in that world.”
    Unnamed speaker · 3:34
  • “So the world models can predict the world around you. It can do a 3D representation of the world.”
    Unnamed speaker · 7:56
  • “The other thing is, you can say, hey, this is my current state. I'm planning to take this action. Can you tell me what is going to happen if I do this?”
    Unnamed speaker · 8:24
  • “We are getting to the point where we can build this general physical intelligence models that you can use for a lot of applications in the physical world.”
    Unnamed speaker · 5:18
  • “It seems impossible if you tell a robotics person today that, hey, you know, programming a robot is just like, go show it.”
    Unnamed speaker · 52:08
  • “I mean, if we didn't have this inflection point with software based L L Ms in the past year, I probably would not believe you, but seeing what happens in the past year, I, I believe it.”
    Unnamed speaker · 52:32
  • “The best way, if you want to, you know, get into robotics, it's just starting, I believe with simulation because that is where, you know, it still feels like you're working with software.”
    Unnamed speaker · 53:46
  • “So I think that's the first thing is get out, start working with the simulator, get exposed to those terminologies and the, you know, the datas that, that you're not used to when you're working with code and software.”
    Unnamed speaker · 54:20
  • VariaCode 18:55–19:50
  • Fingerprint 19:55–20:54
  • GuardSquare 20:45–22:01

About this episode

In recent years, the main impact of AI has been on the digital world, and most notably software engineering. However, its longer-term effect on the physical world may prove even more profound, reshaping how cars drive themselves, how factories run, and how robots move through homes and warehouses. World models are at the center of the movement to bring AI into the physical space. These models take in observations of the current state of the world and predict what comes next, which lets a robot plan its next move or a car anticipate the road ahead. NVIDIA has been building a family of these models called Cosmos, and sits at the frontier of physical AI and robotics. Amit Goel https://www.linkedin.com/in/amit-goel-759146ahas been atNVIDIA https://www.nvidia.com/ for fifteen years, and he is currently the Head of Robotics Ecosystem and Edge AI Product. In this episode, Amit joins Matt Merrill to discuss what separates physical AI from digital AI, how world models plan and predict, the two-tower architecture that lets a robot reason and act at once, why training on video demanded a new data pipeline, and how any developer can start building in this space. Free Physical AI Courses: https://nvda.ws/4xqCfo9 Cosmos: https://nvda.ws/4zJ4iRj GR00T Teleop + LeRobot: https://huggingface.co/blog/nvidia/nvidia-isaac-teleop-and-gr00t17-in-lerobot Sponsorship inquiries: sponsor@softwareengineeringdaily.com mailto:sponsor@softwareengineeringdaily.com The post Physical AI at NVIDIA https://softwareengineeringdaily.com/podcasts/physical-ai-at-nvidia/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.

Transcript

0:00
56:56

9,831 words · 3 speakers

Topics

  • Nvidia
  • Cosmos
  • LLMs
  • LLM
  • Hugging Face
  • Jetson
  • Nest thermostat
  • Boston Dynamics
  • Waymo
  • Alpha Mayo

Page updated · transcript and summary are machine-generated