Skip to content

Why Scaling Prediction Cannot Create Intelligence - Alexander Mattick

Machine Learning Street Talk (MLST)

Hosted by Tim Scarfe

2h 14m
Transcript

Original podcast

Where to listen

Summary

Written by a model from the transcript; not yet reviewed.

This episode of Machine Learning Street Talk features guest Alexander Matic discussing advanced inference techniques, the limitations of energy-based models, and the evolution of deep learning architectures. The conversation explores the historical progression from early parameterization and Monte Carlo sampling to modern diffusion models and flow matching, contrasting them with energy-based models and joint embedding predictive architectures. The guest argues that energy-based models are largely obsolete for generative tasks and unpacks the theoretical underpinnings of deep learning through parametric and functional views. The discussion further examines reinforcement learning, the role of constraints in optimization, and the practical challenges of deploying world models in real-world robotics.

  • Energy-based models are generally impractical for generative tasks due to expensive unnormalized density sampling during inference. 11:40
  • Diffusion models utilize an infinite latent Markov process to avoid complex MCMC sampling steps by modeling a vector field. 27:09
  • Flow matching fixes paths a priori to combine the flexibility of normalizing flows with the superior sampling properties of diffusion. 1:01:54
  • Constrained reinforcement learning utilizes expected costs and safety thresholds to prevent hazardous outcomes in complex environments. 1:43:40

Chapters

  1. 0:01
    Ad Read and Intro

    The episode opens with a commercial break and a brief introduction to the host and guest.

  2. 0:48
    Guest Introduction and Background

    Alexander Mattick is welcomed back to the show and shares his academic and professional history.

  3. 3:54
    Overview of Inference Techniques

    The hosts outline the main inference methods that will be discussed, including energy-based models, diffusion, and flow matching.

  4. 10:45
    Energy-Based Models Explained

    The concept, training advantages, and sampling challenges of energy-based models are detailed, along with their use cases.

  5. 25:00
    Comparing Generative Paradigms

    The discussion contrasts generative efficiency, inference costs, and practical applications across energy-based, diffusion, and flow-based models.

  6. 26:08
    Diffusion and Flow Matching Foundations

    Diffusion models and continuous normalizing flows are introduced, highlighting their optimization perspective and inference efficiency.

  7. 34:22
    Unsupervised Learning and Contrastive Debate

    Mattick critiques contrastive learning, advocates for latent compression, and explains how energy-based framing fits into unsupervised methods.

  8. 41:00
    Closing Thoughts on Scaling and Intelligence

    Mattick reflects on the limits of scaling predictions and the need for smarter inference strategies to achieve true intelligence.

  9. 46:40
    Inference in Practice and Real-World Models

    The hosts explore how inference is applied in large language, vision, and reinforcement learning models, emphasizing test-time adaptation and search.

  10. 49:23
    Large Language Models as Bayesian Inference

    Explains how large language models can be viewed as performing Bayesian inference and the implications for research and application.

  11. 50:43
    Q‑Learning Fundamentals and Variants

    Provides a refresher on Q‑learning, its decomposition into reward and future value, and contrasts SARSA with Q‑star.

  12. 53:57
    Energy‑Based Models: When and Why They Matter

    Discusses the niche scenarios where energy‑based models are useful, such as ratio estimation and physics‑inspired tasks, and their limitations.

  13. 1:37:25
    Data Requirements for Robust RL

    Discussion of the need for exponential data to capture rare deviations and the limitations of relying on expected outcomes.

  14. 1:38:22
    Constraints vs. Scaling in RL

    Debate over whether incorporating constraints or simply scaling models is the better path for advancing RL.

  15. 1:43:21
    Constrained Reinforcement Learning Foundations

    Explanation of constrained RL concepts, including cost thresholds, risk measures, and the challenges of modeling constraints.

  16. 1:47:38
    Practical Challenges of Constraint Modeling

    Illustration of real-world issues such as hallucination, oscillation, and the difficulty of estimating constraints from data.

  17. 1:54:24
    Episode Wrap‑Up and Advertisements

    Closing remarks, thank‑you notes, and a series of unrelated ad reads concluding the episode.

Highlights

  • 4:23

    History of Inference and Samplinginsight

    The guest outlines the historical progression of inference from traditional variational inference and rejection sampling to MCMC, energy-based models, diffusion, and flow matching.

  • 41:03

    Usefulness of Energy-Based Modelsinsight

    The guest argues that energy-based models are largely useless outside of specific physics settings, as they lack practical generative utility.

  • 42:30

    Continuous Decomposition in Generative Modelsstrategy

    The discussion explains why continuous domains like images and audio require bottom-up decomposition via diffusion or flow matching rather than autoregressive steps.

  • 1:09:13

    Parametric versus Functional Viewinsight

    The guest breaks down theories of deep learning into a parametric optimization view and a functional space view, favoring the latter for scalable analysis.

  • 1:30:21

    Critique of Reward is Enoughdebate

    The guest analyzes Rich Sutton's Reward is Enough hypothesis, explaining that while theoretically sound, finding and optimizing arbitrary reward functions is impractical.

  • 1:43:21

    Constrained Reinforcement Learning Formulationstrategy

    The guest details constrained Markov decision processes and the mathematical trade-offs between Lagrangian approaches, projections, and local approximations.

  • 2:04:31

    World Models and Brandingdebate

    The guest and host debate whether world models and JEPA represent genuine architectural breakthroughs or simply rebranding of traditional dynamics models.

Quotes

  • “The MCMC method has an energy, which is an un-normalized density.”
    Unnamed speaker · 10:37
  • “Energy based models do not offer access to the probability of any sample, which is helpful for many applications such as anomaly detection or to correct accumulated errors when unrolling a dynamics model.”
    Unnamed speaker · 11:51
  • “Because a Q-Learn only tells you kind of what to do. It doesn't tell you, like, if you do the different thing, I expect you to be worse by this and that amount.”
    Unnamed speaker · 14:28
  • “The nice thing about energy-based models is you can do certain things cheaper. Namely, kind of the very big one is you can do kind of comparisons between data points.”
    Unnamed speaker · 32:14
  • “If you have a continuous density, like an image, you can still do the auto-aggressive, but it's not longer a natural way of doing things.”
    Unnamed speaker · 43:19
  • “If your work is, let's say, I scale up a large language model, then you probably don't need to think about this.”
    Unnamed speaker · 49:43
  • “Because especially Bayesian inference gives you a lot more structure, a lot more information than what we give it credit for.”
    Unnamed speaker · 50:11
  • “The problem with this is now that you're no longer, you have some form of, effectively you have a counterfactual inference problem where you need to deal with what actually happened versus what happened in our, like, golden state where we walked along the kind of optimal path.”
    Unnamed speaker · 53:42
  • “The idea of a continuous or like any normalizing flow is you have a base density, so a normal distribution.”
    Unnamed speaker · 56:39
  • “If you have a high dimensional function which is constant almost everywhere, and then it's much, much easier to fit if you have a two dimensional function which looks like a heartbeat monitor.”
    Unnamed speaker · 1:22:38
  • Michelob Ultra 0:01–0:23
  • LinkedIn 0:48–0:58
  • Cyberfund 2:30–3:03
  • HMP, thehousewithjordypag.com 2:15:19–2:15:46
  • Apple Watch Series 12 2:15:52–2:16:16

About this episode

Alexander Mattick is a researcher at Fraunhofer IIS and a PhD researcher at the University of Technology Nuremberg (UTN), and a regular on Yannic Kilcher's Discord. He first came on MLST in 2022, after helping research the Yann LeCun and Randall Balestriero episode on interpolation. SPONSOR: --- Cyber Fund built the Monastery to help founders ship products that were impossible a year ago. Applications for Batch 1 are now open. Apply now: https://cyber.fund --- Alexander treats inference as the thread running through modern machine learning: once you have a model, what does it cost to get an answer out of it? He works through Monte Carlo, GFlowNets, energy-based models, diffusion, normalising flows and flow matching, with four short explainers he recorded himself. He is blunt about energy-based models: you can sample from them in principle, but it is rarely worth the compute. JEPA and "world model", he says, are closer to branding than to technical categories. Next: theories of deep learning, none of which he thinks predicts enough yet to guide practice, then reinforcement learning. --- 0:00 Cold open: information is expensive 0:51 Welcome back, Alexander Mattic 2:08 Alexander's research background 2:50 Inference: densities, sampling and Monte Carlo 6:42 GFlowNets, energy functions and MCMC 9:45 Explainer: energy-based models 11:03 Why model a density at all? 17:30 From learned energies to flow matching 25:08 Explainers: diffusion and normalising flows 28:33 Are energy-based models generative? 33:22 JEPA, contrastive learning and collapse 41:13 Why non-language modalities need flows 44:51 Inference as search: branch and bound 49:43 Q-learning and delayed consequences 55:14 Flow matching, optimal transport, Fokker-Planck 1:00:03 Explainer: flow matching 1:01:49 AlphaFold, latents and scale versus architecture 1:07:52 Two families of deep learning theory 1:15:04 What a good theory would predict 1:23:53 The manifold hypothesis and compression 1:28:25 Is reward enough? 1:32:01 Control theory versus reinforcement learning 1:37:22 The Bitter Lesson and expensive information 1:42:08 Constrained RL: the constrained MDP toolbox 1:50:12 Creativity as constrained search 1:55:44 Reality is protean: when abstractions hold 2:00:32 What is a world model? 2:04:38 Prediction is not control 2:08:13 Robot demos, MPC and reliability --- REFERENCES: [6:55] GFlowNets (Bengio et al., 2021) https://arxiv.org/abs/2106.04399 [38:46] Contrastive Self-Supervised Learning (Anand, 2020) https://ankeshanand.com/blog/2020/01/26/contrative-self-supervised-learning.html [38:56] LeJEPA (Balestriero and LeCun, 2025) https://arxiv.org/abs/2511.08544v3 [47:10] RL for Node Selection in Branch-and-Bound (Mattick) https://openreview.net/forum?id=0ez68a5UqI [56:20] Flow Matching for Generative Modeling https://arxiv.org/abs/2210.02747v2 [1:12:41] Disentangling feature and lazy training in deep neural networks https://arxiv.org/abs/1906.08034v4 [1:31:05] Reward is enough (Silver) https://doi.org/10.1016/j.artint.2021.103535 [1:35:12] Learning ReLU networks to high uniform accuracy is intractable (Berner et al.) https://arxiv.org/abs/2205.13531v2 [1:40:20] Dota 2 with Large Scale Deep RL https://arxiv.org/abs/1912.06680v1 [1:45:41] Constrained Update Projection for Safe Policy Optimization (Yang et al., 2022) https://arxiv.org/abs/2209.07089 [1:46:11] SafeMPO (ICLR 2026) https://openreview.net/forum?id=1m0EU6QXj6 [1:50:17] Why Creativity Cannot Be Interpolated https://archive.mlst.ai/paper/why-creativity-cannot-be-interpolated/ [1:51:39] Invalid Action Masking (Huang and Ontañón) https://arxiv.org/abs/2006.14171 [2:00:04] Probability Theory: The Logic of Science (Jaynes, 2003) https://www.cambridge.org/core/books/probability-theory/9CA08E224FF30123304E6D8935CF1A99 [2:01:53] Training Agents Inside of Scalable World Models (Hafner et al., 2025) https://arxiv.org/abs/2509.24527v1 [2:03:43] World Models (Ha and Schmidhuber, 2018) https://arxiv.org/abs/1803.10122v4

Transcript

0:00
2:14:20

25,226 words · 8 speakers

Speaker names marked are inferred from speaking time and the show’s known hosts — they are not voice-verified.

Topics

  • reinforcement learning
  • flow matching
  • MCMC
  • diffusion models
  • JEPa
  • Q-learning
  • continuous normalizing flows
  • optimal transport
  • large language model
  • mean field theory

Page updated · transcript and summary are machine-generated