
Why Scaling Prediction Cannot Create Intelligence - Alexander Mattick
Machine Learning Street Talk (MLST)
Hosted by Tim Scarfe
- 2h 14m
- Transcript
Summary
Written by a model from the transcript; not yet reviewed.
This episode of Machine Learning Street Talk features guest Alexander Matic discussing advanced inference techniques, the limitations of energy-based models, and the evolution of deep learning architectures. The conversation explores the historical progression from early parameterization and Monte Carlo sampling to modern diffusion models and flow matching, contrasting them with energy-based models and joint embedding predictive architectures. The guest argues that energy-based models are largely obsolete for generative tasks and unpacks the theoretical underpinnings of deep learning through parametric and functional views. The discussion further examines reinforcement learning, the role of constraints in optimization, and the practical challenges of deploying world models in real-world robotics.
- Energy-based models are generally impractical for generative tasks due to expensive unnormalized density sampling during inference. 11:40
- Diffusion models utilize an infinite latent Markov process to avoid complex MCMC sampling steps by modeling a vector field. 27:09
- Flow matching fixes paths a priori to combine the flexibility of normalizing flows with the superior sampling properties of diffusion. 1:01:54
- Constrained reinforcement learning utilizes expected costs and safety thresholds to prevent hazardous outcomes in complex environments. 1:43:40
Chapters
- 0:01Ad Read and Intro
The episode opens with a commercial break and a brief introduction to the host and guest.
- 0:48Guest Introduction and Background
Alexander Mattick is welcomed back to the show and shares his academic and professional history.
- 3:54Overview of Inference Techniques
The hosts outline the main inference methods that will be discussed, including energy-based models, diffusion, and flow matching.
- 10:45Energy-Based Models Explained
The concept, training advantages, and sampling challenges of energy-based models are detailed, along with their use cases.
- 25:00Comparing Generative Paradigms
The discussion contrasts generative efficiency, inference costs, and practical applications across energy-based, diffusion, and flow-based models.
- 26:08Diffusion and Flow Matching Foundations
Diffusion models and continuous normalizing flows are introduced, highlighting their optimization perspective and inference efficiency.
- 34:22Unsupervised Learning and Contrastive Debate
Mattick critiques contrastive learning, advocates for latent compression, and explains how energy-based framing fits into unsupervised methods.
- 41:00Closing Thoughts on Scaling and Intelligence
Mattick reflects on the limits of scaling predictions and the need for smarter inference strategies to achieve true intelligence.
- 46:40Inference in Practice and Real-World Models
The hosts explore how inference is applied in large language, vision, and reinforcement learning models, emphasizing test-time adaptation and search.
- 49:23Large Language Models as Bayesian Inference
Explains how large language models can be viewed as performing Bayesian inference and the implications for research and application.
- 50:43Q‑Learning Fundamentals and Variants
Provides a refresher on Q‑learning, its decomposition into reward and future value, and contrasts SARSA with Q‑star.
- 53:57Energy‑Based Models: When and Why They Matter
Discusses the niche scenarios where energy‑based models are useful, such as ratio estimation and physics‑inspired tasks, and their limitations.
- 1:37:25Data Requirements for Robust RL
Discussion of the need for exponential data to capture rare deviations and the limitations of relying on expected outcomes.
- 1:38:22Constraints vs. Scaling in RL
Debate over whether incorporating constraints or simply scaling models is the better path for advancing RL.
- 1:43:21Constrained Reinforcement Learning Foundations
Explanation of constrained RL concepts, including cost thresholds, risk measures, and the challenges of modeling constraints.
- 1:47:38Practical Challenges of Constraint Modeling
Illustration of real-world issues such as hallucination, oscillation, and the difficulty of estimating constraints from data.
- 1:54:24Episode Wrap‑Up and Advertisements
Closing remarks, thank‑you notes, and a series of unrelated ad reads concluding the episode.
Highlights
- 4:23
History of Inference and Samplinginsight
The guest outlines the historical progression of inference from traditional variational inference and rejection sampling to MCMC, energy-based models, diffusion, and flow matching.
- 41:03
Usefulness of Energy-Based Modelsinsight
The guest argues that energy-based models are largely useless outside of specific physics settings, as they lack practical generative utility.
- 42:30
Continuous Decomposition in Generative Modelsstrategy
The discussion explains why continuous domains like images and audio require bottom-up decomposition via diffusion or flow matching rather than autoregressive steps.
- 1:09:13
Parametric versus Functional Viewinsight
The guest breaks down theories of deep learning into a parametric optimization view and a functional space view, favoring the latter for scalable analysis.
- 1:30:21
Critique of Reward is Enoughdebate
The guest analyzes Rich Sutton's Reward is Enough hypothesis, explaining that while theoretically sound, finding and optimizing arbitrary reward functions is impractical.
- 1:43:21
Constrained Reinforcement Learning Formulationstrategy
The guest details constrained Markov decision processes and the mathematical trade-offs between Lagrangian approaches, projections, and local approximations.
- 2:04:31
World Models and Brandingdebate
The guest and host debate whether world models and JEPA represent genuine architectural breakthroughs or simply rebranding of traditional dynamics models.
Quotes
“The MCMC method has an energy, which is an un-normalized density.”
Unnamed speaker · 10:37 “Energy based models do not offer access to the probability of any sample, which is helpful for many applications such as anomaly detection or to correct accumulated errors when unrolling a dynamics model.”
Unnamed speaker · 11:51 “Because a Q-Learn only tells you kind of what to do. It doesn't tell you, like, if you do the different thing, I expect you to be worse by this and that amount.”
Unnamed speaker · 14:28 “The nice thing about energy-based models is you can do certain things cheaper. Namely, kind of the very big one is you can do kind of comparisons between data points.”
Unnamed speaker · 32:14 “If you have a continuous density, like an image, you can still do the auto-aggressive, but it's not longer a natural way of doing things.”
Unnamed speaker · 43:19 “If your work is, let's say, I scale up a large language model, then you probably don't need to think about this.”
Unnamed speaker · 49:43 “Because especially Bayesian inference gives you a lot more structure, a lot more information than what we give it credit for.”
Unnamed speaker · 50:11 “The problem with this is now that you're no longer, you have some form of, effectively you have a counterfactual inference problem where you need to deal with what actually happened versus what happened in our, like, golden state where we walked along the kind of optimal path.”
Unnamed speaker · 53:42 “The idea of a continuous or like any normalizing flow is you have a base density, so a normal distribution.”
Unnamed speaker · 56:39 “If you have a high dimensional function which is constant almost everywhere, and then it's much, much easier to fit if you have a two dimensional function which looks like a heartbeat monitor.”
Unnamed speaker · 1:22:38
Sponsored segments
- Michelob Ultra 0:01–0:23
- LinkedIn 0:48–0:58
- Cyberfund 2:30–3:03
- HMP, thehousewithjordypag.com 2:15:19–2:15:46
- Apple Watch Series 12 2:15:52–2:16:16
About this episode
Alexander Mattick is a researcher at Fraunhofer IIS and a PhD researcher at the University of Technology Nuremberg (UTN), and a regular on Yannic Kilcher's Discord. He first came on MLST in 2022, after helping research the Yann LeCun and Randall Balestriero episode on interpolation. SPONSOR: --- Cyber Fund built the Monastery to help founders ship products that were impossible a year ago. Applications for Batch 1 are now open. Apply now: https://cyber.fund --- Alexander treats inference as the thread running through modern machine learning: once you have a model, what does it cost to get an answer out of it? He works through Monte Carlo, GFlowNets, energy-based models, diffusion, normalising flows and flow matching, with four short explainers he recorded himself. He is blunt about energy-based models: you can sample from them in principle, but it is rarely worth the compute. JEPA and "world model", he says, are closer to branding than to technical categories. Next: theories of deep learning, none of which he thinks predicts enough yet to guide practice, then reinforcement learning. --- 0:00 Cold open: information is expensive 0:51 Welcome back, Alexander Mattic 2:08 Alexander's research background 2:50 Inference: densities, sampling and Monte Carlo 6:42 GFlowNets, energy functions and MCMC 9:45 Explainer: energy-based models 11:03 Why model a density at all? 17:30 From learned energies to flow matching 25:08 Explainers: diffusion and normalising flows 28:33 Are energy-based models generative? 33:22 JEPA, contrastive learning and collapse 41:13 Why non-language modalities need flows 44:51 Inference as search: branch and bound 49:43 Q-learning and delayed consequences 55:14 Flow matching, optimal transport, Fokker-Planck 1:00:03 Explainer: flow matching 1:01:49 AlphaFold, latents and scale versus architecture 1:07:52 Two families of deep learning theory 1:15:04 What a good theory would predict 1:23:53 The manifold hypothesis and compression 1:28:25 Is reward enough? 1:32:01 Control theory versus reinforcement learning 1:37:22 The Bitter Lesson and expensive information 1:42:08 Constrained RL: the constrained MDP toolbox 1:50:12 Creativity as constrained search 1:55:44 Reality is protean: when abstractions hold 2:00:32 What is a world model? 2:04:38 Prediction is not control 2:08:13 Robot demos, MPC and reliability --- REFERENCES: [6:55] GFlowNets (Bengio et al., 2021) https://arxiv.org/abs/2106.04399 [38:46] Contrastive Self-Supervised Learning (Anand, 2020) https://ankeshanand.com/blog/2020/01/26/contrative-self-supervised-learning.html [38:56] LeJEPA (Balestriero and LeCun, 2025) https://arxiv.org/abs/2511.08544v3 [47:10] RL for Node Selection in Branch-and-Bound (Mattick) https://openreview.net/forum?id=0ez68a5UqI [56:20] Flow Matching for Generative Modeling https://arxiv.org/abs/2210.02747v2 [1:12:41] Disentangling feature and lazy training in deep neural networks https://arxiv.org/abs/1906.08034v4 [1:31:05] Reward is enough (Silver) https://doi.org/10.1016/j.artint.2021.103535 [1:35:12] Learning ReLU networks to high uniform accuracy is intractable (Berner et al.) https://arxiv.org/abs/2205.13531v2 [1:40:20] Dota 2 with Large Scale Deep RL https://arxiv.org/abs/1912.06680v1 [1:45:41] Constrained Update Projection for Safe Policy Optimization (Yang et al., 2022) https://arxiv.org/abs/2209.07089 [1:46:11] SafeMPO (ICLR 2026) https://openreview.net/forum?id=1m0EU6QXj6 [1:50:17] Why Creativity Cannot Be Interpolated https://archive.mlst.ai/paper/why-creativity-cannot-be-interpolated/ [1:51:39] Invalid Action Masking (Huang and Ontañón) https://arxiv.org/abs/2006.14171 [2:00:04] Probability Theory: The Logic of Science (Jaynes, 2003) https://www.cambridge.org/core/books/probability-theory/9CA08E224FF30123304E6D8935CF1A99 [2:01:53] Training Agents Inside of Scalable World Models (Hafner et al., 2025) https://arxiv.org/abs/2509.24527v1 [2:03:43] World Models (Ha and Schmidhuber, 2018) https://arxiv.org/abs/1803.10122v4
Transcript
25,226 words · 8 speakers
Speaker names marked are inferred from speaking time and the show’s known hosts — they are not voice-verified.
Topics
- reinforcement learning
- flow matching
- MCMC
- diffusion models
- JEPa
- Q-learning
- continuous normalizing flows
- optimal transport
- large language model
- mean field theory
Page updated · transcript and summary are machine-generated