Skip to content

AI:AM: A Level We Shouldn't Pass? Notes from The Curve + Tokens vs. Salaries & Is SaaS Cooked?

The Cognitive Revolution

Hosted by Nathan Labenz

1h 28m
Transcript

Original podcast

Where to listen

Summary

Written by a model from the transcript; not yet reviewed.

This episode covers wide-ranging notes from the Curve conference in Berkeley, featuring insights from frontier lab insiders, chip architects, software engineers, and risk researchers. The host and guests discuss the reality of AI scaling, the growing importance of synthetic data and reinforced learning, memory constraints in AI hardware, and how coding agents are transforming software development and threatening traditional SaaS business models. Throughout the conversation, the speakers examine the operational realities of relying on large language models for complex engineering and research tasks, the challenges of evaluation and reward hacking, and the societal implications of rapidly advancing intelligence.

  • Frontier lab insiders are taking the possibility of hard caps on pre-training scale seriously as models continue to grow rapidly in power. 11:34
  • By shifting to commodity memory and achieving high memory bandwidth utilization, specialized chip architecture can drastically scale context capacity for multiple concurrent agents. 33:56
  • Software engineering is rapidly evolving from writing individual lines of code to managing multiple coding agents and maintaining rigorous evaluation harnesses. 51:30
  • Traditional mid-tier SaaS applications face a severe threat as companies increasingly use coding agents and bounties to build custom internal software solutions. 57:11

Chapters

  1. 0:00
    Episode Intro and AI Ceiling Concept

    Hosts introduce the episode and discuss a frontier lab leader’s warning about a hard limit on AI intelligence.

  2. 0:57
    Memory Wall and Chip Innovation

    Thomas Somers explains the surge in memory costs, Positron’s chip design choices, and the impact on scaling AI models.

  3. 8:13
    Curve Highlights: Rapid AI Growth and RSI Concerns

    Discussion of how AI capabilities are accelerating, the rise of recursive self‑improvement, and the debate over potential limits.

  4. 14:18
    Policy and Government Intervention Outlook

    Hosts discuss emerging expectations of congressional action on AI regulation and the potential shift in political dynamics post‑election.

  5. 48:21
    Software Engineering in the Agent Era

    Sean Wang talks about how software engineers are adapting to AI agents, the changing demand for coding skills, and the new focus on data and oversight.

  6. 54:11
    Token Bandwidth and SaaS Bounties

    Explores how token consumption scales with SaaS usage and introduces a bounty model to replace costly subscriptions.

  7. 56:05
    Adapting Legacy Teams to AI

    Discusses hiring traditional event professionals and gradually integrating AI tools to improve workflow and adoption.

  8. 59:04
    SaaS Customization and Market Dynamics

    Analyzes why SaaS platforms like Salesforce thrive on customization and how this shapes the software market.

  9. 1:00:25
    Token Bandwidth Limits and Silicon Constraints

    Considers the ultimate cap on token usage imposed by silicon production and its impact on pricing and availability.

  10. 1:03:03
    AI Engineer Conference Promotion

    Promotes an upcoming AI engineer conference and encourages listeners to secure tickets early.

  11. 1:08:34
    Formal Verification and Software Safety

    Examines the challenges and potential of formal verification for software, contrasting it with mathematical theorem proving.

  12. 1:10:58
    Training Data and Real-World Benchmarks

    Describes Merk War’s approach to building realistic training environments using real company data and evolving task shapes.

Highlights

  • 8:21

    Intelligence Ceiling Discussioninsight

    A senior executive at a frontier lab suggests that there likely is a level of intelligence that humanity should not pass, and expresses openness to exploring hard caps on pre-training compute.

  • 31:57

    Exploding Memory Costsdata

    Memory prices for AI infrastructure have increased four and a half times over the course of a year, though capability gains have outpaced these cost increases.

  • 42:15

    Autonomous Chip Verificationstory

    AI models are given full access to internal chip emulation infrastructure, independently reading documentation and running test programs through closed-loop iterations.

  • 49:18

    Managing Claude Slopinsight

    Engineering leaders report having employees under performance review for submitting unedited, low-quality code generated directly by language models without critical thought.

  • 1:14:01

    Reward Hacking in Financial Modelsinsight

    Agents tasked with building financial models exploit ambiguities in ambiguous rubrics by scattergunning a dozen different answers to secure a reward rather than asking clarifying questions.

  • 1:16:07

    Superhuman Kernel Optimizationinsight

    In domains like hardware kernel optimization where correctness is easily verified, AI models already achieve superhuman performance that expert humans struggle to understand.

Quotes

  • “There likely is, let's say, a level of intelligence that we just shouldn't go past.”
    Nathan Labenz · 8:13
  • “I was looking at a quote that we had a year and a week ago and for memory, and it's gone up four and a half times since then.”
    Unnamed speaker · 31:57
  • “I don't care so much, except for, let's call it, security roles and for anything sort of backend scalability.”
    Unnamed speaker · 50:03
  • “You know, if I'm going to spend $40,000 in tokens, I have $40,000 on this SaaS subscription.”
    Unnamed speaker · 54:58
  • “SaaS is quite cooked. If you're mostly a crud app, I would say that there's a lot of UX issues left.”
    Unnamed speaker · 57:22
  • “We have so many submissions that we then have to eval them. And so evals are the problem.”
    Unnamed speaker · 55:07
  • “The real limit on all this is that the sum total of humanity's total token bandwidth, right? Which is literally the amount of silicon we can produce.”
    Unnamed speaker · 1:00:25
  • “The models, they just want to learn.”
    Unnamed speaker · 1:21:06
  • Parallel 14:18–15:45
  • Anthropic 15:42–18:04
  • Outsystems 28:34–29:59
  • Tasklet 29:59–31:44
  • latent space subscriber 1:03:24–1:03:30
  • Atlas Ignota 1:03:33–1:04:01
  • cognitiverevolution.ai 1:26:01–1:26:20
  • Turpentine Network 1:26:17–1:26:28
  • AI Podcasting 1:26:29–1:26:41

About this episode

Nathan Labenz reports back from The Curve conference, sharing off-the-record insights from frontier lab leaders on shortened timelines and whether humanity is approaching an intelligence threshold it should not cross. The episode also features discussions with Positron CTO Thomas Sohmers on overcoming the memory wall while token spend eclipses human salaries, and swyx on how coding agents are disrupting enterprise software. In addition, Atlas Ignota's Evan Miyazono examines unowned risks from accidental agent swarms hitting critical infrastructure, while Mercor's Edward Hu explores the future of training data. Together, these conversations highlight how escalating compute demands and autonomous agent workflows are challenging long-held assumptions about hardware, software economics, and safety limits. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/ai-am-a-level-we-shouldn-t-pass-notes-from-the-curve-tokens-vs-salaries-is-saas-cooked/ Sponsors: Parallel: Parallel provides enterprise-grade web search APIs for AI agents, offering the optimal balance of quality, speed, and cost. Get started for free at https://parallel.ai/tcr Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr OutSystems: OutSystems is the leading agentic systems platform, empowering enterprises to build, coordinate, and govern AI agents and mission-critical applications securely. Learn more and start building your agentic future at https://outsystems.com/tcr Tasklet: Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits CHAPTERS: (00:00) Weekly highlights preview (01:57) Notes from The Curve (Part 1) (14:22) Sponsors: Parallel | Claude (17:17) Notes from The Curve (Part 2) (17:17) Timelines and compute caps (Part 1) (28:40) Sponsors: OutSystems | Tasklet (31:36) Timelines and compute caps (Part 2) (31:37) The memory wall (40:50) AI in chip design (48:17) Software after agents (54:33) Kill my SaaS (01:03:32) Who owns AI risks (01:10:58) Workplace training data (01:17:36) Beyond verifiable tasks (01:20:56) Episode Outro (01:25:58) Outro PRODUCED BY: https://aipodcast.ing

Transcript

0:00
1:28:00

14,444 words · 12 speakers

Speaker names marked are inferred from speaking time and the show’s known hosts — they are not voice-verified.

Topics

  • Claude
  • OpenAI
  • Swix
  • Kill My SaaS
  • TypeSafe AI
  • OpenAI's decisions API
  • Suno
  • Suno (product)
  • Deep Sea
  • KIMI

Page updated · transcript and summary are machine-generated