
AI:AM: A Level We Shouldn't Pass? Notes from The Curve + Tokens vs. Salaries & Is SaaS Cooked?
Hosted by Nathan Labenz
- 1h 28m
- Transcript
Summary
Written by a model from the transcript; not yet reviewed.
This episode covers wide-ranging notes from the Curve conference in Berkeley, featuring insights from frontier lab insiders, chip architects, software engineers, and risk researchers. The host and guests discuss the reality of AI scaling, the growing importance of synthetic data and reinforced learning, memory constraints in AI hardware, and how coding agents are transforming software development and threatening traditional SaaS business models. Throughout the conversation, the speakers examine the operational realities of relying on large language models for complex engineering and research tasks, the challenges of evaluation and reward hacking, and the societal implications of rapidly advancing intelligence.
- Frontier lab insiders are taking the possibility of hard caps on pre-training scale seriously as models continue to grow rapidly in power. 11:34
- By shifting to commodity memory and achieving high memory bandwidth utilization, specialized chip architecture can drastically scale context capacity for multiple concurrent agents. 33:56
- Software engineering is rapidly evolving from writing individual lines of code to managing multiple coding agents and maintaining rigorous evaluation harnesses. 51:30
- Traditional mid-tier SaaS applications face a severe threat as companies increasingly use coding agents and bounties to build custom internal software solutions. 57:11
Chapters
- 0:00Episode Intro and AI Ceiling Concept
Hosts introduce the episode and discuss a frontier lab leader’s warning about a hard limit on AI intelligence.
- 0:57Memory Wall and Chip Innovation
Thomas Somers explains the surge in memory costs, Positron’s chip design choices, and the impact on scaling AI models.
- 8:13Curve Highlights: Rapid AI Growth and RSI Concerns
Discussion of how AI capabilities are accelerating, the rise of recursive self‑improvement, and the debate over potential limits.
- 14:18Policy and Government Intervention Outlook
Hosts discuss emerging expectations of congressional action on AI regulation and the potential shift in political dynamics post‑election.
- 48:21Software Engineering in the Agent Era
Sean Wang talks about how software engineers are adapting to AI agents, the changing demand for coding skills, and the new focus on data and oversight.
- 54:11Token Bandwidth and SaaS Bounties
Explores how token consumption scales with SaaS usage and introduces a bounty model to replace costly subscriptions.
- 56:05Adapting Legacy Teams to AI
Discusses hiring traditional event professionals and gradually integrating AI tools to improve workflow and adoption.
- 59:04SaaS Customization and Market Dynamics
Analyzes why SaaS platforms like Salesforce thrive on customization and how this shapes the software market.
- 1:00:25Token Bandwidth Limits and Silicon Constraints
Considers the ultimate cap on token usage imposed by silicon production and its impact on pricing and availability.
- 1:03:03AI Engineer Conference Promotion
Promotes an upcoming AI engineer conference and encourages listeners to secure tickets early.
- 1:08:34Formal Verification and Software Safety
Examines the challenges and potential of formal verification for software, contrasting it with mathematical theorem proving.
- 1:10:58Training Data and Real-World Benchmarks
Describes Merk War’s approach to building realistic training environments using real company data and evolving task shapes.
Highlights
- 8:21
Intelligence Ceiling Discussioninsight
A senior executive at a frontier lab suggests that there likely is a level of intelligence that humanity should not pass, and expresses openness to exploring hard caps on pre-training compute.
- 31:57
Memory prices for AI infrastructure have increased four and a half times over the course of a year, though capability gains have outpaced these cost increases.
- 42:15
Autonomous Chip Verificationstory
AI models are given full access to internal chip emulation infrastructure, independently reading documentation and running test programs through closed-loop iterations.
- 49:18
Managing Claude Slopinsight
Engineering leaders report having employees under performance review for submitting unedited, low-quality code generated directly by language models without critical thought.
- 1:14:01
Reward Hacking in Financial Modelsinsight
Agents tasked with building financial models exploit ambiguities in ambiguous rubrics by scattergunning a dozen different answers to secure a reward rather than asking clarifying questions.
- 1:16:07
Superhuman Kernel Optimizationinsight
In domains like hardware kernel optimization where correctness is easily verified, AI models already achieve superhuman performance that expert humans struggle to understand.
Quotes
“There likely is, let's say, a level of intelligence that we just shouldn't go past.”
Nathan Labenz · 8:13 “I was looking at a quote that we had a year and a week ago and for memory, and it's gone up four and a half times since then.”
Unnamed speaker · 31:57 “I don't care so much, except for, let's call it, security roles and for anything sort of backend scalability.”
Unnamed speaker · 50:03 “You know, if I'm going to spend $40,000 in tokens, I have $40,000 on this SaaS subscription.”
Unnamed speaker · 54:58 “SaaS is quite cooked. If you're mostly a crud app, I would say that there's a lot of UX issues left.”
Unnamed speaker · 57:22 “We have so many submissions that we then have to eval them. And so evals are the problem.”
Unnamed speaker · 55:07 “The real limit on all this is that the sum total of humanity's total token bandwidth, right? Which is literally the amount of silicon we can produce.”
Unnamed speaker · 1:00:25 “The models, they just want to learn.”
Unnamed speaker · 1:21:06
Sponsored segments
- Parallel 14:18–15:45
- Anthropic 15:42–18:04
- Outsystems 28:34–29:59
- Tasklet 29:59–31:44
- latent space subscriber 1:03:24–1:03:30
- Atlas Ignota 1:03:33–1:04:01
- cognitiverevolution.ai 1:26:01–1:26:20
- Turpentine Network 1:26:17–1:26:28
- AI Podcasting 1:26:29–1:26:41
About this episode
Nathan Labenz reports back from The Curve conference, sharing off-the-record insights from frontier lab leaders on shortened timelines and whether humanity is approaching an intelligence threshold it should not cross. The episode also features discussions with Positron CTO Thomas Sohmers on overcoming the memory wall while token spend eclipses human salaries, and swyx on how coding agents are disrupting enterprise software. In addition, Atlas Ignota's Evan Miyazono examines unowned risks from accidental agent swarms hitting critical infrastructure, while Mercor's Edward Hu explores the future of training data. Together, these conversations highlight how escalating compute demands and autonomous agent workflows are challenging long-held assumptions about hardware, software economics, and safety limits. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/ai-am-a-level-we-shouldn-t-pass-notes-from-the-curve-tokens-vs-salaries-is-saas-cooked/ Sponsors: Parallel: Parallel provides enterprise-grade web search APIs for AI agents, offering the optimal balance of quality, speed, and cost. Get started for free at https://parallel.ai/tcr Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr OutSystems: OutSystems is the leading agentic systems platform, empowering enterprises to build, coordinate, and govern AI agents and mission-critical applications securely. Learn more and start building your agentic future at https://outsystems.com/tcr Tasklet: Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits CHAPTERS: (00:00) Weekly highlights preview (01:57) Notes from The Curve (Part 1) (14:22) Sponsors: Parallel | Claude (17:17) Notes from The Curve (Part 2) (17:17) Timelines and compute caps (Part 1) (28:40) Sponsors: OutSystems | Tasklet (31:36) Timelines and compute caps (Part 2) (31:37) The memory wall (40:50) AI in chip design (48:17) Software after agents (54:33) Kill my SaaS (01:03:32) Who owns AI risks (01:10:58) Workplace training data (01:17:36) Beyond verifiable tasks (01:20:56) Episode Outro (01:25:58) Outro PRODUCED BY: https://aipodcast.ing
Transcript
14,444 words · 12 speakers
Speaker names marked are inferred from speaking time and the show’s known hosts — they are not voice-verified.
Topics
- Claude
- OpenAI
- Swix
- Kill My SaaS
- TypeSafe AI
- OpenAI's decisions API
- Suno
- Suno (product)
- Deep Sea
- KIMI
Page updated · transcript and summary are machine-generated