Skip to content

Software Verification in the Age of Coding Agents

Software Engineering Daily

47m
Transcript

Original podcast

Where to listen

Summary

Written by a model from the transcript; not yet reviewed.

This episode explores how deterministic simulation testing and software verification are evolving to meet the challenges of distributed systems and AI-generated code. The guest and the host discuss why traditional testing often fails to catch unknown edge cases and how platforms like Antithesis use controlled environments, injected faults, and coverage-guided fuzzing to explore thousands of code execution paths at runtime. They also examine how these testing strategies can provide safety nets for autonomous coding agents as the industry shifts toward higher precision specifications.

  • Engineers should write procedural tests that parameterize input spaces to turn testing into an engineering puzzle. 7:14
  • Developers can use worse mock services with a wider range of failure behaviors to test complex systems. 14:44
  • Deterministic simulation testing enables breadth-first search of a software's state space to catch concurrency and state bugs. 22:02
  • Teams should run short 15-minute simulation tests on pull requests and longer stress tests overnight to build confidence incrementally. 40:42

Chapters

  1. 0:00
    Introduction to Antithesis and Episode Overview

    Sean Falconer introduces Carl Svera and outlines the episode’s focus on software verification, reliability tooling, and deterministic simulation testing.

  2. 1:35
    Carl’s Day‑to‑Day Role on the Red Team

    Carl explains his responsibilities as a Red Team member at Antithesis, bridging customer support and product development to find hard bugs in distributed systems.

  3. 4:00
    Deterministic Simulation Testing (DST) Fundamentals

    Carl breaks down DST into determinism and simulation, explaining how Antithesis makes software deterministic and explores state space by branching timelines.

  4. 6:00
    DST vs. Fuzzing and Coverage‑Guided Techniques

    The conversation contrasts DST with fuzzing, highlighting how Antithesis monitors coverage and autonomously focuses on new behaviors to find concurrency bugs.

  5. 8:00
    Real‑World Example: SQLite Wall‑Reset Bug

    Carl recounts how Antithesis quickly discovered a long‑standing SQLite concurrency bug using a generic workload and property‑based testing.

  6. 10:00
    Observability and Debugging Features in Antithesis

    Carl describes the platform’s high‑resolution logs, multiverse debugging, and causality analysis that help users pinpoint and fix bugs discovered by DST.

  7. 12:00
    Hydro and Compile‑Time Formal Verification

    Sean and Carl compare Antithesis’s runtime testing with Hydro’s compile‑time type‑system checks, noting complementary strengths and trade‑offs.

  8. 14:00
    Integrating DST into a Testing Stack

    Carl outlines how DST fits with unit tests, integration tests, and stress testing, emphasizing its role in covering unknown‑unknown state space.

  9. 16:00
    Future of Testing with AI‑Generated Code

    The hosts discuss how AI’s rapid code generation shifts the balance toward verification, requiring precise specifications and automated testing loops.

  10. 44:34
    Encoding Workloads into Antithesis Net

    The speaker explains how to encode workloads into Antithesis to ensure all simulation paths satisfy valid properties, preventing counterexamples.

  11. 45:06
    AI Coding as a Diffusion Process

    The speaker speculates that future AI-generated code could evolve like image diffusion, continuously re‑diffusing entire codebases based on updated specifications.

  12. 46:05
    Antithesis as Verification Solution

    Antithesis is presented as a promising tool to handle verification challenges posed by massive AI‑generated code volumes.

Highlights

  • 8:44

    Testing Unknown Unknownsinsight

    The guest explains that testing usually focuses on regression for known behaviors, while the hardest part of software engineering is testing for unknown unknowns across staple systems.

  • 20:56

    Deterministic Simulation Testingstrategy

    The guest breaks down the two core ingredients of deterministic simulation testing: making the software environment deterministic and simulating branched timelines to test different event orders.

  • 25:11

    SQLite Concurrency Bugstory

    The guest shares a story about using Antithesis to find a ten-year-old concurrency bug in SQLite within just 15 minutes by running inserts and write-ahead log checkpoints in a loop.

  • 30:11

    Multiverse Debuggingstrategy

    The guest explains how platform features like multiverse debugging and causality analysis allow developers to SSH into simulation points and binary search for when a bug occurred.

  • 38:51

    Antithesis-Driven Developmentinsight

    The guest argues that instead of test-driven development, AI coding should use antithesis-driven development where agents write code against broad safety properties that cannot be easily overfitted.

Quotes

  • “The more we made testing something that you can do procedurally, so like, the more testing could feel like procedural generation, the more likely engineers would like to use it.”
    Unnamed speaker · 7:07
  • “We have a bunch of other inputs that we monitor about the system that helps us understand whether or not we're seeing sort of new behaviors or pre-existing behaviors.”
    Unnamed speaker · 24:21
  • “In 15 minutes, we run at a default parallelism of 48.”
    Unnamed speaker · 40:47
  • “I think that as we sort of see the like time it takes code, effectively as the coding time decreases, we basically should be rebalancing that effort into verification and specification.”
    Unnamed speaker · 43:41
  • “and you want to build really good specification. In some ways, like I've been thinking about the reality of coding”
    Unnamed speaker · 45:12
  • “that we could actually completely change software engineering from an iterative process to almost like a diffusion process where you change the specification and you re-diffuse the entire codebase”
    Unnamed speaker · 45:34
  • “And as long as it matches the specification and the, you know, passes this sort of huge, huge net of potential state space, you basically just release that completely net new codebase as the new version of code”
    Unnamed speaker · 45:47
  • “I don't think we're at steady state now. I think things are going to continue to massively change over the next couple of years.”
    Unnamed speaker · 46:22

About this episode

Software testing is widely understood as essential to building reliable systems, and it’s one of the hardest things to get right. A fundamental issue is that tests are good at confirming the behavior a team expects, but the most damaging failures tend to hide in the edge cases no one thought to check during development. Those blind spots are only growing as AI produces more code that no human has closely reviewed. Antithesis https://antithesis.com/ is a software verification company built around an approach called deterministic simulation testing. The platform runs a customer’s software in a fully controlled environment, exploring thousands of different paths through it at runtime. Crucially, the paths aren’t pre-specified and they’re not merely the expected cases. In this way, it can surface rare, damaging bugs that normal testing would never catch. Carl Sverre https://x.com/carlsverre is on the Red Team at Antithesis and he also leads much of the company’s AI strategy. In this episode, Carl joins Sean Falconer to discuss the Antithesis approach to software verification, why reliability tooling has been chronically underinvested, what makes stateful and distributed systems so hard to test, how deterministic simulation testing works under the hood, and much more. Full Disclosure: This episode is sponsored by Antithesis Sponsorship inquiries: sponsor@softwareengineeringdaily.com mailto:sponsor@softwareengineeringdaily.com The post Software Verification in the Age of Coding Agents https://softwareengineeringdaily.com/podcasts/software-verification-in-the-age-of-coding-agents/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.

Transcript

0:00
47:00

9,450 words · 3 speakers

Topics

  • Antithesis
  • SQLite
  • Hydro
  • Fuzzing
  • Red Team
  • SingleStore
  • Netflix
  • Google
  • Chaos Engineering
  • diffusion

Page updated · transcript and summary are machine-generated