Skip to content

Can Cloudflare save the web from AI?

Decoder

1h 18m
Transcript

Original podcast

Where to listen

Summary

Written by a model from the transcript; not yet reviewed.

This episode explores the profound disruptions caused by artificial intelligence to the business model of the internet and corporate structures, featuring Matthew Prince, CEO of Cloudflare. The guest and host discuss how automated bot traffic has recently surpassed human traffic online, threatening traditional advertising-based models and forcing a reexamination of how websites, content creators, and infrastructure are funded. They examine Cloudflare's initiatives to implement payment protocols and block unauthorized scrapers to protect web publishers, while also debating the economic and copyright implications of these shifts. Additionally, the conversation tackles the controversial corporate restructuring at Cloudflare, where thousands of employees were laid off and management evaluated workers using AI-driven performance tracking across builder, seller, and measurer categories.

  • Automated bot traffic has officially surpassed human traffic online, fundamentally disrupting the traditional advertising business model of the internet. 6:45
  • Cloudflare set defaults to block Google's AI training crawlers for free customers to encourage fair compensation for web publishers. 31:57
  • Unique local content and specialized information command significantly higher licensing value from AI labs than generic high-volume news. 24:43
  • Artificial intelligence tools can be utilized to dramatically increase organizational efficiency in measurement and auditing functions. 1:01:35

Chapters

  1. 0:00
    Show Intro and Host Overview

    The episode opens with the host introducing the show, sponsors, and the guest, Matthew Prince, CEO of Cloudflare.

  2. 0:45
    Defining Cloudflare’s Role

    Matthew Prince explains Cloudflare’s core services—speed, security, and infrastructure for the internet—while hinting at its evolving role with AI.

  3. 1:17
    Bots, AI Traffic, and Control

    Cloudflare’s discovery that bots now dominate internet traffic and the company’s tools to allow site owners to block or monetize AI agents are discussed.

  4. 2:07
    AI’s Impact on Business Models

    The conversation shifts to how AI-driven bot traffic challenges traditional advertising revenue models and forces a rethink of internet economics.

  5. 6:00
    Music Industry as a Case Study

    The hosts compare the music industry's shift to streaming and paid models with the proposed AI content economy, debating who benefits.

  6. 10:31
    Micropayments and the 402 Protocol

    Matthew outlines a potential future where content is paid for via microtransactions, reviving the 402 HTTP protocol to monetize AI access to information.

  7. 28:04
    Legal and Regulatory Landscape

    Discussion turns to copyright law, AI training, and the evolving legal frameworks that could shape how AI companies access web content.

  8. 35:12
    Incentives for Content Creation

    The conversation explores how new payment models could incentivize creators, local news, and academic content to thrive in an AI-dominated ecosystem.

  9. 35:43
    Google’s Role and Future Changes

    Matthew details Google’s upcoming transparency measures for its crawler and the broader implications for AI training and web access.

  10. 38:27
    Distribution, Media, and the Future

    The hosts reflect on how distribution shapes content, the challenges of the current media landscape, and potential shifts toward more meaningful information sharing.

  11. 40:51
    Closing Thoughts on AI and Society

    The episode concludes with a broader discussion on AI’s societal impact, the role of media, and the need for new models of knowledge sharing.

  12. 41:59
    Cloudflare’s Token Economy and Media Deals

    Explores how Cloudflare measures content value through tokens and the unequal revenue distribution between media giants and platforms like Reddit.

Highlights

  • 6:45

    Automated Traffic Surpasses Humansdata

    The guest reveals that automated bot traffic has surpassed human traffic online, growing much faster than previously predicted and creating immense demands on infrastructure.

  • 9:48

    Reviving the 402 Protocolstrategy

    The guest explains how Cloudflare is working with companies like Coinbase and Stripe to bring back the 402 payment required protocol, enabling micropayments for bots accessing content.

  • 24:43

    Local Media Economic Valueinsight

    The guest notes that local media outlets are realizing higher revenue from AI licensing deals than digital advertising because AI companies need unique local knowledge to train accurate models.

  • 31:57

    Blocking Google AI Crawlersstrategy

    The guest announces that Cloudflare set defaults across free customers to block Google's AI crawlers unless they differentiate from standard search crawling.

  • 42:35

    Reddit Versus News Valuationdata

    The guest compares licensing valuations between the New York Times and Reddit, arguing that unique user-generated platforms command higher payouts than traditional outlets.

  • 1:01:35

    AI in Internal Measurementstory

    The guest discusses how Cloudflare utilized AI tools to streamline financial reporting and internal audit measurement processes, reducing preparation times from two weeks to minutes.

  • 1:03:50

    Flattening Management Hierarchiesstrategy

    The guest explains that better tooling has allowed managers to handle more direct reports, flattening organizational hierarchy and reducing the need for middle management.

Quotes

  • “Cloudflare found in June that bots now make up more than half of the internet traffic, a number that just keeps going up as more and more AI companies scrape more and more of the web.”
    Unnamed speaker · 1:17
  • “Bots don't click on ads. They don't respond to pretty swirls of paint and what we traditionally think of as a brand.”
    Unnamed speaker · 7:31
  • “Information wants to be free. There's a second half of that quote everyone forgets, which is that information also wants to be expensive because it's hard to generate.”
    Unnamed speaker · 17:00
  • “The 402 protocol will come back to life, and we're going to do, I don't know, crypto micropayments using Stripe powered by Cloudflare.”
    Unnamed speaker · 11:00
  • “AI is only as good as the hands that shape it.”
    Unnamed speaker · 54:50
  • “We have a free version of the service. It takes 30 seconds to leave.”
    Unnamed speaker · 50:31
  • “We took what used to take two weeks and we reduced it down to three minutes.”
    Unnamed speaker · 1:02:47
  • “If you think about musicians, there are two reasons that people become musicians, to get rich and to get famous.”
    Unnamed speaker · 45:07
  • EY 0:00–0:38
  • EY 26:47–26:49
  • EY 53:42–53:47
  • EY 1:19:02–1:19:34

About this episode

Cloudflare discovered in June that bots already make up more than half of internet traffic — and CEO Matthew Prince says that bots could outnumber humans a thousand to one within five years. That's a huge change to the internet, and if the web is going to keep existing, it needs a huge change to its business model. Can Cloudflare use its place in the ecosystem to direct the money back to the people who make content? Read the full interview transcript on The Verge. Links: How I choose which Cloudflare employees to replace with AI | Wall Street Journal Why Cloudflare CEO Matthew Prince is the internet’s unlikely defender | Decoder (2024) Cloudflare will now block AI crawlers by default | The Verge Patreon partners with Cloudflare to block AI crawlers | 404 Media Sony and Warner sue Anthropic in latest AI music lawsuit | Billboard Trump administration supports OpenAI in NYT copyright lawsuit | The Verge Google Zero is here — now what? | The Verge (2024) Subscribe to The Verge to access the ad-free version of Decoder! Credits: Decoder is a production of The Verge and part of the Vox Media Podcast Network. Decoder’s producers are Greg Ott, Kate Cox, and Nick Statt. This episode was edited by Kabir Chopra. Our editorial director is Kevin McShane. The Decoder music is by Breakmaster Cylinder. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Transcript

0:00
1:18:12

17,208 words · 4 speakers

Topics

  • Cloudflare
  • Google
  • Spotify
  • Anthropic
  • Reddit
  • New York Times
  • Wall Street Journal
  • The Verge
  • OpenAI
  • Daily Stormer

Page updated · transcript and summary are machine-generated