Skip to content

⚡️ Which GPU Clouds Are Actually Good? | ClusterMAX 3.0

Latent Space: The AI Engineer Podcast

57m
Transcript

Original podcast

Where to listen

Summary

Written by a model from the transcript; not yet reviewed.

This episode covers the release of ClusterMax 3.0 by the SemiAnalysis team, examining how emerging NeoClouds stack up across performance, reliability, and security. The discussion details severe cybersecurity vulnerabilities found in many cloud providers, the intense competition for GPU compute driven by profitable inference startups and frontier labs, and major industry shifts including Nvidia acquisitions and Huawei's hardware scaling in China. The speakers explore the realities of renting compute in a supply-constrained market and explain why managed cluster rankings differ from bare-metal infrastructure or stock market fundamentals.

  • Install the free ClusterMax CLI tool to check software versions and identify well-documented CVEs on your GPU clusters. 8:56
  • Acknowledge that ClusterMax evaluates managed service layers like Slurm and Kubernetes rather than bare-metal data center construction. 9:39
  • Expect fierce competition for compute as inference providers achieve high gross margins and rent smaller cluster sizes down to 1,000 GPUs. 12:28
  • Understand that ClusterMax rankings are strictly buy-side tools for renting compute and do not reflect stock market valuations. 43:09

Chapters

  1. 0:04
    ClusterMax 1 to 3 Evolution

    Jordan discusses the progression from ClusterMax 1 to 3, highlighting expanded provider coverage, automation, and new benchmark repositories.

  2. 3:25
    Defining NeoClouds and Criteria

    Explanation of what NeoClouds are, the 10 evaluation criteria, and the focus on managed clusters versus bare metal.

  3. 6:10
    Security Challenges in NeoClouds

    Detailed discussion of recurring security issues across generations, the impact on rankings, and the importance of software and network configuration checks.

  4. 12:08
    Strategic Partnerships and Acquisitions

    Analysis of SpaceX–Google deals, Nvidia’s acquisitions of Poolside and Hugging Face, and their implications for NeoCloud competition.

  5. 13:43
    Market Dynamics and Compute Scarcity

    Exploration of how compute demand has shifted, the rise of inference providers, and the decreasing cluster sizes available to NeoLabs.

  6. 18:00
    Future Directions: Inference and Bare Metal

    Plans to expand testing to inference endpoints, bare metal clusters, and broader infrastructure segments beyond managed clusters.

  7. 39:35
    Criticisms and Transparency of Rankings

    Addressing concerns about bias, paid services, and the distinction between buy‑side research and sell‑side recommendations.

  8. 48:41
    Feedback on Self‑Confidence

    The hosts give Dylan constructive feedback about his self‑confidence and discuss how to improve it.

  9. 49:24
    Multi‑Silicon Trends for Next Year

    Exploration of the upcoming multi‑silicon landscape, including Nvidia’s Vera Rubin generation and other chip startups entering the market.

  10. 50:34
    Inference vs. Training Silicon Use

    Clarification of how different silicon types are used for inference and training, with examples from Anthropic and the role of TPUs and GPUs.

  11. 51:28
    Challenges of Heterogeneous Silicon

    Discussion of the benefits and numerical challenges when mixing different chips or generations during training and inference.

  12. 52:19
    Disaggregated RL and Data‑Center Architecture

    Exploration of disaggregated, decentralized RL architectures and the need for sharding models across multiple sites.

Highlights

  • 3:53

    Defining NeoClouds and AI Focusinsight

    NeoClouds are distinguished from legacy hyperscalers by their specific focus on deploying GPUs and accelerators for AI training, inference, and reinforcement learning workloads.

  • 7:48

    Critical NeoCloud Cybersecuritydata

    Security audits revealed shocking vulnerabilities, including exposure of national intelligence data due to outdated software versions and improper network key configurations.

  • 12:49

    Inference Profitability Squeezing Computeinsight

    While early labs were unprofitable, modern inference providers running at high gross margins have intensified the competition for scarce GPU access.

  • 15:07

    OpenAI and Anthropic Dominanceinsight

    OpenAI and Anthropic have surpassed Google's DeepMind in dedicated research and development compute by leveraging large-scale bare-metal hyperscalers and self-built infrastructure.

  • 20:04

    Mutual 90-Day Cancellation Rightsstrategy

    Contract structures like SpaceX's deal with Google feature mutual 90-day cancellation clauses, allowing companies to reclaim compute for research runs when needed.

  • 31:47

    Defensive Nvidia Acquisitionsinsight

    Nvidia's high-value acquisitions of labs like Poolside are defensive moves to retain top engineering talent and prevent competitors from owning key training infrastructure.

  • 36:20

    Distinguishing Buy-Side and Sell-Sidedebate

    The speakers clarify that poor ClusterMax managed-service rankings can coexist with strong data center construction capabilities or separate financial research.

Quotes

  • “With ClusterMax 1, you know, all the testing was hands‑on. This is pre‑CLOUD code, pre‑Codecs, beginning serious.”
    Unnamed speaker · 2:43
  • “We do not take money for any ranking. That would be illegal.”
    Unnamed speaker · 37:30
  • “If you’re going to train a frontier model, you need to have health checks on your cluster. You need to have high reliability.”
    Unnamed speaker · 34:52
  • “We have a dashboard, which we took a redacted screenshot of below where you can get all of our data.”
    Unnamed speaker · 40:49
  • “There's a whole bunch of benefits of using, let's say, cheaper chips capable of high throughput for the forward pass in RL.”
    Unnamed speaker · 51:28
  • “I absolutely expect there to be heterogeneous silicon used across training and inference.”
    Unnamed speaker · 51:47
  • “People are doing RL across many, many sites, right? This data center is doing token generation, that data center is doing token generation.”
    Unnamed speaker · 52:21
  • “Huawei is definitely the best of them all, and then Baidu is spinning out their chip business as a completely separate company.”
    Unnamed speaker · 54:50
  • “They will get to even tens of millions of chips produced by 2028.”
    Unnamed speaker · 56:23

About this episode

Watch the full show on YouTube: https://www.youtube.com/watch?v=MWX36ZYnsm0

Transcript

0:00
57:51

10,733 words · 4 speakers

Topics

  • Nvidia
  • Anthropic
  • OpenAI
  • Huawei
  • FluidStack
  • Tranium
  • AWS
  • AMD
  • Vera Rubin
  • Cerebras

Page updated · transcript and summary are machine-generated