
⚡️ Which GPU Clouds Are Actually Good? | ClusterMAX 3.0
Latent Space: The AI Engineer Podcast
- 57m
- Transcript
Summary
Written by a model from the transcript; not yet reviewed.
This episode covers the release of ClusterMax 3.0 by the SemiAnalysis team, examining how emerging NeoClouds stack up across performance, reliability, and security. The discussion details severe cybersecurity vulnerabilities found in many cloud providers, the intense competition for GPU compute driven by profitable inference startups and frontier labs, and major industry shifts including Nvidia acquisitions and Huawei's hardware scaling in China. The speakers explore the realities of renting compute in a supply-constrained market and explain why managed cluster rankings differ from bare-metal infrastructure or stock market fundamentals.
- Install the free ClusterMax CLI tool to check software versions and identify well-documented CVEs on your GPU clusters. 8:56
- Acknowledge that ClusterMax evaluates managed service layers like Slurm and Kubernetes rather than bare-metal data center construction. 9:39
- Expect fierce competition for compute as inference providers achieve high gross margins and rent smaller cluster sizes down to 1,000 GPUs. 12:28
- Understand that ClusterMax rankings are strictly buy-side tools for renting compute and do not reflect stock market valuations. 43:09
Chapters
- 0:04ClusterMax 1 to 3 Evolution
Jordan discusses the progression from ClusterMax 1 to 3, highlighting expanded provider coverage, automation, and new benchmark repositories.
- 3:25Defining NeoClouds and Criteria
Explanation of what NeoClouds are, the 10 evaluation criteria, and the focus on managed clusters versus bare metal.
- 6:10Security Challenges in NeoClouds
Detailed discussion of recurring security issues across generations, the impact on rankings, and the importance of software and network configuration checks.
- 12:08Strategic Partnerships and Acquisitions
Analysis of SpaceX–Google deals, Nvidia’s acquisitions of Poolside and Hugging Face, and their implications for NeoCloud competition.
- 13:43Market Dynamics and Compute Scarcity
Exploration of how compute demand has shifted, the rise of inference providers, and the decreasing cluster sizes available to NeoLabs.
- 18:00Future Directions: Inference and Bare Metal
Plans to expand testing to inference endpoints, bare metal clusters, and broader infrastructure segments beyond managed clusters.
- 39:35Criticisms and Transparency of Rankings
Addressing concerns about bias, paid services, and the distinction between buy‑side research and sell‑side recommendations.
- 48:41Feedback on Self‑Confidence
The hosts give Dylan constructive feedback about his self‑confidence and discuss how to improve it.
- 49:24Multi‑Silicon Trends for Next Year
Exploration of the upcoming multi‑silicon landscape, including Nvidia’s Vera Rubin generation and other chip startups entering the market.
- 50:34Inference vs. Training Silicon Use
Clarification of how different silicon types are used for inference and training, with examples from Anthropic and the role of TPUs and GPUs.
- 51:28Challenges of Heterogeneous Silicon
Discussion of the benefits and numerical challenges when mixing different chips or generations during training and inference.
- 52:19Disaggregated RL and Data‑Center Architecture
Exploration of disaggregated, decentralized RL architectures and the need for sharding models across multiple sites.
Highlights
- 3:53
Defining NeoClouds and AI Focusinsight
NeoClouds are distinguished from legacy hyperscalers by their specific focus on deploying GPUs and accelerators for AI training, inference, and reinforcement learning workloads.
- 7:48
Critical NeoCloud Cybersecuritydata
Security audits revealed shocking vulnerabilities, including exposure of national intelligence data due to outdated software versions and improper network key configurations.
- 12:49
Inference Profitability Squeezing Computeinsight
While early labs were unprofitable, modern inference providers running at high gross margins have intensified the competition for scarce GPU access.
- 15:07
OpenAI and Anthropic Dominanceinsight
OpenAI and Anthropic have surpassed Google's DeepMind in dedicated research and development compute by leveraging large-scale bare-metal hyperscalers and self-built infrastructure.
- 20:04
Mutual 90-Day Cancellation Rightsstrategy
Contract structures like SpaceX's deal with Google feature mutual 90-day cancellation clauses, allowing companies to reclaim compute for research runs when needed.
- 31:47
Defensive Nvidia Acquisitionsinsight
Nvidia's high-value acquisitions of labs like Poolside are defensive moves to retain top engineering talent and prevent competitors from owning key training infrastructure.
- 36:20
Distinguishing Buy-Side and Sell-Sidedebate
The speakers clarify that poor ClusterMax managed-service rankings can coexist with strong data center construction capabilities or separate financial research.
Quotes
“With ClusterMax 1, you know, all the testing was hands‑on. This is pre‑CLOUD code, pre‑Codecs, beginning serious.”
Unnamed speaker · 2:43 “We do not take money for any ranking. That would be illegal.”
Unnamed speaker · 37:30 “If you’re going to train a frontier model, you need to have health checks on your cluster. You need to have high reliability.”
Unnamed speaker · 34:52 “We have a dashboard, which we took a redacted screenshot of below where you can get all of our data.”
Unnamed speaker · 40:49 “There's a whole bunch of benefits of using, let's say, cheaper chips capable of high throughput for the forward pass in RL.”
Unnamed speaker · 51:28 “I absolutely expect there to be heterogeneous silicon used across training and inference.”
Unnamed speaker · 51:47 “People are doing RL across many, many sites, right? This data center is doing token generation, that data center is doing token generation.”
Unnamed speaker · 52:21 “Huawei is definitely the best of them all, and then Baidu is spinning out their chip business as a completely separate company.”
Unnamed speaker · 54:50 “They will get to even tens of millions of chips produced by 2028.”
Unnamed speaker · 56:23
About this episode
Watch the full show on YouTube: https://www.youtube.com/watch?v=MWX36ZYnsm0
Transcript
10,733 words · 4 speakers
Topics
- Nvidia
- Anthropic
- OpenAI
- Huawei
- FluidStack
- Tranium
- AWS
- AMD
- Vera Rubin
- Cerebras
Page updated · transcript and summary are machine-generated