Skip to content

Chroma and Agentic Retrieval

Software Engineering Daily

52m
Transcript

Original podcast

Where to listen

Summary

Written by a model from the transcript; not yet reviewed.

In this episode, the guest from Chroma and the host discuss the evolution of data retrieval infrastructure for AI applications, the mechanics of context rot, and the release of Context One, a retrieval subagent. The guest explains how Chroma moved from active learning for computer vision to solving the inference-time retrieval and search needs of language models. They examine why traditional single-index search models fail under agentic workloads, how object storage enables database viability, and why partitioning indices per customer or agent is necessary. The conversation also covers the engineering behind Context One, a 20 billion parameter model trained to handle agentic search at high speed and low cost through self-editing context and parallel tool calls.

  • Install ChromaDB locally by running a simple package installation command to start prototyping. 51:55
  • Developers should consider the heavy operational and architectural tax of building a Bring Your Own Cloud deployment model. 41:31

Chapters

  1. 0:00
    Podcast Host Search and Intro

    The episode opens with a call for a new host and an overview of the show’s mission.

  2. 0:33
    Data Retrieval Challenges in AI

    The host introduces the fundamental problem of data retrieval for AI systems and the rise of agentic approaches.

  3. 6:35
    Developer Adoption Path and BYOC

    The discussion covers how developers start with the local DB, move to the cloud, and the considerations around Bring Your Own Cloud deployments.

  4. 12:59
    Context Rot Findings and Context Engineering

    Hammad discusses how model performance degrades with larger context windows and the need for disciplined context selection.

  5. 22:07
    Architecture of Context One Retrieval Subagent

    The host and guest break down the subagent’s toolset, parallelism, and self‑editing context strategy.

  6. 27:24
    Training and Optimizing the Retrieval Model

    Details of reinforcement learning, supervised fine‑tuning, and pruning accuracy improvements are shared.

  7. 33:27
    Open Source Philosophy and Model Release

    Hammad explains why Chroma releases models and data under open source licenses and the benefits of community involvement.

  8. 40:30
    BYOC Deployment Challenges and Guidance

    Hammad shares practical advice on implementing BYOC, addressing security, compliance, and operational concerns.

  9. 44:57
    Intro and Data Privacy Strategy

    The hosts introduce the topic of data privacy and how Chroma’s architecture ensures no open ports on the data plane, using reverse tunnels and push-based models.

  10. 45:31
    Reverse Tunnel Architecture Explained

    The speakers detail how data flows from the data plane to Chroma through reverse tunnels, avoiding direct access and enhancing security.

  11. 46:41
    Kubernetes and BYOC Challenges

    Discussion on the complexities of building on Kubernetes and the difficulties of supporting Bring Your Own Cloud deployments.

  12. 47:36
    Future of Context Management in LLMs

    Exploration of the ongoing need for context management in language models and the vision of integrating intelligence directly into databases.

Highlights

  • 2:58

    Shift From Computer Vision To AI Searchstory

    The guest explains how the company originally started solving active learning for computer vision systems, and how that curation problem translated to inference-time data retrieval for language models.

  • 5:00

    Three Pillars of Chroma Platformstrategy

    The guest outlines the platform's technology stack, including Chroma DB, the SINK managed ETL pipeline, and the Context One model for agentic search.

  • 8:01

    Partitioned Indices Versus Monolithic Indexesinsight

    The guest explains why AI applications require hard-partitioned indices per customer, team, or agent rather than one giant monolithic index, due to cost and accuracy trade-offs.

  • 12:59

    Empirical Finding Of Context Rotdata

    The guest details the Context Rot paper findings, showing a sharp drop-off in model performance as context window utilization increases into the tens of thousands of tokens.

  • 15:48

    Building Databases On Object Storagestrategy

    By leveraging conditional writes introduced on AWS S3 in 2024, Chroma was able to build a database solely on top of object storage to make AI application economics viable.

  • 15:55

    Testing Across 18 Different Modelsdata

    The Context Rot benchmark was tested across 18 models, revealing performance gaps between open-weight models and closed frontier labs that have steadily closed over time.

  • 26:18

    Self-Editing Context With Prune Chunksinsight

    Context One uses a prune chunks tool to allow the model to remove documents from its own context window, reducing token usage and keeping inference fast.

  • 29:41

    Small Models Matching Frontier Qualityinsight

    The guest explains how a purpose-trained 20 billion parameter model can match or beat frontier models on search tasks because search requires specific narrow behaviors rather than raw reasoning.

Quotes

  • “Vector search was an early answer to the retrieval problem, but the rise of agentic systems has raised the stakes considerably.”
    Unnamed speaker · 0:40
  • “I think that for us has been how we've alleviated people's concerns about what we're capturing and storing.”
    Unnamed speaker · 45:21
  • “I think there will always be a need for managing the context of language models.”
    Unnamed speaker · 47:36
  • “I think that system will look like something much different.”
    Unnamed speaker · 51:26
  • Fingerprint 19:44–20:21
  • GuardSquare 20:33–21:12
  • Deepgram Flux TTS 21:14–21:44
  • Chroma 51:55–52:35

About this episode

Data retrieval is a fundamental challenge in AI systems, and the approaches for solving it are still evolving. Vector search was an early answer to the retrieval problem, but the rise of agentic systems has raised the stakes considerably. Agents issue queries at machine speed, decompose complex questions into parallel searches, and require retrieval infrastructure that can keep pace without becoming prohibitively expensive. Chroma https://www.trychroma.com/ is a company building open source infrastructure for AI applications, best known for its widely used database of the same name. The company also published the influential Context Rot paper, which documented how model performance degrades as context window utilization increases, and recently released Context One, a 20 billion parameter retrieval sub-agent trained to do agentic search at frontier model quality but at an order of magnitude lower cost and higher speed. Hammad Bashir https://x.com/HammadTime is the CTO of Chroma, with a background spanning machine learning, computer vision, and data systems. In this episode, Hammad joins Gregor Vand to discuss the origins of ChromaDB, our current understanding of context rot, why a purpose-built small model can match frontier models on search tasks, the philosophy behind Chroma’s open source approach, and where the company sees AI data infrastructure heading. Sponsorship inquiries: sponsor@softwareengineeringdaily.com mailto:sponsor@softwareengineeringdaily.com The post Chroma and Agentic Retrieval https://softwareengineeringdaily.com/podcasts/chroma-and-agentic-retrieval/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.

Transcript

0:00
52:20

10,715 words · 6 speakers

Topics

  • database
  • language model
  • index
  • data plane
  • object storage
  • CPU
  • Postgres
  • Kubernetes
  • GPU
  • search system

Page updated · transcript and summary are machine-generated