Skip to content

The Gap Between AI Spending and AI Value

Software Engineering Daily

54m
Transcript

Original podcast

Where to listen

Summary

Written by a model from the transcript; not yet reviewed.

This episode explores the widespread gap between enterprise spending on artificial intelligence and the durable value actually captured by organizations. The guest and host discuss why mainstream AI tools often fail to transform businesses beyond individual coding assistants and chatbots, breaking down the problem across three distinct layers: the underlying AI models, the complex enterprise systems and fragmented data foundations, and the human change management needed for successful adoption. They also examine what sets apart the small minority of enterprises successfully deploying AI at scale from those that stall out at the pilot stage.

  • Successful enterprise AI adoption requires aligning leadership on clear business metrics rather than treating AI as a magic technology. 18:59
  • Organizations must fix their underlying data fragmentation and governance before expecting agents to deliver reliable operational results. 26:18
  • The most effective enterprises combine internal domain expertise with external specialists in AI models and application development. 31:35
  • Enterprise deployment of AI demands rigorous evaluation frameworks that cover happy paths, edge cases, and robustness across environments. 35:31

Chapters

  1. 0:00
    Episode Introduction and Host Overview

    The episode opens with the host introducing the topic of AI spending versus value and the guest, Emily Hsu, head of enterprise AI at Scale AI.

  2. 0:54
    Emily Hsu’s Background and Scale AI Overview

    Emily shares her career journey from Google to Scale AI and explains Scale’s dual focus on data labeling and enterprise AI applications.

  3. 6:00
    Data Foundation and Integration Issues

    Emily discusses how fragmented, multimodal data and lack of governance create a foundational problem for AI success in enterprises.

  4. 6:41
    Enterprise AI Adoption Challenges

    The conversation shifts to the core problem: why AI is not delivering enterprise-level returns and the three layers where enterprise AI breaks down.

  5. 19:53
    Human‑Technology Alignment and Change Management

    The hosts explore the importance of leadership commitment, change management, and aligning AI tools with existing workflows to avoid failure modes.

  6. 24:09
    Success Patterns of the 6% Winning Enterprises

    Emily outlines the characteristics shared by the small percentage of enterprises that successfully integrate AI, emphasizing data, human, and partnership factors.

  7. 32:43
    Evaluation of AI Agents in Enterprise Contexts

    The discussion turns to how to evaluate AI agents, covering environment setup, test cases, and robustness in non‑deterministic settings.

  8. 40:02
    Scale AI’s Platform and Deployment Services

    Emily explains Scale AI’s skill generative platform, full deployment motion, and how they combine internal expertise with external AI specialists.

  9. 43:35
    Co‑Evolution of AI Agents and Human Workflows

    The hosts discuss how AI agents learn from human interactions, evolve over time, and the balance between automation and human oversight.

  10. 45:32
    Human‑in‑the‑Loop and Liability Considerations

    The conversation focuses on the long‑term role of humans in AI decision‑making, liability, and the need for calibrated confidence signals.

  11. 52:55
    Practical Takeaways for Enterprise Engineers

    Emily offers actionable advice for engineers and managers to increase the likelihood of successful AI adoption, emphasizing pilot use cases and data readiness.

Highlights

  • 8:39

    Frontier Models Lack Enterprise Focusinsight

    The guest explains that frontier lab developers focus heavily on broad capabilities like math and code, leaving professional workflows and operational policies as a narrow pillar in model development.

  • 15:27

    Digitization Technical Debtinsight

    Decades of imperfect digitization meant humans papered over messy documents and fragmented data, but AI agents cannot easily bridge those gaps without proper foundational cleanup.

  • 24:27

    Six Percent Success Ratedata

    Scale AI research shows that only six percent of large enterprises successfully move AI projects from pilot into production and meaningfully integrate them into workflows.

  • 32:48

    Combining Internal and External Expertisestrategy

    The winners strategically blend internal domain knowledge with outside specialists who understand frontier model evaluation and agent development best practices.

  • 34:28

    Evaluating Agentic Environmentsstrategy

    Agent evaluation requires treating the backend systems, APIs, database states, and knowledge bases as part of the test environment alongside the model itself.

  • 38:51

    Simulated Data for Evalsstrategy

    Because real enterprise data involves severe privacy and legal constraints, organizations use anonymized expired data and proxies to build surrogate evaluation environments.

  • 50:58

    Human Liability and Oversightinsight

    Humans must remain responsible and liable for high-stakes AI decisions, maintaining oversight during early deployment before trust allows for greater automation.

Quotes

  • “If you think about enterprise, it's a single word, but there's so many verticals.”
    Unnamed speaker · 11:26
  • X Weather 16:30–17:10
  • Bitdrift 17:46–18:01
  • Warp Build 18:02–18:44

About this episode

It is widely reported that a gap has emerged between enterprise spending on AI and the durable value captured from that spend. Individual employees have enthusiastically adopted coding assistants and chatbots, yet those gains do not seem to be transforming businesses at an organizational level. One of the most important questions in the tech industry today is understanding why AI is not yet delivering returns that match the investment, and what separates the small number of enterprises succeeding from the many that are not. Scale AI https://scale.com/ is known for supplying the human-labeled data behind many frontier models. It now also builds AI applications and agents for large enterprises. That combination of working alongside frontier labs and inside enterprise deployments gives the company a rare view of why enterprise AI may be stalling. Emily Xue https://www.linkedin.com/in/yuan-emily-xue-3483012/ is the Head of Enterprise AI at Scale AI, and previously spent over a decade at Google. In this episode, Emily joins Kevin Ball to discuss the three layers where enterprise AI breaks down, why frontier model benchmarks miss what enterprises actually need, the data foundation problem, how the most successful companies combine internal domain expertise with outside AI specialists, and more. Sponsorship inquiries: sponsor@softwareengineeringdaily.com mailto:sponsor@softwareengineeringdaily.com The post The Gap Between AI Spending and AI Value https://softwareengineeringdaily.com/podcasts/the-gap-between-ai-spending-and-ai-value/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.

Transcript

0:00
54:42

8,951 words · 3 speakers

Topics

  • Google
  • Scale AI
  • Gemini
  • Gen AI
  • GPT
  • Mentos
  • San Diego JavaScript Meetup
  • Laten Space
  • GitHub Actions
  • Sky

Page updated · transcript and summary are machine-generated