Pod Insights Unsupervised Learning with Jacob Effron Ep 50: Fireworks CEO Lin Qiao on Why There Won’t be a Single Model, Will Hyperscalers Win Inference & AI Use-cases with PMF About this episode Lin Qiao, the co-founder of Fireworks.ai, sits down for a deep dive into the future of AI. Lin ran the PyTorch team at Meta, which developed some of the most fundamental open-source AI software in use today. She’s got a riveting perspective on the AI landscape that is a must-listen.
Transcript Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
0:00 Jacob Effron? Lin Xiao is the co-founder and CEO of Fireworks, a company building fast and efficient inference for compound AI systems. She was also one of the creators of PyTorch over at Meta and has raised over $75 million from investors as a Koyan benchmark. On unsupervised learning, we had a fascinating, wide-ranging conversation. We hit on Lin's thoughts on the evolving open source ecosystem, what infrastructure is required to make agents work in 2025. And we also hit on the challenges of building an AI infrastructure company when so many changes are happening at the model layer. I think folks are really going to enjoy this conversation. Without further ado, here's Lin.
0:31 Jacob Effron? Well, Lin, thanks so much for coming on the podcast. Really appreciate it.
0:34 Speaker 01 Thank you for having me.
0:35 Jacob Effron? Yeah, I've been looking forward to this one for a while. And I feel like it's probably a really interesting time to get to chat with you. Obviously, you guys have built just an incredible platform for developers and folks that are building in the AI space. And there's always, it's never a dull moment in the AI world, and I feel like there's always things changing. And, you know, one that I'd be really curious to get your take on is obviously, I think, everyone seems to be talking about test time compute right now. And one, and, you know, many models that seem that they will shortly follow. And I'm wondering, like, how does that change your kind of strategy or product that you offer at Fireworks, like these types of models?
1:06 Speaker 01 Yeah, I think from the inception of Fireworks, we always had that in mind. I think the, like, what is Fireworks? Fireworks is a gen AI platform with a focus on inference. And here, the top level goal is deliver the best quality, lowest latency, and lowest cost. Right? So in the inference stack, and we can, I'd love to dive into details there, because inference is not a single model as a service inference. It's not that simple. So it just kind of allude to test time inference, test time quality scaling law. The inference system we envision in the future is a complex inference system with logical reasoning, with access to a lot of hundreds of small expert models and so on.
1:50 Speaker 01 So the level of problems we're trying to solve is not, hey, you know, this is just API call, in, out, done. It's never this simple.
1:59 Jacob Effron? And so it sounds like you kind of envision a world where, you know, the user puts in a query and like at Fireworks, you're doing the routing and figuring out, like, what is the perfect, like, you know, the best performance model for whatever that query is as a response?
2:11 Speaker 01 Right, so I think Let's take a big step back of what the problem we're trying to solve here, right? So there are a lot of limitation of models. And its limitation comes from the models are not deterministic. It's probabilistic, right? By nature, it's not desirable when you want to deliver factual results, always truthful result to your end user. So controlling that, hallucination is extremely important. And the second is a complex business problem. We have many customers using us. It has to assemble across multiple models across multiple modalities to solve those problems. As we're talking right now, right?
2:52 Speaker 01 So as kind of a lot of application are to consumer, to personal developers, we are not communicating, texting each other. I'm processing audio information and visual information to kind of deliver a good, like, interactive experience, a conversation we're having right now, similar to a lot of GNI-based native applications, consumer personal facing. They also need to process multiple, across multiple modalities. And even within same modality, let's take L M's. There are many different expert L M models specializing doing classifications, summarization, multi-turn chats, tool calling, and they're all slightly different from each other.
3:37 Speaker 01 So that's kind of single models very limited if you want to solve a real-world problem. And the last but not least, single model is very limited in knowledge. Its knowledge is limited by its training data. Training data is finite, not infinite. So a lot of like real information, real-world information lives behind APIs. Yeah. They are public APIs or even proprietary, private APIs within enterprise that there's no way you can access to it without like working directly with enterprise. So across the board, we envision that like the next barrier is how to
4:11 Speaker 01 Go beyond single model as a service, right? So the world needs a notion called compound AI system. As in multiple models across different modalities along with various different APIs. Holding knowledges, including databases, storage systems, knowledge base. Need to work together to deliver the best AI results.
4:32 Jacob Effron? Yeah, but as you think about building these more complex compound AI systems, like what are the tools you think developers will need to, you know, be able to effectively build these?
4:39 Speaker 01 Maybe you can first start talking with like a polar opposite design points, right? So one design is is called imperative. Where you have full control of what is the workflow and what are the inputs, outputs? Everything you want to make it deterministic, right? So so that's kind of one level of design. The other design, so you basically design the the house.
5:06 Jacob Effron? Yeah.
5:07 Speaker 01 Right, dictate that. The other design is called declarative. You define the what, the what problem this system should solve for you. But then you let the system figure out the house. So in general, in the industry, it's not unique to AI. In general, we have these two school of thoughts in approaching the system design. For example, in the database world, it's kind of they're full of those kind of examples. SQL is the example of declarative.
5:35 Jacob Effron? Yeah.
5:36 Speaker 01 Where you data engineers or analysts, they define very like exactly what we want to retrieve out of the database. And database managers figure out the execution plan, the best, most efficient execution plan, right? And then the ETL process is declarative. And you basically define, oh, here's the source and what are the processing logic? How you trigger the next step and how do you do backfield? Very imperative. And those are nothing wrong with either of those approaches. Those are different, just different approaches. And the file was based on our. Pi Torch experience, we want to deliver, ultimate our design principles. We want to deliver the simplest the user experience.
6:21 Speaker 01 And hide all the kind of nitty gritty details on complexity in the backend as much as possible without hurting the speed of iteration. And it's a very fine balance to strike there. So we are leaning more towards a more declarative system with full debuggability and mental abilities.
6:39 Jacob Effron? What are some of the, like, some examples maybe of where you're on the line and you're like, we could do it this way, that would be more declarative or a different way? Like, what are some of those trade offs you have to, or you're thinking about making?
6:49 Speaker 01 Yeah, so for example, where we started, we started with the lowest level abstraction, which is single model as a service. Where today, we provide hundreds of models across large engine model, across audio models, transcription, translations, speech synthesis and alignment, across vision models, where the source can be paid for images, screenshots and so on. And embedding models, image generation models, we're adding video models. So those are kind of the lowest, lowest level of building blocks. And developers can just assemble what they want on top of that, but there's a lot to assemble. A lot of pieces to assemble and the quality control is difficult because they, every week,
7:32 Speaker 01 there are multiple models get released. And they constantly ask us, we want to try this model, what's your suggestion? Should we even try or like, then what is our production stability, right? And how to do version control and all this becomes a problem for them. So, but we start there, so we, we know like, like how, we can understand how the industry is evolving. But then we quickly realized there's a huge gap of usability. Especially when it comes to enterprise and, and there's a huge gap there and we want to fill in the gap.
8:04 Jacob Effron? When you're talking to folks, like, why, why aren't there more Gen AI applications out there?
8:08 Speaker 01 So I think there are a lot of barriers actually. One is, there's no like one model fits all, that's our observation.
8:19 Speaker 01 It's kind of the nature of the training process. The training process is a very opinionated process that you have to pick which subset of problem across like thousands of problem in the world you care the most about and you devote the biggest amount of resource and money to acquire data and make sure your data quality and a variety of data in that space is the best. And which subset of problem you don't care and we're in the middle, right? So because of that, the end result of the model will be really, really, really good at doing things and then really bad at doing certain things, right? So that's just the nature of model training. We believe the future are the hundreds of small expert models.
9:02 Speaker 01 Because when you shrink the problem down to a narrower space, it's inevitably much easier for small model to thrive in pushing the quality. And as a matter of fact, in the open source world, this is really great for the open source community because with open source based models, it gives a lot of control for you to customize your model. There are a lot of model providers, they focus on post-training or fine-tuning and deliver specialized model, giving back to open source community that's really, really good at solving certain problems very well. So we believe the future are hundreds of small models.
9:45 Speaker 01 I think that's kind of where we see the enterprise world is moving towards, they want to have more control. They want to have stability.
9:53 Jacob Effron? In this world of like hundreds of small expert models, I can imagine one world in which, you know, there's so many people taking open source models and tweaking them in some ways and so fireworks just has a big library of them and I'm in enterprise and I have a problem and, you know, you're like, hey, here's a dozen really great models that will orchestrate together. I could also imagine there's probably some cases where you tell an enterprise, hey, you should really fine tune for this use case or you might actually, maybe in some cases, even pre-train a model to some extent. How do you think about where that plays long term? Like, how do you think in five years, like, will enterprises be fine tuning a lot of models or pre-training or how does that evolve?
10:26 Speaker 01 We deeply believe in customization, again, as I mentioned, but the process is it's kind of, it's not, I would say it's not straightforward, like, it's not like, oh, like, one minute done, right? So it's longer than that. So we see a fine balance between
10:42 Speaker 01 Like fine tuning and problem engineering. Right, it's, it's, there's a strong trade off there and that's where we are working on. Because we're believing in customization, we're working on how to make customization extremely easy. So the balance is falling, right, so um it's much immediate. You can see the result immediately, it's much more responsive, interactive using problem engineering. This is kind of, that's the nature of it. And we see a lot of enterprise or developers start with problem engineering, right? So hey, is this model steerable towards the direction I want to do? And I can test it out quickly. Um but then we see thousands of lines of system problems. Yeah. At some point, if you want to continue to steer it, it got stuck. Yeah. Because you don't know where to change and you know, it's gonna all get lost.
11:27 Speaker 01 Uh so, so then um. So solving a very complex system problem management is a problem myself. Yeah. Uh huh, so actually that eventually is the key to how to make customization easy. Is to solve that problem. And, and we are gonna launch a product to solve that problem. Interesting. Yeah, so um and then we see, um then at that phase. Leave that aside, at that phase, you have thousands of lines of system problem, then what do you do? Um and usually that's the time, it's the prime time for you to do, use fine tuning to absorb that system problem into the model itself. Because by that time, you should already proven the model is steerable to solve your problem.
12:10 Speaker 01 You, you are able to let the model follow your instruction. Uh your instruction is pretty much the data you provide to the model. Uh and then, you know, it would be good candidate to do the fine tuning. Uh and that already mean you're probably evolving from pre-part of market fitting to post-part of market fitting, product scaling. And if, once you absorb the long lengthy system prompt into the model, the model can run faster also. And the cheaper and also with high quality. So we see that movement, it's very organic movement. But still, um getting stage of managing, like even kind of uh before you absorbing to, to fine tuning.
12:47 Speaker 01 Managing these long process and problem is still a very challenging thing. So, yeah.
12:52 Jacob Effron? Do you think pre-training ever makes sense for enterprises?
12:54 Speaker 01 Pre-training is kind of the whole entire industry is going to hey, pre-training is being consolidated, right? So into hyperscaler, that's pretty much the reality. We do feel like there are, we do see enterprise, some enterprise do pre-training models because they have, this is kind of core to their business or for other reasons.
13:19 Speaker 01 But, yeah, so, but the differentiation, the question we see a lot of enterprise asking themselves is, or even startup asking, what's the differentiation? Because pre-training is very expensive and RI has to justify you threw a lot of money and the human resources into, into doing this. And if you can just do post-training on top of very strong base model, then RI is much stronger and you are much more agile to test different ideas.
13:43 Jacob Effron? You said, you said an interesting vantage point because you see so many people building on top of fireworks. Like, what use cases for Gen AI do you feel like have product-market fit today?
13:52 Speaker 01 I would say there are, I would classify those use cases in probably two different buckets.
13:59 Speaker 01 But in summary, I think most of use case we've seen successful getting adoption is kind of human in the loop, automation.
14:10 Speaker 01 Not human, other loop automation yet. So my hypothesis is a Gen AI system has to be, it has to be human, debuggable, understandable, maintainable, operable. If a human cannot evaluate it, cannot maintain it or put influence over it and operate that in production, it will be really, really hard to get adoption. So because of that nature, we see a lot of kind of vibrant kind of product offering from all different kind of assistant to human in different categories.
14:52 Speaker 01 We have seen people building assistant to doctors to make their scriving much easier. I've seen assistant to teachers and students and people want to learn foreign languages, educational. We have seen assistant to coding. Of course, that's a highly competitive space, but we have worked very closely, for example, with Cursor, Resource Web and many other great companies. We have seen assistant to, like medical assistant. Apparently, there's shortage of more than million nurses in North America. And there are a lot of patient need being like paying attention to. So that is a whole slew of like to consumer, presumer, like developer space. So that is one.
15:38 Speaker 01 The other is kind of more B two B, as in there are automation, like call center automation. But more like call center automation is a great example. It has, you can build assistant to human agent to make them more productive and answer the questions better or replace human. We have seen a lot of success of building this assistant to help human agents, right? So that's another example. A lot of process of optimizing business logic, business workflow, efficiency and so on.
16:10 Jacob Effron? And then what about on the model side? What have you noticed in terms of the models that are actually getting used by companies?
16:15 Speaker 01 We see a lot of convergence of variation of Llama models. I think that's a testimony to the quality of the model. And it's a very strong base model, very, very good instruction following and very good for tuning. So and of course, meta backing it is very important and see a lot of enterprise adoption there.
16:38 Jacob Effron? Obviously, I'm sure something your enterprise customers think a lot about or, you know, is just like evals. And I'm sure, you know, you obviously have thought a lot about this, like, what's kind of the current state of evals? And, you know, how do you think about the right building blocks to give, you know, enterprises that might have very different use cases and as a result, very different ways of evaluating success?
16:59 Speaker 01 Many enterprise do at the stage of eval is vibe based.
17:05 Speaker 01 Which is not surprising, right? So yeah, yeah, yeah, so they are, but they quickly realize. I mean, for the early stage, product development is kind of more vibe based. I just want to get a sense of how this product is like feel like with, with, with different models and so on, right? And they quickly evolve to the stage that they are consciously building eval. And they know this is kind of an investment area. But for them to stay on top of the state of art, they have to be able to evaluate. And also they have to be able to evaluate once they get into kind of more deeper problem engineering or fine tuning. Like how to just evaluate quality is important. They cannot always go to A B testing.
17:48 Jacob Effron? Right.
17:49 Speaker 01 To, A B testing is ultimate process to, to determine the product impact. But it's longer cycle, so.
17:58 Jacob Effron? It is funny that we talked to some, I think Source Graph's customer of yours said they don't even bother with evals. They just release it out, A B test it, and they'll know pretty quickly from their developers which, which model is better.
18:08 Speaker 01 Yeah, so but the, the evals are very important because.
18:14 Speaker 01 And, and they, for our customers' point of view, they realize investing in generating good eval data set and, and adding more and more to it can give them a very clear view.
18:29 Speaker 01 What matters, what doesn't matter, because like in this process of catching up with the constant moving train of those small expert model. Their, the model is not just every week, there's a better model like beating leaderboard. The model is also getting more and more specialized. And they, they will also, they are, many of our customer, they start from the mod, their product is open ended. They don't have a strong opinion what kind of question they should expect from their user to their they start to get more and more clear opinion. And they're hardened their product design into open-ended product to like specialized product in different product features. And then from those product features, they want to have specialized model to solve this problem. So it's a natural evolution, evolution of product development.
19:14 Jacob Effron? And I guess like, you know, maybe just talking about some of the stuff you guys have built. Obviously, you're taking the great models that exist in the open source world. And then you also build your own models, right? You built F1. Maybe talk a little bit about why, you know, why you decided to do that?
19:28 Speaker 01 So we release F1 under API and that's a model API. Basically, how you use F1 is as if you are using a new model. But F1 is really a complex logical reasoning inference system.
19:43 Speaker 01 So which underneath, we haven't talked too much about what's underneath F1. But it's kind of here is a snippet of what's underneath. So underneath there are actually multiple models. And also there is a logical reasoning steps that we implement in our system. So then actually building a system is very complex. It's not, it's not like just regular single model as a service inference. And also we have to solve a lot of, a lot of quality related problem. Because now you let the model talk with itself, let the model talk with each other.
20:23 Speaker 01 And how, what kind of information they communicate to drive quality and they are sensitive to that, right? And how to do quality control in this complex system? I think the complexity is actually higher than building a database management system.
20:41 Speaker 01 In that sense. And yeah, and then of course, because there are multiple steps involved. And then your overall influence latency and then cost becomes more interesting. So that's our bottom bar, because we're a team like really specializing optimized for that. And then we view the our like when we start a company, we build a lower level of single models of service. That's a building block towards this complex logic reasoning stack.
21:09 Jacob Effron? Interesting. And then obviously, I think a big part of it is function calling as well, right? And like, what are the barriers, I guess, to getting function calling even better today? It feels like we're still kind of in the nascent stages of that in many ways.
21:18 Speaker 01 So function calling, like many, many people recognize that actually, we see a lot of after we launched one. We have, we have a waiting list for people to get in. And we talked with many of them. Apparently, their use case, the majority of use case is actually building agents. And they need function calling. So what is function calling, right? Function calling in basics is an extension point for, for this model to call into other tools to enhance the quality of, of the answer. But function calling is not just about calling into one tool. It's actually very complex if you think about it. Usually, people use those function calling model in the multi-turn chat context, right?
22:00 Speaker 01 The model is, it should be, it need to be able to hold a long context of what has, what the commerce. And then use that context to influence which is as tools call. And oftentimes, they also need to call into multiple tools, up to like, could be up to hundreds of tools, tool selections. And also, they need to, it's not just call one tool at a time. Oftentimes, they also need to call like multiple tools in parallel. And it's sequential, so it's a complex coordination plan in one shot. So our function calling is able to do parallel sequential complex planning. And then orchestrate and execute on that plan.
22:44 Speaker 01 Of course, then, the precision is important, like if I ask the model to like do a general chart of a stock price of top three cloud providers. Underneath, it will return the answer quickly and then show you a chart. But underneath, there's a search to find out top three. There's three parallel function calls to get the stock price, and then one call into the chart to get the chart back. So this is a very simple example, right? But it demonstrates the complexity of orchestration. And the model need to have very strong capability of understanding, if you're plugging your tool, when to call this tool, and how to drive that position is very important. So that makes the tuning process very complicated.
23:25 Speaker 01 But we have been investing this space for now about a year.
23:30 Jacob Effron? Yeah.
23:31 Speaker 01 So yeah, so like, we start to see, like, when we start to release the first function calling model, I think we are ahead of the adoption curve.
23:38 Jacob Effron? Yeah.
23:39 Speaker 01 The use case hasn't kind of fully taken off.
23:41 Jacob Effron? And how do you think about like what, you know, why decide to build that versus, I'm sure if you waited at some point, there'd be, you know, a good open source model with some of those capabilities. Like, how do you kind of figure out when to build things in-house on the model side?
23:52 Speaker 01 Yeah, so we heavily build on top of open source community, right? That's absolutely, it's the case. And we are betting on that because we believe the hundreds of small expert model will come from the open source community.
24:05 Jacob Effron? Yeah.
24:06 Speaker 01 So that is directly aligned with our vision. But at the same time, we're also heavily investing in the compound AI system, where it's all about composing those hundreds of small expert model to solve a complex business task in a very easy way. And the critical part of this composability lies in this layer of orchestration. So the model was intelligent enough to be able to call into different tools. And here, you can think about the general way of thinking about tool is each individual small expert model is also a tool.
24:41 Jacob Effron? Right.
24:42 Speaker 01 In addition to this particular large language model. So then it becomes a critical ingredient of tying everything together. And we will invest in this strategically critical area that we cannot just wait to see what happens.
24:59 Jacob Effron? Where do like the reasoning models end up fitting in? I mean, obviously, at least the ones we've seen today are, you know, I think of them as bulky and generally good at reasoning of all things. Do we end up with like small expert, you know, like test on compute intensive models as well? Or like, do you imagine we'll only kind of route to those in very specific instances? Or like, as I think about these compound systems, I'm just curious like where these O1-like models will fit into that.
25:23 Speaker 01 I think even for reasoning, there are different paths to solve reasoning problems. And there will be different models specializing in different paths. One path is to have a very strong base model itself to do self-inspection. And a lot of technique already being discussed with chain of thought, truth of thoughts, and backtracking and all these different technologies. So that's this one. And there will be a new set of models. They can do logical reasoning, not in the prompt space. And this is something I'm very excited about, but in latent space.
26:06 Speaker 01 Because you can think about, when we think, right? We don't have to talk in words, like our thinking process. We can, but we don't have to, right? So I'm pretty sure when we process a problem in our head, we're probably thinking in a different space.
26:21 Jacob Effron? Right.
26:22 Speaker 01 Right? So similar to the model as well. So there are a lot of active research happening, like to how to make the thinking process much more efficient and much more native to the process. So I'm very excited about that kind of research. There will be other flavor of logical reasoning. So we are not, we don't want to be very opinionated about which one is going to win. Instead, we're going to integrate all these different flavor into our logical reasoning system.
26:51 Jacob Effron? I love that. I guess outside of what you're doing at Fireworks, what do you feel like are some of the other big unsolved problems in AI infrastructure today?
26:59 Speaker 01 I think we have seen a lot of movement into building agentic workflows.
27:06 Jacob Effron? Yeah.
27:08 Speaker 01 And again, I think we're still at the early stage of figuring out what is a right user experience.
27:16 Speaker 01 What is the abstraction, the whole industry, right? What is the right abstraction? What we should hide behind the system? We should expose to developer. We're still kind of in very early stage and there it's, I think it's very experimental right now. But where the abstraction lies will determine the complexity of the infrastructure hiding behind the abstraction. But we, I think we now start to form opinion around that.
27:48 Speaker 01 And in a month or two, we're gonna kind of GA F1. And so, okay, I don't think in process is building F1 is one is kind of, F1 is our own exercise to understand the system abstraction and the complexity of building logical reasoning engine.
28:11 Speaker 01 And from there, when we GA, we want to expose developer facing, plugins and how developer can build their own F1s. So yeah, so I'm very excited about that direction.
28:24 Jacob Effron? That's super cool because obviously you learn the tools and the abstractions that are necessary by building it yourself. And then, you know, you basically allow developers to recreate that. I love that as an approach. I guess you know, a lot of, it feels like on the hardware side, I mean, obviously everyone's using Nvidia, but there's a lot of competitors have been popping up. And I'm curious like what you make of that space and the viability of some of those efforts. I think you guys support AMD chips for inference on the platform. Like, you know, when does that make sense for developers? And how do you see that kind of evolving?
28:52 Speaker 01 In general, we see a kind of scarcity in the developer space that they understand hardware, low-level hardware optimization. And we also see a very
29:07 Speaker 01 Big change in the kind of hardware development cadence. Before, it's kind of every three years, there's a new hardware skill. Not every one year, there's a new hardware skill for each hardware vendors. And then the whole hardware space is moving very fast. And how to kind of access to the best hardware? And by best, I really mean, there's, again, there's no one size fits all. So it really depends on your workload pattern. Even for the accessing the same model, there's no best hardware for one model even. Because your workload distribution is gonna determine where is your biggest bottleneck. And different hardware skill is gonna be the best for solving, kind of removing certain bottleneck the best.
29:49 Speaker 01 So we even at the fireworks layer, we absorb the burden of integrating and determining which hardware is best for what kind of workload. Even for one workload, when you have mixed access pattern, we can route to different hardware. So yeah, so we want to really kind of alleviate the concerns and burden of developer, they should focus on building product. And we will, we will kind of take the complexity of managing and optimizing for the hardware.
30:20 Jacob Effron? Yeah, it seems like you're obviously super focused on these compound systems. You know, there's obviously always gonna be some set of folks that like, you know, just wanna call a Lava model as it is and run it. In that world, you know, it feels like for a while, there was really a lot of competition on the inference side. Like a new model, an open source model would come out and everyone would be offering their prices and it would get lower. You know, in that game, do you think that like, eventually that's the hyperscalers are best suited to win? Just like kind of straight inference on an open source model? Or like, how does that kind of develop?
30:49 Speaker 01 That's a very complex question to answer. I think, I think all hyperscalers, they want to be the Apple, right? So they want to build iPhone. They want to kind of build a vertically integrated stack because they can. So, but again, the direction we're heading towards is we feel like the future lies in hundreds of small expert models. And we want to harvest that energy and build a compound system that leverage the best models. To solve a complex problem. So, in my view, I feel like the hyperscalers, what's their biggest benefit?
31:30 Speaker 01 Their biggest benefit is that to solve the problem requires a lot of resources in terms of money and the human. Those examples are like building data centers, acquiring powers.
31:43 Speaker 01 And install tons and tons of machines, deploy them, light them up. Yeah, and turn up large-scale storage and compute. Cloud provider, please solve those problems, right? So this kind of massive problem at scale.
32:01 Speaker 01 And we specialize in solving problems that requires a lot of the combination of engineering craftsmanship and deep research. And then we can deploy at scale. So I think inference is like that. I work at Meta, it's not like Meta has a thousand people inference team, right? Because it, once you have a, a system that can be horizontally scalable with high production quality. Then the rest is just let it scale. That's the beauty of designing a highly scalable system.
32:41 Speaker 01 So, so in that sense, I think this is squarely in our, you know, our strength and expertise. And also, again, the inference system is not simple. It is actually gonna, we're evolving to, to, to get into this compound logical reasoning inference system. Where there's a lot of complexity in building that. And that complexity is not by throwing people and money to solve, so, yeah.
33:08 Jacob Effron? Yeah, I guess with these smaller expert models, one, you know, one thing I wonder is, you know, as these models get smaller. It feels like there's a big trend to running them locally too. And I'm curious like how you see that playing out over, over the next few years? And then, you know, if that, you know, becomes part of the fireworks platform as well.
33:23 Speaker 01 Yeah. I've seen a lot of argument for running models locally for two reasons. And also, we should talk about what does local even mean, right? One reason is cost-saving, because yeah, you need to pay GPU on cloud and, you know, running on desktop, you don't need to pay. And yeah, so the other is privacy, right? So I have different thoughts on that. I think that makes sense to kind of offload compute from cloud to desktop. I think many application like Zoom, for example, that save a lot of cost for them, right? But offload, offload compute from cloud to mobile is a different story.
34:06 Speaker 01 Because mobile phone has a very, like, it's getting like beefy and beefier, but it has a very limited power. Right, so at Meta, for the application, a lot of application metrics are closely monitored, including cost of time, your power, like, consumption. All this affect application rating and user adoption and experiences. So because of that, the model that can practically deploy to mobile is tiny. Like, 1B, 10B, and very, very small. And those model has limited capability, right? So I think like, between mobile and cloud, this kind of the difference is extremely big.
34:51 Speaker 01 And offload to desktop, I think for many like consumer personal facing application makes sense. But also, like, there's another argument of, hey, you know, it's more private. And privacy is interesting, right? Because we have a lot of personal data is already on the cloud.
35:07 Jacob Effron? Right.
35:07 Speaker 01 Most of our personal data is not on like local desk anymore. So then, kind of, hey, does that even make sense for privacy? Yeah, so that, I think that's a separate concern.
35:19 Jacob Effron? Yeah, super interesting. I mean, I guess on the, on the open source side, you mentioned, obviously, you used to work at Meta. It feels like, you know, obviously, they've been providing a huge service to ecosystem by training these larger and larger models. Like, how far do you think they go on continuing to scale spend on, on open source models before, like, at some point? I mean, I guess everyone's asking this for all the players, not just the open source players. But how do you think about like the, the extent to which it makes sense to keep pushing on, on pre-training spend?
35:43 Speaker 01 I think that's a good question to Zach.
35:49 Speaker 01 No, I'm just excited. We work very closely with Meta team, of course, we came from Meta. And we actually co-design, they're not just open source Llama models, they're also building a standard called Llama Stack.
36:04 Jacob Effron? Yeah.
36:04 Speaker 01 And Llama Stack, the intention is kind of standardize the
36:09 Speaker 01 the tools stack around Llama models. And we co-design with them because we are kind of, we have a lot of information coming from our customer that can help them like make design choices. So, so I think that's kind of ambition from Meta is to build this Android world where everything is nicely standardized and you plug into different components and make it very easy to adopt. And from my vintage point of view, my understanding is Llama 4 is coming soon, right? So there, there, there will be continuous investment from Meta that train. I haven't seen that stopped.
36:55 Speaker 01 Of course, like if it's a general question, when would model providers stop or invest less on pre-training is when there's not much ROI. So why there's not much ROI is because we're hitting a wall of data. Where we're now data, everyone call the same internet data. There's just, we have exhausted synthetic generated data. We have exhausted a combination of multimedia and text data. Then kind of there's less return.
37:25 Speaker 01 Before we hit that wall, I think there will be continuous investment.
37:29 Jacob Effron? When do you think we'll hit that wall?
37:31 Speaker 01 I think we're, it's definitely getting longer and longer. Like from like OpenAI's release point of view, right? So.
37:40 Speaker 01 And I definitely see a lot of investments start to shift from pre-training to post-training, from post-training to inference.
37:47 Jacob Effron? Right.
37:48 Speaker 01 So I think we're already hitting a soft war, not a hard war. So yeah, so I think in general, as an industry, we're the ROI is kind of transitioning from pre-training to post-training to inference.
38:04 Jacob Effron? As you think about like all the sets of tools that are required to make like an agent successful, I mean, there's one world where as fireworks, you can be a one-stop-shop and like build all of them. There's others where you could partner and you go, look, we're not going to build, you know, this part of the thing. Like, how do you think about that?
38:16 Speaker 01 Yeah, so we always are very compatible with imperative agentic tools, right? I think most notably is LanQian.
38:28 Jacob Effron? Yeah.
38:28 Speaker 01 And we have been a very strong partner with them because we are not gonna like, LanQian is doing great work in what they are building. Huge community following and adoption, really awesome. And we just want to simplify the next level above single model as a service by like composing wherever it makes sense, multiple models to solve the problem much better. So and we are fully kind of, we can, we will, we will not change our position, we'll plug into LanQian. To, for example, as one example, there's also, of course, many other examples in the community.
39:10 Jacob Effron? Yeah, makes a ton of sense. As you think about kind of like the competitive landscape today, I mean, I imagine when people think of fireworks, they also probably lump you in with like Together AI and, you know, I know Databricks talks a lot about these like compound AI systems too, like, how do you think about what like differentiates fireworks today? And then, you know, five years from now, like what will ultimately determine who kind of like captures this really interesting space?
39:30 Speaker 01 Yeah, we are probably thinking, so compound AI actually is the term, we didn't coin that term, right? So it's kind of coined by Berkeley. And we know that Databricks is also like looking into that space. So I think it's great, it's a new category that we're defining, right? So definitely multiple players will kind of be in the same space and we are, we're very happy that Databricks also think about that problem. So, so I think that that space is very meaningful because it's a very complex space to like for. Application developer, product developer engineers to get into.
40:11 Speaker 01 And there will be a rich set of tools that emerge in that space to make development much more efficient, and today we are not there yet.
40:20 Jacob Effron? Yeah.
40:20 Speaker 01 So we are determined to kind of be a key player in that space. Other than that, I, I, we are not, we are not in the space of being a GPU cloud, for example, and provide access to cheap GPUs, that's not our play at all. So we actually, we build on top of GPU cloud to, to offer a complex inference stack.
40:46 Jacob Effron? I guess one thing that's, I think's really interesting about the company is, I guess you started in Fireworks like, what, a few months before ChatGPT? And so I'm curious, like, you know, what, like, as you think about the original vision for the company, and then, you know, where you've come through today with this like Gen AI craziness, like, how much of it was the same, you know, sitting there in September twenty-two? How much of it had you had to like change as the world has just changed so much in the last few years?
41:09 Speaker 01 Yeah, um, it's actually, it's, yeah, this is a very interesting question. When I started Fireworks, there was a actually active debate, is AI here or not?
41:21 Jacob Effron? Yeah.
41:23 Speaker 01 Because this pre kind of awareness of foundation models, and it's kind of, oh, there's so many like fragmented AI application, and the data is not there, you know, AI is not there, right? So, but we clearly see that AI is coming, right? So usually hyperscalers are like three to five years ahead of the whole entire industry, and Meta is massive AI powered. And through, clearly the industry is kind of through PyTorch adoption, it's clear, they see that, you know, the wave is coming. So that's kind of the timing about Fireworks. And the Gen AI did skew, ChatGPT heavily skewed the adoption curve of AI.
42:07 Speaker 01 And that's a special kind of AI. So it's very interesting, Gen AI is a different beast, not because it's Gen AI, it's magical, because it's fundamentally changed the
42:18 Speaker 01 The accessibility. So pre GNI, when it's like traditional machine learning or early deep learning models, any company investing AI particularly have to first hire a machine learning team. Because they have to train model from scratch. There's no other alternative. There's no other model that's pre-trained you can build on top of. And that means this team has to curate, spend a lot of time curate data. And then the training process, because training from scratch is kind of take a long time. So they have to have the resource to hire those very scarce talent and through money into training.
42:59 Jacob Effron? And was the original fireworks product like built for those people? Or like, cause you know, that was originally what it felt like was going to be needed to get adoption.
43:05 Speaker 01 Yes, yes. That's what I, when we think about, when we started, we think, we think about that cohort. And then with GNI, it makes it much, it's kind of change the landscape, right? You don't have to hire a big team to carry data to, because it has a, the beauty of GNI is it creates a foundation model that absorb a majority of the knowledge. And then you just build on top of it as is or you tune, right? You have thousands of samples you tune. So then you either don't need to have any machine learning team. You just need to have that application product team to build on top of that directly. Or you have a small machine learning team, right? So this is a massive, massive unblocking of accessibility of, of that technology.
43:49 Speaker 01 And that, that's why we see kind of the adoption curve take off like crazy. Because of that accessibility is fundamentally different before after. So because of that, we, these are focused on it because the pull is much stronger in that direction. And the other side effect is all these models, all GNI models are PyTorch models.
44:07 Jacob Effron? Yeah.
44:08 Speaker 01 And we wrote PyTorch code for years. And we operate PyTorch, large, complex PyTorch model in production with high volume traffic for years. And that's what we're really without doing.
44:19 Jacob Effron? Yeah, very well situated for, for that. You mentioned when you, you know, you, like when you're at a place like Meta, you get kind of a window into the future. Like before, it's obvious. Like, was there something you saw when you were at Meta where you're like, God, this is just going to be clearly so big? You know, shortly thereafter?
44:33 Speaker 01 I think it's more the kind of AI is going to be very big wave. That's clear. So I worked at Meta for seven years, right? It's interesting time. When they joined Meta, they were going through the mobile first transition and the kind of tail end. And then mobile first, and then the industry is also going to mobile first, right? So from desktop to mobile. And mobile first enable consumers to be able to access application everywhere. And that drove up adoption and engagement. And because of that, it produced a lot of data. And a lot of data is powering AI, right? So that's, then the whole entire industry is also following the mobile first, drive a lot of data, and they're power AI. So that trend is very clear.
45:18 Jacob Effron? What AI research do you pay attention to? Maybe even outside of like the core day-to-day of fireworks?
45:23 Speaker 01 I usually pay attention to two kind of research. One is like model system co-design kind of research. Because there is a interesting organizational division, as in there's researchers, they focus on quality only.
45:41 Jacob Effron? Right.
45:42 Speaker 01 There's a system builder like us, we focus on like deliver the best quality with low latency and low cost, right? So like we solve a system problems. But oftentimes, the best ROI is think about that together.
45:56 Speaker 01 And that's how we operate at Meta is kind of the research team and kind of info teams sit very closely to discuss trade-offs, right? And the co-design. So there are a lot of research in the co-design space. I think it's gonna be very appealing to make, to kind of find the best design point across quality, latency, and cost. As a three-dimensional optimization. The other kind of research is kind of fundamentally different disruptive kind of research. As in transformer as a technology is overdue for disruption.
46:32 Speaker 01 Right? So where are the next generation of transformers, right? That's gonna change how we train model, change how we do inference, and so on. So that's very interesting. And also, in the agentic world. How different agents is gonna like communicate with each other? As I mentioned, thinking the latest space, that kind of very new kind of thinking is very, very interesting.
46:58 Jacob Effron? I mean, one challenge that I imagine in building an infrastructure company is that obviously the pace of change is so fast, both like the improvement in models, plus just like how enterprises are thinking about and actually using these things. And I can imagine like one school of thought would be it's changing so fast, like, you know, we'll just kind of quickly update what we're, you know, what our kind of core tools are because of how fast it's changing. And at some point, it will settle down and then we can kind of build the standard set of tooling. How do you think about that tension of like, you know, building for however folks are doing things today? With an, you know, without knowing exactly, maybe in two, three years, like models have all these other different capabilities and we're designing systems in all these other different ways.
47:34 Speaker 01 Yeah, that's a really good question. That's all we think about all the time, right? So we don't want to keep chasing what's coming because like chasing is exhausting. We'll always want to stay on top of the curve.
47:49 Speaker 01 So I think the model capability is definitely evolving, has evolved from two years ago.
47:58 Speaker 01 Especially like if we just look at OpenAI's model capability from G P three to now today. It's there and I differences.
48:08 Speaker 01 And but fundamentally, like, like a few trend that doesn't change, right? So one is, again, anchor back to our vision is we believe the direction is specialization and customization. That doesn't change how the model core capability evolves. Because we just don't believe that one size fits all different workloads with proprietary data and so on. And we believe there's a much better solution if you can customize, if you can steer, if you have control the destiny here. So that's why we built our stack to to heavily kind of enable that in an easy way.
48:52 Speaker 01 Specifically, we offer on top of our inference engine, which is one size fits all. We have an optimizer, file optimizer, that's one size fits one. We take inference workload as input and also your customization objective as input and spit out inference deployment configuration along with potentially a model, adjusted model for you to have the control whether you want to deploy that or not. So this is just a process of we kind of close the loop and make it super easy to customize. So I don't think that's going to change in the future.
49:30 Jacob Effron? Totally. Just what you're customizing may change, but like still the way you will go about it. I think that makes a tremendous amount of sense. Well, we always like to end our conversations with a quick fire round where we get your take on a standard set of questions. And so maybe to start, would love just one thing you think is overhyped in the AI world right now and one thing you think is underhyped.
49:48 Speaker 01 I think overhyped is the perception GNI is magical. It's a recipe of our problems. We should just ask it. Any questions, it's kind of, we'll have the right answer. I think, I think it's going through the correction time right now. And again, that's where kind of we believe like there's no magical one model that solve all the problems the best way or in the correct way.
50:11 Jacob Effron? What's one thing you've like changed your mind on in the AI world in the last year?
50:14 Speaker 01 I think my hypothesis in building this company and how this GNI technology can adapt, adopted. And our go-to-market strategy in my mind is always like in a sequential way. Startups, startups are like GNI native, so they will be on the frontier of adoption. And then we will work with digital native. They have very strong engineering resource and always kind of be more tech-forward. And then traditional enterprise because they have more like other kind of problem to solve. But now we're working with all of them at the same time.
50:50 Speaker 01 And to me, it's a little bit crazy.
50:54 Speaker 01 Of course, the use case are not all the same, but the adoption curve is happening all at the same time. It's very different from what I was imagining.
51:05 Jacob Effron? Totally. I mean, on the application side, it feels like the normal rules of how long it takes to land one big enterprise and then land a few after. It's all out the window, right? Some of these application companies are scaling so fast in the large enterprise. I think there's just a tremendous appetite for it.
51:17 Speaker 01 That's right. That's right. So that's why I feel like we're in this revolutionary way. So a lot of things will be done differently. Not just the application we build will be different. Technology adoption curve will be different. But also, even how to think about go-to-market will be different. The sales cycle is much shorter, and people are open to think the procurement process differently. So it's very interesting, a ripple effect from this massive transformation.
51:48 Jacob Effron? Do you feel like, is there any difference in what you have to build for enterprise versus the startup folks? Or is actually the needs pretty similar?
51:55 Speaker 01 I think, my observation is startups, they are typically, like, they want to get access to, and when we talk about abstraction, right? They want to get access to the low-level abstraction.
52:12 Jacob Effron? Yeah.
52:13 Speaker 01 Because they want to tinker a lot, right? So assemble a lot of things. And for traditional enterprise, I'm not saying this in an exhaustive way, but typically, they want to hire a higher-level abstraction. If they don't need to solve for low-level details they don't need to pay attention to, rather not to pay attention to. So even at the matter, we have those different abstraction layer for different kind of team, because they want to pick and choose, right? So usually, two-level abstraction will suffice. Kind of like they were happy with, at least have two choices. And we see a little bit of those.
52:47 Jacob Effron? Totally. Must be interesting to build both at the same time, as you're rapidly scaling.
52:51 Speaker 01 Right, right. So we need a low-level abstraction for ourselves anyway. So to us, it's not additional overhead, but we see the adoption will happen at different layers.
53:02 Jacob Effron? Yeah. I know it's probably, I won't ask you to pick a favorite, but what is one favorite application that's going to work today?
53:10 Speaker 01 This is a tricky question.
53:14 Speaker 01 I'll say Cursor.
53:17 Speaker 01 Which.
53:19 Speaker 01 But we see a lot of similar company in in kind of productivity space, right? Not just Cursor, but also Source Graph, Z, Cognition, Factory. They're all kind of very forward-thinking company. And I think in that space, it's still early. But you know, that's the space we're very excited about, the whole chat space.
53:40 Jacob Effron? Do you think someone will spend a hundred billion dollars on a on training a model in the next few years?
53:45 Speaker 01 Ooh, maybe. I think it only makes sense there if they are training using a fundamentally different model architecture.
53:54 Jacob Effron? Yeah.
53:55 Speaker 01 It's so disruptive, it's worth that investment.
53:58 Jacob Effron? Yeah. You know, you kind of already mentioned the coding space and then some of those startups. But any other like AI startup or space that you're really excited about outside of fireworks right now?
54:07 Speaker 01 Again, I think the the agentic.
54:13 Speaker 01 The agentic word hasn't been fully figured out yet.
54:16 Jacob Effron? Have you seen any applications like do that well?
54:18 Speaker 01 We have seen like, we have seen early applications. For example, the digital SDRs, the, you know, the digital marketing. And yeah, so we have seen kind of good startups getting great adoption there. But I do, I do think in the future, there will be a lot more complexity in that space. And it was too early.
54:50 Jacob Effron? Well, this has been a fascinating conversation. I'm sure folks will want to pull on all sorts of threads. So I want to leave the last word to you. Where can folks go to learn more about you, about what you're building at Fireworks? The floor is yours.
55:00 Speaker 01 Yeah, so yeah, definitely. So we offer for the benefit of our developer community, we offer a self-service platform. Very simple, just go to fireworks.ai and you can access to our playground and hundreds of model capabilities we have. Yeah, and feel free to reach out to me also. Connect me on LinkedIn and love to hear your use case, your challenges and pain points.
55:25 Jacob Effron? Amazing. Well, thanks so much, Theresa. This was a fascinating conversation.
Previous Ep 49: OpenAI Researcher Noam Brown Unpacks the Full Release of o1 and the Path to AGI Next Ep 51: Former Chief Research Officer of OpenAI Bob McGrew - What Comes Next for AI? Page updated October 12, 2026 · transcript and summary are machine-generated