Pod Insights a16z Podcast a16z Podcast | The Cool Stuff Only Happens at Scale About this episode Distributed computing frameworks like Hadoop and Spark have enabled processing of "big data" sets -- but that's not enough for modeling surprise/rare "black swan" or complex events. Just think of scenarios in disaster planning (earthquakes, terrorist attacks, financial system collapse); biology (including disease); urban planning (cities, transportation, energy power grids); military defense ... and other complex systems where unknown behaviors and properties can emerge.
They can't be modeled based on (by definition impossible) limited data. And parallelization for this is hard. But what if companies and governments *could* answer these seemingly impossible questions -- through simulations? Especially ones where we can directly merge in knowledge and cues from the real world (sensors, sensors everywhere)? CEO of Improbable Herman Narula and Stanford University professor-in-residence at a16z Vijay Pande discuss this and more with Chris Dixon in this episode of the a16z Podcast. And as Herman says, "the cool stuff only happens at scale".
Transcript 0:00 Speaker 2 Hi, this is Chris Dixon, it's the A16Z podcast. I'm here with Herman Narula from Improbable and Vijay Pandey from Stanford, who's also a professor in residence here at A16Z. Um, let's talk about distributed computing. So my my own view, I guess, I'll just start it off is that um over the next few decades, distributed computing will be a particularly important topic because we're now with things like AWS, a wash in computing resources. Um, you know, compute is becoming approaching zero, storage, networking. But most of this is on you know on multiple physical machines, and it's very, very hard uh for software developers to build uh software that that distributes well. Um so we have things like I think of Hadoop and and its successor, we think of successor Spark as frameworks for doing distributed computing for a specific application, which is data processing.
0:53 Speaker 2 And I think we'll probably see more of that kind of pattern uh uh among other verticals. And also at the same time, uh more infrastructure that that helps us, you know, programming languages, frameworks, et cetera, that help us um do this. So so I don't know, maybe VJ, if we could start off. How do you how do you view this?
1:13 Speaker 1 Yeah, I mean, I think this is the key issue, right? Because everyone can program the standard C paradigm on one processor core. Now actually, you know, going to multi-core is not so hard, but start thinking about multiple boxes, thousand boxes, ten thousand boxes, this will drive you insane, right? Uh we can't be spending our time programming thinking about each of these things. We need to have some type of abstraction. And that seems to be the key. That's how we've been successful in general with coding. And Hadoop uh and its successors are great abstractions for certain things, but you can't do map reduce with everything. And I think that's gonna be the challenge. And I think what we're gonna see is verticals moving in in certain directions that can do different things. It's hard to imagine this magic language that solves all this problems. And so picking things that can sort of attack certain key domains actually could have a huge impact.
1:56 Speaker 0 Yeah, I mean, I'd agree. I think uh Vidji, when you say you can't use MapReduce for everything, that that suggests an even deeper issue, which is a lot of the approaches people have today for scale and for parallelizability, they really boil down to problems that are quite easy to scale because they naturally have a simple abstraction, a way of splitting them up. I think the next few years are gonna be about attacking problems which are naturally harder to split, harder to scale. Um and how we go about that. I mean, a whole other layer of reliability is going to be required as well for computations that are more challenging to split up. And that suggests to me that you know, even in terms of the cloud infrastructure that's available, there may be completely different uh characteristics that are necessary. So I think there'll be lots of winners and losers in this space that no one can yet predict.
2:35 Speaker 1 And the thing to emphasize here is that the possible win is huge. The difference between what you could do on one box versus 100,000 boxes. I mean, that's not just like I'm being impatient. That's being transformative.
2:46 Speaker 2 There's a couple things happening, right? One is, I mean, they always say Moore's Laws ending. People are saying that now. I mean, but then again, the people said that every year for the last 50 years. So to the extent that maybe it's slowing down or whatever, that the way you're gonna get additional sort of the effects of Moore of the Law, additional compute, is gonna be going across machines, not with more transistors on a on a single core, number one, right? So you need to. Number two, as you said, like it's a different, like it's not just like you're doing like three times as much. If you're doing a hundred thousand times as much, it unlocks a whole new class of potential applications. Yeah, exactly. So, like from your own area, like I don't know if you could like so in biology, for example. Yeah,
3:19 Speaker 1 in biology, you know, we launched folding at home uh in October of
3:22 Speaker 2 2000 now, so we're coming up to the code. Can you tell us people what that is? Yeah,
3:25 Speaker 1 so folding at home is a large-scale distributed computing project where people uh go to our website, uh folding.sanford.edu, they download the software. And right now, um, we have about 40 petaflops worth of performance out of maybe about you know, maybe 400,000 processors. It's actually interesting if you think. This was
3:41 Speaker 2 inspired by SETI at home.
3:42 Speaker 1 Yeah, inspired by City at home, which also is inspired by GIMPs, and there's a few other ones. I think we were the first to do sort of something in science. I think uh, you know, debate whether finding aliens of science or not. It's uh um but uh you know, something where in biology
3:55 Speaker 2 nonfiction science. Yeah.
3:57 Speaker 1 In biology, you know, the challenge was we wanted to tackle problems that would take, let's say, a million CPU days to do. You know,
4:04 Speaker 2 now would it would you approach did you still need to do that kind of approach today, or could you do it on on a in a cloud computing information?
4:11 Speaker 1 Yeah, I think you could do it on cloud computing, but you think about it, I think Amazon has roughly maybe 300,000 boxes. So if we want to buy all of Amazon,
4:19 Speaker 2 that would be a little pricey.
4:21 Speaker 1 Okay. Yeah. And but you know, you think about what we do with folding at home, it's kind of like a time machine because what we did 10 years ago, people can now do on GPUs. What we are doing now with 10,000 GPUs, people will probably do in the future with maybe a small GPU cluster. But the the paradigm, the programming paradigm, the way you think about it is all the same. And therefore we are taking advantage of Moore's law as time goes on. So
4:41 Speaker 2 what and so how does how does the the future, let's say the future you can actually have access to a million boxes or something like what is that, what does it mean for the applications in biology and healthcare?
4:50 Speaker 1 Yeah, I think there's a couple different ways to think about it. I think there's sort of processing lots of data and then doing calculations, either for us we do a lot of simulation, but I think it's uh people usually think about the data side, but the simulation side's actually pretty intriguing too.
5:02 Speaker 0 It's interesting. I think uh, you know, all that power, it unlocks so many more problems. I mean, how do I test my distributed application? How do I even run uh you know sensible diagnostics on it? Plus, not to mention the skills shortage and people that can really build distributed applications well. Um thinking thinking about that alone, I think will start to create a whole movement of people uh where the this skill set becomes incredibly valuable. Um, even in terms of languages and tools, we're not well served today with good abstractions to think about distributed systems. Most of the ideas that are being used now, like uh active paradigms, for example, which some people may be familiar with. I mean, these are from the 80s, right? Nothing's really changed in how we attack uh distributed systems. So it should be fun to see that revolution. Is there
5:41 Speaker 2 anything promising you see um in terms of you know languages, frameworks, infrastructure, software? I
5:47 Speaker 1 think right now everyone is sort of rolling their own for their domain. I mean, that's true for us. And you know, there's these problems that we all have that are hard to handle sort of in a generic way. Like you have to deal with fault tolerance. You got a million boxes or even just 10,000 boxes. Most likely one of them will die or have some problem along the way. And MPI, which has been the standard in supercomputing HPC, is very fault uh intolerant. You know, the whole job will crash and things like that. And so those paradigms really have to change. And I think you know, the pre prominence of people, uh companies like Google, which have all this on the back end, I think have gotten people thinking about this and map reduce and things like that. Those abstractions have played a huge role. I really am looking forward to seeing where people will go. And I think Hadoop and Spark are a good example, but I think we need much more.
6:31 Speaker 0 Yeah, I mean, these problems are profoundly different from scaling web services. And I I think another interesting point is that we traditionally assume large tech companies are gonna have a hegemony over large compute problems. But with this new space, I wonder whether existing infrastructure really will be that useful. So one
6:46 Speaker 2 pattern we've noticed is that whereas uh industry, meaning probably Google and maybe Facebook led data center innovation and Amazon, sorry, over the last 15 years, uh, we're seeing more and more academic-led stuff. So Spark is an example came out of Berkeley. Um, a lot of interesting stuff at Stanford, Berkeley, MIT, kind of the usual suspects. And I think that the theory we have is that that's because you kind of, you know, industry is very good at a kind of a depth search, right? Like kind of continuing to iterate on something. Um but when you need to go back and fundamentally rethink how you do something, that's probably better done in academia.
7:19 Speaker 0 Completely. But I guess industry always needs it needs motivational problems, right? And while uh the issues within biology are profound academic and potentially commercial importance. I think if you want to get the massive hackers and developers out there behind something, uh we need to start seeing some interesting problems that are kind of solvable, but through interesting innovation are gonna result in new new companies. And I think there's tons of stuff out there, be it in gaming or wherever. I
7:42 Speaker 1 think also there's interesting intersections between academia and industry. At Stanford, there's this pervasive parallelism lab, which brings together um companies in the valley, mostly big companies, but I think uh startups could certainly play a role there too. Because I think academics are interested in these questions, but it's useful to have some grounding for where are the big problems that really need to have the biggest impact right now.
8:01 Speaker 0 Completely. And we see a lot of academics that we speak to in kind of Cambridge Oxford using supercomputer methods right now, unaware that actually a distributed systems approach might be might be more cost-effective or even easier to
8:10 Speaker 2 think about from that perspective. So uh so Herman, you're so your company improbable builds simulations. Can can you talk about you know why are simulations important?
8:18 Speaker 0 Sure, absolutely. Well, I mean, I guess there are many paths to knowledge, right? And one of the ones that people are very familiar with now, uh uh I guess big data collecting huge amounts of information about the world and running pattern analysis on top of it. And another approach, which we're passionate about is completely recreating a phenomenon from the real world. I mean, I guess this is something Vijay would be familiar with from a biological perspective. But imagine being able to model cities, model power grids, model telecoms networks, actually achieving any of that, though, you know, involves solving some of the distributed systems problems we just talked about, and also being able to think about simulation in a totally new. And why
8:50 Speaker 2 would someone want to model us?
8:51 Speaker 0 I mean,
8:51 Speaker 2 uh,
8:51 Speaker 0 sure. So you can you can answer questions, right? Answer those water of questions. What happens if you know a disease is released in this in this crowd? What happens if we shut down this tube station? Questions which governments and companies want to answer, but which you know you can't answer just looking at data, and particularly when you're considering situations that have never happened or trying to project and understand.
9:10 Speaker 2 Yeah, but
9:10 Speaker 0 uh, but I think the problem's even deeper than that, right? Like the problem, the big problem is complexity, you know, stepping away even from technology. The problem is how do we make choices when our world is full of so many interrelated complex systems that no one person can actually hold in their head. And that's where simulation comes into its own, right?
9:26 Speaker 2 It's interesting. Like today, like so one so I'll tell you my pet theory, which is sort of in the same way that so if you go back to like the 80s, machine learning was kind of this rebel enclave of AI, right? Like the the mainstream AI thought you could do use rule-based systems and and this this rebel enclave was like, no, no, no, you need to create you know statistics and have machines that learn. Uh and and now it turns out, of course, that the enclave became the dominant um the the dominant movement. In fact, AI and machine learning are probably basically synonymous today. Today, um, simulations and agent-based kind of reasoning, it's like just kind of like Santa Fe Institute and all these kinds of things. Kinds of quote eccentric um thinkers, the mainstream, if you look at the social sciences, all the mainstream uh kind of thought leaders use analytic approximations. If you look at like macroecon economics, for example, they use you know, whatever they have their set of equations and they have a very, very poor track record in predicting the future.
10:15 Speaker 2 There's these rebel enclaves of agent-based thinking who haven't actually been able to really run their. I mean, so my my own pet theory is a little bit like machine learning
10:23 Speaker 1 in the
10:24 Speaker 2 80s or something, which is until
10:25 Speaker 1 machine
10:26 Speaker 2 learning
10:26 Speaker 1 couldn't
10:26 Speaker 2 happen for real until you had the infrastructure,
10:28 Speaker 1 right? You
10:28 Speaker 2 needed to have massive amounts of data. You couldn't do the kind of things like for just take Google Translate an example, like rule-based systems were better than statistical systems until you had the ability to scan, you know, corpora of millions of books or something, right? Trevor Burrus, I mean,
10:41 Speaker 0 it's particularly important when you consider that the times you're going to want to run simulation, you're often dealing with phenomena that have emergent complexity. The cool stuff only happens at scale, right? So I mean, for example, we're dealing with a group called the Institute for New Economic Thinking at Oxford, and these are some amazing scientists and they want to model the UK housing economy. Now, with 10,000, 20,000 houses, there's only so much, and actors, there's only so much you're going to be able to deduce, right? I mean, I wonder if you actually are there some things in your domain where this emergent complexity property becomes. I think
11:08 Speaker 1 those are the things that we're most excited about because I think uh, you know, analogous to what Chris was talking about, there's tons of um pencil paper analytic work that's done in physics and chemistry, but it's reaching its limits. The approximations you have to make really sort of take out a lot of the things that are the hope for what would be interesting and complicated. So we turn to simulations to give us uh in new insights that we couldn't get from other things. Aaron Powell
11:27 Speaker 0 If anyone listening to this wants to get an example of emerging complexity, I recommend Googling just Conway's game of life. Sure. It's this little cell-based automata that you can play with, but you see so much beauty emerging from such simple rules. Very simple. Yeah, very simple rules.
11:39 Speaker 1 Yeah. So I think actually in physics and chemistry, this is simulation is is a dominant paradigm. It is actually really intriguing to imagine taking this to social areas, social science. Yeah.
11:48 Speaker 2 And what and so if you had so if if you if today you had uh you know your fantasy simulation scenario with like, say, if for example, you know, to model a cell or something. And you explain like how that would work and what what kind of uh uh questions you might be able to answer.
12:01 Speaker 1 Well you know, the cell is interesting because a cell is more like New York City or you know, then like uh a sort of uh dirt path or something like that. There's a lot going on. And it's that complexity that leads to all these emergent properties. And so uh the hope about simulating a cell is that we'd be able to sort of gain some understanding that you couldn't get from just doing the experiment alone. Because if you could just do the experiment, then that would be fine, but there's just so much going on, it's hard to really sort of capture everything. And so I think there's been a lot of excitement in cellular biophysics and cellular biology on the disease side because we think diseeds are sort of systemic problems, not having to do with any one single point. And I think you could imagine these systemic things are sort of what's interesting in the graph, and it's not just simulating a cell and looking at the systemic properties would be simulating a city, where maybe shutting down one bridge has effects sort of all over or and making small changes even could have major effects.
12:47 Speaker 1 And it's those counter to the things are the things that I get excited about because the kind of the things that are obvious the simulation could verify that's not very interesting. It's the discoveries of things that you would never think would be connected, are I think where the real um excitement is. And that's where we've had the biggest wins. Aaron Powell
13:02 Speaker 0 I think it scares people a little bit, um, particularly in some domains like economics. I mean, analytical methods, they they come with a certain degree of certainty and trust, right? Like I can I can exactly explain to you why how and why this works. But emergent complexity is unpredictable. It's scary. The results may not be what you what you expect, and it has the potential to upset a lot of preconceived notions about what's possible.
13:21 Speaker 1 The other aspect that I think uh is interesting is this concept of A B testing that you can't do this in real life as easily. You know, you can't shut down this bridge during this time and see what would happen. And so the ability to do this type of A B testing and optimizing things before you bring into the real world is actually also exciting. Uh I I think finally, I think one of the things that we've seen in terms of uh these types of areas is that it always starts with heuristics. Um when people built bridges, people built bridges in Roman times. You know, they didn't have simulations of bridges or F equals MA or anything like that. But now with the Bay Bridge, you know, that you know, multi-billion dollar thing, you wouldn't just do that empirically. And so I think as the simulations and um sort of analytics become better and better, we don't have to use our best guesses, which is what heuristics are. We can actually really just see what's going to happen. Aaron Powell
14:05 Speaker 0 So there's a common preconception that I wonder if Vidj or maybe you Chris can attack. And it's an interesting idea that you often hear thinking about simulation, which is how do you know the model is right? And could your simulation be useful unless your model is 100% accurate? I mean, what do you think of that, J? Aaron Powell,
14:18 Speaker 1 yeah. I think you know there's a couple of things. One is that there's waves of testing on back data to see if you're right. But you're you're correct that actually the simulation doesn't have to be perfect to be provocative. A, there's the you know, the ideal is something where it's perfect and it gives quantitative predictions, whether of the stock market or of traffic or something like that. But a lot of times things are useful even if it just gives you an idea or a hypothesis or a new insight that you wouldn't have gotten um just by thinking about it or by doing pencil, paper, math. And then that hypothesis can be tested in other ways. Aaron Powell
14:46 Speaker 2 So Herman, um, so some of your customers are are using improbable to simulate cities. Can you talk about like how that might work? Sure, awesome.
14:53 Speaker 0 Um to be quite concrete and to give them a little mention. Uh Matthew Ives at Oxford and the ITRC group um are very interested in modeling uh large city infrastructure. So from their perspective, they see cities as interconnected uh layers of infrastructure where each layer actually isn't as significant as the sum total of all the layers working together in interesting ways. Now, again, the limiting factors and being able to see that emergent complexity and to be able to book them is enough scale and enough detail. So, you know, what we're hoping to do is provide a platform almost an OS where they can build those sorts of simulations. But actually, I think the cool things will happen when we can go a little further than that. Uh not simply creating standalone models, but actually instrumenting real cities. Imagine a simulation powered by sensor data, you know, from millions of sensors placed around a city that actually lets you. So
15:38 Speaker 2 a car in the simulation actually corresponds to a c to a car with a Internet of Things device sitting on it. Um so maybe half the half the entities in the simulation are actually real world entities and half could be completely and modeled or something.
15:51 Speaker 0 And for every event, there are knock-on consequences. If there's a tra if there's an accident or a traffic jam, you know, it's possible to then extrapolate from that potential other outcomes and scenarios that would be of interest to a wide variety of people. I mean, I guess that out of improbable, we don't really believe that a simulation should be this like standalone box where knowledge comes out of. We see it as an operating platform, somewhere that you can actually make decisions and build applications that consume that simulation. That's also something that's been missing. I mean, the community, I think, is very much, Vijay, correct me if I'm wrong here, I might be jumping out of my purview, but the community is very much inspired from uh supercomputer research, which was always about putting in data and getting an answer out. Thinking of simulations in this more like almost web 2.0 app way as quite flexible constructs is a little alien to people in the space at the moment. Aaron Powell And
16:32 Speaker 1 I think we're gonna see more of this because the desire to have one place where you can integrate uh simulation predictions and sort of experimental data, whether that's IoT experiments or whatever, that becomes very powerful because imagine having a million IoT devices. How do you even like understand what's going on and how do you put that together and how do you get a picture of what it means? Aaron Powell
16:51 Speaker 0 Yeah, exactly. I mean, it's quite weird. A lot of customers we've spoken to, they just want to visualize the current state of uh a large complex system, let alone simulating it. Turns out to be quite a hard challenge.
17:01 Speaker 1 And then you know, there are dark aspects you can't see, and if simulation can fill in the gaps between that, then suddenly you have a full picture. Trevor Burrus, Jr. Completely. I
17:08 Speaker 0 mean, there are undoubtedly amazing companies out there that have built massive uh distributed computing infrastructures that live on their own proprietary hardware. The question though is as we start to think about uh the kinds of applications that Chris and Vijay are talking about, where problems are not so easy to paralyze, how useful those uh software infrastructures are going to be in the future. I mean, I think ultimately the companies and solutions that can start dominating the space are gonna have to redo a lot of stuff at quite low less in order to make it effective. So there may be uh a need to throw out some of the work that's gone before. Aaron Powell
17:37 Speaker 2 So you know, one just on the business side of simulations, um, one area that people have been interested in is that um increasingly there's pressure on companies, particularly around sort of core infrastructure like financial services, um uh public safety, et cetera, to do serious disaster uh recovery planning. And that includes both in the case of so, say, cyber attacks, also physical like terrorist attacks. So, you know, if there is a terrorist attack, God forbid, you know, against some financial institutions or something, you know, some we want to make sure that the system as a whole is robust and survives that. And so there's lots and lots of thought being put into this concept of sort of disaster planning. And it seems like an area where simulations can be quite useful. Aaron Powell Yeah.
18:22 Speaker 0 I mean, I think even conceptually, simulations tend to allow you to consider the vulnerabilities in your infrastructure a little bit more objectively than you would if you were being totally analytical, because it doesn't require a human being to, you know, maybe focus on what vulnerabilities seem most obvious. For example, when we look at cascading failures in power grids, which is an area that you know we've explored it with improbable, how those failures arise can often be the accumulation of many, many disparate, like seemingly irrelevant events and and slight vulnerabilities which add up together to cause a big catastrophe. Again, I think this might even relate perhaps to some of the things.
18:57 Speaker 2 Exactly. Exactly. It's like if you look at like as an example, like um the air airplane industry has air travel has gotten much safer over time. Unfortunately, they have a number, they had a number of crashes in order to learn from. Yeah. Um, you know, which of course is tragic, but is also from a disaster recovery planning point of view a positive because they had many data points, right? When you talk about things like massive terrorist attacks, you have one or two data points. So you have no historical pattern from which to train from.
19:24 Speaker 1 Even like a nine or or greater earthquake, let's say in the Bay Area, what do you tell people to do? First, to be really powerful to be able to simulate different possibilities. The second one is in the moment, having IoT like information for what people are doing to be able to feed into making predictions for where to go from here. Trevor Burrus, Jr.
19:41 Speaker 0 Instant decision making. Yeah,
19:42 Speaker 1 that that combination of that you know, you can imagine the team has seen this a hundred times because the simulations have been running over the last like two years. And then in the moment, they're sort of ready for the game plan. They're getting information from IoT and they're sort of in real time deciding what to do based on what they've already run or what the simulation would even predict would be the best thing.
20:00 Speaker 0 Or even considering how a situation like a riot or civil unrest or uh a problem might evolve over time, given its current situation and given the mechanics of that group of people that the simulation is able to model and explore. I mean, these are all things that people don't even dream of doing today. I mean, and to make them possible and usable over so many different domains, I think is an immense challenge. Trevor Burrus,
20:19 Speaker 1 Jr.: And it's not something where you can just have a bunch of guys and say, oh, I think this is what we should do. I mean, uh to be data-driven and to be uh to really use the data on the ground in a way that no human could wrap their head around could really be something f fantastic and have could be the difference between life or death for many people. Aaron Powell
20:32 Speaker 0 And it's another area where you can't just simulate one thing, right? It's not just about the you know the physical effects of the earthquake on building integrity, it's about everything together. Exactly. It's about you know social conditions, it's about weather. Aaron Powell
20:47 Speaker 1 Exactly.
20:47 Speaker 0 And it's it's often the little details that slowly accumulate and add up and make a simulation uh meaningful. I mean, I'm always reminded of the, I don't know how how particularly relevant this would be. I'm always reminded of the prisoner dilemma simulations that have taken place uh a long time ago, very simple simulations. But as you add more detail, the results become completely different. Um you know, when you when you just run uh prisoner's dilemma style games between uh participants, okay, you get one outcome. But when you start to introduce geographic components to those simulations, suddenly it all changes again. I mean, that that's why we need a system, something that'll let people introduce detail at you know, pretty much arbitrary uh scales in order to really get more and more accurate models. Aaron Powell
21:22 Speaker 2 Okay. Thanks, guys.
Previous Company Building in Crypto: A Conversation between Brian Armstrong and Chris Dixon Next What's Next for Education Startups Page updated August 25, 2026 · transcript and summary are machine-generated