Pod Insights Masters of Scale A masterclass in crowdsourcing, w/Duolingo's Luis von Ahn About this episode Crowdsourcing is more than a group of people interested in the same cause, it’s a way to tap skills – and scale – that you don’t have. And when it works, it can be the rocket fuel that launches you to scale further than you could have ever imagined. No one knows this better than Luis von Ahn, founder and CEO of Duolingo. Duolingo is a language app with over 300 million users worldwide, who complete over 7 billion exercises a month. It’s their passion for learning that drives them to create even more content for the app – igniting the rocket fuel that Duolingo needs. Cameo appearance by Elizabeth Sampat (game designer and author).
Read a transcript of this interview at: https://mastersofscale.com/luis-von-ahn/
Subscribe to the Masters of Scale weekly newsletter: https://mastersofscale.com/subscribe
See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.
Transcript 0:01 Speaker 00 We've all spent money on something that seemed incredible but didn't live up to the hype. A kitchen gadget, a car. When it doesn't deliver, the disappointment is real. Marketers suffer from this too. Chasing metrics that look great but don't translate into revenue. LinkedIn calls that bull spend. Invest in what matters and reach the right buyers with LinkedIn Ads. You can target by company, industry, job title, and more. LinkedIn Ads also generates the highest ROAS of all major ad networks. Spend $250 on your first campaign on LinkedIn Ads and get a $250 credit for the next one. Just go to LinkedIn.com/Scale. That's LinkedIn.com/Scale.
0:44 Speaker 00 Terms and conditions apply. The very best founders I know are brilliant at building systems. They connect teams, they remove bottlenecks, and they eliminate single points of failure. And yet, when it comes to their own wealth, most are running a disconnected stack. A tax accountant here and a state attorney there, a wealth manager who doesn't talk to either one of them. Creative Planning was built to fix exactly that. One integrated team of tax professionals, state planners, investment specialists, all coordinated by a dedicated wealth manager who sees your full financial picture and keeps every piece working together.
1:27 Speaker 00 Proactive tax efficiency, estate strategy, investments, all under one roof. Creative Planning, where wealth works together. Learn more at creativeplanning.com/mastersofscale.
1:41 Speaker 01 AI is full of possibilities. Enterprises run on realities. That's why Pega combines the world's most powerful AI with the governance, transparency, and control enterprises demand. So you can move faster, adapt faster, and scale AI with confidence. All the power of AI, none of the uncertainty. Learn more at Pega.com/AI.
2:10 Speaker 02 I was gonna make a gym.
2:15 Speaker 02 That was going to be free.
2:18 Speaker 02 Anybody could go there, imagine, it's a free gym. It's amazing.
2:22 Speaker 03 That's Luis Von On, founder of language learning app Duolingo, describing a game-changing idea he had at the age of 12 to disrupt the fitness world.
2:33 Speaker 02 I thought this was a unique idea that was going to completely revolutionize the world.
2:38 Speaker 03 What was it that made this idea so revolutionary? It would harness the power of gym goers to generate electricity.
2:50 Speaker 02 We're gonna connect all the exercise equipment to the power grid, and we're gonna sell that electricity to the power company. And that's how we were gonna make money.
2:59 Speaker 03 Young Luis feverishly doodled weird and wonderful designs for his free gym. Exercise bikes hooked up to the grid. Treadmills wired to power converters.
3:11 Speaker 03 Bodybuilders could pump out the electrons as they pumped iron.
3:17 Speaker 03 And society could burn calories instead of fossil fuels. But Luis's idea hit a snag.
3:27 Speaker 02 It turns out, this is actually not a very good idea. Humans are not very good at generating electricity this way.
3:33 Speaker 03 That's right. Even the most dedicated gym rat can't generate enough electricity to make the scheme worthwhile. There was also another flaw.
3:43 Speaker 02 I've now realized, a lot of gyms make a lot of their money from people signing up and never showing up.
3:48 Speaker 03 And this is where young Luis's naivete shows through.
3:53 Speaker 03 Traditional gyms are placing a bet that most of their customers will lose their drive. Whereas Luis wanted his gym's motivation to be aligned with his customers. Luis isn't the only person who fell under the electrifying spell of the power pumping gym.
4:09 Speaker 02 Who knows how many thousands of people have had the same idea?
4:12 Speaker 03 Yeah, and actually, I'm one more of those thousands to millions of people in my childhood.
4:19 Speaker 03 And just as you say, you learn that actually the math doesn't work.
4:23 Speaker 02 Right.
4:26 Speaker 03 It's not just unworkable math that makes the gym idea an ineffective use of crowdsourcing. It's the lack of a shared mission bonding the crowd and the gym operator. The crowd is attracted by free gym time, but they're not interested in the fruit of their efforts, the free electricity. They have no shared pride in it, no commitment, and no reason to commit to keeping the lights on and the energy flowing.
4:57 Speaker 03 That's why I believe crowdsourcing can scale your business in unexpected ways, as long as you perfectly align your mission with your crowd's motivation.
5:11 Speaker 04 You've got to have incredible talent at every position.
5:15 Speaker 03 There are fires burning when you're going home. Can you believe it?
5:18 Speaker 05 Such an idiot. And then you go back to, this is totally going to be amazing. There are so many easy ways. So so I have no idea what to do. Sorry, we made a mistake. But you have to time it right. Oops. Working as a free bed compartment. Stuff that just seems absolutely nutballs. 10 years later, we're like, well, that's just how you do it. We haven't made it. Just how you do it.
5:41 Speaker 03 This is Masters of Scale.
5:48 Speaker 03 I'm Reid Hoffman, co-founder of LinkedIn, partner at Greylock, and your host. And I believe crowdsourcing can scale your business in unexpected ways, as long as you perfectly align your mission with your crowd's motivation. To picture what I mean, I want to remind you of a classic scene from the movie Raiders of the Lost Ark.
6:14 Speaker 03 In this scene, rakish archaeologist Indiana Jones descends into an ancient room under the Egyptian desert. Inside, there's a massive model of a long-buried city. Indy has with him an ancient artifact called the Staff of Ra. It's basically a tall pole with a bronze medallion on the top.
6:36 Speaker 03 He places it at a precise position on the map, at just the right time of day. Rays from the sun burst into the room.
6:45 Speaker 03 Pass through the medallion staff and send a ray of light directly to the spot on the map where they'd find the Ark of the Covenant.
7:02 Speaker 03 Everything lined up, and this is the kind of alignment you need to get crowdsourcing right. Crowdsourcing is a way to tap skills and scale that you don't have in-house. When it works, it opens extraordinary opportunities. It lets you scale faster and farther than you ever could on your own. But if you get the alignment wrong, you can end up with anything from a weak fizzle to a fiery disaster. I wanted to talk to Luis Von On about this because his company, Duolingo, is one of the most fascinating examples of crowdsourcing gone right. Duolingo is the app that lets you learn language in bite-sized lessons motivated by achievements and prompts.
7:43 Speaker 03 It has over 300 million users who complete over 7 billion exercises a month, and much of the course content is created by its passionate users. That's why alongside the likes of Spanish, Chinese, and Arabic, you can find more esoteric languages like Esperanto, Navajo, and even Klingon. Luis's story is a masterclass in working with your audience, and he also knows firsthand how learning a language can change your life.
8:13 Speaker 02 I was born in Guatemala, and I grew up there. And I came to college here in the US. But to come to college in the US, you have to take an English proficiency test. The most common one is called a TOEFL. It stands for Test of English as a Foreign Language. I had to take this test, and I had the problem that the country of Guatemala ran out of seats for this test. And I freaked out. I thought I couldn't apply to college anymore. Fortunately, I had enough means to be able to fly to El Salvador to take that test. Guatemala is kind of a dangerous country. At the time, El Salvador was an even more dangerous country. So I had to go there. I had to take the test.
8:49 Speaker 03 Flying into a war zone just to prove he can speak a language? Luis thought, there has to be a better way.
8:57 Speaker 02 And at the time, I thought, I'm going to do something to kill the TOEFL.
9:01 Speaker 03 On arriving in the US, Luis put aside his desire to kill the TOEFL and focused on another ambition, one he and I shared.
9:09 Speaker 02 I wanted to become a math professor. This is what I wanted to do. I read somewhere that you also wanted to become a professor.
9:16 Speaker 03 Philosophy, a little less disciplined than the math professor. So Luis left his home in Guatemala to study at Duke University. Today, he's a consulting professor of computer science at Pittsburgh's Carnegie Mellon University, or CMU. And it was at CMU when Luis was completing his PhD that he caught an impactful lecture by Yahoo's chief scientist, Udi Manber.
9:40 Speaker 02 And he came and gave a talk of 10 problems that they didn't know how to solve at Yahoo. One of the problems really captivated my attention. So Yahoo was one of the first few places that would offer free email accounts and spammers would write programs to obtain millions of email accounts for free because they wanted to send spam from Yahoo accounts.
9:59 Speaker 03 It was the year 2000 and Yahoo was desperate to stop spam bots from opening free accounts.
10:06 Speaker 02 And then with my PhD advisor, a guy named Manuel Blum, who's amazing, we came up with a potential solution, which was these distorted characters that you have to type whenever you're getting an account. It's this thing called a captcha.
10:19 Speaker 03 We've all seen captchas. When you're signing up for something online, you're presented with a set of random letters or numbers bent out of shape as if they're melting. And you retype them in a box. With a bit of squinting and head tilting, it's not too hard for us humans. But at the time, it was impossible for machines. And this was the point. "Bots beware, none shall pass."
10:44 Speaker 02 We built the first prototype and we actually gave it to Udi. And he put it in Yahoo within a week. And I also now understand that for a very large company to do something in one week, it means it was a serious problem for them. So yeah, so from then on, all kinds of websites started using it and it was pretty amazing.
11:01 Speaker 03 Here's something else that's pretty amazing. Luis and Manuel gave away the captcha technology for free. That's right, they let one of the world's biggest internet companies have a tech they were desperate for with no payment. Why? It comes back to that talk by Udi, Yahoo's chief scientist.
11:22 Speaker 02 I now understand that this was a recruiting talk. Back then, I did not realize that. I now understand the reason he came to CMU was, "Yeah, I'll just tell them some stuff that's exciting, and then, you know, maybe they'll come work for Yahoo."
11:34 Speaker 03 Udi knew that the way to recruit engineers wasn't to talk about how great it was to work for Yahoo. Instead, he presented these engineers with the one thing they couldn't resist: an unsolvable problem. Udi wasn't just recruiting in a very clever manner. He was also crowdsourcing for Yahoo. And his precise alignment between engineers' motivation and Yahoo's mission is one you should keep in mind. Yahoo ran with Luis's solution, and pretty soon, CAPTCHAs were springing up everywhere. Want to sign up for an online account? CAPTCHA. Join a mailing list? CAPTCHA. Post to a forum? CAPTCHA, CAPTCHA, CAPTCHA! Charlie. Alpha. Papa. Dango. Charlie. Hotel. Alpha. CAPTCHA.
12:18 Speaker 06 If you want to tell somebody is human or not, you need to challenge response test and put a request to type words obscured and wavy.
12:29 Speaker 06 The test fully automated, both reviled and celebrated. Alphabet, yeah, it's gonna get you. You're a BOT. You can take from me. It's like a digital lock and key. They call CAPTCHA. CAPTCHA. Am I a robot? I'm a human being. And I know that you know what CAPTCHA means. It's completely automated, public Turing test to tell computers and the humans apart.
12:59 Speaker 03 In terms of sheer numbers, it was a success.
13:02 Speaker 02 I did a little back-of-the-envelope calculation of the number of times somebody would type a CAPTCHA online. And the number I came up with was about 200 million times a day. At first, I was pretty proud of myself. I thought, "Look at the impact that this has had." But then I started feeling bad, because each time you type one of these, you waste about 10 seconds of your time. And 200 million times 10 seconds is half a million hours a day. So I started feeling bad about that. And not only that, people kind of hate doing these things.
13:30 Speaker 03 Luis started feeling responsible for millions of hours of lost productivity. But I think he's being far too hard on himself. The CAPTCHA did solve a problem. And a few seconds of annoyance is a price most of us are willing to pay for a life of a lot less spam. So the captcha was a successful, if simple form of crowdsourcing. It's a decent alignment of crowd motivation and company mission. For a 10-second task, it works. I think what really irked Luis was that there was a missed opportunity here, an inefficiency. And from speaking to Luis, I could tell he loathes inefficiency.
14:10 Speaker 03 Follow me on a little tangent while I share Luis's response to one of our lightning round questions. The one thing you wish your phone could do.
14:20 Speaker 02 You know, I started getting a lot of these smart home things. And I have like a million base stations plugged into other stuff. And there's a million things and it doesn't work super well. I wish my phone could just make that work. I mean, it should know where I am, turn on the lights right there. I shouldn't have to say, okay, Google, turn on the lights. I wish this was just farther along. I'm a big fan of this, but it's still pretty annoying. And people kind of make fun of me because, you know, they're like, you could just tap that switch. I'm like, no, no, no, no, listen to this. Okay, Google, turn the lights red and then it works. But I wish my phone could just be better at that.
14:54 Speaker 03 It's the strong aversion to inefficiency that got Luis thinking about how to iterate on the captcha.
15:00 Speaker 02 So I thought, can we make that time count for something useful? During those 10 seconds, the human brain that was solving this was doing something that computers could not do. So it was 10 seconds of very valuable work in some sense.
15:12 Speaker 03 Luis started looking around for problems that could be solved with 10 seconds of human brain power. It was around this time that Google embarked on its quest to digitize every book in existence. And they'd hit a snag.
15:27 Speaker 02 At the time, for older books, things that were written before 1980, the computer could not recognize about 25 to 30% of the words. Luckily, the computer knew it couldn't recognize the word.
15:37 Speaker 03 Luis thought to himself, well, humans can recognize those words. What if we could use those 10 seconds of human brain power to help digitize old books?
15:47 Speaker 02 We could take all of those words and get people to read them for us while they're typing a captcha on the internet. So that was the idea with recaptcha. And it turned out it worked.
15:55 Speaker 03 Luis called this new version Recaptcha. And it was about this time that a lot of websites asked to use it. Some of them are websites you'll have heard of.
16:06 Speaker 02 The first one was not all that big, but actually for our size, it was pretty big. A website was called onlinebootycall.com.
16:12 Speaker 03 Well, okay, you'll definitely have heard of the next one.
16:15 Speaker 02 Pretty soon after that, a website that was starting to get big, they contacted us. They were like, hey, we're implementing a captcha service. We looked around, this seems to be the best one. Can you handle high traffic? And of course, we just said, sure, we can. We didn't realize how high it was.
16:30 Speaker 03 That company?
16:31 Speaker 02 It was Facebook.
16:33 Speaker 03 Other big sites like CNN and BuzzFeed signed up for Recaptcha's translation service, and they were willing to pay. In 2009, Google bought Recaptcha, and it's still harnessing its crowdsourcing power today.
16:47 Speaker 02 They're using it to improve things like the street view, the recognition of the addresses. So basically, anything that computers are not super good at, they kind of turn into a captcha to get a lot of training data.
16:59 Speaker 03 I asked Luis about how the captcha experience helped evolve his thinking about crowdsourcing. Did it occur to you this was like your gym idea? Where it was kind of?
17:09 Speaker 02 Not at first, but eventually I'm like, oh, this is just like that gym. It's pretty similar.
17:13 Speaker 03 Yes.
17:14 Speaker 02 Yeah.
17:14 Speaker 03 And obviously, as we get to Duolingo, it'll be a pattern. But what are the kind of the ways that you put this toolset together where you say, oh, actually, in fact, there's ways that groups of humans can do this in collaboration with computers that make all of this ecosystem better? What's the kind of questions you ask or the ideas that occur to you in solving these problems that make you one of the very unique people of creating these ecosystems?
17:41 Speaker 02 Yeah, I mean, one of the hardest things is finding a problem that is kind of Goldilocks problem in that computers are not quite good at it, but humans are really good at it. And humans can do it pretty quickly, at least in all the systems that I've done like this. I mean, I did a number of these. It was things that humans could do in five seconds, 10 seconds.
18:00 Speaker 03 While Goldilocks wanted porridge that was just the right temperature, Luis wanted problems that were just the right complexity. They had to be too complex for computers, but easy enough for humans to do in a few seconds.
18:13 Speaker 02 And you can turn them into all kinds of different problems. I mean, a captcha is one where you can just, the incentive for the human to do it is because they want to get an email account or buy tickets or whatever. The incentive is just this intrinsic thing. It's a blocker and they have to do it.
18:26 Speaker 03 Those captchas only took 10 seconds to do, but even a 10-second blocker is frustrating. And frustrating your users is never good. There's another problem. With recaptcha, users were actively helping achieve something, arguably something for the greater good. But they may not have been aware of it. And when they found out, well, they weren't always thrilled. This is exactly the reaction one of our producers, Katie Clark Gray, had when she was working on this episode and realized for the first time that recaptchas were more than just an annoyance.
19:00 Speaker 06 I had no idea until we were working on this episode that all of those thousands of recaptchas I've done helped digitize books for Google. Now, I mean, I love books, but I feel very conflicted that I wasn't even aware that I was helping. And like, was it really a good thing? I mean, was it for Google's profit? Am I, am I going to get free access to those books that I helped translate for the computer? Am I going to get a credit? Will I even know which specific books that I helped with? I, I mean, I have questions.
19:28 Speaker 03 Though it's important to remember that one person's "ew" is another person's "cool", just listen to how our executive producer, June Cohen, reacted when she learned the hidden crowdsourcing of the recaptcha.
19:42 Speaker 05 This is so odd because Katie and I agree on most things. I carbon date to an earlier era in the web and in technology. I can actually remember the day I learned the actual scope behind the captcha project, that it wasn't just a clever way to make sure. That I was a human, but it was this grand crowdsourcing scheme that was allowing us to digitize old books. That is brilliant. Like, the fact that Luis Manon and his team had found a way to actually make effective use of this six seconds of my wasted time. I just thought, this is a masterwork in crowdsourcing. It had never occurred to me that you could put,
20:27 Speaker 05 like, people's greatest annoyances to work for the world's greatest challenges. I love it. This makes me hate these six seconds much less.
20:37 Speaker 03 This is something to remember with crowdsourcing. Not everyone will understand your mission. And misunderstanding can cause a buildup of bad sentiment that may poison your project. This is why you need to be clear with your contributors, especially when their efforts take more than a mere 10 seconds. And that brings us to Luis's next leap.
20:58 Speaker 02 I wanted to do something related to education that could help a country like Guatemala, which is why I thought of language learning. So I thought I would do something that would teach people English for free. And I didn't really know how to make money. The first idea, actually, it occurred to me, was something very similar to recapture or very similar to the gym, which was, okay, we're not going to charge people because I don't want to charge people because this is kind of supposed to help the world. So this was actually the first idea of Duolingo. When we launched it, people would be learning a language, but then after a few lessons, we would tell them, hey, you want to practice what you just learned? Here's a news article that is related to what you just learned. In the language that you're learning, can you translate it to your native language?
21:37 Speaker 03 In Duolingo, Luis had found a way to monetize the activity of his users in a way that furthered their mission rather than exploiting them. If you are crowdsourcing for a for-profit business, you need to demonstrate that your mission is the same as that of your contributors, and that the money you make benefits your shared mission. But there was still a problem for Duolingo.
21:59 Speaker 02 There was, again, a flaw, kind of similar to the gym flaw, just can't make that much money off of translation, it turns out. I think it's kind of a [bleep] business model, but that was the original business model for Duolingo.
22:10 Speaker 03 Luis's solution? Classic blitzscaling. Don't worry about the money right now.
22:16 Speaker 02 Over time, we pretty much realized, look, the main thing we're going to do is something to teach a language. We'll figure out how to make money later. And as time passed, we kept raising more and more venture capital. And our investors would tell us the prevailing wisdom at the time was, don't worry about making money, just grow.
22:32 Speaker 03 Fortunately for Luis and his investors, Duolingo's users were motivated by the shared mission, making it easy and free for anyone to start learning a language. They were exactly the passionate core you need if you're letting money take a backseat as you speed down the blitzscaling highway. This passionate core was also critical in the next vital development in Duolingo's business model.
23:00 Speaker 00 We all know the disappointment that comes when a grand gesture falls flat, something that seemed impressive in the moment, but didn't deliver the results you expected. Marketing budgets too often fall into that trap, aiming for big impressions, big reach, but without the revenue to back it. LinkedIn has a name for it: Bullspend. LinkedIn Ads generate the highest ROAS of all major ad networks. They target by company, industry, job title, and more. Spend $250 on your first campaign on LinkedIn Ads and get a $250 credit for the next one. Just go to LinkedIn.com/Scale. That's LinkedIn.com/Scale. Terms and conditions apply.
23:44 Speaker 00 When you've built substantial wealth through your business, it's often tied up in a single equity position. The upside is real, but so is the risk, and knowing when to act isn't always obvious. Creative Planning works with business owners to build a strategy around concentrated equity, when to diversify, how to manage tax risk, and how to protect what you've spent years building. Creative Planning: Where Wealth Works Together. Learn more at CreativePlanning.com/MastersOfScale. Stripe Treasury is your new financial operating system built for what's next. With Stripe Treasury, the place your business accepts payments is also where you can store balances, move funds, and pay vendors worldwide. If you're building for global scale,
24:29 Speaker 00 get a business bank account that lets your money move instantly, whenever and wherever you need it. Learn more at Stripe.com/Treasury. Stripe is not a bank. Banking services provided by Fifth Third Bank, National Association Member FDIC.
24:46 Speaker 03 Before the break, we heard how Luis van On found a Duolingo to help people learn languages for free. Duolingo had tried to make money by capturing the efforts of its language learners to offer a translation service. However, this turned out to be a nonstarter. The translation business was hard. But I would also say it was another, bigger problem. Although now the motivations of the company were in the same ballpark as those of its users, they were not totally aligned. You can be a passionate language learner without caring a jot about providing translation services. So Luis put aside this crowdsourcing idea for Duolingo.
25:27 Speaker 03 However, he was about to hit upon a more unexpected use of crowdsourcing, one that would see Duolingo's motivations perfectly aligned with those of its users. And that would push Duolingo to become the world's most popular language learning app.
25:43 Speaker 02 When we launched Duolingo, you could only learn Spanish and German, and I made the Spanish course. I'm a native Spanish speaker. And then my co-founder is from Switzerland, Swiss German, and he made the German course. And then we hired somebody to do a Portuguese course, and we hired somebody to do a French course.
26:01 Speaker 03 This handcrafted approach was an undeniable and unexpected success.
26:06 Speaker 02 Apple named us the iPhone app of the year. We didn't expect that, and it just kept growing and growing, and people started asking for a ton of languages. And literally every day, we would get requests, multiple requests for, can you add this language or that language? So we quickly realized there was no way we were going to be able to add all the languages that people wanted. And also, it didn't even make much business sense. I mean, adding a course to learn Esperanto, you know, just not going to get that many users.
26:33 Speaker 03 The people who reached out to Luis were deeply motivated to share the wonder of their native languages with other people. Luis thought back to the theme running through his previous ideas.
26:46 Speaker 02 It just occurred to me at some point, in the past, I worked on crowdsourcing. Maybe we can crowdsources. Maybe we can get people to actually help us add the course themselves. And so the first thing I did is, for one of these requests randomly, I just responded. I said, "Look, this is a very small language. Would you be able to help?" They responded pretty much immediately, saying, "I'd love to, and I have friends, and I'm willing to help."
27:09 Speaker 03 Note here how Luis uncovered a hugely valuable resource. In fact, it was the resource that would shoot Duolingo to massive scale. And he found it by reaching out to his passionate crowd of users.
27:24 Speaker 02 We realized, actually, we could get a good number of people helping us add languages to Duolingo. In part because it was a free way to learn languages. I mean, people really saw it. We were very mission-driven and people really saw that.
27:36 Speaker 03 It's the holy grail. Or should that be the Rosetta Stone of crowdsourcing? Routing people who share your mission to gather beneath your banner. To keep them there, you have to make sure you're directing their efforts towards something that engages them. Not something that has no relation to them. Like generating electricity they'll never use, or helping to digitize books they'll never read.
28:00 Speaker 02 What we did is we opened up our tools for creating languages for Duolingo, for adding languages to Duolingo to the world. In the first week, about 50,000 people applied to add a language.
28:08 Speaker 03 Keep this in mind when you reach out to users and offer to be partners with them in a shared mission. You raise the stakes. If you let them down, they'll lose faith in your commitment and your product. And that powerful crowd you've worked so hard to assemble will disperse. As will their willingness to work with you. This is why Luis put so much time and effort into ensuring that every user-generated course Duolingo puts out meets his rigorous standards. To do this, he had to find a way to measure their effectiveness.
28:42 Speaker 02 But first, we would measure just user engagement. So you know, average time that they spent on the course, user retention. So did they come back the next day, et cetera? I did not realize how hard it would be to measure how well people learn. I thought, how hard can it be? You do a pre and a post-test. You just ask them beforehand, you do a little few questions, then you let them learn, and then you ask a few questions afterwards. It is impressively hard to do that. It's taken us years and a significant number of people with PhDs to figure this out.
29:11 Speaker 03 This is a problem that all mission-driven founders must face. How much people like your product is not the same as how effective it is in achieving your mission.
29:23 Speaker 02 For some fraction of the users, we would give them a little test on the app, and then we would have them use the app for a while, and then we would give them another test, and we would see how much they learned. And then we thought, okay, well, we can A/B test against this. So if we teach a little differently, we can see, did the scores go up more or less?
29:39 Speaker 03 But they soon found a problem with this method. At first, the numbers told a story of success. But when they looked a little closer, the story was total fiction.
29:49 Speaker 02 One of the problems we had, probably the easiest way to get your scores to move up, is to make the people who are not very good drop away. But that's not teaching better. That's just like, if you just make the course really frustrating, actually, the less able people drop away, and your scores go up. And then we realized what was going on was, you know what? We were just making the course impossibly difficult. So the ones that stuck around kind of already knew the language. So it turns out to be pretty hard. But by now, we have a much more sophisticated ways of measuring how well people learn. We have a number of measurements, and we can tell, you know, the quality of the different courses.
30:25 Speaker 03 Using these measurements, Duolingo can test a course on a selection of users and tweak it to perfection before releasing it. Maintaining quality is vital to keeping users and contributors engaged. But so is getting out of the way. If your efforts to keep quality up end up blocking more than they help, then your crowd will find somewhere else to realize their passion.
30:50 Speaker 02 I think the reason we have so many volunteers working on some of these kind of smaller courses on Duolingo is because of the mission. I mean, we've shown them over the years that we are not going to suddenly put a paywall in the front of Duolingo, and just nobody will be able to use their stuff. So one of the biggest things is just really, people do it because they believe in the mission.
31:10 Speaker 03 The mission is so important that Duolingo has taken clear, legally binding steps to assure contributors that they're fully committed.
31:18 Speaker 02 At first, we did not have a legal contract with our contributors. It is unclear who owned this data. I mean, it didn't really occur to us. After a while, there was a couple of volunteers that started thinking, well, wait, who owns this? And we had to spend effort and we had to get our lawyers to really craft a good contract. Where now the contract is, so the volunteers actually own all their work, but we get licensed forever to use it.
31:44 Speaker 03 Despite his long history of crowdsourcing, this was something new for Luis. And I believe this legal agreement between Duolingo and its users is a great innovation, launching the company to the next level of scale.
31:58 Speaker 02 This is the first time I really actually had to deal with that. I think part of the reason is because unlike my other projects, most of the effort that people put in is these five-second things that people don't care all that much. In this case, you know, the volunteers spend sometimes years helping us add something. And so this is the first time we had to really deal with that.
32:16 Speaker 03 This clear contract that contributors retain ownership of all content they submit makes them feel secure that Duolingo is aligned with their mission. And if you want to harness the power of your crowd, the most important thing you'll need to do is to make sure your mission and that of your contributors is totally in sync. But there are also other things you can do to keep your crowd motivated. I want to spend the rest of this episode focusing on one way Luis does this. It's something that Duolingo is famous for.
32:51 Speaker 03 Gamification. For example, adding elements from video games,
32:57 Speaker 03 such as leveling up,
33:01 Speaker 03 leaderboards,
33:05 Speaker 03 and scoring points.
33:08 Speaker 02 One of the things that I have found with behavioral economics is, in my sense, the people who know most about how to influence behavior, I would say are not these people who are PhDs in behavioral economics, no offense. I would say it's people who make games.
33:20 Speaker 03 Yep.
33:21 Speaker 02 Those people are amazing. So we take from Duolingo, I mean, we spend a lot of times, it's funny, you can see a lot of the different items in Duolingo. You can trace a lot of the things we do based on what game our product managers are currently playing.
33:37 Speaker 02 So for a while, they were playing Clash Royale, which has these chests. Well, Duolingo is full of chests these days.
33:46 Speaker 02 And so it's this funny thing. You can trace a lot of our elements just to which game people were playing.
33:55 Speaker 03 I do similar thing to even product managers at LinkedIn and other things like, you must be playing some of these simple games to get a sense of them.
34:03 Speaker 02 Yeah.
34:04 Speaker 03 Right. Because you will learn things from them, even if you go, well, the game's kind of boring. It's not that interesting. Like, no, no, no, no. Pay attention to how actually even a very simple game actually retains some people because of the mechanics of the use.
34:17 Speaker 02 It's amazing. Now, I'll tell you, don't get your product organization playing Clash Royale. Suddenly, they won't be as efficient.
34:23 Speaker 03 Got it.
34:25 Speaker 02 We have a little leaderboard, very powerful. We have these little game dynamics. It works a lot.
34:30 Speaker 03 Any lessons from those game dynamics?
34:33 Speaker 02 Yeah, I think leaderboards and progress bars. People just want to fill up progress bars. Early on, we realized the hardest thing about learning a language is staying motivated. And so we rely heavily on gamification for keeping people motivated. So we do a lot of things. For the first four years of Duolingo, the only thing we did was improve user retention. And it's basically adding gamification features. So by now, we run about 2,000 A B tests a year, and most of them are on just little game mechanics. And it really has helped us.
35:02 Speaker 03 Another person who has thought deeply about gamification is Elizabeth Sampat, game designer and author of the book Empathy Engines: Design Games That Are Personal, Political, and Profound.
35:16 Speaker 05 Gamification works best when it's really tied deeply into the user's needs or desires. And what it really fails is when this gamified solutions are actually about the needs and desires of the person making the product, as opposed to the person that's actually going to be using it.
35:32 Speaker 03 Elizabeth says Duolingo is a good example of gamification done right.
35:37 Speaker 05 The thing about Duolingo is the vast majority of people don't have the goal to learn a language. They have the goal to know a language. And the part that's really painful, the part where all the friction is, is having to get to the point where you know the language. So by making that as distracting and sort of lighthearted as you can, it really helps smooth out the friction towards people's goal. And that's really what game design is. It's about figuring out what the actual goal of something is and how to get there. And so much of the time, and this is true for gamification as much as it is for games,
36:20 Speaker 05 when you have a system that isn't giving you the output that you want, that the real question is, have I named the correct goal? What is the actual goal that we're trying to reach here? Because oftentimes, you've actually created a really solid system. You've just created a system for the wrong goal.
36:41 Speaker 03 But even thoughtful gamification can spin out of control. Constant course correction is needed in order to keep gaming elements aligned with your mission. It's something both Luis and I have grappled with. One of the things that we found at LinkedIn was also where game dynamics can lead you astray. So for example, when we started, we wanted to create a game to get people to be more connected. So we put your number of connections at the very top of your profile in a prominent way. And so what happened is everyone started playing it just like progress bars. But then some people went crazy, like 23,000, 25,000, like nobody knows 23,000 people. I mean, literally nobody. But it's like, I'm trying to win that game.
37:25 Speaker 03 And we had a protest when we later retrofitted this and said, okay, at 500, it goes to 500 plus and we're done.
37:31 Speaker 02 Uh huh, uh huh.
37:32 Speaker 03 Like we end the game there as a way that we kind of patch the game dynamic. Have you guys had any kind of similar experiences where the game dynamic led you astray? And you needed to alter it in some way in order to have the better outcome for the ecosystem and the community and the people?
37:48 Speaker 02 Just last year, we took a concept from gaming where the idea is you actually make a leaderboard with strangers, but with strangers that are equally committed to you. And the way to implement that is this concept called leagues. You are with 50 randos in the bronze league. Last one week, the top 10 people go to the silver league. And then if you're at the top 10 there, you go to the next league, et cetera. And so this was very powerful at Duolingo. In fact, just adding this leaderboard increased the average time spent by users on Duolingo by 20%, which is a huge number, given that we have so many users. But it created some people who were basically Only motivated by this leaderboard. And they started doing things that were not good for their learning. I mean, for example, some of them discovered that you can just redo the first lesson
38:31 Speaker 02 a million times, and it happens to be an extremely easy first lesson. And so they just redo that like crazy. We've started patching this and now, you know, if you redo any lesson more than twice, we stop giving you points. And stuff like that. And again, people have complained about this type of stuff. We've had to patch it. And in general, we want to do what's good for the users in terms of actually learning. And we don't want to get these people always learning the word for man a trillion times.
38:57 Speaker 03 You have the game dynamics to help them stay engaged, stay motivated, not to create a different outcome.
39:02 Speaker 02 Yeah.
39:03 Speaker 03 But these are all nudges. The overall drive for your crowd always comes back to the wellspring of your shared mission. It was this shared mission that let Duolingo hold off from needing to generate revenue and focus on attracting mission focused creators.
39:20 Speaker 02 Because we had this mission that we didn't want to really charge for content. Our hands were a little tied on what we could make money with. I mean, the standard way to make money in education is just to charge for people learning. So we thought, well, the first thing we're going to do is just going to add a programmatic ad at the end of a lesson. We have enough users that that was significant. I mean, it's tens of millions of dollars from those little programmatic ads. Then we added a subscription that the only thing it did was turn off the ads at first. Now it does a few more things, but at first it just turned off the ads. Quickly, the subscription made more money than the ads.
39:50 Speaker 03 Generating revenue didn't disrupt the mission to bring free education to as many people as possible. The main concern for Luis is to keep Duolingo focused on the mission.
40:01 Speaker 02 I worry about this. I wholeheartedly believe that our business model is actually better than the just standard charging for content. This may not be intuitive, and I understand many investors will just never agree to this. But in the education category, we are the top grossing app. I mean, we make more money than any other education app, and we are like the one that doesn't charge for the actual content.
40:23 Speaker 03 Luis has also been able to revisit his early ambition to kill the TOEFL, that anachronistic English test he and millions of others have to take if they want to study in the US.
40:35 Speaker 02 We, as a company, have our own version of the test. So what happened is about four years ago, we started getting emails from people saying, hey, thank you for teaching me English. You know, I couldn't learn it before you guys. It's free. Thank you. But now I have a problem. I need to certify that I know English. And we got enough of these emails that we started looking into this.
40:53 Speaker 03 Now Duolingo offers an English test for $49 and is accepted by Yale, Columbia, Johns Hopkins, and around 500 other institutions. And the best part is you don't have to cross borders or brave war zones to take it as it's all done online.
41:12 Speaker 02 I kind of had forgotten my whole TOEFL experience, you know, when I had to take that test in Guatemala, but I quickly remembered. And when we looked at this business, it's a pretty crazy business. It's about somewhere between $5 and $10 billion they are spent by people proving that they know English.
41:26 Speaker 03 This is another unexpected place where the crowd took Luis and Duolingo to. And this is what you want when you take your product to the crowd. And the power of it, they are invested and will do their best to help you improve your product. Feedback will not be coming from just the disgruntled few, or if you're in dire straits, the disgruntled many. Your users will be invested in your shared mission and they will help you course correct well before you hit the jagged rocks of disillusion. I'm Reid Hoffman. Thank you for listening.
42:02 Speaker 00 Global hiring is such a headache when you're building fast, but don't let payroll, compliance, or onboarding slow you down. Companies like 11 Labs and Mistral are scaling globally and effortlessly with Deal. 11 Labs quadrupled their headcount in 2025 and Mistral pays 364 employees across 15 countries, all from one system. Build your global team with Deal. Visit deal.com/mos and start expanding your business today. That's D E E L dot com slash M O S.
42:38 Speaker 00 Every business hits the same wall the moment your business outgrows the systems that got you here. Spreadsheets everywhere, five different logins, data scattered across entities that don't even talk to each other. If your finance team spends more time finding data than using it, that's when you need the Intuit ERP. Intuit Enterprise Suite is the AI-native ERP solution that's powerful, painless, and proven, built for businesses scaling fast without the complexity that usually comes with it. From the makers of QuickBooks, learn more at intuit.com/ERP.
43:18 Speaker 03 Before we get to the rest of our conversation today, I want to spend a moment sharing with you about Masters of Scale Summit, which returns to San Francisco this October 20th to 22nd. So many of us feel overwhelmed right now. It feels like so much of the world is careening and we don't have power. We don't have agency. That's the thing about the Masters of Scale Summit. It's really designed to remind us that we do have agency. Agency to lead, agency to design, to build the relationships and the partnerships that empower us and to make the change that we want to see. We program a stage, we curate an audience, we set a schedule that builds in time for these relationships to be made and to grow.
44:01 Speaker 03 So that by the time you leave summit, you are fired out of a cannon, ready to run through walls to do better in the world with the agency that you actually have. I hope to see you there. Visit mastersofscale.com/summit.
44:22 Speaker 03 Masters of Scale is a Wait What original. The show is recorded remotely with sanitized audio gear. Our executive producers are June Cohen and Darren Triff. Our supervising producer is Jay Punjabi. Our producers are Chris MacLeod, Adam Skuse, Jenny Cataldo, Jordan MacLeod, Catherine Clark Gray, Hallie Bondi, Christina Gonzalez, and Ben Manila. Our editor-at-large is Bob Safian. Original music and sound design by Ryan Holiday and Daniel Nissenbaum. Gaming music by Alison Leighton Brown. Audio editing by Keith J. Nelson. Mixing and mastering by Brian Pugh. Special thanks to Chris Ye, Elisa Schreiber,
45:03 Speaker 03 David Sanford, Said Aspeyeva, Adam Heiner, Emily McManus, Kelsey Capitano, Tim Cronin, Sarah Sandman, Charlie Menezes, and Colin Howarth. Visit mastersofscale.com to find the transcript for this episode and be sure to subscribe to our email newsletter.
45:26 Speaker 06 Hey listeners, Bob here. If you listen to Rapid Response on Masters of Scale, you may be missing half the show because every Friday we release a second Rapid Response exclusively in the Rapid Response feed. The guests and topics are just as compelling and timely, from Ford CEO to NASA's administrator to the lessons from the devil wears Prada. It takes about 10 seconds to find. Just search Rapid Response wherever you listen to podcasts and hit follow to make sure you never miss an episode. I hope to see you there.
Previous Rapid Response: Overcoming institutional hurdles, w/ U.S. Olympic & Paralympic Committee CEO Sarah Hirshland Next Rapid Response: Launching a $500b startup, w/Jet.com's Marc Lore Page updated October 26, 2026 · transcript and summary are machine-generated