a16z Podcast
Original podcast
Where to listenCopy share link
3,469 words · 7 speakers
Hi, so I'm Jason Mars. I think the first thing to do is to introduce myself and the research lab that we run at University of Michigan. So I'm Jason Mars. This is my third year as an assistant professor at Michigan. I co-run the Clarity Lab in Michigan with Ling Jia Tang, she's in the audience. Wave. Awesome. And we have a great team of very, very enthusiastic, high-energy PhD students in Postdocs. We have 18 of them. So it's a challenge and an adventure to mentor them. We've been super duper. Super duper productive in our short time at Michigan. Since at Michigan, we've published papers at the absolute top places in my community. And I encourage you guys to look into this. I mean, we've got some cool stuff here.
There's there's a work where we put candle wax in servers to keep them cooler longer. So check that out. But today I'll be talking about two pieces of work that I think can really be interesting and disruptive to the current uh model for how we interact with intelligent personal assistants. So let's get serious. How many folks in the audience have heard of Sirius? All right, a couple. Cool. Well, now all of we'll get it to 100% by the end of this talk. So there's an odd there's a clear observation, right? Companies like Apple have Siri, uh, Microsoft has Cortana, Google has Google now. Uh and these are intelligent personal assistants. These are changing the way we interact with technology. With wearables uh being the expected future of the market and the future way we interact with technology, and with wearables becoming even more interesting with vision capabilities in addition to audio, uh, it's gonna be ever more important that we we have uh you know natural and intuitive ways to interact with technology.
And so ABI research notes that uh wearables are expected to grow to uh 485 million shipments of wearables like the iWatch by 2018. Uh and with that, as a research team, interesting questions arise, right? So, what are the impact on the cloud computing and the data centers of the world when you have a workload that uses machine learning uh on the critical path of every query? Uh can we scale to millions and billions of users? Apple built a one billion dollar data center at the time that Siri was released. In that first couple weeks, if you remember, there were outages. It was the most expensive data center built at the time, one data center built at the time. Uh and so the question is if we move to a world where we're interacting with our wearables more than we're typing search queries into uh a text box.
Can we actually support that future today? And how do we design the right systems to get there, right? Now, as a researcher and wanting to ask these kinds of questions, we hit a block in the road. These kinds of infrastructures are monopolized essentially by companies like Apple, Google, uh, Microsoft, and so forth. So as a researcher, we could choose to turn around and pick an easier problem, one that we can do in a more constrained environment, or we can do the hard thing. And the hard thing was to get the most representative intelligent personal assistant that we can build uh out of what we can learn from the companies themselves and what's available in the open source community. So we chose the hard thing.
And so we designed Sirius. Uh it's an end-to-end voice and vision personal assistant. Uh on this path, there we had to have a clear vision of exactly what we wanted this system to be capable of. There are three types of queries we wanted to support. Uh, one voice command where you give a command to uh your mobile device or or or uh an object in your space. We also wanted to support voice queries where you ask a question that should be, should have an answer in the you know uh large amounts of data that we have around us. Um so who is the lead singer of YouTube would be one such question. And we wanted to push it even further. Uh we wanted to include voice and vision uh uh in a single query, so you can look at a bar and snap a picture of that bar and say, when does this bar close, right?
Uh and the intelligence would be able to ascertain what bar is it, uh, and then when it actually closed to interact with you. This is the system that we've stitched together using the best stuff around. Uh and so I'll walk you through the life of a query, a serious query. Um so the first class of query to voice query, uh, the recording of the user's voice uh uh is sent from the mobile device, and this is how Siri and Cortan and Google now all function, uh, and that recording uh hits an automatic speech recognition system on the cloud. Uh in Sirius, that speech recognition, that's that the actual text that comes from the speech uh could be discerned as to whether it's a it's a query, it's a question that you need an answer, or if it's a command, an action that you want the mobile device uh to take.
Uh and if it's an action, in the case of this first class of query, it's sent back to the user. Uh the second class of query uses uh a more sophisticated pathway through these services. First, we do speech recognition, uh, and then when it's actually a question for which we need to grab information, we have uh a QA system, a question and answer system that uh uh has as it uh at its disposal the entire corpus of Wikipedia to answer your question. Uh and uh it hits that system, and the answer is plucked from the knowledge base of of Wikipedia and sent to the user. For the final class of query, we also add a new type of service, and this is an image matching service where uh you can extract exactly what's in the image, and I'll describe how this works shortly, uh, and then combine the information from the speech and the image into one query that hits a QA system uh before producing an answer.
And so we've had a lot of fun building the system. It's it's it's been awesome. We went to the companies themselves, we did a lot of investigative research, is what I'm calling it, where we talked to the leaders of important teams, the key teams at Google and Facebook to really understand well, what are the algorithms they're using? Uh so in our in our infrastructure for speech recognition, uh, Google used to use Gaussian mixture models uh uh in the early days, and now everyone is moved to deep neural networks to do the speech recognition uh task. And so we support both of those approaches in Sirius. For the image matching, we use uh computer vision technique, very well known, the surf algorithm that extracts features out of the image and can identify, can match those features to a knowledge base that's out there.
And for QA, uh we chatted with folks at Google, and they said if you want to know what we're doing for our question and answer systems, take a look at Open Ephra. Uh and so we've started with Open Ephra, which is actually the foundation that then became Watson, and we've built a QA system, uh, RQA approach out of that. And the way it looks is uh uh uh we we have a microservice infrastructure where we have each of these uh uh infrastructures as a service. Uh and let's take a quick look at a demo of Sirius. Now, after Sirius gained a good bit of traction, Apple reached out to us and said, you probably should change the name. And so we're now calling it Lucida. Um but let's take a look at how it works. Before I played a video, on the laptop, we have we essentially logged into uh our cloud uh server in our cloud at Michigan, uh, and we actually have mobile apps on iOS and Android, uh, and we're using one of them from an iPad here because the iPad is louder.
Um and Johan Hoswell, who's in the audience, wave real quick, Johan, is actually the one that's doing this demo for us.
Who's the author of James Bond?
Let me find the Ian Fleming.
When was Google's IPO?
Finding the answer for you. August 19th, 2004.
Who invented peanut butter?
Right away. George Washington Carver.
This is an image.
How tall is this building?
Analyzing your image. 56.67 meters.
How many pounds does my car's roof rack hold?
Retrieving your car's information. 100.
And so that's a demo of the actual technology working. Uh and as you could probably guess, you know, the first demos that we showed of this was very cryptic, right? It was we didn't even have a mobile app, so you're talking to a terminal, and there's just a bunch of uh Linux uh uh prompts. Uh and so we've been refining this thing, and we've actually been working on productizing it. And I'll talk a little bit about more about that in a in just a moment. But now that we have an end-to-end personal assistant, we can revisit those questions that I asked earlier in the talk. Uh and so we've studied a number of issues, and I'll zoom through some of this. Uh uh there's in the paper we have over 30 figures that you can really do a deep dive into the Data.
But leave it to a nice observation we made is when you have all of the systems required to do the speech recognition and the question and answering on the pathway of a single query, that query needs to do a lot more computation. And here we get a sense of that scale. In a nutshell, we call it the scalability gap. If you moved all of the queries that were search queries and turn them into intelligent personal assistant queries, we would have to grow our data centers by two orders of magnitude. So if this is a top-down view of that of a of one of Google's data centers, using commodity processors, we'd have to scale it two orders of magnitude. That's not sustainable. We can't do that. And so we wanted to understand well, what's the stuff in this end-to-end intelligent personal assistant?
So we studied it and found that the machine learning aspects of the workload consumed the majority of the execution time, the cycles. So Gaussian mixture models deep learned deep neural networks, a large portion of speech recognition. STEMmer, which is a machine learning, a natural language processing algorithm to get the most rudimentary version of a sentence and conditional random fields are a huge percentage with regex, are a huge percentage of the QA system. And in image matching, of course, it's primarily feature extraction, feature description, which are huge fractions of the execution. And so across the entire workload, 92% of the execution of Sirius could be described by these compute-intensive algorithms. And so we plucked these algorithms out and created a benchmark suite that also is available with Sirius to study accelerator platforms and what future systems should look like.
So we did a quick study. We actually ported all of those algorithms, so we've got the speech recognition, question and answering, and image matching to three accelerator platforms, right? CMP, which is multi-core, GPUs, the Intel fee, and FPGAs. It turns out, and I'll just get to the key essence here. We can get significant speed ups when we port these algorithms to GPU, when we leverage GPUs and FPGAs to build future servers. And what that means is we can bring that scalability gap down. And so we can get about 10x back if we move by taking these, by taking Sirius, we can get 10x back. However, there's still a lot of room to grow. And so we're still studying this problem. Sirius has had a big splash. We've released it to the community, and it's been covered all over the place.
It was number one on GitHub for a couple weeks. There's been a lot of interest. We've gotten emails from all over the world of people interested in taking this infrastructure and putting it to use in their own applications. I mean, and I think what we've done here that I'm super duper excited about is we've taken a technology that you only find in the biggest companies in Apple, Microsoft, and Google, and we've democratized it. We've made it available to everyone on the planet. And so I think that that's hugely powerful. And I think that I think there's a huge opportunity to create companies around this vision of democratizing big infrastructures. Another observation we made, and I only have two slides on this piece, is that deep learning is absolutely the bet that companies are making, right?
So Google has put a lot of resources into deep learning. You know, Facebook, we're designing future data centers and future infrastructures around this one technique, and it's primarily because this one technique solves a lot of problems and solves it better than other approaches. So our lab is specializing and is implementing the latest most sophisticated neural network architectures that we know about from NIPS and ICML and ACL to create an ecosystem of deep learning engines that we call, well, the name keeps changing. It used to be called Intellico when I made this slide, but I think we're gonna change the name yet again. And this is the absolute state of the art, right? Like we recently stood up a sentiment analysis based on recurrent neural networks and long short-term memories.
Uh and we also have a CNN approach from 2014 that we compete with this approach. Um so we're having a lot of fun. Uh in the time left, I want to tell you about what we've done. We wanted to take the next level of impact with the work that we do in Clarity Lab. So we created a company called Clarity Lab Inc. Really, what drives us and what drives the culture in our lab is to actually change the world. It's to actually change future technology. And papers take you part of the way, but to really take it all of the way, you need to get your creations in the hands of the world. And so Clarity Lab Inc. is all about doing that. Um all of the co-founders of Clarity Lab Inc. is actually in the room, so feel free to chat with us. Ling Jia Tang, another professor at Michigan, Mike and Johan, the absolute top students in our lab.
Um is actually the first author and the lead behind Sirius and Jin and Tonic top-tier papers. Gin is a deep learning engine, it's an appliance. We co-designed a GPU box with a software stack for deep learning. Uh, and it can it it beats the pants off of out of the box stuff that you can find, it's published at Isca. We have a beautiful space in uh downtown Ann Arbor. Uh, and uh we were so excited about what we were doing with the company specifically for this talk, we made a video. So I'll play that video now.
The expectations of users and the way they interact with technology is changing. Previously they used keyboard strokes and mouse clicks to interact with computers, but now with wearable devices and mobile devices, users require experiences where they're using natural language, voice, and images to interact with technology. Because we need a higher quality artificial intelligence behind these new kinds of modes of interacting with technology, we've created Clarity Lab Inc. We call it Clink for short.
Clarity Lab has a wide spectrum of products and applications that use artificial intelligence and machine learning as call engine.
One such example is a sentiment analysis engine we developed on state-of-the-art machine learning technology that extracts from a sentence whether it's a positive or negative sentence. Through our partnership with Notion, we directly integrated this state-of-the-art sentiment analysis into their platform. So we
started Notion to bring artificial intelligence to people in a way that makes their everyday life experiences a lot better. And the best place that we see to do that is in their email. Many of us spend hours a day dealing with our email, but really it requires us to do a lot of work as opposed to working for us. So what Notion's able to do is actually look at your email history, which we all have a lot of, and start building an understanding of you. So it's a really personalized artificial intelligence that adapts to you and makes your email become intuitive. What Clink has been able to develop in the research lab at the University of Michigan is cutting edge artificial intelligence that isn't available on the market anywhere else. It gives us the ability to really tap into an engine that can run sort of the highly computational stuff that you need to actually develop these experiences that we're building into Notion.
One of the things we were doing is we're taking Sirius, an open source question-answer system that we developed at the University of Michigan, and we've turned this into a product called Lucida, which is a question answering system that allows people to interact with technology in a natural and conversant
way. Lucida is a customized and tailored personal assistant that we specialize for specific domain. We took Lucida, and we're designing a banking coach that integrates for a company,
BankJoy. So Bank Joy is a Y combinator back startup that is providing APIs to banks and credit unions. We started this company because we were working in the banking industry for a number of years, and we realized it was too hard for developers to build software for banks. Clarity Labs is helping us to do things with our apps that no other mobile banking or Apple Watch banking apps were able to do today by giving us the opportunity to integrate full smart voice user interface technology and personal assistance technology into our app such that we can ask questions like do I have 200 to spend on a TV right now? Or how much money have I spent on groceries in the past month, or send $5 to John. You'd be able to do things with this voice user interface, and it's gonna work really well because of the research that they're doing.
We are really excited to be working on Clarity Labs because they are, in our opinion, they're like our Google labs or our Microsoft research that we, as an early stage startup, have no business having. But we have them, and we're able to build this new next level technology today.
Artificial intelligence is hard. What we provide is the entire end-to-end solution. We are able to interact with a company to understand well, what is it exactly that they want to make a reality in their product, and then we'll show them how our ecosystem of services is the right platform for them to be able to build that feature without requiring that they understand anything about machine learning or AI. Companies don't have to understand the technology. They can just get highly intelligent experiences into their products. And so that's what we're super excited about. Going beyond simply providing algorithms as a service and providing the entire solution where you can build from there. Well, the base is pretty good in this room.
So you learn about Bank Joy and Notion. We're really excited about these companies that we're working with. Final slide. So, what we've done with Clink is we've already got some really interesting collaborations going on with companies. I think this is really, really cool. We have an unprecedented license agreement where, in addition to all the tech we currently have, four years of future tech is also covered in our license. So as we improve it from our research lab, it's exclusively owned by Clarity Lab Inc., the improvements. We have a nice talent pipeline. We're super lean, and for the time being, this is gonna change. We have a little cash in the bank. So thank you. And that's uh my talk.
Page updated August 25, 2026 · transcript and summary are machine-generated