a16z Podcast
Original podcast
Where to listenCopy share link
4,661 words · 3 speakers
See if I need to augment my reality. Yeah, I do. Good. Okay, so my experiences in a group like this, probably there's a handful of you who really know what augmented reality is, and particularly the distinction between AR and VR and such. But there's probably a handful who think you know that and you don't. And then there's another handful who, you know, maybe you know a little bit. But so I want to spend a little bit of time just talking about augmented reality, and then I'll jump into at least a little bit of my work and in being there. So let me start out by saying that uh when I was a kid, I wanted to be a superhero. I wanted to have superpowers, super strength, I wanted to be able to fly, wanted to have X-ray vision, I wanted to be able to teleport, all those things were just so cool. And somehow it never quite happened.
But uh what I ended up doing, oh by the way, and I did actually buy those glasses when I was a kid, embarrassingly enough. Uh I won't ask who else of you might have done that, but um, but yeah, superpowers was just pretty cool idea. So what I ended up doing eventually is creating technologies uh for superpowers. So, you know, maybe I don't have super strength, but you know, I can give people superpowers through certain technologies. And so X-ray vision, seeing the invisible, uh, being being able to change size, large or small, and teleportation, things like that. Um of you know the the woman on the bottom right there. She's actually you go girl, which doesn't sound like a great superhero. So I'm gonna replace her actually with something you know nearer and dearer to my childhood. Um, not exactly superheroes, but you know, you get the idea for teleportation for the Star Trek folks.
So all of those things are done these days. X-ray vision, seeing the invisible changing size, uh teleportation done in the area of augmented reality. So let's just talk a little bit about augmented reality and the sort of areas that people are using or proposing to use AR in. Um, and first of all, just what is augmented reality? Let's sort of start with a basic definition so we're clear on it. My definition of AR is is not really a technical definition, it's an experiential definition, which is integrating digital information into real world experiences in real time. Okay, it's not fundamentally about technology, it's fundamentally about people and their experiences. That digital information is generally delivered visually or or through audio or through haptics. Um there's a number of different kinds of information or different ways the information can be delivered and such.
But it's basically an experiential thing, and it has to do with real world experiences, okay. Um and so this is the thing that's you know gets the message across to most Americans really easily, really straightforwardly. Probably, I expect all of you have seen football at some point, right? And you're used to seeing that yellow first down marker. Now, this may be a surprise to you, but that marker's not actually on the field. There's not some guy who's you know running out in between plays and spraying it on and then you know, undoing it and spot they don't do that. That's actually digitally put on. It's augmenting your experience, okay? And uh another example of that. Here we have the line of scrimmage in blue and the first down marker in yellow. We also have the third and five down there on the field. It looks like it's right on the field, right?
Um, but that also is an augmented reality thing. And you know, who knows? Maybe there was only one guy on the field and all that other stuff is is augmented, you know, it's kind of hard to tell, I guess. But um, so anyway, um here's another example from our lab where this is uh a scene on campus at UC Santa Barbara, and uh, but it's actually augmented. Uh there's something there that's not real. And um, as you'll be able to tell as I move that thing that that tree uh was not really in the scene. Okay, we augmented the scene by putting that tree in there, and we can do things like change the the lighting of the tree in real time and change the position of it and things like that. So a lot of times you don't actually know when when things have been augmented. Now, when people think of augmented
reality, they uh
often think about the devices, uh, because there's a lot of you know PR about the devices, uh head-mounted displays of various kinds, uh, you know, the upcoming HoloLens that most of you probably know about, contact lenses perhaps that can serve as uh display devices, cameras that you might wear, all sorts of things that that uh get a lot of public attention. But again, just keep in mind that augmented reality is not primarily about these devices. You know, that they're they're important now, but it's primarily about our experiences, and these devices are being used to deliver
experiences to us.
Okay, so of course, uh there are a lot of possible applications of augmented reality, and some that are already in use. A lot of interesting things in medicine, uh, in shopping and in real t retail, in learning and training, in gaming. That's probably one of the most uh earliest commercial uses of AR these days. In tourism, in advertising and marketing, that's sort of the other early entry, I guess, in real commercial use for AR these days. And a whole lot of other things in military and entertainment and a lot of areas. So people are building augmented reality systems for a real wide range of application, and again, in sort of real world experiences for people. So I still think of AR as superpowers, though. So just like you know, Superman could see through a patient into the brain and tell the doctor that it's ready to go, you know, that that's that's a really good use of real-world augmented reality.
And another one that I like of Superman, uh, he looks at Jimmy and tells him that he has to go to the dentist. Um, and so I think I can add to my list dentistry as being you know a really good application of AR. But again, AR superpowers. It brings me back to my childhood, and I guess that's why I'm I'm working in it. Okay, so in a quick overview of augmented reality. Uh again, the the distinction uh with some of the virtual reality talks is in augmented reality we stay in the real world, right? We're we're augmenting that experience as opposed to removing you from the the real world and replacing that with something else. Okay, so what I'm gonna talk about now with the remainder of my time is this project uh we call being there, which is augmented reality for remote collaboration, and particularly for remote physical real world collaboration.
Okay, so scenarios where people might be working on something together, but they are remote. For example, a repair or maintenance kind of task, um, where one person is there physically with the equipment and he or she needs to work with a remote expert in order to get the job done. That's sort of our our canonical example of the physical real world kind of remote collaboration that we want to support with our augmented reality. So we think of this as teleportation. It's sort of like you know, Captain Kirk and Spock going down to the planet, dealing with the problem there physically, and then beaming back up, you know, or teleporting an expert perhaps to work with a local novice uh in fixing something, uh, and then he teleports back and maybe goes and works with another novice somewhere else a few minutes later.
So it's that that general idea is what we're trying to do. So the key aspects of teleportation. I mean, if you think of what the Star Trek guys did, um the ability to view and explore independently. So if you're just stuck with my viewpoint, seeing through my eyes, that's that's not really the same as if you were standing here with me and we were looking at something together. So the ability to have an independent viewpoint and the ability to independently explore the area that you're working on is important. Also uh to be able to communicate in the context of a shared reference frame. So I can say when we're together, I can say, okay, give me that thing over there, or put this thing over there. It's very clear what I'm referring to when I when I say those things. Uh if we're just using Skype or the telephone or something, those sorts of reference aren't so obvious.
And then the third thing is the ability to physically manipulate the environment. And so I will admit we are not gonna do that one. We're gonna focus on the first two. Uh there are actually people working on remote ways to manipulate the environment, but that's not part of part of this project. So we're focusing on these two things: the the ability to explore independently for the multiple people, and then to understand your shared context very well. Okay, so in all these scenarios, the the sort of scenario we see we tend to talk about is a local user, maybe you know, the person needing to fix something or where the physical thing is, and then a remote expert somewhere else. Okay, and so between them there's a communication channel. And you know, typically we could call somebody on the phone and collaborate that way.
Uh we could actually take pictures of the thing and text or or email the picture and then talk on the phone. We could use Skype or FaceTime or whatever and video conference, but all of these have limitations. They're really not the same as being there as teleporting, and we want to make that interaction more like that and much richer. So our goal is to provide unobtrusive and easily deployable mobile telecollabor. I can't say that word sometimes, mobile telecollaboration, using 3D-based augmented reality. Okay, and to support rich interaction in unprepared physical environments. So there's some key words in there. Ubtrusive. We we don't want to require people to wear all this stuff and and uh we don't want to sort of have to put markers in the world and set up cameras here and there and all that.
We want this to be very easy and you know, work with whatever device you happen to have on you. And maybe if you have some great devices like a Microsoft HoloLens or or a Google Project Tango or whatever, it'll make the experience better. But we'd like to use you know whatever common devices people have. Um particularly Particularly unprepared physical environments. We want this to you know just pull it out and work anywhere, and not to have to sort of create uh the environment to be to be appropriate first. Okay, so here's the here's a scenario, sort of a cartoon scenario. We've got, say, me uh out here in California, and I want to, I'm I'm working on something physical, and I maybe I run into a problem, and so I need to consult an expert. I need to uh collaborate with this guy, maybe on the East Coast, or maybe crosstown here, or maybe even just in a different location on the building site uh that we're working on.
So I can pull out my my smartphone, and I can uh as we're starting to talk, we're starting to essentially video conference initially together. Um my device is building a 3D model of the scene in front of me. Okay, the the equipment or the area in we're actually doing this physical work. Okay. That model is sent over to the remote guy I'm collaborating with, and he can actually then explore that model separately from the live video. So we can watch the live video, but he can also detach from that, and I might be you know looking over at the left side of a piece of equipment, he can actually go over to the right side and take that viewpoint. He can zoom in, he can zoom out, he can freeze his viewpoint. Uh so it gives him that that first thing I mentioned is important in uh in teleportation, which is the sort of independent ability to explore the scene.
And then the second thing, he gets to do annotations on his screen with his finger or mouse or whatever, and those annotations get sent back to me, the local guy, and they show up on my device in the right place. So he might be, you know, again be referring to, okay, you need to take that widget here and rotate it in that direction, or pull it out and put it in there. Those kind of spatial reference become very clear when he can make annotations that show up correctly tracked in my view. Okay. So here's a video. Do we have audio by the way?
Like overhead.
Thank you.
Perfect, thank you.
This is our prototype system.
The memory, the network content you have there, that goes in one of the slots over here. Do you see what I mean? Yeah. And actually, you need to um unlock that little blue lock first.
And then I go this right here. Perfect.
Got it? Yeah. And the memory. Uh that's in one of those slots down here.
So you get the idea. The guy on the right is the remote expert, and he's exploring that scene uh around the computer chassis there somewhat independently from the local guy's point of view. And when he makes a gesture, the scene freezes momentarily, he can move around to different viewpoints, zoom in, zoom out, all that sort of thing. So the two of these, these two people can collaborate quite successfully on this physical task in a way that's hard to do again with just telephone or or Skype or something like that. So an example scenario, uh, a different scenario would be you know, you break down in the middle of nowhere with your car, and you need to contact, you know, triple A or your your uh your technician or whatever to figure out what's going on because you know nothing about car engines.
And so, you know, another scenario here where we've done experiments with that, you know, scenario of working with a remote person on a on an automobile engine. So here's the guy who broke down on the left, actually my PhD student Ben, um, and then working with a remote expert in the lab who's sort of telling him what to do in that scenario. So here's the whole system, and I and I won't go into much detail about this, but on the left side is the local user, um, and you see various components. There's the camera on the on the very left, uh, the screen of his device at the top on the left there, um, different components of of what his system has to do in order to send video to send 3D information. And the the probably most uh important aspect here is just the what goes on here, what's going on between the two, and the communication is basically video to the remote person, uh, 3D information to the remote person, and then eventually annotations back to the local user, and then of course audio going in both ways.
So the key research challenges in this work are really three primary things. One is doing the environment modeling or the 3D modeling. You know, again, I've got my cell phone out here, I'm talking to the remote guy, building a model of the world here. Now it's it's not gonna be some you know beautiful, pristine, nice 3D model. It's it's kind of messy, it's kind of noisy, uh, and we have to deal with with what we're given there. So how can we model the scene good enough to support this kind of interaction? The second problem is image-based navigation. For that remote person, how can he or she navigate through the 3D space and explore as he or she wishes based on this sort of imperfect 3D information and the live video stream? And then the third problem is the is the annotations. If we draw an annotation from a certain perspective, how should that be properly rendered from the user's perspective, the local user's perspective.
So I'm gonna talk a little bit about the first and third of these with my uh limited available time today. So environment modeling, just sort of a quick overview. There are really two kinds of typical models people might build with images. One is a 3D model, uh, sort of the um that model there from this tracking data, you might build a 3D model. Or if you're just rotating your camera, you might build a panoramic kind of model. And if, for those of you who know about SLAM simultaneous localization and modeling, uh there's some, it's it's a very good technology for building 3D models and knowing where you are at a given point of time in that model. Uh, but there's certain limitations about it, you need translational motion and stuff to get the 3D and all that. So what we end up doing is sort of merging these two things, the the simultaneous localization mapping or slam to get sort of 3D like the uh like this, and then the panorama tracking and modeling to get the 2D here, and we essentially merge them together.
So if someone's just rotating the camera, that's only panoramic information at that time, we can include that in our model. So it's really a mixture of 3D and 2D information glued together appropriately. So we we combine slam and panoramic track tracking and modeling. So here's sort of a sense of how those two come together. We've got a video source that's sending frames or or frames are coming into the camera. Um we've we have feature points from, say, frame one, and we locate feature points in frame two, and presumably we're doing some correspondence so that we have a good good sense that they're matching from frame one and frame two. And so we say, given that correspondence, let's see, is that better estimated by what's what's known as an essential matrix, which is basically a 3D descriptive motion or a 2D descriptive motion, the homography.
So given those two frames, is it really a kind of a 3D thing or a 2D thing? So we basically uh measure the goodness of these two models and say, is it 3D structure for motion, or is it 2D panorama? And we make that decision. Let's say for this pair of images, we said, okay, the essential matrix is a better model of this transition. So we're gonna we're gonna try to get 3D information from these two frames. So we're gonna label that transition as a 3D frame, uh 3D pair of frames. And then we'll do the same thing for subsequent frames as the video goes along. And as you can see, we have a group of uh 3D frames and then a group of sort of 2D information frames. And so we can build structure for motion from the 3D frames, we can build panoramic information from the 2Ds, and then we'll merge them, and we know you know there's a particular frame there where those two kind of are kind of glued together.
Yes, rich.
Are those time correlated? Do you do E's and then H's of the keys of the long runs or are
we? We do, and then we can actually sort of do that in a hierarchy, so we can group them in in different ways. Yes, yeah. And that's typically it works how people tend to move around. You know, you tend to do that way, you don't look randomly frame to frame move differently, basically. Yeah, because we're physical. So uh what we have here is just an example of of uh putting together some 3D stuff, which you can see on the left side of this model, and some 2D stuff, panoramic stuff that you can see on the right side of the model. And as it turns out that is a fine model, even though it kind of looks noisy, and you say, well, why is that right side flat? Well, because that's all that you know the camera was able to give you, and it turns out to be a perfectly fine model to support the experience that we're trying to support here, the the ability to navigate and the ability to make annotations and such.
So here's uh just a little clip of the uh the first part of this, uh, doing this modeling, building the 3D model as we go, and in a moment we're gonna stop translating and only rotate, so it'll start building a panoramic. So right now we're rotating. So now it's gonna build a 2D part.
Okay, so um, and we end up again with a model like this, and then uh what I didn't show before is the locations of the camera, you know, at different points in that model. And all the all that information is useful for what we do with it. Okay, and then the other problem here is annotation rendering. If you uh if if I'm gonna draw on this guy, say I'm gonna draw a mustache on him, but you are gonna look at this guy from a different perspective, say from the side, where should that mustache be drawn in this perspective? Well, you know, presumably there, and that kind of makes sense. If we have this idea of kind of a spray paint model, I have 3D information about the guy, I spray paint this mustache on him, that that's more or less where it's gonna show up. But what if instead of a mustache, I I wanted to do this?
I'm circling the guy's head and say, you know, this is the guy I'm talking about. Where's that going to show up in this spray paint model? It's going to show up like that. Okay. This large, whoops, wrong button. This large blue thing there, most of that circle is actually way in the background somewhere. Okay, so that doesn't work out. What I probably meant to do was this, right? So we have to figure out a way to make 2D annotations on a screen be rendered correctly from different perspectives. Because we have two different people, you know, looking at things from different perspectives. So what did the user intend is the real question. How can we infer this based on 2D gestures and based on the underlying 3D data that we have? So we did a study with 88 users using Amazon Mechanical Turk, and we basically showed them a bunch of 2D annotations like this, and then said, okay, given a different perspective, which actually matches best?
And we use four, maybe sometimes five different models. Spray paint model, what's called as a minimum depth plane, median depth plane, dominant surface plane, and basically figured out you know, in what conditions do one or other of these models make sense. And to cut to the chase here, let me just show you a little video of this. So here a user draws an arrow, it immediately gets regularized. That is we draw a nice clean digital version of it. And now we rotate to a different viewpoint, and that's what we'd like to see from the alternative viewpoint. But this is what the spray paint model gives us. It's complete trash. Okay. This is what the median death plane interpretation, and it's maybe okay. The dominant plane interpretation, not so good. Uh and what the gesture enhanced by surface normal.
So this is the one that you know we're really interpreting the user's intent and using that to help us render things. Now we're just looking at a different direct different viewpoint again and going through these same models with pointing at the printer there. So some of them make no sense at all. And this last one, the gesture enhanced, makes a lot of sense. And yet another viewpoint here. Spray paint's bad. Median depth plane is pretty bad. Dominant plane is really not so good. But again, gesture enhanced, this is really understanding the user's intent. Um makes a lot of sense. Okay, and then one more here. Now here's drawing a circle uh around an object. First thing we do is re regularize that right there. And now let's look at that from a different perspective. The spray paint looks pretty bad from that perspective.
Median depth plane misses the box pretty much completely. Dominant plane misses it as well, and the gesture enhanced, which again is you know, trying to infer the user's intent uh does a pretty reasonable job at uh showing what you were referring to. Okay. So anyway, to summarize, um we're we've been working for a few years now on this framework and prototype to support mobile remote collaboration, and particularly in unprepared environments with commonly available devices. And it involves these three things that I was talking about: 3D modeling, remote navigation, and these world-stabilized annotations. And there are actually lots of variations. So I've described to you sort of the canonical situation of like a local user and a remote expert who have different roles, but there are many different ways we can think about those roles, and it can be end to end and all sorts of variations.
The real challenge here for us moving forward is dealing with dynamic scenes and non-rigid options, non rigid objects. So those are things we're working on now but have not solved at this point. Um but who actually wants this? What are the compelling use cases? So just real quickly, uh, we have a startup, my former PhD student and I, uh working on this to commercialize this. It's called cognate. And um basic idea is initially is targeting field service technicians, and there are over five and a half million field service technicians in the US, and the dynam the uh demographics of field service is such that there are fewer and fewer experts, the kind of older guys and and a lot of young guys, inexperienced guys out in the field, and finding ways to sort of make good use of the limited experts and let people work remotely on problems that can you know cost companies tens or hundreds of thousands of dollars of a day if you know equipment is down, uh, is a really promising market, we believe.
So, leading to faster problem resolution, reduced expert travel, reduced equipment downtime, higher customer satisfaction, all these good things. So uh let me just kind of quickly go through our long term view, business to business, later business to consumer, and then eventually I think there are a lot of possibilities here for consumer to consumer uh remote collaboration. So acknowledge uh several people who've been involved in this work, and I'll look forward to talking about this in the panel session. Thanks
for the
Page updated August 25, 2026 · transcript and summary are machine-generated