a16z Podcast
Original podcast
Where to listenCopy share link
3,562 words · 1 speaker
Hi, I'm Mark Ring. I'm from a brand new startup, Kojitai. I'm one of the co-founders. And it's an honor to be here to get to tell you about what is the passion of my life, which is continual learning. To a large extent, I think a lot of AI today is really a computationally sophisticated party trick. Siri and Sirius and Lucida can answer a lot of questions. But we know that they aren't really understanding in the way that we are. There are many questions that even small children can answer that no AI can even hope to give a reasonable response to. Maybe Jason will disagree with some of these. But nearly any question that requires visualizing how the world works will go well beyond the ability of any AI technology in existence today.
So I see AI as really the next and possibly the final step in the long progression of human automation. The industrial revolution transferred much of the burden of physical labor to machines. Information revolution transferred much of the burden of mental labor to software. Supervised machine learning reduces the burden of a lot of tedious software development, but still requires people to compile data sets for training. Reinforcement learning is the next step which allows agents to learn by trial and error, but I see the ultimate step as being continual learning, which has been the focus of my career.
Continual learning is about building machines that learn on their own, which I think is the best, maybe the only path to real AI. So supervised learning presents inputs and their corresponding outputs to a learning system such as a deep neural network. And a good system will learn to map inputs such as visual images to their appropriate categories. A reinforcement learning system is really a learning agent. The agent takes actions in an environment where it receives positive or negative rewards, and it learns through experience to improve the actions it takes at each step so as to improve its overall long-term reward.
A reinforcement learner maps inputs to actions. And it gets rewards too and it also generally makes predictions about how much war it'll get in the future, which helps them choose good actions in the present, and that's how they can learn to solve tasks. But a continual learner's goal is not to categorize data or to do certain tasks well, but to constantly add to their own knowledge and skills, continually improving themselves and their ability to interact with the world. I think the holy grail of AI is a continual learning agent. It has no specific task but keeps getting better and better. Interacts with the world and learns for itself what to do next and what to learn next, and builds up new knowledge and skills on top of existing knowledge and skills.
And all of this, I think, comes down to one thing, which I call having an isolaminar learning mechanism. It's a single mechanism for every level of learning, so it can be used again and again at every stage of development to learn every possible thing, no matter how simple or complex. The same mechanism for all knowledge and skills. These are really hard to find. The brain is an isolaminar mechanism. You get one brain. No one gets to tinker with it and change it while you're growing up. And humans are excellent continual learners. Think about how we first learn to sit up before we can crawl. We have to stand before we can walk.
We have to learn to reach to an object before we can grab it. Then we can learn to grasp it and hold it. Only then can we learn to scribble and draw shapes. Then finally, much later, we can learn to write. And eventually, after 20 years of this continual learning process, we can finally write something worthwhile. So I've spent my whole career looking for isolaminer mechanisms for AI, and I found many candidates, but I'm most excited about what I call forecasts, which I'll present today. So a lot of what I'm talking about today is due to a long-term collaboration I've had with Rich Sutton, who's generally considered the father of reinforcement learning today and who's also colleague at Kojitai.
Forecast began during a time when he and I were working closely together, searching for an isolaminar mechanism for all knowledge and skills, and I think they represent a major step in that direction. So as their name implies, forecasts are predictions about the future. Their secret is that they encode knowledge as behavior-dependent predictions, meaning that each forecast predicts something about what an agent will perceive if it acts in a certain way. So their subjective predictions. Each one says basically, I will perceive this if I behave like that. Like I will see blue if I look up, or I will see a bright light perception if I press the light switch. But they can also be about quantitative things, like how tired I'll be if I run five kilometers.
But critically, forecasts are an isolaminar mechanism, so you can use them layer after layer from simple to complex. And this is possible because a forecast can make a prediction about another forecast. You can build forecast A to make one prediction and then build forecast B to make a prediction about forecast A. And that's where things start to get complicated. So for example, forecast A could predict whether I would see blue if I look up, which is going to have a high value when I'm outside on a sunny day. Forecast B could predict whether forecast A will have a high value if I go outside. So this is going to have a high value if you're inside on a sunny day. And both A and B are behavioral, behavior conditional predictions that tell me something about the world around me and can change from moment to moment.
So forecasts are deeply grounded in some very nice mathematics, but I promise not to say a word about it. So the important question is then if forecasts are this great isolaminar mechanism, can they go all the way from the bottom to the top? Can we start at the very lowest level, say an agent's raw sensory motor stream, just pixels and motor signals, and knit together real abstract knowledge of the world. To start answering this question, I'm going to give you a demonstration. It's a very abbreviated 10-minute demonstration that leaves out nearly all the details. But I'm hoping it'll give you some inkling, some feeling of what forecasts are and how they can build knowledge.
So the demonstration is really a lot like a thought experiment. It's all going to be done in our heads. It consists of a micro world, which is a simplified environment with a robot agent. It's going to start with the robot's raw sensory motor apparatus and build up knowledge one forecast at a time. And our goal is to build up an abstract description of the microworld with just forecasts, hand designed by me to show you what can be done. So the micro world looks like this when viewed from above. Consists of multiple houses, floor plans, open spaces, barriers, obstacles, and doorways. And the robot looks like this. Cylindrical with two wheels, a camera, a finger that it can extend to touch things.
But remember, this is just a thought experiment. So we're just going to think about this robot and consider what happens as it goes around its environment learning about things. It has only five possible actions. It can roll forward or backward, it can turn left or right, 30 degrees, and it can extend its finger, which retracts automatically. It has only two sensors, a simple camera, nothing fancy, and a touch sensor at the end of its fingertips so it can tell when it's touched something. The robot, critically, has absolutely no built-in knowledge. William James, the 19th-century father of American psychology described new newborn babies' experiences as a blooming buzzing confusion because they have no understanding at all of what all their sensors mean.
In the same way, this robot just gets a lot of unidentified input values. It can't even distinguish between its fingertip and its camera. So we can think of the robot's experiences in three ways. We can imagine looking at the robot from the top and diagram it something like this, where the finger shows which head which direction it's heading. And we can also imagine what the robot sees through its own camera. Floor, walls, doorways, and so on. But keep in mind, when we look at this picture and recognize walls and a floor and a doorway, isn't that because we've had a lifetime of experiences looking at images like this and being able to Interpret them. But the robot, of course, has no experience, so it isn't able to interpret this at all.
It just sees a lot of random pixels
when it moves around and takes actions in its environment. So what I want to show you now is how the robot can start to acquire an understanding of the world like we have through continual learning, and I'll show you this using the overhead perspective, but we all have to be thinking about what the agent is thinking through its own subjective experience. So a continual learning agent, this one has a learning system that maps unidentified sensory signals to unidentified actions. And though it knows it has five actions, it doesn't know what they do. The agent also generates forecasts, predictions about the future at every moment the agent produces new predictions based on both its sensory input and the forecast from a moment ago.
So the challenge is to capture a high level understanding of the micro world using just these and only these ingredients, starting with just the raw sensory motor apparatus, and just through continual learning, adding one forecast at a time, layer by layer, attaining a general abstract understanding of the world that the agent is living in. So we'll start here at the very beginning. What we see here is the robot from above, and on the left is its touch sensor. When the robot extends its finger and makes makes contact, the touch sensor has a high value. And a moment later it returns to a low value. If the finger is extended and doesn't make contact, then the sensor keeps a low value. So the first forecast that I'll introduce is the touch forecast, which predicts the probability that the touch sensor will come on if the agent chooses action five, which extends the finger.
In this position, the probability is low. But in this position, the probability is high. This forecast says if I extend the finger now, the touch sensor will be activated and it's right. A moment later, the touch sensor has a low value again. But the touch forecast still has a high value because the robot's in a place where it could reach out and touch and sense something. So the big question is how can the robot learn to predict the value of the touch sensor? And actually, the answer is easy. The neural network at the agent's core just needs to recognize the camera image when the robot's facing up against the barrier or obstacle, which is a very easy classification task for a deep learning network. So when the robot rotates away, the angle looks very different to the camera, and the network can learn after lots of experience in these situations that the sensor will not come on if the robot extends its finger.
If it rotates back again, it can again recognize that it's facing an obstacle, that it's in the same position as before, and predict it would get a high sensor value if it chooses the extend finger action, which it can also verify. Okay, so that's the first forecast. Now we add a second forecast, touch left, which makes a prediction about the first forecast. The touch left forecast predicts whether the touch forecast will have a high value if the agent rotates to the left. So in this position, it has a high value because if the robot rotates to the left, then the touch forecast will have a high value. The same thing can be done on the other side with a touch right forecast, which predicts whether the touch forecast will have a high value if the rot the robot rotates to the right.
So in this position, the touch right forecast is a high value, predicting that if the agent rotates to the right, then the touch forecast will have a high value. Which again indicates that the agent predicts that the sensor will have a high value if the agent extends its finger, and that can be verified too. Okay. So step by step, we're going to get more and more abstract. Now, this same idea can be carried out not just once, but many times. In fact, we can have a whole series of forecasts, each predicting the previous one in the chain. If we lay these out in a circle like a clock, we can see how informative they can be. Now, regardless where the robot is, where it rotates to, it can keep track of the actions that would be required for it to get to a place where it can have a high value for its touch forecast.
Okay, so we're getting a little more abstract now. That might not seem so interesting, but another way of looking at this is that it now has an awareness of where the obstacles are immediately surrounding it. And it maintains this awareness as it rotates. Now, all of this is possible because of learning, because the supervised learning algorithm inside can learn simple distinctions, simple categories of visual images, and the forecast learning mechanism organizes that learning. Now we can build a new forecast, touch adjacent, which has a high value whenever the robot predicts that it can rotate to a position where it can get its touch forecast to have a high value. So again, we're building up from the raw sensory motor stream, and this forecast predicts whether the robot can make the touch forecast have a high value just by rotating.
Or another way of looking at it is the robot now has a representation that describes when it is adjacent to an obstacle or barrier. So that touch forecast is really useful. And the robot can now predict the number of forward actions using this that it would take to reach a place where the touch adjacent forecast has a high value, and that prediction gives the robot the subjective representation of distance, telling it how far away an obstacle or barrier is in its environment, just using its own actions. As the robot gets closer, the distance forecast continually decreases. And again, the learning required here is well within the abilities of current supervised learning methods. And the forecast learning mechanisms handle how that learning is done using the robot's experiences to generate data.
So this distance estimation ability is very important. It gives the agent for the first time an idea of what it can expect when it sees something from far away. So the distance estimate can now be used as the anchor for another chain of forecasts, like we did before, each making a prediction about the previous forecasts in the chain. Now the robot can rotate in circles and collect visual information that informs it about the number of steps it would take to reach a place where it's adjacent to something it can touch. Getting very abstract now. Now we can think of this more intuitively and say that the robot is now aware of how far away things are in all directions surrounding it. All through continual learning.
The robot can now build on top of these distance estimates. It can use its camera to look along an obstacle and predict whether it can roll forward, maintaining the obstacle on its side. And that prediction tells it something about the geometry of the obstacle, namely whether the obstacle is flat and straight. So our English word for this now very high-level abstract prediction is wall. In other words, the robot now has an understanding of what a wall is that's completely grounded in its experiences in the world. So I think that what we're seeing here is a first glimpse of true computer understanding of the world around us in the way that humans have it. These aren't just arbitrary labels or categories that a programmer is assigned to a node in a graph.
This is verifiable knowledge based on experience in the real world. So the wall forecast can be used in the same way we've used previous forecasts. We can build a distance estimator to the wall and then a chain of forecasts based on the distance estimate. The robot can then learn to estimate not just the distance to some kind of obstacle, but can to predict what experiences it can have once it traverses that distance. So it's the difference between knowing that if you take 10 steps, you'll bump your nose, or knowing that if you take 10 steps, you'll reach a wall. It's a much richer understanding of the world. This richer, deeper understanding of the world is also fundamentally informed by what the robot sees through its camera, through machine learning, what it's learned about.
So now when the robot stands in the middle of a room and rotates, it can make whole, a whole network of predictions about how it could experience its environment around it. So now let's look at some of the things that can be done with this rich information. There are a few examples of grounded abstractions the agent can build now. It can compare the distances to the walls surrounding it and figure out where it is with respect to these walls and whether or not it's enclosed within a room, and what the shape of the room is, and what as well as its size. It can use those abstractions now to build more forecasts, distance maps to doorways, hallways, rooms of different shapes and sizes, and in principle, any distinctive set of forecasts can be predicted.
And those predictions can be used to build ever more abstract forecasts continuing indefinitely. The agent can distinguish different rooms by their predictable characteristics. A certain room, say a kitchen might have a distinctive configuration of walls, and another room might be distinguishable because of its connection to the kitchen through a doorway, which means the robot can learn relationships now between entire rooms. Between sets, configurations of forecasts. It can then distinguish the different houses in the microworld by how the rooms in each house are connected, as well as how the houses are related to each other.
So the challenge was to try to build high level knowledge, starting with just the robot's sensory motor stream, and using the same learning mechanism over and over and over again. And to me, while I was putting this demonstration together, I found it fairly amazing to see how well forecasts can turn raw sensory motor experience into such a rich web of abstractions that discern and organize the properties of the world. This is the first time such high level abstractions have been built with an isolaminar learning mechanism out of raw sensory motor experience. This is just a thought experiment using the simplest robot I could imagine, having just a single touch sensor.
In fact, all the forecasts I showed here come down to making predictions about just one touch sensor. And yet the robot still ends up with this broad awareness of its entire surroundings. So you can imagine what sort of depth of understanding is possible if we had if I used a richer sensory apparatus.
And the regularities we find there become the building blocks of our knowledge. And when machines build up their knowledge through experience with the world the way we do, they'll be able to truly understand the world the way that we do.
There are some essential questions that need to be answered, including when to build new forecasts, when to remove old forecasts, how to deal computationally with massive numbers of forecasts. But these are all relatively soluble problems, and nothing would lead me to believe that they can't be solved within the next few years given sufficient focus and effort. So the result will be the machines will understand the world the way we do. And the question on my mind these days is what do we do with these machines? They'll be capable of improving almost every aspect of our lives, allowing us to have truly smart robots, truly smart cars, truly smart question answering machines, and truly smart search.
So I think maybe a better way of putting the question is is there any area of our lives that will not be impacted by this? Okay, thank you for your attention.
Page updated August 25, 2026 · transcript and summary are machine-generated