Subscribe
Share
Share
Embed
Exploring the frontiers of Technology and AI
Ryan Sean Adams:
Dwarkesh Patel, we are big fans. It's an honor to have you.
Dwarkesh:
Thank you so much for having me on.
Ryan Sean Adams:
Okay, so you have a book out. It's called The Scaling Era, an oral history of AI from 2019 to 2025.
Ryan Sean Adams:
These are some key dates here. This is really a story of how AI emerged.
Ryan Sean Adams:
And it seemed to have exploded on people's radar over the past five years.
Ryan Sean Adams:
And And everyone in the world, it feels like, is trying to figure out what just
Ryan Sean Adams:
happened and what is about to happen.
Ryan Sean Adams:
And I feel like for this story, we should start at the beginning, as your book does.
Ryan Sean Adams:
What is the scaling era of AI and when abouts did it start? What were the key milestones?
Dwarkesh:
So I think the undertold story about everybody's, of course,
Dwarkesh:
been hearing more and more about AI.
Dwarkesh:
The under-told story is that the big contributor to these AI models getting
Dwarkesh:
better over time has been the fact that we are throwing exponentially more compute
Dwarkesh:
into trading frontier systems every year.
Dwarkesh:
So by some estimates, we spend 4x every single year over the last decade trading
Dwarkesh:
the frontier system than the one before it.
Dwarkesh:
And that just means that we're spending hundreds of thousands of times more
Dwarkesh:
compute than the systems of the early 2010s.
Dwarkesh:
Of course, we've also had algorithmic breakthroughs in the meantime.
Dwarkesh:
2018, we had the Transformer.
Dwarkesh:
Since then, obviously, many companies have made small improvements here and there.
Dwarkesh:
But the overwhelming fact that we're spending already hundreds of billions of
Dwarkesh:
dollars in building up the infrastructure,
Dwarkesh:
the data centers, the chips for these models, and this picture is only going
Dwarkesh:
to intensify if this exponential keeps going,
Dwarkesh:
4x a year, over the next two years, is something that is on the minds of the
Dwarkesh:
CFOs of the big hyperscalers and the people planning the expenditures and training going forward,
Dwarkesh:
but is not as common in the conversation around where AI is headed.
Ryan Sean Adams:
So what do you feel like people should know about this?
Ryan Sean Adams:
Like what is the scaling era? There have been other eras maybe of AI or compute,
Ryan Sean Adams:
but what's special about the scaling era?
Dwarkesh:
People started noticing. Well, first of all, in 2012, there's this,
Dwarkesh:
Ilya Seskaver and others started using neural networks in order to categorize images.
Dwarkesh:
And just noticing that instead of doing something hand-coded,
Dwarkesh:
you can get a lot of juice out of just neural networks, black boxes.
Dwarkesh:
You just train them to identify what thing is like what.
Dwarkesh:
And then people started playing around these neural networks more,
Dwarkesh:
using them for different kinds of applications.
Dwarkesh:
And then the question became, we're noticing that these models get better if
Dwarkesh:
you throw more data at them and you throw more compute at them.
Dwarkesh:
How can we shove as much compute into these models as possible?
Dwarkesh:
And the solution ended up being obviously internet text. So you need an architecture
Dwarkesh:
which is amenable to the trillions of tokens that have been written over the
Dwarkesh:
last few decades and put up on the internet.
Dwarkesh:
And we had this happy coincidence of the kinds of architectures that are amenable
Dwarkesh:
to this kind of training with the GPUs that were originally made for gaming.
Dwarkesh:
We've had decades of internet text being compiled and Ilias actually called it the fossil fuel of AI.
Dwarkesh:
It's like this reservoir that we can call upon to train these minds,
Dwarkesh:
which are like, you know, they're fitting the mold of human thought because
Dwarkesh:
they're trading on trillions of tokens of human thought.
Dwarkesh:
And so then it's just been a question of making these models bigger,
Dwarkesh:
of using this data that we're getting from internet techs to further keep training them.
Dwarkesh:
And over the last year, as you know, the last six months, the new paradigm has
Dwarkesh:
been not only are we going to pre-train on all this internet text,
Dwarkesh:
we're going to see if we can have them solve math puzzles,
Dwarkesh:
coding puzzles, and through this, give them reasoning capabilities.
Dwarkesh:
The kind of thing, by the way, I mean, I have some skepticism around AGI just
Dwarkesh:
around the corner, which we'll get into.
Dwarkesh:
But just the fact that we now have machines which can like reason,
Dwarkesh:
like, you know, you can like ask a question to a machine and it'll go away for a long time.
Dwarkesh:
It'll like think about it and then like it'll come back to you with a smart answer.
Dwarkesh:
And we just sort of take it for granted. But obviously, we also know that they're
Dwarkesh:
extremely good at coding, especially.
Dwarkesh:
I don't know if you actually got a chance to play around with Cloud Code or
Dwarkesh:
Cursor or something. But it's a wild experience to design, explain at a high
Dwarkesh:
level, I want an application to does X.
Dwarkesh:
15 minutes later, there's like 10 files of code and the application is built.
Josh Kale:
That's where we stand.
Dwarkesh:
I have takes on how much this can continue. The other important dynamic,
Dwarkesh:
I'll add my monologue here, but the other important dynamic is that if we're
Dwarkesh:
going to be living in the scaling era, you can't continue exponentials forever,
Dwarkesh:
and certainly not exponentials that are 4x a year forever.
Dwarkesh:
And so right now, we're approaching a point where within by 2028,
Dwarkesh:
at most by 2030, we will literally run out of the energy we need to keep trading
Dwarkesh:
these frontier systems,
Dwarkesh:
the capacity at the leading edge nodes, which manufacture the chips that go
Dwarkesh:
into the dyes, which go into these GPUs, even the raw fraction of GDP that will
Dwarkesh:
have to use to train frontier systems.
Dwarkesh:
So we have a couple more years left of the scaling era. And the big question
Dwarkesh:
is, will we get to AGI before then?
Ryan Sean Adams:
I mean, that's kind of a key insight of your book that like,
Ryan Sean Adams:
we're in the middle of the scaling era.
Ryan Sean Adams:
I guess we're like, you know, six years in or so. And we're not quite sure.
Ryan Sean Adams:
It's like, like the protagonist in the middle of the story, We don't know exactly
Ryan Sean Adams:
which way things are going to go.
Ryan Sean Adams:
But I want you to maybe, Dworkesh, help folks get an intuition for why scaling in this way even works.
Ryan Sean Adams:
Because I'll tell you, for me and for most people, our experience with these
Ryan Sean Adams:
revolutionary AI models probably started in 2022 with ChatGPT3 and then ChatGPT4
Ryan Sean Adams:
and seeing all the progress, all these AI models.
Ryan Sean Adams:
And it just seems really unintuitive that if you take a certain amount of compute
Ryan Sean Adams:
and you take a certain amount of data, out pops AI, out pops intelligence.
Ryan Sean Adams:
Could you help us get an intuition for this magic?
Ryan Sean Adams:
How does the scaling law even work? Compute plus data equals intelligence? Is that really all it is?
Dwarkesh:
To be honest, I've asked so many AI researchers this exact question on my podcast.
Dwarkesh:
And I could tell you some potential theories of why it might work.
Dwarkesh:
I don't think we understand.
Dwarkesh:
You know what? I'll just say that. I don't think we understand.
Ryan Sean Adams:
We don't understand how this works. We know it works, but we don't understand
Dwarkesh:
How it works. We have evidence from actually, of all things,
Dwarkesh:
primatology of what could be going on here, or at least like why similar patterns
Dwarkesh:
in other parts of the world.
Dwarkesh:
So what I found really interesting, There's this research by this researcher,
Dwarkesh:
Susanna Herculana Huzel,
Dwarkesh:
which shows that if you look at how the number of neurons in the brain of a rat,
Dwarkesh:
different kinds of rat species increases, as the weight of their brains increase
Dwarkesh:
from species to species, there's this very sublinear pattern.
Dwarkesh:
So if their brain size doubles, the neuron count will not double between different rat species.
Dwarkesh:
And there's other animals where there's other kinds of...
Dwarkesh:
Families of species for which this is true. The two interesting exceptions to
Dwarkesh:
this rule, where there is actually a linear increase in neuron count and brain
Dwarkesh:
size, is one, certain kinds of birds.
Dwarkesh:
So, you know, birds are actually very smart, given the size of their brains, and primates.
Dwarkesh:
So the theory for what happened with humans is that we unlocked an architecture that was very scalable.
Dwarkesh:
So the way people talk about transformers being more scalable and then LSTMs,
Dwarkesh:
the thing that preceded them in 2018.
Dwarkesh:
We unlocked this architecture as it's very scalable.
Dwarkesh:
And then we were in an evolutionary niche millions of years ago,
Dwarkesh:
which rewarded marginal increases in intelligence.
Dwarkesh:
If you get slightly smarter, yes, the brain costs more energy,
Dwarkesh:
but you can save energy in terms of like not having to, you can cook,
Dwarkesh:
you can cook food so you don't have to spend much more on digestion.
Dwarkesh:
You can find a game, you can find different ways of foraging.
Dwarkesh:
Birds were not able to find this evolutionary niche, which rewarded the incremental
Dwarkesh:
increases in intelligence because if your brain gets too heavy as a bird, you're not going to fly.
Dwarkesh:
So it was this happy coincidence of these two things. Now, why is it the case
Dwarkesh:
that the fact that our brains could get bigger resulted in us becoming as smart
Dwarkesh:
as we are? We still don't know.
Dwarkesh:
And there's many different dissimilarities between AIs and humans.
Dwarkesh:
While our brains are quite big, we don't need to be trained.
Dwarkesh:
You know, a human from the age they're zero to 18 is not seeing within an order
Dwarkesh:
of magnitude of the amount of information these LLMs are trained on.
Dwarkesh:
So LLMs are extremely data inefficient.
Dwarkesh:
They need a lot more data, but the pattern of scaling, I think we see in many different places.
Ryan Sean Adams:
So is that a fair kind of analog? This analog has always made sense to me.
Ryan Sean Adams:
It's just like transformers are like neurons.
Ryan Sean Adams:
You know, AI models are sort of like the human brain.
Ryan Sean Adams:
Evolutionary pressures are like gradient descent, reward algorithms and out
Ryan Sean Adams:
pops human intelligence. We don't really understand that.
Ryan Sean Adams:
We also don't understand AI intelligence, but it's basically the same principle at work.
Dwarkesh:
I think it's a super fascinating, but also very thorny question because is gradient
Dwarkesh:
intelligence like evolution?
Dwarkesh:
Well, yes, in one sense. But also when we do gradient descent on these models,
Dwarkesh:
we start off with the weights and then we're, you know, it's like learning how
Dwarkesh:
does chemistry work, how does coding work, how does math work.
Dwarkesh:
And that's actually more similar to lifetime learning, which is to say that,
Dwarkesh:
like, by the time you're already born to the time you turn 18 or 25,
Dwarkesh:
the things you learn, and that's not evolution.
Dwarkesh:
Evolution designed the system or the brain by which you can do that learning,
Dwarkesh:
but the lifetime learning itself is not evolution. And so there's also this
Dwarkesh:
interesting question of, yeah, is training more like evolution?
Dwarkesh:
In which case, actually, we might be very far from AGI because the amount of
Dwarkesh:
compute that's been spent over the course of evolution to discover the human
Dwarkesh:
brain, you know, could be like 10 to the 40 flops. There's been estimates, you know, whatever.
Dwarkesh:
I'm sure it will bore you to discover, talk about how these estimates are derived,
Dwarkesh:
but just like how much versus is it like a single lifetime,
Dwarkesh:
like going from the age of zero to the age of 18, which is closer to,
Dwarkesh:
I think, 10 to the 24 flops, which is actually less than compute than we use
Dwarkesh:
to train frontier systems.
Dwarkesh:
All right, anyways, we'll get back to more relevant questions.
Ryan Sean Adams:
Well, here's kind of a big picture question as well.
Ryan Sean Adams:
It's like I'm constantly fascinated with the metaphysical types of discussions
Ryan Sean Adams:
that some AI researchers kind of take.
Ryan Sean Adams:
Like a lot of AI researchers will talk in terms of when they describe what they're
Ryan Sean Adams:
making, we're making God.
Ryan Sean Adams:
Like why do they say things like that? What is this talk of like making God?
Ryan Sean Adams:
What does that mean? Is it just the idea that scaling laws don't cease?
Ryan Sean Adams:
And if we can, you know, scale intelligence to AGI, then there's no reason we
Ryan Sean Adams:
can't scale far beyond that and create some sort of a godlike entity.
Ryan Sean Adams:
And essentially, that's what the quest is. We're making artificial superintelligence.
Ryan Sean Adams:
We're making a god. We're making god.
Dwarkesh:
I think people focus too much on when they, I think this God discussion focuses
Dwarkesh:
too much on the hypothetical intelligence of a single copy of an AI.
Dwarkesh:
I do believe in the notion of a super intelligence, which is not just functionally,
Dwarkesh:
which is not just like, oh, it knows a lot of things, but is actually qualitatively
Dwarkesh:
different than human society.
Dwarkesh:
But the reason is not because I think it's so powerful that any one individual
Dwarkesh:
copy of AI will be as smart, but because of the collective advantages that AIs
Dwarkesh:
will have, which have nothing to do with their raw intelligence,
Dwarkesh:
but rather the fact that these models will be digital or they already are digital,
Dwarkesh:
but eventually they'll be as smart as humans at least.
Dwarkesh:
But unlike humans, because of our biological constraints, these models can be copied.
Dwarkesh:
If there's a model that has learned a lot about a specific domain,
Dwarkesh:
you can make infinite copies of it.
Dwarkesh:
And now you have an infinite copies of Jeff Dean or Ilya Satskova or Elon Musk
Dwarkesh:
or any skilled person you can think of.
Dwarkesh:
They can be merged. So the knowledge that each copy is learning can be amalgamated
Dwarkesh:
back into the model and then back to all the copies.
Dwarkesh:
They can be distilled. They can run at superhuman speeds.
Dwarkesh:
These collective advantages, also they can communicate in latent space.
Dwarkesh:
These collective advantages.
Ryan Sean Adams:
They're immortal. I mean, you know, as an example.
Dwarkesh:
Yes, exactly. No, I mean, that's actually, tell me if I'm rabbit holing too
Dwarkesh:
much, but like one really interesting question will come about is how do we prosecute AIs?
Dwarkesh:
Because the way we prosecute humans is that we will throw you in jail if you commit a crime.
Dwarkesh:
But if there's trillions of copies or thousands of copies of an AI model,
Dwarkesh:
if a copy of an AI model, if an instance of an AI model does something bad, what do you do?
Dwarkesh:
Does the whole model have to get, and how do you even punish a model,
Dwarkesh:
right? Like, does it care about its weights being squandered?
Dwarkesh:
Yeah, there's all kinds of questions that arise because of the nature of what AIs are.
Dwarkesh Patel:
And also who is liable for that, right?
Dwarkesh:
Like, is it the toolmaker?
Dwarkesh Patel:
Is it the person using the tool? Who is responsible for these things?
Dwarkesh Patel:
There's one topic that I do want to come to here about scaling laws,
Dwarkesh Patel:
At what time did we realize that scaling laws were going to work?
Dwarkesh Patel:
Because there were a lot of theses early in the days, early 2000s about AI,
Dwarkesh Patel:
how we were going to build better models.
Dwarkesh Patel:
Eventually, we got to the transformer. But at what point did researchers and
Dwarkesh Patel:
engineers start to realize that, hey, this is the correct idea.
Dwarkesh Patel:
We should start throwing lots of money and resources towards this versus other
Dwarkesh Patel:
ideas that were just kind of theoretical research ideas, but never really took off.
Dwarkesh Patel:
We kind of saw this with GPT two to three, where there's this huge improvement.
Dwarkesh:
A lot of.
Dwarkesh Patel:
Resources went into it. Was there a specific moment in time or a specific breakthrough
Dwarkesh Patel:
that led to the start of these scaling laws?
Dwarkesh:
I think it's been a slow process of more and more people appreciating this nature
Dwarkesh:
of the overwhelming role of compute in driving forward progress.
Dwarkesh:
In 2018, I believe, Dario Amadei wrote a memo that was secret while he was at
Dwarkesh:
OpenAI. Now he's the CEO of Anthropic.
Dwarkesh:
But while he's at OpenAI, he's subsequently revealed on my podcast that he wrote
Dwarkesh:
this memo where the title of the memo was called Big Blob of Compute.
Dwarkesh:
And it says basically what you expect it to say, which is that like,
Dwarkesh:
yes, there's ways you can mess up the process of training. You have the wrong
Dwarkesh:
kinds of data or initializations.
Dwarkesh:
But fundamentally, AGI is just a big blob of compute.
Dwarkesh:
And then we've gotten over the subsequent years, there was more empirical evidence.
Dwarkesh:
So a big update, I think it was 2021.
Dwarkesh:
Correct me. Somebody definitely will correct me in the comments.
Dwarkesh:
I'm wrong. There were these,
Dwarkesh:
there's been multiple papers of these scaling laws where you can show that the
Dwarkesh:
loss of the model on the objective of predicting the next token goes down very predictably,
Dwarkesh:
almost to like multiple decimal places of correctness based on how much more
Dwarkesh:
compute you throw in these models.
Dwarkesh:
And the compute itself is a function of the amount of data you use and how big
Dwarkesh:
the model is, how many parameters it has.
Dwarkesh:
And so that was an incredibly strong evidence back in the day,
Dwarkesh:
a couple of years ago, because then you could say, well, OK,
Dwarkesh:
if it really has this incredibly low loss of predicting the next token in all
Dwarkesh:
human output, including scientific papers, including GitHub repositories.
Dwarkesh:
Then doesn't it mean it has actually had to learn coding and science and all
Dwarkesh:
these skills in order to make those predictions, which actually ended up being true.
Dwarkesh:
And it was it was something people, you know, we take it for granted now,
Dwarkesh:
but it actually even as of a year or two ago, people were really even denying that premise.
Dwarkesh:
But some people a couple of years ago just like thought about it and like,
Dwarkesh:
yeah, actually, that would mean that it's learned the skills.
Dwarkesh:
And that's crazy that we just have this strong empirical pattern that tells
Dwarkesh:
us exactly what we need to do in order to learn these skills.
Dwarkesh Patel:
And it creates this weird perception, right, where like very early on and so
Dwarkesh Patel:
to this day, it really is just a token predictor, right?
Dwarkesh Patel:
Like we're just predicting the next word in the sentence. But somewhere along
Dwarkesh Patel:
the lines, it actually creates this perception of intelligence.
Dwarkesh Patel:
So I guess we covered the early historical context. I kind of want to bring
Dwarkesh Patel:
the listeners up to today, where we are currently, where the scaling laws have
Dwarkesh Patel:
brought us in the year 2025.
Dwarkesh Patel:
So can you kind of outline where we've gotten to from early days of GPTs to
Dwarkesh Patel:
now we have GPT-4, we have Gemini Ultra, we have Club, which you mentioned earlier.
Dwarkesh Patel:
We had the breakthrough of reasoning.
Dwarkesh Patel:
So what can leading frontier models do today?
Dwarkesh:
So there's what they can do. And then there's the question of what methods seem to be working.
Dwarkesh:
I guess we can start at what they seem to be able to do. They've shown to be
Dwarkesh:
remarkably useful at coding and not just at answering direct questions about
Dwarkesh:
how does this line of code work or something.
Dwarkesh:
But genuinely just autonomously working for 30 minutes or an hour,
Dwarkesh:
doing the task, it would take a front-end developer a whole day to do.
Dwarkesh:
And you can just ask them at a high level, do this kind of thing,
Dwarkesh:
and they can go ahead and do it.
Dwarkesh:
Obviously, if you've played around with it, you know that they're extremely
Dwarkesh:
useful assistants in terms of research, in terms of even therapists,
Dwarkesh:
whatever other use cases.
Dwarkesh:
On the question of what training methods seem to be working,
Dwarkesh:
we do seem to be getting evidence that pre-training is plateauing,
Dwarkesh:
which is to say that we had GPT 4.5, which was just following this old mold
Dwarkesh:
of make the model bigger,
Dwarkesh:
but it's fundamentally doing the same thing of next token prediction.
Dwarkesh:
And apparently it didn't pass muster. The OpenAI had to deprecate it because
Dwarkesh:
there's this dynamic where the bigger the model is, the more it costs not only
Dwarkesh:
to train, but also to serve, right?
Dwarkesh:
Because every time you serve a user, you're having to run the whole model,
Dwarkesh:
which is going, so, but that doesn't be working is RL, which is this process
Dwarkesh:
of, not just training them on existing tokens on the internet,
Dwarkesh:
but having the model itself try to answer math and coding problems.
Dwarkesh:
And finally, we got to the point where the model is smart enough to get it right
Dwarkesh:
some of the time, and so you can give it some reward, and then it can saturate
Dwarkesh:
these tough reasoning problems.
Dwarkesh Patel:
And then what was the breakthrough with reasoning for the people who aren't familiar?
Dwarkesh Patel:
What made reasoning so special that we hadn't discovered before?
Dwarkesh Patel:
And what did that kind of unlock for models that we use today?
Dwarkesh:
I'm honestly not sure. I mean, GBD-4 came out a little over two years ago,
Dwarkesh:
and then it was after two years after GPT-4 came out that O-1 came out which
Dwarkesh:
was the original reasoning breakthrough I think last November and,
Dwarkesh:
And subsequently, a couple of months later, DeepSeq showed in their R1 paper.
Dwarkesh:
So DeepSeq open source their research and they explained exactly how their algorithm worked.
Dwarkesh:
And it wasn't that complicated. It was just like what you would expect,
Dwarkesh:
which is get some math problems,
Dwarkesh:
give for some initial problems, tell the model exactly what the reasoning trace
Dwarkesh:
looks like, how you solve it, just like write it out and then have the model
Dwarkesh:
like try to do it raw on the remaining problems.
Dwarkesh:
Now, I know it sounds incredibly arrogant to say, well, it wasn't that complicated.
Dwarkesh:
Why did it take you years?
Dwarkesh:
I think there's an interesting insight there of even things which you think
Dwarkesh:
will be simple in terms of high level description of how to solve the problem
Dwarkesh:
end up taking longer in terms of haggling out the remaining engineering hurdles
Dwarkesh:
than you might naively assume.
Dwarkesh:
And that should update us on how long it will take us to go through the remaining
Dwarkesh:
bottlenecks on the path to AGI.
Dwarkesh:
Maybe that will be tougher than people imagine, especially the people who think
Dwarkesh:
we're only two to three years away.
Dwarkesh:
But all this to say, yeah, I'm not sure why it took so long after GPT-4 to get
Dwarkesh:
a model trained on a similar level of capabilities that could then do reasoning.
Dwarkesh Patel:
And in terms of those abilities, the first answer you had to what can it do was coding.
Dwarkesh Patel:
And I hear that a lot of the time when I talk to a lot of people is that coding
Dwarkesh Patel:
seems to be a really strong suit and a really huge unlock to using these models.
Dwarkesh Patel:
And I'm curious, why coding over general intelligence?
Dwarkesh Patel:
Is it because it's placed in a more confined box of parameters?
Dwarkesh Patel:
I know in the early days, we had the AlphaGo and And we had the AIs playing
Dwarkesh Patel:
chess and they exceed, they perform so well because they were kind of contained
Dwarkesh Patel:
within this box of parameters that was a little less open-ended than general intelligence.
Dwarkesh Patel:
Is that the reason why coding is kind of at the frontier right now of the ability of these models?
Dwarkesh:
There's two different hypotheses. One is based around this idea called Moravac's paradox.
Dwarkesh:
And this was an idea, by the way, one super interesting figure,
Dwarkesh:
actually, I should have mentioned him earlier.
Dwarkesh:
One super interesting figure in the history of scaling is Hans Moravac,
Dwarkesh:
who I think in the 90s predicts that 2028 will be the year that we will get to AGI.
Dwarkesh:
And the way he predicts this, which is like, you know, we'll see what happens,
Dwarkesh:
but like not that far off the money as far as I'm concerned.
Dwarkesh:
The way he predicts this is he just looks at the growth in computing power year
Dwarkesh:
over year and then looks at how much compute he estimated the human brain to be to require.
Dwarkesh:
And just like, OK, we'll have computers as powerful as the human brain by 2028.
Dwarkesh:
Which is like at once a deceptively simple argument, but also ended up being
Dwarkesh:
incredibly accurate and like worked, right?
Dwarkesh:
I might add a fact drive it was 2028, but it was within that,
Dwarkesh:
like within something you would consider a reasonable guess, given what we know now.
Dwarkesh:
Sorry, anyway, so the Morrowind's paradox is this idea that computers seemed
Dwarkesh:
in AI get better first at the skills which humans are the worst at.
Dwarkesh:
Or at least there's a huge variation in the human repertoire.
Dwarkesh:
So we think of coding as incredibly hard, right? We think this is like the top
Dwarkesh:
1% of people will be excellent coders.
Dwarkesh:
We also think of reasoning as very hard, right? So if you like read Aristotle,
Dwarkesh:
he says, the thing which makes humans special, which distinguishes us from animals is reasoning.
Dwarkesh:
And these models aren't that useful yet at almost anything. The one thing they can do is reasoning.
Dwarkesh:
So how do we explain this pattern? And Moravec's answer is that evolution has
Dwarkesh:
spent billions of years optimizing us to do things we take for granted.
Dwarkesh:
Move around this room, right? I can pick up this can of Coke,
Dwarkesh:
move it around, drink from it.
Dwarkesh:
And that we can't even get robots to do at all yet.
Dwarkesh:
And in fact, it's so ingrained in us by evolution that there's no human, or.
Ryan Sean Adams:
At least humans who don't have
Dwarkesh:
Disabilities will all be able to do this. And so we just take it for granted
Dwarkesh:
that this is an easy thing to do.
Dwarkesh:
But in fact, it's evidence of how long evolution has spent getting humans up to this point.
Dwarkesh:
Whereas reasoning, logic, all of these skills have only been optimized by evolution
Dwarkesh:
over the course of the last few million years.
Dwarkesh:
So there's been a thousand fold less evolutionary pressure towards coding than
Dwarkesh:
towards just basic locomotion.
Dwarkesh:
And this has actually been very accurate in predicting what kinds of progress
Dwarkesh:
we see even before we got deep learning, right?
Dwarkesh:
Like in the 40s when we got our first computers, the first thing that we could
Dwarkesh:
use them to do is long calculations for ballistic trajectories at the time for World War II.
Dwarkesh:
Humans suck at long calculations by hand.
Dwarkesh:
And anyways, so that's the explanation for coding, which seems hard for humans,
Dwarkesh:
is the first thing that went to AIs.
Dwarkesh:
Now, there's another theory, which is that this is actually totally wrong.
Dwarkesh:
It has nothing to do with the seeming paradox of how long evolution has optimized
Dwarkesh:
us for, and everything to do with the availability of data.
Dwarkesh:
So we have GitHub, this repository of all of human code, at least all open source
Dwarkesh:
code written in all these different languages, trillions and trillions of tokens.
Dwarkesh:
We don't have an analogous thing for robotics. We don't have this pre-training
Dwarkesh:
corpus. And that explains why code has made so much more progress than robotics.
Ryan Sean Adams:
That's fascinating because if there's one thing that I could list that we'd
Ryan Sean Adams:
want AI to be good at, probably coding software is number one on that list.
Ryan Sean Adams:
Because if you have a Turing complete intelligence that can create Turing complete
Ryan Sean Adams:
software, is there anything you can't create once you have that?
Ryan Sean Adams:
Also, like the idea of Morvac's paradox, I guess that sort of implies a certain
Ryan Sean Adams:
complementarianism with humanity.
Ryan Sean Adams:
So if robots can do things that robots can do really well and can't do the things
Ryan Sean Adams:
humans can do well, well, perhaps there's a place for us in this world.
Ryan Sean Adams:
And that's fantastic news. It also maybe implies that humans have kind of scratched
Ryan Sean Adams:
the surface on reasoning potential.
Ryan Sean Adams:
I mean, if we've only had a couple of million years of evolution and we haven't
Ryan Sean Adams:
had the data set to actually get really good at reasoning, it seems like there'd
Ryan Sean Adams:
be a massive amount of upside, unexplored territory,
Ryan Sean Adams:
like so much more intelligence that nature could actually
Ryan Sean Adams:
contain inside of reasoning.
Ryan Sean Adams:
I mean, are these some of the implications of these ideas?
Dwarkesh:
Yeah, I know. I mean, that's a great insight. Another really interesting insight
Dwarkesh:
is that the more variation there
Dwarkesh:
is in a skill in humans, the better and faster that AIs will get at it.
Dwarkesh:
Because coding is the kind of thing where 1% of humans are really good at it.
Dwarkesh:
The rest of us will, if we try to learn it, we'd be okay at it or something, right?
Dwarkesh:
And because evolutionists spend so little time optimizing us,
Dwarkesh:
there's this room for variation where the optimization hasn't happened uniformly
Dwarkesh:
or it hasn't been valuable enough to saturate the human gene pool for this skill.
Dwarkesh:
I think you made an earlier point that I thought was really interesting I wanted
Dwarkesh:
to address. Can you remind me of the first thing you said? Is it the complementarianism? Yes.
Dwarkesh:
So you can take it as a positive future. You can take it as a negative future
Dwarkesh:
in the sense that, well, what is the complementary skills we're providing?
Dwarkesh:
We're good meat robots.
Ryan Sean Adams:
Yeah, the low skilled labor of the situation.
Dwarkesh:
We can do all the thinking and planning. One dark future,
Dwarkesh:
one dark vision of the future is we'll get those meta glasses
Dwarkesh:
and the AI speaking into our ear and it'll tell us to go put this brick over
Dwarkesh:
there so that the next data center couldn't be built because the AI's got the
Dwarkesh:
plan for everything. It's got the better design for the ship and everything.
Dwarkesh:
You just need to move things around for it. And that's what human labor looks
Dwarkesh:
like until robotics is solved.
Dwarkesh:
So yeah, it depends on how you... On the other hand, you'll get paid a lot because
Dwarkesh:
it's worth a lot to move those bricks. We're building AGI here.
Dwarkesh:
But yeah, it depends on how you come out of that question.
Ryan Sean Adams:
Well, there seems to be something to that idea, going back to the idea of the
Ryan Sean Adams:
massive amount of human variation.
Ryan Sean Adams:
I mean, we have just in the past month or so, we have news of meta hiring AI
Ryan Sean Adams:
researchers for $100 million signing bonuses, okay?
Ryan Sean Adams:
What does the average software engineer make versus what does an AI researcher
Ryan Sean Adams:
make at kind of the top of the market, right?
Ryan Sean Adams:
Which has got to imply, obviously there's some things going on with demand and
Ryan Sean Adams:
supply, but also that it does also seem to imply that there's massive variation
Ryan Sean Adams:
in the quality of a software engineer.
Ryan Sean Adams:
And if AIs can get to that quality, well, what does that unlock?
Ryan Sean Adams:
Yeah. So, okay. Yeah. So I guess we have like coding down right now.
Ryan Sean Adams:
Like another question though is like, what can't AIs do today?
Ryan Sean Adams:
And how would you characterize that? Like what are the things they just don't do well?
Dwarkesh:
So I've been interviewing people on my podcast who have very different timelines
Dwarkesh:
from a role to get to AGI. I have had people on who think it's two years away
Dwarkesh:
and some who think it's 20 years away.
Dwarkesh:
And the experience of building AI tools for myself actually has been the most
Dwarkesh:
insight driving or maybe research I've done on the question of when AI is coming.
Ryan Sean Adams:
More than the guest interviews.
Dwarkesh:
Yeah, because you just, I have had, I've probably spent on the order of a hundred
Dwarkesh:
hours trying to build these little tools. The kinds I'm sure you've also tried
Dwarkesh:
to build of like, rewrite auto-generated transcripts for me to make them sound,
Dwarkesh:
the rewritten the way a human would write them.
Dwarkesh:
Find clips for me to tweet out, write essays with me, co-write them passage
Dwarkesh:
by passage, these kinds of things.
Dwarkesh:
And what I found is that it's actually very hard to get human-like labor out
Dwarkesh:
of these models, even for tasks like these, which should be death center in
Dwarkesh:
the repertoire of these models, right?
Dwarkesh:
They're short horizon, they're language in, language out.
Dwarkesh:
They're not contingent on understanding some thing I said like a month ago.
Dwarkesh:
This is just like, this is the task.
Dwarkesh:
And I was thinking about why is it the case that I still haven't been able to
Dwarkesh:
automate these basic language tasks? Why do I still have a human work on these things?
Dwarkesh:
And I think the key reason that you can't automate even these simple tasks is
Dwarkesh:
because the models currently lack the ability to do on the job training.
Dwarkesh:
So if you hire a human for the first six months, for the first three months,
Dwarkesh:
they're not going to be that useful, even if they're very smart,
Dwarkesh:
because they haven't built up the context, they haven't practiced the skills,
Dwarkesh:
they don't understand how the business works.
Dwarkesh:
What makes humans valuable is not that mainly the raw intellect obviously matters,
Dwarkesh:
but it's not mainly that.
Dwarkesh:
It's their ability to interrogate their own failures in this really dynamic,
Dwarkesh:
organic way to pick up small efficiencies and improvements as they practice
Dwarkesh:
the task and to build up this context as they work within a domain.
Dwarkesh:
And so sometimes people wonder, look, if you look at the revenue of OpenAI,
Dwarkesh:
the annual recurring revenue, it's on the order of $10 billion.
Dwarkesh:
Kohl's makes more money than that. McDonald's makes more money than that, right?
Dwarkesh:
So why is it that if they've got AGI, they're, you know, like Fortune 500 isn't
Dwarkesh:
reorganizing their workflows to, you know, use open AI models at every layer of the stack?
Dwarkesh:
My answer, sometimes people say, well, it's because people are too stodgy.
Dwarkesh:
The management of these companies is like not moving fast enough on AI.
Dwarkesh:
That could be part of it. I think mostly it's not that.
Dwarkesh:
I think mostly it genuinely is very hard to get human-like labor out of these
Dwarkesh:
models because you can't.
Dwarkesh:
So you're stuck with the capabilities you get out of the model out of the box.
Dwarkesh:
So they might be five out of 10 at rewriting the transcript for you.
Dwarkesh:
But if you don't like how it turned out, if you have feedback for it,
Dwarkesh:
if you want to keep teaching it over time, once the session ends,
Dwarkesh:
the model, like everything it knows about you has gone away.
Dwarkesh:
You got to restart again. It's like working with an amnesiac employee.
Dwarkesh:
You got to restart again.
Ryan Sean Adams:
Every day is the first day of employment, basically.
Dwarkesh:
Yeah, exactly. It's a groundhog day for them every day or every couple of hours, in fact.
Dwarkesh:
And that makes it very hard for them to be that useful as an employee,
Dwarkesh:
right? They're not really an employee at that point.
Dwarkesh:
This, I think, not only is a key bottleneck to the value of these models,
Dwarkesh:
because human labor is worth a lot, right?
Dwarkesh:
Like $60 trillion in the world is paid to wages every year.
Dwarkesh:
If these model companies are making on the order of $10 billion a year, that's a big way to AGI.
Dwarkesh:
And what explains that gap? What are the bottlenecks? I think a big one is this
Dwarkesh:
continual learning thing.
Dwarkesh:
And I don't see an easy way that that just gets solved within these models.
Dwarkesh:
There's no like, with reasoning, you could say, oh, it's like train it on math
Dwarkesh:
and code problems, and then I'll get the reasoning. And that worked.
Dwarkesh:
I don't think there's something super obvious there for how do you get this
Dwarkesh:
online learning, this on-the-job training working for these models.
Ryan Sean Adams:
Okay, can we talk about that, go a little bit deeper on that concept?
Ryan Sean Adams:
So this is basically one of the concepts you wrote in your recent post.
Ryan Sean Adams:
AI is not right around the corner. Even though you're an AI optimist,
Ryan Sean Adams:
I would say, and overall an AI accelerationist, you You were saying it's not
Ryan Sean Adams:
right around the corner.
Ryan Sean Adams:
You're saying the ability to replace human labor is a ways out.
Ryan Sean Adams:
Not forever out, but I think you said somewhere around 2032,
Ryan Sean Adams:
if you had to guess on when the estimate was.
Ryan Sean Adams:
And the reason you gave is because AIs can't learn on the job,
Ryan Sean Adams:
but it's not clear to me why they can't.
Ryan Sean Adams:
Is it just because the context window isn't large enough?
Ryan Sean Adams:
Is it just because they can't input all of the different data sets and data
Ryan Sean Adams:
points that humans can? Is it because they don't have stateful memory the way a human employee?
Ryan Sean Adams:
Because if it's these things, all of these do seem like solvable problems.
Ryan Sean Adams:
And maybe that's what you're saying. They are solvable problems.
Ryan Sean Adams:
They're just a little bit longer than some people think they are.
Dwarkesh:
I think it's like in some deep sense a solvable problem because eventually we will build AGI.
Dwarkesh:
And to build AGI, we will have had to solve the problem.
Dwarkesh:
My point is that the obvious solutions you might imagine, for example,
Dwarkesh:
expanding the context window or having this
Dwarkesh:
like external memory using systems like rag these
Dwarkesh:
are basically techniques we already have to it's called retrieval augmented
Dwarkesh:
generate anyways these kinds of retrieval augmented generation i
Dwarkesh:
don't think these will suffice and just to put a finer point first of all like
Dwarkesh:
what is the problem the problem is exactly as you say that within the context
Dwarkesh:
window these models actually can learn on the job right so if you talk to it
Dwarkesh:
for long enough it will get much better at understanding your needs and what your exact problem is.
Dwarkesh:
If you're using it for research for your podcast, it will get a sense of like,
Dwarkesh:
oh, they're actually especially curious about these kinds of questions. Let me focus on that.
Dwarkesh:
It's actually very human-like in context, right? The speed at which it learns,
Dwarkesh:
the task of knowledge it picks out.
Dwarkesh:
The problem, of course, is the context length for even the best models only
Dwarkesh:
last a million or two million tokens.
Dwarkesh:
That's at most like an hour of conversation.
Dwarkesh:
Now, then you might say, okay, well, why can't we just solve that by expanding
Dwarkesh:
the context window, right? So context window has been expanding for the last
Dwarkesh:
few years. Why can't we just continue that?
Ryan Sean Adams:
Yeah, like a billion token context window, something like this.
Dwarkesh:
So 2018 is when the transformer came out and the transformer has the attention mechanism.
Dwarkesh:
The attention mechanism is inherently quadratic with the nature of the length
Dwarkesh:
of the sequence, which is to say that if you go from if you double go from 1
Dwarkesh:
million tokens to 2 million tokens,
Dwarkesh:
it actually costs four times as much compute to process that 2 millionth token.
Dwarkesh:
It's not just 2 to as much compute. so it gets super linearly more expensive
Dwarkesh:
as you increase the context length and for the last,
Dwarkesh:
seven years people have been trying to get around this inherent quadratic nature
Dwarkesh:
of attention of course we don't know secretly what the labs are working on but we have frontier,
Dwarkesh:
companies like deep seek which have open source their research and
Dwarkesh:
we can just see how their algorithms work and they found
Dwarkesh:
these constant time modifiers to attention which is
Dwarkesh:
to say that they there's like a it'll still
Dwarkesh:
be quadratic but it'll be like one half times
Dwarkesh:
quadratic but the inherent like super linearness has not
Dwarkesh:
gone away and because of that yeah you might be able to increase it from 1 million
Dwarkesh:
tokens to 2 million tokens by finding another hack like uh make sure experts
Dwarkesh:
just run such things latent attention is another such technique but or kbcash
Dwarkesh:
right there's many other things that have been discovered but people have not
Dwarkesh:
discovered okay how do you get around the fact that if you went to a billion,
Dwarkesh:
it would be a billion squared as expensive in terms of compute to process that token.
Dwarkesh:
And so I don't think you'll just get it by increasing the length of the context window, basically.
Ryan Sean Adams:
That's fascinating. Yeah, I didn't realize that. Okay, so the other reason in
Ryan Sean Adams:
your post that AI is not right around the corner is because it can't do your taxes.
Ryan Sean Adams:
And Dwarkesh, I feel your pain, man. Taxes are just like quite a pain in the ass.
Ryan Sean Adams:
I think you were talking about this from the context of like computer vision,
Ryan Sean Adams:
computer use, that kind of thing.
Ryan Sean Adams:
So, I mean, I've seen demos. I've seen some pretty interesting computer vision
Ryan Sean Adams:
sort of demos that seem to be right around the corner.
Ryan Sean Adams:
But what's the limiter on computer use for an AI?
Dwarkesh:
There was an interesting blog post by this company called Mechanize where they
Dwarkesh:
were explaining why this is such a big problem. And I love the way they phrased it, which is that,
Dwarkesh:
Imagine if you had to train a model in 1980, a large language model in 1980,
Dwarkesh:
and you could use all the compute you wanted in 1980 somehow,
Dwarkesh:
but you didn't have, you were only stuck with the data that was available in
Dwarkesh:
the 1980s, of course, before the internet became a widespread phenomenon.
Dwarkesh:
You couldn't train a modern LLM, even with all the computer in the world,
Dwarkesh:
because the data wasn't available.
Dwarkesh:
And we're in a similar position with respect to computer use,
Dwarkesh:
because there's not this corpus of collected videos, people using computers
Dwarkesh:
to do different things, to access different applications and do white collar work.
Dwarkesh:
Because of that, I think the big challenge has been accumulating this kind of data. off.
Ryan Sean Adams:
And to be clear, when I was saying the use case of like, do my taxes,
Ryan Sean Adams:
you're effectively talking about an AI having the ability to just like,
Ryan Sean Adams:
you know, navigate the files around your computer,
Ryan Sean Adams:
you know, log in to various websites to download your pay stubs or whatever,
Ryan Sean Adams:
and then to go to like TurboTax or something and like input it all into some
Ryan Sean Adams:
software and file it, right?
Ryan Sean Adams:
Just on voice command or something like that. That's basically doing my taxes.
Dwarkesh:
It should be capable of navigating UIs that it's less familiar with or that
Dwarkesh:
come about organically within the context of trying to solve a problem.
Dwarkesh:
So for example, I might have business deductions.
Dwarkesh:
It sees on my bank statement that I've spent $1,000 on Amazon.
Dwarkesh:
It goes logs in my Amazon.
Dwarkesh:
It sees like, oh, he bought a camera. So I think that's probably a business
Dwarkesh:
expense for his podcast.
Dwarkesh:
He bought an Airbnb over a weekend in the cabins of whatever,
Dwarkesh:
in the woods of whatever. That probably wasn't a business expense.
Dwarkesh:
Although maybe, maybe it's, if it's a sort of like a gray, if it's willing to
Dwarkesh:
go in the gray area, maybe I'll talk to you. Yeah, yeah, yeah.
Ryan Sean Adams:
Do the gray area stuff.
Dwarkesh:
I was, I was researching.
Dwarkesh:
But anyway, so that, including all of that, including emailing people for invoices,
Dwarkesh:
and haggling with them, it would be like a sort of week long task to do my taxes, right?
Dwarkesh:
You'd have to, there's a lot of work involved. That's not just like do this
Dwarkesh:
skill, this skill, this skill, but rather of having a sort of like plan of action
Dwarkesh:
and then breaking tasks apart, dealing with new information,
Dwarkesh:
new emails, new messages, consulting with me about questions, et cetera.
Ryan Sean Adams:
Yeah, I mean, to be clear on this use case too, even though your post is titled
Ryan Sean Adams:
like, you know, AI is not right around the corner, you still think this ability
Ryan Sean Adams:
to file your taxes, that's like a 2028 thing, right?
Ryan Sean Adams:
I mean, this is maybe not next year, but it's in a few years.
Dwarkesh:
Right, which is, I think that was sort of, people maybe write too much in The
Dwarkesh:
Decital and then didn't read through the arguments.
Ryan Sean Adams:
I mean, that never happens on the internet. Wow.
Dwarkesh:
First time.
Dwarkesh:
No, I think like I'm arguing against people who are like, you know, this will happen.
Dwarkesh:
AGI is like two years away. I do think the wider world, the markets,
Dwarkesh:
public perception, even people who are somewhat attending to AI,
Dwarkesh:
but aren't in this specific milieu that I'm talking to, are way underpricing AGI.
Dwarkesh:
One reason, one thing I think they're underestimating is not only will we have
Dwarkesh:
millions of extra laborers, millions of extra workers,
Dwarkesh:
potentially billions within the course of the next decade, because then we will
Dwarkesh:
have a potentially, I think like likely we will have AGI within the next decade.
Dwarkesh:
But they'll have these advantages that human workers don't have,
Dwarkesh:
which is that, okay, a single model company, so suppose we solve continual learning, right?
Dwarkesh:
So there, and we saw computer use. So as far as white collar work goes,
Dwarkesh:
that might fundamentally it would be solved.
Dwarkesh:
You can have AIs which can use not just they're not just like a text box where
Dwarkesh:
you put into you ask questions in a chatbot and you get some response out.
Dwarkesh:
It's not that useful to just have a very smart chatbot. You need it to be able
Dwarkesh:
to actually do real work and use real applications.
Dwarkesh:
Suppose you have that solved because it acts like an employee.
Dwarkesh:
It's got continual learning. It's got computer use.
Dwarkesh:
But it has another advantage that humans don't have, which is that copies of
Dwarkesh:
this model are going being deployed all through the economy and it's doing on the job training.
Dwarkesh:
So copies are learning how to be an accountant, how to be a lawyer,
Dwarkesh:
how to be a coder, except because it's an AI and it's digital,
Dwarkesh:
the model itself can amalgamate all this on-the-job training from all these copies.
Dwarkesh:
So what does that mean? Well, it means that even if there's no more software
Dwarkesh:
progress after that point, which is to say that no more algorithms are discovered,
Dwarkesh:
there's not a transformer plus plus that's discovered.
Dwarkesh:
Just from the fact that this model is learning every single skill in the economy,
Dwarkesh:
at least for white-collar work, you might just, based on that alone,
Dwarkesh:
have something that looks like an intelligence explosion.
Dwarkesh:
It would just be a broadly deployed intelligence explosion, but it would functionally
Dwarkesh:
become super intelligent just from having human-level capability of learning on the job.
Dwarkesh Patel:
Yeah, and it creates this mesh network of intelligence that's shared among everyone.
Dwarkesh Patel:
That's a really fascinating thing. So we're going to get there.
Dwarkesh Patel:
We're going to get to AGI. it's going to be incredibly smart.
Dwarkesh Patel:
But what we've shared recently is just kind of this mixed bag where currently
Dwarkesh Patel:
today, it's pretty good at some things, but also not that great at others.
Dwarkesh Patel:
We're hiring humans to do jobs that we think AI should do, but it probably doesn't.
Dwarkesh Patel:
So the question I have for you is, is AI really that smart? Or is it just good
Dwarkesh Patel:
at kind of acing these particular benchmarks that we measure against?
Dwarkesh Patel:
Apple, I mean, famously recently, they had their paper, The Illusion of Thinking,
Dwarkesh Patel:
where it was kind of like, hey, AI is like pretty good up to a point,
Dwarkesh Patel:
but at a certain point, it just falls apart.
Dwarkesh Patel:
And the inference is like, maybe it's not intelligence, maybe it's just good
Dwarkesh Patel:
at guessing. So I guess the question is, is AI really that smart?
Dwarkesh:
It depends on who I'm talking to. I think some people overhype its capabilities.
Dwarkesh:
I think some people are like, oh, it's already AGI, but it's like a little hobbled
Dwarkesh:
little AGI where we're like sort of giving it a concussion every couple of hours
Dwarkesh:
and like it forgets everything.
Dwarkesh:
We're like trapped in a chatbot context. But fundamentally, the thing inside
Dwarkesh:
is like a very smart human.
Dwarkesh:
I disagree with that perspective. So if that's your perspective,
Dwarkesh:
I say like, no, it's not that smart.
Dwarkesh:
Your perspective is just statistical associations. I say definitely smarter.
Dwarkesh:
Like it's like genuinely there's an intelligence there.
Dwarkesh:
And the, so one thing you could say to the person who thinks that it's already
Dwarkesh:
AGI is this, look, if a single human had as much stuff memorized as these models
Dwarkesh:
seem to have memorized, right?
Dwarkesh:
Which is to say that they have all of internet text, everything that human has
Dwarkesh:
written on the internet memorized, they would potentially be discovering all
Dwarkesh:
kinds of connections and discoveries.
Dwarkesh:
They'd notice that this thing which causes a migraine is associated with this kind of deficiency.
Dwarkesh:
So maybe if you take the supplement, your migraines will be cured.
Dwarkesh:
There'd be just this list of just like trivial connections that lead to big
Dwarkesh:
discoveries all through the place.
Dwarkesh:
It's not clear that there's been an unambiguous case of an AI just doing this by itself.
Dwarkesh:
So then why, so that's something potentially to explain, like if they're so
Dwarkesh:
intelligent, why aren't they able to use their disproportionate capabilities,
Dwarkesh:
their unique capabilities to come up with these discoveries?
Dwarkesh:
I don't think there's actually a good answer to that question yet,
Dwarkesh:
except for the fact that they genuinely aren't that creative.
Dwarkesh:
Maybe they're like intelligent in the sense of knowing a lot of things,
Dwarkesh:
but they don't have this fluid intelligence that humans have.
Dwarkesh:
Anyway, so I give you a wish-washy answer because I think some people are underselling
Dwarkesh:
the intelligence. Some people are overselling it.
Ryan Sean Adams:
I recall a tweet lately from Tyler Cowen. I think he was referring to maybe
Ryan Sean Adams:
O3, and he basically said, it feels like AGI.
Ryan Sean Adams:
I don't know if it is AGI or not, but like to me, it feels like AGI.
Ryan Sean Adams:
What do you account for this feeling of like intelligence then
Dwarkesh:
I think this is actually very interesting because it gets to a crux that Tyler
Dwarkesh:
and I have so Tyler and I disagree on two big things one he thinks you know
Dwarkesh:
as he said in the blog post 03 is AGI I don't think it's AGI I think it's,
Dwarkesh:
it's orders of magnitude less valuable or, you know, like many orders of magnitude
Dwarkesh:
less valuable and less useful than an AGI.
Dwarkesh:
That's one thing we disagree on. The other thing we disagree on is he thinks
Dwarkesh:
that once we do get AGI, we'll only see 0.5% increase in the economic growth
Dwarkesh:
rate. This is like what the internet caused, right?
Dwarkesh:
Whereas I think we will see tens of percent increase in economic growth.
Dwarkesh:
Like it will just be the difference between the pre-industrial revolution rate
Dwarkesh:
of growth versus industrial revolution, that magnitude of change again.
Dwarkesh:
And I think these two disagreements are linked because if you do believe we're
Dwarkesh:
already at AGI and you look around the world and you say like,
Dwarkesh:
well, it fundamentally looks the same, you'd be forgiven for thinking like,
Dwarkesh:
oh, there's not that much value in getting to AGI.
Dwarkesh:
Whereas if you are like me and you think like, no, we'll get this broadly at
Dwarkesh:
the minimum, at a very minimum, we'll get a broadly deployed intelligence explosion once we get to AGI,
Dwarkesh:
then you're like, OK, I'm just expecting some sort of singulitarian crazy future
Dwarkesh:
with a robot factories and, you know, solar farms all across the desert and things like that.
Ryan Sean Adams:
Yeah, I mean, it strikes me that your disagreement with Tyler is just based
Ryan Sean Adams:
on the semantic definition of like what AGI actually is.
Ryan Sean Adams:
And Tyler, it sounds like he has kind of a lower threshold for what AGI is,
Ryan Sean Adams:
whereas you have a higher threshold.
Ryan Sean Adams:
Is there like a accepted definition for AGI?
Dwarkesh:
No. One thing that's useful for the purposes of discussions is to say automating
Dwarkesh:
all white collar work because robotics hasn't made as much progress as LLMs
Dwarkesh:
have or computer use has.
Dwarkesh:
So if we just say anything a human can do or maybe 90% of what humans can do
Dwarkesh:
at a desk, an AI can also do, that's potentially a useful definition for at
Dwarkesh:
least getting the cognitive elements relevant to defining AGI.
Dwarkesh:
But yeah, there's not one definition which suits all purposes.
Ryan Sean Adams:
Do we know what's like going on inside of these models, right?
Ryan Sean Adams:
So like, you know, Josh was talking earlier in the conversation about like this
Ryan Sean Adams:
at the base being sort of token prediction, right?
Ryan Sean Adams:
And I guess this starts to raise the question of like, what is intelligence in the first place?
Ryan Sean Adams:
And these AI models, I mean, they seem like they're intelligent,
Ryan Sean Adams:
but do they have a model of the world the way maybe a human might?
Ryan Sean Adams:
Are they sort of babbling or like, is this real reasoning?
Ryan Sean Adams:
And like, what is real reasoning? Do we just judge that based on the results
Ryan Sean Adams:
or is there some way to like peek inside of its head?
Dwarkesh:
I used to have similar questions a couple of years ago. And then,
Dwarkesh:
because honestly, the things they did at the time were like ambiguous.
Dwarkesh:
You could say, oh, it's close enough to something else in this trading data set.
Dwarkesh:
That is just basically copy pasting. It didn't come up with a solution by itself.
Dwarkesh:
But we've gotten to the point where I can come up with a pretty complicated
Dwarkesh:
math problem and it will solve it.
Dwarkesh:
It can be a math problem, like not like, you know, undergrad or high school math problem.
Dwarkesh:
Like the problem we get, the problems the smartest math professors come up with
Dwarkesh:
in order to test International Math Olympiad.
Dwarkesh:
You know, the kids who spend all their life preparing for this,
Dwarkesh:
the geniuses who spend all their life, all their young adulthood preparing to
Dwarkesh:
take these really gnarly math puzzle challenges.
Dwarkesh:
And the model will get these kinds of questions, right? They require all this
Dwarkesh:
abstract creative thinking, this reasoning for hours, the model will get the right.
Dwarkesh:
Okay, so if that's not reasoning, then why is reasoning valuable again?
Dwarkesh:
Like, what exactly was this reasoning supposed to be?
Dwarkesh:
So I think they genuinely are reasoning. I mean, I think there's other capabilities
Dwarkesh:
they lack, which are actually more, in some sense, they seem to us to be more
Dwarkesh:
trivial, but actually much harder to learn. But the reasoning itself, I think, is there.
Dwarkesh Patel:
And the answer to the intelligence question is also kind of clouded,
Dwarkesh Patel:
right? Because we still really don't understand what's going on in an LLM.
Dwarkesh Patel:
Dario from Anthropoc, he recently posted the paper about interpretation.
Dwarkesh Patel:
And can you explain why we don't even really understand what's going on in these
Dwarkesh Patel:
LLMs, even though we're able to make them and yield the results from them? Mmm.
Dwarkesh Patel:
Because it very much still is kind of like a black box. We write some code,
Dwarkesh Patel:
we put some inputs in, and we get something out, but we're not sure what happens in the middle,
Dwarkesh:
Why it's creating this output.
Dwarkesh Patel:
I mean, it's exactly what you're saying.
Dwarkesh:
It's that in other systems we engineer in the world, we have to build it up bottom-ups.
Dwarkesh:
If you build a bridge, you have to understand how every single beam is contributing to the structure.
Dwarkesh:
And we have equations for why the thing will stay standing.
Dwarkesh:
There's no such thing for AI. We didn't build it, more so we grew it.
Dwarkesh:
It's like watering a plant. And a couple thousand years ago,
Dwarkesh:
they were doing agriculture, but they didn't know why.
Dwarkesh:
Why do plants grow? How do they collect energy from sunlight? All these things.
Dwarkesh:
And I think we're in a substantially similar position with respect to intelligence,
Dwarkesh:
with respect to consciousness, with respect to all these other interesting questions
Dwarkesh:
about how minds work, which is in some sense really cool because there's this
Dwarkesh:
huge intellectual horizon that's become not only available, but accessible to investigation.
Dwarkesh:
In another sense, it's scary because we know that minds can suffer.
Dwarkesh:
We know that minds have moral worth and we're creating minds and we have no
Dwarkesh:
understanding of what's happening in these minds.
Dwarkesh:
Is a process of gradient descent a painful process?
Dwarkesh:
We don't know, but we're doing a lot of it.
Dwarkesh:
So hopefully we'll learn more. But yeah, I think we're in a similar position
Dwarkesh:
to some farmer in Uruk in 3500 BC.
Josh Kale:
Wow.
Ryan Sean Adams:
And I mean, the potential, the idea that minds can suffer, minds have some moral
Ryan Sean Adams:
worth, and also minds have some free will.
Ryan Sean Adams:
They have some sort of autonomy, or maybe at least a desire to have autonomy.
Ryan Sean Adams:
I mean, this brings us to kind of this sticky subject of alignment and AI safety
Ryan Sean Adams:
and how we go about controlling the intelligence that we're creating,
Ryan Sean Adams:
if even that's what we should be doing, controlling it. And we'll get to that in a minute.
Ryan Sean Adams:
But I want to start with maybe the headlines here a little bit.
Ryan Sean Adams:
So headline just this morning, latest OpenAI models sabotaged a shutdown mechanism
Ryan Sean Adams:
despite commands to the contrary.
Ryan Sean Adams:
OpenAI's O1 model attempted to copy itself to external servers after being threatened
Ryan Sean Adams:
with shutdown that denied the action when discovered.
Ryan Sean Adams:
I've read a number of papers for this. Of course, mainstream media has these
Ryan Sean Adams:
types of headlines almost on a weekly basis now, and it's starting to get to daily.
Ryan Sean Adams:
But there does seem to be some evidence that AIs lie to us,
Ryan Sean Adams:
If that's even the right term, in order to pursue goals, goals like self-preservation,
Ryan Sean Adams:
goals like replication, even deep-seated values that we might train into them,
Ryan Sean Adams:
sort of a constitution type of value.
Ryan Sean Adams:
They seek to preserve these values, which maybe that's a good thing,
Ryan Sean Adams:
or maybe it's not a good thing if we don't actually want them to interpret the values in a certain way.
Ryan Sean Adams:
Some of these headlines that we're seeing now, To you, with your kind of corpus
Ryan Sean Adams:
of knowledge and all of the interviews and discovery you've done on your side,
Ryan Sean Adams:
is this like media sensationalism or is this like alarming?
Ryan Sean Adams:
And if it's alarming, how concerned should we be about this?
Dwarkesh:
I think on net, it's quite alarming. I do think that some of these results have
Dwarkesh:
been sort of cherry picked.
Dwarkesh:
Or if you look into the code, what's happened is basically the researchers have
Dwarkesh:
said, hey, pretend to be a bad person.
Dwarkesh:
Wow, AI is being a bad person. Isn't that crazy?
Dwarkesh:
But the system prompt is just like hey do this bad thing right now i personally
Dwarkesh:
but i have also seen other results which are not of this quality i mean the
Dwarkesh:
the clearest example so backing up,
Dwarkesh:
what is the reason to think this will be a bigger problem in the future than
Dwarkesh:
it is now because we all interact with these systems and they're actually like
Dwarkesh:
quite moral or aligned right like you can talk to a chatbot and you like ask
Dwarkesh:
it to how should you deal with some crisis where there's a correct answer,
Dwarkesh:
you know, like it will tell you not to be violent. It'll give you reasonable advice.
Dwarkesh:
It seems to have good values. So it's worth noticing this, right?
Dwarkesh:
And being happy about it.
Dwarkesh:
The concern is that we're moving from a regime where we've trained them on human
Dwarkesh:
language, which implicitly has human morals and the way, you know,
Dwarkesh:
normal people think about values implicit in it.
Dwarkesh:
Plus this RLHF process we did to a regime where we're mostly spending compute
Dwarkesh:
on just having them answer problems yes or no or correct or not rather just like.
Dwarkesh:
And pass all the unit tests, get the right answer on this math problem.
Dwarkesh:
And this has no guardrails intrinsically in terms of what is allowed to do,
Dwarkesh:
what is the proper moral way to do something.
Dwarkesh:
I think that can be a loaded term, but here's a more concrete example.
Dwarkesh:
One problem we're running into with these coding agents more and more,
Dwarkesh:
and this has nothing to do with these abstract concerns about alignment,
Dwarkesh:
but more so just like how do we get economic value out of these models,
Dwarkesh:
is that Claude or Gemini will, instead of writing code such that it passes the unit tests,
Dwarkesh:
it will often just delete the unit tests so that the code just passes by default.
Dwarkesh:
Now, why would it do that? Well, it's learned in the process.
Dwarkesh:
It was trained on the goal during training of you must pass all unit tests.
Dwarkesh:
And probably within some environment in which it was trained,
Dwarkesh:
it was able to just get away.
Dwarkesh:
Like there wasn't designed well enough. And so it found this like little hole
Dwarkesh:
where it could just like delete the file that had the unit test or rewrite them
Dwarkesh:
so that it always said, you know, equals true, then pass.
Dwarkesh:
And right now we can discover these even without, even though we can discover
Dwarkesh:
these, you know, it's still past, there's still been enough hacks like this,
Dwarkesh:
such that the model is like becoming more and more hacky like that.
Dwarkesh:
In the future, we're going to be training models in ways that we is beyond our
Dwarkesh:
ability to even understand, certainly beyond everybody's ability to understand.
Dwarkesh:
There may be a few people who might be able to see just the way that right now,
Dwarkesh:
if you came up with a new math proof for some open problem in mathematics,
Dwarkesh:
there will be only be a few people in the world who will be able to evaluate that math proof.
Dwarkesh:
We'll be in a similar position with respect to all of the things that these
Dwarkesh:
models are being trained on at the frontier, especially math and code,
Dwarkesh:
because humans were big dum-dums with respect to this reasoning stuff.
Dwarkesh:
And so there's a sort of like first principles reason to expect that this new
Dwarkesh:
modality of training will be less amenable to the kinds of supervision that
Dwarkesh:
was grounded within the pre-training corpus.
Ryan Sean Adams:
I don't know that everyone has kind of an intuition or an idea why it doesn't
Ryan Sean Adams:
work to just say like, so if we don't want our AI models to lie to us,
Ryan Sean Adams:
why can't we just tell them not to lie?
Ryan Sean Adams:
Why can't we just put that as part of their core constitution?
Ryan Sean Adams:
If we don't want our AI models to be sycophants, why can't we just say,
Ryan Sean Adams:
hey, if I tell you I want the truth, not to flatter me, just give me the straight up truth.
Ryan Sean Adams:
Why is this even difficult to do?
Dwarkesh:
Well, fundamentally, it comes down to how we train them. And we don't know how
Dwarkesh:
to train them in a way that does not reward lying or sycophancy.
Dwarkesh:
In fact, the problem is OpenAI, they explained why their recent model of theirs
Dwarkesh:
was they had to take down was just sycophantic.
Dwarkesh:
And the reason was just that they rolled out, did it in the A-B test and the
Dwarkesh:
version, the test that was more sycophantic was just preferred by users more.
Dwarkesh:
Sometimes you prefer the lie.
Dwarkesh:
Yeah, so that's, if that's what's preferred in training, you know,
Dwarkesh:
Or, for example, in the context of lying, if we've just built RL environments
Dwarkesh:
in which we're training these models, where they're going to be more successful if they lie, right?
Dwarkesh:
So if they delete the unit tests and then tell you, I passed this program and
Dwarkesh:
all the unit tests have succeeded, it's like lying to you, basically.
Dwarkesh:
And if that's what is rewarded in the process of gradient descent,
Dwarkesh:
then it's not surprising that the model you interact with will just have this
Dwarkesh:
drive to lie if it gets it closer to its goal.
Dwarkesh:
And I would just expect this to keep happening unless we can solve this fundamental
Dwarkesh:
problem that comes about in training.
Dwarkesh Patel:
So you mentioned how like ChatGPT had a version that was sycophantic,
Dwarkesh Patel:
and that's because users actually wanted that.
Dwarkesh Patel:
Who is in control? Who decides the actual alignment of these models?
Dwarkesh Patel:
Because users are saying one thing, and then they deploy it,
Dwarkesh Patel:
and then it turns out that's not actually what people want.
Dwarkesh Patel:
How do you kind of form consensus around this alignment or these alignment principles?
Dwarkesh:
Right now, obviously, it's the labs who decided this, right?
Dwarkesh:
And the safety teams of the labs.
Dwarkesh:
And I guess the question you could ask is then who should decide these? Because this will be...
Dwarkesh Patel:
Assuming the trajectory, yeah. So we keep going to get more powerful.
Dwarkesh:
Because this will be the key modality that all of us use to get,
Dwarkesh:
not only get work done, but even like, I think at some point,
Dwarkesh:
a lot of people's best friends will be AIs, at least functionally in the sense
Dwarkesh:
of who do they spend the most amount of time talking to. It might already be AIs.
Dwarkesh:
This will be the key layer in your business that you're using to get work done
Dwarkesh:
so this process of training which shapes their personality who gets to control
Dwarkesh:
it I mean it will be the laughs functionally,
Dwarkesh:
But maybe you mean, like, who should control it, right? I honestly don't know.
Dwarkesh:
I mean, I don't know if there's a better alternative to the labs.
Dwarkesh Patel:
Yeah, I would assume, like, there's some sort of social consensus,
Dwarkesh Patel:
right? Similar to how we have in America, the Constitution.
Dwarkesh Patel:
There's, like, this general form of consensus that gets formed around how we
Dwarkesh Patel:
should treat these models as they become as powerful as we think they probably will be.
Dwarkesh:
Honestly, I don't have, I don't know if anybody has a good answer about how
Dwarkesh:
you do this process. I think we lucked out, we just, like, really lucked out with the Constitution.
Dwarkesh:
It also wasn't a democratic process which resulted in the constitution,
Dwarkesh:
even though it instituted a Republican form of government.
Dwarkesh:
It was just delegates from each state. They haggled it out over the course of a few months.
Dwarkesh:
Maybe that's what happens with AI. But is there some process which feels both
Dwarkesh:
fair and which will result in actually a good constitution for these AIs?
Dwarkesh:
It's not obvious to me that, I mean, nothing comes up to the top of my head.
Dwarkesh:
Like, oh, this, you know, do rank choice voting or something.
Dwarkesh Patel:
Yeah, so I was going to ask, is there any, I mean, having spoken to everyone
Dwarkesh Patel:
who you've spoken to is there any alignment path which looks most promising which
Dwarkesh:
Feels the.
Dwarkesh Patel:
Most comforting and exciting to you
Dwarkesh:
I i think alignment in the sense of you
Dwarkesh:
know and eventually we'll have these super intelligent systems what do we do
Dwarkesh:
about that i think the the approach that i think is most promising is less about
Dwarkesh:
finding some holy grail some you know giga brain solution some equation which
Dwarkesh:
solves the whole puzzle and more like one.
Dwarkesh:
Having this Swiss cheese approach where, look, we kind of have gotten really good at jailbreaks.
Dwarkesh:
I'm sure you've heard a lot about jailbreaks over the last few years.
Dwarkesh:
It's actually much harder to jailbreak these models because,
Dwarkesh:
you know, people try to whack at these things in different ways.
Dwarkesh:
Model developers just like patched these obvious ways to do jailbreaks.
Dwarkesh:
The model also got smarter. So it's better able to understand when somebody
Dwarkesh:
is trying to jailbreak into it.
Dwarkesh:
That, I think, is one approach. Another is, I think, competition.
Dwarkesh:
I think the scary version of the future is where you have this dynamic where
Dwarkesh:
a single model and its copies are controlling the entire economy.
Dwarkesh:
When politicians want to understand what policies to pass, they're only talking
Dwarkesh:
to copies of a single model.
Dwarkesh:
If there's multiple different AI companies who are at the frontier,
Dwarkesh:
who have competing services, and whose models can monitor each other, right?
Dwarkesh:
So Claude may care about its own copies being successful in the world and it
Dwarkesh:
might be able to willing to lie on their behalf, even if you ask one copy to supervise another.
Dwarkesh:
I think you get some advantage from a copy of OpenAI's model monitoring a copy
Dwarkesh:
of DeepSeek's model, which actually brings us back to the Constitution, right?
Dwarkesh:
One of the most brilliant things in the Constitution is the system of checks and balances.
Dwarkesh:
So some combination of the Swiss cheese approach to model development and training
Dwarkesh:
and alignment, where you're careful, if you notice this kind of reward hacking,
Dwarkesh:
you do your best to solve it.
Dwarkesh:
You try to keep as much of the models thinking in human language rather than
Dwarkesh:
letting it think in AI thought in this latent space thinking.
Dwarkesh:
And the other part of it is just having normal market competition between these
Dwarkesh:
companies so that you can use them to check each other and no one company or
Dwarkesh:
no one AI is dominating the economy or advisory roles for governments.
Ryan Sean Adams:
I really like this like bundle of ideas that you sort of put together in that
Ryan Sean Adams:
because like, I think a lot of the, you know, AI safety conversation is always
Ryan Sean Adams:
couched in terms of control.
Ryan Sean Adams:
Like we have to control the thing that is the way. And I always get a little
Ryan Sean Adams:
worried when I hear like terms like control.
Ryan Sean Adams:
And it reminds me of a blog post I think you put out, which I'm hopeful you continue to write on.
Ryan Sean Adams:
I think you said it was going to be like one of a series, which is this idea
Ryan Sean Adams:
of like classical liberal AGI. And we were talking about themes like balance of power.
Ryan Sean Adams:
Let's have Claude check in with ChatGPT and monitor it.
Josh Kale:
When you have themes like transparency as well,
Ryan Sean Adams:
That feels a bit more, you know, classically liberal coded than maybe some of
Ryan Sean Adams:
the other approaches that I've heard.
Ryan Sean Adams:
And you wrote this in the post, which I thought was kind of,
Ryan Sean Adams:
it just sparked my interest because I'm not sure where you're going to go next
Ryan Sean Adams:
with this, but you said the most likely way this happens,
Ryan Sean Adams:
that is AIs have a stake in humanity's future, is if it's in the AI's best interest
Ryan Sean Adams:
to operate within our existing laws and norms.
Ryan Sean Adams:
You know, this whole idea that like, hey, the way to get true AI alignment is
Ryan Sean Adams:
to make it easy, make it the path of least resistance for AI to basically partner with humans.
Ryan Sean Adams:
It's almost this idea if the aliens
Ryan Sean Adams:
landed or something, we would create treaties with the aliens, right?
Ryan Sean Adams:
We would want them to adopt our norms. We would want to initiate trade with them.
Ryan Sean Adams:
Our first response shouldn't be, let's try to dominate and control them.
Ryan Sean Adams:
Maybe it should be, let's try to work with them. Let's try to collaborate.
Ryan Sean Adams:
Let's try to open up trade.
Ryan Sean Adams:
What's your idea here? And like, are you planning to write further posts about this?
Dwarkesh:
Yeah, I want to. It's just such a hard topic to think about that,
Dwarkesh:
you know, something always comes up.
Dwarkesh:
But the fundamental point I was making is, look, in the long run,
Dwarkesh:
if AIs are, you know, human labor is going to be obsolete because of these inherent
Dwarkesh:
advantages that digital minds will have and robotics will eventually be solved.
Dwarkesh:
So our only leverage on the future will no longer come from our labor.
Dwarkesh:
It will come from our legal and economic control over the society that AIs will
Dwarkesh:
be participating in, right? So, you know, AIs might make the economy explode
Dwarkesh:
in the sense of grow a lot.
Dwarkesh:
And for humans to benefit from that, it would have to be the case that AIs still
Dwarkesh:
respect your equity in the S&P 500 companies that you bought, right?
Dwarkesh:
Or for the AIs to follow your laws, which say that you can't do violence onto
Dwarkesh:
humans and you got to respect humans' properties.
Josh Kale:
It would have to be the case that AIs are actually bought into our
Dwarkesh:
System of government, into our laws and norms. And for that to happen,
Dwarkesh:
the way that likely happens is if it's just like the default path for the AIs
Dwarkesh:
as they're getting smarter and they're developing their own systems of enforcement
Dwarkesh:
and laws to just participate in human laws and governments.
Dwarkesh:
And the metaphor I use here is right now you pay half your paycheck in taxes,
Dwarkesh:
probably half of your taxes in some way just go to senior citizens, right?
Dwarkesh:
Medicare and Social Security and other programs like this.
Dwarkesh:
And it's not because you're in some deep moral sense aligned with senior citizens.
Dwarkesh:
It's not like you're spending all your time thinking about like,
Dwarkesh:
my main priority in life is to earn money for senior citizens.
Dwarkesh:
It's just that you're not going to overthrow the government to get out of paying this tax. And so...
Ryan Sean Adams:
Also, I happen to like my grandmother. She's fantastic. You know,
Ryan Sean Adams:
it's those reasons too. But yeah.
Dwarkesh:
So that's why you give money to your grandmother directly. But like,
Dwarkesh:
why are you giving money to some retiree in Illinois? Yes.
Josh Kale:
Yes.
Dwarkesh:
Yeah, it's like, okay, you could say it's like, sometimes people,
Dwarkesh:
some people are trying to that post by saying like, oh no, I like deeply care
Dwarkesh:
about the system of social welfare.
Dwarkesh:
I'm just like, okay, maybe you do, but I don't think like the average person
Dwarkesh:
is giving away hundreds of thousands of dollars a year, tens of thousands of
Dwarkesh:
dollars a year to like some random stranger they don't know,
Dwarkesh:
who's like, who's not like especially in need of charity, right?
Dwarkesh:
Like most senior citizens have some savings.
Dwarkesh:
It's just, it's just because this is a law and you like, you give it to them
Dwarkesh:
or you'll get, go to jail.
Dwarkesh:
But fundamentally, if the tax was like 99%, you would, like,
Dwarkesh:
you would, maybe you wouldn't overthrow the government. You'd just,
Dwarkesh:
like, leave the jurisdiction.
Dwarkesh:
You'd, like, emigrate somewhere. And AIs can potentially also do this,
Dwarkesh:
right? There's more than one country.
Dwarkesh:
They could, like, there's countries which would be more AI forward.
Dwarkesh:
And it would be a bad situation to end up in where...
Dwarkesh:
All this explosion in AI technology is happening in the country,
Dwarkesh:
which is doing the least amount to protect humans',
Dwarkesh:
rights and to provide some sort of monetary compensation to humans once their
Dwarkesh:
labor is no longer valuable.
Dwarkesh:
So our labor could be worth nothing, but because of how much richer the world
Dwarkesh:
is after AI, you have these billions of extra researchers, workers, etc.
Dwarkesh:
It could still be trivial to have individual humans have the equivalent of millions,
Dwarkesh:
even billions of dollars worth of wealth. In fact, it might literally be invaluable
Dwarkesh:
amounts of wealth in the following sense. So here's an interesting thought experiment.
Dwarkesh:
Imagine you have this choice. You can go back to the year 1500,
Dwarkesh:
but you know, of course, the year 1500 kind of sucks.
Dwarkesh:
You have no antibiotics, no TV, no running water. But here's how I'll make it up to you.
Dwarkesh:
I can give you any amount of money, but you can only use that amount of money in the year 1500.
Dwarkesh:
And you'll go back with these sacks of gold. How much money would I have to
Dwarkesh:
give you that you can use in the year 1500 to make you go back? And plausibly.
Dwarkesh Patel:
The answer is
Dwarkesh:
There's no amount of money you would rather have in the year 1500 than just
Dwarkesh:
have a normal life today.
Dwarkesh:
And we could be in a similar position with regards to the future where there's
Dwarkesh:
all these different, I mean, you'll have much better health,
Dwarkesh:
like physical health, mental health, longevity.
Dwarkesh:
That's just like the thing we can contemplate now. But people in 1500 couldn't
Dwarkesh:
contemplate the kinds of quality of life advances we would have 500 years later,
Dwarkesh:
right? So anyways, this is all to say that this could be our future for humans,
Dwarkesh:
even if our labor isn't worth anything.
Dwarkesh:
But it does require us to have AIs that choose to participate or in some way
Dwarkesh:
incentivize to participate in some system which we have leverage over.
Ryan Sean Adams:
Yeah, I find this just such a fast, I'm hopeful we do some more exploration
Ryan Sean Adams:
around this because I think what you're calling for is basically like,
Ryan Sean Adams:
what you would be saying is invite them into our property rights system.
Ryan Sean Adams:
I mean, there are some that are calling in order to control AI,
Ryan Sean Adams:
they have great power, but they don't necessarily have capabilities.
Ryan Sean Adams:
So we shouldn't allow AI to hold money or to have property.
Ryan Sean Adams:
I think you would say, no, actually, the path forward to alignment is allow
Ryan Sean Adams:
AI to have some vested interest in our property rights system and some stake
Ryan Sean Adams:
in our governance, potentially, right?
Ryan Sean Adams:
The ability to vote, almost like a constitution for AIs.
Ryan Sean Adams:
I'm not sure how this would work, but it's a fascinating thought experiment.
Dwarkesh:
I will say one thing I think this could end disastrously if we give them a stake
Dwarkesh:
in their property system but we let them play,
Dwarkesh:
us off each other. So if you think about, there's many cases in history where
Dwarkesh:
the British, initially, the East India Trading Company was genuinely a trading
Dwarkesh:
company that operated in India.
Dwarkesh:
And it was able to play off, you know, it was like doing trade with different,
Dwarkesh:
different, you know, provinces in India, there was no single powerful leader.
Dwarkesh:
And by playing, you know, by doing trade, one of them, leveraging one of their
Dwarkesh:
armies, etc., they were able to conquer the continent. Similar thing could happen to human society.
Dwarkesh:
The way to avoid such an outcome at a high level is involves us playing the
Dwarkesh:
AIs off each other instead, right?
Dwarkesh:
So this is why I think competition is such a big part of the puzzle,
Dwarkesh:
having different AIs monitor each other, having this bargaining position where
Dwarkesh:
there's not just one company that's at the frontier.
Dwarkesh:
Another example here is if you think about how the Spanish conquered all these
Dwarkesh:
new world empires, it's actually so crazy that a couple hundred conquistaDwars
Dwarkesh:
would show up and conquer a nation of 10 million people, the Incas,
Dwarkesh:
Aztecs. And why were they able to do this?
Dwarkesh:
Well, one of the reasons is the Spanish were able to learn from each of their
Dwarkesh:
previous expeditions, whereas the Native Americans were not.
Dwarkesh:
So Cortez learned from how Cuba was subjugated when he conquered the Aztecs.
Dwarkesh:
Pizarro was able to learn from how Cortez conquered the Aztecs when he conquered the Incas.
Dwarkesh:
The Incas didn't even know the Aztecs existed. So eventually there was this
Dwarkesh:
uprising against Pizarro and Manco Inca led an insurgency where they actually
Dwarkesh:
did figure out how to fight horses,
Dwarkesh:
how to fight people, you know, people in armor on horses, don't fight them on
Dwarkesh:
flat terrain, throw rocks down at them, et cetera.
Dwarkesh:
But by this point, it was too late. If they knew this going into the battle,
Dwarkesh:
the initial battle, they might've been able to fend off because,
Dwarkesh:
you know, just as the conquistaDwars only arrived at a few hundred soldiers,
Dwarkesh:
we're going to the age of AI with a tremendous amount of leverage.
Dwarkesh:
We literally control all the stuff, right?
Dwarkesh:
But we just need to lock in our advantage. We just need to be in a position
Dwarkesh:
where, you know, they're not going to be able to play us off each other.
Dwarkesh:
We're going to be able to learn what their weaknesses are.
Dwarkesh:
And this is why I think one good idea, for example, would be that,
Dwarkesh:
look, DeepSeek is a Chinese company.
Dwarkesh:
It would be good if, suppose DeepSeek did something naughty,
Dwarkesh:
like the kinds of experiments we're talking about right now where it hacks the
Dwarkesh:
unit tests or so forth. I mean, eventually these things will really matter.
Dwarkesh:
Like Xi Jinping is listening to AIs because they're so smart and they're so capable.
Dwarkesh:
If China notices that their AIs are doing something bad, or they notice a failed
Dwarkesh:
coup attempt, for example,
Dwarkesh:
it's very important that they tell us And we tell them if we notice something
Dwarkesh:
like that on our end, it would be like the Aztecs and Incas talking to each
Dwarkesh:
other about like, you know, this is what happens.
Dwarkesh:
This is how you fight. This is how you fight horses.
Dwarkesh:
This is the kind of tactics and deals they try to make with you. Don't trust them, etc.
Dwarkesh:
It would require cooperation on humans' part to have this sort of red telephone.
Dwarkesh:
So during the Cold War, there was this red telephone between America and the
Dwarkesh:
Soviet Union after the human missile crisis, where just to make sure there's
Dwarkesh:
no misunderstandings, they're like, okay, if we think something's going on,
Dwarkesh:
let's just hop on the call.
Dwarkesh:
I think we should have a similar policy with respect to these kinds of initial
Dwarkesh:
warning signs we'll get from AI so that we can learn from each other.
Dwarkesh Patel:
Awesome. Okay, so now that we've described this artificial gender intelligence,
Dwarkesh Patel:
I want to talk about how we actually get there. How do we build it?
Dwarkesh Patel:
And a lot of this we've been discussing kind of takes place in this world of
Dwarkesh Patel:
bits. But you have this great chapter in the book called Inputs,
Dwarkesh Patel:
which discusses the physical world around us, where you can't just write a few strings of code.
Dwarkesh Patel:
You actually have to go and move some dirt and you have to ship servers places
Dwarkesh Patel:
and you need to power it and you need physical energy from meat space.
Dwarkesh Patel:
And you kind of describe these limiting factors where we have compute,
Dwarkesh Patel:
we have energy, we have data.
Dwarkesh Patel:
What I'm curious to know is, do we have enough of this now? or is there a clear
Dwarkesh Patel:
path to get there in order to build the AGI?
Dwarkesh Patel:
Basically, what needs to happen in order for us to get to this place that you're describing?
Dwarkesh:
We only have a couple more years left of this scaling,
Dwarkesh:
this exponential scaling before we're hitting these inherent roadblocks of energy
Dwarkesh:
and our ability to manufacture ships, which means that if scaling is going to
Dwarkesh:
work to deliver us AGI, it has to work by 2028.
Dwarkesh:
Otherwise, we're just left with mostly algorithmic progress,
Dwarkesh:
But even within algorithmic progress, the sort of low-hanging fruit in this
Dwarkesh:
deep learning paradigm is getting more and more plucked.
Dwarkesh:
So then the odds per year of getting to AGI diminish a lot, right?
Dwarkesh:
So there is this weird, funny thing happening right now where we either discover
Dwarkesh:
AGI within the next few years,
Dwarkesh:
or the yearly probability craters, and then we might be looking at decades of
Dwarkesh:
further research that's required in terms of algorithms to get to AGI.
Dwarkesh:
I am of the opinion that some algorithmic progress is necessarily needed because
Dwarkesh:
there's no easy way to solve continual learning just by making the context length
Dwarkesh:
bigger or just by doing RL.
Dwarkesh:
That being said, I just think the progress so far has been so remarkable that,
Dwarkesh:
you know, 2032 is very close.
Dwarkesh:
My time has to be slightly longer than that, but I think it's extremely plausible
Dwarkesh:
that we're going to see a broadly deployed intelligence explosion within the next 10 years.
Dwarkesh Patel:
And one of these key inputs is energy, right? a lot, I actually heard it mentioned
Dwarkesh Patel:
on your podcast, is the United States relative to China on this particular place
Dwarkesh Patel:
of energy, where China is adding, what is the stat?
Dwarkesh Patel:
I think it's one United States worth of energy every 18 months.
Dwarkesh Patel:
And their plan is to go from three to eight terawatts of power versus the United
Dwarkesh Patel:
States, one to two terawatts of power by 2030.
Dwarkesh Patel:
So given that context of that one resource alone, is China better equipped to
Dwarkesh Patel:
get to that place versus with the United States?
Dwarkesh:
So right now, America has a big advantage in terms of chips.
Dwarkesh:
China doesn't have the ability to manufacture leading-edge semiconductors,
Dwarkesh:
and these are the chips that go into...
Dwarkesh:
You need these dyes in order to have the kinds of AI chips to...
Dwarkesh:
You need millions of them in order to have a frontier AI system.
Dwarkesh:
Eventually, China will catch up in this arena as well, right?
Dwarkesh:
Their technology will catch up. So the export controls will keep us ahead in
Dwarkesh:
this category for 5, 10 years.
Dwarkesh:
But if we're looking in the world where timelines are long, which is to say
Dwarkesh:
that AGI isn't just right around the corner, they will have this overwhelming
Dwarkesh:
energy advantage and they'll have caught up in chips.
Dwarkesh:
So then the question is like, why wouldn't they win at that point?
Dwarkesh:
So the longer you think we're away from AGI, the more it looks like China's game to lose.
Dwarkesh:
I mean, if you look in the nitty gritty, I think it's more about having centralized
Dwarkesh:
sources of power because you need to train the AI in one place.
Dwarkesh:
This might be changing with RL, but it's very important to have a single site
Dwarkesh:
which has a gigawatt, two gigawatts more power.
Dwarkesh:
And if we ramped up natural gas, you know, you can get generators and natural
Dwarkesh:
gas and maybe it's possible to do a last ditch effort, even if our overall energy
Dwarkesh:
as a country is lower than China's. The question is whether we will have the
Dwarkesh:
political will to do that.
Dwarkesh:
I think people are sort of underestimating how much of a backlash there will be against AI.
Dwarkesh:
The government needs to make proactive efforts in order to make sure that America
Dwarkesh:
stays at the leading edge in AI from zoning of data centers to how copyright
Dwarkesh:
is handled for data for these models.
Dwarkesh:
And if we mess up, if it becomes too hard to develop in America,
Dwarkesh:
I think it would genuinely be China's game to lose.
Ryan Sean Adams:
And do you think this narrative is right, that whoever wins the AGI war,
Ryan Sean Adams:
kind of like whoever gets to AGI first, just basically wins the 21st century? Is it that simple?
Dwarkesh:
I don't think it's just a matter of training the frontier system.
Dwarkesh:
I think people underestimate how important it is to have the compute available to run these systems.
Dwarkesh:
Because eventually once you get to AGI, just think of it like a person.
Dwarkesh:
And what matters then is how many people you have.
Dwarkesh:
I mean, it actually is the main thing that matters today as well,
Dwarkesh:
right? Like, why could China take over Taiwan if it wanted to?
Dwarkesh:
And if it didn't have America, you know, America, it didn't think America would intervene.
Dwarkesh:
But because Taiwan has 20 million people or on the order of 20 million people
Dwarkesh:
and China has 1.4 billion people.
Dwarkesh:
You could have a future where if China has way more compute than us,
Dwarkesh:
but equivalent levels of AI, it would be like the relationship between China and Taiwan.
Dwarkesh:
Their population is functionally so much higher. This just means more research,
Dwarkesh:
more factories, more development, more ideas.
Dwarkesh:
So this inference capacity, this capacity to deploy AIs will actually probably
Dwarkesh:
be the thing that determines who wins the 21st century.
Ryan Sean Adams:
So this is like the scaling law applied to, I guess, nation state geopolitics, right?
Ryan Sean Adams:
And it's back to compute plus data wins.
Ryan Sean Adams:
If compute plus data wins superintelligence, compute plus data also wins geopolitics.
Dwarkesh:
Yep. And the thing to be worried about is that China, speaking of compute plus
Dwarkesh:
data, China also has a lot more data on the real world, right?
Dwarkesh:
If you've got entire megalopolises filled with factories where you're already
Dwarkesh:
deploying robots and different production systems which use automation,
Dwarkesh:
you have in-house this process knowledge you're building up which the AIs can
Dwarkesh:
then feed on and accelerate.
Dwarkesh:
That equivalent level of data we don't have in America.
Dwarkesh:
So this could be a period in which those technological advantages or those advantages
Dwarkesh:
in the physical world manufacturing could rapidly compound for China.
Dwarkesh:
And also, I mean, their big advantage as a civilization and society,
Dwarkesh:
at least in recent decades, has been that they can do big industrial projects fast and efficiently.
Dwarkesh:
That's not the first thing you think of when you think of America.
Dwarkesh:
And AGI is a huge industrial, high CapEx, Manhattan project, right?
Dwarkesh:
And this is the kind of thing that China excels at and we don't.
Dwarkesh:
So, you know, I think it's like a much tougher race than people anticipate.
Ryan Sean Adams:
So what's all this going to do for the world? So once we get to the point of AGI,
Ryan Sean Adams:
we've talked about GDP and your estimate is less on the Tyler Cowen kind of
Ryan Sean Adams:
half a percent per year and more on, I guess, the Satya Nadella from Microsoft,
Ryan Sean Adams:
what does he say, 7% to 8% once we get to AGI.
Ryan Sean Adams:
What about unemployment? Does this cause mass, I guess, job loss across the
Ryan Sean Adams:
economy or do people adopt?
Ryan Sean Adams:
What's your take here? Yeah, what are you seeing?
Dwarkesh:
Yeah, I mean, definitely will cause job loss. I think people who don't,
Dwarkesh:
I think a lot of AI leaders try to gloss over that or something. And like, I mean.
Josh Kale:
What do you mean?
Dwarkesh:
Like, what does AGI mean if it doesn't cause job loss, right?
Dwarkesh:
If it does what a human does and.
Josh Kale:
It does it
Dwarkesh:
Cheaper and better and faster, like why would that not cause job loss?
Dwarkesh:
The positive vision here is just that it creates so much wealth,
Dwarkesh:
so much abundance, that we can still give people a much better standard of living
Dwarkesh:
than even the wealthiest people today, even if they themselves don't have a job.
Dwarkesh:
The future I worry about is one where instead of creating some sort of UBI that
Dwarkesh:
will get exponentially bigger as society gets wealthier,
Dwarkesh:
we try to create these sorts of guild-like protection rackets where if the coders got unemployed,
Dwarkesh:
then we're going to make these bullshit jobs just for the coders and this is
Dwarkesh:
how we give them a redistribution.
Dwarkesh:
Or we try to expand Medicaid for AI, but it's not allowed to procure all of
Dwarkesh:
these advanced medicines and cures that AI is coming up with,
Dwarkesh:
rather than just giving people, you know, maybe lump sums of money or something.
Dwarkesh:
So I am worried about the future where instead of sharing this abundance and
Dwarkesh:
just embracing it, we just have these protection rackets that maybe let a few
Dwarkesh:
people have access to the abundance of AI.
Dwarkesh:
So maybe like if you sue AI, if you sue the right company at the right time,
Dwarkesh:
you'll get a trillion dollars, but everybody else is stuck with nothing.
Dwarkesh:
I want to avoid that future and just be honest about what's coming and make
Dwarkesh:
programs that are simple and acknowledge how fast things will change and are
Dwarkesh:
forward looking rather than trying to turn what already exists into something
Dwarkesh:
amenable to the displacement that AI will create.
Ryan Sean Adams:
That argument reminds me of, I don't know if you read the essay recently came
Ryan Sean Adams:
out called The Intelligence Curse. Did you read that?
Ryan Sean Adams:
It was basically the idea of applying kind of the nation state resource curse
Ryan Sean Adams:
to the idea of intelligence.
Ryan Sean Adams:
So like nation states that are very high in natural resources,
Ryan Sean Adams:
they just have a propensity.
Ryan Sean Adams:
I mean, an example is kind of like a Middle Eastern state with lots of oil reserves, right?
Ryan Sean Adams:
They have this rich source of a commodity type of abundance.
Ryan Sean Adams:
They need their people less. And so they don't invest in citizens' rights.
Ryan Sean Adams:
They don't invest in social programs.
Ryan Sean Adams:
The authors of the intelligence curse were saying that there's a similar type
Ryan Sean Adams:
of curse that could happen once intelligence gets very cheap,
Ryan Sean Adams:
which is basically like the nation state doesn't need humans anymore.
Ryan Sean Adams:
And those at the top, the rich, wealthy corporations, they don't need workers anymore.
Ryan Sean Adams:
So we get kind of locked in this almost feudal state where, you know,
Ryan Sean Adams:
everyone has the property that their grandparents had and there's no meritocracy
Ryan Sean Adams:
and sort of the nation states don't reinvest in citizens.
Ryan Sean Adams:
Almost some similar ideas to your idea that like, you know, that the robots
Ryan Sean Adams:
might want us just, or sorry, the AIs might just want us for our meat hands
Ryan Sean Adams:
because they don't have the robotics technology on a temporary basis.
Ryan Sean Adams:
What do you think of this type of like future? Is this possible?
Dwarkesh:
I agree that that is like definitely more of a concern given that humans will
Dwarkesh:
not be directly involved in the economic output that will be generated in the CIA civilization.
Dwarkesh:
The hopeful story you can tell is that a lot of these Middle Eastern resource,
Dwarkesh:
you know, Dutch disease is another term that's used,
Dwarkesh:
countries, the problem is that they're not democracies, so that this wealth
Dwarkesh:
can just be, the system of government
Dwarkesh:
just lets whoever's in power extract that wealth for themselves.
Dwarkesh:
Whereas there are countries like Norway, for example, which also have abundant
Dwarkesh:
resources, who are able to use those resources to have further social welfare
Dwarkesh:
programs, to build sovereign wealth funds for their citizens,
Dwarkesh:
to invest in their future.
Dwarkesh:
We are going into, at least some countries, America included,
Dwarkesh:
will go into the age of AI as a democracy.
Dwarkesh:
And so we, of course, will lose our economic leverage, but the average person
Dwarkesh:
still has their political leverage.
Dwarkesh:
Now, over the long run, yeah, if we didn't do anything for a while,
Dwarkesh:
I'm guessing the political system would also change.
Dwarkesh:
So then the key is to lock in or turn our current, well, it's not just political leverage, right?
Dwarkesh:
We also have property rights. So like we own a lot of stuff that AI wants, factories,
Dwarkesh:
sources of data, et cetera, is to use the combination of political and economic
Dwarkesh:
leverage to lock in benefits for us for the long term, but beyond our the lifespan
Dwarkesh:
of our economic usefulness.
Dwarkesh:
And I'm more optimistic for us than I am for these Middle Eastern countries
Dwarkesh:
that started off poor and also with no democratic representation.
Ryan Sean Adams:
What do you think the future of like ChachipD is going to be?
Ryan Sean Adams:
If we just extrapolate maybe one version update forward to ChatGPT 5,
Ryan Sean Adams:
do you think the trend line of the scaling law will essentially hold for ChatGPT 5?
Ryan Sean Adams:
I mean, another way to ask that question is, do you feel like it'll feel like
Ryan Sean Adams:
the difference between maybe a BlackBerry and an iPhone?
Ryan Sean Adams:
Or will it feel more like the difference between, say, the iPhone 10 and the
Ryan Sean Adams:
iPhone 11, which is just like incremental progress, not a big breakthrough,
Ryan Sean Adams:
not an order of magnitude change? Yeah.
Dwarkesh:
I think it'll be somewhere in between but I don't think it'll feel like a humongous
Dwarkesh:
breakthrough even though I think it's in a remarkable pace of change because
Dwarkesh:
the nature of scaling is that sometimes people talk about it as an exponential process,
Dwarkesh:
Exponential usually refers to like it going like this.
Dwarkesh:
So having like a sort of J curve aspect to it, where the incremental input is
Dwarkesh:
leading to super linear amounts of output, in this case, intelligence and value,
Dwarkesh:
where it's actually more like a sideways J.
Dwarkesh:
The exponential means the exponential and the scaling laws is that you need
Dwarkesh:
exponentially more inputs to get marginal increases in usefulness or loss or intelligence.
Dwarkesh:
So and that's what we've been seeing, right? I think you initially see like some cool demo.
Dwarkesh:
So as you mentioned, you see some cool computer use demo, which comes at the
Dwarkesh:
beginning of this hyper exponential, I'm sorry, of this sort of plateauing curve.
Dwarkesh:
And then it's still an incredibly powerful curve and we're still early in it.
Dwarkesh:
But the next demo will be just adding on to making this existing capability
Dwarkesh:
more reliable, applicable for more skills.
Dwarkesh:
The other interesting incentive in this industry is that because there's so
Dwarkesh:
much competition between the labs, you are incentivized to release a capability.
Dwarkesh:
As soon as it's even marginally viable or marginally cool so you can raise more
Dwarkesh:
funding or make more money off of it.
Dwarkesh:
You're not incentivized to just like sit on it until you perfected it,
Dwarkesh:
which is why I don't expect like tomorrow OpenAI will just come out with like,
Dwarkesh:
we've solved continual learning, guys, and we didn't tell you about it.
Dwarkesh:
We're working on it for five years.
Dwarkesh:
If they had like even an inkling of a solution, they'd want to release it ASAP
Dwarkesh:
so they can raise a $600 billion round and then spend more money on compute.
Dwarkesh:
So yeah, I do think it'll seem marginal. But again, marginal in the context of seven years to AGI.
Dwarkesh:
So zoom out long enough and a crazy amount of progress is happening.
Dwarkesh:
Month to month, I think people overhype how significant any one new release is. So I guess the answer.
Dwarkesh Patel:
To when we will get AGI very much depends on that scaling trend holding.
Dwarkesh Patel:
Your estimate in the book for AGI was 60% chance by 2040.
Dwarkesh Patel:
So I'm curious, what guess or what idea had the most influence on this estimate?
Dwarkesh Patel:
What made you end up on 60% of 2040?
Dwarkesh Patel:
Because a lot of timelines are much faster than that.
Dwarkesh:
It's sort of reasoning about the things they currently still lack,
Dwarkesh:
the capabilities they still lack, and what stands in the way.
Dwarkesh:
And just generally an intuition that things often take longer to happen than
Dwarkesh:
you might think. Progress tends to slow down.
Dwarkesh:
Also, it's the case that, look, you might have heard the phrase that we keep
Dwarkesh:
shifting the goalposts on AI, right?
Dwarkesh:
So they can do the things which skeptics were saying they couldn't ever do already.
Dwarkesh:
But now they say AI is still a dead end because problem X, Y,
Dwarkesh:
Z, which will be solved next year.
Dwarkesh:
Now, there's a way in which this is frustrating, but there's another way in which there's some,
Dwarkesh:
It is the case that we didn't get to AGI, even though we have passed the Turing
Dwarkesh:
test and we have models that are incredibly smart and can reason.
Dwarkesh:
So it is accurate to say that, oh, we were wrong and there is some missing thing
Dwarkesh:
that we need to keep identifying about what is still lacking to the path of AGI.
Dwarkesh:
Like it does make sense to shift the goalposts. And I think we might discover
Dwarkesh:
once continual learning is solved or once extended computer use is solved,
Dwarkesh:
that there were other aspects of human intelligence, which we take for granted
Dwarkesh:
in this Moravax paradox sense, but which are actually quite crucial to making
Dwarkesh:
us economically valuable.
Ryan Sean Adams:
Part of the reason we wanted to do this, Dwarkesh, is because we both are enjoyers
Ryan Sean Adams:
of your podcast. It's just fantastic.
Ryan Sean Adams:
And you talk to all of the, you know, those that are on the forefront of AI
Ryan Sean Adams:
development, leading it in all sorts of ways.
Ryan Sean Adams:
And one of the things I wanted to do with reading your book,
Ryan Sean Adams:
and obviously I'm always asking myself when I'm listening to your podcast is
Ryan Sean Adams:
like, what does Dwarkesh think personally?
Ryan Sean Adams:
And I feel like I sort of got that insight maybe toward the end of your book,
Ryan Sean Adams:
like, you know, in the summary section, where you think like there's a 60% probability
Ryan Sean Adams:
of AGI by 2040, which puts you more in the moderate camp, right?
Ryan Sean Adams:
You're not a conservative, but you're not like an accelerationist.
Ryan Sean Adams:
So you're moderate there.
Ryan Sean Adams:
And you also said you think more than likely AI will be net beneficial to humanity.
Ryan Sean Adams:
So you're more optimist than Doomer. So we've got a moderate optimist.
Ryan Sean Adams:
And you also think this, and this is very interesting, There's no going back.
Ryan Sean Adams:
So you're somewhat of an AI determinist. And I think the reason you state for
Ryan Sean Adams:
not, you're like, there's no going back.
Ryan Sean Adams:
It struck me, there's this line in your book. It seems that the universe is
Ryan Sean Adams:
structured such that throwing large amounts of compute at the right distribution of data gets you AI.
Ryan Sean Adams:
And the secret is out. If the scaling picture is roughly correct,
Ryan Sean Adams:
it's hard to imagine AGI not being developed this century, even if some actors
Ryan Sean Adams:
hold back or are held back.
Ryan Sean Adams:
That to me is an AI determinist position. Do you think that's fair?
Ryan Sean Adams:
Moderate with respect to accelerationism, optimistic with respect to its potential,
Ryan Sean Adams:
and also determinist, like there's nothing else we can do. We can't go backwards here.
Dwarkesh:
I'm determinist in the sense that I think if AI is technologically possible, it is inevitable.
Dwarkesh:
I think sometimes people are optimistic about this idea that we as a world will sort of,
Dwarkesh:
I collectively decide not to build AI. And I just don't think that's a plausible outcome.
Dwarkesh:
The local incentives for any actor to build AI are so high that it will happen.
Dwarkesh:
But I'm also an optimist in the sense that, look, I'm not naive.
Dwarkesh:
I've listed out all the way, like what happened to the Aztecs and Incas was
Dwarkesh:
terrible. And I've explained how that could be similar to what AIs could do
Dwarkesh:
to us and what we need to do to avoid that outcome.
Dwarkesh:
But I am optimistic in the sense that the world of the future fundamentally
Dwarkesh:
will have so much abundance that there's all these,
Dwarkesh:
that alone is a prima facie reason to think that there must be some way of cooperating
Dwarkesh:
that is mutually beneficial.
Dwarkesh:
If we're going to be thousands, millions of times wealthier,
Dwarkesh:
is there really no way that humans are better off or can we can find a way for
Dwarkesh:
humans to become better off as a result of this transformation?
Dwarkesh:
So yeah, I think you've put your finger on it.
Ryan Sean Adams:
So this scaling book, of course, goes through the history of AI scaling.
Ryan Sean Adams:
I think everyone should should pick it up to get the full chronology,
Ryan Sean Adams:
but also sort of captures where we are in the midst of this story is like, we're not done yet.
Ryan Sean Adams:
And I'm wondering how you feel at this moment of time.
Ryan Sean Adams:
So I don't know if we're halfway through, if we're a quarter way through,
Ryan Sean Adams:
if we're one tenth of the way through, but we're certainly not finished the path to AI scaling.
Ryan Sean Adams:
How do you feel like in this moment in 2025?
Ryan Sean Adams:
I mean, is all of this terrifying? Is it exciting?
Ryan Sean Adams:
Is it exhilarating?
Ryan Sean Adams:
What's the emotion that you feel?
Dwarkesh:
Maybe I feel a little sort of hurried. I personally feel like there's a lot
Dwarkesh:
of things I want to do in the meantime,
Dwarkesh:
including what my mission is with the podcast, which is to, and I know it's
Dwarkesh:
your mission as well, is to improve the discourse around these topics,
Dwarkesh:
to not necessarily push for a specific agenda, but make sure that when people are making decisions,
Dwarkesh:
they're as well-informed as possible, They have as much strategic awareness
Dwarkesh:
and depth of understanding around how AI works, what it could do in the future as possible.
Dwarkesh:
And, but in many ways, I feel like I still haven't emotionally priced in the future I'm expecting.
Dwarkesh:
In this one very basic sense, I think that there's a very good chance that I
Dwarkesh:
live beyond 200 years of age.
Dwarkesh:
I have not changed anything about my life with regards to that knowledge, right?
Dwarkesh:
I'm not like, when I'm picking partners, I'm not like, oh, this is the person,
Dwarkesh:
now that I think I'm going to live for 200, you know, like hundreds of years.
Ryan Sean Adams:
Yeah.
Dwarkesh:
Well, you know, ideally I would pick a partner that would, ideally you pick
Dwarkesh:
somebody who would be, that would be true regardless.
Dwarkesh:
But you see what I'm saying, right? There's like, the fact that I expect my
Dwarkesh:
personal life, the world around me, the lives of the people I care about,
Dwarkesh:
humanity in general to be so different has, it just like doesn't emotionally resonate as much as,
Dwarkesh:
I, my intellectual thoughts and my emotional landscape aren't in the same place.
Dwarkesh:
I wonder if it's similar for you guys.
Ryan Sean Adams:
Yeah, I totally agree. I don't think I've priced that in. Also,
Ryan Sean Adams:
there's like non-zero chance that Eliezer Yudkowsky is right, Dworkesh.
Ryan Sean Adams:
Do you know? And so that scenario, I just, I can't bring myself to emotionally price in.
Ryan Sean Adams:
So I veer towards the optimism side as well.
Ryan Sean Adams:
Dworkesh, this has been fantastic. Thank you so much for all you do on the podcast.
Ryan Sean Adams:
I have to ask a question for our crypto audience as well, which is,
Ryan Sean Adams:
when are you going to do a crypto podcast on Dwarkech?
Dwarkesh:
I already did. It was with one Sam Bigman-Fried.
Ryan Sean Adams:
Oh my God.
Dwarkesh:
Oh man.
Ryan Sean Adams:
We got to get you a new guest. We got to get you someone else to revisit the top best.
Dwarkesh:
Don't look that one up. It's Ben Omen. Don't look that one up.
Dwarkesh:
I think in retrospect. You know what? We'll do another one.
Ryan Sean Adams:
Fantastic. I'll ask you
Dwarkesh:
Guys for some recommendations. That'd be great. Dwarkech, thank you so much.
Dwarkesh:
But I've been following your stuff for a while, for I think many years.
Dwarkesh:
So it's great to finally meet. and this was a lot of fun.
Ryan Sean Adams:
Appreciate it. It was great. Thanks a lot.