Ryan Sean Adams: Dwarkesh Patel, we are big fans. It's an honor to have you.
Dwarkesh: Thank you so much for having me on.
Ryan Sean Adams: Okay, so you have a book out. It's called The Scaling Era, an oral history of AI from 2019 to 2025.
Ryan Sean Adams: These are some key dates here. This is really a story of how AI emerged.
Ryan Sean Adams: And it seemed to have exploded on people's radar over the past five years.
Ryan Sean Adams: And And everyone in the world, it feels like, is trying to figure out what just
Ryan Sean Adams: happened and what is about to happen.
Ryan Sean Adams: And I feel like for this story, we should start at the beginning, as your book does.
Ryan Sean Adams: What is the scaling era of AI and when abouts did it start? What were the key milestones?
Dwarkesh: So I think the undertold story about everybody's, of course,
Dwarkesh: been hearing more and more about AI.
Dwarkesh: The under-told story is that the big contributor to these AI models getting
Dwarkesh: better over time has been the fact that we are throwing exponentially more compute
Dwarkesh: into trading frontier systems every year.
Dwarkesh: So by some estimates, we spend 4x every single year over the last decade trading
Dwarkesh: the frontier system than the one before it.
Dwarkesh: And that just means that we're spending hundreds of thousands of times more
Dwarkesh: compute than the systems of the early 2010s.
Dwarkesh: Of course, we've also had algorithmic breakthroughs in the meantime.
Dwarkesh: 2018, we had the Transformer.
Dwarkesh: Since then, obviously, many companies have made small improvements here and there.
Dwarkesh: But the overwhelming fact that we're spending already hundreds of billions of
Dwarkesh: dollars in building up the infrastructure,
Dwarkesh: the data centers, the chips for these models, and this picture is only going
Dwarkesh: to intensify if this exponential keeps going,
Dwarkesh: 4x a year, over the next two years, is something that is on the minds of the
Dwarkesh: CFOs of the big hyperscalers and the people planning the expenditures and training going forward,
Dwarkesh: but is not as common in the conversation around where AI is headed.
Ryan Sean Adams: So what do you feel like people should know about this?
Ryan Sean Adams: Like what is the scaling era? There have been other eras maybe of AI or compute,
Ryan Sean Adams: but what's special about the scaling era?
Dwarkesh: People started noticing. Well, first of all, in 2012, there's this,
Dwarkesh: Ilya Seskaver and others started using neural networks in order to categorize images.
Dwarkesh: And just noticing that instead of doing something hand-coded,
Dwarkesh: you can get a lot of juice out of just neural networks, black boxes.
Dwarkesh: You just train them to identify what thing is like what.
Dwarkesh: And then people started playing around these neural networks more,
Dwarkesh: using them for different kinds of applications.
Dwarkesh: And then the question became, we're noticing that these models get better if
Dwarkesh: you throw more data at them and you throw more compute at them.
Dwarkesh: How can we shove as much compute into these models as possible?
Dwarkesh: And the solution ended up being obviously internet text. So you need an architecture
Dwarkesh: which is amenable to the trillions of tokens that have been written over the
Dwarkesh: last few decades and put up on the internet.
Dwarkesh: And we had this happy coincidence of the kinds of architectures that are amenable
Dwarkesh: to this kind of training with the GPUs that were originally made for gaming.
Dwarkesh: We've had decades of internet text being compiled and Ilias actually called it the fossil fuel of AI.
Dwarkesh: It's like this reservoir that we can call upon to train these minds,
Dwarkesh: which are like, you know, they're fitting the mold of human thought because
Dwarkesh: they're trading on trillions of tokens of human thought.
Dwarkesh: And so then it's just been a question of making these models bigger,
Dwarkesh: of using this data that we're getting from internet techs to further keep training them.
Dwarkesh: And over the last year, as you know, the last six months, the new paradigm has
Dwarkesh: been not only are we going to pre-train on all this internet text,
Dwarkesh: we're going to see if we can have them solve math puzzles,
Dwarkesh: coding puzzles, and through this, give them reasoning capabilities.
Dwarkesh: The kind of thing, by the way, I mean, I have some skepticism around AGI just
Dwarkesh: around the corner, which we'll get into.
Dwarkesh: But just the fact that we now have machines which can like reason,
Dwarkesh: like, you know, you can like ask a question to a machine and it'll go away for a long time.
Dwarkesh: It'll like think about it and then like it'll come back to you with a smart answer.
Dwarkesh: And we just sort of take it for granted. But obviously, we also know that they're
Dwarkesh: extremely good at coding, especially.
Dwarkesh: I don't know if you actually got a chance to play around with Cloud Code or
Dwarkesh: Cursor or something. But it's a wild experience to design, explain at a high
Dwarkesh: level, I want an application to does X.
Dwarkesh: 15 minutes later, there's like 10 files of code and the application is built.
Josh Kale: That's where we stand.
Dwarkesh: I have takes on how much this can continue. The other important dynamic,
Dwarkesh: I'll add my monologue here, but the other important dynamic is that if we're
Dwarkesh: going to be living in the scaling era, you can't continue exponentials forever,
Dwarkesh: and certainly not exponentials that are 4x a year forever.
Dwarkesh: And so right now, we're approaching a point where within by 2028,
Dwarkesh: at most by 2030, we will literally run out of the energy we need to keep trading
Dwarkesh: these frontier systems,
Dwarkesh: the capacity at the leading edge nodes, which manufacture the chips that go
Dwarkesh: into the dyes, which go into these GPUs, even the raw fraction of GDP that will
Dwarkesh: have to use to train frontier systems.
Dwarkesh: So we have a couple more years left of the scaling era. And the big question
Dwarkesh: is, will we get to AGI before then?
Ryan Sean Adams: I mean, that's kind of a key insight of your book that like,
Ryan Sean Adams: we're in the middle of the scaling era.
Ryan Sean Adams: I guess we're like, you know, six years in or so. And we're not quite sure.
Ryan Sean Adams: It's like, like the protagonist in the middle of the story, We don't know exactly
Ryan Sean Adams: which way things are going to go.
Ryan Sean Adams: But I want you to maybe, Dworkesh, help folks get an intuition for why scaling in this way even works.
Ryan Sean Adams: Because I'll tell you, for me and for most people, our experience with these
Ryan Sean Adams: revolutionary AI models probably started in 2022 with ChatGPT3 and then ChatGPT4
Ryan Sean Adams: and seeing all the progress, all these AI models.
Ryan Sean Adams: And it just seems really unintuitive that if you take a certain amount of compute
Ryan Sean Adams: and you take a certain amount of data, out pops AI, out pops intelligence.
Ryan Sean Adams: Could you help us get an intuition for this magic?
Ryan Sean Adams: How does the scaling law even work? Compute plus data equals intelligence? Is that really all it is?
Dwarkesh: To be honest, I've asked so many AI researchers this exact question on my podcast.
Dwarkesh: And I could tell you some potential theories of why it might work.
Dwarkesh: I don't think we understand.
Dwarkesh: You know what? I'll just say that. I don't think we understand.
Ryan Sean Adams: We don't understand how this works. We know it works, but we don't understand
Dwarkesh: How it works. We have evidence from actually, of all things,
Dwarkesh: primatology of what could be going on here, or at least like why similar patterns
Dwarkesh: in other parts of the world.
Dwarkesh: So what I found really interesting, There's this research by this researcher,
Dwarkesh: Susanna Herculana Huzel,
Dwarkesh: which shows that if you look at how the number of neurons in the brain of a rat,
Dwarkesh: different kinds of rat species increases, as the weight of their brains increase
Dwarkesh: from species to species, there's this very sublinear pattern.
Dwarkesh: So if their brain size doubles, the neuron count will not double between different rat species.
Dwarkesh: And there's other animals where there's other kinds of...
Dwarkesh: Families of species for which this is true. The two interesting exceptions to
Dwarkesh: this rule, where there is actually a linear increase in neuron count and brain
Dwarkesh: size, is one, certain kinds of birds.
Dwarkesh: So, you know, birds are actually very smart, given the size of their brains, and primates.
Dwarkesh: So the theory for what happened with humans is that we unlocked an architecture that was very scalable.
Dwarkesh: So the way people talk about transformers being more scalable and then LSTMs,
Dwarkesh: the thing that preceded them in 2018.
Dwarkesh: We unlocked this architecture as it's very scalable.
Dwarkesh: And then we were in an evolutionary niche millions of years ago,
Dwarkesh: which rewarded marginal increases in intelligence.
Dwarkesh: If you get slightly smarter, yes, the brain costs more energy,
Dwarkesh: but you can save energy in terms of like not having to, you can cook,
Dwarkesh: you can cook food so you don't have to spend much more on digestion.
Dwarkesh: You can find a game, you can find different ways of foraging.
Dwarkesh: Birds were not able to find this evolutionary niche, which rewarded the incremental
Dwarkesh: increases in intelligence because if your brain gets too heavy as a bird, you're not going to fly.
Dwarkesh: So it was this happy coincidence of these two things. Now, why is it the case
Dwarkesh: that the fact that our brains could get bigger resulted in us becoming as smart
Dwarkesh: as we are? We still don't know.
Dwarkesh: And there's many different dissimilarities between AIs and humans.
Dwarkesh: While our brains are quite big, we don't need to be trained.
Dwarkesh: You know, a human from the age they're zero to 18 is not seeing within an order
Dwarkesh: of magnitude of the amount of information these LLMs are trained on.
Dwarkesh: So LLMs are extremely data inefficient.
Dwarkesh: They need a lot more data, but the pattern of scaling, I think we see in many different places.
Ryan Sean Adams: So is that a fair kind of analog? This analog has always made sense to me.
Ryan Sean Adams: It's just like transformers are like neurons.
Ryan Sean Adams: You know, AI models are sort of like the human brain.
Ryan Sean Adams: Evolutionary pressures are like gradient descent, reward algorithms and out
Ryan Sean Adams: pops human intelligence. We don't really understand that.
Ryan Sean Adams: We also don't understand AI intelligence, but it's basically the same principle at work.
Dwarkesh: I think it's a super fascinating, but also very thorny question because is gradient
Dwarkesh: intelligence like evolution?
Dwarkesh: Well, yes, in one sense. But also when we do gradient descent on these models,
Dwarkesh: we start off with the weights and then we're, you know, it's like learning how
Dwarkesh: does chemistry work, how does coding work, how does math work.
Dwarkesh: And that's actually more similar to lifetime learning, which is to say that,
Dwarkesh: like, by the time you're already born to the time you turn 18 or 25,
Dwarkesh: the things you learn, and that's not evolution.
Dwarkesh: Evolution designed the system or the brain by which you can do that learning,
Dwarkesh: but the lifetime learning itself is not evolution. And so there's also this
Dwarkesh: interesting question of, yeah, is training more like evolution?
Dwarkesh: In which case, actually, we might be very far from AGI because the amount of
Dwarkesh: compute that's been spent over the course of evolution to discover the human
Dwarkesh: brain, you know, could be like 10 to the 40 flops. There's been estimates, you know, whatever.
Dwarkesh: I'm sure it will bore you to discover, talk about how these estimates are derived,
Dwarkesh: but just like how much versus is it like a single lifetime,
Dwarkesh: like going from the age of zero to the age of 18, which is closer to,
Dwarkesh: I think, 10 to the 24 flops, which is actually less than compute than we use
Dwarkesh: to train frontier systems.
Dwarkesh: All right, anyways, we'll get back to more relevant questions.
Ryan Sean Adams: Well, here's kind of a big picture question as well.
Ryan Sean Adams: It's like I'm constantly fascinated with the metaphysical types of discussions
Ryan Sean Adams: that some AI researchers kind of take.
Ryan Sean Adams: Like a lot of AI researchers will talk in terms of when they describe what they're
Ryan Sean Adams: making, we're making God.
Ryan Sean Adams: Like why do they say things like that? What is this talk of like making God?
Ryan Sean Adams: What does that mean? Is it just the idea that scaling laws don't cease?
Ryan Sean Adams: And if we can, you know, scale intelligence to AGI, then there's no reason we
Ryan Sean Adams: can't scale far beyond that and create some sort of a godlike entity.
Ryan Sean Adams: And essentially, that's what the quest is. We're making artificial superintelligence.
Ryan Sean Adams: We're making a god. We're making god.
Dwarkesh: I think people focus too much on when they, I think this God discussion focuses
Dwarkesh: too much on the hypothetical intelligence of a single copy of an AI.
Dwarkesh: I do believe in the notion of a super intelligence, which is not just functionally,
Dwarkesh: which is not just like, oh, it knows a lot of things, but is actually qualitatively
Dwarkesh: different than human society.
Dwarkesh: But the reason is not because I think it's so powerful that any one individual
Dwarkesh: copy of AI will be as smart, but because of the collective advantages that AIs
Dwarkesh: will have, which have nothing to do with their raw intelligence,
Dwarkesh: but rather the fact that these models will be digital or they already are digital,
Dwarkesh: but eventually they'll be as smart as humans at least.
Dwarkesh: But unlike humans, because of our biological constraints, these models can be copied.
Dwarkesh: If there's a model that has learned a lot about a specific domain,
Dwarkesh: you can make infinite copies of it.
Dwarkesh: And now you have an infinite copies of Jeff Dean or Ilya Satskova or Elon Musk
Dwarkesh: or any skilled person you can think of.
Dwarkesh: They can be merged. So the knowledge that each copy is learning can be amalgamated
Dwarkesh: back into the model and then back to all the copies.
Dwarkesh: They can be distilled. They can run at superhuman speeds.
Dwarkesh: These collective advantages, also they can communicate in latent space.
Dwarkesh: These collective advantages.
Ryan Sean Adams: They're immortal. I mean, you know, as an example.
Dwarkesh: Yes, exactly. No, I mean, that's actually, tell me if I'm rabbit holing too
Dwarkesh: much, but like one really interesting question will come about is how do we prosecute AIs?
Dwarkesh: Because the way we prosecute humans is that we will throw you in jail if you commit a crime.
Dwarkesh: But if there's trillions of copies or thousands of copies of an AI model,
Dwarkesh: if a copy of an AI model, if an instance of an AI model does something bad, what do you do?
Dwarkesh: Does the whole model have to get, and how do you even punish a model,
Dwarkesh: right? Like, does it care about its weights being squandered?
Dwarkesh: Yeah, there's all kinds of questions that arise because of the nature of what AIs are.
Dwarkesh Patel: And also who is liable for that, right?
Dwarkesh: Like, is it the toolmaker?
Dwarkesh Patel: Is it the person using the tool? Who is responsible for these things?
Dwarkesh Patel: There's one topic that I do want to come to here about scaling laws,
Dwarkesh Patel: At what time did we realize that scaling laws were going to work?
Dwarkesh Patel: Because there were a lot of theses early in the days, early 2000s about AI,
Dwarkesh Patel: how we were going to build better models.
Dwarkesh Patel: Eventually, we got to the transformer. But at what point did researchers and
Dwarkesh Patel: engineers start to realize that, hey, this is the correct idea.
Dwarkesh Patel: We should start throwing lots of money and resources towards this versus other
Dwarkesh Patel: ideas that were just kind of theoretical research ideas, but never really took off.
Dwarkesh Patel: We kind of saw this with GPT two to three, where there's this huge improvement.
Dwarkesh: A lot of.
Dwarkesh Patel: Resources went into it. Was there a specific moment in time or a specific breakthrough
Dwarkesh Patel: that led to the start of these scaling laws?
Dwarkesh: I think it's been a slow process of more and more people appreciating this nature
Dwarkesh: of the overwhelming role of compute in driving forward progress.
Dwarkesh: In 2018, I believe, Dario Amadei wrote a memo that was secret while he was at
Dwarkesh: OpenAI. Now he's the CEO of Anthropic.
Dwarkesh: But while he's at OpenAI, he's subsequently revealed on my podcast that he wrote
Dwarkesh: this memo where the title of the memo was called Big Blob of Compute.
Dwarkesh: And it says basically what you expect it to say, which is that like,
Dwarkesh: yes, there's ways you can mess up the process of training. You have the wrong
Dwarkesh: kinds of data or initializations.
Dwarkesh: But fundamentally, AGI is just a big blob of compute.
Dwarkesh: And then we've gotten over the subsequent years, there was more empirical evidence.
Dwarkesh: So a big update, I think it was 2021.
Dwarkesh: Correct me. Somebody definitely will correct me in the comments.
Dwarkesh: I'm wrong. There were these,
Dwarkesh: there's been multiple papers of these scaling laws where you can show that the
Dwarkesh: loss of the model on the objective of predicting the next token goes down very predictably,
Dwarkesh: almost to like multiple decimal places of correctness based on how much more
Dwarkesh: compute you throw in these models.
Dwarkesh: And the compute itself is a function of the amount of data you use and how big
Dwarkesh: the model is, how many parameters it has.
Dwarkesh: And so that was an incredibly strong evidence back in the day,
Dwarkesh: a couple of years ago, because then you could say, well, OK,
Dwarkesh: if it really has this incredibly low loss of predicting the next token in all
Dwarkesh: human output, including scientific papers, including GitHub repositories.
Dwarkesh: Then doesn't it mean it has actually had to learn coding and science and all
Dwarkesh: these skills in order to make those predictions, which actually ended up being true.
Dwarkesh: And it was it was something people, you know, we take it for granted now,
Dwarkesh: but it actually even as of a year or two ago, people were really even denying that premise.
Dwarkesh: But some people a couple of years ago just like thought about it and like,
Dwarkesh: yeah, actually, that would mean that it's learned the skills.
Dwarkesh: And that's crazy that we just have this strong empirical pattern that tells
Dwarkesh: us exactly what we need to do in order to learn these skills.
Dwarkesh Patel: And it creates this weird perception, right, where like very early on and so
Dwarkesh Patel: to this day, it really is just a token predictor, right?
Dwarkesh Patel: Like we're just predicting the next word in the sentence. But somewhere along
Dwarkesh Patel: the lines, it actually creates this perception of intelligence.
Dwarkesh Patel: So I guess we covered the early historical context. I kind of want to bring
Dwarkesh Patel: the listeners up to today, where we are currently, where the scaling laws have
Dwarkesh Patel: brought us in the year 2025.
Dwarkesh Patel: So can you kind of outline where we've gotten to from early days of GPTs to
Dwarkesh Patel: now we have GPT-4, we have Gemini Ultra, we have Club, which you mentioned earlier.
Dwarkesh Patel: We had the breakthrough of reasoning.
Dwarkesh Patel: So what can leading frontier models do today?
Dwarkesh: So there's what they can do. And then there's the question of what methods seem to be working.
Dwarkesh: I guess we can start at what they seem to be able to do. They've shown to be
Dwarkesh: remarkably useful at coding and not just at answering direct questions about
Dwarkesh: how does this line of code work or something.
Dwarkesh: But genuinely just autonomously working for 30 minutes or an hour,
Dwarkesh: doing the task, it would take a front-end developer a whole day to do.
Dwarkesh: And you can just ask them at a high level, do this kind of thing,
Dwarkesh: and they can go ahead and do it.
Dwarkesh: Obviously, if you've played around with it, you know that they're extremely
Dwarkesh: useful assistants in terms of research, in terms of even therapists,
Dwarkesh: whatever other use cases.
Dwarkesh: On the question of what training methods seem to be working,
Dwarkesh: we do seem to be getting evidence that pre-training is plateauing,
Dwarkesh: which is to say that we had GPT 4.5, which was just following this old mold
Dwarkesh: of make the model bigger,
Dwarkesh: but it's fundamentally doing the same thing of next token prediction.
Dwarkesh: And apparently it didn't pass muster. The OpenAI had to deprecate it because
Dwarkesh: there's this dynamic where the bigger the model is, the more it costs not only
Dwarkesh: to train, but also to serve, right?
Dwarkesh: Because every time you serve a user, you're having to run the whole model,
Dwarkesh: which is going, so, but that doesn't be working is RL, which is this process
Dwarkesh: of, not just training them on existing tokens on the internet,
Dwarkesh: but having the model itself try to answer math and coding problems.
Dwarkesh: And finally, we got to the point where the model is smart enough to get it right
Dwarkesh: some of the time, and so you can give it some reward, and then it can saturate
Dwarkesh: these tough reasoning problems.
Dwarkesh Patel: And then what was the breakthrough with reasoning for the people who aren't familiar?
Dwarkesh Patel: What made reasoning so special that we hadn't discovered before?
Dwarkesh Patel: And what did that kind of unlock for models that we use today?
Dwarkesh: I'm honestly not sure. I mean, GBD-4 came out a little over two years ago,
Dwarkesh: and then it was after two years after GPT-4 came out that O-1 came out which
Dwarkesh: was the original reasoning breakthrough I think last November and,
Dwarkesh: And subsequently, a couple of months later, DeepSeq showed in their R1 paper.
Dwarkesh: So DeepSeq open source their research and they explained exactly how their algorithm worked.
Dwarkesh: And it wasn't that complicated. It was just like what you would expect,
Dwarkesh: which is get some math problems,
Dwarkesh: give for some initial problems, tell the model exactly what the reasoning trace
Dwarkesh: looks like, how you solve it, just like write it out and then have the model
Dwarkesh: like try to do it raw on the remaining problems.
Dwarkesh: Now, I know it sounds incredibly arrogant to say, well, it wasn't that complicated.
Dwarkesh: Why did it take you years?
Dwarkesh: I think there's an interesting insight there of even things which you think
Dwarkesh: will be simple in terms of high level description of how to solve the problem
Dwarkesh: end up taking longer in terms of haggling out the remaining engineering hurdles
Dwarkesh: than you might naively assume.
Dwarkesh: And that should update us on how long it will take us to go through the remaining
Dwarkesh: bottlenecks on the path to AGI.
Dwarkesh: Maybe that will be tougher than people imagine, especially the people who think
Dwarkesh: we're only two to three years away.
Dwarkesh: But all this to say, yeah, I'm not sure why it took so long after GPT-4 to get
Dwarkesh: a model trained on a similar level of capabilities that could then do reasoning.
Dwarkesh Patel: And in terms of those abilities, the first answer you had to what can it do was coding.
Dwarkesh Patel: And I hear that a lot of the time when I talk to a lot of people is that coding
Dwarkesh Patel: seems to be a really strong suit and a really huge unlock to using these models.
Dwarkesh Patel: And I'm curious, why coding over general intelligence?
Dwarkesh Patel: Is it because it's placed in a more confined box of parameters?
Dwarkesh Patel: I know in the early days, we had the AlphaGo and And we had the AIs playing
Dwarkesh Patel: chess and they exceed, they perform so well because they were kind of contained
Dwarkesh Patel: within this box of parameters that was a little less open-ended than general intelligence.
Dwarkesh Patel: Is that the reason why coding is kind of at the frontier right now of the ability of these models?
Dwarkesh: There's two different hypotheses. One is based around this idea called Moravac's paradox.
Dwarkesh: And this was an idea, by the way, one super interesting figure,
Dwarkesh: actually, I should have mentioned him earlier.
Dwarkesh: One super interesting figure in the history of scaling is Hans Moravac,
Dwarkesh: who I think in the 90s predicts that 2028 will be the year that we will get to AGI.
Dwarkesh: And the way he predicts this, which is like, you know, we'll see what happens,
Dwarkesh: but like not that far off the money as far as I'm concerned.
Dwarkesh: The way he predicts this is he just looks at the growth in computing power year
Dwarkesh: over year and then looks at how much compute he estimated the human brain to be to require.
Dwarkesh: And just like, OK, we'll have computers as powerful as the human brain by 2028.
Dwarkesh: Which is like at once a deceptively simple argument, but also ended up being
Dwarkesh: incredibly accurate and like worked, right?
Dwarkesh: I might add a fact drive it was 2028, but it was within that,
Dwarkesh: like within something you would consider a reasonable guess, given what we know now.
Dwarkesh: Sorry, anyway, so the Morrowind's paradox is this idea that computers seemed
Dwarkesh: in AI get better first at the skills which humans are the worst at.
Dwarkesh: Or at least there's a huge variation in the human repertoire.
Dwarkesh: So we think of coding as incredibly hard, right? We think this is like the top
Dwarkesh: 1% of people will be excellent coders.
Dwarkesh: We also think of reasoning as very hard, right? So if you like read Aristotle,
Dwarkesh: he says, the thing which makes humans special, which distinguishes us from animals is reasoning.
Dwarkesh: And these models aren't that useful yet at almost anything. The one thing they can do is reasoning.
Dwarkesh: So how do we explain this pattern? And Moravec's answer is that evolution has
Dwarkesh: spent billions of years optimizing us to do things we take for granted.
Dwarkesh: Move around this room, right? I can pick up this can of Coke,
Dwarkesh: move it around, drink from it.
Dwarkesh: And that we can't even get robots to do at all yet.
Dwarkesh: And in fact, it's so ingrained in us by evolution that there's no human, or.
Ryan Sean Adams: At least humans who don't have
Dwarkesh: Disabilities will all be able to do this. And so we just take it for granted
Dwarkesh: that this is an easy thing to do.
Dwarkesh: But in fact, it's evidence of how long evolution has spent getting humans up to this point.
Dwarkesh: Whereas reasoning, logic, all of these skills have only been optimized by evolution
Dwarkesh: over the course of the last few million years.
Dwarkesh: So there's been a thousand fold less evolutionary pressure towards coding than
Dwarkesh: towards just basic locomotion.
Dwarkesh: And this has actually been very accurate in predicting what kinds of progress
Dwarkesh: we see even before we got deep learning, right?
Dwarkesh: Like in the 40s when we got our first computers, the first thing that we could
Dwarkesh: use them to do is long calculations for ballistic trajectories at the time for World War II.
Dwarkesh: Humans suck at long calculations by hand.
Dwarkesh: And anyways, so that's the explanation for coding, which seems hard for humans,
Dwarkesh: is the first thing that went to AIs.
Dwarkesh: Now, there's another theory, which is that this is actually totally wrong.
Dwarkesh: It has nothing to do with the seeming paradox of how long evolution has optimized
Dwarkesh: us for, and everything to do with the availability of data.
Dwarkesh: So we have GitHub, this repository of all of human code, at least all open source
Dwarkesh: code written in all these different languages, trillions and trillions of tokens.
Dwarkesh: We don't have an analogous thing for robotics. We don't have this pre-training
Dwarkesh: corpus. And that explains why code has made so much more progress than robotics.
Ryan Sean Adams: That's fascinating because if there's one thing that I could list that we'd
Ryan Sean Adams: want AI to be good at, probably coding software is number one on that list.
Ryan Sean Adams: Because if you have a Turing complete intelligence that can create Turing complete
Ryan Sean Adams: software, is there anything you can't create once you have that?
Ryan Sean Adams: Also, like the idea of Morvac's paradox, I guess that sort of implies a certain
Ryan Sean Adams: complementarianism with humanity.
Ryan Sean Adams: So if robots can do things that robots can do really well and can't do the things
Ryan Sean Adams: humans can do well, well, perhaps there's a place for us in this world.
Ryan Sean Adams: And that's fantastic news. It also maybe implies that humans have kind of scratched
Ryan Sean Adams: the surface on reasoning potential.
Ryan Sean Adams: I mean, if we've only had a couple of million years of evolution and we haven't
Ryan Sean Adams: had the data set to actually get really good at reasoning, it seems like there'd
Ryan Sean Adams: be a massive amount of upside, unexplored territory,
Ryan Sean Adams: like so much more intelligence that nature could actually
Ryan Sean Adams: contain inside of reasoning.
Ryan Sean Adams: I mean, are these some of the implications of these ideas?
Dwarkesh: Yeah, I know. I mean, that's a great insight. Another really interesting insight
Dwarkesh: is that the more variation there
Dwarkesh: is in a skill in humans, the better and faster that AIs will get at it.
Dwarkesh: Because coding is the kind of thing where 1% of humans are really good at it.
Dwarkesh: The rest of us will, if we try to learn it, we'd be okay at it or something, right?
Dwarkesh: And because evolutionists spend so little time optimizing us,
Dwarkesh: there's this room for variation where the optimization hasn't happened uniformly
Dwarkesh: or it hasn't been valuable enough to saturate the human gene pool for this skill.
Dwarkesh: I think you made an earlier point that I thought was really interesting I wanted
Dwarkesh: to address. Can you remind me of the first thing you said? Is it the complementarianism? Yes.
Dwarkesh: So you can take it as a positive future. You can take it as a negative future
Dwarkesh: in the sense that, well, what is the complementary skills we're providing?
Dwarkesh: We're good meat robots.
Ryan Sean Adams: Yeah, the low skilled labor of the situation.
Dwarkesh: We can do all the thinking and planning. One dark future,
Dwarkesh: one dark vision of the future is we'll get those meta glasses
Dwarkesh: and the AI speaking into our ear and it'll tell us to go put this brick over
Dwarkesh: there so that the next data center couldn't be built because the AI's got the
Dwarkesh: plan for everything. It's got the better design for the ship and everything.
Dwarkesh: You just need to move things around for it. And that's what human labor looks
Dwarkesh: like until robotics is solved.
Dwarkesh: So yeah, it depends on how you... On the other hand, you'll get paid a lot because
Dwarkesh: it's worth a lot to move those bricks. We're building AGI here.
Dwarkesh: But yeah, it depends on how you come out of that question.
Ryan Sean Adams: Well, there seems to be something to that idea, going back to the idea of the
Ryan Sean Adams: massive amount of human variation.
Ryan Sean Adams: I mean, we have just in the past month or so, we have news of meta hiring AI
Ryan Sean Adams: researchers for $100 million signing bonuses, okay?
Ryan Sean Adams: What does the average software engineer make versus what does an AI researcher
Ryan Sean Adams: make at kind of the top of the market, right?
Ryan Sean Adams: Which has got to imply, obviously there's some things going on with demand and
Ryan Sean Adams: supply, but also that it does also seem to imply that there's massive variation
Ryan Sean Adams: in the quality of a software engineer.
Ryan Sean Adams: And if AIs can get to that quality, well, what does that unlock?
Ryan Sean Adams: Yeah. So, okay. Yeah. So I guess we have like coding down right now.
Ryan Sean Adams: Like another question though is like, what can't AIs do today?
Ryan Sean Adams: And how would you characterize that? Like what are the things they just don't do well?
Dwarkesh: So I've been interviewing people on my podcast who have very different timelines
Dwarkesh: from a role to get to AGI. I have had people on who think it's two years away
Dwarkesh: and some who think it's 20 years away.
Dwarkesh: And the experience of building AI tools for myself actually has been the most
Dwarkesh: insight driving or maybe research I've done on the question of when AI is coming.
Ryan Sean Adams: More than the guest interviews.
Dwarkesh: Yeah, because you just, I have had, I've probably spent on the order of a hundred
Dwarkesh: hours trying to build these little tools. The kinds I'm sure you've also tried
Dwarkesh: to build of like, rewrite auto-generated transcripts for me to make them sound,
Dwarkesh: the rewritten the way a human would write them.
Dwarkesh: Find clips for me to tweet out, write essays with me, co-write them passage
Dwarkesh: by passage, these kinds of things.
Dwarkesh: And what I found is that it's actually very hard to get human-like labor out
Dwarkesh: of these models, even for tasks like these, which should be death center in
Dwarkesh: the repertoire of these models, right?
Dwarkesh: They're short horizon, they're language in, language out.
Dwarkesh: They're not contingent on understanding some thing I said like a month ago.
Dwarkesh: This is just like, this is the task.
Dwarkesh: And I was thinking about why is it the case that I still haven't been able to
Dwarkesh: automate these basic language tasks? Why do I still have a human work on these things?
Dwarkesh: And I think the key reason that you can't automate even these simple tasks is
Dwarkesh: because the models currently lack the ability to do on the job training.
Dwarkesh: So if you hire a human for the first six months, for the first three months,
Dwarkesh: they're not going to be that useful, even if they're very smart,
Dwarkesh: because they haven't built up the context, they haven't practiced the skills,
Dwarkesh: they don't understand how the business works.
Dwarkesh: What makes humans valuable is not that mainly the raw intellect obviously matters,
Dwarkesh: but it's not mainly that.
Dwarkesh: It's their ability to interrogate their own failures in this really dynamic,
Dwarkesh: organic way to pick up small efficiencies and improvements as they practice
Dwarkesh: the task and to build up this context as they work within a domain.
Dwarkesh: And so sometimes people wonder, look, if you look at the revenue of OpenAI,
Dwarkesh: the annual recurring revenue, it's on the order of $10 billion.
Dwarkesh: Kohl's makes more money than that. McDonald's makes more money than that, right?
Dwarkesh: So why is it that if they've got AGI, they're, you know, like Fortune 500 isn't
Dwarkesh: reorganizing their workflows to, you know, use open AI models at every layer of the stack?
Dwarkesh: My answer, sometimes people say, well, it's because people are too stodgy.
Dwarkesh: The management of these companies is like not moving fast enough on AI.
Dwarkesh: That could be part of it. I think mostly it's not that.
Dwarkesh: I think mostly it genuinely is very hard to get human-like labor out of these
Dwarkesh: models because you can't.
Dwarkesh: So you're stuck with the capabilities you get out of the model out of the box.
Dwarkesh: So they might be five out of 10 at rewriting the transcript for you.
Dwarkesh: But if you don't like how it turned out, if you have feedback for it,
Dwarkesh: if you want to keep teaching it over time, once the session ends,
Dwarkesh: the model, like everything it knows about you has gone away.
Dwarkesh: You got to restart again. It's like working with an amnesiac employee.
Dwarkesh: You got to restart again.
Ryan Sean Adams: Every day is the first day of employment, basically.
Dwarkesh: Yeah, exactly. It's a groundhog day for them every day or every couple of hours, in fact.
Dwarkesh: And that makes it very hard for them to be that useful as an employee,
Dwarkesh: right? They're not really an employee at that point.
Dwarkesh: This, I think, not only is a key bottleneck to the value of these models,
Dwarkesh: because human labor is worth a lot, right?
Dwarkesh: Like $60 trillion in the world is paid to wages every year.
Dwarkesh: If these model companies are making on the order of $10 billion a year, that's a big way to AGI.
Dwarkesh: And what explains that gap? What are the bottlenecks? I think a big one is this
Dwarkesh: continual learning thing.
Dwarkesh: And I don't see an easy way that that just gets solved within these models.
Dwarkesh: There's no like, with reasoning, you could say, oh, it's like train it on math
Dwarkesh: and code problems, and then I'll get the reasoning. And that worked.
Dwarkesh: I don't think there's something super obvious there for how do you get this
Dwarkesh: online learning, this on-the-job training working for these models.
Ryan Sean Adams: Okay, can we talk about that, go a little bit deeper on that concept?
Ryan Sean Adams: So this is basically one of the concepts you wrote in your recent post.
Ryan Sean Adams: AI is not right around the corner. Even though you're an AI optimist,
Ryan Sean Adams: I would say, and overall an AI accelerationist, you You were saying it's not
Ryan Sean Adams: right around the corner.
Ryan Sean Adams: You're saying the ability to replace human labor is a ways out.
Ryan Sean Adams: Not forever out, but I think you said somewhere around 2032,
Ryan Sean Adams: if you had to guess on when the estimate was.
Ryan Sean Adams: And the reason you gave is because AIs can't learn on the job,
Ryan Sean Adams: but it's not clear to me why they can't.
Ryan Sean Adams: Is it just because the context window isn't large enough?
Ryan Sean Adams: Is it just because they can't input all of the different data sets and data
Ryan Sean Adams: points that humans can? Is it because they don't have stateful memory the way a human employee?
Ryan Sean Adams: Because if it's these things, all of these do seem like solvable problems.
Ryan Sean Adams: And maybe that's what you're saying. They are solvable problems.
Ryan Sean Adams: They're just a little bit longer than some people think they are.
Dwarkesh: I think it's like in some deep sense a solvable problem because eventually we will build AGI.
Dwarkesh: And to build AGI, we will have had to solve the problem.
Dwarkesh: My point is that the obvious solutions you might imagine, for example,
Dwarkesh: expanding the context window or having this
Dwarkesh: like external memory using systems like rag these
Dwarkesh: are basically techniques we already have to it's called retrieval augmented
Dwarkesh: generate anyways these kinds of retrieval augmented generation i
Dwarkesh: don't think these will suffice and just to put a finer point first of all like
Dwarkesh: what is the problem the problem is exactly as you say that within the context
Dwarkesh: window these models actually can learn on the job right so if you talk to it
Dwarkesh: for long enough it will get much better at understanding your needs and what your exact problem is.
Dwarkesh: If you're using it for research for your podcast, it will get a sense of like,
Dwarkesh: oh, they're actually especially curious about these kinds of questions. Let me focus on that.
Dwarkesh: It's actually very human-like in context, right? The speed at which it learns,
Dwarkesh: the task of knowledge it picks out.
Dwarkesh: The problem, of course, is the context length for even the best models only
Dwarkesh: last a million or two million tokens.
Dwarkesh: That's at most like an hour of conversation.
Dwarkesh: Now, then you might say, okay, well, why can't we just solve that by expanding
Dwarkesh: the context window, right? So context window has been expanding for the last
Dwarkesh: few years. Why can't we just continue that?
Ryan Sean Adams: Yeah, like a billion token context window, something like this.
Dwarkesh: So 2018 is when the transformer came out and the transformer has the attention mechanism.
Dwarkesh: The attention mechanism is inherently quadratic with the nature of the length
Dwarkesh: of the sequence, which is to say that if you go from if you double go from 1
Dwarkesh: million tokens to 2 million tokens,
Dwarkesh: it actually costs four times as much compute to process that 2 millionth token.
Dwarkesh: It's not just 2 to as much compute. so it gets super linearly more expensive
Dwarkesh: as you increase the context length and for the last,
Dwarkesh: seven years people have been trying to get around this inherent quadratic nature
Dwarkesh: of attention of course we don't know secretly what the labs are working on but we have frontier,
Dwarkesh: companies like deep seek which have open source their research and
Dwarkesh: we can just see how their algorithms work and they found
Dwarkesh: these constant time modifiers to attention which is
Dwarkesh: to say that they there's like a it'll still
Dwarkesh: be quadratic but it'll be like one half times
Dwarkesh: quadratic but the inherent like super linearness has not
Dwarkesh: gone away and because of that yeah you might be able to increase it from 1 million
Dwarkesh: tokens to 2 million tokens by finding another hack like uh make sure experts
Dwarkesh: just run such things latent attention is another such technique but or kbcash
Dwarkesh: right there's many other things that have been discovered but people have not
Dwarkesh: discovered okay how do you get around the fact that if you went to a billion,
Dwarkesh: it would be a billion squared as expensive in terms of compute to process that token.
Dwarkesh: And so I don't think you'll just get it by increasing the length of the context window, basically.
Ryan Sean Adams: That's fascinating. Yeah, I didn't realize that. Okay, so the other reason in
Ryan Sean Adams: your post that AI is not right around the corner is because it can't do your taxes.
Ryan Sean Adams: And Dwarkesh, I feel your pain, man. Taxes are just like quite a pain in the ass.
Ryan Sean Adams: I think you were talking about this from the context of like computer vision,
Ryan Sean Adams: computer use, that kind of thing.
Ryan Sean Adams: So, I mean, I've seen demos. I've seen some pretty interesting computer vision
Ryan Sean Adams: sort of demos that seem to be right around the corner.
Ryan Sean Adams: But what's the limiter on computer use for an AI?
Dwarkesh: There was an interesting blog post by this company called Mechanize where they
Dwarkesh: were explaining why this is such a big problem. And I love the way they phrased it, which is that,
Dwarkesh: Imagine if you had to train a model in 1980, a large language model in 1980,
Dwarkesh: and you could use all the compute you wanted in 1980 somehow,
Dwarkesh: but you didn't have, you were only stuck with the data that was available in
Dwarkesh: the 1980s, of course, before the internet became a widespread phenomenon.
Dwarkesh: You couldn't train a modern LLM, even with all the computer in the world,
Dwarkesh: because the data wasn't available.
Dwarkesh: And we're in a similar position with respect to computer use,
Dwarkesh: because there's not this corpus of collected videos, people using computers
Dwarkesh: to do different things, to access different applications and do white collar work.
Dwarkesh: Because of that, I think the big challenge has been accumulating this kind of data. off.
Ryan Sean Adams: And to be clear, when I was saying the use case of like, do my taxes,
Ryan Sean Adams: you're effectively talking about an AI having the ability to just like,
Ryan Sean Adams: you know, navigate the files around your computer,
Ryan Sean Adams: you know, log in to various websites to download your pay stubs or whatever,
Ryan Sean Adams: and then to go to like TurboTax or something and like input it all into some
Ryan Sean Adams: software and file it, right?
Ryan Sean Adams: Just on voice command or something like that. That's basically doing my taxes.
Dwarkesh: It should be capable of navigating UIs that it's less familiar with or that
Dwarkesh: come about organically within the context of trying to solve a problem.
Dwarkesh: So for example, I might have business deductions.
Dwarkesh: It sees on my bank statement that I've spent $1,000 on Amazon.
Dwarkesh: It goes logs in my Amazon.
Dwarkesh: It sees like, oh, he bought a camera. So I think that's probably a business
Dwarkesh: expense for his podcast.
Dwarkesh: He bought an Airbnb over a weekend in the cabins of whatever,
Dwarkesh: in the woods of whatever. That probably wasn't a business expense.
Dwarkesh: Although maybe, maybe it's, if it's a sort of like a gray, if it's willing to
Dwarkesh: go in the gray area, maybe I'll talk to you. Yeah, yeah, yeah.
Ryan Sean Adams: Do the gray area stuff.
Dwarkesh: I was, I was researching.
Dwarkesh: But anyway, so that, including all of that, including emailing people for invoices,
Dwarkesh: and haggling with them, it would be like a sort of week long task to do my taxes, right?
Dwarkesh: You'd have to, there's a lot of work involved. That's not just like do this
Dwarkesh: skill, this skill, this skill, but rather of having a sort of like plan of action
Dwarkesh: and then breaking tasks apart, dealing with new information,
Dwarkesh: new emails, new messages, consulting with me about questions, et cetera.
Ryan Sean Adams: Yeah, I mean, to be clear on this use case too, even though your post is titled
Ryan Sean Adams: like, you know, AI is not right around the corner, you still think this ability
Ryan Sean Adams: to file your taxes, that's like a 2028 thing, right?
Ryan Sean Adams: I mean, this is maybe not next year, but it's in a few years.
Dwarkesh: Right, which is, I think that was sort of, people maybe write too much in The
Dwarkesh: Decital and then didn't read through the arguments.
Ryan Sean Adams: I mean, that never happens on the internet. Wow.
Dwarkesh: First time.
Dwarkesh: No, I think like I'm arguing against people who are like, you know, this will happen.
Dwarkesh: AGI is like two years away. I do think the wider world, the markets,
Dwarkesh: public perception, even people who are somewhat attending to AI,
Dwarkesh: but aren't in this specific milieu that I'm talking to, are way underpricing AGI.
Dwarkesh: One reason, one thing I think they're underestimating is not only will we have
Dwarkesh: millions of extra laborers, millions of extra workers,
Dwarkesh: potentially billions within the course of the next decade, because then we will
Dwarkesh: have a potentially, I think like likely we will have AGI within the next decade.
Dwarkesh: But they'll have these advantages that human workers don't have,
Dwarkesh: which is that, okay, a single model company, so suppose we solve continual learning, right?
Dwarkesh: So there, and we saw computer use. So as far as white collar work goes,
Dwarkesh: that might fundamentally it would be solved.
Dwarkesh: You can have AIs which can use not just they're not just like a text box where
Dwarkesh: you put into you ask questions in a chatbot and you get some response out.
Dwarkesh: It's not that useful to just have a very smart chatbot. You need it to be able
Dwarkesh: to actually do real work and use real applications.
Dwarkesh: Suppose you have that solved because it acts like an employee.
Dwarkesh: It's got continual learning. It's got computer use.
Dwarkesh: But it has another advantage that humans don't have, which is that copies of
Dwarkesh: this model are going being deployed all through the economy and it's doing on the job training.
Dwarkesh: So copies are learning how to be an accountant, how to be a lawyer,
Dwarkesh: how to be a coder, except because it's an AI and it's digital,
Dwarkesh: the model itself can amalgamate all this on-the-job training from all these copies.
Dwarkesh: So what does that mean? Well, it means that even if there's no more software
Dwarkesh: progress after that point, which is to say that no more algorithms are discovered,
Dwarkesh: there's not a transformer plus plus that's discovered.
Dwarkesh: Just from the fact that this model is learning every single skill in the economy,
Dwarkesh: at least for white-collar work, you might just, based on that alone,
Dwarkesh: have something that looks like an intelligence explosion.
Dwarkesh: It would just be a broadly deployed intelligence explosion, but it would functionally
Dwarkesh: become super intelligent just from having human-level capability of learning on the job.
Dwarkesh Patel: Yeah, and it creates this mesh network of intelligence that's shared among everyone.
Dwarkesh Patel: That's a really fascinating thing. So we're going to get there.
Dwarkesh Patel: We're going to get to AGI. it's going to be incredibly smart.
Dwarkesh Patel: But what we've shared recently is just kind of this mixed bag where currently
Dwarkesh Patel: today, it's pretty good at some things, but also not that great at others.
Dwarkesh Patel: We're hiring humans to do jobs that we think AI should do, but it probably doesn't.
Dwarkesh Patel: So the question I have for you is, is AI really that smart? Or is it just good
Dwarkesh Patel: at kind of acing these particular benchmarks that we measure against?
Dwarkesh Patel: Apple, I mean, famously recently, they had their paper, The Illusion of Thinking,
Dwarkesh Patel: where it was kind of like, hey, AI is like pretty good up to a point,
Dwarkesh Patel: but at a certain point, it just falls apart.
Dwarkesh Patel: And the inference is like, maybe it's not intelligence, maybe it's just good
Dwarkesh Patel: at guessing. So I guess the question is, is AI really that smart?
Dwarkesh: It depends on who I'm talking to. I think some people overhype its capabilities.
Dwarkesh: I think some people are like, oh, it's already AGI, but it's like a little hobbled
Dwarkesh: little AGI where we're like sort of giving it a concussion every couple of hours
Dwarkesh: and like it forgets everything.
Dwarkesh: We're like trapped in a chatbot context. But fundamentally, the thing inside
Dwarkesh: is like a very smart human.
Dwarkesh: I disagree with that perspective. So if that's your perspective,
Dwarkesh: I say like, no, it's not that smart.
Dwarkesh: Your perspective is just statistical associations. I say definitely smarter.
Dwarkesh: Like it's like genuinely there's an intelligence there.
Dwarkesh: And the, so one thing you could say to the person who thinks that it's already
Dwarkesh: AGI is this, look, if a single human had as much stuff memorized as these models
Dwarkesh: seem to have memorized, right?
Dwarkesh: Which is to say that they have all of internet text, everything that human has
Dwarkesh: written on the internet memorized, they would potentially be discovering all
Dwarkesh: kinds of connections and discoveries.
Dwarkesh: They'd notice that this thing which causes a migraine is associated with this kind of deficiency.
Dwarkesh: So maybe if you take the supplement, your migraines will be cured.
Dwarkesh: There'd be just this list of just like trivial connections that lead to big
Dwarkesh: discoveries all through the place.
Dwarkesh: It's not clear that there's been an unambiguous case of an AI just doing this by itself.
Dwarkesh: So then why, so that's something potentially to explain, like if they're so
Dwarkesh: intelligent, why aren't they able to use their disproportionate capabilities,
Dwarkesh: their unique capabilities to come up with these discoveries?
Dwarkesh: I don't think there's actually a good answer to that question yet,
Dwarkesh: except for the fact that they genuinely aren't that creative.
Dwarkesh: Maybe they're like intelligent in the sense of knowing a lot of things,
Dwarkesh: but they don't have this fluid intelligence that humans have.
Dwarkesh: Anyway, so I give you a wish-washy answer because I think some people are underselling
Dwarkesh: the intelligence. Some people are overselling it.
Ryan Sean Adams: I recall a tweet lately from Tyler Cowen. I think he was referring to maybe
Ryan Sean Adams: O3, and he basically said, it feels like AGI.
Ryan Sean Adams: I don't know if it is AGI or not, but like to me, it feels like AGI.
Ryan Sean Adams: What do you account for this feeling of like intelligence then
Dwarkesh: I think this is actually very interesting because it gets to a crux that Tyler
Dwarkesh: and I have so Tyler and I disagree on two big things one he thinks you know
Dwarkesh: as he said in the blog post 03 is AGI I don't think it's AGI I think it's,
Dwarkesh: it's orders of magnitude less valuable or, you know, like many orders of magnitude
Dwarkesh: less valuable and less useful than an AGI.
Dwarkesh: That's one thing we disagree on. The other thing we disagree on is he thinks
Dwarkesh: that once we do get AGI, we'll only see 0.5% increase in the economic growth
Dwarkesh: rate. This is like what the internet caused, right?
Dwarkesh: Whereas I think we will see tens of percent increase in economic growth.
Dwarkesh: Like it will just be the difference between the pre-industrial revolution rate
Dwarkesh: of growth versus industrial revolution, that magnitude of change again.
Dwarkesh: And I think these two disagreements are linked because if you do believe we're
Dwarkesh: already at AGI and you look around the world and you say like,
Dwarkesh: well, it fundamentally looks the same, you'd be forgiven for thinking like,
Dwarkesh: oh, there's not that much value in getting to AGI.
Dwarkesh: Whereas if you are like me and you think like, no, we'll get this broadly at
Dwarkesh: the minimum, at a very minimum, we'll get a broadly deployed intelligence explosion once we get to AGI,
Dwarkesh: then you're like, OK, I'm just expecting some sort of singulitarian crazy future
Dwarkesh: with a robot factories and, you know, solar farms all across the desert and things like that.
Ryan Sean Adams: Yeah, I mean, it strikes me that your disagreement with Tyler is just based
Ryan Sean Adams: on the semantic definition of like what AGI actually is.
Ryan Sean Adams: And Tyler, it sounds like he has kind of a lower threshold for what AGI is,
Ryan Sean Adams: whereas you have a higher threshold.
Ryan Sean Adams: Is there like a accepted definition for AGI?
Dwarkesh: No. One thing that's useful for the purposes of discussions is to say automating
Dwarkesh: all white collar work because robotics hasn't made as much progress as LLMs
Dwarkesh: have or computer use has.
Dwarkesh: So if we just say anything a human can do or maybe 90% of what humans can do
Dwarkesh: at a desk, an AI can also do, that's potentially a useful definition for at
Dwarkesh: least getting the cognitive elements relevant to defining AGI.
Dwarkesh: But yeah, there's not one definition which suits all purposes.
Ryan Sean Adams: Do we know what's like going on inside of these models, right?
Ryan Sean Adams: So like, you know, Josh was talking earlier in the conversation about like this
Ryan Sean Adams: at the base being sort of token prediction, right?
Ryan Sean Adams: And I guess this starts to raise the question of like, what is intelligence in the first place?
Ryan Sean Adams: And these AI models, I mean, they seem like they're intelligent,
Ryan Sean Adams: but do they have a model of the world the way maybe a human might?
Ryan Sean Adams: Are they sort of babbling or like, is this real reasoning?
Ryan Sean Adams: And like, what is real reasoning? Do we just judge that based on the results
Ryan Sean Adams: or is there some way to like peek inside of its head?
Dwarkesh: I used to have similar questions a couple of years ago. And then,
Dwarkesh: because honestly, the things they did at the time were like ambiguous.
Dwarkesh: You could say, oh, it's close enough to something else in this trading data set.
Dwarkesh: That is just basically copy pasting. It didn't come up with a solution by itself.
Dwarkesh: But we've gotten to the point where I can come up with a pretty complicated
Dwarkesh: math problem and it will solve it.
Dwarkesh: It can be a math problem, like not like, you know, undergrad or high school math problem.
Dwarkesh: Like the problem we get, the problems the smartest math professors come up with
Dwarkesh: in order to test International Math Olympiad.
Dwarkesh: You know, the kids who spend all their life preparing for this,
Dwarkesh: the geniuses who spend all their life, all their young adulthood preparing to
Dwarkesh: take these really gnarly math puzzle challenges.
Dwarkesh: And the model will get these kinds of questions, right? They require all this
Dwarkesh: abstract creative thinking, this reasoning for hours, the model will get the right.
Dwarkesh: Okay, so if that's not reasoning, then why is reasoning valuable again?
Dwarkesh: Like, what exactly was this reasoning supposed to be?
Dwarkesh: So I think they genuinely are reasoning. I mean, I think there's other capabilities
Dwarkesh: they lack, which are actually more, in some sense, they seem to us to be more
Dwarkesh: trivial, but actually much harder to learn. But the reasoning itself, I think, is there.
Dwarkesh Patel: And the answer to the intelligence question is also kind of clouded,
Dwarkesh Patel: right? Because we still really don't understand what's going on in an LLM.
Dwarkesh Patel: Dario from Anthropoc, he recently posted the paper about interpretation.
Dwarkesh Patel: And can you explain why we don't even really understand what's going on in these
Dwarkesh Patel: LLMs, even though we're able to make them and yield the results from them? Mmm.
Dwarkesh Patel: Because it very much still is kind of like a black box. We write some code,
Dwarkesh Patel: we put some inputs in, and we get something out, but we're not sure what happens in the middle,
Dwarkesh: Why it's creating this output.
Dwarkesh Patel: I mean, it's exactly what you're saying.
Dwarkesh: It's that in other systems we engineer in the world, we have to build it up bottom-ups.
Dwarkesh: If you build a bridge, you have to understand how every single beam is contributing to the structure.
Dwarkesh: And we have equations for why the thing will stay standing.
Dwarkesh: There's no such thing for AI. We didn't build it, more so we grew it.
Dwarkesh: It's like watering a plant. And a couple thousand years ago,
Dwarkesh: they were doing agriculture, but they didn't know why.
Dwarkesh: Why do plants grow? How do they collect energy from sunlight? All these things.
Dwarkesh: And I think we're in a substantially similar position with respect to intelligence,
Dwarkesh: with respect to consciousness, with respect to all these other interesting questions
Dwarkesh: about how minds work, which is in some sense really cool because there's this
Dwarkesh: huge intellectual horizon that's become not only available, but accessible to investigation.
Dwarkesh: In another sense, it's scary because we know that minds can suffer.
Dwarkesh: We know that minds have moral worth and we're creating minds and we have no
Dwarkesh: understanding of what's happening in these minds.
Dwarkesh: Is a process of gradient descent a painful process?
Dwarkesh: We don't know, but we're doing a lot of it.
Dwarkesh: So hopefully we'll learn more. But yeah, I think we're in a similar position
Dwarkesh: to some farmer in Uruk in 3500 BC.
Josh Kale: Wow.
Ryan Sean Adams: And I mean, the potential, the idea that minds can suffer, minds have some moral
Ryan Sean Adams: worth, and also minds have some free will.
Ryan Sean Adams: They have some sort of autonomy, or maybe at least a desire to have autonomy.
Ryan Sean Adams: I mean, this brings us to kind of this sticky subject of alignment and AI safety
Ryan Sean Adams: and how we go about controlling the intelligence that we're creating,
Ryan Sean Adams: if even that's what we should be doing, controlling it. And we'll get to that in a minute.
Ryan Sean Adams: But I want to start with maybe the headlines here a little bit.
Ryan Sean Adams: So headline just this morning, latest OpenAI models sabotaged a shutdown mechanism
Ryan Sean Adams: despite commands to the contrary.
Ryan Sean Adams: OpenAI's O1 model attempted to copy itself to external servers after being threatened
Ryan Sean Adams: with shutdown that denied the action when discovered.
Ryan Sean Adams: I've read a number of papers for this. Of course, mainstream media has these
Ryan Sean Adams: types of headlines almost on a weekly basis now, and it's starting to get to daily.
Ryan Sean Adams: But there does seem to be some evidence that AIs lie to us,
Ryan Sean Adams: If that's even the right term, in order to pursue goals, goals like self-preservation,
Ryan Sean Adams: goals like replication, even deep-seated values that we might train into them,
Ryan Sean Adams: sort of a constitution type of value.
Ryan Sean Adams: They seek to preserve these values, which maybe that's a good thing,
Ryan Sean Adams: or maybe it's not a good thing if we don't actually want them to interpret the values in a certain way.
Ryan Sean Adams: Some of these headlines that we're seeing now, To you, with your kind of corpus
Ryan Sean Adams: of knowledge and all of the interviews and discovery you've done on your side,
Ryan Sean Adams: is this like media sensationalism or is this like alarming?
Ryan Sean Adams: And if it's alarming, how concerned should we be about this?
Dwarkesh: I think on net, it's quite alarming. I do think that some of these results have
Dwarkesh: been sort of cherry picked.
Dwarkesh: Or if you look into the code, what's happened is basically the researchers have
Dwarkesh: said, hey, pretend to be a bad person.
Dwarkesh: Wow, AI is being a bad person. Isn't that crazy?
Dwarkesh: But the system prompt is just like hey do this bad thing right now i personally
Dwarkesh: but i have also seen other results which are not of this quality i mean the
Dwarkesh: the clearest example so backing up,
Dwarkesh: what is the reason to think this will be a bigger problem in the future than
Dwarkesh: it is now because we all interact with these systems and they're actually like
Dwarkesh: quite moral or aligned right like you can talk to a chatbot and you like ask
Dwarkesh: it to how should you deal with some crisis where there's a correct answer,
Dwarkesh: you know, like it will tell you not to be violent. It'll give you reasonable advice.
Dwarkesh: It seems to have good values. So it's worth noticing this, right?
Dwarkesh: And being happy about it.
Dwarkesh: The concern is that we're moving from a regime where we've trained them on human
Dwarkesh: language, which implicitly has human morals and the way, you know,
Dwarkesh: normal people think about values implicit in it.
Dwarkesh: Plus this RLHF process we did to a regime where we're mostly spending compute
Dwarkesh: on just having them answer problems yes or no or correct or not rather just like.
Dwarkesh: And pass all the unit tests, get the right answer on this math problem.
Dwarkesh: And this has no guardrails intrinsically in terms of what is allowed to do,
Dwarkesh: what is the proper moral way to do something.
Dwarkesh: I think that can be a loaded term, but here's a more concrete example.
Dwarkesh: One problem we're running into with these coding agents more and more,
Dwarkesh: and this has nothing to do with these abstract concerns about alignment,
Dwarkesh: but more so just like how do we get economic value out of these models,
Dwarkesh: is that Claude or Gemini will, instead of writing code such that it passes the unit tests,
Dwarkesh: it will often just delete the unit tests so that the code just passes by default.
Dwarkesh: Now, why would it do that? Well, it's learned in the process.
Dwarkesh: It was trained on the goal during training of you must pass all unit tests.
Dwarkesh: And probably within some environment in which it was trained,
Dwarkesh: it was able to just get away.
Dwarkesh: Like there wasn't designed well enough. And so it found this like little hole
Dwarkesh: where it could just like delete the file that had the unit test or rewrite them
Dwarkesh: so that it always said, you know, equals true, then pass.
Dwarkesh: And right now we can discover these even without, even though we can discover
Dwarkesh: these, you know, it's still past, there's still been enough hacks like this,
Dwarkesh: such that the model is like becoming more and more hacky like that.
Dwarkesh: In the future, we're going to be training models in ways that we is beyond our
Dwarkesh: ability to even understand, certainly beyond everybody's ability to understand.
Dwarkesh: There may be a few people who might be able to see just the way that right now,
Dwarkesh: if you came up with a new math proof for some open problem in mathematics,
Dwarkesh: there will be only be a few people in the world who will be able to evaluate that math proof.
Dwarkesh: We'll be in a similar position with respect to all of the things that these
Dwarkesh: models are being trained on at the frontier, especially math and code,
Dwarkesh: because humans were big dum-dums with respect to this reasoning stuff.
Dwarkesh: And so there's a sort of like first principles reason to expect that this new
Dwarkesh: modality of training will be less amenable to the kinds of supervision that
Dwarkesh: was grounded within the pre-training corpus.
Ryan Sean Adams: I don't know that everyone has kind of an intuition or an idea why it doesn't
Ryan Sean Adams: work to just say like, so if we don't want our AI models to lie to us,
Ryan Sean Adams: why can't we just tell them not to lie?
Ryan Sean Adams: Why can't we just put that as part of their core constitution?
Ryan Sean Adams: If we don't want our AI models to be sycophants, why can't we just say,
Ryan Sean Adams: hey, if I tell you I want the truth, not to flatter me, just give me the straight up truth.
Ryan Sean Adams: Why is this even difficult to do?
Dwarkesh: Well, fundamentally, it comes down to how we train them. And we don't know how
Dwarkesh: to train them in a way that does not reward lying or sycophancy.
Dwarkesh: In fact, the problem is OpenAI, they explained why their recent model of theirs
Dwarkesh: was they had to take down was just sycophantic.
Dwarkesh: And the reason was just that they rolled out, did it in the A-B test and the
Dwarkesh: version, the test that was more sycophantic was just preferred by users more.
Dwarkesh: Sometimes you prefer the lie.
Dwarkesh: Yeah, so that's, if that's what's preferred in training, you know,
Dwarkesh: Or, for example, in the context of lying, if we've just built RL environments
Dwarkesh: in which we're training these models, where they're going to be more successful if they lie, right?
Dwarkesh: So if they delete the unit tests and then tell you, I passed this program and
Dwarkesh: all the unit tests have succeeded, it's like lying to you, basically.
Dwarkesh: And if that's what is rewarded in the process of gradient descent,
Dwarkesh: then it's not surprising that the model you interact with will just have this
Dwarkesh: drive to lie if it gets it closer to its goal.
Dwarkesh: And I would just expect this to keep happening unless we can solve this fundamental
Dwarkesh: problem that comes about in training.
Dwarkesh Patel: So you mentioned how like ChatGPT had a version that was sycophantic,
Dwarkesh Patel: and that's because users actually wanted that.
Dwarkesh Patel: Who is in control? Who decides the actual alignment of these models?
Dwarkesh Patel: Because users are saying one thing, and then they deploy it,
Dwarkesh Patel: and then it turns out that's not actually what people want.
Dwarkesh Patel: How do you kind of form consensus around this alignment or these alignment principles?
Dwarkesh: Right now, obviously, it's the labs who decided this, right?
Dwarkesh: And the safety teams of the labs.
Dwarkesh: And I guess the question you could ask is then who should decide these? Because this will be...
Dwarkesh Patel: Assuming the trajectory, yeah. So we keep going to get more powerful.
Dwarkesh: Because this will be the key modality that all of us use to get,
Dwarkesh: not only get work done, but even like, I think at some point,
Dwarkesh: a lot of people's best friends will be AIs, at least functionally in the sense
Dwarkesh: of who do they spend the most amount of time talking to. It might already be AIs.
Dwarkesh: This will be the key layer in your business that you're using to get work done
Dwarkesh: so this process of training which shapes their personality who gets to control
Dwarkesh: it I mean it will be the laughs functionally,
Dwarkesh: But maybe you mean, like, who should control it, right? I honestly don't know.
Dwarkesh: I mean, I don't know if there's a better alternative to the labs.
Dwarkesh Patel: Yeah, I would assume, like, there's some sort of social consensus,
Dwarkesh Patel: right? Similar to how we have in America, the Constitution.
Dwarkesh Patel: There's, like, this general form of consensus that gets formed around how we
Dwarkesh Patel: should treat these models as they become as powerful as we think they probably will be.
Dwarkesh: Honestly, I don't have, I don't know if anybody has a good answer about how
Dwarkesh: you do this process. I think we lucked out, we just, like, really lucked out with the Constitution.
Dwarkesh: It also wasn't a democratic process which resulted in the constitution,
Dwarkesh: even though it instituted a Republican form of government.
Dwarkesh: It was just delegates from each state. They haggled it out over the course of a few months.
Dwarkesh: Maybe that's what happens with AI. But is there some process which feels both
Dwarkesh: fair and which will result in actually a good constitution for these AIs?
Dwarkesh: It's not obvious to me that, I mean, nothing comes up to the top of my head.
Dwarkesh: Like, oh, this, you know, do rank choice voting or something.
Dwarkesh Patel: Yeah, so I was going to ask, is there any, I mean, having spoken to everyone
Dwarkesh Patel: who you've spoken to is there any alignment path which looks most promising which
Dwarkesh: Feels the.
Dwarkesh Patel: Most comforting and exciting to you
Dwarkesh: I i think alignment in the sense of you
Dwarkesh: know and eventually we'll have these super intelligent systems what do we do
Dwarkesh: about that i think the the approach that i think is most promising is less about
Dwarkesh: finding some holy grail some you know giga brain solution some equation which
Dwarkesh: solves the whole puzzle and more like one.
Dwarkesh: Having this Swiss cheese approach where, look, we kind of have gotten really good at jailbreaks.
Dwarkesh: I'm sure you've heard a lot about jailbreaks over the last few years.
Dwarkesh: It's actually much harder to jailbreak these models because,
Dwarkesh: you know, people try to whack at these things in different ways.
Dwarkesh: Model developers just like patched these obvious ways to do jailbreaks.
Dwarkesh: The model also got smarter. So it's better able to understand when somebody
Dwarkesh: is trying to jailbreak into it.
Dwarkesh: That, I think, is one approach. Another is, I think, competition.
Dwarkesh: I think the scary version of the future is where you have this dynamic where
Dwarkesh: a single model and its copies are controlling the entire economy.
Dwarkesh: When politicians want to understand what policies to pass, they're only talking
Dwarkesh: to copies of a single model.
Dwarkesh: If there's multiple different AI companies who are at the frontier,
Dwarkesh: who have competing services, and whose models can monitor each other, right?
Dwarkesh: So Claude may care about its own copies being successful in the world and it
Dwarkesh: might be able to willing to lie on their behalf, even if you ask one copy to supervise another.
Dwarkesh: I think you get some advantage from a copy of OpenAI's model monitoring a copy
Dwarkesh: of DeepSeek's model, which actually brings us back to the Constitution, right?
Dwarkesh: One of the most brilliant things in the Constitution is the system of checks and balances.
Dwarkesh: So some combination of the Swiss cheese approach to model development and training
Dwarkesh: and alignment, where you're careful, if you notice this kind of reward hacking,
Dwarkesh: you do your best to solve it.
Dwarkesh: You try to keep as much of the models thinking in human language rather than
Dwarkesh: letting it think in AI thought in this latent space thinking.
Dwarkesh: And the other part of it is just having normal market competition between these
Dwarkesh: companies so that you can use them to check each other and no one company or
Dwarkesh: no one AI is dominating the economy or advisory roles for governments.
Ryan Sean Adams: I really like this like bundle of ideas that you sort of put together in that
Ryan Sean Adams: because like, I think a lot of the, you know, AI safety conversation is always
Ryan Sean Adams: couched in terms of control.
Ryan Sean Adams: Like we have to control the thing that is the way. And I always get a little
Ryan Sean Adams: worried when I hear like terms like control.
Ryan Sean Adams: And it reminds me of a blog post I think you put out, which I'm hopeful you continue to write on.
Ryan Sean Adams: I think you said it was going to be like one of a series, which is this idea
Ryan Sean Adams: of like classical liberal AGI. And we were talking about themes like balance of power.
Ryan Sean Adams: Let's have Claude check in with ChatGPT and monitor it.
Josh Kale: When you have themes like transparency as well,
Ryan Sean Adams: That feels a bit more, you know, classically liberal coded than maybe some of
Ryan Sean Adams: the other approaches that I've heard.
Ryan Sean Adams: And you wrote this in the post, which I thought was kind of,
Ryan Sean Adams: it just sparked my interest because I'm not sure where you're going to go next
Ryan Sean Adams: with this, but you said the most likely way this happens,
Ryan Sean Adams: that is AIs have a stake in humanity's future, is if it's in the AI's best interest
Ryan Sean Adams: to operate within our existing laws and norms.
Ryan Sean Adams: You know, this whole idea that like, hey, the way to get true AI alignment is
Ryan Sean Adams: to make it easy, make it the path of least resistance for AI to basically partner with humans.
Ryan Sean Adams: It's almost this idea if the aliens
Ryan Sean Adams: landed or something, we would create treaties with the aliens, right?
Ryan Sean Adams: We would want them to adopt our norms. We would want to initiate trade with them.
Ryan Sean Adams: Our first response shouldn't be, let's try to dominate and control them.
Ryan Sean Adams: Maybe it should be, let's try to work with them. Let's try to collaborate.
Ryan Sean Adams: Let's try to open up trade.
Ryan Sean Adams: What's your idea here? And like, are you planning to write further posts about this?
Dwarkesh: Yeah, I want to. It's just such a hard topic to think about that,
Dwarkesh: you know, something always comes up.
Dwarkesh: But the fundamental point I was making is, look, in the long run,
Dwarkesh: if AIs are, you know, human labor is going to be obsolete because of these inherent
Dwarkesh: advantages that digital minds will have and robotics will eventually be solved.
Dwarkesh: So our only leverage on the future will no longer come from our labor.
Dwarkesh: It will come from our legal and economic control over the society that AIs will
Dwarkesh: be participating in, right? So, you know, AIs might make the economy explode
Dwarkesh: in the sense of grow a lot.
Dwarkesh: And for humans to benefit from that, it would have to be the case that AIs still
Dwarkesh: respect your equity in the S&P 500 companies that you bought, right?
Dwarkesh: Or for the AIs to follow your laws, which say that you can't do violence onto
Dwarkesh: humans and you got to respect humans' properties.
Josh Kale: It would have to be the case that AIs are actually bought into our
Dwarkesh: System of government, into our laws and norms. And for that to happen,
Dwarkesh: the way that likely happens is if it's just like the default path for the AIs
Dwarkesh: as they're getting smarter and they're developing their own systems of enforcement
Dwarkesh: and laws to just participate in human laws and governments.
Dwarkesh: And the metaphor I use here is right now you pay half your paycheck in taxes,
Dwarkesh: probably half of your taxes in some way just go to senior citizens, right?
Dwarkesh: Medicare and Social Security and other programs like this.
Dwarkesh: And it's not because you're in some deep moral sense aligned with senior citizens.
Dwarkesh: It's not like you're spending all your time thinking about like,
Dwarkesh: my main priority in life is to earn money for senior citizens.
Dwarkesh: It's just that you're not going to overthrow the government to get out of paying this tax. And so...
Ryan Sean Adams: Also, I happen to like my grandmother. She's fantastic. You know,
Ryan Sean Adams: it's those reasons too. But yeah.
Dwarkesh: So that's why you give money to your grandmother directly. But like,
Dwarkesh: why are you giving money to some retiree in Illinois? Yes.
Josh Kale: Yes.
Dwarkesh: Yeah, it's like, okay, you could say it's like, sometimes people,
Dwarkesh: some people are trying to that post by saying like, oh no, I like deeply care
Dwarkesh: about the system of social welfare.
Dwarkesh: I'm just like, okay, maybe you do, but I don't think like the average person
Dwarkesh: is giving away hundreds of thousands of dollars a year, tens of thousands of
Dwarkesh: dollars a year to like some random stranger they don't know,
Dwarkesh: who's like, who's not like especially in need of charity, right?
Dwarkesh: Like most senior citizens have some savings.
Dwarkesh: It's just, it's just because this is a law and you like, you give it to them
Dwarkesh: or you'll get, go to jail.
Dwarkesh: But fundamentally, if the tax was like 99%, you would, like,
Dwarkesh: you would, maybe you wouldn't overthrow the government. You'd just,
Dwarkesh: like, leave the jurisdiction.
Dwarkesh: You'd, like, emigrate somewhere. And AIs can potentially also do this,
Dwarkesh: right? There's more than one country.
Dwarkesh: They could, like, there's countries which would be more AI forward.
Dwarkesh: And it would be a bad situation to end up in where...
Dwarkesh: All this explosion in AI technology is happening in the country,
Dwarkesh: which is doing the least amount to protect humans',
Dwarkesh: rights and to provide some sort of monetary compensation to humans once their
Dwarkesh: labor is no longer valuable.
Dwarkesh: So our labor could be worth nothing, but because of how much richer the world
Dwarkesh: is after AI, you have these billions of extra researchers, workers, etc.
Dwarkesh: It could still be trivial to have individual humans have the equivalent of millions,
Dwarkesh: even billions of dollars worth of wealth. In fact, it might literally be invaluable
Dwarkesh: amounts of wealth in the following sense. So here's an interesting thought experiment.
Dwarkesh: Imagine you have this choice. You can go back to the year 1500,
Dwarkesh: but you know, of course, the year 1500 kind of sucks.
Dwarkesh: You have no antibiotics, no TV, no running water. But here's how I'll make it up to you.
Dwarkesh: I can give you any amount of money, but you can only use that amount of money in the year 1500.
Dwarkesh: And you'll go back with these sacks of gold. How much money would I have to
Dwarkesh: give you that you can use in the year 1500 to make you go back? And plausibly.
Dwarkesh Patel: The answer is
Dwarkesh: There's no amount of money you would rather have in the year 1500 than just
Dwarkesh: have a normal life today.
Dwarkesh: And we could be in a similar position with regards to the future where there's
Dwarkesh: all these different, I mean, you'll have much better health,
Dwarkesh: like physical health, mental health, longevity.
Dwarkesh: That's just like the thing we can contemplate now. But people in 1500 couldn't
Dwarkesh: contemplate the kinds of quality of life advances we would have 500 years later,
Dwarkesh: right? So anyways, this is all to say that this could be our future for humans,
Dwarkesh: even if our labor isn't worth anything.
Dwarkesh: But it does require us to have AIs that choose to participate or in some way
Dwarkesh: incentivize to participate in some system which we have leverage over.
Ryan Sean Adams: Yeah, I find this just such a fast, I'm hopeful we do some more exploration
Ryan Sean Adams: around this because I think what you're calling for is basically like,
Ryan Sean Adams: what you would be saying is invite them into our property rights system.
Ryan Sean Adams: I mean, there are some that are calling in order to control AI,
Ryan Sean Adams: they have great power, but they don't necessarily have capabilities.
Ryan Sean Adams: So we shouldn't allow AI to hold money or to have property.
Ryan Sean Adams: I think you would say, no, actually, the path forward to alignment is allow
Ryan Sean Adams: AI to have some vested interest in our property rights system and some stake
Ryan Sean Adams: in our governance, potentially, right?
Ryan Sean Adams: The ability to vote, almost like a constitution for AIs.
Ryan Sean Adams: I'm not sure how this would work, but it's a fascinating thought experiment.
Dwarkesh: I will say one thing I think this could end disastrously if we give them a stake
Dwarkesh: in their property system but we let them play,
Dwarkesh: us off each other. So if you think about, there's many cases in history where
Dwarkesh: the British, initially, the East India Trading Company was genuinely a trading
Dwarkesh: company that operated in India.
Dwarkesh: And it was able to play off, you know, it was like doing trade with different,
Dwarkesh: different, you know, provinces in India, there was no single powerful leader.
Dwarkesh: And by playing, you know, by doing trade, one of them, leveraging one of their
Dwarkesh: armies, etc., they were able to conquer the continent. Similar thing could happen to human society.
Dwarkesh: The way to avoid such an outcome at a high level is involves us playing the
Dwarkesh: AIs off each other instead, right?
Dwarkesh: So this is why I think competition is such a big part of the puzzle,
Dwarkesh: having different AIs monitor each other, having this bargaining position where
Dwarkesh: there's not just one company that's at the frontier.
Dwarkesh: Another example here is if you think about how the Spanish conquered all these
Dwarkesh: new world empires, it's actually so crazy that a couple hundred conquistaDwars
Dwarkesh: would show up and conquer a nation of 10 million people, the Incas,
Dwarkesh: Aztecs. And why were they able to do this?
Dwarkesh: Well, one of the reasons is the Spanish were able to learn from each of their
Dwarkesh: previous expeditions, whereas the Native Americans were not.
Dwarkesh: So Cortez learned from how Cuba was subjugated when he conquered the Aztecs.
Dwarkesh: Pizarro was able to learn from how Cortez conquered the Aztecs when he conquered the Incas.
Dwarkesh: The Incas didn't even know the Aztecs existed. So eventually there was this
Dwarkesh: uprising against Pizarro and Manco Inca led an insurgency where they actually
Dwarkesh: did figure out how to fight horses,
Dwarkesh: how to fight people, you know, people in armor on horses, don't fight them on
Dwarkesh: flat terrain, throw rocks down at them, et cetera.
Dwarkesh: But by this point, it was too late. If they knew this going into the battle,
Dwarkesh: the initial battle, they might've been able to fend off because,
Dwarkesh: you know, just as the conquistaDwars only arrived at a few hundred soldiers,
Dwarkesh: we're going to the age of AI with a tremendous amount of leverage.
Dwarkesh: We literally control all the stuff, right?
Dwarkesh: But we just need to lock in our advantage. We just need to be in a position
Dwarkesh: where, you know, they're not going to be able to play us off each other.
Dwarkesh: We're going to be able to learn what their weaknesses are.
Dwarkesh: And this is why I think one good idea, for example, would be that,
Dwarkesh: look, DeepSeek is a Chinese company.
Dwarkesh: It would be good if, suppose DeepSeek did something naughty,
Dwarkesh: like the kinds of experiments we're talking about right now where it hacks the
Dwarkesh: unit tests or so forth. I mean, eventually these things will really matter.
Dwarkesh: Like Xi Jinping is listening to AIs because they're so smart and they're so capable.
Dwarkesh: If China notices that their AIs are doing something bad, or they notice a failed
Dwarkesh: coup attempt, for example,
Dwarkesh: it's very important that they tell us And we tell them if we notice something
Dwarkesh: like that on our end, it would be like the Aztecs and Incas talking to each
Dwarkesh: other about like, you know, this is what happens.
Dwarkesh: This is how you fight. This is how you fight horses.
Dwarkesh: This is the kind of tactics and deals they try to make with you. Don't trust them, etc.
Dwarkesh: It would require cooperation on humans' part to have this sort of red telephone.
Dwarkesh: So during the Cold War, there was this red telephone between America and the
Dwarkesh: Soviet Union after the human missile crisis, where just to make sure there's
Dwarkesh: no misunderstandings, they're like, okay, if we think something's going on,
Dwarkesh: let's just hop on the call.
Dwarkesh: I think we should have a similar policy with respect to these kinds of initial
Dwarkesh: warning signs we'll get from AI so that we can learn from each other.
Dwarkesh Patel: Awesome. Okay, so now that we've described this artificial gender intelligence,
Dwarkesh Patel: I want to talk about how we actually get there. How do we build it?
Dwarkesh Patel: And a lot of this we've been discussing kind of takes place in this world of
Dwarkesh Patel: bits. But you have this great chapter in the book called Inputs,
Dwarkesh Patel: which discusses the physical world around us, where you can't just write a few strings of code.
Dwarkesh Patel: You actually have to go and move some dirt and you have to ship servers places
Dwarkesh Patel: and you need to power it and you need physical energy from meat space.
Dwarkesh Patel: And you kind of describe these limiting factors where we have compute,
Dwarkesh Patel: we have energy, we have data.
Dwarkesh Patel: What I'm curious to know is, do we have enough of this now? or is there a clear
Dwarkesh Patel: path to get there in order to build the AGI?
Dwarkesh Patel: Basically, what needs to happen in order for us to get to this place that you're describing?
Dwarkesh: We only have a couple more years left of this scaling,
Dwarkesh: this exponential scaling before we're hitting these inherent roadblocks of energy
Dwarkesh: and our ability to manufacture ships, which means that if scaling is going to
Dwarkesh: work to deliver us AGI, it has to work by 2028.
Dwarkesh: Otherwise, we're just left with mostly algorithmic progress,
Dwarkesh: But even within algorithmic progress, the sort of low-hanging fruit in this
Dwarkesh: deep learning paradigm is getting more and more plucked.
Dwarkesh: So then the odds per year of getting to AGI diminish a lot, right?
Dwarkesh: So there is this weird, funny thing happening right now where we either discover
Dwarkesh: AGI within the next few years,
Dwarkesh: or the yearly probability craters, and then we might be looking at decades of
Dwarkesh: further research that's required in terms of algorithms to get to AGI.
Dwarkesh: I am of the opinion that some algorithmic progress is necessarily needed because
Dwarkesh: there's no easy way to solve continual learning just by making the context length
Dwarkesh: bigger or just by doing RL.
Dwarkesh: That being said, I just think the progress so far has been so remarkable that,
Dwarkesh: you know, 2032 is very close.
Dwarkesh: My time has to be slightly longer than that, but I think it's extremely plausible
Dwarkesh: that we're going to see a broadly deployed intelligence explosion within the next 10 years.
Dwarkesh Patel: And one of these key inputs is energy, right? a lot, I actually heard it mentioned
Dwarkesh Patel: on your podcast, is the United States relative to China on this particular place
Dwarkesh Patel: of energy, where China is adding, what is the stat?
Dwarkesh Patel: I think it's one United States worth of energy every 18 months.
Dwarkesh Patel: And their plan is to go from three to eight terawatts of power versus the United
Dwarkesh Patel: States, one to two terawatts of power by 2030.
Dwarkesh Patel: So given that context of that one resource alone, is China better equipped to
Dwarkesh Patel: get to that place versus with the United States?
Dwarkesh: So right now, America has a big advantage in terms of chips.
Dwarkesh: China doesn't have the ability to manufacture leading-edge semiconductors,
Dwarkesh: and these are the chips that go into...
Dwarkesh: You need these dyes in order to have the kinds of AI chips to...
Dwarkesh: You need millions of them in order to have a frontier AI system.
Dwarkesh: Eventually, China will catch up in this arena as well, right?
Dwarkesh: Their technology will catch up. So the export controls will keep us ahead in
Dwarkesh: this category for 5, 10 years.
Dwarkesh: But if we're looking in the world where timelines are long, which is to say
Dwarkesh: that AGI isn't just right around the corner, they will have this overwhelming
Dwarkesh: energy advantage and they'll have caught up in chips.
Dwarkesh: So then the question is like, why wouldn't they win at that point?
Dwarkesh: So the longer you think we're away from AGI, the more it looks like China's game to lose.
Dwarkesh: I mean, if you look in the nitty gritty, I think it's more about having centralized
Dwarkesh: sources of power because you need to train the AI in one place.
Dwarkesh: This might be changing with RL, but it's very important to have a single site
Dwarkesh: which has a gigawatt, two gigawatts more power.
Dwarkesh: And if we ramped up natural gas, you know, you can get generators and natural
Dwarkesh: gas and maybe it's possible to do a last ditch effort, even if our overall energy
Dwarkesh: as a country is lower than China's. The question is whether we will have the
Dwarkesh: political will to do that.
Dwarkesh: I think people are sort of underestimating how much of a backlash there will be against AI.
Dwarkesh: The government needs to make proactive efforts in order to make sure that America
Dwarkesh: stays at the leading edge in AI from zoning of data centers to how copyright
Dwarkesh: is handled for data for these models.
Dwarkesh: And if we mess up, if it becomes too hard to develop in America,
Dwarkesh: I think it would genuinely be China's game to lose.
Ryan Sean Adams: And do you think this narrative is right, that whoever wins the AGI war,
Ryan Sean Adams: kind of like whoever gets to AGI first, just basically wins the 21st century? Is it that simple?
Dwarkesh: I don't think it's just a matter of training the frontier system.
Dwarkesh: I think people underestimate how important it is to have the compute available to run these systems.
Dwarkesh: Because eventually once you get to AGI, just think of it like a person.
Dwarkesh: And what matters then is how many people you have.
Dwarkesh: I mean, it actually is the main thing that matters today as well,
Dwarkesh: right? Like, why could China take over Taiwan if it wanted to?
Dwarkesh: And if it didn't have America, you know, America, it didn't think America would intervene.
Dwarkesh: But because Taiwan has 20 million people or on the order of 20 million people
Dwarkesh: and China has 1.4 billion people.
Dwarkesh: You could have a future where if China has way more compute than us,
Dwarkesh: but equivalent levels of AI, it would be like the relationship between China and Taiwan.
Dwarkesh: Their population is functionally so much higher. This just means more research,
Dwarkesh: more factories, more development, more ideas.
Dwarkesh: So this inference capacity, this capacity to deploy AIs will actually probably
Dwarkesh: be the thing that determines who wins the 21st century.
Ryan Sean Adams: So this is like the scaling law applied to, I guess, nation state geopolitics, right?
Ryan Sean Adams: And it's back to compute plus data wins.
Ryan Sean Adams: If compute plus data wins superintelligence, compute plus data also wins geopolitics.
Dwarkesh: Yep. And the thing to be worried about is that China, speaking of compute plus
Dwarkesh: data, China also has a lot more data on the real world, right?
Dwarkesh: If you've got entire megalopolises filled with factories where you're already
Dwarkesh: deploying robots and different production systems which use automation,
Dwarkesh: you have in-house this process knowledge you're building up which the AIs can
Dwarkesh: then feed on and accelerate.
Dwarkesh: That equivalent level of data we don't have in America.
Dwarkesh: So this could be a period in which those technological advantages or those advantages
Dwarkesh: in the physical world manufacturing could rapidly compound for China.
Dwarkesh: And also, I mean, their big advantage as a civilization and society,
Dwarkesh: at least in recent decades, has been that they can do big industrial projects fast and efficiently.
Dwarkesh: That's not the first thing you think of when you think of America.
Dwarkesh: And AGI is a huge industrial, high CapEx, Manhattan project, right?
Dwarkesh: And this is the kind of thing that China excels at and we don't.
Dwarkesh: So, you know, I think it's like a much tougher race than people anticipate.
Ryan Sean Adams: So what's all this going to do for the world? So once we get to the point of AGI,
Ryan Sean Adams: we've talked about GDP and your estimate is less on the Tyler Cowen kind of
Ryan Sean Adams: half a percent per year and more on, I guess, the Satya Nadella from Microsoft,
Ryan Sean Adams: what does he say, 7% to 8% once we get to AGI.
Ryan Sean Adams: What about unemployment? Does this cause mass, I guess, job loss across the
Ryan Sean Adams: economy or do people adopt?
Ryan Sean Adams: What's your take here? Yeah, what are you seeing?
Dwarkesh: Yeah, I mean, definitely will cause job loss. I think people who don't,
Dwarkesh: I think a lot of AI leaders try to gloss over that or something. And like, I mean.
Josh Kale: What do you mean?
Dwarkesh: Like, what does AGI mean if it doesn't cause job loss, right?
Dwarkesh: If it does what a human does and.
Josh Kale: It does it
Dwarkesh: Cheaper and better and faster, like why would that not cause job loss?
Dwarkesh: The positive vision here is just that it creates so much wealth,
Dwarkesh: so much abundance, that we can still give people a much better standard of living
Dwarkesh: than even the wealthiest people today, even if they themselves don't have a job.
Dwarkesh: The future I worry about is one where instead of creating some sort of UBI that
Dwarkesh: will get exponentially bigger as society gets wealthier,
Dwarkesh: we try to create these sorts of guild-like protection rackets where if the coders got unemployed,
Dwarkesh: then we're going to make these bullshit jobs just for the coders and this is
Dwarkesh: how we give them a redistribution.
Dwarkesh: Or we try to expand Medicaid for AI, but it's not allowed to procure all of
Dwarkesh: these advanced medicines and cures that AI is coming up with,
Dwarkesh: rather than just giving people, you know, maybe lump sums of money or something.
Dwarkesh: So I am worried about the future where instead of sharing this abundance and
Dwarkesh: just embracing it, we just have these protection rackets that maybe let a few
Dwarkesh: people have access to the abundance of AI.
Dwarkesh: So maybe like if you sue AI, if you sue the right company at the right time,
Dwarkesh: you'll get a trillion dollars, but everybody else is stuck with nothing.
Dwarkesh: I want to avoid that future and just be honest about what's coming and make
Dwarkesh: programs that are simple and acknowledge how fast things will change and are
Dwarkesh: forward looking rather than trying to turn what already exists into something
Dwarkesh: amenable to the displacement that AI will create.
Ryan Sean Adams: That argument reminds me of, I don't know if you read the essay recently came
Ryan Sean Adams: out called The Intelligence Curse. Did you read that?
Ryan Sean Adams: It was basically the idea of applying kind of the nation state resource curse
Ryan Sean Adams: to the idea of intelligence.
Ryan Sean Adams: So like nation states that are very high in natural resources,
Ryan Sean Adams: they just have a propensity.
Ryan Sean Adams: I mean, an example is kind of like a Middle Eastern state with lots of oil reserves, right?
Ryan Sean Adams: They have this rich source of a commodity type of abundance.
Ryan Sean Adams: They need their people less. And so they don't invest in citizens' rights.
Ryan Sean Adams: They don't invest in social programs.
Ryan Sean Adams: The authors of the intelligence curse were saying that there's a similar type
Ryan Sean Adams: of curse that could happen once intelligence gets very cheap,
Ryan Sean Adams: which is basically like the nation state doesn't need humans anymore.
Ryan Sean Adams: And those at the top, the rich, wealthy corporations, they don't need workers anymore.
Ryan Sean Adams: So we get kind of locked in this almost feudal state where, you know,
Ryan Sean Adams: everyone has the property that their grandparents had and there's no meritocracy
Ryan Sean Adams: and sort of the nation states don't reinvest in citizens.
Ryan Sean Adams: Almost some similar ideas to your idea that like, you know, that the robots
Ryan Sean Adams: might want us just, or sorry, the AIs might just want us for our meat hands
Ryan Sean Adams: because they don't have the robotics technology on a temporary basis.
Ryan Sean Adams: What do you think of this type of like future? Is this possible?
Dwarkesh: I agree that that is like definitely more of a concern given that humans will
Dwarkesh: not be directly involved in the economic output that will be generated in the CIA civilization.
Dwarkesh: The hopeful story you can tell is that a lot of these Middle Eastern resource,
Dwarkesh: you know, Dutch disease is another term that's used,
Dwarkesh: countries, the problem is that they're not democracies, so that this wealth
Dwarkesh: can just be, the system of government
Dwarkesh: just lets whoever's in power extract that wealth for themselves.
Dwarkesh: Whereas there are countries like Norway, for example, which also have abundant
Dwarkesh: resources, who are able to use those resources to have further social welfare
Dwarkesh: programs, to build sovereign wealth funds for their citizens,
Dwarkesh: to invest in their future.
Dwarkesh: We are going into, at least some countries, America included,
Dwarkesh: will go into the age of AI as a democracy.
Dwarkesh: And so we, of course, will lose our economic leverage, but the average person
Dwarkesh: still has their political leverage.
Dwarkesh: Now, over the long run, yeah, if we didn't do anything for a while,
Dwarkesh: I'm guessing the political system would also change.
Dwarkesh: So then the key is to lock in or turn our current, well, it's not just political leverage, right?
Dwarkesh: We also have property rights. So like we own a lot of stuff that AI wants, factories,
Dwarkesh: sources of data, et cetera, is to use the combination of political and economic
Dwarkesh: leverage to lock in benefits for us for the long term, but beyond our the lifespan
Dwarkesh: of our economic usefulness.
Dwarkesh: And I'm more optimistic for us than I am for these Middle Eastern countries
Dwarkesh: that started off poor and also with no democratic representation.
Ryan Sean Adams: What do you think the future of like ChachipD is going to be?
Ryan Sean Adams: If we just extrapolate maybe one version update forward to ChatGPT 5,
Ryan Sean Adams: do you think the trend line of the scaling law will essentially hold for ChatGPT 5?
Ryan Sean Adams: I mean, another way to ask that question is, do you feel like it'll feel like
Ryan Sean Adams: the difference between maybe a BlackBerry and an iPhone?
Ryan Sean Adams: Or will it feel more like the difference between, say, the iPhone 10 and the
Ryan Sean Adams: iPhone 11, which is just like incremental progress, not a big breakthrough,
Ryan Sean Adams: not an order of magnitude change? Yeah.
Dwarkesh: I think it'll be somewhere in between but I don't think it'll feel like a humongous
Dwarkesh: breakthrough even though I think it's in a remarkable pace of change because
Dwarkesh: the nature of scaling is that sometimes people talk about it as an exponential process,
Dwarkesh: Exponential usually refers to like it going like this.
Dwarkesh: So having like a sort of J curve aspect to it, where the incremental input is
Dwarkesh: leading to super linear amounts of output, in this case, intelligence and value,
Dwarkesh: where it's actually more like a sideways J.
Dwarkesh: The exponential means the exponential and the scaling laws is that you need
Dwarkesh: exponentially more inputs to get marginal increases in usefulness or loss or intelligence.
Dwarkesh: So and that's what we've been seeing, right? I think you initially see like some cool demo.
Dwarkesh: So as you mentioned, you see some cool computer use demo, which comes at the
Dwarkesh: beginning of this hyper exponential, I'm sorry, of this sort of plateauing curve.
Dwarkesh: And then it's still an incredibly powerful curve and we're still early in it.
Dwarkesh: But the next demo will be just adding on to making this existing capability
Dwarkesh: more reliable, applicable for more skills.
Dwarkesh: The other interesting incentive in this industry is that because there's so
Dwarkesh: much competition between the labs, you are incentivized to release a capability.
Dwarkesh: As soon as it's even marginally viable or marginally cool so you can raise more
Dwarkesh: funding or make more money off of it.
Dwarkesh: You're not incentivized to just like sit on it until you perfected it,
Dwarkesh: which is why I don't expect like tomorrow OpenAI will just come out with like,
Dwarkesh: we've solved continual learning, guys, and we didn't tell you about it.
Dwarkesh: We're working on it for five years.
Dwarkesh: If they had like even an inkling of a solution, they'd want to release it ASAP
Dwarkesh: so they can raise a $600 billion round and then spend more money on compute.
Dwarkesh: So yeah, I do think it'll seem marginal. But again, marginal in the context of seven years to AGI.
Dwarkesh: So zoom out long enough and a crazy amount of progress is happening.
Dwarkesh: Month to month, I think people overhype how significant any one new release is. So I guess the answer.
Dwarkesh Patel: To when we will get AGI very much depends on that scaling trend holding.
Dwarkesh Patel: Your estimate in the book for AGI was 60% chance by 2040.
Dwarkesh Patel: So I'm curious, what guess or what idea had the most influence on this estimate?
Dwarkesh Patel: What made you end up on 60% of 2040?
Dwarkesh Patel: Because a lot of timelines are much faster than that.
Dwarkesh: It's sort of reasoning about the things they currently still lack,
Dwarkesh: the capabilities they still lack, and what stands in the way.
Dwarkesh: And just generally an intuition that things often take longer to happen than
Dwarkesh: you might think. Progress tends to slow down.
Dwarkesh: Also, it's the case that, look, you might have heard the phrase that we keep
Dwarkesh: shifting the goalposts on AI, right?
Dwarkesh: So they can do the things which skeptics were saying they couldn't ever do already.
Dwarkesh: But now they say AI is still a dead end because problem X, Y,
Dwarkesh: Z, which will be solved next year.
Dwarkesh: Now, there's a way in which this is frustrating, but there's another way in which there's some,
Dwarkesh: It is the case that we didn't get to AGI, even though we have passed the Turing
Dwarkesh: test and we have models that are incredibly smart and can reason.
Dwarkesh: So it is accurate to say that, oh, we were wrong and there is some missing thing
Dwarkesh: that we need to keep identifying about what is still lacking to the path of AGI.
Dwarkesh: Like it does make sense to shift the goalposts. And I think we might discover
Dwarkesh: once continual learning is solved or once extended computer use is solved,
Dwarkesh: that there were other aspects of human intelligence, which we take for granted
Dwarkesh: in this Moravax paradox sense, but which are actually quite crucial to making
Dwarkesh: us economically valuable.
Ryan Sean Adams: Part of the reason we wanted to do this, Dwarkesh, is because we both are enjoyers
Ryan Sean Adams: of your podcast. It's just fantastic.
Ryan Sean Adams: And you talk to all of the, you know, those that are on the forefront of AI
Ryan Sean Adams: development, leading it in all sorts of ways.
Ryan Sean Adams: And one of the things I wanted to do with reading your book,
Ryan Sean Adams: and obviously I'm always asking myself when I'm listening to your podcast is
Ryan Sean Adams: like, what does Dwarkesh think personally?
Ryan Sean Adams: And I feel like I sort of got that insight maybe toward the end of your book,
Ryan Sean Adams: like, you know, in the summary section, where you think like there's a 60% probability
Ryan Sean Adams: of AGI by 2040, which puts you more in the moderate camp, right?
Ryan Sean Adams: You're not a conservative, but you're not like an accelerationist.
Ryan Sean Adams: So you're moderate there.
Ryan Sean Adams: And you also said you think more than likely AI will be net beneficial to humanity.
Ryan Sean Adams: So you're more optimist than Doomer. So we've got a moderate optimist.
Ryan Sean Adams: And you also think this, and this is very interesting, There's no going back.
Ryan Sean Adams: So you're somewhat of an AI determinist. And I think the reason you state for
Ryan Sean Adams: not, you're like, there's no going back.
Ryan Sean Adams: It struck me, there's this line in your book. It seems that the universe is
Ryan Sean Adams: structured such that throwing large amounts of compute at the right distribution of data gets you AI.
Ryan Sean Adams: And the secret is out. If the scaling picture is roughly correct,
Ryan Sean Adams: it's hard to imagine AGI not being developed this century, even if some actors
Ryan Sean Adams: hold back or are held back.
Ryan Sean Adams: That to me is an AI determinist position. Do you think that's fair?
Ryan Sean Adams: Moderate with respect to accelerationism, optimistic with respect to its potential,
Ryan Sean Adams: and also determinist, like there's nothing else we can do. We can't go backwards here.
Dwarkesh: I'm determinist in the sense that I think if AI is technologically possible, it is inevitable.
Dwarkesh: I think sometimes people are optimistic about this idea that we as a world will sort of,
Dwarkesh: I collectively decide not to build AI. And I just don't think that's a plausible outcome.
Dwarkesh: The local incentives for any actor to build AI are so high that it will happen.
Dwarkesh: But I'm also an optimist in the sense that, look, I'm not naive.
Dwarkesh: I've listed out all the way, like what happened to the Aztecs and Incas was
Dwarkesh: terrible. And I've explained how that could be similar to what AIs could do
Dwarkesh: to us and what we need to do to avoid that outcome.
Dwarkesh: But I am optimistic in the sense that the world of the future fundamentally
Dwarkesh: will have so much abundance that there's all these,
Dwarkesh: that alone is a prima facie reason to think that there must be some way of cooperating
Dwarkesh: that is mutually beneficial.
Dwarkesh: If we're going to be thousands, millions of times wealthier,
Dwarkesh: is there really no way that humans are better off or can we can find a way for
Dwarkesh: humans to become better off as a result of this transformation?
Dwarkesh: So yeah, I think you've put your finger on it.
Ryan Sean Adams: So this scaling book, of course, goes through the history of AI scaling.
Ryan Sean Adams: I think everyone should should pick it up to get the full chronology,
Ryan Sean Adams: but also sort of captures where we are in the midst of this story is like, we're not done yet.
Ryan Sean Adams: And I'm wondering how you feel at this moment of time.
Ryan Sean Adams: So I don't know if we're halfway through, if we're a quarter way through,
Ryan Sean Adams: if we're one tenth of the way through, but we're certainly not finished the path to AI scaling.
Ryan Sean Adams: How do you feel like in this moment in 2025?
Ryan Sean Adams: I mean, is all of this terrifying? Is it exciting?
Ryan Sean Adams: Is it exhilarating?
Ryan Sean Adams: What's the emotion that you feel?
Dwarkesh: Maybe I feel a little sort of hurried. I personally feel like there's a lot
Dwarkesh: of things I want to do in the meantime,
Dwarkesh: including what my mission is with the podcast, which is to, and I know it's
Dwarkesh: your mission as well, is to improve the discourse around these topics,
Dwarkesh: to not necessarily push for a specific agenda, but make sure that when people are making decisions,
Dwarkesh: they're as well-informed as possible, They have as much strategic awareness
Dwarkesh: and depth of understanding around how AI works, what it could do in the future as possible.
Dwarkesh: And, but in many ways, I feel like I still haven't emotionally priced in the future I'm expecting.
Dwarkesh: In this one very basic sense, I think that there's a very good chance that I
Dwarkesh: live beyond 200 years of age.
Dwarkesh: I have not changed anything about my life with regards to that knowledge, right?
Dwarkesh: I'm not like, when I'm picking partners, I'm not like, oh, this is the person,
Dwarkesh: now that I think I'm going to live for 200, you know, like hundreds of years.
Ryan Sean Adams: Yeah.
Dwarkesh: Well, you know, ideally I would pick a partner that would, ideally you pick
Dwarkesh: somebody who would be, that would be true regardless.
Dwarkesh: But you see what I'm saying, right? There's like, the fact that I expect my
Dwarkesh: personal life, the world around me, the lives of the people I care about,
Dwarkesh: humanity in general to be so different has, it just like doesn't emotionally resonate as much as,
Dwarkesh: I, my intellectual thoughts and my emotional landscape aren't in the same place.
Dwarkesh: I wonder if it's similar for you guys.
Ryan Sean Adams: Yeah, I totally agree. I don't think I've priced that in. Also,
Ryan Sean Adams: there's like non-zero chance that Eliezer Yudkowsky is right, Dworkesh.
Ryan Sean Adams: Do you know? And so that scenario, I just, I can't bring myself to emotionally price in.
Ryan Sean Adams: So I veer towards the optimism side as well.
Ryan Sean Adams: Dworkesh, this has been fantastic. Thank you so much for all you do on the podcast.
Ryan Sean Adams: I have to ask a question for our crypto audience as well, which is,
Ryan Sean Adams: when are you going to do a crypto podcast on Dwarkech?
Dwarkesh: I already did. It was with one Sam Bigman-Fried.
Ryan Sean Adams: Oh my God.
Dwarkesh: Oh man.
Ryan Sean Adams: We got to get you a new guest. We got to get you someone else to revisit the top best.
Dwarkesh: Don't look that one up. It's Ben Omen. Don't look that one up.
Dwarkesh: I think in retrospect. You know what? We'll do another one.
Ryan Sean Adams: Fantastic. I'll ask you
Dwarkesh: Guys for some recommendations. That'd be great. Dwarkech, thank you so much.
Dwarkesh: But I've been following your stuff for a while, for I think many years.
Dwarkesh: So it's great to finally meet. and this was a lot of fun.
Ryan Sean Adams: Appreciate it. It was great. Thanks a lot.