Subscribe
Share
Share
Embed
A podcast from Reserve about the AI revolution: the technology, the power, and the future it's pulling us toward. Reserve co-founder and CEO Nevin Freeman explores the AI boom from two angles: how to invest in it and own your share of the buildout, and where AI may really take us in the long term. But it's really about investing our way out of the permanent underclass.
Thomas: [00:00:00] The main thing I want people to take away is that AI is gonna be a much bigger deal than I think almost everyone thinks. Uh, I think the current market has been proven wrong time and time again for the last five years in being, uh, not bullish enough on AI, and I'm very... and I think that's very related to the fact that our governance of AI is, like, not pricing in the full effects whatsoever.
Hmm. So I think I overall want sort of, um, discussion as a whole to, like, take the impacts of, like, extremely tr- societally transformative AI much more seriously, which is why I think it's, like, good for people to be aware that this is what's coming. On the other hand, I am-- I have some reservations about people aggressively accelerating the AI build-out.
In particular, I think that it's, like, better if we can slow things down somewhat. And so s- it's sort of, like, an awkward position where I want people to be aware of stuff more, but I also don't want there to be, like, massive amount of cash flowing into the AI infrastructure build-out, despite it being a good investment to do so, [00:01:00] obviously.
Romeo: If you fully automate coding, then how long will it take to automate other parts of the research process? Mm-hmm. And how hard will it be to cross qualitatively through the human range? Like, these, these are some of the other things that, um, we've been trying to, to model and think about. And yeah, it do- it does just seem, um, like there's a high probability of, you know, in the next, I would say, like, in the next decade where, you know, we're not extremely confident that we can get to this, like, strong AGI or super intelligence milestones.
But I think off the top of my head, and we ha- we have our numbers out there, um, but, like, seventy, eighty percent probability, I think, that within the decade, um, maybe more eighty percent, um, for, for most of us on the team.
Nevin: So to those out there who are worrying that this might be a bubble, you guys' answer would be absolutely not.
That is, that is the least of our challenges.
Thomas: Uh, I would say [00:02:00] it's-- I would say there's some small chance it's a bubble. Yeah. Like, there's some v- there's a, there's a possibility, which is that-- which is, um, like, I'm right about my theoretical claim that you could have AIs that are better than humans, but I'm wrong that LLMs are, like, a meaningful step towards that.
Right. And so there's a, there's a, there's a chance, I think it's a s- it's not that likely, I think, but I think there is a chance that the LLMs basically peter out. Yeah. And then, like-- and, like, this current data center build-out was, you know, a bad investment and the returns are, are lower than people would have thought.
I think that on average, that's not gonna be the case. We're gonna get intelligence explosion and it's gonna go totally crazy.
Nevin: Thomas, Romeo, thank you for joining us for episode two of Buying Into the Singularity. Uh, for our viewers, uh, Thomas Larsen and Romeo Dean are both researchers at the AI Futures Project, uh, which published, you may have heard of AI 2027, a scenario that they went live with, what, like a year ago or [00:03:00] so at this point?
Thomas: Yeah, last April.
Nevin: Yep. And then they followed it up with, uh, uh, a, a new document called Plan A, uh, or AI 2040, which is, uh, different from 2027. It's, uh, their view on what they think should happen rather than just a prediction of what they think is perhaps most likely to occur. So thank you guys for joining us.
Super excited to dive in and talk about this. Let's start with, um, let's start with a walkthrough of Plan A for anyone who hasn't read it yet, uh, to give, you know, listeners and viewers a little bit of context
Thomas: Yeah. Awesome. Thanks for having us. Um-
Romeo: Yeah. Thank you.
Thomas: So basically, our idea with Plan A is that sort of on this default trajectory of AI that we think is, is gonna happen over the next few years, um, there are gonna be a huge number of big risks, uh, that society will face.
Um, perhaps most importantly, the issue of AI loss of control. Um, but there are many, many [00:04:00] others, um, like concentration of power, like massive geopolitical upheaval. Um, and in AI 2027, our, our, our sort of, uh, predictive scenario, we saw like a bunch of those actually happening and a bunch of bad stuff occurred.
Uh, and so, AI 2040 was basically our vision for how we can sort of minimize those risks as much as possible while still, like, actually achieving a bunch of the upsides.
Nevin: Mm-hmm.
Thomas: Um, there's sort of a few key points of AI 2040. The main one is, uh, basically extending the period of time in which we have roughly human-level AI, but not wildly superintelligent AI, so that we can sort of make as much progress as fast as possible on the core issues that we need to solve before we build superintelligence.
So the reason why it's called AI 2040 is because we delay superintelligence until 2040 and sort of get this big window in which to make progress. What are the pro-- What are the actual problems we sort of need to make progress on? [00:05:00] So there's, um, alignment, um, there's concentration of power, there's AIs taking everyone's jobs, um, and, and a bunch of others.
Our sort of core policy recommendations for actually sort of achieving these goals. So there's this sort of buying time, which mostly happens via, um, international cooperation to develop AI in a, uh, basically s- in s- a way that's more safe and much more transparent and much more open than the current status quo.
Mm-hmm. Um, and we'll-- I'm sure we'll get into the details on, on the exact mechanics of that later. Um, there's this, uh, diffusing AI broadly throughout society, uh, component, which is, uh, we don't want there to be a big gap between w- the AIs that are being used inside the AI companies specifically or for, for recursive self-improvement and the AIs sort of accessible to everyone broadly.
Because most of the risks, most of the most important risks that, that we see [00:06:00] happen from specifically the recursive self-improvement intelligence explosion that could happen inside a single AI company. Mm-hmm. And we think that the risks are much lower from AIs that are sort of broadly deployed in any application besides recursive self-improvement.
And we think there are a huge number of upsides there. In particular, people externally working on things like, uh, you know, making safety progress or, uh generically making the world a better place in a bunch of ways, like with healthcare or with like, you know, AI lawyers or, or whatever, as well as just generic economic growth.
Um, and so in our scenario, AI is very, very broadly diffused. It gets to sort of this like human-level AI, like AGI, um, and then causes this sort of massive societal transformation where, uh, many of the world's problems get solved. GDP growth is humongous. It's like, you know, we get s- you get like GDP doublings in- over the course of a year.
Um, and, uh, so basically you get this massive societal [00:07:00] transformation that's hopefully steered to basically solve a lot of the world's problems and w- in the scenario that we wrote, um, we sort of do get into a bunch of, a bunch of sort of the generic existing problems in the world that get solved. Um- And then sort of the final, maybe main, uh, policy component of our plan is sort of this issue of, um, we have all of this, uh, regulation in place to, um, mitigate the issues and sort of make sure the intelligence explosion happens at a safe pace.
Um, what if those sort of break down? What if, uh, you know, what if a new president comes in and is like, "Everything my predecessor did was terrible- Mm-hmm. "... and I want to sort of, uh, do the reverse of everything that he says." How do we make sure that our, our sort of plan fails as gracefully as possible and doesn't result in, uh, sort of like an immediate intelligence explosion that would happen much faster than it would have before- Mm.
because there was, uh, more compute in the world because of, uh, because there was more time [00:08:00] to build more compute? Um, and our answer there, basically this thing called mutually assured compute destruction, which is maybe the most controversial part of our plan, which is, uh, make sure that if our regime breaks down, uh, the amount of compute in the world gets reduced, and everyone's incentivized to deliberately destroy their own compute in order to reduce the speed of the resulting intelligence explosion to make the sort of failure case, uh, sort of better than
at least as good as the status quo as if, if we hadn't done this at all. Um, so there's a lot more to get into, but that's maybe the very, very high level overall picture. We've sort of... We have both this overall plan, Plan A, and then we have this ex- specific scenario where we sort of describe it being implemented as concretely as we possibly can.
Mm-hmm. Um, of course, there's a bunch of possible implementations of it. There's a bunch of possible empirical uncertainties that we have about the, you know, how the actual implementation might go. Yeah. So we, we sort of outlined our best guess, but there's like a lot more possibilities to [00:09:00] sort of get into and describe.
Nevin: For sure. And something I really appreciate about the way you guys have gone about both your initial scenario, uh, write-up and the scenario around Plan A is, you know, just boldly telling one storyline, right? Because obviously, if you, if you specifically hone in on any given storyline, you're gonna be wrong about a bunch of stuff.
You're making forecasts about these sort of crazy dynamics and so on. Uh, but it seems like a fantastic way to start conversations like this and kind of actually help people think through the, the, the dynamics and, uh, wrap their heads around those pieces. So let's walk through the scenario on a story level.
Um, and, uh... So yeah, so let's kind of just start with setting the stage of where does the story begin and what are kind of the first developments.
Romeo: The method is also... It was helpful for us as well, like as we set out to try and figure out- What to, what to do about [00:10:00] AI, what our Plan A should be. I think there are a lot of points where we were surprised ourselves or, or if you had told us some aspects of our plan back in time when we started writing, we would've been surprised by them.
Um, I think some of them are counterintuitive, but it was part of this process of, like, running into problems as we were-
Nevin: Yeah ...
Romeo: trying to sketch them out in concrete detail, and then realizing there were things that we hadn't thought of and, and, and the solutions that we came up with, um, and dynamics that we realized were at play led, led to a lot of the key, uh, policy ideas, I would say.
Um, but yeah, talking through the scenario as a story, I- maybe we could do, like, a back and forth.
Thomas: Yeah. Do you wanna start with, like, the first section?
Romeo: Yeah, sure. So, um, the early years, 2027 and 2028, are kind of the part of the scenario that have the, the lowest amount of recommendations, um, [00:11:00] and, and a bit more, they're a bit more on the predictive side.
Yep. But we wanted to just work in some of the, like, minimal things that we thought, um, would be helpful for setting up kind of, like, the optionality to do plan, something like Plan A, um, if the need arised. So there's, you know, the AI companies are, continue their massive revenue growth trends, and now they're, you know, starting to get towards being some of the bigg- some of the biggest companies in the world by 2028.
Um, their, the kinda data center capital expenditures keep growing. They're bigger than the, the US military budget already. I think the crossover point's in, like, 2027. And then- Part of, uh, part of all of this growth is leading to AI becoming, like, the biggest election, uh, election topic or biggest issue in the- Mm-hmm
in the 2028 election.
Nevin: Okay.
Romeo: Um, and- And
Nevin: you guys see that [00:12:00] as kind of status quo. Apart from, apart from any, uh, wishes you might have, would you say you think it is gonna be the number one topic when it comes to the 2028 election?
Romeo: I'm, I'm not so sure. I think it'll be a, a bigger topic than it, than it, than it has been in the past.
Nevin: Definitely,
Romeo: yeah. I think I... Yeah, I'm not confident it'll be, like, the biggest issue. Um, but I think there's a pretty good chance. I'm not sure if you agree.
Thomas: Yeah, I think, I think it's, it's always hard to tell with these things. Um, if AI continues sort of on the current trajectory and becomes, uh, like, sort of grows in the way that it did in AI 2027, which it might, uh, then I think, well, there might not even be a 2028 election.
But- Sure ... if there is, then, uh, good chance, I think, AI is at the forefront, especially if we see stuff like massive AI-driven job loss in time. In this scenario, we don't quite see that. Um, it's, like, a bit of a slower timeline than the AI 2027 timeline, but it's, [00:13:00] like, big enough that the effects are sort of rippling out throughout society sufficiently that the voters are really concerned.
I think there's possibility that that does- that doesn't happen, but it's, like, one of the, one of the elements of sort of this specific scenario.
Nevin: Okay.
Romeo: Yeah. Yeah.
Nevin: Okay. So yeah. So beginning from, you know, this first two-year phase, uh, heading into the 2028 election, you guys are proposing that there would be a bill passed by Congress that would start to have some teeth and start to shape how things play out.
Thomas: Yeah.
Nevin: Talk me through the basics of that bill.
Thomas: Yeah. So I think the first thing I wanna say is our actual recommendation is that we should do Plan A, like, the whole thing with the international deal and the sort of, like, all of the verification as soon as possible.
Nevin: Okay.
Thomas: Like, I think if we could basically start on that today, that would be the best-case scenario.
And then sort of what we have in the scenario is, like, a middle ground, which involves a much more sort of moderate form of policy happening, um, uh, in the first few years. Um, so, uh, yeah, what happens in that [00:14:00] bill? Um- In our scenario, we're, we're basically saying that some sort of giant omnibus AI bill, um, happens, uh, because there's a lot of political pressure for Congress to do something, and that most of the stuff in that bill is, uh, sort of r- from our perspective, random policy that doesn't really help address the core problems.
Mm-hmm. But, you know, we are able to at least get some components of, of policy that help with plan A, uh, started. Um, the main components, uh, are one, um, some of the verification stuff, basically, um, means to accelerate, uh, uh, some of the verification R&D that Romeo was talking about, as well as the just, like, directing parts of the national security apparatus to do some of the chip, chip tracking.
Nevin: Okay.
Thomas: And then an- another key part is this transparency component, which is basically creating mechanisms for the public and the government to see what's going on inside AI companies. We think this is, this is gonna become very, very important later. Um, [00:15:00] but even sort of baseline, basic, very, very basic transparency measures seem very, very important in the near term for having sort of governments and the public see Uh, when the intelligence explosion is starting and seeing those early warning signs, um, as opposed to just waiting for the effects of the AI to be seen on the broader world.
Um, so an example is sort of this, this Mythos case. The, the Anthropic, um, you know, produced this model Mythos, which had a, a bunch of cyber capabilities which were very concerning to the national security apparatus. Um, Anthropic just sort of decided to voluntarily do this, um, you know, this basically staged release approach where they first released it to a bunch of trusted providers and then used it to harden their systems before doing a broader public release.
Basically, we think that it would be better if, as soon as possible, the broader public and the government were able to see exactly [00:16:00] what the capabilities and understand exactly what those capabilities look like so that we don't have to rely on AI companies doing sort of like a responsible strategy, particularly once we get to AI capabilities that are much more consequential than cyber.
Cy- like these cyber capabilities are already pretty consequential, but once you get AIs that can, you know, um, do sort of, you know, build bioweapons that could v- you know, cause a, you know, total civilizational collapse or, you know, even m- more worrying, do recursive self-improvement and build a full-on superintelligence.
Mm-hmm. Um, I think once you're sort of at, you know, AIs that are sort of having those amounts of risk, uh, it is very, very important that we see what's going on as fast as possible so that the appropriate response can be taken, um, before it's too late.
Nevin: And so what would that feel like in practice? If you're, y- let's say, kind of from the perspective of inside of one of these labs or from the perspective of like a random outsider who wants to see in, what is it that they are required [00:17:00] to disclose?
'Cause I mean- Yeah ... do- you-- I, I miss-- I don't think you mean that like the model is trained and immediately everyone has access to the sort of un- un- you know, unbridled version of it. So what, what do you mean really?
Thomas: Yeah. So there's a few components. Um, so one is, uh, so w- we want the more extreme version, which we'll get into, uh, later.
Um, the very, very basic version is it'd be nice if, um, there were at least some people within government and potentially some third-party risk assessors that had the full access to the information that's going on inside the lab as soon as possible. So you have, you know, folks like, uh, you know, maybe folks like CAISI, maybe folks like, which is a, um, the Center for AI Standards and Innovation, uh, which is I think maybe the best body of technical experts about sort of AI, uh, and AI safety in the government right now.
Um, as well as external folks like METR, um, uh, and Apollo and, and various others. Um, having those folks get a- access to the internal, um, what's going on [00:18:00] inside the AI companies, and then have the ability to write public reports- Um, as well as private reports to, to particular people who, who, who matter a lot, um, about sort of the state of play inside the AI companies.
What's the ... Like, how much risk of various sorts, both including near-term stuff as well as, uh, more existential stuff. Mm-hmm. Um, and then basically just, like, raising the alarm if there's a big concern. That's the very, very simple baseline version, and then we w- well, there's a lot more that, that we want. Um, uh, but I think that would, like, be much better than the sort of current status quo, where a little bit of that is happening, but only on a very voluntary basis.
Um, and I think the fact that it's a voluntary basis allows AI companies to pressure the people who they allow to give access to write very f- like, more favorable things than they would have otherwise. Mm. Um, and I think basically not having that line of pressure would, would on average make the, uh, information disclosed more trustworthy.
Nevin: Demis Hassabis, [00:19:00] um, founder and CEO of DeepMind, now Google DeepMind, recently proposed, um, something that sounds a little bit like this. Uh, proposed a, an, a regulatory body in the US akin to FINRA, which for those who don't know, I think it's the financial, uh, industry regulatory authority. Um, the, the way that FINRA works is, you know, it's, it's sort of paid for, uh, by the industry participants who have to register with it and pay dues, and then they have to follow all sorts of processes, and it's overseen by, by the government, the regulators.
Um, and he proposed that this body would basically do something like what you're saying, like get access, I think he said a pretty short period of time, like 30 days before something is publicly released, to do that kind of analysis. Do you think that his proposal would basically fill the gap that you're talking about if that were adopted?
Or do you think it misses sort of the key piece of, of how this would need to work?
Thomas: Yeah. [00:20:00] So I think, I think I'm tentatively in favor of his proposal. I think it seems like a good on-the-margin thing. Um, one, one thing that's very important to note is that I think we're more concerned with the internal deployments of the AI system than the external deployments.
Mm-hmm. And so anything that's framed as, or, or sort of tethered to, like, something like 30 days before model release, uh, seems worse than as soon as it's internally deployed, like internally used by the company itself. Mm-hmm. Um, because the internal deployment is where so much of the risk comes from, particularly the, like, intelligence explosion.
Walk us through why
Nevin: that is.
Thomas: Yeah. So in AI 2027, what happened was, uh- An AI company, we call it OpenBrain, um, built AIs that were sufficiently capable that they could automate the, uh, process of building new AIs. Um, and that happened in, you know, [00:21:00] February, March, depending exactly how you count, 2027. Um, then that caused this lab to speed up their AI research capabilities a huge amount.
Um, and at the initial point where you've got AIs that are about as good as humans at doing the whole AI R&D process, you know, the speed up is moderate. It's maybe, like, 3X or 5X, 'cause the AIs are faster and cheaper, but they're not, you know, qualitatively superhuman.
Romeo: Mm-hmm.
Thomas: But that amount of speed up lets you relatively quickly do new training runs, do...
You know, develop new AI training techniques that build AIs that are, um, you know, qualitatively superhuman, much smarter than humans. And those AIs can then come up with even better techniques in sort of a, a process called recursive self-improvement. Recursive because the AIs are, uh, building new AIs, which then build new AIs, and so on.
Um, there are very complicated questions about the exact dynamics of this, and exactly how long it will take, and exactly how smart the AIs will end up being, um, after it. But [00:22:00] my view is that it's pretty likely that, uh, this process happens over the course of a few months to a few years, um, basically between, like, you can automate humans and you have, uh, qualitatively superhuman general AIs that are, like, vastly, vastly, vastly smarter than humans, and can do, uh, like qu- like qualitatively crazy and, like, world transformative, uh, things.
Um- In that case, uh, those AIs obviously pose a huge amount of risk. Um, in particular, I'm very worried about those AIs, uh, taking over. Um, the sort of the classic, like, AI loss of control, AI takeover story, um, if those AIs have sort of misaligned values or values that are not, uh, aligned with general human interests.
But I think even if they are sort of kept under control and those AIs are, uh, aligned with, you know, their creators or hu- humanity more broadly, there are [00:23:00] still huge issues. In particular, um, there's this issue of, uh, well, who are they aligned to? The default scenario, I think, is either they're aligned to the company that builds them or the government that has the most say in their development.
Mm-hmm. Either case, that's very, very scary because it's a insane amount of concentration of power where the specific CEO who's successfully controlled them or the specific president or, you know, a government that's controlled them has this huge amount of power where I think that they can use those AIs to basically get their way and have a ginormous military, political, economic advantage over everyone else, um, and sort of coerce everyone else to, to do what they want.
Um, and so I think that's very, very scary. I think, like, permanent AI-enabled totalitarianism is a, is a real risk from that. Um, and then a third risk, I think, which is maybe not talked about enough, is this risk of massive geopolitical upheaval and conflict during this process, where if it's [00:24:00] the case that, uh, other AI companies or other countries see that this recursive self-improvement process is happening- Mm
and that whoever's doing it is going to get a massive advantage over everyone else, then those countries who currently have a lot of power, they've got nuclear weapons, they've got a lot of economic leverage, are very incentivized. I think they're in what's called, like, a Thucydides Trap where they're losing power because, you know, the country that's building AIs is gaining a lot of power.
Mm. Everyone else is relatively losing power and therefore is incentivized for there to be conflict or negotiation as early as possible while they still are empowered. And so I think you're in this position which is very, very prone to conflict basically, where I'm very, very worried that, uh, given that the US is in the lead right now, other countries that aren't the US are gonna notice that the US is gonna run away with things basically and are incentivized to, uh, have there be a conflict.
And I think, uh, you know, I hope that we don't get World War III, but I think it's a real [00:25:00] possibility- Right ... that we get World War III at the same time as we get all of these other risks during the intelligence explosion.
Nevin: And that, that could happen even if these other risks turn out to, you know, for some lucky reason, um, be, uh, you know, be something that we don't have to deal with.
If everyone believes we have to deal with them, we may have the conflict just the same. That's a good point. Um, and okay. So you can s- I can start to see, um, the viewers can start to see how from that perspective- You care quite a lot about understanding a new model release before it's used internally for AI R&D.
Thomas: Yep.
Nevin: Um, perhaps much more than you care about assessing it before it's used externally if the thing you're most worried about governing is the use of models for AI R&D. Exactly. Is that the point?
Thomas: Yep.
Romeo: It's, it's also quite consistent with this whole scenario that the company might be able to just either not release models for a lot of this period or just release models that they know are gonna meet the regulatory [00:26:00] requirements.
And- Right ... they can make a bunch of revenue, but then the models they're using internally are the smarter, riskier models, and there's this big gap. And, and-
Nevin: Yeah. And it seems like, um, this is no longer hypothetical, just with the Mythos fable situation, and, you know, I think other frontier labs are around the same point right now.
We can already see, like, well, we just don't have access to the most powerful version of the tool already. That's just already a reality, and we just went over that threshold a tiny bit. Well, obviously, if more and more powerful and th- and hence more and more dangerous things are developed, then, you know, it, it seems unlikely that they would be released.
Um, but then, yeah, there's currently no ... W- I mean, actually, maybe there is. Do you guys have an understanding of, like, if you're an employee within, um, a frontier lab, what are the limitations on what you can do with the frontier unreleased versions of the model? Do you know? Are, are, are there sort of governing bodies or principles in place inside of those [00:27:00] companies?
Thomas: My understanding, which, uh, it shouldn't be ... I'm not fully confident in, but my understanding is that, um, fa- that Mythos was internally deployed in Anthropic basically right after it was trained, uh, in February of this year, um, which was several months before it was released- You mean, like- ... to the broader public
Nevin: anyone can, you know, go into the local, you know, s- use com- Claude Code on the command line and just do whatever they want with it? Or like a small group of people is allowed to study it?
Thomas: Uh, no. Uh, it was broadly, like, broadly available for use within Anthropic.
Nevin: Okay.
Thomas: It's unclear to me to what extent there were, uh, like, misuse safeguards on the model.
It's possible that it was sort of what I would call, like, a helpful-only variant or, like, a version of the AI which has no refusals and would just do whatever it's asked. Um, of course, they never release helpful-only models to the public because, uh, they want to sort of guard against misuse of the AIs, and so the AIs do often refuse requests which they perceive as harmful.[00:28:00]
Um- It's possible that they d- deploy, that they, you know, only do the internal deployment with safeguards. I would guess that there's a lot of internal, internal deployment use at the AI companies without the safeguards, um, because people are just so desperate to get as much uplift as possible from the A- AIs that they really just want as capable an AI as possible as soon as possible and- That's
Nevin: what I would want if I was- Yeah
if I was there doing it.
Thomas: Yeah. Yeah.
Nevin: Okay. Okay. Got it. So, so then basically this initial transparency machinery that you envision, um, coming from this initial bill or, you know, or, or wherever it would come from, uh, is in part, in, in, in large part it's there to solve this problem, right? It's, it's like, "Hey, you know, let us evaluate any of this frontier stuff before you do anything internally that could be risky with it."
Thomas: Yep, that's right. And then there's a version of it that's only government-facing, but our preferred version of it is public-facing because we don't just want it to [00:29:00] be some lab people and some government people kept in the loop about what's going on. Mm-hmm. We'd like it to be as, um, as public-facing as possible so that the broader public, um, like the broader scientific community can weigh in on these questions of like is this recursive self-improvement actually gonna lead to crazy super intelligence or not?
Nevin: Okay.
Thomas: Um, and that it's very important for preventing Both, like, the, sort of the government being able to see what's going on helps a lot with the lab running away with everything, but it doesn't help a lot obviously with the government running away with super intelligence. Yep. Um, so that's why we really want the public, um, to know, 'cause then the co- public can help somewhat act as a check on what the government is doing.
Nevin: Okay. Great. Okay. So now let's kind of look at the next phase in the story. What ha- what are the big pieces that happen next?
Thomas: Yeah. So, um, there's sort of, um... So yeah, the first section was, was now to 2028. The next [00:30:00] section is, uh, sort of starting in 2029. So there's a new president that gets inaugurated in 2029, and in our scenario we say that they have, um, basically a bunch of options in front of them for how to navigate this upcoming intelligence explosion, which the transparency mechanisms are telling them and the, the broader public is telling them, like, "Yeah, the intelligence explosion is gonna happen and it's gonna happen in the next few years."
In our scenario, it would've happened by default in 2030, and this president is getting inaugurated in 2029, beginning of 2029, and so has to set... has to decide basically, are, what are we gonna do here? Um, and we lay out a series of options. Um, our favorite option being, of
course, Plan A, which is what the president goes for. Um, and the core thing that happens in Plan A is there is international coordination between the US, China, and several other countries, most importantly the semiconductor supply chain countries- [00:31:00] Mm-hmm ... um, to, uh, enact some AI regulations. In particular, um, this thing we call total research transparency, which is that all AI research that is being done on the major compute clusters is Broadly available to the public, and people can see basically the exact algorithms that are happening on the giant research clusters in, in all of the countries, as well as limits on the pace of particularly algorithmic progress to sort of slow down-
Nevin: So hold up.
So you're saying not just open weights, but open, uh, open source for the, the, the source code itself, for the, the code that trains the models and the, and the, and the code that runs everything?
Thomas: Yeah. So our proposal is very similar... Or, yeah, do you want to jump in, Romeo?
Romeo: I was just gonna say, it's actually, it's actually the, the kind of the opposite of what open source means now in terms of what's shared and what's not.
Right, right. Like, we actually, uh, are proposing closed [00:32:00] weights, um, at, in the mainline scenario. Um, we do outline different transparency options. Um, and we do think that, you know, you could go even, even further on the open side than what we said and also have open weights, but there are maybe more risks than benefits to that.
Nevin: And what is, um... And what's the point of that? So why, why do you want all of that code to be public? And, and what do you get from that if the weights are not public?
Thomas: Yeah. So the core thing that we want to do is we want there to be a safety case-based regime for frontier AI training, where, um, by default there would've been this crazy intelligence explosion, and probably would've built very, very unsafe models which might have then had gone and taken over the world or done other bad things.
Um, uh, what we want there to be is for all people who are developing AI, um, to basically make sure that all AI development that they're doing is under, [00:33:00] is like, you know, taking into acc- is like having... Is like reasonably safe, is not overly risky. Um- Doing that's very, very complicated to implement. It requires a bunch of technical trade-offs like, uh, you know, do I change my architecture to...
You know, current architectures have this very fa- nice property called chain of thought, uh, where, um, which basically helps, helps them be a lot more safe and monitorable and controllable. You could imagine architectures that don't have this property that are sort of have directly pass activations between their layers, and that would, I think, make the AIs somewhat less safe.
But you could imagine there being a big, um, sort of performance boost from this type of thing.
Nevin: And just to break that down for someone who hasn't heard that term, I'm understanding you to mean, you know, current chain of thought is like the, the ... a text LLM is producing a bunch of words in order to sort of reason through and then sort of re-prompt itself and reason through over and over again, and that's [00:34:00] happening in some human language that we all can read.
Um, and so it's easy to inspect. But you could imagine a version where the model instead produces some complicated, you know- Arrays ... in- inscrutable vector or whatever, and it's just doing this reasoning in a way that we can't make sense of what's happening in the reasoning, kind of in the way that it's hard to make sense of what happens in a single pass inside the model or whatever.
Yeah. And so then that would, you know, maybe that could make it faster or, you know, give it more sort of, uh, internal context or whatever, but it might be just, like, much harder to understand what it's doing. Is that- That's right ... a good summary?
Thomas: That's a, a very good summary. Okay. Um, so basically there'll be all sorts of technical decisions like that, um, during AI development.
Um, and then, uh, w- what we want it to be the case is that overall these decisions are being made s- in a way that allows overall risk to be relatively low, which I think will involve, will need to involve some amount of c- coordination [00:35:00] on doing something that isn't the maximally fastest in capabilities and maximum revenue path to more AI.
And so will involve trade-offs that, you know, you want to take the safe- the safety side as opposed to the capability side. And so what the transparency enables Is if everyone doing frontier AI development knows that they can see everything that's going on on their competitors, then it helps a lot with basically the ability to coordinate to do the safe things and not the unsafe things.
Mm-hmm. And so what we want is this sort of regime where you've got all of the major AI companies and governments that have major compute clusters to be bought in to know that their tr- their compute cluster is verifiably transparent, and so is everyone else's. And now everyone is doing the safe things that they've all agreed on and not the super unsafe things, which entails a relatively large slowdown.
[00:36:00] Um, doing a intelligence explosion as fast as possible would obviously incur a lot of risk and be very unsafe. And so there's, there's sort of gonna be this process, and that's what we sort of describe throughout the rest of the scenario. But this regime is set up over 2029 and 2030, and that's sort of the core thing that allows the rest of the scenario to sort of proceed in a reasonably safe manner.
Nevin: This, this approach was one of the biggest surprises for me in what you guys landed on. Um, you know, historically, the, the sort of safety-oriented community has had the general, um, you know, the general opinion of, like Don't, you know, sort of block all communication, you know, between researchers. Don't, don't propagate ideas.
Uh, you know, sort of max secrecy is maybe better because then new innovations don't get adopted elsewhere and the whole thing goes more slowly. Um, uh, but here, you know, I... It sort of seems like, [00:37:00] well, okay, if some frontier company comes up with some innovative new way to, uh, do, you know, you know, instead of training with four modalities, they're training, you know, with eight different types of data or whatever.
Well, if they immediately have to publish their mechanism for doing that, then I assume you're expecting that, you know, others would immediately adopt anything that's functional, and so sort of i- in a way would speed up the pace of innovation, um, on that level. Uh, uh, but, but somehow you think the benefits of, uh, having it occur in public and us all being able to talk about it and so on outweigh what sort of the, the more naive or, or sort of, you know, people who haven't thought it through as much but have a safety, uh, lens would, would have intuitively preferred of like, "Yeah, everyone, like, don't talk about this.
It's better if we don't know how GPT-12 is trained or whatever so that, you know, the other companies can't copy them."
Thomas: Yeah. So I [00:38:00] think that this was just historically a big mistake that the safety community has made, and I disagree a bunch with, I think, the status quo wisdom here.
Nevin: Okay.
Thomas: So I think in, in most cases we'll want to do something that's much more open than I think the traditional safety community h- has wanted.
Though I will note that, uh, there are think- I think there are some ways in which the future could go where the sort of a, a, a secrecy very lockdown is warranted. But, um, some initial points, some like sort of, of the most salient important points in the, in the pro-transparency direction. One of the main ways we sort of expect any sort of, uh, AI regulation to go wrong is that historically the government is just pretty bad at doing things, es- especially, like, regul- regulating complicated technology which they don't understand very well.
And so we want as much as possible to make any sort of government involvement as... have, um, basically as little reliance on the technical expertise of the government, and the transparency helps a huge amount in letting sort of, letting our [00:39:00] regulatory regime rely a huge... like much, much more on third-party risk assessors and the broader scientific community- Mm
and the international community. Mm-hmm. As opposed to, like, a bunch of government... like, the, the few government people who've got the necessary security clearances or whatever to have access to the information. And we think basically the, the broader like epistemic and regulatory discussions about what AIs are, are safe or not will be like vastly, vastly better e- u- under this sort of transparency regime than in the, like, it's super locked down regime.
Romeo: Yeah.
Nevin: Okay. Okay. And so to get to this phase of, um, of basically requiring disclosure of your methods and code and stuff, um- Do you think that that requires an international deal first so that that would happen on a global scale? Or do you think that the US can do this independently of making a deal with China or others?
Thomas: So I think it would be bad for the US, [00:40:00] or AI companies in particular, to unilaterally do this because it would be ceding a massive competitive advantage for, for nothing, uh, and would sort of just accelerate the trailing actors and make the r- Like, you know, right now there's a couple- Yeah ... uh, US AI companies in the lead.
Uh, it would sort of cede a bunch of the competitive lead, um, and then make the race closer and more hairy. I think this is particularly good in this particular regime where we've set up a bunch of this verification infrastructure, which Romeo is talking about, where you've got, um, the US and China agreeing to declare large fractions of their compute supply chain and do sort of this, like, joint regulatory, um, approach where they coordinate to, uh, slow down the intelligence explosion on both of their sides.
I think in that regime, sort of the transparency helps with a huge amount of that implementation and makes it happen better on net. But in the absence of that, I don't think it would be good, um, on average. Though I do think, like, more limited forms of it, like the stuff we were talking about earlier, would be good to do unilaterally.
Nevin: Okay. [00:41:00] So walk me through, walk the listener through, um, what is it like for the US and China to negotiate and reach a deal that would enable something like this?
Thomas: Yeah. So I think the core problem is right now the US is in the lead, um, and China's behind. Um, China isn't gonna wanna lock in anything where they sort of permanently get behind the US, and the US isn't gonna wanna, like, agree to anything where they sort of le- lose their lead and China gets to c- catch up, um, to the same amount.
So you sort of... Maybe it's impossible. The sort of way we've tried to thread that needle is by saying, well, one, because of the research transparency, we'll equalize the algorithms. So you- China will catch up a bunch in terms of the algorithms, but the US will get to maintain their hardware lead, where right now the distribution of compute...
Romeo know- knows the numbers better than me. What's the exact distribution of, [00:42:00] of compute- Yeah ... worldwide relative to US and China?
Romeo: I think at the moment it's, like, five, five or six X, the US, uh, in total AI compute relative to China, and probably by 2029 we think it'll widen a little bit and just be, like, around a, a 10X gap.
Nevin: Oh, so the gap's getting bigger, not smaller.
Romeo: Yeah.
Nevin: Hm.
Romeo: Yeah.
Nevin: Why is that?
Romeo: So there's the export controls-
Nevin: Yep ...
Romeo: and then China's domestic abilities, um, have also been hit by the export controls, just at a different layer on the, on the chip equipment Um, there's imperfect enforcement, but, uh, yeah, based on what it seems like China's gonna be able to produce domestically, they are many years behind in, in the quality of the chips and the-- they will be able to, I, I think compensate pretty heavily with quantity.
Um, even then, I think that's, that's [00:43:00] still kind of the gap that we're looking at. My Chinese compute forecasts are actually like a little bit more bullish than, uh, kind of the people I've been able to get feedback from, um, that are looking at this and actually in some cases I think have better data than me.
Um, but still I... Yeah, I've, I've, I've leaned on the side of, um, China kind of waking up to this, putting a lot of effort into kind of diverting some of their Huawei capacity that a lot of which goes to smartphones now, trying really hard to, to make the specialized like h- high bandwidth memory that they need for AI chips domestically.
Um, uh, and all of this just still falling, falling behind, um, gradually and then there being like slightly better enforcement on the smuggling and some of these other sources of, of compute, um, that China's currently getting. So yeah, that's my, [00:44:00] that's my overall picture. I think A good lower bound is actually just the spending gap between the US and China.
That's, like, the biggest driver of this, of this, like, 5, 6X gap today- Mm ... is that the capital investment is already, like, four... A multiple of, like, four or five. Mm. So just even if they could both buy chips at the frontier, no export controls or anything, maybe that would cause China to increase their investment somewhat.
But there is a massive gap between kinda what the big tech companies in each country are currently able to mobilize.
Nevin: And I know this is an aside, but, uh, if the- if suddenly, uh, Chinese manufacturers were able to purchase EUV lithography machines from ASML- Mm-hmm ... how much would it change that trajectory?
Would it actually be super material, or would it not really matter in the next... I- in the time period we're talking about?
Romeo: So, yeah. I, I think [00:45:00] if they were able to purchase EUV, it would be a really big difference. Um, there's a gap between what it costs to make a chip and what it costs to buy a chip in the US that is a very large gap.
There's, like, a lot of large profit margins.
Thomas: Yeah, what's the Nvidia markup?
Romeo: Um, I think the Nvidia markup is, like, a 4 or 5X multiple.
Thomas: Wow.
Romeo: And that's just the t- layer on top. Um, TSMC and, and the memory providers also are now starting to see, like, similar markups.
Nevin: Okay.
Romeo: I think what would happen if China had EUV machines is kind of like- What you s- I think is what is happening kind of with their solar panel manufacturing where their solar panels are, like, booming.
Um, they're kind of doubling the- their production I think every, like, two years or something at the moment. They're on this very steep trajectory. But all the solar panel companies, like, aren't making any [00:46:00] profit. And- Mm ... I think you might-
Nevin: Just because of the way China runs companies.
Romeo: Yeah, I think- I think that's right.
I don't fully understand it, um, how the- how the system works there. But my guess is that you would s- you would have AI chips available in China, um, at very, very low markups-
Nevin: Mm-hmm ...
Romeo: and very, very high quantities correspondingly. Maybe they would kinda be most constrained by their ability to, like, uh, secure the- the ca- the EUV capacity from ASML.
Um, that might actually keep the situation a bit stable for a while. I'm just thinking through this now. But ASML kind of has a very inelastic supply- Mm ... for increasing their production of- of EUV machines. I see.
Nevin: So even if they were politically allowed to- Yeah ... it would take them a while to just be part of their order flow.
Romeo: Exactly. I think basically they would be outbid by the kind of Western supply chain [00:47:00] that c- currently has more demand behind it in- in pure dollars. But- There's another factor, which is even them getting one EUV machine could also significantly accelerate. They're able to- their ability to, to make, um, similar tools domestically- Mm-hmm
by, like, pulling it apart and reverse engineering things. I think, um, yeah, that, that might be another big factor, so.
Nevin: Last question on this aside- Yeah ... while we're here. Mm-hmm. If, let's say they suddenly were at the front of the line and could buy as many of the machines as they wanted- Mm-hmm ... how, how long do you think it would take them, like institutionally and talent-wise and so on, to get to the point of being able to use them the way TSMC can- Yeah
uh, to, to make all those chips? Is that like a, you know, one year, five year? Like, how long does it take to actually-
Romeo: Yeah ...
Nevin: make it work?
Romeo: It's a good question. I think, um, my [00:48:00] impression is that it would be closer to one year. I think they have already made a lot of progress, of, of the relevant progress just at the older t- technology node that they're, they're working with.
I think the transfer is probably just very high between, like, running the kind of seven nanometer fabs that they have, um, and pushing the limits on what they're doing with that-
Nevin: Okay ...
Romeo: with, with the older DUV technology to, to making it work at, like, a three nanometer fab. I think probably my, my understanding is a lot of the same, like, expertise and engineering, um, talent required would be quite similar.
The, the constraining thing is, is more gonna be, like, ramping up the total volume. Like, there, there will have been an accumulated stock even if, you know, in, in two years' time China's given up, um, access to, to EUV machines. There's like a decade worth of EUV machines that are all in [00:49:00] the Western supply chain and working in fabs.
Right. So I think the ramp-up to being competitive with the total Western kind of, um, production quantities, uh- Will be, um, more on the three to four-year range e- even if they're-
Nevin: Okay ...
Romeo: on a very, um, aggressive ramp of that domestically.
Nevin: And to give some context here of part of what I'm asking is just thinking about if you're in the Chinese position, you know, you have the direct prevention of purchasing the frontier chips, uh, from TSMC, and then you have this indirect, you know, barrier to purchasing the underlying, uh, technology for your own fabs.
I think it's useful every once in a while to just imagine what would that feel like to be like, "Okay, this is the alternate reality we could be in in a few years if we were allowed to purchase these things, and th- and, and because of these decisions of this other country that is milit- [00:50:00] has this military dominance, here's where we are instead."
That is the context in which these negotiations have to happen. Mm. Um, so, um, you know, it's... I imagine there's some, some sort of pissed off yelling and huffing and, you know, like-
Thomas: Yeah ...
Nevin: you know, "Screw these guys." You know, uh, you know, we, we heard a very sort of diplomatic and interesting address from Xi Jinping yesterday to the world.
But behind closed doors, it must be pretty annoying, uh, to be in that situation
Thomas: Yeah, I think that's right. I think, um, I think there's a vision of, like, you know, international cooperation on AI, which is, like, very rosy and very, like, kumbaya and we all come together. And I think the vision that we're trying to depict in Plan A, like, part of the reason why we went for, like, the red color scheme is, like, we're imagining that this will be a very, very gnarly, heated, risky period of, of [00:51:00] negotiation and diplomacy, and that the, like, fundamental driver of a bunch of this...
Like, the fundamental political driver of a bunch of this stuff is everyone is going to be, especially the non-US countries, but including the US, is going to be extremely, extremely freaked out about what AI is... What, what's gonna happen with AI. Yeah. Um, we, we mentioned a lot of issues. The US will be worried about job loss, I think.
The US, we have a population that's probably very anti-AI. The other countries will be worried that the US will, you know, get an insane amount of geopolitical dominance on the back of AI, and will have... Will be, like, strongly, strongly pressuring the US. And so I think the negotiations will be extremely tense.
They will be extremely consequential. They will matter massively for sort of the whole future.
Nevin: Who, who's gonna be negotiating? Is it gonna be one leader and the other having a lot of conversations over the course of a year? Or is it, like, you know, a bunch of, uh, sort of career bureaucrats flying back and forth and having these conversations?
Like, how do you think that that will actually play out?
Thomas: Yeah. So neither of us are experts in this exact field. My guess... So I think there will need to be a huge amount of engagement [00:52:00] between extremely technically competent people in both countries. So there will need to be, I think... If anything like this were to happen, I think there would need to be a bunch of engagement from the top level.
So you'd need, like, you know, Xi Jinping and POTUS to be h- you know, doing handshakes and s- and signing deals, uh, signing probably several deals, or m- many deals. Several over the course of the first year, and then many in the future. And then I think between those events, you would need huge amounts of, like, you know, technical experts within each government.
So the equivalent of, like, um... So, like, right now you've got, like, the OS, the, the White House Office of Science and Technology Policy, and, like, the National Security Council, and, like, those folks. I think a lot of those folks are very, like, technically grounded, technically competent, really know what's going on in AI, really know the people.
Um, I think you'd have a lot of those folks. You'd have a lot of the labs both in the US and in China involved. And I think you'd have those people, like, living together for a year or more basically, or, or at least constantly flying out to different conferences with each other. I think a good analogy [00:53:00] is World War II.
Um, so in World War II when the allies were fighting, um, you know, you had very, very regular communication between, you know, Britain and the US and, and other allies. Um, you also had very regular, like, you know, um, the Casablanca Conference, the Yalta Conference. You had, you had many sort of conferences where you got the big head honchos together and sort of discussed issues at a high level.
Nevin: From opposing sides.
Thomas: Um, no, not from opposing sides. Oh, okay. Well, in some... I mean, you, you, you never got that between, like, Germany and the US and Britain. Okay. But you did get that between the US, Britain, and the Soviet Union, who were in some sense-
Romeo: Mm ...
Thomas: big enemies even at that point, right? Even though they were technically on, on the same side of this war.
Um, and, and the US and the Soviet Union were not at all on the same side in, in the sense of, like, what they, what their fundamental, you know, what they fundamentally wanted out of the geopolitics of post-World War II. The Soviet Union wanted, you know, probably as much territory as they could get, and the US was, uh, was thinking ahead at that point to [00:54:00] the post-World War II sort of geopolitical balance of power, and they wanted more of it relative to the Soviet Union.
Um, and so they were not... They were like fundamentally there was a bunch of zero-sum dynamics that they had to negotiate. Um, and you know, the, the Stalin really, really wanted the Western allies to invade, uh, and open a second front in Europe as soon as possible, right? And the Western allies didn't want that to happen.
They wanted the Russians to do as much of the fighting as possible. And so there were sort of all of these different tensions there, um, and that ended up getting negotiated in these very, very tense conferences w- with very little trust where they were spying on each other all the time. Um, but you know, obviously they had to make a decision.
They were, you know, fighting this existential war- Right ... where if they were not able to come to a deal, it would've been really catastrophic for both sides. And I sort of think, I sort of think that's what it's gonna feel like.
Nevin: Hm.
Romeo: Um- It's against the AIs.
Thomas: Yeah. Yeah. Yeah, yeah, yeah.
Nevin: Okay. Makes sense. Okay, so, so then, and I, I realize we're, [00:55:00] we're kind of going in more detail through this whole scenario, but I think this is a good way to talk through- Yeah
these pieces, and I'll ask some follow-up questions once we kind of hit the end narrative-wise. Um, so what happens next? So now we're at a point of, you know, one way or another, uh, the US and China have, have negotiated a deal or a series of deals. Interesting that you would think it'd be a series. Yeah. But whatever, let's, let's gloss over that.
So what are the main things they've agreed to, and then what happens in the following period?
Thomas: Yeah. So I think I'll, I'll, I'll, I'll sort of like quickly go through the rest of the, the agreements, then I think, Romeo, you can take like the, the crazy- Economic side. Yeah ... economic transformation we see. The sort of the initial set of agreements were, there's the transparency agreement that I mentioned.
There is, um, there needs to be some sort of regulatory agreement made, um, on, in both US and China and other countries that are developing AI, which is the, um, for, uh, basically f- for the AI progress that they're making, we want [00:56:00] them to make sure that that's safe. So there's some sort of safety case-based regime where there's regulators on both sides.
Um, there's like US domestic regulator, Chinese domestic regulator, et cetera, making sure that the companies that are still developing AI are doing so in a reasonably safe way, and are not making, like, huge amounts of algorithmic progress that would lead to very, very scary dynamics where it would, for example, let, um, very small amounts of compute potentially make, like, huge amounts of, uh, capability progress and build very, very scary AIs.
So you'd need sort of this regulation. Then you would need a bunch of verification agreements to make sure that they trust each other Um, so a key worry with all of this is, um, uh, the US is gonna be like, "If we agree to regulate our AI companies, what's stopping China from secretly taking some compute, stowing it away, um, building AI that's much, much more powerful than anything we've got, and then using that for, for their own gain?"
And so there's gonna need to be a bunch of agreements basically [00:57:00] where the US and China and other countries, um, allow inspectors into their data centers and into the chip supply chain to basically see the records, see very detailed information, where you've got this- where you basically have, you know, you've got, you've got this really long semiconductor supply chain.
You've got, uh, Nvidia designing the chips, TSMC manufacturing the chips, um, uh, ASML giving the machines to TSMC to then manufacture the chips, and then many, many, many other sort of smaller providers. A key part of the verification scheme is each of these providers, um, basically declares the records that they've sold to the, you know, upstream providers and what they've bought from the downstream providers.
And then there's this massive audit done to make sure that all of the numbers line up, where TSMC says, "Hey, I bought this many EUV machines", and then ASML is like, "Okay, I sold this many EUV machines." Yep. And, and there's that for a bunch of smaller components as well. And so you make sure that all of that lines up.
And then you hopefully at that point have a very, very accurate understanding [00:58:00] of the number and type of different AI chips in the world. So you know there's exactly this many H100s, there's exactly this many, um, H20s, et cetera, for basically all the chips. Then you see, okay, for all of the data centers that were declared, how many did we find?
Um, and then how many are still missing? And we've got a bunch of guesses about how this will go. We think it's probably pretty viable to get something like 99% of the AI relevant chips. So this wouldn't be anything like laptops or phones- Phones ... but it would be like the H100s.
Nevin: 99 seems totally sufficient.
Thomas: Yeah, yeah, yeah.
Nevin: Seems like you'd be fine at like, what, 90?
Thomas: Um, I think it's, I think it's... If it was 90, I think it would be complicated. I think it might be worrying. '
Nevin: Cause there could be too many secret training runs happening.
Thomas: Current frontier AI companies have something like 10% of the world compute, like Anthropic, OpenAI, I think are a little more probably at this point.
So if you really can only get 90% of the world compute, you still have like an Anthropic-sized entity potentially out there, uh, doing ongoing training, and [00:59:00] so that's like pretty worrying.
Romeo: And because of the research transparency as well, they, like currently Anthropic probably uses something like 50% of the compute, it seems, on, on R&D, but the actual final training runs are, are a smaller percentage.
Um, if you're kind of this covert project and you have this massive compute, you can just wait for the transparent research to happen- Mm. And then just over time-
Nevin: Right ...
Romeo: copy the algorithms and just do the training run. Um, so- It's actually th- this question of what percentage you can get. Like, if it is 90 versus 99 versus 99.9, it kind of changes the equation in terms of the downsides of the transparency.
Nevin: Yeah.
Romeo: Um, and that's one k- key trade-off that's, that, that's happening. But we think because y- you can probably get 99 and because you slow down the pace of the kind of algorithmic [01:00:00] progress, um, the, the total research transparency is, is quite viable. Um- Okay. Yeah.
Nevin: That helps me understand something from earlier.
I just wanna comment on it briefly, and then I'll give it back to you, Thomas. So, um-
You know, you have this proposal of open source but not open weights. And one thought I had had is, like, well, if, if the source is public, then as soon as there's a new innovation, everyone is just gonna use that and train their own version of basically the same model, especially if you have to disclose what kind of data you use, et cetera.
Um, but I guess the point is, well, yeah, that's true, but only the people with the compute would be able to do that, and those are all the parties that we're gonna be able to watch and monitor. Yeah. So if some- so someone ... It w- would have pretty hard time creating a secret data center and, uh, creating a version of the model that, you know, is, is just, is not part of this overall transparent public discussion.
Is that the idea?
Romeo: That's the idea.
Thomas: That's exactly right. Yep.
Nevin: Okay. [01:01:00]
Thomas: Um, and so then when we're getting into this sort of the covert project detection question, so getting compute, preventing them from getting compute is, like, one lever that we have that I think is very promising, but there's a bunch of other levers.
In particular, it's just very, very hard to build, you know, a data center with 1% or t- e- even 10% would be very, like, I think impossible. Um, if you try to build a giant data center that would be costing, you know, tens of billions of dollars in secret, um, even if you can get the chips, uh, actually constructing the data center, getting the money in a way that US spies or Chinese spies or, or whoever can't, can't see, um, evading satellite monitoring, evading, like, heat signature detection.
You know, that, that amount of compute will, you know, require 100 megawatts, a gigawatt, um, like, on that order, uh, which does produce a lot of waste heat, and you need to power it. You need to figure out some way to power it in secret. So there's a bunch of questions basically that any covert project has to navigate that I think makes operating a very [01:02:00] large scale covert project basically infeasible.
Operating a small scale covert project for a sufficiently small project, I think it's of course feasible. And so then we sort of get into this quantitative question of exactly how small is, like, sufficient to be secret. And from our modeling- I feel pretty confident that this basically won't be a big concern, and that if you...
And that basically you can get this number to be small enough for, such that for a 10-year style takeoff that we do in plan A, covert projects really won't be the binding constraint, and the binding constraint will be, you know, many other failure modes which, which we, which we, we should get into, uh, that I think are just, like, much, you know, substantially bigger concerns.
It's not, like, totally... Like, it's important that we do the analysis, that we check that the numbers actually, uh, you know, work out. If it's the case that you do your, you know, chip tracking and it comes out that, "Oh, actually, we can't find 10% of the world's compute, and we have no idea where it is, and there's a lot of heat coming out of that Chinese military base, and we're not sure what, and they're not willing to tell us what's going on in there", [01:03:00] then, you know, then you probably should be pretty worried about covert projects.
But I think sort of assuming that all of the declarations go well and there isn't a bunch of evidence that you see, then I think, um, you shouldn't do covert projects. You shouldn't be super worried about covert projects. Okay. And, uh, yeah. Yeah, Romeo, do you have anything to add to that?
Romeo: Yeah. One, one other kind of situation in which, which I think you could get, you could- should be worried about it, even if you have kind of tracked down 99% or 99.9%.
Um, and I think that is the case where under your transparent research, the kind of pace of algorithmic- Yeah ... efficiency discoveries is just, has not been kept low enough. Um, and you know, today, it seems like algorithmic progress, um, might be around kind of, like, a 10x efficiency gain per year. It's pretty un- unclear.
Um, if that pace of progress were kept for, you know, five years, [01:04:00] um, rather than slowed down, then that's kind of five orders of magnitude of efficiency. That means that the 0.1% of compute could now actually be very scary. Mm-hmm. And, you know, at that scale, we are into the regime where you might have to start worrying, worrying about also, like-
Thomas: Yeah
Romeo: decentralized training runs or even, like, consumer compute- Yeah ... being hooked up in large quantities. So a core part of this whole dynamic is to also be Taking into account how much algorithmic efficiency has been achieved, keeping that low to the extent that you're not able to keep it low, you know, take that into account in the, in the decision-making of, um, yeah, whether to try and increase detection, doing, doing more deals to try and get to those higher levels of confidence.
If you're not able to do that, maybe, maybe it is time to, to figure out the kind of exit to the deal, which we can talk about [01:05:00] later, the, the options there.
Nevin: Um- I find it really plausible to imagine that, you know, you, you, you mentioned that if you require open sourcing the algorithms, you reduce the incentive to make algorithmic progress financially.
I can see that, but it seems really plausible to me that algorithmic prog- progress would go faster in this situation, absent bans on particular research directions, which I know is something that you add to this scenario. Um, but absent any bans, um, it seems, just seems to me like
Kind of in the way that like as a, as a normal person, the, the value of money kind of drops off at some point. It's like, okay, you have, you have a ton of money, like having double that money is like, sure, it's cool, but like it's not gonna change your life that much. Um, it seems like similarly here I would, I would say that there's still a very strong incentive, [01:06:00] uh, financially to, um, to, um, to make progress.
But also just if you, if you think about the engineers in these companies, it's like, okay, there's enough of a financial incentive to have one of these companies- Yeah ... and to try to provide a product, right? It's like, well, we get a lot of, we get a lot of, you know, uh, insights for free from the world. But if you wanna attract good talent to your company, well, what are the people who are super smart who can make progress on this wanna do?
Well, they wanna figure out new stuff. Like, it's incredibly fun to figure out, you know, better ways of doing AI, I think. Yeah. Um, and if you can, if you can figure it out and it's not just a secret in your company, actually it gets broadcast and everyone knows that you figured it out, that's actually, like, a much stronger incentive for the geek type who's actually making this progress.
It- so much so that, you know, in- I remember in the earlier days of DeepMind, I was really surprised at how many papers they were publishing. It's like, "Guys, you're just, [01:07:00] like, making advances and then just publishing them." And OpenAI at the beginning did a lot of that, too. I think it was because that's what they had to allow the researchers to do to attract them and have them be willing to work in that context.
This would, in a way, sort of cr- recreate the sort of academic publishing type situation- Yeah ... uh, for people working inside of extremely well-funded companies. I think that that would mean more thinking and faster progress in the, in the global community. Again, absent any sort of collective decision-making about, we've decided not to research this area.
It's illegal to do this- Yeah ... so don't do that. What do, what do you think of that line of thought?
Romeo: I, I think this is a pretty great kind of argument in response to the critique or concern about the open research transparency proposal being too much of a, being something that will kind of kill progress, um, kill, you know, growth and all these things.
Like, and I think basically the [01:08:00] load-bearing part of the algorithmic slowdown does in fact need to come from the regulatory side- Yeah ... instead of just, um, trusting the- Yeah ... transparency to do enough on this because of the things you're saying. I basically, I basically agree, find it plausible. Um, the regulation is the thing that needs to kind of be titrating the algorithmic progress-
Nevin: Okay
Romeo: in my view, and I think doing that well will be extremely hard. Bala- ba- balancing these risks. Like, there's not just the technical safety cases to be evaluating, but there's the overall strategic situation with respect to potential secret projects, potential, like, time, uh, left in the deal, which is something we haven't really talked about yet, but You know, every, every year going by under the deal, you're incurring some risk of it breaking down.
Mm-hmm. And so you do need to kind of balance, [01:09:00] um, going, going faster and, and potentially making, like, m- incurring a bit more of the risks, but getting more of the benefits earlier on, um, might be quite a, a, a smart thing to do. Um, because y- you can't just rely on the, the deal being stable and lasting forever because of outside factors, like political ones.
Nevin: So maybe we can get into what that sort of collective regulation might look like in a few minutes.
Romeo: Yeah. '
Nevin: Cause first, I, I, I think at this stage of the deal, you are imagining this mutually assured compute destruction regime and people building data centers in opposing countries and all that stuff, right?
Yes. Yeah. Let's talk about
Thomas: that. So this is, this is maybe the most, uh, controversial and weird part of our plan. Um-
Romeo: Yeah.
Thomas: So basically, what happened here was we wrote the initial draft of plan A, uh, like very early on, and it didn't have anything to ... like this. [01:10:00] And then we did a bunch of war games where we tried to implement plan A.
And then what kept happening in the war games was we would do plan A, and then like three years in, or five years in, or two years in, or whatever, a new president would get elected or s- you know, s- a new ... Something would happen politically such that the deal breaks down and everyone goes back to racing.
And then what would happen is that since we built way more compute in the meantime, like over the, those few years, there was maybe 10X more compute, or 100, or like some, some significant factor more compute in the world.
Nevin: More compute than there otherwise would have been?
Thomas: Yeah.
Nevin: What causes that?
Thomas: Or, sorry-
Romeo: Not than there otherwise would have been, just than there
Nevin: was- Just, just than, than there was at the beginning
initially. Than
Thomas: there was at the beginning. Okay. Okay. Because you build more compute over time. Yeah.
Nevin: So ba- basically, the compute scaling trajectory is about the same in this scenario as the default case.
Thomas: Um, so in the default case, if you don't do any governance, there's a singularity, and then you get, like-
Nevin: Okay, okay.
Yeah. But, but absent, absent that effect occurring-
Thomas: Sort of the default trend. Yeah.
Nevin: Yeah, yeah,
Romeo: yeah.
Nevin: This scenario doesn't depend [01:11:00] on really bending the curve of compute build-out.
Thomas: No.
Romeo: No.
Thomas: Yeah.
Nevin: Okay.
Thomas: Basically, you just ... The default compute build-out curve is, is very aggressive, you know? There's like 2X, 3X more compute every year in the world.
Um, and so just by naturally over the course of time, more compute ends up getting built. And- There's this big question of how much faster does the intelligence explosion go if you have more compute? Um, so the way we typically think about this is suppose you have ten X more compute in the world, how much faster does your intelligence explosion go given that you have ten X more?
Um, it's not gonna be ten X because there's other inputs like human labor and world feedback and stuff, but it's also gonna be, you know, it's not gonna be one X, it's gonna be... You're gonna get some amount because compute- Right ... is somewhat useful. Our best guess is, like, five to six x. Like, ten X more compute causes a five to six x increase.
Compute's very, very important, but not the only
Romeo: thing. Especially kind of post, like, full automation, you know, when some of the other inputs might go away. Yeah. Um, I think it might be on the [01:12:00] higher side, like- And
Thomas: earlier might be lower.
Romeo: Yeah.
Thomas: Um, but anyways, basically if you believe that number, that means that let's say you've increased the amount of compute in the world by a hundred X, like, during this deal.
Mm. Then... And let's say the number is five X. Let's say the returns to compute is five X. Then with a hundred X more compute, your intelligence explosion goes twenty-five X faster. So if it would've happened over, you know, two hundred and fifty days, like most of a year, then now it would happen by, in ten days, you know?
So you get, you maybe get an extremely, extremely, extremely fast intelligence explosion because of this major compute build-out. And we basically kept seeing this in the war game, where you would do your deal, it would be great. You'd get a huge amount of safety progress. You'd, you know, get sane regulations, but then eventually it would fail, and then you would get an extremely, extremely vertical curve that is extremely difficult and des- destabilizing and scary.
Um, and so we were like, "Okay, how do we fix this?" And the solution we landed on was, um, basically make sure that if the deal breaks down, [01:13:00] um, there is less compute available in the world, or there's about the s- we sort of revert back to something like the pre-deal amount of compute in the world. And the way we do this is the US data centers mostly get built in a location which could be seized by China if the deal breaks down, as well as vice versa.
So in particular in the scenario, we say the US data centers are built in Mongolia and the Chinese data centers are built in Canada. And so if the deal were to break down, what would happen is the US would go to grab the Chinese data centers, uh, you know, with some, you know, physical, you know, physical people, uh, a- as would China.
But, um, because the o- you know, because China knows that the US would go take their compute, China would deliberately destroy, would deliberately set up mechanisms to destroy or scuttle their own compute so that the US doesn't have it, and the US would also do the same. And so for the big legal AI data centers, they would sort of- there would sort of be the incentives in place to [01:14:00] destroy them.
And then the remaining compute in the world would be all of the consumer hardware that isn't the AI stuff, um, as well as, um, any other stockpiles of compute, uh, that are, like, not in the destroyable zones that are, like, just located in the US or China, um, which we would recommend would be set to be roughly equal to the pre-deal compute ratio so that the sort of incentive...
So that the sort of BATNA to the agreement, such that if the deal were to break down, the sort of status quo that it would ret- return to would be something like, you know, what the case was before the deal, um, to avoid this problem of, like, if, you know, you could just join the deal and then immediately leave.
Um, and we want that to sort of be a, a, a viable- Yeah ... um-
Nevin: Okay. Two, two questions- ... incentive set up ... about that. So, um Concretely, is it like, you know, you have a data center and, like, every single compute tray has, like, a little, you know, dose of thermite that's, like, ready to be [01:15:00] detonated, you know, at a moment's n- notice?
Or is it like, you know, just like, no, you can just, like, send in a missile from afar with, like, you know, a 30-minute- Yeah ... heads-up to tell them to get the people out? Or, like, what, what is the actual destruction mechanism that you imagine?
Thomas: Yeah. So there are a bunch of viable options. We wanna avoid missiles whenever possible because, you know, actual, like, c- c- conventional war between nuclear countries is, is a very scary thing that we want to avoid if at all possible.
Nevin: Yep.
Thomas: So, um, our first line of defense is, uh, cryptography, which is, um, we, we have this proposal, um, which we, which we use, which, which, um, there's some literature about called offline licensing, which is that basically for the GPUs in the clusters, they require a, uh, key to be sent from both the US and China to continue running every, you know, for every certain amount of time, like every few hours.
Hmm. And so if they stop getting the key, then they're set up to automatically stop running. Um, and that's sort of the first line of defense. Hopefully it works, but, you [01:16:00] know, maybe that can be hacked.
Nevin: Where, where you mean set up to stop running, so it's not that you, like, need the key for every single computation.
There's, like, some further module that has to, like, receive the- Yeah ... two of two key in order to, like, enable the system. But, but does it physically destroy the system if it doesn't get its, like, call from home or whatever?
Thomas: No. We would say that it would just, uh, the system would stop working until it got another key.
Yeah. So basically for every N flops or for every N hours- Yeah ... it would need a new key to keep running.
Romeo: There's a paper by, by RAND about this. I think the default proposal is that there's, like, some kind of resource counter or something, and every time, uh ... And, and basically the chip is throttled to, like, uh, only be able to reset that counter with the keys.
And you
Nevin: can't just, like, rip that mechanism off and run it without the- Well,
Romeo: that's, that's the question that would need to be figured out. Like, I think you can only really rely on this if both sides are extremely confident that it's hyper-secure [01:17:00] and, and, and even to, like, physical tampering. Maybe you can kinda solve that physical problem Um, by something else.
Like, you have the chips all in like a, an enclo- like a tamper-proof enclosure, and then separately you have this key mechanism, and so the two together would be hard to- Okay. But like, yeah.
Thomas: But anyways, we don't really wanna rely on this. Yeah. This is- This is like- Sure ... this is the, like the elegant solution- Yeah
which would be, which would be nice if we could make it work. Yeah. The, the sort of, um, more blunt solution is, yeah, you just like have US soldiers in the US data center ready to destroy the compute, and that could, that could be via like, yeah, you've got your little bit of acid or something that you could just like spray on the chip to make sure it works.
You could imagine like tiny explosives. You could imagine all sorts of ways to basically, um, destroy the compute. We would want it to basically just be set up such that they're ... the US has a sufficient workforce there, like some me- members of the National Security apparatus are at the data centers.
They've got probably a bunch of independent [01:18:00] backup, like backup, um, mechanisms, uh, to make sure that they can destroy it at reasonably short notice. Mm-hmm. Specifically, a short enough notice that they can be confident that if China tried to make a move to grab the chips and then like rapidly turn off all the devices, the US would be able to notice that and then destroy the compute before China was able to do that.
Romeo: Yeah. I think like, yeah, it might be worth talking through maybe some of the kind of alternative situations, but just like to motivate this. I mean, you could imagine if the, if, if it was just the reverse, for example, like the US data centers are in Canada. Um, you know, the problem there is that when the deal breaks down, the US can probably quite easily defend its data s- its data centers in Canada with respect to China.
Um, and then, you know, we really don't want there to be like a situation where China really wants to destroy the data centers that are in Canada. It's potentially even worse if they're, you know, on their own soil. [01:19:00] Um, you know, that for, for the geopolitical risk. Um, you could have like, yeah, I don't know, maybe you could have like some, uh, the data centers in Australia or something, but then like, you know, it's unclear exactly where they're, um, who, who would be able to defend that.
You know, Ch- we don't want like the, basically the risk of war and conflict, um, to, to happen after the deal, but we do want it to be the case that it, it, it's like, um, easy to revert to the previous status quo. And so like this is kind of the solution that we currently Think is best and it's like, um, this nice elegant thing where, um, hopefully, you know, we're, we're heavily incentivizing the destruction to be like a self-destruction, like a scorched earth thing.
Mm-hmm. Um- As opposed to bombing. As opposed to bombing.
Thomas: And it seems like in the natural case, if you don't do this, [01:20:00] which we, we admit it's a little weird, but if you don't do this, it does seem like the natural thing is you either get a really fast singularity, but probably whichever side is on the losing end of that really fast singularity will have this massive incentive to sort of intervene to stop that-
Nevin: Right
Thomas: via, including via force. And so you're sort of naturally in this position then where if plan A breaks down, you get war.
Nevin: Right.
Thomas: And you, and you sort of would prefer this sort of much more like elegant, uh, um, hopefully like you, you want a, a version of the deal breakdown that doesn't necessitate there being a complete World War III type scenario.
And you also want to avoid there being a World War III type scenario in the beginning, so.
Nevin: Is there anything in place like this within TSMC's factories? Like if China, if mainland China decides to take military control of Taiwan, uh, is there some way to, uh, destroy what's there without firing missiles, or is that not the case?
Thomas: So yeah, my understanding is the missiles is the plan, and that the US is [01:21:00] basically planning that if, if they lose Taiwan, they blow up TSMC so that China can't get it via missiles, much less elegant.
Nevin: Mm.
Thomas: Um, and, and very sad.
Nevin: Yeah. That would be, that'd be pretty ugly.
Thomas: Yeah. Yeah.
Nevin: Okay. Um, and so and then in this scenario, are you imagining that, um, export controls would be released?
Like once we have this deal, then China could get the best tech and use the best tech on the manufacturing and the, and the just sort of chip consumption level within these special data centers?
Romeo: Yeah. So there's, there's a bunch of context here. We actually don't specify in, in, in the kind of like first couple years of the deal what ha- what exactly happens.
But then we have, um, a whole kind of like agreement that happens in twenty thirty two, um, uh, that, that relates to this that I should explain. So one of the kind of key things that happens after the deal's set [01:22:00] up and like we have the research transparency and the, and the regulation, um, the, the kind of dominant risk now changes from being the risk of, um, kind of the recursive self-improvement and, and AI takeover.
Um, you know, that's kind of been averted, but now we think the dominant risks are like the, the two we've kind of talked about, which is the potential for a secret project There's the p- the, the risk of the deal breaking down. Um, and then another one is there kind of being a risk of an industrial explosion.
Um, so there being kind of dangerous, um, re- or like destabilizing, I think is, is the best word, uh, progress in hardware where either you could imagine there being kind of like R&D that leads to, um, it, [01:23:00] uh, much easier to manufacture AI compute Could be a destabilizing thing. So now, you know, maybe there's a paradigm change.
You no longer need a crazy complex EUV machine to make- Mm. Mm-hmm ... an equivalent piece of AI compute, and now your kind of risk of a covert project being able to do that somewhere- Mm-hmm ... underneath a mountain is elevated. Um, so that's kind of like, uh, uh, of like a hardware R&D, um, risk, uh, uh, vector. And then another one is just, um, that potentially once we do get AIs under the deal that are, um, and robotics capabilities that are able to kind of fully automate, uh, a compute supply chain or, and a, and a robotics supply chain, um, at that point, the kind of doubling times, the default doubling times on the amount of compute and robots in the world might be very fast.
Thomas: We think, [01:24:00] um, we think that once you get sort of the AI capabilities that we're talking about in the early 2030s, the AIs will be able to, um, basically operate robots-
Romeo: Yeah ...
Thomas: um, operate like humanoid robots as well as other type of robots in a way where they can, uh, f-fully automate every part of the building more robots supply chain-
Romeo: Yeah
Thomas: which includes chips. It includes building like motors for the arms. It includes batteries. It includes mining for the raw materials. It includes the transportation. So all parts of that supply chain, we think. You know, all, all of that right now is, is very bottlenecked by human labor. But at this point in the scenario, we s- we think that you basically get the, the AI software such that you can operate robots, such that you can do every part of that, which is a pretty high level of capability.
But once that's the case, then the growth rate, the number of new robots that you can produce every year, will be proportional to the existing supply of robots, um, because the robots can j- do all of the parts of that. [01:25:00] And so that gives you the sort of differential equation of, you know, the growth rate is equal to the existing population.
Yeah. Uh, hopefully you're familiar with the solution to this, which is, you know, exponential growth. Um, uh, and so then the, the parameter here that is uncertain is what the sort of doubling time, as Romeo was saying, of this- Yeah ... exponential growth. So how fast of exponential growth, uh, do we get here? Um, and so we've done some analysis and thinking about this question.
Uh, you can sort of... There's a bunch of ways of, of looking at it. Um- The, the bottom line is I think that you get growth that looks something like a doubling on the order of every one to 12 months probably by default. Um, let's say you get on the more aggressive end, which I think is very plausible. Um, like you- let's say you get a doubling every one month, then, um, basically every year you get, you know, you increase your robot population by a factor of two to the 12, um, which is, which is what, like [01:26:00] 4,000?
Romeo: Yeah.
Thomas: Um, and, you know, very, very quickly basically you'll get to this point where robots are the vast, vast majority of the entire population on Earth. Um, Romeo was saying this will be destabilizing. I think it's very clear that if you had trillions and trillions of robots doing every physical task in the entire economy, this would like radically, radically, radically transform sort of the overall balance of power, and it would probably allow there to be lots and lots of new technology.
Um, it would have massive, uh, impacts on the, uh, the climate- Yeah ... uh, which we, uh, talked about a bit in the scenario. Um, basically it would be very scary. And so in sort of the 2032 negotiation, we... One of the things that the US and China agree upon, and other countries agree upon, are the, uh, are to limit the robot build-out numbers to be something more like one doubling per year, not, not more than that.
Although it changes a little bit over time quantitatively. [01:27:00] Um, and this sort of limits the most extreme, most worrying forms of robot industrial growth, while still allowing there to be a huge number of robots that still have like a pretty huge effect on society for the rest of the scenario.
Romeo: Yeah.
Thomas: Um-
Romeo: So that's like the 2032 in, in, in the particular scenario, um, cap and trade agreement.
So the way that we're proposing for the robot and, and kinda compute, um, populations and, and numbers to be limited is for there to be a global cap on how many robots and, and AI chips can be produced. And then, um, we imagine there being a kinda negotiation in 2032 between all the countries in the world about how the Permits, how those permits are gonna be distributed
Thomas: So like what fraction of the number of robots can China build versus can the US build- Yeah
et cetera. Mm-hmm. Which is gonna be an extremely tense negotiation but is gonna, it's gonna be extremely [01:28:00] important. It's gonna be kind of analogous to the, the London Naval Treaty maybe where between World War I and World War II, the countries agreed to how many tons of ships each of their navies could be.
And, you know, some countries got bigger navies and some countries got smaller navies, and a bunch of the countries with smaller navies didn't like it. Um, and that was extremely important at the time for sort of the balance of, uh, you know, sea power, and the robot power balance will be very, very important here.
And so, you know, we, we make some guesses about how that might go, uh, in the scenario.
Romeo: Yeah. So I don't know. I think, um, from, from our perspective writing the scenario, this was like quite an elegant thing to do, was like say from, from our perspective, we're worried if the number of chips or the number of robots kind of grow beyond this number.
Um, because in the case of, um, yeah, in, in the case of, of both of these, you know, they- they're gonna increase the, the, some of the risks like the covert project risks, for example. [01:29:00] Um- And it's a very elegant thing to just be able to set to s- to... It's, it kind of solves the problem by default, you know, to say, "This is the cap."
Um, there's-
Thomas: In the maximally libertarian way, uh- Yeah ... additionally. Because there's like the... It's basically just a, there's a, the, the governments agree on a particular number. But then because it's a cap and trade regime where the government mints a certain number of waivers and then the private market buys, you know, buys and sells the waivers for robot and compute production as it wants, the market still gets to decide, you know, who gets to build what robots on the basis of who pays the most for each of the different robot waivers.
And there's no government, like, regulator deciding, like, "You in particular can't build any robots."
Nevin: And so the idea is that we'd negotiate sort of which, you know, how, how many of these permits, uh, each country sort of gets to, you know, enjoy the scenery of, in some sense. Because [01:30:00] they get to create that permit and sell it, you know, auction it into the open market.
I guess they can choose whether to sell it domestically or, uh, or allow anyone to participate in the auction. Yeah. Um, and so then basically, i- in theory, every country has this new form of revenue- Yeah ... um, of selling these, selling these permits. And this ends up being really economically impactful in the way the scenario plays out, right?
Yeah.
Romeo: Yeah.
Thomas: Yeah, that's right.
Romeo: That, that's right. So I think basically, um, the, these cap and trade permits are set in a way such that they kind of limit the, the doubling time of the economy to around one year. Um-
Thomas: Which is a massive departure from the status quo. Which is, yeah, that's right. Yeah. Right now, the economy doubles over the, you know, decades.
Romeo: Right. It's extremely, extremely quickly. Um, but yeah. It's like in the, in the first few years, uh, the, the robots and, and AI compute are allowed to, like, grow a little faster than that. And the way it works out with, like, there being kind [01:31:00] of other inputs of the economy that are growing slower, um, kind of all, all nets out at, to, to roughly a doubling per year is what the caps are setting.
The cost of making the robots and the computer chips, um, over time due to kind of Moore's law and other kind of gains from scale, um, dynamics, uh, seem like, and, and they're consistent with the default doubling times of these hardwares being lower than, than, than a year. Um- are, are shrinking over time. But you can only, you know, continue to, to make a fixed doubling and basically the way this works out is like the, the cost of making more robots and compute, uh, becomes dominated by buying this permit.
And then also the, the, uh, the, the [01:32:00] compute and robots are becoming so valuable and capable that, yeah, we think it works out that basically the, the willingness to pay for these permits grows to be, uh, a very large fraction of the economy. Um, and yeah, effectively works like a tax, um, that is capturing this massive difference between the cost to make the compute and the robots and the economic, um, value that they're able to provide.
And then this leads to, yeah, solving the kind of oth- the whole other problem that is being faced, which is, um, the, uh, kind of job disruption and the fact that, you know, even in this slowdown scenario, we are reaching, um, AIs and robots that dominate humans, um, in a lot of, uh, capabilities and, and therefore there needing to be some kind of solution to, to redistributing [01:33:00] all, all that wealth that by default would all be collected by the- Yeah
the AI and robot owners. Do you
Thomas: wanna say a little bit about ex- exactly how much quantitatively we're saying?
Romeo: Yeah. Yeah. So-
Thomas: Uh, happens in each year.
Romeo: Yeah. So according, um, to, to the, to the modeling of, of this particular scenario, the US, we say that the US, China, and the rest of world negotiate for a 35/25/45% split of, of the cap and trade-
Nevin: 35 to the US
Romeo: 25 China and 45 the rest of the world.
Nevin: Okay.
Romeo: And this is, you know, extremely uncertain. We think, yeah, it'll be a very tense negotiation based on all the, the economic and military leverage at play in 2032. Um, think that- But
Thomas: then how much does this let the US fund in terms of, uh, like, uh, the citizens dividend- Yeah.
Romeo: So then this- ... the sort of
Thomas: UBI equivalent?
Romeo: This leads to, uh, the US if they distribute, um... I think [01:34:00] if they distribute something like 70% of that- their cap and trade revenue t- to US citizens, um, in 2035, they can pay all the US adults a million dollars per year, and then in 2040 it's, it's 10 million. Um-
Thomas: Yeah, and this is real dollars too, uh, where, like, we're trying to adjust for inflation in some sense.
Though, of course, the, um, the, uh, literal way of adjusting for inflation is, is a bit tricky because the relative prices of goods will be changing a huge amount in a kind of interesting way. So in particular, goods like land, uh, which, um, you can't make more of, uh, mostly with more robot labor, um, uh, will be getting much more expensive.
Whereas goods like cars or new apartment buildings, um, which you can build lots of more of with, uh, more robot labor, as well as cognitive labor like, you know, legal papers, uh, or like, you know, lawyer work or, or like, [01:35:00] you know, similar cognitive labor, um, that stuff, which can be done by AIs, is getting massively, massively cheaper.
And so you sort of get this effect where, um, we're sort of... We're trying to average it out, but the actual, like, profile of goods that a consumer can, can purchase even with this, like, millions of dol- With this, like, a million dollars per year or $10 million per year is gonna be many, many, many more, like, cars and, you know, uh, physical labor, like things downstream of physical labor and cognitive labor than you, than you would think naively, but much less in terms of land.
So you would still be sort of a similar amount of land maybe that you own today, but the amount of stuff that you could buy, the amount of travel, et cetera, would be much larger.
Nevin: Yeah. Okay. So just putting ourselves in that world, it's like, all right, you start getting a, a million-dollar check every year for doing nothing.
Um, just thinking this through myself, I... One interesting question here is, like, how many, [01:36:00] uh, jobs will not yet yield to, you know, AI systems and robotics, right? Because le- in... I think in this scenario, you say something like 15% or something by economic output. Is that right?
Romeo: Yeah.
Nevin: In this period. So let's just take that fi- Let's just take that number.
So 15% of goods and services still require, you know, human labor to, to deliver them for whatever reason. Um- Imagine that you happen to be one of the people who is in one of those professions going into this. Um, well now you're getting a million dollar a year check for doing nothing, or you could, you know, get like, you know, 1.2 or whatever your salary was going into this to, to keep providing that service.
Well, you're probably gonna quit, right? A lot of people would quit. Um, and so, and then it seems like the market would obviously adjust. It's like, okay, well we need to pay more for those services. Maybe like a lot more. [01:37:00] Yeah. So it's not just land that would be more expensive. It's like all goods and services that require human labor end up being maybe like 10X more expensive because the, the labor that goes into it is gonna be like 10X more expensive for someone.
You know, 'cause you're gonna have to get paid like $2 or $3 million a year or whatever for it to be worth it- Yeah ... to have that job as opposed to just, you know, backpacking and, and spending time with your loved ones and so on. Yeah. Does that seem right?
Thomas: That seems exactly right to me. Um, so I think it might be interesting to go into the specific jobs we think, which will still be human at that point.
Um, so I think the, the, maybe some important ones are, uh, like- Um, people that are doing things that, uh, you know, people just don't want robots to be doing. So, like priests, philosophers, um, various service jobs. Like, maybe you want, uh, you know, your, your house to be cleaned by a human for some reason as opposed to by a robot [01:38:00] because it, like, means more that way.
Maybe you want your kids to be taught by a human as opposed to by a robot because you just, like, value the process of, of human learning.
Romeo: Yeah.
Thomas: Um-
Romeo: But those will become very expensive.
Thomas: Yes. Those will become super expensive, but, you know, p- people will also be super rich, so it may be afford- it will be affordable for, for people.
Um, another very, very important class is jobs that we don't want robots to do, um, because it would be really bad for society if robots were to do them. Um, two very important examples here are, like, AI safety and alignment research. We want there to be a lot of uplift from, from robots, but we don't want to, at this point at least, like, fully trust that the AIs are doing a good job at the problem of AI alignment.
So, we want there to be a large number of, uh, very talented humans working on, uh, s- specifically problems around AI alignment, as well as other problems that we think the AIs might not be very good at solving.
Romeo: Yeah. Um- Like the verification and security for the entire regime, kinda enforcing [01:39:00] the, the transparency in the data centers, um, enforcing the kind of cap and trade regulations on the robots and compute being followed.
Yeah. These kinda things might r- rely, at the end of the day, still on a lot of human monitoring as part of the kind of safety and control case.
Thomas: And then another, another sort of example is persuasion-based, uh, jobs. So, things like being a newspaper writer or a blogger or a podcaster or, or whatever. Um, I think it might be very scary to be in a regime where you have, uh, AIs, um, that are maybe very cheap and very, very capable at persuasion, uh, or, like, addiction or whatever.
Um, sort of like hyper-optimizing your social media feed or your short-form video feed or whatever to be maximally engaging and maximally likely to keep you addicted and, like, buy whatever ads are being served. And if you could have the equivalent of, like, thousands or tens of thousands of humans, uh, which are [01:40:00] AIs, but you'd have the equivalent of that many people, like, in a giant marketing department, like, hyper-optimizing each little thing that you get shown on your screen to keep you, like, maximally engaged and the- doing whatever, you know, that company or government or whatever wanted.
I think that might- That, you know, might be where we're going with AI, and that might be extremely, extremely scary- Mm-hmm ... once you've got AIs that sort of have the capability to that. I think the existing social media feeds are already in some sense a bit scary. A lot of people are addicted to them.
They're not, you know, it's not the end of the world. But I think in the future, when you've got this sort of level of AIs, it might actually be the end of the world, where you just sort of permanently get a huge fraction of the population addicted. And so I think it might be very important to limit the use of very, very strong AI, uh, you know, AIs for things that are, like, addictive or persuasive or, or whatever, and have, you know, the short-form video producers still be humans, the bloggers still be humans, et cetera.
And that might be a, a reasonably large fraction of the population. You know, a lot of people might [01:41:00] want to read a lot of stuff and watch a lot of videos. Uh, and so there might be a lot of demand for, um, a bunch of human labor here. And I think it might be really good for society if humans were to do that labor, and we didn't get sort of trapped in a AI doom loop where almost all of our content that we're being served is AI-generated.
Because it seems like that might just be bad for people's ability to think about stuff and people's- Yeah ... uh, you know, endorsed preferences might come apart a lot from their, you know, what they actually do, their revealed preferences.
Romeo: Yeah.
Thomas: Um,
Romeo: yeah. Anything to add there? I d- I do think then, like, the way it might work out is even with a lot of these legally protected kind of domains and jobs, um- There might still be a lot of, like, AI uplift, um, and kind of the, the number of humans required to do the monitoring and to do the, um, to provide, like, really, really great legal services, um, to do, or, [01:42:00] a, a bunch- Yeah
of the podcasting and stuff might, might just kind of go down over time, um, as there are, there are ways that are optimized to like, you know, keep, keep the humans in control in, in the way that meets the, the kind of relevant regulations here, but then still allows in the background there to be a lot of com- ca- like, s- help from the AIs.
And I think that might kind of net out to a situation. Uh, in, in, in, in the scenario, the way that, the way that I've modeled it, it kind of leads to a situation where in, by like 2035, there's like a third of Americans working, I think, or a quarter or something like that. Um, and then by the end, it's more like 10%, uh, by, by 2040.
And yeah, I think basically the, this all works out where, like, the people that, you know, you can still get a job in some of these areas if you do want it. The [01:43:00] wages there are four or five times the, the dividend.
Nevin: Meaning four or five million dollars.
Romeo: Yeah, if it's in like 2035. Um-
Nevin: And that's in like today's dollars- Yeah
as best as we can sort of- Yeah, as best as- ... splinterize and think about what that- Yeah ... would even mean.
Romeo: Yeah. Yeah.
Nevin: Okay. Um, and so, um, you, you guys are revealing your extreme bullishness on the software and hardware with this answer because you haven't mentioned a single thing really that's like, "Yeah, we still need humans to do this for competence reasons."
Right? Like, you, you sort of seem to think that like at this point in the story, the AIs and robots could do everything functionally better. Yeah. It, you know, it's like we, no, no matter whether it's, you know, servicing your airplane, uh, you know, or whatever the sort of most complicated executive roles are within financial companies, you're like, "Just AI all the way" is, is your view.
Yeah.
Romeo: We have, we have this milestone that we [01:44:00] call a top expert dominating AI, and it's kind of like a very strong definition for AGI. Um, and- In this scenario, we, we're imagining that to be hit, uh, in, in 2035. By default, we think, you know, the, the intelligence explosion, the dynamics we were talking about earlier, like months to, to a low number of years between, um, kind of a very low definition where maybe we're just kind of automating coding to then this milestone of like a top expert dominating AI.
Um- It's,
Thomas: it's maybe worth clarifying sort of the v- like our overall views and then this specific scenario.
Romeo: Yeah.
Thomas: Where like this specific scenario is this like very optimistic vision which probably won't happen, but we hope will happen, um, where, uh, where sort of, uh, we would have had an intelligence explosion and this level of AI happen in 2030, but thanks to all of these governance mechanisms we talked about, it gets [01:45:00] extended, and this particular milestone gets reached in 2035, and then some future milestones like superintelligence get reached in 2040.
Um, our default view is, like out of the scenario context, is very uncertain about the timeline. Uh, I think that it's still very plausible that you get something like AI 2027, which was, you know, you get AIs that are better than all the humans at everything in literally 2027, next year. That still seems plausible to me.
I think it probably won't happen. I think it's maybe ten, ten, twenty percent likely. Um, if you, uh... And then I think what happens in this scenario is like a 2030 intelligence explosion by default. That seems roughly my median, so fifty-fifty that it has happened by then versus not. And then I think there's a bunch of chance that, like, you know, the near term scaling doesn't work out, and you have to wait until the 2030s or maybe even the 2040s before you get sort of extreme levels of AI capability.
It's really, I think, it's, it's really hard, you know, people who've tried to predict the future in the past have historically gotten it very [01:46:00] wrong. Um, so I think w- we tend to try to have a very agnostic view about... Well, we have some, some opinions, but we try to have a broad distribution- Yeah ... of when we sort of make these types of, of general predictions, um, uh, to like sort of be appropriately unconfident.
Um-
Romeo: Yeah.
Thomas: Yeah.
Romeo: We, we, we don't think there's anything like fundamental preventing this though. Like, yeah, I guess like it, it being possible to, to make an AI that cognitively-
Thomas: Yeah ...
Romeo: and then later kind of physically, um, uh, you know- Dominates
Thomas: humans. Yeah ...
Romeo: dominates humans. Like-
Thomas: Yeah, it seems totally doable.
It's just a question of when, and maybe it's a long time, but we think probably not. Yeah. I think probably we're, we're getting there.
Nevin: Okay. So we've talked about, you know, this hypothetical that, um- Goods and services produced by AI and robots become super cheap. [01:47:00] Um, uh, you know, land and, um, maybe certain services that are still human-dependent become very expensive.
Um, what happens to, like, the value of companies? You know, so if, if you were to invest through this with a normal investor motivation of, you know, getting a good return and having wealth that still exists on the other side of this crazy, um, situation, not even, not even mentioning, not even thinking about, like, you know, the singularity possible future.
Just, just up until this kind of point, um, what do you invest in? You know, like, what, which companies do you expect to do well? Or is that kind of unknowable, unknowable from your perspective?
Romeo: This is a kind of pretty unique scenario, um, or un- kind of unlikely, I think, from our perspective. And
Nevin: let's analyze it just from the assumption that this is how it plays out.
Okay, yeah. You know, we're not making a prediction. We're not, we're obviously not giving investment advice here. Yeah. [01:48:00] Um, but just, yeah, i- if the world does, uh, traverse this plan- Yeah ... A scenario, what would be- So- ... the trade expression?
Romeo: So I think because of the, um, transparency measures with AI, I think this creates an AI industry that is mu- highly competitive and more distributed and looks more like some of the global industries today that are highly competitive and make kind of very slim profit margins.
Because of the, the level of growth, though, that they're able to, to drive, I, I still do think that you, um, have AI companies that are involved and, and robotics companies, um, that are basically driving this kind of double- doubling of the economy every year. Um, and that actually AI and robot companies [01:49:00] might do slightly worse than the overall economy because as industries, they are growing, um...
The, the, the overall industry is growing kind of similarly to the entire economy. It's in fact what is driving that level of growth. But it's becoming less concentrated over time. Mm-hmm. And so the, the kind of leading companies, um, actually, like, diverge a little bit from the overall industry's trend. Um, and I think what you actually get in this scenario is you get, like, uh, interest rates in the entire economy tracking, um, the level of growth.
So it will kind of be... I think it'll kind of be a situation where, um- Every- everybody's kind of savings by default will be [01:50:00] growing with the whole economy. And yeah, and, and, and it might kind of be a situation where, like, the level -- the overall level of, of growth in the economy is now so driven by this very, uh, multipolar AI and robot industry, and it's very hard for there to be, like, large, um, alpha in, in investment, I think is my default view.
And it might be, like, a pretty, um, unique period historically from this perspective. Um, because, like, yeah, the, the, the economy is dominated by AI and robots, and both those industries have, like, research transparency. And so it's, like, almost by definition very hard for there to be, um, large, [01:51:00] like, speculative differences in, in, in-
Nevin: What about-
in, in gains ... the whole infrastructure layer that powers all of this, right? Yeah. Because, you know, doesn't, doesn't, um, you know, how do NVIDIA and TSMC and all those types of companies do in this scenario?
Romeo: Yeah. So I, I do think potentially at that layer, at least initially, there's still a lot of, um- Important IP and moats and things, uh, like that.
But very quickly in this scenario, the amount of kind of compute manufacturing capacity that is newly built, um, uh, becomes, it, it, it becomes like within two years that like 90% of the, you know, compute manufacturing capa- capacity is all newly built in the last two years.
Nevin: Because of the robotics explosion.
Romeo: Because of the, yeah, because of the growth rate being fast, [01:52:00] um, b- being this fast. So, uh, and then I think y- you might want to have some of the similar, um, transparency things in place for the AI hardware industry because of some of the, um, potential kind of analogous risks around there being, um, kind of destabilizing amounts of progress there, um, potentially where you, you don't want people to, you know, inven- you don't want there to be hyper-efficient, easy to manufacture AI compute designs, for example.
But even, even putting that aside, um, I think the rate of progress being so fast, um, I think might ... And, and, and there being like so [01:53:00] much AI and robot labor changing the picture and changing, um, the way that the chips are made and potentially also kind of the access to a- the AIs and robots being spread across more companies, meaning that like, you know, now instead of having just Anthropic or just OpenAI having the ability to make really, really great new hardware designs and, and you know, even potentially like, um, overtaking Nvidia in that domain because their internal AI and robotics capabilities have this large gap, um- Hmm
over the rest of the world. That's not gonna happen. Like all of the, all, like there's many, many frontier companies with these capabilities, and I don't think that like the, the moats even in, in some of the hardware areas are very durable in this scenario where the future hardware capabilities will [01:54:00] come, will be downstream of AI and robot capabilities.
Um-
Nevin: Interesting. I did a little bit of a dive recently to think about like as AI technology progresses What will happen to, uh, different types of parties' ability to, to compete in financial markets just in d- you know, even, like, short-term trading. Mm-hmm, mm-hmm. And my conclusion was that the top funds with the most money are going to continue to stay ahead because they are investing the most money.
They're already training their own foundation models on, you know, financial time series data and so on. Um, and so, uh, you know, I went into it wondering, is this gonna be sort of a, you know, leveling effect where suddenly there's no alpha and everyone's kind of on the same playing field? The more I thought through it, the more I was like, "No, it seems like the, the, the entities that are [01:55:00] most advanced, most knowledgeable, have the most money to invest, et cetera, are gonna continue to stay out ahead, at, at least in the near term."
Who knows what happens if you have, like, superintelligence or whatever. So, um, I can kind of a- apply that metaphorically to thinking about TSMC and other chip, you know, the chip designers and so on. Mm-hmm. Where it's like, yeah, okay, um, even-- I mean, if you have the transparency thing, that changes things eventually.
Yeah. Uh, but if we're talking about a 5, 10-year period, it still seems like if you're getting the automation on the software level and the hardware level- Mm-hmm ... they're gonna be applying it themselves the most, like, by way more than any other company that hasn't gotten into that business. So it seems like they're still gonna stay ahead and become, like, enormously valuable.
But, but yeah, maybe at some point, like, the stuff is so general that it's like, well, any company can kind of do anything. Is that, is that sort of like- Yeah ... where this ultimately leads in your intuition?
Romeo: I think that's what I'm imagining is that that transition is [01:56:00] accelerated and things change enough such that, um, yeah, like the, the progress is so AI and robotics like capabilities driven that it, that, that, that transition does happen kinda quickly.
I do think that by default, um, the situation is more of like, yeah, of like a, a snowball effect for, for the larger, for the players that are ahead. Um, and yeah, I think that because in particular the two kind of industries in the, the AI and robotics industries, uh, that will drive so much of the progress if those are not heavily concentrated, um, I do think it would, you know, pull all the other industries kind of towards the same kind of level of [01:57:00] concentration of the AI and robotics industries.
But yeah, there might be some kind of stickiness or, or durable, um, effects because of the things you were mentioning of, like, just already having some of the relevant infrastructure. I just think, you know, the relevant infrastructure and talent will change very quickly- Mm-hmm ... in this scenario.
Nevin: Mm-hmm.
Romeo: Um, yeah.
And- Yeah. And then I think maybe, you know, m- making the comparison to what our default worldview is, is like, uh, you know, it's, it's, it's kind of like in, in, in the default world we expect there to be even just one or, like, a, a couple of companies that, that, you know, do this internal, um, R- and- R&D recursive self-improvement.
And, you know, the, the, the [01:58:00] wealth and economic power just concentrates extremely heavily. Um, and it's like, yeah, like a, almost like a, it's a winner-takes-all situation and I think that's kind of what I think we expect by default. And so both of these, I think, are pretty difficult to kind of reason about from, like, an investing perspective.
I think, um-
Nevin: But it sounds like you're, you're, you lean towards the idea that someone, uh, would maybe do best by just buying the market overall, just a super broad index of everything-
Romeo: Yeah ...
Nevin: uh, because it's very hard to predict and value will accrue across the entire economy in some shape or form.
Romeo: Yeah.
That's what I'd expect for kind of Plan A world, I think. Um-
Thomas: Just one thing I might add is that I would guess that the market is not sufficiently AGI-pilled enough. And so because of that, I would imagine that they, um, that in Plan A, the market in 2030 would [01:59:00] not price in the doublings, like, of the overall economy- Mm
that will soon be coming. And so, uh, y- y- it, it might be the case that one could buy, uh, like, options that are, like, uh, uh, like, basically o- options that are, um, sort of represent the ability to buy if, uh- Yeah ... if the market were to increase a huge, huge amount. Mm-hmm. And people might feel willing to sell you those at really, really cheap prices because they're, like, think it's basically implausible that the overall economy will, like, double- Yeah
uh, ev- every year for a couple years in a row. Yeah. And so, um, generally, in bo- in both Plan A and generally, I would expect that, yeah, I would, I would expect basically the market to underprice, uh, how big, big of a deal AI is gonna be.
Nevin: But it's interesting because right now so many investors in the world are, like, voraciously trying to understand AI and the whole infrastructure build-out in particular because, you know, s- so prices have gone up so much, a bunch [02:00:00] of these companies.
Um, but s- kind of this counterintuitive conclusion that, like, actually if you just invest in, like, a handful of these companies that are making AI infrastructure, at least if we're talking about this, like, 10, 15-year timescale, you might do very poorly compared to just owning an index, uh, across, across everything.
Especially if these policy, uh, directions were to come to pass And cause certain AI related companies to actually be way less valuable than the market is currently anticipating because it's kinda taken away, you know, some of their, some of their, um, trade secrets effectively.
Thomas: Yep.
Romeo: Yeah. I think that, that might be right.
Yeah. I think- With the, with the floor or, like, the, the absolute level of the economy growing so much, I think, like, you would, you would still do well. But I agree, like the relative shift- Yeah ... if something like Plan A happened would make the kind of AI specific things relatively less attractive. Um,
Nevin: yeah.
So [02:01:00] one huge question right now i- for all everyone who's looking at all this stuff is, you know, is this the year 2000? Are we in like sort of a tech bubble? So I have a couple things for you guys to look at for reference. We can put these on the screen. So the first sheet of paper is the Korean stock market index, and you can see in the first plot, that's the entire plot since 2008, right?
So it's gone completely nuts in the past year.
Romeo: Yeah.
Nevin: Uh, but also we've dropped off from the peak significantly. So the other charts show, I think the one year view and the one month view. Just over the last month we're down 25%, uh, for the entire Korean stock market. You know, having been up, uh, uh, you know, I guess we're up 114% still on a one year timeline.
Mm-hmm. Um, so that's kind of, that's one lens and that's, you know, the memory maker Samsung and SK Hynix are largely driving that in, in the Korean market. But that's the entire Korean stock [02:02:00] market, that's not just those two companies, right? Yeah. Imagine if the US entire S&P 500 looked like this chart. It's pretty insane for Korean investors right now.
Yeah. Um, and then the second chart is the Reserve AI infrastructure DTF, uh, with the ticker build out. This is basically an index of the top 25 US traded companies, um, that are, you know, thematically related to the whole build out, right? So we start with Nvidia and we go all the way down, you know, to some of the more, uh, obscure for, for someone who doesn't know about the AI infrastructure build out, um, you know, like Lumentum and, and Coherent.
And again, if you look at this chart over, this is a one year chart, um, we're up, you know, 95% on the year, but we're also down, you know, uh, quite a, quite a bit from the peak over the past month. Mm-hmm. And so at this moment, if you look at, you know, uh, people on [02:03:00] X who are talking about this trade, um, you know, people are posting s- screenshots of their portfolios down, you know, 20, 30, 40, 50%.
Mm-hmm. Um, and you know, everyone's just trying to figure out like, well, you know, yeah, is it like the year 2000 for the tech bubble? Is it like, you know, 2017 for Bitcoin where we all got like way ahead of ourselves and this is all gonna come down a lot? If, you know, so is it gonna come down and stay down for five, 10 years?
You know, was this all just like really a financial bubble or- You know, is it gonna... You know, with, with crypto we have these four-year cycles of it goes up a huge amount, goes down quite, quite a bit. You know, not as much. You know, and every four years you, you make al- new all-time highs. Uh, or is it gonna be something totally novel and different that doesn't look like any of these things that people tend to pattern match to?
So, um, I thought it would be fun for us to just try to draw the chart. And since I'm asking you to stick your neck out and [02:04:00] draw a chart- Oh, boy ... I'm, I'll, I will start. So I want you to... And you can think about it while I- Yeah ... figure out what my answer's gonna be. I-
Thomas: Sorry, for the Plan A scenario?
Nevin: S- yes. Okay.
So in the Plan A world, uh, so this is, this is, this is a condensed version of this chart, right? Yeah. The top 25 AI index, uh- Yeah ... uh, uh, the, the build-out product we call it. So you see it, you know, going up, uh, and then we've retraced a little bit. And this is over one year, okay? And we're gonna draw the chart over the next 10 years of what we think could happen.
And, you know, this, the, the axis goes to 650% from, uh- Oh,
Thomas: it's not, it's not a log scale? Yeah, that's gonna be an issue I think.
Nevin: Yeah, let's make it a log scale.
Thomas: Yeah.
Nevin: For fun, let's draw the chart on the linear scale first.
Thomas: Okay.
Nevin: And then we can, uh, and then we can do it on the log scale. Um, so yeah. But so take a moment to think about these companies- Mm-hmm
you know, that are part of the build-out right now, and think about what might happen over this time period. Um, and then I'm gonna draw sort of [02:05:00] roughly the pattern that I'm, that I'm expecting.
So you can start on the linear scale. Okay, yeah. I mean, I'm just making it up, of course. Yeah. This is all just a- So I would've
Thomas: thought that it
Nevin: basically just- ... sketching exercise.
Romeo: So, okay. So this is-
Thomas: So
Romeo: yeah,
Thomas: how exactly big is this?
Romeo: And th- this is, like, a 10X roughly, right?
Thomas: Okay.
Romeo: So 10X over, like, eight years.
Okay. I guess I could do it, like, dotted, um, [02:06:00] I guess. I
Thomas: think we could, we could just do the dotted that represents the, like, average case or something, and then you add a martingale on top of it or something.
Romeo: Yeah, yeah. I think the average would default to, like, a doubling-ish per year. Seem reasonable?
Thomas: That seems reasonable, yeah.
And
Romeo: then, like, like this or something-
Thomas: Yeah ...
Nevin: on average.
Thomas: Yeah, so that, on average, like, that plus, uh, some random walk. Yeah.
Nevin: And how, how much volatility do you expect?
Romeo: Um.
Nevin: Inasmuch as, like, inasmuch as volatility- I, yeah ... is the world sort of oscillating around in what it thinks is gonna happen. Yeah.
Romeo: I th- I think, like, swings like this where you get, like, that we've just seen where you get, like, a month of gains erased.
But, you know, when, when you're growing so fast, like, a month of g- gains erased looks like a massive- Mm ... you know. Historically, it's like a, it's like a recession. Or, or what are they ... Yeah, like, I think, um, you know, it's like [02:07:00] correction is 10%, and, and, um... Yeah. Anyway, I think basically, like, those kind of swings will even get more and more pronounced because, um, you know, a- as, as the growth rate is larger, those, like, one to two-month swings, I think in, like, calendar time, you know, become, like, a bigger percentage hit.
Yeah. But I don't know. I think, um, I can draw, like, just kind of ran- randomly, like-
Thomas: The, the variance you drew seems, like, naively kind of reasonable to me. Yeah,
Romeo: yeah. Like, not too much
Thomas: or something. But I should say I, I am nowhere near, like, an expert in this sort of thing.
Romeo: Yeah. I don't know. Yeah, I think that it would be, like, it'll actually
The trend will actually, like, accelerate, um, during the intelligence explosion.
Nevin: Right. So this is a lower bound because this is slowed down by various things, whereas-
Thomas: Well, I think the graph- Well, this actually isn't yet because it doesn't get to 2029.
Nevin: Okay. Yeah. That will come into play here.
Thomas: This is what happens after this.
Yeah.
Romeo: I think it can, like, s- it can, like, slow down. It could, like, slow down [02:08:00] until there's an intelligence explosion. Um-
Nevin: Okay. So let's do, um-
Thomas: Maybe while you're, while you're going I'll, I'll just say a quick thing. I think my- The main thing I want people to take away is that AI is gonna be a much bigger deal than I think almost everyone thinks.
I- in particular, I think it's gonna, like, be much, much more transformative than people expect. Mm-hmm. Uh, I think the current market has been proven wrong time and time again for the last five years in being, uh, not bullish enough on AI, and I'm very ... And I think that's very related to the fact that our governance of AI is, like, not pricing in the full effects whatsoever.
Hmm. So I think I overall want sort of, um, discussion as a whole to, like, take the impacts of, like, extremely tr- societally transformative AI much more seriously, which is why I think it's, like, good for people to be aware that this is what's coming. On the other hand, I am ... I have some reservations about people aggressively accelerating the AI build-out.
In particular, I think that it's, like, better [02:09:00] if we can slow things down somewhat. And so just ... It's sort of, like, an awkward position where I want people to be aware of stuff more, but I also don't want there to be, like, massive amount of cash flowing into the AI infrastructure build-out, despite it being a good investment to do so, obviously.
Nevin: Yeah. It seems pretty inevitable. It seems, you know, virtually impossible to, to, to, to ... for capital to not flow into-
Thomas: Indeed ...
Nevin: um, and so I think it is what it is, and, uh, you know, this stuff is all gonna be extremely well-funded. But also, it's, you know, j- just as the, you know, stock market reaction caused people to pay attention to COVID at the beginning, it's like, wow, the stock market is showing that- Mm-hmm
this is being taken seriously. That was a big moment. Mm-hmm. Um, and I think we're having that this year for- Yeah ... for AI, where it's like, holy crap, like, the market says this is real. Yeah. Um, and so I think, uh, it is a, [02:10:00] it is a way to, um, for people to, to take this seriously, I think.
Romeo: Mm-hmm.
Nevin: And you're- y- and when you draw these curves, you're not thinking about just the market overall, you're thinking about, like, so if someone- Yeah
were to hold this AI- Yeah ... basket, uh, you don't think that there's, that it's likely that that's gonna become worthless because these particular companies will become irrelevant, like- Oh,
Romeo: yeah. I guess I should have, um, done that, like, in the context of Plan A, um, maybe these start to go below the overall trend.
Nevin: That's, that's part of my question here. Yeah, yeah,
Romeo: yeah, yeah. Yeah. So- So-
Nevin: I guess I could do, like- So maybe draw two curves. Yeah. One is, like, the S&P 500, and the other is these- Yeah ... AI infrastructure companies. Yeah.
Thomas: Another, another interesting effect I was thinking maybe is in, I think the variance will be higher pre-Plan A.
Yeah. And then once you're on a more capped economy, I feel like the variance w- would decrease a huge amount. Yeah, yeah. Because it's, like, very predictable to all market holders- Okay ... that, like, there'll be this amount- Yeah ... of build out- Okay ... in the next year.
Romeo: Yeah. Um- And then for [02:11:00] this ... Yeah. Yeah. If this is like S&P, then I think the, the, um
It'll maybe be, like, it will have been, like, a little slower- Yeah ... at first or something. But then, like, I do think pretty soon they kind of become very, very similar. Like, the, the kind of- Okay ... like AI ETFs and then, like, the S&P will kind of-
Thomas: Yeah ...
Romeo: be extremely similar. And
Thomas: well, interesting. So the S&P should be below this trend probably for the near future.
Yeah,
Romeo: yeah. But for the near future be lower.
Thomas: Yeah.
Romeo: Um, so yeah. So I guess I can do, like, these spec- I think these specific stocks in Plan A would start to, like, diverge after the deal in 2030 from, like, um, to be on, like, a shallower trend. Basically, that is just, like, different in how much less concentration there's gonna be in the top players.
I think by, like ... Yeah. But, like, towards the end, I think it's gonna be, [02:12:00] like, something like 10x less, um, concentrated.
Nevin: So it'd be more like
Romeo: this
Nevin: or something.
Romeo: So I think this would be, like, yeah, the overall, um ... So let's switch it where, like, before, um, before this point, this line, um, this squiggly line that we already drew is, like, the, the AI specific stocks.
Nevin: Yep.
Romeo: And then it becomes like this one, but, like- Yeah ... very low variance or something. And then, like, the, the S&P one is the w- it would be, like, flatter first, and then kind of more on this trend.
Nevin: Interesting. Yeah. So broadly, m- possibly, uh, someone would do better holding the AI basket until the deal- Yeah ... and better holding the entire market after the deal, in theory.
Thomas: Yeah. Seems reasonable to me. Yeah.
Nevin: That's an interesting outcome. Um, [02:13:00] and um- Uh, but, but, but it's sort of, it's sort of, you know, sort of the simplest takeaway if, if you're right is kind of like just making sure you're in the market generally, right? It's like the way you really lose here is, uh, if your, if your money is in cash.
Yeah. Yes. Um, and, you know, actually, I, I'm gonna force you to draw one more line. What about, um, land, right? You talked about- Oh, yeah ... land sort of completely- Yeah ... completely skyrocketing in value. Oh,
Thomas: we made some estimates of this. Yeah. Do, do you remember the, do you remember the, the quantitative results? So
Nevin: I think
Romeo: it, it
Nevin: basically- And which kinds of land?
Yeah, so- Urban land, rural land? Like, are they... How different are they?
Romeo: So I think-
Thomas: We have projections.
Romeo: Yeah. There's, there's a bunch of different-
Thomas: So it'll be basically flat- ...
Romeo: considerations here ...
Thomas: until the deal, so that part's easy. Yeah. So I can draw that.
Romeo: I think that basically what, what happens is like- Um, I mean, the situation [02:14:00] today is that only about 1% of land, um, is urban land.
And so it'll be hugely... But it'll be, like, hugely, hugely dependent on, I think, regulations, um, about how valuable other types of land are. And, like, I think you'll get a situation where rural land, um, or, or just in general because of the r- the robot, uh, uh, labor, the relevant robot labor, become extremely...
Like, the, the value of what you can cheaply turn la- do with land and turn land into, um, will, will go up by a lot. And then if you're allowed to build, in places where you are allowed to build, um, I, yeah, I think there, there'd just be exploding demand for, f- for [02:15:00] land because it'll be the most scarce part of this.
Um-
Thomas: Like, basically you've got these massive new robot mega cities that the robots- Yeah ... are building-
Romeo: However- ...
Thomas: in the- Yeah ... particular regulatory regimes where- Right ... like the regulatory zones where that's allowed. You have a
Nevin: population where most people don't need to have a job. Yeah. Yeah. So they can move anywhere and do whatever-
Romeo: Yeah
Nevin: fun stuff they wanna do, and you can cheaply pay a robot army to construct any building you can imagine.
Thomas: Yeah.
Romeo: And, and probably transportation has also gotten a lot better and cheaper- Mm. Mm ... um, as part of this. A, a, you know, putting aside regulations. Um, so I think, however, there's another part of this which is that, like, the actual supply of land, there's, like, this big kind of overhang of supply where there's, like, a ton of land that is not being used right now.
Nevin: Mm.
Romeo: There's an economics explorer that we put out together with the scenario, [02:16:00] and one of the tabs has concrete, like, land rental prices for different types of land in the US.
Nevin: Okay. We can, we can- And you can just- We can put that on screen now. So we can... Yeah. For people to sort of get- I can, I can get that up
a taste, and they can download it if they're curious.
Romeo: Yeah. And, and there are some specific assumptions there about the different, um, demand and supply for the different types of land. And so, like, the way it plays out, you c- you can also, like, fiddle with the parameters there. Um, but I think from the top of my head, it will have, like, maybe a slightly steeper slope than this, but then because of the supply, it will look a bit, um,
Nevin: it, it, it would, like, bring it down by maybe 10X.
So I'll just draw-
Romeo: Something that would cross like three, a bit more than three orders of magnitude. So get to like here
Yeah [02:17:00] Slightly steeper I think is my best guess. Um-
Thomas: For like all land averaged overall.
Nevin: Right. And then, yeah, that'll be a pretty interesting distribution probably of different-
Thomas: Yeah ... how different types of areas. And then maybe you've got like specific SEZs that grow much faster than the overall-
Nevin: Right.
Thomas: Yeah ... market, and then a bunch of like super regulated zones which don't, and a bunch of like- Yeah ... places like New York where maybe you get positional goods, reasons for why it, uh, accelerates even more.
Romeo: Yeah.
Nevin: Okay.
Thomas: Yeah
Nevin: And then, um, with this idea of, you know, I think when, when people hear the idea of like, "Oh, yeah, well, you know, we'll just put all of our data centers in Mongolia," the- I mean, there's kind of just like a, "That's crazy.
We would never do that"- Yeah ... um, intuitive reaction. Um, uh, but also I wonder, like, you know, for people who are investing in the build out of all, all these data centers in the US right now, if we did [02:18:00] adopt this policy where it's like, "No, we're not gonna build them in the US, we're gonna build them in some other place," would that make a big difference to that investment thesis or would it not really matter?
'Cause it's like, well, who cares where your data center is, you know? It's really the fact, it's who owns it and what-
Thomas: Yeah ...
Nevin: value it produces- Yeah ... so it doesn't really matter.
Romeo: I, I think it doesn't matter. I think a large percentage of the investment is going towards the, the IT infrastructure, like the, the chips and the servers and the networking.
Thomas: Like, the building is a tiny fraction of the overall- Yeah ... CapEx of the data center. It's, like, almost entirely, like, actually buying the GPUs and then a bit- Yeah ... the other equipment.
Nevin: Right.
Romeo: Yeah. And then ob- I think, yeah, yeah, roughly like 70% is what goes in the data center. Then there's just, like, labor, land.
The same power companies can sell their power things, um, if the data center's being built in Mongolia versus the US. Maybe, like maybe, you know, especially if it's, like, the nat- natural gas generators or, or whatever, or the [02:19:00] solar panels. Maybe the, the labor is, is the thing that changes. Like, yeah, maybe you're trying to hire more people or maybe you're still even hiring the same kind of people but they're just moving to Mongolia for, for a while for the project.
So yeah, I don't, I don't think overall it changes very much.
Nevin: Okay. All right. Since we're short on time, uh, I'll give you guys a few minutes to cover any remaining big points in the scenario and, and policy that we didn't hit. Uh, and then I wanna ask you just a few fun questions at the end.
Thomas: Great. Cool. Yeah, so I think the main one I wanna talk about was, um, the sort of, uh, capability scaling strategy in a coordinated regime.
So, um, in our view, um, sort of we have this thing called Plan A, which we've been talking about. Uh, a different, um, policy perspective is, is that which, like, MIRI Eliezer Yudkowsky has, which is more like shut it [02:20:00] all down now. Um, and these would lead to very different curves, where sort of Eliezer's curve would lead to more like, well, we shut it all down now.
The capabilities curve is basically just flat over time. Um, our preferred policy agenda is more like- Uh, scale, scale, scale as fast as you possibly can make safe. So not as fast as possible. That would be more like a, a, you know, pretty fast singularity. Ours is like a pretty steep line, but not fully steep.
And then, uh, when you're at this point where you, like, really can't scale anymore without it being extremely unsafe, posing a huge amount of misalignment risk, then you do just, like, pause for as long as you can. And the advantage of our curve... So, like, our curve is more like up and then flat, and then Eliezer's curve is more just like flat.
The advantage of ours is that you get more time with smarter AIs, and you can use those AIs to do all sorts of useful stuff. Um, so all of this economic transformation that we were talking about is downstream of having the really smart [02:21:00] AIs, which can be used to, like, you know, solve a bunch of the world's problems.
Um, some particular applications of AIs that we're very excited by are... Uh, so obviously one is AI for AI alignment, so using AI to solve the alignment problem. Um, there's also AI for verification, governance, and epistemics, which are sort of related categories. Um- Uh, so like AIs for governance is sort of just like using AIs in our existing government and regulatory bodies to make them more effective.
Where like right now, I think governments are often, you know, pretty incompetent and pretty bad at doing things. And, uh, I think, uh, there's a hope that if you have pretty good AIs that have reasonable alignment properties in them, you could, uh, rely on them for a bunch of these sorts of decision-making or at least get a bunch of uplift out of the AIs to hopefully make governments sort of operate more efficiently and be more able to do something as complicated as Plan A successfully.
Um, and then another is just this verification angle where Romeo earlier talked a bunch about the [02:22:00] different, um, ways that the US and China might be able to, uh, make agreements without trusting each other by using various forms of technology that could let them be confident that the data centers are doing what they say they're doing.
Um, and AIs could be very helpful at basically making better versions of those technologies, um, and, and a whole bunch of other things that, that I haven't mentioned. Um, and the reason why it's important to make this progress as fast as possible, as opposed to just waiting for a really long period of time, is that we're worried that sort of any sort of AI regulatory regime that majorly impacts things will sort of have a timer associated with it, and it won't be that stable.
Because historically, governments are very fickle. Governments, you know, a new president will be elected and will reverse everything the predecessor did. Uh, you know, countries will just change their minds about stuff, will pull out of the deal. And so, uh, one of the central sort of wh- overall problems we're very worried about is you basically run out of time, your deal breaks down, and then you go back to [02:23:00] racing.
And if that's the case, we want to have made as much progress on these key, key issues as possible. And also, we want to sort of improve the technology that would make it less likely the deal breaks down in the first place, like the verification stuff that I mentioned earlier. So all in all, I think we're pretty excited about this view of like go as fast as possible while still being safe, which in practice will, you know, be, be somewhat slower than as fast as possible.
Um- Uh, because we think that's like the overall sort of globally optimal strategy, taking into account this risk of the deal breaking down. Um, there's a bunch of plausible counterarguments we get into. Maybe the most important one is, uh, this issue of like, how do you know exactly where to stop? How, how do you know what's safe and what's not safe?
Um, where's the capacity for that coming from? And I think, uh, that's one of our biggest worries with the overall plan. We think it's a very legitimate criticism, and we're very, very worried that the government won't [02:24:00] do a good job at this. And this is one of the reasons why we're excited now about the government trying to increase its capacity on AI and trying to increase its sort of understanding of what's going on.
Yeah. As well as, uh, we're sort of hoping that the government can rely on these, this like third-party ecosystem of auditors, um, you know, both nonprofit and for-profit, to sort of supplement the government expertise as much as possible. Um, but if this doesn't work out, then you might be forced into something that's much more conservative and looks much more like the MIRI plan than our plan because you simply don't have the regulatory capacity, the regulatory government capacity to implement anything like our plan.
Um, yeah. Did, Romeo, you wanna add anything to that?
Nevin: Um, I'm gonna keep us moving 'cause we're short on time. Yeah. That was great. Yeah. Um, the... So, um, you mentioned Eliezer Yudkowsky, and for anyone who doesn't know, Eliezer Yudkowsky is kind of the OG, very safety-minded person, uh, who's been sort of shouting for [02:25:00] the longest about the potential big risks here, and just published a book literally called If Anyone Builds It, Everyone Dies.
So he has this very extreme view about, uh, the potential risk. Um, interestingly, the... There's this document he published in 1999 that's kind of, uh, a hidden Easter egg namesake for this show. It's called Staring Into the Singularity, in which he wrote, "Right now, every human being on this planet who has heard of the singularity has exactly one legitimate concern.
How do we get there as fast as possible? What happens afterwards is not our problem. Our sole responsibility is to produce something smarter than we are. Any problems beyond that are not ours to solve." So Eliezer in 1999 was just like balls to the wall, acceleration, get there as fast as possible. That's all that matters.
Um, and that's what he was devoting his full energy towards. Um, and, and then he flipped to be, you know, sort of the polar opposite [02:26:00] of that, uh, as he thought about the problem over the subsequent few years. And I bring this up because, you know, um, it's, it's very bold of you guys to come out and, and talk about the way you see all of this going.
It's a very controversial view. Many people listening to this are gonna think that you're nuts. Um, and in this, uh, in, in this world today, it's like we're still having a really hard time talking about these issues and coming to anything close to agreement about where this technology is gonna go. And, and that's the context in which we have to answer these crazy governance questions is, you know, people thinking that each other are, like, crazy.
Um- Interestingly, just yesterday, uh, Xi Jinping in, uh, addressed the, the, what's called the World AI Conference, but it was a conference hosted in China, sort of China's flagship AI event. And one of the things he said in this address was, uh, [02:27:00] "We must make AI's oversight and governance precise and effective, and constantly refine measures to forestall loss of control."
It's a translation from Chinese to English, so I don't know exactly what the Chinese words were and whether they map perfectly to the sort of term of art, loss of control, that's come to be used in US AI policy circles that you guys use as well. Um, but it sure sounds like, uh, he's making a direct statement about, like, there's this risk of loss of control, this, referring to the same thing that Eliezer Yudkowsky refers to, and you know, when he's freaking out about it in his book.
And so That, if that's true, that's a really interesting development in the sort of discourse landscape around all this, right? It's like, you know, um, and it's kind of interesting timing that you guys happen to have just published, you know, this Plan A where you're, you're taking this all seriously and saying, "We need these extreme measures.
We need to negotiate with China." And then Xi comes out [02:28:00] and says like, "Hey, we need to, we need to address this risk of loss of control." I'm assuming you guys have listened to this speech.
Thomas: I've read it. I haven't listened to it.
Nevin: Read it?
Thomas: Yeah. 'Cause it's in Chinese.
Nevin: Same thing. Um, and so I'm curious, you know, do you think that, uh...
How do you, how do you interpret it? Do you think, uh, it's a very positive sign for the, the possibility of negotiating something on this front? Does it not really change your anticipations at all based on pre this particular speech? Um, how should the world make sense of what he meant- Yeah ... with, with that statement and, and with his overall, um, uh, uh, speech?
I'll, I'll read one more quote. He says, "We should jointly oppose overstretching the national security concept in the field of AI and placing one country's security over that of others." That was, I thought, another interesting statement.
Thomas: Yeah. I think it's a positive update. I think, I think it's, it signals that China is very receptive and perhaps will actively push for some [02:29:00] of the sorts of policies that I would be excited by.
Um, I think it's hard to say exactly what he means, and it's hard to make... It's hard to sort of super strongly, um, take it at face value because o- of many reasons, um, w- one of which is that I'm really not an expert in Chinese politics, and so I have no idea what the, uh, internal political dynamics are that are, like, sort of causing him to actually say this.
Mm-hmm. Um, I think we internally have basically just expected that China will be pushing for a deal because of this sort of argument that the US is by default in a very good position, and so the US g- so the Chinese will want to sort of make a deal while they still can. Um, and so I think that's a bit of evidence for this type of view that, uh, really it's a question of does the US want to or not, and I think the US should want to.
Um, yeah, Romeo, do you have anything to, to add- Yeah ... on the Xi thing?
Romeo: Yeah, I think, uh, I think mostly I, I just agree with that. I think it's hard though to... I think most of the Difficulty [02:30:00] will be in the, in the details, and it will be, um, at the time of a potential negotiation, just very susceptible-
Nevin: Yeah ...
Romeo: um, you know, to, to thorny details about exactly agreeing on how they're gonna verify things, exactly agreeing on, you know, f- things that are consequential for future balance of power and like we were- Yeah
saying earlier.
Nevin: Yeah.
Romeo: You know, the difficulty of agreeing for the US of agreeing to something that would allow China even partially to catch up. Uh, and then from China's perspective, agreeing to anything that locks in any kind of, um, disadvantage. So I think y- the intention... I was, I think, never as much worried about the intention to come to some kind of deal or, or-
Nevin: Yeah
Romeo: as, as I was of the actual- Yeah ... ability to reach a deal.
Nevin: Yeah.
Romeo: Um, but that being said, I do agree that it [02:31:00] was, like, a positive, um, a positive update and somewhat.
Nevin: Okay. And for someone who's watching or listening to this who does kinda think that you guys are crazy in, in your expectations of how, you know, ubiquitous and effective all this stuff will be in the future, what do you say to them?
Thomas: Um, yeah, I think, I think it's very reasonable to, at face value, take the things we're saying and then be like, "Wow, that's so far out of the distribution of the historical trend for, like, how, you know, the historical 2% per year, 3% per year GDP growth has been, the historical trends in technology." Um, I think, I think it's...
So I think it's basically a very reasonable first pass view. I would encourage them to take a look at our detailed argumentation where we sort of lay out the case for a lot of this stuff. Um, one maybe very important high-level argument to make is just from a very, um, from a very zoomed out perspective, I think what we're saying is [02:32:00] actually not a deviation from the overall trends of history.
In particular, we've sort of had these different regimes of growth where, you know, before, uh, agriculture, you know-
Nevin: Right ...
Thomas: growth was very, very slow. The, y- you didn't see 2% GDP growth. You saw GDP growth much, much lower than that if you try to, if you try to calculate it out. Yeah. And you sort of see these- You're
Nevin: saying if you just follow the exponential, then you kind of get to the same con-
Thomas: Yeah.
I would say, in fact, the historical, you see, you see, um, accelerating, uh, rates of growth, which means that your, your, the historical GDP trend has been substantially super exponential. It's only exponential if you zoom in to, like, a very, very particular regime.
Nevin: But what about the technology in particular?
Some people are j- are gonna say, like, "Well, okay. Well, sure. But, you know, you think that, like, you know, as soon as we automate this process, you know, we get to this crazy super intelligent outcome or whatever, or, or that it's even possible to automate the process." Like, what is it... H- how have you guys peered into the nature of the technology?
Like, [02:33:00] what are the key things you believe that you think- Yeah ... if other people believed them, they would be like, "Whoa, okay. Maybe it is gonna go that way." Is- Yeah, yeah, yeah ... are there, like- I think- ... two or three technical assumptions?
Thomas: I think the really core point is that AI is Is that AIs can do everything that humans can do.
There's nothing magical or special about the human brain. I mean, there's something special. There's nothing magical. And what the human brain does is fundamentally a bunch of com- is a really complicated set of computations that are, you know, uh, implemented on our, like, neurons and synapses in our brain.
And fundamentally, a computer can do all of that, and in fact it can do all of that way, way, way better than a c- than a human brain can because there are all sorts of architectural limitations of a human brain. So in fact, like, for example, the human brain is very, very bounded in the amount of energy it can consume.
It's very bounded in the amount of, like, total neurons, total computational capacity it can have. It's specifically bounded in serial computation, so the human brain [02:34:00] is extremely parallel. Um, but it can't do very many, you know, sequential operations. Um, it's had to, like, sort of use the parallel trick as, as, as much as it possibly can.
And for basically all of these limitations, you can just scale those with computers and you can say, "We're gonna add in more artificial neurons. We're gonna add in ... We're gonna make them faster. We're gonna make the clock speeds of the, of the computer faster. We're gonna make the equivalent sort of computations happen much, much, like, many, many m- many more times per second than the human brain architecturally can do.
We're gonna make the equivalent, like, we're gonna make the, you know, system two reasoning parts of the brain just much, much bigger-
Nevin: Right ...
Thomas: such that it can do much more." Like, you can sort of-
Nevin: And I assume that- ...
Thomas: make all this
Nevin: happen ... your view is sidestepping the whole confusing question of, you know, what is a mind and what is a conscious experience and so on.
You don't even have to have a position on, like, are these things gonna have conscious experiences or not. The point is just, like, behaviorally, uh, they, they should have these capabilities and then there's a whole 'nother podcast we could do about, you know, [02:35:00] uh, what, what sort of, like, mental phenomena or, or- Yeah
feeling or whatever may exist in these machines in the future. Is that right?
Thomas: Yeah. I think, I think that's right. Yeah.
Nevin: Yeah. And I, I, I think this is a helpful thing I encourage people to think about, um, to just, just to pull those two things apart, right? Because, you know, you're ... That, that position it, it, it doesn't answer the question of, like, will these, should these things have personhood and, and will they, you know, will they feel things and so on and so forth.
Uh, but the point is, like, well, the, the robots and, and, and AI superclusters and whatever just will be capable of these behavioral patterns of, you know, managing a company better than the CEO of the biggest public company can as of now. Yeah. Um, and, and that's sort of what's key to the things playing out the way that you think they will.
Thomas: Yeah. Exactly.
Romeo: I think, yeah. And I think there's, apart from this kind of argument that there isn't some inherent limit that from ... There's no reason that-
Thomas: [02:36:00] Yeah ... the limit- There is an inherent limit. It's just much higher than where we are at with humans.
Romeo: Right. Right. Exactly. There is an, an inherent limit that is somewhere around the human range for some reason.
Um, that seems very unlikely and, and that's kind of that argument. And then I think separately, if you just look at the trends of recent years in terms of... And, and the trends that are continuing for the next few years in terms of com- compute build-outs leading to larger and larger training runs, and more and more data and learning, um, and then what these different AI systems have been able to do over time Outside of the pure training and, and data that they've been, that they've been fed and environments that, um, they've been trained to do.
Like the, the level of, um, generalization of intelligence outside of the specific training regime. Um, and [02:37:00] then the fact that there's so mu- so many more researchers, uh, going into this field working on how to f- make all parts of the training and learning and data-
Nevin: Right ...
Romeo: more efficient. Um, it just seems like what we've seen, and then the things that we've seen resulting from that in terms of metrics like revenue, um, metrics like the code at the AI company- Yeah
written by the AIs. Um, just a lot of these. And, and our particular kind of timelines model at AI Futures Project that some of our colleagues worked on, um, not Thomas and I, but that, that one uses like, uh, projections of coding time horizon. So the, how, um, how long of tasks in terms of how long it takes humans to complete them can AIs do over time, and there's the famous like graph that METR has measured.
Um, and projecting that forward in, in [02:38:00] our model, um, for when we would be able to fully automate coding, and then trying to reason about, okay, if you fully automate coding, then how long will it take to automate other parts of the research process? Yeah. And how hard will it be to cross qualitatively through the human range?
Like these, these are some of the other things that, um, we've been trying to, to model and think about. And yeah, it do- it does just seem, um, like there's a high probability of, you know, in the next, I would say, like in the next decade where, you know, we're not extremely confident that we can get to this like strong AGI or super intelligence milestones.
But I think off the top of my head, and we ha- we have our numbers out there, um, but like 70, 80% probability, I think, that within the decade. Um, maybe more 80%, um, for, for most of us on the team.
Nevin: So to those out there who are [02:39:00] worrying that this might be a bubble, you guys' answer would be absolutely not. That is, that is the least of our challenges.
Um,
Thomas: I would say it's, I would say there's some small chance it's a bubble. Yeah. Like, there's some v- there's a, there's a possibility, which is that, which is, um, like I'm right about my theoretical claim that you could have AIs that are better than humans, but I'm wrong that LLMs are like a meaningful step towards that.
Right. And so there's a, there's a, there's a chance, I think it's a s- it's not that likely, I think, but I think there is a chance that the LLMs basically peter out. Yeah. And then like, and like this current data center build-out Was, you know, a bad investment and the returns are, are lower than people would have thought.
Yeah. I think that on average, that's not gonna be the case. We're gonna get an intelligence explosion and it's gonna go totally crazy.
Romeo: Yeah. Yeah. Yeah.
Nevin: Okay. Uh, we're, we're pretty much at time. Uh, a couple quick things. In addition to, uh, your scenario, which, uh, for anyone listening is at ai-2040.com, uh, which other...
Are there any other, like, plans or publications or projects that you would recommend [02:40:00] someone looking into if they're trying to understand where AI is going to ultimately lead us, just like basic resources?
Thomas: Yeah. So I think, not to toot our own horns, but I think ai-2027.com, like our predictive scenario, is I think also a, a pretty good place.
Nevin: Yep.
Thomas: Um, in terms of competitor plans, we tend to, you know, not like them as much as ours, but there are a bunch of pretty high-quality ones out there in terms of the amount of thought going into them, though I do disagree with a bunch of their conclusions. So situational awareness, Leopold Aschenbrenner's thing.
Um, Dario has written several essays, maybe most notably Policy on the AI Exponential and, uh, Machines of Loving Grace, um, are sort of maybe his, his two most important essays there. Um, and yeah. Yeah. Are there any other particular essays you think are, are important? I would say those are, like, the first ones I would recommend.
Romeo: Yeah. There's also the If Anyone Builds It, Everyone Dies- Yep ... which kind of describes... Like, we, we have, uh, in AI [02:41:00] 2040 this kind of diagram that also shows kind of the, our other categories of plans that we've- Okay ... in the way that we've bucketed them. Um, and yeah, pl- plan S is kind of like the shut it all down plan that that book corresponds to somewhat.
Um-
Nevin: And who's the top, like, person or organization that, um, you know, that has a very opposing view to you guys but that you respect and would, you know, maybe you read what they say and you think it's worth, like, looking at what they say?
Thomas: I would say probably MIRI.
Romeo: Yeah.
Thomas: Probably Eliezer Yudkowsky and, and their book.
Okay. That's
Nevin: in, that's in the even-
Thomas: That's in the
Nevin: even- ... more risky
Thomas: direction ... even more extreme.
Nevin: Okay. Good ...
Thomas: in the less extreme direction.
Nevin: And then a less extreme direction, what would you say?
Thomas: Yeah. So there's METR, I think-
Nevin: Mm-hmm ...
Thomas: um, is, is pretty reasonable.
Nevin: And that's M-E-T-R for people-
Thomas: M-E-T-R ... who are looking it up.
Yeah. They haven't really published a plan, and they're, they're the ones behind the, the Time Horizon Chart, which we, we just mentioned. Um, but
Nevin: I, I think- So they're less concerned than you?
Thomas: They're less concerned. They're not that much less concerned-
Nevin: [02:42:00] Yeah ...
Thomas: by and large. Um, I think, yeah, in terms of people who are really not concerned- Yeah
Romeo: I, I would guess maybe, like, Anthropic is the representation of being less concerned.
Nevin: Right.
Thomas: Yeah.
Nevin: Where it's like- And- Yeah Yeah ... they, they, they take the problem somewhat seriously, but they sort of think that they can just figure it out as they go. They think,
Romeo: they think they can probably, yeah- Yeah ... solve it- Yeah ... by themselves with not that much, um- Makes sense.
Thomas: Yeah. And so, like, Dario's essays are a pretty good representation of that view.
Romeo: Yeah.
Nevin: Okay then, last question for you. Given that you guys spend your professional lives, you know, imagining these possible futures and they're v- I think very real to you, and this is kind of what you think is gonna happen in the world. Your experience of just walking down the street must be very different from most people because you're expecting this, like, crazy whirlwind of change.
And so I wonder, like, are there just things about the world today and the, [02:43:00] what it's like that you savor because you're like, "Man, you know, we're never gonna have, you know, this sort of thing again." Like, you know, with, with, like, the, the robot explosion. I've s- after reading, you know, what you guys wrote, I'm like, you guys might wanna enjoy the world robot-free 'cause, you know, there's gonna be like a zillion robots if these guys are right.
Uh, what, what do you, what is your experience like in that regard? Huh.
Thomas: I kind of have a boring answer, which is, like, I think we can get the best of both worlds and, like, have some historic preserves where you ban the robots and whatever. Okay. And so you still get the good ex- you get the good experience that you miss.
I see.
Nevin: So whatever it is we want in the future, we'll get that too.
Thomas: But, uh, yeah, what, what, are there, are there particular things, um, Romeo, do you have anything off the top of your head?
Romeo: I think, uh, I think just in general, there's some separation between logically what I think will happen, and then, like, emotionally what you feel day to day.
Nevin: I see.
Romeo: Like, I, I, yeah. I- Yeah ... I think that [02:44:00] my day-to-day, like, life experience from before I was thinking about this stuff, which isn't even that long ago, and now actually hasn't changed that much. Yeah. But then-
Nevin: So you've still yet to fully update on the gut level to all of this. Yeah,
Romeo: yeah. Yeah. And I think it will be hard to until it's actually happening.
Like, I think, um- It'll be, yeah, I think, um, I don't know. I think they're starting, I think it's starting to, to become scarier. Um-
Thomas: I mean, I think Claude Code is a very intuitive sneak peek- Yeah. Yeah ... at what it's gonna, at what the future of, like, agentic, you know, AI labor is gonna look like. You can just offload very long run tasks, and then sometimes it will succeed at them.
Romeo: Yeah. Yeah. Yeah. But I think, um, I think maybe partially intentionally, I, I try to not think about it when I'm at work.
Nevin: Yeah, might go a little crazy if you think about it too hard.
Romeo: Yeah.
Nevin: All right. Well, thank you, guys, for all the time. Really appreciate it. This was fascinating. We, we could talk for several more hours, but [02:45:00] I know you got a flight to catch.
Um, so, uh, appreciate you joining the show and, um, yeah, for anyone who wants to learn more about this scenario and this policy, again, check out, uh, ai-2040.com.
Romeo: Cool. Thank you so much. Thanks, Nevin. Thanks for having us.
Nevin: Yeah.