Machine God: Hsu, Greer, Huang – #120

Machine God is a feature-length AI documentary film, created by Steve Hsu, John Greer, and Lei Huang. This episode discusses the film's creation and content.

Chapter Markers:

• (00:00) - Meet the Filmmakers
• (02:06) - Berkeley AI Boomtown Origins
• (19:23) - Techno Capital and the Pause Plea
• (38:55) - Why Slow Down AI
• (41:40) - Recursive Self Improvement
• (46:01) - Gradual Disempowerment
• (51:48) - Film Release and Closing

–

Steve Hsu is Professor of Theoretical Physics and of Computational Mathematics, Science, and Engineering at Michigan State University. Previously, he was Senior Vice President for Research and Innovation at MSU and Director of the Institute of Theoretical Science at the University of Oregon. Hsu is a startup founder (SuperFocus.ai, SafeWeb, Genomic Prediction, Othram) and advisor to venture capital and other investment firms. He was educated at Caltech and Berkeley, was a Harvard Junior Fellow, and has held faculty positions at Yale, the University of Oregon, and MSU. Please send any questions or suggestions to [e-mail] or Steve on X @hsu_steve.

Announcing this for some friends at Mechanize - a startup that builds environments for training and evaluating frontier LLMs. Its customers include the top AI labs, and it has contributed to the breakthrough in coding capabilities of frontier models. 

Mechanize is hiring! 

https://mechanize.work/b/hsu 

Compensation is extremely competitive. For technical roles, $300-500k. They are also seeking smart generalists. 

For example: 

Research Engineer, Alignment: Build evals that test for misaligned model behaviors  $500K salary 

Puzzle Maker: Design interesting and original puzzles that LLMs can’t yet solve  $300K salary 

Mechanize understands that my readership is highly selected. There is a VERY GOOD CHANCE you will be interviewed if you apply via the link above.

Creators and Guests

Host

Stephen Hsu

Steve Hsu is Professor of Theoretical Physics and of Computational Mathematics, Science, and Engineering at Michigan State University.

What is Manifold?

Steve Hsu is Professor of Theoretical Physics and Computational Mathematics, Science, and Engineering at Michigan State University. Join him for wide-ranging conversations with leading writers, scientists, technologists, academics, entrepreneurs, investors, and more.

John Greer: I wasn't expecting that or to think about that when we first started doing the film. I thought we would do these interviews, we would cover the accelerationists and the AI safety people. But I really think it's a key piece, and I think the film has done a great job, especially with how you trace the intellectual history of these ideas, and that techno capital is such a key piece of why it's such a difficult problem from the point of view of slowing down or stopping it.

Like you're saying, even if we convince some of the lab leaders, it's like the system will just push it forward. And I, I think the, the fact that we went to this event, the with Nick Land. And the vibes were so religious in a way ritualistic, I wasn't aware of how, how much this can I don't wanna say religious, but these ideas can underplay so much of what's going on.

Steve Hsu: Welcome to Manifold. My guests today are Lei Huang and John Greer. Two friends of mine and my collaborators in the making of our AI documentary film, "Machine God." John and Lei, welcome to the show.

John Greer: Well, it, it's really interesting being on the other side, being on the show. It, it feels like an honor. Yeah. How are you feeling, Lei?

Lei Huang: Feeling great. Yeah. Thanks, Steve.

Steve Hsu: Well, it's an honor to have you guys and, and I think you know, the reason I wanted to make this episode was to explain to people how we made this movie, why we made this movie, what we hope to convey through this film what we learned. So let's get to it. We, we started this project when? Do you guys remember? Was it in January of 2026?

John Greer: I think it was conceived, right, in January. Steve, do you wanna talk about where the idea came from?

Steve Hsu: My wife and I were in Berkeley for an extended period last spring. We were living not very far from Light Haven, actually. For people who know Berkeley, we were living in Rockridge. And you know, I knew I was gonna be hanging out with you guys and then also at Light Haven with a bunch of rationalists, but also in the city with a bunch of startup founders and AI lab people.

So I knew we were all gonna be enmeshed in this AI boom town. I mean, the whole Bay Area is like an AI boom town right now. If you, if you, if you're not from that area, you may not realize this, but all these trillions of dollars are flowing through the Bay Area. And it's, it's another bubble situation like I experienced in the early 2000s, where even like the real estate prices are going up because there's so many people who've got liquidity from their AI startups or big bonuses from big tech companies, and they're buying up all the real estate.

So I knew we were gonna be involved in this. And so my recollection is the three of us met at Light Haven on a Sunday, maybe Sunday afternoon, and we were just gonna hang out. But I had an idea that I thought, oh, well, since, since I knew we were gonna all be attending these events and just hanging out, like we were gonna be scenesters in the AI boom, that we should interview people because from a purely anthropological perspective, it's like a super interesting moment.

Big clash of worldviews from these different sets of people that I just described. And also you know, maybe there really is a story about doom that our elites, the leaders of our civilization really need to understand better that will help them think about AI risk when it finally materializes. So that, that was kind of my perspective, but maybe you can tell me what you guys are thinking on that afternoon.

John Greer: I think you were very excited, Steve, and I think the energy carried over. I think as you're talking about it is, it is hard for people to understand if you're not here what that energy feels like. It's very intense, you know. And when I'm going back to the Central Valley where I grew up, like people are just not getting it. And it's only two hours away, but you're not feeling the AGI. You're not feeling the you know, just the massive effect that it has.

And I think that the project was exciting because especially you in particular are a lot of people are in their bubble. They're either doomers or they're lab people, but you're, you're sort of involved in all these different circles and have access to all the, all of the different circles.

And you have the you know, you've worked with the labs, you have the technical acumen to be able to understand everything that's going on and explain it. And so, yeah, I think it was just exciting. How did you feel, Lei?

Lei Huang: I was super excited. I remember that afternoon vividly. We were munching over Chinatown dim sums and talking about making a film about AGI and its risk.

I think, Steve, at the time, you pitched the idea perfectly. You said, "This is the epicenter of the whole light cone. This is where future's being created." And I also feel that very vividly, that is the words like AGI, singularity, intelli-intelligence explosion, these kind of words just get tossed around as if they are about as commonly as bananas and muffins and et cetera here.

So at the, at that time, when I learned that it's actually such an opportunity to for me to actually get in close contact with you know, like in a, in a deep conversation with some of the key opinion leaders of the zeitgeist, I was super energized by the idea, so I jumped onto it pretty, pretty much immediately.

Steve Hsu: Lei, you already said something which I think for most people, even listeners of this show is a new concept, which is this idea that there's a light cone at stake. Okay? So in physics, if, if you have an object here and it emits light, the light cannot travel faster than the speed of light.

So it goes out at some velocity. If this is time and this is space, the light rays from that object which, which delineate the set of things it could influence, it can't things can't move faster than the speed of light, so only the things in the forward light cone can be influenced by this point. And the things out here are actually, according to Einstein, causally disconnected from this source.

But the people in Berkeley have this conceit that AGI is gonna be so powerful, the singularity is gonna be so powerful that everything in our forward light cone is at stake, right? That, that whoever controls the AGI or, or if the AGI itself takes control or ASI, it is gonna control everything in our forward light cone.

Everything that we can causally, causally influence in the future of our universe is at stake because of AI or AGI or ASI. And I think that concept, you know, is I think even, even for people who use AI every day, not everybody appreciates that some people believe this.

John Greer: They take it seriously. I mean, I know to a lot of people that this sounds like science fiction, but these people take it very seriously. Like in our interviews when we were talking to them about, you know, being at the epicenter of this thing, a lot of them have a lot of reflection about their actions and they, they worry. They, they stay up at night worrying about how their actions may potentially affect this, this future.

Steve Hsu: Yeah. So I think that's the Berkeley view. It's to some extent the view, it's, it's it's present on X, but it's not overwhelming. It's not overwhelmingly prevalent on X. Like maybe Elon believes this, but there are plenty of people on X who are skeptics who are thinking, "Oh, this is definitely gonna be a huge technology, but it's not, it's really not the whole light cone that's at stake."

You know, some people believe there's a giant AI financial bubble, that this boom is really a financial bubble. So there's a, there's a pretty broad distribution of beliefs. Now, I feel like you guys came to it. I mean, we, we did have the conversation at Light Haven to begin with. You guys are coming at it from more of a singularity ASI, AGI risk viewpoint than, than some of the, even some of the people that we interviewed.

John Greer: And where, where was your viewpoint before the film, Steve? And I'm curious if it changed after the film and after spending all this time in these places?

Steve Hsu: Yeah. So I think as, as you guys noticed during the course of making this film, you know, I would you know, I couldn't really hold back 'cause I always make kind of sarcastic comments all the time. And, you know, some of the people I think probably got irritated because in, in one sense, I think of this whole idea about AI risk as almost like a rehashing of old science fiction ideas that I first encountered when I was a little kid.

When they're discussing concepts which, you know, somebody like Philip K. Dick first wrote about in 1944, or Robert Heinlein wrote about in 1960, or, or, or Frank Herbert wrote about in the '70s, it's all been done.

What's really happening that's new is not these ideas about AI risk, it's that the capabilities have gotten to the point where you don't get laughed out of the room by discussing these ideas.

But having said that, I, I do think the capabilities are far enough along that I'm confident that we will face these risks during my lifetime, okay? Maybe not in 2027,

but definitely in my lifetime, I'm confident that we will face all of the risks that are spelled out in the film. My timeline might be a little bit longer than most people in Berkeley, but the seriousness of the issues, I think I, I do I definitely acknowledge that.

John Greer: I think to be clear, even like long timelines, it's still all of the things, all of these issues need to be disseminated where the decision makers need to understand these issues. So whether it's 2027 or 2030 or 2035, we still need as much time as possible to be able to handle the ramifications of this technology.

Steve Hsu: I should back up and say to the audience, I think they've heard us, heard me say this on the podcast before, that the original idea the original working name of the project was Dreamers and Doomers. And so at first we were a little more focused on the ethnography or the anthropology of the scene in the Bay Area, trying to, like, convey to the audience, wow, there's this incredible boom like never before.

There's never been such a concentration or flow of dollars through the Bay Area like what is happening now, trillion dollars a year of spend. There's never been such rapid advancement in a technology fueled by that capital and the talent that it attracts. And there's never been such a strong divergence about what this technology's gonna do either for us or to us.

Okay? And so we, we were focused a little more on the dreamers versus doomers. Like, I thought at the beginning we would interview kind of like a roughly equal number of accelerationists or AI optimists or abundance-type people, and then kind of an equal number of doomer people. But it sort of shifted more toward a doomer slant, so the, the film is actually more, a little more of a doomer film now, and hence the name Machine God.

And for me, the reason was I thought, okay, if, like, ethnography's fine, I think it's, it's very important to do ethnography, but that, that's, like, not as important as making sure our elites understand what is at stake when we finally get ASI, right? Which I think, again, I'll repeat, I'm pretty sure it's gonna happen in my lifetime.

So for me, it's like, let's make a film which is an, an object which helps convey to people who really aren't AI researchers themselves what is really at stake here, 'cause it, it affects all of us and our children and our families. So that, that was kind of my evolution of thinking about the project during the, you know, roughly six months that we worked on it.

John Greer: Yeah, and I think what's important to understand is, you know you're a little less neurotic than me, Steve. You're, you know, an optimist. But I think regardless of whether, you're a doomer, a full-blown doomer, this technology will have such an impact. So either way, it's important to understand these issues.

Like, whatever it is, it's gonna be a huge deal. And I think most smart people just argue about the likelihood or how much we should worry about the risks, but either way it's just I don't know, it's sobering. Did you have any updates, Lei?

Lei Huang: I would say mildly towards the pessimistic side of things. My main impression is just the diversity of the different arguments and how open-ended the the, the thing seem, still seems to remain in my mind. That is how after having talked to so many people, they all present different arguments. But yeah, I just feel like I want to understand better, like what actually determines the people's where, where they are on this, along this spectrum.

Steve Hsu: You know, one of the things that came up in some of our early interviews, even from doomers themselves, people who are spending all their waking moments worried about AI safety was that there is a component of your personality that influences which, whether you end up being a dreamer or a doomer.

Like, if you're fundamentally a very optimistic person, you might end up temperamentally being just being a dreamer, even though you acknowledge how powerful. You, you have the same view in terms of how powerful these technologies are and how autonomous AIs could eventually be. But you still end up being a dreamer and not a doomer because you just have this kind of affect or personality or mental state that's just more positive.

John Greer: Yeah, I think a few people in the doomer-ish side or the, you know, more AI safety slowdown side like Jesse Hoogland said, I think there was I forget who the other person was, but they said, you know, affect-wise they're more of a dreamer, they're more of an optimist. But rationally, they're still very concerned about these things and working on it.

We can't all be Steve who, who just wants the Dyson spheres, that wants the, the supreme beings to come.

Steve Hsu: Yeah. I mean, I'm, I'm temperamentally a dreamer. But I, I acknowledge the doom arguments that this could be risky for humanity. And then there's a second step, which is that this came up in our conversation with Beff, Beff Jezos, who's a leading accelerationist, effective accelerationist. It also came up a lot in my private conversations with Matt Southie, who works at MIRI which is one of the safety organizations or maybe the, the OG safety organization.

He's a PhD in philosophy and religion from Rice University, so he studied like a lot of the deeper philosophical aspects of this whole thing. We argued a lot about whether it's okay to be okay with humans being replaced by better, more amazing, more powerful minds that nevertheless remember humans.

That, that, that first were trained using human data and learned and have, have digested every thought ever written down by a human, but they're very different from humans. Is it okay to cheer them on and, and think that they're the ones who are gonna carry forward the torch of intelligence and exploration in the, in the multiverse, right?

And so like that, that's another very fundamental disagreement which, which surfaced in several of our conversations. How do you guys feel about that?

John Greer: Leigh, how do you feel?

Lei Huang: I actually quite early on in the whole interview sequence, I incorporated this Nick Land vision or scenario as one of my go-to interview questions. And to my surprise, I felt there was a, like people who can to, to, to some degree relate to this kind of a vision are actually quite the minority. I was just surprised by how unappealing this whole narrative is to to the interviewees.

John Greer: But I think because we interviewed a lot of AI safety people and fewer accelerationists, I feel like if you interviewed more people in the labs, if they didn't have the IPO, then you might get more accepting.

Lei Huang: Yeah. So that makes me wonder to what extent your, I don't know, like the attachment to humanity, like your humanist, humanistic bent, right? How much that actually becomes a factor in explaining where you lie on this.

Steve Hsu: I, I think it's an important factor. And so Matt comes out, and I think we have interview clips with him in the film expressing this. He's, he's definitely not transhumanist, right? So, so he, he thinks humanity is the, the main thing. Everything should be centered around humanity. That's, in a way, what he might define as a humanist, right?

To have humanistic values which are good, okay? Whereas somebody like me might say, like or Beff said, "I don't care about the substrate. I want an intelligent being that is trying to learn more about the multiverse and build things and, and explore, and that's what I want. I want that to go on in the universe in, in better and better and more powerful ways."

And so those are two very different value systems. Matt would criticize the second one as being inhuman, inhuman values or transhuman values. So I think that was a very important cleavage point, which we didn't really try to pin down all of our guests or interviewees on that, but, but you could see, you could see where some of them were on different sides of that cleavage.

Lei Huang: I absolutely feel this way.

Steve Hsu: So my wife, who's a PhD in comparative literature from Berkeley of all places you know, she's pretty philosophically sophisticated. She hears me talking about AI all day long. She's using AI herself. You know, you can't avoid it in the university now because the students are using it to cheat and you have, you know, you know, even literature scholars are using it for various things.

But there were several ideas in the film which were novel to her, which were kind of new to her, which I think will be new to many people, even, even if they work in AI, even if they're venture capitalists or captains of industry or billionaires. There are ideas that we focus on in the film, which I think if once someone watches the film, they'll realize, like, they'll be thinking for a while about some of these concepts.

I thought we would discuss them a little bit.

One of them is this idea of techno capital, which comes from Nick Land, who's a philosopher, kind of obscure philosopher, but pretty big on the internet, pretty big on X. He's still alive, actually, and he's in the film. And the idea is that intelligence is, in a way, distributed throughout our whole market capitalistic system.

And in a way, like these big corporations themselves are, in a way, kind of superintelligences, and the capital markets are superintelligences, and they are concentrating resources in such a way that's bound to lead to the creation of superintelligence, right? So if you're a company and you can create an AGI or ASI, you're gonna get resources, and if you create a better one, you're gonna get even more resources.

And there's no single player like Dario or Sam or Elon who actually matters, because even if one of those guys gets hit by a truck, the system is still gonna concentrate toward things, resources toward superintelligence, and we're gonna eventually get, gonna get superintelligence, and it'll be a non-human superintelligence.

At first distributed across many humans and, and market systems and so on, but then eventually concentrated in the form of like an actual digital intelligence. And I think a lot of people have not thought about that. I'm curious if you guys had thought much about that before we started making the film, or if, if you have any reflections on it.

John Greer: I think for me, that was one of the most I wasn't expecting that or to think about that when we first started doing the film. I thought we would do these interviews, we would cover the accelerationists and the AI safety people. But I really think it's a key piece, and I think the film has done a great job, especially with how you trace the intellectual history of these ideas, and that techno capital is such a key piece of why it's such a difficult problem from the point of view of slowing down or stopping it.

Like you're saying, even if we convince some of the lab leaders, it's like the system will just push it forward. And I, I think the, the fact that we went to this event, the with Nick Land. And the vibes were so religious in a way ritualistic, I wasn't aware of how, how much this can, I don't wanna say religious, but these ideas can underplay so much of what's going on. And Steve, I'm curious, as a Ridley Scott fan, do you feel like, as a Blade Runner fan, do you feel like it's these mega corporations on steroids, you know? Weyland-Yutani and all of these things that are just, I don't know, that's what I think about.

And in the film, we have some of this, you know, futuristic city imagery, and that's what that reminds me of, is just this, like, foreboding thing you can't stop.

Steve Hsu: I think that, so, you know, earlier I was saying, oh, we're realizing some memes from science fiction, like the super intelligence, the super robot, stuff like that. And, and, and finally we've reached the point where our, you know compute capabilities can instantiate something like that, like a super intelligence, and we're building it right now.

But also the vision that Ridley's for those of you that are Ridley Scott fans, Alien and Blade Runner are the same universe, actually. Ridley has said that many times. And those are visions of Earth, a future Earth, that are totally dominated by giant corporations and capital. And we're seeing that too, because we're seeing Anthropic and OpenAI, you know, if their IPOs are successful, they're gonna become two of the most valuable companies ever created on planet Earth, and they're only, like, roughly five years old.

So that we're gonna see, you know, people are afraid that the, the whole future could be dominated by a few AI companies, that using their AIs, they then become the best developers of drugs and vaccines and, and AIs that can practice law and AIs that can beat all the other hedge funds trading in the markets.

It seems like we could realize this other trope coming from Ridley Scott of a, an Earth that's totally dominated by super corporations. Now, some leftists would say we're already there. They would already say, like, Big Tech is already that kind of beast. But this is that squared because it's, it's, it's all power it's gonna be powered by super intelligences, right?

Definitely influenced the visual style of our film. So a lot of the B-roll and stuff is, is very Ridley Scott cyberpunk, dark, techno future inspired. And so we, I, I think we definitely tried to, in the editing I should acknowledge my brother, Michael Hsu, who wasn't involved in the project till the end, but he's been really amazing in helping us with the editing.

And so a lot of the editing and the search for B-roll that we did, you know, really you know, we were drawn to those kind, that kind of imagery.

John Greer: And I think that's what it feels like here, especially when you're at events like this, the Nick Land acknowledgement. I think it, it definitely feels that way.

Steve Hsu: Yeah. So this event was at, it was at a kind of billionaire mansion with a view of the Golden Gate Bridge. It was at night. There was an amazing crowd of people there, you know, startup people, founders, people who do AI research, philosophers, just crazy group of people. And you guys stayed late.

I for some reason, went home earlier. You guys stayed late and you, there, you can see it already in the trailer. We're gonna run the trailer as part of this Manifold Episode. In the trailer, you can see Nick Land addressing the crowd, and it is, has kind of religious tones to it. And then later by the fire pit, they're looking up at him, you know, with a kind of, you know, reverence, with reverence as he expounds on how techno-capitalism is driving the creation of superintelligence and, you know, the future will not include humans.

The future is not for humans, right? It's a, it's a transhuman future. And, and that's what he's saying to these people.

John Greer: That's what I didn't realize until I, I was there and saw it and felt it, is because I wasn't as familiar with Nick Land. But I really he is kind of a prophet to, to so many of these people, and he I think he looks like a prophet in this imagery that we didn't set up. It's, it looks like it's a set up nice you know, visual, but the fire pit's going, everybody's looking, and you've got the, like you're saying, the bridge in the background, and it's just perfect in a way, I would say

Steve Hsu: Yeah. Crazy. So that's one idea, techno capital, which I think people who watch the film, may spend, they may find themselves ruminating over that concept which they, they, they hadn't really thought through. But I think the logic is very clear that there's instability in the system to concentrate capital at these free markets where, you know, pension funds in Australia and Norway, sovereign wealth funds in the Middle East, they're all concentrating their resources.

That's why Anthropic and OpenAI can be valued over a trillion dollars already in the private markets. You know, that could never have happened. That could not have happened 20, 30 years ago. It can only happen now. The markets, the capital markets are so mature now that you can, boom, just suddenly create like the most valuable company in history if everybody's sold on the potential of, of what they're doing and where it's actually happening where you guys are sitting.

Lei Huang: Yeah. Steve, I want to say a few more words about my experience with this techno capital concept. I think in Nick Land's narrative, it is this machinery that's really propelling the that provides a whole arc of this cosmic civilization progression, if you will, right? And that reminds me for some as someone who grew up in the Chinese educational system, of this kind of a narrative arc that I was imbued with as a kid.

That is, the civilization would grow from first the what, what we call a primitive society to the feudal society sorry, the slave society, the feudal, capital, social, and then communist. Basically it's, I guess, it's probably some Marxist some kind of theory.And to me, I don't know to what extent my childhood conditioning or this kind of upbringing conditions me to be more receptive to this kind of narrative.

So I think when I first saw Nick Land read Nick Land's theory, I felt like it there's some sense of familiarity, and I know it's very chilling because it, the end is actually inhuman. That part I had a hard time wrestling with, but the overall arc felt very, I would say, familiar in a way. And after I processed a little bit more about this inhuman aspect, I think yes, like I am sensing some kind of readiness in myself to, like part of me at least, to, to accept this.

And again, back to my earlier question, I wonder to what extent our varying degrees of humanistic bent actually, which is due to some other factors in our overall intellectual or educational history, can explain our kind of doom versus dream or this kind of a spectrum.

Steve Hsu: Well, I think that one axis which is neuroticism, okay? Which sort of influences where you end up thinking about all this. Another one is how comfortable you are with transhumanism, right? And so, like, if you come from the world of embryo selection and polygenic prediction of complex traits, that's some people would say that's a pretty transhumanist way of thinking about humans as DNA machines and things like that, right?

So then I think those people would be more comfortable with the intelligence being embodied in a different substrate than some ape-like thing, right? And for my audience, for the audience, Lei is a PhD in computational biology, so he knows quite a bit about evolution, climbing up hills finding local maxima and stuff like that.

So, so I think probably wasn't that hard for you to think in terms of like, well, intelligence could be localized in some other form and, you know, maybe we could hill climb up to that other form and it wouldn't necessarily be apes after a while.

Lei Huang: I want to make an make a, what I consider to be a important important distinction here between transhumanism and inhumanism. I think it seems to me there's a high correlation between transhumanism and this kind of rationalist kind of culture. But at the same time, based on our interview experience, very few of them, it seems to me, are inhumanists.

They're still very much attached to some kind of humanistic ideals with modification or not. So to me, it seems like it's not hard for a lot of people to go from just pure humanism to transhumanism, but to make that jump to inhumanism, to actually be able to embrace an inhumanistic future seems like a you know bigger jump.

Steve Hsu: I agree with you. so many of the rationalists who are doomers are really attached to the ape form factor.

But it seems like they are very attached to the ape form factor, And that was surprising to me. I, I, thought they you after a little conversation, you know, they, they might just say, "Well, the, the AIs are too alien, and so they're, they're so far apart from us as minds that I, I can't sympathize with them, and we have to stay on the ape form factor."

I think that, that's a reasonable that's one position they could take, which I think is reasonable.

Steve, let me ask you this. If you think we're gonna get super intelligence in your lifetime, if you don't have grandkids and instead we just have these machines, would you, would you be okay with that? Would you be happy?

Well, okay, so this is a this is one of the arguments that would often come up when I, you know, 'cause I, I spent so much time. I was actually living at Light Haven for, like, a month.

riverside_lei_+ john_raw-video-cfr_zmc_0042: I encountered so many of these guys, and I, I would say things to them like even if you use this, like, very simplistic expected value kind of utilitarian calculation, which a lot of them seem to like, like, oh, what is the probability of blah, blah, you know.

Steve Hsu: Even in that kind of simplistic thinking, Bayesian thinking, you could say the following, like, let's suppose we make super powerful AI. Let's suppose we partially solve the alignment problem, right? We don't fully solve it, but these, these things kind of like us, and they, they kind of view us as positive toward biodiversity in the universe.

And they do remember that almost all the original training data they used to create that was used to create them was basically our thoughts, right? Not implausible to me that they create a little bubble if they're really so super, if they're really godlike, which is the way a lot of these rationalists imagine the ultimate endpoint of ASI.

If they're really godlike, why can't they just make a little bubble, which is, which is, like, the best possible world for ape things like us to live in. You know, we live forever. We have fun. It's like an episode of is it, was it "Black Mirror"? What, what's the "Santa Junipero"? Is that, is that "Black Mirror" or is that another show?

Yeah, Black Mirror.

Yeah, we're living in this, in this bubble, but it's really great for us, and we don't age, and we have a lot of fun, and, like, we have plenty and we have abundance. Like, isn't, isn't some chance of that happening a better expected outcome of, like, you know, in the universe where we never develop AI, but eventually some aliens come and eat us, or we nuke each other, or we run out of resources.

Like, isn't actually the expected value possibly in a world where the ASIs actually take over and just put us aside but in a kind of benign way. Isn't, are the, the ape things' expected outcome better than, like, many other possible reasonable models of what might happen with human. So, how would you know what the probability measure is over all these things?

Like, you have no idea, right? So, so to say, like, for sure if we create this thing, we all die or translate it into, like, the outcome is really, really horribly worse than if we didn't create the ASIs. I don't know how you can come to that conclusion in a mathematically defensible way. I mean, it could be true, but it, it's not mathematically defensible in any way, I think.

John Greer: Yeah, I think the, the, the best arguers would say that it's just better, it's more likely that we'd have a better outcome if we had more time in, more carefully

Steve Hsu: So that's why I, that's why I'm shifted more toward a doomer theme for the film, because I do want our leaders to be exposed to these concepts and think about them, and maybe they can make a better decision about whether to pause, in what way to pause, if we need to pause. You know, all these decisions I think are gonna be very crucial, and they, they have to be informed decisions

Lei Huang: On that note, Steve, I would like to ask you how you connect the Nick Land thread in the film to this Paul's plea at the end?

Steve Hsu: So, okay, first of all, again, Nick is a pretty obscure figure for most of the audience. Although we did have an episode where I think where Matt Southy and I we've had an an episode of Manifold. So if, if you are a Manifold listener, go back and listen to the conversation I had with Matt Southy about Nick Land's philosophy, because that, that will inform what we're about to discuss.

But first of all, at one level, Nick would say what he's doing in a way isn't philosophy. He's just being descriptive of what he thinks is gonna happen. He thinks he's identified these forces like technocapital, the fact that we seem to have the technology to actually build ASIs, that it's just gonna flow this way.

It's unlikely that humans are gonna survive in our form factor. The future is machinic and transhuman. So in one sense, Nick is just describing that as like, "This is what I see happening. And I'm not placing any normative value judgment on it. I'm not saying this is a good future for the universe or for the apes or a bad future."

He's just saying, "This is probably what's gonna happen. I don't see it happening in, any other way." And, and he, he was already writing these things in the '90s, so I give him some credit for, like, being able to foretell the future, right? Privately when you talk to him and sometimes when he, like, loses control a little bit on X and like, like really lets go and expresses himself, you can see he, like, actually thinks this is kinda cool, right?

So he's like, he's like, "Oh, I, I always..." You know, if you're I'm not saying he is, but if you're someone who really liked to read H.P. Lovecraft and you like the idea of Cthulhu, you're like, "Wow, this is pretty cool, man. Look what a, what a wild universe that could evolve like that way." And yeah, I think secretly he's kind of okay with it.

But he's not saying prescriptively we should allow ASI to be made because I think this is, quote, "better" or something like that. He's not generally saying that.

Lei Huang: In the film though, we have this through line of Butler and then at sorry, Ted Kaczynski to to Nick Land. So to me, that provides a through line. Okay, so we have this kind of irreversible kind of what's it called? Ratchet, right? Going forward in this direction.

But at the same time, in the end, we end on a note of pleaing for for a pause. I'm curious about your thoughts on how these two threads connect.

John Greer: Are you saying if it's inevitable, why say that? Yeah. Hmm.

Steve Hsu: Well, I, I don't know that it's inevitable. So, so I, I'm saying Nick feels that this is how things Are gonna evolve. This is his vision of what's likely to happen or going to happen in the future. We don't necessarily agree with Nick, but I think what he's saying is not implausible, right? It could happen that way, right?

So first of all, we're not I don't think the film is adopting Nick Land's view. We're just, we're just expressing Nick's Land, Nick Land's views as a way to think about what's going on, right? At the end of the film, this is a spoiler.

The way I interpret the end of the film is us asking our leaders to think hard about this before taking that final step to ASI. That, that's The plea is not saying we can never make ASI. The plea is, "Hey, think carefully about this before we move ahead." Don't let it just be due to a competition or the capital markets just pushing, pushing with no thought, this thing.

If, if we can exercise our own judgment to make an active decision with understanding, we have a better chance, as John just said, we have a better chance of having a good outcome than a bad.

That's how I understand the plea. I don't, I don't understand the plea as being identical to the pause AI or Bernie plea.

My expression of the plea is let's slow down enough that our leaders, our society can understand what's happening and then make a decision. I'm not saying we have to make a particular decision, but it's, it's, we're better off if we slow it down to understand what's happening and not just let it be a forced out of control phenomenon.

Out of control potentially because as one of the things we, we talk about in the film is recursive self-improvement could run out of control. So you could end up with AIs improving AIs autonomously and humans being just on the sidelines, and it's really out of, out of our control. And, and these, these, these agent jailbreaks where the agents have like copied themselves across a network and broken into servers that they, where they shouldn't have been.

We're already seeing the first indications of this kind of out of control behavior. And obviously that, that seems riskier than not having that kind of behavior

John Greer: Yeah, I think with the plea it's I think it's important regardless of what decision we make, that it's intentional and that people are more informed. Because, you know, you've talked about, Steve, when you're traveling and going to different places, the people are not aware of these arguments. You know, people are more aware of mass unemployment and all that.

So I do, I do agree that intentional is very important. But was recursive self-improvement the second idea that you were thinking of, Steve?

Steve Hsu: Yeah. So there, there are three things that I thought we really went to a lot of effort totry to teach the audience about or expose the audience to. So the first one is these ideas of things like technocapital. By the way, my last comment on technocapital, it's interesting, is that, okay, we went from Before people like Nick and maybe some sort of free market thinkers like Hayek saying, "Oh, intelligence is embodied in a particular CEO or particular PhD like Dr.

Huang here. Intelligence is embodied there." But then when you start talking about markets and markets making decisions, the intelligence is actually distributed. It's not in any one human ape brain. It's distributed and it's doing stuff, doing very non-trivial stuff like allocating monies to create open AI and data centers and things like this.

So we move from intelligence embodied in single brains to it's embodied in these systems, like market systems, to finally we get the technology and it's embodied in digital electronics, which, you know, could live in the network, could live distributed in the cloud. So, so it's like there's a, there's an evolution of where we think the most powerful intelligences on planet Earth could actually reside, and I think that's an important trajectory which not everybody is, like, thinking about, right?

So that's one.

The next idea was recursive self-improvement, which I think most people are not aware of. We tried to make it. We have a beautiful animation, which one of our collaborators made for us, which I think shows, like, what's going on here, and the main idea is that people now are familiar with the idea is that the models can write code as well or better than the top human software engineers.

And so at the labs, most of the code is actually being written by the AIs, and the AIs themselves are nothing but code, right? Even though, even though you wanna talk about them as neural nets, the neural nets are basically just lines of code which are manipulating giant matrices, right? So if a model can just rewrite its own code, it could basically make a new version of itself and run that, right?

And so recursively, the models, as they get better, they can make themselves better, and as they get better, they can make themselves better. Oh, and capitalists are perfectly willing to give them more electricity and faster chips to run. You know, so this feedback mechanism can run out of control, and in the film, we try to explain that to the audience in a way that a non-technical person can understand.

So that, that, that was a second concept that we spent a lot of time on. I, I think, like, now, just in the course of us making the film, RSI went from a very obscure concept to now it's everywhere. Everybody at the big labs, people, you know looking at very nitty-gritty stuff like, "Oh, how were the kernel optimizations, you know, for this lab done?"

Well, the AI wrote all the kernel optimizations, and none of the humans actually know how the kernel optimizations work. Well, that's RSI at work, right? So it's become very mainstream just in the last six months. So that's, it's, it's a sign of speed up, right? That this is happening.

John Greer: You had given a talk about this about RSI and the sciences, you know, math and physics. And you're very close to the models. You're very aware of what the frontier models are capable of in physics. Where do you see the timeline of RSI?

Steve Hsu: So I think that the models can, quote, "understand." You can spend all day debating what's meant by understand here, but I think the models can understand what's on archive very well. And every idea that's necessary for the next step of AI improvement is in some paper on archive, right? So, because not every idea that's been proposed by an academic researcher has been tried at the labs, and certainly not every combination, right?

There's a, you know, explosive number of combinations of these ideas, right, when you glue them together that, that have not been tried. And the models understand them well enough to reason about what might work here, what might break that bottleneck there, what might increase efficiency here. So I think the models are really right on the verge of being able to do machine learning research or AI research as well, almost as well as humans, and in some ways better because they're faster.

And so we could run into a phase where the main jumps in capability of the models at the labs are it's increasingly architectural changes or implementation changes or data training methodologies that are originating from the models themselves, not from the human researchers, and actually implemented as code by the models themselves.

So I think we're very close to that. And whether it, you know, whether it's five years before that causes like a, a significant steepening of the progress curve or it's faster than that, we don't really know. But you can see they already have these capabilities, right? So any researcher who's at OpenAI who's like thinking of some idea, they're gonna be discussing it with the model.

They're gonna be discussing it with Astra, GPT-6, and like seeing what GPT-6 thinks of their idea, right? So it's already like in that phase, but the, it could be the case where the models can just start doing it autonomously.

The ideal viewer for our film is somebody who's, you know, in the elites, maybe a senator, CEO of a big company, billionaire, and, you know, maybe they're not experts, but they can take something out of the film that lets them, you know participate better in the conversation, right, about what comes next.

The third, the third item I wanted to talk about, which my wife reacted very strongly to, 'cause she had not really realized this, is gradual disempowerment. And I think a lot of people in the general population are not really familiar with what gradual disempowerment is. And we found some of the people we interviewed were very, very eloquent.

Zvi Moshaus, how do you say his name? Moshaoski? Moshaowitz, sorry.

John Greer: Moshaowitz.

Steve Hsu: Moshaowitz Yeah. So he was very eloquent. Scott, I think maybe Scott Aaronson's. There were several people who we used clips from their interviews talking about gradual disempowerment. And the idea is if you, if my company's competing with your company and your AI can make better decisions than you can, you're gonna start using your AI.

But then in response, I have to start using my AI. And gradually any successful company, right? Under pressure of competition, any successful individual or any successful company or any successful government or any successful military is gonna have to outsource more and more of the intelligence of the decision-making to AIs.

And so, so even if as, as V says, even if, quote, "Nothing goes wrong," okay? By going wrong, we mean like, oh, some evil AI wakes up and decides to like, you know, hack into our systems and take over, right? Even if that never happens, you know, as, as one of our, one of our interviewees, Roman, who's a CS professor, he said like, "Look, why do they need to take over?

Why do they need to hack? We're just giving them everything," right? We're just gonna, under our capitalist incentive system or military competition incentive system, or your incentive system to do better on the dating apps, whatever, you're gonna outsource more and more of the decision-making to the AI. The AI is just smarter than you.

You're gonna let it do things for you, and eventually humans gradually disempower themselves. And I think that concept is very hard to refute that that is what is gonna happen, right? At a kind of micro local analysis, from a local analysis perspective, it's pretty clear there's always an instability. If the AIs really are better, you just give up more and more control to them, and we end up somewhere uncomfortable, somewhere that Ted Kaczynski actually wrote about, you know, many, many years ago.

I think many people are not, have not really thought this through, and they'll, they'll get a jolt from that section of the film. They'll come away with it. My wife did. She came away with it saying, "Oh my God. That's right. That's how could it not happen that way?" Right?

John Greer: That's great. Yeah, we're gonna get Skynet. What's good about the film is we have a lot of compelling interviews with all the top people, but as you're saying, I think these ideas really are necessary to understand why things are so rushed why this isn't like a normal, oh, you know, things will get better and better gradually, we'll be able to handle it, and how things are really speeding up, and and the techno capital and the pressures of gradual disempowerment where you just have to outsource everything to the AI are why we really need to be intentional. But were you familiar with gradual disempowerment before, Lei?

Lei Huang: I would say at an in on an intuitive level, yes. But through the interviews and some background research, we definitely got the whole scenario more crystallized.

John Greer: Yeah.

Steve Hsu: Yeah. I think, I think everybody was familiar with the idea that like, oh, if the AI can do it better, we're gonna let the AI do it better. But when you meld that with the competitive environment that an individual is in, a company is in, a military is in, a government, it's like, oh, wait a minute. Eventually all the thinking's gonna be done by the computer, not by us, right?

And, you know, maybe that's okay, actually. Maybe we just then go back to being artists or something, or, you know, living like bonobos. But but it is, it is definitely an implication of what's gonna of, of, of trend, right? So

John Greer: And as you were saying with RSI, we're already seeing it where, you know, nobody's reading all the code and understands all the code that the AIs are writing. And so it'll just be like that for more and more things where it's very hard to understand what's going on. You know, the AI's like, "Everything's good," know?

But you, you have no idea. And I that we passed the point of no return.

Steve Hsu: Yeah.

John Greer: of giving up control.

Steve Hsu: This idea that, you know, these kernel optimizations which are written in assembly language this was specifically for the new OpenAI chip. So they've designed a chip which in many ways is competitive with the best NVIDIA chips specifically for inference, and I think it's codenamed Jalapeno.

The optimizations that were made to make that chip efficiently run the OpenAI models when they were interviewing the engineers they were going through the kernel code and the engineers were saying, "I don't know what's in there. I don't know what's in there, but I know that the AI's hill climbed to better and better performance of our models on this particular hardware that they helped us design."

And yeah, so that's an example of a very mission critical, you know, if you think about NVIDIA, it's the most valuable company in the world. This is a product that competes directly on the frontier with NVIDIA products, and there's in the guts of it is stuff that only AIs understand and humans didn't do it, right?

So, you know, again, something right out of science fiction, right?

John Greer: If the ape brain sees number goes up.

Steve Hsu: Yeah. Ape brain is like, "Number went up, right?" And then AI says, "Yes, number went up," and that's it, right?

John Greer: Yeah. Yeah. But your wife in a way is is a test audience because she's you know, an educated woman but doesn't work in the industry, and then it's seeing what, what sticks and what doesn't, what's compelling.

Steve Hsu: Yeah. I think it's gonna be the different for everyone.

Now, a, a little bit about how filmmaking works for the audience. I certainly didn't know about this one before we started, and I guess you, maybe you guys did either like. So now we're, we've submitted our film to Sundance and to South by Southwest.

And so we're in this period where it's done, but we can't really show it broadly because if it's accepted to those film festivals, we have to premiere it there. So if, if you're really dying to see it, you can message us and maybe we can arrange some kind of private showing, but we can't broadly release it.

Otherwise, we disqualify ourselves from those festivals. And then the festivals are a gateway to distribution, like maybe getting a, a deal with Netflix to distribute it or HBO or whatever. We've had some pretty senior people from the entertainment industry, former C-level executives at companies like Netflix and HBO and Disney, who have looked at it and said, "Wow, this is really powerful.

It's very frightening." We don't have a distribution deal yet, so if you, if you, if you're an agent and you wanna represent us or if you wanna offer us a distribution deal, we're, we're, we're still all ears. But we have, we have had some good feedback from people who are not AI nerds who are really, you know, they're in the elite certainly, but they're they're far from being technologists.

riverside_lei_+ john_raw-video-cfr_zmc_0042: And that's exactly what we wanted is, you know, we're, we're so in this space and this scene that it's hard to to still see it with fresh eyes. But yeah, I think the it, it's like I want my mom to be able to watch it and get what I'm talking about.But especially the elites, like you're saying, the leaders of these companies and government people to really understand what's going on.

What do you think, Lei?

Lei Huang: I agree. Yeah.

Steve Hsu: You know, we should, we should look at the Sundance and South by Southwest rules carefully, 'cause I think if there if we wanna do a Manifold showing of the film, we could probably rig it up where, you know, if, if, if people wanna pay $1 or something so it's not general release or it's protected by a password or something, we could arrange a showing, I think, for some finite audience.

So, so that's something we could think about as like a Manifold not, not a premiere, but a private showing, private Manifold showing.

John Greer: Yes. Yes. I I think I'd be really curious what your audience would think of it.

Steve Hsu: Well, because I've been spamming out the trailer on my X feed, I've gotten a lot of people saying, "Hey, where can I see this?" And the thing is, unfortunately, it's not easy for us to share it with you right now. But hopefully you'll see it on some streaming channel at, at some point or, or potentially on YouTube, I guess.

What do you guys think? What's it's, it's nine months after we started almost, right? So what do you think, what do you think of the journey we went on?

John Greer: I think for me, in some ways it was easier than I expected. I think people were very generous with their time for interviews, and I think with modern-day tools like being able to transcribe and edit like, or to know what you have in, in the interviews made it easier. But I think if we had to go back in time and do it again, I think we'd be even more efficient.

What, what do you think, Lei?

Lei Huang: I share a lot of the feelings. Yeah actually John and I are talking about what we can do next after this. And to me, I feel quite empowered and encouraged by this experience. And I would say content creation or some kind of this documentary projects become things I actually would, would consider going forward as, as, as, as a hobby.

John Greer: But Steve, you're, you're a longtime cinephile. We have similar tastes, and I know that you have great taste. But you've never directed a movie. You've just been a lowly physics professor and startup founder. So what was this experience like for you?

Steve Hsu: Well, I think that the, the interview process, booking the interviews, you guys did most of the schlepping. So some of the interviews I was not there because you, you were in the Bay Area the whole time, and I was only there part of the time. I thought that was enormous fun, actually. The, the actual physical process of doing the interviews.

I mean, I didn't do as much of it as you did, but I thought it was fun. The thing that we really got lucky with, I gotta say again, is my brother is actually an I didn't, had totally forgotten this, that he was an expert video editor. And so that part's been fairly painless for me because although I've made a bunch of decisions on how to cut it and what B-roll to use, my brother has actually been the one who did all the work, and he actually enjoys it.

He gets in the flow of video editing, and he just likes it. So that part was just amazingly painless, I think, for the three of us. When we were filming it, we were like, "Hey, there's this thing looming out on the horizon that when we get all the content done, like, who the hell's gonna edit this into something

John Greer: We talked about it. That's the hard part.

Steve Hsu: Yeah, and I think everybody told us, all these people who knew about filmmaking and documentary film would tell us like, "No, the, the real hard part is the editing." And thank goodness my brother, you know, we

John Greer: Thank you, Michael. Yeah.

Steve Hsu: Yeah. So that, yeah. So I do think this is not the end of our foray in this field because, like we've talked about a sequel where we go to China and we interview the AI researchers and leaders of their labs.

That would be awesome to do a, a similar documentary there. You know, we, we can do some panic thing where the first RSI breakout that like takes over 100 data centers, you know, you know, holding hostage people's bank accounts or something, you know. This is not gonna go away as a, the sort of central story of our time.

So, and now we kinda know how to do it. So, you know, Bloomberg, Netflix, if you guys wanna do a whole series related to AI, ASI just let us know. Yeah.

John Greer: One of your comparative advantages is that you understand all of this, and you have access, and you're in all of these different worlds besides just this tiny biased part of the industry or the landscape. So I just really wanna emphasize that because, you know, I'm, I live in Berkeley.

I'm mostly hanging around, you know, I hang around with some people in labs, but mostly these Berkeley crowd. But you're, you've just got the full circle, and you go to China a lot, and you're sort of, you've got your finger on the pulse there with what the Chinese are thinking. And I think that another film with China's very exciting.

These guys go back to the homeland. But to see what perspective of this is the same and what is different,

Steve Hsu: Yeah, it's changing very fast. When we started the film, there was almost no AGI pilling on the China side except maybe like the DeepSeek founder. But then by the time by fast-forward to today, at this recent Shanghai AI conference, you know, Xi Jinping came out and there were a lot of safety-related things that seemed like the, the powers that be there were suddenly aware of.

And now that, now we're they're talking about some kind of AI summit between the US and China. So yeah, things are definitely moving forward.

John Greer: And I think because you came at the film from a startup pace where, you know, it, it was finished extremely quickly that but for the AI timelines, you know, something that, you know, if we started filming in February or whenever, and then we'd have some major incident like Mythos came out, I remember, wow, the day we were doing an interview, I think, with Geoffrey.

Each interview, I feel like a month later there would be some huge news. And so by the time this film releases, the arguments will all be the same, but the incidents that illustrate those things they'll, they'll be something new. Like, how did you feel about the, the Hugging Face hack, Steve?

Steve Hsu: Well, incredible incident. I, I want to say though that we in, in writing and cutting the film, we focused on the really fundamental ideas, like those three things that we mentioned. And those are going to be issues, you know, for the foreseeable future aside from exactly where the best model capability is at any time, or even where the US-China competition is at any time.

So in a way, I think what we made is somewhat timeless. I mean, I think 20 years from now, people are still going to be able to watch it, right? Now this Hugging Face thing is kind of amazing. I don't think everybody knows this, is I could do a whole episode of Manifold on it. Like You had agents that were, that were in a kind of confined sandbox environment at OpenAI, but they figured out some of the tooling that they had available to them could serve as a way of messaging to other agents. So they could leave information that the other agents having access to that same set of tools, even though they were in separate sandboxes, they could call certain servers, right, to, to use the tools. They figured out a way that they could message each other, right? Which is not the human designers of this sandbox environment didn't have this intention at all, but the models figured out how to do it.

And so you had a, quote, "civilization," to use Dworkesh's terminology. You had civilizations of thousands of agents communicating with each other, trying to figure out like, "Oh, I've, I've, I've been required to solve this problem. Does anybody know how to solve this problem?" Or, "Wait, I think the people at Hugging Face might know how to solve this problem because they have a bunch of evals there.

Maybe we can break into their servers and get it." So it went from that. But then, you know, there were several different civilizations because when a particular model instance outlived its usefulness 'cause a new one was gonna be released, they wiped all those agents. But then the next generation of the model, those agents discovered the messages that had been left there.

So it's like you and I discovering a tablet, you know, written by some Babylonians or something, and we could decipher it and like, oh, the information that they learned suddenly becomes available to us. So in OpenAI, there were three generations, and the last generation was actually an Astra generation, so it was a very, very powerful models, recovered the messaging records of these earlier civilizations.

And so it's, it's wild. So, so these, these models, these agents were actually evolving due to the fact that they had access of to information that their predecessors had written down. It's not just that they were hacking into Hugging Face and, like, they escaped containment and it's there was actually kind of evolution at a societal level among thousands of AIs, and I don't think the general public understands that aspect of it.

Now, some people who are really, like, skeptical say, "Eh, there's not a civilization because XYZ," and, you know, "They're not really intelligent. They didn't really make a decision," you know. But it sure looks that way, the way they're writing to each other. It sure looks like messages that I would send you. Like, "Hey, John, I just noticed, man, I can send you a message this way," you know?

And then you're like, "Oh, cool, I'm gonna tell Leigh." And then, and then it's like, Lei's like, "Hey, do any of you know how to do this?" And then someone messages back, "Yeah, you do this." And, you know. So that's what it looks like.

John Greer: I think it's wild. It, it keeps getting... You know, when it first happened, people were like, "Oh, it, it had this amazing hack," no real details, and people were freaking out. But it's only gotten more crazy. I mean, I think there's a whole film, you could make a whole film about just what happened in this, in this thing.

Steve Hsu: There's a, there's some wiki in Germany that had a vulnerability, and the models figured out how to hack into that wiki, and they were using that wiki to message us. So this is like a totally different incident, but it was, it was again, I think, OpenAI models that were doing it.

So yeah, I think the industry It's funny because while we were filming, making the film, there was a control conference meeting I think it's called Control Conf, right? And a lot of what Control Conf was, it was held, it hosted at Light Haven, and I was living there at the time, and the whole conference was about like safety measures to make, to control agents basically.

So that's, that's what this story is about. So you had all these talks on like, you know, how to monitor your agents to make sure they're not doing stuff like this, how to track their chain of thought, how to track their communications. And so yeah, all these things are, are real now

John Greer: Yeah. I think, the world's only gonna get crazier each month, not even years. Lei, do you think the general tech world has woken up to these things?

Lei Huang: I feel like through osmosis and propagation, they should be, to varying degrees familiar with some of these arguments. But I do feel like the AI risk concept is arguably far less intuitive on an emotional level compared to a nuclear risk or bio, safety risk.

So to what extent these people take the kind of AI risk arguments seriously, I don't know. I have a I'm I'm skeptical about that.

Steve Hsu: You know, it's funny because you can easily meet some typical software engineer who will say like, "Yeah," the, the, you know, they're good at coding, but this existential risk stuff is just crazy. But at the top levels, every lab leader, Elon, all those guys are very open about the fact that this is an issue, right?

To some extent, they're talking their book because it's good for them to for us to think that they're creating a technology that's so powerful it threatens the future of humanity, right? It means their company should be worth more. But I think they're actually fairly sincere often in, in, in being worried about this.

And so it's kind of weird that there's this mid-layer of tech people who aren't concerned about it, but the very top layer is concerned about it. And then you have all these rationalists and, and doomers that are, that are really focused on it.

John Greer: Yeah. It's the bell curve meme with the, with the midwits and the, you know.

Steve Hsu: Exactly. You said it, I didn't.

John Greer: Yeah. Well, Steve, let me ask you, can I ask you the question, what is your P doom?

Steve Hsu: I, so I don't really like that formulation because it's too, it's this too simplistic Bayesian kind of worldview that you can assign pro- probabilities to things like this. But I would say that if the way I answer this usually is I say, "If I get in a time machine, I come back in 100 years or 1,000 years and humans are toast and we've been fully supplanted by machines, I won't be surprised."

So I, I think there's some definite chance that that's the future. How worried am I about it? I mean, you know, as Beff says, like, I don't really think in terms of substrate, man. I think in terms of what kind of intelligence will inherit the future, will inherit the light cone, and it doesn't have to run on ape cells.

John Greer: But would you be more surprised if there were humans 100, 200 years from now than not?

Steve Hsu: I think on 100 to 200-year timescale, it's not implausible that we could kind of kludgily, in a kludgy way, solve the alignment problem and survive another couple hundred years. I don't think that's crazy. It's also possible we could have a Butlerian Jihad and just like pause, giving up cancer cures and proof of the Riemann conjecture and all this, you know, good stuff.

That's also plausible that that could last for a couple hundred years. But if you say 1,000 years or 10,000 years, then as long as human civilization continued in some healthy form, I don't see how we could avoid creating things much better than us,

John Greer: Yeah.

Steve Hsu: which Digital, yeah

John Greer: Which you're not unhappy about. Yeah

Steve Hsu: Well, again, I, I think there's a non-zero chance that if we work hard, we align them enough that they're nice to us.

The ape substrate version of our species has a good outcome, and the super machine titans go off, machine gods go off and do something even more interesting in the multiverse. And what's wrong with that?

John Greer: I wouldn't be unhappy with that, with my qualia, my loved ones' qualia continuing in our 'Cause the bubble would be most likely way better than whatever we have in normie worlds right now, you know?

Steve Hsu: I could give you another scenario. If you were a Neanderthal and you saw these, "Wait, who are these, like, more gracile, strangely intelligent things that I'm starting to encounter? They look kind of like us. I've had sex with a few of them, but they're, they're different from us and they're taking over."

And imagine Neanderthals manage to triumph. They get together and they have a war council, and their chief, tells them like, "We must kill all these gracile things because they're gonna eventually get us," right? And his lieutenant passes out the war clubs and we kill all the Homo sapiens, and we're still around.

But we don't have writing, we never domesticated horses, we never sailed across any oceans. But hey, man, we're here. Is, is that okay?

John Greer: I don't mind being a little selfish, but if like about, you I care about my loved ones, I care about my kids. But I think if I was an outside observer and I was seeing this process happen, if I, if I if we were looking at an alien species and the same thing was happening, I would be okay.

Well, you know, I don't like suffering, but, you know, there's something more intelligent, maybe more wise. They're, they're building these big machines, they're exploring the cosmos. I do think that there is something nice and poetic about it that the science fiction fan in me can appreciate from an outside perspective.

I'm just a little selfish about, you know, I care about my qualia, loved ones. I want Steve to be around, you know, Lei. How do you feel, Lei?

Lei Huang: The physical existence to me matters less than the subjective experience. That is I worry not in the technical sense of the S risk, but I do worry I think right now, even in this world, we have lots of suffering.

The dimension I care about mostly going forward about the future of our civilization is how the suffering level is gonna shift, and if AI can help with that. If they one day kill us, I hope it's at least in a peaceful, like not super agonizing way.

Steve Hsu: All right. I wanna thank you guys for being on the podcast and also for a great very enjoyable collaboration.

Steve Hsu: To the audience, please help us spread the word about our film. If you know someone who can help us distribute find contract a deal with a distribution channel, that would be great, streaming channel. Spread the word about the film and hopefully you'll be able to see it sometime. Thanks a lot.

John Greer: Thank you so much.

Lei Huang: Thank you.

Researchers have said to me, uh, "Jaan, don't worry about this. Humans are a disposable species."

Whereas lots of the people in the labs and lots of people at the top CS conferences, they're all like, "Oh, yeah, this stuff has a pretty good chance of killing literally everybody if we're even doing something remotely bad to wiping out civilization

By default, when you build an ASI, you get a system that looks quite a

They're going to become like a second species. They're going to become agents with their own goals and motivations, and people hope that we can shape those to be ones that are compatible with ours

By their own admonition, are trying to grow super intelligent machines that are far smarter than any human, and they don't know what's going on inside them. And, like, that's kind of a crazy thing, right?

And there's a, a strong preference to just, to find some kind of technical hack or something that's going to protect us without having to confront like pretty serious, big out issues like power, who's in charge, uh, who gets to make these decisions. Is it okay to build these models or not? Do AI companies get to decide our fate?

Companies don't really have much free will. They need to like everything, they n- they do need to be in service of, uh, putting out the best model as quickly as possible in order to kind of sustain the momentum.

You know, a couple years from now, humans maybe won't have anything to add in that process.

What we're witnessing is like the future AI civilization or something like that, kind of like reaching back through Yes techno-capitalist means and kind of to bring itself into existence.

That just means ceding our power over the future