Subscribe
Share
Share
Embed
Steve talks with Skype founder and global tech investor Jaan Tallinn.
Show Notes
Steve talks with Skype founder and global tech investor Jaan Tallinn. Will the coronavirus pandemic lead to better planning for future global risks? Jaan gives his list of top existential risks and describes his efforts to call attention to AI risk. They discuss AGI, the Simulation Question, the Fermi Paradox and how these are all connected. Do we live in a simulation of a quantum multiverse?
Rationality
Additional Resources
Creators & Guests
What is Manifold?
Steve Hsu is Professor of Theoretical Physics and Computational Mathematics, Science, and Engineering at Michigan State University. Join him for wide-ranging conversations with leading writers, scientists, technologists, academics, entrepreneurs, investors, and more.
Steve: Our guest today is Jaan Tallinn. Jaan was trained in theoretical physics in Estonia. He was a founder of Skype, and is a prominent super angel-investor in technology start-ups all around the world. As an example, he was an early investor in the AI company, DeepMind, which was later acquired by Google. DeepMind is famous for many things, but one of the things they’re famous for is having created the first super-human Go AI called AlphaGo.
Steve: I’d like to start, Jaan, with just some personal details about you for our audience. Our audience is mostly, I would say, it’s some distribution of scientists and university professors, technologists, start-up people … but it’s very diverse. Also, some writers and artists and things like this. So not everyone will be familiar with you, so I just thought I’d go with your background just a little bit.
Steve: Now, my impression of you, if I were to describe you to other people, is that you’re basically, at this point in your life, living … You’re doing what I would call living the dream. Because you live in your beautiful home in Tallinn, Estonia, but oftentimes, you seem to be jet-setting around the world. And I mostly run into you at really fancy meetings full of all kinds of interesting people discussing interesting topics. But maybe you could just describe for a moment, what a day or a month or a year is like for you these days.
Jaan: Yeah. First of all, thanks for having me. Yeah, I travel about 40% of my time. And when I’m home, I work from home. So there’s very little impact that this current crisis is having on me, except that the travel is not happening, really. And I’m going to try to maximize the amount of time that I spend on conversing with people whose ability to think, I respect.
Steve: So, one of the focuses of yours that I think we’re going to spend maybe most of the time on, is existential risk for humanity. You’re trying to do good for humanity by addressing and raising awareness about existential risk. If you were to categorize the various buckets in which you’re spending your time right now, how much of it is on that specific thing, versus looking at companies, versus other activities?
Jaan: For the last decade or so, it has been sort of 50/50 on investing and professional time. Half of it goes to investing, and half of it goes into preparing talks, for example, or doing philanthropy, et cetera. And for the last year or two, I’ve been trying to reduce the amount of time that I spend on investing, so I have hired someone who’s doing most of the leg work investing, in order to grant me more time for actually saving the world.
Steve: Yes. I think when I first met you, I was really … obviously I knew about Skype and things like your tech start-up activities. But I was really impressed that you seem to have reached a stage in your life where you’re really actually at least 50% focused on trying to save the world. And very few people, even people who are billionaires, and have all the time in the world, and resources to do it, are trying to do that in the effective way that you are. So that was what, I think, impressed me the most.
Jaan: I’m kind of surprised how few people there are. Like one of my first meetings, when I got caught into this existential risk reduction business, was with Peter Thiel. And if I remember correctly, he said that, “Look, it’s going to be … ” I went to him with a laundry list of items that I could be doing with my time and brand, and very high up there was, I should be trying to recruit wealthy people for this cause, because nobody seems to be paying attention to that.
Jaan: And if I remember correctly, Peter said this: “You’re not going to be successful doing that like this. There’s no point in talking to them.” And then, he was right. I ended up talking to about a thousand billionaires or so, and yeah, I had very little to show for it. [inaudible] interest was like Elon Musk and Dustin Moskovitz, they got interested in this cause completely independently of me.
Steve: I see. This COVID-19 situation is an interesting test, because here, of course, the risk is, it’s not really existential. But it’s very concrete. So the question is, what fraction of billionaires … and I think already there’s some positive signals from people like Bill Gates and some others, that there will be some participation by the fantastically wealthy, to help solve this problem. But of course, existential risk is something where we may be looking out 100 years, or 200 years, and trying to do something now. And I think that’s much harder for even these smart billionaires, to wrap their brains around.
Jaan: Oh, yeah. The big problem with so-called tail risks, risks that are conceptually there, but we have no experience with them, is that they’re very hard to intuitively grasp, and be … I mean, even I find it difficult to have … be concerned to my system one, so to speak, that there might be a big danger just across the horizon. In that sense, I do think that the COVID crisis is, I call it minimum viable global catastrophe, because it totally qualifies, I think, as a global catastrophe while still having relatively mild, still, effects. Although, this is of course, relative. I’m just linking that in terms of possible global catastrophe that we could be having, this is one of the mildest.
Steve: Yeah. Someone actually claimed to me, or advocated the viewpoint, that this is actually okay. It’s bad in that a lot of people are going to die, and there’s some problems for the economy. But in the long run, it’s quite good, because as you said, it’s a minimum viable crisis. So, in terms of the number of people that are going to die, and the economic damage, it’s just enough that it’ll focus attention, at least on this particular risk of pandemics, and maybe a more dangerous kind of virus in the future. But we will be able to get through it, and maybe we’ll build systems. We’ll put some systems in place that’ll be useful when something much more serious comes along.
Jaan: Exactly.
Steve: So, do you have any personal, maybe non-obvious, to use the dialogue terminology … any non-obvious near-term implications of coronavirus that, they’re not widely understood?
Jaan: I do think and hope that one day … I’m not sure if it’s very non-obvious, but I do think that there is this value of having something external to humanity having like species-wide problem, and something that’s external to human affairs, is calling the bullshit so to speak. So if somebody’s making outrageously wrong predictions, you will see that they are wrong in just a few weeks’ time. Conversely, it kind of surfaces good, clear thinking, and good leadership. Because the effects of that are going to be seen very quickly. In that sense, it’s like a war. Right?
Jaan: War is obviously a horrible thing, but the sort of good side effect of that, of a war, is that you will get leaders that are able to save lives, because if … just go through as generals until we find someone who will win battles. So I do think that there might be, although obviously much more milder, effect from this crisis than it would be from an actual war.
Jaan: Whoops, I lost your audio.
Steve: Oh, okay. Let me just repeat that. I forgot what I said. I was saying that this is a very interesting experiment, because every country in the world is affected. So after the whole crisis is over, you’ll be able to conclude who did the right thing, who was properly calibrated, who understood things the fastest, and reacted effectively.
Steve: The problem I have is that, other than a small group of rationalists and very sort of data-driven smart people, the rest of the world might not take the best essence from this. And there will be very active propagandizing and retconning of history and events. People have very strong motivations to do this. So while the story, I think, is knowable, that the analysis of it is knowable, whether it will be known by large numbers of people on the planet, I’m not sure.
Jaan: Oh, yeah. I agree. There’s definitely a lot of incentives to misrepresent the history, et cetera, once this is over. However, the nice thing is that people who care actually, about the truth, they can go through the history and look up who the good leaders were. In that sense, at least, in this smaller circle, there will be knowledge, like who seem to have done better job than the others. And if there’s a mixed catastrophe, in that sense, the world will be somewhat better prepared, because we know who managed to do it well
Steve: Yeah, I agree with that. And this is a little bit ghoulish, but I was just thinking that after going through an intense period of time, where I myself was kind of trying to analyze the epidemiology, and trying to figure out what are the proper well-calibrated projections one can make about this … A lot of my thinking is recorded on my blog, so people can go and check on it.
Steve: But I started to think this is going to be a longer-term thing, and maybe there are actually some pretty huge start-up opportunities coming from this. Because if you properly predict what the world will be like one or two years from now, post-COVID, maybe there are some rather obvious start-up ideas, which could be big winners. And the people who act on those, who are properly calibrated and act on those ideas, are going to win. Have you given that any thought?
Jaan: A little bit, but I’ve done it most in the context of what public stock will do well.
Steve: Yes.
Jaan: And there’s certain, like a sub-section of the rationality community, that I know that have been giving a lot of thought into what the proper prices, how the stock market should react. And then basically, having … literally profiting from the fact that they didn’t, which, an interesting fact, that the markets generally are regarded very highly in rationality circles. But now they were, people were like, “Wait a minute. Why aren’t they reacting?” Like, “Okay, I guess I’d better make my bets now.”
Steve: Yeah. This is a very interesting point. I’m a little bit older than I think the main body of people that are in the rationality movement, so I’ve seen a couple of financial crises before. In fact, did some kind of deep analysis of some of them. So I was fully aware that efficiency of the market is limited. And it can mis-price assets for very long periods of time. So, I wasn’t surprised.
Steve: One of the very first bits of frenetic analysis I had to do when I first heard about all this, was to figure out when I should get out of … You know, sell all my equity positions and go to cash. That was the very first thing I had to figure out what to do. I think I performed reasonably well. I basically sold very close to the peak, actually. So that turned out pretty well for me.
Steve: But one point I want to make for our listeners is … we have a lot of academics and traditional, kind of more mainstream scientists and such, that listen to this podcast. And I detect that there isn’t a very strong cross-fertilization between what I call the rationalist movement, which is typically younger people.
Steve: Maybe a little more focused in tech, but who are really focused on trying to perfect their calibration, reasoning about the world, how to do Bayesian updates, how to really cross this data properly and come to the most robust conclusions … There isn’t that much cross-fertilization between these other communities, which I think would be very open to the rationalist worldview. But they’re just isn’t … they don’t tend to go online and read those websites that the rationalists are on.
Steve: And this even extends to, for example, the hedge fund community. Which obviously clearly needs good predictions for what’s going to happen. So one advertisement, maybe we’ll put some links in the show notes to some of the rationalist websites. I’ll encourage some of the scientists and academics and others to go there, read their stuff.
Jaan: Yeah, I think the LessWrong website and community has been doing a significantly above average job, when it comes to digesting the information from the current crisis and then in forming interesting opinions.
Steve: Right. And even a third sliver of culture here, is the super … which overlaps a little bit with the other ones that I mentioned, but there’s also the Superforecaster community. And I think last time I checked, maybe that was like a month ago or a little more, it seemed to me their essential values of their predictions were pretty well calibrated. In other words, they seemed to have predictions by mid-February or early March, that looked to me like they’re going to be reasonably correct. So, it’s a whole other community that’s trying to get things right.
Jaan: And a third one is Meticulous, that I am also a small investor there. But actually, one of the concrete things that I did in this crisis, I just gave them a donation, so they could better build up their specific pandemic-focused predictor market there.
Steve: Right. I’m going to tell a little bit of a story that contains a little bit of confidential information. But I think our listeners will like it, so maybe you’ll like it too, Jaan. It has to do with the British response, the UK response, to this crisis. And this has been reported somewhat in the media, somewhat intensely in the UK media. But because there are invested interests there, I don’t think it was reported really very accurately. But I have some insight into this.
Steve: What happened there is they had a pre-existing pandemic plan, which was formulated I think as early as 2011. But then it was re-studied again in around 2014, 2016, in actual, official government-sponsored studies and simulations. That’s where this herd immunity idea came from. That maybe the smartest thing to do would just be to let it … You know, cocoon the most vulnerable people, and let it sweep through the younger, less vulnerable population, and then you wouldn’t have to stop the economy to deal with the virus.
Steve: So, the present government basically inherited that plan from previous administrations. It was embedded already in the civil service, et cetera, et cetera. And it was only through very kind of rationalist thinking, where they realized, “Okay, but these early analyses have a problem, that they were simulating kind of a flu. They didn’t actually know what we were going to confronting now. And there were key parameters that were off, relative to coronavirus, like this need for ventilators.
Steve: So they, after a while, realized … and there was a very complicated back and forth struggle, which was really updating on data among different groups in the government … that led them to eventually move away from this herd immunity plan that they had inherited. And it really was a real-time situation, where a government was trying to figure out what was going on.
Steve: They had a plan in place, but it was the wrong plan. And they had to figure out, both through simulation and through what physicists would call Fermi estimates, of just trying to get the most robust prediction of what’s going to happen based on their early data. But they finally then, moved away from herd immunity. And they’re basically executing kind of lockdown, like most countries are, now.
Jaan: Yeah. When I heard about their original plan, my reaction was that I’m glad on a mental level, that there are countries who are doing some other things than the rest of the world. So we will see [inaudible] incorrect. It wasn’t obvious at all to me that the initial plan was incorrect.
Jaan: Because even if you think about, like what are the actual numbers, if we assume that the infection fatality rate is something like 1% all-in, then that gives you roughly the same mortality that you get in the world naturally, like 1% of people. So, it’s not like obvious that you should be super afraid of it. But there are second order questions indeed, like what kind of effect, what the second order effects will be, that might tip the scales.
Steve: I would even say, although it’s very cold-blooded, that is you do a full-blown cost-benefit analysis, and you price lives, or you price quality adjusted years of life qualities, that you could argue that our response has been somewhat overboard on this. Maybe from a purely numerical perspective, we should have let it sweep through, and just done our best to cocoon the vulnerable people. You could make that argument. If someone really wants to make that argument, I wouldn’t call them a monster or logically incorrect. They may just have different parameter values for pricing qualities than me.
Jaan: I agree.
Steve: And the additional subtlety of this is if you’re a politician, you might just say, “Look, I don’t want this really traumatic stuff to be happening during my term as prime minister or president.” So you might err very strongly toward the side of just having as little trauma as possible for your society, even if this cost-benefit analysis doesn’t justify it.
Jaan: Yeah. That seems true, as well. In that sense, I do think that it’s interesting to see you got differences between democratic countries and authoritarian countries. Although I’m not sure how big their … I mean, authoritarians still have to … big enough. They’re not like completely free in their actions and policies.
Steve: Yeah. I think it’s exactly right. I think that one level of analysis might have said, “Hey, the popularly elected people have to really worry about this issue of optics.” But in China also, I think she was very worried that he would get tarred with … if they had responded poorly to this, then that would be a big huge stain on his record, too.
Steve: So everybody has to worry about that, I think. Except maybe like in North Korea, or something. Do you have any more thoughts on coronavirus? Do you want to maybe give us your central sort of prediction for what the world will look like in two months, in six months, in one year?
Jaan: Yeah. I’m more than a little concerned that this lockdown regime will have to last much longer than I would like. It might have to last for like more than a year, because of the vaccine isn’t there. Before the vaccine, there isn’t really a big reason. I hope that I’m wrong, that treatments will be available, that will take the edge off the virus.
Jaan: I do think that it’s valuable to compare the crisis to a previous global crisis like World War II. And unless I’m wrong, I think the world bounced back pretty quickly after World War II. So in that sense, I’m not super concerned about long-term effects on economy. I do think that there will be long-term effects on various sectors on economy.
Jaan: I think it’s totally possible that the commercial air traffic will take a long time to recover, as people get settling into their new forms of doing work. Even so, there’s like [inaudible] a joke recently that … there was a questionnaire, like “Who was responsible for your policy of working from home? CEO, CTO, or COVID-19?”
Steve: Yeah. Just to throw out a couple of ideas … One is, I think there’s a pretty decent chance that we might not get a vaccine for a year or well over a year. It’ll be interesting to see whether governments get into a kind of Manhattan Project mode, where they’re really trying out-of-the-box things, and expending large-scale resources to try to get the vaccine here earlier. And maybe even be much more willing to take risks in testing the safety and efficacy of the vaccine. So that’s an interesting question for me.
Steve: Another one is whether the remote education, remote meetings, video conferencing, all of that … because of this event, everybody’s exposed to it, and it passes a tipping point where it becomes much more acceptable. Virtual conferences, virtual ways of interacting, I think people may develop. So that’s an interesting question for me.
Steve: And then a third interesting question for me is how this will affect relationships between the West and China. Who will win the propaganda war for pinning blame on this, for the virus, and what it’ll do to globalization, and things like that.
Jaan: Yeah. Personally, I really, really hope that having encountered a species-wide problem gives a good angle for people to advocate greater cooperation among species, like global cooperation. But yeah, it remains to be seen, how it will fall out in the end.
Steve: I think that having struggled with this, in a sense, kind of small non-existential crisis, I hope it will leave us with better systems in place to deal with the big ones that are likely to come. And I think probably, public intellectuals and leaders like yourself should be out there emphasizing that this was a predicted and predictable risk that actually materialized. So maybe it’ll make people a little more comfortable in going further out in the tale, or further forward in time, in preparation and planning for [crosstalk].
Jaan: I think it’s … like one of the struggles, especially in AI risk community, the things that we struggle with is this, I got this universal, “Oh, this is just science fiction” counter-argument. I do think that this crisis will give some counter-arguments to that, that everything entail risks, or science fiction risks.
Steve: Right. The vivid one I remember is people … I forgot who said this. I think it was probably a pretty prominent AI researcher, who said this. It’s like, “Existential AI risk is like worrying about the overpopulation on Mars,” or something like this.
Jaan: Andrew Ng.
Steve: Oh, was it him? Okay, yes. Yeah. So let’s turn to that. Because my sense is, correct me if I’m wrong … that AI risk is the main existential risk that you’re concerned with. Because I think you’ve also spoken about biological existential risk, but I think AI risk is the one that you spend most of your cycles thinking about.
Jaan: Yes. I do think that the top three risks that might wipe out the humanity, in my view, are AI, bio, and unknown unknowns. Because both AI and bio are less than 100 years old, when it comes to considering them as global risks.
Steve: Where do you put just the old one, that’s been around for a while, just nuclear war?
Jaan: Based on what I know right now, I think it’s very unlikely to be an existential risk. It turns out, if you even … if you detonate all the nuclear arsenal that humanity has, it will be … definitely will not kill everyone by just, from the first order effects. There might be some second order effects that would lead to collapse of civilization, and then going out with a whimper, so to speak.
Jaan: But of course, there’s still like non-zero chance that people having … analyzing this, are wrong. So nuclear still is like non-zero existential risk. Interestingly, I think nuclear was the first existential risk that humanity encountered. And I think there was a tech-gate in the 30s, from the mid-30s to mid-40s, during which nuclear was was like full-blown existential risk. Because that was the risk between … that was the period between the invention of the nuclear chain reaction, and the first test of the nuclear chain reaction.
Jaan: It wasn’t clear whether this planet would survive the first nuclear detonation. And there was the first existentialist report that humanity has produced. It’s called LA-602, concerns about the ignition of the atmosphere. That concern was that that first nuclear detonation would cause the nitrogen in the atmosphere to fuse and thereby basically turn the entire plant into a nitrogen bomb, basically.
Steve: Yeah. [inaudible] my central projection would be, nuclear war would be really terrible for well-being, and would set civilization back potentially quite a lot. But as a true existential risk, it doesn’t seem likely that even a full-blown nuclear exchange, say between the US and the Russians, or China … would actually eliminate humanity from the earth. We would maybe take a long time to recover from it, but I don’t think it’s truly existential.
Jaan: Another evidence for that is that the humanity has done a lot of nuclear blasts already, in terms of testing. So, the environment had been able to absorb them over time, at least. For what it’s worth, that’s another evidence.
Steve: Yes. Now back to that, what you were calling maybe the first existential risk report. I have to share a little anecdote. That analysis of whether the first nuclear explosion could ignite this chain reaction with nitrogen in the atmosphere, burning the atmosphere … That was a serious thing to be considered, because at that time, knowledge was very limited as to the physics of how this all worked.
Steve: And that calculation was done in a room which was next to Oppenheimer’s office on the 4th floor, 3rd or 4th floor, of the Physics building at Berkeley, called LeConte Hall, where I spent … I did my PhD there. And at the time, when I was a grad student, those offices, the old Oppenheimer offices, had been turned into offices for graduate students.
Steve: I didn’t have an office there, but I had friends who would sit in this office where … I don’t know if the blackboard was still the same blackboard, but it was that room where basically I’d done all these calculations for that very first existential risk analysis of nuclear weapons. So I actually spent a lot of time in that room.
Jaan: That’s an awesome story. I do think that it’s like a super commendable thing that they did. Once in a while, I’m not going to mention any names, but there’s this argument that, “oh, there have always been like doomsday predictors, and that therefore, if next one comes along, you just shouldn’t listen to them.” I think the same argument can be shown to be incorrect when you try to apply it to the Manhattan Project people trying to do the analysis, whether we are actually going to blow up the planet.
Jaan: Because it’s clear that if you go to that room to talk about, and say like, “Look, there have always been the doomsday sayers. Why are you here?” Like, “Should have been doing something productive.” That argument just doesn’t cut when you have a good reason to be concerned.
Steve: Yeah. In this podcast series, we’ve done quite a bit on climate science. And I frankly don’t see that as anything close to an existential risk. Because … well, okay, there’s some parameter space where maybe there’s some very non-linear evolution of the climate, and that it really is a problem for the human species, but seems like that risk is relatively low. And yeah, you might argue that the people that are really passionately pushing climate change risk as the most important thing, should actually reorient a little bit, and think more heavily about things like AI.
Jaan: Yeah. Another approach to that being … another angle that I’ve been taking is to reframe AI risk in terms of concern about the environment. Because I do think that if AI ends up being … manifesting as existential risk, it will manifest through sudden changes in the environment. Or put it yet another way, like the climate models that indicate the concern about the future, they implicitly assume that humans are still in charge over the climate in 50 or 100 years.
Jaan: And I think it’s not obvious at all that we will be. And if we are not in charge, well, we should project this concern to AI. And AI, whether … We know that humans, as long as we can stay in our biological form, we will continue to be concerned about the environment. But if not, AI will not be concerned about the environment at all. The reason why we send robots to space and radioactive areas, is that they do not care about the environment.
Steve: And they, in fact, may want to continue spewing huge amounts of carbon into the atmosphere, so that they can power their calculations at giant data centers.
Jaan: I think it’s more likely that they will just get rid of the atmosphere. Because almost all engineering that they might want to do is probably easier to do in a vacuum. And also, correct me if I’m wrong, but I think it’s easier to get rid of the waste heat if you don’t have atmosphere between-
Steve: Yeah, that’s probably right. Let’s back up a second for our listeners, who maybe are not all very familiar with this concept of AI risk and the singularity. Maybe we could start with the most basic pitch you give. So, you’re at some meeting and Joe Billionaire or Joe President of Big Country ends up sitting next to you at a dinner table, and asks you what you work on. What’s the elevator introduction to this subject that you give to those people?
Jaan: I have a different pitches when it comes to that. Sometimes I have even done this thing that, where I ask people two questions. A: Do you have children? And B: Can you program? Then I have like four different quadrants-
Steve: You have to fine tune the message.
Jaan: Yeah. Then I can fine tune the message, based on those two bits of information. And those are my pitch and my metaphors I’ve evolved over time. But I think one of the recent ones that I’ve been trying on people is that … There’s this game in rationality community called Taboo the Word. The idea is that words have different meanings, or different connotations. So in order to make the communication clearer, you might want to change the … just taboo the word, and use something, different words in its stead, that don’t have this connotation.
Jaan: So, one way of looking at AI, and this become the start of the pitch, is that it’s the process of delegating human decisions to machines. And there is this intrinsic trade-off between trying to be more productive, more competent, more capable, and staying in control. Every CEO knows that. Whenever they need to get more stuff done, they need to delegate. Yet, whenever they delegate it, they yield control, and they have less control over what’s going to happen in the future.
Jaan: In some ways, humanity’s that CEO. We are in the process of reducing the control that we have over the future in order to get more stuff done. And you can see, can imagine, this slider that basically drops over time as it proceed to the future. And the problem is that there might be some critical areas where we really, really want to remain under control.
Jaan: One is AI development itself. Once we no longer are in charge of AI development, well, the smartest systems here will no longer be human-controlled or human-developed, even. So, it’s hard to imagine what they’re going to do. The second thing is the control over the environment. We really need those narrow parameters so [inaudible] remaining space remain in place. If we are going to delegate away control over the environment, we might go extinct in a matter of minutes.
Steve: I like that way of introducing it a lot, because it doesn’t have to invoke this leap to AGI. Right? What I think a lot of people who are not super technical, and even many people who are super technical, have a problem with … is that leap to AGI. And if you just say, “We have these mechanistic systems or digital systems that we’re handing control off to,” at some point, it becomes an issue. And the trend is obviously clear, the way it’s heading. So we should just, need to think harder about that. I like that very much.
Jaan: The most recent minimum viable danger is AI definition I heard from, was from my friend Andrew Critch. He defines an AI, he called pre-potent AI, and there are basically just two assumptions there. One is that this AI is going to have as big impact on the environment as humanity has had, at least as big.
Jaan: And second is, that it’s unstoppable. It’s either unstoppable because it’s too smart for us to stop it, or there’s systemic reasons, that like just, it’s very hard to stop internet, for example. I think it’s still possible to stop internet if you really wanted to, as a species. But it’s going to be really hard, and it’s not obviously doable. So if you just have those two things, this logically implies that the effect on environment is going to be just as big as humans are already concerned with.
Steve: Now, some people would already want to argue that something like free market capitalism or neoliberalism, is already that kind of thing. That it’s, independent of the actions of George Soros or Bernie Sanders, or whatever … There is this monolithic thing which is building more and more powerful markets and machines and things around the Earth, and impacting the environment. And humans are, in a sense, kind of powerless to stop it. Because it-
Jaan: Yeah. I think that in some ways, they would be right. But a really big thing is that the system that has human components in it still works at the speed of human thinking. So if you’re going to replace wet carbon with silicon, you might be facing one billion time speed-ups that are off by a factor of one billion. So in that sense, the climate change will no longer be like a problem of the century. It might be a problem of today.
Steve: Yes. Although there is a continuum between that sort of mushy what-where, one hundred millisecond time scale, to the super fast silicon time scale. Because humans now are relying … we have this kind of cyborg situation, where humans are relying, for at least chunks of their thinking or decision-making, on silicon. So there’s a continuum as we head in that direction.
Jaan: Yeah. AI impact actually has an organization called AI Impact, has done some very interesting analysis when it comes to trying to look at what are the discontinuities that we might be seeing from continued technological progress, and looking at what were the past discontinuities … I think that because one, they found it wasn’t explosive power, that the introduction of nuclear detonation caused something like, was it like four or six hundred years of progress?
Steve: Yes.
Jaan: … over a couple of years?
Steve: Yes. So, in the spirit of rationalism, I think we mentioned Andrew Ng, whom we both know, a moment ago. He’s a smart guy. Right? We might disagree with him, but he’s a smart guy. What is the strongest argument against being concerned about AI risk?
Jaan: I have two arguments that I would be least surprised if they were … ended up being true. And one of them is a totally crazy one. And one of them is a very reasonable one. The reasonable one is that, look, we are facing the world with many, many problems. And AI could be super helpful in alleviating those, and addressing those. In fact, that’s one of the reasons why I’m focusing on AI safety, rather than bio-safety.
Jaan: Because if we fix all bio risk, we still have the AI risk to contend with. However, if we fix AI risk, and we’ll get very powerful AIs, we’re probably going to be able to also fix the other risks, including bio risk. So I think that is a reasonable thing, and we might just alter so many problems that currently the world is burdened with.
Jaan: So in that sense, if you’d actually do the EV calculation, depending on what your parameters are, you might end up in a situation where it’s just worth taking the risk of killing everyone. Which, I don’t believe that that argument is true, but I certainly wouldn’t give like zero percent to that being true.
Jaan: And the other, like the really crazy one, I think one way of putting it is that humans might be very cheap to keep around.
Steve: And we’re fun.
Jaan: Yeah. For AI, we are not fun. We’re just like statues, standing in place, and not doing anything. But yeah, if you look around, if the universe is real, then almost all of the resources are outside of this planet. So, the reason why … I think AI’s almost entirely interested in the rest of the universe rather than the Earth. The big problem is that it will, by default, use as many resources as it can on this planet, in order to get to the rest of the resources out there.
Jaan: And there are some reasons why … I’ve heard some interesting arguments why moral realism might be true. So there might be theoretically, a lot of … somebody who is interested in tiny marginal gains, at the cost of peak loses for other agents, then if you have this trait, it means that you are much harder to cooperate with.
Jaan: Therefore, if you have some uncertainty about the need of future cooperation, for example, the AI might not be the first one. There’s this wonderful story by Scott Alexander, called Demiurge. This older brother, as he describes the situation where the AI wakes up and starts thinking about, “I’m not the first one here, probably not. What should I do?” And if an AI actually ends up reasoning theoretically, what are the correct things to do, I would not be super surprised if it discovers that being nice to other agents is reasonable when cheap.
Steve: Right. I like that train of thought. Just to throw a couple of things out there, this idea that most of the resources are far from Earth. And there might be some initial transition period where, to get off Earth, AI has to do stuff, and maybe that has bad side effects for humans, or something. But in the long run, the AI’s going to maybe potentially go off into the rest of the universe.
Steve: There’s an old but very, pretty well known in physics calculation, that Dyson did. Freeman Dyson just passed away. But he was trying to understand what is the total amount of computation that could be done in the universe, before the end of time. And is it infinite or finite?
Steve: So, he starts just asking these questions of, “Okay. If you’re an expanding universe, and entropy is taking over eventually, how many flops, or how much calculation could you actually do? How much subjective time could the AI experience before the end of the universe?” And it really focuses the idea that the AI will be focused on these much more long-term mega-scale things, then necessarily what’s happening on Earth. So, I like that earlier part of your perspective.
Jaan: And once more, as we are in this mindset, there are some other interesting observation that like there’s a fundamental trade-off in computer science, between memory-heavy and the CPO-heavy computations. It’s possible that you treat the functions, or the goals of AI, that kind of emerge in different parts of the universe if its sufficiently large. They will actually enough differ, whether that the goals are memory-heavy or … that is, memory-heavy meaning basically, you need a lot of matter-
Steve: Yes.
Jaan: … or they are kind of CPO-heavy, computation-heavy, in which means that you need a lot of time. So there might be kind of interesting trading happening between different super-intelligent AIs, if they find themselves places in the universe that are incompatible, relatively incompatible with their utility function, that they end up [inaudible].
Jaan: Anyway, this is a philosophical narrative that I find super fascinating to think about, kind of prime representative of ideas that I find … There is a like a category of ideas which are … A, Crazy sounding, but B, super important if true. And I naturally find myself very attracted to those ideas.
Steve: Yeah. Well, just as a meta-remark, it’s amazing to me how few people though, allow themselves to actually expend a fair amount of effort or energy in that direction that you just described. It’s pretty natural for people who do like theoretical physics, or theoretical computer science or something, to be willing to think about these things. But so many scientists I meet are so practical or focused on their day-to-day lab problem, that they just don’t allow themselves ever to even head in that direction, even for short periods of time.
Jaan: First of all, I definitely do have this luxury of not being on economic rails and having to solve a stream of incoming problems like every day. On the other hand, I do notice that there’s like a significant over-representation of physicists in these AI safety/philosophy circles, which I find curious.
Steve: Well, I think we like, and are willing to examine things from first principles. So it’s very similar to why this question of “Oh, how much computation could happen before the universe dies of heat death?” That’s very similar. Right? So yeah, I think it’s not too surprising to me that physicists are interested.
Jaan: Another kind of interesting observation … again, I think I heard it from Andrew Critch first of all, is that physicists routinely use this concept of face space-
Steve: Yes.
Jaan: … or state space, where you basically have this abstract space that describes the current parameters of your system. And then as the system evolves, it controls a line in this parameter, parametric space, or face space. And if you can imagine the planet, like all the temperature is like one axis, and humidity’s another axis, et cetera.
Jaan: You put Earth in this state space, and then you look at the evolution of our time. There is going to be this line in this space, and around it is a narrow tunnel, which corresponds to parameters, within which, human survival is possible. The game is, don’t touch the wall.
Steve: Yeah. Yeah, exactly. There’s a region, a favorable region that you just don’t want to leave.
Jaan: Yeah, yeah, yeah. That’s very physicist kind of thought, even though Andrew Critch is a mathematician.
Steve: Yeah. The other thing you, in your earlier remarks, said, I wanted to tease out a little bit, which I really liked … I this question of the character of the early AIs, when they emerge and what they’re like, or maybe … You said it in terms of their own game theory analysis, or maybe that was Scott Alexander.
Steve: But it seems to me, going back to the earlier days of what Eliezer Yudkowski was trying to do, where his people tried to do provably safe AI, that kind of mathematics. I was always very sort of pessimistic that anything like that would actually be possible. And I could give you some arguments for why, but it seems like, rather than make sure that we have a provably safe AI, this is the idea that the AI has some kind of character that makes it a little bit more beneficent, or a little bit more favorable toward humans.
Steve: That little sliver of difference might make all the difference in the end, as to whether humans survive. As they’re leaving the planet, they’re nice to us. That seems more likely to be decisive to me than the other thing.
Jaan: I think the most general way of putting this thing is that we want to be able to predict some invariance about AI. Or like this ecosystem of AIs, once it starts self-generating. I got a theory; every once in a while I come back to this argument I had almost 10 years ago with a pretty prominent AGI developer who basically wants to accelerate the AI timelines. And he said to me that, “Jaan, this AI safety concern is completely pointless. Because once we have AIs that are smarter than us, what they will do will be as unpredictable for us than what we will do is unpredictable to rabbits.”
Jaan: Then my question was like, “Wait a minute. Why are you involved in a project that is trying to randomize the world?” Shouldn’t he be trying to make the world a better place, rather than a random place? So I do think that this, the responsibility of technology developers in general, think about not only effects, but also side effects, or potential bad effects of their work.
Steve: Right. I agree with you, that this unpredictability is the very central aspect to this. So I think this provably safe AI direction just is unlikely to bear fruit.
Jaan: To defend this a little bit, I think you might have provably safe AIs in a way that you have provably very limited AIs. So you can constrain their effects on the state. So, yeah, I could have a provably safe AI. I’m not sure if I would call it AI. But I’m going to prove that this thing, given those constraints, like this IO behavior, is just not going to do certain things.
Steve: Yes.
Jaan: Of course, there are some assumptions that always have to go into your proofs, but just as an example of provably safe AI doesn’t necessarily mean that these completely unconstrained things, unconstrained systems that they have proven is good for humanity. It might be also proofs of very hard constraints.
Steve: Yes. I agree with that. But going back to the psychology of this AGI researcher that you were talking to … I don’t know if you said 10 years ago, but some time ago. Both in my self and in others, who develop new technology, encounter this kind of, I don’t know what to say … this sort of Nietzschian or Promethean idea that, “You know, look, this is just an amazing hack, or amazing gizmo. Let’s just build it. And let’s not worry about the risk, because it’s just so cool, I want to do it. It’s just an expression of my human Promethean spirit, that we’re just going to … ” “Oh, fire. Wow, that could be pretty dangerous.” “No, no, no. We got to do it.”
Steve: And you could even refine that a little by saying like, “Well, who cares about human utility? I’ll just transfer my utility to the existence of these better beings. If we make them, and they go on, it doesn’t really matter what happens to us. I’m just as happy for that timeline, or that future path for the universe.”
Steve: Do you think that’s the right characterization of the psychology of these people? I mean, obviously, there could be some people just don’t believe in AI risks. They’re just totally unconcerned about it for that reason. But some people, I think, are fully aware. They fully intellectually grasp that there is this risk and unpredictability there. But they just can’t stop themselves from pushing as hard as they can still, in that direction.
Jaan: I think there are two separate arguments there. One is that it just feels really cool to invent new things, and I totally identify with that. In fact, when I was at DeepMind, I was going to … Constantly, I was struggling with how interesting the discussions were, and the inventions were. On the other hand, trying to wear my safety hat and kind of just shaving off years from the timeline, every time they invented something. So I’m very familiar with that. And I really don’t blame other people for feeling like that.
Jaan: This thing that, “Oh, we’re just step in evolution, and we’re going to create something that’s better.” That piece is just pure rationalization to me. It’s usually very seldom people have thought, put more than 10 minutes thought into that argument.
Jaan: And I think it’s just the burden of proof will lie with them, if they claim that this thing is going to be good. I call into human values or some kind of general values. Then they have to show why. They are making a prediction. And I’m claiming that this is sort of … what’s the word? How Carl Sagan put it. Extraordinary claims require extraordinary evidence. They’re making an extraordinary claim.
Steve: The period of history which reminds me very much of what’s going on now, is when they made the jump between the ordinary fission bomb to the hydrogen bomb, which was much more powerful. In fact, in a sense, unlimited in its power, potentially. There was a group of physicists who said, “Why do we possibly need this hydrogen bomb, other than as a weapon of murder? What could possibly come from having this?”
Steve: So there was some disagreement between how hard the US should pursue that next step of development. But what ultimately happened when they came upon this so-called Ulam Teller design, this innovation that’s … clearly, people saw it had the potential of actually solving a technical problem, Oppenheimer, who had been an opponent of going that next step … moral, ethical opponent of it … said, “Well, you know what? When you see something that’s technically that sweet, you just have to do it, and then worry about the consequences later.”
Steve: He literally said that. So I think that’s, in a way, “Oh, if it’s technically sweet … ” Like, you show me some really beautiful arrangement of neural nets that, “Wow, this will really be able to insubstantiate conscious self-awareness in the thing.” “Oh, yeah, we got to do it. How could we not do that? We got to check that out.”
Jaan: Yeah. That strikes me like, thinking that is completely understandable psychologically, but somewhat more than lacking when it comes to consequentialism.
Steve: Yes. Now, you’re familiar with the Fermi Paradox, about why other-
Jaan: Yes, I’m aware.
Steve: Yeah. So, do you think AI risk is … is it plausible for you that AI risk is one of the … Well, actually, how do you regard the interplay between development of AI and the Fermi Paradox?
Jaan: My views have evolved over time, and I expect them possibly to continue evolving. My current position is that we are just alone in this Hubble volume. So the reason like any significant interplay between AI … I mean, there is this interplay, like going back to our earlier point about AI’s reasoning about other AIs, if we are alone in this Hubble volume, then this AI will [inaudible]… the probability of it actually encountering other AIs in this Hubble volume is going to be very low.
Jaan: So there should be like second order effects from it to reason about theoretic situation where it turns up in making this kind of … The argument that it will keep humans around because it’s cheap and theoretically correct thing to do, this argument, it makes it weaker. So, yeah, I think there are some interplays there, but I don’t expect strong interplay. I definitely don’t expect right now that AI risk is something that can routinely terminate civilizations because then we will just see AI as coming [crosstalk].
Steve: Yeah. That’s where I was heading. But then I realized, yeah, but then we would see AIs, unless they, for some reason they’re hiding from us, or something.
Jaan: Yeah. In general, I do think that if we expect the universe to be teeming with pre-spacefaring civilizations, then I don’t think any particular scenario will be like a argument why we don’t see anyone. Because if there are many of them, some of them will evade these scenarios.
Steve: Let me throw another one at you. And I want to be respectful of your time, so we probably don’t have too many more minutes. But if you believe in the possibility of AGI, whether it’s 100 years from now, or 1000 years from now, it seems to me … and just tell me how convincing you find this argument … that if you believe in that trajectory, then the idea that in our future light cone, there will eventually be lots of AIs, but then also potentially lots of simulations of less capable beings, automatons …
Steve: If you think that’s a plausible future light cone for us, it seems to raise substantially the odds that we might ourselves be in a simulation right now. And I’m curious how you feel about that.
Jaan: Oh, 50/50.
Steve: Okay. That’s fair. That’s fair.
Jaan: I find that there is … I do believe two different arguments, and they sort of cancel out, when it comes to intuitive surprise. One is, I do believe in simulation argument. I do think that there’s a lot of historical simulations that are happening in the multiverse. But also, believe in multiverse.
Jaan: I do think at least, even … Like my friend Max Tegmark has this entire categorization of different multiverses. And you only need one of them to be true. You just need the universe to be sufficiently large for every once in a while, there being like a podcast like that. So, if the universe is sufficient large in some dimensions, then some of our copies are simulated. Some of our copies are real. So, it makes this in a single universe, there is a super dramatic question: Are you simulated or not? Is like one bit of information, so therefore, it’s like very … makes you kind of uneasy. Or if you start thinking of yourself as a group of people that are just spread over the universe, or over the multiverse, then yes, some of them are simulated and some of them are not. It’s just a fraction, therefore less dramatic question.
Steve: Yeah. I agree with you. And I actually, some time ago, I wrote some blog posts, formalizing this a little bit. Where, if you give the standard simulation argument, but you have in mind classical computers that the simulations would be run on … Given that, it seems plausible now that quantum computers will eventually be made and will work, one should think this through a little bit more.
Steve: And then it seems like, in addition to having some non-trivial probability that we live in a simulation, there’s this non-trivial probability that we live in a simulation in a quantum computer, in which case our simulation is a multiverse, and a quantum multiverse. So that didn’t seem to have been considered very much, because I think it was only just as AI became more and more plausible as technology got better, it’s only very recently that super powerful quantum computers, which … inside them, if you understand quantum computing, you realize inside of them, there’s an un-decohered coherent state inside … un-decohered pure state inside that giant super quantum computer. And that thing is actually then, a quantum multiverse. Hasn’t been discussed-
Jaan: Although …
Steve: Go ahead.
Jaan: Although, I do think that it’s, if you start counting simulations and if you assume that there is concern about resource use is universal, then I guess you should expect most of your copies to be located in universes that are fairly poor resolution, when it comes to micro-world. It kind of like … Yeah. I mean, I’m a former game developer. And what I do in games, it’s just like you simulate just the field of view. And what happens behind your back is just very poorly, roughly simulated. So there might be a lot of things that feel real to conscious observers, but when you look at the mechanics of simulation, it’s just like smoke and mirrors.
Steve: Yep. I agree with that. Actually, to get into the real theoretical physics weeds, I wrote some papers with a guy called Tony Zee, who’s another well known theoretical physicist, about this question of if in fact there is a kind of resource constraint, are there implications of that on the true nature of Hilbert space. Whether Hilbert space really is this fully continuous complex space, or whether it could be … have some hidden discretization that we haven’t found yet, even in the structure of Hilbert space.
Steve: And the other place where it has some implication is in the measure problem in many-worlds quantum mechanics. Where it’s not clear how the measure is imposed. The measure might be imposed by resource constraints by the thing which is actually doing the simulation. Anyway, it’s getting a little bit into the esoteric, there.
Jaan: It’s getting too interesting.
Steve: Yeah. I think we’re over time, so let me conclude by saying it’s been great having you on the show. We’d love to have you back some other time. And anything you want to throw out there that we should put in the show notes … Given that our audience is not primarily rationalist. A lot of people in academia, a lot of scientists. A lot of writers and journalists and things like that. What’s a thing that you’d like them to read about AI existential risk, or-
Jaan: I do have this link collection that I update every once in a while. It’s almost like a meta-link collection. Because it includes links to link collections. So yeah, it’s just JaanTollinnonline/xrisk.
Steve: Great. We will include that our show notes. And I want to thank you, and I hope that we get out from under this COVID problem so you can return to your super awesome lifestyle, traveling the world.
Jaan: Thank you very much.
Steve: All right. Take care, Jaan.
Jaan: You, too. Bye.
Steve: Bye.