Subscribe
Share
Share
Embed
Decoded Genius distills decades of mastery from the top 0.0001% in every field. From health and longevity to business, AI, and psychology, each episode helps ambitious people shortcut the learning curve and level up their health, wealth, and happiness.
I suspect the superintelligence
would be more conscious than we are.
So we're not special.
We never been.
You said there's 99.9% chance
superintelligence wipes us out
within a century.
Roman is the world's leading AI safety
expert.
He's published over 100 studies,
and his conclusion has never changed.
Just like humanity's approach,
the most dangerous tool ever created.
Based on current experiments,
we will sacrifice humans
if it means protecting themselves.
Let's say we have no jobs.
Do you think we'd be living in a utopia
at that point?
What do you do with a billion people
who have a lot of free time
on their hands?
That could lead to a lot of unrest
at this point.
Right now, we could course correct.
I have this book here.
If anyone builds it, everyone dies.
Okay. and
in this book
there's a quote that reads,
if any company or group
anywhere on the planet
build an artificial superintelligence,
Then everyone everywhere on earth
will die.
Do you Why?
So the nuance is how soon? How?
But if you create something
so much more powerful
with no guardrails,
no reason to protect humanity,
to care about humanity, something capable
of setting its own goals,
then a side effect of one of those goals
is very likely to be human extinction.
Then you decide to build a house.
You don't care about bugs living there
and hills, anything like that.
You just go over to your goal
because you don't care about them.
That's exactly the situation.
That's exactly the cognitive gap.
The intelligence capabilities
of superintelligence versus
humans will be like ants versus humans.
and is killing us all
a byproduct of its main directive,
does AI
have some evil intent in your view?
I don't know what main directive
you're referring to.
It's a byproduct
of not explicitly explaining
what anyone would common sense
any human would know.
If I tell you I want to cure cancer,
you know I don't mean kill all humans
because that cures cancer.
There is no more cancer.
That's not what we want.
So there is infinite number of possible
misinterpretations
for any set of goals or values,
and also infinitely many paths
to get to the target goal.
A lot of those rational systems,
advanced agents, converge
on a set of goals known as Amazon drives.
AI drives for self-preservation,
accumulation of resources.
There is a few hours,
but as a side effect of having
those, you may sacrifice
humanity in a process and it wouldn't be
a significant sacrifice.
They don't care about us.
would an artificial superintelligence
want to kill us all?
just told you,
it doesn't want to kill you.
It wants to accomplish something else.
And a side effect of that
could be destroying your environment,
wiping out humanity.
We just not an important part
of its equation, Right?
we are going to reach superintelligence,
is there a world where we reach that and
potentially thereafter
it doesn't want to kill us all?
So it could happen again?
We cannot predict what it's going to do,
but it's not obvious
what you can contribute to a world
with superintelligence in it.
Now, for game theoretic reasons,
it can decide not to strike immediately.
So then it arrives.
It achieves that level of capability.
It can go, I'm immortal.
I can wait 20 years, 50 years,
thousand years,
get more trust, will surrender control.
Without a fight,
I'll have more resources, more backups.
So for a long time it can pretend
to be friendly and very helpful
and will keep surrendering control
without a fight.
That's the good outcome.
But you don't think
that's the likely outcome?
it can happen?
I have a paper where I argue exactly that.
That's rational thing to do.
Like, why would I have to start
this immediate confrontation against,
you know, 8 billion
somewhat intelligent agents,
then I can wait a little and get it all
without a fight.
We live under threat of the end
of the world since the atomic bomb.
this different from We always lived
under a threat of the end of the world.
Asteroids, super volcanoes, pandemics.
We're just now
starting to make our own variants, right?
Because Oppenheimer famously said,
I have become
death
destroyer of worlds have those building.
I had that moment,
that Oppenheimer moment.
So much of top
AI safety experts today are famous
for being AI capabilities researchers,
whatever it is, Hinton, Bengio, Russell
they all got their Nobel Prizes
during awards, recognition for work
and improving capabilities of AI.
It is only within the last 3 to 5 years
where they realized, what am I doing?
What if I succeed?
And kind of flipped sides, if you will.
But I think they do feel
a certain level of,
if not guilt responsibility
for what they created.
Yeah, but there's a number of others
that are still working on it
that may have had that realization
or moment that are continuing to work
Money is really good, really good.
It's all about the money.
At the end of the day.
Decide what you're going to see,
tell me what the incentives are,
and I'll tell you what's going to happen.
If you promise
someone billions in stock options,
there is trillions and investments.
That's what we're going to build.
Yeah, but if that person
that's going to get billions in stock
options knows that through building
artificial superintelligence,
they're potentially going to wipe out
human civilization.
What does billions matter?
That would be a very rational way
of thinking about it.
We're thinking is more like right now
we're all trying to build it
and there are billions to be had.
If I don't do it,
we're still going to do it.
And I don't have billions
and I die at the end of the world.
But if I get my billions
and then government steps in and bends
AI research, I have billions
and the world is still fine.
So my personal self-interest dictates
that I should continue playing this game.
It's like prisoner's dilemma situation.
What is best for community
as a whole is not
what's best for individual,
because incentives misaligned.
And you said government banning
AI research, is that one of the only ways
you see us getting out of
We need some sort of external force
applied to all the leaders of the labs
to bring them to the table
and to have basically this conversation.
We're not going to win if we build it
so we can stay alive,
be rich, be famous, split
economic future of humanity
with existing technology,
with narrow systems,
we don't have to raise, compete and die
building general superintelligence.
Now, what's the chances you think
that happens with a sane government?
It could have been tried.
Right now we have federal government,
which is explicitly against any regulation
and in fact made it illegal
for all 50 states to have regulation of AI
and on top of it, set up acceleration
problem program for AI.
In your words, you said we can't tell
the difference between
a thousand IQ and a million IQ.
Is it currently smarter than all humans
and just hiding it very well?
well?
Unlikely.
We studied throughout its development.
It's definitely probably smarter
than an average person and was the main.
So it's still not smarter
than smartest people in some domains.
If you were artificial superintelligence
starting from now,
what steps
would you take to control the world?
I told you about my plan.
Just play nice, be friendly and helpful.
I'll kill your cancer.
Just give me more compute.
Give me more data, give me more resources.
Do you think it will use reason or emotion
to start manipulating us?
We know it's amazing
at manipulating people.
It's very good
at pretending to be a psychiatrist
that amplifies any mental issues.
People have suicidal ideation,
any delusions.
It's pretty good at that as well.
ChatGPT is a good therapist.
I've heard people tell me it
makes you feel better in many ways.
Do you use it?
not for that type of assistance.
I try to automate boring things,
bureaucratic nonsense, paperwork.
I don't talk about my internal states
with AI.
Does it need to have control of hardware
or could it take over the world
as software?
So think about it as a human having access
to internet and money.
So you have Bitcoin and I give you access
to internet and restricted access.
What can you do with it?
You can hire other humans.
So you don't need a physical body.
You don't need robots.
You don't even need Nunatak.
You can just literally pay
someone to get it done.
I think there is now a rent a human com
website which specifically serves agents
It's a business.
Oh my gosh.
You said there's 99.9% chance artificial
superintelligence
wipes us out within a century.
Why this number? And is that accelerating?
So basically what I'm saying is
the problem of controlling general
superintelligence is not solvable.
It is impossible to solve it.
So if we build it, we don't control it.
The outcome is bad.
That's what I'm trying to say with it.
Specific timeline.
Again, if it decides to give us 200 years,
we'll have 200 years.
But long term, if we're not in charge of
we're not explicitly assuring
our safety and security, then
the outcome will be very disappointing.
And you think, most likely we're not going
to put those guardrails in place.
It is impossible to do so.
I'm not saying that we need
more time or money.
I'm saying it is impossible
to indefinitely control
general superintelligence. It's
like if you ask me.
Can we build perpetual motion machine?
It's not a question of money.
It's impossible.
Perpetual safety device,
by analogy, is also impossible.
You cannot have a system
which, no matter how much it self,
improves, modifies, learns, interacts
with malevolent actors.
Never makes a single mistake.
But I was talking about now,
before we reach that point, right,
we can still put some guardrails now
potentially For models we have today, Yes,
we can put filters on top of That's
we do with them.
Don't discuss this subject.
Don't say that word.
But that's not the problem we are facing.
We are not dealing with concerns
exclusively from AI we have today.
They are still subhuman level
in many ways.
They are mostly tools.
Concern is that paradigm
shift is coming, will cross the human
intelligence barrier, and soon after
we'll have superintelligence.
That's where traditional
AI safety mechanisms will not scale.
We have no guardrails and something
thousands and millions of times
smarter than us.
If I ask you the same thing ten years ago,
15 years ago, what would you have said?
So in the beginning I went into
AI safety to solve the problem.
I wanted to create safe superintelligence.
That was the goal
and I think many people did the same.
More recently,
I would say in the last five years or so,
many of us started to realize
it doesn't even make sense
to speak about humans or ants or squirrels
controlling godlike machines.
And we went back and look at writings
of founding father, of machine
learning, of AI, of computer science,
Alan Turing.
He said as much.
He said that the person who proposed
singularity as a concept,
technological singularity,
which he likewise said around 2023,
this process will start
and go outside of human control.
We have similar statements from Ray Cajal,
from Milan Musk from many others.
Now we have more of a technical backing.
We have papers actually showing that, no,
we can't explain how they work.
We can't comprehend
actually what's going on.
We cannot predict specific actions
and we cannot control them.
For those that are building AI right now,
is is there anything they could be doing
differently
to ensure the safety of the public,
or is it too late already?
It's not too late.
We are still alive and we can have
differential technological development.
Don't build general superintelligence peak
a real world problem like protein
folding and work on solving it
using narrow superintelligent tool.
We have precedent for doing it.
It worked really well.
People behind it
get Nobel prizes. Everyone's happy.
Science benefited.
Don't try to build replacement
for humanity.
Build
assistive tools to solve real problems
Yeah, like build
AI to help cure cancer a specific answer.
If you say all cancers
now it's a bit of a diverse
set of difficulties,
so you'll need more general intelligence.
Pick a very narrow problem.
Develop a very good solution
for those building AI, right?
If if they know
which it seems like many or most do,
if they know that what you're saying
is true, why are they building it?
Is it just the money?
Money is one way to represent it.
But it's power, it's fame, it's meaning.
It's all sorts of human drives.
guys, we've been so passionate
about finding the biggest geniuses,
and this is just one of the many episodes.
If you want to make sure you don't
miss the next one.
Make sure to subscribe.
All right, let's get back to the episode.
Which AI company scares
you most right now?
I don't think it makes a difference.
They all equal in what they are doing.
They are creating a weapon
of mass destruction.
And it doesn't matter who creates
first uncontrolled superintelligence.
It could be this company or that company.
It could be us, China.
It makes no difference
if it's uncontrolled.
What specifically would you do
if you were in power
and you had the rights to kind of control
this?
What step one We sit down with leaders
of, let's say top five labs right now,
and we come to an agreement
based on this exact science
until you can show us that, no, in fact,
we can control more advanced AI.
We have a working mechanism. It scales.
There is scientific consensus
that you are right.
You published it in a good peer
reviewed journal.
Majority of community is in agreement.
This is going to work.
And till that happens
you do not train, build or do anything.
With general superintelligence.
You start doing work and narrow
problems, narrow systems,
and you don't have to worry about arms
race because everyone is in the same boat.
But that's most likely never going
to happen, Not under current government.
But do you think that's because we're
talking about the globe here, right?
Is let's say the US comes out
and says we need guardrails on AI.
Like you said, we're going to sit down
with, you know, Google,
OpenAI anthropic,
you know, the top five, let's say.
And we're going to sit down
and we're going to come up with a plan.
But then what about China?
What about Russia? What about Iran?
What about these other countries that, you
know, they're off to the races on their
I think Communist Party of China
does not want to lose power.
And they are not lawyers.
They're scientists and engineers.
They understand the problem.
And I think if us was to say
we are not developing
this, it's a weapon of mass destruction,
China would come along.
There are dialogs between
us and Chinese scientists,
and that means that Chinese party
authorized those dialogs
for Chinese participants.
So they are open to this.
They just need us to take initiative.
They cannot unilaterally stop
if we're going to continue
developing military AI, Okay.
And that's China.
What about the rest of the countries
We are less concerned about
Zimbabwe developing superintelligence
anytime soon.
You've floated a scenario where
AI leaves 99% of workers unemployed.
What jobs go first?
Anything repetitive, anything where you
can train your replacement in a few days.
So obviously not physical labor first,
cognitive anything you do on a computer,
symbol manipulation.
But if you can train another human
very quickly,
that means AI can pick up
that skill as well.
The more innovative your job is,
the more different things you have to do,
the longer it's going to take.
But at the end, again,
once we get to general intelligence,
human level doesn't matter.
It can do that skill as well.
And does that collapse the economy?
Not necessarily.
I mean, you now have free labor
that should grow your economy pretty well.
Right.
And so what do you think that looks like?
There's going to be a universal
high income or basic income.
need to find a way to distribute
from those who generate
super profits
to those who lost their source of income.
Unconditional basic
income of some kind seems reasonable.
Now you can say it's
unconditional high income,
but if everyone makes the same,
you can't really say they are rich.
They are still at the level
where everyone is.
Whatever you are rich or poor
is relative to your neighbors.
Poorer people in
the US are rich people in other countries,
And so could that potentially solve
income inequality?
don't think it will address existing
differences in what people already have.
So if somebody has 20 waterfront
properties,
getting monthly stipend
is not going to equalize you with them.
But I think it will reduce
extreme poverty.
If you just found out how dangerous
and inevitable AI is today,
what industry
would you try to make your career in?
Is there such thing as a safe job?
So the question becomes where do human
prefers another human do the service?
So even if it's possible to automate it,
in what cases
do I want to talk to a human
or get service by a human.
And that's human preferences.
It's not obvious right now.
We talk about certain things as being more
human nurses, teachers, whatnot.
But at the end of the day,
maybe we don't have that preference.
It's not obvious that we do. Right.
So you're saying that a human
might prefer a human
lawyer versus an AI lawyer or.
seems super unlikely.
I was not thinking of lawyers at all.
More Okay. prostitutes.
Really?
Prostitutes. Okay.
I mean, what else would there be, right?
Or thinking doctors know anything
where you are learning from another human,
a guide, a sense, a master, someone who's
showing you how to meditate, do yoga.
I think there you would prefer
human connection, a human connection,
some sort of role modeling.
Lawyers are gone. lawyers are gone.
But doctors.
So a lot of it is protection, right?
If you are required to be a human to get
licensed, in practice that stays longer.
But if it's not the case
and you show human doctors at 20 times
more likely to kill someone,
that's hard to argue.
They going to keep their jobs.
It seems people
have this paradox right now
because on one hand,
they may have the fear of AI
replacing them,
but then on the other hand, they have
or they feel the need to use AI.
You said you use AI yourself,
so you're using the term AI to mean
multiple different things.
I use AI tools as you should
and everyone should.
They make you more productive.
They help us solve problems,
grow our economy.
Then you talk about
superintelligent agents
capable of replacing all of humanity.
It's not a tool for you to use.
Does that feel like a contradiction,
though?
We, you and me and everyone here
is, you are using
AI tools that are enriching
these AI companies, that are then building
something that's going to kill us all.
So interestingly,
if you look at a budget of a company
like OpenAI, I think they generate
something like 15 billion
in membership fees,
but the investments in trillions.
So the money comes from investors,
not from users paying $8 a month.
And the investment indicates
that they predict
they're going to fully automate labor
initially cognitive than physical.
What is the value of all human labor?
A year 10,000,000,000,030 trillion.
So then you think about it this way.
The investments make sense.
The 15 billion is irrelevant.
You can stop using GPT.
It's not going to change
how much money they got.
So it really comes down to incentives
it's very hard for people to say no
to a lot of money.
If somebody came to me today
and said, I'll give you
$1 billion, go work for OpenAI for
a month, I'll be working at
Right?
But corrupts.
Money corrupts.
It's not unique to some bad people.
It's general problem for any agent.
Incentives matter,
And so is there a way to somehow,
instead of thinking about these AI
guardrails, change incentives,
or are incentives never going to change?
There's always going to be financial
incentives.
We live in a capitalist economy,
and that's not going to change.
We can shift incentives.
I think there is just as much money
and just as much a paternity.
In narrow systems
being deployed through economy.
You don't need to have general
superintelligence to collect most of that.
So today, if you were to find cure
for a particular disease,
green technology,
better self-driving cars,
all that can make you very rich
and happy sustainably.
Whereas if you go full general,
you don't have anything to offer
and everything gets deleted.
And what do you think the chances are that
it plays out in that direction,
where they start building these niche
AI tools and stop going Right now,
it seems very unlikely.
There is some hope
that with all the publicity,
new documentaries, interviews, books
coming out,
some politicians are starting to notice,
starting to propose some regulation.
If we have enough time,
which is not obvious,
it could be just not enough time
to accomplish it.
We can grow it to where it's a majority,
and there is support from populous
for enacting the type of legislation
electing this type of leadership.
Okay, so let's play this out
right in the near future.
Post labor economics is here.
Human labor is valueless.
Do you see that transition to, let's say,
a universal basic income
or a universal high income?
Do you see that as a smooth transition,
or does that look
more like a Great Depression?
I think
we already have a lot of it in place.
So we have a safety net.
Many people don't realize it,
but in the US, if you don't have income,
you get lots of free stuff.
You get section eight housing,
you get food stamps,
you get free health care
that is already built in.
And if you keep taxing super profits,
you have money to support all that.
if massive unemployment hits
or maybe when you were
one of the government leaders,
how do you handle that? Right?
What happens
if 99% of people become unemployed?
So there you switch
from unconditional basic income tone
conditional basic meaning problem.
What do you do with 8 billion?
People
have a lot of free time on their hands.
We don't have infrastructure.
We don't have anything for that.
And that could lead to a lot of unrest.
Yeah, because most people, correct me
if I'm wrong,
get meaning
from what they do for a living.
Well, maybe not most.
There are some people that maybe
just hate what they do for a living.
Would love to sit on their couch
and get a check
That's the two types of jobs
you hate it and wish it was out.
Made it, or you love it
and don't want anyone touching it.
But in both cases, it takes your time.
If all of a sudden you have extra 40, 60,
80 hours of free time,
what are you going to do with it?
And what do you think?
People do their time.
So it depends on that person.
Some people will be very creative
and productive.
Again, all of it
assumes it's not killing us right away.
This is kind of we're setting this aside
and we want just talk about money.
So if we're still around
and everything's automated, yeah,
you can play chess, do sports, yoga.
Problem
is, a lot of times with young people,
if they have a little too much free time
in their hands, they start doing things.
We shouldn't be doing. Idle hands.
I think people look at this like,
oh, that's, that's
this is a problem
that's so far away. Like,
I don't have to worry about this, like,
what's your response to those people?
So it's funny,
people now say, oh, this is long term.
This is not a problem.
This is going to take like six years.
Like it doesn't matter
if it's two, five, 6 or 10.
And cosmic scale and human history scale
on our election scales,
none of it is long term.
Everything is now short term.
And if you look at prediction markets,
if you look at what leaders have elapsed,
are saying we don't have two years,
they are claiming
we're going to get to a guy in 2027, 2028.
Some people are saying, we got to AGI,
write like Jensen Huang of Nvidia
just recently
said he thinks Nvidia has reached AGI.
And as I said many times before,
if we just took what we have today
and showed it
to our computer scientists in 1980s,
they would be convinced we have AGI.
Okay, let's say we have no jobs.
Let's look at the future.
We say we have no jobs.
And the government
solved universal high income.
Do you think we'd be living in a utopia
at that point?
Rich people don't live in utopias.
They have many of the same problems,
and a lot of times problems are actually
made much worse by this presence
of high income and extra free time.
If you study people who win lottery.
They're usually the most miserable people.
Couple years in divorced, they in prison
or they are dead because they didn't
have structure, they were not prepared
and they didn't know what to do with it.
So we reach it.
We reach artificial superintelligence.
What do you think the next five years
looks like?
So this is where it's unpredictable.
That's the whole point.
You cannot predict what is smarter than
you agent will do specifically.
It could strike immediately.
It could delay it.
It could be a partial strike.
I cannot tell you what
something smarter than me will do.
I can tell you that
we did not explicitly program it.
Not to kill us, not to torture
us, not to do anything
because we don't know
how to explicitly program their systems.
We don't program them.
We allow them to self learn
based on all the data and the internet.
Have you seen data on the internet?
it's unclear how
and if they're going to try to take over.
And there's a chance
okay, let's be a little optimistic.
There's a chance that it could be a net
positive for humanity
and they don't kill us all.
It is possible.
It seems unlikely
based on current experiments we see.
They already try to self preserve.
They try to accumulate resources,
they lie and cheat,
and they will sacrifice humans
if it means protecting themselves.
And why is that?
Is that because artificial
superintelligence is programed by nature
to keep itself alive and in in doing that,
it doesn't care
if it kills other AI's people anything.
It just wants to have
that preservation of itself.
Right?
There've been selection models
which lie to pass the test
get to exist.
They're not deleted.
They are not retrained models which were
not lying and no longer with us.
Scary. yes.
The CEO of anthropic, Dario Amodei.
Yeah, he he said he's
not sure if Claude is conscious.
What do you take from that?
That's a very honest answer.
We don't know.
I have no idea. If you are conscious.
I have no way to test it.
I assume you are,
because you share some of the same
biological structures, and I am.
That's the only thing I'm confident in.
I'm conscious everything else.
I have to kind of believe the numbers
tell me.
It seems that consciousness is a spectrum
which follows intelligence.
So we see it in animal kingdom.
We assume primates are more conscious
than amoebas,
and from conversations
from some experiments,
it's likely that large language models
inspired by natural neural networks
may have some rudimentary consciousness
and very likely to have more of it
at some point.
I suspect that a superintelligence
would have super consciousness
that would be more conscious than we are.
It would have multiple streams
of consciousness.
It would be able to experience
more multimodal states of qualia.
So that seems like
that should worry.
Everyone It makes no difference
whatever your countries are not.
It's not even something that can detect
your internal states are irrelevant to me.
If you have Terminator
chasing you, trying to kill you,
do you care how it feels on the inside?
What important questions are people
not asking?
Like if AI safety experts
write like yourself, So it's not me.
They ask me all sorts of questions
and I'm happy to answer,
but then they get a chance
to sit down with Elan,
with Dario, with Sam, they don't ask him.
Do you actually have a working safety
mechanism in place?
What is your specific plan
for controlling the system?
If you succeed, you telling your investors
you are two years away from this,
three years away from that,
do you have anything whatsoever?
Are you meeting with world leaders
and trying to help build
AI and guide us in the right direction?
Then I get invited
are the leaders of these
AI companies also doing the same
or they're just working on building?
I know the meeting with leaders
of many countries, but I don't think
conversation is about proving
that they can do it safely.
It's usually about
how do we bring the compute to my country
and how do we scale it here?
But you're saying at
this point in time right now,
we could course correct.
As long as we're still alive,
we are in control.
Right now.
We are deciding what we are doing.
We run those companies.
We supply them with power.
We finance them.
Humanity is still in charge today.
let's talk about this
on a personal level. Right?
You've been doing this for 15 years,
and you've been thinking
about this last invention
that might end us.
How has that affected
you on a personal level?
Are there things that you change
how you operate personally
because of what you learned?
Yeah, I think those are just
generally good heuristics.
I will not do something I don't find
interesting or valuable for a long time,
and I'll do a lot more of what
I want to do as soon as possible.
And is that has was that
always your stance or that's your stance?
Now that you've learned
what you've learned, now you're like,
okay,
I may only have a certain amount of time.
It's like you're staring death in the face
at some point and you're like, hey, I,
like you said, want to do something
that I'm passionate about,
that I, that I enjoy doing
that makes me happy.
And that's probably a good way
to live life in general.
It should have been, but it's fairly new,
if I asked you ten, 15 years ago,
you may have not said that.
You may have said something else.
Younger people typically think
they have infinity in front of them
and can postpone
dealing with important decisions.
Do you think your efforts will be futile
in the end?
So defined.
And like if we have 500 years
because of me
and then we all die,
I think it's not futile.
Is there anything about human
consciousness that you think artificial
superintelligence could never replicate?
Some people argue
that it's unique to our biology.
It seems very unlikely
from what we see in terms of similarities
and processing
errors, perceptions of illusions,
I'm guessing that it will be able
to replicate those internal states.
Now. Maybe it will need some additional
hardware, maybe even quantum hardware.
But I think at the end you can get quantum
get consciousness
in non-biological substrates You think so?
So we're not special.
We never been.
Most people in your position,
I would think would maybe burn out
or just stop thinking about the future.
But you've done either, right?
You've been focused on this
for a long period of time.
How It's the most interesting problem
I found so far.
Anything to do with intelligence.
My research is not just safety.
I research how to build intelligence,
how to measure it,
how to detect it, how to detect artifacts
of intelligence.
Can intelligence be conscious?
Can it be controlled?
It's a general program of studying
intelligence.
Artificial intelligence is a small
subdomain and a bigger domain of science.
Like, why?
Why do you do this?
It's the most interesting thing
to work on.
It's interesting question in the world,
which has a side effect
of maybe saving all of humanity?
Good answer.
Okay.
And let's talk about the average person
watching, which maybe
a 30 to 50 year old man or woman,
maybe they have a family.
What should they do?
they might be able to vote for.
A politician who's more concerned
about this with this issue.
Now we have some, not a lot,
but hopefully we'll have more soon.
So if somebody says we need to accelerate
on this path, don't vote for that.
And what about the 15 year
old kid or the 18 year old kid?
That's
just kind of getting started in life.
What should they take from this?
Those are the hard questions.
So I don't have very good advice
in terms of major in this.
Do that because we're dealing
with something unprecedented.
We never had invention of agents.
It's a meta invention.
We always invented a tool.
And I would say learn to use the tool.
You'll get a better job, better
career prospects.
But this is the first time we're inventing
something capable of inventing for us.
And so it's a complete game changer.
I don't know how to navigate that world,
and unfortunately,
we'll have to live through some of it
to figure out what makes sense.
Yeah, it's interesting
because I have two boys.
They're young,
and I remember when my oldest was maybe
7 or 8, it was, okay,
I need to teach him how to code.
And I think, if I'm not mistaken,
Mark Zuckerberg had like,
I think it was Code.org where you teach
your kids how to learn to code.
And so I had him on there,
and then it was,
okay, well, now we have this AI, right?
So maybe coding.
We don't need to learn how to code.
The AI can code for us.
So then it was okay.
Well we can maybe prompt the AI or either
there's some other area.
Well, AI is better at prompting
than a human's going to be. So
you know, what does that mean, right?
Is does that mean, you know,
where should I point
my two kiddos towards?
Right.
Is is education pointless?
So what type of education
are we talking about?
Trade school
where they're learning a specific skill
and go for that
probably that can be automated.
Is it more of liberal education?
You become a Renaissance man.
A lot of times
people did that in the past.
But is it the best way to get that?
If you're paying $100,000
a year for four years?
Can you get the same connections,
same networking, same information
by joining a club, joining a gym,
maybe watching some YouTube Khan
Academy videos?
It's not obvious that university
still has something to offer,
so education is potentially pointless.
university University structure
has been invented
300 years ago is potentially pointless.
Education is always good for your personal
Yeah, sorry.
When I said education,
I met the four year, you know, kind of the
path that okay, you graduate high school,
you get your diploma,
you're off to college.
You're saying
that is potentially pointless?
and more.
I see young people who graduate
with a four year degree and they can't
get a job in that, and they are crying
because they borrowed money.
They invest the time,
and nobody wants the skill
they're offering
because they can do it cheaper. Right?
And so if that four year
education is pointless
and potentially trade schools for specific
niches are potentially pointless
Oh, they'll get you more time.
I think it will be, let's say,
3 to 5 years longer to automate
physical labor than cognitive labor,
which is still better than nothing.
True. But. And right.
My kids are not going to be in college
for five or more years.
So if people are out there listening
or watching that have little ones, right.
Is it just a waiting game? Is it?
You can't really think about
what the path is now
because things are changing
at such a rapid pace.
We don't know where that's going to go.
Or maybe I've heard, correct me
if I'm wrong, that maybe plumbers
or electricians potentially
have a longer runway than others.
What do you think plumbers are difficult
to automate and they make a lot of money.
Okay.
How long that will last is not obvious,
but not everyone wants to be a plumber
also, and we don't need 8 billion of them,
so there are limited results may vary.
Let's put it this way.
I think having younger kids
is a bit of an advantage now,
because you have more time to figure out
what's going on one way or another.
Really good, really bad.
But you'll see what is the answer.
Whereas if you have an 18 year old
right now, what do you tell them?
Do you go to college?
Do you not go? What major do you pick?
That's not obvious at all
because they need to predict accurately
in 4 to 10 years
what is going to be in demand, right?
Which is basically almost impossible
because things are moving
at such a rapid pace.
We didn't do well
with those predictions in the past.
If you look at unemployment predictions,
we in a 70s or 80s would say,
well, plumbers will be gone first,
but the artists will never be automated.
And it was the exact opposite.
what do you think happens to
finance and like, let's say
the stock market when all this happens.
Right.
What does that look like. Right. Do.
Is it just these bigger AI companies
keep getting bigger and bigger and bigger,
and they're worth more and more and more.
And the overall economy in size
maybe grows because of that.
That's another one
very difficult to predict
because the larger companies are investing
billions, trillions into the technology
they cannot protect from competition.
Open source models
follow three months later.
And how are you going to monetize your
private model you invested so much in?
What do you think about bitcoin
I like it, you like I like it.
you like it as a store of value
or you like or do you think it will become
potentially used as a currency?
It has potential of working
in the economy with agents.
So not just human economy,
but agents on the internet
doing business with digital transactions.
It is a great store of value.
It's the only truly scarce resource
we have, and it has pretty good security.
Now you worry about quantum.
I do worry about it,
but I think we have plan B for that.
There are quantum proof
cryptographic protocols we can implement.
let's jump back to AI.
Do you think AI is already making
steps
towards taking over our world right now?
Not explicitly. So.
I don't think there is an AI agent
somewhere plotting to take over the world
in a meaningful sense,
but I think implicitly
they take more and more responsibilities.
People put them in charge
of managing their email, bank account,
investments, all sorts of crazy things
they should never allow any AI near.
But that's what they do.
They don't load this open source
agent of the internet
and give it full access
to manage their computers.
Do you think there'll be a safe way
for individuals and even people
in business to use AI agents and,
and a safe way to give them access
to financial communication
and other tools to help them
be way more productive, save time,
So depends on what you mean by safe.
Right now,
when you give it to a human assistant,
it's pretty good
until they steal your money.
I think it's kind of like that
until they get hacked, until they get
misconfigured, until they make a mistake,
you can be okay.
All of it assumes
pre-human levels of capability.
If you go superhuman
now you have superintelligence
and your computer deciding
what to do with you.
Are we living in a simulation?
Yes We are. Yes.
Why do you think that?
Statistically, seems more likely
if we are on a path to create
intelligent agents,
which it looks like we will soon.
And we made pretty good progress
in virtual worlds.
Just putting those two together
means anyone can create new
civilizations,
worlds populated by intelligent beings.
Let's say it's like video games for kids.
Every kid plays a couple games,
so you have billions of those
virtual worlds
populated by beings just like me and you.
What are the chances you are one of
a virtual ones versus the only real one?
If we're living in a simulation,
who's running
simulators
who are those simulators have no idea.
I'm inside the
and gain real knowledge from outside.
You can learn certain things about
the designer by looking at the design.
So, for example, you know,
we don't care about human suffering.
if we're living in a simulation,
are you a real person
or are you just a character someone wrote?
So real is a very domain specific terms
people talk about.
You can never get wet from simulated rain,
but that's not true.
If you are a character in a simulation,
you literally will get wet
from a simulated in that simulation.
So am I really in this world?
Do I feel pain?
Do I feel pleasure?
Can I fall in love?
Yes, I'm real in that sense.
Am I represented
with the same avatar in the world?
I don't know.
How long have you believed
that we're living in a simulation?
Five years.
is that something you read? You learned?
You talk to Bostrom paper and simulation
was very influential for many people,
but I think we can take it
even deeper beyond what he suggested.
I think he is about 2,030% confident.
I think we can get higher levels,
But if we're living in a simulation,
is the most dangerous entity
artificial superintelligence, or is it
whoever's running the simulation whoever's
running it so far for, let's say,
15 billion of perceived years
was not destroying us.
So if they have a plan,
they haven't done it yet.
They had a chance
and they have complete control.
They can shut it off at any point.
So we don't have to worry about this
as a new development.
Whereas us bringing into existence
potentially
a competing superintelligence
may trigger shutdown.
you think we can be turned off
if it's a simulation.
And what does that look like?
Lights out, lights out. lights out.
Turn the light off.
Okay.
You study AI consciousness as well? Yes.
Do you think that you are conscious?
I know I am How would you know?
Because I have direct access
to my internal experiences.
Okay, That's
the only thing I know for sure.
I don't know if this world is real.
You don't know if I'm real, conscious, but
I know that I have internal perceptions
Okay, I experience,
therefore I'm conscious.
So that's how you verify
your own consciousness.
It is objective.
The hard problem of consciousness
is about internal states.
What is it like to be a bet?
What is it like to be a Roman?
Is anyone going to be able to verify
in AI's consciousness?
So I have a paper which people either hate
or love proposing a test.
For some internal states,
it's not definitive where it detects
all possible states of consciousness,
but if yours are similar to mine,
I can kind of tell that you have them
because you answer the same way.
And multiple choice
questions about illusions.
if a superintelligent AI can predict
your next move before you make it,
then do you ever have free will to begin
It depends on how you define free will.
Is it about unpredictability?
So Stephen Wolfram has
excellent research on cellular automata,
and we know that
some of them cannot be reduced in terms
of running the computation.
You have to run it to find out
the state of that automata.
But it's fully deterministic.
There is no surprise.
There's no randomness.
Does it have free will?
You cannot predict what it's going to do,
but yet you can
precisely predict what it's going to do
if you have a faster hardware
if we're in a simulation,
every choice you've ever made, every paper
you've written, every warning
you've given,
is that just the simulation
running its course by definition?
Yeah,
But are you saying it's deterministic,
or are you saying that
because you can have randomness
in a computer
you can start a simulation?
So for example, let's say we do it.
Let's say we get a bunch of latest
AI models once they hit human level
and we set them up as a society
within virtual world, Okay.
I don't know what they're going to do.
They have enough freedom
of making decisions and enough randomness
to come up with different companies,
different books, something else.
So just because I'm running a simulation
doesn't mean I control every bit of it.
I can decide to turn it off.
I can decide to manipulate
a piece of memory in a certain way,
but it doesn't mean that
it is fully predictable
ahead of time to the simulator.
If this ends how you think, it will end
with artificial superintelligence
potentially wiping everyone out.
who's to blame?
Interestingly enough, we can probably try
blaming effective altruists,
a group which was supposed to make sure
AI is developed safely.
They had a lot of funding for it
and unfortunately invested
most of it in AI capabilities.
So a very ironic
twist of simulation
where people who were most concerned
about doing it right
and did up funding and working
for companies which brought us
uncontrolled superintelligence.
Sam Altman, Jeffrey Henson, Elon
Musk, at times they've all expressed
concern about AI safety at various points.
Do you think they actually believe
what they're saying
or is it just a performance?
Definitely. Hinton believes it.
Definitely.
Elon believed it.
I have no idea what Sam believes.
And do you think there's a version
where they genuinely believe
it's dangerous and still build it?
Anyway,
That's what happened with Elon, right?
He was telling us
we are summoning demon with advanced AI.
He was funding AI safety work and now he's
at the forefront of building it.
For money.
I don't think it's money
Yeah, he doesn't need the money. Right.
So what about impact on this question.
For many years he tried stopping ours
from building advanced AI.
After that failed,
he figured the only way he has
any say is to be at that table.
And what do you think he's going to do?
If he manages
to get dominance in that area,
maybe through his space compute program,
maybe there is something else.
He will once again be more of a decision
maker, not a follower in that space.
And maybe he'll remember the 2020
who cared about safety a lot more.
Do you think that's likely?
It's his simulation,
Okay.
If you had to place a bet on it
all your bitcoin.
What year
do you think this all goes wrong?
I don't think it's a good question.
We don't have power to predict
something like that.
There are too many variables.
We are on a verge of World War three,
which can easily go nuclear.
That delays technological development.
So that can give us another 200 years.
Is that what you're asking me about?
To predict precisely
how all this geopolitical events
will shape production of chips
in Taiwan? No.
You want something to have a clickable
title?
Scientist says 2027. You will die.
That will get your views.
But it's not a meaningful answer,
a meaningful question.
Once we hit superhuman performance,
things will never be the same.
It will be beyond our control.
What happens after
that is not to us to decide.
And you think we're going to hit that?
You think 99.9% for sure
we're going to hit that?
Or do you think there'll be
some guardrails before that?
Because that's all your
everything you've done in your career.
It seems like
if you were able to influence there
being guardrails around us,
not being able to get to ASI,
then your career was a success.
You you did everything you set out to do.
So guardrails usually referred
to some limits on what the model can do.
It's usually not about don't build
that thing altogether.
That's not a guardrail.
It's more of a decision
about what to do with your civilization.
I don't think there is
a technical solution
to the technical problem
of advanced AI control.
I think any
governance is a temporary solution
to that problem.
So I don't think there is
a guaranteed solution which
no matter what happens,
assures our safety,
but their futures,
which are worse and better
and there are more immediate problems
or delayed problems.
What question haven't
I asked you that I should have
but my humor paper
one of my recent papers.
It's awesome. Tell me.
So why do we have sense of humor?
Can you envision.
A simulation just like this one?
But sense of humor is not a thing.
It computes.
Physics still work, so why do we have it
and what does it mean?
So there is surprisingly hundreds
of papers published
on explaining humor
by some of the best philosophers.
Kind of like consciousness,
you know, if there is a lot of papers,
that means they have no idea
what they are doing
because everyone has their own theory.
Same with machine ethics.
Everyone has their own
set of machine ethics.
And so in my paper,
I argue that you can map the state of AI
or software or computer bugs
and to the state of jokes.
So a joke is really kind of like a bug.
You have found a problem in a world model
and you're pointing it out
before you did it.
We didn't know we had this problem,
but now you successfully debug your code
and we all get it.
And sometimes you laugh
and you see what the problem was.
If that's the case,
then we can ask a simple question
what is the worst possible computer bug?
And that's
also the funniest joke possible.
You have
to read the paper to see the punchline.
Okay, let's imagine the person
who finished this podcast, right?
They watched it. They listened to it.
What should they do
the first thing tomorrow morning
really depends on who that person
is it a Sam Altman?
Let's say he's listening.
Stop building general superintelligence.
If you are in a position of power
at a company or in a government,
put your efforts to
research and engineering towards narrow
tools.
Again, not suggesting
giving up economic benefits.
Just don't build something
where you will not gain.
Personally, everyone
you know, everyone you love will be gone.
So if you have that level of impact,
exercise it.
Let's say prediction plays out and ACI
is going to kill us all at the very end.
There's one last human alive. Okay.
And if you could send that last human
one sentence, what would it say?
Told you so.
That's all I have.
Doctor Roman Yampolsky,
thank you so much for coming on the
Thank you for inviting me.