Subscribe
Share
Share
Embed
Epoch AI is a non-profit research institute investigating the future of artificial intelligence. We examine the driving forces behind AI and forecast its economic and societal impact. In this podcast, our team shares insights from our research and discusses the evolving landscape of AI.
Here's a fun fact. This is crazy.
So real GDP is not a quantity.
As new goods get introduced,
they can cause the relationship between our
wealth and our willingness to sacrifice
consumption for safety to go the other way,
making us willing to sacrifice less
consumption for safety because the
consumption tastes so damn good.
This is sort of a pessimistic view,
but economic theory is primarily a
destructive project.
How do we detect the economic singularity?
Okay, yeah, that's a big one.
Hello, my name is Anson.
I am a researcher at Epoch AI,
and today I have the pleasure of speaking
with Phil Trammell. Phil is an economist at
Stanford University,
and he has thought a lot about all sorts of
scenarios involving economic theory and AGI,
and knows a lot about different kinds of
thought experiments relating to this topic.
I thought it'd be really great to have you
on. Welcome, Phil.
Thank you, Anson.
Let's start with explosive growth.
Most discussions about AI and explosive
growth so far have focused on one particular
family of economic growth models,
which is the semi-endogenous family.
What do you think about that? What are some
issues with how most people have been
thinking about AI and explosive growth from
an economic perspective?
I think the biggest issue is that not enough
people are thinking about it at all.
There's been a lot of anchoring on a
particular semi-endogenous growth model,
the Jones model. And the Jones model,
the constraint on growth,
in particular on technology growth,
is just a lack of research inputs.
You have researchers,
your R&D.
They develop better technology and that that
speeds growth.
The only reason why we don't have really,
really fast R&D already is because we only
have so many people doing it.
So if we automated it,
if we created lots of AIs,
lots of robots doing R&D,
we would have really fast growth.
One issue is that that might not be the whole
story,
right? It could be that a so-called
Schumpeterian model is closer to correct in
some contexts. This is a model in which the
main thing constraining growth is that every
innovator needs a patent or a,
you know, a trade secret or something.
They need a temporary monopoly on what
they're producing in order to justify the
cost of developing the innovation.
And so on those views,
growth has to proceed slowly enough.
The sequence of new innovations has to come
sparsely enough for it to be worth doing any
innovation at all. And yeah,
I mean, I think we see this potential worry
in how the LLM market has unfolded where
because each one has so little moat.
They keep pouring money into making better
and better models,
but very quickly, you know,
someone makes an even better one and steals
the lead and everyone quickly switches over
to the competitor.
So there have been these worries about how
sustainable this is exactly. And in these
models, R&D is, for simplicity,
typically presented as being just a function
of capital expenditure.
So if you ask, taking the model totally
literally,
what happens if you automate R&D?
Well, nothing because it's already all about
the lab equipment.
That's going too extreme.
Again, I think the Jones story is is closer
to right if you have to pick one.
But there's been so much work anchored on the
Jones model, that I think probably taking a
look at other approaches would be
informative.
Given the Jones model,
or a model kind of in that class,
I think we are sort of led astray by a few
things.
But let's maybe say a bit about two in
particular.
One is that in the Jones model,
there's not a hard limit on how
much research can be parallelized at a given
time.
So if you scale up the number of researchers
really fast,
the rate of technological progress goes to
infinity with the research inputs.
You know, maybe not one for one. Maybe the
rate of progress is like the square root of
the inputs or something,
but still a sudden really arbitrarily big
surge in the inputs means that you make
really fast technological progress.
And I think that that just can't be right.
If you think about an assembly line,
bringing in loads of engineers and management
scientists or whatever to to speed up its
production.
The more you stuff in,
the faster progress they'll make.
But as the number of engineers goes to
infinity,
the rate of progress that they can make in
five minutes,
it just hits a ceiling.
Right. So I guess like another example here
would be like, no matter how many GPT-3 that
OpenAI is running, they can't just
arbitrarily push up the rate of algorithmic
progress by running a trillion instances of
GPT-3.
That's right. And importantly,
I mean, no LLM can single-handedly carry on
AI research at all. Right.
With GPT-3 alone, they can't really make any
progress,
right? You need people and GPT-3,
or even just people.
So I think it's actually like a little bit of
a different case, but it's another example of
gross complementarity,
at least.
Right. So it would be that even if we had
GPT-3 plus lots of humans,
if we scaled that up arbitrarily,
we still wouldn't be able to just get
arbitrary algorithmic progress.
Yeah, yeah, that would be the tighter
analogy.
I see. Okay. So should we just totally scrap
this Jones Law of motion then?
Is this law of motion just fundamentally
going to be way too aggressive?
No. So well, it's very simple.
And it's, you know, for many purposes I think
it makes reasonable predictions and
so we shouldn't always scrap it.
And to the extent that it leads us astray,
it could lead us astray in either direction.
And I think this is a common pattern i seem
to run into.
When you think of a bottleneck that models
haven't been incorporating before,
on the one hand, the lesson could be "Oh,
well,
actually, growth won't be as explosive as we
thought,
because once we relieve some other
bottleneck, we'll just slam into this new one
we'd forgotten about". But on the other hand,
to the extent that we've been constrained all
along by the bottleneck in question without
realizing it then, you know,
if this is a new margin on which we could get
more growth than we had been anticipating
because it could be relieved.
It could be getting relieved more quickly
than before. So in this case,
the reason why scaling up the number of
people and computers working on algorithmic
progress, you know, just running GPT-3
wouldn't make progress go to infinity is the
high latency that different human brains have
between each other,
right? It's hard to efficiently divide a big
project among lots of different people who
only have faint understandings of what,
you know,
part they are playing in the big kind of,
you know,
the big orchestration. And we've been getting
better over time at collaborating.
And, you know, the internet allowed for
collaboration with people with complementary
skills in very different parts of the world.
And you know, whatever.
I mean, obviously, if you go back far enough,
you know, the invention of language and
writing and everything obviously lowered
latency in some sense.
And that's been furthering growth all along.
But the rate at which that bottleneck is
being relieved could rise pretty radically,
I would think, in the event of you know,
neural nets,
kind of virtual brains that are themselves
getting bigger and bigger.
So instead of having one,
two, three, four employees,
you have, you know, fully integrated minds
that that are,
you know, two, three,
four times bigger. So,
so assuming a sort of constant rate of
penalty for parallelization,
you know, a constant sort of exponent on the
research inputs in the Jones model would be
leading us astray in the sort of pessimistic
direction.
If you want to think through the problem
carefully, we probably shouldn't be using the
Jones model because it doesn't even work well
as a bound,
right? It could be leading us astray in one
direction or the other. But if you want
simplicity and if you're not going to be
extrapolating kind of too far out of sample,
then it's still, you know,
kind of simple and insightful.
Yeah. I guess the tricky thing for us is that
we're thinking about AI, and that almost
necessarily entails us having to extrapolate
a lot of things pretty far out of bounds of
the regimes that are being tested. Yeah. So
one question here that is also pertinent to
models like Tom Davidson's full takeoff model
and the GATE model is that we might get
declining returns to scale over time as we
make further and further technological
progress. And I know that you've been
thinking a little bit about this.
So how fast has this decline been?
What do we know about this question?
I think you're referring to this point that,
you know,
ideas get harder to find.
And the rate at which ideas are getting
harder to find might itself change over time.
Right. So that's something else that the
Jones model doesn't feature.
And I think that's a much smaller weakness
than this parallelizability point.
But it's still a strong assumption and I
think probably an incorrect assumption.
Yeah. Something I used to think was that in
the grand scheme of things,
the rate at which ideas were getting harder
to find was itself growing kind of
non-negligibly, and that you could see that
by comparing the,
you know, the function from growth and
research inputs to growth in productivity
outputs in the distant past to in the more
recent past.
So there's this famous Cramer paper from 1993
that if you look over like the million years
of the Malthusian past,
he estimates that basically ideas aren't
getting harder to find at all. You know,
that every time you double the population,
you double the rate at which they are coming
up with proportional productivity
improvements. The data is really sketchy,
obviously,
but I think even after accounting for that,
it seems pretty robust that this idea is
getting harder to find effect does not appear
very strongly in the distant past,
if anything like Cramer's data are to be
accepted.
And by contrast, this well-known paper,
"Are ideas getting harder to find?" from 2020
finds that over the last century you have to
be growing the research inputs kind of three
times faster than the,
you know, your target rate of growth in
productivity. So I thought "Okay,
well, so that's a big change".
And if you project it forward,
then the rate at which ideas get harder to
find will keep on rising until eventually we
just hit hit the ceiling of technological
maturity or before that.
Right. If there is some technology levels
that are feasible,
but that we kind of just,
we never really get to because we slow down
before we were anywhere close,
you know?
I'm thinking about it more.
I think that's probably not right.
And the big difference between that analysis
of the distant
past and the sort of analysis in the "Are
ideas getting harder to find"-
paper, is that in the past,
they're looking at how quickly we can create
more copies of the same object,
namely the human,
the human body. And as I was saying before,
modern growth is mainly not about that,
but about creating new kinds of goods and
higher quality goods.
Another difference, of course,
is that modern growth is often driven by
profit-seeking R&D, right.
Which can be constrained by that creative
destruction issue I was getting at before.
So the issue here would basically be if a few
hundred years ago,
we come up with the idea of patents and we
have a sort of entrepreneurial culture take
hold where people are trying to invent new
products,
new production processes and everything and
capitalize on that.
That's going to really boost growth at the
time,
because it's something people could have been
doing all along that they weren't, and now
they're doing it. But that process is just
capped by something different from the kind
of old Malthusian, purely
resource-constrained thing driving growth,
right. So you get this burst of extra growth
that starts picking up and the industrial
revolution and so on. And that adds a lot.
But if it itself starts slowing down for
whatever reason,
that's just like unrelated to the rate at
which ideas are getting harder to find or
whatever. Right? Then in a model which only
has room for ideas getting harder to find,
you'll pick that up as ideas getting harder
to find.
Okay, so there was a paper,
actually, by James Bessen and a co-author I'm
forgetting just recently,
which makes this claim,
actually, that ideas haven't even been
getting harder to find.
Over the last century,
sort of properly understood,
but that R&D has been contributing ever less
per per unit of
of R&D, R&D expenditures or R&D effort.
It's been contributing ever less to
productivity growth. Because for whatever
reason, it's sort of unfolded in a direction
where new R&D efforts have ever more so been
cannibalizing past ones which both
discourages investment to begin with.
And of course, it itself means that you're
adding less to the you know.
Yeah. If the big new thing is that you come
up with Amazon or something,
but that makes you a multi-billionaire.
And so it warrants like billions of dollars
in investments and how to get all the
logistics right. But actually everyone
without you,
everyone would just be shopping at Walmart
and the consumer surplus and everything would
just be like a penny less,
then you're not actually increasing output
very much. You're just sort of like
transferring it by winning the race a second
faster. But both parties still run almost the
entire race.
And that's been happening all along.
But that kind of thing,
according to Besson,
is happening more now. And so if you're just
looking at a sort of time series of R&D
inputs to productivity growth over time,
you'll think "Oh,
ideas are getting harder to find",
but that's not really what's going on. Now,
I don't know if that's entirely right,
okay. But it's another example of how I think
you're really comparing apples and oranges
when you look at modern growth,
where it's all about ,
you know, kind of on the intensive margin.
More and different kinds of things per person
than the old Malthusian kind of regime.
And so I don't think you can infer that the
rate at which ideas are getting harder to
find is itself rising.
Right. So I guess the takeaway is we would
like to be able to answer this question,
but then the data is not really comparable
between different regimes,
both on the R&D inputs and the R&D output
side.
There is also this sort of omitted variable
bias kind of issue that's going on,
that kind of makes it not directly
comparable. Well,
okay, that's kind of unfortunate because we
would like to know what these bottlenecks are
for, say, hardware efficiency and software
efficiency.
I guess it's just too hard to say. Okay
changing tacks a little bit.
You've written a paper about how new products
and new varieties of goods is an important
consideration when we're thinking about the
real impacts of economic growth.
What exactly is missing from the current way
that we think about economic growth?
Yeah, the problem is that we try to flatten
growth in the goods and services we
enjoy into a single variable,
and put a number to how much that variable is
growing. I'm not making a standard critique
of GDP here.
I mean, it's somewhat standard,
but I'm not making the particular standard
critique that it leaves out all kinds of
things,
sources of value other than consumption,
like leisure or friendship or whatever.
That's true. But what I'm saying is that even
when it just comes,
even when it comes to consumption,
you know, the part of all that matters,
that's just the knickknacks that we consume
in a very concrete way and enjoy.
You just can't flatten things into this
single dimension.
I think a really simple way to see that is to
just say,
okay, let's say we only had one consumption
good:
horse. Okay, so we're in the Golden Horde,
something.
One of the empires that followed Genghis
Khan's when it when it split up.
And, you know, they got a lot of use out of
their horses.
They could ride them,
but they could also, you know,
use their skin and their milk and everything.
And let's say they could only consume horses.
The quality of the consumption basket that
they got to enjoy. It rises as the number of
horses rises,
like you'd rather have five horses than one,
but it only rises toward a ceiling,
and that ceiling is below what a typical
modern middl-classs American gets to enjoy.
And so whatever number you put to consumption
or GDP in the horse economy.
If your index is going to be homogeneous of
degree one,
if it's going to have the natural feature
that if you double all the stuff in it,
then you double the number you assign to
consumption. So if you've got five horses,
and then you go from that to ten horses,
then you've doubled consumption,
right? Whatever number you put on how much
consumption five horses count for,
as horses go to infinity,
consumption is going to rise to infinity.
But the quality of the consumption basket is
still going to be below what's presumably the
finite consumption basket that,
you know, the middle-class American gets.
And the issue is, no matter how many loaves
of bread that I have in 1800.
It doesn't matter because I would swap no
number of loaves of bread for my modern day
laptop.
Yeah, I mean, at least it's totally possible
to have that preference.
Yeah, I think that's a common preference.
Yeah.
Yeah.
There was this previous paper that I think
you were involved in together with Leopold
Aschenbrenner on whether or not we could try
to speed through the time of perils.
So, relating existential risk and growth.
What does this imply for that?
Yeah, this paper and some recent papers by
Chad Jones,
they all are centered around this calculation
of how much consumption we might be willing
to sacrifice for safety over time as we get
richer.
And the assumption is that as as we get
richer,
our marginal utility in consumption falls
because,
you know, we've got these concave utility
functions. And we have more to lose if we die
because we're better,
we're doing better and better. And so for
both reasons, we'll be willing to sacrifice
more consumption for safety over time.
And if that effect is strong enough,
it could mean that we're willing to tolerate
really rapid AI development,
which could get out of hand and be disruptive
or even existentially catastrophic.
Right now we're willing to take that risk,
but in the future,
we won't be, at least if we survive to a
future where we're rich enough to kind of
want to make the tradeoff the other way. I
think that argument probably has a fair bit
of truth to it. It's not just a theoretical
point that as we get richer,
we get willing to sacrifice a lot of
consumption for safety.
We see it in medicine being a luxury good.
As people get richer and as societies get
richer,
we spend not just more on medicine,
but a larger fraction of our incomes on
medicine. On the other hand,
these trends can reverse.
So, yoknow,ow health care spending is a
fraction of GDP actually has fallen a bit in
the US in recent years.
And in other advanced countries and,
you know,
it's always very easy to predict a trend
correctly right before it turns around for
one reason or another. But the deeper issue
with putting too much stock in this argument
is that, as we were just saying,
in principle,
as new new goods get introduced they can
cause the relationship between our wealth
broadly construed and our willingness to
sacrifice consumption for safety to go the
other way. New products can kind of start
making us willing to sacrifice less
consumption for safety because the
consumption tastes so damn good,
right. Now again, on balance historically,
I think this effect has tended not to win
out. But if AI nudges the direction of
technological development so that,
yeah, we're getting more copies of the old
things.
And so we're kind of satiating in those,
but we're inventing these new things that are
really desirable, you know,
because, they're like immortality pills or
they're wires to the head that just give you
mind-blowing joy or something.
Then, you know, it's totally consistent with
a reasonable-looking utility function to
just, you know, be like the rat pushing the
heroin button or something and getting more
expected utility out of a path that involves
more short-term pleasure,
but a lot of it, and a higher chance of
disaster than than the safer path you know.
At least for a non-negligible discount rate.
I hope that doesn't happen.
You know, I hope that growth is more in this
kind of satiating,
safety-promoting direction in the near
future.
Because I care about the distant future and
getting to it.
But it's at least a logical possibility that
it goes the other way. And I think,
you know, I mean, this is sort of related to
some of the stuff we were talking about
before. But yeah, I think we can be led
astray,
and I certainly have been led astray by,
by kind of doubling down too hard on a simple
one-dimensional model of things in which
basically all the automation or AI can do is
speed us up or slow us down on some
trajectory that's already set in stone.
There's lots of different dimensions of
technological development and paths we could
go down. And by speeding up one a bit more
than the others relative to how fast they've
been proceeding relative to each other in the
past, a lot of our intuitions might break.
Moving on to the GATE model.
A core part of this model is the amount of
production that you get and it's is
determined strongly by the fraction of tasks
that are automated.
So there's this question,
though, about whether or not this framing
even makes sense. So I'm curious what you
think about that. Does the fraction of tasks
as the framing for these growth models hold
much water or doesn't it really make that
much sense.
Yeah. I think qualitatively they can be
pretty insightful.
And I don't want to say the whole approach is
worthless or anything like that,
but certainly I don't currently think they're
as useful as I used to think.
For starters, I just think it's sort of
telling the very first task based model in a
form anything like we know them today was
Zeria's in 1998,
and it came out right when the O*NET task
database of US occupations was
first put out, I think in 1997 by the Bureau
of Labor Statistics.
So it's like, well, people collected all this
data thinking "Might as well,
it could be useful for something." And people
come up with a model that makes use of this
data. If we collected other data,
we probably would have come up with some
other model. I don't think there's a deep
fact that
work subdivides into these nuggets that we
call tasks and that we can sort of think
about plowing through one by one and seeing
what happens to growth.
A simple example of how that can go wrong was
explained to me by Pamela
Mishkin at OpenAI. Who was one of the
co-authors on this paper that makes heavy use
of a task based model:
"GPTs are GPTs".
Daron Acemoglu wrote a paper kind of making
use of their estimates of automatability by
LLMs, by task. And yeah,
it just seems this really rich sort of
resource. And so I and some other people
where I'm working at Stanford,
at least we were and I think we are still,
trying to follow up on that.
And I talked to Pamela and she was like "Oh,
this whole framework doesn't really make much
sense anyway". "What do you mean?" And she
sent me an email explaining it,
and I didn't get it. So then we had a Zoom
call and this was her example.
Let's say I've got two tasks.
Write an email to this moron Phil.
She didn't put it that way. To write an email
to him explaining the limitations of the
task-based model. Task B is have a Zoom call
with Phil going through the limitations of
the task-based model in more detail.
If all she had to do was task A,
maybe an LLM could have done it better and
faster. But given that task B is coming,
she felt and it certainly it could be the
case, that if the LLM can't do task B,
have the Zoom call,
it would take her more time to read and
internalize and remember the particular
choices of terminology that the LLM would
produce in the email,
so that we can discuss it on common terms
over the Zoom call.
Right. Instead of having to keep going back
and saying, "Wait, what's Phil getting at?
Oh yeah, the LLM used this term for
something." That would take more time.
So given that task B is coming up,
the most efficient workflow is for her to be
the one that writes the email,
right?
So like an eval or a just a human-like
evaluation of a given
task on whether an LLM can do the task would
predict that if someone's job consists of
task A and task B and then we introduce an
LLM that you found could do task A,
their productivity would double.
Right. They could just do task B all day,
they'd get twice as much of it done. Maybe
half of them would be fired. I mean it
depends on the effect or whatever. But
actually it would have no effect at all in
this case.
Right? Everyone would just keep on doing task
A.
So yeah. So if these sorts of effects are
significant enough,
I think that means that our projections from
task-based models could be wrong in at least
three big ways. One is that the growth
impacts of automating technologies,
including LLMs or AI in general,
could be delayed relative to what
you naively would have thought. Because it's
got to automate A and B before you see any of
the effects, right? Second bias is that when
they come,
they'll be more sudden than you would have
thought. And the third,
more sort osubtly,ly gets back to this
latency thing.
This is what I'm thinking about all the time
now, so I have to work it into everything.
The message of Pamela's example is that
there's a kind of increasing returns to scale
and that it's more productive to have the
same mind doing tasks A and B.
And that might outweigh the fact that if you
tried to outsource task A in isolation,
this alternative system could way more
efficiently or quickly or something,
perform task A. Despite that,
it's still, it's so beneficial.
There's a kind of a positive spillover,
where doing task A makes the factor of
production, in this case her body and brain,
more productive at task B,
and that positive spillover is so big that it
totally displaces the use of the LLM at task
A. Well, we can currently take advantage of
these sort of positive spillovers or these
cases of learning by doing from concrete task
to concrete task.
We can take advantage of them up to the size
of a given human's capacity for work.
Some kinds of work maybe can be subdivided.
Is there something you can just take out of
your day and sort of ask someone else to do?
But not that much. And so what that's telling
us is that we have these spillovers where
it's really helpful to have the same mind
doing all the different things. And,
yeah, we can take advantage of the increasing
returns that this allows for, but only up to
the size of what a single person can do in a
day or in a year or in a life.
And so with just sort of bigger systems with
really big neural nets that can do a lot of
things simultaneously and have really big
memories, that can use the learning they got
from writing that email to Joe six years ago
on the other side of the world to inform the
Zoom call that they have with Phil about
task-based models tomorrow.
Yeah. You would expect to see even more
growth than you would predict from a naive
task-based model where the best you can do is
just automate all the tasks one by one, and
then kind of get a world in which sort of
everyone was still the same size.
Yeah. Is this not a question,
though, of whether or not the tasks and
owners happen to be at the right level of
abstraction?
Couldn't I, say, make an argument about how,
for example,
if we look at which occupations existed in
the past before the Industrial Revolution,
and we compare that to the tasks of today,
indeed,
like maybe a large fraction of those jobs,
using the precursor to O*NET,
that wouldn't have been so bad that it's just
totally useless?
I guess you're not saying that this is
totally useless, but then is this not just a
question of O*NET being at the wrong level of
abstraction?
I see what you're saying. So a few things.
One is if we're talking about GATE in
particular, one thing it can do in kind of a
best-case scenario is to be improved on over
time as data comes in.
And so we could see like where it looks like
this is the function from model size to
fraction of tasks automatable or whatever.
Right. And if we're looking at the occupation
level and there's only like three,
then it's just not going to be useful like
that.
And you're going to have to turn to other
ways of indexing our progress,
like SWE-bench or the task time length
growing or whatever.
But you're not going to be able to use use
O*NET because the right level of granularity
is just unreasonably large.
On the other hand, I mean I don't think
there's one objective right level of
granularity. Because for some purposes,
like with A and B,
it could be that some people have questions
that need follow-up emails and that don't
require follow-up Zoom calls.
And you want to be able to capture the fact
that Pamela is going to be able to just ask
the LLM to handle those.
So I don't know, my lesson is more like
you've got to be careful when thinking about
your application of a task-based model rather
than that there is a right task-based model,
but it's just at a coarser level of
granularity or something.
I think that's very much the kind of thing
that I was trying to get at.
Which is like, if I'm trying to do this
coding task,
maybe the description at O*NET-level would
look more like doing coding,
doing data visualization.
But then really the thing that we actually
care about is more like it's able to write
this particular block of code in this Python
notebook,
or it's able to write this Python script. And
what I meant by level of abstraction was
maybe we need to actually step down to a
different thing that's more amenable to
capturing that kind of automation. So it's
less of a problem of the framing of a
fraction of tasks, but maybe it's more like,
you know,
the particular data that we have,
the things that we're trying to use to
measure these things,
are quickly becoming out of date. Maybe in
the same way that like the precursor to O*NET
was out of date and they had to come up with
a new thing.
When you say "it quickly becoming out of
date",
do you mean just because the nature of work
is changing fast,
or do you mean that it's becoming out of date
because day to day,
the list of tasks that you're doing is
different?
Because you know, "code this particular
thing" is different from "code that
particular thing".
I think it's becoming out of date in the
sense that we are getting a better sense of
what kinds of tasks AI is automating over
time.
And maybe in the past,
we just didn't really have this
understanding. Now we're realizing more and
more just how often these task descriptions
might be compared to the actual test
automation progression.
I see, I see. Yeah. Okay.
So we should cut things up in a way that
corresponds more closely to how we think
things will actually unfold? So that things
will be cleaner:
where it can fully do A,
B and C before it can even do a little bit of
D?
Exactly. I think that would be the hope. But
then I think it's also kind of hard to do
that. I'm not sure how I would cut things up.
Yeah. No, that's a good thought. I haven't
thought about how we might categorize tasks
in a way that's more amenable to tracking AI
automatability.
Okay, so speaking of the GATE model,
would you say that the biggest problem with
the GATE model or the greatest source of
uncertainty is coming from the AI R&D module?
Or do you think it's actually not just that?
Well, I can only speak for myself.
I think the biggest source of uncertainty is
the assumption that there's some amount of
compute where we can be confident that a
model trained with that much compute would be
able to fully automate anything in
particular.
And secondly, that we can interpolate between
here and there
such that, with the kind of 20% of the
compute to the threshold to the finish line,
we could automate whatever percent of the
tasks,
right? This isn't an issue with GATE in
particular.
It's inherited from Tom's takeoff
speeds-model,
I think. But it would be really nice to have
a have a threshold like that,
right? Where we could say,
"Okay, we know we'll get to AGI,
to a system that can automate AI R&D or that
can do something else you're interested in".
And then the interpolation just seems totally
made up. But I've,
I've never I've never thought that whole
methodology was very well grounded.
I think the interpolation in indeed is pretty
made up.
I guess the issue is just like,
what do we do instead?
Well, I'll tell you. I mean,
we can't do it now,
right? But you know, I think it will start to
be the case that full-on O*NET tasks will be
automated and full-on sub-classifications of
work time,
certain types of eventually whole
occupations.
But before that, you know categories from
time-use surveys or from timesheets,
whatever. That they'll start to be
eliminated.
And then we'll have the data to extrapolate
rather than interpolate,
right. Instead of assuming a finish line and
then putting a point in the middle and then
fitting a curve. We'll be able to say every
time we double the model size,
we tend to cut the remaining net tasks in
half or whatever the relationship will look
like. At the moment, there's that "GPTs are
GPTs"-paper.
They just ask an LLM for its guesses about
what it could do.
And then they have people try to judge it,
but they adjust the prompt that they give the
LLM to kind of match the people.
And then they extrapolate. In principle any,
you know what I mean?
Like for anything, the people.
Whatever, there's a whole long conversation.
But so even if you fully trust those
estimates, that's sort of a single data
point, right? That's like what GPT-4 could
automate. How partially it could automate
each O*NET task.
People haven't done a follow-up one for
GPT-5,
or for a reasoning model.
They haven't recapitulated the whole
methodology of that paper.
Yeah. I mean, like the Anthropic Economic
Index and other data on LLM usage that's
starting to come out from the other providers
might let us sort of continuously start to do
this sort of extrapolation.
And, you know, they're not just anchoring on
O*NET tasks,
right. Like Anthropic,
but they also have their own categorizations
that might sort of cluster more naturally.
They have this like endogenous clustering
thing where they apply some ML to try to
figure out what people are using the models
for.
And that's kind of maybe responding to your
point about how we can sort of create on the
fly a task, a way of carving up work into
tasks that better corresponds to
automatability. That approach will have its
flaws as well.
But I think from at least a certain
perspective,
it'll be more grounded than the sort of set
end point and then interpolate-approach that
we've been having to make use of so far.
But it's still early stages for that. And,
you know,
yeah, I mean in a really fast takeoff
scenario then there won't be time,
right? Because we'll move so quickly from
being able to automate almost nothing to
being able to automate everything at once. So
maybe it's just intrinsically hard to
predict. But yeah, I think probably not.
I think probably we will be able to start
doing extrapolations that are a bit more
empirical as time goes on and as we start
kind of developing a track record of
automating tasks completely.
So of course you can track progress and start
to extrapolate in this sort of more empirical
way. Not just by looking at the dimension of
tasks,
but by looking at task time length.
Right. So that's METR have their famous study
of this.
I think that's just great.
That's amazing because it's the first real
example of of starting to do what we would
have liked to be able to do all along.
Yeah, track progress on something that,
in principle,
would eventually get you to full automation
of at least some class of tasks.
And we're far enough along already that we
can draw a line through the data that we have
and sort of make an extrapolation.
It's not like we just have 1 or 2 data points
right at the beginning, like maybe with
O*NET. You might be able to,
you know, carve up the space of work or the
space of R&D or the space of AI R&D in other
ways as well.
I think Situational Awareness has been
criticized by some for
being as hand-wavy as it is about the kind of
extrapolation by equivalent to human age,
basically, right? As smart as a high schooler
or as smart as a college student.
And I think that that framing is not
justified in that essay.
But it could be, or something like it could
be.
More experienced workers are paid a lot more
than more junior workers.
If you know one thing about a person to
predict their wages.
Right. Even more than you know,
education or race or gender or whatever,
age explains so much. And what that's telling
us is that there's this,
a lot of tacit knowledge that people accrue
through life.
Well, okay, so maybe the way things will
unfold is that AI's first get good at
learning the sorts of things you can learn
from books which they have access to no less
than a college student. In fact,
they have better access in some sense.
And so the people right out of college with
no kind of tacit experience will be first
automated. And then as they start to get more
sample efficient and they're deployed for
longer and in those kind of early-stage roles
they accumulate the data and that means that
they can displace the people that had only
had three years of real world experience.
And then you could see that creeping up.
Maybe. I'm not saying that's what's
happening. You hear anecdotes,
of course, about young software engineers
finding it harder to get jobs.
But it could be that a story along the lines
sketched out in Situational Awareness comes
to have an empirical grounding,
in which case we'll have this other way to
extrapolate, right? And there might be others
as well that I haven't thought of.
I guess it reminds me of like AI 2027,
where they're using this METR data for
forecasting the timeline to a superhuman
coder. As far as I understand.
And then instead of just doing this for a
superhuman coder,
which I guess makes the most sense given what
kinds of tasks were included in the METR
study, perhaps we could try to generalize
this somehow by getting more data or just
generalizing the study to a broader fraction
of tasks in the economy.
Yeah, but still looking at task length.
Yeah. Right. Yeah. That would be kind of
expanding the scope of that particular
dimension, the task length dimension.
Right.
How much should we update on economic theory
anyway?
Yeah. Yeah, that's a profound question.
Something I've had to wrestle with having
sort of specialized in it.
Definitely less than I used to think.
This is sort of a pessimistic view,
but a view I've somewhat come around to is
that economic theory is primarily a
destructive project.
And what I mean by that,
I mean, well.
It doesn't sound promising.
Well, yeah, but it's not so bad.
What I mean is, if it's good for anything,
it's most useful for pointing out that
intuitions that you might have had don't
actually hold in general.
Okay. Sort of coming up with counterexamples
or sort of curious models in which something
you thought was inevitable turns out not to
hold.
And every now and then on further
investigation,
hopefully empirical investigation,
you'll find that this theoretical curiosity
is actually relevant. So it can sort of open
your mind.
But it's a much more limited role than being
able to actually,
deduce the truth from first principles,
right?
You know, a classic example is like minimum
wages,
right? You'll see people saying,
as an economist,
I know X. As the price of something goes up,
people want people want less of it.
And so minimum wage is going to cause
unemployment. That's not something,
you know, as an economist.
That's something you know as a shopper or
something. Everyone kind of has the intuition
that typically when things get more
expensive,
they'll be less demanded.
You didn't need a PhD to learn that.
If you learned anything in a PhD either it
was the the tools to empirically test whether
that's true in some contexts,
the fact about whether it's true in some
contexts because you did actually apply the
tools,
or the theoretical insight that it's not true
by necessity.
So it turns out that there are such things as
Giffen goods,
or you can very commonly have
backward-sloping supply curves for labor.
Because if someone gets paid more,
they might just work fewer hours,
right. Sort of buy back their own leisure
with some of the money.
And so labor markets can look all sorts of
ways from first principles.
So this theory can shake you out of this kind
of intuition that you might have had walking
into econ 101 or walking out of econ 101.
And, you know, I think actually,
I mean, I think the empirical case on minimum
wages is just sort of messy.
In some cases they do probably cause some
unemployment,
in other cases not. And it depends a lot on
magnitudes and all the rest of it.
There's some people who triumphally say
"Economics is wrong.
Minimum wages don't cause unemployment." I'm
not saying that. I'm just saying I think if
economic theory has anything to add there,
it's destructive of the common-sense
intuition we all have about prices and
quantities that we know from everyday life.
And so generally when thinking about AI and
growth,
there's some constructive points.
The Jones model can, you know,
at least sort of to some extent,
if you squint, explain the past and so it can
serve as a nice basis where you can swap out
the Ls with Ks and see what full automation
would do to growth.
But beyond that, I think it doesn't have that
much to add,
even though I'm trying to add a little bit.
And if it does have something to add,
it's by identifying ways in which the future
could be surprising.
Which then will have to just kind of look
into more with whatever empirical tools we
have.
So I guess some of the ways in which you've
applied this is to say things like what is
the impact of AGI on the probability of
explosive growth and on wages.
And my understanding is that you thought that
it's kind of ambiguous just from a purely
theoretical perspective. I'm curious if
that's right?
And also whether you think you have a
different perspective if you take all of the
empirical evidence into account?
Like what's your overall view and what's your
purely
theory-let's-try-to-find-the-counter-examples
kind of view?
Yeah. Well it's all a little bit mixed
together.
But one thing that I think definitely is in
this like "Aha,
here's a theoretical curiosity point" is that
real GDP is such a bizarre chimera of a
variable that you could have full automation
and really explosive growth in every
intuitive sense of the term and yet real GDP
growth could go down.
An example of why it might at least not go up
that much,
which I think it probably won't all kind of
work out this way but I don't think this is
crazy, is that you get this effect where
there's this common pattern you find where
new goods, just as they're introduced,
have a really small GDP share.
Because they have zero GDP share before
they're introduced,
right. At first they're really expensive -
we're not very productive at making them.
And as the price comes down,
as we get more productive,
the price falls but the quantity rises
faster,
right. So the elasticity of demand,
I always get this backwards,
the elasticity of demand is greater than one.
Yeah. Every time the price falls a little
bit,
the quantity rises a lot.
So the dollar value of the good rises.
So the share is rising. And after a while it
goes the other way.
Once the goods are really abundant,
at least relative to everything to everything
else. Every time we have the price [go up],
the quantity only rises a
little bit because we're basically satiated
in it. So you get this hump:
new goods - small share;
goods that have been around for a medium
length of time that we're mediumly productive
at -
high share, they dominate GDP;
old goods like food -
small share. So we're kind of continually
going through this hump.
Everyone's familiar with Baumol's cost
disease.
But the way it's usually presented is that AI
might have less of an effect on growth than
you might have thought,
because we'll be bottlenecked by the few
things that have not yet been automated that
you still need people for.
And actually, you can have Baumol after after
full automation.
Because, so remember the hump,
right?
Real GDP growth at a given time is the
weighted average of the growth rates of all
the goods where the weightings are the GDP
shares.
The GDP shares will be dominated by the goods
that we're intermediately productive at in
this view. Right. So let's say for every good
you have
its own specific technology growth rate.
Like how quickly it can be produced is some
arbitrary function of its current technology
level. Okay. So it can be hyperbolic.
You can have A dot equals A squared
something.
So for every good, there is some finite date
by which we'll be able to produce infinite
quantities of it in finite time. So it'll be
free. So GDP share will be zero.
And we just go through these ever higher
index goods,
ever more complex goods over time.
And at any given time,
all of GDP are the goods that have a
productivity level of five or whatever
happens to be in the middle as far as GDP
shares go. So some effect like that can
produce something like a Baumol effect even
after full automation. Yeah.
I think it would be pretty weird if that kept
the absolute number low.
Like anything as low as the current number
sort of indefinitely.
But yeah, the idea that maybe it causes
measuredreal GDP growth to
not be that high for a while when the world
is starting to look remarkably different.
That doesn't seem crazy to me.
And maybe it's worth knowing and just kind of
having as a scenario in your back pocket in
case things start looking weird and anyone
says "What are you talking about?
I don't see the numbers." I'm trying to be
cautious and all,
butthat's an example of,
like, destructive economic theory.
Do we have any quantitative sense of what the
hump looks like?
Yeah, that's a good question.
I mean, again, there's that Besson paper and
you could just do a bunch of case studies by
good. Yeah, I should look into that more
quantitatively.
Okay. And then kind of to go back to the
thing of what's your overall view then?
Oh, I didn't get to the wages either. I
should say something about wages,
but yeah, I don't know.
My overall view on on what?
The probability of explosive growth over the
next,
say, five decades.
I think, if if by explosive you mean like an
order of magnitude higher than now,
that I think is more likely than not.
I mean, with 25 years of 30% growth,
the world's very
different. In fact, I was just punching in
the numbers on my phone on the way here.
So 25 years of 30% growth is so different
that even our intuition that you can do 1.3
to the 25, or you can do e to the 0.3 times
25 will be about the same.
That's broken down. So 1.3 to the 25 is 700.
The economy is 700 times bigger in some
sense.
E to the 0.3 times 25 is 1800,
right?
So just the additional compounding from
continuous growth at an annualized 30% a year
more than doubles the number. Anyway,
so in a world that different,
some funky effect where we're bottlenecked by
these new goods,
which will be really advanced because they're
the kind they'll be like some weird
nanobot-type thing that it takes a lot of
serial steps
to to create. And we need it -
we can't make up for a lack of it with
large quantities of all the goods that we've
been able to produce in the meantime.
So we're kind of like bottlenecked by the
scarcity of this,
of this new thing. But anyway some weird
effect like that or some natural resource
constraint or some regulatory imposition,
like
maybe we really do satiate and we start
wanting to just really make the world very
safe and stable even if it means throttling
growth to only 30% a year or whatever.
All of that roughly cancels out to me at the
moment,
I think, with the counterarguments,
which are also valid.
That, you know, due to international
competition people will continue to race with
each other for military purposes,
even if they would prefer each to be safer.
So that's where about where I land.
I'm kind of even odds once you press too far
beyond order of magnitude.
And how much does your answer change if we
don't consider the economic theory aspect of
this? We don't consider those,
you know the hump of the productivity and,
shares. Would your answer change much?
Oh, sorry. I should have said before:
50 years is long enough that I do think we'll
be able to develop robots and AI that can do
what we can do.
But some of the uncertainty will also just be
that that doesn't work out.
And then, yeah, it's a little hard to
separate it out.
I mean, digging into the theory of what
chain-weighting is and so on has made me
pretty viscerally feel like real GDP is a
much slipperier concept than than I ever used
to think. I mean, here's a fun fact.
This is crazy. So real GDP and lots of real
variables like inflation-adjusted variables,
real capital or whatever,
let's say real GDP,
is not a quantity. What do I mean?
Yeah. It's not. Here's what I mean.
Imagine a timeline of some economy.
So, the US from 1950 to 2025,
75 years.
Okay. And imagine an alternative timeline
with an alternative economy living it out
that's exactly the same as the US in 1950,
at the beginning,
in its own 1950, and exactly like the US in
2025,
at the end in year 75.
But in the middle things happened in a
different order.
So the microwave was invented in 2006,
and the iPhone came out in 1971.
And, you know, the distribution of wealth
changed hands,
evolved in a different way. But at the end,
it's exactly the same.
Everyone's got the same preferences.
Exchanges the same goods and services for the
same dollar bills.
Atom for atom. Everything unfolds exactly the
same in 2025 and in the 1950 on both
timelines. Okay. Timeline A,
timeline B.
Unless people have homothetic preferences,
meaning that the fraction of their income
they spend on each good is constant,
no matter how rich they are.
So no luxuries or inferior goods,
which is completely wrong,
right? You don't spend the same fraction on
food when you're starving as when when you're
richer. But unless people have homothetic
preferences,
that are the exact same preferences across
the population,
and totally stable over time.
Unless those three conditions are met,
there is a timeline B on which real GDP
growth chain-weighted across the years with
perfect measurement is any number.
Okay.
Isn't that crazy? I mean,
even the fact that there could be any
variation means that,
to my mind, real GDP is not a quantity.
Because it's like baking in the history. You
see what I'm saying?
Like a yardstick should,
it shouldn't matter the order in which you
measure things. It should order things in the
same way.
But the order in which things happen can
change what share of GDP a given good was
while it was growing quickly,
right?
So let's say there's two of us and one of us
is going to be rich one year,
and the other one is going to be rich the
other year. And the stuff that I like more,
I'm going to bid up the price.
I've got a lot of clones that have my
preferences and you've got a lot of clones.
We bid up the price more of the things we
like when we're rich.
The way things happen is that the things we
like are growing quickly in absolute units
while we happen to have the money.
So our preferences are kind of mostly
determining what GDP is.
And the things you like are growing quickly
when you and your clones have the money.
Real GDP is going to be higher across the two
years than if it's the other way,
where the things I like grow when I'm poor
and vice versa..
And it's that kind of effect that can mean
that you can scramble things up so that as
long as people depart from perfect
homotheticity,
constant preferences,
same across population,
then real GDP can be any number.
So maybe I've overinternalized this.
But given that I've overinternalized this,
I sort of feel like I can't separate the
theory from the overall opinion I think.
Yeah. I guess it's like a funny way of
framing it though. I think I would still call
it a quantity, but it's just it depends on
the history.
Because, I mean you know,
in in physics we could say the work done is
not a function of state.
It's not just these are the things that you
have and it's only determined by that.
It also depends on the path it took
historically.
But then there are also other quantities that
are just dependent on the state. Like the
internal energy or something.
Yeah.
I think that's sort of like a word choice,
though.
I don't think that really makes a difference
to your claim. So,
new topic. How do we detect the economic
singularity?
Okay. Yeah, that's a big one.
I don't have anything super original.
But I have something a little bit original.
William Nordhaus,
this Nobel Prize-winning economist,
has a paper from 2021 asking "Are we
approaching an economic singularity?" And his
answer was no,
because he observed that if we were,
we would see a bunch of macroeconomic
variables moving in sort of predictable
directions.
So we would see the capital share rising,
for instance.
I think that's just the most robust one. If
we're approaching a world of full automation
where we've got these robots and computers
that can do everything people can do,
and there's just more of them then the share
of income paid to their owners has to be
higher than the share paid to labor.
And it's got to grow ever higher as their
relative quantity increases.
But other examples like that.
So he then plots these things and finds that
they're all kind of as flat as they ever
were. So he says the singularity is not near.
And methodologically,
I think that's a great insight.
It can be. So the issue with it is that it
can be a really lagging indicator of whether
basically this tells us is a singularity
underway.
And it's sort of like being a weather
forecaster and just sort of looking out the
window and saying whether it's raining.
But I mean, if you start to see all of them
moving in the same direction a little bit,
that should be an update. And it'll only be
super lagging on the most
fully software intelligence explosion comes
first then it starts to impact the real
economy very discreetly later sort of
scenario.
Which I think is a possible scenario.
But it's good to know that in other scenarios
you can extract some real information from
the macroeconomic variables well in advance.
So anyway, Nordhaus did that.
One very small thing I've done is just get
the analyses up to date and expand on them a
little bit, add a few kind of similar
variables to the list and find that they're
all just as flat as they were when Nordhaus
measured them.
I guess that's not that surprising.
Which is not surprising,
but I can report that on Nordhaus's
methodology we're not seeing any more
evidence than we did in 2021 or 2015 when he
first wrote the paper actually. A sort of
subtler thing is that you can
look at what's called a network-adjusted
capital share.
The network-adjusted capital share is a
feature of a good.
So the capital share is a feature of an
economy, right? What fraction of all income
is received in exchange for the use of
capital.
Network-adjusted capital share asks:
for every dollar of revenue spent on that
good, if you trace it all the way back down
the supply chain,
how much of it is paid out in exchange for
value added by capital,
as opposed to value added by labor or taxes?
What was the thing that we were tracing?
A dollar of revenue.
Okay.
For example, Starbucks sells a cup of coffee.
You give them $5. Some of it is paid to the
people working at Starbucks.
So, you know, maybe $0.20 or let's say $1.
Some of it is received as profits by the
people who own the physical infrastructure of
Starbucks and the brand,
you know, the shareholders of Starbucks.
So that's another dollar. And let's put aside
taxes and $3 are spent on intermediate
inputs, say. I'm making up these numbers,
but you know.
So on the coffee beans and the cups and the
electricity to keep the lights on and all of
that. Now, as far as the Starbucks balance
sheet is concerned,
those are capital expenses.
But in reality, they're not all capital
expenses because we haven't traced them down
the chain. You've got to ask of every dollar
they get in revenue for coffee beans:
how much goes to the owners of the firm for
just owning the capital of the firm?
How much goes to the people working at the
firm for their labor,
and how much is spent on intermediate inputs
that that firm uses?
If it's a farm, you know,
then the tractor or something,
and then, okay, for every dollar they give
the tractor, you know,
they give John Deere. How much is spent on
capital labor and intermediate inputs at John
Deere. And so of that original $5 that you
gave,
you can in principle ask,
okay, it's all ultimately going to capital or
labor like all the way down the supply chain.
I meanin some sense, it never ends,
right?
You're cutting up the pennies ever smaller.
If you're going to try to do this like
manually, you'd have to sort of stop after a
while,
right? But you can say okay. Yeah.
What's the capital kind of contribution in
some sense to this cup of coffee as opposed
to the labor? So the reason this is relevant
is so one neat thing is that it sounds
totally intractable to compute.
There's this A.J. Jacobs book in which he...
You know A.J. Jacobs? He writes these sort of
funky books in which he does some sort of
stunt and writes about it.
And in one of them, he tried to thank
everyone in the world for his cup of coffee.
So at the beginning, he's thanking the
barista. But a few chapters in and he's,
like, thanking the people who made the paint
for the truck that carried the whatever.
I mean I haven't read the book,
but that's my understanding. So,
yeah, it sounds intractable. It turns out the
US Bureau of Economic Analysis and the OECD
similarly create these big input-output
tables every year where they actually try to
figure out relatively coarsely,
but still at a level of granularity like 87
industries or something I think it is,
how much of every dollar of revenue goes to,
if we're like an average firm in that
industry,
the immediate labour capital and then all the
intermediate inputs across all the other
industries. So you get a big matrix and it
turns out that you can work out what the
network-adjusted capital share is for any
item in the matrix with some matrix inversion
and stuff. You kind of do that infinite sort
of tracing down the chain, and then you can
plot what that is over time for any given
good.
You might want to know if a good is
approaching having a network-adjusted capital
share of one. For a few reasons.
One is that if it's a kind of good that can
later drive the growth of everything else.
If it's like semiconductors or whatever,
if we've like fully automated semiconductor
production, we think that that will then
drive the intelligence explosion, which
drives the automation of other things.
This will actually be more of a leading
indicator, right?
You'll actually be able to do some weather
forecasting instead of just looking out the
window. But secondly,
you might just be interested in it because if
a good if a good has a network-adjusted
capital share of one,
everyone on earth could die and it would
still keep getting cranked out,
right? And yeah, I mean,
that's the kind of thing that could have a
big effect on the world.
Having self-replicating in some sense in this
sort of grand supply chain sense.
Having self-replicating drones would be
really important for military purposes,
right? Or robocops or somethin.
You know, if you don't no longer need the
support of your population to have have the
equipment to suppress them,
maybe dictatorship will be more stable.
Or if some omnicidal maniac wants to create
little killer machines that can
self-replicate. That'll all.
People can intervene to shut it down or
something, but at least you've crossed some
threshold of it being more worrying when you
have a closed loop in which no one needs to
actively intervene to keep the system going.
Which then leaves its mark on the world.
So anyway, I just thought that would be kind
of interesting, compute that and invert that
matrix and look at that network-adjusted
capital share for different goods over time.
And it turns out they're all basically as
flat as a board.
Yeah. For semiconductors,
that's not at the industry level that's
computed annually. It's only every five
years,
unfortunately. So it's always really out of
date. But yeah,
it's like 50-50. It's sort of always been
50-50.
So there you go. So you got one more little
data point that maybe the singularity is not
near, at least if it's going to be an
industrially intense singularity early on.
I'm curious, if we kind of went back in time
here and we tried to apply a similar style of
thinking to whether or not we could have
detected the Industrial Revolution happening.
Suppose we went to like 1700 in France and we
tried to detect the Industrial revolution
coming in advance with some kind of leading
indicator, what would people have done?
How well would you have done?
Yeah. I mean, in practice,
I think the people at the time would not have
done well.
Yeah.
I mean, even if they'd kind of known to keep
an eye out for something like that.
I do want to share a bit about the economists
of 1700s France,
the Physiocrats they were called.
They created the first of those input-output
tables,
I think, as far as we know,
that the BEA now studiously collects every
year or five years. And in some ways they had
some sort of free market thoughts which
Adam Smith incorporated into Wealth of
Nations.
But they also had this crazy idea,
at least a lot of them did,
that the real wealth came from the earth.
You know, it came from the land,
and everything else that people do,
that's just some icing on the cake.
That's rearranging the wood into a chair and
stuff and,
yeah, you've got to do that down the line.
But to really get more wealth,
you've got to speed up the beginning of the
pipeline.
And so all these moves to the cities and all
those first glimmers of industrialization,
they were all moving the wrong way. So
everyone had to get back to the countryside
for economic growth -
it's like the worst possible advice you you
can imagine! And maybe an early example of
economic theory gone awry.
But anyway, I think the things to do would
have been first
to look around for what's going on in other
countries.
Sort of an obvious example,
but don't assume that what's happening in
France is the best guide to what will be
happening in France in a few decades or in a
century or something. Likewise,
I'm sort of realizing my own
folly as I'm speaking,
but I was looking at the US for all these
network-adjusted capital shares. But there
are other countries like Japan where there's
a fair bit more automation,
at least in a lot of sectors.
And because that demonstrates that
something's feasible, that means that even if
for whatever reason it's not been implemented
here, it might come very quickly.
As industrialization came over the English
Channel a bit later.
One question that I have about explosive
growth is that a lot of the framing is
usually around increasing the number of AI
researchers.
But you also mentioned that there are these
kinds of other returns to scale.
So the framing I want to take here is:
what exactly are the returns to tacit
knowledge? So in particular,
suppose you could either 10x the human
population on the one hand,
and on the other hand,
you could allow all humans to share all of
their tacit knowledge with each other.
So which do you think would have a bigger
effect on economic growth, and what would
that look like? Is this like a growth effect
in the long run?
If we maintain this,is there some kind of
level effect here?
How do you expect this to work out?
I expect that doubling the population would
have a bigger effect on on growth.
I mean, again, mind you,
the increasing returns to scale that
specialization could allow for or just better
coordination
through one person's knowledge immediately
being accessed by another,
that's something that we,
on some level have been improving on over
time. I mean, not that quickly.
But as I mentioned, I think the invention of
the internet and modern telecommunications
and just all the academic infrastructure we
have for sharing papers,
and even on the tacit knowledge front I just
think we've been getting better at
communicating over time.
And if that's had an effect,
it's been small enough that we've been able
to not be completely crazy when sort of
talking about the economy in really broad
brushstrokes. Even proposing that it's a
significant source of increasing returns to
scale such that with these big-brained AIs,
one 10x:ed AI is going to be
a lot more productive than ten smaller ones.
I think it's true. But the idea that this
would be such a large effect that it would
outweigh the raw replication effect – that
with ten times more people you'd have kind of
at baseline, a ten times larger economy.
And here the replication argument does work,
mind you. Because with ten times more people
with the same tastes,
they're going to make ten times as much
stuff. Yeah, I think I'd be very surprised if
this effect of kind of mind melding
was big enough to outweigh the full-on
elasticity of one "double the population,
double output" replication point.
But I don't know. I mean,
obviously we'd be really pushing the boundary
to develop AIs that could immediately share
all their tacit knowledge.
But actually there is another kind of way you
can shed some light on this,
which is just by looking at the returns to
working longer hours.
Something I've sort of been interested in is
why exactly is it that in some professions
you make so much more by working really long
hours?
You know, as a lawyer or whatever.
Often you'll have an early investment banker
or something, you'll have people working like
80-hour weeks. And that's really miserable.
And you think, "Well,
why can't you just get a normal job where
you're working like 40 hours a week but get
paid half as much?" But actually you get paid
less than half as much,
or the job is not available at all.
And I mean, sure, there are some genuine
workaholics who would want to work that much,
but I think there are people for whom it sort
of feels suboptimal.
And the reason why that's what we end up
doing it's got to be because of this sort of
increasing returns to scale thing.
Where it's more productive to have this
breadth of tasks associated with a project
located in a single brain.
If you can't all get it into a single person,
then you make do with as few as you can so
that you can have the integration of the
disparate parts of the project. So that's
kind of like sharing tacit knowledge with one
other person. It's just yourself for another
workweek crammed into the same workweek.
And yeah the CRS's view,
the constant returns to scale view,
would say that working 80 hours gets you
twice the salary or less if you're kind of
overworked or something.
And the increasing returns to scale view says
that it gets you more than twice the salary.
And it does. But it doesn't get you more than
four times the salary.
See what I'm saying? So,
the primary effect is,
I think, just the fact that you're working
more hours.
Is that the case, though?
I guess, in this case I wonder how comparable
it is.
If we're really sharing the tacit knowledge
across the entire economy,
of all the possible agents in this economy.
And it feels potentially a lot bigger than
just sharing with one person and stuffing it
within the same week.
It's a good point. It's a good point. I would
wonder how much everyone needs everyone
else's tacit knowledge,
right? Because to a large extent,
we're just doing jobs that require
independent knowledge bases.
But I can't rule it out.
Maybe someone knows something that rules it
out, but I definitely take the point that it
might be a bit bigger than the investment
banker-returns to scale.
But it's a lower bound.
Intuitively, for me, it's something close to
an upper bound. But yeah,
I don't have more to say about that.
Which is going to be more valuable if we
consider the thought experiment of two half
Jupiter brains versus a Jupiter brain,
which one would be more valuable?
So, we all know that having twice as many
people leads to
more than twice as much output through
agglomeration.
You know, people cluster in cities nowadays
almost entirely because of each other.
Not because of some natural resource,
some port or something that the city happens
to be next to. And so everyone gets
to go to their own favorite kind of barber
and eat at their own favorite kind of
restaurant. Whereas if we were all just
living in towns of 100 people,
we'd all have to kind of make do with
something generic.
On current margins, economists tend to think
that this effect is pretty small,
right? That you get benefits from
specialization up to maybe the size of a
small city or something or a small country.
But beyond that, it's basically constant
returns to scale.
One argument for this is that the gains from
trade between pretty similar countries with
similar natural resource endowments and so on
are estimated to be pretty small.
It's sort of funny, economists are
associated,
I think, with the idea that tariffs are
really horrible and that there's all these
gains from trade. And implicitly that's an
argument about increasing returns to scale.
Because if there was constant returns to
scale, then why bother trading between
countries? Each country can just sort of chop
the land in half and each country produces
half as much on its own. And that is in fact
what is typically estimated.
The gains from trade between the US and
Canada or something,
it's maybe a few percent of GDP.
Which is a lot in the grand scheme,
but it's not that much.
And so all the recent sort of fretting about
tariffs and stuff.
There can be large short-run losses if you've
built up a trade network and then you
suddenly sever it, because the supply chain
has to be reorganized.
But in the long run, standard models,
standard estimates predict that the effect
should be small. Okay.
On the other hand, I think in the long run
the effects could be pretty large after all.
Interesting.
My sort of pet theory of this would be that a
large part of what's happening,
as we've developed more advanced technology,
is that we make better use of our latent
capacity for specialization.
So as academic fields have gotten more
specialized,
they've been making use of the fact that you
had more academics to fill all these
specializations. One thing we could have done
is just keep on calling everyone a natural
philosopher or whatever. Like basically all
the academics were back in the Middle Ages,
or theologian or lawyers.
But they had like four kinds of academics.
And then just have lots of people
largely duplicating each other's work. And if
there'd been this big influx in academics and
people hadn't yet come up with,
you know, molecular biology and all the rest
of it, then as they did so you'd be getting
all of these gains.
Which on some level you could attribute to
the development of molecular biology,
but on some level you've got to attribute it
to the fact that there's more people now to
take up this kind of further specialisation.
And likewise, if you suddenly moved a bunch
of people from small towns into a big city.
And you just had a bunch of
middle-of-the-road barbers and diners and so
on that hadn't yet specialised,
there would be these gains up to the degree
of specialisation that the size of the city
allowed for. But it wouldn't all be realised
at once. So, if a view like that is right,
I think it kind of helps to explain what
would otherwise be a bit suspicious.
Which is that, when moving from full
individual-level autarky where everyone's
self-sufficient. It's all,
you know, everyone has their own little house
on the prairie, right?
There's no gains from trade at all. And
moving from that to a modern
sort of small city or small country.
Assuming it's not got oil. That's that's a
big case where where there are gains from
trade. I should have said. But that's that's
clearly not about specialization.
That's just about a necessary natural
resource.
But yeah, you get all these gains from trade
up to that size and then it just sort of
suddenly levels off. And that's just a fact
of nature.
It seems a little suspicious to me that,
in the whole space of ways to arrange matter
and energy to produce things of value,
you get strong benefits from specialization
up to the size of a small modern city in 2025
and then no gains at all after that?
I would guess that with enough time,
if we just stagnated at current population
sizes, we would develop more and more
specializations and we would continue to
urbanize globally and not just increase the
fraction of people in cities but the fraction
of people in the biggest cities.
And in time we would extract ever more
benefits of specialization.
But just what has happened over the last,
I don't know, century or something, is that
populations have grown really quickly and
urbanization has grown really quickly.
And that has sort of created this big
overhang,
if you like, of potential specialization to
to be exploited.
That's kind of one big thought experiment.
Here's another thought experiment.
It seems to me that the welfare capacity of a
being,
of a brain, probably tends to grow
superlinearly in the size of the brain.
So in the EA [Effective Altruism] community,
in the animal welfare community,
there's a pretty strong inclination that it
goes the other way.
And a lot of research that seems to support
the conclusion that it goes the other way.
People will, like Rethink Priorities when
coming up with their animal welfare weights,
say there are two things that matter.
There's how intensely a given creature can
feel pleasure or pain,
like on a utilitarian perspective.
How intensely it can feel pleasure and pain,
and its list of capacities for valenced
experience. Can it feel depression? Can it
feel anxiety? Can it feel elation?
And they find that small creatures can check
off a surprisingly large fraction of the
boxes that large ones can,
including ourselves.
And the intensity on some level doesn't seem
to be that much smaller.
You know, little creatures can squeal like
they're just about,
asintensely feeling pain or pleasure as we
can.
And so they think, "Okay,
well, so all this extra gray matter that
we've got isn't adding much to our welfare
capacity." And I think this is neglecting a
dimension of welfare capacity that is
analogous to the size of a population.
Which I call the "size" of the experience. So
my thought experiment is
if you imagine a split-brain case.
You know, these split-brain patients that on
some level seem to have two separate streams
of experience. If someone's submerged in an
ice bath,
say. So, they're experiencing some pain all
over their body. And then you cut their
corpus callosum. Now suddenly you have two
streams of experience,
each of which is only feeling cold on half of
a body.
On the Rethink Priorities-type view,
you've basically doubled the amount of pain
in the world because you now have two beings
that check off the same list and have the
same intensity. But I would say that's crazy.
That you've just by one snip,
you've doubled the amount of pain in this
bathtub. No. You've probably left it about
the same,
or if anything, diminished it a little bit to
the extent that there are sort of
psychologically sophisticated kinds of
suffering that can only arise when you have
the hemispheres communicating. And likewise I
just think it's a bit absurd to think that
there's this intense non-monotonicity where
if you disaggregated my neurons into lots of
small mouse brains and then even smaller fly
brains,
that the sum of the welfare capacities across
the whole set of creatures would rise and
rise and rise until suddenly it was dust.
And then, it's down to zero.
I think it's probably more integration,
more complexity means more capacity
for value itself. Okay,
so I've got more thought experiments along
those lines. So, I've got my economics
thought and I've got this brain thought.
So what does this all add up to?
And yeah, what we were talking about before,
the parallelizability thought.
About how, at least eventually on some
margin,
latency could be a really strong bottleneck.
And so you'd get way more out of one big chip
than two small chips or one automated
researcher that in some sense has a double
brain rather than two people with normal
brain side by side. All of it seems to add up
to the possibility that in a sort
of radical future where we're turning Jupiter
into some giant computer
two half Jupiter brains could could be a lot
less valuable than one whole Jupiter brain.
I don't think this kind of extends all the
way up.
Apparently the galaxy is like 100,000 light
years across.
So unless we're engaged in this very long,
slow dance where we're all playing little
parts that can't communicate with each other
for a very long time,
you probably at some point get plain old
constant returns to scale.
Because you just kind of can't communicate
efficiently over far enough distances.
But but you'd have to get pretty big for the
speed of light to start being an issue here.
And if you get increasing returns to scale
for a long way,
I think that has some potential implications.
One of them is that we should perhaps expect
a more peaceful future than we might have
anticipated, if you believe the liberal peace
hypothesis that countries that benefit from
being able to trade with each other are less
likely to go to war.
Well, the gains from trade are just going to
rise over time as we make use of our capacity
for specialization. At least if we don't grow
into space or something more quickly than we
can absorb that capacity.
So if the increasing returns to scale thing
actually starts showing up more,
then maybe that's kind of good news on the
peace front. Another implication maybe is
that if you're thinking of trying to invest
for a really,
really, really, really long time to turn
matter and energy into hedonium or whatever.
Then you should be more risk tolerant than
you otherwise would have been. Because it's
better to own one whole Jupiter brain than
two half Jupiter brains.
I guess this is true prudentially as well,
if you're just trying to be a kind of
hedonist, like a selfishly hedonist futurist
and you want to turn yourself into a Jupiter
brain. Then you want to increase the chance
you get the whole.
You care about getting the whole one more
than like a, 50-50 chance of getting half.
I don't know all the places this could go.
It's obviously opening a massive can of
worms,
but I think it's sort of fun to think about.
Yeah. On the peace thing,
do you think this is a core explanation for
why we've seen fewer and fewer individual
nation states over time?
If we compare say, 1800 to today,
is the reason actually just returns to scale?
And it's all just like flowing through this.
Yeah. I've wondered a little bit about that.
I don't think it necessarily tells us how big
the returns to scale are.
Because, you know, as long as it's true that
to some extent we all benefit from being able
to coordinate better. Then there's going to
be some pressure toward having larger states.
And with enough time,
just like over the course of evolution even
traits with very small fitness advantages can
eventually become dominant.
So, yeah, I don't think it tells us anything
quantitatively.
But states are kind of effective mechanisms
for coordinating groups of people.
And they've been getting bigger over time.
And this is an example of something I was
saying before.
That like this sort of latency bottleneck or
whatever you want to call it,
coordination bottleneck,
has been around all along.
It doesn't feature in most of our growth
models,
really. Certainly not the Jones.
And it's been getting relieved over time.
And so to some extent,
we've got to attribute the growth that we've
had so far to the ongoing relief of this
bottleneck.
Which means that if in the event of full
automation and
literal or almost literal galaxy brains that
constraint's getting relieved more quickly,
that would be a kind of new dimension on
which growth could proceed faster than
before.
So this is not necessarily Phil's theory for
why world government is going to exist if you
just go with the long-run equilibrium?
Oh, no. Well, it's an argument for thinking
that that eventually a world government is
more likely than you would have thought
otherwise.
Sure. But whatever form that takes,
I mean, it could be that countries just
collaborate more over time or something or
trade more with each other over time, and the
regulations sync up and it ends up kind of
functionally looking like a world government.
I don't know, this prediction would have
seemed a lot more sensible a few decades ago
or something, or maybe 1990.
And since then, I think for the first time in
history basically,
we've seen an increase in the number of
countries in the world, right?
Not a huge increase, but Czechoslovakia split
up and,
you know, various countries have split up and
they haven't merged.
Like all of history?
Maybe it's overstating it,
but actually I was reading something a while
back thatthis past 35 years,
basically has been unique on record.
We have no idea like this average size of
tribes,
but yeah, it's been anomalous as far as the
records go,
I think. And you know,
there are more wars now than there were
ten years ago or whatever.
So, you never know what the future holds.
But this is just one argument on the on the
pile for the thought that,
in the long run if the more efficient
arrangement wins out,
it'll probably involve kind of more
integration and fewer states.
Because the degree of increasing returns to
scale will itself rise over time as we make
better use of our latent capacity.
Okay. I think this is a good place to end.
Thank you for joining us on the podcast,
Phil.
Thank you, Anson.
And thank you all for tuning in to Epoch AI's
podcast.
We look forward to you joining us for future
episodes.