The Circuit

Ben Bajarin and Jay Goldberg discuss Jay's recent article and flesh out a thesis related to AI's impact on different segments of the semiconductor industry. 

What is The Circuit?

A podcast about the business and market of semiconductors

[Ben Bajarin]: Hello everyone, welcome to another
episode of The Circuit. I am Ben Beharon.

[Jay Goldberg]: Hello world, I'm Jay Goldberg.

[Ben Bajarin]: We, I had a joke a couple of
days ago, Jay and I were talking about where

[Ben Bajarin]: I would introduce myself as Jay
Goldberg, just to throw everybody off and see

[Ben Bajarin]: what the response would be. But
then we concluded that might not be a good

[Ben Bajarin]: idea.

[Jay Goldberg]: I think you've just accomplished
confusing everybody.

[Ben Bajarin]: Yes. All right. Ben Beharon is
the one talking now. There you go. All right.

[Ben Bajarin]: So it's been an interesting week
on the back of a couple of semiconductor earnings

[Ben Bajarin]: calls, ASML and TSMC. This created
a lot of different talking points, questions.

[Ben Bajarin]: Something that we're going to
dive into today is something we've talked a

[Ben Bajarin]: little bit about before. I wish
we had the wherewithal to go even deeper on

[Ben Bajarin]: this thesis, but it's something
that Jay and I are working on as well. Relative

[Ben Bajarin]: to the impact that AI has on
a number of different computing segments, not

[Ben Bajarin]: just data center, but edge, we've
done some specific, like I said, episodes on

[Ben Bajarin]: this, but Jay recently wrote
a thesis. on this at Digits to Dollars, which

[Ben Bajarin]: we can link to in the show notes.
So I'm just going to let Jay outline his thesis

[Ben Bajarin]: because I followed some of the
people that were questioning him on Twitter

[Ben Bajarin]: and seeing the discussion. So
lob the thesis out on us and then let's unpack

[Ben Bajarin]: it.

[Jay Goldberg]: So I approach this topic of
AI semis. Obviously, everyone's talking about

[Jay Goldberg]: it a lot. We've done a few episodes
on it. And I realized that we sort of addressed

[Jay Goldberg]: different parts of the elephant.
And so I wanted to take a step back and sort

[Jay Goldberg]: of look at the whole thing.
And I think from a high level strategic perspective,

[Jay Goldberg]: there are really three questions
around AI semis today. And the, the first one

[Jay Goldberg]: is will AI be additive to the
addressable market, the TAM for semiconductors,

[Jay Goldberg]: or will it just cannibalize
CPUs and other stuff and market stays the same?

[Jay Goldberg]: Second question is how will
the market for edge inference, or excuse me,

[Jay Goldberg]: for inference in general, shape
up? What's that competitive dynamic gonna look

[Jay Goldberg]: like? And the last question
is can anybody displace NVIDIA, who is clearly

[Jay Goldberg]: the market leader and the dominant
player in this space right now? And I think

[Jay Goldberg]: those are the three sort of
fundamental questions that are gonna determine

[Jay Goldberg]: some important trends for the
next decade of semiconductor. market and who

[Jay Goldberg]: the winners and losers are going
to be.

[Ben Bajarin]: Okay, I'm gonna start actually
with your third point as a starting point.

[Ben Bajarin]: So this is something I've been
thinking about and I haven't asked you this

[Ben Bajarin]: question but I've been talking
to other people. We talked a couple episodes

[Ben Bajarin]: ago when we talked about ARM
and the data center about everybody sort of,

[Ben Bajarin]: everybody who's in the know,
unanimously agreeing that much of the workload

[Ben Bajarin]: in AI is gonna move from training
to inference. And in some cases I've seen folks

[Ben Bajarin]: and investors on the sell side,
estimate this to be, you know, 77, 78% of the

[Ben Bajarin]: market today in terms of dollars
is training. Um, but that could move to a 40,

[Ben Bajarin]: 60% split 50 over the next five
years. So people are assuming that training

[Ben Bajarin]: will still be important. Let's
again, let's say it's 50%, 60% great. but a

[Ben Bajarin]: huge portion of that moving dollars
moving to inference. So the question is, if

[Ben Bajarin]: we believe that, and then perhaps
over the 10 year timeframe, even more moves

[Ben Bajarin]: to inference, maybe that's a
20% training, 70% inference, does it even matter

[Ben Bajarin]: if people compete with Nvidia
in training? Because there's so much other

[Ben Bajarin]: opportunity for inference, why
would we not be focusing more on that versus

[Ben Bajarin]: trying to compete with Nvidia?

[Jay Goldberg]: So I think there's a few parts
to that. A lot of that is around inference,

[Jay Goldberg]: which is kind of my second question.
But in terms of Nvidia's position in the market,

[Jay Goldberg]: I think we should probably break
down a little bit what dominance means. Today,

[Jay Goldberg]: they are clearly dominant in
training, close to 100% of the market. The

[Jay Goldberg]: only real

[Ben Bajarin]: Yep.

[Jay Goldberg]: competition they have there
are a few internal things at like Google, who

[Jay Goldberg]: also is using Nvidia for training
as well. So that's part of it. then the question

[Jay Goldberg]: is, can they extend their position
and training into inference?

[Ben Bajarin]: Mm-hmm.

[Jay Goldberg]: And I think we're at a place
now where most people assume that AI equals

[Jay Goldberg]: GPU

[Ben Bajarin]: Mm-hmm.

[Jay Goldberg]: and GPU equals Nvidia.

[Ben Bajarin]: Mm-hmm.

[Jay Goldberg]: We can debate who has a better
product and is AMD's MI 300 competitive versus

[Ben Bajarin]: Right.

[Jay Goldberg]: whatever. But I think fundamentally,
it comes down to the fact that the big customers

[Jay Goldberg]: in particular are not going
to ever settle for a world in which they are

[Jay Goldberg]: completely dependent on Nvidia,
especially when Nvidia controls both the silicon

[Jay Goldberg]: and the software layer around
CUDA. That's just an unacceptable, untenable

[Jay Goldberg]: position for the customers.
And so no matter what, this current status

[Jay Goldberg]: quo has to change. Now, I'm
not bearish on Nvidia by any means. They're

[Jay Goldberg]: gonna do fine, but absolutely,
they are not going to be able to extend that

[Jay Goldberg]: 100% share into inference. There's
no way that's gonna happen for a lot of reasons.

[Jay Goldberg]: Can they get displaced out of
training? I think... I think everybody's exploring

[Jay Goldberg]: that right now. And

[Ben Bajarin]: All right.

[Jay Goldberg]: I think it's gonna be tough.
I think it's actually gonna be pretty tough.

[Jay Goldberg]: But if you start to look at
some of these alternative software frameworks

[Jay Goldberg]: that are coming out, PyTorch
and Triton are on everybody's lips. I think

[Jay Goldberg]: there are a lot of people in
marketing who just assume, okay, PyTorch is

[Jay Goldberg]: coming and it's going to displace
CUDA and suddenly we're going to have a little

[Jay Goldberg]: bit of evening out in market
share and you can run PyTorch on an AMD or

[Jay Goldberg]: whatever. And that's going to
disrupt Nvidia's hold on training. I actually,

[Jay Goldberg]: I'm not convinced. I'm not convinced.
I think

[Ben Bajarin]: I

[Jay Goldberg]: there

[Ben Bajarin]: agree.

[Jay Goldberg]: is, there's a world in which
you just run PyTorch on CUDA. Right. Uh,

[Ben Bajarin]: Mm-hmm.

[Jay Goldberg]: they don't necessarily, in some
ways they compliment each other. Uh. I think.

[Jay Goldberg]: You have the software on training
is still very complex. I know a lot of companies

[Ben Bajarin]: Mm-hmm.

[Jay Goldberg]: have tried to get into it. It's
just, it's going to, it's going to be, it's

[Jay Goldberg]: going to be a battle. Um, and
I think it's just for a lot of companies, it's

[Jay Goldberg]: just not going to be worth it
as, especially because like you said, inference

[Jay Goldberg]: is going to become. The majority,
I would say, I would argue the vast majority

[Jay Goldberg]: of spend over

[Ben Bajarin]: Yep.

[Jay Goldberg]: time. And

[Ben Bajarin]: Yeah.

[Jay Goldberg]: so it there's how much effort
do you want to spend trying to disrupt training

[Jay Goldberg]: when that's, you know, 10% of
your spend five years from now,

[Ben Bajarin]: Exactly.

[Jay Goldberg]: right? You're gonna be much
more focused on the inference side and training

[Jay Goldberg]: stuff is pretty complicated.

[Ben Bajarin]: agreed.

[Jay Goldberg]: Unquestionably big companies,
Google comes to mind in particular, but others

[Jay Goldberg]: as well, they're gonna find
alternatives. They're not gonna use training.

[Jay Goldberg]: They're not gonna do all their
training on Nvidia long-term probably. But

[Jay Goldberg]: I think for the enterprise,
like why, you know, if you're an enterprise,

[Jay Goldberg]: you're a bank and you wanna
run training on your data, Like that's hard

[Jay Goldberg]: enough. You have to find AI
people, developers to just do that. And then

[Jay Goldberg]: to go the extra mile and like
fight the trend and create new software frameworks

[Jay Goldberg]: and integrations. Some people
will do that, but I think a lot of people just

[Jay Goldberg]: find going with the default
Nvidia for training is the easiest solution.

[Ben Bajarin]: Yeah. So that was kind of like
roundabout where I was getting to, right? Which

[Ben Bajarin]: is, I see the allure in competing
with them today and trying to compete with

[Ben Bajarin]: them. There's at least half a
dozen companies that we know of. There's probably

[Ben Bajarin]: more attempting this. It's not
a trivial problem. PyTorch, you're right, is

[Ben Bajarin]: gaining... significant sort of
share in terms of academia and research and

[Ben Bajarin]: people who are using it. But
again, that's not the only thing here that's

[Ben Bajarin]: being used. So it really does
feel like there's a still strong reason to

[Ben Bajarin]: be bullish on Nvidia for GPU.
But what you said is exactly right. One of

[Ben Bajarin]: the big questions is, what do
they do with Grace Hopper and how do they continue

[Ben Bajarin]: to move their companion CPU part?
Um, because if you, if you unpack Jensen's

[Ben Bajarin]: vision, and this is where I,
this is the thing I think is going to be the

[Ben Bajarin]: hardest for most companies to
really kind of grasp in their, their strategy

[Ben Bajarin]: to compete with Nvidia is Jensen
labels this under accelerated computing. And

[Ben Bajarin]: in my mind, he does not limit
accelerated computing to even the GPU or the

[Ben Bajarin]: CPU there's networking parts.
And I think there's other elements of AI Asics

[Ben Bajarin]: Nvidia can make. So if they are
still. a dominant platform player and a dominant

[Ben Bajarin]: seller than of GPUs who compete,
who bring companion parts to diversify these

[Ben Bajarin]: workloads. It's actually again,
a really strong story because his focus is

[Ben Bajarin]: just, let's just accelerate computing
and any bit of Silicon that can do that. Let's

[Ben Bajarin]: just go make it Nvidia. And I
think it's hard because who else is going to

[Ben Bajarin]: have that stack? Very few people,
maybe AMD, right? Maybe Intel could have all

[Ben Bajarin]: those pieces, but Jensen's much
more focused. And that's the part I think is

[Ben Bajarin]: just gonna be really, really
tricky at the end to bite off all those pieces.

[Ben Bajarin]: Whether he's successful in those
other areas, I don't know, that's debatable.

[Ben Bajarin]: But I think his vision's very
clear and his knowledge of the problem is also

[Ben Bajarin]: very clear.

[Jay Goldberg]: Yeah, so let me break that down.
There's a couple of things there. One, today

[Jay Goldberg]: when people ask me, like, why
is Nvidia so dominant in AI? The sort of default

[Jay Goldberg]: knee-jerk answer is to say CUDA.
CUDA is a software layer that in between the

[Jay Goldberg]: operating system and the chip,
and you can get a whole bunch of optimizations

[Jay Goldberg]: to run your systems much better
because you have CUDA. And I think the advantage

[Jay Goldberg]: that CUDA has conveyed to Nvidia
I think we could reasonably argue that is slowly

[Jay Goldberg]: being diluted by all the things
I talked about before. CUDA was really important

[Jay Goldberg]: in the early days of AI for
enabling all of this. Its sustainability as

[Jay Goldberg]: a durable competitive advantage
is probably peaking. That's how I'll put it.

[Ben Bajarin]: Mm-hmm.

[Jay Goldberg]: It's very strong, but it's probably
at its peak and it's going to wane. So then

[Jay Goldberg]: the next question will be, oh,
does that mean I can shorten video now? And

[Jay Goldberg]: I think the answer is no. setting
aside cyclical factors because they're going

[Jay Goldberg]: to blow up in a few quarters
because they always do but like in terms of

[Jay Goldberg]: Secular trend I think Nvidia
is still in a really good position and You're

[Jay Goldberg]: right. It's all the reasons
you stated they have this whole stack They

[Jay Goldberg]: have they have all the pieces
it will become very easy to just buy an AI

[Jay Goldberg]: system from Nvidia if you can
afford it You just get everything from them

[Jay Goldberg]: and for a lot of companies that
will be very appealing maybe not the hyperscalers

[Jay Goldberg]: but maybe them too, but certainly
for the enterprise like you just You buy everything,

[Jay Goldberg]: you buy a rack or two of Nvidia
solution. Plus on top of that you have all

[Jay Goldberg]: their software offerings. Their
models which are trained, their software frameworks

[Jay Goldberg]: which are trained for specific
industries and they have a dozen now. I think

[Jay Goldberg]: that's a really compelling vision
and I think that speaks to Jensen, like you

[Jay Goldberg]: said, really understands this
and knows where it's going. He's multiple steps

[Jay Goldberg]: ahead. The one pushback I would
give on that though is not everybody's going

[Jay Goldberg]: to want that. Because again,
it conveys a pretty high degree of lock-in,

[Jay Goldberg]: right? If AI is as important
as everybody seems to think it is. there's

[Jay Goldberg]: a risk in being so dependent
on somebody for an entire solution. And historically,

[Jay Goldberg]: those kinds of dependencies
don't last. That being said, it will appeal

[Jay Goldberg]: to enough people and Nvidia
is so far ahead on so many software fronts

[Jay Goldberg]: that I think Nvidia will do
just fine, even if CUDA goes away. I think

[Jay Goldberg]: they're in

[Ben Bajarin]: Oh,

[Jay Goldberg]: a really

[Ben Bajarin]: for sure.

[Jay Goldberg]: good position.

[Ben Bajarin]: Totally.

[Jay Goldberg]: There's gonna be a lot of friction
about people complaining about locking and

[Jay Goldberg]: all that, but still there's
a lot of strong appeal there.

[Ben Bajarin]: I totally agree. I mean, on the
merits of just the product quality themselves,

[Ben Bajarin]: right, that they're building,
it's hard to do what they're doing when it

[Ben Bajarin]: comes to these specific workloads.
So by there alone, right, I think my broader

[Ben Bajarin]: sort of just view is, one, we're
probably a couple years away, or sorry, not

[Ben Bajarin]: a couple years away, maybe closer
than that, but NVIDIA is much now more quickly

[Ben Bajarin]: going to be... a $100 billion
company in revenue sooner than we thought,

[Ben Bajarin]: thanks to this trend. That increases
their capex. That increases their ability to

[Ben Bajarin]: become a priority share at TSMC.
I think they're number four priority now or

[Ben Bajarin]: so, but increasing in that. It
just gives them so much leverage with, again,

[Ben Bajarin]: well-execution, good vision,
and a large software stack that it's hard to

[Ben Bajarin]: be displaced where they are today.
is part of my view. But that leads me to this

[Ben Bajarin]: sort of broader question you
and I have been circling around, which is,

[Ben Bajarin]: there are other areas of growth
to the data center that's not GPU. And I think

[Ben Bajarin]: we both believe GPU spend of
that is going to grow probably faster than

[Ben Bajarin]: CPU, but both are relevant. But
we've asked this question before about how

[Ben Bajarin]: much additional lift to the data
center, Tam, is AI going to bring? If it's

[Ben Bajarin]: a big number, like I've seen
numbers anywhere from, you know, 30 billion

[Ben Bajarin]: to an extra a hundred billion
over the next, you know, 10 years. So if it's

[Ben Bajarin]: a big number, I kind of feel
like if I was somebody trying to increase my

[Ben Bajarin]: strategy in the data center,
I would want to go after this green field of

[Ben Bajarin]: growth versus go for these areas
where people are more entrenched.

[Jay Goldberg]: Yeah, I think, well, I think
this gets into sort of the second question,

[Jay Goldberg]: which is inference, right? I
think going after training right now, especially

[Jay Goldberg]: for a startup is, um, I don't

[Ben Bajarin]: Yes.

[Jay Goldberg]: want to say suicidal, let's
call it challenging,

[Ben Bajarin]: challenging.

[Jay Goldberg]: very challenging, but the, the
bigger market will be inference.

[Ben Bajarin]: Mm-hmm.

[Jay Goldberg]: It's going to be a very big
market and it just can't be run on GPU entirely.

[Jay Goldberg]: Right. The

[Ben Bajarin]: Yes.

[Jay Goldberg]: economics will not work out
even if,

[Ben Bajarin]: Right.

[Jay Goldberg]: you know, especially with supply
conditions the way they are today. But there

[Jay Goldberg]: are other factors too, right?
A lot of these AI, I mean, AI is just software.

[Jay Goldberg]: And so you're going to run your
normal corporate software workload, and you

[Jay Goldberg]: can have some AI functionality
in it. There are a lot of times when the software

[Jay Goldberg]: architecture is going to dictate
that means running the AI on CPU alongside

[Jay Goldberg]: the other part of the workload.
So AI is not all GPU. Some of it's going to

[Jay Goldberg]: be on CPU. A lot of companies
are going to build accelerators for this. The

[Jay Goldberg]: hyperscalers I think are, you
know, are pretty invested in, in accelerators.

[Jay Goldberg]: That's an important category
for a lot of them. And they're going to, they're

[Jay Goldberg]: going to, that's, they're going
to run inference. In the cloud on accelerators,

[Jay Goldberg]: because the numbers just work
out much better that way.

[Ben Bajarin]: Mm-hmm.

[Jay Goldberg]: I haven't quantified this yet.
We've been talking about this a lot. But I

[Jay Goldberg]: think my intuition is that the
market for inference for generative AI is going

[Jay Goldberg]: to be the economics are going
to be so challenging that the only way it's

[Jay Goldberg]: going to work at is if you can
push a lot of that inference onto device, onto

[Jay Goldberg]: the edge. Because there and
the key thing there is the consumer, the customer

[Jay Goldberg]: is paying for the CapEx, they're
buying a phone, they're buying a PC that has

[Jay Goldberg]: some AI functionality and running
it on their device.

[Ben Bajarin]: Mm-hmm.

[Jay Goldberg]: That's going to offload it.
Doesn't have to be run in somebody's cloud.

[Jay Goldberg]: And I think that's the only
way this really works out given the way that

[Jay Goldberg]: generative AI is taking off.
And so, yeah, that's, that's where the opportunity

[Jay Goldberg]: is in those, in those areas
around inference.

[Ben Bajarin]: And I'd add another element that
we highlighted last week that I still think

[Ben Bajarin]: is just one of the most fascinating
things to think about is the other argument

[Ben Bajarin]: that why all of this can't continue
to be done in the cloud is scarcity of resource

[Ben Bajarin]: of energy. And

[Jay Goldberg]: Right.

[Ben Bajarin]: so to your point, right, CapEx,
it gets offloaded if I'm now using another

[Ben Bajarin]: device, my edge device, my car,
my PC, my automotive, my camera that's sitting

[Ben Bajarin]: right on a stoplight, um, it's,
it's the one handling that power, right? So

[Ben Bajarin]: I'm offloading power as a part
of that as well, because I think you could

[Ben Bajarin]: make the strong argument like
we have that we just don't have the grid for

[Ben Bajarin]: all of this to be run, you know,
in, in the cloud, especially amongst the top

[Ben Bajarin]: three hyperscalers, let alone
people include in, in broad terms, Apple as

[Ben Bajarin]: a top four U S hyperscaler, you
know, they, it's, you just can't run all this

[Ben Bajarin]: in the cloud. So I think that's
an important reason, which again, goes back

[Ben Bajarin]: to your other point, right? A
thesis of, of on device, which we did a whole

[Ben Bajarin]: segment on. And I think now we've
seen a handful of demos and a bit of extra

[Ben Bajarin]: conversations on device. It's
definitely not there today. But at some point

[Ben Bajarin]: in time, you'll be able to do
a whole lot more of this on device, and it

[Ben Bajarin]: will feel not that far off from
the things you've experienced in cloud-centric

[Ben Bajarin]: experiences.

[Jay Goldberg]: Yeah. I think, I think that's,
that's how this is going to work out. And I

[Jay Goldberg]: think that it's just, it's just
too cumbersome and the, the workloads are too

[Jay Goldberg]: big to really be run any other
way. It has to be some significant portion

[Jay Goldberg]: of offload from the cloud.

[Ben Bajarin]: So the question here that sort
of plagues me is, it feels like it's gonna

[Ben Bajarin]: be very hard to run this hybrid
AI environment that a lot of people are talking

[Ben Bajarin]: about, meaning that I provide
a service, I'm a cloud provider, and I realize

[Ben Bajarin]: that I've got tons of capabilities
here to do what I wanna do in terms of this

[Ben Bajarin]: service, what people are paying
for. But yes, I want to offload that to the

[Ben Bajarin]: device, but I need to know how
much that device can be offloaded to that device.

[Ben Bajarin]: And that's not going to be an
all things equal scenario, right? Devices that

[Ben Bajarin]: are five years old are going
to need a whole lot more cloud help than devices

[Ben Bajarin]: that are one year old. But people
talk about this hybrid environment. I want

[Ben Bajarin]: to have some of it in the cloud
and some of it on device. I feel like that's

[Ben Bajarin]: a really tricky architecture.
to talk about because again, it feels like

[Ben Bajarin]: the service needs to know, well,
I can't offload that or it will be a terrible

[Ben Bajarin]: experience or it's a capable
device, I can't offload 80%. I just don't know

[Ben Bajarin]: how this gets worked out but
that's kind of how people talk about this hybrid

[Ben Bajarin]: cloud on device today. Could
change in three years, but it feels very tricky,

[Ben Bajarin]: very complex to me to do that.

[Jay Goldberg]: I think you just described Android.

[Ben Bajarin]: Well, that's a whole different
issue in my brain,

[Jay Goldberg]: Yeah,

[Ben Bajarin]: but you're 100% right.

[Jay Goldberg]: I won't start my Android rant,
but I think when you talk about the edge, we're

[Jay Goldberg]: really talking about three things.
We're talking about PCs, iPhone, and Android.

[Jay Goldberg]: Yes, there's cameras, those
will come, and then there's automotive someday

[Jay Goldberg]: further out. But for the moment,
what we're really concerned about is laptop

[Jay Goldberg]: and phone. And at their event,
what? Two months ago, AMD actually... started

[Jay Goldberg]: talking about that, including
some, some neural processing blocks in their

[Jay Goldberg]: seat, in their laptop CPUs.
Apple's been doing it for a while. Obviously

[Jay Goldberg]: they have some in the phone
as well.

[Ben Bajarin]: Mm-hmm.

[Jay Goldberg]: Right. That's, that's really
where this edge inference is going to take

[Jay Goldberg]: place is in those kinds of devices.
And so what's going to, what's it going to

[Jay Goldberg]: take to accomplish that is,
uh, Microsoft has to get windows to the point

[Jay Goldberg]: where it can do it.

[Ben Bajarin]: Mm-hmm.

[Jay Goldberg]: Apple has to, it has to do likewise
for both Mac OS and iOS. And I would say, uh,

[Jay Goldberg]: Microsoft is clearly fully invested
in it and they're, they love generative AI.

[Jay Goldberg]: And I think

[Ben Bajarin]: Yeah.

[Jay Goldberg]: I have to imagine that there
are teams inside Microsoft working pretty heavily

[Jay Goldberg]: to bring that. transform or
to support into Windows sooner rather than

[Jay Goldberg]: later. Apple already done it
for the Mac. And so once those frameworks get

[Jay Goldberg]: set up, I think it actually
can happen pretty quickly. We will have this

[Jay Goldberg]: problem with Android where there
are going to be a lot of devices that can't

[Jay Goldberg]: run it for years.

[Ben Bajarin]: Mm-hmm.

[Jay Goldberg]: And I think that's just one
more thing that has to add that to the list

[Jay Goldberg]: of problems that Google has
with Android. Because when Apple thinks it's,

[Jay Goldberg]: you know, consumers really want
generative AI support on the iPhone, they'll

[Jay Goldberg]: launch it, you know, if it's
not in this iPhone, it'll be, you know, whenever

[Jay Goldberg]: six months after they think
consumers are ready for it, right?

[Ben Bajarin]: Yeah. Well, and

[Jay Goldberg]: So

[Ben Bajarin]: it's.

[Jay Goldberg]: I would be surprised if they
don't talk about something along these lines

[Jay Goldberg]: in September with the new iPhone.

[Ben Bajarin]: Sure. So there was a report,
I'm sure everybody who listens to this saw

[Ben Bajarin]: that Apple, Mark Gurman wrote
it at Bloomberg and just saying that they are

[Ben Bajarin]: working on their own GPT model.
It sounds like it's a pretty large model in

[Ben Bajarin]: terms of overall size, like definitely
large enough that whatever they're. building

[Ben Bajarin]: data set wise is not going to
run on device at that size. I mean, roughly

[Ben Bajarin]: the stuff we've tried today seems
to be a successful if it's in the 10 to 15

[Ben Bajarin]: billion parameter range. Anything
north of that is crashing devices and running

[Ben Bajarin]: exceptionally slow. But that's
today, right? That's not where we'll be in

[Ben Bajarin]: two to three years. But it shows
kind of what's possible on device versus the

[Ben Bajarin]: size of cloud. But Apple, 100%.
right, is going to want to do this. And arguably,

[Ben Bajarin]: like I said, they are a top four
hyperscaler. They can create that cloud to

[Ben Bajarin]: device infrastructure as good
as anybody, if they want, um, to handle their,

[Ben Bajarin]: their device fragmentation, which
as you point out is not as, not nearly as,

[Ben Bajarin]: as tricky as Google's.

[Jay Goldberg]: Yeah, Apple technically is capable
of doing it today. There's no question. If

[Jay Goldberg]: they wanted to get generative
AI working on the iPhone, it could happen today.

[Jay Goldberg]: I think the question for Apple
is, or the question that Apple is asking is

[Jay Goldberg]: more, what are we going to use
this for? Apple doesn't like to add features

[Jay Goldberg]: that consumers don't care about.

[Ben Bajarin]: Right.

[Jay Goldberg]: How much do consumers really
care about generative AI? How important is

[Jay Goldberg]: it to the user experience, the
human experience, excuse me,

[Ben Bajarin]: Right.

[Jay Goldberg]: on an iPhone? And there are
a lot of people who think that Apple is behind

[Jay Goldberg]: in AI as well, because they
don't have generative AI today. I think they're,

[Jay Goldberg]: they're trying to figure out
as am I, and a lot of people, like what is

[Jay Goldberg]: actually really useful for.
And I actually had a Twitter debate with somebody

[Jay Goldberg]: who was saying, you know, AI
is going to be really important, everyone's

[Jay Goldberg]: going to want it. And I said,
well, how much more would you pay for a phone

[Jay Goldberg]: that does

[Ben Bajarin]: Sure.

[Jay Goldberg]: chat GPT or stable diffusion
on the phone? And he says, well, I already

[Jay Goldberg]: paid chat GPT $20 a month. And
I'm like, yeah, I understand that. But that's

[Jay Goldberg]: not the question. You like generative
AI, you're willing to pay for it. How much

[Jay Goldberg]: though are you willing to pay
for it to work on your phone in airplane mode?

[Jay Goldberg]: And I don't think anybody knows
the answer to that question yet.

[Ben Bajarin]: So there's another though, part
of this that feeds into this that I think is

[Ben Bajarin]: fascinating. And this again speaks
back to companies who are prioritizing on device,

[Ben Bajarin]: those who are not be playing
both sides of cloud to device in terms of their

[Ben Bajarin]: roadmap, is you exactly rightly
point out, I don't think anybody is gonna pay

[Ben Bajarin]: more for these things. I think
it's gonna have to just be an evolution of

[Ben Bajarin]: the silicon's... roadmap to increase
those features. But if I'm a silicon designer

[Ben Bajarin]: today, so let's just say this
is MediaTek, Qualcomm, and Apple, to some degree

[Ben Bajarin]: Intel and AMD on PCs, you do
have to sort of make some decision about how

[Ben Bajarin]: much transistor budget you're
gonna throw to something like an NPU, because

[Ben Bajarin]: it's a relevant decision, right?
If I believe that I need that to compete, I

[Ben Bajarin]: gotta take that from something
else, right? I gotta take it from my GPU blocks.

[Ben Bajarin]: my CPU blocks, right, something
else, right? So they need to make calls on

[Ben Bajarin]: how important that is by how
much diary are they're gonna commit to these

[Ben Bajarin]: things going forward. And that's
a fascinating trade-off, right, that I think

[Ben Bajarin]: people are stuck with these next
two years, when again, you and I can argue

[Ben Bajarin]: all day, people will 100% be
willing to pay for those silicon bits in the

[Ben Bajarin]: data center more. They're not
gonna pay for them on device. And so that's

[Ben Bajarin]: just a fascinating dynamic of
maybe how fast the capabilities develop at

[Ben Bajarin]: the edge, when again, you've
got to make these trade-offs with your transistor

[Ben Bajarin]: budget.

[Jay Goldberg]: Yeah, I think that's a really
good way of framing it. And I think that sort

[Jay Goldberg]: of throws light on what I was
saying about Apple is, Apple at this point

[Jay Goldberg]: doesn't seem to see the need
to throw those blocks, those resources at it.

[Jay Goldberg]: They just don't see it as necessary.

[Jay Goldberg]: And I don't think anyone knows
when that will change.

[Ben Bajarin]: Well, we'll see. I mean, I think
we'll see with two fundamental things in my

[Ben Bajarin]: opinion, that'll happen this
fall, right? Qualcomm is going to unveil their

[Ben Bajarin]: new PC chip, PC chip architecture.
My hunch is that's going to be loaded with

[Ben Bajarin]: tops. Um, Apple has historically
only bumped their tops up. If I recall two

[Ben Bajarin]: to three, you know, uh, I think
they went from 10 to 15. Right. So maybe five

[Ben Bajarin]: or so. So, so what they throw
at tops in next generation products will be

[Ben Bajarin]: telling. Right. If it's goes
from 15 to 30, that's a pretty big jump. Right.

[Ben Bajarin]: But if it goes from 15 to eight
to 18, you know, you'll, you'll again, you'll

[Ben Bajarin]: just sort of see how they're
prioritizing. So, um, that's an interesting

[Ben Bajarin]: bit, but, but you're right. I
think that's, that's really the call they've

[Ben Bajarin]: got to make is, is how much C
block they throw to this stuff and it, and

[Ben Bajarin]: could it be premature, you know?

[Jay Goldberg]: Yeah, Apple, I think, typically
tends to be a little bit conservative in jumping

[Jay Goldberg]: on the latest feature, right?
The original iPhone was a 2G phone deep into

[Jay Goldberg]: the 3G era. That's the classic
example. And they've had AI on their phone

[Jay Goldberg]: longer than anybody, right?
They don't call it that. It's a neural engine.

[Jay Goldberg]: And it's there for a very specific
purpose to help with image processing. I don't,

[Jay Goldberg]: yeah, it would be interesting
to see it to what degree they change it, right?

[Jay Goldberg]: Or do they just sort of keep
going at their steady incremental pace?

[Ben Bajarin]: Yeah. So I want to end on just
circling back to the question at hand about

[Ben Bajarin]: is there even a way to sort of
put a model around the additional lift of dollars

[Ben Bajarin]: that's coming to AI? I think
we agree we can't do it on device. But to that

[Ben Bajarin]: point, in a number of earnings
calls, as well as some commentary of some of

[Ben Bajarin]: the people who are doing this,
it appears that it The answer is others haven't

[Ben Bajarin]: done this work either. That these
models actually haven't been built or at least

[Ben Bajarin]: in a way that they're confident
they will portray it to customers and or investors

[Ben Bajarin]: that they think these custom
AI parts or their accelerator bits or whatever

[Ben Bajarin]: could lead to this much more
revenue. Like in terms of, I guess their model

[Ben Bajarin]: guide, it seems like everyone
kind of believes like we do. There's something

[Ben Bajarin]: there. There will be some additional
tam in the data center, but no one's really

[Ben Bajarin]: done that work yet is basically
what I'm saying. So as an answer to the question,

[Ben Bajarin]: I'm floating the no yes, it could
be, but no, that hasn't worked been done. It's

[Ben Bajarin]: very vague speaking in vagueness.

[Jay Goldberg]: So I will start by saying, the
answering the first part of your question,

[Jay Goldberg]: which is, yes, somebody can
build this model. I don't think anybody has

[Jay Goldberg]: yet. You and I have toyed around
with this. I think our model is as advanced

[Jay Goldberg]: as pretty much anybody's. And
it's an important, important area. I tend to

[Jay Goldberg]: think, and I've been debating
this a lot with you and with other people and

[Jay Goldberg]: myself lately. I'm coming down
on the side that AI is additive to the semiconductor

[Jay Goldberg]: TAM.

[Ben Bajarin]: Mm-hmm, agree.

[Jay Goldberg]: And we're recording this on
Friday, July 21st, TSMC reported last night.

[Jay Goldberg]: And they made some comments
about seeing AI servers cannibalizing CPU servers

[Jay Goldberg]: in their, in the data they track.
But, but what they're really saying is what,

[Jay Goldberg]: what they, what they really
saying is because hyperscaler capex budgets

[Jay Goldberg]: for new data centers are fixed.
We haven't gotten into the new budgeting cycle.

[Jay Goldberg]: that's going to accommodate
this increase in

[Ben Bajarin]: Mm-hmm.

[Jay Goldberg]: AI needs. So that's a very short
time window because they say long-term AI is

[Jay Goldberg]: going to be huge in the data
center. I will say anecdotally, from what I

[Jay Goldberg]: can tell, I've heard a lot of
the big hyperscalers are accelerating, pulling

[Jay Goldberg]: forward their data center physical
plant expansion because they need AI, they

[Jay Goldberg]: need it soon, right? So I was
talking to somebody recently who owns a plot

[Jay Goldberg]: of land, it's sort of a... tier
two or tier three data center location.

[Ben Bajarin]: Hmm.

[Jay Goldberg]: Critically, it has power, it
has electricity and they're in a region where

[Jay Goldberg]: the main tier one location is
out of power. So they're on a grid, they have

[Jay Goldberg]: electricity and they just won
the lottery. They've been sitting on this piece

[Jay Goldberg]: of land for years and now suddenly
they have customers, they have all the hyperscalers,

[Jay Goldberg]: all those usual suspects plus
many more knocking on the door saying, hey,

[Jay Goldberg]: can I get in? Let's get going,
I need this, right? And... it's, you know,

[Jay Goldberg]: they have to build a plant,
they have to build the building. So it's not

[Jay Goldberg]: going to come in, you know,
next three months, but

[Ben Bajarin]: Right.

[Jay Goldberg]: it is clearly to me, additive
to whatever the hyperscalers are doing, they're

[Jay Goldberg]: adding data centers that they
know they're adding to the plan. And so it's

[Jay Goldberg]: not going to happen this quarter,
but over the, I would say over the next year,

[Jay Goldberg]: we're going to see this spike.

[Ben Bajarin]: Yeah. Nope. I agree. I'm, I'm
aligned with that. I think what questions I've

[Ben Bajarin]: heard, which again, I'm it's
fine that nobody knows this, but just in terms

[Ben Bajarin]: of, of people being aware of
the questions is really just how, how much

[Ben Bajarin]: additional capex could be thrown
to this. Like there's a reasonable amount,

[Ben Bajarin]: again, knowing that we're up
against limitations of physical space, we can't

[Ben Bajarin]: get enough wafers to meet that
demand. So yes, it will grow, but it's not

[Ben Bajarin]: going to go. It's not going to
be a hundred percent. right, growth year over

[Ben Bajarin]: year. So I think the understanding
the amount at which it can grow and then where

[Ben Bajarin]: are those pockets that might
get spent quickest I think are at least helpful

[Ben Bajarin]: if you're trying to come up with
where might these dollars go over the next

[Ben Bajarin]: year or two, knowing where the
constraints are. And I think that's a better

[Ben Bajarin]: way to kind of. look at this
question about how much, because again, I go

[Ben Bajarin]: back to this, people are throwing
these astronomical numbers out over the next

[Ben Bajarin]: five years. And I keep asking,
well, do we even have the land and the grid?

[Ben Bajarin]: Can they even build out fast
enough to meet that revenue number? But there's

[Ben Bajarin]: some reasonable amount of growth
that's going to come from AI to these data

[Ben Bajarin]: centers. And so I think if the
CapEx goes up, I don't know, I'm just making

[Ben Bajarin]: this up, but let's just say we
landed on it's 5% to 8% a year in terms of

[Ben Bajarin]: your flexible budgets. it's a
helpful way to look at where that growth can

[Ben Bajarin]: come from, being, again, additive
to a number where they were already spending.

[Jay Goldberg]: Yeah, I agree. I agree. We need
a little bit more work to put a precise number

[Jay Goldberg]: on it, but we're getting closer.

[Ben Bajarin]: Yeah.

[Jay Goldberg]: it's going to be, I think, a
meaningful amount for a lot of companies.

[Ben Bajarin]: Yes, no, I agree. Um, I will
be at an event next week with one of the top

[Ben Bajarin]: three hyperscalers and, uh, we'll
have a chance to talk to many of their customers.

[Ben Bajarin]: So we can, we can talk about
that maybe on the next episode or episode after

[Ben Bajarin]: that. When, uh, when I can share
more of what I've learned, but this is top

[Ben Bajarin]: of mine. This will be amongst
my top questions in, uh, in, in CapEx spend

[Ben Bajarin]: for AI specific stuff, so more
on that later then. Um, all right. Well, thanks,

[Ben Bajarin]: uh, everybody for listening.
Review our podcast, give us likes, tell your

[Ben Bajarin]: friends, share us on socials,
et cetera. Uh, we appreciate everybody listening.

[Jay Goldberg]: Thank you for listening everybody.