Subscribe
Share
Share
Embed
A podcast about the business and market of semiconductors
[Ben Bajarin]: Hello everyone, welcome to another
episode of The Circuit. I am Ben Beharon.
[Jay Goldberg]: Hello world, I'm Jay Goldberg.
[Ben Bajarin]: We, I had a joke a couple of
days ago, Jay and I were talking about where
[Ben Bajarin]: I would introduce myself as Jay
Goldberg, just to throw everybody off and see
[Ben Bajarin]: what the response would be. But
then we concluded that might not be a good
[Ben Bajarin]: idea.
[Jay Goldberg]: I think you've just accomplished
confusing everybody.
[Ben Bajarin]: Yes. All right. Ben Beharon is
the one talking now. There you go. All right.
[Ben Bajarin]: So it's been an interesting week
on the back of a couple of semiconductor earnings
[Ben Bajarin]: calls, ASML and TSMC. This created
a lot of different talking points, questions.
[Ben Bajarin]: Something that we're going to
dive into today is something we've talked a
[Ben Bajarin]: little bit about before. I wish
we had the wherewithal to go even deeper on
[Ben Bajarin]: this thesis, but it's something
that Jay and I are working on as well. Relative
[Ben Bajarin]: to the impact that AI has on
a number of different computing segments, not
[Ben Bajarin]: just data center, but edge, we've
done some specific, like I said, episodes on
[Ben Bajarin]: this, but Jay recently wrote
a thesis. on this at Digits to Dollars, which
[Ben Bajarin]: we can link to in the show notes.
So I'm just going to let Jay outline his thesis
[Ben Bajarin]: because I followed some of the
people that were questioning him on Twitter
[Ben Bajarin]: and seeing the discussion. So
lob the thesis out on us and then let's unpack
[Ben Bajarin]: it.
[Jay Goldberg]: So I approach this topic of
AI semis. Obviously, everyone's talking about
[Jay Goldberg]: it a lot. We've done a few episodes
on it. And I realized that we sort of addressed
[Jay Goldberg]: different parts of the elephant.
And so I wanted to take a step back and sort
[Jay Goldberg]: of look at the whole thing.
And I think from a high level strategic perspective,
[Jay Goldberg]: there are really three questions
around AI semis today. And the, the first one
[Jay Goldberg]: is will AI be additive to the
addressable market, the TAM for semiconductors,
[Jay Goldberg]: or will it just cannibalize
CPUs and other stuff and market stays the same?
[Jay Goldberg]: Second question is how will
the market for edge inference, or excuse me,
[Jay Goldberg]: for inference in general, shape
up? What's that competitive dynamic gonna look
[Jay Goldberg]: like? And the last question
is can anybody displace NVIDIA, who is clearly
[Jay Goldberg]: the market leader and the dominant
player in this space right now? And I think
[Jay Goldberg]: those are the three sort of
fundamental questions that are gonna determine
[Jay Goldberg]: some important trends for the
next decade of semiconductor. market and who
[Jay Goldberg]: the winners and losers are going
to be.
[Ben Bajarin]: Okay, I'm gonna start actually
with your third point as a starting point.
[Ben Bajarin]: So this is something I've been
thinking about and I haven't asked you this
[Ben Bajarin]: question but I've been talking
to other people. We talked a couple episodes
[Ben Bajarin]: ago when we talked about ARM
and the data center about everybody sort of,
[Ben Bajarin]: everybody who's in the know,
unanimously agreeing that much of the workload
[Ben Bajarin]: in AI is gonna move from training
to inference. And in some cases I've seen folks
[Ben Bajarin]: and investors on the sell side,
estimate this to be, you know, 77, 78% of the
[Ben Bajarin]: market today in terms of dollars
is training. Um, but that could move to a 40,
[Ben Bajarin]: 60% split 50 over the next five
years. So people are assuming that training
[Ben Bajarin]: will still be important. Let's
again, let's say it's 50%, 60% great. but a
[Ben Bajarin]: huge portion of that moving dollars
moving to inference. So the question is, if
[Ben Bajarin]: we believe that, and then perhaps
over the 10 year timeframe, even more moves
[Ben Bajarin]: to inference, maybe that's a
20% training, 70% inference, does it even matter
[Ben Bajarin]: if people compete with Nvidia
in training? Because there's so much other
[Ben Bajarin]: opportunity for inference, why
would we not be focusing more on that versus
[Ben Bajarin]: trying to compete with Nvidia?
[Jay Goldberg]: So I think there's a few parts
to that. A lot of that is around inference,
[Jay Goldberg]: which is kind of my second question.
But in terms of Nvidia's position in the market,
[Jay Goldberg]: I think we should probably break
down a little bit what dominance means. Today,
[Jay Goldberg]: they are clearly dominant in
training, close to 100% of the market. The
[Jay Goldberg]: only real
[Ben Bajarin]: Yep.
[Jay Goldberg]: competition they have there
are a few internal things at like Google, who
[Jay Goldberg]: also is using Nvidia for training
as well. So that's part of it. then the question
[Jay Goldberg]: is, can they extend their position
and training into inference?
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: And I think we're at a place
now where most people assume that AI equals
[Jay Goldberg]: GPU
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: and GPU equals Nvidia.
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: We can debate who has a better
product and is AMD's MI 300 competitive versus
[Ben Bajarin]: Right.
[Jay Goldberg]: whatever. But I think fundamentally,
it comes down to the fact that the big customers
[Jay Goldberg]: in particular are not going
to ever settle for a world in which they are
[Jay Goldberg]: completely dependent on Nvidia,
especially when Nvidia controls both the silicon
[Jay Goldberg]: and the software layer around
CUDA. That's just an unacceptable, untenable
[Jay Goldberg]: position for the customers.
And so no matter what, this current status
[Jay Goldberg]: quo has to change. Now, I'm
not bearish on Nvidia by any means. They're
[Jay Goldberg]: gonna do fine, but absolutely,
they are not going to be able to extend that
[Jay Goldberg]: 100% share into inference. There's
no way that's gonna happen for a lot of reasons.
[Jay Goldberg]: Can they get displaced out of
training? I think... I think everybody's exploring
[Jay Goldberg]: that right now. And
[Ben Bajarin]: All right.
[Jay Goldberg]: I think it's gonna be tough.
I think it's actually gonna be pretty tough.
[Jay Goldberg]: But if you start to look at
some of these alternative software frameworks
[Jay Goldberg]: that are coming out, PyTorch
and Triton are on everybody's lips. I think
[Jay Goldberg]: there are a lot of people in
marketing who just assume, okay, PyTorch is
[Jay Goldberg]: coming and it's going to displace
CUDA and suddenly we're going to have a little
[Jay Goldberg]: bit of evening out in market
share and you can run PyTorch on an AMD or
[Jay Goldberg]: whatever. And that's going to
disrupt Nvidia's hold on training. I actually,
[Jay Goldberg]: I'm not convinced. I'm not convinced.
I think
[Ben Bajarin]: I
[Jay Goldberg]: there
[Ben Bajarin]: agree.
[Jay Goldberg]: is, there's a world in which
you just run PyTorch on CUDA. Right. Uh,
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: they don't necessarily, in some
ways they compliment each other. Uh. I think.
[Jay Goldberg]: You have the software on training
is still very complex. I know a lot of companies
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: have tried to get into it. It's
just, it's going to, it's going to be, it's
[Jay Goldberg]: going to be a battle. Um, and
I think it's just for a lot of companies, it's
[Jay Goldberg]: just not going to be worth it
as, especially because like you said, inference
[Jay Goldberg]: is going to become. The majority,
I would say, I would argue the vast majority
[Jay Goldberg]: of spend over
[Ben Bajarin]: Yep.
[Jay Goldberg]: time. And
[Ben Bajarin]: Yeah.
[Jay Goldberg]: so it there's how much effort
do you want to spend trying to disrupt training
[Jay Goldberg]: when that's, you know, 10% of
your spend five years from now,
[Ben Bajarin]: Exactly.
[Jay Goldberg]: right? You're gonna be much
more focused on the inference side and training
[Jay Goldberg]: stuff is pretty complicated.
[Ben Bajarin]: agreed.
[Jay Goldberg]: Unquestionably big companies,
Google comes to mind in particular, but others
[Jay Goldberg]: as well, they're gonna find
alternatives. They're not gonna use training.
[Jay Goldberg]: They're not gonna do all their
training on Nvidia long-term probably. But
[Jay Goldberg]: I think for the enterprise,
like why, you know, if you're an enterprise,
[Jay Goldberg]: you're a bank and you wanna
run training on your data, Like that's hard
[Jay Goldberg]: enough. You have to find AI
people, developers to just do that. And then
[Jay Goldberg]: to go the extra mile and like
fight the trend and create new software frameworks
[Jay Goldberg]: and integrations. Some people
will do that, but I think a lot of people just
[Jay Goldberg]: find going with the default
Nvidia for training is the easiest solution.
[Ben Bajarin]: Yeah. So that was kind of like
roundabout where I was getting to, right? Which
[Ben Bajarin]: is, I see the allure in competing
with them today and trying to compete with
[Ben Bajarin]: them. There's at least half a
dozen companies that we know of. There's probably
[Ben Bajarin]: more attempting this. It's not
a trivial problem. PyTorch, you're right, is
[Ben Bajarin]: gaining... significant sort of
share in terms of academia and research and
[Ben Bajarin]: people who are using it. But
again, that's not the only thing here that's
[Ben Bajarin]: being used. So it really does
feel like there's a still strong reason to
[Ben Bajarin]: be bullish on Nvidia for GPU.
But what you said is exactly right. One of
[Ben Bajarin]: the big questions is, what do
they do with Grace Hopper and how do they continue
[Ben Bajarin]: to move their companion CPU part?
Um, because if you, if you unpack Jensen's
[Ben Bajarin]: vision, and this is where I,
this is the thing I think is going to be the
[Ben Bajarin]: hardest for most companies to
really kind of grasp in their, their strategy
[Ben Bajarin]: to compete with Nvidia is Jensen
labels this under accelerated computing. And
[Ben Bajarin]: in my mind, he does not limit
accelerated computing to even the GPU or the
[Ben Bajarin]: CPU there's networking parts.
And I think there's other elements of AI Asics
[Ben Bajarin]: Nvidia can make. So if they are
still. a dominant platform player and a dominant
[Ben Bajarin]: seller than of GPUs who compete,
who bring companion parts to diversify these
[Ben Bajarin]: workloads. It's actually again,
a really strong story because his focus is
[Ben Bajarin]: just, let's just accelerate computing
and any bit of Silicon that can do that. Let's
[Ben Bajarin]: just go make it Nvidia. And I
think it's hard because who else is going to
[Ben Bajarin]: have that stack? Very few people,
maybe AMD, right? Maybe Intel could have all
[Ben Bajarin]: those pieces, but Jensen's much
more focused. And that's the part I think is
[Ben Bajarin]: just gonna be really, really
tricky at the end to bite off all those pieces.
[Ben Bajarin]: Whether he's successful in those
other areas, I don't know, that's debatable.
[Ben Bajarin]: But I think his vision's very
clear and his knowledge of the problem is also
[Ben Bajarin]: very clear.
[Jay Goldberg]: Yeah, so let me break that down.
There's a couple of things there. One, today
[Jay Goldberg]: when people ask me, like, why
is Nvidia so dominant in AI? The sort of default
[Jay Goldberg]: knee-jerk answer is to say CUDA.
CUDA is a software layer that in between the
[Jay Goldberg]: operating system and the chip,
and you can get a whole bunch of optimizations
[Jay Goldberg]: to run your systems much better
because you have CUDA. And I think the advantage
[Jay Goldberg]: that CUDA has conveyed to Nvidia
I think we could reasonably argue that is slowly
[Jay Goldberg]: being diluted by all the things
I talked about before. CUDA was really important
[Jay Goldberg]: in the early days of AI for
enabling all of this. Its sustainability as
[Jay Goldberg]: a durable competitive advantage
is probably peaking. That's how I'll put it.
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: It's very strong, but it's probably
at its peak and it's going to wane. So then
[Jay Goldberg]: the next question will be, oh,
does that mean I can shorten video now? And
[Jay Goldberg]: I think the answer is no. setting
aside cyclical factors because they're going
[Jay Goldberg]: to blow up in a few quarters
because they always do but like in terms of
[Jay Goldberg]: Secular trend I think Nvidia
is still in a really good position and You're
[Jay Goldberg]: right. It's all the reasons
you stated they have this whole stack They
[Jay Goldberg]: have they have all the pieces
it will become very easy to just buy an AI
[Jay Goldberg]: system from Nvidia if you can
afford it You just get everything from them
[Jay Goldberg]: and for a lot of companies that
will be very appealing maybe not the hyperscalers
[Jay Goldberg]: but maybe them too, but certainly
for the enterprise like you just You buy everything,
[Jay Goldberg]: you buy a rack or two of Nvidia
solution. Plus on top of that you have all
[Jay Goldberg]: their software offerings. Their
models which are trained, their software frameworks
[Jay Goldberg]: which are trained for specific
industries and they have a dozen now. I think
[Jay Goldberg]: that's a really compelling vision
and I think that speaks to Jensen, like you
[Jay Goldberg]: said, really understands this
and knows where it's going. He's multiple steps
[Jay Goldberg]: ahead. The one pushback I would
give on that though is not everybody's going
[Jay Goldberg]: to want that. Because again,
it conveys a pretty high degree of lock-in,
[Jay Goldberg]: right? If AI is as important
as everybody seems to think it is. there's
[Jay Goldberg]: a risk in being so dependent
on somebody for an entire solution. And historically,
[Jay Goldberg]: those kinds of dependencies
don't last. That being said, it will appeal
[Jay Goldberg]: to enough people and Nvidia
is so far ahead on so many software fronts
[Jay Goldberg]: that I think Nvidia will do
just fine, even if CUDA goes away. I think
[Jay Goldberg]: they're in
[Ben Bajarin]: Oh,
[Jay Goldberg]: a really
[Ben Bajarin]: for sure.
[Jay Goldberg]: good position.
[Ben Bajarin]: Totally.
[Jay Goldberg]: There's gonna be a lot of friction
about people complaining about locking and
[Jay Goldberg]: all that, but still there's
a lot of strong appeal there.
[Ben Bajarin]: I totally agree. I mean, on the
merits of just the product quality themselves,
[Ben Bajarin]: right, that they're building,
it's hard to do what they're doing when it
[Ben Bajarin]: comes to these specific workloads.
So by there alone, right, I think my broader
[Ben Bajarin]: sort of just view is, one, we're
probably a couple years away, or sorry, not
[Ben Bajarin]: a couple years away, maybe closer
than that, but NVIDIA is much now more quickly
[Ben Bajarin]: going to be... a $100 billion
company in revenue sooner than we thought,
[Ben Bajarin]: thanks to this trend. That increases
their capex. That increases their ability to
[Ben Bajarin]: become a priority share at TSMC.
I think they're number four priority now or
[Ben Bajarin]: so, but increasing in that. It
just gives them so much leverage with, again,
[Ben Bajarin]: well-execution, good vision,
and a large software stack that it's hard to
[Ben Bajarin]: be displaced where they are today.
is part of my view. But that leads me to this
[Ben Bajarin]: sort of broader question you
and I have been circling around, which is,
[Ben Bajarin]: there are other areas of growth
to the data center that's not GPU. And I think
[Ben Bajarin]: we both believe GPU spend of
that is going to grow probably faster than
[Ben Bajarin]: CPU, but both are relevant. But
we've asked this question before about how
[Ben Bajarin]: much additional lift to the data
center, Tam, is AI going to bring? If it's
[Ben Bajarin]: a big number, like I've seen
numbers anywhere from, you know, 30 billion
[Ben Bajarin]: to an extra a hundred billion
over the next, you know, 10 years. So if it's
[Ben Bajarin]: a big number, I kind of feel
like if I was somebody trying to increase my
[Ben Bajarin]: strategy in the data center,
I would want to go after this green field of
[Ben Bajarin]: growth versus go for these areas
where people are more entrenched.
[Jay Goldberg]: Yeah, I think, well, I think
this gets into sort of the second question,
[Jay Goldberg]: which is inference, right? I
think going after training right now, especially
[Jay Goldberg]: for a startup is, um, I don't
[Ben Bajarin]: Yes.
[Jay Goldberg]: want to say suicidal, let's
call it challenging,
[Ben Bajarin]: challenging.
[Jay Goldberg]: very challenging, but the, the
bigger market will be inference.
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: It's going to be a very big
market and it just can't be run on GPU entirely.
[Jay Goldberg]: Right. The
[Ben Bajarin]: Yes.
[Jay Goldberg]: economics will not work out
even if,
[Ben Bajarin]: Right.
[Jay Goldberg]: you know, especially with supply
conditions the way they are today. But there
[Jay Goldberg]: are other factors too, right?
A lot of these AI, I mean, AI is just software.
[Jay Goldberg]: And so you're going to run your
normal corporate software workload, and you
[Jay Goldberg]: can have some AI functionality
in it. There are a lot of times when the software
[Jay Goldberg]: architecture is going to dictate
that means running the AI on CPU alongside
[Jay Goldberg]: the other part of the workload.
So AI is not all GPU. Some of it's going to
[Jay Goldberg]: be on CPU. A lot of companies
are going to build accelerators for this. The
[Jay Goldberg]: hyperscalers I think are, you
know, are pretty invested in, in accelerators.
[Jay Goldberg]: That's an important category
for a lot of them. And they're going to, they're
[Jay Goldberg]: going to, that's, they're going
to run inference. In the cloud on accelerators,
[Jay Goldberg]: because the numbers just work
out much better that way.
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: I haven't quantified this yet.
We've been talking about this a lot. But I
[Jay Goldberg]: think my intuition is that the
market for inference for generative AI is going
[Jay Goldberg]: to be the economics are going
to be so challenging that the only way it's
[Jay Goldberg]: going to work at is if you can
push a lot of that inference onto device, onto
[Jay Goldberg]: the edge. Because there and
the key thing there is the consumer, the customer
[Jay Goldberg]: is paying for the CapEx, they're
buying a phone, they're buying a PC that has
[Jay Goldberg]: some AI functionality and running
it on their device.
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: That's going to offload it.
Doesn't have to be run in somebody's cloud.
[Jay Goldberg]: And I think that's the only
way this really works out given the way that
[Jay Goldberg]: generative AI is taking off.
And so, yeah, that's, that's where the opportunity
[Jay Goldberg]: is in those, in those areas
around inference.
[Ben Bajarin]: And I'd add another element that
we highlighted last week that I still think
[Ben Bajarin]: is just one of the most fascinating
things to think about is the other argument
[Ben Bajarin]: that why all of this can't continue
to be done in the cloud is scarcity of resource
[Ben Bajarin]: of energy. And
[Jay Goldberg]: Right.
[Ben Bajarin]: so to your point, right, CapEx,
it gets offloaded if I'm now using another
[Ben Bajarin]: device, my edge device, my car,
my PC, my automotive, my camera that's sitting
[Ben Bajarin]: right on a stoplight, um, it's,
it's the one handling that power, right? So
[Ben Bajarin]: I'm offloading power as a part
of that as well, because I think you could
[Ben Bajarin]: make the strong argument like
we have that we just don't have the grid for
[Ben Bajarin]: all of this to be run, you know,
in, in the cloud, especially amongst the top
[Ben Bajarin]: three hyperscalers, let alone
people include in, in broad terms, Apple as
[Ben Bajarin]: a top four U S hyperscaler, you
know, they, it's, you just can't run all this
[Ben Bajarin]: in the cloud. So I think that's
an important reason, which again, goes back
[Ben Bajarin]: to your other point, right? A
thesis of, of on device, which we did a whole
[Ben Bajarin]: segment on. And I think now we've
seen a handful of demos and a bit of extra
[Ben Bajarin]: conversations on device. It's
definitely not there today. But at some point
[Ben Bajarin]: in time, you'll be able to do
a whole lot more of this on device, and it
[Ben Bajarin]: will feel not that far off from
the things you've experienced in cloud-centric
[Ben Bajarin]: experiences.
[Jay Goldberg]: Yeah. I think, I think that's,
that's how this is going to work out. And I
[Jay Goldberg]: think that it's just, it's just
too cumbersome and the, the workloads are too
[Jay Goldberg]: big to really be run any other
way. It has to be some significant portion
[Jay Goldberg]: of offload from the cloud.
[Ben Bajarin]: So the question here that sort
of plagues me is, it feels like it's gonna
[Ben Bajarin]: be very hard to run this hybrid
AI environment that a lot of people are talking
[Ben Bajarin]: about, meaning that I provide
a service, I'm a cloud provider, and I realize
[Ben Bajarin]: that I've got tons of capabilities
here to do what I wanna do in terms of this
[Ben Bajarin]: service, what people are paying
for. But yes, I want to offload that to the
[Ben Bajarin]: device, but I need to know how
much that device can be offloaded to that device.
[Ben Bajarin]: And that's not going to be an
all things equal scenario, right? Devices that
[Ben Bajarin]: are five years old are going
to need a whole lot more cloud help than devices
[Ben Bajarin]: that are one year old. But people
talk about this hybrid environment. I want
[Ben Bajarin]: to have some of it in the cloud
and some of it on device. I feel like that's
[Ben Bajarin]: a really tricky architecture.
to talk about because again, it feels like
[Ben Bajarin]: the service needs to know, well,
I can't offload that or it will be a terrible
[Ben Bajarin]: experience or it's a capable
device, I can't offload 80%. I just don't know
[Ben Bajarin]: how this gets worked out but
that's kind of how people talk about this hybrid
[Ben Bajarin]: cloud on device today. Could
change in three years, but it feels very tricky,
[Ben Bajarin]: very complex to me to do that.
[Jay Goldberg]: I think you just described Android.
[Ben Bajarin]: Well, that's a whole different
issue in my brain,
[Jay Goldberg]: Yeah,
[Ben Bajarin]: but you're 100% right.
[Jay Goldberg]: I won't start my Android rant,
but I think when you talk about the edge, we're
[Jay Goldberg]: really talking about three things.
We're talking about PCs, iPhone, and Android.
[Jay Goldberg]: Yes, there's cameras, those
will come, and then there's automotive someday
[Jay Goldberg]: further out. But for the moment,
what we're really concerned about is laptop
[Jay Goldberg]: and phone. And at their event,
what? Two months ago, AMD actually... started
[Jay Goldberg]: talking about that, including
some, some neural processing blocks in their
[Jay Goldberg]: seat, in their laptop CPUs.
Apple's been doing it for a while. Obviously
[Jay Goldberg]: they have some in the phone
as well.
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: Right. That's, that's really
where this edge inference is going to take
[Jay Goldberg]: place is in those kinds of devices.
And so what's going to, what's it going to
[Jay Goldberg]: take to accomplish that is,
uh, Microsoft has to get windows to the point
[Jay Goldberg]: where it can do it.
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: Apple has to, it has to do likewise
for both Mac OS and iOS. And I would say, uh,
[Jay Goldberg]: Microsoft is clearly fully invested
in it and they're, they love generative AI.
[Jay Goldberg]: And I think
[Ben Bajarin]: Yeah.
[Jay Goldberg]: I have to imagine that there
are teams inside Microsoft working pretty heavily
[Jay Goldberg]: to bring that. transform or
to support into Windows sooner rather than
[Jay Goldberg]: later. Apple already done it
for the Mac. And so once those frameworks get
[Jay Goldberg]: set up, I think it actually
can happen pretty quickly. We will have this
[Jay Goldberg]: problem with Android where there
are going to be a lot of devices that can't
[Jay Goldberg]: run it for years.
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: And I think that's just one
more thing that has to add that to the list
[Jay Goldberg]: of problems that Google has
with Android. Because when Apple thinks it's,
[Jay Goldberg]: you know, consumers really want
generative AI support on the iPhone, they'll
[Jay Goldberg]: launch it, you know, if it's
not in this iPhone, it'll be, you know, whenever
[Jay Goldberg]: six months after they think
consumers are ready for it, right?
[Ben Bajarin]: Yeah. Well, and
[Jay Goldberg]: So
[Ben Bajarin]: it's.
[Jay Goldberg]: I would be surprised if they
don't talk about something along these lines
[Jay Goldberg]: in September with the new iPhone.
[Ben Bajarin]: Sure. So there was a report,
I'm sure everybody who listens to this saw
[Ben Bajarin]: that Apple, Mark Gurman wrote
it at Bloomberg and just saying that they are
[Ben Bajarin]: working on their own GPT model.
It sounds like it's a pretty large model in
[Ben Bajarin]: terms of overall size, like definitely
large enough that whatever they're. building
[Ben Bajarin]: data set wise is not going to
run on device at that size. I mean, roughly
[Ben Bajarin]: the stuff we've tried today seems
to be a successful if it's in the 10 to 15
[Ben Bajarin]: billion parameter range. Anything
north of that is crashing devices and running
[Ben Bajarin]: exceptionally slow. But that's
today, right? That's not where we'll be in
[Ben Bajarin]: two to three years. But it shows
kind of what's possible on device versus the
[Ben Bajarin]: size of cloud. But Apple, 100%.
right, is going to want to do this. And arguably,
[Ben Bajarin]: like I said, they are a top four
hyperscaler. They can create that cloud to
[Ben Bajarin]: device infrastructure as good
as anybody, if they want, um, to handle their,
[Ben Bajarin]: their device fragmentation, which
as you point out is not as, not nearly as,
[Ben Bajarin]: as tricky as Google's.
[Jay Goldberg]: Yeah, Apple technically is capable
of doing it today. There's no question. If
[Jay Goldberg]: they wanted to get generative
AI working on the iPhone, it could happen today.
[Jay Goldberg]: I think the question for Apple
is, or the question that Apple is asking is
[Jay Goldberg]: more, what are we going to use
this for? Apple doesn't like to add features
[Jay Goldberg]: that consumers don't care about.
[Ben Bajarin]: Right.
[Jay Goldberg]: How much do consumers really
care about generative AI? How important is
[Jay Goldberg]: it to the user experience, the
human experience, excuse me,
[Ben Bajarin]: Right.
[Jay Goldberg]: on an iPhone? And there are
a lot of people who think that Apple is behind
[Jay Goldberg]: in AI as well, because they
don't have generative AI today. I think they're,
[Jay Goldberg]: they're trying to figure out
as am I, and a lot of people, like what is
[Jay Goldberg]: actually really useful for.
And I actually had a Twitter debate with somebody
[Jay Goldberg]: who was saying, you know, AI
is going to be really important, everyone's
[Jay Goldberg]: going to want it. And I said,
well, how much more would you pay for a phone
[Jay Goldberg]: that does
[Ben Bajarin]: Sure.
[Jay Goldberg]: chat GPT or stable diffusion
on the phone? And he says, well, I already
[Jay Goldberg]: paid chat GPT $20 a month. And
I'm like, yeah, I understand that. But that's
[Jay Goldberg]: not the question. You like generative
AI, you're willing to pay for it. How much
[Jay Goldberg]: though are you willing to pay
for it to work on your phone in airplane mode?
[Jay Goldberg]: And I don't think anybody knows
the answer to that question yet.
[Ben Bajarin]: So there's another though, part
of this that feeds into this that I think is
[Ben Bajarin]: fascinating. And this again speaks
back to companies who are prioritizing on device,
[Ben Bajarin]: those who are not be playing
both sides of cloud to device in terms of their
[Ben Bajarin]: roadmap, is you exactly rightly
point out, I don't think anybody is gonna pay
[Ben Bajarin]: more for these things. I think
it's gonna have to just be an evolution of
[Ben Bajarin]: the silicon's... roadmap to increase
those features. But if I'm a silicon designer
[Ben Bajarin]: today, so let's just say this
is MediaTek, Qualcomm, and Apple, to some degree
[Ben Bajarin]: Intel and AMD on PCs, you do
have to sort of make some decision about how
[Ben Bajarin]: much transistor budget you're
gonna throw to something like an NPU, because
[Ben Bajarin]: it's a relevant decision, right?
If I believe that I need that to compete, I
[Ben Bajarin]: gotta take that from something
else, right? I gotta take it from my GPU blocks.
[Ben Bajarin]: my CPU blocks, right, something
else, right? So they need to make calls on
[Ben Bajarin]: how important that is by how
much diary are they're gonna commit to these
[Ben Bajarin]: things going forward. And that's
a fascinating trade-off, right, that I think
[Ben Bajarin]: people are stuck with these next
two years, when again, you and I can argue
[Ben Bajarin]: all day, people will 100% be
willing to pay for those silicon bits in the
[Ben Bajarin]: data center more. They're not
gonna pay for them on device. And so that's
[Ben Bajarin]: just a fascinating dynamic of
maybe how fast the capabilities develop at
[Ben Bajarin]: the edge, when again, you've
got to make these trade-offs with your transistor
[Ben Bajarin]: budget.
[Jay Goldberg]: Yeah, I think that's a really
good way of framing it. And I think that sort
[Jay Goldberg]: of throws light on what I was
saying about Apple is, Apple at this point
[Jay Goldberg]: doesn't seem to see the need
to throw those blocks, those resources at it.
[Jay Goldberg]: They just don't see it as necessary.
[Jay Goldberg]: And I don't think anyone knows
when that will change.
[Ben Bajarin]: Well, we'll see. I mean, I think
we'll see with two fundamental things in my
[Ben Bajarin]: opinion, that'll happen this
fall, right? Qualcomm is going to unveil their
[Ben Bajarin]: new PC chip, PC chip architecture.
My hunch is that's going to be loaded with
[Ben Bajarin]: tops. Um, Apple has historically
only bumped their tops up. If I recall two
[Ben Bajarin]: to three, you know, uh, I think
they went from 10 to 15. Right. So maybe five
[Ben Bajarin]: or so. So, so what they throw
at tops in next generation products will be
[Ben Bajarin]: telling. Right. If it's goes
from 15 to 30, that's a pretty big jump. Right.
[Ben Bajarin]: But if it goes from 15 to eight
to 18, you know, you'll, you'll again, you'll
[Ben Bajarin]: just sort of see how they're
prioritizing. So, um, that's an interesting
[Ben Bajarin]: bit, but, but you're right. I
think that's, that's really the call they've
[Ben Bajarin]: got to make is, is how much C
block they throw to this stuff and it, and
[Ben Bajarin]: could it be premature, you know?
[Jay Goldberg]: Yeah, Apple, I think, typically
tends to be a little bit conservative in jumping
[Jay Goldberg]: on the latest feature, right?
The original iPhone was a 2G phone deep into
[Jay Goldberg]: the 3G era. That's the classic
example. And they've had AI on their phone
[Jay Goldberg]: longer than anybody, right?
They don't call it that. It's a neural engine.
[Jay Goldberg]: And it's there for a very specific
purpose to help with image processing. I don't,
[Jay Goldberg]: yeah, it would be interesting
to see it to what degree they change it, right?
[Jay Goldberg]: Or do they just sort of keep
going at their steady incremental pace?
[Ben Bajarin]: Yeah. So I want to end on just
circling back to the question at hand about
[Ben Bajarin]: is there even a way to sort of
put a model around the additional lift of dollars
[Ben Bajarin]: that's coming to AI? I think
we agree we can't do it on device. But to that
[Ben Bajarin]: point, in a number of earnings
calls, as well as some commentary of some of
[Ben Bajarin]: the people who are doing this,
it appears that it The answer is others haven't
[Ben Bajarin]: done this work either. That these
models actually haven't been built or at least
[Ben Bajarin]: in a way that they're confident
they will portray it to customers and or investors
[Ben Bajarin]: that they think these custom
AI parts or their accelerator bits or whatever
[Ben Bajarin]: could lead to this much more
revenue. Like in terms of, I guess their model
[Ben Bajarin]: guide, it seems like everyone
kind of believes like we do. There's something
[Ben Bajarin]: there. There will be some additional
tam in the data center, but no one's really
[Ben Bajarin]: done that work yet is basically
what I'm saying. So as an answer to the question,
[Ben Bajarin]: I'm floating the no yes, it could
be, but no, that hasn't worked been done. It's
[Ben Bajarin]: very vague speaking in vagueness.
[Jay Goldberg]: So I will start by saying, the
answering the first part of your question,
[Jay Goldberg]: which is, yes, somebody can
build this model. I don't think anybody has
[Jay Goldberg]: yet. You and I have toyed around
with this. I think our model is as advanced
[Jay Goldberg]: as pretty much anybody's. And
it's an important, important area. I tend to
[Jay Goldberg]: think, and I've been debating
this a lot with you and with other people and
[Jay Goldberg]: myself lately. I'm coming down
on the side that AI is additive to the semiconductor
[Jay Goldberg]: TAM.
[Ben Bajarin]: Mm-hmm, agree.
[Jay Goldberg]: And we're recording this on
Friday, July 21st, TSMC reported last night.
[Jay Goldberg]: And they made some comments
about seeing AI servers cannibalizing CPU servers
[Jay Goldberg]: in their, in the data they track.
But, but what they're really saying is what,
[Jay Goldberg]: what they, what they really
saying is because hyperscaler capex budgets
[Jay Goldberg]: for new data centers are fixed.
We haven't gotten into the new budgeting cycle.
[Jay Goldberg]: that's going to accommodate
this increase in
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: AI needs. So that's a very short
time window because they say long-term AI is
[Jay Goldberg]: going to be huge in the data
center. I will say anecdotally, from what I
[Jay Goldberg]: can tell, I've heard a lot of
the big hyperscalers are accelerating, pulling
[Jay Goldberg]: forward their data center physical
plant expansion because they need AI, they
[Jay Goldberg]: need it soon, right? So I was
talking to somebody recently who owns a plot
[Jay Goldberg]: of land, it's sort of a... tier
two or tier three data center location.
[Ben Bajarin]: Hmm.
[Jay Goldberg]: Critically, it has power, it
has electricity and they're in a region where
[Jay Goldberg]: the main tier one location is
out of power. So they're on a grid, they have
[Jay Goldberg]: electricity and they just won
the lottery. They've been sitting on this piece
[Jay Goldberg]: of land for years and now suddenly
they have customers, they have all the hyperscalers,
[Jay Goldberg]: all those usual suspects plus
many more knocking on the door saying, hey,
[Jay Goldberg]: can I get in? Let's get going,
I need this, right? And... it's, you know,
[Jay Goldberg]: they have to build a plant,
they have to build the building. So it's not
[Jay Goldberg]: going to come in, you know,
next three months, but
[Ben Bajarin]: Right.
[Jay Goldberg]: it is clearly to me, additive
to whatever the hyperscalers are doing, they're
[Jay Goldberg]: adding data centers that they
know they're adding to the plan. And so it's
[Jay Goldberg]: not going to happen this quarter,
but over the, I would say over the next year,
[Jay Goldberg]: we're going to see this spike.
[Ben Bajarin]: Yeah. Nope. I agree. I'm, I'm
aligned with that. I think what questions I've
[Ben Bajarin]: heard, which again, I'm it's
fine that nobody knows this, but just in terms
[Ben Bajarin]: of, of people being aware of
the questions is really just how, how much
[Ben Bajarin]: additional capex could be thrown
to this. Like there's a reasonable amount,
[Ben Bajarin]: again, knowing that we're up
against limitations of physical space, we can't
[Ben Bajarin]: get enough wafers to meet that
demand. So yes, it will grow, but it's not
[Ben Bajarin]: going to go. It's not going to
be a hundred percent. right, growth year over
[Ben Bajarin]: year. So I think the understanding
the amount at which it can grow and then where
[Ben Bajarin]: are those pockets that might
get spent quickest I think are at least helpful
[Ben Bajarin]: if you're trying to come up with
where might these dollars go over the next
[Ben Bajarin]: year or two, knowing where the
constraints are. And I think that's a better
[Ben Bajarin]: way to kind of. look at this
question about how much, because again, I go
[Ben Bajarin]: back to this, people are throwing
these astronomical numbers out over the next
[Ben Bajarin]: five years. And I keep asking,
well, do we even have the land and the grid?
[Ben Bajarin]: Can they even build out fast
enough to meet that revenue number? But there's
[Ben Bajarin]: some reasonable amount of growth
that's going to come from AI to these data
[Ben Bajarin]: centers. And so I think if the
CapEx goes up, I don't know, I'm just making
[Ben Bajarin]: this up, but let's just say we
landed on it's 5% to 8% a year in terms of
[Ben Bajarin]: your flexible budgets. it's a
helpful way to look at where that growth can
[Ben Bajarin]: come from, being, again, additive
to a number where they were already spending.
[Jay Goldberg]: Yeah, I agree. I agree. We need
a little bit more work to put a precise number
[Jay Goldberg]: on it, but we're getting closer.
And
[Ben Bajarin]: Yeah.
[Jay Goldberg]: it's going to be, I think, a
meaningful amount for a lot of companies.
[Ben Bajarin]: Yes, no, I agree. Um, I will
be at an event next week with one of the top
[Ben Bajarin]: three hyperscalers and, uh, we'll
have a chance to talk to many of their customers.
[Ben Bajarin]: So we can, we can talk about
that maybe on the next episode or episode after
[Ben Bajarin]: that. When, uh, when I can share
more of what I've learned, but this is top
[Ben Bajarin]: of mine. This will be amongst
my top questions in, uh, in, in CapEx spend
[Ben Bajarin]: for AI specific stuff, so more
on that later then. Um, all right. Well, thanks,
[Ben Bajarin]: uh, everybody for listening.
Review our podcast, give us likes, tell your
[Ben Bajarin]: friends, share us on socials,
et cetera. Uh, we appreciate everybody listening.
[Jay Goldberg]: Thank you for listening everybody.