[Ben Bajarin]: Hello everyone, welcome to another episode of The Circuit. I am Ben Beharon.
[Jay Goldberg]: Hello world, I'm Jay Goldberg.
[Ben Bajarin]: We, I had a joke a couple of days ago, Jay and I were talking about where
[Ben Bajarin]: I would introduce myself as Jay Goldberg, just to throw everybody off and see
[Ben Bajarin]: what the response would be. But then we concluded that might not be a good
[Ben Bajarin]: idea.
[Jay Goldberg]: I think you've just accomplished confusing everybody.
[Ben Bajarin]: Yes. All right. Ben Beharon is the one talking now. There you go. All right.
[Ben Bajarin]: So it's been an interesting week on the back of a couple of semiconductor earnings
[Ben Bajarin]: calls, ASML and TSMC. This created a lot of different talking points, questions.
[Ben Bajarin]: Something that we're going to dive into today is something we've talked a
[Ben Bajarin]: little bit about before. I wish we had the wherewithal to go even deeper on
[Ben Bajarin]: this thesis, but it's something that Jay and I are working on as well. Relative
[Ben Bajarin]: to the impact that AI has on a number of different computing segments, not
[Ben Bajarin]: just data center, but edge, we've done some specific, like I said, episodes on
[Ben Bajarin]: this, but Jay recently wrote a thesis. on this at Digits to Dollars, which
[Ben Bajarin]: we can link to in the show notes. So I'm just going to let Jay outline his thesis
[Ben Bajarin]: because I followed some of the people that were questioning him on Twitter
[Ben Bajarin]: and seeing the discussion. So lob the thesis out on us and then let's unpack
[Ben Bajarin]: it.
[Jay Goldberg]: So I approach this topic of AI semis. Obviously, everyone's talking about
[Jay Goldberg]: it a lot. We've done a few episodes on it. And I realized that we sort of addressed
[Jay Goldberg]: different parts of the elephant. And so I wanted to take a step back and sort
[Jay Goldberg]: of look at the whole thing. And I think from a high level strategic perspective,
[Jay Goldberg]: there are really three questions around AI semis today. And the, the first one
[Jay Goldberg]: is will AI be additive to the addressable market, the TAM for semiconductors,
[Jay Goldberg]: or will it just cannibalize CPUs and other stuff and market stays the same?
[Jay Goldberg]: Second question is how will the market for edge inference, or excuse me,
[Jay Goldberg]: for inference in general, shape up? What's that competitive dynamic gonna look
[Jay Goldberg]: like? And the last question is can anybody displace NVIDIA, who is clearly
[Jay Goldberg]: the market leader and the dominant player in this space right now? And I think
[Jay Goldberg]: those are the three sort of fundamental questions that are gonna determine
[Jay Goldberg]: some important trends for the next decade of semiconductor. market and who
[Jay Goldberg]: the winners and losers are going to be.
[Ben Bajarin]: Okay, I'm gonna start actually with your third point as a starting point.
[Ben Bajarin]: So this is something I've been thinking about and I haven't asked you this
[Ben Bajarin]: question but I've been talking to other people. We talked a couple episodes
[Ben Bajarin]: ago when we talked about ARM and the data center about everybody sort of,
[Ben Bajarin]: everybody who's in the know, unanimously agreeing that much of the workload
[Ben Bajarin]: in AI is gonna move from training to inference. And in some cases I've seen folks
[Ben Bajarin]: and investors on the sell side, estimate this to be, you know, 77, 78% of the
[Ben Bajarin]: market today in terms of dollars is training. Um, but that could move to a 40,
[Ben Bajarin]: 60% split 50 over the next five years. So people are assuming that training
[Ben Bajarin]: will still be important. Let's again, let's say it's 50%, 60% great. but a
[Ben Bajarin]: huge portion of that moving dollars moving to inference. So the question is, if
[Ben Bajarin]: we believe that, and then perhaps over the 10 year timeframe, even more moves
[Ben Bajarin]: to inference, maybe that's a 20% training, 70% inference, does it even matter
[Ben Bajarin]: if people compete with Nvidia in training? Because there's so much other
[Ben Bajarin]: opportunity for inference, why would we not be focusing more on that versus
[Ben Bajarin]: trying to compete with Nvidia?
[Jay Goldberg]: So I think there's a few parts to that. A lot of that is around inference,
[Jay Goldberg]: which is kind of my second question. But in terms of Nvidia's position in the market,
[Jay Goldberg]: I think we should probably break down a little bit what dominance means. Today,
[Jay Goldberg]: they are clearly dominant in training, close to 100% of the market. The
[Jay Goldberg]: only real
[Ben Bajarin]: Yep.
[Jay Goldberg]: competition they have there are a few internal things at like Google, who
[Jay Goldberg]: also is using Nvidia for training as well. So that's part of it. then the question
[Jay Goldberg]: is, can they extend their position and training into inference?
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: And I think we're at a place now where most people assume that AI equals
[Jay Goldberg]: GPU
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: and GPU equals Nvidia.
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: We can debate who has a better product and is AMD's MI 300 competitive versus
[Ben Bajarin]: Right.
[Jay Goldberg]: whatever. But I think fundamentally, it comes down to the fact that the big customers
[Jay Goldberg]: in particular are not going to ever settle for a world in which they are
[Jay Goldberg]: completely dependent on Nvidia, especially when Nvidia controls both the silicon
[Jay Goldberg]: and the software layer around CUDA. That's just an unacceptable, untenable
[Jay Goldberg]: position for the customers. And so no matter what, this current status
[Jay Goldberg]: quo has to change. Now, I'm not bearish on Nvidia by any means. They're
[Jay Goldberg]: gonna do fine, but absolutely, they are not going to be able to extend that
[Jay Goldberg]: 100% share into inference. There's no way that's gonna happen for a lot of reasons.
[Jay Goldberg]: Can they get displaced out of training? I think... I think everybody's exploring
[Jay Goldberg]: that right now. And
[Ben Bajarin]: All right.
[Jay Goldberg]: I think it's gonna be tough. I think it's actually gonna be pretty tough.
[Jay Goldberg]: But if you start to look at some of these alternative software frameworks
[Jay Goldberg]: that are coming out, PyTorch and Triton are on everybody's lips. I think
[Jay Goldberg]: there are a lot of people in marketing who just assume, okay, PyTorch is
[Jay Goldberg]: coming and it's going to displace CUDA and suddenly we're going to have a little
[Jay Goldberg]: bit of evening out in market share and you can run PyTorch on an AMD or
[Jay Goldberg]: whatever. And that's going to disrupt Nvidia's hold on training. I actually,
[Jay Goldberg]: I'm not convinced. I'm not convinced. I think
[Ben Bajarin]: I
[Jay Goldberg]: there
[Ben Bajarin]: agree.
[Jay Goldberg]: is, there's a world in which you just run PyTorch on CUDA. Right. Uh,
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: they don't necessarily, in some ways they compliment each other. Uh. I think.
[Jay Goldberg]: You have the software on training is still very complex. I know a lot of companies
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: have tried to get into it. It's just, it's going to, it's going to be, it's
[Jay Goldberg]: going to be a battle. Um, and I think it's just for a lot of companies, it's
[Jay Goldberg]: just not going to be worth it as, especially because like you said, inference
[Jay Goldberg]: is going to become. The majority, I would say, I would argue the vast majority
[Jay Goldberg]: of spend over
[Ben Bajarin]: Yep.
[Jay Goldberg]: time. And
[Ben Bajarin]: Yeah.
[Jay Goldberg]: so it there's how much effort do you want to spend trying to disrupt training
[Jay Goldberg]: when that's, you know, 10% of your spend five years from now,
[Ben Bajarin]: Exactly.
[Jay Goldberg]: right? You're gonna be much more focused on the inference side and training
[Jay Goldberg]: stuff is pretty complicated.
[Ben Bajarin]: agreed.
[Jay Goldberg]: Unquestionably big companies, Google comes to mind in particular, but others
[Jay Goldberg]: as well, they're gonna find alternatives. They're not gonna use training.
[Jay Goldberg]: They're not gonna do all their training on Nvidia long-term probably. But
[Jay Goldberg]: I think for the enterprise, like why, you know, if you're an enterprise,
[Jay Goldberg]: you're a bank and you wanna run training on your data, Like that's hard
[Jay Goldberg]: enough. You have to find AI people, developers to just do that. And then
[Jay Goldberg]: to go the extra mile and like fight the trend and create new software frameworks
[Jay Goldberg]: and integrations. Some people will do that, but I think a lot of people just
[Jay Goldberg]: find going with the default Nvidia for training is the easiest solution.
[Ben Bajarin]: Yeah. So that was kind of like roundabout where I was getting to, right? Which
[Ben Bajarin]: is, I see the allure in competing with them today and trying to compete with
[Ben Bajarin]: them. There's at least half a dozen companies that we know of. There's probably
[Ben Bajarin]: more attempting this. It's not a trivial problem. PyTorch, you're right, is
[Ben Bajarin]: gaining... significant sort of share in terms of academia and research and
[Ben Bajarin]: people who are using it. But again, that's not the only thing here that's
[Ben Bajarin]: being used. So it really does feel like there's a still strong reason to
[Ben Bajarin]: be bullish on Nvidia for GPU. But what you said is exactly right. One of
[Ben Bajarin]: the big questions is, what do they do with Grace Hopper and how do they continue
[Ben Bajarin]: to move their companion CPU part? Um, because if you, if you unpack Jensen's
[Ben Bajarin]: vision, and this is where I, this is the thing I think is going to be the
[Ben Bajarin]: hardest for most companies to really kind of grasp in their, their strategy
[Ben Bajarin]: to compete with Nvidia is Jensen labels this under accelerated computing. And
[Ben Bajarin]: in my mind, he does not limit accelerated computing to even the GPU or the
[Ben Bajarin]: CPU there's networking parts. And I think there's other elements of AI Asics
[Ben Bajarin]: Nvidia can make. So if they are still. a dominant platform player and a dominant
[Ben Bajarin]: seller than of GPUs who compete, who bring companion parts to diversify these
[Ben Bajarin]: workloads. It's actually again, a really strong story because his focus is
[Ben Bajarin]: just, let's just accelerate computing and any bit of Silicon that can do that. Let's
[Ben Bajarin]: just go make it Nvidia. And I think it's hard because who else is going to
[Ben Bajarin]: have that stack? Very few people, maybe AMD, right? Maybe Intel could have all
[Ben Bajarin]: those pieces, but Jensen's much more focused. And that's the part I think is
[Ben Bajarin]: just gonna be really, really tricky at the end to bite off all those pieces.
[Ben Bajarin]: Whether he's successful in those other areas, I don't know, that's debatable.
[Ben Bajarin]: But I think his vision's very clear and his knowledge of the problem is also
[Ben Bajarin]: very clear.
[Jay Goldberg]: Yeah, so let me break that down. There's a couple of things there. One, today
[Jay Goldberg]: when people ask me, like, why is Nvidia so dominant in AI? The sort of default
[Jay Goldberg]: knee-jerk answer is to say CUDA. CUDA is a software layer that in between the
[Jay Goldberg]: operating system and the chip, and you can get a whole bunch of optimizations
[Jay Goldberg]: to run your systems much better because you have CUDA. And I think the advantage
[Jay Goldberg]: that CUDA has conveyed to Nvidia I think we could reasonably argue that is slowly
[Jay Goldberg]: being diluted by all the things I talked about before. CUDA was really important
[Jay Goldberg]: in the early days of AI for enabling all of this. Its sustainability as
[Jay Goldberg]: a durable competitive advantage is probably peaking. That's how I'll put it.
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: It's very strong, but it's probably at its peak and it's going to wane. So then
[Jay Goldberg]: the next question will be, oh, does that mean I can shorten video now? And
[Jay Goldberg]: I think the answer is no. setting aside cyclical factors because they're going
[Jay Goldberg]: to blow up in a few quarters because they always do but like in terms of
[Jay Goldberg]: Secular trend I think Nvidia is still in a really good position and You're
[Jay Goldberg]: right. It's all the reasons you stated they have this whole stack They
[Jay Goldberg]: have they have all the pieces it will become very easy to just buy an AI
[Jay Goldberg]: system from Nvidia if you can afford it You just get everything from them
[Jay Goldberg]: and for a lot of companies that will be very appealing maybe not the hyperscalers
[Jay Goldberg]: but maybe them too, but certainly for the enterprise like you just You buy everything,
[Jay Goldberg]: you buy a rack or two of Nvidia solution. Plus on top of that you have all
[Jay Goldberg]: their software offerings. Their models which are trained, their software frameworks
[Jay Goldberg]: which are trained for specific industries and they have a dozen now. I think
[Jay Goldberg]: that's a really compelling vision and I think that speaks to Jensen, like you
[Jay Goldberg]: said, really understands this and knows where it's going. He's multiple steps
[Jay Goldberg]: ahead. The one pushback I would give on that though is not everybody's going
[Jay Goldberg]: to want that. Because again, it conveys a pretty high degree of lock-in,
[Jay Goldberg]: right? If AI is as important as everybody seems to think it is. there's
[Jay Goldberg]: a risk in being so dependent on somebody for an entire solution. And historically,
[Jay Goldberg]: those kinds of dependencies don't last. That being said, it will appeal
[Jay Goldberg]: to enough people and Nvidia is so far ahead on so many software fronts
[Jay Goldberg]: that I think Nvidia will do just fine, even if CUDA goes away. I think
[Jay Goldberg]: they're in
[Ben Bajarin]: Oh,
[Jay Goldberg]: a really
[Ben Bajarin]: for sure.
[Jay Goldberg]: good position.
[Ben Bajarin]: Totally.
[Jay Goldberg]: There's gonna be a lot of friction about people complaining about locking and
[Jay Goldberg]: all that, but still there's a lot of strong appeal there.
[Ben Bajarin]: I totally agree. I mean, on the merits of just the product quality themselves,
[Ben Bajarin]: right, that they're building, it's hard to do what they're doing when it
[Ben Bajarin]: comes to these specific workloads. So by there alone, right, I think my broader
[Ben Bajarin]: sort of just view is, one, we're probably a couple years away, or sorry, not
[Ben Bajarin]: a couple years away, maybe closer than that, but NVIDIA is much now more quickly
[Ben Bajarin]: going to be... a $100 billion company in revenue sooner than we thought,
[Ben Bajarin]: thanks to this trend. That increases their capex. That increases their ability to
[Ben Bajarin]: become a priority share at TSMC. I think they're number four priority now or
[Ben Bajarin]: so, but increasing in that. It just gives them so much leverage with, again,
[Ben Bajarin]: well-execution, good vision, and a large software stack that it's hard to
[Ben Bajarin]: be displaced where they are today. is part of my view. But that leads me to this
[Ben Bajarin]: sort of broader question you and I have been circling around, which is,
[Ben Bajarin]: there are other areas of growth to the data center that's not GPU. And I think
[Ben Bajarin]: we both believe GPU spend of that is going to grow probably faster than
[Ben Bajarin]: CPU, but both are relevant. But we've asked this question before about how
[Ben Bajarin]: much additional lift to the data center, Tam, is AI going to bring? If it's
[Ben Bajarin]: a big number, like I've seen numbers anywhere from, you know, 30 billion
[Ben Bajarin]: to an extra a hundred billion over the next, you know, 10 years. So if it's
[Ben Bajarin]: a big number, I kind of feel like if I was somebody trying to increase my
[Ben Bajarin]: strategy in the data center, I would want to go after this green field of
[Ben Bajarin]: growth versus go for these areas where people are more entrenched.
[Jay Goldberg]: Yeah, I think, well, I think this gets into sort of the second question,
[Jay Goldberg]: which is inference, right? I think going after training right now, especially
[Jay Goldberg]: for a startup is, um, I don't
[Ben Bajarin]: Yes.
[Jay Goldberg]: want to say suicidal, let's call it challenging,
[Ben Bajarin]: challenging.
[Jay Goldberg]: very challenging, but the, the bigger market will be inference.
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: It's going to be a very big market and it just can't be run on GPU entirely.
[Jay Goldberg]: Right. The
[Ben Bajarin]: Yes.
[Jay Goldberg]: economics will not work out even if,
[Ben Bajarin]: Right.
[Jay Goldberg]: you know, especially with supply conditions the way they are today. But there
[Jay Goldberg]: are other factors too, right? A lot of these AI, I mean, AI is just software.
[Jay Goldberg]: And so you're going to run your normal corporate software workload, and you
[Jay Goldberg]: can have some AI functionality in it. There are a lot of times when the software
[Jay Goldberg]: architecture is going to dictate that means running the AI on CPU alongside
[Jay Goldberg]: the other part of the workload. So AI is not all GPU. Some of it's going to
[Jay Goldberg]: be on CPU. A lot of companies are going to build accelerators for this. The
[Jay Goldberg]: hyperscalers I think are, you know, are pretty invested in, in accelerators.
[Jay Goldberg]: That's an important category for a lot of them. And they're going to, they're
[Jay Goldberg]: going to, that's, they're going to run inference. In the cloud on accelerators,
[Jay Goldberg]: because the numbers just work out much better that way.
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: I haven't quantified this yet. We've been talking about this a lot. But I
[Jay Goldberg]: think my intuition is that the market for inference for generative AI is going
[Jay Goldberg]: to be the economics are going to be so challenging that the only way it's
[Jay Goldberg]: going to work at is if you can push a lot of that inference onto device, onto
[Jay Goldberg]: the edge. Because there and the key thing there is the consumer, the customer
[Jay Goldberg]: is paying for the CapEx, they're buying a phone, they're buying a PC that has
[Jay Goldberg]: some AI functionality and running it on their device.
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: That's going to offload it. Doesn't have to be run in somebody's cloud.
[Jay Goldberg]: And I think that's the only way this really works out given the way that
[Jay Goldberg]: generative AI is taking off. And so, yeah, that's, that's where the opportunity
[Jay Goldberg]: is in those, in those areas around inference.
[Ben Bajarin]: And I'd add another element that we highlighted last week that I still think
[Ben Bajarin]: is just one of the most fascinating things to think about is the other argument
[Ben Bajarin]: that why all of this can't continue to be done in the cloud is scarcity of resource
[Ben Bajarin]: of energy. And
[Jay Goldberg]: Right.
[Ben Bajarin]: so to your point, right, CapEx, it gets offloaded if I'm now using another
[Ben Bajarin]: device, my edge device, my car, my PC, my automotive, my camera that's sitting
[Ben Bajarin]: right on a stoplight, um, it's, it's the one handling that power, right? So
[Ben Bajarin]: I'm offloading power as a part of that as well, because I think you could
[Ben Bajarin]: make the strong argument like we have that we just don't have the grid for
[Ben Bajarin]: all of this to be run, you know, in, in the cloud, especially amongst the top
[Ben Bajarin]: three hyperscalers, let alone people include in, in broad terms, Apple as
[Ben Bajarin]: a top four U S hyperscaler, you know, they, it's, you just can't run all this
[Ben Bajarin]: in the cloud. So I think that's an important reason, which again, goes back
[Ben Bajarin]: to your other point, right? A thesis of, of on device, which we did a whole
[Ben Bajarin]: segment on. And I think now we've seen a handful of demos and a bit of extra
[Ben Bajarin]: conversations on device. It's definitely not there today. But at some point
[Ben Bajarin]: in time, you'll be able to do a whole lot more of this on device, and it
[Ben Bajarin]: will feel not that far off from the things you've experienced in cloud-centric
[Ben Bajarin]: experiences.
[Jay Goldberg]: Yeah. I think, I think that's, that's how this is going to work out. And I
[Jay Goldberg]: think that it's just, it's just too cumbersome and the, the workloads are too
[Jay Goldberg]: big to really be run any other way. It has to be some significant portion
[Jay Goldberg]: of offload from the cloud.
[Ben Bajarin]: So the question here that sort of plagues me is, it feels like it's gonna
[Ben Bajarin]: be very hard to run this hybrid AI environment that a lot of people are talking
[Ben Bajarin]: about, meaning that I provide a service, I'm a cloud provider, and I realize
[Ben Bajarin]: that I've got tons of capabilities here to do what I wanna do in terms of this
[Ben Bajarin]: service, what people are paying for. But yes, I want to offload that to the
[Ben Bajarin]: device, but I need to know how much that device can be offloaded to that device.
[Ben Bajarin]: And that's not going to be an all things equal scenario, right? Devices that
[Ben Bajarin]: are five years old are going to need a whole lot more cloud help than devices
[Ben Bajarin]: that are one year old. But people talk about this hybrid environment. I want
[Ben Bajarin]: to have some of it in the cloud and some of it on device. I feel like that's
[Ben Bajarin]: a really tricky architecture. to talk about because again, it feels like
[Ben Bajarin]: the service needs to know, well, I can't offload that or it will be a terrible
[Ben Bajarin]: experience or it's a capable device, I can't offload 80%. I just don't know
[Ben Bajarin]: how this gets worked out but that's kind of how people talk about this hybrid
[Ben Bajarin]: cloud on device today. Could change in three years, but it feels very tricky,
[Ben Bajarin]: very complex to me to do that.
[Jay Goldberg]: I think you just described Android.
[Ben Bajarin]: Well, that's a whole different issue in my brain,
[Jay Goldberg]: Yeah,
[Ben Bajarin]: but you're 100% right.
[Jay Goldberg]: I won't start my Android rant, but I think when you talk about the edge, we're
[Jay Goldberg]: really talking about three things. We're talking about PCs, iPhone, and Android.
[Jay Goldberg]: Yes, there's cameras, those will come, and then there's automotive someday
[Jay Goldberg]: further out. But for the moment, what we're really concerned about is laptop
[Jay Goldberg]: and phone. And at their event, what? Two months ago, AMD actually... started
[Jay Goldberg]: talking about that, including some, some neural processing blocks in their
[Jay Goldberg]: seat, in their laptop CPUs. Apple's been doing it for a while. Obviously
[Jay Goldberg]: they have some in the phone as well.
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: Right. That's, that's really where this edge inference is going to take
[Jay Goldberg]: place is in those kinds of devices. And so what's going to, what's it going to
[Jay Goldberg]: take to accomplish that is, uh, Microsoft has to get windows to the point
[Jay Goldberg]: where it can do it.
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: Apple has to, it has to do likewise for both Mac OS and iOS. And I would say, uh,
[Jay Goldberg]: Microsoft is clearly fully invested in it and they're, they love generative AI.
[Jay Goldberg]: And I think
[Ben Bajarin]: Yeah.
[Jay Goldberg]: I have to imagine that there are teams inside Microsoft working pretty heavily
[Jay Goldberg]: to bring that. transform or to support into Windows sooner rather than
[Jay Goldberg]: later. Apple already done it for the Mac. And so once those frameworks get
[Jay Goldberg]: set up, I think it actually can happen pretty quickly. We will have this
[Jay Goldberg]: problem with Android where there are going to be a lot of devices that can't
[Jay Goldberg]: run it for years.
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: And I think that's just one more thing that has to add that to the list
[Jay Goldberg]: of problems that Google has with Android. Because when Apple thinks it's,
[Jay Goldberg]: you know, consumers really want generative AI support on the iPhone, they'll
[Jay Goldberg]: launch it, you know, if it's not in this iPhone, it'll be, you know, whenever
[Jay Goldberg]: six months after they think consumers are ready for it, right?
[Ben Bajarin]: Yeah. Well, and
[Jay Goldberg]: So
[Ben Bajarin]: it's.
[Jay Goldberg]: I would be surprised if they don't talk about something along these lines
[Jay Goldberg]: in September with the new iPhone.
[Ben Bajarin]: Sure. So there was a report, I'm sure everybody who listens to this saw
[Ben Bajarin]: that Apple, Mark Gurman wrote it at Bloomberg and just saying that they are
[Ben Bajarin]: working on their own GPT model. It sounds like it's a pretty large model in
[Ben Bajarin]: terms of overall size, like definitely large enough that whatever they're. building
[Ben Bajarin]: data set wise is not going to run on device at that size. I mean, roughly
[Ben Bajarin]: the stuff we've tried today seems to be a successful if it's in the 10 to 15
[Ben Bajarin]: billion parameter range. Anything north of that is crashing devices and running
[Ben Bajarin]: exceptionally slow. But that's today, right? That's not where we'll be in
[Ben Bajarin]: two to three years. But it shows kind of what's possible on device versus the
[Ben Bajarin]: size of cloud. But Apple, 100%. right, is going to want to do this. And arguably,
[Ben Bajarin]: like I said, they are a top four hyperscaler. They can create that cloud to
[Ben Bajarin]: device infrastructure as good as anybody, if they want, um, to handle their,
[Ben Bajarin]: their device fragmentation, which as you point out is not as, not nearly as,
[Ben Bajarin]: as tricky as Google's.
[Jay Goldberg]: Yeah, Apple technically is capable of doing it today. There's no question. If
[Jay Goldberg]: they wanted to get generative AI working on the iPhone, it could happen today.
[Jay Goldberg]: I think the question for Apple is, or the question that Apple is asking is
[Jay Goldberg]: more, what are we going to use this for? Apple doesn't like to add features
[Jay Goldberg]: that consumers don't care about.
[Ben Bajarin]: Right.
[Jay Goldberg]: How much do consumers really care about generative AI? How important is
[Jay Goldberg]: it to the user experience, the human experience, excuse me,
[Ben Bajarin]: Right.
[Jay Goldberg]: on an iPhone? And there are a lot of people who think that Apple is behind
[Jay Goldberg]: in AI as well, because they don't have generative AI today. I think they're,
[Jay Goldberg]: they're trying to figure out as am I, and a lot of people, like what is
[Jay Goldberg]: actually really useful for. And I actually had a Twitter debate with somebody
[Jay Goldberg]: who was saying, you know, AI is going to be really important, everyone's
[Jay Goldberg]: going to want it. And I said, well, how much more would you pay for a phone
[Jay Goldberg]: that does
[Ben Bajarin]: Sure.
[Jay Goldberg]: chat GPT or stable diffusion on the phone? And he says, well, I already
[Jay Goldberg]: paid chat GPT $20 a month. And I'm like, yeah, I understand that. But that's
[Jay Goldberg]: not the question. You like generative AI, you're willing to pay for it. How much
[Jay Goldberg]: though are you willing to pay for it to work on your phone in airplane mode?
[Jay Goldberg]: And I don't think anybody knows the answer to that question yet.
[Ben Bajarin]: So there's another though, part of this that feeds into this that I think is
[Ben Bajarin]: fascinating. And this again speaks back to companies who are prioritizing on device,
[Ben Bajarin]: those who are not be playing both sides of cloud to device in terms of their
[Ben Bajarin]: roadmap, is you exactly rightly point out, I don't think anybody is gonna pay
[Ben Bajarin]: more for these things. I think it's gonna have to just be an evolution of
[Ben Bajarin]: the silicon's... roadmap to increase those features. But if I'm a silicon designer
[Ben Bajarin]: today, so let's just say this is MediaTek, Qualcomm, and Apple, to some degree
[Ben Bajarin]: Intel and AMD on PCs, you do have to sort of make some decision about how
[Ben Bajarin]: much transistor budget you're gonna throw to something like an NPU, because
[Ben Bajarin]: it's a relevant decision, right? If I believe that I need that to compete, I
[Ben Bajarin]: gotta take that from something else, right? I gotta take it from my GPU blocks.
[Ben Bajarin]: my CPU blocks, right, something else, right? So they need to make calls on
[Ben Bajarin]: how important that is by how much diary are they're gonna commit to these
[Ben Bajarin]: things going forward. And that's a fascinating trade-off, right, that I think
[Ben Bajarin]: people are stuck with these next two years, when again, you and I can argue
[Ben Bajarin]: all day, people will 100% be willing to pay for those silicon bits in the
[Ben Bajarin]: data center more. They're not gonna pay for them on device. And so that's
[Ben Bajarin]: just a fascinating dynamic of maybe how fast the capabilities develop at
[Ben Bajarin]: the edge, when again, you've got to make these trade-offs with your transistor
[Ben Bajarin]: budget.
[Jay Goldberg]: Yeah, I think that's a really good way of framing it. And I think that sort
[Jay Goldberg]: of throws light on what I was saying about Apple is, Apple at this point
[Jay Goldberg]: doesn't seem to see the need to throw those blocks, those resources at it.
[Jay Goldberg]: They just don't see it as necessary.
[Jay Goldberg]: And I don't think anyone knows when that will change.
[Ben Bajarin]: Well, we'll see. I mean, I think we'll see with two fundamental things in my
[Ben Bajarin]: opinion, that'll happen this fall, right? Qualcomm is going to unveil their
[Ben Bajarin]: new PC chip, PC chip architecture. My hunch is that's going to be loaded with
[Ben Bajarin]: tops. Um, Apple has historically only bumped their tops up. If I recall two
[Ben Bajarin]: to three, you know, uh, I think they went from 10 to 15. Right. So maybe five
[Ben Bajarin]: or so. So, so what they throw at tops in next generation products will be
[Ben Bajarin]: telling. Right. If it's goes from 15 to 30, that's a pretty big jump. Right.
[Ben Bajarin]: But if it goes from 15 to eight to 18, you know, you'll, you'll again, you'll
[Ben Bajarin]: just sort of see how they're prioritizing. So, um, that's an interesting
[Ben Bajarin]: bit, but, but you're right. I think that's, that's really the call they've
[Ben Bajarin]: got to make is, is how much C block they throw to this stuff and it, and
[Ben Bajarin]: could it be premature, you know?
[Jay Goldberg]: Yeah, Apple, I think, typically tends to be a little bit conservative in jumping
[Jay Goldberg]: on the latest feature, right? The original iPhone was a 2G phone deep into
[Jay Goldberg]: the 3G era. That's the classic example. And they've had AI on their phone
[Jay Goldberg]: longer than anybody, right? They don't call it that. It's a neural engine.
[Jay Goldberg]: And it's there for a very specific purpose to help with image processing. I don't,
[Jay Goldberg]: yeah, it would be interesting to see it to what degree they change it, right?
[Jay Goldberg]: Or do they just sort of keep going at their steady incremental pace?
[Ben Bajarin]: Yeah. So I want to end on just circling back to the question at hand about
[Ben Bajarin]: is there even a way to sort of put a model around the additional lift of dollars
[Ben Bajarin]: that's coming to AI? I think we agree we can't do it on device. But to that
[Ben Bajarin]: point, in a number of earnings calls, as well as some commentary of some of
[Ben Bajarin]: the people who are doing this, it appears that it The answer is others haven't
[Ben Bajarin]: done this work either. That these models actually haven't been built or at least
[Ben Bajarin]: in a way that they're confident they will portray it to customers and or investors
[Ben Bajarin]: that they think these custom AI parts or their accelerator bits or whatever
[Ben Bajarin]: could lead to this much more revenue. Like in terms of, I guess their model
[Ben Bajarin]: guide, it seems like everyone kind of believes like we do. There's something
[Ben Bajarin]: there. There will be some additional tam in the data center, but no one's really
[Ben Bajarin]: done that work yet is basically what I'm saying. So as an answer to the question,
[Ben Bajarin]: I'm floating the no yes, it could be, but no, that hasn't worked been done. It's
[Ben Bajarin]: very vague speaking in vagueness.
[Jay Goldberg]: So I will start by saying, the answering the first part of your question,
[Jay Goldberg]: which is, yes, somebody can build this model. I don't think anybody has
[Jay Goldberg]: yet. You and I have toyed around with this. I think our model is as advanced
[Jay Goldberg]: as pretty much anybody's. And it's an important, important area. I tend to
[Jay Goldberg]: think, and I've been debating this a lot with you and with other people and
[Jay Goldberg]: myself lately. I'm coming down on the side that AI is additive to the semiconductor
[Jay Goldberg]: TAM.
[Ben Bajarin]: Mm-hmm, agree.
[Jay Goldberg]: And we're recording this on Friday, July 21st, TSMC reported last night.
[Jay Goldberg]: And they made some comments about seeing AI servers cannibalizing CPU servers
[Jay Goldberg]: in their, in the data they track. But, but what they're really saying is what,
[Jay Goldberg]: what they, what they really saying is because hyperscaler capex budgets
[Jay Goldberg]: for new data centers are fixed. We haven't gotten into the new budgeting cycle.
[Jay Goldberg]: that's going to accommodate this increase in
[Ben Bajarin]: Mm-hmm.
[Jay Goldberg]: AI needs. So that's a very short time window because they say long-term AI is
[Jay Goldberg]: going to be huge in the data center. I will say anecdotally, from what I
[Jay Goldberg]: can tell, I've heard a lot of the big hyperscalers are accelerating, pulling
[Jay Goldberg]: forward their data center physical plant expansion because they need AI, they
[Jay Goldberg]: need it soon, right? So I was talking to somebody recently who owns a plot
[Jay Goldberg]: of land, it's sort of a... tier two or tier three data center location.
[Ben Bajarin]: Hmm.
[Jay Goldberg]: Critically, it has power, it has electricity and they're in a region where
[Jay Goldberg]: the main tier one location is out of power. So they're on a grid, they have
[Jay Goldberg]: electricity and they just won the lottery. They've been sitting on this piece
[Jay Goldberg]: of land for years and now suddenly they have customers, they have all the hyperscalers,
[Jay Goldberg]: all those usual suspects plus many more knocking on the door saying, hey,
[Jay Goldberg]: can I get in? Let's get going, I need this, right? And... it's, you know,
[Jay Goldberg]: they have to build a plant, they have to build the building. So it's not
[Jay Goldberg]: going to come in, you know, next three months, but
[Ben Bajarin]: Right.
[Jay Goldberg]: it is clearly to me, additive to whatever the hyperscalers are doing, they're
[Jay Goldberg]: adding data centers that they know they're adding to the plan. And so it's
[Jay Goldberg]: not going to happen this quarter, but over the, I would say over the next year,
[Jay Goldberg]: we're going to see this spike.
[Ben Bajarin]: Yeah. Nope. I agree. I'm, I'm aligned with that. I think what questions I've
[Ben Bajarin]: heard, which again, I'm it's fine that nobody knows this, but just in terms
[Ben Bajarin]: of, of people being aware of the questions is really just how, how much
[Ben Bajarin]: additional capex could be thrown to this. Like there's a reasonable amount,
[Ben Bajarin]: again, knowing that we're up against limitations of physical space, we can't
[Ben Bajarin]: get enough wafers to meet that demand. So yes, it will grow, but it's not
[Ben Bajarin]: going to go. It's not going to be a hundred percent. right, growth year over
[Ben Bajarin]: year. So I think the understanding the amount at which it can grow and then where
[Ben Bajarin]: are those pockets that might get spent quickest I think are at least helpful
[Ben Bajarin]: if you're trying to come up with where might these dollars go over the next
[Ben Bajarin]: year or two, knowing where the constraints are. And I think that's a better
[Ben Bajarin]: way to kind of. look at this question about how much, because again, I go
[Ben Bajarin]: back to this, people are throwing these astronomical numbers out over the next
[Ben Bajarin]: five years. And I keep asking, well, do we even have the land and the grid?
[Ben Bajarin]: Can they even build out fast enough to meet that revenue number? But there's
[Ben Bajarin]: some reasonable amount of growth that's going to come from AI to these data
[Ben Bajarin]: centers. And so I think if the CapEx goes up, I don't know, I'm just making
[Ben Bajarin]: this up, but let's just say we landed on it's 5% to 8% a year in terms of
[Ben Bajarin]: your flexible budgets. it's a helpful way to look at where that growth can
[Ben Bajarin]: come from, being, again, additive to a number where they were already spending.
[Jay Goldberg]: Yeah, I agree. I agree. We need a little bit more work to put a precise number
[Jay Goldberg]: on it, but we're getting closer. And
[Ben Bajarin]: Yeah.
[Jay Goldberg]: it's going to be, I think, a meaningful amount for a lot of companies.
[Ben Bajarin]: Yes, no, I agree. Um, I will be at an event next week with one of the top
[Ben Bajarin]: three hyperscalers and, uh, we'll have a chance to talk to many of their customers.
[Ben Bajarin]: So we can, we can talk about that maybe on the next episode or episode after
[Ben Bajarin]: that. When, uh, when I can share more of what I've learned, but this is top
[Ben Bajarin]: of mine. This will be amongst my top questions in, uh, in, in CapEx spend
[Ben Bajarin]: for AI specific stuff, so more on that later then. Um, all right. Well, thanks,
[Ben Bajarin]: uh, everybody for listening. Review our podcast, give us likes, tell your
[Ben Bajarin]: friends, share us on socials, et cetera. Uh, we appreciate everybody listening.
[Jay Goldberg]: Thank you for listening everybody.