Limitless: An AI Podcast

Inference is becoming more important than pre-training in AI chips, including how pre-fill and decode work and why more compute is shifting toward serving models. 

Today we walk through Etched’s ASIC system for transformer inference, its claims around efficiency and throughput, and the tradeoff between specialization and general-purpose GPUs like NVIDIA’s. 

We also look at custom chip efforts from companies like OpenAI, Google, and Amazon, and argues that inference demand may keep growing as AI agents and long-running workloads expand.

------
🌌 LIMITLESS HQ ⬇️

EMAIL US:           info@limitless.fm
NEWSLETTER:    https://limitlessft.substack.com/
FOLLOW ON X:   https://x.com/LimitlessFT
SPOTIFY:              https://open.spotify.com/show/5oV29YUL8AzzwXkxEXlRMQ
APPLE:                 https://podcasts.apple.com/us/podcast/limitless-podcast/id1813210890
RSS FEED:           https://limitlessft.substack.com/

------
TIMESTAMPS

0:00 Inference’s New Frontier
2:14 Training Versus Inference
5:19 Etched’s Bold Bet
7:58 Building the Whole Rack
10:48 TSMC and the Hard Problems
13:29 Why Inference Matters
14:59 The Transformer Risk
17:02 OpenAI’s Jalapeno Chip
18:59 Why Accelerators Keep Winning
22:28 The Market Is Underpricing It
23:10 NVIDIA Is Still in the Game
24:56 Vertical Integration Wins

------
RESOURCES

Josh: https://x.com/JoshKale

Ejaaz: https://x.com/cryptopunk7213

------
Not financial or tax advice. See our investment disclosures here:
https://www.bankless.com/disclosures⁠

Josh works with Anthropic as a contractor. All views expressed are his own and do not represent Anthropic, its leadership, or its affiliates. Nothing in this episode is investment advice.

Creators and Guests

Host
Ejaaz Ahamadeen
Host
Josh Kale

What is Limitless: An AI Podcast?

Exploring the frontiers of Technology and AI

Josh:
So at limitless we are always in pursuit of alpha in pursuit of the thing that's

Josh:
around the corner the interesting investment opportunity and i'm very excited

Josh:
to say that we found a new one

Josh:
and it comes in the form of this company named etched and it comes in the form

Josh:
factor of this thing called inference over the weekend i know we spent a lot

Josh:
of time while people were shooting off fireworks reading about this one small

Josh:
company that's intention is to change the way that we look at inference forever in fact

Josh:
They're just a couple of 24-year-olds who have already dethroned NVIDIA across a series of benchmarks.

Josh:
This is in the world of inference that I think a lot of people aren't really

Josh:
paying too much attention to. A lot of people are still focused on pre-training

Josh:
and they think NVIDIA GPUs are the be-all end-all. But we've seen this trend

Josh:
popping up of companies like Google building their custom accelerators, companies like Amazon.

Josh:
And now we have a series of startups you might have heard like Cerebris or Grok,

Josh:
which was acquired by NVIDIA for a tremendous amount of money.

Josh:
Cerebris just went public a few weeks ago.

Josh:
And there's a lot to unpack here, both as an economic opportunity,

Josh:
but also as just a really cool opportunity for a new frontier in AI,

Josh:
which is this accelerated inference chip architecture that everyone seems to be gunning towards.

Ejaaz:
Now, in order to understand everything we're going to describe in this episode,

Ejaaz:
I think it's important to start off with what on earth is inference.

Ejaaz:
So typically, when you use a model, you write a prompt, you send that prompt,

Ejaaz:
and suddenly, magically, an answer appears from the LLM, whether you're using Claude or ChatGPT.

Ejaaz:
Now, what happens when you click enter is the following.

Ejaaz:
Your prompt gets sent to a server. A server rack has a bunch of different AI

Ejaaz:
chips, commonly known as these GPUs. That's what you associate NVIDIA with.

Ejaaz:
And what this GPU does is it firstly reads your entire prompt and processes it.

Ejaaz:
And this is known as something called pre-fill. So the chip does this.

Ejaaz:
It reads your prompt, processes it.

Ejaaz:
And then the second thing that it does is it draws on memory that it has for

Ejaaz:
your entire conversation, the context that you have about you,

Ejaaz:
the context that you gave it in previous prompts in that exact same conversation.

Ejaaz:
And then it starts generating a response to you, token by token, one at a time.

Ejaaz:
And this is something called decode. So typically, if you interact with AI chips,

Ejaaz:
what's happening on the backend is this pre-fill and decode type process.

Ejaaz:
And that's what generates your answer at the end. And that's effectively what

Ejaaz:
inference is as an output.

Ejaaz:
So now when we're talking about chips in general,

Ejaaz:
You think of NVIDIA, you think of GPUs, and you think, okay,

Ejaaz:
well, these chips are primarily used for AI training. And recently,

Ejaaz:
as you just mentioned, Josh, a bunch of these different AI labs have announced

Ejaaz:
that they're building their own AI chip.

Ejaaz:
And the question then becomes, what does this AI chip specifically optimize

Ejaaz:
for? And the answer is very simple.

Ejaaz:
It's inference. And this has been getting more busy over the last couple of

Ejaaz:
months. OpenAI recently announced that they're building their own jalapeno chip.

Ejaaz:
You've got Anthropic that's rumored to be building their own in terms with Samsung.

Ejaaz:
You've got Cerebrus and Grok, as you just mentioned.

Ejaaz:
And now we have this new startup called Etched, which is building a brand new

Ejaaz:
chip which competes pretty effectively with NVIDIA's ability to perform inference.

Josh:
Yeah, there's a key difference between training and inference.

Josh:
Training is that thing that happens one time, and it normally takes months long.

Josh:
So when you hear a company is training the new GPT or training the new quad

Josh:
mile, that's what it is. It's using this pre-training.

Josh:
Inference is a totally different animal. And a really interesting thing that

Josh:
I learned about doing research about this is that the amount of inference demand

Josh:
as of I believe two years ago was about one-third and it was two-thirds of pre-training.

Josh:
Now that's flipped to be two-thirds inference, one-third pre-training,

Josh:
and it's expected by the end of this year even it's going to be heading towards

Josh:
80 percent which is a really interesting stat considering that NVIDIA holds

Josh:
roughly 75 percent of the total chip share

Josh:
but they're not actually optimized for this and And that is a signal because,

Josh:
NVIDIA's share has actually gone up in terms of how many, what percentage of

Josh:
the AI world is using NVIDIA GPUs, when the reality is that they're not optimized

Josh:
for this type of inference.

Josh:
So as inference demands are going higher, NVIDIA's share is also going higher,

Josh:
but NVIDIA chips are not optimized for this inference.

Josh:
And that's a signal showing that it's really the only thing available.

Josh:
No one's actually figured out how to build these custom chips at scale.

Josh:
Therefore, the entire market must buy NVIDIA chips. But there's this new...

Josh:
Series of companies that's been sprouting up like etch that we're going to talk

Josh:
about that is going to solve this problem and their intention is to do so at

Josh:
a scale that's large enough to offset this kind of asymmetry that we're seeing in the market.

Ejaaz:
Did you know that uh for the average frontier ai lab and it's funny that i say

Ejaaz:
average because there's like literally three of them

Ejaaz:
around 40 to 50% of their entire compute availability is now put towards inference.

Ejaaz:
Just think about that for a second. Like typically you'd think that you'd use

Ejaaz:
the majority of that compute to train the next big bad model,

Ejaaz:
the next Mito 6 or whatever that might be, but it's actually being used to serve up the model.

Ejaaz:
And that has happened exponentially more as people start to spin up these AI

Ejaaz:
agents, which work 24 seven for you.

Ejaaz:
So inference has become this super important thing and optimizing the hardware

Ejaaz:
around that has now become the new moat.

Ejaaz:
Forget about pre-training, it's all about inference. It's actually how you make

Ejaaz:
smarter models. We talk about Chinese AI models a lot on this show and we think

Ejaaz:
about like the fact that they don't have NVIDIA GPUs.

Ejaaz:
So have they been able to train models that are 90% of their capability of some

Ejaaz:
of these frontier American models?

Ejaaz:
It's because they've got really creative with inference. So inference is actually

Ejaaz:
the next sector and it's not a moat that NVIDIA has, as you mentioned.

Ejaaz:
Josh, I think we need to get into what some of these startups are doing and

Ejaaz:
why they're so competitive to NVIDIA. Because if I'm listening to this,

Ejaaz:
right, I'm thinking NVIDIA has like, what, a $5.2 trillion market cap.

Ejaaz:
It's the most valuable company on earth.

Ejaaz:
How on earth can a ragtag group of Harvard dropouts actually beat these guys?

Josh:
Well, it seems like it's impossible, right? But then I look at the website of

Josh:
Etch and I listen to these guys talk and they're unbelievable.

Josh:
And these are two 24 year olds that managed to somehow build a company large

Josh:
enough to seriously threaten a lot of these big incumbents and

Josh:
i think the the idea of the company the main ethos is baked around an idea that

Josh:
i actually didn't even know was a reality

Josh:
um because as we're talking about the demand and inference going up they refer

Josh:
to the nvidia gpu this is the godfather of ai this is how everything is trained

Josh:
and typically when nvidia gpus use inference they're only achieving about 30 to 40

Josh:
utilization which is crazy there is 60 to 70 of the chip that's totally unused

Josh:
and the market's It's just saying, okay, well, I guess that's the best we have for today.

Josh:
We're just gonna go and train with NVIDIA GPUs or in serve inference.

Josh:
And the reality is that there is a huge amount of.

Josh:
Improvements, both in efficiency and throughput that you can create on these chips.

Josh:
And that's what this team set out to do. They said, we're going to build a chip

Josh:
that is close to 100% efficient and utilization. And they do that by doing a

Josh:
lot of really interesting things around thermals and around vertical integration.

Josh:
And that's kind of the idea for this company, Etch. Now, they just came out

Josh:
of stealth. They have been in business for about three years now.

Josh:
I believe this started in 2023.

Josh:
And you can imagine how difficult it would have been in 2023 three to convince

Josh:
a series of investors as what were they then maybe 21 years old,

Josh:
that they need to invest not just a couple million dollars but a hundreds of

Josh:
millions of dollars in order to actually make this company reality fast forward

Josh:
three years turns out they've did it

Josh:
they've gotten over a billion dollars in customer contracts

Josh:
they've raised over 800 million dollars of funding and the early tests that

Josh:
they have on the server rack are showing that it has true state-of-the-art output

Josh:
on latency on power efficiency and on inference workloads

Josh:
and that is kind of the basis of this company, who these people are,

Josh:
and what they're working on right now.

Ejaaz:
Now, I'm sure you're listening to this and you're thinking, well,

Ejaaz:
guys, this is a private company. I don't know how I could get access to it.

Ejaaz:
And trust me, I feel your pain.

Ejaaz:
There are actually public ways that you could potentially get exposed to the

Ejaaz:
success of what Etch is building and a bunch of the other companies that we're

Ejaaz:
about to mention on this episode.

Ejaaz:
We'll get to that in a bit. But first, I want to talk about what breakthroughs

Ejaaz:
these kids, and I literally mean that, they're 24 years old,

Ejaaz:
made over the last three years that has given them such an insane valuation

Ejaaz:
When they haven't even released a proper product just yet.

Ejaaz:
And the answer is very simple. They haven't actually built a chip.

Ejaaz:
In fact, they're not building a chip.

Ejaaz:
They're building an entire chip rack. And that's their whole thesis.

Ejaaz:
What they did was they looked at how inference worked. They looked at how NVIDIA

Ejaaz:
GPUs performed and they saw, as you just mentioned, that it only utilizes 30 to 40 percent.

Ejaaz:
Imagine paying $50,000 to $150,000 for this machine and it only works 30 to

Ejaaz:
40 percent of its true capacity.

Ejaaz:
You'd be pretty annoyed at that ROI, right?

Ejaaz:
So they looked at the entire process and they thought, hmm, it's not just good

Ejaaz:
enough to build a good chip.

Ejaaz:
We have to build the entire system that can be placed inside a data center that

Ejaaz:
allows for 80 to 90 percent inference utilization.

Ejaaz:
So here are the two things that they did. Number one, they figured out this

Ejaaz:
mind-blowing thing, how to use less voltage to get the same smart answer that

Ejaaz:
you get from Claude or a GPT, for example.

Ejaaz:
And the way that they did this was they redesigned the entire chip to base around

Ejaaz:
the transformer architecture, which is what all LLMs are based off of.

Ejaaz:
Now, let's say in the future, you get an AI model that doesn't use the transformer

Ejaaz:
model. Well, unfortunately, you can't use that chip. So it's hyper-specialized.

Ejaaz:
And what they were able to achieve was a low voltage for this.

Ejaaz:
Now, there's this equation, right? I'm not going to get too technical on you, but

Ejaaz:
It's voltage, or maybe it's power equals voltage squared.

Ejaaz:
So the fact that they halved the voltage for their chip means that they use,

Ejaaz:
they require 75% less power to power their chip.

Ejaaz:
So the long story short is you need so much less energy to achieve the same

Ejaaz:
amount of smart answer that you get from your AI model.

Ejaaz:
What does this mean? In practice, well, you save tens to hundreds of millions

Ejaaz:
of dollars, or you can spit out way more tokens

Ejaaz:
per second, which means that you can serve millions of more users,

Ejaaz:
which is exactly what Cerebris offers, which is exactly what Grok offers,

Ejaaz:
but in a much more efficient way without losing intelligence for your model.

Ejaaz:
Finally, the second thing that they achieved was they looked at the memory of

Ejaaz:
a chip and they were like, this is hugely inefficient.

Ejaaz:
And they redesigned it from scratch and they call it cluster scale memory.

Ejaaz:
And what this means is, you know how you add memory to a chip typically?

Ejaaz:
Well, they also have a shared memory pool between their different chips.

Ejaaz:
And the long story short, what that means is they can move data super quickly

Ejaaz:
in a second, which means that you get a faster answer.

Ejaaz:
It's all optimized completely around getting you a quicker answer that is of

Ejaaz:
the same intelligence and capability as your Claude or GPT.

Josh:
This comes in the form of a very specific type of chip. Like when we're talking

Josh:
about these NVIDIA chips, that's a GPU.

Josh:
And then what we're talking about here is more of an ASIC. It's something that

Josh:
is application specific and built specifically for this.

Josh:
And what's funny is, as I was listening to one of the podcasts that they were

Josh:
discussing, they actually, they referenced Bitcoin mining ASICs as one of the

Josh:
inspirations to prove that it was possible because Bitcoin mining ASICs are

Josh:
very specific computers for a very specific type of math.

Josh:
And they're able to do so with so much more efficiency and when you can edge

Josh:
out that efficiency over the scale the amount of tokens per second you could

Josh:
generate at scale using these is tremendously higher so anyone who's looking at this on a

Josh:
performance per watt basis or performance per flop i guess you could say with these,

Josh:
gpus or these accelerated processors it's going to be a financial no-brainer

Josh:
to do this and as i was listening to the stories of this team it was unbelievable

Josh:
so first of all they're working with tsmc um already they managed to convince

Josh:
tsmc that their technology was good enough to convince them to start to do this run

Josh:
and they're in bangalore and they are.

Josh:
Half of the team is there. Half of the team is in the States.

Josh:
They're working 12 hours a day over there. Then they pass over the work to the US.

Josh:
They're working 12 hour night shifts, and they're going 24 hours a day.

Josh:
And they finally get an opportunity to test this chip on TSMC.

Josh:
And I remember the conversation that they were having.

Josh:
They're like, yeah, we called up TSMC. It was the middle of the night,

Josh:
and they were doing this kind of live feed.

Josh:
And you're looking at a chip, and it either will light up green or red based

Josh:
on which wafer is good, which wafer is bad.

Josh:
So the idea is you want the whole chip to light up green, or most of them to

Josh:
lit up green the entire thing lit up red not a single one of them worked and the problem was that.

Josh:
There was this and this is this is a little technical so i'm just going to abbreviate

Josh:
here because i didn't fully understand myself but basically there is these like

Josh:
clock signals that exist within it that need to be synchronized

Josh:
and i didn't realize how difficult chips were you just this is when i realized

Josh:
like oh my god this is like actually a really difficult problem this is why no one's doing it

Josh:
they needed to align these two clock signals within 50 picoseconds and I'm like,

Josh:
okay, what's a picosecond?

Josh:
That's 50 trillionths of a second.

Josh:
And light itself travels about one and a half centimeters during that time.

Josh:
So they're really optimizing for these things at the speed of light.

Josh:
And a couple of engineers actually quit. They said it was impossible.

Josh:
You will never be able to solve this.

Josh:
And the team went off and solved it two weeks later. And I think it's a testament

Josh:
to how one difficult the problem is, but two, how cracked this team is,

Josh:
is that the fact that they're working 24 hours a day, they are actually in the

Josh:
factory with half of the team.

Josh:
They're back in the U.S. with the other half, and they are solving these seemingly

Josh:
impossible problems that are enough to get people to actually quit.

Josh:
It's a testament to how impressive they are and specifically what it takes.

Josh:
When I'm thinking about this from a generalized investment angle,

Josh:
I'm like, okay, who else is in this game? Google has their TPUs.

Josh:
Amazon has their Tranium chips. I know for a fact they're not doing this.

Josh:
They do not have people sleeping in the factory. They do not have people like

Josh:
really heads down with their sole purpose of building these chips.

Josh:
And one of the things I found interesting that they mentioned is when they're,

Josh:
They're hiring people who are excited because this company lives or dies by these chips winning.

Josh:
And a company like Google, in the case that TPUs don't work out,

Josh:
the company's still fine. So I really found that kind of an inspiring story

Josh:
on how impressive this team was and how difficult it really is to build these

Josh:
low voltage inference chips.

Ejaaz:
The question that then pops in my mind is, why are they going so hard at this?

Ejaaz:
Why do they believe so strongly that inference is the moat and why it's so important?

Ejaaz:
The answer can actually be revealed by the investors that they have on that

Ejaaz:
cap table. Brian Johnson, who is the health and longevity guy,

Ejaaz:
right? Like, what's he doing with this?

Ejaaz:
What's he doing on the cap table here? You've got Jane Street,

Ejaaz:
which is effectively a quant slash hedge fund, the best in the world.

Ejaaz:
And then you look at Peter Thiel, you look at a few others, TSMC,

Ejaaz:
by the way, through their own venture fund is also invested in this.

Ejaaz:
And you start to think, hmm, what might be the problem that they're solving?

Ejaaz:
And the answer is very simple.

Ejaaz:
It's everything. If you have a fasted chip that is spitting out tokens at lightning

Ejaaz:
speed, but at the same amount of intelligence, guess what?

Ejaaz:
You can solve that research problem five times faster. Guess what?

Ejaaz:
Oh, you're looking for a cure for this science problem or for this particular disease?

Ejaaz:
We can solve it faster because we spit out the most tokens per second without losing intelligence.

Ejaaz:
And that's the main pitch. Brian Johnson says it here in his own tweet,

Ejaaz:
breaking news for people who want to look hot, be young and not die.

Ejaaz:
A few years ago, two college dropouts told me that they could accelerate longevity

Ejaaz:
by building a faster AI chip.

Ejaaz:
And that single sentence is their entire thesis. If they build out the better

Ejaaz:
machinery and chip architecture that allows you to go use this AI stuff much

Ejaaz:
quicker, you can end up beating the

Ejaaz:
clods and GPTs that run on current NVIDIA GPUs if they run on these GPUs because

Ejaaz:
they can do the problem faster.

Ejaaz:
And that's the main unlock here and why this is so impressive in my opinion.

Josh:
Yeah. Okay. So there is like this edge case that I think I wasn't even accounting

Josh:
for really until we spoke about it earlier, EJS, which is that they're making

Josh:
a very specific bet on this one specific architecture. Transformer.

Josh:
Yes. On the transformer architecture. So since the beginning of time,

Josh:
basically since GPT-2, all of today's frontier models run on this thing called

Josh:
the transformer architecture. And it is a specific type of architecture.

Josh:
You've probably heard of it, or we've talked about it on the show.

Josh:
And it's basically this recursive learning thing where it goes through this

Josh:
like latent space and then it generates some words and it's the next token prediction

Josh:
it's kind of how we've always predicted next tokens,

Josh:
this is entirely built upon the fact that that is going to continue because

Josh:
my understanding and you just correct me if i'm wrong but this hardware specifically

Josh:
built for that and because it's specifically built for that architecture the

Josh:
payoff is pretty high they can get maybe a 10 to up to 50 times multiple

Josh:
in terms of efficiency because they're hyper specialized but that is under the

Josh:
assumption that this is going to continue to be the primary architecture that

Josh:
his language models run on top of

Josh:
in the case that that shifts, my understanding is that this is actually hard-coded into the chips.

Josh:
They would need to rebuild a lot of the stack in order to kind of solve for

Josh:
this. So is this an existential risk or is this like, how could you think about this?

Ejaaz:
No, no, it's very accurate. So let's say hypothetically a year from now,

Ejaaz:
someone, let's say Andre Carpathy discovers a brand new AI design architecture

Ejaaz:
for a chip and it's not a transformer.

Ejaaz:
You know, hey, look, I built this new AI model and it runs on a different design than the transformer.

Ejaaz:
Etched chips won't work anymore, like if you run those newer models on their chips.

Ejaaz:
What they've done is they've hard-coded the computation graph,

Ejaaz:
which is basically the algorithm, onto the silicon itself.

Ejaaz:
Now, if you look at an NVIDIA GPU, yeah, it's really underutilized at 30% to

Ejaaz:
40%, but you get the flexibility of being able to run whatever model architecture

Ejaaz:
that you want in the future.

Ejaaz:
You can't do that with etched chips. So,

Ejaaz:
They kind of need to redesign from scratch. You're going to have to like,

Ejaaz:
you know, that story you were telling of them going to Bangalore,

Ejaaz:
they need to redo that entire process all over again.

Ejaaz:
So it is a big bet, but maybe it might be the right one because other companies

Ejaaz:
themselves, Josh, are also going down this route, including a little known company known as OpenAI.

Ejaaz:
They announced a few weeks ago that in partnership with Broadcom,

Ejaaz:
they're going to be building their own purpose-built LLM chip known as Jalapeno.

Ejaaz:
And what was interesting about this announcement is the chip is optimized around,

Ejaaz:
you guessed it, inference, how to serve models and tokens faster,

Ejaaz:
but it's hyper-optimized around ChatGPT specifically.

Ejaaz:
But there is a slight difference between the chip that they're building and

Ejaaz:
what Etched is building, and it's the following, which is they didn't hard-code

Ejaaz:
the transformer architecture, which I thought was super interesting.

Ejaaz:
They allowed it to be general, but hyper specialized for GPT specifically.

Ejaaz:
And you might wonder, like, why are they doing it? And how are they doing it?

Ejaaz:
Well, the how that they're doing it is they're open air, they own the models,

Ejaaz:
they know how these models work and how to load tokens for it.

Ejaaz:
So they're like, okay, I know what kind of requests or prompts our users have for them.

Ejaaz:
We know how to process that we'll build a chip and rack system hyper optimized for that.

Ejaaz:
But the why you're doing that is what I mentioned earlier, which is if they

Ejaaz:
can own the chip architecture and serve ChatGPT for much cheaper and much faster,

Ejaaz:
they can solve problems.

Ejaaz:
And that ineffectively becomes the better model if you compare it to a Frontier

Ejaaz:
AI lab that doesn't have their own AI chips.

Ejaaz:
And that's why I'm actually more bullish on a Frontier model lab specifically

Ejaaz:
integrating vertically with their own chip versus Etched, who has the issue of

Ejaaz:
They now need to either get acquired by a Frontier AI lab to have that vertical

Ejaaz:
integration, or they're ending up serving multiple

Ejaaz:
labs where they can't hyper-specialize the inference workload.

Ejaaz:
And that's the main difference.

Josh:
Yeah. And also what was really impressive is the speed in which they taped this

Josh:
thing out. The norm is about one and a half to two years to make this happen.

Josh:
They did this with the help of Broadcom in, I believe, yeah,

Josh:
nine months. And the time it takes to have a baby, they birthed a jalapeno.

Josh:
So that's pretty impressive, I will say.

Josh:
And it's interesting too, because this isn't the first accelerator chip that they've had.

Josh:
I mean, they had this famous deal with Cerebris, which is now publicly traded.

Josh:
And Cerebris is basically playing in the same exact mode. It's like,

Josh:
hey, we can serve tokens very quickly and very efficiently.

Josh:
And that's kind of what they've been using Cerebus for, but it seems like that's not enough.

Josh:
And I think this is the general theme as we're kind of shifting over to looking

Josh:
at this through an investment lens is that there is no limit to the amount of

Josh:
inference capability that we can have right now.

Josh:
It seems like any time that anyone comes up with any sort of efficiency improvement

Josh:
or any increase in the amount of tokens that could serve per second, it just gets eaten up.

Josh:
And when you think about the trends, this makes sense. It's like very economically

Josh:
viable to pay a huge premium for this because when you think about the frontier

Josh:
models, every time, what's one of the things that we talk about?

Josh:
It's the duration of a task that it can do.

Josh:
So we went from being able to just type in and you get a response in a couple

Josh:
seconds to a couple minutes to a couple hours.

Josh:
Now we're at days, weeks, and even months for some tasks.

Josh:
And if you're running this tremendously difficult problem or if you're trying

Josh:
to solve, if you're trying to migrate a huge code base or if you're trying to

Josh:
do these really complicated technical tasks,

Josh:
compressing a few months or a few years into half that time,

Josh:
into a third of that time, is not only a huge amount of savings, but it's a huge amount of.

Josh:
Acceleration that you can get as a company in terms of how much progress you

Josh:
can make quickly and if you're a company like open ai whose goal is to serve these customers

Josh:
being able to serve double the amount of customers during the same amount of

Josh:
time is a huge efficiency unlock so

Josh:
having the ability to have this accelerated inference ability where you can

Josh:
serve tokens quicker more efficiently more effectively seems like it's going

Josh:
to be a very important trend that i don't see ending soon so we saw the cerebrus chart and

Josh:
cerebrus actually didn't do too well after the IPO.

Josh:
It wasn't super high. It hasn't been doing well.

Josh:
But the reality is, is that does that feel right to you when you see this chart

Josh:
and you look at the demands that we're talking about? Like, is Cerebra's properly

Josh:
priced here down 35 and a half percent?

Ejaaz:
No, because I think, and it's the reason why we decided to make this episode,

Ejaaz:
I think a lot of people are unaware that inference is actually the new mode

Ejaaz:
for how to train a better model, but also how to optimize sending tokens to a lot of people.

Ejaaz:
I think people of the majority of people are stuck in the mindset that you just

Ejaaz:
use an LLM, maybe like you use Google.

Ejaaz:
At most, you probably have less than a percentage of people on the entire earth

Ejaaz:
that has spun up an agent and runs it autonomously, even for an hour.

Ejaaz:
And the trend is very clear. You will have a bunch of these AI models working

Ejaaz:
autonomously for you for hours or days at a time. And guess what?

Ejaaz:
It's going to burn a lot of tokens. And guess what? You want it to serve as

Ejaaz:
many tokens as you can as quickly as you can, because you will beat the competition.

Ejaaz:
You will get to the answer quicker. And that means you can do more work,

Ejaaz:
et cetera, et cetera, and solve all your problems. So the point is,

Ejaaz:
if you want to achieve that, you need a different chip architecture completely.

Ejaaz:
And NVIDIA, the daddy of all companies that are building these GPU architectures,

Ejaaz:
hasn't figured out that problem right now. And so you have these companies like

Ejaaz:
Cerebris that are publicly traded.

Ejaaz:
You have these companies like MediaTek, who is helping design some of these

Ejaaz:
specific chips that are around their fronts.

Ejaaz:
Now that's a chart, right? You're up 180% year to date.

Ejaaz:
You've got Broadcom as well. Let's take a look at Broadcom. Broadcom is the

Ejaaz:
company that is actually doing a lot of design.

Ejaaz:
Look at this. It's basically up 10% year to date, or less than 10% year to date.

Josh:
So I think- That's funny. That's what it's up today.

Ejaaz:
Yeah. Wait, really?

Josh:
It's like today is the total or half of that today.

Ejaaz:
That's funny. So the point I'm trying to make is I think it's an asymmetric

Ejaaz:
bet that is sitting in front of everyone's faces right now.

Ejaaz:
Everyone is obsessed with memory, the memory bottleneck, which is very much,

Ejaaz:
you know, a big deal. Everyone's looking at power. They're like,

Ejaaz:
we can't power these GPUs. We can't power these data centers.

Ejaaz:
But a lot of people are forgetting that a bulk of profit margins that come to

Ejaaz:
a ton of these AI labs when they eventually IPO is going to come from inference.

Ejaaz:
Anthropic themselves is rumored to become profitable this quarter,

Ejaaz:
by the way, because of the profit margins that they're making on inference specifically.

Ejaaz:
So if you believe in agents, if you believe in autonomous work in the future,

Ejaaz:
you have to bet on inference chips. And these are the companies that are currently available.

Ejaaz:
And you've got companies like Etched, which hopefully comes out of private placement soon.

Josh:
Yeah, I will say, don't count out NVIDIA either, because it's not like they're

Josh:
unaware that this has happened. In fact, they were ahead of the trend,

Josh:
and they acquired that little company named Grok for, what was it, $20 billion.

Josh:
Yeah, so they are not in the dark about this. NVIDIA is very smart. Jensen is very clever.

Josh:
He is fully aware of the situation at hand. It's just it's difficult to move

Josh:
a giant company to do this at scale.

Josh:
So you have to imagine that Grok acquisition was step one. Grok was kind of

Josh:
similar to what all these other companies are doing with the very specific accelerator

Josh:
chips that are meant for inference.

Josh:
You have to imagine they're working now very hard to integrate those into their

Josh:
chips to create this new line that allows for more optimized inference,

Josh:
less general purpose, more narrow band.

Josh:
But these are the main players in the space. And it's funny,

Josh:
it seems like everyone's kind of doing it to varying degrees.

Josh:
We have Google has their TPUs that we talk about, They have the Ironwoods.

Josh:
Then we have Ace Amazon, who has their terrarium chips.

Josh:
So there's a lot of large companies doing this, but it seems like the velocity

Josh:
of these smaller ones, like the Cerebris, like Grok, like Etched,

Josh:
is really, they're moving so quickly because they're small and nimble.

Josh:
And you have to imagine that, like a company like Etched, if they're not going

Josh:
to get acquired, they're just going to continue to explode in terms of valuation,

Josh:
so long as they can make these at scale.

Ejaaz:
So my bet on Etched is they're an amazing company, but they will eventually

Ejaaz:
get acquired by either an Anthropic or an OpenAI or maybe even Google.

Ejaaz:
But one of the Frontier Labs will. And the sole reason behind that is in order

Ejaaz:
to build the best chip at inference, you need to be one and the same with the

Ejaaz:
actual model that is serving the tokens itself.

Ejaaz:
And their entire philosophy, the founders have said like on podcasts,

Ejaaz:
is they want to build the best inference product and they need to be close to

Ejaaz:
the model lab. So that's my bet.

Ejaaz:
In, let's say under three years, they get acquired, if not sooner.

Josh:
Vertical integration baby i mean that's the way it goes i always tell the end

Josh:
of time refer to the apple m series chips i'm like this is how good it can be

Josh:
if you vertically integrate you can change the entire uh,

Josh:
way that a product line works if you can figure out how to vertically integrate

Josh:
these and that's clearly what everyone is trying to do open out with their jalapeno

Josh:
everyone has their own chip everyone's got their own asic and i think with that

Josh:
that is the inference episode that is kind of where we stand

Josh:
that is where the ball is rolling towards it is inference it is very fast compute

Josh:
is answering your questions and your very long questions as fast and efficiently

Josh:
as possible and at the end of the day it's it's really just an efficiency thing

Josh:
it's like if you can get more performance per watt if you could generate higher

Josh:
intelligence tokens then you can you can basically win and there is no limit to the demand in which

Josh:
there is going to be over the next i don't know how long um in terms of generating

Josh:
tokens so yeah very bullish on the company wish i could participate in etch

Josh:
but i can participate in some others which i am strongly going to consider and

Josh:
yeah i think that that's the episode.

Ejaaz:
That's that's pretty much it i have uh placed personal bets accordingly uh across

Ejaaz:
some of the companies that we've spoken about i wish i could have gotten access

Ejaaz:
to etch but i did not um private these private companies man i need to figure out how uh

Ejaaz:
how all these other podcasters do it, man. Invest like the best.

Ejaaz:
They're just absolutely killing it.

Josh:
We got to get a Limitless fund.

Ejaaz:
I know, we need a Limitless fund if anyone wants to help us raise that.

Ejaaz:
But speaking of requests to our listeners, I don't know if you've heard,

Ejaaz:
but Limitless is in the market for sponsorships.

Ejaaz:
And we've actually received a bunch of outreach from you folks,

Ejaaz:
but we're always hoping to hear from more of you.

Ejaaz:
So if you are someone in a position, if you like the content that we hear about,

Ejaaz:
and if you're someone in a position that wants to help support us,

Ejaaz:
please reach out. We would love to partner with you. We get so much support

Ejaaz:
from our fans and listeners, so much engagement, and it might be the best place

Ejaaz:
to feature your product or service.

Ejaaz:
Or if you know of someone that might be interested, please let them know.

Ejaaz:
Send us a DM. We're on X. There's an email in the description below.

Ejaaz:
Just reach out to us. We read everything, even comment. Let us know.

Ejaaz:
We would greatly appreciate your support. But that is it for the episode.

Ejaaz:
Wherever you're listening to this, by the way, if you could thumbs up,

Ejaaz:
if you could subscribe, if you could give us a rating, leave us a comment,

Ejaaz:
say hello. If you disagree with us, let us know.

Ejaaz:
And I guess that's everything, Josh.

Josh:
That's it. Yeah. Thanks everyone so much for watching and we'll see you next one.

Ejaaz:
See you guys.