Frankcx: Hey, everyone.
Last week, I said that AI conversations were moving from cost to control.
Well, cost just crashed too.
The best AI is getting dramatically cheaper, and the AI you can
run on your own device is also getting dramatically better.
this week's question is simple: If being smart gets cheap, are the big AI companies
still gonna charge a premium for?
From cost to control to what's left.
find out
Welcome back to The Local Host.
I'm Frank, and this week, I am your local host again.
This is show colon 0002 port speak.
hey, let's do a little bit of around the horn with, Jacob, Chauncey, and Robert.
And we know Neal is, on a important call, so he will join us.
You know, customers come first, but he'll, join us as soon as he's available.
So hey, Jacob, how are
Jacob: Hi, I'm doing great, Frank.
Yeah, I'm Jacob.
I work for Microsoft, so that's an important disclosure.
I love telling stories about on-device productivity, including AI.
Excited to be here.
Frankcx: Awesome.
Chauncey: Awesome.
Yep.
Chauncey Larson, tech nerd, also worked for Microsoft
Frankcx: And our hockey fan from Florida.
Robert: It almost seems like an oxymoron.
I am a little bit of both.
And I'm Robert, and, I too am, employed by Microsoft under full disclosure,
I'm gonna do my best to match Frank's, electric energy on today's show
Chauncey: You are bringing the best hat of the crew though, that's for sure
Frankcx: yeah, that, the Copilot, "I am your father."
Chauncey: Love it
Frankcx: Hey, let's get into it.
It's been a wild week.
I swear every day I look at the news and I see something else that's coming in around
AI and it's just an ever-evolving, piece.
But we have this section of the show we call the rundown, which
is kind of news of the week.
Hopefully you're listening to this and you understood that, as we're recording
this, it's been reported that NVIDIA, is thinking of buying Hugging Face.
What's Hugging Face?
Well, Hugging Face is essentially where all the AI models are stored, from
cloud-based models that can be deployed up to a trillion parameters, all the way
down to what can run on a Copilot+ PC.
If you want to find it, there's a little slider bar that says, "I have this much
memory," or, "This is the kind of device I have," and it'll tell you essentially
the model that you can download.
It's literally where… It's the App Store of, models.
And it's super interesting that NVIDIA, I always say is essentially the company
that's selling the shovels for you to go out and mine AI, is also the company
that's now becoming the claim where you come in and tell, the AI where it is that
you've, found AI where you can go get it.
hey, Neil.
We were just talking about, NVIDIA buying Hugging Face.
Let's, see if anybody has any opinions towards this announcement
Jacob: The discourse online has been pretty rampant about this.
Kind of felt like it was a matter of time.
It seems like, people are generally positive to Nvidia.
I think it's a better option than some of the other model providers, and
model creators because Nvidia has been moving in an open source direction
already with some of their in-house models, Nemotron and things like that.
So I think it's par for the course.
Probably a good thing, but that's my take.
Robert: Does it dilute their focus though?
I mean, I don't think Hugging Face makes money,
Chauncey: Yet.
It doesn't make money yet
Jacob: I heard they actually do make money.
They do cover their costs, but they have a ton of investment from a
whole bunch of different areas, so that's not their focus, I don't think
Chauncey: do they just make money?
'Cause that might be also the other thing, right?
Frankcx: I think it's a double down, right?
I mean, I think it's a double down into the idea of what an open weight model
is and why it's important to NVIDIA.
I mean, NVIDIA is known as the company that essentially builds cloud-based
AI, but they're also, with their announcements and deployments of DGX
Spark and RTX Spark coming, you know, the idea would be is that now you have
models that can run wherever you want intelligence to be, whether that's on
your device or in your data center.
And Hugging Face is the tool that deploys those different solutions.
So to me, it aligns really well with what NVIDIA is doing.
But, you know, I think, time will tell if they change
Jacob: Well I
mean, there's just a risk they lose their users, like the
community forks and like…
Chauncey: Yeah
Jacob: I think they've just got the users and the models,
and so that's like the market.
Neil, you were gonna say something?
Neil: I saw an analogy that NVIDIA's chips, right?
These are the ovens, and Hugging Face, they have the library of recipes,
and if you control the oven and the recipes, you can start, really
delivering outsized value to the market.
I think it's also important to note that this isn't, NVIDIA's first attempt, to,
Chauncey: Hmm
Neil: they actually invested, in Hugging Face back in 2023, alongside
Salesforce, Google, Amazon, and IBM.
Last year, NVIDIA wanted to follow on with another 500 million.
Hugging Face said, "No, we don't wanna give outsized control to a single
investor." Potential hot take here would be, did the, breach of Hugging Face,
lead potentially to this acquisition?
Frankcx: Oh
Neil: need, we do need support from
Hmm
to,
Chauncey: Interesting
Neil: some of the security and, and maximize, the, the value
of this entity we've created"?
So couple talking points there.
This isn't, out of the blue.
NVIDIA actually has, quite a extensive history, including being on Hugging
Face's cap table, three years ago
Robert: Does that become a hotspot for the regulators if they
own the recipes and the ovens?
Chauncey: I mean there's still-- Yeah, but there's still a decent
amount of competition here.
Maybe not great competition, right?
But there are still other platforms out there.
So what grounds do the hawks have to stand on in this case?
Jacob: there's nothing preventing those users from going to another
Chauncey: Yeah
Jacob: And that's why I think,
Chauncey: Or even starting one
Jacob: especially for things like, uncensored models that
strip away some of the guardrails.
Like for that portion of the local AI community, I think they will go elsewhere,
away from the corporate entities
Frankcx: Well, leaning into what Neil was saying around they have a history
of doing this, I used to work for this company called Silicon Graphics
that was making big time, you know, graphics engines, and then NVIDIA
comes along and literally scales it so that they give it to everybody.
So it's like they have a history of finding out ways to scale their
business to bring everybody into the market, and not just somebody
that can own a data center.
So I think that that's a big play for an NVIDIA here is like their customer
isn't just the big giant data centers, it's literally you and me, and, you know,
anybody that can afford a 3090 card that's used or buy a new RTX Spark device, you
know, that's who they're playing for.
So I like it personally.
I think it helps our story, for local AI,
Robert: hopefully it'll
Chauncey: fingers
Robert: the prices of RAM down
Frankcx: Right.
Robert: Maybe they can
Chauncey: Mm-hmm.
Frankcx: You're right.
Yeah, I mean, that is an interesting… Maybe that's a show coming up of like, is
local AI gonna take some of the heat and pressure off a data center deployment?
You know, like, how do we balance that?
But there will always be, in my mind, cloud AI, but, what level of it balances
in the future is up to be seen, So moving on to the next topic of news that I'm
bringing to the table today is around Qwen, which is our favorite Chinese model.
At least it's one of mine.
It's pretty capable.
It's got amazing intelligence.
It scales or quantizes, if I'm saying that right.
But the interesting thing that Qwen, it had released a model
3.8 a little over a week ago.
We benchmarked it.
But it also released last week or this week this interesting model
called Qwen3.8-Flash.NEXT, which is actually a preview towards Qwen4.
But it has some really interesting memory capabilities, and I know Jacob, you've
played around a little bit with this, so maybe you can kind of tell us what you
Jacob: I'm very, very passionate and interested in just like experimentation,
'cause at this point I'm just learning, trying to be a sponge as much as possible.
So the moment that it dropped, I had my agents working on the best way to figure
out how to install it, 'cause I knew that it was 170, 180 billion parameters,
and I was like, that seems like too much for what I can fit on a consumer card.
it seems like it's just too big.
I've got that external 3090 that I was thinking about running it on.
And, I realized that there's a lot that I'm still learning, about this lookup
table, these Ngram tables that are essentially like available for being
able to… Well, they just don't need the same availability as the rest of the
model does, is the way I think of it.
They can use slower memory, they can use system RAM, and I heard some
folks in the community talking about being able to use SSD storage for it.
so that was what was cool for me, and I had some success with running the
bulk of those model weights on SSD in the Ngram tables so that it could look
up and then a small amount on VRAM
Robert: Now, is this an MOE model, Frank?
Is it a mixture of experts model
Jacob: it is an MoE model technically.
I think there are 6 billion parameters that are active at a time.
I'll have to double-check.
And then there's 51 billion parameters that are in that n-gram table, and then
there's another 120 are available that should be on some fast RAM as well,
but it doesn't need to be VRAM because they're not active all the time being MoE
Frankcx: Yeah, it feels like even though you look at the size of the model and
you're like, "That can't run on my device," the way it loads the portions
of the model that you need are kind of in the chunks that you're asking
for, not like everything all at once.
So yeah, it's a,
Robert: Which is the whole premise of a MoE model, right?
It's a combination of a bunch of smaller models together,
Chauncey: Mmhmm
Robert: and it could pull from any one of them.
there's a component that lives in there that says, "Oh, let's use this
model for this task and this model for that one," which is awesome.
It's super cool
Jacob: Yeah, I think there's something like
Frankcx: that,
Jacob: 512 experts, and each one of those experts is a certain size in this model.
Frankcx: Right
Jacob: I'm still learning, but it's very interesting.
Frankcx: I tried to run it on the CPU on my device, which is a Xeon-based
device, and it was getting like four tokens per second, which is like
barely… Like, if you think of a token as like three quarters of a word,
that's like somebody is slurring their words, like they're having trouble
ran it on the 3090, it got up to 50 tokens per second, but because
it was just chunking along, it had trouble finishing its tasks.
So, there's definitely some tuning that has to be done.
I think in the end, what I found is it's not built for the 3090.
It has this weird lookup table that's 26 gigs and, a 3090 only has 24 gigs
of VRAM, so it just doesn't fit right.
It'll be interesting when, RTX Spark or DGX Spark or, other devices that ha- Like,
Neil, I'd be interested if you tested it on a Mac that has unified memory, how
that would run, because it seems like it's more geared towards that type of hardware
than it is something like a 3090 card.
Neil: Got my homework for next week, so
Frankcx: All right
Neil: received some feedback folks liked the, our ability to translate
technical topics into business terms.
I think pausing for a moment on Mixture of Experts, all the rage right now.
Maybe this analogy fits, maybe it doesn't.
I like to view Mixture of Experts as, analogous to the Encyclopedia Britannica
bookshelf at my grandparents' house.
You've got
Frankcx: Oh, right
Neil: of all
Chauncey: All right
Neil: topics.
If you know
Frankcx: B through C-A
Neil: You don't need to open all of the books at the same time.
So, just, again, wanna pause for a moment, define Mixture of Experts.
Frankcx: love it
Neil: this is going to allow us to leverage larger parameter models
without, running into some of the memory bandwidth constraints that we've
seen with, previous, models before Mixture of Experts became, top of mind
Robert: And I'll share some stuff later on,
Chauncey: Oh, nice
Robert: this week with MoE models
Frankcx: Oh, nice.
Neil, given that you're talking about unified memory and kind of memory
management, I think you've been doing some stuff with Apple as well.
So what can you tell us about some of the Mac stuff or the news
that you're excited about for Mac?
Neil: So big announcements this week from the Apple world.
They announced M5, M6 processors, Mac Minis, which became all the rage with the
OpenCL moment earlier this calendar year.
We're continuing to see Apple double down on this local AI moment.
The twenty twenty-six was supposed to be the year of agents.
I think one can make an argument that it's becoming the year of local and hybrid AI.
Um, Apple continues to, um, innovate at the silicon layer, uh, dropping
down to a two you look at some of the models like, Kimi V3, right, and three
hundred and fifty gigs, you see almost a direct parallel between the, the
type of silicon and, and these types of MoE models that we're seeing released.
Ran some tests on an M3 Max, versus the thirty ninety.
So Frank, thanks for sharing those benchmark tests.
I, I think it boils down to what are you trying to get
out of these local AI models?
Ran those five scenarios, and the consistent takeaway was anything
that's bandwidth bound, memory bound, Mac is outpacing the competition.
When it comes to raw compute, NVIDIA's, you know, taking the cake, right?
Apple is two to six X slower.
So again, that's a, a couple generations back, but you're seeing Apple continue
to double down on these, bandwidth, memory bound constraints, and you're
seeing NVIDIA to, you know, continuing to outpace the competition when
it comes to, to raw compute power.
So it'll be interesting to see the shift in paradigm and, again, lots of,
excitement for Mac enthusiasts, this week.
Curious any other takes from the crew here on, Apple's recent announcements?
Chauncey: I mean, I would say as a, fellow hardware nerd and just tracking
these other guys, like, this is only just beneficial to the entire ecosystem, right?
If Apple's going this way, Windows has to follow.
Like, as an ecosystem, of course, if you think about Surface and Dell and
Lenovo and everything else that's out there, like, it's so critical that
we all continue to match this space.
So I'm always excited when new things come out because that means that there's
gonna be new things on the horizon for everyone else and we get some, I mean,
it's just gonna continue to get better.
Just imagine the amount of power we have now and what we're able to achieve,
which all you guys are talking about so far of, like, testing locally.
What we'll be able to do in just a year or two years it's
just exponential from here out.
Robert: I
wouldn't exactly say Windows is following, right?
But just to be clear, I mean, there's some pretty amazing things that are
happening in the Windows ecosystem Look, Neil and I have had these conversations.
It was just cloud, then it was cloud and device, and now it's cloud, edge, and
device in sort of a three-tiered model.
once you get to the phone, you
know, bets are off
Frankcx: I think the thing that I see is, look at what happened this past week.
Two things: NVIDIA looking at Hugging Face, that's a double down on local
AI and open-weight models, and then Apple, another multi-trillion
dollar company, doubles down on local AI themselves by releasing
these Macs that are in that space.
So I would argue, and I'm not with Microsoft anymore, but I would argue
that the governance that Microsoft is gonna bring to these local AI spaces, I
think that's why a lot of people should be excited about what RTX Spark could
bring, 'cause you're bringing this capability of what CUDA and speeds are,
but you're bringing the governance of what you need, because many enterprise
companies are scared to death of running,
Robert: And they should be, yeah.
Well, not Linux, but
Of running agents for sure
Chauncey: Yeah
Jacob: I was talking to a customer
Frankcx: being able to have that governance that Microsoft can bring over
top of what local AI can, bring from an intelligence standpoint, there's some
exciting things just around the corner
Jacob: I was talking to a company in Japan this week, and they were pretty frank
with me, like: "Hey, right now we can't run agents at all in our organization."
And they are very AI forward.
So AI forward, they actually have a partnership now to run local
models on all of their laptops.
So they're running local models everywhere, but those local models
are not oper-- They're-- it's all just connecting to the cloud or to
specific outputs and not agented tasks.
So they're like: "We want to run like Scout or things like OpenClaw or, you
know, a whole host of other agents." And, I think there's a continued story
that, Windows will continue to add value with Microsoft Execution Containers,
Agent 365, all of that goodness.
Frankcx: Well, you guys will get a kick out of this story.
You may not know this, but I've been playing around with doing a little bit
of Uber driving myself, just on like a,
Robert: In the sprinter?
Frankcx: I want, I wanna get down into Cap Hill and see what the
cool kids are doing, so I'm driving them around, dad in his Volvo.
I picked up this guy, and we had this conversation, and I'm like: "Hey, what
do you do?" And he's like: "Oh I run an aerospace company." I'm like: "Oh,
that's pretty cool." I was like: "Are you guys using much AI?" He's like:
"I use it to write emails, but I can't use it for engineering because we're
doing contracts with Boeing and the defense side that I can't share the IP
of what we're doing with the cloud."
And I was like: "Well, do you know much about local AI?" And he's like: "Well,
tell me." And I'm like, here's this Uber driver telling him about local AI.
He used to be in this band called Fifth Angel, and he was the lead
singer for it, and it's this Seattle hair metal band that came out
Robert: may- maybe his next band will be called Air Gap
Chauncey: Ooh, nice.
That's good.
Robert: Hair, you got a
Chauncey: That's good.
Robert: gap
Frankcx: gap.
Chauncey: Robert coming up with the band names, man
Frankcx: Yeah.
Does anyone else have any news they wanna share?
All right, if not, it feels like we should do a baseline.
Baseline to me is like, Chauncey, I know you're interested in making sure we're all
talking speak that everybody understands.
Maybe you can give us a little bit of a clue
Chauncey: This is also referred to as the section of the podcast where
it's like, I have no idea what you guys are actually talking about.
Can you please explain it?
So can you do a couple things?
One, I know some of it.
Like what's the difference when we're-- Well, first of
all, like what is quantization?
So you kind of mentioned quantizing earlier.
Like, I think it's probably really worth diving a little bit into the
details of what quantization means.
And then I would actually love to know your thoughts on the
difference in cost per token and how we're thinking about that too.
Frankcx: Yeah, I mean, I hear a lot of people talking about, the $25 token or the
$6 token or, like, you know, what is that?
Nobody's paying $6 for a token.
Well, it comes down to it's $6 per million tokens.
And what the funny part is that this past week, you know, Grok, which I think is
Neil's favorite AI, like it's, I called it last week your drunk uncle, but your
drunk uncle sobered up and became a pretty capable, model at $6 per million tokens,
which honestly is like a fifth of the cost of what, OpenAI and Claude charge
for their, frontier models at $25 a token.
But I think the point is, is that because of the ability to start
running these models locally, it's starting to drive down, you know…
And when Grok comes out with a $6 per million, you know, token cost, that
really makes everybody else look at it like, "Hey, what are you doing
there? And how come you're not charging as much as we are?" Well, you know,
they're not the premier, you know, AI, but they are showing themselves
to have just as much capability.
And Neil, maybe you can talk a little bit about Grok, but it'd be like, you know,
it feels like there are models out there, because they're all sharing algorithms,
they're all becoming kind of equally smart
Neil: Yeah, it's becoming more of a commodity value-based discussion.
The days of unlimited tokens are over, right?
As we mentioned, currently work at Microsoft and we have set token limits.
And so continuing to route every prompt through frontier models, Opus 5, GPT
Sol 5.6 no longer becomes practical.
So, shifting some of the development needs to Groq 4.6
stretches my token budget further.
And if it can do, as good or close to as good of a job, if I can get more
done with my existing token budget, that's why, Groq, the drunken uncle,
that sobered up, is now becoming, one of my favorite models to work with.
not all tokens are equal, dependent upon which models you choose.
And I think Groq offers, great value at this moment in time
Chauncey: I'm really interested to see where we go with, model
choice and making it easier.
I mean, like the average person-- Your developers, obviously, they're gonna
know, like, "Oh, I should probably choose something cheaper," and whatnot.
But like the average person's just gonna go, "Oh, this says it works better.
I'm gonna click on work better." Right? And, but ultimately, like I
would say companies would value either Microsoft or whoever making this kind
of orchestration layer that says, "Hey, for sure, I'm building a PowerPoint,
I'm gonna have this level of model.
But if I'm just answering a simple question, I should be able to like
dynamically switch to another model."
I think that's where I'm really excited to see some of this happen.
And I know we have some auto settings in, Cowork and in
Scout, but not quite there yet.
It's still a pre-selected handful of models versus if it can really
start dynamically going across whatever your organization says.
Like, I want only these six models to ever run.
I think that's when we're gonna start seeing some pretty cool things happen.
Robert: it goes down a whole other layer that general
users don't understand, right?
Chauncey: Well, yeah.
Yeah
Robert: know, sonnet or whatever, then there's like medium, max,
Chauncey: Yep.
Robert: extra large, extra max,
Frankcx: Right
Robert: big max,
Chauncey: Plus
Frankcx: wondering, to bring it back to Chauncey's original question
around quantization, I'm wondering who can take a shot at that.
And when you quantize something, does it make it less
Jacob: Well I'll do my first pass, and then I'll get corrected by Robert or
Neil, is probably gonna be my choice here.
The way I think of quants is I think of it like any other compression.
Like you're making the model and the weights take up less space,
a result, you are potentially cutting some of the detail.
Just like when you have a JPEG image that's compressed, you may
be cutting out some of the detail.
It's doing some calculations of, "Hey, we're grouping these pixels
together. They look the same." It's saving you space on disk.
It's saving you space in RAM.
And so if a model like was a seven billion parameter model, like its full weights
would probably be like double that.
They would probably be like 14 gigs.
if you, compress it down to Q8, then it might be seven gigs.
So it might cut the space in half, that's kinda how I think about it, is like you
may be trading off some precision, you may not be able to trust the outputs in the
same way, in a world where you have models where you're running them to do a task,
and then you have another model checking its work, then maybe it makes sense.
Chauncey: So
Robert: Yep,
Chauncey: it
Robert: The trade-off is accuracy.
How do you
Chauncey: Hmm
Robert: more, you know, tighter without losing the accuracy?
And that's the trick that goes with it.
And look, we're even seeing some models get down to one-bit
quantization, which is insane.
But that's where the technology is going.
But that, that's always, as I understand it, that's the challenge is how do you
make it and use less, but still maintain the accuracy and the credibility of the,
Chauncey: S-
Robert: the answers you get?
Chauncey: how do you choose then?
Like, if I have this option of a frontier version that's gonna be obviously the most
powerful, the most intelligent, and like, that's kind of like what I would think I
would want all the time versus something that is more efficient, that is gonna be
smaller, that I can maybe run locally.
Like, if there is kind of that more gut reaction of just like,
well, then how do I choose?
Like, what would be your kind of answer then?
Jacob: I would say the highest quant that you can the space that you have.
So like the benchmarks for like 3.8, 27 billion parameters that we talked about
last week, and, I think Robert, you might have some more to share about this week.
Like those benchmarks were generally ran at full precision, like no quant at all.
So you want it to be as precise as possible, but there's a
trade-off in speed and efficiency and what can physically fit.
That's my take.
Frankcx: Yep.
Well, getting to that, I think it's time to go into the tension of the week, and
I know that Robert, you said you'd be willing to be on the hot seat, so I'm
Robert: Why not, eh?
Give it a go.
Frankcx: it
Robert: I'm always anxious to get into the fight.
So listen, we've been talking about AI running it locally.
Now we can run it ourselves.
The models keep getting better.
it, just obviously saves money than using the ones… But the big cloud
models keep getting cheaper as well.
so the big question is: where are the AI companies eventually
gonna get their money?
What are they gonna charge for?
Wendell at Level 1 Tech says the answer is speed.
Not smarter answers, just the same answer, but faster.
the example I used, earlier off air was, you know, I'm gonna order something
from Amazon, it'll come in two days.
If I want it in a day, I'm gonna pay, two ninety-nine to get it.
So we're already starting to see it, in the AI world.
If you wanna wait, you pay less.
If you want it right now, you pay more.
Same AI, just a different clock.
And so, you know, the twenty-five dollar, token is dead, right?
I think within 18 months, the only thing that frontier labs are-- really
gonna be able to sell is the speed, and not so much of the intelligence.
I experienced some of this this week, in loading, a thirty billion
parameter model, onto one of my devices, and I was blown away, at the
speed and the accuracy and, just the capability of it and how far it's come.
So, in plain English, soon being smarter will not be enough to
charge more for, you're gonna have to be faster, and that's my take.
Frankcx: Yeah.
I mean, you know, for me, I think of it as like Neil's… He's Mr.
Analogy, so I'll bring in the analogy of like, to me, like a cloud AI
is like hiring a consulting firm.
They can bring in a lot of people.
They can do a lot of work.
They can scale very well.
They're expensive.
They don't know you very well, so they have to take the time to learn
you, you know, what you're doing.
if you think about local AI, could literally go out and hire
an MBA from business school and bring them in and train them.
They're smart.
They're just as smart as who you could bring in from a consulting group,
but they just can't do as much work.
They don't have the same volume that they can do.
You could hire more and more of them, but then you have to take
on the responsibility of managing them and taking care of them.
for me, it's this, type of analogy that the AI is getting smart enough
that you can run it locally, and then you're able to control privacy.
That person isn't gonna take that information and share it with
some other consulting agency.
It's sovereign, meaning that, it's controlled as to where
it goes and where it lives.
It's local, so if something happened, like, there was a storm and, things got
knocked out, that consulting company that, flies in every week would not be
able to get to you, but that business person can come to work that day.
So there's a lot of advantages of having local.
But that doesn't mean that the cloud's ever gonna go away.
It's gonna be there for when you need it and the tough problems.
But I would argue that AI being balanced is the way we're
looking at AI in the future
Chauncey: To your point about, the analogy too, going back to speed,
a bigger team of individuals may or may not come to that answer faster.
We can maybe throw more people at that project, so you get that ability
to get either a more cohesive answer or a faster answer, because of
that ability to throw more at it.
Is that kinda the mindset here?
Maybe?
Frankcx: Yeah, I would agree
Jacob: I don't know if I believe the assertion fully, Robert, because
Robert: not say it with enough conviction?
Chauncey: Ding, ding, let's get it on
Jacob: I feel like everyone's just trying to acquire users, and so it's a race
to the bottom temporarily while they're trying to prove that their models or
their values, is where the user should be.
it's to the point a couple weeks ago, there's this Ox Alpha, like kind of
like a stealth model on OpenRouter, so you can choose if you're, running
a cloud model via OpenRouter, you can choose this Ox Alpha and it gives you a
anonymous model that you don't know what it is, but it's free to run and use.
So you're, using a model, you don't know what it is, you don't have any trust.
It's in the cloud, it's not local, but, it's free to use.
And so they're trying to acquire
Robert: Model roulette is what that's called.
Jacob: What's that?
Robert: That's called model roulette.
Jacob: Exactly.
Robert: Good Lord
Jacob: but everyone wants users, so, like, they're gonna continue lowering the cost,
but I don't think that that's sustainable.
I'm not into the economics of it with these companies, but, like, it just
doesn't feel like that's sustainable.
At some point, they're gonna, like, entrench that they've got their
users and the prices go back up
Neil: I'll offer a perspective in terms of speed.
Robert, the argument that speed is going to be what these frontier labs
offer, how does that shift when we get into asynchronous workflows?
If I can craft my prompt to build the code I need overnight, why would I
pay a premium for a frontier model?
Why would I pay the outsource consultant, leveraging the example?
Yes, maybe my MBAs take, much longer to execute that task, but if I can set
them up proactively and achieve the desired result, certainly under some
time constraint pressures, yeah, maybe I need to outsource and execute a sprint.
Does speed become less of a factor when we start talking about,
orchestration and autopilot workflows that can accomplish work when
Robert: But is,
Neil: lid is shut?
Robert: is that a builder perspective or is that a general
everyday user perspective?
Neil: I would say right now it's a builder perspective.
I'm seeing AI from a knowledge work perspective, a more one in
one out, a more sophisticated Ask Jeeves, Google search, if you will.
Once we start talking asynchronous workflows, which developers are, are
Robert: Yeah.
Yep
Jacob: Yeah
Neil: Right to that,
Jacob: You
Neil: think it's the developer workflows where speed doesn't matter.
I'm used to crafting a prompt, letting my device cook for 10 minutes, 15 minutes.
My benchmark test took an hour and a half this morning,
Hmm Hmm
acceptable.
So,
Jacob: Yeah.
It's like,
Neil: may not matter for developer
Jacob: Frank, if you're tossing, one of those tasks like, you know, build an
app, build whatever it is that you're building these days, like your four tokens
per second example of that, 180 billion parameter model that you're running.
Like, could theoretically just say, "Hey, I'm gonna be gone for the
weekend," like build me something
Frankcx: Right
Jacob: come back and you've got something that's smarter and better and free so
Frankcx: Especially these days when you're saying, like, you're kicking off
seven different Claude Costa sessions and you're like, "Well, just go to
Robert: Fleet.
You've got a fleet, yeah
Frankcx: You're like, "Can you keep working on
Robert: Does,
Frankcx: please?"
Robert: Does this equation change based on what we saw from NVIDIA and Hugging Face?
Now, maybe not in that exact scenario, but thinking about
consolidation in the industry does that change your on any of this?
Chauncey: Wait, so just kind of rehashing that thought.
Does NVIDIA potentially acquiring Hugging Face affect this speed versus like
They're only gonna
be building for speed
Robert: Yeah, but not that specifically, but just in
general industry consolidation.
I mean, look at all the model providers we have, the hardware
manufacturers, the data centers.
Like for example, Microsoft makes models, build data centers, they provide services.
I mean, are we gonna see more of that?
And is that gonna affect your answer?
Frankcx: there's some tools out there that are like these routers
now that can essentially look at… They're like intelligence routers.
They look at the question that's being asked, and then they determine what
model or what tool, this can run on.
And I feel like within an industry, if you had your own ability to route to
local because you knew that job that you were requesting based on the prompt
could be handled by a local AI or a cheaper model, then you would, right?
So I feel like router, types of, intelligence is gonna be involving
local AI in the future as well
Robert: And does
Chauncey: Hmm
Robert: equation change, I'm gonna throw another curve ball
here, with a three-tier model?
If you have your device running a local model, you have a server a
server or blade in your house that can run close to a frontier model,
that change things even further?
Chauncey: from a revenue perspective, like
Robert: a, from a, what are they gonna charge for?
Chauncey: Yeah.
Yeah, well that's, that's, yeah.
Yeah
Robert: I could probably get some pretty serious speed off of it
Jacob: You're,
Chauncey: I mean it's, wonder even, like, how we ended up making
a decision, if you think about even analogous to Office, right?
At one point, we had Office running locally, and, everyone questioned why the
heck we moved, we being Microsoft, moved to going to a cloud subscription-based
model for, and now that that subscription model is king for everyone, right?
That is not just, a Microsoft thing.
Subscription is king across the board.
And so how do we start looking at models in that same fashion?
Are we reverting?
I don't think so.
I think there's still gonna be a play of individuals that are gonna want access to
the newest, the latest, the greatest, the fastest of what the cloud has to offer
and to that provides that scale, right?
Frankcx: Yeah.
Chauncey: where can the cloud grow?
It can grow
Frankcx: Yep.
No, I think that's another show, because I feel like there's this
whole idea of, like, butts in seats equals subscriptions times the number
of people you have, and that doesn't equal an AI model when it comes to
Chauncey: Right
Frankcx: But, hey, let's, let's do a little vote around the horn.
I'll jump in first.
So I'm gonna support, Robert's, theory that the $25 token is, a
thing of the past, especially since Grok and others are already leading.
I don't know what that means towards the economy and all the investment
banking that takes place around AI.
Hopefully they can pay their bills, but, but that's the way I feel about it.
Jacob?
Jacob: I think that it's gonna get expensive again before it gets cheaper.
Local is here, we all think it, but I think the market's not gonna be fully
there and the costs are gonna go up again, especially as licenses, like open
source licenses are in the mix of like what local models are able to do legally.
I think that some of the Chinese AI firms are going to… Well, we're seeing them
add more, and not as open of licensing.
So you're gonna see more commercial customers want to-- they'll be paying
more for it because it's in way higher demand before it goes down.
So I don't agree with you yet, Robert, yet.
Frankcx: All
Robert: On the side
Frankcx: Chauncey?
Chauncey: Yeah, I'm on the same boat as Jacob.
Demand's gonna be high enough to continue to maintain a higher price for now.
Robert: Must be in marketing
Frankcx: Yeah, they want the
Chauncey: Or, yeah.
Frankcx: And then, Neil
Neil: Speed matters when we're talking security.
If we have bad actors using these frontier models, we need access to
these frontier models to combat.
I'm going to agree with the stance that frontier models deliver speed.
The use cases for speed vary.
I think there will continue to be a premium for frontier models
specifically to combat rising threat actors leveraging the same models, us
Jacob: And people will pay for that, all right?
Frankcx: All right, Robert, you get the last word.
Robert: I'm
Frankcx: know where your
Robert: I'm holding firm.
Now, what we might see is different tiers of speed you pay for.
I think regardless of where it goes, and I do think speed is gonna
be, the first, shoe to drop here.
Over the next six to 12 months, I think we're gonna see some significant changes
in how this looks, is good for us 'cause it gives us stuff to talk about
Frankcx: Yep.
right, let's, wrap up with on my device.
Anybody have anything cool that they're working on?
Robert: Here's the short story.
Frankcx: problem
Robert: week I talked about my, struggle to try and find out how we can get,
on-device coding, a code agent running.
And I of tripped into it or backed into it by mistake.
The short version of the story is I loaded up a twenty-seven billion
parameter model, connected it in, thank you Jacob, into Copilot CLI.
I typed in, "Let's make something," and then I got distracted.
I turned on autopilot, which essentially gave it full, permission to do
whatever it wanted, and I walked away.
And I came back, and it said, " Well, you didn't respond in ten minutes, so
I built you something." And it built an online coding agent, and it said, " You
should use a-- this MoE thirty billion parameter three point eight Qwen
model." and, and the rest is history.
There's a lot more to it, but it
Chauncey: Oh, that's awesome.
Frankcx: All right, gentlemen, I hope you all have a terrific weekend.
So is no place like 127.0.0.1.
Robert: hang with you fellas
Frankcx: now
Chauncey: See you.
Jacob: Thanks, team
Chauncey: Thanks guys