Subscribe
Share
Share
Embed
MIT’s Manya Ghobadi on AI Economics: Why Token Pricing Misses the Point
Host Dan Mitchell interviews Manya Ghobadi, MIT CSAIL associate professor and founder/CEO of Systalyze, about making AI practical, efficient, and economically viable at scale. Ghobadi explains that AI “models are just math,” with prompt cost driven by model architecture and matrix multiplications, and offers a simple view of LLMs as next-token probability predictors. She argues efficiency should be a first-principles focus, critiques token-based pricing as a flawed proxy for value, and advocates resource-based thinking centered on GPU requirements, deterministic performance, and self-hosted open-source models for enterprises. Drawing on conversations with over 1,000 Fortune 500 AI leads, she notes organizations struggle to measure ROI and maintain efficient GPU clusters as models and libraries change. She contrasts academia’s long-term innovation with industry’s resource-driven progress, calls agentic AI overhyped, and highlights performance and energy efficiency as underestimated, urging listeners to challenge the status quo and choose tools wisely.
00:00 Mainly AI Episode 3 - Models Are Just Math: The Economics of AI at Scale
01:26 AI Is Just Math
04:59 The Economics of AI
14:14 Academia vs. Industry
21:34 Most Overhyped AI Trend
24:19 Most Underestimated AI Trend
27:25 5-Year Prediction
Creators and Guests
What is Mainly AI?
Welcome to Mainly AI — the show where artificial intelligence meets real-world impact. Join your host Dan Mitchell as he and guests explore the latest breakthroughs in AI and other emerging technologies, and how they’re changing business, science, and your everyday life. From startup founders to academic minds, we bring in the experts shaping the future — and break it all down in a way that's smart, sharp, and seriously fun. There's nothing to fear. AI can help humanity in so many ways! It's an exciting time to be alive.
Dan Mitchell: Welcome back to Mainly AI.
I'm your host, Dan Mitchell, and
today we're diving into a side
of artificial intelligence that
doesn't get nearly enough attention.
While much of the industry is focused on
building larger models and chasing the
next breakthrough capability, today's
guest is focused on a different question:
do we make AI practical, efficient,
and economically viable at scale?
Joining me today is Manya Ghobadi,
associate professor at MIT's
Computer Science and Artificial
Intelligence Laboratory, or CSAIL.
She's the founder and CEO of Systalyze
and a researcher whose work spans AI
systems, cloud infrastructure, networking,
and machine learning efficiency.
And before joining MIT, Manya
worked at both Google and Microsoft
Research, and today she sits at a
unique intersection of academia,
entrepreneurship, and enterprise AI.
Manya, welcome to Mainly AI
Manya Ghobadi: Thank you for having me
Dan Mitchell: Absolutely.
So you once told me that
models are just math, and that
statement has stuck with me.
What do you mean when you
say models are just math?
Manya Ghobadi: the way I think
about AI models is different than
the good old database systems.
The complexity of a database query
is determined based on how, what
the operation this query requires.
If it's a simple select or
if it's a joint operation.
To me, AI is flipping that.
The complexity of an AI prompt is
actually not dictated by what is
the prompt, it's actually what AI
model that prompt is going through.
So think of it like this.
Who is the president of the United
States versus what is the solution
of this differential equation?
If they go through the same model, they
are going through the same architecture,
they're going to go through the same
mathematical matrix multiplications.
And so to me, AI is just math is because
AI is dictated by the architecture.
Underneath those architecture
is a series of algebra, mostly
dominated by matrix multiplication.
And so it's almost like a flip in the
world of old systems versus new systems,
where before the complexity of the
prompt or the query would determine
how long this query is going to take,
whereas here is the complexity of the
underlying AI model that's gonna determine
how long this query is gonna take.
This is very loose, but this is
sort of foundationally accurate
Dan Mitchell: Okay, so for someone
who uses something like a ChatGPT or
a Claude every day, what's a simple
mental model that they can think about
for how an LLM works in the first place?
Manya Ghobadi: Mm-hmm.
LLMs are basically predicting a next
token, or let's be more precise, the
probability of what the next token should
be based on all the previous tokens.
And so you've probably seen that when
you type something in ChatGPT, it's
just typing, like, word after word.
It actually is predicting token
after token, and a token is
basically a series of characters,
like say three to four characters.
It's not actually a word.
We do see words, and we speak in words,
but for an LLM, it's all about a token.
This is the unit that The
model is basically predicting.
So ChatGPT or Claude, the way they
work is, given a series of previous
tokens, let's say the cat would
like to go to, then there would
be a series of predictions and the
probabilities associated with them.
A box, the fence, red, book,
and then probabilities.
The word or the token with the
highest probability would be
predicted as the next token.
The cat would like to go to the box.
Maybe box has the highest probability.
Then that becomes the
next context we call it.
Then the LLM would predict the next token
and the next token and the next token.
So it's a probability of what
would be the next token given
the set of previous tokens.
Dan Mitchell: Got it.
Thanks for that.
Yeah, a lot of us take it for
granted that it just works, so if
AI is ultimately math, then every
interaction requires computation, right?
So is that where the
economics of AI begins?
Manya Ghobadi: Precisely.
So how do you do these predictions?
We said matrix multiplications and some
mathematical optimization functions.
Okay.
GPU is now the perfect architecture,
the perfect hardware, because it
was originally designed for gaming.
And in gaming, you also have a
lot of matrix multiplications.
And so it just happens that this
is a perfect hardware for this
particular, we call it a workload.
And so, yes, what drives the economy
and the cost of when you type
something into ChatGPT and you get
charged is on where is the model.
The model sits on a bunch of GPUs, and
these GPUs are expensive, but they're
performing the mathematical matrix
multiplications very quickly for you
Dan Mitchell: So, you know, we
spent the last few years talking
about AI capabilities, right?
This one can code, this
one can make images, right?
But it feels like now we're talking
about AI economics, and you hear
people use the term tokenomics.
are we entering an era where efficiency
matters as much as the intelligence?
Manya Ghobadi: I would argue that
efficiency already matters, we should
think about efficiency from day one.
I've heard tokenomics before.
It is an artifact of the
charging mechanism that
frontier labs have invented.
Token pricing.
Think about it.
Why are we paying per token?
Why aren't we paying per request?
Why aren't we paying per number of GPUs?
What drives this concept of
if the LLM model generated 20
tokens, I should be paying, 20
tokens worth to a model provider?
What if the model is not clever, and
it takes multiple prompts for the model
and multiple responses for the model to
reach the high-quality response to me?
Why should I be paying that?
So I think we should have started
thinking about efficiency from day
one, and we shouldn't necessarily
submit to this notion of everything
should be measured by tokens and
everything should be priced by tokens.
Even if you put that on the side, even
if we accept with the token pricing is
the right way of advancing AI to the
next level and having everybody on the
planet to use AI, there is not enough
resources for the entire world to be using
AI and embedding AI into our day-to-day
tasks without really even thinking
about optimizations as first principles.
Right now, the GPUs, there's
shortage on hardware.
There's new models that are
coming every day, new use cases.
If we don't even think about
efficiency right now, we're
already losing the AI game
Dan Mitchell: That makes sense.
Iâ¦
Talking to different leaders about
their token economics, normally they
just point at their API bill, right?
And usually they've got a kind of
overwhelmed look on their face when
they do that, and, it's part of,
like, that bigger picture of how do
you get AI and be able to do the ROI.
So, as part of building your research or,
and building your company, you talk to a
ton of people about the problems they were
having, trying to solve with AI, right?
And so organizations have invested heavily
in AI infrastructure, especially as an
alternative to the endless consumption of
these frontier model tokens in the cloud.
How do they know they're getting
the most value from those
investments that they've made?
Manya Ghobadi: The sad reality is that
they don't And I want to solve that.
That's a very deep
research interest for me.
On given a particular use case,
let's say an internal ChatGPT,
there's an organization that is
interested in building an internal
ChatGPT system for themselves
or a document processing system.
What should be the investment
on AI, and what would be the
expected return of investment?
It seems to me right
now that we don't know.
I've had more than 1,000
conversations with AI leads across
many, many Fortune 500 companies.
And you're right, everybody starts by
saying, "Yes, we have a subscription
to Anthropic and to OpenAI, and
the bill is coming, and hopefully
it's going to do something for us."
Every day, the bill can grow, and as
more employees adopt the tools, and then
they start thinking, "Actually, maybe
we should have a cap per employee."
And that actually already sort of defeats
the purpose of using AI company-wide.
So I'm completely against this
way of innovating with AI.
The way I think we should have started
with AI is let's deploy AI models, and
let's charge based on the GPUs, that
unit that is performing the compute.
And so let's embrace open source models.
Let's ditch token pricing.
It's not really a good model.
It just doesn't give you a way
of measuring what is expected.
The model can go crazy.
And hasn't it happened to you?
It happens to me, that the model is
just going crazy, and it's just likeâ¦
And I'm like, "Fin- end, end.
Don't even produce more tokens.
Don't want you anymore."
That just doesn't make any sense to me.
At a large enterprise scale, there
should be self-hosting open source
models and being able to leverage all
of these open source models and not
even thinking about token pricing, just
thinking about, "How many GPUs do I need?"
Maybe the GPUs are available through
cloud providers, maybe the GPUs are
available through an AI factory.
So obtain the GPUs, understand what
is the required number of GPUs for
a particular task, and then you
have a deterministic price, and you
have a deterministic performance.
That's what we are trying
to do at Systalyze.
We are trying to bring the mathematics
back to the calculations and say,
"Given a particular use case and
given a particular hardware, what
would be the expected performance?
How long it's going to take for this
particular model to be, producing
a response on this hardware?"
Like I said, the hardware can be
in an AI factory, the hardware can
be in cloud, the hardware can be in
somebody el- your neighbor's house.
That doesn't matter.
What matters is to be able to enable these
enterprises to think in a very rigorous
way of here's what I would like to do,
here's the required hardware, hence this
is the amount of money and investment
that I need to put into this one use case
Dan Mitchell: Yeah I think a little
bit about the airline industry.
they own these planes, and
owning the planes is not enough.
Their profitability is centered around
how effectively those planes are used,
how many people are on every flight.
But it sounds like AI is just not there.
They don't have it down to a science yet,
and they're still trying to figure it out
Manya Ghobadi: Yes.
It's almost like we're all chasing the
dragon here, and we get distracted by it.
We look at, okay, there's this
new model and it's like $27
per million tokens, it's okay.
There is no evidence that if a
model generates more token, it means
the model is more intelligent, and
we should separate those things.
It's very, very important to think
about AI model from a capability
perspective, not from how much garbage
or how much tokens they're generating
Dan Mitchell: Yeah, it's
definitely a, a good perspective.
you wouldn't just pay somebody to do a
lousy job they were working at your house
and they just continued to bill hours for
doing repairs and didn't do a good job.
So yeah, I think that figuring out how to
hold models accountable for their quality
and efficiency is an important step.
Manya Ghobadi: And that goes back
to the math because a modelâ¦
So a bigger model performs
more mathematical computations
than a smaller model.
So it is possible to argue and say,
"If this model is performing more
mathematical calculations, more matrix
multiplications, then you should be
paying more because under the hood
you would be needing bigger GPUs."
So there is a way to connect these by
thinking about the underlying resources
required to execute this particular model.
So these, these are loosely connected,
but simplifying it simply to the number
of tokens that have been generated is a
little bit of a disservice to the users
Dan Mitchell: Got it.
Okay, well, changing gears a little
bit, you're in a, a unique position,
as I mentioned earlier, and you're
seeing frontier research, you're
talking to your startup customers,
you're with MIT students all the time.
Who's moving faster right
now, academia or industry?
Manya Ghobadi: Perhaps the best way to
think about your question is there's
probably two sides of your question.
One is maybe think about where AI
was generated to begin with . Who
came up with, let's say, you know,
the original ideas behind AI?
And so maybe longer term ideas,
academia is still a better place
to innovate from that perspective.
Right now, LLMs are, the dominating
AI conversation, but maybe not.
Maybe LLM is not really the ultimate
architecture or type of models that we're
going to use for AI in the next decade.
What would be that next jump in
technology or architecture or design?
That's probably come-
gonna come from academia.
But because LLMs are very large and
running them requires a lot of resources,
and academia does not have those
massive resources of GPUs, back to our
conversation on the economy, industry
is making a lot of progress on larger
scale experiments and observations.
And there should be a world when
we are managing and merging these
two efforts together, and there's
a lot of, initiatives around this.
But to me, it's almost like most
immediate innovations and most
immediate observations that would
lead into innovations are coming from
industry, and my colleagues in various
industries are even publishing, so
they're contributing to the community.
But I think the next generation,
next jump in technology is
likely to come from academia
Dan Mitchell: Got it.
Yeah.
a lot of academic research
never becomes a company, right?
And your startup is a culmination
of multiple years of research.
So, what was the hardest part
of making that transition?
And was there a moment when you knew
Systalyze was solving a business
problem and not just a technical one?
Manya Ghobadi: Good question.
It came to me somewhere in just a couple
of years ago when I realized that even
my students are sometimes struggling
with keeping an AI model up to date and
always running and always highly efficient
and performant in our lab inside MIT.
So a little bit of backstory, I have a
couple of GPUs, some number of GPUs at MIT
for our research because our research is
about optimizing and efficient AI systems.
And every time that we would write
a paper, we would bring up the
system, we would run experiments,
and then, some other student
will just, take over the testbed.
And it turns out that between, like
even within a few weeks of running
this particular experiment, switching
to another experiment, or then a new
model is released or a new library
is released, I noticed that even my
students who are the top of the echelon
would sometimes struggle of keeping
this testbed highly efficient, and we
continuously have to maintain this.
That, coupled with a series of
conversations with members of the CSAIL
community called CSAIL Alliances, where
we presented some of our ideas and some
of these pain points, and there were AI
leads that were already thinking about
maintaining their own on-prem or cloud
GPU clusters, and they were sharing
the same pain points that every time
that there's a new model, every time
there's a new library, every time that
there's a new even open source tool,
it takes a while to put everything
together and to make sure that the
solution that I have created right now
on these GPUs and this GPU cluster is
actually as efficient as it can be.
And so it was realizing a pain point that
I've seen internally in my group and then
validating it with some early adopters.
Dan Mitchell: Got it.
Cool.
Cool, I love it.
so I guess one question I have for
you: How do you personally stay
current when AI is evolving so quickly?
what is your secret resource?
Where do you collect as much information
and learn as much as possible?
Manya Ghobadi: I'm constantly worried
about falling behind, and it's almost
humanly impossible to read every
blog post, every archive paper,
all of these conference papers.
There's so many breakthroughs that
I don't think I'm at the ⦠I
don't think I'm every day I
know e- exactly what's going on.
My secret weapon is that I'm lucky that
I have smart students, and they are
It's almost like m- you're
parallelizing your effort of staying
on top of the knowledge base.
So I think I'm a few days behind.
I still have to have the students to
absorb what's going on, and then the
students filter the key ideas, and then
through communications with students
is where I oftentime find out about
there's this new technique that was just
proposed, or there's this new model.
Did you hear that, like, say 9:00 AM
there's a new model that is released?
I'm probably in meetings, but then around
noon or lunchtime have a conversation, and
then I'm lucky that I get that information
very quickly through, through students
Dan Mitchell: So speaking of those
students, what do MIT students
understand about AI that enterprise
leaders may be underestimating?
Manya Ghobadi: Probably it
has something to do around the
capabilities of the models and
the opportunities on optimization.
Oftentimes, industry leaders tend to
accept that here's the model and here's
the price and here's how it operates.
Whereas the students, or in my
case, it's a combination of my
students and my amazing engineers
at Systalyze, I should add.
Most of them are former MIT students
or current even MIT students.
So I think the younger generation
doesn't necessarily accept the status
quo, whereas an enterprise leader often
probably is busy leading that business.
And so they would accept that, "Okay,
I guess this is how this model runs,
and I guess it requires this many
resources, and I guess this is the price.
Let's figure out how to
build a business around it."
Whereas engineers and students
would start by saying, "Why?
Is there a way for me to make it better?"
Dan Mitchell: Great.
Great.
Well, there's a lesson for the
enterprise leaders who are listening.
Take a lesson from some MIT students.
so let's get to industry.
What would you say is the most
overhyped AI trend right now?
2026, what is the most overhyped AI trend?
Manya Ghobadi: And by overhyped
you mean it probably does
not deserve to be overhyped?
I hesitate to make an precise prediction
at the fear of what if in a year that
becomes the next big, jump in technology.
But maybe I can talk about some of
the directions that I often hear about
that perhaps should be more grounded.
One is agentic AI, where everybody
talks about agentic AI, and it's like
overpowering all of our conversations.
And I think it's going to change the
way that we're thinking about the world.
I have no doubt about that.
But the details, like if you ground the
word agentic, and if you actually think
about what is ⦠every conversation,
what are we even talking about when we
say, "Let's agentify this application"?
I think there's a lot of nuances there
that sometimes even a simple Python script
that's automating a process that has
nothing to do with AI could be advertised
as here's the new agentic workflow.
And I think that's where maybe the
fallacy is to embrace the technology,
embrace the, the potential, but also
don't forget and be a little bit more
precise about when we say agentic,
what are we actually talking about?
Is it possible to over-hype it and say,
maybe, Dan, this conversation is also
an agentic conversation because you are
having a, AI companion taking notes.
And so you and I can go and
advertise this conversation as we
just had a fully agentic podcast.
I think that would be a little
bit of over-hype, right?
Dan Mitchell: that agentic AI is
definitely on the buzzword bingo
card, probably the free space in the
middle if I were to qualify it as
how much it's overused today, in the
industry and in, in the terminology.
And, and, to that end, I would say that
people are using it more for marketing,
and there are a smaller percentage of
people who can actually deliver agents.
but, such is the, the hype cycle, right?
It'll, it'll eventually come back down.
People have already moved
on to everything claw.
So, agentic will, will get grounded.
But on the inverse of that, what
is the most underestimated AI
trend that we should be looking at?
kind of break the news, you
heard it here first from Manya.
What is the underestimated
AI trend that's coming?
Manya Ghobadi: Let me think about that.
There's many
Dan Mitchell: Well, you
can pick three if you want.
That's fine too
Manya Ghobadi: I think something that
has been overlooked, and I can say
that with certain level of certainty
because I've spent the last, eight
years of my life thinking about this,
is how much work can be done with
the current available resources.
So the, the trend is often more is better.
The more money you spend on AI is better.
But I think what's being overlooked is
the capabilities that we have today,
and the power that we have today can
achieve more than what we are doing.
And maybe this connects us to
the performance optimization.
I don't think it's overlooked, but I
think it should be wider, widely adopted.
There are papers that we see once
in a while, there's efforts that
we see once in a while, but the
fact of the matter is oftentimes
backs to the chasing the dragon.
I think it getsâ¦
Maybe energy efficiency is a
better term of putting here.
It's like performance efficiency and
energy efficiencies, and it is somewhatâ¦
People are worried about energy, people
are worried about performance, but
it oftentimes becomes the second t-
part of the conversation sometimes.
And I think that is, has to, that
c- type of conversation, that
type of mentality has to change.
There isn't enough power in the world to
enable agentifying all of our applications
Dan Mitchell: Yeah, for sure.
For sure.
Manya Ghobadi: But, also there is, if
we actually think about, there is room
to optimize and there is room to pack.
There is resources, and I think there
is a way for us to actually double
the capabilities of the current power
plants by packing more work into
the GPUs, by optimizing the way that
we are building these solutions.
So if we continue doing the way
we are doing things, there isn't
any, like, we're almost doomed.
But if we actually think about
optimization as a first principle and
bringing that into every application
that we're building, every AI model
that's running, if we change the
pricing model into, like, related to
the optimization, related to resources,
I think that opens up the routes
that i- it's impossible right now
Dan Mitchell: Okay.
Well, let's hope that
we're able to go that way.
I'll double down on a prediction ask here.
So, you've spent your career
studying the systems that
power modern computing and AI.
If we sit down again five years from
now, what's one thing about today's
AI landscape that you think we'll
look back and we'll say, "I can't
believe we used to do it that way"?
Manya Ghobadi: A lot has changed actually.
Maybe I answer it from an
academic perspective and then
from a, business perspective.
Academic perspective, I already
said, I can't believe I used to
write every word of every paper.
I used to generate every plot manually
with PowerPoint or with Excel.
I can't believe I used to do that.
I can't believe I used toâ¦
Like, every slide, I actually used to
build everything, write everything.
Like, every word would-
was generated by a person.
So that's, that's already happened.
But let's extend that to
the rest of our lives.
Still right now, you and I
write p- emails to each other.
We write texts to, to each other.
I think all of that is going to be gone.
Purchasing, a lot of decision-making,
a lot of perhaps hopefully mundane
decision-making that could have been
automated, you and I are gonna look
back and be like, "I can't believe we
would be, like, typing texts to each
other, or we would be sending each
other documentations or designs orâ¦"
All of that is hopefully going to be taken
by, let's use the hyped word, of agents.
A lot of our tasks, and hopefully we,
you and I, will do the creative tasks.
Like right now, probably during the
day, we are doing some creative work
and some mundane repetitive work that
can be automated, and I'm hoping that
in the future, in five years, agents
are taking care of the repetitive
work, and you and I only have the
creative component of our days.
And we expand that, and look back and
say, "Oh, my God, I can't believe that
I was wasting my time responding to
emails that could've been generated
automatically, or scheduling things
on calendar, or building slide decks."
Dan Mitchell: I believe it.
I believe it 100% as someone who formerly
owned a flip phone and then came with
predictive text, and that was wonderful.
And then came the ability to type
on a keyboard on a phone, and then
autocorrect, which is a love-hate
relationship for me, right?
Sometimes you don't wanna use the word
duck, but it autocorrects to it anyway.
And, uh, and so let's hope that
you're right, that that efficiency
all the mundane tasks just goes away.
No more mundane tasks, just focus on
creativity and thinking about new ideas.
well, finally, if listeners take
away one thing from our conversation
today, what do you hope that is?
Manya Ghobadi: I would like
to encourage the listeners to
Challenge the status quo.
If you are an actual AI user,
challenge the status quo more
Be hungry on the use of AI,
but also be observant and keep
asking, "Why is it done this way?
Why am I choosing this particular model?
Why am I going to this
particular provider?"
As opposed to following the herd
of, "Today there's a new model and
everybody should use it," or, "My
friend told me about this one feature
or one service that I should use it."
I think AI is moving so fast
that it's very hard for all of
us to sit back, think deeply.
And if there is one thing that I would
like to encourage the listeners to
do is take a moment and think about
AI and how it can help your business,
then think about the capabilities that
are out there, and then choose wisely.
There are many options.
Not always the one that gets
hyped up is the correct choice
Dan Mitchell: That's great advice.
All right.
Well, this has been Mainly AI.
Manya, thank you so much
for joining us today
Manya Ghobadi: Thank you for having me
Dan Mitchell: Absolutely.
if you like what you've heard, be sure
to like and subscribe to our podcast.
And until next time, this is
Dan Mitchell with Mainly AI