Dev Rishi: Like, I'm sure in the next three to six months, we're going
to see great open source variants that do something very similar.
So open, you're thinking of like,
Daniel Reid Cahn: arms race is good, basically.
Dev Rishi: I think arms race is like, net good for everybody, like in, like, for all
Daniel Reid Cahn: consumers.
Oh, awesome.
So we can just jump in.
Well thanks so much for joining us I'm here with Dev Rishi Dev did his
undergrad and master's at Harvard in CS went on to work at Google,
where he worked on Firebase, Kaggle, and then Google Cloud AI, which you
mentioned became Vertex AI just towards the tail end of your time there.
Yeah.
And then you started Predabase, which you founded to do a couple of things
you started with, if I understood, the low code framework Ludwig.
Mm hmm.
Yeah.
And then you guys also produced Lorax, and now you focus on Predabase.
fine tuning and hosting language models that are task specifically trained.
Is that
Dev Rishi: right?
Yeah, that's pretty much perfect.
You know, when we started Predabase in 2021, it was on top of Ludwig,
which is a low code framework for building deep learning models.
And we were convincing everyone why deep learning mattered.
And then in 2023, everyone cared about deep learning, but just one variant
of them with large language models.
So we really, I think, pivoted the company to be very focused on LLMs.
And our, you know, unique views is how do we help you fine tune those models
and how do we help you deploy them?
Daniel Reid Cahn: So language, Curious you weren't just focused on
language, became focused on language.
Do you think that language is special, or do you think people
are focused on it today because that's, like, what's possible?
Dev Rishi: I think two things.
One, I think language is special.
I also think we're focused on it because we're very early.
Like you know, the industry is early and as like startups were also quite early,
like if you think about a lot of the kind of core models that have existed for a
while, the transformer architecture I think has been applicable for language
and like computer vision in particular.
Now you've applied it to audio, you've applied it to video and multimodal,
but like the types of models people actually put into production pre OpenAI
were probably like BERT, LongFormer, Decilbert, like these variants that
were pre trained deep learning models.
And then like, you know, the VIT and Vision Transformers from Google.
Those are like the two areas.
And so I think language is special because it's like, because it's possible
to get started very early on that.
And we had a lot of prior art to be able to build with.
And a lot of use cases are transcribed in language today.
You
Daniel Reid Cahn: say it's easy to get started in language.
Like, why is language hard?
Dev Rishi: For two reasons.
One, I think there is that history and background already.
So we like have a lot of tasks that are already defined.
A lot of the types of tasks we see people looking to apply LLMs for today.
Four years ago, companies were trying to do a lot of those same types of tasks, but
using different model variants like BERT.
So The use case imagination, the evaluation criteria, the
datasets themselves, I think are already set up nicely in language.
There's more prior art there, I think is really what I want to say, actually.
Daniel Reid Cahn: Because, I mean, I tend to think the other, the other way
to look at it is just like information theory, and like how condensed information
is, that like, if a picture's worth a thousand words, like, that's awesome, but
it has like a ton of pixels, you know?
Yeah.
So, You know, you now have like megabytes of data that really
represent only a thousand words.
A thousand words is pretty small.
I mean, like I, given that you guys have focused on infrastructure and
tooling, like, do you think that there is, you know you know, is it a good
place to be in the language space or is it sort of like scary that we're all
handicapping ourselves by just focusing on this relatively easier problem?
Dev Rishi: To be honest, it's both.
Like I feel like it is a good place to be in the sense that the compute
infrastructure and layout, I think is much more optimized for being able to do.
The types of production, lower latency, and higher throughput applications
we'd want with language models.
You know, even like, again, four years ago on like, V100s, T4s, you
could effectively serve some of these types of like smaller language models.
Today, you can get the type of throughput you'd want.
I kind of struggle to imagine if you're a really fine tuning multimodal video
models, what kind of throughput you'd actually wanna be able to do, right?
Yeah.
If you're a security company and you have like security camera footage
from all of your different offices, I dunno if there's realistically a
great way to be able to process all of that video that comes in daily for
many different cameras very quickly.
However, if you're a bank or if you're, you know, an and you have many emails
that go through, we know we can scale up that infrastructure up and down.
So
Daniel Reid Cahn: just to clarify, you're saying if I am.
A security company and I want security cameras.
Yeah.
I can't use multimodal language models because they're super heavy and
powerful and if my goal is just to see that someone walked in the door like
that's crazy overkill, way too slow.
So.
I think it's harder
Dev Rishi: to use in production.
Like I think you can start to use it but what you can't do is probably
throw all of your video feeds directly out of it without it being
really either cost prohibitive.
Especially given kind of like the crunch on what hardware
you need to be able to run on.
Whereas I think you can go into production with something like email use cases,
where people have just as many emails that get sent probably as like, you know,
security camera footage as an example.
Daniel Reid Cahn: I mean, there also are general purpose,
like, image models, right?
Yes.
Like segment anything model, like SAM.
I don't know if that's outdated now, but.
Dev Rishi: Yeah, so I guess, it makes sense.
And that's where actually, honestly, that's where I see the progression though.
It's like language really kind of first.
CV and computer version and images next, but then there's like even higher fidelity
of like images mixed with audio mixed with video Yeah Where I think things
like more or less get heavier and heavier over time and you asked is it like?
Convenient or is it scary and I said both the reason I'd say it's scary is
because I think the space is moving so quickly That like what you want to
make sure to do is like Offer benefits that aren't gonna get commoditized
at the infrastructure layer, right?
like like Yeah, it's possible to do language modeling that means a lot
of people are doing language modeling means the infrastructure for it's
getting cheaper And cheaper and so how do you think about your mark?
Daniel Reid Cahn: Yeah, whereas like very few companies are doing
video because it's so freaking hard
Dev Rishi: Yes,
Daniel Reid Cahn: so language you mentioned you want to
avoid being commoditized.
I guess
Just correct me if I'm wrong, but like, it seems like model
hosting is getting commoditized.
Is it?
Yeah,
Dev Rishi: I think that there's I mean, I think that there are some differences
that model hosting providers will go ahead and try and usurp There's
two key ones that I think about.
The first is like performance on throughput and latency.
And so some performance like Groq I think is a great example of
one that has gone to the hardware layer to be able to optimize that.
The second I would say is around workflows.
For like model hosting, but usually there's more advanced types of use cases.
For us, the workflow we're most interested in is when people have
multiple fine tuned models, which is where we developed Lorax, a framework
for hosting many fine tuned LLMs.
But we've seen other people invest in hosting solutions that like, allow
you to do shadow deployments, A B testing, you know, incremental rollouts,
blue green deployments, and others.
Just the part that I think is getting commoditized is like if you're just doing
base model inference and that's it, it's hard for me to understand why you pick one
provider versus another other than cost
Daniel Reid Cahn: and quality.
I mean, if you're, I, I, I'm totally with you, by the way, I think I won't list all
of them, but there are other companies today offering free APIs of base models.
That must be really hard to compete with.
Dev Rishi: Yes.
I think that in my view, it's not actually something I want to compete with at all.
So when I think there are companies and I actually think some of these
companies have made meaningful amounts of revenue even but I don't know if
it's necessarily high margin revenue.
That's probably the thing that needs to get litigated.
Right.
But essentially what you have is you sign up for these massive pre commits
to GPU clusters, you optimize the throughput of your models a lot, and
then you hope that you get really good utilization across those, so
that you can more or less squeeze the lowest price per million tokens.
Daniel Reid Cahn: Yeah, whereas with, so Lorax, can you just
quickly explain what Lorax is?
Dev Rishi: Definitely.
So Lorax is essentially our serving framework at Predabase
that we open sourced of last year.
It's a multi Lora serving serving infrastructure, and what that really
means is, We observed that when most people were fine tuning LLMs, they were
doing a parameter efficient or lower fine tuning, where you only customize
a small subset of the weights of the model, usually much less than 1%.
Daniel Reid Cahn: So the theory, just to clarify, is like, you can fine tune an
entire model, but the vast majority of the language model is just like reading
English, understanding basic concepts.
And so realistically, like, like a human, you could have very specialized humans
who are really damn good at their job, but 99 percent of their brain is identical.
So we think like 99 percent of the model.
Can be standardized, 1%.
Dev Rishi: I think that's a great analogy, yeah.
Actually, Databricks just put out a paper today on LoRa fine
tuning versus full fine tuning.
Those are generally like the two types of domains, and obviously
there are different variants.
There's LoRa, there's DoRa, there's quantized LoRa, and a few others.
Daniel Reid Cahn: like when I first saw the lower paper, it's
really hard to underestimate.
Like it was freaking surprising as hell.
Like, I do not think it was like remotely obvious.
Like,
Dev Rishi: I don't think it's like, I don't think it's intuitive.
And like the key intuition is used, like, you know, essentially discover
a small transformation, a small subset of weights that you can go ahead and
apply against like the overall model.
And then that is very customized towards the tasks that you're looking to do.
But then I think like,
Daniel Reid Cahn: even in the paper, I'm just saying like, they, they were, you
know, they acknowledge at some point, they're like, we know Laura is not going
to work for everything, but we haven't yet found a case where it doesn't work.
Dev Rishi: From our standpoint, I've been really impressed with the broadness,
the broad applicability for it.
And I think actually the intuition goes towards you.
I'll talk a little bit about why I see that broad applicability in a
second, but the intuition, I think, is actually very similar to what
you said, which is, let's imagine you had, you know, two human brains.
99 percent of the overlap is actually quite similar.
A lot of, like, you know, both of us understand the
fundamentals of English language.
We can both see that these mics are black and, you know, that we're
sitting on blue chairs, for example.
But there are things that make us very unique.
It's a very small subset of, like, the actual weights or
neurons, like, in our brains.
And what the paper from Databricks that was released earlier today, I think
actually indicated was, Lora fine tuning is not necessarily as good at full fine
tuning when you have lots and lots of data, and you want to relearn something
extremely specialized, but it also is not as prone to catastrophic forgetting.
And what I think that really is, what I think is important there is, The kind of
base model knowledge is very important.
That foundational understanding of English language, the understanding
of like these, you don't want to overwrite a lot of those.
And that's why when we actually talked to customers a year ago, I talked to
some customers that were like, I don't believe in fine tuning, because I
fine tuned and my model got way worse.
Like I thought, you know, worse than base model performance.
Like, and this was a common thing.
I think it was because the same things we saw with BERT models in 2017 with
catastrophic forgetting and others.
What we did in Lor Land was you know Lor Land was a project where we essentially
picked out 27 different datasets.
They represented tasks from legal clause classification to question answering
to comment class to content moderation.
And we fine, we LoRa fine tuned, in fact, we queue LoRa fine tuned just for
efficiency quantized LoRa, for across 27 different tasks like Mistral 7 billion.
And initially our goal was just to show, hey, Actually, you
know, your LoRa fine tuned model is better than your base model.
But what we actually saw that was for 25 out of those 27 tasks,
it was significantly better than the base model, up to the point
where it was kind of matching or exceeding GPT 4 level performance.
And so I agree with the authors of the QLoRa paper where they said you
know, I don't know, I'm sure it's not applicable for everything, but we're not
sure what it's not applicable for yet.
We found LoRa to be extremely applicable across, you know, many, many different
types of tasks that we've benchmarked.
Daniel Reid Cahn: Yeah, that makes sense.
And then I guess the other big benefit is you mentioned, like, the efficiency
thing, which is like, if you're doing, if you're offering a base model, You
can scale to a lot of GPUs, which means you have like some threshold
of the number of requests per minute you can handle across the platform.
So let's say a company is like, you know, we're going to offer, you know, inference
to a thousand companies, each get a thousand requests per second or whatever
it is, thousand requests per minute.
You know, and therefore we need a million requests per minute to handle.
We're going to get those GPUs on all the time, and as long as we get
utilization high, we can make money.
The problem is, If you're in fine tuning land, you can't do that generally, right?
Because you would need to have, you know, if you, you need each company
to have, let's say, their own fixed thousand requests per second, I guess,
in your, in LoRa, what's really cool here is that you can host many different
LoRa's on a single machine, right?
Dev Rishi: That's exactly right, yeah.
So I think the origins of Lorax really came in because we wanted to have a
free trial version of our product.
We wanted anyone to be able to try out fine tuning.
And it turned out that wasn't that big of a deal.
You know, I could allow, we could even give away GPU access for
fine tuning because the models are pretty inexpensive to train.
The 27 ones that we benchmarked in Loraland were, on average, 8 or
less to fine tune for a given run.
So they weren't that expensive to be able to give away for free trial access, but
what was gonna get a lot more painful was once you fine tune that model,
it's I was a lot more concerned about people deploying them to test them.
Because if every single person who is fine tuning a model had to spin up a
new GPU, that was gonna get very kind of painful and restrictive for us,
because there's only a limited set of, you know, A100s that I would say the
world has, and also, you know, a much smaller subset of that that we have.
Daniel Reid Cahn: Yeah, yeah.
Dev Rishi: And so we needed this technology to allow free trial users.
This is the origin story for Lorax.
We needed a way to be able to allow free trial users to be
able to test out their models.
Because it probably wouldn't be good enough in a trial to say, Hey,
congrats, you fine tuned a model, now pay us in order to be able to go
ahead and try out a single inference.
So we developed this way where we could host a single base
model, call it Mistral 7 billion.
And then, you know, Daniel could come in with his fine tuned Lora, and I
could come in with my fine tuned Lora, and we'd just exchange those Loras
to be on top of the same base model.
And that was, you know, where Loras came out and a lot of our kind of,
I'd say, innovations in the open source have really been how do we make
that exchange extremely efficient?
Which is
Daniel Reid Cahn: crazy cool, by the way.
I mean, like, it's, I think it's hard to overestimate like the impact of
like, typically, I mean, the big thing is about cold start time to me, which
is like, if I don't have dedicated compute, and I am a free trial user,
or like, At Slingshot, we play with a lot of new techniques constantly.
We might train 20 models just with minor differences on some experiment, and we're
like, let's run inference in all of them.
Either we have to, you know, if you're just doing arbitrary compute, the
cold start time to load 70 billion parameters into memory can be huge.
You know?
Exactly.
First, allocate the machine and if I wanna make.
Ten requests to ten models, then that means that like either I need
to get ten machines in parallel, each one taking 15 minutes to spin up.
No, no one has that kind of compute, you know, or you go to Predabase
and then you're like, Hey, can you load all 10 onto one machine?
And it's like, sure, I'll do each one.
It'll take me half a second.
Dev Rishi: Yeah, exactly.
You know, when we first launched Lorax, we thought one of the use cases would
be People who had many, many different fine tunes for many different use cases.
So an example, you wanted to fine tune for many different customers.
Turns out the earliest use case is just AB testing different versions of the models.
And that's obvious to say like in retrospect.
But usually when you have fine tuned, you didn't just fine tune one model,
you've kind of created a workflow, you've tweaked a few different parameters.
Or at least
Daniel Reid Cahn: like for us.
multiple checkpoints if we're like, Hey, did more training actually help?
All right.
I'm being really nice.
So let me ask on the harder side, I guess a little bit.
There's one other company doing this OpenAI.
Are they a competitor?
Dev Rishi: OpenAI view in two ways.
I actually love when people use OpenAI.
I think they are like, look, I think they are a competitor.
I think that The way that I see it is today the vast majority of
workloads that exist on GenAI are on a proprietary model like OpenAI.
And I think, you know, open source is a very small percentage.
I think that there is absolutely roles for both of these to exist.
So I don't think it's like an either or set, solid market.
I don't think everything is open source.
I don't think everything is open AI.
I just expect the market, like the overall pie, to grow.
And I expect the overall market share of open source models to grow.
The reason I expect that market share of open source models to grow, and at
Predabase we do all open source LLMs and fine tuning of open source LLMs.
The reason I expect that is, I think it's a very natural progression.
OpenAI is extremely easy to use.
We have to give them a lot of credit for that, right?
Almost all of our customers started on OpenAI.
And they decided to go ahead and fine tune a smaller task
specific model for three reasons.
Cost, latency, or control.
And so when somebody isn't an active user of OpenAI, I almost
get a little bit nervous.
Because I'm like, well, do you know what your GenAI use cases are going to be?
Is there any reason you haven't already experimented at that level?
Whereas some of our most qualified users are saying things like, I have lots of
GenAI I have lots of OpenAI GPT Four load.
It's just getting very expensive or very slow at scale, I'm getting
throttled on late limits, or I want to go ahead and, or, or, you know, I
built my POC in prototype there, but to actually go into production, my
company isn't going to allow me to use that external service, and I want to
find a unit smaller open source model.
Daniel Reid Cahn: That actually makes a ton of sense, yeah.
I like the framing of like, I want to know that you've used OpenAI.
Dev Rishi: Yeah, they're a competitor and they're also, I'd
say, a gateway drug to the source.
I think that's
Daniel Reid Cahn: interesting.
So last week it hasn't been published yet, but my last podcast was with this guy
Talfen Evans, a friend of mine at DeepMind who works on pre training of Gemini.
And I asked him similarly, like, is there going to be one model to rule them all?
And he surprised me by saying no.
And he was like, no, I think that companies like OpenAI care only
about the very biggest models.
I think there will be a space for smaller models.
But just to ask you, I mean, Do you think there will be one model to rule them all?
Dev Rishi: No, I don't think the history of machine learning
has ever been like that.
And I don't see that like really going forward.
The main reason I think that there won't be.
One model to rule them all is, I think that there will necessarily
be a price performance versus quality trade off, like over time.
Like, I think you will necessarily see larger models are a little bit
better, from like a, you know, quality standpoint in a general setting, and
smaller models are cheaper to serve.
And I think that we see this same experience today, people distill
these larger models into smaller tasks like models once they have
very specific data sets as well.
Just to be
Daniel Reid Cahn: clear, I think there are like two separate things here.
Yeah.
The first being like, will there be one general purpose model
that's so powerful that we could just prompt it to do everything?
Hmm.
Which I think is the, is that, that's definitely You know,
up for a lot of contention.
I think there's also the element of just like performance, like in terms of like,
it might be that that happens eventually, but it just is a long time in the future.
The other question then is just like, what if OpenAI's models are so good that fine
tuning them is just so freaking good, even if you took their smallest possible model?
What do you think?
I mean, they don't have a small model right now.
GPT 3.
5 is the smallest.
Dev Rishi: Yeah.
Daniel Reid Cahn: Do you think there's a reason why they're not going smaller?
Dev Rishi: I think, I mean, so I'm just going to speculate it to
what OpenAI would say, but I think that's actually pretty aligned.
Like, I think their core mission is around, you know
essentially building towards AGI.
And I think like if that is your core mission, one thing I've learned
as a startup is you have to be extremely focused on what you do.
And I'm not convinced what OpenAI wants to be able to solve, necessarily, is
like, how to be able to deploy the most optimized, small, task specific,
fine tuned models that can be deployed, you know, as individual instances that
people have high degrees of control with inside their VPC, versus, how
do we go ahead and solve alignment issues for my single, large, big model?
So that's why I think OpenAI probably is, like, I think that it's, Not antithetical,
but it's a, it's a li it's a detraction from focus on their core mission.
I
Daniel Reid Cahn: think it's interesting 'cause Han said the same thing.
Yeah.
And he's coming from a, the opposite.
He's coming from a big company.
Right.
You know, literally Gemini.
But he had a similar conclusion.
I think you're both wrong.
I'm like so confused by this.
Mm.
Like OpenAI had smaller models for a long time.
They don't right now.
Yeah, but they did.
That's one thing.
And secondly, yes, their focus is on AGI long run, but they've also
been very consistent about this foundation model mentality where
they want people fine tuning, right?
So what I wonder is like if they believe themselves to be a foundation model
where people are going to train on top, doesn't that necessarily mean that they,
you know, want lots of fine tunes and you know, even if that's I don't know.
Dev Rishi: I think the piece that I feel like is most critical
for this is like the G in AGI.
Like my view is OpenAI is going to solve things like in as
generalized of a setting as possible.
But there are like very specific, like domain specific models that
are foundation models as well that people go ahead and like train.
Code Llama being like one of the most like popular subsets, right?
Yeah,
Daniel Reid Cahn: but I mean the most popular code model is still AGI.
OpenAIs.
It's still OpenAI.
Yeah.
Through GitHub Copilot.
Dev Rishi: But I think, like, what I, But I think if you had, like,
a low latency code classification use case, as an example.
I'm just mentioning,
Daniel Reid Cahn: like, a good example here is, like, CodeX is super low latency.
It's their low latency distillation of GPT, basically.
Yeah.
Right.
Dev Rishi: Yeah.
But I think that like, will OpenAI want to offer smaller distillation
versions of their own models?
Daniel Reid Cahn: They are CodeX, I'm just saying.
Like, they're, let's literally CodeX.
Dev Rishi: Yeah, but I haven't seen this like across like I haven't seen
this be a key focus for them, right?
Like I like what if I think about like what OpenAI's key business is,
Codex exists I think that there are so many other like opens like like
for example, I think ServiceNow.
starcoder You know, I think there's so many of like the code foundation model
space is itself so quite like competitive, and I think you're going to start to
see this across like every single subset of domain, like medical, legal, you
know low latency hardware on device.
Maybe OpenAI will go ahead and start to chew in some of this.
I think there's always a risk.
Anytime you have like a hyperscaler, there's always a question of
like, well, why can't they do X, Y, Z different things?
But I feel like what they're going to be, like, if you OpenAI focused
on today, I feel like they're focused on, you know, How do we get
to the next level of performance from GPT 4 or GPT 5, for example?
And like, how do we
Daniel Reid Cahn: But then GPT 4.
0, I mean And
Dev Rishi: how do we get, and how do we get through some of these developer
Like, I think they're, what they're most interested in is From a research
standpoint, building towards AGI, which is where I see that performance side.
And from a business standpoint, building the developer ecosystem.
So I think one of the big things about GPT 4.
0 was, how do we stop, like, how do we make it, how do we eliminate some
of the frictions we consistently hear from developers about using OpenAI?
That's
Daniel Reid Cahn: fair.
And I think that's what they care about.
You know, they can do that without it being their central focus, because
it's not that far off, but I guess the point here being, if they really
wanted to have the best ecosystem to host their product, Lorax is basically,
and lots of loras of small models.
That could be a very different business model that they might
or might not wanna touch.
Dev Rishi: I think that's true, but I think in that world they'd have to compete
with decentralized model essentially.
Where today, you know, you have many different institutions that are
putting out some of these very domain specific like foundation models.
Like Yeah, but I, for code and like, just to mention, so open.
Daniel Reid Cahn: We have a partnership with OpenAI.
We also train our own models.
But from what we've heard I believe it was not confidential.
They mentioned that they had 150 partners at the time that were
partnering to fine tune GPT Four.
I think.
I could be getting that wrong.
I could be.
Anyway, but their philosophy when we had spoken to them was
very much like we want AGI.
We want to cover every use case.
We could do that through one model to rule them all, but our actual main plan is
probably support these foundation models, like have 150 perhaps or whatever, a
thousand companies where we have like, you know, maybe specific AI for immunology.
startup that builds their immunology model on top of GPT Four.
Personally, I think about this so much because it affects my space,
but like, I'm so conflicted.
Like on one hand, are we stuck relying on OpenAI because their models are
going to be so freaking good that the only choice is to fine tune them?
Or, you know, is OpenAI focused so much on generality, the biggest fish,
the general purpose assistance that we should, you know, Not rely on them
Dev Rishi: empirically.
Hasn't it been opening, has been focused on generality though?
Like if you think about their subset of usage today, I would like Hazard a
guess that 98% of usage, if not more, is probably like G PT 3.5, turbo G, PT
four and like the, you know, G PT four.
Oh.
Maybe like it's very much these general purpose models rather
than small task specific like I'm not confident about that.
Daniel Reid Cahn: I would have to look into it.
Yeah.
Obviously I don't know, but I think of demos, yes.
I mean, you're probably right, 98 percent of usage might even just be
demos, but I do think among companies I tend to talk to, I'm not sure.
You might be right, but I wouldn't be nearly as confident about the
Dev Rishi: 98%.
I would say of OpenAI usage, I think that's true.
What I think people do is then build a lot of workflows so that they can then
customize those models, or they, you know, graduate to the point where they're
using a different open source model.
Daniel Reid Cahn: Could be I just and then the other thing is
we were talking about pricing.
Yeah.
I When we were talking about GPT 4.
Oh, you were not surprised by the huge price decrease
Dev Rishi: Yeah, Look, I think if you look at the history of like OpenAI's
price cuts over the last 18 months, you have seen pretty substantive
cuts in price and pretty substantive like improvements in latency.
And I think that it's actually very good for the ecosystem that
happens, because every time that there is A massive cut in price.
You see the next release of like six to twelve open source models
over the next six months that are catching up at quality, but
a smaller footprint than others.
And so I, look, I expect GPT Four will get cheaper too,
like over the next six months.
Daniel Reid Cahn: I do think the coolest thing, like the kind of craziest
thing here to notice with OpenAI has been, they have been at the forefront.
Yeah.
But there's also some phenomenon, there's like the classic you know, four
minute mile story of like the first time someone runs the first minute
mile, it's considered impossible.
And then as soon as they do, other people know it's possible.
And I feel like GPT 4.
0 was like a huge shock to the ecosystem.
The minute it comes out, it becomes possible.
Similarly with prices.
Were you shocked by GP by the audio model, by the way?
Dev Rishi: I would think I was most surprised that was the low latency.
You know, that was the part that was, yeah, I think the idea that they were
working on like heavily just like following what they had been doing you
know, over the last six months with multimodal video adding in kind of audio
towards that, like that part wasn't.
so surprising.
The piece that I think was actually the most interesting was where
they were able to get to like kind of near real time response rates.
I actually so
Daniel Reid Cahn: disagree.
I personally, I don't know.
I had a, so a couple of weeks back I had Chris Gagne from Hume on the podcast.
And I asked him, actually, I don't think it was on the podcast, but I did ask
him about when do you think we'll have end to end language models for audio?
And I don't remember his exact answer, but he was like, He and
I actually fully agreed on this.
I really thought we were like far away.
I thought two to three years.
And why is that?
Do you
Dev Rishi: think it's like a function of the training data?
Like model architectures don't well support it or something
Daniel Reid Cahn: else?
The like, I still have no idea how GPT 4.
0 did it, but biggest challenges I'd imagine are training data.
There's way more text data out there than audio.
Yeah.
And then text data is so much cleaner.
Like we were talking about with images, like text.
has exactly the right signal.
Most of the time, text is grammatically correct.
It has sentence structures, it has meaning.
Think about, like, essays.
People thought about it.
They, like, wrote the thing they wanted to say.
It's not just, like, a speech.
Audio tends to be more this, like, speech format.
It tends to have a lot of artifacts.
And then fundamentally, like, architecture wise, you need some way
to tokenize the speech and then return back from speech space back, sorry,
from token space back to speech space.
No one's really built great ways to do that.
I think, like, OpenAI, OpenAI.
was the closest, right?
They built Whisper, which at least was speech to text.
And they were, as soon as they built it, it was like state of the art,
super impressive until other people started building similar things.
Do you think
Dev Rishi: Whisper was, for example, a way to be able to collect data?
To be able to start to train kind of an end to end audio model?
Daniel Reid Cahn: Totally could be.
Dev Rishi: Like, I actually think that if you consider some of the
work that they had done in video generation, you think about Whisper,
like, the idea that they got the end to end audio model was actually, for
me, a little bit less surprising.
I think.
The idea that the end to end audio model was able to do, like, generation
in such a quick time, like, sub 200 milliseconds, that, I think, was actually
really, really, that's the part that I would say, like, I didn't expect.
If I had to expect what I would see from an audio model, I think I would
have expected to see something clunky.
Daniel Reid Cahn: But I,
Dev Rishi: I've never
Daniel Reid Cahn: even seen a high latency audio model.
That's why I'm like, even if, personally, if they had released
a crazy high latency audio model, I would have had my mind blown.
Dev Rishi: You've seen some of these startup demos, though, right?
Where somebody is, like, chatting with somebody on a phone, like, on calls.
Text.
Daniel Reid Cahn: to speech.
The language modeling, language modeling right underneath it is not.
Dev Rishi: Yeah.
Daniel Reid Cahn: But I think like the idea is like language model
generally is always pre trained on like prediction of next tokens.
So conceptually that makes sense because you can have a lot of examples of
like question answer message response.
Yeah.
Whereas with audio, you know, usually most audio, if you take
a second of audio in the next second, it'll be the same speaker.
If you were trying to learn how to have a conversation from this podcast right
now, you know, me and you talking, That would be so much harder than taking a
transcript of the same thing, you know?
Dev Rishi: You know, it's interesting, again, I can't speculate on how they
made the training data, but what you just said is like traditional
modeling is speech to text to speech.
Yeah.
And it does feel like that actually helps you create some synthetic data
sets for doing intent audio modeling.
So I'm betting they
Daniel Reid Cahn: either use some synthetic data or the other possibility,
which I think is phenomenal if they were able to pull it off, is learn almost
entirely from text and then like 0.
1 percent of training being speech and then somehow get the whole
thing to work is I think when you watch the demo It's clearly not
perfect like speech generation.
It's clearly it's something it's clearly like way better than anything We've seen
before Yeah but I also by the way give a huge amount of credit that I think they
didn't try to like cherry pick the demos and try To show the best possible cases
like the demos show things go wrong.
They show it via v1.
They show all the problems
Dev Rishi: Did the latency surprise you like it was crazy
surprising to very surprising But I
Daniel Reid Cahn: mean, I'm also surprised that they were able to I mean For a single
announcement to be you know, we made our model faster, better, and cheaper,
all at once, like, usually you get one, maybe two, but faster, better, and
cheaper, I don't want to like, you know, blow too much steam up their ass, but,
Dev Rishi: I don't know, I was shocked.
I do think the better is parts that are getting litigated right now, though,
in some of the Better in the sense of
Daniel Reid Cahn: audio is really what I meant.
Okay, in audio that makes sense.
Increased capabilities.
Yeah,
Dev Rishi: yeah.
One of the things we've been doing is 0 like in the same kind of fine
tuning leaderboard that we've had.
And so we're going to be releasing those results in a few days, but I
think it's like, I think it remains to be seen how much better, for sure.
Daniel Reid Cahn: Yeah, in terms of like accuracy, obviously okay,
I want to ask about fine tuning.
Do you think fine tuning is hard?
Dev Rishi: I think that there's, I don't think it has to be hard,
but I think there's practically two things that are hard about it today.
Okay.
The first is the data.
Like, I think that's always been one of the struggles for it.
Which is like making sure that you have either a good completions
dataset, maybe that's a little bit less challenging, or a good instruction
fine tuning supervised dataset.
This was like the problem for machine learning in 1990.
It's like still the problem with fine tuning and machine learning in 2024.
Then this, the thing, the part that I think has gotten a lot easier
is the tweaking of the algorithms.
Used to be the case that like, if you were building like a model, it was like this,
Weird arts slash highly experimental setup where you're tweaking every parameter
from like learning rate and batch size towards like your regularization
lambda to like anything, right?
You like everything was fair game and you're going ahead and throwing around
you're throwing at it with fine tuning the like scope of the number of parameters
that you actually need to adjust.
I think has gotten to be a lot smaller in order to be able to see
kind of meaningfully good results.
Daniel Reid Cahn: Like what are they?
Dev Rishi: I think like target modules tends to matter.
The lower rank I think also matters, which is like the lower rank will
go ahead and correlate towards the size of the adapter and capacity.
How many?
Yeah, exactly.
And then, you know, I think like classic things like the number of app boxes
as well as your learning rate that you might wanna work for, which is
like not necessarily a hyper parameter as much as it is maybe a business
Daniel Reid Cahn: What about app size and learning rate?
Dev Rishi: Oftentimes, I think now you use like automatic batch
sizes and automatic learning rate schedulers with fine tuning jobs.
At least that's what like we typically do, like in Predabase as a default.
We do like out of batch sizing because it's really a function
of the hardware that you're on.
Size of the model and size of the data sets, input sequences.
And so rather than having to do a lot of experimentation, you can do
some of that in a warm up phase.
Yeah.
So, I think like, like, I think the tricky part of fine tuning
there has gotten less difficult.
I think the infrastructure still tends to be a bit of a pain for people.
For sure.
Like, I've But, I don't know, I, I
Daniel Reid Cahn: do have to say, like, depending on the model,
like, the first time our team fine tuned GPT Four, we diverged.
Like, our loss just went up instead of down.
Yeah, yeah.
And we were looking at that, we were looking at the curve, and we were like,
Wait a minute, isn't this curve supposed to go down, like, decrease loss, not up?
Yeah.
It could, I mean, GPT Four is fine tuned.
massive so it's particularly tricky, but I mean we do think about person I mean not,
not a lot, I think we think about rank, we think about learning rate, we think about
batch size, and we think about epochs.
Yeah.
Epochs you can compare with checkpoints because you just train multiple Right.
Exactly.
We do find learning rate can make a huge difference.
Yes.
And a way that you know, a difference that's impossible to really detect,
like, it's really hard to know.
You're like, hmm, this one seems smarter, this one seems nicer, and you're
Dev Rishi: like,
Daniel Reid Cahn: that makes no sense because all I did
was change the learning rate.
Dev Rishi: And I should be fair, like, I think learning rate schedules aren't
perfect, so I actually do think, like, playing with learning rate is a
fundamental factor within, But like, I mean, I remember, like, the biggest
difference I think about is like, I remember in early versions of Predabase
where you could build these deep, like, where you could build any, like, model.
And then you have to think about, like, the whole architecture.
You have to think about the whole architecture, you have to think about,
like, learning rate of subparameters, you have to think about dropouts and,
like, regularization rates, you have to think about so many different things.
Choosing your activation function and your And that was before
you got to a good enough level.
Like that was before you even started to see something like, you know, it
wasn't like the optimization step.
It was like the, I want to see any value step at all.
Yep.
Today, I just chatted with somebody today who like, you know, is, is former
colleague of mine can be critical.
Like he, he's like honest, I would say about products and it was like, Hey,
what was your experience on Predabase?
And, you know, he told me a couple of things they thought could be
improved, but his main point was.
I didn't expect it to be that this easy.
I didn't expect it to be yeah.
And he said, I didn't expect it to like it this much either in terms
of like just the fine tuning piece, because now you actually can fine tune.
And usually there's a lot of these parameters you want to tweak, but that
first bit that you fine tuned in a platform like ours, or if you fine tune
kind of, you know, externally that has the right kind of default settings,
you're probably going to actually immediately get a lift that Laurel
and it was all our basic defaults.
We didn't optimize the hyper parameters at all, which is kind of crazy.
And I think that like, like lack of hyper parameter optimization need.
is probably the part of fine tuning that has gotten easy and why fine
tuning is no longer very difficult.
The parts that are hard are the data and the underlying
infrastructure, not the algorithms.
Daniel Reid Cahn: I mean, I think the data is really freaking hard, though.
I saw you guys did some partnership with Gretel.
Dev Rishi: Yes, yeah, we just did a webinar, I think, a couple days
ago in a partnership with Gretel.
And I think one of the main motivations there is I think synthetic data is
getting better and better kind of around this training workflow and we see a
lot of people use it in different ways.
So that helps anything that helps the data set creation side of it.
I want to be able to be very front and center of because that means more people
can start to use us for fine tuning.
Daniel Reid Cahn: Totally agree.
You know, on the data side, I have to say so the challenge for us on the
data side, I definitely think like when we, like at Slingshot Our, our
biggest challenges on machine learning are definitely data related, but the
way that we frame it is basically like.
We are trying to achieve, and I think this is true of a lot of AI companies,
a task for which no data exists.
So if you're training a general purpose chat assistant like ChatGPT, or if
you're trying to train Hume or whatever kinds of specialized, you know, a
legal assistant, those don't exist.
Right?
There are no legal AIs, right?
And so if you're trying to show, like, what would a great answer from, you
know, an AI doctor sound like, well, you can look at what doctors say.
But doctors say and ha.
Doctors make mistakes all the time.
Doctors forget to ask the right question.
They're constrained by the amount of time they can spend.
They can't write out long answers because of those constraints.
They, you know, et cetera, et cetera.
Yeah.
And so you're basically like, if I could just find a data set of a billion examples
of people going to the doctor, asking a question, getting a perfect answer, right?
Boom.
All I need to do is walk over to Predabase and train with that data.
That would be phenomenal, right?
And I'm sure you guys can handle that.
But where the hell do I get a billion examples?
Like, even if I got a billion examples of doctors talking to patients, I still
wouldn't have, you know, an AI doctor.
Dev Rishi: Yeah, I don't have an easy answer here, honestly.
And like, this is, again, where I feel like I'm most interested in, like,
tools for the data infrastructure side of things to be able to advance.
What I will say is we see people maybe not as complicated as like there's no such
thing as an AI doctor, but we see people that are like, I don't have label data.
And the tricky thing is one way or another you have to bootstrap it.
One way that we've seen people bootstrap it is okay, the risky way you can
bootstrap it is you go to a subset of your traffic and just launch with something
subpar and you collect like, you know, like some user signal on kind of what
you want to be able to get feedback from.
That could be one approach, depending on which industry you're in.
One thing we've seen is people actually find GPT 4 quality with some
edits in post processing is actually roughly where they might want.
So maybe the case is like you can get, and this is not like, it's something
I've seen like as a repeating pattern.
Maybe the case is like GPT 4 can sound close enough to an AI
doctor for what you might want.
Maybe that's not the case, but like in some cases GPT 4 is at
the quality where you'd want.
And the real concern is like, I just can't use GPT 4 live in production
because like, you know, it's the most, like one of the most expensive
models that exist out there.
Like it's.
It's really slow and rate limited and can be like maybe outside
the organizational policy.
So we've seen people bootstrap with GPT four for data collection, distill that
model down into smaller open source model.
Daniel Reid Cahn: Yeah.
I was going to ask, distillation.
I mean, I think one, one thing I also want to ask about, so I tend to be
very optimistic about AI and also pretty pessimistic just because like
AI workloads out there in the world.
Like AI you know, still pretty much hype.
Like, I'm very excited for AGI.
I'm very excited for where we're going.
I think technology is freaking phenomenal.
I think it's exciting.
You could really see progress, but most people I don't think should be using
AI at work all that much personally.
Like, I don't, I don't know about like AI writing, all that kind of stuff.
And one thing that gives me a little pause here is that it does
seem like a lot of the time, the use cases we're talking about with
AI are still kind of the old ones.
I know you guys pivoted.
So Pre pivot, you were focused on those churn prediction,
revenue projections type stuff.
Dev Rishi: Yeah, I mean pre pivot we were focused on, we're an ML platform,
we can help you do any type of model.
Which, by the way, is a very broad like value proposition.
The types of things people would come with was, yeah, like churn prediction, lifetime
value, like these are the types of things people knew to use machine learning on.
Daniel Reid Cahn: Yeah.
Dev Rishi: And so we saw a lot of that.
So
Daniel Reid Cahn: I wonder, similarly, like, you know, you were excited about
deep learning because you've got it.
You're like, Oh my God, this thing is like intelligent.
It could do anything.
Why would you want to calculate customer lifetime value when you
could literally like understand everything about your customer?
Isn't that way more interesting?
So similarly, I mean, do you think some of the use cases that we're imagining for AI
are just limited from this deep learning?
Same like past point of view of like, you know, before we could
do lifetime value prediction.
Now we can still do it.
But with deep learning, you know, and the analogy being like, I can
go through a blog post and find, you know, a list of tags for SEO.
And it's like, yeah, yeah.
But is that really why we want to build AI?
Dev Rishi: Yeah, but I'm actually not convinced that's a bad thing.
Which is to say like, I think that over a short period of time, like we're so
early within the phases, we have to remember that most people didn't really
have a good lifetime value return prediction like machine learning model.
That's why they were looking for these innovative solutions to be able to do it.
I talk to customers still like, you know, that have to do classification
tasks or extraction tasks over email.
The way that this happens today is there is a back office function
somewhere that's like, you know, maybe offshore going through a lot of these.
So is it?
As flashy to say like, you know, what we're going through is blog
posts and extracting tasks for SEO.
We're going through emails for compliance.
We're going through transcripts to be able to do that.
No, but we have to recognize that the industry is like, we're, we're
like, we're, the industry is in early stages of AI adoption full, full stop,
whether that's like deep learning, AG or anything else along those lines.
And I actually think the biggest thing we can do is start to solve
some of these narrow, boring tasks so we can get to the cooler stuff.
What I actually don't love is when you see like this really interesting
flashy demo from a company that's like, here's our bot that's going to
teach us how to make more revenue.
But it's like, all right, have you figured out how to like even just
solve customer support or something else along those lines, you know,
these classic use cases within it.
So to me, like we were trying to say like, this is going to be a
thing in 2021 using an old class of maybe like technologies now.
Not all of them are bad, like long former, but still actually quite effective
models, but like using an older class and trying to show this is the value.
Daniel Reid Cahn: Yeah.
Dev Rishi: And now if we can get, you know, and I would say like, maybe 1
percent of organizations got it, right?
Like, what were the, what were the number of organizations
using deep learning pre GPT Four?
Not many.
And so what we can do is increase that 1 percent to like 20%, 30%, or
50 percent even doing those tasks.
I think that's a massive win over, over the past year.
So there's not like the
Daniel Reid Cahn: boring work that still needs to get automated.
Yeah, let's do the boring work.
Let's just get it over with.
Dev Rishi: Yeah, let's do the boring work because that boring
work is taking up a lot of time.
And like, like I actually, I almost think like it's not necessary, like, do
we want to go ahead and skip five steps forward or do we want to go ahead and
like build and solve the problems that are sort of, you know, in front of us today?
I am actually personally just most excited about.
For 20 years, people have been talking about being able to automate some parts
of these workflows and now we actually see, you know, more than the most
advanced companies trying to do that.
And I think that's, that's a great place for us to land as an industry,
I think, over the next few years.
Daniel Reid Cahn: I hear that.
I was, I was talking to my father in law earlier today.
He runs an IT company in the Bay Area.
And he was telling me, like, we're finally adopting AI for,
like, our ticket understanding.
A lot of our, you know, agents get these, like, support tickets and
they have to read through the whole thing and they miss the context.
And click one button.
And AI can, like, read it, understand it, make it so much faster, customers
get support faster, our team is happier.
And I'm like, yeah, that's nice, but, you know, what do you really want from AI?
And he was like, I want AI that could solve the ticket,
Dev Rishi: you know?
Yeah, exactly.
And do you think that that's too narrow or non creative, or
how do you think about that?
I
Daniel Reid Cahn: don't know.
No, I mean, I think, like, he's, he's very pragmatic.
Yeah.
Like obviously similar to like what you're describing, would he want, you know,
preta base to be able to actually help him host models that solve his use case?
Like of course he wants to solve his problems now he has problems
now that AI can solve now.
Yeah.
But it also seems like if AI right now can go read through a boring
ticket and understand it enough to get a human to move from taking
10 minutes to solve it to five.
Yeah.
And then, you know, very soon it moves from 10 to zero.
You know, there does seem like.
You know, the exciting thing to me, I have to at least be excited from a sci fi point
of view about this leapfrogging about the point at which this ticket, you know, the
person emails in and they're like, I'm having trouble with signing into zoom.
And then the guy says, like, Oh, yeah, here are three things you can try.
And like GPT four isn't there yet.
Like he did experiment with like, what if GPT four wrote the answer?
It's just not there yet, right?
Yeah, team is smarter.
But he also knows, you know, for a lot of those tickets, We can actually
fully automate them just with technology that's not quite here yet.
That's kind of where I'm, you know.
Dev Rishi: I think the place that I come from is like I remember getting into,
like, democratizing AI in 2016, 2017.
And I think that AI has never been underhyped.
Like, people talk about the hype cycle for AI in 2020 in 2023 2022.
We started Protobase in 21 and I remember thinking, man, we're in
machine learning, this is like one of the hottest spaces, like, in the world.
And I remember thinking that when I was at Kaggle Kaggle in 2019 and 2018.
It's never been underhyped.
I would say it's consistently underdelivered in economic value outside
of the top percentage of companies.
And so if the way we get there is we solve some of these narrow use
cases, I would love that a lot more than we kind of build the next series
of kind of sci fi esque use cases.
But, you know, every comp like the, you know, the construction companies
are still consistently going through every manual invoice themselves.
Like sci
Daniel Reid Cahn: fi guy?
Just curious.
Dev Rishi: I do like sci fi.
Okay.
Yeah, yeah, I do like sci fi.
Daniel Reid Cahn: I I mean, I Look, I think there's,
Dev Rishi: I like sci fi, but I've spent so much time with the customers
that are like, look, AI seems great, but here's my, like, here's
the problem that I actually have.
And it's the same things that we haven't solved in 2016, 2018, 2020.
And now I see the solution like here.
Daniel Reid Cahn: Although I don't know, I, I go back and forth.
Cause I do get it.
I do get that.
Like, I think we are going to go through some sort of market crash,
some bubble popping, some, you know, investors saying, show me the money.
What's going on.
Yeah.
On the other hand, you look at these, AI hype cycles, like we're talking about,
and you're like, it's not really a cycle.
It's just been hyped.
But the truth is, like, it was pretty hyped during the big data era.
Yeah.
Then it got more exciting during the ML era, and then it got more
exciting when Transformers came out, and then it got more exciting
when GPT came, GPT Four came out.
Right.
It's not like we've gone through some hype cycle where it was exciting and then
not, and then more exciting and then not.
It seems like it's only gotten higher and higher.
And the reason why is because I think like we moved from a place where
like AI can automate back office tasks to now AI doctors, like, I
also wonder just economically, like if we actually delivered on all the
back office tasks, that be nearly enough to account for the investment
and the hype and the excitement?
Or do we really need AI doctors for, you know, That's a good
Dev Rishi: question.
I feel like we should put a top three consulting or accounting
firm on that to figure out like what the costs directly are.
The nice thing is once you solve with AI, it's kind of like recurring value, right?
And so like whatever you solved in a year, like you basically get the same.
It's like the beautiful thing about Sass in some ways.
So I imagine over some horizon of time, the answer is probably
yes, but it'd be interesting to compare that against investment.
I mean, I will say like, I'm very excited about the really cool end to
end multimodal models that we have.
I've seen these end to end physical world models, too.
Like, I think there's some amazing areas where AI is gonna go.
Daniel Reid Cahn: Self driving cars, by the way.
Where are all the self driving cars?
I just
Dev Rishi: think that, like, I actually think that the ecosystem is not currently
limited in the imagination on that.
Daniel Reid Cahn: Okay.
Dev Rishi: And so I think, like, the funding exists for
those types of environments.
I think that the progress is happening.
Maybe it could happen faster.
I think the progress is happening kind of in a startup landscape there.
And so I'm not so concerned that, you know, the Fortune 500s of the world
are probably starting to think a lot more about back office task automation.
Because I think by the time that they figure that out, I'd love
for some of these physical world models to then be like, Okay, and
here's how we can actually help you.
Like, end to end physical world models?
Incredible.
Like, I think, you know, what, incredible in like the line of thinking.
Do I think there'll be something someone could use there in six months?
No, and I don't want AI to be a disappointment just because,
you know, they haven't they haven't quite gotten there yet.
Daniel Reid Cahn: Yeah.
I mean, I also think like, like the reason why I'm excited about Predabase
is because I think there are a lot of sci fi models that we can actually deliver
on, hopefully on Predabase, meaning like that we can get to that point of
like, you know, not just we were able to do some like tag identification,
but you know, there are probably a lot of really high impact things that
can be delivered with fine tuning.
Totally.
I think You know, I do think there has been some lack of creativity, personally,
I think there has been some, like, I hear way too often when I talk to people about
AI, you know, this, this idea, like you said, about the 98 percent of compute of
GPT Four requests being on the base model.
Most people I talk to assume, yeah, that AI is just base models.
I hear a ton from AI engineers about, you know, I think rag has sort of fallen
a bit, but for a while that was like, rag, rag, rag, don't ever fine tune.
And I'm wondering, like, is it just because fine tuning is hard?
You know, were people not targeting hard enough use cases
to bother with fine tuning?
Would Predabase make more money if people tried more hard shit on Predabase?
Dev Rishi: I actually just think that people think fine tuning is hard.
I don't think it actually is hard.
I think it's hard because the data
Daniel Reid Cahn: is what I mean to say.
The data is, yeah.
Like, if I said, I can call GPT Four to anonymize some text versus I can fine
tune a model to anonymize some text.
Calling GPT Four is so easy.
It is so insanely easy.
For the latter, maybe I have to talk to Gretel, create some synthetic data,
train a llama model on Predabase.
Like, that could take me, real effort and thought.
Dev Rishi: For sure, but I think that's actually why no one
starts off fine tuning a model.
They always start off like with GPT Four, you know, or GPT 3.
5, and then the base open source model, and then fine tuning is kind of
next in that progression life cycle.
And I am actively thinking about ways we can move fine tuning up
closer to the beginning or at least something there like that data prep
requirement becomes less painful for the user, kind of on their side.
But I think the fact that this progression exists.
is it's like very logical.
And to me, like some of the limitation in thinking is just.
We have like this cascade of people that are going through these steps still and
I feel like we've just started to see the fine tuning wave Starting right now.
Like it's like the very early days of like where people are like
fine tuning actually makes sense
Daniel Reid Cahn: Yeah,
Dev Rishi: and like I think I think like launches like, you know, like the
Databricks paper on like Laura fine tuning and it's just full fine tuning I think
like Laurel and I think all these things help advance that ecosystem Also just
Daniel Reid Cahn: like this shout out to Harvey fine tuning
10 billion tokens for legals.
Yes their whole point is about fine tuning.
They are fine tuning.
Everyone likes to make these distinctions.
That is fine tuning.
What they're doing is fine tuning and that's phenomenal because they're trying
to take on an insanely hard use case by getting 10 billion tokens of data.
That's not easy.
Exactly.
I feel like again, not the training part.
I think training, we could debate how hard it is, but the point is the
data is definitely the hard part.
Dev Rishi: Yeah.
I feel like a year and a half, the thing was like pre training and that
wave actually didn't really make a lot of sense to me because it's like,
the vast majority of organizations probably don't need to pre train.
Some definitely do, the vast majority definitely know, but fine tuning I think
makes all the sense in the world to me.
I feel like we're in the very early like eight eras of that and so it's
not like it's, I think we're gonna see a lot more creativity in terms
of the ideas that got brought on.
The direction I'm most interested in the sci fi like creativity is right
now people still think about fine tuning as like one per use case style.
Like I'm going to fine tune a model to do this.
What I'm really interested in is like we have many diff like we have the ability
to fine tune many models very easily, very cheaply, create these adapters,
and now with Lorax you can serve many of these adapters very cheaply.
So what does it look like if you didn't have like one fine tune,
but you had a hundred fine tunes, or four hundred fine tunes, that
all do slightly different things?
maybe, but can be served just as easily as a single base model.
And then what you really have is, you know what, I think some of these mixture
of expert models have exploited, which is the ability to understand what part
of a model architecture you actually want to be able to use to answer a question.
Daniel Reid Cahn: So can you imagine for me, like even if it
were 10, like how would you use?
10 Lauras for a use case.
Dev Rishi: Yeah.
So I think that there's two ways that I can think about it, but let's just
imagine that you were in that customer.
Like, let's imagine you just wanted a customer service bot, right?
Like customer service is actually many different tasks, right?
There's like, hey, what is the triage level and priority for this?
What is the routing area that it should go to?
How do I write a response back to this user?
You know if they want to cancel their subscription, how do I, it's like a
lot of these different types of tasks.
Right now I think that your two options are you like trying to fine tune models
per task and then like deploy all those and figure out like a orchestration
for them all that lives in business logic, which is like, hey, first do
this step and do this step or that would
Daniel Reid Cahn: be like first class by the email.
Okay, it looks like a cancellation email.
Yeah, run the cancellation bot and then run the response bot.
Dev Rishi: Yeah, yeah.
And then like, obviously, there's like maybe 16 other steps that go.
Daniel Reid Cahn: And then the problem is that the person says like, Okay.
Cancel my order, but only if it's not able to arrive by Labor Day.
'cause I'm not in town by Labor on Labor Day.
Exactly.
But if you're able to tell me that it's gonna be the day before Labor
Day, then I can tell my neighbor to come by the house and pick it up.
Yeah.
And they're just like, oh shit, what do we do?
And then
Dev Rishi: they're gonna have, and then they're gonna say something
like, oh, by the way, like what was the what, what delivery mechanism
is gonna, is it gonna be like FedEx or it gonna be dropped off my door?
Like they have all these like interper questions, right.
And so I think solving a use case like that.
It weirdly reminds me of building old school assistants where like,
I worked at the Google Assistant for a little while too, right?
And the way you used to do it would be basically you map everything to an intent.
You map intents to like fulfillment logic and slot filling.
And it kind of reminds me of that.
You have to like build a deterministic logic of like fine tuned model X to
this, fine tuned model Y to this.
So that
Daniel Reid Cahn: was like someone says, turn the light off, turn the lights off.
And then it's like, okay, this person's trying to like change
the status of the lights.
Which way?
Oh, off.
Off is one of the on off options.
Yeah.
Dev Rishi: That's the way we built a lot of AI systems historically, right?
Conversational AI, in particular.
But I think what would be really interesting is if you had specialized
adapters that were well trained for all of these different tasks, served
off top of a single base model, and kind of a router logic, like a routing
layer, that understood for a given question that the user is trying to
do, which specialized model should I go ahead and hand this off to?
So
Daniel Reid Cahn: I'd love to dive deeper there, I don't, I
know we're running out of time.
I want to ask just some last questions.
Curious, first AGI.
Bye.
Do you have plans for AGI?
Dev Rishi: Do I have plans for AGI?
Retire early, I think?
I'm not sure actually, so With AGI At Predabase, we build like infrastructure
and tooling, so I would love for people to get closer and closer towards AGI
building fine tuned LLMs, maybe using this routing approach that I was talking about.
I feel like what I'm still missing is a really good, crisp definition for AGI.
If it's just the Turing test, I feel like we're probably, you
know, around that right now.
Daniel Reid Cahn: Contentious issue.
Contentious podcast on, are we, Like, how close are we to passing it?
Well, is the Turing test outdated?
But I, I,
Dev Rishi: Whether it's outdated or not, I wonder if, like, I think The Turing
test especially is like a measure maybe of like, you know, I would say like,
it's some bar around intelligence.
Daniel Reid Cahn: Sure.
Dev Rishi: I feel like we're hovering around it, whether we've
passed it, whether we're not.
No one talks to GPT Four and says, This is, you know, the furthest
thing I've ever heard from a human.
Like, there is some kind of criteria where they're close, but Let
Daniel Reid Cahn: me just, just for your sake for now, let's stick
with one definition, which is like, work that can be done by a human
remotely, can be done by an AI.
Dev Rishi: Yeah, I think we're very, I think in some, like,
it depends on the work, but I think we're very close to that.
Can your job be done
Daniel Reid Cahn: remotely?
Dev Rishi: No, but I wish.
Daniel Reid Cahn: I don't know, I, I, I don't know, I, I think this is like
my, my father in law, you know, his type company that, you know, he'd be, you know,
He'd be very much affected, but I, like, the other way to look at it would be the
point at which humans become useless.
Like, where you're like, hey, I would love to solve world hunger and then you
have this AI, like, that's cute, you know?
Right.
Like, leave it to me, because I'm smarter and faster, and
Dev Rishi: I think there's, like, the way that the debate usually breaks up
is, like, one, that would be terrible and there'd be mass unemployment in others,
and then the second being, humans will find the next series of things that they
want to go ahead and spend time doing.
Similar to, like, Industrial Revolution, I'm no longer spending time on
agriculture, what do I spend time on?
Daniel Reid Cahn: Yeah.
Dev Rishi: And I probably put myself a little bit more in the latter bucket,
in terms of like, I think that if we end up like getting to the area where
AGI establishes some base level of productivity for the overall economy,
then I think people will choose different ways to spend their time.
Some of them will be productive and expand the Pareto Frontier, some
of them will be for leisure, and I think both of those are good things.
Daniel Reid Cahn: But you're gonna go for the latter.
You're gonna retire?
Dev Rishi: We'll see.
We'll see how everything goes with, like, the next with this with Predabase.
But you know, I think that I would love to be able to have a I
Daniel Reid Cahn: heard Sam Altman say at some point, I don't know if
he was being facetious, but he was like, I'll just have a lot of kids.
Dev Rishi: If that was the flower case, I mean, I think if I got to the
point where like AGI was able to do, like, let's say 70 percent of my role.
Daniel Reid Cahn: Yeah.
Dev Rishi: I don't think I'd backfill 70 percent of it would just work.
I think I'd like mix in some work life balance in there.
More work life balance than I'd say I have today.
Daniel Reid Cahn: Very nice.
And then just in terms of like resources for keeping up with AI,
it sounds like you have some, if you had like an ML engineer looking for,
Stuff to check out, things to read.
Dev Rishi: Yeah I mean, like, there's some obvious things.
Like, I think Andrew Wing's been putting together some great, like, short form
courses on YouTube with deep learning AI.
We did one on efficient serving for LLMs.
We have a fine tuned LLM newsletter.
And I hate to say it, but I actually think I get a lot of
my LLM news on Twitter still.
Daniel Reid Cahn: And
Dev Rishi: it's, like, weirdly actually something where I, like,
feel like I see, like, probably everyone I follow right now is an ML
influencer of some sort or the other.
But The space is moving so quickly, I feel like things just get, like,
organically shared on social, so.
Other newspapers I read newsletters that include The Sequence and a few others
but, I would say places to start would probably be, check out our fine tuned
newsletter like, on top of Predabase.
Great name.
Yeah, exactly.
We have shirts that say the future is fine tuned, so we just called our newsletter.
I
Daniel Reid Cahn: love that.
Yeah.
That is such a great idea to get a t shirt.
We'll get
Dev Rishi: you a shirt.
Yeah, we'll make sure that you have one.
Daniel Reid Cahn: Alright, well this was awesome, Dev.
Thanks so much for joining us.
Dev Rishi: Yeah, you got it, Daniel.
Thanks for having me.
Daniel Reid Cahn: Awesome.