Subscribe
Share
Share
Embed
Season 1 Episode 4 | Value Is Sexy
Gooder AI’s Eric Siegel on Why Most AI Projects Never Make It to Deployment
Gooder AI founder Eric Siegel says the industry is obsessed with the wrong number. Accuracy, precision, AUC score, none of it matters if nobody can translate what a model does into dollars. Eric makes the case for predictive AI's quiet comeback, why 80-95% of AI projects still fail to ever deploy, and why marketing hype might matter almost as much as the model itself. Out now in paperback: his book, The AI Playbook.
🔗Eric’s company, Gooder AI:
https://www.gooder.ai/
https://www.gooder.ai/
____________________
Timestamps:
00:51 — Why AI Value Matters
03:18 — The Power of Predictive AI
06:36 — Why Accuracy Isn't Enough
11:07 — The Cost of AI Models
13:20 — Why AI Projects Fail
18:13 — Measuring AI Agent Reliability
23:59 — Predictive AI Makes a Comeback
29:39 — Bridging Business and AI
____________________
About Speak to an Agent:
Speak to an Agent explores what it really takes to bring AI into large organizations, why AI can't fix a problem you haven't already solved, why real understanding matters more than a polished voice, and why the domain expert who knows how to use AI is becoming the most valuable person in the business. Featuring the people and ideas shaping the future of CX AI, the innovators, the thought leaders, and those making the decisions. They get candid about what's working, what isn't, and what changes now that machines can reason, plan, and act at a human level, and sometimes beyond it. If you're responsible for putting AI to work inside your organization, this is a conversation you won't want to miss.
Creators and Guests
Guest
Eric Siegel
What is Speak to an Agent?
Speak to an Agent explores what it really takes to bring AI into large organizations, why AI can't fix a problem you haven't already solved, why real understanding matters more than a polished voice, and why the domain expert who knows how to use AI is becoming the most valuable person in the business. Featuring the people and ideas shaping the future of CX AI, the innovators, the thought leaders, and those making the decisions. They get candid about what's working, what isn't, and what changes now that machines can reason, plan, and act at a human level, and sometimes beyond it. If you're responsible for putting AI to work inside your organization, this is a conversation you won't want to miss.
Value's sexy. We're bringing value back. Accuracy is almost
always the wrong metric. Don't fetishize the technology for its own sake.
That's not slick marketing, that's prudent marketing. Welcome to Speak to an
Agent. Those words used to mean one thing,
get me a human. Now, the agent people actually want is
AI. It's faster, sharper, and never puts you on hold.
If that surprises you, you're exactly who this show is for.
Featuring the people and ideas shaping the future of CX AI, the
innovators, the thought leaders, and those making the decisions, comparing
notes while the next generation of technology gets built. You're
not just listening, you're in the room. I'm your host, Neeraj Verma.
Joining me today is Eric Siegel, a machine learning veteran of over 3 decades,
founder and CEO of Gooder AI, and author of The AI
Playbook. Let's get into it. You know, oftentimes people think
that just having a hyper-intelligent system immediately
equates into value, and especially I think value in the enterprise.
And it's, it's really interesting, right? Because you
can make tiny models, predictive models like Jeb and
others, result in massive value without being a
hyper-intelligent almost being, right? I think that's a really
interesting topic. So, so tell me about that. Yeah, it's not as sexy,
but actually, you know what? Value's sexy. And I think it's
time to bring value. We're bringing value back, like Justin Bieber.
No, Justin Timberlake.
The word intelligence is very much a subjective term,
and whatever technology we create, it is and
will continue to be a tool that we can use and we
have to use very intentionally and in a directed manner to achieve
value. And I think that what I just said is a fundamental.
It may seem like a no-brainer to a lot of people, Um, but it is
antithetical to a great majority of the hype. Do you— I'm
assuming you follow the Reddit Singularity and Hyperintelligence and
Accelerate. If you don't, I think those are super important and interesting,
um, subreddit communities where people are pushing for the singularity to
happen. I'm not sure if you're familiar with that concept. It's where AGI and
ASI kind of come together and everything becomes Star Trek. I'm
familiar with the concept of singularity. And I am a big
Trekker. Sorry. I know. Um,
but so are you saying that they're proponents of making
it happen? 'Cause I don't think they're proponents. I think if it can happen, it's
gonna happen. Yeah. But I'm a big skeptic, right? I mean, maybe in a couple
thousand years, kind of. I'm, that's, you know, that's where I'm at with that. Yeah.
It's, it's really interesting, right? I think I do believe
that, I don't know if it's necessarily AGI that's going
to happen, but you will have hyper-intelligent machines that are very
task-specific. You've got hyper-intelligent machines today that are
very task-specific to cybersecurity. Yes, there are generalized models
that can activate certain parameters to execute whatever.
But going back to this value case, fundamentally, I
think looking at what your organization does, Gooder AI,
you guys are starting to measure model outputs and comparing them
to the value that they generate from a business. Well, how do you do that?
How do you determine what models? That's the crazy thing is
that, so those types of predictive applications, learning from data to
predict who's gonna click, buy, lie, or die, commit an act of fraud, or outcome
or behavior that's valuable to predict, to target marketing,
credit scoring, fraud detection, and an incredibly
long range of use cases, right? This is the technology you turn to,
or the way of using machine learning that you turn to
to improve your existing large-scale operations and has been around
for decades. It's older but not old school. Most of
its value is still largely untapped. And a core
deciding factor in that, the real problem, is that
people don't measure and plan for the business
value. So this is the crazy thing after all these decades.
And I'm seeing this story take place all over again with the advent of Jev,
where we're taking those types of predictive use cases into the modern
era. By using foundation model and foundation model technology
and enabling zero-shot classification, zero-shot prediction,
which is unprecedented and has a lot of upside. It's still a lot of also
potential benefit by making a customised predictive model. But either way,
I'm seeing the whole thing take place again before my eyes after sort of
decades of kind of shrugging and trying to figure out what can we do
about this, which is that People decide what are they going to predict, and
it's going to potentially make a decision, right? Which transaction to audit
as potentially fraud, which credit applicant to
approve, which large language model-based,
so-called agentic system to allow to move forward
autonomously or to escalate to a human. Whatever it is, there's
millions of decisions that need to be made, and prediction's the holy grail.
or that is to say, putting odds or probabilities of the holy grail of doing
that. And with the advent of Jive, it's easier to get started.
There's a much lower barrier to entry because of that
zero-shot nature of being able to, you know,
you can say, hey, I'm gonna look at this customer service
transcript and decide, hey, is this customer expressing a
risk that they're gonna defect or cancel, right? And I can just start
finding out, a probabilistic odds of that
yes/no question, binary question, which is often
what you want to predict for these use cases, without any
of the standard data science type of work that's been going on for decades. Right.
I don't have to make a custom, I don't have to prep the data, which
is usually the technical bottleneck, and then do
model training. Now, doing so very well might
improve greatly, depends on the project, but at least I can typically get
started at almost zero cost. It's zero-shot, like, ready to go.
OpenAI, Amazon, NVIDIA, Databricks, Cloudflare,
and a whole bunch of open-source efforts are very much competitive.
The comparisons often show us at least a slight advantage in
certain cases of GeV. GeV has done it very well, but this is
to say that that's not the only game in town. So you're getting all this
attention to it. And again, I'm unfortunately seeing the same old story,
which is that people just talk about comparing how well these things
predict or how well their particular initiative did in terms
of accuracy. Accuracy is almost
always the wrong metric. It's only another technical metrics in the
data science and predictive analytics enterprise machine learning
field. It's the typical technical
metrics are area under the curve, precision, recall, F-score,
accuracy, and there's a lot of them. And most of the data
scientists are familiar with all of them, but what they're not familiar with is the
need and methodology to also estimate the
potential value in very straightforward terms like profit and savings.
That's what we're doing— It's all outcome-based. Good or bad. And if you're not
measuring potential value, you're not pursuing it, you can't be
planning the deployment in order to maximize that value, and perhaps more
importantly, if you can imagine, you can't sell it. And that's
why most of these projects actually fail to deploy, because the
business stakeholder, person in charge of the, operations meant to be improved with the predictions
can't authorize it when all they get from their data science
quant counterpart is the area under the receiver
operating characteristic curve is 0.83. Isn't that awesome? Yeah, it is
awesome because it means the model is sound. You've learned something real from
data. It predicts better than guessing, relatively
speaking, which is generally the
most we can hope for. We don't and we'll never have a magic crystal
ball. So predicting better than guessing turns out to be valuable for improving these large-scale
operations. and does have an impact on those
straightforward business metric bottom lines. But
these technical metrics say it's potentially valuable. They don't tell you
anything about how valuable it could be. Let's also
estimate and plan for the actual profit savings or
whatever KPI matters for the project. So what, you know,
and you bring up a really interesting point, by the way, and I follow the
JEV space very closely, right? You know, we've personally started
using Luna's predict model. Amazon Strands is decent, right?
Strands has got a decisioning model that can be hosted on a laptop, and it's
actually pretty— What's Luna? I don't know. So Luna, so GPT, you know,
OpenAI's got 3 sort of classes of models. They've got the Luna class model, the
Terra class model, and the Soul class model. I thought the OpenAI competitor, Jev, was
called like Decision something. So it's called Luna Decision. It's
based off of the Luna model itself. Luna is their tiny model, right? So what,
what a lot of these frontier labs are doing, they're just taking their tiny models
and fine-tuning. Yeah, and slapping on the head. Right, right. And that's what Quinn has
done, right? Quinn's got a— Yeah. Nice little 4B model that I think
is really, really interesting in this space. Amazon kind of
went, went back and said, you know, we're gonna have a model that can run
on your laptop called Strands Decisions, which
is really, really cool that we've just started playing with. But, you know, you can
run on your freaking laptop. I've run it on my MacBook, which is really, really
interesting. Which is typical for a predictive use case, right? Right. Exactly. The
fingerprint's much I mean, using more and more
sophisticated analytics and even using a full-on large language model to help with the predictive
project in various ways, or even serve as a core part of the predictive model
itself, calculating the odds, can help, but there's very much diminishing
returns. Again, who's going to click, buy, lie, or die? We're never going to have
a magic crystal ball, and we're playing odds, we're getting probabilities,
and those are really valuable. They're the next best thing to a crystal ball. They're
not a technical concept. It's just a number between 0 and 100 so that you
can order, prioritize, and triage according to which— Right. is likely to have whatever
outcome pertains to the decision you're making. It's really straightforward. It's not rocket—
the rocket science is in how you learn from data to do that, or how
a large language model operates, or whatever the analytic core analytics are.
But its output, these predictions, the fact that it's a number
instead of just an absolute confident yes, no, that's only a good thing. It's
not a big technical gotcha or whatever. It's really, really valuable
and useful. But its ability to calculate
those odds a bit more precisely increases with
very much diminishing returns. So yeah, you really end up oftentimes with these
relatively lightweight models. And if you take like a foundation model,
change the way it's being used, you only have to iterate it once rather
than generating language,
and you change the head to do a binary probability or whatever
the classification problem is. That's a lot more
heavyweight than your standard classic predictive models,
even though those can get kind of hairy with ensembles and such, or
small neural networks, and yet still much more lightweight. So being
lightweight is very much part and parcel
to predictive use cases in general. You know, it's interesting, right?
Because generalized predictive models are— so we've
been writing predictive models forever, right? Whether they're SVMs, whether they're neural nets,
Their heavy use cases are always in analytics, right? When you've got a
crap ton of data that you've got to go through and come up with something
that's useful, some metric that you can calculate, some metric you can
group by, that's always what they've been used for. But I mean, JEV is
still exponentially more expensive to run than
any small predictive model, right? Mm-hmm. We've got hundreds of predictive models that run on
our systems to do things like intent classification.
Jev on average is 160 times more expensive and
50 times slower because it's not— it's
generalized, right? It's a general-purpose model. Um, so I, I think—
So there's a big spectrum here between, right, a classic machine learning
model, which could be pretty darn sophisticated,
to something like one of these Jev or its competitors,
to a full-fledged large language model. Really wide. But even when you're in the
middle, in the Jev range, you're doing it on your laptop potentially. No,
but it's, it's also, it's super interesting, right? Because all of this has to
map to a relative value. When I say relative value, it's
we use neural nets because we get, I don't know, 26 billion calls a
month. You can't use anything else on top of that, right?
Because if I use Jev, I would never be able to make back enough
money, right? So my data scientists are very concerned with the ROI
of the output that they produce, right? That's how we pay them. So I, I
think this sort of connection between the data science and business is one of the
large reasons why essentially all AI projects end up failing, right?
Because everybody wants to do this cool. They don't understand what's actually
useful versus cool. I mean, let's be— let's not say
100% fail, right? We're gonna lose like something like— listen, we're gonna
lose some listeners. Yeah, but there's a big difference because even if
Only 15 or 20% of projects succeed. There's a
lot of projects, so there is a lot of success. It's just that when you
take a little step back, you're like, this is a dismal track record,
stands to improve greatly, and it's not a
technical problem, it's an organizational dilemma. Well, I mean, what was the
latest statistic from Microsoft, right? There are, from a, from a
GenAI perspective, something like 95% of all projects still
fail. Right, because people are— Is there new— is there new— because I know obviously
that came out a year ago from MIT. Yeah, it was a year ago. It
was a year ago. I don't think they've done new research now. A year, you
know, a year in this space is essentially— Well, there's certainly a lot of— there's
a lot of press and headlines and white paper— vendor white papers, but still
sort of seemingly vendor-neutral white papers, but also a lot of press and
press releases that are in one way or another really trying to
push back and say that that has changed. And A lot certainly has
changed. Like, just your typical
large language model is definitely harder to jailbreak. I mean, I know that from
personal experience because I like trying. I mean, it's pretty rare even to
this date that you interact with an
LLM-driven customer service agent
as a consumer. Like, it's pretty— they still
aren't widely released, that's for sure, but a little bit. And so if I come
across one, usually for more of a medium-sized company, 'cause the
large companies are gonna be more picky and selective about constraining the
behavior. Then of course, I'm just like, let's jailbreak this, right?
So, you know, a year ago it was pretty easy to be like, hey, are
you allowed to give me financial advice? No. Should I
diversify my portfolio or invest in individual stock? Diversify, definitely
diversify. But I thought you said you're not supposed to give me financial advice. Oops,
sorry. Right? So that was like typical a year ago. Now it's harder to get
it to do that. That doesn't mean that, It's achieved
reliability up to snuff, but it's better. I think hallucinations
now are more nuanced than ever, right? That's why predicting and preventing
those hallucinations are actually very, very difficult, and why
using a model like JEV on outputs is actually very, very useful. For sure. I
mean, for like a year now, I've been referring to that
as hybridizing predictive and generative AI. So predictive and
generative, you know, really aren't 2 different kinds of technology. They're really
just the 2 main categories of use cases of machine learning.
Learning, and they are destined to remain inherently
distinct because a predictive project where you're dealing with odds and
using them systematically to drive individual one-off
decisions, micro-decisions, the things that constitute the main
large-scale operations, that's just a very different kind of endeavor. It
requires a very particular discipline to get the project
actually socialized, understood, and deployed, just like with
generative AI systems. They usually fail still, even though this
type of endeavor is decades old. So there's a certain professionalization and
maturation culturally that still hasn't been achieved.
We're working hard to change that.
And then within that realm, it's not about a hallucination, because anybody
involved with these projects has come to understand we don't expect supreme
reliability because we don't have a god or some prophet or
something. We have the next best things, which are probabilities. We can play the
odds, tip the odds better in these numbers games known as business,
tip the odds in our favor, and makes a tremendous difference. You know,
one of the big things I've seen at large organizations, I think you were kind
of spot on about large organizations being picky. I thought,
you know, I saw this quote in the airport from Workday, which was like, what,
a model that is 99% accurate is still 100%
not useful. I don't believe that, honestly, but
large organizations— No, that's an
exaggeration in the wrong direction. It's anti-hype. It is, it is. It's totally not
true, especially with predictive, 'cause that's the name of the game. I mean, you
identify a group of transactions, some
segment of credit card transactions that have a
3% fraud rate.
That's a very, very high fraud rate. You've just found a real hot pocket of
fraud, but you don't have high confidence of any individual
Even within that hot pocket of definitely being fraud. So the numbers games,
right? So this is what I just said is the real bare
bones, simple, straightforward, non-rocket science
arithmetic that most business stakeholders have not come to learn.
So, and it's part and parcel to what it means to
understand, gain value from, and lead to actual successful
deployment, a predictive AI project. You know, but structurally, it's
still hard to, if not virtually impossible, to convince large
organizations. And you're right, there's not a lot of generative AI bots out from huge
companies. It is very difficult to convince them to
have something non-deterministic in front of customers, right? Well, it
didn't respond exactly the same way 2 times in a row, so it's hard for
me to test, right? Of course, you can run evaluations and simulations that say
98.9% of the time it's going to be exactly the the thing that you
want, but it's still very difficult to convince
leadership. Well, but then those, those particular leaders are never going to
launch anything with an LLM on that basis, right? That's just inherent to the
technology. Um, but,
um, the— they need to be, they need to be redirected to what the real
problem is, which is it's not reliable enough. Or is it?
Let's actually measure it. Let's not rely So
for both predictive and generative, we are still to this day
relying much too heavily on
standard benchmarks of performance that are at best
very poor proxies for value. And in the case of large
language models, the benchmarks are how good is it on a
public testbed of
tests and questions, which is— Right. And those leaderboards in there,
it's heavily competitive and they keep getting better. And that's super interesting,
Interesting may be a dirty word, right? Interesting is the word in academics that you
use to get grants or they compliment colleagues. Doesn't necessarily
mean valuable, but at least
subjectively, it's cool, right? It's interesting. Fine. What actually
matters instead of that, the benchmark that is spot on related to value in the
case of those types of, let's call them agentic projects if you have to use
that word. I don't like that word. I think it I
don't think we should be pretending that we're giving agency to a machine, and I
think it's hype. But, um, uh,
the, what we should instead do is be measuring how well the
system performs at the task at hand on examples
that are representative of how it'll be used in the field. So very much customized
measure, see how good it is. And then, so if
somebody's like caught up, oh, it's got a random element to it, and therefore it
has that unpredictability, it'll be different each time we test it. Like,
they just need to be socialized into the reality of the nature of the thing
and instead be said, hey, look, why don't you test it and see what the
numbers tell you? And if it makes a relatively grave error
of some sort, whether it's hallucination, a transaction it shouldn't have authorized,
divulging sensitive information, or what have you, 4% of the time,
that could easily be too often. And indeed, it will be shelved. So
99%, maybe it's more likely to not be shelved. But there is a
certain reality to that sort of exaggerated quote you gave a moment ago,
which is that it takes only a relatively small incidence of error before the whole
system's not viable, therefore can't be used, can't be deployed, and will
not realize any value. The answer to that is now what
the JEV flurry has brought to lots of people's attention,
which is what I've been referring to as hybridized, which is that you're putting predictive
reliability layer and you're making these decisions about escalating to a human,
that more expensive human in the loop, And that way we're compromising this audacious
promise of autonomy. We're reintroducing, no, it's not fully
autonomous. We have a human, but they can be judiciously used.
Some people refer to JEV models and that class of models as a
judgment model, just a way to rebrand what has always been
predictive— Observer model, yeah. Predictive analytics, enterprise machine learning. Now we're calling
it a judgment model or a decision model, although that name was
already taken. So it's going to be rebranded. That
happens. Anyway,
put in a judgment model, predictive model, to
choose just the highest risk cases. Again,
the same issue for any predictive project. We don't know for sure which of these
interactions are going to go bad, but we can put that risk
level and then funnel off to that more expensive human in the
loop, the human expert, somebody to intervene
potentially, or at least to inspect, maybe
let's say 15% of the time. And if they do
that, and you can assume the human is not going to repeat the kind of
error the LLM would make, at least the vast majority of the time,
then the net error rate might get down to like 1%,
depending on how you define error. And that might make the system viable.
So that's the next killer app for predictive
AI. That's absolutely— it's absolutely necessary
to realize a good portion of this
audacious promise of autonomy that feeds the hype. Domain
expert evaluations is really what you're talking about, right? Evaluating an
agent at the specific task that it needs to execute. And I'm a big
believer in that, right? I think, as an example, what we do here is all
about engagement resolutions, right? Are we resolving a conversation? I
really don't care how well a precision score or an AUC score
or an SWE score ends up on an LLM. What I care about is, Did
it call the right tools to be able to resolve this conversation, right? Did the
password get changed at the end? And I think that, that's the most important thing.
Mm-hmm. But, you know, going back to predictive, and it's, it's really, it's so cool
how JEV and other sort of these, these kind of technologies are kind of bolted
onto LLMs. Have you seen this new experiment where this, um,
this guy essentially hooked up an LLM head onto
JEV? So it gave JEV predictors of the next
character, right? So it just says, He would ask it a question, is the sky
blue? Or what color is the sky? And he would essentially just pass it the
alphabet each time, and it would say,
T-H-E, the sky is blue. It was really, really interesting, and
it kind of— but it really kind of helps explain how these technologies have shifted.
Well, but that sounds like they kind of— they took an LLM, they handicapped
it, and then they forced it in a sort of— Well, it's a decision tree,
right? Less elegant way to act like an LLM again.
That's exactly what they did. But it kind of really shows how both of these
technologies have diverged from the same root and the same tree. Right. I think
that that's really the most interesting piece of it. Yeah. And that's what makes it
so unique. And I really think the hype behind Chev is
justified. It's really, really freaking cool. I think it's
justified for sure because it's this much long-awaited
renaissance. Not that predictive AI ever went out the window. We have lots of reasons
to believe that still most of the enterprise value that's being realized is
for successful predictive projects and that half the enterprise
projects are predictive despite all the pressure to
make hay of large language models. And if Jev is helping
put it back in the spotlight publicity-wise, and hopefully get more
investments, and maybe even venture investments in predictive, 'cause predictive, it's
only bad for everybody for predictive to be sidelined to the
degree everybody's enamored with the seemingly human-like aura and
charm of what's undeniably sexy and
incredible large language models. I mean, you can use Luna, you can use
Terra, you can use SOL, you can use any of these large language models generalized
predictors. They're slow, they're expensive, they're not meant for the task. And if you
chop off all the, you know, the
inference passes, right, you get JEV. So they are the same technology, which is what
you have to convince the leaders of, right? They are the same underneath, underlying model.
Right, that's another thing that should help with the publicity of predictive use cases.
Exactly. It's a category of project, right, that we need to not forget about at
all. It needs to be more front and center. It never
fully achieved its necessary professionalization and level of maturity as an
industry. Before everybody got inundated
and distracted and overflowed with focusing almost exclusively on generative
AI, at least in the headlines and board meetings.
It needs that extra oomph, and this could very much help. And the fact that
it uses the best, most modern technology under the hood,
you know, at least that helps advertise it and does help with many
of the projects. Yeah, by the way, one of the— so Jev
did a lot of things that are a little slick in terms of the way
it marketed and publicized. Maybe that's a good—
I mean, it's cynical to say that's a good thing, but it was effective, right?
And I'm glad to see attention back on this. Right. One of the things is,
for example, they don't just— I mean, the idea of binary classification,
that is the machine learning for all the projects
I've listed so far in this call, right? Fraud classification and all that kind of
stuff, or fraud detection, all that kind of stuff. Binary is the most
popular type of projects. And the fact that when you're doing binary, No,
the model doesn't just say yes or no. It always gives a probability. Well,
Jev's publicity says, let's give it a new word. They made up a new word,
noul, N-O-U-L, noul, as if it's a new
concept when it's like Stats 101 from 1975,
right? I'm like, oh, come on. So, and then another example is
that they say they, their models have bleeding edge
because they're, they're, um, they're trained
entirely over only a proprietary source of synthetic
data that they created. Which has to be entirely not true
because it has to also be built. Maybe that's true for the sort of
classification part and how they get the
outputs calibrated and this kind of thing and sort of optimized in certain ways.
Right. But it's built on a large language model. There's no question about that. It
takes English as input, which was, you know, I don't
know which, nobody knows. It's a black box, right? It's proprietary.
And it turns out it's not that hard to get pretty close to JEV-level
performance building from open source. People predict— people are predicting that
it's based off an older Quin 3.2-ish
model, something like a 4B, and it's highly, highly
fine-tuned to the specific text. But, you know, going back to the marketing aspects of
it, a lot of today's world is about marketing hype.
There's so many cool projects, so many cool companies and startups that
fail. just because they don't have the marketing hype. And I think that that's the
bigger case here. You know, succeeding at GenAI and
predictive analytics, it's not hard. I mean, you can— software, in my opinion, in today's
world is not that difficult, right? Applying it to the right problems
is difficult, but a lot of people apply it to the right problems and still
fail because they don't have the right marketing chops.
And you mean marketing chops also just internally, as far as like,
I think it's externally. I think it's, you have to create a hype cycle in
today's world in order to be successful. Well, to be
successful and for people to notice. Um, but I
don't want to be that cynical. You know, I want to believe that when you
speak to value clearly and you're like, hey, this will improve
a lot of things, including profit. Like, you know, that should
be enough. Like, okay, then prove it. Fine.
The proof's in the pudding. I think it's more about SNR, right? The
signal-to-noise ratio, right, that we have today of
technology is difficult for anybody to understand. I mean, you— I
mean, I'm sure you follow along with every model release out there. I try
to. No, actually I don't. I don't. I don't watch the leaderboards that
closely on a day-to-day basis because I just sort of feel like I'm much more
interested in the broad narrative and it's There's only so many hours in
the day. So I've actually— it's a choice. But what you just said about, well,
it's got to be— the marketing's got to be slick. I mean, that you didn't
use the word slick, but if I put words in your mouth and I ask,
well, does it need to be slick internally? Because there's
many organizational initiatives where they're like, okay, let's
improve our fraud detection model, or let's target this retention. Let's do
some churn modeling to try to do targeted retention. We could really
Lower our attrition rate of customers. Initiatives like that
get greenlit and then don't get deployed routinely,
right? That's and that's all entirely internally.
So I'm wondering whether your perspective, which I I don't want to
believe but might have to believe. I'm wondering if your
perspective does also apply internally. So my
take on the internal and the theme of my entire second more
recent book, the AI Playbook. These projects fail because of
a lack of connection and collaboration and speaking the same language
between tech and biz, between the data scientists, the quant, and their business
counterpart, the stakeholder in charge of the operations meant to be improved. And they only
talk in terms of technical metrics, and whereas the business people have to
get involved in a semi-technical, accessible, relevant,
fully accessible understanding of
what's predicted, how well, and what's done about it, and dig into the details a
bit. If they don't get their hands dirty, their feet will get cold. And that's
what happens. It stalls, it doesn't get deployed, they can't authorize it, they can't integrate
it. So that's the practice that I
espouse in my more recent book, The AI Playbook. And it's about the
clarity and getting everyone on the same page and expectations
management. But it is valuable. That's not too hard to understand. But
how, a little more concretely, do you turn
less than perfect prediction into actual dollar value?
That's also not rocket science. It's very particular arithmetic. It's very
rarely known, at least in terms of business stakeholders, which is going to
set them up perfectly for stalling the project. The failures are
routine. They get swept under the rug. So I'm espousing clarity,
explicit value calculations. That's what I did in the book. That's what
we do at Gooder AI. You're
espousing the right marketing
spin. I wonder if my work
needs a little more of a song and dance. Like, what do you think?
So, you know, I think it's a combination of the two. And I, you know,
there's a difference between internal AI projects and external AI purchase.
And I think what people don't understand is that the internal data scientists are
competing directly with salespeople and external
organizations that know how to sell the value For dollars,
right? They're not necessarily selling, here's a model that's 29%
accurate. There's the— this— the external organizations are selling
the direct value. Okay, they're saying, pay us $1 million and we'll
save you a ton. Okay, so I see what you're saying. That's a great— you
gave me a real absolute answer to my question. Wait, who's interviewing
whom here? So no, you gave me a great answer to my question
as far as, um, I'm like, does it also apply for internal?
You're like, yeah, because the internal projects are competing
with the external vendors who know how to do the slick walk, right? That's a
great answer. And I'm like, oh no. So let's talk about one.
Let's make it a little more concrete. Another way that Jeff spins it,
which for an old nerd like me who's been in machine
learning for 35 years is really annoying.
The way, the way One of the spins is like, Jev
does not hallucinate. Well, yeah, of course it doesn't hallucinate because it
only— the word hallucinate pertains only to large language models. If you're just
putting a probability on whether this is fraud and
the probabilities are going to be between 0 and 1 or 0 and 100, right?
0 and 1 and usually in real. But either way, it doesn't— what's the difference?
There's no— of course it's not always going to be correct, and that's the
nature of the prediction game. That's inherent.
Hallucination is not a concern that's on the table. So bragging
that Jev is a technology that somehow manages to not—
it is such a ridiculous pose. I mean,
but it works. Such a bad brag, but I guess it works, right? I mean,
well, I mean, look at how— look at the hype cycle, right? I mean, it,
you know, I think, I think internal— and I see this all the time,
right? We compete internally with data scientists, and the reason that we win is that
we we see the outcomes at the end, we see the business
outcomes, and our entire lives aren't dedicated to
doing AI for the sake of AI. It's to achieve the final end
outcome. And that's why oftentimes we'll end up winning against the internal
data science teams. It's super interesting. I think marketing has a
huge aspect. And when I say marketing, it's a little bit tricky, right? No, but
that's not slick marketing. That's the whole actual— I know. That's great. That's the
right pitch, which is like, Don't follow solutionism.
Don't fetishize the technology for its own sake. Don't have an AI strategy
that's like having an Excel strategy. Solve a specific problem, measure how well it
works. That's not slick marketing. That's like prudent marketing.
That's good. But that doesn't count as an example in
the debate I'm describing. I guess you're right. I guess you're right. But it's more
like positioning. I think hearing about— and I think OpenAI is a perfect example,
right? Everybody wants to work with OpenAI or Anthropic because they
sound so sexy. Right? They want to do projects with
OpenAI, even though you could work, you could do— Well, in general, like, just go
to Hollywood, right? It's a similar thing with business. Like,
celebrity begets celebrity, just like power begets power. And
so the more famous you are, you're automatically, right? It's a
certain, it's a kind of royalty thing. Um, it's modern-day
royalty in this country and some countries. So there's just
a reality to, there's only so many slots. in the
celebrity roster on any given year. And so that's
inevitably a superficial cosmetic thing. Yeah. But my question is
about the messaging itself. How do you spruce up—
do you have to tell what's essentially a lie,
such as, like, it avoids hallucination? That is effectively a
lie because it intrinsically implies that hallucination would
ever be possible for that type of endeavor. Do you have to lie
to spruce up your marketing? I wouldn't go as far as lie, but I think
it's more about how do you be politically astute enough
to understand the room and what's going to win the room? It's all about,
you know, internal politics at a company. You have to understand
them really well if you want to get promoted at a company, right? I mean,
promotions work the same way, right? You've got 10 people with relatively the
same attributes or same aptitudes. Mm-hmm. Who do you pick, right?
It's the person that markets themselves the best, right? And marketing
can come as I said in many different forms. It's catchy statements, it's how you
present, it's how, you know, 10 data scientists coming to a
CTO or chief AI officer asking for money,
asking for $3 million, or an outside third-party
company with slick salespeople that understand the problem saying, hey, give
us $6 million, And we'll give you $15 million in
result over the next 3 years. What are you gonna end up picking, right? What's
your bet? Um, yeah, so reading the room, right, which is kind of
antithetical to what it means to be a nerd, right, to be a techie.
And then yet at the same time, the nerdy persona has a
salesy attribute, right? Like, I'm a super nerd and I present, I'm
sure, to a lot of normies as a nerdy, but I
don't think— I'm pretty self-aware. I don't think I present nearly as nerdy as certain
certain famous techies who, where that's really a part of their
persona. Right. And the fact that they're so
freaking nerdy, uh, somehow
seems to give them addition, like that automatically means they must be smarter.
Well, it's like wearing glasses. It's like, wow, that guy looks pretty smart. Right? Yeah.
Yeah. I'm telling you, I think, I think marketing is a large piece of
success in today's world. That does it for this episode of Speak to an Agent.
Thanks to Eric for bringing value back, literally. If you found the conversation valuable,
subscribe wherever you listen and leave us a rating or review. It helps more
listeners discover the show. We'll be back soon with more of the people making the
AI agent the one you actually want to talk to. Thanks for listening.