Subscribe
Share
Share
Embed
Season 1 Episode 5 | Small Bets, Big Trust | Selin Güven
Why Deloitte's Selin Güven Says AI Adoption Needs Patience
Deloitte's Selin Güven says trust is earned in small bets, not big rollouts. From hallucination risk to what a bot should never do, she's seen it all. Her advice: start small, get it right, then scale.
🔗Connect with Selin:
https://www.linkedin.com/in/selin-g%C3%BCven-9a651415b/
https://www.linkedin.com/in/selin-g%C3%BCven-9a651415b/
___________________
Timestamps:
00:49 — From Chatbots to AI Agents
00:49 — From Chatbots to AI Agents
03:10 — Building the Right AI Foundations
04:30 — AI Won't Fix Broken Processes
11:03 — Measuring AI Agent Success
14:04 — How AI Transforms Workforces
19:27 — Where to Start With AI
____________________
About Speak to an Agent:
Speak to an Agent explores what it really takes to bring AI into large organizations, why AI can't fix a problem you haven't already solved, why real understanding matters more than a polished voice, and why the domain expert who knows how to use AI is becoming the most valuable person in the business. The show features the leaders, founders, and technologists building and deploying AI at scale, and they get candid about what's working, what isn't, and what changes now that machines can reason, plan, and act at a human level, and sometimes beyond it. If you're responsible for putting AI to work inside your organization, this is a conversation you won't want to miss.
- (00:49) - From Chatbots to AI Agents
- (03:10) - Building the Right AI Foundations
- (04:30) - AI Won't Fix Broken Processes
- (07:25) - AI Guardrails and Security
- (09:25) - Testing AI Agents at Scale
- (11:03) - Measuring AI Agent Success
- (14:04) - How AI Transforms Workforces
- (19:27) - Where to Start With AI
Creators and Guests
What is Speak to an Agent?
Speak to an Agent explores what it really takes to bring AI into large organizations, why AI can't fix a problem you haven't already solved, why real understanding matters more than a polished voice, and why the domain expert who knows how to use AI is becoming the most valuable person in the business. Featuring the people and ideas shaping the future of CX AI, the innovators, the thought leaders, and those making the decisions. They get candid about what's working, what isn't, and what changes now that machines can reason, plan, and act at a human level, and sometimes beyond it. If you're responsible for putting AI to work inside your organization, this is a conversation you won't want to miss.
AI isn't magic. It also won't fix your
problems. It would just make it worse if you do not solve them
before. Welcome to Speak to an Agent.
Those words used to mean one thing, get me a human. Now,
the agent people actually want is AI. It's faster,
sharper, and never puts you on hold. If that surprises you, you're
exactly who this show is for. Featuring the people and ideas
shaping the future of CX AI. The innovators, the thought leaders, and
those making the decisions. Comparing notes while the next generation of
technology gets built. You're not just listening, you're in the
room. I'm Niraj Verma. Today's episode is guest hosted by Sasha Pogoman.
Joining Sasha is Selin Güven, Manager for Conversational AI and
GenAI Infusion at Deloitte Consulting. Let's get into it. I
built bots that really led you through the process,
and now it's really different, I would say, um, because
the whole agentic world really needs so much
adaption also that the agent adapts to
what the customer wants. And yeah, that really
changed also our approach, how we go into
projects and also talk with customers about their
expectations. Yeah. And you're basing that on a variety
of different experiences at IBM and then working for BMW, right?
Yes. Yeah, that was fun times. I've
like learned robotic process automation at first. built
really bots that were mostly in the background. They were not
conversational, right? They just automated business processes. Right.
And you had a clear rule set which you
implemented. And this is now
changing with conversational AI. And IBM was my first
step where I really got hands-on in
implementing chatbots. And this is where I got my first
experience and it made me really excited and
what the future of that looks like. Yeah, 100%. And, you know, there's
so many synergies between the robotic process automation
and what we are doing today in end-to-end transformation journeys.
But take us through the early days, like what you hinted
to, the chatbots as silos and what you found there.
Yeah, so chatbots in the past,
they really had a defined set of rules and really
a lot of if-else constructions. I remember those days.
Yeah, it was just, I think, but more easy for us as
consultancy to implement those because the
bot really had just one or two ways to solve a problem
and not 100 multiple ways. And yeah, it was,
I think, a little bit less stressful for us to really
see whether the bot is successful or not. Yeah, and in the early days you
had the bot as a silo, right? And now you're coming in with all your
experiences in robotic process automation. And let's
talk a little bit about the synergies there between data
integration, evaluation, and then the go-live.
Yeah, so for robotic automation Robotic process automation, sorry.
We have kind of a similar approach actually, because
once we go into a client and see what
their processes look like, we really have to know
what are the details, where are— where is the data coming from, what are the
interfaces, are your APIs ready? Because
this is actually the first step where we think
most of the clients do not think
about in the beginning, right? They have to have the data ready. They have to
have a great foundation to have the system
working. Because then if you have the processes defined, the
data is ready, everything gets so much easier. Oh
yeah. Yeah, it's not then that hard to
implement the bot because you know how
everything is working. And we
see in a lot of client projects that I got into that
the clients were not sure about their processes and we had to start from
the beginning. Right. I was gonna ask that, like when you walk
into a customer for the first engagement, what is the
uncomfortable truth you often have to share? Where do
you need to point them to or where do you need to pick it up
with them? Yeah. Oh, that's hard. I
think I would have more than one, but I would say
AI isn't magic and it also
won't fix your problems. It would just make it
worse if you do not solve them before,
right? That's why I also mentioned the foundation.
You really have to think about, do I have the
right data? where is my single source of truth?
We see that when we implement RAC implementations.
The knowledge in the company is distributed in
different applications, but where is actually the source
of truth? Yeah. We need to know that. And the fun fact is
you always kind of see
where the product is missing something, right?
seeing all of the edge cases than customers. When customers first try
the bot, you realize, oh, this is the use case I haven't
thought about in the beginning. Okay, now we have to add that. And
there is a lot of iteration going on, so it won't be perfect at your
first try. Yes. We need a lot of testing
and regression testing, especially in AI,
to have a really good solution at the end. Yeah. And
trust is a very hard thing to earn when you put such technology in
front of customers. Walk us through that a little bit.
What are you experiencing? Yeah, it was also
hard for me to trust it in the beginning. Oh yeah. So
we also had some downsides, or we saw them when
we first implemented our AI projects because somehow
it worked in 8 out of 10 cases. But then in those 2 cases, it
started to hallucinate. And this is really
important to communicate to the customer that you need
a longer time to perfect the system,
to have less hallucinations and to always
re-engineer your prompt. Yeah. And maybe even
in some cases do a hybrid approach.
That's what we do sometimes. to
reduce hallucination, to control the agent in their
behavior, right? To have some variables defined in the
background to see, are these really collected?
And then go on with the process. Otherwise, you may share some
personal data and hallucination should not be there in
that case. Hallucination is one thing, of course, but then there's
also others saying you always put a human in the loop, right?
That's also probably not the ideal solution, but a pathway
to get there. Maybe walk us through a little bit on the technical
foundations of evaluation, guardrails,
observability, and escalation design.
Yeah, so guardrails are really important, especially when you work for
large companies. You don't want to share sensitive data of your
customers. And what we do is really,
as I said, the approach that
you first start with your easy use cases
that are really repetitive and there is no
high risk of sharing anything that you shouldn't share.
Yeah. And then really think about how can we control the
agent in his behavior that in those cases,
hallucination or anything else that the bot should not
do does not happen. And what I think
is a learning also for myself from the different projects I was on, we
always defined what bots should do or should be
capable of doing, but not what should the bot
not do. Very good point. This is really important
because if you connect an agent to a large language model,
it basically knows everything, right? And it would share everything.
it's really important to say this is
information that you should not share, right? Or this is advice that you should not
share. We first really try to automate everything end to
end and see, are there some risks in the
process? And if yes, maybe first start with
forwarding it to a human and then see if you can somehow control
it. But otherwise it would just be too risky, especially
for a large enterprise. Yeah, you could probably do all of that in custom
code and design that on your own. But
walk us a little bit through on the capabilities that you're leveraging
from simulations all the way into managing
these guardrails so that you with confidence are able
to put out these bots into the wild. And maybe
also speak about not everything probably needs to be agentic, right?
Yes, yeah, definitely. Big learning also from us
because we do a lot of testing and testing became a huge
part of our process. It wasn't that much of a focus
in the past projects. Yeah. But now with agentic, testing
takes so much more time. Once you've developed something, a
small use case, and it works,
but then you add another use case and change the prompt, Yeah,
it can be that the first use case you've tested does not work anymore.
And the happy path. But also I liked what you said before, it's like
describe what the bot should not be able to do, right? Test for the
bad cases as well. Yes. So very in line with that. And we also
have like test automation going on and another large
language model or another agent is testing our agent that we've
built. Yeah, but also again,
there could be hallucinations also from the other agent that is testing.
So how can we test really consistently
is where we also put deterministic variables in there to see,
are those information captured and collected from the
agent, I would say. And when you engage in
these projects, you are
approaching it from a transformation journey, right? So you're
elevating processes to a new level, to a new customer experience.
Help us understand a little bit on how that works in
line with what we just discussed in terms of the testing and frameworks
and new processes that are being established.
Yeah, so what we do is first look
which use case is the most beneficial actually, right?
So we don't want to start with a use case that has no
business value. You want to measure the bot at the
end if it's successful or not. Right. You want to define the KPIs.
And for that, you really need
a great use case which may not have a lot of dependencies at
the beginning because you really want to go live really fast to test
it out, to learn from it, and then go back and reiterate.
And yeah, that then comes also to play with your team.
You have to have a team that right at the beginning
takes care of this bot and really goes along with
us. Because for us as consultancy, it's really important that we one
day can hand over the bot to other teams.
Yeah. And yeah, with that,
you are actually set to go. So we then start implementing the first use
case. see whether it's going good or not, and then
reiterating the whole process and then making it bigger and
bigger. And KPIs are really important to measure. So you
always need a team that just measures
the success of the bot and see whether it handles the
processes right or not. Yes. And as we've heard from Jack
Roberts here on Ferplatics, for example, sometimes the metrics are changing as well, right?
So you need to look at it much more holistically in order to do so.
Yeah, what we have implemented now, we always had the customer
satisfaction score, for example, but we
thought, okay, we do not only want to capture the
satisfaction. We also want to know how much
effort was it for customers to solve her or
his problem. Right. So what we've also integrated now into our process is
integrating the customer effort score. And asking how
much effort did it take you to handle the process with the bot.
Very good point. Yeah, but I think still the customer
satisfaction is another really great KPI that
really shows if your bot is successful. And what we always do is
end-to-end automation rate, because at the end of the day, how many
processes and how many use cases could the bot really
handle from start to end? Without any human in the loop.
Yes. Yeah. And that's a very good segue because you
have delivered some of the most prestigious projects that are
handling interactions end-to-end, all the way into the
backend services, all the way into outbound services
that, that are attached to it. And you're coming across
a lot of leaders who are thinking about what does that do to my
workforce? And maybe we can speak a little bit about that, on
how you consult these leaders in terms of
transforming their workforce. Yeah, so
it's, I would say, a different kind of work. And what we
do is really also define right at the beginning the
target operating model. Because at the end
of the day, their teams will take over what we've implemented
and they need to understand how everything is working. Yes. And
also all of the automation that is going on, right? I really
like that, the target operating model that you're building out, the
center of excellences within your clients in
order for them to self-serve and be less dependent on you
operating it for them. But at the same time, they are,
their workforce most of the time is also going through a massive
transformation and it oftentimes, it needs a partner
that helps them navigate not only the technical complexity
complexity, but also the transformation journey? Yeah,
there are actually a lot of things that we do there. Defining
new processes that go through the departments
and also new roles. You really have to have
AI experts and also testing experts. You have to have a
new strategy that goes on within your company.
So there is a lot of things you have to define in the
company. to make that successful also
after we leave as a consultancy. And
yeah, that's basically a lot of steps. I
cannot— I think I cannot list them all right now. You'll have to think about
data security, right? In all your decisions you have to take, you have to
think about governance. Oh yeah. So there are a
lot of aspects in the company that will
change after you go live with an AI agent and
Yeah, agreed. And you're operating in some of the most
regulated industries, right? It's not like you're doing all the
easy stuff, right? You're going where the real pain points are.
Yes. Yeah. And I think we see at
projects that especially the data security part
and having the confidence
that this AI agent and trust— we talked about trust in the
beginning— is there. still needs a little bit of time.
So we really have to teach the teams in the
organization, and what are the risks? What
do you have to do when something fails? And how does the
escalation path in an organization look like?
Everything, right, that needs to be defined in an
organization, we do. Yeah, yeah. And I mean, you advise
across industries and maybe from all your broad
experience there, Where are you seeing the most unglamorous
GenAI use cases that are creating massive values?
I think there are several. One is
especially— I think authentication is quite a boring topic,
especially for the customer service. They're just collecting
data, right? And you need to authenticate in a way
that you are really the person who is calling. to then continue
with the process. And with that, like an AI agent
that just collects some data of you and then just
forwards to a human that can handle the complex tasks is just a
huge benefit, right? It would just also save a lot of
time in the beginning. Very standardized, encapsulated, and then but high
scale, right? Yeah. And what we also do for a lot of use cases
in the beginning is RAG. knowledge
management. A lot of customers call because
they just can't find information. They don't know where to
find it. And if you just set up a really
simple RAG system at the beginning, you can already save a lot
of calls, a lot of time for the customer service.
And the bot can just handle those easy FAQ questions.
Yeah, but it's a very good point there. But what we
are also seeing at the same time, it's not necessarily
possible to put all knowledge on one layer, but maybe you also need to slice
and dice it per use case and make sure that you can segment knowledge properly,
right? So definitely. And
RAG is also a complex topic, could be,
because I had a client in the insurance sector
and the documents and contracts
for insurance are so complex even for RAG systems, you need a
really good structure for a bot to understand it. Yeah. And also there are
paragraphs that are linked to each other, right? You need to define a good
logic and system that can handle those
links between the documents. So you need also some metadata
to include there that the bot really understands.
Okay, I have this kind of snippet, but then again, this
other snippet is also relevant. to answer because otherwise
the answer would not be complete. It may be true but not complete, but we
always want to have the full answer. Yeah, makes sense. And
when you would speak to a leader who feels behind at the
moment, everyone is racing towards AI,
what could be a small first step for them to get
started? I would say always
think about the use case where you have everything
prepared and which is not that complex but gives
a big business value. This is something I
recommend. And when you go live with the first simple
use case, you learn so much. You learn so much about your customers.
You learn how they, how they communicate with the
bots, and then you can adapt, right? Then you see what are
the edge cases that we need to think of. Yeah. And this is,
I think, the tip I would give. Really think about a use case that is
simple but brings big business value. Could be that
authentication use case that we touched on earlier, right?
But then I think paving the way for a full transformation journey and not
having to rebuild everything, but really try to get
this established in what we discussed earlier, the operating
model, getting the foundations right with that first use case. is a
really good starting point and looking at it more holistically.
Maybe one thing that we haven't touched on yet from your perspective, is there
anything that you would give as advice or a topic
that you would bring up for the audience? I would say
don't be too fast in implementing AI
agents and having the expectation that everything goes
perfectly. You will learn from your first use case.
And then you have to reiterate and test
to make it a perfect system. And then you, you
take it one step at a time. I think this is what we are seeing
from all the projects. Everyone doesn't have like a golden
path to it, but they need to embrace on their own journey.
And it's like a little puzzle that you put in place one piece at a
time. Yes. Yet people can
learn a lot from others who have already embraced on that journey
and see what works, what doesn't work. And that's
probably something that you are also bringing to the table.
Yes, we have a lot of experience. We are working with like
many different industries, and I think every industry differs a bit,
right, with their products and how the processes look like. And
when you really dive deeper, you really see
that The bot— there is not a
standardized process for any AI agent. You really
have to communicate with the team, with your client,
how their internal processes look like and how their customer service at the end
of the day is working. Yeah. Because they
handle the processes so differently. They know everything about your customer
and their frustrations, where the process is
And it's really important to communicate to the
customer service and to the teams that currently handle the process,
that have that customer relationship, to learn from
them and then put that into your AI agent. And
probably in line with that, it's very important to also make a proper stakeholder
management, set expectations, pave the road— the
roadmap towards success. Definitely.
Expectations is a good word because we often
see that expectations are really high and we try
to fulfill those expectations. We just talked about that
expectation management with your stakeholders.
And but what's important is that you really take them onto their
own individual journey and help them navigate
through this and put the puzzle pieces in place. Yes,
and that's, I think, the challenge for us. I really like
working with different clients, having to solve different projects and
processes, and I think that's like our
magic that we put into the projects, right? And yeah, just
share our experience that we had with other clients and
build great AI agents. And not getting started is not an option,
huh? No, no, no, no. So we have to start now
transforming all of the companies. I mean, we're
focused on Germany, but, um, yeah, I think, uh, the
whole world needs to start and see what are the capabilities of AI.
That does it for this episode of Speak to an Agent. Many thanks to Selin
for keeping it honest about trust. If you found the conversation valuable,
subscribe wherever you listen and leave us a rating or review. It helps more
listeners discover the show. We'll be back soon with more of the people making the
AI agent the one you actually want to talk to. Thanks for listening.