Beyond Chatbots: How Twilio Uses AI to Strengthen Human Connection | Vinnie Giarrusso

Show Notes

Can AI assistants actually enhance human connection?

As Season 1 of Chain of Thought comes to a close, Conor Bronsdon and Vinnie Giarrusso (Twilio) explore the transformative potential of AI assistants in the workplace.

Discover how these assistants function as "async junior digital employees," taking on specific tasks and contributing to the organizational structure. But will AI assistants ultimately replace human connection? Vinnie argues the opposite is true, suggesting that AI can liberate employees from mundane tasks, allowing them to focus on building meaningful relationships and providing personalized experiences.

This thought-provoking conversation takes a philosophical turn as Vinnie explores how AI could revolutionize education while potentially disrupting traditional mentorship roles. He shares his vision for a future where AI democratizes information and empowers individuals to personalize their learning journey. Finally, learn how Twilio and Galileo are partnering to shape the future of AI and what this collaboration means for both companies.

Chain of Thought will be taking a break for the holidays, but we'll see you back here on January 8th for the start of Season 2!

Chapters: 00:00 Twilio's AI Agent Platform

06:34 Ensuring Accuracy and Trustworthiness

09:49 Challenges and Failure Modes

17:39 Future of Fully Autonomous Agents

22:18 Human-AI Collaboration and Mentorship

31:24 Education and Democratization of Information

32:58 Partnership with Galileo

39:54 Conclusion and Season Wrap-Up

Follow:

Conor Bronsdon: https://www.linkedin.com/in/conorbronsdon/

Vinnie Giarrusso: https://www.linkedin.com/in/vinniegiarrusso/

Show notes: Twilio Alpha: ⁠https://twilioalpha.com

OWASP GenAI: https://genai.owasp.org

What is Chain of Thought | AI Agents, Infrastructure & Engineering?

AI is reshaping infrastructure, strategy, and entire industries. Host Conor Bronsdon talks to the engineers, founders, and researchers building breakthrough AI systems about what it actually takes to ship AI in production, where the opportunities lie, and how leaders should think about the strategic bets ahead.

Chain of Thought translates technical depth into actionable insights for builders and decision-makers. New episodes bi-weekly.

Conor Bronsdon is an angel investor in AI and dev tools, Head of Technical Ecosystem at Modular, and previously led growth at AI startups Galileo and LinearB.

[00:00:00]
Introduction and Welcome
Vinnie Giarrusso: I think that there's a lot of worry about agents taking work away and resulting in like a less personal experience. But actually, I think it's, I think it's the opposite,
I think that there's a huge opportunity here to use these assistants in taking away the boring technical work and the sort of like mundane tasks,that would ordinarily be done by a human. Instead of letting them take that over and like what the human can really focus on is that human to human interaction,
Conor Bronsdon: Welcome back to chain of thought for the last episode of season one. I'm your host, Conor Bronsdon. And today we have a special guest, Vinnie Giarrusso, principal software engineer at Twilio, Vinnie, welcome back to the show.
Vinnie Giarrusso: Hey, thanks, Conor. Thanks for having me here. Excited to hang out today.
Conor Bronsdon: Yeah, I'm really excited to have this conversation with you, learn more about the [00:01:00] work you're doing at Twilio with AI agents and to have you back on the podcast for listeners who may have listened to last week's episode, we actually featured Vinnie as part of our panel of guests at Galileo's productionized 2.
0 conference. he had conversations about applied AI lessons from deploying AI at enterprise scale. and his work at Twilio, but there's lots more that Vinnie's working on around Twilio's AI assistance.
Twilio's AI Agent Platform
Conor Bronsdon: Can you start by giving us some background on how Twilio is currently using AI agents?
Vinnie Giarrusso: so Twilio, we're building a low code,autonomous agent platform. And so what that really means is we're building a set of tools for our developers to deploy their AI systems at scale. Cool. and leverage all the other cool Twilio stuff that they're, you know, used to getting from us. You know, we provide APIs on top of this.
We manage the deployment. We manage the invocations. we manage all the sort of like backend,heavy stuff for you. so really what we're focusing on is providing a lot of value to just getting an agent [00:02:00] out there,testing it, asking it questions around your domain and just seeing, how fast people can go from zero to one, essentially.
Conor Bronsdon: it sounds like Twilio is really empowering its customers to build their own agents with this framework you're establishing. Can you add a little depth around how exactly Twilio's tools provide for this purpose?
Vinnie Giarrusso: Twilio's agent platform is, like a fully featured agent platform. So we give you, the ability to use tools, we give you the ability to use knowledge sources, you can upload your documents, you can crawl web pages,you can do all of that sort of thing. We handle the memory management for you.
and on top of all of that, we, we put an API layer in front of it as well. So you can handle all of your deployments through the API. You can do all of your uploads through the API. You can talk to your assistant through the API. we actually call them Twilio AI assistants. The, the industry term is, is agents.
but yeah, I mean, so, so we provide all that on top of, you know, the, the Twilio API. [00:03:00] And then, you know, the other great thing about Twilio is we have all the channels as well. So not only do we have this,AI platform that we're building.
So that our, our developers can get in contact with, with their customers. But we're doing it in a way so that you can use the same Twilio channels that you're used to using, SMS, voice, WhatsApp,conversations, all of that sort of thing. all of those are natively integrated into our platform as well.
So not only is it a platform for, building the agent, but also that next step as well. It's like, what do you, okay, so now you have this agent that can do some cool stuff like. How do you get it to talk to people? Um, the answer is the same way that, you know, the rest of your Twilio apps work,
Conor Bronsdon: I like that Twilio is differentiating a little bit from the rest of the industry and saying these are AI assistants.
But I want to learn a bit more about how Twilio actually arrived at this approach and this tech stack to build agents. What were the lessons you learned along the way and how did you get to where AI assistants are today?
Vinnie Giarrusso: regarding the assistant versus agent thing,that definitely was an [00:04:00] intentional decision on, on our part to lean more towards,assistants. we really feel like, especially at the beginning, these agents are going to play sort of a side by side role,with humans.
So these agents necessarily aren't going to,replace the human role, but instead sort of augment that human role with additional capabilities that give it sort of like superhuman powers. and that's, that's sort of how we're, we're thinking about it and we're framing it.
they can interact with the humans,who were maybe tasked for, accepting or rejecting,certain actions by that agent or however you have the setup. that's where our mind is around, uh, assistants versus agents.
Conor Bronsdon: Yeah, I think you're, you're spot on that there are these stages that are happening. And a lot of folks want to jump to this idea of fully autonomous agents that are The equivalent of high level employees.
And there's a lot of work to do along the way. a lot of the hype around AI has been that, Oh, it's this [00:05:00] magical solve for you, but there's still a lot of work to be done to set up non deterministic systems in a way that actually works for businesses.
Vinnie Giarrusso: I mean, we see agents that are fully capable of doing amazing things today. You know, there are, of agents that can use computers. You know, recent releases of agents that can generate videos and long form videos and things like that. and that's amazing. I love that innovation.
Like, that is truly incredible to see some of the innovations that are coming out. But when we think about sort of like where the enterprise is sitting,the enterprise is going to be like a little bit more careful and in our approaches, we're going to be a little bit more measured and conservative and what we put out there and what we expect our agents to do.
and so we're, we're really leveraging that. we know that our customers are expecting to be a little bit more careful. And, and I love this sort of like side by side, you know, with, certain parts of the industry pushing forward really quickly, other parts of the industry looking at that and saying, okay, like that's really cool, but I think that might be like a year [00:06:00] out for us, but keep doing what you're doing.
And we're going to focus on this thing over here until like all that stuff gets settled out. I think that that's probably going to be like the general trend for a little bit now. everyone is,is working with the new latest and greatest and it's pretty amazing. But when it comes to like actually getting the work done for businesses, it's a little bit of a different story.
It's a little bit less of the wild west, so to speak, you know.
Conor Bronsdon: And I know enterprises in particular have to really consider. because they already have established distribution, established trust with their customers, accuracy, trustworthiness.
Ensuring Accuracy and Trustworthiness
Conor Bronsdon: how is Twilio actually ensuring that the AI agents or assistants that are being built through its platform are accurate and trustworthy?
Vinnie Giarrusso: This is a tricky one for us, actually. So for us as a platform, it's not quite as straightforward as us being able to run evals and say like, this works, release it. We do do that in some cases. So we, you know, run evals on like a set [00:07:00] of, cases that, you know, sort of generalize across, you know, some set of our customers, however,you know, we can't predict how people are going to use our platform.
We can't really predict how people are going to want to use it. And so what we see happen is,and we see new use cases come in and, you know, they might use different knowledge sources than we've seen before. You know, those knowledge sources might be formatted in a different way or contain information like kind of far away from each other.
and in that way, it gives us a lot of viewpoint into like what is accurate and how accuracy really relates to the specific application. because at a base level, you know, the base models, You know, we expect to accurately answer random questions, right? but for a business, those questions get very specific.
Those questions can be specific to, like, you know, how much does a text message cost, in Spain, for example, right? and that's a much different question than, you know, how much does a Text message from, you know, another country, maybe. And in that way, like, [00:08:00] accuracy is not so much, like, does the agent accurately answer the question,you know, how much is a text message, but it has to be very specific to, where the user is, and maybe the user hasn't, told them where they are.
And so the agent might need to be smart enough to say, like, oh, well. I have, you know, this list of pricing, but I'm not really sure exactly what it's referring to because I'm not sure which one of these to pick. And so then the agent, you know, instead of saying, like, I don't know, or, you know, picking one at random, accuracy really comes from the agent being able to turn around then and say, like, Oh, actually I have, you know, a variety of pricing options.
Pricing. Um, do you happen to know where, this is going to be located or do you happen to, have any more information about that? And so for us, like accuracy is a little bit more difficult to track down because it can come, in the form of a very specific question that gets a very specific answer, or it can come through a series of, you know, messages back and forth, how accurate is it over a course of a conversation as it's trying to figure out exactly what his job is and exactly what it's supposed to be doing here.
[00:09:00] Mm
Conor Bronsdon: really appreciate that detailed example. Since I think it illustrates well some of the challenges with agents when they get into production, there are all these edge cases that are just hard to correct for when you're building your initial wireframe and kind of getting things going and not until you start getting the different Approaches and data from users.
Do you really realize, Oh, we need to correct this piece. and it
speaks to the need to continually be evaluating and continually be improving upon your applications, your AI agents, are there particular . Failure modes that you've noticed as common or challenges that you're seeing that have to be addressed, whether it's in pre production or in production
Vinnie Giarrusso: s
Challenges and Failure Modes
Vinnie Giarrusso: o one of the biggest challenges for us has been the sort of like rag retrieval pipeline. it's pretty easy to go to, you know, like the Lang chain docs and get like a rag pipeline [00:10:00] going very quickly. but as you mentioned, as you start to scale it out, as you start to add more knowledge sources and more use cases, and then you add users on top of those sources that are asking questions that maybe you hadn't thought of before, this is really when the challenge starts to Uh, you start to notice things like Well, the, the agent couldn't figure it out and we're not really sure why.
Right. So we will go into our reservability tools and, look through that trace and see like, okay, like how did it, what did it pick out? Like what, did it even pick the correct knowledge source, first of all? And if it picked the correct knowledge source, like what did the chunks look like? did the top K come back in a way that was set it up for success to answer that question.
And then a lot of Examples that we've seen that's not always the case where it comes back with a great set of top chunks, or it's not always the case where it picks the correct knowledge source, for example, we're relying on. A couple of things to help that one is like the amazing research that's going out by, so many of the teams out there that are putting out [00:11:00] white papers around this sort of thing.
You know, there was a great one about contextual search put out the anthropic which gave us a lot of really good information and a lot of great ideas and there's like a lot of work going on in this area. And it really shows sort of like how big this can be when you start dumping you know, 10, 15, 20 knowledge sources in an LLM, like the context becomes really important.
Like you, it's when you first deploy an application and you start testing it, it's pretty easy to be like, Oh, that, that works really well. that works well for several use cases. but. It doesn't often generalize super well. And that's, that's where we've seen the need for, especially like context aware,you know, retrieval has been like a game changer for us.
that is like increase the accuracy of our retrieval. I don't want to give any hard numbers, but like a lot,a positive amount that makes us excited. That's one of them. Like knowledge, knowledge retrieval is like one of those big areas that we've [00:12:00] seen,vagueness and failure modes abound.
and it's a tricky one,but there's a lot of really great research going on and it's getting better and better every day.
Conor Bronsdon: I don't want to generalize too much here, but I think this is something where we really underestimate the work that needs to be done to do this well at times. And the way I kind of think about this is, look, we as humans or as an LLM that's having a conversation. We may have a framework that we come in with of like, Oh, this is the knowledge I have.
This is how I would approach this problem, but the devil is in the details. And, you know, there's a reason this phrase always here. And to your
point, getting that context in and making sure that context is held throughout, is so crucial. And I'm curious about the specific. Processes or tools that Twilio is using for experimentation around this for monitoring it for debugging.
How are you approaching that process with these [00:13:00] agents?
Vinnie Giarrusso: Well, for, for monitoring, primarily we use Galileo. you know, we've, we've been using Galileo observe, Basically, since day minus one, it was one of the very first things we integrated. It was one of the very first tools that we, you know, had set up and that was, that really set us up for, for success. having that visibility from day one really,you know, allows the team to have good ideas.
And so for us, like, Monitoring and debugging also have like a direct impact on innovation. because when we see these failure modes happen, if we're not, logging those appropriately, or we're not really like capturing the things that are happening, we can't quite figure out. Exactly what happened, and we can't also quite figure out exactly what we should do about it.
so Galileo has been like a critical part in, in helping us get through that. especially with, you know, the,with the rag retrieval pipelines and stuff like that, like all of the observability tools in there have been super, [00:14:00] super helpful for us to, to get in there and, and figure that kind of stuff out.
Vinnie, I would love to understand More about how Twilio has established best practices around either your approach to evals and observability or the tooling you're leveraging to upgrade this entire process. How are you approaching this?
there's two sides to it. Uh, one side is how we approach evals, you know, for ourselves. and the other side is, you know, how do we want to expose that to our customers. And, in a lot of ways, there's, there's quite a bit of overlap. And so, a lot of what we do at Twilio is we'll take something that we see,we'll build something around it to solve our own problem, and we'll see,What we need to improve on, you know, what areas does that actually solve?
And then is this something that maybe our customers could use as well? And so with our evals, we did exactly that. we started with, building sort of like an open source framework for ourselves to use around evals and this was specifically [00:15:00] for when we started this it was for security evaluations but it has since evolved into evals more generally What we found was that, this was really useful for us.
and we, we wanted to open source this so that our customers could also use this. And so we have a, an open source package now,that's based out of the,Twilio alpha, GitHub,and NPM namespace that, allows our customers to, run evals,up against our own AI assistants. And we feel that this is really important because we can run evals ourselves and we can say, it does X, Y, and Z under, you know, A, B, and C scenarios.
you know, some percentage of the time. but it's a lot more convincing when you put it in the hands of people to play with themselves, when you put it in the hands of people to like use their own use cases and ask their own questions and like really refine that for themselves and start to figure out like, where are the gaps here?
where the gaps may be in my knowledge sources, where the gaps may be in my, my prompting strategy [00:16:00] and. do I have all of the context,for my agent to be able to accurately answer this question? Like, maybe you find out that you don't through your evals and maybe, you have some work to do in other areas.
So, putting this out, has been really helpful for us. it's helped us shore up a lot of things on our side and we, we hope that people get a lot of good use out of it for themselves too.
Conor Bronsdon: are there particular insights that you've already drawn from the way Twilio customers are using AI systems today?
Vinnie Giarrusso: you know, one insight is that,I think in most cases, these agents, these assistants are going to be used side by side. And so a lot of our initial customers right now are, maybe not thinking like we want to, fully deploy these and just, you know, let it go ham. I don't think anyone at this stage really wants that.
you know, what, what we're sort of like putting out for people is the ability to say, like, okay, here is our goal. Here is, what we want our assistant to do. Here are the things that it has access to. I need to go test that, right? that process is really eye opening for us because [00:17:00] we can't know in advance,how people are going to use our platform.
We can't know in advance what use cases are going to come our way. And honestly, we are pleasantly surprised, like, all the time. Like,every time, you know, uh, you know, someone new comes to use cases are always interesting. it's like, oh wow, like that's super cool. Like,you know, when we think about assistants, we tend to think about, maybe a customer service assistant or something like that.
But a lot of the use cases we see coming our way aren't necessarily that. and especially some of the ones that are more forward thinking,get us really excited for, for like what's about to come.
Conor Bronsdon: Let's dive right into that forward thinking piece.
Future of Fully Autonomous Agents
Conor Bronsdon: Obviously today, a lot of agentic systems are built with a human in loop in mind where there is this validation step, but increasingly we're seeing a rise in multimodal systems. We're seeing people experiment with, frameworks that have agents talking to one another and supporting one another. What are your thoughts on what the future looks like for [00:18:00] fully autonomous agents?
Vinnie Giarrusso: Well, I think, in the long term future, like fully autonomous, you know, in the sort of like short to medium term future, I think what that looks like is, you know, we're going to have these assistants be able to, do tasks on their own, they may be scoped to do, smaller tasks and have access to maybe another assistant that has, scoped to smaller tasks and its own goal of itself.
there might be a layer above that, that's sort of managing this cohort of agents,that has very specific things that it can do. And I think at least in, you know, the short term, like, one strategy,I think could probably work well that I haven't tested this. I'm literally just sort of spitballing on the future.
Um, is sort of like a stop mode or like a, like a request help mode, you know, for the assistant. Like we want the assistants to be fully autonomous to a point , right? Like there are things that an assistant can be trusted with [00:19:00] to do. just kind of fully on their own. And then there are things that at least today, assistants should not be trusted to do fully on their own.
especially in the early phase. and so a pattern that I think we're going to see sort of evolve is, you know, have these assistants working, doing their thing, and then they'll get to a point where they're like, Oh, actually, I need, you know, for this step, I need human supervisor approval, or, I need to check in, Regarding the series of steps that I took,and what that looks like on the other side, like, who knows, right?
Like, you know, maybe, you know, it sends a recording of what the assistant was doing and then the human can overview and, and whatever. And then while that's happening, You know, we'll call you back in, in, in a couple of minutes or like, hang on the line while we approve this or, we'll send another correspondence.
I'm not really sure what that looks like. But,I think that measured approach, at least in the short term is sort of where I think we might be going. but definitely in the future, I think, these agents will be talking to each other. from one system to another system internally, uh, you know, even [00:20:00] internally to other external systems.
And then those agents talking to other agents like this is going to get complex pretty quickly. And I think that it will really model and mirror kind of how we do things today. and I think that, you know, similarly to how we do things today, where, you know, we don't always have,the maybe necessary permissions to do something or it might be outside of our job role specifically to do something and we have to, you know, maybe go over to another business function to, get that taken care of and worked out and take a look at.
I think that that's probably how this is going to end up where today we're, we're super stoked that, you know, assistants and agents can do basically anything and everything, but that that's probably not how it's going to, you know, for a while now.
Conor Bronsdon: I absolutely agree. I think a really good framing for the short term of these AI assistants or AI agents is async junior digital employees, where there are certain tasks that they can go jump on and there are certain things you need to give them [00:21:00] feedback on
Vinnie Giarrusso: Yeah. I love that.
Conor Bronsdon: yet.
Vinnie Giarrusso: I love that. What did you say? async junior digital employee? Yeah, that's
Conor Bronsdon: Async junior digital employees, because we're, we're creating this kind of. digital employee class that are these AI agents and the goal with it is, Hey, I'm not going to jump on live to solve their problem, but I'm going to, I'm going to get to your point, a failure mode, or, uh, I'm going to have a, a check in point where I go, Oh, here's how I correct.
Here's how I adjust. and I think where we're going to increasingly have to spend a lot of time. Is not just on the kind of challenge of scaling that individual agent to senior level. That's a ton of folks are working on that problem. There's a lot of thinking happening there, but also in the. Structures of how those async junior digital employees function within the larger organization. If I'm managing a team of 10 AI agents doing different tasks, that is a lot of human overhead that I have to then put towards [00:22:00] making sure all that is railed properly. So how can I then have. That hopefully more autonomous senior level agent that we are developing.
That's going to have multiple steps, now start to be the judge within that,framework, how, how does that all work together? it's going to be a fascinating mix
Human-AI Collaboration and Mentorship
Conor Bronsdon: and something I've had a conversation with folks about a couple of times is also how this plays into our broader hiring practices.
I think we're already seeing this within tech where. A lot of junior employees are kind of struggling to take that next step because people in tech are hiring senior, I think, in part with this anticipation of the future of saying, Hey, I want more senior folks who I can scale with systems who I can scale with. You know, AI tooling and who can really help solve these problems for me. And there's going to be this interesting problem for us where we've trained a lot of junior engineers. We've trained a lot of junior product marketers. We've trained a lot of, [00:23:00] you know, junior technical PMs who are coming out of college or coming off their first job. And. Getting them to the level of, Hey, I can now be a senior who's managing this team of agents, this, this team of different processes that we've built up from these digital employees. How, how do we get them to that next step? And right now, I think we're, we not only have this challenge around how do we get all these agents working together, but also how do we make sure that our. Our junior folks within the tech ecosystem have that opportunity to scale so they can also start managing these teams of agents, which I really think is the future, at least in the midterm.
Vinnie Giarrusso: Yeah. I think that something interesting happens here too. I think that there's a lot of worry about agents taking work away and resulting in like a less personal experience. But actually, I think it's, I think it's the opposite,
I think that there's a huge opportunity here to use these assistants in taking away [00:24:00] the boring technical work and the sort of like mundane tasks,that would ordinarily be done by a human. Instead of letting them take that over and like what the human can really focus on is that human to human interaction, right?
Like everyone's had a tough time on you know A customer service call where it seemed like they didn't really understand like your problem I mean this happens to me. I don't know half the time when I, when I call, I'm not going to say any particular company, but I'm always like, okay, actually, you don't really understand exactly what I'm asking.
And I understand that you're sort of like reading from the script and you're like trying to figure out like how I fit into this like playbook that you have, but that's not what I really need. And so what I think that's going to happen is like, when we do have these human to human interactions, the human that's there is not necessarily worried about like, oh, did I memorize like All of the, you know, technical, you know, things that could possibly happen before this.
And, and like really by the time that problem gets to you, like hopefully the, the [00:25:00] AI assistant has like sussed out some of those technical problems. And now maybe like what you're dealing with is like, I have a human to human problem right now. I have a, I have a human that has like an actual issue and we can be like a little bit more like empathetic and compassionate at that moment because we're not necessarily worried about like, you know, ticking off all the boxes on our sheet that says like, did you ask them this?
Did you ask them this? Like, have you gone through this? Like we can safely assume like by the time it's gotten to a human, like those things are true. And you know, maybe you have a report that The AI assistant like generated for you when you, get that human call tells you what happened before that.
But I think that there's like, a huge opportunity to, have a really positive experience and, really build some positive experiences that we haven't had the opportunity to before,even simply because that human agent just didn't have time to sit and talk to you about the problem.
Conor Bronsdon: Yeah. I think you're spot on. There's this
massive opportunity space [00:26:00] here to create more human interactions by automating all these, I'll call them annoying tasks in a way. And we have to be intentional about how we approach this. but it's a big opportunity. And I think, you know, hopefully for folks who are maybe more on the junior side or listening, I'm not trying to scare you. I think there is this big opportunity that Vinnie's describing where if you are building up the right skillset here, and in particular, I think if you are leveraging AI to help you learn,and by that, I mean, not just the agent approach that we're talking about, but like, Hey, if you are onboarding to a new company. Throw it all into your favorite GPT or cloud or whatever you want to use. Like have it help you synthesize what you're doing through? Have it help you learn that there is such an opportunity to scale ourselves with machines today.
And that opportunity is only going to continue to increase. And to your point, we have to be careful to ensure that we're not.
getting rid of these human to human interactions instead of enhancing them. but it's a really exciting thing that's happening right now. And
if you're in [00:27:00] this space today, I think you are in absolute the right position to make an amazing impact on what the future of work looks like and what we get to do as a species in a lot of ways.
Yeah.
Vinnie Giarrusso: I mean, that is my hope, right? Like my hope, there's definitely a lot of, you know, sort of fear around, like, how is this going to impact, the future of work? and that is a big question. It's a question we should definitely be asking ourselves at this point.
but, you know, I choose to sort of believe that, This will open up a lot of personal interactions for us. It'll open up a lot more human to human interactions, in a way. Even though, you know, maybe some of it is going to get taken care of, just straight up by an assistant. Like, if you have a problem that only requires, you know, Hey, can you check my thing?
Cool. Like, that's great. but in those moments where you really need a human, maybe now you don't have to wait an hour. maybe now you don't have to wait, so long to get ahold of somebody so, I'm hopeful for that and, and, you know, with regards to like the the junior [00:28:00] engineer out there who's, you know, maybe wondering how does this, impact me.
the opportunities are massive at this point, like, if you can be, you know, amongst the first to, you know, stand side by side with AI and use these new tools to your advantage, like, this is going to open up a lot of doors, you know, it's going to open up a lot of opportunities, the, the speed at which you can learn,is unparalleled today, the speed at which you can, you know, learn.
Write. Code, um, is unparalleled today. The speed at which you can get feedback is unparalleled today, but that doesn't mean that we don't need, that senior mentorship anymore. one of my colleagues,Dominic Condell was talking about this,in a court that he had where, the senior to junior,engineer relationship, There's an opportunity for this to kind of go away, unfortunately, like, like the mentorship aspect,might start to slip away.
And, you know, on both sides, like, It's good for the people [00:29:00] who, like, really have a lot to learn, because for the first time really, like, you have a personalized, like, mentor. you have, you know, someone, uh, not someone, but, uh, something. You know, some, some being of some kind.
Conor Bronsdon: them. It's happening. Yeah.
Vinnie Giarrusso: It's already happened, yeah. You know, you have some, some, some system that you can interact with, that can teach you personally, it can sort of really get to know you and,know, where you might need help and where you might need guidance, but that is certainly not a replacement for, the direct interaction with, with a senior engineer.
And, you know, for the senior engineers, it's really easy to forget what it was like to be a junior engineer. it's really easy to forget what it was like before, like you were an expert in the thing that you're an expert in. even though like now we're here, in this stage with, the ability for our colleagues to just kind of figure things out, on their own with the help of AI.
It's not a [00:30:00] replacement for, that mentorship. And I, I really liked, you know, what Dom said about that because that's something that, has been on, on my mind. As well, like, not only,our other industries sort of grappling with, like, what does AI mean for the future of our job,you know, as engineers, like, we have to ask ourselves the same questions.
Like, are we eliminating? roles for junior engineers. Like, are we eliminating the historical mentorship that goes on? you know, even in just the conversations that we have day to day, like when we start having those conversations with AI, it's like, I'm not necessarily like sharing all of my conversations that I have with my team, although I do share some of them that are either like funny for some reason, or like spectacularly good for some reason.
but it's easy to like, let that opportunity to slip away and,It's good to keep in the back of our heads too.
Conor Bronsdon: Yeah, I think as we continue to spend more and more time making sure AI has the context it needs and it's bringing in the right context windows, we also need to be that intentional about our human interactions to your point where it's so easy to [00:31:00] overfocus on, oh, here's a technological solution and, you know, We'd be careful not to let it eat away at the established work we've done to build these great mentorship relationships and build these learning opportunities.
And I absolutely agree with you. I think there's a lot of work to be done there on the future. And I know there are a lot of smart people thinking about it, but I appreciate you getting philosophical with me and kind of making sure we address it here.
Education and Democratization of Information
Vinnie Giarrusso: I mean, the education space is one that I'm particularly excited about for AI. Like, I come from a non traditional sort of computer science background myself. And, you know, on my bookshelf right over there, you know, I can peep at a number of, you know, textbooks that you might find in your college classroom, right?
And when I was going through those, I kind of went through them on my own and I wasn't really, I mean, these books are thick, man. Like, it's like, what do I even look at here? Like, how do I even like start to frame a context around this? Like, I have no idea. and that's even with the advantage of having access to those books.
Right. So like, we have to start from the place where like. [00:32:00] I was advantaged enough to be able to buy that book. You know, I was advantaged enough to be able to read that book and, like, understand that book, and, you know, go ask my colleagues, like, Hey, what, what does this mean? Like, what does this symbol mean?
Like, how do I interpret this equation? I'm super excited about the way that this is going to change education. Like, I'm super excited about the way that this is going to help teachers. I'm, I'm really excited about the ways that, like, You know, kids and,adults are going to be able to use these systems to personalize their own education on things that are important to them.
the democratization of information is something that I'm super excited about to see evolve,in the space. And,education is certainly one of the biggest ones that I think is going to be,a really cool sort of area of this advancement.
Conor Bronsdon: Totally. We are in this period of kind of reestablishing and rebuilding norms around how we approach information systems in this new era. And there's massive opportunity and a lot of work to be done.
Partnership with Galileo
Conor Bronsdon: and with that context in [00:33:00] mind, I would love to understand more about how you see that partnership evolving between Twilio and Galileo.
Vinnie Giarrusso: I mean, I hope our partnership is one where we, we continue to push each other forward. you know, we, we continue to work together and look forward at the advancements that are being made and say, like, how do we integrate these into our system? And how are you integrating them into your system?
And what is the Delta that we're sort of missing? and that, that's one of the things that, like, I've loved working with the Galileo team is that sort of back and forth around like, hey, like, we saw that you're using, you know, this thing, like, How is it going? Like, how is it working for you, you know, and then us, you know, coming back and saying, well, you know, we really like that, but we'd also like this other thing that we happen to notice while we were using it and that's sort of back and forth is like, like, it's such a great relationship to have between, you know, two teams of, of like really talented people because like, we can go, we can go quick, you know, we can go fast.
We can, we can sort of like hit on things and, you know, especially, you know, with where [00:34:00] we sit in terms of like, the amount of use cases we see and the customers that we see. And As it relates to like multi modal and all those things and multi channel, like, I really look forward to, to pushing the envelope with y'all and I really look forward to seeing like, What can we do next?
how far can we go? Like, how far can we bring, you know, visibility into these platforms? How far can we bring testability into these platforms? How far can we bring security into these platforms? Like, let's go, you know? I'm so excited about, that. Like, um, the Galileo team,is like, obviously, world class, right?
And it's such a privilege for us to be able to, to work with, that caliber of people,and to be able to push the industry forward together. And, and I really hope that, you know, we continue that,going forward. Yeah.
Conor Bronsdon: right back at you, Vinnie, it is such a distinct pleasure having the opportunity to learn from the work Twilio is doing and with that in mind, I'd love to understand more about how Twilio is leveraging Galileo today. In [00:35:00] your process and how you see the potential for that to continue to evolve.
Vinnie Giarrusso: Primarily we use Observe,and Observe for us is like, you know, I was trying to think about this, this earlier, and I was like, what does it, what does it mean to us, right? And the best, example that I can think of is that like, Galileo is sort of like our head chef in the kitchen.
You know, like, we have, we have a bunch of servers out front. literally servers, right? But in the restaurant equation, they're also servers. So this works pretty well. We got a bunch of servers out front that are sending stuff in, you know, they're taking stuff out, you know, whatever. And when, you know, if you've ever worked in a kitchen, when something comes back, uh, to the kitchen, like, that head chef is right there, right?
That head chef, that executive chef knows everything there is to know about what's going on. You can ask them, like, hey, is there some random ingredient in this dish? And the answer is, you know, whatever it is. But, you know, of course they know. They also know how to do the job of, like, everybody else in the kitchen, right?
So your executive chef can, you know, in the same way that they can [00:36:00] act as a line cook, they can also act as, you know, they can, get the food going between the expo and the front of house or, they can even, take on, other roles in the kitchen. Galileo is like, that's, when I think about like, at least the observe platform, like the ability to just go in there and see everything and not only see everything, but be able to gain insights too.
Like the insights that we get from, from the Luna metrics, like we're, we're always in there looking at one thing or another, right. Either it's, you know, context adherence or complexity or,accuracy, or, you know, any of the tone, like that sort of thing. Like, that's really important. and you also sort of get that, you know, from a head chef too, right?
Like you, you can also in the same way, you can go to the head chef and say like, Hey,can you, uh, remake this dish? Or, Hey, can you tell me how you did this? You can also ask a head chef for insights into like how they might improve on that you know, use one ingredient over another or why one ingredient works [00:37:00] better in this case than another ingredient, right?
And I realize I'm taking this example pretty far, but,
Conor Bronsdon: I love it. It,
Vinnie Giarrusso: but, um, you know, that's, that's sort of how I see it. Like, Like our, your observe platform is basically the go to for any question that we have from any level of our team, right? Like from engineer to PM to business partner, like everybody who has access to Galileo has a specific reason to go in there and look for something.
And they have the ability to find what they're looking for. and so for us, that's invaluable because we don't really have any other tools that. Scale that well across a team, you know, there are tools that work well for our engineers And there are tools that work well for our PM's and there are tools that work well for our business partners There are very few that work Equally well across all of them And so for us like that's that's like critical for us
Conor Bronsdon: I love to hear that. I I'm so glad [00:38:00] Galileo has been able to help fill that niche for you. Has Galileo also influenced Twilio's approach to AI agent development and evaluation?
Vinnie Giarrusso: Oh, absolutely. Absolutely. the visibility that that gives us provides us with the answers that we need. and in the same token, it also provides us with,areas that,are opportunities. Like, for example, like, you know, if we're looking at something that has a low, chunk adherence or a low context adherence, like we, we can figure out why, and by figuring out why, that shapes the way that we sort of like frame the question to ourselves.
why did this happen? It's like, well, that happened because our, you know, maybe our assumptions about what would happen in that case, turned out to be wrong. Or, you know, we expected this to happen and in a very small percentage of cases like this happened. And so, you know, leveraging those insights gives us the ability to [00:39:00] take like a,a more rounded approach to like how we're moving forward and what types of things we might,expose to people.
For example, like the ability to, put of our customers is a direct result of, our team,trying to gain those insights out of,a bunch of other stuff, and so like we, you know, for us, we take a look at that and we say like, okay, how do we build something around this?
And, and even if we do, what is sort of like, what are we trying to get out of that? And what are we trying to put in the hands of our customers and what do they want out of it? And so in a way, like all of these, like observe and protect and evaluate, like they're all interrelated.
They all play a role in the way that we think about a holistic product. they play a role in how we think about the future,based on what we see today. So really, I mean, all of those are like critical to how we think about things and how we plan,
Conclusion and Season Wrap-Up
Conor Bronsdon: It's been such a joy, you know, having this conversation with you, Vinnie, and I know our whole [00:40:00] team at Galileo is so thankful to be able to work with you and the rest of the team at Twilio on this as we build the future of AI. It's so exciting to have such an incredible partner in the space and to see all the work that you and your team are doing around AI assistance and beyond. thank you so much for coming on the show today. It was a distinct pleasure to have you.
Vinnie Giarrusso: Ah, thank you so much. Uh, you know, we, we feel the same way. And,it's been such a great ride with you guys and, really looking forward to seeing what we can build together.
Conor Bronsdon: Yeah, I think we're just getting started. And I would love to give our listeners an opportunity to learn more about what you're up to or the work that you're doing at Twilio. Where is the best place they can go?
Vinnie Giarrusso: Yeah,for my team,twilioalpha. com, that'll give you sort of like the latest up to date and what we're working on. you know, we're releasing blog posts and papers and, you know, new, new code and,experimental code and all sorts of stuff. that's the place to go to, to check out what's going on our team.
I also work with the OWASP foundation. there's a lot of security initiatives going on,in that space as well. Um, so,
vinnie-giarrusso_2_12-16-2024_140544: [00:41:00] genai. owasp. org.
Vinnie Giarrusso: is another one, you can go to find,the latest work, uh, that's going on in the security space as well.
Conor Bronsdon: Fantastic. Vinnie. Thank you so much. We will link all of this. everything that Vinnie's discussed will be in the show notes. So listeners make sure to check it out and everyone. thank you so much for tuning into season one of chain of thought. This is the final episode of this first season. So this is, this is our wrap moment. we'll be back on January 8th for the start of season two of the podcast. And we've got a lot of incredible guests in mind. We'll have to have Vinnie back for sure. And I think there'll be a lot of folks who you may recognize from X or from LinkedIn and from elsewhere, who you'll be very excited to hear from. A huge shout out to all of you for tuning in and for everyone who has supported the show, your likes, your reviews, your comments, we've gotten such positive feedback and we're learning so much from this process. So thank you. as I said, we've got big things in store for season two. things are not going to slow down for chain of thought and we'll be there to help you and your teams navigate AI throughout 2025. Thank [00:42:00] you again. And Vinnie, thank you so much. It's been a fantastic chatting with you.
Vinnie Giarrusso: Likewise. Thanks a lot.
Conor Bronsdon: to you all January 8th. Happy holidays, everyone.