Got Agents? Agentic Workflows & Architecture | Weaviate, Unstructured & crewAI

Show Notes

AI agents have quickly emerged as the next ‘hot thing’ in AI, but what constitutes an AI agent and do they live up to the hype?

Join Brian Raymond, founder & CEO at Unstructured.io, Bob van Luijt, co-founder & CEO at Weaviate, and João Moura, founder at crewAI as they discuss the shift to agentic workflows, dissect their architecture, and tackle real-world challenges in agent deployment. 

From data management tips to generative feedback loops, this episode is your essential guide to operationalizing agents effectively.

Chapters:

00:00 Defining AI Agents

01:16 Components of Agentic Architecture

02:16 Challenges and Solutions in Agent Deployment

03:58 Data Management and Quality Issues

05:23 Operationalizing Agents in Production

06:56 API and Security Considerations

09:04 Multimodal Information and Agentic Workflows

12:42 Future of Agentic Workflows

20:20 Best Practices for Agentic Strategies

25:30 Generative Feedback Loops

28:29 Agentic Evaluations

Follow:

Yash Sheth: https://www.linkedin.com/in/yash-sheth- Bob van Luijt: https://nl.linkedin.com/in/bobvanluijt

Brian Raymond: https://www.linkedin.com/in/brian-s-raymond

⁠⁠⁠⁠⁠⁠⁠João Moura: https://br.linkedin.com/in/joaomdmoura

Show notes:

⁠⁠⁠Watch all of Productionize 2.0: https://www.galileo.ai/genai-productionize-2-0

What is Chain of Thought | AI Agents, Infrastructure & Engineering?

AI is reshaping infrastructure, strategy, and entire industries. Host Conor Bronsdon talks to the engineers, founders, and researchers building breakthrough AI systems about what it actually takes to ship AI in production, where the opportunities lie, and how leaders should think about the strategic bets ahead.

Chain of Thought translates technical depth into actionable insights for builders and decision-makers. New episodes bi-weekly.

Conor Bronsdon is an angel investor in AI and dev tools, Head of Technical Ecosystem at Modular, and previously led growth at AI startups Galileo and LinearB.

Got AI Agents Panel Podcast; ft Brian Raymond, Founder & CEO; Unstructured.io; Bob van Luijt, Co-Founder & CEO, Weaviate; João Moura, Founder, crewAI; and Yash Sheth, Co-Founder & COO, Galileo
[00:00:00] Hey folks. Welcome back to the chain of thought podcast. Episode two. We want to start by giving a massive thank you to everyone who has already rated and reviewed the podcast for episode one last week. We've been overwhelmed by the support. Thank you so much. This week on the podcast, we're bringing, you got agents a session from our productionized 2.0 virtual event. It was recorded, live in front of an audience of over 2000 people from more than 60 countries around the world. AI agents have captured everyone's attention and imagination. Emerging as the next hot thing in AI. To help explain agents and why they've exploded onto the scene.
We're joined by Brian, Raymond, founder, and [e-mail]. Bob co-founder and CEO at V8 Joe Maura founder at crew AI. And ya'll chef co-founder and COO of Galileo is our session. Host, Josh, take it away. the term agent itself is Quite [00:01:00] overloaded.
Defining AI Agents
as we go deeper into it, I'd like to set the record straight and set a baseline of what are agents?
what do we mean by agents? And, who better than, than Joe here, founder of true AI, to, uh, tell us, about his perspective that, All right. So AI agents. That's a good question. I feel like everyone can like agrees and disagrees at it at the same time.
It's a little weird for me. It's very simple for me. Having an agent is about having agency. So regular flows. That is, if there's been that, that call. LLMs, that is not an agent for me. for me, an agent is something that controls the flow of the application. It uses a LLM as a brain to kind of like decide and have some reasoning on how to go about things.
But in the end of the day, it's deciding what tools to use it, when to use it, how to format the answer, how to go about a thing. And that would be, for me, the base definition of what an agent is. Like, uh, something that has [00:02:00] agency. Absolutely.
Components of Agentic Architecture
And to, to think of that, you know, there's a, you know, we think of agents as like a planning phase and execution phase, but when we go into the architecture itself, like what are some of the key components, and this is a question for all three of you.
when we think about building agents, what are some of the most essential components of the agentic architecture and it's, as we, you know, it's already been established as well beyond rag plus prompts, right? So, love to get your perspective, sir. Uh,look, I think the interesting thing is, is that, you know, this is a kind of a special moment.
We're almost at the 2 year mark. on the from the release of chat. And we started out by using just what's in memory in the models, right? And we dust it off that meta paper. And we hooked that sucker up, right to, uh, to all the crazy stuff Bob's been building, right? And the thing was alive, right? So he's He helped counter some of the three problems of just using what's in memory and the models, which that models are frozen time.
They don't have access to private data, and they tend to make stuff up. Right. And so. Rag [00:03:00] became, you know, a dominant paradigm.
Challenges and Solutions in Agent Deployment
interesting, like, you know, in order to actually, put these things into use, but, to actually drive towards like multi step process automation, which is kind of how I think about the business value of it.
Right. you need to take the models, you need to take the prevailing rag architecture, and then you need to be able to, set that into motion. I think that's, what's really exciting about what the folks over at are doing and others, which is. highly variable, depending on the types of agents that you're working on.
Right? and that you're wanting to set in motion. And so, you know, from like, you know, from that standpoint, it depends on like, you could think about like, what's going into like a Tesla and the self driving mode, right? As agentic, it perceives the world, it makes decisions, and then it takes action, right?
Which might be different. From a trading bot that a hedge fund, might use. Right. And so depending on the types of agents, right? different frameworks are needed, different model hosting and serving capabilities are [00:04:00] required, right? And so what going all the way down to like the cloud architecture of the hardware?
I think it's fairly variable depending on what, you know, what the actual business or operational use case is For the agent and like what data needs to make available to it, and a host of other sort of related issues. Bob, you want to go next? Yeah, I don't have much to, to add there.
So, uh, thanks for having me by the way, Yash, but I think what, Joe said is, is very interesting, right? So it's like a,if something has agency, so it can make a decision, are we going to go left or are we going to go right? Right. And rather than saying, if this then go left, uh, else go right.
There's some, some reasoning to it. Right. So the, I think it's, it's as simple as that. A
Data Management and Quality Issues
nd the reason why it's so important is because if we look at data management and those kinds of things, we kinda, We kind of got stuck in that world, right? So, low data quality, not getting to insights from your data.
That's just still the number [00:05:00] one problem with all the exciting stuff that's happening. So I recently had a conversation with a, with a CIO and he said, like, he said, it's all great. What you guys are building over there. So wonderful. He said, I still have like 15 ERP systems and I can't, you know, I can't make heads or tails of it.
Right. So if you can help me solve that, then that's it. And that's where, you know, the agents or the, as I like to say, the agentic way of doing stuff comes in. And that's where the value is going to come from. And let's bear in mind that. Rag, I, you know, I run with it, so I'm very happy with rag.
Right. You can, you can imagine why, but the thing is, it's also one dimensional or one directional I should say. And, and there's a lot of exciting stuff that's happening thing, you know, thanks to these agentic way of thinking, where are we going to loop that information back into the database, right. And then we really start to build smarter systems.
And that is extremely exciting. And that goes back because now I can go back to that CAO and say like, you know, I don't Maybe, there's a way to help you solve that, [00:06:00] that problem that you have. So it's a, you know, it's exciting times and that's what the agentic way of, of thinking is, uh, is solving. Yeah.
Operationalizing Agents in Production
And to add a few things to that, I, yeah, I think you got an ear in the head, like, uh, We definitely have seen kind of like a, a few plateaus as we go, but like, we keep like finding these other things that are the locks, more stuff. And honestly, I think like agents, like as the start are usually kind of like straightforward, right?
Like, Hey, I have an LLM. I have to, I have a task. I would give you an outcome. What I find though, is that as you bring this things into production, especially like an enterprise companies, then another, like a few other needs, Hard to pop up, and those are more like regular kind of like, uh, how can I say a regular technology needs?
We have seen this before, right? So it's like, oh, I want a caching layer. I want my agencies in the same tools over and over. Oh, I want a memory layer. So it's not only about the caching. It's about being able to do the rag. Pull the context, create kind of like a long term memory of everything that I have done and the [00:07:00] things that I have learned, so I don't need to keep relearning them and everything along the way.
Uh, and then also, of course, you go into the regular guardrails and you want to have delegation and you want to have everything in between. So, uh, I do find that these architectures, at least they tend to start simple, but once that you go into production, especially under your company, Security is another layer and distribution is another layer.
Dan, like you start to add a bunch of other components that kind of like can get the complexity to just skyrocket a little bit. Exactly. Right. I think that's, that's, this is great. Cause like I want to cover, I mean, data is the backbone of, our agent agentic flows.
API and Security Considerations
But I think, uh, what I'm also seeing is that, for example, APIs, let's say we want an agent to call an API effectively.
do we have a standardized spec for that API? Is that API, you know, changing over time? How do we make sure that the agent is able to call that API effectively? Like [00:08:00] these are. I mean, this is just one small example. Then, as Joe mentioned, like security permissions, if the, if you want the agent to act on our behalf, how do we make that auth token sort of pass through towards the end system and get that?
get the action, approved from the end system, like who is authorized to make these agents take actions on their behalf. Some of these things like, you know, we are still in the early stages, but in order to like truly make agents operational in production, Some of the questions from the audience was also, uh, similar, like, how do we have agents interact with ERP systems like effectively, right?
If I can quickly say something there, Yash is like, I recently had a conversation with somebody was super interesting. And this person was, he was proposing. To like have an add on to the, to the restful architecture. Right. So that he said, like, if we see the return headers that are coming in an API call, he was proposing ideas where he would say like, who's going to consume this [00:09:00] information and if then the return header would say, well, it's actually not a human or like system button.
A model that's gonna consume the information, then how you would format what was coming back would be different. Right. And so I love how people are thinking about that and how people are even, you know, going back in time to, in this case, restful architectures to say, like, can we do something there to rather saying, like, you know, it's going to be consumed by humans going to be consumed by.
An application or by model. So it's like a, um, there's a lot of exciting stuff happening there. And, uh, because I think that more and more of these interactions will be handled by, uh, by models. Absolutely. And, I think, one other kind of, aspect that came up also in, in our previous discussions is, with agentic, uh, workflows.
Multimodal Information and Agentic Workflows
The dependency on multimodal environments becomes even broader and even bigger. Like, I think question for you, Brian on like, how do you see kind of just. the challenges with [00:10:00] digesting multimodal information being kind of, a blocker in some ways for some agentic workflows today.
And, and, and what are we doing to, to get past that? I kind of break it down into three different buckets and you could break this down a number of different ways. first is data availability. So for example, like Bob and I were at a conference last, week where a CIO said, actually, my number one problem right now for adopting generative AI is RBAC.
It's role based access controls and just managing that across, theirenormous organization. What a boring topic and a boring issue,to be blocking generative AI adoption, but like, it's real. Why didn't you say that Brian? Why didn't you say that But like, it's true, right? Like this stuff, so data availability. Data quality, structuring it. That's what Bob was just talking about. And,that's where we see, embedding natural language or, image data, whatever it might be, generating triples, for example, indexing in the graph, try and make rag work better, whatever it might be.
[00:11:00] changing how APIs deliver payloads. and then. how you're actually serving that to the model. And this, right now, like for example, when we're using multimodal models. We're working with best in the business. We're still looking at like 20 or 30 seconds for inference on just like basic stuff on just piling PDF pages and boring stuff like that that we do.
Right? If you want this to be sub millisecond. For, like a almost a human feel to this as, sensordata's coming in right? Then you need to be really thoughtful about like what sort of cocktail of models that you're using to power those models. Oh. Um,because this is something that, chip and the other panels was talking about in the last, in the LA in the last panel, which is that like, look, Sub 7 billion sub 2 billion sub billion text to text models might work absolutely fabulously for a lot of tasks. Right? And so being really thoughtful about which models you're giving, which workloads in order to deliver that [00:12:00] business value. But you got to solve across that whole value chain. If you're going to do that.
I think stepping back, what I'm seeing, at least from like the, over the summer and into the fall are lots of startups emerging to go grab data in the wild. So like kind of solve old problems, go, go rip, you know, go crawl, scrape websites, rip HTML, deliver it to the models. And then the stuff that companies like, you know, I'm in part of the value chain that are structured, which is like get private data.
And deliver that and, you know, as as cleanly structured of a manner, as you possibly can. To help the models be successful, because the end of the day, the math is never going to change. You want to deliver to the context window. Exactly what the model needs, nothing more as efficiently as possible if you want to, minimize cost time and also, maximize performance.
And accessing data is, absolutely important. And like, yeah, just a question keeps coming up is like, how do we even [00:13:00] let the right person access the right data, and manage access that way. But from the accuracy standpoint, right?
Like Brian, that, uh, that you mentioned, maybe, uh, Bob, I'd love to, and I know you have your hand raised as well. So, you know, feel free to chime in here, but Bob, how do we ensure like agentic workflows are going to need more, specificity on the data that was provided, as part of the rag outputs and, you know, how, how are folks, how are developers solving that today?
Future of Agentic Workflows
before answering your question, I really would like to like to comment something on the, on the previous point, because the, I think we have a lot of people now listening and building, you know, coming up with ideas and building these kinds of applications. And I think, so things like RBAC, et cetera, are more like, you know, things that people need to go into, into production.
Right. I really would like to urge people to just don't think back in time. Don't think backwards. Right. so we see that a lot of. A lot of things that I see is like how people try to implement this is that they just basically slap on the old techniques to something new. But I really [00:14:00] believe that with AI, we are like in this new paradigm, how we do stuff, right?
So there are even new ways of doing, for example, RBAC now. What that means and how that works, I don't know, because if I would know I, you know, we probably would be doing something there. But the point is that this is the time, right? To come up with new ways of to think about, because, you know, our back is very binary, right?
So it's like access or not, right? But what if it's like certain type of information or nuances of information that you're getting from the model? And I think that, you know, startups that will emerge. That's all that. Right. So, but not in like the old way, but like in a new way, just leveraging the models to do, for example, RBAC or whatever, that's, I really would like to urge people to, you know, to keep thinking that step ahead.
Speaker: And then to answer your question, like how. how do people do it? Well, so one of the, challenges that we have right now with, like with chunking and those kinds of things is [00:15:00] that sometimes the true value comes from. More information where we don't have any access to right.
So let me give you an example. So let's say that you have like email and so we have like, we have like a customer with a big use case with an email and I happen to be using the application as well. So if I go like, okay. You know, uh, what time is my flight today? Right. Then, you know, it does like a hybrid search of the database.
it's properly chunked. It goes into the model and then it will return. I don't know, uh, 1 15 PM. But if I then ask the question, the follow up question, so. You know, what terminal is it from, then, you know, the answer I get, like, I, you know, I didn't know that information is not available. And the, one of the things that we've seen right now is that the chunking, and yes, sometimes it's hard and sometimes depends on the information, but kind of the techniques are now kind of here, right?
How to, how to deal with [00:16:00] is that. the bringing together also had like what the stuff with the crew, et cetera, is doing is like bringing together different ways and different sources of information to really, truly help the end user. that is stuff where I really would love to see more being built.
so rack one dimensional rack. So I often just draw it like as a line, right. From the, from the query to the, to the answer has brought us very far. But now we need to start to bring in loops, right? So what if you have more questions? What if we need to query multiple times?
And that is something we at we've had, they put a lot of work in that. So, so we'll probably get to that later. But the, the feedback loops to, you know, to get might be multiple sources of information, do multiple queries from the database. That is going to be tremendously important because that's where the true value is.
And let's not forget if we talk about. uh, what you just mentioned, right? With this one dimensional structure is like we only talk about reading, right? Reading information. So I have a query, you get a [00:17:00] result and it's presented to you. But what about writing? What if the agent you're talking to or interacting with is, you know, what time is my appointment?
And it says, Oh, you don't have an appointment today. And what if I don't say, okay, can you can you book an appointment for me? Right? So, That kind of stuff, a lot of work needs to be done there and it's happening, of course, it's happening, of course, but I would love to see everybody that's also listening to us right now, just, uh, when we're done with our panel to get their hands to the keyboards and, uh, and help us build more of these kind of, uh, tools.
I
Speaker 2: mean, after the
Speaker: panel, after the panel, not now,
Speaker 2: don't stop yet.
Speaker 3: yeah, I was going to share just for sake of, just candor, uh, failure and a success that we've had, or let's say, uh, a hopeful, a probable success, on the failure, or at least, we haven't cracked the code yet. We've been trying to use agents to help automate the setup of connectors. So I'm going to pull from Salesforce or S3.
Et cetera. Right. and it's a pain in the ass. Like we have, [00:18:00] we've sat live with like lots and lots and lots of users and like the fastest folks have been able to set up a connector is like 15 to 20 minutes a lot because it requires like emailing and admin to get, you know, certain permissions and so on and so forth.
It's a giant pain in the ass. Right. So we're like, Oh, well, let's use some, let's use, you know, agents for this. And, it's proven incredibly difficult to have that work. So that's like the one we haven't cracked quite yet. in order to like productionize it and put it, put it out in the wild for our users to, to leverage on the success side.
We're hoping to move this into prod some more time around, like after Thanksgiving around, like reinvent 1st of December, but this isn't like prop, it depends on how strict you are with the definition of agents, but with the definition I gave you earlier, we're looking at like dynamic pipelines where models are sampling data, like say you have a fresh, you're doing a fresh poll from S3, you sample it, you spin up, you run a bunch of different chunking techniques and parallel.
Hello. [00:19:00] You have models judge, which, which kind of chunking strategy it prefers for that particular type of data and then decides to implement that for that particular job to do the same thing for embedding model selection is like you could if you wanted to for better model selection, which. Might be problematic, but like for all these cocktails and models that we that are our users rely upon to go from raw to like rag ready data so I can do a baton pass to Bob, right?
you may have 6 or 8 different models along that, and it might vary wildly depending on the type of data that you're pulling from, right? On like, which cocktail models and which settings make the most sense. And so we're trying to automate that or at least make those dynamic by inserting. One or more models to help perceive, you know, decide and enact in terms of like adapting pipeline configurations autonomously.
Speaker 2: Amazing.
Speaker 4: Yeah, no, I want to, I want to teach you one thing that Brian said that I think is super relevant and that is we are seeing a lot of people kind of like, going after like use case and an agenda again [00:20:00] and getting burned, right? And sometimes it's not even about the two that they're choosing is about like, it's again, I think it go back to what we're talking starts very simple, but once that you get like, you're like, all right, now I want to get a reliable consistent results are like, oh.
All right, so there's a lot more that I need to put into this in order to get what I actually need from this. So there's definitely something that we are seeing where like, uh, some of our customers are coming to us saying like, Hey, we do this. We have this thing. It kind of works. It's very promising, but it's not up to our standards.
Like we need a few extra things. and other customers come up with like this crazy use cases, right? They're looking for a hundred percent accuracy. And honestly, right now it's not the time for that. you don't get a hundred percent accuracy with AI agents. That's not how, how things goes.
Uh, you can definitely get humans in the loop and that brings you very close to it. but it's something that need to be mindful of. So whenever someone shows to me, it's like, Hey, I wanna do accounting. I was like, well, you can definitely try everybody. You're gonna have some humans in the loop there.
They don't, they don't pretend that [00:21:00] they're gonna kinda like automate the whole thing.
Speaker 2: Absolutely.
Best Practices for Agentic Strategies
Speaker 2: And Joe, to that effect, right, like, moving on from the data to the application side, like what are some of the learnings or best practices for thinking about, uh, you know, framing agentic flows, uh, so that, you know, you're not going down a rabbit hole, uh, the best practices that we can share with all our listeners today on getting the agentic, strategy right from the beginning.
Speaker 4: I'd love to talk about that. So from everything that we're seeing out there, we are seeing that it's definitely, it's, it's definitely cross vertical. So like there's a lot of internal automation and operations that are like going on and then you have like sales, marketing and research and coding.
So there's kind of like a, there, there's a healthy distribution, but what we're finding is that there is a, there is basically a normal distribution on. the pattern of the use cases. So a lot of them are usually a combination of [00:22:00] research, analysis, summarization, reporting. Sometimes it's not all of those together.
Sometimes it's kind of like mix and match, but those kind of like those work like a charm. So if it's like, hey, I want to pull data from somewhere, from an ERP, from the CRM, from somewhere else, and when I conduct some sort of analysis, I want to have a summarization of sorts. And I'm going to put off a report, either JSON or whatever it might be, so that I can push into another system.
those ones kind of like work pretty well. then the other thing that people are doing that are kind of working great is This idea of like bringing human the loop when necessary. Like, Hey, this is kind of like a use case. We actually want to have human the loop. Like we're already saving a bunch of time on this, but I want to make sure that we get someone in there.
I would say that's the other, the other thing in the final thing that I want to mention is the way that you roll out those things in companies. Uh, so we had, uh, we have a great interview that went out last week with, the commercial CTO [00:23:00] of GemAI from PwC. And that's something that they have to talk like from the ground up because it's a big company and we are rolling out things like that.
People start wondering, like, all right, what role does AI agents play in here? And how do I interact with them? So there's definitely a human component on, like, the companies that are promoting people to feel like you manage these agents now, and they're tooling your tool set. I can like, having more success with that than others.
Speaker 2:
Final Thoughts and Conclusion
Speaker 2: I know we're coming up on time, but the 1 thought that I'd love to, end with is for each 1 of us. highlight what needs to be addressed to make agentic workflow successful, like what each one of us think is kind of one thing that, we would love to have solved in the coming months, for agentic workflows to see, you know, the light of day, literally, uh, across, All kinds of use cases.
Speaker 4: So, um, is from everything that I have seen, the one thing that I think is a must [00:24:00] for this future of GenTech to live to its full potential is.
It needs to be fast and simple for companies to build them. Uh, there's, there's many other things that I could mention here, like enterprise readiness, security, everything. And like, and all the other things are true, but I think being fast and simple is the most important thing because in the market that's moving so fast, if you say that companies need to spend like, Five, 10 engineers for three months with a product manager and a designer to get like something like a POC done that doesn't work, but if you make it that simple and fast to try things and throw things in the wall and see what sticks, that is cruise up the economic in a good way where now don't think it's going to move as fast as the market and then double down on my work.
So. I would say that would probably be the most, uh, the most important thing from my point of [00:25:00] view for, the future of agents can, like, lead to its full potential.
Speaker 2: All right, Bob, you're next, but it gets harder for you and me, Brian.
Speaker: I'm gonna, I'm gonna be a little bit self serving answering this question, but the, And, and we've, we're super bullish on this concept of generative feedback loops.
and very soon you can already read about it and I'll, I'll share a link in a bit, but the, uh, very soon you can play around with it. And it basically means that you can, rather than prompting the model, you prompt the database. what you're basically going to say is that, for example, when you create a collection in the database, let me give you the simplest example that you can think of, they say, okay, let's say that you have products and e commerce, like every product should be in English.
Right. American English. Then if you store something in Spanish, then, it just updates that for you. It says like, well, you know, and then you can choose as you want to keep the original content to do an app search or those kinds of things. But now also think about, for example, data management, right?
So we have like a, we now work with a data sets, like a factory data set. And it says like, you know, uh, [00:26:00] all temperatures from the oven should be in, in, in Celsius. And then if somebody starts in the Fahrenheit. It just does an absurd and it's just like, well, you know, it's now it's now in Celsius as well.
And so you prompt database rather than the, model, and then you get full cross support. So the, uh, the database can read more from the database. It can update in the database. If you can even delete, if you wanted to, And it can create. So. that is something we're super, super, super bullish on.
And that's just, that's in the domain of the database. So basically if you would be interacting to the stuff, you know, that, the folks at the crew or the instructions are building, then basically then the database gets smarter, right? And we, we call that agentic. Workflows, right? So, so the generative feedback loop is a form of an agentic architecture.
And, that is something I'm extremely bullish about. So, so very soon more, uh, for, and also for people to play around with.
Speaker 2: That'd be awesome. Can't, can't wait to try it.
Brian.
Speaker 3: I think just the pace with which models are [00:27:00] improving is still really exciting and rumors are that GPT five is going to drop in December. Right. And, and we're going to have even more performant and faster multimodal models.
I think that's the long pole in the tent here for a lot of this. I think this stuff that job is working on is. Absolutely essential to being able to operationalize this in the hands. But like, the end of the day, like, it comes down to performance, which is what we were talking about earlier, and a lot of the performance rides on the power and quality of these models.
Like, as we figure out ways to shrink them back down again, that's great, right? It means that they could proliferate and we could push them to edge devices. But, at the end of the day, it's like my mom thinks it's a piece of crap and she doesn't use it like she doesn't use Siri. we're not there yet.
Right? And so I think that we're driving in that direction. I think we're like, it's a shallow trough of just it was a shallow trough of disillusionment on on agents for like months in 2023 and like they're kicking ass. But I think that we're driving hard towards that reality and I'm, I'm [00:28:00] bullish that in 2025 we're going to start getting there with a higher win rate.
Speaker 2: Amazing. And I'll, just, add on to that because like, you know, it's great to be able to build agents quickly, in a more robust, data driven manner. but in the end, from my perspective, I think we all want to build applications that can make it into production very quickly.
and, in the spirit of this conference today, I think, One of the things that, I would love to really solve for personally, as well as, my, the wonderful team and Galileo is, how can we enable not only building agents fast through a better data driven perspective, improving access to more data, but also having the right set of metrics that people can rely on.
Right from the beginning so that you're not building a POC in three months and then you're, spending a year trying to productionize that trying to make it better and useful as you said, Brian, it should be robust and useful right from the get go. And if you don't have the right signals for the entire [00:29:00] agentic chain, it's because it's even more complex than a simple rag setup.
it's going to be very hard for us to. Go from the to see the production stage, without the set of observability and metrics. So, you know, the team and Galileo is also innovating and training models and systems on algorithms on what do we call agentic evaluations? And we, we had a, uh, I had a talk earlier with chip and, and, uh, and we've been about this, but we've been thinking hard about this.
we have some ideas to lots more to come in this space, but I think. Without the right set of, trustworthy metrics that teams can rely on, along with the data infrastructure and an easy way to orchestrate them. these are all the key building blocks here, to build agents.
Speaker: Yeah. And we now know that the best evil for an agentic framework is Brian's mom. So that's a,
Speaker 2: Love it. Love it. Amazing. Uh, Well, thank you all. This was an amazing chat. We all [00:30:00] had fun and I'm sure our listeners and audience as well. and you guys have been super engaging on the chat as well, which is amazing. thank you all for, for being here. This is awesome.
Speaker: Cheers.
Thanks everyone for tuning into episode two of chain of thought. If you want to check out more sessions from productionize, you can find [e-mail]. We'll be back next week with predictions about gen AI in 2025 and much more.
Conor Bronsdon: If you are enjoying chain of thought, leave us a quick rating and review on your podcasting app of choice or share the show with a friend that you think might enjoy it. Thanks so much and we'll see you next week.