[00:00:00] Anna: in an industry that is moving at breakneck pace. [00:00:02] To me, vendor lock-in is a kiss of death when it comes to competitive agility. a one year agreement might as well be a decade long, vendor contracts, creates this dangerous complacency, there's this invisible cost this tendency to, okay, well I have this tool that does what it needs, so set it and forget it. Right? there are certain things that you can consider buying and certain things that you can consider building. If it is something that, is a core competency of your business, you cannot afford to be complacent. [00:00:32] Don't be inhibited by what the AI tool can offer you, your imagination. Is the limit, right? we're shifting from being execution focused to now being very design focused. You're not driven by tasks, but you're driven by like, how do I architect solutions now? [00:01:15] In This Episode --- [00:01:15] Phil: What's up everyone? Today we have the pleasure of sitting down with Anna Abalon, VP of Operations at Civic Technologies. And as up episode we cover how AI flipped the build versus buy decision and why in-house AI provides better economics and control. [00:01:30] We'll also cover how Anna moved BI workloads out of dashboards and into LLMs using role-based AI guardrails across MCP servers, all that and a bunch more stuff after a quick word from two of our awesome partners. [00:01:43] ​ [00:03:47] Phil: Anna, thank you so much for your time today. Really excited to chat. [00:03:50] Anna: Hi Phil. I am super excited to spend the next hour or plus with you a fellow Canadian here. Don't get that often, so. [00:03:58] Phil: Yeah, I had just a [00:04:00] few Canadians actually on the show. So yeah, it is super cool to have another fellow Canadian. Um, a lot of folks come from like the Bay Area when we're chatting with like folks in tech and in and in marketing. So yeah. Nice little switch. Uh, we had a fun episode for folks. Um, [00:04:15] 1. How AI Flipped the Build Versus Buy Decision --- [00:04:15] Phil: I wanted to maybe start with something that I love about your perspective specifically. [00:04:19] AI has kind of quietly exposed this illusion of complexity that's kept a lot of companies hooked on overpriced MarTech, uh, for a bunch of years. So I, I've had plenty of conversations on the show, but this like build versus buy [00:04:33] Anna: Mm-hmm. [00:04:33] Phil: um, it's pretty ops heuristic to have like a person say, you know, you should build, if it's core and manageable to the company. [00:04:42] But you should buy if it's complex and not differentiating for you. You don't disagree with that rule itself. But you have an interesting thought on like how you apply that in practice. And most companies assume that like AI systems are way too complex to build internally yourself, especially marketers that are [00:05:00] less technical. [00:05:01] Um, so they default is just like buying and shopping for ai. MarTech, you are arguing the opposite, that with today's tools building is way easier, faster, and cheaper than a lot of vendor contracts. Uh, I love your take here and it's almost like you're suggesting we rethink where teams should be drawing that line on like, what is manageable, what is too complex? [00:05:20] So chat, chat to us about that like a bit more. Wh why do you think modern AI has lowered the complexity threshold so far? That teams should be building a lot more. [00:05:29] Anna: Yeah, I love that question, Phil. Um, and I think it's you. The, when you an, when you ask the question, you know, if you ask that question 12 months ago versus asking that question now you're gonna get very different answers. Right? Um, the truth is that the AI tools have, that we have today have greatly democratized, um, building and evolving faster than our decision making framework. [00:05:52] Right. Um, the tooling landscape has changed so significantly in the last couple of months. I mean, we were doing bill versus buy [00:06:00] analysis like six months ago, but. So much has changed between then and now. Um, so examples are, you know, today you have, um, uh, like MCP for system integrations, right? People are talking about like N eight N and Lindy and the likes for workflow automation. [00:06:16] Then just like couple weeks ago you had chat GPT launch agents cloud launching skills, and then all of these have pre-built capabilities, right? Um, and then plus you have like LLM galore, right? And LLM direct access. You no longer need to be building your own custom rag pipeline or complex vector databases, uh, database infrastructure. [00:06:38] So much of like the heavy of technical overhead has now been greatly simplified by these, um, AI tools. And so I'd say that the democratization of that, it has lended. Has created a ton of opportunities, particularly for various functions within ops, to really look and break down their [00:07:00] various workflow and not to look at just externally for workflows, uh, sorry for solutions, but to take a look and see what is feasible given like this whole landscape of AI tools. [00:07:13] Phil: Yeah. [00:07:13] 1.2 Redrawing What “Complex” Means --- [00:07:13] Phil: Where do you think this like new line is then? Like if you're internal and you're thinking about like where that line is for complexity, what do we outsource or look to [00:07:22] Anna: Mm-hmm. [00:07:23] Phil: How is the team made up? Like who is a specialist in certain area and would we build in house? Like when does it still make sense today, in your opinion, to buy versus build with ai? [00:07:34] Anna: Yeah, I mean, going back to, you know, the, um, the core competency, right, that we talked about, complexity is less about, um, just the technical complexity. Um, it's about, you know. Combined with like core competency is what you are looking to solve, like core to your business, right? Again, these AI tools have brought down complexity, [00:08:00] but at the same time you wanna focus on is it core? [00:08:03] Could I turn around and see other people in my, you know, particular function, use this tool and, you know, could this have, could this provide value proposition to customers of ours? Right? Like, are those core, is it a core competency of yours? Um. I'd say that, you know, that there's a couple of, like differentiators, like for us, we looked at, um, analytics, right? [00:08:27] We use custom analytics, um, to answer a lot of like different various business questions. So for us, that was core, um, complexity. It was fairly low given all of the different MCP integrations that we were looking at today. So in that case, you know, for us it was smart. Let's go ahead, let's build it. Um, for something like, um, an AI support bot, right? [00:08:49] Um, it is not fundamentally core for us. Um, and when you look at the complexity, and when I say complexity, there's a myriad of different things like the technical part, but [00:09:00] also the function that it serves. Like we. You know, one day you're serving, um, consumer end user another day you're serving developers. [00:09:08] Like trying to build support bot that can be effective, um, through various different mediums and engage effectively with very different audiences is extremely complex. Um, so for us that was high complexity and in that case, it just was not worth for us to build that. So I would say focus again. You know, look the take, rethink the complexity part, right? [00:09:32] Again, look at the new landscape of tools and then really surmise like, is what you are thinking about core to your business? Are you, um, simply buying and feeding a competitive solution? Or are you developing yourself and creating a competitive moat around you? [00:09:50] Phil: Yeah, that's such a good point. Like the core competency aspect, uh, often feel. Folks will think of it as. Is it a core competency that the company could then [00:10:00] resell to our core audience? Um, I think that an interesting layer to that, and you kind of teased it out, is also this idea that, is it a core competency that maybe I don't resell to customers or end users, but I can resell to other people in my company, in ops or in other departments. [00:10:18] And like the example I go to is like a, at a short stint@wordpress.com, like big, um, enterprise team. We had like a data team that was central servicing a bunch of different product teams and they built out an internal experimentation tool. And it wasn't something that, you know, a lot of WordPress end users could have found a lot of value in. [00:10:39] But we built it internally because all these different like teams within the company could then use that internal tool versus having like customer support, have their own AB testing tool, product has their own marketing, has their own. So we are all under one umbrella. Do you kind of extend that like core competency outside of just like end users and also 'cause like when you're an ops, [00:11:00] like your end users are customers, but they're also people like operators within the company. [00:11:04] Right. [00:11:05] Anna: yes. And I think that, you know, you kind of hit exactly the, one of the biggest problem in ai, right? How organizations fail to adopt AI is because, you know, everyone is sitting in silo. Everyone uses a specific AI solution trying to drive like certain automation within their function. But to really achieve success as an organization with AI is to kind of not find like universal, but to really get adoption right across the organization. [00:11:36] Um, so I think if you can find like workflows that, or the way in which you build workflows with these AI tools as something that, you know, other teams can also leverage. You know, whether like various different mcps that they're plugging in, like different workflow building tools. To allow others to or share in your success and allow them to see what is possible [00:12:00] in this like breakneck speed that we're moving in. [00:12:03] That is critical. [00:12:05] Phil: Yeah, yeah. So true. So fast. We're, we're recording this and like last night I was just playing around with the OpenAI Atlas, like the, the browser plugin for GPT. And by the time this comes out, there's probably gonna be like two new advancements already. [00:12:20] 2. Why In House AI Provides Better Economics And Control --- [00:12:20] Phil: Uh, I wanna ask you about off the shelf AI models, like, you kind of talked about that a little bit, but like the common wisdom when you're inside an ops team right now is to say like, let's not reinvent the AI wheel. [00:12:31] Let's just buy an AI solution that's already been trained on. Way more data than we can ever get our hands on as a company. Why would we try to rebuild something with our tiny little data set, especially if we're a startup? Um, your take is that most of these tools are overpriced when it comes to AI tooling. [00:12:49] And, you know, maybe you'll agree that like a lot of them are just wrappers and, and not necessarily like their own proprietary data sets. Um, and a lot of them are under contextualized, like they're not trained [00:13:00] on your data. And so if, even if there is a lot of data, doesn't mean that it's necessarily representative of your customer base. [00:13:06] So they solve the wrong problems for too many companies. Why do you think that AI vendors and the whole like market right now has kind of become bloated and in-house AI gives better economics and control. [00:13:19] Anna: Yeah, I mean, so many people look at AI as being the, the silver bullet to all of their organization's problems, right? So I swear, I, you drive like around the Bay Area and there's just billboards for different AI solutions. You could not drive down the 1 0 1 without getting hit by a couple. Um, so my overall take, um, and I'm, you know, call me a skeptic, um, what you want, like my overall take is that again, we are in an industry that is moving at breakneck pace. [00:13:53] To me, vendor lock-in is a kiss of death when it comes to competitive agility. We're moving so [00:14:00] incredibly fast that vendor contracts, and most of those that I've seen are annual contracts. I mean, in where we're at right now, a one year agreement might as well be a decade long, like in typical business terms, right? [00:14:13] Um, vendor contracts, in my opinions, creates this dangerous complacency, um, in the environment that we're in. Um, once you sign an annual contract, there's this tendency to, okay, well I have this tool that does what it needs, so set it and forget it. Right? Um, but the AI landscape evolves so incredibly fast. [00:14:32] Like you're gonna miss out, um, like you said, right? Buy like in a month when this comes out, like Atlas will be a thing of the past, or, you know, there'll be two new models. Like it just moves so fast. Like every day I come back, you know, something, what we, what I did last week is now, you know, old news and it just moves so fast. [00:14:49] So you, you can't, you can't afford to be complacent. Um. And so I think there's this invisible cost to finding the vendor, right? And we talked about [00:15:00] earlier, um, there are certain things that you can consider buying and certain things that you can consider building. If it is something that, um, that is core to your set of like, it's a core competency of your business, you cannot afford to be complacent. [00:15:16] It is the kiss of death. Um, so I'd say, you know, look at tools, do your build versus buy carefully, right? Assess your core competency and then, um, you know, like see if it's something that you could potentially build. [00:15:31] Phil: I really like your perspective, Anna. Like [00:15:33] 3. How to Treat AI as an Insourcing Engine --- [00:15:33] Phil: most stories that we have on the podcast about AI are like automation and replacing a bunch of tasks, and in a lot of cases, like replacing jobs entirely. Your take is way more refreshing. It's like you're more focused or excited about the aspects of AI that are. [00:15:49] Are reclaiming ownership, like bringing back ops capabilities back inside the business, um, that a lot of times were outsourced in the last couple years, uh, or like [00:16:00] decade even. Why are you more excited about AI making insourcing profitable again, like versus simply just thinking of outsourcing stuff to ai, it's maybe not a core competence for a lot of our users. [00:16:12] Um, and maybe chat about like what convinced you that AI had reached the point today where insourcing could outperform outsourcing both financially and strategically. [00:16:23] Anna: Yeah, I mean, um. Earlier we talked about like vendor, um, vendor contracts and like looking at different vendors. Like one of the things that you know, while your vendors may not learn specifically from the data that you feed into it, what they're doing, and this is again leaning to focus on your comp, uh, your core competency, they are optimizing right with every single interaction. [00:16:49] They are optimizing. They're dev, they're learning from every single edge case that their users are injecting. They are continuously iterating on their product. With your insight, you are locked in, [00:17:00] right? And you, while they are continuously optimizing, um, both performance and their product offering. Um, at the end of the agreement, what you walk away with is the content that you inject, that you injected. [00:17:12] You are not walking away with like their entire rag system. You are walking away with what you fed into the system. So, um, it's this massive missed opportunity. And again, just going back to cost like this, um, this, uh, by outsourcing you, you miss out on this like invisible cost, um, that I think is so crucial. [00:17:33] Um, I will say like, so for insourcing, um, insourcing versus outsourcing. Um, what's interesting, while we did outsource, um, you know, the AI support bot function, there was this element of insourcing too. And what do I mean by that? Um, I don't think of AI as replacing jobs. I think of this, I think of AI as how it frees your team and in particular, operations so big, right? [00:17:59] Um, it [00:18:00] frees your team to do the work that really truly moves the needle for your organization. When you think about operations, there's, you know, what typically comes to mind is, oh, there's a lot of like repetitive tasks, like things, you know, they're just kind of like a cog in the machine, right? Like making sure things move as they should, like that the organization's running smoothly. [00:18:19] Well, people often miss out on, because we're so bogged down with like the operational task is the strategic element of operations. Like, who has the time when you're so focused on making sure the machine is running? Um, AI has afforded us the opportunity to free up various different aspects of operations to really like, think more strategically, um. [00:18:43] So, uh, you know, whether, so going from, um, in the case with like outsourcing the AI support, um, AI support bot as an example, rather than building our own, um, we previously had, um, BPO contracts, um, to provide level one support, [00:19:00] right? By outsourcing to, um, use an AI support bot, we're able to elevate our internal support team to really become experts at digesting, you know, the interactions, content management, context management, um, and how users can, um, how users are interacting with our AI bots and how to make those bots like work smarter more effectively. [00:19:24] So in a way, we are insourcing for our team whilst outsourcing in that particular scenario. Um. The other thing, um, that we did, you know, the, what we insource was we previously had, um, you know, very expensive, like bi solutions, right? I know you have some experience, um, working for a BI company, um, yourself. [00:19:45] Um, like they're expensive and they're expensive for a reason, right? Um, they, you can create a multitude of dashboards that your entire organization relies on. Um, but you know, on the flip side, we also to support that you need, [00:20:00] like data engineers, you need like, um, data scientists and um, engineers, right? [00:20:05] Um, so by insourcing that part, that bi visualization tool function, not only did we see savings, but we were able to, um, really elevate our team to be able to interact with our data in a much more meaningful way. So we've enhanced, like, we've enhanced the way that we articulate. Analytics and um, really also open up access to the analytics for the rest of the organization. [00:20:35] So I'd say that I don't think insourcing is eliminating jobs. I really think it's about reclaiming time for high value work within your organization and specifically for operations. [00:20:47] Phil: I appreciate the, the practical examples there. So you, you kinda shut it out, like AI bots for support use cases, and then you talked about this BI tool, uh, outsourcing versus insourcing. [00:21:00] Um, maybe we can touch on both of those. Uh, [00:21:02] 4. Moving BI Workloads Out of Dashboards and Into LLMs --- [00:21:02] Phil: let's start with bi, like replacing expensive BI tools is a massive operational swing, especially in companies that all across departments are using the same central BI tool. [00:21:15] Talk about that journey. Like what did that look like? The tools, the sequencing, uh, the early friction, I'm sure that you must have had, especially with like the data, uh, counterparts on, on the data team, how to performance, speed and cost kinda shift once you own these systems fully in-house versus the, the outsourcing that you were doing previously. [00:21:33] Anna: Yeah. Um, I, I mean I think for us, like as an organization and we're relatively small, right? Um, the breaking point for us is I think the speed to getting the data. I don't think people were territorial about, you know, their control over the analytics for the organization. Um, there are lots of requests and demands for data. [00:21:58] And again, when you're a small [00:22:00] organization, maybe you have like one engineer or like one data scientist or data engineer, um, and you're working with an imperfect tool. Um. I say imperfect in the sense that with what you're, what is available to you, legacy BI tools are so imperfect. Um, so the breaking point for us was realizing that our, our ex, our legacy BI tools were really built for yesterday's question and not for tomorrow's decisions, right? [00:22:29] Um, these tools are meant to look backwards. Like, let's look at what were Q3, Q4 sales, right? They're not there to synthesize like, Hey, from your data, we think that these are the trends that you should be focusing on. So in a way, like they're so imperfect and then they create this like, bottleneck of demand because they're not the easiest and most like user friendly, right? [00:22:53] Um, now we have the ability, you know, through, um, MCP orchestration, we can bring not [00:23:00] just the, the data warehouse that we used to pipe to, um, our BI tools, but we can bring it into an LM client. Whether you use Nexus or you use Claude or whichever. Um, and you can, you, you can use natural language to interface with your data. [00:23:19] Um, it's, you know, beyond just let tell me what Q4 sales is, it's now, all right, let's bring in Q4 sales data and then let's bring in like usage patterns and then let's bring in also, um, website traffic activity and any CRM like, you know, um, activity, like logged activity that we have for, with this particular customer. [00:23:38] You can bring all of that together and achieve this level of synthesis that I don't think was their previously. And you know, to to another point, like there's so many tools out there. Every single, single one of these disparate services that I talked about have some level of AI automation, but they all act independently. now [00:24:00] it's not just. Do you have an AI BI tool? It's that I can bring that BI tool and I can merge it together, not merge it, orchestrate it together with all of my other service to not just get information, but now I can execute a workflow that produces an end product. Um, and so to me it really shifted from understanding the limitations of Legacy BI tools to what we can do with, um, the, the AI tools that are at our disposal today has been absolutely game changing. [00:24:34] Phil: It's super cool. Fascinating story. Um, yeah, like you said, I, I have like a, a, a short stint early in my career at a, at a bi startup. So it is really cool to hear like the journey of, of bringing that in house and the whole discussion about centralizing all the different sources of data. And you kind of talked about like bringing that in, piping it in from, from the warehouse. [00:24:56] Um, I wanna talk about like owning the data layer. A little bit [00:25:00] more. So let's, let's unpack that BI example a bit. Um, so look, I wanna make sure I have this right. So your, your team like replaced enterprise BI with a self-serve analytics stack, and you've got, um, LLMs on top of that, some automation. And like obviously there's tons of cost savings, like once you get to the stage that you're at today, um, maybe chat a bit about more, like how that changed and how your teams are accessing using data today. [00:25:27] Like did it make things faster? Is it better because there's more context that you can give folks or like folks are just freed up to do more creative things? Like, just curious if you can chat about that a bit more. [00:25:39] Anna: absolutely. So, you know, we talked about earlier. Ai, really? AI does like what, 80%, right? We all love to say it does 80% of the work, but then now that 20%, you still have that 20%, um, of what you do. You have this time that's freed up to really [00:26:00] elevate and be more strategic. Right? So historically, you would maybe reach out to your data engineer. [00:26:05] I have this particular request that is beyond what is in our existing dashboards. The typical turnaround time, if you're lucky, you know, if you're in their good graces, maybe he'll turn it around overnight for you. Now, I don't need that anymore. I can, I can have a full blown conversation with, you know, Claude or with Nexus Civic Nexus and dive deeper layer on layer about the data. [00:26:28] And who is the best person to engage with that data. Not, you know, not hating on any data engineers at all, but I know my business problems. I know in real time what I want to ask, that you know, what I want out of my data, the questions, the burning questions that I have, right? And I want. You know, the, the, the LM to also engage with me, like, Hey, we noticed this, like, have you thought about this? [00:26:52] Or maybe it's this. Right? Um, so how has that, you know, impractical terms, um, how has that translated for [00:27:00] our team? So let's start with like go to market, right? Go to market, use a myriad of different, different services already. But I'd say data enrichment is probably one of the biggest ones. Um, you have your CRMs, you have like, you know, all of your other, um, tools where you get customer informations, but typically those are not infused with usage information, right? [00:27:23] And then, um, conversion information. But now we can bring all of those things together. I can bring all of our event like usage, event data, you know, website event data and CRM data together. Into one LLM conversation, um, that has transformed the way, you know, yes, they're still building out campaigns, but they can build more nuanced campaigns and at much faster speed. [00:27:48] They can tackle a problem at 11:00 PM independently if they want to. They don't need to wait for anybody else. Um, so that's huge. Um, and our product teams, man, like when you think [00:28:00] about product usage, right? You, you look at like, oh, you know, do we do these like various, like very systematic KPI tracking, but now I can say, Hey, like, I wanna understand what makes a regular user versus a power user. [00:28:14] Like, get down to the nuance, like what, how many clicks per second? Like what type of, you know, tool calls or do they make within an MCP, et cetera. Then help me understand how I can convert these users into, like, the power users really, like, just peel back the layers. Um, real time like having, like talking to someone who just knows every single thing about, you know, we're like talking to Postgres, like right there. [00:28:39] You know, just tell me everything. Um, so those have evolved into how do we make UX better? How do we, you know, help make, um, help improve that conversion funnel so that when someone comes to our product, they know how to use it and how do we make a power user out of them? Um. That translates to, you know, like, um, [00:29:00] uh, subscriptions, revenues, and so forth, which leads me to our finance team, right? [00:29:04] Um, you know, in the world of AI where we're making all of these, you know, API calls, like those things can kind of run awry pretty quickly. So, um, our finance team certainly appreciates having real time access to, um, you know, how expensive are things Getting, like, where are our costs at? You know, they're not beholden to like a daily report, a monthly report. [00:29:26] They can ask the lm like, tell me where, where we are right now. Tell me, you know, are we about to hit our limit? What is like, what is our rate? You know, like our, our rate, um, at like, you know, for tool calls. Um, so there's a lot of like immediate impact that we're seeing. Um, but again, I think it speaks to like how people are using that and the amount of time that they've managed to save, um, and what they're doing with that. [00:29:52] That really speaks volumes. [00:29:53] ​ [00:31:31] Phil: I'll play the, the counterpoint there, like devil's advocate on like, something you said was like. [00:31:37] 5. Guardrails That Keep AI Querying Accurate --- [00:31:37] Phil: It's, it's so much better to be able to give the person who has the business context access to data because, like, no shade at the data engineer, but like, they don't understand the marketer's world. [00:31:48] They don't understand the product marketer's world, the product person's world. Right. Um, the counter argument to that though is. The product person doesn't understand the semantic layer and the data layer [00:32:00] and all the different fields and the tables and how this is calculated and how everything moves into the warehouse and that, how that goes into the, the BI tool. [00:32:08] So like those are the two sides of the coin there. And I'm curious to ask you like about guardrails and machine made insights, like when we talk about letting AI take over a lot of. BI stuff people get really nervous because, like, especially the data engineer folks, like self-serving marketers, like it's a dream. [00:32:27] It's amazing. But a lot of companies put it in practice and then it's like marketers are creating spreadsheets and like presentations and they're like, look, look at this insight and that insight and the data engineer is like, cringing. 'cause they're like, no, you used the wrong query there. Like, that's the wrong table. [00:32:42] Like that's the wrong field for active users. You should be using [00:32:45] this one. And our data dictionary is like outta sorts. So like what advice do you have there? Like what, what are your thoughts on like the, the counter argument of like making sure you've got good guardrails in place so that people are trusting in the, the, the AI [00:33:00] insights that come outta that tool. [00:33:01] And it's not like a limiting factor to have, you know, AI generate, uh, SQL queries. [00:33:07] Anna: Um, I, I love that question. Um, because you know, like I said earlier, people look at AI and it's a silver bullet. You can do anything with ai just 'cause you can ask. It does not mean you can get it right. Um, or what you get, you better validate. Um, so yes, there are some things that still hold true today, right? [00:33:27] So when you said data dictionary, yes you need schema documentation. Your AI can at read minds. Like if you don't have good schema documentation, it will infer, it will best infer what it can out of what you have set up. And, you know, the age old saying garbage in, garbage out too, right? Make sure your data is structured and that you are limiting the, um. [00:33:50] You, you wanna make sure that it's structured well so that it is able to easily synthesize as well. Um, so the, so providing the schema [00:34:00] documentation allows the LLM to really understand like the underlying data relationship. Um, you also met, you know, when we talk about like data engineers, like there, it has not taken a data, a data engineer's job, right? [00:34:12] They are helping at best, but they're not taking their job because it's the data engineer that really truly understands like how the various different tables in your database, like how they all interact with each other. So in prac, in, in our company, um, as a general practice, we still have the engineers develop. [00:34:33] Um, and when I say engineers, like it could be your engineers or your, um, business analyst, like develop those very critical business critical prompts, right? Those are the ones that, you know, your board looks at it and your, um, you know. They are, they are like the backbone of, um, how your company is performing. [00:34:52] So these business critical prompts are still designed at a very high, like high, not high, um, detailed SQL [00:35:00] level by those who understand the data intimately. Um, now prompts are also utilized by the rest of the organization, but by ha, by giving them access to the data, they can run these prompts and then they can start a interacting with, um, the data posts, like post prompt, right? [00:35:20] Start asking more questions. Um, but one of, we also, you know, have like best practices, like things like chain of thought prompting, right? If you start to, and you know, you don't even have to, um, think your AI might be hallucinating. I think it's just general good practice. Like, tell me how you got there. [00:35:36] Like, show me your reasoning, right? And like slowly you can unpack that. I mean, I'll give you a couple of examples of where, um, for us, like, you know, things that we learned from like. Just if you set up your data tables to have very different naming conventions, um, to what you, you know, the, like employees of the organization are used to, then your l LMS going to tell you what [00:36:00] the data table says and you're gonna be like, I have no idea what you're talking about. [00:36:03] Right. So that's where you get like, you know, um, uh, things like user users versus accounts or whatnot, like, you know, jobs versus tasks, like those can get super confusing and the inference gets a little bit dangerous when you're using natural language to query. Um, so make sure that you, um, you know, that you're still like all those business critical prompts, those like high, um, high impact ones are still being done by your data engineer team. [00:36:29] And then use things like, um, a chain of thought prompting and then human in the loop review to, you know, to, um, as you're using AI to massage like the, the analytics. Um, but yeah, I think, you know, general like. Like those, um, uh, very simple, like schema documentation cannot be missed. Um, I will say, you know, in a month maybe things change, but, um, uh, like try to keep it tight. [00:36:58] Right? So you talked about guardrails. [00:37:00] Um, and then in our MCP world, we also have tool calls. The more you limit these things, like the less hallucination they have, um. Then we also apply guardrails for very different purposes. Um, guardrails in the sense within our product is about, um, uh, data privacy, um, and also security. [00:37:19] So we also have that too. There are certain things, like for example, if we don't want, um, you know, members of the team to have access to like PII, um, or people who aren't running like secure machines to have access to PII, you can establish guardrails that can prohibit that. Um, and then you can also, um, you know, say like certain, um, like, uh, certain tool calls are also off limits for different roles, et cetera. [00:37:42] Um, so those guardrails that we have within Civic Nexus, um, it has really great like, um, data privacy and security and compliance, um, impact for us too. [00:37:55] Phil: Such a cool overview there. I'm sure there's a lot of stuff that folks have gone down [00:38:00] that similar path. Have also caught like the, the whole semantic discussion, the whole thing about like accounts versus users versus people versus, you know, however other term is coming from different, uh, the, yeah, it's easier said than done for sure. [00:38:17] Um, the, [00:38:18] 6. Using Role Based AI Guardrails Across MCP Servers --- [00:38:18] Phil: the part with this whole journey that you went on, that for me is harder to do, is. Like getting to that decision in the first place, like once you've got approval, like, alright, doing it is, is tricky for sure. Like you just talked about some of the obstacles there, but getting there in the first place, it's like there's a political element to like winning the buying in battle, right? [00:38:43] Like replacing long, longstanding vendor relationships. Especially when like people inside the company have been using the school for like three, four plus years or like certified users in that tool and you're coming in and you're like pitching this idea of replacing internal systems. It's not just a [00:39:00] technical discussion. [00:39:01] Right. Sadly, a lot of the time it's, it's a political one. How did you make the business case internally that we are gonna replace this off the shelf BI tool, we're gonna build something internally with ai. What, what strategies helped you kind of turn a lot of the skepticism, I'm sure that you faced into momentum? [00:39:19] Anna: yeah. Our journey was an interesting one, um, in that, I think with the legacy BI tool, because we're a small company, we're very agile. Um, the, the supply was not meeting the demand. Um, so we were frustrated with the tools that we had. Um, not just that too, it's, it's also BI tools, the way that they price these things. [00:39:42] Um, the more successful you become, the more expensive they get, which you, you know, kind of begs reason. Um, so it. There was a, there was a lot of like, I think, um, momentum behind, like there has to be a better tool, right? But at the time when we were [00:40:00] looking like, you know, call it a year ago or so, um, you're just jumping from one legacy BI tool to another legacy BI tool really. [00:40:07] Um, and then it becomes like a price war. Um, so there wasn't a good time, but when we introduced Civic Nexus, um, and we built Civic Nexus before we, um, walked away from our BI vendor, um, when we built Civic Nexus, which is, um, an, or like an MCP orchestration platform, allowing you to, um, you access all of your various tool within your LLM right to engage with all of your tools unique, um, individually or together. [00:40:41] When we built that, we noticed that, oh my goodness, like we can access all of our, you know, database, uh, or we can access our database not just for like event data, but as we said earlier, you know, your CRM, your support ticketing system, um, you know, your, your Confluence, your internal Confluence pages, [00:41:00] your Jira, um, you can access it all and interface them together. [00:41:05] Wow. The, uh, like your imagination just starts running wild. Um, so the challenge there was more that, yeah, we could do all of these things. Um, but what about compliance? Right? So what I, you know, we are a privacy first organization. Always have been. We, you know, um, are also very identity focused. Um, we are extremely protective of data privacy for our users. [00:41:33] Um, and so for us, like, yes, it's great, but. Um, you know, so I don't, I, I don't wanna give too much access to everybody, right? Who knows? Like someone might leave their laptop on and then, or like, you know, um, go to a website that they shouldn't, like all the what ifs, right? The prompt injections. Like you hear about the horror stories, like on a daily basis. [00:41:55] Um, so when we looked at Nexus, yeah, wanna use [00:42:00] that for this, but then what about, you know, like what are some of the safety concerns that we have, um, by granting so much access, right? And that's where like guardrails, we talk, we hear about guardrails a lot. Um, and for us, like within Nexus for each single MCP server, and let's just take Postgres or Redshift or AWS like as individual cps, you now have the ability to go in and cater and design your guardrail. [00:42:27] So for example, if I want to, you know, I. Strike out email from like, from a certain person's access to an MCP server. I can do that if I want to. Um, you know, black out all PIII can do that. Um, so being able to, at a granular role-based access level, um, design, these guardrails have really changed the way that we view, um, you know, our ability to use MCP server to access, um, data in a trusted way. [00:42:58] So that was compliance was [00:43:00] our biggest hurdle. And, you know, for us, um, civic Nexus didn't just like help with enabling AI coordination. It made AI compliance very manageable for us as well. [00:43:12] Phil: Yeah, it's such an important component to the build versus buy debate, privacy, compliance, and especially as we talk about build versus buy for AI solutions, like you're turning the key over potentially to a third party to just get everything that's in the warehouse. Like it, it gets really scary for companies that are in regulated industries like FinTech, finance and health tech. [00:43:36] Like, uh, my short stand in health tech was like a whole opening of like PHI and PII, like, I spent most of my career in just like SaaS startups and like, you know, like privacy was important, but it was just like, you know, nothing compared to how important it is at a health tech company. Like every single employee does PHI training. [00:43:55] Like it's just a different ball game when it comes to building versus buying. And [00:44:00] that's, that's an element that's, I feel like always gonna be in favor of like the, the building part. [00:44:05] Anna: That's, uh, yeah. For us, you know, there's so many services out there and I'd say that we probably reject, you know, 99%, I don't know if it's an exaggeration, but close like to 99% of them because they don't meet our compliance requirements. We're not interested in sending our users information to a third party that could potentially, you know, do something with it. [00:44:31] So guardrails are huge, and to be able to bring it in-house, that was just absolutely mind blowing. [00:44:38] Phil: Yeah. Um, crazy, crazy journey for sure. Um, I, [00:44:43] 7. Ops People are Creators of Systems Rather Than Maintainers of Them --- [00:44:43] Phil: I'm curious to ask you maybe like, taking a step back on this idea of turning operators on the ops team into builders. Like you talked about how you've built your ops teams to ship a lot more like product orgs. Um, bring us inside those teams for a bit maybe in like, what does that look like day to day? [00:45:03] How do you cultivate, uh, a mindset where s people see themselves as creators and builders of systems rather than just like maintainers and people just go from like ticket to ticket and service other [00:45:15] Anna: Yeah. Yeah. Um, I love this part because, um, like it's changed the way I operate and I love, you know, talking to people and, um, mentoring, um, people as well about this. Um, you know, within operations we see support, operations, sales, product, um, and business operations. Um, and like we really want to elevate these individuals so that the company as a whole can also accelerate, right? [00:45:46] Um, so what we've, what we focused on and what I always, um, not preach is the right word, um, but what I always talk, talk about is don't be inhibited by what the AI tool can offer you, [00:46:00] your imagination. Is the limit, right? Think about ultimately what is your OB objective, and that will help inform, um, the design of what you want to do with ai. [00:46:14] So we're shifting from being execution focused to now being very design focused. And I think that that is happening across many different organization. You are now, you know, taking your ops team and creating, um, an an environment where you're not driven by tasks, but you're driven by like, how do I architect solutions now? [00:46:35] How do I design workflows that really help? You know, myself, like to self scale and then also my team and the organization, uh, the organization to scale better. Um, so I, I, I love like, you know, being able to, as we talked about like AI tools have democratized like building, I mean, how many of us have like, vibe coded on the side, like doing things that were just like [00:47:00] unimaginable before, right? [00:47:01] Like 12 months ago, like maybe I was joking that three years ago I was trying to, you know, use Midjourney and teach Midjourney to my child and now she can make a whole video on Soar, right? She can write, she can vibe code on She is, she is couple, she is less than two digits. Like she can vibe code unlovable, like she can do all of these things, but it's not, I don't think it's enough for people to just like one for one. [00:47:27] I wanna be able, like this is my day to day, like, use AI to replace and do my day-to-day. No, it's now think creatively out of the box, you can do so much more. The sky is the limit. Um, just think about ultimately what your objective is and then design from there. Um, AI tools, they, the landscape transforms so quickly at breakneck speed. [00:47:51] The tools that you are inhibiting you today will not exist tomorrow. So I'd say continuously, like, you know, be [00:48:00] creative, um, think about your ideal, think about your objective, and then the workflows will kind, will work itself out. So I love this subject and I can talk forever about it. [00:48:11] Phil: I love it. I, [00:48:12] 8. Why Natural Language AI Lowers the Barrier for First-Time Builders --- [00:48:12] Phil: I love the example of, uh, teaching your daughter mid journey and then now she's in lovable and, and vibe coding stuff. It's, it's such a cool story. And it made me think of, uh, I just recently had an episode with, uh, one of the founding members of women in ai. So like a community of empowering women that are building, but also just like learning ai. [00:48:33] And she talked a lot about this idea and like a lot of studies have come out showing that like, women are a lot slower to adopt AI compared to their male counterparts. And if we don't do something about it, the gender gap that we see in tech could amplify with ai. What are your thoughts there? 'cause like, obviously you're women in ai in tech, you're doing crazy stuff. [00:48:58] You're teaching your daughter [00:49:00] who's like super young and like the women listening right now that are just like, like I've never vibe coded anything. Like, and he just said like, we've all vibe coded something on the weekend. Like I haven't, like I haven't taken that leap yet. I don't have time to do it. [00:49:12] What advice do you have for those women? [00:49:14] Anna: Yeah. Um, I, I, I think. My best piece of advice is that what AI is lending is this, you know, if you're not technical, this is the perfect opportunity. This is what AI is allowing you to do. All of the super high technical stuff that you know, that you, that you don't have the abilities for is now available to you through natural language. [00:49:39] And I think if you can think in a process, you know, in a process manner, and you have the creativity, um, like you can start like just, this is how I talk, I talk, I talk to my daughter, think about like all the cool things you wanna do, right? And then use your words and start like just explaining it. And that's really what you're doing when you're vibe [00:50:00] coding. [00:50:00] Um, all the stuff like how do I build a backend database? How do I build an authentication layer? You know, there are services where like, I'll get an ex, an example, one of our other products, civic Off, right? Or even for Civic Nexus. You can go to our Dove docs and copy and paste an AI prompt, and AI will build that entire integration for you. [00:50:19] It's so easy now, right? Learning where to access those might be a challenge for some who are super new and you know, to, to the industry. But there's so many tools out there, and I, even the word tools I think is frightening for some, but just use natural language. That's all I can say. Like, we are in a world now that you can do so much with natural language. [00:50:39] So start composing your thoughts. Start thinking about like, what do you want that user experience? What is like the, like the functional objective of what you're trying to achieve? Like, like start thinking about what the value is of what you're creating and then use your words and the rest will kind of come into, come into place. [00:50:58] Um, as [00:51:00] far as like gender gap and, you know, for women in tech, and I've been in tech for a long time, um, you know, I started out. Um, in, you know, traditional finance. Um, and then I moved into, um, blockchain payments. And so I've been through like all of the industries that are heavily male dominated and have never been more excited. [00:51:20] Like this. AI is this very sweet spot of, for people who have like some technical fluency and then also the ability to really like, form cohesive, um, business thought and logic. It's, it's this revolutionary time and I think that, you know, everybody, men, women included, should all really try their hand at it. [00:51:43] Phil: Love your answer, Anna. Such a, such great advice, like everyone says, you know, like, just jump in, do something, come up with a cool idea and just like do it. But like, there's like something missing in that advice between like the come up with a cool idea and then just do it. Like in the middle there, your advice is just [00:52:00] like, explain it with words. [00:52:01] Like, try to figure out semantically, what does that mean? Like that crazy idea you have, like this, this thing you want to build. Like everyone has cool ideas, but I feel like that step of just like sitting down, writing out what that means, explain it in words like that's, that's half the job right there. [00:52:18] Like, turning that into code. Like that's where these tools come in and then you can kind of discover that part of the world. But yeah, really, really great advice. Uh, didn't plan to ask you that question, just kind of like jumped on a whim [00:52:30] Anna: I love it. I love it. [00:52:31] Phil: yeah, it's, [00:52:31] 9. Technical Literacy Requirements for Next Generation Operators --- [00:52:31] Phil: it's related to the, like the. The operator of tomorrow. [00:52:35] I guess if, if you wanna call it like a ai, like you said, is completely redefining and, and in our time, like what it means to like create stuff, but also lead operations. Um, a lot of folks listening are in ops, marketing ops, data ops. Um, what qualities do you think really distinguish this next generation of operators? [00:52:55] You know, a lot of folks talk about curiosity, but there's like technical literacy that's super [00:53:00] important here. Is there something else entirely? What are your thoughts there? [00:53:04] Anna: I mean, I think, you know, as I said earlier, like right now, the sweet spot is for anyone who has like little, you know, some mid-level technical literacy, it, it is the sweet spot. But I think that barrier to entry is gonna get lowered over time. Um, genuinely because, and we talked about at the beginning of the show, we are moving at breakneck speed. [00:53:25] Um, part of the challenge is really keeping up and with anything, right? To keep up with something that is like, that is so demanding. Um, and you know, like the, with like the amount of technical literacy that is required, you have to be interested in the subject and you have to be curious. Um, curiosity to me is the most important, um, because curiosity then feeds imagination. [00:53:55] And to me those are like the two like. [00:54:00] Like if you were to succeed, I think curiosity and imagination, um, are like the building blocks for success in this area. Um, you know, I don't think about like, do I need to, in order to be a successful AI company, should I go hire someone with like AI experience? Um, for what we're doing, I think that there's so many, so much potential within your existing organization, right? [00:54:23] You start with those who, like you said, like that fit the mold of, have some level of technical literacy and then let's work with them to identify, you know, what are your typical workflows look like. Well, what is it that you really trying to get out of your workflows? And then let's think about, okay. So an example would be for the BI tools. [00:54:41] I'm not looking to build dashboard one for one because as we said, they hallucinates. I have tried spent hours, days, nights, like building like one for one dashboard. They look pretty, but are they right? Do they conk out? Do they refresh on demand? Like, there's so many challenges, but ultimately, what was I trying to achieve [00:55:00] with those dashboards, right? [00:55:01] The idea was that I wanted to get data insight to people who wanted it at a time that it mattered, right? So that they can make data-driven decisions. Now, where do we spend most of our time? Most of us, like in an organization, like regardless of what function you're in, you are spending a, you know, a good amount of your day in Slack. [00:55:23] So I will meet you where you're at. I will give you exactly what you need, where you're at, you know, court 25% of your day. So rather than building the dashboards, like I asked myself the question like, well, what type of data? Who wants what type of data out of a dashboard? Let me separate them out and let's just feed it into Slack. [00:55:43] So by using Civic Nexus and orchestrating that workflow together, I brought the data in, I brought Slack in, right? And then I built a whole flow where now we are pulling data right out with like very sophisticated SQL queries, right? Nobody's like querying in plain language [00:56:00] there and then feeding that directly into Slack. [00:56:02] I do not need a visualization dashboard anymore. So I think, you know, there's, I challenge everyone to think about, yeah, you did this before, but is that what you really want? What was the objective and goal? And could we meet that in some other way? Um, and to really think more creatively when looking at your workflow. [00:56:24] Phil: Love, love the challenge, especially folks in, in MarTech that have been using the same systems for so long. Like really think about whether you've been doing this the right way and if there is a better way. Um, and this has been such a fun conversation. I feel like we could keep chatting. There's a lot of topics here that you're obviously really intelligent and super well researched on. [00:56:45] Um, [00:56:46] 10. Why Creative Practice Strengthens Operational Leadership --- [00:56:46] Phil: we are running outta time though, so I got one last question for you. You're a vp, a team leader, you're a seasoned ops pro, but you're also a full-time mom and a part-time vibe coder. You're also an avid wheel thrower. One question we ask everyone on the show is how do you remain happy and successful in your career, and how do you find balance between all the stuff you're working on while staying happy? [00:57:07] Anna: Yeah. Um, I love this question. Um, you know, you don't get it all the time, but, um, I think it's super important for me to have a balance of both. Social and social and solo creative outlets. I, um, work is, I, I love the challenges at work. I love constantly learning new things at work, but, you know, it's, it can be kind of, um, uh, oh taxing on the mind. [00:57:31] And so I, I love a creative outlet. So when it comes to wheel throwing, I love sitting in front of a wheel and it's just me, the wheel, my clay, and then what I want to create that day. And for anyone who's like ever thrown at a wheel, like you fail so many times before you can, like, you know, what you thought was gonna be a bold, turned out to be a pinch pot. [00:57:52] And so you, it like, it's humbling. And so not only is it creative, but you learn from the failures and you [00:58:00] learn to iterate. Um, so I love that outlet, um, that outlet. And it really helps me build resilience over time. And I think some of that translates into my professional life too. [00:58:12] Phil: Very cool. And, uh, yeah, I, I actually didn't know what wheel throw in was the first time you mentioned it. And, uh, now that you're explaining it with clay and you see like all the videos on Instagram, like just so many people doing wheel throw in, it's something I have to give a shot because it does look super fun. [00:58:27] Anna: It's, uh, I mean, it's like a golf game. You're, you're committing three hours to this spinning thing and Yeah, just everyone's doing the same thing. So it's, uh, I, I recommend, highly recommend it. [00:58:40] Phil: Awesome. I got a young daughter myself, uh, is two and a half years old, so, uh, I think a lot younger than than yours, but I hope one day to get into like AI tooling and, and teacher about that side of the world too 'cause Yeah, it's, it's super exciting [00:58:55] Anna: And also safety. [00:58:57] Phil: Yeah. Yeah. Especially [00:59:00] like AI generation images, like yeah, that's a good call out. [00:59:04] Anna: Absolutely. [00:59:05] Phil: Awesome man. Thanks so much for your time. Really appreciate it. [00:59:08] Anna: Thank you.