Subscribe
Share
Share
Embed
Candid conversations with the builders shaping the future of engineering.
Braintrust dives into the operational realities of running high-performing engineering organizations, from production readiness and migrations to AI adoption and operational excellence.
Hosted by Ganesh Datta, CTO & Co-founder of Cortex
Steve Flanders (00:00):
But beyond engineering, I mean, there are so many other parts that I think are way too slow. I mean, any process that you have today is likely mostly manual, not automated. So unless you're infusing AI or maybe even redoing those processes, do you need a PRD in this new AI world? You need to answer some of these questions, I think. But if you don't fix the other stages, then you're either going to create a bottleneck somewhere else like the code review part, or you're going to be slow because the steps that come before the code itself are actually slowing you down.
Ganesh Datta (00:35):
You are listening to Braintrust by Cortex, where we explore how engineering leaders blend AI, platforms, and culture to build high-performing software teams. I'm your host, Ganesh Datta, CTO and co-founder of Cortex, an engineering operations platform designed to help organizations continuously improve their operational maturity and reduce developer friction. In each episode, we go deep with CTOs, VPs of engineering, and technical leaders who've been in the trenches navigating the tension between speed and quality, building reliability at scale, and figuring out how to lead through major platform shifts. Whether you're running a team of 10 or a thousand, this is your space to learn from people who've made the hard calls and live to talk about it. Today, I have on with me a expert in observability, Steve Flanders, great to have you on.
Steve Flanders (01:31):
Thanks so much for having me. Yeah, so I've been in the observability space here for almost two decades and have kind of seen it all from end user managing these systems to actually designing them from an engineering perspective. In addition to the observability space, I'm also leading AI transformation currently for my business unit. So how do you infuse all the great AI technology that's available and actually try to get to delivering customer outcomes faster really at the end of the day is the goal. But that's a little bit harder than it sounds and maybe something that we'll talk about today.
Ganesh Datta (02:03):
Absolutely. The other thing I noticed is you've written the book on open telemetry literally, and we can see right behind you. If you want to give us a quick plug on the book before we dive in.
Steve Flanders (02:14):
Yeah, yeah. So before I joined Splunk, which is part of Cisco now, I was part of the founding team at Omnision, which is now the Splunk APM product line. Two of the things that we did is build Splunk IPM and contribute to OpenTelemetry. We actually built the OpenCensus service that's now the OpenTelemetry collector. And I've been involved in that project since the very beginning, something that I'm actually very passionate about. And yeah, I wrote a book on this topic, both observability and OpenTelemetry and bridging that together and hopefully you'll take a look at
Ganesh Datta (02:44):
It. Definitely would recommend folks take a look. Well, because you're the expert in observability and now thinking about AI, I think a conversation about both of those will be really, really fascinating. I would love to start with something you've talked about recently, which is this idea that code was never the bottleneck, but it was the most expensive constraint in software engineering. I think it echoes a lot of similar sentiments that we hear from folks like Dora. Honestly, almost the exact same tagline that we have in our website. So it clearly is a very kind of obvious, well, not obvious, but it's a very important thing for folks to understand that we're not optimizing for the same set of constraints anymore. But what did you mean by it was the most expensive constraint?
Steve Flanders (03:25):
Yeah, so I mean, if you think about it from where does the money go from a business perspective, your engineering organization is quite expensive. You're hiring senior talent at a premium cost with the hopes of being able to deliver customer value as quickly as possible with a high quality bar. I mean, it kind of depends on where you are. Startups try to add new capabilities or offer differentiation where enterprises might focus a lot more on security and quality and reliability type aspects. But you need to get a pretty solid team in order to achieve those business outcomes at the end of the day, and that's not cheap. So the hope is that with AI technology, you can commoditize some of this. Learning a programming language is easy. Being able to run and operate enterprise software at enterprise scale is not. It requires a whole bunch of subject matter expertise, just domain experience over time, and a large enough network of other engineers you can work with.
(04:23):
But AI is starting to transform that a bit, at least for more of the junior work. You can start delegating tasks to it. You can start solving at least simpler tasks with it. And so it's kind of freeing up a constraint that existed before, which is, hey, I have to wait a pretty long time to get this code. Now, to be fair, the coding aspect though, I mean the engineering thing is a important resource, but generating the code often isn't the slowest part. There's an entire software development life cycle, and that's only one piece of it. Maybe the most important because you don't get the customer value at the end of the day if you don't release the code. But getting to that code and getting it to production also takes a very long time. And that's not really where AI has been focused yet.
(05:11):
I mean, it's starting to evolve a little bit. We're seeing product management or design use cases starting to pop up a little bit more now. So I think it will get there. But the initial focus was on the engineering part because the hope is that you could move faster and hopefully do it in a cheaper way. We'll see whether or not it's cheaper long-term though. Yeah.
Ganesh Datta (05:29):
And to that point, I think what's interesting is AI is obviously affecting every part of the SDLC directly or indirectly. When you think about a lot of the studies over the last five to 10 years, there was this general concept that developers spend a very small percentage of their time coding. And I think that's kind of what you're describing is it's a very small percentage of the SDLC. I do think a lot of the time we were spending heading into the actual process of writing code was important because the cost of writing code, like you said, doing the wrong thing was expensive because engineers are expensive. And so if you spend even a week building the wrong thing, it was really, really bad. And now you have compression in different parts of the SDLC where maybe before you would've had to have three meetings before you go write a line of code because it was really important, but now you can just prototype it and then go have a conversation about it.
(06:19):
But then suddenly the constraints are just shifting. And I think that's what you're describing is Goldratt's Theory of Constraints. The constraint just changes. You fix one and something else becomes a constraint. So where are you seeing now that constraint move? What do you think is going to break next if it hasn't broken already?
Steve Flanders (06:35):
Yeah, I mean, if we stick to just the coding aspect for a second, I think I've seen two things post-code that is pretty interesting. So one is code review time is going up because a lot of these PR pull requests are larger in size. And so it's much harder for a human to actually be able to review it and ensure that it's doing the right thing. So now you have the code, but it's slower to actually ship it to production because it's not as easy to review, or maybe you just don't have the right quality gates to ensure that it's doing what you expect out the other end. The other kind of category from an engineering perspective is maybe you are getting those code reviews done quickly because you found a way to generate smaller MRs. But when it gets to production, you end up running into quality resiliency, or I think the new DORA metric's called rework rate.
(07:23):
You have to rework what you actually shipped. And I guess if you can do that very rapidly, maybe it's not that big of a concern, but if you can't, then the rework work actually has a negative impact to the business. But beyond engineering, I mean, there are so many other parts that I think are way too slow. I mean, any process that you have today is likely mostly manual, not automated. Even stitching together things. If you assume that a product manager is generating, let's say, a product requirements document, and then they hand it to engineering, which does an engineering requirements document, and then you have some sort of architecture review of that, and then you have to go through some security and compliance team, each one of those stages has their own processes that you need to follow. And any of them can be slow or all of them can be slow.
(08:09):
So unless you're infusing AI or maybe even redoing those processes, do you need a PRD in this new AI world? You need to answer some of these questions, I think. But if you don't fix the other stages, then you're either going to create a bottleneck somewhere else like the code review part, or you're going to be slow because the steps that come before the code itself are actually slowing you down.
Ganesh Datta (08:31):
I completely agree. And in fact, I would go a step further and say that I think historically would've been some of the quality gates like code reviews are honestly becoming unsustainable. I don't think it's going to be possible in the near future to truly review every PR that we're producing, even if those PRs are smaller and smaller. I think smaller PRs will make it easier for AI to review those things. But I think many organizations are maybe holding onto an older way of working even when the forward pressure of the amount of code we're producing is changing. And so the validation I think is kind of shifting, like you said, to two different ends of the spectrum. It's like, are we building the right thing and the PRDs and things like that? And maybe how that transforms is yet to be seen. And then in production, it's like, well, did it do the things that we wanted it to do?
(09:19):
But code reviews and those kind of human in the loop processes that we had before are maybe not going to last as long as we think it might. And so on that note, on the observability side, you mentioned rework rate as being one example, but rework assumes that you know that something was wrong and you changed it and you went back and reworked it. What does it look like now on the observability or the path to production side of the house? How are those constraints looking like in practice? Because we're producing way more code than we ever did. So what are you seeing in that post-code part of the life cycle today? How does that look?
Steve Flanders (09:59):
Yeah. I think what we're seeing is that maybe we didn't have the best guardrails to begin with. At least most companies don't have the best guardrails of getting code into production. And so you're starting to see the cracks in that because you are seeing failure rate increase or customer found defects increase or whatever you call that bucket of work. So that's a bit of a problem. And then from an observability perspective, if you're observing your systems, it can be hard to know. Is this because of AI-generated code or is it because that intent is not correct or is it something else? I guess a good example I can give is that a lot of observability products are adding AI capabilities like some sort of chat assistant, for example, to help you query over your data and hopefully get you to root cause and remediation faster.
(10:47):
That's great because a lot of observability platforms are kind of proprietary in some way, shape or form. They have a query language or some way of getting that data or representing it, and you need to understand the system that can take time. So I think there's benefits to the AI feature sets that are being added. But the problem is if you are ingesting telemetry data that is not well correlated, doesn't have the right metadata, doesn't have the right data, you're asking AI to search over nonsense almost. And so it can get you like, here's the results, but it may not be the answer or the problem that you're actually trying to solve. So data hygiene becomes a bigger problem that people weren't really worried about before. Before it was just collect as much telemetry data as possible. Hopefully you can query over it. Now it's make sure you're ingesting the right telemetry data that gives you the context and correlation.
(11:39):
Otherwise, it's garbage in, garbage out when it comes to AI capabilities being built in your observability platform.
Ganesh Datta (11:45):
What does meaningful data mean? Especially from the context of AI, having focused now on the AI transformation aspect of things, you must have an interesting perspective on what AI is particularly good at parsing over.
Steve Flanders (11:58):
Yeah. I mean, again, sticking to the observability use case, you've probably heard of the three pillars of observability, traces, metrics, logs. Of course there's more data than that, but if we just use that as the basic premise, metrics tell you that something happened. They don't necessarily tell you why it happened. Usually logs will tell you why something happened. You get a little bit more of the behavior or understand what changed, but you don't understand how do you correlate those two things together? If you're ingesting metrics and logs separately, which most people do today, and unless you have metadata or some way to stitch those two things together, it's very, very hard to go from a symptom to a root cause from a metric to a log basically at the end of the day. And so with AI, there's kind of your problem. If you tell AI, "Hey, my CPU is high, I want to understand why," it needs to be able to query over the logs that are relevant to whatever host or application you're on and that has the high CPU utilization.
(12:54):
It needs to understand was there a code change? Did we do a code push to production or something else? Is this just happening on one host or another? If you don't have that metadata, that context and correlation, it can just say, "I see high CPU and it looks like it's happening in this region." That's not good enough. That doesn't get you to the actual root cause or the remediation aspects. So I think spending more time upfront understanding what you're collecting and how you're collecting it matters. And if I now take the observability use case and go back to engineering, it's no different. You have to define your intent upfront. It means your engineering requirements document has to be more fleshed out than it ever has before, because if you don't define the intent correctly to the AI, there's no way you're going to achieve the outcome on the other end that you asked it to do in the first place.
Ganesh Datta (13:44):
That makes sense. Yeah, it's interesting because a lot of observability practices were designed for humans in the entire loop, code run by humans, code reviewed by humans, dashboards consumed by humans, alerts triaged, and incidents resolved by humans. And every part of that stays is changing. And so each part of that step has new validation requirements, it has new inputs going into it. And what it sounds like is similar to how a lot of the industry's talking about harness engineering, where the more you can feed in verification systems to the active writing code, the better the code will be. And a lot of those verification systems are product requirements and things like that and ways to validate it. But it sounds like what you're saying is observability, the design of the observability of that feature that you're implementing should be part of that initial set of designs so that in production you can validate with AI.
(14:43):
And so it's thinking about a different state of the SDLC from an AI lens that we haven't thought about before. Is that the right way to think about it?
Steve Flanders (14:49):
I think so. I mean, I think it was always there. And just like the example I gave from the engineering side, it's just becoming more apparent like the cracks in the system, like the telemetry data. Telemetry's always been important. Context and correlation has always been the goal of observability products to get you quicker to ideally remediation. But most people just want to get to root cause where they can bring in their own subject matter expertise and then fix the issue themselves. But at massive scale, as the data increases and as you have AI querying over it, it's just the telemetry data becomes so much more important than it was before that you have to look at it. And maybe before the AI era, the driving factor was more cost. How do I get rid of the noise and the non-valuable data? In the case of AI, it's not only do I need to do that, but I need to make sure it's enriched properly, that I have the context and correlation that I need to answer the questions that I'm going to have in order to get my systems back online.
(15:45):
So I see it as just showing more of the cracks that have always been there. I don't think this is a new problem. I just think it's much more apparent because of what AI introduces.
Ganesh Datta (15:54):
100%. I mean, like Dora said in their recent report, AI is just amplifier. It's making things that were bad worse and making things that were good better. And it's true across every organization. I definitely, definitely believe that. Maybe zooming into this into more of a tactical lens, let's say you're shipping a new feature. I don't know, let's say, let's pick a simple example, maybe not simple, but you're sending Slack messages and you're like, okay, I want to design observability upfront. What does good actually look like? If you were to give the listeners advice on how to think about, I want to instrument this feature and I want AI to be able to get to the root cause and remediate things, how should I be thinking about instrumenting a feature like sending a Slack message? Or pick another example if you can
Steve Flanders (16:43):
Do that.
Ganesh Datta (16:43):
Sure.
Steve Flanders (16:44):
Any example. I mean, look, my personal belief is, especially in the observability market, some standard like OpenTelemetry is the way to do this. That's my personal belief. It has nothing to do with the company that I'm at or what have you. I believe that having a standard is how you can achieve some of this. And that's no different than standardizing on an AI tool chain at the end of the day. You get the benefits by having a single way of defining your skills, doing your code reviews, things to that extent. But what OpenTelemetry provides is not only will generate the telemetry data, it actually adds, it's called Semantic Conventions, fancy word for metadata, a standardized way to tag your data across these different telemetry sources. Now, where you send the data is the next most important thing. So if I'm using OpenTelemetry to generate, process, and emit my telemetry data, then the only other question remaining is where am I storing it?
(17:33):
And that could be one or multiple destinations. But a lot of people today will kind of separate. They'll send their metrics to one destination, their logs to another, and traces to a third. That's okay because optimizing your data store for the usage pattern that you have kind of makes sense. Some vendors will offer you one store where you can ingest all three. Some give you unique stores. That's irrelevant. The question is, can you actually query over all of that data in a reasonable way? Things like high cardinality becomes a problem. How can I get multiple dimensions at once, but do it in a cost-effective way at scale? It's because as I increase my number of services, I'm going to increase my amount of telemetry data. As I have more load on my applications, I'm going to increase my telemetry data. All of that has a cost.
(18:19):
And unless you can get value out of that cost, you don't want to incur that cost at the end of the day. So you're striking the balance between sampling and querying over that data. But if you don't standardize, let's say that you ingest metrics in Otel format, but you ingest logs in some other format and you can't correlate between them, that's where I think you're going to have a very bad time in the AI era because AI may be able to query over both data sources, but it won't be able to efficiently, or even with a high quality bar, be able to correlate things for you. It'll start to make assumptions, and maybe you will make the wrong decisions based on those assumptions, or maybe you won't get the insights that you need in order to solve the problem in front of you, which defeats the purpose of having an observability product at the end of the day.
(19:07):
So really the summary in my mind is you need a standard. OpenTelemetry is trying to help solve this problem. It doesn't have to be OpenTelemetry. There's plenty of other open source things out there like Prometheus for metrics, for example, or Fluent D, FluentBit for logs. There are other ways to solve it, but you need to find a way to standardize. This is easier, I think, for smaller companies, startups, SMBs. It's harder for larger companies because if you're in a brownfield environment, trying to change the way that you generate or collect telemetry data is very, very hard. If you acquire a bunch of companies and they use different tech stacks, again, it's hard to consolidate onto one. So I'm not trying to say this is an easy problem, but you have to find a way to at least get basic context and correlation across all the data that you're collecting.
(19:54):
Otherwise, AI really isn't going to help you. If anything, it might make it harder for you to achieve observability.
Ganesh Datta (20:01):
Absolutely. I mean, it sounds like, to your point, something like this might be difficult to roll out across a large organization, but if we believe, and I think the data shows us that although AI is helping us move faster, it is increasing incidents and rework, then things that can provide leverage in improving that side of the house, of improving quality, improving reliability are worth the investment, then standardizing on something like this that can make AI better at solving these kinds of issues for you seems like a high leverage thing for organizations to be investing in. Yes. At least that's what it sounds like.
Steve Flanders (20:39):
I agree. Yeah. I mean, look, this could be solved in the backend as well. You could do this as a post-processing step, but there's a cost incurred with that too. Maybe that's the easiest thing to do today as we're starting to figure out where AI is transforming observability or again, for large enterprises where change is hard and changing telemetry data is non-trivial. So I don't want to say it's the only way, but I believe it's the long-term sustainable way. You need to find a path to get there and fix it more on your side where the telemetry generation is occurring. Otherwise, at scale, I think it's going to be very difficult.
Ganesh Datta (21:12):
Absolutely. Yeah, it sounds like it's similar to most things AI. It's not that we necessarily need new technologies or new ways of doing things. It's going back to the basics and it's less AI is making us change how we do observability at an implementation level and more, again, people process culture. How do we think about this stuff? How do we standardize on the right patterns? How do we instrument things earlier and think about that earlier in the life cycle? So it's more of a people process culture thing than an actual tools technology thing.
Steve Flanders (21:43):
Yeah. And I think we focused a lot on the telemetry side because that is going to remain important. But long-term, I think that there will be observability changes. Once you have the data in your observability platform, what's the next thing you want to do? Typically build dashboards and alerts. Well, a lot of the dashboards today are static. You have your own dashboards. They're different than my dashboards and they might solve the same problem. A lot of dashboards are not used. They just sit there, you build them once and they sit in the system for years. Then you have alerts, you have alert storms, or you don't have context or correlation across them. All of those go back to what you were referring to before. You assume human in the loop. But is that going to be true as we move forward? As MCP architecture takes off, if you have more agentic flows, do I need to have static dashboards?
(22:30):
Could I have a dynamic dashboard? Could I have a way of having these alerts be handled and handle aggregation of these so I get one alert instead of a thousand alerts? Maybe this will look different in the future. And we haven't seen a lot of it yet because again, I think change is hard, but I really think that the experiences on top of the observability platform are going to have to evolve as AI becomes more mainstream.
Ganesh Datta (22:55):
Related to that, one of the things that I've been thinking about is a lot of people don't know where to start. They want to augment their observability stack and their operations with the AI. And to this point, it's like there are things you could be doing to be improving the actual metrics and traces and logs you're collecting so that AI can do better about it. But let's say one of the problems that we have today is people don't really know what code they're writing anymore. If they don't know what code they're writing anymore, they don't necessarily know how to instrument it. And so if you have this chicken or the egg problem, what can people do to break that loop of the chicken or the egg?You've shipped a bunch of code into prod and it's like this slop, for lack of a better word, and you know things are breaking.
(23:36):
How do I go back and say, okay, these are the things I need to go back and now add instrumentation for? How should people think about that?
Steve Flanders (23:43):
Yeah, I mean hopefully if you're going down this path of taking more of an AI first approach, whether it's vibe coding or agentic workflows or however you're doing it, doesn't matter. Hopefully you are checking in your intent with the code that you're actually generating. Because the idea should be that A, you can go back to it later and hopefully be able to achieve a similar outcome using that intent. But B, someone else could actually go look at that code and again, start where you left off and build on top of it. The next thing I think, maybe it's something like a scorecard. How do I score a repository and say, do I have a good agent's markdown file? Do I have the right intent here? Did I think about security and compliance? Did I think about performance? Did I think about quality gates, unit test coverage, integration testing, end-to-end testing?
(24:33):
How do I make sure that's kind of standardized across every code repository that I have? Those things I think can help you. The good news is if you know what you're looking for, you mentioned observability, for example, in your code, AI is more than capable of adding OpenTelemetry to your existing code. OpenTelemetry is an open source project on GitHub. It's actually really good at it. So if you're like, "Hey, I'm having trouble solving these types of problems. I'd like to use OpenTelemetry and at least provide metrics, let's say, into my application, please add it." And you add that to your intent specification, it will go ahead and enrich your existing exact application code for you. Now in practice, it may be a little bit harder than that if I'm being fully transparent. It will add instrumentation. That doesn't mean it's the right instrumentation. That doesn't mean if you add the instrumentation and you don't ask it to confirm that performance doesn't go through the roof because you added that instrumentation, it brings down your application, for example.
(25:31):
So I think you need someone senior enough to define the right intent upfront and to think about all the different scenarios of the implications of that intent. Adding instrumentation is one thing. Adding overhead is another. Are you thinking about both? Just because you have traces, metrics and logs doesn't mean that you have observability. What are you actually instrumenting? Is it entry and exit calls? Is it function level things? Do you have infrastructure metadata like what host or pod or Kubernetes environment it's running on? It's tagged onto there. The intent part becomes so important. But the good news is it is possible with AI to do this pretty quickly. You just have to be able to think through and reason with the AI about what intent you're trying to drive.
Ganesh Datta (26:18):
And to your point, I think it's become easier now for organizations like experts within organizations to scale that influence and those practices across the org. You have an SRE team define skills that are embedded as part of your standard engineering tool chain that know how to do things, that are using the right tooling and based on your own internal practices. And so I think in a lot of ways, some things have gotten harder, but I do think things like this have definitely gotten easier or easier to enable across the organization, which is really interesting.
Steve Flanders (26:49):
I assume standardization is the only thing that I'd add, right? I've seen a lot of large companies that they have different CICD systems or maybe different business units do things differently. That becomes harder because now you're implementing the same solution multiple times. But again, going back to standardization, if you can find a way within your company to have the same tool chain, you're adding it once and everyone can benefit. Absolutely agree. A DevProd or DevEx team can go a long way to enable this or an SRE team as well. But it just becomes harder if you have three different ways to commit code and three different CI/CD systems versus
Ganesh Datta (27:25):
One. I'll use opportunities to plug. I released a new framework for measuring AI accelerated development last week after a ton of research. And one of the pillars in that is non-product engineering initiatives. And I think it's stuff like this, like standardization of tool chain, getting people to adopt certain practices across all repos and whatnot. It's a thing that people talk about a lot. But I think to your point, now with AI, the amount of leverage you can get by standardizing things and driving these non-product engineering initiatives is so much higher because it just creates a compounding effect. So I think it's in most organizations' best interest to really invest in this kind of standardization practices and projects now more than ever before.
Steve Flanders (28:08):
Yep. Completely agree with you.
Ganesh Datta (28:10):
Maybe my last question, listeners who are listening to this might hear this and say, "Oh my God, this sounds like a lot of work. I got to standardize telemetry. I got to figure out what patterns we're going to use. I have to go back and re-instrument things or remove traces and all this stuff. I'm going to do all this stuff. My bosses are telling me I just got to focus on shipping right now." I think that's flawed, but is that the right way to think about it? Is it one or the other or do these things feed into each other?
Steve Flanders (28:38):
Yeah. Look, I get the pressure of leadership coming top down and demanding that we go faster and fix all these things all at once. But think about any problem. Let's say you have a customer problem that you're trying to solve and you're on the engineering side. You're taught very early on that it's iterative software development. You're not trying to build the car and only ship it when the car is fully ready. If I can get you a skateboard quicker, I'm going to get you a skateboard so you can get going and then I'll build you a bicycle next and we'll go to a motorcycle and maybe I'll get you a car in the future. You have to have some iterative software developments or better definition of MVP or whatever you want to call it in order to be sustainable here. So this is true in the AI world as well.
(29:23):
You're not going to change your entire SDLC overnight. You're not going to fix all of your telemetry data for observability overnight. It is going to take time. The question is, what is your strategy? What are you trying to achieve? And working back from there, what are some milestones that you could hit along the way? I think AI tooling, especially internal AI tooling might be an easy way to talk through it. Initially, a lot of people just focused on adoption and usage. I didn't care if you were getting value with it. I cared that you were actually trying to use the AI tools and trying to learn them, discover, figure out what works versus what doesn't. Eventually you reach a tip. Ping point where you have enough of the organization that are actually on the tools themselves and are using it on a regular basis. Okay, well now we can move on.
(30:07):
How can I start showing some value on top of that? In the case of engineering, maybe it's I want to increase the number of pull requests that are AI assisted, so I'm going to start tracking that KPI and start working with my leaders to kind of improve it over time and get it to some baseline, 50%, 70%, whatever it is. Again, getting to AI assisted pull requests doesn't necessarily mean that you're being that much more productive, doesn't necessarily mean you're delivering customer outcomes, but it's getting you one step closer. Then you can look at your SDLC and be like, "Hey, I have a slow part over here," whatever it is, product requirements document, engineering requirements document. "I'm going to take an AI first approach here. How can I infuse AI into just this one piece of the SDLC? And then I'll figure out how to stitch it to the other pieces later.
(30:52):
"Taking that iterative approach is how you build the momentum to actually get to the desired outcome that you want to achieve. Same is true for telemetry data. You don't have to convert everything to open telemetry overnight. In fact, in my book, I cover the difference between greenfield and brownfield. If you're in a brownfield environment, the easiest thing you can do is deploy the open telemetry collector between your applications and wherever you're sending your telemetry data. If you can put it in the middle there, don't change anything else, don't re-instrument your app, don't change your agents, just have your agents point to the OTel collector and have the OTel collector send directly to your backend. You've already provided value because it can add the semantic conventions. It can reroute data for you. It can process the data for you. It's one step closer to improving where you currently are.
(31:38):
And I think that's what matters. So the pressure from leadership isn't going to go away by any means, but if you can start showing like, Hey, I'm thinking through this, I'm taking the right steps to actually improve it, long-term you will achieve the outcomes that you're trying to. I
Ganesh Datta (31:51):
Think that makes sense. And I think it aligns also with leadership outcomes. If your goal is to move fast, it's one of those things where if we know that the downside risk of more AI assisted development is regressions and quality and rework and things like that, which we know, then actually investing in some of this stuff upfront can lead to higher adoption and more throughput and all the things that you want because you have more confidence in the systems, you're able to resolve things more quickly, and it can actually lead to the right feedback loop over time. So it's not going to be either or, it's those things feed into each other. It's just a matter of how do you prioritize? How do you do it incrementally, like you said, and can deliver that value over time. Well, Steve, it was awesome having you on.
(32:30):
Thanks for sharing your expertise and wisdom around observability and how things are changing with AI. It was really great having you on the podcast.
Steve Flanders (32:37):
Thanks so much for having me.
Ganesh Datta (32:45):
Thanks so much for listening to this episode of Braintrust. If this resonated with you, do me a favor. Share it with another engineering leader who's wrestling with these same challenges. And if you want to continue the conversation or learn more about how we're thinking about engineering operations platforms at Cortex, reach out to us at cortex.io. Thanks for listening and we'll catch you on the next one.