Subscribe
Share
Share
Embed
The development world is cluttered with buzzwords and distractions. Speed, focus, and freedom? Gone.
I’m Nicky Pike. And it’s time for a reset.
[Dev]olution is here to help you get back to what matters: creating, solving, and making an impact. No trend chasing, just asking better questions.
What do devs really want?
How can platform teams drive flow, not friction?
How does AI actually help?
Join me every two weeks for straight talk with the people shaping the future of dev.
This is the [Dev]olution.
Jason Baum (00:00):
You ask the testers or people who are creating tests. The creation of the test isn't necessarily the problem, it's the maintenance of that test. And so we're solving the wrong problem by saying AI, write the test for me. Instead of how can we make sure that this is the right test and manipulating the test and getting the results and finding those signals through the insights. That's where testers spend most of their time and not necessarily in the creation authoring of the test itself.
Nicky Pike (00:30):
This is Devolution, bringing development back to speed, back to focus, back to freedom. I'm Nicky Pike. Okay, so everyone is talking about AI. It's everywhere these days, but here's what nobody's saying out loud. AI may be making some developers slower. Yeah, you heard that right. A recent study found that experienced developers are taking 19% longer with AI tools and everyone is pretending that this didn't happen. Meanwhile, enterprises are throwing millions of AI tools while this study claims that developers are spending more time debugging AI hallucinations than actually shipping code. With me today is Jason Baum. He's an expert in developer productivity and quality. Jason has worked extensively in the space of automation and testing, and he's here to kind give us the unfiltered truth about what's really happening when AI meets software.
(01:19):
But before Jason and I get started, here's the challenge I'm throwing on the table. Now, hang on because I'm going to be slinging facts here like we're slinging hash browns. Stack Overflow's 2025 survey just dropped in. 84% of developers are using AI tools, but only 33% are trusting the output. That is a massive trust crisis that we're seeing right now. The Qodo study found that 78% are claiming productivity gains, but only 66% are saying that they're almost correct AI solutions that need constant debugging. And the real kicker? Fastly found that senior developers are shipping two and a half times more AI generated code than juniors, but 28% say fixing it offsets most of their time savings. So how the hell do we navigate this paradox where everyone's adopting AI that they don't trust, then they're claiming that they're getting productivity gains, but they're spending more time on debugging. Hey Jason, welcome to the Devolution, man. How are you doing?
Jason Baum (02:12):
Hey Nicky, thanks so much for having me. I'm doing great.
Nicky Pike (02:15):
Awesome. Well, I'm just going to dive right into the questions we've got a lot of talk about here. So first off, AI has been our sparring partner for years. It's been helping us train. It's been taking the hits for us and it's been making us better. But it seems like that AI is now kind of wants its own fight card. It's ready to step in the ring and start throwing hands for the title shot. How do you think that AI went from punching bag to main event?
Jason Baum (02:37):
I think that it's interesting, the evolution, sort of where AI came from to where it is today, and I think the fact that it's now the main event, I think, is part hype, right? There's a lot of hype initially still. I think if you just literally just walk down the hate and start yelling, I have an AI company, they just throw money at you VCs, but there's true benefits to it that we've all seen as an assistant. It's wonderful. I think where we start having the issues that you were talking about was that trust, right? It's how much are we asking it to do? And we're sort of running before we walk, I think a little bit with AI.
Nicky Pike (03:15):
Well, and I think there's this aspect of when it first came out, it was immature. People were looking at it and then you started seeing all these fear claims coming out, taking people's jobs. This is going to take over everything. We don't need developers anymore. And then we're starting to see that caused people to disprove. It started getting into AI and they found out, well, crap. Yeah, there is some really helpful things here. There is some truth to this, but there is also still a lot of hype and it really depends on who you're talking to.
Jason Baum (03:41):
Yeah, definitely. I did a podcast. When ChatGPT first came out, everyone was all over that story. Everyone was playing with it, trying to figure out use cases, and it was hard. I think at first it was like, I don't know, I can ask it what I should do when I go on a trip to North Carolina. It was just these very basic things. I was interviewing a CEO and he was saying that there was this story about how a company was going to try having ChatGPT be its CEO for a week. And everyone was like, well, how'd it do? And he said it did great. They didn't lose any speed. And then that same CEO who I was interviewing lost his job a week later.
Nicky Pike (04:21):
Man.
Jason Baum (04:21):
It's an unfortunate incident. So I always say he was the first one to lose his job to AI, but I think it was because of what he said. But I don't know how true that whole thing of AI coming for your jobs is yet. Certain jobs maybe. I don't know if we want to get into all of that, but I do worry about junior developers and stunting growth, and I think there's a whole lot of issues that come out of it and benefits too.
Nicky Pike (04:49):
I imagine we probably will get into that a little bit later, but to me it's almost in this COVID pandemic scenario, when I look back at COVID, the people that I most felt sorry for were kind of like the high school kids. The ones that lost their high school, they lost their graduation because of this transition period. I think we're seeing the same thing today with AI. We've got a bunch of guys coming out and gals coming out with CS degrees that were in this interim time between when AI first started to where now it's becoming popular. They're unprepared, they don't know about AI. They're hearing all this stuff and buzz in their head about, Hey, your jobs are now worthless. Which by the way, I don't believe that's at all true. And yeah, we'll talk about that a little bit later, but there's a lot of fear there.
(05:31):
But now the universities are starting to pick up and say, okay, well, we got to start teaching some AI, but then we're hesitant because we don't want you cheating with AI. I feel bad. I think there are going to be some bodies left on the ground when it comes to people that are just coming out if they're not willing to adapt and learn. And when I ask people those questions, are you willing to adapt and learn? Well, we don't know. It's a big topic. Well, you're in computer science, man, that's your job almost, it's secondary education. Everything's always changing.
Jason Baum (05:59):
Well, think about the calculator. This is kind of what I tell some of my friends who are not in the space. When we were in school, there was a big deal about not having your calculator for tests, and the calculator was almost like a negative thing, but you still had to learn how to use it. It was really weird. And then it's like you're required to use a graphing calculator for certain classes. So it was very confusing. Is the calculator good? Is the calculator? Calculator is just a tool. Just because you use a calculator doesn't mean you don't know how to do math. In fact, you need to know how to do math in order to use a calculator properly. And I think it's the same thing with AI. It's like we actually need to really learn concepts a whole lot. And that's why I think senior developers are finding better use out of AI than some of the junior developers. They haven't gotten those tools yet to be able to use AI successfully. So I think it's something similar that's going on.
Nicky Pike (06:51):
I love that analogy because believe it or not, I keep in touch with my old sixth grade teacher. She's a very important person in my life, but I remember her telling me, well, you got to learn this stuff. You can't carry around a calculator in your pocket all the time. Well, first thing I did was, Hey, look at my phone. But you're absolutely right. If you don't know the basics, then you can't use a calculator appropriately and you're going to get wrong answers. When I look at AI today, it is a junior developer. It is an intern fresh out of college. It's got all the theory, but it doesn't have any experience. And when you've got junior developers trying to learn or utilize another junior developer, bad things can happen. So you've got to learn a new skillset almost about how to guide. You're still going to have to have the senior developers come in and help tell what right is and provide oversight. So that's a great analogy, Jason.
Jason Baum (07:35):
And I think it's evolving. We're at the beginning. The time that's going by is very short, yet it feels like the progress that is being made. I think for a little bit there in 2024 there, this "have we hit the ceiling?" conversation, there's not much progress, and then all of a sudden agentic AI is changing that conversation again and speeding things up and wow. Now AI is not just this five years ago, we're talking about ML, right? It's all machine learning, data input, data back and very, very rudimentary. And then I guess early days ChatGPT, now we have natural language prompting. We're still pulling from all the data in the world now, but from a certain set time. So that's going to be old data that we're talking to. And then they've updated that evolved, and now we're task driven. Now it's this agentic AI that can generate apps for us, write tests, create docs. That's a bigger cognitive load now, not just for the AI that's actually doing some of the work, but for us. As developers, you need to, this is a whole lot to manage in addition to your day job.
Nicky Pike (08:47):
Yep, absolutely agree. And I mean, it's a huge debate right now. We're watching this play out on LinkedIn. We're watching it play out at conferences. There's this whole idea that AI is either improving your quality or it's hindering your quality. There was a survey that was put out by METR that found that experienced developers are actually working 19% slower with AI tools, but then Qodo comes out and says, well, 78% are claiming productivity gains. I mean, what's the reality here? Are developers spending more time debugging AI generated code or are they getting faster? It's hard to tell.
Jason Baum (09:16):
And I think it's different from your run of the mill. Hey, I'm going to create this app for fun, versus I'm creating something in production for my enterprise company. I think it just comes down to risk. That's what testing's all about, right? It's risk mitigation because there's always going to be bugs released in production. We're never going to catch all the bugs. So it's like, what's your risk tolerance? What are we focusing on? And then what can we get away with versus what can we absolutely not get away with? And there's always that line, and that's fun about I think the testing field, but what drives probably developers crazy and their bosses crazy.
Nicky Pike (09:54):
Yep. I actually just had posted something on this on LinkedIn and it is that how are we viewing the market? We see a lot of negative people coming out and saying, Hey, vibe coding is horrible. It's going to be full of bugs and production issues. And I think there's two things that we're looking at. One is perception. It's almost like we're expecting more out of AI than we would another coworker or a junior developer. But then there's also this whole aspect of democratization and bringing it over to the hobbyist. I mean, I do this talk on how what we're seeing with vibe coding or democratization of coding, very equivalent to the DIY boom. You're not having people going out. Your citizen developers aren't going out there and creating Fortune 500 auth systems or handling payment systems. They're creating a workout tracker or something for their kids.
(10:37):
But when we're looking at LinkedIn and we're seeing this, I think a lot of the senior developers can't get outside the view of we're professionals and we write code for Fortune thousand companies and we have to be secure. They're not seeing, well, AI can help you, but vibe coding's not for you. It's for the citizens. Maybe the people in marketing and HR that don't have that technology, but they still want to get some stuff that's been sitting on their backlog for six months out and they can't get their development teams to work on. What are your opinions on that?
Jason Baum (11:03):
Yeah, definitely. I mean, it's for the side projects, it's for the dev rail team. I think that's a great way, something great for them because a lot of dev rails aren't necessarily writing code all day. Our job is to be out there talking about what we're seeing in the field and talking about the tools and not necessarily always coding and using apps, the app itself. So I think it keeps dev rails on their feet. I heard Angie Jones talk about this a little bit about what vibe coding is doing and the evolution of AI and agentic AI in general. It's changing the conversations for dev rails, from needing to educate developers to needing to educate everyone because everyone now is a developer, and maybe your Fortune 100 developer doesn't want to hear that, but that's the truth. That's what's happening. Anybody can create an app these days, which is I think, good and bad.
Nicky Pike (11:59):
Yeah, I mean, I agree with you. Now, I tend to think that words have meaning. So I say that AI is making everyone a builder. I do think there's a very big difference between somebody that can vibe code an app versus someone that's been trained and can actually polish an app. So I tend to use developer versus builder, but we're kidding ourselves. If we think that only developers or people with CS degrees are going to come up with great app ideas, I personally think we're in the greatest period of software development that we've ever seen of this boom. I mean, this vibe coding is giving people the ability, in my view, to I've got a great app, but I don't have the technical skills to build it. I'll go vibe code it, is it going to be secure? Probably not. Is it going to be performant? Probably not. But I've got this working prototype now that I can go get funding with, which maybe I wouldn't have been able to do before. I'd have to go find and have money out of my pocket, but now I can have this working prototype, get some funding, and with that funding, hire the developer to come in and make it better for me.
Jason Baum (12:52):
You know what? That's the best use case. I think you just came up with the ultimate one that's for your non-technical founder who has a great idea and now can demo it. Basically. I think it's going to change pitching forever, which is really interesting. I think now VCs are going to expect to see the app. Give me the prototype. Where is it rather than a presentation?
Nicky Pike (13:15):
Yeah, this is the new Wireframe, man. This is a working wireframe and kind of going back to the senior developers and them being a little bit more skeptical, the Stack Overflow study came out, and again, this is one of those paradoxes. It found that 84% of developers are using AI, but only 33% trust it and only 3% highly trust it. I mean, to me, that's like saying I use this parachute daily, but I don't trust that it's going to open for me when I jump out of the plane. So how do we reconcile this mass disconnect, do you think?
Jason Baum (13:41):
Oh, there's so much that I want to unpack about that survey, but the first is, who did it? So data is interesting and why companies choose to do surveys, and there's always, I try to look at it very objectively because there's no one who is completely neutral. I think data can be read many ways and for Stack Overflow, that's very interesting. I don't want to badmouth Stack Overflow. Stack Overflow's wonderful. I obviously use Stack Overflow. A lot of people use Stack Overflow, but who's hurting with AI? It's places like Stack Overflow. I don't think Reddit's going anywhere or Stack Overflow's going anywhere or Google are going anywhere. But AI is where you go now for your answers most of the time. So having a study that's going to paint a certain picture is interesting to me. That said, I think there's probably a lot of truth to the data itself, the 84% adoption and 33% trust. Can we talk about the word trust real quick?
Nicky Pike (14:41):
Yeah, and I think you're probably going to lead me into another point, but absolutely.
Jason Baum (14:45):
Yeah. Okay. So trust is an interesting word. When the GPS came out, this is going to make us sound old, but whatever, I don't care. I was a navigator in the car and I had the big map and I had to figure out which way we go. And usually we end up lost or my wife and I would get into arguments, and then we got our Garmin and I named it my wife's name. So when the Garmin would lead me the wrong way, I'm like, ah, we can get in a fight again. But seriously, but GPS, when it came out, I think there was a certain level of trust. We all of a sudden had to give this tool to lead us the right way, and now we're completely reliant on it. Nobody uses a map anymore. All the GPS on our phones are built in. And so I think it's sort of this same thing that we're experiencing a bit with AI. It's sort of like, okay, take the wheel, let's go. But then at the same time, once it has the results, we're like, well, this isn't right. This isn't right. But we still are comfortable enough to say, alright, let's see your draft. Let's see where we're going to go with it. What do you think, Nicky?
Nicky Pike (15:53):
I find it very interesting to me because we've seen this talking about trust, talking about the fact that this came from Stack Overflow and they're the ones bringing up trust. But I watched the senior developers, the guys that really know their craft, saying "There's absolutely no way that I would trust AI code. It doesn't know what it's doing." I submit to you, go find me a single developer that hasn't went on to Stack Overflow and use something that they weren't quite sure about because it said it worked. What's the difference here? And I guess that's kind of where I'm going to is are we setting these unreal expectations on AI? It's almost like we're believing it's going to come in as a sage and just fix all our problems, but I don't think we're looking at it realistically. AI is just another tool. As you said, it's got to be told what good is and the fact that we expect perfection of it. And we'll get into another study where we're talking about they find that AI generated code is like 41 to 48%. I've got it in my notes. I'll go look it up full of bugs. Okay, well, is that any different than a human developer? I've never met anyone that has written perfect code that's bug free that they never went in to change. So why are we expecting a different thing from AI here?
Jason Baum (17:01):
It's a wonderful point actually, the expectations, and that certainly impacts trust. But if I was to unfortunately paint a broad picture of developer practices, developers are pragmatic. So they're going to use a tool that reduces friction. And even if they don't fully trust it, if it reduces friction, they're going to try it and they're going to use it. And then there's the other thing is the hype around it. If you don't use it so you at least understand it, I mean, I don't know how you could possibly be a developer today and have never have used an AI tool. That is actually more interesting that only 80, what was it, 84% have used it. I'm surprised it's not a hundred percent, at least have used it.
Nicky Pike (17:47):
I mean, you'd have to see how the questions were framed because are they using it to actually generate code? Are they considering that using AI or did they post, Hey, I've got this problem into ChatGPT or Claude and say, fix this for me. That's still using an AI, but maybe that's not considered in. I'm with you. I'm going to say it's probably closer to 98, 99%.
Jason Baum (18:05):
I would think it would have to be at this point.
Nicky Pike (18:07):
Yep, I absolutely agree. So on that same topic when we're talking about this with senior developers, again, the Qodo's survey found that when we're looking at teams that are using AI, 81% of those teams are using AI for code review. They're seeing quality improvements versus 55 that just aren't, they're seeing no improvements, but developers are reporting that AI misses these critical context windows 60% of the time. So 81% are using it, but they feel that it's missing critical context, 65%. This is one of the things I think with AI that's very confusing to a lot of people is what makes sense. All of these studies are saying different things.
Jason Baum (18:45):
That's a great point too. Yeah, I don't think anybody knows what to think. That's the problem. I think there's truth to all of it, right? I think we've said it now a few times where we're going to use these tools. The tool is not perfect. There's the human. The human has to have the knowledge to go back, review the code, find the bugs that are inevitably going to be there. All of this is time. So did we reduce time or is it the same time?
Nicky Pike (19:14):
Right.
Jason Baum (19:15):
We just didn't write it, but now we have to do a much longer code review because perhaps it's not our code, or is it the same? I don't know. I think that maybe there's time savings, but it must be very small right now anyway.
Nicky Pike (19:29):
Well, again, it's that perception and that scope window. Where are we focusing? Because going back to that METR study, right, found that 19% of senior developers were actually slower using AI because they had to go back and generate the code. Now I remember reading that survey. The first thing I thought was, okay, this is true, I believe this, but this was narrowly scoped. It was looking at senior developers on known complex code bases, things that they already know really well. So of course it's going to slow 'em down, but they're just looking at code generation. What if we went beyond that? Because I do think that AI is helpful. I mean it could be helpful in things like testing, documentation is a huge one and even help kind of debugging and finding those patterns. What do you think the productivity boost is going to really bring if we look outside of just code generation?
Jason Baum (20:11):
I think that it absolutely saves times for many use cases now with writing, documentation. I mean, I know open source projects, I won't say by name, but just translating docs has been a big problem and is very expensive. And I know of a couple of projects who have used AI to translate their docs into several different languages. I think that's actually a great use case. And I think productivity there would be very high. So I think it really depends.
Nicky Pike (20:45):
You go into a conference and you ask a hundred developers, you walk into a customer, you ask their development base. How many of you like documentation, right? You're not going to see a single hand get raised, but it's one of those things.
Jason Baum (20:55):
Poor tech writers.
Nicky Pike (20:56):
Yeah, you're right. I mean, this is not something that people went to college and university for to learn how to document. But it is vitally important not only to understand what the developer wrote, but also for archeology purposes for new developers, when you rotate out, the thing that I find most interesting is like, okay, you don't want to use it for code generation, but use it to document your code because you can use AI to become living documentation. Even those developers that absolutely go in and they write documentation once the feature's out of their mind and they're working on something else, nothing comes back. They don't go back and re document or update that code generally. But what if AI is running in the background and every time you hit a submit and you do a PR, it comes in and it automatically looks through and recreates code for you? That's a huge savings. And from a developer standpoint, look at that, man. I mean, your manager wants you to document your code. Nobody's doing it. All of a sudden you look like a rockstar because your code's fully documented, but I didn't have to put any extra effort into it.
Jason Baum (21:52):
Yeah, definitely. I mean, we're seeing a lot of use cases I think outside of just code where AI has made huge improvements, so much so that we want to go back to that original conversation about it replacing humans. Unfortunately, I do think there are certain use cases where we're seeing people lose their jobs due to AI because the value is there already. I don't know if I would be one of those companies. I'm hearing of founders who are starting a company and only relying on AI like one senior developer and that's it. And everything is being run by AI. I don't think we're there yet. I think those founders, I think that's a company that's going to not make it. But yeah, certainly we're on our way.
Nicky Pike (22:37):
I think the first challenge you would hear, I actually agree with you, but the first challenge you would hear is have you heard the story of Base 44?
Jason Baum (22:44):
No.
Nicky Pike (22:45):
So Base 44 is one of those vibe coding platforms. It was created by, and I'm going to probably say his name wrong, Moore Show Low, and he sold this to Wix for $80 million and he was a sole founder. He used AI to help write it. It's a tool to use AI to help other people create code. And he sold this for $80 million and within six months he had a quarter of a million users. He was bringing in 3 million in ARR. But the one thing that people are going to say, well, look at that. He did it well, yeah, but he was also a highly intelligent, he really understood code and he was able to use AI. So yeah, there's going to be corner cases. I'm with you. I think AI is going to get there at some point as we keep training it. And these models keep getting better. But I mean, what's your take on that? Now everybody's thinking, well, I don't have to be technical. I can go out and create this 80 million application and get rich. I don't think it's there.
Jason Baum (23:35):
I don't think it's there today. Like we were saying before, I think it's going to help with the pitch to the VCs certainly. But I don't think we're at the point where now all of a sudden you have this $84 million company. I don't think it really works that way. And I think even in the evolution, I don't think it's going to work that way. I think it's going to make it easier for some people who might not have had a chance before to get in the game. But I don't think it's all of a sudden going to give everybody an 84 million company. Wouldn't that be one nice?
Nicky Pike (24:05):
No, but at that point, what does money really mean if I could just use AI?
Jason Baum (24:09):
I think it goes back to your expectations of AI. What are these expectations we're putting on it? And again, we're still in its infancy. That's the funny thing. It's only been here. This version of what we have of AI is new relatively.
Nicky Pike (24:21):
Yeah. I don't even know if we can say this is the tip of the iceberg man. I think it's just started snowing and the temperatures just drop below freezing when it comes to this stuff because we're seeing new use cases. There's this other rocket ship taking off that's in conjunction, which is MCP servers. And man, we're seeing a lot of great movement there and what it can do. And to me that's both encouraging, but it's also a little bit scary when we see, okay, we're doing MCP servers that can now control your computer. Well, okay, that brings to mind Terminator, right? Is anybody ever seen Terminator when you start doing that? But then I see other useful parts. A lot of people are using Granola and things like that to record meetings and MCP servers that can go in and let your agent summarize all your week. That's hugely helpful. So what's the balance do you see there between scary and extremely helpful?
Jason Baum (25:05):
I think MCPs can be extremely helpful. I've been on a team that's developed one and a very good one at that to work with our rest API and certain endpoints and that's it. And it was using your LLM, not an LLM. We were hosting your LLM.
(25:25):
I think that's actually a really great example of an MCP that can be secure limited, but still very practical and useful. It makes using the rest API incredibly easy so that you could get your data. What I am worried about with MCP in general, from a broad stance, I mean it is a security nightmare. We're talking unauthorized access data, leakage problems with supply chain attacks, manipulation risks. There's so many risks that it introduces that. I think companies at first were like, Ooh, MCP, we need to do that, go do that. And then probably the dev rail team or some other team was creating these MCPs and then all of a sudden we found out the risks. And I think companies are starting to pump the brakes a bit. We're not hearing about it as much. There certainly still is the hype, but I don't see a lot of companies just rapidly putting them out there they were a few months ago because all of a sudden, I think they were realizing the risks they were introducing into the market. And people are smart. They're going to take advantage of these things and it can be scary, I think, looking at this.
Nicky Pike (26:39):
There's that big story in the news where AI went out and deleted a production database because the tests were working. So instead of fixing the code or fixing the test that well, let's just change the database to help our test pass. And I think that's one of the dangers.
Jason Baum (26:52):
Hey, it did its job.
Nicky Pike (26:52):
Yeah, it did exactly what you asked it to do. And that's one of the dangers. And I think we're seeing that. So like Coder itself, we see that same thing. And we're coming up with a product called AI Bridge to kind of help be that gateway between agents and data. What's going to control the MCP service? How do we put security and governance around that? How can we track what MCP servers are doing? Because again, you've got to be very careful. Shadow IT has existed for years. We're seeing Shadow AI now. You got to be very careful about what your devs are out there doing to try to improve their day. And I don't think that devs are being malicious in this, and it's not just in the dev world, it's across the board. We're asking people to do more with less. We're asking them to be this famed 10X developer. What companies want is somebody that works. 10X is hard for the same amount of money. So how do we actually do that? Well, AI is providing that ability. Your thoughts on that?
Jason Baum (27:42):
And just in general, practical, non just everyday use. If you're using an MCP server, let's say I'm going to book a ticket for an airline. I'm going to talk about that North Carolina trip. Right now, I'm going to have it. I'm going to use an LLM, it's going to tell me what things I could do. Now I'm going to say great book the tickets.
(28:04):
Okay, it's going to come back and it's going to say it's going to need a credit card. So now are you going to put your credit card? Do you feel comfortable putting your credit card, your information, your TSA number, all these secure things. It's your passport if you need it. I mean, think about what you're now giving this MCP server that might not be secure, that can be manipulated. I'm a risk averse person on the project leadership of one of the biggest testing libraries that there is. I don't know if I would go take that risk. I think it's the same thing when we're using it for development.
Nicky Pike (28:46):
You brought up testing. I want to come back to that, but I mean, do you even know that you're not right now? Is United or Delta using an MC preserver? Are we giving them their details? I don't think we know. There's almost a sense of inherent trust or willful ignorance about what's going on with some of the companies we do. So all those same things that we worried about in the past with security on the internet, are they being amplified by AI? Because to your point now we've got agents, we've got things that are quote unquote kind of self thinking, well, if I give an AI an agent my credit card number and it decides that, well, I need to go do something that you asked me to do, I'm just going to go out and buy a bunch of stuff for you. I mean, it's not outside the realm of possibility of that happening.
Jason Baum (29:25):
No, it's absolutely something that could happen. I don't know if a lot of people are thinking about it that way because ooh, shiny new object, this is cool, want to play with it. Tons of hype and not really realizing the full risk. And like you said, the risk might already be introduced and we don't even know it, which there probably is that I like what you said, willful ignorance. I think that that is the best description for what it is.
Nicky Pike (29:49):
Yeah, I mean, it's no different than the people that said, well, I'll never have an Alexa in my house or anything like that. I don't want it spying on me enlisting. Well, again, I'll bring back the cell phone. Do you have a cell phone? It's happening already. And to pretend that it's not, you almost have to be a Luddite to get rid of that risk. But if you're that way, then you're missing out on all the potential and all of the enhancements that are coming to our lives because of this technology.
Jason Baum (30:12):
I mean, I've been in marketing for a long time. I've sat in the marketing org for a long time as a developer advocate. I'm all familiar with retargeting. And when that was first starting, I thought that was invasive. Gosh, now I'll just be sitting next to my wife. We'll just be scrolling, doing the doom scroll. And I might have said something out loud to somebody and now all of a sudden it's showing up in my phone in an ad, an Instagram or something. How does it know that? And I know how it does that, but gosh, we're really at this point where nothing is safe.
Nicky Pike (30:47):
Yeah, I mean, how creepy is that? Hey, me and you're sitting here talking. Hey, we talked about some equipment before we got on the show. I guarantee you that you'll probably start seeing some ads pop up on Amazon. Amazon or something like that. Yep. Well, you brought up testing. You brought up the largest testing company. So let's talk about testing specifically and what AI's role is in that. So a lot of developers and myself included, we're hearing a lot about AI's ability to do self-healing testing or AI driven test automation. I mean, is that something we should trust or is this really just kind of going out there and creating more false positives that are just going to need to be caught by humans?
Jason Baum (31:20):
I think when we're talking about self-healing, it sounds magical, but what it's usually doing is just making issues, more issues instead of fixing root causes. Sort of similar to the analogy that deleted the database. I think it's something similar to this. It's going to try to fix the issue by causing more issues. So I'm not really sure if self-healing is something I would use or when a company says that they do self-healing, I don't know if I necessarily trust it. I think AI driven test automation is definitely powerful for speed, but I think we're going to see a lot of false positives that might be real problems. The worst thing is when we're testing the wrong thing, when the real bug is going out, because we're spending all our time here and we should be spending our time here. And I think of a major outage that happened, I guess now over a year ago. I won't call the company out. They've had enough issues on it, but we all know who I'm talking about.
Nicky Pike (32:19):
If you were traveling that day, everybody knows exactly what you're talking about.
Jason Baum (32:22):
You saw blue screen, blue screen, everyone that saw that blue screen. Ironically I was at Dev Rail Con and the night before, I was talking about all this because I get a little preachy when it comes to quality and code because all you hear is get it out quick. Speed, speed, speed. But what you don't hear is speed, speed, speed. But make sure it's good. That's the missing component. And I wonder who's in charge of quality? I ask that question to people all the time, who's in charge of quality? And I have gotten about a million answers. The one I get the most that I absolutely honestly hate is everyone's responsible for quality. Can I say bullshit?
Nicky Pike (32:59):
Yeah.
Jason Baum (32:59):
I'm going to say bullshit.
Nicky Pike (33:00):
Bullshit. Call it out.
Jason Baum (33:03):
I think that is bullshit. That's just something we say, and we don't actually mean it. If your boss is like, get this out faster. We need to deploy more. You're going to focus on speed and quality is lost. So who is responsible for it? And I think with AI, I think that makes the problem 10 times worse.
Nicky Pike (33:22):
Alright, so let me throw a counter at you because I do agree with you. So my counter would be, again, it goes back to perception and what your expectations are. I do agree. If you bring in any testing tool, anything that does automation, you would be outside your mind to fully trust it to begin with, right? You've got to go through testing cycles. You got to go through configuration and tuning cycles. Why would we look at AI any differently? Do I want it going out and running on my test? Am I going to blindly accept it? No, but I do see AI's, their ability to look at patterns and find the corner cases, things that would take us days, weeks, or even months to go through with our human eyes and process. It can do those things very quickly. So I see the benefit there, but then again, I'm also the guy that's saying, Hey, use it, test it. You don't blindly trust it.
Jason Baum (34:08):
And I think that's where there is benefit to AI is analysis. Look, I used to work for a test vendor, one of the most popular test vendors for the largest companies in the world, Fortune 100s. That's the bread and butter. Hell, mostly Fortune 50, the biggest of the big. And then there's tons of risk there, lots of money. And money means risk because if those apps crash, you don't want to be going to Disney and have your app crash. You don't want the Disney app down. That would cause a lot of money to walk out the door. So companies like that rely on certain cloud vendors and they're going to want to get the right results. When they're looking at their tests, they're going to want to see that test data and focus on where are we seeing the biggest risk that we can mitigate with our tests that where are we finding the bugs? And that test data analysis is the best use case, I think, of AI.
(35:11):
A natural language. Let's talk to it. Let's get insights into what is actually happening. What I see companies going to these cloud vendors saying is, can you write my test for me? Can the AI create the test? That's the wrong use case because we don't want AI test cases. Now we have issues with the test cases, and that's not really a number one problem. The number one problem is the maintenance. Number one problem is figuring out what to do with all this stuff. What are the signals we're looking for? AI can help us get those signals and get them faster.
Nicky Pike (35:47):
So when you say we don't want AI tests, written tests, so this goes back to the code generation parts. Let me throw this at you because we look at Microsoft, Hey, 33% of all our code is AI generated. Okay? But there's a difference between AI generated versus AI created. AI generated means I'm telling it to do something and it's going out and it's typing the code for me, which I think is a great productivity. Why should I have to sit and manually type all this stuff out when I've got something that can do it a thousand times faster than I can and generate all this code? I see that same thing in test. And maybe that's what you're saying. I don't know. Are you seeing a difference between AI generated versus AI created when it comes to testing?
Jason Baum (36:26):
Yeah, a little bit for sure. And again, I think it goes back to the problem we're trying to solve. You asked the testers or people who are creating tests. The creation of the test isn't necessarily the problem, it's the maintenance of that test. And so we're solving the wrong problem by saying AI, write the test for me, instead of how can we make sure that this is the right test and manipulating the test and getting the results and finding those signals through the insights. That's where testers spend their most of their time and not necessarily in the creation authoring of the test itself.
Nicky Pike (37:01):
I'm going to throw this out there. For those that are listening, put in the comments. What are you guys thinking? Are you wanting to see something, help you find the corner cases and help write the test? Or are you looking for, as Jason says, which I kind of agree with here, are you wanting to something, Hey, we've written the test, we know what they do, but as things change, we want to help update those tests and kind of maintain 'em. I think that'd be an interesting topic in the comments. So get in there. Tell us.
Jason Baum (37:23):
I'd be curious to see who answers and how they answer. Because I think that also depends, right? I think you ask an STET, they might say what I said. You ask a developer who doesn't want to spend time authoring any tasks. Yeah, they're probably going to say, let AI do it.
Nicky Pike (37:39):
Well, that's the great thing about AI right now. You walk in and you ask five people a question about AI, you're going to get seven different answers, right?
Jason Baum (37:45):
A hundred percent.
Nicky Pike (37:46):
Yeah, absolutely. Well, and you bring up a good point on this is enterprises, when we're talking about AI in the enterprise, they're absolutely scared to death of AI breaking things. Again, going back to that story of deleting the production database. So we're seeing that 41% of new code is actually AI generated, and this is according to Fastly, but it all requires these substantial edits. So again, going back to the perception issue, there's this fear of AI driven bugs making into production. How do you think enterprises should approach that? They want to ensure quality, but they've got this FOMO of missing out on AI and the potential that it could bring to you in the testing pipeline.
Jason Baum (38:20):
From talking to companies, the one thing I would ask of them is what I said before is why? Why do you want to, what is the problem you are trying to solve? Because a lot of times I'll just hear "What's the AI plan? What are we doing for AI? How are we going to incorporate AI?" Okay, but why? Once you incorporate it, depending on what you're using it for, start small. Don't just give the keys to the car to the, you wouldn't let a 7-year-old drive. AI is sort of like a 7-year-old right now. It's not there yet. It's not fully developed. It's still learning. Don't hand your mission critical code paths to AI and focus, focus where the edits and oversights are naturally built in documentation, code reviews. I think the company needs to create governance. What AI is allowed, what it's not, where it's allowed, where it's not and truly define it because we're not getting any sort of oversight from anyone anytime soon from the government or anybody other than the company itself and how we're going to be using it right now. It's sort of like the Wild West.
Nicky Pike (39:25):
Yeah, I mean, I think you hit it right on the head, and I don't remember the exact set, but it's something like 80 or 90% of companies that go out and try to implement AI. Those projects fail. And I don't think it fails because of technical reasons. I think it fails because companies don't come in with a plan. Exactly what you said. What are the outcomes that we're looking from using AI? Is this to aggregate our data into a single place? Is it to improve developer productivity? Is it to catch corner cases that maybe we haven't thought of? Coder's CEO Rob Whitely says it's the airplane phenomenon. Somebody read it in a magazine, AI's doing it, Google's doing it, so therefore we must, and they try to bolt it onto their infrastructure and onto their processes. And I can guarantee you that's going to fail because AI is capable of a lot more than you think it is.
(40:11):
And if you go in and do that, you're going to see it break shit. Same thing with junior developers. You're not going to bring a new intern in or a new developer on his first day and say, here, go revamp our whole credit card system. So why would you do that with AI? Be pragmatic about it, be responsible. Come with a plan now. I think it'll grow. I mean, we're watching the rate of improvement that we're seeing in AI is phenomenal, but come into this with how are you going to put guardrails around AI? How are you going to restrict the things that it can access? How are you going to make sure that it's not making calls out to the internet or things of that nature? Guardrails and boundaries for AI are going to be extremely important, which means that you almost have to rethink your infrastructure. You've got to have an infrastructure that's built for this. Agree or disagree?
Jason Baum (40:58):
A hundred percent.
Nicky Pike (40:59):
And I come from a platform engineering background, so I always think engineering first or infrastructure first. And I do believe that there needs to be an infrastructure first approach to AI. Build things where you can guardrail it, build a moat around it, and love your point, start small. I mean, you're not rolling this out to the organization. I'm rolling this out to two or three developers and letting them go through and I'm going to iterate and find what I need to do. I'm going to sandbox this thing and slowly roll that out as we go forward. I mean, have you seen this in enterprises? Have you seen an enterprise that has kind of taken this approach and done it right? And you don't need to name names, but...
Jason Baum (41:34):
I'm trying to think off the top of my head. I have seen slow roll one feature type releases sort of, I'm guessing for them, sort of an MVP, if you will, of how they were going to use, I don't know from a practical standpoint, from an actual workflow standpoint.
Nicky Pike (41:50):
We've been talking about AI for years, but AI really didn't become a thing except for the last two or three years. So this is still new. Like I said, it's just started snowing and we're talking about icebergs already. But the other...
Jason Baum (42:01):
I think it became useful.
Nicky Pike (42:03):
I agree.
Jason Baum (42:03):
I think it was sort of for fun. And now all of a sudden the for fun is now actually practical.
Nicky Pike (42:09):
Yeah, that's a great way to put it. I'm going to steal that from you, right? I mean, you talk about, well, AI wasn't there three years ago, and you'll get people jumping on. It's like, no, we've been talking about AI for years. Well, it didn't become available. It didn't become useful until the last two or three years. And now we're just seeing this hockey stick of things coming out. It's the new.com boom. But actually, this is something that in my opinion, is going to absolutely change our lives. But there's a human aspect to this. There's a human aspect, especially when we start talking developers. You're being a dev rail guy. I mean, we're seeing AI's been a sidekick for a couple of years. You got Cursor, you got Windsurf, those type of things. But we're seeing AI of step up now as we're talking about to take over more tasks. We've got 46% of developers who actively distrust AI accuracy, but a majority, 51% are using it daily. What's the line between making things easier and the developers becoming overly reliant on AI?
Jason Baum (43:03):
Oh gosh, I worry about this just in general terms. I think about this with kids a lot actually who are using AI and we hear, are they going to learn properly? Are they going to be stunted?
(43:14):
And I think the same thing with regards to developers over-reliance is going to possibly lead to skill atrophy. I don't know if that's true. I don't know if it's riding a bike. I think to a degree it is. It's a language. If you don't use your language, you might forget it, right? I think I see this all the time. I'm a Duolingo person, so I was using Duolingo for one year for a language, and then I stopped using it and man, did it go fast. And it took so long to learn and then it's gone like that. And that's sort of like any routine that you have. And I think the same thing. If you're not using it, you lose it. And debugging is a muscle that needs exercise. I think there's a lot of things that I worry about with overreliance.
Nicky Pike (43:59):
And I do see that, I mean, I'm probably going to get in trouble with this, but there was a comedian back there that used to do this bit on drugs taking weed and how it slowed him down and he did this bit on cocaine and people, well, why do you want to use cocaine? Well, it enhances your personality. And he's like, well, yeah, but what if you're an asshole? I see the same thing with AI. I tell people, AI is there to amplify your experience. If you have no experience, it's going to amplify the fact that you have no experience and you're going to get yourself in trouble. And when we look at this, again, big fear AI is going to take all the developers. I absolutely see the exact opposite of that. I see it because AI is helping people go so much faster that you're going to have to have more oversight. I do see it changing the role of software engineering to be less typist, less syntax, more will engineers, writing the blueprints, writing the architect, and overseeing quality. But I think we're going to need more of that now. We can do things on a five, 10, a hundred X scale of what we were able to do just two or three years ago.
Jason Baum (45:03):
Absolutely. I'm a big proponent, but at the same time, what I said before is true too. I think it's going to be great, but I think we got to be careful.
Nicky Pike (45:12):
So as a dev rail guy, how are you going out there? How are you looking to help people kind of preserve that intuition, those problem solving skills that we see in developers in this AI world? You're worried about skill atrophy from a dev rail perspective and what you're doing. How do we prevent that?
Jason Baum (45:28):
Yeah, I mean, for my own team it was let's use our, what side projects are you working on? What applications are you working on? How are we fixing certain problems? What MCP are we creating? And keep them on their toes. So still working with AI, but also my team was never going to be like, I wasn't worried about skill atrophy. We're talking about lifelong testers. I don't worry about them as much as the ones who are coming in, like you said, the junior ones. I think people who are coming, you said like COVID such a great analogy where I'm worried about the people who are fresh, the people who have never done it before, the people who don't have that real world experience without AI. It's sort of like kids who never don't know the world without a phone or the internet, instant answers. There's a lot of skills that we might have that they will never have.
(46:29):
And I don't know if that's true or maybe that's just my own bias towards it, but I think we have to force people to learn the, it's always kind of like, well, why are we learning basic math? When am I going to need to know basic math? Well, you need to know it all the time and you also just need to know how to problem solve. That's really what we're teaching. So I think it's teaching, problem solving and then that general curiosity that will keep people focused on the tactical and tangible versus this tool that can solve all your problems.
Nicky Pike (47:05):
Well, and you brought up a great point as well about the whole kids in technology, and I wonder if we're going to see the same thing there. I can say, you look at my parents very hesitant, barely know how to use a laptop. I use it every day. I know you got kids basically coming out of the womb knowing how to use an iPad and get around things. I compare it to Kubernetes, right? I've been in this industry for 30 years. I built platforms. Kubernetes was a hard thing for me to pick up, but I see all these guys that I was working with 10 years, 20 years younger than me, that it's like, no, this is second knowledge, man. They're coming out ready for this. I wonder if we're going to see the same thing with AI. Yes, you are going to know things as an SD or a software developer that they may never know, but they're going to know how to manipulate and use AI in ways that really are kind of our mindset now.
Jason Baum (47:53):
That's such a good point. It's so interesting and I don't know the answer, I'm just now genuinely curious what it's going to lead to. What's so crazy about this, right? This is one of those things we won't know what the answer for maybe another 20 years.
Nicky Pike (48:08):
Yeah. Well, man, I don't know the way it's growing. It'd be wild to me if we don't know the answer for another 20 years. I think in five or seven years we're going to see dramatic. But that's just my prediction and I'm not good at predictions to be honest with you. But on keeping that skill atrophy, I think we're going to have to see software development teams basically borrow from and adopt things from other industries. A good example that comes to mind is like hospitals. They've got their morbidity and mortality boards. When something bad happens, I think we're going to see those kinds of deliberate practices have to come into the software team where to prevent that atrophy. Hey Jason, today you're going to explain this 500 lines of code, what it does, why it does it, and why it works or shouldn't work. And I think we're going to have to start seeing those types of practices come into development teams rather than everybody go to their laptop, everybody sit at their desktop and write code.
Jason Baum (48:57):
Yeah, a hundred percent code review.
Nicky Pike (48:59):
Well, and going back to your point about senior versus junior, I mean the data is showing that senior developers actually trust AI a little bit more because they have that ability to spot the mistakes that it makes faster when you've got these mixed experience levels. Going back to the MMB conferences that you see from hospitals making way in the software teams, do you have any ideas about how teams could actually balance that adoption without leaving the juniors behind? Because we can't ignore the juniors. We've got to do something for those guys because they're the next crop of senior engineers. How do we prepare them for what's coming?
Jason Baum (49:32):
Real world mentorship. You have the seniors mentor juniors while reviewing AI output. Teach them as you are reviewing, and then AI can teach the juniors how to debug and not just generate code so that there's this kind of constant flow of education that we could sort of help these juniors along as we're using it.
Nicky Pike (49:54):
Yep. My advice to the senior engineers out there has changed your tone a little bit. Quit being the grumpy old man, telling people to get off their lawn. Embrace the fact that this is happening. You're not stopping it. So be looking to pass your experience and your knowledge off to the juniors because here's what's going to happen if you badmouth it, if you don't take the time, the juniors are going to do it anyway and they're going to learn bad habits and they're going to learn bad mistakes, and it's going to give you the ability to say, well, I told you AI sucked. No, they used AI as a tool. What really sucked was you didn't take the time to mentor these guys. You didn't take the time to explain to them what good looks like so that they can follow in your footsteps. So that's my advice to the senior developers. It's happening whether you want it to or not. So I'm not saying embrace it and say it's the greatest thing since sliced bread, but prepare your juniors for it. They're going to need the help.
Jason Baum (50:43):
Yep, a hundred percent.
Nicky Pike (50:44):
All right, buddy. Well, we're coming to the end of this, so I like to end each one of these with a prediction. So what is your prediction for the adoption of AI and development over the next year? Are developers going to start trusting it more or will it continue to just be a tool that remains this oversight burden?
Jason Baum (51:02):
I think trust will increase, but incrementally, and I think AI will become like a default assistant in IDEs, but I don't think we're going to see the disappearance of oversight. I think oversight. It's not going to go away anytime soon, and I think the winners are going to be tools who are transparent and explain why they output what they did.
Nicky Pike (51:24):
To the best of their ability. I agree with you.
(51:26):
Yep. All right. Last one. Give me your spiciest take. What happens when the first major security breach comes that's directly attributed to AI code? Are we prepared for that?
Jason Baum (51:36):
I don't know if we're prepared for that. I don't think we are, to be honest. Not to scare everybody. I think it's actually very possible, very real. We kind of were talking about those risks earlier before that MCP itself introduces and a whole bunch of other things. I think it's going to force enterprises to adopt stricter AI governance. I think that's that we need to have that in order to get to stricter governance, which there is none of right now. It's very Wild West, as I was saying before, and I think it's kind of similar to Equifax reshaping security practices. I think it's going to be something similar.
Nicky Pike (52:13):
Yep. I agree. It's inevitable. It's going to happen. Again, I caution everybody to look at this when it happens because it's absolutely going to think about that perception problem. Look at it from a lens outside of professional because it's not any different than security things that we see with humans. So be prepared for it. What worries me is going to be the scale of the breach, especially when we start talking agentic and the things that can happen. If you're not putting the boundaries, you're not putting the guardrails around it. I think the potential for the blast radius of that breach being way larger than it would've been in the past, but we'll have to wait and see.
Jason Baum (52:51):
Yep. Yep. It's going to happen.
Nicky Pike (52:54):
So what do you think, Jason? Welcome to the Devolution. Are you a full fledged member now? Are you going to be one of the great men out there talking on the back end?
Jason Baum (53:02):
Absolutely. This was so much fun. Thank you so much for having me.
Nicky Pike (53:05):
Oh, it was great, man, and I hope you enjoyed it as much as I did. This was a great conversation. Any parting thoughts before we leave?
Jason Baum (53:11):
With great... What's the Spider-Man line?
Nicky Pike (53:15):
With great power comes great responsibility.
Jason Baum (53:16):
With great power comes great responsibility. Thank you. It would've been much better if I just had that, but the fumble is good too. That's what I like in this too. I think all have a lot of power now at our fingertips, and I think it really depends on how we use it, and I think we need to use it cautiously, and I think we need to use it for the right reasons, and I think a lot of great things can come from it.
Nicky Pike (53:40):
To do another great quote, trust, but verify. I'm always looking for something to help make me more productive, but don't blindly trust. Look for ways to put a leash on the AI beast when you need to, but don't be fearful of it either. It's going to change our lives. Well, Jason, I appreciate you being on the show and for all the people that are out there listening, make sure you hit like and subscribe so that you get notified when we're bringing up these next ones. And I'm sure that at some point we're going to be talking to you again, Jason.
Jason Baum (54:09):
Thanks so much for having me, Nicky. I would love to come back.
Nicky Pike (54:12):
All right, thanks. Thank you for listening to Devolution. If you've got something for us to decode, let me know. You can message me, Nicky Pike, on LinkedIn or join our Discord community and drop it there. And seriously, don't forget to subscribe. You do not want to miss what's next.