Lori MacVittie (00:03.406) Welcome back to Pop Goes the Stack, my fellow digital janitors, prompt whisperers, and anyone currently staring at a 4,000 line pull request titled simply Fix Stuff. I don't know, it passed the test. Ha ha. Yeah, today we're gonna talk about vibes, specifically vibe coding. Because who doesn't want to talk about that? And the question is, is vibe coding creating an unsustainable mountain of unmaintainable AI-generated code, or slop as the kids say, that will break in the future? Are they reasoning or just regurgitating bad Stack Overflow answers? And the even bigger question, how much tech debt are they actually generating? So Joel is here as usual. Hi, Joel. Joel Moses (00:50.977) Hello, Lori. Thank you for agreeing to service my PR request you mentioned. Lori MacVittie (00:55.222) Yeah, I'll commit that later, somewhere. Joel's gonna break down some of what the studies are showing. And we've got Narayan here to talk about these benchmarks and the new kind of tech that Narayan Veeramani (01:10.013) Hi everyone. Lori MacVittie AI is producing. Joel Moses (01:12.595) That's awesome. Yeah, this is a fantastic topic. I really love this one. Lori MacVittie (01:17.1) I love it. I love it. Joel Moses So I mean in the end analysis, we have all these AI systems to help us be better coders. That's the promise of them. We have these systems that also promise to give time back to us so we can do interesting things and we can increase the number of things that we do with these systems. Joel Moses (01:36.463) And you know they're generating fairly high quality in a lot of studies, but the question remains, are they actually helping us solve our technical debt problem? Or are they simply moving it along and creating new methods of technical debt that we're gonna have to address in the future? So I'm gonna put the question to you, Narayan, what do you think? Do the systems that we use today, the AI coding systems, are they hurting us or are they helping us? Narayan Veeramani (02:06.729) Thank you Joel for having me first of all. So yeah, I think in my opinion, like AI as Lori pointed out, AI is generating code incredibly fast. But unfortunately we human beings, we still don't have that, you know, the fancy neural implant in our heads, which lets us process code at machine speed. You know, the example I can think of is, know, someone passes over a like a boring 500 page tax document and tells us like, Hey, can you go and review this document for me? And that's the situation that I think programmers are facing today with vibe code. I mean, there is this new sort of tech debt, which you mentioned, Joel, which I think we are calling it as comprehension debt, where it's generating this massive code, which functionally it seems to work okay till there is a production problem. But no one understands how it works. And that's the challenge we have, right? And that's where I think the burden on the developer, from the coding perspective, has gone down. But all it has done is put more pressure on post-production maintenance and support. And this is what I believe is like causing that workplace friction for senior developers. Joel Moses (03:28.459) Interesting. So let's dig into some of what that means. So you know, SWE-bench verified is one of the common benchmarks that's used to determine the quality of the output from these coding systems. And that has actually skyrocketed from roughly sixty percent accuracy kind of the beginning of last year to nearly one hundred percent accuracy. Now I'll leave behind whether you can actually verify something as 100% accurate. I don't think perfection exists in coding, whether it's AI generated. I think Lori MacVittie Oh, uh Joel Moses there's always gonna be something that's flawed about a system, but Lori MacVittie (03:59.635) My code was always perfect. You take that back. Joel Moses Oh, okay. Lori MacVittie Perfect. Narayan Veeramani (04:05.225) Hehehehehe. Joel Moses I'll take that Lori MacVittie Nothing wrong with it. Joel Moses from you, Lori. Lori MacVittie That's right. Joel Moses (04:08.993) But again, one of the studies that we've read suggests that that is true, that the code quality that's output is consistently good until the system has some reason that it relies on memory rather than reasoning. And that can be simply that the state isn't cleared from prior sessions, and so it begins to rely, the models begin to rely on things that are buried inside of, you know, their training sets. And then the code quality begins to creep back down again. So there's that. And then there's of course the other thing, which is that these elements typically do really good at designing tests for unit tests, but designing like architectural tests, which is I think what you were talking about, that they are less good at. Can you give me some stories from your time working with these tools? Narayan Veeramani (05:09.353) Yes, Joel, I think what you raise is valid. I think these are pretty good in, you know, in writing code. But I think in addition to everything you said, I think there is also another root cause to this issue, I believe, which is I think it comes back to not providing adequate context as part of the prompt. What I mean by that is like, you know, with vibe code, what happens is the user or the vibe code user use some ambiguous prompt. Let me give you a concrete example here, right? Say if you have a simple example like, Hey, you know, Claude Code, write me code to apply promotion on Thanksgiving. Now this prompt itself, it doesn't provide any context that the US and Canada have different Thanksgiving days, right? So the agent in this case is just going to make a wrong assumption that, Hey, you know, you don't have any global customers. It's just going to assume US Thanksgiving, generate the code, and as well as the test is going to be generated based on that wrong assumption. And then the test shows up as green. Again, AI generated test. The test is fine, Lori MacVittie (06:22.091) Ha ha ha. Narayan Veeramani but it's based on the wrong assumption, in my opinion. Joel Moses (06:24.535) Right. Narayan Veeramani And that's where you end up with the wrong result, where the Canadian customer ends up getting the promotion on US Thanksgiving Day, right, which is incorrect. Again, that's the logical bug I'm referring to. Narayan Veeramani (06:35.859) Which is a little bit different than I think the point you made is valid. But I think to me the fundament of the crux of the problem is vibe code doesn't mean taking a shortcut as far as engineering rigor goes, you still have to make sure you provide the context as part of the requirements, as part of the prompt so that the tool is not making the wrong assumptions, both when generating the code as well as generating the test. And by the way, I'm not making up this example. I literally tried this on Claude Code and the agent actually defaulted to US Thanksgiving. And the funny part is I asked Claude later on like, Hey, how did you even generate the test? It just said like it used the code to actually generate the test. And again, which is a problem, right? You don't want the Lori MacVittie Ha ha ha ha. Joel Moses (07:26.913) Right. Narayan Veeramani code to be the basis for generating tests, Joel Moses Yeah, you want the spec Narayan Veeramani which we all, right, Joel Moses to be the basis, right? Lori MacVittie Ha ha ha ha. Narayan Veeramani Exactly the spec to be the valid issue, right? So yeah. Lori MacVittie (07:35.766) Ha ha ha, I wrote the code and then I tested it based on what the code did. So Narayan Veeramani (07:38.415) Exactly, right. Lori MacVittie of course the code is gonna be right. Joel Moses (07:40.023) Right. Narayan Veeramani Exactly. Lori MacVittie I mean, that's like Joel Moses Yeah. Lori MacVittie like how does how does that even work? And you kind of inserted a word in that in the tech debt, the comprehension debt. And I think Narayan Veeramani Yeah. Lori MacVittie that is a very important point. It's not just the volume, it's also, right, just some of the code. People are, Joel Moses Yeah. Lori MacVittie (08:01.594) they have trouble following it. And I mean in most enterprises there are standards, right? Narayan Veeramani (08:05.385) Mm-hmm. Lori MacVitti You can't use even certain constructs. The reason being that newer developers would not understand them. Because some languages like Narayan Veeramani Exactly. Lori MacVittie (08:14.738) if you've written in C, you know what I'm talking about, right? You cannot win the obfuscated C contest in an enterprise, that, it's not good. So there's standards around that that help readability, but they're still following that logic through. Joel Moses (08:28.983) Yeah. Lori MacVittie How do you do that? Do we have to walk through generated code to make sure we understand how it works? Like actually step through, debug, or read through it? Or why is the debt there? Right? Joel Moses (08:40.983) Yeah. Lori MacVittie Is no one looking at this code? No reviews? What? What's going on? Narayan Veeramani (08:46.961) Yeah, I think the, Lori MacVittie (08:47.596) No one knows. That's it. Narayan Veeramani Yeah, exactly. Joel Moses (08:48.899) No one knows. Lori MacVittie Yeah. Narayan Veeramani At the end of the day, I think like to me, like the AI is, it's pretty smart, right? The frontier models are pretty Lori MacVittie (08:56.588) Yeah. Mm-hmm. Narayan Veeramani good in both generating code as well as generating test cases. But to me at the end of the day, it's simply not like a Jedi mind reader. It cannot really figure out magically on, hey, you know what the missing context is. Like if you're a 30 year old company and then you have your own enterprise standards, your own legacy systems, Narayan Veeramani (09:16.453) your own single sign on, your own logging observability standards. It doesn't know what you have behind the scenes. If you don't give it instructions, it's going to make assumptions and it's going to make wrong assumptions about the code as well as the test cases it generates. And that's what happens. And that creeps into as a production issue. Joel Moses Yeah. Narayan Veeramani And that's where, I believe, where you end up a engineer being pulled in at 3 a.m. in the morning to debug an issue. And yeah, I mean, guess what? There is not even debugging or tracing available in the code for them to be able to triage. Lori MacVittie (09:50.56) Right. Joel Moses (09:50.914) Yeah. Lori MacVittie Right. Narayan Veeramani And that's where the developer ends up wasting time cycles, I think. Joel Moses (09:54.794) Yeah. You know, if Lori MacVittie (09:56.962) Wasn't Joel Moses we're sharing stories, I've got a story to tell you Lori MacVittie Ah, okay. Joel Moses and I think you're right. Like missing context, that's a bad thing. But I think that these systems also don't necessarily code architecturally. Sometimes they code efficiently, but not Narayan Veeramani (10:09.598) Right. Joel Moses necessarily knowing what the impact to the architecture surrounding it is. And I'll give you an example. I saw an unreviewed vibe coding process that was gonna do an asynchronous Joel Moses (10:22.835) management of a database. So it used async calls to call out to a database. The coder was wanting to fetch a nested list of profiles and their corresponding order histories. And the AI produced some rather beautiful code. It was grammatically flawless, it was highly readable. It used utilizes sequential asynchronous awaits inside of a tight loop, you know, very efficient. And in isolation the code looks great. It passes the standard mock testing, but in production it hammers the enterprise database with thousands of sequential requests when what it really needed to do was a single clean batch join, right? Just something Lori MacVittie (11:01.378) Mm-hmm. Joel Moses architecturally kind to the database. But no, what it did is it generated the most efficient mechanism for creating lots of sequential queries, which was incorrect. The prompt was right, but the architectural sensitivity wasn't contained in the prompt. Lori MacVittie (11:18.84) Well, but you know how many, I mean humans have done this, right? Joel Moses (11:22.558) Oh, yes. Lori MacVittie I mean, I've seen some horribly written, right, database queries. I mean, Narayan Veeramani Mm-hmm. Lori MacVittie that is a different art. There's a reason that database administrators, right the people who know databases, are separate. Right? They keep put them in little rooms, they feed 'em special food because they take care of the databases Joel Moses Ha ha ha. Lori MacVittie and they understand how to make-. They actually, they do, don't they? No? Joel Moses (11:45.008) Ahh, DB admin chow is not bad stuff. Lori MacVittie (11:46.486) I was, Joel Moses It's pretty good. Lori MacVittie (11:46.486) Okay, Narayan Veeramani (11:47.049) Ha ha. Lori MacVittie yeah! Right, but they understand how to do exactly what you're saying, which is "this is the most efficient way to do a query," which is different from "this is the most efficient way to code it," right? And so if you're not aware of those, I guess, which is your point, right, architecturally that Hey, you're talking to a database. You need to be efficient in your query, not just in your code, Joel Moses (12:10.891) Yeah. Lori MacVittie you know, but Joel Moses (12:10.891) The AI system has no idea how busy the database actually is. So it'll write something that it knows will do the job. It just doesn't fit in with all the other things the DB is doing. Lori MacVittie (12:23.018) Even if Narayan Veeramani (12:23.312) Right. Lori MacVittie it's busy not busy, you shouldn't be hammering it like that. That's an inefficient I mean, Joel Moses (12:27.779) Granted. Lori MacVittie come on, this is SQL 101. Like, you know... Joel Moses (12:27.779) Granted. Narayan Veeramani (12:31.177) Yeah. So Joel Moses So let, yeah, Narayan, please. Narayan Veeramani yeah, Joel, like the question is, I mean you know, just to kind of deep dive into that issue that you're talking about, is that, would you say like it's more to do with not specifying clear instructions or architectural boundaries and constraints to the LLM? Because I think the LLM does know how to write efficient queries. I mean, it's just a question of pushing it or motivating it or giving it instructions on or nudging it in that direction, right? Joel Moses (13:04.769) Yeah. Narayan Veeramani I mean, yeah. Joel Moses Well, I mean, it also is highly dependent on the model, whether the model is key to generate efficient loops or whether it's key to generate you know, kind calls to external systems. Lori MacVittie (13:16.108) Ha ha ha. Narayan Veeramani (13:17.063) Right. Yeah. Joel Moses You know, let me cover another thing, and again, this is another piece of that tech debt and developer experience that we were promised so much, and I wanna see if you think that AI is delivering on the promise. Joel Moses (13:31.14) These AI systems are supposed to save developers a ton of time. Narayan Veeramani (13:35.913) Mm-hmm. Joel Moses (13:36.187) And there's a BairesDev survey called Dev Barometer that looks at what AI is doing to developers over a period of years. And last year, about 12% of developers said that that they write code with AI. This year it's upwards of 42%. Now, when you look at what those developers who are using AI actually do, 80% of developers now spend less than half their time writing new code. But they also report that they didn't really get anything back. What they're doing with the time that they got back is reviewing the code that was generated. So instead of being able to do a lot more with these dev systems, what they're doing is actually spending the time reviewing it in order to productionize it. So what do you think? Is that, does that fit with what you know? Narayan Veeramani (14:29.585) Absolutely, and that goes back to what we started this discussion with on regarding comprehension data, right? Where you, and I think like, I mean, Lori and Joel, you had touched on this in one of your prior episodes on the need to have these iterative, you know, reviews, Narayan Veeramani (14:47.139) not wait till you end up with like, you know, 50,000 lines of code and then hand it to a developer and say, Hey, you know, go and review this. Right? That's one aspect of it where you need to have those, you know, whether it be through hooks or whatever it is, mechanisms, to continually review the code as it gets generated rather than kind of like this big bang approach. And then the second part of it is do use, like use AI, to do the grunt work, whether it be the, you know, the vulnerability scans or even the upfront code reviews, all of that needs to happen. And again, going back to what you discussed earlier, moving some of this upfront or shifting to the left rather than waiting to the, you know, to the end and putting all the pressure on the developer at the end. Joel Moses (15:37.411) Mm-hmm. Narayan Veeramani (15:38.269) And by the way, like at least the review part is okay. I think the worst part to this, Joel, is this code goes into production without a proper engineering review. And then the coder, the engineer gets a call at 3 a.m. to debug. And the worst part is the guy who vibe coded it has no idea of how the code was built either. That's because they don't know what the architecture was, what the components are, how it's all talking to each other. Joel Moses (15:48.748) Right. Narayan Veeramani (16:07.261) I think that's the worst nightmare, like even than the review, right like being pulled in the last minute. And that's where I think like, you know, I think that's where I think moving some of this engineering rigor upfront in the development life cycle would be super helpful instead of putting all the pressure on the developer later on to go and like, Hey, you know, here you go, inherit like this massive code base go figure. Right? Joel Moses (16:37.987) Mm-hmm. Narayan Veeramani And that's the situation we are dealing with now. Yes, AI is helping with reducing the burden on coding, but it's just shifting it to the review process and the debugging phase. Joel Moses (16:51.969) Right. Lori MacVittie (16:52.35) Yeah, and it's, I think it's important. We're talking specifically about engineering, about software engineering, about developers, but you've also got all of these other people, like all of the operations folks are generating scripts for automation, they're generating code for agents. Who's reviewing that? Like that's Narayan Veeramani (17:12.281) Exact. Lori MacVittie there's no Joel Moses (17:13.517) Mm-hmm. Lori MacVittie review in that. Narayan Veeramani Right. Lori MacVittie (17:14.386) They have to start, everyone needs to start acting like engineers if you're going to be using AI to generate something and move that review as it's gotta be part of your you can't actually say go until it's been reviewed by a human being. Narayan Veeramani (17:29.413) Exactly, exactly. I think the, and that goes back to that whole point, right? At the end of the day, you cannot just have the AI to review the code because I mean, if there is an audit, you cannot have the Lori MacVittie (17:38.264) Course it's good. Narayan Veeramani agent to go and put the agent in front of the auditor, right? You do need a human to actually take the ownership and the accountability. Lori MacVittie (17:49.506) Yeah, well that's why they don't, I mean, you're not supposed to review your own code. No one ever Narayan Veeramani (17:53.767) Yeah. Ha ha. Lori MacVittie said, Hey, review your own code. Tell me it's okay to put-, no you don't do that. Someone else has to Narayan Veeramani (17:58.471) Yeah, Lori MacVittie do it. Ha ha ha. Narayan Veeramani Agree. Yeah. Joel Moses (17:59.981) Yeah. Lori MacVittie We're running up against time 'cause we always do when we talk about these great topics, but we do want to leave listeners with some kind of takeaway, some kind of learning, some kind of key point. So, you know, Joel, why don't you kick us off? What should people be taking away from this? Joel Moses (18:18.839) Well, you know, I think my takeaway from this is AI can draft code at a scale the world has never seen, but it can also produce tech debt almost as fast. Especially if you don't do what the developer should do, which is review the code and understand the code and figure out exactly how to operationalize and maintain the code later on. It doesn't obviate the need to do that. The other thing that I'm taking away from this is, you know, one of the drudgery parts of developer life is developing tests and that AI, left to its own devices, tends to develop tests based on the code that it generates, which is probably not a great practice. So one of the things that a developer might want to do is to whenever they're writing their prompt to generate the code, include a link to the spec and ask it to generate a test based on the spec rather than the code. I think that that's a good way to jumpstart some of these issues. But in the end analysis, you know, it's one of those cases where the developer should know you break it, you bought it. You generate it, you vibe code it, you maintain it. Lori MacVittie (19:32.067) Ahhh. Narayan, what about you? What do you want people to take away? Narayan Veeramani (19:34.065) Yeah. Wholly agree with everything Joel said. I want to hone in on the generating the testing from the requirements and not from the code. Definitely, you know, we want to have that separation of concerns. It's kind of like you don't want the student grading their own homework and getting an A plus, like giving themselves an A plus grade. And then I think the second thing I would say is moving some of the engineering rigor upfront in the development lifecycle. And then thirdly, I think Lori you hit on something interesting, which made me think, not every vibe coder has engineering proficiency. Right? So, because there are many citizen developers who are actually building, I mean, vibe coding. So we've got to bake that rigor into the development experience itself with the right abstraction and the tooling and kind of give them a guided skill and let them describe what they are trying to build. And the skill brings in the right authorization, logging, security, integration patterns so that the vibe coder doesn't need to know all the complexity. End of the day, one final line I would say is stop using the senior engineers as the cleanup crew Lori MacVittie/Joel Moses Ha ha ha ha. Narayan Veeramani and let the AI do the mechanical work. Let the senior engineers do high value tasks, right. Yeah. Joel Moses (20:54.174) Spoken like a true senior engineer. Narayan Veeramani Yes. Ha ha ha. Lori MacVittie (20:54.156) Yeah. I was gonna say, gee, I wonder what Narayan's role is. But agree, right? Reviews are important and we have to keep recognizing that as people are, vibe coding kind of implies non-engineers too. So we need to develop some kind of systems and I like the idea of embedding it right in the experience Narayan Veeramani (21:18.991) Right. Lori MacVittie (21:19.138) for them to make sure that, you know, there's at least a modicum of like real testing and decent code being generated. But at the end of the day, we're gonna have tech debt because, well, it's tech debt. Joel Moses (21:34.807) Fact of life. Lori MacVittie You can't get away from it, but let's try and reduce it. Everybody chip in, Narayan Veeramani (21:39.463) Agree. Lori MacVittie do your part. Let's try and reduce it. well until next time I'm gonna have to say that is a wrap for Pop Goes the Stack. Lori MacVittie (21:48.024) Thanks for listening or watching. Please subscribe, so when your dashboard goes green, you'll at least have company.