Subscribe
Share
Share
Embed
Learning to apply solid engineering principles to AI-assisted software development.
Hello, everyone. Thank you for joining. My name is Jonathan Hall and just a really brief intro into what this is. I'm a software developer. I have been for many, many years, and AI is all the rage these days, and I have been intentionally kind of stick...
Jonathan Hall:Staying away from that until fairly recently. I've seen a lot of the hype about, oh, Claude's gonna take over the world, and 87% of code at Microsoft was written by AI or all the... All these silly things. And I've been kind of a skeptic. Not not that I think it's a bad idea, but, like, I didn't wanna ride the hype train in and I didn't wanna be the one on the bleeding edge doing all the tinkering.
Jonathan Hall:I'd rather let someone else do the tinkering and then I can learn from their mistakes. And that's kind of why I'm here. I have Paul Hammond joining me. Hi. Hi, Paul.
Paul Hammond:Hey. Hey. Glad to Thanks be
Jonathan Hall:for thanks for joining. I am hoping to learn from some of your mistakes today. Yeah. Yeah. The goal of this recording is in the second is basically for me to learn, get some ideas from you.
Jonathan Hall:I have been using Claude code for a couple months maybe, fairly heavily. I I use it for a lot of stuff, but I still find a lot of rough edges. I'm hoping you can help me and and whoever's listening or watching come away from this with some some tangible things we can do better. So before we dive into all that, Paul, why don't you tell us who you are?
Paul Hammond:Sure. Yeah. Thanks for inviting me. And, yeah, I've been in the industry now for about twenty years, there or thereabouts. And for most of that time now for at least a good, I don't know, maybe thirteen years or so, I've been doing what you would probably you'd recognise as kind of XP practices, test driven development, pair programming, all of that kind of good stuff, continuous integration and all that stuff on a pretty regular basis.
Paul Hammond:So I kind of came at AI from a perspective of somebody who is used to working in a way where I have like really high confidence with my tests and a really good kind of relationship with them. And yeah, my initial kind of experiences with AI were not amazing, I think it's fair to say. It's funny you mentioned tinkering because I think that's exactly what I did. I I tend to be an early adopter of things. I'm I like shiny things.
Paul Hammond:What can I say, you know? And yeah, and I found when I when I first started, I mean, AI has come a long, long way in the last couple of years. Right. And then I would say probably about a year or so ago, AI started becoming more as I think it has for a lot of developers, maybe maybe it began as, you know, it's slightly impressive auto complete, you know, that's that's kind of how it began. So I'd be in, like, Versus Code at the time.
Paul Hammond:And it was suddenly, oh, this is a bit better than auto complete. And it's give me suggestions that sometimes are right. But most of the time, they were not not great, actually, you know, suggestions. And sometimes it was just kind of annoying and it would just get in my way and like, you know, it's it's not what I want. And then over time, I found myself moving towards Cursor.
Paul Hammond:And initially, again, I had some early kind of good impressions of Cursor. But again, you know, I soon hit issues. I soon found that I was losing control of the confidence that I'm used to. Right. So I would expect a lot of your kind of listeners are probably, you know, people who understand things like Test Driven Development.
Paul Hammond:But for maybe people who are not as familiar, maybe I could just describe a little bit about... Again, I don't I don't know how much detail to go into, but for people who may be not familiar with these techniques, so kind of stepping back from AI for a moment. You know, with TDD, you kind of invert the usual kind of the process on its head. So a lot of people would think if I'm writing automated tests, I'm going to write a feature, going to write code for it. And then to prove that the test work sorry, that the feature works, I'll write a test.
Paul Hammond:Right? And that seems kind of logical. And for myself, I, you know, before I did TDD, seemed perfectly logical to me as well. Yeah. But Test Driven Development flips that the other way around.
Paul Hammond:And so you begin with, you know, a failing test. The test is always going to fail at the beginning. You then make the test go green, and then you see if there's any opportunities for refactoring as you go. And another thing that I do and just something to kinda mention that's kinda relevant to how I'm using Git AI now. When I first started doing TDD, I watched a video called TDD Where Did It All Go Wrong by Ian Cooper.
Paul Hammond:Are you familiar with that that video?
Jonathan Hall:Been a long time, but I've seen it. Yeah.
Paul Hammond:Yeah. And in that video, he he basically explains that where where a lot of people make mistakes is they get obsessed with this idea of a unit. What is a unit? And they think that because I want to test my unit in isolation, my unit is code, it's a function or it's a class or a method. And they think, well, because my unit is code and I want to test that in isolation, I have to mock out all collaborators to that unit.
Paul Hammond:Right? And you end up in this position where you have tightly... Tests are tightly coupled to implementation details, and it slows you down instead instead of speeding you up and kinda gets in your way. So he recommended not doing that and instead make your test based on behavior instead. The idea being that you only mock out where you need to to to to prove a given behavior.
Paul Hammond:So all your tests have focused on business behaviors instead and and things that if they failed, the business would recognise why that's a bad thing. That's the way I look at it, you know?
Jonathan Hall:And
Paul Hammond:so I... Yeah, when I first started... Oh, and just to kind of explain to people why going test first kind of makes a big difference. Because you go through that cycle, and the trigger for kind of new implementation code is a failing test. Over time you you get into this really...
Paul Hammond:It's like a rapid feedback loop that you get into. And the way that I would typically code, again, forgetting AI for a moment, because it informs the way I use AI now. But the way I would do it is I would have my IDE on a big... Well, in one screen. And then I'd have another screen or a big widescreen.
Paul Hammond:And on the other screen, have my test running in watch mode. And the experience that I would have is if I... Like, I I create bugs all the time when I'm writing code, but I spot them instantly. That's the that's the key because I've got my test running in real time. And because I'm always following this process, you end up with this suite of really amazing kind of tests that give you this confidence to make changes over time.
Paul Hammond:And when you make a mistake, you find out usually a second or two later, and that's such a powerful thing. And then, yeah, and I... Yeah, I've been using these techniques for a long time, came to AI, and basically missed that. You know, when I first started using Claude code, I was having this buzz around Claude code. And I started using it, and it was like, well, it does too much.
Paul Hammond:Right? So I would ask it to do something, and then it'd run off and go and do five other things. Didn't ask it. Yeah. Have you seen this behavior?
Jonathan Hall:Oh, yeah. For sure. All the time.
Paul Hammond:Yeah. And so that that was one thing that I had a problem with. And then very early on, I would find, okay, it's the way I describe it is like an interesting parlor trick. Like it's it's it was fascinating. And it was like really kind of in some ways very impressive, but not not genuinely useful is the way I'd see it.
Paul Hammond:You know, goes off and changes 50 files. And you're like, okay, the thing that I asked you to do seems to be working. But I don't know if you'd broken three other things when you did that now. And so I guess what what I started doing was experimenting a bit in what little spare time I have to young kids, but in what little spare time I have, I would just try stuff. And so I thought, well, okay, first thing for me is that I need to have I need to regain this confidence that I'm used to.
Paul Hammond:So I started in Claude, you can you can create skills, right? So I'm sure you're familiar with skills in in Claude. Yeah. And and I created... I'll I'll send you the link now, but I know obviously for...
Paul Hammond:If if you if people Google, if you Google city Paul, that's my name on GitHub and dot files, you you can find my dot files. Oh, is there a chat here? Don't know if I can send you this. Is there a chat? I don't see it.
Paul Hammond:Oh, there is. Sorry, I see it. Yeah. I'll send you that so you can see it. But then what I started doing, so I have my dot files repository, and I started adding some of these kind of rules, these skills to my dot files.
Paul Hammond:And one of the very first ones was a test test first kind of skill TDD, you know. And quite quickly, I started getting... I started seeing that it changed the behavior of Claude, and it was suddenly working in a way where, it was... So what I found was the the TDD approach with Claude is slightly different to when a human does it. Right?
Paul Hammond:So when we do it, it's always one test at a time. Right? Mhmm. So, you know, it's one failing test at a time. One thing...
Paul Hammond:I I initially forced Claude to work that way, but I actually found that, and this is just through experimentation, tinkering, as you said before, and I found that, making it go test first was more important than having the single test at a time, which might sound might sound weird at first because it's not true TDD as a human would do it. But what I found was that I think there were three kind of big advantages to TDD. Well, a test first approach, shall we say, with AI. So one is that it constricts the scope of the work that it's it's going to do. So it it stops from running off and doing a million things at once.
Paul Hammond:You give it the scope. Your job is to build this feature test first, right, and to to go in this way. The second thing that it does is it gives it a way to instantly validate validate some work because it itself sees it go from red to green. And that gives gives you some kind of confidence. And then I guess the third thing is that it leaves you with that regression suite, which again, if you're focusing the tests on behavior, which again is another thing that I make it do, seems to work really well.
Paul Hammond:Oh, and I guess one other thing is it gives you a natural kind of, validation point to get the AI to basically say, okay, now I'm going to check what you've just done. Mhmm. I just realized, think I talked for about ten minutes there without even...
Jonathan Hall:No, that's perfect. Yeah. Interesting. So I've gone through a very similar journey and I guess before I sort of compare and contrast what I've done, I'm curious. How happy are you with what you've come up with?
Paul Hammond:Pretty happy, actually, I would say. It's not perfect at the moment. It can still... One of the things that I found is that sometimes Claude will just forget to follow your rules, which is a bit annoying. It seems to pretty much pretty consistently go test first now, but it will sometimes forget some of the more subtle rules, and sometimes I have to remind it.
Paul Hammond:And there's certain things that I I have got into the habit of doing, which we can go into some detail on that I think make quite a big difference. But I will say that, I can tell you now that I haven't written any code, any production code, aside from... I think there's one function I wrote, and that was because I was on a stand up, but it was just easier to just... It was like a tiny function. Aside from that, it it was late November was when I last wrote code of myself, and that's including in my paid gig right now.
Paul Hammond:We've been delivering stuff a rate that I've never really seen in my career. But still a way still in a way that I think the quality is still high. And... But it but it very much requires... There's a skill to it.
Paul Hammond:Do you know? There's definitely a skill to it. It's not it's nothing even close to one shot, and it's not... What I do is not vibe coding in any way. You know, it's not it's not remotely similar, I I think.
Jonathan Hall:So I've done some... I've I've gone on a similar journey. I started with autocomplete and Copilot and, you know, it was great for for, like, refactoring where I needed to do, you know, the same chain in a 156 places. It was good at that. It it would get 80% of them right, and the other 20% I would catch it as I was doing it.
Jonathan Hall:Yeah. And then... Yeah. I think I think it was late last year, probably November or so that I started... I tried Claude code first time.
Jonathan Hall:And first just within Versus Code. And I remember I think the first day I tried it, I told it... I can't remember the task, but it was like I had a to do comment above a function that I had been intending to refactor for a while. I was like, I'm gonna just gonna try. Tell it to do this thing.
Jonathan Hall:And it's... You know, for twenty minutes, it went and did a bunch of stuff, came back, and it worked. Like, I had tests around it already, so it was it was a pure refactoring. It worked. Yep.
Jonathan Hall:The code was absolutely horrible. Like, was redundant and, you know, bad abstractions, but it worked. But I was both impressed and mortified at the same time. So I quickly deleted that. And since then, there was a while I was trying a single TDD skill similar to what you have.
Jonathan Hall:Claude helped me write it, and I hated that thing. I found that... So before before I get to that, let me start with... I'd probably just using Claude dot m d and telling it write tests first, and then... You know?
Jonathan Hall:And it would always forget. And I would say, where's the test? They'd say, oh, you're right. I forgot to write a test first. Let me let me do...
Jonathan Hall:Yeah. Whatever. Like, that that was, like, every single time that would that would be what happened. So I added a TDD skill. It didn't work very well.
Jonathan Hall:I can't remember the exact failure modes, but what I've what I've ultimately landed on right now, and I'm not perfectly happy with it either, but I think we can prepare some notes here and maybe I can get some suggestions from you. I actually have three TDD skills now. Have a red skill, a green skill, a refactor skill. Mhmm. I found that by using a single skill, it would it would often do the implementation before the test.
Jonathan Hall:Okay. Like like, because it had all the context for everything together, it it wasn't isolated enough to force it to do the test first. By splitting it into preskills, I very consistently have it writing a test first. The failure there is that it often does more than one test first. And sometimes that's okay, depending on what it's doing, you know, like if it's...
Jonathan Hall:I don't know. If it's a simple transform, I don't necessarily care that you have 25 tests first. When there's 25 different behaviors, then I'm a little bit more skeptical of that approach. And then the refactor fails mostly by refactoring things that are way out of scope. Yep.
Jonathan Hall:So I'm not perfectly happy with what I have, but it mostly works. The the big tweak I've done since I started it with my three skills is I added a gate at the very beginning that it had to tell me what behavior it was going to test first because it was frequently creating bogus tests. Like, it would just test that, you know, it wasn't a behavior. It was it was an implementation detail or or something like that. Right?
Jonathan Hall:So by having it described to me before it goes into to the red green refactor, describe that this is the behavior I'm going to test and wear, it still get... It still makes wrong assumptions fairly frequently, but I catch them very early. And so I don't mind that, right? That's better than catching at the end after it's spent fifteen minutes doing the wrong thing. Yep.
Jonathan Hall:So I'm curious how how would I describe as compared to your experience and if you have any any additional skill suggestions to make.
Paul Hammond:Yeah. So I guess one thing to ask. So like whenever I whenever I start any piece with Claude now, the very first thing I do, I always have it in plan mode. So I guess that's the first question. Do you go through the plan mode with it?
Paul Hammond:You do that every time.
Jonathan Hall:It depends. I was using the plan mode more heavily and I use it a little bit less often now with these three skills. Like I don't need to go back into plan mode as often.
Paul Hammond:Yeah. Yeah. So typically what I do... So I actually have... My TDD skill is just one skill.
Paul Hammond:And I also have a refactor agent. I have a refactor skill, but I I tend to use the agent, more. And the agent, I... I'm quite happy with... Again, I used, Claude to help me create that agent.
Paul Hammond:But I've got, again, I'll send you this link here, I don't know. I can maybe read out some of the things in there. But what are the things that I do in in this particular agent? This is one of the first agents that I wrote that I was really happy with, and I felt like I felt like I knew I'd done a fairly good job on it because it would push back sometimes, which is a behavior I hadn't seen in Claude previously. You know, I think it's gotten better now.
Paul Hammond:Like, the actual models have gotten a bit better at that now as well, I I think. Although I've been using this ever since, so I don't know if it's just because I'm using this. But it has... In in this particular agent, it's got a load of rules in there. And one of the things...
Paul Hammond:And I'm looking at it now and it says... So there's a section on balance, and it says, like, say no refactoring needed when code is clean. Recommend refactoring only when it adds value. Distinguish semantic from structural similarity. That's one that I think is quite important.
Paul Hammond:Provide concrete examples with reasoning. And then I have a bunch of examples in there about, like, decision making. There's like a decision making mini framework in there. So it says here, again, I'm just reading this out, but decision making questions. And then it says, for each potential refactoring value check, will this genuinely make the code better?
Paul Hammond:Semantic check. Do the similar code blocks represent the same concept? And I've got an example in there that shows basically the same code, but but expressing semantically different meaning and and explaining how to me that's what DRI was about. DRI was never about code duplication as such. It's about knowledge duplication.
Paul Hammond:Right? Right. And and so I felt like this particular agent was one that it was one of the first I built actually. And I was I was really with Claude code, did build it with Claude. But I was I was happy with it because I found that sometimes it would it would almost argue back at me, and I'd be saying, oh, well, what...
Paul Hammond:You know, is there any way of cleaning this up? And he'd push back and say, no. I... Here are the reasons why I don't think you need to. But then sometimes it will refactor and it makes a lot of sense, you know, when it does seem to do a good job.
Paul Hammond:But one of the things that I would recommend, and this is something that certainly the way that I kind of do it right now is I still I always start every feature with I want I want the plan. So I go through that. And I always tell it a few things. So to be honest, I should probably improve this a bit because my own really, I should have a I don't know if I have a I think I have a plans. I can't remember if I create a plan skill, even if I didn't, I'm not using it.
Paul Hammond:So I should I should probably build one. But but the way that I kind of do it is so every now and then, Claude will go for these compaction events. Right? Yeah. And what what often happens there because it's it's obviously trying to compact the conversation and so that it's got kind of more room for more context.
Paul Hammond:But sometimes it doesn't do the best job in that compaction event. And so I found in the past, you know, things are going really well, then it gets to that compaction point. And then suddenly it seems to forget what it was just doing or, you know, it loses it. So one of the things that I started doing, and this seems to have made quite a big difference actually. And this is one thing that I would definitely recommend if you're not doing is I I start in plan mode, and I get it...
Paul Hammond:I I go for all of this, and I usually just give it a reminder. I just say just, you know, read the... Read up on the testing rules. Make sure you're always going to test first. And even though I probably shouldn't need to do that, I've just got into the habit of reminding it, you know, and I say that.
Paul Hammond:And then I say, you know, always go in small increments, ask me ask me frequently to validate for verification. And then I get it to commit that to a plan m d file. Right? And and I tell it this is... We're gonna work on this incrementally.
Paul Hammond:The plan m d will be an incremental file. As we work through the feature, you're gonna update that file. And sometimes I'll even say to it as well, make sure that there's enough context in that file so that if if your context was completely lost, you'd be able to regain it again. To be honest, I really should have a plan document for that. I've just got it kind of...
Paul Hammond:I've just got into the habit of just just telling it that, you know. Yeah. But that one has made a really big difference. The amount of times as well, you get to like a compaction event and I just say I'll reread the plan and then it just instantly remember remembers where it was and carries on again. I I know that some tooling exists around this this problem as well.
Paul Hammond:And I think there's one called beads, and there's a few of them, but I haven't really played with those. And the main reason being that this just seems to work really well for me. And guess just because it seems to work, I haven't really felt the need, you know, to try some other tooling that I know people have built to kind of deal with that problem. But yeah, I don't know if because of the way I'm working there. I don't know if that has something to do with why it seems to stick to the red green refactor quite well for me.
Paul Hammond:Just to ask as well, are you using... I take it, are you using, like, Opus 4.6? Because that's the model I am now. The time... Yeah.
Jonathan Hall:Lately. I mean, that's that's pretty new, but but, yes, I am now. Yeah. I think I am. I might be using Sonnet.
Jonathan Hall:I I I started switch... I ran out of credits one one week. So I was like, let me let me go to the cheaper models at least for some of the time. So, yeah, I use different models for different things. Yeah.
Jonathan Hall:So I've I've been less formal about what you just described, but I have found also that, like, once I have a plan that I like, store it somewhere because Claude's gonna forget. Mhmm. And then as I as I work through it, I instruct Claude to update that plan, remove the things that are already completed. And and that's one that... That's kind of an aside, but I find it's it's really hard to convince Claude that a to do list is not a done list.
Jonathan Hall:It loves putting check marks and things that are completed.
Paul Hammond:Yeah, it does. Does do that.
Jonathan Hall:I just have to remind it all the time.
Paul Hammond:Yeah, yeah, it's it's definitely not not 100%, but it's... I I have gotten to the point now where I mean, I built... There's a tool that I built. Again, it's like a free open source thing. I I keep saying to myself, need to record some videos showing how this thing works.
Paul Hammond:A tool called Scenarist, like scenarist.io. It's not... It's just a testing tool. But that was was the first significant thing that I built entirely using Claude. I mean, that's from the ground up using Claude, and it's it's a it's a testing tool that can help with a lot of the problems that you get with the modern frameworks like Next.
Paul Hammond:Js and so on because those frameworks, I think, is... Have treated testing as like a a second class citizen, which really annoys me, you know, coming from the TDD background. And so, this is a tool that, you know, significantly helps. But if you read through the code on that, it's it's quite a significant... There's a lot going on and it's built in a...
Paul Hammond:I'm actually using like a hexagonal architecture internally so that I can build adapters for different frameworks. So Next. Js is just an adapter. I've got one for Express. I've got one for well, there's two for Next is the pages route or an app router.
Paul Hammond:And then I could quite relatively easily now using Claude, I could build, you know, adapters for pretty much any framework. Won't go into the detail on what it really does, but it's it's powerful. But alongside that, there's, like, three fully built apps within that... It's a mono repo, three fully built apps there that consume the testing library to prove that it works. And they they run as part of the test suite as well.
Paul Hammond:So, you know, on every pipeline, it would be so expensive for me to build that in terms of my time. You know, this... They're actually working gaps. You know what I mean? They're not trivial because they need to not be trivial.
Paul Hammond:So for me to do that in my own would just be I mean, I could have done it, but not in the yeah, it's like take ages and it would just be unrealistic to finish it, I think.
Jonathan Hall:You know? So I'm looking through the repo you shared with me and of course, I'll put that in the description or in the show notes for anybody listening who wants to look to. So you've got skills, agents, and command. So I'm... You you clearly have more experience with Claude than I do.
Jonathan Hall:I've barely started playing with skills just a few weeks ago. I'm aware that agents exist. I don't even know what commands are.
Paul Hammond:I think think commands now have been kind of superseded by skills, to be honest. Think I think I think they kind of came. So funnily enough, because my setup, the way that I kind of work, I have the dot files that I've shared and they exist globally in my system. And then you can kind of override things specifically. But actually, as it happens, now that you mentioned the commands, I don't really...
Paul Hammond:I haven't really used them too much recently. I think I was doing that earlier on. These things there like create adapter, for example, and it tells Claude how to create a new adapter in the system. But probably I should just convert them to skills. One thing I will mention, though, this is another really kind of interesting thing.
Paul Hammond:There's a docs folder there, and if you click on there, there's a ton of docs, probably maybe too many. But I've got into the habit of creating these ADRs as well. So if you go if you click on the ADRs and find any random one, so they've got like thin adapters, real integration tests, right? So there's a full... So I've got an ADR agent as well in my doc files.
Paul Hammond:What... Again, another thing that I found... If I don't... If you document stuff as you're going and you link to the docs from the Claude MD file as well, that can be quite useful. And Claude will just pick up those things.
Paul Hammond:So, you know, in a project that I'm working on now, this one is a closed source one, I'm working on a new kind of micro SaaS framework is my idea. So the idea is that my thinking was, you know, off is a capability, newsletters, email, all of that, you know, these these are all kind of capabilities. Payments is a capability, right? And so, again, it's kind of it's not quite hexagonal architecture, but it's similar, really. You have it's the same concept.
Paul Hammond:You have basically a port and an adapter. It's not quite the same as hexagonal because there's no kind of shared domain logic between the apps and stuff, but whatever. But the idea is that, using Claude, I was thinking, well, it'd be amazing if I could create this kind of framework that just builds, you know, and just say, I'm going to build a new application, SaaS product, and I need off any payments, payments will be Stripe. Do you know I mean? Orf will be authentic or off zero.
Paul Hammond:And I can just pull in the adapters and then boom, I've got that. And I just focus on the core problem of what that app does. So I'm building out an app now that is a consumer of that framework to prove it, and it's it's all kind of coming together. But that's, again, like, there's no way I could do the level of work that I've been able to do without these tools.
Jonathan Hall:I I started to have the same experience like I I still have Versus Code open occasionally. Not all the time like I used to. But it's mostly for browsing code. It's not it's rarely for typing code.
Paul Hammond:Yeah, same. Yeah, there is one one other thing I just wanted to mention, actually, Jonathan, that I think, you know, in terms of like things that are working really well, and this is something that is fairly new to me. Think I can't remember exactly when I did it, you know, find out for Git. It's, again, I'll share a link, but in in my doc files, I have a specific skill, which is mutation testing. And this one, yeah, this one is really interesting because it's working really well.
Paul Hammond:And like, it's it's a bit kind of crazy because, again, maybe for people who are not familiar with the concept. So if you imagine normally when you're going through... When you're trying to if you're trying to assess, you know, are my tests any good? Code coverage is a well known metric, right? And I do I do enforce code coverage, but it's not the problem that people have, I think, is, you know, coverage itself doesn't mean your tests are any good.
Paul Hammond:Right? And because all it does, it just proves that your you code runs. So your tests run that code at some point. Right. So it doesn't you can have 100% coverage and your test could be rubbish.
Paul Hammond:What mutation testing does is it's kind of a way of actually testing your tests to see that they're good. And by that, what it does is it will the idea is that it looks at your actual code. Say if you've got some logic, right? So if customer is great... Customer's age is greater than 18, do this, right?
Paul Hammond:Otherwise, do that, right? What it will do is it will mutation testing frameworks like Stryker that that currently exist, they will go through your code, and it will see that greater than. And it'll say, I'm going to now change that to a less than and run the tests and to see whether your test catches that error or not. Mhmm. And, of course, it should.
Paul Hammond:You... What you want is to see a test fail. When a test doesn't fail, that's called a a surviving mutant, and you want to get rid of these surviving mutants.
Jonathan Hall:Mhmm.
Paul Hammond:Now tooling does exist. So Stryker, for example, is a well known mutation testing framework, and it's really powerful. But the problem with Stryker is it... It's very slow, unfortunately, because it has to go through and it changes all this code in line, and it's expensive to do that. So there are kind of ways of kind of handling it so that it tests the diff and stuff, but I've always found problems with it.
Paul Hammond:And what I have seen is when teams use it, they'll run it kind of overnight and then look at the... Which is still valuable, but it's not the same as having the feedback right there and then. And I was thinking about this fairly recently, and I was like, well, maybe I could just create a skill which just does this. Right. And that's what I've done.
Paul Hammond:And I was a little bit cheeky. Actually, I'll just be honest with like what I did was it's quite funny. Stryker have got on their website and in their docs, they've actually got a page that lists all of the mutations and the mutators that they test.
Jonathan Hall:So
Paul Hammond:I just pointed Claude at that. And I said, take these, this list of mutators and turn that into a skill. Basically, you know, that's what it's done. But what it actually does, it actually does this in runtime. So it will look at the code that's changed on your on your branch.
Paul Hammond:Or if you're doing pure trunk based development, know, this is the code that you've just changed since your last commit. And it will say, okay, I'm gonna I'll just change these in line and you can actually see it doing it. But it's it's also clever enough to kind of give you feedback on so it'll it'll do some mini report at the end, and it'll say, these mutants survived. But then it'll sometimes say, actually, this probably doesn't matter because it's just... It's hard to think of an example off the top of my head, but some examples.
Paul Hammond:And remember, there's some CSS examples, and I'm not really bothered about that. Do you know what I mean? Like, I am... If CSS is a behavior, you know, if it's showing an element term red, then okay, I do care about that if it's an error message. But quite often I don't don't need a test for CSS.
Paul Hammond:Right? So So it'll go in there and it'll it'll detect these things and it'll actually say, I don't think these need testing, but what do you think? Do you know what I mean? It's yeah, would definitely recommend trying that out. It's very...
Jonathan Hall:I had in the back of my mind to do mutation testing and fuzz testing also. Yeah. With an LLM. I feel like it's a perfect match for that sort of thing. Yeah.
Jonathan Hall:Haven't done it yet. I do want to ask you about your TDD skill and and even your Claude MD mentioned Test Driven Development is is mandatory. Always TDD, he says, in your skill specifically for features, bug fixes, refactoring. TDD is non negotiable. I started with that hard line and I have found that there are times when I actually don't want TDD and maybe I'll back up a little bit.
Jonathan Hall:One thing I found is that trying to teach an LLM to do TDD and some of these other things has taught me more about these things. And I have found that there are definitely places where I actually don't want TDD. A simple example is deleting dead code.
Paul Hammond:Oh, yeah, yeah, Yeah.
Jonathan Hall:And I try to I tell Claude, go delete this dead code. It's like, let me write a test for that. I'm like, wait, what kind of test are you going to write? You can maybe wrap your test around that somehow, but it's it's weird. So I'm curious if you've run into similar things.
Jonathan Hall:Are there places where you maybe maybe you're this hard lined in the in the configuration so that it doesn't mess up? But do you relax that sometimes when you're interacting with Claude?
Paul Hammond:Yeah. And that's that's a really good it's funny because when you said it initially, I was like, when would you not want to do CDD? But then you gave a really good example. It's like, yeah, no, that does make sense. Yeah.
Paul Hammond:Yeah. And exactly that. Like, I it's funny because I think if I were just telling it to delete the code, I would I would kind of maybe I just trust it too much at this point. Would almost expect it to to not go test first, but it could well do. I get it.
Paul Hammond:And if I see it doing that, like you can quickly hit escape and then it's like, no, just you can ignore the rule for now. Do you know what
Jonathan Hall:I mean?
Paul Hammond:It's fine. Think there's some of it like another example. Yeah, remember the number examples. I have like a certain pattern as similar to builder pattern. I tend to my kind of preferred kind of coding style is a bit more functional.
Paul Hammond:So I don't really do the objects or kind of builder pattern so much, but it's similar thing. It's like a similar concept in in a more functional way. And then I remember one seeing it, you know, build these kind of mock kind of builders, you know, for mod data specifically. And it was doing them test first. And I was like, no, you don't need to do that.
Paul Hammond:Like that's for testing. You know what I mean? So you don't need to.
Jonathan Hall:That's been another example. Yeah, it'll try to do test first for the tests. Yeah. Or like a little helper that's not in the testing library, but it's only for tests and it's what, you know. Yeah.
Jonathan Hall:So there are like, I keep finding edge cases where the hard line always TDD rule doesn't actually follow through.
Paul Hammond:Yeah. Yeah, it's I think a lot of that as well. It's just it's common sense, isn't it? You get to a point where you, you know, like, again, even before doing, you know, A. I.
Paul Hammond:And stuff, these these points where, you know, you can violate a rule for a certain reason. Do you know what mean? Like, to me, that's that's the kind of sign of an engineer that's I don't know. You kind of get into a point where things have really kind of fallen in place for you when you know where you can violate a rule. Do you know what I mean?
Paul Hammond:It's... Right.
Jonathan Hall:Yeah. I think it goes back to kind of what I was saying that, like, trying to teach two year old, basically. All these rules that I feel like they're ingrained to me helps me, you know, sort of shakes out the corner cases and the ed cases and and realize that it's not nearly as black and white as maybe I thought it was in my head. When when the when the rubber hits the road, there are more eddies than I realized.
Paul Hammond:Yeah. So one of those is a kind of question that I have for you, which is it's one of those things that I've been thinking about it recently. And I'm not I'm not 100% sure where I stand on this one right now. I mean, I know where I am now. But I don't know whether my perspective has changed or what.
Paul Hammond:And that is how far how far do you feel the need to to check every line of code that the LLM is is writing? And, yeah, I'm interested in what you think on that.
Jonathan Hall:I try to at least review everything it does, but it doesn't necessarily mean every single line of code. And I think it kind of depends on what I'm doing and how quote unquote important it is. You know, I don't mean that in the sense of like, oh, if this code stopped working, I wouldn't care. But, like, is it a security path or or, you know, something like that? Yeah.
Jonathan Hall:Some of the changes are pretty... A lot of code is is honestly repetitive, right? You know? You know, if I'm telling Claude to do something, implement this pattern over here, then I don't necessarily need to feel I don't feel like I need to review it as as thoroughly as if it's like what... We're inventing this new thing together.
Jonathan Hall:Mhmm. And I wanna make sure, you know, I wanna read it really closely. I guess it's similar to how I review code from humans, though. Like, if somebody's if somebody's adding a new, I I don't know, a new widget that copies the same pattern we do everywhere else, it's just displaying x y z instead of a b c like it did over there. Just, know, I want make sure the shape looks right and it and it feels like, you know, it has the right methods on it or whatever, you know?
Jonathan Hall:And then, you know, again, that that's probably fine. If we're building a new auth endpoint or some weird concurrency thing or something like that, I'm going to review it a lot more closely. Yeah. So I guess that's kind of my feeling. I'm not I don't trust Claude very far.
Jonathan Hall:I mean, I'll be honest with that. Like, it makes a lot of stupid mistakes.
Paul Hammond:Yeah. Yeah. Yeah, I think it's probably probably similar to I guess how I feel like. I still feel like I mean, foresee to work in these kind of smaller increments. I still do that.
Paul Hammond:And then I still think it's a good idea anyway, you know. And at the moment, I I I do check the code that it writes. I make it do pull requests. That's something that I do. And I go through and I just...
Paul Hammond:But most of the time, it's it's a pretty quick. I'm just looking for obvious things, you know, similar like you say to if you're reviewing a human's code. I saw all of a discussion but prefer pairing with humans than than that. But yeah, it's it does make me think, though. Yeah, maybe
Jonathan Hall:remember, there are times when I will ask Claude, does the code do this thing? Rather than like manually going to verify, like, did you consider this edge case? You know, does it handle a four zero four error correctly? You know, I'll ask questions like that. And then, you know, I usually trust its response in that case.
Jonathan Hall:Yeah. So I guess that means I don't review the code 100% of the time or else I wouldn't be asking those questions to Claude.
Paul Hammond:Yeah. Do you find... Would you... Do you still review the tests or do you treat that the Yeah. Same
Jonathan Hall:I'm I'm kind of picky about my tests. I want my tests to be a certain structure and I want to make sure... I I don't necessarily get bothered by, like... So in in Go, I mostly use Go. But in in Go, there's a pattern called table driven test where you, like, you have a single test runner and you pass, you know, 20 different inputs and output expectations through the same thing.
Jonathan Hall:Yeah. And usually in my and one of the common idioms in the go is that you use a library basically does a diff between two different structures, right? Or two objects rather than asserting each member of the object one by one. Claude often likes to do the list of assertions rather than the whole thing. And sometimes I just let it go.
Jonathan Hall:So that's a case where, like, I would never write it that way. I wouldn't... You know, if it's just one or two things, wouldn't... It's the kind of thing that I wouldn't block a pull request for, like, if I was reading a human's code. So maybe I...
Jonathan Hall:I'm I'm probably actually stricter with Claude than I would be with human because, like Yeah. There's no feelings to be hurt. But so there are things like that where I'm a little bit more relaxed than I would be if I was writing my own code. If it's more than two or three things, if it makes the function hard to read, then I will have Claude fix it. But so there's little things like that where I'm a little bit less picky about the exact format of my tests than I would be if I was writing it myself.
Jonathan Hall:But the part of that's like just a pragmatism thing, like, is it worth my effort to tell Claude to change this thing that doesn't really hurt the readability and, you know, it doesn't really matter.
Paul Hammond:Yeah, it's interesting to... It will be interesting as well to see how things kind of pan out long term because I still have this kind of perspective, like, I want it to be kind of... I I still... The mental model I still have right now is, if Claude just disappeared overnight, I should be able to pick it up and continue relatively easily. Mhmm.
Paul Hammond:But the longer I go without actually writing code, the more I start thinking, well, do I need... You know, it changes your perspective a bit. It's like, I shared some code that was written by Claude on LinkedIn, today, actually. And somebody, pointed out, oh, here's an example. Like, it it was the repository I shared with you just now, and it's quite a big repo with loads of stuff going on it.
Paul Hammond:And I I think the code quality is pretty high there. But if you dig deep enough, you're gonna find the odd thing for sure. You're... Nothing's perfect anyway. And he found he found a function that was quite long.
Paul Hammond:I think it's about a 100 lines long or something. And initially, I thought, yeah, okay, that's that's bad, and I could make it better. But I was kind of thinking, well, does it matter as much now? I don't know. Does it matter as much?
Paul Hammond:Because because because Claude, the tests are good, and I know I trust the tests. And the tests are really comprehensive there. Like, I have, like, three full apps consuming, proving that every single edge case works in all of these apps as well. So I'm like, well, I don't know if that matters as much as it as it did.
Jonathan Hall:I generally don't care about function length until I until it confuses and then I just refactor it at that point. Yeah. So that's what that's whether I wrote it. You know, maybe I write a function and, you know, every I add a block here and I add a block there. And over time, it's now 300 lines long.
Jonathan Hall:Until it bothers me, it doesn't bother me. And when it does, I refactor. So that's kind of the approach I would take for that anyway. Like, discover the shape of your code. Don't don't mandate it.
Jonathan Hall:And so, yeah, I I personally wouldn't care very much about that unless I have to read the code for my own comprehension, and then I'm gonna... In the process of reading that, I'm gonna refactor it anyway. So
Paul Hammond:yeah, I think. Yeah. And again, I think that's a mature approach kind of shining through its experience shining through that, I think. One other thing I just wanted to ask, if you don't mind, is and it's a separate thing, but well related, obviously, to AI. But it's more, do you worry at all about the whole?
Paul Hammond:Do you worry about your own job? At any point? I mean, it's, it's Yeah, it's these two sides to this. Do you worry about yourself, your own position? And the the other thing and, you know, this is the final thing that I'd probably want to ask you, but it's about it's about juniors and people new to the industry and the impact it has has on them.
Paul Hammond:Guess if we start about yourself first and, you know, and what you I'm just interested what you think on that.
Jonathan Hall:Yeah. So do I worry? Yes. I have two worries and you touched on both of them. I do worry about my own job, but not in the way that a lot of people are talking about in the hype cycle.
Jonathan Hall:I'm I'm mostly worried that there might be a big recession coming, as the AI bubble, if that's what it is. I believe it is, but, you know, we we don't know until things are are done. Mhmm. But assuming there's a bubble and it pops, there's gonna be a lot of retraction in money flow. There's already been some of that.
Jonathan Hall:Right? A lot of layoffs have been happening in the last couple of years. And there's... So couple that with the perception, or maybe it's just wishful thinking, maybe it's not even perception, but certainly talk about AI making everybody redundant and more efficient. And I think in a lot of cases, AI is an excuse to lay people off, not the reason.
Jonathan Hall:But that doesn't really necessarily matter. As long as there are enough people out there who think that AI can replace me or my, you know, demographic of software developer, they may stop hiring me and be like, for a while. I think it won't take long for them to realize that that's a big mistake. But can I survive that long? That's the that's the worry.
Jonathan Hall:Yep. Can I survive twelve months to to two years while the industry is realizing, oh, they do actually need my skills? And that's so that's the worry. It's not that my skills are obsolete. I don't believe they are.
Jonathan Hall:I believe that the skills of people with our sorts of experience are being magnified with with AI, not replaced. But I don't know if the industry recognises that yet. So that's that's the concern for me.
Paul Hammond:Yeah, I think it's a really good answer, honestly. Like it's it's I agree with the the magnifier kind of amplifier kind of thing. I think it's it's definitely true. I think that's why people that may become like because I'm hearing more on the grapevine people who previously were not talking about, you know, Test Driven Development are suddenly talking about it. Do you know what I mean?
Paul Hammond:And it... It's these skill... Skills that have always worked so well because they're now being amplified to the extent that somebody like myself suddenly becomes so much more productive. Yeah. People have to start taking notice now, do you know what I mean, of those skills.
Paul Hammond:Yeah. But yeah, at this point in time, I... My own kind of perspective on this is it. There's still a big part of me that just loves the tech because I just I can't help myself. Right.
Paul Hammond:Just just, you know, I right now, I think it definitely requires people of our kind of level of skill to be able to run these so effectively. Whether it'll be that way longer term, I don't know. Five years from now, that's who knows where we're going to be five years from now. It's hard to say.
Jonathan Hall:I don't really have any fear that AI will replace my skills, actually. Because the the hard part has never been writing the code anyway. We can write the code faster now. Yeah. And as long as we know how to keep Claude or our AIs on a short enough leash, we can we can iterate very quickly.
Jonathan Hall:But it's the engineering skills that a Claude doesn't have, you know, knowing when to apply which patterns. It can it can copy patterns all day long, but it doesn't know which pattern belongs where all the time. Yeah. But then to to your other concern that you you brought up, juniors and, you know, people starting their careers. Long term, that's a bigger concern, although I I don't I don't feel doom and gloom like some people have been saying.
Jonathan Hall:I think that software development software development's always been a pretty easy thing to get into if you wanted to. Just get a computer and learn to tinker, and then you're... And in a few years, you're a software developer. It's kind of been the path. I think it's gonna be more like becoming a medical doctor in the computer that you have to dedicate yourself to, like, eight years of hard study, or maybe it's not eight, but certain number of years of hard study, develop those skills because because a lot of...
Jonathan Hall:Mean, a lot of people are gonna try to take the shortcut. They already are. You know, vibe coding is the shortcut. Just, oh, I can just tell ChatGPT to build a website for me and look, I'm selling shoes online now or whatever.
Paul Hammond:Mhmm.
Jonathan Hall:Oh, whoops. Someone stole your credit card. You know? Yeah. That sort of stuff.
Jonathan Hall:That's going to be the pattern. Doing it correctly is still a skill, but it's no longer as accessible as it was. So I think it's raising the barrier to entry, which is going to make make it a harder career to get into, which will be good in a sense for people like you and me who are already here. If we can survive over that that potential dip, you know, if we can survive that dip, then there's going to be, I think, a bigger demand for developers. Yeah, I think I think I think AI is increasing the demand for developers because people are realizing more things they can build and build them faster and at the same time making it harder to become one.
Jonathan Hall:So it's attacking the supply and the demand side of things.
Paul Hammond:Yeah. Yeah. There's also the whole for people like us as well. There's a whole legacy like, you know, most systems are not built in in we know this, right? Because like I know prior to the, you know, the AI stuff, I would chat to you on LinkedIn.
Paul Hammond:And, you know, we were always, you know, talking about these kind of skills and the amount of people who would push back. And you see it when, you know, well, I'm contracting now, so I move around and it's very rare to to join an organisation where the tests are in really good shape and where the confidence is high. And some the vast majority, at least in my experience, the vast majority of systems in the wild, you can't just you can't apply all these techniques out of the box. You actually need to know how to how to take the kind of existing code and turn it into something that, you know, you can then apply these techniques to. So the whole Michael Fevers working effectively with legacy code, you know, building skills around seams and things like that.
Paul Hammond:That's that's there's a lot of value, I think, essentially there where people like me. Yeah. You see? So, yeah, it going to be interesting few years, I think, for sure.
Jonathan Hall:For sure. Yeah. So I think you've also touched on a lot of people are trying to everybody's virtually everybody is trying to apply AI to the current skill set they have. And if they have... If they aren't skilled with testing and with building robust engineering software projects, the AI is just going to make it worse, not better.
Jonathan Hall:It'll do the same thing faster, which means it'll be bigger messes to clean up. So, yeah, for those of us who are focused on that and can find people willing to pay us to do that, there will... I don't think there's going be any shortage of work. I think the question is, will people want to pay us for it?
Paul Hammond:Yeah. In time, I think they will, but you say. Right.
Jonathan Hall:Well, Paul, I want to thank you for taking the time to talk with me. Before we close off, is there anything you'd like listeners to be aware of? I know I know that you shared your GitHub. We'll put a link to that in in the show notes or on the description wherever you're you're listening or watching this. Anything else you'd like to share?
Paul Hammond:There is there is a site, the SaaS thing that I told you about that I'm still working on. Obviously, right now it's not open to the public. There is a landing page there, which is chipin.online. I don't know if that would translate so well to to American audiences. In The UK when...
Paul Hammond:I don't know if that's just a UK slang term, but in The UK when, say, if it's somebody's leaving party or something and everybody puts money together to buy them a present, we say we're going to chip in. So we're all going to... I didn't
Jonathan Hall:say that here too.
Paul Hammond:You said that. Okay. Okay. Right.
Jonathan Hall:I didn't
Paul Hammond:know if it's okay. Fine. So yes, it's the whole concept is based around that basically. But there's no one even though you can see a login link there, if you're looking at it right now, you you can't actually log in. Well, I can log in, but it's sign up and sign ups are not open yet because I'm still working on it.
Paul Hammond:Right.
Jonathan Hall:Are you doing that that off code closely?
Paul Hammond:Yeah. Yeah. But it's it's it's coming along. And so, yeah, hopefully before too long, that'll be an actual thing that people can can use and check out. Cool.
Paul Hammond:It'll be awesome.
Jonathan Hall:Launching in The UK. So if you're in The UK, out chippin. Online. If you're not, you can still read it and maybe later it'll be open to other places, I imagine, if it's successful. Cool.
Jonathan Hall:Awesome. Well, thanks again, Paul. It's been a pleasure. We've we've followed each other on LinkedIn for quite a while. It's nice to finally put a voice to the face and to the content.
Jonathan Hall:So thanks for joining me. Yeah.
Paul Hammond:No problem. It's been good. Thank you.
Jonathan Hall:Great.