Subscribe
Share
Share
Embed
Host Ran Aroussi (Old School / New Tech) shares a recent “war story” from his agency Automaze: a hard-to-reproduce mobile bug where calls were sometimes dropped only on a client’s devices, which the team couldn’t reproduce for 2–3 weeks. After two long sessions using a Droid Factory setup with agents and a “small council” debate between models (e.g., Fable and Sol), he got the team unstuck, found the issue, and then turned the session logs into an internal guide and public article on his methodology. Key practices include distrusting agent conclusions (treat “preexisting/flaky/unrelated” as hypotheses), demanding proof (including his Proof library requiring video evidence), using extensive unit and end-to-end tests, asking for a numeric confidence level before production, controlling environments, handling merge conflicts with full review/testing, switching models for independent review, and re-spec’ing from scratch to detect drift from the implementation plan.
00:00 Show Begins
00:58 The Sneaky Mobile Bug
02:46 Agents Find The Fix
05:23 Never Trust Preexisting
08:29 Proof Over Promises
10:14 Confidence Level Gate
12:47 Model Debate Reviews
17:08 Merge Conflicts Standards
21:23 Spec From Scratch
25:49 Ten Rules Summary
31:30 Token Usage And Wrap Up
-----
Old School / New Tech is produced by VarOps.
00:00 Show Begins
00:58 The Sneaky Mobile Bug
02:46 Agents Find The Fix
05:23 Never Trust Preexisting
08:29 Proof Over Promises
10:14 Confidence Level Gate
12:47 Model Debate Reviews
17:08 Merge Conflicts Standards
21:23 Spec From Scratch
25:49 Ten Rules Summary
31:30 Token Usage And Wrap Up
-----
Old School / New Tech is produced by VarOps.
- Subscribe to the newsletter – the written version of this show, weekly: varops.com
- Read the books – Company-Scale Agentic AI (for operators and the C-suite) and Production-Grade Agentic AI (for engineers): varops.com/books
- Work with us – VarOps installs resident AI inside mid-market companies. It runs where your work already happens, sees the whole operation, and what it learns belongs to you: varops.com/transform
Creators and Guests
What is Old School / New Tech?
Old school thinking meets new tech — an unfiltered podcast, hosted by Ran Aroussi and a co-host you won’t see coming.
Live every episode. No production lag, no editing. Business, AI, tech, startup ideas, and whatever's worth talking about — through the lens of someone who's been building production systems for 35+ years
Welcome to Old School New Tech, I'm your host Ran Aroussi and this is going to be episode nine, I think, nine or 10. And I'm quite surprised that I made it thus far and that I was able to stick around and keep talking about the different things that I'm doing and working on and so on and so forth. I had this whole plan to talk about some cool stuff this episode, but then something happened this week and I decided that maybe that's what I'll be talking about instead because it's both timely and I think it it can give you kind of a quick hint as to how a real software development for production environment works. So, as you know, I have a software development agency. This is one of the things that I do.
Speaker 1:It's called Automaze. We work with different clients and we have this one client that they have a mobile application. It has something to do with communication and making calls and all that fun stuff. And we were circling a bug for the past maybe two or three weeks. Circling around a bug would be probably the best way to describe it because every time we thought we had fixed it, the team thought it was fixed, then something else that's related to the same problem kind of came up.
Speaker 1:And I was called upon as the agent whisperer to try to figure out what the issue is. And I wrote about it on my ex, so you know, I'm just going to share this article here. I'm just going to zoom, so if you're watching this, you can follow along. So basically what happened is that I was called in to figure out how to help get this thing unstuck because it was a very, very sneaky bug, and it had to do something with calls being dropped, but only sometimes, and only on the client's devices, to be honest. We were unable, other than maybe once or twice, we were unable to ever reproduce this bug.
Speaker 1:So while I was thinking that this whole it works on my machine era was over, apparently we still have these quirks every now and again, especially with mobile applications. But anyway, I was brought in to take a look and see how can I get things kind of moving along? And after a session of about, I think I had like two fifteen hour sessions or something along those lines with my trusty agents, I'm using Droid Factory with a consortium of both Fable and Soul, GPT 5.6 soul, both at high, and I have this I don't use a lot of skills or anything like that, but I have this one skill that allows me to talk to both of these and have them argue each other and try to probe and figure out what the problem might be. So while I was working on that, I was able to both find the issue, fix the issue, and get the team unstuck. And I don't think it has a lot to do with I'm a better coder or anything like that.
Speaker 1:I think that my approach to software development was what got me past this issue. I was able to basically find the problem. And then what I did, because I got these requests from the people on my team, what I did is I took another agent to go over the entire logs of the session that I had that where I found and fixed the problem. And I asked it to, Hey, write a guide about how I talk to AI agents. If you see anything special that I'm doing that maybe people can learn from.
Speaker 1:I wanted to write an article about that. Well, started with an internal document that I gave to the team. Obviously had a lot of examples that are specific to the client, but then I had it rewritten into this article with obviously the client information is not disclosed, but the methodology that I'm using and kind of the rules that I'm using when I approach such case is what I want to discuss. So that was a lot of intro to basically go over how I talk to AI agents and how I attack and locate and fix something that were unable to reproduce for weeks. It wasn't that we were dealing with a few hours and weren't able to reproduce it.
Speaker 1:We weren't able to reproduce it for literally two or three weeks. So the first thing that the agent kind of surfaced from my interaction with it is that I don't trust it in a good way. So I trust it completely to write code, but I never trust it when it says, okay, I found the bug, it is a preexisting issue, so it couldn't be this thing. This line of code predates the bug or it can come up with different excuses, just like a human developer would. And I never let it slide.
Speaker 1:I even posted about it later. So you can see that I even taught my agent to finally realize that a passing test doesn't mean that it's proof that everything works. So that was kind of a funny thing to see that it was able to learn and pick up on my disbelief of it, because the agent will believe itself. So if you give a capable model a task, it will do a genuinely good job, but it will also cut corners. And that's what they're trained to do.
Speaker 1:They're trained to get back to us and tell us, yeah, we're good to go. So it will make sure that test pass, even if it needs to create mock APIs or whatever it is that it needs to connect to. And yeah, so don't believe it. So every time that it came out with something like it's a preexisting, I wanted to treat it as a hypothesis and not a finding. So that means that whether or not it's pre existing, whether or not it's part of this coding session, it's unacceptable Because when it ran, we had for this part of the testing, we had 90 end to end tests for that specific feature.
Speaker 1:85 of them passed and the agent wrote it that it had an explanation ready because the five failures are pre existing and they had nothing to do with what we were working on. And it all sounds really nice and good, but I asked them to clear those tests anyway to really probe and figure out what the problem is. And when it went down to a rabbit hole, I figured out, oh yeah, these are actually bugs, it's not just failed tests and not just skip tests, It's an actual issue and it resolved it and it was definitely part of the problem. So I would say that whenever the agent tells you this is unrelated, use your judgment here because yeah, if you change the button's color and the login stopped working, it may very well be unrelated, but it also might be the case where it forgot to close an HTML tag or didn't write the proper function or just because it's only a color change and on its surface it has nothing to do with the bug, do not believe it. Always ask for proof.
Speaker 1:And I think I mentioned it once or twice or more that we have our own internal well, it's not internal, but we created it internally. You get it on GitHub. It's called Proof and it's a library and a skill that the agent can install and every test that comes back to you and says it's passing, it needs to include video proof, whether it's video proof of the terminal, everything that happened or a browser or a mobile application or a desktop application, it will create a video of the thing actually working. So that's very important. So pre existing play key, it can get back to you with something, oh, once we update the API key to the real one instead of the testing one, it should work.
Speaker 1:Don't take its word for it, because while I don't necessarily take a look at the code anymore, I do impose a lot of restrictions on what I will accept as done. So by the time all these restrictions are met, so we have unit tests in the thousands and end to end tests in the hundreds, and I ask for a video proof and I ask for linters and I ask for profiling. Once all of that has proven to pass and work and looks good, then whether or not the code is pretty doesn't really matter. What matters is that it did pass all these criterias. Another thing that I noticed I do a lot is I keep asking the agent every time that he comes back and says it's done, I keep asking what's your confidence level that we can go to production with this?
Speaker 1:And you can see that immediately it will backtrack on its assumptions and it will start telling you, no, no, it's not really ready. Let me take another look. I do have some open items. So it will immediately go back from you're good to go. It will go to kind of covering its own ass and telling you that, no, we need to run a plethora of additional tests and different metrics.
Speaker 1:So it's really cool that every time I ask, okay, what's the confidence level? And it comes back to me with a number. Sometimes the number cannot be 100% without a human intervention, so sometimes we can do all the testing that we want with mobile simulators and everything, but at the end of the day to get really 100% we need to test it on a real device that I can hold in my hand. So sometimes the most we can get is to 90%, but I want the agent before it hands off any sort of remaining percentage points to the human. I want it to achieve the maximum we can achieve and more often than not it can get to 100% or very close to.
Speaker 1:So anything under 5% less of the ceiling, I'm not going to let it off the hook. So, for example, for a mobile app, eventually we do need to test it on device. So let's say the maximum the agent can confidently reach is 90%. Anything less than 85%, I will not consider it as done and I will not move it to a human testing. Same goes for if it's a CLI script that the agent can test end to end without any human intervention, then anything less than 95 to 98% before a human jumps in and runs their own tests, anything other than that is unacceptable.
Speaker 1:So the agent's job is to keep working until it can confidently come back to me and tell me that, okay, we're at 100% or whatever number that we can reach. And another thing that I'm doing, and now I've already created it as a skill, but another thing that I used to do is I used to switch between the top models. So I don't necessarily use the best models for the actual coding itself. Once the plan is set and once all the checks are done by state of the art models like Fable and Sol, a lot of the coding can be done by GLM or by Kimi. So these are great models and they can do the job they were tasked to do, but I wouldn't trust them with any sort of judgment, that sort of thing.
Speaker 1:So I used to kind of ask the state of the art model, let's say I'm using GPT. Before I had the skill, it was GPT 5.5. I'm asking, okay, go take a review, do a review, tell me what you think, what's your opinion about the application, where do you find room for improvement or simplification or anything like that. Even after all the tests are passing and all the constraints have been met, I still go and ask what their opinion is. And then I take this output, I switch models and it's very easy to do with the Droid, that's one of the main reasons why I'm using them instead of directly with Codex or with Quad.
Speaker 1:Then I switched model to, again, I had the scale, it was Opus 4.8. And I'm literally telling you this is what GPT had to say. What do you think? And then it will immediately try to one up it and try to either disprove some of GPT's claims, which is a good thing because if they can be disproven, that's great, or find additional things to show that I'm better than GPT. It'll try to find additional things and then I do another round where I'm giving output back to GPT and now I have it as a skill.
Speaker 1:You know what, I'll just put the skill on my website in the next few days. I'll just kind of make sure that it's ready for public consumption without any personal information disclosed, but essentially what I'm doing is I'm letting them argue with each other for up to five turns and then get back to me. And yes, it's expensive, but it is worth it for them to be able to come to terms on, okay, they're both in agreement on these issues. Sometimes you can see Claude saying, I'm not as concerned as GPT on one, two, three, but you might as well do it. And sometimes it will just say, no, no GPT is wrong.
Speaker 1:And then when I go back to GPT and says, Claude told me you're wrong. Sometimes GPT will say, you know what, Claude is right. And it's nice to see that the labs have created a built in animosity between the models and it only serves me as the one that kind of the puppet master of the agent management. Again, I mentioned this earlier: always show me, never tell me. I don't care what's your confidence level, I don't care about you've added a test and you're saying that the test is good and that it's working, show me.
Speaker 1:I want to see the video or as far as I'm concerned, nothing happened. So this is another part of the requirements that we've been adding. I don't recommend you with every commit you're going to run all the tests because if you have enough tests, it could be something that takes hours. But what I do is I always ask it, okay, between commits, when we're not talking about a release between different commits while we're still in development, run five or 10 or 20 based on how large the change was, just random end to end tests just as a smoke test that there are no regressions, nothing bad happened during development. But of course, we do a fully merge into the release candidate branch or the main branch, then everything has to run and everything has to pass.
Speaker 1:So this is just another part of the standard that I hold my age to. Whenever there's a merge conflict, don't just try to fix the merge. We're a team, we're almost 30 people working on the code and sometimes you inevitably two people will work on the same file, modify the same file and there will be some merge conflicts. Some merge conflicts can simply be fixed by, know, tending to those five lines or whatever it is that got overridden by the two developers. And the agent will just try to find the compromise in between.
Speaker 1:Again, this is unacceptable. Whenever there's a PR merge, one developer has to ask the agent to merge the other PR into their working branch, fix the merge and then run the full code review, the full testing as if it's the strangest contribution to an open source software. You have to run all the tests, but again it's a different manifestation of the same principle that I don't want to accept your word for it. Maybe in a year's time from now, maybe we'll be able to blindly accept whatever the agent is saying, but right now without all these restrictions and this framework, I will not accept its word for it. And if you cannot verify something, if the agent cannot verify something, I ask it to just tell me and then we can go off a brainstorm session.
Speaker 1:You can tell me what you're experiencing and what might be the problem. Maybe I can offer some idea, but this is time to kind of step back. But I also want to cover another thing is that all of what I just described, you can and we are automating, so we have this software factory that we use and it has the exact same methodology, but software factories, we tend to not use them while an application is still in, let's call it exploration mode. So while we're still working on something that doesn't have its full shape yet, because once you have the full shape of the application, you can start using just simple tickets. Hey, do this, add this feature, run this feature.
Speaker 1:But until we fully have a grasp of what it is that we're actually building, we are still in an IDE like environment, specifically some of the people on the team use Versus Code, Cursor, Droid, Dell, Orca. So we have a plethora of tools, but we all use the same skills and methodology in the factory. Due to this experiment and this experience this week, we are going to start enforcing it on the exploration phase while the developers are with their IDE. So this is something that's very important and it's proven that if you use this approach and you don't let the agents bullshit you, for lack of a better word, then literally I was able to kind of circumvent something that we were kind of fighting around and trying to solve for the better part of the past two, three weeks. And I was able within net of about thirty to thirty five hours, which is essentially three days of, as I mentioned, was two very long sessions, about fifteen hours each.
Speaker 1:Most of it was just agents working, so I wasn't sitting in front of the agent kind of steering it for fifteen hours. That would be insane. But that's one of the things that we are going to start implementing in local development as well. I also noticed that this is something that I do a lot is I spec the entire application from scratch. So before I even take a look at the code and say, okay, I understand what the software needs to do and I understand that it's not working, but let's ignore the fact that we have anything.
Speaker 1:How would we write it? Let's take another look maybe at the PRD from scratch and the implementation plan and what would be the ideal way the code would look like. And I kind of sit with the agent for a brainstorming session to create the most efficient version of the implementation plan. And then I asked it to compare it to what was actually implemented. And I can tell you that two things happen more often than not.
Speaker 1:One is that the new implementation plan is in many, many cases almost identical to the original implementation plan. So that was never a problem of or that was almost never the problem of we wrote a wrong implementation plan to begin with. So what happened would definitely have to be drift from the implementation plan. It's the agent kind of convincing the developer, 'hey, let's do this now, let's work on that, let's kind of basically creating a drift and going in directions where we weren't planning on going. And having this grounding and saying, okay, tell me how far away the existing code is from the implementation plan that we had or that we just recreated, you can see that a lot of times it will find the issues immediately.
Speaker 1:Well, immediately sometimes it can take hours, but it's agent's work, not you going over the code with the agent, but it will find the problem, which means that the problem was due to drift. Wrote it here and maybe it sounds like I'm touting my own horn here, But yes, I still think that I'm a better engineer than the agent. I'm a far worse coder than the agent, but I'm much better engineer than the agent because I'm able to keep steering the agent, and I've noticed this repeatedly, steering the agent to the simpler solution. And I actually, I even have it as a keyboard shortcut to automatically tell the agent, I want you to find a simpler way to do this, suggest different ways of doing this and see what existing code can be leveraged, what can be reused. And it's just that simplification is what makes me the better engineer.
Speaker 1:A good engineer is a lazy engineer, who wants less code and someone who wants a lot less maintenance. And it works for a lot of industries where the laziness is what makes things more efficient. So, yeah, so when you make sure that you still are at the wheel and you're the one that is responsible to steering the agent in the right direction, you will get better results. And this is not a new thing. This is something that has been going on for years.
Speaker 1:Used to do code reviews for junior developers and we used to check their PRs and steer them in the right direction. Now the developers are the agents. And I'm certain that we will have less work and agents will be able to exercise more judgment as getting better. But right now we do need to remember that there's still just a loop that runs and figures out what the next tool should be in order to, well not brute force, but conceptually brute force their way into a working application. So if I had to summarize, and I did summarize, these would be the 10 kind of rules of how I work with agents and how I recommend that anyone else would use and talk to their agents.
Speaker 1:As I mentioned, I don't have a lot of skills. I have like a browser skill. I have maybe a handful of more, but the one that I keep going for is this small council, as I call it, of now it's Fable and Sol to kind of debate each other on who's doing a better code review and who's fixing problems better. So it's not about setting up the perfect environment with the million skills and NCP servers and what have you, it's just making sure that you have the methodology right. So the first rule is match the model to the stakes.
Speaker 1:Use big models and literally bring in the big guns for hard problems, and if you can bring in two independent models to review each other, that would be really, really good. Again, it's expensive, and if you're developing an application that you're the only user, that no one cares. Just have it work and working sometimes it's good enough. But if it's anything that needs to go to production and people are going to use it, bring in the big guns, pay the big bucks and just do it. And for just pure coding, you can definitely use the lesser, well not lesser, but the cheaper models, let's call it that way, the cheaper models like Kimi, K3 and GLM.
Speaker 1:Yeah, they're really good. Preexisting, flaky, unrelated, these are not an answer that I will ever accept. This means that we need to start an investigation to figure out why they happen, whether or not they're preexisting. I love that they start a new session with no knowledge of previous sessions and they just decide that something is pre existing, so I never accept it as an answer. Number three: ask for confidence level per code base as a number and ask it what do you estimate that we can realistically achieve without the human actually running a test.
Speaker 1:Many, many cases it's going to be 100%, so ask it for a confidence level. When it says, okay, I'm at about 75, ask it, what do you need to do now to get to 100? And it will come up with a plan. And then you take this plan and you go back to item number one, which is switching between the models until they come up with the best plan. When your environment can't reproduce it, you need to stop, obviously start trusting environment, but you need to, and again, with mobile apps, it's harder for anything server related or application or website.
Speaker 1:It's easy because all of your users experience the same and are using the same application. For mobile apps, it's kind of different because of, different network speeds, different OS versions, different phone models, so there are many different variables that can come into play, but whenever you can control the environment, control the environment and if you're going to end up deploying as Docker image, if it's a web application or something like that, do the development inside that Docker image and not work on your machine then transfer it into a Docker image. Make sure that you control the environment as much as possible. Ask, number six, ask for observed evidence. As I mentioned, video proof, logs, I don't want summaries.
Speaker 1:So that's another thing. Make sure that you understand what the agent is telling you. So if you don't understand, go back, explain this to me, write it as prose rather than as code, or if it's easier for you, write it as code rather than prose. Whatever makes you fully understand what the agent has done. Treat both the docs and the PR as completely self contained.
Speaker 1:They need to be able to fully first be fully in sync with what was happening. PR needs to tell the full story of the session of what was going on, what was fixed, why, etc. And anything in the documentation that is no longer true immediately after a PR is merged needs to be updated. The next one goes without saying, make sure that you always keep your code in sync with your team, if you work with a team. So if there is a merge conflict, bring their code into yours and make sure that the combined version works.
Speaker 1:Never do a full, just a simple rewrite. And yeah, whenever the agent cannot verify something, it needs to be very specific. So demand a very specific list of what is truly unverifiable without you as the human jumping in and running your own test. Yeah, I guess that would be it because apparently this has been working for me for the past, I don't know, two, three years now with Agents and more aggressively in the past few months. Someone asked me yesterday and I ran a quick check.
Speaker 1:I realized that I'm personally consuming in my local machine about 300,000,000 tokens every day, which is kind of insane. That's not including the software factory that runs on the server and the teams. Apparently, I don't know if that's normal or not. Feel free to tell me how much you spend, I would love to know. But yeah, that's what I wanted to talk to you about, kind of give you a war story the past, literally from the past few days, and hope you find this useful.
Speaker 1:If you did find it useful, please let me know. And until then, I'll see you on the next episode. Have a great time and have fun coding. See you next time. Cheers.