Subscribe
Share
Share
Embed
We follow the journey of Faith Forster as she creates an AI native tech startup & product.
Welcome back to the Building Out Loud It podcast with me, Randy Silver and
Faith Forster:Thanks Forster.
Randy Silver:And this week, we're joined by a special guest.
Ed Biden:I'm Ed Biden. I'm one of the cofounders of Hustle Badger.
Randy Silver:And, Ed, we know Hustle Badger really well, but do a brief intro. Tell us what Hustle Badger is.
Ed Biden:Hustle Badger teaches super pragmatic skills for product managers and anyone who wants to learn how to build with AI.
Randy Silver:And you've been doing this a lot. I took your course. It's a fantastic course that we definitely highly recommend, but you've also been helping Faith out a little bit behind the scenes as an advisor with the work that she's doing building Discoveree. We wanted to get you in this week to use a special expert to follow-up on some of the questions we came up with last episode. Before we jump into that, Faith, last time we talked, you had been going full steam on lots of feedback.
Randy Silver:You talked to lots of people. How's that been going? Give us a quick update and then we'll jump into everything else.
Faith Forster:I had 20 or so interviews last week. So loads of feedback. I also had people responding to the survey. It's been so useful. So many things I can do to make it that bit better.
Faith Forster:But before I invest too much more into making updates on those features, the main thing is just getting these agents to work really well. The value proposition around driving commercial and customer value and outcomes is really resonating. The other clear message is that they need to trust the data behind that, and that's really driven by these agents. And so I need to make sure these agents are working really well before I spend too much more time on sort of tweaking around with the features.
Randy Silver:One question about the feedback you get. When I do these things, there's always the temptation when you've had the conversation to say, oh, yes. That's obvious. Let me go tweak that and fix that right away. And then there's the, I need to wait and see what priority is this?
Randy Silver:How many people are actually saying this? Do I get contradictory feedback about this? Where do you fall on that spectrum? How do you deal with temptation?
Faith Forster:Oh, that is such a great point to raise. The first couple of people that gave me feedback, I jumped straight in and made those changes. I kind of regretted doing it. And I waited and got more feedback and really thought more about that feature, and I'm going to completely change it. It has really helped me step back and rethink how these features should work in general, how it fits together rather than just tweaking here and there.
Randy Silver:Ed, you've got a Hustle Badger is a nascent company itself. How have you dealt with that before we jump into the whole agent thing?
Ed Biden:I've definitely made that same mistake of jumping into building things too early. I think if you're the one who's doing the user research and also doing the building, that's very easy because you're just so excited to be able to go and change things and make it better. As soon as you hear that there is something you can make better, you just dive straight in. At the moment, I'm not having as much time to work on the Hustle Badger site itself as I'd like to. When I do, the things that I build are a little bit more considered, which is nice, but I prefer to be spending more time building myself.
Randy Silver:It's so nice to hear you both say that because, you know, we're we're experienced product leaders. We talk about these things all the time. We know about recency bias. We know of what we're supposed to do, but, yeah, the temptation is real. So, Faith, you were saying the whole application of Discoveree that you're doing is underpinned by a whole army of agents, and it's not working perfectly at this point.
Randy Silver:So what's going on? Where are you?
Faith Forster:So I have done the proper logging, but the way that Replit shows that logging is not very easy. It's also not proactive in notifying you when something's gone wrong. So, for example, I ran out of credits on Perplexity, and so the agents just weren't working, and I didn't know I wasn't notified by Perplexity or my own application. But there's still quite a number of issues around JSON passing coming through from some of the LLMs. There's some really practical debugging stuff I need to work through.
Faith Forster:It's really not been easy to identify and work out how to go about fixing it. I still haven't got OpenAI to give any output any of the requests that have gone through, and I'm not sure why. There is some work to be done.
Randy Silver:Last time we talked, you were starting to think about moving off of Replit and starting to play with a more professional development environment, an IDE. How's that gone?
Faith Forster:I moved the code base over to AntiGravity, so I was able to do that reasonably easily. I've been syncing it with GitHub regularly anyway, so all the code is sort of backed up in GitHub, even though Replit was the primary interface I was using. I was able to then download that straight into AnyGravity. I also had to give it access to the databases and to the API keys I was using, but that didn't take too long. I then tried to get AnyGravity to connect to there's sort of two main eval tools you can use, or there's two leading ones, I think, Arise and Langfuse.
Faith Forster:So I connected into Arise, and I just couldn't get it to work. I couldn't get the keys to authenticate properly. I tried every way possible. Used all the documentation. I then gave up and went back to Replit and thought, could I connect this into Replit and see if that works?
Faith Forster:So I tried doing that again. I hit the same roadblock. The API keys just wouldn't authenticate. So I think there's an issue on Arise's side. Unfortunately, don't have any customer support.
Faith Forster:There's not much I can do without contacting their sales team, which I don't think will help. So I will try Langfuse, but I'm also thinking at this point, the main issue is just debugging. I do think I need to rethink the way the agents work in general, which I'll get into a little bit later.
Randy Silver:We've both been under your course that we talked about earlier, and it's a great course. You start with very basic stuff when you're using Lovable, a very what you see, what you get type approach to things. Then you move up through Replit, through Cursor as a full IDE. You do a whole bunch of different stuff. Talk us through these different tools and these different approaches and why as you start to get to the point where faith is, you need to upgrade.
Ed Biden:I usually think about there being four main categories of AI tools that you can use to build things. Firstly, you've got your your general LLM site, Chatty, BT and Claude, and, of course, they're great for just answering questions, probably expects, helping you plan things. And I assume everyone kind of knows what they're like. The second plat type I think about is agent or workflow builders. So this is things like NAN, Zapier, Relay.
Ed Biden:They're really good for visualizing a back end and designing that in a drag and drop way without coding, but they don't have any kind of front end. So they're really good for internal processes or plugging into an existing front end, but they're not gonna do everything that say Faith wants to do, where she wants to have a front end that customers can play with. The third group then, what I call the prototyping tools. So that is things like Lovable, Replit. I mean, there's so many of these now.
Ed Biden:You've got v zero, BOLT, Base 44. They turn a prompt into a website, which is just awesome. Right? And as well as doing that, they also deploy it to the Internet and make it kind of available for everyone. And I think that's a really important distinction because it's something that the IDEs don't do.
Ed Biden:So another thing to be aware of about the prototyping tools is they are all browser based. They all work in the cloud. So then I can kind of contrast them with the IDEs, which is the fourth category. So IDE is an integrated development environment. That is where professional developers will write their code.
Ed Biden:That is based on your machine. So you have to download that as a separate application to write the code, and you're gonna have all the files or the code files themselves on your machine as well. Now, the benefits of doing that is that they generally give you finer grain control over what you're building. And the prototyping tools, a lot of people find they get to a certain point where there's just so much complexity in what they're building that those prototyping tools can't handle it. And they start creating duplicate code, they start deleting code, So they start kind of like throwing up bugs that you think you've already fixed or creating bugs in unrelated areas.
Ed Biden:And that obviously like really slows down how fast you can build and makes it very difficult to progress. So that's why I recommend that at some point, if you're really doing this seriously, you're going to be developing a product for multiple users, not just yourself, not just internally, you're going be doing that over a kind of long period. You should graduate to an IDE. That does come with its own complexities. They're not as easy or as intuitive as the prototyping tools.
Ed Biden:Actually just viewing what you've built locally is a lot more complicated and then deploying is a lot more complicated as well. So there is additional complexity there. The trade off is the functionality you get. I should also say that as of February 2026, there's kind of a fifth category of tool emerging, which is probably going to replace most or perhaps even all of those existing four categories. And these are the tools where Codecs and Anthropix CoWork, where they have got light coding and some kind of LLM all in one place.
Ed Biden:That I think is gonna take over quite a lot of both development tasks, but also not development tasks.
Randy Silver:Sorry, one more question about this. I've been hearing lately about swarms of agents. Where does that come into things?
Ed Biden:My understanding of the swarms of agents, that one of the reasons this kind of fifth category is emerging is these are effectively like lightweight IDEs, and they've got a number of bits of functionality that lots of people are finding useful, not just agents. So one of those bits of functionality is that they will spawn up sort of sub agents to do particular tasks. So when you put a prompt in, it won't just be like one thread that you're having a conversation in like you are in a chatbot like ChatGPT. You'll give coworker prompt, and it will then spawn up different agents to go and do different things. So that obviously can increase your capacity significantly, but has its own challenges with context switching and planning and kinda like keeping everything going in the right direction.
Randy Silver:Faith was talking earlier. I think Evals is one of those things that's not widely well understood yet, and it's not something that's necessarily easy to use. So just give us the potted history. What is an Eval and how how is Faith gonna make it work?
Ed Biden:Evals are basically QA for non deterministic systems. So as soon as you start working with AI, you're making an API call to OpenAI or Anthropic, and you're getting back some kind of response like that because that is non deterministic. So same input will give you different responses. That is obviously a challenge to make sure that you're actually getting what you want out of it. And so Evals very simply is just a set of techniques that allows you to, systematically measure and quantify the results that you're getting so that you know that your product is getting better even though it's not deterministic.
Randy Silver:Is this fundamentally different from test driven or behavior driven development? It sounds like an applied way of doing that from my old school brain.
Ed Biden:Yeah. An engineer or someone that had actually studied computer science would probably tell you lots of reasons that they are different. But for me, when I'm thinking about building products, yeah, this is this is basically the same.
Randy Silver:So so functionally, what are you doing? What makes a good eval? Why is this such a, you know, considered a hot skill and a new thing?
Ed Biden:I think it's considered a hot skill because everyone's building AI products, and testing those is a little bit, you know, trickier and different from testing a non AI deterministic system. I think the other thing is that at the heart of Evals is human judgment. And so it puts product really at the center of that testing process because someone's got to decide what good is. And when you're doing some traditional code, deterministic coding, you can write that down in a set of logical statements and anyone can test it. So that can be dedicated QA, that can be a developer, it can be a PM.
Ed Biden:When you're dealing again with AI systems, Evals are all based around what someone's judgment about like, yes, this is good and no, this other answer is not good and you can explain why. So to give you I I usually teach Evals in a few different stages, but basically what you're doing is you're starting out with gathering traces, you know, sort of like what is going on in the system, principle of observability. If you're going to kind of work on any software system and fix bugs in it, you've got to understand or be able to see what's going on at each step in the workflow so you can see where the problems are happening. And AI systems are no different, but instead of calling it a log, we call it a trace. So a trace is like a single user interaction between a human and an agent and that will include the input from the human, it'll include the final output from the agent, but it'll also include all the intermediate steps that, you know, if the agent's calling tools or making API calls and, you know, metadata about when they did it and what browser they're on and all that sort stuff.
Ed Biden:Once you've got that trace or a series of traces rather, is to run through and see where is the output, not what you expect or not as good as you want it to be. You can start doing that in a fairly unstructured way, just making quick notes and you want to go through, I don't know, 50, a 100 traces, something like this. Once you've got that many notes, so that might take, you know, an hour or two, then you can cluster those into groups. And you can focus on saying that, okay, well, if I had to kind of cluster these into four, five groups, what would be the main buckets of failure that I'm seeing? So once you've got to that stage, even like as soon as you've got the traces and you you can review what the traces are, you're gonna start seeing things where like, this definitely doesn't make sense.
Ed Biden:You're immediately gonna see ways that you can improve the product. And then when you cluster them and you're starting to go from purely qualitative data into that kind of semi quantitative data, you can say like, well, actually I can see that this is the biggest cluster and therefore you know how to start prioritizing where you want to fix things. And you should be able to see also where in the process it's going wrong. Is it a particular if you've got multiple agents, is that particular agent that's failing? Is it the interaction between them?
Ed Biden:Whatever it is. So from that point then, you've got your clusters, you know roughly how big they are, What you can do, and this is the sort of the the Evals bit, is you can design tests that then look for those failure modes in an automated basis. Benefit of doing that is that anytime you make a change to your agents, to your products, you can run all of your traces as test cases. You can see the outputs again, and then you can have your eval score whether they whether that failure mode still exists or not. Right?
Ed Biden:And that allows you then to kind of in an automated way, kind of review lots of test cases over multiple versions. So it gives scalability to the whole process of building AI systems.
Faith Forster:What I'm getting at the moment, the logs is just like a series of events. It's just this failed because the JSON passed it. This did this. This did this. That's quite different from in an Evals tool.
Faith Forster:It almost creates a map where you can actually see the inputs, the different steps that were taken, and then the output. And so it gives you a much clearer picture of what's happened, where something's gone wrong and why. It's very, very different from just a list of stuff that you need to sort through and work out what's going on.
Ed Biden:What an eval tool is doing is it's effectively, it's taking all of those logs, but it's then putting it in a lightweight visual structure. So you can see this API call was called by this agent and you can see what order everything happened in. So you see both the hierarchy and the order of things being flown through. And that is just much, much easier for people to have a look at and be like, oh, okay, it was all fine till this step, rather than look at just like a wall of like machine output logs and figure out which one has gone wrong.
Faith Forster:And where in the process. It's that visibility of the context of what's happening and where things are going wrong. That's what they refer to as observability. So you're able to understand exactly what's happening and where and be able to isolate much quicker what you need to fix.
Randy Silver:Let's talk a little bit more about getting good at Evals. The process that you talked about, philosophically, it's not new. It's still UAT and diagnosis and synthesis and then prioritization, although it's done slightly differently. How do you get good at writing evals in this case, though? It's a different technical approach, but same mindset as what I'm hearing.
Ed Biden:Yeah. Absolutely. I mean, I think there are some emerging guidelines for how you do these exact How many traces do you need to review before you start clustering your failure modes? Well, somewhere between one and two hundred in a test environment. How do you get your LLM judge to mark whether a trace passes or not?
Ed Biden:Well, you should probably design it as a binary pass fail criteria rather than as a grade. So there are some sort of like fairly well worn guidelines like that. The other thing is like, like any skill, you just need to go and make an agent and build some Evals and run the process and get used to how it works. And I think this is one of the benefits of working in say one of the more agentic workflow platforms, know, something like NAN, is that you do get those traces out of the box. You you can see every run, you can see where in the system things worked or failed, the inputs and outputs of each individual node.
Ed Biden:And they also have some nice but basic kind of ways of inputting test cases and then quantifying your Evals as well, if you want to take that step.
Randy Silver:I heard some horror stories early on in AI assisted development where you found out that the system was essentially writing things that would pass test cases or faking the tests so that it would report success even though it clearly wasn't doing it. Is this something where we're just transposing that to a higher degree of difficulty, or is this meant to completely get rid of that as a problem?
Ed Biden:It should get rid of the problem. Right? I've definitely seen that sort of thing before. Right? Where you you try and connect up a lovable front end to some kind of back end, and it's like, here's the response.
Ed Biden:You're like, that is definitely not what is coming back from my agent, and it's just faking the whole response from the webhook. Yeah. LLMs will do that, and the the remedy to that is having that observability so you can actually see what did you make the API call? Does that API call work? What response was there?
Ed Biden:Was that response actually shown in the front end? You've gotta have that sort of ability to dive in and see exactly what is happening at any particular point.
Randy Silver:And is this something that you do once as a discrete testing phase or do you have this running in production all the time as kind of like a dashboard?
Ed Biden:What a lot of people are doing is they are doing both kind of like time based and then version based testing. They are time based as in every week, they will take a set of traces that happened that week from live data, and they will run it through their Evals just to see, like, is user behavior changing? Because we're all using AI in different ways. Right? As as we get kind of more used to working with it, that kind of tells you, like, is human behavior changing of the test cases or the failure modes we're seeing changing?
Ed Biden:That gives you kind of like the live data on humans. And then the other thing they're doing is saying like, three days every week I'm testing my system, but I'm doing that on a set of test cases that I have standardized and I'm going to keep standard from, you know, version to version so I can track between versions whether the system is improving or not.
Randy Silver:So Faith, how's it going for you? What questions do you have today?
Faith Forster:I still need to get the setup working properly. So the next step for me is to connect in LingFuse see if I can get that working. If not, I'm going to default back to just trying to get a better dashboard out of Replit so I can at least get a better understanding of what's going on in the logs. There's been quite a bit of change very recently, like in the last week or so, where there is now a new standard for skills for agents. So this is kind of almost the input, all the resources or approaches or even like domain knowledge that an agent should be leveraging when it does its job.
Faith Forster:That's now set up as a standard framework. So that's something that I'm going to look at using as a way of seeing if that will improve the performance of the agents. And then there's also now a more established concept around sub agents and agents, where you've got different agents performing tasks, but you have planning agents sitting across them and sort of orchestrating that towards a given goal. And so I'm also going to look at the way I've structured the agents to work and see if I can, because it is a little bit clunky at the moment. It does feel a little bit too manual, even within the context of the tasks it's doing.
Faith Forster:I have had that feedback from some people. So I'm going to just take a step back and see if I can change the way that it's structured and orchestrated across the different jobs that it's doing and see if I can make that a lot smoother. There is a balance, though, and it's something I'm debating and I've been talking through. What's that balance between full automation and the convenience of that versus the user being in control of what's happening? Where do you draw that line?
Faith Forster:I don't think full automation is the right answer in this context. I think the PMs and the product teams need to be working alongside the agents to work out what are the option needs, what are the problems, how do we solve these problems, have a level of control and input and collaboration as a group. Because a lot of the interaction you have with an LLM isn't by nature individual. It's more of a philosophical question of what's the right balance between the agents doing a lot of the heavy lifting for the teams and the teams being able to then use that context and apply judgment and make good decisions versus the convenience of having the agent do the work for them. That's partly question of comfort for the existing teams so that they're adopting it, but also a bit of forward looking view of where is this heading and what's the right expectation of agent based tools going forward.
Randy Silver:So Faith, what's the next step? What are you going to be doing over the next week?
Faith Forster:I've got a few more interviews to finish off today, and then I've purposely kept my calendar free so I can just get back into the building side and focus on not just these agents, but then updating the features. So my plan at this point, I've got some companies who have said they're very interested in being part of beta. I've also got a very experienced CTPO who works with piggyback companies, which is part of my initial target market, I hadn't met before, but spoke to as part of AlphaTesting. And he said himself that there's huge potential here to make this work really well for the owned companies who are very focused on driving commercial outcomes, but doing that across a portfolio of products. So I will also work with him to shape that portfolio view.
Faith Forster:I think there's probably two to three weeks of work just to get this back into the build mode and get this working, get the product working the way we want it to for the next round.
Randy Silver:But from the technical side of that discussion, last time we talked to you, you just downloaded Google's anti gravity tool. You were playing with that. You didn't love it, as I understand it. So you're playing with REPL. What's the next step in sticking with that level of tool versus a full on IDE?
Randy Silver:What didn't work for you?
Faith Forster:Yeah, it is definitely a leap. An IDE is a level above. I found it uncomfortable. The reason I found it uncomfortable is in Replit, when you've put in a prompt and you ask it to build something, you can actually see the logic and the reasoning. It talks that out in natural language.
Faith Forster:And so you can see the way it's thinking, the assumptions it's making, what it's doing. Particularly the assumptions it's making, that's where I found a lot of the issues or incorrect implementations of things. As I've read through it and said, Oh, no, we can't do that because I'm like, Well, no, that's not correct. You should be able to do that. The intention was that we should be set up in that way.
Faith Forster:I feel like I've got a little bit more control over what the platform is doing. With AnyGravity, it doesn't do that natural language. It shares that thinking in code. And so I'm not able to understand it very easily. And it's making me quite uncomfortable.
Faith Forster:I also think for the next round, at least, I need to make some fairly major changes to the features. Integrative is better at, or IDE tools are better at very specific focused changes. I want to make fairly major changes to the features, so I think that would be easier for me to do a replet than it would be in IDE. And then I come back to an IDE to do the fine tuning afterwards. So at the moment, I think I'm going to keep progressing with Replit and then maybe look to use the IDE.
Faith Forster:But I hope that at some point in the not too distant future, I'll have an engineer who can help me with that because it is a little bit beyond what I'm comfortable with.
Randy Silver:Fantastic. It sounds really interesting and look forward to hearing what's going on next week.