Subscribe
Share
Share
Embed
We follow the journey of Faith Forster as she creates an AI native tech startup & product.
Welcome back to Building Out Loud with me, Randy Silver. And, Faith, for anyone who hasn't seen this week after week that we've been doing this, we're following you on your journey to build a new startup, doing it with AI as your co pilot on all this. Last week when we finished chatting, you were about to send things out to 60 people to do a trial, give you some feedback. How's that going?
Faith Forster:It went out. It took me longer than I expected to get it out because it was just hard work getting these agents to actually do their job, which we'll talk about today. I've had a number of calls with people and had some really useful feedback, not just on the product, but actually on all the considerations around like selling it in and implementing it and how it places or displaces other tools in their stack.
RandySilver:Okay. Well, let's leave that there this week because we wanna hear a ton about it, but we're gonna come back to that next episode. This week, we're gonna dig under the covers a bit because as you said, it took you a little bit longer than you had hoped, and you've been complaining for the last couple of weeks about developing these agents. And I've played around with building things in AI, but I haven't developed my own agents yet. So I'm fascinated.
RandySilver:I'm sure other people are. So why don't you give us a look under the covers, tell us about what it's been like developing agents and give us a little bit of a demo.
Faith Forster:It's been frustrating. I do have a moment. I messaged you during the week. When I actually got the agent to do the job, it was a mix of air punching euphoria.
RandySilver:The GIF you sent me was not air punching, it was falling down dead.
Faith Forster:There's definitely been some explaining to the mix. And then also, just sending a good laugh at myself. Does this mean I would make a good engineer or a bad one? Don't know. But it was quite satisfying when it actually worked, but oh my God, it took a lot.
Faith Forster:I had to completely re engineer the way the whole platform works, which is why I thought it'd be really interesting to get into that today. This is Discoveree. I'm going to talk through one particular example, which I think would be quite useful to share. In settings, you can actually see all the agents and change the prompts. It can also now change the LLM.
Faith Forster:So I'll talk a little bit about that as well. So one of the examples where I've built an agent to do the heavy lifting is to gather all the information on what is changing very, very quickly with your competitors. So you can stay ahead of that. Actually, there's another agent in the onboarding process where you put your product in your website. It then searches all the review on comparison sites to find where users have compared your products to another product and gathers those as competitors.
Faith Forster:And then it also looks at other products that have a similar functionality or are serving a similar audience and it captures those as adjacent products. And that's all user validated. So you can say yes or no, on whether or not those are the right platforms and answer those. What we really wanted to do is on a very regular basis, be scanning and checking what's changed with those competitors. And this is something we need to be aware of.
Faith Forster:When I originally set it up, I just had one agent that was the competitor agent and it in theory was supposed to pull this information together on what's happening with your competitors. Literally wasn't working. It wasn't loading. It would just hang. And I knew to get agents to work well, you need them to do a very, very narrow job and then set up a series of agents and then get them talking to each other.
Faith Forster:And so I started breaking down what I was wanting that one agent to do into what became, I think, eight or nine agents doing the job. So if you take this as an example, Approval Max is one of the products that was adjacent to text. So we now have an agent who pulls their website and their help center. So all it does is find those links and populate those here. We then have another agent that pulls a summary.
Faith Forster:So it looks at the product's website. It also does a quick check on review sites. It gives a summary of kind of what this product does and its key differentiators. It also identifies which countries they're actively marketing to. We then have a separate agent that then also looks for the customer segments that they're targeting.
Faith Forster:So that's already three agents. We then have this news and updates section, product releases, new integrations, many market updates. I had just put this into two agents in settings. Now we've got a page with sources on that page. We have trusted sources for news publications, blogs.
Faith Forster:And then we've also got a separate section for trusted sources for the review and comparison platforms that are relevant to this particular product as well. So I set up two agents to actually find that list. Then I've got a separate agent who's actually then checking those platforms on a regular basis to see if there's any new announcements or changes. So you can see that that didn't work. So something's still not quite right with that agent.
Faith Forster:We've then got a pricing section. So this has been one of the areas that's been really quite frustrating actually. So there's a separate pricing agent now. There you go. That worked well.
Faith Forster:So this checks for the actual products own website for their pricing information, but also checks on anything that people have talked about in reviews and also actual news when there's been an announcement. This one is actually quite complex. Like you have different pricing for different markets, different currencies. You've got different pricing for monthly versus annual for different customer segments. Some products sell through marketplaces and there's different pricing there again.
Faith Forster:So it's actually not an easy thing to find all the right information. We've then got key features, separate agent again, we've got integrations. This one is working well now. And then we've got another one that does a summary of the customer reviews. Let's see if this updates properly.
Faith Forster:This is actually a really good example of where the AI did an amazing job with very little input for me. So I went through and said, we want separate sections for each of these. I'm going to have an agent doing its job in each. I didn't say what I wanted. I just said, we wanted a summary of customer feedback and it designed this itself.
Faith Forster:And it's actually done an amazing job in that it's got each of the platforms. It's then given a summary of what customers love about this product and what they don't love about the product. And then some example quotes as well. I didn't ask it to break it up in that way, but it's actually a really great way to present it.
RandySilver:I've had this experience once or twice when I've used AI to help me build things where I knew generally what I wanted. I didn't know the specifics. If I was sitting down as a product manager for the first time and trying to spec this out and write a PRD type thing, I would review of existing things, look for best cases and things like that. Some of these things like this is a solved problem. This is something that there are good patterns for.
RandySilver:It's very easy to take something like this and say, is there anything else I want? Is there anything that's unnecessary? And react to it. But we don't necessarily need to build this stuff from scratch every single time. And it's really nice when there is a good pattern out there that we can just draw from and use as a starting point.
Faith Forster:As I've been going through and building different parts of this platform, there's a couple of examples like this where it's just now that it's very little input from me. Usually, yeah, I'll ask it to do something and then it'll be, you haven't quite got this right, or you've forgotten this bit, or what about this? It just takes a few iterations to get it to what I was expecting. Even though I thought I had made that clear in the prompt. There's definitely been times where it's completely missed things that were in the prompt and just didn't get to it.
Faith Forster:But then there's been other times where it's just like, what? So when this refreshes, it talks to the LLM to get the updates. We can tell users in real time what it's doing. But it's taken quite a few attempts asking it to do that. I actually said to the agent, are you getting this information from the LLM?
Faith Forster:I was like, oh, no, we need to set up a webhook for that. I'm like, okay, so all of the parts of the product are supposedly doing that. What's happening is like, oh, no, we hard coded. So I had to then say, can you please go through and find all the places you've hard coded it and take it out?
RandySilver:Yeah. There's the unit tests or the behavior driven development on this kind of stuff of this is what I'm expecting. And it gives you the appearance of that. It'll satisfy you at the highest level, but it's not actually doing what you want. The best advice I've gotten is treat this like the laziest developer ever.
RandySilver:They've coded something that looks good to pass a demo review, but not necessarily something that actually
Faith Forster:works. Sometimes it just nails it and you're like, amazing. Well done. You absolutely have to triple check everything without a doubt.
RandySilver:It would be nice if giving positive feedback for doing the job right, like actually writing the hook instead of hard coding and things like that resulted consistently good behavior. Like you could train a pet.
Faith Forster:I've been working with the Repla agent now for quite some time, and it's quite noticeable that it does inherently understand the nature of the platform and what we're trying to do in some of its responses. And then at other times, I'll talk this through a little bit more, but one of the things we've had to put a lot of focus on to get the agents to the point where they've started to do their job is making sure that everything's properly logged so that we can see what's actually happening and have that feedback that we can then use to improve it.
RandySilver:One thing I've seen online a few times in terms of advice is a way of working with Replit in this case and giving it some sort of master document about how to work together. Some basic rules like do not hard code things where it should be dynamic.
Faith Forster:I didn't author it, but the Replit platform is pretty good at documentation actually. Rules like don't let the agents hallucinate. It captures that and then documents it properly. I can really clearly see the design system is all set out. The technical spec of the platform is available.
Faith Forster:I literally just copied it to put it on our website. So it is pretty good at documenting this. At the moment, our error rate from the Gemini platform is a really good dashboard. The error rate was at 48%. I managed to get it down to 23%.
Faith Forster:I just checked and it's currently at 19%, which is high. And the OpenAI API also has a clear log that you can see. And not a single request had come back with output. Perplexity doesn't have a log and neither does Claude, but I actually contacted the Perplexity customer support assistant agent. So it came back saying, oh, the logs have to sit on your side and this is how the log should be set up.
Faith Forster:Which was really useful because I could just feed that into the agent and make sure it was actually doing what it's supposed to be doing. Cause I kept saying the agent, are you logging this? And then they're like, yeah, we're logging. Yeah. And then I'd refresh.
Faith Forster:I'm like, well, what just happened? Oh, I can't see anything that happened. Look guidance like that from Perplexity agent was actually quite useful to feed into it saying, please set up all the logs like this, which it did for Perplexity. And I said, well, should we be doing this for all of them? So yeah, you could, but unless you really need to, you shouldn't.
Faith Forster:I'm like, we need to set it up. The thing about having a customer support agent is it's all generic guidance. You ask about what's happening in your account. It's like, oh, I can't tell you what's happening your account, but this is the most likely causes, and this is what you should be doing. But in that case, it was quite useful.
Faith Forster:I thought it's worth sharing this because it really digs into how I've had to sell the agents. I have laid out how each of the agents work, but this is something that can be changed. If you're a platform that gets a lot of feedback on review and comparison sites, then you might want to be checking it every day. But if you're more of an enterprise product, it's not really a thing. You might want to have a monthly check.
Faith Forster:And so that's something that we've made quite flexible so you can make it more relevant to your product because it does have a cost implication. These are each of those agents. You can see them here. Then when I click on them, so these are each of the possible triggers. So this is running in the background that the users can also manually refresh it and it should be working on a schedule, but it looks like it's not there.
Faith Forster:You can also change which agent is doing the job. So perplexity is by far the most robust in terms of doing a thorough web search and finding the right sources and being out of reference though. So I've got this set to plexig mode, but it's also the more expensive one. And so you can change it to Gemini or Jacky TV if you want to. You can also see the prompt so you can actually see what it's been asked to do.
Faith Forster:A user can then tailor this and make it more relevant to their particular organisation. If they've got a specific scenario that they need to either check for or avoid, they can do that. And you can also see how often it's being run and when. Yeah. So you can see there are failed ones.
Faith Forster:Just back to the conversation we were having. So I asked it to create this, the ability to see the prompt and edit it, which it did. And I didn't quite believe it. Last week when I was working on trying to get these agents to work properly, actually went through and effectively audited what the prompts were in each agent, because I had already set it up so that we now had an agent who was finding the trusted sources as a dynamic list. And so I needed each of these agents that were then checking those sources to be referencing that list.
Faith Forster:But a few of them still had sort of effectively hard coded. We're going to check these four platforms. And so I said, this still is not correct. And the agent come back saying, Oh, the code that's running the agent is working properly. It is checking the trusted sources.
Faith Forster:It's just what we can see here on the UI is separate. It hasn't been updated properly. And I was like, we shouldn't be using something that's hard coded instead of referencing this. Like, oh yeah, you're right. And so it's had to completely rearchitect it so that this is the source of truth.
Faith Forster:And then the other thing that it did, which again, took a number of attempts is this ability to switch between LLMs. That was a pretty major effort to rearchitect the whole flow. Now all rooted through a router that takes the request. It then works out which one to send it to. It has a setting by default.
Faith Forster:The user can change it, but also the users can choose to bring their own LLM keys, which is easier from a security perspective. But if they're doing that, they don't necessarily use all of the same LLMs that we've got available here. If in that example, don't have Perplexity, but they do have Gemini or OpenAI, it now recognizes there's not a Perplexity Key for this organization, and it has a preference flow where if perplexity is unavailable, then Gemini is the next best one. So we use Gemini if those keys are available. If they're not available, then I'll use OpenAPI.
RandySilver:This this all sounds like you're getting into the territory of writing evals. Is that something you've been playing around with yet?
Faith Forster:I still haven't researched what an eval is and how that works, but I think the robust logging would now be in place as the first step. So you can see here the cost that's all been properly tracked. So again, this was another thing that we set this up. Some of the agents were going through here, but quite a few of them weren't. And so that was part of that re architecture and it all being routed properly through one channel was making sure it was all properly tracked and monitored for usage and cost.
Faith Forster:I think potentially we surface the performance of each of the LLMs or each of the agents. I'm not sure. At the moment, I'm using it to get it to just work and do the job it's supposed to do. I think that the next step will then probably be making sure there's some sort of continuous review and improvement.
RandySilver:I'm just curious. You sending me gifts and randomly during the week of you ecstatic and exhausted from this. You're talking about this. You're totally into this, but it's also obviously a bit of an obsession in a very positive way. How many hours a day are you spending on this?
RandySilver:How long does it take to do this
Faith Forster:In some ways, surprisingly quick. A day is a long time in terms of the progress that you can make and how much better things can get. The first time I got that competitive agent to actually show the pricing data and show the summary, that was me air punching. It's like, yes. It had only been like, I don't know, two or three days of trying to get it to work, but that's a long time of tweaking the prompt and time feels a little distorted within the context of getting it to work.
Faith Forster:It seems like it should be a simple job. And you never really know when you're going to get there. You keep thinking each improvement you make, this will be it and it's done. Then it's not.
RandySilver:So there's no story points on this at this point. You're not estimating specific I
Faith Forster:was trying to test this to get it out to the alpha users. And I said, yeah, I'll be with you in a couple of days. And then it's like, early next week. That became Wednesday afternoon. You get caught in so many rabbit holes.
RandySilver:Come on. You're working with developers. You always double the estimates, so you need to do that for yourself as well.
Faith Forster:It's also good discipline. I'm setting myself false deadlines to just get it done and get it out. You spend so much time fine tuning this and getting it perfect and then find out that actually it's completely the wrong functionality. So I think it's healthy to have those deadlines.
RandySilver:All right. Let's wrap up this week's episode. What are you gonna be doing this week? What should we talk about next week?
Faith Forster:From the conversations I've had last week, I've got a much better idea of what's valuable, the pricing, the messaging. So we're about to get new website live, which this time I created Lovable, and it is a thousand times better than Squarespace. And then going through all the feedback forms that people have completed and working out what that means in terms of where we take the direction of the product next.
RandySilver:Fantastic. Well, I look forward to catching up with you on that next week.