Subscribe
Share
Share
Embed
We follow the journey of Faith Forster as she creates an AI native tech startup & product.
Hey. We're back with the Building Out Loud podcast with me, Randy Silver, and
Faith Forster:Athe Forster.
Randy Silver:Where every week we talk about Faith's journey building her new app Discoveree and how it's going. And you had a big week last week. What happened?
Faith Forster:I did. It was so exciting. Honestly, I'm still buzzing. So I had lunch with Anton, the founder of Lovable on Friday. It was organized to his investors, and I'd say he's such a genuinely good person.
Faith Forster:It was just so lovely to meet him as a person, but also, wow. What a guy. Super, super impressive.
Randy Silver:Fantastic. And, you know, you've gone past what you can do with Lovable in terms of this, but you've used Lovable quite a bit as well for other things. So what's your experience with it very quickly?
Faith Forster:Lovable is great for the right use case. It's not set up well enough to support a production level app, certainly not a B2B one, but I've used it to build my website. I use it to build my investor pitch deck, which is the second thing that started happening last week. So I think for the right things, it's great. It's a really great tool, but not production level apps just yet.
Randy Silver:Not just yet. Investor pitch deck. That's impressive. Give me two seconds. Where are you on it?
Randy Silver:You've done this before. What's it what's it been like this time?
Faith Forster:After the reaction I got from Alpha and the number of people who signed up, I thought, oh my god, there's something in this. And so I just assumed that it would take six months to raise a round and therefore I need to start thinking about who I want to speak to and what that pitch looks like. I reached out to Dan from Creedom, who were also Lovable's investors. I had met him before and mentioned I'd had this idea a little while ago. And he said, well, when you're ready, let's have a chat.
Faith Forster:Part of my job is to help you get to the point where you're investment ready. And so I got some really great feedback from him on the pitch deck and then shared it with just a small number of investors I know. It's very interesting the difference in reaction. Of them I just didn't hear from at all. I've already got feedback from a couple who I wasn't expecting a reaction so quickly from them, to be honest.
Faith Forster:So their reaction was, You're pre revenue. We only invest post revenue. And I was like, Yes, sorry. I didn't realize I would get the answer so quickly. Give me a month or two and I will be post revenue, which is fine because I'll then go back and have a chat to them.
Faith Forster:But I've got some really useful feedback from them on what I need to be able to demonstrate by that time. I've now reached out to a few others, but more to get their advice on who the right investors are to talk to.
Randy Silver:Okay. So we will probably talk much more about this, about what makes a good deck, how do you go out and find the right people, what's the conversation like, all that because I've never tried to do a raise. I'm really curious about all this. But I also want to get an update with where things are going because you've been getting a whole bunch of feedback. Last time we checked in, you were having a little bit of a tough time with some of the agents and getting them to work.
Randy Silver:And you're all smiley and happy, it's not just that you had a nice lunch last week. Things are going well, aren't they?
Faith Forster:They are going well. I have finally finally cracked the back of getting these agents to it. It has been many weeks in the making. So I have set up the evals tool. So I'm using LangFuse now, which worked exactly how I expected.
Faith Forster:Unlike Arise, I just connected it and all of sudden the traces were going in. Yay! Yeah, yeah, which is how it should work. But actually that wasn't the main reason I got the agents working actually. The main reason that really changed the game is I started working with Claude to understand what I should do with agents.
Faith Forster:There's now agent skills that's been set as a standard. We've got MCP servers, hooks. There's a lot of things about agents that I didn't properly understand. I was finding quite overwhelming, to be honest, and I really didn't know where I should be trying to go with this. And so I uploaded the code files for the agents, what the issues were that I was having, and it made a bunch of recommendations about what I should do differently, which I then uploaded into Replen and said, this is what Claude thinks.
Faith Forster:What do you think?
Randy Silver:How dare you talk to Claude behind my back?
Faith Forster:Some of it's like, yes, that's obvious. We should definitely do that. That's a great win. There was a few other things. It's like, no, Claude's wrong here, and here's the data.
Faith Forster:So all the failure rates and data points that I wasn't able to access and wasn't understanding, it then shared. Made some alternative recommendations. So I was like, yeah, great, build that. It's working so much better. Given that we spent a lot of time talking about this, I thought it'd be worth just showing length views in any Vals platform and how that's different from logs.
Randy Silver:Absolutely. I just want to ask you one question before you bring that all up. So there's always been the debate for product people of how technical do you need to be? Do you need to be able to code? And you're at the point now where you're doing this without a technical co founder.
Randy Silver:You are building everything. And week by week, you're becoming more technical on this. From what it sounds like, if you went under the hood of this and try to pick apart any given subroutine, any given piece of code in it, I'm not sure that that's what you would be able to do, but you're able to talk to the tools in such a way as you would talk to developers. What do you think now? Where do you fall on this opinion?
Randy Silver:Do products people need to or founders? Do you need to be technical or do you need to be technical enough? What does that even mean?
Faith Forster:So this is part of the conversation we had on Friday at the lunch with Anton. There was 12 of us in total. We had a conversation about the future of product development, and that was one of the things that came up. There were quite a number of AI experts, engineers in the room, and they said themselves, It won't be long now. We won't need people who can code.
Faith Forster:One of the founders that was in the room, he said that he's now purposely recruiting graduates who have learnt to code, but are not wedded to the practices of code, I guess, and don't see themselves as artists in the art of code. They're much more open minded about how we leverage the tools that we have now. And he's making a really conscious choice to recruit less experienced people who are more open minded. I think back to your question around how technical do we need to be? You do need to understand the architecture.
Faith Forster:There is absolutely no way I would have gotten to the stage that I'm at if I didn't understand the principles of a multi tenant environment and the data security that's required for a B2B enterprise app. I'm also now getting a lot more comfortable with some of the technical terms. And if I don't understand them, that's okay. I just ask the agent to explain it. There's not a lot that's going on that I don't understand about how the whole product is set up and architected.
Faith Forster:And actually now, with this AI agent skills framework, we're actually documenting that properly as skills MD files so that that knowledge is there. So as the agent is building more things, it's doing it in a consistent way with the framework we've already set up. And that's something that I've checked.
Randy Silver:Not having devs who are wedded to a certain way of doing it Sounds good in the abstract, but production outage or you need to audit something. Yeah. I I don't know what you're gonna do when and even if you've got everything documented, people learning the architecture from scratch and coming in and looking at first time and saying, okay, so this is how it works and this is how it's documented versus this is how it actually works in some cases. Yeah. I don't know.
Randy Silver:It feels like going on a tightrope. It's great until it isn't sort of thing.
Faith Forster:It is something I've been thinking about quite actually. I've reached out a couple of people I met at the lunch to get their input on this a little bit more. And just in terms of thinking about the structure of my team going forward once I raise. When I say I need someone who is more technical, they assume I meet a co founder or a CTO. And I'm like, I'm not sure I do need that at the moment.
Faith Forster:Actually, I need a doer. The most important things at the moment are actually, like, the APIs, data security. Those are the things that I'm worried about. I actually wanted an engineer taking care of. I'm not sure yet.
Faith Forster:I need someone sort of shaping, vibing from a leadership perspective, the technical side.
Randy Silver:It's gonna be fascinating to follow. You promised to show how the Langfuse and the evals were working and how the agents are working. Let's go straight into that.
Faith Forster:I'm gonna show you behind the hood first in So this is what you see behind scenes in Replit. Over here is the prompts. So this is where I have the conversation, the agents that build things. Across the top here, you've got different windows, which you can see the preview of the app, you can see the database, and this is where you can see the logs. This is where I published the app, just an introduction, and you can see the logs of everything that's happening at this time.
Faith Forster:If I click over here, you'll see all the errors basically of anything that's happened recently. So these are from this morning actually. So there's still some JSON passing issues. We've got something going on with the competitor summary. This is all the information I get.
Faith Forster:I get the timestamp. I get the fact that this has come from a user and I just get this really simple statement, which is really hard to decipher most of the time. So this is where I was having a lot of trouble. Like when you hit the tick button here, it's just this wall of red and you're like, oh my God, what's going on? What do I do?
Faith Forster:How do I fix this? It's quite intimidating. I will then share Langfuse. This is what I now get in LangFuse. When I set LangFuse up, I put the API keys in.
Faith Forster:It then started populating all the traces here. I had to specifically ask Replit to also send the input and the output so we can get a bit more information about what was happening. You can see here the time it's taking as well, the number of tokens it's used. If I go down, you can see this one has had an issue. I can click on that.
Faith Forster:This is the Competitive Features Agent. I haven't seen this one yet. So you can see it took almost three minutes. It's failed. Gemini didn't return any text.
Faith Forster:This is happening in quite a few things. It was actually one of the biggest failure rates I was seeing. So I'm going to have to talk to Replit now and work out why that was the case and make sure we've got proper fallback happening. So that's where I've had the biggest benefits with the agents and the fixes that I've made is just having spoken to Claude, got its opinion and then shared that with Replit. I was able to have the conversational prompt for Replit to say, well, no, that's not true.
Faith Forster:Of the different agents, quite a few of them were working quite well. The ones that had the biggest issues, competitor intelligence one was failing 48% of the time. And often it's because Gemini is not feeding the information. So one of the things I requested is verbatim quotes from reviews, example. And Gemini doesn't like that very much.
Faith Forster:It blocks it because it thinks that it might be a copyright issue. And so, that's where it's failing quite a lot. And so, we've changed it now so there's fallback. So, we try Gemini first. If that doesn't work, then we try Perplexity, which is better set up or doing exactly this and having quotes that you can attribute to sources, but it is more expensive.
Faith Forster:And then if that doesn't work, we fall back to OpenAI. So we've got a proper system in place where we are going to get a result.
Randy Silver:What you showed was it's not an overall analysis of here's all the issues that all the errors that are occurring. Here's an understanding, a synthesis. It is just more detail in making each error more understandable. That's the the fundamental thing.
Faith Forster:Yes, exactly. It's just the visibility of what's happening and why it went wrong. Not necessarily giving you the reason why it's gone wrong, but it's giving enough data that I can then feed that back into Replit, into the agent and say, this has happened. Why can we fix it.
Randy Silver:So what are you doing with that information? You're then feeding it back in. Okay. So it's not something that you then sit and do the tracing of the root cause on your own. It's, I'm seeing this.
Randy Silver:Tell me why this might be happening.
Faith Forster:Yeah, exactly. It's not a magic wand. So, rep will often come back and say, like the verbatim quote issue. It's like, Oh, well, if we just don't use quotes and instead give a summary, then we'll avoid this issue. And like, Well, no, because there's a lot of value in it having a verbatim quote.
Faith Forster:And so we need to find a way to make that work. You can't just take what Replit does at surface level because often it will just be finding a quick fix for the problem. You actually have to challenge it and think about how do we solve this in a way that's still creating value for the users.
Randy Silver:Interesting one, but this feels more like a feature than a product in that it feels like this should be something that Replit does. The thing you're using to build should have a way of checking itself. These things are all evolving. This is a step along the way.
Faith Forster:These platforms are not necessarily people are using them to build all sorts of things. So they're not necessarily building an army of agents like I have. They're not necessarily designed to build agents. They're designed to build platforms or tools. I don't know to what extent that's influencing their roadmap, what they're choosing to work on.
Faith Forster:I will say though, a noticeable improvement in the replet agent in the last week, because they are now using Opus 4.6, the LLM. But there is a noticeable improvement in how well it can reason and solve problems and give recommendations, which I think has also helped.
Randy Silver:It is interesting. The way you're talking about though, it's kind of mirroring software development. You know, what I've seen is older school shops where you would have very much write a bunch of specs, it over the wall, you'd have devs do it, then it would go to QA and then it would go back to devs and all that. And then as things developed, you got more towards QA built into the dev process with behavior driven and test driven development and writing the cases first and doing that and you know there's the hallucination problem that it can write something that passes the cases without actually working and we've seen that in the past but as maturity comes you'd think that this goes in that direction and repl dot it even if it's more about building platform than agents at some point these all kind of standardize around the fact that it can build, it can debug, it can feedback, it can do some of the synthesis. I'm just excited and curious to see where this all goes.
Randy Silver:So the agents are working better than they ever have before. Are you gonna show us that too?
Faith Forster:I have also, as you know, been working on all the feedback I've gotten from Alpha to build out the beta version. And I have to say, I am buying
Randy Silver:through it.
Faith Forster:It is incredible. I thought maybe it might take me two to three weeks. I've had maybe two or three days to work on it and I'm a solid way through the feedback and what I need to update in the platform. But actually, the shape it's taking is getting to the point it is far beyond what I thought was possible when I started this project, which was only three months ago. I just couldn't have imagined we would have a platform that would be this sophisticated and do this much on behalf of product teams.
Faith Forster:I'll give you a little sneak preview on some of that because a lot of it's still work in progress. We were previously calling it Competitors and Adjacents and now call it Competitor Intelligence. One of the things I got really positive feedback in Alpha was the data collection to give a lot better insight into your competitor landscape. What I had there was quite static. It was pulling lots of different information together, but it was information that wouldn't change very often.
Faith Forster:And it was just there. So that's still the case. I've still got these competitor profiles. We've got the website link and the help center documentation, which we pull information from. We've also got a threat level so you can choose is this one to watch?
Faith Forster:Is it very competitive or is it a big threat? And that then changes what we do with the feedback which you'll see on the first page we'll go back to. So we've got this sort of summary position which we had before. We've now have also included financial updates on these products. The pricing is now working far better for the agents and you can see we've got the different currencies we can flick between them.
Faith Forster:We've got the key features, the summary. So that's supposed to be linking to the Help Center documentation. I'm still refining that. And we've got customer reviews as well. What we've also got on top of this now is a strategic analysis.
Faith Forster:So there's now an agent that's basically looking at all of that other profile information and making an assessment about what this means for your product. So it's giving its own threat assessment now, and we'll update that accordingly. It's also found some strategic gaps and made some recommendations here.
Randy Silver:Sorry, the strategic gaps, those are gaps in what DEXT has versus Expensify in that case?
Faith Forster:Yes, exactly. We may or may not want to do something about it and actually make suggestions here on whether or not we should act on it. So, high, medium or low. So, we've got those profiles for all our competitors. We've also got it for adjacent products, and we've got that profile for our own product as well.
Faith Forster:We've then got agents that now basically look across all of that information across all of those competitors and give you this summary. It gives a summary of the competitor landscape in general and how our product is performing against other products or how we're positioned against other products. We've also got strategic imperatives and risk areas, which when I pulled this out, was like, oh, this really resonates for Dex. This sounds really true. But actually one of the last Alpha calls I had was just after I built this on Thursday, I think.
Faith Forster:And he just logged in and he was like, this is exactly what I was hoping for. And he read through these risk areas. Like, that's what I would have said. Like, that's pretty spot on. He was also very impressed with how accurate the competitor list itself was now.
Randy Silver:So these are things that are all being dynamically generated. They're not you don't start populating on day one as the onboarding. It's you just say, this is my company. This is my URL, and it starts to pull everything else.
Faith Forster:Yeah, exactly. It's all triggered. And I've actually now set it up so that if, for whatever reason, any of the sections of the profiles have not populated, that it keeps attempting to populate it until there is information there because that impacts this overall assessment. And actually, on that call with that alpha tester, while we were talking, a new competitor popped up. It does also continue to search for any additional new competitors in the background.
Faith Forster:And it was one that he hadn't been aware of. While we were talking, he then clicked through to their website and had a quick look. He's like, wow, no, you're right. This is one we should be aware of. We should be monitoring.
Faith Forster:In the conversation, I said to him, I can't tell you how happy I am. Actually, suggested it was something I've been thinking about anyway. You'll notice we've also changed the layout, the format. It's cleaner. It's more like a markdown fight because that's where things are going with agent based tools.
Faith Forster:But he did say, you know, this is a lot of text. Can you break it up?
Randy Silver:It is looking more like individual slides in a board deck, to be honest. It is this is the type of things you would expect to see discussed not just with the product team, but at a management level. So this is yeah. It's really interesting.
Faith Forster:It's that balance between, like, asking agents to do tasks and pulling information together, and then having agents on top of that who are actually analysing that and giving you advice based on that information. It has just uplifted, I think, the whole proposition and the whole platform to be a genuine strategic partner, as well as doing the doing for you. Other things we've got here now we've got this competitor comparison. So, the different information that we've collected in the profile, you can use that to choose the Y and X axis, but it then will readily change your competitive landscape for you and you can quickly see exactly where you are versus something that compares any of those dimensions. And then down the bottom here, it's got product recommendations.
Faith Forster:So based on all of that, what are the differentiation opportunities? What does it think we should be doing as strategic investments? What are some of the things that competitors are a little ahead of us on? And we might want to invest a little bit in there to get to parity, but it's not necessarily a strategic driver for us. And then what are the known gaps?
Faith Forster:So payroll, HR tools, we know we don't have that. Expensify has that. That's okay.
Randy Silver:What you were originally talking about was that you wanted to be able to assign these directly to individual product teams or align this to the things that they were working on. So is that what the the plus signs next would lead to?
Faith Forster:Any of these recommendations, can either assign it to a product team for them to work on, or we can put it into the product strategy space, which I'm going to update next.
Randy Silver:Fantastic. It's going really fast. What's made the difference for you?
Faith Forster:The reason the builds are moving so fast is I've had so much feedback, and I've had quite a bit of time to reflect on that and think about what that means for the product and what I should do. So I've literally got a Word document where I've just bullet pointed the different changes I want to make, and I'm just working through them. And it's amazingly quick when you've actually thought about it first and you're very considered in the prompts that you're putting in.
Randy Silver:Okay. So it was a very exciting week last week just on the social front, on the meeting front, on the investment front, as well as on the building front. What's on tap for this week? What are we gonna talk about when we check-in next time?
Faith Forster:Yeah. The priority for this week is absolutely getting the platform ready for beta. I've got the first call with one of our beta testers, some of their team, to introduce the product to the team on Thursday. So I want to make sure this is in decent shape for that. I've got more people who have reached out about joining the beta as well, waiting until next week to start sort of talking to those people.
Faith Forster:But yeah, the focus this week is just getting the beta product ready.
Randy Silver:Fantastic. Hey, well, this is moving really fast. I can't wait to see where you are next week. We'll see if we're ready to talk about investment then or maybe the week after. But yeah, I can't wait to see where you get to this week.