WEBVTT 00:00:00.000 --> 00:01:32.000 What does it take for two people to build software that might once have required an entire engineering team? Our guest today has gone from leading large organizations at Google to building a startup with one co-founder. He and I discuss how they work with coding agents and the engineering decisions that still require human attention. Welcome to Agents and Engineers. I'm your host, Dan Gerlanc. Today's guest is Allen Hutchison. Allen is a co-founder and CTO of Vycari, where he builds AI agents that help with everyday tasks. He previously spent 21 years at Google, most recently as a distinguished engineer at Google DeepMind working on AI agents and developer tools. He was part of the founding team behind the Gemini CLI and built the Antigravity SDK. In this episode, we also discuss. How Vycari uses design docs, nightly audits, and agents that pick up follow-on issues to keep development moving. Why Allen is building agents into familiar interfaces like calendars and to-do lists. How he chooses models for different tasks, including using more capable models to write skills for less expensive ones. How Allen chooses models for different tasks, including using more capable models to write skills for less capable and less expensive ones. Enjoy the show. 00:01:32.000 --> 00:01:35.000 Allen, thanks for joining us today. 00:01:35.000 --> 00:01:36.000 I'm happy to be here. 00:01:36.000 --> 00:01:55.000 So I know you had worked on the Gemini CLI, so you've got a little bit of a longer background probably than a lot of folks have in this world. Tell me a bit about your journey into agentic engineering and AI. 00:01:55.000 --> 00:03:37.000 Well the journey into AI is actually a very long one. I was an cognitive science major at Indiana University in the mid nineties and my concentration was artificial intelligence. of course that was like during the the drought, the ML drought, and it was a very long time before we got to a point where machine learning capabilities matched what we hoped at the time it could be. but throughout my career I worked in a variety of machine learning kinds of roles. I was in the Google Maps group for a long time, about a decade where we used machine learning extensively for creating semantic data out of imagery and other sensors. But you know, as the AI boom started to really happen just after the pandemic. I moved into a group at Google called Core ML, which was an attempt at the company to concentrate all of our machine learning infrastructure into a single team. And during that time I was responsible for our Compilers and runtime and ML performance teams and things like that. For a time I led the entire organization. this was you know right around the same time that ChatGPT first came out, and Google was consolidating DeepMind and the Google Brain Research Team to start really focusing on Gemini. We had a product at that point, or soon after that, called BARD, which was a precursor to Gemini. And Yeah. 00:03:37.000 --> 00:03:40.000 Yeah, I remember Bard in the early days. 00:03:40.000 --> 00:06:36.000 And and you know, I just really felt like this was a great time to get back into hands-on engineering. So I had been senior director for a long time, managing, you know, very large teams. And I felt like this was an opportunity to move back into engineering overall. And so I started moving my career into a new organization that we called the AI Developer Team in Google Cloud. That team was eventually And yeah, around that time I became a distinguished engineer in DeepMind. But the agentic engineering piece of this really started with Gemini CLI. So there was an engineer on my team named Taylor Mullen who had been tasked with playing around with new ideas for developer tooling with AI. at the time that organization included Kaggle and Colab. and the Gemini APIs and we were trying to think about like what we could do for developers to help with incorporating AI into their workflows and into their tooling. Claude Code came out right around the same time we were playing with some of these ideas. And you know, Taylor wrote the first prototype and we were all just tickled that we could write a React app in a TUI. and w started to think about what we could do with that. And we, you know, there were two or three of us working on this initially, putting together the initial agentic loop, building the initial tools, and We launched that product I think I think we started working on it in March and we launched it in June, if somewhere around that time frame. about two years ago. And that was just a very, very rapid development project for Google overall. And we took advantage of the speed to be able to, you know, get something really interesting out there and have our point of view out there. I think one of the things that was really interesting with that project is we made it open source, which was a decision that I felt really strongly about because I wanted I wanted developers to have a example of an actually complex AI application. Like a lot of the examples that you see on frontier model websites or in developer documentation is by necessity quite brief. You know, it's like here's how you make your model calls and or here's a toy example on function calling or something like that. And I wanted people to see a much more fleshed out and complicated agentic AI system that they could use for inspiration for learning, those kinds of things. 00:06:36.000 --> 00:06:46.000 That makes sense, yeah. It's like that meme where it's like draw the owl and there's two lines and then the rest of the owl in the next frame. 00:06:46.000 --> 00:07:38.000 Yeah, and it's it's it was a great ride. Like I it was a lot of fun to develop in the open. I was just looking at Gemini CLI at the repo the other day and it's still at, you know, a hundred thousand stars on GitHub, which was just a meteoric rise for a product like that. I it was very challenging, you know. Building an agentic AI product meant that you also had a lot of agentic AI pull requests. and issues filed and we were probably not prepared for the the volume of third party contributions that people wanted to make to that product. so it was very difficult in the first few months in particular to steer it with a you know with a product vision and trying to trying to build that product out. But it was a it was a lot of fun. 00:07:38.000 --> 00:07:55.000 that can be challenging when you're operating within what are the goals of the product internally, even if it's open source and there's lots of folks who want to take it in other potential directions. 00:07:55.000 --> 00:09:17.000 Yeah, for sure. I mean, like, everybody had an opinion and by that time the team had grown quite a bit, so we probably had I don't know, dozens to low twenties of people working on it full time and Yeah, there was a there was a lot to learn as well. Like that yeah, two years ago agentic loops were still pretty exotic and models hadn't really been trained for function calling to any particular degree. and you know, the the patterns that we've started to see in these kinds of tools today were really being discovered. Yeah just the invention of like auto mode that Claude Code has today is a huge step forward because people get so tired of approving tool calls in these products. And yeah, this f the first versions of all of these agentic development to products, even including like Cursor and the the the IDE based ones, you know, had you like saying yes to every single tool call and it was so draining and it was so tiring that people kind of got this like got these blinders on about saying yes to tool calls and they would just say yes to everything or they would run in YOLO mode or something else and then we've seen, you know, all kinds of stories about what happened to people's systems from that. 00:09:17.000 --> 00:09:34.000 Yeah, actually the first agentic software product I built was actually an open source tool to allow you to configure er to configure your accepts f and rejects for Claude Code by parsing the bash commands. 00:09:34.000 --> 00:09:44.000 And parsing bash commands is like very tricky. Yeah, there's all kinds of exotic syntax in bash that the model knows and you don't. And 00:09:44.000 --> 00:09:45.000 Yeah. 00:09:45.000 --> 00:09:53.000 like trying to patch the or to parse those accurately and understand the intent of the command is is really, really hard. 00:09:53.000 --> 00:10:17.000 Yeah, for fortun fortunately I was able to pull in a library. There was a Go library that has an in that parses bash, but there were still other decisions to be made like what do you do with Here Docs or things like that? Or I mean at some point the model can embed whatever it wants unless you're fairly restrictive. 00:10:17.000 --> 00:10:44.000 Yeah, I did something similar in Gemini C L I right after I made the transition to an IC role and I built a policy infrastructure in Gemini CLI based on like YAML files for policy. that worked fairly well because it gave users a lot of control on what kinds of commands or what subsets of commands that they were comfortable with allowing the agent to run. we we included both like prefix matching and 00:10:44.000 --> 00:10:50.000 Yeah. 00:10:44.000 --> 00:10:50.000 regular expressions, which you know gave you a lot of flexibility, but still bash commands are hard. 00:10:50.000 --> 00:11:08.000 Yeah. Yeah, exactly. Like you wanna be you don't wanna say, allow all commands that start with nice because that's meaningless, right? You n you need to start after that, but there's a lot of permutations of things that can get in 00:11:08.000 --> 00:11:21.000 Mm-hmm. 00:11:08.000 --> 00:11:21.000 there. This is slightly unrelated, but did you said you were working on some of the compiler infrastructure. Correct me if I'm wrong, but did you work on XLA? Okay. 00:11:21.000 --> 00:12:11.000 I managed the team that built XLA. I didn't contribute to it directly though. There was a great set of really talented engineers who worked on XLA and XLA is a really interesting product because, you know, it tries to create really good compiled ML programs for a wide variety of infrastructure from CPUs and GPUs of multiple manufacturers to our internal TPU product. And that was that was a really neat set of things to work on. I I wish I had had more opportunities to get deeper into that particular technology, but at the time my management responsibilities were so large that I just didn't have the the time Yeah. 00:12:11.000 --> 00:12:21.000 Yeah, I feel like a lot of folks don't realize that all that work actually goes into powering a lot of the training and inference that gets done. Yeah, 00:12:21.000 --> 00:12:23.000 Yeah. 00:12:23.000 --> 00:12:42.000 I I s similarly managed a team that we continue to me maintain the fork of Theano which was one of the original symbolic differentiation compilation tools. 00:12:42.000 --> 00:13:52.000 Yeah, it was it was intro you know, one of the things that was really interesting about the time at XLA is this is when developers started to really take off with custom kernels for ML development. And, you know, we'd started to see a lot of publications on particular custom kernels or particular techniques for training in particular that would accelerate the training, you know, propagation passes and those kinds of things. And the the demand internally for custom kernels for TPUs started to grow really significantly. we worked alongside the team that publishes the Jax Harness to try to create, you know, infrastructure for developers to bring their own custom kernels to TPUs and to be able to support custom kernels in CUDA or in you know, NVIDIA's GPU infrastructure and in AMD's GPU infrastructure. So there were a lot of challenges for that team, just because of the broad base of hardware and languages that they needed to try to support. 00:13:52.000 --> 00:14:09.000 Yeah, those are those are tough problems for sure. So changing subject a little bit, tell me a little bit about what you're building at your new company that you started, Vycari 00:14:09.000 --> 00:16:52.000 Yeah, Vycari is a an agentic AI company that is focused on building agents that everybody can use. one of the things that I think a lot about are the user interfaces that we have for for agentic AI today. Most user interfaces for interacting with a model or interacting with an agent happen through some kind of chat interface. And That makes sense given where you know Transformers developed from and how those models work, but isn't necessarily the right interface for most people to interact with an agent. And so we've spent a lot of time thinking about how do we build really familiar tooling for everyday users to interact with agents and to help them to accomplish tasks throughout their day, whether that's scheduling or meal planning or you know, helping with fitness goals or just taking notes or managing to do lists, any of those kinds of tasks, all things that agents are really, really good at. But if If you don't understand the capabilities of the agent and the way it calls tools, you often have a lot of trouble composing a prompt that will get the agent to do the thing you want. What we do instead is we build user experiences around these kinds of mini apps that have the agent deeply embedded within them and allow users to interact with something that feels a lot more familiar and a lot more comfortable than the plain chat box. Now we do have a chat box as well because I think like you have to. But My hope is that over the next few years the industry moves to something more maybe something that acknowledges the kind of last fifty years of human-computer interaction research that we've had and starts to build back towards more guided experiences in structured applications that you know blend the capabilities of agentic AI with the you know structured data and deterministic systems that we've used for a really long time. time. And so that's what we're hoping to do with our agent in at Vycari. And I think that yeah, the first agent we'll release is a personal assistant and aimed at consumers and then from there we're looking at other experiences that we can bring to bear with the same kind of philosophy and the same ideas. 00:16:52.000 --> 00:17:04.000 So do you think of it kind of like where the iPhone really brought app specific functionality to different activities people were doing, but with agents, 00:17:04.000 --> 00:17:05.000 Yeah, I think 00:17:05.000 --> 00:18:40.000 yeah. 00:17:05.000 --> 00:18:40.000 yeah, I I think that's a good analogy. I think that yeah, today we look at the agent as the product in a lot of cases, across the industry. And I I think over time we will look at the agent as a component of a product. Yeah, I I would like people to think of our assistant as an assistant product and whether or not it's implemented with an AI agent. should be you know irrelevant to most users. It it should carry about as much weight as the programming language we use to implement it. It's a detail. I think that agents are great for handling kind of non-deterministic input and you know making decisions guided by your development infrastructure and your prompts to make good decisions at runtime that you couldn't you couldn't have foreseen as a developer. You know, agents are great at that. Agents are great at like search tasks. You know, the user comes in with a vague prompt and you give the agent, you know, several corpora to search over and give it the tools to do that and put it in a loop and let it run until it has enough data and you get really, really good results back. Much better than just a standard like nearest neighbor search over a vector database or something like that. So, you know, I love having agents built into products, but I don't as much love the agent as a product. 00:18:40.000 --> 00:19:06.000 Yeah, I think some of it is I mean, especially on the consumer side, but also within the developer side as well in different cases, people want their problem solved. It's not really like you said, what the programming languages don't really care so much. It's more is it doing this thing that I want it to do in a way that I can understand. 00:19:06.000 --> 00:21:02.000 Yeah. And I do think, you know, the the chat interface is is well suited for a lot of development tasks. You you know, it takes the place of conversations you would have already had as a developer to build whatever piece of software you're actually building. especially if you take the approach of you know, working with an agent to flesh out a design doc, for example, and then, you know, having another agent implement that design doc. It's a very natural pattern from software engineering. But that that doesn't mean that it's also the right pattern for like figuring out what groceries you need to buy next week or, you know, making sure that you remember a bill is due two weeks from now, or, you know, figuring out what meals to cook this week or which parent is gonna go to open house for which kids school and those kinds of problems. Like I think those are handled much better in those kinds of interfaces that we're really familiar with, like calendar interfaces or to do lists and, you know, adding kind of agentic magic to the background of that turns those things into really, really smart tools. And that's something I'm just super excited about because, you know, I would love to, you know, love to have a very smart calendar that understands my priorities, that understands when I wanna schedule meetings, who I wanna prioritize, you know, how I want to guard my time, like all of those kinds of things. or a really smart to do list, for example, that understands again, understands my priorities, my projects, my goals, and then, you know, moves not only keeps track of my to do list items, but moves them forward when it can. and I think those are the kinds of tools that we want to bring to people from from our agent at Vycari. 00:21:02.000 --> 00:21:18.000 Yeah, I think of calendar booking type tools and I know some people like them, but personally there was never really a consistent set of time that I wanted to say, hey, here's available. It probably depended on well what's going on that week. 00:21:18.000 --> 00:21:20.000 Yeah. 00:21:20.000 --> 00:21:21.000 That seems like something 00:21:20.000 --> 00:21:21.000 And 00:21:21.000 --> 00:21:23.000 you could address with an agent that understands 00:21:23.000 --> 00:21:24.000 Mm-hmm. 00:21:24.000 --> 00:21:28.000 the greater context of what's going on, like how much energy 00:21:28.000 --> 00:21:31.000 It's 00:21:28.000 --> 00:21:31.000 do you need, things like that. 00:21:31.000 --> 00:22:34.000 It's a great example because we actually have a calendar booking tool as part of the product suite. And it's the first time where I felt like the calendar booking tool did the did the work of a human scheduler. because what I tell the the instructions I give to the tool are, you know, I want to have this many, no more than this many meetings a day. I'd prefer them to be in these time slots. I want to make sure that when you do schedule something, I have buffer time on either side. Like all of those kinds of things that you would tell a human assistant, those are all instructions that I've given to the agent. And then when someone wants a meeting, I just send them a link and it presents times that are consistent with what I've already expressed to the agent. and of course a really nice thing in our system is that if you know two users of our system need to meet, the agent can just handle that on its own. And the the people never have to get involved because it's got access to everybody's calendar and preferences. Yeah. My agent will call your agent. 00:22:34.000 --> 00:23:06.000 Yes, our our people will talk to your people. So in building this, I mean you've been running large engineering orgs from before. How does it feel again to be writing code or how how have you how does it change now working with agents in a small startup versus what you've been doing for a decent amount of time? 00:23:06.000 --> 00:27:25.000 I think the changes are a couple. a f a few years ago I started shifting the way I was spending my time to focus more and more on my individual contributions and my own coding projects, whether internally at Google with things like Gemini CLI or externally with open source projects that I support. And The thing that I found most interesting with this particular exercise, now Vycari has only been around officially for like a month. but we've landed well over a thousand PRs in that time and built a very sophisticated set of infrastructure and systems with just two people. And so part of this is velocity. the fact that you can move really quickly, that you can have you know multiple agents working in the code base at any given time. I think part of it is the freedom of decision making. at any large company situation, you know, you have a lot of stakeholders. You have a lot of people who have input or have decision making authority over any given project. And here, you know, it's just two of us and it's very easy for us to get on the same page really quickly. I think agentic AI has given us the third leg of a trend that's been happening. I'm kind of mangling an analogy here, but if you think about software development over the last couple of decades, there are a couple of trends that have really made it easy to start a new software company. the first being the availability of open source developer tooling. You know, moving from closed source to open source tooling has been a huge boon. regardless of the language that you write in today, you're almost certainly using open source tooling for your development process. the second is the development and prevalence of cloud infrastructure. So you no longer have to make those capital expenditures to to buy machines or to stand them up in data centers and those kinds of things. It's very easy to get up and off the ground with serverless information. infrastructure or with virtual private servers or whatever. And then the third thing, which is what has just happened, is the advent of agentic development tools. And those three things together mean that anyone with a really good idea can bring that idea from concept to production with a very low budget and you know very small number of people, if any additional people, by using these three these three trends and how they've come to come to bear in the industry. I think it's a really exciting time to be starting companies because of the very low capital requirements, the the ease of getting started, you know, the ability to, you know, essentially bring on an infinite number of software engineers through through the agentic tooling. The the pitfalls are still that you do need, you know, some people who understand systems design and architecture to drive those agents, especially if you want a system that's gonna scale over time. you need to have the right checks and balances in your S DLC to ensure that the the agents aren't running amok and and then you need to you know make the right decisions about your underlying cloud infrastructure and where you're going to where you're gonna develop your plu your your product to you know, which infrastructure you're gonna use to develop your product. But it you can bring all that together a lot easier than you could even five years ago. now, you know, a startup like ours with two people, we can make really tangible progress. whereas you know, five years ago doing a similar kind of product suite, I would have expected to probably hire, you know, twenty to fifty engineers just to you know, just to get things ready to go to market. 00:27:25.000 --> 00:27:38.000 Yeah, that that whole side of things is pretty amazing the amount of leverage an individual developer can have without the associated scaling cost. 00:27:38.000 --> 00:28:47.000 Yeah. And the yeah, the thing that I find really interesting here is and I have no idea how this is gonna gonna shape up in the end, but the costs of running even frontier model agents, while they can be quite high from a token perspective or an API perspective, they're still not nearly as high as hiring human developers. even if you're looking at like offshore or contract developers, you know, your your agents on a per hour per output basis come in, you know, orders of magnitude cheaper, even if you're paying like frontier model API costs. I I estimate that you know the code that I've written for Vycari at API costs is somewhere around fifteen thousand dollars. I mean that's a rough estimate based on like C C usage, comparing API prices for Claude Code, but but that's that's saying something. You know, I couldn't I couldn't hire engineers for that money. 00:28:47.000 --> 00:28:49.000 Not not for not for very long or very many. 00:28:49.000 --> 00:29:04.000 No, not for not for more than a few days. And and and of course, like with the labs subsidizing token costs through subscriptions, like the cost is much, much lower than than that in reality. 00:29:04.000 --> 00:29:09.000 Yeah, that that you could get on a single 00:29:09.000 --> 00:29:29.000 Yeah. And, you know, I do run out of tokens almost every every week. so I feel like I'm really getting the value out of that subscription. But you know, if you supplement that with subscriptions from other labs, then you end up in a pretty good pretty good place. 00:29:29.000 --> 00:29:52.000 What kind of strategies are you using to to make that agent development process as efficient as possible? Like how are you approaching SDLC or determining when the agents can start new issues, merge things in? 00:29:52.000 --> 00:30:46.000 So we have a couple of standing rules in the repository that go along with this. first, you know, we you don't implement anything without a design doc. And so this is a little bit of human training and a little bit of prompting in our agent md files, but you know, any major feature requires a planning document before it can be implemented. that process is great because you talk to the agent about what you want to build, you go back and forth over a document that, you know, humans can read and understand. And you end up with a pretty good artifact that we actually check into the repo. so we keep our planning documents in the repo next to the code in a planning directory that then stands as this like durable documentation about the implementation of the repo. 00:30:46.000 --> 00:30:54.000 Is there a specific skill or process you have for building those or just kind of a back and forth with the agent? 00:30:54.000 --> 00:35:35.000 It's mostly back and forth with the agent. you know, it starts with talking about what you want to build, having it go look at the the repo itself. It r they often read a lot of the planning docs, understand how this particular idea fits with everything else. I think that the bigger piece, the the one that that we've had to be really careful about is once you commit that planning doc, it can go out of date. And so one of the things that we do is a nightly job that compares the claims in our planning documents to the implementations and the code. And this process looks at like all of the changes from the previous 24 hours. It finds the planning docs that might be impacted by those changes, and then spot checks other planning docs along the way and it and it submits a PR in the morning with the updates to all of the planning docs to keep them in. to keep them consistent with the implementations in the code base. That's really, really important because a lot of the time, you know, I'll point an agent at the planning docs and say, you know, this here's here's here's what we're planning to build or here's the documentation for what we've built. And it gives the agent a lot of context without causing it to, you know run through a large code base on its own and infer that context from from the code itself. the other thing that we end up doing is we've we've our approach to using issues has changed quite a bit. humans file very few issues in our repository anymore. if we're going to plan and build a big feature, we don't set it up as an epic, you know, with like 25 issues for each step or anything like that. We have one issue that says like implement this planning document. the way we close the circle with this is that During the course of a PR review, inevitably new things will come up. And the reviewer will flag, hey, this is inconsistent or you should handle it this way or whatever. And the agents are instructed to like leave no follow-ups behind. So a follow-up can't just be an acknowledgement in a PR comment. It has to have an issue filed for it as well. And so the vast majority of the issues filed in the repo are follow-ups from PRs. The nice thing about that is the agent that files that issue has all of the context for why it's needed and what code it should change and how it should be built. And a lot of the time we don't build it right at the moment because it would double an already very large PR or yes, send us off on a side quest that we don't really want to take on at that particular moment. And so the agent will file an issue with a design already in it. And we have another process that runs continuously 24 hours a day that picks up those pre-planned issues and implements them And that just results in new PRs that land in the repo. And that's that process has worked really well for us. I'd say a third of our PRs are probably landed, you know, completely autonomously this way with an agent that picks up an issue that's ready to go, it implements it, it submits a PR. another agent reviews the PR, they go back and forth, which is always fascinating to watch. and when that PR is is you know ready to submit, that's the point where I look at it and I read through the discussion, I read through some of the code. Mostly there's certain files that I really want to keep an eye on and I watch for those in particular. You know, database schema changes, changes to our agentic loop, changes to our tools. And, you know, I review that and then I submit the PRs. I don't let the agents submit their own PRs. I feel like that's not quite a place where I'm willing to trust them yet. but Yeah, we've had really good success here. we use greptile for code reviews and I think it's really important to have like a separate system with its own context and its its own perspective, so to speak, you know, doing the the PR reviews. But it's fascinating to watch them argue with each other. like Claude kind of gets out of gets bent out of shape by Greptile a lot of the time, which is really funny. And then it'll come back to me and it'll say, like, I just don't agree with this greptile and you need to do something about it. 00:35:35.000 --> 00:35:53.000 Yeah, I've had some battles with locally I've used Roborev, which basically anytime there's a commit, it will kick off a adversarial review and sometimes I'll put it in goal mode and be like, all right, go back and forth till this is done. 00:35:53.000 --> 00:36:13.000 Greptile gives you a s a confidence score, a mergeability score on on a scale of one to five. and for our agentic PRs, I give them a goal of getting Greptile to five out of five. And that gives me a lot of confidence that that PR is ready and safe to merge into the code base. 00:36:13.000 --> 00:36:24.000 And for your processes that are running continuously, do you have those running on your local machine or are you using some kind of cloud agents to kick those off? 00:36:24.000 --> 00:37:52.000 We have a combination. we're moving more and more towards hosting our own sandboxed agents to do these tasks. But right now the split is whether or not the agent needs access to to internal data and to our database. so if we look at the first set, which is mostly what I've been talking about here, which is our like auto dev pipeline, that runs primarily in claud. cloud instances and on a timer. So I r I run the pipeline every I forget every fifteen or twenty minutes and its job is to just move a state machine one tick forward. we store state mostly on the issues so that I don't have to keep any external data. And this is all codified in a set of skills that I wrote for running the system. for sis for agents that need access to data, so for example every night we have a reflection agent that runs across user trajectories in our system. And what it's looking for there is like was the user's request successfully handled? Did anything go wrong in the tool calling? Were there problems? Yeah, there's a rubric that it scores the trajectories against. And it's surf for the product itself, 00:37:52.000 --> 00:37:55.000 This is for the ba the product. Yeah. 00:37:55.000 --> 00:38:40.000 yeah. And it surfaces bugs It surfaces it actually f a lot of the time it finds bugs and writes failing tests for it and creates PRs with failing tests for me. it will surface issues as well. It it'll keep track kind of longitudinally of issues that we're watching or experiments that we're running like in shadow mode, and it'll gather data for those and append it to a GitHub issue that represents that particular experiment. But, you know, those tasks really need access to the trajectory data. And so they run in a sandbox that's inside our infrastructure rather than running in in anthropics infrastructure. 00:38:40.000 --> 00:38:48.000 And going back a little with to the planning documents, so these are basically every day getting updated to the state 00:38:48.000 --> 00:38:49.000 Yeah. 00:38:49.000 --> 00:38:51.000 of the repo as it exists. 00:38:51.000 --> 00:39:57.000 Yeah, every night we do this consistency pass. it's one of several like AI audits that I run on the code base every night. Yeah, so we audit design docs and planning docs, we audit architectural principles, things like you know, agents are really excited to repeat themselves and so you know, we have one of the audits we do is for like dry, yeah. Let's extract interfaces and you know, co consolidate ideas or patterns into a single library and things like that. We run automated, you know, type checking and linting and all those kinds of things. And then we have agents that look at the output of those. and then On a less regular basis we run a variety of other like security audits and things like that that again often produce issues or sometimes produce PRs directly. but that gives us kind of the pipeline of stuff that goes into that auto development process. 00:39:57.000 --> 00:40:05.000 How do you distinguish between design and planning docs? Is or or is that a distinction you make? 00:40:05.000 --> 00:40:25.000 It's it's not, I kinda use the two terms interchangeably. I I think potentially a a planning doc hasn't been implemented yet and a design doc has. but beyond that the it's one artifact that includes the technical design as well as the product goals for that particular feature. 00:40:25.000 --> 00:40:33.000 So you'll mix when you're doing a planning document, you'll generally mix the technical and product designs together. 00:40:33.000 --> 00:42:24.000 I think it's really important to do that because you want to give the agent the context of why it's building this particular feature. Yeah, if I'm building a you know calendar system, I want the agent to understand that I'm building a calendar system, not that I'm building a tool that accesses a calendar. and that product definition inside the planning doc helps with that a lot, I think. I don't have any data that that measures this right now, but like at least anecdotally I feel like it gives the agent a lot more context on what we're trying to achieve as well as like how we're going to achieve it. And that comes in because like the way I use these agents in particular is I work with like a a tech lead persona agent. And that's usually a I've been using Fable a lot for this. I've been experimenting with Astra since it came out recently for this as well. But I interface with that agent and then instruct it to design with the next tier down Opus or Sol and implement with the next tier down Sonnet or Luna. And so most of the time when I'm working on the code base, I have a single instance of this tech lead agent that I work with. And then I'll have subagents, I'll have dozens of subagents running on a bunch of different threads, either planning new features. or implementing parts of those plans. And so the standard that I give to the agent is, you know, design for the lower tier model for implementation. and then I go back to the mid-tier model, the Opus and Sol models for for code reviews. 00:42:24.000 --> 00:42:31.000 And do you run these all locally generally as well, or you're y using cloud agents? 00:42:31.000 --> 00:43:14.000 No, I run a lot of this locally right now. because I have a really fast local machine and we have like one of the big challenges we have right now is I hold a high a high ratchet bar for code coverage by unit testing and our our system is covered at like ninety two percent right now. but that means that we've got even on a Well, like on a normal GitHub worker our our C I C D takes fifteen minutes, which is way too long. And on my local machine I can do it in about four and so I I take advantage of that. 00:43:14.000 --> 00:43:31.000 I believe that. Yeah, there is high overhead to CI C D. I mean I personally still like running CI ha and having an independent check. I know 00:43:31.000 --> 00:43:32.000 Mm-hmm. 00:43:32.000 --> 00:43:47.000 some people are like or there is some I know I think Steve Yeaggy had written recently about rethinking CI CD. Do you ha do you have any thoughts on that? 00:43:47.000 --> 00:46:17.000 I I do and I think I think it's necessary, because agents can mess up or the people driving agents can mess up. Yeah, right now the agents are only as good as their input and if you don't tell the agent to check on something or if the code base is big enough that it doesn't fit in context, then you know, the agent is gonna miss an issue and that's why we have unit testing. now most of the time the the unit testing that I run catches issues before it gets into the repo. And so running running it in GitHub workers is is rarely finding new issues, but it's it's it stands as that gatekeeper for for the the quality of the code and the integration at the code base level. I think the big thing that we're starting to realize is that the monorepo isn't serving us very well. And so we've started to s to really aggressively split microservices out of our monorepo into their own repos with their own CIs. Because first of all, that forces the agents to have like a more defined interface in between these different components, which is very helpful. that's a thing that they can understand really well and it's a very small amount of data that they have to load in their context. They don't have to worry necessarily about the implementation of another service that they're interacting with. secondly, it allows us to isolate unit testing on a per on a per microservice basis, which means that the CI runs a lot faster and the the agent is better able to reason about that particular part of the code base if it's independent and in its own repository. and then third, by doing this we're allowing the humans to iterate a lot faster. yeah, nothing's worse than getting stuck like forty minutes behind in a merge queue when you're trying to, you know, see what you're doing live in the product. And like our product, we publish a new image every f every on every merge, and we update the prod version of the product every 15 minutes. And so, you know, a feature can go from idea to product in half an hour, assuming that your CI is fast enough. Yeah. 00:46:17.000 --> 00:46:36.000 Yeah, I think it's interesting because personally I always historically I would always say that the reason to use different repos is largely your CI. Because unless you're using a very specialized monorepo tool or you're somewhere like Google or Meta that 00:46:36.000 --> 00:46:37.000 Mm-hmm. 00:46:37.000 --> 00:46:49.000 has their own Custom tooling that can only run your tests. At some point, it becomes kind of overwhelming for the humans to figure out what's going on in the one. 00:46:49.000 --> 00:48:06.000 Well and I I I believe really firmly that you wanna you want your repo to be as you Not in a pejorative sense, but you want it to be as generic as possible. Like I want my Python codebase to look like every other open source Python code base that exists. Because that's the overwhelming amount of data in the training set for these models. The moment I start building like really fancy, you know, monorepo like CI segmentation GitHub actions, like I could do that, I think, pretty sure. but like it starts to skew away from what the agents know really, really well. And then you're going to have to spend a lot of time either building guardrails around that system or, you know, fighting with the agent about like, no, I really want you to do it this way when it tries to do something in a different way. And so in a lot of cases, I think the best thing to do is to just break it up into different repos, make them look just like a a repo does anywhere else in its training set and then build like tools and skills to make it easy to extract functionality from from your your repo as it grows and move it into a new one as a microservice and set that repo up so it works just like all the other ones do. 00:48:06.000 --> 00:48:26.000 Are you working off of more modern? I don't know if more modern is the correct word, but I know a lot of companies today are choosing to go like Cloudflare Vercel versus the more traditional AWS or GCP cloud offerings. 00:48:26.000 --> 00:49:53.000 no, we do right now for almost all of our stuff, it's Python running on VPSs. And so we've got a full Linux machine under the covers. We get from Hostinger. they're pretty cheap and easy to spin up. They've got two locations in the US, and so we get some geographic diversity and network diversity from that. and they work really well for us. I think because we're using Python, obviously, like Vercel wasn't really an option. you know, I think Cloudflare workers are potentially an option, but we've we've stuck really to really traditional kind of you know, it's a it's a it's a SQL database, it's a Postgres database, it's you know, Python middleware and Python web interfaces and That's working well for us. And again, it's like this is stuff that's in that's very heavily represented in the training sets for the models. I think that over time, yeah, we may need to move to like serverless runtimes for parts of the system and and it's architected in such a way that we could make those decisions down the line. I think the the biggest risk will be what we do with Postgres over time, but we aren't anywhere near a point where Postgres can't handle the traffic that we're gonna throw at it. Like the first step is just throw it on a really big machine. 00:49:53.000 --> 00:50:07.000 Yeah, I remember seeing Modal was running some massive part of their system on Postgres just for a very long time. like individual function calls or something like that. so 00:50:07.000 --> 00:50:17.000 Yeah, and I mean there's there's this new breed of like cloud blade cloud based Postgres service providers. There's things like Supabase or Neon, I think it's called Neon. There's a there's a few. 00:50:17.000 --> 00:50:20.000 There you go, there's Neon, yeah. 00:50:20.000 --> 00:50:54.000 And like Google Cloud has one. I assume AWS has one. And so there are ways to like scale that Postgres database, but you know, going from going from that to, you know, a a decentralized, you know, big table style database isn't gonna be a huge leap for for these these projects. And and frankly it's not a huge leap for the models. Like you can redesign your data model and then migrate your data at a later point. 00:50:54.000 --> 00:51:14.000 I like that you have a traditional stack of Python and Postgres on VPSs. I feel like a lot of times people make things more this isn't necessarily an agent thing specific, but more complex than it needs to be given the problems they're trying to solve. 00:51:14.000 --> 00:51:47.000 It's you know, it's something that I was a little sensitive to as I started talking to folks about leaving Google and doing a startup. Yeah, one of the one of the things people would tease me about is like, you're you're never gonna ship something because you're gonna worry about serving it at a million QPS. And I wanted to make sure that the infrastructure didn't you know, followed that, you ain't gonna need it. principle. Like if I get to a point where I need to worry about a million QPS on this platform, like I what I welcome that challenge. Like that's gonna be awesome. 00:51:47.000 --> 00:51:53.000 Right. Yeah, it's like you then then you have pl you should have plenty of resources to deal with the problem. 00:51:53.000 --> 00:52:57.000 Yeah, yeah. And you know, I think the thing that hasn't necessarily made its way into everyone's consciousness yet is that as good as the agents are at building software, they're even better at refactoring software. like three weeks into our our our build phase, we had, you know, hundreds of thousands of lines of code and I decided w we needed to refactor it. And I did it in a weekend with Fable, you know, s shifting around, you know, forty thousand line PRs, which I never would have even attempted manually or with the pre agentic tooling. And so, you know, when it comes time to refactor for, you know, a a different kind of database structure or it comes time to refactor because we want to pull a microservice out of the main repo, you know, the the nice thing is that the agents can handle that really, really well. 00:52:57.000 --> 00:53:20.000 Yeah, there was someone I chatted with on the show recently where they were talking about how they had changed the domain model but the database tables still had the old name and a customer had asked about that and historically that probably would be something that wouldn't be super high high priority to change, but 00:53:20.000 --> 00:53:22.000 Mm-hmm. Yeah. 00:53:22.000 --> 00:53:43.000 With agents you can go and do it. So your code quality, I feel like, and the semantics of your system can really match in a way that you would be hard to justify ever historically if you've got all these other priorities. Is that the thing you're gonna do? Probably not. 00:53:43.000 --> 00:54:18.000 Yeah, exactly. Like it's a lot easier to keep naming consistent. It's a lot easier to keep you know, your database columns consistent with the intent that the the column supports. Like yeah, I I a hundred percent agree. I did a set of migrations yesterday that I never I never would have done without, you know, an agentic tool to help me land those those migrations. Like it was it was trivial to do with an agent and it would have been days to weeks to do without. 00:54:18.000 --> 00:54:28.000 Have you run into any cases where the agents build something that you didn't want them to build? 00:54:28.000 --> 00:57:24.000 I don't know that I've run into cases where they've built things I didn't want them to build. I ha a a funny story with the the auto dev pipeline is that I when I first started building out this concept, I was using it on a open source repository that I manage. It's a an agentic plugin for the Obsidian note taking app. and I Like I said earlier, I manage the state machine through labels on GitHub issues. And that's fine. I'm the only maintainer in the product. I'm the only one who can put labels on issues. but what I didn't anticipate is that the agent would reply to any issue that was filed, not just the ones that I filed. And so when users filed issues in the repo, the agent would come back with like design proposals. but with a little bit of tweaking, it actually turned out to be a real boon because the agents would answer user questions much better than I would and would ask good questions for additional data points from the users to figure out like why it was not working for them. So that was like unexpected but I decided not to turn it off in the end because I felt like the the advantages were higher than the disadvantages. although I think it probably put some people off to see like an immediate response to their bug that says like, you know, created by Claude or something at the bottom. I think outside of that, I've I've had some cases in Yeah, yeah. Well, I've had some cases in that repo in particular where it's building in TypeScript for an electron app as a plug-in where it'll go and build things that don't use the standard plugin infrastructure and and I had to build some like really significant skills to, you know, guide the agents into like don't don't don't reinvent something that's already in the standard plug in library. like just use the thing that's already there instead. I did well this is one that I saw yesterday. I saw a one-liner I have a script where an agent needs to validate a URL. And I I had an automated bug fix come through yesterday and I looked at the diff and the way the agent was validating the URL was a w a Python one-liner. So it loaded the entire Python interpretator interpreter to just validate a URL. but we already had like JQ running in the system and it's like that's a much better solution, like just use JQ. And and that's actually what the automated bug fix was, but like that was a case where like something slipped by and I didn't notice it and it would have been would have been better to do in a different way. 00:57:24.000 --> 00:57:49.000 Yeah, I th I forget where I saw this, but I think someone had gone through all of the agent MDs or skills in various open source repos, and it was interesting to see the things where they're say, don't do this, which I imagine is issues that had repeatedly come up in the repo for some specific agent behavior. 00:57:49.000 --> 00:58:04.000 Yeah, yeah, and I think, you know, we've got some of those don't do this kind of things in our agents MD as well. for us a lot of it is, you know, be very careful about database migrations. 00:58:04.000 --> 00:58:16.000 Yeah, the the database part is still, I think, my experience has been having knowledge of proper database design still comes in handy. 00:58:16.000 --> 00:59:19.000 Mm-hmm. Yeah, I think I think there's that. I think there's the the agents when you're working in parallel you start to have problems with like schema changes and the ancestry of schema changes. Like you'll have three different agents to try to branch off of the same version of the database and you end up having to build tooling in GitHub to find find and assign the ancestry changes to a a news you know, a new version of your schema, for example. And those kinds of things are Those are challenges that you wouldn't have as often with a human development team just because they would probably talk to each other about, hey, I'm gonna change the database. There'd probably be, you know, a discussion or a Slack channel or something where they agree what the new schema's gonna look like and the agents just kinda throw schema changes in as the features are developed. 00:59:19.000 --> 00:59:37.000 How do you think about both for coding and within a product or within the product sense, which models you're gonna use? And I know you'd written a blog post about looking at different size models and different 00:59:37.000 --> 00:59:38.000 Yeah. 00:59:38.000 --> 00:59:43.000 times to reach to different places. 00:59:43.000 --> 01:01:35.000 So in our product, we use in a lot of cases we use the least expensive models that are available. our orchestrator in our agent for our product is built on the the low-tier models across the frontier lab. So it's Gemini Flash Lite, it's Claude Haiku, and it's GPT Luna. And we actually run a portion of our traffic on each of those agents and we score the the trajectories independently of each other to understand like which one's doing best at which time and which one's the cheapest for us to run. but like these agents are or these these models, these I like to call daily driver models because you know, calling them like low tier models just seems insulting. they're very, very good. Yeah, the they're typically equivalent to the the Flash or Sonnet level from a year ago. you know, they're often equivalent to the Frontier level from like two years ago. And so, you know, if you're thinking about an agent that's going to orchestrate personal productivity, it's often all you need. And, you know, we've benchmarked these daily driver models for the tasks that we have. And in most cases they do quite well in our harness against the more expensive tiers of of models, whether it's Flash or Pro or Yeah, s sonnet or or opus or Terra or Sol. Wish we had better, more consistent names, but 01:01:35.000 --> 01:01:38.000 Yeah, right. 01:01:38.000 --> 01:04:39.000 the other thing about this though that's really interesting is it gives us a path to doing fine tuning with open weights models. a lot of the open weights models are kind of in that same tier as the this daily driver tier of models and some of the ones that have come out recently in that tier, yeah, the new Gemma models or some of the the Flash like GLM models and and DeepSeek models, like there's a really interesting opportunity there for us to fine-tune those to the orchestration tasks that we have in our products and then you know, take on serving inference those models on our own. It gives us some nice isolation from you know what the what decisions the labs might make around their their model offerings in the future. But for right now, like these are also very inexpensive. Yeah, I think at this point we've run 200 million tokens through this set of models and I think we've spent, I don't know, like $60 or something like that. so we can support a lot of users for you know a really small amount of money on that class of models. Now there are some cases where you want to make sure that you're using a more a more intelligent model or more capable model. And one of the ones that stands out for us is writing skills. So part of what our orchestrator does is if you ask the model to do something, it will write a skill to do it for you. And if you use Luna to write a skill for Luna, you don't get nearly as good a result as if you use Terra to write a skill for Luna. or you use Gemini Flash to write a skill for Flash Lite, or and it doesn't have to be the same model family, but like if you want to go one tier up to write the procedure for the model one tier down. That results for us in just much more comprehensive skills that are easier for the orchestrator to follow. and it is really interesting because we tell the the mid tier model to write for that that lower tier model or that daily driver model. We tell it like you're writing a skill for this other model that has this level of of intelligence and to keep that in mind when you're writing the skill. and so it leads to skills that are a little bit more pedantic and a little bit more prescriptive. But that's exactly what we want with the the tier of model that we're using day to day. So we do that in the product. And then there are cases where we'll ramp up to either the mid tier or the high tier frontier models when it's called for. But we want to be really careful about cost because we want to offer our users a inexpensive product. Like my goal is to offer users a five dollar a month subscription. And you know, we're not going to be able to do that if we're running on, you know, Fable. So 01:04:39.000 --> 01:04:42.000 Yeah. Yeah. All right, yeah. 01:04:42.000 --> 01:06:02.000 And you know, and that's also why we're focused on the kinds of tasks we're focused on right now. You know, we're not doing a lot of computer use tasks because those are incredibly expensive. I think in the blog post for Astra, you know, OpenAI was talking about how they've brought down the cost of computer use to and I don't have the post in front of me, but I think it was like ten dollars at a sixty percent success rate. and like cool, that's great. But like day to day for a consumer, what are they willing to spend ten dollars and only have a sixty percent chance that it's going to pass? I don't I don't know. And I I think that we're better off trying to make tool calls deterministic and integrations deterministic so that s lighter weight models can use them effectively. So that's in our product side. On the development side, yeah, I talked about this a little bit earlier, but I tend to work with the whole range of models. and I use the frontier tier, the Fables and the Astras to interact with me and to orchestrate a whole bunch of sub agents that are working at planning and implementation. 01:06:02.000 --> 01:06:16.000 This is one thing I often like to end on with folks, but is what's something you're proud of that you built that might not be immediately apparent to everyone? 01:06:16.000 --> 01:07:21.000 I mean one of the things that I'm really proud of with this new product is the agent harness itself. This is, gosh, the fourth or fifth agent harness that I've implemented over the last couple of years. And it's taken like everything I've learned from all the way back to Gemini CLI days to the agent I implemented in Obsidian to the work I did on like the Antigravity SDK to going into this particular product and building an agent that's specifically geared towards these daily driver models. So there I mean there are a lot of things in this this harness that you know check for bad output or yeah work around specific bugs in some of these models or, you know, a wide variety of techniques that I'm using here to make sure that this agent performs really well with the model that we're targeted at. And I'm I'm super proud of that and like no one's ever gonna see it. 01:07:21.000 --> 01:07:50.000 Mm-hmm. Well, Allen, thanks so much for joining us today. It was great to have you and hear a bit about the journey as well as what you're working on now. Also all the links to Vycari as well as things we chatted about will be in the episode notes, so folks should be able to find them after that. But thanks again. It's great to have you. 01:07:50.000 --> 01:08:15.000 Thank you. This was a lot of fun.