Whoa, whoa, whoa, okay, so we're not product engineers. Now we're factory engineers. This is an article written by Zach Lloyd of Warp. It's a memo that he actually sent to Warp employees. So we're gonna read this article together and see what he's talking about. Factory engineers, oh goodness. We're just X engineers now. All right, so this is a memo I shared with the Warp Team about what building Warp needs to look like. It's a combo of context and what I'm hearing from customers, what I think the future of development looks like and how the Warp Team needs to adjust cutting edge. Oh, the future of development looks like. This is, of course, the million dollar question we're always trying to answer. And my answer to that is product engineering, but I'm curious what Zach Lloyd is going to say to his Warp Engineers. This is better with Kent where I teach you durable skills so that we can all get better together and have skills that will last even as AI agents get better. And yeah, okay, we'll see what Warp is doing with factory engineering. We'll see how they define that. So over the past year, we've moved from AI autocomplete to interactive coding agents. Yeah, we have. I haven't looked at like typed in a code editor with autocomplete for six months at least. Over the next six months, the paradigm is going to evolve again to automated development. Okay, self-driving code bases is another thing that I heard. This was from the cursor CEO who talks about this and cursor in general is trying to build kind of a self-driving code base. So I'm on board so far. I think that that, depending on how you define it, that actually could be in the future. And the world of automated development, the job of engineers is no longer to write code. Yeah, we've definitely moving in that direction. It's not even to build products. Now, I'm gonna disagree with you there, but let's see what you mean. It's to build an internal machine, a cloud software factory that builds the products for them. The goal of this factory is to ship an incredible product. Wait, it is to ship an incredible product. Okay, but we're not building product. Okay, but the day-to-day job of the engineer. Okay, so the factory builds the product, the engineer builds the factory. Okay, okay, that's interesting. This reminds me of things that Elon Musk says, like the hard part is the Tesla builds the factory that builds the cars. But like the factory is the hard part. Okay, I am actually, like this matches a lot of what I've been thinking about recently too, where the role of a product engineer is to build the system, wherein the agents can play, and then the agents' output is the product. I'm not sure that you can reasonably get there without having a deep understanding of the product. And so we probably overlap on the way that we're thinking about building products, because really it's the product that matters. And even beyond that, it's not even the product that matters. It's the job to be done. If we're talking about jobs theory, it's actually being hired by users to accomplish whatever job that they have to be done. But I guess like the day-to-day job of the engineers to build that factory, wherein the agents can build the product. So, okay, let's keep going. Success is measured not by how many features and engineers ships, that's a failure metric. And I'm not sure if I would call it a failure metric, but I would say it's not a very useful one. It's measured by the percentage of all changes that are shipped automatically, and at what cost automatically means agents has done all the work, from triaging to specking to implementing and reviewing, verifying monitoring. This is why it's a bad metric, because it's very gameable, especially with AI. You can ship all kinds of features, but that is running the wrong direction really fast as bad. If a change can't yet be shipped automatically and many currently can't, then the goal is semi-automation with an agent doing triage verification, even if a human needs to step in and review, have agents do as much as possible. Okay, I'm on board with this, but I'm a little bit, yeah, like, okay. I'm not totally on board with this necessarily. I think it's okay if it's shipped automatically, like how it's shipped matters much less than what is shipped, and that you're actually building the right thing. So, I would not measure success this way. It's not measured by a percentage of all changes that are shipped automatically. That is maybe like a part of the purview of a product engineer who's building a factory, like how well can we ship those features? And if you're going to kind of silo individuals where you say, okay, you're in charge of the factory, your job is to make it turn out features and bug fixes, whatever, automatically. And then you're in charge of the product, and your job is to make sure that the things that go into that machine are the right things. If you're going to segment things that way, then okay, I can make sense of that. But if you're building the factory without understanding what features are going to be coming into it, then you're going to be unsuccessful at doing that effectively. Like the output, okay, input is the idea, the output is the implementation. That is not going to be as successful if you are not like building or aware of what the product is that you're trying to build. But okay, have agents do as much as possible. I'm with you. It provided that the value offsets the cost, then yeah, that makes sense. The job of every engineer is to improve the efficiency of their team's product factory. Factory efficiency is roughly measured by shipped product over inference costs and human time costs. Yeah, I'm with you there. Companies are going to assess the efficacy of their factories in terms of return on investment. If they spend a dollar on automation, does their business get more? Yeah, exactly what I was just saying. Shipping product is an imperfect measure of value. That's fine for right now. Yes, exactly. So the ROI is really like, it is expensive to create these factories and running a feature through the factory is very expensive because of the agents and everything. This is no different from even a decade or more ago where your team lead was responsible for making this system that the other engineers on the team could, and not just the team lead, the architect as well, but this system where other members of the team could work efficiently in there. Now the difference is that instead of humans working in that system, it's agents and that's fine, but like the ROI doesn't change. How effective is that factory at taking a feature and coming out with an implementation that implements that feature well? And then the real question and the differentiator is now a lot of it is, are you actually solving the real problem? So how good is the input? Garbage in, garbage out kind of thing. Okay, so the days of giving all engineers unlimited token budgets to spend on interactive coding, agents are ending, totally agreed. I think that token efficiency will be a big topic in coming years, even as we end up using more tokens. I think that we actually will be using more tokens in the future, but we also will be more token conscious. If that makes any sense, like we'll be cost conscious and really focused on the ROI. Like, is it worth sending this feature through the product factory to get the implementation on the other side and we'll be measuring how much that costs and how much value we're gonna get out of that. And honestly, again, like that's no different from a decade ago when the product factory was humans working inside of that system. Instead, companies are going to treat software production as a variable cost, not R&D expense. It has been R&D expense for a lot of people. So far, we're just all experimenting. It's going to show up as cogs on the P&L because they are going to want to see the marginal gain of investing more and more in their software factories. Okay, yep, I'm writing this because this is mentally what we need to embrace at warp. Every engineer must stop thinking that they are making changes to our code base and our product directly and instead view everything through the lines of improving our factory. I can kind of get on board with this. I, with some of this, I really don't think that we need to stop thinking that we're making changes to our product. Your product really has to be the center of everything that you're doing. That said, if you are an internal engineer working at a company unlike internal software to speed up your team, maybe you're working on Infra or you're working on internal tools or something, you are a product engineer still. It's just that your users are not the external users who are paying you. Instead, you're actually a cost center and you have to prove your worth by actually taking the thing that you're building and productizing it internally. I worked at PayPal and one of the big challenges that we had there was getting people to use the design system and component library that had been produced. And so I worked on that to make it more useful and actually solve the problems and the pain points of our users who were not external PayPal users, they were internal PayPal developers. And so it's the exact same thing. If you're making a software factory, then you are still a product engineer. It's just that the users of your software factory are whoever is inputting the features that need to happen. So I really think that you can't do that effectively without understanding what the user's needs are and the user's needs are ultimately the product that is being built. And so I take issue with the idea that we're gonna stop thinking about the product and only ever think about improving the factory. I don't think that you can improve the factory if you stop thinking about the product. But okay, maybe there's some semantics. I'd love to talk with Zach about this a little bit later. So yes, while we are building and improving the factory, you are still responsible for improving the product directly. After all the product we ship is what creates value for customers and users, thank you. But whenever our factory fails, we need, yes, okay, we need to learn from the failure and try next time to get further into the automation. Boom, right on with you. Okay, over time, the percentage of automation will go up and up until our job is purely improving efficiency not pulling cars off the line. Yeah, okay, I'm with you there, that makes sense. I think, I wouldn't say that this is necessarily walking back, but he's using very extreme language to make a point. So that's okay. This is extra important for work because our company is now in the factory business. We are running a factory for improving our open source warp terminal. That is almost a million devs relying on it, improving. And we are shipping a platform called Oz to help other companies replicate this workflow on their most important products. The factory business is the only software productivity business that's going to matter. If we're selling that, we better embrace its ethos ourselves. This makes a ton of sense. And it absolutely agree with Zach writing this memo for warp engineers. They are trying to build a piece of what will be the part of the factory that other companies use. And so adopting this mindset makes a lot of sense for them. Still, they are building a product. It is a product that will have users. And so definitely understanding how users are using that product will shape the factory that you build. I don't think that we can reasonably make a single factory that works for all possible software cases. I think that every software factory is going to be unique to the particular company because it's going to take shape or be shaped by the product that is being built. So there we go. What needs to change the automation mandate? First, we must measure our throughput and efficiency. Measuring is good. We must religiously look at how much product we are shipping, autonomously, and how much it costs us to ship it. I think that's hugely valuable. A lot of times we just use up a bunch of tokens and ship it without coming back around and saying, was it worth it to use up those tokens for this particular feature? So yes, I think measuring that is very valuable. We're not doing that yet. The most important metrics to track right now are the percentage of fully automated tasks we are completing and the cost of completing them. OK, so I think the percentage of automated tasks-- to me, that matters a lot less, but I can see why he's trying to push in this direction to incentivize the building out of the factory because what you measure is what either gets gamed or hopefully improves. And so yes, this is kind of pushing in the right direction, but I think you could overdo this. And you wind up drastically increasing costs, which is why it's also good to compare that to the cost of completing it. So you're trying to increase the fully automated task while decreasing the amount that that cost, which will hopefully make this factory smoother and grease the wheels, so to speak. Second, we must force ourselves to approach everything automation first. That means every time we use an interactive agent, aka human in the loop, to write code, we view it as a failure to learn from. OK, for every task we should follow, the factory workflow and only pull staff off the factory floor where we need to. Right now, we will need to do this often. That's very pragmatic and practical, that makes sense. And that's fine, but the goal is due to this less and less. Yes, I at least appreciate the pragmatic view of this that right now we're not really there. I know that a lot of you watching right now are probably thinking, we're never going to get there. Will you always need a human in the loop? I'm not sure that I completely agree or maybe we need to define what human in the loop really means. For you, does that mean that the human has to press the merge button? Does that mean that the human has to read the code? A lot of this is going to be different, and I think that this is going to change significantly over time. But I think what Zach is trying to get across here is that if you have built out a really great software factory, then you should be able to do a lot without humans in the loop. And by measuring how much we need to include humans in the loop, we can figure out where the rough edges are. And I don't know if he's going to get to this. I haven't read this whole article yet. But what I would also add to this is you need to also measure how frequently you didn't have a human in the loop. And that was bad. How frequently would a human in the loop have solved some sort of problem? Because if you don't, then you're just going to think, oh, well, we've optimized for humans not being in the loop. And I don't know why we have to keep on going back and fixing things. But if you're measuring that and you're actually thinking about that, then you can not only improve the software factory itself in its ability to do things without humans, but you can actually also improve the output, which I think is really valuable. So it's more than just how much of tasks are fully automated and the cost of completing them, but also how well are they actually accomplished. And that, again, is where thinking about the product actually matters a lot. OK, sweet. So factory workflow is simple. triage agent runs and tries to understand the repo, the issue. If it determines the task is automatable, hand it to the implementation. If it's ambiguous or whatever, have the agents spec it, OK? Yeah. This is interesting. Because if it's ambiguous to the agent, then I feel like it would need to have some input from a human. So I'm not sure-- I would like to dive a little deeper on what your spec agent is actually doing. If it's ambiguous-- oh, OK, if it needs specs because of ambiguity. OK, yeah. And if it's ambiguous, maybe even you get to this phase and you're like, OK, I'm going to build a spec. Oh, it's still ambiguous. I can't build a spec for this. Now we're going to get the human in the loop and rerun or just decide to park the issue for now. And so this phase is where you're going to be thinking about, OK, so how well are we creating those specs? So we've got the software factory. And so the solution to this ambiguity and the human in the loop, I think the idea is once the idea gets into the software factory, it should be able to do the whole thing. And if it's ambiguous, then it's going to spit it out. And so the way that you solve this problem is you improve your process at creating those tasks initially in the first place. OK, so then if necessary, if the spec runs, spec agent human reviews, the spec passes to an implementation agent. That's another one that will be improved by improving your process for how you get specs into that factory. OK, implementation writes code, code review, agent reviews code, yep, verification agent does computer use it. Yup. In fact, I've been doing this stuff for quite a while now with cursor cloud agents. I've got cursor implementing it. And then I've got cursor and code rabbit reviewing it. Cursor cloud also has computer use. They're like an actual box for the cloud that handles verification. And then it will take screenshots and even videos and stuff, and so the human review can review the code itself, of course, if you need to. And then depending on the situation, some needs it more than others. I will often review migration scripts and stuff like that. That seems to be high risk if that goes wrong. And then verification output. So watch the video that was generated, whatever. And if necessary, go back to previous steps. CI/CD, I kind of include that as part of the review process. But yeah, that's a part of that. And then ship it, I would say, especially early on, but actually forever, ship it with feature flags. And this will actually solve a number of problems. Users get really overwhelmed if you keep on shipping stuff to production and changing their workflows and stuff, especially for something like Warp that is so workflow heavy. And so I would say ship with feature flags would be really kind of a necessity here. For one, it helps to ensure that this feature that was completely written by the software factory, especially now we're still working on, it doesn't ship out a feature that breaks everything. And we can easily roll back or fail forward. But then the other part of this is it doesn't mess up people's workflows as much, because you can slowly open it up as an experiment to the users who were originally asking to solve that particular problem. So yeah, ship it with feature flags. And then monitor agent runs and creates issues if it needed to-- if need be completing the loop. Yes, I would say this doesn't quite get to what I was hoping that they would say. I think this is kind of review everything that happened and see if you can improve that software factory. So like feed this back in as part of the loop before improving the factory. But this is why you need to continue to think about the product. Because once it's been shipped, you need to make sure that the thing you shipped actually solves the original problem. So like, yes, we're talking about the software factory and improving that. But there's a whole other piece that is kind of left out of this article, which I don't think is just the product manager's job anymore, where it's the input and making sure that those are specced well or at least non-ambiguous. And so that we don't have to boot those out and get the human in the loop in the first place. And it could be that the agent thinks it understands what that ambiguous thing was. And then it ends up building the wrong thing. And so if it does make it into that software factory and out it comes, the thing that you built, you need to verify that that actually matches the expectations of whoever is experiencing that problem, which is why the feature flags are so useful. And in addition, having some sort of metrics or something so that we can feed that back into the beginning and go through again, also like bugs and that sort of thing. So a lot of that is having some sort of channel of feedback, having a connection with actual users who are playing around with it, all of that really matters. And if you completely ignore the product and you're just focused on this piece, then you're going to be missing a lot of things that need to happen in here that are going to improve the process of inputs and outputs of that factory. So the workflow should already look familiar. It's what we built for our 60K star open source repo. Congratulations, Warp. That's exciting. You can see how that's working at build.warp.dev. My view is that it's half working, but we haven't fully embraced it yet. Look at build.warp.dev and see 1,300 issues in the ready to implement state and wonder, well, what are we waiting for? Let's have an inch and implement them. We need to make it fully work. OK, so new task issue, triage runs, triage outcome. Park for now, if it decides we should park it, just park it. Needs human clarification, OK, good. Needs spec due to scope or ambiguity. Specage runs, needs revision. Oh, the human decides whether it needs revision. And once it's approved, then, or if it was originally automatable, then the implementation agent runs, code review agent runs. If it's not ready, I would actually put this arrow back to the implementation agent. I find that I like to put feedback back into the implementation agent, because it has the context needed to make whatever improvements. It's the same thing with people. If you are reviewing somebody else's PR and you see a problem, it's better for you to go to them, rather than just implement the fix yourself. Unless it's really simple. But yeah, in general, I'd put that right there. And then we go through this, verification and human reviews. And I would put the CI and CD-- I don't know, maybe you're trying to save money on CI and CD. But I wouldn't want to bring the human in the loop until after CI and CD has said, yeah, this is-- I'm talking about your test, your build, those sort of automatable or automations. That I feel like should be before the human gets in the loop. Like, what a waste of time it would be if the human says, yep, it's approved. Oh, CI failed. Let's send it back through the thing again. And now-- oh, significant things had to change. Humans back in the loop again. So move that. Ship it with feature flags. Monitoring agent runs, issue detected, yes, create issue. Yeah, I like that there's a monitoring agent. I'm very curious what that monitoring agent is actually doing, what kind of observable metrics are just like encoded into all the features and stuff. Every time we need to bring a human into the loop, our platform Oz needs to record that and have our self-improvement agents try to improve the flow to make it less likely a human has to intervene again. Likewise, we need our self-improvement agents running over factory conversations, looking for patterns, token waste, and suggesting efficiency gains. Boy, that-- I agree with this in general, but one thing that I'm worried about, if we're looking for patterns of token waste using a self-improvement agent that's looking over the conversation, you're going to need to have some mechanism for summarizing that conversation. Otherwise, you're like paying double because you're looking over the tokens twice. We should be e-valuing different harnesses and prompts and noting which works best for the next execution. I think that's interesting. Yeah, that is an interesting-- so with that, those e-vals will have to be really good. OK, our primary job as engineers at warp is to make sure this workflow is running smoothly and that Oz provides the best experience possible for supporting it. If we get this right, the factory will eventually reach a state of recursive self-improvement. That's the golden path. That would be amazing. And I do believe that that's a good north star. Where you get to that point and now you're not building a software factory. You're building a software factory that builds software factories, which, wow, that would be pretty amazing. This may sound like an embrace of treachery. I don't know, I don't think so. Or maybe an idealistic but unrealistic vision of the world. I would go more on this side of things. But I would say that you're never going to do really awesome things without trying something that nobody's really done before or you haven't done before. So I don't mind the vision. I do think that it is idealistic. But I don't know, like there are a lot of things where idealistic and now we're flying in the air. Like we are no longer getting to tinker with code or that we're going to spend all of our time reviewing agent-generated slob, I don't read that from this at all. Actually, I think that it's a really interesting challenge to see and debug. OK, why did I have to enter the loop here? Or why didn't I enter the loop here when I should have? I view it differently. I view it as solving the most important problem there is in software engineering. OK, here we go. The last most important problem there is in software engineering. Ooh, sounds a lot like the last software engineer blog post that I wrote recently that-- yeah, I'm interested in this. I call it metaengineering, which is engineering the system in which coding agents can most effectively build and ship useful stuff. That is fascinating. Yeah, the last most important problem there is in software engineering. I actually don't disagree with this, except I really feel like we're leaving out an important aspect that the product is the thing that matters-- or not even the product, but solving the user's pain point, being hired by the user to accomplish a job, that is the most important thing. The way we accomplish that happens to be with software in our industry, and the way that we get to that software solution happens to be with this metaengineering of building a software factory. So I can see that. But ultimately, the most important part of everything is not going to be the implementation detail. But it's going to be how well we solve the problem. And if metaengineering is it, then that's great. But maybe there's something that comes after this. But it's always going to be in service of the product that you're trying to build. And I just don't think that you can build a reasonably good software factory without understanding the problem that you're trying to solve in your software. Yes, there's going to be a ton of pain in the short term where agents fail, or we have to review their slot. But we must make ourselves experience this pain so we can figure out how to automate it away if we don't someone else will completely agree. All the hard things are-- nothing we're doing is easy, all that stuff. And you have to do it the hard way first to really do it well. So completely agree with that. Our mission has always been to empower developers to ship better software more quickly, letting them all build, manage, and tune cloud software factories is the final product for this. Ooh, ooh, I'm sure he didn't mean to do this. But I love that this is an article titled, We Are Now Factory Engineers, Not Product Engineers. But what gets the last word right here? Product. It all comes back to product. So yes, we are building software factories. I actually agree with so much of this article. And I see this as the direction where we're going. And honestly, I think that we're going to stop reviewing code in the future and instead review systems. And we'll say, OK, here's the thing that needs to happen. And when we're looking at the review process, it's going to say, well, here is the system before the system was now changed. Here's how it was changed. This is the diff between the previous system and the system as a result of these changes. And that is a big part of the software factory. And maybe in the future, the software factory is going to be handling reviewing those system changes. And you could say, OK, well, if no system changes were necessary, then humans don't need to look at it. We've got good primitives and whatever. And OK, that's great. And so you can avoid the human in the loop there. So I see the software factory as a really great output of the energy that you put into solving the user's problem, because you can solve it faster, all of that stuff. But really, it comes down to the product. And if you are a software factory engineer, then you are still a product engineer. You are just engineering. Your product is the software factory. And to understand your users' problems that they're trying to solve, you need to understand the ultimate product that your end users are paying you for, where your paycheck comes from. And this is all the sorts of stuff that we talk about on become an epic product engineer podcast that you should absolutely take a look at Sean's episode. In particular, it talks about a lot of things like tying your vision to and what you actively do to the income of the company. There are so many really great conversations on become an epic product engineer. So definitely look at and subscribe to that podcast. And then with Better With Kent, here I am just trying to help you navigate the very rapidly changing software landscape. I do think that a lot of this article has some really, really great points. But there are some pieces that are missing that are the durable things that you need to do to be a very employable and valuable software developer in this age, and that is really caring about and understanding the users' problems and the products that you're building to solve those problems. This has been Better With Kent. Thank you so much for giving me some of your time. I hope that you like, share, subscribe, and let me know what you think about the future of software development. And with that, we'll be good. So thank you so much for getting better with me.