Software Delivery in Small Batches

Adam & Joe discuss the proposed additions in the 12.1 factor app, the challenge with dev-prod parity, and the missing security factor.

Show Notes

Adam welcomes Joe Kutner to the show. Joe is a Software Architect at Salesforce Heroku. He's been writing and speaking about the 12 factor app for years and continually evaluating the practices at Salesforce Heroku.

Adam & Joe discuss the misconceptions of the 12 factor app and the gaps between the 12 factor app and continuous delivery requirements. This is really a follow up discussion to the earlier episodes on the 12.1 Factor app.

Joe Kutner's Links
Adam Hawkins' Links
Get Involved!

Follow @smallbatchesfm on Twitter and tweet me with your comments. Want a topic covered on the show? Then call +1-833-933-1912 and leave your request in a voice mail. Preference goes to to listener requests.

Support The Show!

Tell me about you in the listener survey! Rate this show on iTunes. Share this episode with your friends and colleagues. Feedback is love, so send some my way.

Creators & Guests

Adam Hawkins
Software Delivery Coach

What is Software Delivery in Small Batches?

Adam Hawkins presents the theory and practices behind building a high velocity software organization. Topics include DevOps, lean, software architecture, continuous delivery, and interviews with industry leaders.

[00:00:00] Hello and welcome. I'm your host Adam Hawkins. In each episode, I present a small batch, with theory and practices behind building a high velocity software organization. Topics include dev ops, lean, software architecture, continuous delivery, and conversations with industry leaders. Now let's begin today's episode.

[00:00:25] Today, I'm happy to speak with Joe Kuttner. Joe is an architect at Salesforce Heroku, where he guides engineering best practices. Also, he's been writing and speaking about the toll factor rep for years now, I invited Joe on the show to discuss the twelve factor app in my proposed additions to what I call the 12.1 factor app.

[00:00:44] If you recall, I did some episodes on dead product parody, config, and logging, but no worries if you haven't listened to those episodes yet. I mentioned my high-level points from those episodes throughout our conversation. But if you haven't listened to them, then please check them out after this episode.

[00:01:01] We discussed Joe's opinion on applying a 12 factor app guidelines to modern software and what he would add given today's landscape.

[00:01:08] We also spoke a lot about dev product parody in a way that you may not expect. And that caught me off guard. So maybe the same thing will happen to you. Now I give you my conversation with Joe Kuttner.

[00:01:24] Adam Hawkins: Joe. Thanks for coming on the show.

[00:01:26] Joe Kuttner: Hi, good to be here.

[00:01:27] Adam Hawkins: So I've already introduced you in my own words. Why don't you introduce yourself in your own words?

[00:01:34] Joe Kuttner: I've been working on the heroku platform for about six years. I started as the Java language owner. Uhso I was curating the Java experience on the heroku platform and making sure that Java developers felt at home and were using the tools that they sound natural.

[00:01:51] Joe Kuttner: I was doing that for a long time, and now I'm an architect. So I'm working with a bit more broad down the platform and guiding some of the engineering best practices that we hold dear as heroku.

[00:02:02] Adam Hawkins: Yeah. So I guess that's where the twelve factor app comes in. I wanted to have you on the show because I heard your interview on software engineering radio, and that happened to coincide with some of the episodes I've done on this podcast.

[00:02:13] Adam Hawkins: And what I called the 12.1 factor app specifically like 0.1, because it's a, it's an addition. It's not necessarily like changes, but I think it's a broadening of the scope. And after listening to your interview, I think there's some alignment between us on where we think that it can be expanded upon in some of the, not necessarily the shortcomings, but the changes in which the practices apply in the current context.

[00:02:40] Adam Hawkins: Maybe we can start off by talking about how has that context different in your experience versus over the years, how have you seen the application of these principles change as the technologies change?

[00:02:50] Joe Kuttner: When the 12 factor app concept was first launched, I think it was about eight years ago. The state of the industry was quite different.

[00:02:59] Joe Kuttner: I think the apps I was working on were deployed to a server running in a closet down the hallway. We weren't even on the cloud yet, let alone talking about the different options for what infrastructures of service and whether you're using a path that you built yourself or some vendor paths. Part of what the 12 backdrop was trying to do was teach people, the architecture designs and principles that were best suited for cloud deployment.

[00:03:25] Joe Kuttner: Because when we were deploying in those onto the metal on a server that was in the same building as us whatever. Uhthose resources were really precious. We had to fill out some it form to get a new server and it took two weeks to get it set up. There was an actual cost directly associated with that. We were treating our servers themselves, but the applications that ran on them like pets, we were trying to keep them running all the time. We didn't want to restart them, that kind of thing.

[00:03:52] Joe Kuttner: But today the infrastructure that we run on is disposable, right? It's much more like livestock or cattle where we can buy new ones when we need one at market and we can dispose of, you know, something's not working. Right. We get. Yeah. So it's a very, it's a very different model.

[00:04:07] Joe Kuttner: And that was just emerging when the 12 factor app was first launched. I think it did a great job of helping bring people into that state of mind. But as you say, that was a long time ago, I've been giving talks about the 12 factor app for at least five years. And even for me, but the way I talk about it has changed in that time. So I think that says something about it.

[00:04:28] Joe Kuttner: Oh, yeah. How so?

[00:04:29] Joe Kuttner: Well, when I first started speaking about it at conferences and writing blog posts, I was targeting the Java community. I mean, I wasn't alone. It looks like Josh long and the spring team were trying to bring that ecosystem forward to where ecosystems like Ruby already were.

[00:04:47] Joe Kuttner: The tools in that ecosystem are lagging behind to say the least, we were really focused on tools like for in the Java ecosystem. It was the eclipse IDE and aunts, which was very poor at managing dependencies. And we were trying to move people from those into a way of doing development that was more modern, again, very much based on ecosystems like Ruby and node Js.

[00:05:11] Joe Kuttner: But that ship has sailed. The Java ecosystem has very much modernized. So technologies like spring boots and other web frameworks have those kinds of principles that we expect that are more akin to rails than some of the servers of the past. So when I talk to people, there are points that I would hammer on five years ago, but I don't even need to talk about anymore.

[00:05:32] Joe Kuttner: And there are other things that have moved to forefront like securities.

[00:05:36] Adam Hawkins: I think the success of the 12 factor app is vindicated in the fact that people just work this way by default. Now you see cloud services design in this way platform as a service designed in this way, engineers tend to just generally think in this way, at least this has been my experience. Encountering something that was not really in accordance with these practices would surprise me. And I think that would surprise many of the people I've worked with also.

[00:06:00] Adam Hawkins: So my question to you is how do you think the changes or the adoption of technologies like Docker and the container orchestration have changed the way that the toll factor app is applied?

[00:06:13] Joe Kuttner: They've changed it, but in some sense, they've drawn from it too. Right. So I think that's important to think about as well, many of the Docker commands look like Heroku commands, Docker run, heroku, run, you know, that kind of thing. Right. And that's because they were drawing from those same principles.

[00:06:30] Joe Kuttner: They really got us thinking about. Taking things to the next level, because you can't run your own Heroku locally, but Docker is it's Docker on your local machine. It's Docker in the cloud. It's whatever, wherever you're running. That changed the way we think about things like dev prod parody. And in some ways maybe we started going too far with that.

[00:06:52] Joe Kuttner: You and I have talked about crazy Docker, compose the animals that have tried to mimic an entire complex infrastructure, many different microservices and databases, all working together locally. And at times, trying to sustain that kind of dev prod parody at that extreme ends up being detrimental to your developer productivity, that you spend more time keeping that in sync with the production environment than actually just writing code that solves the problem that your business needs to solve.

[00:07:20] Adam Hawkins: Yeah, that was probably one of my biggest contention points I had after working with the 12 factor app for awhile. The inflection point for me was when moved away from working on say monolithic applications where there largely one apple, one code base is just one of everything it gets employed and that's it. Nobody has to think about anything.

[00:07:41] Adam Hawkins: But as systems become more complicated, that it becomes more code bases like interdependencies between services become really important. The way that environments are constructed, become really important rules of engagement between the services are really important when it comes to say, can things be deployed in any order or do they have to be deployed in a certain order because there's dependencies? Like what things. I think it's important for us to also consider the context for these things is something, we talked a little bit about pre-show, which was when the guidelines were started, the landscape was different, right?

[00:08:14] Adam Hawkins: So if we consider what dev prod parody meant, say eight years ago, it could be something as simple as using the same version of Java on your machine that you would use in. Because things were deployed in a much simpler infrastructure, you know, you'd have say a bare metal server. Maybe you have a VM just simpler things, but now you, it's more common to have some kind of container orchestration. You'll have Kubernetes for whatever. And then if you take it sort of the other, the other end and think, well, if I'm deploying to some sort of function as a service, like Lambda or something like that, what does dev product parody mean for that?

[00:08:47] Adam Hawkins: Because you can't really have. Say a local dev environment when you're completely dependent on this cloud service. So there's definitely some tension between trying to achieve dev product parity for the sake of achieving dev prod parity versus focusing on the developer productivity, which is something that you mentioned in the pre-show.

[00:09:08] Adam Hawkins: If you just read the guidelines today, it doesn't really seem. Like, does it come to me? What do you think?

[00:09:15] Joe Kuttner: Yeah, I think it's definitely a subtext, right? I think the spirit of dev pod parity is, well, there's really two things. One is developer productivity in the sense that if we rewind 10 years ago or whatever, people are writing Java code and they're deploying it into these large on-premise servers that are running proprietary web servers, that you can't even run on your developer.

[00:09:36] Joe Kuttner: So you're doing all these crazy things to try and actually get your app running locally. And then you find that everything works locally. You deploy it to prod nothing works. Right. And that was a drain on developer productivity. So what dev product parody gave us early on is we're moving away from that model.

[00:09:55] Joe Kuttner: Oh, Hey. Yeah, like it's really easy to reproduce that bug. It's really easy to make changes, be confident in them, set up my CICB pipeline, whatever it is and get them into prod and be confident that everything's going to work.

[00:10:07] Joe Kuttner: So when we're thinking about today and more modern architectures, which by the waythe problem that prevents us from having to dev prod parody. That's because we've moved to the cloud, right? Like now that all these resources are disposable, it's like, oh yeah, let's have these microservices because we can, so this journey brought us full circle or something. It's important to keep in mind that like at the point that dev product parity is not increasing your developer productivity, then there's something wrong.

[00:10:35] Joe Kuttner: If you need to use a tool like what it's rails or spring boot or, or node js where it's hot loading your code and you can just make changes, see it appear in the browser or whatever that's going to improve your development cycle.

[00:10:47] Joe Kuttner: The other thing is about disposability. So what you gain from having dev prod parody is a reproducible environment. If you're really sticking to dev prod parody, it means it's easy to stand up a new production environment or a new test environment because it's the same thing everywhere. And you followed these principles and, and you can do that. It's easy to set up a new developer machine and someone who's just joined your company can commit and ship something to prod on the first day or whatever.

[00:11:12] Joe Kuttner: So again, if the way you've set up your workflow, your inner loop, your CICD, if that allows you to have excellent developer productivity and that disposability that's, what's important. Keep those things in mind as you're finding that happy spot.

[00:11:30] Adam Hawkins: Yeah. So the way that I kind of find whatever that happy point is, is start by doing whatever is the least effort required to get something that approximates the target environment.

[00:11:42] Adam Hawkins: So for example, if say I am building something that's shipped as a Docker image and it's running somewhere, then at a minimum, you'll have some version of whatever your runtime is in the Docker. If you don't want to develop an container, no big deal in my mind, just use the same version of whatever that runtime is on your local machine, until you find some regression somewhere in this process, then if you can prove that, okay, there was an issue somehow related to the runtime. Or like the setup inside the container image, whatever.

[00:12:15] Adam Hawkins: Then maybe you move some of your development or some part of your overall pipeline to them check that bit and then continually expand out as necessary as you find regressions. Whereas the extreme end of this, you have front end developers who are running their web application inside a container inside Kubernetes to just look at CSS and their browser, because that's how it executes in production. But speaking to your point about developer productivity, then this is the kind of thing that goes out the window, but they feel like they should stick to this parody thing because supposedly that's a good thing, but no, maybe they never questioned the aspect of the parodies is only important if it's helping you with the productivity.

[00:12:56] Joe Kuttner: Yeah. I like to say that everything in software is about layers and it's one, I think it's one of those things where you have to cut it off at some layer. So if you are a front end developer working on some CSS thing, the layer you probably need to cut it off at is above the Kubernetes stack.

[00:13:12] Joe Kuttner: I think it's, it's tricky to find that a lot of times, especially when you think about full stack developers, where you really are, you know, sometimes need to work through the entire stack of layers.

[00:13:24] Joe Kuttner: And like very often I think that's to reproduce a really tricky bug or something like that. So having that ability to reproduce the environment is what's important. Not necessarily using it continuously as part of your development work.

[00:13:38] Adam Hawkins: Yeah. See, and that's one of the other things that I think it kind of leaves the practices open to interpretation in my opinion. Like you said, you may not want to be doing this continually, but for people who have read the text, just follow that because that's what it says. They don't start to question why they shouldn't do it in certain cases, we talked a little bit about. We're working in a product that's composed of multiple services.

[00:14:03] Adam Hawkins: And if you're going to run one naturally, it probably has dependents on others, other services that might be maintained by the same team or other engineers, the company, you know, whatever. And then, okay, because in a production environment, all the services are running then to start a development environment for one, you need to run all of the bits dependencies, and then all of its dependencies and so on and so forth.

[00:14:25] Adam Hawkins: And then you get into this tangled mess where if you're not even familiar with any of these other services, but your application expects these to be available, then what do you do? I think if people blindly focus on the goal of achieving dead part parody, then they're led down this path of definitely pitfalls. And it definitely, it's something that's unsustainable, it's unsustainable approach in my mind. And have you encountered anything like that in your experience?

[00:14:50] Joe Kuttner: Oh yeah. Oh yeah, totally. And even if it is possible at getting about an organization that has a hundred or more services that are all working together. If a single developer needs to be able to own operate and understand all of those services, you do not have a scalable organization.

[00:15:05] Joe Kuttner: So at a minimum, it's a people problem, but sometimes it's just a technology problem. I always like to ask people, especially that are using Lambda or other functions as a service platforms, how are you running locally? The most common answer I get is that I'm not. I'm always very curious to hear what people's development strategy is for those kinds of platforms, because it seems to be like unit tests and shipped to prod or something. So that's a technology constraint, right? Like you can't run AWS locally or whatever the platform you're running on, there's constraints.

[00:15:38] Adam Hawkins: You get into sufficiently advanced fakes, right? Like local, or is it called a local cloud or these versions that exist purely for these purposes. But I wouldn't want to make myself entirelly dependent. That level of fakery, but I think that fakes and like mock versions of these services and development environment are really important. Especially if you're developing a single-page application and you're just burning the front end, you call out to some other APIs, right?

[00:16:04] Adam Hawkins: Like you don't care, because part of the other problem too, is that say not everything has test or development versions of the services that you connect to. Say Stripe. For example, they have their test motor development mode that you can use. But if you're integrating with some other provider, you don't have an option, then you want, you have different accounts maybe, and everybody has different accounts. It just doesn't make sense.

[00:16:23] Joe Kuttner: You know, here's your one metric service instance, stand up a new one every time you want to spin up a new development machine or something. Yeah.

[00:16:33] Adam Hawkins: So the other aspect of this I'd like to get your, your view on is if you consider the practices now, like what the technology landscape is and what expectations are on the modern software developer at this point, what, if anything, would you add to these practices?

[00:16:51] Joe Kuttner: Number one is security. It's so complicated now. When I think about the apps that I was building 10 years ago, 15 years ago, Point security was an afterthought. It's ups was not even a thing or at least sort of an advanced thing, but today that's got to be done upfront. When you are building out your models, you need to think about what type of events I need to be logging such that I can trace back and see if there was a suspicious activity. I need to have all those proper security mechanisms upfront. They need to be a part of that initial design process.

[00:17:21] Adam Hawkins: So that's definitely the number one thing that I would say. And what would that look like in practice? I'm guessing this would be something like the 12 factor app, does image scanning or some sort of automated way to ensure that the applications free of known security vulnerabilities or meet some level of compliance, the automated testing as opposed to manual inspection?

[00:17:41] Joe Kuttner: Yeah. You know, and sounds about right. I was going to write it up today, I would take a good look at frameworks like rails and spring boot. These web frameworks do a really good job of taking care of a lot of those things for you in a way that they didn't even just a few years ago. In some sense, it's about using those tools properly, not circumventing.

[00:18:02] Joe Kuttner: One of the most common things that I see. And you can see this on stack overflow where people will actually recommend this, where it's like disabling certificate checking, because you're trying to connect to a service. And it's like, ah, it doesn't work on my local machine. So here's the code snippets just disabled.

[00:18:22] Joe Kuttner: It's hard. People, especially more junior developers who don't understand how this whole negotiation and verification process works. That's really confusing. So I think it's about making the right decisions upfront, especially for people who do understand those things. You're setting up a development environment and a production environment where you don't have to take a step like disabling security or disabling certificates or whatever it is.

[00:18:48] Adam Hawkins: Okay. So that's security, anything else?

[00:18:51] Joe Kuttner: Yeah, the other one is telemetry and I think more broadly like observable. We need to think about our applications as spacecraft when we've deployed these things into the cloud, they're not coming back. So whatever we have sent them with the instrumentation that we have sent them with, that's all we've got. That's the only way we're gonna know caused that weird bug or God forbid that they're compromised security problem. That's the only way we're going to get records of what happened.

[00:19:20] Joe Kuttner: So that kind of instrumentation that allows you to gain insight into how your applications are performing, it needs to be just as essential as many of the other factors like dependencies and config.

[00:19:33] Adam Hawkins: Yeah. That was one of the things that I also flagged in the 12.1 factor app, which was these applications have to have telemetry, especially if the pitch is these practices enable continuous deployment dusty. You know, predicate on continuous delivery. Then by design, you need telemetry. You need telemetry to understand that the application is functioning correctly. Each telemetry, to verify that whatever change that you've released is in fact, doing whatever is supposed to do, you know, all of these sort of higher level engineering outcomes like continuous delivery are predicated on telemetry. This sort of leads me into the next question regarding security aspect of this.

[00:20:11] Adam Hawkins: If we consider the idea of sort of the overall CICD pipeline as a factor of the application, like when you design your application, you have to design it in such a way that area X can be exercised by some automated test pipeline area B so on.

[00:20:27] Adam Hawkins: One of these areas is compliance. Checking. If you don't think about that from the beginning and baked it into sort of the automated pipelines that ship your application, then it's going to be more difficult than if you didn't, especially at the biggest. And this speaks to some of the other things like testing, for example, like it's all assumed right that we'll have automated testing, but what kinds of things should be covered by the test? So one of the things that I call out in the additions to the config factor is the idea of having some kind of dry run mode for the application.

[00:21:01] Adam Hawkins: I don't know if you have worked with Ruby or like dynamic languages, for example, but I would be hit by problems where I had some air in the say of production, that RB file. But that file was only evaluated until I went to prod. So I do, it actually was on, it was on heroku. No, I would push to Heroku and then the app would blow up in production because of this air. And that was the first time that I'd ever been executed.

[00:21:23] Adam Hawkins: So after hitting these kind of stumbling block more and more times, okay, now there's a better way to do this. And I came to the conclusion that having this dry run mode was a fantastic way to smoke tests that there's like free of these kinds of regretions.

[00:21:37] Adam Hawkins: If you bake that into the application, and then you can bake that into the delivery pipeline and you can assert that the application is free of these types of progressions.

[00:21:47] Joe Kuttner: Yeah. I think that aligns pretty well with what we were talking about with regard to dev prod parity. When we were talking about it, but we actually think about it is like modes, right?

[00:21:56] Joe Kuttner: I have this development mode where it's maybe a little bit like prod, but I still get the hot loading of code and things like that. And I can adjust that until I have either a production mode or as you said, a dry run mode. And I think in the same sense, as you were saying with the dev prod parody, I think it really comes down to disposability, but really the, the characteristic is about reproducibility and being able to have those different environments and to do those things easily.

[00:22:22] Adam Hawkins: Hmm. I think the more that we talk about the debt product parody factor, the more that I think it's more open to interpretation than I originally thought, but it's one of these bigger things that you can, like in some way it can come under that umbrella. So when I talk about the dev product parody, I try to be as explicit as possible in that say, if your application connects to a database, then you should use the same version of that database locally.

[00:22:46] Adam Hawkins: Now this is trivial, it's trivial, easy, trivial, easy compared to how it used to be, because you can use a Docker image to run any version of anything at any point in time. And you can have all these versions running at the same time, too. So you can even cross versus. That is the basic level. You know what you mentioned earlier of say runtime, parody and dependency, parody of system packages, application packages, all these kinds of things.

[00:23:10] Adam Hawkins: That's where my mind goes in terms of dev pod parody, but it seems like you have the broader vision for what's part of this whole thing.

[00:23:17] Joe Kuttner: Yeah. I often call debt fraud, parody the gateway factor, because you work towards that. And again, I totally hear you on how it can be up to interpretation, but as you work towards that, you're doing all these other things, right? Like separating your configuration from your code so that you're not connecting to your production database by accident from your local workspace.

[00:23:37] Joe Kuttner: Your build release, run process, which is part of your CICB and lends to your development environment, the way you bind the ports, rather than like, you know, port buying is actually a really good example because that's another one where a lot of people read that and they're like, what the heck is this about?

[00:23:51] Joe Kuttner: I don't understand, but you have to go back 15 years ago when people were deploying, well, some people still do this, but for files running server, right? Like their application had no concept of handling HTTP requests. It was just this app with handlers it was dropped into a running web server that did all the HTTP stuff.

[00:24:10] Joe Kuttner: And the poor binding factor was really about inverting that model so that your web server became a dependency of your application. And this is all very natural. It's a Ruby developers for the most part, but in other language ecosystems, it's not.

[00:24:23] Adam Hawkins: Yeah. That's the funny part, like how much of these, like interpretation of these practices is that influenced by maybe the language you grew up in or the ecosystem that you got most of your experience and as you say, working with Ruby is far different than what it was working with Java, especially 10 years ago. Yeah. Yeah. When the 12 factor app was written, Ruby was the Vanguard and the majority of the industry was doing PHP and Java actually thinking back, I think the majority of industry is still doing PHP and Java, but that doesn't change. The fact that the state of the universe is very. So just to, as a little click reality check there on that one.

[00:24:59] Adam Hawkins: So speaking of PHP, so PHP and I an interpreted in the context of something else, right? So to use the analogy of Java and more files, you have your PHP at code, but then your application server or something like engine X or Apache.

[00:25:14] Adam Hawkins: So in this case, PHP violate the 12 factor app and the strict definition, unless you were somehow going to bundle your quote app as the combination of say, like internet/Apache and PHP code.

[00:25:27] Joe Kuttner: I think it depends how you're using PHP. I think PHP like Java has this long history and a lot of different ways to do the same thing.

[00:25:34] Joe Kuttner: Right. But there is definitely advances in that ecosystem. I'm a little bit out of my wheelhouse here, but I think it's composer that has changed how the dependencies are managed in PHP. There are definitely parts of that ecosystem that have come forward. And I have friends that are PHP developers and swear by some of the best practices.

[00:25:53] Joe Kuttner: I think any tool that continues to be used in production, there are reasons for it. And some people get a lot of value.

[00:26:02] Adam Hawkins: Oh, yeah. One thing has been really helpful for me as just somebody who has moved across a lot of different languages. A lot of different stacks is using the 12 factor app guidelines just as a yardstick to assess how different technologies mesh with this way of working.

[00:26:18] Adam Hawkins: And one of the pain points is when you deal with the runtime, say like PHP, where you have like application level configuration, like whatever PHP code you have, but then you have the inf files. And you have to also configure to set stuff like ThreadPool limits and blah, blah, blah, as totally separate. And of course, those likely don't support some level of environment, local configuration. Then you end up doing some shim type solution, which will be the environment variables and modify the files before starting the thing.

[00:26:43] Adam Hawkins: And you get into this shim layer, then you go to other technologies where like, oh, and they just work that way. You don't have to think about any of this kind of stuff. And to me, this is a virtuous feedback loop between the 12 factor principles and software design.

[00:26:57] Adam Hawkins: And I'm curious, I want to get your view on this and all that you go, which is one thing I propose. And the top 0.1 factor app is the adoption of hexagonal architecture. And this speaks to your point about disposability.

[00:27:11] Adam Hawkins: The reason why I suggest hexagonal architecture is because it enforces the creation of the boundaries between the service, the code that we're writing. And I think as it's called in 12 factor language back in services. Such that if you have this boundary in place and you have the configuration, like separate configuration with environment variables, then in theory, you should be able to say, just change the environment, variable, then use a different version of this. Like you could use a fake in development. You could use this version in staging or this version of prod or whatever.

[00:27:41] Adam Hawkins: But that requires you actually adopt a certain way of architecting your code as an application developer. So how have you experienced that kind of feedback loop and how have you seen that play out in your experience.

[00:27:54] Joe Kuttner: Yeah, I think the backing services has this intention to it, which is really about decoupling and very similar the way we think about a microservice architecture and what we were saying with, if you're in an organization with a hundred services, you shouldn't be responsible for all of them. That's very often the way you should treat your database or a third-party service.

[00:28:13] Joe Kuttner: And that decoupling of the application from whether it's a database or another microservice or a third party metric service or whatever it is, that's the spirit of that factor. The use of environment variables to do that, I think is a good recommendation. But even on Heroko, we're finding that doesn't get us where we need to go sometimes.

[00:28:34] Joe Kuttner: That can at times make it difficult to manage secrets among your team where it's, you don't want everybody to have access to the same things. It can also make it difficult to like roll credentials and things like that. So there are other models you can write these things to disk and have them read from disk, but those are in some sense, implementation details.

[00:28:53] Joe Kuttner: The important part is that that decoupled.

[00:28:55] Adam Hawkins: Exactly. Once you have it decoupled, you don't care where the configuration comes from in a sense, right? Like if it's this shim thing that I mentioned, that's doing this, or some config map or secret mounted as a volume inside your brand is potted. It gets it's all fine. It doesn't really make a difference because the focus on separation of concerns, if you take that factor up the levels of abstraction, I think that's where you get the most, just the most benefit from that because of the positive effects it has on your application.

[00:29:25] Adam Hawkins: Well, Joe, thank you so much for taking the time to come on the show and talk to us about the 12 factor app and have little back and forth about the twelve 0.1 factor app seems that we are overall aligned in how we think about some of the stuff, but it's always great to hear somebody else's perspective on this.

[00:29:40] Adam Hawkins: So is there anything you'd like to leave the audience with before we go?

[00:29:44] Joe Kuttner: Yeah, something I haven't been spending a lot of my time on lately is a cloud native buildpacks project, which is very much related to the 12 factor app. Build packs are the mechanism that we use on Heroku to make the platform application aware.

[00:29:58] Joe Kuttner: So when you send your source code to Heroku, we detect what kind of app it is. We prepare it for production. That's what allows you to just have source code that you work on locally. Send it up to the server and poof. It's also what allows you to focus on processes and your application rather than servers and infrastructure and things like that.

[00:30:17] Joe Kuttner: So the cloud native build packs project is an effort to both modernize the build pack concept and bring it into the Docker and container ecosystem, but also democratize it because today go packs, run on Heroku and that's it. Well, there's some other platforms like cloud Foundry and Deus and things like that, but probably to build packs is by design its own standalone project.

[00:30:39] Joe Kuttner: And it's something that you can adopt without using Heroku. It's something that you can run locally. You can build it into your own CICB process and bring your applications into the container ecosystem without breaking the fourth wall, which I use as a euphemism for writing a Docker file and keep moving down the stack and taking on all these concerns. Buildpacks offer a better abstraction, but still play well with the container EcoSys.

[00:31:05] Joe Kuttner: I told you to give it a try, build backs that IO, we have a CLI that runs on Linux, windows, and Mac, and you can run Heroku, build packs on your local workstation.

[00:31:14] Adam Hawkins: Great. We'll have to give it a try. I may do an episode on that one in the future. Well, Joe, thank you so much for talking and hope to talk to you.

[00:31:21] Joe Kuttner: That's great to be here.

[00:31:21] Adam Hawkins: Thanks that wraps up this batch. Visit small badges RFM for the show notes. Also find small batches of FM on Twitter and leave your comments in the thread for this episode. More importantly, subscribe to this podcast for more episodes, just like this one. If you enjoyed this episode, then tweeted or posted to your team slack rate this show on iTunes.

[00:31:45] Adam Hawkins: It also supports the shell and helps me produce more small batches. Well, I hope to have you back again. The next episode. So until then, happy shipping,

[00:31:58] Adam Hawkins: one to learn more about dev ops that wasting your time and sign up for my free email My course combines the best from the DevOps handbook, accelerate and years of software delivery experience. You'll learn the three ways of DevOps and the four KPIs of software delivery performance.

[00:32:16] Adam Hawkins: More importantly, I'll show you how to put that theory into practice. That means shipping better software faster. Sign up today at

[00:32:30] Adam Hawkins: Like the sound of small batches? This episode was produced by pods worth media. That's