Subscribe
Share
Share
Embed
Derek Comartin & Adam Hawkins discuss designing individual services in a distributed system for fast flow and team autonomy.
Show Notes
Adam welcomes Derek Comartin to the show. Derek is a Microsoft Most-Valuable-Professional and distributed systems architect. He also co-hosts the Loosely Coupled podcast.
Derek discusses the ideas behind his talk the "Loosely Coupled Monolith" and "Context is King: Finding Service Boundaries". Great high level conversation on how software design principles inside applications scale out to distributed systems as a whole.
Adam Hawkins' Links
Derek Comartin's Links
ā
Support this podcast on Patreon ā
Creators & Guests
What is Software Delivery in Small Batches?
Adam Hawkins presents the theory and practices behind software delivery excellence. Topics include DevOps, lean, software architecture, continuous delivery, and interviews with industry leaders.
[00:00:00] Hello and welcome. I'm your host Adam Hawkins. In each episode, I present a small batch, with theory and practices behind building a high velocity software organization. Topics include dev ops, lean software architecture, continuous delivery, and conversations with industry leaders. Now let's begin today's episode.
[00:00:25] Today I'm speaking with Derek Coleman. Derek is a four time Microsoft, most valuable professional. He has two decades of software development experience focusing on software architecture, CQRS event, sourcing, messaging, and.net development. I found Derek on Twitter through a link to a blog post on the loosely coupled monolith that peaked my attention.
[00:00:46] That led me to his YouTube video, titled context is king finding service boundaries. That's when I knew I was hooked and I could tell that Derek and I are cut from the same. Both of us have similar ideas. This goes back to a talk I gave in 2014, titled application architecture, boundaries, optics rules, and patterns, the same ideas in the talk apply to services.
[00:01:10] So I invited Derek on the show to discuss his rules for designing distributed systems, because dear listener, as you know, the strong correlation between high velocity, software delivery and service, autonomy. This also speaks to Conway's law and team typologies, but Hey, we've already done an episode on that.
[00:01:29] Basically, if you want to go fast, then you've got to decompose systems and that requires to finding context and setting boundaries. So with what that I give you my conversation with Derek Cole Martin.
[00:01:43] Adam Hawkins: Derek welcome to the show.
[00:01:45] Derek Cole Martin: Thanks. Thanks for having me.
[00:01:46] Adam Hawkins: So I wanted to have you on the show to discuss some of your work online related to like the breaking up of monoliths, the construction of microservices, sort of the relationship we can, software architecture and the services, and sort of the idea of founded context around services and how that fits into the overall architecture of systems.
[00:02:08] Adam Hawkins: So for the listener, why don't you give a little bit background on kind of the systems that you worked with and how you came to kind of like your philosophy on softwares? Yeah. So what you're alluding to was kind of a series of blog posts and videos that I had recently about creating a loosely coupled monolith.
[00:02:31] Derek Cole Martin: This is kind of like what I called it. I didn't want to use the term modular in the reason being is because most of the, I'm not saying it's not, there's not content out there that does this, but a lot of the content that was seeing relate to a monolith, ultimately having the term, like a big ball of mud, like there's just this equation of people thinking that monolith is there's, there's no way you can create a, a modular monolith and nevermind modular, because I get the feeling when people think modular, they think of, okay just components that are how they're even loosely coupled. Nobody really knows. They're just, there's some invisible boundary.
[00:03:11] Derek Cole Martin: So what I wanted to describe what they loosely call monolith was that you can take a lot of the principles that are being used that have always been used in SOA. And that are semi use the microservices, depending on what version of microservices that you're talking about.
[00:03:33] Derek Cole Martin: And basically just define clear boundaries about what your system does and then communicate asynchronously through those boundaries.
[00:03:42] Derek Cole Martin: So a monolift doesn't necessarily even need to be that everything needs to be done in process. It's just taking this idea that. You can have a single code base with well-defined boundaries, even within that code base. And when you want to communicate between those boundaries, do it asynchronously.
[00:03:58] Adam Hawkins: Yeah, for sure. And I like the concept of sort of a loosely coupled model within the sense that I think that there's maybe kind of an assumption or at least this has been in my own experience like a monolith is a monolith in a sense that it's going to be like one process as deployed it's doing everything. But even inside like one boundary of a whole service, you can have one process that's doing this or one process that's doing that. Like they can maybe I'll be sharing the same code, but there's nothing that says, like all of these things have to be highly coupled into one cohesive unit of code and all run and deployed in that way?. Is that kind of what you're getting at?
[00:04:37] Derek Cole Martin: Exactly right on the money. So that's the thing. And I've mentioned this a few different times at conferences and have discussions with people as I'm still trying to figure out what this came from, is that the idea that you have a single repo that represents a single project that gets deployed as a single unit where all these things just became that they have to be a one to one to one and they don't, at all.
[00:05:00] Derek Cole Martin: Like, so. In my examples that I was using is, I actually have two top level executable processes. One being just the, them, both being entry points. So one being safe you're creating a web app. If it's served, if it's just an HTTP API. That could be that's one kind of interface to the application. And then the other being where the loosely coupled part comes in with messaging is something that's just strictly picking up message from a message broker and then dispatching them to whatever relevant part of the system is good. Basically needs to handle those.
[00:05:37] Derek Cole Martin: So even when you start thinking about microservices and again, I, that's why I said like, it depends what your definition is, is because I think there's this... It's like, I think if you were to ask around, like, what are microservices? Like, what are the, the kind of the tenants of microservices?
[00:05:53] Derek Cole Martin: Well, there's tenants of SOA. There's kind of, there's defined rules, guidelines that you need to be following if you like that are kind of established. And those were by, I believe Don box and like early, mid two thousands, but there really isn't that for microservices. It's kind of good and bad that there isn't a strict definition.
[00:06:16] Derek Cole Martin: But I th unfortunately I think with a lot of things in our industry, There's concepts and terms that just have wildly different meanings to people.
[00:06:24] Adam Hawkins: Like, so end-to-end tests or integration tests?
[00:06:27] Derek Cole Martin: Yeah. Anything to do with testing, like good luck. Trying to get a clear definition of what that means to people.
[00:06:33] Derek Cole Martin: Rest is just become synonymous with HDP. I mean, you might as well, not even say I don't even bother saying it anymore. Cause it, it doesn't mean that. I don't think it means what you think. It means type thing that mean. There's just so many things that are that way. And I feel like microservices are one of them, where I just go back to more of the, the tenets of SOA and the biggest one being autonomy.
[00:06:58] Derek Cole Martin: And that's kind of goes with the loosely coupled monolith is still having autonomy, even within boundaries within that model.
[00:07:07] Adam Hawkins: Yeah. So before we get into the boundaries and context, I think it's good if we set some terms, defines my things that we're talking about. So I like how you brought up SOA, because one thing that I've kind of been going back and forth in my head for a long time ,really, is what is the actual difference between microservices and service oriented architecture in the sense of like, is it actually. Technically different in terms of like how it's done or is it just the fact that like, say, is it SOA if you have 10 or as a microservices, if you have a hundred, like, where's the boundary, what's like sort of this, the spectrum of these things, because at least sort of how I would approach it, this architecture principles I would have used to build a service oriented architecture are the same that I would do in microservices.
[00:07:58] Adam Hawkins: Just that if you have a thousand, maybe you have to change the way that you work, like as an engineering team support them. But like on the technical level, the architecture level, you would still, you know, enforce a boundary between services, still have an API, so we'll have autonomy, so we'll have like data ownership, all these things that I don't think they would be different if I had two or 200, you know.
[00:08:18] Adam Hawkins: So I'm gonna guess, based off of the work, I've seen that we share the, kind of the same definition of a service, which is in my mind, it's independently developed unit of software that's deployed and communicates with others over an API, like kind of broad, but it's sort of, I think it fits for me. How do you see services?
[00:08:38] Derek Cole Martin: Pretty much the same way. I mean, I don't, I can't really give you a definition of what the difference is between a regular service and a microservice. Like, what is that size have, I mean, if you go back to the whole, okay, two pizza team thing, like I don't even, I don't even know what that means.
[00:08:59] Derek Cole Martin: Like, I mean, it's relative you get what it means, but it's really about what can an individual team can take responsibility on to develop and manage. And if that's there, I mean, if that's there they own it, and they have to deal with it, it's whatever they can, like whatever their capacity is. So if you have really small teams or a little bit larger teams, I don't, I don't, I just think the term is so weird.
[00:09:29] Derek Cole Martin: I think maybe what happened and why it happened is because... In the SOA world, things got a little sideways with enterprise service buses and that kind of dictating where traffic flowed, rather than now, it's more of a, what you call dumb pipes, smart end point like smart end points that like you have messages to the understand whether they need to deal with them or not. There's no, there's no orchestration of routing. But at the same time, again, this is just my assumption on the feeling of when I read blog posts and Twitter, and that's not to say I'm in my, I'm in my own bubble clearly on Twitter, but with microservices and even back in the kind of SOA 2.0 version, is that there, they're got this push to be everything to be synchronous.
[00:10:20] Derek Cole Martin: So I'll use it, like I am not speaking for inside knowledge or even how this works, but I'll use the example of Uber had their blog posts about their... I don't know if you read that article about DOMA.
[00:10:34] Adam Hawkins: I was to the, and even the one that fuse before that, where they're like, oh, we have 2000 or 3000, whatever.
[00:10:40] Derek Cole Martin: A little over 2000 microservices in the beginning of the article kind of makes a lot of sense to me because it said like, okay, we started off as a monolith and then it just. I don't know how quickly it exploded, but it exploded to 2000 services. But it makes sense because if you are in an organization that is growing, I'm assuming here, that rapidly and the economy is law.
[00:11:07] Derek Cole Martin: This is going to happen. This is you're going to end up with multiple teams working on these things. And if you're trying to rip apart a monolith, that is tightly coupled, I'm making assumptions here. I'm just not using really Uber. Lets you think of any system that's getting large and it's really growing.
[00:11:25] Derek Cole Martin: That's tightly coupled. If you have a lot of inner process communication, how are you going to rip that out and make separate services? While you're going to call from service a to service B that you, you know, mean you pulled service B out, you're going to now have to call it synchronously over HTP. Or something over the network.
[00:11:48] Derek Cole Martin: So then you end up with a distributed monolith ultimately, and I'm not saying there is that, but I will say if it isn't, then they did a really poor job explaining that the, how they used messaging, because they never really mentioned in that post anything, then our PC. There was one little kind of graphic that showed a mention of kind of events, but I don't know how that fix fit into the mix, but I, I, I'm always curious to know what the outside world right now that people are developing microservices, how much they are using RPC and how much are they are using messaging.
[00:12:24] Adam Hawkins: Hmm. Yeah. So deciding on how you communicate between these different services is actually really important in terms of your overall architecture. You know, if it's going to be synchronous, it's going to be asynchronous.
[00:12:36] Adam Hawkins: The last time I went through this exercise was breaking up monolith and it was really lucky in the sense that the, like I've been working with the company for a while and there was pretty well stablished.
[00:12:49] Adam Hawkins: What I call like fault lines inside the monolith, where you could split things out where there was already some kind of quasi API in the sense that there was a class definitions or the method calls. And you could sort of imagine that, okay, this could be just, you know, cut out here, moved over here and you know, you could do that.
[00:13:04] Adam Hawkins: And when we did it, we just, this was back in 2014, 2015. I can't remember. But we had decided to use thrift. For RPC all you know, between all the services. So I broke up pretty well, but you know, the definite other areas, of course async and then the question is, are you going to have a centralized, like messaged broker or are you going to have an independent streams? You know, how do you decide sort of the level of independent vert, like decentralized versus centralized? You know, so like, if you think about it, where do you kind of fall on that spectrum?
[00:13:39] Derek Cole Martin: So how did it work out with everything being RPC and when it got split up?
[00:13:45] Adam Hawkins: It was, it was fine. Like for the, it was, we had also the ability to like background certain things. When we had say, you know, like a request come in, some web server user does something it's going to have to send an email that just queued up a background job to execute asynchronously, but when the job executed, it would make a synchronous call to email service to send some email. And if it failed, it would retry.
[00:14:09] Adam Hawkins: But the issue with the, the synchronous thing was not necessarily from my experience there, like the operability, but the expandability it's like when you, when other parts of a system have to hook into other parts of the system. It's nice not to have to actually make multiple RPC calls, but say, Hey, I'm publishing a message to this place.
[00:14:31] Adam Hawkins: Anybody who needs whatever needs it can listen to it and then do something, right?
[00:14:36] Derek Cole Martin: Yeah. I guess it depends on how how much explosion happens of these services? I think Uber is probably a little bit extreme in their example of having two monoliths or whatever it was that goes into 2000. But even in their posts, they mentioned like, okay, well troubleshooting, you had to basically look at, I don't know, like a dozen different services.
[00:14:57] Derek Cole Martin: Somewhere around there. I can't remember exactly the number. They said it was at least a dozen that you were using to trace through request or whatever the issue is and that sounds horrific to me.
[00:15:12] Adam Hawkins: Oh yeah. I know you when we, when we did this exercise, we went from like a one model monolith to something like 10 different services, but the typology was designed specifically such that there was no cycles and the sort of the traffic through this, or at least a minute, like as minimum as possible, certainly not in the synchronous paths, because as you say, if you're going through. you know, More than two, basically something that's just like, ah, there's too much. Right. You can't go more than one dependency deep to actually care to debug it. You get, then you get into the, you know, the questions of, okay, who is what in a big enough organization is like, whose team is this? Like who are the people that maybe even operate this? I have no idea. Right. And then that's when you get into, like distributed ball of mud, which was even worse than a centralized ball, but in my opinion, yeah, I agree.
[00:16:00] Adam Hawkins: So let's bring the conversation back here to sort of defining the service and setting the context, right? So you have series of blog posts on context is king also YouTube video when it kind of fits into the idea of a loosely coupled monolith, is that, you know, if you have a loosely couple of monolith, You can run one processes, this or one processes that whatever, you know, and then even if you want it to actually that separate those things out specific, like entirely you could. Right. But that all comes down to following, you know, strong software architectural principles in the beginning.
[00:16:35] Adam Hawkins: So what's kind of the list that you have a numerated so far?
[00:16:40] Derek Cole Martin: So you alluded to it, boundaries are like the most important thing to get right. But I find on the easiest to get wrong. So it's kind of unfortunate and there's really no, like there's a lot of strategies I have for doing that, but it's not really a, like, there's no, there's no prescription on how to do it.
[00:17:00] Derek Cole Martin: It's really having to understand the domain that you're in, the problem that you're trying to solve, collaborating a ton with the business, so that you can really get a grasp on. Like what the surroundings look like. My analogy for this is, especially when you don't know domain that you're working in. And you're trying to figure out where these boundaries are. To me, my analogy is you're going into a dark room. That's pitch black, and you have a flashlight. You've never been in that room before, but you're just pointing that thing around, trying to kind of get an idea of where the walls are, how tall the ceiling is. And as you kind of work around the room more and more, you get to get a bigger picture.
[00:17:42] Derek Cole Martin: Like you get a better mental model of what that looks like. And at first, if you try to find those boundaries, you know, meaning you just shine that light just in one side, you're not going to get it right. So it takes, it takes a little bit upfront to, to kind of feel what, where those are especially. And there's a ways of doing this. There's some of the ways I've mentioned before it's and this goes back to domain-driven design.
[00:18:07] Derek Cole Martin: It's the language people use. It's how it's kind of a, an easy giveaway sometimes. How the term things. I've noticed in our world, there's a tendency to create services around entities. So like entity services, meaning I have a product and I just have this one service that does everything for a product. And that's not really the case.
[00:18:33] Derek Cole Martin: So my example of this, when I worked in distribution was I used to say a product, isn't a product, isn't a product. What I mean by that is depending on who you talk to you in that organization, in that distribution company that worked with they viewed a product differently. So if you're talking to somebody in sales about a product, they care about the sale price, they care about margins the care about who like their customers, different pricing levels, maybe, how all that works. If you go to talk to somebody in purchasing and you say the word price, they're not talking about a sale price, they don't even care about the sale price relative. They're talking about, if you say price, they're talking about costs, they're talking about the vendor price, like the cost to you. And then if you're talking to somebody out on the warehouse floor, they care about none of that. They care about locations, bins, Isles, a whole whack load of different things. And then accounting, that's a whole like the, a combination of stuff, right. But a product isn't a product, isn't a product.
[00:19:34] Derek Cole Martin: So this idea that you have one service that just is the owner of all the data related to a product will end up making, you have to be coupled to that thing. Versus is if every service that say you have a, like a shipping service and a purchasing service and a sales service, they would each have a quote-unquote version of a product entity with owning the particular piece of data that they should own.
[00:20:05] Derek Cole Martin: Right. So, like language is a kind of a giveaway when you're talking to people that this is okay, somebody said this, you said this, do you even care what that person is referring to? Like when they're talking about the sale price or et cetera. So a lot of it's just in communication to kind of figure out where those, where those boundaries are.
[00:20:24] Derek Cole Martin: So that's probably, but again, it's not prescriptive. It's you got to do a lot of legwork to get your mind into the domain.
[00:20:32] Adam Hawkins: Oh, yeah. And this is why I feel especially bad for teams that start with microservices, because how do they know where those boundaries are? Especially, I would really like your analogy of you're in a dark room and you're exploring with a flashlight because when you're in a new context and new to domain, you don't fully understand it. And you only come to understand it over sufficiently long enough timeline, given enough experience enough trials, like successes and failures, as you find what does and doesn't work. If you're going to just start from T zero and say, oh, I'm going to have this over here. And this over here, they're going to talk like this. This is how everything's going to flow. You're going to eventually going to have to pay back that debt in some way, because azar things aren't going to fit the initial model of the world that you had when you created it.
[00:21:20] Adam Hawkins: So like when you start from zero and you build up a system and it gets larger, then of course, if as you say, Conway's law will happen, you're either going to try to find ways to split it, or you're going to create new stuff outside of that. So people can actually do the work that they need to do.
[00:21:37] Adam Hawkins: But yeah, your product, isn't a product, isn't a product gives away like the other part of this, I think while I see where people making mistakes is that they assume. Service boundaries should be around like data entities themselves and not necessarily to get the functionality they provide in the whole system. Right. Because if you're making a service, that's just like, Hey, I can store products. I can get products. It's basically just rest in a different form. But what does that do besides exposing a database over a different interface? How does that, how is that useful?
[00:22:09] Derek Cole Martin: So, yeah, that's the other piece that I kind of tried to mention is that it's behavior and data go hand in hand. So. While data is useful to look at it. Like if you're looking at, say, for example, that you have this product and you're like, okay, we have price costs, all this stuff. And you're like, okay, clearly this isn't going to be just one thing. It's going to be broke up in a small cotext. It really is then starting to look at the behavior around the data because they have to go hand in hand if you just have data, well, guess what? You have a database, right? Like that's all it is. And if you have a service that doesn't own any data, I, to me, that's not a definition of a service. I don't know what we call that, but it's not a service.
[00:22:47] Derek Cole Martin: So data behavior, hand in hand, now the piece of like, okay, well, what behavior belongs where. And you're alluding to it is, is get rid of this notion of that it's all about entities. Like we were alluding to it's it's about what's what capabilities do your service provide, right? And then once you start defining what those capabilities are and grouping them together, okay. These are set feature sets that we need where? What pieces of data are gonna either change state? I mean, what pieces of data do we need to own? And you start figuring that out. Then I think that's where you start to develop a boundary.
[00:23:26] Derek Cole Martin: And like I said, I think you're going to probably end up getting some stuff wrong and putting some behavior in places that you don't ultimately shouldn't be. But that's just, that's just with building more aha moments along the way that you realize, oh, that was a, that was a bad idea. It's like, they're just, it's just going to happen as you gain more insights into the domain. And as you communicate more and really understand the problems that you, yeah, that you start to kind of redefine things.
[00:23:57] Derek Cole Martin: So that's why I said it's, it's, it's really hard. It's really important to get. Right. But it's really hard to get right. Right from the beginning.
[00:24:04] Adam Hawkins: Yeah, but there's really like important rules of thumb, like first principles that you have to adopt to actually set the boundaries in the first place. It's like I come at it from the perspective of, if you're trying to create a new service, well, that to me, kind of communicates that you have the concept of a bounded context. You know, you have some hard boundary, you can draw that inside this, that has data autonomy, it's full control. It could be a data store or any number of data stores, whatever you have defined, some sort of behaviors or functionalities that you can offer to some consumers of your service. You have an API to do that and you can develop and deploy this thing. Independently.
[00:24:45] Adam Hawkins: Now, if you, if you don't have that model in your mind, you're more likely to do things that are just horrible mistakes. Like, oh, I will just read some data from some other things database, or I will communicate through like, you know, some number of like back doors or like non advertised API as you'll do all these things that sort of just violate the principles of the boundary, but if you don't understand what needs to be in the boundary in the first place, then you can go wrong in so many different ways that don't actually give you that isolation, but just end up smearing out different parts of the system across just different code bases.
[00:25:24] Derek Cole Martin: Yeah. And then I, I think too, is that, and he's he, even if you kind of get into this entity, a service idea is that you end up just yeah with a ton of RPC and depending on, I guess the system, there's a lot of implications, right. That with, so do you, if you have, for example, some user makes a request it's coming to your service and then you need to make, say, subsequent call three calls to three other services. If any one of them has high latency, or if anyone comes in on, on available, like you're dealing with all that and how do you deal with it? Right. Like, it's, it's just another thing that you're dealing with because you're making kind of that distributed monolith. I feel like that is the case that a lot of monolith that get broken up, go into is making breaking things apart and making things RPC.
[00:26:17] Derek Cole Martin: And that was kind of the point of my posts is really saying, okay, if you don't do that, if you start this way and you do want to carve things off, you are, you were kind of, you're independent to begin with. So it's, it's really just about the strategy I was showing was okay you can be asynchronous. But you can still roll these things up into two top level executable and technically that could have only been one process that's really running, but it just so happens that the all get deployed together, but there's nothing stopping you, like you said earlier from saying, okay, we're going to take this one off and deployed independently.
[00:26:48] Derek Cole Martin: Cause maybe it gets, I mean more traffic or has a different load or et cetera, for whatever reason and you, because maybe you start revving that one differently and you don't want to deploy it with everything else.
[00:26:58] Derek Cole Martin: Maybe you, I mean, There's more, maybe you're developing more of a risk to deploy those monolith so you want to have a different strategy so you can let some. Pieces that aren't changing much. Leave them be and then deploy I mean, the pieces that are a little bit more.
[00:27:14] Adam Hawkins: Yeah. That's one of the other factors I think is really important in setting up sort of the boundaries because the boundaries are, there's no technical boundaries, but then there's also organizational boundaries. Right. And, you know, on ways law laws, a law in the sense that if you're going to be hard for you to create a new service, if it's spanning across like three or four, you know, and different teams, I mean, we really want to localize these services to. Like one, one team and avoid spreading them across multiple teams wherever possible.
[00:27:41] Adam Hawkins: So like you have the boundary from the technical sense of like API in this whole context, but then you have the boundaries from the organizational sense as well. And like one thing you mentioned, the change ability is that there are some, you know, like there's some pieces of code that they might change every day, you know, even multiple times a day, And some code that may not even change more than three, like once every three years.
[00:28:05] Adam Hawkins: So like the level of like, maybe for want of a better term, how much you care about the thing that changes every three years compared to the thing that changes every day, there's a different level of commitment. You have to keeping those code bases, you know, well-maintained so like you could, you should also consider changeability and like maintenance concerns when you're setting up these contexts and boundaries and your system.
[00:28:29] Derek Cole Martin: Yeah. And I mean, that's the, that's a thing too. It's like it context does matter.
[00:28:33] Derek Cole Martin: So like I said, as the way things are today in your software and your organization or whatever you're delivering is not obviously going to be the same as what it was. I'm thinking of like the software I'm working in now, it's it was Greenfield five years ago.
[00:28:48] Derek Cole Martin: So it's clearly very different from what it was and how it's got deployed has changed drastically over time too. And when you just said like, oh yeah, I mean, there's parts of the system that change very frequently, which is very true for us. And there's parts of the system that don't change. If not, I couldn't even tell you the last time parts of it have changed because they're just, they're not really the core piece of the puzzle for us.
[00:29:11] Derek Cole Martin: They're just kind of the fringe, a lot of it related to kind of configuration setup that just, it just it's there. It's what is what it is. It does. It doesn't need to change.
[00:29:24] Adam Hawkins: Yeah. So I want to get your opinion on the asynchronous part of this. So like when, and this kind of speaks to one of the questions I have or points I brought up earlier with like, what's the difference between service oriented architecture and microservices?
[00:29:36] Adam Hawkins: And like when you get into microservices, it's not necessarily the software architecture that changes. It's the infrastructure concerns that change because like managing one versus managing a thousand requires it sort of just brings in the assumption that like, to me, I don't know how you feel about this, but when I hear microservice.
[00:29:51] Adam Hawkins: I don't really think software architecture, I think infrastructure architecture, which implies that you're going to do something like Kubernetes or containers or whatever, you know, there's just going to be something like that, as opposed to, Hey, I have one thing, maybe I'm deploying into a VM and you know, it's a different, whole different approach there.
[00:30:07] Adam Hawkins: But as far as async messages go, where do you stand on centralized message brokers or buses for all the different services or like different, different, like messaging streams for the different services?. You know, like if you're say using something like rabbit MQ, you can have a centralized broker and set up all the rules there, or you can do like, it needed to be wise. So you can have, you know, Kinesis streams or flag each per service or something like that. How do you feel about this area of software architecture as it relates to SOA or microservices?
[00:30:43] Derek Cole Martin: Yeah. So that's the comment I hear the most often is like, okay, well, yeah, you're going to messaging. So now you're just completely reliant on a, a message broker it's like, well, yeah, yeah, they have to get distributed somehow. But the idea is still being right, is that you're you don't care who the consumers are. You're just publishing messages and whoever the consumers are so be it.
[00:31:07] Derek Cole Martin: Now, whether you need to. Concern yourself with. Okay, well, where are these message brokers that like are just each, is there a central broker? Is there, I really don't honestly don't have a hard opinion on it because I think it really depends. Generally I've been more of the, especially now with everything being so redundant in the cloud, it's that I'm not so concerned with having a single say like you, whatever you're using, if it's rabbit MQ, or obviously this is not a cue Kafka, but there's a log, but still, I guess if you're in the cloud, I'm not so concerned with having a single single point, I guess it all depends on the number of services that you have, right?
[00:31:52] Derek Cole Martin: Like if you have this little cluster over here, that's five different service or whatever. Say it's 20 microservices here. And those do a lot interactions then. Yeah. Maybe they mean where that message broker is, but you don't even necessarily that can all be abstracted from you too. So I don't, I don't really have a, I think it, I think it depends on the, just the, probably the volume and the size. There was a post recently from I just seen it this weekend. A food delivery service, not Uber eats like a bite squad or something.I don't know where I see. I may skip the dishes maybe. And they mentioned that the move from rabbit to kafka and the reasons that they moved from rabbit were I believe just the volume, the, it couldn't handle the volume they were producing, but they have a whole blog post on it. And I think it makes sense. It's like the, the went through great lengths of trying to basically reimplement what they had with rabbit using Kafka and seemingly from their posts that it's worked well for them, but they had a lot availability issues. So I think that. I think it's one of these things that you grow with. I don't think there's any set rule on you. Okay. You have many, or you have one.
[00:33:15] Derek Cole Martin: I think it depends on the like the ownership model of this particular thing, because if you're going to have say, you know, X number of services and you're going to have to share some common communication method between them.
[00:33:27] Adam Hawkins: Somebody in the entering, team's going to happen on it. And this kind of becomes one of these like horizontal supporting layers that platform, team, ops team, whatever you want to call them, dev ops team, blah, blah, blah. They're going to end up being the arbitrary of that bit of infrastructure. And then how does that relate to the other teams that are publishing messaging, consuming messages? And then if they want to add a new service, like what's the process sort of gets into like one of the other, the other factor, that's kind of the governance model of how the engineering team is going to approach like all of these different services, which I don't know how it's been in your experience, but in my experience, teams don't tend to think of that kind of governance.
[00:34:08] Adam Hawkins: Like what are we going to do if we have a new service or this or that, it's just, oh, let's just start doing it and see what happens. But you've got to do a little bit of planning upfront. Have you seen something like.
[00:34:17] Derek Cole Martin: Well, I think if it's I think, yeah, I think there's a lot of things that we don't think of as developers. I think that's probably, I think it's gotten better though in that sense about, cause I remember a long time ago it's like developers didn't touch infrastructure, right. Where that's, that's come a heck of a long way. Yeah. So, so yeah, but I mean, I think that, I think there's a piece of that is not even just necessarily. Where that infrastructure lives, who owns it. But it's, it's even to the level of how you are using it. So for example, like if, what those streams of data look like and what those, like in, in what those, where you're publishing stuff.
[00:35:02] Derek Cole Martin: So for example, is. If you've ever done anything like this is kind of a, maybe a, a good or bad analogy depends how it goes here. If you're talking about something like event sourcing and you that's another term, unfortunately, that I think has lost a little bit of its definition, maybe, or I think it's getting a little bit has, depending on who you ask, we have a little bit of a different definition to it.
[00:35:33] Derek Cole Martin: Mine, however, is that of a, an event stream. So when people say event streams, I immediately think of event sourcing. I think of the series of events for a particular instance of some entity. So if I'm talking about the prototypical example, here is of a bank account. It's bank account 1, 2, 3, 4. That's my big account number.
[00:35:56] Derek Cole Martin: My that's my stream of all my transactions. There's not one global stream for all transactions ever. It's not partitioned that there's a, there's a stream for that particular, I mean, bank account or whatever it is. If you have an order. It's that order ABC 1, 2, 3, 4, whatever that number is. So how stuff gets partitioned because even that, when you're talking about what I'm alluding to with event sourcing, you have to decide how you're going to partition that.
[00:36:27] Derek Cole Martin: And you're going to have to decide what that model looks like. So I think the same thing applies to however you're publishing this. And where are you on public messages too. So if we're talking about Kafka, if you're talking about like a log, if you're talking about using something like events, store DB as essentially a cue, cause they can do kind of competing consumer stuff too, or whether you're using rabbit or SQS and SNS.
[00:36:55] Derek Cole Martin: I don't, I mean like, yes, somebody has got to manage all of it, but even beyond that, like even before that, it's how is all this getting partitioned? Which is a whole nother concern. Right?
[00:37:05] Derek Cole Martin: So in that ultimately is kind of that between teams, this middle ground of where, where does this all like who's responsibility is this.
[00:37:14] Adam Hawkins: Yeah. And also like what's the sort of the contract between the consumer, the publisher and the consumer. I know I read Martin Clemens, designing data intensive applications, and that book actually changed the way I thought about software because it introduced, I was not familiar with Kafka at the time, but it introduced the idea of there'll be no, the global law. You can replace nothing right there, everything there.
[00:37:36] Adam Hawkins: But the other part of it was Avro with the idea of these like backwards and forward compatible schema is that you could publish a message onto this log at one point in time and then X number of, you know, years or whatever versions down the line still get, you know, something that you could consume.
[00:37:53] Adam Hawkins: And so if you're doing something like RPC or rest HQ, PJ's on whatever you're going to define some, some requests and response format. Whereas if you do messaging, you still have to, you still have to consider that.
[00:38:07] Derek Cole Martin: Yeah, well, yeah, for sure. Like if you're talking about rabbit, right? Like, okay, you consume the message. See you later, it's gone. If you're talking about something like Kafka that you have, whatever retention is not forever that I'm aware of anyways.
[00:38:19] Derek Cole Martin: But if you're using again, if you're using something like events or. And you're creating streams that are consumable. All those events don't go away. They're like if you're creating some projection and you're going back to the beginning of time, well, that event what you created four years ago, whatever that scheme is, it's, that's still what it is. So you need to, yeah, you need to handle that scheme, especially if you're dealing with really old events.
[00:38:47] Adam Hawkins: And one of my favorite aspects of that particular architecture is the idea of. You could recreate state at a particular point in time by just replaying all these streams. So if you're dealing with like say transactional systems or whatever, you just need to have a snapshot of whatever, how many databases are at that point in time and hope that they're all in sync.
[00:39:09] Adam Hawkins: And I think one of the levers that you can pull on in microservice architecture, you know, like if you can adopt like really full asynchronous stuff, you have. Just a really scalable architecture from like the operable side, but also like from the sort of the Conway's law side of being able to change it as teams grow or go away, new requirements come and all that.
[00:39:34] Adam Hawkins: So I think that your focus on asynchronous messaging is really important. And how did you come to that as really important point in the architecture?
[00:39:47] Derek Cole Martin: So this actually just came up the other day and I thought about it for a little bit was because a lot of this stuff way back when I used to do the same type of thing, but in process. And obviously when you're doing stuff in process, if something fails, well, then you have your own code, not library code or whatever you're using that's sitting on top of whatever message broker. You have to implement all your own retry logic, timeouts, circuit breakers, like all this stuff you have to implement yourself. If you're generally trying to write this all yourself, like to do it all in process.
[00:40:25] Derek Cole Martin: To say like, okay, I have this event and I'm going to find all him at the code base. I mean, however you determine what the handlers are for it and the system and you kind of invoke them. But if each, I mean, if, if one, if you have three different events and one of them fails.
[00:40:43] Derek Cole Martin: Well, does the entire initial requests that started that whole thing fail. So that's actually, I think how it started for me is, okay, well I need, I want to have everything be done in isolation, but the cool thing about what I don't think is leveraged enough is thinking about the same paradigm that you can use for messaging for whether it be an event or a command that you're getting from a queue is that you can take that exact same paradigm and think of it as in-process like big, almost becomes indistinguishable. So I mean by that is if you have a request, come in, say from you have an HTTP API and you convert that HTP requests to what I will call just an application request. If you will.
[00:41:33] Derek Cole Martin: So you have some object or message that represents kind of the data that the request of the data. And then if that gets dispatched, so you have some dispatcher and there's a bunch I'm assuming this exists in probably every language and platform, but you mean you have a dispatcher that will take that message and then send it to the, basically the right handler for it. And this is all done in process, but you can take that same notion and then apply that to. Okay, well, instead of sending this in process, I'm going to deliver this to a queue and then that's, what's going to process that message is going to be the exact same handler that you were previously doing it in process.
[00:42:10] Derek Cole Martin: So you end up just, when you start writing your application, everything becomes of handling messages pretty much like right from the get-go. So that's how I kind of, I think started going down this road of asynchrony between things is it's just the being able to deploy things in singly or not deployed execute things in single units, like in isolated and let them have, let them dictate the things that you need, like retries and timeouts, et cetera.
[00:42:40] Adam Hawkins: Yeah. And that speaks to leveraging the capabilities of different technologies like retries and these things. Oh yeah. You can put them in a queue and you'll get all of these other things for free. And then you mentioned single units of functionality, which then feeds back into a better software architecture because now you have the ability to develop and test them independently. You can really say with high certainty that, Hey message X comes in a, B, and C happen. All these, you can just say, boom, I know exactly what's going to happen. And you can then chain all those things together and whatever, you know, it becomes easier to isolate which then feeds into the fact that you can separate them out. You can move these systems in different places and it creates this really nice virtuous feedback loop that if you start to think about it feeds into all these other beneficial areas.
[00:43:29] Derek Cole Martin: Yeah. Thank you. You touched on a big one. There's the testing of it and I think we're also will lead people immediately into is thinking of it kind of automatically brings you into this kind of CQRS. Cigarettes is kind of a gateway drug is what I like to call it. It's super, super simple people conflate it with a pile of us stuff, other stuff like we were taught, like what we've been talking about, but it really is just as simple as commands and queries, separate things entirely.
[00:43:58] Derek Cole Martin: But I found that once you start getting into it and you, especially, if you're applying it with messaging, you get into this mode of vertical slices. So you stop thinking about layers. And you start thinking about verb like vertically. So I always like to say, if you think of like your service or your application, whatever case may be, and you have layers, let's say you have the typical data access layer. You have a business logic layer or domain layer. You have some UI, whatever, whatever the heck you want to call these layers.
[00:44:33] Derek Cole Martin: But generally these things are thought of as kind of like application wide type. When you start getting into vertical slices, you stop thinking about that. And you start thinking about the unit, the single unit of what the piece of functionality is that you want to provide.
[00:44:50] Derek Cole Martin: And then within that unit, you think about layers. So what that, that perf it's, it makes things so much easier to test for one, but two is, if you start organizing code this way, you 100% stop organizing code by layer. And you start putting things that are related together. So you start putting features together. You start putting things very close together. So depending on what language you're in, obviously this works differently, but whether they're in a single folder or single fire. So when you're talking about, oh, I need to go edit the system. I've got to add some feature to some ex or modify some existing feature, say it's to some thing in shipping related shipping.
[00:45:34] Derek Cole Martin: You just, you go to that folder, that's 100% related to that feature and you know exactly what you're doing. You're not jumping around projects. You're not jumping around different files. You're just going specifically to the heart of what you're actually dealing with.
[00:45:45] Adam Hawkins: Yeah. And this vertical slice sort of starts to bring you into the like domain driven design type of thinking, because now you have this sort of isolated area where things are all related to the same thing you probably speak the same internal language inside there, and that when you go vertical slice aid, a vertical slice B they're different on purpose because they are for different things and they can, you know, be their own shape or adopt different even like internals in the software architecture for what makes sense inside that particular box.
[00:46:17] Adam Hawkins: Like I totally agree with your point that like application layers that's like for internal teams, you know, like you might have somebody inside, one of these teams working on these vertical slices, who's better to infrastructure are better at CSS. I mean, please don't ever make me write CSS, but you know, you have people who like to do that. And they're good at that, so great. Like you need those people on your team, like let's, you know, empower them to work at that level inside this whole vertical slice or context of this particular function. Or feature, you know?
[00:46:48] Derek Cole Martin: Yeah. Like a vertical slice can be as big or as slim as you want it to be. It could be a collection of features that could be a single like command.
[00:46:56] Derek Cole Martin: And you just touched on it, which is once you start doing that as well, is that you start deciding dependencies and you kind of mentioned like the shape or the internals within that unit within that vertical slide. So, for example, in this has been, there's been many benefits of doing this. One of the biggest reasons that this is paid off is being able to do that is technology moves so quickly. Your skills change, your knowledge changes. So all of a sudden, if you want to say, okay, we want, we we've implemented some new library or something, that's making our life a lot easier, or we want to stop using one and start using something else, or just how we're doing something. You don't need to change in, rip out the entire service or application to do that.
[00:47:42] Derek Cole Martin: Right. You're not ripping it. I don't, I got to rip out the entire data access, like no, You're doing that, you can start doing that on a per vertical slice feature basis. Right. Cause those things are there. They're siloed. They're, they're pretty much independent generally for the most part. And yeah, you're going to have some shared things between them. You'll like, you'll have some shared concerns, like maybe some data models or et cetera, but for the most part, the are they're little, they're little slices of cake so that, you know what I mean? In those little slices, you can decide what you want to do.
[00:48:12] Adam Hawkins: Yeah. I mean, and that's really what it comes down to when we're talking about the idea of like context and boundaries is that they're there exactly for that reason, if it's inside one code base or across independent services, or even, you know, across groups of services, you know, these, the only thing that I think really matters in software architecture is boundaries. Cause if you, if you don't have that, then you don't, you don't have anything, you know, and if you, if you don't have. The boundaries. You don't have interfaces. You don't have APIs. If you don't have APIs, then what do you, frankly, what are you doing? You know?
[00:48:45] Derek Cole Martin: Yeah. If you don't have boundaries, I like the way you safe. You don't have boundaries. I don't know what, yeah. You don't have anything at that point.
[00:48:50] Adam Hawkins: Right. Because you, I mean, if you come from a system or an environment that has these well-defined interfaces, these boundaries, and then. You know this like ball of mud or like just things were smeared. You just look at as like, I dunno now, what is this? This is, how do you make sense of it? Right? Cause it's here and there and all this and all that is just, it doesn't work. That's why I got, I really liked your video. Context is king. And just the idea of like really beating on the fact that boundaries are really, really important and that it's really worth it to like in the beginning, explore the problem domain with the flashlight, learn the room, forgot what the stuff is and put those things in place. Yeah. So we're out of time now. Close there.
[00:49:33] Adam Hawkins: Yeah. I don't think there's anything else I like to ask you. Is there anything else that you'd want to add to the conversation before we sign off? Good. Okay.
[00:49:42] Derek Cole Martin: Well, Derek I'm sure we could talk more about this. I can tell that you and I are kind of kindred spirits in this in this discussion. So it was a pleasure talking to you. Is there anything you'd like to leave the audience with before we go?
[00:49:56] Derek Cole Martin: Sure. If you want to check out any more of this, just my blog codeopinion.com has pretty much everything we've been talking about. So that's pretty much my focus.
[00:50:04] Adam Hawkins: And you have a podcast too, right?
[00:50:07] Derek Cole Martin: Yeah. So James and other fellow Canadian, we have a podcast called the loosely coupled show. So you can find that on wherever you get your podcasts from, basically. So it's the loosely coupled show.
[00:50:20] Adam Hawkins: What do you talk about on the show?
[00:50:21] Derek Cole Martin: Everything that we're talking about right now, basically and we have guests, you should, we should definitely get you on there. But yeah, it's a little bit all over the map, but a lot of it obviously is about software.
[00:50:32] Adam Hawkins: Yeah. All right. Well, Derek, thank you so much for coming on the show. It was fun and hope to talk to you again.
[00:50:37] Adam Hawkins: Appreciate it. Thanks.
[00:50:40] Adam Hawkins: That wraps up this batch, a small batch of that FM for the show notes. Also find small batches FMon Twitter and leave your comments in the thread for this episode. More importantly, subscribe to this podcast for more episodes, just like this one.
[00:50:54] Adam Hawkins: If you enjoyed this episode, then tweet it or posted to your team slack or rate this show on iTunes. And it helps me produce more small batches. Well, I hope to have you back again for the next episode. So until then, happy shipping.
[00:51:13] Adam Hawkins: Want to learn more about dev ops than wasting your time and sign up for my free email course atfreedevopscourse.com. My course combines the best from the dev ops handbook, accelerate and years of software delivery experience. You'll learn the three ways of DevOps and the four KPIs of software delivery performance. More importantly, I'll show you how to put that theory into practice. That means shipping better software faster. Sign up today at freedevopscourse.com.
[00:51:46] Adam Hawkins: Like the sound of small batches? This episode was produced by pods worth media. That's podsworth.com.