Subscribe
Share
Share
Embed
Screaming in the Cloud with Corey Quinn features conversations with domain experts in the world of Cloud Computing. Topics discussed include AWS, GCP, Azure, Oracle Cloud, and the "why" behind how businesses are coming to think about the Cloud.
Anthony: Now, it’s about not only getting them out of the hole that they’ve dug, but also changing the mindset, sort of get in line with really how they should be working in the cloud. And that’s, honestly, the hardest part is changing minds and dealing with the politics.
Corey: Welcome to Screaming in the Cloud. I’m Corey Quinn. I’m joined today by someone who came to my attention relatively recently by releasing simultaneously the best and cheesiest internal marketing video for something that I think I’ve ever seen. Tony Esper is the founder at Cloud Effect, which is a consultancy. Tony, thanks for joining me.
Anthony: Thanks a lot. It’s great to be here.
Corey: So, I’m wondering at this point, what is the story behind that? Because all I know is, I showed up one day in Slack, and you had this great video that you had dropped in that had no company affiliations next to it, it just talked about features of a Golden VPC model, and it looks like the best video effects from something released in the ‘90s. It was amazing. How did it come to life?
Anthony: [laugh] I’m glad you liked it. Well, I had built a Golden VPC module. I had actually brought in Service Catalog for Terraform OS—this basically allows you to run Terraform as a Service Catal—an engine for service catalog, so you can build up your products and service catalog with Terraform. And I had designed this Golden VPC module for my company at the time, and I released it. And I was super pumped about it because it took a long time, and we really needed it, and I was very excited about it, and so I was like, I have to promote this. So, the only—the best way I could think about promoting it was creating a YouTube video for it. I like to do video editing on the side, so I said, “Sure, why not?” And then, in the back of my mind, I was like NBA theme song. I needed to put the NBA theme song, and then I just went from there.
Corey: It was perfect for what it is. And I’ll throw a link to it in the show notes just because it was spectacular beyond absolute belief. So, other than creating these videos—and apparently accompanying modules; like, “I want to do a video, but first I’ll create a module, so I have something to make a video about,” which I assume was what your creative process looks like—but other than that, what is it that you do?
Anthony: I do a lot of stuff. Well, obviously I’m in cloud computing. I’ve been in cloud computing since 2012. I’ve worked with some of the biggest integrators.
Corey: My condolences.
Anthony: [laugh]. I’ve actually had a great ride. I love it. You know, I’m an infrastructure system admin engineer in the background, and then I went through VMware, then I ended up bec—you know, joining a company that was on AWS. I worked under the CIO, I ended up checking into AWS, and I was like, [explosion noise] you know, this is the next big thing. So, I just made the total leap, ended up getting back into consulting and said, “I wanted to focus on Amazon.” And that was in 2012. So, it’s been quite the ride.
Corey: Oso makes it easy for developers to build authorization into their applications. With Oso you can model, extend and enforce your authorization as your applications scale. Organizations like Intercom, Headway ProductBoard and Pagerduty have migrated to Oso to build fine-grained authorization, backed by a highly available and performant service. Check out Oso today at osohq.com today.
Corey: I came from the infrastructure path myself. Did you have that same, I guess, moment of ‘what has happened to this industry,’ that I did, where, historically, everything that I did was at most, if it wasn’t completely manual, it was, “Okay, here’s a small shell script,” and that would be the end of it. And suddenly, it feels like you wake up one day, and oh, if you don’t write code, you’re about to have a bad time. And if you do write code, you’re going to have a slightly better time, but not by much. And suddenly, it was, well, I guess I’m becoming a software engineer, whether I want to or not.
Anthony: It completely happened that way. And I actually really like it. I’m thankful that I was able to, sort of, get into that. I mean, it started off with just shell scripting, right, but now, you know, you start getting into all this configuration languages, then it was a Puppet, Chef, and then it started moving into, okay, hey, CloudFormation. Now, if you’re not, you know, writing Terraform, like, what are you doing?
And now, I think even more, you’re getting into Python. So, Python scripting has become huge with automating things through the SDK. So—you know, through the [unintelligible 00:04:24] library, so I’m actually really enjoying it. I think I would have had to have been forced to get back into development, software development, and I think this is—because it really applies it. It applies it to what you do, right? So, it’s not just as if you’re just writing it for whatever; you’re really leveraging it to do your daily work, so I’m actually enjoying it.
Corey: And oh, it’s not doing what it’s supposed to be doing today. Let’s dive in and fix it really quickly. There’s a very short cycle time. Also, let’s be clear, we talk about the languages we wind up writing in, but a quick check of most of our GitHub repos. Oh, it’s 94% YAML. Of course it is. Because it feels like that has become the world’s most common programming language, whether we want it to or not.
It is, in fact, not. What is, is—believe it or not—Microsoft Excel.
Anthony: it’s a markup.
Corey: Yeah, it seems like everyone has some critical business process that’s stuffed over in that. I am curious as far as going in the consulting direction you have, which… I went in a slightly different direction where all I do is purely advisory. You are actually getting your hands dirty and writing code in the muck—for lack of a better term—for a lot of your customers. How did they wind up approaching the cloud? Was it something that has already been entrenched when you got there, or were you brought in to effectively lead the charge, or a little from both?
Anthony: I’d be so happy if I was able to get in there and greenfield. That is just not the case. And I’ve worked with so many different clients. I’ve worked with clients in a vast array of industries all over the United States, and I get in there in all different maturity levels. And so, what I really looking to do probably in the later stages right now of my current career is I tried to mature them.
So, I really—I can probably tell within a couple days where they’re at—maybe less—and then from there, once I understand where they’re at in their AWS maturity process, then I’m thinking about how I can get to the next stage. Some of the companies I’ve worked with, they’re super mature, very, very mature, some of the large insurance companies that I’ve worked with, they’re highly, highly mature, building things that I love, like, roles as a service, where you’ll put a request to get a role created for your account through repo, and then it will do a series of checks to make sure all your permissions are right, and then issue that role to your account. I mean, awesome stuff that is really solving problems that the enterprise really needs to solve. But I’ve also worked at companies where they’ve dug themselves in, in ways they shouldn’t have, and now it’s about not only getting them out of the hole that they’ve dug, but also changing the mindset to sort of get in line with really how they should be working in the cloud. And that’s honestly the hardest part is changing minds and dealing with the politics.
Corey: Have you found that there are some decisions that are easier to back out of than others, like sort of the one-way door approach? I take a look at even the stuff that I set up for internal use when I started on this path myself seven years ago, and there are things I would love to just burn to the ground and start over with, except I can’t for a variety of reasons. These were very difficult, if not impossible decisions to back out of, that I didn’t even realize that I was making at the time. And conversely, you then encounter people in the world who were convinced that, “Well, this is how our CI/CD process works and there’s no way we could ever migrate off—what do you mean you just reimplemented a new one?” There are things that are trivial to switch off of, and then there are things that are impossible. The hard part is, I think, discerning what those tipping points are.
Anthony: Yeah, and that’s where you really need diligence when you get into this stuff. And what the heck, Corey? Why would you build something that you later wanted to—realize was a mistake, and you need to back out of. What were you thinking, man?
Corey: Yeah, you should have been smarter the first time. Also—
Anthony: [laugh].
Corey: —we don’t need QA or testing if you just write code correctly the first time. I don’t see why this is a problem. Yeah. I… and heaven forbid functional requirements wind up shifting.
Anthony: Tests schmesh.
Corey: There’s no way to win. It’s one of those areas where consulting is absolutely at its most valuable. It’s, great, okay. So, we’re late coming to this particular part of the world. What best practices have emerged? What should we learn from that other people have gotten wrong, is one of the best questions. Unfortunately, it feels like we get tagged in after that would have been a terrific question, and now we have to either backtrack or find a way to make do with the way that things are today.
Anthony: That’s right. Yeah, you get in there when it’s—they’ve already started rolling. And that’s kind of one of the major issues with the cloud is that it’s just so easy to get rolling, in the sense that you could just get in there and start clicking around. Oh, I’ve heard a new term, by the way, Corey, at this recent client I’m working on, called ClickOps, which is when you click—[laugh]—
Corey: I love the term.
Anthony: [laugh].
Corey: Yeah, I came up with a few years ago. I’m not sure who started it, but I had a meme that blew up, which was, like, the gat—the small-brain galaxy brain explosion about Infrastructure as Code, where, like, Phase One is clicking around to the console; Phase Two is Terraform or CloudFormation; Stage Three is the CDK; but the final form is ClickOps, which is using the console, but then lying about it. And that is the way that everything seems to progress. Everyone loves to say, “Oh, we’re full IaC. Except for these parts over here. Don’t look at them.” And of course, people are clicking around in the console to build these things.
Anthony: Well, sometimes a little click is okay.
Corey: Exactly. And that’s the slippery slope we all go tumbling down.
Anthony: Yeah, it’s a little of a slippery slope. I think writing IaC is really important if you need to have something repeatable, and you need to—other people need to work on it, and you want it in a state. I think that’s when you really need it, especially you keep developing repeatable stacks, or you need other people to keep developing on it, and it needs the land in a state. If it’s a one-time thing, you know, maybe not so bad. So, I think there’s definitely a line and some give and take.
Corey: I found a terrific way to sort of square the difference. It’s hanging out on AWS’s serverless app repo—which nobody remembers exists, including AWS—and it’s effectively a ClickOps detector. It winds up watching CloudTrail management events for anything that is not set to a read-only event and has a console identifier as the user agent. And then it just fires off an alert saying, “ClickOps detected.” And I love that because I don’t want to stop people necessarily from doing these things, but I want to know that it’s happening because oh, okay, that is drifter on something that is sensitive. Let me jump in and make sure that that’s not turning into something disastrous, just from a reportability perspective, if nothing else.
Anthony: There’s a ClickOps detector?
Corey: There absolutely is. And—
Anthony: [laugh].
Corey: Getting that working more and more effectively with time has been helpful. What I love personally is when they first launched Amazon Q—you know, the bad chatbot that nobody likes the pops up all the time?—it was firing off non-read-only events into the whatever account you happen to be logged into in that browser session at that moment, and suddenly, it was spewing stuff into the logs. And I started screaming about it, and they realized, “Oh, we just built this last night, so yeah, let us fix that now.” And then suddenly it stopped. But it was awful.
Anthony: And you program it to set off a sound effect [annoying klaxon sounds]?
Corey: Oh, that’s the beautiful thing. The chatbot fires off on an SNS notifier that I intercept with Slack, but you can do it with a bunch of things. Oh, yeah. Build out a big klaxon and a siren, though, that’s hard to do with remote and work from home approaches these days. Like, you have to shipping them individually to everyone, and mandating they keep them up, and monitor them to make sure it’s there. Suddenly, you wind up with a turtles-all-the-way-down type of architecture. Not that I’m saying I wouldn’t do it. I think it’d be hilarious. It just, you know, has some shipping logistical challenges.
Anthony: What’s ‘turtles all the way down?’ What is that?
Corey: Oh yeah, like the idea that oh, you have a giant—the world is built in the back of a giant turtle. Great. Awesome. What’s the turtle standing on? Another turtle. And it just winds up effectively being turtles all the way down. Some wit was said that about a theory of what the universe was. And you know, honestly, it’s about as plausible as almost anything else. You wind up with complexity layered on top of complexity layered on top of complexity, and fundamentally, no one knows how the whole thing works.
I mean, that’s inherently the problem I always had moving up the stack into some of the modern languages and frameworks. Like, I wind up running some application someone wrote on Node. I grab it off a GitHub, and it has a whole bunch of stuff scrolling by after I do ‘npm install’ and it’s, wow, that’s an awful lot of stuff I never understand, I’m never going to dig into. Sure hope it’s doing the things that you want it to be doing. Primarily, it’s just—it seems—just a way to get Dependabot never to shut up about anything on GitHub ever again and obscure things that are actual problems with, “Oh, this thing that will never see anything but a sanitized input now has a problem. Let’s freak out about it.”
Anthony: Ugh. Yeah, I know what you mean, it’s a lot of, like, complexity you run into sometimes, especially, you know, some of the software you pull off the public interwebs, where it’s just all of a sudden, you know, it’s pulling this and pulling that when you go to do the install, and you have no idea what’s going on. Reminds me of, actually, Account Factory for Terraform, which is pulling in a lot of under-the-covers repos and libraries that you didn’t even realize are there. And then you have to go—when you actually need to solve, like, a deep problem, you have to go troubleshoot that to the nines, you know? So, I know what you mean. Turtles on turtles.
Corey: No one is excited by the prospect of building permissions – except for the people at Oso. With Oso’s authorization as a service you have building blocks for basic permissions patterns like RBAC, ReBAC, ABAC and the ability to extend to more fine-grained authorization as your applications evolve. Build a centralized authorization service that helps your developers build and deploy new features quickly. Check out Oso today at osohq.com today.
Corey: Oh, yeah. Then you have this just a really unfortunate situation where you’re—like, at least in my experience, I’ve been kicking around Kubernetes lately for a talk I’m preparing. So, I built a cluster locally. And the problem I’m running into now is that, okay, this just sort of kind of works, but I don’t know how it breaks, and I don’t know how to expect it to behave when it breaks. And coming from an ops background, I think the reason that it would be very hard for anyone to displace AWS in a lot of shops is you have an entire generation of engineers who know exactly what happens when it breaks. The status page will not update, you’re going to start seeing weird issues.
You’re going to check social media and back channels to see if anyone else can do this. Check Downdetector, which AWS hates, but they’re not going to come up with anything better for us. Eventually they admit the problem, and if you can find that, like, showing up in the glacial record at the same time. And then, okay, now we know how this progresses and what it looks like. But with other providers, you get to experience this anew.
Like, okay, what’s going on? How is it going to manifest as a breakage in an underlying system? Is the API just going to lie to me? Is it going to timeout? It’s going to throw an error? Is it, worst of all, going to tell me everything’s fine? And it’s a matter of not just understanding how something works, but understanding the patterns when it doesn’t work.
Anthony: Yeah, and I think that’s really getting under the covers. And for me, I don’t know what better way there is to figure that out than just running with it. And when it does break, you have to end up getting in there and figuring it out. And I mean, I think that’s kind of how we ended up learning a lot is things break, and when they do break, you have to kind of get under there and, you know, figure out what it is that broke by figuring out how this thing works. And I think that can be said for a lot of what we run today in the cloud, mostly a lot of software applications and things like that.
Now, you’re talking about people running Amazon, if they want to see, like, what happened, they go to Downdetector, they go to all these status pages, and they can sort of see, you know, what’s the story here, and what’s down, right? But then when you run these kind of software stacks—like you were saying, you spun up a Kubernetes, locally—you’re not going to be sure where to go when you troubleshoot, right? Is that what kind of what you’re saying?
Corey: Sort of. It comes down first to—like, whenever I was dealing with outages back in my hands-on keyboard running things days, like, the first question was always, “Okay, is this a global problem, or is this something that has to do with my code running in production, possibly the last change that wound up happening?” And the sooner I can divide the world into those two equal parts and figure out what side it is, the better everyone’s going to be. Because until then, roughly half your team is going to be scurrying around trying to figure out the answer to that question. And I don’t necessarily need precise answers about what’s broken in AWS right now in this particular region, but as long as I know that suddenly, there’s a bunch of chatter on social media about people seeing weird things, I can then theorize that it’s probably not my specific environment.
Whereas if everyone else is having it just fine, and suddenly it’s me, okay, what did we just change? What’s happened? Did something get pushed recently? Did we run out of some resources in the account? Are we smacking into rate limits? It helps us decide very quickly, what side of the world we’re on as far as problem resolution goes.
Anthony: Quick deduction.
Corey: Yeah. And then you have things like Kubernetes, which are the least cloud-native thing in the world, where it’s great, we’re going to abstract a cloud provider on top of an existing cloud provider. And then it’s, okay, is it underlying Amazon problems? Is it that we’re cosplaying as something we don’t understand and then Kubernetes is broken yet again, for some reason? Or is it the application riding on top?
And this is sort of—at least in my experience—where modern trend of observability came from. How do you wind up getting valid answers from ephemeral things that stopped existing 20 minutes before you actually think to look at it? Anything that had put out had better be enough to start doing the diagnosis. I don’t love it, but it’s definitely keeping me agile in my old age.
Anthony: Are you enjoying your Kubernetes Lab?
Corey: I’m running it locally, and that fine. The reason I didn’t do it in the cloud, to be very direct, is despite my level of expertise with AWS billing structures, I wasn’t convinced that I wasn’t going to sign up for a $5,000 oopsie just because something wound up getting carried away. I’ve already found that this ridiculous board cluster is spitting out a gigabyte a logs a day to some random thing.
Anthony: That’s because you do too many ClickOps, Corey.
Corey: It must be.
Anthony: [laugh].
Corey: It absolutely must be. And good, now we just wind up running the imperative commands with the Kubectl ‘apply.’ Great. Awesome. Maybe that’ll work. Maybe it won’t. Now, you have these files that diverge from what’s actually running, and everyone’s back in hell. Yeah, it’s the same problems, just written different ways.
Anthony: And are you enjoying your local build? Are you feeling good about it? You’re doing pods and running containers? And—
Corey: I’m also—because I’m building this on a bunch of Raspberries Pi built into a few different structures here, and I’m reminded of a whole class of problem that I haven’t had to deal with in a decade when I was in data centers, which is things like, oh, I need to order some cables. Oh, Amazon keeps telling me they’re delayed another day, so I finally go to the store and buy some, and problem solved. I have to deal with weird issues around failing drives and strange power fluctuations. These are all things that cloud providers have done an excellent job of abstracting away. Expanding a disk volume is an exercise in a bunch of rote steps that should work the way you’d hope, and in practice, often don’t.
Whereas it’s an API call away and then just expand the file system, and you’re done on an EC2 environment, if that’s even how you want to treat these long-term things, rather than just changing some Terraform somewhere and not thinking about it again. It’s a whole series of problems that I’m not saying we’ve forgotten how to solve them, but it’s awfully nice for the past decade not having to think about these problems in any meaningful way. I mean, yeah, I’m saving a bunch of money on Cloud bills, but I also have spent a lot more time chasing these things down than I would have otherwise.
Anthony: I guess, maybe part of the dream of the cloud has come true. We’re putting more time developing good stuff than having to go and, you know, backfill and maintain.
Corey: Well, now the dream has been replaced. So, now we just write a whole bunch of Lambdas to glue together service gaps in AWS services because apparently they’re not allowed to talk to each other because they think an NDA applies to other service teams.
Anthony: [laugh].
Corey: Great. Awesome. That has been a constant source of frustration and concern. I’m curious to get your take. If someone’s getting into the cloud these days, and—as a serious way as an enterprise, not some random startup that’s chasing an idea—what should they do to avoid so much of the pain that people who have adopted earlier have experienced on their behalf?
Anthony: Look at Well-Architected Frameworks, start strategizing, talk to somebody who knows what they’re doing, and start planning. Before you get into it in a serious way, like actually having apps that are part of your corporate workloads actually running in the Cloud: plan. So, it’s like building a city or a town, right? I could go out, and I could just start building stuff. Like, I could go get some, you know, whatever, start building houses, places, and doing things like that, but if you didn’t plan it out, right, if you didn’t decide, okay, here’s where I’m going to put the town square, here’s where I’m going to put my residential zone, here’s where I’m going to put the, you know, the commercial zone, here’s where I’m going to put a park, here’s where I’m going to put these things, you end up with a city that’s all over the place, right, and a mess because you decided to just go at it.
I really think of clouds kind of like city planning. Like, you want to plan out, where’s my InfoSec going to be, you know? Are we going to be doing shared CI/CD? Where’s that going to live? How’s the account structure going to work? How are people going to actually work in the cloud when they get in the cloud? How are they going to actually start getting in there and developing? How are we going to provision access, right? How are the workload accounts going to be provisioned? How’s everything going to be structured, right?
And you want to start thinking about that. How do we start training people on how we start leveraging the cloud when they come in? What’s our current skill gaps, right? Let me look at my human resources right now. Like, where are we with the cloud? Like, how much do they understand? Do they understand a little bit? Do they understand—are they HashiCorp experts? Like, where are we with them right now? And so, how do we get them from A to B, right?
And so, I would definitely advise taking a very strategic approach, and working with a expert that knows how to plan a strategy for AWS, rather than thinking that, you know, you can go and sort of watch a couple YouTube videos, or, you know, think it through and actually just start going at it by clicking next, next. And, you know, I think a lot of people get into timelines, too. I think they start getting rushed, sometimes for necessity. But even if you take just even some small-time to plan before you execute, I think you’re going to be in a lot better of a position, even if you do have timetables to meet because of whatever reasons.
Corey: I had a client recently where their account structure was an absolute thing of beauty. I mean, you look at most companies that have been around for a while on the cloud, and you have one weirdly named account that still has a whole bunch of resources in it because it used to be their only account, tied to a Gmail address that their founder used when they registered the company many years ago, and it’s still tied their amazon.com account—
Anthony: [laugh].
Corey: —great, and yeah, and they’ve been trying to move things off of it. But this was one of the absolute most beautiful account structures I’ve ever seen. It was, “Wow, you folks are great. How’d you do this?” And they said, “Well, we sort of dragged our feet and didn’t really get into cloud until a couple of years ago.” “Oh. Yeah, then you did a smart thing and talked to people who’ve been down this path already, rather than decided to YOLO it by, ‘I’m going to click and find out.’” which is often a great way to do things in a test account, but some of these mistakes have repercussions, and these decision points that you don’t even realize you’re crossing will come back to haunt you.
Anthony: It’s a mess. It does come back to haunt you. And I think [laugh] you hit the nail on the head. Don’t YOLO [laugh]. Just do not YOLO it. Like, this—I see so much shooting from the hip, and it’s like, what are we doing here? I think you just have to have that certain mindset. I don’t know what it is, I don’t know—I mean, to be a hundred percent honest with you, and not to pat myself on the back, but I have it. Like, I really like to have forethought and planning. I mean, I’m an architect by nature, so I—you know, by—whatever you know what I mean?
So, my AWS nature is to plan things. My Lucidchart account, I probably have [laugh] you know, I probably have hundreds and hundreds of, you know, architecture diagrams, and topologies, and flowcharts, and things like that. So please, no YOLO. Don’t shoot from the hip. I think people have a mentality where they just want to… they just want to get it done, right, and they’re just going to next, next, next, and then have it up, and then just try to—you just—you know, you don’t—you’re not thinking about what services to use, like, how to actually build it, from the account topologies down through the application stacks, through, where’s our CDI CI/CD going to live?
How are we doing identity management? Where are we storing our repos? What are the conventions that we’re going to use? What’s our tagging structure? I mean, all this type of stuff. So—how are we doing InfoSec? I mean, all this kind of stuff, that is a later thought that really should be in the forefront before you even dip your toe in the water. That’s why I love it when I told you I can get in there on a greenfield. Because if it’s greenfield, then we can actually get it done right. I don’t know what it is with the mentality of shoot from the hip.
Corey: In my experience, you’ll see a lot of ground-up adoption. It’s not usually something that is strategic when people are first dipping their toes, at least historically. So, it’s someone in their spare time, okay, I’ll spin this small thing up and see what happens, and suddenly it becomes load-bearing in production, almost by accident. Other times, I think a lot of that is the AWS documentation. Reading through a lot of the well-architected guides on how to build these things from scratch, day one, it has so many different overly complicated sounding things, and they do it terrible job of explaining it.
If you follow the docs on how to get federated identity setup with the service formerly known as AWS SSO, it is complete gobbledygook unless you have a background in managing directory services. But it’s not actually that complicated; they just do a terrible job of describing how to do this, and explaining why you want it. And by default, it just leads you to oh, here’s how to generate a strong password and credentials for the API in the root account, which you should never do. Now, go ahead and build IAM users instead, which you then get yelled at, which you should never do. And it becomes this, great, then stop making it easy for me and make the easy path, the low friction path, the right way.
Anthony: Mmm.
Corey: And I think that they’ve just failed that across the board. I mean, look at all the things that your Golden VPC does, and how finicky and annoying it is to dial that in the first time. But you now have built a template where it’s push the button, grab a cup of coffee, by the time you’re back, it’s up and running, and there’s nothing else to worry about. Why do you have to build that mechanism? Why isn’t that something that you can, while you’re clicking around in the horrible AWS console, maybe have that as something that’s a global pattern that fits 90% of use cases?
If you’re that exception case, like, “Oh, we’re a bank. We need to be more secure.” Great. You already have a security apparatus. You should understand this. Whereas if you’ve’re a kid getting started, maybe you shouldn’t have to configure $2,000 worth of security services a month by accident, and discover that the free tier does not mean what you think it did. There’s a spectrum in user experience that the console—today at least—is completely blind to.
Anthony: I think they’ve tried to do a better job, especially with landing zones from Control Tower. I mean, they’ve, they’ve—they’re—[laugh] I think, you know, who’s kn—like, let’s go back, you know, ten years, twelve, fifteen years. Like, did they ever expect to be where they are right now? Like, I’m trying to give Amazon a little bit of a benefit of doubt right now. I mean, yeah, they’re a billion-dollar organization—
Corey: I used to, but they’re worth one-and-a-half trillion dollars in market cap. It feels like they could hire someone who has better opinions on the UX than I do. I would fix it this way, but I’m not the one-and-a-half trillion-dollar company. And their response is, “We’re frugal. We can’t hire anyone.” “Okay.”
Anthony: Okay, you want a really good example of way over—like, crazy complexity that just got built up over time? I think there’s parts of Amazon in the back-end that have been a lot of just, from the ground up, you know, a little bit of, you know, shoot from the hip, look at System Manager.
Corey: Oh, my God, yes.
Anthony: Ever dove into that?
Corey: Which part of it? Because it feels like there’s 17 services in there, most of them pretty decent, all called ‘Systems Manager,’ but they bear little, if any, relation to one another.
Anthony: [laugh] It’s crazy. And they don’t even work for, like, half o—most of them really do not work for a multi-account model, which is baffling. Like, I cannot do—I had to build a whole custom solution in order to do, you know, inventory and patching through System Manager for a multi-account model. Didn’t exist. There wasn’t even a—what are they called? The solution blueprints or whatever that you could go spin one up? It didn’t even exist. I had to go and make it.
Corey: Yeah, you want to patch things individually by hand in every account. Sometimes you’ll see the same thing, but it doesn’t realize there are other regions. Like, they have damn near 30 of them now, which each is sort of its isolated own cloud account within your Cloud account. So, how do you—you multiply those and get a matrix, and wow, you’ve got a lot of clicking to do if you’re not going to automate this in some way. Why do we have to automate it? Why can’t they?
Anthony: Proprietary mark open System Manager documents.
Corey: Oh, dear Lord.
Anthony: Yeah. I mean, that was a ball of wax.
Corey: And that’s what you have to use to opt out of AI. And there’s no validator that says you got it right. Like, they’re going to train on AI services unless you get the magic incantation right, and hope for the best.
Anthony: Oh, really? I was not even aware of that.
Corey: It’s fun. See, things buried in terms of service, they’re always pleasant.
Anthony: [laugh].
Corey: Ugh. I really want to thank you for taking the time to speak with me. If people want to learn more about how you view these things, and reach out to you for help, where’s the best place for them to find you?
Anthony: Find me on LinkedIn.
Corey: Awesome. And we’ll definitely put a link to that in the show notes. LinkedIn is the modern resume. Everyone has that. It’s the… it has replaced, in many respects, the formal resume. In fact, some people just submit the ‘generate this as a PDF button’ on LinkedIn, and here you go. It’s as valid as anything else that tells a story. Thanks so much for being so generous with your time. I really appreciate it.
Anthony: Corey, it’s been great. I appreciate you having me on the show. It’s been great to talk shop with you.
Corey: Tony Esper, founder at Cloud Effect. I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud. If you’ve enjoyed this podcast, please leave a five-star review on your podcast platform of choice, whereas if you’ve hated this podcast, please leave a five-star review on your podcast platform of choice, along with an angry comment that you’re of course going to have to type in by hand because automation around these things had never occurred to you until this exact moment.