What risks does your company need to be protected against?
If it's the risk of a fire in a data center
or a minor electrical outage, guess what?
Multi-AZ is fine.
You don't need to be multi-region because again,
it's multiple data centers when you're Multi-AZ.
Welcome to Screaming in the Cloud.
I'm Corey Quinn.
You may know my guest today from the reInvent stage or from various
YouTube videos or, you know, blogs, if you're more of the reading type.
Seth Eliot was a principal solutions architect
at AWS and is currently between roles.
Seth, you're free.
What's it like to suddenly not have to go to work at an enterprise
style company, which I think is where you spent your entire career.
No, instead I'm working full time doing interviews.
[laughs] It's another job to have.
It's a lot less process, a lot fewer stakeholders
have to sign off on these things most of the time.
Be your own boss, you know, work from home.
Well, what's that like?
But, you know, hey, you know, after hearing this conversation, anyone out there
thinks they could use my skills as a cloud architect with a focus on resilience.
I'm on LinkedIn.
That's Eliot with one L and one T.
And we will, of course, toss that into the show notes.
Backblaze B2B Cloud Storage helps you scale
applications and deliver services globally.
Egress is free up to three times the amount of data you have stored,
and completely free between S3 compatible Backblaze and leading CDN
and compute providers like Fastly, Cloudflare, Vultr, and CoreWeave.
Visit backblaze.
com to learn more.
Backblaze.
Cloud Storage.
You've been talking for a while now about something
that I've always had a strange relationship with.
Specifically, the Well-Architected Framework Tool Team Process, etc.
at AWS.
What is your involvement with that whole Well-Architected sphere?
So, actually, I was, uh, I had a really interesting job we could
talk about in another question about it as an AWS Solutions Architect
embedded in Amazon.com, not working for AWS, but working for Amazon.
But my first job in AWS was as Global Reliability Lead for AWS Well-Architected.
So there were five pillars at the time.
There are six now.
Let's see if I can remember them.
Operational Excellence, Performance, Security,
Reliability, Cost, and the sixth is Sustainability.
Uh, cost might be of some interest to you, Corey.
I'm not sure.
But I joined that team as Reliability Lead.
There is.
Computers tend to do interesting things in that sense, but we'll roll with it.
A question that I always had when I first heard about
the Well-Architected Framework coming out was, "Oh great,
I might be about to learn a whole bunch of things."
And I've got to be direct, I was slightly disappointed when I first saw
it because it's, well, this is just, stuff is how you're supposed to build
on the cloud, who's worked in operations and doesn't know these things.
And then I started talking to a whole bunch of AWS customers, and it turns
out lots of them, lots of them have not thought about these things because
not everyone is an old grumpy sysadmin like I am and where you sort of learn
exactly why through painful experience you do things the way that you do.
Yeah, exactly.
Um, you know, certainly the performance pillar would play a role
here in terms of this recording and the lag issue, but yeah, all
the pillars are important, and they're a really great resource.
Basically, they're just a set of best practices that
everyone should be knowledgeable about in building the cloud.
And if you take a conscientious decision to not do one
of those best practices, we want you to make sure it's a
conscientious decision and not just a decision of omission.
Um, what I found, though, about it, and you mentioned this when you
talk about the Well-Architected tool is that people tend to think
of Well-Architected as the Well-Architected tool, and it's not.
It's- Well-Architected is the best practices, and the tool is a great resource
by which you could sit down with six of your friends that you work with from
all different parts of the company and do a multi-hour, sometimes multi-day deep
dive review going through each of the best practices and checking the boxes.
And that actually is useful if you're actually the
right people there and have the right conversations.
But what I was finding was people were not doing that.
They were just doing as a checkbox exercise, ticking,
ticking off in an hour, and that's not useful.
So when I joined, I wanted to bring the
best practices to the people, democratized.
So a couple of things that I did, I led the effort to rewrite my pillar,
and all my pillar leads followed suit so that it's now on the web.
It's not a PDF, not an 80 page PDF, but it's on the web with
hyperlinks to each section and hyperlinks to each best practice.
So I want to know about just disaster recovery, I could zip right in there.
And the other thing I did is, like you said, I got on stage.
I wrote blogs.
I wanted to get the best practices into people's hands, how they
wanted it, whether they wanted to do a review or not, and reviews
are useful, don't get me wrong, but whether they wanted to review
or not, I wanted to make sure they had those best practices.
It always felt like something of a maturity model,
where it's easy to misinterpret the, okay, what is your
maturity along a variety of axes and different arenas?
The goal is not to get a perfect score on everything, it's to
understand where you are with these, along these various pillars, to
use your term, and figure out where you need to be on those pillars.
Not everything needs to excel across the entire universe of Well-Architected.
If you, if you're making that decision, it should be a
conscious decision rather than, "Well, apparently cost
is not something we care about here after the fact."
You'll care about it when you get your bill, right?
Um, exactly.
So, that's why I, I presented the best practices as I did.
That's why I would talk one-on-one with customers.
I mean, basically, a Well-Architected review only works well.
If you're having a conversation, if you're just going to
a checkbox, checkbox exercise, then don't even bother.
It's not going to really tell you anything.
So I decided there's too many ways to have conversations.
One is with the tool, and there's another at a whiteboard, and there's
another on a stage, and there's all kinds of ways to have conversations.
And I'm talking about conversations with the engineers and the business, right?
Like, like, if we're going to, like, disaster recovery objectives, guess what?
That's a business decision.
That's not a technical decision.
It's a technical implementation, but your business better, you know, you better
work with them to understand what those disaster recovery objectives are.
In the time before cloud, I was- I wound up being assigned
to build out a mail server for a company that I worked at.
That's where I come from is relatively large scale email systems.
And my question was, "Okay, great.
What is the acceptable downtime for this thing?"
And their answer was, "No downtime is acceptable."
It's- okay, so we'll start with five nines and figure it
out from there, which you won't get in a single facility.
So we'll have that in multiple data centers because it's before cloud.
That'll start at around $20 million.
Um, when can I tap the first portion of the budget, which led to a negotiation?
It turned out what they meant was we kind of like it to mostly be
available during business hours with an added bonus outside of that.
Okay.
That, that dramatically lowered the cost,
but it was a negotiation with the business.
There is kind of this lever that, you know, and then we
talked about pillars being in intention with each other.
There's this lever cost versus resilience, right?
And it's not always that.
You can definitely add resilience without additional
cost by doing certain smart things and optimizations.
But very often it's that cost resilience lever, and I try to talk to customers
about it and say, you have to decide where you want that lever to be.
There's no magic formula that you get, you know,
lowest cost and highest resilience at the same time.
You were embedded in an Amazon team before you wound up moving over to AWS.
Were you working on the equivalent of the
responsibility model or the actual responsibility model?
I confess I don't know much about the internal workings of the Amazon behemoth.
I was embedded in several teams.
I actually started at Amazon in 2005 when it
was this relatively small company in Seattle.
I remember in 2005, like when I moved here and I'd like go to like
a gym and they say, "Oh, we have discount for Microsoft members.
Are you a Microsoft member?"
I said, "No, I work for Amazon."
They're like, "Nah, we don't have a discount for them."
Like you're too small.
[laughs] So, uh, I started out on the catalog team deep into, you know, um, it'd
be the inner workings of later worked on the team that launched Prime Video.
It was called Amazon Unbox at the time.
Uh, if anybody out there remembers that.
I, uh, then took a break for a little period of time.
If you want to call working at Microsoft
a break, it was actually quite rewarding.
For a brief half decade or so.
Yeah, it worked out.
[laughs] I worked at Amazon Fresh, and I worked in international technology.
And then the role I was telling you about before was the most interesting
one where I was embedded as one of only two AWS focused solution architects
in all of Amazon.com, not Counting AWS, but in Amazon.com working across
Amazon teams in, in, in obviously in Seattle, but in Tokyo, Beijing,
Luxembourg, and Germany on their adoption and modernization on AWS.
That was a really cool and interesting job, and I got to
see how, you know, teams build with AWS, a wide variety.
And the number one thing, interestingly enough, is that the cool
thing about internally at Amazon is that cost is a first class metric.
You'd be happy to know that.
Everybody I talk to when I'm talking about system
design and architecture, they ask about cost.
"Should we use Lambda?
Lambda seems cool.
I don't want to maintain servers."
Okay, how much is it going to cost?
We'd have to work through those numbers and make a decision.
Is it worth the cost?
So cost is a first class metric inside Amazon.com.
Which makes a fair bit of sense.
I mean, obviously, things like performance and
durability are as- reliability are as well.
The idea of having a, in fact, you folks, I don't forget if you, I don't
remember the exact timing on this, so you may have been at AWS by then, but
they came out with a shared responsibility model for resilience as opposed
to the one for security, which I always somewhat cynically tended to view
as, well, you have to put it in a shared responsibility model because
when someone gets breached, you need to drag that out for 45 minutes.
You can't just say, you left it misconfigured.
It's your fault.
We wash our hands of you.
Well Corey, you're looking at one of the
co-authors of that shared responsibility model.
So there was already a shared responsibility model for security,
and Alex Livingston and myself wrote a white paper on disaster
recovery, which became very, very popular in late 2020 and early 2021.
And we could talk about why, but we wrote a white paper on disaster recovery,
and in there, we put the shared responsibility model for resiliency,
and that since has been backported into the reliability pillar, too.
And it's important to actually to say, hey, look, you talk about myths, right?
There's a myth On some people's part, it says to be, make it more resilient,
we put it in the cloud, we put it in the cloud, it's now more resilient.
That, that, you know, that is not necessarily so.
You put it in the cloud and you take these steps, being those
steps, being the best practices in the reliability pillar and
the operational excellence pillar, and now it's more resilient.
You can't just count on the cloud to do everything for you.
One of the inherent tensions behind the way to approach resilience
is it cuts, in the, it cuts against the grain of the idea of the cost
pillar, where you get to make an investment decision of do you invest
in saving money or do you spend the money to wind up boosting your
resilience, and it's basically rarely an all-or-nothing approach.
But it's always been tricky to message that from the position of AWS because
it sounds an awful lot like, "We would like to see more data transfer and
maybe twice as many instances in different regions or availability zones.
That would be, well, that's the right way to do it."
It rings a little hollow though.
I have no doubt that was in no way, shape or form your intention at the time.
As I said, it's sometimes a lever.
I mean, but it's sometimes not like if you're running on EC2
instances and you switch over to like Kubernetes and put yourself
on pods, you could be using a lot less EC2 instances and have
a lot more pods in place and you make yourself more resilient.
So it's not always more cost, but if you're talking about disaster
recovery, a question I used to get a lot is, "Do I need to be multi-region?"
Okay, I'm Multi-AZ, multi availability zone, and for folks
that might not know, that means I'm in multiple physical
data centers, but I'm still within a single region.
Uh, that makes me highly resilient if I've taken, or highly available,
I should say, if I've taken the right steps to architect it.
Do I need to be multi-region?
And when I go multi-region, yeah, I'm going to be setting up infrastructure
in another region, and that's probably going to increase my costs.
So when answering that question, I always ask, what
risks are you trying to protect yourself against?
That's again, it's a business question, right?
What risks does your company need to be protected against?
If it's the risk of a fire in a data center
or a minor electrical outage, guess what?
Multi-AZ protection.
is fine.
You don't need to be multi-region.
Because again, it's multiple data centers when you're Multi-AZ.
So then people say why do I need to be multi-region?
And the first thing that comes to mind is, oh, a comet, you know, wiping out
the eastern seaboard, or a nuclear bomb, or, you know, something like that.
And guess what?
That's never happened yet.
I mean, uh, yesterday's performance is no guarantee of
tomorrow's returns, but no comets, no nuclear bombs.
Right.
There's also the idea that, okay, assume that you do lose the entire U.
S.
East Coast somehow.
How much are people going to care about your specific web app?
I'm going to guess not that much in that extreme example.
There's a question of what level disaster are you bracing for?
Yeah, we talked about that too.
Your disaster recovery strategy is part of your business continuity plan.
Do you have a plan for getting workers in front of keyboards?
Do you have a plan for your supply chain if the eastern seaboard is wiped out?
If not, don't worry about the tech right now.
It's not important.
But I was standing in front of a crowd once, teaching a course, and I gave the
whole, you don't have to worry about nuclear bombs for your service, probably.
And then I realized I was like literally in Washington, D.C.
talking to public sector.
I'm like, well, maybe some of you do.
Maybe that is on your business continuity plan.
One thing that I'll see a lot when people try to go with multiple
regions is that they'll very often, "Ah, we don't want a single point of
failure, so we're going to use multiple regions with a second region."
And then after a whole lot of work and expense and
time, they built a second single point of failure.
That's funny.
But, um, but there is a reason, there is a legitimate reason to go multi-region.
And you know what it is.
Do you care to venture a guess on what that is?
In many cases, there's the idea of a few things.
You can separate out, uh, different phases of what's
going on in your environment at different times.
You are, there is a, there's a hard control plane separation.
So if there's a region-wide service outage, theoretically,
you can wind up avoiding that by being in multiple regions.
And, of course, there's always the question of getting closer to customers, too.
Oh, yeah, well, that's not a resilience issue.
That's a performance issue, and that's a legitimate reason to go multi-region.
But, yeah, the number one thing that the actual risk we have
seen, and across all cloud providers, is not a slam on AWS.
All cloud providers is an event, and a
service, or a cloud provider owned network.
And if you have a hard dependency on that service, or, unfortunately,
if you have a hard dependency on other services that depend on that
service, then you need a plan to be able to go to another region.
So, I mentioned how our Disaster Recovery
White Paper became really popular in 2021.
That's because in December 2020, there was a Kinesis event in US East 1.
And I don't know how many people use Kinesis, uh, but that wasn't,
but a lot of AWS services apparently at the time used Kinesis.
So like, I think CloudWatch, I don't, I could be wrong,
double check me on that, but- So, people were affected.
And so if you need to protect yourself against
that, and again, it's all cloud providers.
And actually, AWS likes to show data that is objectively true that they
have the least number of these events of all the major cloud providers.
But if you need to protect yourself against
those events, you need to be multi-region.
Well, when you do talk about multiple cloud
providers, there's also the question of, okay, great.
Well, Amazon themselves is a single point of failure.
The credit card payment instrument that I have on file for my
AWS account is in fact a single point of failure to some extent.
And I'll see companies in some cases storing, uh, rehydrate the business
level backups in another provider, where they're certainly not going
to be "active" active, but they don't have to shut down as a going
concern in the event that something catastrophic happens AWS-wide.
Yeah, again, it's about risk assessment.
If you're afraid of your cloud provider going belly up
in some reason, I think that's a pretty, pretty low risk.
But you're right.
Taking the step of just doing backups of your data and infrastructure to another
cloud provider without any of the operational ability to bring it up quickly.
You're just, this is the, "Oh crap," recovery scenario.
I know it's going to take a long time.
My recovery time is going to be extended, but this is like, I
don't care that it's a long RTO because it's protecting myself
against a risk that's probably never, never going to happen.
That seems legitimate, but yeah, I was just simply trying to
say not protect yourself against your cloud provider or protect
yourself against an event in a region of your cloud provider.
I know of at least one company that winds up having to rehydrate the backups
level of infrastructure and other providers specifically so they don't have
to call it out as strongly as a risk factor in their quarterly filings.
In some cases, it's just easier to do it and stop trying to explain this
to the auditor every quarter and just smile, nod, and do the easier thing.
It comes down to being, again, a business decision.
And that can be a fairly low effort to implement.
It's going to cost you.
Data transfer is going to cost you.
Data storage on the other provider is going to cost you.
But, yeah, unlike a full- so, you know, when I talk about disaster
recovery, I adopted the model that it pre-existed before mid-AWS of four
strategies, backup and recovery, which is the one we're talking about now,.
Very easy to do, but very long recovery times
and longer recovery points, although not always.
And then moving towards shorter recovery times, you could have a pilot light,
or you could have a warm standby, or you could have an active active, which I
consider to be, uh, both a high availability and disaster recovery strategy.
Some of my colleagues at AWS considered it not to be
a disaster recovery strategy, but that's an argument.
I dunno if it's worth getting into.
One thing that I found when I was doing my own analysis for the stuff
that I built, I have an overly complicated system that does my newsletter,
publication every week, and it's exclusively in US West 2, in AWS in Oregon.
And the services that I use are all, as it turns out, Multi-AZ.
So, there's really no reason for me to focus on resilience in any
meaningful sense, because if Oregon is unreachable as a region for
multiple days, well, that week I can write the newsletter by hand
because I think I'm going to have a bigger story to talk about
than they released yet another CloudFront edge pop in Dallas.
No, it's part of the plan, Corey.
They're bringing you down so you can't write about the outage.
But honestly, um, you are resilient, you're highly available, but you're
not, you don't have a disaster recovery strategy because you don't need one.
So, you know, just to be clear, you know,
resilience has that high availability piece.
I'm in multiple availability zones.
I could, I could tolerate component level failures versus
disaster recovery where I need to stand myself up somewhere else.
One mistake I see people making, especially in the multi-cloud
direction, is, okay, we're an e commerce store, so we're going to
either build entirely on Azure, or we're going to go multi-cloud,
so in the event that AWS has an outage, we're not exposed to it.
In practice, what they do is they expose themselves to everyone's
vulnerability, unless they're extremely careful, and if they're, I
don't know, using Stripe to process payments, Stripe is all in on AWS.
So great, we're now living on Azure, but our
payment vendor has the dependency on AWS.
So when there's an actual serious outage, you wind up
with dependency issues that are several layers removed.
Some cases, your vendors know about it, and in many more, they don't.
So when we see things like that giant S3 issue about seven years ago,
well, that's one of those things where everyone's learning an awful lot
about the various interchained dependencies as you go down this path.
Though on the plus side, for most of us, on that day
just the internet is having a bad day, so we don't have
to spend a lot of time explaining why we alone are down.
It's, there's safety in numbers.
First of all, you know, you're still talking about S3 from over a decade ago.
I need to educate you on more recent events,
so you have more recent stuff to talk about.
But, uh, as for your point of dependencies, that's so true, because
we have customers that'll look at, we used to publish, well, we
publish SLAs, but those are not, you know, guaranteed numbers.
Those are just a cost agreement of what any any provider is going to credit you.
And we used to publish the design for reliability.
They since took them away.
It used to be part of the reliability white paper.
And you'd see the number of nines designed for EC2 and S3, and people
would like to try to take those numbers and do the availability map.
Availability map says if I have redundant ones, I could
like, you know, if I have two things that are four nines
and I put them redundantly, then I now have eight nines.
But if I have two things that are four nines
and they're in parallel with each other.
I have to multiply the errors times each other and I have less than four nines.
But, and, and that availability math is well known.
And, and, but you try to do that, it is the way to madness, right?
Because yeah, you've done that with all the components
in AWS, but how about those third party providers?
How about DNS?
How about, The internet, you know, having problems?
You know, like, you're not accounting for all those other things.
So I think you're not getting a real number when you do that.
And even when you're doing DR testing, you have these scenarios where, okay,
we've tested our ability to fail between regions, but what you haven't done
is tested your ability to fail between regions when everyone else in that
region is doing something similar, because it's not just you doing this at
three o'clock on a Tuesday afternoon, suddenly there is a service-wide outage.
We'll, we'll avoid picking on S3 further, but when everyone is starting
to evacuate, you often see, like, even an older issue, we saw that with
EBS volume failures in US East 1, I want to say in 2012, where suddenly
there was the herd of elephants problem that we all learned a lot from.
So Thundering Herd is more of an issue if you're in an
availability zone in a region and that availability zone is
having issues and there's only two more availability zones to go.
So everybody's going to those two availability zones.
That's a real, uh, Thundering Herd issue, especially if
you're looking for EC2 availability, instance availability.
Your- the type you want if you're not flexible,
might be gone by the time you get over there.
When you're talking about multi-region, It's
less so because, especially if you're in the U.
S., you have multiple regions, four commercial regions.
So A, there's no guarantee everybody's even going to the same region.
But B, most people aren't even failing over, right?
Not everybody has a multi-region strategy.
So we actually haven't seen thundering herd happen with multi-region failure.
What we have seen happen is control plane dependency.
So, I actually added this into the reliability pillar pretty late.
It was like the last two best practices I added to my, you
know, to the reliability power, which was don't take hard
dependencies on the control plane if you could help it.
Because the way this works is for every service you use,
if it's a regional service like S3 or EC2, there's a
data plane and there's a control plane in that region.
Data plane is basically the stuff running all the time to service actual
requests, control plane of the CRUD operations, create, modify, update, delete.
The gotcha is, if you're using a global service, like Route 53, at
the time when I last was at AWS, had a single control plane in U.S.
East 1.
And so what happened was, I think this was a 2021 outage,
maybe event, or 2022, we saw an outage in US East One.
It was a network outage and it brought down the control
plane for Route 53 so that people couldn't modify the Route
53 records, which was how they planned to do a failover.
So they couldn't fail over.
Now there are solutions to this.
The solution is choose a data plane strategy instead.
Since then, AWS has come out with Application Recovery Controller, which I
want to hear from you, Corey, what you think of the cost benefit of that is.
It's a little spendy, but you could also roll your own Application
Recovery Controller by doing something like creating a CloudWatch alarm.
Connecting that to a Route 53 health check and having that
CloudWatch alarm not check for health because that's not reliable,
but literally check, is there an object with this name in S3?
If not, alarm.
And then you could delete that object, data plane operation,
the alarm will go off, data plane operation, the Route 53
health check will go off, data plane operation, and it'll swap
It's very helpful.
I, I do like the application recovery controller.
The challenge is it starts at two grand a month, which means for
small scale experiments that, that gets a little pricey just to kick
the tires on and really get a lot of hands on experience with it.
But for the large scale sites that use it, it's, it's who cares money.
They're thrilled to be able to have something like that.
So it's just a question of who the product is actually for.
On the topic of control planes, one of the challenges I've run into in the
past is, it's not just, is the control plane available, but is it latent?
At some point when you have a bunch of folks spinning up EC2
instances, yeah, the SLA on the data plane of those instances is
still there, but it might take 45 minutes to get enough capacity
to spin up just by the time that your request gets actioned.
Yeah, and that's taking a dependency on a control plane.
Even, even if you're Multi-AZ, if your plan is I need
to use auto scaling to spin up EC2 instances in the two
remaining healthy availability zones, that's control plane.
If you want to avoid that, you need to be statically stable and have capacity.
If your costs are three availability zones, then having full capacity in two
of them means you're 50 percent over provisioned in a given availability zone.
Math works, right?
So that's something you have to be willing to pay for.
Or take the dependency on the control plane and it'll probably work.
But you're, you're, you're taking on more risk.
And this, again, is driven by business need.
If you were to take a look at the entire resiliency landscape,
as my last question, this is something I'm deeply curious about.
What do you see people getting wrong the
most that you wish they wouldn't in 2024?
I think in general, what we're looking at is people not understanding
how the cloud is different, I think, when they're moving from on prem.
I'm not talking about your mature folks in the cloud,
but folks looking to adopt cloud for the first time.
It needs to be explained to them that an availability zone
is not only a data center, it's multiple data centers.
And the other availability zone, guess what?
That's a completely separate set of data centers.
So like if your on prem strategies to be in two data centers, whoo two, that
are like 400 miles apart, and that's a really far distance, so there's no chance
those two are going to be affected by each other, even a thousand miles apart.
Guess what?
When you move to AWS or, ya know, at least with AWS's
availability zone model, if your two availability zones are not
400 miles apart, they're only between 10 and 30 miles apart.
But AWS has put in a lot of effort, and I've seen some of these reports.
I've seen the reports include the geological survey and the
floodplain analysis, so that these availability zones are not
sharing the same floodplain, and that if any disaster happens, it's
unlikely that It should affect more than one availability zone.
So guess what?
You don't have to be a thousand miles apart.
You don't have to be 400 miles apart.
Being 30 miles apart is giving you almost that same benefit.
Now, let's talk to your regulator, your auditor, and convince them of that
too, so you don't have to set up in another region a thousand miles away.
I really want to thank you for taking the time to speak with me about all this.
Uh, given that you're currently on the market, if people want to
learn more or potentially realize that, "Huh, we could potentially
use a cloud architect with a resiliency emphasis where we work."
Where's the best place for them to find you these days?
Well, I mean, I'm, you know, just search for me on
your, on your, on your, uh, search engine of choice.
Seth Eliot, E-L-I-O-T.
One "L."
One "T."
Throw AWS on the end of that, you'll probably find
stuff related to me, especially my LinkedIn account.
That's a good way to reach me.
Awesome.
And we will, of course, put links to that.
In the show notes.
Thank you so much for taking the time to speak with me.
I really appreciate it.
Oh, thank you.
It's been a pleasure.
Seth Eliot, Principal Solutions Architect, currently between roles.
I'm Cloud Economist Corey Quinn, and this is Screaming in the Cloud.
If you enjoyed this podcast, please leave a five
star review on your podcast platform of choice.
Whereas if you hated this podcast, please leave a 5 star review on your podcast
platform of choice, along with an angry, insulting comment because despite being
in four different regions, you didn't take all of the control plane access away
from Dewey, who pushed a bad configuration change and brought you down anyway.