We’re used to talking about networks, applications, code, and network.
We’re not used to talking about and even have
vocabulary for what makes something confusing?
What makes something ambiguous?
And that is what makes for effective incident analysis.
Welcome to Screaming in the Cloud.
I’m Corey Quinn.
I’m joined this week by John Allspaw, who’s—well, he’s done a lot of things.
He was one of the founders of the DevOps movement—although I’m sure someone’s
going to argue with that—he’s also written a couple of books, The Art of
Capacity Planning and Web Operations and the foreword of The DevOps Handbook.
But he’s also been the CTO at Etsy and has gotten his Master’s in Human Factors
and System Safety from Lund University before it was the cool thing to do.
And these days, he is the founder and principal at Adaptive Capacity Labs.
John, thanks for joining me.
Thanks for having me.
I’m excited to talk with you, Corey.
This episode is sponsored in part by my day job, the Duckbill Group.
Do you have a horrifying AWS bill?
That can mean a lot of things.
Predicting what it's going to be.
Determining what it should be.
Negotiating your next long term contract with AWS.
Or just figuring out why it increasingly resembles a
phone number, but nobody seems to quite know why that is.
To learn more, visit duckbillgroup.
com.
Remember, you can't duck the duck bill, Bill.
And my CEO informs me that is absolutely not our slogan.
So, let’s start at the beginning here.
So, what is Adaptive Capacity Labs?
It sounds like an experiment in auto-scaling, as is every
use of auto-scaling, but that’s neither here nor there.
I’m guessing it goes deeper.
Yeah.
So, I managed to trick, or let’s say convince some of my heroes, Dr.
Richard Cook and Dr.
David Woods, these folks are what you would call heavies in the
human factors, system safety, and resilience engineering world, Dave
Woods is credited with creating the field of resilience engineering.
And so what we’ve been doing for the past—since I left Etsy is bringing
perspectives, techniques, approaches to the software world that are, I guess,
some of the most progressive practices that saved other safety, critical
domains, like aviation, and power plants, and all of the stuff that makes news.
And the way we’ve been doing that is largely through the lens of incidents.
And so we do a whole bunch of different things, but that’s the
core of what we do is activities and projects for clients that
have a concern around incidents; both, are we learning well?
Can you tell us that?
Or can you tell us how to understand incidents and analyze
them in such a way that we can learn from them effectively?
Generally speaking, my naive guess, based upon the times I
spent working in various operations role has been, “Great.
So, how do we learn from incidents?” Well, if
you’re like most of the industry, you really don’t.
You wind up blaming someone in a meeting that’s called blameless, so
instead of using the person’s name, you use a team or a role name, and
then you wind up effectively doing a whole bunch of reactive process
work that in long enough timeline and enough incidents ossifies you
into a whole bunch of processes and procedure that is just horrible.
And then how do you learn from this?
Well, by the time it actually becomes a problem, you’ve rotated
CIOs four times and there’s no real institutional memory here.
Great.
That’s my cynical approach, and I suspect it’s not entirely
yours because if it were, you wouldn’t be doing a business in
this because otherwise, it would be this wonderful choreographed
song-and-dance number of, “Doesn’t it suck to be you?
Da-da.” And that’s it.
I suspect you do more as a consultant than that.
So, what does my lived experience of terrible companies
differing in what respects from the folks you talk to?
Oh, well, I mean, just to be blunt, you’re absolutely spot on.
[laugh] . The industry is terrible at this.
Well, crap.
I mean, look, the good news is, there are inklings, there are signals for
some organizations that have been doing the things that they’ve been told
to do by some book or website that they read, and they’re doing all the
things and they realize, “All right, well, whatever we’re doing doesn’t
seem to be—it doesn’t feel—we’re doing all the things, checking the boxes,
but we’re having incidents”—and even more disturbing to them is we’re
having incidents that seem as if—it’d be one thing to have incidents that
were really difficult, hairy, complicated, and complex, and certainly
those happen, but there is a view that they’re just simply not getting
as much out of these sometimes pretty traumatic events as they could be.
And that’s all that’s needed, yeah.
In most companies, it seems like, on some level, you’re
dealing with every incident that looks a lot like that.
Sure, it was a certificate expired, but then you wind up
tying into all the relevant things that are touching that.
It seems like it’s an easy, logical conclusion.
Oh, wow.
It turns out in big enterprises, nothing is straightforward or simple.
Everything becomes complicated, and issues like that
happen frequently enough that it seems like the entire
career can be spent in pure firefighting reactive mode.
Yeah, absolutely.
And again, I would say that just like these other domains
that I mentioned earlier, there’s a lot of, sort of, intuitive
perspectives that are, let’s just say, sort of unproductive.
And so in software, we write software; it makes sense if all
of our discussions after an incident trying to make sense of
it, is entirely focused on the software did this, and Postgres
has this weird thing, and Kafka has this tricky bit here.
But the fact of the matter is, people and—engineers and non-engineers—are
struggling when an incident arises, both in terms of what the
hell is happening, and generating hypotheses, and working through
whether the hypothesis is valid or not, adjusting it if signals
show up that it’s not, and what can we do, what are some options?
If we do feel like we’re on a good [unintelligible] productive thread
about what’s happening, what are some options that we can take?
That opens up a doorway for a whole variation of other questions.
But the fact of the matter is, handling incidents, understanding really,
effectively, time-pressured problem solving, almost always amongst
multiple people with different views, different expertise, and piecing
together across that group what’s happening, and what to do about it,
and what are the ramifications of doing this thing versus that thing?
This is all what we would call above-the-line work.
This is expertise.
It shows up in how people weigh ambiguities, and things are uncertain.
And that doesn’t get this lived experience that people have,
it just we’re not used to talking about—we’re used to talking
about networks, and applications, and code, and network.
We’re not used to talking about and even have
vocabulary for what makes something confusing?
What makes something ambiguous?
And that is what makes for effective incident analysis.
Do you find that most of the people who are confused about these things
tend to be more aligned with being individual contributor type engineers,
who are effectively boots-on-the-ground, for lack of a better term?
Is it high-level executives who are trying to understand why
it seems like they’re constantly getting paraded in the press?
Or is it often folks somewhere between the two?
Yes.
[laugh]
. Right?
Like there is something that you point out, which is this contrast
between boots-on-the-ground, hands-on keyboard, folks who are resolving
incidents, who are wrestling with these problems, and leadership.
And sometimes leadership who remember their glory days of being
an individual contributor sometimes are a bit miscalibrated.
They still believe they have a sufficient
understanding of all the messy details when they don’t.
And so, I mean, the fact of the matter is, there’s
the age-old story of Timmy stuck in a well, right?
There’s the people trying to get Timmy out of the well, and then
there’s what to do about all of the news reporters surrounding the well
asking for updates and questions, and how did Timmy get in the well?
These are two different activities.
And I’ll tell you pretty confidently, if you get Timmy out of the well, pretty
fluidly, if you can set situations up where people who ostensibly would get
Timmy out of the well are better prepared with anticipating Timmy is going
to be in the well, and understanding all the various options and tools to get
Timmy out of the well, the more you can set up those and have those conditions
be in place, there’s a whole host of other problems that simply don’t go away.
And so, these things kind of get a bit muddled.
And so when you say ‘learning from incidents,’ I would separate
that very much from what you tell the world externally from your
company about the incident because they’re not at all the same.
Public write-ups about an incident are not the results of an analysis.
It’s not the same as an internal review, were the review to be effective.
Why?
Well, first thing is you never see apologies on internal
post-incident reviews because who are you going to apologize to?
It’s always fun watching the certain level of escalating transparency as
you go up through the spectrum of the public explanation of an outage,
to ones you put internal customers, to ones you show under NDA to special
customers, to the ones who are basically partners who are going to fire
you contractually if you don’t, to the actual internal discussion about it.
And watching that play out is really interesting.
As you wind up seeing the things that are buried deeper and deeper, yeah, you
wind up with this flowery language on the outside, and it gets more and more
transparent, and at the end, it’s, “Someone tripped and hit the emergency power
switch in a data center.” And it’s this great list of how this stuff works.
Yeah.
And to be honest, it would be strange and shocking if they weren’t different.
Because like I said, the purpose of a public write-up is entirely
different than an internal write-up and the audience is entirely different.
And so that’s why they’re cherry-picked.
There’s a whole bunch of things that aren’t included in public
write-up because the purpose is, “I want a customer or potential
customer to read this and feel at least a little bit better.”
Or really, I want them to at least get
this notion that we’ve got a handle on it.
“Wow, that was really bad, but nothing to see here, folks.
It’s all been taken care of.” But again, this is very
different, the people inside the organization, even
if it’s just sort of tacit, they’ve got a knowledge.
Tenured people who have been there for some time, see connections, even
if they’re not made explicit, between one incident to another incident.
To that one that happened—“Remember that one
that happened three years ago, that big one?
Oh, sorry, you’re new.
Oh, let me tell you the story.
Oh, it’s about this and blah, blah, blah.
And who knew that Unix pipes only passes 4k across it.”
Blah, blah, blah, something—some weird, esoteric thing.
And so our focus, largely, although we have done projects with companies
about trying to be better about their external language about it, the
vast majority of what we do and where our focuses is, is to capture
the richest understanding of an incident for the broadest audience.
And like I said at the very beginning, the bar is real low.
There’s a lot of, I don’t want to say falsehoods, but certainly a lot of
myths that just don’t play out in the data about whether people are learning.
Whenever we have a call with a potential
client, we always ask the same question.
Ask them about what their post-incident activities look like, and they tell
us and throw in some cliches, and everyone—never want a crisis go to waste.
And, “Oh, yes.
And we always try to capture the learnings and we put them in a
document.” And we always ask the same question, which is, “Oh.
So, you put these documents, these write-ups in an area?”
Oh, yes, we want that to be shared as much as possible.
And then we say, “Who reads them?” And that tends to put a bit of a
pause because most people have no idea whether they’re being read or not.
And the fact is, when we look, very few of these write-ups are being read.
Why?
I’ll be blunt: because they’re terrible.
[laugh]
. There’s not much to learn from there because they’re not written to be read.
They’re written to be filed.
And so we’re looking to change that.
And there’s a whole bunch of other things that are unintuitive,
but just like all of the perspective shifts, DevOps, and continuous
deployment, they sound obvious, but only in hindsight after you get it.
That’s characterization of our work.
It’s easy to wind up, from the outside, seeing a scenario where things go
super well in an environment like that, where, okay, we brought you in as
a consultant, suddenly, we have better understanding about our outages.
Awesome.
But outages still happen.
And it’s easy to take a cynical view of, okay, so other than talking
to you a lot, we say the right things, but how do we know that
companies are actually learning from what happened as opposed to
just being able to tell better stories about pretending to learn?
Yeah, yeah.
And this is, I think, where the world of
software has some advantages over other domains.
And the fact is, software engineers don’t pay any attention
to anything they don’t think the attention is warranted,
or they’re not being judged, or scored, or rewarded for.
And so there’s no single signal that accompanies learning from incidents.
It’s more like a constellation, like, a bunch of smaller signals.
So, for example, if more people are reading the write-ups.
If more people are attending group review meetings.
In organizations that do this really well, engineers who start attending
meetings, we ask them, “Well, why are you going to this meeting?” And they’ll
report, “Well, because I can learn stuff here that I can’t learn anywhere else.
Can’t read about it in a runbook, can’t read about it on the wiki,
can’t read about it in an email, or hear about it in an all-hands.”
And that they can see a connection between, even incidents handled
in some distant group, they can see a connection to their own work.
And so those are the sort of signals—we’ve written about this on our blog—those
are the sort of signals that we know that progress is building momentum.
But a big part of that is capturing this, again, this experience.
Usually, we’ll see, there’s a timeline, and this is when memcached
did X, and this alert happened, and then blah, blah, blah, blah, blah.
Right?
But very rarely are captured the things that, when you ask an
engineer, “Tell me your favorite incident story.” People who will
even describe themselves, “Oh, I’m not really a storyteller, but
listen to this.” And they’ll include parts that make for a good story.
Social construct is, if you’re going to tell a story, you’ve
got the attention of other people, you’re going to include
the stuff that was not usually kept or captured in write-ups.
For example, like, what was confusing?
A story that tells about what was confusing, well—“And then we
looked, and it said, ‘zero tests failed.’”—this is an actual case
that we looked at—“It says ‘zero tests failed.’ And so, okay.
So, then I deployed.
Well, the site went down.” “Okay, well, so
what’s the story there?” “Well, listen to this.
As it turns out, at a fixed font, zeros, like, in Courier or
whatever, have a slash through it and at a small enough font,
a zero with a slash through it looks a lot like an eight.
There were eight tests failed, not zero.” So, that’s about the display.
And so those are the types of things that make a good story.
We all know stories like this, right?
The Norway problem with YAML.
You ever heard of that Norway problem?
Not exactly.
I’m hoping you’ll tell me.
Well, so lay [laugh] it’s excellent, and of course it works out that the
spec for YAML will evaluate the value no—N-O—to false as if it was a boolean.
Yes, for true.
Well, but if your YAML contains a list of abbreviations for countries, then
you might have Ireland, Great Britain, Spain, US, false instead of Norway.
And so that’s just an unintuitive surprise.
And so, those are the types of things that don’t
typically get captured in incident writeups.
There might be a sentence like, “There was a
lack of understanding.” Well, that’s unhelpful.
At best.
Don’t tell me what wasn’t there.
Tell me what was there.
“There was confusion.” Great.
“What made it confusing?” “Oh, yeah.
N-O is both ‘no’ and the abbreviation for Norway.”
Red herrings is another great example.
Red herrings happen a lot; they tend to stick in people’s
memories; and yet, they never really get captured.
But it’s, like, one of the most salient
aspects of the case that ought to be captured.
People don’t follow red herrings because they know they’re a red herring.
They follow red herrings because they think it’s going to be productive.
So therefore, you better describe for all your colleagues
what brought you to believe that this was productive.
Turns out later—you find out later that it wasn’t productive.
Those are some of the examples.
And so if you can capture what’s difficult, what’s
ambiguous, what’s uncertain, and what made it difficult,
ambiguous, or uncertain, that makes for good stories.
If you can enrich these documents, it means people who maybe don’t even
work there yet, when they start working there, they’ll be interested; they
have a set expectation they’ll learn something by reading these things.
This episode is brought to you by Gitpod.
Ever feel like you spend more time fighting
your dev environment than actually coding?
“Works on my machine issues” are too familiar and
the VDI setup in your organization drives you mad?
Gitpod brings automated, standardized development
environments to your laptop—and the cloud.
Describe your dev-environment-as-code and
streamline development workflows with Automations.
At the click of a button, you get a perfectly configured
environment and you can automate tasks and services like
seeding your database, provisioning infrastructure, running
security or scanning tools or any other development workflows.
You can self-host Gitpod in your cloud account for free in under
three minutes or run Gitpod Desktop locally on your computer.
Gitpods automated, standardized development environments
are the fastest and most secure way to develop software.
Gitpod is trusted by over 1.5 million developers including
some of the largest financial institutions in the world.
Visit gitpod.io and try it for free with your whole team.
And also, let me know what you think about it.
I've honestly been looking for this for a while,
and it's been on my list of things to try.
I'll be trying it this week.
Please, reach out, let me know what you think.
There’s an inherent cynicism around… well, from at least from my side
of the world, around any third-party that claims to fundamentally shift
significant aspects of company culture, and if the counter-argument to that
is that you and DORA and a whole bunch of other folks have had significant
success with doing it, it’s just very hard to see that from the outside.
So, I’m curious as to how you wind up telling stories about that
because the problem is inherently whenever you have an outsider
coming into an enterprise-style environment, is, “Oh, cool.
What are they going to be able to change?” And it’s hard to articulate
that value, and not—well, given what you do, to be direct—come across as an
engineering apologist, where it’s well, “Engineers are just misunderstood,
so they need empathy, and psychological safety, and blameless post-mortems.”
And it sounds to crappy executives, if I’m being direct, that, “Oh,
in other words, I just can’t ever do anything negative to engineers
who, from my perspective, just failed me or are invisible, and there’s
nothing else in my relationship with them.” Or am I oversimplifying?
No, no.
I actually think you’re spot on.
I mean, that’s the thing is that if you’re talking with leaders—remember, a.k.a.
People who are, even though they’re tasked with providing the resources and
setting conditions for practitioners—the hands-on folks who get their work
done—they’re quite happy to talk about these sort of abstract concepts,
like psychological safety and insert other sorts of hand-wavy stuff.
What is actually pretty magical about incidents is that these
are grounded, concrete, messy phenomena that practitioners
have, and will remember; they’re sometimes visceral experiences.
And so that’s why we don’t do theory at Adaptive Capacity Labs.
We understand the theory, happy to talk to you about it,
but it doesn’t mean as much without the practicality.
And the fact of the matter is that the engineer apologist is, “If
you didn’t have the engineers, would you have a business?” That’s
at the flip side; this is, like, the core unintuitive part of the
field of resilience engineering, which is that Murphy’s Law is wrong.
What could go wrong almost never does, but we don’t pay much attention to that.
And the reason why you’re not having nearly as many incidents
as you could be is because, despite the fact that you make it
hard to learn from incidents, people are actually learning.
But they’re just learning out of view from leaders.
When we go to an organization and we see that most
of the people who are attending post-incident review
meetings are managers, that has a very particular signal.
That tells me that the real post-incident review is happening outside
that meeting, it probably happened before that meeting, and those people
are there to make sure that whatever group that they represent in their
organization isn’t unnecessarily given the brunt of the bottom of a bus.
And so it’s a political due diligence.
But the notion that you shouldn’t punish or be harsh on
engineers for making mistakes completely misses the point.
The point is to set up the conditions so that
engineers can understand the work that they do.
And if you can amplify that, as Andrew Schaffer has said,
“You’re either building a learning organization, or you’re
losing to someone who is.” And a big part of that is you need
people; you have to set up conditions for people to
give detailed story about their work, what’s hard.
This part of the codebase is really scary, right?
All engineers have these notions: this part is really scary, this part
is really not that big of a deal, this part is somewhere in between.
But there’s no place for that outside of the informal discussions.
But I would assert that if you can capture
that, the organization will be better prepared.
The thing that I would end on that is that it’s a bit of a rhetorical
device to get this across, but one of the questions we’ll ask is,
“How can you tell the difference between a difficult case—a difficult
incident—handled well, or a straightforward incident handled poorly?”
And from the outside, it’s very hard to tell the difference.
Oh, yeah.
Well, certainly if what you’re doing is averaging how long these things take.
But the fact of the matter is that all the people who were
involved in that, they know the difference between a difficult
case handled well, and a straightforward one handled poorly.
They know it, but there’s nowhere, there’s no
place to give voice to that lived experience.
So, on the whole, what is the tech industry missing when it
comes to learning effectively from the incidents that we all
continually experience and what feels to be far too frequently?
They’re missing what is captured in that age-old
parable of the blind men and the elephant.
And I would assert that these blind men that the king sends out—“Go find
an elephant and come back and tell me about the elephant”—they come back
and they all have—they’re all valid perspectives, and they argue about,
“No, an elephant is this big flexible thing,” and other one is, “Oh,
no, an elephant is this big wall,” and, “No, an elephant is a big flappy
thing.” If you were to make a synthesis of their different perspectives,
then you’d have a richer picture and understanding of an elephant.
You cannot legislate—and this is where what you brought up—you
cannot set ahead, a priori, some amount of time and effort.
And quite often what we see are leaders saying, “Okay, we need
to have some sort of root cause analysis done within 72 hours
of an event.” Well, if your goal is to find gaps, and come
up with remediation items, that’s what you’re going to get.
Remediation items might actually not be that good
because you’ve basically contained the analysis time.
Which does sort of feel, on some level, like it’s very much aligned
as—from a viewpoint of, yeah, remediation items may not be useful as
far as driving lasting change, but without remediation items, good luck
explaining to your customers that will never ever, ever happen again.
Right, yeah.
Of course.
Well, you’ll notice something about those public write-ups;
you’ll notice that they don’t tend to link to previous
incidents that have similarities to them because that would
undermine the whole purpose, which is to provide confidence.
And a reader might actually follow a hyperlink to say, “Wait a minute.
You said this wouldn’t happen again.”
Turns out it would.
Of course, that’s horseshit.
But you’re right.
And there’s nothing wrong with remediation items, but if
that’s the goal, then that goal is—you know, what you look
for is what you find, and what you find is what you fix.
If I said, “Here’s this really complicated problem and I’m only giving you an
hour to describe it,” and it took you eight hours to figure out the solution.
Well then, what you come up with in an hour
is not actually going to be all that good.
So, then the question is, how good are the remediation items?
Quite often what we see is—and I’m sure you’ve had this
experience—an incident’s been resolved and you and your
colleagues are like, “Wow, that was a huge pain in the ass.
Oh, dude.
I didn’t see that coming.
That was weird.
Yeah.” And one of you might say, “You know what?
I’m just going to make this change because I don’t want to be woken
up tonight, or I know that making this change is going to help things.
I’m not waiting for the post-mortem.
We’re just going to do that.” “Is that good?” “Yep.” “Okay, yeah, please do it.”
Quite frequently, those things, those actions, those aren’t
listed as action items, and yet it was a thing so important
that it couldn’t wait for the post-mortem—arguably the most
important action item—and it doesn’t get captured that way.
We’ve seen this take place.
And so again, in the end, it’s about those who have the lived experience.
The live experience is what fuels how reliable you are today.
You don’t go to your senior technical people and say, “Hey, listen.
We got to do this project.
We don’t know how.
I want you to figure out—we’re going to—let’s say we’re
going to move away from this legacy thing, so I want
you to get in a room, come up with two or three options.
Gather a group of folks who know what they’re talking about.
Get some options, and then show me what the options.
Oh, and by the way, I’m prohibiting you from taking into account
any experience you’ve ever had with incidents.” It sounds ridiculous
when you would say that, and yet, that is what [unintelligible]
. So, if you can fuel people’s memory, you can’t say
you’ve learned something if you can’t remember it.
At least that’s what my kids’ teachers tell me.
And so yeah, you have to capture the lived experience,
and including what was hard for people to understand.
And those make for good stories.
That makes for people reading them.
That makes for people to have better questions about it.
That’s what learning looks like.
If people want to learn more about what you have to say
and how you view these things, where can they find you?
You can find me and my colleagues at adaptivecapacitylabs.com
where we talk all about the stuff on our blog.
And myself, and Richard Cook, and Dave Woods are also on Twitter, as well.
And we’ll, of course, include links to that in the [show notes] . John,
thank you so much for taking the time to speak with me today.
I really appreciate it.
Yeah, thanks.
Thanks for having me.
I’m honored.
John Allspaw, co-founder and principal at Adaptive Capacity Labs.
I’m Cloud Economist Corey Quinn, and this is Screaming in the Cloud.
If you’ve enjoyed this podcast, please leave a five-star review on
your podcast platform of choice, whereas if you hated this podcast,
please leave a five-star review on your podcast platform of choice,
along with a comment giving me a list of suggested remediation
actions that I can take to make sure it never happens again.