So legacy is really just a certain slice of rapid growth in
applications and infrastructure that's sort of an unmanageable mess now.
Welcome to Screaming in the Cloud.
I'm Corey Quinn.
This promoted episode is sponsored by a long time, I wouldn't even say
friend, so much as antagonist slash protagonist slash symbiotic company.
With things I have done as I have staggered through the ecosystem.
There's a lot of fingers of blame that I can point throughout the course of my
career at different instances, different companies, different clients, etc, etc.
That have shaped me into the monstrosity that I am today.
Far and away, the company that had the most
impact on the way that I speak publicly is Puppet.
Here to accept the recommendation for what I've become and how it's
played out is Nigel Kirsten, a field CTO at Puppet, or THE field CTO.
I don't know how many of them they have.
Nigel, welcome to the show and how unique are you?
Thank you, Corey.
Well, I'm, you know, reasonably unique.
I think you get used to being, you know, one of the few Australians
living in Portland who's decided to move away from the sunny
beaches and live in the grey wilderness of the Pacific Northwest.
This episode is sponsored in part by my day job, the Duck Bill Group.
Do you have a horrifying AWS bill?
That can mean a lot of things.
Predicting what it's going to be.
Determining what it should be.
Negotiating your next long term contract with AWS.
Or just figuring out why it increasingly resembles a
phone number, but nobody seems to quite know why that is.
To learn more, visit duckbillgroup.
com.
Remember, you can't duck the duck bill, Bill.
And my CEO informs me that is absolutely not our slogan.
So to give a little context into that ridiculous intro, I was a
traveling contract trainer for the Puppet Fundamentals course for an
entire summer back in, I want to say 2014, but don't hold me to it.
And it turns out that when you're teaching a whole bunch of students
who have paid, in many cases, a couple thousand dollars out of
pocket for, to learn a new software, where in some cases they feel
like it's their taking their job away because they viewed their
job rightly or wrongly as running the same script again and again.
And then the demo breaks and people are angry.
And if you don't get a good enough rating, you're not invited to continue.
And then the company you're contracting through hits you with a stick.
It teaches you to improvise super quickly.
Okay.
So, I wasn't kidding when I said that Puppet was in
many ways responsible for the way that I give talks now.
So, what do you have to say for yourself?
Well, I have to say, you know, congratulations for surviving
opinionated defensive nerds who think not only you, but your
entire product, your demo could be replaced by a shell script.
It's a tough crowd.
It was an experience, and some of these were community based, and some of
them were internal to a specific company, and if people have heard more
than one episode of this show, I'm sure they can imagine how that went.
I gave a training at Comcast once, and set a personal challenge for myself
of how many times could I use the word Comcastic in a three day training?
And I would work it in and talk about things like the schedule
parameter in Puppet, where it doesn't guarantee something's
going to execute in a time window, it's the only time it may
happen, if it doesn't fire off and then it isn't going to happen.
It's like a Comcast service appointment, and then they just all kind
of stared at me for a while, and credit where due, that was the best
user rating I ever got from people sitting through one of my trainings.
So thanks for teaching me how to improv at basically what
could have been a very expensive mistake on Puppet's part.
It accidentally worked out for everyone.
Brilliant.
Yes.
You would have survived teaching the spaceship operator to that sort of a crowd.
Oh, I mostly avoided that thing.
That was an advanced Puppetism, and this was Puppet Fundamentals because
I just need to be topically good at things, not deep dive good at things.
But let's dig into that a little bit.
For those who are, who have not had the
pleasure of working with Puppet, what is it?
Sure.
So Puppet is a pretty simple DSL.
You know, DSLs aren't necessarily in favor these days.
Domain specific language for those who have not caught up on that acronym.
Yes.
So, a programming language designed for a specific task.
And, you know, instead we've decided that, you know, the
world will rest on YAML, and we've absorbed a fair bit of YAML
into our ecosystem, but there are things that I will still
stand by are just better to do in a programming language.
If X, then Y, for example, is just easier to express when
you have actual syntax around you, and you're not sort of
forcing everything to be in a data specification language.
So Puppet's pretty simple in that it's a language that lets
you describe the state that infrastructure should be in.
And you can do this in a modular and composable way.
So I can build a little chunk of automation code, hand it to Corey, Corey
can build something slightly bigger with it, hand it to someone else.
And really this sort of collaboration is one of the reasons why Puppet's
sort of been at the center of the DevOps movement, which at its core is not
really about tools, it's about reducing friction between different groups.
Back when I was doing my traveling training shtick, I found that I had to figure
out a way to describe what Puppet did to folks who were not deep in the space.
And the analogy that I came up with, that I was particularly
partial to, was, imagine you get a brand new laptop.
Well, What do you do with it?
You install your user account and go through the setup.
You install the programs that you use, some which have licenses on it.
You copy your data onto it.
You make sure that certain programs always run on startup,
because that's the way that you work with these things.
You install Firefox, because that's the
browser of choice that you go with, etc, etc.
Now imagine having to do that for, instead of one computer, a thousand of them.
And instead of a laptop, their servers.
And that is directionally what Puppet does.
Absolutely.
This is the one I used for my mother as well.
Like, I was working around Puppet for years before, and I used,
the way I explained it was You know, when you get a new iPad,
you've got to set up your Facebook account and your email.
Imagine you had 10, 000 of these.
And she was like, and I was like, you know, companies like Google,
company like big banks, they all have lots and lots and lots of computers.
And she was like, they run all those things on iPads.
And I was like, this is not really where my analogy was going.
But
right.
And increasingly though, that seems like the world has shifted in some
direction where when you explain that to your mother and she comes back with,
Well, wouldn't they just put the application into Docker and be done with it?
Oh dear.
But that seems to be in many ways that the direction that the zeitgeist
has moved in, whether or not that is the reality in many environments,
where when you're just deploying containers everywhere through the
miracle of Kubernetes, if you'll pardon the dismissive scorn there, that.
You just package up your application, shove it into a container, and
then hurl it from the application team over to the operations team
like a dead dog cast into your neighbor's yard for him to worry about.
And then it just sort of takes a, it sort of takes up the space of you don't
have to manage state anymore because everything is mostly stateless in theory.
How have you seen it play out in practice in the last five years?
I mean, that's a real trend.
And you know, like, the size of a container
should be smaller than an operating system.
And the reality is, you know, like, I'm a sysadmin.
I love operating systems.
I nerded out on operating systems.
They're a necessary evil.
They're terrible, terrible things.
Registry keys, config files, you know, like,
they're a pain in the neck to deal with.
And if you look at I think what a lot of operations folks missed about
Docker when it started was that it didn't make their life better.
It was worse.
It was like this actual sort of terrible tool chain where
you sort of tied together all these different things.
But really importantly what it did is it put
Control into the hands of the developers.
And it was the developers who were trying to do
stuff, who were trying to shift applications.
And I think Docko is a really great technology.
In the sense of, you know, developers could ship value on their own.
And that was the huge, huge leveling up.
You know, it wasn't the interface.
It wasn't the user experience.
Wasn't all these things.
It was just that the control got taken away from the IT
trolls in their basement going, No, don't touch my servers.
And instead given straight to the developers.
And that's huge because it let us ship things faster.
And that's ultimately the whole goal of things.
The thing that really struck me the most from conducting the
trainings that I did was meeting a whole bunch of people across
the country in different technological areas of specialty,
in different states of their evolution as technologists.
And something that struck me was just how much people wound
up identifying the with the technology that they worked on.
When someone is the AIX admin and the AIX machines are
getting replaced with Linux boxes, there's this tendency
to fight against that and rebel rather than learning Linux.
And I get it.
I'm as subject to this as anyone is.
And in many cases, that was the actual pushback
that I saw against adopting something like Puppet.
If I identify my job as being the person that runs all of these
But carefully curated scripts that I've spent five years building.
And now that all gets replaced with something that is
more of a global solution to my local problem, then it
feels like the thing that made me special is eroding.
And we see that with the migration to cloud as well, when you're the storage
admin and it just becomes an API call to S3, that's kind of a scary thing.
And when you're one of the server hugger types, again, as guilty as
anyone of this, and you start to see cloud coming in as like a rising
tide that eats up what it was that you became known for, it's scary.
And it becomes a foundational shift in How you view yourself.
What I really had a lot of sympathy for was
the folks who'd been doing this for 20 years.
They were, in some cases, a few years away from retirement.
And they'd been doing basically the same set of tasks every year for 25 years.
It's one year of experience repeated 25 times.
And they don't have that much time.
time left in their career, intentionally, so they want to
retire, but they also don't really want to learn a whole bunch
of new technologies just to get through those last few years.
I feel for them, but at the same time Me too,
totally.
But what are you going to do?
But without sounding sort of too dismissive there, like, I think it's a
natural tendency for us to identify with the technology when, if that's
what you're around all the time, you know, mechanics do this, truck drivers
with brands of trucks, people like build attachments to the technology
they work with, because we fit them into this bigger Techno social system.
But I have a lot of empathy for the people in enterprise
jobs who are being asked to change radically because
the cycle of progress is speeding up faster and faster.
And as you say, like they might be a few years away from retirement.
I think I used to feel a lot more differently about this when I was
really hotheaded and, you know, sort of much more of a tech enthusiast.
And that's what I identified with in terms of, you
know, it's okay for a job to just be a job for people.
It's okay for someone to be doing a job because They get good
health care and good benefits and it's feeding their family.
Like, that's an important thing.
You can't expect everyone to always be incredibly passionate about
technology choices in the same way that I think, you know, many
of us who live, live on Twitter and hang out in this space are.
Oh, I have no problem whatsoever with people who want to show
up for 40 hours a week ish, work on their job, and then go
home and have lives and not think about computers at all.
There's this dark mass of developers out there that basically never
show up on Twitter, they aren't on IRC, they don't go to conferences.
And that's fine.
I have no problem with that.
And I hope I don't come across as being overly dismissive of those folks.
I honestly wish I could be content like that.
I just don't hold still very well.
But yeah, so I think you touched on a few interesting things there.
And some of those we sort of cover in the state of
DevOps report, which is coming out in the next few weeks.
Indeed.
And the state of DevOps report
is started off at Puppet.
And they've now done it for, what,
10 years?
This is the 10th year, which is completely crazy.
So I was looking at the stats as I was writing
it and it's 10 years of stated DevOps reports.
I think it's 11 years of DevOps Weekly, Gareth Roshgrove's newsletter.
It's 12 or 13 years of DevOps days that have been going on.
You know, this is longer than I spent in primary and high school put together.
It's kind of crazy that the DevOps movement's still kind of chugging
along, even if it's not necessarily the coolest kid on the block.
Now that, you know, GitOps, SRE, Flavor of the Month, various kinds of
permutations of how we work with technology have perhaps got a little bit
cooler, but it's still very, very relevant to a lot of enterprises out there.
Yeah, as I frequently say, legacy is a condescending engineering
term for it makes money, and there's an awful lot of that out there.
Forget cloud, there are still companies wrestling
with, do we explore this virtualization thing?
And that was something I was very against back in 2006, let's be very honest.
I am very bad at predicting the future of technology.
And I can see it for small niche edge workload
cases, where you have a bunch of idle servers.
But for the most part, who's really going to use this in production?
Well, basically everyone, because that in turn is what the cloud runs on.
Yeah, I think we can safely say I got that one hilariously wrong.
But hey, if you aren't going to make
predictions, then what's the matter?
But the industry pushes you in these directions.
So there was this massive bank in Asia who I've been working with for
a long time, and they were always resistant to adopting virtualization.
And then it was only four or five years ago that I visited them, they
were like, right, we're Okay, it's time, we're rolling out VMware.
And I was like, so I'm really curious, what exactly changed in the last year
or two, in like 2014, 2015, that you decided virtualization was the key?
And I'm like, Oh, there was
this jackwagon who conducted this training.
Yeah, no, no, sorry.
I can't fend credit for that one.
They couldn't order one rack unit servers with CD drives anymore because
their whole process was actually provisioning with CDs before that point.
Welcome to the brave new world
of pixie booting, which is kind of hard.
So yeah, virtualization is easier.
I, you know, sometimes people have to be dragged into various ways of
technological advancement, which gets to the real thing I want to cover.
Since this is a promoted episode where you're talking about the state
of DevOps report, I'm almost less interested in what this year's
has to say specifically than what you've seen over the last decade.
What's changed?
What was true 10 years ago that is very much not true now?
Bonus points, you can answer that without
using the word Kubernetes more than twice.
So I think one of the big things was that
we've definitely passed peak DevOps team.
Like if you may remember, there was a lot of arguments
and there's still regular, is DevOps a job title?
Is it a team title?
Oh, I was very much on the
no side until I saw how much more I would get paid as a DevOps
engineer instead of a systems administrator for the exact same job.
So, you know, I shut up and I took the money.
I figured that the semantic arguments are great, but.
Yeah,
and that's exactly what we've, we've written
in the report and I think it's great.
The sysadmins, you know, we were unloved, you know, we were
in the basement, we weren't paid as much as programmers.
The running, you know, the joke used to be for developers DevOps meant I don't
need ops anymore, but for ops people it was I can get paid like a developer.
In many cases, oh, well, systems administrators don't want to learn how to code.
It's Yeah, you're remembering a relatively narrow slice of
time between the modern era where systems administrator types
need to be able to write in the lingua franca of everything,
which is of course YAML, as far as programming languages go.
And before that, to be a competent systems administrator,
you needed to have a functional grasp of C.
And there is only a limited window in which a bunch of bash
scripts and maybe a smidgen of Perl would have carried you through.
But the deeper understanding is absolutely
necessary, and I would argue always has been.
And this is great because you've just linked up with
one of the things we found really interesting about the
report is that, you know, when we talk about legacy.
We don't actually mean the oldest shit, because the
oldest shit is the mainframes, it's a lot of bare
metal applications, a lot of that in big enterprises.
We're still waiting for an AWS 400 to replace some of that.
Well, it's administered by real systems engineers, you know, like the
people who wrote C, who wrote kernel extensions, who could debug things.
What we actually mean by legacy, is we mean late 90s to late 2000s, early
2010s, stuff that was put together by kids who, like me, happened to get a job
because you grew up with a computer and then the dot com explosion happened.
You weren't necessarily particularly skilled and a lot of people were, they
didn't go through the apprenticeships that mainframe folks and systems engineers
actually went through and everyone just held this stuff together with, you
know, duct tape and dental floss and then now we're paying the price of it all.
Like way back down the track.
So legacy is really just a certain slice of
rapid growth in applications and infrastructure.
That's sort of an unmanageable mess now.
Oh, here in San Francisco, legacy is
anything prior to last night's nightly bill.
It's turned into something a little ridiculous.
I feel like the real power move as a developer now is to get a
job, go in on day one, rebase everything in the Git repository to a
single commit with a message, legacy code, and then follow that up.
And that's the power move and that's how it works.
And that's also the attitude we wind up encountering in a lot of places.
And I don't think it serves anyone particularly well
to tie themselves so tightly to that particular vision.
Yep, absolutely.
But this is a real problem, this space.
And one of the things we found in the Stead Devils report is that Let me
back up a little and give a little bit of methodology of what we actually do.
We survey people about their sort of performance
metrics, you know, like how quickly can you do deploys?
What's your sort of mean time to recovery?
Those sorts of things.
And what practices do you actually employ?
And we essentially go through and do statistical analysis
on this and everyone tends to end up in three cohorts.
They separate pretty easily of low, medium and high evolution.
And so one of the things we found is that everyone
at the low level has all sorts of problems.
They have issues with what does my team do?
What does the team next to me do?
How do I talk to the team next to me?
How do I actually share anything?
How do I even know what my goals are?
Like fundamental company problems.
But everyone at all levels of evolution is stuck on two big things,
not being able to find enough people with the right skills for
what they need and their legacy infrastructure holding them back.
The thing that I find the most compelling is the idea of not
being able to find enough people with the skills that they need.
And I'm going to break my own rule and
mention Kubernetes as a prime example of this.
If you are effective at managing Kubernetes in production, you will make.
A very comfortable living in any geographical location
on the planet, because it is incredibly complex.
And every time we've seen this In previous trends, where you need to get
more and more complexity and more and more expertise just to run something,
it looks like a sawtooth curve, where at some point that complexity gets
abstracted away and compressed down into something that is basically a
single line somewhere, or it happens below the surface level of awareness.
My argument has been that Kubernetes is something no
one's going to care about in roughly three years from now.
Not because we're not using it anymore, but because it's
below the level of awareness that we have to think about.
In the same way that there aren't a whole lot of people on the planet these
days who have to think about the Linux virtual memory management subsystem.
It's there and a few people really care about it, but
for the rest of us, we don't have to think about that.
That is the infrastructure underneath our infrastructure.
Absolutely.
I used to make a living, and it's ridiculous
looking back at this for a year or two.
Doing high performance custom compiled apaches.
I was really, really good at this.
Well, yeah, Apache is a great example of this.
Where, back in the 90s, to get a web server up and running,
you needed to have three days to spare, and in depth
knowledge of GCC compiler flags, and hope for the best.
And then RPM came out.
And then, okay, then YUM, or other things like that on top of it.
And then things like Puppet started showing up.
And we saw, oh great, now Insure installed.
Great.
And then we took a step beyond that, and it was, oh, now it's just a
Docker run, whatever it is, and these days, yeah, it's a checkbox in S3.
So, so let me make, get your Kubernetes prediction down, right.
So you're not, you're predicting Kubernetes is going to go
away like, you know, Apache and highly successful things.
It's not an open stack failure state.
It's a Apache invisibility state.
Absolutely.
My timeline is a bit questionable.
Let's be fair, but it's on the aggressive side, but yeah,
I think that Kubernetes is inherently too complex for most
people to have to wind up thinking about it in that way.
And.
We're not talking small copies, we're talking big ones, where you're not
in a position, if you're a giant blue chip fortune 50, to hire 2, 000
people who all know Kubernetes super well, and you shouldn't have to.
There needs to be some flattening of all of that High level complexity.
Without the management tools though, with things like Puppet and the
things that came before in a bunch of different ways, we would all not be
able to get anything done because we'd be too busy writing and assembly.
There's always going to be those abstractions abstractions, and very few
people understand how it works all the way down, but that's in many cases okay.
That's civilization, you know?
Like, do you understand what happens when you
plug in something to your electricity socket?
I don't want to know.
Like, I just want light.
And more to the point, whenever you flip the switch, you don't
have that doubt in your mind that the light is going to come on.
So if it doesn't, that's notable.
And your first thought is, oh, I The light
bulb is out, not the utility company is down.
And when we talk about the cloud being utility computing
Has someone put a Kubernetes operator in this
light switch that may break this process?
Well, okay, IoT does throw a little bit of a crimp into those words, but yeah.
Let's talk more about the state of DevOps report.
What notable findings were there this year?
So one of the big things that we've seen for the last couple of years has
been that most companies are stuck in the middle of the evolutionary progress.
And, you know, anyone who deals with large enterprises knows this is true.
Like whatever they've adopted in terms of technology, in terms
of working methods, you know, agile, various different things.
Most companies don't tend to advance to the higher levels.
Most places stay mired in mediocrity.
So we wanted to dive into that and try and work out why are most
companies actually stuck like this when they hit a certain size.
And it turns out the problems aren't technology or DevOps.
They're like really fundamental problems like We don't have clear goals.
I don't understand what the teams next to me do.
We did a bunch of qualitative interviews as well as the quantitative work in the
survey with this report and we talked to one group of folks at a pretty large
financial services company who were like, Our teams have all been renamed so
many times, if I need to go and ask someone for something, I literally page up
and down through ServiceNow trying to find out where to put the change request
and they're like, how do I know where to put a network port opening request
for this particular service when there are 20 different teams that might be
named the right thing and some are obsolete and I get no feedback whether
I've sent it off to the right thing or to a black hole of enterprise despair.
Here at the Duck Bill Group, one of the things we do with,
you know, my day job is we help negotiate AWS contracts.
We just recently crossed 5 billion of contract value negotiated.
It solves for fun problems such as how do you know that your
contract that you have with AWS is the best deal you can get?
How do you know you're not leaving money on the table?
How do you know that you're not doing what I do on this podcast
and on Twitter constantly and sticking your foot in your mouth?
To learn more, come chat at duckbillgroup.
com.
Optionally, I will also do podcast voice when we talk about it.
Again, that's duck bill group.com.
That doesn't get better with a lot of modernization.
I mean, I feel like half of my job, and I'm not
exaggerating, is introducing Amazonians to one another.
Corporate communication between departments and
different groups is very far from a solve problem.
I think the tooling can help, but I've never been a big
believer in solving political problems with technology.
It doesn't work.
People don't work that way.
Absolutely, like one of my earliest times working at Puppet
doing sort of higher level sales and services and support.
Huge national telco walk in there, we've got the
development team, the QA team, the infrastructure team.
In the course of this conversation, one of
them makes a comment about using apt get.
And the others were like, what do you mean?
We're on RHEL.
And it turned out production was running on RHEL, the QA team running
on CentOS, and the developers were all building everything on Ubuntu.
And because it was Java apps, they almost didn't have
to care, but you know, write once, debug everywhere.
History doesn't repeat,
but it rhymes.
Before Docker, so much of development and startup land was, how do
I make my MacBook Pro look a lot more like an EC2 Linux instance?
And it turns out that there's an awful lot of work that goes
into that, that maybe isn't the best use of people's time.
And we start to see these breakthroughs and
these revelations in a bunch of different ways.
I have to ask, this is the 10th year that
you've done the State of DevOps report.
At this point, why keep doing it?
Is it inertia?
Are you still discovering new insights every year on top of it?
Or is it one of those things where, well, someone
in marketing says we have to do it, so here we are?
No, actually, it's not that at all.
So definitely we're going to take stock after this year because
10 years feels like a really good point to sort of, it's a nice
round number and, you know, a certain kind of number system.
Mainly the reason is, you know, a lot of my job is going and
helping big enterprises just get better at using technology.
And it's funny how often, like, I just get folks going, Oh, I read this thing.
Like, people who aren't on, you know, the bleeding edge,
constantly discussing these things on Twitter or whatever,
but the Stead DevOps support makes its way to them.
And they're like, Oh, I read a thing there about how, how
much better it is if we standardized on one operating system.
And that made a really huge difference to what we were actually doing,
because you had all this data in there showing that that is better.
And honestly, that's the biggest reason why I end up doing it.
You know, it's the fact that, It seems to be a tool that has
made its way through to very hard to penetrate enterprise folks.
And they'll read it and managers will read things like that are like if you
set clear goals for your team and Get them to focus on optimizing the legacy
environment You will see returns on it and like I'm being a little bit facetious
in the tone that I'm saying because a lot of this Stuff does feel obvious
if you're you know You know, constantly swimming in this stuff day to day.
But it's not just the practitioners who it's
just a job for in a lot of big companies.
It's true of a lot of the management chain as well.
They're not necessarily going out and reading up on, you
know, modern agile IT management practices day to day for fun.
You know, they go home and do something else.
One
of
my favorite conferences is Gene Kim's DevOps Enterprise Summit.
And the specific reason behind that is these are very large companies that
go beyond companies in some cases to institutions where you have the US Air
Force as a presenter one year, and very large banks that are 200 years old.
Every other conference, it seems, more or less involves people
getting on stage to deliver conferenceware and tell stories
that make people at those companies feel bad about themselves.
Where it's, we're Twitter for pets and this is how we deploy software.
Or the ever popular, this is how Netflix does stuff.
Yeah, Netflix has basically no budget constraints as far
as hiring engineering folks go, and lest we forget, their
failure mode is someone can't watch a movie right now.
It's not exactly the same thing as the ATM starts
spitting out the wrong balance in the streets.
And I think that there's an awful lot of Discussion where people look at the
stories people tell on conference stages and come away feeling bad from it.
Very often I'll see someone from a notable tech company talk about
how they do things and, wow, I wish my group did things like that.
And the person next to me says, yeah, me too.
And I check and they work at the same company.
And it's, Like, the stories we tell are
not necessarily the stories that we live.
And it's very easy to come away discouraged from these things.
And that goes triply so for large enterprises that are regulated,
that have significant downside risk if the technology fails them.
And I love watching people getting a chance to tell those stories.
Let me jump on that really quickly.
Please, by all means.
One is, you know, having done four years at Google,
things are a shit show internally there too.
You talk about it like it's prison.
I like it.
Like People get horrified when they turn up and they're like, Oh, what?
It's not all gleaming, perfect software artifacts,
you know, delivered from the hand of Errs.
But I think what Gene has done with DevOps Enterprise Summit is
fantastic and how people share more openly their failure states.
But even there, and this is an interesting result we found from a few
years ago, State Dev Support, even those executives are being more
optimistic because it's sort of beaten into you as a senior executive.
You're putting on a public face.
And even when they're trying to, Share the warts and all story.
They can't help but put a little bit of a positive spin on it,
because I've had exactly the same experience there, where someone's
up there telling a war story, and then I look, turn to the
person next to me, and they work at that same 300 year old bank.
And they're like, actually it's much much worse than
this and we didn't fix it quite as well as that.
So I think, you know, the big tech companies
are terrible inside, unless they're Netflix.
And the big enterprises are also terrible, but they're also No, no, I've
talked to Netflix people too.
They do terrible things internally there too.
No one talks about the fact that their
internal environments are always tire fires.
And there are two stories, the stories we tell publicly and the reality.
And if you don't believe me on that, look at
any company in the world's billing system.
As much as we all talk about Agile and various implementations thereof, when
it comes to things that charge customers money, we're all doing Waterfall.
Absolutely.
Because mistakes show when you triple charge someone's
credit card for the cost of a small country's GDP.
It's a problem.
I want to normalize those sorts of things more.
I'm looking forward to reading this year's report just because it's, it's
interesting to see how folks who are in environments that differ from the
ones that I get to see experience this stuff and how they talk about it.
Yeah.
And so one of the big results I think there for
big companies that's really interesting is that.
One of the sort of anti patterns is having lots of different types of teams.
And I kind of touched on this before about, you know, having
confusing team titles being a real problem and not being able
to cross organizational boundaries quickly is really, really,
you know, it's a huge inhibitor and cause source of friction.
But it turns out the pattern that is actually really great
is one that the team topologies guys have discovered.
If you've been following what Matthew Skelton and Manuel Paez have been
doing for a while, they've basically been documenting a pattern in software
organizations of a small number of team types of a platform team, value
stream teams, complicated subtest system teams, and enabling teams.
And so we worked with Manuel and Matt on this year's report and asked a
whole bunch of questions to try and validate the team topologies model.
And the results came back and they were just incredibly strong.
Because I think this speaks to some of the stuff you mentioned before
that no one can afford to hire an army of Kubernetes developers.
And whatever the hottest technology is in five years, most
big companies can't hire an army of those people either.
And so the way you get scale internally before those things become
commoditized is you build a small team and create the situation
where they can have outsized leverage inside their organization.
Like, get rid of all the blockers to fast flow
and make their focus self service to other people.
Because if you're making all of your developers learn distributed systems
operations arcane knowledge, like, that's not a good use of their time either.
It's really not.
And I think that that's something that gets lost a lot is, I've never
yet seen a company beyond the very early startup stage where the
AWS bill exceeded the cost of the people working on the AWS bill.
Payroll is always a larger expense than infrastructure
unless you're doing something incredibly strange.
And, uh, Oh, I want to save some money on the cloud bill is very
often offset by the sheer amount of time that you're going to have
to pay people to work on that because contrary to what we believe
is engineering hobbyists, people's time is very far from free.
And it's also the opportunity cost of if you're going to work on this
thing instead of something else, well, is that really the best choice?
It comes down to contextualizing what technology is doing as well
as with what's happening over in the world of business strategy.
And without having a bridge between those, it doesn't seem to work very well.
Absolutely.
It's insane.
Like, it's literally insane that as an industry we will optimize 5%, 3
percent of our, you know, infrastructure build or application workload
and yet not actually re examine business processes that are causing
your people to spend 10 percent of their time in synchronous meetings.
You know, like you can save so much more money and achieve
so much more by actually optimizing for fast flow and
getting out of the way of the people who cost lots of money.
So one last topic
that I want to cover before we call it an episode.
You talk to an awful lot of folks and it's easy to point at
the aspirational stories of folks doing things the right way.
But let's dish for a minute.
What are you seeing in terms of people not using the cloud properly?
I feel like you might have a story or two on that one.
I do have a few stories.
So in this year's report, one of the things we wanted to find out of,
like, are people using the cloud sort of, you know, in the way we think of
cloud, you know, elastic, consumption based, all of these sorts of things.
We use the NIST metrics, which I recognize can be a little controversial,
but I think you've got to start somewhere as a certain foundation.
It turns out just about everyone is using the public cloud.
And when I say cloud, like, I'm not really talking about
people's internal VMware that they've rebadged as cloud.
Like I'm talking about.
the public cloud providers.
Everyone's using it, but almost no one is taking
advantage of the functionality of the cloud.
They're instead treating it like an on premise
VMware installation from the mid 2000s.
They're taking six weeks to provision instances, they're importing all of
their existing processes, they keep these things running for a long time.
If they fall over, they're going to have to re install themselves.
One person is like tasked with, hey, do you
know how pet number 45 is actually doing here?
They're not really treating any of these things
in the way that they're actually meant to.
And I think we forget about this a lot of the time when we talk about
cloud because we jump straight to cloud native, you know, the sort of
bleeding edge of folks and serverless, highly orchestrated containers.
I think if you look at the actual numbers, the vast majority
of cloud usage, it's still things like EC2 instances on AWS.
And there's a reason because it's a familiar paradigm for people.
We're definitely going to progress past there, but I think it's easy to leave
the sort of You know, the people in the middle behind when we're talking
about cloud and how to improve the ecosystem that they all operate in.
Part of the problem too is that whenever we look at how
folks are misusing cloud, it's easy to lose sight of context.
People don't generally wake up and decide I'm going to do a terrible job today
unless they work in, you know, Facebook's ethics department or something.
Instead, it's very much a people are shaped by the constraints
they're laboring under from a bunch of different angles
and they're trying to do the best with what they have.
Very often, the reason that a practice or a policy exists is because once
upon a time there was a constraint that may or may not still be there and
going forward the way that they have seemed like the best option at the time.
I found that the default assumption that people are generally smart and doing
the right thing with the information they have carries you a lot further.
in many respects, than what I did as a terrible junior consultant,
which is, oh, what moron built this, uh, invariably to said moron,
and then the rest of the engagement rapidly goes downhill from there.
Try and assume good faith, and instead when you see
something that makes no sense, ask, why is it like this?
Rather than, why is it like this?
Counts for a lot.
It's the fundamental attribution bias.
You know, it's why we think all other drivers on the road are terrible,
but we actually had a good reason for swerving into that lane.
This isn't how I would
have built it, so it's awful.
Yeah, exactly.
And in some cases though, there are choices that are objectively
bad, but I try to understand where they came from there.
Company policy historically around things like data centers
trying to map one to one to cloud often miss some nuances.
But hey, there's a reason it's called a digital
transformation, not a project that we did.
And I think you've got to always have empathy for the people on the ground.
I quite often have talked to folks who've got like
a terrible cloud architecture with the deployment.
I'm like, well, what happened here?
And they went, well, we were prepared to deploy this whole thing on AWS.
But then Microsoft salespeople got to the CTO and we got told
at the last minute we're redeploying everything on Azure.
And so these people were often, you know, you're given a week or two to
pivot around a decision that doesn't necessarily make any sense to them.
And there may have been a perfectly good reason for the CTO to do this.
They got given really good kickbacks, you know, like in terms of
bonuses for like how much they were spending on the infrastructure.
I mean discounts.
But people on the ground are generally doing the best with what they can do.
They end up building crap.
It's because, you know, our system, society,
capitalism, everything else is at fault.
I have to say, I'm really looking forward to seeing the observations
that you wound up putting into this report as soon as it drops.
I'm hoping that I get a chance to speak with you again about the
findings and then I can belligerently tell you to justify yourself.
That was my favorite follow ups.
Brilliant.
If people want to get a copy of the report for themselves
or learn more about you, where can they find you?
Just head straight to Puppet.
com and it will be on the banner on the front of the site.
Excellent.
And we'll, of course, put a link to that in the
show notes, if people can't remember puppet.
com.
Thank you so much for taking the time to speak with me.
I really appreciate it.
Awesome.
No worries.
It was good to catch up.
Nigel Kirsten, Field CTO at Puppet.
I'm cloud economist, Corey Quinn, and this is Screaming in the Cloud.
If you've enjoyed this podcast, please leave a five
star review on your podcast platform of choice.
Whereas if you've hated this podcast, please leave a comment.
Leave a 5 star review on your podcast platform of choice, as
well as an insulting comment telling me that Comcastic isn't a
funny word, and tell me where you work, though we already know.