I'm going to try something new.
This was something that I can do very well.
I think I'm, you know, uniquely suited for being able to talk about
cost and articulate cost, but do it with an architecture view.
Welcome to Screaming in the Cloud.
I'm Corey Quinn.
My guest today spent nine long years working at AWS.
Before pivoting into his current job.
Eric Pullen is a cloud economist here at The Duck Bill Group.
Eric, thanks for agreeing to talk to me, you know, in public.
Thanks, Corey.
I'm glad to be here.
And yeah, I think it's only fair that I come
on your podcast since I work for you now.
Exactly.
It's one of those power imbalance, forced way through.
It's like, "Hey, you're being voluntold to do a thing."
I love being voluntold things.
So that's great.
Backblaze B2B Cloud Storage helps you scale
applications and deliver services globally.
Egress is free up to three times the amount of data you have stored,
and completely free between S3 compatible Backblaze and leading CDN
and compute providers like Fastly, Cloudflare, Vulture, and CoreWeave.
Visit backblaze.
com to learn more.
Backblaze, Cloud Storage, Backblaze.
Built better.
So where do you come from?
Uh, most of the time when I talk to folks who are interested in working here,
their early career, it's, it's a delicate thing to say, but cloud economists
don't just spring fully formed from the forehead of some God somewhere.
There, there's, there's no such thing as a junior person who is
likely to succeed in this role by any angle that I can spot it from.
You've been in this industry a dreadfully long time.
What's your story?
So I built a data centers to start with.
So I, you know, out of college, I worked for a medium sized company and built
a couple of data centers, did all the sort of standards that you would expect.
And I really had to learn the sort of financial side of IT when I did that.
I was a director of IT and, you know, sort of ran the department.
So everything from power and cooling to networking and, Storage
and all the way up to basically the OS levels where we would stop.
And, and so you had to learn not only all the technology aspects,
which we all learned, and you know, that's the part I was most
interested in, but also I had to learn about, uh, what does
CapEx mean and what does OpEx and how do we deal with that?
And that was an interesting addition to my sort of repertoire
of, of, uh, skill sets, but it was also great because I got
to learn not only the IT side, but also the business side.
And I think that's a great learning point for a lot of people
that maybe they don't get, especially if they're junior.
Maybe they don't have the ability to even look at some of these things.
Like, you know, what does the budgeting process
look like for a midsize or even enterprise size?
I mean, those are interesting aspects to, to, uh, any IT type job.
It feels like this is a similar path to the one that I went down and
everywhere I look these days, it feels like the door to do that sort of thing.
Has long since closed.
You're fresh out of school, you're gonna go build data
centers and become the director of the IT department.
That sounds like a 90s to 2000 era story.
Now you do that, the question is, oh great was the company just negligent?
Like, we found a kid out back, we're gonna let
them go ahead and run all the computers here.
And it feels weird now that I'm the old guy because back
then it was, I was the young kid and I was doing this stuff.
And I do think there is a level of that, that one,
a lot of companies don't need that aspect anymore.
I mean, how many companies are building data centers anymore?
So even understanding or needing that technology skillset is just gone.
Folks are insisting that cloud repatriation is a big thing.
Even AWS, a couple of weeks ago, made a whole stink about this, but
only in response to a regulator that said, well, this is happening.
Where is it happening?
The, the data center company stocks are flat.
Is this, is the cloud repatriation serverless?
What's the deal here?
And I look at the server repatriation and all the stuff
that goes with it and people are just going crazy about it.
Let's be honest, these enterprises, they move at glacial speeds and
they've moved to cloud, which is great, and it took them years to do it.
There is no way they're bringing that stuff back on prem in a short order.
So I think it's premature at best to say that I think a lot of
those companies aren't even interested in running that again.
Is that the core business?
I mean, that's the thing you, I think people miss a lot of times.
Like is your core business, healthcare and life sciences
companies, example, is that your core business to run data centers?
No.
It's to design drugs and to do drug trials.
And do you even care what a data center is?
No.
You just need to run compute.
You need a lot of it nowadays, especially with all the GenAI,
but more specifically the old AI stuff that they would do.
And that's great.
You need that, but you don't need to understand and run those data centers.
And I think that is an art that is going away for a lot
of younger people that are getting into this industry.
That's why I struggle when people ask, How did you break into this?
I don't know that I could tell you an answer that would
be applicable for somebody that's coming out of college
today, because a lot of those conditions just didn't exist.
My breakthrough job was being the network admin at Chapman
University, because I, they liked my personality when I
interviewed and the technical interviewer was out sick that day.
They called to offer me the job 20 minutes after I left.
So I crammed like hell for three months and realized
that I had gotten to the level I needed to be.
And a year later, someone approached, Hey,
you want to come work in corporate life?
Why, why would I want that?
Well, we'll give you a 50 percent raise just for showing up.
Okay, I'm going to corporate life.
I'll take that every day.
No, I, and I'm the same way.
I mean, I kind of fell into mine as a career.
I was working in IT when I was in college, and I did it at the, you know,
campus computer centers, you know, trying Slackware on an old, I don't
know, HP desktop of some sort and learning Unix through Next, uh, if you
remember the old, uh, Nextstations and the cubes and the slabs and stuff,
that's how I learned Unix and then Linux and then other things later.
But, you know, I got to learn all that because it was
just stuff to play with because I was working on it.
to save money for college tuition at that time, which was great.
I mean, it was a great environment today.
That's a little bit more, you know, esoteric.
I think that.
Their are whole curricula aimed at these things too.
Oh, you want to work in computers.
Great.
Join the 5,000 other students at this university that are
doing the exact same thing and try and distinguish yourself.
It's funny that you mentioned that I actually taught a course in college once.
Just out of, uh, you know, an old teacher reached out to me at
the University of Lowell and said, "Hey, would you be interested
in being an adjunct professor for this database course?"
Which I was not huge into database, but it was an interesting thing to try.
Glad I did it.
Man, that's a difficult thing to do.
And also it's, It's difficult for me to articulate
to them where this technology was going.
This was even years ago at this point.
But, you know, to be able to show people, you
know, where we started and where we are today.
I mean, the original database for me was, you
know, Lotus 1, 2, 3, or ebase and other things.
And, you know, nowadays you've got a database for everything.
So it's just kind of a crazy shift that we've seen.
But I do think that, you know, when we're looking at, you know,
anybody trying to get into this industry, my suggestion and the
number one thing I do tell people all the time is try to gain
as much knowledge as you can in a broad set of environments.
Don't limit yourself to just working in startups or just working at
enterprises because each of those have positives and negatives, but you need
to learn from all of those different sets to learn the different skill sets
you're going to need if you want to continue to grow up in the IT world.
So after doing this for a period of time,
You decided, you know what I'm going to do?
I'm going to go work for an online bookstores, computer division.
You know, it's funny.
I mean, I was, I enjoyed what I did before I
went to AWS, but I wanted to see what AWS was.
And so in 2015, this was still early days of cloud for a lot of people.
And for us, I was working at a company where we were moving things to
cloud and it was great, and I kind of convinced that team that we need
to go down this road after having built data centers and all that.
So I wanted to see what that was like.
And ProServe had just started up inside of AWS.
And so their professional services group was one where you could work remotely.
And so I was an early remote worker for AWS and a lot of their positions,
even at that time, were not remote friendly, but this is one that was.
And so for me, it was a great opportunity to stay where I live, which is
in Kentucky and still be able to experience Amazon and learn from them.
And so, uh, joined ProServe in 2015.
And I, I really loved it.
I got to work with some of the biggest clients, uh,
you know, it was an interesting time for AWS because
a lot of the tooling didn't exist that does now.
I mean, there was, you know, all the stuff that we do to
create all these accounts and all that stuff that didn't
exist and, you know, transit gateways were not around, we
were, you know, Cisco had a appliance that you could install.
And it was.
We're going to do a whole lot of horrible
things with EPC peering and hope for the best.
It was bad, but it was also exciting.
Come work in our cloud.
An in depth knowledge of several routing protocols is required.
What are you people doing over there?
You really aren't going to like the answer to that.
I'm pre-VPC guy, right?
I remember when we had EC2 Classic, and I was running just Linux running
on a, I think it was Flask Linux at the time, and running it on a
single, you know, one or two machines that had nothing but a security
group protecting them, which was at the time seemed like a good idea.
But now with VPCs and all the constructs that they have, it's, it's
even better and more secure, but for us, it was, it was a godsend
that we even had that capability, but when I started at ProServe,
the thing that I learned very quickly is there wasn't that many
people that knew this stuff outside of people that worked in AWS.
So you got a community very quickly inside of the world and
you had to sort of run that community and learn from all
those different people very quickly because consultancies,
other outside entities just didn't know how to run a cloud.
And so we were doing as much teaching as we were
building because you had to help all these clients just
to understand what are they even going to do on this?
So, you know, and then the other aspect of that is that a lot of people
missed even in the initial onset is how much is it going to cost me?
We'll worry about that part later.
Everybody was like, Oh, just make it work.
And we did.
And then the cost starts to come up and it's like, wow, this is expensive.
Now, some companies, I will say.
I worked with many that actually said, well, it's cheaper than my
on prem chargeback model that the company was using at the time.
And that was an interesting conversation to have with
the internal IT teams to say, look, I can get the compute
for half the cost, AWS, because of this weird chargeback.
And I can get it today.
And they're not going to sit there stonewalling me
for eight months, telling me no every chance they get.
And it was the rise of Shadow IT.
Large enterprise that was doing chargebacks.
I mean, they learned a lot through cloud migrations that all of the
business units were basically saying, no, I'm not going to do this anymore.
And it was amazing to watch them have to shift.
And in some cases they all shifted straight to cloud as well,
because they're like, well, they can do it cheaper than me.
I mean, I have all this sort of layers of complexity
and maybe we just move everything to cloud.
And some of those.
Large enterprises did that and you heard that it reInvents
and main stages and those things, which is all great.
But it was a lot of effort to get in those early days to even get people
to understand what the cost sort of dimensions were around things.
And right sizing was always the first thing that people looked at.
But honestly, right sizing is only a piece of the puzzle.
And I think that's where a lot of companies now are starting to experiment with.
Other pieces, uh, as far as cost to go.
What else did you do while you were there?
Nine years is a long time to work anywhere.
And if you're know anything about AWS, you typically, if
you're there for very long, you don't stay in one position.
I mean, there's probably a few exceptions to that, but most people, especially
in the sales org, which technically I was in the sales org inside of AWS.
You kind of move around because one, you just want to do
more things, but also because you just, you need to, to
sort of keep your career moving on an upward trajectory.
So spent a few years doing ProServe, learned a ton.
Uh, and then I got this opportunity to join the Well-Architected team.
And that for me was a great experience to try to grow my brand.
Was the team itself well architected?
Or was it one of those "cobbler's children have no shoes" problem?
It was an interesting group.
Uh, I will say there was some of the smartest people that
I worked when I first started in that team, we're, we're
from all over the org, and we have all over the world.
Uh, we had team members that went to Australia and
then West Coast and East Coast and the team was great.
I loved working with that team.
It's like a lot of positions though, it's difficult
to articulate your value to the organization.
When there's not hard numbers behind it, we didn't have a sales quota.
We didn't have clients that we were working with where if
they use Well-Architected, they would suddenly increase.
So I think we're Well-Architected.
And even today, I mean, I, I still live architected and I believe in
a lot of its principles, but where it's kind of gone off the rails
a little bit is, is it about the concepts of Well-Architected, or
is it about some tool that we built helps you with those concepts?
Cause I think it's moved towards the tool and less from the concepts.
Tool's kind of a lofty term if I'm being direct.
It's a checklist.
That is effectively what it is.
I was curious when the whole Well-Architected Framework
came out, so when I dove into it, it was, oh, this is just
a formalization of the stuff everyone should be doing.
There were no surprises lurking around in there.
And it's one of those things, I will say this, for all of the faults of
the tool, it was a way for people to not have Excel spreadsheets with the
questions in them, which was an early days problem that we had of that, right?
So, I mean, I think the tool was beneficial in that sense.
It gave you a sort of a mechanism to use that was easy to repeat.
That part was great, but using metrics against the tool, I think, is a bad idea.
Like, how many, you know, uh, essays are doing Well-Architected reviews?
I don't know if that's a metric we should measure.
If a customer doesn't want to do it, then they shouldn't do it.
But, but again, going back to the Well-Architected,
the framework itself was the important part to me.
And I think that's where a lot of people didn't
necessarily understand that that was the key component.
And for a lot of enterprises, they never looked at
it in a holistic way when they look at workloads.
And so you have operation security and performance and cost, all those things.
That's a different way to view it when you look at it in a totality.
And every large enterprise I went into, even
mid-sized companies, we start asking those questions.
They didn't know how to answer the questions from one person's perspective.
We would have to bring together teams to answer all the questions to
even understand one singular workload that they were doing on AWS.
And I think that was the power of LoL Architect.
It was the conversation.
It was getting those teams together to ask those questions.
The tool and all the other things that go with it were just secondary to that.
And I think that's where Well-Architected can be powerful as a
base level of knowledge, but it's something that you have to do.
It's not a tool that you can run.
And I think that's where a lot of people got confused with Well-Architected.
I think that there's been a lot of noise about it, and whenever
you have a formula like that, or a framework, there's always some
folks who are wanting to take it beyond the precepts upon which
it was based and into now the process becomes the reason land.
And that is where I think people start to
go awry with a lot of things, honestly.
And yeah, it's true.
And the other thing is, and if you know anything about
the internals of AWS, you end up with competing things.
Like there was also the Cloud Adoption Framework.
Oh, don't worry.
You see, you get competed both by AWS outside of the company too.
It's a weird sort of system where people are encouraged to
generate new instead of adding on to what already exists.
And I think that's another area where it's kind of
problematic for these frameworks to continue and grow.
The sort of normal mode of operation is, Oh, that thing has reached a lifetime.
Let's do something new because they won't, you know, people want to show value.
That's, I get why the system is built the way it is internally,
but I also understand that it makes it difficult for customers
that, well, is this the thing I should use or this other thing?
Or what, what's going on?
This service hasn't seen a feature release in three years.
Is there someone there keeping the lights on?
Or are you about to come out with a whole series of things at reInvent for it?
Or Is it going to be merged into something else?
Or is it just going to sort of sit there doing what it does?
And service deprecation is something very new for AWS.
I mean, we're just now seeing the beginnings of that.
And I think it's a good thing.
Oh, it's a great thing.
I wish they communicated more effectively as opposed
to having to like go out and find it individually.
But if they can solve that, yeah, turn the stuff off that doesn't work.
And I think there's a lot of teams that wanted things to
be turned off and couldn't get them turned off because.
There was a real sort of hesitation in
turning things off because we didn't do that.
Like that was a, not an Amazonian thing, right?
We built it as kind of a contract to have services running for our clients.
I think what they're learning now is it's just, you can't do that forever.
I mean, there are certain things, you know, having the old
cloud search platform, like why is anybody still running that?
Like, if you're not moving on to something
else, then you need to be told to move on.
And, you know, you give them timeframes and all that.
They definitely need to articulate this better, but there's
no reason to get rid of stuff that's not being used heavily.
They do, in some cases, shut it down to
new users as an early step, which is great.
And I get the sense that there are still customers on these things.
We're like, well, we're definitely going to be turning it off on X date.
No, they're not Google, where with Google, when Google
does a deprecation, it will be ripped out from under you.
When AWS does it, I suspect at some point you drag your heels as a customer
hard enough, they will fly a team of engineers out to help you migrate off.
And
I have actually been first party to that.
And I have seen where we have had instances where we said,
look, we're going to have to do upgrades on X, Y, or Z platform.
Usually it's some managed service platform that we, that they offered.
Uh, and those are ones where they said, look, if there was an
exception, because it was a particularly important client or had a
particularly important workload, they would make exceptions for that.
Now, they would put all kinds of rules and sort of
parameters around it, which is great, but they would do that.
And I do think that's a big positive for AWS is that they're always willing
to kind of bend over for customers and make sure that they're always
capable of doing what they need to do to keep running their business.
So, after nine years, you decided, "You know what I should do?
I'm going to go work with that guy with a big mouth who basically insults
everything that the company I currently work for does and instead of working on
the computers directly, we're going to go talk about the money aspect instead."
Uh, what's wrong with you?
Uh, I don't know.
Uh, I actually, so, I look at it this way.
I wanted to do something different.
I wanted a new challenge, but I also saw inside of AWS,
things are changing and it doesn't take a rocket scientist
to see that things are kind of changing internally at AWS.
It could be for the good, it could be for the bad.
I can't articulate yet whether it is or isn't.
I can tell you though, From my perspective as an employee of AWS,
at that time, it was the risk level has increased at AWS, so layoffs
and return to team, return to office, all these things you hear about
in the public media about AWS, that weighs on you as an employee.
That's certainly something you look at.
Plus, I would just wanted to do something new.
And so I had been talking with Mike actually for a while, and we conversed
about, you know, different roles over time a period of time, and it
was like, it just made sense now that I'm going to try something new.
This was something that I could do very well.
I think I'm, you know, uniquely suited for being able to talk about
cost and articulate costs, but do it with an architecture view,
and I think that's what's really needed for any cloud economist is,
you've got to understand the technology and what's underlying it.
And for a lot of these companies, you have to
understand where the technology even originated from.
And I think, you know, that's where I built those data centers.
I know how that stuff used to run.
I know how it runs in cloud.
And I also know where you can spend money that doesn't make sense.
And so for me, it was just a great opportunity and really enjoyed my time here.
I think I didn't even realize how, um, I don't think I was unhappy at AWS.
I think I enjoyed what I did, but I think
I'm even more excited now than I used to be.
And my wife is joking with me now.
She's like, you seem much happier now than you have been in a few years.
And I think it's not anything other than I'm just interested in doing
a fun new thing and I'm able to grow and learn new things at my own
pace without having to worry about sort of the bureaucracy that is AWS.
Again, there's reasons for all that bureaucracy,
but nice to not have that, I will tell you.
You're also the vanguard of a new approach that we're taking with our clients.
Historically, most of our engagements have been
fixed term, uh, very defined, relatively short.
You're embedded full time at one of our larger customers.
Uh, 40 hours a week, and you're digging very deep into not just
the architectural optimizations, but helping get some of the
changes driven across the finish line, working cross functionally.
And it's, it's a new experiment for us, and it's, it's
going very well to the point where this is, this is
not the only one like this that we now have in flight.
Imagine that.
And And it was, it was fascinating to me when we were
trying to figure out what this role was going to look like.
Something you, you, you stumbled onto very early on in
our conversations was that you, you thought beyond the big
obvious services, EC2, RDS, how to optimize those things.
Great.
Yeah.
Everyone optimizes those things.
There are tools that do it.
We're not going to come up with much new ground on EC2 optimization.
Yeah.
We're just not, but you focus on other areas, uh, primarily, which resonates
with my own hobby horses as well, as far as how I approach these things.
Yeah.
And I think it's really strange that, you know, I mean, you look
at the bill and, you know, we, we all should, if we're looking
at AWS cloud spend, we should be looking at our overall bill.
And, you know, I just like to look at the top 20.
Like, let's just take a really quick and easy
look at what are you spending your money on?
And what I find is every enterprise is a little bit different,
but obviously RDS and EC2 and maybe S3 are the top three.
That's typical.
Uh, you may see CloudWatch up there, although I hope people are working
towards reducing that because there's a lot of money being spent
on CloudWatch that I think is a little too much being spent on it.
But, you know, either here or there on that one.
But, but then you start looking down, look at
the number five and six and seven on the list.
Some of those can be really big drivers of
cost, and people just aren't even focused on.
Things like Databricks Migration Service, or Manage Kafka, MSK, or it
could even be other services like, uh, maybe you're using chatbots,
you know, and you're using Uh, other types of sort of, you know,
ElastiSearch services, other things where maybe there's opportunities
to save money there that nobody's looking at because it's just not at
the top of the list and yeah, you know, maybe it's another, you know,
30K a month that you could save just by optimizing MSK, let's say.
Or Dynamo is another interesting one that, you know,
I've spent some time recently diving deep into Dynamo.
There's a lot of cost optimizations to be done there, and
the tooling for that just doesn't seem to really exist.
Unlike RDS, unlike RIs, and all that stuff where you can pull these levers.
Some of these don't have many levers.
So you have to go much deeper in from an architectural perspective
to really understand what can we even do to save money on this?
Is there even an opportunity to save money?
You have the concept of consulting skill as well, of when
you see a large, a large billing item, your immediate
response is not, This is broken and shouldn't be this way.
Your immediate response is to have questions about it.
Okay.
What is this doing?
What is the business use for it?
How does this interplay with other things?
Is this a misconfiguration?
Is it the mistake or is it underpinning an incredibly important, valuable
business function and we should not look directly at it lest it stop working?
Exactly.
And I think that's another area where a lot of people
that work in, especially in FinOps space, but simply as
an architecture team, if you're an architect on a team.
You should be able to articulate to your upper
management what the value of the thing is.
Like somebody looks and says, you're spending 2
million bucks a month on Dynamo as an example.
That's a lot of money to spend on Dynamo.
But if it's running a service that's generating
100 million dollars a month, who cares?
Like your way spending under what you're
bringing in, it doesn't make any sense then.
Versus the other way to look at it, if you're spending a
couple million dollars a month on Dynamo on a service, It's an
inventory service for the pins that you have in your break room.
Uh, something's wrong.
Like you, you know, there is an articulation of what the value is.
Uh, and even in days from low architect that we try to help companies, you
know, articulate some of the value of a workload, but specifically when
it comes to, you know, looking at it from a spin off standpoint is like,
one of the things that you're spending money on doesn't even make sense.
And so we look at things like database migration service.
Great service from AWS.
It can do CDC, so you can have all of your data on prem
running to cloud all the time, and it stays in sync.
That's a great service to have.
It's not cheap, but it is.
But are you even using it for that?
Or is it a service where you installed it, you moved some data to the
cloud, you moved all your services to the cloud, and you forgot about it?
So now you're paying all this money for something
where, what's the perceived value of that?
And so I think there's a lot of conversations when you look at Further down
the bill is what are these, you know, what's the use case for this stuff?
And is there a use case for it?
And then having that conversation about articulating its value versus its cost.
I think that's an area where a lot of companies just don't see that yet.
Enterprise itself is a skill.
And at a small company, great, you can more or less talk to one or two
people that'll have the answer to any question you've got in their heads.
Whereas in enterprise land, you get to go exploring.
Through a whole bunch of different folk, each have a piece of the
puzzle and you've shown a an adroit ability to navigate that well.
I have a lot of experience in it, I guess, and I think that people tend
to, to know that I understand what they're dealing with in their eyes.
And I think that sympathy is a lot, goes a long way when you're
doing these kinds of engagements where you understand that
this person that you're talking to only has the ability to
affect one very narrow slice of that enterprise organization.
And so talking to that person in the way that they understand things
is a way that you can gain their trust enough that they can tell you
what's even going on, because otherwise you're just another consultant
that's wanting to change their world and is it for the better or not?
Who knows?
But you're, if you're able to sympathize with people and
especially as they're telling you about their problems.
And a lot of these FinOps conversations are
they know maybe there's issues here or there.
They don't know how to fix it, but they know that there is a concern there.
And so you're helping them uncover that.
Say, Oh, you see this cost is increasing, but it's because
of a team that's maybe adjacent to you that's causing that.
Well, how can we open up that conversation?
And me being sort of a third party to that is also at a huge advantage.
Any consultancy you've ever worked in, I'm sure this is the case.
It's like sometimes just being the guy that says, "I have this name
behind me because we're the FinOps team that we're going to help you,
and we're going to uncover these savings," gives you a lot of cachet
with these different groups to say I want to cut across those boundaries.
I need to have conversations at different levels.
It's always been critical from my, my view of the world to
be able to delve into what the technology is actually doing.
There's, there's an engineering prerequisite of understanding how these
systems talk to each other, what the, what the, what the constraints are.
And understanding how making a cost effective change could have
detrimental effects on performance, because, you know, once you
bring something down, you're not allowed to save money ever again.
And it's funny you mentioned performance, because the pillar that
I led for Well-Architected was the performance efficiency pillar.
And if you read through, like, Well-Architected as an example, you know, there's
all this push and pull between the different pillars, like cost and performance.
Yeah, I can make it super performant, but it
also is going to cost you an arm and a leg.
Or when you look at reliability, it's like, I can make this the most reliable
system across multiple regions and, but it's going to cost a lot of money.
And so you have to kind of recognize what's the value that.
of the service you're providing so that you can
articulate what the cost should be for that service.
Like, again, if you have a service where it's counting the number of pins in
the break room, maybe it doesn't need to be multi region, you know, but if
it's counting, you know, I don't know the number of drugs that you're shipping
out to pharmacies, that's an important number that should never go away.
And you may need to...
You're not spending enough on that.
Go spin up some managed NAT gateways.
So we get fewer questions.
Exactly.
So I think it's It's always good to find out what the reason is behind
spend, and sometimes that conversation has nothing to do with IT, right?
This is a business problem.
You're starting to understand why they're doing what
they're doing, and you may even uncover that this problem
doesn't need to be solved in the way they're solving.
So now you've affected an architectural change through
conversations you have about why they're spending so much.
And I think that's another critical piece of what we do,
especially in these newer engagements that Phil is doing is,
we're trying to dive much deeper into that and say, you know,
it's not just enough to say you can save some money with RIs.
You have to look at why you're doing what you're doing.
And it takes time to do that.
It's not a, you know, one week engagement.
You can find all those things out.
It's, it's across the board.
You have to have conversations with the organization.
Organizations generally tend to be fairly poor at
articulating internally the context for a lot of these things.
You'll have some engineers spinning things up that are wildly expensive without
justification, but what I always find more interesting is when I encounter folks
who are well intentioned engineers, but they're treating it like it's their
own personal money, where they think for some reason they can't spend more than
200 a month on their own development environment there, where it's, I assure
you, you cost significantly more than the infrastructure upon which you work.
We need this working.
Don't worry about the cost.
You're really aiming at the wrong part of the story.
And it's funny you bring that up because I have had many
occasions where I've seen both sides of that, right?
I've had people that are unwilling to spend 200 on You know, some
AWS service to try it out and to see if it will fix a problem.
They don't have to build a whole nother service for great or
you see the other side, which is someone that is doing some
minor scripting and they spun up a, you know, 12 XL instance.
So run that EC2 assist because they don't care.
It's not their money, and they wanted it to be fast, even though it's
way overkill for whatever the thing is, they were running, and they
leave it running forever because they don't want a pull shutting it down.
There's a weird dichotomy you see, and I think that's
probably true for their personal finances as well.
Maybe they don't think about it for their own money.
Yeah, I've noticed my personal EC2 dev box has gotten larger over the years.
It's now a C7G large and it's, um, that, that's
a little beefy to be running all the time.
But, you know, in the, I need something there that it works.
I don't have to wait for things to, uh, to complete.
I'm spoiling myself.
And that's true.
And spoiling yourself sometimes, like you said, is needed to save you
time and effort if your job is to do building scripts or whatever.
And then the money kind of works itself out.
It's when you see hundreds or thousands of those that
you start to wonder about, you know, and it's, that's
when you start going, well, why are you even doing this?
And that's a question that you have to kind of unfurl the answer to.
It could be cultural inside of a company.
It could be that they've been dictated that this is the way they do things.
And so the first person on the team built a 12XL, as everybody else does too.
A stupendous number of things out there just, uh,
workloads are running on a single instance size.
Okay, why that?
Well, the developer who was building it that day, five years ago, just
pulled something out of the ether, which when you're developing is fine,
but it becomes those load bearing to dos in the backlog of, oh yeah,
we should go and figure out what the right scale is for this thing.
But once it was up and working, it's suddenly in
production and well, that's not a high priority right now.
There are features to ship.
And the other thing that I've seen at a lot of enterprises
over the years is this sort of ability that the enterprise
wants to have to say, we're going to have standards.
And so they want to pick a size of like, EC2, or maybe it's a higher level
service like RDS or something, like, every RDS instance is going to be a 8XL.
Why?
They're just doing it because they want to set a standard and there's
a group that sets standards in large enterprises, and they're used
to doing that when they had on prem, it made a lot of sense, right?
Economies of scale, I'm going to buy the same size server or the farms
that I build internally, okay, that made sense to have one standard size.
It doesn't make any sense in AWS.
I will caveat that there.
The time we're in it does is, are you getting a screaming deal on a highly
specific private pricing agreement on this one instant size family and region?
Because if so, that shoots a hole in everything we just said.
I suspect you're not, and yeah, usually when that
happens, you know going in that that's the story.
And in my experience, what I've seen when working in AWS is the
ones where the, you take advantage of that is weird instance
types, GPUs, obviously a lot of people, but I thought with F1s.
So if you have, you know, certain industries where FPGAs
are heavily used for certain tasks, and so they would get
private pricing for those F1 instances, makes total sense.
But that's a very sort of esoteric use case.
It's a weird choice for a bastion host, but okay, you do you.
Yeah, exactly.
Whatever.
Uh, but I do think that, you know, a lot of times companies
are just built around standardization when they're enterprises.
And it gets to a certain point where that made sense for them.
But I do think that's an area where you have to look
at it and say, what is, is this making sense anymore?
And it's the old days too, with standardization
around database engines, as an example, right?
Everybody was standardizing on Oracle or Microsoft SQL or whatever.
It's like, yeah, that's a great concept.
And I get why they would do that because a lot of the heavy lifting.
Call legal.
We need to figure out what the license terms are on this.
But now with open source engines and things like Aurora and,
and, and MySQL and Postgres and all those, it's like, pick
what makes sense and let the provider run a lot of that.
Low level infrastructure, you know, replicating data.
I don't know if you remember, but I remember how difficult that was
across data centers, like having an Oracle box and using GoldenGate
to ship data from one place to the other, it broke all the time.
And now you have multi AZ for something like
RDS, you don't even have to think about it.
It just works.
And, you know, that's an area where I think a lot of people,
especially coming into the industry now, if they're new is.
They don't even know how difficult some of these things were years ago.
I spent months on that at multiple companies, just trying to get
replication between two sites that aren't that far apart working.
And it was brutal.
In some cases, in the same facility, just in a different rack.
Yeah, we had a Macross facility.
We did Golden Gate across two data centers, and it was a pain in the butt.
They didn't even, like, think about file replication.
Look at how easy something like EFS is today.
Or, you know, there's even ONTAP and other things.
That's a pain in the butt to build in a datacenter,
and if people haven't experienced that...
I mean, it is Well, ONTAP was the solution that you used back then.
Waffle's still one of my favorite file systems.
But you had to spend top dollar for NetApps to make it work.
And you still had to have all these restrictions on how the two buildings
were interconnected and there's all kinds of rules around that stuff.
Whereas with AWS, you just, you know, click a
button and say, I want a file system and it does it.
DFS is magic.
I mean, I, you know, I remember the early days of EFS when it got pre announced.
It was one of the first ones that they pre announced that didn't come out
because it didn't work so well from what I was told, but then later, when
it did come out, and now you look at it as a service today, it's a great
foundational service, and it works really well for a lot of different use cases.
It's not perfect for everyone, but if you're
needing NFS, man, it's cheap and it's fast.
It's cheap, it's fast, it replicates super well.
The only problems I've had with it are more
or less when I have made poor life choices.
Like, okay, why does it take me seven seconds to log into
my box every time when I put my home directory on EFS?
Well, yeah, maybe it's time to redo my crufty decade
old ZSHRC, which makes apparently 1500 calls to, oh,
across the wire for that, which it never expected to do.
It's maybe stop doing horseshit and see maybe
it works better and wouldn't you know, it did.
You know, that's another area where I always find it
kind of amazing that Amazon does eventually get around to
building services on top of other foundational services.
I mean, you look at like EFS is a great
service, but you also look at what uses EFS.
You know, you can use it with Lambda.
You can do all these interconnections, but then you even think about, I
remember early days when EFS first got announced, the reason I wanted it is
I had an enterprise that had FTP servers or secure FTP servers at the time,
and I wanted a quick and easy way to have a centralized file system because
that was a pain in the butt at the time, and EFS looked like a great option.
Well, it wasn't because of a whole bunch of restrictions that it had.
But lo and behold, Amazon says, "Oh, we could
build a file transfer service that frontends S3."
I requested that service when I was at BlackRock because it was, I
think, my fourth company where I had built the same thing where it's,
all right, you're going to use lib notify or inotify on an FTP box to
make sure the transfer is done and then validate it and wait a while and
make sure the connection's closed and then go ahead and throw it into S3.
Can I just pay you to make this go away?
Because it's always for big enterprises.
It's not for you, shopping about, sharing with your mates, but for
enterprises trying to solve for enterprise workloads, it is cheap as free.
And people don't realize how tedious that process was to manage file transfer.
It always broke.
Yeah, it broke and it broke in ways you couldn't figure out.
And it's just, and now having the file land in S3,
which is where everyone wants it to land anyway.
At the end of the day, to then kick off whatever other process.
I think there's that whole aspect of things when people look
at on-prem workloads that do FTP and do file manipulation
and then push it through a process and a workflow.
That stuff is dead simple now in a cloud environment.
And it's an API call away, clickety clickety, and it
costs pennies per hour, and you don't have to worry.
And it's so straightforward now for people, and again,
this is the history of people coming to the industry today
don't even understand how difficult some of this stuff was.
Now people come in and go, why would you do
that when you can just push the file to S3?
I would have loved to have done that over the years, but
every other enterprise didn't want to touch S3 at the time.
Yeah.
Go ahead and just teach that giant bank over there how to use S3 instead of FTP.
It's like, do I look foolish?
I wish someone had told me.
Nobody ever wants to deal with that stuff.
And I think, you know, that's technology changes.
That's great.
But being able to still have some of the old messages
still work in a cloud friendly way is a great advantage.
And I've seen that, you know, with AWS, I think that's one of the
areas where they really shine is they're trying to bridge that gap
between what we used to have to do and what you could do today.
And I think there's a lot of areas there that are optimization
efforts that you can look at when companies are moving to cloud.
So this whole repatriation and moving stuff back is like.
Yeah, did you think about all the stuff we're going to have to do again?
You just moved a whole bunch of workloads into cloud, dude.
It's okay, maybe yours are better ways to do these things instead
of having to have that fleet of servers to build a message cube.
Have you heard of SQS?
It amazes me too, when you read some of those stories, and I've read a few of
them, where it's like, did you just not try to optimize before you moved back?
There are a few cases where I could see that it makes sense.
I mean, if you're just, all you're doing is, you know, I don't know,
some really simple website and maybe you could get away with just
one single server and, you know, you're only serving a hundred users.
Okay, maybe it, maybe it doesn't make sense.
Let's be clear.
It can be done.
We have, this is not just knowledge of the ages.
We, we still know how to run these servers at scale.
It's just a question of, is that where you want to invest time and effort?
It's a decision every company has to make.
My assumption is, is that people are doing what's right for them.
And I don't see a lot of people talking about what that cost is.
I mean, every story I've read about is like,
we already had teams that could do this work.
Great.
What if those teams go away?
I don't know too many companies that are spending
more on infrastructure than they are on people.
Agree.
And it's, it's a weird, Set up to say you're going to save
that much effort, you know, by moving it back on prem.
I don't know.
It's just that whole repatriation stuff.
It just doesn't make sense to me.
It does not.
Can I just say, I'm glad that you're here.
I appreciate the chance to work with you and I'm
enjoying, I'm enjoying seeing how this plays out.
If people want to learn more, Where's the best place for them to find you?
Well, they can find me at The Duckbill Group.
And so feel free to come to duckbill.com, and you can find
out more about reaching out to one of us as a cloud economist.
I would be happy or Mike or Corey or any of us can
reach out and, uh, find out what we can do to help you.
Cause I would love to have more effort than just the one project I'm working on.
I would love to do many more of these.
And I think that there's opportunities abound for different enterprises
that want to engage us or even small to mid sized companies.
Everyone can save money if you're in cloud.
And maybe again, you can look at those non, you know, EC2 RDS type things, and
maybe we can help you find areas to save money that you didn't even think about.
So.
I really want to thank you for taking the time to speak with me.
I appreciate it.
Thanks Corey, I appreciate it.
Eric Pullen, Cloud Economist here at the Duckbill Group.
I'm Corey Quinn, also Cloud Economist here at the
Duckbill Group, and this is Screaming in the Cloud.
If you enjoyed this podcast, please leave a five
star review on your podcast platform of choice.
Whereas if you hated this podcast, please leave a 5 star review on your
podcast platform of choice, along with an angry comment that you'll no
doubt be charged way too much money for because you don't realize what that
podcast platform is charging you because eh, that's someone else's problem.