I think honestly I love hiking.
Um, I've been section hiking the Appalachian
Trail and Corey joined me on this current leg.
And I think what I love most about it is the community
Welcome to Screaming in the Cloud.
I'm Corey Quinn, and today's guest has had just about enough of me.
Caleb Hurd is many things.
He's periodically a cloud economist here at Duckbill, but he was also
my hiking buddy for a 60-mile section hike of the Appalachian Trail.
Caleb, thank you for continuing to speak with me.
You're welcome, Corey.
You're welcome.
It's a pleasure to be here, and, uh, I would say six days on the trail
with you is very much like s- Screaming in the Cloud and Last Week
in AWS, in that you don't really know why you do it, but then at the
end of it you feel like you should have done it and you appreciate it
This episode is sponsored in part by my day job, Duckbill.
Do you have a horrifying AWS bill?
That can mean a lot of things.
Predicting what it's going to be, determining what it should be, negotiating
your next long-term contract with AWS, or just figuring out why it increasingly
resembles a phone number, but nobody seems to quite know why that is.
To learn more, visit duckbillhq.com.
Remember, you can't duck the Duckbill bill, which my
CEO reliably informs me is absolutely not our slogan.
Thank you.
I, I still remember a few of the things you said along the way.
Uh, for example, y- you confessed that you thought that a lot of
my jokes on Twitter were AI-driven until you started talking to
me and realized, no, I talk exactly like my Twitter account does.
Yeah, pretty much.
Sarcasm remains my first language spoken at home.
Yeah, we wound up doing, uh, what was it?
10 miles a day on average for six days, uh, at the end of which everything hurt.
It was fun.
Uh, carrying, carrying obsessively about how much everything
we carried weighed was certainly a new experience.
And, and then talking to folks who were doing three,
four times our mileage, good for them, good for them.
I, I, I choose happiness in life.
Yeah, I don't think managing ounces that you carry in your pack is a new
experience for you at all because that's exactly what you do tuning AWS bills.
Um, I feel like managing weight is like you load everything in you need, and
then halfway through the trip you start throwing it off the trail, not that we
would ever litter, and you end up at the end of it with what you exactly needed.
Oh, it was… It's great too.
My back is super strong from carrying AWS marketing all these years.
So I don't know why I was so focused on being ultra-lightweight.
Y- you had a knack, though, for finding very expensive things along the trail.
Just people had littered, and you, well, you picked
them up, and one, for, to be a, a good citizen.
Like, you found a headlamp that was not inexpensive and also
a pound and a half just because someone hated themselves.
I can see why they wanted to leave that behind.
Please don't litter in the woods.
But yeah, you, you had a knack for being very lucky around things like that.
Yeah.
It's true.
And for the listeners, uh, for the podcast, day one, I was convinced
that Corey had memorized all of his jokes and AI-generated content.
Uh, day two, I started to suspect it might
actually be coming out of him naturally.
Day three through six, I realized this is actually just
how Corey talks, and in fact, AI is the one catching
up, and he probably doesn't generate any of his content
Not really.
Sometimes I'll workshop titles, and that's, that's about enough.
Honestly, in this case, I don't even need to.
The title writes itself.
Appalachian Cloud Trail.
It's beautiful.
A wonderful podcast title.
I mean, the best part of it was we didn't have
to think about clouds for the entire trip.
The internet can't find us there, and y- you learn a lot about yourself
when you're, you know, huddled in a tent in a rainstorm, or in your case,
trying fruitlessly to convince me to switch to a hammock all the time,
or, you know, when my sleeping pad exploded under me at 10:30 at night.
Uh, then as I'm lying there in the cold, dark rocks, I get
to hear the American singing retrievers 100 yards away.
Uh, the only difference between them and coyotes is my
unwillingness to admit when I've made a terrible mistake.
And it, it was just a, a series of fun things that at the
time didn't seem like fun, but now I'm nostalgic about
them because anything's better than dealing with AWS bills.
I don't disagree, and what, what really the highlight of the
trip, and you probably should take a step back, and we should,
we should walk them through it in chronological order, but,
uh, was being saved by Shepherd, our, uh, our shuttle driver.
Yeah.
Why don't you begin that story?
'Cause otherwise it looks like I'm dunking on
you, and it's more fun if you dunk on yourself.
Please, take it away.
We'll get to the cloud stuff.
Don't worry.
But I want… I… This is the story worth telling.
It's, it's phenomenal.
It's, there's the three part, and so I'll try to go through
it rapidly, and you, uh, jump in when I miss a detail.
But we started off parking my vehicle at the bottom, and then we
had a shuttle driver drive us to the top, and his name was Tom.
The bottom, top.
There was a lot of uphill in, in a lot of our hiking.
It was… There… We, we had what?
Something like a mile in elevation?
Oh, it was, it was something crazy.
I don't even remember.
It was, it was, it was significant, and every day was hard.
It… There m- we might've had one flat day out of the six.
Like, it was pretty wild.
Or one flat night when my sleeping pad exploded, but we'll get there.
Please continue.
We definitely had a flat night.
So, uh, a couple days in, I don't know, night two,
I can't remember which exact night, I slept great.
I heard nothing other than the, the coyotes,
which apparently are, uh, not coyotes.
But apparently in the middle of the night, Corey's
sleeping pad blew up, and early in the night, too.
It was, like, 11:00 PM or something like that.
Eight-inch blowout.
We're not talking a slow leak that you get to find.
We're talking all the air left in about a second and a half.
Uh, in fact, the next morning you asked
if I had gone to sleep with ice skates on.
That's right.
It was pretty wild.
It was an unrepairable, uh, thing.
So of course Corey, uh, he had a great night.
He managed to turn his backpack into a, a sleeping bag.
Oh, I s- I left a review on the Durston subreddit.
Uh, it was the Kwaka, uh, 55, and I said this is a solid 8
out of 10 as a backpack, uh, 2 out of 10 as a sleeping pad.
That's exactly right.
So we were trying to figure out the next day how to not
have Corey sleeping on the ground, and I was like, "You
know what? I should text Tom, our shuttle driver," 'cause-
And oh, I wanna be clear.
Before, before this point, you were extraordinarily kind.
You're like, "Well, you should've woken me up."
Like, why?
Like, like my… One of our… Like, we both have young kids.
Part of the reason we wanted to go get lost in the woods.
It's like I… Like, they're, like, standing by the side of your hammock.
"My blanket fell on the floor. I had a bad dream. I want some water." Like, no.
I… There's nothing you can do.
I'm just gonna…
Like, I'm already not getting a great night's sleep.
Why do I make it both of us?
Because I'm not a jerk.
That's true.
He did let me sleep, but, um, that's, that's accurate.
So the next day, me with my full night's rest in a beautiful
hammock, by the way, which you should switch to sometime, Corey.
We woke up and…
Well, he- Corey didn't wake up.
He was already awake.
We texted Tom, and basically we're like, "Hey, could you shuttle
us into this town nearby, Hot Springs?" And we can buy a sleeping
pad at, you know, wherever- 'Cause we were shooting two nights
out, and I wasn't gonna do that to my back
… right.
No.
That's, that's not, that's not how…
Nobody wants to do that for multiple nights in a row.
And, uh, Tom is like, "Hey, um, why don't I just run you one out?"
And so next thing you know, we hike to a road, and our shuttle driver
runs us a sleeping pad out that Corey, uh, uses the rest of the trip.
It was phenomenal.
So that was, tha- those are the first two parts of the story that involved Tom.
I liked it because it d- it wasn't as lightweight, but it also didn't explode.
Uh, w- we'll get there.
There is an advantage to a sleeping pad that functions.
And then after we rolled into Hot Springs, slightly before we
rolled into Hot Springs, I realized that I had left my phone at the
campsite where we had just walked out of, and there was no way that
I was gonna hike, you know, multiple miles back to go retrieve it.
I started doing the math in my head of, like, "How much do phones
cost these days?" So we ended up at a hostel to do laundry.
It's a VM
iron.
What's, that doesn't sound like a bad thing to litter in the woods.
And maybe we're uplifting the technological
development of the local squirrel population.
Why not?
That's right.
No, I was leaving a relic to be discovered by future generations up there.
But anyway, so we mention it to Tom, who was at the hostel.
He happened to be one of the people running the hostel, and, uh, his
trail name is Shepherd, which we quickly discovered why it was Shepherd.
Uh, and he's like, "You know what?
Let me just trail run back.
I can just trail run back and get your phone for you." And next thing you
know, this guy, who's an absolute legend, trail runs back to, uh, where
we camped at, retrieves my phone, and runs back before the sun even sets.
It was wild.
It's super easy to picture this guy because picture a guy that can
just run up the side of a mountain like that to find a phone with
weak-ass directions for where this is, find it, and come back.
That's exactly what he looks like.
Um, but it was great.
He's like, "So how will I know what's your campsite?" "Well,
there's a tree that looks like it's about to give out at
any moment and obliterate the fire pit," which, okay, great.
He knows nothing about us, whether we know what we're talking
about in the woods, and he's, he's walking around the area.
And then he sees that giant tree about to obliterate a fire.
He's like, "Yep, this is definitely the site. Holy
crap." Yeah, like, we, you started setting up.
You saw it from the other side, and you said, "Yeah, well, let's
move about 20 feet that way. Uh, choose life. Fine." A- and he
just did th- he, he was, he pulled a rabbit out of a hat for us.
It was the, it was the weirdest thing And you could, like for us
it's this legendary story, and for him it was any given Tuesday.
Yeah.
I think honestly I love hiking.
Um, I've been section hiking the Appalachian
Trail, and Corey joined me on this current leg.
And I think what I love most about it is the community.
Like, just people are willing to just jump in and help.
If you need something out on the trail, you can't just
go to a city, you know, to a town casually and grab it.
People, you know, I… For example, um, there was a guy passing us whose water
filter gave out, and so Corey and I, Corey and I both have water filters.
His is better than mine.
I won't admit that.
Don't tell Corey that.
It'll get to his head.
And so it was like, "You know what? I can use that water filter.
I'll give you my water filter." But, and it's not just me.
Like, the whole trail has that communal feeling to it,
which makes the whole experience even that much better.
Yeah, we ran to an old guy with a dog who was, uh, three miles from water.
We gave him the last of ours, 'cause we're, we're young enough
that we could, we could still deal with getting there, getting
a little dehydrated, and then just drinking three gallons of it.
Um, we want… Yeah, it's, it's the thing you do.
Like, who needs it more?
Exactly.
I have a similar feeling doing, uh, cloud economy work in that there's so
few people in this space, especially ones that have heavy duty engineering
backgrounds, and it's a similar community where everybody just jumps in.
Like, there's a lot, uh, there's a lot of sharing of knowledge,
there's a lot of opinions, there's a lot of esoteric information
that's out there, and every time I come across somebody in my space
that works in this field, they just freely give that knowledge.
It's one of the things I enjoy about working in this space.
Yeah.
Uh, uh, one thing that I found, the, the reason I like doing stuff like this
is it's been a month since we got back, and I'm sti- it's weird how quickly you
get accustomed to it, but y- but I'm also making it a point to notice it now.
I dunno if you're listening to this, uh, you're just hearing my
voi- my dulcet tones in a mic, but if you're watching this on the
YouTubes, uh, this is my first recording from our new Duckbill office
in downtown, and I'm sitting in a comfy chair at a desk with a sparkle
water in my hand, and everything is comfortable, everything is clean.
Uh, I don't have to dig a hole if I wanna poop.
I mean, I can, but that's generally frowned upon in the office environment.
Uh, uh, yeah, it… And I don't have to care how much something weighs.
It's if I'm gonna… Am I gonna take this home tonight or not?
Well, weight isn't really a factor on that.
So it, it's just, it's a different way of living on some point.
At some point it's, it makes you more thankful for what you've got.
Like, I, I still wander the grocery store in awe compared to how p-
how much of a pain in the neck it is to feed yourself in the woods.
Yeah, no, I think that's a great… Like, perspective's
everything, and there sh- it's so easy to go through life fast.
I think the, the best way I can describe coming off the trail, and it
being a month in, this is probably, this feeling's already passed for me.
Um, but the first probably solid week, uh, when I
come back from a long hike, everything feels fast.
Meetings feel fast, conversations feel fast, cars feel fast.
You know, even schedule, like even my calendar, everything just feels fast.
And when you, when you're out on the trail, you, everything's serial.
You have, like, one task to do There's no multitasking and, and
everything happens in order, and your brain can follow it in order.
And it's a real reminder that, like, cognitive
overload in your day-to-day, it's a real thing.
Like, simplifying what you have in front of you, uh, and,
and doing it serially really is an effective strategy.
Yeah.
What, what's also wild is I, I then got to spend a whole bunch of time harassing
you after the fact with, uh, references to the callback jokes we had on it.
Like, we, we decided that everything was heavy.
Why don't we just build gear full of helium, uh, so that it actually
has a negative weight, and that means you could take a lot more with it.
And like three days later, I'm at home and I sit bolt upright and text you.
It's, "I figured it out.
We're gonna sell this, and we're gonna brand it, and here's how.
Most sleeping bags are filled with down.
These are the only ones that are filled with up." And you had
the courtesy of not responding to that, and that was great.
Uh, uh, thanks to the magic of AI image editing, I wound up finding,
uh, like I did some rain gear that looked like Rudolph, and I
called it Rudolph the Red-Nosed Reindeer, and that was awesome.
I wound up with a blister on my heel when I switched away
from toe socks because it turns out I am lactose intolerant.
Ah.
That was, that was one of my favorites.
It was great.
It was just great being able to throw the dumb jokes
at someone who will absolutely smack them back.
No, it was, it was… We had a phenomenal time.
It was, uh, it really was… It's something I wish I could do more
of, like, and not just me, I wish, I wish everybody, I wish there
was, like, a public holiday where everybody went and spent a week
on the trail, 'cause I think we would all view life differently.
Right, and what do you do when you're trying to
climb up a mountain that never seems to freaking end?
Well, you come up with trail facts about Tom.
For example, that, like, he doesn't use GPS positioning himself,
the satellites just realign to match his trail out of respect.
Uh, he passes the white blaze, they repaint themselves.
You know, usual stuff, one would expect, Chuck Norris fact
style, only Tom is still alive, 'cause that man will never die.
Well, and that's true.
I re- I refuse to believe those headlines.
Um, another example of having time is we were headed up one particularly
bad, uh, climb at the end of the day, so we were both exhausted, and this was
towards the end of the hike, so we were even more exhausted, and I think I
turned to Corey, and it was like, "I think we've got about an hour of climbing.
You should tell me a joke," and he proceeded
to tell me the world's longest joke.
And it, I think it is titled The World's Longest
Joke, and I kid you not, it was at least, what?
An hour in the telling, Corey?
Oh, yeah, it had to be.
Yeah, and I didn't feel one single step.
Uh, like, time just basically teleported me to the top, uh, when he
finally got to the punchline, which was very much worth it, by the way.
Exactly.
Maybe one of these days when I have nothing better to
do, I'll record an hour-long podcast of just that joke.
I, you know, I'm here for it, 100%.
I would, uh, I would be your audience member.
I could hop on, and, uh, laugh and interject appropriately.
Yeah.
It, it was an awful lot of fun, just doing something completely different.
And then we went back to work and worried about cloud finance and the rest.
And one of the reasons I love working with you is
that you've been doing this almost as long as I have.
Uh, you were the guy that went on stage a while back and talked
about putting Kubernetes clusters in every Chick-fil-A restaurant.
I don't know if that's the secret ingredient
in their sandwiches or not, but good job.
That's, uh, sort of everything tastes better with Kubernetes.
Like, maybe it's like trail hunger, where you're,
uh, you're hungry, so everything tastes better.
Uh, with Kubernetes, it's the, it's the suffering that
infuses the sandwich that makes everyone just enjoy it more.
Maybe that's the knack.
That's right.
It's like, uh, lab-grown diamonds versus real diamonds.
It is the suffering that makes the, uh, that makes the diamond.
Jesus.
Is there a con- is that a conflict diamond?
Well, not until I fight you for it.
Yeah, exactly.
That's funny.
No, so, uh, yeah, it's… I've been doing this for a long time.
I think I, I really enjoy doing cloud economy work.
Uh, I, I do consulting for Duckbill.
Um, I also have a full-time job, keeps me busy too.
And o- one of the reasons I enjoy it so much is because, um, in
order to be really good at it, you have to have walked the gauntlet
of racking and stacking as an engineer so you know op-prem pieces.
You have to walk the gauntlet of operational work, keeping systems
alive, because cost is only one element of consideration for an
organization, and if you bust through the door and you're like, "Hey,
these are the f- you know, you could, you could save a ton of money
if you could just turn production off between the hours of 11:00
and 6:00 AM," people are gonna look like at you like you're crazy.
Like-
The United States Social Security Administration
basically does, but that's neither here nor there.
It's some sort of mainframe backup job that means that they
have actual, I'm not kidding, business hours for their website.
I don't doubt that in the least.
I mean, you know what?
Maybe they're actually ahead of the curve.
Maybe everybody should take their example.
Maybe it's not them needing to come- The
computers are tired, Caleb.
Let them sleep.
Yeah, you know, everything comes back around, right?
Maybe we should all regress to 1980s models.
That's, that's one of the things I've been helping people
with recently is convincing them not to go back to on-prem.
Oh, it's funny.
We've had a number of clients approach us to this, and
I, I swear every one of them is gonna think that we're
calling them out, but no, no, this is a larger trend.
And one of the fun things that you and I got to do because we're
hiking through the woods with not much else to talk about at one
point, was try to out trauma each other with what's gonna suck the
most that people don't see coming about building out a, a data center.
And we talked about things like rack nuts, and bad batches
of cables, and Dell screwing things up, as is their
nature, and Cisco doing the same, as is their nature.
And then you have remote hands, which are great, and you have smart hands, which
says something really insulting about remote hands by a juxtaposition there.
And, and all the, the trauma and the rest, and we mo- and, like, the
speed of Jason, Jason APIs, uh, which is when you have a guy named Jason
who opens a ServiceNow ticket via fax and eventually implements it.
And I'm not saying that, that on-prem is always a bad idea, but the companies
that have never done it before are opening up a bag of pain that the clouds
have done such a good job abstracting away that people don't realize it's
there or believe that it exists to the degree that we're talking about it.
You and I collectively have damn near 30 years of
experience working in data centers and the like.
Uh, the fact that we do not work there anymore is not because we are stupid
This episode is sponsored by my own company, Duckbill.
Having trouble with your AWS bill?
Perhaps it's time to renegotiate a contract with them.
Maybe you're just wondering how to predict
what's going on in the wide world of AWS.
Well, that's where Duckbill comes in to help.
Remember, you can't duck the Duckbill bill, which I am reliably
informed by my business partner is absolutely not our motto.
To learn more, visit duckbillhq.com.
Yeah, I mean, at the end of the day, like at least your cloud
bill edges up over time, and you get to investigate it and
figure out why, and then troubleshoot it and lower that bill.
But with data centers, like you can wake up tomorrow and, you
know, it can be a, a catastrophic issue or outage and or a-
Through no fault of your own.
Well, well clouds take outages too.
Yeah, but the difference there is, one, everyone is gonna be down at the same
time, and two, if it's a foundational infrastructure issue, they have a team of
the best people in the world that are working to fix that faster than you will.
It's… There, there's no if, ands, or but around that.
It's a different universe.
Yeah.
My DR strategy if us-east-1 goes down is I'm gonna go, uh, grab Corey,
and we're gonna shelter it out in the Appalachian Mountains in a cabin.
And or depending on whether the world is still survivable at that
point, we will also make all the money consulting people with very
expensive problems all of a sudden because people generally don't plan…
Well, like we've seen the regional outages in the Middle East.
Yeah, geopolitics matter.
If, if we see a us-east-1, uh, regional outage, uh, that says something
concerning from a physical infrastructure layer after it goes about a week
'cause this is about 100 square miles in northern Virginia we're talking about.
Yeah.
Actually, you got a good point.
If us-east-1 was down for more than a week,
you would see Duckbill become- Pacific
Coast Trail.
Pacific Coast Trail.
Yeah, we don't wanna be on the Appalachian at that point.
Yeah, exactly.
Or and or once we come back from the trail, we'll, we'll retool
Duckbill to focus on migrating people out of the cloud onto on-prem.
Ducks are migratory.
I mean, you know, arguably we could make that argument.
But, uh, more duck less bill now.
More duck less bill.
Amazing.
So what was your, uh, what was your favorite
part of, uh, of hiking with me, Corey?
It's weird because at the time, the thing you look forward to the most is
we hit the halfway point and managed to eat like 5,000 calories in a meal.
That was awesome.
Uh, there was one, uh, place in Hot Springs that was dialing in their
pizza oven, and it was… They were so good at… They… There was a
sign that said, uh, "Best pizza in town." Uh, that was a fucking lie.
Uh, I love the fact that both of us were starving and
still wouldn't eat that piece of crap, and then we went
to the actual pizzeria, and it was freaking incredible.
Everything else at that place was amazing,
but it was, uh, but it was very much a eh.
Curiously, the two of us got to meet up with Chris, uh, my podcast producer.
So it was, it was fun being able to sit down
and talk a little bit of shop about this.
We tried to convince him to use Kubernetes for something, mostly as a prank,
and yeah, it was fun just being able to let go of a lot of the work stuff.
It, it was also nice to not have to worry about the complex things and instead
worry about the simple things that are a little bit more survival-oriented.
Like, "Okay, I don't like the look of those clouds.
We should probably find a decent camping spot so
we're not surprised off the side of the mountain.
Uh, uh, wow, we're getting a little low on water.
How do we believe that, uh, that next water source ahead is gonna
be sufficient, or should we use this crappy one while we've got
it?" Uh, there's, it's… These are decisions that are great.
Frankly, you also sleep like the dead after your
second night out there just because you're bone tired.
Yeah, even, even if it is on a backpack.
Yeah, you know, we take what we can get.
Uh, I, I did send… To their credit, uh, it was a NEMO Tensor Elite.
This thing's about a $250 sleeping pad that weighs about as much
as a postage stamp, and it's, it's a great pad, but it's delicate.
And to their credit, they replaced the entire thing with a, uh,
position of just write the RMA number on it in permanent ink.
Great, we're mailing you a new one.
Sorry for the trouble, which was super appreciated.
But also now the problem I've got is one of these already failed me once.
Would I trust it as my only sleeping option going forward?
I don't know.
It's a, there's a question of faith.
Uh, and this of course is similar to cloud providers.
When, uh, you know how a cloud provider fails and how
they handle that failure from a business perspective.
Um, when you're dealing with a new provider, well, you don't know
how that's going to break, and furthermore, there's a terrific
opportunity for them to either gain or lose you as a customer forever.
Uh, I'm saying nice things about NEMO.
Like, their Tensor product line is pretty great.
It's delicate, but it worked super well.
If they'd been a jerk to me, oh, that would be the headline instead,
and I would've been tearing into them, but they did the right thing
There's a… People, I think cloud providers forget the human component.
Like, Google can't wait to not talk to humans as fast as humanly possible
Do I get, uh, do I get to split the, uh, product placement fee for Nemo there?
You know, I, I miss the days when I used to sponsor this with
something other than Duckbill itself, uh, just because it's great.
Hey, why not?
It's… At some point you start to wonder,
like, whether the audience is no longer aligned.
But well, you, you went from ad- advertising enterprise
SaaS and observability products to camping gear.
Do… Bad business or do you know something we don't?
Like, eh, you never know.
No, but it's nice not to have to be Cloud Billy Mays,
uh, pitching for something different every week.
Yeah.
No, it was a great time, and there's so many parallels between, um, you know,
camping and doing cloud economist work from, you know, planning and managing
the load you're gonna carry to, you know, going through AWS bills and trying
to understand services and how that impacts organizations, and the communal
aspect of the trail versus the communal aspect of helping these organizations.
You know, everybody piling and helping.
Some people being resistant to, you know, some people
being like, "You're here to just to make me look bad."
It's like, "No, no, I'm not here to make you look bad.
I really am here to try to help the organization."
Well, you have to be careful how far you
stretch that analogy to the breaking point.
Otherwise, it starts to sound like a really bad long form LinkedIn post.
Uh, what being brutally surprised by a raccoon taught me about B2B sales.
Like, I- Uh, I mean, I… You know, I could see that.
I could see being brutally surprised by a raccoon helping me with B2B sales.
I think I would hate both things just as equally much.
No, raccoons are great, except by surprise.
That's right.
Oh, man.
But, um, but yeah, it was, it was, it was a good time
So I, I guess my question now is since we're, we're back and ostensibly
having to think about work again, what do you see that's exciting coming
down the pike right now as far as the wide world of cloud economics?
I, I have a list of my own, but, uh, this is
not one of those areas that's right or wrong.
Yeah, it's a great question.
One of the things that I've seen emerging that I have mixed
feelings about, I used to have stronger feelings about it, but
now I think there's arguments that could be made for both sides,
is I'm starting to see serverless, uh, within the AWS ecosystem.
And when I say the word serverless, I'm referring to, uh, Lambda, I'm referring
to Mongo or, uh, DynamoDB and other serverless functions that don't require
constant loads being run, even like Aurora Serverless to a certain extent.
I'm starting to see these functions start to become
more part of the main production systems that I find.
Yeah.
This, this is gonna sound weird.
I, I wanna, I wanna flag this out because companies
have been using these things for 10 years at this point.
We are not late to, we are not late to the adoption curve, but
what we're seeing is this, say, being a significant part of the
core production application line, a significant driver of spend.
Uh, yes, it was replacing batch jobs a long time ago.
Now it's in the critical path in a way that
it wasn't previously in large enterprises.
Yeah, small companies have been doing this forever, but
that's, that's the beautiful part of being a small company.
Uh, you can put a funny animal on your website, "Oops,
we're down," and there's the raccoon looking embarrassed.
And great, it comes back up an hour later, things are generally okay.
Enterprises don't have that luxury.
Which I'm a big fan of.
Like, the small projects that I built for myself, for example, I built
them all serverless 'cause I don't know what kind of load I'm gonna get.
I wanna be able to scale up if it, whatever I'm working on goes
bananas, and I wanna be able to not pay a bunch of money if it
doesn't So it's kind of an ideal solution for a lot of things I build.
Um, I think for the first time about a month ago, I came across
a customer who probably 90% of their bill, and they were a fairly
significant spend per year, annual spend per year, but probably
80 to 90% of their services were all serverless technologies.
They didn't… Uh, I think ECS was probably the
closest thing that I would say was, like, constant.
They had, like, a minimum number of ECS jobs that they had
kind of tuned their application down to for some things.
It was interesting 'cause evaluating that customer, there were
areas that I could give advice on, you know, how they were
doing CloudTrail and CloudWatch and some CSP pieces, et cetera.
But, uh, I couldn't really tune down their CPU and memory loads,
um, because, because they were so s- heavy on serverless, they
were already using pretty much exactly what they needed to.
Now, the negative downside of that is, um, because people aren't,
haven't historically been running most of their loads in these particular
technologies, A, the one-to-one ratio of serverless is more expensive.
Like, if you took the per call cost for, you know, Lambda or the
per call cost for, um, DynamoDB, and you compared it to running your
own large cluster doing that thing, y- it's always gonna be a two
to three, sometimes four to one ratio as far as dollars to dollars
Where that's a little deceptive is that people don't run perfectly
efficient solutions when they manage themselves If you have perfect
foreknowledge and can run it exactly 100% with no overhead,
and your people's time to manage that cluster is free,
yeah, sure, maybe, but we're talking math, not reality.
Yeah, so what was really interesting was, like, where I started,
started to see it break down a little bit is the tooling isn't
there yet to really manage serverless technology at scale.
So if you've got thousands and thousands of Lambdas, tuning that,
managing it, controlling it, you kinda destroy the 80/20 rule, right?
So the nice thing about traditional ways of solving cloud problems is
the top 20% of your services are the heavy hitters, they're expensive.
Um, you… If you tune those top 20%, you're gonna get 80% of
your bill impacted, so that's- that's sort of the 80/20 rule.
Don't optimize the tiny thing that gets fired once a month.
Maybe optimize the thing that gets fired 18 times per user request.
And also question your architecture if that's what it's doing.
Also true.
But the, um, but the serverless approach tends to kind of
peanut butter your code, repos, services, et cetera, and stretch
it out, and so you no longer really have that 80/20 rule.
If you wanna go tune something, you really have to go
and revisit 1,000 Lambdas or 1,000 different services to
go, you know, change how you're doing a particular thing.
So I think there's a sprawl problem that a lot of people experienced
when containers first hit the market, that people are finally figuring
out, like, that sprawl and making things too easy and too e- easy to
manage will translate into making more services, which creates sprawl.
So I think my opinion with this trend, I really like it.
As a cloud economist, I love the idea that people are choosing a technology
that is efficient and cost-effective, but I also think, as a technologist
that's run these systems before, I think you're gonna see people start
hitting a very similar sprawl problem that they hit with containers.
All
this has happened before, and it, it doesn't… The, the rise
of vibe coding is gonna make this worse instead of better.
Like, "Oh, you have some bad code? We have a solution to that.
It's called a lot more code." And it builds on top of everything.
It doesn't get additive.
It doesn't refactor on its own.
It's let's continue to build, build, build, build, build.
The foundation could be rotten.
Yeah.
Observability, uh, in those sorts of environments becomes a lot harder, too.
I mean, one of the upsides of a monolith is that, you know, you instrument
it to write to logs in a opinionated way, and it doesn't matter what
somebody does in the future, for the most part, you're gonna be able to
go get that information, that observability, and go drill down into it.
But when you've got 2,000 Lambdas that were written over
a period of five years by, you know, 15 different people-
Coverage is uneven.
It's weird that the, some of the best environments you see are the
most… Some of the best environments you see architecturally are
some of the newest just because people have, people have gotten into
it, uh, not having to learn the early mistakes the rest of us did
Yeah, again, I, I strongly agree with that.
But I think there's… Every new trend, every new technology,
there is some downside that we just don't see yet.
I, I haven't seen, in the last 10 years in this field, I haven't seen any trend
emerge that doesn't have some sort of payoff later that you just don't know yet.
So me personally, if I was at the helm of a very large enterprise
organization, I, I would be fine being m- maybe not a late adopter, but I
wouldn't wanna be the tip of the spear on a lot of these new technologies.
No, spend your innovation tokens where it
matters, on the thing your company actually does.
Uh, Caleb, I, I wanna thank you for taking the time to speak with me.
If people wanna learn more, where's the best place for them to find you?
Yeah, you can hit me up on LinkedIn.
Um, I think it's… Let me actually look up my LinkedIn.
I should have this memorized here.
It's Caleb R Hird, H-I-R-D.
Um, but yeah, they can hit me up on LinkedIn.
Awesome.
Or find you on the trail.
Caleb, thank you for putting up with me as long as you
have and not murdering me out in the middle of the woods.
It's been, it's an absolute pleasure.
Yeah.
You too, Corey.
It's always fun.
Caleb Hird, cloud economist here at Duckbill.
I am Corey Quinn, chief cloud economist here at
Duckbill, and this is Screaming in the Cloud.
If you've enjoyed this podcast, please leave a
five-star review on your podcast platform of choice.
Whereas if you hated this podcast, please leave a five-star review on your
podcast platform of choice, along with an angry, insulting comment that then
will be gathered, printed out, and hand-delivered to us on the trail by Tom.