I wouldn't read it.
That's the first reason.
So, if I wouldn't read it, I can't expect others to read it either.
So, that was the first reason.
The second reason is I think, for me, sitting down and writing is the
hard part, and like I said earlier, I started with the diagrams, right?
I started with the diagrams with the intention that once
I create the diagram, I'll be forced to write the content.
It did happen.
I was dragging my feet trying to put words on the page.
Welcome to Screaming in the Cloud.
I'm Cory Quinn, and my guest today is not typical of most guests I have on the
show, and I don't even slightly care 'cause I am tickled think to have him.
Malith Rajapske is a DevOps engineer who self-describes
as making interesting visualizations and, okay, great.
That can go in a lot of different directions, but what first
came to my notice is a blog post that he put out about, oh, let's
call it three weeks ago or so from before the recording of this
show that purports to explain how the Managed NAT Gateway works.
Insert obvious, "it runs on money joke," here, but it had some
embedded visualizations that I think did a better job of explaining
how this thing actually works than anything AWS has ever done.
Malith, Thank you for speaking to me.
Yeah, great to be here.
This episode is sponsored in part by my day job, the Duckbill Group.
Do you have a horrifying AWS bill?
That can mean a lot of things.
Predicting what it's going to be.
Determining what it should be.
Negotiating your next long-term contract with AWS.
Or just figuring out why it increasingly resembles a
phone number, but nobody seems to quite know why that is.
To learn more, visit duckbillgroup.com.
Remember, you can't duck the Duckbill bill.
And my CEO informs me that is absolutely not our slogan.
So I have to ask, where did you come from?
This is the sort of thing that usually someone puts out after
a long period of time having worked in the space, having done a
bunch of more traditional blog posts that you know fit in markdown,
and you can just slap into some third party system as they are.
And then they wind up partnering with some sort of designer to build
something like this in my case, 'cause I don't have that skillset.
And then it kind of comes out and usually lands with a
thud, and you have basically done absolutely none of that.
Where did this idea come from?
Yeah, I guess we need to go a couple months back.
This
was around November that I'm talking about,
and my story is quite common these days.
I was made redundant around November and had all this time, right?
And for the first two months it was okay.
I really enjoyed being, you know, like very slothful doing nothing, but
then you had Jerry come out, come in, and a lot of these people from
LinkedIn, you know, coming out of the woodworks telling you how they're
gonna be very productive during the year, sort of made me feel bad.
And I was thinking, "okay, what do I do?" Right?
I have all this time on my hands, and I was thinking maybe
let's look at some sort of infra project that I could do, but it
turns out that at least for me, they weren't that interesting.
Imagine you spin up all these infra, then you had
just bring it back down once you're done with it.
And so I did not really want to do any infras or project.
Sick of backend projects.
Couldn't think of anything I really wanted
to, and then finally it was front-end.
I did a bit of front-end before, like in my own personal
time, so I definitely could do some front-end project.
I just did not know what.
Then one day I was just making circles around the corridor trying
to get in my steps, and then I remembered this blog by Josh Comer.
If you don't know who Josh Comer is, he just
makes some the most incredible front-end content.
It's very interactive.
It's sort of like going to a digital carnival.
At least that's what it feels like to me.
I thought I could do something like that, something
really interactive, but for the infra space.
I started creating this around the end of Jan, and I think I spent
around two months, and then published it on the 20th of March on Reddit.
And then you saw it.
I have to ask, it's first, there's a lot to unpack there.
I'm going to get there in a minute, but first I want to ask,
of all the topics you could have picked to write about, why
pick something as truly curse it as the Managed NAT Gateway?
There's so much out there in the AWS space.
This is one of the first deep dive treatments I've seen
on this thing, and I tore apart its economics in a post
a few years back, but this is something else entirely.
Yeah, so there's multiple reasons.
The first reason is I thought it was a bit easier than the
first topic that I was looking at, which is transit gateways.
I know if it was transit gateways, it'd be much, much longer and I couldn't
give it the treatment that it deserves with the skills that I had at the time.
Oh, do you think that's fun?
Go for Cloud WAN, and then at that point you have
an entire career building your blog post over that.
Yeah, I looked at that as well.
I think not yet, and then I also knew that I needed
to write about technology that someone might be
interested in, and here's the thing, I knew, right?
I knew that you are in the Reddit AWS subreddit,
and so this is sort of engineered in a way.
I knew that if I wrote about NAT gateways, it'll catch your eye.
I know how much you love NAT gateways.
And so, yeah, and the gambit paid off.
It worked.
It was, I tend to have cast a pretty wide net.
I caught your blog post in my intake system in three different places.
The first one I saw was my Reddit filter, which
great, that it all centralized to one place.
So I popped over to Reddit and wound up commenting, and this is amazing.
The, I also caught It made Hacker News, and it was also posted on
Bluesky, which I wound up finding slightly later 'cause I haven't
really finalized that search pattern yet, which is neither here nor
there, and it is spectacular because first off, for those who aren't
familiar, I spend a lot of time making fun of the Managed NAT Gateway
because its pricing is terrible on both the high and the low end.
On the low end, it costs four and a half cents per hour whenever you're
running it, and it likes to spin itself up as a part of other stacks.
So it costs about 35 bucks a month, and there is no free tier for this.
So if you're building something for funsies in a free account as you view it,
you're suddenly getting a $35 bill surprise just for the absolute hell of it.
And at large company scale, you're also charged four and a
half cents per gigabyte that passes through it, which does not
generally discount with volume and also can be significant.
I've talked to companies that are spending high tens of thousands of dollars
every month on this, and collectively hundreds of thousands of dollars.
When I'm doing a client project, one of the first things I look at
around NAT gateways with the bill for them as high is the ratio.
What are the instance hours compared to the traffic passing through them?
And both ways people get very upset to the point where
Ben Whaley over at Chime Financial created AlterNAT.
Which is effectively using one of these as a fallback option
when you have when you're running na instance in front of it.
And there's the aptly named FCK-NAT Project, FCK-NAT, which I think
has a bit more adoption in the industry, but no one likes these things.
They serve a valuable purpose, which is letting traffic in and
out of private subnets, but they charge for it like they're
made out of the most precious substance on earth, and it, maybe
it's like, "well, we built this based on customer requests," was
the response a distinguished engineer at AWS gave publicly years ago, to which
someone responded, not me, that, "Well, apparently you built the wrong thing
then because this is not what I want or need for this very simple use case."
So it is maligned, but it is also a phenomenal feat of technical engineering.
It leverages a bunch of internal AWS constructs like Hyperplane, and
everyone tends to use them, but not a lot of folks know how they work.
And what I really appreciated about this is, I didn't know what
your background was, but I had the sneaking suspicion, please
correct me if I'm wrong, that it wasn't based around networking.
No.
It's, the reason I ask is that there, when people start to dive deep on
things, they tend to focus on the areas historically, the smart ones anyway,
they tend to focus on the areas that they have not spent a lot of time on.
I got into networks during the Great Recession for that exact reason.
It was always the thing, it's like, "and then you put whatever a
subnet mask is supposed to be over here. Don't know what it does, but
if you get the numbers wrong, half your stuff doesn't work," and that
was basically a way of understanding what's going on under the hood.
And the reason I suspected you didn't have a networking background is
because this strikes me as someone trying to fill in a gap in their
own knowledge, because no one else cares enough to do stuff like this.
To be perfectly honest with you, it was masterfully done.
Thank you.
Yeah, like you said, I mean, I know basics of networking.
I would not, I don't know what a spanning tree is.
I've heard it.
It's the reason you plug into a network, and
for some reason, for 30 seconds, nothing works.
So you're convinced you didn't plug it in correctly, and then
there's rapid spanning tree, which is only, what, five to 15 seconds.
It causes the same thing, but They basically serve the same
purpose is to aggravate the hell out of network people.
Right.
This was also learning.
I was learning quite a bit as well while I was doing
this, but mainly she saw what I was talking about.
It's a basic of networking, but I was trying to focus more on the
cost aspects and trying to find a way to make it interesting because
cost is quite a boring subject for most people at least, I think.
Believe it or not.
I used to as well, but please continue.
Yeah.
So that's why I had those interactive elements.
So I tried to sort of grab the attention of the reader by having them do that
exercise of dragging the NAT gateway and the elastic IP, and so that would
sort of make the reader more interested to go to the more dry stuff.
At least that's how I planned it.
It's what I like about, so I, for those, fortunately for
most people listening to this, it is an audio podcast.
So I can't just assume everyone's watching a video and throw up a
screenshot of this thing, but it is absolutely worth visiting, and it
will be featured prominently in the show notes when I put this thing out.
But it's a blog post that explains the how
the Managed NAT Gateway works and what it is.
The baseline stuff that more or less you can see in a bunch
of other places, increasingly, some of it generated by AI and
giving factually incorrect information along the way for funsies.
I wouldn't be raving about it if that's where all that this was.
Additionally, on top of it, there are a number of diagrams popping
around the thing that you can, it's not necessarily obvious, but
you can pick it up and drag and drop the Managed NAT Gateway.
The first one is, "Here's a public subnet.
Here's a private subnet.
Drag and drop the NAT gateway to the place that it lives in.
Does it live in the public subnet or the private subnet?"
And if you get it right, it winds up doing the typical thing that
you would expect, and then you, if you iterate through it, like,
"Okay, now attach where the elastic IP goes," and so on and so forth.
And that is, it effectively takes you through the steps in a highly visual,
highly interactive way that frankly, I'm just stuck marveling at here.
Yeah.
Thanks.
Yeah, so I wanna talk about actually creating it.
In my opinion, creating the diagrams was not the hard part.
In fact, I started with the diagrams first 'cause actually
writing the content around it was a tough part, at least for
me, especially because this was my first blog post and trying
to make sure that everything flows, everything makes sense.
That took a lot of time.
It does, and it works super well.
What I find fascinating is that the interface is very clean.
It is well designed and visually appealing, with the obvious
exception for the crap-ass icons that AWS uses for their services.
It got bad enough that, for a brief stint, I had a designer do some in-house
icons for the more commonly used services in the Last Week in AWS icon style.
A bucket that looks like it's something Billy the Platypus
might carry around, for example, but it makes sense.
It flows, it works super cleanly, and I have to ask,
what is your front-end background that got you here?
I studied chemical engineering at university, and I was looking at, you
know, I really did not enjoy the practical side of chemical engineering.
I really liked the theory, but anyways, we had this one
unit where we had this programing class, and I hated it.
I was like, the first six weeks I hated it.
It was, I know if you know how much JavaScript, I dunno how much
JavaScript, you know, but do you remember this key word in JavaScript?
It's sort of like objects.
I do not.
I, to be clear, my programming languages extend primarily to brute
force and enthusiasm, and that can carry you surprisingly far, but
I don't have a formal computer science education in that direction.
Right, right.
But anyways, this key word was so annoying and once I, but I spent
some time, I understood it, and then I started liking programming.
But still, I continued on with my degree, and
once I graduated, I wanted to do something else.
And so I picked up, you know, I went to
YouTube, how can I break into this industry?
And the most common video that I saw was, "Hey, learn CSS, JavaScript and
HTML, and you can apply as a front-end engineer," and you know, it's possible.
Where we wind up having the interview process being a whole bunch of
algorithmic questions and the rest, and the actual job is, "Okay, can you
center a div?" which is apparently way harder than you'd think in CSS.
But then things like Tailwind apparently make it far easier
to do, if I'm understanding the grousing of front-end
engineers who let me hang out with them sometimes.
Yeah, that's sort of correct.
Yeah, and I spent, yeah, so I spent around maybe a year or so improving
my front-end skills, but I did not become a front-end engineer.
Somehow I became a DevOps engineer, but I still kept those
skills sharp in case I would need them, and it just so
happens that, you know, I needed it for this blog post.
This episode is sponsored by my own company, The Duckbill Group.
Having trouble with your AWS bill?
Perhaps it's time to renegotiate a contract with them.
Maybe you're just wondering how to predict
what's going on in the wide world of AWS.
Well, that's where The Duckbill Group comes in to help.
Remember, you can't duck the Duckbill bill, which I am reliably
informed by my business partner, is absolutely not our motto.
I'm sorry.
I'm still sort of boggling at that trajectory there, because foundationally
I don't think I've ever spoken to a DevOps person before
who came from a traditionally front-end background.
I used to be in the DevOps SRE space myself, managing teams and whatnot.
I love shooting my mouth off at the DevOps days
conferences 'cause I love the sound of my own voice.
And that was an awful lot of fun, but I don't, I've met
people who came from a systems engineering background.
I came from the Ops side of the world, which is basically CIS admins who
realized that by changing your job title, you can get a 40% raise, so do
it, but the first programming language most of these folks have encountered
have been things like Python, or in some cases, Go these days is popular.
C is where I first started learning many, many years ago because
we all have to haze different generations in different ways.
But I never heard of JavaScript being applied as to
the idea of, I guess, the traditional DevOps roles.
I mean, that's not where most of the tools are written in.
It's fascinating to think about.
Yeah.
And now that you say it like it, it is true, but then again, you
have a lot of adoption of JavaScript as well in the AWS space, so
for example, CUK is now all, is mainly JavaScript untied script.
A lot of these other libraries are in Java.
So Java, it's popular, right?
It's getting, and I think it's getting more and more popular because
for a lot of people, I think it's like the gateway programming
language, at least now, compared to many, many years ago.
It's so odd just seeing the, I guess the juxtaposition where I've
seen some beautiful interactive designs of blog posts in a bunch
of different places, but they don't tend to get into the nitty
gritty of AWS services, much less something that goes this deep.
I don't mean to start a war in Seattle, which I tend to accidentally do every
time I open my mouth, but with all of the people they hire on their DevRel teams
at AWS, with their massive documentation teams, the training and certification
group, all of 'em do excellent work, I wanna point out, why does it feel like
whenever they're describing how these things work, they're basically trying
to describe a painting by gesticulating with their hands, as opposed to,
"Well, here's a real quick animation that makes sense. In fact, now you try
it and see if that helps it set in," which by the way, it absolutely does.
You said this took you a few months to build.
They've had years.
Where is their version of this?
Yeah.
You know, good question.
I also agree with you.
I feel like the documentation is not too great.
If you look at some of the other products, I'm not talking about cloud product
providers, but in general, like, Next.js, I really like reading their document.
It's very clear.
I wish more documentation was like that.
I agree wholeheartedly.
Vercel's doing some fascinating work there, and
they sweat a lot of details that other folks don't.
I mean, to be clear, I am slightly biased.
My friend Cody, who's the voice behind Killed by Google, works there,
and I periodically make it a point to go dark in their doorstep.
Yeah.
and you know, right.
If you see that document, it's actually a delight.
You know, you actually like reading these document, but when
I look at AWS documentation, it feels like, it's like work.
I mean it is work, but it feels like more than it needs to be.
My problem with it, I think, is that so much of it is written
from the perspective of either trying to be exhaustive and act
like a library, like API library documentation and list every
call, or it just gives this topical glossing treatment of it.
But it becomes remarkably clear that whatever it is that has
led to this did not come from someone sitting down to use the
product, who was smart at many things but had not used this
product before, and tried to accomplish a number of common tasks.
That's where the gulf seems to be.
That the people writing the documentation are very often far
too close to the product that they're attempting to document.
Like, take a giant step back and assume someone is far less smart
about the specifics of this thing you've worked on than you are.
How would you explain it to someone new, and every bit of documentation
I've seen officially on the Managed NAT Gateway presupposes that you are
already deep in the AWS networking weeds, even though this is one of the
first AWS networking constructs are going to encounter 'cause you find
it staring at you on your bill and what you thought was a free account.
Yeah.
To be honest, I actually don't know what the answer.
It's hard, right, for them as well.
Who is their audience, right?
Maybe assume that you know, they assume that the audience
is very technical, and so they don't want to write too
much information that might just turn off the wheel.
I'm not too sure myself, but I think there is some
opportunities to make it at least more interesting.
I would agree.
That is, it's one of those big problem spaces where if you'd look
at this through most common lenses, this stuff is incredibly boring.
No one cares about how this networking stuff works internally until
they decide they're tired of not knowing and they want to go diving into
it, and even then it's a slog because so much of this stuff is so dry.
It's, you didn't just build something with a great
visualization approach, you take the user through it from
the simple to the complex, and you're engaging about it.
That is something that's very hard to find.
I would still like this blog post, even if it just had typical
static images on it, but the fact that it goes so far beyond
that means that it's something that is significantly neat.
In fact, when it was posted on Bluesky
and on Reddit, you saw the response to it.
Most people's response to people said, "Hey, I wrote a blog post,"
is complete and utter silence because it just feels like it's, odds
are it's gonna be part AI generated, gonna get some things wrong,
and basically people are just looking for clicks, attention or clout.
This legitimately comes from a place of wanting to educate people and it shows.
Yeah, and I think I also was inspired by Josh Comer's
website because I've done many courses, right?
But I feel like I forget about them once I'm done.
But Josh's one is so different.
I'm still going and just looking stuff up because I just enjoy it.
That's sort of what I wanted to create as well.
There's the education book part, but I also wanted
other people to sort of experience, you know, this sort
of joy that you can actually get from a good website.
You know, because the web is really interactive,
and I feel like it's not well utilized.
Yeah, it's very much the, it's become more or less a, you download a more,
a static blog post that also comes along with 75 megabytes of JavaScript
when all is said and done, loaded serially, because of course it is,
and 95% of it is basically there to act as a surveillance apparatus.
One of the fastest and best days on the internet that I can recall,
it was the day that GDPR came into force and so many companies
had not planned sufficiently far ahead, like the major news
organizations, that for a couple of days, they turned their website
into pure text only, and it was lightning fast to load anything.
It was, this is what we could have had except we somehow decided it's
impossible to advertise things to people without basically spying on them and
building a fairly complete picture of their entire lives, which, I'm sorry.
I have challenges with that entire model, but this is, if you actually have
used that platform for something in a way that is creative, fun, and at
best of all, which I don't know if it's clear or not from the conversation,
from everything I can tell you're not trying to sell me anything.
No, I'm not.
I wish I could sell you something, but I can't.
I don't have anything to sell.
Yeah.
Another question I have for you is that you mentioned at the
end that this was your first blog post, and that really was
sort of the second big shock of all of this, because normally
like how, as you mentioned, this took months for you to write.
Why didn't you start with something, I guess, easier
from a production perspective to put out there?
Namely, you know, text on a screen just hypothetically there.
Yeah, because I feel like I wouldn't read it.
That's the first reason.
If I wouldn't read it, I can't expect others to read it either.
So that was the first reason.
The second reason is I think for me, sitting down and writing is the hard part.
And like I said earlier, I started with the diagrams, right?
I started with the diagrams with the intention that once I
create the diagrams, I'll be forced to write the content.
It did happen.
I was dragging my feet trying to put words on the page.
Yeah, I think that you have a gift here.
You're opinionated in how you write.
The things I take issue with on some level on
this are, I guess, not what you might expect.
Your big warning at the top where you say, "The intended audience. This blog
post assumes you already have a strong grasp of networking in the cloud,"
and for 98% of blog posts on stuff like this, I would wholeheartedly agree
you have made it so approachable that I don't believe that that holds true.
Worst case, I feel like you could have put maybe a hyperlink
in there when you talk about what a route table is.
For example, too, if you don't know what it
is, here's the link to the Wikipedia page.
Go read and come back.
The, for folks to figure this stuff out, like honestly,
I feel like that warning at the top might scare away
people who otherwise are perfectly situated to pick it up.
The other that I, the other thing that I'm wondering on is
great, all you link at the bottom is your now X account,
I suppose, your Bluesky account, and then LinkedIn.
Great.
Where's the other blog posts?
I wanna read more of what it is that you're working
on, and at the moment, this is the only one.
Yeah.
So, I am working on another blog post, but I was initially
thinking maybe I might start writing about VPC endpoints.
It feels like a natural next step, but I thought maybe I wanna
do something else instead, rather than just, how do I say this?
Rather than just
going sequentially.
I'm using CloudFlare, and I really love the product.
I feel like I really want to write something about it.
Especially, especially comparing CloudFlare workers
with something like CloudFront, Lambda@Edge.
I feel like there's some content that I can write over there.
So that's gonna be my next blogpost and the one I'm working on.
If you enjoyed this sort of content, there are other content creators
like this in the space, but not AWS, but I think you'll still enjoy it.
One of them is Sam Rose.
Sam Rose creates some incredibly interactive computer science related topics.
And then there's Benjamin from Plan Skill.
I don't know if you've seen his blog post on latency and .io, yeah?
Have you seen it?
I was at their office when, before they put that out, I was talking to
a few folks over there about how to contextualize some of this stuff.
Yeah, I know Planet Scale very well.
Huge fan of what they do.
Yeah, I love that blog post.
That was amazing.
I could tell as someone who's done a bit of this visualization, I could
tell that must have taken such a long time for him to put that together.
And then at least in the Infra and back-end sphere, I think that's it.
And then there's Josh Comer.
If you wanna have a look at his stuff, I would highly recommend it.
I was inspired by his blog post.
I wanna put links to all of this stuff in the show notes
as well, 'cause I think If it inspires you to do things
like this, I want more people to read these things.
Definitely.
I really wanna thank you for taking the time to speak with me.
If people wanna learn more, where should they go to keep up to date
and figure out when your next blog post drops where they can read it.
Yeah, so I am quite, well, I'm relatively active on Bluesky and on Reddit.
I'm not that active on Twitter, though.
No, I used to be extraordinarily active and turns out that populations migrate.
Platforms change, and people decide where they wanna invest their attention.
The fact that so many people found this worthy of investing its
attention and tells me that you've written something phenomenal.
Thank you.
And we'll of course, put links to all of those things in the show notes.
Thank you so much for taking the time to speak with me.
I appreciate it.
Likewise.
Malith Rajapakse, DevOps engineer who just
excels at telling fantastic stories through art.
I'm cloud economist Corey Quinn, and this is Screaming in the Cloud.
If you've enjoyed this podcast, please leave a five
star review on your podcast platform of choice.
Whereas if you've hated this podcast, please, leave a five
star review on your podcast platform of choice, along with an
angry, insulting comment that still requires 75 megabytes of
JavaScript to load, so I can fully appreciate the lack of content.