Between rough and UV and things like identic modern Python development
is so different than what people have expected from the past.
It's in like the typings are getting better with every version.
Python's event loop.
Is ridiculously performing now.
Welcome to fork around and find out the podcast about
building, running, and maintaining software and systems.
Hello everyone.
And welcome to fork around and find out I am your host, Justin Garrison.
And with me as always, it's Autumn Nash.
Day's special episode.
We have Ada Lund talking about how chaos monkeys and
just controlled chaos can be a good thing for you.
And, and you, you really hooked me with this
story about Rachel Ray taking down a website.
And so we'll get into that in just a second.
Ada, tell us a little about yourself.
What do you, what do you do?
What have you been doing tech?
And what are you kind of.
So, hi, I'm Ada.
I am a lead engineer at a company called DataVance.
I work on a variety of systems.
I've been, I'm kind of a smorgasbord.
I've done everything from web dev.
I got my start as anyone does in Montana, working on like PHP, WordPress
websites, and then shifted into like the usual suspects, 2016, 2017, early
Angular and JavaScript, like React development, uh, taught at code school.
Then I shifted into like, Test and then machine learning engineering
and dev ops and i've just i've kind of been all over the place I
can reasonably impersonate about six different types of engineers.
That's fun
I'm like six devs in a trench coat.
What I really like to do though
It really does feel like that some days where i'm just like yeah, I can
author a front end am I as good as like a Seasoned web engineer who is
like, you know, spend 12 years doing CSS and reacting, things like that.
No, can't do I know, know enough to know that, you know, how to avoid like
accidentally putting use effect into an infinite render loop sometimes.
I feel like that's all of being an engineer.
You're just like faking it.
So you make it and just pray that like, that's my entire backstory.
That's my super in depth in one area.
And that's the only area you work in.
Like.
You're just a Jack of all trades trying to
pretty much every other field.
You just have to know enough to know what questions to ask, right?
Cause you're like, I just need to know which direction
should I go in so I can discover what the answer is.
And as long as you can get down that path, you're fine.
That's really what it
is.
And it's funny because even when you talk to a lot of
those specialists in tech, that's how they got there.
It's just, they kept going down the same rabbit hole.
It's also how, you know, half of us have ADHD because we live in a rabbit hole.
Why, I, yes, why, why, yes, I was diagnosed with autism at the age of five.
Me too, me too.
And it shows in this very room that I'm in.
Um, this is not a special
subject at all.
I think it just means you have great hobbies,
okay?
Like I'm counting like 12 keyboards behind you.
Oh, it gets so much worse if I turn.
Like, okay, okay, hold on, we're doing this, we're doing this.
Okay, so, yeah, those photos are not AI, those photos are just This room for,
for anyone that is listening to this podcast, I'm sorry, but also this is
like a whole studio in, in what looks like a large lock walk in closet.
It's, it's the third bedroom of the house,
and this is 15 years of gear addiction.
Everybody knows it.
It's called gas gear acquisition syndrome.
I have a different kind of gas.
This is, uh, It's it's a studio.
It's so it's funny because a lot of how I got into tech actually was
through this kind of thing Because these same sort of skills used in
like music engineering audio engineering music production these sorts of
things Are a lot of the same general skills general problems something
that you'll use in tech and that you'll Find very useful in tech, you
know, it's this mixture of very scientific, very technical, very detail
oriented work and a degree of taste or artistic like choice to some degree.
And I think even, even if you're writing the like most
boilerplate back end service, there's a degree of that, right?
This, I think this is part of the reason why I fell in love with.
Like software engineering and building systems and like coding particularly
building like internal tooling and stuff like that so much was because it's this
blend of very precise technical in depth knowledge Like you can really get into
the nappy side of it and all that sort of thing and then but also there's just
this Expressivity there's this creativity with it that makes it so much fun.
You can explore and build just wild and wacky things And we'll
get to that but it's one of those things where also just like I
originally graduated in 2013 with an English degree, and I thought
it was terrible at math, at like science, at all these things.
My dad used to have a saying, which is, you know, I love you some, but if
you tried, tried to land a rocket on the moon, you'd land it in Detroit.
And he wasn't wrong.
I, I just did not get along with math well for the longest time.
And then on kind of actually because of my partner that I've
been with now for 11 years seeing her level of learning,
I was just like, well, I have that level of learning.
And you know what?
I, I always have like admired folks like Rupert Neve and I forget the name
of the CEO of Atom Audio and all these guys that have these like very like.
Technical but not necessarily traditional backgrounds.
Although I think the CEO of Adam Audio actually
was like a full physicist for a long time.
And I want to be able to go into that.
Like I want to learn how to write like DSP and how to write like
software plugins, VST plugins for like audio and stuff like that.
And so I went back to school for Sciences and I ended up in finishing
a computer science math dual post back and my work in audio actually
funded a majority of like the textbooks and like a good chunk of tuition
and stuff like that and as well as just like working in tech the entire
time in Montana because I don't know if it's still true, but like
back from like 2014 to like 2020 in Montana, there was just like every
place needed some sort of just like WordPress, PHP, journal dev person.
And there's, it was a small, you know,
growing startup scene up there in Montana.
It's funny how this all like comes in place and, and I still love to
do it because it's like, I'll get done with work for the day and I'll
come over here and it will, it will be very much like similar skills.
So I'm still kind of working that muscle, but it's a very different outlet,
the different keyboard.
You just, it's just from QWERTY to 88 keys.
QWERTY to, to, to knobs and whatever.
Well, as I decided to come up with at a given time
before we before we dive too deep into the
technical side, do you, are you creating music?
Do you mix it?
Do you produce?
I mean, I know usually it's like all of the
above, like which, what do you usually focus on?
Do you like?
I like doing mixing and mastering for other folks.
For the most part, I've always felt more comfortable kind
of as a studio rat kind of, you know, behind the scenes and
that I do commercial audio for Clients and stuff like that.
I have a decent client base.
I built up like,
so we could hire you for a fork road and find out theme.
Yes.
Yes, actually you could.
It's very fun to me to explore and to create these sounds,
I guess, kind of diving in then to the major topic here,
which is just like those rabbit holes and things like that.
If I were to have a like background specialty, it would be building
tooling and in particular, like internal tooling, test tooling.
And that leads to the, I guess, today's story
with like Rachel Ray and things like that.
So back in 2020 when I got down here, got hired
at the Zebra on the test team, uh, as an SDET.
Was tasked with doing, with implementing performance testing.
And the reason was because they had been crawled by rachel ray rachel ray
has a web crawler And it had almost taken the site down like everything
was red lining They had to like impromptu like the dev office team was
just like oh god We gotta spin up replicas, you know We had a kubernetes
cluster couldn't but we had to change the policy on the fly her company
runs Web crawling and my guess is it's to Find links and useful things that
they can put in or maybe just to kind of do some web indexing stuff I'm,
not exactly certain why but it's because of how the caching and everything
was Configured and because of how things were set up at the time with
the auto scaling, we ran over Kubernetes cluster at this point in time.
It wasn't like the Kubernetes side.
The DevOps side was not well understood.
It was just as happens with many sites, applications, APIs.
It had grown up over time, kind of organically.
It was a Django monolith, and it's just It was architected to handle what
it had handled thus far They had never encountered an instance of being web
crawled by someone like rachel ray A lot of times what you find with software
is the current implementation is the best effort implementation Given the
the the the problems that have been encountered thus far in the scope and the
scale that has been encountered thus far It's only when you hit a oh Situation
like that and you find that things like performance testing are actually Super
useful because it's very difficult to replicate situations like that, right?
Or let's say, for example, you do a Super Bowl ad takes the site offline, right?
Or something like that.
You just can't predict.
It's very hard to and it's even harder to like emulate that level of usage.
So that's where kind of the forking around comes in.
And in many ways you don't want to emulate that, right?
Like that's just like pre optimization of like,
Oh, we assume we're going to get a million hits.
It's like, well, like let's just build to what we're actually getting.
Right.
And so like staying within bounds is a smart idea just for time spent.
Right.
And I feel like a lot of people have spent
way too much time on their auto scaling.
When they don't need to, like you could have saved all that engineering
effort on auto scaling on just like over provisioning once and that would
have been okay for you in a lot of cases in the case of this web crawler
comes and it must be hitting the site a lot because I mean you obviously
you have users on the site in general that open links and go to the pages.
But what is the web crawler doing that makes it all of a sudden?
Oh, one instance of a crawler makes us have to scale up the website.
Exactly.
And it's one of those things where in this case, it was the fact that
we were shipping back a lot of HTML back and forth with the cash.
And that was just absolutely just redlining everything.
And also just the way it was crawling about the pages and
all the way we had the links put out and all those things.
It was just absolutely redlining it.
And so in that case, spending more money on the
infrastructure wasn't going to solve the problem.
You had to re architect the actual.
Functionality of the application.
Exactly.
We had to find a way to rearchitect things from the ground up and just be like,
okay, we need to actually be much more efficient with how we're serving this.
If we're going to use the cash to cash HTML, it can't be shopping
back and forth, you know, entire documents and stuff like that.
It needs to be very explicit, like snippets and things of that sort.
Otherwise network eats everything.
This is something that I've always found true, which is network eats everything.
Even still today, we have much better networks
than we did a decade ago, two decades ago.
Network still eats everything.
And yeah, you don't want to pre optimize, you don't want to solve for
problems that are entirely theoretical, and over budgeting is definitely
a good way to do that, like, during my data structure and algorithms
class, uh, we had a challenge to write a breadth surf First graph
problem and being the overconfident chaos gremlin that I was, I was
just like, I'm going to do it for C and I kept running into segfaults
memory errors was trying to be smart about how I was reallocating memory.
So finally just lost my patience.
I allocated what ended up being like two gigabytes of memory and it
just read the entire file and the professor didn't realize that I'd
done that and it handled their, it handled the largest edge case, but
it was very much brute force like, okay, this is going to work like
if you feed me a bigger than two gigabyte text file, then we can talk.
But at the same time, it's like when you start hitting those
limitations, when you write things to solve certain problems, always
make sure to leave yourself escape hatches and means of reconfiguring,
rewriting, reimplementing things easily such that you can go back and.
Address potential scale issues and at the same time like performance testing
is one of the things where people are just like That's like a tomorrow problem.
We don't need to know that up front I always say that if you have
the right tooling like if performance testing frameworks were more
friendly you would write all your unit integration more so your
integration tests as Performance tests because then you could execute
them at scale and you could get that synthetic insight You could get
the checking boxes side of tests, which is does this break or not?
But also you can get the performance metrics You can get particularly
with things like open telemetry and stuff like that You can get
much more detailed insight to how your application is functioning.
It's memory usage all these other things that let you Identify issues
much more accurately beforehand So you can minimize the oh, oh no muffins
it's one of those things where This got down to the performance testing.
So I'm implementing this and the thing that I find is that a
lot of these performance testing frameworks are very rough goes.
They tend towards one of two extremes.
There are frameworks that try to be, let you effectively write integration
tests and execute them at scale, but they're terrible resource hogs.
They execute pretty slowly and they're just kind of a bear to set up.
Or they're much kind of lighter, faster tools, but they
generally only let you hit a single, like, end point.
They're pretty limited in what they can do.
This is what put me on this path of writing
my own performance testing framework.
It's one of those things where, as I've continued down this over the
past five or so years, what I've increasingly realized is we really do
need a new class of test tooling, which frameworks in that they simulate.
User interaction at varying degrees of scale and to me this ideal
framework is something that lets you Write things as very small compact
Unit or integration tests basically you compose them as workflows And
then you execute them at the scale of a performance test and that allows
you to scale up scale down So if you really only need just kind of basic
integration testing stuff, you can kind of bring it down The amount of
concurrency, but if you need to really see where issues at scale might
lie, where those little, you know, hiding demons and goblins in the code
might lie, you can scale it up and you can actually do those things.
And you don't have to change the code.
You just have to change a config or CLI arg was this
output.
Like
a direct results of, of what was happening
where you're just like, Oh, site went down.
We fixed it.
We found it was caching problem.
Let me go build a performance framework.
Like what, what kind of led, like, there's, it seems like a gap in there.
Yeah, that's that.
And that's what I was saying was like performance testing tooling tends to
those two extremes where you have tools that are let you kind of have that
realistic usage, but they're not very friendly to use or tools that let you
just kind of simple usage, but they're very limited in what they let you do.
And we tried using one before that was which we started
with locust and locust does a lot of things, right?
It's very quick to write tests in But to get any reasonable scale out of
it, you have to run it distributed Uh, it's not very resource efficient
So it doesn't play well with a much more resource constrained environments
of something like a kubernetes cluster, especially back in 2020 And it
has a very spicy uh distributed architecture where all Of the workers must
have a copy the exact same copy of the test So you can't just like on the
fly update the test or rerun a job easily You have to basically redo the
deployment or you have to you know If you need to make a change to a test
you have to redeploy things and things of that sort That gets kind of clunky.
It also means that locust has a very nice web ui At the time I was using
it, that web UI was really how they wanted you to interact with it.
Well, developers were okay with UIs, but a
lot of times we prefer command line tools.
And trying to get things running via the
command line was just a rough, rough go.
There are other tools that we wanted to try using, like K6.
K6 probably strikes the best balance, but again, if you
want to use it inside a Kubernetes cluster, You have to use
their custom operator, and that is also a bit of a rough go.
And so for me, it was these continual
frustrations, just like, hey, this roadblock.
Oh, well, this doesn't quite work.
Well, I have to write all this other code
around it to get it to work in our environment.
I have to write all this other code to work to get it to work in environment.
Well, I can run it like this locally, but then I have to do all this
other changes and work to get it to work in a distributed environment.
And that's what pushed me to that.
So what happened was I think it was a time is like the third
time in as many weeks, we're trying to run a test and we kept
running into out of memory issues and our cluster with focused.
And so finally I go to my, my manager.
I'm just like, do you trust me?
Yeah.
It sounds like I'm going to go give me like two weeks.
I'm going to go write our own performance testing framework.
And then I'll be back.
And then I came back two weeks later and we
deployed it and it did the thing and it worked.
Wait, you accurately predicted how long it would take you to write something.
That is, I think that's a big story here.
This is
the, the way engineering is, uh, uh, people
have been like, wait, you made that engineering?
I was just like, Um, because I was at that point, I was motivated.
Have you ever had that when you're just like, I'm going to make this
engineering deadline work because I have spiked, spiked driven development.
So powerful.
Spite and ADHD hyper focused, like, you know what I mean?
Like when it's something that you're like, I have to figure
that like, it bothers you, like you try to go to bed and you're
like, that's all you can think of, like, and you're just like,
I mean, yeah, you should, you should have said
three weeks and slept once or twice, but instead.
It was, it was that and like by that point, so we wanted to get
the results into stats D and this was another thing that I found
with a lot of performance testing frameworks is if you wanted to
get your results somewhere that they didn't support, it was just.
Utter pain trying to get them to do I had basically written an entire
frameworks worth of code already around locust Getting our results
into stats d the way I had to get it done So that it didn't completely
tank the memory because locust was already being a memory hog.
I had to use grpc Well locust uses g event and g event and grpc do
not play nice together and they would just hang the program unless you
brutally monkey patch a couple of things and i'm just like Look, Linda.
I didn't ask for this.
I didn't.
This is actually when I was just getting hacked out on tech twitter too.
And I posted some random thing about it.
And one of the G event devs comes into my
mentions and is just like, what do you mean?
And I'm just like, I don't know!
You write the library, you tell me!
Why don't you like GVENT?
I don't like GVENT because it's breaking things right now, okay?
Like, you have the most fire tech twitter and now tech blue sky, like
Oh gosh, we didn't talk about my in house stuff.
It's made my life for like
three
years But that's that's why we love it and stand for it I
give full credit to like being cold water and like kat dinnings and
uh, Cory and angie and those folks were just like getting me to because
before that I was very much like I have all the spicy thoughts, but
i'm very terrified of saying them because I don't want to lose my job
because I know how the internet works and if I say something like to You
know, I if I mouth off to a particular vc bro or something like that.
I I i'm gonna ruin my entire career And I just got out of montana.
I don't want to be kicked out of tech Right, like I worked
for five years to get above, you know poverty line in
montana Which montana poverty is its own special kind of?
Like ask primogen about it.
It's it's it's bad Like, uh, it's funny because danny and i dan thompson
I we both share this background of like we both worked great cards at
gas stations And I remember the moment where I started like that push
to get into tech was it wasn't working But I was on one of my runs and
I was running down the street I stopped at the gas another gas station
same chain And talked with one of the guys there He was a friend of
mine and 20 minutes later it turned out he got robbed And I got shot.
That's when I was just like I can't do this, like, this English
major isn't paying off, like, I gotta go, I gotta go elsewhere.
I'm, like,
There's a stark difference, right?
There is, like, the people that came from middle class
families and, like, have never, you know, experienced
poverty and, like, all they know is tech and fancy stuff.
And then there's the people that have English degrees and theater
degrees and art degrees and we've all clawed our way out of
poverty and we're just, like, They let us do this every day.
like, you know what I mean?
Like every day you're like, wait, is this my life?
And then I'm just like, oh, but I have spicy thoughts.
And I'm like, don't get fired.
Don't get fired.
Like, you know, . And then I hang out with Justin too
much and then the spicy thoughts come out accidentally
as someone who was almost fired for my spicy thoughts online.
Uh, I will say, don't, don't do it if you don't have a , a backup
. I mean, it's one of those things though, right?
Where you're just like, you're so, like, you
start off so scared and then eventually like.
I like y'all's spicy thoughts.
So I'm just saying that
But it is just wild that is like I feel like I found my people of spicy
thoughts and like Different backgrounds in tech and it's like we're
obviously all like diverse and different But the similarities are just fire.
Like, you know what I mean?
Like, you're just like, ah, like when you know what it's like to be the poor kid
and we've all like got spicy takes and just, I don't know, like they're just,
you can tell, like we were all built to be like a certain type of people and
see, it's the thing where it's like, when I was growing up as a kid,
I got very like, like, I don't hide the fact that when I grew up in
my household as a kid, like economically, we were, my parents had
busted her butt and we were, we were very much on the like advantage.
Side, but on top of that, like we had house
issues putting this in a very appropriate way.
You know, my, my dad, he's, he's worked through it so hard now, but
he, he had to deal with alcoholism that made growing up really hard.
And, uh, so I'm not used, I was not used to saying also those spicy thoughts for
the longest time, because if I mouthed off to my dad, it did not turn out well.
Yeah,
my family was very religious, but like either half my family
was very religious or the other half was like very like did
drugs and craziness, so it was like you had to be perfect to
stay in the safety, you know what I mean, of the religious side.
So I feel you, like I was always like the opinionated child.
And it was not good.
I would love bringing that side into technology too, because
there are a lot of people that don't feel they belong in tech
because they're like, Oh, my, my dad also went to AA, right?
My mom was in rehab through most of my junior high and high school and beyond.
Um, it's like, this was, this is how people live.
Like this is the reality of a lot of people's lives
and they don't think they can break out of it.
They don't think that they can.
Do something to get themselves out of that situation,
which you, which can be dangerous for their lives.
And in all three of us on this, on this show right
now are like, we've all come through some stuff.
We've all broken out of things.
I remember plenty of welfare Thanksgiving's, right?
Like people bringing us the food to like, be able to have something to eat.
So
embarrassing to have to like use an EBT card and all the things.
But I think that like.
You can have your flavor of heart, right?
Like it doesn't have to, everybody's background doesn't have to
be exactly the same to know that you've been through struggles.
And I don't think like the typical, like whether you grew up in poverty
or middle class, there's still things that can happen in your life.
That makes, we're not trying
to like outpour someone here, right?
Like there isn't that like notion of
people like my friends will be like, Oh, well,
like, I know you're going through this thing.
But this thing happened to me and I'm like,
there's no competition on pain or struggle, right?
Like we're all in it together.
Like, and there are different levels of hard.
And I think what brings us together is the fact
that we got through and we tried to create it.
Like when you come up as a kid, that
doesn't, that can't like really be yourself.
You know what I mean?
And then when you get to like, you've made a life that is better than what
you grew up with and a safe place that you can be yourself and go seek out
community, that's such a win because there's so many ways that you could have
picked an unhealthy way to deal with those problems, you know what I mean?
So the fact that we've got through it, we figured out how to build
that life, that we could have created our own safe spaces and then
found our safe people, that is such a like, it's an underrated.
Win, you know what I mean?
Exactly.
And it's one of those things where there's not a day I wake
up now and I don't feel incredibly lucky to do what I do.
And I just love that you're out there being just like yourself.
Like, like every time you just like a
little bit more 80 shows and just like, yes.
And I'm like, if you come from them in their mentions, I will find you.
I'm the same way.
Like, don't come for my people.
Don't come for my people.
Like, Josie's seen this too.
Like, I will light someone on fire if you come for my people.
But like, how dope
is that?
That we've created this cool community and we have each other's back.
Like, that's so dope.
And like, I got an art degree.
Justin was going to be a mechanic, but he's also
this smart dude with like, a mathematics degree.
You had an In like an English degree.
And we all ended up in the same place.
That's rad.
So my postbacs were actually NCS and applied math.
Hey, there fellow math nerd.
I wanted to do pure math, but it would have been another
year because of how they have the courses lined up.
I mean, it's university of Montana.
I was, I was similar.
Like I wanted to do pure physics, but the
way it lined up, I had to have the math.
And so I was like, all right, fine.
I think this is, I want to graduate in four years.
That's cool.
Sometimes Justin just pops out and I'm like, I know he's
smart, but he just like says something and I'm like.
What goes on?
And then you're on like, the internet and you're like, I built this
thing with Python and it did this crazy like, hydro testing stuff.
And like, I'm just like, you know, what, like, do you sleep?
I do actually, I promise.
I promise I actually sleep, but no, it's one of the things where
it's just like the math side, I used to be terrified of math.
And then the thing you end up reeling is that math at the upper levels
is a lot more creative and open to interpretation and stuff like that.
And even if it's like applied stuff where you're writing systems
of equations or you're writing solvers to solve kind of these.
very practical applied things.
So University of Montana, case in point, their math department
is very much oriented along the side of computational biology.
It's really handy back, uh, right at 2020 because I knew
how to write my own models and everything for COVID.
I did so.
It's, it's one of those things where it
gets so much more fun as a large part of it.
It's just, it's, I will still always say this, like the,
when I step into my complex analysis class, which for those
that don't know, complex analysis is basically calculus.
With complex numbers and proofs, which sounds like three of the
worst things you could ever put together, but it's really cool.
Yeah,
you just gave me, like, anxiety.
I had so much anxiety walking into that class, right?
But then, you realize unless you do these things, because at this
point you've probably taken, like, a multivariable calculus class.
And you have had to solve things like basically what
are called path integrals, basically finding a path.
The hardest
class I've ever taken in my life.
Yeah.
Multivariate.
Like I, I almost died.
It takes like four or five pages to solve these things.
And then you realize with complex analysis, it's like a half paragraph proof.
And you're just like, I did all that math for what?
So at this class, you could show me, Hey, guess what?
There's an easier way.
You didn't leave with
that.
You ever felt like you got like the answers in like a weird way.
Like when I was younger, I was really good at math.
And then I got convinced that I couldn't do math because I would
always find like the weird creative way to solve an equation.
And they're like, that's not how we learned in class.
And it's crazy.
Like, if you look at the studies of how like girls, Don't do
well in math because of the way that like we're taught and
that we're, we're doing it wrong or just the other things I
purposely got the degree that had less calculus in it to avoid it.
Cause I was convinced I was bad at math.
And then I ended up learning math backwards through
like programming and like different like computer, like.
Compu, like, things because the, like, you know, they
share the same, like, kind of functions and, like, wording.
And I was like, this is just a bunch of set of instructions the whole time.
It's basically, and that's the thing for me, is so much of
math in schools, it is, The problem is a teaching problem.
It is how these subject matters are taught and the fact that we do
not allow for students to engage in alternative methods of solutions.
Because when you're an upper level math, that is encouraged.
That's as long as you can prove it.
As long as you can write out your logic and you can QED that thing.
Cool.
But if the rote memorization that is expected until you get to the point, the
fact that we purposely make learning the mathematics and learning computation.
Suffering up until you get to that level is just the most backwards thing to me.
It's frustrating It's so deeply frustrating because if I had a dollar
for every person i've met that is actually fantastic at mathematical and
computational thinking but has been just Absolutely clobbered into thinking
that they're terrible at it because they have had teacher teacher after teacher.
How many of
them were
spicy?
Neurospicy or exactly, exactly.
And it's so wild because like Einstein, like Sebastian Bach, like
all these, like every big person in history that had some like
monumental like change or figured out like something that we are
going to learn in textbooks, we're told the same thing and we're
very neurospicy, but we only teach for like a very small subset.
Of like the population
for me.
That's the biggest thing about the work that we do.
Like I said, that creative aspect is that it encourages
this exploration and this alternative problem solving.
And it's just, it's so incredible.
Like.
When I worked at the code school and if that code school had not unfortunately
failed, I would still be working there because I loved that job so much.
Seeing adults find that like childlike wonder and love of
learning and desire to explore again after they've been told
for their entire lives that there is one way to do things.
This is the way we do things.
If you don't, you are punished.
It is the best feeling in the world.
I still Live for that.
That's a lot of why I like to build things like internal tools and
stuff like that that bring joy to programmers because it lets them
explore and be creative or like libraries that let them solve cool
things.
Just allowing someone to be creative again into into
be a beginner again is a huge win in so much of this.
But you just mentioned something that I did want to get back to.
Your performance testing framework that you built in two weeks, because
we kind of stopped the story there and went on a detour here, but then
you, you now built this thing and it sounds like it was successful.
Is this something that's open source?
Is that something that other people, it is now there's a version
of it that is open source.
Yeah.
So I basically, there was an original version I
wrote that was a. Decent implementation on its ideas.
The newest version that I'm working on is called hyperscale.
It's under the hyperlite, hyper lite organization on github.
It is, I am working on it.
The CLI is working.
It's available as a Python package.
You can run it.
I got a bunch of stuff working in the past couple weeks.
It is very much interactive development.
It has a full CLI.
It has basically, it's all Python written and what it is is
when I originally wrote initial version, which was called hera,
which is a terrible name, it's the best thing I come up with.
'cause like it does many things.
It's, you know, I have a huger gone.
Okay, I'll just steal from that name.
I thought it was really cool.
It,
it's, it's math.
Nerding.
It's best nerding and like, I was just like, cute are gone.
Yeah.
I just, the thing with it though was like.
Every time you try and implement something like this, it
gets part of the way to the actual vision that you have.
Right.
And you again, it gets down to this idea of like best implementation
at the time, given what you know, given the limitations that you have.
So it did well, but there was a lot of baggage with it.
It was really clunky.
It was not going to work as well distributed.
And there were other like architectural decisions that I made with
respect to like pythons multiprocessing that were not That made it
feel really janky to use in my view and this newer version hyperlite
it uses a very different Architecture and very different like
implementation and it has a much more simplified api and it's just
it's so much cleaner and nicer to use The UI is very informative.
It's entirely custom written terminal.
All those terminal graphics are done via a component
library that is in the hyperscale library itself that
I have written, which FYI, your, your website's down
hyperscale.
dev
hyperscale.
dev.
Uh, that is not a website that's up.
Actually.
It shouldn't be.
Okay.
I'm on the guide.
That's where it
was
linked.
Oh, you're on the GitHub.
Hmm, interesting.
Oh, it's, I'm actually working on updating that right now.
I need to update the screenshots and things like that.
But, if you do like a pip install or my preferred tool of
choice, uh, uv, which by the way, shout out Charlie Marsh.
Goodness gracious, that, that man single handedly
changing the entire face of Python development.
I didn't know you could use uv over pip, that's awesome.
I love UV so much.
I have never felt such love for a package manager.
Dwayne, you were so passionate about that.
It made my heart happy.
You were like, look.
I love how fast it is.
I love how easy it is to set up virtual environments.
I love how good the resolver is.
I love how clean and concise the messages are.
I love the fact that it has built in log files that are actually nice.
This isn't to say, again, past efforts like Poetry and stuff like
that, best implementations given what we knew at the time, but UV.
Just nails it.
This is the second episode of the episode with Sam.
Also, we were talking about UV for a little while and just
like how, how much better it is than what came before it.
UV is so good.
Like Charlie Marsh is single handedly changing
the entire face of Python development.
Y'all y'all got to understand like between rough and UV
and things like identic modern Python development is so
different than what people have expected from the past.
It's in like the type pins are getting better with every version.
Python's event loop.
Is Ridiculously performing now.
It's every version since like 3.
10. It's gotten so much faster so much better and it's just it's good You know
and it's like with the uh, the sub interpreters that are finally coming in 3.
14 Well, they've been there since 3.
13 But they're going to be publicly accessible as an executor in 3.
14 And you better bet this framework's going to have that that we're going
to switch over to that the python development is really just The language
is growing up and it's changing in a way that is absolutely fantastic.
And I just
because Python was out there longer than most people think it
was, but it's grown so much in the last couple of versions.
And that was always everybody's complaint that it wasn't performing enough.
And now they're putting so much effort in
everything that you just mentioned as far as performance goes, right?
Like rough and UV, they're all written in rust.
Like, which is a very interesting take on tooling for
a language that's not written in the language, right?
Always, always historically has been like, don't do that, right?
If you're writing something for the Ruby ecosystem, it should be
written in Ruby because we're going to maintain it going forward.
And this idea that's like, actually, let's just pick the more performant
language or something that is, is better to distribute with CLIs.
Like, like just let's write it and go or rust or something else.
Knows how Python works.
Maybe that's the better route for some of these things.
And that seems really interesting.
It kind of makes sense though, because just think about it.
When they make languages, I don't need to be compiled.
They're usually on top of language that is compiling, right?
So it's like just another version of us kind of learning new ways to build
things and to make them more performant, but make it be more accessible because.
Let's be real.
If we're going to, if I'm going to teach someone Python over Java and
teaching them how to account for garbage collection or C and pointers, you
know, it's just, they, they're all on top of assembly, you know what I mean?
Or like whatever, kind of like, it's all just another abstraction.
Yeah.
And that's the thing is like, I, I do actually get in a lot of ways, why.
Rust would be a good choice, particularly because like C and C and
C Python, all those things that comes with a lot of package with it.
And it's just like, if you've ever looked at the C
Python internals, it's a lot, it's a lot to take in.
And so having messed with Py03 and Rust, it's just, it's
such a nice, clean interface to build all the core guts that
really matter as something super performant, very memory safe.
Like that makes a lot of sense because you don't want to
have to worry about memory leaks with a package manager.
You don't want to have to worry about, you know,
global interpreter and all these other things.
You don't want to have to worry about all of that.
It should just work.
And then you have a nice interface.
That seems to be where people really want to put rest.
Like, I think some people look at rest and they're like, I just want
to build everything in it, but as far as big companies and really like.
When we're starting to rewrite, like, whole programs in it,
people are picking, like, very specific places that you need the
memory safety, you need the performance, and just rewriting that.
So I think Rust is going to be one of the few languages that you will
pick a part of your architecture or program and completely rewrite
that section in it, but still maintain the language in everywhere else.
But just the important guts, like a kernel or memory node
or, you know what I mean, like this, like package managers.
That will get rewritten in Rust, even while they maintain
the programs in other areas in the same language, because I
think that's just going to be the new implementation for Rust.
As a general question, one of the benefits
obviously of Rust is like, Oh, it's memory safe.
It's because you're manually managing the memory.
You don't need a garbage collection.
You don't need to do all this stuff.
But then also I think that like CLI tools.
They run for less than a second.
Most of the time, right?
Like I'm running it for a couple seconds or something.
And, and maybe the memory safety would help me with doing
some passing information through that I don't trust.
Like it's, it's arbitrary.
And how many times was there a critical vulnerability in a font?
Because fonts are basically like rendering
engines that you can pass different data to.
Like there's all these like weird places where it's like, Oh.
I don't know the inputs to this thing, but I know what the output should be.
And if we send it garbage in, we get vulnerabilities on
the outputs and maybe like rust is going to help there.
But how much of the other things that rust does
really matter for something like UV for something?
And this is something I'm generally like, I don't,
I don't know the answer to infinity listeners.
No.
Like why are there benefits?
What are the benefits are there to something like this?
I've used like go as a garbage collected language.
And it's like, I don't care because I don't
run a CLI long enough to collect garbage on it.
Right.
And it's, it's one of those things where it's, it's, it's like,
for me with, when I take a lot, look at Rust and writing things
with command line tools and like the, that, that safety and that
performance and things like that, you would think it wouldn't matter.
But it's, it's, it's one of those things where.
When you, particularly in things like Python, when you have things
like global interpreter lock kind of lurking around that can
potentially result in some unsafe situations in various environments.
And other things too, it's just the way that I've heard people
argue for Rust in this case is when it comes down to things like
that, where it's, you know, a couple of seconds of interaction,
what's really the benefit in terms of performance and speed up and
why safety when it's just, you know, a couple of command line R's.
It's not like we're ingesting, you know, 50
terabytes of data and things of that sort.
The biggest thing I've heard is that it's it's the stability in terms
of releases is what it is, which is, it's a long term investment
and developers being able to improve and expand upon the tool.
And being able to release consistently and knowing that things won't break
So that way when you get a new version of the tool, you know that it's if
there are bugs they're not going to be like memory violations and stuff
like that and it is also security in some cases because There are a lot
of cases where folks will try to abuse They'll try to use CLIs for exactly
what you're saying, you know, garbage and malicious data out, right?
I can fuzz a CLI or something and see what happens.
You
would be surprised Data Vance is a healthcare company.
Rather large.
It's a realistic security consideration.
We have to lock down a lot of stuff inside our images So that people
can't run it because people will try to see what they can get away
with injecting into for example, you know Maven or java or java or uv
or pip or other things To see if they can get some access somewhere
else that that program has license to run in so, you know And then just
also but also just on top of that it's one of the things where it's um,
The other part of it I think is when you're releasing something like a
command line tool The expectation is that it should just work, right?
You should be catching memory violations and stuff like that out in live and
tools like languages like rust and stuff like that Um really tend to help
with Preventing that sort of thing early on, it's kind of shifting it left.
I mean, as a, as a, as a consumer of a lot of these tools and
occasional writer of some of them, like just the distribution
model is so much better for compiled languages where I tried to
install a, uh, background remover that was written in Python.
And I was like, I give this a 70 percent chance
of succeeding and sure enough, it failed.
Right.
They would not even go through the install.
Cause like, I, I can't get the right dependencies in your.
Distribution, whatever.
And I'm like, yep, that's what I thought getting a binary
and only relying on the C libraries or something, you
know, is, is, is usually going to be better for me.
Are you, are you doing performance tuning, tuning
and, and stuff at DataVant or are you doing.
Like a data back end.
What's your, what's your role there?
I'm internal services, internal tooling, running and going.
Actually, it's funny that you bring up going,
so going API CLI services, things of that sort.
Um, I've also written a whole slew of course, of internal
tooling and like Python and things of that sort for them.
Data is very multifaceted.
So we have parts of the organization that run very heavily on very legacy, like.
net and stuff like that.
Then we have parts of the org that run almost exclusively in Python.
Good swath of the org services are exclusively in Python.
And then we have another part of the org that uses
a lot of Java and like the thing that my team does.
So you're an enterprise.
Yeah.
We're an enterprise.
That's all those, all those layers of your
enterprise.
This is.
And so what my team does is we bridge all that by writing
a lot of like deployment tooling and stuff like that
in Golang and Golang is interesting because Golang.
Not necessarily before Rust, but it certainly became
hugely popular with a lot of infrastructure stuff.
So if you want to work with like infrastructure libraries
like etcd or it's like the Kubernetes API and stuff like
that, probably the best way to do it is something like Go.
And particularly if you want to turn it into a service or a CLI tool that
interacts with those things, that isn't to say that you can't do those things.
This was like docker was built in go, which
was like maybe a bad call for them early on.
Right.
And it was just like, they did that really early and they
went all in and now everything in that ecosystem, if you're
interacting with the docker API, Kubernetes, whatever, it's all go.
Huge part of it.
And it's one of those things where, so for my team, the reason we went
with something like go as opposed to Python or something like that was
again, that redistributability, the build pipeline and stuff like that.
The fact that we can just build a binary, set it as cross
platform, build a different architecture and just deploy the thing.
Whereas with Python, You have to worry about selecting the right Docker image.
You have to worry about building things for the right, you know, architecture.
You have to worry about, you know, everything being installed and
having available like versions that are available and compatible if
you have a package, but that package doesn't support Mac, for example,
which was a problem when, uh, like M ones and stuff like that came
out because the underlying C code hadn't really become compatible yet.
That could be an issue.
It's just one of those things where.
I in particular so it's funny because I write a lot of python, but going
is actually my favorite language Go and rust and these other languages
just let you take care of stuff So you just don't have to worry about it
and there's an upfront cost particularly with rust with rust the upfront
cost is Hi, because you have to learn about a set of concepts like the
barrow checker and things of that sort that are just kind of intense.
They're, they're taxing their up front.
It's not like CRC where it's kind of, you know, fork around and find
out you learn about how not to mess up memory as you go as you get
the, that, uh, you know, as you encounter segfaults and stuff like
that, which, you know, As you get experienced enough in c and c and
things of that sort you start learning more and more So it's like when
i'm running c anymore I try and use like i'm very careful with it But
I try and use things like shared corners and stuff like that Even if
they are less performant than just mallocing or reallocating memory.
I know how to do that as well Uh, I also know how to do unhinged things with
like compile time template programming and c that No one should ever do.
Can you tell me about those things?
Oh
lord.
Oh yes, yes.
Uh, that's, that all came from a professor we had my last year called
Oliver Sering, who remains one of the most brilliant people I've ever met.
But good lord, the man.
The things that man knows how to do with C are truly unhinged and beautiful.
It's things like, you will compile a program, and you can use template
programming to, at compile time, whilst determining the types, compiling
the program, to use, like, constructs like recursion, to do, like, loops and
things of that sort, to allocate memory, and, like, templating constructs.
Calls and to perform operations and things of that sort.
So the way Oliver and I would use this is basically for like
tensor libraries for scientific computing and things of that sort.
We would use that to basically set up tensor computing
ahead of time because you have a static file there.
So you know the size, the params and all that
stuff as you would need to do at compile time.
And it can just take care of that in compile
time so it doesn't have to do it during run time.
Would I recommend doing that for like a C server program?
Like should you go, you know, build your Drogon app with fancy You
know, compile time vectors and stuff like that, probably not, you
know, is it maybe a useful trick to do for scientific computing
where a lot of times you have access to the day set ahead of time
and it's you kind of those parameters are known, potentially, it's a
trade off and this really gets down to the idea of like optimization.
And things of that sort again, kind of going back
to the performance framework and stuff like that.
The thing you end up learning about optimizing frameworks, particularly
with like performance frameworks, like hyperscale is optimization.
You can reduce the amount of work.
Of course, that's one way of doing it,
which is just you remove work that's done.
You can shift the work elsewhere.
You can do it at a different step at a different place
at different time and get out of the way of things.
Right.
Or you can like.
Distribute the work right you can you can kind of spread it out So you chose
to get done more quickly and then you bring it back together And one of the
things I found with python is that there's a lot of introspection features that
are available So that you can introspect from types and arguments of functions
and stuff like that So one of the things I found also with things like locusts
and stuff like that was that they were doing a lot of dns Calls and stuff
like that while they were running these tests I was just like well, it's like
for example, if it's like a static string, right if i'm just hitting hd Pbin.
org slash get.
That's a hard coded string.
Why do I keep doing these DNS lookups?
Like, why am I invalidating the cache or anything of that sort?
Like, this, this is a static address.
So one of the things Hyperscale lets you do is you can set default
arguments and then specify a, using a type hint from types that the
library provides you, that this is like a URL or something like that.
What hyperscale will do is shifting that work, it will go in before it
starts running the test, it will do the DNS lookups, or it will like
serialize and turn your data into bytes or your headers or stuff like that.
And again, it's that idea of shifting work and just like, hey, if you're going
to tell me that this is just static and you're not changing this, this isn't
dynamic, we can optimize that ahead of time and that makes it much faster.
DNS lookups
can be a big performance bottleneck if you are like, Oh, you know what?
This I missed the cash.
Let me go look it up.
Oh, it's not in my local network, especially if you're
running like your performance testing containers, right?
And you're basically starting with a clean slate every time.
And you're like, Oh, now I need to wait to go find the next answer.
If that downstream server is affecting your
performance results, that's not accurate.
Necessarily.
You're like, Oh, I must have shift something that is really slow.
Now it's like, actually, well, no, you just DNS was slow.
I think we can make sure you get that ahead of time.
Exactly.
And some of those things are your performance testing, the DNS server,
or your performance testing, your actual API and stuff like that.
And it's also just things like, are your performance
testing like your server or your performance testing?
How fast Python can serialize a string, which is.
Fast dish, but you know, still a overhead or, and this even
comes into a bigger role when you have things like HTP two and
things like that where the serialization of headers and all
these things and the encoding and the HPAC headers and all that
stuff can get very expensive if you have to repeatedly run it.
It's one of those things where I went through and I wrote, I took a Python
HTTP two library, and I basically wrote a bunch of it, kind of removed a bunch
of bottlenecks and efficiencies and things of that sort, simplified it down.
So that is.
Much faster.
I then took the HPAC library that it had that was pure Python and I
rewrote that so it kind of optimized that and turned it into its own thing.
It's still a good amount of overhead where as if I pass
in just static headers and I do all that ahead of time
so that I'm just passing in the preencoded headers.
That's saving a lot of work other things too with a lot of
these frameworks I I think even within locusts you could have
different types of locusts making different types of requests
But each locus could have like you could have an http locust.
You could have a fast http locust You could have a grpc locust.
Well, the way modern services work is they You could have
a web page that is making calls to any number of these
things and then on top of that You also have the web page.
That's a part of your performance, right?
So why not make it so that you, when you're running these workflows,
you can use multiple clients within a single given workflow.
So I can simultaneously and admittedly from the same concurrency pool, but still
at the same times test at the multiple like horizontal layers of that stack.
So I can test using like a playwright integration
that my web UI can handle a thousand concurrent users.
And then I can test at the API layer that, you know, all those back
end calls that I'm making can also handle that additional traffic.
And things of that sort, and I can test like it goes on down the line, right?
Or if I need to make some sort of HTTP three
or UDP call, I can do that as well, right?
And it lets you just be much more creative and expressive with your tests
and really get down to what am I really needing to test and what do I really
need to see that's the thing with simulation frameworks that differentiates
them so much from more traditional performance testing tooling is this
idea of really focusing on how do I get the insight that I need as opposed
to how fast can I hit something and it's what differentiates them from
traditional like Unit and integration performance testing frameworks
and that it's does this break check the box versus what kind of data?
Can I get back from this?
What's my memory usage over here in the server?
What happens when I make this extra udp call what
happens when I add a probabilistic variation, right?
And there's like a 50 chance and we'll make this request
because i've Put that in my test in the workflow, right?
So I can add a little probabilistic bit in that like chaos testing side.
Or what happens when I add a mutation and it inserts some
malformed headers or some extra junk data at the HTTP request?
Does that trash us?
Does that really hit us?
I feel like a lot of the future for a lot of these
things is shifting from here's a bunch of data.
You make sense of it to hear some understanding that if you ask the right
question, hopefully, you know, the answer to that without just like throwing
a bunch of numbers at you, because as much as people can understand and
do complicated math, people aren't good with numbers generally, right?
If you just throw a whole, like, just throw a bunch of numbers
at me, like, I don't know, let computers do the numbers, right?
And so letting the computers figure out what the
meaningful answer or meaning of that number set is.
Is probably a better way to interact with any of
these testing tools of saying like, was it faster?
Yes or no.
Okay.
What does that matter?
Like, does it matter?
Because I ran this one on a Raspberry PI and that one on a, on
a, you know, 84 core server, like, well, like it's faster, right?
So that's no, like, that's not actually what we're trying to get at.
We're trying to see like, how much does this
cost or how much do I need to scale it up?
Or how many replicas do I need?
Or more.
Practical like business applications to things
outside of the code, not just here's a bunch of data.
Let's apply the data it's okay.
Well, how does that data fit into the rest of the
organization or a Java application versus Python versus, you
know, uh, uh, Rachel Ray, uh, crawling my website, right?
Like all those things are real world concerns that aren't just
numbers.
Exactly.
And that's the thing is that at the end of the day, a lot of software
that we write comes back down to dollar value and things of that sort.
And the metrics that we get, we have a glut of metrics nowadays with testing.
It's much more challenging because testing tooling as is right
now does not do as good a job of exposing just, and I mean
just bare line exposing metrics and numbers as it should.
It's kind of the first step.
That business logic step.
Is really what's key and that gets down more at least in my view to like Having
good integrations with reporters and things of that sort so like for hyperscale
it has integration with like 31 different reporting options data dog postgres
db, you name it and that just gives you that freedom to be able to tell that
business story because if i'm just outputting a bunch of Timing results to a
json file that doesn't tell a great story but if I can publish that to a data
dog dashboard and then have that dashboard set up such that it's Like you were
saying it shows that dollar value of like This latency is costing us this many
dollars this many requests Because of how much memory is using is costing us
this much That tells a story that's something that I can go to a organization
leader or something of that sort and say hey This is the dollar value.
This is the story that it's telling like the reason why We need
to optimize this cash even though rachel ray seems like one off
instance is because the story that's telling is that Our ui is
costing us money because of how much we're shipping back and forth.
It's instability So that it's an issue once right now But it's going to become
more and more of an issue as we try and ship more and more data back and forth
as this Application goes and as we keep following this pattern of shipping
these very complex documents back and forth using this cache It's going to slow
down other things and it's going to become more than just Rachel Rick calling.
It's going to be the average user hits the webpage and it
takes seconds to load because it has to go out to the caching.
Hope that it's there.
Hope that that document's hot and ship it back.
I feel like testing and security because it's harder
to show business value until something bad happens.
is always underrated and it's so hard to get
funding and for people to take seriously.
And then something bad happens and they magically want you to save
the world and you're like, dude, I've been trying to tell you.
It just sucks because being reactive costs so much more money.
You know, being reactive makes you so much more vulnerable,
but It's like you can talk to your blue in the face
about being proactive and people don't want to be like,
and that's exactly what this sort of understanding helps you to do, right?
Because if you have some of the frameworks and data in place, like
I can't tell you how many interesting problems I spent half a day.
Investigating how much money it cost us and realize I'm like, Oh,
this is only like 100 bucks like this is like as interesting as
a problem is that is it is not worth the time fixing and it is
not like we can just spend another 100 next month and it's okay.
I will go find another problem that's actually worth spending time on
Corey Quinn heads up the duck bill group.
And this is something that he always impresses, which is
like we're aiming to optimize items that are, you know,
millions of dollars off your AWS bill, not hundreds.
It's a couple hundred bucks a month.
That's pocket change for where you know that we're
looking at hitting the big items, and it's the same thing.
It's something that you learn when you really start pursuing optimization in
general, which is this idea of performance in general, which is like what is
the amount of effort you're putting into this task versus what is the return?
And can you actually quantify that right?
Is it really worthwhile?
A lot of times, like I've seen developers just go on perfectionists.
Just ideals.
Yeah, ideals get in the way.
And admittedly, I used to tend to that side myself.
I, I remember back, I was at the code school.
I got in a debate one time with a fellow instructor over whether or
not it was more important to teach four loops versus the math methods.
Just like, well, if they don't know for loops, they don't know basics.
It's just like, okay, yeah, fine, but then we
need to get them on board the methods as quickly.
It's just like, but there's, they're less performant.
They're not, they're not as good.
And it's just like.
It's like, who cares?
Like, it's so much more readable and clean.
And this is what's going to be expected of them at the job.
And that was one of the times where it was just, it was
this awakening of just like, the business value, right?
The amount of money spent and that time it takes the developer
to write a good, clean, well maintained, correct for loop that
hopefully doesn't break or have a one off error, whatever it is.
It is just so much more than it would take like it's so
much more than the cost of the iota of compute more that
it uses to run that map statement or that filter state.
And that is like you mentioned, like the readability,
like the maintenance of software is so expensive and.
And that is why, like, that is what I want this podcast to be about.
It's all, it's not about writing code.
It's not about like the fun, like cool new tools.
Like this is like the legacy podcast, right?
This is all like, Hey, the stuff that makes money, right?
Like that is, that is the stuff that in so many developers, they get into it.
Cause they're like, I see all this money that I get and get paid
by being a developer, but they never want to boil their own.
Like job function down to a dollar amount, right?
They say like all the other people, like the sales
people, I can buy a dollar amount on the marketing people.
I can put a dollar amount on legal, all these other like functions in
the organization they can look at and say like, Oh, those all are here.
Cause they cost or make money in some way, but I'm the developer.
And I feel like this is hard thing where, yeah, like
developers need to just open their eyes and say, Oh,
how much money is the feature you're working on worth?
And how much is it worth?
In a year or two or two, you know, like, like that is where you start to
extrapolate it out and say, now maintenance matters and now performance matters.
And now all that other stuff matters more, the longer it exists.
I think we're in this realm of like the toxic 10 times engineer,
like everything is so technical and I'm more technical than you.
And look at how I wrote what could have been.
Three lines of code and one line of code and writing code as fast as
possible and not understanding the business impact or just the way
that it impacts a huge code base in general, like we are lacking giving
people context and we are lacking forcing engineers to be able to
understand business stakeholders and the fact that what you build, right?
So like, if you're just worried about flexing tech it Technically,
you're not even making sure you're building the right thing.
You're not making sure that you're building a thing that brings value.
They just want to flex technically.
Well, and we've always praised the firefighters, right?
We've always been reactive when he always said, Oh, wow, you fixed that outage.
Couldn't.
Congratulations.
Like, yeah.
But the person over that's like, here's the emails I spent
six months ago that said we're going to get into this problem.
I'm
the six months ago person who's like, bro,
like, I'm not just, but think about it.
It's a team sport.
Just because you wrote the code doesn't mean you're fixing it.
So if it's shitty code, or if it's very unreadable,
when somebody else has to come up behind you on call.
and try to figure out what is wrong here, you're
costing engineering hours in that moment, right?
Like, it's not just about how fast you can write it, but is it going to cause
other people to, like, and just like Ada was talking about earlier, like,
when you re architect things, right, you're asking for engineering hours.
So when you go and rebuild something, build it right.
Take in the context and then go back and make sure that you're
building it in a flexible way that will scale and that is
right for what this product is now and what it's going to be.
Moving forward and people don't take that in and then
they're like, where did we get all this technical debt, bro?
You built it into it.
Like, you know what I mean?
Like it's you can't always account for the future But
with where we are now and the amount of technology and
how much infrastructure grew you can at least try, right?
You can at least and we don't put any emphasis on that as a developer It's
about like how fast can you build it yada yada yada, but it's not enough
foresight like We need to be able to be the people that have context,
bring in the requirements, talk to stakeholders, figure out your business
value and how to communicate that to your leadership and stakeholders.
And we have lost that as engineers because, but look,
bro, I can write 500 lines of code in like an hour and.
Completely burn things down because I tried it and I built it fast.
That's the thing.
And it's just like, this is something that,
okay, we're getting into spicy take time.
There's, there's a two pronged side of this, which is that one, we have
Glorified the writing of code itself without and what most people don't
realize is that every line of code that you write has a cost it is debt
that you are writing and when you write a bunch of stuff quickly and
make a bunch of decisions it'll work for now you are not just solving
something you're taking out a loan against If it's just you your future
self, but if it's not just you the entire rest of the org potentially the
rest of the business and at some point you are going to have to pay that
debt and you had better really hope that you get to pay that debt before
things blow up before things get big because otherwise you end up with
things like myspace where they literally had to get put out of business
because they could not rearchitect their app to to beat the competition
I just I love you so much, like, you just be like, you just, you
understand my heart, like, just, dude, like, like, you sit in
rooms, and you're just sitting there arguing with, like, three
different engineering managers, and you're just like, bro, like,
You can give me two extra weeks to write this now, get it okay.
Like get it done.
Well, or yeah, I can sign this out in two weeks, but we're
going to be back here in two months and it's going to be bad.
It's going to be miserable.
And
think about the amount of engineering hours.
If you do a double, like if there's 10 engineers doing it, right.
Think about how much engineers are paid.
Think about the hours you're going to take to redo it.
And to like, that's so much money.
Or just like when people try to tell you that like the cloud is
like certain parts of the cloud are more expensive than on prem.
Some of them, they really are, right?
But they don't count in like the people cost.
So like if you have to hire a DBA or multiple DBAs, that also costs money.
So you have to weigh them in a actual like
You have to really think about it, right?
Like in way, all of the costs and the technical debt and everything.
So sometimes like people don't even realize how expensive
cloud is going to get because they didn't take everything into
consideration, but then they just automatically assume on premise
cheaper because they're only thinking about hardware, right?
So it's like, there's just so many technical,
like gives and takes that people forget to do and.
We have just incentivized the work fast and fail or work.
What is it?
The fail?
What is that stupid thing?
Move fast
and break things?
Yeah.
And like, which sometimes you do have to break stuff
to learn how they work, but you don't have to just be
completely obnoxious and just a bull in a tie in a shop.
This, like what you said earlier, this is a creative art.
Like it is almost an art, you know what I mean?
And it is very much problem solving and see how things work.
And you can, you can nicely break things.
You know what I mean?
Like you can break things in a way where you're learning and
you're like, okay, if I like put this print statement here and
then like, you know, common out this line of code, but you don't
have to take a whole thing down just because you're an idiot.
Like, it's the difference between like building, making a sculpture and
taking the time to make sure that it's, you know, placed well, that you
aren't invading your, your fellow neighbor's properties and stuff like that.
And just building a sculpture and putting down everybody else's.
Houses around that like that.
We have a guy out here in the neighborhood that does metal sculpting, and this
is an HOA neighborhood, but he does it in such a way where the HOA is completely
unbothered by it, even though he shows it all out there, because he is so
careful about how he constructs things, and he's so conscious about when he
does it, what time of night, how much noise he's making, all these other things.
And it is, good lord, if developers exercised that amount of general
situational awareness when they built things, we'd be in such a better state.
And when you all end up in on call or something
gets messed up, who do you want to help?
The dude who is considerate or the dude who breaks stuff
and is just a jerk and thinks they know everything?
Right?
Like, who are you going to stay on that call with?
Who are you going to?
You know what I mean?
It is a team sport.
You need each other.
I've exited calls because of individuals like that because
it's just like, okay, I'm going to come back once we've.
We need a little bit more time to let that, like, simmer a little bit.
We need some time.
There's a couple other people here on the call.
I'm going to take a break real quick
because I've got another thing to attend to.
I'm going to come back, and we're going to handle this like mature adults.
I think eventually we're going to get to the
point where these things make us better engineers.
Like, we know what makes us better engineers, and I
think eventually that will be, you know, looked on well.
But honestly, like, okay, low key, not to get too spicy.
I just hope that Dodge goes so wrong that we can get rid of, like, can I just,
I want something to point to, to be like, did you, did you see it though?
Cause like, I'm so tired of existing in toxic tech, bro engineering.
And just, you know what I'm like, we all know we've all built things and we
know that you need the emotional intelligence and you need communication skills.
And leaders can tell us this.
How many times has Kelsey and Angie and all these amazing things.
People put these in talks and told the world that this is what we need.
And it's important to building technology and still we're here.
So I'm just like, what level of like foreground and find out,
do we need to do for it to trickle down into like hiring?
That's the other thing too, is it's just like.
The other part is like the VC capitalism thing, which is where the Elon Musk
bro side really comes in, which is the, ah, we just need to, you know, move fast
and break things, we need to scale up and these, it's the emphasis on features.
I am so sick of new features and not just the AI stuff.
All the AI stuff is the worst offender.
New features being pushed down.
On upon me that I literally have an entire studio in the other bedroom
That is two thousands kit only is completely disconnected from the
internet If I want to install something I have to come over to another
computer insert a usb drive copy the the installer That you know is from
that I download from whatever vst plugin maker that I may want to use
or things like that So if I want serum, I have to go over to serum's
website download put on the usbs to come over and install it and it like
it makes this whole thing where It's just like I don't want your updates.
I don't want Roll into cloud pestering me to install or to log
back in because it can't remember that I logged in 16 minutes ago.
I have an old iMac and I gave it to my kid and I was like, never touch
the update button and then he did one day and I was like, oh gosh,
I was like, don't do it.
But okay, do you ever watch those VC dudes in the Elons and then
just be like, Tell me you've built some cool stuff for two seconds
and you've never maintained anything in a real environment.
You know what I mean?
Have we all just been waiting for them to figure it out?
Like, you know, just
in, in many, many places, especially the last 15, 20 years, you know, post.
com boom, sort of like this new wave of cloud, people have jumped.
Jobs and technology so frequently that they haven't had to maintain anything
me and my friend were talking about that dude Also, okay.
There's a different level of like tech bro confidence, right?
Like they will come in they will delete 16 lines of something do
some other stuff and with entire confidence And not even think about
the future, because for one, if they just pretend to be confident,
the ramp up period for so long was like, what, six months to a year?
So they weren't expected to know anything for six months to a year.
Then they just had to survive for like a year and
they were already job hopping to the next job.
You know what I mean?
So they like, sometimes they weren't even actually like, we
don't even know if they were good at like the first two jobs.
Like, you know what I mean?
And then they're just like in there, like advising about stuff.
And like, sometimes I walked into rooms and people will
like, as an essay, these are all people that supposedly
have been in, you know, industry for 15 to 20 years.
And then like senior essays or like L6s or
L7s would walk in and tell you something.
And I'm like, bro, have you ever built anything in production?
That's a scary feeling.
Oh, I've had that.
But you're over here telling so many people how to build
things with this very like Let's use a Cassandra database.
I used to work on Cassandra.
Oh my God.
Don't get me started on the scaling of it.
Oh, everybody thinks Noah's scale is fun
until they have to actually maintain on.
We are already over time.
Ada, thank you so much for coming on the show.
Where should people find you to hear more of your spicy
takes?
I believe it's adalund.
dev on Blue Sky.
That's where you want to find me for the most part.
My github is slash adalund.
That's where you can find me there.
Those are really the two places I'm at nowadays.
And I just want to say thank you so much for this.
Yeah, Autumn, I'm with y'all.
We living through the consequences of your actions is really what it is.
I'm so excited to finally virtually like meet you cause like, look
I have like been one of her biggest fans for like years, okay?
Like just The spicy takes, like I live for them.
And, and you are in the, uh, we have a starter
pack on Blue Sky with guests on the show.
So if anyone wants to, we're you're already in there.
We're recording this a couple weeks before it comes out, but Yeah.
So if people are on Blue Sky, uh, f afo fm, there
is a starter pack that has all of our guests.
We're, I think we're gonna break it up by year.
'cause there is a limit.
We can't scale it past 150 or whatever.
So I
just, I, I love that feature.
Talk about things that are actually like.
Thoughtful and we'll put together a starter pack.
Like blue sky has been a nice experience.
And
also like, I also hear the people that are abusing it in various ways.
And it's like, you can't, no feature ever.
We all know that you build software and you can
never intend how people are going to use it.
We'll use it however they
want to with whatever intentions they have.
And that is unfortunate, but we are trying to use it for good.
Yeah, exactly.
It's like the performance testing
framework that can be used to DDoS the site.
That is, that is exactly how locusts exhibit you.
That is consequences of your actions, right?
And it's one of the things where it's just like, but yeah, thank you for this.
I'm so glad I get to meet y'all.
I'm so glad I got to meet you.
Thank
you everyone for tuning into this episode.
Your homework for this, this week is to please rate the show
and leave a comment with whatever you developed out of spite.
We would love to see
like your best programming memes.
Cause we need, like, I think we need like a, a
meme competition of like your best nerdy meme.
I'm immediately going to go for Emperor Palpatine's let the hate flow through.
This is why y'all follow Ada.
Okay, Gem.
so much, everyone.
Thank you, Ada, for coming on the show.
We'll talk to you all next week.
Y'all take
care.
Later.
Thank you for listening to this episode of Fork Around and Find Out.
If you like this show, please consider sharing it with
a friend, a coworker, a family member, or even an enemy.
However we get the word out about this show
helps it to become sustainable for the long term.
If you want to sponsor this show, please go to fafo.
fm slash sponsor and reach out to us there about what
you're interested in sponsoring and how we can help.
We hope your systems stay available and your pagers stay quiet.
We'll see you again next time.