Because I like being able to tell stories out of what people spend, just because
it is—as we say around here—it’s the sum of all of your engineering decisions.
Because everything you do, there’s a price tag on it.
And knowing how you got there, and that you can optimize the
architecture by looking through the bill is super fascinating to me.
Welcome to Screaming in the Cloud.
I’m Corey Quinn.
I’m joined this week by my colleague, Amy Arumbulo Negrette,
who’s a cloud economist here at The Duckbill Group.
Amy, thank you for taking the time to, basically, deal with my
slings and arrows instead of the ones that clients throw your way.
It’s perfectly fine.
It’s not as if we are… not the kindest people within the Slack channels anyway.
So, I am totally good.
[laugh]
. This episode is sponsored in part by my day job, the Duckbill Group.
Do you have a horrifying AWS bill?
That can mean a lot of things.
Predicting what it's going to be.
Determining what it should be.
Negotiating your next long-term contract with AWS.
Or just figuring out why it increasingly resembles a
phone number, but nobody seems to quite know why that is.
To learn more, visit duckbillgroup.com.
Remember, you can't duck the duck bill, Bill.
And my CEO informs me that is absolutely not our slogan.
[laugh] . So, you’ve been at The Duckbill Group, as of the time of
this recording, which when you’re releasing things in the future,
it’s always a question of how long will you have been here by then?
No.
We’re playing it straight here from a perspective of, as of
the time of this recording, you’ve been here six months, all
of which, of course, have been during the global pandemic.
So first, what’s that been like?
It has been very loud.
And that’s to say, I live in a house with five other people in it, so it’s
one thing for me to be a remote worker and just being at my desk, working
quietly, but also having to manage noise that you can’t really control,
it’s been an extra level of stress that I could possibly do without.
It’s fine.
[laugh]
. One of the whole problems with the pandemic, from our perspective,
has been that we’ve run this place as a full remote operation since
it was started, and people come at this from a perspective of,
“Well, this whole experience we’ve had with working remote is awful.
It’s terrible.
No one likes it.
I’m not productive.” Let’s be very clear here.
There’s been a global pandemic; this is not like most years, and there
are stressors and things that absolutely suck about this that don’t
normally impact the remote work story quite the way that they have.
I totally agree.
At least before, in one of my previous companies had an office
in Chicago, so I would be there once a week, but effectively
I was remote because that was an all meetings type of day.
And the difference between that and now is that you had very
explicit work hours; you had client hours; your work sometimes
brought you out the house, if you had to go on travel, or on-site.
This is just everything is done within the same ten feet
of basically where you work, and you eat, and you sleep,
if you have really unhealthy living habits like I do.
And while I’m trying to get better at it, I’m
also not the best at having time-based boundaries.
I’m only good with physical boundaries.
So, I
have to turn off the work computer, to turn on the fun computer, which
are physically next to each other, but I have to look in a different
direction.
And that is as close as I get.
Yeah.
It’s the good screen versus the bad screen model of, “Oh, yeah, we’re going
to stop doing work now and just move our gaze slightly to one side and look
at the fun screen and work on those things instead.” And at first, I was
trying to be militant when we started the whole pandemic thing and working
full remote of booting people of, “Hey, all right, it’s quitting time.
Go home and stop it.” The other side of that, though,
is some people are, in some ways, using work to escape.
So, we’ve modified our approach to get the work done.
If you’re working consistently more than 40 hours
to get it done, let us know; that’s a problem.
But let people work when they want to work,
how they want to work and be empathetic humans.
And that carries surprisingly far.
Now, the question, of course, becomes, does this
scale to a company that has 50,000 employees?
I don’t know.
That’s never been a problem that anyone has asked me to solve.
But it works for us at small scale.
I find that the attitude working here has been really understanding
as to, we know what our lives are like and we know what kind of
work that we actually have to get done within a certain time period.
And all of us make those.
We don’t feel the need to explain how we were able
to get that work done or what time slots happen.
You’ll see me and Jesse—one of the other cloud economists—it’s like we’ll
be hitting document at the same time at—for my time would be closer to
later in the evening, but that’s just because I spent most of the day
either taking care of kids or handling house management sort of duties.
So, having that flexibility on where my schedule goes without having to answer
that question of, “Well, why were you working at that hour?” I feel gives me
a lot more control and takes one less thing away that I have to worry about.
One approach that we’ve always taken here has been
that we treat people functionally like adults.
And that’s sort of an insulting way to frame it.
Like, “What are you saying.
That a lot of employers treat their staff like kids?” Well, basically, yes,
is the short answer, where it’s, “great, we’re going to trust you with root
in production, or a bunch of confidential customer data, but we’re also not
going to trust you to make a $50 purchase on the company credit card without
a bunch of scrutiny because we don’t trust that you’re not embezzling.”
The cross-incentives of different organizational structures are
so twisted at that point that it’s very hard to self-correct.
I mean, our approach going into this was always
never to go down that dark path, and so far, so good.
Will it bite us someday if we continue to grow to that 50,000 person company?
Undoubtedly.
But I have to believe it can be done.
I would also think the pressure of managing 50,000
people would break every single person in this company.
Just thinking about it gives me panic.
Oh, yeah.
When every person becomes effectively,
what, 500 people, that’s divisional stuff.
I don’t think that anyone wants to stick around and see a
small company go through those kinds of transformations because
functionally, it becomes such a radically different place.
For better or worse, that’s not something we
have to worry about, at least not anytime soon.
We work really well with our fairly tight teams, I think.
And it’s I think it’s one of the virtues of the type of work that we do.
You don’t need a team of 15 people looking at one document.
No.
And invariably, it seems like that tends to slow things down.
Let’s talk a little bit about something that makes you a
bit of an outlier insofar as when you are only dealing with
a small number of people everyone’s inherently an outlier.
You’re the only cloud economist at The Duckbill
Group with a background as a software engineer.
Yeah.
I did not realize that when I was joining.
So, this deal with my infrastructure background is that I’ve
only ever done enough infrastructure to support my applications.
Like, Pete, I come from a startup background initially in my career,
where you had to manage your application, you had to manage your own
routes, you had to manage your own database connections, your own storage
connections, and all of that, which, looking back on it and saying it
out loud, sounds like a really bad idea now, but this was life before
infrastructure as code and it was the quote-unquote, “Wild West” of San
Francisco startups, where you can make a product out of basically anything.
And the anything I went into was fantasy football of all things.
So, I always made sure to have enough knowledge so that if
I knew something broke, that I could blame it on networking,
and then I would be able to show the paper trail to prove it.
That was the extent of my knowledge.
And then I started getting into the serverless space, where I
started building things out of cloud services, instead of just
spinning up more EC2 because I was young and impressionable.
And that gave me a lot more understanding
of what infrastructure engineering was like.
But beyond that, I build APIs, I do business
logic; that’s really where my comfort levels are.
For the rest of us, we seem to come from a background of grumpy Unix
sysadmin types where we were running infrastructures, but as far as
the code that was tied into it, eh, that was always the stuff we would
kind of hand-wave over or we’d go diving into it as little as possible.
And that does shape how you go through your career.
For example, most companies are not going to wind up needing
someone in that role until they’ve raised at least a Series
B round, whereas in many cases, “Oh, you’re an engineer.
Great, why don’t you start the company yourself?”
From a software engineering perspective.
It’s a different philosophy in many respects, and one that I
think is a little bit on the strange side if that makes sense.
It is.
It’s extremely strange how completely dependent the two
are on each other, yet the mindset to get into either
and of that level of engineering is completely different.
My husband and my father are both on the infrastructure side,
and they’ve tried to explain networking to me my entire life,
and I just—the minute the word subnet comes up, my brain is gone.
It’s like I’m replaying Star Trek episodes in my head
because I can no longer handle this [laugh] conversation.
That’s how a lot of us feel about various code constructs at some point.
For better or worse, we’ve made our peace with it, and let’s be very direct
here for a minute, we’ve learned to talk our way around customer questions
that go too deep into the software engineering space, by and large.
What’s it like on the other side of that, where there’s an expectation that
you have a lot more in-depth infrastructure experience than perhaps you do?
Or isn’t there at that expectation?
There is, I think a lot of that is just because of the type of industry this is.
Cloud consulting is always infrastructure first because that is
what the cloud is selling; they are selling managed infrastructures.
They are giving you data center alternatives,
but they’re not giving you are full-blown apps.
And whenever they do—let’s say Lightsail—it’s an expensive thing
that you, somewhere in your mind go, “I could build that cheaper.
Why am I paying for this service?”
So, when I am on the phone with clients and they have a situation
that is obviously going to be a software solution, where their
infrastructure is growing, but it’s because their software has a specific
requirement, either for logging, or for surge, that they’re using
either Elasticsearch, or Kubernetes, or CloudWatch Metrics for, and
it’s turning into an expensive solution, it gives me an inside, “Well,
this is the kind of engineering effort that’s going to need to happen in
order for you to write all of these problems and to reduce these costs.
And these aren’t as simple as hitting an option within AWS console to
bring all of that down.” It’s always going to be seen as more of an effort,
but you also get a bit of empathy from the engineers you’re talking to
because you now are explaining to them that you understand what they built.
You understand why they built it a specific way,
and you’re just trying to give them a path out.
You mentioned the now antiquated idea of going on-site and talking to clients.
I mean, before pandemic, Mike and I would head out to a lot of our clients
for the final wrap-up meeting, or even in some cases kickoffs, because it made
sense for us to do it and get everyone in the same room and on the same page.
And over the past year, we’ve found ways to solve for these problems in ways
that I don’t necessarily know are going to go away once the pandemic is over.
Is it more effective for us to travel somewhere and sit
down in the same room with people, who in many cases
have to travel in for wherever they live themselves?
I don’t know.
There is going to be a higher bandwidth story there, of course,
and the communication is going to be marginally more effective,
but is it going to be so effective that it’s worth more or less
throwing a wrench into everyone’s schedule for that meeting?
That leaves me somewhat unconvinced.
One of the strange things is that previously, I would go on-site
to clients and fly out to where they are because as many startups
as there are within Chicago, and the [unintelligible] , and within
Illinois, I’m always being sent to New York, or Atlanta, or Denver,
for some reason because they’re far and there’re planes there.
But we always end up having to talk to some amount of people that
don’t even work in that time zone, or maybe even then in this
country, so we’re talking to resources in Asia, resources in Europe,
which meant we were flying people in to be on somebody else’s phone.
And I’m glad to not do that.
I’m glad to not have to hang out in an airport, there is a burrito place in
O’Hare that I truly enjoy, but that is the one thing I miss about traveling.
I almost have my punch card done and it stopped
right before then, but I’m kind of okay with that.
I was chasing the brass ring of airline status and all the rest for a long time.
I can’t wait to finally go and hit the next tier and the rest, and
where, well, the pandemic through all of that into a jumble and I take
a step back and look at it and, you know, I don’t miss it as much.
What I do miss is that the opportunity, in my case, to get
away from everyone that I spend all of my time with now,
just for a day or two, and clear my head and recenter myself.
But there are probably ways to do that that doesn’t
keep me on the road for 140,000 miles a year.
I think, or at least I hope, that this will give us a chance
to as an industry just reevaluate how we treat travel.
A lot of clients treated it as essentially a status level that came with
your engagement where we need you on-site so we can show off we have
consultants coming in on-site so frequently to give us personal reports,
even though we are all in a room on a conference call with other people.
So hopefully, even if they’re not forcing that every other week—sometimes
weekly, depending on what your engagement is—type of cadence on travel,
then maybe it’ll just increase the quality of life for some of us.
It would be nice.
It would be super nice.
I honestly don’t see them forcing that anytime soon, but once everyone
gets vaccinated, and there’s a successful pediatric vaccine that comes
out, it’s like, I don’t see them, just letting us stay at home and continue
doing our job the way we have been for the past year, going on two years.
So, dialing back into the mists of the distant past, it’s always
a question of where do cloud economists—or clouds economist,
depending upon how we choose to mis-pluralize things—come from.
And everyone here is a different story and there’s not
a whole lot of common points between those stories.
You, for example, spent some time doing work with NASA.
What was that about?
There’s a lot of misconceptions about working
for NASA like you need to be a doctor.
[laugh] . And trust me, you don’t.
I knew a lot of people who work there that they basically got their
degree, and then they just did code work forever and they are lifers there.
And it’s such an interesting place to be because, on one hand, you have
that mission of space and exploration and trying to do better by the
world, but also, it’s still a federal agency and there’s still a lot of
problems with federal agencies in that how you get paid is essentially
at the beginning of the year, that’s when all the budgets are done.
So, you can’t do the startup thing where you go, I’m going to try a bunch
of things, and if one doesn’t work, I’m going to pivot to something else
because you’re essentially answering taxpayers and they don’t let you do that.
No one wants their taxes going to someone who
tried a thing and then found out they messed up.
Which is unfortunate, but also a really hard reality of the way these work.
Here at the Duckbill Group, one of the things we do with,
you know, my day job, is we help negotiate AWS contracts.
We just recently crossed five billion dollars of contract value negotiated.
It solves for fun problems such as how do you know that your
contract that you have with AWS is the best deal you can get?
How do you know you're not leaving money on the table?
How do you know that you're not doing what I do on this podcast
and on Twitter constantly and sticking your foot in your mouth?
To learn more, come chat at duckbillgroup.com.
Optionally, I will also do podcast voice when we talk about it.
Again, that's duckbillgroup.com.
I must confess that I’m somewhat disappointed that you opened with,
“You don’t really need to be a rocket scientist to work at NASA,”
just because, honestly, I was liking the mystique of, “Oh, yeah.
You need to be a rocket scientist to understand AWS billing constructs.”
But I suppose if I’m being honest, that might be a slight overreach.
I knew a lot of people there who had multiple PhDs, and they
could barely keep their computer on, so really, I’m finding that
I respect very smart people, but it also does not imply your world
intelligence anywhere else outside of that very specific field.
One of the really weird things about having worked there—I worked at NASA
Ames as part of their IT department—one of the things we did as outreach
was, we did a booth over at SiliCon Valley Comic Con once, and it was great.
We had a vintage display of old electronics, like CRT monitors, and full
keyboards, and all of this nonsense, and kids would go, “These are so old.
Why would anyone use this?
It’s so boring.” [laugh]
. And the entire IT department showed up to volunteer.
We’re like, “No, you don’t understand why it’s interesting.
It’s great.” It was so, so hard to watch young people just [laugh]
not be interested in what the past of digital devices were.
Very sad.
And on the other hand, we did get a lot of interest, but it was also having
to have that conversation in real life can be a little disheartening.
It really is.
But it was fun because it was one of the few things
that you don’t really get to see NASA at a comic book
convention, so that was actually a really cool thing to do.
Also, we got free tickets, so that was great.
So, what was your background before you got
to the point of, “You know what I want to do?
Work for a consultancy, whose entire mascot is a platypus,
and from there, go ahead and fix AWS bills,” which sounds
like, to folks who aren’t steeped in it, the worst thing ever?
What series of, I guess, decisions led you here?
I know you don’t remember this, but we actually met at Serverlessconf, and
you opened your talk with, “I am a cloud economist, a title I completely made
up.” Your talk was right before mine, so that’s why you didn’t remember I was
there because I was actually on the backstage getting prepared for my talk.
That’s right.
I would have been breathing into a paper bag right
before or right after my talk, trying not to pass out.
People say, “Oh, you won’t be nervous once
you give enough talks.” I’m still waiting.
It never happens.
I did finally stop having blackouts, so that’s an improvement.
It gets better, but it never goes away.
And when you told me that, and I saw the listing,
I’m like, “I don’t know what this job is.
There’s an easy way to find out what the job is, and that’s to apply.”
And that is when I started going through the process of
applying, and then you hired me some months afterwards.
And the thing that I found out about looking through AWS
billing is that I found out I have a very specific skill set,
in trying to find a discount while looking through receipts.
This is a thing I thought only applied to my personal life because
I don’t really like paying retail prices for anything, so I’m
always looking for a way to squeeze out another 20, 30% out of
something because 10% is really just taking taxes off of stuff.
And the fact that I was able to apply that very specific skill set
to an actual technical job is so much fun for me because I like being
able to tell stories out of what people spend, just because it is—as
we say around here—it’s the sum of all of your engineering decisions.
Because everything you do, there’s a price tag on it.
And knowing how you got there, and that you can optimize the
architecture by looking through the bill is super fascinating to me.
So, now that it’s been six months, is the job what you expected it to be?
Is it something radically different?
Is it something else?
The tone of our engagements have actually changed within the
past six months, partially because of the way AWS has made
some organizational changes internally as far as we can tell.
But really, it’s also what types of companies are
finding out that they need a service like this.
Before, when I was interviewing, and when I started, I was
talking to Pete and Jesse about the types of engagements you do.
It was for larger companies; they’re looking for some amount of
savings, and then we run some tooling and then we get back to them.
And now it’s turning into… some are relatively small companies, companies that
wouldn’t get use out of an EDP, for example, because they’re spending so low.
But also other companies, they don’t even really want this saving specifically,
they want validation on what the process is like, they want validation
on their unit economics and what the cost allocation strategies are.
So, it’s fascinating what people actually want now
that they understand that that option is out there.
One thing that you mentioned a minute ago, was the idea of going and
giving talks—in the before times, at Serverlessconf and things like
that—how do you find that that has changed for you over the past year?
And how are you viewing a slow, but effectively guaranteed
in the long enough timeframe, return to normalcy?
So, when everything went virtual, it was a
really hard transition for a lot of communities.
I’m part of AWS Chicago, I’m an organizer at a meetup group called
Write/Speak/Code where our whole deal was to give women and people
of underrepresented genders the opportunity to learn how to do
open-source, how do you learn how to do technical speaking, we helped
them with their CFPs, and all of that required a physical community
in order for us to be able to give each other that type of support.
Well, we can’t do that anymore.
The last big event we did was specifically around
getting everyone set up into the organization.
So, we do one big one every year where we tell you how to do slides,
we tell you how to do everything, and then the rest of the year
smaller meetups where we do feedback and prompts and other types of
support events.
Now, that we can’t do that, we found that a lot of the people who ran
these events, they’re extroverts and they’re social to begin with, so they
got burnt out very quickly, which meant we not only had to find new ways of
supporting them but also reaching out to members and making sure everyone
could still do the types of thing we promised we’d help them to do.
The other issue is that with things going virtual, there used to be
clear lines on the types of events you could apply to, which were
events that I could reach, that my company would pay for, what have you.
But now everything’s virtual, I accidentally applied to a meetup
that I spoke at that was based in Australia, which meant I was
talking at 10 o’clock at night, just because I explained to
them the mix-up and essentially begged to get an earlier slot.
And it’s interesting because it presents both a wide array of
opportunities, but it also means that there’s now so much noise,
and so much burnout and fatigue from these one direction types
of conferences, which are long Zoom meetings, which is basically
everyone’s workday now, which is just full day’s worth of Zoom meetings.
And it’s hard to get people interested.
And really, what these events did best was give people who generally
don’t have that type of visibility—like me—who, I am a female engineer,
and I am a person of color, so my opportunities aren’t always going to
be as well as they could be, but this visibility also gave me a boost.
So, when we lost out on physical events, my organization personally
lost out on a lot of things that we could do for those people, which
is unfortunate, but it’s also what was happening, really, everywhere.
Now, we’re gearing more towards how to do less Zoom meeting type of events,
and we’re now using a tool called gather.town, which lets everyone go into
a space, you can walk around, you can drop in and out of conversations like
you would in a hallway, and it has this cute little eight-bit kind of avatar
feel to it so it looks kind of like a game, but it’s also—if you go around
a large group of people, it pulls up everyone’s picture and then you’re
suddenly speaking to each other over voice without having to physically
join in or wait for a breakout room or wait to be let into a different room.
So, it’s been difficult to try to find ways of managing it, but it’s
also been very interesting seeing the tools that had come out of this.
Now, once we start going back to physical events and on-site events,
personally, I have a lot of anxiety about that just because my
allergies and everything makes it hard for me to do travel in the
first place, but also, I’m not really sure how everyone’s going to
react being in the same space anymore, or how full they’re going to be.
So, to me, it does cause me a little bit of anxiety and
I’m going to wait a little until things are more settled
and are a little more stable than they are right now.
That said, I believe AWS Midwest is doing
something physical at the end of this year.
But I’m not entirely positive about that.
One of the things I want to avoid is going back to an old style of re:Invent,
where it’s only open to folks who are able to spend $2,000 on a ticket, travel
to Las Vegas, and get the time off and afford the hotel stay and the rest.
One thing I loved about the 2020 version of that was that
everyone was coming from a baseline of its full remote.
There was no VIP ticket option that got you
a better experience than anyone else had.
And I do worry, on some level, that as soon as they can,
they’re going to go running back to a story like that.
I hope not.
I hope not, too, because one of the great things about virtual
re:Invent was everyone saw the same things at the same time.
And you didn’t have to give up one panel because
you’re too busy being in line for another panel.
And I’ve never liked that type of convention activity where
you know you’re actively giving something up just because
the line for the thing you want to get to is so super long.
That said, they could have done better a bit on the website.
The website was really hard to navigate
and confusing and the schedule was weird.
If they get that kind of usability fixed and a little more reasonable
so that things are surprisingly searchable and easier to navigate, and
they have more social type of events instead of AWS is going to—it’s
going to announce another service that they’re calling a product, even
though it’s just… it’s a service enhancement, and that’s all it is.
I would like for all of that to kind of be
streamlined so it’s not just more announcements.
I can read announcements.
I don’t need to watch two hours’ worth of announcements.
I really hope that at some point, some of
the AWS service teams learn that, “Hey.
We could announce services anytime of the year,” rather than in a three-week
sprint that leaves no one able to pay attention because there’s just too much.
And it’s hard because it’s hard when you have
to hold a release because re:Invent’s coming up.
Or you have to be part of their Developer Relations Group where
you have to do all of your training and all of your docs right
beforehand, just so that it’s prepared for this launch that people
may not hear about because it gets drowned out in all the other noise.
And that’s sometimes part of the entire problem.
Yeah.
Thank you so much for taking the time to speak with me.
As always, it’s a pleasure.
If people want to learn more about what you’re up to, where can they find you?
They can hire me for engagement here.
[laugh] but also—
Good answer.
—I do technical talks.
I don’t have anything lined up right now, just because it’s
spring in my brain took a break, like everyone else did.
I’m on Twitter as @nerdypaws because that was a
handle I had since college and have not changed.
Excellent.
And we will, of course, leave a link to
that in the [show notes] , as we always do.
Thank you so much for taking the time to speak with me.
It’s always a pleasure, and it’s deeply appreciated.
It’s always a good time talking to you, Corey.
Amy Arumbulo Negrette, cloud economist here at The Duckbill Group.
I am Cloud Economist Corey Quinn, and this is Screaming in the Cloud.
If you’ve enjoyed this podcast, please leave a five-star review
on your podcast platform of choice, whereas if you’ve hated this
podcast, please leave a five-star review on your podcast platform
of choice, along with a comment telling me why you do in fact need
to be a rocket surgeon in order to properly work on AWS bills.