Like I said, the product life cycle, when you're building
something new, you want to go as fast as possible.
When you're launching it, you want it to be as reliable as possible.
Once you're launched, once you're reliable,
then you can start focusing on costs.
Welcome to Screaming in the Cloud.
I'm Corey Quinn.
If there's one thing that I love, it is certainly not AWS billing,
but for better or worse, that's where my career has led me.
Way back in episode 51, I had Dann Berg, the cloud
ops analyst at Datadog, and now he's back for more.
Things have changed.
He's now a senior cloud ops analyst and I'm hoping my jokes have gotten better.
Dann, thanks for being bold enough to come on and find out.
Yeah, I'm excited to see, uh, if these jokes have
gotten better, that's the main reason for coming back.
Exactly.
Because it turns out that, uh, death taxes and AWS bills
are the things that are inevitable and never seem to change.
Yeah, they just keep coming.
They never stop, and they're always, uh, slightly different than you expect.
I guess just like death and taxes.
This episode is sponsored in part by my day job, the Duck Bill Group.
Do you have a horrifying AWS bill?
That can mean a lot of things.
Predicting what it's going to be, determining what it should be, negotiating
your next long term contract with AWS, or just figuring out why it increasingly
resembles a phone number, but nobody seems to quite know why that is.
To learn more, visit duckbillgroup.
com.
Remember, you can't duck the duck bill, Bill.
And my CEO informs me that is absolutely not our slogan.
So when we spoke back in, I want to say 2019 is when it aired, so probably
that ish is when we had the conversation, if not a little bit before that.
You were effectively a team of one, and as
mentioned, had the CloudOps Analyst title.
Now you are a Senior CloudOps Analyst, which I assume just means you're older.
Is the team larger as well?
What does that process look like?
How has it evolved in the last couple of years?
Yeah, it's been interesting, especially being at a single organization and
that organization being Datadog, that to be able to grow the team a little bit.
So as you said, it was just me.
Now it's a total of four people, including myself.
So three others.
And Yeah, it's been interesting just in terms of my own professional
development, being able to identify what needs to be done, how much
capacity I have, and being able to grow it over time, especially in this
fairly new space of being specifically focused on cloud cost billing.
So kind of that bridge between engineering and finance,
which itself is kind of a fairly new space still.
It is.
And my favorite part of having these conversations with folks who
have no idea what this space is, is learning, when I was starting out,
how to talk about this in a way that didn't lead down weird paths.
It's, Oh, you, you save money on Amazon bills.
Can you help me save money on socks?
It's like, no, well, yes.
Get the prime card.
It gives you 5 percent off, but, but no.
And.
Yeah, I talk about Camel, Camel, Camel and other ways of
working around the retail side, but that's not really what I do.
It's similar to back when I was doing SRE style work.
I made it a point never to talk about being someone involved in working
in tech, or suddenly you're the neighborhood printer repair person.
Similarly, you have I guess gone in a strange direction because you
weren't, to my recollection, someone who had a strong SRE background.
That's not where you came from in the traditional sense, is it?
No, not an SRE background at all.
Yeah, I mean, it's really interesting.
So talking about this space, I mean, people are calling
it a lot of different things, cloud economics, The term
FinOps, Financial Operations, is being used a lot now.
Cloud financial management is another popular one.
Oh, the swing a dead cat, you'll hit 15 different words.
And I give my advice on that, even though I hate some of the terms, is cool.
If people are going to pay you to have a title, even if you think it's
ridiculous, you can take the money or you can die on a petty naming hill.
And here we are.
Yeah, and it's interesting because the role that I
was hired for at Datadog was very much this niche.
Very specific role that I didn't realize
was a niche, very specific role at the time.
So previously I was at a company and I was building out their data center.
So I was working with vendors, buying servers, sometimes going
on site, installing, racking those, uh, dealing with RMAs.
And I was getting more involved as their cloud usage was growing and
bringing some of those Hardware capitalization costs procedures to the cloud.
And so I found myself in this kind of niche
role in my previous company and a data dog.
They basically had the exact same role that was dealing with all of the
billing stuff around the cloud, kind of from an engineering perspective
because it was on the engineering team, but working closely with finance.
And I was like, Oh, these are the skills that I have.
And it kind of fit perfectly and it wasn't until after I got to Datadog and
kind of was doing more research about the specific space that I discovered
just how wide open it was and I mean, meeting you was one of the earliest
things I did in the industry, discovering the FinOps Foundation and a few
other things is kind of like opened my eyes to this as an actual career path.
It's an expensive problem that isn't going away anytime
soon, and it is foundational and core to the entire
rest of how companies are building things these days.
My argument has been for a while that when it comes to
cloud, cost and architecture are the exact same thing.
You don't have the deep SRE architect background,
but you're also now a member of a four person team.
Does everyone on the team have the same skill set as you, or do you wind
up effectively tagging in subject matter experts from different areas?
How is the team composed?
People love to ask me this question and I strongly believe
there's no one way to do it, but what's your answer?
Yeah, I mean, the team works very much in terms of everybody
kind of taking on tasks that they need to do, but we did
hire for specific skill sets when we tried to find people.
So, the first person that we hired, we wanted them to
have more of a developer, engineer type personality.
Background writing code, stuff like that.
The third hire, we were looking for somebody that was more of kind of a
generalist, I see myself more as a generalist in the space, anything that's
going on, I can pick it up and make some progress on it and build something out.
And then the fourth person, we were lacking some of the deeper FP& A or finance.
Experience.
And so we found somebody with more of that kind of background
and less of the engineering experience, but they were eager
to kind of move from finance into more of an engineering role.
And I feel like this is the perfect role for that because
I feel like there are a lot of non engineers that want to
break into engineering and don't really know how to do it.
And if you are in finance, in FB& A, finding one of these more cloud cost
optimization specific roles is the great way to bridge that gap, I feel.
The last time we spoke, I was independent, doing this all myself.
And it turns out that taking all of the things that make me, me,
And trying to find those in other people is a relatively heavy
lift, even if you discount the things like obnoxious on Twitter.
So how do you start decomposing that?
Well, now we're a dozen people and we've found ways to do it.
But by and large, in our experience, for the way that we
interact, and I want to get to that in a second, is that
it's easier for us to teach engineers how finance works.
than it is the opposite direction.
And there are exceptions to that.
And as we scale, I can easily see a day in the
near future where that is no longer the case.
However, we also have two very specific styles of engagement.
We do our cost optimization projects, where we go into
an environment and, Oh, fix this, turn that thing off.
Do you really need eight copies of those four petabytes of data?
Oh, you didn't realize they were there.
Great.
Maybe delete it.
And we look like wizards from the future and things are great.
The other project that we do is contract
negotiation with AWS, especially at large scale.
It's never as simple as people would have you believe, because,
oh, you're doing co marketing efforts, and you're, you have a
very specific use case, and there are business partnerships on
15 different levels, and that all factors into how this works.
It's nuanced and challenging, and of course, because it's a series of
anecdata, I can't really tell too many stories in public about that.
But those are the two things that we wind up focusing on.
You are focusing on a very different problem.
You're not moving from company to company, basically re
implementing the same global problem, solving it locally for them.
You are embedded in an account for the
duration, almost four years now, by my count.
And, okay, I guess I could just do a whole bunch of cost optimization
projects on a quarterly basis in an environment like that?
Doesn't seem like it solves the problem in any meaningful way.
Yeah, well, I mean, that's such an interesting question.
Just in terms of, yeah, if you're doing consulting, you're kind of starting
from square one every time you get a new contract, a new engagement,
and being at the same company for, you know, Like you said, about four
years going on four years now, you really have a chance to dive in and
think about, okay, what does it mean to work cloud cost optimization
into just the regular business cycle of like how it works, right?
Because I mean, you have the triangle that everybody's familiar with.
Things can either be like cheaper, faster, efficient, and at different stages
in a product life cycle, you want to be focusing on these areas more or less.
And so on our team, kind of, The different things
that I'm thinking about is first is visibility, right?
Is you want to provide engineers visibility
into their cost and not just numbers, right?
Actionable visibility, where if something
needs to change, they need to do something.
They know what that is.
And a lot of times that means not just cost, but also efficiency.
So like, these are the metrics that this
particular application should be scaling against.
As this application grows, as usage grows, are we remaining as cost efficient?
Then there's also the piece, as you're saying, like discovering things within
the infrastructure that like, Hey, if we make this change, or if you turn this
off, if we do things this way, we'll save a bunch of money, let's do those.
There's things like reservations, committed use discounts
for DCP, all of those kinds of things we manage.
And then dealing closely with verifying our bill, working with finance, FP& A.
on cost modeling, forecasting, both short term, like within a
month, like what are we going to be at the end of this month?
And it's the 10th right now.
And also like, what does our next quarter look like?
What are our next two years look like?
And that kind of bleeds into the contract
negotiations, those kind of things as well.
So I mean, it's setting up the cycles of how do you prioritize this work?
What is the company focusing on at the time and what can you do when
the company is not focusing explicitly on deciding to save money?
One of the more interesting aspects of my work that I didn't expect is whenever
I wind up starting an engagement or even in the prospect stage, I love asking
the dumbest possible questions I can think of because it turns out they're not.
And the most common one that I always love to
start with is, Oh, okay, your AWS bill's too high.
Why do you care?
And that often takes people aback, but once you dig down underneath the surface
just a little bit, it becomes pretty clear that the actual goal is not that it's
too much money, because, spoiler, payroll always costs more than infrastructure.
Instead, it's, how do I think about this?
How do I rationalize what the additional costs are going to be, uh, per,
you know, Active Users or whatever metric it is you're choosing to use.
And how do you wind up forecasting that?
Because the old days of data centers where you well, we're going
to spend a boatload of money and then we'll have capacity for the
next two years, maybe down to 18 months, depending on growth, that's
easier for companies to rationalize around rather than this idea
of incremental cost on a per unit basis, but not exactly, because
it also turns out that architecture changes and Problems of scale,
AWS pricing changes from time to time, all tend to impact that.
What I think is not well understood in this space is that, yeah, if you have a
20 percent overage this month, people are going to have some serious questions,
but they're also going to have those same questions if you're 20 percent low.
Yeah, I mean, understanding why people care about the cost is definitely
the first step, because with a single company, so it's just I'm constantly
looking at the numbers, rather than Understanding exactly what motivations
a company has to contact somebody like you, like a consultant, right?
Because usually I imagine that it's going to be a bill, maybe two bills, three
bills come in, and they keep going up and up and up, and they need to go down.
And they're going to have an explicit reason why it needs to go down.
Finance is going to say like, Margins are X, Y, and Z or revenue has done this.
Our costs can't do this.
There's going to be explicit reasons because if there aren't reasons, then
they shouldn't necessarily be focusing on costs at that moment in time.
And what you want to do is kind of have, I mean, this is way more complicated
than just saying it out loud, but have a culture of like, Cloud Cost
Mindfulness, where people aren't just spinning up resources willy nilly.
But also, like, my goal is for people not to have to really think about cost
that much, other than just, like, in a way that helps them do their work.
Because, I mean, I want engineers to be
able to build stuff and build stuff fast.
That's what the cloud is all about.
But I also want to be able to do it in a way that isn't inappropriately high.
I have my thoughts on this, and I've shared them before,
and I'll dive into them again, but how do you approach that?
If Datadog makes a grievous error and hires me to write code somewhere as an
engineer, what is the, I guess, cost approach training for me as I wind up
going through my onboarding as part of an SRE team or an application team?
I mean, this feels so basic as to not even be
the right answer, but honestly, visibility.
Is the easiest and best thing that you can give people.
And so we've built out some visibility
reports that engineers get on a regular basis.
We also meet with our top, what is it?
10 or 15 spending internal engineering teams on a monthly basis to
go over those costs so that they understand what they're looking
at so that we understand the context behind it so that we can.
Understand what's on the roadmap going forward so that when things in the
costs happen, we're aware, and then we're just staying on top of things.
And if we have questions, we have an open
dialogue with engineers and things like that.
In an ideal space, it would be great to have.
Uh, cost, I guess, more fit into the product
development lifecycle in a more deeply ingrained way.
But at the same time, like I really don't want to serve as a gatekeeper.
Our goal is not to like stop any sort of engineering
process and we haven't needed to do anything like that.
Although I guess every company is going to be
different in terms of what their needs are.
But yeah, I'm, I'm totally happy kind of being a little bit more.
reactionary in terms of looking at the numbers and responding and then
proactive just in terms of the the regular communication with people.
I tend to take the perspective that engineers need to
know enough about cost to maybe fill an index card at most
because You don't want them, I guess, over fixating on it.
Left to my own devices, in my personal account, I'll see a 7 a month
bill, and oh, I'm going to spend two weeks knocking that down to
4, and of course I can do it, but is that the best use of my time?
Absolutely not.
Very often, what is a lot of money to an
engineer is absolutely not to the business.
And vice versa, when you bring in a data science team, it's, Oh yeah, we need at
least four more exabytes of data because we never learned to do a join properly.
Yeah, maybe don't do that.
Like, understanding the difference between those, those two approaches is key.
I've always been of the mindset that I would rather bias for letting
developers build an experiment and have things that catch outsized things
quickly than trying to wind up Putting a culture of fear around cost,
because I'd much rather see whether the thing they're trying to build is
possible to build, then go back and optimize it later once that's proven out.
But again, this is a nuanced thing.
Everyone seems to think I have this back
pocket answer that will apply to all companies.
And you've been doing this at Datadog for
almost four years with a team of people.
I am an outsider.
I see the global trend.
I see what works in different ways in different companies.
But the idea that I can sit down and say, Oh, well, clearly the thing
you're doing is completely wrong because that's not how I think about it.
is the hallmark of a terrible consultant.
There are, there are reasons that things are the way that they are, and
it's generally not that people are expecting to do a terrible job today.
You know, unless they work in the Facebook ethics
department, which is neither here nor there.
Yeah, I mean, like I said, the product lifecycle, when you're
building something new, you want to go as fast as possible.
When you're launching it, you want it to be as reliable as possible.
Once you're launched, once you're reliable,
then you can start focusing on costs.
It's kind of like, you know, Not the universal
rule, but kind of the flow that I tend to see.
So as you're at a company that is regularly Innovating,
creating new products, going through that cycle.
You're going to have these kind of periods, as well
as you have the products that have been around.
There's a lot of legacy code, there's a lot of stuff
going on that maybe isn't the best or some efficiency.
Work that has been deprioritized for whatever reason
that maybe it's time to start considering doing this.
So keeping track of all of that.
And like I said, if for whatever reason the business wants to focus on cloud
cost efficiency, or a team has decided that in a particular quarter, Or for
a particular reason they want to focus on that, being able to assist as much
as you can, being able to save all that work so that there's kind of like a
queue that you can go to when it is time to focus on cost efficiency stuff.
So here's a fun one for you.
As of the time of this recording, it's a couple weeks old, but if you're
anything like what we do here for some of our more sophisticated clients,
we do occasionally build out prediction models Models of economics that wind
up defining how some architectural pattern should be addressed, etc, etc.
What's always fun is the large clients who have
this significant level of spend on outlier service.
Like every once in a while, it was great that we got to
do a deep dive into the Washington Post's use of Lambda,
because normally Lambda is a rounding error on the bill.
They had a specific challenge, and they did
a whole blog post on this for the AWS blog.
I believe the Monitoring Tools blog, but don't take my that at face value.
I never remember which AWS blog is which, because
AWS doesn't speak with a single voice on anything.
But yeah, most of the time it's block, tackle,
baseline stuff that is the big driver of spend.
But a few weeks ago, they changed the pricing dimensions for S3
Intelligent Tiering, where there's no longer a monitoring charge.
for objects that are smaller than 128 kilobytes, and there's no 30 day minimum.
So the fact that those two things went away removed almost every
caveat that I can picture for using S3 intelligent tiering, which
means that for most use cases, That should now be the default.
I imagine you caught that change as well since that's
one of those wake up and take notice no matter what time
of the world it is where you are when that gets dropped.
How did that change your modeling or did that
not significantly shift how you view any of this?
No, I mean, I think part of our role within the organization is to pay
attention to stuff like that and then to just have those conversations
with the teams that I know were either exploring intelligent tiering.
We do some pricing.
Modeling for different products, S3 storage for different types.
So updating those and being like, Hey, this might be
something we want to actually use and explore now similar.
And I guess more of something that I actively worked on that
I consider in the same category is when Amazon announced
savings plans as replacing convertible reservations, right?
Cause at first they announced and being like, okay,
well, it's going to automatically like rebalance.
between different instance families across regions
too, which convertible RIs could never do it.
And it's going to be the exact same price for
a compute savings plan as a convertible RI.
And we were kind of like, what's the catch?
And we spent a few weeks like Doing a deep dive, working with our
data science team, kind of like being like, where is the catch here?
Yeah, the real catch is that you can't sell it on the secondary market if it
turns out you bought the wrong thing Which if that's your plan A, then good
luck.
Yeah, we we definitely don't use that secondary market I don't have as much
experience there Although i'm sure some people can use it to their advantage.
Almost no one does, in fact The reason that it exists, my pet theory,
is that once upon a time companies would I would try and classify
some of the reserved instance purchases as capital expenditures,
which there has since been guiDannce from regulatory authorities
not to do that, but at the time the fact that you could sell it to
a third party on the secondary market helped shore up that argument.
If you're listening to this and you're classifying
some of your RIs as CapEx, please don't do that.
Feel free to reach out to me.
I can dig out the actual regulation and send it to you.
There are two of them.
It's a nuanced topic.
If you're listening to this and have no idea
what I'm talking about, God, do I envy you.
Yeah, definitely don't do that.
There was a lot that was interesting about savings plans.
When I was read in on a month or so in
advance of them being announced, it was great.
I want to see this and this and these other things too.
And some of those things came to pass.
It was extended to work with Lambda.
I don't believe that that is financially useful
in almost every case, but it doesn't need to be.
Because so much of cloud economics, from
where I sit, is psychological in nature.
Where, oh, we have this workload that lives on EC2 instances and we
want to move it to Lambda, but we already bought the reserved instances,
so we're not going to do it because of some cost fallacy, which is
not much of a fallacy when it's that kind of money in some cases.
Okay, great.
Now if it can migrate to Lambda and still wind up getting the
discounts you've paid for, you have removed an architectural barrier.
And that's significant.
Now I want to see that same thing apply to, oh, if
you move from EC2 to RDS or DynamoDB or anything else.
That should be helpful too.
But whatever you do, don't do what SageMaker did and launch their own
separate savings plan that is not compatible with the compute savings plans.
So effectively, it's great.
You're locked in architecturally to one or the other,
Because machine learning is, once again, a marvelously
executed scam to sell pickaxes into a digital gold rush.
I mean, I like savings plans a lot, and we've been slowly, as
convertible, our eyes have expired, replacing them with savings plans.
And I think that it is pushing the other cloud
providers forward, because we're definitely multi cloud.
And so that's really useful, and I hope more people take on the platform.
Compute Savings Plan type model just because it makes our lives so much
easier or makes my life so much easier in terms of like planning it, selling
the commitment internally, just everything about it has made my life easier.
So, I mean, how many years later are we?
I definitely haven't found any big gotchas, I guess, in
the secondary market, but that doesn't really impact me.
Yeah, I spent a lot of time looking for it too, doing deep analyses of, okay,
for which instance classes in which regions is there a price discrepancy?
And I finally got someone to go semi on
record and say, yeah, there should not be any.
Please ping us if you find one.
Okay, great.
That is enough for me to work with.
Exactly.
We got that too and didn't believe it.
So we were like downloading price sheets and
like doing comparison, doing all that stuff.
Oh, trust but verify.
And when we're talking this kind of money, I don't trust very far.
They make mistakes on billing issues from time to time.
And I get it.
It's hard, but there are challenges here and there.
I mean, I'm glad you mentioned a minute ago that you are multi
cloud because my position on that has often been misconstrued.
I think that designing something from day one to work
on multiple cloud providers is generally foolish.
I think that unless you have a compelling reason not to go
all in on one cloud provider, that's what you should do.
Pick a cloud, I don't care which, and go all in.
Conversely, you have a product like Datadog, where your customers
are in multiple clouds, and first, no one wants to pay egress to
send all of the telemetry from where they are into AWS, and secondly,
they're not going to put up in many cases with their data going
to a cloud provider they have explicitly chosen not to work with.
So you have to meet your customers where they are.
In your case, it is absolutely the right thing to do.
And Twitter often gets upset and calls me a hypocrite on stuff
like this, because Twitter believes that two things that are,
uh, that take opposite positions cannot possibly both be true.
But the world is messy.
Yeah, and I mean, the nice thing about us being in
multiple clouds is we are our own biggest user, right?
And that's actually one of the reasons why I love working
at Datadog is because I get to use Datadog all the time.
And not only that, Datadog is on everything and we have all of our products.
It's I'm very spoiled with all of this, but I mean,
we are running in these different cloud providers.
We are using Datadog and those different cloud providers.
And, and that is just helping everything overall.
In addition to like supporting customers that are in
each cloud, because that is a huge reason as well.
Here at the Duck Bill Group, one of the things we do with,
you know, my day job, is we help negotiate AWS contracts.
We just recently crossed 5 billion dollars of contract value negotiated.
It solves for fun problems such as how do you know that your
contract that you have with AWS is the best deal you can get?
How do you know you're not leaving money on the table?
How do you know that you're not doing what I do on this podcast
and on Twitter constantly and sticking your foot in your mouth?
To learn more, come chat at duckbillgroup.
com.
Optionally, I will also do podcast voice when we talk about it.
Again, that's duckbillgroup.
com.
One of the problems that I keep running into across the board is
that, With things like Datadog, and again, not to single you out,
every monitoring vendor to some extent has aspects of this problem.
It's that when I'm a customer and I'm hooking my accounts up to
Datadog, I want you to tell me about things that are going on.
But the CloudWatch charges can be so egregious on the customer
side, where It is bizarre and frankly abhorrent to me when I wind
up paying more for the CloudWatch charges than I am for Datadog.
And let's be clear here, I am in fact a Datadog customer.
I pay you folks money.
Not a lot of money, but I pay you money.
Because I have certain things that I need to know
are working for a variety of excellent reasons.
And The problem that I keep smacking into on this is it's not your fault.
It's not anything you can do.
In fact, you are one of the better providers as far as not only not
being egregious with the way that you slam the CloudWatch endpoints,
but also in giving guiDannce to customers on how to tune it further.
And I really wish that more folks in the,
in your space would do things like that.
It always bugs me when I wind up using a tool that tries to
save money that in turn winds up costing me more than it saves.
Yeah, yeah, it's tricky there.
I have less experience myself setting up Data dog and
running it in my own infrastructure as I'm more digging
deep into like the cost stuff and us using the cloud.
So I can't speak to that specifically, but yeah, you're
not the first person that I've heard have that experience.
And again, it's not your fault at all.
I've been beating up the cloud watch team for years on this,
and I will continue to do so until I'm safely dead, which
depending on Amazon's level of patience might be in mere minutes.
In the larger picture wise, we have to remember
that we're super early in the cloud adoption.
Even looking at the cloud economics, FinOps,
cloud cost optimization world, right?
I feel like most businesses at this stage in their journey
are still in data centers and they're dealing with the problem
of how do we move to the cloud and do it cost efficiently?
How do we set everything up?
And that's where, like, the world is right now.
And I think that dealing with, okay, we are 100 percent running in the cloud.
What are the processes that we have in place?
How do we think of finance and the finance organization, not through the
lens of, we once had data centers and now we don't, but how do we look
through that in the lens of, okay, we are cloud native from day one.
What does the finance department look like?
And dealing with those problems is really interesting
because Datadog has never been in a data center, right?
We are cloud native from the very beginning.
And so it was interesting for me to join the company and
build up a lot of these processes because it is different
than what a lot of other people were dealing with and doing.
And it presents some really interesting problems and questions
that I think are going to be the foundation for the next
decade of building companies and operating in the cloud.
I always love having conversations with folks who
are building out teams to handle these things.
Because usually the folks I keep talking to, or who want to
have conversations like this, are building tools themselves
to solve this problem through the miracle of SaaS, where they
will bend over backwards to avoid ever talking to a customer.
And we're all dealing with the same AWS APIs.
There's not that much of a new spin you can put on most of these things.
But understanding what customers are actually trying to do instead of
falling down the rabbit hole trap of, hey, turn off those idle instances that
are all labeled DR site because you probably don't need them, is foolish.
And after a few foolish recommendations, tooling doesn't get there.
I'm a big believer that tools can assist
the process and narrow down what to look at.
I believe they shouldn't have to exist.
I think that the billing dashboard should be a hell of a lot better
natively than having to pay a third party to make sense of it for me.
But by and large, I do believe this is a problem
that is best solved from a consultative approach.
I mean, when I started this place, I was planning to build out some software.
Tried doing it, called Duck Tools, and wound up mothballing the whole thing,
because what we were building was not what the industry claimed to want,
and frankly, educating people into a position where then they see the value
and only then will they buy has never been a game that I wanted to play.
Yeah, I really like that article that you guys published about
exploring that product and the reason why you decided not to pursue
it, but it's super interesting in terms of where the industry
is going and building out those tools because I found that there
isn't really any sort of new thing that you can do with the tools.
All the tools that exist for looking at your costs are largely the same.
The main differences that I've seen is that the UI is
slightly different and they have different sales teams.
And if the sales teams are better, they're
going to get more of the market share.
And if the sales teams are not as good, it's going to be a smaller market share.
And like, It's weird to, to be in this industry for as long as
we have been and seeing like, okay, well, Anderseen Horowitz just
funded this new company and this other company got invited into
a Y Combinator or like all of these things that are happening.
And I'm kind of like, Okay, but what is this tool really doing differently?
And there are a few of them that are, that are doing something
kind of innovative and different, but there's also a few
that are just like, this is a space where people are in.
There's money here, we're doing the same thing, but
we got our sales team and we'll carve out our little
corner and then we'll get acquired and that'll be that.
Although I guess we're just at that stage of innovation in this space, I guess.
Yeah, I have no earthly idea what the story is around how these
companies plan to differentiate because it seems to me that they're
directly attempting to compete with Cost Explorer, which it's taken
some time for that thing to improve to the point where it is now and
it'll take further time for it to improve beyond it, but long term, I
don't think you're going to outrun AWS on a straight line like that.
Yeah, I mean, like, when you work for one of these third
party cost tooling things, and you're working with one
of your customers, and they're like, how do I view this?
And it's kind of like, that is the easiest
thing to find in Cost Explorer as well.
It's, it's, I can't imagine being like, Well, you should pay
me thousands, tens of thousands, hundreds of thousands of
dollars a month to view it here when like Cost Explorer is free.
And I think Cost Explorer is, it doesn't do everything, but it's
gotten a lot better at what it does and it could probably solve
90 percent of people's problems without using a third party tool.
You are at significant scale in multiple clouds.
So the answer that these companies always give is, ah, but we provide a single
dashboard so that you can look at costs across multiple providers in one place.
Is that even slightly useful to you?
Man, if you need dashboards, get a dashboard tool.
Don't get this crazy cost analysis tool, right?
I mean, like there are some great, dashboard solutions that you can
get where you can connect your detailed billing, cost and usage report,
whatever cloud provider is calling it, but like that really detailed
gigabytes per hour report, and then visualize it, build reports, do all
that kind of stuff, because that's not something that the tooling does
well right now in terms of like building out cost dashboards and stuff.
But that's also right now.
It could in the future.
Yeah, if you're a BI tool, wind up passing
out templates that normalize these things.
I am so tired of building it all from scratch in Tableau myself.
If you're Tableau, sell me a whole bunch of things that
I can use to view this stuff through so I don't have to
wind up continually reinventing that particular wheel.
Yeah.
Ugh, I like your approach.
I didn't know the answer when I was asking the question.
I was about to learn something if you'd gone the other direction,
but nope, it's good to know that my impressions remain intact.
Yeah, I mean, I've used different tools in the past.
Again, I hesitate to name any of them, but there's a
few in the space that I feel like everybody, if they're
in the space, they know which tools I'm talking about.
Yes, we do.
And yeah, I've used them.
They're okay.
A few of them are okay.
A few of them are better than others.
But I mean, I was trying to evaluate the value add over me manually
setting some things up and having some sort of visualization and just
like the value add in terms of what they were charging even if it was
like a significantly smaller percent of the bill because that alone
like percent of bill is such a difficult kind of cost model to do.
I hate that.
Pricing is hard.
Let's start there.
Yeah.
I hate the percent of bill because then it's let me get this straight.
I'm paying you a percentage of things like Data transfer
charges that I know are fixed that I can't optimize.
I'm paying you a percentage of my AWS Enterprise Support subscription.
I'm paying you a percentage of the Marketplace stuff, and so on and so forth.
And it doesn't work.
At some point of scale as well, it's, I could hire a team
of 20 people and save money versus what you're charging me.
The other side of it goes, Ah, we'll start,
we'll charge you percentage of savings.
Well, then you wind up with people doing a whole bunch of things.
Like before they bring you in, they'll make a bunch
of ill advised Reserved Instance Purchases or Savings
Plan Purchases, you can then unwind after the fact.
When I was setting this place up, I looked
long and hard at different billing models.
The only thing I found that worked is Fixed Fee, The End.
Because at that point, suddenly everyone's on board with, hey,
let's solve the problem and then get out as soon as possible.
We're not trying to build ourselves a forever
job nestled in the heart of your company.
And it's the only model I found that removes
a whole swath of conflicts of interest.
And that's the hard part.
We have no partners with anyone in this space, including AWS
themselves, just because as soon as we do, it becomes extremely
disingenuous when we suggest doing something for your sake that happens
to benefit them, such as maybe back that S3 bucket up somewhere.
Well, okay, if we're partnered with them, does that mean
we're trying to influence spend the other direction?
It just becomes a morass that I never found it worth the time to deal with.
Yeah.
But that SaaS.
Yeah, that that makes a lot of sense and I haven't actually thought about
pricing model for consulting in the space that closely but I mean when you're
charging a percent of bill or percent of savings you have the opportunity
to Screw the customer right through through all the things that you were
saying if you charge a fixed fee you have the possibility of Undervaluing
yourself which the only one that's true in that case is you potentially and
if you're okay with that risk and you're okay with those dollars That's great.
Cause yeah, if you're able to be like, okay, here's the
services that I do, here's the fixed cost, done, done.
It just sets everybody's expectations for the relationship in a much
better way that you're not constantly worried about like upsells and
other things that might happen along the way that screws the customer.
And that's the hardest part, I think, is that people
lose sight of the entire customer obsession piece of it.
That's one of the things Amazon gets super right.
I wish more companies embrace that.
Dann, I want to thank you for taking so much time out of your
day to suffer my slings, arrows, and half formed opinions.
If people want to learn more about who you are
and what you're up to, where can they find you?
Yeah, I have a website you guys can go to that links kind of everywhere else.
It is Dannb.
org, and I spell my name with two n's, so d a n n b.
org, and I have LinkedIn, I have Twitter, I have a monthly newsletter.
That is not really about FinOps or anything, but I really enjoy it.
I've been doing it for like a year now that you should sign up for.
And links to that will of course be in the show notes.
Dann, thanks again for your time.
I really appreciate it.
Yeah.
Thanks so much for having me again.
It's been a blast.
It really has.
Dann Berg, Senior CloudOps Analyst at Datadog.
I'm cloud economist Corey Quinn, and this is Screaming in the Cloud.
If you've enjoyed this podcast, please leave a 5
star review on your podcast platform of choice.
Whereas if you've hated this podcast, please leave a 5 star review on your
podcast platform of choice, along with a comment featuring a picture of
several corkboards full of post it notes and string, and a deranged comment
telling me that you have in fact finally found the catch in savings plans.