Subscribe
Share
Share
Embed
The Product Engineering Podcast explores how software teams can move beyond shipping features and start delivering measurable outcomes. Hosted by engineering leader James Charlesworth, each episode covers practical ideas for product engineering, engineering leadership, team operating models and building products that create real impact.
In 1961, a guy called Robert McNamara
entered the US Department of Defense,
rich in his belief in
quantitative analysis.
Robert McNamara previously worked at the
Ford factory and while he was there he
really pioneered the use of data and
systems analysis to drive efficiency in
the production of cars.
And he took that belief in data and
analysis to his new job in the Department
of Defense in the 1960s and
he joined the Vietnam War.
Now when he was put in the charge of the
war he applied the same approach that he
had learned at Ford.
So he turned up to Vietnam and he started
measuring things like logistics networks,
which areas of the map were under US
control, troop numbers,
all that kind of stuff.
Robert McNamara declared quite
confidently that the war was going well
and that we were going to win.
Now spoiler alert, the US did not win the
Vietnam War and one of the reasons was
that the data that McNamara had available
was basically just that, it was the data
that he had available and what he ended
up doing is ignoring all of the more
important stuff like the morale, the
sentiment of the Vietnamese people, all
the kind of stuff that you can't really
measure and that gave rise to something
called the McNamara fallacy.
The McNamara fallacy is something that a
lot of organizations can fall into where
if you over index on things that are easy
to measure then you end up ignoring the
things that are difficult to measure just
because they're difficult to measure.
(Music)
Hi, my name is James Charlesworth and
this is the Product Engineering podcast
where we talk about the intersection
between product management and
engineering and today we're
going to talk about measuring.
How do you measure a
product engineering team?
Because there's different metrics that
you can use and they're often split into
output metrics and outcome metrics and as
the McNamara fallacy shows us, output
metrics are often easier to measure but
they don't always tell the full story and
if you only measure the things that are
easy to measure then you will miss the
bigger picture and you'll miss what's
actually more difficult which is the
outcome, the product outcomes you're
trying to drive or the business outcomes
that you're trying to drive and we fall
into this trap because output metrics
often feel good and they're often very
very available in the
tools that we already use.
So if you're an engineer or an engineer
manager then the tools you use every day
can give you data and they can give you
data that can cause you to
fall into the McNamara fallacy.
So for example, let's say you're using
Jira as your project management tool.
Jira can tell you things like cycle time,
it can tell you burn down, velocity,
story points, all that kind of stuff.
GitHub can tell you how many pull
requests are going in, how long they're
staying in review how many lines of code,
people love that when
you measure lines of code.
Those kinds of metrics are very very
easily available to engineering leaders
and they can lead you into the trap of
thinking that just because those things
are easy to measure that's what you
should be indexing yourself on and it's
often ignoring things that are more
difficult to measure.
So let's quickly talk about the
difference between output
metric and an outcome metric.
An output metric is measuring how
efficiently or how well
you're producing things.
So you know lines of code even though
people hate that as a metric it is a
fairly pure output metric because lines
of code will tell you like how much code
is being shipped to production.
Door metrics will tell you how successful
your changes are, how regularly they're
going out and how often
they need to be rolled back.
Story points will tell you how
effectively you're capturing those
features into stories and
pushing them through the board.
Sprint metrics will tell you how
effective your sprints are being and how
accurately you're predicting the amount
of work that will be shipped.
These things are all output metrics
because they're taking something from a
feature definition into some working code
and then measuring how
efficient that process is.
So that's a little bit
like a production line.
Take the example of
the Ford production line.
On a production line you've got lots of
stages and they're similar to the stages
in software So a production line you
might have like assembling the chassis
and then adding the
engine and that sort of stuff.
I don't really know how cars are made but
I imagine it's a several step process
like that and so by looking at all of
those individual steps and how they all
fit together you can measure how
efficiently you're building cars in a
factory and you can tweak various parts
of that process to
make it more efficient.
So output metrics are very very useful in
manufacturing but they're not as useful
in product engineering and the big
difference is that when a Ford makes a
car they know what
that car is going to be.
The design has already been put in place.
It's already been tested.
All they need to do is make more cars
more efficiently and at a higher rate but
in products engineering you often don't
know what features
you should be building.
So with all of the validation and
discovery in the world you can never 100%
know if the thing is going to drive the
business outcomes that
you want in your company.
So you can't treat software engineering
purely as a kind of
production line system.
Now that's not true for all projects.
Some projects are incredibly predictable.
Some projects like maybe a migration
project if you're migrating a database
from one technology to
another that's fairly predictable.
You know what the end
state should look like.
You know you know if you're
re-platforming a product or something you
know that the thing you're going to
should look in a certain way because
you've got a reference to
what it should look like.
So if you're doing a migration or
re-platforming then fine treat it as a
production line and measure how
effectively you're doing it.
But most products
development is not like that.
Most product development has got a lot of
unknowns in it and it's not simply a case
of going from A to B
extremely efficiently.
So you also need to
consider outcome metrics.
Now outcome metrics are basically derived
from the business value that you're
trying to get from a product.
So that could be if you're a SaaS company
your outcome metrics could just be your
SaaS metrics like you know churn, net
retention revenue, net revenue
retention that sort of thing.
Or they could be product metrics so
product adoption, net promoter score,
customer feedback metrics, feature usage,
activation rate all those kinds of things
that product managers are
much more familiar with.
They tend to be outcome metrics because
if your product activation rate goes up
that's probably the
outcome that you want.
You want more people to use your product.
If you can get more people to create like
stickiness inside your product so they
don't want to leave it, they've got very
high net promoter score
that will reduce churn.
So those kinds of outcome metrics do tie
into the business metrics that ultimately
keep businesses alive.
But they are measured in a very very
different way to how
output metrics are measured.
You can't measure churn by
looking at nine lines of code.
So you need to have both of them.
It's horses for courses basically.
You need both output metrics and outcome
metrics and a lot of engineers fall into
the trap of over indexing on the output
metrics just because
they're easier to measure.
They're not just easier to measure,
they're also easier to attribute as well.
Because an outcome metric something like
the product activation rate you know how
many people are using a feature, that
feature could have been built by many
many teams at your organization.
And so if your output like your
activation rate goes up that could have
been because the customer enable team did
a really good job or it could be because
the product team designed a really good
feature or it could be because the
engineering team
asked the right questions.
There's lots of ways that
those outcomes can go up.
Whereas output metrics you can break it
down very very easily.
So if you are looking at something like
the number of pull requests that are
going in or the number of story points
that are happening, you can break that
down to an individual team
level very very effectively.
You can look at a group of five or six
engineers and you can compare their
sprint metrics to another team sprint
metrics and you can just decide that this
team needs a little bit of help or this
team maybe could help this other team.
That's much much harder
to do with outcome metrics.
It's very difficult if customer feedback
comes in and says we find
this feature difficult to use.
It's very very difficult to pin that to
one individual area of your teams and say
okay that feature was difficult to use we
just need to go and
make a tweak over here.
So it's very very difficult and it's also
very difficult to
attribute outcome metrics.
But again don't fall into the magnum r
fallacy of assuming that just because
they're difficult to measure and
difficult to attribute that you shouldn't
be measuring them
because you absolutely should.
Both of these metrics are important and
you need to take both of them together.
And product engineering really is about
combining output
metrics and outcome metrics.
So you still have an efficient system.
It still is important to measure how
efficiently you're putting product out
but you also need to combine that with
how well are we achieving the outcomes
that we want to in our product.
I'll give you an
example from my experience.
I was once managing two engineering teams
that were working on two
very different projects.
One of the engineering teams was working
on a framework update.
So if I remember right they were
upgrading the version of React.
So it was using the component if you know
much about React like they used to be
this old way of creating React components
which would use like a
component class-based syntax.
They were migrating the
code base to use hooks.
So that was a massive migration project
that took them about a sprint or two.
I had another engineering team that were
changing the navigation
inside the user interface.
So both of these engineering teams had a
piece of work that was taking them about
two weeks and they both did a fairly good
job of delivering it.
So the migration team that were moving
this version of React, they did, they
estimated it pretty correctly and I could
track how efficiently the pull requests
were going in, how much refactoring was
happening, how much code was changing and
I could track the number of bugs that
were coming in as a
result of that migration.
So the doorware metrics for
that team were easy to track.
They weren't all perfect
but they were easy to track.
The team that was changing the navigation
on the other hand, again I did have all
the output metrics for them.
I could see that they were doing their
work but what happened is we put the
navigation changes in and users
absolutely hated it.
It caused a lot of
resentment from a lot of people.
Whenever you change navigation that
people have built up muscle memory
around, then you can initially at least
achieve a negative product outcome.
So the team that were changing the
navigation achieved a negative product
outcome even though they had actually
delivered that project on time.
And that's difficult to console yourself
with basically because we wanted to
celebrate the success of both of these
teams and the one that had done the
migration project, that was very easy to
celebrate the success of.
We knew what we were trying to do, we got
there, we tested it, it was working,
congratulations, you've done that
migration, you can
move on to something else.
The team that were changing the
navigation, again they did everything
they were asked to do
but people hated it.
And so they can't move
on to something else.
They then need to go back through another
cycle and iterate on that navigation and
fix a bunch of stuff
because people didn't like it.
And so that's difficult because
ultimately one of those teams did exactly
what you were supposed to do in terms of
outcomes and the other one didn't.
The other one, even though they did
exactly what they were asked to do, they
didn't drive the correct outcome, they
were trying to make the navigation easier
to use and they ended up making it harder
for a lot of people.
And so measuring these two teams and
celebrating their success, it was
incorrect to do that just based on how
like efficiently they
delivered their work.
And that's the important thing about all
kinds of delivery metrics or output
metrics is they tell you how efficiently
you're building things but they don't
tell you anything about whether that's
the right thing to be building.
And so if you just celebrate like the
efficiency, then you're not going to
celebrate the actual changes
that matter to the business.
And this is how you end
up with feature factories.
So I said before that product engineering
is not like building a car in a Ford
factory and ultimately that's incredibly
true and you can't treat a team as if
they are a production line because then
you will end up
celebrating the wrong things.
You will end up celebrating a production
line being super efficient and they've
built all their cars by 5pm and so
everyone can go home happy.
You can't do that with a team.
If you do that, you end up with a feature
factory model where your team is only
there to turn
requirements into working code.
And particularly in the world of AI, AI
can now turn requirements into working
code extremely efficiently.
So if you're judging your ability of your
engineering team to turn some
requirements into code and then AI comes
along, you might think well why do I even
need this engineering team?
And a lot of software engineers at the
minute are getting extremely nervous
about that very thing because they've
tied their entire purpose into the
ability to move code through a factory
and that factory is now automated.
Robots have come in and they've started
doing all those individual pieces.
So it's more important now than ever to
change the model about how we celebrate
success and how we measure engineering
teams so that we're not
treating them as feature factories.
We're treating them as teams combined,
products and engineering teams that are
driving towards an outcome and we're
judging them based on the outcome that
we've asked them to achieve.
So the mistake I made with that team that
I'd asked to change the navigation was
that I was holding them accountable to
how quickly they could do that project,
should have been holding them accountable
to how easily users can
find the new navigation.
That was the correct thing to hold them
accountable for and I would have just
given them as much time as possible until
we've seen signals that they've actually
meaningfully improved the product.
And if it took them three times as long
that's basically okay because if you say
to teams it can take you a bit longer but
as long as you achieve that outcome then
they will naturally try and make it take
a little bit quicker because they will
iterate more, they'll do more
experiments, they'll try and find out
more before they commit too much time
into things because you told them that
I'm not measuring on your cycle time or
the number of code like lines of code you
write, I'm measuring you on when people
can find this more easy to use.
And that brings me on to my next point
about measurement because when you are
measuring outcomes it's extremely
important to work out how you're going to
measure them before
you start doing the work.
Output is fairly easy because you know
that, you know you've already got the
tools in place to measure things like
cycle time or pull request throughput so
you don't need to decide that you can
just know that you've got those metrics
there but outcomes you need to decide
before you start investing time into
something how are we going to measure
this outcome because they are more
difficult to measure and again just
because they're more difficult to measure
does not mean they can be ignored they
just need a little bit more thought they
need a little bit more time spent up
front working out how are we going to
measure the success of this project how
are we going to measure the impact of
this feature and then as they start
working you can bring that measurement in
and you can try and get the team to
iterate earlier and more quickly so that
they have a higher chance of achieving
that outcome because after all measuring
things only really matters when those
measurements can change a decision.
Measuring things after the fact is okay
if you want to look back and do a
retrospective on some work but you really
want to be measuring things from day one
because if you're not measuring
constantly through the entire process
then you're not going to be able to
respond to those measurements and outcome
metrics particularly are often lagging
indicators so often it takes a long time
for changes to show up in outcome metrics
particularly around things like user
feedback or net promoter score surveys
and things like that user feedback won't
come in until you've shipped and people
have had enough time to play with the
thing that you've built so they're
lagging indicators or they can be lagging
indicators so it's even more important to
get those lagging indicators brought
forward and get some of that user
feedback earlier in the cycle.
Output metrics are often leading
indicators so you can tell every single
day how much code was shipped that day
you need to be able to try to get it so
that every single day you can tell how
far we towards the outcome that day.
So I want to close out this episode by
giving you a question that you should ask
of your engineering team your product and
engineering team and that question is did
the product get better because most
outcome metrics are trying to answer that
question we all want to build great
products and we all want those great
products to be used by lots of people who
stick around and stay in our product and
potentially drive business revenue as
well and so in order to do that we need
the products to constantly be getting
better and you can't tell that just by
measuring how efficiently you're changing
that you also need to be able to measure
actually how well it is improving.
So ask yourself that question do you have
a way of measuring outcomes and if you
don't what are those things you can do
what things can you put in place that
will help you measure the impact of the
work you're putting out.
So I'll leave you with that question I
hope you're enjoying the podcast so far
these first few episodes are just solo
episodes with me so I can get a bit of
rhythm into this but we will be having
guests on and exploring some other
people's opinions in the future as well
so stick around pepodcast.com you can
find it on all of the common streaming
services my name is James Charles Wirth
and I'll see you in the next one.