Subscribe
Share
Share
Embed
课代表立正的官方Podcast
深度访谈，有用干货，亲身验证的「真本事」
Superlinear Academy创始人，Maven Top AI Instructor
前Statsig布道师（OpenAI收购），腾讯副总监，Meta，Amazon；康奈尔经济学博士
社区：Superlinear.Academy
课程：ai-builders.com
个人：lizheng.ai
okay um
statsig was uh created by a cervix
facebook engineers um and data
scientists
uh with the intention of bringing a set
of uh
infrastructure tools that we use inside
facebook
to everyone so inside facebook the
development infrastructure is very well
thought out and a lot of investment has
gone in
and that enables teams to move really
really fast
make product decisions really really
fast and then ship only
the right features to the users
and what we want to do at statsig is
give the same power
to those teams that may not be able to
like actually invest
so much in infrastructure tools to
to get to the right decisions for their
own product
what we do is give a set of development
tools
that allows you to first build uh
your new features and when you build we
want to make it really really
easy to move fast and decouple product
features from
uh code shipments and so
for example if you have a mobile team
and a server team
uh that are building the same feature
they need to talk to each other
and they need to be able to like launch
the same feature at the same time
it's very hard to coordinate a bunch of
those things
especially because of dependencies with
like apple and android
google play and so on so
what we do is we allow these things
called feature gates that allow you to
build features separately and then turn
them on
whenever you're ready and then the
second thing is we allow you to measure
the impact of each of the features on
your product so when i when we talk
about the measurement
this is important to understand that
it's not just about like the surface
level metrics
it's about when you ship a new feature
how does it affect the overall company
level in the business level metrics
and we allow you to like simple uh with
using simple measures um
uh simple tools measure those and get to
the right decision
and then once you get to the right
decision you can either ship your
product or
decide to go iterate on your product and
so the idea
is to like uh make it so easy to run
these kind of
tests uh in in in language like this
experiments and
a b tests uh so much so that
you should be running every feature as a
test
and that allows you to like get to 10x
more tests than most other
tools out there and then that means
you're already
like on a well on your way to like
trying out various different things and
having your product lead the way in
growing it rather than
waiting for marketing or sales teams to
grow your product
the last thing is also arriving at
causation so when you have features
behind feature gates and when the tool
can give you a really good way to like
debug
why metrics are going in a particular
direction
um it gets you faster
at arriving at like why
your products are doing uh what they're
doing and then be able to like simplify
some of your debugging
for uh on without relying on data
scientists
the the the other thing i want to
mention here is that
the team that built all of this came
from facebook and
we've seen a lot of the the way product
growth happens
inside these companies and so a lot of
experience has gone into
building these tools not just that we
also make it like extremely reliable and
scalable
because the tools have served some
pretty large companies
so let's start um so when you first go
into static.com
you see our landing page and landing
page gives you an overview of like
uh what we're trying to do basically
what we want to do is to make
uh make it easy for all products to
build and measure
what they're building and how it affects
their product metrics and then based on
those
make decisions to ship or not to ship or
to go iterate on the product
um so you can read a lot more about what
we have going on here we also talk a
little bit about a product and the
documentation
and our team and so on but once you
sign in we land you in our console which
is where
you configure our product the first
thing you want to notice
is that we have a list of projects that
you can create
and you can create as many projects as
you want these projects
are scopes and these scopes have
people from teams invited so kind of
like each team has its own set of
features and metrics and so on so they
don't step on each other's toes
what i'll do is i'll start showing off
our demo project so we have a demo
project
where all the data is filled in so first
thing i want to show off is feature
gates
so what this does is like you have a set
of features that you want to build and
every time you write code you want to
create uh
your code you want to put that code
behind the feature gate and what that
allows you to do is to decouple code
launches from feature launches
and so whenever you're ready to open up
a feature
you will then go turn on a feature or
turn it off if something goes wrong
here's a view of all the features
or the feature gates that we have and
you can turn them on or off easily you
can also see
how well they're doing in terms of like
how many checks are happening on a daily
basis
i'll pick one example to go into and
drill
so once you go into a feature gate
you will notice a few things here so the
first thing is i'll actually tell you
your daily exposures and then it also
gives you a toggle to like either turn
on or off
um as you need and then once you come in
here you see a set of rules
so the rules are determine how a feature
is
opened uh for people and allows you to
target a specific feature to
specific audiences and these targeting
criteria we have
quite a few of them and each of them
follows this kind of like this
workflow of ruleset so if then else
condition
in this particular case you have this
this feature is open to employees only
the way we determine employees only is
by
checking if an email if their email the
user email contains
any of swagger.com that means that
the user is an employee and that means
we want to like open this feature to 100
now and once that rule fails it'll go to
the next rule set and so on and so forth
and if you want to add more conditions
it's relatively easy
you can add conditions based on all of
these criteria so we have user id
app version browser name browser version
country uh
country means like you can actually like
target the feature only to a specific
set of countries
we have email environment we also have
ability to like either
chain existing gates like either failed
target gate or passes target gate
we have operating systems so you can
target like ios or android um so for
example you can do ios android
um only that means you want to only open
this feature to mobile
uh users and so on another one of those
is
time so you can actually say like you
know open this feature only
after a specific day so if ever you're
doing like a memorial day sale or
something you can
you can then time your feature um
so that's this is the way you do
features uh and
you can get these features to specific
people once you create a gate you also
have the ability to like look at the
history of the changes and be able to
like restore
to a well-known good state we also give
you
ability to like take uh code snippets
you can copy this code and post
paste it in your client or our server
depending on the language and the
environment that you're using
then once you have set up a feature gate
one of the things that you're going to
notice is like
there are some people that are being
exposed to the feature and there are
some people that are not exposed to the
feature
statsig takes advantage of that fact and
then is able to provide
a set of features and metrics uh for the
people that are exposed to the feature
and
people that are not exposed to the
feature using this analysis tool called
pulse
pulse gives you kind of like the
cumulative effect of
uh how a particular feature is
performing so in this particular case
you can see like
uh 10.45 percent of the people are being
exposed to this feature
and then the remaining are not and then
if you go down
it'll actually show you how that
particular feature is doing
across a bunch of metrics so in this
particular case you will see like
a weekly stickiness is is actually down
quite significantly by 4.38 percent
and anytime you see red uh values that
means it's
statistically significantly uh
underperforming
and then anytime you see green values in
this particular case there are no green
values
um that will be significantly performing
better
and then the gray ones are those that
are directionally correlated but not
quite have reached the statistical
significance so
now you can actually look at this
feature like turns out this feature is
doing really poorly
and you don't want to launch this
feature so that's the kind of decision
that you want to make
and you can drill go back into pulse and
you can see like uh we make
pulse available for every feature that
you have shipped and uh so you can be
able to like make the right product
decisions um
based on every feature that you want to
watch
moving on we have another feature called
dynamic
config dynamic config allows you to like
provide
real time and runtime configurations uh
based on targeting so one example is
localization so
for example if you have a set of strings
you want to provide the default value in
english
so you know you have like us followers
and share and so on these are all
provided in english and then imagine in
the future
um your your localization team comes
back and says like
hey we had some spanish strings um and
you can simply add a new condition
where you say like if the user is coming
from any of these countries mexico spain
argentina or so on um then you want to
provide
spanish strings and then you can
actually go update your configuration
this way
this allows you to like very easily go
change the behavior of the application
without having to write code or ship
new versions and this is easily
extensible to like other languages
and so on the next thing i want to show
off
is holdouts so sometimes when you're
launching
lots and lots of features during a small
period of time especially if you have
teams that are moving really fast uh you
want to understand what is a cumulative
effect of all of the features that
you've launched during a specific time
so what we do is we provide you with
this thing called hold out
which you can set up at the beginning of
the half or the quarter depending on how
much time you want to spend to
measure this thing and you can make a
holdout be global and assign like
you know a small percentage of the
people that are held out from every
feature so in this particular case we
have
uh two percent holdout so what that
means is like
any new feature that you're building um
in that
in that quarterly period will be behind
uh this holdout so those features will
respect this whole
so in this particular case let's go back
to the dark point home page that i
showed you
and here you'll see that we have
this gate is respecting a list of
holdouts and one of them is the
you know the h1 product hold out which
is two percent
um so this is very useful because uh
our pulse analysis engine automatically
picks out
holdouts and will also tell you the
cumulative effect
of all the features that were launched
during a specific period of time
and so pulse is going to tell you like
okay well the purchase rate in the last
six months has gone up by
quite significantly but the dau and the
mau and the
wau have actually fallen so this is
gives you an overview of like all of the
the product work that
your teams have done very very very
helpful in like coming up with uh
you know the uh how one feature
either impacts positively or negatively
other features we also provide
a set of metrics metrics are very useful
to like you know when you're logging new
events
you can see how each of these metrics
are doing
so in this particular case you know
product view if i remove so you can see
like all
a set of uh features and you can also
see a set of events that you're logging
and these events um are really useful to
like you know debug
um your product integration
then what we also provide is set of
derived metrics in this particular case
we give you a list of all the user
level metrics like daily active weekly
active monthly actives
one of the important features here is we
actually overlay a set of feature
updates um so whether you've launched a
new feature or turned off a new
uh an old feature um and it's overlaid
with the metrics so that kind of gives
you a
clean idea of like okay what could have
changed any kind of inflection in your
metrics in this particular case you see
the
the metric going down and then we made
some changes and then the metrics
started going up
and now you know the list of all the
changes is captured right here so it
gives you a really easy way to start
debugging
any issues and
then finally i want to show off
integration
so our product integrates with like lots
and lots of data aggregators so that if
you have data in any of these
places you should be able to like
quickly integrate that and the statsig
will pull that data
and then if you are a developer we have
lots of documentation around how you can
use
um you know walk-through guides of like
how you can create your new feature
uh how do you make your code dynamic and
then so on and so forth
we also have a list of client and server
sdks
so whichever technology you're using we
have a solution for you there
we also have http rest apis that you can
call directly
if we don't support your sdks so that's
that brings us to the end of
this demo cool