Subscribe
Share
Share
Embed
课代表立正的官方Podcast
深度访谈，有用干货，亲身验证的「真本事」
Superlinear Academy创始人，Maven Top AI Instructor
前Statsig布道师（OpenAI收购），腾讯副总监，Meta，Amazon；康奈尔经济学博士
社区：Superlinear.Academy
课程：ai-builders.com
个人：lizheng.ai
But after writing the book, I realized I
knew this by talking with people, by
looking at different notes, for example,
taking the Facebook analytics book or by
making mistakes. There was no book out
there that I can learn all this
information from. Data consumption is
probably going to be disrupted a lot
using AI. Like instead of having so many
dashboards or so many like ways to look
at data, it's probably going to be a
very different interface.
>> The first one is I think all the data
people right now have the potential to
become a real builder. Hello everyone.
My name is Ming. I'm one of the
co-authors of the book Ghost uh Data
Analyst Playbook. Today, I'm just so
excited and honored to have all of you
here celebrating the launch of the book.
To kick things off, I want to first give
a huge shout out to our host today,
which is Pylon. This is their office
here. So, Pylon is um is a support uh
platform that is dedicated built for B2B
companies. In my opinion, it's the best
and the only support platform that
connect all the tools you use to talk to
the customers like you know Discord,
Slack, chat, email and everything. And I
have known these three co-founders from
very early on like two and a half years
ago when there were only three people
building pylon from a loft near Oracle
Park. Now the team has grown to of 60 to
70 people team and they just raised
their series B. So if your team is also
using any of these kind of Slack teams
and all these tools and feel free to
talk to them because they're going to
give you the best customer support
platform ever. or if you are considering
joining them uh feel free to talk to
their folks as well. And secondly, I
also want to give a huge shout out to
our sponsors today uh which is three
ventures and Philis venture is a venture
capital firm based in San Francisco.
They have been invested in ambitious
funders really turning this ambitious
ideas into uh go to market advantage. So
they have invested a lot of great data
infra companies like Oman also we have
Brian here. Brian is from the ad
ventures who is leading their AI over
there. So if you're interested to learn
more about the ventures feel free to
talk to him afterwards. Another one I
want to give a huge shout out is
Felicis. Unfortunately uh Nancy from
Felicity couldn't make it today and
Felicius is also a VC firm has been
around for almost 20 years. They have
invested in a lot of great companies
like Shopify, Credit Karma, Notion and
recently have been investing in like
core and all these great companies. Uh
if you're interested in their in
learning more about their companies,
feel free to check them out as well. And
lastly, I al also want to give a warm
welcome to Julie.
Thank you so much for coming to moderate
our uh session today. So I think a lot
of people here already know about Julie
and I'm joking with Julie people here
not for us but for her. So I have known
Julie since my Facebook days. I still
remember the days we were in the meeting
talking about how to improve the quality
of notifications with Chris Cox. And uh
Julie now is the founder of thumb which
is a product analytics platform. It's
not like a traditional product analytics
platform but it's actually help you
understand what is going on behind your
trend. So they have a lot of models
behind the code and also to my opinion I
think it's the best design product and
tools out there as well. Thanks to Julie
and also Julie is uh the bestselling
author of the making of a manager and is
also one of my favorite books as well. I
learned so much when I was doing this
transitation from IC to a manager. Uh so
I will hand the mic to Julie to get
things started.
>> Hello everyone. Thank you so much for
being here. I just want to say that I'm
actually very honored to be invited to
an event. Um often people are like oh
you know talk about management or talk
about this app but it's actually goal
since starting to be accepted as part of
the data and analytics platform
community. So thank you so much and
thank you to uh Joe and and Bening. It's
super incredible that these people who
have worked in such amazing places and
know so much about everything from
experimentation to PLG playbooks to how
to scale and come up with the ways that
that we make these uh scalable the
problems that they put all of their
knowledge into the book. I know many of
you guys probably have already checked
it out or bought it, but highly
recommend you all get on Amazon and get
a copy of this book. Um, so we're also
going to be able to open this up for for
questions in just a little bit. Um, but
first let me introduce uh my esteemed uh
guests on this panel. So as you guys
know um we have my lead and she is
somebody who I think needs almost no
introduction. She's been in so many
places heading growth uh data at notion
from Facebook to mother duck. um and she
is also very involved in the startup and
uh analytics ecosystem. So when she
gives you a recommendation I always take
it very seriously.
>> Thank you.
>> Uh Joe we have here all of us I think
knew each other to some extent at Meta.
Uh but Joe has almost a decade of
experience especially in data
engineering. Scott is like a data
engineering wizard. One of the favorite
projects, one of the most influential
projects at Meta was this idea of this
gran fixed granularity framework of
which he and Yen worked on uh which
really helped us be able to scale uh
their analytics and the ability to get
fast answers from data. Um so I'm super
excited to hear him talk about some of
his playbooks uh in in the conversation
ahead. And finally with
>> Dr. reasons and um he is a super
prominent voice in the data and growth
community. Um we all know that he's got
an incredible uh advocate community
member with um the the way that he has
uh been at 10 cent Meta and Amazon and
he has 300,000 followers for his content
channel for his Chinese podcast. Um he's
super passionate now about AI education
as well as uh been one of the great
leading voices for experimentation. So
very excited to have these three. Please
give them a warm welcome.
>> All right, let's get down to brass tax.
So what made you decide that we all
needed this book that you decided to
write?
>> That's a great question. Um so I think
the reason why I think this book is
super important is is actually totally
based on my on my own experience. When I
was at Facebook, uh I was working on
product analytics. I never worked on
growth data stuff before. Then I joined
notion as the first ghost data scientist
and I felt really lonely because at
Facebook there was so many data
scientists. There is something called
workplace which is like a Facebook feed.
You can learn so much from other data
scientists about what they are doing. So
you can learn so much from people who
are doing similar things as me. But that
notion because I was one of the earliest
data scientists joining the company. So
I had nobody to turn to where I had any
questions and I just don't know who is
actually working on data. So instead I
had to actually turn to the network I
had. I started harassing my friends from
Facebook ask them questions and also
read a lot of blogs trying to figure out
what things should I do in order to
understand the growth factor. What kind
of metrics most important for a grow
stage company like notion afterwards I
was able to figure it out by myself but
it was very painful and inefficient.
That's why I wish if I had something
kind of like a framework help me to get
started. I think it's going to be great.
I think that's one of the motivations
why we started to create this book.
>> And Joe, maybe you can share a little
bit more about both that motivation, but
also how did you guys get this done in
between your your very busy day jobs.
>> Maybe uh you may have heard me share
this story. Um but I joined Meta in 2015
and I was uh I came from a very non uh
technology company. I used to work in
the east coast for a spinal implant
engineering company. Uh we had to pay
for our soda. So very very traditional
right and uh I was doing traditional
business intelligence and I came to meta
and trying to really understand what was
the value that analytics uh provides for
meta right so the question I was
wrestling with was what if they fire
everyone uh in in analytics right will
the company actually lose anything or
will they continue to continue to
sustain so
this question for a few months I
realized that oh we are actually like
helping launch success uccessful
products. That's what analytics like
core value prop is right. So if we are
to launch successful products through
analytics then it means we need to
understand product development better so
we can apply analytics better right so
if you're a sports analytics person you
should know the rules the game and whatn
not if you're sales analytics you
probably understand sales right so
similarly for product analytics you need
to understand product the the gap I saw
was most of the other domains like
accounting finance whatever domain you
think about had very standardized theory
behind them And over a period of time,
over decades, they've all been like kind
of fine-tuned a lot. But for product
analytics that did not exist in the
minds of people at few places, right? My
motivation was okay, how do we uh have a
playbook that that kind of captures all
this knowledge like at Metam amazing
people everyone here might uh like
acknowledge. So how do we get that and
put together a playbook was the
motivation that how it evolved within
Madan? That was my motivation to even
step out and think about how do we now
bring this to outside meta right we see
data scientists data engineers in so
many companies working on growth working
on product analytics but the gap is we
don't have the theory and we don't have
the parts in terms of how do we actually
build this and actually apply this for
uh driving product impact so that was
the motivation uh you know it's it's a
long answer uh now to your other
question about like how did this happen
I think many initially initiated this
she was like hey like you know I have
this thought process about maybe writing
something and like YC was in static so
static was kind of helping us think
through this so that's how we started
but it was a very long process a lot of
brainstorming weekends and nights and
all that stuff actually we we were
always working on a notion doc or
whatnot right and then when we actually
physically saw the book that was just
super strange was like oh my goodness
this is like actually like real it took
a while but it's good to see this happen
>> yeah I actually want to share an
anecdote behind how the whole thing got
started. So actually two years ago I
caught with with BJ uh so the CEO of
Stasik and also right now the CTO of
open air apps. So I caught with him at
the event. I was like this year I really
want to write more things share my
thoughts and all that kind of stuff. Um
and then his answer to me is like why
don't we write a book together like a
book? He's like yes a book let's write a
book together. And then he wrote a very
detailed schedule for me about how to
write a book. And but look at scared of
melt. It's impossible. But that's why I
decided to actually have my good friends
who actually helped me out. Uh so this
is some anecdote behind it. What was
your motivation? And I'm also curious if
you had one particular anecdote that
when you think about why the industry
needs a book like this, what what comes
to mind?
>> I got to realize the industry needs a
book like this. After writing it, I knew
growth was important, but I guess to add
up my anecdotes. I remember VJ's
timeline was uh few weeks or a month per
chapter uh with our full-time job. And
the reason this book exist is because of
our ignorance. We didn't know how hard
it is. Uh knowing how hard it is, I
think the book would never get started.
Maybe this
>> don't tell people that.
>> If you have the right expectation can be
done, but it's very hard. I I feel like
it's almost like a startup, right? You
go in there with ignorance, but along
the way you try to do something amazing.
So I gravitated towards growth in my
career at Amazon. I was economist
building models but I was thinking my
value as a data person is because I can
do hard things. I can do modeling. Then
Facebook uh that got thrown away. I
remember like my first job was at build
logistic regression model or rental
force model and uh my manager now was
happy about how slow I was even though I
was much faster than I was Amazon he put
couple data points and wrote a note
published it tps and uh that was it and
I realized oh the value of data is to
make better decisions it's not about
doing hard things but uh why do you need
to make better decisions and uh in And I
realized growth is why we have jobs and
why the companies are earning money.
Growth is what's important. So I
gravitated towards growth. But after
writing the book, I realized I knew this
by talking with people by looking at
different notes. For example, taking the
Facebook analytics book or by making
mistakes. There was no book out there
that I can learn all this information
from.
>> Let's get into some of the secrets of
this book. Um, Nan, I'm going to start
with you. You know a lot about PLG.
You've been doing it for a very long
time. So, what do most people or
companies get wrong about product
growth?
>> That's a great question. Um, I think a
lot of people get it wrong when they saw
PLG just organic growth. They just rely
too much on word of mouth without
investing in marketing or even picking
up the good leads uh full of sales. I've
seen people who, you know, like, okay,
our is growing. we're going to let that
up girl but actually I think the real PJ
is like a fly like wheel. So basically
you create this kind of working model
from the early believers early doctors
and then you actually need the marketing
to actually to help to amplify that
impact to get to the people who wouldn't
have discovered uh you organically. All
these kind of leads like good leads who
become your champion and then you should
send them to sales and then they become
enterprise to you and on the other side
they actually one is it proves you know
your your product is inter is enterprise
ready and on the other side is also a
great story to actually boost your PG uh
funnel as well because people have more
multiple trust in your product. Uh so I
think that's probably one of the biggest
mistake I've been seeing people.
>> How do you know if you're doing it well?
What metrics or KPIs should we have for
how you measure whether we're we're
executing well?
>> Oh, that's a good question. How do you
define whether you're doing well or not?
I think really depends on the stage you
are in because through the whole product
development cycle, there are so many
stages from the MVP all the way to the
PMF like product marketing fit and all
the way to kind of like grow stage to
mature stage at different stage you want
to have different metrics to make sure
it can match your health. So, don't just
money copy whatever match out there
because I understand your business. I
think it's the most important thing for
you to measure. Well,
>> Joe, I'm going to move on to you. Um,
because this is a great followup and you
just mentioned product market fit
>> and I know you thought a lot about this.
We were talking about retention earlier.
>> How do you know if we have product
market fit even before the growth stage
and can you help us define it more
quantitatively as well?
>> Absolutely. I think book has more
details but we'll give a high level
playbook right I think even the
definition of product market fit has to
be thought a little bit like deeply it's
just because we've heard the term a lot
sometimes we just use it in passing
right we just say oh we have product
market fit but uh if you think deeply
it's not really like a like a
destination it's not like oh I achieved
product market fit I'm good no it's not
the case it is continuum um you would
have achieved product market fit but you
want to make sure you sustain your
product market fit right so there are
factors that affects it your product may
change and you may lose product market
fit your market might change and you
might lose product market fit so you
want to be very cautious about just even
the term right think deeply how do you
measure quantitatively I think again
retention is the king is how we say we
think uh that's the strongest indicator
for product market fit right I was
explaining actually Julie earlier I I
used to do this uh boot camp or
analytics camp at Meta for anyone new in
analytics and I used to do the retention
class and one of the slide I used to
have was around emphasizing the
importance of retention and I had three
quotes one was from Julie the other one
is from Alex Schulz and then from script
all the leaders at Madic they deeply
care about retention no one is going to
sign off on any product without looking
at retention so that's really the
strongest indicator for retention. But
the story does not end there.
Understanding retention and
understanding if you have a sustainable
growth, right? For example, you may have
good retention for and maybe a product
is used by 10 people. But does that mean
you have product market f you can't
claim oh you have great retention but
how many people you have 10? I'm not
sure right. Uh again the answer depends.
So you want to understand you have uh
some amount of growth there right? some
set of users uh and that really depends
on your business. Maybe it's a SAS
business and 10 users are great but if
you have a customerf facing product then
probably not right so you got to think
about that and then the third factor we
think about is understanding uh
engagement on the product right so to
see if there is deeper engagement on the
product and um that's a very broad term
it can be substituted with many things
maybe it's about having strong
monetization or maybe it's about having
engagement um so it really again depends
on the product but you can abstract into
these factors and put them through a
playbook and understand like okay these
are the metrics or these are the factors
I'm going to like look at to understand
or evaluate product market fit right and
then having metrics for each of these
and putting that together to to really
come up with a quantitative way to say
how is my product market fit but again
the important thing is to have uh some
consistency in monitoring this right So
you don't want to get to the state and
then just like move on. Make that as a
guardrail, if you will, uh to just have
that monitored over a period of time to
make sure you still have
>> Now for the million-dollar question cuz
I get asked this a lot. What is exactly
a good retention number? Is 13%, 20%,
what guidance can you give us here?
>> So I think there is no real magic number
uh for retention. It really depends on
your uh product. So we give few
guidelines to understand this. One is
like is your retention improving over
time an example you launch a product in
Jan maybe you know your retention is x%
and ideally you want to improve the
product experience over a period of time
so ideally in June your product should
have a much better experience and users
in that June cohort should have much
better retention than users in the Jan
code. So that's one way to understand if
your product has uh better retention. So
there are also benchmarks to simplify
understanding. There are a lot of
industry benchmarks. Say social apps
should have x% and commerce app should
have y%. So there are some benchmarks
that you can absolutely use. So I would
think of three factors. One factor is is
it improving over time. Second factor is
maybe like this industry benchmark.
Third factor is really understanding
within your product ecosystem how is
this feature or this product's attention
doing? Um would give you a better grasp
if you have good retention or not. But I
don't think it's a multi-million dollar
question to get the answer right. But
these are the ways in which you can kind
of get an understanding of like what
good means actually.
>> Yeah. Also I just want to add to this
because also the definition of retention
just bear so much because everything
based on active means and every company
has their own definition of activeness.
Some just think you think as long as you
log in is called active. Someone think
you have to spend that session of at
least 3 seconds that's called active.
There are a lot of nuances over this.
That's already really hard to compare
apple to apple.
>> You're the king of experimentation. Can
you give us some advice on when should
companies start experimenting and um
what's a healthy amount that is for
example the right amount of
experimentation versus let's say too
little or too much?
>> Oh, good question. Um I try to simplify
things because like I I know as data
people we hate oversimplification but I
think to easy to remember as a starting
point maybe it helps. So at sic we serve
B2B companies and B2C companies and we
have two versions of this story. The
first version is uh your sample size is
dependent on your expected uh impact. If
you're measuring 50% impact you probably
just need a sample of 30. But if you are
measuring 1% incremental then you need a
lot of sample and I think there's a
problem I remember if you have over
10,000 people in your sample then you
can measure something like one to 5%. So
that is oversimplification also. It's
also related to how many experiments is
a good amount of experiments. I actually
consider coverage to be a more important
metric than number of experiments. And
the reason is if you can cover 100% of
your new features, you cannot cheric.
The danger in doing experimentation is
uh people cherrypick. People only want
to confirm theirel and this becomes a
culture issue. So they only put up their
best candidates to experiment and
experimentation creates no value in that
case because if uh you expect this to be
positive, experimentation confirms this
is positive. Experimentation doesn't
create any value. The value is you
expect this to be positive and realize
it's actually negative. then you
actually need to think update your
mental model or update your product road
map. Uh so I do consider coverage to be
important and at Facebook we can almost
do 100% coverage especially on the
important big features that is not hard
to do but I realize most company do not
have the right uh infrastructure.
I I'm going to turn this into a static
pitch even though I left the company.
The technical insight is you need to
have feature flags and experimentation
as one system. So feature flags and
experimentation is the same object. The
every feature you use a feature flag you
get an experimentation for free. If you
have this setup, you can do 100%
coverage. But the system is actually uh
easy to start, hard to scale.
>> Okay. Well, I'm going to move it back to
you.
>> So you're an active advisor and
investor. So when you look at companies
and you evaluate founders, what do you
look for in terms of helping you
understand if they're data rigorous,
data informed, what what's a good
profile of a founder that you'd be
excited to back?
>> Um, that's a great question. So actually
for most founders I've talked so far,
most of four kind of on the extremes of
a spectrum is either it's like, okay, I
don't really care about data. I know
every single customer I have. I talk to
all of them. uh or on the other uh
selection, I want to lock every single
detail of my customer. I'm so afraid of
missing any details about what they're
trying to do with my product. And to be
honest, I think a lot of people think
data is just like a data problem. I
think that's a wrong way to think about
it because data problem is actually a
business problem. It's very important
when you think about what data actually
need uh what kind of an I want to do is
really around what kind of business this
questions you want to answer to begin
with and why this question is very
important for you to answer right now.
That's why in order to answer this
question that's why you need to make
sure you have the river in the data
analysis. But if some question is not
something that's super important even
the data is like trash it's probably
okay for now. That's just my opinion. It
sounds like it's just it's very
important to actually understand what is
this data going to tell me and what it
goes back to what you guys were saying
before about it's all meant to drive
decisions and if you can keep what
actions or what decisions you have in
mind we're going to do a better job of
understanding even does this data
question does it matter that much makes
sense why don't you tell us a little bit
more about data storytelling I think you
are out there you make a lot of content
you you evangelize you're very effective
at that how do you think about
storytelling as a skill for people who
work in data.
>> So I think there are two things like
storytelling as a data person and data
storytelling. They're not always the
same. I heard this opinion from my uh
manager and I strongly agree that the
two most under appreciated or
underdeveloped skills of a data person
is storytelling and making charts. A
strong charts just tells you all the
things you need to know and it's very
compelling and storytelling is this. I
don't remember how many meetings I went
into and if I put a table or if I put
charts even though I can make very
informative charts five minutes probably
after people walk out out of the meeting
and ask them do you remember my points
they don't I think start talking to the
human brain we just don't remember the
facts they might not that well but once
you can do it kind of story I found this
to be super effective on designers
actually whenever I talk to a designer I
always kind of try to screenshot the
user journey And then it became a pen
for the designer. If I just say this
conversion rate is 12% is too low they
don't feel anything but if I say oh this
stat is broken that's why I believe the
conversion rate is so low people fix it
immediately. So I think story drives
resonance drives empathy and drives
action.
>> Can you say more about charts and what
is a really good chart?
>> I think a really good chart uh says one
point and one point only. I see the
mistake a lot of data folks might make
is we try to overload information on
charts. We try to make a chart that is
so sophisticated it tells 10 story at
the same time. This ties to the action
part, right? Think about what kind of
action you want to inform or you want to
recommend and just have my chart. If you
have three saries or three data points
or conclusions, make three charts and
make it very clear and annotate. You
shouldn't just have a line chart and
don't say anything. have it very elegant
meaning uh the information you want to
present is very obvious and you don't
have any redundant information but also
use color use annotation use lines for
example if you are saying there is a dip
then point out the dip and point out the
trend so I think the strongly opinion
chart is a good chart
>> someone's like you know your story and
then find the chart that's the best
representation of the point you're
trying to make
>> the minute you said that I remember like
it was not comfortable for me as a data
scientist is to have opinions.
>> But I think a good data person should
have opinions.
>> Makes sense. Joe,
>> can you say more about what it's like as
a data engineer? Like how do you think
about what a really good healthy
engineering culture using data looks
like?
>> From an engineering perspective, it's
really like aligning the team towards a
specific goal. Um and then having a way
to track these goals and making sure
we're able to uh progress towards that
goal I think is is really key. Um which
I think uh I've seen that at meta being
done quite well that we have specific
goals and then translate that into a
metric and then focusing on that one
metric uh as a team. I think the builds
us up to make sure we are all focusing
on the right direction and not going in
different tangents. Now for that to
happen really like having the right uh
tools type is key right making sure it's
much easier for teams to be able to like
run experiments and then evaluate those
experiment results is important and then
really understanding like what's the
landscape of the product is important
right as an engineer we want to know
like okay am I on track to towards the
goal the product is is something that
the leader care about the manager care
about engineer care about everyone cares
about this one or three metrics. So
having these key data artifacts, having
a dashboard that kind of like helps you
like rally around is is important and
being able to understand the data in
deeper level by engineering team is
important, right? Like they they launch
a new feature, they want to quickly see,
hey, is this feature working or not? Are
people using this or not? Where is the
drop off? All of those standard
questions, you want an easy way for
engineers to be able to answer. I think
when you reduce that friction then it
becomes much easy for engineers to
actually use data to really even for
them to just navigate their way around
because when you remove that it becomes
really hard to actually access the data
and get information then people are not
going to be like motivated to actually
go with data but then if you actually
provide a system that makes it easier
and ask your team hey do you want to use
data to make decisions I don't think
anyone is going to like negate that
everyone is going to want to use data so
really reducing the friction
brings in or improves the rigor in terms
of using data.
>> I hear you say that it's like people
naturally are curious. People want
answers and they want to be able to
build the best products and if you're
helping to support an engineering team
>> being able to provide you mentioned
lowering friction for them so that they
can actually follow their natural
curiosity
>> and build the best things possible. This
is a question for all of you because of
course we're in the AI era and that's
changing everything. uh maybe for good
or for bad. I'm very curious to hear
about what do you think will change like
what are your biggest predictions for
how teams will use AI and how that will
change the way that people make
decisions or use data. Start with maybe
Joe.
>> Oh, sure. I can uh share some thoughts.
I think two different things. One is
even use a whatnot but eventually you
are building some product to serve some
users and grow the product. You still
need to have data. product analytics is
probably still going to stay and
probably going to grow even bigger than
it is right now uh for teams to launch
successful products right because now we
have so many Asian so many tools and
whatnot and I believe every product
still needs to understand their
customers I don't think there is a way
to substitute that what is one way to
understand your customer if you have 10
customers great you can go and talk to
them but probably you don't want to have
10 you probably want to have more right
one of the theme that we have is data is
the voice of the user right so we want
to have right data to understand our
user AI or nonAI I think that's still
going to stay but how is AI going to
influence data I think um that that's a
key question here right data consumption
is probably going to be disrupted a lot
using AI like instead of having so many
dashboards or so many like ways to look
at data it's probably going to be a very
different interface maybe a
conversational interface you're going to
have some companion who is going to kind
of help you answer some questions about
your product in a much more informed
standardized way. I think maybe we're
two hops away from it probably but
eventually I think that that's one uh
one path that we will go and I also see
a similar pattern happening in the data
production side of things right how do
you create the right uh like data
warehouse or how do you create the right
uh data models and schemas for your
product how do you properly log data I
think all of those today are very domain
specific and it's all based on the
strength of the engineering team but I
believe it'll eventually be like an
agent that will you know help understand
some of these patterns much better and
it will probably do the logging for you
and it'll probably help you build the
right data warehouse and so on. So both
in the data production side and in the
data consumption side AI definitely have
a strong influence.
>> Why do you what do you think?
>> May I have one more version?
>> Just kidding. I'm going to do two
minutes version. I have so much things
to talk about but I'll pick one that is
relevant.
>> Just look at the chart. This is I think
like teacher AI but for this audience I
think the most relevant thing is the
definition of data is very strange text
is data right but when we think about
data we think about a number but data is
not number data is everything AI just
enabled us to analyze or use that
nonquant
and it's more important why is that
because people are doing things much
faster now not only experiments takes a
lot of sample size But what about
retention? You need a month to get the
monthly retention and then you just get
one data point. So we need to have ways
to analyze non-quantitative data and
drive useful insights and I think we can
do that actually. So the role of data
scientists do not treat us as a number
or math scientist. We are like actual
data scientist.
>> Can you give some examples of
non-quantitative data that you think
will become bigger?
>> My first invested company is from my
friend. He actually was a head of growth
for a lot of company. He was also I
think the first 10 machine learning
engineer at Instagram as well. He spent
billions of dollar trying to grow. But
the company he started was about user
research. And uh when when I first heard
about that I was like oh this is
something that we couldn't do before AI
now we can do because we always want to
talk with customers but how to actually
extract useful information from all
those talks and how to in our forite
have smart trigger to trigger the right
uh feedback that is hard to do but now
it's easy to do well not easy but
possible.
>> How about you?
>> I want to talk about from probably
different angles. The first one is I
think all the data people right now have
the potential to become a real builder.
>> Oh thank you. Uh because to be honest in
my opinion data people always probably
the most technical folks outside of
engineering team. So because we we
understand even SEO right there are some
logic you understand for loop while loop
understand conditions that's all the
codes about basically right. Uh and of
course engineer know some HTML CS they
know how to build something fancier. uh
but now we have AI right so I think to
me uh because data team they have this
kind of very solid foundation of
understand the data understand the
business very well we are also kind of
the communication center of talking with
all the stakeholders we're actually in a
perfect place to become a builder uh
that's something I'm trying actually
experiencing at my current company uh I
recently just launched a feature into
production and I realized oh my god
that's how it feels like when you
actually keep on using your feature
that's how it looks like to become
engineer of course it's not that easy
But it's not impossible. So I think I
really encourage everybody here to
really try to get your hands dirty. Try
try something you never done before. Not
only on an analytics but also on
something else. Build a data feature for
your team. Right now I'm trying to
implement all the statistics in our
product as well. I build a feature just
by myself. I'm not just doing a mock
account. This is P value. This is how
the confis supposed to look. I just
build it. So I really think AI makes it
very possible. The second thing I think
is uh also it's a great time to build a
PLJ company. The reason why is also has
to do with AI. I remember last year a
lot of uh BC and like boys start think
oh we should have more forward
deployment engineer. I don't know
whether you have seen that on LinkedIn.
Yes of course humans are still very
important. They still should help the
customers get on boarding and be
successful and all that kind of stuff.
But to be honest, because right now AI
makes it so easy to understand your
docs, to give you instructions as long
have good docs, good demos, it actually
makes P way much easier. So a lot of
barriers you just don't understand how
to how to use this product right now.
Just ask and they will give you an
answer, give you a demo and you can get
started and just make sure you have good
dogs out there. The LM can pick it up. I
think that's the most important
foundation for you to build a good P
company these days.
>> Fantastic. Very inspiring. I think this
room is going to go out. We're going to
be builders, bookw writers, go out there
and um build the application. I'm going
to turn it over to the audience now for
questions. Um if you have a question,
raise your hand and I'll come send the
microphone over.
>> So, anybody who ask a good question,
we're going to hand over um a signed
copy of our book for free.
I'll do that. I don't know.
>> I'll send the mic right.
>> So, I know we talked a little bit about
sort of like DS Kabi builders. Um, one
thing that I've been seeing a lot over
the past couple years is that DS are
really good at building AI in
particular. And so, we talked about sort
of like the intersection between AI and
data science, but from the AI engineers
that I've worked with, those with a
background in data science uh are the
best. So I'm curious as people that have
spent a lot of time doing data science,
what do you think the like zeitgeist
needs to understand from the data
science perspective? How do we get more
people to understand that data science
is valuable for this new?
>> So the question is uh how do we actually
make you understand data science is
actually very important?
>> Yeah, why don't we tell that story?
>> That's a great question. Do you want to
start with the story to telling? I'm
actually not a good person for this
because I always tell data scientists to
stop being a data scientist to be a
builder actually because what we do is
quite indirect right and uh I think we
strive in large organizations with
complex information like people just
cannot understand data we have a
specialty of understanding it well and
we can make better recommendations and
indirectly drive up actions I don't like
that I want to just make the action
happen so I always tell data scientists
is to be builders but not the other way
around. I don't tell people that the
data scientists are important in this
state of change. I hope all data
scientists can directly implement the
change. They don't have to rely on
engineer to make the change happen.
>> So just to refram
as something who has some specialty I
think data and AI you should all treat
them as tools. basically the tools you
can use to inform your business decision
because again everything what are doing
here is you want to drive this growth
right you want to make money so whatever
you do data AI whatever you can have on
your hands just do it that's hello this
is a question for my you mentioned
earlier that PG doesn't mean you don't
do marketing right can you give some
thoughts in terms of what kind of
marketing uh you are talking about
because there is paid ads there is
influencer maybe investment in the
community because you have gone through
the notion in the early days what sort
of the marketing are more effective
especially in the early days can you
>> actually I think it depends on which uh
industry you're in for example notion
has benefited a lot from influencer
marketing especially during the pandemic
time when everybody is rolling oh also
our head of marketing of notion is right
there so if you're curious about
marketing look for I mean
and thanks for coming and so we learned
a lot from that also came did authentic
job writing docs about notion make sure
you understand how to use notion we all
know notion has a pretty steep learning
curve uh I think that's all helps and
also pilot actually on other hand they
did a lot of linking marketing and they
did so successfully they didn't spend
single dollar on marketing at all just
the co-founders they make post seriously
every single day uh and I'm very
impressed
And that's how they become successful.
And nowadays for example because of all
this kind of AEOG game and people start
building uh the kind of content just for
the models. I've heard some stories
people will have a hidden website on
under a domain where they just have all
this kind of thresh blogs written by LLM
to be read by LLM so that the search
algorithms going to be picked up. I
think there really depends on uh
different stages and different industry
and what you are actually good at and
also who do you have in the house.
>> Other questions. First of all,
congratulations again for publishing a
book. I have a question for
>> like I'm the target today. I should get
a book just for answer the question
>> when you're first joining uh notion and
then how to identify oh this is the most
important question if we want to do
Rob's what is that posture? Oh, that's a
good question. So, I think when I first
joined Notion, uh, of course, I was kind
of at loss about I don't know what's
going on with a startup just trying to
make sure I have a job. I was so worried
and got fired seriously because I just
didn't know whether everything I learned
from Facebook or Microsoft going to be
applicable in startup development. So,
in order to understand what was the most
important thing to do, I think it's
really important to talk with
stakeholders. Uh, it's really to
understand what's on top of their mind.
uh what is the things that really
blocking that that nobody else can do
maybe I have the edge to do it that's
why I identified experimentation so
that's how I identified actually
experimentation is one of the blockers
of people launching boot features people
always want to back in a while dry had
millions of users but they only run
probably fewer than 10 experiments back
then and then I realized maybe this is
the muscle I can help build within the
company and so I started talking with
successively with different vendors did
vendor shopping for the first time
myself, signed a contract for the first
time myself, did negotiation all by
myself and then launched the
experimentation campaign with the
company to really start building and
fostering that kind of culture.
>> Hi uh my name is Miranda and thanks for
hosting this. First of all, I find a lot
of things very resonate and myself I'm a
full-time data science leading a team
working on better influencer marketing
and the new AEO and GEO and I think
speaking of text and analytics I think
IC application as well. on even a lot of
like influencer social comment all of
those and I think I've also resonated a
lot on oh data science can be a builder
but I realize today a lot of line get
blurred so I'm curious about every one
of you what do you think could be the
next skill or the most important skill
for have data science stand out in the
organizations because right now data
science or data engineering lines
getting blurred
>> data science PM lines getting blurred a
lot flourish and everyone can do
everyone's job. It's like all hands on
the deck all the time. How do you guys
think about the unique or the best skill
that you think stand out among this
broad or competitive space?
>> I have a short answer then I think my
answer for the general population is the
same for all data scientists is agency
and taste. When you have agency, you can
do a lot of new things that other people
are not thinking about doing or just
hasn't been able to execute and taste is
ultimately what makes you unique and
makes you having better decisions. I
would add one thing probably curiosity.
I think that's actually the drive behind
everything. You have to be really
curious about what is my business doing
and you know like why this the numbers
actually going differently than I
thought or why this picture looks so
weird why we actually make our blog
looking this way why there's no
background there just be curious ask
questions and then even you're not the
only expert on this people will answer a
question and we'll tell you this is why
we're doing this or maybe they will oh
actually we never thought about this
maybe we should do it that's how
contribute
>> I think is it's still very very
important even though the lines get
blurry and whatn operate really taking a
step back and thinking from first
principles what does it take to solve
the problem and how do you approach and
build it I think is still going to be
critical
>> thank you
>> my name is Jerry uh my question is I
think people talk about the blur roles
and there's different skill sets that
data science needs to have now in the AI
era what's your prediction for the job
market for data scientists um I think
there's entry level mid-level and uh
going higher level I think going higher
level I guess it's all about decision
making and influence. Uh but what's your
prediction? It feels like the market for
all the tech jobs going to shrink.
That's my immediate reaction. It could
be wrong.
>> Well, I'm still hiring several folks.
Oh, that's a great picture on my job
post. So, anybody who is interested in
joining trust as a data center for go to
market or as an data analyics engineer,
feel free to talk to me as well to
answer your question. Um I do think so
one thing I would say is so these days
previously what I'm hiring I also look
at okay how many years of experience and
what things they have on the resume what
kind of companies they work at before
but this is I think more relaxed on that
to be honest uh as folks just mentioned
what I care really about is really the
agency the curiosity and whether they
can actually use the right tool to to
get the things done I think that's very
important so that's why like even in my
job post I ask folks to share a story
where they're able to just unblock
themsel proactively like even this not
the thing that's supposed to do or very
typical problem uh I think that actually
going to show me what this person is I
think that's very important especially
in the AI age again hold coding is not
the most important thing
>> I think evidently it does
disproportionately impact junior roles
as an example static put up a position
for junior data scientist which was
actually quite rare people do not hire
junior data scientist anymore we got I
think 30,000 applications within a day
and uh we can only go through 1050 right
it was not like two years ago so I think
there was an impact and uh
disproportional impact roles but
actually I invited Holly to another
panel and H answer to a similar question
once there was no data scientist 20
years ago data scientist is a invented
job title it's not like a farmer right
it's not like a toy cleaner even it's a
invented So I I do think the meaning of
our role is going to change. So whether
you are a senior or junior person go for
what problem you are solving for. Maybe
the senior person we are hiring for you
have a better mental model about how the
business should work. So you have better
judgment. Junior person maybe you are
more versatile. You can code and you can
do analysis and you have a statistical
training. So now analysis actually is
rigorous. So I think if you break down
what to make is valuable especially
because skills are getting cheaper as
well. So then maybe you have a better
chance of landing a job.
>> Last question.
>> So first my question is about uh I think
we have been talking about a lot about
using AI to help data scientists but I
want to ask another question. In this
era there are more and more agent
products coming out right. How do you
measure them? Especially if the agent
product is having it for example like
you know has a agent called roofers
right? So these are the traditional
business which already have its um set
of uh measurements like KPIs. So but
when they are launching a new agent
product how do you think should they
change the KPIs because if they measure
using the traditional KPIs you will see
the gains probably not worsening the
cost right because developing an agent
is very very expensive. So I think this
problem is getting more and more popular
around Silicon Valley really you want to
think about what does success look like
for the agent right and then translating
it to how do you measure that for
instance what problem are you trying to
solve and understanding again the
fundamentals I believe are not going to
change much I know there's a lot of
frenzy around like you know this agent a
lot of buzz words and all that stuff but
just to take a step back I think the
fundamentals are still fundamentals in
my opinion you still want to understand
your users you still want to have the
same measurement I don't think you want
to have some new metric discovered just
to understand agents and its success
essentially right. So end of the day
agent is going to solve some problem. So
essentially what does that problem look
like? I think even shity is still they
just put on like weekly active users and
so on so forth. So I think it's still
the same. Also, it just makes it uh I
think cognitively less stressful to
measure success, right? Because we've
been uh used to a way of understanding
success for a product and measure them.
I'm sure there's going to be a few
things that might evolve, but in my
opinion, still understanding a users,
understanding the fundamentals and
having standard metrics would still
apply. I some of the question was about
AI evaluation, how to evaluate uh the
quality or effectiveness of an agent.
But the second half of your question, I
was wondering why do you need that agent
at all? If you don't see the business
outcome positively increase, I think the
reason you want a agent is either you
can save cost, do it faster or you can
do new things that you couldn't do
before. If you didn't any new things and
there's no PPI, there's no improvement.
Don't use agents. Why are you using?
>> Yes and no. I would say because you
still see all those big companies are
still pushing for every organization's
goal.
>> I think there is an investment part of
it. People are afraid of not catching
up. So we pick the investment of uh
let's use AI for the sake of using AI
even though it may not be effective but
we treat it as a invest investment for
the future. I think from that and golden
treated as investment.
>> All right. Thank you. Um, let's give a
big round of applause to uh, thank you
everyone for joining us tonight. I think
we'll all stick around for a little bit.
And is there going to be a signing? Hang
on.
>> There's no signing. We're going to head
over the book to the folks who just
asked great questions and we're gonna
share some cake.
>> Fantastic. All right. Well, we will stay
around for a little bit longer and we
can continue conversations in small
groups. All right. Take your run.