Subscribe
Share
Share
Embed
课代表立正的官方Podcast
深度访谈，有用干货，亲身验证的「真本事」
Superlinear Academy创始人，Maven Top AI Instructor
前Statsig布道师（OpenAI收购），腾讯副总监，Meta，Amazon；康奈尔经济学博士
社区：Superlinear.Academy
课程：ai-builders.com
个人：lizheng.ai
all right it's four o'clock now let's
get started hi everyone this is Yen
welcome to the lining lesson for the
course from users to builders AI powered
productivity for T
RS first several Logistics here this is
a half an hour Zoom meeting we will
spend like 20 to 25 minutes on the
lecture and then use the remaining for
the
Q&A um during the lecture feel free to
put your questions in the zoom chat
window and then use the emoji react
reactions to vote for others questions
we will answer the most popular
questions in the end the meeting is
recorded and you will receive an email
from Maven about where to download the
recorded video and
audio now let's jump right into the
topic AI is the new buzz word everyone
is talking about it and big companies
are pouring money in it but if you
really use chbt you may find actually
quite often that chpt is dumb it's lazy
it forgets about things it didn't do
what I asked you to do so we become
upset sometimes painful and curious why
is others char so smart and so
powerful well you're at the right place
this is because we used it wrong and
it's not our fault to some extent chat
gbt is flawed in design this product is
designed to support and encourage
conversations or chats it makes sense
because that's the most intuitive way of
interaction however this form of product
would naturally lead to certain usage
that makes gen forget about things lose
track of details and appear dumb and
lazy in this lesson you will learn about
a core concept from relevant research
that can explain this all it will help
you make sense of the unexpected and
undesired behaviors and naturally leads
to a counterintuitive but simple trick
if you use it you will find the AI to be
smarter and help you better on most
tasks
immediately but in order to do that we
we can't treat chbt as a blackbox
anymore we don't come up with a random
explanation or try out different tricks
and wishing it to work in this lesson we
will tackle the lazy and forgetful
problems of AI systematically beginning
with understanding what is inside
trbt not trbd is still developing very
fast we hope what you learned today will
still be relevant in a few years so we
need to go to the
research the most relevant research of
chbt began about six years ago it began
with a foundation model and then
alignment was added to make it something
similar to today's CH gbt in order to
effectively use this tool we also need
to add proper prompting and a solid
understanding of context Windows this
four are most practical and fundamental
components of using
chbt understanding each component will
help us um will make our AI smarter and
our work more effective in the full
course we we will go over each of them
but today we only have about 30 minutes
so we'll still try to do it but focus on
one thing the contact window because it
can help with all of your prompts it's
often overlooked but it's Central to
explain and correct a lot of the
problems of
jni after understanding the concept we
will then introduce a simple but very
effective trick to effectively manage
the context window you may be familiar
with the G on the left which is often
the case for people not managing their
contact window it gave short answers not
willing to help but with the same model
and even the same prompt after we use
this simple trick the gbt immediately
becomes more willing to help and even
smarter this is the power of contact
window management and let's begin the
journey by first understanding what is a
contact
window contact window is a concept
introduced very early in original GPT
paper it has its meaning derived from
the original along with time even in
open eyes on documents but in this
course a contact window is simply what
trpd sees when it generates
responses what does that mean let's take
a look at basic
example the simplest way to use trbd is
you ask it a question and gives you an
answer in this example it ask it I have
many large files to copy from one
computer to another computer what should
I do this sentence is my question and is
also what chpd sees when generating the
response therefore in this example this
sentence is the context window again
this is a probably 90% but not 100% the
same as original definition in the paper
but in the context of this course a
context window is what chbd sees when it
generates
responses we just saw a simply the case
of we ask a chpd one question things
begin to get more interesting when chpd
generates a response and we ask a
followup question I cannot upload the
file to the cloud storage what are my
options in this case the the context
window will change to the re red
rectangle it would include the chat
history and latest the
question this might be a little bit
counterintuitive for some people so
let's stop here and make a clarification
on a common
misconception trbt is a good product it
gives people an impression that we're
talking with a human so it's natural for
us to imagine oh chbd has a memory
storing its understanding of the world
storing our chat history our requests
and so on and then it reads my question
and tries to answer it by recalling from
the memory and reasoning based on
information in this case the context we
though would be the latest only the
latest question not including the
previous
conversation however this is a natural
and common
misconception open AI here uses a clever
trick to make chb appear to have memory
under the hood what happened is open AI
includes the chat history as part of the
context window and ask gbt to answer the
question based on all the information
including the historical conversations
and the latest question in other words
chat gbt implements memory by appending
chat history to the context window when
we ask CH gbt a new question what
happens is the answer from the last
round and the latest question was added
to the contact window nothing else no
memory no database no state so in the
example above the contact window would
be my first question tr's first response
and my second question
this is a very important clarification
actually it immediately causes two
problems the first problem is that the
contact window is finite chat gbt or any
gen so far can only process a limited
number of
input GPT 3.5 had a limit of 4K tokens
which is about 3,000 words and gradually
expanded to 16k gbd4 had a limit of 8K
tokens and gradually expanded to 120 8K
but still kept a limit of output window
of 4K tokens these contact Windows
appear large but for certain tasks they
can be quickly consumed and causing
issues for example here the user asks
chbt to write a Python program with an
actal requirement that for some reason
don't use the library of npy the AI then
outputs a Python program which can
easily cost hundreds sometimes a
thousand words of tokens
then the user did a few rounds of
iterations to ask the AI to add the
feature of A and B up to this point the
contact window may still hold entire
chat history showing the red rectangle
but assume that it's close to the upper
limit of the context window in this
situation then the user asks can you
also do c in the program then in order
to still fit the contact window and
focus on the latest and potentially the
most important requests try to be PD
drops the first conversation and moves
the contact window down to only keep the
latest
conversations and this causes a problem
the request of I cannot use npy is now
not in the context window anymore and
chbd has no knowledge on that anymore so
it's totally possible that in the latest
response it may begin using numpy and
from the users perspective trag fails to
satisfy our requirements it forgets
about things it's not as smart as the of
the conversation this is all caused by
the fact that the contact window is
limited and it cannot accommodate
infinite
chat the finite size of the problem of
the context window is easy to recognize
and mitigate when the conversation is
long and AI appears less smart we can
consider restarting a
conversation but a lot of times even
when the conversation is short we still
feel the AI is lazy and dumb and that's
because of another more subtle but
impactful limit gbt has difficulties
processing long or or unorganized
content for this long or unorganized
cont contacts Windows gbt may miss
certain requirements especially those
need attention to the details in the
previous example even if the entire
conversation can fit into the context
window it is still not uncommon that you
gbt may give a a codee doing a and c and
forget about B one way to understand
that is we can treat gbt as an intern
that have limited intelligence when it
spends the bring power on somewhere it
has to spend less power somewhere
else and in this example because the
requirements are scattered everywhere
Sean as the green texts gbt needs to
spend actual Intelligence on recognizing
what are the actual requirements from
this long and Massy taxt and that
distract it so it couldn't spend as much
Intelligence on the actual problem
solving and this is just the one side of
the story another potentially even worse
side is the majority of the contact
window is actually bad answers we were
not satisfied with gpt's previous
answers and that's why we further
iterate to add new requests so from the
perspective I probably answer latest
question all the previous answers are
useless or even wrong to some
extent this again distracts GPT it needs
to figure out oh this taxs are actually
wrong where are the problems and how to
fix them as humans we all know that this
is sometimes even harder than getting
something from scratch so overall it is
this scattered around the green prompts
and the long distracting red promps that
degrades gbt's
Intelligence on unfortunately this is
even not the end of the story when we
see AI BEC dumb and lazy what is our
first reaction we chat more give it more
requests and hope this could correct the
problem but now you understand the
context window and its limits you can
see how this is not only not helping but
actually make things
worse because now we have even longer
chat histor to put in the context window
which means it's more likely to get cut
off and more distracting the
requirements are further scattered
around and the bad previous responses
are even longer that makes everything
worse so it's not self correctable in
the case of AI becomes dumb or forgetful
following our first reaction of chatting
more will make it even
worse I'll pause here to review what
just happened we now learn from L
research that the contact window is a
core concept deciding how smart CH GPT
may appear and how effective we could
use it based on this concept what
previously appears frustrating upset and
even weird behaviors now suddenly begin
to make sense when we chat with chat gbt
is nothing more than appending our chat
history to its contacts window and more
chat ends up with a Messier contact
window which will result in a dumb or
forgetful or lazy AI our first reaction
is to chat more to correct it but it
simply doesn't work it's not really our
fault
is mostly rooted from how chbt is
designed it's designed to encourage us
to chat and that inevitably leads to
this
situation but does that mean that we
have to live with it not necessarily
before touching the actual solution
let's detour a little bit to learning to
drive for a moment when we learn about
driving in many cases our first reaction
is actually wrong for example when the
tire blows humans first reaction is to
hit the brake
but it's wrong the best practice is
actually to live it alone and gradually
decelerate the question is how can we
know what is the best practice and how
can we really do that in real
life the answer is simple through
training training tells you how a
technology Works internally tells you
what's the best to practice and why and
helps you practice until make it a habit
until you remember it in your muscle
memory and it also applies to chbt to
properly use chbt we need training too
we need training to correct our firstly
reaction and use good practice to
replace
it and for our specific problem the
solution based on our previous
understanding is we need to manage our
contact window
intentionally instead of blindly
accepting trb suboptimal design we
should proactively think about what our
contact window should look like is it in
the best shape for G to work well if we
follow this mindset it will be easy to
come up with a much more effective
prompt or context window for example
simply put all of the requirements in
one paragraph with nothing else write a
Python program to do a b and c I can I
cannot use
npy but how do we do that does that mean
that we need to start a new conversation
every time when I need to chat that's
not stupid fortunately we have a hidden
feature in the chat gbd UI helps us to
do this contact window
management and that is this small pencil
button below our question it means edit
we can click here to edit the current
prompt and there will be a button say
save and submit clicking it will change
this prompt in the context window rather
than append it to the context window
let's take a look at example here a side
note is we are using gbd 3.5 here
because it has a smaller contact window
and it's easier to trigger those
behaviors gbd4 has the same issue and
benefits from the same trick it's just
not that friendly for this 30 minute
long lining
course come back to our example it's
following the previous example of
copying files from a computer to another
here we further ask chbt oh I cannot
upload the files to the cloud storage
what are my options chbt gives a very
short answer without any detailed
instructions but if we use the addit
trick not how we change the prompt here
we simply copy paste the second prompt
after the first prompt making it a
complete request oh I want to copy files
from computer to another and I cannot
upload the files to the cloud then
suddenly Char becomes more diligent and
intelligent it gives details and
specific and organized answers this is
very different behavior from before and
note here we still use the same model
even the prompt is the same
what was changed is the content of
contact window we intentionally make it
precise and brief and this is the core
of context window
management we can continue the
conversation back to the chat style if
we further ask chbt following the
previous conversations I'm copying files
from a Mac to a PC what shall I do then
chbt gives a bunch of solutions the
third of them is clock storage and
actually we literally just said I cannot
upload files to the cloud in the
immediate previous round stupid right
how to fix it you tell me contact window
management instead of relying on chbt to
append everything and construct its own
contact window we constructed by
ourselves simply append I'm copying the
files from a Mac to a PC to our previous
prompt and that's it gbd suddenly become
smart just like a different Ai No cloud
storage anymore
so let's reflect on this example why did
it work what is the internal mechanism
so that I can use it somewhere
else in the chat style we have the asks
or requirements scattered everywhere and
GPT needs to spend its intelligence on
recognizing and understanding the
requirements but in the added style the
context is well organized with claim
statements on what needs to be done so
GPT could save the intelligence to
really focus on solving the problem in
the chat style Cha's imperfect or
incorrect answer becomes a major part of
the prompt and that's both a waste and
sometimes a distraction of the context
window on the contrary this is not
included in the edit style and we have
efficient use of the context window
meanwhile thanks to the editing feature
we still keep the same ease of
use actually there's the bonus benefit
of the editing style that is the
reusability when we write prompt to help
chbt accomplish some task The Prompt is
not only to explain what we expect from
it but more importantly to correct its
undesired behaviors for example due to
some reason we cannot use numpy in the
company's computer but trb likes to
solve the problem using this Library so
we need to include this requirement in
the prompt however when we use the chat
style it's nearly impossible for us to
use the prompt because such kind of
Correction of unexpected behaviors are
scattered all around the place across
many conversations but for the addit
style use we always have the latest and
greatest The Prompt at hand summarizing
all the requirements and Corrections
ready to use this is especially helpful
in AI assisted programming which we will
cover in the full
course and what we just experienced is
not only a journey to effective use of
chbt but also a great example of
training before training we Face a lazy
forgetful and dumb AI rely on our first
reaction to use it and blame open AI for
when we see unexpected
behaviors after the training we
understood how it works internally we
made made sense of all those behaviors
and learned about the best practices
which is addit not chat the training
lets us know the best way to use chbt is
actually to not chat and in the full
course we will also help you to
exercises to form a habit to make it a
muscle
memory overall the general principle is
for more complicated tools you need more
training it applied to bikes cars planes
and AIS and you need people who knows
the ins and outs and people who know how
to teach to train
you in the full course we will also go
over different components of chbt and
have a comprehensive overview on how to
prepare yourself in terms of mindset
knowledge and practical tricks like
addit not chat we already had the first
two cohorts sold out and the current
cohort will begin from July
22nd all right this is it for linning
course hope you enjoy this and I'm happy
to take any questions let me go over the
chat window to figure out what are the
questions about uh Google so the
question is didn't Google recently void
this issue with the infinit window model
uh yes in the recent I think that's in
last a few days Google proposed this um
but it's my understanding is still in
the pretty research phase after it it
gets into production hopefully we will
have less burden in manually managing
all this context window but before that
this is a pretty easy easy to use and
effective
trick uh and I saw that some of the
questions are already answered and Eugen
is also online to help answer the
questions and the question could you
point us to some resources where we can
dig more into the foundation of L
understand more critical Concepts like
concept Windows this is a pretty new
field and I will say most of resources
online is pretty not unsystematic and uh
at least for this field the this this
makes us painful and we like to have a
systematical overview of what was
important from our perspective so to
direct answer Nick your question um the
best place probably is our our course
but if we see some other resources they
are not a very systematic one on top of
my mind um if I see in the future
probably we can post it somewhere um
such as in the community of our course
what is the target audience of this
course what can we get out of this
course in the end if we are uh talking
about the main course the target
audience of this course is uh the
Professionals in the IT industry by that
I mean we need to have some not basic
knowledge of python such as we know how
to you need to know how to run python
programs you need to know how to read
simple python programs and install
python libraries you don't need to be a
professional developer um for example
PMS TPMS engineering managers uh
analysts the or the target audience of
this course um but we do require some
basic python knowledge so if uh you're
out of the it Prof industry that might
not be the best course for
you and what we can get out of this
course in the end is the general idea is
Gen gen is a pretty effective tool to
boost our productivity but it's actually
there are a lot of petful and nuances as
we just demonstrated in this process if
we understand better from the foundation
and know about the best practices this
will help us a lot in boosting our
productivity my personal experience is
Boost like two times to five times of my
productivity and we' like to also share
this with our the course audience that
is the target outcome of this
course so on us sandbox versus chat
interface that's a great question I I
actually I think using sandbox is a
pretty good way of using uh the gbt API
or chat gbt uh some background is
sandbox is a tool provided by openai
it's developer oriented instead of
showing you a chat interface it in
directly invokes the underlying GPT API
so it's a little bit different from the
chat interface in a few ways
uh difference one is chpt is a product
built on top of GPT API so it has quite
some additional prompts and limits and
features but the sandbox doesn't have
that which means you may not have the
alignment or Safeguard of chbt or other
limits of char BT so you may be able to
get into the dark side of GPT which
means it may not be that safe and the
second is second difference is there is
also some limit POS on the CH on the
chbt for example gbd4 has a contact
window of
128k but in chbt you cannot use that
long before that you will be cut by Char
saying that oh your message is too long
but in the sendbox you are able to use
that the third difference is the uh
maintenance of chat history in chbt web
interface chbt will maintain the context
window as well as the chat history for
you but in the sendbox a little bit more
manual process
which is good thing and from the
perspective of of this course because it
forces you to manually manage the
context window and the fourth uh
potential difference is the price charb
is a subscription based $20 per month
but sendbox is uh per
token so overall it's a pretty
interesting idea to use sendbox I if for
anyone having haven't tried that before
I encourage you to try that it's quite
unique and fun experience and that might
be the right solution to to the context
meal
management do we have an agenda of this
course yes uh in terms of cohort dates
we have that in the latest slide and
feel free to also scan the QR code so
that you get to uh the cord homepage and
know more about the course such as the
uh the the uh syllabus the expected
outcome target audience we have every
everywhere we have everything
there s on using iterations of a highly
structured prompt by edit and or having
start by asking questions to improve
context of prompt then incorporating
into
prompt um that's more about prompt
engineering that's my understanding of
this question um and we do have some
tricks around that but that may be out
of the scope of this context window but
overall yes having the prompt formulated
in a more structured way is definitely
helpful for making trbt smarter as we
mentioned before tra is like it's pretty
much like an intern actually I really
like this analogy it's it's energetic it
doesn't know what is tiest and it can
work 247 but it needs some handh holding
and this kind of PR engineering is a
great example of handholding you cannot
assign a very complicated task to an
intern and expect him or her to
accomplish the task perfectly you have
to do some handholding uh and
potentially even babysitting and prompt
engineering and contact window
management are very good examples of
star
Point um and what might be a more
beginner noice oriented geni
course that really depends on your
background and the intention of what do
you want want to learn and my suggestion
is actually if you want even more
beginner or novice um course probably of
course is not the best option directly
go to the the chat.com or other similar
products begin using that and begin to
feel the capability envelope of geni is
probably the best um thing to do to to
begin with because this will give you a
feeling and
then you also give you some questions
with the questions it will be easier for
you to figure out oh what I want from
this a course like that what are the
pinpoints what can the AI bring to me
what I I think getting those questions
might be more important than figing out
the answers so to directly answer a
question probably the best course is
sign up a account using chbd from today
it seems that the price of course
increases per cohort will that be a
trend um it sounds like a you already
answered
that what are your thoughts on the
content of the course potentially
becoming outdated in mod models iterate
for example context window and
information infinite window model that's
that's a great question uh sounds like a
e already addressed that but generally
the idea are two folds the first is the
content of course can be be basically
divided into two things one is the
underlying research and another is based
on those research that derve the tricks
and skills um for a lot of fundamental
things they will not change in a long
time for example the initial com initial
gbt model has been proposed like six
years but it's the basic PR principle
Still Remains the Same and another side
is we do see this field is growing very
fast that's also reason why ourselves
are Inus as in Inus as of this field and
we keep up with the field every day um
we try to figure out what are the most
important tricks and principles and
distill them into the course to make it
up to
dat um due to the limit of time we will
take the last question here um Yen have
you played around cloud. a pick out of
curiosity yes I played with it and it's
it's quite interesting uh in terms of
two aspects the first is I found that
the difficulty of managing my contact
window is much less for cloud V3
especially Opus more more um
specifically for a lot of prompts and
contact Windows gbt may be lazy so we
need to do more tricks to make to like
massage those prompts and contact
Windows to make it work but for cloud V3
we don't need to do that tricks from
this perspective we can see a trend that
people also realize the problem with
contact window and Make it try to make
it more user friendly and from another
perspective I also feel gbts 4 is still
a little bit smarter than Cloud a floud
V3 but that may be just my feeling based
on limited
tests so due to the limit of time uh
that's it for the Q&A and really
appreciate everyone's time and interest
for attending this course we look
forward to see you in the actual course
thanks a lot bye