Subscribe
Share
Share
Embed
课代表立正的官方Podcast
深度访谈，有用干货，亲身验证的「真本事」
Superlinear Academy创始人，Maven Top AI Instructor
前Statsig布道师（OpenAI收购），腾讯副总监，Meta，Amazon；康奈尔经济学博士
社区：Superlinear.Academy
课程：ai-builders.com
个人：lizheng.ai
okay it's a good time to do some
exploration on the application of
agentic AI in finance here we'll go over
a series of demos to build some back
testing platform which is uh very
fundamental but important topic in
trading
firms and we will probably go over the
data downloading platform building
strategy development and back testing
processes all right so we are at the
cursor interface we have nothing really
in our workspace and here we have a
screen set up with the cursor on the
left side and a tool written by me on
the right side it's a realtime speech
Rec recognition tool where you could
just dictate the prompt to it given
today we will have quite a few long
prompts it may greatly accelerate the
process so we can click stop and
immediately get the speech recognition
results so as the first step we could
explore some data downloading it's
pretty similar with our previous demos
so here we just adct dictate to the to
the
AI we try to build some back testing
platforms but as the first step we need
to collect some data could you do the
folling for me let's download the stock
data for say Nvidia meta and apple in
2024 and save them as a CSV file The
Columns of the CSV file may include say
date
ticker open close low high and volume I
only need a daily level data for
2024 after downloading please also do a
visualization so that I could easily
cross check the correctness of the
data after that we just copy paste this
into cursor and note that we are in the
composer mode this little switch between
normal and agent are an agent here we
choose claw 3.5 but feel free to use
latest 03 mini which is also compatible
with agent mode then we click submit
cursor will then do the analysis on this
first it realizes according to the
guidance from the do cursor rules file
oh probably I need to do some planning
because this is not that easy task so it
performs some planning this is the
current task download and viiz stock
data and steps it decompos it into four
steps which is pretty much the idea of
chain of thoughts and then Mark the
progress here so that it has the
motivation and the
the pr good practice of reporting the
progress which is good then the next
step is Claud then write a um python
file called stock data config the
requirements create a new VMV virtual
environment install the requirements and
run the data Note that in this entire
process I didn't do anything it's fully
autonomous and after this it seems that
the download has has completed if we go
to this CSV file we will be able to see
that okay there are those columns the
dates looks correct and the ticker also
looks correct this is some crosschecking
just as if you have a real human intern
you would do the same thing and the AI
also provid provides a visualization
about the stock price the range the
trend and the xais or look Sim uh pretty
reasonable so I think it it did a pretty
good job in collecting data and now it's
a good time to move to the
implementation stage of the back testing
platform now let's think about
implementing a back testing platform it
has two inputs the first input is the
data which is the collected stored in
the CSV file and the second input is
some alha strategy here we're interested
in day level prediction that is the
alpha would predict the profit for the
next
day and the simulation would also be
performed on The Daily level to simplify
the problem setting we make two
assumptions the first assumption is
Let's ignore or the transaction cost the
weird stuff like a slipage we just don't
consider them and the second assumption
is we have a very simple um position
sizing strategy for example when we have
when we have an ARA input Alpha we just
do a clipping to make it within minus
one to positive one and then we do a
linear scaling to the capto hand you can
assume that we have $100,000 in the
beginning of the year and in the end of
the year you need to report some key
metrics reflecting the um return
volatility and stability of the strategy
and your task is of course you need to
implement that and second is in order
for me to easily check the
correctness let's use some dummy
strategy to verify the correctness of
the platform for example Buy and Hold
and um just just do
nothing and note that we have the data
in the data we have three stocks so
potentially you you do want to um apply
the strategy on all of the three stocks
and do the
average all right so we get a very long
prompt here it seems that we don't have
quite very obvious speed magnation
mistake so I'll just uh click submit
here
then it says let's plan it first which
is a good practice according to our
experiments this is this good practice
will dramatically improve the
effectiveness on especially solving
complicated problems this helps the AI
to stay top on the plans and real really
make uh reasonable movements across
different steps requirements input
assumptions steps in progress all right
then it takes some time to build the
back tester. pii we can take a
look and note that our position here is
AI manager we are in the position of
crosschecking the correctness of whether
AI did some the ai's job assess its
quality rather than really diving deep
into the technical details this is the
key to make us really scalable
okay so it seems that the AI already
finished implementation it performs some
back testing which generates a back
testing
results and give us a portfolio value
over
time Buy and Hold strategy for strong
performance okay looks like it gives us
um three kinds of metric one is the
return one is the shop ratio one is the
max draw down that makes
sense I do have a concern though that
the performance looks too good to be
true let's ask AI because this at least
rings a bell in my brain whether this is
really the case can you double check
uh we could use this can you double
check the implementation is correct that
strong performance of such a simple
strategy sounds too good to be
true all
right ah the relative relative
return calculation was incorrect it was
was relative to the current days Capital
not the previous days
Capital then let's see whether this will
fix this
issue so this is an example of how we
could cross check ai's quality um a lot
of people have concerns that oh the AI
may may may make mistakes May
hallucinate so it they are hesitant
hesitating on whether to use AI but one
um argument is
that this is not something new to the AI
actually if you have a human intern you
will do the
same interns also make mistakes but the
key is manager need to be responsible
for the final result final deliverable
so that it's mandatory to put in some
mechanism to double check the
quality okay it says oh it turns out
this is actually
correct because
all of
the
stocks did pretty
well huh it looks really the case for
the Blue Line it's like starting from
50 and ending about 120 more than 100%
and apple it's not two times but still
pretty good and for the meta is from 350
to more than 600 wow nearly 700 now so
so it's indeed the case that if you just
a Buy and Hold You could get really good
gain okay it passes the rational check
okay then let's compare some more
classical strategies for
example good now I'm convinced that this
implementation is correct now it's the
time for us to implement three classical
strategies for example one 5day momentum
Factor two 5day moving average crossover
three 20-day moving average
crossover and I not only want the
reported metrix but also need a re a
report in markdown format with embedded
visualization so that it's easy for me
to understand the result and comparison
contrast between different methods
different strategies we also want um
some analysis provided by you which me
Inspire
us so we'll just copy paste here no
speed recognition mistake click
submit then it first decides to
implement the
strategies meanwhile we could just click
the accept
all on
this and here's the
strategy momentum 5 day a moving average
crossover all right and it provides a
new script acting as the main function
to invoke this back testing process
all right and now let's run the analysis
and it looks like there was some error
so the AI moves the implementation from
one file to
another also remove the old one it's
interesting that those kind of technical
stuff are not done by us it's done by Ai
and if you take a look at all of the
instructions or prompts we give to the
AI it's all about whats all about work
quality problem definition scope of the
project this kind of high level
stuff and now the AI uses run strategy.
pii produces something but the it looks
like it hits some issue that the daily
return is not a number
so the strategies are not getting the
historical data need to calculate
momentum uh let me fixing back tester
this is another example that we don't
need to care too much about the
debugging but can focus on defining the
problem defining the whats rather than
really thinking about the
house okay still it needs some time to
fix
this
and this is as we said before a mindset
of AI
managers actually to the opposite of
what a lot of people
believes the key to using AI is not to
be super technical it doesn't really
matter that you know the ins and outs of
Transformers what is mad attention and
how to fine-tune a model it's not it's
quite the opposite in order to
effectively use AI you really need to
understand the um how to be a good AI
manager how to communicate with AI
Define a problem when to jump in to
unblock Ai and when to check the quality
to make sure there is no hallucination
those are the most important skill set
rather than knows the ins and outs
because most of us are not hardcore AI
researchers we are just users of AI and
it's our learning that the the most
important thing to use the AI
effectively is actually the manager's
mindset and here we see that what we
while I was talking I was still looking
at what AI is doing it's quite
interesting it sounds like the AI
actually um did some experiments and saw
something really weird unexpected so it
decides to put in more debug information
this is a good mindset it's like do not
just um guess randomly guess where is
the issue let's put in more debug
information and use this information
to figure out where's the root CA to
localize the
problem and it says it it found several
issues so this is another another aspect
of what we just said so on one hand a
good manager needs to be really focused
on the high level thinking make sure the
direction of the project is correct on
the other hand when your team member
whether that's an intern or an AI is
getting blocked is also your
responsibility to unblock them and I'm
currently still observing I'm still
trying to see whether the AI is getting
blocked or is it making progress in the
case that it's getting blocked I do need
to jump in and help it out
still doubling the portfolio
value to the
subract actually according to our
previous analysis it may be expected
that the Buy and Hold strategy is uh
doubling the portfolio value in the end
of the year but in the case that you see
it in the beginning of the year that
might be
problematic so this is some high level
uh prompt to make sure it did the the
correct cross
check okay and it looks like
still p hold immediately on day one all
right so this is definitely wrong
yeah and we could actually also check
the some intermediate results
yeah this doesn't look right trading
day and a side note is in the case that
you need to calculate the 5 day or 20day
moving average you may need the at least
the five or 20 days data so it might not
be that surprising if you if you have
like zero position for the first few
days you can can potentially make the
simulation
longer yeah it looks like it's it
realizes this issue oh insufficient
history so it's it's more or less the
same as we just
mentioned and I do think we just saved
it um somehow because it's sounds like
the AI was blocked in some intermediate
debugging code for from some time on it
puts a limit on the simulation time
within 3 days and that actually prevents
the calculation of several
features and now it adds uh some warm up
dayses now run the analysis again to see
the final results
okay it looks more reasonable
now and it is doing some final
Improvement let's see whether we
have some more data yeah it looks
good and where is the
report this is strategy report and we
could use the open preview to check it
okay Buy and Hold this is definitely not
correct I believe this is we haven't
really Rong the script
yet okay it now tries to
check okay it tries to check this um
correctness it looks like there's no
issue and now it tries to um do
some something that a good employee
would do but now due the limit of time
we we don't want to do which is to
actually improve the strategy um but we
also need to fix this report because I
believe it's not up to date
yet now the final ask I I think what you
what you have done is awesome um my
understanding is we've got a correct
implementation but feel free to correct
me if that's wrong now the problem is I
don't believe the strategy report was
updated can you update this report
potentially either using your code or
using your
intelligence and include all the
generated visualization and Analysis and
tables in
it okay strategy report. empty
so it turns out it tries to use
the like manual stuff but one limitation
is this is prone to hallucination so we
really wanted to use at least some
numbers from the real strategy to
it uh one ask is could you potentially
use some templates but fill in the
actual numbers of the say return and
other metrics
into this uh document to prevent any
potential
hallucination but you can still
definitely do your own analysis
somewhere all
right I mean you could potentially build
a script uh build a template and use
the code you just wrote to update
it all right now we get a pretty
detailed
report and I still want to really want
to see the visualization can you include
the visualization in editing the
report but let's review the report while
the AI is doing its own stuff it has
overview generated date performance
summary which I believe should be
correct because we are generating from
the code but in in the reality it's
definitely always a good practice to
inspect what's going on there Buy and
Hold and positions just a or spot check
this is good because it allows as humans
to indeed check what's going on there
and
realizations all right now we have this
we have a bunch of realizations with Buy
and Hold strategy with different return
daily return for different strategies
and Signal Evolution so I would say this
is a pretty comprehensive and useful
report or different the cumulative
return comparison value comparison so I
would say that's pretty good um let's
review what just happened we started
from a very simple request of
downloading data and then build the back
testing platform then we ask you to
implement three um alha strategies and
get the them implemented back tested and
cross checked in this entire process we
only care about the problem definition
and scope definition rather than the KN
house but just like human interns our AI
may get stuck at somewhere so we do need
to um at the right timing go to help it
out unblock it so that we could push
forward the progress but as we can see
here the entire process is I would say
extremely smoother uh extremely smooth
and quite smoother than human interns
because I I don't expect that intern
would write all this code like this
generate a report finish all the
implementations within what half an hour
so I hope you enjoy this demo and hope
this is inspiring to you as well