1
00:00:00,000 --> 00:00:00,630
Hello, everyone.

2
00:00:00,630 --> 00:00:04,533
Thank you for joining us for this
month in React, which is not going

3
00:00:04,533 --> 00:00:06,363
to be particularly React heavy.

4
00:00:06,423 --> 00:00:09,963
Mark and I have been talking for a
couple of weeks now about doing a,

5
00:00:09,963 --> 00:00:14,463
like, bonus episode of sorts to talk
about AI and how we're using it.

6
00:00:14,973 --> 00:00:18,093
So we are just, instead of a bonus
episode, we're, we're just gonna

7
00:00:18,093 --> 00:00:19,503
do that for this month, for April.

8
00:00:19,563 --> 00:00:21,063
Yeah, and apologies for last month.

9
00:00:21,063 --> 00:00:24,224
We had a recording problem,
and we, it was completely

10
00:00:24,224 --> 00:00:26,144
unsalvageable, just nothing to save.

11
00:00:26,324 --> 00:00:26,864
Big bummer.

12
00:00:26,954 --> 00:00:28,664
But yeah, so I am Carl.

13
00:00:28,994 --> 00:00:32,534
I am joined this month by Mark
Erickson and Swizec Teller.

14
00:00:33,114 --> 00:00:36,024
yeah, we're gonna talk about AI
because it's been a huge part of

15
00:00:36,024 --> 00:00:40,554
each of our workflows for the last,
like, ranging from, like, three

16
00:00:40,554 --> 00:00:42,624
to six months to a year or more.

17
00:00:43,054 --> 00:00:44,074
Let's do some intros first.

18
00:00:44,074 --> 00:00:46,954
I guess Mark and I are reasonably
well known, but I'm Carl.

19
00:00:47,014 --> 00:00:52,174
I am a staff level software engineer
and engineering manager and community

20
00:00:52,174 --> 00:00:56,134
lead here at Reactiflux, where I
do events like this and build code

21
00:00:56,134 --> 00:00:57,754
to keep the community operating.

22
00:00:58,024 --> 00:00:58,774
I'm Mark Erickson.

23
00:00:58,864 --> 00:01:03,604
My day job is ReplayIO, where we've built
a time traveling debugger for both humans

24
00:01:03,604 --> 00:01:06,664
and agents with ReplayMCP now available.

25
00:01:07,024 --> 00:01:07,984
Please check out our blog.

26
00:01:07,984 --> 00:01:12,064
I just put up a blog post on how Replay
found a bug faster than Dan Abermov did.

27
00:01:12,274 --> 00:01:13,804
I am still the Redex maintainer.

28
00:01:13,804 --> 00:01:16,624
Honestly, I haven't done much
Redux stuff in the last few months

29
00:01:16,624 --> 00:01:19,564
because all my brain space has
been taken up with day job work.

30
00:01:19,864 --> 00:01:22,434
And also, I'm going around to a
whole bunch of conferences this year.

31
00:01:23,154 --> 00:01:23,254
I'm Swiz.

32
00:01:23,874 --> 00:01:29,634
I work at Plasmidsaurus, we are a
DNA sequencing as a service company.

33
00:01:29,964 --> 00:01:33,774
We do a lot of React for really fancy
data visualizations for stuff like,

34
00:01:34,254 --> 00:01:38,124
"Hey, wh- how do you visualize a few
million data points in the browser and

35
00:01:38,124 --> 00:01:39,744
make it work smooth?" Stuff like that.

36
00:01:40,314 --> 00:01:45,234
And these days, I'm kind of more
of a manager than an IC really.

37
00:01:45,564 --> 00:01:50,304
And I've been thinking a lot about what
kind of engineers get hired these days.

38
00:01:50,694 --> 00:01:54,924
We've been hiring a lot and using
more and more AI to write the code.

39
00:01:55,524 --> 00:01:55,734
Yep.

40
00:01:55,824 --> 00:01:56,454
Believe that.

41
00:01:56,754 --> 00:01:57,084
Cool.

42
00:01:57,084 --> 00:01:57,414
Yeah.

43
00:01:57,504 --> 00:02:00,204
So we were just chatting a
little bit about what shape

44
00:02:00,204 --> 00:02:01,674
this conversation's gonna take.

45
00:02:01,764 --> 00:02:04,614
Just to level set a little
bit for everyone listening.

46
00:02:04,884 --> 00:02:08,754
We're gonna start off kind of at the
point of, like, what convinced us that

47
00:02:08,754 --> 00:02:13,884
AI was a tool worth taking seriously and,
you know, getting AI pilled as it were.

48
00:02:14,244 --> 00:02:18,114
Go from there into how we're using it
now, what problems we're using it to

49
00:02:18,114 --> 00:02:22,584
solve, with what tools, as well as kind
of, like, what aren't we using, what

50
00:02:22,584 --> 00:02:24,504
don't we find useful and compelling?

51
00:02:24,894 --> 00:02:27,954
And go from there to, like, landscape
of, like, what tools are available,

52
00:02:27,954 --> 00:02:31,374
what's out there, where do we think
it's gonna go, and then kinda close out

53
00:02:31,374 --> 00:02:35,604
with what do we think the impacts are
gonna be on the industry more broadly.

54
00:02:36,124 --> 00:02:38,884
Mark, you wanna start us off
talking about what convinced

55
00:02:38,884 --> 00:02:40,804
you that AI was worth using?

56
00:02:41,374 --> 00:02:41,734
Sure.

57
00:02:41,944 --> 00:02:47,134
A year ago, I was dead set that I would
never, ever allow AI to write code for me.

58
00:02:47,134 --> 00:02:48,784
It was a fate worse than death.

59
00:02:48,784 --> 00:02:50,344
It was destroying my career.

60
00:02:50,344 --> 00:02:51,794
I refused to do it.

61
00:02:52,094 --> 00:02:57,014
And in fact, I actually wrote a 15,000
word blog post over the weekend that

62
00:02:57,014 --> 00:03:01,454
I haven't published yet that will give
the long form version of this story.

63
00:03:02,024 --> 00:03:07,004
The short form is, over the summer last
year, I cautiously started using AI to

64
00:03:07,004 --> 00:03:09,764
explain an existing code base to me.

65
00:03:10,064 --> 00:03:13,634
You know, just give me some architecture
docs, walk me through the data flow.

66
00:03:14,294 --> 00:03:18,374
And then there was a three-day period
in late August that blew my mind.

67
00:03:18,824 --> 00:03:23,684
On a Tuesday, I asked it to write some
redux unit tests for me because my

68
00:03:23,684 --> 00:03:28,034
brain was too tired to write actual
code, and it did, and I was stunned.

69
00:03:28,394 --> 00:03:33,134
On Wednesday, there was a node compression
library that I've been trying to replace,

70
00:03:33,134 --> 00:03:37,064
but the alternative didn't have all the
features we needed, and it's Rust-based.

71
00:03:37,424 --> 00:03:40,364
And I tried asking the AI to
write the feature for me in

72
00:03:40,364 --> 00:03:42,164
the Rust library, and it did.

73
00:03:42,224 --> 00:03:45,104
It actually didn't quite work right,
and the maintainer had to turn down

74
00:03:45,104 --> 00:03:50,054
the PR, but this was the first time
I saw an agent actually just crank

75
00:03:50,054 --> 00:03:53,534
along and spit out a bunch of code
and happily make a bunch of updates.

76
00:03:54,014 --> 00:03:55,934
And I thought I had a, a
good understanding of what

77
00:03:55,934 --> 00:03:57,104
that process looked like.

78
00:03:57,284 --> 00:04:00,284
And then when I saw it in person
for the first time, my jaw dropped.

79
00:04:00,974 --> 00:04:05,294
And then on a Thursday, I needed to
write some AST-based linting code.

80
00:04:05,594 --> 00:04:07,094
I know what ASTs are.

81
00:04:07,094 --> 00:04:08,084
I've used Babbel.

82
00:04:08,084 --> 00:04:11,594
I understand the concepts, but
it's kind of complicated and

83
00:04:11,594 --> 00:04:13,394
fiddly, and we had a custom setup.

84
00:04:13,784 --> 00:04:15,674
I was like, "Could this do it for me?

85
00:04:16,124 --> 00:04:16,904
" And it did.

86
00:04:17,324 --> 00:04:20,264
And it did it much faster
than I could as a person.

87
00:04:20,744 --> 00:04:22,244
And my worldview got destroyed.

88
00:04:22,724 --> 00:04:22,994
Yep.

89
00:04:23,084 --> 00:04:23,864
Sounds familiar.

90
00:04:24,104 --> 00:04:27,464
I mean, that, that sounds
like an amazing experience.

91
00:04:27,464 --> 00:04:33,104
For me, the first was way back in
the, uh, Stone Age where you had

92
00:04:33,104 --> 00:04:38,744
to talk to ChatGPT and then copy
paste the output to try to run it.

93
00:04:39,194 --> 00:04:44,324
And it was like, I think it was the
holidays, and I was writing a book feeling

94
00:04:44,324 --> 00:04:47,534
kind of discouraged, and I was like, "I
wonder how many words I'm writing per

95
00:04:47,534 --> 00:04:49,574
day." And it's like, I can write Python.

96
00:04:49,604 --> 00:04:55,184
It's not that interesting to parse a
markdown file, go through, get history and

97
00:04:55,184 --> 00:04:57,524
see how many words you added every day.

98
00:04:58,034 --> 00:05:01,544
Um, so I was like, "Maybe ChatGPT
can just write this for me.

99
00:05:01,924 --> 00:05:07,904
" And I asked it, and the code,
the code didn't work, but it ran.

100
00:05:08,414 --> 00:05:10,304
And I thought that was really cool.

101
00:05:10,544 --> 00:05:14,174
So I talked to it a little bit
more, and we ended up with a really

102
00:05:14,174 --> 00:05:19,424
nice ... It was an extremely ugly
code that I would never write myself.

103
00:05:19,874 --> 00:05:23,594
I think it ended up still taking
two or three hours, but it was a lot

104
00:05:23,594 --> 00:05:25,274
more interesting than me doing it.

105
00:05:25,874 --> 00:05:31,484
Um, and I ended up with a nationalization
of how my, how my book is doing, and then

106
00:05:31,484 --> 00:05:33,314
I wrote a few more scripts like that.

107
00:05:33,944 --> 00:05:39,104
And then I started using it at work to
write my database migrations, because-

108
00:05:39,494 --> 00:05:41,504
Please tell me, was it
still just the ChatGPT UI?

109
00:05:41,984 --> 00:05:42,644
It was.

110
00:05:42,734 --> 00:05:45,434
Uh, this was, this,
this was before Copilot.

111
00:05:45,644 --> 00:05:49,484
I was like, "Hey, ChatGPT, I have this
SQL query, please write migration."

112
00:05:49,794 --> 00:05:56,274
And it wrote the migrations or I ended
up, uh, later on just copy pasting table

113
00:05:56,274 --> 00:06:01,734
definitions from, like, DBR or DataGrape
or whatever, go look at the current tables

114
00:06:01,734 --> 00:06:06,204
and be like, "I have these tables. Please
write migration for ... " I think we

115
00:06:06,204 --> 00:06:11,244
were using Connects at the time. "Please
write migration to add these columns

116
00:06:11,484 --> 00:06:13,344
and then copy paste back and it worked.

117
00:06:13,884 --> 00:06:16,284
"And I was doing that for
a while and it was amazing.

118
00:06:16,734 --> 00:06:18,624
I never wrote a migration
again in my life.

119
00:06:18,954 --> 00:06:19,314
Yeah.

120
00:06:19,374 --> 00:06:22,044
I started doing a lot more
migrations because it just became

121
00:06:22,044 --> 00:06:23,964
so much easier to actually do them.

122
00:06:23,964 --> 00:06:25,584
There's such a pain,
there's so mechanical.

123
00:06:25,734 --> 00:06:29,874
That was an early one for me too that
was like ... I guess I actually had been,

124
00:06:29,874 --> 00:06:33,804
like, scoping my work to avoid migrations
because I hate doing them so much.

125
00:06:34,194 --> 00:06:36,924
I had a similar experience of just
like, " Oh my God, this works for that.

126
00:06:36,924 --> 00:06:39,684
I can do so many more of them now
because I don't have to do it.

127
00:06:40,284 --> 00:06:43,614
"Yeah, it's a little funny because
actually database migrations are

128
00:06:43,614 --> 00:06:47,064
a big part of why my career looks
the way it does, because I did,

129
00:06:47,094 --> 00:06:49,974
like, one in, you know, my first
year as a software engineer.

130
00:06:49,974 --> 00:06:51,174
I went," Wow, I hate this.

131
00:06:51,174 --> 00:06:52,524
You can't guarantee anything.

132
00:06:52,524 --> 00:06:54,474
There's just, you just gotta try it.

133
00:06:54,924 --> 00:06:57,354
"And so then I went more front
end because you don't have to do

134
00:06:57,354 --> 00:06:59,034
database migrations on the front end.

135
00:06:59,634 --> 00:06:59,994
Yeah.

136
00:07:00,084 --> 00:07:02,934
I had a similar type
of experience as Mark.

137
00:07:03,024 --> 00:07:05,544
I guess I, I got on it much
later than you did, Swizz.

138
00:07:05,634 --> 00:07:10,584
Sometime around, like, August, July,
August, September of last year, you

139
00:07:10,584 --> 00:07:14,334
know, I, I had done kind of like
ChatGPT prototypes or whatever, like

140
00:07:14,394 --> 00:07:18,534
Claude, having it write some HTML
in line and just, like, prove out a

141
00:07:18,564 --> 00:07:23,664
concept or write a first draft of a
script, like, prove out does this work?

142
00:07:23,664 --> 00:07:27,714
Give me off the blank page and give me
something to grow from and diagnose.

143
00:07:28,374 --> 00:07:32,754
And then I'd been hearing people talk
about Claude code, so I finally gave

144
00:07:32,754 --> 00:07:36,174
that a shot in, like, I think around
July, August for the first time.

145
00:07:36,594 --> 00:07:41,514
And just watching it, you know, like,
Mark, like you said, of just watching

146
00:07:41,514 --> 00:07:46,494
it churn through and do things and
also having it be, like, native

147
00:07:46,494 --> 00:07:50,634
on the command line and watching
it just run exactly the same Bash

148
00:07:50,634 --> 00:07:54,234
commands that I would to diagnose
something and, like, read the files.

149
00:07:54,384 --> 00:07:59,381
It was like, "Oh, this is doing exactly
the same process I would to resolve

150
00:07:59,381 --> 00:08:04,421
this bug, but it's doing it five times
faster than I can." That was my big,

151
00:08:04,421 --> 00:08:07,961
like, light switch moment of, like, "Oh,
I need to take this really seriously."

152
00:08:07,961 --> 00:08:11,921
For me, it's kind of like I
really hate watching it work.

153
00:08:12,251 --> 00:08:15,041
I've been using AI for a while
doing, like," Oh, can you

154
00:08:15,041 --> 00:08:16,331
write this function for me?

155
00:08:16,331 --> 00:08:19,151
Or can you, like, do little small things?

156
00:08:19,571 --> 00:08:24,911
"But my workflow really changed when
Cursor launched Slack background agents.

157
00:08:25,331 --> 00:08:29,411
So I started in Slack, like, when you
get ... Partially, I'm a PM, so I get

158
00:08:29,411 --> 00:08:33,161
a lot of requests that are like, " That
is definitely not a priority right now.

159
00:08:33,161 --> 00:08:34,391
We're not gonna work on that.

160
00:08:34,991 --> 00:08:40,811
"But I can now go at Cursor, do the
thing, and I just get a PR with the

161
00:08:40,811 --> 00:08:45,581
implemented small thing that I never
would've taken the time to do myself.

162
00:08:46,241 --> 00:08:49,281
And I, I find that amazing
because watching it work, I think

163
00:08:49,281 --> 00:08:50,991
is, for me, is too distracting.

164
00:08:51,351 --> 00:08:51,471
Yeah.

165
00:08:51,471 --> 00:08:54,381
So I like it when it's fully
somewhere in the cloud.

166
00:08:54,381 --> 00:08:55,371
I don't have to worry.

167
00:08:55,671 --> 00:08:57,261
I just review the code when it's ready.

168
00:08:58,101 --> 00:09:01,971
My workflow is very hands-on
and human in the loop.

169
00:09:01,971 --> 00:09:05,601
Like, I'm sitting there watching that
thing like a hawk and having conversations

170
00:09:05,601 --> 00:09:10,041
with it, which I realize is not how
most people are using AI at this point.

171
00:09:10,041 --> 00:09:14,721
I think a lot of people are very much
on the, in the cloud, multi-agents, how

172
00:09:14,721 --> 00:09:18,201
many of these th- things can be run in
parallel, which I now have opinions on.

173
00:09:18,591 --> 00:09:21,561
Part of it is ... Well, a lot,
a lot of it's the point that I'm

174
00:09:21,561 --> 00:09:27,171
making the draft blog post, which
is understanding is still critical.

175
00:09:27,351 --> 00:09:31,011
And I'm a very, I'm a very firm
believer in, you know, the fundamentals

176
00:09:31,011 --> 00:09:33,801
and understanding and building
a mental model of the system.

177
00:09:34,341 --> 00:09:39,141
And I personally, for me,
I want my brain engaged.

178
00:09:39,291 --> 00:09:43,461
I want to be thinking through the
problem, and then I'm using the

179
00:09:43,461 --> 00:09:46,101
agent to amplify my own abilities.

180
00:09:46,101 --> 00:09:48,531
Now, don't get me wrong, there's
definitely been a few moments where

181
00:09:48,531 --> 00:09:49,731
I was, like, you know, out and about.

182
00:09:49,731 --> 00:09:52,041
It's like, you know, it actually would be
kind of nice if I could just pull up my

183
00:09:52,041 --> 00:09:53,751
phone and tell an agent, go do this thing.

184
00:09:54,021 --> 00:09:54,921
Like, I, I, don't get me wrong.

185
00:09:54,951 --> 00:09:55,761
I get the appeal.

186
00:09:56,151 --> 00:09:56,361
Yeah.

187
00:09:56,391 --> 00:09:59,991
But in terms of day-to-day
development work and how I approach

188
00:09:59,991 --> 00:10:05,631
programming, I want my brain
active, engaged, and thinking, not

189
00:10:05,631 --> 00:10:07,251
just handing it off to an agent.

190
00:10:08,601 --> 00:10:08,901
Yeah.

191
00:10:09,231 --> 00:10:09,711
I hear that.

192
00:10:09,711 --> 00:10:11,781
I think I use it in both ways.

193
00:10:11,961 --> 00:10:16,191
There are a lot of things that, like
you said, so it's just, like, it's not

194
00:10:16,191 --> 00:10:19,221
quite high priority enough to justify.

195
00:10:20,331 --> 00:10:22,701
And that's where I'll use, like,
more of a background agent.

196
00:10:22,731 --> 00:10:28,791
Like, you know, here's 15 one line
descriptions and, like, just from the one

197
00:10:28,791 --> 00:10:30,501
sentence, it's clear enough what's needed.

198
00:10:30,501 --> 00:10:32,451
Like, you know, adjust the size of this.

199
00:10:32,931 --> 00:10:35,361
Add an element that controls this.

200
00:10:35,601 --> 00:10:38,841
Like, those are mechanical enough
that it's really just, like, getting

201
00:10:38,841 --> 00:10:40,311
it right, making sure it compiles.

202
00:10:40,611 --> 00:10:44,661
I don't currently have a functioning
workflow for that exactly.

203
00:10:44,931 --> 00:10:48,021
What I'm doing right now in the last
four or five weeks just, like, hasn't

204
00:10:48,021 --> 00:10:50,541
really been of that type of development.

205
00:10:50,841 --> 00:10:53,841
So I am still, day-to-day, most
of what I've been doing is very

206
00:10:53,841 --> 00:10:55,311
much, like, the mech suit variant.

207
00:10:55,311 --> 00:10:57,421
You know, people talk about,
like, automated versus mech

208
00:10:57,421 --> 00:10:59,591
suit, like robot or mech suit.

209
00:11:00,021 --> 00:11:02,691
And yeah, so I'm driving it
pretty closely, but I'm not really

210
00:11:02,691 --> 00:11:04,761
reviewing its output very strictly.

211
00:11:04,761 --> 00:11:07,971
I'm, you know, I tell it to use
subagents, and then every so often

212
00:11:07,971 --> 00:11:11,061
I'll, you know, tell it to write me
a report, like, how is this working?

213
00:11:11,061 --> 00:11:11,691
What is it doing?

214
00:11:11,691 --> 00:11:14,691
Ask it some targeted questions to
make sure that the mental model I

215
00:11:14,691 --> 00:11:16,641
have matches what's actually there.

216
00:11:17,121 --> 00:11:19,671
But that's a little different, I
think, than what you described, Mark.

217
00:11:19,671 --> 00:11:22,821
Like, I'm not actually, like, I
don't read almost any of the code.

218
00:11:23,211 --> 00:11:26,271
I try and think of it more from,
like, my engineering manager,

219
00:11:26,271 --> 00:11:28,551
tech lead, product manager hat.

220
00:11:28,611 --> 00:11:33,081
I do really try and sit in engineering
manager/product manager role and

221
00:11:33,081 --> 00:11:37,191
say, like, "Okay, how did those
people talk to me as an engineer?

222
00:11:37,191 --> 00:11:38,931
Like, they were not technical.

223
00:11:38,931 --> 00:11:41,961
They didn't understand anything of
what was being built, and yet it

224
00:11:41,961 --> 00:11:45,511
was their job to make sure it was
functioning as intended." So that's

225
00:11:45,511 --> 00:11:49,531
very much how I try to think about
my work now is from that perspective.

226
00:11:49,531 --> 00:11:54,841
It's like I, instead of reading a line
of code to ensure that patterns are

227
00:11:54,841 --> 00:11:59,221
followed as I intended, like, set up
a lint rule and then, you know, maybe

228
00:11:59,221 --> 00:12:03,691
it's only 80% is good, it's not gonna
catch all the subtleties, but, like,

229
00:12:03,751 --> 00:12:06,751
my experience of working on a team
is, that's kinda how it works anyway.

230
00:12:06,751 --> 00:12:09,871
Like, if it's not captured in the
lint rule, eventually it will break.

231
00:12:10,021 --> 00:12:15,001
Like, eventually, someone will
have been onboarded and not ha- be

232
00:12:15,001 --> 00:12:18,991
deeply steeped in the history of the
project or just be tired that day

233
00:12:18,991 --> 00:12:21,361
and they forget about it or whatever.

234
00:12:21,751 --> 00:12:24,931
So, like, that was very much my
perspective as, uh, you know, when

235
00:12:24,931 --> 00:12:29,011
I've been a tech lead is, like, if bad
code goes out, you know, the blameless

236
00:12:29,011 --> 00:12:30,811
postmortem, it was a process failure.

237
00:12:30,811 --> 00:12:33,691
This should not have been
permitted by the automated checks.

238
00:12:33,931 --> 00:12:37,321
And so that's kind of how I'm thinking
about it is just remove the abstraction

239
00:12:37,321 --> 00:12:41,581
a little bit and guarantee the
outputs and then work on automated

240
00:12:41,581 --> 00:12:46,381
tracking of automated evaluation of
the code quality to make sure that's

241
00:12:46,381 --> 00:12:47,941
at a level that I need it to be.

242
00:12:48,301 --> 00:12:48,811
Yeah, same.

243
00:12:48,811 --> 00:12:50,761
I think that's my perspective as well.

244
00:12:50,761 --> 00:12:56,041
I've ... Going up into tech lead was
really kind of almost broke me as an

245
00:12:56,041 --> 00:13:01,201
engineer because I was like, "Oh my God,
all of this terrible, awful code, but

246
00:13:01,291 --> 00:13:07,981
it works and it's fine and everything
is okay and the team just handles

247
00:13:07,981 --> 00:13:11,761
it.  And if anything is wrong later,
we just fix it. It's totally fine."

248
00:13:12,181 --> 00:13:14,191
And I kind of treat AI the same way.

249
00:13:14,289 --> 00:13:18,751
I, I do a lot of driving from, like,
a product perspective and for critical

250
00:13:18,751 --> 00:13:23,161
features or, like, super gnarly
business requirements, I go into the

251
00:13:23,161 --> 00:13:27,571
ID and I drive it kind of like what
Mark was describing, reviewing every

252
00:13:27,571 --> 00:13:29,221
line of code, clicking yes, et cetera.

253
00:13:29,821 --> 00:13:32,041
But they do really well with follow-ups.

254
00:13:32,191 --> 00:13:35,221
Cursor gives you the wrong
thing back in a Slack thread.

255
00:13:35,551 --> 00:13:39,061
You just say, "Ed curs- ad cursor,
go fix it. " A lot of the times,

256
00:13:39,061 --> 00:13:41,071
those fixes are actually, "Oh, yeah.

257
00:13:41,431 --> 00:13:45,271
Now that I'm holding a working MVP,
it doesn't actually feel right.

258
00:13:45,271 --> 00:13:48,031
I totally didn't even think
of several features that it

259
00:13:48,031 --> 00:13:49,171
needs before it's useful."

260
00:13:49,711 --> 00:13:52,351
I also really like the code review flow.

261
00:13:52,741 --> 00:13:55,771
I don't know exactly what we did,
but we hooked it up so that you can

262
00:13:55,771 --> 00:13:59,731
do a PR review, like a code review
for your cursor agent session.

263
00:13:59,941 --> 00:14:04,471
And in GitHub, you just go at cursor, fix
this, or you give it information basically

264
00:14:04,471 --> 00:14:08,491
the same as you would with a team
member, and then you just get follow-up,

265
00:14:08,611 --> 00:14:10,471
commits, and it fixes all the things.

266
00:14:10,981 --> 00:14:12,841
That's very much been
my experience as well.

267
00:14:13,081 --> 00:14:16,741
One of my biggest struggles as a,
as an engineer is, like, I remember

268
00:14:16,741 --> 00:14:20,131
hearing about this sort of a stereotype
of, like, give it, give this task

269
00:14:20,131 --> 00:14:22,861
to a junior engineer because they
don't know it's impossible, you know?

270
00:14:23,131 --> 00:14:26,641
The sense of as you get more
experienced, you get to see more of

271
00:14:26,641 --> 00:14:30,721
the complexity, and the complexity
makes it harder to take action because

272
00:14:30,721 --> 00:14:33,211
you're now evaluating trade-offs
instead of just kind of ignoring them

273
00:14:33,211 --> 00:14:34,231
because you don't know they're there.

274
00:14:34,651 --> 00:14:37,891
And I would get so stalled into
analysis paralysis and, like,

275
00:14:37,891 --> 00:14:39,211
what's the correct way to do this?

276
00:14:39,211 --> 00:14:40,231
How do you do this best?

277
00:14:40,231 --> 00:14:42,901
What's gonna have the best maintenance
trade-offs and the lowest, you

278
00:14:42,901 --> 00:14:44,621
know, the tech debt and whatever?

279
00:14:45,281 --> 00:14:49,901
And that's just such a
difficult way to actually solve,

280
00:14:50,051 --> 00:14:51,341
especially a novel problem.

281
00:14:51,341 --> 00:14:53,681
Like, if you understand
a problem well, great.

282
00:14:54,251 --> 00:14:55,751
You can specify everything in advance.

283
00:14:55,811 --> 00:14:58,661
But, like, if you're working at the
edge of your knowledge, at the e- edge

284
00:14:58,661 --> 00:15:02,471
of your understanding, then, like,
you're gonna go the wrong way at first.

285
00:15:02,471 --> 00:15:05,681
And it's so painful to work on something
for three weeks and then go like, "Oh,

286
00:15:05,681 --> 00:15:09,011
shit. This is just completely the wrong
architecture and I need to start over."

287
00:15:09,251 --> 00:15:11,411
But it's so much less painful
now with AI because, like,

288
00:15:11,441 --> 00:15:12,911
great, what a wonderful learning.

289
00:15:12,911 --> 00:15:16,871
Let me take that, let me have it write
a three-page document about what we

290
00:15:16,871 --> 00:15:22,031
learned, what the new problems are, go
back and forth, interact with it about,

291
00:15:22,241 --> 00:15:27,911
like, designing a new data model and
architectural, you know, process flow.

292
00:15:28,241 --> 00:15:29,981
I actually just did this
in the last, like, week.

293
00:15:30,011 --> 00:15:34,091
I've been benchmarking local LLMs because
I want to better understand which ones

294
00:15:34,091 --> 00:15:40,121
work for what tasks so I can not have
all of my AI usage be based on some

295
00:15:40,121 --> 00:15:42,011
mystery frontier model in the cloud.

296
00:15:42,641 --> 00:15:45,641
And I just, like, you know, I started
with just, like, "Hey, give me a

297
00:15:45,641 --> 00:15:49,691
script that will run this model. Start
up a, you know, LLM server and then

298
00:15:49,691 --> 00:15:53,651
run prompts against it. " And then,
like, that grows, "Oh, I need, I

299
00:15:53,651 --> 00:15:55,031
wanna be able to kill it and restart.

300
00:15:55,091 --> 00:15:59,231
Oh, I wanna be able to, you know,
version the prompts as I change them.

301
00:15:59,261 --> 00:16:03,581
Oh, I need to have it, you know, run
a code evaluator when I'm giving it,

302
00:16:03,611 --> 00:16:05,711
like, a code generation challenge."

303
00:16:06,191 --> 00:16:09,221
Eventually, it became un-
unmanageable because of the tech debt.

304
00:16:09,221 --> 00:16:12,461
It's, you know, it grew and expanded and
I learned more about what was needed.

305
00:16:13,091 --> 00:16:16,691
And so I just started over and I
said, "Great. This is a Python ball

306
00:16:16,691 --> 00:16:21,791
of mess and it sucks and I hate it."
So let's talk about the architectural

307
00:16:21,791 --> 00:16:26,261
flow of it and now let's add some lint
rules and let's rewrite it in effect.

308
00:16:26,411 --> 00:16:29,891
So it's actually like a high quality
code base with, like, resumability

309
00:16:29,921 --> 00:16:31,361
and scheduling and whatever.

310
00:16:31,361 --> 00:16:34,151
And now it's working way better
and it's, like, maintainable.

311
00:16:34,151 --> 00:16:35,201
It's been really powerful.

312
00:16:35,861 --> 00:16:39,221
I freely admit that the very
hands-on style that I've got

313
00:16:39,281 --> 00:16:41,471
is intentionally learning.

314
00:16:41,681 --> 00:16:43,871
I know it's possible to go faster.

315
00:16:43,931 --> 00:16:46,571
I know that it's possible
to delegate a lot more.

316
00:16:47,051 --> 00:16:53,441
I am treating it much more as an extension
of my IDE and keyboard than I am as a

317
00:16:53,441 --> 00:16:55,991
junior developer delegating the work.

318
00:16:56,351 --> 00:16:57,011
That's fine.

319
00:16:57,041 --> 00:16:58,061
I am good with that.

320
00:16:58,181 --> 00:17:00,461
That is what works best
for me and my brain.

321
00:17:01,151 --> 00:17:04,541
You might start doing that less
and less as you use it more.

322
00:17:04,631 --> 00:17:09,071
I started with doing that a lot more,
and then eventually I realized, wait

323
00:17:09,071 --> 00:17:12,281
a minute, I'm just clicking yes on
everything and then giving it a follow-up

324
00:17:12,281 --> 00:17:16,151
prompt to change the things I didn't
actually like, but click yes on anyway.

325
00:17:16,841 --> 00:17:19,691
I certainly didn't think a year ago
I would be using any of this at all.

326
00:17:19,691 --> 00:17:23,111
I was, I was convinced that was a red
line that I would, I would never, ever

327
00:17:23,141 --> 00:17:24,971
cross on pain of death, and here I am.

328
00:17:25,451 --> 00:17:29,051
But also, like, I mean, I've, I've found a
workflow that I will be hopefully blogging

329
00:17:29,051 --> 00:17:32,591
about in the next day, next couple
days that does actually work for me.

330
00:17:33,221 --> 00:17:38,771
And the hands-on aspect of it
is what I get good results out

331
00:17:38,771 --> 00:17:40,271
of, and it's what fits my brain.

332
00:17:40,931 --> 00:17:42,831
Well, okay, let's take
that as jumping off point.

333
00:17:42,831 --> 00:17:44,091
Like, what tools

334
00:17:44,091 --> 00:17:47,601
Let's do, like, super concise,
just, like, list what tools and

335
00:17:47,601 --> 00:17:50,031
what models you're using day-to-day.

336
00:17:50,361 --> 00:17:56,301
So my, my own setup, my intro was the
kilocode VS code extension with, you

337
00:17:56,301 --> 00:18:00,531
know, it was probably either Sunat 35
or Sonat4, whatever was available around

338
00:18:00,531 --> 00:18:06,111
September-ish last fall because I didn't
want to go anything command line at first.

339
00:18:06,111 --> 00:18:08,241
I wanted to stay in a
graphical environment.

340
00:18:09,081 --> 00:18:11,131
I tried ClaudeCode for about a day.

341
00:18:11,311 --> 00:18:14,671
Tried the VSCode extension, which didn't
work at all, and then I tried the command

342
00:18:14,671 --> 00:18:16,501
line tool, and I, I did not like it.

343
00:18:16,801 --> 00:18:21,061
I tried the OpenCode command
line tool, also did not like it.

344
00:18:21,241 --> 00:18:24,481
How do y'all deal with copy paste
in a command line environment?

345
00:18:24,531 --> 00:18:27,021
So what I have now, I'm using OpenCode.

346
00:18:27,651 --> 00:18:29,721
My personal laptop is Windows.

347
00:18:29,721 --> 00:18:32,721
My work laptop is Windows,
but I work in WSL.

348
00:18:33,021 --> 00:18:38,601
So I actually serve OpenCode from
within the WSL environment, and then I

349
00:18:38,601 --> 00:18:42,213
use a third party web UI for OpenCode.

350
00:18:42,218 --> 00:18:44,871
OpenCode has a very nice
server client distinction.

351
00:18:44,871 --> 00:18:48,891
The text client is just one
of the possible clients.

352
00:18:49,251 --> 00:18:54,651
So I found one called Code Nomad,
which is very good, works great.

353
00:18:55,041 --> 00:18:59,061
So I actually serve Code Nomad
plus OpenCode added the WSL side,

354
00:18:59,391 --> 00:19:02,931
and then I just hit local host
whatever port in my browser on the

355
00:19:02,931 --> 00:19:07,971
Windows side, which also avoids any
cross-platform file shenanigans as well.

356
00:19:08,451 --> 00:19:12,561
So I've got my chat sessions
and my tabs open in the browser.

357
00:19:12,771 --> 00:19:15,621
That's now my development environment.

358
00:19:16,401 --> 00:19:19,731
And then I still have VS Code open
for looking at some of the diffs and

359
00:19:19,731 --> 00:19:24,921
editing that my ... I much prefer a Git
graphical client called Fork, but it

360
00:19:24,921 --> 00:19:29,541
doesn't work well in Linux, so I've just
stuck with the built-in VS Code stuff.

361
00:19:29,541 --> 00:19:32,721
I actually don't like it, but I've been
too lazy to go find a better alternative.

362
00:19:32,811 --> 00:19:36,051
Model-wise, I've basically
been on whatever the latest

363
00:19:36,051 --> 00:19:37,371
and greatest anthropic is.

364
00:19:37,371 --> 00:19:39,681
We have corporate keys,
da, da, da, da, da, da.

365
00:19:39,891 --> 00:19:44,511
Also, I have somewhat intentionally
tried to avoid model hopping.

366
00:19:44,931 --> 00:19:48,501
I don't want to be running a bunch of
evals every other week and saying, "Oh,

367
00:19:48,501 --> 00:19:53,391
this one provides me a 3% increase in
such and such a benchmark. Clearly, I

368
00:19:53,391 --> 00:19:58,581
need to switch my entire workflow to a
different model." and granted, OpenCode

369
00:19:58,581 --> 00:20:02,901
does let you just pick and choose what
model you want, but I'm trying to get

370
00:20:02,901 --> 00:20:06,801
something that's good and consistent
and that I know works, not chasing

371
00:20:06,801 --> 00:20:10,291
the hypothetical maximum performance.

372
00:20:10,831 --> 00:20:11,011
Okay.

373
00:20:11,011 --> 00:20:17,461
So you're, you're on OpenCode
and using latest Claude Anthropic

374
00:20:17,461 --> 00:20:19,441
models just through API usage?

375
00:20:19,921 --> 00:20:20,851
O- Opus 46, yeah.

376
00:20:21,061 --> 00:20:21,811
That's the other thing.

377
00:20:21,871 --> 00:20:27,001
Uh, it's, it's API keys, not the various,
like, you know, $20, $100, $200 max plans.

378
00:20:27,331 --> 00:20:30,811
So I, I read about people getting the
resets and it's like, I haven't do with

379
00:20:30,811 --> 00:20:32,161
that because I'm also not paying for it.

380
00:20:32,731 --> 00:20:33,061
Right.

381
00:20:33,361 --> 00:20:33,691
Cool.

382
00:20:33,721 --> 00:20:34,081
Okay.

383
00:20:34,141 --> 00:20:37,951
It's like knowledge that my experience
may not be universally shared.

384
00:20:38,491 --> 00:20:38,791
Okay.

385
00:20:38,881 --> 00:20:39,241
Yeah.

386
00:20:39,541 --> 00:20:41,041
Yeah, Swizz, what's
your workflow look like?

387
00:20:41,071 --> 00:20:43,171
I'm curious, I'm very curious
about yours because I think you're

388
00:20:43,171 --> 00:20:47,701
by far the most advanced agent
user between the three of us.

389
00:20:47,821 --> 00:20:51,061
Which is really funny because I
was surprised to learn earlier

390
00:20:51,061 --> 00:20:55,441
this week that 97% of my code
is AI because I really talked.

391
00:20:55,981 --> 00:20:56,851
I was like, "What?"

392
00:20:57,361 --> 00:21:01,981
But I actually have an
extremely simple setup.

393
00:21:02,221 --> 00:21:06,031
I use Cursor, I keep it updated
to whatever the latest version.

394
00:21:06,031 --> 00:21:08,731
So when it pops up with, "Please
update," I click the button.

395
00:21:09,031 --> 00:21:14,611
I think the latest Opus model or
whatever it is, it's like something 4.6.

396
00:21:14,671 --> 00:21:16,111
I think it's an Anthropic model.

397
00:21:16,651 --> 00:21:23,131
We have a Team subscription, so
CompanyPays gives us an infinite budget,

398
00:21:23,611 --> 00:21:25,561
but I think I'm actually using, like

399
00:21:25,561 --> 00:21:26,761
It's actually funny.

400
00:21:26,761 --> 00:21:28,711
I don't know ... We were just
talking about this today.

401
00:21:28,711 --> 00:21:33,451
I don't know how this happened, but on
the Cursor Leaderboard, I have the second

402
00:21:33,451 --> 00:21:39,751
most AI usage of all of the engineers, and
I have the least amount of dollars used.

403
00:21:40,381 --> 00:21:40,861
Fascinating.

404
00:21:40,951 --> 00:21:41,551
Interesting.

405
00:21:41,581 --> 00:21:44,671
I have literally spent $25 this month.

406
00:21:45,061 --> 00:21:50,821
So anyway, I use Cursor, I use
@cursor on Slack a lot, and I

407
00:21:50,821 --> 00:21:53,521
use @cursor on GitHub a lot.

408
00:21:53,971 --> 00:21:58,411
We also use Linear, and I've
started more and more delegating

409
00:21:58,411 --> 00:22:00,211
my linear tickets to cursor.

410
00:22:00,211 --> 00:22:04,501
So look at the linear ticket, be like,
"Eh, I didn't write this well, add a

411
00:22:04,501 --> 00:22:09,511
little bit more context so that a dumb bot
can know where to go fix things or what

412
00:22:09,511 --> 00:22:13,351
to change, and then just click delegate
to cursor, and then I review the PRs."

413
00:22:13,981 --> 00:22:15,061
That's my workflow.

414
00:22:15,181 --> 00:22:20,371
I try to keep it really simple and easy,
because like Mark said, I'm not looking

415
00:22:20,371 --> 00:22:23,101
to super maximize my productivity.

416
00:22:23,101 --> 00:22:27,451
I'm more looking for, you know, as a
manager, I'm supposed to stay on SiteQuest

417
00:22:27,451 --> 00:22:32,581
anyway, so this is a really good way
to code between during, uh, between

418
00:22:32,581 --> 00:22:36,091
and during meetings when I'm supposed
to be paying just like half attention.

419
00:22:36,661 --> 00:22:37,441
Yeah, that's really interesting.

420
00:22:37,441 --> 00:22:42,031
I, I, I like that you two are like
kind of two ends of the spectrum here.

421
00:22:42,481 --> 00:22:47,431
So as you use it as like very much
in a managerial capacity, like, you

422
00:22:47,431 --> 00:22:50,701
know, the same as you would ping a
colleague, you ping cursor instead

423
00:22:50,701 --> 00:22:52,351
and just say, "Hey, do look at this.

424
00:22:52,501 --> 00:22:53,611
" 
Yeah, pretty much.

425
00:22:53,611 --> 00:22:54,901
I, I've been doing that a lot.

426
00:22:54,961 --> 00:22:58,051
And it's all within collaboration
platforms, I guess is like

427
00:22:58,111 --> 00:22:59,221
a big distinction here.

428
00:22:59,221 --> 00:23:01,836
It's all in space- A lot of it is.

429
00:23:02,251 --> 00:23:02,641
Yeah.

430
00:23:02,671 --> 00:23:03,061
Okay.

431
00:23:03,241 --> 00:23:08,021
We're now experimenting with building
like a feedback loop so that we would

432
00:23:08,021 --> 00:23:13,391
have a bot that Sentry sends us errors and
we're thinking of having a bot that looks

433
00:23:13,391 --> 00:23:17,711
at those errors, figures out how to fix
them, and then just issues a PR when we

434
00:23:17,711 --> 00:23:20,021
get, so that we could have automatic PRs-

435
00:23:20,201 --> 00:23:20,471
Yeah.

436
00:23:20,471 --> 00:23:26,561
for, at least for some er- no, like, you
know, probably not for super critical,

437
00:23:26,561 --> 00:23:31,421
crazy, important things, but there's a
lot of errors that happen where it's not

438
00:23:31,421 --> 00:23:33,161
that important, but it's nice to fix.

439
00:23:34,031 --> 00:23:34,931
Yeah, definitely.

440
00:23:35,081 --> 00:23:37,031
Right, the long tail of Century issues.

441
00:23:37,211 --> 00:23:41,981
But when I'm like actually hands-on
coding, I use it a lot to write my tests

442
00:23:42,521 --> 00:23:48,401
because I know what I want to test and
I can describe the situation to set

443
00:23:48,401 --> 00:23:55,391
up, but I hate doing the grunt work of
setting up your database in just the

444
00:23:55,391 --> 00:23:57,701
right way to have 50 models, et cetera,

445
00:23:58,271 --> 00:23:58,511
yeah.

446
00:23:58,511 --> 00:23:58,521
Yeah.

447
00:23:58,521 --> 00:23:58,691
Okay.

448
00:23:58,691 --> 00:23:59,111
Interesting.

449
00:23:59,217 --> 00:24:02,351
I, I kind of split y'all's experience
a little bit, or I'm working

450
00:24:02,351 --> 00:24:05,381
towards ... I feel like I'm closer to
where Mark is right now and I'm trying

451
00:24:05,381 --> 00:24:06,631
to move towards where you are, Swiz.

452
00:24:07,451 --> 00:24:11,741
Yeah, I almost exclusively am
using Claude code with Opus.

453
00:24:12,011 --> 00:24:16,241
I've played around a bit with ... Like
I, I used Copilot, you know, GitHub

454
00:24:16,241 --> 00:24:21,611
Copilot, like once to, like, scaffold a
prototype that I was experimenting with.

455
00:24:22,211 --> 00:24:24,431
I don't like the experience
of it being in a web browser.

456
00:24:24,461 --> 00:24:26,411
Like, something about it ... I don't know.

457
00:24:26,411 --> 00:24:31,031
I also don't like code spaces or,
like, remote development across SSH.

458
00:24:31,541 --> 00:24:34,961
So, like, this may be just my own,
like, biases and preferences, but I

459
00:24:34,961 --> 00:24:37,031
really like just ... It's right here.

460
00:24:37,031 --> 00:24:38,171
It's on my machine.

461
00:24:38,171 --> 00:24:39,941
It's an environment that I have set up.

462
00:24:40,211 --> 00:24:43,091
I know exactly what's
available to it and what's not.

463
00:24:43,601 --> 00:24:47,531
Just, like, I guess I've prioritized,
at least in that where I'm trying to

464
00:24:47,531 --> 00:24:51,341
use it like a Mac suit if speeding up
my own pr- individual productivity,

465
00:24:52,241 --> 00:24:55,511
I just want it to be predictable
and understandable for myself.

466
00:24:55,631 --> 00:25:00,971
To me, that's very much just been Claude
code with whatever defaults pretty much.

467
00:25:01,001 --> 00:25:04,391
I've played around a little bit
here and there, but yeah, I'm trying

468
00:25:04,391 --> 00:25:07,691
to work towards ... One of my many
projects that I have in flight is

469
00:25:07,751 --> 00:25:09,551
a personal, like, orchestrator.

470
00:25:10,091 --> 00:25:15,731
So I had started out kind of my foray
into, like, earnest AI focused, using it

471
00:25:15,731 --> 00:25:20,291
to the extent that I am now, as opposed
to more limited reading every line of

472
00:25:20,291 --> 00:25:22,511
code, going in and editing it myself.

473
00:25:23,741 --> 00:25:26,141
That's only really been since,
like, February when I started

474
00:25:26,141 --> 00:25:28,001
playing this AI agent game.

475
00:25:28,601 --> 00:25:32,681
And it's been really interesting thinking
about ... Because putting an agent in

476
00:25:32,681 --> 00:25:38,411
a simulated world and, like, having it
autonomously play a game and do so in an

477
00:25:38,411 --> 00:25:42,551
interesting way, you know, effectively
and interestingly, really just, like,

478
00:25:42,551 --> 00:25:45,161
got a bunch of wheels turning in my head.

479
00:25:45,431 --> 00:25:50,441
And so, like, ClaudeCode works really well
for me for, like, pushing the envelope,

480
00:25:50,441 --> 00:25:56,081
like, expanding what a project is, what
it can do, and I have really enjoyed

481
00:25:56,081 --> 00:26:00,011
using background agents more, like, very
much like what you described to us of

482
00:26:00,011 --> 00:26:04,031
just, "Hey, oh, fix this. This is broken.
Here's this bug." I don't currently

483
00:26:04,031 --> 00:26:08,421
have anything like that operational,
but I did get my little personal

484
00:26:08,421 --> 00:26:10,371
orchestrator working for a minute.

485
00:26:10,621 --> 00:26:14,641
It was just wor- working on the GitHub
API and I just said, like, "Here's a

486
00:26:14,641 --> 00:26:21,091
repo, like, go, you know, read the issues,
triage them, prioritize them, take a task,

487
00:26:21,151 --> 00:26:23,791
open a PR, and wait for me to review it.

488
00:26:24,061 --> 00:26:28,171
" And it worked, but I didn't give it
any guardrails, so it ended up, like,

489
00:26:28,231 --> 00:26:33,211
redoing the same GitHub issue multiple
times, or, you know, it would re-review

490
00:26:33,211 --> 00:26:37,021
the same PR over and over again with,
like, a page and a half of a comment.

491
00:26:37,081 --> 00:26:39,961
And so I was like, "All right,
okay, this is proof of concept, it

492
00:26:39,961 --> 00:26:42,721
works, but, like, clearly I need
some additional things set up."

493
00:26:43,021 --> 00:26:45,721
And I guess one, another thing I
wanna say, like, kind of what you

494
00:26:45,721 --> 00:26:50,521
said was about, like, using Sentry
Data as an input for what to work on.

495
00:26:51,031 --> 00:26:51,991
I think that's the frontier.

496
00:26:51,991 --> 00:26:54,301
Like, that's what is
currently being explored.

497
00:26:54,301 --> 00:26:55,741
Like, how do you do that effectively?

498
00:26:56,131 --> 00:26:56,371
Yeah.

499
00:26:56,401 --> 00:26:57,631
You need the feedback loops.

500
00:26:57,631 --> 00:27:01,831
So I was reading a lot of how to
build an agent papers the other day,

501
00:27:02,251 --> 00:27:08,671
and the main inputs are basically
rag, memory, and feedback loops,

502
00:27:09,331 --> 00:27:10,651
and from there, it can do a lot.

503
00:27:11,011 --> 00:27:16,351
So Cursor, for example, Cursor Cloud
agents through Slack, they actually,

504
00:27:16,351 --> 00:27:20,911
when they're developing the feature,
they will fire up a browser and go

505
00:27:20,911 --> 00:27:25,051
test it, see if they can actually do
the thing, and if they can't, they

506
00:27:25,051 --> 00:27:26,881
will then c- continue iterating.

507
00:27:27,271 --> 00:27:29,671
And in the end, the PR
doesn't just have code.

508
00:27:29,671 --> 00:27:33,931
It has screenshots and videos of
the working feature, which is what

509
00:27:33,931 --> 00:27:38,401
I require from all of my engineers,
and it makes reviews so much easier.

510
00:27:39,001 --> 00:27:40,861
And there was another ... Oh, yeah.

511
00:27:40,861 --> 00:27:46,081
So I think the longest I've managed
to do, to have it spin when I asked

512
00:27:46,081 --> 00:27:47,311
it to build an entire feature.

513
00:27:47,311 --> 00:27:53,131
So, like, I would write a maybe 200
word, two or 300 word prompt in Slack

514
00:27:53,131 --> 00:27:58,981
to add cursor, and it took ... I think
it spent 45 minutes to come back with

515
00:27:59,011 --> 00:28:03,151
the working feature, and that was
amazing, because I was in a meeting

516
00:28:03,151 --> 00:28:05,911
for that whole time, and then I just
looked at the PR and gave it feedback.

517
00:28:06,421 --> 00:28:06,661
Yeah.

518
00:28:07,081 --> 00:28:07,741
That's incredible.

519
00:28:08,191 --> 00:28:12,961
So my day job, we're coming at it from a
similar but also sort of opposite angle.

520
00:28:12,961 --> 00:28:16,501
We, we have built a time driver to
bugger, and originally the premise was

521
00:28:16,501 --> 00:28:21,271
that by making a DVR style recording,
you as a person can go in and do all

522
00:28:21,271 --> 00:28:24,121
the investigating in the lines of code
and the print statements and everything

523
00:28:24,121 --> 00:28:28,471
else so that eventually you as a
person figure out why this was broken.

524
00:28:28,771 --> 00:28:33,571
So we shipped an MCP a couple months
ago, and I've already seen some very

525
00:28:33,571 --> 00:28:38,131
real examples of agents being able
to go in and solve bugs that they

526
00:28:38,131 --> 00:28:39,511
wouldn't have been able to otherwise.

527
00:28:39,751 --> 00:28:42,301
I actually put up a post
about this a week ago.

528
00:28:42,811 --> 00:28:48,511
Dan Abermov had filed an actual React
bug saying that the used deferred value

529
00:28:48,511 --> 00:28:52,381
hook sometimes fails in production,
it's stuck, like a render behind.

530
00:28:52,621 --> 00:28:56,311
And he had a repro and he said, uh,
"I've had my agent try to look at it,

531
00:28:56,641 --> 00:29:00,241
but I can't find the answer." A month
later, he comes back and files like

532
00:29:00,241 --> 00:29:04,241
a four-line fix deep in the guts of
React Scheduler to actually fix it.

533
00:29:04,661 --> 00:29:08,171
And he posted on BlueSky later, and
apparently what he had to do was

534
00:29:08,201 --> 00:29:13,541
rebuild the React Library with a bunch
of console logging added so that his

535
00:29:13,541 --> 00:29:16,691
agent could look at the prod build
and eventually trace what was going

536
00:29:16,691 --> 00:29:18,431
on and figure out how to fix it.

537
00:29:18,731 --> 00:29:22,931
So I'm like, "That would be a great
marketing post comparison." So I took

538
00:29:22,931 --> 00:29:28,241
his example, I made replay recordings
of the working dev build and the failing

539
00:29:28,241 --> 00:29:32,411
prod build, handed them to an agent,
and I said, "Here's a bug report.

540
00:29:32,531 --> 00:29:34,061
Here's the two replay recordings.

541
00:29:34,241 --> 00:29:35,471
The issue is somewhere in React.

542
00:29:35,741 --> 00:29:38,111
Can you find it?" took 10 minutes.

543
00:29:38,591 --> 00:29:41,321
So I'm like, so then I'm like, okay,
well now let's make it like, you know,

544
00:29:41,321 --> 00:29:43,151
something resembling a proper experiment.

545
00:29:43,661 --> 00:29:47,531
So I took the same two recordings
and I spun up four simultaneous

546
00:29:47,531 --> 00:29:49,751
sessions with differing instructions.

547
00:29:50,141 --> 00:29:53,531
The first one was just a basic, here's
a bug report, go investigate, actually

548
00:29:53,561 --> 00:29:55,601
less context, the proof of concept.

549
00:29:56,051 --> 00:30:01,961
The second one, I gave it like a eight
step investigative process to follow.

550
00:30:02,441 --> 00:30:06,911
The third one had a few paragraphs
just naming some concepts and

551
00:30:06,911 --> 00:30:10,301
Reacts internals, like not even file
structure, but just things like,

552
00:30:10,301 --> 00:30:12,401
you know, schedulers, fibers, lanes.

553
00:30:13,001 --> 00:30:15,881
And then the last one also
explicitly listed some of the

554
00:30:15,881 --> 00:30:17,741
replay MCP tools we have available.

555
00:30:18,221 --> 00:30:23,171
All four of the sessions found the
same bug and suggested the correct fix.

556
00:30:23,891 --> 00:30:29,231
They took respectively 28,
17, eight, and seven minutes.

557
00:30:29,651 --> 00:30:33,671
So on the one hand, you know, ob-
obligatory sales pitch, replay recording,

558
00:30:33,671 --> 00:30:35,501
finding bugs, it's awesome, it's great.

559
00:30:35,501 --> 00:30:36,701
We're building some cool stuff with it.

560
00:30:37,151 --> 00:30:41,681
But it also showed me a lot
about the value of the prompts

561
00:30:41,681 --> 00:30:43,871
and the context that you give.

562
00:30:44,261 --> 00:30:48,491
There was another example I was working
on where someone had made a, an example

563
00:30:48,491 --> 00:30:52,751
NextJS app with some example bugs in it
and was asking an agent to find them.

564
00:30:53,171 --> 00:30:56,591
And in one of them, there's a, there's
a double loading screen that's caused

565
00:30:56,591 --> 00:31:00,161
by mixing and matching suspense
and Tanstack query loading state.

566
00:31:00,641 --> 00:31:06,071
The AI always suggests the use suspense
query hook, but apparently if you do that,

567
00:31:06,071 --> 00:31:08,171
it leads to a hydration mismatch error.

568
00:31:08,501 --> 00:31:11,951
And the real answer is to do
some server pre-fetching instead.

569
00:31:11,951 --> 00:31:14,381
So like you have to think
bigger, think architecturally.

570
00:31:14,951 --> 00:31:16,031
So I did the same thing.

571
00:31:16,031 --> 00:31:19,091
I tried, you know, making some
re- replay recordings and feeding

572
00:31:19,091 --> 00:31:20,831
in the, the varying instructions.

573
00:31:21,221 --> 00:31:26,351
My agents mostly got the same base
level used suspense query error.

574
00:31:27,101 --> 00:31:31,301
And then I tried one more session
where I fed in the NextJS and

575
00:31:31,311 --> 00:31:33,911
TanStack query skills files.

576
00:31:34,211 --> 00:31:38,681
And now the agent actually said,
"Well, useSuspense query is the

577
00:31:38,681 --> 00:31:42,161
initial fix, but you really ought
to do server pre-fetching instead."

578
00:31:42,581 --> 00:31:47,231
So again, like that, that taught
me a lot about the proper context.

579
00:31:47,471 --> 00:31:52,901
That also says a lot as we're trying
to build our own CI debugging agent

580
00:31:52,901 --> 00:31:55,841
that looks at test recordings, so
clearly we need to give it, you

581
00:31:55,841 --> 00:31:58,661
know, good prompts, user-based
context, all that kind of stuff.

582
00:31:59,111 --> 00:32:02,601
So, I mean, I think even something
like that can maybe help explain

583
00:32:02,601 --> 00:32:04,941
some of the differences and
results that different people get.

584
00:32:05,421 --> 00:32:11,031
Yeah, no, the impact of prompting and
available context is, like, so massive.

585
00:32:11,331 --> 00:32:16,461
As we're talking workflow, something I do
very regularly is, you know, start with

586
00:32:16,461 --> 00:32:19,911
a prompt, start with, like, if I'm gonna
add a new feature or redesign something

587
00:32:19,911 --> 00:32:24,651
or what, you know, a large task, not a
bug fix, you know, a sprints project,

588
00:32:24,711 --> 00:32:31,131
not a ticket, then I'll start off with,
like, a 200-word, like, prompt of, like,

589
00:32:31,131 --> 00:32:32,181
"Here's what we're trying to build.

590
00:32:32,181 --> 00:32:33,771
Here's how I think we will need to do it.

591
00:32:33,981 --> 00:32:35,901
Here's a couple of parts that I
think are gonna be important." You

592
00:32:35,942 --> 00:32:40,041
know, you know, have it explore,
do the implementation, review, you

593
00:32:40,041 --> 00:32:41,811
know, and then I manually confirm.

594
00:32:41,961 --> 00:32:44,961
And then I'll tell it, like ... And
I guess I'll do this at a couple of

595
00:32:44,961 --> 00:32:47,961
points throughout as it's exploring
and, like, evaluating things.

596
00:32:47,961 --> 00:32:50,451
I'll ask it, like, "How
was the documentation?

597
00:32:50,451 --> 00:32:54,711
Like, did you find everything you
needed?" And that has been, like, totally

598
00:32:54,711 --> 00:32:59,451
transformative for, I think for token
usage as well, because, like, just by

599
00:32:59,451 --> 00:33:02,091
asking that, it'll say, like, "Oh yeah,
it was all ... Here's the three files I

600
00:33:02,091 --> 00:33:04,281
referenced and I got everything I needed."

601
00:33:04,281 --> 00:33:08,181
Or, like, "Oh, yeah, this file was
out of date. You know, I did the wrong

602
00:33:08,181 --> 00:33:11,211
thing at first and then I had to go
back and rework it. " So I'd recommend

603
00:33:11,211 --> 00:33:15,081
that we update this" and it makes those
recommendations just based off its

604
00:33:15,081 --> 00:33:17,511
own experience of reading the code.

605
00:33:18,171 --> 00:33:22,671
And so, like, just by doing that fairly
regularly as the code base evolves,

606
00:33:22,731 --> 00:33:26,541
I manage to keep, you know, the
documentation pretty well up to date.

607
00:33:27,081 --> 00:33:30,471
Are you asking it to write its
findings back into documentation

608
00:33:30,561 --> 00:33:32,001
or are you updating the docs?

609
00:33:32,241 --> 00:33:35,661
I'll read it with a skeptical
eye of, you know, like, does this

610
00:33:35,661 --> 00:33:39,771
sound like it is a real problem or
not, a real area for improvement?

611
00:33:39,771 --> 00:33:41,871
And then I'll just say, like, "Yeah,
hey, can you update those docs?"

612
00:33:41,961 --> 00:33:47,901
Often, I'll come at it with a pretty
clear intention of, like, we just

613
00:33:48,441 --> 00:33:51,411
revamped how scheduling works.

614
00:33:51,441 --> 00:33:56,331
Let's analyze the architectural overview
document that we have and see if

615
00:33:56,331 --> 00:33:58,251
it's still accurate and then tell it

616
00:33:58,252 --> 00:34:02,091
... 
You know, so, like, I will do some
documentation specific projects with that

617
00:34:02,091 --> 00:34:05,601
in mind, but for the most part, yeah,
just, like, ask it how the experience

618
00:34:05,601 --> 00:34:10,491
of the familiarizing itself with the
code base is, and then give it free

619
00:34:10,491 --> 00:34:12,681
reign to improve that for the most part.

620
00:34:13,131 --> 00:34:13,461
Yeah.

621
00:34:13,581 --> 00:34:18,111
I think one thing I would be worried
with that we have some people in the

622
00:34:18,111 --> 00:34:23,241
team experimenting with that, and they
all agree that after a few iterations,

623
00:34:23,241 --> 00:34:28,401
like, a few weeks later or a few months
later, you end up with very spaghettified

624
00:34:28,401 --> 00:34:33,951
documentation that's essentially just AI
slob that humans definitely don't wanna

625
00:34:33,951 --> 00:34:38,571
read, but even the AI barely wants to
look at it and read it because it just

626
00:34:38,841 --> 00:34:40,911
keeps getting low, lower and lower signal.

627
00:34:40,911 --> 00:34:44,721
So I think ... I don't have a solution
for that, but it's a thing I've heard.

628
00:34:45,201 --> 00:34:45,651
That's fair.

629
00:34:45,651 --> 00:34:48,891
I haven't really evaluated that recently.

630
00:34:49,161 --> 00:34:54,021
I did come at it with trying to
say, like, "Here's five documents at

631
00:34:54,021 --> 00:34:57,841
these levels of abstraction in the
code base." So I do try to give it a

632
00:34:57,841 --> 00:35:02,401
little bit of overall stru- structural
guidance, but that's a good thing to,

633
00:35:02,491 --> 00:35:03,571
that's a good thing to look out for.

634
00:35:03,571 --> 00:35:04,861
I haven't really evaluated it.

635
00:35:05,341 --> 00:35:08,041
That's also how I get my own
mental model of the code.

636
00:35:08,041 --> 00:35:11,491
So I guess, like, I don't necessarily
write it, but I will read it, and if

637
00:35:11,491 --> 00:35:15,061
it's nonsense spaghetti, then it's like,
"No, we gotta do this again." Like-

638
00:35:15,721 --> 00:35:19,651
That also ties into the
longer term memory question.

639
00:35:19,771 --> 00:35:22,801
So, you know, the problem is
ev- every session is fresh.

640
00:35:22,801 --> 00:35:26,641
It knows nothing, you know, every,
literally everything is being injected,

641
00:35:27,001 --> 00:35:30,241
you know, agents, MD, you know,
whatever rules, files, et cetera.

642
00:35:30,691 --> 00:35:35,641
So how does an agent even know that
you have these nice architectural docs

643
00:35:35,641 --> 00:35:37,471
that you've been keeping up to date?

644
00:35:37,741 --> 00:35:42,541
Or how does it know that, you know,
last week, a week earlier, whatever,

645
00:35:42,781 --> 00:35:46,441
you made these particular decisions,
and that's why we ended up here.

646
00:35:46,831 --> 00:35:49,291
Or it has no idea what
any of your code base is.

647
00:35:49,291 --> 00:35:54,781
Let's go read 30 files to re-
form a brand new, fresh set of

648
00:35:54,781 --> 00:35:56,881
memories that compress the context.

649
00:35:57,511 --> 00:36:00,571
And some of this can be,
can be dealt with partially.

650
00:36:00,571 --> 00:36:05,341
There's ... I, I, I highly recommend
tools that will do AST-based scanning

651
00:36:05,341 --> 00:36:08,731
of the code base and preferring those
for what, for loading chunks of code

652
00:36:09,001 --> 00:36:13,861
rather than just, like, blindly whatever
built-in read file tools are there.

653
00:36:14,311 --> 00:36:16,621
But that is actually where I'm
running into the limits of my

654
00:36:16,651 --> 00:36:18,841
own personal workflow right now.

655
00:36:19,171 --> 00:36:25,321
I have, you know, dozens of feature
and research docs that I've generated.

656
00:36:25,651 --> 00:36:27,811
I have daily progress docs.

657
00:36:27,811 --> 00:36:30,001
I have sub-task handoff docs.

658
00:36:30,301 --> 00:36:34,411
There's a lot of very valuable information
in there, and the AI has absolutely

659
00:36:34,411 --> 00:36:36,931
no clue that any of that exists.

660
00:36:37,111 --> 00:36:41,221
So I'm n- I'm now f- starting to feel the
need that I need to have some kind of,

661
00:36:41,671 --> 00:36:47,431
you know, review sweep process, some kind
of tool to, you know, index the markdown

662
00:36:47,431 --> 00:36:52,711
file, something to form that longer term
memory structure, and I keep bookmarking

663
00:36:52,711 --> 00:36:56,251
hundreds of tools and saying I'm gonna go
investigate them and haven't done so yet.

664
00:36:56,851 --> 00:37:02,521
If only you had a tool that's really
good at summarizing large pieces of text.

665
00:37:03,511 --> 00:37:05,491
I, no, I, I actually
have done some of that.

666
00:37:05,551 --> 00:37:08,671
Um, I just haven't actually settled
on one and tried installing it yet.

667
00:37:09,361 --> 00:37:14,011
One thing I found also really helps
for context and stuff like that is

668
00:37:14,281 --> 00:37:16,711
actually structuring your code well.

669
00:37:17,251 --> 00:37:22,471
So having a vertically oriented
architecture, small balls of mud that

670
00:37:22,471 --> 00:37:28,741
are self-contained rather than one large
horizontally sliced ball of mud makes

671
00:37:28,741 --> 00:37:33,361
it easier for humans, and it also really
works for the AI when you can say,

672
00:37:33,811 --> 00:37:35,881
"Go add a feature to this directory."

673
00:37:36,031 --> 00:37:41,251
And it's the, like, basically use your
directory structure as an index for

674
00:37:41,311 --> 00:37:43,651
where to find different kinds of code.

675
00:37:44,911 --> 00:37:48,241
That sounds suspiciously like software
engineering practices, and I was

676
00:37:48,241 --> 00:37:49,591
told those don't apply anymore.

677
00:37:50,221 --> 00:37:50,941
Yeah, right.

678
00:37:50,941 --> 00:37:55,041
Like, the main thing that I've been trying
to keep in mind here of using it, and,

679
00:37:55,181 --> 00:37:58,421
and this was, this kind of talk goes back
to your recent blog post of, you know,

680
00:37:58,421 --> 00:38:02,221
where you talk about how much code A-
AI is writing for you, and to just talk

681
00:38:02,221 --> 00:38:08,871
to it like, these models were trained
on things like GitHub discussions and PR

682
00:38:08,901 --> 00:38:11,031
reviews and code comments and whatever.

683
00:38:11,031 --> 00:38:16,131
So, like, it's not some new,
esoteric, crazy, unpredictable thing.

684
00:38:16,161 --> 00:38:20,871
Like, the more you talk to it, just like
it's a competent engineer, the more it

685
00:38:20,871 --> 00:38:22,581
will behave like a competent engineer.

686
00:38:22,581 --> 00:38:29,331
And so it has all of these assumptions
about, you know, social norms within

687
00:38:29,331 --> 00:38:34,281
the context of, you know, text documents
describing code and documentation.

688
00:38:34,641 --> 00:38:39,171
And so the, the more you understand the
social norms that it are deeply baked

689
00:38:39,171 --> 00:38:41,181
into its training, the better it does.

690
00:38:41,181 --> 00:38:45,021
So, like, yeah, like, I saw these,
you know, memory startups where it's

691
00:38:45,051 --> 00:38:48,411
like, "Oh, we will automatically
generate documentation for your code

692
00:38:48,411 --> 00:38:49,611
base and make sure it's up to date."

693
00:38:49,611 --> 00:38:53,991
And it's like, I'm already doing that with
a Read Me document tagged with a GitHash

694
00:38:54,051 --> 00:38:57,721
of how, when it was last updated for free.

695
00:38:58,111 --> 00:39:02,911
Like, if you have a couple of folders,
a couple of directories full of code

696
00:39:02,911 --> 00:39:06,721
that is, like, pretty well isolated with
a Read Me document in there, like, I

697
00:39:06,721 --> 00:39:09,991
don't need to tell the AI to look for
the Read Me because it knows to do that.

698
00:39:09,991 --> 00:39:11,881
Of course, it knows to look for a Read Me.

699
00:39:12,481 --> 00:39:17,191
So there's definitely a lot of
difficulty with keeping the signal

700
00:39:17,191 --> 00:39:19,921
high, but, like, that's not new.

701
00:39:20,101 --> 00:39:20,491
I don't know.

702
00:39:20,491 --> 00:39:23,311
Like, that just sounds so much
to me, like, working on a team

703
00:39:23,311 --> 00:39:27,031
of eight engineers and, like, not
everyone has the same shared context.

704
00:39:27,031 --> 00:39:30,841
Not everyone reads every
email or every Slack message.

705
00:39:30,841 --> 00:39:35,641
And so, like, as far as the challenge
of, the, the challenge of keeping

706
00:39:35,701 --> 00:39:41,101
a team up to date on best practices
and recent decision making, versus

707
00:39:41,191 --> 00:39:45,961
keeping an AI at the same level, like,
that, those feel really similar to me.

708
00:39:45,961 --> 00:39:49,831
It's just engineering best
practices and communication norms.

709
00:39:50,431 --> 00:39:54,001
With, with the major difference
that AI will actually go read the

710
00:39:54,001 --> 00:39:56,011
documents you tell, you ask it to read.

711
00:39:57,661 --> 00:39:58,711
Right, right.

712
00:39:59,011 --> 00:40:02,731
Like, one of my, like, you know,
war stories was I was a contractor

713
00:40:02,731 --> 00:40:05,371
somewhere and they kept having, I kept
having to do a rework because, you

714
00:40:05,371 --> 00:40:10,141
know, QA was overwhelmed and, like,
you know, they finally get around to

715
00:40:10,141 --> 00:40:12,301
reviewing PRs and, oh, it's broken.

716
00:40:12,361 --> 00:40:13,321
Oh, gotta go back.

717
00:40:13,351 --> 00:40:15,151
Oh, there's all these
conflicts now because

718
00:40:15,201 --> 00:40:18,811
And so, like, despite being
a contractor, I sat down.

719
00:40:18,811 --> 00:40:20,941
It's like, no, we need to
fix our review process.

720
00:40:20,941 --> 00:40:22,381
We need to fix our merges and whatever.

721
00:40:22,741 --> 00:40:25,501
Spent, like, two weeks doing that,
get everyone on the same page, man.

722
00:40:25,981 --> 00:40:30,061
Next day, the, you know, tech lead
employee at the company, the guy

723
00:40:30,061 --> 00:40:33,871
who should have been doing that
process is like, "Nah, this is too

724
00:40:33,871 --> 00:40:36,871
much. Like, I'm just gonna force
push." And he took down production.

725
00:40:37,351 --> 00:40:42,121
So it's so it's like, uh, when
people talk about AI doing stuff,

726
00:40:42,121 --> 00:40:47,044
it's like, "I don't know, man. Have
you ever worked with engineers?"

727
00:40:47,049 --> 00:40:50,831
Uh, so this, this does tie into a lot of
the conversations that I had last week.

728
00:40:50,831 --> 00:40:53,771
I, I was at both the AI engineer
and React Miami conferences.

729
00:40:53,801 --> 00:40:58,961
And the general theme of the discussions
there between the talks and the people

730
00:40:59,021 --> 00:41:04,091
and then, you know, all the different
chatter online is we built ... And

731
00:41:04,121 --> 00:41:07,061
I also ranted about this in my
15,000 Word unpublished blog post.

732
00:41:07,841 --> 00:41:14,021
We built a bunch of software development
practices for people, the agile manifesto,

733
00:41:14,021 --> 00:41:18,671
linear issues, PR reviews that require
multiple people to stamp them or, you

734
00:41:18,671 --> 00:41:22,991
know, as a way to share knowledge,
standups, like these are all people-

735
00:41:22,991 --> 00:41:28,661
based processes, and it made sense when
humans are the limiting factor and you

736
00:41:28,661 --> 00:41:32,351
have to be very intentional about making
sure you're writing the correct code.

737
00:41:32,831 --> 00:41:36,041
Like, not just does the code work, but
are we even spending the time and effort

738
00:41:36,041 --> 00:41:37,421
on the right thing in the first place?

739
00:41:37,931 --> 00:41:43,811
And we now essentially have a generate
infinite amounts of code button, and now

740
00:41:43,811 --> 00:41:48,311
we're finding out that we've moved the
bottleneck further down the chain into all

741
00:41:48,311 --> 00:41:50,801
the verification and QA sides of things.

742
00:41:51,281 --> 00:41:56,231
And so this shows up in, oops, GitHub
is overwhelmed because we've literally

743
00:41:56,261 --> 00:42:01,091
10Xed the number of PRs that are being
pushed or, wait, we've got a bunch of

744
00:42:01,091 --> 00:42:05,081
senior engineers, but they're literally
spending all their time reviewing PRs

745
00:42:05,081 --> 00:42:10,331
generated by the junior devs with AI, much
less what happens when you have a dark

746
00:42:10,331 --> 00:42:14,801
factory of agents cranking out code twenty
four seven and no one ever looks at it.

747
00:42:15,281 --> 00:42:22,091
Like, we haven't figured out what the
resulting processes ought to be, and

748
00:42:22,091 --> 00:42:24,791
one of the points I make in my blog
post is that I think right now we're

749
00:42:24,791 --> 00:42:28,871
all operating under the assumption that
there's no limit for how fast we can

750
00:42:28,871 --> 00:42:34,871
go, and in reality, we're gonna figure
out the actual limit is maybe, like, 3X

751
00:42:34,871 --> 00:42:41,411
of what it used to be, but we need to,
like, accept that and plan for it instead

752
00:42:41,411 --> 00:42:45,041
of thinking that if we shove everything
in one end of the pipe simultaneously,

753
00:42:45,041 --> 00:42:46,031
it all comes out the other end.

754
00:42:46,901 --> 00:42:47,651
I have opinions.

755
00:42:47,951 --> 00:42:48,341
Um-

756
00:42:49,931 --> 00:42:50,021
Yes.

757
00:42:50,021 --> 00:42:54,611
I think I wanted to say first is I really
like, Carl, you mentioned that you start

758
00:42:54,611 --> 00:42:59,831
sessions with a really long 200 word
prompt that gives all the context, doesn't

759
00:42:59,831 --> 00:43:04,571
just tell the agent what to do, but also
why it's doing it in, like, other context.

760
00:43:04,901 --> 00:43:05,171
Same.

761
00:43:05,171 --> 00:43:07,991
This turns out works really
well with humans as well.

762
00:43:08,201 --> 00:43:10,901
They give you much better work
if you tell them why they're

763
00:43:10,901 --> 00:43:12,401
doing it or what the goal is.

764
00:43:12,821 --> 00:43:15,431
And sometimes if you tell, if you
tell them what the goal is, they

765
00:43:15,431 --> 00:43:18,701
might even come back with, "Your
solution is dumb. We should do this

766
00:43:18,701 --> 00:43:22,991
other thing instead," which, wow, can
you imagine using engineers' brains

767
00:43:22,991 --> 00:43:24,461
for thinking, not just writing code?

768
00:43:24,821 --> 00:43:28,781
But I think the other thing is, we
are generating a lot of code, and we

769
00:43:28,781 --> 00:43:30,881
think the bottleneck is code review.

770
00:43:31,151 --> 00:43:32,861
I think there's kind of two bottlenecks.

771
00:43:32,861 --> 00:43:37,091
One is that when you're moving really,
really fast, you have to actually

772
00:43:37,091 --> 00:43:41,961
be more careful about working on the
right things because if you're just

773
00:43:41,961 --> 00:43:46,581
digging yourself into a hole faster,
we just had an example recently where

774
00:43:47,061 --> 00:43:50,331
we broke Agile, we were like, "Oh yeah,
we know what we, what we need to do.

775
00:43:50,571 --> 00:43:55,461
Let's just have Claude crank out 6,000
lines of code in one gigantic PR."

776
00:43:55,761 --> 00:43:58,941
First of all, we couldn't review that,
so we had to break it up into a bunch of

777
00:43:59,001 --> 00:44:02,511
sub-tasks and then turn that into sub-PRs.

778
00:44:02,961 --> 00:44:07,851
Turns out that took another extra day of
work and everyone was distracted while

779
00:44:07,851 --> 00:44:11,511
doing this, so we weren't focusing on what
we were actually supposed to be doing.

780
00:44:11,781 --> 00:44:13,371
So I think distraction is still bad.

781
00:44:13,401 --> 00:44:15,321
Multitasking is still bad.

782
00:44:15,321 --> 00:44:17,601
We're not as good at
multitasking as we think we are.

783
00:44:18,321 --> 00:44:22,461
And the other thing is we, when we
split all of that up, we found that

784
00:44:23,061 --> 00:44:29,181
Cloud made a tactical mistake very
early on, chose the wrong architecture

785
00:44:29,181 --> 00:44:31,611
for how to build something with React.

786
00:44:31,821 --> 00:44:35,931
And now you have a lot of code that needs
a lot of rework to actually make work,

787
00:44:35,931 --> 00:44:39,681
and you're wasting reviewing time, you're
wa- you're wasting a lot of time that

788
00:44:39,681 --> 00:44:45,471
could have been solved if we started with
a small task to set up a small thing and

789
00:44:45,471 --> 00:44:49,041
talked about it, or maybe even talked
about the architecture before we wrote

790
00:44:49,551 --> 00:44:54,411
six, 7,000 lines of code, and then just
wrote the correct code the first time.

791
00:44:54,861 --> 00:44:55,791
Yeah, 100%.

792
00:44:55,971 --> 00:44:59,031
We're hitting communication
bottlenecks much faster now.

793
00:44:59,061 --> 00:44:59,811
Yeah, like Mark said.

794
00:44:59,931 --> 00:45:02,751
Communication and thinking and
planning was always the job.

795
00:45:03,291 --> 00:45:03,531
Yeah.

796
00:45:03,561 --> 00:45:03,771
Yep.

797
00:45:04,041 --> 00:45:06,621
Except now you're running,
like, 10 times faster.

798
00:45:06,621 --> 00:45:10,251
So if you take the wrong step or as
your first step, you're just running

799
00:45:10,281 --> 00:45:13,701
in the opposite direction of where
you wanna go, but really, really fast.

800
00:45:13,971 --> 00:45:14,301
Right.

801
00:45:14,391 --> 00:45:14,691
Yeah.

802
00:45:14,991 --> 00:45:15,381
Right.

803
00:45:15,831 --> 00:45:19,371
It's very easy to get in
extremely the wrong place in

804
00:45:19,371 --> 00:45:20,811
exactly the same amount of time.

805
00:45:21,051 --> 00:45:23,751
That touches on something I've
said before in this podcast.

806
00:45:24,021 --> 00:45:28,101
I think that the future of, like,
review is gonna be less code review.

807
00:45:28,701 --> 00:45:31,281
The architectural side of things
is still gonna be important.

808
00:45:31,311 --> 00:45:32,721
Like, it's not just, does it work?

809
00:45:32,721 --> 00:45:33,981
Does it do what you intended?

810
00:45:34,101 --> 00:45:36,231
You can't fully treat it as a black box.

811
00:45:36,231 --> 00:45:40,041
The box has to be translucent, you
know, at least a little bit to avoid

812
00:45:40,101 --> 00:45:44,421
major architectural problems that are
gonna cause a production outage or make

813
00:45:44,421 --> 00:45:46,161
it impossible to scale or whatever.

814
00:45:46,191 --> 00:45:48,621
Like, it's not all going
to be, does this function.

815
00:45:49,041 --> 00:45:53,661
But I do think, does it function is
going to be a much greater part of this?

816
00:45:54,021 --> 00:45:59,181
My sense is that it will rely a
lot less on reviewing the code so

817
00:45:59,181 --> 00:46:06,771
much as getting a plausible mental
model of the architecture, and then

818
00:46:06,861 --> 00:46:10,581
it, you know, and then statically
verifying that it does what's intended.

819
00:46:10,581 --> 00:46:14,151
So, like, my, my theory here is that
we're gonna start seeing a lot more

820
00:46:14,391 --> 00:46:16,971
end-to-end testing and user acceptance.

821
00:46:17,451 --> 00:46:20,451
A thing I did recently that I
haven't fully executed on yet,

822
00:46:20,481 --> 00:46:21,981
but ... Well, I, I did a lot of it.

823
00:46:21,981 --> 00:46:28,341
But so a, a project that I have, I
fully rewrote the CICD so that instead

824
00:46:28,371 --> 00:46:34,831
of PR checks, review, approve, merge,
ship, instead of, you know, that, like,

825
00:46:34,831 --> 00:46:39,221
trunk-based development, where every PR
gets shipped individually, I move to more

826
00:46:39,221 --> 00:46:44,471
of a, like, Git Flow type architecture
where it's like, no, here's the version.

827
00:46:44,471 --> 00:46:45,611
This is going to go out.

828
00:46:46,121 --> 00:46:47,741
Uh, here's our release candidate.

829
00:46:48,191 --> 00:46:51,761
And then that single release candidate
gets pushed to a staging environment,

830
00:46:51,791 --> 00:46:55,721
and then I do the user acceptance testing
of does this function as I need it to.

831
00:46:56,111 --> 00:46:59,261
And then once I have that, once I've
done that manual testing, great.

832
00:46:59,261 --> 00:47:01,271
I know this code is
working as I meant it to.

833
00:47:01,751 --> 00:47:05,351
So now I'm going to have it write a bunch
of end-to-end tests to verify that it will

834
00:47:05,351 --> 00:47:07,361
continue to do that when I make changes.

835
00:47:07,691 --> 00:47:10,571
And I think that's gonna be
kind of the new cycle of, like-

836
00:47:10,661 --> 00:47:12,401
I want that, but on every PR.

837
00:47:12,821 --> 00:47:13,541
Yeah, right.

838
00:47:13,601 --> 00:47:17,351
And I guess to me, like, that's where
manual testing is now the bottleneck.

839
00:47:17,441 --> 00:47:20,891
Like, you cannot ... At the end of
the day, this needs to function,

840
00:47:21,281 --> 00:47:22,301
this needs to solve a problem.

841
00:47:22,331 --> 00:47:25,121
Any code you write needs to solve
a problem that real humans have.

842
00:47:25,451 --> 00:47:28,121
Or I guess, you know, sure,
other agents or whatever.

843
00:47:28,181 --> 00:47:31,751
I've experimented with
automated end-to-end testing.

844
00:47:32,261 --> 00:47:33,341
It doesn't quite work yet.

845
00:47:33,341 --> 00:47:36,911
I think the models are, or at least the
models I was trying were too slow, but

846
00:47:36,911 --> 00:47:39,131
the idea is that instead of telling that

847
00:47:39,131 --> 00:47:44,171
Instead of writing a test as what the
features that should be there, it's

848
00:47:44,171 --> 00:47:46,511
more about, can the user do a thing?

849
00:47:47,081 --> 00:47:51,791
And then because you have a computer use
model, it can look at the browser and

850
00:47:51,791 --> 00:47:54,701
it's automatically testing both your UX.

851
00:47:55,301 --> 00:47:58,601
Like, if the model can't figure out
how to do what your user is supposed

852
00:47:58,601 --> 00:48:02,921
to be, drunk users won't figure it out
either, or tired users or whatever.

853
00:48:03,281 --> 00:48:08,201
So you just point it at the browser
and you'd say, "Go, like, go place an

854
00:48:08,201 --> 00:48:12,101
order." And at the end of the day, as
long as it can place the order it wants

855
00:48:12,101 --> 00:48:16,181
to place, you can keep iterating on the
implementation, you can keep iterating

856
00:48:16,181 --> 00:48:20,351
on the design, you can keep iterating
on the UX, you just get tests that are

857
00:48:20,351 --> 00:48:22,751
based on can the agent place an order.

858
00:48:23,231 --> 00:48:23,411
Yeah.

859
00:48:23,411 --> 00:48:24,281
Oh, that's really interesting.

860
00:48:24,281 --> 00:48:27,401
That's, that's a really good variant on
what I was just kind of talking about

861
00:48:27,401 --> 00:48:31,181
because, like, sure, static end-to-end
tests that are verifying, like,

862
00:48:31,181 --> 00:48:35,501
this button with this test ID exists
and the flow completes as expected.

863
00:48:35,501 --> 00:48:38,171
Like, sure, okay, it
functions, but is it usable?

864
00:48:38,621 --> 00:48:42,161
That's an interesting usability
automated usability testing because

865
00:48:42,161 --> 00:48:43,991
that kind of goes back to what
we were talk- talking about with,

866
00:48:43,991 --> 00:48:46,301
like, documentation and code notes.

867
00:48:46,481 --> 00:48:48,191
Like, just make it intuitive.

868
00:48:48,491 --> 00:48:51,371
Can an agent intuit how
to achieve a general task?

869
00:48:51,401 --> 00:48:52,241
Oh, that's really interesting.

870
00:48:52,661 --> 00:48:56,831
If you've ever used VCR in Rails,
that's kind of where I got the idea.

871
00:48:57,311 --> 00:49:02,381
With VCR, you do full integration testing
with calling remote APIs, but you don't

872
00:49:02,381 --> 00:49:05,981
wanna call them every time, so you record
the responses and you keep them locally.

873
00:49:06,521 --> 00:49:11,141
So you then write tests against
those local responses, but if the

874
00:49:11,141 --> 00:49:15,071
API ever changes or anything, you
just regenerate them and you find all

875
00:49:15,071 --> 00:49:18,401
of the bugs in your code that don't
fit the new, newly released API.

876
00:49:19,271 --> 00:49:22,841
And you could do something similar
with this where you have browser-based

877
00:49:23,471 --> 00:49:27,641
acceptance testing, and it records
what it's doing so that you don't have

878
00:49:27,641 --> 00:49:29,381
to spend as many tokens every time.

879
00:49:29,861 --> 00:49:32,471
And you just d- when you change
the UI, you delete it and you

880
00:49:32,471 --> 00:49:34,001
see if you can do the flow again.

881
00:49:34,511 --> 00:49:35,081
Yeah.

882
00:49:35,241 --> 00:49:35,871
Totally smart.

883
00:49:35,991 --> 00:49:36,351
Interesting.

884
00:49:36,351 --> 00:49:38,151
Doesn't quite work yet, but
that would be really cool.

885
00:49:38,571 --> 00:49:41,241
We've kind of been going into this
direction, but let me, like, sort of,

886
00:49:41,241 --> 00:49:43,101
like, you know, refocus a little bit.

887
00:49:43,157 --> 00:49:46,791
Like, so what do we think the
current landscape of tools,

888
00:49:47,241 --> 00:49:49,311
of AI tools looks like?

889
00:49:49,491 --> 00:49:53,041
We've kind of been discussing that
in, in, we've been dancing around it.

890
00:49:53,491 --> 00:49:55,321
And I guess where do we
think it's going next?

891
00:49:55,621 --> 00:49:56,851
Going or ought to go?

892
00:49:57,121 --> 00:49:57,847
Either, both.

893
00:49:57,852 --> 00:49:58,111
Both.

894
00:49:58,621 --> 00:50:02,011
My general inclination as the person
who found one tool set and is stuck

895
00:50:02,011 --> 00:50:05,641
with, and is sticking with it and is not
actively trying to change what he's doing.

896
00:50:06,181 --> 00:50:11,281
Certainly what I'm seeing online is
turning towards automate all the things,

897
00:50:11,521 --> 00:50:14,401
which I think is a very understandable
impulse for a software developer.

898
00:50:14,401 --> 00:50:17,401
If you can do things once, you can
do it in a loop, you can run it in

899
00:50:17,401 --> 00:50:21,031
parallel, you can distribute it, you
can have an agent do it, et cetera.

900
00:50:21,691 --> 00:50:25,771
And I think it kind of ties into the
go as fast as possible at all costs

901
00:50:25,771 --> 00:50:29,401
for the purpose of going as fast
as possible, direction that I see.

902
00:50:30,061 --> 00:50:35,191
I mean, like this bit about, you know,
the dark factories, the idea that a team

903
00:50:35,191 --> 00:50:41,101
never even looks at the code, that it's
truly all agents all the time, and all you

904
00:50:41,101 --> 00:50:45,901
say is, "Do something that accomplishes
some goal, and you never even look at it,

905
00:50:45,901 --> 00:50:48,391
" is sort of an ultimate outcome of that.

906
00:50:48,871 --> 00:50:52,021
I don't think it's a good one,
but I see how it's the idea

907
00:50:52,021 --> 00:50:53,461
taken to a logical extreme.

908
00:50:53,848 --> 00:50:57,331
I, I also have been pretty ... I
found a tool and I'm sticking with it.

909
00:50:57,391 --> 00:51:00,331
I've been noticing ClaudeCode
pulling in lots of ideas

910
00:51:00,331 --> 00:51:01,501
that I've just been thinking.

911
00:51:01,531 --> 00:51:05,491
Like, as it evolves, it's like,
this is a productized version of

912
00:51:05,491 --> 00:51:06,811
how I've already been using it.

913
00:51:06,901 --> 00:51:12,901
You know, like they added a /btw command
that lets you ask a question and it

914
00:51:12,901 --> 00:51:17,581
will ... You cannot use tools and it
does not go into the context history.

915
00:51:17,581 --> 00:51:19,111
So it's just like even do a little aside.

916
00:51:19,111 --> 00:51:21,721
Like if you wanna ask, like, "What's
this thing doing?" I had already

917
00:51:21,721 --> 00:51:24,811
been doing that by way of, like,
I'd ask it a question and then

918
00:51:24,811 --> 00:51:27,601
I'd go back and just, you know,
restore the conversation to that.

919
00:51:27,871 --> 00:51:31,531
And similarly, two months ago, I had
been saying, like, I would actually spin

920
00:51:31,531 --> 00:51:36,361
up a Claude code instance that was on
Haiku because I just need to summarize

921
00:51:36,361 --> 00:51:40,021
this and it's easier and I wanna manage
my usage a little bit more precisely.

922
00:51:40,021 --> 00:51:44,941
And, like, now I don't have to do that
because I just say, use a subagent and

923
00:51:44,951 --> 00:51:48,541
ClaudeCode is smart enough to say, "Oh,
I'm just summarizing. Let me use Haiku."

924
00:51:49,051 --> 00:51:52,711
So I've just been watching, like,
kind of what the agent itself is doing

925
00:51:52,711 --> 00:51:57,841
versus what I need to tell it to do, or
proactively manage, converge a little bit.

926
00:51:57,841 --> 00:52:02,641
So that's, that's had me wondering,
what outside of the agent, what is

927
00:52:02,641 --> 00:52:04,591
not going to live within the agent.

928
00:52:04,741 --> 00:52:09,421
And I think where I've landed on
that is, like, everyone's talking

929
00:52:09,421 --> 00:52:12,601
about memory the last six weeks
or so, and I think that's wrong.

930
00:52:12,661 --> 00:52:14,551
I think that's just gonna be
conventions and documentation.

931
00:52:14,551 --> 00:52:15,631
It's just knowledge.

932
00:52:16,021 --> 00:52:19,351
I think what's going to persistently
live outside of an agent harness, as it

933
00:52:19,351 --> 00:52:26,881
were, is, like, incoming information,
like a data stream, like GitHub issues,

934
00:52:27,121 --> 00:52:32,471
text, you know, the Slack chat, a
Century Alert, a production alarm.

935
00:52:32,501 --> 00:52:36,581
If you have a data stream that is
categorizing those and prioritizing

936
00:52:36,581 --> 00:52:41,771
them, and then giving that to an agent,
to an autonomous agent that is just

937
00:52:41,771 --> 00:52:44,531
working, like, those are different things.

938
00:52:44,531 --> 00:52:48,911
I think you cannot have a single
generic agent that is able to connect

939
00:52:48,911 --> 00:52:50,141
to every possible data source.

940
00:52:50,141 --> 00:52:52,961
So that to me feels like a
boundary that's gonna be pretty

941
00:52:52,961 --> 00:52:54,491
strong for the coming months.

942
00:52:55,061 --> 00:52:55,361
I don't know.

943
00:52:55,361 --> 00:52:59,801
I think for me, I feel like going
super deep into what, at least on

944
00:52:59,801 --> 00:53:03,761
the internet, a lot of people are
talking about various MD files and

945
00:53:03,761 --> 00:53:05,741
orchestrations and all the crazy stuff.

946
00:53:06,131 --> 00:53:10,061
I feel like that's a little bit
like ID skins and WIM shortcuts.

947
00:53:10,061 --> 00:53:14,561
It's like, yeah, sure, it's cool, but
I, I think you're over complicating it.

948
00:53:14,861 --> 00:53:19,211
The models are gonna keep getting
stronger and better, so I would

949
00:53:19,211 --> 00:53:22,091
just think they're gonna be able
to do bigger and bigger tasks.

950
00:53:22,091 --> 00:53:25,871
I think feedback loops, like what you
said, what is outside of the model?

951
00:53:25,961 --> 00:53:28,241
Building really strong
feedback loops is important.

952
00:53:28,601 --> 00:53:33,581
I think memory documentation inside the
code or making the code itself easier

953
00:53:33,581 --> 00:53:39,821
to research or giving it MCP tools to,
like, notion or linear or whatever,

954
00:53:39,821 --> 00:53:41,741
you keep your organizational memory.

955
00:53:42,281 --> 00:53:46,211
Having access to organizational
memory will help agents as well.

956
00:53:46,451 --> 00:53:46,661
Yeah.

957
00:53:46,721 --> 00:53:51,551
I don't think we're ever gonna come to
full automation of anything, really.

958
00:53:52,061 --> 00:53:56,651
I think there's a strong gelman
amnesia when it comes to th- these

959
00:53:56,651 --> 00:54:01,841
things where agents will fully
automate every job except the ones

960
00:54:01,841 --> 00:54:03,401
I'm strict, super familiar with.

961
00:54:03,791 --> 00:54:06,281
That is just way too complicated
and there's too many deals.

962
00:54:06,971 --> 00:54:10,061
Everyone feels that way about
their job because newsflash,

963
00:54:10,061 --> 00:54:11,231
they're all too complicated.

964
00:54:11,771 --> 00:54:14,261
I wanna explain that reference briefly
for people who aren't familiar.

965
00:54:14,861 --> 00:54:18,911
Gelman and Amnesia, that refers to
the idea of, like, you're reading a

966
00:54:18,911 --> 00:54:22,721
newspaper and you see a story about your
professional industry or something you're

967
00:54:22,721 --> 00:54:26,501
very knowledgeable about, and you see
everything that it got wildly wrong, and

968
00:54:26,501 --> 00:54:29,601
you're like, "Who even wrote this? What
are they doing? These incompetents."

969
00:54:29,631 --> 00:54:32,921
And then you read the next article, and
you're not familiar with it, and you

970
00:54:32,921 --> 00:54:35,861
just take it super credulously, and it's
like, "Oh, yes, look at these," you know.

971
00:54:36,101 --> 00:54:39,701
It's a, like, I think that's a pretty
good way of thinking about how people

972
00:54:39,701 --> 00:54:41,021
are talking about AI right now, yeah.

973
00:54:41,441 --> 00:54:45,581
And I think, from my perspective,
go hard as long as you can.

974
00:54:45,911 --> 00:54:51,161
I think the bubble is going to
collapse in the next one to two years.

975
00:54:51,731 --> 00:54:56,531
Tokens are gonna become super expensive,
and we're not gonna use as many of these

976
00:54:56,531 --> 00:55:00,641
tools long-term as we are right now, but
we're gonna have a lot of infrastructure,

977
00:55:00,761 --> 00:55:02,591
like, just like we did after the dot-com.

978
00:55:02,831 --> 00:55:05,261
That's part of why I've been
really interested in local

979
00:55:05,261 --> 00:55:06,611
models, because I agree with that.

980
00:55:06,611 --> 00:55:10,301
Like, they're being heavily subsidized
right now, and I think in, I don't know,

981
00:55:10,301 --> 00:55:11,771
about two years, I don't know, we'll see.

982
00:55:11,771 --> 00:55:15,101
It's been very ... People are talking
about it, it feels like the wheels

983
00:55:15,101 --> 00:55:19,571
are falling off right now with, like,
ClaudeCode is just clamped down usage on

984
00:55:19,571 --> 00:55:21,941
the 20 d- $20 a month plan aggressively.

985
00:55:22,331 --> 00:55:28,361
They grew from nine billion ARR to
30 billion ARR in a quarter, so I'm

986
00:55:28,451 --> 00:55:30,081
imagine they have some scaling issues.

987
00:55:30,711 --> 00:55:31,041
Right?

988
00:55:31,371 --> 00:55:32,451
Yeah, clearly.

989
00:55:32,751 --> 00:55:35,901
One of the points I made in that draft
blog post that I'm hoping to publish this

990
00:55:35,901 --> 00:55:41,331
week, the cats out of the bag, like, even
if OpenAI and Anthropic were to utterly

991
00:55:41,331 --> 00:55:45,531
and completely collapse and go out of
business today, the technology exists.

992
00:55:46,101 --> 00:55:53,091
And whether it's, you know, boutique
hosting of models or local LLMs, the

993
00:55:53,091 --> 00:55:54,381
technology is not going to go away.

994
00:55:54,411 --> 00:55:59,301
And even if it only, even if it
never gets better at all, the t- the

995
00:55:59,361 --> 00:56:04,821
capabilities stay exactly the way they
are today, it's still curly enough

996
00:56:04,821 --> 00:56:06,681
to upend large parts of our society.

997
00:56:07,251 --> 00:56:10,521
So then the question becomes, w-
what are you doing with it and

998
00:56:10,521 --> 00:56:11,871
how do you, how do you respond?

999
00:56:12,321 --> 00:56:15,741
Well, that's a good ... We should wrap up
soon, but that is an excellent jumping off

1000
00:56:15,741 --> 00:56:17,931
point for our last little subject here.

1001
00:56:17,931 --> 00:56:24,891
Uh, what do we think the impacts of LLMs
on engineering as an industry, software

1002
00:56:24,891 --> 00:56:29,781
engineering as an industry, and maybe more
broadly, societally, if we wanna go there?

1003
00:56:30,351 --> 00:56:32,721
I would say it's raising
the bar for everything.

1004
00:56:32,781 --> 00:56:36,411
Like, what counts as a minimum
viable product is gonna get

1005
00:56:36,411 --> 00:56:38,091
more fancy and more complicated.

1006
00:56:38,601 --> 00:56:43,161
What counts as a junior entry-level
engineer, or what the expectations are for

1007
00:56:43,161 --> 00:56:48,921
a junior entry-level engineer are gonna
go up, and I think we're gonna need a lot

1008
00:56:48,951 --> 00:56:54,081
more, um, product sense, and a lot less
being just really good at writing code.

1009
00:56:54,531 --> 00:56:55,671
Yeah, I'm inclined to agree.

1010
00:56:56,181 --> 00:56:59,751
I am honestly worried about the
impacts in society as a whole.

1011
00:57:00,051 --> 00:57:04,161
We've seen lots of debate and studies
about, you know, have, have cell phones

1012
00:57:04,161 --> 00:57:07,881
effectively destroyed the minds of
youth and the ability to focus and

1013
00:57:08,271 --> 00:57:12,651
social bul- like electronic bullying
and, like, very ... Like, why did teens

1014
00:57:12,651 --> 00:57:14,931
get very depressed starting in 2012?

1015
00:57:14,931 --> 00:57:20,091
Like, that sort of thing where
a technology possibly had

1016
00:57:20,751 --> 00:57:22,371
large-scale societal effects.

1017
00:57:23,061 --> 00:57:25,761
And I'm not saying that LLMs are,
you know, inherently destructive or

1018
00:57:25,761 --> 00:57:30,261
inherently bad, but a lot of times
the technology gets invented and

1019
00:57:30,261 --> 00:57:33,921
then it changes society in ways we
were very unable to predict early on.

1020
00:57:34,461 --> 00:57:38,841
In this case, I think we can predict
a number of ways, and there's probably

1021
00:57:38,841 --> 00:57:39,921
a whole lot of other stuff, too.

1022
00:57:40,341 --> 00:57:44,091
So I genuinely worry about college
students being able to get away

1023
00:57:44,091 --> 00:57:46,821
without having to develop critical
thinking skills even less than they

1024
00:57:46,821 --> 00:57:48,681
were able to get away with it before.

1025
00:57:49,071 --> 00:57:51,201
I have concerns about that sort of thing.

1026
00:57:51,801 --> 00:57:55,791
And, you know, I'm towards the tail
end of my career, I've done my thing,

1027
00:57:55,791 --> 00:57:57,741
I learned, I gained my experience.

1028
00:57:58,161 --> 00:58:01,341
I don't know how junior devs are
going to gain some of that experience.

1029
00:58:01,851 --> 00:58:04,521
Maybe it is they need
different experience.

1030
00:58:05,301 --> 00:58:08,331
I don't know what the job
pathways look like at that point.

1031
00:58:08,871 --> 00:58:11,541
So, like, I mean, don't get me
wrong, I'm excited to be able to

1032
00:58:11,541 --> 00:58:14,991
use AI to crank out some of the
things that I've had in my head.

1033
00:58:14,991 --> 00:58:21,081
I've found some of the same joy in being
able to have a multi-hour flow session

1034
00:58:21,081 --> 00:58:25,791
and build stuff just with AI editing
the files instead of me and my fingers.

1035
00:58:26,511 --> 00:58:29,881
But I can also look around and say,
"Yeah, there's a bunch of unexpected

1036
00:58:29,881 --> 00:58:31,441
consequences floating around as well."

1037
00:58:31,891 --> 00:58:35,881
I mean, in terms of experience,
it's not like I know how to code

1038
00:58:35,881 --> 00:58:39,781
without a compiler, and I would
definitely not enjoy coding without

1039
00:58:39,781 --> 00:58:41,191
a, without garbage collection.

1040
00:58:41,521 --> 00:58:47,251
Yeah, I think that's closer
to my theory on this.

1041
00:58:47,311 --> 00:58:50,911
I think it will, you know, the societal
impacts are gonna be something a lot.

1042
00:58:50,911 --> 00:58:51,451
We'll see.

1043
00:58:51,961 --> 00:58:55,111
I think the societal impacts will come
mostly not from software engineering, so

1044
00:58:55,111 --> 00:58:59,551
I think I don't understand those quite
well enough to opine on it too much.

1045
00:58:59,971 --> 00:59:02,641
I do care a lot about, like,
sociology and economics and whatever.

1046
00:59:02,641 --> 00:59:06,001
So I do have thoughts and opinions, but
I th- maybe it's a, maybe it's my Gelman

1047
00:59:06,001 --> 00:59:09,661
new amnesia of like, I know those well
enough that I'm like, "Nope, that's way

1048
00:59:09,661 --> 00:59:13,231
too complicated. I'm not no opinion." But
I think we take for granted a little bit

1049
00:59:13,231 --> 00:59:18,361
how much we're standing on the shoulders
of giants already and like, this is a big

1050
00:59:18,361 --> 00:59:20,971
giant to climb the shoulders of for sure.

1051
00:59:21,001 --> 00:59:22,651
Like, not trying to discount that.

1052
00:59:22,921 --> 00:59:25,051
People talk about how it's gonna
change our brains and like, you

1053
00:59:25,051 --> 00:59:26,521
know, destroy humanity and whatever.

1054
00:59:26,521 --> 00:59:30,541
And like, I don't know, like, we
kind of did a lot of that already.

1055
00:59:30,811 --> 00:59:33,691
Like, can you navigate
in your city without GPS?

1056
00:59:34,351 --> 00:59:39,151
There's a lot of stuff that we've
already lost that used to be

1057
00:59:39,151 --> 00:59:42,121
like, canonically like, you know,
you used to have a navigator.

1058
00:59:42,121 --> 00:59:44,761
If you wanted to do a road trip,
like, you would have spent a week

1059
00:59:44,761 --> 00:59:47,791
planning a route and then, like,
God help you if you miss a turn.

1060
00:59:48,361 --> 00:59:51,451
Same thing for, like, finance
and, like, the spreadsheet.

1061
00:59:51,781 --> 00:59:54,601
I don't know, this is gonna be, this is a
broad, more broadly applicable thing than

1062
00:59:54,601 --> 00:59:57,781
any of those past technological leaps.

1063
00:59:58,471 --> 01:00:01,921
Another thing that I learned recently,
going on the societal level again,

1064
01:00:03,031 --> 01:00:06,751
there was a ... In 2023, there was an,
like, international, like, literacy

1065
01:00:06,931 --> 01:00:09,181
survey done, levels and whatever.

1066
01:00:10,081 --> 01:00:15,481
And I think level one was can
read, but struggles with, like,

1067
01:00:15,481 --> 01:00:20,311
finding knowledge in text or, like,
following multi-step instructions.

1068
01:00:20,731 --> 01:00:25,411
And 28% of Americans m- met
that were, you know, were tested

1069
01:00:25,411 --> 01:00:26,671
at that level of literacy.

1070
01:00:26,671 --> 01:00:30,691
And, like, it said anything below level
three is considered partially illiterate.

1071
01:00:31,771 --> 01:00:35,431
And the stats were half, half-

1072
01:00:35,431 --> 01:00:39,721
54% of Americans are less than
sixth grade reading level.

1073
01:00:39,991 --> 01:00:41,941
They will not do well with LLMs.

1074
01:00:42,361 --> 01:00:42,661
Right.

1075
01:00:42,991 --> 01:00:45,091
But, like, you know, we
did that before LLMs.

1076
01:00:45,181 --> 01:00:46,261
That was 2023.

1077
01:00:46,651 --> 01:00:47,701
Like, okay, sure.

1078
01:00:47,701 --> 01:00:50,911
Like, there's gonna be weird shit
going on since then now, but, like,

1079
01:00:50,911 --> 01:00:54,361
God, we already ... Like, as far as
destroying society, like, we've done

1080
01:00:55,021 --> 01:00:57,121
a pretty good job of that without AI.

1081
01:00:57,271 --> 01:01:00,691
And for context, that's
54% of adult Americans.

1082
01:01:00,901 --> 01:01:02,551
It doesn't, like, count
children and stuff.

1083
01:01:03,031 --> 01:01:03,271
Yeah.

1084
01:01:03,481 --> 01:01:03,841
I don't know.

1085
01:01:03,841 --> 01:01:07,081
It's just, like, anytime I hear
people talk about, like, the downfall

1086
01:01:07,081 --> 01:01:09,511
of, like, civilization or whatever,
it's like, "I don't know. Have

1087
01:01:09,511 --> 01:01:14,611
you looked at civilization lately?
It's not doing so great." So, I

1088
01:01:14,611 --> 01:01:15,661
don't know, it is gonna be a lot.

1089
01:01:15,661 --> 01:01:17,911
It's gonna be challenging and difficult.

1090
01:01:17,971 --> 01:01:21,241
I guess going back to engineering,
software engineering, you remember

1091
01:01:21,241 --> 01:01:25,271
the '90s and 2000s, 2010s era,
like, stereotypes of software

1092
01:01:25,271 --> 01:01:29,231
engineers, like, it's, like, super
autistic, like, zero social skills.

1093
01:01:29,291 --> 01:01:30,611
They are a human computer.

1094
01:01:31,571 --> 01:01:34,091
And now we don't have that anymore.

1095
01:01:34,121 --> 01:01:37,451
We, you know, democratize software
engineering and now everyone does it.

1096
01:01:37,511 --> 01:01:41,591
We've always tried to get
something basically like an LLM.

1097
01:01:41,621 --> 01:01:46,121
Like, that's what people wanted out
of software engineers this whole time.

1098
01:01:46,991 --> 01:01:51,251
So, I think it's gonna be ... I
like the analogy of, like, compilers

1099
01:01:51,251 --> 01:01:54,911
and garbage collection and stuff,
because it's just a new tool.

1100
01:01:54,911 --> 01:01:58,091
It's a new level of abstraction,
and now it's much further

1101
01:01:58,091 --> 01:02:00,581
removed from the hardware.

1102
01:02:02,351 --> 01:02:03,251
But, yeah, I don't know.

1103
01:02:03,281 --> 01:02:06,011
People talk about, like, the d- it's
gonna destroy, like, the upskilling, it's

1104
01:02:06,011 --> 01:02:09,491
gonna destroy junior engineers, and it's,
there's not gonna be any pathway in.

1105
01:02:09,551 --> 01:02:13,271
And, like, I don't know, maybe
that's true, but at the end of

1106
01:02:13,271 --> 01:02:17,831
the day, I'm gonna end on, like, a
positive thought or, like, advice.

1107
01:02:18,431 --> 01:02:20,861
Like, this is all communication.

1108
01:02:20,951 --> 01:02:23,081
It's just more and more communication.

1109
01:02:23,141 --> 01:02:26,951
If you wanna lock in your software
engineering career, get good at

1110
01:02:26,951 --> 01:02:31,901
communicating, get good at deeply
understanding and articulating

1111
01:02:31,961 --> 01:02:37,181
a problem, not the code, not the
patterns necessarily, but, like,

1112
01:02:37,211 --> 01:02:42,071
understand what an architecture
is good at or bad at or what the

1113
01:02:42,071 --> 01:02:45,011
performance profile of this or that are.

1114
01:02:45,011 --> 01:02:47,471
And, like, now, you don't even need
to understand it at the same level.

1115
01:02:47,471 --> 01:02:51,611
Like, I've been really excited about
AI is because I have such a broad and

1116
01:02:51,611 --> 01:02:55,241
shallow expertise, and now I can use it.

1117
01:02:55,481 --> 01:02:59,261
Like, it used to be that I had all
this, like, vague notional expertise

1118
01:02:59,291 --> 01:03:03,611
in a wide range of stuff, and it's
like, "Damn, I wish I could use that,

1119
01:03:03,611 --> 01:03:07,991
but I would need a team of 15 people
in order to be productive with this.

1120
01:03:08,351 --> 01:03:09,461
And now I don't.

1121
01:03:09,731 --> 01:03:11,441
So, like, I don't know,
that's exciting for me.

1122
01:03:11,501 --> 01:03:13,541
And I think that's just
generally exciting.

1123
01:03:13,541 --> 01:03:18,581
So if you're just good at communicating
and you learn a domain and you get good at

1124
01:03:18,581 --> 01:03:22,541
articulating problems within that domain,
then, like, I think you're gonna be fine.

1125
01:03:22,991 --> 01:03:27,641
I think superpower right now is if you're
a domain expert who can kind of code.

1126
01:03:28,211 --> 01:03:31,511
There is certainly still value
in, you know, greater engineering

1127
01:03:31,511 --> 01:03:34,181
expertise and understanding how
to ship stuff to production.

1128
01:03:34,361 --> 01:03:36,911
At some scale, that
becomes a domain expertise.

1129
01:03:37,181 --> 01:03:37,481
True.

1130
01:03:38,081 --> 01:03:38,381
Yeah.

1131
01:03:38,501 --> 01:03:39,101
Extremely true.

1132
01:03:39,731 --> 01:03:40,751
So yeah, I don't know.

1133
01:03:41,051 --> 01:03:41,891
Uh, I'm excited.

1134
01:03:41,891 --> 01:03:42,011
I

1135
01:03:42,011 --> 01:03:43,061
can build more stuff.

1136
01:03:43,541 --> 01:03:45,731
I can build so much ... I
have built so much more.

1137
01:03:45,791 --> 01:03:48,641
I've explored so many more things
and gone, "Oh, this is harder than

1138
01:03:48,641 --> 01:03:51,671
I thought. I'm gonna set it down."
Instead of just never doing it and,

1139
01:03:51,671 --> 01:03:54,071
like, fantasizing about what it
could have been for years, you know?

1140
01:03:54,281 --> 01:03:54,491
Yeah.

1141
01:03:54,551 --> 01:03:54,851
Same.

1142
01:03:55,151 --> 01:03:55,511
All right.

1143
01:03:55,571 --> 01:03:56,081
Cool.

1144
01:03:56,561 --> 01:03:57,191
Thank you all.

1145
01:03:57,251 --> 01:03:58,451
Thank you both for joining me.

1146
01:03:58,481 --> 01:04:00,161
Thank you everyone in the
audience for listening.

1147
01:04:00,431 --> 01:04:01,271
Appreciate it a lot.

1148
01:04:01,571 --> 01:04:02,591
Hopefully this was informative.

1149
01:04:02,861 --> 01:04:03,191
Cool.

1150
01:04:03,221 --> 01:04:03,551
All right.

1151
01:04:03,551 --> 01:04:06,311
Well, this has been this
month in React, quote unquote.

1152
01:04:06,491 --> 01:04:11,051
I think we may be exploring different
formats in the future because, I don't

1153
01:04:11,051 --> 01:04:15,371
know, who's following individual software
library development as closely anymore.

1154
01:04:15,881 --> 01:04:17,561
But yeah, thanks so much for listening.

1155
01:04:17,651 --> 01:04:20,681
If this is a show that you get value
from, please send it to a coworker,

1156
01:04:20,711 --> 01:04:22,361
send it to a friend, leave us a review.

1157
01:04:22,661 --> 01:04:24,791
And yeah, I'll see you next month.

1158
01:04:25,031 --> 01:04:25,301
All right.

1159
01:04:25,421 --> 01:04:25,931
Take care.