1
00:00:00,010 --> 00:00:02,161
Welcome to another episode of Empower Apps.

2
00:00:02,161 --> 00:00:04,076
I'm your host, Leo Dion.

3
00:00:04,076 --> 00:00:07,201
Today I'm joined with Graham Lee. Graham,

4
00:00:07,201 --> 00:00:09,855
Thank you so much for joining us again.

5
00:00:10,319 --> 00:00:11,166
it's great to be back.

6
00:00:11,166 --> 00:00:12,172
Thanks for having me back.

7
00:00:12,172 --> 00:00:13,629
yeah, how are you doing?

8
00:00:13,799 --> 00:00:15,300
Good, good, good.

9
00:00:15,430 --> 00:00:18,742
it was good to catch up iOSDevUK.

10
00:00:18,856 --> 00:00:29,065
and I really liked your presentation and I think, you know, we've done a lot of episodes
already on AI, but I think there's there's a I think you bring an interesting perspective

11
00:00:29,065 --> 00:00:30,186
I wanted to cover today.

12
00:00:30,186 --> 00:00:35,310
but before we get into that, let's go ahead and I'll let you go ahead and introduce
yourself.

13
00:00:35,847 --> 00:00:36,517
Sure, thank you.

14
00:00:36,517 --> 00:00:37,958
so I'm Graham.

15
00:00:37,958 --> 00:00:39,359
I live in the UK.

16
00:00:39,359 --> 00:00:44,000
I it's pretty hard for me to keep track of what I do 'cause it keeps changing a lot.

17
00:00:44,000 --> 00:00:56,284
But at the moment I am running a kind of one person company called Chiron Codex, that is
about understanding what the kind of shift towards LLM-based tools and AI augmented

18
00:00:56,284 --> 00:01:01,205
software engineering means for us as software engineers and means for our industry.

19
00:01:01,233 --> 00:01:08,893
So I'm writing a book that's about interaction patterns in AI augmented software
development.

20
00:01:08,893 --> 00:01:11,198
I've got an app called Axminster.

21
00:01:11,198 --> 00:01:12,879
I'm sure we'll talk about that later.

22
00:01:12,879 --> 00:01:15,630
That's in review at the moment, so it might be out.

23
00:01:15,630 --> 00:01:21,974
Hoping it's be gonna be out by the time that this episode airs, but you can you never tell
with the App Store review process.

24
00:01:21,974 --> 00:01:27,617
do some like consultancy for engineering organizations helping them to like

25
00:01:28,739 --> 00:01:35,084
make sure that their processes are going to be things that they actually want to improve
by using AI.

26
00:01:35,084 --> 00:01:42,558
Yeah, the way I think of it is that like AI can be an accelerator, and acceleration has
both an amount and a direction, right?

27
00:01:42,558 --> 00:01:47,832
So you've got to like make sure you're pointing in the right direction before you kind of
you know push the pedal down.

28
00:01:47,832 --> 00:01:50,443
otherwise you're just gonna do the wrong thing much faster.

29
00:01:51,281 --> 00:01:53,544
Yeah, yeah, that's a good point.

30
00:01:56,119 --> 00:02:04,093
What's your his how has your perspective changed on AI since the beginning of the decade,
I guess?

31
00:02:04,093 --> 00:02:10,915
And were you surprised by its rise or were you like or what what has surprised you what
has shocked you?

32
00:02:10,915 --> 00:02:12,376
What do you think people get wrong?

33
00:02:12,376 --> 00:02:13,616
That kind of stuff.

34
00:02:14,050 --> 00:02:18,436
No, that's a really interesting topic to explore because it has changed quite a lot.

35
00:02:18,436 --> 00:02:27,353
I mean I started doing what we was then called machine learning back in about twenty
sixteen I think.

36
00:02:27,353 --> 00:02:35,861
and I was working for a financial technology startup in the the realm of giving financial
advice.

37
00:02:35,861 --> 00:02:38,853
So it was like software mediated financial advice.

38
00:02:38,999 --> 00:02:41,722
to like individual people.

39
00:02:42,323 --> 00:02:45,722
And in the UK, like I'm sure it's regulated in many countries.

40
00:02:45,722 --> 00:02:50,089
I'll just quickly explain that like in the UK, financial advice is regulated.

41
00:02:50,089 --> 00:02:53,012
You have to be a professional financial advisor to do it.

42
00:02:53,012 --> 00:02:57,857
And you have to basically sign your name to the advice you give.

43
00:02:57,857 --> 00:03:05,728
And it's like your reputation and your career that's on the line if that advice turns out
to be like wrong or fraudulent or anything like that.

44
00:03:05,728 --> 00:03:15,444
and that has meant there's quite a bit of in some ways risk aversion towards adopting new
software, but it also means that there's a lot of opportunity because someone who is

45
00:03:15,444 --> 00:03:22,798
willing to like try out the software can basically do whatever they want as long as
they're willing to read the thing that comes out at the end of it and sign their name on

46
00:03:22,798 --> 00:03:22,908
it.

47
00:03:22,908 --> 00:03:27,973
so yeah, about ten years ago, we kind of built a machine learning model.

48
00:03:27,973 --> 00:03:30,388
That was like back when you like literally took

49
00:03:30,388 --> 00:03:41,254
a bunch of CSV files and a Python script running a module called pandas or something
like that and like made your own neural network and then trained it from scratch.

50
00:03:41,254 --> 00:03:53,665
And we trained it on the case history of like a particular financial advisor so that we
could categorize what advice they would give in like you know in another situation.

51
00:03:53,665 --> 00:03:58,185
Or we could train it on another financial advisor and compare the two and see if they were
consistent.

52
00:03:58,185 --> 00:04:03,149
see if a particular document was like an outlier for compliance purposes, all this kind of
stuff.

53
00:04:03,149 --> 00:04:08,734
It was a load of work and like the results were kind of you know, they they were good.

54
00:04:08,734 --> 00:04:16,850
It was interesting that they it could do it, but it like a huge risk to kind of you know,
base your entire financial advice business on this.

55
00:04:16,850 --> 00:04:21,363
It was a a useful kind of side tool, like a a kind of cross check.

56
00:04:22,172 --> 00:04:29,526
And then when generative AI and large language models came out, firstly I'll hands up, I
did not see that coming.

57
00:04:29,526 --> 00:04:36,090
Yeah, we were in the realm of an AI tool could tell you whether a picture had a hot dog in
it, eighty five percent of the time, and they'd be

58
00:04:36,090 --> 00:04:38,312
wrong the completely wrong the other fifteen percent.

59
00:04:38,312 --> 00:04:45,032
And then we suddenly went to you can like do some kind of cool demo, so you can get it to
give it give you a summary

60
00:04:45,032 --> 00:04:53,080
of a Shakespeare play as a limerick, or you can get it to write a Valentine's card with
the name and some interesting attributes about like someone.

61
00:04:53,080 --> 00:05:02,952
but at no point did I think yeah this is going to be kind of useful for generating working
software until the point where it was.

62
00:05:02,952 --> 00:05:03,369
yeah

63
00:05:03,369 --> 00:05:10,391
and even when I was kind of like evaluating these tools kind of on software teams as part
of like a software engineering experience, I was going,

64
00:05:10,391 --> 00:05:17,214
Yeah, that yeah, it can generate code, but the code doesn't compile, it doesn't do the
right thing, this is never gonna be useful.

65
00:05:17,214 --> 00:05:18,135
wait, hold on a second.

66
00:05:18,135 --> 00:05:26,083
It suddenly became useful and there was a kind of shift that was imperceptible when you're
in it, but looking back on it was very quick.

67
00:05:26,083 --> 00:05:34,002
sometime around twenty twenty three, they went from being interesting demos to actually
useful, valuable tools.

68
00:05:34,200 --> 00:05:35,071
Right, right.

69
00:05:35,071 --> 00:05:37,782
Everybody was adding it to their website and their app.

70
00:05:37,782 --> 00:05:38,363
~

71
00:05:38,363 --> 00:05:49,000
and I so one of the things too I've noticed is a lot of people have a negative and not I'm
not disagreeing with this, but a lot of how people have a negative opinion of AI because

72
00:05:49,000 --> 00:05:57,836
they're kind of force-fed it and it's usually like prose, which it's getting better at,
but it's still not great.

73
00:05:57,836 --> 00:06:02,539
Image generation, video generation, music gener right, that kind of stuff.

74
00:06:02,539 --> 00:06:03,567
And it's like

75
00:06:03,567 --> 00:06:07,768
That stuff is okay to garbage.

76
00:06:08,289 --> 00:06:17,812
And those of us who are like software developers, I think part of it is just the people
who make this stuff are software developers.

77
00:06:17,812 --> 00:06:20,433
So it's they understand what they're training, right?

78
00:06:20,433 --> 00:06:32,718
But like for those of us, it's like wow, like the unless you're philosophically against
AI, like it is extremely useful.

79
00:06:32,760 --> 00:06:36,032
For a lot of people getting a lot of code out.

80
00:06:36,173 --> 00:06:40,356
a and there is there is like, as we'll talk about today, there is a right way to do it.

81
00:06:40,817 --> 00:06:51,638
so like yeah, it seems like that to us it's like, yeah, we're on the cutting edge of where
AI will be, whereas everybody else is kind of experiencing like dregs of AI training, I

82
00:06:51,638 --> 00:06:52,458
guess.

83
00:06:53,436 --> 00:07:06,533
Yeah, and I I think that there's there's also a part of that reaction and it's like you
know, it's totally like valid reaction and yeah, I don't want to kind of like sound like

84
00:07:06,853 --> 00:07:13,577
kind of sort you know AI boosters saying, right, this is gonna like change the way that we
do the entire society and yeah.

85
00:07:13,819 --> 00:07:16,350
you know, there are contexts where it makes more sense.

86
00:07:16,350 --> 00:07:19,483
And like what I want to kind of come back to is like the

87
00:07:19,483 --> 00:07:20,704
the idea of creativity.

88
00:07:20,704 --> 00:07:22,545
the reason AI music does suck.

89
00:07:22,545 --> 00:07:25,636
And the reason that AI music sucks is that it is generic.

90
00:07:25,636 --> 00:07:30,057
you say, I want, a a three and a half minute hip hop track.

91
00:07:30,057 --> 00:07:36,381
It will give you like what is the most middle of the road three and a half minutes of hip
hop like you can possibly imagine.

92
00:07:36,381 --> 00:07:46,706
and there is yeah, there are kind of different s kind of schools of thought on whether
software is like you know an an art a kind of creative

93
00:07:46,994 --> 00:07:50,338
mindset or whether it's a tool, whether it's like a constructive thing.

94
00:07:50,338 --> 00:07:52,270
And the fact is that both are true.

95
00:07:52,270 --> 00:07:54,752
Like you know, software is engineering.

96
00:07:54,752 --> 00:08:03,957
That is the combination of you know art and design and science to creating artifacts that
people use and want to use.

97
00:08:03,957 --> 00:08:11,091
A lot of software is entirely what you might call like blue-collar code, right?

98
00:08:11,091 --> 00:08:12,948
You know, if if what you're doing

99
00:08:12,948 --> 00:08:25,333
is taking some JSON from a web service, maybe paginating it, caching it, like and then
kind of ordering it by date so that you can show it in some kind of timeline feed.

100
00:08:25,614 --> 00:08:32,656
The bit where you design the timeline feed and what the user can do with that is you know
creative and artistic.

101
00:08:32,656 --> 00:08:38,088
And the bit where you just like load some JSON from a web service is entirely generic.

102
00:08:38,088 --> 00:08:39,008
And it doesn't matter

103
00:08:39,008 --> 00:08:40,885
whether that software is

104
00:08:40,885 --> 00:08:46,526
generic in fact putting too much creativity in there is probably the wrong thing to do
because we we know how to solve that problem.

105
00:08:46,526 --> 00:08:50,609
We've solved it three thousand million times in the last, tw six months or whatever.

106
00:08:50,826 --> 00:08:51,637
Right, right, right.

107
00:08:51,637 --> 00:08:57,251
isn't the joke that iOS development is basically you have fancy formatter for JSON,

108
00:08:57,251 --> 00:08:58,099
Ha ha.

109
00:08:58,217 --> 00:09:03,254
That's what I've heard like non I guess non mobile developers kinda say that.

110
00:09:03,254 --> 00:09:04,699
So like yeah.

111
00:09:04,699 --> 00:09:10,277
yeah, been through parts of my career where all I'm doing is putting JSON in a table view
over and over again.

112
00:09:10,277 --> 00:09:11,351
Yeah.

113
00:09:11,915 --> 00:09:15,500
So let's talk let's so let's talk about that and I I like this term.

114
00:09:15,500 --> 00:09:19,590
I think it's still valid the software centaur.

115
00:09:19,590 --> 00:09:25,905
I'll let you go ahead and explain that and kind of where we are as far as that's
concerned.

116
00:09:26,101 --> 00:09:26,511
Sure.

117
00:09:26,511 --> 00:09:37,897
So yeah, Garry Kasparov, who was the Russian chess grandmaster who was beaten by an AI
system and not a large language model, a different like approach to AI, in the nineteen

118
00:09:37,897 --> 00:09:41,697
eighties, which was an IBM call computer called Deep Blue.

119
00:09:41,697 --> 00:09:50,149
He created this term the centaur to refer to humans whose capabilities were

120
00:09:50,149 --> 00:09:52,270
augmented by AI.

121
00:09:52,270 --> 00:09:59,273
And what he was referring to, or like he was kind of extrapolating from his experiences in
chess.

122
00:09:59,273 --> 00:10:10,908
Like every chess grandmaster who's playing today has much higher scores than Garry
Kasparov ever achieved, like, yeah, when i in his professional career and all of his

123
00:10:10,908 --> 00:10:11,978
contemporaries.

124
00:10:11,978 --> 00:10:18,448
Like they are better and the the score is kind of you know like weighted by based on

125
00:10:18,448 --> 00:10:20,820
Who you win against and what this their score is and everything.

126
00:10:20,820 --> 00:10:27,905
So it's the people playing chess today are objectively better at playing chess than people
who were playing chess in the 1980s.

127
00:10:27,905 --> 00:10:28,926
Why is that?

128
00:10:28,926 --> 00:10:41,676
It's because they now use the AI tools that defeated Kasparov in the 80s to play chess, to
learn how to play chess, to develop new strategies and to get to understand positions and

129
00:10:41,676 --> 00:10:43,298
to get better at chess.

130
00:10:43,298 --> 00:10:46,820
So a centaur in the chess world.

131
00:10:46,820 --> 00:10:58,633
realm is a better chess player than any human who doesn't use AI because they're actually
expanding their craft, they're expanding their tool set and their understanding using

132
00:10:58,633 --> 00:10:58,974
chess.

133
00:10:58,974 --> 00:11:02,504
So my question is what's a software centaur look like?

134
00:11:02,504 --> 00:11:02,704
Yeah.

135
00:11:02,704 --> 00:11:09,466
What what is what is the way to become a better software engineer through using AI tools?

136
00:11:09,466 --> 00:11:13,926
Because the alternative, which is a fear that a lot of people have and it's justified, is

137
00:11:13,926 --> 00:11:24,025
that we become worse software engineers because every time we get a problem we just paste
the problem into the you know into like Claude Code or whatever tool we're using and then

138
00:11:24,025 --> 00:11:32,852
like hit yes every time it asks a permission question and then commit the result or like
ship the result to the App Store without understanding what's happened in the intervening

139
00:11:32,852 --> 00:11:33,652
time.

140
00:11:35,468 --> 00:11:37,130
isn't the answer yes?

141
00:11:37,731 --> 00:11:38,932
Isn't the answer

142
00:11:38,932 --> 00:11:46,180
yes, software will get worse and yes, software will get better because depending on the
company and w the willingness.

143
00:11:46,360 --> 00:11:47,191
Yeah, absolutely.

144
00:11:47,191 --> 00:11:51,654
So there will be some places that want to kind of get better at craft.

145
00:11:51,654 --> 00:11:56,067
There'll be some places that want to like just ship more and ship faster.

146
00:11:56,067 --> 00:11:56,957
Yeah.

147
00:11:56,957 --> 00:11:59,590
there'll be some places that just want to fire a bunch of people.

148
00:11:59,590 --> 00:12:05,768
yeah, so that's another point is you know, in my opinion yeah, i it in in my opinion,

149
00:12:05,768 --> 00:12:08,079
like technology has never stolen a job, right?

150
00:12:08,079 --> 00:12:10,411
You know, employers have stolen jobs and they

151
00:12:10,411 --> 00:12:11,682
might use technology to do it.

152
00:12:11,682 --> 00:12:13,523
But anyway, like

153
00:12:13,792 --> 00:12:16,534
Yes, there will be a vast variety of experiences.

154
00:12:16,534 --> 00:12:20,066
and so it depends on to some extent and always has done.

155
00:12:20,066 --> 00:12:28,822
Whenever people have said like, TDD is the way that you need to do software engineering
or you need to use functional programming or like whatever this week's intervention is,

156
00:12:28,822 --> 00:12:37,608
you know, you need to use like memory safe languages or type safe languages or whatever,
well you have to demonstrate that that's better and like you have to demonstrate that you

157
00:12:37,608 --> 00:12:39,649
create better products as a result.

158
00:12:39,649 --> 00:12:41,830
So if you know, if we want to say

159
00:12:42,548 --> 00:12:45,546
there's a better way of doing software using AI.

160
00:12:45,546 --> 00:12:51,818
We kind of have to qualify what it means to be better and like demonstrate that
qualification.

161
00:12:54,019 --> 00:12:56,833
so yeah, let's talk about that.

162
00:12:56,833 --> 00:13:04,792
We'll jump right in and say what what are some keys to making better software with AI?

163
00:13:06,351 --> 00:13:21,320
So it gives us the opportunity to use all of our understanding of AI, also to sorry, all
of our understanding of software, but also in some senses to use all if everybody else's

164
00:13:21,320 --> 00:13:22,580
understanding of software.

165
00:13:22,580 --> 00:13:26,973
Like, yeah, these things are trained on like you know, huge amounts of knowledge.

166
00:13:26,973 --> 00:13:34,839
And that means that like if we understand sort of how our software development process
works as a system, but we maybe don't

167
00:13:34,839 --> 00:13:42,872
understand one particular aspect of it, we can use these tools to do a passable job at
that.

168
00:13:42,872 --> 00:13:48,535
So like, you know, as I said, when you ask it to generate music, you get super gener
generic music out.

169
00:13:48,535 --> 00:13:56,039
but like if I don't understand a particular part of software engineering, I'm not that
good at operations.

170
00:13:56,039 --> 00:13:59,841
Like, you know, I've written a couple of like apps that have

171
00:13:59,978 --> 00:14:09,033
Docker containers and Kubernetes clusters and all of the buzzwords in and I've either like
well in fact every time I've done that, professionally, I've been working in a company

172
00:14:09,033 --> 00:14:16,067
where there are ops people who can help me get it right or can like you know give me a
platform on which I do that right.

173
00:14:16,067 --> 00:14:28,586
But like if I if I'm working on it and I want to just like kind of produce a prototype or
maybe I'm working in a team where everybody has like iOS experience and nobody has

174
00:14:28,586 --> 00:14:35,470
the kind of depth of back end skills to get that operations part, that kind of deployment
part correct.

175
00:14:35,470 --> 00:14:37,171
We can do an average job of that now.

176
00:14:37,171 --> 00:14:42,493
Yeah, we can do a generic Kubernetes back end without actually understanding what it is.

177
00:14:42,493 --> 00:14:48,356
I'm also like really bad at yeah, user experience and design.

178
00:14:48,356 --> 00:14:54,639
I have no graphical skills at all, but I can do an average job of it now.

179
00:14:54,639 --> 00:14:54,980
And I

180
00:14:54,980 --> 00:14:58,047
can also like see when the model

181
00:14:58,047 --> 00:15:01,518
I know, I can generate something that looks bad and go, okay, that looks bad.

182
00:15:01,518 --> 00:15:06,269
I know that I actually need to go and find a pre professional to do this.

183
00:15:06,269 --> 00:15:16,792
So I can go and collaborate with an ops person if my like generic backend stuff doesn't
work, or I can go and collaborate with like a designer if my like generic approach to

184
00:15:16,792 --> 00:15:18,033
design doesn't work.

185
00:15:18,033 --> 00:15:26,917
But I have the opportunity to at least like explore that space and understand my problem
in much more detail.

186
00:15:26,917 --> 00:15:34,065
Rather than just treating a designer as a completely alien person who speaks a different
language and going, can you just make this look good?

187
00:15:34,065 --> 00:15:38,676
I can say, right, hi here's what I've tried and I can see that these things are bad.

188
00:15:38,685 --> 00:15:39,325
~

189
00:15:39,325 --> 00:15:44,801
yeah, let's now have the conversation where we combine our expertise to make a great
thing.

190
00:15:46,175 --> 00:15:57,338
So it sounds like you're saying like the floor has been raised, but like how and speed is
faster, but how can we get better quality?

191
00:15:57,338 --> 00:16:11,387
Is it just basically what you're saying is everybody has more I'm trying to like put this
all together, but everybody has more experience accumulated because AI brings that to the

192
00:16:11,387 --> 00:16:11,968
table?

193
00:16:11,968 --> 00:16:14,269
Is that is that correct?

194
00:16:15,105 --> 00:16:15,605
It's great.

195
00:16:15,605 --> 00:16:26,998
everybody has access to I mean, so like you know, t taking the internet, pre AI, everyone
had access to tons of information about writing software, right?

196
00:16:26,998 --> 00:16:35,090
Yeah, there was Stack Overflow, there's Experts Exchange, there were all sorts of like
forums on particular technologies, there were discussion groups, mailing lists, like

197
00:16:35,090 --> 00:16:36,821
Apple's developer forums.

198
00:16:37,041 --> 00:16:41,411
and you needed to like know what you were searching for, know where to search it.

199
00:16:41,411 --> 00:16:45,153
like know how to kind of make sense of what he was saying.

200
00:16:45,153 --> 00:16:54,109
Or, and this is relevant to the kind of LLM story, like take something that looked roughly
correct from Stack Overflow and paste it into your code and not understand how it worked.

201
00:16:54,109 --> 00:16:56,920
Like that was that that has been considered.

202
00:16:57,791 --> 00:17:01,404
I ~ well I I obviously don't know firsthand.

203
00:17:01,660 --> 00:17:07,512
but yeah but like it was quite a lot of work to do all of that research.

204
00:17:07,512 --> 00:17:13,077
Now doing like a kind of first pass of that research is really quick.

205
00:17:13,077 --> 00:17:19,173
You know, that that, if anything, is a more impactful use of AI than even generating code.

206
00:17:19,173 --> 00:17:24,658
It's just like generating answers to questions based on a corpus of training data.

207
00:17:24,658 --> 00:17:32,811
And we know that there's like hallucinations and we know that there's risks that the
answers aren't valid, so we still need to kind of apply our critical

208
00:17:32,811 --> 00:17:38,102
skills, but we get to the point where we have something that we can critically analyse way
earlier.

209
00:17:38,292 --> 00:17:41,563
so yeah, we have more access to a load of different skills.

210
00:17:41,563 --> 00:17:47,554
We also have the opportunity to use that to augment our own learning.

211
00:17:47,554 --> 00:17:51,616
Like a specific example, I do know something about software testing.

212
00:17:51,616 --> 00:17:56,157
Like I wrote a book on test driven development like a number of years ago.

213
00:17:56,398 --> 00:17:57,619
and so I know how to

214
00:17:57,619 --> 00:17:58,920
write automated tests.

215
00:17:58,920 --> 00:18:03,921
I know how to write UI tests and integration tests, unit tests, performance tests, blah
blah blah.

216
00:18:03,921 --> 00:18:07,922
There's a technique that I used to not use very much.

217
00:18:07,922 --> 00:18:14,725
I understood that it's value, but I just didn't really understand how to apply it called
mutation testing.

218
00:18:14,725 --> 00:18:25,379
And what mutation testing is, is you take your existing test suite and you take the
software that you're testing and you just change something in the test.

219
00:18:25,379 --> 00:18:26,385
So maybe like

220
00:18:26,385 --> 00:18:28,567
You've got sorry, you change something in the code.

221
00:18:28,567 --> 00:18:33,509
So maybe you've got some code that adds two numbers together and you subtract them
instead.

222
00:18:33,509 --> 00:18:37,352
Or you always return zero from that and you like don't actually do the addition.

223
00:18:37,352 --> 00:18:39,273
And you run your tests again.

224
00:18:39,974 --> 00:18:44,597
If your tests still pass, then your tests aren't very good because like

225
00:18:44,597 --> 00:18:50,261
you have some like behavior that you depend on in your app that doesn't actually get
tested.

226
00:18:50,261 --> 00:18:53,703
and so what you then do is you refine your

227
00:18:53,835 --> 00:18:59,955
your tests so that they would catch that problem and then you know that they're working as
like regression checks.

228
00:18:59,955 --> 00:19:10,035
And I didn't really do that before, but now, like maybe sort of two years ago, 18 months
ago or something, I just kind of prompted a model like, hey, how how would mutation

229
00:19:10,035 --> 00:19:12,205
testing work in this code base?

230
00:19:12,405 --> 00:19:21,066
And it like came up with a Python script that would like make changes to the code and then
it ran the tests and it said, okay, so these

231
00:19:21,066 --> 00:19:25,851
six things didn't get caught, so we're going to delete the tests that test them and
redesign them.

232
00:19:25,851 --> 00:19:28,393
And I was like, okay, now I understand this thing.

233
00:19:28,493 --> 00:19:28,874
And now

234
00:19:28,874 --> 00:19:31,837
I use mutation testing all the time on pro on my projects.

235
00:19:31,837 --> 00:19:38,163
It's given me the ability to apply a new technique to my code.

236
00:19:38,163 --> 00:19:45,750
Not because I've told, told told a model, hey, go off and do this, but because it's helped
me understand how to do it from how I get the value out of it.

237
00:19:46,211 --> 00:19:46,633
Right.

238
00:19:46,633 --> 00:19:50,304
It's it's almost like teaching by example, I guess.

239
00:19:50,587 --> 00:19:51,720
Yeah, totally.

240
00:19:51,720 --> 00:19:52,951
at that kind of teaching by example.

241
00:19:52,951 --> 00:19:56,131
They're also good at Socratic teaching.

242
00:19:56,131 --> 00:20:01,584
So you can get them to like just generate a bunch of questions about your code or about
your product or about ~

243
00:20:01,584 --> 00:20:11,459
you know, like a problem statement you've got from a customer or something, and just you
know, check that you actually understand everything that that is written down there.

244
00:20:11,640 --> 00:20:19,843
Yeah, when we did the episode with Donny, he was talking about this grill me skill that
I've now used that basically asks you a ton of questions to confirm

245
00:20:19,843 --> 00:20:20,603
stuff.

246
00:20:20,603 --> 00:20:21,723
So yeah.

247
00:20:21,723 --> 00:20:30,306
I do I do want to cover one more thing about how and I this is the part I thought was
interesting about your talk.

248
00:20:31,066 --> 00:20:37,260
How the history of computing and your experience with Cocoa and Objective-C

249
00:20:37,322 --> 00:20:43,214
affects how you look at AI and how LLMs have affected software development.

250
00:20:44,299 --> 00:20:45,589
Yeah, that's really interesting.

251
00:20:45,589 --> 00:20:49,812
I was reflecting on that this morning.

252
00:20:49,812 --> 00:21:02,923
What Objective-C or like object oriented programming in general is about modelling a
software system as a collection of like little independent computer programs that

253
00:21:02,923 --> 00:21:06,826
communicate by sending messages to get a big thing done.

254
00:21:06,826 --> 00:21:12,651
And a lot of the techniques that you use are effectively like role playing bits of
computer.

255
00:21:12,651 --> 00:21:12,871
Right?

256
00:21:12,871 --> 00:21:22,815
Yeah, you you like i if you're using what are called CRC cards, which is class
responsibility and collaborator, you're designing an object by saying, I've got this type

257
00:21:22,815 --> 00:21:26,037
of thing that's in my code, what does it do?

258
00:21:26,058 --> 00:21:29,180
And what other objects are going to talk to it?

259
00:21:29,340 --> 00:21:40,199
And within that context, like if you think of each kind of object as an independent actor
that's just receiving messages and then doing stuff and then responding.

260
00:21:40,544 --> 00:21:50,830
It's very easy to fit the idea of an LLM into that model because an LLM is you know
effectively an independent actor that receives stuff and then like does something with it,

261
00:21:50,830 --> 00:21:54,942
in this case, generating text or tool uses.

262
00:21:55,482 --> 00:22:03,135
so you can you it doesn't kind of make it this is about like writing code in a different
way.

263
00:22:03,135 --> 00:22:08,047
It makes it about this is using a computer in a different way, or even this is using a
computer

264
00:22:08,093 --> 00:22:12,662
In the same way I was already doing it, but with like different possibilities.

265
00:22:12,662 --> 00:22:25,473
and then, yeah, as we'll come on to and I mentioned in my introduction, like my book is
about patterns and the patterns and pattern languages of programming came from the object

266
00:22:25,473 --> 00:22:33,216
oriented programming world and it was a new approach to design that was based on
understanding how

267
00:22:33,216 --> 00:22:42,374
certain shapes of problem have shared solutions and how you can identify that those
problems are the same and apply the same solution to them.

268
00:22:42,936 --> 00:22:43,727
Yeah, yeah.

269
00:22:43,727 --> 00:22:52,944
Well, I like I like the part about how the Apple I, I don't know how many people know
that, but the Apple I is basically a kit, and it's like, yeah, you have to put your

270
00:22:52,944 --> 00:22:56,557
software all together and how that's rapidly changing.

271
00:22:58,378 --> 00:23:11,148
So yeah, like segueing from that, there's the concept of like having to build it from
scratch, and like one of the things you talk about is how

272
00:23:11,274 --> 00:23:17,870
The job was never about typing, it was never about putting the thing together so much as
delivering quality software.

273
00:23:18,751 --> 00:23:27,419
how should like let's just start with indie developers, how should they think about that
as far as building building their apps in particular?

274
00:23:28,135 --> 00:23:38,138
Yeah, so I like at the high level, one thing that I want to sort of I want to caution
against a particular interpretation, and this ends up being like good news for indie

275
00:23:38,138 --> 00:23:38,868
developers.

276
00:23:38,868 --> 00:23:46,690
I want to like caution against the interpretation that like all software engineering jobs
are gonna go away because we can now automate it.

277
00:23:46,690 --> 00:23:47,530
~

278
00:23:47,630 --> 00:23:55,712
because like for how or rather like, you know, everyone can write their own software, so
why would they buy any software from you?

279
00:23:55,920 --> 00:23:59,998
for how long has it been possible like to grow your own vegetables?

280
00:23:59,998 --> 00:24:03,450
Like something like twenty thousand years, right?

281
00:24:03,450 --> 00:24:04,071
Yeah.

282
00:24:04,071 --> 00:24:07,757
Do you grow all of your own vegetables or do you buy vegetables from the store?

283
00:24:09,203 --> 00:24:11,011
Yeah, you well, depending.

284
00:24:11,011 --> 00:24:14,069
Some people garden, some people buy yeah, exactly.

285
00:24:14,069 --> 00:24:18,371
even people who garden, for the most part, buy some stuff from the store.

286
00:24:18,371 --> 00:24:19,081
Either because

287
00:24:19,081 --> 00:24:29,695
it requires resources they don't have like particular soil types or you know, s more space
than they need, or because they want carrots and they've only planted onions.

288
00:24:29,705 --> 00:24:30,556
Right, right.

289
00:24:30,556 --> 00:24:41,131
so yeah, but if twenty thousand years after we invented like agriculture, people are still
buying vegetables from a store, or like yeah

290
00:24:41,131 --> 00:24:54,196
coming down a bit more r relevant to software, like we've been teaching literacy to the
entire of you know, certainly Western societies for like a century or more now.

291
00:24:54,196 --> 00:24:56,331
people still buy books from other people.

292
00:24:56,331 --> 00:25:00,771
Like you could just write all of your own books, but people still read other people's
books.

293
00:25:00,771 --> 00:25:01,352
~

294
00:25:01,352 --> 00:25:04,725
kitchens now and people still go out and eat yeah.

295
00:25:04,725 --> 00:25:05,155
exactly.

296
00:25:05,155 --> 00:25:07,617
yeah, you can put you can cook at home, people don't.

297
00:25:07,617 --> 00:25:08,756
They go out to restaurants.

298
00:25:08,756 --> 00:25:10,359
That's a another great example.

299
00:25:10,359 --> 00:25:19,256
the idea that because an individual can write software, they will write all of the
software that they ever need to use is like false.

300
00:25:19,256 --> 00:25:31,276
There will still be a market for selling things that are you know, i i where it's quicker
like supermarket example is quicker to buy it off a shelf than it is to build it yourself.

301
00:25:31,601 --> 00:25:38,183
the kind of artisan example, you will get a nicer thing if you buy it from someone else
than you would do if you made it yourself.

302
00:25:38,183 --> 00:25:40,143
You might get a more reliable thing.

303
00:25:40,143 --> 00:25:44,304
And so those attributes are still going to be valued.

304
00:25:44,424 --> 00:25:46,325
and that's what we're gonna have to provide.

305
00:25:46,325 --> 00:25:57,184
taking a particular h you know, I don't know how many to do apps there are on the store,
but that seems like one where you might think and I certainly did like think when I s

306
00:25:57,184 --> 00:26:00,048
started considering how

307
00:26:00,826 --> 00:26:02,818
AI would affect software engineering.

308
00:26:02,818 --> 00:26:08,483
I used to think there's no more market for to-do apps because everyone wants them to work
a slightly different way.

309
00:26:08,483 --> 00:26:13,688
So everyone can just get their own bespoke to do app that does exactly what they want.

310
00:26:13,688 --> 00:26:15,408
The thing is most people won't do that.

311
00:26:15,408 --> 00:26:20,433
They either won't want to do it or they won't feel like they have time to do it or they
won't be motivated to do it.

312
00:26:20,433 --> 00:26:26,176
or and it'll be cheaper or better for them to buy someone's notes app from the store.

313
00:26:26,176 --> 00:26:35,554
So even though that seems like like a kind of low hanging fruit for everybody gets their
own version that does exactly what they want, in practice that won't happen.

314
00:26:35,554 --> 00:26:45,747
And people's well, some people will use the reminders app that's built into iOS or the
Notes app or whatever, but some people will still buy other ones because it's better for

315
00:26:45,747 --> 00:26:49,069
them than trying to do it themselves.

316
00:26:49,953 --> 00:27:07,286
So my question for you then is there a middle ground opportunity there for I don't know go
into the cooking analogy but like you know meal kits for instance or like like

317
00:27:08,023 --> 00:27:12,936
You here's a s seed kit or something like that for your garden or something like that.

318
00:27:12,936 --> 00:27:22,686
'Cause I I'm just thinking it was a Ben Thompson article a few weeks ago that I've been
dwelling on about like dynamic the idea of a dynamic app per se.

319
00:27:22,686 --> 00:27:24,827
do you think there's a place for that?

320
00:27:24,827 --> 00:27:27,439
And how would that how how do you think that would look?

321
00:27:27,542 --> 00:27:38,455
So yeah, the the kind of analogy yeah, but we were talking about Apple's history and how
we went from the Apple I kit computer to effectively, better fully integrated Apple II,

322
00:27:38,455 --> 00:27:43,938
Apple III and then completely changed the world with the Macintosh and then the graphical
user interface and everything.

323
00:27:43,938 --> 00:27:52,864
let are there example from Apple's history, what is the HyperCard or the AppleScript of
the AI process?

324
00:27:52,864 --> 00:27:53,204
world.

325
00:27:53,204 --> 00:28:04,403
is there a a world where you have bits of software components that are you know written by
software engineers and bits where you kind of you know either glue those together or add

326
00:28:04,403 --> 00:28:13,380
your own components that are created by LLMs or where there's just like gaps in your
architecture and you just say to an LLM, right, this thing needs you know, this type of

327
00:28:13,380 --> 00:28:16,062
information, this thing needs it y this type of information.

328
00:28:16,062 --> 00:28:22,025
Your job is to like you know, is to adapt those two things together and just kind of

329
00:28:22,025 --> 00:28:25,156
play computer every time that question comes up.

330
00:28:25,417 --> 00:28:25,698
~

331
00:28:25,698 --> 00:28:30,321
from a yeah, from a technological perspective, that absolutely makes a lot of sense.

332
00:28:30,321 --> 00:28:37,235
in some ways, that is what the new version of Siri is supposed to do with like App
Intents.

333
00:28:37,235 --> 00:28:38,205
You know, I come up

334
00:28:38,205 --> 00:28:46,209
with a c a question, and the and the contextual Siri but understands all your personal
kind of context and information and what you're doing on your phone.

335
00:28:47,369 --> 00:28:57,524
Yeah, I come up with a question like I'm I've got a meeting with Leo in forty five
minutes, work out which of these tasks on my app I can actually start writing before like

336
00:28:57,524 --> 00:28:58,874
the call starts.

337
00:28:59,054 --> 00:28:59,304
And

338
00:28:59,304 --> 00:29:03,215
and then, you know, it uses all that information to to do that.

339
00:29:03,936 --> 00:29:12,643
In practice, like the the the risks are are are the same as they've always been, but with
a lot more

340
00:29:12,643 --> 00:29:20,966
cognizant of them now and what makes it difficult is the security model because you're
basically saying, right, this little bit of arbitrary information over here, which might

341
00:29:20,966 --> 00:29:32,781
be confidential, it might you know, it might need to not change outside of certain
situations, is now like interfaced with this model that will take some random like string

342
00:29:32,781 --> 00:29:36,552
of text and just interpret it as a set of instructions and then do it.

343
00:29:36,552 --> 00:29:40,832
Which is much more you know it that's

344
00:29:40,832 --> 00:29:50,575
more like going back to the world where y your computer just had a basic prompt and you
could type in any program you wanted, even though the program itself is now free form

345
00:29:50,575 --> 00:29:51,315
text.

346
00:29:51,315 --> 00:29:57,677
You've got to think about what are the implications, what are the the impacts, what are
the risks of doing that.

347
00:29:57,677 --> 00:30:05,536
And I honestly think that a lot of you know, because of the attitude to security

348
00:30:05,592 --> 00:30:16,122
particularly for s a company like Apple and their approach to privacy, the risk of
allowing you to do anything with that kind of tool is such that we won't really get

349
00:30:16,122 --> 00:30:21,298
anything more than sandboxes, even though there's like huge usability opportunities there.

350
00:30:22,702 --> 00:30:24,683
Do you think that's the right thing to do?

351
00:30:26,110 --> 00:30:26,419
no.

352
00:30:26,419 --> 00:30:37,879
I you know, I I think that so I I come from a background like before I was in the Apple
world, the reason I came into the Apple world was that I was like a huge free software

353
00:30:37,879 --> 00:30:46,616
person and basically like back in two thousand four, Mac OS X was like Linux, but it
carried on working when you plugged in a monitor and you didn't have to like r

354
00:30:46,616 --> 00:30:48,268
edit a device driver file.

355
00:30:48,268 --> 00:30:50,619
So I come from that world where

356
00:30:50,806 --> 00:30:53,969
I think of it as my computer and I'm free to do what I want.

357
00:30:53,969 --> 00:31:04,446
And so I even on devices like the iPhone and the iPad, I you know, don't like the
restrictions on what I'm allowed to do as a user and on like the software that I'm allowed

358
00:31:04,446 --> 00:31:05,267
to use.

359
00:31:05,267 --> 00:31:16,491
So but then I also acknowledge that I'm a particular type of person who like has put a
load of effort into understanding this stuff in depth and so has

360
00:31:16,491 --> 00:31:18,272
much better understanding of the risks

361
00:31:18,272 --> 00:31:32,231
and everything there and like a lot of us are in a position of making choices on behalf of
people who you know while they certainly could understand those choices, you know we're

362
00:31:32,231 --> 00:31:39,707
not kind of making any comments on their abilities or intelligence, we're making comments
on what they choose to focus their time on and it ain't computers.

363
00:31:39,707 --> 00:31:40,687
So we're

364
00:31:40,687 --> 00:31:45,200
we're being trusted to make decisions on their behalf and so we have to make sure we're
making the good ones.

365
00:31:45,318 --> 00:31:45,688
Great.

366
00:31:45,688 --> 00:31:46,269
Yep.

367
00:31:46,269 --> 00:31:56,793
Yeah, I think some people just that's not their thing and they don't want to customize and
that's that's the benefit of Apple to a lot of people is like it works outside the box.

368
00:31:56,914 --> 00:31:59,735
Or it works out of the box, excuse me.

369
00:31:59,895 --> 00:32:00,536
In theory.

370
00:32:00,536 --> 00:32:01,877
In theory anyways.

371
00:32:01,877 --> 00:32:07,900
so yeah, let's talk about like having control as a software engineer.

372
00:32:07,900 --> 00:32:13,072
Because you know, there's the whole idea of like meat proxies and stuff.

373
00:32:13,502 --> 00:32:15,203
that term's been thrown around.

374
00:32:15,203 --> 00:32:23,948
But like where how do you make sure that you're in control and you're using the LLM to be
actually productive?

375
00:32:24,169 --> 00:32:25,179
And not just letting

376
00:32:25,179 --> 00:32:26,690
it tell you what to do.

377
00:32:27,202 --> 00:32:28,023
No, absolutely.

378
00:32:28,023 --> 00:32:40,675
And I c the first time I heard this meat proxies phrase, I I thought of times in my life
when like I have been employed as a software engineer to take like text that you know, a

379
00:32:40,675 --> 00:32:49,516
problem report from Jira that a customer has like written or a you know, product manager
has written and then an analyst has interpreted and then my job is to turn that into

380
00:32:49,516 --> 00:32:56,087
something that a tester is going to test and thinking, Well, I have been the meat proxy
already, just like the this was well before

381
00:32:56,087 --> 00:32:57,727
AI even existed.

382
00:32:57,777 --> 00:33:01,788
right, the so this comes down to responsibility, right?

383
00:33:01,788 --> 00:33:09,999
The yeah, you have to understand what your responsibility is and to whom that
responsibility lies.

384
00:33:09,999 --> 00:33:20,542
And for me, yeah, if you like thinking about kind of the Agile Manifesto, like the first
principle is that like your your primary responsibility

385
00:33:20,542 --> 00:33:25,945
is the delivery of valuable software to your customers, or working software.

386
00:33:26,065 --> 00:33:28,656
that is kind of true.

387
00:33:28,656 --> 00:33:41,008
I would then like you know add some kind of ethical dimension and say that I also have a
responsibility to society and to like the other people who are impacted by like by my

388
00:33:41,008 --> 00:33:46,645
software existing and saying that I have to do things that are beneficial to them as well
as to

389
00:33:46,645 --> 00:33:50,728
the the customer, it can't just be about making more money for someone.

390
00:33:51,810 --> 00:33:55,073
within that so there's my fundamental framework.

391
00:33:55,073 --> 00:34:04,431
I am trying to deliver something that is of value to some people and that is generally
beneficial to the the world in which those people exist or to the society in those in

392
00:34:04,431 --> 00:34:06,081
which those people exist.

393
00:34:06,442 --> 00:34:08,323
How am I going to do that?

394
00:34:08,384 --> 00:34:15,311
And if I can't answer that question, how am I going to do that in the way that I am
writing my software

395
00:34:15,311 --> 00:34:17,702
then I am not writing my software in the right way.

396
00:34:17,702 --> 00:34:27,144
So if i i i if the customer wants f wants their software to do a particular thing and
that's what they find valuable, I have to make sure that the software does that particular

397
00:34:27,144 --> 00:34:38,198
thing, which means I have to like A deliver that functionality and B like validate that
that actually exists and that it does it in a way that the customer can use and is happy

398
00:34:38,198 --> 00:34:39,198
to use.

399
00:34:41,131 --> 00:34:51,915
Can I just make so firstly, c could I just like pay someone else to do all of that and
then not inspect their work and just like, you know, outsource it and just give whatever

400
00:34:51,915 --> 00:35:00,321
the outsource agent delivers to my customer without like looking at it, without evaluating
it, without asking any questions.

401
00:35:00,582 --> 00:35:02,923
Yes, but I probably wouldn't want to.

402
00:35:03,263 --> 00:35:05,976
What's the difference here?

403
00:35:05,976 --> 00:35:11,319
The the only difference is that rather than giving it to someone else, I'm prompting an
LLM.

404
00:35:11,840 --> 00:35:23,036
So if I don't feel comfortable hiring a contractor and then making the contractor talk to
the customer directly, I shouldn't feel comfortable about using an LLM and making the LLM

405
00:35:23,036 --> 00:35:24,587
talk to the customer directly.

406
00:35:25,587 --> 00:35:26,518
at this.

407
00:35:28,021 --> 00:35:36,012
Do you think that's the perspective of a lot of the leading AI companies out there, or do
you think they're trying to figure out a way not to have to deal with that?

408
00:35:36,410 --> 00:35:38,452
There's a bunch of different takes.

409
00:35:38,452 --> 00:35:46,146
I think like a and they've also different people have put this in different ways based on
how it lands, really.

410
00:35:46,146 --> 00:35:47,516
~

411
00:35:47,537 --> 00:35:57,816
so there there is there has been a take and I think like Dario Amodei, the CEO of
Anthropic, used to say this a lot and has pivoted away from it.

412
00:35:57,816 --> 00:36:00,688
That like, yeah, we'll be able to make kind of

413
00:36:00,688 --> 00:36:02,520
a bunch of different knowledge workers redundant.

414
00:36:02,520 --> 00:36:06,467
We'll be able to eliminate a bunch of jobs because you can just get computers to do it.

415
00:36:06,467 --> 00:36:07,146
~

416
00:36:07,146 --> 00:36:09,102
that would be taken negatively?

417
00:36:09,583 --> 00:36:11,104
Well, so

418
00:36:11,645 --> 00:36:16,819
in in some ways that should be that could be a positive thing, right?

419
00:36:16,819 --> 00:36:26,693
You know, if you think about like Star Trek, nobody has a job in Star Trek because you
know, the everything can be automated so people do the things that they want to do.

420
00:36:26,693 --> 00:36:35,751
Like, you know, no one's being paid to like be an officer in Starfleet, for example, or
nobody is being paid to

421
00:36:35,778 --> 00:36:37,928
grow the wine on Picard's Vineyard.

422
00:36:37,928 --> 00:36:39,148
They do it because

423
00:36:39,148 --> 00:36:41,009
because that's what they want to do.

424
00:36:41,229 --> 00:36:53,174
The difficulty with saying we're going to like make everybody unemployed is a transition
where we still have a society where you need to like have money to do stuff and that money

425
00:36:53,174 --> 00:36:54,264
comes from doing work.

426
00:36:54,264 --> 00:37:02,037
and that's why like making people unemployed is bad, is because we haven't got any like
safety net for though for people who are unemployed.

427
00:37:02,037 --> 00:37:03,366
We haven't got any

428
00:37:03,366 --> 00:37:04,909
We're unemployable.

429
00:37:05,599 --> 00:37:07,019
or unemployable, right.

430
00:37:07,019 --> 00:37:14,639
so within that framework, like and particularly a lot of software engineers, I am one of
them, enjoy writing code.

431
00:37:14,639 --> 00:37:17,182
You know, it's something that we like doing.

432
00:37:17,182 --> 00:37:25,774
Now just because we like doing it, it doesn't that doesn't mean that there is a
requirement for somebody else to pay us to do it.

433
00:37:25,774 --> 00:37:29,699
Like, yeah, we I have no entitlement to be hired as

434
00:37:29,699 --> 00:37:32,590
a a a as a code writer because I like writing code.

435
00:37:32,590 --> 00:37:38,912
but if someone's gonna threaten the entire basis of like not only code writing but all of
no knowledge work.

436
00:37:38,912 --> 00:37:47,421
And so not only can I no longer do that for money anymore, but I also can't do any of the
other things I could possibly do without skill anymore, then yes, I'm gonna be threatened

437
00:37:47,421 --> 00:37:49,135
and I'm gonna feel bad about that.

438
00:37:49,135 --> 00:37:55,156
But a different way to look at it, the way that I have used on my blog, and I know you've
got a link to this so it'll be in the show notes.

439
00:37:55,196 --> 00:37:56,056
and

440
00:37:56,260 --> 00:38:11,527
the and I'm not unique in this is there's a an idea in economics called Jevons paradox
which Jevons was industrial age like mid nineteenth century he observed that as it became

441
00:38:11,527 --> 00:38:19,591
more efficient to to extract coal demand for coal went up basically like if you make it
more efficient

442
00:38:19,591 --> 00:38:21,102
to use the resource

443
00:38:21,337 --> 00:38:22,628
you increase demand for resource.

444
00:38:22,628 --> 00:38:25,899
You can also see that if you've ever driven around Los Angeles.

445
00:38:25,899 --> 00:38:27,920
Los Angeles has super bad traffic.

446
00:38:27,920 --> 00:38:34,903
some people can like not get to work in a day because they get in a traffic jam for eight
hours and they have to turn around to go home.

447
00:38:35,023 --> 00:38:36,184
~ and

448
00:38:36,184 --> 00:38:40,385
the way that LA has tried to fix this is by building wider and wider roads.

449
00:38:40,466 --> 00:38:43,707
And you know, you feel wider roads means more cars.

450
00:38:43,915 --> 00:38:44,876
so we should have less traffic.

451
00:38:44,876 --> 00:38:47,025
But people go, right, the road's better now.

452
00:38:47,025 --> 00:38:51,999
I can drive to this like destination, so they get on the road and then there's more
traffic.

453
00:38:51,999 --> 00:39:00,751
so increasing the d the efficiency of using the resource, which in this case is the roads,
inc increases demand so you

454
00:39:00,751 --> 00:39:01,822
get more traffic.

455
00:39:01,822 --> 00:39:04,203
Well, that should apply to knowledge work as well.

456
00:39:04,203 --> 00:39:12,202
If you make it more if you make it easier to employ human intelligence, there should be an
increase in demand for human intelligence.

457
00:39:12,202 --> 00:39:18,467
There are way more knowledge workers in the internet era than there were in the kind of
pen and paper era.

458
00:39:18,467 --> 00:39:25,442
And there were way more knowledge workers in the pen and paper era than there were in the
era when everyone except religious people was illiterate.

459
00:39:25,442 --> 00:39:34,790
So yeah, we another way of looking at is that we're going to unlock a whole new collection
of jobs in knowledge work and we don't even know what those are yet.

460
00:39:36,658 --> 00:39:38,100
Yeah, I agree with that completely.

461
00:39:38,100 --> 00:39:39,553
We don't know what those are yet.

462
00:39:39,553 --> 00:39:44,532
despite the fact that we've created fields like prompt engineering and all that.

463
00:39:44,532 --> 00:39:47,366
Like yeah, I don't or context engineering, whatever.

464
00:39:47,366 --> 00:39:51,250
It's like I think we're still in the baby stages w with regard to that.

465
00:39:51,250 --> 00:39:58,792
those might be kind of tasks that become part of the new work, even if they don't become
whole careers and jobs in their own right.

466
00:39:58,792 --> 00:40:04,616
It might be that prompt engineering is something that everyone has to learn at grade
school before they can, go to university.

467
00:40:04,616 --> 00:40:11,738
Or it might be that a whole bunch of different knowledge work jobs have prompt engineering
as a skill that you need.

468
00:40:12,044 --> 00:40:13,835
Yeah, yeah, exactly.

469
00:40:13,955 --> 00:40:17,118
So let's get into patterns.

470
00:40:17,118 --> 00:40:28,449
well that was probably one of my favorite parts because a lot of those were like is it's
and I think we I told you this after the talk, but like when we talk about like design

471
00:40:28,449 --> 00:40:33,696
patterns, you know, visitor pattern, etc., it's almost like, you implement that pattern.

472
00:40:33,696 --> 00:40:36,145
Like you have to learn the pattern so you implement it.

473
00:40:36,145 --> 00:40:39,310
But it's in fact more like what I found is I I

474
00:40:39,310 --> 00:40:49,620
implement the pattern and then I realize it's a pattern and that's what I'm finding with a
lot of what you talked about in your talk about like these certain patterns that we follow

475
00:40:49,620 --> 00:40:51,181
and I'm like, yeah that that makes sense.

476
00:40:51,181 --> 00:40:52,823
That's exactly what I do.

477
00:40:52,823 --> 00:41:05,033
But yeah what like is that it's that kind of your perspective on it and then what what
have you found is the the biggest lesson you've learned with these

478
00:41:05,430 --> 00:41:19,497
So I presented an early part of so talking about my interaction patterns in AI augmented
software development and I presented an early one of these to a workshop at conference,

479
00:41:19,497 --> 00:41:23,719
the XP 2026 conference in São Paulo, back in April.

480
00:41:23,719 --> 00:41:28,375
and a load of the feedback I had was people saying

481
00:41:28,375 --> 00:41:29,586
I have that pattern.

482
00:41:29,586 --> 00:41:35,272
I the thing that I have described in the in this pattern is something that they were
already doing.

483
00:41:35,272 --> 00:41:46,811
And that is great news when it comes to patterns because what you're trying to do is as I
said earlier, you're trying to like define the like universal characteristics of some

484
00:41:46,811 --> 00:41:54,528
category of problem and then say, so if you have a problem that fits in this category,
here's a solution that you can apply.

485
00:41:54,549 --> 00:42:00,703
And here's the opportunities and the costs of using that solution, here's here's the other
things it unlocks.

486
00:42:00,703 --> 00:42:14,568
and so what the AI patterns are about is me reflecting on my work with AI tools, like my
colleagues, other people in the community who going, right, you know, I've or like yeah,

487
00:42:14,568 --> 00:42:18,694
people post on LinkedIn and say, Hey, I've discovered this thing I can do with

488
00:42:19,301 --> 00:42:23,083
an AI tool and you know, and I go, yeah, yeah, you're not the first person.

489
00:42:23,083 --> 00:42:26,265
There's a common yeah, there there's a common thread through this.

490
00:42:26,265 --> 00:42:34,569
Or people saying, I'm stuck on this problem, and then I can say, Well, yeah, my yeah, the
people I was working with like a couple of months ago were stuck on that problem.

491
00:42:34,569 --> 00:42:37,189
We came up with this solution, you can try that.

492
00:42:37,230 --> 00:42:44,463
so really like the the fundamental thing that I've learned is that like we're still in a
very exploratory phase.

493
00:42:44,539 --> 00:42:46,100
with using AI tools.

494
00:42:46,100 --> 00:42:57,119
We're still in the everyone is like either paying the whatever it is eighteen dollars a
month to Claude and using that, or they're like, you know, downloading Gemma 4 or Qwen or

495
00:42:57,119 --> 00:42:59,681
something onto their computer and they're trying that.

496
00:43:00,222 --> 00:43:08,888
and everyone is kind of going through this like this same set of both like technical and
also emotional

497
00:43:09,052 --> 00:43:12,845
changes as they learn how to how to work with these tools.

498
00:43:12,845 --> 00:43:14,766
and that that will settle down over time.

499
00:43:14,766 --> 00:43:31,320
but also a lot of what people need help with is or the a lot of the benefit that people
can get is the going from type in a problem statement and get some source code out to type

500
00:43:31,320 --> 00:43:34,743
in a problem statement and

501
00:43:34,743 --> 00:43:45,420
references to your company's way of working or your project documentation or your coding
style or whatever it is so that the thing you get out at the end, A works in terms of

502
00:43:45,420 --> 00:43:54,245
being able to like integrate it into your app or your project, and B is more likely to be
something that you would want to like use and want to rely on.

503
00:43:54,245 --> 00:43:58,929
Going back to that point about responsibility, you know, the reason that we have

504
00:43:58,929 --> 00:44:09,822
coding standards and testing practices and code review and all these things is because we
want to give our customers the things that we're comfortable with maintaining and that

505
00:44:09,822 --> 00:44:12,787
we're like proud to put our names behind.

506
00:44:12,807 --> 00:44:24,344
So a lot of it is about kind of getting there and not just having some random Stack
Overflow style like function that just sits in space and like doesn't match the way that

507
00:44:24,344 --> 00:44:26,301
we work, which comes from

508
00:44:26,301 --> 00:44:29,800
our understanding of our responsibilities to the people that we work with.

509
00:44:30,700 --> 00:44:33,726
Yeah, so let's get into some of these patterns.

510
00:44:33,726 --> 00:44:45,559
first of all, prompt management basically the idea is is having something either like
create a prompt for you or record a prompt or give examples of a prompt, right?

511
00:44:45,559 --> 00:44:48,083
And let me know if I'm off base on any of these.

512
00:44:48,083 --> 00:44:48,853
no, that's yeah,

513
00:44:48,853 --> 00:44:49,313
that's fair.

514
00:44:49,313 --> 00:44:59,675
So yeah, there there is so we talked about prompt engineering as like something that was
very briefly a job that people could have and these days is more like a skill that a lot

515
00:44:59,675 --> 00:45:01,286
of people need.

516
00:45:01,286 --> 00:45:11,098
And so the prompt management patterns are essentially they aren't a guide to prompt
engineering because that changes every time there's a new model or like new capabilities

517
00:45:11,098 --> 00:45:12,068
in the tools or whatever.

518
00:45:12,068 --> 00:45:15,077
What they are is a framework for

519
00:45:15,077 --> 00:45:16,518
how you think about prompt engineering.

520
00:45:16,518 --> 00:45:19,921
So the core of the prompt management tool set

521
00:45:19,921 --> 00:45:21,403
is Bake Off.

522
00:45:21,403 --> 00:45:27,678
And Bake Off is like it it is the scientific method applied to prompts.

523
00:45:27,678 --> 00:45:39,517
So you know it's very like we know that models are non deterministic, or at least like if
you use them with the like the ways that you typically use them are non deterministic and

524
00:45:39,517 --> 00:45:43,160
they'll randomly choose like likely tokens to complete your stream.

525
00:45:44,494 --> 00:45:56,324
so Bake Off says like if you've got something that kind of that works like to some certain
extent and then you think you want to like make a change, for example, editing your prompt

526
00:45:56,324 --> 00:46:06,393
to like add a a different rule or to change the wording or to make something explicit that
wasn't before, try your thing out a bunch of times before you make the change.

527
00:46:06,393 --> 00:46:06,873
Make

528
00:46:06,873 --> 00:46:10,802
only that change, try your thing out again a bunch of times.

529
00:46:10,802 --> 00:46:12,373
and then compare the results.

530
00:46:12,373 --> 00:46:18,495
Now, yeah, that has a a like a word that people use, which is an eval.

531
00:46:18,495 --> 00:46:28,926
You what you're doing there is performing an eval on your entire LLM setup with like two
different prompts and seeing how that changes.

532
00:46:28,926 --> 00:46:36,991
But what's important is the process, it's the identifying what problem you have or what
like improvement you want to make.

533
00:46:36,991 --> 00:46:43,773
hypothesising that a particular change will make that improvement and then making only
that change and testing whether it does or not.

534
00:46:44,033 --> 00:46:54,697
so like within that you then have a continuous improvement loop that applies to prompts
but it also applies to you know evaluating different models or to changing like the

535
00:46:54,697 --> 00:47:00,549
parameters of a model or to like switching your harness tools or whatever.

536
00:47:01,326 --> 00:47:01,787
Okay.

537
00:47:01,787 --> 00:47:03,188
Yeah, I like that.

538
00:47:03,749 --> 00:47:15,857
so next we got I think this is a quick one, memory management, just the idea of like
somehow exporting part of context into something that can then be read out ~ anywhere

539
00:47:15,857 --> 00:47:16,608
else.

540
00:47:16,958 --> 00:47:27,706
so yeah, the a model itself has like all of the weights that were created by feeding it
tons and tons of training data and then

541
00:47:27,706 --> 00:47:39,287
those are tweaked and biases added by a process called fine tuning, where basically humans
kind of set goals and then evaluate the output and then like nudge the model so that it

542
00:47:39,287 --> 00:47:42,549
goes more towards particular types of output than others.

543
00:47:42,549 --> 00:47:51,054
And then what happens is that like all of that is saved to a a really big file and that is
the model and that is the thing that you use in inference.

544
00:47:51,054 --> 00:47:53,375
And it never changes again.

545
00:47:53,375 --> 00:48:00,007
So that model would always work the same way given the same input information.

546
00:48:00,007 --> 00:48:05,316
so the memory is about choosing information

547
00:48:05,322 --> 00:48:17,265
in your project, in your context, in your organization, that is relevant to the tasks
you're trying to get the model to do that you attach to the prompt so that it does it in

548
00:48:17,265 --> 00:48:21,629
ways that are appropriate for for your tasks.

549
00:48:21,749 --> 00:48:30,601
And and here I have to admit, like, it is very hard to talk about AI without pretending in
some way that it's intelligent.

550
00:48:30,601 --> 00:48:33,339
and so I'm going to use words very loosely

551
00:48:33,339 --> 00:48:35,490
That sound like human

552
00:48:35,490 --> 00:48:38,231
cognition words and really are just analogies.

553
00:48:38,231 --> 00:48:46,275
But like what you're trying to do with memory management is to get the model to learn, and
I'm using my scare quote fingers as I say that.

554
00:48:46,275 --> 00:48:49,676
learn what the way is that you want it to work.

555
00:48:50,077 --> 00:48:51,267
It's not learning at all.

556
00:48:51,267 --> 00:48:54,416
You're just feeding it like extra rules or whatever,

557
00:48:54,416 --> 00:48:55,177
Right.

558
00:48:55,279 --> 00:48:58,310
every time you ask it a question or every time you tell it to do something.

559
00:48:58,490 --> 00:48:59,010
Yep.

560
00:48:59,010 --> 00:49:00,180
Discover Opportunities.

561
00:49:00,180 --> 00:49:01,021
I really like this one.

562
00:49:01,021 --> 00:49:07,993
I feel like I used that a lot early on when I was trying to get ChatGPT to do prose.

563
00:49:08,234 --> 00:49:09,229
And then

564
00:49:09,229 --> 00:49:25,751
and then it was like once I started l using prose less, I've been yeah, using this less,
but then I think it's really good for basically the kind of stuff not so much code you're

565
00:49:25,751 --> 00:49:26,925
gonna disagree, but

566
00:49:26,925 --> 00:49:27,506
I I think it

567
00:49:27,506 --> 00:49:34,856
is good for code, but I don't think you get as much bang for your buck compared with
images or text or video, et cetera.

568
00:49:35,067 --> 00:49:48,983
So yeah, so I w and I think that the patterns in the discover opportunity section are some
of the best things for people who are like maybe kind of coming to generative AI tools for

569
00:49:48,983 --> 00:49:49,473
the first time.

570
00:49:49,473 --> 00:49:57,455
Like you know, maybe you've been kind of sceptical about how well they'd work, but yeah,
you know, your employer is saying you have to use this now and so you're kind of coming on

571
00:49:57,455 --> 00:50:01,090
board or you know, you're just kind of ready to take a go.

572
00:50:01,802 --> 00:50:15,052
These are the patterns that most that are a easiest to adopt because they don't actually
make any changes to your code, and also help to help you to understand how you can

573
00:50:15,052 --> 00:50:16,464
integrate these tools into your workflow.

574
00:50:16,464 --> 00:50:19,165
So like the first one is Just Ask.

575
00:50:19,266 --> 00:50:25,451
that's what we were talking about earlier with the like the Socratic method for doing like
reviews or for doing education.

576
00:50:25,451 --> 00:50:26,261
Just

577
00:50:26,580 --> 00:50:29,442
Ask the model questions about your project.

578
00:50:29,442 --> 00:50:31,924
Ask the model so I need to add this feature.

579
00:50:31,924 --> 00:50:34,625
Where is the view that has these controls on it?

580
00:50:34,625 --> 00:50:39,248
where is the view model that shows this data?

581
00:50:39,248 --> 00:50:46,253
How does this like field get into the data that goes to the back end?

582
00:50:46,253 --> 00:50:52,129
What would happen to performance if I changed this list of lists?

583
00:50:52,129 --> 00:50:58,973
into dictionary or if I used SwiftData instead of a flat file for this data.

584
00:50:59,089 --> 00:50:59,694
Right.

585
00:50:59,694 --> 00:51:04,107
and the others it the other patterns in that collection are kind of similar to that.

586
00:51:04,107 --> 00:51:07,499
They're all about like finding out things.

587
00:51:07,499 --> 00:51:15,124
The most interesting one is Disclose Ambiguity because that's about finding out where the
gaps in your prompt are.

588
00:51:15,124 --> 00:51:17,097
So you know if you

589
00:51:17,097 --> 00:51:22,958
you want the LLM to generate some software, don't just like type your prompt in and see
what it does.

590
00:51:23,018 --> 00:51:29,019
Type your prompt in and then ask what are the hidden assumptions in this prompt?

591
00:51:29,059 --> 00:51:29,859
And then you might

592
00:51:29,859 --> 00:51:33,402
f you might find that you've like missed some stuff out and you can answer it.

593
00:51:33,402 --> 00:51:37,983
You might find that your customer doesn't know what they want and you have to go back to
them and say like hey

594
00:51:37,983 --> 00:51:39,134
so what do we actually do here?

595
00:51:39,134 --> 00:51:43,175
and that's that yeah that is where we start to get into the kind of centaur

596
00:51:43,175 --> 00:51:55,762
superpower thing because then we're we're out of the kind of turnkey do what our customers
tell us to into do what our customers want f us to do ~

597
00:51:55,762 --> 00:51:59,794
and collaborate with them on designing great software that solves their problems.

598
00:52:00,858 --> 00:52:05,840
I wanna hop ahead to quality improvement because that's the part I'm really interested in.

599
00:52:05,840 --> 00:52:11,065
A lot of the gains people see in AI are typically speed.

600
00:52:11,365 --> 00:52:18,658
Where where can AI improve the quality of your best code, let's say?

601
00:52:18,841 --> 00:52:21,722
Yeah, so and then let's like let's combine the two.

602
00:52:21,722 --> 00:52:26,763
So the first thing and a very common pattern is the You Review pattern, right?

603
00:52:26,763 --> 00:52:32,785
You know, you you tell your LLM your prompts, you generate a bunch of code, and then you
sit there and you try and read all the code.

604
00:52:32,785 --> 00:52:37,846
and A, these things can generate code much faster than like any of us can type.

605
00:52:37,846 --> 00:52:40,126
and so there's way more code to review.

606
00:52:40,126 --> 00:52:42,100
and B, that's really dull.

607
00:52:42,100 --> 00:52:42,890
we none of us got

608
00:52:42,890 --> 00:52:46,311
into software engineering because we wanted to read other people's code all day.

609
00:52:46,311 --> 00:52:49,954
We would in it because we wanted to write code and make the computer do interesting
things.

610
00:52:49,954 --> 00:52:57,186
So so all we've done is like made software engineering slower and more boring by like
adopting this tool.

611
00:52:57,186 --> 00:52:57,986
That doesn't sound great.

612
00:52:57,986 --> 00:52:59,797
~ and then people have

613
00:52:59,797 --> 00:53:02,627
the great idea, well, why don't we just get the model to review the code?

614
00:53:02,627 --> 00:53:08,549
Because if the LLM can like write the code really fast and the LLM can read the code really
fast, then

615
00:53:08,549 --> 00:53:18,482
We'll just get it to do both bits, but then it's effectively marking its own homework and
any problems that you had prompting it are going to remain in the output because the your

616
00:53:18,482 --> 00:53:24,544
your instructions for make sure this does the thing that I wanted are based on the same
wrong instructions.

617
00:53:24,544 --> 00:53:32,299
So then we have to work out, okay, how do we get this thing to do things better and faster
because we want both.

618
00:53:32,299 --> 00:53:36,712
And part of my answer turns out to be to not always use an LLM.

619
00:53:36,963 --> 00:53:47,158
there's a pattern called Replace Vibes with Tools, where I say if you've got some
deterministic part of your process, like say you want to v verify that some software does

620
00:53:47,158 --> 00:53:50,572
a particular thing, don't ask a model, does it do this thing?

621
00:53:50,572 --> 00:53:51,943
Write a test.

622
00:53:51,943 --> 00:53:54,826
Because that test is objective.

623
00:53:54,826 --> 00:53:56,337
It is deterministic.

624
00:53:56,337 --> 00:53:58,826
Every time you run that test, you will get

625
00:53:58,826 --> 00:54:03,168
a pass if the software actually does that thing you'll get a fail if it doesn't do that
thing.

626
00:54:03,168 --> 00:54:13,579
or if I'm not entirely sure what my customer meant and I can't get my model to like fill
in the gaps because they don't know what the customer means.

627
00:54:13,579 --> 00:54:16,305
You know, it doesn't know what the customer means either.

628
00:54:16,305 --> 00:54:18,136
It's just like reading this text.

629
00:54:18,136 --> 00:54:19,837
Why don't I just build a prototype?

630
00:54:19,837 --> 00:54:23,138
So like yes, these things are great at generating code.

631
00:54:23,138 --> 00:54:25,199
We don't need that code to go into production though.

632
00:54:25,199 --> 00:54:26,616
We could build a prototype.

633
00:54:26,616 --> 00:54:28,680
show that to the customer, go, is this what you mean?

634
00:54:28,680 --> 00:54:31,024
They could go, well, no, actually I meant this.

635
00:54:31,024 --> 00:54:37,686
And then we could go and build the software and throw away what the model wrote and just
build whatever it is we've learned from that experience.

636
00:54:37,727 --> 00:54:38,268
Yep.

637
00:54:38,268 --> 00:54:39,710
Yeah, I like that.

638
00:54:40,334 --> 00:54:49,201
Yeah, a good point about prototypes too, because like I mean I'm just thinking of like a
boss who's just like, just go do this and then you do it and

639
00:54:50,278 --> 00:54:51,082
That's not what I wanted.

640
00:54:51,082 --> 00:54:52,275
Yeah.

641
00:54:52,275 --> 00:54:53,225
yeah, that's not what I want.

642
00:54:53,225 --> 00:54:53,506
Right.

643
00:54:53,506 --> 00:55:04,312
You they're like frustrated or asking details and at the end of the day, like a prototype
for both us when we talk to an LLM, but also for people who are our bosses.

644
00:55:04,312 --> 00:55:07,604
It's good to have like an actual example to see.

645
00:55:08,008 --> 00:55:08,819
Yeah, so there used to be

646
00:55:08,819 --> 00:55:14,027
this phrase that sometimes a month in the library can save an hour in the lab.

647
00:55:15,191 --> 00:55:15,691
yeah, it's

648
00:55:15,691 --> 00:55:15,942
true.

649
00:55:15,942 --> 00:55:18,796
Just tr just try the thing out and see whether you were right or not.

650
00:55:19,466 --> 00:55:23,208
were there any other patterns you wanted to cover before we move on?

651
00:55:24,111 --> 00:55:32,034
I think that like that's a good overview of what the whole like pattern catalogue is
about.

652
00:55:32,034 --> 00:55:38,218
Like that you know, going into the details of all of them is perhaps just going to glaze
people's eyes over.

653
00:55:38,218 --> 00:55:49,194
Like yeah, the idea is that in each of these cases there is a thing a thing that many
people are trying to do with an LLM and there is an answer to, well, here's how you can do

654
00:55:49,194 --> 00:55:50,485
that and then

655
00:55:50,653 --> 00:55:53,895
Here's some new problems you get because you've solved that problem.

656
00:55:54,682 --> 00:55:56,343
Yeah, yeah, fair enough.

657
00:55:56,343 --> 00:56:03,269
are there any anti patterns you've seen that people do that you're like, No, that's not
the way we're supposed to do it?

658
00:56:04,625 --> 00:56:13,050
So like obviously the big anti pattern from a software engineering perspective is vibe
coding.

659
00:56:13,050 --> 00:56:15,632
yeah, that I have no pro Right.

660
00:56:15,632 --> 00:56:16,752
Okay, yeah.

661
00:56:16,752 --> 00:56:27,903
So I think it was Andrej Karpathy came up with the with the term and it basically means
just generating software by typing

662
00:56:28,943 --> 00:56:36,634
like what you want, you know, typing a problem specification into an LLM and and then
getting software out and then just running that.

663
00:56:36,634 --> 00:56:41,659
So it means like not having an engineering process.

664
00:56:41,659 --> 00:56:43,540
and I have no problem with people doing that.

665
00:56:43,540 --> 00:56:55,777
I you know, I think that like this is one of the big moments in the history of computing
where computing becomes an empowering tool that enables more people to do more things.

666
00:56:55,991 --> 00:57:06,257
But going back to what I was saying about responsibility, as software engineers, our
responsibility is to make good quality software that does what our customers want and that

667
00:57:06,257 --> 00:57:08,319
benefits society.

668
00:57:08,319 --> 00:57:17,519
You can't guarantee that software that you've never like read or interacted with and was
just generated by a language model has those attributes.

669
00:57:17,519 --> 00:57:23,722
Yeah, you can't guarantee that it has like the security qualities that people need, the
performance qualities that it

670
00:57:23,722 --> 00:57:25,542
actually does what they want.

671
00:57:26,443 --> 00:57:27,913
but that was the cheap answer, right?

672
00:57:27,913 --> 00:57:28,174
Yeah.

673
00:57:28,174 --> 00:57:36,486
Saying the vibe coding is an anti pattern to a podcast for like developers is not really
kind of giving anyone any new information.

674
00:57:36,486 --> 00:57:46,173
An interesting like anti pattern in the way that people are using these things in like in
software development contexts is the rules file.

675
00:57:46,173 --> 00:57:48,127
So if you have like an AGENTS.md file

676
00:57:48,127 --> 00:57:49,808
or a CLAUDE.md file.

677
00:57:49,808 --> 00:57:58,302
Yeah, people put all of the things that they want the LLM to pay attention to when it's
generating code in that.

678
00:57:58,532 --> 00:57:59,082
two problems.

679
00:57:59,082 --> 00:58:09,607
Like one is as this file gets big, it fills with thing like it fills up context and not
every rule in there is relevant to every task you have.

680
00:58:09,607 --> 00:58:12,008
Like, you know, if you're doing kind of

681
00:58:12,404 --> 00:58:19,360
if you're doing changing the data schema, then you need your migration rules, you don't
need your UI automation rules.

682
00:58:19,360 --> 00:58:20,220
~ and

683
00:58:20,220 --> 00:58:22,473
so you should separate those things out.

684
00:58:22,473 --> 00:58:35,554
Another thing that people do is they kind of give there was a workshop and you know I I'm
not kind of like calling this out but there was there was a a workshop at the conference

685
00:58:35,554 --> 00:58:36,505
where

686
00:58:36,833 --> 00:58:45,578
someone was showing their rules file and it said, like, you are a staff developer at this,
you know, company with like expertise in this areas.

687
00:58:45,578 --> 00:58:49,110
and they said, Well, why would I have a junior developer if I can have a staff developer
for free?

688
00:58:49,110 --> 00:58:50,697
~ you know it's kinda

689
00:58:50,697 --> 00:58:53,376
the whole you are thing, I think that's dated.

690
00:58:53,376 --> 00:58:54,269
I don't think that's a thing

691
00:58:54,269 --> 00:58:55,150
anymore.

692
00:58:55,257 --> 00:59:00,258
Yeah, so it turns out that what that is good for, is increasing alignment.

693
00:59:00,258 --> 00:59:10,803
So if you have particular safety guardrails, you want your model to do things in
particular ways, giving it a role makes it align better with those safety guardrails.

694
00:59:10,803 --> 00:59:15,564
But it actually reduces it like the accuracy with which it completes tasks.

695
00:59:15,564 --> 00:59:17,185
So if you look at kind of

696
00:59:18,077 --> 00:59:29,588
benchmarks like Terminal-Bench or SWE-bench or something, a a rules file that like gives
a role to the model means doing getting a worse outcome on those benchmarks than not

697
00:59:29,588 --> 00:59:30,349
bothering.

698
00:59:30,349 --> 00:59:35,153
So again this goes back to the idea of a Bake Off and actually testing any of the changes
you make.

699
00:59:35,345 --> 00:59:37,016
Yeah, yeah, yeah, totally.

700
00:59:37,455 --> 00:59:45,086
all right, so we're one hour in, and I think I should probably give you an opportunity to
talk about your project.

701
00:59:45,086 --> 00:59:45,546
yeah.

702
00:59:45,546 --> 00:59:50,486
~ what have you been working on lately to help with this?

703
00:59:50,486 --> 00:59:57,910
about the we've talked about the book that I've just actually been through all of the kind
of technical and editorial review comments from the publisher on that.

704
00:59:57,910 --> 00:59:59,310
So that should be coming out.

705
00:59:59,310 --> 01:00:03,424
It's published by Apress, currently due to be out March twenty twenty seven.

706
01:00:03,424 --> 01:00:11,115
So, you know, maybe pre order it as an early Christmas present for all of your friends,
but like you'll they'll have to wait until Easter to actually get it.

707
01:00:11,345 --> 01:00:14,739
But I've also been working on an app called Axminster.

708
01:00:14,739 --> 01:00:27,892
and Axminster is a way to visualize the conversation that you have with an LLM and to get
some of this information out of Markdown files that you know we've already talked about

709
01:00:27,892 --> 01:00:30,791
the problem with like reading large code files.

710
01:00:30,892 --> 01:00:34,014
Reading large Markdown files is not much more fun.

711
01:00:34,775 --> 01:00:40,529
And getting that into a visual canvas so that you can kind of see what the different types

712
01:00:40,617 --> 01:00:47,153
that the model is going to generate are and you can change that, you can relate them
together.

713
01:00:47,274 --> 01:00:52,068
And it's based on and you know you might be saying why are you talking about this in 2026,
Graham?

714
01:00:52,068 --> 01:00:58,624
It's based on UML, which is a modeling language from the nineteen nineties.

715
01:00:58,624 --> 01:01:02,087
But it turns out that like UML has a lot of benefits.

716
01:01:02,087 --> 01:01:04,783
It has a well defined

717
01:01:04,783 --> 01:01:06,044
semantic structure.

718
01:01:06,044 --> 01:01:13,739
So the different lines and arrows and boxes on a UML diagram mean something precise in the
context of software design.

719
01:01:13,739 --> 01:01:25,727
So if you have a UML diagram and then you feed that information to an LLM, the LLM knows
exactly what you mean by all of those boxes and arrows.

720
01:01:25,727 --> 01:01:31,051
And so it can generate code that does exactly what you want.

721
01:01:31,051 --> 01:01:33,052
It also has a built-in

722
01:01:33,907 --> 01:01:40,889
constraint language, which is a way of encoding rules and correctness information directly
into the diagram.

723
01:01:40,889 --> 01:01:51,822
So you can say, like if you have a state diagram, so you have a state machine that like
has events that move a system between different states, and you have a guard that says

724
01:01:51,902 --> 01:01:57,163
like this transition only happens when this condition is true.

725
01:01:57,323 --> 01:02:01,304
You've basically put a constraint on the LLM so that when it generates code

726
01:02:02,010 --> 01:02:05,844
the code has that logic built into it as part of the design.

727
01:02:05,844 --> 01:02:17,507
So I think that this is a really powerful way to move beyond walls of Markdown files and
chatbot interfaces and get into a partly visual, partly textual model of the software that

728
01:02:17,507 --> 01:02:19,299
we're building with AI tools.

729
01:02:20,019 --> 01:02:31,457
So yeah, so basically you have like this MCP server that takes the diagrams and turns it
into code and but also code can turn it into diagrams.

730
01:02:31,837 --> 01:02:38,892
and then how are the di how are the diagrams I mean know UML, but I mostly know it
graphically.

731
01:02:38,892 --> 01:02:44,471
Is there like a language or do you use Mermaid or what what exactly are you doing?

732
01:02:44,471 --> 01:02:51,338
No, so there's no kind of well, so the documents are stored in an XML format called XMI,
which is a

733
01:02:51,338 --> 01:02:52,022
Okay.

734
01:02:52,022 --> 01:02:53,823
a standard format that's interchangeable.

735
01:02:53,823 --> 01:02:57,184
But no, the model doesn't work with the XMI files.

736
01:02:57,184 --> 01:03:08,998
The model works with the MCP server, which exposes a c so like let's back up for people
who don't understand or don't haven't heard the phrase MCP, it's model context pro

737
01:03:08,998 --> 01:03:09,628
protocol.

738
01:03:09,628 --> 01:03:10,492
It is

739
01:03:10,492 --> 01:03:22,581
a server client protocol that's designed for the client to be a tool that has an LLM in it
and for the server to provide capabilities or information to the LLM that's running in the

740
01:03:22,581 --> 01:03:23,442
client.

741
01:03:23,442 --> 01:03:35,970
So in this case it gives tools that allow the model to ask questions about the diagrams
about your software design and to make changes to those diagrams and to that software

742
01:03:35,970 --> 01:03:36,791
design.

743
01:03:36,791 --> 01:03:37,564
So

744
01:03:37,564 --> 01:03:43,636
The way that round trip works is when you say, please generate code that conforms to this
model, Axminster is running.

745
01:03:43,636 --> 01:03:48,146
You're looking at the model on the you know in the window if if you want.

746
01:03:48,146 --> 01:03:53,538
and the model is asking, okay, so what classes are in this design?

747
01:03:53,538 --> 01:03:56,999
And it gets a list of classes, and then it says, What attributes are on this class?

748
01:03:56,999 --> 01:04:00,370
So it gets a list of like instance variables and operations.

749
01:04:00,370 --> 01:04:02,156
What constraints are there?

750
01:04:02,156 --> 01:04:06,733
on this operation and then it gets the rules and then it can generate the code from there.

751
01:04:06,733 --> 01:04:12,791
So it is effectively it's not doing it in a visual format, but it's effectively reading
the diagram.

752
01:04:14,140 --> 01:04:15,002
Right, right.

753
01:04:15,002 --> 01:04:16,526
Yeah, I like that a lot.

754
01:04:16,526 --> 01:04:19,772
so this is a macOS app, SwiftUI.

755
01:04:20,676 --> 01:04:21,527
you wrote your own

756
01:04:21,527 --> 01:04:24,253
MCP server, right?

757
01:04:24,356 --> 01:04:28,288
so like that is not actually that complex.

758
01:04:28,288 --> 01:04:32,473
Like MCP is just kind of JSON in, JSON out.

759
01:04:32,694 --> 01:04:37,027
yeah, and it's like they used to be in fact it's simpler than it used to be.

760
01:04:37,027 --> 01:04:45,391
The early version had like this handshake where the client says, Right, I I'm this version
and I have these capabilities and the server said, I'm this version, I have these

761
01:04:45,391 --> 01:04:47,202
capabilities and then they agreed.

762
01:04:47,202 --> 01:04:48,683
That doesn't even happen anymore.

763
01:04:48,683 --> 01:04:51,204
So like there's there's two

764
01:04:51,248 --> 01:04:52,349
facilities I use.

765
01:04:52,349 --> 01:04:55,031
One is well yeah, what one is tools.

766
01:04:55,031 --> 01:05:03,676
So the client says list all the tools that available and the server responds with a list
and there's a little description of what each one does and the LLM can read that and so it

767
01:05:03,676 --> 01:05:05,698
knows when to use each tool.

768
01:05:05,698 --> 01:05:08,160
And the other is prompts.

769
01:05:08,160 --> 01:05:17,406
So you can add like like agent skills give you the ability to kind of define custom
reusable prompts for an LLM.

770
01:05:17,495 --> 01:05:23,056
An MCP server can contain a collection of prompts that are relevant to its activities.

771
01:05:23,056 --> 01:05:32,577
So yeah, there's like prompts for disclosing ambiguity in models for applying constraints
to models based on business rules.

772
01:05:32,577 --> 01:05:36,158
So basically go you know, applying a bunch of these patterns that we've been talking
about.

773
01:05:36,158 --> 01:05:44,580
There are built in prompts in Axminster so that your when you connect your like coding
assistant to it, you can use those prompts from inside the chat box.

774
01:05:45,297 --> 01:05:45,958
Okay.

775
01:05:45,958 --> 01:05:47,399
Yeah, that makes total sense.

776
01:05:47,399 --> 01:05:58,370
Did you learn anything during the development of this app that you were like, I didn't
realize this is a certain AI pattern I've been using or anything about software

777
01:05:58,370 --> 01:06:00,211
development in this space?

778
01:06:01,075 --> 01:06:02,476
I I learned a a lot.

779
01:06:02,476 --> 01:06:13,483
I have had because I'm an old school like Cocoa person who've like learned Objective-C and
AppKit and everything, I'd kind of always gone, yeah, SwiftUI looks interesting.

780
01:06:13,483 --> 01:06:17,395
I already know how to solve this problem using these other tools, so I'm gonna go and do
that.

781
01:06:17,395 --> 01:06:25,461
And on this thing I decided right, I am like going all in on SwiftUI, Swift 6, like you
know, modern concurrency

782
01:06:25,461 --> 01:06:26,783
patterns, everything.

783
01:06:26,783 --> 01:06:29,484
so I had to learn how all of that works.

784
01:06:29,484 --> 01:06:37,486
particularly until Xcode 27, models integrating with Xcode didn't do a great job
at doing idiomatic Swift.

785
01:06:37,486 --> 01:06:39,306
I don't know whether you found that.

786
01:06:39,336 --> 01:06:40,206
I certainly did.

787
01:06:40,206 --> 01:06:48,439
Like things like Swift Concurrency, they would just go, Hey, I've got a bunch of like
async await errors, so I'm just gonna write @MainActor on everything and then hope that

788
01:06:48,439 --> 01:06:49,139
it all goes away.

789
01:06:49,139 --> 01:06:53,370
~ Yeah, so I had to learn

790
01:06:53,376 --> 01:06:58,060
if it's even worse that it was using locks and stuff and it's like, Come on, seriously?

791
01:06:58,060 --> 01:06:59,061
Yeah.

792
01:06:59,061 --> 01:07:11,421
had to learn how Swift Concurrency works and like make sure that I had a good model of the
concurrency in my app, like so that I could express that to the LLM.

793
01:07:11,421 --> 01:07:14,763
yeah, I you know, I learned a ton about the technology.

794
01:07:15,044 --> 01:07:22,041
I learned a ton about App Store Review because like it it turns out that I have never been
the admin on a

795
01:07:22,041 --> 01:07:23,281
App Store Connect account before.

796
01:07:23,281 --> 01:07:31,473
I've always been a developer and someone hasn't been the app so I haven't actually added
an app to TestFlight or put an app into review before.

797
01:07:31,473 --> 01:07:34,857
yeah, we've all done anything we've done, we've all done for the first time, right?

798
01:07:34,857 --> 01:07:35,879
Like that that's nothing

799
01:07:35,879 --> 01:07:37,039
to be ashamed of.

800
01:07:38,066 --> 01:07:38,929
Cool.

801
01:07:38,929 --> 01:07:41,776
so what's the status of Axminster right now?

802
01:07:42,362 --> 01:07:46,189
so it went into review on Thursday.

803
01:07:46,189 --> 01:07:48,794
it's still awaiting review.

804
01:07:48,794 --> 01:07:52,940
so hopefully it'll be in the store by the time this this episode airs.

805
01:07:53,384 --> 01:07:55,119
Is it closed source?

806
01:07:55,119 --> 01:07:57,083
What's the monetization?

807
01:07:57,196 --> 01:07:57,727
Right.

808
01:07:57,727 --> 01:08:01,571
okay, so yeah, it is a a fixed upfront cost.

809
01:08:01,571 --> 01:08:04,554
I still haven't decided what that cost is gonna be.

810
01:08:04,554 --> 01:08:08,216
so I will work that out by the time we by the time it goes live.

811
01:08:08,227 --> 01:08:10,359
but yeah, like one time cost.

812
01:08:10,539 --> 01:08:12,631
the what was the other question?

813
01:08:14,401 --> 01:08:15,052
think you answered it.

814
01:08:15,052 --> 01:08:18,898
It was just like yeah, it's all closed source I assume and

815
01:08:18,898 --> 01:08:19,288
source.

816
01:08:19,288 --> 01:08:19,749
Right, yeah.

817
01:08:19,749 --> 01:08:32,517
So currently yes, I'm considering like open sourcing the part that represents the UML
model itself, so the in memory model, so that other people can build r related tools.

818
01:08:32,918 --> 01:08:41,854
if you're you know, if your listeners are interested in that then yeah, they should get in
contact and like let me know what they do with it and I'll see if there's interest in

819
01:08:41,854 --> 01:08:42,714
doing that.

820
01:08:43,089 --> 01:08:43,910
Cool.

821
01:08:44,081 --> 01:08:46,085
anything else before we close out?

822
01:08:48,005 --> 01:08:54,526
so yeah, I suppose like I just want to hit on the point that like this is somewhat
related.

823
01:08:54,526 --> 01:08:59,209
The Axminster kind of visual approach is somewhat related to spec driven development.

824
01:08:59,209 --> 01:09:09,633
but I want to kind of close off the idea that we're going back to a world of waterfall
development where we like, you know, spend ages writing project documentation and doing

825
01:09:09,633 --> 01:09:12,182
designs and things, and then only like after

826
01:09:12,182 --> 01:09:16,853
everyone in the company and all of the presidents and whatever have signed off that we
actually get to write a line of code.

827
01:09:16,853 --> 01:09:18,373
that's not what I'm proposing at all.

828
01:09:18,373 --> 01:09:22,536
What I'm because the you know these generative tools are incredibly quick.

829
01:09:22,536 --> 01:09:34,390
What I'm proposing is having a little think before you write any software and thinking
about what software you want to write and then encoding that thinking into like writing

830
01:09:34,390 --> 01:09:35,480
the software.

831
01:09:35,480 --> 01:09:39,121
so like, you know, I gave an example in my talk.

832
01:09:39,569 --> 01:09:43,671
where I did about an hour of planning before I write wrote some code.

833
01:09:43,671 --> 01:09:55,146
but I went through a bunch of alternative designs and the pros and cons of those and
selected some and explored their like impacts and whatever in that hour using the LLM as a

834
01:09:55,146 --> 01:09:56,257
tool to enable that.

835
01:09:56,257 --> 01:10:07,091
And that's what I'm like that's the kind of workflow that Axminster is geared towards is
the just like you know, spending an hour or so thinking about the problem you solve.

836
01:10:07,091 --> 01:10:08,966
So that you do a better solution.

837
01:10:09,561 --> 01:10:11,171
So it's funny you mentioned that.

838
01:10:11,171 --> 01:10:18,579
This weekend, I've been putting together a PR for just a a spec document.

839
01:10:18,579 --> 01:10:21,982
And I decided I just figured I'd try it.

840
01:10:21,982 --> 01:10:23,762
Let's burn some tokens.

841
01:10:23,764 --> 01:10:24,677
I had it run.

842
01:10:24,677 --> 01:10:29,829
So I'm ta I'm working with Claude and then I'm having Claude review the PR.

843
01:10:29,950 --> 01:10:32,782
And it's interesting how like

844
01:10:33,444 --> 01:10:39,246
There's some things that I think were legitimate, but I almost felt I was getting to a
point of analysis paralysis.

845
01:10:39,246 --> 01:10:39,546
Where

846
01:10:39,546 --> 01:10:52,370
it's like or over like one thing I I actually prompted in the last comment, the last PR,
I'm like, we're getting a little too specific about edge cases, but also things that like

847
01:10:52,370 --> 01:10:54,020
will need to be a little bit flexible.

848
01:10:54,020 --> 01:11:00,896
Cause I've run into issues where I've created a spec document and it said, Well, you said
you were gonna it was

849
01:11:00,896 --> 01:11:02,217
This is how it's supposed to be.

850
01:11:02,217 --> 01:11:04,990
And I'm like, Well, yeah, but that's before I knew about X, Y, and Z.

851
01:11:04,990 --> 01:11:10,425
It's like there needs to be a little bit of flexibility in these documents where it's
like, No, no, no, no, no.

852
01:11:10,425 --> 01:11:16,678
this is this doesn't have to be in the document because it's really not that strict of a
rule, so to speak.

853
01:11:16,678 --> 01:11:17,438
Yeah, absolutely.

854
01:11:17,438 --> 01:11:22,481
So one of the patterns I have in the book is Defer the Details, which is just about
saying, Yeah, we don't need to answer that.

855
01:11:22,481 --> 01:11:24,377
don't worry about it.

856
01:11:24,557 --> 01:11:26,023
Or like let's see let's

857
01:11:26,023 --> 01:11:28,875
see whether people complain when when we put that in TestFlight.

858
01:11:29,052 --> 01:11:29,732
Yeah yeah yeah.

859
01:11:29,732 --> 01:11:30,204
Yep.

860
01:11:30,204 --> 01:11:31,215
Yeah, exactly.

861
01:11:31,215 --> 01:11:34,388
Let's f let's experiment and see what people say.

862
01:11:35,250 --> 01:11:36,942
Graham, thank you so much for coming on.

863
01:11:36,942 --> 01:11:38,273
This was awesome.

864
01:11:38,329 --> 01:11:39,001
Absolute pleasure.

865
01:11:39,001 --> 01:11:40,813
It's I really enjoyed the conversation.

866
01:11:40,813 --> 01:11:41,644
Thank you.

867
01:11:41,819 --> 01:11:50,239
Yeah, we'll put links to your website and blog posts, etc.

868
01:11:50,239 --> 01:11:53,023
Hopefully we'll have an App Store link in the show notes as well.

869
01:11:53,023 --> 01:11:55,726
anything else you want to plug before we close out?

870
01:11:56,394 --> 01:11:57,489
nope, that sounds good.

871
01:11:57,489 --> 01:11:59,234
thanks again for the conversation.

872
01:11:59,441 --> 01:11:59,941
Yeah.

873
01:11:59,941 --> 01:12:03,183
if people have any questions or follow-up, let us know.

874
01:12:03,183 --> 01:12:08,815
you can find me I'm at BrightDigit.com, leogdion on the socials.

875
01:12:08,815 --> 01:12:18,258
if you have it, definitely you should think about subscribing to the newsletter as well as
the YouTube channel and like it and subscribe the podcast as well.

876
01:12:18,418 --> 01:12:21,029
thank you so much and I look forward to talking to you later.

877
01:12:21,029 --> 01:12:22,159
Bye everybody.