1
0:0:0,06 --> 0:0:2,44
Michael: Hello and welcome to PostgresFM, a weekly show about

2
0:0:2,44 --> 0:0:3,36
all things PostgreSQL.

3
0:0:3,62 --> 0:0:5,98
I am Michael, founder of pgMustard, and I'm joined as usual by

4
0:0:5,98 --> 0:0:7,64
Nik, founder of PostgresAI.

5
0:0:7,64 --> 0:0:8,34
Hey, Nik.

6
0:0:8,54 --> 0:0:9,26
Nikolay: Hi, Michael.

7
0:0:9,719999 --> 0:0:13,34
Michael: And we also have Radim Marek with us, who is a consultant

8
0:0:13,34 --> 0:0:17,04
and Postgres enthusiast, whose BoringSQL site we have mentioned

9
0:0:17,04 --> 0:0:18,46
many times on the show.

10
0:0:18,68 --> 0:0:19,66
Welcome, Radim.

11
0:0:20,22 --> 0:0:20,8
Radim: Hello guys.

12
0:0:20,8 --> 0:0:21,96
Thank you for having me.

13
0:0:21,96 --> 0:0:22,9
Nikolay: Thank you for coming.

14
0:0:22,96 --> 0:0:23,74
Michael: Yeah, absolutely.

15
0:0:24,18 --> 0:0:29,18
So we wanted to talk about a newish project you have kicked off

16
0:0:29,18 --> 0:0:33,8
called RegreSQL about regression testing SQL queries or SQL

17
0:0:33,8 --> 0:0:34,3
queries.

18
0:0:34,86 --> 0:0:35,64
Very cool.

19
0:0:35,66 --> 0:0:39,82
It got quite a lot of uptick on Hacker News, a lively discussion

20
0:0:39,84 --> 0:0:40,54
on there.

21
0:0:40,64 --> 0:0:41,14
Congrats.

22
0:0:41,36 --> 0:0:45,28
But also, I'd be interested in discussing with you about how

23
0:0:45,28 --> 0:0:48,36
you got into the topic, like regression testing in general.

24
0:0:48,7 --> 0:0:50,04
Where would you like to start?

25
0:0:50,54 --> 0:0:53,04
Radim: Okay so that's the tough topic to start with.

26
0:0:53,36 --> 0:0:57,28
So effectively it wasn't my project or it didn't start it as

27
0:0:57,28 --> 0:1:1,36
my project it actually has started as a complementary project

28
0:1:1,4 --> 0:1:6,3
to a book called The Art of Postgres, which I still believe is

29
0:1:6,3 --> 0:1:9,9
one of the best things you can do to learn how to work with Postgres.

30
0:1:9,96 --> 0:1:11,02
Nikolay: Dimitri, right?

31
0:1:11,74 --> 0:1:12,54
Radim: Yes, Dimitri.

32
0:1:12,62 --> 0:1:16,56
And I'm trying to get in touch with him forever, like last 4

33
0:1:16,56 --> 0:1:20,2
years, But I'm not able, so maybe if he hears this podcast that

34
0:1:20,2 --> 0:1:22,04
I will have a chance to talk to him.

35
0:1:22,84 --> 0:1:27,1
So I did start with the project because I was kind of a bit worried

36
0:1:27,1 --> 0:1:31,12
about how people deal with the SQL queries in their code.

37
0:1:31,38 --> 0:1:34,74
Because usually it's either generated by ORM and you don't see

38
0:1:34,74 --> 0:1:41,02
it or it was a string hard-coded in my case Go file so you can

39
0:1:41,02 --> 0:1:42,76
see it in TypeScript files.

40
0:1:44,44 --> 0:1:48,2
And all the testing around this one originally kind of bothered

41
0:1:48,2 --> 0:1:52,2
me because you have an infrastructure which was difficult to

42
0:1:52,2 --> 0:1:54,3
start, set up, and everything.

43
0:1:54,4 --> 0:1:56,62
And I said there must be a simple way.

44
0:1:57,26 --> 0:2:1,52
Something that actually running a simple query doesn't take minutes

45
0:2:1,56 --> 0:2:6,14
or seconds to do all the infrastructure tiered up, tiered down.

46
0:2:6,78 --> 0:2:10,52
And I found RegreSQL that is doing this job.

47
0:2:10,52 --> 0:2:12,58
And I tried to fix the project.

48
0:2:13,94 --> 0:2:16,96
And fast forward, I think 4 years, because it's actually quite

49
0:2:16,96 --> 0:2:18,06
a long time ago.

50
0:2:18,74 --> 0:2:22,5
I found another use cases because I'm working on some educational

51
0:2:22,84 --> 0:2:27,42
products and I needed to test whatever
users are hitting the

52
0:2:27,52 --> 0:2:29,48
metrics that I'm interested.

53
0:2:29,54 --> 0:2:33,34
And then I kind of figure out,
yes, this is actually where it

54
0:2:33,34 --> 0:2:37,34
can come in because it was comparing
whatever the output the

55
0:2:37,34 --> 0:2:40,84
correctness of the query and that
was only step away because

56
0:2:40,84 --> 0:2:44,94
then I realized okay every single
incident is based on increased

57
0:2:45,02 --> 0:2:49,96
time on SQL queries and that was
just a you know small step to

58
0:2:49,96 --> 0:2:52,98
go from something that is predictable
because time is unpredictable.

59
0:2:54,22 --> 0:2:57,98
I got into something predictable
and this is where I started.

60
0:2:57,98 --> 0:3:0,58
So I started committing, I started
adjusting it.

61
0:3:0,86 --> 0:3:4,74
And right now I'm actually working
on a version 2.0, which is

62
0:3:4,74 --> 0:3:10,06
a big update in terms of functionality
and in terms of developer

63
0:3:10,08 --> 0:3:10,58
experience.

64
0:3:11,4 --> 0:3:14,2
Nikolay: But how do you approach
the testing?

65
0:3:15,08 --> 0:3:16,58
So I understand the tool.

66
0:3:17,02 --> 0:3:20,2
For regression testing, it's super
important these days, especially

67
0:3:20,22 --> 0:3:23,6
because there are no more excuses
not to have 100% coverage of

68
0:3:23,6 --> 0:3:28,68
tests and proper coverage, not
just like good-looking coverage.

69
0:3:29,18 --> 0:3:35,36
And performance testing in CI,
and even before your AI assistant

70
0:3:35,5 --> 0:3:38,3
commits and pushes code, that should
have happened before probably.

71
0:3:38,42 --> 0:3:41,64
How do you approach the fact that
we need a lot of data to have

72
0:3:41,64 --> 0:3:42,48
proper plans?

73
0:3:43,82 --> 0:3:47,9
Radim: Okay, so this is effectively
a never-ending topic, how

74
0:3:47,9 --> 0:3:53,16
even developers can get to something
which is representative

75
0:3:53,46 --> 0:3:54,36
of their database.

76
0:3:54,66 --> 0:3:57,78
I actually call this 1, I think
that happened somewhere on Hacker

77
0:3:57,78 --> 0:4:0,26
News with somebody from Supabase.

78
0:4:0,86 --> 0:4:5,78
We came up to a topic which was
called, it works on my database.

79
0:4:6,76 --> 0:4:8,04
That's the same issue.

80
0:4:8,24 --> 0:4:14,18
But to answer your question, as
I said, initially I naively thought

81
0:4:14,18 --> 0:4:18,88
I can do a time, But time is all
over the place and that wasn't

82
0:4:18,88 --> 0:4:19,3
working.

83
0:4:19,3 --> 0:4:22,54
So the only way how to approach
this 1 is buffers and cost.

84
0:4:22,54 --> 0:4:24,72
Nikolay: Let's pause here because
I think it's super important.

85
0:4:24,72 --> 0:4:26,86
I think industry doesn't understand
it yet.

86
0:4:27,72 --> 0:4:30,7
We chatted about buffers with Michael
2 years.

87
0:4:30,7 --> 0:4:34,04
Finally, Postgres 18 has it by
default and explain and analyze,

88
0:4:34,04 --> 0:4:36,3
this is great, like absolutely
great.

89
0:4:36,62 --> 0:4:40,28
So, industry doesn't realize how
important it is to shift focus

90
0:4:40,28 --> 0:4:42,38
from timing to buffers still, right?

91
0:4:42,58 --> 0:4:44,16
Why timing is not enough?

92
0:4:44,44 --> 0:4:48,04
Why is it not so good when you
have to control performance.

93
0:4:48,08 --> 0:4:50,04
What's your point of view on this?

94
0:4:50,5 --> 0:4:52,32
Radim: Okay, so timing.

95
0:4:52,84 --> 0:4:56,52
Timing is a perfect metric because
this is what you see if you

96
0:4:56,52 --> 0:4:57,94
come in the middle of incident.

97
0:4:58,14 --> 0:5:0,92
Timing is actually the easiest
to spot metric.

98
0:5:1,02 --> 0:5:4,62
Queries are timing out, Hopefully
all of us have, you know, statement

99
0:5:4,64 --> 0:5:8,46
timeouts, transaction timeouts,
everything in place.

100
0:5:8,6 --> 0:5:10,54
Nikolay: And humans need the good
timing.

101
0:5:10,56 --> 0:5:12,18
This is our final goal, right?

102
0:5:12,6 --> 0:5:13,1
Radim: Exactly.

103
0:5:14,04 --> 0:5:18,5
But the problem is, A, if you want something fast, explain alone

104
0:5:18,5 --> 0:5:20,98
doesn't give you timing because that's just, you know, what the

105
0:5:20,98 --> 0:5:21,9
planner thinks.

106
0:5:22,2 --> 0:5:27,66
And EXPLAIN ANALYZE can't give you something that is predictable.

107
0:5:27,74 --> 0:5:31,04
So if you want to do a regression testing, You can't rely if

108
0:5:31,04 --> 0:5:36,78
you are running on, I don't know, underutilized AMD machine or

109
0:5:36,78 --> 0:5:37,56
ARM machine.

110
0:5:39,34 --> 0:5:43,48
Or, you know, everybody these days, we have Apple Silicon Macs

111
0:5:43,48 --> 0:5:45,02
and they are super fast.

112
0:5:45,02 --> 0:5:46,86
Their I/O is fast.

113
0:5:47,22 --> 0:5:49,94
So timing is generally, that's the-

114
0:5:50,14 --> 0:5:51,48
Nikolay: Doesn't work in CI.

115
0:5:51,82 --> 0:5:52,78
Radim: It doesn't, yes.

116
0:5:52,82 --> 0:5:56,82
Well, it doesn't even work in production because you will get

117
0:5:56,88 --> 0:5:58,76
a big spread of queries.

118
0:5:58,78 --> 0:5:59,86
You still have outliers.

119
0:6:0,28 --> 0:6:2,92
1 customer can hit 10,000 rows.

120
0:6:2,92 --> 0:6:4,6
1 customer hits 5 rows.

121
0:6:4,6 --> 0:6:5,44
How do you...

122
0:6:5,46 --> 0:6:7,6
Nikolay: What's the locking weight on locking acquisition as

123
0:6:7,6 --> 0:6:8,1
well?

124
0:6:8,26 --> 0:6:9,26
Yeah, yeah.

125
0:6:10,16 --> 0:6:10,8
That's great.

126
0:6:10,8 --> 0:6:11,3
Yeah.

127
0:6:11,94 --> 0:6:12,8
I think it's even

128
0:6:12,8 --> 0:6:19,26
Michael: worse in CI because flaky tests are a nightmare.

129
0:6:19,4 --> 0:6:24,12
So even if, for example, we accepted double the time, if we said

130
0:6:24,12 --> 0:6:28,4
like fail this test if it's double the time and that worked 99%

131
0:6:29,02 --> 0:6:33,34
of runs, when it fails but for a bad reason, when it fails, it's

132
0:6:33,34 --> 0:6:37,08
picked the same plan, no indexing changes, nothing has actually

133
0:6:37,08 --> 0:6:39,96
changed, but there was just some something else happening on

134
0:6:39,96 --> 0:6:43,38
the machine at that moment, and it took, it kind of blitzed past

135
0:6:43,38 --> 0:6:47,22
its p99, we got that 1 time it was slow, and the test fails,

136
0:6:47,44 --> 0:6:50,38
that's not a flaky test, that's a test that sometimes fails when

137
0:6:50,38 --> 0:6:51,78
it shouldn't, like false positives.

138
0:6:51,78 --> 0:6:54,56
And those are always the nightmare tests.

139
0:6:54,88 --> 0:6:59,34
Nikolay: Before we proceed to the solution, we all agree, buffers,

140
0:6:59,38 --> 0:6:59,88
right?

141
0:6:59,96 --> 0:7:2,94
Still like let's pause maybe, I have a crazy idea, I'm thinking

142
0:7:3,08 --> 0:7:4,44
for several years about this.

143
0:7:4,44 --> 0:7:6,18
Maybe it's time to implement it.

144
0:7:6,18 --> 0:7:9,84
Probably it's a better topic for our hacking sessions with Kirk

145
0:7:9,84 --> 0:7:12,54
and Andrey we have usually on Wednesdays live.

146
0:7:12,88 --> 0:7:17,9
But your opinion, I'm very interested to hear your opinion.

147
0:7:18,28 --> 0:7:22,12
What if we still keep analyzing timing, but imagine, you know,

148
0:7:22,12 --> 0:7:23,42
wait events, right?

149
0:7:24,0 --> 0:7:27,54
Which usually are used in aggregated form, like how many sessions,

150
0:7:27,78 --> 0:7:28,94
aggregate, and so on.

151
0:7:28,94 --> 0:7:32,94
What if that analysis would be
brought to explain and analyze

152
0:7:33,62 --> 0:7:36,36
to understand how this timing was
spent exactly.

153
0:7:36,38 --> 0:7:39,3
Was it heavyweight lock acquisition
waiting?

154
0:7:39,86 --> 0:7:42,54
Or I/O was too slow this time?

155
0:7:43,14 --> 0:7:47,72
If in this case, wouldn't it help
to extract the, like to structure

156
0:7:48,12 --> 0:7:53,22
the whole latency, to split it
to pieces, and in CI context,

157
0:7:53,82 --> 0:7:58,46
extract pieces which are like unpredictable,
okay, this doesn't

158
0:7:58,46 --> 0:8:1,28
matter, okay, waiting time, it
doesn't matter, waiting on lock

159
0:8:1,28 --> 0:8:1,78
acquisition.

160
0:8:1,98 --> 0:8:7,54
But here we see a lot of actual
CPU non-waiting, which usually

161
0:8:7,54 --> 0:8:8,76
is null, right?

162
0:8:9,34 --> 0:8:10,74
But there is a trick there.

163
0:8:11,54 --> 0:8:15,02
But anyway, wouldn't it help to
keep in the, like, still keep

164
0:8:15,02 --> 0:8:19,18
timing as main metric for regression,
testing in CI for performance?

165
0:8:19,84 --> 0:8:23,6
Radim: I believe in theory that
would work, but for me the complexity

166
0:8:23,96 --> 0:8:28,5
behind this is already implemented
on the cost and buffer level.

167
0:8:29,16 --> 0:8:33,38
So I'm not sure if the benefit
would actually be having better

168
0:8:33,38 --> 0:8:33,88
metrics.

169
0:8:34,92 --> 0:8:39,26
Because what I found out is, and
we can get to the discussion

170
0:8:39,28 --> 0:8:41,14
is, you know, buffers are fixed.

171
0:8:41,14 --> 0:8:45,6
If you have a predictable data
set that you start with, and you

172
0:8:45,6 --> 0:8:48,24
have a ideal state scenario.

173
0:8:48,34 --> 0:8:51,14
I think that's the important ideal
state because production is

174
0:8:51,14 --> 0:8:52,36
never in ideal state.

175
0:8:52,36 --> 0:8:55,58
You will have a bloat, you will
have a, so, you know, my regression

176
0:8:55,58 --> 0:8:57,1
testing is never- Always changing.

177
0:8:57,8 --> 0:9:0,04
Exactly, everything is changing,
so it depends.

178
0:9:0,32 --> 0:9:4,12
But if you want to do a regression
testing, you just need a matrix,

179
0:9:4,12 --> 0:9:6,8
1 or 2 of them, which are predictable.

180
0:9:7,36 --> 0:9:10,02
Buffers are always predictable.

181
0:9:10,64 --> 0:9:14,28
Nikolay: Yeah, I'm just, I'm so
so interesting because you came

182
0:9:14,34 --> 0:9:17,96
and talked what I was like shouting,
shouting on every event

183
0:9:17,96 --> 0:9:18,46
basically.

184
0:9:18,72 --> 0:9:22,72
Buffers, I even have t-shirt with
buff, explain what is buffers,

185
0:9:22,72 --> 0:9:23,94
where buffers is bold.

186
0:9:23,98 --> 0:9:24,14
By the

187
0:9:24,14 --> 0:9:26,28
Michael: way, it can depend a little
bit, unfortunately.

188
0:9:26,28 --> 0:9:28,78
It depends what you mean by buffers,
but like if you include

189
0:9:28,78 --> 0:9:32,62
planning buffers, which are sometimes
reported, like there are

190
0:9:32,68 --> 0:9:37,54
states that, like for example,
if for example, autovacuum just

191
0:9:37,54 --> 0:9:40,68
ran, you might end up with a different,
if analyze's just run,

192
0:9:40,68 --> 0:9:42,88
you might end up with a different
number of planning buffers

193
0:9:42,88 --> 0:9:45,36
than if it hasn't just run or if
it hasn't run recently.

194
0:9:45,38 --> 0:9:48,04
So there are like slight variations.

195
0:9:48,9 --> 0:9:50,64
So yeah, it might not be perfect.

196
0:9:50,8 --> 0:9:53,3
But cost is an interesting 1, for
sure.

197
0:9:53,3 --> 0:9:57,1
Nikolay: So, okay, what I'm going
to do, I will postpone my comments

198
0:9:57,1 --> 0:10:0,92
about why not only buffers, and
maybe let's return to timing,

199
0:10:0,92 --> 0:10:4,9
and we will discuss maybe later,
but let's agree with you, like

200
0:10:4,9 --> 0:10:10,08
buffers is the thing that doesn't
depend on the state so much.

201
0:10:10,08 --> 0:10:13,98
So we just check hits and reads of shared buffers, also temporary,

202
0:10:13,98 --> 0:10:14,66
like everything.

203
0:10:15,06 --> 0:10:15,74
It's I.O.

204
0:10:15,8 --> 0:10:17,72
Database is all about I.O.

205
0:10:17,72 --> 0:10:18,22
Right?

206
0:10:18,42 --> 0:10:22,44
But still the question remains how like your approach, like you

207
0:10:22,44 --> 0:10:26,82
need a lot of data to reproduce the executor behavior so you

208
0:10:26,82 --> 0:10:28,86
will see proper number of buffers, right?

209
0:10:28,86 --> 0:10:31,16
You cannot do it with database sizes small.

210
0:10:32,52 --> 0:10:33,62
Radim: Yes, that's the part.

211
0:10:33,62 --> 0:10:36,82
It works on my database because developers they will start with

212
0:10:36,82 --> 0:10:37,32
hundreds.

213
0:10:37,42 --> 0:10:40,24
Cardinality of the data is going to be very small.

214
0:10:41,6 --> 0:10:45,52
And I haven't seen a single company that wouldn't have a problem

215
0:10:45,52 --> 0:10:51,5
with how to get a kind of representative set of data to developers

216
0:10:51,66 --> 0:10:52,16
machines.

217
0:10:52,36 --> 0:10:56,18
Well, I know there are attempts, but like if you take an average

218
0:10:56,38 --> 0:11:0,08
software team, they usually have huge issues on this one.

219
0:11:0,24 --> 0:11:4,08
So my issue, and that's actually part of the reaction to Michael's

220
0:11:4,08 --> 0:11:8,0
comment, is I do always have to start with a clean slate.

221
0:11:8,0 --> 0:11:11,12
So for me, this is not a production, so I have to count that

222
0:11:11,12 --> 0:11:13,64
the vacuum ran and the statistics are up to date.

223
0:11:14,54 --> 0:11:18,88
The original solution, which I described, fixtures, they are

224
0:11:19,18 --> 0:11:22,9
the obvious solution, so you know I will always have to improve

225
0:11:22,9 --> 0:11:24,84
the ways how to improve the fixtures.

226
0:11:25,4 --> 0:11:30,98
But the way how to do it, and this is actually my roadmap towards

227
0:11:30,98 --> 0:11:34,04
version 2.0, is to do a snapshot building.

228
0:11:34,86 --> 0:11:37,26
You've probably seen, well you actually seen because you have

229
0:11:37,26 --> 0:11:39,44
discussed it, that instant database clones.

230
0:11:39,48 --> 0:11:44,18
That article wasn't a coincidence, it was actually for me need

231
0:11:44,18 --> 0:11:49,34
how to get a fast start state for the regression testing, and

232
0:11:49,34 --> 0:11:50,94
I needed something that works.

233
0:11:51,06 --> 0:11:54,76
This is why I had to dig, and then I actually, same as you guys,

234
0:11:54,76 --> 0:11:58,32
I was surprised it's already there, you know, because it wasn't

235
0:11:58,32 --> 0:12:2,22
something I was aware actually exists in version 18, but suddenly

236
0:12:2,28 --> 0:12:3,54
everything was done.

237
0:12:3,9 --> 0:12:7,44
So it's not for production use, which you mentioned correctly,

238
0:12:7,54 --> 0:12:8,72
because there are still issues.

239
0:12:9,18 --> 0:12:16,24
But if you fix a easy to get starting point, if you imagine you

240
0:12:16,24 --> 0:12:21,34
have a predictable build, which actually is coming, it's now

241
0:12:21,34 --> 0:12:24,44
already committed, but it's coming for version 2.0 as part of

242
0:12:24,44 --> 0:12:25,38
the developer experience.

243
0:12:26,04 --> 0:12:28,86
And you have a fresh statistics, you can run the query.

244
0:12:29,68 --> 0:12:32,9
And if your configuration, Postgres configuration is actually

245
0:12:32,9 --> 0:12:36,38
a very interesting part because you can change the cost, how

246
0:12:36,38 --> 0:12:41,42
it's calculated but you will always get the same buffers.

247
0:12:42,4 --> 0:12:42,9
Exactly.

248
0:12:44,18 --> 0:12:47,14
Nikolay: If you talk about hits plus reads probably the distribution

249
0:12:47,16 --> 0:12:47,96
will be different.

250
0:12:47,96 --> 0:12:50,74
Also hits might be hitting the same buffer multiple times.

251
0:12:50,74 --> 0:12:52,44
So there are nuances there.

252
0:12:53,0 --> 0:12:56,18
Radim: But the total, it's like
always the same.

253
0:12:56,28 --> 0:13:1,96
So, and you Nik, you said it,
we all of us came from a spindle

254
0:13:1,96 --> 0:13:2,36
disk.

255
0:13:2,36 --> 0:13:8,0
So you remember times where you
try to hit the control C on the

256
0:13:8,0 --> 0:13:12,08
query and it was just flushing
WAL files on a disk.

257
0:13:12,38 --> 0:13:15,36
And we can say that I/O is cheap
these days, but it's not.

258
0:13:15,36 --> 0:13:18,22
It's still the only metric that
you have to do.

259
0:13:18,24 --> 0:13:21,64
And I/O, whatever there's a contention,
will affect time.

260
0:13:22,1 --> 0:13:24,96
So for me, certainly buffers is
number 1.

261
0:13:24,96 --> 0:13:26,66
So there is nothing more predictable.

262
0:13:27,26 --> 0:13:29,64
Cost is very close.

263
0:13:30,06 --> 0:13:34,28
And I actually did a, I haven't
done like a scientific research

264
0:13:34,28 --> 0:13:38,8
on the topic, but if you take the
1 version of Postgres and you

265
0:13:38,9 --> 0:13:43,04
run through different architectures
and same settings, the variance

266
0:13:43,04 --> 0:13:44,14
is surprisingly low.

267
0:13:44,14 --> 0:13:44,44
Not all

268
0:13:44,44 --> 0:13:44,94
Nikolay: settings.

269
0:13:45,06 --> 0:13:48,38
There is a specific set of settings
because you can have much

270
0:13:48,38 --> 0:13:51,02
less memory, shared buffers can
be very small, you just need

271
0:13:51,02 --> 0:13:54,96
to set proper effective cache size,
proper all the planner settings

272
0:13:54,96 --> 0:13:55,56
and work memory.

273
0:13:55,56 --> 0:13:57,1
We have an article about it.

274
0:13:57,26 --> 0:13:59,96
And then I agree with you, it will
be the same planner behavior

275
0:13:59,96 --> 0:14:0,86
for the same version.

276
0:14:0,86 --> 0:14:1,6
That's it.

277
0:14:1,64 --> 0:14:2,14
It's great.

278
0:14:2,14 --> 0:14:2,84
It's magic.

279
0:14:4,54 --> 0:14:9,82
You can have Raspberry Pi and repeat
the haver of huge like 800

280
0:14:9,96 --> 0:14:10,78
core machine.

281
0:14:11,04 --> 0:14:11,76
It's insane.

282
0:14:11,98 --> 0:14:12,48
Radim: Exactly.

283
0:14:12,66 --> 0:14:13,48
This is what I did.

284
0:14:13,48 --> 0:14:18,14
I actually took my old Raspberry
Pi, I took Hetzner, I took a

285
0:14:18,14 --> 0:14:22,06
GCP machine And the difference
was something around 2%.

286
0:14:22,12 --> 0:14:26,78
So right now I have a threshold
of 10% by default, but I'm looking

287
0:14:26,78 --> 0:14:28,64
forward to actually make it configurable.

288
0:14:29,28 --> 0:14:30,12
That's great.

289
0:14:31,1 --> 0:14:32,28
And let's let people decide.

290
0:14:32,28 --> 0:14:37,04
But and then the answer to all
your initial questions is I don't

291
0:14:37,04 --> 0:14:40,74
have to care about timing because
in production, if you would

292
0:14:40,74 --> 0:14:45,28
have 1000 transactions per second,
I/O will get saturated, you

293
0:14:45,28 --> 0:14:47,78
know, And that will naturally raise
the timing.

294
0:14:47,9 --> 0:14:52,28
So if you care about buffers, you
don't have to care about timing,

295
0:14:52,28 --> 0:14:56,12
because timing is just an extrapolation
of problems that happened

296
0:14:56,12 --> 0:14:58,3
before actually the query returns.

297
0:14:58,44 --> 0:15:0,96
Nikolay: Some people say, like,
let's do proper full-fledged

298
0:15:1,1 --> 0:15:5,46
benchmarking just to study behavior
of planner for a few queries.

299
0:15:5,46 --> 0:15:8,36
It's wrong idea because you have
so many moving parts around.

300
0:15:8,48 --> 0:15:12,24
You just narrow down to single
plan, single session, right?

301
0:15:12,24 --> 0:15:14,62
And focus on what matters.

302
0:15:15,14 --> 0:15:16,7
But here, like my...

303
0:15:16,86 --> 0:15:17,58
I have...

304
0:15:17,58 --> 0:15:19,2
Before I had that question.

305
0:15:20,42 --> 0:15:23,6
So you mentioned create database in Postgres 18, you mean strategy,

306
0:15:23,68 --> 0:15:24,18
right?

307
0:15:25,12 --> 0:15:28,6
When you can copy-on-write clone and when you have create database

308
0:15:28,64 --> 0:15:33,76
with clones, template 1 database, but not in regular form, but

309
0:15:34,02 --> 0:15:36,26
basically very fast because of copy-on-write.

310
0:15:36,26 --> 0:15:40,1
Does it mean you use ZFS or something, Btrfs or what?

311
0:15:40,4 --> 0:15:42,44
Radim: Well, it depends what you want.

312
0:15:42,62 --> 0:15:45,04
I'm not forcing people to use anything.

313
0:15:45,3 --> 0:15:48,74
I'm just offering an option if you want fast because you know,

314
0:15:48,76 --> 0:15:53,16
another part of frustration was that everything in your integration

315
0:15:53,3 --> 0:15:55,62
test that touches the database is slow.

316
0:15:56,78 --> 0:15:57,12
Yeah.

317
0:15:57,12 --> 0:15:59,96
And the way how people normally use it, it's slow, but you know,

318
0:15:59,96 --> 0:16:2,62
running most of the queries it's like literally milliseconds.

319
0:16:3,04 --> 0:16:3,56
Nikolay: Yeah, yeah.

320
0:16:3,56 --> 0:16:6,76
And we have new 1 branching, we have other copy-on-write branching,

321
0:16:6,76 --> 0:16:7,7
so that's great.

322
0:16:7,8 --> 0:16:11,88
And here, if you compare, do you know about DBLab or not?

323
0:16:12,7 --> 0:16:13,46
Radim: Yes, I know.

324
0:16:13,46 --> 0:16:18,84
Nikolay: Yeah, so I'm comparing the various approaches and this

325
0:16:18,86 --> 0:16:22,44
approach, like when we have multiple logical databases and physical,

326
0:16:22,78 --> 0:16:27,28
is great because this is exactly what we try to achieve with

327
0:16:27,28 --> 0:16:31,32
the DBLab, but not many people realize what you just said.

328
0:16:31,32 --> 0:16:34,78
I hope with this episode more people will realize.

329
0:16:35,22 --> 0:16:39,14
And the thing is that if you allocate this machine, you can have

330
0:16:39,14 --> 0:16:40,06
a huge database.

331
0:16:40,76 --> 0:16:46,64
This wonderful Postgres 18 feature, if You use ZFS, we don't

332
0:16:46,64 --> 0:16:48,54
care about time anymore, almost.

333
0:16:49,08 --> 0:16:50,72
So we focus on buffers.

334
0:16:51,62 --> 0:16:54,44
Postgres doesn't know which file system we have.

335
0:16:55,32 --> 0:16:59,76
And then you can achieve constant price for many, many tests

336
0:16:59,76 --> 0:17:0,38
in parallel.

337
0:17:0,38 --> 0:17:4,78
Unlike Neon or Aurora, thin cloning, you need to pay.

338
0:17:5,02 --> 0:17:8,42
If you run 10 tests in parallel, for example, you can have 10

339
0:17:8,44 --> 0:17:12,66
ideas from LLM, you need to pay 10 times more for compute.

340
0:17:12,94 --> 0:17:15,52
Yes, storage is solved, but compute is not.

341
0:17:15,66 --> 0:17:19,28
Here, create database, 10 databases in parallel, it's good.

342
0:17:19,28 --> 0:17:22,48
The only thing compared to DBLab is that it's not possible, you

343
0:17:23,42 --> 0:17:27,42
cannot have a major upgrade inside a clone and have independent

344
0:17:27,44 --> 0:17:28,9
versions in parallel.

345
0:17:29,34 --> 0:17:33,0
But Maybe some more differences, but in general, it's great that

346
0:17:33,0 --> 0:17:34,78
it comes to Postgres naturally.

347
0:17:35,28 --> 0:17:36,76
This feature is a great feature.

348
0:17:36,82 --> 0:17:39,58
I hope it will get traction in CI environments.

349
0:17:40,08 --> 0:17:42,74
Radim: I think it's actually great to have this natively.

350
0:17:42,9 --> 0:17:44,24
I'm not saying it's perfect.

351
0:17:44,44 --> 0:17:45,64
It's just an option.

352
0:17:45,86 --> 0:17:49,78
Nobody is forced into a third-party
solutions, everybody will

353
0:17:49,82 --> 0:17:53,68
pick whatever they want and, you
know, I'm not against Neon or

354
0:17:53,68 --> 0:17:57,54
anything, I think you evaluate
your requirements and you say

355
0:17:57,54 --> 0:18:0,1
what's the best approach you want
to take.

356
0:18:0,1 --> 0:18:2,52
Nikolay: Well with Neon it's also
the same, like you can, you

357
0:18:2,52 --> 0:18:5,74
just pay for compute but also you
can study planning and behavior,

358
0:18:5,74 --> 0:18:6,92
put it into CI.

359
0:18:7,9 --> 0:18:12,38
The thing is that you need to focus
on buffers because cold startup

360
0:18:12,38 --> 0:18:13,44
will be different, right?

361
0:18:13,44 --> 0:18:15,62
But we don't care anymore if you
focus on buffers.

362
0:18:15,62 --> 0:18:16,62
That's the magic, right?

363
0:18:16,62 --> 0:18:17,32
That's great.

364
0:18:17,4 --> 0:18:18,78
And cost, also cost.

365
0:18:18,94 --> 0:18:21,84
Because this is what says how planner
things.

366
0:18:21,88 --> 0:18:25,32
The only problem is there are some
queries which are out of this

367
0:18:25,32 --> 0:18:25,82
methodology.

368
0:18:26,36 --> 0:18:33,0
For example, when people who deal
with RLS, they know this problem,

369
0:18:33,0 --> 0:18:33,28
right?

370
0:18:33,28 --> 0:18:37,86
If you have a query with counting
million rows and there is a

371
0:18:38,44 --> 0:18:43,3
hidden filter involving current
setting, it's going to be executed

372
0:18:43,66 --> 0:18:45,9
for each row And it's not buffers.

373
0:18:46,28 --> 0:18:47,66
It's pure CPU time.

374
0:18:48,08 --> 0:18:50,14
But these are exclusions, right?

375
0:18:50,14 --> 0:18:52,54
We have only limited number of
them, right?

376
0:18:53,62 --> 0:18:54,12
Radim: Exactly.

377
0:18:54,22 --> 0:18:58,26
I would say before hitting that
problem, that is 99% of problem

378
0:18:58,26 --> 0:18:59,68
we can solve much easier.

379
0:18:59,68 --> 0:19:0,32
Nikolay: I agree.

380
0:19:0,66 --> 0:19:5,22
And still you can troubleshoot
it on this machine because getting

381
0:19:5,22 --> 0:19:8,6
those million rows is much faster
if you have this copy-on-write

382
0:19:8,6 --> 0:19:9,74
and your methodology.

383
0:19:9,96 --> 0:19:14,12
I mean, what you said, it doesn't
mean you should stop thinking

384
0:19:14,12 --> 0:19:14,92
about timing, right?

385
0:19:14,92 --> 0:19:17,88
You still can think about timing,
right?

386
0:19:18,42 --> 0:19:19,6
Radim: You can, it will hit you.

387
0:19:19,6 --> 0:19:22,36
That's what's going to hit you
in production at the end of the

388
0:19:22,36 --> 0:19:25,58
day, because this is what the user
perceives when they render

389
0:19:25,58 --> 0:19:30,46
the page waiting for the output
or something, but it's not predictable.

390
0:19:31,24 --> 0:19:31,8088
Yeah, The

391
0:19:31,8088 --> 0:19:34,6
Nikolay: problem is we gave this
tool, DBLab, to many hundreds

392
0:19:34,6 --> 0:19:37,84
of engineers and noticed, I think
thousands already, and noticed

393
0:19:37,84 --> 0:19:42,14
that some people expect that it's
full reproduction of production

394
0:19:43,1 --> 0:19:43,6
behavior.

395
0:19:43,78 --> 0:19:48,24
They say, why my Index is being
created 3 times longer than in

396
0:19:48,24 --> 0:19:48,74
production.

397
0:19:49,12 --> 0:19:51,98
Well because quite different environment,
right?

398
0:19:52,54 --> 0:19:56,98
So there is some adjustment needed
in in a mindset to test like

399
0:19:56,98 --> 0:19:58,48
this, right?

400
0:19:58,74 --> 0:20:1,24
Well, okay enough about thing calling
and branching.

401
0:20:1,24 --> 0:20:2,72
Let's think about tooling.

402
0:20:2,96 --> 0:20:7,9
Michael: Yeah, I had a question
going back to cost tolerance.

403
0:20:7,92 --> 0:20:12,6
You mentioned allowing for 10%
higher costs, and that confused

404
0:20:12,62 --> 0:20:16,7
me slightly because I was thinking
on CI we'd always be running

405
0:20:16,7 --> 0:20:21,02
on the same, effectively a clone
of the same Database static

406
0:20:21,02 --> 0:20:24,68
state you mentioned why would the
cost why do we need to allow

407
0:20:24,68 --> 0:20:26,54
for 10% higher costs potentially

408
0:20:27,26 --> 0:20:31,86
Radim: that was initially my naive
protection effectively so

409
0:20:31,86 --> 0:20:36,62
before I did the benchmarking I
said you know 10% literally I

410
0:20:36,62 --> 0:20:40,26
just there was no measuring I just
said 10% sounds reasonable.

411
0:20:41,32 --> 0:20:44,44
When I did measuring, I found,
as we discussed, you know, there

412
0:20:44,44 --> 0:20:48,34
is slight variance, but it's not
actually, I think on average

413
0:20:48,34 --> 0:20:49,34
was below 2%.

414
0:20:49,84 --> 0:20:53,24
Nikolay: I would like to see details
here because I think there

415
0:20:53,24 --> 0:20:54,56
should be 0 difference.

416
0:20:56,32 --> 0:20:59,82
Let's talk, I'm going to follow
up with you on this because there

417
0:20:59,82 --> 0:21:2,72
are some cases which I don't know,
I'm super interested to understand.

418
0:21:3,14 --> 0:21:3,64
And

419
0:21:3,72 --> 0:21:4,22
Radim: yeah.

420
0:21:4,5 --> 0:21:6,54
I will just, you know, just to
finish the topic because, you

421
0:21:6,54 --> 0:21:11,82
know, I did extensive testing in
terms of establishing it I haven't

422
0:21:12,38 --> 0:21:15,28
dig into a reason because you know
some were our machine some

423
0:21:15,28 --> 0:21:19,8
were very you know Raspberry Pis
so that wasn't comparable and

424
0:21:19,8 --> 0:21:23,2
that was the where the difference
came from Some queries were

425
0:21:23,22 --> 0:21:23,42
0.

426
0:21:23,42 --> 0:21:25,38
Nikolay: But Planner doesn't know
about hardware.

427
0:21:25,38 --> 0:21:27,16
It doesn't know anything about
hardware.

428
0:21:27,8 --> 0:21:31,22
Radim: We actually had this discussion
on the Prague Postgres

429
0:21:31,22 --> 0:21:34,9
meetup 2 months ago, that it will
be actually very interesting

430
0:21:34,96 --> 0:21:39,3
to go into a differences between
ARM and Intel processors.

431
0:21:39,76 --> 0:21:43,78
What actually is, if there is some
default somewhere, I have,

432
0:21:43,78 --> 0:21:48,12
for example, seen that I/O startup
time, again, it doesn't involve

433
0:21:48,12 --> 0:21:49,64
buffers, but the I/O

434
0:21:49,64 --> 0:21:51,72
Startup times on the cold I/O

435
0:21:51,76 --> 0:21:54,36
Are much slower on ARM in general.

436
0:21:54,96 --> 0:21:57,0
I believe there might be something
else.

437
0:21:57,04 --> 0:21:57,98
Nikolay: Different code paths.

438
0:21:57,98 --> 0:21:58,48
Exactly.

439
0:21:59,24 --> 0:22:0,14
Right, right.

440
0:22:0,14 --> 0:22:0,4534
This I can imagine, yeah.

441
0:22:0,4534 --> 0:22:2,16
But it's not about hardware, just
code paths.

442
0:22:2,16 --> 0:22:3,34
The architecture might matter.

443
0:22:3,34 --> 0:22:4,26
Yeah, that's interesting.

444
0:22:4,66 --> 0:22:5,16
Yeah.

445
0:22:5,74 --> 0:22:8,44
What's also interesting here, I
think we touched an interesting

446
0:22:8,44 --> 0:22:8,94
topic.

447
0:22:9,52 --> 0:22:10,68
Could zoom in there.

448
0:22:10,68 --> 0:22:12,02
It's because it's super interesting.

449
0:22:12,04 --> 0:22:15,84
When they say 10%, I think it's
because in CI, you need to define

450
0:22:15,84 --> 0:22:18,96
some thresholds to define where,
when it's failed, when it's

451
0:22:18,96 --> 0:22:19,78
passed, right?

452
0:22:19,9 --> 0:22:24,9
And it's when you work manually,
you have some like thresholds

453
0:22:24,96 --> 0:22:27,84
in your mind, you don't realize,
but then you need to code it

454
0:22:27,84 --> 0:22:28,26
basically.

455
0:22:28,26 --> 0:22:29,62
And this is a challenge, right?

456
0:22:30,06 --> 0:22:30,3
Radim: Exactly.

457
0:22:30,3 --> 0:22:31,72
This is where it started from.

458
0:22:31,72 --> 0:22:35,8
And this is actually where I hit
a wall with the original release

459
0:22:35,8 --> 0:22:39,6
of the tool because people didn't
understand, you know, you said

460
0:22:39,6 --> 0:22:44,08
it, people didn't understand buffers,
why they matter so much.

461
0:22:44,2 --> 0:22:47,54
People didn't understand why something
is changing.

462
0:22:47,9 --> 0:22:49,6
And I had to start thinking again.

463
0:22:49,6 --> 0:22:52,6
And this is, you know, I think
if you would go to original blog

464
0:22:52,6 --> 0:22:56,02
post, now there is a notice that
I had to take the fixtures out

465
0:22:56,02 --> 0:23:0,04
of the test because people were
kind of linking it together and

466
0:23:0,04 --> 0:23:3,16
they were getting different data
sets and they didn't use it

467
0:23:3,16 --> 0:23:3,66
correctly.

468
0:23:3,84 --> 0:23:6,96
My fault was that I didn't specify
properly that that was the

469
0:23:6,96 --> 0:23:8,14
issue with the blog post.

470
0:23:8,24 --> 0:23:12,54
But this is why the version 2.0
is actually heading in the direction

471
0:23:12,98 --> 0:23:15,52
because we have touched on the
topic.

472
0:23:15,6 --> 0:23:20,34
You can have a multiple snapshots
and those snapshots will give

473
0:23:20,34 --> 0:23:24,6
you a different experience but
the same baseline on a given snapshot

474
0:23:24,84 --> 0:23:28,46
should be stable or more or less
stable over time.

475
0:23:28,9 --> 0:23:31,56
And then it depends because your
business will change, business

476
0:23:31,56 --> 0:23:32,64
requirements will change.

477
0:23:32,64 --> 0:23:36,78
Then for example, yesterday I committed
a change which is a selective

478
0:23:36,78 --> 0:23:37,16
update.

479
0:23:37,16 --> 0:23:41,12
So you can literally link a change
in baseline to a commit so

480
0:23:41,12 --> 0:23:46,42
you can say and blame what actually
was that kind of historical

481
0:23:46,44 --> 0:23:47,98
trend change on a query.

482
0:23:48,28 --> 0:23:51,3
So this kind of adds another dimension
because suddenly you don't

483
0:23:51,3 --> 0:23:54,64
have a good blame only who broke
something.

484
0:23:54,84 --> 0:23:59,62
You actually can trace it back
to how did it affect the performance.

485
0:24:0,04 --> 0:24:3,12
And this was actually on one of the
posts, Michael, you had on

486
0:24:3,12 --> 0:24:8,92
your blog about this optimization
from 7 milliseconds to something

487
0:24:9,14 --> 0:24:11,52
you hit the index scan.

488
0:24:11,52 --> 0:24:14,24
And that was actually a big inspiration
because I said, you know,

489
0:24:14,24 --> 0:24:17,36
this is actually very important
because the reason that query

490
0:24:17,36 --> 0:24:19,94
does 7 milliseconds, that doesn't
matter.

491
0:24:20,34 --> 0:24:25,12
But if you save so many buffers,
you should be able to trace

492
0:24:25,12 --> 0:24:27,28
it back even on the positive outcome.

493
0:24:27,28 --> 0:24:30,72
So at 1 point, there can be a layer
which will take all those

494
0:24:30,72 --> 0:24:31,22
optimizations.

495
0:24:31,8 --> 0:24:34,92
And this is where we will get to
very interesting topic and that's

496
0:24:34,92 --> 0:24:36,42
the topic in this AI.

497
0:24:37,04 --> 0:24:38,8
Nikolay: Right, experiments, right?

498
0:24:39,62 --> 0:24:41,92
Radim: Experiment, well, I wouldn't
call it experiments.

499
0:24:42,24 --> 0:24:45,82
I call it set reality because I 
believe most people-

500
0:24:45,82 --> 0:24:49,54
Nikolay: I mean, code is written
by AI only right now, right?

501
0:24:49,6 --> 0:24:50,1
Radim: Yes.

502
0:24:50,14 --> 0:24:50,64
Yeah,

503
0:24:51,7 --> 0:24:51,74
Nikolay: yeah.

504
0:24:51,74 --> 0:24:53,6
And how do we control it, right?

505
0:24:53,74 --> 0:24:58,26
So as I said, we need not only
100% coverage, but it should be

506
0:24:58,26 --> 0:25:0,68
real tests should be inside, right?

507
0:25:1,28 --> 0:25:3,14
Including performance, I agree.

508
0:25:3,9 --> 0:25:4,4
Yeah.

509
0:25:4,74 --> 0:25:7,76
Radim: So, yeah, we had a discussion
with Michael in London during

510
0:25:7,76 --> 0:25:8,76
a PGDay.

511
0:25:10,08 --> 0:25:16,08
If you have a context which AI
don't have at 1 point, you can

512
0:25:16,08 --> 0:25:20,72
then see regular SQL actually in
a very different light, because

513
0:25:20,8 --> 0:25:24,22
it knows the query, it knows the
baseline, it has a snapshot,

514
0:25:24,52 --> 0:25:26,88
and it has a context, which is
the schema.

515
0:25:27,12 --> 0:25:27,62
Nikolay: Exactly.

516
0:25:27,88 --> 0:25:28,6
And statistics.

517
0:25:29,1 --> 0:25:29,54
Radim: Exactly.

518
0:25:29,54 --> 0:25:30,36
You have statistics.

519
0:25:30,44 --> 0:25:33,92
So you have suddenly much more
data, which you need.

520
0:25:34,74 --> 0:25:38,56
And I had to, you know, all of
us, all of us have to think about

521
0:25:38,56 --> 0:25:40,92
how AI is going to change our work.

522
0:25:40,92 --> 0:25:41,62
I'm thinking

523
0:25:42,18 --> 0:25:43,52
Nikolay: every day since 2017.

524
0:25:45,36 --> 0:25:45,86
Radim: Exactly.

525
0:25:45,86 --> 0:25:49,34
I have a weekly session which I
do and you know, I'm trying to

526
0:25:49,34 --> 0:25:51,68
digest the news and do this 1.

527
0:25:51,74 --> 0:25:55,84
But there are things that just
confirmed that this is the right

528
0:25:55,84 --> 0:26:1,4
direction because the joke is 84%
of developers use some sort

529
0:26:1,4 --> 0:26:7,36
of assisted coding or the vibe
code or whatever, we all do it.

530
0:26:7,36 --> 0:26:7,82
Yes.

531
0:26:7,82 --> 0:26:12,08
But the same 84% of people, they
don't write SQL queries anymore.

532
0:26:12,44 --> 0:26:12,94
Nikolay: Yes.

533
0:26:13,14 --> 0:26:16,8
Radim: If they were bad in writing
SQL queries, you know how

534
0:26:16,8 --> 0:26:19,5
a LLM SQL queries look without
the context.

535
0:26:20,22 --> 0:26:24,24
I actually found it, you know,
this is the deformation you have

536
0:26:24,24 --> 0:26:26,88
when you look into a, how Planner
works.

537
0:26:26,88 --> 0:26:30,06
I found it very funny because it
doesn't differentiate whatever

538
0:26:30,06 --> 0:26:34,62
you're running Postgres 12, 14,
18, it will mix all that advice,

539
0:26:34,74 --> 0:26:38,58
it kind of reads the blog post
which is just you something wrong

540
0:26:38,8 --> 0:26:40,78
and people will just go with it.

541
0:26:41,24 --> 0:26:48,28
And if you put this 1 in context
of a 10 times more pull request,

542
0:26:48,52 --> 0:26:50,34
Who's going to review it?

543
0:26:51,04 --> 0:26:54,1
And now you see where all this
coming through because if you

544
0:26:54,1 --> 0:26:57,38
don't have time to review it, if
you don't have the human aspect

545
0:26:57,38 --> 0:27:0,06
of reviewing it, and if you don't
have the DBA knowledge, I think

546
0:27:0,06 --> 0:27:3,22
the lost DBA knowledge is actually
kind of key here.

547
0:27:3,74 --> 0:27:10,32
You need something that will govern
that a your tests are fine,

548
0:27:10,32 --> 0:27:13,94
you can write unit tests, integration
tests on a couple rows,

549
0:27:13,94 --> 0:27:16,96
but how do you do it if you have
to return 5,000 rows?

550
0:27:17,0 --> 0:27:19,9
This is very difficult And this
is what RegreSQL actually

551
0:27:19,9 --> 0:27:21,32
was doing ever since start.

552
0:27:21,32 --> 0:27:24,38
You can take a 5,000 rows and
compare them.

553
0:27:24,72 --> 0:27:27,8
And it can suggest you, you know,
order is wrong or maybe, you

554
0:27:27,8 --> 0:27:31,28
know, there is a wrong field and
it will notice it.

555
0:27:31,28 --> 0:27:35,34
So actually that comes somewhere
between unit test and integration

556
0:27:35,44 --> 0:27:40,12
test, it can measure or control
that your data and your contract

557
0:27:40,12 --> 0:27:41,18
is still valid.

558
0:27:41,4 --> 0:27:45,66
And this is where I believe the
direction of the tool is actually

559
0:27:45,66 --> 0:27:46,74
having a big future.

560
0:27:46,8 --> 0:27:48,22
Nikolay: That's absolutely great.

561
0:27:48,34 --> 0:27:52,4
I just like everything you say
makes total sense, absolute sense.

562
0:27:52,74 --> 0:27:53,24
Yeah.

563
0:27:53,44 --> 0:27:56,26
That's great that we like think
in the same direction.

564
0:27:56,6 --> 0:27:58,0
That's all I can say.

565
0:27:59,22 --> 0:28:0,26
Radim: Well, thank you.

566
0:28:0,42 --> 0:28:3,9
But I would say, you know, this
is, if you have experience dealing

567
0:28:3,9 --> 0:28:7,04
with production system, this is
what you will effectively arrive

568
0:28:7,04 --> 0:28:7,54
to.

569
0:28:7,9 --> 0:28:8,26
Yeah.

570
0:28:8,26 --> 0:28:11,76
Because you have to generalize,
you have to find underlying issues.

571
0:28:11,82 --> 0:28:14,56
Nikolay: We need guardrails and
protection, automated protection

572
0:28:14,62 --> 0:28:15,84
at a larger scale.

573
0:28:15,92 --> 0:28:21,24
We cannot say, oh, let's not, let's
prepare to run those pipelines,

574
0:28:21,38 --> 0:28:21,88
right?

575
0:28:21,88 --> 0:28:23,86
Because they're expensive and long.

576
0:28:23,96 --> 0:28:27,54
We need to make them cheap and
fast, radically, right?

577
0:28:27,66 --> 0:28:30,38
So we can run like 50 of them,
right?

578
0:28:30,38 --> 0:28:34,74
And check all the things and throw
away all hallucinated stuff,

579
0:28:34,74 --> 0:28:34,96
right?

580
0:28:34,96 --> 0:28:39,36
This is this is the way yeah This
is the way And then comes to

581
0:28:39,36 --> 0:28:42,08
in many details including for example
There are some hard problems

582
0:28:42,08 --> 0:28:44,58
which come from production side
not from development side.

583
0:28:44,58 --> 0:28:49,82
For example plan flips You know
like and then it's different

584
0:28:49,82 --> 0:28:53,7
so but it's a different problem
if we split these problems anyway,

585
0:28:53,72 --> 0:28:58,3
I agree the most danger to performance
right now is the AI.

586
0:28:58,36 --> 0:28:59,7
But it's also great.

587
0:28:59,7 --> 0:29:3,14
But, Like, I mean, it's wild west
right now, right?

588
0:29:3,68 --> 0:29:7,5
So we need some practices and tools
to be established.

589
0:29:8,44 --> 0:29:12,7
Radim: But just to kind of put
it in contrast, I actually believe

590
0:29:12,7 --> 0:29:16,08
most issues are actually very simple
because most of the incident

591
0:29:16,08 --> 0:29:20,32
they start, you know, I'm not sure
if I'm the only 1, but most

592
0:29:20,32 --> 0:29:23,4
incidents you see are literally
simple, missing index.

593
0:29:24,8 --> 0:29:28,04
Or somebody rewrites index without
understanding how it should

594
0:29:28,04 --> 0:29:28,68
be rewritten.

595
0:29:28,78 --> 0:29:31,8
And you know, they obviously verify
it locally.

596
0:29:31,8 --> 0:29:35,38
It works fine because they have
a small database but they don't

597
0:29:35,38 --> 0:29:39,06
notice that the query exploded
100 times in terms of buffers

598
0:29:39,06 --> 0:29:41,12
retrieved and that will have an
impact.

599
0:29:41,12 --> 0:29:44,68
Well, 100 times is easy, that usually
you can survive it.

600
0:29:44,76 --> 0:29:48,28
Nikolay: Well, important point
here also I noticed if you say

601
0:29:48,28 --> 0:29:50,74
1000 buffers people don't understand
you.

602
0:29:50,74 --> 0:29:55,68
If you say 8 megabytes they start
understanding how much it is,

603
0:29:55,68 --> 0:29:56,18
right?

604
0:29:56,72 --> 0:30:1,46
So you just multiply by block size
And this is magic.

605
0:30:2,42 --> 0:30:3,96
So simple magic, you know.

606
0:30:4,74 --> 0:30:5,28
Radim: It is.

607
0:30:5,28 --> 0:30:9,1
That's, and you know, this is,
I don't want to sound as a promotion.

608
0:30:9,44 --> 0:30:11,14
I'm building like it's called labs.

609
0:30:11,14 --> 0:30:13,84
So, you know, we both have labs
of some sort.

610
0:30:14,16 --> 0:30:16,44
Nikolay: For experimentation, it's
right.

611
0:30:16,44 --> 0:30:16,94
Yeah.

612
0:30:17,7 --> 0:30:23,26
Radim: Because for me, and you
mentioned it before, developers

613
0:30:23,46 --> 0:30:27,38
want same environment as production
and they don't understand

614
0:30:27,44 --> 0:30:28,38
why it's different.

615
0:30:30,06 --> 0:30:34,94
And how many of us, I think 3 of
us, we are lucky because we

616
0:30:34,94 --> 0:30:39,52
had access to a really busy production
environments and we know

617
0:30:39,52 --> 0:30:40,44
how it looks like.

618
0:30:40,44 --> 0:30:44,34
But if you take an average developer,
during their career, they

619
0:30:44,34 --> 0:30:49,28
will never actually touch environment
that has more than 10 TPS

620
0:30:49,6 --> 0:30:53,3
and maybe gigabyte of data, they
don't simply see it.

621
0:30:53,32 --> 0:30:57,68
This is why, you know, if you trace
the BoringSQL block, everything

622
0:30:57,92 --> 0:31:2,56
comes down to, this is fine on
your database, but it will eat

623
0:31:2,56 --> 0:31:7,12
your life on production because
it won't work.

624
0:31:7,12 --> 0:31:9,1
Nikolay: Yes, I compare it to bicycle.

625
0:31:10,76 --> 0:31:15,88
It's hard to be DBA because you
need to learn, but where to learn?

626
0:31:15,88 --> 0:31:19,26
If you don't have access to terabyte
scale and heavy workloads,

627
0:31:19,9 --> 0:31:25,96
if the only practice you have is
production, it's basically learning

628
0:31:26,0 --> 0:31:28,12
bicycle, how to ride bicycle on
highway.

629
0:31:30,48 --> 0:31:33,18
Some people survived and they are
called DBA now.

630
0:31:33,82 --> 0:31:36,5
Others they say the database is
slow, it's dangerous, don't touch

631
0:31:36,5 --> 0:31:36,76
it.

632
0:31:36,76 --> 0:31:40,36
If you have an environment where
you can safely make mistakes,

633
0:31:40,92 --> 0:31:44,44
I saw magic when people have access
to terabyte scale clones

634
0:31:45,24 --> 0:31:50,42
and they start to do really crazy
stuff and realize it's wrong,

635
0:31:50,44 --> 0:31:51,68
but nobody affected.

636
0:31:52,4 --> 0:31:57,08
And this is how learning is actually
works, you know, it works.

637
0:31:57,8 --> 0:32:1,96
So yeah, you're basically 100%
resonating in my mind what you

638
0:32:1,96 --> 0:32:2,46
do.

639
0:32:2,72 --> 0:32:6,16
Radim: And then take whatever you
do, because I'm actually following

640
0:32:6,16 --> 0:32:10,66
your post on LinkedIn about self-driving
Postgres.

641
0:32:11,2 --> 0:32:15,08
Now you take, people won't write
SQL queries.

642
0:32:15,66 --> 0:32:19,7
They will not, let's forget that
it's a regression testing framework,

643
0:32:20,0 --> 0:32:24,02
they will not have this experience
and they will never hit issue

644
0:32:24,44 --> 0:32:27,38
that Postgres would be badly tuned.

645
0:32:27,52 --> 0:32:28,02
Yeah.

646
0:32:28,78 --> 0:32:33,48
That suddenly increases the gap
and sure, there will be AI which

647
0:32:33,48 --> 0:32:37,96
will help but you still need a
manual or human operator at 1

648
0:32:37,96 --> 0:32:41,78
point or somebody who will be able
to diagnose right when wrong.

649
0:32:41,98 --> 0:32:45,04
Nikolay: Because real testing shifted
to production that's why.

650
0:32:45,72 --> 0:32:47,86
And we just need to shift it left
again.

651
0:32:48,54 --> 0:32:48,82
Yeah.

652
0:32:48,82 --> 0:32:49,32
Radim: Exactly.

653
0:32:49,34 --> 0:32:54,14
So I think it's all about clarity
and giving people, sure, there

654
0:32:54,14 --> 0:32:57,66
will be, I think we will never
have 10 times more DBAs.

655
0:32:58,04 --> 0:33:0,52
We will have probably 10 times
less DBAs.

656
0:33:1,08 --> 0:33:4,36
But we need to make sure how the
new people will actually learn

657
0:33:4,36 --> 0:33:8,64
those skills and how they will
learn them predictably without

658
0:33:8,68 --> 0:33:12,94
that fear, without that, you know,
being scared and who knows

659
0:33:12,94 --> 0:33:14,76
what will happen in 5 to 10 years.

660
0:33:14,76 --> 0:33:15,69
Nikolay: Yeah, I agree.

661
0:33:15,69 --> 0:33:16,62
I agree.

662
0:33:18,4 --> 0:33:21,6
And also it's very resonant to
what happened to code and containers.

663
0:33:21,82 --> 0:33:24,86
Containers is the way to have reproducible
tests.

664
0:33:25,24 --> 0:33:26,2
It's standard, right?

665
0:33:26,2 --> 0:33:27,94
We just need the same for data.

666
0:33:28,78 --> 0:33:30,68
And copy-on-write is the key.

667
0:33:31,62 --> 0:33:33,94
But also methodologies should be
adjusted, buffers.

668
0:33:34,28 --> 0:33:35,82
So everything is so.

669
0:33:35,82 --> 0:33:38,04
And I agree, DBAs is like sysadmins.

670
0:33:38,56 --> 0:33:39,52
It's in the past.

671
0:33:39,86 --> 0:33:44,32
People need to learn Postgres who
build stuff and know how to

672
0:33:44,7 --> 0:33:47,78
operate bigger databases coming
from various angles.

673
0:33:47,78 --> 0:33:51,3
I have front-end engineers who
already start understanding things.

674
0:33:51,82 --> 0:33:53,8
Or Supabase, Supabase is a great
example.

675
0:33:53,86 --> 0:33:57,92
He attracted a lot of front-end
engineers to learn SQL and so

676
0:33:57,92 --> 0:33:58,52
on, right?

677
0:33:58,52 --> 0:34:0,22
That's great, I think.

678
0:34:0,56 --> 0:34:2,84
Radim: And that's the boring part,
you know, I have to make the

679
0:34:2,84 --> 0:34:5,74
joke because, you know, for me
literally the direction where

680
0:34:5,74 --> 0:34:9,3
I'm going is those boring things
are with us for 40 years, they

681
0:34:9,3 --> 0:34:11,74
will be with us for quite some
time.

682
0:34:12,7 --> 0:34:16,5
And they just work, those skills
are reusable, no matter if you've

683
0:34:16,5 --> 0:34:18,28
read them in LLM or not.

684
0:34:18,34 --> 0:34:19,9
Somebody needs to understand them.

685
0:34:20,46 --> 0:34:21,22
Nikolay: Yeah, yeah.

686
0:34:21,22 --> 0:34:23,9
So can you explain what your tool
does?

687
0:34:24,52 --> 0:34:29,28
So assuming we understood this,
that we need to focus on buffers

688
0:34:29,34 --> 0:34:33,34
during optimization and establish
baselines focusing on buffers

689
0:34:33,34 --> 0:34:34,08
and cost.

690
0:34:34,84 --> 0:34:39,66
And we managed to achieve the same
data using this feature in

691
0:34:39,66 --> 0:34:40,58
Postgres 18.

692
0:34:40,84 --> 0:34:44,32
What does your tool do and who
should use it?

693
0:34:44,68 --> 0:34:49,2
Radim: Okay, So the main goal of
the tool is to find all your

694
0:34:49,44 --> 0:34:56,6
SQL queries and help you in some
way to write a plan because

695
0:34:56,6 --> 0:35:0,16
you know queries need parameters,
it needs to have a way, So

696
0:35:0,16 --> 0:35:5,04
you need to write a fixtures or
provide your dump database, anonymized

697
0:35:5,06 --> 0:35:5,56
database.

698
0:35:5,9 --> 0:35:6,88
That's your choice.

699
0:35:6,88 --> 0:35:8,74
I'm not actually forcing you in
anything.

700
0:35:8,74 --> 0:35:10,74
You have to get it to predictable
state.

701
0:35:11,2 --> 0:35:12,24
That's your snapshot.

702
0:35:12,8 --> 0:35:16,62
Then you run it on your queries
and based on your plan, each

703
0:35:16,62 --> 0:35:18,66
query can have 1 to n plans.

704
0:35:18,9 --> 0:35:20,28
That means what's the variation.

705
0:35:20,38 --> 0:35:25,16
So you can have LLM to generate
you plans, you know, using random

706
0:35:25,16 --> 0:35:25,44
values.

707
0:35:25,44 --> 0:35:28,86
You can have a fuzzy generated
plans and that means every single

708
0:35:28,86 --> 0:35:31,02
plan will execute that query once.

709
0:35:31,5 --> 0:35:35,76
What the tool does, it captures
the expected, the original tool

710
0:35:35,76 --> 0:35:38,48
did it also, it captures the data.

711
0:35:38,48 --> 0:35:42,52
So what you have is right now it's
a JSON with all the data

712
0:35:42,52 --> 0:35:43,36
that it produced.

713
0:35:43,5 --> 0:35:46,4
This is your baseline And that's
the contract for correctness

714
0:35:46,4 --> 0:35:47,26
of the queries.

715
0:35:48,04 --> 0:35:49,9
Then you have the functionality
for baseline.

716
0:35:49,94 --> 0:35:55,22
So right now the version 1.0 does
EXPLAIN only, but version 2.0

717
0:35:55,38 --> 0:35:58,14
will have a full EXPLAIN ANALYZE
because that's what I need for

718
0:35:58,14 --> 0:35:58,64
buffers.

719
0:35:59,44 --> 0:36:1,16
And it records that data.

720
0:36:1,7 --> 0:36:1,92
Nikolay: Yeah.

721
0:36:1,92 --> 0:36:6,66
What about other settings, modern
settings like memory and serialize?

722
0:36:8,74 --> 0:36:12,44
Radim: That is interesting because
again, this is the part of

723
0:36:12,44 --> 0:36:16,16
the 2.0 roadmap because snapshot
can't be consistent.

724
0:36:16,46 --> 0:36:20,18
You have to have a data, you have
to have a now even configuration.

725
0:36:20,42 --> 0:36:24,3
So I didn't, I'm not saving all
the configuration, all the metadata

726
0:36:24,52 --> 0:36:27,82
about where it's running and how
it's running, but I'm actually

727
0:36:27,82 --> 0:36:29,44
saving this as part of the snapshot.

728
0:36:29,44 --> 0:36:33,74
So if something changes, like if
you change a configuration,

729
0:36:35,22 --> 0:36:39,12
what's the cost of sequential scan,
it will warn you.

730
0:36:39,12 --> 0:36:43,2
You change something and I will
probably give you wrong results.

731
0:36:43,2 --> 0:36:45,26
So you still have to interpret
the data.

732
0:36:46,42 --> 0:36:50,22
At 1 point you arrive to a scenario
where you have a good Commit

733
0:36:50,22 --> 0:36:53,52
and you can distribute the snapshots
for testing.

734
0:36:53,52 --> 0:36:57,72
So ideally, you can have a couple
of versions of the snapshots.

735
0:36:57,72 --> 0:37:0,7
You can have a small snapshot,
which will have 100 megabytes,

736
0:37:0,7 --> 0:37:1,48
200 megabytes.

737
0:37:1,48 --> 0:37:5,78
You can have a mid-size 5 gigabyte
snapshot, which you can use.

738
0:37:6,28 --> 0:37:9,78
And then you can have a large snapshot,
50 gigabytes or something.

739
0:37:9,96 --> 0:37:14,8
And then every time you run test,
you will get your report, what

740
0:37:14,8 --> 0:37:15,88
actually has changed.

741
0:37:16,12 --> 0:37:18,54
And that change is based on the
cost of the buffers.

742
0:37:19,2 --> 0:37:20,14
Nikolay: Yeah, that's great.

743
0:37:20,14 --> 0:37:23,94
And you capture also version, I
think, right?

744
0:37:23,94 --> 0:37:24,64
And settings.

745
0:37:25,08 --> 0:37:25,58
Yes.

746
0:37:25,84 --> 0:37:29,26
And some like statistics and structure
of table, which indexes

747
0:37:29,26 --> 0:37:30,66
we have and so on, right?

748
0:37:30,9 --> 0:37:34,94
We're basically in the same, like,
met at some point, it resonates

749
0:37:34,94 --> 0:37:38,08
a lot, but you come from development
more, I come from production

750
0:37:38,08 --> 0:37:38,58
more.

751
0:37:38,76 --> 0:37:42,54
And in production, what also matters often, including buffers,

752
0:37:42,74 --> 0:37:46,32
for example, if you think about index-only scans, heap fetches

753
0:37:46,32 --> 0:37:47,46
matters a lot, right?

754
0:37:49,54 --> 0:37:53,22
And this is the state of production which can be bad or good.

755
0:37:53,8 --> 0:37:58,38
And capturing from production the state, some statistics might

756
0:37:58,38 --> 0:37:59,68
make sense as well.

757
0:38:0,04 --> 0:38:1,96
Do you see somehow it could play or not?

758
0:38:1,96 --> 0:38:6,24
For example, when last time this table was vacuumed, how much

759
0:38:6,76 --> 0:38:11,12
dead tuples it has or how much bloat accumulated, something like

760
0:38:11,12 --> 0:38:14,94
these things, you know, or it's outside of your scope?

761
0:38:15,24 --> 0:38:18,0
Radim: I spend so much time thinking about this topic.

762
0:38:18,08 --> 0:38:22,12
And if I would ask, if you would ask LLM, that would say, yes,

763
0:38:22,12 --> 0:38:23,48
that's your natural direction.

764
0:38:23,48 --> 0:38:28,02
But I actually proactively said no to this because this is a

765
0:38:28,02 --> 0:38:28,98
different domain.

766
0:38:29,44 --> 0:38:32,46
You know, you have to guarantee what's broken on the development

767
0:38:32,52 --> 0:38:33,02
side.

768
0:38:33,16 --> 0:38:36,08
You have to make sure that contract, which is also a performance

769
0:38:36,1 --> 0:38:41,0
or buffers, is the same or better and you track it, but you can't

770
0:38:41,0 --> 0:38:42,24
replace this in production.

771
0:38:42,24 --> 0:38:45,78
Because in production you can have a latency, you can have a

772
0:38:45,78 --> 0:38:48,3
replica, which run in different availability zones, you can have

773
0:38:48,3 --> 0:38:51,88
a network issues, You can have, you know, so many issues that

774
0:38:51,88 --> 0:38:52,98
can happen in production.

775
0:38:53,36 --> 0:38:57,6
And there are tools that I effectively, I can't compete with.

776
0:38:57,66 --> 0:39:1,26
And I think that wouldn't be, that wouldn't be my goal.

777
0:39:1,26 --> 0:39:4,24
And because my angle is education effectively.

778
0:39:5,34 --> 0:39:8,54
For me, it stops with CI.

779
0:39:9,18 --> 0:39:12,88
Nikolay: Yeah, it again resonates 100%, because what you do,

780
0:39:12,92 --> 0:39:16,88
you say, let's split the system into pieces and study each piece

781
0:39:16,88 --> 0:39:17,38
separately.

782
0:39:18,42 --> 0:39:23,24
And when we understand each piece, like, behavior fully, then

783
0:39:23,24 --> 0:39:25,14
we can go and increase complexity.

784
0:39:25,68 --> 0:39:30,08
And this, that's why you shouldn't use pgbench to study plans,

785
0:39:30,96 --> 0:39:31,38
right?

786
0:39:31,38 --> 0:39:32,44
Because pgbench is

787
0:39:32,44 --> 0:39:33,08
Radim: very cool

788
0:39:33,08 --> 0:39:34,28
Nikolay: to test the whole system.

789
0:39:34,28 --> 0:39:37,62
And also that's why you shouldn't bring too much from production.

790
0:39:38,1 --> 0:39:43,36
Study your system first, how it works, and then let's go to production,

791
0:39:43,36 --> 0:39:46,9
understanding our system and knowing what it's capable of already.

792
0:39:47,08 --> 0:39:48,26
Then we bring complexity.

793
0:39:50,66 --> 0:39:54,66
This is like one of the oldest scientific methods, René Descartes,

794
0:39:54,66 --> 0:39:55,16
right?

795
0:39:56,14 --> 0:40:0,0
Let's split the pieces, study each piece, and then composition

796
0:40:1,4 --> 0:40:2,54
to have whole understanding.

797
0:40:2,9 --> 0:40:6,72
Without this split, it's super hard and you always have too many

798
0:40:6,72 --> 0:40:7,9
moving pieces, right?

799
0:40:8,16 --> 0:40:12,24
So I like this, but still I'm from production, so I care about

800
0:40:12,26 --> 0:40:14,44
the state of production, you know, it's different.

801
0:40:15,06 --> 0:40:15,56
Yeah.

802
0:40:16,26 --> 0:40:16,76
Cool.

803
0:40:16,82 --> 0:40:18,52
Michael: I think we're talking about different things.

804
0:40:18,52 --> 0:40:19,78
They were, I really liked that.

805
0:40:19,78 --> 0:40:21,36
This is about regression testing.

806
0:40:21,36 --> 0:40:25,12
This is about not breaking things that like ourselves through

807
0:40:25,12 --> 0:40:25,62
development.

808
0:40:25,76 --> 0:40:28,86
This is not breaking correctness, whether that's a human changing

809
0:40:28,86 --> 0:40:32,06
a query or an LLM chatbot changing a query.

810
0:40:32,32 --> 0:40:34,9
So correctness is like primary importance, that makes sense.

811
0:40:34,9 --> 0:40:40,4
And ideally not affecting performance, but we can't measure production

812
0:40:40,4 --> 0:40:42,54
performance in test, in development.

813
0:40:42,8 --> 0:40:45,06
So all we can do is get a proxy for it.

814
0:40:45,06 --> 0:40:50,24
I don't think we can anticipate every single potential performance

815
0:40:50,38 --> 0:40:52,7
issue in development that could hit production.

816
0:40:52,72 --> 0:40:56,06
Nikolay: We've produced really difficult problems.

817
0:40:56,8 --> 0:41:0,38
Yeah, with point-in-time recovery, sometimes you can reproduce

818
0:41:0,38 --> 0:41:2,38
really hard problems at macro level.

819
0:41:3,38 --> 0:41:5,96
Michael: Reproduction is different from anticipation though.

820
0:41:5,98 --> 0:41:9,14
Anticipating, so to get it into your regression test you would

821
0:41:9,14 --> 0:41:13,08
have to anticipate every possible issue and I think personally

822
0:41:13,08 --> 0:41:16,56
I really like the direction of let's do 1 thing at a time, let's

823
0:41:16,56 --> 0:41:18,22
tackle the common things, let's

824
0:41:18,42 --> 0:41:18,93
Nikolay: look at the costing.

825
0:41:18,93 --> 0:41:19,1233
I'm not

826
0:41:19,1233 --> 0:41:19,64
Michael: arguing with the

827
0:41:19,64 --> 0:41:22,58
Nikolay: whole concept, I'm just saying that also is possible,

828
0:41:22,58 --> 0:41:26,82
it's just much, much heavier, much more expensive, time consuming.

829
0:41:27,44 --> 0:41:30,98
In development we need to move faster to write a lot and Check

830
0:41:31,08 --> 0:41:33,62
dozens of ideas at the same time like in parallel.

831
0:41:33,62 --> 0:41:38,1
This is what we need right and this is perfect approach So the

832
0:41:38,1 --> 0:41:41,06
Michael: last the last thing I wanted to say is I've personally

833
0:41:41,26 --> 0:41:45,06
I personally think that This hasn't been a topic that's been

834
0:41:45,06 --> 0:41:47,22
particularly of interest to people for many years.

835
0:41:47,22 --> 0:41:49,84
I've expected it to be like I thought it should be.

836
0:41:49,84 --> 0:41:53,14
I've worked at a company previously that made tools in this area

837
0:41:53,14 --> 0:41:57,04
that weren't very successful and I was actually pleasantly surprised

838
0:41:57,18 --> 0:42:1,02
by how much positivity and interest there was in on the Hacker

839
0:42:1,02 --> 0:42:4,34
News discussion and in your in your revamp of this tool.

840
0:42:4,34 --> 0:42:8,0
So for me, it's actually feels like people are ready for this

841
0:42:8,0 --> 0:42:9,14
topic for some reason.

842
0:42:9,14 --> 0:42:11,34
I don't know what changed or what.

843
0:42:11,68 --> 0:42:13,94
Nikolay: What is writing there?

844
0:42:13,94 --> 0:42:14,68
Who knows?

845
0:42:15,04 --> 0:42:17,46
Michael: Do you think that's what it is, Radim, from your perspective?

846
0:42:18,6 --> 0:42:22,32
Radim: I think this goes to, there's not 1 answer.

847
0:42:23,36 --> 0:42:30,02
First is going back 1 topic, how many percent of incidents you

848
0:42:30,02 --> 0:42:31,42
deal are really production-based?

849
0:42:31,98 --> 0:42:39,36
Like no matter if it's human or LLM, you know, I'll be just guessing.

850
0:42:40,08 --> 0:42:45,8
90% of all incidents are change management that didn't go well

851
0:42:46,4 --> 0:42:47,44
and something happened.

852
0:42:47,44 --> 0:42:50,78
So this is why the regression testing
actually when you Microsoft

853
0:42:50,8 --> 0:42:53,8
you can't measure but if you take
a snapshot something which

854
0:42:53,8 --> 0:42:58,12
is now working you should actually
do something that is measurable.

855
0:42:58,18 --> 0:43:1,3
You measure if you're improving
on performance.

856
0:43:1,36 --> 0:43:5,94
So it's you know sure contract
is important but actually if you

857
0:43:5,94 --> 0:43:12,52
start with a 0.0, you can technically
say we on average go up

858
0:43:12,52 --> 0:43:14,16
and down in terms of performance.

859
0:43:14,48 --> 0:43:16,42
This is very easy to measure.

860
0:43:16,82 --> 0:43:19,6
And you know this 1 for example,
you know how many times you

861
0:43:19,6 --> 0:43:22,96
double guess when you upgrade from
Postgres 17 to 18, what's

862
0:43:22,96 --> 0:43:23,6
the impact?

863
0:43:24,02 --> 0:43:28,46
You can suddenly put a number and
if you somehow put all your

864
0:43:28,46 --> 0:43:33,74
queries and do that number, you
can put that number into a bucket.

865
0:43:34,4 --> 0:43:39,86
And the second 1 is no matter what
you do, I was asking myself,

866
0:43:39,86 --> 0:43:43,4
you know, why nobody thought about
this 1 before?

867
0:43:43,5 --> 0:43:46,8
I actually don't understand why
Nobody did this because it's

868
0:43:46,8 --> 0:43:47,3
predictable.

869
0:43:48,34 --> 0:43:51,34
The metrics were there, but nobody
did it.

870
0:43:51,34 --> 0:43:56,26
And from my experience and the
frustration I had was all the

871
0:43:56,26 --> 0:44:0,68
incidents were very easily avoidable
because the metrics were

872
0:44:0,68 --> 0:44:4,26
not in the 10% margin, they were
in 1000%.

873
0:44:5,02 --> 0:44:8,36
Every single incident started with
1000% difference.

874
0:44:8,72 --> 0:44:11,62
Nikolay: I have also 1 question
about this tooling.

875
0:44:12,56 --> 0:44:15,86
So how do you capture queries to
test?

876
0:44:16,52 --> 0:44:21,14
Because for example, if we have
ORM, it produces various queries.

877
0:44:21,54 --> 0:44:24,4
We can see pg_stat_statements, normalized
queries in production,

878
0:44:24,72 --> 0:44:28,5
but for each query we might have
multiple plans depending on

879
0:44:28,5 --> 0:44:29,0
parameters.

880
0:44:29,72 --> 0:44:32,22
So how to find those queries?

881
0:44:32,54 --> 0:44:33,74
I have my own methodology.

882
0:44:34,28 --> 0:44:36,5
It's actually described in some
articles.

883
0:44:36,5 --> 0:44:40,14
We could collaborate maybe a little
bit on it, but what's your

884
0:44:40,14 --> 0:44:40,64
approach?

885
0:44:41,12 --> 0:44:42,09
Radim: This is the difficult part.

886
0:44:42,09 --> 0:44:43,38
I'm not going to lie.

887
0:44:43,66 --> 0:44:49,64
I come from Go background, and
if you take, ORMs are not that

888
0:44:49,64 --> 0:44:51,62
popular in Go, I would say.

889
0:44:52,08 --> 0:44:55,74
For a framework like SQLX and things
like this, 1 day they encourage

890
0:44:55,74 --> 0:44:59,2
you to write SQL queries by hand.

891
0:44:59,48 --> 0:45:1,1
And this is actually where it started.

892
0:45:1,1 --> 0:45:5,56
So, effectively, my first commits
were about supporting multiple

893
0:45:5,56 --> 0:45:7,08
queries in 1 file.

894
0:45:7,26 --> 0:45:8,94
So I have a different base.

895
0:45:8,94 --> 0:45:12,74
There are even frameworks like
SQL C, I think that generates

896
0:45:12,9 --> 0:45:15,34
code based on the queries or vice
versa.

897
0:45:16,26 --> 0:45:19,44
But that would keep the target
market very slow.

898
0:45:19,46 --> 0:45:23,14
So the obviously next step, and
that's the support of the different

899
0:45:23,14 --> 0:45:23,64
frameworks.

900
0:45:23,92 --> 0:45:26,78
So I think in my article, I did
a SQL alchemy.

901
0:45:27,88 --> 0:45:30,76
That was 1 that actually allows
you to save the queries.

902
0:45:30,94 --> 0:45:34,12
And this, if you remember, we talked
about the plans.

903
0:45:34,3 --> 0:45:38,08
And plans are that variation of
arguments.

904
0:45:38,36 --> 0:45:38,72
Nikolay: Right.

905
0:45:38,72 --> 0:45:42,16
Radim: So you see, you have an
artificially generated snapshot,

906
0:45:42,16 --> 0:45:44,54
which is fixed, and then you need
to generate the plans.

907
0:45:44,54 --> 0:45:48,92
And plans variation, if you write
1 plan, your query testing

908
0:45:48,94 --> 0:45:50,66
is not going to be this 1.

909
0:45:50,66 --> 0:45:54,38
So you need to generate, and this
is actually where LLM can help

910
0:45:54,38 --> 0:45:54,88
you.

911
0:45:55,16 --> 0:45:58,28
Because suddenly you have a tool
which can create the variation,

912
0:45:58,42 --> 0:46:0,06
create even hallucinate data.

913
0:46:0,06 --> 0:46:4,34
And you will be testing queries
with a, I would actually be quite

914
0:46:4,34 --> 0:46:7,58
happy to have hallucinated parameters
because it will show you

915
0:46:7,58 --> 0:46:9,64
whatever something is changing
or not.

916
0:46:9,72 --> 0:46:11,92
Nikolay: It's a proactive approach
actually, because my approach

917
0:46:11,92 --> 0:46:14,58
is to take everything from production
because we deal with really

918
0:46:14,58 --> 0:46:20,04
large systems and we need to understand
when we, for example,

919
0:46:20,06 --> 0:46:23,16
perform major upgrade, there are
no plan flips, so planner or

920
0:46:23,16 --> 0:46:24,5
plan flips are positive.

921
0:46:25,32 --> 0:46:28,78
But what you say, like, let's use
LLM to generate parameters

922
0:46:28,82 --> 0:46:31,22
which probably will happen next
month, right?

923
0:46:31,86 --> 0:46:33,9
Like some edge cases we haven't
met before.

924
0:46:33,9 --> 0:46:35,64
I like this a lot as well.

925
0:46:35,86 --> 0:46:36,36
Yeah.

926
0:46:36,6 --> 0:46:36,82
Radim: Yeah.

927
0:46:36,82 --> 0:46:39,12
So you have to use what you are
afraid of.

928
0:46:39,12 --> 0:46:43,94
So if we are afraid of randomized
queries, then why not use it?

929
0:46:43,94 --> 0:46:47,46
And effectively said, yeah, I need
a variation of thousand plans.

930
0:46:47,64 --> 0:46:50,7
Nikolay: Yes, again, like the problem
is like, again, I have

931
0:46:50,74 --> 0:46:54,4
slightly different opinion here
because I'm coming from production

932
0:46:54,4 --> 0:46:54,9
experience.

933
0:46:55,2 --> 0:46:59,76
I would say we can generate like
thousands and thousands of various

934
0:46:59,86 --> 0:47:3,66
cases, maybe 1,000,000 queries to
test, but it will increase the

935
0:47:3,66 --> 0:47:7,66
longevity and the price we pay
for CI and testing, right?

936
0:47:7,66 --> 0:47:9,56
So we need probably to prioritize
some queries.

937
0:47:9,56 --> 0:47:10,28
How to prioritize?

938
0:47:10,28 --> 0:47:11,5
We need to study production.

939
0:47:12,84 --> 0:47:13,32
No?

940
0:47:13,32 --> 0:47:13,78
Radim: Exactly.

941
0:47:13,78 --> 0:47:14,28
Okay.

942
0:47:15,06 --> 0:47:18,4
You need to understand your business
requirements, you need to

943
0:47:18,4 --> 0:47:21,74
understand your outliers, you need
to understand your smallest

944
0:47:21,74 --> 0:47:24,44
customers, you need to understand
your biggest customers, you

945
0:47:24,44 --> 0:47:26,6
need to understand the patterns
which are happening.

946
0:47:26,64 --> 0:47:28,82
Nikolay: That's also an interesting
comment, because I was thinking

947
0:47:28,82 --> 0:47:31,48
about like this tooling, Oh, by
the way, parameters right now

948
0:47:31,48 --> 0:47:34,82
is just an idea, it's not implemented
yet, a collection of...

949
0:47:34,9 --> 0:47:36,36
Radim: No, that's already...

950
0:47:36,96 --> 0:47:38,26
The queries are already parameterized.

951
0:47:38,86 --> 0:47:39,84
Yeah, I understand the

952
0:47:39,84 --> 0:47:44,74
Nikolay: picture, but if I have already a large 10TB database

953
0:47:45,04 --> 0:47:48,42
with a complex workload and I want to start using your tool,

954
0:47:48,9 --> 0:47:53,4
will it help me to create a set of queries, example queries,

955
0:47:53,4 --> 0:47:54,4
which will be tested?

956
0:47:56,06 --> 0:47:58,68
Radim: Queries we have to extract from somewhere.

957
0:47:59,06 --> 0:48:3,08
So no, I would say answer is no, because right now I assume you

958
0:48:3,08 --> 0:48:6,34
have a queries or way how to capture the queries, either by having

959
0:48:6,34 --> 0:48:12,8
them in the SQL files or by being able to have a CI task that

960
0:48:12,8 --> 0:48:16,18
will generate those SQL files from your ORM.

961
0:48:16,24 --> 0:48:20,08
Nikolay: Yeah, I would like to have an additional call with you

962
0:48:20,08 --> 0:48:23,04
to discuss because I have an approach maybe we could connect the

963
0:48:23,04 --> 0:48:27,78
dots here and see if it can land because this methodology probably

964
0:48:27,78 --> 0:48:33,06
works with your tool naturally, you know, how to inject this

965
0:48:33,06 --> 0:48:34,94
approach to an existing big project.

966
0:48:35,34 --> 0:48:38,68
We deal mostly with projects already starting, hitting walls,

967
0:48:38,68 --> 0:48:41,02
they come to us for help.

968
0:48:41,28 --> 0:48:45,3
So I understand completely how to start from scratch, but when

969
0:48:45,3 --> 0:48:49,74
you have something already, there it can be difficult.

970
0:48:50,54 --> 0:48:54,18
Another thing I wanted to point out, what I hear, you say smaller,

971
0:48:54,52 --> 0:48:55,74
medium-sized, bigger.

972
0:48:56,04 --> 0:48:58,66
Sometimes we have only 1 database and that's it.

973
0:48:58,66 --> 0:49:1,58
Some systems like SaaS, which is installed only once.

974
0:49:2,5 --> 0:49:6,18
But sometimes we develop software which has many installations,

975
0:49:6,34 --> 0:49:7,66
small, big ones.

976
0:49:7,9 --> 0:49:8,86
So it's interesting.

977
0:49:8,86 --> 0:49:13,76
I think the approach when you define some reference databases,

978
0:49:14,38 --> 0:49:18,28
this works better when we have multiple installations of various

979
0:49:18,28 --> 0:49:18,62
sizes?

980
0:49:18,62 --> 0:49:19,9
Is it what you think?

981
0:49:19,9 --> 0:49:23,38
Radim: Yes, as I said, there's no universal answer how to generate.

982
0:49:23,72 --> 0:49:27,1
And many people actually wrote me about fixtures being completely

983
0:49:27,18 --> 0:49:27,68
wrong.

984
0:49:28,5 --> 0:49:32,72
But what I'm trying to get past is I don't care about the features.

985
0:49:33,16 --> 0:49:35,94
I care about the way how to generate a snapshot.

986
0:49:36,74 --> 0:49:41,66
And if you have the same cardinality and same data, you can then

987
0:49:41,66 --> 0:49:42,48
measure correctness.

988
0:49:42,72 --> 0:49:47,72
So obviously the investment in a tool like RegreSQL is actually,

989
0:49:47,84 --> 0:49:48,68
it's a big 1.

990
0:49:48,68 --> 0:49:51,96
You have to start using it and you will get dividends of months

991
0:49:51,96 --> 0:49:52,44
on.

992
0:49:52,44 --> 0:49:55,56
You might get some dividends as you go, but actually the true

993
0:49:55,56 --> 0:49:56,98
value will only go.

994
0:49:56,98 --> 0:50:0,14
So the developer experience and then whole life cycle, that's

995
0:50:0,14 --> 0:50:3,12
maybe a version 3.0, because you know, there needs to be much

996
0:50:3,12 --> 0:50:3,84
more support.

997
0:50:3,84 --> 0:50:8,4
So the tool itself is not important as much, it's the whole experience

998
0:50:8,72 --> 0:50:11,56
in the process because methodology, exactly.

999
0:50:11,72 --> 0:50:14,06
Nikolay: So what I'm actually interested in, like let me share

1000
0:50:14,06 --> 0:50:17,68
like to be honest with you and what I have in my mind listening

1001
0:50:17,68 --> 0:50:18,36
to you.

1002
0:50:18,68 --> 0:50:21,24
For example, GitLab, they use our approach.

1003
0:50:21,7 --> 0:50:24,86
Many engineers say it's great and we have case study publishers.

1004
0:50:25,52 --> 0:50:26,5
Everything is great.

1005
0:50:26,68 --> 0:50:29,86
But listening to you, I realize actually we always cared about

1006
0:50:29,86 --> 0:50:33,16
the biggest GitLab.com installation which is huge, multiple,

1007
0:50:33,18 --> 0:50:34,06
it's huge.

1008
0:50:34,9 --> 0:50:36,72
We focused on solving that problem.

1009
0:50:36,9 --> 0:50:42,1
We solved it, not fully, but solved how to test queries there.

1010
0:50:42,5 --> 0:50:43,94
It can be proved always.

1011
0:50:44,1 --> 0:50:46,88
But now I'm thinking, actually, GitLab has hundreds of thousands

1012
0:50:46,88 --> 0:50:49,82
of other installations and they have reference.

1013
0:50:52,2 --> 0:50:54,94
We actually should care about that as well.

1014
0:50:55,32 --> 0:50:59,24
So it's interesting because we could generate different sizes

1015
0:51:0,08 --> 0:51:4,7
of database and cover it by testing them as well.

1016
0:51:5,22 --> 0:51:9,36
So it's interesting because I was always thinking about the biggest

1017
0:51:9,36 --> 0:51:12,66
1, you know, because usually problems usually there.

1018
0:51:13,08 --> 0:51:13,58
Right.

1019
0:51:14,34 --> 0:51:14,76
Radim: Exactly.

1020
0:51:14,76 --> 0:51:16,24
You hear about the biggest ones.

1021
0:51:16,24 --> 0:51:18,56
This is where, you know, the production breaks, the incident

1022
0:51:18,56 --> 0:51:20,72
starts, the query, which was fine.

1023
0:51:21,14 --> 0:51:24,28
It sounds similar, but it's a very different problem.

1024
0:51:24,56 --> 0:51:27,98
But you need to find a some sort of boundary sometimes you can't

1025
0:51:27,98 --> 0:51:30,7
solve all the issues and maybe you need to find a different solution

1026
0:51:30,78 --> 0:51:32,78
Nikolay: right right right yeah so it's

1027
0:51:32,78 --> 0:51:37,7
Michael: just out of interest Nik what would you expect to break

1028
0:51:38,04 --> 0:51:39,56
on a smaller instance?

1029
0:51:39,64 --> 0:51:42,04
Is it that it's just like a different shape, for example, on

1030
0:51:42,04 --> 0:51:47,38
GitLab.com's main instance, maybe there's tons of 1 type of object

1031
0:51:47,4 --> 0:51:50,46
and in a different customer's installation there's a different

1032
0:51:50,46 --> 0:51:52,96
distribution and therefore plans are different?

1033
0:51:52,96 --> 0:51:53,8
Or is it like...

1034
0:51:53,8 --> 0:51:54,86
Nikolay: I can tell you.

1035
0:51:55,02 --> 0:51:58,74
So what we think about, like, we have various kinds of testing,

1036
0:51:58,74 --> 0:52:2,56
like, red data testing, It's like ultimate final testing with

1037
0:52:2,56 --> 0:52:6,08
basically production clones, which only a few people can have

1038
0:52:6,08 --> 0:52:6,58
access.

1039
0:52:6,78 --> 0:52:11,12
But also, you cannot download this database due to size and also

1040
0:52:11,12 --> 0:52:13,04
restrictions, PII and so on.

1041
0:52:13,14 --> 0:52:16,56
But what if you need to care about multiple installations in

1042
0:52:16,56 --> 0:52:22,28
this case, the direction DBLab is moving in is having full-fledged

1043
0:52:22,9 --> 0:52:28,48
database branching Git-like, but also cloning between multiple

1044
0:52:29,12 --> 0:52:29,62
setups.

1045
0:52:30,36 --> 0:52:31,82
And We already run on MacBook.

1046
0:52:32,08 --> 0:52:35,94
So I'm thinking if you care about smaller databases as well,

1047
0:52:35,94 --> 0:52:39,28
developers could basically download reference database and iterate

1048
0:52:39,28 --> 0:52:42,38
much faster because I hate how CI is organized right now.

1049
0:52:42,7 --> 0:52:48,42
If AI is hallucinating, in production if I hit it, it's an absolute

1050
0:52:48,42 --> 0:52:48,92
nightmare.

1051
0:52:49,28 --> 0:52:53,04
But hitting even in GitLab CI or
GitHub Actions, hitting problems

1052
0:52:53,04 --> 0:52:54,38
there, it's already too late for
me.

1053
0:52:54,38 --> 0:52:56,08
I want to iterate much faster.

1054
0:52:56,1 --> 0:53:1,82
I want all the tests to be run
before push, git push, right?

1055
0:53:1,84 --> 0:53:5,74
In this case, if we have a DBLab
of a smaller size, you download

1056
0:53:5,74 --> 0:53:10,08
it and like don't hunt only for
the biggest case, medium size,

1057
0:53:10,08 --> 0:53:10,9
small size.

1058
0:53:10,96 --> 0:53:13,08
I have like a terabyte or so here.

1059
0:53:13,08 --> 0:53:17,36
I can have a good example database
on my laptop and iterate much

1060
0:53:17,36 --> 0:53:20,54
faster and before push I already
have much better code, you know.

1061
0:53:21,42 --> 0:53:24,34
Michael: Yeah, so catch more issues
earlier and only catch the...

1062
0:53:24,44 --> 0:53:26,5
Nikolay: Shift left what you can
shift left.

1063
0:53:26,84 --> 0:53:27,74
Michael: Yeah, that makes sense.

1064
0:53:27,74 --> 0:53:28,24
Nikolay: Okay.

1065
0:53:28,66 --> 0:53:30,96
So it's saying let's test earlier.

1066
0:53:31,5 --> 0:53:33,7
Ideally, developers should test.

1067
0:53:33,96 --> 0:53:38,42
Usually they don't have time, but
AI, I have no excuses to AI.

1068
0:53:38,42 --> 0:53:39,74
AI should test everything.

1069
0:53:40,24 --> 0:53:41,6
I pay for it, right?

1070
0:53:41,98 --> 0:53:43,72
Tokens, not time, right?

1071
0:53:44,54 --> 0:53:45,06
That's it.

1072
0:53:45,06 --> 0:53:50,44
I think it's a perfect time to
think about all levels, and if

1073
0:53:50,44 --> 0:53:53,72
you can shift left to your laptop
to your machine, you should

1074
0:53:53,72 --> 0:53:54,36
do it.

1075
0:53:56,04 --> 0:53:59,56
So future DBLab versions will be
able to clone data between,

1076
0:53:59,6 --> 0:54:1,5
to pass snapshots between 2 DBLabs.

1077
0:54:1,56 --> 0:54:5,28
You can have it on your machine
and just extract and then do

1078
0:54:5,28 --> 0:54:8,86
something, extract back and share
with your colleagues some tests

1079
0:54:8,86 --> 0:54:9,38
and so on.

1080
0:54:9,38 --> 0:54:12,74
Yeah, anyway, I'm still in agenda
all the time because it's actually

1081
0:54:12,74 --> 0:54:13,5
my topic.

1082
0:54:13,94 --> 0:54:15,54
I'm so glad you came actually.

1083
0:54:16,86 --> 0:54:19,6
Radim: There's actually so many
topics and I think we can go

1084
0:54:19,6 --> 0:54:24,14
for another hour because you know,
it's like the support.

1085
0:54:24,24 --> 0:54:27,0
I think last night I pushed was
the same 1.

1086
0:54:27,0 --> 0:54:31,42
You take the snapshot, you run
it against migration, either a

1087
0:54:31,42 --> 0:54:32,92
SQL file or command.

1088
0:54:32,92 --> 0:54:36,36
So you can actually trigger a third-party
system and then test

1089
0:54:36,36 --> 0:54:39,96
the same just to test the migration
how it looks like and You

1090
0:54:39,96 --> 0:54:42,8
can start there so many angles
on this topic alone.

1091
0:54:42,8 --> 0:54:43,52
Nikolay: Oh, yes

1092
0:54:44,2 --> 0:54:47,46
Radim: You know, I don't say I
have answers for everything, but

1093
0:54:47,48 --> 0:54:48,58
I just feel like I

1094
0:54:48,58 --> 0:54:48,79
Nikolay: hit a topic.

1095
0:54:48,79 --> 0:54:49,12
Radim: Let's have

1096
0:54:49,12 --> 0:54:51,92
Nikolay: part 2 about testing of
schema changes.

1097
0:54:52,36 --> 0:54:55,26
Let's have part 2 just about it,
because it's a big topic.

1098
0:54:55,68 --> 0:55:0,02
This is where, as you said, change
management, this is where

1099
0:55:0,02 --> 0:55:2,98
danger comes from usually in instance,
right?

1100
0:55:2,98 --> 0:55:5,68
So let's just discuss it separately
because I think it's worth

1101
0:55:5,68 --> 0:55:6,18
discussing.

1102
0:55:6,82 --> 0:55:9,52
Michael: Radim, is there anything on the regression testing side

1103
0:55:9,52 --> 0:55:12,94
that you wish we had asked about or that you wanted to mention

1104
0:55:13,32 --> 0:55:14,48
before we wrap up?

1105
0:55:15,3 --> 0:55:18,46
Radim: I think we have covered most of the things.

1106
0:55:19,9 --> 0:55:21,3
Nikolay: Where to start for people?

1107
0:55:21,42 --> 0:55:22,65
Radim: Where to start for people?

1108
0:55:22,65 --> 0:55:24,26
Nikolay: You know, first steps.

1109
0:55:24,96 --> 0:55:26,68
Radim: Follow, first step, follow the project.

1110
0:55:26,68 --> 0:55:28,6
I think that's the first good step.

1111
0:55:28,78 --> 0:55:33,4
I will definitely make it public and do some, you know, sessions

1112
0:55:33,74 --> 0:55:38,04
around the version 2.0 because this is where the core developer

1113
0:55:38,04 --> 0:55:40,96
experience will come and I will actually need feedback on that.

1114
0:55:40,96 --> 0:55:42,88
So I'm actually open to any feedback.

1115
0:55:42,88 --> 0:55:45,72
You can email me at any point or open a GitHub issue.

1116
0:55:45,72 --> 0:55:48,5
I would be actually grateful to see that.

1117
0:55:48,92 --> 0:55:51,98
But I would say pick your 10 queries.

1118
0:55:51,98 --> 0:55:53,22
Don't start with anything.

1119
0:55:53,22 --> 0:55:58,66
Find a way to pick 10 queries which you believe are, maybe not

1120
0:55:58,66 --> 0:56:2,24
small, I would say representative of something that's breaking.

1121
0:56:2,68 --> 0:56:6,6
Try with them, try to create a plans, try to try the tool, how

1122
0:56:6,6 --> 0:56:11,32
it works, what might be the flow, and try to live with it.

1123
0:56:11,32 --> 0:56:15,8
And then have a rule that every single handwritten or LLM written

1124
0:56:15,8 --> 0:56:18,46
query needs to be covered.

1125
0:56:18,74 --> 0:56:22,12
If you start with that, you will start seeing value in a couple

1126
0:56:22,12 --> 0:56:24,4
of weeks, a couple of months, because you will have that check

1127
0:56:24,4 --> 0:56:25,88
mark, everything is fine.

1128
0:56:26,32 --> 0:56:30,58
And then in future, hopefully all the features and the developer

1129
0:56:30,58 --> 0:56:35,08
flow will support that you will get much more values and you

1130
0:56:35,08 --> 0:56:39,04
will actually get your time back in terms of safer production.

1131
0:56:39,34 --> 0:56:40,52
Nikolay: Confidence level builds up.

1132
0:56:40,52 --> 0:56:44,24
Yeah, I mean, a small set of queries, but it's a good idea to

1133
0:56:44,24 --> 0:56:47,06
understand which queries matter most based on some production

1134
0:56:47,32 --> 0:56:48,06
or something.

1135
0:56:48,74 --> 0:56:52,84
And you have a website boringsql.com so this is where you have

1136
0:56:52,84 --> 0:56:56,3
blog and so on and the project you're talking about is RegreSQL,

1137
0:56:56,84 --> 0:56:57,34
right?

1138
0:56:57,72 --> 0:56:58,16
Radim: Exactly.

1139
0:56:58,16 --> 0:56:59,28
Yeah, great, great.

1140
0:56:59,82 --> 0:57:0,06
Cool.

1141
0:57:0,06 --> 0:57:2,9
Right now it's not featured, I'm preparing a section.

1142
0:57:2,92 --> 0:57:5,54
Nikolay: You have a huge roadmap I guess.

1143
0:57:7,36 --> 0:57:9,8
Radim: You can't even imagine and this is actually where I loved

1144
0:57:9,8 --> 0:57:9,92
it.

1145
0:57:9,92 --> 0:57:10,64
Nikolay: I can.

1146
0:57:12,72 --> 0:57:14,48
Yeah, that's great, that's great.

1147
0:57:14,54 --> 0:57:17,14
And the project is in Go, so that's great.

1148
0:57:17,14 --> 0:57:19,99
Yeah, good, good, I think, finishing words, right?

1149
0:57:19,99 --> 0:57:21,32
Unless Michael has something.

1150
0:57:21,82 --> 0:57:24,2
Michael: No, I'll link to everything in the in the show notes

1151
0:57:24,2 --> 0:57:27,28
if anybody needs anything that
we've mentioned and yeah thanks

1152
0:57:27,28 --> 0:57:29,24
so much for joining us Radim it's
been great.

1153
0:57:29,54 --> 0:57:32,86
Radim: Yeah thank you it was very
interesting and I loved talking

1154
0:57:32,86 --> 0:57:35,32
because you know it's an important
topic.

1155
0:57:35,38 --> 0:57:36,38
Nikolay: Thank you for coming.

1156
0:57:36,58 --> 0:57:37,9
Radim: Okay, thank you guys.