1
00:00:00,060 --> 00:00:02,660
Michael: Hello and welcome to Postgres.FM, a weekly show about

2
00:00:02,660 --> 00:00:03,540
all things PostgreSQL.

3
00:00:03,660 --> 00:00:06,220
I am Michael, founder of pgMustard, and this is Nik, founder

4
00:00:06,220 --> 00:00:06,800
of Postgres.AI.

5
00:00:07,040 --> 00:00:08,220
Hey Nik, how's it going?

6
00:00:09,240 --> 00:00:10,060
Nikolay: Going great.

7
00:00:10,120 --> 00:00:11,940
I'm very glad to see you.

8
00:00:12,100 --> 00:00:12,720
How are you?

9
00:00:12,720 --> 00:00:13,220
Michael: Likewise.

10
00:00:13,580 --> 00:00:14,840
I'm good, thank you.

11
00:00:15,060 --> 00:00:16,720
And you chose the topic this week.

12
00:00:16,720 --> 00:00:17,900
What are we talking about?

13
00:00:17,980 --> 00:00:21,940
Nikolay: Yeah, I think it's very interesting to discuss the level

14
00:00:21,940 --> 00:00:26,020
of automation we have in terms of all, you know, my position

15
00:00:26,040 --> 00:00:29,600
against managed Postgres, and in this case, it probably will

16
00:00:29,600 --> 00:00:34,280
be opposite, like saying that it's not enough what we have in

17
00:00:34,280 --> 00:00:37,780
terms of managed Postgres and also in terms of Kubernetes operators

18
00:00:38,080 --> 00:00:44,440
and other automation projects Postgres ecosystem has right now.

19
00:00:45,100 --> 00:00:53,600
So why I was thinking about it, imagine 2011, Heroku was started,

20
00:00:53,720 --> 00:00:55,320
Heroku Postgres was started.

21
00:00:55,640 --> 00:01:03,140
2013, RDS Postgres was released in November, I think, and at

22
00:01:03,240 --> 00:01:04,420
the end, I guess, right?

23
00:01:04,740 --> 00:01:11,320
So then this was a foundation of growth of interest, I think.

24
00:01:11,320 --> 00:01:14,080
Like some people say it's because of JSON or something.

25
00:01:14,200 --> 00:01:18,200
I agree with those arguments, but I think the central reason

26
00:01:18,200 --> 00:01:23,760
of why Postgres started to grow in 2014, 15 and by now is that

27
00:01:24,840 --> 00:01:30,060
before that, backend engineers, developers, they were always

28
00:01:30,060 --> 00:01:33,620
complaining how difficult it is to set up Postgres and configure

29
00:01:33,740 --> 00:01:37,120
it and backups and replication that just didn't want to deal

30
00:01:37,120 --> 00:01:42,100
with it and RDS and Heroku before that, they brought automation

31
00:01:42,380 --> 00:01:44,440
for basic things, right?

32
00:01:44,540 --> 00:01:50,180
And this, I think, simplified lives of a lot of engineers.

33
00:01:50,600 --> 00:01:51,360
That's great.

34
00:01:51,420 --> 00:01:54,440
In 2020, Supabase was released.

35
00:01:55,160 --> 00:02:00,100
And I think a new wave of audience was brought to Postgres ecosystem

36
00:02:00,180 --> 00:02:01,660
of front-end people actually.

37
00:02:02,360 --> 00:02:05,900
Because now it's not only Postgres automated, it's very well

38
00:02:05,900 --> 00:02:06,400
automated.

39
00:02:06,740 --> 00:02:10,700
Other components are very well automated, like REST API, real-time

40
00:02:10,760 --> 00:02:11,780
component, right?

41
00:02:11,880 --> 00:02:14,040
These things, authentication component.

42
00:02:14,540 --> 00:02:20,300
So you immediately start working on front-end, forgetting about

43
00:02:20,300 --> 00:02:20,800
backend.

44
00:02:21,820 --> 00:02:26,140
So I admire Supabase for bringing a lot of frontend guys to

45
00:02:26,140 --> 00:02:26,640
Postgres.

46
00:02:27,400 --> 00:02:28,200
That's great.

47
00:02:29,920 --> 00:02:32,980
But at some point they need to learn SQL, I'm pretty sure.

48
00:02:33,700 --> 00:02:34,740
So this is fine.

49
00:02:35,380 --> 00:02:39,980
And now we have AI builders, and this is another wave of users

50
00:02:40,080 --> 00:02:42,100
and basically not humans already sometimes.

51
00:02:42,260 --> 00:02:46,920
So we hear from Supabase and Neon
that a lot of clusters which

52
00:02:46,920 --> 00:02:53,560
are created these days are created
by request from like Cursor

53
00:02:53,560 --> 00:02:55,460
or something, Vibe coding, right?

54
00:02:56,320 --> 00:02:59,920
And so many, many clusters, many
of them are small and maybe

55
00:02:59,920 --> 00:03:04,440
it won't go anywhere because it's
just experiments, prototyping

56
00:03:04,640 --> 00:03:05,460
and so on.

57
00:03:05,720 --> 00:03:08,260
But some clusters grow and they
lack attention.

58
00:03:08,740 --> 00:03:12,480
So with RDS, there was a big shift.

59
00:03:12,740 --> 00:03:17,780
We talked about startup teams who
don't have DBA and they are

60
00:03:17,780 --> 00:03:21,540
fine with it until some point and
this is where Postgres.AI professional

61
00:03:21,540 --> 00:03:25,140
services catch them, right, quite
often.

62
00:03:25,320 --> 00:03:29,160
But now we talk about even lack
of backend engineering team completely,

63
00:03:29,260 --> 00:03:33,500
for example, with Supabase or
even without like lack of engineering

64
00:03:33,520 --> 00:03:34,740
team completely somehow.

65
00:03:34,740 --> 00:03:35,240
Right.

66
00:03:35,320 --> 00:03:39,220
And only guys who understand product
and try to wipe code this,

67
00:03:39,840 --> 00:03:43,580
sometimes with security breaches,
like recently some app was

68
00:03:44,320 --> 00:03:48,000
like storing data in, in Firebase,
right.

69
00:03:48,000 --> 00:03:48,800
Google Firebase.

70
00:03:49,000 --> 00:03:51,000
And that was not secure at all.

71
00:03:51,340 --> 00:03:52,940
5 million users registered.

72
00:03:53,000 --> 00:03:54,440
That was big scandal.

73
00:03:55,080 --> 00:03:59,080
So yeah, anyway, but this is part
of the topic security.

74
00:04:00,440 --> 00:04:05,140
So what I feel is a demand of much
higher automation than just

75
00:04:05,140 --> 00:04:06,360
RDS or Supabase.

76
00:04:06,820 --> 00:04:09,560
Some new level of automation should
be present.

77
00:04:10,460 --> 00:04:15,480
And if you look at enterprise sector,
there is Oracle with this

78
00:04:15,480 --> 00:04:20,920
idea of autonomous database, self-driving
database for many years,

79
00:04:21,100 --> 00:04:21,300
right?

80
00:04:21,300 --> 00:04:22,320
On 1 hand.

81
00:04:22,660 --> 00:04:27,620
On another hand, there are academic
papers like 1 from Carnegie

82
00:04:27,620 --> 00:04:33,360
Mellon University by Andy Pavlo
from 2017, which discusses what

83
00:04:33,540 --> 00:04:36,360
self-driving database management
systems are.

84
00:04:36,960 --> 00:04:38,640
And there's a question.

85
00:04:38,800 --> 00:04:42,840
If you think about zillions of
Postgres clusters, which should

86
00:04:42,840 --> 00:04:47,780
be highly automated And when experts
look at them, everything

87
00:04:47,780 --> 00:04:51,600
should be already transparent and
obvious how to fix and move

88
00:04:51,600 --> 00:04:52,100
on.

89
00:04:52,660 --> 00:04:53,560
What is this?

90
00:04:53,560 --> 00:04:55,220
Like what is self-driving Postgres?

91
00:04:55,440 --> 00:04:56,260
I was thinking.

92
00:04:56,720 --> 00:05:01,400
And to answer that I performed
several waves of research, of

93
00:05:01,400 --> 00:05:05,040
course, with deep research from
cloud and open the ChatGPT,

94
00:05:05,460 --> 00:05:05,960
right?

95
00:05:06,000 --> 00:05:09,020
Latest models, I paid to everyone
a lot of bucks already.

96
00:05:09,520 --> 00:05:12,660
So I was thinking, what is, what
could it be for Postgres?

97
00:05:13,440 --> 00:05:17,080
To answer that, I performed a search
looking at Oracle, first

98
00:05:17,080 --> 00:05:17,720
of all.

99
00:05:18,220 --> 00:05:25,080
And I just asked to, you know,
deep research is when they perform

100
00:05:25,080 --> 00:05:29,740
Google searches or Bing searches
and analyze hundreds of sources

101
00:05:30,060 --> 00:05:34,300
and then write some kind of report
like some student would write.

102
00:05:34,300 --> 00:05:37,960
It might have issues, of course,
with report, but it gives you

103
00:05:37,960 --> 00:05:39,980
a lot of links at least and some
summaries.

104
00:05:40,520 --> 00:05:45,680
So my question was, what people,
after all those years of building

105
00:05:45,680 --> 00:05:50,600
autonomous Oracle, what do people
really like and what they like

106
00:05:50,600 --> 00:05:51,920
less, right?

107
00:05:52,540 --> 00:05:54,520
What's the- What did you find?

108
00:05:54,520 --> 00:05:55,580
Michael: What did they say?

109
00:05:55,660 --> 00:05:57,900
Nikolay: Yeah, and 1 more comment.

110
00:05:57,900 --> 00:06:03,740
In 2013 or 14, I think I was attending
autonomous Oracle webinar

111
00:06:04,600 --> 00:06:08,680
and I was completely shocked.

112
00:06:09,220 --> 00:06:13,300
They promised autonomous Oracle,
but they only talked about clustering

113
00:06:14,280 --> 00:06:19,460
logs, organizing like better log
analysis from hundreds or thousands

114
00:06:19,460 --> 00:06:21,180
of sources and not only database.

115
00:06:21,740 --> 00:06:24,560
And I was like, where is autonomous
Oracle here?

116
00:06:25,380 --> 00:06:29,440
And 10 years since then, almost
more than 10 years, I was thinking

117
00:06:29,440 --> 00:06:30,560
it's stupid.

118
00:06:30,720 --> 00:06:35,720
Now I changed my mind and I hope
you and our audience will understand

119
00:06:35,720 --> 00:06:36,220
why.

120
00:06:36,400 --> 00:06:43,120
So what I found is that people
appreciate a lot self-patching.

121
00:06:43,740 --> 00:06:47,440
It's like many minor releases,
like if new release comes out

122
00:06:47,440 --> 00:06:51,140
with security patches, it's not
a headache at all.

123
00:06:51,240 --> 00:06:55,580
And this kind of automated RDS,
you can just define maintenance

124
00:06:55,580 --> 00:06:58,480
window, yeah, with some caveats.

125
00:06:59,480 --> 00:07:01,240
Michael: Minor versions only, I
think?

126
00:07:01,320 --> 00:07:03,240
Or have they done major versions
now?

127
00:07:03,320 --> 00:07:07,120
Nikolay: Major versions yeah automation
of major version upgrades

128
00:07:07,120 --> 00:07:10,580
and here we definitely have something
to discuss I mean we discussed

129
00:07:10,580 --> 00:07:15,560
it already and I mean my team like
we we had very good recent

130
00:07:16,400 --> 00:07:19,400
cases where our customers had the
0 downtime upgrades.

131
00:07:19,400 --> 00:07:21,080
It's like, we're very happy.

132
00:07:21,180 --> 00:07:22,860
I hope some blog posts are coming.

133
00:07:23,220 --> 00:07:26,260
Michael: 0 downtime is very different
to fully autonomous though.

134
00:07:26,260 --> 00:07:27,840
Like very, very different.

135
00:07:29,280 --> 00:07:30,320
Nikolay: From fully autonomous

136
00:07:31,020 --> 00:07:33,920
Michael: or self-driving a major
upgrade is very different.

137
00:07:34,760 --> 00:07:36,820
Nikolay: Let's do 1 more step back.

138
00:07:36,820 --> 00:07:38,500
What is self-driving car?

139
00:07:38,520 --> 00:07:39,560
Michael: Yeah, great.

140
00:07:40,440 --> 00:07:47,020
Nikolay: There are 6 levels defined
by SAE, like kind of standard

141
00:07:47,020 --> 00:07:47,720
or something.

142
00:07:47,980 --> 00:07:50,740
So there are 6 levels from 0 to
5.

143
00:07:50,940 --> 00:07:54,500
0 means not autonomous at all,
manual, regular car.

144
00:07:54,920 --> 00:07:56,700
And then 5 is fully autonomous.

145
00:07:57,040 --> 00:08:00,600
And looking at first few levels,
I realized interesting thing.

146
00:08:00,600 --> 00:08:05,060
So they talk about not each feature
particularly, but of combination

147
00:08:05,060 --> 00:08:05,580
of features.

148
00:08:05,580 --> 00:08:12,420
For example, level number 1, it
could be either either adaptive

149
00:08:12,660 --> 00:08:16,300
cruise control, like so keeping
my maintaining speed, but safely,

150
00:08:16,300 --> 00:08:16,800
right?

151
00:08:17,080 --> 00:08:20,100
Or maintaining lane, but not both.

152
00:08:20,100 --> 00:08:24,280
If it's both it's already the level
number 2 right and there

153
00:08:24,280 --> 00:08:28,820
are several levels and we and for
example this paper Carnegie

154
00:08:28,820 --> 00:08:33,640
Mellon Andy Pavlo's paper from
from 2017 and discusses how

155
00:08:33,640 --> 00:08:37,160
to map it to database management
systems right

156
00:08:37,680 --> 00:08:41,600
Michael: so a little bit not not
much in my opinion but a little

157
00:08:41,600 --> 00:08:45,460
bit yeah for sure and this this
paper I looked at this in advance

158
00:08:45,560 --> 00:08:48,160
you shared these car things I'll
link them up in the show notes

159
00:08:48,160 --> 00:08:48,900
as well.

160
00:08:49,440 --> 00:08:55,760
It struck me that level 4 to level
5, so the last step, is a

161
00:08:55,760 --> 00:08:56,680
huge jump.

162
00:08:57,540 --> 00:09:00,780
It's like here's loads of features
in the car that will help

163
00:09:00,780 --> 00:09:05,080
the driver and then level 5 is
suddenly and now the driver does

164
00:09:05,080 --> 00:09:05,580
nothing.

165
00:09:06,180 --> 00:09:11,240
So it's like that feels to me like
a whole like a maybe a potentially

166
00:09:11,480 --> 00:09:15,880
huge chasm that maybe there's like
a hundred more levels in between

167
00:09:15,880 --> 00:09:18,860
4 and 5 that we're going to need
to break down at some point.

168
00:09:18,920 --> 00:09:23,320
But like I felt like it was very,
a very hand wavy way of saying

169
00:09:23,320 --> 00:09:25,840
we might be really close to this
because we already have level

170
00:09:25,840 --> 00:09:28,940
4 features, we're very close to
having level 5.

171
00:09:28,940 --> 00:09:32,300
And it feels to me like there might
be like a, like I was unclear,

172
00:09:32,460 --> 00:09:37,200
for example, in a level 5 car,
whether a human could still take

173
00:09:37,200 --> 00:09:41,040
control if needed or was there
absolutely no way of doing that

174
00:09:41,040 --> 00:09:44,580
like that that feels to me like
a level that wasn't defined and

175
00:09:44,580 --> 00:09:48,140
maybe there's like other I believe
there'll be

176
00:09:48,140 --> 00:09:50,860
Nikolay: yeah yeah let me explain
how I see it.

177
00:09:50,860 --> 00:09:57,420
And I think if you ask several
people, they will answer differently.

178
00:09:57,660 --> 00:10:01,720
And I also heard Kubernetes ecosystem
tries to map it as well.

179
00:10:01,720 --> 00:10:06,680
And some operators, they claim
they have a very high automation,

180
00:10:06,760 --> 00:10:09,620
but many people say they are not,
they're not, they don't have,

181
00:10:09,620 --> 00:10:11,040
they don't have, and so on.

182
00:10:11,040 --> 00:10:15,060
So in terms of cars, let me like
propagate it.

183
00:10:15,060 --> 00:10:18,280
So number 2, like both these options,
for example, and this is

184
00:10:18,280 --> 00:10:20,060
what Tesla autopilot, for example,
does.

185
00:10:20,060 --> 00:10:21,140
I use it a lot.

186
00:10:21,820 --> 00:10:25,520
Like you just turn it on, but you
must sit and keep basically

187
00:10:25,520 --> 00:10:30,420
officially you must keep hands
on wheel and be ready to take

188
00:10:30,420 --> 00:10:32,020
control any second basically.

189
00:10:32,020 --> 00:10:32,520
Right.

190
00:10:32,680 --> 00:10:34,140
But it's still, it's, it's great.

191
00:10:34,140 --> 00:10:35,960
It maintains lane and speed.

192
00:10:36,080 --> 00:10:41,320
Like you just relaxes and, and,
spend much less effort.

193
00:10:41,320 --> 00:10:44,440
And I think we can think about
this in databases as well.

194
00:10:44,480 --> 00:10:48,580
Next level is level number 3.

195
00:10:49,180 --> 00:10:53,140
And level number 3, it's everything
is automated, but you still

196
00:10:53,140 --> 00:10:55,700
need to be to be ready to take
control.

197
00:10:55,760 --> 00:10:59,240
And this is what for example, Tesla
full self driving is, well,

198
00:10:59,240 --> 00:11:03,760
this is this is like, you not everything
automated.

199
00:11:04,200 --> 00:11:06,240
It's like under supervision of
you.

200
00:11:06,460 --> 00:11:08,480
Michael: You have just put I'm
just putting it up.

201
00:11:08,480 --> 00:11:11,400
No, no, no, it's it's not quite
that it's like level 3.

202
00:11:11,400 --> 00:11:13,880
It says here, for example, is a
traffic jam chauffeur.

203
00:11:14,140 --> 00:11:18,080
So it can handle like, basic traffic
conditions like a traffic

204
00:11:18,080 --> 00:11:18,580
jam.

205
00:11:18,780 --> 00:11:22,660
But for example, it doesn't account
for like all weather conditions

206
00:11:22,680 --> 00:11:25,280
or like there's a bunch of other
like potential things.

207
00:11:25,280 --> 00:11:27,700
So it's, it's not like it's

208
00:11:27,900 --> 00:11:28,600
Nikolay: the limitations.

209
00:11:29,060 --> 00:11:29,560
Michael: Exactly.

210
00:11:30,300 --> 00:11:30,780
And

211
00:11:30,780 --> 00:11:33,180
Nikolay: you still need to take
control if needed.

212
00:11:33,280 --> 00:11:35,260
Basically, you need to be ready
to take control.

213
00:11:35,280 --> 00:11:36,800
This is level 3, I agree.

214
00:11:37,800 --> 00:11:41,460
There are limits, but it can bring
you fully automatically from

215
00:11:41,460 --> 00:11:42,160
point to point.

216
00:11:42,160 --> 00:11:44,680
This is what Tesla full self-driving
does.

217
00:11:44,860 --> 00:11:48,540
You sit in the driver's seat, and
from point to point you can...

218
00:11:48,960 --> 00:11:54,300
Basically, you can enjoy full automation
of some whole ride,

219
00:11:54,600 --> 00:11:59,440
but if some bad condition occurs,
then you need to take control

220
00:11:59,440 --> 00:12:00,300
and fix things.

221
00:12:00,800 --> 00:12:06,260
For example, if we map to full
major upgrade, full major upgrade,

222
00:12:06,260 --> 00:12:07,580
downtime and so on.

223
00:12:07,580 --> 00:12:10,640
By the way, when we think about
autonomacy, we also bring some

224
00:12:10,640 --> 00:12:12,840
additional features like 0 downtime.

225
00:12:12,880 --> 00:12:16,880
It's like it could be in place
and without downtime, but somehow

226
00:12:16,880 --> 00:12:20,320
our mind wants some good features
additionally to autonomous.

227
00:12:20,800 --> 00:12:24,980
So yeah, this is like natural desire
to have good stuff.

228
00:12:26,040 --> 00:12:28,820
But we can imagine, for example,
we have a whole thing automated

229
00:12:30,240 --> 00:12:34,800
And under many circumstances, like
in many cases it will work,

230
00:12:35,140 --> 00:12:39,140
but in some edge cases it won't
and you will need to take control

231
00:12:39,140 --> 00:12:42,440
and make some decisions before
proceeding on or even postponing

232
00:12:42,560 --> 00:12:43,340
the whole procedure.

233
00:12:43,460 --> 00:12:46,600
This is very similar to Tesla full
self driving and I experienced

234
00:12:46,600 --> 00:12:46,920
it.

235
00:12:46,920 --> 00:12:49,280
It really can drive you from point
to point.

236
00:12:49,280 --> 00:12:53,800
But for example, if you go inside,
for example, my property,

237
00:12:53,800 --> 00:12:56,520
I have some roads inside it, it
won't be able to drive there

238
00:12:56,520 --> 00:12:59,480
at all because it's like, oh, this
is already not a road, not

239
00:12:59,480 --> 00:13:00,760
proper road, you know.

240
00:13:01,620 --> 00:13:07,420
So there is another level 4, it's
also conditional, but there

241
00:13:07,420 --> 00:13:11,940
you like what my perception again
might be wrong, there you can

242
00:13:11,940 --> 00:13:15,120
go to backseat and sit there.

243
00:13:15,460 --> 00:13:20,320
You are allowed to relax completely,
fully, but again it will

244
00:13:20,320 --> 00:13:21,880
work only under some conditions.

245
00:13:21,900 --> 00:13:23,700
For example, there is Waymo.

246
00:13:23,920 --> 00:13:25,960
I tried it multiple times in San
Francisco.

247
00:13:25,960 --> 00:13:26,640
It's amazing.

248
00:13:26,640 --> 00:13:31,300
You go to its Jaguar, you go to
backseat And everything is fine.

249
00:13:31,560 --> 00:13:35,840
But if like some, we saw this YouTube
videos or something, multiple

250
00:13:35,860 --> 00:13:40,980
Waymo machines, the cars, they
just create a traffic jam themselves

251
00:13:41,000 --> 00:13:43,480
and like basically deadlock, right?

252
00:13:44,440 --> 00:13:48,260
Michael: So like if they sense
that they can't drive or they

253
00:13:48,260 --> 00:13:51,660
spot something they can't deal
with, they'll stop as a safety

254
00:13:51,660 --> 00:13:52,160
precaution.

255
00:13:52,440 --> 00:13:53,360
And then what do you do?

256
00:13:53,360 --> 00:13:54,380
You have to get out?

257
00:13:54,380 --> 00:13:54,680
Nikolay: Yeah.

258
00:13:54,680 --> 00:13:55,880
In this case, yes.

259
00:13:56,340 --> 00:13:58,820
In the case of Waymo, you're a
passenger, yes.

260
00:13:58,820 --> 00:14:03,740
So you can imagine, For example,
if we map it to major upgrades,

261
00:14:03,740 --> 00:14:06,020
for example, imagine there is a
procedure for develop, there

262
00:14:06,020 --> 00:14:10,020
is a vendor who can intervene and
can take control sometimes

263
00:14:10,600 --> 00:14:11,260
if allowed.

264
00:14:11,680 --> 00:14:15,960
But the passenger in this case,
those who asked to perform full

265
00:14:15,960 --> 00:14:20,140
major upgrades, They are passengers,
they cannot make decisions,

266
00:14:20,340 --> 00:14:24,520
but this is good for them because
their mind is spent for product

267
00:14:24,520 --> 00:14:25,880
development, for example, right?

268
00:14:26,480 --> 00:14:32,820
But thanks to like million miles
already of experiments and real-life

269
00:14:33,960 --> 00:14:37,980
experience for these cars, If something
goes wrong, safety first,

270
00:14:38,000 --> 00:14:41,820
it will just abandon the trip,
I mean postpone it and cancel

271
00:14:41,980 --> 00:14:44,440
and another car will come later,
right?

272
00:14:44,480 --> 00:14:45,520
So this is approach.

273
00:14:45,740 --> 00:14:50,040
But It's like whole thing encapsulated,
like black box for you.

274
00:14:50,540 --> 00:14:53,900
You don't go down and don't make
decisions according to some

275
00:14:53,900 --> 00:14:54,980
diagram of decisions.

276
00:14:55,940 --> 00:14:57,460
But it has also limitations.

277
00:14:57,500 --> 00:15:01,020
And I think Waymo is perfect example
for level 4, because it

278
00:15:01,020 --> 00:15:05,380
works in San Francisco, in some
areas, but you cannot drive to

279
00:15:05,380 --> 00:15:09,520
San Jose, which is like slightly
more than 1 hour usually drive,

280
00:15:09,780 --> 00:15:11,460
because it's outside of coverage.

281
00:15:12,800 --> 00:15:16,220
And this is like, this can happen
here as well, major upgrades,

282
00:15:16,220 --> 00:15:21,400
but if there are some extensions,
for example, it's outside coverage,

283
00:15:23,640 --> 00:15:25,080
we don't support this kind of upgrade
because this extension,

284
00:15:25,080 --> 00:15:29,340
like, I don't know, Citus or TimescaleDB,
it requires an additional

285
00:15:29,340 --> 00:15:32,960
approach and we don't have it covered
here, right?

286
00:15:33,580 --> 00:15:38,700
So this is what I think here, like
we can map it and why not?

287
00:15:38,800 --> 00:15:46,100
But my main insight was looking
at deep research of feedback

288
00:15:46,220 --> 00:15:50,160
from Oracle users, DBAs and engineers.

289
00:15:51,580 --> 00:15:54,940
They say, upgrades are great, both
minor and major.

290
00:15:55,760 --> 00:16:01,320
Security is great, like security
control, automatic procedures

291
00:16:01,560 --> 00:16:03,140
to level up security.

292
00:16:03,740 --> 00:16:06,900
These kinds of maintenance things
are great, but when we talk

293
00:16:06,900 --> 00:16:11,060
about smart, like index advising
and so on, it's hit and miss

294
00:16:11,060 --> 00:16:11,560
situation.

295
00:16:13,260 --> 00:16:17,820
And this, I had an aha moment because
I was thinking, actually,

296
00:16:18,160 --> 00:16:22,960
if you look at what all people
try to do, they try to invent

297
00:16:23,640 --> 00:16:27,780
configuration tuning, automated,
with machine learning and AI.

298
00:16:27,780 --> 00:16:30,400
This is what Andy Pavlo was doing
with OtterTune.

299
00:16:30,460 --> 00:16:35,640
And by the end of, OtterTune was
closed, but by the end of Auto-Tune

300
00:16:35,640 --> 00:16:38,860
live, I already noticed the shift
which happened to Postgres

301
00:16:38,860 --> 00:16:39,600
AI earlier.

302
00:16:39,860 --> 00:16:44,220
Attention to query tuning and optimization,
like creation of

303
00:16:44,220 --> 00:16:44,720
indexes.

304
00:16:45,040 --> 00:16:49,900
And I see pganalyze is doing a
great job, not resisting to LLMs,

305
00:16:50,080 --> 00:16:51,240
which is also great.

306
00:16:51,740 --> 00:16:56,880
And also some teams go inside Postgres
and try to make planners

307
00:16:56,920 --> 00:16:57,420
smart.

308
00:16:58,140 --> 00:17:01,480
And I had, like, with configuration,
it's pretty straightforward

309
00:17:01,500 --> 00:17:02,980
for me, it's Pareto principle.

310
00:17:03,460 --> 00:17:11,040
Take pgtune.leopard.in.ua, very
simple configuration, 80% of

311
00:17:11,040 --> 00:17:14,520
job done in 1 person, like really,
really fast.

312
00:17:14,520 --> 00:17:18,480
It's just heuristic-based, rule-based
approach for OLTP or like

313
00:17:18,480 --> 00:17:18,980
anything.

314
00:17:19,340 --> 00:17:23,940
And it's good enough for many,
for many, like we don't need any

315
00:17:23,940 --> 00:17:25,240
machine learning and so on.

316
00:17:25,240 --> 00:17:27,960
Michael: And even when you say
for many as well, I think it's

317
00:17:27,960 --> 00:17:30,400
also about time.

318
00:17:30,620 --> 00:17:33,400
So it's for a while, it will then
be good.

319
00:17:33,400 --> 00:17:35,540
And so for long, it will be good
enough.

320
00:17:35,540 --> 00:17:36,760
So it's like...

321
00:17:36,820 --> 00:17:37,270
Nikolay: Yes, I agree.

322
00:17:37,270 --> 00:17:38,040
Yes, I agree.

323
00:17:38,100 --> 00:17:40,360
And then you need, but then you
need to tune.

324
00:17:40,520 --> 00:17:43,120
My way of tuning is to conduct
experiments.

325
00:17:43,180 --> 00:17:47,180
You know this very well, like how
to make experiments faster,

326
00:17:47,240 --> 00:17:49,600
cheaper, reproducible and so on.

327
00:17:49,600 --> 00:17:51,260
Like these kind of things.

328
00:17:51,620 --> 00:17:56,580
Michael: Is it worth, cause I think
self-driving, like probably

329
00:17:56,580 --> 00:18:00,300
level 4, many people would count
as self-driving, but there's

330
00:18:00,300 --> 00:18:04,000
still the kind of enough caveats
that a lot of the benefits of

331
00:18:04,000 --> 00:18:04,900
self-driving cars.

332
00:18:04,900 --> 00:18:06,340
Let's go back to cars briefly.

333
00:18:06,740 --> 00:18:09,520
I'm really excited and optimistic
about self-driving cars.

334
00:18:09,520 --> 00:18:13,900
I love the idea of being able to
get on with something else while

335
00:18:13,900 --> 00:18:14,760
something drives me.

336
00:18:14,760 --> 00:18:19,240
I don't mind that it might not
be like the fastest way or like

337
00:18:19,240 --> 00:18:21,600
it might not drive like completely
optimally.

338
00:18:22,420 --> 00:18:23,400
Nikolay: Not the safest.

339
00:18:23,900 --> 00:18:24,720
What about safest?

340
00:18:24,720 --> 00:18:27,100
Michael: Probably the safest but
not the fastest sorry.

341
00:18:27,180 --> 00:18:30,860
Probably safer than me but maybe
it would be a bit more gentle

342
00:18:30,860 --> 00:18:34,860
you know maybe it wouldn't take
the yellow light when a human

343
00:18:34,860 --> 00:18:36,820
driver would, you know, that kind
of thing.

344
00:18:37,040 --> 00:18:42,980
But I love that you can just watch
a movie or chat to a friend

345
00:18:42,980 --> 00:18:47,360
easily or play a board game in
the back, you know, you do whatever

346
00:18:47,360 --> 00:18:49,920
you want, you get so much time
back, especially in America where

347
00:18:49,920 --> 00:18:52,500
a lot of the time is spent driving.

348
00:18:52,660 --> 00:18:55,960
It makes so much sense to me that
self-driving cars are like

349
00:18:55,960 --> 00:18:58,760
a huge unlock for a lot of people.

350
00:18:59,640 --> 00:19:05,420
But largely only at stages 4 and
5, like at that highest level.

351
00:19:05,600 --> 00:19:09,440
Like cruise control is great, but
I still have to concentrate.

352
00:19:09,480 --> 00:19:10,760
I still have to be watching the
road.

353
00:19:10,760 --> 00:19:12,500
Like I don't actually gain that
much.

354
00:19:12,500 --> 00:19:16,600
And if we go back to Postgres,
I feel like a lot of the automation

355
00:19:16,940 --> 00:19:18,620
features are great.

356
00:19:19,200 --> 00:19:19,460
Nikolay: Yes.

357
00:19:19,460 --> 00:19:20,840
Michael: But we still have to concentrate.

358
00:19:20,840 --> 00:19:22,080
We still need the person.

359
00:19:22,140 --> 00:19:23,420
We still need the DBA.

360
00:19:23,840 --> 00:19:25,018
And as long as we still need the

361
00:19:25,018 --> 00:19:25,135
Nikolay: driver, and as long

362
00:19:25,135 --> 00:19:26,740
Michael: as we still need the driver,
and as long as we still

363
00:19:26,740 --> 00:19:30,760
need fail-safes down to humans,
all I see is kind of this like

364
00:19:30,760 --> 00:19:35,140
gradual need for maybe maybe this
is where the driving analogy

365
00:19:35,140 --> 00:19:39,280
breaks down a little bit but maybe
fewer humans per Server to

366
00:19:39,280 --> 00:19:43,380
like maybe the DBA team for a company
will be smaller on average

367
00:19:43,380 --> 00:19:46,000
compared to how it was in the past
you know that and I think

368
00:19:46,000 --> 00:19:50,320
we've already seen that over time
but I'm struggling with like

369
00:19:50,320 --> 00:19:54,960
the that last step until we get
to those which which feels to

370
00:19:54,960 --> 00:19:58,260
me like a long way off especially
given like the experience you're

371
00:19:58,260 --> 00:20:01,620
saying about like Oracle and the
experience we saw with very

372
00:20:01,620 --> 00:20:04,340
smart people trying to automate
a lot of this stuff from, you

373
00:20:04,340 --> 00:20:08,420
know, with a lot of AI, but kind
of not even LLM stuff, right?

374
00:20:08,420 --> 00:20:12,780
Like, a lot of the research in
this area has been machine learning

375
00:20:13,100 --> 00:20:20,440
and other longer researched AI
methodologies that have lots of

376
00:20:20,440 --> 00:20:24,440
real-world use cases and even there
we've seen mixed results

377
00:20:24,640 --> 00:20:30,720
and it really I feel like the model
struggling to employ In the

378
00:20:30,720 --> 00:20:33,940
experience I've had talking to
customers that use Auto-Tune for

379
00:20:33,940 --> 00:20:37,740
example, I feel like the Constraints
for example were not as

380
00:20:37,740 --> 00:20:43,500
clear as driving or the slightly
different use cases or the performance

381
00:20:43,500 --> 00:20:46,480
trade-offs that different people
have in different cases are

382
00:20:46,480 --> 00:20:49,540
subtle enough that you can't set
the exact same guardrails for

383
00:20:49,540 --> 00:20:50,040
everybody.

384
00:20:50,240 --> 00:20:55,320
And at that point, it breaks down
enough that well, and 1 more

385
00:20:55,320 --> 00:21:00,140
addition, performance cliffs are
so real that if you change 1

386
00:21:00,140 --> 00:21:02,520
thing, and it looks like it's going
to be great as soon as you

387
00:21:02,520 --> 00:21:05,580
hit a cliff it's then a disaster
and then recovering from those

388
00:21:05,580 --> 00:21:07,500
disasters is actually a real problem.

389
00:21:08,300 --> 00:21:09,250
Nikolay: So I feel like...

390
00:21:09,250 --> 00:21:11,340
Troubleshooting disaster is also
a problem.

391
00:21:11,540 --> 00:21:13,300
Root cause analysis is a problem.

392
00:21:13,320 --> 00:21:17,080
Michael: And arguably they get
harder when you involve automation

393
00:21:17,520 --> 00:21:20,860
because the more that it's automated
the less people actually

394
00:21:20,860 --> 00:21:23,100
know what was changed when and
why.

395
00:21:23,400 --> 00:21:24,850
So I think it can get tricky.

396
00:21:24,850 --> 00:21:28,880
Nikolay: I disagree here with you
because you must be an expert.

397
00:21:29,160 --> 00:21:34,340
But if you have automation, you
move much faster with high level

398
00:21:34,340 --> 00:21:35,040
of automation.

399
00:21:35,240 --> 00:21:41,180
Take Cursor, give it a lot of pieces
together, and explain how

400
00:21:41,180 --> 00:21:43,480
you approach methodology of analysis.

401
00:21:43,520 --> 00:21:47,200
This is like expert needs to bring, and then you move much faster.

402
00:21:48,000 --> 00:21:52,540
But this is like, I was trying to explain how I moved to this

403
00:21:52,540 --> 00:21:53,380
area completely.

404
00:21:53,440 --> 00:21:53,940
Right.

405
00:21:53,940 --> 00:21:54,200
Michael: Yeah.

406
00:21:54,200 --> 00:21:54,700
Okay.

407
00:21:54,800 --> 00:21:59,180
Nikolay: So people say in Oracle, this works that 1 doesn't.

408
00:21:59,380 --> 00:22:00,080
What works?

409
00:22:00,600 --> 00:22:01,960
Quite simple things.

410
00:22:02,860 --> 00:22:06,700
I, I said like upgrades, maintenance, security stuff.

411
00:22:06,700 --> 00:22:09,740
Well, not simple, but they are boring, you know.

412
00:22:09,960 --> 00:22:12,020
Of course, replication and backups.

413
00:22:12,180 --> 00:22:20,160
Like for me, HA and DR, It's like auto steering and maintaining

414
00:22:20,160 --> 00:22:21,700
speed and maintaining lane.

415
00:22:22,700 --> 00:22:23,200
Basics.

416
00:22:23,560 --> 00:22:27,540
Cars must do this, so database must have good HA and DR.

417
00:22:27,540 --> 00:22:30,920
If we look closer, actually, there are issues with both the HA

418
00:22:30,920 --> 00:22:35,920
and DR, which will prevent us from reaching very high level of

419
00:22:35,920 --> 00:22:36,420
automation.

420
00:22:36,660 --> 00:22:38,820
But we can dive into this later.

421
00:22:38,940 --> 00:22:43,300
But anyway, boring stuff lacks automation.

422
00:22:43,780 --> 00:22:47,980
Interesting, like, Remember I mentioned level 1 and 2, they talk

423
00:22:47,980 --> 00:22:49,400
about combinational features.

424
00:22:49,700 --> 00:22:52,720
So if we start analyzing each feature particularly, we cannot

425
00:22:52,720 --> 00:22:54,180
apply the same classification.

426
00:22:55,680 --> 00:22:59,560
You know, because classification talks about combinational features.

427
00:22:59,680 --> 00:23:00,420
Michael: Yeah, sure.

428
00:23:01,500 --> 00:23:04,840
Coming back actually, just to make sure I understand, what have

429
00:23:04,840 --> 00:23:06,600
Oracle done in terms of security?

430
00:23:06,820 --> 00:23:09,560
Can you give some example automation features?

431
00:23:10,080 --> 00:23:11,780
Nikolay: Well, I know a little.

432
00:23:11,840 --> 00:23:16,280
I would say what we should do in Postgres and This is like least

433
00:23:16,280 --> 00:23:19,820
topic I would like, I'm ready to discuss now.

434
00:23:19,820 --> 00:23:22,360
Because this is in the roadmap, but right now we're focusing

435
00:23:22,360 --> 00:23:23,340
on different areas.

436
00:23:24,020 --> 00:23:28,000
I can just speculate on this, like identify potential threats,

437
00:23:28,000 --> 00:23:29,720
like checking permissions, roles.

438
00:23:30,760 --> 00:23:35,500
For example, I know organizations which use the same super user

439
00:23:35,500 --> 00:23:40,180
for all human DB engineers, and this is quite easy to identify

440
00:23:40,240 --> 00:23:41,180
or any organization.

441
00:23:41,440 --> 00:23:47,620
I had a couple of them, on consulting contracts, which passed

442
00:23:47,620 --> 00:23:49,020
through IPO process.

443
00:23:49,300 --> 00:23:52,460
Before that you have audit, right?

444
00:23:52,540 --> 00:23:54,800
And during audit, they ask specific questions.

445
00:23:54,800 --> 00:23:57,900
Some of them are quite like silly, I would say, but some of them

446
00:23:57,900 --> 00:23:58,940
are quite good enough.

447
00:23:58,940 --> 00:24:05,000
And you just inspect your pg_hba.conf, you inspect a user model,

448
00:24:05,080 --> 00:24:07,940
you inspect how multi-tenancy is organized.

449
00:24:07,940 --> 00:24:09,940
We had an episode about it, right?

450
00:24:10,460 --> 00:24:13,140
So these kinds of things can be automatically analyzed and so

451
00:24:13,140 --> 00:24:13,260
on.

452
00:24:13,260 --> 00:24:13,820
I don't know.

453
00:24:13,820 --> 00:24:15,460
I don't know details about Oracle.

454
00:24:15,460 --> 00:24:19,900
I just saw feedback that these
stuff engineers really appreciate

455
00:24:20,220 --> 00:24:24,440
and they appreciate less automated
configuration, automated creation

456
00:24:24,440 --> 00:24:25,060
of indexes.

457
00:24:25,960 --> 00:24:27,200
And this was appreciated

458
00:24:27,200 --> 00:24:30,940
Michael: less or appreciate less
or it gets it wrong.

459
00:24:30,940 --> 00:24:32,720
When it gets it wrong, it's more
painful.

460
00:24:32,720 --> 00:24:34,000
Like what's, do you see what

461
00:24:34,000 --> 00:24:34,620
Nikolay: I mean?

462
00:24:35,060 --> 00:24:35,900
Mixed results, mixed results.

463
00:24:35,900 --> 00:24:39,500
I don't know, like there's lack
of trust in some minds.

464
00:24:39,840 --> 00:24:41,820
Like, I don't know.

465
00:24:42,640 --> 00:24:45,300
Michael: I, for example, I catch
up with customers from time

466
00:24:45,300 --> 00:24:49,600
to time just to hear what they're
doing, what they like about

467
00:24:49,600 --> 00:24:50,840
products, what they don't.

468
00:24:51,500 --> 00:24:57,340
And I was speaking to a customer
of mine that did try and use

469
00:24:57,340 --> 00:24:58,680
OtterTune for a while.

470
00:24:59,200 --> 00:25:02,860
And it'd be interesting, I'd love,
maybe we should invite somebody

471
00:25:02,860 --> 00:25:06,580
from that team on to discuss what
happened there.

472
00:25:06,580 --> 00:25:07,080
Yeah.

473
00:25:07,820 --> 00:25:08,960
Why did it shut down?

474
00:25:08,960 --> 00:25:10,820
Nikolay: I can tell you why.

475
00:25:11,760 --> 00:25:14,620
People don't need configuration
tuning that much.

476
00:25:15,480 --> 00:25:15,980
Michael: Okay.

477
00:25:16,060 --> 00:25:18,840
Well, I also think there might
be other issues.

478
00:25:18,840 --> 00:25:21,180
So there's definitely like not
needing

479
00:25:21,180 --> 00:25:24,400
Nikolay: I tell you the big need
in configuration tuning exists

480
00:25:24,440 --> 00:25:28,200
only if you have, say, 10, 000
plus clusters.

481
00:25:28,940 --> 00:25:33,640
Then you can say, okay, we are
going to save like 5 to 10% of

482
00:25:33,640 --> 00:25:37,160
money just with tuning or workload
will be reduced.

483
00:25:38,460 --> 00:25:45,520
We know, you know very well, a
really bad plan can screw all

484
00:25:45,520 --> 00:25:48,060
efforts of configuration tuning.

485
00:25:48,520 --> 00:25:50,440
Michael: Well, yes, but I think
it's worse than that.

486
00:25:50,440 --> 00:25:55,200
I think also in addition to it
not being needed that often and

487
00:25:55,200 --> 00:25:58,440
therefore kind of subscription
based models not working that

488
00:25:58,440 --> 00:26:04,440
well, also when it, well this customer
was telling me, they moved

489
00:26:04,440 --> 00:26:09,720
from a mental state of, when something
goes wrong, let's dive

490
00:26:09,720 --> 00:26:13,520
into what happened, straight to,
when something goes wrong now,

491
00:26:13,700 --> 00:26:15,320
what did OtterTune change?

492
00:26:15,480 --> 00:26:17,160
And that was like a real shift.

493
00:26:17,160 --> 00:26:17,720
It became

494
00:26:17,720 --> 00:26:18,340
Nikolay: like a...

495
00:26:18,340 --> 00:26:18,840
Trust.

496
00:26:19,140 --> 00:26:22,760
Michael: Trust, but also that must
be based on the fact it made

497
00:26:22,940 --> 00:26:26,020
changes in the past that made things
worse.

498
00:26:26,400 --> 00:26:31,660
So it's not trust for no reason,
it's not kind of like distrust

499
00:26:31,840 --> 00:26:32,860
for no reason.

500
00:26:32,860 --> 00:26:36,220
It's every now and again if you
change something, you hit a performance

501
00:26:36,220 --> 00:26:36,680
cliff.

502
00:26:36,680 --> 00:26:37,620
And it's unexpected.

503
00:26:37,860 --> 00:26:40,640
Or maybe it's not even, like, always
a performance cliff.

504
00:26:40,640 --> 00:26:43,500
Maybe it has another unintended
consequence that you care about

505
00:26:43,500 --> 00:26:44,000
more.

506
00:26:44,700 --> 00:26:46,320
Probably not in a lot of these
cases.

507
00:26:46,320 --> 00:26:50,440
But I think they talked about,
for example, making the mistake

508
00:26:50,440 --> 00:26:53,400
of letting it configure some parameters
that even affected, like,

509
00:26:53,400 --> 00:26:54,960
durability and things like that.

510
00:26:54,960 --> 00:26:58,940
So if you're changing depending
on what you allow it to change,

511
00:26:58,940 --> 00:27:00,960
there might be unintended side
effects.

512
00:27:00,960 --> 00:27:05,080
And I think that's, like, putting
parameters around, like, what

513
00:27:05,080 --> 00:27:09,220
you do, you will and won't let
it do, is actually harder than

514
00:27:09,220 --> 00:27:10,240
it sounds, I think.

515
00:27:10,240 --> 00:27:11,400
Like, it's very hard.

516
00:27:11,680 --> 00:27:16,420
Nikolay: To perform, like, enterprise
approach for making a change,

517
00:27:16,980 --> 00:27:18,420
it's extremely complex.

518
00:27:19,140 --> 00:27:24,940
I'm very grateful that 7 years
ago, I was working with Chewy.

519
00:27:25,920 --> 00:27:30,300
They were preparing for IPO, and
I remember CTO was ex-Oracle,

520
00:27:31,560 --> 00:27:35,760
and discussions we had and resistance
they had in any change

521
00:27:35,760 --> 00:27:36,420
I proposed.

522
00:27:36,900 --> 00:27:39,180
It taught me this enterprise approach.

523
00:27:39,400 --> 00:27:40,240
I'm very grateful.

524
00:27:40,240 --> 00:27:42,420
It was great experience for me.

525
00:27:42,560 --> 00:27:46,780
And I realized, oh, actually, if
you want to be serious about

526
00:27:46,780 --> 00:27:51,540
changes, any small change should
be very thoroughly tested, like

527
00:27:52,200 --> 00:27:57,940
experiments, experiments, all risks
must be analyzed, and then

528
00:27:58,480 --> 00:28:02,140
there should be a plan to mitigate
if a risk, like a non-risk

529
00:28:02,240 --> 00:28:03,220
occurs, right?

530
00:28:03,740 --> 00:28:07,780
And AI doesn't help here almost,
you know, like this is framework

531
00:28:07,780 --> 00:28:09,740
you need to build without AI first.

532
00:28:09,920 --> 00:28:10,520
But this

533
00:28:10,520 --> 00:28:11,540
Michael: is, yeah.

534
00:28:12,180 --> 00:28:13,460
I don't necessarily agree.

535
00:28:13,460 --> 00:28:16,080
I think AI could really help with
these things.

536
00:28:16,560 --> 00:28:20,500
When I say AI, I'm including machine
learning and not just the

537
00:28:20,500 --> 00:28:21,960
latest LLM stuff.

538
00:28:22,500 --> 00:28:26,820
I just think we need to define
constraints really clearly and

539
00:28:26,820 --> 00:28:30,040
define what we care about really
clearly and make it really clear

540
00:28:30,040 --> 00:28:33,480
we care more about reliability
and durability than we do about

541
00:28:33,480 --> 00:28:33,980
performance.

542
00:28:34,440 --> 00:28:35,580
So, like,

543
00:28:35,580 --> 00:28:36,600
Nikolay: yes, yes.

544
00:28:36,880 --> 00:28:38,300
Michael: And that's almost always
true.

545
00:28:38,300 --> 00:28:41,040
And I think that might be more
core to the reason that these

546
00:28:41,040 --> 00:28:45,160
performance tuning tools aren't
haven't yet succeeded because

547
00:28:45,160 --> 00:28:49,420
we haven't yet nailed the reliability,
durability stuff.

548
00:28:49,860 --> 00:28:55,120
So that would be my theory as to
like why they didn't like necessarily

549
00:28:55,380 --> 00:28:55,880
succeed.

550
00:28:56,460 --> 00:29:00,260
Because if we help, even if they
did help performance almost

551
00:29:00,260 --> 00:29:03,760
all the time, if they ever hurt
reliability, that's not a tradeoff

552
00:29:03,760 --> 00:29:05,920
most organizations are willing
to make.

553
00:29:06,100 --> 00:29:10,020
And that's a difficult thing to
tell a tool that's trying to

554
00:29:10,020 --> 00:29:11,360
optimize for better performance.

555
00:29:12,280 --> 00:29:12,780
Nikolay: Yeah.

556
00:29:13,100 --> 00:29:14,020
Yeah, I agree.

557
00:29:14,180 --> 00:29:15,840
And durability has issues.

558
00:29:16,200 --> 00:29:19,900
There is a good article from Sugu
just published on Sugu's blog

559
00:29:19,900 --> 00:29:24,280
about synchronous replication issues
and we also know the very

560
00:29:24,280 --> 00:29:28,400
good talk by Alexander Kukushkin
about issues with synchronous

561
00:29:28,400 --> 00:29:28,900
replication.

562
00:29:30,180 --> 00:29:36,160
So durability is a must thing to
have and targets, I agree with

563
00:29:36,160 --> 00:29:39,900
you, targets, durability, availability,
must be reliability,

564
00:29:40,280 --> 00:29:42,480
must be number 1 before performance.

565
00:29:42,780 --> 00:29:47,220
By the way, I also remember from
that research, people appreciate

566
00:29:48,100 --> 00:29:51,840
automated analysis and control
of costs.

567
00:29:53,360 --> 00:29:55,940
And this can be initially quite
simple.

568
00:29:57,400 --> 00:30:03,120
I remember actually talking to
1 huge organization, their database

569
00:30:03,120 --> 00:30:06,680
director told me, you know what,
it's cool stuff, what you're

570
00:30:06,680 --> 00:30:11,520
showing in terms of experiments
and performance tuning, like

571
00:30:11,520 --> 00:30:14,460
query tuning experiments with Database Lab
and so on.

572
00:30:14,460 --> 00:30:18,740
But the number 1 problem we have
is abandoned instances.

573
00:30:19,740 --> 00:30:23,340
And how to stop doing that and
lose a lot of money.

574
00:30:23,680 --> 00:30:26,060
And big organizations, it's a big
problem.

575
00:30:26,680 --> 00:30:31,160
And yeah, so cloud providers still
don't offer good tools.

576
00:30:31,240 --> 00:30:35,760
It still takes a lot of effort
to understand the cost is how

577
00:30:35,760 --> 00:30:37,560
the structure of spending right?

578
00:30:38,200 --> 00:30:40,680
Practically actively like usually
it's a little late.

579
00:30:40,680 --> 00:30:41,640
They are not interested.

580
00:30:41,640 --> 00:30:46,420
Yeah, right. So anyway, what my
aha moment back to it was

581
00:30:46,420 --> 00:30:47,120
Michael: yeah, sure

582
00:30:47,420 --> 00:30:53,560
Nikolay: people like from academia
from really great people, right?

583
00:30:53,560 --> 00:30:54,260
Great minds.

584
00:30:54,260 --> 00:30:56,080
They try to build really cool stuff.

585
00:30:56,980 --> 00:31:01,820
Like let's have automated parameter
tuning, automated indexing,

586
00:31:02,300 --> 00:31:06,480
or even let's go inside Planner
and create an adaptive query

587
00:31:06,480 --> 00:31:06,980
optimizer.

588
00:31:07,740 --> 00:31:09,260
Michael: I saw even more extreme.

589
00:31:10,440 --> 00:31:15,480
In the paper, they were talking
about choosing whether tables

590
00:31:15,480 --> 00:31:21,680
should be row oriented or column
oriented based on the workload

591
00:31:21,680 --> 00:31:22,460
they're observing.

592
00:31:22,500 --> 00:31:25,660
So it's like trying to attack really
cool

593
00:31:25,760 --> 00:31:26,160
Nikolay: areas.

594
00:31:26,160 --> 00:31:27,240
This is great.

595
00:31:27,260 --> 00:31:30,260
And this has been always so.

596
00:31:30,480 --> 00:31:34,860
Academia guys, they attack like
really detached from reality,

597
00:31:35,200 --> 00:31:35,700
things.

598
00:31:36,660 --> 00:31:41,200
Meanwhile, I realized we implemented
with multiple teams already

599
00:31:41,200 --> 00:31:44,540
automated indexing, and this is
what people really need.

600
00:31:44,540 --> 00:31:49,120
And lately I realized in consulting
we almost always say you

601
00:31:49,120 --> 00:31:53,160
need automated re-indexing, but
we didn't have a polished solution.

602
00:31:53,160 --> 00:31:55,460
We have multiple ways to do it.

603
00:31:55,460 --> 00:32:00,040
I always said you take this tool,
like someone from our team

604
00:32:00,040 --> 00:32:03,140
developed as a side project and
then polish it.

605
00:32:04,220 --> 00:32:07,560
And it's only about bit reindexes
and this and that.

606
00:32:07,800 --> 00:32:10,560
And also estimates, bloat estimates
might be off.

607
00:32:10,560 --> 00:32:14,980
We know very well if there are
estimates, not just startuple

608
00:32:15,840 --> 00:32:20,240
numbers or real fucking like reindex
numbers from a clone, right?

609
00:32:20,380 --> 00:32:23,540
And I realized, actually, this
problem is not solved.

610
00:32:24,280 --> 00:32:25,740
And this is a boring problem.

611
00:32:26,580 --> 00:32:27,660
And we can solve it.

612
00:32:27,660 --> 00:32:30,840
That's why, number 1 thing right
now we are going to release.

613
00:32:30,840 --> 00:32:33,780
We already, like, it's about to
be released.

614
00:32:33,800 --> 00:32:36,300
We call it pg_index_pilot.

615
00:32:37,060 --> 00:32:39,940
And there is a good roadmap inside
of this whole project.

616
00:32:40,400 --> 00:32:45,360
And it's going to be real simple.

617
00:32:46,640 --> 00:32:53,840
And A guy who part-time works with
me right now, Maxim Boguk,

618
00:32:54,720 --> 00:32:59,960
1 of the most experienced DBAs,
Postgres DBAs I know, much more

619
00:32:59,960 --> 00:33:01,240
experienced than I.

620
00:33:01,880 --> 00:33:06,140
He created, basically I consider
prototype, I said we fork it

621
00:33:06,140 --> 00:33:08,840
and then we started iterating on
it.

622
00:33:08,920 --> 00:33:12,040
The idea is simple, I call it a
Boguk number.

623
00:33:12,240 --> 00:33:17,860
You take index size, divide by
the number of life tuples in this

624
00:33:17,860 --> 00:33:18,360
table.

625
00:33:19,540 --> 00:33:22,220
Let's forget for a while for partial
indexes.

626
00:33:22,860 --> 00:33:24,560
And we have some ratio, right?

627
00:33:25,840 --> 00:33:29,340
When we just created the index,
let's consider this ratio as

628
00:33:29,340 --> 00:33:29,840
perfect.

629
00:33:30,580 --> 00:33:33,620
And consider tables which exceed,
like, say, a million rows.

630
00:33:33,820 --> 00:33:35,440
Over time, you will see...

631
00:33:35,460 --> 00:33:38,560
This number is nothing, like, it
costs nothing to check it.

632
00:33:38,560 --> 00:33:40,960
You can put it to monitoring for
all indexes.

633
00:33:41,040 --> 00:33:43,320
It's super fast to get a number
of...

634
00:33:43,320 --> 00:33:44,940
Michael: Because these aggregates
are stored.

635
00:33:45,060 --> 00:33:46,160
They're stored, right?

636
00:33:46,160 --> 00:33:46,920
Nikolay: Yeah, yeah, yeah.

637
00:33:46,920 --> 00:33:47,800
It's from system catalogs.

638
00:33:47,800 --> 00:33:48,340
System views.

639
00:33:48,340 --> 00:33:49,780
Immediately you get it.

640
00:33:49,780 --> 00:33:52,400
And then over time you see degradation
of this number.

641
00:33:53,420 --> 00:33:53,900
Why?

642
00:33:53,900 --> 00:33:57,180
Because some pages inside the index,
they are not fully...

643
00:33:57,440 --> 00:33:58,480
They are not full.

644
00:33:58,860 --> 00:34:01,100
They're half empty and so on, like
they're sparse.

645
00:34:01,100 --> 00:34:01,560
Yeah.

646
00:34:01,560 --> 00:34:05,040
So it means at some point you say,
oh, it's time to re-index.

647
00:34:06,040 --> 00:34:09,140
And the best, there are pros and
cons of this approach compared

648
00:34:09,140 --> 00:34:13,100
to like say traditional based on
bloat estimates everyone is

649
00:34:13,100 --> 00:34:13,600
using.

650
00:34:13,860 --> 00:34:19,160
The big con, like a couple of big
cons of this approach, it required

651
00:34:19,160 --> 00:34:21,900
us some effort to get away from
super user.

652
00:34:21,900 --> 00:34:22,780
We did it.

653
00:34:23,500 --> 00:34:25,660
And oh, 1 more thing.

654
00:34:27,040 --> 00:34:30,920
On purpose, we said this component
is going to be inside, like

655
00:34:30,920 --> 00:34:32,720
self-driving inside database.

656
00:34:33,400 --> 00:34:38,360
We don't need external means like
something installed on EC2

657
00:34:38,360 --> 00:34:39,400
instance or lambdas.

658
00:34:39,400 --> 00:34:40,440
We don't need anything.

659
00:34:40,440 --> 00:34:41,400
It will be inside.

660
00:34:41,480 --> 00:34:44,580
This means it's running inside
PL/pgSQL code.

661
00:34:45,040 --> 00:34:49,940
We know since Postgres 12 or earlier,
stored procedures have

662
00:34:49,940 --> 00:34:51,400
transaction control, right?

663
00:34:51,820 --> 00:34:54,840
So we can go, but we need to index
concurrently, right?

664
00:34:54,840 --> 00:34:59,840
So with index concurrently, we
cannot wrap it inside a transaction

665
00:34:59,920 --> 00:35:00,420
block.

666
00:35:00,900 --> 00:35:04,320
So unfortunately we need something
like dblink, right?

667
00:35:04,760 --> 00:35:09,340
And it's a challenge not to do
it properly on RDS, for example,

668
00:35:09,340 --> 00:35:14,200
because dblink, you need to expose
password, and you don't want

669
00:35:14,200 --> 00:35:16,580
to store it in plain text.

670
00:35:16,900 --> 00:35:22,700
So I remembered very old trick
I used a lot of years ago, dblink

671
00:35:22,740 --> 00:35:23,640
over postgres_fdw.

672
00:35:24,920 --> 00:35:27,180
And this is how we do it right
now.

673
00:35:27,500 --> 00:35:30,580
And there is another kind of limitation.

674
00:35:31,100 --> 00:35:33,900
To start this thing to work, you
need baseline.

675
00:35:35,600 --> 00:35:40,120
So baseline means you need to index
everything first or to bring

676
00:35:40,120 --> 00:35:43,260
this data from clone which we are
this is an idea where I'm going

677
00:35:43,260 --> 00:35:47,780
to implement very soon so to avoid
full reindex because sometimes

678
00:35:47,800 --> 00:35:53,220
our customers have like 10 plus
terabytes databases and it's

679
00:35:53,220 --> 00:35:57,240
not cool to ask them to re-index,
it will take forever and so

680
00:35:57,240 --> 00:36:02,380
on, there's also a big impact when
you re-index quite fast.

681
00:36:02,460 --> 00:36:06,160
So, But a good benefit from this
approach, it's not only about

682
00:36:06,160 --> 00:36:06,660
B-tree.

683
00:36:10,160 --> 00:36:17,840
You can take care of GiST, SP-GiST,
and even HNSW and others if

684
00:36:17,840 --> 00:36:18,500
they degrade.

685
00:36:18,900 --> 00:36:22,760
And like basically, yeah, basically
we measure kind of storage

686
00:36:22,760 --> 00:36:23,920
efficiency for index.

687
00:36:23,920 --> 00:36:25,100
It's super trivial.

688
00:36:25,240 --> 00:36:27,820
And I think I believe into this
simple approach.

689
00:36:28,140 --> 00:36:29,940
I think we are going to have it.

690
00:36:30,020 --> 00:36:34,300
And I think also like I talked
about this last couple of weeks

691
00:36:34,300 --> 00:36:39,260
ago with Andrey and Kirk on Postgres
hacking on Postgres TV.

692
00:36:40,080 --> 00:36:42,540
And we started doing something
mind-blowing.

693
00:36:42,720 --> 00:36:45,000
Andrey just said, let's just implement
merge.

694
00:36:45,400 --> 00:36:50,920
Because you know, B-tree and B-tree implementation
in Postgres, it has

695
00:36:50,920 --> 00:36:51,700
only split.

696
00:36:52,680 --> 00:36:54,260
It cannot merge pages.

697
00:36:55,840 --> 00:36:59,940
And since Andrey's PhD and his
work is in the area of indexes,

698
00:37:00,140 --> 00:37:04,560
it was great to have ability to
start this work, and I'm looking

699
00:37:04,560 --> 00:37:06,220
forward to, like, every 1.

700
00:37:06,220 --> 00:37:06,720
Michael: Wait.

701
00:37:08,260 --> 00:37:12,660
So, like, if, for example, an area
of the index starts to get

702
00:37:12,660 --> 00:37:16,400
sparse because we've deleted a
bunch of let's say we've deleted

703
00:37:16,400 --> 00:37:19,700
some historic data and we're not
going to insert historic data

704
00:37:19,700 --> 00:37:23,680
back in that part of the index,
it could proactively, like, it's

705
00:37:23,680 --> 00:37:24,780
kind of self healing.

706
00:37:26,240 --> 00:37:26,820
Wow, cool.

707
00:37:26,820 --> 00:37:30,480
Nikolay: So, our project would
be archived at some point, I hope.

708
00:37:30,820 --> 00:37:36,760
So, I'm not an expert in Oracle
since I have never been an expert

709
00:37:37,060 --> 00:37:39,060
of Oracle, but also not a user.

710
00:37:39,120 --> 00:37:42,640
The last version I used was in
2001 or 2002.

711
00:37:42,640 --> 00:37:45,100
It was 8i.

712
00:37:46,440 --> 00:37:50,940
But people say Oracle doesn't have
this need in re-indexing.

713
00:37:52,200 --> 00:37:54,520
SQL Server has this need.

714
00:37:54,520 --> 00:37:56,100
Over time, health declines.

715
00:37:56,280 --> 00:37:57,540
We talked about it so much.

716
00:37:57,540 --> 00:37:59,540
I said, this is mantra.

717
00:38:00,300 --> 00:38:02,160
Everyone needs re-indexing at some
point.

718
00:38:02,160 --> 00:38:05,900
And we know Peter Geoghegan's work
and Anastasia Lubenikova in 2013-14,

719
00:38:06,340 --> 00:38:09,520
they implemented deduplication
and other improvements.

720
00:38:09,520 --> 00:38:10,280
That's great.

721
00:38:10,380 --> 00:38:11,860
But still, there is no merge.

722
00:38:12,380 --> 00:38:14,360
So pages cannot be merged

723
00:38:15,020 --> 00:38:15,060
Michael: automatically.

724
00:38:15,060 --> 00:38:19,000
And there was like some work, there
was also some work I think

725
00:38:19,000 --> 00:38:23,540
from Peter Geoghegan and others to
help avoid page splits in more

726
00:38:23,540 --> 00:38:24,040
cases.

727
00:38:24,340 --> 00:38:24,685
Yeah, that's great.

728
00:38:24,685 --> 00:38:27,440
And not just the deduplication
work, but yeah, the bottom up

729
00:38:27,440 --> 00:38:27,940
deletion.

730
00:38:28,140 --> 00:38:31,620
So yeah, kind of avoiding them
getting into this state was 1

731
00:38:31,620 --> 00:38:34,700
thing, helping them heal when they
do is another.

732
00:38:35,080 --> 00:38:38,680
It also strikes me as a lot of
this stuff could live in core,

733
00:38:38,680 --> 00:38:39,180
right?

734
00:38:39,280 --> 00:38:42,040
We already have some automation
features, right?

735
00:38:42,040 --> 00:38:45,920
We already have autovacuum, which
does about 3 or 4 different

736
00:38:45,920 --> 00:38:46,420
jobs.

737
00:38:46,560 --> 00:38:50,380
So we have some groundwork here
already.

738
00:38:50,380 --> 00:38:55,280
We have tools like pg_repack and
pg_squeeze, not in core, but some

739
00:38:55,280 --> 00:38:58,020
ideas of moving more of their features
into core.

740
00:38:59,280 --> 00:39:03,120
So we do have some of this in core,
some of it in popular extensions

741
00:39:03,120 --> 00:39:04,940
that are supported by many clouds.

742
00:39:05,080 --> 00:39:09,960
So that feels like it is the project
is already naturally going

743
00:39:09,960 --> 00:39:10,820
in this direction.

744
00:39:11,780 --> 00:39:15,540
Maybe slowly and maybe like not
all the areas you want it to,

745
00:39:15,540 --> 00:39:18,660
but is that like What does the
end goal look like here?

746
00:39:18,760 --> 00:39:21,240
Is it being in core ideal?

747
00:39:21,780 --> 00:39:25,820
Nikolay: In my opinion, we just
need to solve very complex problems.

748
00:39:26,780 --> 00:39:31,220
I know in some big companies, managed
Postgres services, sometimes

749
00:39:31,640 --> 00:39:35,840
1 experience DBA is responsible
for a thousand clusters.

750
00:39:36,820 --> 00:39:37,480
A thousand.

751
00:39:37,660 --> 00:39:38,440
It's insane.

752
00:39:38,520 --> 00:39:41,640
But we need to be prepared to be
responsible for a million clusters

753
00:39:41,640 --> 00:39:44,240
because AI builders will bring
us a lot of clusters.

754
00:39:44,540 --> 00:39:46,620
Like the time changing really fast.

755
00:39:46,720 --> 00:39:50,740
So Postgres need like not to lose
the game by the way right now,

756
00:39:50,740 --> 00:39:56,140
if you check Hacker News trends,
it was growing and not only

757
00:39:56,140 --> 00:39:59,440
about job postings as I usually
mentioned, but everything, all

758
00:39:59,440 --> 00:40:02,800
discussions on Hacker News, it
was growing until the beginning

759
00:40:02,800 --> 00:40:04,140
of last year, 2024.

760
00:40:04,940 --> 00:40:06,860
And then we have slight decline.

761
00:40:07,400 --> 00:40:10,440
And I think Postgres has right
now huge challenge.

762
00:40:11,200 --> 00:40:13,280
It needs to be much more automated.

763
00:40:14,180 --> 00:40:16,280
If things needs to be in core,
it should be in core.

764
00:40:16,280 --> 00:40:18,600
But not everything should be in
core we know autofailover is

765
00:40:18,600 --> 00:40:23,140
still not in core right Patroni
right but Patroni Patroni I just

766
00:40:23,140 --> 00:40:28,700
asked Kukushkin did has he considered
automation of post resume

767
00:40:28,700 --> 00:40:31,240
and switch over without downtime
this is because this is what

768
00:40:31,240 --> 00:40:35,960
people want and expect for highly
automated system he said no

769
00:40:36,280 --> 00:40:41,900
and I started making joke because
I said Patroni is outside of

770
00:40:42,040 --> 00:40:46,920
Postgres because it's not the job
of Postgres to do HA or to

771
00:40:46,920 --> 00:40:51,780
failover and now I expect you Patroni
maintainer to tell me that

772
00:40:51,940 --> 00:40:55,580
automatic switcher 0 downtime switcher
is no job for Patroni

773
00:40:55,580 --> 00:40:57,940
so I need to implement another
layer on top of it, right?

774
00:40:57,940 --> 00:40:59,620
This is what actually we do already.

775
00:41:00,060 --> 00:41:03,200
For 0 downtime upgrades, we do
this.

776
00:41:03,200 --> 00:41:06,900
And by the way, I can share this
already on our consulting page,

777
00:41:06,900 --> 00:41:10,260
we mentioned GitLab, but also Supabase
and also Gadget.dev.

778
00:41:11,400 --> 00:41:16,960
These companies took tech from
us And I have official quotes

779
00:41:16,960 --> 00:41:20,420
from their managers, so I can share
this news.

780
00:41:20,420 --> 00:41:25,360
I think we developed really great
automation for 0 downtime upgrades,

781
00:41:25,640 --> 00:41:31,080
which are not only 0 downtime,
but also, of course, 0 data loss

782
00:41:31,080 --> 00:41:31,740
and reversible.

783
00:41:32,200 --> 00:41:35,500
And reversible without data loss
as well.

784
00:41:35,500 --> 00:41:39,220
So these are the perfect properties,
but it requires a lot of

785
00:41:39,440 --> 00:41:39,940
orchestration.

786
00:41:40,520 --> 00:41:44,640
Fortunately, since Postgres 17,
things are improving and less

787
00:41:44,820 --> 00:41:46,160
pieces need to be automated.

788
00:41:46,400 --> 00:41:51,180
So let me go back and explain my
humble, finally, and my vision

789
00:41:51,180 --> 00:41:52,580
of what I have right now.

790
00:41:53,000 --> 00:41:57,180
I realize that boring stuff needs
to have much higher level of

791
00:41:57,180 --> 00:41:57,680
automation.

792
00:41:57,900 --> 00:41:58,780
This is 1.

793
00:41:58,860 --> 00:42:02,160
Second, I realized PostgreSQL CI
team, this is exactly what we

794
00:42:02,160 --> 00:42:07,100
are doing because with consulting
people bring us these topics.

795
00:42:08,000 --> 00:42:13,840
And I realized also that in every
area if we think about automation

796
00:42:13,940 --> 00:42:17,680
of feature we can apply simplified
approach for classification.

797
00:42:18,480 --> 00:42:25,780
So if every single step must be
executed manually, like CLI call,

798
00:42:25,960 --> 00:42:32,280
like pg_dump call or something,
pg_upgrade call, or SQL snippets,

799
00:42:32,580 --> 00:42:36,920
if they are to be executed manually
by engineer, this is manual.

800
00:42:38,400 --> 00:42:42,260
If there are bigger pieces, which,
like, say, whole procedure

801
00:42:42,260 --> 00:42:48,220
consists of 2 or 3 big pieces, and they are combined, and engineer

802
00:42:48,220 --> 00:42:51,520
only makes choices to proceed or not between pieces.

803
00:42:51,620 --> 00:42:55,280
For example, our major upgrade consists of 2 huge pieces.

804
00:42:57,180 --> 00:43:01,660
Physical to logical with upgrade, they are bundled due to some

805
00:43:02,240 --> 00:43:02,960
specific reasons.

806
00:43:03,400 --> 00:43:08,300
And switchover, 2 big steps, and inside them there is a high

807
00:43:08,300 --> 00:43:09,120
level of automation.

808
00:43:09,340 --> 00:43:12,480
You just call playbook, it executes, right?

809
00:43:13,180 --> 00:43:16,400
In this case, it's already can be considered level 1, say.

810
00:43:16,420 --> 00:43:19,320
Say level 1, or like 2 maybe, I don't know.

811
00:43:19,460 --> 00:43:26,440
And then if we can fully relax and go to passenger seat and just

812
00:43:26,440 --> 00:43:29,720
say, okay, I approve that, that you need to do everything, but

813
00:43:29,720 --> 00:43:30,720
you will do it yourself.

814
00:43:30,720 --> 00:43:33,840
I mean, Postgres or additional system will do everything yourself.

815
00:43:34,200 --> 00:43:36,420
Like full major upgrade with switchover.

816
00:43:36,960 --> 00:43:41,660
In this case, like say at level 2 and then, and you're in passage,

817
00:43:41,660 --> 00:43:44,640
but there are limitations if it will encounter some problem,

818
00:43:45,240 --> 00:43:47,180
it will stop, revert, postpone.

819
00:43:47,860 --> 00:43:49,400
And you need to approve things.

820
00:43:50,140 --> 00:43:54,320
Highest level, if this feature, like if system decides, oh, it's

821
00:43:54,320 --> 00:43:58,100
time to upgrade, and then it schedules it, like, oh, we have

822
00:43:58,100 --> 00:44:02,220
low activity time on the weekend, and you even, like, you are

823
00:44:02,220 --> 00:44:05,740
notified about it, You probably can block it, but it moves on

824
00:44:05,740 --> 00:44:06,240
itself.

825
00:44:06,340 --> 00:44:08,080
This is the highest level of automation.

826
00:44:08,420 --> 00:44:13,580
And for back to remixing, which I chose this as lowest hanging

827
00:44:13,580 --> 00:44:14,080
fruit.

828
00:44:14,240 --> 00:44:15,340
Everyone needs it.

829
00:44:15,340 --> 00:44:19,820
Nobody in my own community, maybe after our episode, people from

830
00:44:19,820 --> 00:44:23,160
RDS, CloudSQL, I know some of them are listening to us, will

831
00:44:23,160 --> 00:44:24,600
rush into implementing this.

832
00:44:24,840 --> 00:44:28,080
I just see everyone needs it, nobody has it in terms of managed

833
00:44:28,080 --> 00:44:29,720
Postgres, nobody offers it.

834
00:44:30,060 --> 00:44:35,540
So I am highest level automation for pg_index_pilot right away.

835
00:44:35,540 --> 00:44:37,360
So it will decide when to re-index.

836
00:44:37,360 --> 00:44:38,680
It will re-index.

837
00:44:38,680 --> 00:44:42,240
It will control activity level so not to saturate disks, and

838
00:44:42,240 --> 00:44:43,180
so on, and so on.

839
00:44:43,180 --> 00:44:44,340
You can check roadmap.

840
00:44:44,440 --> 00:44:47,080
I explained it in readme of this project.

841
00:44:47,080 --> 00:44:48,300
And this is open source.

842
00:44:48,660 --> 00:44:51,820
Because I actually truly believe at some point it won't be needed,

843
00:44:51,820 --> 00:44:52,460
this project.

844
00:44:53,420 --> 00:44:57,320
If Andrey's idea will work for merge, maybe, I don't know.

845
00:44:57,340 --> 00:44:58,280
This is a dream, right?

846
00:44:58,280 --> 00:44:59,660
So forget about index bloat.

847
00:45:00,960 --> 00:45:01,460
Right?

848
00:45:01,740 --> 00:45:02,240
Yeah.

849
00:45:02,620 --> 00:45:04,340
I guess it will be super hard.

850
00:45:04,860 --> 00:45:08,540
I tried to research why it hasn't been done and I didn't see

851
00:45:08,540 --> 00:45:09,040
why.

852
00:45:10,040 --> 00:45:11,020
Maybe it's scary.

853
00:45:11,880 --> 00:45:15,760
This is basics, like foundation
of whole Postgres, of any cluster,

854
00:45:15,760 --> 00:45:16,080
right?

855
00:45:16,080 --> 00:45:16,580
B-tree.

856
00:45:16,840 --> 00:45:18,840
Michael: Yeah, well, I guess when
would it be done?

857
00:45:18,840 --> 00:45:21,400
Would it be done by a Background
worker like autovacuum?

858
00:45:21,740 --> 00:45:25,760
Or would it be like, at what stage
does it make sense to do?

859
00:45:25,760 --> 00:45:27,440
Nikolay: Well, it should be synchronous.

860
00:45:27,900 --> 00:45:29,760
Oh, I know, this is a good question.

861
00:45:29,760 --> 00:45:31,800
I'm going to ask Andrei about this.

862
00:45:31,940 --> 00:45:32,040
Yeah.

863
00:45:32,040 --> 00:45:33,300
Because split is synchronous.

864
00:45:34,060 --> 00:45:34,440
Michael: Yeah.

865
00:45:34,440 --> 00:45:36,900
I also think the thing you brought
up just at the end there is

866
00:45:36,900 --> 00:45:37,440
super interesting.

867
00:45:37,440 --> 00:45:40,240
Like, controlling the rate to not
saturate disks.

868
00:45:40,240 --> 00:45:43,380
Like, this is an interesting thing
with tradeoffs again.

869
00:45:43,380 --> 00:45:47,820
Like, if you're on Aurora Postgres
and you're paying some amount

870
00:45:48,040 --> 00:45:52,980
per IO, like, for you probably
don't want your indexes rebuilt

871
00:45:53,140 --> 00:45:54,320
constantly all the time.

872
00:45:54,320 --> 00:45:56,900
What you really want is to fix
the root cause.

873
00:45:57,340 --> 00:45:59,100
It's probably an application issue.

874
00:45:59,100 --> 00:46:02,660
Why are you updating the same Row
thousands of times a second.

875
00:46:03,100 --> 00:46:04,700
Nikolay: What's the root cause of

876
00:46:04,700 --> 00:46:05,420
the bloat?

877
00:46:06,340 --> 00:46:07,980
Michael: So, we just had a

878
00:46:08,520 --> 00:46:11,980
Nikolay: case of updating multiple
times a queue like workload.

879
00:46:12,100 --> 00:46:13,160
This can be identified.

880
00:46:13,520 --> 00:46:14,180
Yes, yes.

881
00:46:14,200 --> 00:46:14,700
Michael: Yeah.

882
00:46:14,760 --> 00:46:17,800
So, I'm excited to see what you
build.

883
00:46:17,900 --> 00:46:20,740
I think you're right that more
automation in Postgres would be

884
00:46:20,740 --> 00:46:23,400
good, more automation in extensions
around Postgres would be

885
00:46:23,400 --> 00:46:23,860
good.

886
00:46:23,860 --> 00:46:27,100
But a lot of the issues I see are
kind of application source

887
00:46:27,560 --> 00:46:28,060
issues.

888
00:46:28,180 --> 00:46:32,480
And even if we make Postgres completely
self-driving, there is

889
00:46:32,480 --> 00:46:35,740
still this kind of application
level issue that will be...

890
00:46:35,740 --> 00:46:39,860
Nikolay: Mistake there can put
all your efforts to the ground,

891
00:46:39,860 --> 00:46:40,360
yeah.

892
00:46:40,520 --> 00:46:41,120
Michael: Yeah, exactly.

893
00:46:41,120 --> 00:46:41,680
I agree.

894
00:46:42,200 --> 00:46:47,200
Nikolay: That's why in our, like,
I envision, I identify 25 areas

895
00:46:47,200 --> 00:46:48,300
in my blog post.

896
00:46:48,360 --> 00:46:48,580
Yeah.

897
00:46:48,580 --> 00:46:50,280
Office during our launch week.

898
00:46:50,800 --> 00:46:54,480
Maybe we will adjust those areas,
of course, like it's not final

899
00:46:54,480 --> 00:46:54,980
list.

900
00:46:55,200 --> 00:46:57,600
And we are targeting 3 right now.

901
00:46:58,180 --> 00:47:02,300
Like I hope we will expand this list to 5 to 6 next year.

902
00:47:02,640 --> 00:47:06,260
Once we have quite good building blocks, we'll think about whole

903
00:47:06,260 --> 00:47:06,760
system.

904
00:47:06,880 --> 00:47:10,020
But the central part of all these building blocks is new monitoring

905
00:47:10,080 --> 00:47:14,340
system, which is good both for AI and humans.

906
00:47:14,680 --> 00:47:16,800
And This is what we're already actively building.

907
00:47:16,800 --> 00:47:20,040
We replaced all our observability tooling used in consulting

908
00:47:20,080 --> 00:47:23,300
with new monitoring system, which is called Postgres AI monitoring.

909
00:47:23,300 --> 00:47:24,700
We talked about it separately.

910
00:47:26,400 --> 00:47:32,880
This is going to be the source of insights, why things are wrong.

911
00:47:33,720 --> 00:47:38,040
We cannot self-derive application yet, Although in some cases

912
00:47:38,440 --> 00:47:41,760
might be so, because if for example, Supabase, they control

913
00:47:42,560 --> 00:47:45,600
not only Postgres, but also REST API level, right?

914
00:47:45,600 --> 00:47:50,260
So some things might be down there, but in general case, we cannot.

915
00:47:50,340 --> 00:47:53,860
So in this case, we need just to advise and so on, but eventually

916
00:47:55,160 --> 00:47:59,600
things like serverless, like with cell workloads or so, I see

917
00:47:59,600 --> 00:48:02,740
that can be together with database eventually.

918
00:48:03,160 --> 00:48:06,400
In this case, we can discuss full self-driving something.

919
00:48:06,940 --> 00:48:07,440
Right.

920
00:48:07,540 --> 00:48:09,860
But it's, we are very far from that, I think.

921
00:48:09,960 --> 00:48:10,380
Yeah.

922
00:48:10,380 --> 00:48:10,680
Yeah.

923
00:48:10,680 --> 00:48:12,460
And, and we have limited resources.

924
00:48:12,560 --> 00:48:17,320
I wanted to admit that I'm targeting very narrow topics right

925
00:48:17,320 --> 00:48:17,820
now.

926
00:48:17,860 --> 00:48:24,640
So very boring but really doable and We see demand Like things

927
00:48:24,640 --> 00:48:26,740
like partitioning help with partitioning.

928
00:48:26,820 --> 00:48:31,880
It's nightmare to do it for arbitrary engineer We say Oh declarative

929
00:48:31,880 --> 00:48:35,280
partitioning, but for example partitions are not created automatically

930
00:48:35,540 --> 00:48:38,260
Okay, there's pg_partman, but you need to combine this pieces

931
00:48:38,260 --> 00:48:40,820
something like The level of automation there is terrible.

932
00:48:41,580 --> 00:48:43,820
Michael: Yeah, but these are the kind of things I'd love to like

933
00:48:43,820 --> 00:48:44,880
a couple of things.

934
00:48:44,960 --> 00:48:47,220
I'd love to see more of this go to core.

935
00:48:47,960 --> 00:48:51,340
I think there has been improvements in partitioning core, many

936
00:48:51,340 --> 00:48:53,340
over the last few years and continuing.

937
00:48:54,280 --> 00:49:00,060
And the other thing is, I think there's a lot of focus on fewer

938
00:49:00,060 --> 00:49:00,460
people.

939
00:49:00,460 --> 00:49:04,900
With this automation, a lot of the focus is on we can do more

940
00:49:04,900 --> 00:49:06,340
with less humans.

941
00:49:06,740 --> 00:49:10,440
And I'd love to see more effort into what would happen if we

942
00:49:10,440 --> 00:49:13,440
didn't try and reduce the humans but instead try to increase

943
00:49:14,060 --> 00:49:14,740
the reliability.

944
00:49:15,480 --> 00:49:16,660
You know what I mean?

945
00:49:17,980 --> 00:49:22,460
And I think there's a lot of talk about how do

946
00:49:22,460 --> 00:49:23,740
Nikolay: we cut costs.

947
00:49:24,960 --> 00:49:28,700
We have already I think 5 companies who are very fast growing

948
00:49:28,700 --> 00:49:29,600
AI companies.

949
00:49:29,800 --> 00:49:31,060
Very well known already.

950
00:49:31,720 --> 00:49:34,620
And I see a very different approach to Postgres.

951
00:49:34,700 --> 00:49:37,640
For them it's a natural choice,
like let's start with it, but

952
00:49:37,640 --> 00:49:41,140
then they, oh, we need to partition
terabyte size tables.

953
00:49:42,620 --> 00:49:45,040
Let's estimate work, oh, that much
work?

954
00:49:45,140 --> 00:49:47,940
Okay, maybe we should migrate to
a different database system.

955
00:49:49,340 --> 00:49:51,020
They are moving really fast.

956
00:49:51,180 --> 00:49:53,800
And they are not attached to Postgres,
basically.

957
00:49:53,800 --> 00:49:55,320
So I'm scared about Postgres.

958
00:49:55,320 --> 00:49:56,960
That's why I'm saying automation.

959
00:49:57,800 --> 00:50:00,080
There should be a massive shift
right now.

960
00:50:01,100 --> 00:50:03,420
And resources of my team is not
enough.

961
00:50:04,060 --> 00:50:06,160
So that's why I'm talking about
it.

962
00:50:06,240 --> 00:50:08,560
I'm going to put whole focus on
it.

963
00:50:08,560 --> 00:50:13,820
During consulting, we also choose
only paths that are fully automated,

964
00:50:14,060 --> 00:50:15,640
highly automated, fully automated.

965
00:50:15,760 --> 00:50:20,580
And I'm looking for some use cases,
more use cases, more maybe

966
00:50:20,580 --> 00:50:25,280
partners who think similarly, right?

967
00:50:25,880 --> 00:50:29,540
This is a call for Postgres ecosystem,
like it's not enough right

968
00:50:29,540 --> 00:50:32,133
now, like we're not...

969
00:50:32,133 --> 00:50:33,960
Postgres might lose this game again.

970
00:50:33,960 --> 00:50:38,860
Postgres won multiple challenges,
like object-oriented, NoSQL.

971
00:50:40,140 --> 00:50:45,900
But AI, everyone thinks about AI
only about like pgvector or

972
00:50:45,900 --> 00:50:46,640
storing vectors.

973
00:50:46,640 --> 00:50:47,360
It's not.

974
00:50:48,720 --> 00:50:50,060
Not everyone needs vectors.

975
00:50:50,880 --> 00:50:54,840
People build some app, it needs
database, maybe with vectors,

976
00:50:54,840 --> 00:51:00,560
maybe without vectors, but what
they expect is higher level automation

977
00:51:00,600 --> 00:51:01,500
scale easier.

978
00:51:02,380 --> 00:51:08,160
So I'm super happy to see new innovations
in the area of sharding,

979
00:51:09,560 --> 00:51:10,060
right?

980
00:51:10,600 --> 00:51:10,840
Yeah,

981
00:51:10,840 --> 00:51:13,740
Michael: I think that's actually
more, I do think that's more

982
00:51:13,740 --> 00:51:15,700
important, partly for just the
marketing story.

983
00:51:15,700 --> 00:51:16,280
Nikolay: Then partitioning?

984
00:51:17,240 --> 00:51:18,140
Michael: For those, yeah.

985
00:51:18,520 --> 00:51:21,800
Because I think we talked, like,
if you think about partitioning

986
00:51:21,980 --> 00:51:26,820
if you I think for example when
we spoke to through when we had

987
00:51:26,820 --> 00:51:30,520
our hundredth episode and spoke
to Notion, Figma and Adyen.

988
00:51:31,500 --> 00:51:32,000
Yeah.

989
00:51:32,140 --> 00:51:34,980
Notion skipped partitioning and
went straight to sharding.

990
00:51:35,540 --> 00:51:39,780
They decided that with all of the
partitioning downsides, and

991
00:51:39,780 --> 00:51:43,640
then there are quite a few, quite
a few limitations, for their

992
00:51:44,020 --> 00:51:49,000
multi-tenant architecture it actually
made more sense to shard

993
00:51:49,060 --> 00:51:50,200
instead of partition.

994
00:51:50,740 --> 00:51:54,140
That felt like it wasn't 1 of these
things where they were moving

995
00:51:54,140 --> 00:51:55,940
super quick and made a rash decision.

996
00:51:56,080 --> 00:51:59,000
It felt like a really considered
engineering decision.

997
00:51:59,120 --> 00:52:01,100
And I actually think it was the
right call.

998
00:52:01,100 --> 00:52:06,860
And I wouldn't be surprised if
more of these companies are coming

999
00:52:06,860 --> 00:52:12,780
in and building fairly seamless
sharding that doesn't add any

1000
00:52:12,780 --> 00:52:16,460
complexity at the application level,
I could see people tempted

1001
00:52:16,480 --> 00:52:19,120
into that, even if it wasn't for
good engineering reasons, just

1002
00:52:19,120 --> 00:52:22,080
for the marketing reason of, I
don't have to think about scaling

1003
00:52:22,080 --> 00:52:24,520
this, and I don't have to deal
with tailback tables.

1004
00:52:24,620 --> 00:52:27,660
Nikolay: You know this normal distribution
meme, right?

1005
00:52:27,980 --> 00:52:28,940
Yeah, yeah, yeah.

1006
00:52:28,940 --> 00:52:32,720
Usually it's like unexpected on
the right side, where it's expected.

1007
00:52:32,800 --> 00:52:37,680
So I don't know where, in which
camp I am.

1008
00:52:38,000 --> 00:52:42,440
I was thinking Postgres 1 node
is cool enough, then I was thinking

1009
00:52:42,440 --> 00:52:44,700
it's not cool enough because of
performance cliffs.

1010
00:52:44,920 --> 00:52:48,920
Now I'm thinking maybe it depends
because in some cases like

1011
00:52:48,920 --> 00:52:52,080
yeah it's much safer to avoid all
performance cliffs, not just

1012
00:52:52,080 --> 00:52:57,440
allowing like 100 plus thousand
transactions per second on 1

1013
00:52:57,440 --> 00:52:57,780
node.

1014
00:52:57,780 --> 00:53:00,420
You just, and how it's called?

1015
00:53:00,600 --> 00:53:01,460
Resiliency, right?

1016
00:53:01,460 --> 00:53:05,340
If 1 node is down, only part of,
like, it's just 1 shard, right?

1017
00:53:05,500 --> 00:53:09,440
Maybe yes, but at the same time,
it's so cool to see projects

1018
00:53:09,480 --> 00:53:11,760
which require only 1 node.

1019
00:53:12,080 --> 00:53:13,020
They are isolated.

1020
00:53:13,140 --> 00:53:17,340
They don't need a whole, like,
bunch of shards and clusters,

1021
00:53:18,080 --> 00:53:22,600
cluster of clusters, or cluster
term is heavily overloaded, right?

1022
00:53:22,900 --> 00:53:26,820
Yeah, and I think, oh, that's really
the power to have just 1

1023
00:53:26,820 --> 00:53:29,200
single node, sometimes without
replicas.

1024
00:53:29,380 --> 00:53:33,340
I see projects without replicas,
Because cloud times change,

1025
00:53:33,340 --> 00:53:33,840
right?

1026
00:53:34,220 --> 00:53:39,300
And I think I'm open, you know,
like I know my perception over

1027
00:53:39,300 --> 00:53:41,660
years changes, sometimes it's a
pendulum.

1028
00:53:42,600 --> 00:53:48,260
So it depends, you know, like regular
answer, normal answer from

1029
00:53:48,260 --> 00:53:49,540
consultants, it depends.

1030
00:53:50,380 --> 00:53:50,680
Yeah.

1031
00:53:50,680 --> 00:53:54,140
But sharding, it's really great
to see that it's coming from

1032
00:53:54,140 --> 00:53:56,020
multiple teams, there will be competition.

1033
00:53:56,880 --> 00:54:01,720
So yeah, the here, the future looks
bright, but who will be helping

1034
00:54:01,920 --> 00:54:04,940
to choose proper schema for sharding?

1035
00:54:06,240 --> 00:54:06,600
Michael: Well,

1036
00:54:06,600 --> 00:54:06,940
Nikolay: yeah.

1037
00:54:06,940 --> 00:54:08,760
And rebalance properly and normally.

1038
00:54:08,760 --> 00:54:13,180
Well, yeah, so I think they who
build this think about this as

1039
00:54:13,180 --> 00:54:16,720
well, automation, and the problem,
like, to choose time for rebalancing

1040
00:54:16,960 --> 00:54:17,920
and fully automated?

1041
00:54:18,660 --> 00:54:21,820
Michael: Well, we talked to Sugu,
he said it's inevitable that

1042
00:54:21,820 --> 00:54:24,480
you're gonna have to change your
sharding scheme at some point.

1043
00:54:24,520 --> 00:54:27,760
It's just designing for that upfront
seems really important.

1044
00:54:27,900 --> 00:54:29,200
Vitesse handles it.

1045
00:54:29,200 --> 00:54:32,460
So yeah, Interesting times ahead.

1046
00:54:33,160 --> 00:54:36,360
I'm a bit worried about the complexity
of these systems.

1047
00:54:36,820 --> 00:54:37,320
Personally,

1048
00:54:37,720 --> 00:54:38,970
Nikolay: I quite like Let's start

1049
00:54:38,970 --> 00:54:39,520
Michael: with simple things.

1050
00:54:39,520 --> 00:54:39,780
Yeah.

1051
00:54:39,780 --> 00:54:43,780
I quite well, starting simple, but also I like the idea of if

1052
00:54:43,780 --> 00:54:47,420
I'm a Postgres user, I can still understand the system, like

1053
00:54:47,420 --> 00:54:50,500
roughly what's going on, even if I'm a full stack developer,

1054
00:54:50,500 --> 00:54:54,140
even if I do the front end, the back end, like, I like, I know

1055
00:54:54,140 --> 00:54:55,680
it's already very complex.

1056
00:54:55,680 --> 00:54:58,740
I know there's already a lot of internals that you kind of need

1057
00:54:58,740 --> 00:55:00,780
to know about to make a performance system.

1058
00:55:01,220 --> 00:55:03,800
But I hope we can hold on to that for a while.

1059
00:55:03,800 --> 00:55:04,300
Well,

1060
00:55:05,060 --> 00:55:09,800
Nikolay: I truly believe that what my team does is going to help

1061
00:55:09,800 --> 00:55:12,480
because we observe many problems.

1062
00:55:13,080 --> 00:55:17,380
And every time I'm saying guys like we need to write down some

1063
00:55:17,380 --> 00:55:23,300
how to and So show with experiment so users understand what's

1064
00:55:23,300 --> 00:55:23,800
happening.

1065
00:55:24,060 --> 00:55:26,100
Write some how-to for next time.

1066
00:55:26,380 --> 00:55:29,860
And when we're writing how-tos recently, we write it both for

1067
00:55:29,860 --> 00:55:30,920
humans and AI.

1068
00:55:31,240 --> 00:55:35,240
So next time some RCA is happening, root cause analysis is happening.

1069
00:55:36,060 --> 00:55:40,440
If you have our how-tos injected to your Cursor, for example,

1070
00:55:40,440 --> 00:55:46,160
it's going to take into account some situations which are written

1071
00:55:46,500 --> 00:55:49,340
and how to reproduce them, how to troubleshoot them, right?

1072
00:55:49,400 --> 00:55:52,200
So I think something else is coming.

1073
00:55:52,200 --> 00:55:58,260
I'm not going to spoil it, but RCA and troubleshooting is 1 thing

1074
00:55:58,260 --> 00:55:59,940
we will attack early.

1075
00:56:00,040 --> 00:56:07,540
We are preparing pieces for this, you know, and 1 of key indicators

1076
00:56:07,540 --> 00:56:11,360
of success here is that non-experts understand what's happening.

1077
00:56:12,440 --> 00:56:13,780
You know, because...

1078
00:56:13,780 --> 00:56:15,100
Michael: Yeah, well, that's what I've...

1079
00:56:15,100 --> 00:56:18,120
That's the area that I specialize in.

1080
00:56:18,120 --> 00:56:22,020
You know, I'm actually betting a little bit on humans staying

1081
00:56:22,020 --> 00:56:24,960
in the loop for quite a long time and that there will always,

1082
00:56:24,960 --> 00:56:28,600
well not always, but for a long time, there'll be categories

1083
00:56:28,680 --> 00:56:32,300
of issue that we still need somebody to look into.

1084
00:56:32,780 --> 00:56:38,100
And I kind of feel like the level of Postgres knowledge, the

1085
00:56:38,100 --> 00:56:40,520
median level of Postgres knowledge for people having to look

1086
00:56:40,520 --> 00:56:42,940
into those issues is probably going to go down.

1087
00:56:43,860 --> 00:56:47,580
Based on all the trends that you're talking about, It depends.

1088
00:56:48,480 --> 00:56:51,600
If we only have a few experts that are shared between lots of

1089
00:56:51,600 --> 00:56:53,580
companies, maybe that's not true.

1090
00:56:53,760 --> 00:56:58,180
But if we do have a lot of individuals starting with web coding

1091
00:56:58,180 --> 00:57:02,280
or starting with like doing a full stack kind of single person

1092
00:57:02,280 --> 00:57:06,100
companies running the whole product start to finish.

1093
00:57:07,500 --> 00:57:10,580
Those guys, like, they have to know a lot about a lot of things.

1094
00:57:10,840 --> 00:57:12,040
They can't know that

1095
00:57:12,040 --> 00:57:12,540
Nikolay: much.

1096
00:57:15,720 --> 00:57:16,700
Have you seen numbers, Supabase shared, how many clusters

1097
00:57:16,720 --> 00:57:18,460
were registered and how fast it is.

1098
00:57:18,460 --> 00:57:18,960
Yeah.

1099
00:57:19,940 --> 00:57:23,200
Can you imagine how like, yeah, or average knowledge of Postgres

1100
00:57:23,200 --> 00:57:23,480
there.

1101
00:57:23,480 --> 00:57:23,940
Yeah.

1102
00:57:23,940 --> 00:57:29,180
But yeah, but the idea is I, I,
my vision is that we, we collect,

1103
00:57:29,820 --> 00:57:33,020
knowledge pieces, we experiment
with, automatic experimentation

1104
00:57:33,080 --> 00:57:33,420
and so on.

1105
00:57:33,420 --> 00:57:37,880
But then obviously this is what
I see with our customers some

1106
00:57:37,880 --> 00:57:43,340
human is needed to explain properly
to other humans to answer

1107
00:57:43,340 --> 00:57:47,220
questions properly you know to
build trust and confidence and

1108
00:57:47,220 --> 00:57:47,860
so on.

1109
00:57:48,180 --> 00:57:50,940
Michael: Yeah, but my question,
I guess the question then is

1110
00:57:50,940 --> 00:57:52,060
where do the tools live?

1111
00:57:52,060 --> 00:57:55,180
Like, can the end user use a tool
to get help?

1112
00:57:55,180 --> 00:57:58,980
Like, or does the Supabase team
use a tool to get help?

1113
00:57:58,980 --> 00:58:02,020
Or does the consultant that the
Supabase team employ?

1114
00:58:04,080 --> 00:58:05,900
Nikolay: My answer is, who's the
tool for?

1115
00:58:05,900 --> 00:58:09,400
Michael: And I know I'm just saying
that, like, it depends who

1116
00:58:09,400 --> 00:58:12,700
you count as the user in terms
of how much their Postgres knowledge

1117
00:58:12,700 --> 00:58:12,740
is.

1118
00:58:12,740 --> 00:58:14,340
I'm talking about that end user.

1119
00:58:14,340 --> 00:58:19,040
Nikolay: Yeah, for example, with
DBLab we already went this path.

1120
00:58:19,040 --> 00:58:23,760
We moved from a couple of guys
answering all the details for

1121
00:58:23,760 --> 00:58:27,180
Backend engineers have in terms
of how plan works and what to

1122
00:58:27,180 --> 00:58:29,000
do about it, which index to create.

1123
00:58:29,220 --> 00:58:34,580
We now have DBLab Backend engineers
experiment themselves.

1124
00:58:35,220 --> 00:58:40,440
And if something is unclear, only
then they call an expert for

1125
00:58:40,440 --> 00:58:40,940
help.

1126
00:58:41,200 --> 00:58:41,520
Right.

1127
00:58:41,520 --> 00:58:45,100
But like 90% of the questions answered
by Backend engineers without

1128
00:58:45,140 --> 00:58:49,080
involving expert, Postgres experts,
you know, and expertise in

1129
00:58:49,080 --> 00:58:51,200
Backend engineering minds grow,
grows as well.

1130
00:58:51,200 --> 00:58:56,460
I I'm just thinking this approach
we had for query optimization

1131
00:58:56,920 --> 00:59:00,340
can be applied at ground of schema
for many other areas.

1132
00:59:00,340 --> 00:59:00,840
Yeah.

1133
00:59:01,980 --> 00:59:02,380
Michael: All right.

1134
00:59:02,380 --> 00:59:03,560
Probably enough for today.

1135
00:59:03,620 --> 00:59:04,440
Nikolay: Thank you so much.

1136
00:59:04,440 --> 00:59:08,640
We went much deeper in specific
areas than I expected and I enjoyed

1137
00:59:08,640 --> 00:59:09,240
it a lot.

1138
00:59:09,240 --> 00:59:10,120
Thank you so much.

1139
00:59:10,400 --> 00:59:10,900
Michael: Nice.