1
0:0:0,06 --> 0:0:2,72
Nik: Hello, hello, this is Postgres.FM,
as usual.

2
0:0:2,8 --> 0:0:6,3399997
I'm Nik, PostgresAI, and as usual
with me Michael, pgMustard.

3
0:0:6,68 --> 0:0:7,46
Hi, Michael.

4
0:0:8,26 --> 0:0:9,059999
Michael: Hi, Nik.

5
0:0:9,559999 --> 0:0:11,42
Nik: We have a great guest today.

6
0:0:11,719999 --> 0:0:13,16
Did we have you already?

7
0:0:13,48 --> 0:0:15,04
Álvaro from Ongres.

8
0:0:15,72 --> 0:0:16,220001
Álvaro: Hello.

9
0:0:16,46 --> 0:0:18,28
No, it's my 1st time here.

10
0:0:18,54 --> 0:0:20,64
At least as a speaker, not as a
listener.

11
0:0:21,7 --> 0:0:22,54
Nik: Thank you for coming.

12
0:0:22,54 --> 0:0:24,439999
I know you're a very active listener.

13
0:0:25,08 --> 0:0:30,02
We definitely have you on Postgres
TV channel, on YouTube channel,

14
0:0:30,02 --> 0:0:31,2
talking about licenses.

15
0:0:31,22 --> 0:0:32,46
That's why I'm confused.

16
0:0:32,48 --> 0:0:33,3
Do you remember?

17
0:0:34,06 --> 0:0:34,56
Álvaro: Yeah.

18
0:0:35,08 --> 0:0:39,0
Nik: And I know we will touch a little
bit licenses today, just knowing

19
0:0:39,0 --> 0:0:39,64
the topic.

20
0:0:39,760002 --> 0:0:41,02
Yeah, thank you for coming.

21
0:0:41,4 --> 0:0:45,4
I'm especially excited that today's
topic will connect so many

22
0:0:45,4 --> 0:0:46,9
dots, right?

23
0:0:47,16 --> 0:0:52,12
Performance, sharding, Kubernetes
operator, what your company

24
0:0:52,12 --> 0:0:52,84
is doing.

25
0:0:53,08 --> 0:0:55,68
It's so open source and true open
source, right?

26
0:0:55,68 --> 0:1:0,1
I'm very excited, looking forward
to dive into this chat because

27
0:1:0,1 --> 0:1:4,62
many topics I'm very interested
in myself and my company also,

28
0:1:5,32 --> 0:1:6,72
they are connected today.

29
0:1:7,72 --> 0:1:9,92
Álvaro: No, thank you very much
for having me.

30
0:1:9,92 --> 0:1:14,22
It's been a lot of episodes listening
with you over the year

31
0:1:14,34 --> 0:1:16,96
and now being present here, it's
very exciting for me.

32
0:1:16,96 --> 0:1:17,82
So thank you.

33
0:1:18,54 --> 0:1:22,58
Nik: And today we will talk about
benchmark you haven't published

34
0:1:22,58 --> 0:1:23,08
yet.

35
0:1:23,479996 --> 0:1:27,68
So thank you for coming with results
being still prepared.

36
0:1:27,84 --> 0:1:31,36
That's great because I know we
scheduled this episode before

37
0:1:31,36 --> 0:1:33,4
I think you decided to work on
it.

38
0:1:33,4 --> 0:1:34,2
So this is like...

39
0:1:34,2 --> 0:1:35,46
Álvaro: Right, that's correct.

40
0:1:36,98 --> 0:1:38,56
Nik: Action happening in real time.

41
0:1:38,56 --> 0:1:39,3
That's great.

42
0:1:39,52 --> 0:1:44,14
And yeah, I received your pictures,
3 charts about benchmark

43
0:1:44,14 --> 0:1:47,72
data, benchmark results this morning,
so I'm catching up.

44
0:1:47,98 --> 0:1:49,46
Where should we start actually?

45
0:1:49,6 --> 0:1:52,44
Because this is a great topic,
right?

46
0:1:52,44 --> 0:1:55,18
So how Postgres can scale, what
numbers we can achieve.

47
0:1:55,58 --> 0:1:58,94
Maybe we should start mentioning
that Michael and I had an episode

48
0:1:58,94 --> 0:2:3,48
about our company benchmarks on
1 node, no sharding at all, just

49
0:2:3,48 --> 0:2:7,72
1 node, we achieved 4 million TPS
a couple of years ago.

50
0:2:7,72 --> 0:2:9,44
It was a couple of years ago, right?

51
0:2:9,9 --> 0:2:14,52
With SELECT only primary key
lookup just to see how it's not

52
0:2:14,54 --> 0:2:19,24
super huge compared to some other
database systems, which might

53
0:2:19,24 --> 0:2:22,48
be faster in specific workloads,
but it's great that general

54
0:2:22,48 --> 0:2:23,98
Postgres can achieve that.

55
0:2:24,48 --> 0:2:29,8
And you can give us numbers much
bigger because you involve sharding,

56
0:2:29,8 --> 0:2:30,3
right?

57
0:2:31,42 --> 0:2:32,2
Álvaro: That's correct.

58
0:2:32,38 --> 0:2:36,26
If you want, I can start a bit
with my thesis about sharding

59
0:2:36,26 --> 0:2:37,62
in general for Postgres.

60
0:2:38,94 --> 0:2:43,84
I think it's a field where we're
seeing a lot of innovation happening,

61
0:2:43,86 --> 0:2:44,86
and I love this.

62
0:2:45,22 --> 0:2:48,98
Innovation and competition are
great, are what drives us forward.

63
0:2:49,4 --> 0:2:53,74
Nik: Let me interrupt you with
plugging in episodes with Sugu,

64
0:2:53,74 --> 0:2:57,94
Multigres I think we had 1 or
2, and definitely we had 2 episodes

65
0:2:57,98 --> 0:2:59,68
with Lev PgDog.

66
0:2:59,76 --> 0:3:4,02
So There is an innovation starting
last year, which is great.

67
0:3:4,08 --> 0:3:5,78
And I know you have your own opinion.

68
0:3:6,58 --> 0:3:7,84
Apologies for interrupting.

69
0:3:8,08 --> 0:3:8,86
Let's continue.

70
0:3:9,72 --> 0:3:9,96
Álvaro: Yeah.

71
0:3:9,96 --> 0:3:10,2
Yeah.

72
0:3:10,2 --> 0:3:11,52
I do have my opinion.

73
0:3:11,98 --> 0:3:16,3
Basically databases, they take
a long time to mature.

74
0:3:16,64 --> 0:3:20,28
Databases need to be the places
where you can trust your data,

75
0:3:20,28 --> 0:3:22,7
that it's gonna be there, that
you're durable, and they obviously

76
0:3:22,72 --> 0:3:25,62
obey the rest of the properties
that you expect from a database.

77
0:3:26,14 --> 0:3:30,3
And typically new databases are
risky or tend to lag adoption

78
0:3:30,3 --> 0:3:31,58
because they need to mature.

79
0:3:32,08 --> 0:3:35,24
When you're sharding, my thoughts
keep being the same.

80
0:3:35,34 --> 0:3:39,24
You really need the mature and
trust solution, a boring, I would

81
0:3:39,24 --> 0:3:42,1
say, technology to run your shards
because now you don't have

82
0:3:42,1 --> 0:3:43,68
1 cluster, you have many clusters.

83
0:3:44,28 --> 0:3:47,72
Each shard, each portion of the
database, it's a cluster on its

84
0:3:47,72 --> 0:3:48,22
own.

85
0:3:48,42 --> 0:3:51,96
So the software that manages all
that needs to also be well-established,

86
0:3:52,28 --> 0:3:53,04
let's say.

87
0:3:53,68 --> 0:3:56,94
And again, in competition and innovation
are absolutely great

88
0:3:56,94 --> 0:3:58,24
and very, very welcome.

89
0:3:58,74 --> 0:4:1,8
But I believe that there's a great
technology out there, open

90
0:4:1,8 --> 0:4:5,6
source for sharding Postgres that's
been in the market for, I

91
0:4:5,6 --> 0:4:8,16
guess, like more than 10 years
called Citus.

92
0:4:9,24 --> 0:4:10,58
And Citus is just great.

93
0:4:10,76 --> 0:4:13,94
Might not be perfect, but it's
really good enough or more than

94
0:4:13,94 --> 0:4:14,68
that even.

95
0:4:15,14 --> 0:4:18,9
And a lot of people may have forgotten
about it or put it a little

96
0:4:18,9 --> 0:4:19,46
bit aside.

97
0:4:19,46 --> 0:4:23,86
Myself, the same several years
ago, until I started with thinking,

98
0:4:23,9 --> 0:4:27,46
OK, people need to shard Postgres
and people need to scale Postgres.

99
0:4:27,94 --> 0:4:32,56
And not only because of the performance
directly, because as

100
0:4:32,56 --> 0:4:37,22
you demonstrated, you can run very
high TPS on a single node.

101
0:4:37,38 --> 0:4:40,46
But sometimes it's also about the
blast radius.

102
0:4:40,92 --> 0:4:45,06
Like if you're running a SaaS shop
or an e-commerce and your

103
0:4:45,06 --> 0:4:47,82
database cluster goes down for
whatever reason.

104
0:4:47,9 --> 0:4:51,44
Sometimes it's a configuration
issue with the network, whatever

105
0:4:51,44 --> 0:4:52,16
it is, right?

106
0:4:52,16 --> 0:4:55,02
Your cluster is down, then your
whole thing is down.

107
0:4:55,58 --> 0:5:0,78
If you shard and you split your
database into, let's say 16 logical

108
0:5:0,84 --> 0:5:1,5
shards, right?

109
0:5:1,5 --> 0:5:2,22
Or workers.

110
0:5:2,68 --> 0:5:6,06
And then 1 goes down because of
a configuration error, then it's

111
0:5:6,06 --> 0:5:8,94
only a small percentage of your
users that are effectively down,

112
0:5:8,94 --> 0:5:11,56
which it's not good, but it's definitely
not 100%.

113
0:5:11,64 --> 0:5:14,18
Your PR is completely different
night and day, right?

114
0:5:14,18 --> 0:5:17,44
So a lot of people now need to
shard either because of performance

115
0:5:17,44 --> 0:5:20,42
reasons, because they cannot scale
the writes to a single primary,

116
0:5:20,82 --> 0:5:25,46
and or because of high availability
and blast radius reasons.

117
0:5:26,28 --> 0:5:28,88
And we always had Citus there.

118
0:5:29,38 --> 0:5:31,16
And Citus is fully featured.

119
0:5:32,02 --> 0:5:35,16
It has almost everything that you
need for sharding, for running

120
0:5:35,16 --> 0:5:38,34
complex queries that go down, they're
pushed down to the workers,

121
0:5:39,0 --> 0:5:42,48
and has the tools for rebalancing,
for dealing with hot shards,

122
0:5:42,74 --> 0:5:45,48
for almost all operational activities
that you need to perform

123
0:5:45,48 --> 0:5:47,86
on a sharded database and it just
works.

124
0:5:48,32 --> 0:5:52,44
I believe because their initial
focus was more OLAP style or

125
0:5:52,44 --> 0:5:57,24
mixed OLTP OLAP that people started
considering Citus only for

126
0:5:57,24 --> 0:6:0,8
that and it's definitely good fit
for that but it's also really

127
0:6:0,8 --> 0:6:1,7
good for OLTP.

128
0:6:3,06 --> 0:6:4,24
Nik: 2 points here.

129
0:6:4,24 --> 0:6:7,32
1st of all, you said we always
had.

130
0:6:7,54 --> 0:6:11,14
Let me disagree with you because
when I actually was looking

131
0:6:11,14 --> 0:6:13,62
into Citus, it was several years
ago.

132
0:6:13,96 --> 0:6:17,66
The Key features were not open
source.

133
0:6:17,8 --> 0:6:19,4
For example, online rebalancing.

134
0:6:19,94 --> 0:6:22,58
So we didn't have it in the open
source area.

135
0:6:23,26 --> 0:6:26,42
It was like you need to have a
contract with them to have online

136
0:6:26,42 --> 0:6:30,24
rebalancing, so it quite quickly
becomes out of scope if you

137
0:6:30,24 --> 0:6:33,02
want to rely on open source and
avoid vendor lock-in.

138
0:6:33,08 --> 0:6:34,1
Now it's different.

139
0:6:34,2 --> 0:6:36,24
It has changed a few years ago.

140
0:6:36,34 --> 0:6:39,88
All features after acquisition
by Microsoft, at some point all

141
0:6:39,88 --> 0:6:42,6
features went to open source, which
is great.

142
0:6:43,26 --> 0:6:45,26
But it's not like we always had,
right?

143
0:6:45,26 --> 0:6:46,78
Álvaro: Yeah, you're absolutely
right.

144
0:6:46,8 --> 0:6:47,8
And thanks for that.

145
0:6:47,8 --> 0:6:52,5
I think, if I'm not mistaken, Citus
version 11 was the 1st 1

146
0:6:52,5 --> 0:6:54,82
that included everything as open
source.

147
0:6:55,24 --> 0:6:57,32
And that was, I think, 2022.

148
0:6:57,64 --> 0:7:2,22
So 4 years ago, 4 years and 0.5
ago, that was released as fully

149
0:7:2,22 --> 0:7:3,58
open source and has been since
then.

150
0:7:3,58 --> 0:7:4,7
So yes, you're right.

151
0:7:4,74 --> 0:7:5,68
For this podcast,

152
0:7:6,22 --> 0:7:9,5
Nik: you're right also, because
this podcast exists for years.

153
0:7:9,72 --> 0:7:10,18
Exactly.

154
0:7:10,18 --> 0:7:12,62
So, good.

155
0:7:13,38 --> 0:7:16,52
And the 2nd point is OLAP versus
OLTP.

156
0:7:16,56 --> 0:7:21,34
My concern, like, last 4 years,
I think I didn't dive into Citus.

157
0:7:21,34 --> 0:7:25,96
I remember, diving means like I
run it, inspect myself, the benchmark

158
0:7:25,96 --> 0:7:26,44
and so on.

159
0:7:26,44 --> 0:7:29,26
I did it before we started podcast
actually.

160
0:7:29,26 --> 0:7:32,9
I remember I was reporting some
bugs And since then I didn't

161
0:7:32,9 --> 0:7:37,58
dive into it, but my concern is
we didn't see great examples

162
0:7:38,4 --> 0:7:43,82
of some well-known companies, SaaS
companies, for example, running

163
0:7:43,82 --> 0:7:44,32
it.

164
0:7:44,54 --> 0:7:48,76
Like, you know, if somebody like
GitHub scale would be running

165
0:7:48,76 --> 0:7:51,3
Citus, I wouldn't have any concerns
at all.

166
0:7:51,72 --> 0:7:52,94
What are the great examples?

167
0:7:53,16 --> 0:7:54,6
OLTP style.

168
0:7:56,04 --> 0:7:56,92
That's my concern.

169
0:7:56,92 --> 0:7:59,86
And I think it's still like, I
agree, like what numbers you bring,

170
0:7:59,86 --> 0:8:0,8
and this is interesting.

171
0:8:0,8 --> 0:8:2,46
We dive into them today.

172
0:8:2,8 --> 0:8:6,42
And I was in talks with Citus team
some time ago, maybe a couple

173
0:8:6,42 --> 0:8:7,32
of years ago.

174
0:8:7,48 --> 0:8:12,54
And they presented me some examples
which were not OLTP.

175
0:8:12,62 --> 0:8:17,6
For example, some search for, I
think, lawyers of video, where

176
0:8:17,6 --> 0:8:22,22
I thought like, actually lawyers
can wait 10 seconds to find

177
0:8:22,22 --> 0:8:24,3
problematic materials and so on.

178
0:8:24,38 --> 0:8:28,58
It's not OLTP when we know 200 milliseconds,
our very 1st episode

179
0:8:28,58 --> 0:8:33,04
in this podcast means that humans
already notice delay.

180
0:8:33,46 --> 0:8:36,18
Álvaro: So I think there's 2 angles
to what you're saying.

181
0:8:36,18 --> 0:8:40,36
So 1 is the technical 1, and the
other 1 is the marketing or

182
0:8:40,36 --> 0:8:41,98
trust 1, let's say.

183
0:8:42,28 --> 0:8:46,02
On the trust 1, I would say you're
right, meaning that there's

184
0:8:46,02 --> 0:8:47,22
no such great examples.

185
0:8:47,22 --> 0:8:49,54
And this actually piggybacks on
what I was saying.

186
0:8:49,54 --> 0:8:52,24
It looks like it's a technology
that didn't get the momentum

187
0:8:52,24 --> 0:8:56,1
it could have had, or it could
still has, because of the technological

188
0:8:56,32 --> 0:8:58,04
means that it conveys, right?

189
0:8:58,18 --> 0:9:1,72
I can tell you that we have right
now significant examples of

190
0:9:1,72 --> 0:9:2,64
this kind of usage.

191
0:9:2,64 --> 0:9:6,92
We have now a customer running
Citus on StackGres that is actually

192
0:9:7,02 --> 0:9:10,62
growing database towards 2.2 petabytes
by the end of the year.

193
0:9:10,92 --> 0:9:11,12
And...

194
0:9:11,12 --> 0:9:13,54
Nik: Wait, hold on 1 2nd, 2.2 petabytes?

195
0:9:14,44 --> 0:9:14,94
Yes.

196
0:9:15,48 --> 0:9:17,46
It's just 1 big Citus cluster?

197
0:9:18,08 --> 0:9:18,28
Yes.

198
0:9:18,28 --> 0:9:19,24
Álvaro: Running on StackGres?

199
0:9:20,02 --> 0:9:20,52
Yes.

200
0:9:21,04 --> 0:9:21,1
Nik: Okay.

201
0:9:21,1 --> 0:9:22,06
How many nodes?

202
0:9:23,0 --> 0:9:23,5
Álvaro: 24.

203
0:9:24,78 --> 0:9:30,26
Each node is, I'm not sure if it's
64 or 96 cores, and 0.5 a

204
0:9:30,26 --> 0:9:31,94
terabyte or more of RAM.

205
0:9:33,34 --> 0:9:34,5
Nik: Wow, okay.

206
0:9:34,86 --> 0:9:36,6
Álvaro: Those numbers are actually
quite impressive.

207
0:9:37,36 --> 0:9:42,44
It's an IoT system and they ingest
data massively from millions

208
0:9:42,44 --> 0:9:45,52
and millions of devices that send
heartbeats every 5 minutes.

209
0:9:46,1 --> 0:9:48,72
And they have a 10 day retention
period.

210
0:9:49,34 --> 0:9:52,84
So basically they're deleting 1
10th of the database and ingesting

211
0:9:52,84 --> 0:9:54,94
1 10th of the database every day.

212
0:9:55,44 --> 0:9:58,74
Right now they're in ramping up
period, and as of today, they're

213
0:9:58,74 --> 0:10:3,9
ingesting some 30 plus terabytes
per day, and this is gonna grow

214
0:10:3,9 --> 0:10:5,74
to 200 terabytes per day.

215
0:10:5,74 --> 0:10:8,08
So they're growing by 6X by the
end of the year.

216
0:10:8,92 --> 0:10:11,26
And yeah, this is running on a
single Citus cluster.

217
0:10:11,64 --> 0:10:14,98
We can discuss whether this is
a little bit OLTP or OLAP.

218
0:10:15,06 --> 0:10:16,22
It's a bit of both.

219
0:10:16,36 --> 0:10:17,98
Ingestion is OLTP.

220
0:10:18,84 --> 0:10:21,28
Queries, there are a little bit
of both.

221
0:10:21,28 --> 0:10:25,38
Also OLTP queries when they're
accessing health information about

222
0:10:25,38 --> 0:10:26,4
a given device.

223
0:10:26,88 --> 0:10:31,16
Or more aggregated reports, like
how many devices are down or

224
0:10:31,16 --> 0:10:32,22
having some issues.

225
0:10:32,44 --> 0:10:35,74
So it's a bit of a mixed workload,
but I think it works.

226
0:10:35,74 --> 0:10:39,52
Again, I agree, there are not so
many public great examples about

227
0:10:39,52 --> 0:10:40,46
using Citus.

228
0:10:40,52 --> 0:10:44,06
Citus also introduced some releases
ago, the support for sharding

229
0:10:44,06 --> 0:10:48,84
by tenant, which is actually directly
towards SaaS enterprises,

230
0:10:48,9 --> 0:10:49,14
right?

231
0:10:49,14 --> 0:10:51,98
The SaaS applications where you
don't shard by row, you shard

232
0:10:51,98 --> 0:10:55,02
by tenant, which is a great fit
for those cases too.

233
0:10:55,24 --> 0:10:57,94
Typically, these cases where you
create like a lot of tables

234
0:10:58,0 --> 0:10:59,08
for every tenant.

235
0:10:59,28 --> 0:11:1,74
So it's another way of isolating
those workloads.

236
0:11:1,98 --> 0:11:4,18
But from a technical perspective
and speaking.

237
0:11:5,22 --> 0:11:8,54
Nik: Yeah, I just, I just, I'm,
I'm, I'm smiling at my glasses

238
0:11:8,56 --> 0:11:11,8
because they said, they started
arguing with me saying that there

239
0:11:11,8 --> 0:11:16,62
are public well-known names like
Heap and the Cloudflare named

240
0:11:16,62 --> 0:11:20,14
on, on Citus pages for, for
years.

241
0:11:20,14 --> 0:11:24,18
But, but both of these cases are
analytical cases, 1st of all,

242
0:11:24,18 --> 0:11:27,38
and both these cases went away
from Citus, as we know.

243
0:11:27,62 --> 0:11:33,06
Like, I was at Meetup when Cloudflare
explained how they moved

244
0:11:33,06 --> 0:11:35,64
away from Citus to ClickHouse.

245
0:11:36,06 --> 0:11:39,44
And I know Heap Analytics moved
somewhere else as well.

246
0:11:39,56 --> 0:11:42,04
Heap Analytics was a great case,
but again, analytics.

247
0:11:42,72 --> 0:11:45,56
And Cloudflare, it was analytics,
DNS analytics.

248
0:11:45,78 --> 0:11:46,86
So it's not OLTP.

249
0:11:47,2 --> 0:11:48,7
It can be HTAP.

250
0:11:48,7 --> 0:11:49,4
Yeah, I agree.

251
0:11:49,4 --> 0:11:50,7
And your case is interesting.

252
0:11:50,74 --> 0:11:54,02
It's not what I want to see, like
I said, GitHub or some social

253
0:11:54,02 --> 0:11:56,46
media or something, when there
are many humans working.

254
0:11:56,52 --> 0:12:1,82
Your case is a great case for sharding
Specifically because it

255
0:12:1,92 --> 0:12:6,78
shows exactly where the problem
with the biggest pain of single

256
0:12:6,78 --> 0:12:8,5
cluster is a lot of writes.

257
0:12:9,38 --> 0:12:14,18
This case is highlighting this
particular use case, which is

258
0:12:14,18 --> 0:12:14,66
great.

259
0:12:14,66 --> 0:12:18,52
But I'm still like, my heart is
with social media, SaaS, like

260
0:12:18,52 --> 0:12:21,86
these things, like e-commerce,
online stuff, where a lot of humans

261
0:12:21,86 --> 0:12:26,18
try to deal with some system and
then they want greater responsiveness

262
0:12:26,92 --> 0:12:28,28
of all API calls.

263
0:12:28,28 --> 0:12:31,18
Álvaro: So let's talk about that
because part of my response

264
0:12:31,2 --> 0:12:34,08
also was coming from a technical
angle and from a technical angle

265
0:12:34,08 --> 0:12:38,56
we've been running some benchmarks
and 1 of the questions whether

266
0:12:38,56 --> 0:12:42,56
to say the system is ready for
OLTP or not is the number of concurrent

267
0:12:42,56 --> 0:12:45,32
queries that it can run and the
latency at which those queries

268
0:12:45,32 --> 0:12:45,82
run.

269
0:12:46,32 --> 0:12:51,82
And Citus can run on millions of
transactions per 2nd, even under

270
0:12:51,82 --> 0:12:53,5
1 millisecond p50.

271
0:12:54,62 --> 0:12:55,94
I would call that OLTP.

272
0:12:56,8 --> 0:12:57,7
Nik: I agree here.

273
0:12:57,7 --> 0:13:1,68
If you check very fast SELECTs,
we have benchmarks.

274
0:13:1,78 --> 0:13:6,94
We did it multiple times for actually
all sharding systems, open

275
0:13:6,94 --> 0:13:10,46
source sharding systems we can
reach, including SPQR, Citus,

276
0:13:10,84 --> 0:13:14,74
Multigres actually we also tested,
PgDog, something else.

277
0:13:15,3 --> 0:13:20,26
And Citus, you checked for latency
overhead, off latency, latency

278
0:13:20,26 --> 0:13:21,66
overhead benchmark, simple.

279
0:13:22,54 --> 0:13:26,74
Which like if primary key lookup is
sub millisecond, you don't want

280
0:13:26,74 --> 0:13:29,66
another few milliseconds to be
introduced.

281
0:13:29,7 --> 0:13:33,96
And Citus was actually looking
very well.

282
0:13:34,64 --> 0:13:38,04
It was introducing like 200 microseconds
or so, as I remember.

283
0:13:38,8 --> 0:13:42,14
Álvaro: Yeah, I actually have drilled
down on the CPU consumption

284
0:13:42,2 --> 0:13:44,0
per layer for this kind of queries.

285
0:13:44,44 --> 0:13:48,84
And the Total aggregated CPU consumption
per query, including

286
0:13:49,5 --> 0:13:53,4
the coordinator, including the
worker, including the connection

287
0:13:53,4 --> 0:13:58,22
pooling, excluding the network,
is somewhere in between 300 to

288
0:13:58,44 --> 0:13:59,02
400 microseconds.

289
0:13:59,34 --> 0:14:0,52
Nik: Microseconds, exactly.

290
0:14:0,54 --> 0:14:1,02
Yeah, yeah.

291
0:14:1,02 --> 0:14:1,5
Okay.

292
0:14:1,5 --> 0:14:4,46
Slightly different from what I
remember from our benchmarks but

293
0:14:4,46 --> 0:14:8,48
well the point here is that it
can be very quickly eaten by something

294
0:14:8,48 --> 0:14:12,48
like network complexity right you
won't notice this small overhead

295
0:14:12,48 --> 0:14:16,0
behind something else like
just hops between yeah

296
0:14:16,0 --> 0:14:18,72
Álvaro: the networks that can be
worse than that yeah yeah

297
0:14:18,74 --> 0:14:20,06
Nik: that's I agree yeah

298
0:14:20,34 --> 0:14:23,24
Michael: Álvaro when you query
when you're doing that check are

299
0:14:23,24 --> 0:14:25,9
you going through a coordinator
in Citus?

300
0:14:26,54 --> 0:14:29,7
Because I know now you can query
directly 1 of the workers.

301
0:14:30,06 --> 0:14:31,9
Would that reduce the overhead?

302
0:14:33,52 --> 0:14:37,96
Álvaro: So, yes, the usual way
you wire up a Citus cluster is

303
0:14:37,96 --> 0:14:42,18
you have workers, which are where
you split the data into and

304
0:14:42,18 --> 0:14:47,5
then you have the coordinator,
which acts as a metadata entry

305
0:14:47,5 --> 0:14:49,46
point, but also as a query router.

306
0:14:49,96 --> 0:14:52,7
And so it is the entry point by
default.

307
0:14:53,14 --> 0:14:56,26
It's true that Citus at version
11 introduced a functionality

308
0:14:56,46 --> 0:14:59,44
where you can query also the worker
nodes directly.

309
0:14:59,44 --> 0:15:1,1
There's 2 reasons to do this.

310
0:15:1,28 --> 0:15:4,78
1, if you know from the application
perspective, the sharding

311
0:15:4,78 --> 0:15:7,66
algorithm and you know which shard
to talk directly, which is

312
0:15:7,66 --> 0:15:10,86
probably not a good idea, but technically
sometimes you can do

313
0:15:10,86 --> 0:15:11,16
this.

314
0:15:11,16 --> 0:15:14,72
It is a good idea actually, but
the algorithm is a little bit

315
0:15:14,72 --> 0:15:16,06
internal to Citus, right?

316
0:15:16,06 --> 0:15:19,4
If it would be like clear, exposed
algorithm, then it would be

317
0:15:19,4 --> 0:15:21,9
very good for applications that
want to implement that algorithm,

318
0:15:21,9 --> 0:15:23,16
then talk directly to Shards.

319
0:15:23,16 --> 0:15:25,68
But at least Citus gives you the
opportunity to do that if you

320
0:15:25,68 --> 0:15:26,24
want to do so.

321
0:15:26,24 --> 0:15:27,72
You can target directly the workers.

322
0:15:27,72 --> 0:15:30,04
And if you target directly the
worker, there's no query routing.

323
0:15:30,04 --> 0:15:32,98
It goes directly and resolves the
query locally, if it's local

324
0:15:33,04 --> 0:15:35,66
or fans out to other workers if
required.

325
0:15:36,22 --> 0:15:39,58
But the other reason is to try
to scale the single coordinator.

326
0:15:39,96 --> 0:15:43,82
And I believe this actually also
piggybacks a little bit with

327
0:15:43,82 --> 0:15:47,5
what you were saying, Nikolay, that
Citus maybe was abandoned

328
0:15:47,5 --> 0:15:49,5
by some project before, like Cloudflare.

329
0:15:50,22 --> 0:15:53,42
And Citus can scale workers horizontally
as much as you need.

330
0:15:53,42 --> 0:15:56,52
I was referring to this 24 node
cluster before, and there's been

331
0:15:56,52 --> 0:16:0,12
reports of much larger clusters,
like 60 nodes or more.

332
0:16:0,3 --> 0:16:2,12
And they scale very well.

333
0:16:2,26 --> 0:16:4,84
However, the coordinator remains
only 1.

334
0:16:5,9 --> 0:16:11,18
And if you outrun the coordinator,
then you're hitting a ceiling.

335
0:16:11,52 --> 0:16:14,58
That ceiling is normally quite
high and actually we've measured

336
0:16:14,58 --> 0:16:17,14
that ceiling, but it exists, right?

337
0:16:17,22 --> 0:16:19,92
So that's why Citus introduced
functionality, you can query the

338
0:16:19,92 --> 0:16:22,62
routers directly, and then you
have like essentially multiple

339
0:16:22,64 --> 0:16:23,14
coordinators.

340
0:16:23,6 --> 0:16:26,84
But I don't think that's the way
to go either, because then you're

341
0:16:26,84 --> 0:16:28,12
now contending workloads.

342
0:16:28,26 --> 0:16:32,5
So you have nodes that act both
as query routers, which consumes

343
0:16:32,5 --> 0:16:36,38
CPU, consume resources, notable,
they require connections, and

344
0:16:36,38 --> 0:16:39,72
they spread all the connections
to other nodes, making also connection

345
0:16:39,72 --> 0:16:40,54
math harder.

346
0:16:41,2 --> 0:16:43,62
And they're also serving data and
queries.

347
0:16:43,84 --> 0:16:49,54
So it's a bit of a mixed workload
that I believe is hard to dimension

348
0:16:49,78 --> 0:16:53,04
and do capacity planning around
that and makes everything much

349
0:16:53,04 --> 0:16:53,68
more complicated.

350
0:16:53,68 --> 0:16:55,74
So it's not a perfect solution
either.

351
0:16:56,48 --> 0:16:59,74
Nik: I think we forgot an important
piece.

352
0:17:0,72 --> 0:17:5,14
Why Álvaro at all talking about
Citus, right?

353
0:17:5,14 --> 0:17:6,3
This is a good question.

354
0:17:6,3 --> 0:17:7,64
You're doing something else.

355
0:17:7,64 --> 0:17:9,66
Why are you talking about Citus
at all?

356
0:17:9,72 --> 0:17:11,46
Citus is in the hands of Microsoft.

357
0:17:12,56 --> 0:17:13,66
What happened here?

358
0:17:14,54 --> 0:17:19,02
You developed StackGres, and then
somehow these paths joined,

359
0:17:19,02 --> 0:17:19,52
right?

360
0:17:19,6 --> 0:17:22,66
They met these things and you have
examples.

361
0:17:23,68 --> 0:17:26,7
That's why you mentioned this 2
petabyte example, right?

362
0:17:26,88 --> 0:17:29,6
Can you explain a little bit what's
happening, why you focus

363
0:17:29,6 --> 0:17:30,98
on Citus so much lately?

364
0:17:32,5 --> 0:17:35,78
Álvaro: Well, it's not me, it's
what customers and users request.

365
0:17:36,42 --> 0:17:40,32
I see a growing concern and need
to scale Postgres beyond the

366
0:17:40,32 --> 0:17:43,4
single primary for the reasons
I mentioned before, write limitations

367
0:17:43,94 --> 0:17:45,36
and also blast radius.

368
0:17:45,78 --> 0:17:48,62
And that's why we are also seeing
an explosion of new sharding

369
0:17:48,62 --> 0:17:49,84
technologies for Postgres.

370
0:17:49,84 --> 0:17:53,5
So this is a real need and is driven
by market demand.

371
0:17:53,5 --> 0:17:57,26
We are seeing a lot of people coming
to us asking how we can

372
0:17:57,26 --> 0:17:58,4
shard our database.

373
0:17:59,44 --> 0:18:2,44
And StackGres, it's a platform for
running Postgres, right?

374
0:18:2,44 --> 0:18:5,28
That's it, it's fully featured,
it's really advanced, but it's

375
0:18:5,28 --> 0:18:6,6
a platform for running Postgres.

376
0:18:6,66 --> 0:18:10,14
And so we saw this need and we
saw early that we had to provide

377
0:18:10,14 --> 0:18:11,86
users a way to shard.

378
0:18:11,98 --> 0:18:15,04
And actually we were in a very
privileged position, I would say,

379
0:18:15,04 --> 0:18:18,66
because what we've done in StackGres
with single clusters is

380
0:18:18,66 --> 0:18:22,2
orchestrate and allow you to run
the whole stack of components

381
0:18:22,2 --> 0:18:25,46
that you need to run Postgres,
connection pooling, backups, monitoring,

382
0:18:25,6 --> 0:18:28,38
all that, all automated and all
packaged.

383
0:18:28,44 --> 0:18:33,28
So deploying it is extremely easy
while also following best practices

384
0:18:33,28 --> 0:18:35,78
that we have from our company's
operational experience.

385
0:18:36,48 --> 0:18:40,18
But then we saw that creating a
Citus cluster involved a lot

386
0:18:40,18 --> 0:18:43,86
of manual work because it's no
longer 1 cluster.

387
0:18:44,14 --> 0:18:47,06
Now, if you think about it, and
the model we follow in StackGres

388
0:18:47,12 --> 0:18:51,78
is that every component of the
Citus cluster, the workers, the

389
0:18:51,78 --> 0:18:56,14
coordinator, all of those are a
Postgres cluster in itself, typically

390
0:18:56,14 --> 0:18:59,02
with more than 1 instance also
for high availability purpose,

391
0:18:59,02 --> 0:18:59,52
right?

392
0:18:59,6 --> 0:19:2,54
And they need to be wired up precisely
with the same configuration

393
0:19:2,64 --> 0:19:5,74
parameters across many places or
otherwise it will not behave

394
0:19:5,74 --> 0:19:6,84
as a single cluster.

395
0:19:7,16 --> 0:19:10,82
So it's not only a lot of work
to wire up as let's say a 24,

396
0:19:10,86 --> 0:19:14,18
48 node the Citus cluster but also
to do it right.

397
0:19:14,66 --> 0:19:17,8
And With our automation and orchestration
that we built for a

398
0:19:17,8 --> 0:19:19,54
StackGres, we could do this work.

399
0:19:19,78 --> 0:19:23,2
And now the reality is that if
you use a StackGres and want to

400
0:19:23,2 --> 0:19:26,12
create a sharded cluster with Citus,
by the way, we support all

401
0:19:26,12 --> 0:19:29,94
the other sharding technologies
in StackGres, not only Citus,

402
0:19:30,18 --> 0:19:32,78
even though it's the most used
and the most feature complete,

403
0:19:32,78 --> 0:19:34,98
I would say solution for sharding
that we have.

404
0:19:35,34 --> 0:19:39,86
But with the 13 lines of YAML,
really high level, you can create

405
0:19:39,86 --> 0:19:42,84
a Citus cluster of 60 nodes
in a few minutes.

406
0:19:43,28 --> 0:19:46,06
You can even do from a web console,
Click, click, click, boom,

407
0:19:46,06 --> 0:19:47,62
and you get the sharded cluster.

408
0:19:47,78 --> 0:19:50,72
There's nothing else compared to
this other than the managed

409
0:19:50,72 --> 0:19:53,04
service and Microsoft offers on
Azure.

410
0:19:53,92 --> 0:19:56,82
And that's the gap we wanted to
cover because we saw demand and

411
0:19:56,82 --> 0:19:57,88
it's paying off.

412
0:19:58,02 --> 0:20:1,36
I would say a significant percentage
of our new customers and

413
0:20:1,36 --> 0:20:3,28
users come for sharding.

414
0:20:4,54 --> 0:20:8,0
It is very widespread and it's
a need that few people are focused

415
0:20:8,0 --> 0:20:11,6
on solving, especially using a
technology that is proven and

416
0:20:11,6 --> 0:20:13,52
well maintained as of today like
Citus.

417
0:20:13,94 --> 0:20:14,18
It's a

418
0:20:14,18 --> 0:20:17,32
Michael: good point about Citus
being on Microsoft's platform

419
0:20:17,32 --> 0:20:21,58
and it could because it's AGPL
it means other platforms are

420
0:20:21,58 --> 0:20:25,78
going to really struggle probably
just can't offer it right so

421
0:20:25,92 --> 0:20:30,06
actually it makes sense that the
if the only kind of 10 year

422
0:20:30,06 --> 0:20:32,74
old sharding actually it's maybe
it's not the only it because

423
0:20:32,74 --> 0:20:36,64
it wasn't SPQR quite old as well
as you say they feature complete

424
0:20:37,14 --> 0:20:40,44
sharding solution on Postgres if
that's 10 years old but you

425
0:20:40,44 --> 0:20:44,28
can't get it on any managed services
except for 1, actually,

426
0:20:44,54 --> 0:20:48,42
it makes sense that there's demand
for, okay, how do we self-manage

427
0:20:48,64 --> 0:20:49,12
this?

428
0:20:49,12 --> 0:20:52,1
Nik: We need to revisit the topic
of AGPL misconception.

429
0:20:52,58 --> 0:20:53,5
How to say?

430
0:20:54,02 --> 0:20:58,62
So AGPL allows you to run it on
your managed service, but you

431
0:20:58,62 --> 0:21:2,08
cannot modify it without not publishing
source code.

432
0:21:2,62 --> 0:21:4,94
And right, this is 1st thing.

433
0:21:4,94 --> 0:21:7,8
Let me explain my words and then
Álvaro you can correct me if

434
0:21:7,8 --> 0:21:8,2
I'm wrong.

435
0:21:8,2 --> 0:21:10,94
I'm just curious if I understand
it right.

436
0:21:11,14 --> 0:21:14,12
And then Álvaro is not doing a
managed service at all.

437
0:21:14,54 --> 0:21:19,3
StackGres is just like operator
plus of course there is enterprise

438
0:21:19,3 --> 0:21:21,1
edition with support and so on.

439
0:21:21,1 --> 0:21:24,64
This is like money coming from
support, but it can be installed

440
0:21:24,64 --> 0:21:25,74
anywhere, right?

441
0:21:25,9 --> 0:21:29,68
Like having AGPL components doesn't
hurt at all.

442
0:21:29,84 --> 0:21:34,7
And, And finally, I know StackGres
was on AGPL until lately, right?

443
0:21:34,84 --> 0:21:36,18
Can you elaborate this?

444
0:21:37,2 --> 0:21:38,16
On 1 bit.

445
0:21:38,44 --> 0:21:39,34
Yeah, am I right?

446
0:21:39,34 --> 0:21:41,08
It's not an AGPL anymore?

447
0:21:41,58 --> 0:21:44,24
Álvaro: It's very good insider
information, yes.

448
0:21:44,24 --> 0:21:45,68
So, 1st of all, AGPL.

449
0:21:46,06 --> 0:21:47,58
I'm not a lawyer, okay?

450
0:21:47,68 --> 0:21:51,42
But I love AGPL and GPL or in
general copyleft licenses.

451
0:21:52,04 --> 0:21:54,88
I believe people say they are viral,
that they restrict your

452
0:21:54,88 --> 0:21:55,28
freedom.

453
0:21:55,28 --> 0:21:56,84
I believe it's exactly the opposite.

454
0:21:56,84 --> 0:22:0,92
They provide forward freedom warranties,
meaning that whatever

455
0:22:0,92 --> 0:22:4,3
software is published under a copyleft
license, it is providing

456
0:22:4,3 --> 0:22:6,6
a warranty that will never be closed
source.

457
0:22:6,6 --> 0:22:9,26
It will always remain under such
a license.

458
0:22:9,4 --> 0:22:12,14
So it's a forward warranty for
users of Freedom.

459
0:22:12,4 --> 0:22:17,02
Whereas a license like the Apache
2 or BSD, they can be turned

460
0:22:17,02 --> 0:22:20,28
into proprietary software, and
that software may cease to exist

461
0:22:20,28 --> 0:22:21,62
as an open source software.

462
0:22:22,2 --> 0:22:23,44
That's why I like them.

463
0:22:23,44 --> 0:22:27,44
Now, the reality is that a lot
of people believe that they cannot

464
0:22:27,44 --> 0:22:30,4
be used, for example, as a managed
service, but you put it very

465
0:22:30,4 --> 0:22:30,9
well.

466
0:22:30,92 --> 0:22:34,84
You can perfectly well use an AGPL
software while offering a

467
0:22:34,84 --> 0:22:35,66
managed service.

468
0:22:35,74 --> 0:22:38,6
The only thing that the license
force you to do is that whomever

469
0:22:38,6 --> 0:22:42,1
interacts over the network with
your software need to provide

470
0:22:42,1 --> 0:22:45,72
the same warranties that you got
for that software, which means

471
0:22:45,72 --> 0:22:47,06
publishing the source code.

472
0:22:47,08 --> 0:22:50,86
So if you use Citus, and even with
Citus, it's arguable because

473
0:22:50,86 --> 0:22:52,62
you don't talk directly to Citus.

474
0:22:52,94 --> 0:22:56,32
But even if we do a little more
loose interpretation, or because

475
0:22:56,32 --> 0:23:0,04
I have Citus, then I'm bound by
these terms directly, it would

476
0:23:0,04 --> 0:23:3,48
suffice to publish the source code
of Citus, which is a link

477
0:23:3,66 --> 0:23:5,9
and a disclosure that you use Citus,
right?

478
0:23:6,14 --> 0:23:10,52
Then if you patch Citus and then
you improve Citus, even if you

479
0:23:10,52 --> 0:23:13,68
don't submit upstream, your users
have the right to get those

480
0:23:13,68 --> 0:23:16,38
benefits too, which is why I love
this kind of licenses.

481
0:23:17,4 --> 0:23:19,96
But some lawyers disagree with
me, probably they're right because

482
0:23:19,96 --> 0:23:21,3
they're lawyers, I'm not.

483
0:23:21,88 --> 0:23:26,5
And not many people want to use
AGPL on their managed services.

484
0:23:26,58 --> 0:23:30,36
But in my understanding, limited
understanding, I think that's

485
0:23:30,36 --> 0:23:31,9
not necessarily correct.

486
0:23:32,54 --> 0:23:35,58
Nik: Imagine if Postgres was on
AGPL version 3.

487
0:23:35,82 --> 0:23:39,38
RDS would need to publish all the
changes they made to implement

488
0:23:39,48 --> 0:23:41,5
rds_superuser and so on.

489
0:23:42,9 --> 0:23:44,36
It's simplified as well.

490
0:23:45,06 --> 0:23:48,18
Michael: But this is a really great
point because I think Postgres

491
0:23:48,18 --> 0:23:51,18
has benefited from being on a permissive
license in terms of

492
0:23:51,18 --> 0:23:52,54
how much usage it's got.

493
0:23:52,54 --> 0:23:55,32
And maybe we're seeing the exact
opposite with Citus.

494
0:23:55,32 --> 0:23:58,44
Maybe we're seeing that AGPL is
actually costing in terms

495
0:23:58,44 --> 0:23:59,16
Nik: of usage.

496
0:23:59,16 --> 0:23:59,94
Repelled usage.

497
0:24:0,12 --> 0:24:0,52
Repelled, yeah.

498
0:24:0,52 --> 0:24:3,68
Like Google, for example, have
internal, at least in the past,

499
0:24:3,68 --> 0:24:7,58
they had internal directive that
let's avoid the GPL version

500
0:24:7,58 --> 0:24:8,98
3 software completely.

501
0:24:9,52 --> 0:24:11,34
Álvaro: Yeah, it's a big if.

502
0:24:11,52 --> 0:24:12,74
We don't know the answer.

503
0:24:13,52 --> 0:24:17,96
Most likely, because The market
has this, what I believe it's

504
0:24:17,96 --> 0:24:22,86
a misconception, but this general
wisdom that AGPL or GPL licenses

505
0:24:23,04 --> 0:24:27,38
are this viral and so on, then
yes, it looks like copyleft software

506
0:24:27,8 --> 0:24:32,42
is perceived as unideal and permissive
licenses are winning.

507
0:24:33,06 --> 0:24:35,94
And certainly this is the reason
I don't want to avoid the question,

508
0:24:36,22 --> 0:24:41,82
yes, StackGres, and I can announce
this publicly, it's gonna be

509
0:24:41,82 --> 0:24:46,08
licensed under the Apache 2 license,
instead of AGPL, as much

510
0:24:46,08 --> 0:24:50,08
as I wish to do otherwise, but
the market and the users demand

511
0:24:50,08 --> 0:24:51,36
this, so we're happy to

512
0:24:51,36 --> 0:24:52,28
Nik: do that and we will

513
0:24:52,28 --> 0:24:55,02
Álvaro: do that in the next days
actually.

514
0:24:55,52 --> 0:24:59,34
Nik: While still Citus remains
on AGPL and it's just plugged

515
0:24:59,34 --> 0:25:0,12
in, right?

516
0:25:0,78 --> 0:25:4,28
Álvaro: Yep, we support, in StackGres
we support more than 200

517
0:25:4,28 --> 0:25:4,78
extensions.

518
0:25:4,78 --> 0:25:8,1
They come with a lot of licenses
and it's a user's responsibility

519
0:25:8,48 --> 0:25:12,16
to, to know the license they're,
they're dealing with and their

520
0:25:12,18 --> 0:25:12,6
wording.

521
0:25:12,6 --> 0:25:15,04
We just provide the software and
the platform.

522
0:25:15,16 --> 0:25:18,74
Nik: It's like a big Lego and some
pieces are different colors

523
0:25:18,74 --> 0:25:19,78
and so on, right?

524
0:25:20,32 --> 0:25:20,82
Álvaro: Yeah.

525
0:25:21,46 --> 0:25:25,58
And to complete the picture, I
think I want to go back a little

526
0:25:25,58 --> 0:25:27,42
bit to what you were mentioning,
Michael.

527
0:25:28,38 --> 0:25:32,04
That also relates maybe to 1 of
the reasons, or potentially,

528
0:25:32,04 --> 0:25:35,08
I don't know exactly the reasons,
but why I believe that maybe

529
0:25:35,08 --> 0:25:39,86
some companies abandoned Citus
at some point, which is that Citus,

530
0:25:40,02 --> 0:25:43,14
the workers, where you split the
data, you can, again, you can

531
0:25:43,14 --> 0:25:47,06
add as many as you need, but coordinator
is only 1.

532
0:25:47,68 --> 0:25:51,66
And if you saturate the coordinator,
then you're out of luck,

533
0:25:51,66 --> 0:25:51,98
right?

534
0:25:51,98 --> 0:25:55,92
You can use, as we said, you can
query the nodes directly, the

535
0:25:55,92 --> 0:26:0,6
workers directly, but that it's
complicated to operate and you

536
0:26:0,6 --> 0:26:2,78
still also will get as far as you
can.

537
0:26:3,42 --> 0:26:7,44
This is something we saw some time
ago and decided to do something

538
0:26:7,44 --> 0:26:8,18
about it.

539
0:26:8,32 --> 0:26:11,0
And in the most recent version
of StackGres, StackGres

540
0:26:11,0 --> 0:26:14,12
1.19, we have introduced something
called query routers.

541
0:26:15,06 --> 0:26:18,34
Query routers does nothing that
we have actually patched on Citus

542
0:26:18,34 --> 0:26:19,28
or anything like that.

543
0:26:19,28 --> 0:26:20,58
It's unmodified Citus.

544
0:26:20,58 --> 0:26:24,02
It's just an architectural pattern
that we identified that we

545
0:26:24,02 --> 0:26:27,94
could apply to Citus to allow to
have essentially multiple coordinators,

546
0:26:28,26 --> 0:26:29,16
put that simple.

547
0:26:30,06 --> 0:26:34,3
So a query router is like a coordinator,
so it can route queries.

548
0:26:34,54 --> 0:26:37,18
It doesn't have data, it just route
queries.

549
0:26:37,36 --> 0:26:39,82
The only thing it cannot do is
the DDL operations.

550
0:26:39,96 --> 0:26:42,28
The DDL operations go through the
coordinator.

551
0:26:42,9 --> 0:26:45,82
But now you can add as many query
routers as you want.

552
0:26:46,08 --> 0:26:50,32
And so the limitation that Citus
may have had until now with

553
0:26:50,32 --> 0:26:52,5
a single coordinator, it's gone.

554
0:26:52,8 --> 0:26:55,84
And now you can scale by adding
as many query routers as you

555
0:26:55,84 --> 0:27:0,52
want and scale, not only now the
worker side, the layer where

556
0:27:0,52 --> 0:27:4,58
it keeps the data, but also the
router layer to almost any size.

557
0:27:4,66 --> 0:27:7,6
I'm publishing some benchmarks
that we have been running in the

558
0:27:7,6 --> 0:27:8,9
last days about this.

559
0:27:9,64 --> 0:27:14,88
And ballpark numbers, 64 core workers
running simple queries

560
0:27:15,06 --> 0:27:19,44
indexed by primary key, but not
fully in memory, but with a high

561
0:27:19,44 --> 0:27:20,24
hit ratio.

562
0:27:20,68 --> 0:27:25,76
A single coordinator with 3 workers,
the coordinator being also

563
0:27:25,76 --> 0:27:30,64
64 cores, was maxing out at around
550, 000 transactions per

564
0:27:30,64 --> 0:27:33,9
2nd with a pretty good latency,
like under 1 millisecond.

565
0:27:35,08 --> 0:27:36,08
But that's it.

566
0:27:36,34 --> 0:27:39,78
More than 500000, a little bit
over 500000, and the coordinator

567
0:27:39,96 --> 0:27:40,66
is saturated.

568
0:27:41,18 --> 0:27:44,92
The reason why this number is because
these queries are quite

569
0:27:44,92 --> 0:27:48,42
fast to process, and the proportion
of overhead per query processing

570
0:27:48,44 --> 0:27:49,68
is maxed out, right?

571
0:27:49,74 --> 0:27:53,1
It's a lot of overhead per very
small query, per tiny query.

572
0:27:53,1 --> 0:27:55,76
Of course, if you hit all the queries
and they involve I/O.

573
0:27:55,76 --> 0:27:59,38
And reading work, you can scale
further before hitting the limit

574
0:27:59,38 --> 0:28:0,14
of the coordinator.

575
0:28:0,14 --> 0:28:3,54
So we designed this benchmark to
actually find the limit as soon

576
0:28:3,54 --> 0:28:6,1
as possible of the coordinator
at still very high rates, right?

577
0:28:6,1 --> 0:28:8,16
More than 500000 transactions per
2nd.

578
0:28:8,52 --> 0:28:10,28
So coordinator is maxed out.

579
0:28:10,28 --> 0:28:14,5
Then we add 2 coordinators with
the same 3 shards and then 3

580
0:28:14,5 --> 0:28:18,58
coordinators with the same 3 workers,
sorry, the data nodes.

581
0:28:18,7 --> 0:28:21,18
And the performance basically scaled
almost linear.

582
0:28:21,18 --> 0:28:26,02
So from 550, it went to 1, 050,
000 and then to 1, 500, 000.

583
0:28:26,66 --> 0:28:28,22
So pretty much linear growth.

584
0:28:28,66 --> 0:28:32,16
Michael: When you say 3 coordinators,
Do you mean 3 query routers

585
0:28:32,16 --> 0:28:32,86
in the new?

586
0:28:32,86 --> 0:28:33,34
Yeah.

587
0:28:33,34 --> 0:28:33,76
OK.

588
0:28:33,76 --> 0:28:34,26
OK.

589
0:28:34,28 --> 0:28:34,98
Álvaro: Yeah, sorry.

590
0:28:35,0 --> 0:28:37,86
I use the terms a little bit interchangeably,
because for me,

591
0:28:37,86 --> 0:28:40,46
they're conceptually the same,
even though they're not.

592
0:28:40,58 --> 0:28:41,34
You're right.

593
0:28:41,34 --> 0:28:42,74
It's query routers, yes.

594
0:28:42,74 --> 0:28:44,0
So 3 query routers.

595
0:28:44,2 --> 0:28:45,72
The coordinator remains there.

596
0:28:45,72 --> 0:28:50,56
And actually, I am now recommending
that if you use this functionality

597
0:28:50,78 --> 0:28:53,3
with the StackGres, with the query
routers, that you keep the

598
0:28:53,3 --> 0:28:56,64
coordinator small configuration,
small number of cores, just

599
0:28:56,64 --> 0:29:0,24
to run the DDL through there, and
do not use a coordinator for

600
0:29:0,24 --> 0:29:1,06
query routing.

601
0:29:1,06 --> 0:29:3,04
And then use a layer of query routers.

602
0:29:3,04 --> 0:29:6,68
You can start with 1 if you want
and then scale them up as your

603
0:29:6,68 --> 0:29:8,48
needs grow, but they're already
prepared.

604
0:29:8,48 --> 0:29:11,82
You already, they use a different
service in Kubernetes, so it's

605
0:29:11,82 --> 0:29:14,18
just a way to route the queries
and entry point.

606
0:29:14,34 --> 0:29:15,92
And then you already have that
established.

607
0:29:15,92 --> 0:29:19,6
Your application points to the
service that will balance across

608
0:29:19,6 --> 0:29:21,14
all the query routers you may have.

609
0:29:21,14 --> 0:29:24,08
And then it's just a matter of
changing 1 character in your YAML

610
0:29:24,08 --> 0:29:27,9
file, say, query routers 1, query
routers 2, and you would scale

611
0:29:28,02 --> 0:29:28,52
automatically.

612
0:29:29,48 --> 0:29:33,08
Michael: And You mentioned a while
back that StackGres supports

613
0:29:33,08 --> 0:29:34,46
multiple ways of sharding.

614
0:29:34,7 --> 0:29:38,08
Could I use query routers with
any of them or is this a Citus

615
0:29:38,08 --> 0:29:39,72
specific thing?

616
0:29:39,72 --> 0:29:40,96
Álvaro: It's a very good question.

617
0:29:40,96 --> 0:29:42,42
This is Citus specific.

618
0:29:42,84 --> 0:29:47,16
So if we look at the internals,
Again, this is not, we didn't

619
0:29:47,16 --> 0:29:48,94
touch Citus source code.

620
0:29:48,94 --> 0:29:51,3
We could have if we needed, but
it was not needed.

621
0:29:51,3 --> 0:29:53,1
The building blocks were surprisingly
there.

622
0:29:53,1 --> 0:29:55,82
Actually, what I'm surprised that
nobody saw this before.

623
0:29:56,2 --> 0:30:0,52
For us, it was crystal clear because
it all started with the

624
0:30:0,52 --> 0:30:4,76
concept that every worker can function
as a query router, as

625
0:30:4,76 --> 0:30:5,8
a coordinator, right?

626
0:30:5,8 --> 0:30:7,32
We just call them query routers.

627
0:30:7,54 --> 0:30:9,72
So that was already working there.

628
0:30:9,72 --> 0:30:13,14
The way this works is because they
implement the metadata that

629
0:30:13,14 --> 0:30:15,3
Citus replicates across all nodes.

630
0:30:15,36 --> 0:30:18,0
So you know whom to talk to, essentially.

631
0:30:19,14 --> 0:30:22,44
And there's another property, which
is what we combined into

632
0:30:22,44 --> 0:30:24,58
this pattern that we call the query
router.

633
0:30:24,72 --> 0:30:28,2
There is a property that is called
shouldhaveshards, and you

634
0:30:28,2 --> 0:30:29,56
can set it to false.

635
0:30:30,06 --> 0:30:33,64
Shards are the way that Citus internally
distributes the data,

636
0:30:33,64 --> 0:30:33,96
right?

637
0:30:33,96 --> 0:30:37,06
So also to clear, to make the context
clear for everybody.

638
0:30:37,36 --> 0:30:40,46
On Citus, there is workers, which
let's say they are the physical

639
0:30:40,58 --> 0:30:44,16
nodes that contain the data, physical,
virtual, you name it,

640
0:30:44,16 --> 0:30:46,12
the nodes that contain the data.

641
0:30:46,38 --> 0:30:48,84
But the data in reality is split
into shards.

642
0:30:49,06 --> 0:30:53,52
And then you map shards to workers,
at least 1, but could be

643
0:30:53,52 --> 0:30:54,44
more than 1.

644
0:30:54,64 --> 0:30:58,22
Sometimes some other people call
this virtual shards versus real

645
0:30:58,22 --> 0:30:59,1
shards, right?

646
0:30:59,34 --> 0:31:2,56
Citus terminology is workers,
like the physical thing, and shards

647
0:31:2,56 --> 0:31:3,96
like the virtual thing.

648
0:31:3,96 --> 0:31:7,62
And typically we recommend a higher
number of shards to worker

649
0:31:7,76 --> 0:31:10,96
because it helps you also scale
when you add more workers, you

650
0:31:10,96 --> 0:31:13,82
just move some of the shards to
other workers.

651
0:31:13,82 --> 0:31:15,86
You don't need to re-shard the
whole thing.

652
0:31:15,86 --> 0:31:16,24
Right.

653
0:31:16,24 --> 0:31:17,86
It's much, much better pattern.

654
0:31:17,86 --> 0:31:20,5
Actually, by the way, I'm diving
maybe a little bit too deep

655
0:31:20,5 --> 0:31:22,86
into that topic, but I use the
magic number 120.

656
0:31:23,36 --> 0:31:24,86
Most people use powers of 2.

657
0:31:24,86 --> 0:31:27,96
I think that's a mistake, because
then you can only grow by powers

658
0:31:27,96 --> 0:31:31,92
of 2, which by the growth of powers
of 2, can mean to grow a

659
0:31:31,92 --> 0:31:34,24
lot when maybe you don't need to
grow that much.

660
0:31:34,34 --> 0:31:37,88
Versus having 120 is a number that
has a lot of divisors, and

661
0:31:37,88 --> 0:31:41,2
so you can grow from 30 to 40,
for example.

662
0:31:41,2 --> 0:31:43,94
You don't need to go from 30 to
60, you need to double because

663
0:31:43,94 --> 0:31:46,7
they're still evenly divided across
all your workers.

664
0:31:47,18 --> 0:31:51,72
If you take a worker that can act
as a coordinator or a router,

665
0:31:51,78 --> 0:31:55,6
and you set this property
shouldhaveshards to false, then

666
0:31:55,6 --> 0:31:59,16
Citus will not place any shards
on it, it will be data-less.

667
0:31:59,68 --> 0:32:5,4
So essentially a data-less worker,
and that's a good router.

668
0:32:5,46 --> 0:32:6,88
So the pattern was just there.

669
0:32:6,88 --> 0:32:11,02
We just enabled it through architecture
and automation in StackGres.

670
0:32:11,82 --> 0:32:13,22
Michael: Simple, I like it.

671
0:32:13,66 --> 0:32:16,32
Álvaro: So of course, there's more
meat to it when you look into

672
0:32:16,32 --> 0:32:18,78
the internal details, but conceptually
this is it.

673
0:32:19,28 --> 0:32:19,78
Just

674
0:32:20,24 --> 0:32:22,4
Nik: don't write proxy, just use
Postgres.

675
0:32:25,64 --> 0:32:26,92
In yet another area.

676
0:32:27,8 --> 0:32:28,94
Yeah, it's interesting.

677
0:32:29,04 --> 0:32:34,16
Speaking of benchmarks, these numbers,
what concerns me that

678
0:32:34,16 --> 0:32:39,0
we talk about only read-only queries,
I would like to see the

679
0:32:39,96 --> 0:32:44,12
series 2 of benchmarks talking
about writes, because as we already

680
0:32:44,12 --> 0:32:50,72
discussed, the case you have in
hand is 2.2 petabytes.

681
0:32:52,42 --> 0:32:57,18
It's about writes, not reads, because
scaling reads is less interesting,

682
0:32:57,44 --> 0:32:58,72
less challenging, right?

683
0:32:59,12 --> 0:33:3,42
Álvaro: It is actually challenging
And again, I will not claim

684
0:33:3,42 --> 0:33:4,7
that Citus is perfect.

685
0:33:4,74 --> 0:33:7,36
And it is really good, but it's
not perfect.

686
0:33:7,36 --> 0:33:8,5
I guess nothing is perfect.

687
0:33:8,8 --> 0:33:10,14
And it requires tuning.

688
0:33:10,44 --> 0:33:12,62
I'm not gonna say this worked out
of the box.

689
0:33:12,62 --> 0:33:14,08
No, it requires tuning.

690
0:33:14,1 --> 0:33:18,06
And 1 of The hardest part of Citus
actually is connection tuning,

691
0:33:18,28 --> 0:33:22,12
because when you look at Citus,
at any distributed database for

692
0:33:22,12 --> 0:33:25,6
that matter, any sharded database,
it's no longer a single thing,

693
0:33:25,6 --> 0:33:26,1
right?

694
0:33:26,14 --> 0:33:29,94
Now you're dealing with connections
from the client to the coordinator

695
0:33:29,96 --> 0:33:33,58
or the query routers, and from
them to the workers.

696
0:33:34,06 --> 0:33:36,76
And sometimes these queries actually
have a big fan-out.

697
0:33:36,76 --> 0:33:41,3
For example, on this particular
case of the 2.2 petabyte database,

698
0:33:41,64 --> 0:33:46,42
I just cannot mention the customer
name, but it's a big telco.

699
0:33:46,96 --> 0:33:50,02
They are using COPY to ingest data.

700
0:33:50,02 --> 0:33:53,1
And that's amazing because it's
the most efficient way to ingest

701
0:33:53,1 --> 0:33:54,12
data into Postgres.

702
0:33:54,92 --> 0:33:59,64
And they're using 120 shards on
24 nodes, which divides evenly.

703
0:34:0,24 --> 0:34:7,06
Now, if you do a COPY of 50, 000
records, a single connection,

704
0:34:7,06 --> 0:34:10,04
so a single connection tries to
ingest 50, 000 records via a

705
0:34:10,04 --> 0:34:13,22
COPY, that COPY is going to create
24 sub copies.

706
0:34:14,16 --> 0:34:18,96
Because by law of probability,
all 24 shards are going to receive

707
0:34:18,96 --> 0:34:19,46
data.

708
0:34:19,64 --> 0:34:21,98
1/24th of the 50, 000 records.

709
0:34:22,38 --> 0:34:25,58
Because the way that data is sharded
is by taking a hash of the

710
0:34:25,58 --> 0:34:29,86
primary key, and that hash is a
random, let's say, it's randomly

711
0:34:29,86 --> 0:34:31,78
distributed, so it's going to touch
all the shards.

712
0:34:31,96 --> 0:34:36,14
So 1 connection from 1 client now
will create 24 connections.

713
0:34:36,38 --> 0:34:39,46
Citus implements this very well,
just do 24 sub copies.

714
0:34:39,52 --> 0:34:42,72
But you can imagine the potential
multiplication of connections

715
0:34:42,72 --> 0:34:44,7
that may happen here under high
concurrency.

716
0:34:45,48 --> 0:34:50,04
But by applying proper tuning and
doing proper connection management,

717
0:34:50,14 --> 0:34:51,42
this works very well.

718
0:34:51,5 --> 0:34:52,78
Nik: What is proper here?

719
0:34:53,68 --> 0:34:56,02
Álvaro: Oh, well, it's not an easy
exercise.

720
0:34:56,26 --> 0:35:0,06
The connections from the coordinator
or query routers to the

721
0:35:0,06 --> 0:35:2,98
worker nodes, they're bound by
several parameters.

722
0:35:3,42 --> 0:35:6,98
Essentially, Citus has some kind
of, let's say, connection pooling

723
0:35:7,04 --> 0:35:11,2
system to communicate between the
coordinator or coordinator

724
0:35:11,2 --> 0:35:12,02
to the workers.

725
0:35:12,4 --> 0:35:16,94
But it's not like PgBouncer, because
they open backends that

726
0:35:16,94 --> 0:35:19,78
are Citus is Postgres, for the
good and the bad.

727
0:35:19,84 --> 0:35:22,32
There's many good things and a
few bad things.

728
0:35:22,66 --> 0:35:26,84
And so if there's a client talking
to the coordinator, that creates

729
0:35:26,84 --> 0:35:28,52
a backend on the coordinator, right?

730
0:35:28,52 --> 0:35:30,14
And that backend needs to be open.

731
0:35:30,14 --> 0:35:34,24
Now, this backend may now own multiple
sockets to multiple connections

732
0:35:34,28 --> 0:35:35,04
to the workers.

733
0:35:35,06 --> 0:35:38,3
And there's several parameters
to tune whether you want a single

734
0:35:38,3 --> 0:35:41,32
connection per worker, whether
you want a number of connections

735
0:35:41,32 --> 0:35:41,88
per worker.

736
0:35:41,88 --> 0:35:45,84
You can cache connections already
open per worker, but then there's

737
0:35:45,84 --> 0:35:49,64
also like a global pool so that
you don't want to do more than

738
0:35:49,64 --> 0:35:52,76
X number of connections for a given
coordinator across all the

739
0:35:52,76 --> 0:35:53,26
workers.

740
0:35:53,56 --> 0:35:57,2
And so you need to really understand
that we all know that Postgres

741
0:35:57,66 --> 0:36:2,22
doesn't work well with a huge number
of connections because it

742
0:36:2,22 --> 0:36:3,5
creates a lot of backend.

743
0:36:3,9 --> 0:36:7,0
And so you need to understand the
fan-in rules, the fan-out rules,

744
0:36:7,08 --> 0:36:9,48
whether to put connection pool,
for example, in front of the

745
0:36:9,48 --> 0:36:11,5
coordinator or not, and so on.

746
0:36:11,5 --> 0:36:15,58
So the math ends up being non-trivial
sometimes, especially if

747
0:36:15,58 --> 0:36:17,32
you're talking about huge workloads.

748
0:36:17,44 --> 0:36:20,14
Like for small workloads, even
if you're starting with again,

749
0:36:20,14 --> 0:36:22,84
maybe hundreds of thousand transactions
per 2nd, maybe more or

750
0:36:22,84 --> 0:36:26,14
less trivial, like any server,
modern server will deal comfortably

751
0:36:26,14 --> 0:36:26,52
with that.

752
0:36:26,52 --> 0:36:29,84
But when you grow to these sizes,
you really need to size everything

753
0:36:29,92 --> 0:36:30,64
around this.

754
0:36:30,64 --> 0:36:34,64
But what's actually very fun of
this particular case is that

755
0:36:34,64 --> 0:36:37,64
as of today, on this particular
case, we'll need to blog about

756
0:36:37,64 --> 0:36:38,14
this.

757
0:36:38,24 --> 0:36:43,0
Right now, the database is perceived
as slow, but they're running

758
0:36:43,14 --> 0:36:45,96
some software that I think you're
familiar with called pg_ash,

759
0:36:47,08 --> 0:36:52,58
and it's reporting all the time,
waits are mostly ClientRead.

760
0:36:53,86 --> 0:36:57,2
They don't have an ingestion pipeline
fast enough to saturate

761
0:36:57,2 --> 0:36:57,88
the database.

762
0:36:58,26 --> 0:37:0,94
The database is faster than their
ingestion pipeline.

763
0:37:1,62 --> 0:37:5,14
Because again, COPY is really optimized
in Postgres and can

764
0:37:5,14 --> 0:37:6,98
ingest huge amounts of data.

765
0:37:7,18 --> 0:37:11,64
And we have here 24 shards, really
beefy servers that can ingest

766
0:37:11,64 --> 0:37:13,76
data at really, really high speeds.

767
0:37:13,94 --> 0:37:17,72
We've seen, don't remember the
numbers, but multiple gigabytes

768
0:37:17,72 --> 0:37:19,1
per 2nd being ingested.

769
0:37:19,34 --> 0:37:21,32
And the database is just fine.

770
0:37:21,94 --> 0:37:24,36
But the ingestion pipeline is not.

771
0:37:24,96 --> 0:37:25,82
Nik: That's interesting.

772
0:37:25,9 --> 0:37:29,8
So I see, I hear 3 different directions
here.

773
0:37:29,8 --> 0:37:33,72
1st of all, this benchmark in your
blog post, which is coming,

774
0:37:34,54 --> 0:37:37,44
I guess you will publish all the
details as usual, right?

775
0:37:38,2 --> 0:37:39,26
Álvaro: Yes, of course.

776
0:37:40,84 --> 0:37:44,2
Nik: This is our company's common
value, we share it a lot.

777
0:37:45,14 --> 0:37:47,84
Álvaro: Fully reproducible, StackGres
being open source, we'll

778
0:37:47,84 --> 0:37:49,14
open source all the code.

779
0:37:49,4 --> 0:37:52,76
Nik: It's Postgres's open source
and benchmarks should be open

780
0:37:52,76 --> 0:37:54,64
so anyone could repeat them.

781
0:37:55,16 --> 0:37:55,66
Yes.

782
0:37:55,9 --> 0:37:57,88
Yeah, so that's great.

783
0:37:58,58 --> 0:38:2,32
SELECT only because this is to
compare with different benchmark

784
0:38:2,32 --> 0:38:6,44
which is not so open, right, and
not about open source.

785
0:38:7,24 --> 0:38:8,04
That's great.

786
0:38:8,24 --> 0:38:11,76
And there, I guess, this is interesting
to continue exploring

787
0:38:12,04 --> 0:38:14,28
the limits.

788
0:38:14,44 --> 0:38:16,96
And I think maybe it will also...

789
0:38:17,1 --> 0:38:20,74
I expect more benchmarks to be
done because we have multiple

790
0:38:20,74 --> 0:38:23,85
systems competing in purely open
source areas.

791
0:38:23,85 --> 0:38:26,98
Some of them are AGPL, some of
them are Apache licenses, we mentioned

792
0:38:26,98 --> 0:38:27,48
already.

793
0:38:29,1 --> 0:38:31,74
It's interesting to start comparing
them.

794
0:38:31,88 --> 0:38:34,78
It's great that it's happening
and this is maybe the beginning

795
0:38:34,78 --> 0:38:36,24
of some comparison.

796
0:38:36,6 --> 0:38:37,94
It's time to start comparing.

797
0:38:38,74 --> 0:38:42,68
Right now your bet is fully on
Citus, like almost fully.

798
0:38:43,08 --> 0:38:46,66
I hear you that StackGres supports
other sharding as well.

799
0:38:46,86 --> 0:38:49,6
And I guess at some point you can
start supporting those new

800
0:38:49,6 --> 0:38:50,5
alternatives as well.

801
0:38:50,5 --> 0:38:51,04
Why not?

802
0:38:51,04 --> 0:38:53,8
It's just a Lego style thing, right?

803
0:38:53,8 --> 0:38:54,9
This is 1 direction.

804
0:38:56,82 --> 0:39:1,16
Read-only benchmarks, which I guess
it's not so exciting.

805
0:39:1,62 --> 0:39:4,94
2nd would be probably writing benchmarks
like COPY.

806
0:39:5,38 --> 0:39:9,3
When you say COPY, you mention
specific production case, but

807
0:39:9,3 --> 0:39:13,04
it would be great also to have
some benchmark showing, to start

808
0:39:13,04 --> 0:39:14,74
playing with and showing capabilities.

809
0:39:15,3 --> 0:39:17,38
Do you plan to have this benchmark?

810
0:39:18,26 --> 0:39:21,54
Álvaro: We could do that, but I
believe we will start by publishing

811
0:39:21,68 --> 0:39:25,58
some, the data that we can publish
about 1 particular case that

812
0:39:25,58 --> 0:39:28,66
we have already, like this 1, because
I think that already shows

813
0:39:28,66 --> 0:39:29,28
a lot.

814
0:39:30,06 --> 0:39:34,1
And so to be honest, we have
focused quite a bit on benchmarks.

815
0:39:34,6 --> 0:39:37,44
So let me be clear, we've done
2 different benchmarks.

816
0:39:37,44 --> 0:39:40,62
Both are going to be published
and will be fully reproducible

817
0:39:40,76 --> 0:39:41,96
back by code, right?

818
0:39:42,26 --> 0:39:44,18
And so full transparency on everything.

819
0:39:44,44 --> 0:39:48,04
The 1st 1 is a benchmark to demonstrate
the ceiling of the single

820
0:39:48,04 --> 0:39:51,34
coordinator and to see how we can
scale with query routers, right?

821
0:39:51,34 --> 0:39:54,52
Like I just mentioned the numbers,
it can go from 0.5, essentially

822
0:39:54,52 --> 0:39:58,24
500000 to 1.5 million, so almost
linear scalability by going

823
0:39:58,24 --> 0:40:1,02
from 1 coordinator to 3 query routers,
right?

824
0:40:1,46 --> 0:40:6,44
But then we saw actually that for
this particular workload, like

825
0:40:6,44 --> 0:40:10,96
very, very small primary key queries,
mostly in memory with a

826
0:40:10,96 --> 0:40:16,72
non-uniform distribution, that
you end up needing a similar number

827
0:40:16,72 --> 0:40:20,54
of cores on the routers layer than
on the shards or the, sorry,

828
0:40:20,54 --> 0:40:21,88
the workers layer, right?

829
0:40:22,12 --> 0:40:26,48
And so we did some tuning and around
this number that the ideal

830
0:40:26,48 --> 0:40:33,68
is to have 6 routers of 32 cores
each and 3 workers of 64 cores

831
0:40:33,68 --> 0:40:34,04
each.

832
0:40:34,04 --> 0:40:37,4
So that's the same number of cores
on both layers, but 6 routers

833
0:40:37,4 --> 0:40:38,36
and 3 workers.

834
0:40:38,52 --> 0:40:40,62
And we call this a unit of a scaling.

835
0:40:41,0 --> 0:40:44,84
That package, let's say it's well
tuned so that both layers are

836
0:40:44,96 --> 0:40:47,26
close to saturation but before
saturation.

837
0:40:47,26 --> 0:40:50,68
So they operate at the operational
regime and all math about

838
0:40:50,68 --> 0:40:53,54
connection, tuning connection,
pooling, and all that is well

839
0:40:53,54 --> 0:40:54,04
tuned.

840
0:40:54,34 --> 0:40:58,5
And that's basically delivering
1.5 million transactions per

841
0:40:58,5 --> 0:41:0,36
2nd, that unit of a scale.

842
0:41:0,72 --> 0:41:4,44
And then we decided to multiply
this unit of the scale by 2,

843
0:41:4,44 --> 0:41:5,9
by 4, and by 8.

844
0:41:6,34 --> 0:41:10,56
And this is a 2nd benchmark that
we'll be publishing that also

845
0:41:10,56 --> 0:41:13,78
compares to some data points of
another sharding solution, which

846
0:41:13,78 --> 0:41:16,02
has achieved those numbers and
more actually.

847
0:41:16,56 --> 0:41:20,24
And it also shows really good behavior,
like basically to share

848
0:41:20,24 --> 0:41:24,92
the numbers by multiplying this
by 8x, it passes over 10 million

849
0:41:24,92 --> 0:41:26,14
transactions per 2nd.

850
0:41:26,84 --> 0:41:33,74
With 1.something millisecond p50
latency and 6 milliseconds p99,

851
0:41:35,02 --> 0:41:37,32
at 10 million transactions per
2nd.

852
0:41:37,46 --> 0:41:41,62
To put this into perspective, who
needed to do 10 million transactions

853
0:41:41,68 --> 0:41:42,38
per 2nd?

854
0:41:42,5 --> 0:41:45,36
Probably just a few sites on earth
do more than that.

855
0:41:45,48 --> 0:41:46,88
Nik: So it's...

856
0:41:47,7 --> 0:41:48,48
Maybe more.

857
0:41:48,52 --> 0:41:50,66
We have gigantic workloads these
days.

858
0:41:50,74 --> 0:41:52,54
I think here I disagree.

859
0:41:52,96 --> 0:41:54,02
A lot of more.

860
0:41:54,38 --> 0:41:57,94
We see crazy startups which skyrocket
these days.

861
0:41:58,52 --> 0:42:0,04
And their users are agents.

862
0:42:0,62 --> 0:42:1,92
And they, like, just...

863
0:42:2,44 --> 0:42:4,94
Álvaro: Yeah, but most of the time
what I see is that agents

864
0:42:4,94 --> 0:42:5,88
are small databases.

865
0:42:5,9 --> 0:42:7,38
There are many of them, but a small...

866
0:42:7,38 --> 0:42:10,12
We're talking about single cluster,
right?

867
0:42:10,64 --> 0:42:15,92
But anyway, my point is that anyway,
reaching 10 million transactions

868
0:42:15,96 --> 0:42:19,82
per 2nd with very decent latency
numbers, right?

869
0:42:20,64 --> 0:42:22,46
It's an excellent result.

870
0:42:23,18 --> 0:42:26,56
It's home to most of the clusters
we've ever seen.

871
0:42:27,04 --> 0:42:29,68
And so it's a valid solution for
almost anyone.

872
0:42:29,68 --> 0:42:31,96
And That's actually my takeaway,
right?

873
0:42:31,96 --> 0:42:35,9
It can scale almost as much or
as much as you want with really

874
0:42:35,9 --> 0:42:37,62
decent numbers on this pure OLTP.

875
0:42:38,4 --> 0:42:40,6802
Nik: So I have 4 directions already.

876
0:42:40,6802 --> 0:42:42,1
So you will publish.

877
0:42:42,18 --> 0:42:43,6
I know you will publish this.

878
0:42:43,6 --> 0:42:45,54
You will probably will iterate
this.

879
0:42:45,82 --> 0:42:49,52
I guess this benchmark also not
cheap to run, right?

880
0:42:49,54 --> 0:42:51,8
You need to provision those resources.

881
0:42:52,1 --> 0:42:52,9
Álvaro: Yeah, exactly.

882
0:42:52,9 --> 0:42:54,18
But this is a good point.

883
0:42:54,44 --> 0:42:58,58
We're still a bootstrapped startup,
and we strive to be, we have

884
0:42:58,58 --> 0:43:0,4
to be, very cost efficient.

885
0:43:1,26 --> 0:43:5,02
So running this benchmark, I don't
have the final data, but it's

886
0:43:5,02 --> 0:43:5,78
very approximate.

887
0:43:6,34 --> 0:43:12,08
We've spent around $5, 000 to $6,
000 on running this benchmark,

888
0:43:12,16 --> 0:43:14,58
scaling to 10 million transactions
per 2nd.

889
0:43:14,58 --> 0:43:16,42
Nik: It's AWS or where?

890
0:43:16,84 --> 0:43:17,34
Álvaro: Yep.

891
0:43:17,78 --> 0:43:21,2
Nik: And you just use Spot because
it's not reliable in this

892
0:43:21,2 --> 0:43:21,7
case.

893
0:43:22,36 --> 0:43:26,12
Álvaro: Yeah, it's hard because,
and we had some problems when

894
0:43:26,12 --> 0:43:28,3
you deal with such a large number
of instances.

895
0:43:28,4 --> 0:43:33,22
So on the 10 million transactions
per 2nd, this involved 72 instances

896
0:43:33,28 --> 0:43:36,98
plus the drivers for the workload,
so like around 90 instances.

897
0:43:37,54 --> 0:43:39,74
Just 1 instance behaving wrongly?

898
0:43:40,92 --> 0:43:43,3
Nik: The probability of failure
is too high.

899
0:43:44,34 --> 0:43:44,84
Álvaro: Yeah.

900
0:43:45,12 --> 0:43:48,74
So, yeah, Spot would have most
likely had mean that the test

901
0:43:48,74 --> 0:43:50,72
needs to be repeated over and over
again.

902
0:43:51,76 --> 0:43:52,4
Nik: I see.

903
0:43:52,54 --> 0:43:52,96
That's great.

904
0:43:52,96 --> 0:43:55,82
And that's great cost-wise as well
for 10 million.

905
0:43:55,96 --> 0:43:59,36
So once again, 1 direction is SELECT
only primary key lookup,

906
0:43:59,38 --> 0:44:0,92
trivial workload, I would say.

907
0:44:1,56 --> 0:44:6,94
And I think this is very adjacent
to our benchmarks we did multiple

908
0:44:6,94 --> 0:44:11,04
times for latency overhead is very
in this area, like very fast

909
0:44:11,04 --> 0:44:11,54
queries.

910
0:44:11,6 --> 0:44:15,36
2nd, I still think at some point
there should be some benchmarks

911
0:44:15,58 --> 0:44:18,74
for INSERTs or write workloads,
right?

912
0:44:19,64 --> 0:44:23,64
3rd thing is that you mentioned
many connections is a trouble.

913
0:44:24,52 --> 0:44:30,36
I think, I know Andres Freund did
a lot of great work in Postgres

914
0:44:30,36 --> 0:44:35,08
13 and especially I think 14 to
improve the situation when you

915
0:44:35,08 --> 0:44:36,6
have thousands of connections.

916
0:44:36,98 --> 0:44:42,16
Because before that, thousands
of idle connections affected performance

917
0:44:42,18 --> 0:44:42,9
so badly.

918
0:44:42,94 --> 0:44:44,08
Now it's not so bad.

919
0:44:44,38 --> 0:44:45,64
What does it mean still?

920
0:44:46,5 --> 0:44:48,02
We don't have threads, right?

921
0:44:48,58 --> 0:44:52,62
What's happening if you have 10,
000 connections on single Postgres?

922
0:44:52,72 --> 0:44:54,94
I think this research is very interesting.

923
0:44:54,96 --> 0:44:58,2
I'm talking for myself because
we have a lot of interesting directions.

924
0:44:58,22 --> 0:44:59,44
We have benchmarks.

925
0:44:59,96 --> 0:45:2,82
Our focus mostly on single node
clusters.

926
0:45:3,54 --> 0:45:7,68
Our company is dealing with some
self-implementation, like application

927
0:45:7,74 --> 0:45:8,54
side sharding.

928
0:45:9,0 --> 0:45:11,98
But it's so interesting, I see
some interesting directions here

929
0:45:11,98 --> 0:45:13,22
to benchmark further.

930
0:45:13,82 --> 0:45:19,08
And finally, I'm thinking if you
stop at 10 million, I'm going

931
0:45:19,08 --> 0:45:25,4
to explore the possibility to have
10 million TPS primary key

932
0:45:25,4 --> 0:45:27,42
lookups for a single node.

933
0:45:28,08 --> 0:45:32,06
This probably will encourage you
to go to 50 million or something,

934
0:45:32,28 --> 0:45:34,04
spend a few thousand dollars.

935
0:45:34,14 --> 0:45:38,86
Because if, yeah, I will put it
to midterm to-do list.

936
0:45:39,0 --> 0:45:41,44
If we achieve that, you will be
forced

937
0:45:41,44 --> 0:45:42,02
Álvaro: to cover.

938
0:45:42,02 --> 0:45:44,7
But you know, if you want to do
that, you know, you only have

939
0:45:44,7 --> 0:45:45,56
2 weeks, right?

940
0:45:46,4 --> 0:45:48,48
Nik: No, no, no, no, no.

941
0:45:48,48 --> 0:45:50,02
Go ahead, publish your results.

942
0:45:50,02 --> 0:45:53,2
This is an interesting, friendly
competition and I think some

943
0:45:53,3 --> 0:45:56,3
systems will need it to join.

944
0:45:56,32 --> 0:45:58,58
What I like about this is that
data is shared.

945
0:45:58,58 --> 0:46:0,22
It's about open source components.

946
0:46:0,46 --> 0:46:1,72
Everything is open source.

947
0:46:1,74 --> 0:46:2,78
Data is open source.

948
0:46:2,78 --> 0:46:3,74
Benchmarks are open source.

949
0:46:3,74 --> 0:46:4,72
They can be revisited.

950
0:46:4,74 --> 0:46:5,42
This is great.

951
0:46:5,42 --> 0:46:7,86
This is very good style of competition.

952
0:46:8,3 --> 0:46:10,06
Everyone benefits from, right?

953
0:46:11,42 --> 0:46:14,34
So I'm not going to chase you right
now.

954
0:46:14,34 --> 0:46:16,24
I'm just saying maybe later.

955
0:46:16,78 --> 0:46:22,06
Álvaro: No, I'm very open if anyone
wants to sponsor workloads

956
0:46:22,16 --> 0:46:26,12
to be tested on Citus and see how
Citus can scale to from 10

957
0:46:26,12 --> 0:46:30,42
to 50 million transactions per
2nd or to read-write workloads We

958
0:46:30,42 --> 0:46:32,72
would be more than happy to help
in that regards.

959
0:46:33,48 --> 0:46:34,12
Michael: So yeah.

960
0:46:34,7 --> 0:46:38,24
I think it's, I think what, from
1 of the charts that you sent

961
0:46:38,24 --> 0:46:43,54
over, it's quite an interesting
chart of how the throughput changes

962
0:46:43,58 --> 0:46:44,44
as you scale up.

963
0:46:44,44 --> 0:46:46,88
So I think if you, if you, you
mentioned these units of work,

964
0:46:46,88 --> 0:46:52,06
if, if you add a 2nd unit of work
and the TPS exactly doubles

965
0:46:52,06 --> 0:46:56,5
if it's linear like as you add
more units it's very easy to extrapolate

966
0:46:56,98 --> 0:47:0,84
that of course it can get to 50
I just need 5 times more but

967
0:47:0,84 --> 0:47:5,22
if it isn't if it isn't quite efficient
if you lose 10 each time

968
0:47:5,22 --> 0:47:10,66
if it if it feels like at doubling
the instance size only 1.5x

969
0:47:11,32 --> 0:47:15,26
is the TPS, then it's less believable
that it can scale infinitely.

970
0:47:15,72 --> 0:47:17,78
What kind of numbers were you seeing
there?

971
0:47:19,3 --> 0:47:20,86
Álvaro: So that's a really good
question.

972
0:47:20,86 --> 0:47:25,78
Scaling is slightly sublinear,
with most systems are.

973
0:47:26,2 --> 0:47:32,72
And on our scaling from 1 to 8,
it was 1.94 every time we doubled.

974
0:47:34,06 --> 0:47:34,56
Nik: Okay.

975
0:47:34,64 --> 0:47:35,14
Yeah.

976
0:47:35,5 --> 0:47:39,32
There is a big challenge of scaling
client in this case.

977
0:47:39,38 --> 0:47:39,88
Yes.

978
0:47:40,52 --> 0:47:41,34
That's it.

979
0:47:41,55 --> 0:47:42,44
I mean, I

980
0:47:42,44 --> 0:47:43,86
Álvaro: can go very deep into this
topic.

981
0:47:43,86 --> 0:47:45,14
I guess it's not the moment.

982
0:47:45,48 --> 0:47:49,4
But it's a really, really interesting
capacity planning we had

983
0:47:49,4 --> 0:47:50,58
to do to achieve these numbers.

984
0:47:50,58 --> 0:47:54,52
Again, it sounds like, oh, just
put more nodes and yeah, you

985
0:47:54,52 --> 0:47:56,9
scale 10 million, you just need
to hit enter, right?

986
0:47:56,98 --> 0:47:58,28
No, it wasn't that case.

987
0:47:58,32 --> 0:47:59,84
We had to do a lot of tuning.

988
0:48:0,06 --> 0:48:2,86
The number of connections is actually
mind-blowing because even

989
0:48:2,86 --> 0:48:6,24
though these queries are really
fast and take hundreds of microseconds

990
0:48:6,42 --> 0:48:10,38
to run in reality across all the
fleet, across the CPU aggregated

991
0:48:10,38 --> 0:48:11,64
all the fleet, right?

992
0:48:11,72 --> 0:48:14,02
The number of connections that
you need, the sessions that you

993
0:48:14,02 --> 0:48:16,02
need to drive this is very, very
large.

994
0:48:16,02 --> 0:48:18,46
It's many, many thousands of sessions.

995
0:48:18,82 --> 0:48:23,48
And then they multiply because
if each query router is, I think

996
0:48:23,48 --> 0:48:27,1
where our optimal regime was around
400 connections per query

997
0:48:27,1 --> 0:48:30,7
router on 32 cores, which by the
way, if you look at the numbers,

998
0:48:30,7 --> 0:48:34,64
it's always like the usual rule
for tuning Postgres, right?

999
0:48:34,64 --> 0:48:38,22
The maximum number of connections
is a multiple of the number

1000
0:48:38,22 --> 0:48:38,6
of cores.

1001
0:48:38,6 --> 0:48:41,4
In this case, the multiple is around
7 because they're very,

1002
0:48:41,4 --> 0:48:44,58
very CPU intensive queries because
they almost not touch disk,

1003
0:48:44,58 --> 0:48:45,02
right?

1004
0:48:45,02 --> 0:48:46,88
So the multiple is a bit high.

1005
0:48:46,88 --> 0:48:50,32
Anyway, do you think of 400 connections
coming to each query

1006
0:48:50,32 --> 0:48:54,52
router of which there are 48 on
the 10 million transactions per

1007
0:48:54,52 --> 0:48:56,1
2nd run, right?

1008
0:48:56,5 --> 0:49:2,06
And now each 1 is gonna have outgoing
the same number of connections

1009
0:49:2,58 --> 0:49:5,22
in the same number of sessions,
because they're all, but they

1010
0:49:5,22 --> 0:49:6,52
all go to all shards.

1011
0:49:7,2 --> 0:49:12,18
So the incoming connections at
the shards are the whole fleet.

1012
0:49:12,18 --> 0:49:15,36
So you're seeing the 400 multiplied
by 48.

1013
0:49:15,86 --> 0:49:18,3
Those are incoming connections
to every worker.

1014
0:49:19,7 --> 0:49:23,44
So to actually deal with this,
we had to introduce connection

1015
0:49:23,54 --> 0:49:25,5
pooling before the workers.

1016
0:49:26,32 --> 0:49:30,06
And it's also been an exercise
because StackGres by default uses

1017
0:49:30,06 --> 0:49:33,28
PgBouncer, and PgBouncer is single-threaded.

1018
0:49:33,92 --> 0:49:36,24
So were we saturating 1 thread?

1019
0:49:36,54 --> 0:49:37,04
Definitely.

1020
0:49:37,72 --> 0:49:40,9
So actually this benchmark required,
if I'm not mistaken, 15

1021
0:49:40,9 --> 0:49:45,84
PgBouncers per worker, running
with a SO_REUSEPORT functionality

1022
0:49:46,42 --> 0:49:49,94
and all handled by StackGres as
a custom container.

1023
0:49:50,42 --> 0:49:53,94
So it worked beautifully well,
but you need to do all the dimensioning

1024
0:49:54,14 --> 0:49:56,78
correctly, otherwise it will not
achieve these numbers either.

1025
0:49:57,44 --> 0:50:0,56
Michael: Did you say 15 PgBouncers
per Worker?

1026
0:50:1,34 --> 0:50:1,84
Álvaro: Yep.

1027
0:50:1,86 --> 0:50:2,72
Michael: Oh my goodness.

1028
0:50:4,3 --> 0:50:4,8
Okay.

1029
0:50:5,14 --> 0:50:5,64
Okay.

1030
0:50:5,74 --> 0:50:6,24
Álvaro: Good.

1031
0:50:6,34 --> 0:50:9,68
Because we're dealing with, I think,
19, 000 incoming connections.

1032
0:50:11,76 --> 0:50:15,2
And they're all sending queries
at a really high pace.

1033
0:50:15,86 --> 0:50:18,98
Nik: What would it take to reach
100 million TPS?

1034
0:50:20,32 --> 0:50:22,22
Álvaro: Well, there's 2 things.

1035
0:50:22,48 --> 0:50:24,24
1 is just to keep multiplying.

1036
0:50:24,28 --> 0:50:28,58
I think this unit of scale is very
well tuned as of now, and

1037
0:50:28,58 --> 0:50:29,94
it just keep multiplying.

1038
0:50:30,14 --> 0:50:33,4
As of now, it should be an easy
exercise, other than money-wise.

1039
0:50:34,42 --> 0:50:38,48
But I believe the scalability is
gonna start going, is gonna

1040
0:50:38,48 --> 0:50:40,82
start going less and less over
time.

1041
0:50:41,12 --> 0:50:44,0
And this is, again, because of
how Citus works.

1042
0:50:44,48 --> 0:50:45,94
As I said, it's not perfect.

1043
0:50:46,28 --> 0:50:48,48
You may need to question whether
you need more than 10 million

1044
0:50:48,48 --> 0:50:49,66
transactions per 2nd.

1045
0:50:50,28 --> 0:50:53,62
But it's, so Citus is gonna create
a backend per session, per

1046
0:50:53,62 --> 0:50:54,12
router.

1047
0:50:54,86 --> 0:50:57,78
And this, again, we were just discussing
this, is gonna create,

1048
0:50:57,8 --> 0:51:1,36
that multiplied by the number of
routers will become the number

1049
0:51:1,36 --> 0:51:3,42
of incoming connections to each
worker node.

1050
0:51:3,42 --> 0:51:8,26
So instead of having 19, 200 connections,
if we do this 10 times,

1051
0:51:8,42 --> 0:51:12,56
that will require almost 200, 000
connections incoming per node.

1052
0:51:12,56 --> 0:51:13,94
Not all of them are active.

1053
0:51:14,24 --> 0:51:17,08
Are active actually are that number
divided by the number of

1054
0:51:17,08 --> 0:51:21,6
workers, but still a very high
number, a very, very, very fun

1055
0:51:21,6 --> 0:51:24,34
difference between the fan-in,
the number of connections incoming

1056
0:51:24,34 --> 0:51:27,52
to the workers through the poolers
versus the number of outgoing

1057
0:51:27,52 --> 0:51:30,96
connections, the pool connections
to Postgres, which needs to

1058
0:51:30,96 --> 0:51:35,82
remain constant at around 600 in
the case of the workers or 800,

1059
0:51:35,94 --> 0:51:38,34
something like this, because they
have double the CPUs, right?

1060
0:51:38,94 --> 0:51:43,7
And this difference between this
fan-in and fan-out creates inefficiency

1061
0:51:44,28 --> 0:51:45,74
in the workers themselves.

1062
0:51:45,86 --> 0:51:47,74
So it will start saturating earlier.

1063
0:51:47,9 --> 0:51:51,02
They also create inefficiencies
in the routers because they need

1064
0:51:51,02 --> 0:51:52,22
to talk to more nodes.

1065
0:51:53,08 --> 0:51:55,46
And so, will it reach there?

1066
0:51:55,68 --> 0:51:56,6
I don't know.

1067
0:51:56,72 --> 0:52:0,04
If someone wants to pour the money
into it, we can test it out.

1068
0:52:0,04 --> 0:52:4,64
I believe it may reach to a very
high number, maybe with a slightly

1069
0:52:4,64 --> 0:52:9,3
less efficiency, but it will reach
maybe 50 million, maybe more

1070
0:52:9,76 --> 0:52:12,04
with more effort, but it will do
so.

1071
0:52:12,18 --> 0:52:16,64
It's still compared to other sharding
solutions that can reach

1072
0:52:16,64 --> 0:52:17,38
those numbers.

1073
0:52:17,56 --> 0:52:21,24
I don't think all can, but out
of those that we have tested,

1074
0:52:21,76 --> 0:52:25,64
it doing so, this 10 million transactions
per 2nd, is using 0.5

1075
0:52:25,64 --> 0:52:29,68
of the silicon compared to other
sharding solutions.

1076
0:52:30,06 --> 0:52:32,96
So even though the scaling may
be slightly less linear because

1077
0:52:32,96 --> 0:52:36,82
of this Citus connection model,
which is Postgres connection

1078
0:52:36,82 --> 0:52:41,26
model, it needed 0.5 of the cores
to achieve those results.

1079
0:52:41,82 --> 0:52:44,52
So it's still very impressive,
right?

1080
0:52:44,76 --> 0:52:47,64
Now, Do I see a future different
than this?

1081
0:52:47,64 --> 0:52:48,3
I do.

1082
0:52:48,54 --> 0:52:53,46
I believe there may be an option
where the transport that Citus

1083
0:52:53,46 --> 0:52:57,98
uses could benefit from a multiplexing
transport, something like

1084
0:52:57,98 --> 0:53:2,06
gRPC, which other of the modern
solutions are adopting.

1085
0:53:3,34 --> 0:53:7,86
And that significantly reduces
the fan-in and fan-out connection

1086
0:53:7,9 --> 0:53:9,78
problem from the routers to the
workers.

1087
0:53:10,84 --> 0:53:15,38
So if I were to complete my initial
thesis that Citus is a great

1088
0:53:15,38 --> 0:53:20,14
solution as of today that is very
mature, well proven, featureful,

1089
0:53:20,42 --> 0:53:24,86
and so on, and that new solutions
may take time to mature.

1090
0:53:25,88 --> 0:53:30,68
Maybe, maybe, and it's a big if,
by just improving a bit Citus

1091
0:53:30,68 --> 0:53:34,6
architecture, it could take sharding
even to the next level if

1092
0:53:34,6 --> 0:53:38,3
needed, given that already it reaches
levels that are more than

1093
0:53:38,3 --> 0:53:41,76
enough for mostly everybody, but
even for those who may need

1094
0:53:41,82 --> 0:53:46,12
an even bigger scale, then maybe
It's just a matter of improving

1095
0:53:46,12 --> 0:53:46,62
Citus.

1096
0:53:47,54 --> 0:53:50,64
Nik: Yeah, I have only 1 question
to ask in the end.

1097
0:53:50,64 --> 0:53:55,78
I know you're comparing to other
benchmarks, and I'm just curious

1098
0:53:55,92 --> 0:53:59,48
in terms of, I saw some pictures
you shared with us before this

1099
0:53:59,48 --> 0:54:2,76
recording that was involving money,
like some dozens of thousands

1100
0:54:2,76 --> 0:54:4,7
of dollars to have this cluster.

1101
0:54:5,02 --> 0:54:9,4
I'm very curious, do you compare
with humans involved with support

1102
0:54:9,44 --> 0:54:14,36
or just pure open source and plus
cloud costs and that's it?

1103
0:54:15,78 --> 0:54:20,0
Álvaro: Yeah, so I did 2 comparisons
and they will be clearly

1104
0:54:20,14 --> 0:54:23,04
identified in the blog post that
will announce these benchmarks.

1105
0:54:23,8 --> 0:54:28,3
And so there's the pure infrastructure
cost, which is silicon,

1106
0:54:28,58 --> 0:54:32,06
like number of CPUs, where essentially
it's like around 55%.

1107
0:54:33,48 --> 0:54:36,18
So almost 0.5 the cost in terms
of CPU.

1108
0:54:36,68 --> 0:54:41,42
But then we can compute the total
cost to be very clear.

1109
0:54:41,42 --> 0:54:43,92
And there's nothing here to hide.

1110
0:54:44,06 --> 0:54:47,88
I'm comparing right now with a
benchmark that was run and published

1111
0:54:47,88 --> 0:54:48,54
by Neki.

1112
0:54:49,34 --> 0:54:52,86
And Neki is a managed service,
and they publish their list price.

1113
0:54:52,86 --> 0:54:55,36
And their list price is not the
instance cost.

1114
0:54:55,52 --> 0:54:59,16
It is the instance cost plus their
managing fee or their margin.

1115
0:55:0,04 --> 0:55:4,02
And when comparing with that, the
cost is not 0.5.

1116
0:55:4,3 --> 0:55:8,7
It's depending on what you consider
is between 0.333 or 0.2.

1117
0:55:8,94 --> 0:55:12,5
Actually, a well-dimensioned system
should be just 5 times cheaper.

1118
0:55:12,5 --> 0:55:16,08
Basically, 0.333 is if you compare
on-demand cost, which nobody

1119
0:55:16,08 --> 0:55:20,3
should run like a production cluster
sharded on on-demand cost.

1120
0:55:20,42 --> 0:55:24,44
And if you use a basic saving plans,
then you'll be 5 times cheaper.

1121
0:55:24,56 --> 0:55:25,06
Yeah.

1122
0:55:25,08 --> 0:55:28,16
And this is obviously a managed
service to a software comparison.

1123
0:55:28,58 --> 0:55:32,6
You may discuss that there may
be humans involved in the total

1124
0:55:32,66 --> 0:55:34,3
TCO calculation.

1125
0:55:35,02 --> 0:55:39,56
But my take is that a managed service
and software that does

1126
0:55:39,56 --> 0:55:42,42
a managed service for you is not
so much different.

1127
0:55:42,44 --> 0:55:46,22
Actually, I call StackGres a managed
software, a managed database

1128
0:55:46,22 --> 0:55:49,92
software, because when you say
a managed service, there's no

1129
0:55:49,92 --> 0:55:51,4
human typing, right?

1130
0:55:51,96 --> 0:55:55,44
There may be, if there's a support
ticket or something happens,

1131
0:55:55,44 --> 0:55:57,44
but the operation is just software.

1132
0:55:57,44 --> 0:56:0,36
So it's essentially the same for
me.

1133
0:56:0,8 --> 0:56:8,94
Nik: I know some humans can send
kill -9 in very big companies offering

1134
0:56:8,94 --> 0:56:10,66
managed services sometimes.

1135
0:56:11,0 --> 0:56:13,12
That kind of support is not welcome.

1136
0:56:13,94 --> 0:56:15,04
Not welcome, right?

1137
0:56:15,04 --> 0:56:16,9
kill -9 from support.

1138
0:56:18,28 --> 0:56:21,82
Michael: I also think these machines
are so big, like we're talking

1139
0:56:21,82 --> 0:56:25,36
about sharding and we're talking
about multiple huge machines

1140
0:56:25,46 --> 0:56:26,82
really, like they're expensive.

1141
0:56:27,34 --> 0:56:28,78
Often you don't pay this.

1142
0:56:28,78 --> 0:56:30,26
If you're actually a company,

1143
0:56:30,4 --> 0:56:30,9
Nik: saving

1144
0:56:31,1 --> 0:56:34,24
Michael: those machines, you would,
you would have like savings

1145
0:56:34,24 --> 0:56:37,64
plans either through PlanetScale
and or bring your own cloud

1146
0:56:37,64 --> 0:56:38,2
type thing.

1147
0:56:38,2 --> 0:56:41,32
Nik: So you're comparing not apples
to apples, but the numbers

1148
0:56:41,32 --> 0:56:46,2
you provided on your side, it could
be less with savings plan

1149
0:56:46,26 --> 0:56:50,24
if you pay in advance some number
of years to AWS.

1150
0:56:50,66 --> 0:56:53,76
On the other hand, you need probably
to roughly multiply it by

1151
0:56:53,76 --> 0:56:59,04
2, thinking about humans and the
support needed to surround this

1152
0:56:59,04 --> 0:57:0,04
solution, right?

1153
0:57:0,54 --> 0:57:1,22
Very roughly.

1154
0:57:2,78 --> 0:57:4,24
Álvaro: Yeah, so that's what I
said.

1155
0:57:4,24 --> 0:57:6,82
And so 1st of all, that's why I
want to give both numbers.

1156
0:57:6,82 --> 0:57:10,12
So you can 1st of all, compare
from the pure Silicon perspective

1157
0:57:10,12 --> 0:57:13,6
to pure architecture perspective,
and then from the sale price.

1158
0:57:13,94 --> 0:57:18,42
But as I said, For me, a managed
software, it's just software.

1159
0:57:18,58 --> 0:57:20,52
A managed service is just software,
right?

1160
0:57:20,86 --> 0:57:23,72
And if the software automates and
runs everything for you, which

1161
0:57:23,72 --> 0:57:25,58
is the case in both cases, right?

1162
0:57:25,58 --> 0:57:30,18
Because all managed services, there's
no humans doing operations,

1163
0:57:30,36 --> 0:57:31,4
It's only software.

1164
0:57:31,4 --> 0:57:35,1
So humans only intervene in the
case of issues.

1165
0:57:35,9 --> 0:57:37,36
And in those cases, yes.

1166
0:57:37,36 --> 0:57:41,48
And you can get like a, we sell
enterprise, the StackGres Enterprise,

1167
0:57:41,48 --> 0:57:42,6
which is exactly that.

1168
0:57:42,6 --> 0:57:44,32
Support for those cases, right?

1169
0:57:45,1 --> 0:57:47,56
And that's something optional,
but that's something you would

1170
0:57:47,56 --> 0:57:49,02
add on top of that.

1171
0:57:49,02 --> 0:57:51,6
And then the calculation will not
be 5 times cheaper, it will

1172
0:57:51,6 --> 0:57:52,08
be

1173
0:57:52,08 --> 0:57:52,66
Nik: maybe, I

1174
0:57:52,66 --> 0:57:54,86
Álvaro: don't have the numbers
exactly, 3 times cheaper, 4 times

1175
0:57:54,86 --> 0:57:57,08
cheaper, but still obviously much
cheaper.

1176
0:57:57,44 --> 0:57:59,94
Nik: Yeah, thank you for this clarification,
that's interesting.

1177
0:58:0,48 --> 0:58:3,48
Álvaro: And sorry, also to add,
and that's because talking about

1178
0:58:3,48 --> 0:58:6,66
cloud, but if you talk about on-prem,
which is definitely in

1179
0:58:6,66 --> 0:58:10,06
the space where we also, the StackGres
works anywhere on-prem,

1180
0:58:10,32 --> 0:58:14,08
then the math is even much more
into your favor.

1181
0:58:14,84 --> 0:58:15,53
Nik: Right, if you own hardware.

1182
0:58:15,53 --> 0:58:15,72
If you own

1183
0:58:15,72 --> 0:58:18,92
Álvaro: hardware and data centers,

1184
0:58:19,28 --> 0:58:23,3
Nik: then it's open source, you
can troubleshoot with your engineering

1185
0:58:23,36 --> 0:58:24,34
team if needed.

1186
0:58:24,34 --> 0:58:24,68
That's

1187
0:58:24,68 --> 0:58:25,18
Álvaro: great.

1188
0:58:25,765 --> 0:58:26,265
Thank

1189
0:58:26,85 --> 0:58:29,44
Nik: you so much.

1190
0:58:29,44 --> 0:58:30,12
Very interesting.

1191
0:58:30,18 --> 0:58:34,68
And also I see several paths for
future work.

1192
0:58:35,06 --> 0:58:38,46
Maybe we should have another episode
at some point in the future

1193
0:58:38,76 --> 0:58:41,52
to revisit the topics in some directions.

1194
0:58:41,6 --> 0:58:42,26
Álvaro: Any time.

1195
0:58:42,44 --> 0:58:44,98
I would be more than happy to come
back.

1196
0:58:45,66 --> 0:58:46,4
Nik: Thank you so much.

1197
0:58:46,4 --> 0:58:47,26
It was interesting.

1198
0:58:47,72 --> 0:58:49,02
Very interesting to me.

1199
0:58:49,6 --> 0:58:51,04
Álvaro: Thank you for inviting
me.