1 00:00:00,060 --> 00:00:02,440 Michael: Hello and welcome to Postgres.FM, a weekly show about 2 00:00:02,440 --> 00:00:03,300 all things PostgreSQL. 3 00:00:03,480 --> 00:00:05,740 I am Michael, founder of pgMustard, and I'm joined as usual 4 00:00:05,740 --> 00:00:08,100 by my co-host Nikolay, founder of Postgres.AI. 5 00:00:08,100 --> 00:00:08,800 Hey Nikolay. 6 00:00:09,340 --> 00:00:10,460 Nikolay: Hi Michael, hi. 7 00:00:11,000 --> 00:00:14,360 Michael: And today we are joined by special guest Gwen Shapira, 8 00:00:14,380 --> 00:00:17,440 co-founder and chief product officer at Nile, to talk all things 9 00:00:17,440 --> 00:00:17,940 multi-tenancy. 10 00:00:18,600 --> 00:00:20,580 Hello Gwen, thank you for joining us. 11 00:00:21,020 --> 00:00:23,600 Gwen: Thank you for having me, it's very exciting for me to be 12 00:00:23,600 --> 00:00:24,480 in your show. 13 00:00:24,760 --> 00:00:26,940 Michael: Oh, we're excited to have you, thank you. 14 00:00:27,240 --> 00:00:30,860 So to start perhaps you could give us a little bit of background 15 00:00:30,860 --> 00:00:33,940 and what got you interested in this topic of multi-tenancy 16 00:00:34,040 --> 00:00:34,700 in general? 17 00:00:34,960 --> 00:00:40,720 Gwen: As many things it started with an incident but this time 18 00:00:40,720 --> 00:00:46,440 not actually one of mine, so when my co-founder and I started Nile 19 00:00:46,440 --> 00:00:48,660 we actually started with a very different idea. 20 00:00:49,440 --> 00:00:52,900 After about 9 months, we were like, this idea is not working 21 00:00:52,900 --> 00:00:53,460 very well. 22 00:00:53,460 --> 00:00:56,520 We developed some things, we're not finding the market we hoped 23 00:00:56,520 --> 00:00:57,180 to find. 24 00:00:57,560 --> 00:01:02,580 We were sitting in the Hacker Space over in Mountain View, and 25 00:01:02,580 --> 00:01:05,500 we're like, what did we learn? 26 00:01:05,500 --> 00:01:09,560 We talked to 200 companies all doing SaaS. 27 00:01:09,640 --> 00:01:12,240 What have we found out as a result? 28 00:01:13,520 --> 00:01:20,040 And the things that called to us is that very early on, basically 29 00:01:20,340 --> 00:01:24,020 when the first lines of code are getting written, you have to 30 00:01:24,020 --> 00:01:25,820 choose a multi-tenancy model. 31 00:01:26,820 --> 00:01:32,120 And then about 2, 3 or 4 years later, depending on how fast you're 32 00:01:32,120 --> 00:01:35,100 growing, you have to change it. 33 00:01:36,380 --> 00:01:41,760 And we heard a lot of stories on what caused people to change 34 00:01:41,760 --> 00:01:46,220 it and whether they regret earlier choices or they're like, we 35 00:01:46,220 --> 00:01:50,020 didn't know better and how things went for them. 36 00:01:50,020 --> 00:01:55,200 And then we started looking in different blogs and we found so 37 00:01:55,200 --> 00:02:02,800 many by very famous companies with either incidents where something 38 00:02:02,800 --> 00:02:07,540 that was done to a single tenant caused a whole chain of events 39 00:02:07,540 --> 00:02:11,260 that took down their entire system, sometimes for days. 40 00:02:13,320 --> 00:02:18,100 And also a lot of slightly better stories, how we sharded our 41 00:02:18,280 --> 00:02:19,780 highly multi-tenant database. 42 00:02:20,740 --> 00:02:26,600 And we found story after story after story on how people had 43 00:02:26,600 --> 00:02:29,620 to re-architect their entire database, which is, as you guys 44 00:02:29,620 --> 00:02:33,480 know, extremely painful to do after you're a successful company 45 00:02:33,480 --> 00:02:34,580 3 years in. 46 00:02:35,380 --> 00:02:37,780 And we're like, this is a good problem. 47 00:02:38,460 --> 00:02:39,640 So many people have it. 48 00:02:39,640 --> 00:02:40,740 It is so common. 49 00:02:41,040 --> 00:02:46,640 My past was in databases, not as much Postgres, more Oracle and 50 00:02:46,640 --> 00:02:47,140 MySQL. 51 00:02:47,900 --> 00:02:52,340 But I've seen this problem again and again in all kinds of companies. 52 00:02:52,680 --> 00:02:56,280 My co-founders seen this problem again and again in all kinds 53 00:02:56,280 --> 00:02:57,240 of companies. 54 00:02:57,660 --> 00:02:59,040 This is such a good problem. 55 00:02:59,040 --> 00:03:01,220 Everyone has it and nobody's working on it. 56 00:03:01,220 --> 00:03:03,140 Why is nobody working on it? 57 00:03:04,340 --> 00:03:07,220 And that's how we got into it. 58 00:03:07,540 --> 00:03:08,260 Michael: Yeah, nice. 59 00:03:08,300 --> 00:03:12,320 Should we go back then in terms of how do you like to describe 60 00:03:12,360 --> 00:03:15,180 the different models or the different options that people have 61 00:03:15,180 --> 00:03:17,240 at the in the early stages? 62 00:03:18,360 --> 00:03:23,420 Gwen: Yeah, so everyone basically starts with 1 out of 2. 63 00:03:23,420 --> 00:03:28,080 And I'm using AWS terminology, even though there is other terminology 64 00:03:28,200 --> 00:03:29,720 that people apply to it. 65 00:03:30,160 --> 00:03:36,680 AWS calls it the pooled model versus isolated model. 66 00:03:37,720 --> 00:03:43,980 And in a pooled model, you basically create your tables as normal, 67 00:03:44,240 --> 00:03:49,520 and then add a tenant ID Column to each and every 1 of your tables, 68 00:03:49,540 --> 00:03:50,060 pretty much. 69 00:03:50,060 --> 00:03:53,720 Some, maybe not everyone, some have shared data, but most of 70 00:03:53,720 --> 00:03:57,080 your tables are going to end up with a tenant ID Column that 71 00:03:57,080 --> 00:04:01,100 tells you which tenant this Row belongs to. 72 00:04:01,980 --> 00:04:07,040 Very easy when you start out and all you have to do is sprinkle 73 00:04:07,060 --> 00:04:10,920 some work clauses every now and then and you're pretty much good 74 00:04:10,920 --> 00:04:11,320 to go. 75 00:04:11,320 --> 00:04:13,120 How hard can it possibly be? 76 00:04:13,700 --> 00:04:18,220 The places where it gets you is that you have no solution. 77 00:04:18,420 --> 00:04:20,820 Everyone is in 1 big pool. 78 00:04:21,600 --> 00:04:27,660 And if you have a problem tenant, if someone grows really, really 79 00:04:27,660 --> 00:04:32,360 large and suddenly you need to Query start getting slow for them, 80 00:04:33,060 --> 00:04:36,920 If you need to do an upgrade and 1 Customer absolutely refuses 81 00:04:37,160 --> 00:04:41,100 to accept changes or needs their own time window. 82 00:04:41,680 --> 00:04:45,040 There is a lot of different ways you may discover that by putting 83 00:04:45,040 --> 00:04:48,900 all your Customers in 1 big pool, you save a lot of effort, you 84 00:04:48,900 --> 00:04:49,920 save a lot of money. 85 00:04:49,920 --> 00:04:52,700 This is by far the cheapest option you're going to have. 86 00:04:53,260 --> 00:04:55,300 It's shared resources in 1 Database. 87 00:04:56,640 --> 00:05:00,940 But you are not allowing yourself to do anything specific for 88 00:05:00,940 --> 00:05:03,280 any 1 Customer should they need it. 89 00:05:04,120 --> 00:05:07,120 The other approach is basically the reverse. 90 00:05:07,900 --> 00:05:11,740 It gives every Customer its own Database or sometimes its own 91 00:05:11,740 --> 00:05:12,240 schema. 92 00:05:12,380 --> 00:05:16,080 This still counts as isolated, even though It's not all that 93 00:05:16,080 --> 00:05:16,580 isolated. 94 00:05:16,800 --> 00:05:19,500 You share quite a lot in that scenario. 95 00:05:19,960 --> 00:05:21,360 But the schema is separate. 96 00:05:22,100 --> 00:05:26,660 And it's quite a bit easier to move them out if needed, if the 97 00:05:26,660 --> 00:05:27,680 schema is separated. 98 00:05:28,780 --> 00:05:32,680 And In this scenario, first of all, there is a nice benefit that 99 00:05:32,680 --> 00:05:35,660 you get help from a lot of popular frameworks. 100 00:05:36,160 --> 00:05:40,620 I think Ruby has an, I think it's called apartment plugin for 101 00:05:40,800 --> 00:05:43,040 having this kind of multi-tenancy model. 102 00:05:43,380 --> 00:05:44,440 Django has something. 103 00:05:44,440 --> 00:05:47,660 So a lot of very popular frameworks have something that helps 104 00:05:47,660 --> 00:05:48,740 you in that model. 105 00:05:49,400 --> 00:05:54,460 But if you accidentally grow to a large number of databases, 106 00:05:54,800 --> 00:05:56,260 it starts being very painful. 107 00:05:56,280 --> 00:06:00,280 Obviously, a database with a large number of objects is no longer 108 00:06:00,280 --> 00:06:01,880 as easy to work with. 109 00:06:02,320 --> 00:06:06,480 Suddenly, you start learning how much space and memory the catalog 110 00:06:06,480 --> 00:06:08,640 can really take when you have connections. 111 00:06:09,320 --> 00:06:12,860 Suddenly you start learning that pg_dump can take a very long 112 00:06:12,860 --> 00:06:13,360 time. 113 00:06:13,940 --> 00:06:19,040 If you actually have a database for each tenant, then doing any 114 00:06:19,040 --> 00:06:24,120 kind of maintenance on 100 databases is already not fun. 115 00:06:24,200 --> 00:06:28,000 If you end up going into a thousand databases, it's really not 116 00:06:28,000 --> 00:06:28,500 fun. 117 00:06:29,040 --> 00:06:33,300 And if you think about it, a lot of SaaS have customers in the 118 00:06:33,380 --> 00:06:35,960 hundreds of thousands, not just thousand. 119 00:06:36,580 --> 00:06:41,940 So it becomes very painful exactly when you grow. 120 00:06:43,260 --> 00:06:43,520 Nikolay: Yeah. 121 00:06:43,520 --> 00:06:47,280 For example, upgrade includes dump restore schema. 122 00:06:47,640 --> 00:06:52,620 And we've had cases with 250,000 tables, 2 million indexes. 123 00:06:52,960 --> 00:06:55,700 Well, indexes are there not dumped, but tables. 124 00:06:55,760 --> 00:06:56,260 Gwen: Exactly. 125 00:06:56,320 --> 00:06:58,300 This is worse than no dump, right? 126 00:06:58,580 --> 00:07:00,260 If only we could dump indexes. 127 00:07:00,660 --> 00:07:03,500 Nikolay: Yeah, and then you need to update statistics after upgrade 128 00:07:03,580 --> 00:07:04,900 for all of those tables. 129 00:07:05,280 --> 00:07:06,480 It's a nightmare, honestly. 130 00:07:07,200 --> 00:07:07,580 Gwen: Yeah. 131 00:07:07,580 --> 00:07:10,640 On the other hand, at least you get to do it to a customer at 132 00:07:10,640 --> 00:07:11,120 a time. 133 00:07:11,120 --> 00:07:14,020 Imagine that everyone is in 1 really big database and now you 134 00:07:14,020 --> 00:07:15,900 have to upgrade all of them together. 135 00:07:16,700 --> 00:07:17,200 Nikolay: Yeah. 136 00:07:18,380 --> 00:07:18,880 Yeah. 137 00:07:19,200 --> 00:07:20,920 It's painful, all this. 138 00:07:21,860 --> 00:07:22,300 Gwen: Yeah. 139 00:07:22,300 --> 00:07:27,180 And usually, and then there is the mixed model where I think 140 00:07:27,180 --> 00:07:32,660 pretty much everyone ends up with where you basically have, you 141 00:07:32,660 --> 00:07:37,740 start with the pool model, and as you grow, you shard it, but 142 00:07:37,960 --> 00:07:41,160 you're actually pretty smart, and you realize that not all customers 143 00:07:41,160 --> 00:07:45,640 are of the same size, and you can have some dedicated shards 144 00:07:45,660 --> 00:07:50,040 for your biggest or most sensitive or most demanding customers. 145 00:07:51,460 --> 00:07:56,680 And this is, I think, if you look 5 years into the life of a 146 00:07:56,680 --> 00:08:00,560 company, I would say that this is the dominant model. 147 00:08:00,920 --> 00:08:07,660 Some variation of we have shared databases with pool and then 148 00:08:07,660 --> 00:08:13,720 some dedicated databases with specific customers. 149 00:08:15,040 --> 00:08:16,560 Nikolay: What is the problem we're trying to solve? 150 00:08:16,560 --> 00:08:18,860 Is it security or performance or both? 151 00:08:19,280 --> 00:08:25,160 And if we go back to some customers, they share this pool, it 152 00:08:25,160 --> 00:08:26,840 affects security goal, right? 153 00:08:26,840 --> 00:08:28,680 So we don't achieve it. 154 00:08:28,860 --> 00:08:29,360 Gwen: Absolutely. 155 00:08:29,380 --> 00:08:32,420 So this is 1, First of all, this is 1 driver that people actually 156 00:08:32,420 --> 00:08:34,140 start with the isolated model. 157 00:08:34,280 --> 00:08:37,680 They know that they're going into a sensitive area. 158 00:08:37,740 --> 00:08:42,480 They're focusing on SaaS for healthcare, SaaS for finance. 159 00:08:43,660 --> 00:08:50,440 Those companies definitely start with isolated model and try 160 00:08:50,440 --> 00:08:53,500 to figure out how to manage large number of databases. 161 00:08:53,940 --> 00:08:57,320 A lot of times those companies don't become huge. 162 00:08:57,440 --> 00:09:04,400 There is that many hospitals in the United States, but they still 163 00:09:04,400 --> 00:09:08,800 have to build all the tooling to manage a large number of isolated 164 00:09:08,820 --> 00:09:09,320 databases. 165 00:09:10,840 --> 00:09:14,440 For other companies, it's more complicated. 166 00:09:15,060 --> 00:09:19,700 I would say maybe 70% of the time, the reason for eventually 167 00:09:19,920 --> 00:09:23,900 moving customers out and sharding and isolating would be performance. 168 00:09:25,080 --> 00:09:29,200 It's amazing how many performance problems can be solved by just 169 00:09:29,200 --> 00:09:31,260 having less data in each database. 170 00:09:34,300 --> 00:09:41,340 On the other side, there are the story where 2 years in, suddenly 171 00:09:41,520 --> 00:09:45,560 a very sensitive customer shows up or you want to sell into, 172 00:09:46,080 --> 00:09:52,000 like you thought you were building a normal CRM or some kind 173 00:09:52,080 --> 00:09:57,940 of a rug database, but then a healthcare company shows up, a 174 00:09:57,940 --> 00:10:02,840 bank shows up, or even worse, a government shows up, and they 175 00:10:02,840 --> 00:10:09,620 show up with a list of demands and since they usually have good 176 00:10:09,620 --> 00:10:13,300 amounts of money to back those demands there is a lot of incentive 177 00:10:13,660 --> 00:10:16,220 to figure out a solution for them. 178 00:10:17,020 --> 00:10:17,520 Michael: Nice. 179 00:10:17,860 --> 00:10:22,920 We have 1 kind of ugly duckling in the Postgres world that I'm 180 00:10:22,920 --> 00:10:24,960 not sure quite fits either of these models. 181 00:10:24,960 --> 00:10:28,400 I wonder if it's worth discussing row level security briefly, 182 00:10:28,580 --> 00:10:32,280 because if I was to bucket it based on those definitions, it's 183 00:10:32,320 --> 00:10:35,640 kind of the pooled model in a way because all of the data is 184 00:10:35,640 --> 00:10:38,340 together but there is some isolation 185 00:10:39,160 --> 00:10:40,420 Gwen: between tenants 186 00:10:40,940 --> 00:10:42,320 Michael: absolutely yeah 187 00:10:42,740 --> 00:10:46,120 Gwen: yeah I have a love-hate relationship with RLS I think a 188 00:10:46,120 --> 00:10:47,300 lot of people do. 189 00:10:47,300 --> 00:10:50,400 Because you're right, on 1 hand, it's absolutely a lifesaver 190 00:10:50,800 --> 00:10:51,920 in the pooled model. 191 00:10:52,120 --> 00:10:58,200 Developers make mistakes as joins get, and conditions get more 192 00:10:58,200 --> 00:10:58,700 complicated. 193 00:10:58,940 --> 00:11:04,700 It's very easy to misplace a work clause and actually leak data 194 00:11:05,140 --> 00:11:06,560 that you don't want to. 195 00:11:07,260 --> 00:11:11,600 So RLS will prevent you from doing it if you do it right. 196 00:11:12,440 --> 00:11:16,300 It turns out that a lot of times the rules could get complicated 197 00:11:17,040 --> 00:11:18,620 and then it leads to bugs. 198 00:11:19,060 --> 00:11:22,340 It also turns out that a lot of times the rules get complicated 199 00:11:22,360 --> 00:11:24,340 and it leads to terrible performance. 200 00:11:25,440 --> 00:11:29,760 And 1 thing that developers really don't realize, I'd say almost 201 00:11:29,760 --> 00:11:33,340 no developer realizes it until they run into it. 202 00:11:35,020 --> 00:11:40,680 The work conditions that RLS introduces are not optimized like 203 00:11:40,680 --> 00:11:42,420 the work conditions that you introduce. 204 00:11:43,080 --> 00:11:48,700 Because Postgres, thankfully, is very good about security, It 205 00:11:48,700 --> 00:11:51,600 treats the work conditions in RLS differently. 206 00:11:52,200 --> 00:11:56,180 I think they have, they call it security conditions or something 207 00:11:56,180 --> 00:11:56,880 like that. 208 00:11:57,660 --> 00:12:02,020 And they are very, very conservative on how they optimize it 209 00:12:02,020 --> 00:12:03,740 and how they plan for it. 210 00:12:04,180 --> 00:12:05,260 This has benefits. 211 00:12:05,540 --> 00:12:09,320 You get very strong security guarantees, very few bugs as a result. 212 00:12:09,320 --> 00:12:10,140 This is fantastic. 213 00:12:10,580 --> 00:12:16,160 On the other hand, the plan would be sometimes significantly 214 00:12:16,500 --> 00:12:20,020 worse than what you would come up with if you were to look at 215 00:12:20,020 --> 00:12:23,000 it really hard and do it yourself. 216 00:12:23,740 --> 00:12:27,700 And with RLS, there is basically no way to force the plan you 217 00:12:27,700 --> 00:12:28,200 want. 218 00:12:28,860 --> 00:12:36,800 You cannot set, enable, disable different rules because, again, 219 00:12:36,820 --> 00:12:41,120 the main overriding rule is that we're very conservative on how 220 00:12:41,120 --> 00:12:43,760 we optimize those RLS conditions. 221 00:12:44,380 --> 00:12:47,940 So Some people call RLS a performance killer. 222 00:12:48,560 --> 00:12:52,960 I wouldn't necessarily go this far, but you can definitely run 223 00:12:52,960 --> 00:12:57,100 into gotchas and you need to be aware that it's not a normal 224 00:12:57,440 --> 00:12:59,280 wear conditions that you're looking at. 225 00:13:01,160 --> 00:13:01,560 Michael: Yeah. 226 00:13:01,560 --> 00:13:02,060 Nice. 227 00:13:02,700 --> 00:13:08,920 Nikolay: So what does Nile offer today and what is the ideal 228 00:13:09,560 --> 00:13:13,220 solution to all this in Postgres context of course? 229 00:13:13,780 --> 00:13:19,400 Gwen: Yeah, so basically we wanted to do maybe 3 things. 230 00:13:19,640 --> 00:13:24,900 First of all, give isolation while not degrading the developer 231 00:13:24,940 --> 00:13:25,440 experience. 232 00:13:26,180 --> 00:13:31,940 So for example, we partitioned data by tenant out of the box 233 00:13:32,500 --> 00:13:38,440 for you, completely transparently, because we know that a bit 234 00:13:38,440 --> 00:13:41,280 later on, you're going to want it, and it's going to be a pain 235 00:13:41,280 --> 00:13:42,480 in the ass to edit. 236 00:13:44,060 --> 00:13:45,660 We shard it Transparently. 237 00:13:46,160 --> 00:13:50,440 Basically, your database may be spread across multiple different 238 00:13:50,440 --> 00:13:50,860 charts. 239 00:13:50,860 --> 00:13:55,740 We will route the queries for you and make sure that they are 240 00:13:56,140 --> 00:13:57,100 working as you should. 241 00:13:57,100 --> 00:14:03,580 So, in a way, you get the model you will have anyway in 4 to 242 00:14:03,580 --> 00:14:08,520 5 years, but you're getting it from the get-go and without doing 243 00:14:08,520 --> 00:14:11,880 a lot of the work because we are doing a lot of the management 244 00:14:11,980 --> 00:14:12,680 for you. 245 00:14:13,140 --> 00:14:18,780 The other thing is that we have done some work to basically bypass 246 00:14:19,020 --> 00:14:21,640 RLS and still give you isolation. 247 00:14:22,280 --> 00:14:29,340 So the queries, you kind of do the same RLS set tenant ID equal. 248 00:14:30,040 --> 00:14:32,540 We use that to actually direct queries. 249 00:14:33,400 --> 00:14:37,200 We rewrite the queries immediately to the partitions that we 250 00:14:37,200 --> 00:14:39,360 know has the data for that tenant. 251 00:14:39,860 --> 00:14:43,700 So we have a small extension that kind of replaces table names 252 00:14:43,700 --> 00:14:45,820 with partition names in the query itself. 253 00:14:47,380 --> 00:14:53,980 And this vastly improves performance in the majority of cases. 254 00:14:54,140 --> 00:14:57,980 I mean, we've seen it in a bunch of cases, especially if you 255 00:14:57,980 --> 00:15:02,940 have slightly weird indexes that RLS may conservatively not use. 256 00:15:03,580 --> 00:15:10,080 The improvement is quite stark, depending, obviously, on table 257 00:15:10,080 --> 00:15:10,580 sizes. 258 00:15:10,760 --> 00:15:13,820 You can get a benchmark that proves anything, so I don't want 259 00:15:13,840 --> 00:15:15,640 to throw numbers out there. 260 00:15:16,980 --> 00:15:22,620 But obviously, if you break down a table with a million rows 261 00:15:22,820 --> 00:15:27,500 into thousand tenants with a thousand rows each, then you can 262 00:15:27,500 --> 00:15:30,700 show quite a, you can see where I'm going with that. 263 00:15:31,720 --> 00:15:35,020 The other thing we did, and that was probably the most work, 264 00:15:35,020 --> 00:15:39,260 and this is still work in progress, is allow moving tenants around. 265 00:15:39,440 --> 00:15:43,040 Because 1 of the biggest problems is that the tenant gets large 266 00:15:43,320 --> 00:15:44,020 or noisy. 267 00:15:44,640 --> 00:15:49,940 And you want to give it its own machine, moving it is usually 268 00:15:50,080 --> 00:15:51,040 a long downtime. 269 00:15:51,300 --> 00:15:54,100 If you catch it after the tenant is already large. 270 00:15:54,480 --> 00:16:00,240 By doing the compute storage separation, we can basically make 271 00:16:00,240 --> 00:16:00,980 it transparent. 272 00:16:01,360 --> 00:16:06,500 It's a latency spike while we're holding off some queries, while 273 00:16:06,500 --> 00:16:11,980 we're moving things like setting up sequence ideas, moving, pointing 274 00:16:12,040 --> 00:16:15,640 into a different compute into the same part of the storage. 275 00:16:16,160 --> 00:16:20,240 But it's essentially a no downtime operation. 276 00:16:21,000 --> 00:16:26,500 So we think it's a huge deal because again, it's just a problem 277 00:16:26,500 --> 00:16:29,240 that we keep seeing again and again. 278 00:16:30,600 --> 00:16:33,980 Nikolay: Yeah, I'm curious, is it all open source what you build 279 00:16:34,120 --> 00:16:35,740 or only parts of it? 280 00:16:35,740 --> 00:16:37,620 Gwen: Right now it's mostly hidden. 281 00:16:38,680 --> 00:16:45,600 We have started registering parts of it under a Apache license 282 00:16:46,240 --> 00:16:46,740 veto. 283 00:16:47,300 --> 00:16:51,800 So yeah, the goal is to open source it and we already publicly 284 00:16:52,020 --> 00:16:54,360 declared that it's going to be open source. 285 00:16:54,520 --> 00:17:00,060 We have not a date, but a point of completion where we plan to 286 00:17:00,060 --> 00:17:00,840 open it. 287 00:17:01,100 --> 00:17:01,360 Nikolay: Yeah. 288 00:17:01,360 --> 00:17:05,020 So I've heard about this, several interesting things here. 289 00:17:05,020 --> 00:17:10,160 So 1 is extension for this, I guess it's called, in your documentation, 290 00:17:10,160 --> 00:17:14,660 it's called RLS virtualization or how's it called? 291 00:17:15,140 --> 00:17:16,980 Gwen: We call it tenant virtualization. 292 00:17:17,540 --> 00:17:21,480 The extension itself, I think we called it Karnak. 293 00:17:21,960 --> 00:17:26,480 We call everything after stuff in Egypt and Karnak is a famous 294 00:17:26,480 --> 00:17:26,980 temple. 295 00:17:28,320 --> 00:17:33,600 Nikolay: So data is stored in separate tables and in the same 296 00:17:33,600 --> 00:17:38,540 database, but extension rewrites queries to basically route query 297 00:17:38,760 --> 00:17:40,640 to proper table, right? 298 00:17:40,640 --> 00:17:42,300 Is this based on? 299 00:17:42,660 --> 00:17:48,240 Gwen: It's in separate partitions And we rewrite queries to go 300 00:17:48,240 --> 00:17:49,580 to the correct partition. 301 00:17:50,660 --> 00:17:55,800 We basically bypass RLS, we bypass the planner trying to make 302 00:17:55,800 --> 00:17:56,620 those calls. 303 00:17:56,880 --> 00:18:01,420 We found out that with a large number of partitions, this is 304 00:18:01,780 --> 00:18:03,180 significantly more efficient. 305 00:18:03,180 --> 00:18:03,740 Nikolay: I see. 306 00:18:03,740 --> 00:18:05,280 So it's Postgres partitioning. 307 00:18:05,280 --> 00:18:05,960 I see. 308 00:18:06,060 --> 00:18:10,660 And you mentioned also another thing you mentioned here is sharding, 309 00:18:11,200 --> 00:18:11,700 right? 310 00:18:12,180 --> 00:18:12,680 Gwen: Yeah. 311 00:18:12,720 --> 00:18:16,140 So we use foreign data wrappers to allow... 312 00:18:17,380 --> 00:18:20,140 So we have 2 things in the architecture. 313 00:18:20,460 --> 00:18:24,180 First of all, we have a proxy, it's a routing proxy. 314 00:18:24,240 --> 00:18:29,500 So it keeps track of every connection, which tenant is the current 315 00:18:29,500 --> 00:18:35,860 tenant and it routes it to the shard that has the correct tenant 316 00:18:35,860 --> 00:18:36,600 in it. 317 00:18:37,440 --> 00:18:42,480 And then we also have some cases where some developers want to 318 00:18:42,480 --> 00:18:44,680 write queries that touch multiple tenants. 319 00:18:45,040 --> 00:18:49,240 Those are not going to be as fast, but we do allow them by use 320 00:18:49,240 --> 00:18:51,160 of foreign data wrappers. 321 00:18:52,360 --> 00:18:58,660 And mixing partitions with our own partitioning rules with foreign 322 00:18:58,660 --> 00:19:03,400 data wrappers and still keeping things efficient, we didn't want 323 00:19:03,540 --> 00:19:08,080 the planner on any machine to be aware of all the partitions 324 00:19:08,600 --> 00:19:14,380 in all the other shards, because it just explodes the planning 325 00:19:14,380 --> 00:19:16,420 time in ways that we saw as unacceptable. 326 00:19:17,200 --> 00:19:25,040 So what we did is represent each shard with a table and then 327 00:19:25,040 --> 00:19:29,660 put hierarchical table inheritance on top of it. 328 00:19:30,660 --> 00:19:36,280 And the end result is basically a union all between the table 329 00:19:36,280 --> 00:19:39,560 with the table inheritance that points to all those other shards, 330 00:19:41,040 --> 00:19:42,600 and the table with the partitions. 331 00:19:43,520 --> 00:19:47,920 Now this gives us basically predicate pushdown because the planner 332 00:19:47,920 --> 00:19:52,700 will push the plan to all those different charts. 333 00:19:53,200 --> 00:19:56,540 The other charts know that they have partitions, which the source 334 00:19:56,540 --> 00:19:57,760 planner didn't know. 335 00:19:58,120 --> 00:20:02,360 They will plan correctly with all the partitions, but they only 336 00:20:02,360 --> 00:20:03,980 know about a subset of partitions. 337 00:20:05,020 --> 00:20:09,700 So we see it's a bit hacky and it's a bit weird to explain. 338 00:20:10,360 --> 00:20:14,040 And we think we can do better with some modifications to Postgres, 339 00:20:14,060 --> 00:20:19,440 which we have not done yet, but this does give us predicate pushdown, 340 00:20:20,140 --> 00:20:24,240 fairly fast planning, and the ability to do queries that cross 341 00:20:24,240 --> 00:20:28,320 tenants in situations where this is required, essentially. 342 00:20:28,320 --> 00:20:32,660 Nikolay: Yeah, if you don't involve 2PC, just rely on front-end 343 00:20:32,660 --> 00:20:35,120 wrappers, no two-phase commit. 344 00:20:35,600 --> 00:20:39,440 I'm curious what kind of anomalies can happen there. 345 00:20:40,080 --> 00:20:44,680 Gwen: Yes, and we prevent a lot of things that could cause anomalies. 346 00:20:45,080 --> 00:20:50,140 So we do have a transaction coordinator, but in order to not 347 00:20:50,140 --> 00:20:54,280 overload and also not overcomplicate our architecture, we limit 348 00:20:54,280 --> 00:20:54,860 some things. 349 00:20:54,860 --> 00:21:00,360 So, DDL has to be done on a single tenant, and you cannot mix 350 00:21:00,700 --> 00:21:02,000 cross-tenant queries. 351 00:21:02,120 --> 00:21:06,280 Sorry, DML, INSERTs and UPDATEs have to be done on a single tenant, 352 00:21:06,280 --> 00:21:10,840 and you cannot mix in a Transaction cross, so the moment you 353 00:21:10,840 --> 00:21:13,500 start a Transaction, you have to know what tenant you're working 354 00:21:13,500 --> 00:21:14,000 on. 355 00:21:14,340 --> 00:21:17,120 And then we route it to the correct shard, which has the correct 356 00:21:17,120 --> 00:21:21,000 table, and everything has the absolutely correct guarantees. 357 00:21:21,600 --> 00:21:26,100 If you need to do something cross-tenant, you do not involve 358 00:21:26,180 --> 00:21:28,100 it with any kind of UPDATE. 359 00:21:28,460 --> 00:21:32,320 You could still be exposed to some anomalies, I agree, because 360 00:21:32,320 --> 00:21:36,780 it could be ongoing Transactions from other people in other places. 361 00:21:36,820 --> 00:21:42,600 So you get the basic Read Committed guarantees that Postgres 362 00:21:42,880 --> 00:21:46,520 gives you, I believe, but not anything more than that. 363 00:21:47,280 --> 00:21:53,300 But again, we believe that cross tenant Queries are rare and 364 00:21:53,300 --> 00:21:57,620 mostly done in analytical cases, where you do reporting where 365 00:21:57,620 --> 00:22:00,640 it's slightly less critical to have those. 366 00:22:01,020 --> 00:22:05,440 Nikolay: So you forbid the writing to 2 shards in 1 Transaction? 367 00:22:05,900 --> 00:22:08,000 Gwen: If you want to write to a shard, it's fantastic. 368 00:22:08,300 --> 00:22:12,180 You tell us what tenant you're writing data into, and we will 369 00:22:12,180 --> 00:22:13,580 direct you to the correct shard. 370 00:22:13,580 --> 00:22:16,760 Nikolay: I mean, if there is a Transaction which needs to write 371 00:22:16,760 --> 00:22:23,000 to 2 different shards, this is a big problem, because without 372 00:22:23,000 --> 00:22:23,500 PC. 373 00:22:24,720 --> 00:22:25,660 Gwen: Yes, exactly. 374 00:22:25,840 --> 00:22:28,660 And we don't let you do that, essentially, in order to avoid 375 00:22:28,660 --> 00:22:29,040 anomalies. 376 00:22:29,040 --> 00:22:29,640 Nikolay: I see. 377 00:22:29,760 --> 00:22:33,760 Another question here, have you considered the approach used 378 00:22:33,760 --> 00:22:34,420 in Vitesse? 379 00:22:34,640 --> 00:22:36,720 As I understand it, maybe I'm wrong. 380 00:22:37,040 --> 00:22:41,320 Where mostly for analytical Queries maybe, to avoid distributed 381 00:22:41,320 --> 00:22:45,660 Transactions, data is brought synchronously from 1 shard to another. 382 00:22:45,660 --> 00:22:48,920 And we have it locally, like basically kind of Materialized U 383 00:22:49,020 --> 00:22:51,880 on top of logical replication, for example, or something. 384 00:22:52,200 --> 00:22:55,640 And it has eventual consistency approach, of course, but you 385 00:22:55,640 --> 00:23:00,120 can just join it in 1 Postgres, in 1 shard, right? 386 00:23:00,140 --> 00:23:01,660 Have you considered this approach? 387 00:23:01,840 --> 00:23:02,300 Gwen: Y.K. 388 00:23:02,300 --> 00:23:03,780 We have considered it. 389 00:23:04,700 --> 00:23:10,680 I think maybe CitusDB has something similar, if I remember correctly. 390 00:23:10,680 --> 00:23:12,280 I'm not 100% sure. 391 00:23:12,740 --> 00:23:16,080 But yeah, it's something that we were like, yeah, this is a good 392 00:23:16,080 --> 00:23:18,460 idea that we may examine in the future. 393 00:23:19,280 --> 00:23:23,820 It's definitely, we are trying to build something useful gradually, 394 00:23:24,280 --> 00:23:28,440 and we understand that early on, it's almost safer to have a 395 00:23:28,440 --> 00:23:32,180 bunch of limitations that over time will resolve, rather than 396 00:23:32,440 --> 00:23:37,080 allow people to do something unsafe and also build a kitchen 397 00:23:37,080 --> 00:23:37,580 sink. 398 00:23:37,660 --> 00:23:41,140 Nikolay: It sounds to me like Postgres versus MySQL approaches 399 00:23:41,820 --> 00:23:45,200 because MySQL approach, you remember MyISAM? 400 00:23:45,560 --> 00:23:50,540 Maybe you don't remember, but it was like quite bad. 401 00:23:50,540 --> 00:23:55,080 You need to run a repair table all the time because it's not 402 00:23:55,080 --> 00:23:56,760 ACID and so on. 403 00:23:56,980 --> 00:23:58,220 It allowed too much. 404 00:23:58,980 --> 00:24:00,420 Gwen: Exactly, yeah. 405 00:24:00,540 --> 00:24:02,340 And Marathon had a lot of issues. 406 00:24:02,360 --> 00:24:06,320 I mean, there is a reason why InnoDB became extremely popular. 407 00:24:06,820 --> 00:24:07,800 Nikolay: Right, right. 408 00:24:07,900 --> 00:24:11,120 Michael: But also, with a multi-tenancy use case, I think you're 409 00:24:11,120 --> 00:24:14,440 quite right that, well, we, I mean, you'll find out soon enough, 410 00:24:14,440 --> 00:24:18,340 right, if lots of people want these cross-tenant or cross-shard 411 00:24:18,420 --> 00:24:22,800 queries which are by definition cross-tenant queries and if they 412 00:24:22,800 --> 00:24:25,600 don't, if you don't need to worry about it, you save a bunch 413 00:24:25,600 --> 00:24:27,900 of effort having to even implement that. 414 00:24:27,900 --> 00:24:29,680 So yeah, I like that a lot. 415 00:24:30,300 --> 00:24:34,580 Nikolay: Yeah, last comment here is I'm excited to see that finally 416 00:24:34,600 --> 00:24:38,620 Postgres ecosystem receives some tension in the area of sharding. 417 00:24:38,620 --> 00:24:43,180 I guess it's just time has come and more and more databases became 418 00:24:43,780 --> 00:24:45,420 too large to be handled. 419 00:24:45,420 --> 00:24:46,520 Gwen: I'm almost surprised. 420 00:24:46,960 --> 00:24:49,040 I'm honestly surprised it took that long. 421 00:24:49,160 --> 00:24:51,840 I mean, again, if you look at MySQL. 422 00:24:51,900 --> 00:24:53,540 Nikolay: We just, it's just unfortunate. 423 00:24:53,860 --> 00:24:57,260 First time I touched this topic, it was 2006 immediately when 424 00:24:57,260 --> 00:24:59,080 we started working with Postgres, honestly. 425 00:24:59,440 --> 00:25:04,280 And There was a PL/Proxy from Skype at that time already, but 426 00:25:04,280 --> 00:25:08,300 it required you to write everything in functions. 427 00:25:08,400 --> 00:25:10,700 Gwen: Partitioning didn't exist back then. 428 00:25:10,960 --> 00:25:11,460 Nikolay: Existed. 429 00:25:11,580 --> 00:25:13,980 It was based on inheritance. 430 00:25:14,480 --> 00:25:14,640 It 431 00:25:14,640 --> 00:25:16,960 required much more manual. 432 00:25:17,700 --> 00:25:18,880 It was fun, actually. 433 00:25:19,660 --> 00:25:21,440 You understood it better, you know? 434 00:25:22,240 --> 00:25:25,020 But yeah, but it was not convenient, 100%. 435 00:25:25,640 --> 00:25:26,740 Not super convenient. 436 00:25:26,980 --> 00:25:31,260 It's just, I see that it's just unfortunate how it turned out 437 00:25:31,260 --> 00:25:32,320 in Postgres ecosystem. 438 00:25:32,320 --> 00:25:34,420 And now definitely there is huge pressure. 439 00:25:34,460 --> 00:25:36,880 Many companies need partitioning... 440 00:25:37,400 --> 00:25:37,740 Need sharding. 441 00:25:37,740 --> 00:25:38,260 And it seems 442 00:25:38,260 --> 00:25:39,960 Gwen: like How would history work? 443 00:25:39,960 --> 00:25:44,760 Imagine that YouTube picked Postgres and not MySQL as their first 444 00:25:44,760 --> 00:25:45,260 database. 445 00:25:46,580 --> 00:25:50,140 Nikolay: Yeah Google or Facebook they both chose MySQL somehow 446 00:25:50,140 --> 00:25:50,640 Yeah. 447 00:25:52,040 --> 00:25:54,140 Michael: The test would be for Postgres first, right? 448 00:25:54,140 --> 00:25:54,640 Gwen: Exactly. 449 00:25:55,680 --> 00:25:57,580 It could have turned out so differently. 450 00:25:57,720 --> 00:25:58,220 Michael: Yeah. 451 00:25:59,320 --> 00:26:00,420 Funny, isn't it? 452 00:26:01,020 --> 00:26:04,740 Changing topics slightly or LLMs are quite top of mind at the 453 00:26:04,740 --> 00:26:09,880 moment, are you seeing people's initial choices change as a result 454 00:26:09,880 --> 00:26:14,120 of asking for advice earlier from our robot friends? 455 00:26:15,540 --> 00:26:21,260 Gwen: Oh my god, we're seeing so many weird things, it's just 456 00:26:21,820 --> 00:26:24,120 unbelievable how much things are changing. 457 00:26:24,600 --> 00:26:28,900 First of all, we're having people show up on our Discord and 458 00:26:28,900 --> 00:26:34,120 say things like, I'm using Nile because my LLM thought it's a 459 00:26:34,120 --> 00:26:34,820 good idea. 460 00:26:36,660 --> 00:26:40,760 And I don't really know Postgres, so I need some help, but my 461 00:26:40,760 --> 00:26:43,360 LLM assured me that this is still a good idea. 462 00:26:44,700 --> 00:26:48,580 Like a lot of people are just like people who are very much beginners, 463 00:26:49,440 --> 00:26:53,160 like maybe I can say when I started developing, the first time 464 00:26:53,160 --> 00:26:57,940 I had to use a database, my company sent me to a 3 weeks database 465 00:26:58,240 --> 00:26:58,740 class. 466 00:26:58,780 --> 00:27:01,720 I think it was Oracle about 20 years back. 467 00:27:01,980 --> 00:27:06,140 And I came back a lot more confident that I know how to use Oracle 468 00:27:06,600 --> 00:27:09,660 and not to leave transactions open for too long because people 469 00:27:09,660 --> 00:27:10,780 will yell at me. 470 00:27:11,520 --> 00:27:16,520 These days people don't do the three-week class before they try 471 00:27:16,520 --> 00:27:17,460 using a database. 472 00:27:17,880 --> 00:27:21,740 So you see a lot more, people are using a database earlier on 473 00:27:21,740 --> 00:27:26,340 and they do expect more hand-holding from the vendors. 474 00:27:26,580 --> 00:27:32,620 Like the LLMs give them advice up to a certain point, but eventually, 475 00:27:33,080 --> 00:27:35,980 if things are slow, they will come to you and say, hey, why is 476 00:27:35,980 --> 00:27:36,920 my query slow? 477 00:27:37,960 --> 00:27:40,520 I'm sure you guys have seen your share of that. 478 00:27:41,840 --> 00:27:49,340 I'm also seeing people use Postgres for their LLMs in different 479 00:27:49,340 --> 00:27:49,840 ways. 480 00:27:49,920 --> 00:27:52,420 And this is really exciting to me. 481 00:27:52,420 --> 00:27:57,940 People use Postgres via MCPs, people use Postgres with vectors, 482 00:27:58,260 --> 00:28:01,960 people building AI applications on top of Postgres. 483 00:28:02,440 --> 00:28:04,040 We're seeing a lot of that. 484 00:28:06,100 --> 00:28:10,220 And, I mean, personally, I'm really excited that people can program 485 00:28:10,460 --> 00:28:15,200 with LLMs, not knowing a lot about Postgres, not knowing even 486 00:28:15,200 --> 00:28:21,780 a lot about software engineering at all, and still get reasonable 487 00:28:22,800 --> 00:28:23,860 security guarantees. 488 00:28:24,480 --> 00:28:31,640 You don't need to know to ask your LLM about RLS or about, Are 489 00:28:31,640 --> 00:28:35,640 you sure this query actually properly isolates tenants? 490 00:28:36,900 --> 00:28:41,920 And it's also interesting how much the results differ when people 491 00:28:41,920 --> 00:28:44,880 use different LLMs. 492 00:28:46,260 --> 00:28:51,820 Like I would say syncing models do fairly well iterating in order 493 00:28:51,820 --> 00:28:55,320 to get good code and checking their own results, again, given 494 00:28:55,320 --> 00:28:57,460 via MCP access to a database. 495 00:28:58,340 --> 00:29:04,220 I would say that if you use HRGPT 4.0, you will get a lot of 496 00:29:04,220 --> 00:29:08,100 random hallucinatory stuff still in your code. 497 00:29:08,680 --> 00:29:11,000 Nikolay: Yeah, it's funny. 498 00:29:11,000 --> 00:29:15,660 I already told Michael, we had the cases doing consulting, But 499 00:29:15,660 --> 00:29:17,680 like maybe already almost a year ago. 500 00:29:17,680 --> 00:29:23,620 I started noticing that people send us Like some we are building 501 00:29:23,620 --> 00:29:24,640 this part of database. 502 00:29:24,640 --> 00:29:25,800 Can you review it? 503 00:29:25,800 --> 00:29:30,740 We are reviewing we use different lamps supporting this review 504 00:29:31,040 --> 00:29:36,960 and then we have a call and I'm curious, code looks great, I 505 00:29:36,960 --> 00:29:42,440 mean schema looks great but something is off, right? 506 00:29:42,440 --> 00:29:46,420 And then we have a call and I see they open tabs and charge a 507 00:29:46,420 --> 00:29:51,280 PT, Claude there as well so I realize they used LLM to create 508 00:29:51,280 --> 00:29:54,380 schema and then send us for review and we use a lens to review 509 00:29:54,380 --> 00:29:59,880 it and then There's like a 4 party process, you know It's it 510 00:29:59,880 --> 00:30:04,300 leads to good place, but someone needs to jump with proper expertise 511 00:30:04,440 --> 00:30:06,880 and say, this is not a good approach. 512 00:30:07,660 --> 00:30:08,560 Gwen: This is hilarious. 513 00:30:08,560 --> 00:30:12,340 Do you think at some point you and your customer can just step 514 00:30:12,340 --> 00:30:15,100 out and let the LLMs figure it out between themselves? 515 00:30:15,100 --> 00:30:19,000 Nikolay: Well, there's a problem here because like it's great, 516 00:30:19,000 --> 00:30:23,920 but there is like it does 80% of drop in 1% not not even 20% 517 00:30:24,280 --> 00:30:28,140 of effort very quickly, but there is 20% of problems, which again, 518 00:30:28,140 --> 00:30:31,780 like, I don't know, maybe in next few years, it will change. 519 00:30:31,780 --> 00:30:32,860 I think it will change. 520 00:30:32,860 --> 00:30:38,000 But right now I feel my internal LLM trained much better than 521 00:30:38,000 --> 00:30:38,960 than Charger PT. 522 00:30:39,440 --> 00:30:41,180 Gwen: You trained your own LLM, right? 523 00:30:41,180 --> 00:30:43,040 Nikolay: Yeah, no, I mean my own. 524 00:30:43,620 --> 00:30:46,360 Gwen: Oh yeah, no, but I think you actually trained your own. 525 00:30:46,360 --> 00:30:47,460 Nikolay: Yeah, we experiment. 526 00:30:48,920 --> 00:30:50,440 We do some stuff, we experiment. 527 00:30:50,600 --> 00:30:55,400 We have some things like we have, we start with fine tuning, 528 00:30:55,680 --> 00:31:00,200 moving to our own LLM, but not yet there still. 529 00:31:01,260 --> 00:31:05,400 So yeah, there is a lot of stuff can be done there to properly. 530 00:31:06,040 --> 00:31:10,680 It's hard to compete with Cloud and they have very high pace. 531 00:31:11,120 --> 00:31:14,320 With Cloud 4 release, I see, wow, it's really great. 532 00:31:14,320 --> 00:31:17,720 But it's still missing many things you learn from practice, which 533 00:31:17,720 --> 00:31:18,620 were not discussed. 534 00:31:18,780 --> 00:31:20,580 That's why they don't bring it. 535 00:31:21,580 --> 00:31:24,220 Many problems were not discussed yet. 536 00:31:24,520 --> 00:31:29,080 And you explore them if you have a lot of data and heavy workloads. 537 00:31:30,040 --> 00:31:35,680 Gwen: So you're saying that even training on the Postgres mailing 538 00:31:35,680 --> 00:31:39,600 list doesn't have all the information in it, essentially? 539 00:31:39,880 --> 00:31:40,780 Nikolay: Well, yes. 540 00:31:41,280 --> 00:31:44,820 For example, random problem, we recently touched it. 541 00:31:45,060 --> 00:31:47,220 There is a buffer pool in Postgres. 542 00:31:47,780 --> 00:31:52,040 And there are 128 basically partitions. 543 00:31:52,720 --> 00:31:55,220 So you can have 128 locks. 544 00:31:55,640 --> 00:31:59,280 And if you have a huge buffer pool, this becomes bottleneck. 545 00:31:59,440 --> 00:32:03,580 And some people say, let's maybe make it tunable, configurable. 546 00:32:05,500 --> 00:32:09,380 But when you start researching this topic, you end up finding 547 00:32:09,380 --> 00:32:12,940 some recent, well, recent last August and September last year, 548 00:32:13,280 --> 00:32:17,120 conversation in hackers mailing list, which is open-ended. 549 00:32:17,380 --> 00:32:20,500 It's not complete, because somebody needs to run benchmarks and 550 00:32:20,500 --> 00:32:26,060 prove that this is worth having new setting. 551 00:32:26,880 --> 00:32:27,540 And that's it. 552 00:32:27,540 --> 00:32:29,440 There's a patch proposed, but that's it. 553 00:32:29,440 --> 00:32:31,360 No experiments yet. 554 00:32:31,680 --> 00:32:34,460 Gwen: This is, by the way, 1 of the reasons I was so excited 555 00:32:34,480 --> 00:32:38,600 about your LLM approach, because if you think about what is the 556 00:32:38,600 --> 00:32:43,480 bottleneck for doing a lot of the database improvement things, 557 00:32:43,780 --> 00:32:48,360 and I am feeling it Very personally, running benchmarks is hard. 558 00:32:49,020 --> 00:32:51,000 Properly planning a benchmark is hard. 559 00:32:51,000 --> 00:32:56,000 LLMs can help with that, but again, they sometimes just go off 560 00:32:56,000 --> 00:32:56,680 the rails. 561 00:32:57,700 --> 00:33:02,920 And even if they help plan that, I don't know LLMs that actually 562 00:33:03,460 --> 00:33:07,400 run benchmarks to the point where they provision the machines 563 00:33:07,440 --> 00:33:11,980 in AWS and know that you have to provision a separate machine 564 00:33:11,980 --> 00:33:15,520 as a database and a separate machine, maybe a few of them, to 565 00:33:15,520 --> 00:33:19,340 drive the workload and they both need to have appropriate resources. 566 00:33:19,940 --> 00:33:23,700 Like all those and then don't get me started on analyzing the 567 00:33:23,700 --> 00:33:27,480 results, which is kind of 99.9% of the work. 568 00:33:28,660 --> 00:33:32,220 So the fact that you actually kind of started your LLM for I 569 00:33:32,220 --> 00:33:36,820 have an LLM that can actually do benchmarks, is just, I think 570 00:33:36,820 --> 00:33:41,980 this will be the biggest breakthrough in both people tuning their 571 00:33:41,980 --> 00:33:47,500 own Postgres, and also Postgres as a community being able to 572 00:33:47,500 --> 00:33:49,500 advance the state of the art? 573 00:33:49,540 --> 00:33:51,000 Nikolay: Let me share sad news. 574 00:33:51,580 --> 00:33:55,240 Like, I'm not giving up, but it's a roller coaster. 575 00:33:55,240 --> 00:33:58,880 I spend more than we spend, like the team of maybe 5 engineers 576 00:33:58,900 --> 00:34:02,840 spent more than 1 year trying to achieve that. 577 00:34:03,740 --> 00:34:08,240 We achieved many things, but first of all, we chose Gemini because 578 00:34:08,240 --> 00:34:09,300 they gave us credits. 579 00:34:09,640 --> 00:34:11,140 And I think it was a huge mistake. 580 00:34:11,140 --> 00:34:12,740 Gemini has a lot of problems. 581 00:34:14,360 --> 00:34:20,520 Suddenly you have 500 errors, which are so many problems. 582 00:34:21,100 --> 00:34:26,780 It's just not mature product, Gemini, and it has hallucinations 583 00:34:27,040 --> 00:34:27,880 all the time. 584 00:34:27,880 --> 00:34:31,160 It's good, for example, for JSON, working with JSON, because 585 00:34:31,160 --> 00:34:34,280 when you need to run experiment, we decided to choose JSON as 586 00:34:34,280 --> 00:34:35,140 config format. 587 00:34:35,460 --> 00:34:39,860 It writes much better than GPT-4, 4.0 and so on. 588 00:34:39,960 --> 00:34:43,040 But many things, it just hallucinates. 589 00:34:43,260 --> 00:34:44,860 It invents all the time some things. 590 00:34:44,860 --> 00:34:47,120 It just makes up results all the time. 591 00:34:47,400 --> 00:34:51,960 And we have a system to control it but it bypasses all the time. 592 00:34:52,060 --> 00:34:53,460 Like it's silly hard. 593 00:34:53,500 --> 00:34:59,180 So then yeah, we experimented with additional DeepSeek, Lama 594 00:34:59,480 --> 00:35:02,480 and we've fine-tuned a lot. 595 00:35:02,680 --> 00:35:07,480 All versions of GPT, all modern fresh versions, we also bring 596 00:35:07,480 --> 00:35:08,600 them all the time. 597 00:35:08,720 --> 00:35:12,680 And Claude, Claude is much better, we just added it to this system 598 00:35:12,780 --> 00:35:13,600 we have. 599 00:35:14,060 --> 00:35:19,800 But After 1 year, I decided, you know what, I'm like, benchmarks 600 00:35:19,900 --> 00:35:21,320 is extremely hard topic. 601 00:35:21,860 --> 00:35:23,560 And we cannot trust it anymore. 602 00:35:23,560 --> 00:35:30,940 I mean, we cannot trust LLM to create precise configuration and 603 00:35:30,940 --> 00:35:32,720 process results fully. 604 00:35:32,980 --> 00:35:38,540 So we decided that LLM is just more like connection thing. 605 00:35:38,680 --> 00:35:41,540 When you engineer benchmark, expert needs to engineer. 606 00:35:41,540 --> 00:35:47,900 I don't trust any LLM for now, because any experiment, we plan 607 00:35:47,900 --> 00:35:52,360 to publish maybe 15 to 20 experiments in our blog last year. 608 00:35:52,360 --> 00:35:54,880 And if you open our blog, you see just 1 experiment. 609 00:35:55,320 --> 00:35:59,640 And even there we screwed up and someone on Twitter said this 610 00:35:59,640 --> 00:36:02,220 is not right and we quickly corrected, which is good. 611 00:36:02,320 --> 00:36:06,640 And we had achievements like interesting things, bottlenecks 612 00:36:07,420 --> 00:36:08,560 popped up here and there. 613 00:36:08,560 --> 00:36:10,940 It's really fun to iterate with LLM. 614 00:36:11,060 --> 00:36:14,440 But once you allow to think, to design experiment and to treat 615 00:36:14,440 --> 00:36:20,880 results, in 99% of time we have wrong, wrong results, wrong conclusions 616 00:36:21,040 --> 00:36:21,760 and so on. 617 00:36:21,760 --> 00:36:26,400 So for now we are thinking, okay, this is just, accelerator of 618 00:36:26,400 --> 00:36:30,880 performing experiments, but design and understanding results 619 00:36:30,960 --> 00:36:33,140 should be in human brain for now. 620 00:36:33,540 --> 00:36:34,040 100%. 621 00:36:34,120 --> 00:36:38,080 Gwen: I love the fact that Benchmarks is also hard for robots, 622 00:36:38,240 --> 00:36:39,040 to be honest. 623 00:36:39,060 --> 00:36:40,940 It's so hard for humans, right? 624 00:36:41,600 --> 00:36:45,960 Nikolay: Yeah, and we collect so many artifacts, But somehow 625 00:36:45,960 --> 00:36:47,260 it's super hard still. 626 00:36:48,100 --> 00:36:50,240 So you always think where is the bottleneck? 627 00:36:51,180 --> 00:36:53,040 And simple question, right? 628 00:36:53,560 --> 00:37:00,480 But for now, it's extremely hard to let LLM find bottleneck and 629 00:37:00,480 --> 00:37:01,620 draw proper conclusions. 630 00:37:02,780 --> 00:37:05,740 Gwen: And honestly, if you just had an LM that always said it's 631 00:37:05,740 --> 00:37:09,180 a network, it would be correct about 80% of the time. 632 00:37:10,400 --> 00:37:11,600 Nikolay: What if it's local? 633 00:37:12,340 --> 00:37:15,640 If it's, there's no network, everything through Unix sockets 634 00:37:15,660 --> 00:37:17,740 and we had these cases as well. 635 00:37:18,390 --> 00:37:20,860 So I agree with you in production. 636 00:37:20,900 --> 00:37:24,960 Like in production, yes, but in experiments when we learn Postgres 637 00:37:24,960 --> 00:37:28,660 behavior on single machine running pgbench locally, we don't 638 00:37:28,660 --> 00:37:29,560 care sometimes. 639 00:37:30,060 --> 00:37:32,620 So there's no network there sometimes. 640 00:37:33,340 --> 00:37:35,140 So it's hard. 641 00:37:35,140 --> 00:37:37,800 So I had many moments of frustration. 642 00:37:38,860 --> 00:37:42,880 But it's so good, like I still believe that iterations are great. 643 00:37:42,880 --> 00:37:46,660 So if you say this is great benchmark, just check it on new version. 644 00:37:46,960 --> 00:37:48,380 Just changing 1 thing. 645 00:37:48,900 --> 00:37:49,820 This is good. 646 00:37:50,280 --> 00:37:54,960 We have automation, we have interface, and it repeats the process 647 00:37:54,960 --> 00:37:56,260 of analysis again. 648 00:37:56,580 --> 00:37:58,580 This is where LLM helps a lot. 649 00:38:01,420 --> 00:38:05,220 Because without LLM, you could just, you have some form, and 650 00:38:05,220 --> 00:38:07,320 oh, we don't have this parameter programmed. 651 00:38:07,360 --> 00:38:08,980 It's not exposed in the interface. 652 00:38:09,020 --> 00:38:09,840 It's bad. 653 00:38:09,880 --> 00:38:13,740 With LLM, you have freedom to change things and iterate based 654 00:38:13,740 --> 00:38:15,260 on existing good benchmarks. 655 00:38:16,160 --> 00:38:18,940 So, yeah, I cannot say we are there yet. 656 00:38:18,940 --> 00:38:22,780 This project, like, right now is we are, like, thinking about 657 00:38:22,780 --> 00:38:29,340 next level of it, where I think we will let LLM, we will give 658 00:38:29,340 --> 00:38:35,120 it less freedom, you know, that's the key, and control more by 659 00:38:35,740 --> 00:38:36,500 human brain. 660 00:38:37,700 --> 00:38:39,520 Gwen: Human in the loop kind of thing. 661 00:38:39,520 --> 00:38:40,640 Nikolay: Yeah, yeah, exactly. 662 00:38:40,640 --> 00:38:41,140 Exactly. 663 00:38:41,220 --> 00:38:46,720 So design and first analysis, only human should be there. 664 00:38:46,720 --> 00:38:49,960 But once you have confidence that you're moving in the right 665 00:38:49,960 --> 00:38:53,680 direction and you just need to iterate and expand, for example, 666 00:38:53,680 --> 00:38:56,820 to different versions, platforms, everything, this is where you 667 00:38:56,820 --> 00:38:57,540 can relax. 668 00:38:57,600 --> 00:38:59,180 You already verified results. 669 00:38:59,480 --> 00:39:02,060 You can say just repeat, but on different something. 670 00:39:02,520 --> 00:39:04,760 This is where LLM already can bring you. 671 00:39:04,760 --> 00:39:06,360 It can just speed up everything. 672 00:39:06,660 --> 00:39:10,320 You can throw it to this benchmarking process and have like 10 673 00:39:10,320 --> 00:39:13,220 experiments running in 10 different versions or something. 674 00:39:13,280 --> 00:39:16,460 Gwen: And I think this is also kind of almost the general directions 675 00:39:16,640 --> 00:39:19,020 that agents are taking shape. 676 00:39:19,280 --> 00:39:23,300 I mean, people started to 2025 was supposed to be the year of 677 00:39:23,300 --> 00:39:23,980 the agent. 678 00:39:24,720 --> 00:39:28,620 I think it's almost becoming the year of the human in the loop 679 00:39:28,620 --> 00:39:29,660 with the agent. 680 00:39:29,720 --> 00:39:30,220 Yeah. 681 00:39:30,780 --> 00:39:35,080 Like all the successful products I see are you tell the agent 682 00:39:35,080 --> 00:39:36,140 to do some stuff. 683 00:39:36,820 --> 00:39:38,620 You ask it, please plan something. 684 00:39:38,620 --> 00:39:39,720 You give it feedback. 685 00:39:39,960 --> 00:39:43,260 You then say, okay, now that we have a good plan, go and execute 686 00:39:43,260 --> 00:39:43,700 on it. 687 00:39:43,700 --> 00:39:46,660 You come back an hour later, you had your coffee. 688 00:39:47,020 --> 00:39:48,500 Okay, let's see what you've got. 689 00:39:48,500 --> 00:39:49,460 Here's some feedback. 690 00:39:49,640 --> 00:39:51,440 Go fix some stuff. 691 00:39:52,120 --> 00:39:57,600 I think it's always every successful product is a bit like this. 692 00:39:57,800 --> 00:39:58,200 Nikolay: Right. 693 00:39:58,200 --> 00:40:02,000 But sometimes humans start using different LLM when reviewing 694 00:40:02,000 --> 00:40:02,820 things, right? 695 00:40:03,560 --> 00:40:04,780 Being lazy, right? 696 00:40:06,500 --> 00:40:07,000 Gwen: Yes. 697 00:40:08,420 --> 00:40:09,220 Nikolay: That's interesting. 698 00:40:10,840 --> 00:40:15,520 So I'm not sure how successful it will be, but my gut tells me 699 00:40:15,520 --> 00:40:18,740 that we need to move, move, move in this direction anyway and 700 00:40:18,740 --> 00:40:22,280 have some, I don't know, like more experiments and so on. 701 00:40:22,280 --> 00:40:26,260 I hope we will have more soon to publish, I mean, and start iterating. 702 00:40:27,880 --> 00:40:33,640 But yeah, it was a rollercoaster last year, So now we are rebuilding 703 00:40:33,640 --> 00:40:34,140 stuff. 704 00:40:35,460 --> 00:40:36,920 We will see how it works. 705 00:40:37,660 --> 00:40:41,160 And for example, 1 of the experiments we must do, I think, is 706 00:40:41,160 --> 00:40:43,660 to conduct various benchmarks for RLS. 707 00:40:44,800 --> 00:40:45,800 Because it's obviously... 708 00:40:45,800 --> 00:40:46,320 That would 709 00:40:46,320 --> 00:40:47,860 Gwen: be a fantastic example. 710 00:40:48,160 --> 00:40:48,660 Yes. 711 00:40:49,760 --> 00:40:52,280 And I mean, this is something that humans with experience are 712 00:40:52,280 --> 00:40:56,960 pretty good at finding cases where you're like, is RLS going 713 00:40:56,960 --> 00:41:04,340 to actually be an issue and have the stories that then the LLM 714 00:41:04,340 --> 00:41:06,180 can go implement and test. 715 00:41:06,340 --> 00:41:10,580 Nikolay: Yeah, we had several cases and also Supabase has public 716 00:41:10,580 --> 00:41:12,680 materials, blog posts about this. 717 00:41:12,720 --> 00:41:16,720 Obviously, there is already quite known case when you have like 718 00:41:16,720 --> 00:41:20,040 current_setting function inside RLS expression. 719 00:41:21,100 --> 00:41:28,700 And this is, yeah, if you select count 1000000 rows, it's terrible. 720 00:41:28,860 --> 00:41:31,560 And it's quite easy to fix actually. 721 00:41:31,560 --> 00:41:35,040 But yeah, so these kinds of experiments to collect them and see 722 00:41:35,040 --> 00:41:35,540 how. 723 00:41:35,580 --> 00:41:38,680 Actually, my goal with this experiment I think will be to prove 724 00:41:38,680 --> 00:41:42,420 that they are not a problem if you do it right. 725 00:41:42,900 --> 00:41:43,400 Gwen: Interesting. 726 00:41:43,980 --> 00:41:48,680 I would contribute, I think there was a recent post on the bug 727 00:41:48,680 --> 00:41:53,720 tracker where basically the optimizer, the planner refused to 728 00:41:53,720 --> 00:41:59,380 use, I think it was a GiST index or a GiN index due to the belief 729 00:41:59,380 --> 00:42:04,080 that the RLS optimization is incorrect, is unsafe. 730 00:42:04,900 --> 00:42:06,800 I can look it up and send it to you. 731 00:42:06,900 --> 00:42:08,480 Yeah, it's interesting. 732 00:42:08,480 --> 00:42:10,700 But yeah, that can also be interesting. 733 00:42:10,940 --> 00:42:15,620 Like if you have a fix for Postgres, then you can, it will obviously 734 00:42:15,720 --> 00:42:17,920 be nice to showcase. 735 00:42:18,820 --> 00:42:19,320 Nikolay: Right. 736 00:42:19,600 --> 00:42:25,280 I think also if you, so you don't use RLS, you said bypass it, 737 00:42:25,280 --> 00:42:25,760 right? 738 00:42:25,760 --> 00:42:27,280 Gwen: We bypass it, yeah. 739 00:42:27,880 --> 00:42:28,860 Nikolay: Yeah, that's interesting. 740 00:42:28,860 --> 00:42:34,660 I'm curious if you, in this mixed schema when we have partitions 741 00:42:34,780 --> 00:42:41,140 or shards, and RLS doesn't make sense at all to involve RLS additionally 742 00:42:41,280 --> 00:42:46,480 locally if some of shards have a mixed pool of customers. 743 00:42:46,500 --> 00:42:49,600 Gwen: Okay, 1 of the questions that I have in mind, and this 744 00:42:49,600 --> 00:42:53,220 is something that we're trying to help figure out for our users, 745 00:42:53,900 --> 00:42:57,680 often on top of the tenant, you still have permissions for specific 746 00:42:57,780 --> 00:42:59,120 users in the tenant. 747 00:42:59,120 --> 00:43:02,560 Like you have an admin that can do anything, and then you may 748 00:43:02,560 --> 00:43:05,680 have someone who is not allowed to see some rows at all because 749 00:43:05,680 --> 00:43:07,940 they're too sensitive, all this kind of stuff. 750 00:43:08,600 --> 00:43:13,380 And our users ask us, they can do it with RLS inside those partitions, 751 00:43:14,500 --> 00:43:19,440 or they can do it in their application. 752 00:43:19,780 --> 00:43:23,540 There is a lot of application level tools or like middleware 753 00:43:23,680 --> 00:43:26,060 kind of tools that will do it for them. 754 00:43:26,180 --> 00:43:31,520 Is it better to do it in the app layer or in Postgres with RLS 755 00:43:31,840 --> 00:43:34,900 is a good question that I don't have an immediate answer for. 756 00:43:35,380 --> 00:43:35,660 Nikolay: Right. 757 00:43:35,660 --> 00:43:39,900 So there are several layers of multi-tenancy basically. 758 00:43:40,840 --> 00:43:41,340 Yeah. 759 00:43:41,400 --> 00:43:44,160 Not multi-tenancy, but different layers. 760 00:43:45,020 --> 00:43:49,860 And If your customer being a tenant for you, they might have 761 00:43:49,860 --> 00:43:54,460 tenants for them, but also inside them, they might have additional 762 00:43:55,600 --> 00:43:59,760 clusters or segments of users inside each tenant. 763 00:43:59,820 --> 00:44:04,340 So it's apartments and rooms inside apartments. 764 00:44:05,220 --> 00:44:05,720 Gwen: Nice. 765 00:44:06,100 --> 00:44:07,160 I like that. 766 00:44:08,860 --> 00:44:09,360 Yeah. 767 00:44:09,520 --> 00:44:13,030 This is 1 of the things that make multi-tenancy confusing, right? 768 00:44:13,030 --> 00:44:17,520 Because it's almost like Matryoshka kind of scenario. 769 00:44:17,800 --> 00:44:18,300 Nikolay: Yes. 770 00:44:18,620 --> 00:44:21,600 This term, by the way, used in Postgres ecosystem multiple times, 771 00:44:21,600 --> 00:44:25,520 starting with, you mentioned GiST original paper by Hellerstein 772 00:44:25,900 --> 00:44:32,020 mentions that Matryoshka.., RD-tree, Russian doll tree, so basically 773 00:44:32,020 --> 00:44:32,130 Matryoshka 774 00:44:32,130 --> 00:44:32,240 tree. 775 00:44:32,240 --> 00:44:33,140 Gwen: That's true. 776 00:44:33,840 --> 00:44:39,220 And also in embeddings, they have the Matryoshka type embeddings 777 00:44:39,340 --> 00:44:42,320 where you can make them of any size. 778 00:44:42,800 --> 00:44:44,200 Nikolay: Yeah, I read about this as well. 779 00:44:44,200 --> 00:44:45,140 Yeah, it's funny. 780 00:44:45,580 --> 00:44:46,560 So yeah, great. 781 00:44:47,120 --> 00:44:50,180 And what are your plans for AI? 782 00:44:50,180 --> 00:44:54,580 I saw MCP servers already with some integration. 783 00:44:55,640 --> 00:44:57,900 Gwen: We basically have 3 directions. 784 00:44:58,360 --> 00:45:03,220 1 is MCP server and making it public, giving it authentication. 785 00:45:03,600 --> 00:45:06,560 Right now, it's open-source, you can run it on your own, but 786 00:45:06,560 --> 00:45:10,180 we're not hosting it, so we should start hosting it at some point. 787 00:45:11,400 --> 00:45:15,400 So things that just make people who use LLM, make it easier for 788 00:45:15,400 --> 00:45:16,580 them to use it. 789 00:45:16,840 --> 00:45:19,820 Other thing we've done that we think is very useful for LLMs, 790 00:45:19,820 --> 00:45:24,040 this we already have, just make it 0 time to create new databases. 791 00:45:24,640 --> 00:45:28,220 Because LLMs just 0 time and 0 cost. 792 00:45:28,440 --> 00:45:30,300 Because they love creating a lot of them. 793 00:45:30,300 --> 00:45:33,880 Every time something goes wrong, okay, let's try from scratch 794 00:45:34,120 --> 00:45:36,560 with a new database kind of situation. 795 00:45:37,540 --> 00:45:39,400 So we're making it fast and cheap. 796 00:45:39,440 --> 00:45:42,080 The other thing that we're still working on is really to make 797 00:45:42,080 --> 00:45:44,200 our documentation more LLM friendly. 798 00:45:45,240 --> 00:45:48,540 LLMs.txt is absolutely not enough. 799 00:45:49,760 --> 00:45:53,720 It is actually, it creates a very large file. 800 00:45:53,720 --> 00:45:55,680 The LLMs tend to get lost in it. 801 00:45:55,680 --> 00:45:58,180 We need to figure out how to make it better. 802 00:45:59,440 --> 00:46:03,220 And then also, everyone is kind of thinking, can we have our 803 00:46:03,220 --> 00:46:03,980 own agents? 804 00:46:04,340 --> 00:46:06,800 Can we do something around that? 805 00:46:07,540 --> 00:46:09,360 We're kind of thinking about that. 806 00:46:09,900 --> 00:46:14,540 Something we already have that is useful is just that with the 807 00:46:14,540 --> 00:46:19,540 multi-tenant model, the embedding, the vector indexes are much 808 00:46:19,540 --> 00:46:20,040 smaller. 809 00:46:20,740 --> 00:46:25,120 And this is a huge deal for people building those agents and 810 00:46:25,120 --> 00:46:25,620 LLMs. 811 00:46:26,440 --> 00:46:26,780 Nikolay: Yeah. 812 00:46:26,780 --> 00:46:32,120 And actually, you mentioned, so this zero startup goal, you mentioned 813 00:46:32,120 --> 00:46:34,440 separation of compute and data. 814 00:46:34,860 --> 00:46:36,240 How is that achieved? 815 00:46:36,500 --> 00:46:38,540 Using what approach? 816 00:46:39,160 --> 00:46:39,940 Can you elaborate? 817 00:46:39,940 --> 00:46:40,520 Gwen: Oh, yeah. 818 00:46:40,520 --> 00:46:41,000 Sorry. 819 00:46:41,000 --> 00:46:45,780 This is more than a two-minute answer, but the short one is that 820 00:46:45,780 --> 00:46:49,960 you kind of, you patch Postgres, and then you find a better way 821 00:46:49,960 --> 00:46:55,240 to do your storage and basically wrap every Postgres storage 822 00:46:55,240 --> 00:46:58,700 function with an equivalent with your storage. 823 00:46:59,340 --> 00:47:03,940 And then you also need to apply the WAL continuously to the 824 00:47:03,940 --> 00:47:04,860 storage layer. 825 00:47:05,180 --> 00:47:08,300 Nikolay: Great job doing this like in actually 20 seconds. 826 00:47:09,320 --> 00:47:10,720 I understood very well. 827 00:47:10,720 --> 00:47:11,220 Yeah. 828 00:47:11,280 --> 00:47:15,400 So, and do you have plans to make this open source as well? 829 00:47:16,800 --> 00:47:17,820 Gwen: Yes, absolutely. 830 00:47:17,860 --> 00:47:21,540 I mean, this is a first of all, as you know, WAL readers have 831 00:47:21,540 --> 00:47:27,280 to be registered, ours already is, and with the open-source license. 832 00:47:28,380 --> 00:47:31,400 And then, yeah, we are planning to open 833 00:47:31,400 --> 00:47:31,880 Nikolay: source our 834 00:47:31,880 --> 00:47:33,720 Gwen: storage layer, our extension. 835 00:47:34,200 --> 00:47:37,940 I think at this point it's about seven different patches we've made 836 00:47:38,080 --> 00:47:38,800 on Postgres. 837 00:47:39,160 --> 00:47:42,320 I don't think we'll want to open source it as a Postgres fork, 838 00:47:42,660 --> 00:47:46,100 because the number of patches is quite small and we are maintaining 839 00:47:46,220 --> 00:47:46,680 it. 840 00:47:46,680 --> 00:47:51,020 I think everything from Postgres 12 to 18 at this point, or maybe 841 00:47:51,020 --> 00:47:53,500 13 to 18, something along those lines. 842 00:47:53,680 --> 00:47:55,580 Nikolay: All supported versions, I guess. 843 00:47:55,960 --> 00:47:56,460 Yeah. 844 00:47:56,460 --> 00:47:57,040 That's great. 845 00:47:57,040 --> 00:47:57,400 Yeah. 846 00:47:57,400 --> 00:47:59,440 Well, looking forward to checking it out. 847 00:47:59,440 --> 00:48:02,880 And actually, one more question for me, maybe last one. 848 00:48:02,980 --> 00:48:06,800 Are you open to some benchmarks we probably will do? 849 00:48:07,200 --> 00:48:09,680 We plan to do some benchmarks with various platforms. 850 00:48:10,520 --> 00:48:14,240 It all started after acquisition of Neon and there's a discussion 851 00:48:14,680 --> 00:48:15,560 on LinkedIn. 852 00:48:15,600 --> 00:48:18,900 So I thought about trying some benchmarks for different platforms. 853 00:48:19,540 --> 00:48:23,800 How do you think, like what are your thoughts about it? 854 00:48:24,280 --> 00:48:26,060 Gwen: Oh my God, this is very scary. 855 00:48:26,140 --> 00:48:29,120 We are benchmarking ourselves all the time. 856 00:48:29,180 --> 00:48:33,220 So I'm keenly aware of exactly like what benchmarks make us look 857 00:48:33,220 --> 00:48:36,020 good and what benchmark make us look bad. 858 00:48:36,020 --> 00:48:39,800 And I think that's also how we react to benchmarks in social 859 00:48:39,800 --> 00:48:40,300 media. 860 00:48:40,640 --> 00:48:44,020 Unless it's a benchmark that shows something in Postgres itself 861 00:48:44,040 --> 00:48:47,540 and like clearly attempting to educate and help people. 862 00:48:48,260 --> 00:48:52,280 You can design a benchmark to make anyone in the world look bad. 863 00:48:52,280 --> 00:48:53,252 Nikolay: Benchmarketing, it's called. 864 00:48:53,252 --> 00:48:53,508 Michael: You can design 865 00:48:53,508 --> 00:48:53,860 Nikolay: a benchmark to 866 00:48:53,860 --> 00:48:55,240 Gwen: make everyone look good. 867 00:48:55,240 --> 00:48:55,740 Exactly. 868 00:48:56,440 --> 00:49:01,800 So, I will, at any given time, I can publish my benchmarks that 869 00:49:01,800 --> 00:49:07,720 makes Nile look fantastic, and you can run benchmarks that you 870 00:49:07,720 --> 00:49:12,780 decided how to build them, and may be very realistic and may 871 00:49:12,780 --> 00:49:15,900 even expose a problem that we have that I didn't know about. 872 00:49:16,840 --> 00:49:19,080 But, yeah, in general, I love benchmarks. 873 00:49:19,920 --> 00:49:23,620 I just have opinions on how benchmarks are used by marketing 874 00:49:23,620 --> 00:49:24,120 people. 875 00:49:25,580 --> 00:49:26,660 Michael: Very good answer. 876 00:49:27,260 --> 00:49:30,880 For anybody interested in a bit more about Niall's architecture, 877 00:49:31,120 --> 00:49:35,140 Gwen gave a really good talk at PGConf.dev recently and the video 878 00:49:35,140 --> 00:49:37,800 just went up on YouTube so I will put that in the show notes 879 00:49:37,800 --> 00:49:40,020 for anybody that wants a deeper dive there. 880 00:49:40,840 --> 00:49:45,560 There was also a good talk I saw at a PgDay Paris event by Pierre 881 00:49:45,580 --> 00:49:48,560 Ducroquet, I'm not sure if I'm pronouncing that anywhere near 882 00:49:48,560 --> 00:49:52,360 correctly, all about multi-tenant database design, especially 883 00:49:52,460 --> 00:49:56,260 focusing on something we didn't focus much on today, which was 884 00:49:56,600 --> 00:50:01,620 the downsides of schema per tenant design, including things like 885 00:50:01,620 --> 00:50:04,300 observability and monitoring, which I thought was really fascinating. 886 00:50:04,360 --> 00:50:07,360 So anybody considering going down that route definitely check 887 00:50:07,360 --> 00:50:09,940 out that video and I'll put that in the show notes as well. 888 00:50:10,120 --> 00:50:12,900 So yeah thank you so much Gwen I think we're out of time it's 889 00:50:12,900 --> 00:50:13,940 been a real pleasure. 890 00:50:14,060 --> 00:50:16,380 Gwen: It's been a pleasure Thank you for having me on.