1 00:00:05,049 --> 00:00:08,449 Michael: Hello and welcome to PostgresFM, a weekly show about all things PostgreSQL. 2 00:00:08,529 --> 00:00:11,320 I am Michael, founder of pgMustard, and I'm joined as always by 3 00:00:11,320 --> 00:00:13,140 Nik, founder of Postgres.ai. 4 00:00:13,140 --> 00:00:13,160 Hey, Nik 5 00:00:13,718 --> 00:00:14,328 Nik: Well, Michael 6 00:00:14,908 --> 00:00:17,823 Michael: Hello, and we have a special guest today, Tudor 7 00:00:17,903 --> 00:00:20,561 Golobenco from Xata, CTO there. 8 00:00:20,881 --> 00:00:21,921 Really nice to have you, Tudor. 9 00:00:21,921 --> 00:00:22,691 Thanks for joining us 10 00:00:23,133 --> 00:00:24,553 Tudor: hi Michael, hi Nick, hi everyone. 11 00:00:24,693 --> 00:00:26,283 Thank you so much for having me 12 00:00:27,020 --> 00:00:28,260 Michael: It is our pleasure. 13 00:00:28,350 --> 00:00:31,370 So to kick us off, I wonder if you could give us a bit of the 14 00:00:31,400 --> 00:00:35,320 background story of Xata and take us through to where we are today 15 00:00:36,095 --> 00:00:40,377 Tudor: Yeah, so I will actually start with what it is today because we did a 16 00:00:40,407 --> 00:00:45,747 change once, like a pivot or like a-- I call it more like a technical pivot. 17 00:00:46,227 --> 00:00:53,696 So it's a Postgres platform which has as differentiating features a very fast 18 00:00:53,726 --> 00:00:59,826 copy-on-write branching, so it can create a branching in a matter of seconds, even 19 00:00:59,836 --> 00:01:07,256 for databases that, that are terabytes in sizes, and scale to zero which means that 20 00:01:07,806 --> 00:01:12,926 if a database is inactive, the compute part shuts down, and then on the next 21 00:01:12,936 --> 00:01:15,286 connection, it automatically wakes up. 22 00:01:15,316 --> 00:01:17,166 And this wake up is very fast. 23 00:01:17,176 --> 00:01:18,726 We're targeting one second. 24 00:01:19,316 --> 00:01:25,556 So that makes it very convenient, especially for developer workflows. 25 00:01:25,556 --> 00:01:30,876 So these non-production use cases, things like if you want a Postgres branch for 26 00:01:30,876 --> 00:01:37,006 each pull request or if you want to create an, sort of an ephemeral database for each 27 00:01:37,056 --> 00:01:40,176 agent run or for your agent and so on. 28 00:01:40,786 --> 00:01:43,556 Now, these are our more common use cases. 29 00:01:44,276 --> 00:01:48,606 But on the Xata platform we can and do run production workloads as well. 30 00:01:48,606 --> 00:01:52,933 So the differentiating features are on the developer experience if you 31 00:01:52,933 --> 00:01:57,243 want, but you can use it as a normal Postgres hosting provider as well. 32 00:01:57,973 --> 00:02:00,183 So that's what Xata is today. 33 00:02:00,303 --> 00:02:02,363 It didn't really start like that. 34 00:02:02,413 --> 00:02:09,063 The initial idea five years ago my partner Monica Sarbu, she wanted to do 35 00:02:09,063 --> 00:02:11,473 a database that's really easy to use. 36 00:02:11,473 --> 00:02:15,933 It was a lot of inspiration from Airbase or spreadsheets and because they were 37 00:02:15,933 --> 00:02:22,053 like like developer using Airbase as a database or Google Sheets as a database. 38 00:02:22,453 --> 00:02:27,693 So the initial idea was to make one that's as easy as to use as a 39 00:02:27,703 --> 00:02:31,503 spreadsheet but that scales up so you don't have to switch at some point. 40 00:02:31,503 --> 00:02:33,553 So having features from both. 41 00:02:34,113 --> 00:02:39,133 And we did that for a while with some moderate success. 42 00:02:39,243 --> 00:02:44,383 But at some point, we decided that it's pretty hard to build a 43 00:02:44,443 --> 00:02:47,443 long-term business on that audience. 44 00:02:47,493 --> 00:02:51,503 And it n-never really took off as a primary database 45 00:02:51,513 --> 00:02:53,993 for, like for more production 46 00:02:54,407 --> 00:02:54,757 Nik: This… 47 00:02:55,133 --> 00:02:57,163 Tudor: use as teams and things like this. 48 00:02:57,217 --> 00:02:59,367 Nik: Apologies for interrupting, because you're packing a lot of 49 00:02:59,377 --> 00:03:01,573 stuff in one a very short time. 50 00:03:01,943 --> 00:03:05,953 I just wanted to pause a little bit here and it- this reminds me a bit IO, right? 51 00:03:05,963 --> 00:03:08,833 Th- they also like drag and drop CSV file. 52 00:03:09,173 --> 00:03:13,473 You don't think about schema or anything, but it's Postgres behind it. 53 00:03:13,683 --> 00:03:16,123 So you had similar thing, right? 54 00:03:16,655 --> 00:03:19,605 Tudor: Yeah, but p.io were actual Postgres. 55 00:03:20,055 --> 00:03:23,635 Ours, we were actually hiding Postgres behind our own API. 56 00:03:23,635 --> 00:03:29,065 So it was a REST API that you interacted with data a bit like 57 00:03:29,095 --> 00:03:31,315 PostgREST or something like this. 58 00:03:31,865 --> 00:03:36,445 Nicer, I would say, but but that, that type of interfaces. 59 00:03:36,445 --> 00:03:36,835 Yeah 60 00:03:37,239 --> 00:03:40,529 Nik: So it's I like this idea a lot but Bit.io was closed. 61 00:03:40,529 --> 00:03:43,929 I remember like a very short notice, like two weeks notice or something. 62 00:03:43,929 --> 00:03:46,219 People were like outraged. 63 00:03:46,229 --> 00:03:51,216 That was yet another not fully Postgres platform, but I have something, 64 00:03:51,226 --> 00:03:54,146 like I store my data somewhere and but then it's closed and I have 65 00:03:54,156 --> 00:03:55,686 very short notice to grab my data. 66 00:03:55,836 --> 00:03:59,096 And you are telling us right now that also this model didn't work well, right? 67 00:04:00,176 --> 00:04:00,486 In terms of- 68 00:04:00,524 --> 00:04:01,124 Tudor: yeah. 69 00:04:01,794 --> 00:04:05,454 Yeah, like there's definitely companies that do-- that address 70 00:04:05,494 --> 00:04:07,464 this sort of audience successfully. 71 00:04:07,834 --> 00:04:10,734 Like I'm thinking of Supabase, for example, right? 72 00:04:11,044 --> 00:04:16,234 But they have like millions of developers and such, and out of those millions, 73 00:04:16,234 --> 00:04:22,004 you're going to have a small minority that will become large databases, so 74 00:04:22,004 --> 00:04:24,044 they will pay a lot and so on, right? 75 00:04:24,364 --> 00:04:26,804 And we had a few tens of thousand. 76 00:04:26,854 --> 00:04:29,914 I think at the peak we had 50,000 users. 77 00:04:30,804 --> 00:04:35,884 Turns out of those, there's not enough paying customers 78 00:04:36,034 --> 00:04:37,644 to sustain such a business. 79 00:04:37,694 --> 00:04:38,804 It didn't work for 80 00:04:38,890 --> 00:04:39,560 Nik: Thank you for sharing. 81 00:04:39,560 --> 00:04:40,760 This is super interesting. 82 00:04:40,930 --> 00:04:43,050 50,000 it's actually a lot. 83 00:04:43,570 --> 00:04:43,830 But 84 00:04:44,010 --> 00:04:44,250 Tudor: yeah. 85 00:04:44,270 --> 00:04:45,170 It was quite a lot. 86 00:04:45,330 --> 00:04:48,550 We had the growth in the number of users, but that didn't 87 00:04:48,550 --> 00:04:50,670 reflect in the growth in revenue. 88 00:04:50,968 --> 00:04:52,878 Nik: Many of them are not paying. 89 00:04:53,128 --> 00:04:53,428 Most 90 00:04:53,530 --> 00:04:53,980 Tudor: Yeah. 91 00:04:54,488 --> 00:04:54,758 Nik: Yeah 92 00:04:55,636 --> 00:04:56,536 I feel you well. 93 00:04:56,536 --> 00:04:59,866 I understand this problem also, like from different kinds of businesses. 94 00:04:59,866 --> 00:05:00,146 Yeah. 95 00:05:00,405 --> 00:05:02,435 It's hard to find business model on this. 96 00:05:03,425 --> 00:05:04,685 Unless you're Supabase. 97 00:05:04,685 --> 00:05:05,725 This is a good comparison. 98 00:05:05,725 --> 00:05:06,415 Thank you so much. 99 00:05:06,425 --> 00:05:06,765 It's super. 100 00:05:06,855 --> 00:05:11,365 Like before th-this recording I said we were, we're going to focus 101 00:05:11,365 --> 00:05:14,785 on technical stuff, but the start is very different than what… 102 00:05:15,655 --> 00:05:16,315 and I like it. 103 00:05:16,315 --> 00:05:16,335 Yeah. 104 00:05:16,645 --> 00:05:17,425 Thank you so much. 105 00:05:18,190 --> 00:05:21,490 Michael: I think it's really useful context to then dive into the technical 106 00:05:21,490 --> 00:05:26,610 side because I remember one of your posts that I thought was really good 107 00:05:26,720 --> 00:05:29,912 about the architecture which I guess might have been the architecture all 108 00:05:29,932 --> 00:05:34,222 along, but it explained how you could offer a free plan and how the free 109 00:05:34,242 --> 00:05:38,002 plan didn't actually cost you very much because of your architectural choices. 110 00:05:38,382 --> 00:05:42,282 And that, that was a really nice post from the business perspective but also 111 00:05:42,282 --> 00:05:45,312 from a technical perspective, like understanding how those trade-offs 112 00:05:45,312 --> 00:05:46,732 actually help you as a business. 113 00:05:47,012 --> 00:05:49,962 Tudor: Yeah, we-- So on the new platform, we don't have a free tier. 114 00:05:49,966 --> 00:05:53,756 this is again, the business side, and our experience with the old platform. 115 00:05:53,766 --> 00:05:56,146 So we call the old platform Zeta Lite. 116 00:05:56,156 --> 00:05:57,216 It's now closed. 117 00:05:57,266 --> 00:06:01,316 We migrated the users that we could and so on, but it's now 118 00:06:01,316 --> 00:06:02,826 closed, so now it's only Zeta 119 00:06:03,040 --> 00:06:08,750 but the old platform had a had a free tier and when doing the new platform, we 120 00:06:08,750 --> 00:06:11,180 were not very eager to add another one. 121 00:06:11,290 --> 00:06:15,660 Although with our current architecture, we can do it again because we have 122 00:06:15,660 --> 00:06:19,080 this scale to zero, which is also very efficient from that point of view. 123 00:06:19,800 --> 00:06:24,670 But we're not very eager because it adds a lot of noise and like I was saying earlier 124 00:06:24,690 --> 00:06:27,760 you have some growth, but as long as… 125 00:06:27,830 --> 00:06:31,230 If that doesn't really result in in the type of business that 126 00:06:31,230 --> 00:06:35,000 you want, you get a bunch of feedback that's not really useful. 127 00:06:35,280 --> 00:06:38,910 And it confuses things a little bit more on the business side. 128 00:06:39,010 --> 00:06:41,800 We might come back to that decision and maybe eventually 129 00:06:41,800 --> 00:06:44,890 we'll add again a free tier plan, but currently we don't have one. 130 00:06:45,926 --> 00:06:49,306 Nik: Yeah, let's shift to more technical stuff because I know you 131 00:06:49,306 --> 00:06:53,096 use CloudNativePG, you have branching, you have very interesting open 132 00:06:53,096 --> 00:06:56,506 source tools like pgStream developed and others like pgRole, right? 133 00:06:56,992 --> 00:07:00,292 I'm very curious to talk with you about any of them. 134 00:07:01,266 --> 00:07:04,426 Tudor: Yeah, maybe we start at the Xata Store. 135 00:07:04,636 --> 00:07:07,926 It's our storage engine that makes a lot of this possible. 136 00:07:08,256 --> 00:07:11,976 And then we can go about how we use CloudNativePG to get 137 00:07:12,026 --> 00:07:13,736 the very fast wake-up times. 138 00:07:13,746 --> 00:07:15,196 That's also, I think, interesting. 139 00:07:15,576 --> 00:07:17,296 Nik: So copy on write and branching, 140 00:07:18,036 --> 00:07:18,106 right? 141 00:07:18,686 --> 00:07:19,146 Tudor: yeah. 142 00:07:19,367 --> 00:07:24,557 so we do copy-on-write branching, but we do it at the underlying volume level. 143 00:07:24,757 --> 00:07:28,997 So because our feature set is pretty similar with Neon, right? 144 00:07:28,997 --> 00:07:32,507 Also like copy-on-write branching and then scale to zero with 145 00:07:32,507 --> 00:07:34,027 fast wake-up times, these 146 00:07:34,105 --> 00:07:36,045 Nik: decouple storage and compute, right? 147 00:07:36,647 --> 00:07:37,397 Tudor: Yeah, the 148 00:07:37,485 --> 00:07:40,225 Nik: And you re- and you recently made everything open source, 149 00:07:40,705 --> 00:07:42,805 Apache 2.0 or no, or not everything 150 00:07:42,861 --> 00:07:47,821 Tudor: it not quite everything, but enough to run the whole platform yourself. 151 00:07:48,341 --> 00:07:53,751 But it won't scale to the same levels that we scale our own platform, and you won't 152 00:07:53,751 --> 00:07:56,891 get as fast times as with our platform. 153 00:07:56,891 --> 00:08:01,151 I can go into more details about exactly why, but we have op-open 154 00:08:01,151 --> 00:08:04,831 source enough to be, to be running it yourself and to be useful. 155 00:08:05,391 --> 00:08:07,641 But it's not quite everything open source in there 156 00:08:08,961 --> 00:08:09,261 Nik: Cool. 157 00:08:09,291 --> 00:08:13,471 So if we talk about storage, like what's interesting there 158 00:08:13,708 --> 00:08:17,048 Tudor: Yeah, so Zeta Store, we call it like the Zeta Store and it's-- this 159 00:08:17,048 --> 00:08:19,218 is like one of the proprietary pieces. 160 00:08:19,938 --> 00:08:22,898 And we have quite a long history into that. 161 00:08:22,998 --> 00:08:28,108 We tried a bunch of products and Kubernetes tools for managing 162 00:08:28,138 --> 00:08:29,928 storage, the storage layer. 163 00:08:30,598 --> 00:08:35,368 As in the end, we decided to build our own, but also use a bunch of ZFS 164 00:08:35,368 --> 00:08:37,308 functionality, so it's based on that. 165 00:08:37,798 --> 00:08:42,648 The way it works is, and I'm not sure how familiar you are with ZFS, it's 166 00:08:43,022 --> 00:08:44,412 Nik: let me interrupt you here. 167 00:08:44,982 --> 00:08:49,052 my company, Postgres.ai, has Database Lab Engine, which implemented 168 00:08:49,392 --> 00:08:54,192 database branching in 2019 before Neon and before Xata, right? 169 00:08:54,252 --> 00:08:57,612 And I have a lot of questions related to your experience with database 170 00:08:57,612 --> 00:09:01,502 branching and how you convince users that it's really needed. 171 00:09:01,742 --> 00:09:05,582 Because we do it seven years and some of them are solved very easily, 172 00:09:05,582 --> 00:09:08,812 but some of them still struggle and are like, "Oh, production data, 173 00:09:08,842 --> 00:09:11,232 we need to change it," and so on. 174 00:09:11,242 --> 00:09:15,382 Let's talk about this separately, but I just to give you context 175 00:09:15,442 --> 00:09:16,542 that you probably don't have. 176 00:09:17,482 --> 00:09:17,922 Tudor: yeah. 177 00:09:17,922 --> 00:09:17,932 Yeah 178 00:09:18,232 --> 00:09:19,262 Nik: And it's on ZFS. 179 00:09:19,370 --> 00:09:19,380 Yeah. 180 00:09:19,926 --> 00:09:22,006 Tudor: Oh, are you using ZFS? 181 00:09:22,006 --> 00:09:24,056 I thought you are using BTRFS. 182 00:09:24,540 --> 00:09:25,096 Uh, no 183 00:09:25,096 --> 00:09:27,606 Nik: it was not stable when we started, and I don't know 184 00:09:27,660 --> 00:09:29,160 Tudor: Gotcha. 185 00:09:29,160 --> 00:09:29,580 Okay. 186 00:09:29,620 --> 00:09:30,890 I was actually remembering 187 00:09:30,900 --> 00:09:31,360 wrong then 188 00:09:31,716 --> 00:09:32,766 Nik: It was actually interesting. 189 00:09:32,766 --> 00:09:36,826 When I created it first, it was out came out of ex-experience with GitLab. 190 00:09:37,326 --> 00:09:41,176 We just needed to give backend engineers capabilities to experiment 191 00:09:41,216 --> 00:09:47,256 and see ex-explain plans not touching a few database experts, right? 192 00:09:47,256 --> 00:09:48,216 And it's, it was great. 193 00:09:48,216 --> 00:09:48,916 It worked great. 194 00:09:48,926 --> 00:09:54,136 And then we scaled it and the problem always was that first of all, production 195 00:09:54,136 --> 00:09:56,716 data how to protect this, right? 196 00:09:56,866 --> 00:10:02,526 And also when Neon started working with branching, it's very different. 197 00:10:02,526 --> 00:10:07,806 So we do it very locally, and we achieve static cost for all clones and branches. 198 00:10:07,806 --> 00:10:11,486 So you have, you pay once and you have everything, like mu-multi-tenancy 199 00:10:11,486 --> 00:10:13,186 in one virtual machine, basically. 200 00:10:13,756 --> 00:10:17,906 You have very different model, so I'm very curious how it works and what's your… 201 00:10:18,416 --> 00:10:20,116 what do you observe how people use it? 202 00:10:20,556 --> 00:10:25,456 And because my honest take, I spent a lot of time in this area, 203 00:10:25,836 --> 00:10:29,686 market is still not understanding that branching is super powerful. 204 00:10:29,996 --> 00:10:30,966 And I hope AI will 205 00:10:30,974 --> 00:10:31,354 Tudor: I think-- 206 00:10:31,566 --> 00:10:32,136 Nik: but yeah. 207 00:10:32,786 --> 00:10:36,806 Tudor: I think it's it's it's growing pretty nicely. 208 00:10:37,286 --> 00:10:42,216 But yeah the, like the use cases that we see most often are branch per PR. 209 00:10:42,886 --> 00:10:47,859 So they do PR and have some sort of preview environments. 210 00:10:48,039 --> 00:10:52,459 This could be, like Vercel or their own stuff or different 211 00:10:52,459 --> 00:10:58,899 other platforms for ephemeral environments or preview environments. 212 00:10:59,459 --> 00:11:04,369 But then they want also the database to be part of that with the data, right? 213 00:11:04,439 --> 00:11:07,889 So then a platform like ours makes that possible. 214 00:11:07,929 --> 00:11:10,279 You have the database with the data. 215 00:11:10,769 --> 00:11:13,489 It doesn't have to be the exact production data. 216 00:11:13,899 --> 00:11:19,089 And the way we typically recommend people to do is we tell them, 217 00:11:19,089 --> 00:11:20,909 "Look, keep production where it is." 218 00:11:20,949 --> 00:11:22,199 Let's say it's RDS. 219 00:11:22,769 --> 00:11:25,399 And then we have, this is why we develop pgStream. 220 00:11:26,143 --> 00:11:28,303 We create a replica of it. 221 00:11:28,473 --> 00:11:31,603 And pgStream can copy the data once. 222 00:11:31,793 --> 00:11:35,993 So for small databases, we recommend people to set up a nightly job 223 00:11:36,123 --> 00:11:38,903 or a weekly job or something like this because it's simpler. 224 00:11:39,583 --> 00:11:44,563 For large databases, pgStream can also do the initial snapshot and then 225 00:11:44,563 --> 00:11:47,403 keep it up via a logical replication. 226 00:11:48,185 --> 00:11:48,205 Nik: And DDL? 227 00:11:48,225 --> 00:11:48,765 And 228 00:11:50,021 --> 00:11:51,001 Tudor: and DDL, 229 00:11:51,413 --> 00:11:52,223 Nik: you solve DDL. 230 00:11:52,223 --> 00:11:56,813 I looked at it briefly, and I'm very curious, like, how… if 231 00:11:56,823 --> 00:11:59,293 you do it and what's your scale? 232 00:11:59,303 --> 00:12:00,603 Is it working really well? 233 00:12:00,623 --> 00:12:03,633 If it's working really well, why it doesn't go to core 234 00:12:03,633 --> 00:12:05,423 of Postgres in that form? 235 00:12:06,289 --> 00:12:06,549 Tudor: Yeah. 236 00:12:06,569 --> 00:12:10,447 I think there's some work in progress to make it in in, In co- 237 00:12:10,799 --> 00:12:15,239 Nik: it-- there are several, there, there are several ways, and if you 238 00:12:15,239 --> 00:12:18,715 check recent activities it's not super progress-- it's not progressing 239 00:12:19,267 --> 00:12:20,437 Tudor: And not progressing, yeah. 240 00:12:20,541 --> 00:12:25,191 actually a little bit of backstory on, on pgStream. 241 00:12:25,831 --> 00:12:30,981 We were doing in the old product, so Zetalight, we were doing, we were 242 00:12:30,981 --> 00:12:34,941 offering this REST API, and then most of it was powered by Postgres. 243 00:12:35,531 --> 00:12:41,801 But we also had a free text search with, BM25 and relevance and so on. 244 00:12:42,181 --> 00:12:44,801 And that was actually searched from Elasticsearch. 245 00:12:44,811 --> 00:12:48,711 So we wanted to have this replication between Postgres and Elasticsearch. 246 00:12:49,561 --> 00:12:54,638 And we could do, because we had like our own API for modifying the schema and such, 247 00:12:54,638 --> 00:12:56,778 you could do it in both at the same time. 248 00:12:57,558 --> 00:13:04,568 But that's also tricky because if you replicate data but the schema 249 00:13:04,568 --> 00:13:08,618 changes come out of order, then again you have a problem because 250 00:13:08,618 --> 00:13:12,908 it's not going to fit the schema, the classical CDC type of problems, right? 251 00:13:13,328 --> 00:13:16,458 So it's actually better if the schema changes are in the same 252 00:13:16,458 --> 00:13:18,968 replication stream as the actual data. 253 00:13:19,258 --> 00:13:20,328 It's better that way. 254 00:13:20,768 --> 00:13:24,708 So that's why we did this event hook approach, listening for schema 255 00:13:24,708 --> 00:13:27,258 changes and then apply them that. 256 00:13:27,738 --> 00:13:31,898 There, with Zetalight we had this advantage that we, you couldn't 257 00:13:31,898 --> 00:13:34,358 do arbitrary schemas in Postgres. 258 00:13:34,358 --> 00:13:38,798 It it was a particular set of data types and so on. 259 00:13:38,808 --> 00:13:42,778 It was a little bit more into our control, so we knew we could 260 00:13:42,788 --> 00:13:46,358 test all possible schema changes and make sure they replicate 261 00:13:46,358 --> 00:13:48,258 correctly and these sort of things. 262 00:13:48,878 --> 00:13:51,808 But then we generalized this approach a little bit more. 263 00:13:52,186 --> 00:13:52,436 Nik: Yeah. 264 00:13:52,436 --> 00:13:53,286 So that's interesting. 265 00:13:53,286 --> 00:13:56,026 So you have limited set of what's happening, right? 266 00:13:56,776 --> 00:14:00,176 And then using event triggers, you, Yeah. 267 00:14:00,286 --> 00:14:01,576 Tudor: Yeah, that's our approach. 268 00:14:01,576 --> 00:14:04,376 And of course, it needs-- we need to make sure it's in the right 269 00:14:04,556 --> 00:14:06,526 place in the replication stream. 270 00:14:07,506 --> 00:14:12,426 And the way we do it is we write in a kind of, in a shadow table. 271 00:14:12,436 --> 00:14:17,446 We write what has changed and which w- with what altered statement that 272 00:14:17,456 --> 00:14:22,606 happened, and that gets put into the logical replication stream. 273 00:14:23,066 --> 00:14:27,056 And then pgStream reads it from there and applies it on 274 00:14:27,056 --> 00:14:28,846 the target in the right order. 275 00:14:28,896 --> 00:14:30,276 That's the general approach. 276 00:14:30,398 --> 00:14:33,468 Nik: And I'm very curious at what scale it is working and are you 277 00:14:33,478 --> 00:14:37,458 fully satisfied or there are some like plans to improve or something? 278 00:14:37,654 --> 00:14:38,084 Tudor: Yeah. 279 00:14:38,124 --> 00:14:43,274 We have people using pgStream at like four or five terabytes of data. 280 00:14:43,714 --> 00:14:48,024 And the initial snapshot takes like 24 hours or something like this. 281 00:14:48,024 --> 00:14:51,864 And then And it b- that depends on a number of factors or something like 282 00:14:51,864 --> 00:14:53,894 this, and then it's kept up to date. 283 00:14:53,924 --> 00:14:58,054 But then it, like it, it's from company to company, and sometimes 284 00:14:58,054 --> 00:15:01,864 we have to work on pgStream a little bit , to make it work for them. 285 00:15:01,944 --> 00:15:06,584 But in the meantime, we had we it had enough battles, let's say that 286 00:15:06,624 --> 00:15:10,584 now we're feeling like, it covers already the majority of, uh, of, 287 00:15:10,634 --> 00:15:11,764 cases pretty well. 288 00:15:12,008 --> 00:15:12,148 Nik: Yeah. 289 00:15:12,288 --> 00:15:13,368 Battle proven already, right? 290 00:15:13,964 --> 00:15:14,964 Tudor: Yeah, to some degree. 291 00:15:15,014 --> 00:15:17,224 Like there, there are still problems. 292 00:15:17,234 --> 00:15:19,394 The CDC logical replication 293 00:15:19,444 --> 00:15:19,824 Nik: oh yes 294 00:15:19,886 --> 00:15:24,406 Tudor: is still quite a pla- I w- I would love if DDL would be supported in core 295 00:15:24,942 --> 00:15:28,632 Nik: everyone waits for it s- but it's super not trivial as I saw 296 00:15:28,662 --> 00:15:30,622 last time when I checked this topic. 297 00:15:31,252 --> 00:15:31,812 That's cool. 298 00:15:31,822 --> 00:15:35,752 I like s-small non-non-technical side note side question. 299 00:15:36,272 --> 00:15:39,002 These pieces are… Like you mentioned some proprietary stuff 300 00:15:39,002 --> 00:15:41,082 about p- pgroll and pgstream 301 00:15:41,152 --> 00:15:42,622 Tudor: Yeah, these are fully open source, 302 00:15:42,692 --> 00:15:44,952 Nik: yeah, that's and Apache 2.0, right? 303 00:15:45,134 --> 00:15:45,534 Tudor: Yes. 304 00:15:45,644 --> 00:15:45,984 yeah. 305 00:15:46,684 --> 00:15:50,164 Yeah, and especially pgStream, it's important for us to be open 306 00:15:50,164 --> 00:15:53,654 source because we recommend people to run it in their environment 307 00:15:53,964 --> 00:15:55,884 because it also does anonymization. 308 00:15:55,884 --> 00:15:58,354 This is the, you know, like kind of what we started from. 309 00:15:58,664 --> 00:16:03,314 It's important to do anonymization so that when you do this copy-on-write 310 00:16:03,314 --> 00:16:08,604 branching, the data is already anonymized and you have branches, very fast branches 311 00:16:09,664 --> 00:16:12,114 that don't contain the sensitive data. 312 00:16:12,234 --> 00:16:15,934 Like it's up to you what what transformation rules you set and so on. 313 00:16:15,944 --> 00:16:21,784 But in many cases, it's like 80% of the data is fine as it is, and you have 314 00:16:21,784 --> 00:16:26,534 some data that it's sensitive and you need to transform it and make sure it's 315 00:16:26,534 --> 00:16:28,704 not in any developer copy and so on. 316 00:16:28,714 --> 00:16:33,324 And this I think is the, is the perfect balance of things if you want. 317 00:16:33,614 --> 00:16:36,764 But it does depend on this replication story. 318 00:16:37,054 --> 00:16:41,144 And we want to run it in their environment so they have full control over it. 319 00:16:41,234 --> 00:16:43,124 That's why it needs to be open source really. 320 00:16:43,916 --> 00:16:44,226 Nik: Yeah. 321 00:16:44,546 --> 00:16:45,566 this answers my question 322 00:16:46,534 --> 00:16:49,834 Michael: on the anonymization front, I saw you acquired a company. 323 00:16:49,834 --> 00:16:51,264 I saw some blog posts I think you wrote. 324 00:16:51,264 --> 00:16:52,534 Was it beginning of this year? 325 00:16:52,594 --> 00:16:54,974 Is that what you've then ro- rolled into this 326 00:16:55,756 --> 00:16:57,486 Tudor: Yeah, partially partially. 327 00:16:57,566 --> 00:17:02,386 The it's called Privacy Dynamics, this company, and they were, like, really 328 00:17:02,386 --> 00:17:04,986 deep into the anonymization front. 329 00:17:05,406 --> 00:17:07,176 it's like there's levels to this stuff. 330 00:17:07,196 --> 00:17:12,256 In pgStream we already had the anonymization, which was 331 00:17:12,306 --> 00:17:15,246 simple column rule-based. 332 00:17:15,256 --> 00:17:19,586 Like you can say, okay, this column, you can hash it like this 333 00:17:19,866 --> 00:17:22,516 deterministically and, it's hashed. 334 00:17:23,136 --> 00:17:28,686 But then like for example, for HIPAA compliance and so on, you need to do 335 00:17:28,686 --> 00:17:35,116 the type of anonymization where if you, le-let's say there's patient's data and 336 00:17:35,116 --> 00:17:42,866 you have a woman that's 46 years old and so on, and has this email address 337 00:17:42,866 --> 00:17:46,966 or something like this, and if you just anonymize the email address, but it's 338 00:17:46,966 --> 00:17:52,126 the only woman in the data set with those characteristics then you haven't 339 00:17:52,146 --> 00:17:56,126 really anonymized anything because you can, like even if the name is not 340 00:17:56,126 --> 00:17:59,966 there, there's enough information, unique information there that you 341 00:17:59,966 --> 00:18:03,616 can still work backwards what it is. 342 00:18:03,966 --> 00:18:10,206 So they doing these sort of things where they see exactly combining the 343 00:18:10,206 --> 00:18:15,016 data set into groups and figuring out, okay, we need to anonymize this 344 00:18:15,016 --> 00:18:17,796 data so that no group is really small. 345 00:18:17,916 --> 00:18:22,566 Every group has more than 10 data points and such, so you 346 00:18:22,566 --> 00:18:24,316 cannot find the individual. 347 00:18:24,426 --> 00:18:28,526 And this is like more like, like HIPAA compliant sort of things and such. 348 00:18:28,946 --> 00:18:32,496 So that's not, that's also something that's not in pgStream at the moment. 349 00:18:32,506 --> 00:18:36,826 Maybe we will roll out more of those features in there that we set up 350 00:18:36,836 --> 00:18:41,836 more like manually for customers that actually need this complex stuff. 351 00:18:43,232 --> 00:18:44,042 Nik: Yeah, that's great. 352 00:18:44,642 --> 00:18:47,832 So these two pieces are, they're interesting themselves. 353 00:18:47,832 --> 00:18:50,422 They could they could be separate episodes, honestly. 354 00:18:50,902 --> 00:18:54,612 This is what I… Yeah, because if we dive deeper, there are so many levels. 355 00:18:54,612 --> 00:18:56,382 As I said I'm in that space as well. 356 00:18:56,842 --> 00:19:00,342 We usually use some other tools, like we didn't build. 357 00:19:00,342 --> 00:19:02,792 And we-- I remember also there was a company, Tonika. 358 00:19:02,792 --> 00:19:04,292 Yeah I'm curious what happened to it. 359 00:19:04,362 --> 00:19:05,232 Als-also 360 00:19:05,238 --> 00:19:06,868 Tudor: They still exist, I think, right? 361 00:19:07,122 --> 00:19:07,372 Nik: Yeah. 362 00:19:07,422 --> 00:19:10,312 So a lot of… if you look to what they build, you understand 363 00:19:10,312 --> 00:19:11,942 that so many layers go there. 364 00:19:12,042 --> 00:19:12,402 It's like 365 00:19:12,650 --> 00:19:12,880 Tudor: yeah. 366 00:19:12,880 --> 00:19:13,520 Absolutely. 367 00:19:13,570 --> 00:19:13,920 Absolutely. 368 00:19:14,110 --> 00:19:14,440 Yeah. 369 00:19:14,530 --> 00:19:18,700 And Tonic AI I think they still exist, but it's just proprietary and they 370 00:19:18,700 --> 00:19:23,040 open source like a simple version of it, but it doesn't scale at all. 371 00:19:23,080 --> 00:19:23,630 So it's, 372 00:19:23,660 --> 00:19:26,570 Nik: and important to also remember, like use cases are 373 00:19:26,570 --> 00:19:28,080 very different for branching. 374 00:19:28,080 --> 00:19:33,610 If you just want to… Like you said you don't recommend to take production and if 375 00:19:33,610 --> 00:19:37,070 it's like simple development it's better to have some like preview environments. 376 00:19:37,080 --> 00:19:39,890 It can be some synthetic data, for example, right? 377 00:19:39,900 --> 00:19:41,420 You don't need to synchronize it all. 378 00:19:41,760 --> 00:19:45,879 But If you bring data for a various like closer to production testing, 379 00:19:45,889 --> 00:19:50,429 and especially if it's re- related to performance testing, I'm-- I, I was 380 00:19:50,429 --> 00:19:55,359 always concerned about what plans we will be getting because if you change 381 00:19:55,359 --> 00:19:57,769 data, I stop trusting those plans. 382 00:19:58,333 --> 00:19:58,993 Tudor: Yeah, of course. 383 00:19:59,412 --> 00:20:05,302 Nik: So I usually try to convince people to keep like red data a- as is production 384 00:20:05,302 --> 00:20:09,322 data and just limit access and consider it as part of production and still have 385 00:20:09,342 --> 00:20:14,262 branching and test it closer to production because in… this is the only way you can 386 00:20:14,552 --> 00:20:17,732 have really realistic performance testing. 387 00:20:17,782 --> 00:20:18,862 What you think about this? 388 00:20:19,918 --> 00:20:20,248 Tudor: Yeah. 389 00:20:20,518 --> 00:20:25,958 So there, there is this deterministic hashing which basically means if you 390 00:20:26,808 --> 00:20:31,168 hash a value, it will always hash to the same thing, which is also 391 00:20:31,168 --> 00:20:33,258 useful for referential integrity. 392 00:20:33,258 --> 00:20:36,488 Let's say, like if you have a foreign key on a string column or something 393 00:20:36,488 --> 00:20:42,358 like this, then you want it to anonymize the same in both places. 394 00:20:42,828 --> 00:20:44,888 So that helps a little bit with this. 395 00:20:44,998 --> 00:20:48,978 But yeah, with the Postgres planner, there can be a lot of nuances around 396 00:20:49,096 --> 00:20:51,306 Nik: we, as I said, this could be whole episode. 397 00:20:51,506 --> 00:20:55,216 If we dive deeper, like when you say hashing, okay pgStream takes 398 00:20:55,216 --> 00:20:56,906 data from RDS, understand this. 399 00:20:57,516 --> 00:21:00,226 You have ability, like you copy anyway, right? 400 00:21:00,516 --> 00:21:05,256 But if you talk about your own platform, you have physical data directory, 401 00:21:05,926 --> 00:21:06,346 Tudor: You're right. 402 00:21:06,346 --> 00:21:06,746 Yes. 403 00:21:07,136 --> 00:21:07,486 Yeah 404 00:21:08,266 --> 00:21:12,936 Nik: W- when you touch it with any update, with anything, it's better, 405 00:21:12,986 --> 00:21:16,556 like you have, like we have think cloning branching on physical level. 406 00:21:16,996 --> 00:21:20,096 And if you start touching it, you already will Mi- Michael will tell 407 00:21:20,096 --> 00:21:23,596 better what, what will happen with plans because he builds pgMustard, right? 408 00:21:23,656 --> 00:21:25,716 Which is focusing only on, on the plans. 409 00:21:26,306 --> 00:21:30,236 So if you physically - change something, you like, different 410 00:21:30,246 --> 00:21:32,986 number of blocks and so on, it's already different plans, right? 411 00:21:33,536 --> 00:21:33,856 Tudor: Yeah 412 00:21:34,176 --> 00:21:37,086 Nik: Relpages can… reltuples numbers will become different. 413 00:21:37,086 --> 00:21:38,496 Statistics might start different. 414 00:21:38,586 --> 00:21:44,206 Even if you don't, if you say update some row set column equals the same 415 00:21:44,206 --> 00:21:48,296 column ID something, logically nothing changed, but physically it changed. 416 00:21:48,296 --> 00:21:52,216 And eventually if you massive scale, it can drift plans 417 00:21:53,090 --> 00:21:55,530 Tudor: Yeah, and logical replication already like kind 418 00:21:55,530 --> 00:21:56,970 of throws that away because 419 00:21:57,270 --> 00:21:57,410 Nik: Right. 420 00:21:57,620 --> 00:22:02,650 But if it's, if we go back to ZFS and thin cloning, the copy-on-write, 421 00:22:03,240 --> 00:22:06,800 I, I-- my point was always like, let's not touch the data and we will 422 00:22:06,810 --> 00:22:08,840 have ability to have this, the exact 423 00:22:08,938 --> 00:22:11,268 Tudor: and you can you can still do that. 424 00:22:11,278 --> 00:22:14,528 You can have… like if you run production on Xata what we typically 425 00:22:14,528 --> 00:22:20,208 recommend is have a prod project and then something like a staging project 426 00:22:20,208 --> 00:22:23,858 or something like this, and pgStream between them to do the anonymization. 427 00:22:24,608 --> 00:22:28,198 But then the prod project still supports copy-on-write branching, 428 00:22:28,248 --> 00:22:30,478 and that would be like ZFS 429 00:22:31,172 --> 00:22:32,222 Clone type of thing. 430 00:22:32,222 --> 00:22:36,562 And you can use that but you would only use this, use it when you 431 00:22:36,562 --> 00:22:39,202 actually need it for like performance, 432 00:22:39,334 --> 00:22:42,114 Nik: And like I, I see Michael wants to ask something. 433 00:22:42,114 --> 00:22:45,794 But I will ask very last question because I sit on this topic seven years already, 434 00:22:45,794 --> 00:22:51,084 so I like, I'm super curious to find guys who have like also spent time there. 435 00:22:51,504 --> 00:22:52,504 Last question from me. 436 00:22:52,504 --> 00:22:53,464 Unit economics. 437 00:22:53,494 --> 00:23:00,114 If we want to build like pipelines which will test performance plans, right? 438 00:23:00,594 --> 00:23:06,694 If we keep data in Zetta production and build these pipelines will 439 00:23:06,754 --> 00:23:10,504 each every CI pipeline lead to provisioning on the whole node? 440 00:23:10,894 --> 00:23:13,434 Will I be paying separately for whole… For if I run 441 00:23:13,434 --> 00:23:17,484 10 pipe- or 100 pipelines, you… I need to pay for every one, right? 442 00:23:18,474 --> 00:23:18,884 Or it's 443 00:23:18,954 --> 00:23:22,044 Tudor: Because we do the separation of storage and compute, right? 444 00:23:22,044 --> 00:23:26,084 And on the storage side, there's this thin copy, right? 445 00:23:26,084 --> 00:23:27,974 So you only pay for the difference. 446 00:23:28,014 --> 00:23:28,694 That's on the 447 00:23:28,752 --> 00:23:30,272 Nik: so it basically doesn't change. 448 00:23:30,282 --> 00:23:32,532 If I don't change data, it's the same price. 449 00:23:32,532 --> 00:23:32,572 I 450 00:23:32,696 --> 00:23:33,236 Tudor: Right. 451 00:23:33,856 --> 00:23:36,376 With, of course, some gotchas there as well, but at the 452 00:23:36,376 --> 00:23:37,586 high level, let's say that. 453 00:23:38,056 --> 00:23:43,446 And on the compute side they would we use Kubernetes so that would be-- they will 454 00:23:43,446 --> 00:23:49,316 be starting on different compute nodes potentially, but not each new branch 455 00:23:49,316 --> 00:23:51,776 will be a new node necessarily, right? 456 00:23:51,786 --> 00:23:53,756 They will fill existing nodes and such. 457 00:23:54,356 --> 00:23:58,726 And there you only pay if you use our Xata Cloud platform because 458 00:23:58,736 --> 00:24:01,336 you also have bring your own cloud, which is a little bit different. 459 00:24:01,616 --> 00:24:07,056 But if you use our cloud platform you only pay for the time the branch is active. 460 00:24:07,466 --> 00:24:12,156 And with the scale to zero that that, it's like for development it makes a 461 00:24:12,156 --> 00:24:16,796 lot of sense because it wakes up in a second when you connect to it, right? 462 00:24:16,796 --> 00:24:19,946 And you can set like a five minutes expiration time. 463 00:24:19,946 --> 00:24:22,736 This is like a, a lot of our customers do. 464 00:24:23,526 --> 00:24:25,166 It's like you pay almost nothing. 465 00:24:25,266 --> 00:24:26,736 I actually have a blog post. 466 00:24:26,816 --> 00:24:29,716 You can run thousand… It's just how the math works. 467 00:24:29,716 --> 00:24:35,866 You can run a thousand branches for one dollar basically a tenth of a cent 468 00:24:35,896 --> 00:24:38,496 each if they run for five minutes. 469 00:24:38,723 --> 00:24:39,573 Nik: yeah, that's great. 470 00:24:39,793 --> 00:24:43,213 So this is much better than what I observe with thing cloning 471 00:24:43,213 --> 00:24:45,603 at, in Aurora and Neon, right? 472 00:24:45,643 --> 00:24:48,573 Because there is absolutely it, it's linear scaling. 473 00:24:48,573 --> 00:24:53,253 If you wa- if you want many pipelines in CI to test, you 474 00:24:53,283 --> 00:24:54,983 amplify your spending budgets. 475 00:24:55,063 --> 00:24:59,313 And this limits, This is super important because this makes engineers to 476 00:24:59,313 --> 00:25:01,203 stop testing enough, and they start 477 00:25:01,483 --> 00:25:01,833 Tudor: Yeah. 478 00:25:01,833 --> 00:25:02,213 Yeah. 479 00:25:02,293 --> 00:25:05,663 And it discouraged to have a branch for each PR, which is like 480 00:25:05,853 --> 00:25:07,333 Nik: so that's the kudos for that. 481 00:25:07,333 --> 00:25:07,973 This is cool. 482 00:25:08,003 --> 00:25:11,123 I, as I understand, it's not like a constant price, but definitely you've 483 00:25:11,123 --> 00:25:14,563 thought about this and because this is super important to make testing 484 00:25:14,573 --> 00:25:17,433 more like intensive and complete. 485 00:25:17,655 --> 00:25:17,765 Tudor: Yeah. 486 00:25:18,675 --> 00:25:18,975 Yeah. 487 00:25:19,295 --> 00:25:19,635 Yeah. 488 00:25:20,045 --> 00:25:20,295 Yeah. 489 00:25:20,295 --> 00:25:24,755 So definitely Neon ch-- Neon, like you get 20 branches for free. 490 00:25:24,805 --> 00:25:26,135 You still pay the compute. 491 00:25:26,175 --> 00:25:27,275 They're not completely free. 492 00:25:27,275 --> 00:25:34,205 You would pay for the compute but above 20 branches you pay a fixed amount for 493 00:25:34,205 --> 00:25:36,355 each of them, even if they are inactive. 494 00:25:36,755 --> 00:25:37,765 So we don't do that. 495 00:25:37,825 --> 00:25:38,825 We don't charge that. 496 00:25:38,845 --> 00:25:43,285 That's why you can have a thousand branches with only $1, which I don't 497 00:25:43,285 --> 00:25:47,015 know, maybe in a year we'll have another call and I will say that 498 00:25:47,015 --> 00:25:48,565 does- that doesn't work economically. 499 00:25:51,545 --> 00:25:52,055 Nik: I you start to add nuances. 500 00:25:52,055 --> 00:25:53,745 I know which exactly which nuance. 501 00:25:53,745 --> 00:25:58,895 If you have some branch or clone or like it holds snapshot, and 502 00:25:58,895 --> 00:26:02,525 if production runs away into the future, you keep… You start paying 503 00:26:02,525 --> 00:26:04,245 for all data to be stored, right? 504 00:26:04,371 --> 00:26:04,961 Tudor: Yeah, Exactly. 505 00:26:05,195 --> 00:26:05,815 Nik: That it is. 506 00:26:05,815 --> 00:26:07,405 I know this problem very well 507 00:26:08,001 --> 00:26:12,461 Tudor: Yeah like sometimes hard to explain because you create a branch, 508 00:26:13,081 --> 00:26:17,031 its cost is zero on the storage, and then you delete all the data into it, 509 00:26:17,821 --> 00:26:22,751 and then the-- you would expect-- you wouldn't expect the cost to go up, but 510 00:26:22,751 --> 00:26:26,561 that's actually what happens because now it's different from the parent, right? 511 00:26:27,081 --> 00:26:31,011 So it's like, "Hey, why am I paying more by deleting data," right? 512 00:26:31,041 --> 00:26:32,391 That, that can be confusing 513 00:26:32,853 --> 00:26:36,513 Nik: this is exactly what we also ha-had and still having. 514 00:26:36,533 --> 00:26:37,563 Yeah, that's cool. 515 00:26:38,453 --> 00:26:38,713 Cool. 516 00:26:38,773 --> 00:26:38,993 Yeah. 517 00:26:39,543 --> 00:26:42,503 Yeah, Michael, I, I apologize you had questions. 518 00:26:42,563 --> 00:26:44,833 I, I'm just super excited to talk about 519 00:26:44,879 --> 00:26:47,779 Michael: I was gonna ask more, but actually now I'm thinking, 520 00:26:47,779 --> 00:26:52,939 I think this is great in terms of using Xata for your developer, 521 00:26:53,418 --> 00:26:57,318 environments, let's say like pre-production those early maybe per 522 00:26:57,348 --> 00:26:59,118 agent, per branch, that kind of thing. 523 00:26:59,618 --> 00:27:04,258 But I think you've also got some interesting stuff on the developer 524 00:27:04,268 --> 00:27:08,928 experience of running Postgres in production, like pgRole for example. 525 00:27:09,198 --> 00:27:12,418 But I suspect you'll have opinions on a lot of the other things that 526 00:27:12,748 --> 00:27:16,708 are currently quite hard to do as a developer running Postgres in production. 527 00:27:16,708 --> 00:27:19,568 So I'd, I wondered about getting your opinion on some of those. 528 00:27:19,578 --> 00:27:23,398 So the big one for me, I think still that's difficult for people 529 00:27:23,398 --> 00:27:25,428 to manage is major version upgrades. 530 00:27:25,858 --> 00:27:29,618 Like that, like things like that, I wonder if you have any takes or any, 531 00:27:29,648 --> 00:27:34,098 any interesting features that most managed services don't offer or things 532 00:27:34,108 --> 00:27:36,458 that you want to do in the future to make things like that easier? 533 00:27:38,038 --> 00:27:42,448 Tudor: Yeah, we did put a lot of effort into the schema changes part of it. 534 00:27:42,478 --> 00:27:48,358 It felt to us like one of the biggest pain points in, in Postgres. 535 00:27:48,358 --> 00:27:54,948 There's a lot of gotchas there as well, like you do a schema change and it works 536 00:27:54,948 --> 00:27:59,038 on your small database that you test with, but then you run it in production and it 537 00:27:59,038 --> 00:28:01,138 locks the whole table for a long time. 538 00:28:01,138 --> 00:28:05,638 So then you have to do this process and orchestrate multiple 539 00:28:05,678 --> 00:28:09,798 PRs, change the application, do that, do the backfill, and so on. 540 00:28:10,548 --> 00:28:13,608 So that w- that's how, like how we started pgroll. 541 00:28:13,748 --> 00:28:18,574 It's also related to the old platform, which again, it was all about making 542 00:28:18,594 --> 00:28:21,744 the database as easy as possible to use. 543 00:28:22,009 --> 00:28:25,044 so then we had an API to do schema changes. 544 00:28:25,214 --> 00:28:29,354 And that was not as, you couldn't do as much as as Postgres can do. 545 00:28:29,534 --> 00:28:34,154 But everything that you did there was guaranteed to be safe. 546 00:28:34,204 --> 00:28:36,584 You could never lock your database and such. 547 00:28:37,624 --> 00:28:41,334 But then again, you hit the problem that people want to do those things. 548 00:28:41,334 --> 00:28:44,344 So it's like that Postgres functionality was not added for nothing. 549 00:28:44,344 --> 00:28:47,134 And then you end up having to support everything, which was, 550 00:28:47,294 --> 00:28:48,754 whi- which was our problem. 551 00:28:49,264 --> 00:28:55,274 So yeah, pgroll is is like it's an attempt to make that as as simple as possible. 552 00:28:55,674 --> 00:28:56,514 It's working. 553 00:28:56,534 --> 00:29:01,234 It also has some, some gotchas that currently we're not super focused on 554 00:29:01,234 --> 00:29:05,384 it because we're focused primarily on the Xata platform itself with copy 555 00:29:05,384 --> 00:29:09,254 and write branching and supporting many agents and these sort of things. 556 00:29:09,254 --> 00:29:10,434 That's where our focus is. 557 00:29:10,444 --> 00:29:13,244 So we didn't have that much time for pgroll lately. 558 00:29:13,964 --> 00:29:15,464 Yeah, that's wh- that's where we are. 559 00:29:15,970 --> 00:29:21,760 Nik: So now focus is to support to give database to agents, right? 560 00:29:21,900 --> 00:29:22,530 To-- This is 561 00:29:22,560 --> 00:29:22,980 Tudor: Right. 562 00:29:23,200 --> 00:29:24,690 And that can be in two ways. 563 00:29:24,690 --> 00:29:28,340 One is during development, like you give like I was saying 564 00:29:28,490 --> 00:29:29,530 earlier, branches per PR. 565 00:29:30,700 --> 00:29:34,660 Now with agents, you create a lot more PRs, and we're seeing, like we're seeing 566 00:29:34,660 --> 00:29:39,490 companies that have two engineers, and they have 250 open PRs, and they're 567 00:29:39,490 --> 00:29:41,560 actually all making progress and such. 568 00:29:42,110 --> 00:29:44,610 We see people that that don't do reviews. 569 00:29:44,620 --> 00:29:48,020 It's only AI reviews, and they get merged automatically. 570 00:29:48,280 --> 00:29:49,690 The tests are running. 571 00:29:49,690 --> 00:29:53,850 A lot of these things are happening now and will probably happen more, right? 572 00:29:54,130 --> 00:29:55,440 So that's one. 573 00:29:55,850 --> 00:30:00,960 And the second way is like AI platforms that need a Postgres database 574 00:30:00,960 --> 00:30:03,220 as part of their product right? 575 00:30:03,260 --> 00:30:09,210 And then they want to spin up like an FML database for that application, which could 576 00:30:09,220 --> 00:30:11,630 be an experiment that they throw away. 577 00:30:11,640 --> 00:30:15,780 So then you want these cheap databases that spin up and down 578 00:30:16,510 --> 00:30:18,900 very dynamically and scale a lot, 579 00:30:19,340 --> 00:30:20,210 scaling to 580 00:30:20,492 --> 00:30:21,972 Nik: Yeah. 581 00:30:21,972 --> 00:30:24,492 Yeah, these guys even don't need HA at all, right? 582 00:30:25,282 --> 00:30:27,672 It's just single node no failover 583 00:30:28,094 --> 00:30:28,894 Tudor: Yeah, exactly. 584 00:30:28,944 --> 00:30:29,554 Exactly. 585 00:30:29,614 --> 00:30:33,324 U-usually you don't need replicas for those use cases, yes 586 00:30:33,956 --> 00:30:35,856 Nik: Yeah, so no high availability needed. 587 00:30:35,936 --> 00:30:37,736 Maybe just backups and that's it, right? 588 00:30:37,796 --> 00:30:40,016 Like you also feel this like new demand 589 00:30:40,076 --> 00:30:40,656 Tudor: yeah. 590 00:30:40,726 --> 00:30:44,116 Yeah, and we do take advantage of that because the way we solve 591 00:30:44,146 --> 00:30:49,196 these very fast wake-up times is by doing what we call warm pools. 592 00:30:49,246 --> 00:30:52,086 We keep CNPG clusters ready to go. 593 00:30:52,986 --> 00:30:59,606 And then when there's a connection for a hibernated branch or database we take 594 00:30:59,606 --> 00:31:06,106 one already provision clustered from from the pool, and we connect it to the 595 00:31:06,106 --> 00:31:11,453 right ZFS volume on the ZetaStore storage node, and then serve it to the user. 596 00:31:11,473 --> 00:31:13,873 And that's how, that's how it happens so fast. 597 00:31:14,697 --> 00:31:20,367 But this means the clusters in the pool have to all look the same because 598 00:31:20,367 --> 00:31:23,637 if they don't look the same and and they have, they have different 599 00:31:23,647 --> 00:31:27,247 RAM and so on, it's obvious, but also different Postgres settings. 600 00:31:27,537 --> 00:31:30,667 If that causes a restart, then you did nothing because you 601 00:31:30,677 --> 00:31:32,347 need to restart it then too. 602 00:31:32,407 --> 00:31:33,927 So we take advantage of that. 603 00:31:34,047 --> 00:31:37,487 The cluster pools have no, , no read replicas. 604 00:31:37,557 --> 00:31:41,417 It's-- They're very fast, but it is what it is, 605 00:31:41,487 --> 00:31:42,407 Nik: how fast it is? 606 00:31:43,137 --> 00:31:45,087 Tudor: So like we're targeting a second. 607 00:31:46,025 --> 00:31:49,945 pro-- Yeah, provisioning a, a new cluster is the simplest operation 608 00:31:49,975 --> 00:31:52,425 because we just take one of the, from the pool and give it. 609 00:31:52,455 --> 00:31:54,275 It's it's, yeah, also second. 610 00:31:54,505 --> 00:31:58,165 But it's that most of the cost there is really the control plane. 611 00:31:58,175 --> 00:32:01,965 We have a single global control plane region, so we have to go to it and so on. 612 00:32:02,485 --> 00:32:03,335 But that's fast. 613 00:32:03,615 --> 00:32:08,645 The more-- The wake-up is actually more interesting because there you cannot 614 00:32:08,645 --> 00:32:10,955 just take one from the pool and serve it. 615 00:32:11,445 --> 00:32:14,625 You have to take one of the, from the pool, connect it to the right 616 00:32:14,625 --> 00:32:16,535 volume, and then serve it, right? 617 00:32:16,555 --> 00:32:22,045 And this is what the, this is what the Xata store enables be- and this 618 00:32:22,065 --> 00:32:26,451 separation of storage and compute to some degree And have a bit of a trick 619 00:32:26,481 --> 00:32:31,581 that Cloud Native PG doesn't support any of this, so we are maintaining kind of 620 00:32:31,661 --> 00:32:34,381 our fork of it to enable some of that. 621 00:32:34,401 --> 00:32:38,791 But there's the instance manager in in Cloud Native PG, and we modified it to 622 00:32:38,791 --> 00:32:45,261 essentially wait for the PGDATA folder to show up, and until then it's waiting. 623 00:32:45,261 --> 00:32:47,831 So the pod is completely provisioned. 624 00:32:47,831 --> 00:32:52,171 It's just waiting for that folder to show up before letting Postgres start. 625 00:32:52,911 --> 00:32:56,761 And behind the scenes, we mount it in the right place, and as soon as 626 00:32:56,761 --> 00:33:01,811 that shows up there, the instance manager will let Postgres start. 627 00:33:02,291 --> 00:33:06,641 And Postgres starts in 300 milliseconds or so. 628 00:33:06,641 --> 00:33:10,361 If the, it depends as well if the, it, if the checkpoint was 629 00:33:10,361 --> 00:33:11,791 done and then things like this. 630 00:33:12,341 --> 00:33:15,911 But if it doesn't have to do the recovery, it's actually really fast. 631 00:33:15,971 --> 00:33:19,811 But what's expensive is provisioning the whole stuff in Kubernetes 632 00:33:20,091 --> 00:33:24,791 around it, the pod and the init containers and these sort of things. 633 00:33:24,855 --> 00:33:26,285 that's where the time was going. 634 00:33:26,695 --> 00:33:29,795 So that's why we do all that thing, and then the connection 635 00:33:30,095 --> 00:33:33,315 the, the work that we do at the connection time is actually minimal. 636 00:33:33,465 --> 00:33:34,465 That's why it's so fast. 637 00:33:34,541 --> 00:33:35,261 Nik: That's impressive. 638 00:33:35,331 --> 00:33:35,891 That's cool. 639 00:33:36,501 --> 00:33:37,021 I have a 640 00:33:37,061 --> 00:33:37,651 tricky question. 641 00:33:38,131 --> 00:33:38,231 Oh. 642 00:33:38,241 --> 00:33:39,261 okay, Michael first 643 00:33:39,539 --> 00:33:41,549 Michael: I maybe have the same tricky question. 644 00:33:41,959 --> 00:33:47,659 I wondered about your-- i-in a development environment, you probably 645 00:33:47,659 --> 00:33:52,089 want the same extensions as production, or you might you might need them 646 00:33:52,089 --> 00:33:53,589 in order to test various things. 647 00:33:54,009 --> 00:33:57,579 And also, you might have changed a lot of configuration parameters. 648 00:33:57,579 --> 00:33:58,769 I'm wondering how… 649 00:33:59,549 --> 00:34:05,579 maybe it's not quite relevant to the same pooling system, but does that, that system 650 00:34:05,579 --> 00:34:12,149 you just described limit your use of extensions and changing of configuration 651 00:34:12,955 --> 00:34:14,425 Nik: I just want superuser. 652 00:34:14,425 --> 00:34:14,735 That's 653 00:34:14,825 --> 00:34:15,295 Tudor: Oh, okay. 654 00:34:16,775 --> 00:34:17,655 two different questions. 655 00:34:17,655 --> 00:34:19,305 Let me start with the one from Michael. 656 00:34:19,726 --> 00:34:20,706 Yeah it does. 657 00:34:20,726 --> 00:34:24,826 Because the, the problem is you cannot keep a cluster pool. 658 00:34:24,826 --> 00:34:25,796 This is how we call them. 659 00:34:25,826 --> 00:34:30,426 You cannot keep a cluster pool for each combination of, like you have instance 660 00:34:30,436 --> 00:34:36,376 size, and then you s- have a Postgres setting or list of - extensions and so on. 661 00:34:37,546 --> 00:34:39,766 Preload libraries is more relevant. 662 00:34:39,776 --> 00:34:42,716 Because the extensions, they can be on the image, you just 663 00:34:42,716 --> 00:34:44,796 don't enable them or enable them. 664 00:34:44,796 --> 00:34:45,336 That's fine. 665 00:34:45,336 --> 00:34:46,316 That's not a problem. 666 00:34:46,316 --> 00:34:51,436 But if you need different set of preload libraries, that is a problem. 667 00:34:51,736 --> 00:34:56,316 So we just chose to enable more more common stuff in the 668 00:34:56,326 --> 00:34:58,286 preload libraries by default. 669 00:34:58,996 --> 00:35:02,086 And then you have to, like you have to keep in mind that if you are not 670 00:35:02,116 --> 00:35:07,766 on the happy path and we don't have a cluster pool for you, it still works. 671 00:35:07,766 --> 00:35:08,666 It's just slower. 672 00:35:08,906 --> 00:35:13,406 It will take a few seconds then s- six to 10 seconds instead of one 673 00:35:13,436 --> 00:35:14,836 if you're not on the happy path. 674 00:35:15,346 --> 00:35:18,096 And then the other thing is that we're-- we, we haven't done 675 00:35:18,096 --> 00:35:23,656 yet, but we want to do, which is auto-scaling for the cluster pools. 676 00:35:24,416 --> 00:35:29,366 Because some some configurations, if you want, are going to be a lot 677 00:35:29,366 --> 00:35:31,406 more common than others, right? 678 00:35:31,406 --> 00:35:36,076 The default settings that we have, you do a, a cluster pool 679 00:35:36,076 --> 00:35:39,246 of 100 waiting to go, right? 680 00:35:39,506 --> 00:35:44,316 So that you can do 100 at once and you don't-- y- it's not overloaded. 681 00:35:44,886 --> 00:35:48,926 If you have a more complex one, it could be just one is enough, right? 682 00:35:48,966 --> 00:35:53,686 Because you start that one, and then you immediately provision 683 00:35:53,686 --> 00:35:55,106 the replacement for it. 684 00:35:55,146 --> 00:36:00,136 That will take 10 seconds, . As long as someone else doesn't do the same operation 685 00:36:00,136 --> 00:36:01,776 in those 10 seconds, you're okay. 686 00:36:01,776 --> 00:36:03,676 So it's enough to keep one around. 687 00:36:04,316 --> 00:36:08,666 But making this -- via static configuration is definitely annoying. 688 00:36:08,666 --> 00:36:12,486 So that's why we want to do some sort of auto-scaling based on that. 689 00:36:12,536 --> 00:36:18,266 So depending on the demand that we've seen historically, we would make these 690 00:36:18,386 --> 00:36:23,756 pools smaller or bigger, and that would automatically optimize costs for us. 691 00:36:23,946 --> 00:36:27,976 That, that's something that we haven't done yet but want to look into it. 692 00:36:28,966 --> 00:36:34,746 Yeah, superuser is like I happen to know your opinion on that. 693 00:36:34,776 --> 00:36:37,276 We don't give superuser to to our user. 694 00:36:37,276 --> 00:36:38,416 We give the, give up s- 695 00:36:39,028 --> 00:36:39,158 Nik: But will you think about it at least? 696 00:36:40,336 --> 00:36:42,516 Tudor: Yeah, the, the issue is because we run on Kubernetes 697 00:36:44,632 --> 00:36:47,132 it's pretty hard to secure it at 698 00:36:47,132 --> 00:36:47,462 that level. 699 00:36:47,472 --> 00:36:49,692 If you can run copy program on 700 00:36:49,774 --> 00:36:51,464 Nik: Let's wrap up this question. 701 00:36:51,464 --> 00:36:53,984 I have much more interesting and we don't have a lot of time. 702 00:36:54,024 --> 00:36:54,414 So 703 00:36:54,616 --> 00:36:54,626 Tudor: Mm-hmm. 704 00:36:54,834 --> 00:36:56,004 Nik: question to you, tricky one. 705 00:36:56,614 --> 00:36:59,984 If you started this platform to build this once again right now from scratch, 706 00:37:00,034 --> 00:37:01,184 would you choose Kubernetes still? 707 00:37:03,106 --> 00:37:06,626 Tudor: We do use a lot of stuff from Kubernetes and it's good. 708 00:37:06,706 --> 00:37:08,356 Yeah, I'm happy with-- Overall, 709 00:37:08,356 --> 00:37:08,526 I'm 710 00:37:08,728 --> 00:37:09,208 Nik: most important. 711 00:37:09,268 --> 00:37:09,868 Okay, cool. 712 00:37:10,178 --> 00:37:10,388 Are you 713 00:37:10,466 --> 00:37:10,716 Tudor: I'm happy 714 00:37:10,848 --> 00:37:11,818 Nik: cloud native PG? 715 00:37:13,492 --> 00:37:18,052 Tudor: I'm-- Yeah, you know, it's like that's more on the fence, I would say 716 00:37:18,612 --> 00:37:19,202 for me. 717 00:37:19,324 --> 00:37:20,084 Nik: opinion about it 718 00:37:21,052 --> 00:37:22,492 Tudor: I know your opinion about it. 719 00:37:22,872 --> 00:37:24,522 Nik: Yeah and more is coming actually. 720 00:37:24,862 --> 00:37:28,282 Literally this weekend I didn't work only on this PG SimCity 721 00:37:28,814 --> 00:37:29,934 Tudor: Oh, that's awesome by the way. 722 00:37:30,332 --> 00:37:30,972 Nik: Yeah, thank you. 723 00:37:31,552 --> 00:37:33,612 It's, it was a simple one prompt. 724 00:37:33,612 --> 00:37:34,539 It was unexpected. 725 00:37:34,605 --> 00:37:41,729 but this thing like what's happening, we have a le- a, a winning leader among 726 00:37:42,079 --> 00:37:48,099 Kubernetes operators, which is completely like ignoring the facts like about 727 00:37:48,099 --> 00:37:50,359 split brains and data loss and so on. 728 00:37:51,319 --> 00:37:54,259 Do you use synchronous replication if you have multi-node setup? 729 00:37:55,171 --> 00:37:55,191 Tudor: It's configurable. 730 00:37:55,331 --> 00:37:55,481 It's 731 00:37:56,929 --> 00:37:59,559 Nik: It's configurable, but it's not on def- by default, 732 00:38:00,301 --> 00:38:00,571 Right? 733 00:38:00,571 --> 00:38:02,391 It's not d- it's not on by default. 734 00:38:03,051 --> 00:38:06,781 And it w- which means I have demonstration of split brain is still possible 735 00:38:06,781 --> 00:38:08,871 in multiple f- in multiple ways. 736 00:38:09,621 --> 00:38:12,151 Michael: Tudor, do you mean you let your users configure it? 737 00:38:12,953 --> 00:38:13,393 Tudor: yeah. 738 00:38:13,689 --> 00:38:14,769 We can enable it for them 739 00:38:15,041 --> 00:38:18,221 Nik: And in Cloud Native it's not enabled by default, it's configurable. 740 00:38:18,221 --> 00:38:18,741 It's the same. 741 00:38:18,741 --> 00:38:19,741 Yeah, that's the problem. 742 00:38:19,741 --> 00:38:25,531 And so haven't you already… Like maybe I'm over like maybe we don't need proper 743 00:38:25,531 --> 00:38:28,031 HA, split brain protection and so on. 744 00:38:28,071 --> 00:38:32,221 Like maybe this is like it's okay to run this thing and if your users, 745 00:38:32,281 --> 00:38:35,571 if you don't know about your users suffering may- at your scale, that 746 00:38:36,101 --> 00:38:37,491 this is good enough already, right? 747 00:38:37,491 --> 00:38:38,231 Like it's okay. 748 00:38:38,541 --> 00:38:39,511 What's your take on this? 749 00:38:40,299 --> 00:38:44,029 Tudor: No, I think it's definitely a problem that like I'm, yeah I'm 750 00:38:44,029 --> 00:38:47,379 also a little bit disappointed by the response to that issue. 751 00:38:47,429 --> 00:38:52,339 I was expecting, like to be closed sooner. 752 00:38:52,419 --> 00:38:57,989 I understand it's tricky because in CNPG there's no there's no proxy layer. 753 00:38:58,229 --> 00:39:03,249 So they rely on Kubernetes services to expose things and to move them around. 754 00:39:04,039 --> 00:39:07,729 That makes it, I think, a little bit more complex for them 755 00:39:07,739 --> 00:39:10,269 compared to Patroni, for example. 756 00:39:11,739 --> 00:39:14,029 The funny thing is we do have a gateway 757 00:39:14,067 --> 00:39:16,287 Nik: it's not complex, it's impossible. 758 00:39:17,237 --> 00:39:18,037 It's impossible. 759 00:39:18,037 --> 00:39:20,327 You can… There is no consensus algorithm used. 760 00:39:20,627 --> 00:39:24,447 There is only quorum, and there is mixing of concepts like quorum 761 00:39:24,447 --> 00:39:26,147 is needed to handle transactions. 762 00:39:26,257 --> 00:39:29,397 It's not n- it's not for deciding who is the primary. 763 00:39:30,137 --> 00:39:31,837 So I have demonstrations. 764 00:39:31,857 --> 00:39:33,947 You can have multiple-- You have two primaries. 765 00:39:34,207 --> 00:39:38,347 You can have split brain, you can have data loss, and more to be published soon. 766 00:39:38,654 --> 00:39:42,804 I'm pissed off because m- because of their popularity clients keep asking us. 767 00:39:43,184 --> 00:39:46,484 I literally have right now two asks from two different companies to 768 00:39:46,484 --> 00:39:49,434 explain, should we use CloudNativePG because it's super popular. 769 00:39:50,624 --> 00:39:53,524 And now, like I meet you and you use it at scale. 770 00:39:54,114 --> 00:39:57,894 So you don't have problems with no-nobody lost data, no-nobody 771 00:39:57,904 --> 00:40:00,004 dealt with split brains. 772 00:40:01,024 --> 00:40:07,194 Do we have… M-maybe we have already good cloud resources which don't experience 773 00:40:07,194 --> 00:40:09,294 like outages, like maybe th-this is it. 774 00:40:09,734 --> 00:40:12,894 Maybe we don't need the HA replicas may-maybe even, right? 775 00:40:12,904 --> 00:40:15,664 Like it's already good enough to leave a single node. 776 00:40:16,504 --> 00:40:20,364 So there is no such a problem as HA, high availability, no more such problems. 777 00:40:20,794 --> 00:40:24,894 Or you had such issues like split brains or data losses because 778 00:40:25,284 --> 00:40:26,244 there's no consensus algorithm 779 00:40:27,924 --> 00:40:34,834 Tudor: Yeah, it's like I, like I would have to go back and look how often. 780 00:40:34,884 --> 00:40:40,364 Like I do remember maybe one case where to be honest, we couldn't 781 00:40:40,374 --> 00:40:42,504 figure out exactly what happened. 782 00:40:42,808 --> 00:40:45,728 in these things there's always a combination of factors. 783 00:40:46,208 --> 00:40:49,898 But I do remember once we thought, like this might actually be the 784 00:40:49,898 --> 00:40:52,528 split brain scenario hitting us. 785 00:40:52,538 --> 00:40:55,696 So like I, I think it's a bug really. 786 00:40:56,046 --> 00:40:59,806 Nik: Unfortunately, in their DNA, they like, it's like this is a basic decision. 787 00:40:59,806 --> 00:41:01,166 If you read recent post. 788 00:41:01,226 --> 00:41:02,776 I have an answer unpublished. 789 00:41:02,866 --> 00:41:04,686 I'm going to publish it, so more stuff is 790 00:41:04,998 --> 00:41:05,708 Tudor: It's coming 791 00:41:05,796 --> 00:41:07,416 Nik: And I sent you a LinkedIn request. 792 00:41:07,416 --> 00:41:08,146 Please accept it. 793 00:41:08,146 --> 00:41:10,806 Let's talk, let's keep in touch on this topic because I want the 794 00:41:10,806 --> 00:41:14,786 industry understand that consensus algorithm is needed, and either 795 00:41:14,856 --> 00:41:19,356 CloudNativePG needs to be, to change or people should stop using it. 796 00:41:20,286 --> 00:41:22,956 Or maybe we like, again, like one node is enough. 797 00:41:23,126 --> 00:41:25,566 We don't need the HA, everything is good. 798 00:41:25,766 --> 00:41:31,316 AWS and Google Cloud they stopped being unreliable as it was 10 to 15 years ago. 799 00:41:31,588 --> 00:41:35,958 Tudor: Yeah, on, on our side, we're already I told you we are on a fork of it. 800 00:41:36,088 --> 00:41:39,488 It's not that I would, I would not recommend people to use our fork 801 00:41:39,508 --> 00:41:43,598 because we would, we, we-- It's open source as well, but we didn't make 802 00:41:43,598 --> 00:41:48,818 it with the goal of becoming, like a CNPG successor or anything like that. 803 00:41:49,508 --> 00:41:55,098 But we did we did go a bit in a different direction when it comes to backups. 804 00:41:55,268 --> 00:42:01,028 For example, we're using pgBackRest, and we made it in the operator itself 805 00:42:01,028 --> 00:42:04,988 rather than the plug-in solution, which has a number of disadvantages. 806 00:42:05,648 --> 00:42:10,888 So we have the, like we have the precedent of us going a little bit in different 807 00:42:10,888 --> 00:42:13,198 directions regarding these things. 808 00:42:14,208 --> 00:42:16,088 Also, we do have a gateway anyway. 809 00:42:16,088 --> 00:42:18,528 We need a gateway, like a SQL gateway. 810 00:42:18,528 --> 00:42:24,008 So then, like the main benefit that you are proxy-less actually 811 00:42:24,008 --> 00:42:25,578 doesn't really apply to us. 812 00:42:26,298 --> 00:42:29,448 So it might, that might be also something that we would choose to 813 00:42:29,448 --> 00:42:30,938 do a little bit different there. 814 00:42:31,246 --> 00:42:31,446 Nik: Yeah. 815 00:42:31,446 --> 00:42:35,146 And by default CloudNative PG has a Barman or, for backups or what? 816 00:42:35,198 --> 00:42:38,558 Tudor: I- it's a plugin that's-- It used to be, Barman in the core, but 817 00:42:38,578 --> 00:42:40,948 then now it's Barman as a plugin. 818 00:42:41,728 --> 00:42:46,078 I think as of right now it's the only, though, plugin that exists. 819 00:42:46,078 --> 00:42:47,858 I think there's one for pgBackRest, 820 00:42:47,908 --> 00:42:48,458 but in… 821 00:42:49,083 --> 00:42:52,273 Nik: So if you replace and if you pl- if you have pgBackRest, this 822 00:42:52,273 --> 00:42:53,953 next step is just to add Patroni 823 00:42:55,279 --> 00:42:56,939 Tudor: Yeah, you're right 824 00:42:57,445 --> 00:42:59,495 Nik: And then my concerns are fully resolved. 825 00:42:59,495 --> 00:43:01,155 I'm okay. 826 00:43:01,155 --> 00:43:02,775 Sensor algorithm works well and so on. 827 00:43:03,045 --> 00:43:06,145 But I understand it can be super challenging at your scale because if 828 00:43:06,145 --> 00:43:11,235 you have a lot of smaller clusters and so on ETCD or something, like you 829 00:43:11,235 --> 00:43:13,355 sh- you need to properly engineer it. 830 00:43:13,355 --> 00:43:17,945 I think it's done by some guys, but it's not straightforward to 831 00:43:18,005 --> 00:43:19,255 not-- like, because we don't… 832 00:43:19,475 --> 00:43:24,395 If we have a lot of small clusters, we cannot afford ETCD for everyone, right? 833 00:43:24,435 --> 00:43:26,815 We need to have shared ETCD, and there are some 834 00:43:26,815 --> 00:43:27,265 limitations 835 00:43:28,085 --> 00:43:28,275 remember. 836 00:43:28,901 --> 00:43:29,471 Tudor: exactly. 837 00:43:29,545 --> 00:43:31,385 Nik: But it's like it would be natural step. 838 00:43:31,695 --> 00:43:36,995 Like next time, like you bump into a couple of other weird cases and maybe you 839 00:43:36,995 --> 00:43:40,255 should add Patroni, something like this. 840 00:43:40,317 --> 00:43:40,767 Tudor: Yeah. 841 00:43:40,887 --> 00:43:41,217 Yeah. 842 00:43:41,627 --> 00:43:46,267 but that's, like there's the advantage that we don't really do this yet, but it 843 00:43:46,267 --> 00:43:51,247 does make sense to have a bit of different configuration for your development 844 00:43:51,257 --> 00:43:53,827 branches versus your production branches. 845 00:43:54,487 --> 00:44:02,067 Because the the thousands of branches and so on, the really large numbers don't 846 00:44:02,077 --> 00:44:06,087 even have replicas like you're saying, because that like the point is that 847 00:44:06,117 --> 00:44:08,297 you shut them down, they come back up. 848 00:44:09,057 --> 00:44:12,437 The storage is separated and it's secure no matter what. 849 00:44:12,437 --> 00:44:14,337 It doesn't depend on Postgres at all. 850 00:44:14,857 --> 00:44:19,067 So then, you just don't use read replicas at all for those use cases. 851 00:44:19,257 --> 00:44:23,907 And the production ones are just much fewer and then you can 852 00:44:23,957 --> 00:44:25,467 Nik: And computer's smaller, right? 853 00:44:25,467 --> 00:44:28,957 So like less bu- shared buffers is small, everything like this right? 854 00:44:29,757 --> 00:44:34,707 But then you can fool the planner and make it work as to choose 855 00:44:34,707 --> 00:44:36,037 the same plans as in production. 856 00:44:36,037 --> 00:44:40,517 This is magic I realized some time ago, and it's super cool that you 857 00:44:40,517 --> 00:44:44,797 can s- have a tiny me- like memory small, shared buffers is tiny, but 858 00:44:44,827 --> 00:44:46,647 you say effective cache size is huge. 859 00:44:46,957 --> 00:44:49,677 Other p- planning parameters are huge, some other things, and 860 00:44:50,057 --> 00:44:52,347 you get exactly same plans as in 861 00:44:52,347 --> 00:44:52,867 production. 862 00:44:53,747 --> 00:44:57,687 Yeah, and then you need to just adjust you, you need s- small mind 863 00:44:57,687 --> 00:45:03,477 shift and engineer minds to make them focus on buffers numbers mo- more 864 00:45:03,477 --> 00:45:05,997 because timing might be different. 865 00:45:06,037 --> 00:45:09,467 This is what we talk about buffers three years already or four years? 866 00:45:10,617 --> 00:45:10,827 Yeah 867 00:45:11,513 --> 00:45:12,783 Michael: I'm super conscious of time, 868 00:45:12,833 --> 00:45:15,603 but Tudor, is there any, any last things or shout-outs you 869 00:45:15,603 --> 00:45:16,893 wanted to give on the Xata side? 870 00:45:16,893 --> 00:45:19,593 Like I'm very aware we haven't even talked about DeltaX, 871 00:45:19,593 --> 00:45:20,893 which looks really promising. 872 00:45:21,457 --> 00:45:23,027 Tudor: Maybe we do another one in a few, 873 00:45:23,229 --> 00:45:24,339 Nik: Yeah, let's have part two 874 00:45:25,059 --> 00:45:25,599 quite soon. 875 00:45:25,649 --> 00:45:25,969 Yeah. 876 00:45:26,719 --> 00:45:27,729 It's, it was cool. 877 00:45:27,739 --> 00:45:28,789 And thank you for coming. 878 00:45:28,789 --> 00:45:30,149 I enjoyed to- a lot. 879 00:45:30,169 --> 00:45:34,169 I think we do similar things, like in terms of where we ha- headed. 880 00:45:34,419 --> 00:45:36,329 I think everyone should use database branching. 881 00:45:36,339 --> 00:45:39,838 It's super cool, we will have all the links in show notes, right? 882 00:45:39,878 --> 00:45:42,808 And so I encourage everyone to try out if you haven't. 883 00:45:42,808 --> 00:45:46,488 And I guess you just need to tell your agent to try to go and use it, right? 884 00:45:46,508 --> 00:45:46,898 That's it. 885 00:45:47,976 --> 00:45:49,246 Tudor: Yeah, pretty much, yes. 886 00:45:49,658 --> 00:45:50,168 Nik: That's cool. 887 00:45:50,468 --> 00:45:51,238 Thank you for coming. 888 00:45:51,248 --> 00:45:52,538 Let's have part, part two soon. 889 00:45:54,120 --> 00:45:54,330 Michael: Yeah. 890 00:45:54,330 --> 00:45:55,330 Thanks so much, Judah 891 00:45:56,294 --> 00:45:56,684 Tudor: Thank you. 892 00:45:56,714 --> 00:45:57,664 It was really nice. 893 00:45:58,514 --> 00:45:59,024 Thanks both