1 0:0:0,06 --> 0:0:2,72 Michael: Hello and welcome to Postgres.FM, a weekly show about 2 0:0:2,72 --> 0:0:3,54 all things PostgreSQL. 3 0:0:3,6 --> 0:0:7,7999997 I am Michael, founder of pgMustard, and I am joined by 2 special 4 0:0:7,7999997 --> 0:0:8,299999 guests. 5 0:0:8,92 --> 0:0:13,78 Firstly, Salma El-Sayed, recently graduated from the Mansoura 6 0:0:14,059999 --> 0:0:18,0 University in a computer and control engineering degree and Google 7 0:0:18,0 --> 0:0:21,14 Summer of Code participant working on B-tree merge for Postgres. 8 0:0:21,18 --> 0:0:22,26 Welcome Salma. 9 0:0:23,099998 --> 0:0:24,439999 Salma: Hi. 10 0:0:25,24 --> 0:0:25,58 Michael: Thank you 11 0:0:25,58 --> 0:0:26,26 so much. 12 0:0:27,04 --> 0:0:28,34 Yeah, great to have you. 13 0:0:28,34 --> 0:0:32,18 And also Kirk Wolak, who is a software architect at KiraSoft, 14 0:0:32,58 --> 0:0:35,54 Google Summer of Code mentor, and a co-host with Nik on the 15 0:0:35,54 --> 0:0:37,26 YouTube Hacking Postgres series. 16 0:0:37,42 --> 0:0:39,14 Hi, Kirk, nice to have you too. 17 0:0:39,38 --> 0:0:42,54 Kirk: Michael, it's good to finally be able to be in 1 of these 18 0:0:42,54 --> 0:0:43,22 with you. 19 0:0:43,26 --> 0:0:43,94 Thank you. 20 0:0:44,72 --> 0:0:47,54 Michael: Yeah, I've watched many of your sessions as well, so 21 0:0:47,64 --> 0:0:50,36 this feels like a collab maybe. 22 0:0:50,6 --> 0:0:51,3 Kirk: Kind of. 23 0:0:51,38 --> 0:0:53,2 Michael: I wondered actually if we could start with you, Kirk. 24 0:0:53,2 --> 0:0:56,6 I wonder if you could give us a little bit of background on being 25 0:0:56,6 --> 0:0:58,22 a Google Summer of Code mentor. 26 0:0:58,32 --> 0:1:0,58 From your perspective, how does that work? 27 0:1:0,66 --> 0:1:2,08 For people that aren't familiar with it? 28 0:1:2,08 --> 0:1:3,1 What does it mean? 29 0:1:3,82 --> 0:1:4,28 Kirk: Sure. 30 0:1:4,28 --> 0:1:8,3 In fact, I just did a lightning round as a mentor because of 31 0:1:8,3 --> 0:1:11,76 how we integrated AI, because they're really curious about moving 32 0:1:11,76 --> 0:1:12,26 forward. 33 0:1:12,56 --> 0:1:16,78 So 1st off, Google Summer of Code is an initiative to help bring 34 0:1:16,78 --> 0:1:19,38 new people into the open source world. 35 0:1:19,64 --> 0:1:22,54 And the ultimate goal is to create new contributors. 36 0:1:23,32 --> 0:1:26,18 And I'm taking that completely to heart. 37 0:1:26,28 --> 0:1:30,6 So my goal for Salma is to make sure she not only does something 38 0:1:30,6 --> 0:1:35,04 big, but we also make it so that she contributes in the future. 39 0:1:35,24 --> 0:1:38,86 So much so that we're arranging for other companies to support 40 0:1:38,86 --> 0:1:39,9 her in these efforts. 41 0:1:40,24 --> 0:1:43,68 So that long-term we absolutely have a contributor. 42 0:1:43,9 --> 0:1:47,12 My position, it's my 1st time mentoring and I'm getting help 43 0:1:47,12 --> 0:1:52,5 from Andreas Karlsson, Andrey Borodin from the hackers thing, 44 0:1:52,5 --> 0:1:56,2 and there's a couple other people, Andrei Lepikhov, I believe 45 0:1:56,2 --> 0:1:57,8 saying all these names is hard. 46 0:1:57,8 --> 0:1:59,5 I call them together, the Andres. 47 0:2:0,04 --> 0:2:2,38 All 3 people I work with are called Andre Great. 48 0:2:2,52 --> 0:2:5,6 Anyways, but yeah, so the Google Summer of Code Initiative is 49 0:2:5,6 --> 0:2:6,54 for that purpose. 50 0:2:7,04 --> 0:2:11,0 And we actually, with Nik, because of all the hacking, we actually 51 0:2:11,0 --> 0:2:12,6 had 5 or 6 projects. 52 0:2:13,1 --> 0:2:15,32 And this was by far the hardest project. 53 0:2:15,64 --> 0:2:20,22 This was fixing a 30-year-old Postgres problem that was too hard 54 0:2:20,22 --> 0:2:24,34 for the original PhDs who wrote the paper, Yao et al. 55 0:2:24,96 --> 0:2:29,16 And Shasha later on how we actually handle our B-tree stuff. 56 0:2:29,18 --> 0:2:33,68 So it left behind a gaping hole and I'm 1 of these people I have 57 0:2:33,74 --> 0:2:37,4 not as much programming in Postgres world but I've been chief 58 0:2:37,4 --> 0:2:40,76 architect of software and doing software since I was a teenager 59 0:2:40,76 --> 0:2:44,98 professionally so for me once I got access to AI's and then a 60 0:2:44,98 --> 0:2:49,54 good programmer like Salma I'm like the world is my oyster and 61 0:2:49,54 --> 0:2:51,94 then of course you learn their lessons from there. 62 0:2:53,56 --> 0:2:56,32 Michael: Yeah, calling this a big project I think might still 63 0:2:56,32 --> 0:2:57,44 be understating it. 64 0:2:57,44 --> 0:3:2,18 This is a huge ambitious undertaking But let's come back to that 65 0:3:2,18 --> 0:3:4,28 because I wanted to hear from Salma as well. 66 0:3:4,28 --> 0:3:7,7 Just from the basics, what made you interested in doing Google 67 0:3:7,7 --> 0:3:8,36 Summer of Code? 68 0:3:8,36 --> 0:3:9,18 Why Postgres? 69 0:3:9,56 --> 0:3:11,68 Were you looking for a project this big? 70 0:3:11,68 --> 0:3:13,64 From your perspective, what's it been like? 71 0:3:16,08 --> 0:3:18,62 Salma: I was interested in databases. 72 0:3:18,86 --> 0:3:24,0 I was studying the CMU course, it's Database Introduction to 73 0:3:24,0 --> 0:3:24,5 Database. 74 0:3:25,08 --> 0:3:28,54 It's from the University of CMU, but it's online. 75 0:3:29,28 --> 0:3:33,66 And I was working on a project implementing B+ tree for the BusTub 76 0:3:33,9 --> 0:3:34,4 project. 77 0:3:34,64 --> 0:3:38,74 It's the database management system they have project. 78 0:3:38,74 --> 0:3:40,74 We implement it for it. 79 0:3:42,18 --> 0:3:46,48 And also I know some people from my university who got accepted 80 0:3:46,62 --> 0:3:48,44 to Google Summer of Code last year. 81 0:3:48,82 --> 0:3:55,22 So I wanted to check what's going on in GSoC and check the organizations. 82 0:3:55,76 --> 0:4:0,8 And then when I found Postgres, I found a project about the 83 0:4:0,8 --> 0:4:4,04 B+ tree and I found it interesting. 84 0:4:4,54 --> 0:4:8,82 Then I didn't know anything about the internals of the Postgres 85 0:4:9,14 --> 0:4:14,2 at that set point, but I sent Kirk an email asking about what 86 0:4:14,2 --> 0:4:19,12 should I do 1st, how should I prepare, if my knowledge and my 87 0:4:19,12 --> 0:4:21,8 level is good for this project? 88 0:4:22,48 --> 0:4:27,18 And he responded with the hacking sessions he, Andrey and Nik 89 0:4:27,18 --> 0:4:27,68 had. 90 0:4:28,28 --> 0:4:32,42 And I watched the sessions and asked him again and sent another 91 0:4:32,42 --> 0:4:37,06 email asking questions about B-trees and the bloat problems, the 92 0:4:37,06 --> 0:4:39,66 ideas they discussed in the videos. 93 0:4:41,12 --> 0:4:43,6 And that's when everything started. 94 0:4:43,68 --> 0:4:48,8 We kept talking back and forth until GSoC started in May. 95 0:4:50,66 --> 0:4:52,94 Michael: Yeah, so it sounds like a match made in heaven. 96 0:4:52,94 --> 0:4:56,12 Kirk's crazy enough to suggest a project this huge and ambitious 97 0:4:56,12 --> 0:4:59,18 and you're crazy enough to be interested in it. 98 0:4:59,24 --> 0:5:2,3 Okay I'll start and see how this is working. 99 0:5:2,58 --> 0:5:5,18 Kirk: So there's a magic truth into that. 100 0:5:5,46 --> 0:5:9,88 I've been running and managing developers for the last 30 years. 101 0:5:10,26 --> 0:5:13,7 And 1 of the secrets I've learned in managing developers is you 102 0:5:13,7 --> 0:5:17,2 only give them enough information that helps their confidence, 103 0:5:17,86 --> 0:5:21,14 and you hide the things they're going to run into into the future 104 0:5:21,48 --> 0:5:22,9 until they get there. 105 0:5:23,04 --> 0:5:26,72 Otherwise you overwhelm them and they give up and and having 106 0:5:26,72 --> 0:5:31,78 such a let's say naive student was so helpful in the beginning 107 0:5:31,78 --> 0:5:33,84 as she just learned last week. 108 0:5:35,02 --> 0:5:36,36 So yes. 109 0:5:37,02 --> 0:5:39,88 Michael: There is definitely, I can definitely see that that 110 0:5:39,88 --> 0:5:43,78 there are benefits to that side of things but it reminds me of 111 0:5:43,78 --> 0:5:47,8 that if it's a meme or something or maybe just a post that was 112 0:5:47,8 --> 0:5:50,28 just funny it said something like we don't do this because it's 113 0:5:50,28 --> 0:5:53,24 easy we do this because we thought it would be easy something 114 0:5:53,24 --> 0:5:53,94 like that 115 0:5:54,0 --> 0:5:59,54 Kirk: yes exactly and and if you want I can give you the this 116 0:5:59,54 --> 0:6:3,58 is the cool part and this is my kind of metaphor for helping 117 0:6:3,58 --> 0:6:5,42 everyone understand what's going on. 118 0:6:5,5 --> 0:6:11,04 All right, fixing a B-tree and doing B-tree merge and page merging 119 0:6:11,04 --> 0:6:15,38 and keeping it perfectly clean is absolutely well studied and 120 0:6:15,38 --> 0:6:20,14 it's easy in a single user environment with only 1 thread that 121 0:6:20,14 --> 0:6:21,4 does the reading and writing. 122 0:6:21,72 --> 0:6:25,8 Okay, the only thing that makes this complicated is that you 123 0:6:25,8 --> 0:6:30,04 have potentially 10, 000 other people reading the same structure 124 0:6:30,42 --> 0:6:32,02 while you're changing it. 125 0:6:32,26 --> 0:6:35,94 And the magic, and this is the part that we had to understand, 126 0:6:36,4 --> 0:6:40,52 the magic is simply 1st we are limited in how big of changes 127 0:6:40,52 --> 0:6:45,56 we can make at every step so that anybody who notices the change 128 0:6:45,6 --> 0:6:46,36 can self-correct. 129 0:6:47,02 --> 0:6:51,3 And then next is to turn it into the small number of steps into 130 0:6:51,3 --> 0:6:56,52 the future until eventually all the changes are made and the 131 0:6:56,52 --> 0:6:59,98 B-tree is back to normal and you didn't break anybody who's out 132 0:6:59,98 --> 0:7:0,68 there running. 133 0:7:0,7 --> 0:7:3,62 Because you can't break a search, you can't break an insert, 134 0:7:3,62 --> 0:7:4,9 you can't break a delete. 135 0:7:5,14 --> 0:7:8,04 If you break these things, obviously you're breaking them in 136 0:7:8,04 --> 0:7:8,8 other threads. 137 0:7:9,06 --> 0:7:11,28 So that's the only thing that makes it hard. 138 0:7:11,28 --> 0:7:15,7 If we could just do it and do it all on our own, For example, 139 0:7:15,76 --> 0:7:19,0 if we could actually send a message to everyone scanning an index 140 0:7:19,0 --> 0:7:23,24 right now and say stop, restart after I'm done, and then we make 141 0:7:23,24 --> 0:7:25,7 the changes and push them, that would be easy. 142 0:7:25,76 --> 0:7:26,23 All right? 143 0:7:26,23 --> 0:7:29,32 But imagine having a 30-minute query get stopped and have to 144 0:7:29,32 --> 0:7:29,82 restart. 145 0:7:30,04 --> 0:7:31,38 That would be horrible implementation. 146 0:7:31,84 --> 0:7:32,58 Hey, Nik. 147 0:7:33,4 --> 0:7:33,72 Nik: Hello. 148 0:7:33,72 --> 0:7:34,22 Hello. 149 0:7:34,54 --> 0:7:36,36 Yeah, apologies for being late. 150 0:7:36,5 --> 0:7:38,14 This is 1st time so far. 151 0:7:38,3 --> 0:7:39,38 Michael: Good to have you here. 152 0:7:39,38 --> 0:7:44,8 I think we should even, maybe we could go back a step and look 153 0:7:44,8 --> 0:7:48,34 at Why even bother now? 154 0:7:48,34 --> 0:7:51,86 Like Postgres has got this far without B-tree merge. 155 0:7:52,7 --> 0:7:53,48 Kirk: Good question. 156 0:7:53,48 --> 0:7:57,4 Michael: It could conceivably continue for quite some time without 157 0:7:57,4 --> 0:7:57,6 it. 158 0:7:57,6 --> 0:8:4,04 Like what problems is it causing and why are the existing ways 159 0:8:4,04 --> 0:8:5,82 of handling that maybe not sufficient? 160 0:8:6,28 --> 0:8:9,04 Or why, if this is such a big undertaking on such a difficult 161 0:8:9,06 --> 0:8:11,6 thing, the payoff must be big too, right? 162 0:8:11,6 --> 0:8:13,4 Like what's the benefit? 163 0:8:13,7 --> 0:8:17,72 Kirk: Yeah, so to say the payoff must be big too isn't necessarily 164 0:8:17,72 --> 0:8:18,76 the same thing. 165 0:8:19,94 --> 0:8:23,2 The people who need it the most are the people who can afford 166 0:8:23,2 --> 0:8:24,04 it the least. 167 0:8:24,28 --> 0:8:28,3 So for example, this started because Andrey Borodin mentioned 168 0:8:28,62 --> 0:8:32,92 they have a table that's so large at his company that if they 169 0:8:32,92 --> 0:8:39,66 do reindex concurrently, 1 index takes over 72 hours to reindex 170 0:8:39,82 --> 0:8:42,3 concurrently to get rid of this bloat. 171 0:8:43,14 --> 0:8:46,26 So the thing is, and then there's also times where there's no 172 0:8:46,26 --> 0:8:49,18 getting rid of the bloat if it was a one-time set of deletes 173 0:8:49,54 --> 0:8:52,48 and you're not doing a bunch of inserts back in the beginning 174 0:8:52,48 --> 0:8:55,9 part of the index you're not cleaning that up it's just gonna 175 0:8:55,9 --> 0:8:59,48 stay there forever so part of the problem is bloat impacts space 176 0:8:59,48 --> 0:9:3,82 usage but that every time you make space bigger you also start 177 0:9:3,82 --> 0:9:6,42 impacting time, especially if it's empty space. 178 0:9:6,82 --> 0:9:9,72 So now your search is taking, you're reading more buffers, you're 179 0:9:9,72 --> 0:9:13,1 doing more work than you have to do to get and collect these 180 0:9:13,1 --> 0:9:13,6 things. 181 0:9:13,86 --> 0:9:17,9 So the 1st thing is, it was hard in the past, Very few people 182 0:9:17,9 --> 0:9:20,98 understood all of the complexity and therefore they weren't willing 183 0:9:20,98 --> 0:9:21,94 to make the change. 184 0:9:22,04 --> 0:9:26,14 AI is available now that can explain all of this complexity better. 185 0:9:26,68 --> 0:9:31,24 And even a guy like me, who's got 40 years experience elsewhere, 186 0:9:31,56 --> 0:9:34,44 I can now come in and point that experience at a problem with 187 0:9:34,44 --> 0:9:39,24 the help of AI and wrap my arms around it and go, oh I see what 188 0:9:39,24 --> 0:9:40,32 the problem is. 189 0:9:40,44 --> 0:9:42,04 Everyone's trying to do it quickly. 190 0:9:42,44 --> 0:9:46,96 We need to do it slowly and if we make the change slowly then 191 0:9:46,96 --> 0:9:48,04 it's possible to do. 192 0:9:48,04 --> 0:9:48,98 What's the benefit? 193 0:9:49,4 --> 0:9:52,58 Postgres ends up with self-healing indexes. 194 0:9:53,24 --> 0:9:54,8 That's how I refer to them. 195 0:9:54,8 --> 0:9:57,6 As you naturally, in fact, where does this cleanup belong? 196 0:9:57,6 --> 0:10:0,36 It belongs in vacuum and the delete code. 197 0:10:0,42 --> 0:10:3,22 When you delete records, it just marks them as deleted. 198 0:10:3,74 --> 0:10:7,9 When you actually remove them from there and you leave a page 199 0:10:7,9 --> 0:10:10,92 mostly empty, that's the logic we're writing and they're going 200 0:10:10,92 --> 0:10:14,24 to be able to put that right back in the vacuum and say, oh, 201 0:10:14,24 --> 0:10:18,58 let's push these tuples over here onto this very empty page already 202 0:10:18,7 --> 0:10:22,42 and let's start the unlink process so the vacuum process which 203 0:10:22,42 --> 0:10:26,54 already does a lot of work that Postgres counts on can now pick 204 0:10:26,54 --> 0:10:30,96 this stuff up and then let's we don't need the reindex go ahead 205 0:10:31,94 --> 0:10:36,56 Nik: let me explain my perspective When I was studying B-tree in 206 0:10:36,56 --> 0:10:42,54 school, 20 plus years ago, they told us, books like Chris Date, 207 0:10:42,54 --> 0:10:48,58 Aho Ullman, other books, they told us that B-tree is almost balanced 208 0:10:49,46 --> 0:10:52,36 tree with a lot of children in each node, right? 209 0:10:52,36 --> 0:10:56,16 Almost balanced means that from root to leaf it's always n or 210 0:10:56,16 --> 0:10:57,18 n plus 1. 211 0:10:57,6 --> 0:11:2,72 And there are algorithms that rebalance it when we insert data 212 0:11:2,78 --> 0:11:5,14 or remove data from leaves. 213 0:11:5,66 --> 0:11:11,38 So it means that when you insert and usually when we have integer 214 0:11:11,38 --> 0:11:16,22 4, int8, always on the right side, right? 215 0:11:16,86 --> 0:11:22,16 And when you insert and the right side leaf becomes already almost 216 0:11:23,48 --> 0:11:26,46 full, it's split, right? 217 0:11:26,72 --> 0:11:28,28 And then rebalancing happens. 218 0:11:28,46 --> 0:11:32,26 And vice versa, when we delete everything, some leaves are empty, 219 0:11:32,48 --> 0:11:35,58 the balance happens when leaves are being deleted. 220 0:11:37,2 --> 0:11:41,68 But for me, some time ago, it was a big surprise that only the 221 0:11:41,68 --> 0:11:43,94 1st part is implemented in Postgres. 222 0:11:45,4 --> 0:11:51,1 And only then realization came how actual index bloat is happening. 223 0:11:52,18 --> 0:11:56,54 And we know that there are optimizations in Postgres 13 and 14 224 0:11:56,84 --> 0:12:0,56 for deduplication, but it's not solving that. 225 0:12:1,08 --> 0:12:5,9 As you mentioned, Kirk, if you update or delete, especially if 226 0:12:5,9 --> 0:12:10,12 you delete, updates also like kind of deletes plus inserts, 227 0:12:10,12 --> 0:12:10,44 right? 228 0:12:10,44 --> 0:12:15,7 If you delete entries in leaves, which are in the middle of the 229 0:12:15,7 --> 0:12:21,06 whole range, It creates basically empty space, right? 230 0:12:21,06 --> 0:12:22,36 And it stays there. 231 0:12:22,54 --> 0:12:23,66 A fun correction. 232 0:12:24,64 --> 0:12:28,66 In the heap, if whole page is empty, it will be truncated if 233 0:12:28,66 --> 0:12:29,96 it's at the end. 234 0:12:29,96 --> 0:12:34,06 In B-tree, it will be truncated even if it's in the middle, 235 0:12:34,08 --> 0:12:34,58 right? 236 0:12:34,94 --> 0:12:37,66 But it should be fully empty and this is rare. 237 0:12:37,66 --> 0:12:39,2 Kirk: So- That's the magic. 238 0:12:39,72 --> 0:12:44,06 Current Postgres only deletes a page from a leaf page if it's 239 0:12:44,06 --> 0:12:45,46 100 percent empty. 240 0:12:45,6 --> 0:12:50,26 If it's 99.999, meaning there's just 1 record out of 8, 000, 241 0:12:51,06 --> 0:12:53,66 it's going to hold that page open, okay? 242 0:12:53,86 --> 0:12:57,16 And they don't want to go through the trouble of moving that 243 0:12:57,16 --> 0:12:57,84 1 record. 244 0:12:57,88 --> 0:12:58,38 Yeah. 245 0:12:59,16 --> 0:13:1,74 Nik: How many references can be in a page? 246 0:13:1,74 --> 0:13:3,18 8, 000 is a page. 247 0:13:3,18 --> 0:13:4,6 Kirk: Yeah, that's a page size. 248 0:13:4,6 --> 0:13:5,28 Yeah, but. 249 0:13:5,28 --> 0:13:6,14 Nik: Yeah, yeah. 250 0:13:6,6 --> 0:13:10,04 And yeah, anyway, but so this is how bloat and it cannot be removed 251 0:13:10,04 --> 0:13:10,74 by vacuum. 252 0:13:10,92 --> 0:13:12,54 It's just empty space, right? 253 0:13:13,08 --> 0:13:18,96 And this means that UUID version 4 has advantage over UUID version 254 0:13:18,96 --> 0:13:19,46 7. 255 0:13:20,6 --> 0:13:21,74 Kirk: No, hold it. 256 0:13:21,74 --> 0:13:23,36 UUID 7 had the advantage. 257 0:13:23,36 --> 0:13:24,52 It went to the right. 258 0:13:24,62 --> 0:13:26,04 Oh, I get what you're saying. 259 0:13:26,04 --> 0:13:26,54 Yes. 260 0:13:27,04 --> 0:13:30,48 Nik: Our normal way of thinking, UUID version 7 has advantage 261 0:13:30,48 --> 0:13:35,72 because data locality, all fresh data stays in fewer pages. 262 0:13:36,82 --> 0:13:40,56 But this also means that if you have int4 or int8 primary 263 0:13:40,56 --> 0:13:45,1 keys, delete the data in the middle, it leads to bloat and that's 264 0:13:45,1 --> 0:13:45,52 it. 265 0:13:45,52 --> 0:13:47,86 Without merge algorithm, right? 266 0:13:47,86 --> 0:13:53,56 And the UUID version 4, you have some chances to have a new insert 267 0:13:53,56 --> 0:13:56,14 happening to the same leaf, maybe. 268 0:13:56,16 --> 0:13:57,58 Anyway, this is my understanding. 269 0:13:57,66 --> 0:14:0,76 It was a big surprise for me that Postgres doesn't have split, 270 0:14:1,3 --> 0:14:2,24 not split, merge, right? 271 0:14:2,24 --> 0:14:2,74 Kirk: Merge. 272 0:14:3,32 --> 0:14:4,32 And their argument... 273 0:14:5,38 --> 0:14:7,96 The other key point I want to make is because of the research 274 0:14:7,96 --> 0:14:11,3 I did and listening to Hacker's comment before. 275 0:14:12,08 --> 0:14:15,64 So you understand sometimes programmers justify their decisions 276 0:14:15,8 --> 0:14:17,44 after the fact, okay? 277 0:14:17,8 --> 0:14:21,3 I'm a software developer, I confess, I've done this myself. 278 0:14:21,6 --> 0:14:25,24 Anyway, so 1 of the pieces of feedback I got from Bruce 279 0:14:25,6 --> 0:14:31,26 was, oh, they tested theoretically that merging at 50% empty 280 0:14:31,68 --> 0:14:33,62 just causes more page splits. 281 0:14:34,08 --> 0:14:37,0 And I'm thinking to myself, what a straw man argument. 282 0:14:37,44 --> 0:14:42,42 Okay, nobody in their right mind should take 2 50% empty pages 283 0:14:42,66 --> 0:14:45,36 and merge them and create 100% full page. 284 0:14:45,48 --> 0:14:47,62 That's why we have things like fillfactor. 285 0:14:47,9 --> 0:14:50,82 Of course that page is just gonna split on the next insert. 286 0:14:51,3 --> 0:14:55,76 So, but yes this gets back to the arguments against bothering 287 0:14:55,76 --> 0:14:57,08 to solve this problem. 288 0:14:57,62 --> 0:15:1,76 I think the problem has become easier to solve in today's world. 289 0:15:2,26 --> 0:15:5,38 Michael: Just to put the other perspective across a little bit, 290 0:15:5,38 --> 0:15:8,56 I think I agree that it's good to, we should definitely approach 291 0:15:8,56 --> 0:15:11,96 new hard problems, like if we, someone's got to at some point, 292 0:15:11,96 --> 0:15:13,1 right, it makes sense. 293 0:15:13,38 --> 0:15:17,72 But, This is like incredibly difficult part of the code. 294 0:15:17,72 --> 0:15:18,9 It's B-tree indexes. 295 0:15:18,9 --> 0:15:21,64 It affects every Postgres installation out there. 296 0:15:21,68 --> 0:15:24,96 The risk of making mistakes is like really bad, right? 297 0:15:24,96 --> 0:15:26,94 We're moving things around in a B-tree. 298 0:15:27,26 --> 0:15:28,84 That means ordering matters. 299 0:15:28,86 --> 0:15:30,36 And like we can have corruption. 300 0:15:30,36 --> 0:15:33,54 We could miss entries in a scan and think there isn't data, or 301 0:15:33,54 --> 0:15:36,26 we could see an entry twice and get duplicated data. 302 0:15:36,26 --> 0:15:41,0 So the like risk of it going wrong is so high that like it has 303 0:15:41,0 --> 0:15:46,44 to be perfect and it is it has to be like foolproof like real 304 0:15:46,74 --> 0:15:47,24 solid. 305 0:15:47,36 --> 0:15:50,38 So then you have to say then what's the like if we're going to 306 0:15:50,38 --> 0:15:54,4 risk doing any solution to this It like what's the benefit? 307 0:15:54,4 --> 0:15:57,08 So I think then we do have to come back to like how do we solve 308 0:15:57,08 --> 0:16:2,06 this currently and I think Andrey's 72 hour reindexing this concurrently 309 0:16:2,48 --> 0:16:7,08 must be such a big table and like put on a big index as well 310 0:16:7,08 --> 0:16:12,44 right that it must be like an absolute outlier and most big indexes 311 0:16:12,56 --> 0:16:16,74 still should be like in the order of tens of minutes or hours 312 0:16:16,74 --> 0:16:19,62 like like single-digit hours I would have thought 313 0:16:19,62 --> 0:16:21,56 Kirk: but and why have to 314 0:16:21,58 --> 0:16:24,34 Michael: okay so the downsides there are downsides right like 315 0:16:24,34 --> 0:16:28,26 it you can only run 1 at a time, you need the space again, but 316 0:16:28,26 --> 0:16:30,18 like that's only disk space, right? 317 0:16:30,3 --> 0:16:31,56 Nik: Are you talking about reindexing? 318 0:16:32,02 --> 0:16:33,88 Kirk: Yeah, yeah, it's reindexing concurrently. 319 0:16:34,22 --> 0:16:37,92 Nik: It's been horizon pinned, this is a key problem. 320 0:16:38,04 --> 0:16:38,54 Yeah. 321 0:16:39,52 --> 0:16:44,82 So you fight bloat in 1 index and cause bloat everywhere in that 322 0:16:44,82 --> 0:16:45,32 database. 323 0:16:46,52 --> 0:16:49,08 Michael: But there's like, we've talked, Nik, we've talked many 324 0:16:49,08 --> 0:16:52,2 times about, like, this is a reason to partition or keep your 325 0:16:52,2 --> 0:16:53,04 tables smaller. 326 0:16:53,04 --> 0:16:56,88 Like, there are other design solutions around this area. 327 0:16:56,98 --> 0:17:1,7 So I think I've also personally seen indexes that were 99% bloat 328 0:17:1,7 --> 0:17:3,34 because of, like, access patterns. 329 0:17:3,34 --> 0:17:7,48 So you can get these extreme cases that you're talking about. 330 0:17:7,48 --> 0:17:9,18 I've seen it in real world workplace. 331 0:17:9,52 --> 0:17:9,86 Nik: Simple. 332 0:17:9,86 --> 0:17:14,44 Yeah, simplest example is when we have like a long history of 333 0:17:14,44 --> 0:17:17,32 something like orders in e-commerce and then they decide to clean 334 0:17:17,32 --> 0:17:20,74 up, they don't need orders exceeding 1 year or 2 years, they 335 0:17:20,74 --> 0:17:21,98 delete all that data. 336 0:17:21,98 --> 0:17:28,38 But since int8 or UUID version 7 primary key, bloat stays 337 0:17:28,38 --> 0:17:31,26 in index and the only way is to rebuild it. 338 0:17:31,68 --> 0:17:32,18 Yeah. 339 0:17:32,2 --> 0:17:33,3 So the simplest example. 340 0:17:33,48 --> 0:17:37,12 Maybe I'm not right, because if you delete whole data from the 341 0:17:37,12 --> 0:17:37,62 past. 342 0:17:37,68 --> 0:17:40,46 Michael: Imagine if customers on like a software as a service 343 0:17:40,46 --> 0:17:42,8 application, you delete all of their data because they leave 344 0:17:42,8 --> 0:17:44,22 the service and that removes 345 0:17:44,22 --> 0:17:44,72 Nik: like 346 0:17:44,72 --> 0:17:47,28 Michael: 80% of, like maybe they're a big customer leaves and 347 0:17:47,28 --> 0:17:51,56 80% of every page goes like that's a very easy way of Removing 348 0:17:51,56 --> 0:17:56,12 it moving a lot of every page, but not 100 percent of every anyone 349 0:17:56,12 --> 0:17:58,16 page I get that this problem. 350 0:17:58,38 --> 0:18:1,46 I do get that this problem exists, But I'm just pushing back 351 0:18:1,46 --> 0:18:5,44 to say is it as bad a by the way to add to your to add to your 352 0:18:5,44 --> 0:18:7,44 case It also pollutes the cache. 353 0:18:7,44 --> 0:18:10,38 I think that's a big deal, especially in the days of memory getting 354 0:18:10,38 --> 0:18:13,78 more expensive There are other knock-on impacts of carrying all 355 0:18:13,78 --> 0:18:18,04 of this wasted space around but I'm still of the opinion that 356 0:18:18,04 --> 0:18:21,5 the bar should be extremely high for attacking this. 357 0:18:21,5 --> 0:18:24,18 Kirk: Well, okay, I'll give you your argument temporarily. 358 0:18:24,96 --> 0:18:28,98 Let me give you the opposite side of our argument for how hard 359 0:18:28,98 --> 0:18:30,84 it turned out not to be. 360 0:18:30,84 --> 0:18:34,46 Now, granted, our patch isn't complete, but we had a working 361 0:18:34,46 --> 0:18:38,16 prototype in the 1st 0.5 of the Google Summer of Code project 362 0:18:38,64 --> 0:18:39,14 timeline. 363 0:18:41,26 --> 0:18:42,26 Michael: Okay, let's wait. 364 0:18:42,26 --> 0:18:43,86 So should we switch to solution then? 365 0:18:43,86 --> 0:18:47,54 And Salma, do you want to talk us through what was your approach? 366 0:18:47,54 --> 0:18:50,64 Like how did you go about designing this and why did you make 367 0:18:50,64 --> 0:18:52,1 certain design decisions? 368 0:18:53,4 --> 0:18:55,36 Salma: Okay, we had a lot of discussions. 369 0:18:55,76 --> 0:18:59,7 1st we had a lot of prototype and a lot of 1st ideas. 370 0:19:0,82 --> 0:19:6,16 Our 1st design idea was to merge 2 pages together. 371 0:19:7,54 --> 0:19:13,38 So we have a left page and a right page and we move all data 372 0:19:13,38 --> 0:19:16,46 from the left page to the right 1. 373 0:19:17,26 --> 0:19:23,16 And we use the left page as a direction, leaves the data in it 374 0:19:23,68 --> 0:19:26,26 as a direction for backward scans. 375 0:19:27,4 --> 0:19:32,5 So when a scanner, when a scan read the right page, then it was 376 0:19:32,5 --> 0:19:37,32 waiting between the 2 pages, and we merged these 2 pages together. 377 0:19:37,74 --> 0:19:45,92 Then after the merge happened, it landed on the left page, which 378 0:19:45,92 --> 0:19:47,26 we removed its data. 379 0:19:47,52 --> 0:19:52,8 So our 1st idea was to keep the data in the left page so this 380 0:19:52,8 --> 0:19:55,8 backward scan can read it and go on. 381 0:19:55,8 --> 0:19:59,16 No need to recover, no need to do any extra work. 382 0:19:59,38 --> 0:20:3,98 But When we proposed this to the hackers, they said that this 383 0:20:4,34 --> 0:20:8,44 is a corruption to the index because we have repeated data and 384 0:20:8,44 --> 0:20:9,98 it will cause a lot of problems. 385 0:20:10,52 --> 0:20:16,78 Also, when vacuum cleans the index, So if it's going to clean 386 0:20:16,78 --> 0:20:21,08 these 2 or only clean the data or the right page. 387 0:20:22,12 --> 0:20:25,82 So MUSECUR try to find another solution. 388 0:20:26,68 --> 0:20:32,64 So the solution we are working on, and we sent the 1st concept 389 0:20:32,64 --> 0:20:41,24 about hackers, was instead of keeping the left page as as hidden 390 0:20:41,24 --> 0:20:46,08 data, we only keep it only to route the scans, the forward and 391 0:20:46,08 --> 0:20:48,06 backward scan to how to recover. 392 0:20:48,9 --> 0:20:57,34 When a scanner read the right page, it's about to read the left 393 0:20:57,34 --> 0:21:1,48 page, which is now a tombstone, it doesn't contain any 394 0:21:1,48 --> 0:21:1,98 data. 395 0:21:3,18 --> 0:21:7,98 Actually, the scan in this position have a scan opaque data saved 396 0:21:8,16 --> 0:21:11,5 and not overridden yet. 397 0:21:12,34 --> 0:21:18,94 So this scan opaque contains the TID, a list of TIDs from the 398 0:21:18,94 --> 0:21:22,4 right page read before the merger actually happened. 399 0:21:23,22 --> 0:21:30,98 So we are using this to save these TIDs in another list. 400 0:21:31,94 --> 0:21:37,28 So when we read the right page, we know which TIDs we read and 401 0:21:37,28 --> 0:21:40,9 which values we read and which we haven't yet read. 402 0:21:41,3 --> 0:21:49,32 So we go back, read the page again and eliminate all the values 403 0:21:49,34 --> 0:21:52,7 we haven't seen before, which are saved. 404 0:21:53,16 --> 0:21:58,64 So anything except the ones we saved in the list we have. 405 0:21:59,02 --> 0:22:1,76 Kirk: So again, what we called the original merged away page, 406 0:22:1,76 --> 0:22:3,3 we called those ghost records. 407 0:22:3,4 --> 0:22:5,72 They were ghost copies of the original records. 408 0:22:5,94 --> 0:22:7,76 And let's understand 1 thing. 409 0:22:8,0 --> 0:22:12,94 If I have 100 scans going on and I merge 2 leaf pages together 410 0:22:13,18 --> 0:22:16,64 and nobody notices because they were in different parts of the 411 0:22:16,64 --> 0:22:17,14 tree? 412 0:22:17,44 --> 0:22:18,66 Does anyone care? 413 0:22:19,86 --> 0:22:24,3 No, because by the time they get to those pages, they'll be correct. 414 0:22:24,72 --> 0:22:29,76 There's just a tiny amount of time where we could actually do 415 0:22:29,76 --> 0:22:33,56 the merge while people are currently have these things read in 416 0:22:33,56 --> 0:22:37,36 memory and they're processing them waiting to read either the 417 0:22:37,36 --> 0:22:40,94 next or the previous page and that's what Salma was talking about. 418 0:22:40,96 --> 0:22:44,58 So if we have scans that are touching the points that we're doing 419 0:22:44,58 --> 0:22:47,74 this to, those are the only ones we care about. 420 0:22:47,8 --> 0:22:51,46 If they're 1 before it or 1 on the other side of it, we don't 421 0:22:51,46 --> 0:22:54,52 care about those scans because they'll be correct by the time 422 0:22:54,52 --> 0:22:55,98 our lock lets go. 423 0:22:56,2 --> 0:23:0,16 Unfortunately for the guys who read in the page that we're going 424 0:23:0,16 --> 0:23:3,68 to merge away, they read in and they processed 4 records off 425 0:23:3,68 --> 0:23:4,5 of that page. 426 0:23:4,7 --> 0:23:8,3 Then they read the next page and now those 4 records have already 427 0:23:8,3 --> 0:23:9,84 been added to that page. 428 0:23:9,84 --> 0:23:12,28 That would cause the error of duplicate records. 429 0:23:12,72 --> 0:23:13,98 We can't allow that. 430 0:23:14,22 --> 0:23:19,2 So Salma explained, all we did was we kept in memory the records 431 0:23:19,2 --> 0:23:22,22 that we read from the previous page, the 4 that we accepted. 432 0:23:22,58 --> 0:23:26,68 And then on this page, right as we go to process this, we know 433 0:23:26,68 --> 0:23:28,88 to save this because we detect the flag. 434 0:23:29,1 --> 0:23:32,98 And then when we read all these new records plus these 4, we 435 0:23:32,98 --> 0:23:37,62 subtract these 4 off and only add the new records to our scan. 436 0:23:38,08 --> 0:23:41,58 And so the forward scan continues and it didn't miss a beat. 437 0:23:41,58 --> 0:23:45,44 And we do effectively the exact same thing in reverse for a backwards 438 0:23:45,44 --> 0:23:45,94 scan. 439 0:23:46,18 --> 0:23:49,54 That's the only special cases we have to worry about. 440 0:23:49,78 --> 0:23:53,0 Right at that edge case and it's only when they're reading. 441 0:23:53,0 --> 0:23:55,32 So what did we do to reduce the risks? 442 0:23:55,8 --> 0:23:59,52 1, we'll only merge 2 leaf pages together. 443 0:24:0,04 --> 0:24:4,02 If you try to merge more than 2, you introduce undetectable errors. 444 0:24:4,48 --> 0:24:8,0 Everything we do, we should make detectable and then we should 445 0:24:8,0 --> 0:24:8,5 handle. 446 0:24:8,68 --> 0:24:13,18 So by merging only 2 pages together, we limit how much we can 447 0:24:13,18 --> 0:24:17,26 actually fix, but at the same time, we limit what we can break 448 0:24:17,36 --> 0:24:21,9 to a known set and that's then what we implement for the code. 449 0:24:21,98 --> 0:24:25,92 Next we make sure that these 2 leaf nodes are only pointed at 450 0:24:25,92 --> 0:24:29,64 from the same parent so this way we don't have to repair the 451 0:24:29,64 --> 0:24:31,1 upper structure of the tree. 452 0:24:31,32 --> 0:24:34,84 This is another 90, 80, 20, 90, 10 hack. 453 0:24:35,02 --> 0:24:37,1 Why are we doing it that way? 454 0:24:37,12 --> 0:24:40,22 Because again, I don't want to repair anything more. 455 0:24:40,44 --> 0:24:45,06 I just want to work on these 3 nodes so that this way the 2 leaf 456 0:24:45,06 --> 0:24:46,94 pages can be turned into 1. 457 0:24:47,12 --> 0:24:50,5 Everyone who used to point at this 1 will now point to this 1 458 0:24:50,5 --> 0:24:55,72 and slowly this page will become empty and disappear just like 459 0:24:55,72 --> 0:24:59,72 a deleted page but very slowly through the process. 460 0:25:0,06 --> 0:25:5,58 So by minimizing how much we do in any 1 amount of workload and 461 0:25:5,64 --> 0:25:10,42 focusing it correctly, what we do is we expose the surface area 462 0:25:10,8 --> 0:25:11,78 of the problems. 463 0:25:12,32 --> 0:25:14,88 And then our job is Hansel and Gretel. 464 0:25:15,04 --> 0:25:19,92 We got to drop enough candy or bits on these leaf nodes so that 465 0:25:19,92 --> 0:25:23,56 the scans can detect that oh this changed while I was standing 466 0:25:23,56 --> 0:25:28,16 on it if it changed while I wasn't standing on it I don't care 467 0:25:28,38 --> 0:25:32,02 and then I can't fix I can do that small change by the way the 468 0:25:32,02 --> 0:25:35,78 next change it has to happen I have to wait to the transaction 469 0:25:36,26 --> 0:25:41,1 ID of all transactions passes in time. 470 0:25:41,4 --> 0:25:45,8 The same way deleting a page goes from half-dead to deleted and 471 0:25:45,8 --> 0:25:48,08 then from deleted to free space. 472 0:25:48,12 --> 0:25:52,12 It has to wait that transaction ID passing in order for the next 473 0:25:52,12 --> 0:25:52,96 pass to work. 474 0:25:52,96 --> 0:25:54,64 And these are the baby steps. 475 0:25:54,76 --> 0:25:58,94 And that's when I compare it to trying to do road work and put 476 0:25:58,94 --> 0:25:59,7 up a detour. 477 0:26:0,06 --> 0:26:2,8 If you want to put a detour on a road that's currently active 478 0:26:2,8 --> 0:26:6,1 without jamming the traffic, you have to let traffic still go 479 0:26:6,1 --> 0:26:7,66 through that's already on the road. 480 0:26:7,8 --> 0:26:11,44 You start backwards and put the detour flags, the furthest ones 481 0:26:11,44 --> 0:26:15,6 out 1st, and bring them in, and then you slowly start putting 482 0:26:15,6 --> 0:26:19,78 cones out there to force the traffic to take the new path. 483 0:26:20,18 --> 0:26:24,0 Anybody who's caught in front of you, it's only those cars that 484 0:26:24,0 --> 0:26:28,1 you're dropping cones on in front of that have to react to this 485 0:26:28,1 --> 0:26:28,6 merge. 486 0:26:28,94 --> 0:26:31,96 Everyone else who comes after everything's in place, they'll 487 0:26:31,96 --> 0:26:33,56 end up on the detour. 488 0:26:34,1 --> 0:26:37,58 And then once you lift the detour away, and that's vacuum's job, 489 0:26:37,66 --> 0:26:41,64 vacuum's job lifts all the cones away and it goes and picks up 490 0:26:41,64 --> 0:26:43,02 all the detour flags. 491 0:26:43,26 --> 0:26:46,92 And now whoever was on the detour flag finishes and the rest 492 0:26:46,92 --> 0:26:50,02 of the people finish on the normal highway, and from that point 493 0:26:50,02 --> 0:26:51,24 forward, nobody notices. 494 0:26:53,26 --> 0:26:58,7 Nik: So I know major hackers participated, like Peter Geoghegan and 495 0:26:58,7 --> 0:27:0,56 Robert Haas, others. 496 0:27:1,58 --> 0:27:5,78 And there was a big question, as I understand, how can you prove 497 0:27:5,8 --> 0:27:11,1 reliably some guarantees that everything will be correct always, 498 0:27:11,4 --> 0:27:13,64 even on all edge and corner cases? 499 0:27:14,66 --> 0:27:16,3 Was this answered or not? 500 0:27:17,26 --> 0:27:19,06 Kirk: We're still answering it, right? 501 0:27:19,06 --> 0:27:22,82 The answer is to keep the changes small and to make sure all 502 0:27:22,82 --> 0:27:25,9 the different code detects those changes they just identified 503 0:27:26,06 --> 0:27:31,16 a bug where in the middle of a B-tree merge somebody did an extra 504 0:27:31,16 --> 0:27:35,78 delete and the code coming in didn't detect the flag on the page 505 0:27:35,92 --> 0:27:37,34 that it was merged away. 506 0:27:37,64 --> 0:27:39,52 So it tried to process the page. 507 0:27:39,52 --> 0:27:44,0 But yes, there's pieces there we're still going to have to solve. 508 0:27:45,04 --> 0:27:49,48 Nik: What bothers me a lot is my AI is finding bugs in Postgres 509 0:27:49,48 --> 0:27:51,24 19 every day right now. 510 0:27:51,76 --> 0:27:53,04 And that's much easier. 511 0:27:53,2 --> 0:27:57,04 It's easier to find a bug than to prove reliably that there are 512 0:27:57,04 --> 0:27:57,76 no bugs. 513 0:27:59,32 --> 0:28:1,74 Kirk: Isn't it fundamentally impossible to prove a negative? 514 0:28:2,44 --> 0:28:6,54 Nik: Well, that's why those theoretical foundations exist. 515 0:28:7,12 --> 0:28:8,84 It's proving something like mathematics. 516 0:28:11,04 --> 0:28:14,62 This is the question they ask, like Peter Geoghegan, right? 517 0:28:16,02 --> 0:28:20,06 So some, like, ideally it would be, okay, Postgres follows just 518 0:28:20,06 --> 0:28:23,12 those articles from 80s, right? 519 0:28:23,12 --> 0:28:27,98 Let's, let's, Let's just, based on that, we know everything is 520 0:28:27,98 --> 0:28:28,28 fine. 521 0:28:28,28 --> 0:28:31,42 But there are nuances in current implementation, it's impossible 522 0:28:31,5 --> 0:28:33,9 to, there is no full match, obviously. 523 0:28:34,64 --> 0:28:35,14 There's, 524 0:28:35,86 --> 0:28:38,98 Kirk: we already just found a new limitation of our approach. 525 0:28:39,16 --> 0:28:42,88 It turns out there's 2 B-tree versions for file versioning. 526 0:28:43,18 --> 0:28:46,38 There's an older version that doesn't support the flags we're 527 0:28:46,38 --> 0:28:49,58 using and we can't do the merge on those B-trees, clearly. 528 0:28:49,84 --> 0:28:52,66 So that's going to be 1 of the flags that we have. 529 0:28:53,08 --> 0:28:57,42 The real question becomes simply, is the process sound? 530 0:28:57,74 --> 0:29:1,1 Does it set up the right locks and the pins in the right order? 531 0:29:1,1 --> 0:29:4,44 We've already got the logging working, where it's pushing out 532 0:29:4,44 --> 0:29:8,6 the WAL log and we can crash the server and recover it in the 533 0:29:8,6 --> 0:29:11,54 middle of the merge process and then the vacuum on the other 534 0:29:11,54 --> 0:29:13,36 end can finish cleaning it up. 535 0:29:13,38 --> 0:29:17,04 It's coming together and we're just past the halfway mark of 536 0:29:17,04 --> 0:29:18,3 Google Summer of Code. 537 0:29:18,34 --> 0:29:22,2 Now, that said, I'm not expecting this to get published by the 538 0:29:22,2 --> 0:29:25,94 time November rolls around and Google Summer of Code is done, 539 0:29:25,96 --> 0:29:26,28 okay? 540 0:29:26,28 --> 0:29:29,56 We know that this is gonna take longer, and part of it is, is 541 0:29:29,56 --> 0:29:35,72 Nik, I'm expecting you to tell your AI super agents to go out 542 0:29:35,72 --> 0:29:39,66 there and crush this code looking for the edge cases, right? 543 0:29:39,96 --> 0:29:45,06 And the other side of this equation is, we'll never know it's 544 0:29:45,06 --> 0:29:46,3 perfect, right? 545 0:29:46,3 --> 0:29:48,54 At some point we know it'll be well tested. 546 0:29:49,08 --> 0:29:52,76 Nik: Some proof is needed as I understand this is, this would 547 0:29:52,76 --> 0:29:57,94 be, I wanted to ask Salma, this is probably 1 of the hardest 548 0:29:57,94 --> 0:29:59,94 projects of Google Summer of Code. 549 0:30:0,06 --> 0:30:5,2 I participated in Google Summer of Code in 2006, exactly 20 years 550 0:30:5,2 --> 0:30:5,66 ago. 551 0:30:5,66 --> 0:30:7,66 And since then I kept an eye on it. 552 0:30:8,68 --> 0:30:11,54 And this looks like the most challenging in terms of how fundamental 553 0:30:11,64 --> 0:30:12,04 it is. 554 0:30:12,04 --> 0:30:13,76 How does it feel from your side? 555 0:30:13,82 --> 0:30:18,0 I know guys like Robert Haas and Peter Geoghegan proposed to 556 0:30:18,0 --> 0:30:19,36 change the project, right? 557 0:30:20,42 --> 0:30:21,8 Because of complexity. 558 0:30:22,58 --> 0:30:23,66 How does it look? 559 0:30:24,12 --> 0:30:25,22 Isn't that scary? 560 0:30:27,04 --> 0:30:30,92 Salma: Yes, but from the start Kirk told me this is already the 561 0:30:30,92 --> 0:30:35,92 problem and we will not have it done with the end of the Summer 562 0:30:35,92 --> 0:30:36,38 of Code. 563 0:30:36,38 --> 0:30:42,12 I mean it's it's hard but I truly I'm loving from doing it with 564 0:30:42,12 --> 0:30:42,62 Kirk. 565 0:30:42,84 --> 0:30:47,36 I want to see it make progress and hackers have their insights 566 0:30:47,5 --> 0:30:49,32 on it and their views. 567 0:30:49,54 --> 0:30:51,58 We do a lot of work on it. 568 0:30:51,58 --> 0:30:56,06 It's a little bit, yes, hard, not a little bit, but it's not 569 0:30:56,12 --> 0:30:58,42 that hard, like I'm scared of it. 570 0:30:59,76 --> 0:31:3,08 Kirk: Also, understand, I think I mentioned this with you, Nik, 571 0:31:3,08 --> 0:31:4,4 and Salma knows this. 572 0:31:4,46 --> 0:31:7,68 What's our definition of success when it comes to Google's Summer 573 0:31:7,68 --> 0:31:8,82 of Code being done? 574 0:31:8,88 --> 0:31:12,6 Is it that this gets published, accepted, and there's confetti 575 0:31:12,72 --> 0:31:13,5 in the streets? 576 0:31:13,86 --> 0:31:14,36 No. 577 0:31:15,42 --> 0:31:20,48 My definition of success is we've defined a language by which 578 0:31:20,48 --> 0:31:24,42 we can now talk about making this happen in the future. 579 0:31:24,76 --> 0:31:28,92 Meaning we've identified the core issues, we can talk about it. 580 0:31:28,94 --> 0:31:34,44 Next, we started quantifying the impact on performance from the 581 0:31:34,44 --> 0:31:38,82 standpoint of how much is it really slowing down every B-tree 582 0:31:38,86 --> 0:31:39,36 search? 583 0:31:39,44 --> 0:31:43,22 Look, if it's gonna cost every B-tree search 20% efficiency, 584 0:31:43,68 --> 0:31:47,5 like no, let's not do this to answer Michael's question, right? 585 0:31:47,5 --> 0:31:52,16 If this isn't a very small delta hit on searching, then it's 586 0:31:52,2 --> 0:31:53,12 not worth it. 587 0:31:53,12 --> 0:31:57,1 But on the other hand, if we don't develop the language, we don't 588 0:31:57,1 --> 0:32:1,12 develop the protocols for testing it for performance, We don't 589 0:32:1,12 --> 0:32:5,58 develop the process by which people can discuss it and review 590 0:32:5,64 --> 0:32:6,14 it. 591 0:32:6,36 --> 0:32:10,24 If we get all that done, I think this is a huge successful Google 592 0:32:10,24 --> 0:32:12,1 Summer of Code project in my opinion. 593 0:32:12,34 --> 0:32:16,1 And if we have that plus a rough working prototype, Hallelujah. 594 0:32:16,46 --> 0:32:20,86 That's something we can carry forward and maybe it falls on to 595 0:32:20,86 --> 0:32:24,28 much more experienced people than us, just to make sure of the 596 0:32:24,28 --> 0:32:24,78 correctness. 597 0:32:25,2 --> 0:32:30,4 But I'll be honest, that was the limited version that kept me 598 0:32:30,4 --> 0:32:30,9 motivated. 599 0:32:32,64 --> 0:32:36,88 As we've made the progress we've made, my understanding of these 600 0:32:36,88 --> 0:32:40,42 B-trees and why they made the decisions they made has skyrocketed. 601 0:32:41,6 --> 0:32:46,06 And I'm becoming more and more confident in our approach because 602 0:32:46,06 --> 0:32:50,5 I'm starting to really understand what it is we have to do, what 603 0:32:50,5 --> 0:32:52,0 those breadcrumbs are. 604 0:32:52,3 --> 0:32:54,36 Now, is there a lot of testing involved? 605 0:32:54,64 --> 0:32:55,14 Absolutely. 606 0:32:55,64 --> 0:32:59,82 Could this affect third-party tools that work on indexes? 607 0:33:0,3 --> 0:33:0,78 Absolutely. 608 0:33:0,78 --> 0:33:4,9 If they don't know what a merged away page is or a merged page, 609 0:33:4,9 --> 0:33:7,8 and they're not looking for that stuff, they could make mistakes. 610 0:33:8,4 --> 0:33:8,9 Absolutely. 611 0:33:9,06 --> 0:33:11,08 We don't want that to happen either. 612 0:33:11,2 --> 0:33:14,66 But on the flip side, Core never worries about the extensions 613 0:33:14,72 --> 0:33:15,46 per se. 614 0:33:15,56 --> 0:33:18,28 It's the extensions job when that version comes out to be up 615 0:33:18,28 --> 0:33:18,9 to date. 616 0:33:18,9 --> 0:33:22,36 I don't want to worry too much about it, but I honestly think 617 0:33:22,36 --> 0:33:23,08 we're close. 618 0:33:23,24 --> 0:33:25,54 We have the core concepts in place. 619 0:33:25,8 --> 0:33:27,54 That's the part I'm impressed with. 620 0:33:29,2 --> 0:33:29,7 Also, 621 0:33:30,4 --> 0:33:33,52 Salma: at the very beginning, Andreas turned to me in our 1st 622 0:33:33,52 --> 0:33:38,68 meeting that it's not only about completing this project, it's 623 0:33:38,68 --> 0:33:42,66 about getting new people to contribute to Postgres. 624 0:33:46,0 --> 0:33:49,7 So I will just take care of Postgres, so they contribute to 625 0:33:49,7 --> 0:33:50,4 new steppasses. 626 0:33:51,46 --> 0:33:52,48 Nik: Yeah that's great, that's great. 627 0:33:53,3 --> 0:33:55,08 So how many months left? 628 0:33:55,52 --> 0:33:58,14 You mentioned November, right? 629 0:33:58,7 --> 0:34:1,1 Salma: I think it's the beginning of November. 630 0:34:1,64 --> 0:34:6,3 Nik: Since it's already not much time left, what should we expect 631 0:34:6,3 --> 0:34:8,8 in terms of prototyping this thing? 632 0:34:9,48 --> 0:34:13,58 Kirk: So the way that we did this to get it working was, as we 633 0:34:13,58 --> 0:34:17,32 read the next page, if we detected we were stepping in the middle 634 0:34:17,32 --> 0:34:20,82 of a quagmire, meaning the change happened under our feet, we 635 0:34:20,82 --> 0:34:24,14 could just reach back in memory, look at the records we just 636 0:34:24,14 --> 0:34:26,2 added, and apply a fix up algorithm. 637 0:34:26,2 --> 0:34:27,6 That was completely working. 638 0:34:27,98 --> 0:34:30,06 And we had the stuff working inside a vacuum. 639 0:34:30,06 --> 0:34:31,16 So that was good. 640 0:34:31,22 --> 0:34:35,66 What we found is parallel scans would not work this way because 641 0:34:35,66 --> 0:34:39,68 you'd have to pass all of the TIDs from the previous read to 642 0:34:39,68 --> 0:34:40,74 a different thread. 643 0:34:41,06 --> 0:34:45,04 And all of that messaging would just destroy the parallel thread. 644 0:34:45,06 --> 0:34:48,54 So what we've done is we backed up the truck and we reanalyzed 645 0:34:49,3 --> 0:34:53,94 the situation and we realized all we really need is the 1 pivot 646 0:34:53,94 --> 0:34:54,44 tuple. 647 0:34:54,96 --> 0:34:59,54 So on a forward scan we need the last tuple on the page that 648 0:34:59,54 --> 0:35:4,0 we're going to merge away and on the backward scan we need the 649 0:35:4,0 --> 0:35:9,6 1st tuple of that page so we know what to pre-process if we reread 650 0:35:9,6 --> 0:35:10,32 that page. 651 0:35:10,68 --> 0:35:12,28 So let's do backwards scan. 652 0:35:12,34 --> 0:35:16,52 I just did this page, our right page, before the merge happened 653 0:35:16,82 --> 0:35:20,38 and I know the 1st tuple and even in a parallel scan this is 654 0:35:20,38 --> 0:35:20,88 great. 655 0:35:20,98 --> 0:35:25,88 I go and I read the next page backwards and it was merged away. 656 0:35:26,38 --> 0:35:30,92 That flag tells me I have to go back and reread the merged page 657 0:35:30,92 --> 0:35:37,66 into but I only process the records that are less than that current 658 0:35:38,16 --> 0:35:40,7 record that we saved from the previous page. 659 0:35:40,84 --> 0:35:43,68 Those are all the ones that were inserted because we only move 660 0:35:43,68 --> 0:35:46,16 the keys to the right in the tree. 661 0:35:46,3 --> 0:35:48,24 And they're sorted, which is beautiful. 662 0:35:48,24 --> 0:35:50,9 And they're sorted uniquely because the last sort key is the 663 0:35:50,9 --> 0:35:53,18 TID, the row ID in the table. 664 0:35:53,3 --> 0:35:59,02 So now we pull that up, we reread that page, we strip off everything 665 0:35:59,02 --> 0:36:2,3 we don't want to read again that we already read and we just 666 0:36:2,3 --> 0:36:4,14 add the remaining records in. 667 0:36:4,14 --> 0:36:7,46 So now with this change we have to go back and rewrite our existing 668 0:36:7,6 --> 0:36:10,96 routine but now that means we now have parallel scan will be 669 0:36:10,96 --> 0:36:14,44 working shortly and I'm thinking within a week or so parallel 670 0:36:14,44 --> 0:36:18,42 scan will be working and the normal scan will be working using 671 0:36:18,42 --> 0:36:23,9 this new logic, which means at this point we have the WAL logging, 672 0:36:24,22 --> 0:36:28,52 parallel scan, regular scan, and the core handling. 673 0:36:28,74 --> 0:36:32,7 Now we have to find the nuanced places, the delete records, update 674 0:36:32,7 --> 0:36:35,74 records, things like this that touch these pages and we have 675 0:36:35,74 --> 0:36:38,98 to find the other touch points to make sure that they're honoring 676 0:36:39,4 --> 0:36:42,54 the new flags we've created in these things. 677 0:36:42,72 --> 0:36:46,4 Once we're done with that it's looking good but in the next week 678 0:36:46,4 --> 0:36:50,38 or 2, I believe Salma, and I'm speaking for you, Salma, you can 679 0:36:50,38 --> 0:36:50,86 chime in. 680 0:36:50,86 --> 0:36:53,44 How long do you think it's going to take you to rewrite all of 681 0:36:53,44 --> 0:36:55,04 that code you spent months on? 682 0:36:55,04 --> 0:36:56,92 I'm giving you 2 weeks so far. 683 0:36:56,92 --> 0:36:58,84 Go ahead, tell me how wrong I am. 684 0:36:59,2 --> 0:37:2,24 Salma: I have done before 2 weeks, but yes. 685 0:37:2,58 --> 0:37:4,54 Kirk: Now you know why I love her 686 0:37:6,9 --> 0:37:11,8 Nik: so you mentioned with WAL logging what's left then WAL 687 0:37:11,8 --> 0:37:13,92 is very difficult 688 0:37:15,3 --> 0:37:18,16 Kirk: It's it's it was only difficult correct me if I'm wrong 689 0:37:18,16 --> 0:37:22,76 the difficult part of WAL was 1st off It's a complete mind change 690 0:37:22,96 --> 0:37:26,26 in the problem You're solving because now you're working on playback 691 0:37:26,88 --> 0:37:31,26 and in capturing stuff and then the testing required that Salma 692 0:37:31,26 --> 0:37:36,18 had to learn how to set up a WAL transfer and a recovery process 693 0:37:36,18 --> 0:37:37,12 and do all that. 694 0:37:37,12 --> 0:37:40,02 Other than that, what do you think on the WAL logging stuff 695 0:37:40,02 --> 0:37:41,46 in making sure that it worked? 696 0:37:42,36 --> 0:37:44,04 Salma, what was hard? 697 0:37:44,64 --> 0:37:49,68 Salma: Nothing in particular, but we had to add a new nbtree 698 0:37:49,68 --> 0:37:55,32 resource manager, because the nbtree only had 1 slot 699 0:37:55,32 --> 0:38:3,22 left, the left slot, and we needed 3 slots for logging, 1 slot 700 0:38:3,22 --> 0:38:9,72 for logging the merge, and 2 for when vacuum cleaning the merged 701 0:38:9,72 --> 0:38:12,08 away page and when cleaning the merged page. 702 0:38:12,12 --> 0:38:13,34 So we needed 3. 703 0:38:13,7 --> 0:38:18,54 1st, we did it like with multiplexing this only slot, but it 704 0:38:18,54 --> 0:38:19,44 wasn't clean. 705 0:38:19,82 --> 0:38:25,6 So we asked on Discord that I can do another, a new resource 706 0:38:25,6 --> 0:38:26,1 manager. 707 0:38:26,84 --> 0:38:31,32 And we did a new nbtree2, like following heap2, which is 708 0:38:31,32 --> 0:38:32,3 already introduced. 709 0:38:32,78 --> 0:38:35,88 Kirk: And for those who are not up on that new resource manager, 710 0:38:36,04 --> 0:38:38,5 effectively, WAL is blocks of data. 711 0:38:38,56 --> 0:38:41,72 When you read in a WAL block, there's a bunch of flags that 712 0:38:41,72 --> 0:38:43,82 tell you how to interpret that data. 713 0:38:44,04 --> 0:38:48,14 That's the resource manager layer of it, is it looks and says, 714 0:38:48,14 --> 0:38:50,12 oh, this is a B-tree WAL. 715 0:38:50,18 --> 0:38:55,32 In this case, it's a B-tree merged away WAL record right so 716 0:38:55,32 --> 0:38:58,92 it has to know how to read this in and process it and we had 717 0:38:58,92 --> 0:39:2,34 to teach it that by adding our new types in. 718 0:39:2,58 --> 0:39:6,9 Nik: I somehow haven't followed the project recently and miss 719 0:39:6,9 --> 0:39:9,02 that WAL is already being in work. 720 0:39:9,34 --> 0:39:9,84 Yeah. 721 0:39:10,76 --> 0:39:11,5 I'm impressed. 722 0:39:12,38 --> 0:39:15,78 So if everything goes well, what will be left? 723 0:39:16,64 --> 0:39:21,22 Besides a theoretical question that everything is reliable, which 724 0:39:21,22 --> 0:39:22,66 is a tiny question. 725 0:39:22,66 --> 0:39:23,82 Michael: Quite a big deal. 726 0:39:23,94 --> 0:39:24,44 Nik: Yeah. 727 0:39:25,46 --> 0:39:25,84 Yeah. 728 0:39:25,84 --> 0:39:26,78 What will be left? 729 0:39:26,98 --> 0:39:29,08 Kirk: Besides the question of, is it correct? 730 0:39:29,54 --> 0:39:29,92 What will 731 0:39:29,92 --> 0:39:30,6 Nik: be left? 732 0:39:30,82 --> 0:39:33,84 Kirk: Honestly, it's the things we just talked about, right? 733 0:39:33,84 --> 0:39:37,5 It is literally the parallel scan and getting that in there, 734 0:39:37,5 --> 0:39:40,18 plus finding other touch points or edge points. 735 0:39:40,64 --> 0:39:45,18 And then in my book, the next layer is performance testing and 736 0:39:45,18 --> 0:39:47,34 having you send your guys at it? 737 0:39:47,86 --> 0:39:52,64 Nik: Actually, now, just once we talk through all this, I have 738 0:39:52,64 --> 0:39:58,14 a great idea to point my new harness to find those bugs. 739 0:39:58,14 --> 0:40:0,64 Like, I'm pretty sure we will find some. 740 0:40:1,32 --> 0:40:5,26 But it won't prove the theoretical correctness of everything, but 741 0:40:5,92 --> 0:40:7,44 it may be to help a little bit. 742 0:40:7,44 --> 0:40:12,18 My question is what will be left to feel the prototype complete? 743 0:40:13,2 --> 0:40:15,34 If all is there, what's what else left? 744 0:40:15,34 --> 0:40:15,84 Nothing? 745 0:40:16,4 --> 0:40:19,54 Kirk: Well, this last 2 week cycle is going to be most of it. 746 0:40:19,54 --> 0:40:22,16 And then getting people to give us feedback and testing. 747 0:40:22,54 --> 0:40:25,62 That's what I want this broadcast to talk about. 748 0:40:25,76 --> 0:40:30,34 Because this is, I think this is the UUIDv7 of Google Summer 749 0:40:30,34 --> 0:40:31,04 of Code. 750 0:40:31,24 --> 0:40:34,42 In PG18, that was the 1 feature everyone understood. 751 0:40:34,92 --> 0:40:35,46 It's all. 752 0:40:35,46 --> 0:40:36,96 Nik: That feature was simple. 753 0:40:37,12 --> 0:40:37,54 Kirk: It was. 754 0:40:37,54 --> 0:40:40,58 Nik: That feature was simple, and you need to make efforts to 755 0:40:40,58 --> 0:40:41,26 use it. 756 0:40:41,32 --> 0:40:45,54 Unlike this feature, which is hard and everyone is supposed to 757 0:40:45,78 --> 0:40:47,32 benefit from it by default. 758 0:40:48,34 --> 0:40:52,08 So I understand why you say this, but this is for me, it's quite 759 0:40:52,08 --> 0:40:52,58 opposite. 760 0:40:53,42 --> 0:40:57,74 And this podcast I wanted to do because this is the most impressive 761 0:40:57,84 --> 0:41:1,4 Google Summer of Code I saw in 20 years, as I feel it. 762 0:41:1,4 --> 0:41:8,3 It's like taking articles from 80s, using AI, attacking a hard 763 0:41:8,3 --> 0:41:8,8 problem. 764 0:41:9,8 --> 0:41:11,46 And it definitely deserves attention. 765 0:41:11,98 --> 0:41:15,12 I definitely will point my harness to find bugs. 766 0:41:16,16 --> 0:41:18,84 Kirk: We need people to actually be looking at this and giving 767 0:41:18,84 --> 0:41:19,5 us feedback. 768 0:41:20,98 --> 0:41:23,94 Michael: Yeah, I actually wanted to commend you for the amount 769 0:41:23,94 --> 0:41:25,94 of interest you'd already got. 770 0:41:26,4 --> 0:41:30,66 The level of detail that people have given comments and Advice 771 0:41:30,78 --> 0:41:34,9 already from senior people and people that have worked on B-tree 772 0:41:34,9 --> 0:41:39,64 deletion in the past as well It's been incredible, but I think 773 0:41:39,64 --> 0:41:43,58 I'm a bit confused because I think some of their feedback I can't 774 0:41:43,58 --> 0:41:46,82 tell if it's been addressed yet So I saw 1 comment from Peter 775 0:41:46,82 --> 0:41:50,44 saying what it boils down to is the same TID must never exist 776 0:41:50,44 --> 0:41:52,08 in any 2 index tuples that 777 0:41:52,08 --> 0:41:56,02 Kirk: was the ghost record That was the issue with ghost records. 778 0:41:56,64 --> 0:41:58,68 Michael: Yeah, so that's no longer true. 779 0:41:58,78 --> 0:42:0,28 Kirk: Yeah, we don't do it that way anymore. 780 0:42:0,28 --> 0:42:1,28 We fixed that 781 0:42:1,88 --> 0:42:2,38 Michael: Yes, 782 0:42:2,52 --> 0:42:6,0 Kirk: it became an invariant and they didn't even appreciate 783 0:42:6,04 --> 0:42:9,16 the fact that nobody's supposed to look at those records except 784 0:42:9,28 --> 0:42:12,28 the people that are getting their toes stepped on in the scan. 785 0:42:12,5 --> 0:42:15,54 Everyone else further from the scan would never look at them. 786 0:42:16,12 --> 0:42:21,04 And they blasted it, which is okay, because the technique that 787 0:42:21,04 --> 0:42:24,1 Salma came up with is actually more efficient. 788 0:42:25,16 --> 0:42:27,98 So I appreciate the feedback, but keep going. 789 0:42:27,98 --> 0:42:30,02 If there's other feedback you'd like to hear, but for the most 790 0:42:30,02 --> 0:42:31,94 part, most of these things have been answered. 791 0:42:32,84 --> 0:42:35,58 Michael: Yeah I think that's the kind of thing though that does 792 0:42:35,58 --> 0:42:39,64 help not prove that an issue can't exist but as soon as you allow 793 0:42:39,64 --> 0:42:43,62 for 2 TIDs to point to this like as soon as you allow for 2 records 794 0:42:43,62 --> 0:42:46,96 to point to the same TID you introduce the possibility of corruption. 795 0:42:46,96 --> 0:42:50,14 So I do think that's the kind of design decision that helps avoid 796 0:42:50,32 --> 0:42:52,94 whole categories of bug, which is quite nice. 797 0:42:53,32 --> 0:42:55,74 Kirk: Salma, did you have to modify amcheck? 798 0:42:59,16 --> 0:42:59,66 Salma: Yes. 799 0:43:0,28 --> 0:43:4,92 Yeah, we have to let it know about the Merged Away page And most 800 0:43:4,92 --> 0:43:8,86 of a lot of a lot for a lot of parts of the code it was part 801 0:43:8,86 --> 0:43:9,02 of 802 0:43:9,02 --> 0:43:9,68 Nik: the core 803 0:43:9,96 --> 0:43:13,58 Salma: read the high key from pages and the few and 804 0:43:14,18 --> 0:43:15,6 Kirk: We don't have any tension 805 0:43:16,24 --> 0:43:18,82 Salma: Yeah, we don't pay attention to the rest of a page So 806 0:43:18,82 --> 0:43:22,7 we have to tell a lot of parts of the code that they just skip 807 0:43:22,7 --> 0:43:27,04 the merged away page the same way it does with the deleted 808 0:43:27,04 --> 0:43:27,44 page. 809 0:43:27,44 --> 0:43:28,48 Kirk: Or half-dead. 810 0:43:28,58 --> 0:43:29,08 Yep. 811 0:43:29,54 --> 0:43:30,04 Yes. 812 0:43:30,72 --> 0:43:31,22 Perfect. 813 0:43:31,24 --> 0:43:32,82 But no, back to your point. 814 0:43:32,86 --> 0:43:36,64 So not, not only did we get feedback, we had to fix amcheck. 815 0:43:36,82 --> 0:43:40,58 But then after she fixed it, by teaching it how to recognize 816 0:43:40,64 --> 0:43:46,26 our pages But then it came back and said no you cheated You didn't 817 0:43:46,26 --> 0:43:49,6 teach it how to inspect your pages to make sure they're actually 818 0:43:49,6 --> 0:43:50,1 correct. 819 0:43:50,66 --> 0:43:53,0 And so Salma went and worked on that. 820 0:43:53,0 --> 0:43:55,02 And so let's be clear. 821 0:43:55,08 --> 0:43:58,9 We are standing on the shoulders of everybody in the Postgres 822 0:43:58,94 --> 0:43:59,44 community. 823 0:43:59,76 --> 0:44:2,72 And this kind of feedback is helpful for us. 824 0:44:2,72 --> 0:44:5,42 Yeah, we bit off way more than we could chew by ourself. 825 0:44:5,8 --> 0:44:9,24 And if we were alone on a stranded on a desert island with a 826 0:44:9,24 --> 0:44:13,46 Cray supercomputer and an AI, I don't think we're gonna get near 827 0:44:13,46 --> 0:44:17,52 the best result we get by interfacing and working with the community. 828 0:44:18,74 --> 0:44:20,04 Michael: Nice, yeah, agreed. 829 0:44:20,38 --> 0:44:24,14 Salma: We just got an email, someone reviewed and pointed out 830 0:44:24,14 --> 0:44:25,52 a bug we need to fix. 831 0:44:25,52 --> 0:44:30,04 And this is the kind, also the kind of review we need because 832 0:44:30,32 --> 0:44:34,92 It pointed out to some part of the vacuum that I didn't know 833 0:44:34,92 --> 0:44:35,58 it existed. 834 0:44:35,82 --> 0:44:39,72 So yeah, it helps a lot to point out to this. 835 0:44:40,34 --> 0:44:44,28 Nik: I would also check maybe pgstattuple extension and 836 0:44:44,64 --> 0:44:45,6 some other extensions. 837 0:44:46,5 --> 0:44:50,52 Yeah, I'm definitely pointing my harness after I'm done with 838 0:44:51,14 --> 0:44:51,92 my bugs. 839 0:44:52,54 --> 0:44:54,7 If I'm done with bugs, because they keep... 840 0:44:55,64 --> 0:44:58,48 Kirk: Squeeze this in between the big runs and the bugs. 841 0:44:59,18 --> 0:44:59,86 Nik: Sounds good. 842 0:44:59,86 --> 0:45:0,88 Yeah, Sounds good. 843 0:45:1,62 --> 0:45:2,12 Great. 844 0:45:2,96 --> 0:45:4,9 I'm very impressed with progress. 845 0:45:5,14 --> 0:45:10,14 I was expecting like attacking only part of it, but it's obviously 846 0:45:10,16 --> 0:45:11,62 attack of the whole thing. 847 0:45:11,74 --> 0:45:13,68 It's impressive, especially. 848 0:45:14,44 --> 0:45:18,58 Kirk: So Michael, with all that, are you as nervous now as you 849 0:45:18,58 --> 0:45:19,54 were at the beginning? 850 0:45:21,06 --> 0:45:23,82 Michael: I think the closer you 2 get to something committable, 851 0:45:24,0 --> 0:45:25,5 the more nervous I'll be. 852 0:45:26,32 --> 0:45:31,1 So don't treat my nervousness as any kind of sign of your progress. 853 0:45:31,16 --> 0:45:36,18 But I would agree that this is a really impressively ambitious 854 0:45:36,26 --> 0:45:40,32 project, and if it's about learning and inspiring people to become 855 0:45:40,32 --> 0:45:44,38 committers or contributors in the future, I think you've succeeded. 856 0:45:44,38 --> 0:45:45,8 And we wanted to talk about it, right? 857 0:45:45,8 --> 0:45:48,34 It's an interesting enough piece that we wanted to talk about 858 0:45:48,34 --> 0:45:52,58 it here so I'm really pleased that you're attacking this I think 859 0:45:52,58 --> 0:45:55,68 you've set expectations well on the chance of getting anything 860 0:45:55,68 --> 0:45:59,42 committed here but it sounds like you 2 are getting dangerously 861 0:45:59,48 --> 0:46:3,98 close so I should be worried but that that's a compliment right 862 0:46:3,98 --> 0:46:8,0 the fact that I I was worried when senior hackers were making 863 0:46:8,0 --> 0:46:10,74 changes to B-tree code in 13, 14. 864 0:46:11,2 --> 0:46:15,28 Turned out some of it was like the best work I've ever seen. 865 0:46:15,42 --> 0:46:18,22 Nik: 14.0, you remember this? 866 0:46:19,12 --> 0:46:21,96 Michael: I remember the, yeah, but that was slightly different 867 0:46:21,96 --> 0:46:22,58 code, right? 868 0:46:22,58 --> 0:46:24,94 Like the reindex concurrently stuff, yeah. 869 0:46:25,14 --> 0:46:30,36 But the bottom-up deletion, that was Peter Geoghegan as well, that 870 0:46:30,36 --> 0:46:33,48 is another attempt at trying to avoid the problem in the 1st 871 0:46:33,48 --> 0:46:37,86 place of B-tree bloat but without having to worry about merging 872 0:46:38,14 --> 0:46:41,16 so it was trying to avoid splitting it's the from the other side 873 0:46:41,16 --> 0:46:44,06 of it I was so impressed by that work I'm really impressed that 874 0:46:44,06 --> 0:46:47,6 there weren't major issues with it So it's this kind of super 875 0:46:47,6 --> 0:46:49,4 scary stuff that touches everything. 876 0:46:49,44 --> 0:46:52,6 They're the changes I like the most because everybody benefits 877 0:46:52,6 --> 0:46:54,02 without having to do anything. 878 0:46:54,28 --> 0:46:57,88 But they're the changes I find most scary because everybody's 879 0:46:58,0 --> 0:46:59,94 affected without having to do anything. 880 0:47:0,16 --> 0:47:3,94 So yeah, it's impressive that you've even taken this on, and 881 0:47:3,94 --> 0:47:6,3 I'm really impressed with your progress and the feedback you've 882 0:47:6,3 --> 0:47:6,8 got. 883 0:47:6,88 --> 0:47:9,84 And hopefully, Salma, the whole point was to get you interested 884 0:47:9,84 --> 0:47:12,28 in contributing to Postgres, and it sounds like you are. 885 0:47:13,04 --> 0:47:13,54 Thanks. 886 0:47:14,06 --> 0:47:16,38 Nik: I know about other plans, right? 887 0:47:16,38 --> 0:47:19,82 Not only about this complex project, but maybe smaller features 888 0:47:19,82 --> 0:47:21,36 and fixes and so on. 889 0:47:21,36 --> 0:47:22,06 That's great. 890 0:47:22,12 --> 0:47:25,76 Kirk: Salma's kind of found a sponsor and she's in the background 891 0:47:25,76 --> 0:47:30,16 also picking up some of our LSN drop table logging and some of 892 0:47:30,16 --> 0:47:33,84 the other hacker stuff that Andrey and Nik and I are doing on 893 0:47:33,84 --> 0:47:34,54 the side. 894 0:47:34,54 --> 0:47:38,08 And she's gonna help push some of those through because we just 895 0:47:38,08 --> 0:47:41,82 keep coming up with new ideas and we don't babysit the old ones 896 0:47:41,82 --> 0:47:43,04 through all the commitfest, right? 897 0:47:43,04 --> 0:47:44,54 And there's a lot of work there. 898 0:47:45,06 --> 0:47:48,48 So she's picking up some extra work and skills doing that as 899 0:47:48,48 --> 0:47:48,98 well. 900 0:47:49,4 --> 0:47:54,24 Nik: It's time to start joining our sessions on hacking sessions 901 0:47:54,24 --> 0:47:54,9 on YouTube. 902 0:47:55,24 --> 0:47:56,82 Kirk: Oh, look at that. 903 0:47:57,26 --> 0:47:57,62 Nik: Yeah. 904 0:47:57,62 --> 0:48:2,8 So to wrap up, let's maybe repeat what people can do to help. 905 0:48:3,26 --> 0:48:6,58 Just review the patch, test it, find bugs and so on, right? 906 0:48:6,58 --> 0:48:7,62 And spread the word. 907 0:48:7,7 --> 0:48:10,68 Also, we have a playground in terms of visualization. 908 0:48:11,68 --> 0:48:13,76 To play with it a little bit, yeah. 909 0:48:13,84 --> 0:48:15,42 Kirk: So yeah, check out the visualizations. 910 0:48:15,72 --> 0:48:17,04 They're still accurate for now. 911 0:48:17,04 --> 0:48:19,64 I'll update them in 2 weeks when we get the patch. 912 0:48:19,64 --> 0:48:23,94 But at least go through and do a plus 1 if you like the idea 913 0:48:23,94 --> 0:48:25,76 that we're working on it, Michael. 914 0:48:26,0 --> 0:48:26,98 Anybody else? 915 0:48:27,04 --> 0:48:31,96 Okay, plus 1 these because the more replies people get and see 916 0:48:31,96 --> 0:48:35,58 in the hackers email chain, the more likely they're going to 917 0:48:35,58 --> 0:48:38,04 crack it open and take a look at what's going on. 918 0:48:38,08 --> 0:48:39,56 And we want that attention. 919 0:48:39,96 --> 0:48:44,48 Again, we admit it may not get committed, but the flip side is 920 0:48:44,48 --> 0:48:47,84 that the more people that look at it and realize that this is 921 0:48:47,84 --> 0:48:50,28 not crazy, the more it helps us. 922 0:48:50,28 --> 0:48:51,78 So those are the key things we need. 923 0:48:51,78 --> 0:48:55,32 We just need that extra attention, whatever anyone else there 924 0:48:55,32 --> 0:48:56,02 can do. 925 0:48:56,04 --> 0:48:59,96 And again, I want to say kudos to Google Summer of Code, because 926 0:49:0,06 --> 0:49:2,5 This wouldn't have happened if it wasn't for that. 927 0:49:2,5 --> 0:49:3,94 It's my 1st time mentoring. 928 0:49:4,6 --> 0:49:7,9 I feel bad for Salma, but I did my best. 929 0:49:8,56 --> 0:49:10,3 Salma: No, you are a great mentor. 930 0:49:11,0 --> 0:49:11,5 Nik: Great. 931 0:49:11,68 --> 0:49:12,56 Thank you for coming. 932 0:49:12,56 --> 0:49:13,48 It was interesting. 933 0:49:13,62 --> 0:49:15,82 It's a very challenging and super interesting project. 934 0:49:15,82 --> 0:49:17,04 I'm rooting for it. 935 0:49:17,38 --> 0:49:19,58 And I'm going to help when I can. 936 0:49:20,38 --> 0:49:21,0 Thank you. 937 0:49:21,0 --> 0:49:22,22 Kirk: Awesome, thank you guys. 938 0:49:22,36 --> 0:49:23,26 Have a good day. 939 0:49:23,86 --> 0:49:24,24 Salma: Thank you. 940 0:49:24,24 --> 0:49:25,26 Michael: Likewise, congrats. 941 0:49:25,44 --> 0:49:26,6 Good to meet you both.