1 00:00:00,046 --> 00:00:02,926 Michael: Hello, and welcome to Postgres FM a week. 2 00:00:02,926 --> 00:00:03,616 Share about all things. 3 00:00:03,646 --> 00:00:09,946 Postgres girl, I am Michael founder of PG mustard, and this is my co-host ly founder of Postgres AI. 4 00:00:10,306 --> 00:00:12,416 Hey ly, what are we discussing today? 5 00:00:12,866 --> 00:00:17,576 Nikolay: Hi, Michael, we are reacting to requests and we are discussing vacuum. 6 00:00:17,709 --> 00:00:20,439 Maybe we, we are only starting to discuss vacuum, 7 00:00:20,529 --> 00:00:22,549 Michael: yeah, absolutely deep, deep topic. 8 00:00:22,601 --> 00:00:25,931 But yes, this was, as you say, I think this was one of the first ones. 9 00:00:25,931 --> 00:00:33,371 Somebody, uh, tweeted us once, once we said we were open to talking about things, people wanted to hear a vacuum 10 00:00:33,371 --> 00:00:35,921 was a, is always a topic that people want to talk about. 11 00:00:36,281 --> 00:00:41,911 It seems like it's something that people don't necessarily know about when they first encounter Postgres, 12 00:00:42,181 --> 00:00:44,221 but they quickly learn about one way or the other. 13 00:00:44,561 --> 00:00:46,181 And as you get more and more advanced, I. 14 00:00:46,976 --> 00:00:51,726 There still seem to be things that people can learn about vacuum, even people with many years of experience. 15 00:00:51,866 --> 00:00:52,646 Nikolay: Right, right. 16 00:00:52,656 --> 00:00:57,401 We had a couple of requests and I wanted to thank everyone who provides feedback to us. 17 00:00:57,401 --> 00:01:01,181 So we see it almost all every day already, especially on Twitter. 18 00:01:01,181 --> 00:01:05,691 And it's so good to, to read it and to see what people think about our show. 19 00:01:06,166 --> 00:01:13,966 And also as usual, I ask everyone to like in the place they listen or watch, we have a video version, which is uncut by 20 00:01:13,966 --> 00:01:21,206 the way, longer, a little bit on YouTube postgres.tv is a short web address, but of course it's available on iTunes and so on. 21 00:01:21,206 --> 00:01:25,161 And I encourage everyone to like this show or this episode right now. 22 00:01:25,221 --> 00:01:33,106 And also please share in, social networks you use or working groups where you discuss Postgres or engineering of everything. 23 00:01:33,226 --> 00:01:34,366 Thank you so much, everyone. 24 00:01:34,816 --> 00:01:42,069 So, speaking of Postgres TV channel, we also have A great guests coming almost every week. 25 00:01:42,279 --> 00:01:43,879 It's called open talks. 26 00:01:43,999 --> 00:01:51,199 The idea is we invite people who, presented some interesting talk at some conference, but recording was not done. 27 00:01:51,739 --> 00:01:56,809 And one of the such talks recently was Hannah crossing X Skype. 28 00:01:57,019 --> 00:02:00,079 He created a lot of, of positive related stuff at Skype. 29 00:02:00,139 --> 00:02:01,399 And, he's now at Google. 30 00:02:01,987 --> 00:02:06,047 and the talk was titled, do you vacuum every day? 31 00:02:06,276 --> 00:02:07,176 It was great. 32 00:02:07,296 --> 00:02:14,360 Like, it's very great deep at the, and at the same time, very simple material, everyone can watch it and understand a lot of stuff. 33 00:02:14,365 --> 00:02:21,550 So I, I, again, like encourage a hundred percent everyone should watches so it's, it's directly related to our today topic. 34 00:02:21,691 --> 00:02:22,286 Michael: I watched that. 35 00:02:22,286 --> 00:02:23,486 I thought it was fantastic. 36 00:02:23,536 --> 00:02:27,941 He managed to keep it, as you say, kept it simple, but I definitely still learnt a few things during it. 37 00:02:28,061 --> 00:02:30,401 I think it was about an hour, maybe a little bit longer. 38 00:02:30,431 --> 00:02:33,721 So it is definitely a deep dive, but, definitely were worth watch. 39 00:02:33,767 --> 00:02:39,938 Nikolay: Yeah, we don't, we don't limit our speakers, with timing unlike at conference, when we have very strict constraints. 40 00:02:39,938 --> 00:02:45,278 So we can spend a couple of hours, definitely if you want, and we have material and desired. 41 00:02:45,308 --> 00:02:45,728 Definitely. 42 00:02:45,728 --> 00:02:53,763 So, but our show is roughly 30 minutes because, as usual, I want to say big hello of those who are running 43 00:02:53,763 --> 00:02:57,783 or riding bicycle, or especially walking their dogs. 44 00:02:57,813 --> 00:02:58,953 I know some people. 45 00:02:59,268 --> 00:03:03,442 from feedback, I've learned it and I, I think docs must love our show. 46 00:03:03,503 --> 00:03:05,063 Michael: Yeah, it's their excuse to get out. 47 00:03:05,183 --> 00:03:07,283 Uh, I think dogs and elephants would be friends too. 48 00:03:08,183 --> 00:03:09,083 Keep it in the spirit. 49 00:03:09,148 --> 00:03:09,448 Nikolay: right. 50 00:03:10,588 --> 00:03:11,358 So vacuum. 51 00:03:11,588 --> 00:03:12,188 Where to start. 52 00:03:12,188 --> 00:03:12,588 What do you think. 53 00:03:12,838 --> 00:03:14,938 Michael: I think probably right at the beginning. 54 00:03:14,938 --> 00:03:15,178 Right. 55 00:03:15,178 --> 00:03:16,738 So why do we have VA? 56 00:03:16,888 --> 00:03:17,998 Like, what is vacuum? 57 00:03:18,058 --> 00:03:18,958 Why do we have it? 58 00:03:19,018 --> 00:03:22,078 Some people kind of see it as a negative part of Postgres. 59 00:03:22,078 --> 00:03:26,692 I see it as something we need because of some of the design decisions, right. 60 00:03:26,692 --> 00:03:30,472 At the beginning of Postgres, I personally really like it. 61 00:03:30,472 --> 00:03:34,012 I, I think we've gained so much from some of those design decisions. 62 00:03:34,987 --> 00:03:38,587 But this one is to do with multi versions, concurrency control. 63 00:03:38,587 --> 00:03:38,827 Right? 64 00:03:38,827 --> 00:03:46,369 So MVCC the fact that we keep old versions of nearly, I was about to say rose then, but I guess tuples 65 00:03:46,835 --> 00:03:49,024 Nikolay: rogue version, or you say doubles or 66 00:03:51,785 --> 00:03:52,705 Michael: which is correct? 67 00:03:52,705 --> 00:03:53,695 I say the correct one. 68 00:03:53,845 --> 00:03:54,715 Nikolay: I don't know. 69 00:03:54,835 --> 00:03:55,705 I heard both. 70 00:03:55,915 --> 00:03:59,275 So which we need to choose. 71 00:03:59,760 --> 00:03:59,980 Michael: So. 72 00:04:00,500 --> 00:04:01,800 I'm gonna say tuple. 73 00:04:02,180 --> 00:04:02,820 I don't know why. 74 00:04:02,840 --> 00:04:06,200 Nikolay: I, will choose two Okay. 75 00:04:06,200 --> 00:04:09,710 Taps, uh, let's let's let's see use taps, so right. 76 00:04:09,710 --> 00:04:19,152 A tap is a physical version of a row and each row can have multiple tops at the same and one transaction sees only one of them. 77 00:04:19,362 --> 00:04:24,792 It may happen that some tap, which is physical version of a row is not visible to any transaction. 78 00:04:24,792 --> 00:04:26,169 So basically it's dead. 79 00:04:26,246 --> 00:04:26,576 Right. 80 00:04:26,636 --> 00:04:33,831 And, you can check when the two was created, because you can select, uh, XME X, max, CT. 81 00:04:33,836 --> 00:04:36,171 I D we, we discussed CT ID some time. 82 00:04:36,885 --> 00:04:43,375 it's very convenient sometimes to know that there are hidden system columns in each table, they are created automatically. 83 00:04:43,375 --> 00:04:47,995 You cannot, for example, you cannot use CT I D or XME in, in your column names. 84 00:04:48,455 --> 00:04:54,355 So, so it's like reserv words, but you can select, uh, and see the value of XME X max, and the CT. 85 00:04:54,355 --> 00:05:04,305 I D CTD is a physical address like page and offset X mean is when the transaction idea which created the two, but X max is slightly more complex. 86 00:05:04,675 --> 00:05:08,515 It's usually zero means like a row is live, but it can be not zero. 87 00:05:08,520 --> 00:05:11,775 It can be something different, some transaction which was rolled back. 88 00:05:11,875 --> 00:05:13,135 So X max is present. 89 00:05:13,165 --> 00:05:14,965 You can see it in your transaction. 90 00:05:15,475 --> 00:05:22,782 Obviously this tool is live this like, so it means that when you select Xen X, max, you have a feeling of double. 91 00:05:23,652 --> 00:05:29,652 Right and the next time you select, you can have different values of XME X marks. 92 00:05:29,682 --> 00:05:34,438 XME no, but marks and CTD, they can change, , with hidden columns. 93 00:05:34,708 --> 00:05:35,878 So, right. 94 00:05:35,908 --> 00:05:42,059 So if, uh, twofold becomes dead, it's a problem because it still occupies some space. 95 00:05:42,118 --> 00:05:46,561 This is a key problem that, leads us to the, the need of vacuuming, right? 96 00:05:46,681 --> 00:05:50,101 And also like this very simple exercise. 97 00:05:50,101 --> 00:05:57,955 I recommend everyone who starts working with Paulus, create a table with just single column, uh, ID intro eight, for 98 00:05:57,960 --> 00:06:03,725 example, and, puts a row where ID equals one and then check I. 99 00:06:03,987 --> 00:06:07,827 and then just update is not changing values. 100 00:06:07,857 --> 00:06:11,547 Say ID equals ID and you will see CT ID will change. 101 00:06:11,547 --> 00:06:15,157 So this is how you can feel that new tap is created. 102 00:06:15,727 --> 00:06:23,655 Every time you try to do update, even if you try and enrolled back, tap will be created, but new tap will be marked that instead of old one. 103 00:06:24,605 --> 00:06:25,725 Michael: I didn't realize that that's 104 00:06:25,789 --> 00:06:26,089 Nikolay: Yeah. 105 00:06:26,119 --> 00:06:26,239 Yeah. 106 00:06:26,269 --> 00:06:30,499 Or you can insert an inside transaction and then roll back your transaction. 107 00:06:30,499 --> 00:06:31,789 So cancel your insert. 108 00:06:32,329 --> 00:06:34,636 And this insert will produce some tops. 109 00:06:34,966 --> 00:06:38,816 Again, this shows us that testing on production is not good. 110 00:06:38,816 --> 00:06:45,896 Some, some people say, oh, we will insert some data and then roll back our transaction, not to disturb production, but you disturb anyway, 111 00:06:45,896 --> 00:06:55,921 because you will produce, fresh dead twos and, you'll make a vacuum to make, to vacuum, to make more work than, than without your actions. 112 00:06:56,191 --> 00:06:57,091 Testing on production. 113 00:06:57,091 --> 00:06:58,111 Not is not a good idea. 114 00:06:58,222 --> 00:07:01,252 Michael: And this is the reason we are discussing this part of it is. 115 00:07:01,912 --> 00:07:09,382 One of the core tasks that vacuum has and presumably where it got its name from is then going and looking for these 116 00:07:09,467 --> 00:07:17,998 dead two pools and, vacuuming them up or, removing them so that, reclaiming the space, I guess is there is one that's. 117 00:07:18,215 --> 00:07:25,565 Nikolay: By the way to finish this idea about testing the one type of workload, which you can do not disturbing physical 118 00:07:25,595 --> 00:07:29,685 layout and not making out vacuum to work more is cancel, delete. 119 00:07:29,685 --> 00:07:30,825 So you delete a lot. 120 00:07:30,895 --> 00:07:31,795 Then roll back. 121 00:07:32,065 --> 00:07:33,145 You will produce a lot of high. 122 00:07:33,745 --> 00:07:41,305 Uh, a lot of records will go to wall, so it'll put some stress to replication, but physical layout won't change 123 00:07:41,305 --> 00:07:45,745 because your transaction will just put X, max new X max value. 124 00:07:46,015 --> 00:07:49,885 Then when it got canceled with transaction idea marked as canceled. 125 00:07:49,885 --> 00:07:50,725 So it'll. 126 00:07:51,400 --> 00:07:56,830 It will logically will equal to zero, meaning that these twos, these twos are still alive. 127 00:07:57,130 --> 00:07:59,170 So this is a very interesting kind of workload. 128 00:07:59,170 --> 00:08:04,170 I sometimes use on regular clones when we need to have many iterations. 129 00:08:04,635 --> 00:08:11,875 Of workload, but we don't want our physical layout to be changed so to start from the same point every time. 130 00:08:12,305 --> 00:08:16,510 So this is the only kind of workload that won't disturb physical layout and vacuum. 131 00:08:16,557 --> 00:08:16,827 Right. 132 00:08:16,977 --> 00:08:19,447 Sorry for maybe it's kind of off topic a little bit. 133 00:08:19,634 --> 00:08:26,181 , but I, I want everyone just to start thinking about, uh, pH physical rogue versions, which call apples when they 134 00:08:26,391 --> 00:08:29,751 think about performance, because it's very, very related. 135 00:08:29,751 --> 00:08:34,246 You cannot optimize performance, not understanding VCC and apples, right. 136 00:08:34,612 --> 00:08:35,392 Michael: Yeah, absolutely. 137 00:08:35,479 --> 00:08:35,839 So. 138 00:08:36,174 --> 00:08:37,314 Where do you want to go next? 139 00:08:37,314 --> 00:08:39,384 Do you want to look to talk about performance a little bit? 140 00:08:39,384 --> 00:08:45,415 Or do you want to talk about the different like, , so one of vacuums tasks is to, free up space. 141 00:08:45,431 --> 00:08:56,501 Nikolay: Yeah, which which, which tasks vacuum has, uh, first of all, cleaning up the twos, tops, tops, and second is, uh, 142 00:08:56,531 --> 00:09:00,671 preventing transaction ID wraparound, which is called freezing. 143 00:09:01,181 --> 00:09:05,337 And third one additional one is recalculation of table statistics. 144 00:09:05,517 --> 00:09:13,996 The, the statistics of data to help the planner to understand which plan to choose based on, on data stats. 145 00:09:14,026 --> 00:09:14,326 Right. 146 00:09:14,379 --> 00:09:16,299 And this is also like analyzed part, 147 00:09:16,689 --> 00:09:17,019 Michael: Yeah. 148 00:09:17,019 --> 00:09:18,069 So that happens. 149 00:09:18,069 --> 00:09:24,819 If you, you can, you can, additionally do vacuum analyze can't you, but also auto vacuum does this. 150 00:09:24,909 --> 00:09:27,879 So I wasn't sure if you were gonna bring this up, so yeah. 151 00:09:27,901 --> 00:09:28,111 Nikolay: Yeah. 152 00:09:28,111 --> 00:09:28,261 Yeah. 153 00:09:28,261 --> 00:09:36,087 Well, I switched us to auto vacuum actually, uh, implicitly, and this is of course, uh, this is what auto vacuum can do by the way. 154 00:09:36,092 --> 00:09:41,637 It's, it's interesting that it, sometimes it does just analyze for a table when time comes. 155 00:09:41,637 --> 00:09:41,877 Right. 156 00:09:42,177 --> 00:09:44,877 But sometimes it, it chooses to do vacuum analyze. 157 00:09:44,877 --> 00:09:46,557 So both actions at once. 158 00:09:46,611 --> 00:09:49,551 It's interesting that both are, are possible for auto vacuum. 159 00:09:49,602 --> 00:09:50,712 And that vacuum is great. 160 00:09:50,742 --> 00:09:53,022 It's it allows you to forget about vacuuming. 161 00:09:53,022 --> 00:09:56,356 I remember times when, uh, vacuum wasn't present in POS. 162 00:09:56,536 --> 00:10:00,886 So you, if by default you are in trouble. 163 00:10:01,726 --> 00:10:03,556 if you don't vacuum what happens? 164 00:10:04,055 --> 00:10:06,391 Michael: Yeah that's, that's, a really good point, actually. 165 00:10:06,396 --> 00:10:08,611 So there might be people listening who are, who. 166 00:10:09,256 --> 00:10:14,637 Maybe they're relatively new to progress or they've started a project and they've not had to worry about vacuum. 167 00:10:14,667 --> 00:10:21,837 And that's probably because for so far, at least auto vacuum has been, you know, it's, it's got certain default 168 00:10:21,837 --> 00:10:24,897 settings that probably a bit conservative in general. 169 00:10:25,152 --> 00:10:29,952 But when you're first getting started, when the tables are small, um, they're okay. 170 00:10:29,952 --> 00:10:30,402 They're fine. 171 00:10:30,402 --> 00:10:35,022 They'll keep you they'll keep this from being a problem for longer than, uh, as you say, if they don't have it. 172 00:10:35,022 --> 00:10:40,722 And I think, uh, as far as I can tell, unless you're extremely advanced and know exactly what you're doing, 173 00:10:40,962 --> 00:10:43,602 you really shouldn't be turning auto vacuum off. 174 00:10:43,962 --> 00:10:49,332 So if anybody suggests doing that, that should set off, big, big sirens and warning signals. 175 00:10:50,112 --> 00:10:50,782 Place of work. 176 00:10:50,902 --> 00:10:51,142 I think. 177 00:10:51,267 --> 00:10:51,597 Nikolay: Right. 178 00:10:51,597 --> 00:11:02,287 Well, uh, in my opinion, based on my like 17 plus experience of working with pores in, uh, quite large setups with hundreds of thousands transactions 179 00:11:02,287 --> 00:11:11,917 per second, multi dozens of terabytes of data and so on in all TP context, when a lot of users are present and they need very good performance. 180 00:11:11,917 --> 00:11:18,967 So my experience, my opinion is, uh, Uh, is not enough for almost everyone. 181 00:11:19,297 --> 00:11:22,147 So, so it should be tuned in the very beginning. 182 00:11:22,147 --> 00:11:27,307 For example, uh, only three auto workers is not enough for modern servers. 183 00:11:27,427 --> 00:11:27,697 Okay. 184 00:11:27,697 --> 00:11:32,527 If, if it's, if it's a laptop or some very, very small, uh, system. 185 00:11:32,902 --> 00:11:38,962 Three workers is probably enough, but since every time you produce the twos and we discuss just discussed you 186 00:11:39,292 --> 00:11:42,652 many transactions produce the twos you need cleanup. 187 00:11:42,652 --> 00:11:47,152 So it means that cleanup is almost like constant work that Pogs needs to do. 188 00:11:47,422 --> 00:11:48,532 So my recommendation is. 189 00:11:49,312 --> 00:11:55,462 Check how many course you have and allocate least 30% of those scores to, to WAM. 190 00:11:55,462 --> 00:12:03,472 So if you have 12 course, four workers, if you have, uh, I don't know, like 40, um, how many, like 96 191 00:12:03,472 --> 00:12:06,732 course, uh, it means that you need to have 30 workers. 192 00:12:06,732 --> 00:12:15,462 At least if, if you have a very big server and, uh, I'm, I'm excited to see that some, uh, some settings default 193 00:12:15,467 --> 00:12:20,532 settings were recently changed, uh, cost limit auto it's. 194 00:12:20,622 --> 00:12:21,372 It's very complex. 195 00:12:21,372 --> 00:12:28,212 Like, uh, it's very one setting depends on another and so on, but, uh, roughly a cost limit on cost. 196 00:12:29,127 --> 00:12:31,167 Uh, associated with auto vacuum. 197 00:12:31,527 --> 00:12:35,367 We are very, very, very old defaults. 198 00:12:35,997 --> 00:12:45,807 And, but roughly speaking, uh, you had only add maybe by its per second of reads, uh, for all workers that you have not more, this is like 199 00:12:45,807 --> 00:12:51,207 quarter, uh, and, uh, it was changed in PO August, maybe 12 or, or, or. 200 00:12:51,717 --> 00:12:52,947 Plus plus minus one. 201 00:12:53,457 --> 00:12:56,697 Uh, and it was like became 10 times more. 202 00:12:57,057 --> 00:13:00,357 So now, now roughly 80 max per second, but it's not enough 203 00:13:00,357 --> 00:13:03,927 for big, big setups with modern discs. 204 00:13:03,987 --> 00:13:07,984 You probably want like half terabyte or even even bigger, quota. 205 00:13:08,524 --> 00:13:09,364 So you need to tune 206 00:13:09,364 --> 00:13:10,054 farther father. 207 00:13:10,324 --> 00:13:13,644 Michael: Yeah I think there were quite a few good tuning guides. 208 00:13:13,674 --> 00:13:14,184 Aren't there. 209 00:13:14,544 --> 00:13:15,594 We could, I can link to it. 210 00:13:15,714 --> 00:13:19,194 I know at least one really good one in terms of giving people advice on. 211 00:13:19,730 --> 00:13:20,090 Go on. 212 00:13:20,180 --> 00:13:27,020 Nikolay: I like couple of articles from, I don't remember who, who wrote it, but it was on second quant blog. 213 00:13:27,770 --> 00:13:32,570 when not vacuum doesn't vacuum, when vacuum does, does vacuum understanding basics of vacuuming. 214 00:13:32,570 --> 00:13:35,930 This is, this is like very like great basic material. 215 00:13:36,290 --> 00:13:43,310 And I, I even open it to very often when I need to explain something to others, I always mention this, these couple of posts. 216 00:13:43,640 --> 00:13:47,780 So let's link them in, in encourage everyone to, to read it. 217 00:13:48,380 --> 00:13:53,240 But I also wanted to say that, like, if you don't tune it in the very. 218 00:13:55,340 --> 00:13:56,510 will come, right. 219 00:13:56,600 --> 00:13:57,200 Eventually 220 00:13:57,875 --> 00:13:58,505 Michael: Yes, 221 00:13:58,760 --> 00:13:59,930 Nikolay: the two risks, right, right. 222 00:14:00,080 --> 00:14:03,370 Transaction idea up around and blo two risks 223 00:14:03,495 --> 00:14:03,885 Michael: Yep. 224 00:14:03,950 --> 00:14:11,600 and that, like, that manifests itself in gradually gradually slow performance or, you know, like various things will start 225 00:14:11,600 --> 00:14:14,750 to degrade at that point, disc like will start to increase. 226 00:14:14,750 --> 00:14:17,660 Like, there's, there's a few different ways that that will manifest. 227 00:14:18,110 --> 00:14:27,353 Um, and I, I actually wanted to say like, oh, before we move on from auto vacuum, The the best advice I ever heard was it's a bit like exercise. 228 00:14:27,713 --> 00:14:30,353 If you, if it hurts, you're not doing it enough. 229 00:14:30,533 --> 00:14:37,193 So it's the, if auto vacuum looks like it's getting in the way of things, so sometimes I think people see it running and that's 230 00:14:37,193 --> 00:14:39,803 blocking other things, or it's taking ages, things like that. 231 00:14:40,013 --> 00:14:42,593 If you're in that situation, don't turn it off. 232 00:14:42,593 --> 00:14:43,883 I think some people get tempted. 233 00:14:44,083 --> 00:14:51,643 To turn it off so that, so that the problem goes away, the, the solution is more, make it more aggressive, make it happen more frequently. 234 00:14:51,643 --> 00:14:52,393 Exactly. 235 00:14:52,753 --> 00:14:57,873 So, um, that, that seems to be what, what trips people up, around this 236 00:14:57,973 --> 00:14:58,263 Nikolay: yeah. 237 00:14:58,313 --> 00:14:58,943 Funny word. 238 00:14:58,943 --> 00:15:06,018 When I describe what to do with AKI, I also use, uh, aggressive and sometimes some company they have interesting bot in, 239 00:15:06,078 --> 00:15:09,138 in slack that say, no, you shouldn't use word aggressive. 240 00:15:10,518 --> 00:15:14,688 choose another one and options but I agree like OWA tuning. 241 00:15:15,353 --> 00:15:18,863 It's two things make it aggressive, move faster, right? 242 00:15:18,863 --> 00:15:19,673 So aggressive. 243 00:15:19,823 --> 00:15:21,353 Like what does it mean? 244 00:15:21,773 --> 00:15:22,763 Move faster. 245 00:15:22,763 --> 00:15:29,453 So give more quota don't limit risk discretes, and so on. 246 00:15:30,273 --> 00:15:34,143 and cost limit cost delay, these like set of settings. 247 00:15:34,203 --> 00:15:35,193 There are several settings. 248 00:15:35,783 --> 00:15:38,633 so also at vacuum settings, depend on vacuum settings. 249 00:15:38,693 --> 00:15:41,153 If it it's minus one, it means get it from there. 250 00:15:41,158 --> 00:15:44,483 So like, that's why I, I see, I say it's quite complex. 251 00:15:44,488 --> 00:15:52,490 You need to, to spend some time understanding these like hierarchy of settings, but, this is one thing, another thing is frequency. 252 00:15:53,090 --> 00:15:56,870 So like speed quarters, aggressiveness and frequency. 253 00:15:57,380 --> 00:16:06,320 Frequency is very important because if you allow it move fast, but you keep a defaults like 10% of that twos it's a lot. 254 00:16:07,670 --> 00:16:11,210 I, 1% below T we need to go down. 255 00:16:11,210 --> 00:16:13,990 Like we need to, remove the chips more often. 256 00:16:14,129 --> 00:16:14,369 Michael: Yeah. 257 00:16:14,369 --> 00:16:19,499 So just for anybody wondering, that's like by default, I think all tables. 258 00:16:19,589 --> 00:16:27,599 Uh, so the like basically any table has to grow by 10% in order for it to trigger another auto vacuum, which when, when the tables are a hundred or 259 00:16:27,617 --> 00:16:33,375 Nikolay: has, has, sorry, needs to accumulate at least 10% of thats, 260 00:16:33,999 --> 00:16:34,669 Michael: great point. 261 00:16:34,889 --> 00:16:35,109 Yes. 262 00:16:35,235 --> 00:16:35,715 Nikolay: right? 263 00:16:36,255 --> 00:16:37,365 It can grow. 264 00:16:38,355 --> 00:16:40,185 For example, a Pand leak situation. 265 00:16:40,190 --> 00:16:49,695 It was fixed by the way in PO 13, maybe when, when this analyzed part of auto didn't trigger because we don't have dead twos and special 266 00:16:49,695 --> 00:16:53,295 setting was added at additional logic for Pand leak situations. 267 00:16:53,775 --> 00:17:01,485 But the, the, like at least 10% of dead twos by default, and in my opinion should be 1% in OTP, 268 00:17:02,234 --> 00:17:04,154 Michael: Or maybe not even like, should it Def. 269 00:17:04,649 --> 00:17:06,389 Should it definitely be a percentage. 270 00:17:06,389 --> 00:17:11,159 Like I've seen people switch completely to a raw number OFTU pools instead. 271 00:17:11,219 --> 00:17:16,679 So they, they don't have this degradation over time as it, as it, as those relations 272 00:17:16,890 --> 00:17:17,280 Nikolay: right. 273 00:17:17,480 --> 00:17:19,590 So yeah, it's also not simple. 274 00:17:19,590 --> 00:17:25,020 There is scale factors, also set of settings for, for analyze, for vacuum part. 275 00:17:25,020 --> 00:17:26,250 And also there is a. 276 00:17:27,075 --> 00:17:32,755 Threshold is like some absolute number, like 50 by default, as I remember, 50 by twos. 277 00:17:32,985 --> 00:17:41,955 And, uh, there is some formula based on these two based off these two settings that gives you real threshold that when like defining when not 278 00:17:41,955 --> 00:17:50,071 vacuum can triggers, , well, this static, number approach is, you know, like there are many topics people have different opinions about, for 279 00:17:50,071 --> 00:18:00,441 example, some, I, I see several groups of people and you still see them that say, uh, uh, ATO vacuum default auto vacuum is a very silly algorithm. 280 00:18:00,474 --> 00:18:11,574 , for example, it doesn't understand that on, on weekends we have much more space for work for auto vacuum or at nights it's, it doesn't respect 281 00:18:11,604 --> 00:18:16,194 the time of day or of week or, or the day week, and we should change it. 282 00:18:16,194 --> 00:18:21,004 So they switch off photo vacuum and, uh, implement their own, algorithm and run vacuum. 283 00:18:21,101 --> 00:18:22,921 Or do both. 284 00:18:23,097 --> 00:18:25,347 Michael: Both both seems like really smart to me. 285 00:18:25,347 --> 00:18:25,677 Right? 286 00:18:25,677 --> 00:18:32,842 So if you know when your business is quiet and you can afford most of the time to be able to do it, , a manual vacuum analyze, 287 00:18:33,202 --> 00:18:40,402 it's not that you're doubling the work by keeping auto vacuum on because when auto vacuum then does trigger, it has less to do there. 288 00:18:40,402 --> 00:18:41,812 There's there's less work. 289 00:18:41,812 --> 00:18:44,512 So I, I do see the logic to that myself. 290 00:18:45,232 --> 00:18:45,592 Nikolay: Yeah. 291 00:18:45,892 --> 00:18:50,882 Also, if you do it, with your script, it means that you run so called manual. 292 00:18:50,882 --> 00:18:54,152 Of course it's a full, automated this case, but it's so called manual vacuum. 293 00:18:54,152 --> 00:18:58,682 And in recent versions of POS it has benefit unlike auto vacuum. 294 00:18:58,682 --> 00:19:01,997 It can process indexes of a table in parallel. 295 00:19:02,027 --> 00:19:11,532 So multiple processes and it's like, ization is a great to improve vacuuming on largers it's it's inevitable way I would say. 296 00:19:11,832 --> 00:19:15,192 And it, it leads us to the topic of partitioning, but maybe 297 00:19:15,197 --> 00:19:16,302 that's a little bit postpone. 298 00:19:16,932 --> 00:19:17,592 Right, right. 299 00:19:17,892 --> 00:19:19,032 But, uh, I agree. 300 00:19:19,422 --> 00:19:20,052 Well, I don't. 301 00:19:20,112 --> 00:19:25,992 I, I don't like the idea of turning a vacuum completely and relying on your custom inhouse tool. 302 00:19:26,052 --> 00:19:30,582 It may be dangerous, but combination sounds good. 303 00:19:30,582 --> 00:19:40,262 But, if you run vacuum, if you connect to POS using P SQL and run vacuum, it's UN throttled, or thro is a good way, is a good way to name quotas differently. 304 00:19:40,322 --> 00:19:40,652 Right. 305 00:19:40,670 --> 00:19:50,610 So auto vacuum is throttled too much by default, but manual vacuum is not throttled at all because vacuum cost delay is zero by default. 306 00:19:50,734 --> 00:19:51,944 It means Thero link is not. 307 00:19:52,322 --> 00:19:57,692 and my question is, can you put your system down in terms of situation of this Coyo? 308 00:19:58,082 --> 00:20:04,322 Just doing some vacuum, maybe in multiple, uh, processes, multiple tables at once. 309 00:20:04,982 --> 00:20:09,122 I, I tried I on modern hardware on one, like NBME disks. 310 00:20:09,122 --> 00:20:09,902 I could not do it. 311 00:20:10,592 --> 00:20:12,752 maybe on older hardware, it's possible. 312 00:20:12,752 --> 00:20:15,830 I saw problems related to performance when vacuum was. 313 00:20:16,360 --> 00:20:17,470 Aggressive right. 314 00:20:17,527 --> 00:20:25,268 Moving Too fast, using too much of our disco or capacity, but, on modern disks, we, we recently tried it. 315 00:20:25,358 --> 00:20:33,188 Like we tried to prove that, unleash auto vacuum was not a good idea, but we failed, always had room, even if we used. 316 00:20:33,743 --> 00:20:36,623 Almost all, all CPUs are doing some work. 317 00:20:37,073 --> 00:20:37,613 It's interesting. 318 00:20:37,733 --> 00:20:38,753 Maybe I'm wrong here. 319 00:20:38,753 --> 00:20:42,717 It's an interesting exercise to see where the limit is. 320 00:20:43,130 --> 00:20:46,088 Michael: That sounds like a, that doesn't sound like a failure at all. 321 00:20:46,088 --> 00:20:49,788 That sounds like a successful experiment where you learn something really important. 322 00:20:50,059 --> 00:20:52,742 Nikolay: well, there is always bottleneck somewhere, right? 323 00:20:52,982 --> 00:20:59,224 If we cannot, , situate our dis it means that probably we have some, we, we spend some time in code. 324 00:20:59,382 --> 00:21:01,752 Loading CPU more than could be like, I don't know. 325 00:21:01,752 --> 00:21:02,708 Like it's interesting. 326 00:21:02,708 --> 00:21:11,356 I'm just trying to say that on modern disks, you probably should almost unleash it to move faster if you, especially, if you have a lot, of course. 327 00:21:11,416 --> 00:21:12,316 Michael: There were a couple more. 328 00:21:12,316 --> 00:21:17,536 So I think partitioning actually might be worth discussing briefly, probably not the depths of it, but I think a lot of 329 00:21:17,536 --> 00:21:21,346 people think partitioning is gonna help them with performance. 330 00:21:21,706 --> 00:21:23,476 And my experience is different. 331 00:21:23,481 --> 00:21:27,946 My experience is the main argument for it is all around maintenance. 332 00:21:27,976 --> 00:21:30,305 It's all around being able to delete. 333 00:21:30,356 --> 00:21:31,509 I'm getting big thumbs up here. 334 00:21:31,509 --> 00:21:32,339 So yeah, go on. 335 00:21:32,349 --> 00:21:32,679 Nikolay: Yes. 336 00:21:32,679 --> 00:21:33,159 And no. 337 00:21:33,596 --> 00:21:35,569 So I agree like B three is great. 338 00:21:35,581 --> 00:21:37,679 the height B three grows very slow. 339 00:21:37,889 --> 00:21:43,379 So if we have, uh, hundred gigabytes, if we have 10 terabytes, the difference is not that big. 340 00:21:43,409 --> 00:21:47,369 We have, we need just additional, uh, um, buffers. 341 00:21:47,759 --> 00:21:50,849 To be checked to perform index scan, right. 342 00:21:51,209 --> 00:21:52,919 Index on the scan, if you talk about it. 343 00:21:53,309 --> 00:22:04,679 But, the problem is definitely related to maintenance because if we have 10 terabyte table, a vacuum is taking a day, maybe depending 344 00:22:04,679 --> 00:22:08,669 on, on your speed and, and the power of disks and, and CPU and so on. 345 00:22:09,209 --> 00:22:11,609 But, uh, like it's, it's it's problem itself. 346 00:22:11,659 --> 00:22:12,619 Michael: partition table. 347 00:22:12,624 --> 00:22:12,949 Right? 348 00:22:13,094 --> 00:22:13,694 Nikolay: no partition. 349 00:22:13,694 --> 00:22:14,444 Of course I I'm. 350 00:22:14,444 --> 00:22:21,824 I'm I'm trying to understand like how partitioning can be connected to vacuuming and, uh, one how partitioning can help vacuuming. 351 00:22:22,214 --> 00:22:28,594 Of course, if a table is partitioned, the problem is that auto vacuum cannot,, process even. 352 00:22:29,029 --> 00:22:36,619 Indexes of a table yet it maybe it'll be implemented in future because, uh, regular vacuum can do it, but also like even if it 353 00:22:36,619 --> 00:22:44,329 processed the, the indexes in parallel, for example, you have 10 indexes and you process, uh, using 10, 10 CPUs, 10 processes. 354 00:22:44,329 --> 00:22:44,539 Right. 355 00:22:44,929 --> 00:22:52,189 But, , heap itself, it's hard to paralyze it also, I think it's possible, but it's maybe like it's complex task right now. 356 00:22:52,189 --> 00:22:52,909 It's not possible. 357 00:22:52,909 --> 00:23:00,614 So if you have a huge table the process of vacuuming it with all its SynXis will be single credited by, by auto vacuum auto vacuum. 358 00:23:01,184 --> 00:23:06,819 But if it, you partition it, you can benefit from having many auto vacuum workers. 359 00:23:07,179 --> 00:23:12,729 Of course, if you tuned it from default free to a bigger number and you should on larger systems. 360 00:23:13,149 --> 00:23:16,869 So this greatly improves the speed of vacuum. 361 00:23:17,097 --> 00:23:25,884 but not only this of course, if your table is partitioned, you can re-index and create index or recreate indexes, uh, much faster, but 362 00:23:25,884 --> 00:23:31,164 also the state of cash improves because data is localized, much better. 363 00:23:31,164 --> 00:23:38,034 New data is present in pages where mostly new data or some pages have all data. 364 00:23:38,364 --> 00:23:45,894 And if these twos tops are not changed in, in some page, it's, it's all frozen, all visible. 365 00:23:45,976 --> 00:23:47,517 Auto vacuums keeps it right. 366 00:23:47,697 --> 00:23:48,507 It's so good. 367 00:23:48,567 --> 00:23:50,757 And maybe it's even not present in cash. 368 00:23:50,757 --> 00:23:54,807 So you have more space in your buffer pool and page cash for newer data. 369 00:23:55,077 --> 00:23:58,367 So cash efficiency increases as well. 370 00:23:58,467 --> 00:23:58,817 Right? 371 00:23:59,027 --> 00:24:08,880 And this directly improves performance, but also one topic I only recently realized is that, ind indexing and re-indexing it affects vacuum. 372 00:24:08,953 --> 00:24:15,571 If creation of some index or recreation, some index takes hours during this period, vacuum cannot delete freshly that twos 373 00:24:16,081 --> 00:24:20,251 twos which became that tops, which became that only recently. 374 00:24:20,881 --> 00:24:21,241 Right? 375 00:24:21,246 --> 00:24:26,221 And this is a problem because indexing indexing, they hold horizon. 376 00:24:26,611 --> 00:24:27,631 Michael: E even if it's 377 00:24:28,000 --> 00:24:29,290 Nikolay: Yes, concurrently as well. 378 00:24:29,470 --> 00:24:36,230 There was an attempt to, uh, optimize this in posts 14 for index and index concurrently. 379 00:24:36,590 --> 00:24:40,580 And, it was so great, like index and index concurrently. 380 00:24:40,623 --> 00:24:42,093 They, they don't hold Xin two. 381 00:24:42,373 --> 00:24:46,153 So all the two, which became that recently vacuum can clean. 382 00:24:46,398 --> 00:24:46,758 Great. 383 00:24:47,088 --> 00:24:58,197 But in recently in June, in Pogs 14.4, this triggered, this releases understand immediate back fix needed, 384 00:24:58,197 --> 00:25:01,014 and this functionality was reverted in Pogs 14. 385 00:25:01,014 --> 00:25:01,734 Unfortunately. 386 00:25:01,844 --> 00:25:06,600 So the rule of fam you don't want indexation to be very long. 387 00:25:06,645 --> 00:25:15,411 So you need partitioning for faster vacuuming to, to, run it parallel And to avoid blood because what, like, we didn't discuss what blood is, right? 388 00:25:15,411 --> 00:25:20,091 Blo happens when you accumulate too many that to, to taps, right. 389 00:25:20,331 --> 00:25:23,151 And then a vacuum, delete them a lot of them at once. 390 00:25:23,151 --> 00:25:26,331 And you have a lot of space free space inside pages. 391 00:25:27,448 --> 00:25:27,958 Michael: Yes. 392 00:25:28,018 --> 00:25:31,638 And, and I think we probably should, because we're on the topic of indexes. 393 00:25:31,638 --> 00:25:38,827 Talk about, we we've talked mostly about table bloat so far, so, that row versions, dead row versions, but there are 394 00:25:38,827 --> 00:25:45,217 also, there's also index bloat, which is subtly different in my opinion, because you can, you've got the same entries. 395 00:25:47,137 --> 00:25:48,337 Like freeing them up. 396 00:25:48,487 --> 00:25:56,037 Doesn't do as much good as in, in the table or is in the, in the heap, in the heap, they can be reused very easily. 397 00:25:56,067 --> 00:25:59,277 Another row, a row version can be inserted there. 398 00:25:59,587 --> 00:26:06,907 It doesn't matter what, but in a, in an index because it's, , it or in a bere index, for example, Adding more 399 00:26:06,957 --> 00:26:11,727 rows can split pages and vacuum won't UNS spit the pages. 400 00:26:11,727 --> 00:26:17,197 So, even if you vacuum as very strongly or very aggressively, you. 401 00:26:17,434 --> 00:26:17,784 Nikolay: rebel. 402 00:26:18,557 --> 00:26:19,037 Michael: It does. 403 00:26:19,097 --> 00:26:19,607 Yeah, exactly. 404 00:26:19,607 --> 00:26:24,287 You won't get it back to the same size as it was at at first. 405 00:26:24,287 --> 00:26:33,927 And therefore you might need to re-index from time to time if you want to remove index bloat as well, or, or you can stay on as on top of it as you can. 406 00:26:34,077 --> 00:26:38,560 And there's some good optimizations for this in Postgres 30, and I believe maybe 14. 407 00:26:38,830 --> 00:26:41,821 So yeah, it it's just something that people sometimes don't realize. 408 00:26:42,063 --> 00:26:47,327 Nikolay: Yeah, we talked about it in both in both B three optimizations. 409 00:26:47,357 --> 00:26:50,717 I think they started in POG, August 12, even in or 13. 410 00:26:50,717 --> 00:26:55,852 And we discussed it on Pogo TV channel with on a, and then with Peter Gagan. 411 00:26:56,092 --> 00:27:00,622 It's it's like so great that this optimization happen that very fundamental. 412 00:27:01,037 --> 00:27:05,807 Affecting every post set up, so good D duplication and so on. 413 00:27:06,077 --> 00:27:13,161 I, I agree, but, I think, blog sometimes as useful, both in hip and, uh, indexes like, if in inhibit 414 00:27:13,341 --> 00:27:17,081 can help to have more often, hip only two updates. 415 00:27:17,126 --> 00:27:17,966 More optimized. 416 00:27:18,056 --> 00:27:22,779 That's not, not without need to touch indexes, but in indexes, I think it's also useful. 417 00:27:22,779 --> 00:27:27,446 Otherwise, why do we have default fuel factor in indexes 90 not hundred, right? 418 00:27:27,806 --> 00:27:31,286 I'm not like, like I'm not expert in indexes. 419 00:27:31,336 --> 00:27:39,686 We have other experts like Peter Gagan and, and regen, for example, or like, by the way, the way, , Inus is a great. 420 00:27:40,351 --> 00:27:41,851 Set of topics, right? 421 00:27:41,881 --> 00:27:52,816 We, we should talk about them at some point, but related to vacuum, I, I think you're right, over time indexes health degrades anyway, even in 422 00:27:52,846 --> 00:28:01,811 like in modern versions, latest versions of Postgres, it's much better situations improve, improve blood growth decreased, but still we need to 423 00:28:01,871 --> 00:28:05,601 perform so-called index maintenance and rebuild indexes from time to time. 424 00:28:05,651 --> 00:28:11,861 even if our vacuum is well tuned and, uh, we have the latest version, we still need to rebuild indexes. 425 00:28:11,951 --> 00:28:20,231 And when we rebuild indexes, we want to be very fast because of Xin horizon, because we affect, uh, all tables by the way. 426 00:28:20,561 --> 00:28:27,174 So if we have, if we build one index and if it takes many hours, indexes and our database are affected. 427 00:28:27,264 --> 00:28:34,531 So vacuum cannot clean tops in all tables, all indexes systems, freshly dead tubes and interest to, to that thats. 428 00:28:34,536 --> 00:28:36,633 So it's, it's huge global problem. 429 00:28:36,663 --> 00:28:39,025 So you want indexing to be fast. 430 00:28:39,062 --> 00:28:40,209 Michael: Yeah, super interesting. 431 00:28:40,509 --> 00:28:47,899 I had one last thing that I thought we couldn't finish this episode without talking about briefly and, or even if it's just a public service announcement. 432 00:28:48,019 --> 00:28:54,439 There is a parameter for vacuum or I think it should really be called a different thing, but it's, it's 433 00:28:54,444 --> 00:28:58,219 called vacuum full and you probably never want it. 434 00:28:58,279 --> 00:29:02,859 Like, I,, I could imagine people very happily using Postgres for decades and never using it. 435 00:29:03,249 --> 00:29:10,779 But I, I just wanted to say, because I think it can trip people up thinking, well, like the name suggests that it's gonna do very, a very comprehensive 436 00:29:10,779 --> 00:29:14,369 version of vacuum, but, because of the locks it takes it's, it's, 437 00:29:14,435 --> 00:29:16,055 Nikolay: I use it all the time. 438 00:29:16,115 --> 00:29:17,108 I use it all the time. 439 00:29:17,132 --> 00:29:17,612 Michael: go on. 440 00:29:17,612 --> 00:29:18,242 Well for, 441 00:29:19,260 --> 00:29:20,280 Nikolay: really, I 442 00:29:20,422 --> 00:29:20,742 Michael: you joking? 443 00:29:21,230 --> 00:29:21,520 Nikolay: well. 444 00:29:21,570 --> 00:29:27,401 So I use it all the time because it's related to how we estimate how we see how much blood we have. 445 00:29:27,851 --> 00:29:30,611 You know, that blood estimation is not a simple task. 446 00:29:30,704 --> 00:29:31,274 , scripts. 447 00:29:31,334 --> 00:29:33,824 All, everyone has various versions of scripts. 448 00:29:33,924 --> 00:29:36,234 They have lightweight, but they can be very wrong. 449 00:29:36,234 --> 00:29:44,304 For example, I, I can show you how I can, uh, this, the script says we have 40% of blood, but blood is zero because the table is just created. 450 00:29:44,694 --> 00:29:45,624 It's easy to do. 451 00:29:46,014 --> 00:29:48,304 It's related to alignment aging. 452 00:29:48,380 --> 00:29:59,940 So if you want to understand real blood numbers, The only there is a top two top extension extension, but I had issues with, using it some time ago. 453 00:30:00,090 --> 00:30:07,310 So since we are big fans of real experimenting and using clones and think loans C doesn't matter in this case, what do we do? 454 00:30:07,310 --> 00:30:10,903 We just create clone and promote it to primary. 455 00:30:10,903 --> 00:30:16,286 So it's real clone writeable, and we vacuum full whole database or specific tables. 456 00:30:16,586 --> 00:30:19,106 And we compare numbers before and after. 457 00:30:19,931 --> 00:30:20,231 Right. 458 00:30:20,411 --> 00:30:22,271 And we can say a hundred percent. 459 00:30:22,511 --> 00:30:25,571 We know that blood is this current blood, is this, 460 00:30:25,974 --> 00:30:29,524 Michael: So, and, and, also, presumably we're talking about test environments here. 461 00:30:29,524 --> 00:30:30,501 We are not talking about. 462 00:30:30,508 --> 00:30:36,158 Nikolay: Of course detached instance, not product it's, it's not test because it's production data, but it's like 463 00:30:36,278 --> 00:30:39,692 closer to production to get a real measurement of vacuum. 464 00:30:39,732 --> 00:30:43,032 This, this is very simple, straightforward, brute force approach. 465 00:30:43,224 --> 00:30:43,373 useful. 466 00:30:43,627 --> 00:30:45,067 Michael: Yes, I okay. 467 00:30:45,067 --> 00:30:45,877 I agree. 468 00:30:45,877 --> 00:30:51,157 I was thinking, uh, purely for production use case, but you are kind a hundred percent more as usual. 469 00:30:51,877 --> 00:30:53,877 Is there anything else you wanted to make sure we talked about today? 470 00:30:53,921 --> 00:30:59,316 Nikolay: No, let's make some summary, maybe like recommendations, uh, understand V VCC of course. 471 00:30:59,316 --> 00:30:59,586 Right. 472 00:30:59,646 --> 00:31:00,606 Just like read about it. 473 00:31:00,606 --> 00:31:04,956 Understand there are many good materials around then tune auto vacuum. 474 00:31:05,611 --> 00:31:09,951 And then there are, there, there is a separate topic of monitoring maybe next time, right. 475 00:31:10,491 --> 00:31:16,431 Also run, create indexes from time to time, probably an automated fashion. 476 00:31:16,731 --> 00:31:23,833 It's all, it's not a simple task, also worth separate discussion, , and maybe use some tools like PPG pack or 477 00:31:23,863 --> 00:31:28,583 PPG squeeze to deal with blood, in tables themselves. 478 00:31:28,583 --> 00:31:28,823 Right. 479 00:31:29,648 --> 00:31:30,068 What else? 480 00:31:30,068 --> 00:31:31,418 Ah, partitioning. 481 00:31:32,408 --> 00:31:40,770 there's a rule of fam, it's like an empiric rule, , not based on like some logic, but it's based on experience of many people if the table grows. 482 00:31:40,770 --> 00:31:45,690 So, and, or has chances to grow over hundred gigs, it should be partitioned. 483 00:31:45,967 --> 00:31:46,807 Michael: Yeah, I like this. 484 00:31:46,807 --> 00:31:48,607 And I think you word one more rule. 485 00:31:48,612 --> 00:31:49,207 Didn't you like? 486 00:31:49,207 --> 00:31:52,867 So it's, uh, if you're thinking about hundred gigabytes, think about partitioning. 487 00:31:52,897 --> 00:31:55,987 If you're thinking about a terabyte, think about shouting. 488 00:31:56,047 --> 00:31:56,737 Was that your role? 489 00:31:57,045 --> 00:31:57,555 Nikolay: Shouting. 490 00:31:58,185 --> 00:31:58,485 Yeah. 491 00:31:58,635 --> 00:31:59,055 Yeah. 492 00:31:59,055 --> 00:32:01,485 it was, yeah, it was, uh, discussion too. 493 00:32:01,875 --> 00:32:02,625 It's not my role. 494 00:32:02,625 --> 00:32:06,091 It's like it started in different places and then, 495 00:32:06,109 --> 00:32:06,619 Michael: sounds good. 496 00:32:06,619 --> 00:32:07,489 It's easy to remember. 497 00:32:09,409 --> 00:32:09,949 Wonderful. 498 00:32:09,979 --> 00:32:10,639 Well, thank you. 499 00:32:10,639 --> 00:32:11,719 Thanks again to everybody. 500 00:32:11,719 --> 00:32:15,019 Who's been sharing this on Twitter and wherever else. 501 00:32:15,089 --> 00:32:16,284 We really appreciate it. 502 00:32:16,554 --> 00:32:18,864 And yeah, looking forward to talking to you again next week. 503 00:32:18,933 --> 00:32:19,353 Nikolay: Thank you. 504 00:32:19,353 --> 00:32:19,773 Bye.