1 00:00:00,049 --> 00:00:03,769 Michael: Hello and welcome to Postgres fm, a weekly show about all things Postgres qr. 2 00:00:04,099 --> 00:00:05,569 I'm Michael, founder of PG Mustard. 3 00:00:05,569 --> 00:00:07,819 This is my cohost Nikola, founder of Postgres ai. 4 00:00:08,090 --> 00:00:10,009 Hey, Nikola, what are we talking about today? 5 00:00:10,384 --> 00:00:11,014 Nikolay: Hi Michael. 6 00:00:11,014 --> 00:00:11,814 It's your turn. 7 00:00:12,084 --> 00:00:13,414 Tell us, tell us please, 8 00:00:13,549 --> 00:00:15,559 Michael: Oh, um, flip it on me. 9 00:00:15,559 --> 00:00:15,979 Yes. 10 00:00:15,979 --> 00:00:17,179 I, I picked this one. 11 00:00:17,179 --> 00:00:24,409 I think we're gonna call it row estimates, but this is, uh, yeah, based on, so this is about the planner, uh, query planner. 12 00:00:24,439 --> 00:00:28,849 And this is again, based on a listener suggestion or request. 13 00:00:29,330 --> 00:00:35,179 Uh, they didn't ask about this specifically, but they did ask about nested loops causing performance issues. 14 00:00:35,690 --> 00:00:43,360 And, looking at any one part, Performance issue would be, quite a dull podcast, I think, maybe a better video. 15 00:00:43,750 --> 00:00:58,150 But it really, a lot of these, especially when it comes to people thinking that the nested loop is the problem, it's normally due to a bad row estimate at some or an inaccurate row estimate at some point, and a bad planner choice based on that low estimate. 16 00:00:58,510 --> 00:01:03,235 So I wanted to dive into how Progres is making those decision. 17 00:01:03,615 --> 00:01:12,624 And also some things that we can do as users to help it get better statistics and to help guide it in the right direction or, or not. 18 00:01:12,624 --> 00:01:16,704 Things that we don't, maybe don't have at our disposal at the moment that other, other databases do. 19 00:01:16,764 --> 00:01:19,044 So that's, that's the kind of thing I was hoping to talk about. 20 00:01:19,685 --> 00:01:21,964 Nikolay: Sounds good, but what's wrong with nest loops? 21 00:01:22,505 --> 00:01:22,774 They. 22 00:01:24,199 --> 00:01:31,220 Michael: Yeah, and this is the, this is the, um, reason I didn't want to call this, you know, nested loops causing issues. 23 00:01:31,399 --> 00:01:34,440 It's mostly, so yeah, so what, what's wrong with them is a good point. 24 00:01:34,440 --> 00:01:36,390 And I think it's about trade-offs, right? 25 00:01:36,390 --> 00:01:46,660 So the reason we have multiple, we, a lot of the time the planner has multiple choices for the joint algorithm it chooses, and even the scans that it does, so it's not even just about joins. 26 00:01:46,760 --> 00:01:49,070 It has a lot of choices and different options and. 27 00:01:49,910 --> 00:01:51,290 Sequels declarative. 28 00:01:51,740 --> 00:01:56,840 It can make its mind up, out the, the fastest or most efficient way to, to do that. 29 00:01:57,263 --> 00:01:59,753 , nested loops have some real advantages. 30 00:01:59,753 --> 00:02:02,093 They've, they're very quick to get started. 31 00:02:02,123 --> 00:02:03,193 They're very flexible. 32 00:02:03,198 --> 00:02:05,363 They can be used in a lot of different cases. 33 00:02:05,813 --> 00:02:07,823 But when the're two. 34 00:02:08,933 --> 00:02:12,413 Relations that are being joined get large. 35 00:02:12,713 --> 00:02:18,353 There can be more efficient or quick or quicker ways of joining those in Postgres. 36 00:02:18,953 --> 00:02:22,013 We have hash joins and we have merge joins. 37 00:02:22,523 --> 00:02:25,113 I think that's it, for the join algorithms. 38 00:02:25,718 --> 00:02:25,928 Nikolay: Yeah. 39 00:02:26,258 --> 00:02:29,088 So nested loops are good when, uh, of. 40 00:02:29,348 --> 00:02:33,758 Relation on one side is smaller, quite small, right? 41 00:02:33,968 --> 00:02:36,008 A low number of rows on one side, 42 00:02:36,788 --> 00:02:37,658 Michael: Yeah, exactly. 43 00:02:38,348 --> 00:02:39,038 Both sides. 44 00:02:39,038 --> 00:02:45,158 One side and if, if, even if one side's large, if it's indexed, an nested loop can still be really good. 45 00:02:45,158 --> 00:02:45,398 Right? 46 00:02:45,768 --> 00:02:46,378 Nikolay: mm-hmm. 47 00:02:46,688 --> 00:02:52,478 . And if they are both quite large, uh, nest, look, probably it's not a good idea to, to have it. 48 00:02:54,458 --> 00:02:55,328 Michael: Yeah, exactly. 49 00:02:55,328 --> 00:03:02,258 So we'll probably be better off with a hash join or in some cases if they're both sorted, a merge join, I guess. 50 00:03:02,328 --> 00:03:04,248 But sometimes we don't have those options, right? 51 00:03:04,253 --> 00:03:05,238 Sometimes the only. 52 00:03:05,778 --> 00:03:18,828 If we, if we're not doing an quality operation, I think there, so it's, but equally, when people are looking into this, if there's a good plan and a bad plan, if it, it, that's often when people think it's the nested loop that's the problem. 53 00:03:19,038 --> 00:03:20,658 It, there was another option, right? 54 00:03:20,658 --> 00:03:27,708 There was a better plan, a better path that could have been chosen but wasn't because it was being scored higher cost. 55 00:03:27,708 --> 00:03:30,048 The costs were estimated to be higher. 56 00:03:31,223 --> 00:03:35,753 A big input for those costs is how many, I'm gonna say Rose. 57 00:03:35,753 --> 00:03:36,233 I guess. 58 00:03:36,263 --> 00:03:43,763 Uh, this might be another argument for Tules, uh, are being returned at each, um, or tuple, sorry, back to that old one. 59 00:03:44,138 --> 00:03:44,558 Nikolay: Yeah. 60 00:03:44,888 --> 00:03:46,058 Your version isles. 61 00:03:46,058 --> 00:03:48,578 I propose Tules, but you switched. 62 00:03:48,608 --> 00:03:48,878 Okay. 63 00:03:49,148 --> 00:03:56,583 So, if you see nest loop and, we think it should be different, we suspect, , some issues with statistics, right? 64 00:03:58,078 --> 00:04:04,178 and first thing to check is to try to analyze, the table to recollect statistics or no? 65 00:04:04,448 --> 00:04:04,988 What do you think? 66 00:04:05,558 --> 00:04:19,488 Michael: Well, this is, I think where we get into it's probably worth us explaining up top, analyze, I guess what you're talking about there is the process Postgres has for gathering statistics so you can analyze a relation. 67 00:04:19,488 --> 00:04:25,908 I learned today actually looking into it, you can analyze just a single column, and there's a lot of flexibility 68 00:04:25,983 --> 00:04:26,973 Nikolay: Or expression 69 00:04:27,873 --> 00:04:28,383 Michael: Nice. 70 00:04:28,443 --> 00:04:29,253 Okay, cool. 71 00:04:30,123 --> 00:04:37,053 , and also it, it happens automatically as part of auto vacuum or auto analyze, I guess is the, part of 72 00:04:37,063 --> 00:04:37,793 Nikolay: Well, yeah. 73 00:04:37,793 --> 00:04:40,913 Auto vacuum has three, three, um, tasks. 74 00:04:41,383 --> 00:04:41,803 Michael: Yep. 75 00:04:42,843 --> 00:04:43,673 Nikolay: So vacuuming. 76 00:04:44,418 --> 00:04:47,418 preventing, preventing transaction idea routes or freezing. 77 00:04:47,448 --> 00:04:55,128 And also third option is, uh, o auto analyzing like recollecting statistics. 78 00:04:55,938 --> 00:04:56,358 And 79 00:04:56,718 --> 00:04:57,528 Michael: With two, 80 00:04:58,128 --> 00:05:01,598 Nikolay: yeah, VA, what, what is fourth? 81 00:05:01,788 --> 00:05:03,708 Michael: Does the VisibiliT map count as 82 00:05:03,808 --> 00:05:07,788 Nikolay: Well, yeah, but it's can be considered as a part of vacuum. 83 00:05:08,748 --> 00:05:09,318 Michael: Okay, cool. 84 00:05:10,434 --> 00:05:19,034 Nikolay: Maintaining free face map and visibility map, but, uh, when analyzing, analyzing might happen together with auto vacuuming, but it may happen also. 85 00:05:19,830 --> 00:05:22,950 Like auto, auto automatizing, uh, separately. 86 00:05:23,520 --> 00:05:26,309 So everything happens in inside box, right? 87 00:05:27,119 --> 00:05:28,919 Many options, but, right. 88 00:05:28,919 --> 00:05:34,020 So not every time auto Im runs to process some table electrical statistics. 89 00:05:34,020 --> 00:05:36,934 It, it, it's controlled by several settings. 90 00:05:36,984 --> 00:05:39,844 Threshold scale factor, related to analyze part 91 00:05:40,909 --> 00:05:43,009 Michael: Yeah, and I think it's fairly recent. 92 00:05:43,014 --> 00:05:50,929 I didn't actually check, but one of the newer versions lets us set an analyze scale factor even for insert only workloads. 93 00:05:50,929 --> 00:05:51,559 So times 94 00:05:51,709 --> 00:05:52,069 Nikolay: right? 95 00:05:52,069 --> 00:05:52,849 There was a problem. 96 00:05:53,119 --> 00:05:59,569 There was a problem for append only use cases when you just add, add a lot of data and that's it. 97 00:05:59,669 --> 00:06:02,196 And, uh, , WM didn't process it. 98 00:06:02,251 --> 00:06:07,601 well, VM probably is not needed because you don't produce delete, uh, tops if you just ensure right. 99 00:06:07,661 --> 00:06:09,621 But, uh, Aly. 100 00:06:10,281 --> 00:06:12,971 Recollecting statistics is definitely what I would like to have. 101 00:06:13,361 --> 00:06:21,491 And, , actually, many examples, educational examples and so on, they, they show like, okay, let's load something to a table and. 102 00:06:22,010 --> 00:06:28,374 before, uh, you, like you, you must, uh, run, analyze yourself to have, proper plans. 103 00:06:28,974 --> 00:06:31,644 But now you can just wait a little bit, right? 104 00:06:32,544 --> 00:06:35,394 and it will do a job and then you can play with it. 105 00:06:35,399 --> 00:06:41,124 But of course, uh, if you want to control, probably you just need to say, analyze a table name and that's it. 106 00:06:42,267 --> 00:06:42,657 Michael: Yep. 107 00:06:42,837 --> 00:06:44,817 So actually the reason I wanted to. 108 00:06:45,102 --> 00:06:49,982 Nikolay: 11 or 12 when it was like couple, couple of versions ago, maybe three versions ago. 109 00:06:51,072 --> 00:06:52,692 Michael: Wow, that's getting quite a long time ago. 110 00:06:52,965 --> 00:06:54,135 I remember when that was new. 111 00:06:54,225 --> 00:06:54,525 Even 112 00:06:54,545 --> 00:06:58,875 Nikolay: I, I remember dfe, I, I, I, I cannot pronounce the last name, sorry. 113 00:06:58,965 --> 00:07:07,536 Dar Dfe participated in this, discussion and, uh, and he, he, he was also in, uh, active in the Russian speaking community as well. 114 00:07:07,536 --> 00:07:11,656 So I remember this case and it was quite like, so obvious improvement. 115 00:07:11,661 --> 00:07:18,412 Like it was like one of those, changes when, when you have feeling, how come it didn't happen before, right? 116 00:07:19,367 --> 00:07:19,757 Michael: Yep. 117 00:07:20,357 --> 00:07:21,167 But that's the nice thing. 118 00:07:21,167 --> 00:07:22,337 It does keep getting better. 119 00:07:22,337 --> 00:07:35,244 And the, the reason I wanted to, be specific about the analyze you were talking about there is that there's, a few places people might see that word and there's analyze itself as a, as a, a keyword that gathers statistics. 120 00:07:35,514 --> 00:07:37,254 You might see vacuum analyze. 121 00:07:37,784 --> 00:07:41,264 as like a kind of parameter to vacuum that is the same thing. 122 00:07:41,294 --> 00:07:52,944 So it, that's doing both, vacuum and analyze, but you might also see, explain, analyze, which I guess we're gonna talk about in a moment, uh, which is not related at all, just totally different thing. 123 00:07:53,367 --> 00:08:00,684 so naming things is difficult and, sadly that's just one we need to, like, once you, once you know what it does, you realize it's unrelated. 124 00:08:00,714 --> 00:08:02,921 But, it can be confusing for beginners. 125 00:08:03,646 --> 00:08:04,006 Nikolay: right. 126 00:08:04,086 --> 00:08:06,456 terms, words are overloaded everywhere. 127 00:08:06,936 --> 00:08:09,636 , the statistics term was also overloaded. 128 00:08:09,636 --> 00:08:13,242 We can talk about statistics, double statistics, and the user statistics. 129 00:08:13,542 --> 00:08:19,733 This, statistics, statistics which collects, which keeps, the column and expression stats. 130 00:08:19,733 --> 00:08:23,103 And this is what is calculated when you run, analyze. 131 00:08:23,513 --> 00:08:26,993 And also in, in explain plan, we also see some statistics, right? 132 00:08:27,053 --> 00:08:32,543 We can also call it some, so like the, the terms are overloaded. 133 00:08:33,168 --> 00:08:36,948 there, there are too few words in in language, unfortunately in any language. 134 00:08:37,357 --> 00:08:38,017 but, okay. 135 00:08:38,227 --> 00:08:48,072 So, back to our topic, if we see nested loop and we say we, we would like to have something else like probably, we. 136 00:08:49,217 --> 00:08:52,307 Need to check it with, analyze, like, what will you do? 137 00:08:52,337 --> 00:08:55,607 Like just run, analyze, and double check the plan or what 138 00:08:56,522 --> 00:08:59,252 Michael: Yeah, so there's a, there's a, so I think goes back a step. 139 00:08:59,252 --> 00:09:01,112 How are you even spotting that problem? 140 00:09:01,112 --> 00:09:06,242 I would suggest that you're, you should be running, explain analyze, and trying to work. 141 00:09:06,302 --> 00:09:08,582 So that's faint analyzed buffers. 142 00:09:08,582 --> 00:09:08,762 Yeah. 143 00:09:08,762 --> 00:09:11,222 Ideally, maybe even VERBOs format. 144 00:09:11,252 --> 00:09:12,302 Jason maybe. 145 00:09:12,752 --> 00:09:13,052 No. 146 00:09:13,052 --> 00:09:13,442 Um, 147 00:09:13,637 --> 00:09:17,147 Nikolay: on Twitter there was a new saying like in buffers with. 148 00:09:18,752 --> 00:09:19,352 Michael: I like it. 149 00:09:20,092 --> 00:09:22,802 I think I saw you had a t-shirt like that as well, so that's fun. 150 00:09:23,672 --> 00:09:23,852 Nikolay: Yep. 151 00:09:23,942 --> 00:09:30,692 So, so, so when explaining last buffers to check before you run, analyze to re recollect statistics. 152 00:09:31,007 --> 00:09:47,719 Michael: Yeah, but specifically explainer alone won't help us here because explainer alone only gives us the estimated number of, rows being returned at each stage of the execution plan, but with, explain, analyze buffers, but explain, analyzes the important part for this topic. 153 00:09:48,304 --> 00:10:02,773 Because that'll give us not only the estimated number of rows, but also the actual number of rows, being returned and it's mismatches there that can really, help us diagnose where this problems are rising, where the bad estimates are. 154 00:10:03,127 --> 00:10:04,717 so yeah, that would 155 00:10:04,742 --> 00:10:07,622 Nikolay: me, let me, let me, let me disagree with you here. 156 00:10:08,142 --> 00:10:09,602 At least, at least partially. 157 00:10:09,662 --> 00:10:20,022 Uh, you are talking about, , like step forward examples when we can run, explain large buffer so we see all details both for planning and execution and we understand something is wrong. 158 00:10:20,202 --> 00:10:26,262 We don't want to have nest loop here, but what if our execution takes a hundred years? 159 00:10:27,062 --> 00:10:27,942 Michael: Yeah, of course. 160 00:10:28,387 --> 00:10:34,477 Nikolay: So if you say like, just explain is not, is not helpful, it's still helpful. 161 00:10:34,477 --> 00:10:35,977 It'll show nested loop. 162 00:10:35,977 --> 00:10:40,577 It will give us, understanding how planner thinks about our data. 163 00:10:41,057 --> 00:10:42,647 And we know our data, right? 164 00:10:42,652 --> 00:10:44,717 We know how many rows there and there. 165 00:10:44,717 --> 00:10:49,667 So we say like, no, I would, I would if I, if I were you, apply the planner, right? 166 00:10:49,667 --> 00:10:51,107 I I would do it differently. 167 00:10:51,107 --> 00:10:53,687 So just explain in some cases and. 168 00:10:54,572 --> 00:10:59,622 And, and it also can be help helpful right in, in some extreme cases when execution is long. 169 00:11:00,212 --> 00:11:01,082 Michael: Very, very good point. 170 00:11:01,172 --> 00:11:01,442 Yep. 171 00:11:01,492 --> 00:11:02,272 Completely agree. 172 00:11:03,002 --> 00:11:05,492 But then I agree with you about the second port of call. 173 00:11:05,552 --> 00:11:24,922 It's often if you can see the relations involved and you can run and, and you think maybe analyze hasn't run recently on those, and you can just, uh, it, there's, it is not a locking operate or it has very, very light locks so you can run, analyze, Tables in productional columns even, with, with very low risk. 174 00:11:25,042 --> 00:11:27,952 And that might be, that might be enough to solve your problem. 175 00:11:28,132 --> 00:11:30,472 Uh, it might be that the statistics were out of date. 176 00:11:30,892 --> 00:11:31,702 Nikolay: yeah, we can check. 177 00:11:31,702 --> 00:11:36,682 We can, we can check how, when analyze, uh, when. 178 00:11:37,672 --> 00:11:46,012 In logs if we, if, if auto, auto is, is log is enabled and in recent versions threshold is 10 minutes, it should be zero, always zero. 179 00:11:46,012 --> 00:11:46,462 Zero. 180 00:11:46,462 --> 00:11:52,822 Like all, everything, all rows, all and log interest related to auto vacuum should be there, in my opinion. 181 00:11:53,342 --> 00:12:03,769 But we also have produced that all tables at user tables and we can check, , it has four, columns, two pairs, one pair for vacuuming and one pair for. 182 00:12:04,651 --> 00:12:05,221 timestamps. 183 00:12:05,611 --> 00:12:06,961 All, all pairs are timestamps. 184 00:12:07,021 --> 00:12:08,401 I'm, I'm not talking about count. 185 00:12:08,851 --> 00:12:11,608 Count is quite silly, metric, right? 186 00:12:11,608 --> 00:12:12,318 So, okay. 187 00:12:12,318 --> 00:12:17,188 Five, five times, well, zero means something, but five, 10 when, right. 188 00:12:17,548 --> 00:12:19,708 So two pairs of timestamps. 189 00:12:19,708 --> 00:12:27,704 One pair for vacuum, one is for auto vacuum, one is for manual vacuum, and two, times stepss for analyze, one for auto. 190 00:12:29,279 --> 00:12:37,469 means like analyze job of auto com and second is for manual analyze and uh, we can see when it happened, right? 191 00:12:37,799 --> 00:12:39,269 It might still be happening. 192 00:12:39,269 --> 00:12:45,989 So also we're checking PPG start activity and you will see it there. 193 00:12:45,989 --> 00:12:46,739 Auto com. 194 00:12:46,744 --> 00:12:51,029 Uh, colon analyze or vacuum analyze so you 195 00:12:51,329 --> 00:12:52,109 Michael: Yeah, good point. 196 00:12:52,109 --> 00:12:53,299 It could be happening right now. 197 00:12:54,599 --> 00:12:56,549 Nikolay: activity the query. 198 00:12:57,199 --> 00:13:03,629 , it has, it has details for auto, vacuum jobs as well, so it's, it's, it's seen as regular session, 199 00:13:05,099 --> 00:13:11,049 Michael: What's the like if on huge databases, how long do you tend to see analyzed taking out? 200 00:13:11,669 --> 00:13:17,569 Nikolay: Well, it depends on, on, how detailed you want your statistics to be. 201 00:13:17,869 --> 00:13:30,369 By default, we have, default statistics target hundred, like meaning a hundred buckets, but many people increase it, for example, 2000 sometimes more, and the bigger default statistics. 202 00:13:31,419 --> 00:13:37,539 Is the longer, analyze will be, and it'll, it'll produce a noticeable, disc cayo. 203 00:13:37,539 --> 00:13:47,049 Of course, the reading data, if it's not cashed in, in the buffer pool and, and or, uh, p cash, and also individual, columns can be tuned. 204 00:13:47,519 --> 00:13:55,289 So Defo statistics target, it's like cluster wide setting, but some columns can be adjusted additionally with Al Table, I don't remember, like, 205 00:13:56,269 --> 00:13:57,359 Michael: Or statistics. 206 00:13:57,364 --> 00:13:57,779 Yeah. 207 00:13:57,869 --> 00:13:58,289 Nikolay: like that. 208 00:13:58,289 --> 00:14:02,999 You can say, I want a thousand buckets here, but a hundred buckets there and only 10 there. 209 00:14:03,479 --> 00:14:05,399 Something like this, it's like fine tuning. 210 00:14:05,549 --> 00:14:11,019 Usually I recommend like, avoid it unless you are a hundred percent do no. 211 00:14:11,409 --> 00:14:15,489 In my, in my experience, I saw cases when people tuned it. 212 00:14:16,299 --> 00:14:22,002 But they didn't we are not, um, very, clear in their intentions. 213 00:14:22,002 --> 00:14:29,292 You know, it was like kind of experimenting and then it was abandoned and then you just have some leftover of past experiments. 214 00:14:29,297 --> 00:14:29,712 That's it. 215 00:14:30,192 --> 00:14:34,182 So, uh, systematic approach should be like, you know, what you are doing. 216 00:14:34,182 --> 00:14:37,602 And here we definitely want more detailed statistics. 217 00:14:37,872 --> 00:14:40,392 So analyze can, depends on. 218 00:14:40,792 --> 00:14:46,186 Table size and on number of buckets you want to collect for each, expression or, or comb. 219 00:14:47,121 --> 00:14:47,421 Michael: Yeah. 220 00:14:47,511 --> 00:14:50,031 So that's in fact going on to the statistics target. 221 00:14:50,031 --> 00:15:09,594 I think that's a useful, next, so if Analyze doesn't fix your problem, there is, there's something interesting if you, if you are aware now where the problem is, then if you, if you know the distribution of that data, if you've got a very skewed distribution, that's where I see, Extending or So increasing that target? 222 00:15:09,624 --> 00:15:09,864 What? 223 00:15:10,344 --> 00:15:13,251 Well, I, I don't know if I've, I have the same opinion as you. 224 00:15:13,251 --> 00:15:21,141 I, I personally don't think it's as sensible to change the global statistics target from, like increasing it from a hundred to a thousand. 225 00:15:21,141 --> 00:15:29,111 But, but I've, I've seen some people have success with doing it at a, you know, with specific columns that they know are, distributed a certain way. 226 00:15:29,381 --> 00:15:32,431 Nikolay: before I started to work with, uh, thin. 227 00:15:33,233 --> 00:15:36,939 . I worked on the, holistic approach to this problem. 228 00:15:36,939 --> 00:15:43,756 So for example, you say, okay, I have my database, I have many clones of it, and I have, I have my workload. 229 00:15:43,756 --> 00:15:49,186 I can reproduce it reliably, like definitely can do it, at least, uh, part of it, right? 230 00:15:49,666 --> 00:15:54,376 And then I, we, we were able to run, for example, 10 vm. 231 00:15:55,591 --> 00:15:57,391 uh, in parallel to justice. 232 00:15:57,481 --> 00:15:59,671 We can do it sequentially, of course, but why? 233 00:15:59,731 --> 00:16:01,501 If we can do it in parallel, why not? 234 00:16:02,191 --> 00:16:05,641 By the way, there will be question, do we have the same hardware everywhere? 235 00:16:05,641 --> 00:16:14,571 So, so we need, we also added some micro benchmarks to compare like baseline cpu, same disk, same and so on, but to, to. 236 00:16:14,931 --> 00:16:18,646 Pluralization is of benchmarks, is quite interesting topic itself. 237 00:16:19,186 --> 00:16:27,674 So, and then you just compare, you, you take, uh, 10 values for your knob in this case. 238 00:16:27,936 --> 00:16:32,256 Uh, the full statistics target and you then you compare many, many things. 239 00:16:32,256 --> 00:16:35,286 For example, starting from regular latency and throughput. 240 00:16:35,796 --> 00:16:42,036 You can also use producer statements to analyze particular query groups and see deviations. 241 00:16:42,036 --> 00:16:46,551 Uh, And it's, it's, it's very, this, this is very good holistic approach. 242 00:16:46,821 --> 00:16:47,331 By the way. 243 00:16:47,331 --> 00:16:50,227 It's also possible to do it, for using thin loans. 244 00:16:50,227 --> 00:16:59,827 But you need to focus of course on buffers, not on timing because, uh, timing will be quite volatile metric, but buffers will be, reliable. 245 00:16:59,827 --> 00:17:08,837 So it's possible to, to do something there and see how, amount of work in terms of iOS or buffers will change depending on the four statistics target. 246 00:17:08,897 --> 00:17:18,861 And in this approach, uh, the goal was, okay, we want to improve queries and if we change global value, which queries will be improved and which queries maybe will. 247 00:17:19,721 --> 00:17:20,771 , we'll have some penalty. 248 00:17:20,771 --> 00:17:20,981 Right? 249 00:17:21,941 --> 00:17:25,781 Of course, you need to also to compare, analyze in this experiment. 250 00:17:25,781 --> 00:17:26,951 And this is holistic approach. 251 00:17:26,951 --> 00:17:28,181 Quite interesting actually. 252 00:17:28,181 --> 00:17:29,891 If I like, I like this approach. 253 00:17:29,891 --> 00:17:45,701 It's like I, I consider such approaches as enterprise ready because, uh, you answer all questions, cover all topics here, and then you, you can make decision with a lot of data, but, uh, it requires efforts of course, right. 254 00:17:46,516 --> 00:17:47,256 Michael: And, and. 255 00:17:48,341 --> 00:17:55,511 I haven't seen as many cases where the, where other queries are slowed down in these cases. 256 00:17:55,511 --> 00:17:57,521 Definitely in other performance cases. 257 00:17:57,671 --> 00:18:04,001 But I hadn't considered a really good point you made around the speed of analyze being very de 258 00:18:04,781 --> 00:18:05,201 Nikolay: Well, 259 00:18:05,236 --> 00:18:07,391 Michael: this and, and then what? 260 00:18:07,391 --> 00:18:13,267 That's, that's crucial for things like, major version upgrades because if we, when we do a major 261 00:18:13,432 --> 00:18:14,562 Nikolay: plan changes. 262 00:18:14,567 --> 00:18:15,382 Plan changes. 263 00:18:15,412 --> 00:18:17,962 Michael: Well, we, we don't have statistics, do we? 264 00:18:17,962 --> 00:18:19,852 We don't get the statistics automatically. 265 00:18:19,852 --> 00:18:24,025 So we have to run, analyze, before we can like, as part of the downtime. 266 00:18:24,175 --> 00:18:26,785 But there's no, I don't see you any way around that. 267 00:18:26,935 --> 00:18:40,042 So if we've now increased our statistics target, to a point where it's gonna take half an hour to run, analyze across our entire database, then that's another half an hour of downtime when every time we do a major version upgrade. 268 00:18:40,072 --> 00:18:40,932 Nikolay: That's, that's not. 269 00:18:41,677 --> 00:18:42,577 Michael: No, exactly. 270 00:18:42,577 --> 00:18:54,677 So it's, it's considering these other, I, I hadn't thought so much about that, but that makes me think even more that the kind of going after each, only increasing on a kind of case by case basis might be more sensible. 271 00:18:55,187 --> 00:18:55,547 Nikolay: right? 272 00:18:55,667 --> 00:18:58,067 And, and, and analyzing sta in stages. 273 00:18:58,067 --> 00:19:00,347 I saw cases when, uh, didn't help. 274 00:19:00,347 --> 00:19:04,997 So like to sit in some intermediate state, it was not good at all. 275 00:19:04,997 --> 00:19:11,044 So, conclusion was, let's do the last step, like maximum statistics and just wait more. 276 00:19:11,044 --> 00:19:11,494 And that's it. 277 00:19:12,334 --> 00:19:26,633 It would be great if pos uh, upgrade, PJ upgrade would, um, just expert and import PPG statistic, understanding changes if they are happen between, uh, versions by, by the way, back to like, I, I, I. 278 00:19:27,171 --> 00:19:38,281 I wanted to, uh, interrupt in, interrupt you in one more place, but I didn't, so you mentioned that, uh, if we have issues, probably we need to, to change this. 279 00:19:38,281 --> 00:19:42,451 We, like we, we started to discuss, uh, uh, this default statistics target. 280 00:19:42,511 --> 00:19:46,118 Uh, no, but, maybe there is another mitigation action. 281 00:19:46,118 --> 00:19:50,408 For example, we can tune not to vacuum, to recollect statistics more often. 282 00:19:51,348 --> 00:19:51,678 Michael: Yep. 283 00:19:53,658 --> 00:19:58,982 Nikolay: I would start from there, maybe not from, from, change of, uh, bucket number of buckets. 284 00:20:00,242 --> 00:20:00,512 Michael: Yeah. 285 00:20:00,572 --> 00:20:02,942 Well, and it depends a little bit on the problem, right? 286 00:20:02,942 --> 00:20:09,841 Depends if it's a distribution issue or if it's a data being a stale issue or if it's a correlation issue. 287 00:20:09,931 --> 00:20:11,971 So there's the, another I 288 00:20:12,001 --> 00:20:13,291 Nikolay: we go there, before we 289 00:20:13,291 --> 00:20:13,681 Michael: oh, go on. 290 00:20:14,254 --> 00:20:14,614 Nikolay: right. 291 00:20:14,644 --> 00:20:19,064 So it's a question what it is it and answer question. 292 00:20:19,064 --> 00:20:19,894 We need to experiment. 293 00:20:20,494 --> 00:20:22,054 That's why I started. 294 00:20:22,684 --> 00:20:25,834 Will you do an analyze manual analyze to. 295 00:20:27,241 --> 00:20:27,361 . Right? 296 00:20:27,451 --> 00:20:27,661 Okay. 297 00:20:27,661 --> 00:20:32,052 We see that, OWA did it yesterday and we inserted a lot, for example. 298 00:20:32,922 --> 00:20:36,252 So obviously we should run manual analyze, right? 299 00:20:37,122 --> 00:20:40,722 And, and double check the plan, compare before and after, right? 300 00:20:41,697 --> 00:20:45,597 Michael: I mean, I feel like I, I want, I want to say yes, but equally I feel like it's a trap. 301 00:20:46,872 --> 00:20:49,062 Nikolay: You feel it, eh, you because you know me already. 302 00:20:49,062 --> 00:20:49,302 Right. 303 00:20:49,362 --> 00:20:50,922 But it, it is, it is trap. 304 00:20:50,922 --> 00:20:51,432 It is trap. 305 00:20:51,822 --> 00:20:55,122 Well, 99.9% of GBS will will do it. 306 00:20:55,122 --> 00:21:02,892 And I personally will do it as well, but I don't like it because, if I late, like we, we want to understand what's happening. 307 00:21:02,952 --> 00:21:03,192 Right. 308 00:21:03,282 --> 00:21:04,902 And this is one way ticket. 309 00:21:05,022 --> 00:21:05,502 Always. 310 00:21:06,432 --> 00:21:07,602 Michael: you can't undo it, you mean? 311 00:21:07,752 --> 00:21:08,382 Nikolay: Exactly. 312 00:21:09,072 --> 00:21:10,872 Michael: But we do then know what the problem was. 313 00:21:10,872 --> 00:21:11,712 At least then we know 314 00:21:11,802 --> 00:21:19,662 Nikolay: Well, right, but, but how can you be a hundred percent sure that you won't have any questions to previous version of statistics? 315 00:21:19,667 --> 00:21:24,102 Maybe you have, maybe you would, you would like to check other plants as well, right? 316 00:21:24,492 --> 00:21:25,452 You not randomized. 317 00:21:25,482 --> 00:21:25,992 And that's it. 318 00:21:26,112 --> 00:21:27,222 Doors clo closed. 319 00:21:27,342 --> 00:21:30,242 Bye-Bye, . It's not good, right? 320 00:21:30,282 --> 00:21:30,972 That's why. 321 00:21:31,002 --> 00:21:31,572 That's why. 322 00:21:31,752 --> 00:21:32,382 Guess what? 323 00:21:33,372 --> 00:21:33,852 Think loss. 324 00:21:34,857 --> 00:21:37,677 Michael: well, just to, just to question that slightly. 325 00:21:37,677 --> 00:21:42,157 If we have a point in time recovery, for example, and we. 326 00:21:43,647 --> 00:21:49,180 Then res if we want to, let's say, restore to a previous time before we ran analyzed. 327 00:21:49,180 --> 00:21:51,190 Can we get the old statistics back? 328 00:21:51,310 --> 00:21:51,880 Nikolay: exactly. 329 00:21:51,885 --> 00:21:52,330 Yes. 330 00:21:52,330 --> 00:21:55,660 The only problem with this approach is just like terabyte per hour. 331 00:21:56,090 --> 00:21:56,580 Michael: Yeah. 332 00:21:56,585 --> 00:21:56,620 Yeah. 333 00:21:56,680 --> 00:21:56,830 Okay. 334 00:21:57,520 --> 00:21:57,970 Nikolay: That's it. 335 00:21:57,970 --> 00:22:02,620 So if you have 10 terabytes, wait 10 hours for to, to try again, and then you run, analyze. 336 00:22:02,620 --> 00:22:04,750 And that's why I think loans and database like. 337 00:22:05,690 --> 00:22:09,460 Michael: Yeah, so I think my answer to your, your question is still yes. 338 00:22:09,460 --> 00:22:10,060 If it's. 339 00:22:10,495 --> 00:22:14,035 If this is like a production issue, we don't like, 340 00:22:14,470 --> 00:22:16,260 Nikolay: uh, we, we need to heal us up. 341 00:22:16,260 --> 00:22:16,500 Yeah. 342 00:22:16,705 --> 00:22:17,605 Michael: Yeah, exactly. 343 00:22:18,325 --> 00:22:19,165 I would, I would. 344 00:22:19,195 --> 00:22:27,560 But equally, really good point and worth thinking, worth take, taking a few, a split second to think about the consequences before hitting, , execute. 345 00:22:27,845 --> 00:22:31,291 Nikolay: oh, by the way, is analyze transactional. 346 00:22:32,276 --> 00:22:32,766 Michael: I don't. 347 00:22:32,941 --> 00:22:36,091 Nikolay: In this case, you can detach one of clones. 348 00:22:36,751 --> 00:22:37,051 Right? 349 00:22:37,081 --> 00:22:38,011 Make it, uh, read, 350 00:22:38,356 --> 00:22:39,646 Michael: roll back. 351 00:22:40,291 --> 00:22:41,881 Nikolay: Well, yes, exactly. 352 00:22:41,881 --> 00:22:44,431 I think it is because it's just a P statistics table. 353 00:22:44,431 --> 00:22:48,541 It should be, I don't remember a hundred percent, but it should be an transactional. 354 00:22:48,541 --> 00:22:51,121 So you just begin, analyze, check your plan, roll back. 355 00:22:51,871 --> 00:22:52,231 Why not? 356 00:22:53,911 --> 00:22:54,381 It should. 357 00:22:55,496 --> 00:22:55,986 Michael: Cool. 358 00:22:56,056 --> 00:22:56,686 Very well. 359 00:22:56,741 --> 00:22:58,891 Nikolay: maybe, maybe I'm, I'm terribly wrong, but 360 00:23:00,121 --> 00:23:01,021 Michael: It's a great idea. 361 00:23:01,021 --> 00:23:02,281 Someone, someone check it. 362 00:23:02,371 --> 00:23:03,851 Well one of us will check it and we'll 363 00:23:03,901 --> 00:23:04,171 Nikolay: right. 364 00:23:04,291 --> 00:23:09,191 In this case, you cannot rate and, and, uh, return, , basically results, statistics once again. 365 00:23:09,191 --> 00:23:14,921 And, and, uh, check using regular, traditional post not thin loans, which I, I like. 366 00:23:15,700 --> 00:23:16,376 so, okay. 367 00:23:16,431 --> 00:23:24,463 this is recover bit, like how we can play, with many different query plants and see what's happening. 368 00:23:24,463 --> 00:23:29,323 Uh, if we, now, now let's talk about, uh, correlation. 369 00:23:29,433 --> 00:23:31,490 You, you wanted to talk about. 370 00:23:32,185 --> 00:23:36,755 Michael: Well, yeah, I guess we've only covered a couple of the different ways that this could be a problem. 371 00:23:37,655 --> 00:23:49,020 Another famous one, I think we have discussed it on here before, is, let's say we are looking at two columns that are highly correlated and by default, Postgres is gonna assume I'll go on. 372 00:23:49,560 --> 00:23:56,400 Nikolay: each time I try to invent some example, I'm becoming either racist or sexist or something I, I 373 00:23:56,505 --> 00:23:57,975 Michael: Well, I've got a good one for you. 374 00:23:58,095 --> 00:24:00,375 Car, car makes, car makes and models. 375 00:24:00,375 --> 00:24:17,665 For example, if we say where, where car make is, Toyota and car model is Prius, those are gonna be extremely highly correlated, I'm not aware of any, Prius that aren't Toyotas, but by default, Postgres is gonna assume that those are. 376 00:24:18,885 --> 00:24:35,635 Uh, independent ver variables and is gonna look at the proportion of rows that are Toyotas if they're in the most common values and proportion they're in the pr , the app Prius is and work out a, uh, a much lower estimate of the total than, than actually the case. 377 00:24:35,875 --> 00:24:38,035 So that's, probably the. 378 00:24:39,100 --> 00:24:42,160 Well currently the best use of extended statistics. 379 00:24:42,160 --> 00:24:45,520 So we can now think thanks to people like Thomas Fundra. 380 00:24:45,520 --> 00:24:47,320 I think there's a few more people involved. 381 00:24:47,654 --> 00:24:52,754 as of a few versions ago, we can now spec, like we can educate Postgres about these. 382 00:24:53,489 --> 00:24:54,679 Nikolay: Special object. 383 00:24:55,319 --> 00:25:03,999 There are three are three options, and in this case, uh, I don't remember the words, but like something about distinct, various, most common values. 384 00:25:03,999 --> 00:25:07,975 And the third is about this, uh, like, one like depends on another. 385 00:25:08,535 --> 00:25:08,715 , right? 386 00:25:08,715 --> 00:25:16,815 So functionally dependent or in, in this case, I would choose that third option because obviously Prius means Toyota. 387 00:25:17,025 --> 00:25:22,510 So probably, and uh, if we don't do that, just multiplies, okay, we have. 388 00:25:23,660 --> 00:25:25,520 Like 20% of Toyotas. 389 00:25:25,970 --> 00:25:31,160 Okay, we have 1% of Prius, like multiplication, very low number in this case. 390 00:25:31,640 --> 00:25:32,660 Bad idea obviously. 391 00:25:32,810 --> 00:25:41,536 And also if our query includes, for example, I don't know, like, Nissan and Prius, like obviously we should have zero. 392 00:25:42,466 --> 00:25:42,886 Michael: Yep. 393 00:25:42,996 --> 00:25:46,566 Nikolay: uh, POGS will tell multiplication will give some non-zero number. 394 00:25:46,571 --> 00:25:48,546 Also wrong in this case. 395 00:25:48,546 --> 00:25:48,936 Definitely. 396 00:25:49,146 --> 00:25:53,436 But the question is how to, like, how to, okay. 397 00:25:53,496 --> 00:25:57,101 Again, I, I usually, try to find some holistic approach, right? 398 00:25:57,131 --> 00:25:59,681 How to build some framework. 399 00:26:00,564 --> 00:26:01,464 you are right by the way. 400 00:26:01,464 --> 00:26:03,174 That's obviously. 401 00:26:03,888 --> 00:26:08,178 , if we just statistics or increase number of buckets. 402 00:26:08,628 --> 00:26:11,778 Uh, the no, none of queries, uh, should slow down. 403 00:26:12,228 --> 00:26:13,698 It should only help, right? 404 00:26:14,418 --> 00:26:16,458 Michael: I think so, but I'm not a hundred percent sure. 405 00:26:16,578 --> 00:26:17,058 Nikolay: right. 406 00:26:17,538 --> 00:26:24,408 But, the framework, uh, why I talked about it, the framework also ask question, are there any parts of our workload, which. 407 00:26:25,938 --> 00:26:28,218 From our change, this is the idea. 408 00:26:28,218 --> 00:26:35,148 We should check everything and ensure that we improved something, but we have everything else. 409 00:26:35,148 --> 00:26:36,198 At least the same. 410 00:26:36,228 --> 00:26:38,028 May not improved, but at least the same. 411 00:26:38,538 --> 00:26:41,421 And, here I also, ask holistic question. 412 00:26:41,688 --> 00:26:44,146 how to understand, that we needed. 413 00:26:44,725 --> 00:26:48,415 . we need to guess that this column depends on that, or we 414 00:26:48,645 --> 00:26:48,885 can 415 00:26:50,340 --> 00:26:54,960 Michael: but based on the query plan, we can often see that that's where a misestimate is happening, right? 416 00:26:54,960 --> 00:27:02,125 Like, I mean, I, I'm definitely somebody who's guilty of having a hammer, and then everything looks like a nail. 417 00:27:02,305 --> 00:27:07,465 So that's my, like, that's the typically the way I've seen it, but be easier to, to spot. 418 00:27:08,224 --> 00:27:09,904 so yeah, that, that would be my point. 419 00:27:09,904 --> 00:27:12,634 But where, where is that the place you'd look as well? 420 00:27:13,489 --> 00:27:14,899 Nikolay: I have just question. 421 00:27:14,899 --> 00:27:15,499 No, no answers. 422 00:27:16,054 --> 00:27:16,474 Michael: Okay. 423 00:27:17,164 --> 00:27:23,884 Well, we, um, because this is a common issue, inquiry plans, and it's, it's not always clear from the query plan. 424 00:27:24,679 --> 00:27:30,439 Where the issue is, we, we have a tip in PG mustard for particularly bad row estimates. 425 00:27:30,529 --> 00:27:36,739 Especially when, well, only when there's, they're part of a slow part of the query plan, a slow subry. 426 00:27:37,216 --> 00:27:46,696 and we link out to a blog post that we've written going through, kind of like helping people try this, then try that, or, you know, this going through basically the things we've discussed here. 427 00:27:46,696 --> 00:27:51,306 So I will link it up, but I think I need to update the blog post based on a few things we've mentioned here. 428 00:27:51,552 --> 00:27:52,962 it's a few years old 429 00:27:53,812 --> 00:27:55,312 Nikolay: Yeah, so, so I like that. 430 00:27:55,312 --> 00:28:03,882 Uh, you also started to, uh, advertise your product because we, we invest a lot of efforts to try to help, people and to improve things, tooling and so on. 431 00:28:04,092 --> 00:28:07,300 I think, this, great statistics is, very, very under. 432 00:28:08,059 --> 00:28:11,645 because you need to make a lot of efforts to get into there. 433 00:28:11,645 --> 00:28:21,740 So if, for example, uh, explain lies would suggest, or like if pigeon Masters suggests, more explicitly what to do, uh, more people would, will, would, start using it, right? 434 00:28:22,130 --> 00:28:23,658 So we need some, help. 435 00:28:24,005 --> 00:28:24,906 that would, do it. 436 00:28:25,146 --> 00:28:34,388 Actually, we don't have helpers for indexes, so it's maybe, and it's still like, rather some works in progress, uh, trying to improve and they do it. 437 00:28:34,604 --> 00:28:36,691 but, very interesting que questions. 438 00:28:37,256 --> 00:28:37,556 Michael: Yeah. 439 00:28:38,636 --> 00:28:51,580 Another wall, another thing that co the, the last thing I wanted to make sure we covered as part of this, it feels, odd not to, is that sometimes people get to the end of their tether with this and, and, you know, maybe ex uh, create statistics doesn't yet support their use case. 440 00:28:51,580 --> 00:28:55,570 Like we've talked before about two correlated columns, but from different tables. 441 00:28:55,980 --> 00:28:59,700 I think you mentioned quite a good idea using materialized views for that. 442 00:28:59,790 --> 00:29:00,895 But,, If, if and 443 00:29:01,180 --> 00:29:05,020 Nikolay: As a, as a breach, uh, to have an index on columns from different tables. 444 00:29:05,020 --> 00:29:05,350 Yes. 445 00:29:05,350 --> 00:29:10,840 Well, well, it's like quite also silly thought, but it it, it's what we have 446 00:29:12,025 --> 00:29:14,395 Michael: Can we do create statistics on materialized views? 447 00:29:14,395 --> 00:29:15,775 I didn't even think about that. 448 00:29:16,165 --> 00:29:16,555 Great. 449 00:29:16,705 --> 00:29:17,545 So that makes sense. 450 00:29:17,550 --> 00:29:17,635 Then. 451 00:29:18,040 --> 00:29:22,180 Nikolay: And even on, even on foreign data wrapper tables, so foreign tables. 452 00:29:22,690 --> 00:29:23,770 Michael: Yeah, makes sense. 453 00:29:24,057 --> 00:29:33,537 the, well, the thing I wanted to mention though is that sometimes people as a last port of call or because they've used to using them in previous databases, also would like hints for this. 454 00:29:33,637 --> 00:29:35,107 , so I know, I know people often talk about 455 00:29:35,377 --> 00:29:36,907 Nikolay: Don't mix with hints. 456 00:29:37,057 --> 00:29:40,567 Hints, you mean to, to, to command the, the executor. 457 00:29:40,567 --> 00:29:40,957 What to do. 458 00:29:40,957 --> 00:29:41,407 The planner. 459 00:29:41,407 --> 00:29:41,677 What to 460 00:29:41,677 --> 00:29:43,477 Michael: not necessarily to command them 461 00:29:43,507 --> 00:29:46,387 Nikolay: or HIPO says it's time to, to do that. 462 00:29:46,537 --> 00:29:46,927 To create 463 00:29:47,062 --> 00:29:48,262 Michael: No, I, I meant 464 00:29:49,297 --> 00:29:49,747 Nikolay: regular 465 00:29:49,747 --> 00:29:50,137 database. 466 00:29:50,587 --> 00:29:50,827 Okay. 467 00:29:51,067 --> 00:29:51,397 Like an 468 00:29:51,412 --> 00:30:02,994 Michael: yeah, I did mean regular ones because I think, there's PG hemp plan that I think one of the things they let you do is give an idea of the number of RO you're expecting, uh, from certain things. 469 00:30:02,994 --> 00:30:06,140 So I think there's an interesting, area there. 470 00:30:06,140 --> 00:30:16,677 And I did see a talk by Robert HARs that, when somebody asked about hints, he, he was, seemed very open to the, to specifically for row estimates, not for other things. 471 00:30:16,887 --> 00:30:18,957 So that was very, that was super interesting to me. 472 00:30:19,654 --> 00:30:20,014 Nikolay: Yeah. 473 00:30:20,014 --> 00:30:27,703 And the common approach, uh, to hints, from the core hackers is, we don't want them right 474 00:30:27,823 --> 00:30:32,470 in ,but still people, needed sometimes. 475 00:30:32,860 --> 00:30:36,021 So I, I don't have a hundred percent, like strong opinion here. 476 00:30:36,021 --> 00:30:38,181 I, but let me tell you some small story. 477 00:30:38,721 --> 00:30:42,411 Uh, we added hints to database lab, so database, database, non-production. 478 00:30:42,801 --> 00:30:51,701 And we, the idea was, to allow people to experiment more and see what would, like what if, uh, approach, what if. 479 00:30:51,727 --> 00:30:53,257 Plan would be different. 480 00:30:53,437 --> 00:30:58,774 And so, but sometime later, sometime past people started to use it. 481 00:30:58,924 --> 00:30:59,494 There is comment. 482 00:30:59,524 --> 00:30:59,734 Okay. 483 00:31:00,814 --> 00:31:03,124 People started to ask, uh, okay, it's good. 484 00:31:03,184 --> 00:31:07,834 I found good, better plan, optimized Now why we don't have it on production. 485 00:31:08,521 --> 00:31:08,671 Right. 486 00:31:09,231 --> 00:31:16,411 And I realize that you, if you add something to lab or you also need to think if, like some, there will be some people. 487 00:31:17,446 --> 00:31:19,546 We, we'll want the same on production. 488 00:31:19,546 --> 00:31:23,292 So in, they're already like, okay, should we edit to production? 489 00:31:23,292 --> 00:31:25,392 Maybe No , right? 490 00:31:25,452 --> 00:31:25,782 Maybe. 491 00:31:25,782 --> 00:31:26,142 Yes. 492 00:31:26,419 --> 00:31:27,799 there are different opinions here. 493 00:31:28,594 --> 00:31:30,364 Michael: Maybe we should do a whole episode on that. 494 00:31:30,364 --> 00:31:32,194 Actually, that's, that feels like a good one. 495 00:31:32,719 --> 00:31:32,809 Nikolay: Mm-hmm. 496 00:31:34,264 --> 00:31:34,864 Michael: Wonderful. 497 00:31:34,984 --> 00:31:36,774 Was there anything else you wanted to make sure we covered? 498 00:31:37,302 --> 00:31:37,652 Nikolay: no. 499 00:31:38,038 --> 00:31:38,668 Analyze more. 500 00:31:39,399 --> 00:31:39,699 Michael: Yeah. 501 00:31:39,939 --> 00:31:40,629 Absolutely. 502 00:31:40,844 --> 00:31:42,404 well, thanks everybody for listening. 503 00:31:42,404 --> 00:31:43,244 Thank you, Nikolai. 504 00:31:43,454 --> 00:31:47,444 And see you next week or have a, have a good Christmas for everybody who's celebrating. 505 00:31:48,644 --> 00:31:49,334 Nikolay: Thank you. 506 00:31:49,334 --> 00:31:49,994 Likewise.