1 00:00:00,060 --> 00:00:02,060 Nikolay: Hello, hello, this is Postgres.FM. 2 00:00:02,860 --> 00:00:05,140 I'm Nikolay, Postgres.AI. As usual, 3 00:00:05,140 --> 00:00:07,040 my co-host is Michael, pgMustard. 4 00:00:07,360 --> 00:00:08,100 Hi, Michael. 5 00:00:08,400 --> 00:00:09,560 How was your week? 6 00:00:10,680 --> 00:00:11,400 Michael: Hi, Nikolay. 7 00:00:11,560 --> 00:00:12,540 I'm good, thank you. 8 00:00:12,540 --> 00:00:13,400 How was yours? 9 00:00:14,280 --> 00:00:14,780 Nikolay: Perfect. 10 00:00:15,060 --> 00:00:17,340 Very active and a lot of stuff is happening. 11 00:00:18,220 --> 00:00:21,820 So we needed to miss last week because I had even more stuff 12 00:00:21,820 --> 00:00:25,400 last week, but I'm happy we continue, right? 13 00:00:25,400 --> 00:00:26,400 We don't stop. 14 00:00:27,100 --> 00:00:27,740 Michael: Oh, yeah. 15 00:00:27,740 --> 00:00:28,240 Nikolay: Yeah. 16 00:00:28,460 --> 00:00:33,500 I remember in the beginning I was always against skipping any 17 00:00:33,500 --> 00:00:38,160 week because for me it would be a sign that we probably stop 18 00:00:38,960 --> 00:00:41,980 Which I don't want so Yeah, right now. 19 00:00:41,980 --> 00:00:46,040 I'm already we already proved that like during couple of years 20 00:00:46,040 --> 00:00:50,200 we a Couple of years almost how many years? 21 00:00:50,900 --> 00:00:51,620 Michael: 3 maybe 22 00:00:51,620 --> 00:00:56,700 Nikolay: almost 3 It's like this this July it will be 3 years 23 00:00:57,040 --> 00:01:01,400 and I already proved to myself, we proved to ourselves that if 24 00:01:01,400 --> 00:01:04,820 we skip 1 or 2 weeks it's not game over. 25 00:01:05,340 --> 00:01:08,480 Michael: Yeah, this is me as the European convincing you it's 26 00:01:08,480 --> 00:01:10,020 okay to have a week off every now 27 00:01:10,020 --> 00:01:10,440 Nikolay: and again. 28 00:01:10,440 --> 00:01:11,640 Yeah, exactly, exactly. 29 00:01:12,180 --> 00:01:13,820 Okay, if we stop, that's it. 30 00:01:13,860 --> 00:01:15,040 I don't want it. 31 00:01:15,780 --> 00:01:16,280 Yeah. 32 00:01:16,320 --> 00:01:16,820 Good. 33 00:01:17,640 --> 00:01:24,160 And today, this was my choice and the topic is like, it's less 34 00:01:24,160 --> 00:01:28,020 technical, but although we will talk about technical stuff as 35 00:01:28,020 --> 00:01:35,420 well, and topic is how Managed Postgres services, how they help 36 00:01:35,420 --> 00:01:37,360 us or don't help us. 37 00:01:37,360 --> 00:01:40,920 Customers, I mean, I'm in a different situation probably, but 38 00:01:40,920 --> 00:01:44,540 of course, sometimes I'm just a customer or I'm on customer side. 39 00:01:45,060 --> 00:01:51,320 And there's a problem when the fact that we cannot have access 40 00:01:51,480 --> 00:01:58,860 to the cluster, and we have some issue, there's a whole big class 41 00:01:58,860 --> 00:02:00,900 of problems how to deal with it. 42 00:02:01,840 --> 00:02:06,100 And maybe we should create some best practices how to deal with 43 00:02:06,100 --> 00:02:12,120 support engineers from RDS, Cloud SQL, I don't know, and all 44 00:02:12,120 --> 00:02:13,760 others, right? 45 00:02:14,380 --> 00:02:15,900 And let me start from this. 46 00:02:16,260 --> 00:02:21,420 I learned important lesson, I think in 2015, 16, when I first 47 00:02:21,420 --> 00:02:22,320 tried RDS. 48 00:02:22,860 --> 00:02:29,120 I liked it a lot because of the ability to experiment a lot. 49 00:02:30,380 --> 00:02:35,460 Before cloud, it was really difficult to experiment because for 50 00:02:35,460 --> 00:02:41,140 experiments you need machines of the same size usually for full-fledged 51 00:02:41,240 --> 00:02:44,820 experiments for a very limited amount of time. 52 00:02:45,940 --> 00:02:49,900 15 years ago or so, we were buying servers and putting them to 53 00:02:49,900 --> 00:02:53,400 data centers and experiments were super limited. 54 00:02:53,460 --> 00:02:55,620 Cloud brought us this capability, great. 55 00:02:56,040 --> 00:03:00,720 And with RDS I quickly learned how cool it is to just create 56 00:03:00,720 --> 00:03:05,320 a clone, check everything, how it works, throw it out, and then 57 00:03:05,320 --> 00:03:06,880 rinse and repeat many times. 58 00:03:07,360 --> 00:03:10,680 And then when you deploy, you already studied all the behavior. 59 00:03:11,480 --> 00:03:19,140 And I remember I was Creating a clone, but then it was so slow. 60 00:03:20,220 --> 00:03:24,560 RDS clone, I think it was 2016, maybe 15. 61 00:03:25,120 --> 00:03:26,140 Why is it slow? 62 00:03:27,340 --> 00:03:29,980 Okay, a cluster is like maybe 100 gigabytes. 63 00:03:31,260 --> 00:03:34,100 Today it's a tiny cluster, not tiny, small. 64 00:03:34,540 --> 00:03:37,700 But back in those days it was quite already a big 1. 65 00:03:38,860 --> 00:03:44,940 And I restored and somehow it takes forever to run some SELECT. 66 00:03:45,940 --> 00:03:50,840 And experienced AWS users know very well this phenomenon. 67 00:03:50,860 --> 00:03:57,440 It's called lazy load, because the data is still on S3, and you 68 00:03:57,440 --> 00:04:01,780 have EBS volume which only pretends to have data, but data is 69 00:04:01,780 --> 00:04:04,060 still there, lazy loading in the background. 70 00:04:04,820 --> 00:04:07,680 And I reached support because we had good support. 71 00:04:08,420 --> 00:04:12,260 And the engineer said, oh, let's diagnose, it's some kind of 72 00:04:12,260 --> 00:04:12,760 issue. 73 00:04:13,240 --> 00:04:17,220 So it was hard to understand what's happening and so on. 74 00:04:17,440 --> 00:04:22,320 And I spent maybe an hour or so with that engineer, support engineer, 75 00:04:23,760 --> 00:04:25,360 who was not really helpful. 76 00:04:25,580 --> 00:04:26,080 Right. 77 00:04:26,980 --> 00:04:31,580 And, and then someone, I don't know, like maybe my experience 78 00:04:31,680 --> 00:04:36,320 of managing people by that time, I was already, Hey, I had already 79 00:04:36,320 --> 00:04:39,960 3 companies created in the past. 80 00:04:39,960 --> 00:04:44,180 So I like learn something about cycle psychology and so on. 81 00:04:44,180 --> 00:04:48,140 What I did, I just closed the ticket and opened another 1. 82 00:04:48,280 --> 00:04:53,440 Although usually any support would hate it, like don't duplicate, 83 00:04:53,440 --> 00:04:53,940 right? 84 00:04:54,320 --> 00:04:58,020 But this helped me solve the problem in a few minutes because 85 00:04:58,020 --> 00:05:00,660 another engineer told me, oh, that's just lazy load. 86 00:05:01,680 --> 00:05:07,080 And I googled it, I quickly educated myself, okay, what to do? 87 00:05:07,080 --> 00:05:10,740 Oh, just SELECT * from your Table to warm it up. 88 00:05:11,480 --> 00:05:11,980 Okay. 89 00:05:12,740 --> 00:05:15,920 And since then I have a rule and I share it with my customers 90 00:05:15,920 --> 00:05:16,560 all the time. 91 00:05:16,560 --> 00:05:22,400 If you are a managed Postgres service and you need to deal with 92 00:05:22,400 --> 00:05:25,200 support sometimes, it's like roulette, right? 93 00:05:25,200 --> 00:05:26,700 Like it's 50 50. 94 00:05:26,820 --> 00:05:28,380 It can be helpful, can be not. 95 00:05:28,380 --> 00:05:31,240 If it's not helpful, don't spend more than 10 minutes and just 96 00:05:31,240 --> 00:05:34,900 close the ticket, say thank you and open another one because if 97 00:05:34,900 --> 00:05:40,760 it's a big company who has big support, probably you will find 98 00:05:41,720 --> 00:05:43,520 another engineer who will be more helpful. 99 00:05:43,520 --> 00:05:47,040 Actually, I use this rule with in other areas of my life as well, 100 00:05:47,040 --> 00:05:51,400 for example, talking to some support people in like bank, right? 101 00:05:51,940 --> 00:05:55,800 Credit cards, debit cards, anything, it's not helpful? 102 00:05:55,800 --> 00:05:59,440 Okay, thank you, and you can just call again and another person 103 00:05:59,440 --> 00:06:01,180 will probably help you much faster. 104 00:06:02,080 --> 00:06:03,620 What do you think about this problem? 105 00:06:04,160 --> 00:06:07,640 Michael: Yeah, I think you must have different banking services 106 00:06:07,640 --> 00:06:10,580 to us because if we need to call the bank, you're guaranteed 107 00:06:10,600 --> 00:06:12,540 to be waiting 20 minutes on hold. 108 00:06:12,740 --> 00:06:14,360 Nikolay: So- Oh, yes, it's terrible. 109 00:06:14,480 --> 00:06:15,640 It can be hours. 110 00:06:15,900 --> 00:06:19,320 I think we'll have a day when someone will create an AI assistant 111 00:06:20,380 --> 00:06:24,520 serving on human side, not on company side. 112 00:06:25,940 --> 00:06:26,780 Michael: Oh, interesting. 113 00:06:26,960 --> 00:06:27,460 Yeah. 114 00:06:28,140 --> 00:06:28,365 Nikolay: Yeah. 115 00:06:28,365 --> 00:06:33,320 So they should wait on that line and ask me to join only if everything 116 00:06:33,320 --> 00:06:39,220 is ready and small details already negotiated, some approval 117 00:06:39,220 --> 00:06:40,440 is needed and that's it. 118 00:06:40,440 --> 00:06:41,260 Michael: Yeah, so 119 00:06:41,520 --> 00:06:43,520 Nikolay: maybe one day we will have such systems. 120 00:06:44,160 --> 00:06:46,980 Michael: Yeah, I think at big companies that makes a lot of sense, 121 00:06:47,320 --> 00:06:49,540 at smaller ones much less so. 122 00:06:49,540 --> 00:06:52,580 I think there are some smaller managed services out there. 123 00:06:52,720 --> 00:06:54,520 But yeah, maybe this problem happens less. 124 00:06:54,520 --> 00:06:58,200 I was gonna ask, because sometimes they have the ability to escalate, 125 00:06:58,200 --> 00:06:59,660 right? Do you have any tips? 126 00:07:00,060 --> 00:07:03,400 So let's say you've got a support engineer that wasn't able to 127 00:07:03,400 --> 00:07:04,400 work out the issue. 128 00:07:04,400 --> 00:07:04,600 Nikolay: Do you 129 00:07:04,600 --> 00:07:06,940 Michael: have any tips for getting them to escalate the problem 130 00:07:06,940 --> 00:07:07,780 to a second tier? 131 00:07:07,780 --> 00:07:11,720 Or do you always go to, like, let's open another ticket and hope 132 00:07:11,720 --> 00:07:12,420 that they 133 00:07:12,620 --> 00:07:13,320 escalate it? 134 00:07:13,320 --> 00:07:14,840 Nikolay: That's a great, great question. 135 00:07:15,040 --> 00:07:16,240 And I think we... 136 00:07:17,120 --> 00:07:20,640 ...Position where, I don't know about RDS, by the way, but what 137 00:07:20,640 --> 00:07:25,400 I see in many cases, there is no such ladder built yet. 138 00:07:25,740 --> 00:07:31,480 So in case of big corporations, banks and so on, there is such 139 00:07:31,480 --> 00:07:31,960 option. 140 00:07:31,960 --> 00:07:34,840 You can ask to like senior manager, blah, blah, blah. 141 00:07:34,840 --> 00:07:37,860 Especially if you go offline, it's definitely an option always. 142 00:07:37,860 --> 00:07:38,100 Right. 143 00:07:38,100 --> 00:07:42,540 So please let me speak to like another person and you escalate 144 00:07:42,640 --> 00:07:43,200 and so on. 145 00:07:43,200 --> 00:07:47,840 Like, but what I observe and recently what happened, I, we had 146 00:07:47,840 --> 00:07:52,400 a client who experienced some weird incidents. 147 00:07:54,000 --> 00:07:59,480 And those incidents require you to have low-level access, which 148 00:07:59,480 --> 00:08:00,700 you don't have on RDS. 149 00:08:01,080 --> 00:08:04,640 You need to see where Postgres spends time, for example, like 150 00:08:04,640 --> 00:08:06,660 using Perf, for example, or something. 151 00:08:06,980 --> 00:08:10,320 But you cannot connect, so it's all in their hands. 152 00:08:11,040 --> 00:08:15,020 And you need also to grant them approval to allow them to connect 153 00:08:15,020 --> 00:08:16,500 to your box and so on. 154 00:08:16,720 --> 00:08:18,540 So a lot of bureaucracy here. 155 00:08:18,700 --> 00:08:20,820 And I told them like you need to escalate. 156 00:08:22,060 --> 00:08:27,500 And of course, like it's normal, but I don't see this option 157 00:08:27,500 --> 00:08:28,000 working. 158 00:08:28,680 --> 00:08:32,320 If like if you say escalate, it looks like they don't understand 159 00:08:32,320 --> 00:08:34,780 how, like what's happening here, right? 160 00:08:34,920 --> 00:08:35,420 Michael: Really? 161 00:08:35,900 --> 00:08:39,960 Nikolay: Well, you can try, like you can try and have some problem 162 00:08:39,960 --> 00:08:43,620 and some difficult problem, bring some difficult problem and 163 00:08:43,620 --> 00:08:44,440 try to escalate. 164 00:08:44,480 --> 00:08:45,340 Will it work? 165 00:08:45,360 --> 00:08:47,020 Is there any official option? 166 00:08:47,020 --> 00:08:50,060 Because if it's not official and it works sometimes, it's okay. 167 00:08:50,060 --> 00:08:51,700 Again, it's like gambling. 168 00:08:51,820 --> 00:08:56,280 Like I said, like it's similar to closing and reopening the issue 169 00:08:56,280 --> 00:08:59,420 and hoping next engineer will be more helpful. 170 00:09:00,200 --> 00:09:01,880 Escalation is also not guaranteed. 171 00:09:01,880 --> 00:09:04,400 It's like In many cases, it's good, right? 172 00:09:04,400 --> 00:09:07,940 Because they are probably, they will try to solve. 173 00:09:08,860 --> 00:09:13,700 Well, I also have several, actually, I have several recent cases, 174 00:09:13,700 --> 00:09:14,340 very interesting. 175 00:09:14,340 --> 00:09:17,220 I cannot share all of them, but let me share another 1. 176 00:09:17,320 --> 00:09:23,600 Another company, they are on different, not RDS, not CloudSQL, 177 00:09:24,280 --> 00:09:32,800 and they had issues, a bunch of them, like 10 issues, different 178 00:09:32,800 --> 00:09:33,300 kinds. 179 00:09:33,460 --> 00:09:38,420 1 issue eventually was identified with mutual effort as don't 180 00:09:38,420 --> 00:09:41,740 run backup push or how you call it on the primary. 181 00:09:41,820 --> 00:09:44,160 If system is loaded, do it on replicas. 182 00:09:44,760 --> 00:09:48,420 We talk about it from time to time when we touch backups, right? 183 00:09:48,480 --> 00:09:50,520 And this was an issue on that platform. 184 00:09:51,040 --> 00:09:55,220 But what I observed is, like, trying to work with engineers, 185 00:09:55,280 --> 00:09:59,800 support engineers, and also ultimate escalation if you go to 186 00:09:59,800 --> 00:10:06,500 CTO or CEO level and say, oh, look, like, you know, CTOs are 187 00:10:06,500 --> 00:10:07,360 talking, right? 188 00:10:08,040 --> 00:10:09,440 And this is ultimate escalation. 189 00:10:09,620 --> 00:10:12,880 And it's also not helpful sometimes, right? 190 00:10:12,880 --> 00:10:15,980 In that case, it's like there was some chunk of disappointment, 191 00:10:16,800 --> 00:10:21,300 what I observed, like this was feedback I heard. 192 00:10:21,740 --> 00:10:26,320 Right, so escalation is interesting, but my point is like, we 193 00:10:26,320 --> 00:10:31,740 probably need to learn about escalation ladder and practices 194 00:10:32,680 --> 00:10:34,940 from other businesses, obviously, right? 195 00:10:35,700 --> 00:10:42,980 And I still think it's not fair that customer pays bigger price 196 00:10:42,980 --> 00:10:44,280 and doesn't have control. 197 00:10:45,800 --> 00:10:49,380 Michael: Yeah, sure, well actually on this topic, I was gonna 198 00:10:49,380 --> 00:10:54,800 ask, do you think this is less of an issue as for the... 199 00:10:54,800 --> 00:10:57,780 There are managed service providers that give more access. 200 00:10:57,980 --> 00:11:01,640 Like we had an episode on super user, for example, and it's come 201 00:11:01,640 --> 00:11:02,620 up a few times. 202 00:11:02,920 --> 00:11:06,940 Obviously that's still not like you're talking about running 203 00:11:06,940 --> 00:11:11,920 perf for example but I'm guessing a whole category of issues 204 00:11:11,920 --> 00:11:16,280 just don't exist if you've got superuser access So is it less 205 00:11:16,280 --> 00:11:17,540 of an issue on those? 206 00:11:17,620 --> 00:11:18,900 Nikolay: I will tell you a funny story. 207 00:11:18,900 --> 00:11:19,800 It was with CrunchyBridge. 208 00:11:20,200 --> 00:11:23,300 I respect CrunchyBridge for 2 reasons, already for 2. 209 00:11:23,300 --> 00:11:24,800 It was 1, now for 2. 210 00:11:24,800 --> 00:11:26,340 1 is super user. 211 00:11:26,400 --> 00:11:31,160 I don't know any other managed service yet which provides you 212 00:11:31,160 --> 00:11:31,720 super user. 213 00:11:31,720 --> 00:11:32,420 It's amazing. 214 00:11:33,340 --> 00:11:37,500 You can shoot off your feet very quickly if you want. 215 00:11:37,960 --> 00:11:39,160 It's freedom, right? 216 00:11:39,720 --> 00:11:42,980 And another thing is that they provide access to physical backups, 217 00:11:43,580 --> 00:11:44,720 which is also nice. 218 00:11:45,440 --> 00:11:51,900 This is true freedom and honoring the ownership of database and 219 00:11:51,900 --> 00:11:52,560 so on. 220 00:11:52,800 --> 00:11:58,280 Because without it, maybe you own your data, but not database. 221 00:11:58,940 --> 00:12:02,760 You can dump, but you cannot access PGDATA, but physical backup, 222 00:12:02,760 --> 00:12:03,260 nothing. 223 00:12:03,540 --> 00:12:08,320 And also, you own your data conditionally because if bugs happen, 224 00:12:08,560 --> 00:12:09,860 you even cannot dump. 225 00:12:11,060 --> 00:12:11,560 Right? 226 00:12:12,340 --> 00:12:13,580 And that sucks completely. 227 00:12:14,380 --> 00:12:20,840 And I'm talking about everyone except CrunchyBridge, all managed 228 00:12:20,860 --> 00:12:25,440 services, they all like steal ownership from you. 229 00:12:25,680 --> 00:12:26,400 That sucks. 230 00:12:26,980 --> 00:12:28,080 So, the final story... 231 00:12:28,080 --> 00:12:28,340 I think 232 00:12:28,340 --> 00:12:30,520 Michael: there is at least one other, but like, I think they're 233 00:12:30,520 --> 00:12:30,860 quite small. 234 00:12:30,860 --> 00:12:33,180 I think maybe Tembo give super user access. 235 00:12:33,180 --> 00:12:33,740 I haven't 236 00:12:33,740 --> 00:12:34,520 Nikolay: actually checked. 237 00:12:34,540 --> 00:12:35,580 Oh, maybe, yeah, maybe. 238 00:12:35,840 --> 00:12:38,460 Yeah, apologies if I missed something. 239 00:12:38,680 --> 00:12:39,140 Yeah, let 240 00:12:39,140 --> 00:12:39,780 Michael: us know. 241 00:12:39,940 --> 00:12:43,240 Nikolay: Of course, I work with a lot of customers and expanding 242 00:12:43,340 --> 00:12:48,220 my vision all the time, but of course, it's not 100% coverage. 243 00:12:48,760 --> 00:12:49,640 Michael: Yeah, of course. 244 00:12:49,640 --> 00:12:50,640 Nikolay: Definitely not. 245 00:12:51,000 --> 00:12:52,700 Michael: And definitely the big ones don't. 246 00:12:53,040 --> 00:12:53,800 Nikolay: Right, exactly. 247 00:12:53,800 --> 00:12:58,220 And they say this is for your own good, but it's not. 248 00:12:58,580 --> 00:13:01,030 So let me talk a little bit about CrunchyBridge. 249 00:13:01,030 --> 00:13:02,240 It was super funny. 250 00:13:02,240 --> 00:13:07,540 We needed to help 1 customer and reboot a standby node. 251 00:13:08,560 --> 00:13:11,760 And it turned out CrunchyBridge doesn't support rebooting, restarting 252 00:13:11,880 --> 00:13:13,460 Postgres on standby nodes. 253 00:13:14,720 --> 00:13:18,740 They support it on primary or whole cluster, but not specific 254 00:13:18,740 --> 00:13:19,580 standby node. 255 00:13:19,700 --> 00:13:20,960 It was very weird. 256 00:13:22,060 --> 00:13:25,220 I think it's because they just didn't do it somehow. 257 00:13:25,460 --> 00:13:27,320 Like, it should be done, it should be provided. 258 00:13:28,040 --> 00:13:30,400 But we could not afford restarting whole cluster. 259 00:13:30,400 --> 00:13:31,740 We needed just 1 replica. 260 00:13:32,380 --> 00:13:37,160 And then I said, OK, we have super user. 261 00:13:39,480 --> 00:13:39,920 Michael: Yeah. 262 00:13:39,920 --> 00:13:42,180 Nikolay: What we can do, copy from program, right? 263 00:13:44,140 --> 00:13:45,640 Michael: So you crashed the server. 264 00:13:46,020 --> 00:13:47,040 Nikolay: Not crashed, why crash? 265 00:13:47,040 --> 00:13:51,000 pg_ctl restart, like, it's all good. 266 00:13:51,000 --> 00:13:53,640 Just a -m fast. 267 00:13:54,020 --> 00:13:55,220 All good, all good. 268 00:13:55,760 --> 00:13:57,780 Yeah, there are some nuances there. 269 00:13:58,080 --> 00:14:00,380 Michael: But on that, let's go back to the topic briefly, because 270 00:14:00,380 --> 00:14:01,080 it's relevant. 271 00:14:01,120 --> 00:14:01,820 Nikolay: Let me finish. 272 00:14:01,820 --> 00:14:06,140 Copy from program doesn't work on replicas because it's a writing 273 00:14:06,140 --> 00:14:06,580 operation. 274 00:14:06,580 --> 00:14:07,080 Oh! 275 00:14:08,680 --> 00:14:10,240 Michael: So you had to contact support, right? 276 00:14:10,240 --> 00:14:11,860 That's where I was going with this. 277 00:14:12,280 --> 00:14:14,620 Nikolay: Well, support says this feature is not working. 278 00:14:15,040 --> 00:14:15,900 I mean, it's not supported. 279 00:14:15,900 --> 00:14:17,720 Michael: But they could do it for you, no? 280 00:14:17,720 --> 00:14:18,170 Nikolay: No, no, no. 281 00:14:18,170 --> 00:14:20,940 I needed the part of automation we were building. 282 00:14:21,880 --> 00:14:23,520 It was part of bigger picture. 283 00:14:23,520 --> 00:14:25,140 And we needed this ability. 284 00:14:26,780 --> 00:14:30,420 So what we ended up doing is copy to program, writing to 285 00:14:30,420 --> 00:14:31,080 Michael: a log. 286 00:14:31,240 --> 00:14:33,760 Nikolay: And this worked on Replica, but we were blind a little 287 00:14:33,760 --> 00:14:34,260 bit. 288 00:14:34,280 --> 00:14:41,660 But then I talked to the developers and realized we had an easier 289 00:14:41,660 --> 00:14:42,720 path in our hands. 290 00:14:42,720 --> 00:14:44,560 It's python -u. 291 00:14:46,100 --> 00:14:50,160 Anyway, if you have super user, you can hack yourself a little 292 00:14:50,160 --> 00:14:50,660 bit. 293 00:14:51,220 --> 00:14:52,760 It's your own right. 294 00:14:52,760 --> 00:14:55,760 If you broke something, don't do it. 295 00:14:56,240 --> 00:14:57,640 Michael: Yeah, it's a really good point. 296 00:14:57,640 --> 00:14:59,100 So that was kind of my questions. 297 00:14:59,100 --> 00:15:02,980 If you've got more access, I presume there are fewer issues that 298 00:15:02,980 --> 00:15:04,560 you need support for. 299 00:15:04,960 --> 00:15:08,080 But that does raise a good question because there's kind of 3 300 00:15:08,080 --> 00:15:10,340 times you need to contact support, right? 301 00:15:10,640 --> 00:15:13,780 We've got an issue right now, maybe urgent, maybe not. 302 00:15:15,060 --> 00:15:17,340 I've got a question, how does something work? 303 00:15:17,500 --> 00:15:20,200 And then the third category is feature requests. 304 00:15:20,280 --> 00:15:23,320 Like I'd like to be able to do this, but which we can't currently 305 00:15:23,320 --> 00:15:23,820 do. 306 00:15:25,840 --> 00:15:28,860 My experience of feature requests or like looking at different 307 00:15:28,860 --> 00:15:32,600 forums of different managed service providers of where they ask 308 00:15:32,600 --> 00:15:34,940 people to go to request and vote on features. 309 00:15:35,460 --> 00:15:37,200 It looks a little hit and miss. 310 00:15:38,040 --> 00:15:43,100 How like what's your do you have any advice in terms of how to 311 00:15:43,100 --> 00:15:43,780 do that? 312 00:15:44,060 --> 00:15:49,360 Nikolay: We have 2 paths here advice to whom to users or to platform 313 00:15:49,360 --> 00:15:49,860 Michael: users. 314 00:15:49,980 --> 00:15:52,740 I'm thinking for people listening mostly users. 315 00:15:54,040 --> 00:15:56,400 Nikolay: Well it's a bad state right now. 316 00:15:56,400 --> 00:16:01,460 Again, I think managed services should stop hiding access. 317 00:16:01,460 --> 00:16:05,220 They should like, they build everything on top of open source. 318 00:16:05,800 --> 00:16:09,440 And they charge for operations and for like support, good, good, 319 00:16:09,440 --> 00:16:15,580 good. But hiding access to purely open source pieces, it's like, it 320 00:16:15,580 --> 00:16:16,980 sounds bullshit to me. 321 00:16:16,980 --> 00:16:18,000 Complete bullshit. 322 00:16:18,340 --> 00:16:20,520 Actually, it makes me angry even. 323 00:16:23,500 --> 00:16:26,780 So amazing, like yesterday I saw an article from Yugabyte. 324 00:16:26,820 --> 00:16:32,060 Yugabyte suddenly, I feel it like Tembo actually released DBA 325 00:16:32,080 --> 00:16:34,340 AI going outside of their platform. 326 00:16:34,340 --> 00:16:36,260 And Yugabyte did a similar thing. 327 00:16:36,260 --> 00:16:40,300 They went outside of their Metaverse product and platform, and 328 00:16:40,440 --> 00:16:45,660 they started offering a tool for 0 downtime upgrades, compatible 329 00:16:45,760 --> 00:16:50,080 with Postgres running on many managed service providers like 330 00:16:50,080 --> 00:16:53,900 RDS, CloudSQL, Supabase, CrunchyBridge and so on. 331 00:16:53,920 --> 00:16:54,880 And that's great. 332 00:16:55,200 --> 00:16:56,020 That's great. 333 00:16:56,200 --> 00:17:00,080 They did wrong a little bit because they called things like blue-green 334 00:17:00,420 --> 00:17:02,040 deployments while it's not... 335 00:17:02,040 --> 00:17:05,340 They did similar mistake as RDS did, we discussed it, right? 336 00:17:05,380 --> 00:17:06,400 They, this... 337 00:17:06,567 --> 00:17:09,660 Michael: I, yeah, but I saw your tweet about this and I'm going 338 00:17:09,660 --> 00:17:11,920 to defend them because I don't think it's their fault. 339 00:17:11,920 --> 00:17:14,720 I think the problem is RDS broke people's understanding. 340 00:17:14,720 --> 00:17:16,160 Look, I'm, wait 341 00:17:16,160 --> 00:17:17,075 Nikolay: a little bit. 342 00:17:17,075 --> 00:17:18,400 I'm going there, exactly. 343 00:17:18,820 --> 00:17:20,280 I'm going exactly there. 344 00:17:21,040 --> 00:17:24,440 So blue-green deployments, according to Martin Fowler, 15 years 345 00:17:24,440 --> 00:17:28,720 ago, he published an article, they by nature must be symmetric. 346 00:17:29,680 --> 00:17:31,120 Michael: We did an episode, remember? 347 00:17:31,260 --> 00:17:33,740 Nikolay: Yes, exactly, Criticizing RDS implementation. 348 00:17:34,280 --> 00:17:36,220 And Postgres definitely supports it. 349 00:17:36,220 --> 00:17:39,240 We implemented this, like some customers use it. 350 00:17:39,240 --> 00:17:39,940 That's great. 351 00:17:40,840 --> 00:17:44,460 And what my point is like, probably you go by it, hit the same 352 00:17:45,300 --> 00:17:46,260 limitations we hit. 353 00:17:46,980 --> 00:17:50,340 On RDS you cannot change things, it's not available. 354 00:17:50,740 --> 00:17:53,660 And since you don't have low-level access, you cannot change 355 00:17:53,720 --> 00:17:54,640 many of things. 356 00:17:55,080 --> 00:17:56,940 And this limits you so drastically. 357 00:17:57,840 --> 00:18:02,620 And it feels like some weird pendulum key. 358 00:18:02,620 --> 00:18:05,960 If you want RDS, okay, good, I understand. 359 00:18:06,820 --> 00:18:12,740 But you cannot engineer the best approach for upgrades. 360 00:18:13,500 --> 00:18:15,640 And you need to wait how many years? 361 00:18:15,800 --> 00:18:17,640 Okay, blue-green deployments, they released. 362 00:18:18,060 --> 00:18:20,280 I see better path for blue-green deployments. 363 00:18:20,980 --> 00:18:24,880 And it's my database and I cannot do it, I need to go out of 364 00:18:24,880 --> 00:18:25,380 RDS. 365 00:18:25,920 --> 00:18:30,660 At the same time, if they provided access, more access, opening 366 00:18:30,660 --> 00:18:36,360 gates for additional pieces of changes, it would be possible 367 00:18:36,360 --> 00:18:41,040 to engineer blue-green deployments for me or for third parties. 368 00:18:41,040 --> 00:18:45,380 Like, okay, you go buy this third party, they want to offer or 369 00:18:45,380 --> 00:18:49,820 sell some product or tool compatible with RDS, but since they 370 00:18:49,820 --> 00:18:54,960 don't have access to recovery target LSN and so on, they are 371 00:18:54,960 --> 00:18:56,460 very limited, right? 372 00:18:59,140 --> 00:19:02,380 Michael: Yeah, but It might be exactly for that reason. 373 00:19:02,780 --> 00:19:06,180 If we're talking about the reason for needing it, one of the reasons 374 00:19:06,180 --> 00:19:11,080 is migrating off, migrating out, then you can see the incentives. 375 00:19:13,360 --> 00:19:16,400 Nikolay: And for upgrades, things are becoming much better in 376 00:19:16,400 --> 00:19:17,380 PostgreSQL 17. 377 00:19:17,380 --> 00:19:21,660 And blue-green deployments, it's kind of not only for upgrades. 378 00:19:21,660 --> 00:19:24,960 If we eliminate the upgrade idea, we can implement blue-green 379 00:19:24,960 --> 00:19:27,000 deployments on any platform right now. 380 00:19:27,440 --> 00:19:34,100 Because you can skip many LSNs in the slot and just... 381 00:19:34,740 --> 00:19:35,720 How is it called? 382 00:19:37,000 --> 00:19:39,600 Not promote, because promote is different. 383 00:19:40,080 --> 00:19:40,740 I forgot. 384 00:19:41,600 --> 00:19:47,520 Like shift position of logical slot and synchronize its LSN position 385 00:19:47,520 --> 00:19:50,820 with the position we need, and then from there we can already 386 00:19:51,140 --> 00:19:53,200 perform this dance with blue-green deployments. 387 00:19:53,420 --> 00:19:54,120 It's doable. 388 00:19:54,520 --> 00:19:58,020 But if you want upgrades, OK, we need to wait until 17, because 389 00:19:58,020 --> 00:19:59,980 there is low risk of corruption. 390 00:20:00,860 --> 00:20:01,760 You mean 18? 391 00:20:02,320 --> 00:20:02,820 17. 392 00:20:03,420 --> 00:20:07,220 17 has pg_createsubscriber CLI tool. 393 00:20:07,540 --> 00:20:12,700 And it also officially supports major upgrades on replicas, logical 394 00:20:12,700 --> 00:20:13,200 replicas. 395 00:20:13,980 --> 00:20:19,640 So yeah, These 2 powerful things give us great path to upgrading 396 00:20:20,220 --> 00:20:23,040 really huge clusters using 0 downtime approach. 397 00:20:23,940 --> 00:20:27,180 Well, near 0 downtime, unless you have PgBouncer. 398 00:20:27,340 --> 00:20:30,040 If you have PgBouncer, you have Pulse Resume, then it's purely 399 00:20:30,040 --> 00:20:30,740 0 downtime. 400 00:20:31,640 --> 00:20:36,720 Anyway, my point is, since they partially vendor perform this 401 00:20:36,720 --> 00:20:39,620 vendor lock-in, they hesitate opening gates. 402 00:20:39,620 --> 00:20:42,700 Customers cannot diagnose incidents and they also cannot build 403 00:20:42,700 --> 00:20:46,860 tools and third parties like Yugabyte or for example Postgres 404 00:20:47,020 --> 00:20:52,280 probably would also build some tools compatible with many other 405 00:20:52,280 --> 00:20:52,780 platforms. 406 00:20:53,000 --> 00:20:54,880 Not other, we don't have platform, right? 407 00:20:56,280 --> 00:21:00,760 We help customers regardless of location of their Postgres database. 408 00:21:00,860 --> 00:21:03,460 So if it's RDS, okay, CloudSQL, okay. 409 00:21:03,740 --> 00:21:09,480 But building tools for them, it's very limited right now because 410 00:21:09,480 --> 00:21:12,440 we don't have access to many things and we don't have super user 411 00:21:12,440 --> 00:21:13,240 and so on. 412 00:21:13,320 --> 00:21:16,160 So yeah, that's bad. 413 00:21:16,160 --> 00:21:21,360 But back to support, if, like, my main advice is just gambling 414 00:21:21,500 --> 00:21:22,000 advice. 415 00:21:22,360 --> 00:21:23,460 Just gamble, guys. 416 00:21:23,460 --> 00:21:28,140 Michael: Well, I have some, like, I think a lot of people have 417 00:21:28,140 --> 00:21:32,960 very high trust when they request features, like, very, or very 418 00:21:32,960 --> 00:21:36,820 high belief that people will understand why they're asking for 419 00:21:36,820 --> 00:21:39,960 it and I don't I think a lot of people don't include context 420 00:21:39,960 --> 00:21:42,500 when they're asking for they don't include why they want the 421 00:21:42,500 --> 00:21:45,200 feature or what it's preventing them or what it might cause them 422 00:21:45,200 --> 00:21:49,460 to do if they if they can't get it or what their alternatives 423 00:21:49,600 --> 00:21:50,320 are going to be. 424 00:21:50,320 --> 00:21:55,240 So I think sometimes when you make products people just ask for 425 00:21:55,240 --> 00:21:58,520 features and you have to ask them why do you want this, like 426 00:21:58,520 --> 00:22:01,400 what are you trying to do because without that context it's really 427 00:22:01,400 --> 00:22:04,680 hard to know which of your potential solutions could be worth 428 00:22:04,680 --> 00:22:06,360 it or if it's worth doing at all. 429 00:22:06,360 --> 00:22:09,680 But most vendors I've seen just don't ask that question, like 430 00:22:09,680 --> 00:22:11,180 people ask for a feature 431 00:22:11,260 --> 00:22:11,400 Nikolay: or 432 00:22:11,400 --> 00:22:13,860 Michael: you know or a new extension to be supported or something 433 00:22:13,860 --> 00:22:17,800 and there's no, there's no, even if that extension has multiple 434 00:22:17,800 --> 00:22:20,680 use cases, there's no question back as to why they want that 435 00:22:20,680 --> 00:22:21,160 feature. 436 00:22:21,160 --> 00:22:22,240 Like it's so... 437 00:22:22,240 --> 00:22:23,040 Nikolay: Value, right? 438 00:22:23,040 --> 00:22:23,540 Goals. 439 00:22:23,800 --> 00:22:25,580 Michael: Yeah, well exactly. 440 00:22:25,680 --> 00:22:28,520 And sometimes 5 people could want the same feature but it's all 441 00:22:28,520 --> 00:22:30,360 for different reasons and that's like... 442 00:22:30,360 --> 00:22:33,560 Nikolay: Which shows bigger value if there are many different 443 00:22:33,560 --> 00:22:34,060 reasons. 444 00:22:34,820 --> 00:22:37,820 Michael: Yeah, or maybe an issue. 445 00:22:37,820 --> 00:22:41,000 Like maybe it's actually less of a good idea because they're 446 00:22:41,000 --> 00:22:42,780 actually going to want different things from it. 447 00:22:42,780 --> 00:22:45,920 Like it's going to be harder to implement, unless it's an extension 448 00:22:45,920 --> 00:22:48,080 and you get them all straight away. 449 00:22:48,080 --> 00:22:53,240 But I think in terms of customers asking for things, I've not 450 00:22:53,240 --> 00:22:57,220 seen this work from managed service providers specifically, but 451 00:22:57,260 --> 00:22:59,960 for products in general, I think it is helpful to give the context 452 00:22:59,960 --> 00:23:01,860 as to why you're asking for something. 453 00:23:02,440 --> 00:23:06,620 The only other thing I had to add from my side was if and when 454 00:23:06,620 --> 00:23:09,620 you are considering migrating to a managed service provider, 455 00:23:09,620 --> 00:23:13,520 so either at the beginning or when you've got a project up and 456 00:23:13,520 --> 00:23:13,780 running. 457 00:23:13,780 --> 00:23:17,640 I see quite a few people on Reddit and places at the moment looking 458 00:23:17,640 --> 00:23:20,600 at moving self-hosted things to managed service providers you 459 00:23:20,600 --> 00:23:24,860 know as they're gaining a little bit of traction and My I've 460 00:23:24,860 --> 00:23:28,360 seen a I've seen at least 1 case go badly wrong when the person 461 00:23:28,360 --> 00:23:31,940 didn't contact support At the beginning of the process, you know 462 00:23:31,940 --> 00:23:34,640 They tried to do everything self-service and actually it would 463 00:23:34,640 --> 00:23:36,980 have been helpful for them to contact support earlier. 464 00:23:37,440 --> 00:23:39,080 I think there's 2 good reasons for that. 465 00:23:39,080 --> 00:23:42,260 1 is to make sure the migration goes smoothly, but the second 466 00:23:42,260 --> 00:23:43,940 is test the support out. 467 00:23:44,600 --> 00:23:46,120 How Does it work for you? 468 00:23:46,120 --> 00:23:47,140 Is it responsive? 469 00:23:49,000 --> 00:23:50,880 What kind of answers do you get? 470 00:23:50,980 --> 00:23:52,760 Is it helpful, that kind of thing? 471 00:23:52,760 --> 00:23:55,260 Nikolay: Yeah, we need to write some automation to periodically 472 00:23:55,280 --> 00:23:57,940 test all the supports using LLMs. 473 00:23:58,140 --> 00:23:59,440 I'm joking, of course. 474 00:24:00,020 --> 00:24:03,900 But I just know, it's your database. 475 00:24:04,200 --> 00:24:09,780 Even if you have, like, consider them a cat, like, microservices, 476 00:24:10,020 --> 00:24:11,600 it's not pet, it's cattle. 477 00:24:12,400 --> 00:24:19,400 But it's still like you, you like being maybe DBA, DBRE, SRE, 478 00:24:19,400 --> 00:24:24,060 doesn't matter, back-end engineer, you are very interested to 479 00:24:24,140 --> 00:24:27,100 take proper care of database and so on. 480 00:24:27,640 --> 00:24:29,640 And support, you're just one of many. 481 00:24:29,760 --> 00:24:31,600 Your database is one of many. 482 00:24:31,820 --> 00:24:34,240 And they also have their own KPIs. 483 00:24:35,460 --> 00:24:37,540 Like your question closed, okay, goodbye. 484 00:24:37,760 --> 00:24:41,940 And also like, okay, do this, and so on. 485 00:24:43,480 --> 00:24:48,240 And since we don't have accesses and so on, and kind of, I just 486 00:24:48,240 --> 00:24:51,760 feel the big need, like this is a big imbalance. 487 00:24:52,440 --> 00:24:56,980 If you ask something support, they can help. 488 00:24:56,980 --> 00:25:03,460 I saw many helpful support attempts, very helpful, very careful, 489 00:25:03,720 --> 00:25:06,280 but it's rare, right? 490 00:25:06,500 --> 00:25:11,260 And Postgres experts also rare Not many 491 00:25:11,380 --> 00:25:12,180 Michael: right yeah, 492 00:25:12,580 --> 00:25:16,720 Nikolay: and and this like closing ability to third party for 493 00:25:16,720 --> 00:25:21,040 example if somebody is involving us we immediately say okay this 494 00:25:21,180 --> 00:25:24,440 you need to put pressure on their support we cannot help. 495 00:25:25,320 --> 00:25:27,660 Michael: Okay so what do you mean by putting pressure do you 496 00:25:27,660 --> 00:25:30,540 mean like following up regularly what do you mean by putting 497 00:25:30,540 --> 00:25:31,240 pressure on? 498 00:25:31,240 --> 00:25:35,500 Nikolay: The opening, escalating and so on, like explaining why, 499 00:25:35,500 --> 00:25:40,060 for example, like a big company can have various support engineers. 500 00:25:40,800 --> 00:25:41,300 Yeah. 501 00:25:41,600 --> 00:25:46,780 And for example, if there is a hanging query in its RDS, it's 502 00:25:46,780 --> 00:25:51,720 a recent little story, And they suddenly say, okay, we solved. 503 00:25:52,580 --> 00:25:53,820 Query is not hanging. 504 00:25:55,520 --> 00:25:57,520 And I wonder, how come? 505 00:25:57,520 --> 00:26:00,900 It was hanging because it cannot intercept signal, blah, blah, 506 00:26:00,900 --> 00:26:02,220 blah. It was hanging many hours. 507 00:26:02,900 --> 00:26:04,400 How did you manage it? 508 00:26:04,900 --> 00:26:07,200 OK, support said RDS support. 509 00:26:07,200 --> 00:26:08,720 We managed to, OK. 510 00:26:09,640 --> 00:26:10,880 Did restart happen? 511 00:26:11,320 --> 00:26:12,300 Yes, it did. 512 00:26:12,340 --> 00:26:16,100 And in logs, we see the signs of kill -9. 513 00:26:16,920 --> 00:26:18,940 So this is what support engineer did. 514 00:26:18,940 --> 00:26:21,360 This support engineer should be fired, right? 515 00:26:21,740 --> 00:26:26,120 In RDS team, this is my opinion, but I'm just saying it's 516 00:26:26,120 --> 00:26:29,960 hard to build super strong support team and it will be always 517 00:26:29,960 --> 00:26:34,180 lacking and it would be great if company would allow third party 518 00:26:34,820 --> 00:26:36,540 people help. 519 00:26:36,820 --> 00:26:40,280 If you check other aspects of our life, for example, if you have 520 00:26:40,280 --> 00:26:45,820 a car or you have recently I replaced tankless heater in my house. 521 00:26:46,560 --> 00:26:49,400 If you go to vendor, sometimes vendor doesn't exist. 522 00:26:49,400 --> 00:26:54,100 For example, my solar is very old, but anyway, you can variety 523 00:26:54,240 --> 00:26:57,540 of service people who can help and do maintenance. 524 00:26:58,860 --> 00:27:03,560 If company, Even RDS is limiting maintenance aspects only to 525 00:27:03,560 --> 00:27:04,540 their own staff. 526 00:27:05,380 --> 00:27:09,140 It always will be very limited because Postgres expertise is 527 00:27:09,140 --> 00:27:10,320 limited on the market. 528 00:27:11,120 --> 00:27:13,180 They should find a way to open gates. 529 00:27:13,780 --> 00:27:14,480 This is my... 530 00:27:14,480 --> 00:27:16,420 It's already a message to platform builders. 531 00:27:17,640 --> 00:27:18,140 What? 532 00:27:18,840 --> 00:27:22,120 Michael: Well, I mean, I understand where you're coming from 533 00:27:22,120 --> 00:27:22,760 as a... 534 00:27:22,860 --> 00:27:25,660 Nikolay: I'm coming too, it's not from, it's too, it's future. 535 00:27:26,400 --> 00:27:29,060 Michael: I mean, I understand where you're coming from that they 536 00:27:29,060 --> 00:27:32,120 can't hire all of them and actually there's benefit in terms 537 00:27:32,120 --> 00:27:34,640 of other people being able to provide support. 538 00:27:34,640 --> 00:27:38,940 But if Postgres expertise is so limited, where is everyone else 539 00:27:38,940 --> 00:27:40,460 going to get their support from? 540 00:27:41,040 --> 00:27:41,840 It's not... 541 00:27:42,040 --> 00:27:42,620 It's open 542 00:27:42,620 --> 00:27:43,780 Nikolay: market and competition. 543 00:27:44,640 --> 00:27:45,420 Michael: Yeah, exactly. 544 00:27:46,800 --> 00:27:48,980 So you're saying there is plenty of Postgres expertise? 545 00:27:50,020 --> 00:27:54,160 Nikolay: Well, the company should only benefit if they open the 546 00:27:54,160 --> 00:27:58,940 gates and allow other people, help other people whilst they are 547 00:27:58,940 --> 00:28:00,200 still on the same platform. 548 00:28:00,440 --> 00:28:05,640 Because otherwise, concern and level of disappointment about 549 00:28:05,640 --> 00:28:08,600 support can raise until the point they go off. 550 00:28:08,600 --> 00:28:10,620 Which is actually probably not a bad idea. 551 00:28:11,140 --> 00:28:16,040 I also believe that slowly our segment of market will start to 552 00:28:16,040 --> 00:28:19,600 realize that there should be like this self-managed, there is 553 00:28:19,600 --> 00:28:21,660 managed, but probably there is something in between. 554 00:28:22,040 --> 00:28:25,360 And I know this is some work I cannot share is happening. 555 00:28:25,640 --> 00:28:29,720 So something in between where you truly own but still have benefits 556 00:28:29,760 --> 00:28:30,760 of managed services. 557 00:28:30,860 --> 00:28:33,800 This should happen and I think multiple companies are going in 558 00:28:33,800 --> 00:28:34,460 this direction. 559 00:28:35,600 --> 00:28:40,460 Michael: Or, and I'm seeing this more from kind of smaller companies, 560 00:28:40,600 --> 00:28:45,260 quite established in terms of like their database and team but 561 00:28:45,280 --> 00:28:50,520 not brand new startups necessarily, Moving to services where 562 00:28:50,900 --> 00:28:53,560 factoring in support as one of the main things they're looking 563 00:28:53,560 --> 00:28:54,820 for in a service provider. 564 00:28:54,840 --> 00:28:59,140 I think in the past, like people would factor in, people look 565 00:28:59,140 --> 00:29:00,520 at a lot of things, right? 566 00:29:01,680 --> 00:29:02,060 Nikolay: Price, 567 00:29:02,060 --> 00:29:05,160 Michael: ease of use, region, 568 00:29:05,220 --> 00:29:05,720 Nikolay: like, 569 00:29:06,380 --> 00:29:10,320 Michael: yeah, they look for a bunch of features but don't always 570 00:29:10,320 --> 00:29:15,560 factor in support as one of those key factors and I think I like 571 00:29:15,560 --> 00:29:20,360 to see when people do factor that in and take it seriously. 572 00:29:20,640 --> 00:29:22,480 So that's the alternative, right? 573 00:29:22,720 --> 00:29:25,280 Is pick your managed service provider partly based on how good 574 00:29:25,280 --> 00:29:25,680 their support is. 575 00:29:25,680 --> 00:29:28,900 Nikolay: I'm talking about absolutely new approach when a service 576 00:29:28,900 --> 00:29:33,080 is not self-managed, not managed, but it's very, very well automated 577 00:29:33,240 --> 00:29:36,800 and you can hire if you're not satisfied with some company who 578 00:29:37,120 --> 00:29:41,880 helps you maintain it, you can switch the provider of this maintenance 579 00:29:41,880 --> 00:29:42,680 work, right? 580 00:29:42,980 --> 00:29:43,680 This should be 581 00:29:43,680 --> 00:29:44,780 Michael: like co-managed. 582 00:29:45,800 --> 00:29:46,600 Nikolay: Yeah. Co-managed. 583 00:29:46,840 --> 00:29:47,780 Yes, exactly. 584 00:29:47,860 --> 00:29:51,760 It's, it's, It's great because market is growing and competition 585 00:29:51,900 --> 00:29:56,520 is growing and we see, like I just provided a few examples about 586 00:29:56,520 --> 00:30:00,560 several managed services, we see bad examples all the time and 587 00:30:00,920 --> 00:30:02,120 the problem is systematic. 588 00:30:02,260 --> 00:30:06,360 It's not just like some company is bad and others are good or 589 00:30:06,360 --> 00:30:07,040 vice versa. 590 00:30:07,040 --> 00:30:11,120 It's systematic problem rooted in the decision to close the gates 591 00:30:11,120 --> 00:30:13,760 and not allowing others to look inside. 592 00:30:14,380 --> 00:30:16,900 Michael: I also think providing good support is expensive. 593 00:30:17,620 --> 00:30:19,560 Deep Postgres expertise is expensive. 594 00:30:20,280 --> 00:30:23,500 I'm a bit surprised by your experience with escalation. 595 00:30:25,240 --> 00:30:29,880 Most companies I see do have escalation paths, but I don't deal 596 00:30:30,320 --> 00:30:33,240 with Postgres managed service providers support that often. 597 00:30:33,240 --> 00:30:36,860 So I'm surprised to hear they don't have good escalation paths. 598 00:30:39,520 --> 00:30:43,340 But yeah, if that's the case, I feel like there must be opportunity 599 00:30:43,380 --> 00:30:44,080 for people. 600 00:30:44,100 --> 00:30:46,260 And I know some do really. 601 00:30:46,960 --> 00:30:49,620 Nikolay: I have a question about this area. 602 00:30:49,740 --> 00:30:54,060 About like, if you, you're also running something on cloud, GCP, 603 00:30:54,140 --> 00:30:54,520 right? 604 00:30:54,520 --> 00:30:55,020 Yeah. 605 00:30:55,920 --> 00:30:56,380 Cloud. 606 00:30:56,380 --> 00:30:58,260 Do you have Kubernetes? 607 00:30:58,260 --> 00:30:58,660 Yeah. 608 00:30:58,660 --> 00:31:00,260 You use it, okay. 609 00:31:00,260 --> 00:31:03,580 So Can you, so you use GKE, right? 610 00:31:04,420 --> 00:31:04,920 Yeah. 611 00:31:05,380 --> 00:31:08,680 Google Cloud Engine or Cloud Kubernetes Engine, right? 612 00:31:08,680 --> 00:31:09,180 Yeah. 613 00:31:09,440 --> 00:31:13,940 So if you go to Compute Engine, they call it Compute Engine, 614 00:31:13,940 --> 00:31:14,440 right? 615 00:31:14,440 --> 00:31:15,900 Where you can see VMs. 616 00:31:16,160 --> 00:31:16,660 Yeah. 617 00:31:17,080 --> 00:31:21,380 Do you see VMs where this Kubernetes cluster is running? 618 00:31:22,200 --> 00:31:23,160 I guess yes. 619 00:31:24,080 --> 00:31:26,140 Michael: See the pods and the, yeah, see the pods. 620 00:31:26,140 --> 00:31:27,600 Nikolay: No, not the pods, the VMs. 621 00:31:27,780 --> 00:31:29,660 Can you SSH to those VMs? 622 00:31:30,900 --> 00:31:32,620 Michael: Oh, I have a show, Yeah. 623 00:31:34,140 --> 00:31:38,100 Nikolay: So Google provides Kubernetes engine, automation, everything, 624 00:31:39,340 --> 00:31:41,180 and you still have SSH access. 625 00:31:42,720 --> 00:31:43,220 Yeah. 626 00:31:44,020 --> 00:31:47,040 Why cannot be done the same thing for managed Postgres? 627 00:31:48,140 --> 00:31:48,640 Michael: Oh, okay. 628 00:31:48,640 --> 00:31:49,140 Yeah. 629 00:31:51,280 --> 00:31:52,060 Well, Good question. 630 00:31:52,900 --> 00:31:55,580 Nikolay: If you have SSH access, well, you can break things. 631 00:31:55,580 --> 00:31:57,460 Well, okay, I know. 632 00:31:57,560 --> 00:31:58,300 I know. 633 00:31:58,300 --> 00:32:01,920 If I open my car, I can break things there as well. 634 00:32:02,020 --> 00:32:03,240 This is interesting, right? 635 00:32:05,840 --> 00:32:11,820 I know companies who provide services to tune, maintain Kubernetes 636 00:32:11,860 --> 00:32:12,360 clusters. 637 00:32:13,100 --> 00:32:15,920 And this is a perfect example, because for them, There is great 638 00:32:15,920 --> 00:32:17,220 automation from Google. 639 00:32:18,560 --> 00:32:19,520 Everything is automated. 640 00:32:20,380 --> 00:32:25,640 But if customers have specific needs, and Google cannot meet 641 00:32:25,640 --> 00:32:29,100 those needs because they have limited hands, number of hands 642 00:32:29,100 --> 00:32:34,140 still, right, and attention, and so on, Company can hire another 643 00:32:34,140 --> 00:32:38,000 company who are experts in this particular topic, they can go 644 00:32:38,000 --> 00:32:41,640 and they have everything and they have SSH access to this fully 645 00:32:41,640 --> 00:32:42,580 automated thing. 646 00:32:43,220 --> 00:32:44,180 Interesting, right? 647 00:32:45,340 --> 00:32:45,620 Michael: Yeah. 648 00:32:45,620 --> 00:32:49,300 Well, any last advice for people, like actual users? 649 00:32:49,940 --> 00:32:53,140 Nikolay: Well, yeah, I know I'm biased towards platform builders 650 00:32:53,140 --> 00:32:57,560 because I'm upset and angry and I hope I explain origins of my 651 00:32:57,560 --> 00:33:03,340 anger but Yeah, put as much pressure to support as possible. 652 00:33:04,080 --> 00:33:08,320 Politely, but very firmly and explaining. 653 00:33:09,280 --> 00:33:14,840 Like, I think it's possible to, you had a great point that reasons 654 00:33:15,040 --> 00:33:18,580 and final goals need to be explained, right? 655 00:33:19,080 --> 00:33:22,700 And also risks, like what will happen if we don't achieve this? 656 00:33:23,100 --> 00:33:26,400 Sometimes up to okay, we consider switching to different like 657 00:33:26,400 --> 00:33:28,460 approach, provider or something. 658 00:33:29,100 --> 00:33:35,400 Yeah, I think just like people should be more like detailed and 659 00:33:35,440 --> 00:33:39,340 putting more pressure to support to squeeze details from them. 660 00:33:39,340 --> 00:33:43,080 In this case, I'm very interested because many like managed Postgres 661 00:33:43,080 --> 00:33:47,400 users come to us more and more recently and they ask for help 662 00:33:47,860 --> 00:33:52,440 and if support is doing their job great, it helps us as well 663 00:33:52,440 --> 00:33:57,500 because yeah, it's like beneficial for all because we help to 664 00:33:57,500 --> 00:34:02,540 level up health of Postgres clusters, get rid of bloat, add some 665 00:34:02,540 --> 00:34:04,840 automation, tuning and so on. 666 00:34:05,080 --> 00:34:09,000 But if support does poor job, well, customer starts looking at 667 00:34:09,000 --> 00:34:11,260 different direction where to migrate, right? 668 00:34:12,040 --> 00:34:17,960 And yeah, so my advice to users, pressure, details and so on 669 00:34:18,980 --> 00:34:19,680 To support. 670 00:34:19,780 --> 00:34:22,360 Michael: Is there anything to be gained in the cases where they 671 00:34:22,360 --> 00:34:23,980 give exceptional support? 672 00:34:24,560 --> 00:34:27,380 You mentioned rare cases where the support is very good. 673 00:34:27,380 --> 00:34:31,420 Is there anything that we can do in those cases to, like, not 674 00:34:31,420 --> 00:34:34,380 just say thank you, but say this was really good or feedback 675 00:34:34,440 --> 00:34:34,780 that this 676 00:34:34,780 --> 00:34:39,180 Nikolay: is what I liked a lot is when support engineers formatted 677 00:34:39,180 --> 00:34:43,900 responses very well and I knew it's not a little m actually but 678 00:34:43,900 --> 00:34:47,600 maybe partially but It was human behind that for sure because 679 00:34:47,600 --> 00:34:48,500 I saw it. 680 00:34:49,200 --> 00:34:51,480 Well actually, who knows these days, right? 681 00:34:52,480 --> 00:34:56,560 Yeah, and in this case I would say thank you for well-formatted, 682 00:34:56,880 --> 00:35:00,360 well-explained, well-structured response and so on. 683 00:35:00,560 --> 00:35:01,060 Definitely. 684 00:35:01,340 --> 00:35:06,560 So you try to find good things and mitigate my anger, calm me 685 00:35:06,560 --> 00:35:06,980 down. 686 00:35:06,980 --> 00:35:08,040 Thank you so much. 687 00:35:08,200 --> 00:35:09,340 Thank you for everything. 688 00:35:11,820 --> 00:35:12,340 Michael: Well, it's good. 689 00:35:12,340 --> 00:35:13,060 It's interesting. 690 00:35:13,260 --> 00:35:14,440 I found this one interesting. 691 00:35:14,440 --> 00:35:15,060 Thank you. 692 00:35:15,060 --> 00:35:18,480 Nikolay: Yeah, less technical discussion today, but I hope it 693 00:35:18,480 --> 00:35:20,540 provokes some thoughts a little bit. 694 00:35:21,220 --> 00:35:23,620 I think changes are inevitable. 695 00:35:23,920 --> 00:35:27,620 I'm very curious in which direction the whole market will go 696 00:35:27,620 --> 00:35:29,360 eventually, but let's see. 697 00:35:30,060 --> 00:35:30,800 Michael: Me too. 698 00:35:31,260 --> 00:35:31,760 Nikolay: Good. 699 00:35:31,860 --> 00:35:34,820 Michael: Well, have a good week and catch you next time. 700 00:35:34,820 --> 00:35:35,320 Nikolay: Thank you. 701 00:35:35,320 --> 00:35:35,820 See you.