1 00:00:00,220 --> 00:00:01,010 Daniel: And the best part 2 00:00:01,010 --> 00:00:02,429 of it is, it actually works. 3 00:00:03,135 --> 00:00:04,005 Like that's, that's what 4 00:00:04,005 --> 00:00:05,145 I love about Amazon. 5 00:00:05,145 --> 00:00:06,985 Like, oh, it's so complicated. 6 00:00:07,175 --> 00:00:08,295 Uh, there's so much scale to 7 00:00:08,504 --> 00:00:09,934 it and it continues to work. 8 00:00:15,245 --> 00:00:16,964 Corey: Welcome to Screaming in the Cloud. 9 00:00:17,155 --> 00:00:18,244 I'm Corey Quinn. 10 00:00:18,435 --> 00:00:20,645 A couple of years back, I had my 11 00:00:20,645 --> 00:00:22,975 annual charity t shirt focus on 12 00:00:23,014 --> 00:00:25,274 S3 as the eighth wonder of the 13 00:00:25,274 --> 00:00:27,415 world, because it legitimately is. 14 00:00:27,445 --> 00:00:30,705 It is an amazing service that has in many 15 00:00:30,705 --> 00:00:33,839 cases transformed the way that We store 16 00:00:33,839 --> 00:00:36,260 things the way we improperly use things as 17 00:00:36,269 --> 00:00:38,000 message queues or databases that perhaps 18 00:00:38,019 --> 00:00:40,480 shouldn't be, and myriad other things. 19 00:00:40,760 --> 00:00:42,559 However, I firmly believe that there is 20 00:00:42,569 --> 00:00:45,219 nothing so perfect that it cannot be made 21 00:00:45,230 --> 00:00:48,480 fun of, nitpicked to death, or in other 22 00:00:48,480 --> 00:00:50,489 ways dragged through the public square. 23 00:00:50,800 --> 00:00:52,809 My guest today apparently 24 00:00:52,809 --> 00:00:54,209 feels somewhat the same. 25 00:00:54,480 --> 00:00:56,529 Daniel Gjerlach is the Chief 26 00:00:56,550 --> 00:00:58,519 Innovation Officer at Plarion. 27 00:00:58,709 --> 00:01:00,019 Daniel, thank you for joining me. 28 00:01:00,139 --> 00:01:00,749 Daniel: I'm excited. 29 00:01:00,749 --> 00:01:02,429 I'm a big fan of your shitposting, Corey. 30 00:01:02,550 --> 00:01:03,080 Corey: Well, thank you. 31 00:01:03,080 --> 00:01:04,209 I try to be as well. 32 00:01:04,220 --> 00:01:05,069 Cause otherwise I'd get 33 00:01:05,069 --> 00:01:06,269 really bored really fast. 34 00:01:06,289 --> 00:01:07,080 Cause let's face it. 35 00:01:07,140 --> 00:01:08,470 Our industry is boring if 36 00:01:08,470 --> 00:01:09,640 you take it too seriously. 37 00:01:09,800 --> 00:01:12,509 This episode has been sponsored by our 38 00:01:12,520 --> 00:01:15,440 friends at Panoptica, part of Cisco. 39 00:01:15,735 --> 00:01:17,585 This is one of those real rarities 40 00:01:17,604 --> 00:01:19,485 where it's a security product that 41 00:01:19,485 --> 00:01:22,035 you can get started with for free, 42 00:01:22,585 --> 00:01:25,105 but also scale to enterprise grade. 43 00:01:25,585 --> 00:01:26,225 Take a look. 44 00:01:26,565 --> 00:01:28,134 In fact, if you sign up for an enterprise 45 00:01:28,134 --> 00:01:30,274 account, they'll even throw you one of 46 00:01:30,284 --> 00:01:33,205 the limited, heavily discounted AWS skill 47 00:01:33,215 --> 00:01:35,728 builder licenses they got, because believe 48 00:01:35,728 --> 00:01:38,480 it or not, unlike so many companies 49 00:01:38,480 --> 00:01:40,839 out there, they do understand AWS. 50 00:01:41,200 --> 00:01:43,710 To learn more, please visit panoptica. 51 00:01:43,760 --> 00:01:46,380 app slash last week in AWS. 52 00:01:46,730 --> 00:01:48,390 That's panoptica. 53 00:01:48,390 --> 00:01:51,480 app slash last week in AWS. 54 00:01:51,960 --> 00:01:53,760 You had a post that came out about a week 55 00:01:53,760 --> 00:01:55,700 before this recording titled things you 56 00:01:55,700 --> 00:01:58,430 wish you didn't need to know about S3. 57 00:01:58,625 --> 00:02:01,145 And I saw that come across my desk, 58 00:02:01,145 --> 00:02:03,065 and okay, great, let's look into this, 59 00:02:03,095 --> 00:02:05,275 because I've seen blog posts with 60 00:02:05,285 --> 00:02:07,615 similar titles somewhat frequently 61 00:02:07,625 --> 00:02:09,584 over the years, and it's, I bet you 62 00:02:09,585 --> 00:02:11,615 didn't know this one weird trick, and 63 00:02:11,645 --> 00:02:13,814 invariably there's not a whole lot of 64 00:02:13,814 --> 00:02:15,955 new information hidden in those posts. 65 00:02:16,094 --> 00:02:17,654 There was a lot of new information 66 00:02:17,655 --> 00:02:19,585 hidden in this post, so I absolutely 67 00:02:19,585 --> 00:02:20,985 wanted to talk to you about it. 68 00:02:21,135 --> 00:02:22,555 Where did this post come from? 69 00:02:22,575 --> 00:02:23,345 I guess is probably the 70 00:02:23,355 --> 00:02:24,655 best starting point for us. 71 00:02:24,785 --> 00:02:25,085 Daniel: Right. 72 00:02:25,175 --> 00:02:27,115 So, so, so before I jump into that, I 73 00:02:27,115 --> 00:02:29,089 actually don't think there was much. 74 00:02:29,380 --> 00:02:31,120 New in the post, there was 75 00:02:31,120 --> 00:02:33,000 something new for everyone. 76 00:02:33,000 --> 00:02:34,740 Like everyone found something interesting 77 00:02:34,860 --> 00:02:37,349 and the genesis of it was we were 78 00:02:37,350 --> 00:02:39,760 trying to build some more detailed 79 00:02:39,870 --> 00:02:41,430 risk analysis about S3 at Plarium. 80 00:02:42,059 --> 00:02:43,830 So I went and started having a look 81 00:02:43,830 --> 00:02:45,599 at like, how does, how does it work? 82 00:02:45,639 --> 00:02:47,110 Make sure we get everything right. 83 00:02:47,129 --> 00:02:47,930 Make sure we've got all 84 00:02:47,970 --> 00:02:48,980 the details correct. 85 00:02:49,140 --> 00:02:50,760 And so I started testing 86 00:02:50,790 --> 00:02:52,250 S3, started playing with it. 87 00:02:52,410 --> 00:02:54,060 And every time I found a quirk, 88 00:02:54,490 --> 00:02:56,170 I went, Oh, that's, that's not 89 00:02:56,170 --> 00:02:57,510 exactly how I thought it would work. 90 00:02:57,805 --> 00:02:58,915 I would send it to my engineering 91 00:02:58,975 --> 00:03:00,845 team and they would go, no, that's 92 00:03:00,865 --> 00:03:02,025 not right, that can't be right. 93 00:03:02,175 --> 00:03:03,685 And so the more I kept going, the 94 00:03:03,685 --> 00:03:05,825 more of these quirks I would find. 95 00:03:06,015 --> 00:03:07,075 And that's how we ended 96 00:03:07,075 --> 00:03:08,135 up making the list. 97 00:03:08,135 --> 00:03:10,015 It's just someone would always go, I 98 00:03:10,025 --> 00:03:11,404 thought it worked differently in some way. 99 00:03:11,534 --> 00:03:12,384 Corey: When I say a lot of 100 00:03:12,385 --> 00:03:13,234 this was new information, I 101 00:03:13,244 --> 00:03:14,204 mean, some of it was new to me. 102 00:03:14,255 --> 00:03:15,624 Uh, if I haven't seen it 103 00:03:15,624 --> 00:03:16,955 before, it's new to me. 104 00:03:17,135 --> 00:03:18,615 I don't know that there were necessarily 105 00:03:18,615 --> 00:03:20,745 any groundbreaking revelations that, 106 00:03:20,885 --> 00:03:22,804 The S3 team is gonna be reading this 107 00:03:22,804 --> 00:03:24,785 and going, holy crap, it does what? 108 00:03:25,134 --> 00:03:26,784 But it, but it addresses a bunch of 109 00:03:26,784 --> 00:03:29,664 things that I had either not been 110 00:03:29,664 --> 00:03:31,404 aware of or in some cases not thought 111 00:03:31,404 --> 00:03:32,724 about for a while, and others should 112 00:03:32,724 --> 00:03:33,954 never bothered to think about it all. 113 00:03:33,954 --> 00:03:35,634 I mean, your first point's terrific where 114 00:03:35,634 --> 00:03:38,874 you talk about S3 buckets are the S3 API. 115 00:03:39,054 --> 00:03:40,584 It never occurred to me to dig into 116 00:03:40,584 --> 00:03:42,924 the, the question of why the AWS command 117 00:03:42,924 --> 00:03:45,684 line interface has a sub command of S3 118 00:03:45,684 --> 00:03:48,654 and a separate sub command of S3 API. 119 00:03:49,015 --> 00:03:49,825 What is that? 120 00:03:49,825 --> 00:03:51,845 I know I use the latter when I want to 121 00:03:51,915 --> 00:03:54,135 handle weird modifications to buckets and 122 00:03:54,135 --> 00:03:56,655 whatnot, and the S3 subcommand, generally 123 00:03:56,655 --> 00:03:58,914 when I just create or delete a bucket and 124 00:03:58,954 --> 00:04:01,485 work with objects within it, but it had 125 00:04:01,555 --> 00:04:02,975 never occurred to me to delve into the 126 00:04:02,975 --> 00:04:04,575 nuances of why in the way that you did. 127 00:04:04,725 --> 00:04:05,035 Daniel: Yeah. 128 00:04:05,385 --> 00:04:07,374 And I'm not clear on the difference 129 00:04:07,374 --> 00:04:08,594 between why there's two command 130 00:04:08,674 --> 00:04:10,295 lines, but I think the big difference 131 00:04:10,295 --> 00:04:11,785 is the end point that you end 132 00:04:11,785 --> 00:04:14,285 up sending API operations to. 133 00:04:14,605 --> 00:04:17,054 Typically it's a central end point. 134 00:04:17,495 --> 00:04:18,825 So, again, Controls the whole 135 00:04:18,834 --> 00:04:20,575 service, and then you provide the 136 00:04:20,575 --> 00:04:22,575 parameters that you want it to use. 137 00:04:22,870 --> 00:04:24,440 Here we basically send the 138 00:04:24,440 --> 00:04:25,890 API calls to the S3 bucket. 139 00:04:26,050 --> 00:04:28,380 Now, I'm not sure if underlying that 140 00:04:28,410 --> 00:04:31,130 is some big general endpoint, but for 141 00:04:31,130 --> 00:04:32,710 the user, that's the way it looks. 142 00:04:33,000 --> 00:04:36,329 And so when you are able to go and then 143 00:04:36,450 --> 00:04:38,410 delete that bucket by just sending it 144 00:04:38,410 --> 00:04:40,499 a HTTP request, That's something that 145 00:04:40,499 --> 00:04:42,429 people don't necessarily expect to happen. 146 00:04:42,619 --> 00:04:43,940 Corey: I would honestly expect that not 147 00:04:43,950 --> 00:04:45,499 to happen just because CloudFormation 148 00:04:45,530 --> 00:04:47,520 likes to lose its mind every time you 149 00:04:47,559 --> 00:04:50,169 try to delete a stack that has a bucket 150 00:04:50,169 --> 00:04:52,640 in it that has a lingering thing there. 151 00:04:52,830 --> 00:04:54,409 And they recently fixed the ability 152 00:04:54,590 --> 00:04:55,879 to, Oh, you can just go ahead and 153 00:04:55,880 --> 00:04:57,109 delete the stack now and check 154 00:04:57,109 --> 00:04:58,834 the box to orphan the bucket. 155 00:04:59,114 --> 00:05:00,565 No, I want you to clean the 156 00:05:00,575 --> 00:05:01,745 thing up and get rid of it. 157 00:05:01,745 --> 00:05:03,325 I'm telling you explicitly, go 158 00:05:03,325 --> 00:05:04,854 ahead and blow out the data. 159 00:05:05,094 --> 00:05:06,224 I don't care. 160 00:05:06,234 --> 00:05:08,155 This is all for scratch stuff anyway. 161 00:05:08,304 --> 00:05:09,994 And I understand why you don't want 162 00:05:09,995 --> 00:05:11,124 to make it too easy to do that in 163 00:05:11,125 --> 00:05:13,395 production by accident, but there are 164 00:05:13,575 --> 00:05:15,484 different use cases for different things. 165 00:05:15,655 --> 00:05:15,865 Daniel: Yeah. 166 00:05:16,094 --> 00:05:17,455 And I think that's still true. 167 00:05:17,544 --> 00:05:19,164 The bucket still needs to be emptied. 168 00:05:19,294 --> 00:05:21,034 The interesting thing in my blog post 169 00:05:21,034 --> 00:05:23,495 was that if you accidentally say, give 170 00:05:23,575 --> 00:05:27,035 S3 star permissions, Uh, on a bucket, you 171 00:05:27,035 --> 00:05:30,365 can actually just send a delete request to 172 00:05:30,365 --> 00:05:32,615 the bucket and the bucket will be deleted. 173 00:05:32,785 --> 00:05:34,264 Now, obviously, permissions have 174 00:05:34,275 --> 00:05:35,744 to be wildly misconfigured for 175 00:05:35,744 --> 00:05:36,865 something like that to happen. 176 00:05:36,994 --> 00:05:37,604 Corey: Yeah, and who would ever 177 00:05:37,605 --> 00:05:39,055 screw up IAM permissions, right? 178 00:05:39,195 --> 00:05:39,755 Daniel: Exactly. 179 00:05:39,755 --> 00:05:40,374 That's what everyone's been 180 00:05:40,374 --> 00:05:41,694 doing for a very long time. 181 00:05:41,905 --> 00:05:43,124 And that's, that's the point here. 182 00:05:43,124 --> 00:05:45,104 It's like, AWS has done a really 183 00:05:45,104 --> 00:05:46,955 good job over the years of 184 00:05:47,104 --> 00:05:48,734 removing many of the foot guns. 185 00:05:48,905 --> 00:05:50,794 I couldn't see a use case for 186 00:05:50,875 --> 00:05:53,614 where a bucket could be deleted by 187 00:05:53,625 --> 00:05:55,044 literally anyone on the internet. 188 00:05:55,174 --> 00:05:56,414 So that's a foot gun that I think 189 00:05:56,715 --> 00:05:57,804 could be removed in the future. 190 00:05:57,924 --> 00:05:59,134 Corey: I am very surprised by that. 191 00:05:59,394 --> 00:06:00,405 Did you test whether it works 192 00:06:00,414 --> 00:06:01,594 if there's data in the bucket? 193 00:06:01,724 --> 00:06:02,784 Daniel: I didn't, but, 194 00:06:02,785 --> 00:06:04,380 but I expect it doesn't. 195 00:06:04,560 --> 00:06:06,330 I still think you've got to remove all 196 00:06:06,330 --> 00:06:08,820 the objects inside before you run it. 197 00:06:08,960 --> 00:06:09,870 Corey: That would at least make it a 198 00:06:09,880 --> 00:06:12,110 little bit better in that I don't, I 199 00:06:12,110 --> 00:06:13,950 can't think of any buckets in my accounts 200 00:06:13,950 --> 00:06:15,660 that are purely empty with not a single 201 00:06:15,700 --> 00:06:17,380 object or version stored within them. 202 00:06:17,539 --> 00:06:19,329 Daniel: But I expect if you have S3 203 00:06:19,340 --> 00:06:21,249 star permissions misconfigured, then 204 00:06:21,250 --> 00:06:23,170 you can probably just Send delete 205 00:06:23,280 --> 00:06:25,130 requests for all the objects as well. 206 00:06:25,290 --> 00:06:26,160 Corey: Yeah, I guess by the point 207 00:06:26,170 --> 00:06:27,120 you can delete, you can definitely 208 00:06:27,120 --> 00:06:28,450 do a list and see the buckets. 209 00:06:28,480 --> 00:06:30,020 Uh, frankly, at that point, there's no 210 00:06:30,020 --> 00:06:31,260 reason you couldn't also just send a 211 00:06:31,260 --> 00:06:33,259 lifecycle policy up as well and configure 212 00:06:33,260 --> 00:06:34,450 it to just blow everything away. 213 00:06:34,570 --> 00:06:35,829 Daniel: It's just a, I mean, 214 00:06:35,989 --> 00:06:37,259 it's working as expected. 215 00:06:37,409 --> 00:06:38,859 You've literally given anyone 216 00:06:38,860 --> 00:06:39,739 on the internet all the 217 00:06:39,739 --> 00:06:41,010 permissions for the S3 bucket. 218 00:06:41,325 --> 00:06:42,784 It's just, I just don't think there's 219 00:06:42,784 --> 00:06:44,534 a use case for something like that. 220 00:06:44,684 --> 00:06:45,664 Corey: You found a whole bunch 221 00:06:45,664 --> 00:06:47,284 of strange things in this. 222 00:06:47,314 --> 00:06:48,655 Uh, I, it's been a while since 223 00:06:48,655 --> 00:06:49,934 I thought about this, but I have 224 00:06:49,934 --> 00:06:51,614 found clients at smaller scale 225 00:06:51,614 --> 00:06:52,765 where this becomes significant. 226 00:06:53,025 --> 00:06:55,434 At, at large enough scale, all, all 227 00:06:55,444 --> 00:06:57,914 weird billing misconfigurations become 228 00:06:57,914 --> 00:06:59,884 small enough not to matter because 229 00:06:59,884 --> 00:07:01,354 they would have gotten fixed otherwise. 230 00:07:01,604 --> 00:07:03,875 Uh, You wind up with the Schrodinger's 231 00:07:03,875 --> 00:07:05,945 Objects, as you call them, of incomplete 232 00:07:05,955 --> 00:07:07,925 multi part uploads for objects. 233 00:07:08,255 --> 00:07:10,185 If the upload fails midway through, 234 00:07:10,194 --> 00:07:11,505 those objects that have already been 235 00:07:11,505 --> 00:07:13,515 received by default sit around forever. 236 00:07:13,685 --> 00:07:14,974 They don't show up in the console, 237 00:07:14,975 --> 00:07:16,215 they don't show up under a list 238 00:07:16,215 --> 00:07:18,025 objects without very specific 239 00:07:18,035 --> 00:07:20,634 parameters, and they do charge you. 240 00:07:20,980 --> 00:07:22,720 So I have seen in the early days when 241 00:07:22,720 --> 00:07:24,210 I was working at a much smaller scale, 242 00:07:24,570 --> 00:07:26,230 yeah, someone said that, alright, I have 243 00:07:26,230 --> 00:07:28,920 a 1TB bucket, why am I being charged 50TB? 244 00:07:29,109 --> 00:07:30,839 And incomplete multi part uploads 245 00:07:30,870 --> 00:07:32,520 were the issue, which at that 246 00:07:32,520 --> 00:07:33,549 point became clear that there's 247 00:07:33,550 --> 00:07:35,120 something systemically wrong here. 248 00:07:35,279 --> 00:07:36,680 Figure out what keeps dying 249 00:07:36,689 --> 00:07:38,139 trying to upload things and 250 00:07:38,150 --> 00:07:39,270 make sure that gets fixed. 251 00:07:39,290 --> 00:07:40,740 And also, here's a lifecycle 252 00:07:40,740 --> 00:07:42,230 policy to clean those out 253 00:07:42,240 --> 00:07:44,150 to fix the end result of it. 254 00:07:44,390 --> 00:07:46,130 But it's been a while since I've seen 255 00:07:46,130 --> 00:07:47,630 that because most folks are not gonna 256 00:07:47,630 --> 00:07:49,040 be spending a hundred million dollars a 257 00:07:49,040 --> 00:07:52,610 year on AWS and discover that $20 million 258 00:07:52,610 --> 00:07:55,160 of it is incomplete, uh, S3 uploads. 259 00:07:55,160 --> 00:07:56,750 That just, that isn't a thing that 260 00:07:56,750 --> 00:07:57,950 happens here in the real world. 261 00:07:58,100 --> 00:08:00,020 Daniel: No, uh, and I think that the 262 00:08:00,025 --> 00:08:01,850 interesting thing here is that you could 263 00:08:01,855 --> 00:08:03,410 have it happen by accident, where you 264 00:08:03,470 --> 00:08:04,910 end up with a bucket with all these 265 00:08:04,910 --> 00:08:06,350 objects that you don't know about, but 266 00:08:06,350 --> 00:08:08,630 also if you allow anonymous people on the 267 00:08:08,630 --> 00:08:10,520 internet to dump stuff in your buckets. 268 00:08:10,870 --> 00:08:12,240 then they could potentially 269 00:08:12,240 --> 00:08:13,100 do it on purpose. 270 00:08:13,330 --> 00:08:15,000 Corey: Oh yeah, it's, and there is, it is 271 00:08:15,000 --> 00:08:16,620 the least discoverable thing in the world. 272 00:08:16,630 --> 00:08:19,690 There is no way, except maybe S3 Storage 273 00:08:19,690 --> 00:08:21,260 Lens, if you really go looking, to 274 00:08:21,260 --> 00:08:23,799 figure out what, how many of those 275 00:08:23,800 --> 00:08:25,499 you have account wide and whether 276 00:08:25,500 --> 00:08:26,730 or not you are being, and what it 277 00:08:26,730 --> 00:08:27,939 is you're being charged for those. 278 00:08:28,089 --> 00:08:30,220 It's one of those very, you have to 279 00:08:30,220 --> 00:08:32,110 know the secret passcode to get in 280 00:08:32,169 --> 00:08:34,380 to the hidden speakeasy in order for 281 00:08:34,380 --> 00:08:36,439 that to begin making sense or being 282 00:08:36,439 --> 00:08:37,429 something you'd even, that would 283 00:08:37,429 --> 00:08:38,970 even occur to you that might exist. 284 00:08:39,189 --> 00:08:40,319 Daniel: Yeah, and like you said, 285 00:08:40,319 --> 00:08:41,659 though, you can put a lifecycle 286 00:08:41,679 --> 00:08:43,140 policy on all of your buckets to 287 00:08:43,140 --> 00:08:44,540 protect against this kind of thing. 288 00:08:44,699 --> 00:08:46,170 It's just that I'm not sure that 289 00:08:46,230 --> 00:08:47,999 anyone does that by default. 290 00:08:48,339 --> 00:08:49,689 Corey: I did not know, for example, 291 00:08:49,689 --> 00:08:51,270 that multi part upload listings 292 00:08:51,270 --> 00:08:53,709 will return the principal ARNs. 293 00:08:53,860 --> 00:08:55,040 That was novel to me. 294 00:08:55,139 --> 00:08:55,979 Daniel: Yeah, that's a fun one. 295 00:08:56,010 --> 00:08:58,720 And look, there's not much confidential 296 00:08:58,730 --> 00:09:01,880 about a principal ARN, but in some cases 297 00:09:01,920 --> 00:09:03,949 an attacker, like, wants to do something 298 00:09:03,980 --> 00:09:05,790 and they don't know what the resource 299 00:09:05,790 --> 00:09:07,959 identifier is that they need to target. 300 00:09:08,245 --> 00:09:09,904 And so when you leak these kind of 301 00:09:09,905 --> 00:09:11,145 bits of information all over the place, 302 00:09:11,415 --> 00:09:13,405 there's some very specific edge cases 303 00:09:13,405 --> 00:09:15,225 in which, hey, knowing a resource 304 00:09:15,225 --> 00:09:16,975 name or knowing a full ARN is really 305 00:09:16,975 --> 00:09:18,774 important from an attacker's perspective. 306 00:09:18,955 --> 00:09:21,084 Corey: What I've found is that when I've 307 00:09:21,084 --> 00:09:23,895 talked to the S3 team, it is pretty clear 308 00:09:24,025 --> 00:09:25,514 that, I mean, they put a good spin on it, 309 00:09:25,515 --> 00:09:27,435 don't get me wrong, but it is abundantly 310 00:09:27,435 --> 00:09:30,085 clear that in nobody's wildest dreams, 311 00:09:30,105 --> 00:09:33,045 when this service was built and What damn 312 00:09:33,045 --> 00:09:35,085 near 20 years ago now, uh, that what it 313 00:09:35,115 --> 00:09:37,205 would grow into, what it would become. 314 00:09:37,525 --> 00:09:39,824 And for every other AWS service where 315 00:09:39,824 --> 00:09:41,294 I've spoken to service teams, they 316 00:09:41,295 --> 00:09:42,964 learn more about their services from how 317 00:09:42,964 --> 00:09:45,265 customers use or in some cases misuse 318 00:09:45,265 --> 00:09:47,864 them than was ever accounted for in any 319 00:09:47,865 --> 00:09:49,564 planning document that could have existed. 320 00:09:49,685 --> 00:09:52,204 Daniel: I think AWS, the S3 history, 321 00:09:52,444 --> 00:09:54,444 is one of the fun parts of this. 322 00:09:54,605 --> 00:09:55,725 It's obviously one of the most 323 00:09:55,725 --> 00:09:57,765 robust, most used services that's 324 00:09:57,765 --> 00:09:58,985 been around for a long time. 325 00:09:59,125 --> 00:10:00,735 But because of that, it's got some 326 00:10:00,735 --> 00:10:02,865 of the sort of the archaeology 327 00:10:03,055 --> 00:10:05,304 of its past that's now gone away. 328 00:10:05,305 --> 00:10:06,535 For example, ACLs. 329 00:10:06,715 --> 00:10:09,404 Now all resources in AWS are protected 330 00:10:09,404 --> 00:10:11,215 by fancy policies that are very well 331 00:10:11,215 --> 00:10:12,675 defined and very well understood. 332 00:10:12,834 --> 00:10:15,420 But In the past, S3 had this concept 333 00:10:15,420 --> 00:10:17,069 of ACLs, which is now turned off 334 00:10:17,069 --> 00:10:19,189 by default, but you can still do 335 00:10:19,269 --> 00:10:21,150 interesting things with it, things 336 00:10:21,150 --> 00:10:22,660 that you perhaps don't expect. 337 00:10:22,819 --> 00:10:25,949 And the mental model for that is very 338 00:10:25,949 --> 00:10:28,550 different to what it is for IAM policies. 339 00:10:28,740 --> 00:10:29,290 Corey: Absolutely. 340 00:10:29,290 --> 00:10:32,319 I was always so confused by a bucket ACLs. 341 00:10:32,349 --> 00:10:33,920 I inadvertently reported what I 342 00:10:33,930 --> 00:10:35,770 thought was a security issue, politely, 343 00:10:35,810 --> 00:10:37,110 because I'm never sure of what I'm 344 00:10:37,110 --> 00:10:39,469 looking at when I didn't understand the 345 00:10:39,479 --> 00:10:42,209 interplay of an ACL with an IAM policy. 346 00:10:42,445 --> 00:10:44,225 And was, and found very quickly that 347 00:10:44,235 --> 00:10:46,405 nope, the problem is, is that I'm a fool. 348 00:10:46,605 --> 00:10:48,665 Okay, great, I can own and accept that. 349 00:10:48,905 --> 00:10:50,954 But I'm also never the only fool. 350 00:10:50,955 --> 00:10:53,075 I generally have good company in 351 00:10:53,085 --> 00:10:54,685 people making poor decisions as 352 00:10:54,685 --> 00:10:56,245 I travel throughout the industry. 353 00:10:56,385 --> 00:10:57,805 Daniel: Uh, do you remember the 354 00:10:57,814 --> 00:10:59,525 old authenticated users group? 355 00:10:59,835 --> 00:11:00,725 in ACLs. 356 00:11:00,885 --> 00:11:02,925 Corey: With the checkbox next to it in 357 00:11:02,925 --> 00:11:04,835 the S3 console, people would click it. 358 00:11:04,865 --> 00:11:06,665 I know I did in the very early days, 359 00:11:06,675 --> 00:11:08,934 assuming, oh, any user authenticated to my 360 00:11:08,934 --> 00:11:11,175 AWS account should be able to look at this 361 00:11:11,175 --> 00:11:13,015 bucket because it's a company wide thing. 362 00:11:13,134 --> 00:11:13,374 Yeah. 363 00:11:13,375 --> 00:11:14,755 Turns out that meant every 364 00:11:15,075 --> 00:11:17,225 AWS S3 user on the planet. 365 00:11:17,345 --> 00:11:17,515 Daniel: Yeah. 366 00:11:17,515 --> 00:11:19,354 And that was a fun one where people 367 00:11:19,645 --> 00:11:21,685 could make that mistake very legitimately 368 00:11:21,685 --> 00:11:23,495 going like it's authenticated users. 369 00:11:23,505 --> 00:11:24,185 It must be ours. 370 00:11:24,195 --> 00:11:25,435 It's not everyone on the planet, 371 00:11:25,585 --> 00:11:26,715 but I think that's part of the 372 00:11:26,735 --> 00:11:28,624 interesting archeology of S3. 373 00:11:28,865 --> 00:11:29,034 And so. 374 00:11:29,715 --> 00:11:31,275 acls have a bunch of those 375 00:11:31,395 --> 00:11:32,895 interesting quirks like that. 376 00:11:33,045 --> 00:11:33,885 For example. 377 00:11:34,125 --> 00:11:35,805 The other one that I, uh, that I ran 378 00:11:35,805 --> 00:11:38,505 into was that you can pri provide 379 00:11:38,505 --> 00:11:41,265 permissions to people based on the 380 00:11:41,295 --> 00:11:43,755 email address associated with the 381 00:11:44,055 --> 00:11:45,735 root user of their AWS account. 382 00:11:45,915 --> 00:11:47,655 And there's a fancy error message that 383 00:11:47,655 --> 00:11:49,725 comes up and tells you if that email 384 00:11:49,725 --> 00:11:52,005 address doesn't exist in the AWS database. 385 00:11:52,280 --> 00:11:54,020 So you can basically figure out if an 386 00:11:54,020 --> 00:11:56,220 email has an AWS account associated 387 00:11:56,220 --> 00:11:58,010 with it, which is another thing to work. 388 00:11:58,230 --> 00:11:59,540 Corey: That is wild to me because remember 389 00:11:59,540 --> 00:12:01,540 the canonical user ID where you used to 390 00:12:01,540 --> 00:12:04,560 have to, where there was this giant, I 391 00:12:04,570 --> 00:12:06,250 don't know what the hell it was, it was 392 00:12:06,250 --> 00:12:09,290 some huge alphanumeric string as I recall. 393 00:12:09,530 --> 00:12:11,840 That was the canonical user that owned 394 00:12:11,860 --> 00:12:13,960 the bucket, because S3 significantly 395 00:12:13,960 --> 00:12:16,610 predates IAM and everything else. 396 00:12:16,810 --> 00:12:17,900 It's part of the, basically, 397 00:12:17,900 --> 00:12:19,930 the fossil record at this point. 398 00:12:20,150 --> 00:12:22,289 But it's, it was always a separate 399 00:12:22,449 --> 00:12:24,809 AWS user identity in some ways. 400 00:12:24,830 --> 00:12:25,679 I never saw it used for 401 00:12:25,679 --> 00:12:27,790 much other than S3 policies. 402 00:12:27,890 --> 00:12:28,960 That really messed with me. 403 00:12:29,070 --> 00:12:30,060 Daniel: Yeah, I think it's 404 00:12:30,060 --> 00:12:32,220 a 32 character hex string. 405 00:12:32,420 --> 00:12:34,230 And another fun thing about that is 406 00:12:34,360 --> 00:12:36,050 if you find that string anywhere, for 407 00:12:36,050 --> 00:12:38,690 example, uh, you know, objects listing, 408 00:12:38,830 --> 00:12:40,869 you take that canonical string, you 409 00:12:40,870 --> 00:12:42,959 chuck it in an IAM policy, save it. 410 00:12:43,109 --> 00:12:45,450 And when you see that policy come back up 411 00:12:45,450 --> 00:12:48,190 again, it'll have that string resolved, 412 00:12:48,339 --> 00:12:51,620 uh, to the ARN of that canonical user. 413 00:12:51,840 --> 00:12:53,350 Corey: Wild to me that, 414 00:12:53,350 --> 00:12:54,600 that, that still works. 415 00:12:54,600 --> 00:12:56,080 I mean, it makes sense that it does. 416 00:12:56,665 --> 00:12:58,705 S3 is so much lore around it now 417 00:12:58,705 --> 00:13:00,135 that people legitimately don't 418 00:13:00,135 --> 00:13:01,755 know when I'm shitposting or not. 419 00:13:01,955 --> 00:13:04,075 I'll talk sometimes about how S3 used 420 00:13:04,075 --> 00:13:06,275 to basically have a BitTorrent endpoint 421 00:13:06,335 --> 00:13:08,045 if you enabled it on a bucket, and 422 00:13:08,045 --> 00:13:09,655 people thought I was making it up. 423 00:13:09,715 --> 00:13:10,750 That was one of the few deprecations 424 00:13:10,750 --> 00:13:13,174 that AWS has done, because it turned 425 00:13:13,174 --> 00:13:15,179 out approximately nobody needs it. 426 00:13:15,310 --> 00:13:17,630 It's not how we transfer large files 427 00:13:17,630 --> 00:13:19,450 across the modern internet anymore. 428 00:13:19,620 --> 00:13:20,610 For at a time when a lot of 429 00:13:20,610 --> 00:13:21,660 folks were highly bandwidth 430 00:13:21,660 --> 00:13:23,470 constrained, that was not nothing. 431 00:13:23,600 --> 00:13:24,609 Daniel: Yeah, it's actually 432 00:13:24,609 --> 00:13:25,919 still in the documentation. 433 00:13:26,109 --> 00:13:27,430 That's one of the things I ran into. 434 00:13:27,670 --> 00:13:28,499 Oh, that can't be right. 435 00:13:28,529 --> 00:13:29,940 Went and tested it and found out it 436 00:13:29,970 --> 00:13:31,950 was deprecated, but it's still hanging 437 00:13:31,950 --> 00:13:33,400 around in, uh, in some writing. 438 00:13:33,569 --> 00:13:35,230 Corey: The documentation is basically 439 00:13:35,230 --> 00:13:37,190 engraved upon stone tablets. 440 00:13:38,110 --> 00:13:39,510 As I recall, they have a 441 00:13:39,510 --> 00:13:41,370 couple of versions of the API. 442 00:13:41,370 --> 00:13:42,730 Like one is like a 2012 date. 443 00:13:42,730 --> 00:13:43,750 And I think it's the last time it 444 00:13:43,750 --> 00:13:45,479 was updated where you still have to 445 00:13:45,480 --> 00:13:47,279 specify for some things, a version 446 00:13:47,280 --> 00:13:50,620 string that references a date that is 447 00:13:50,670 --> 00:13:53,240 older than my elementary school child. 448 00:13:53,409 --> 00:13:53,789 Nice. 449 00:13:54,060 --> 00:13:54,550 Daniel: That's fun. 450 00:13:54,985 --> 00:13:55,915 Corey: Few things are better for 451 00:13:55,915 --> 00:13:57,495 your career and your company than 452 00:13:57,495 --> 00:14:00,365 achieving more expertise in the cloud. 453 00:14:00,515 --> 00:14:02,665 Security improves, compensation goes 454 00:14:02,665 --> 00:14:04,965 up, employee retention skyrockets. 455 00:14:05,385 --> 00:14:07,605 Panoptica, a cloud security platform 456 00:14:07,605 --> 00:14:10,024 from Cisco, has created an academy 457 00:14:10,024 --> 00:14:12,124 of free courses just for you. 458 00:14:12,545 --> 00:14:14,065 Head on over to academy. 459 00:14:14,085 --> 00:14:14,355 panoptica. 460 00:14:15,625 --> 00:14:17,325 app to get started. 461 00:14:17,775 --> 00:14:19,095 It's it, you can't change 462 00:14:19,095 --> 00:14:19,895 these things very much. 463 00:14:20,224 --> 00:14:22,344 I was talking with Jeff Barr once, 464 00:14:22,415 --> 00:14:24,764 and he made a great observation 465 00:14:24,764 --> 00:14:25,974 that I asked if we turned it into 466 00:14:25,974 --> 00:14:27,194 a blog post, and he wrote the intro 467 00:14:27,194 --> 00:14:28,234 for it, which was lovely of him. 468 00:14:28,594 --> 00:14:31,514 But he talked about the idea that S3 469 00:14:31,535 --> 00:14:34,234 at this point has become a generational 470 00:14:34,234 --> 00:14:36,344 service, where they have no idea 471 00:14:36,354 --> 00:14:38,304 what's in any given S3 bucket. 472 00:14:38,344 --> 00:14:40,344 They aren't scanning stuff, and there are 473 00:14:40,374 --> 00:14:42,864 encryption practices and policies in place 474 00:14:42,864 --> 00:14:44,604 to prevent them from ever doing this. 475 00:14:44,800 --> 00:14:46,589 But it's definitely something 476 00:14:46,589 --> 00:14:47,469 they have to think about. 477 00:14:47,480 --> 00:14:48,469 Which, if you don't know what's 478 00:14:48,469 --> 00:14:50,130 in a given bucket, maybe it's a 479 00:14:50,130 --> 00:14:52,040 bunch of shitposting meme images. 480 00:14:52,360 --> 00:14:53,490 Maybe it's incredibly 481 00:14:53,490 --> 00:14:54,949 important bank records. 482 00:14:55,069 --> 00:14:56,449 Maybe it's the nuclear codes. 483 00:14:56,490 --> 00:14:57,589 They just don't know. 484 00:14:57,740 --> 00:15:00,660 So they have to not lose data, they have 485 00:15:00,660 --> 00:15:02,939 to make sure that it is accessible via 486 00:15:02,980 --> 00:15:05,730 a variety of embedded API calls that 487 00:15:05,730 --> 00:15:07,770 are never going to get updated anywhere. 488 00:15:08,069 --> 00:15:09,650 And they have to make plans for 489 00:15:09,650 --> 00:15:11,469 this to still be there in 500 years. 490 00:15:11,489 --> 00:15:13,530 Because as long as the account, the 491 00:15:13,530 --> 00:15:15,870 bill gets paid on an account, who's to 492 00:15:15,870 --> 00:15:17,439 say whether something's right or wrong? 493 00:15:17,620 --> 00:15:18,299 Lord knows I have a 494 00:15:18,300 --> 00:15:19,439 bunch of old S3 buckets. 495 00:15:19,439 --> 00:15:20,499 I have no idea what's in them. 496 00:15:20,500 --> 00:15:21,489 I'll never touch again. 497 00:15:21,660 --> 00:15:22,819 And they round up to less than a 498 00:15:22,819 --> 00:15:24,620 penny, so I don't particularly care. 499 00:15:24,839 --> 00:15:26,640 But those things have to still exist. 500 00:15:26,920 --> 00:15:27,709 Daniel: And the best part 501 00:15:27,709 --> 00:15:29,465 of it is It actually works. 502 00:15:29,825 --> 00:15:30,705 Like that's, that's what 503 00:15:30,705 --> 00:15:31,845 I love about Amazon. 504 00:15:31,845 --> 00:15:33,685 Like, oh, it's so complicated. 505 00:15:33,875 --> 00:15:35,065 Uh, there's so much scale to it 506 00:15:35,065 --> 00:15:36,995 and it continues to work, but I 507 00:15:36,995 --> 00:15:38,475 really would love to touch on your 508 00:15:38,675 --> 00:15:40,345 encryption point if you don't mind. 509 00:15:40,675 --> 00:15:42,244 Cause I think that's, that's another 510 00:15:42,265 --> 00:15:44,485 area where they mistake assumptions 511 00:15:44,565 --> 00:15:46,475 about how S3 encryption works. 512 00:15:46,765 --> 00:15:48,445 And one of my friends was actually 513 00:15:48,445 --> 00:15:50,025 talking to a CISO after a major 514 00:15:50,025 --> 00:15:51,935 break and the CISO was telling them. 515 00:15:52,245 --> 00:15:54,055 Hey, it's okay that our objects 516 00:15:54,074 --> 00:15:55,985 got stolen because we've got 517 00:15:56,155 --> 00:15:57,795 encryption at rest enabled. 518 00:15:57,805 --> 00:15:59,115 So it's, it's perfectly fine. 519 00:15:59,295 --> 00:16:00,735 And so people's mental model for 520 00:16:00,735 --> 00:16:03,564 encryption is the file is encrypted. 521 00:16:03,954 --> 00:16:06,025 It goes away when the attacker 522 00:16:06,025 --> 00:16:07,435 tries to open the file. 523 00:16:07,689 --> 00:16:08,920 It'll be a bunch of garbage. 524 00:16:08,920 --> 00:16:09,990 They won't have the keys, so 525 00:16:09,990 --> 00:16:11,319 they won't be able to decrypt it. 526 00:16:11,500 --> 00:16:13,270 But that's not exactly how 527 00:16:13,270 --> 00:16:14,610 encryption works in S3. 528 00:16:14,689 --> 00:16:16,110 I can see you want to say something. 529 00:16:16,579 --> 00:16:17,860 Corey: Oh, last year I wrote 530 00:16:17,900 --> 00:16:19,160 a whole article on this. 531 00:16:19,170 --> 00:16:19,670 It's on my site. 532 00:16:19,670 --> 00:16:20,760 I'll put a link to it in the show notes. 533 00:16:20,790 --> 00:16:22,699 S3 encryption at rest does not 534 00:16:22,710 --> 00:16:24,389 solve for bucket negligence. 535 00:16:24,670 --> 00:16:26,880 And I go into a whole spiel on 536 00:16:26,910 --> 00:16:28,650 exactly what you're talking about. 537 00:16:28,680 --> 00:16:29,340 You're right. 538 00:16:29,500 --> 00:16:31,199 I always found that encryption at rest 539 00:16:31,230 --> 00:16:33,550 in the cloud context to be basically a 540 00:16:33,750 --> 00:16:35,890 box checking exercise and little more. 541 00:16:36,240 --> 00:16:37,880 Because, okay, if you can break into 542 00:16:37,880 --> 00:16:40,090 an AWS datacenter, steal the drives, 543 00:16:40,090 --> 00:16:42,030 get out alive, uh, have stolen them 544 00:16:42,030 --> 00:16:44,180 from the proper places to reassemble 545 00:16:44,220 --> 00:16:45,999 the, uh, the severed objects into 546 00:16:45,999 --> 00:16:47,670 various ways and recombine them, 547 00:16:47,910 --> 00:16:49,720 you kind of earned it at that point. 548 00:16:49,910 --> 00:16:52,795 That's not really my Encryption at 549 00:16:52,795 --> 00:16:54,245 rest matters a hell of a lot more for 550 00:16:54,245 --> 00:16:55,475 laptops that you're going to leave 551 00:16:55,475 --> 00:16:57,215 at the coffee shop or in your car. 552 00:16:57,455 --> 00:16:59,995 They matter a lot more for your crappy 553 00:16:59,995 --> 00:17:01,444 data center where the security guard 554 00:17:01,444 --> 00:17:04,194 forgets to go and lock the door at night. 555 00:17:04,494 --> 00:17:06,484 There are, those are going to be 556 00:17:06,494 --> 00:17:07,925 areas where it absolutely matters. 557 00:17:08,075 --> 00:17:10,234 With this, it just isn't a realistic 558 00:17:10,275 --> 00:17:11,715 threat model, because regardless of 559 00:17:11,715 --> 00:17:13,525 how well encrypted at rest something 560 00:17:13,525 --> 00:17:16,075 is, it still is going to be returned 561 00:17:16,185 --> 00:17:18,025 via the API when it's requested, 562 00:17:18,175 --> 00:17:19,425 assuming the permissions are right. 563 00:17:19,535 --> 00:17:21,334 There are exceptions with KMS 564 00:17:21,334 --> 00:17:21,984 encryption in certain ways. 565 00:17:22,065 --> 00:17:22,695 Please continue. 566 00:17:22,865 --> 00:17:24,835 Daniel: And that's exactly the point 567 00:17:24,884 --> 00:17:28,705 here, is In S3, if you don't have access 568 00:17:28,705 --> 00:17:30,545 to the key, it actually works as an 569 00:17:30,555 --> 00:17:33,855 access control mechanism rather than you 570 00:17:33,875 --> 00:17:35,505 getting back a bunch of garbage data. 571 00:17:35,715 --> 00:17:37,074 So if you don't have access to the 572 00:17:37,074 --> 00:17:38,685 key, you just don't get the object. 573 00:17:38,855 --> 00:17:40,314 The only way you get the object is 574 00:17:40,315 --> 00:17:41,685 if you have access to the key, in 575 00:17:41,685 --> 00:17:43,395 which case you get it in plain text. 576 00:17:43,815 --> 00:17:45,175 And so if your data gets walked, 577 00:17:45,175 --> 00:17:46,245 it gets walked in plain text. 578 00:17:46,585 --> 00:17:47,945 Corey: Do you believe that there is a 579 00:17:47,945 --> 00:17:50,535 risk of AWS using its privileged position 580 00:17:50,575 --> 00:17:52,825 to scan the contents of S3 buckets? 581 00:17:53,115 --> 00:17:54,265 I know that people love to have 582 00:17:54,265 --> 00:17:55,325 conspiracy theories around this 583 00:17:55,335 --> 00:17:56,684 all the time, that they're looking 584 00:17:56,685 --> 00:17:58,655 through all the data you put in S3. 585 00:17:58,950 --> 00:18:00,610 I have a rough idea, at least, of 586 00:18:00,610 --> 00:18:02,400 what general order of magnitude 587 00:18:02,400 --> 00:18:03,630 of compute power it would take to 588 00:18:03,630 --> 00:18:06,260 actually do that, and I have some sub 589 00:18:06,260 --> 00:18:08,460 questions about your conspiracy theory. 590 00:18:08,729 --> 00:18:09,859 But again, you focus on this 591 00:18:09,859 --> 00:18:10,870 stuff a lot more than I do. 592 00:18:10,870 --> 00:18:11,470 What are your thoughts? 593 00:18:11,720 --> 00:18:13,439 Daniel: Yeah, I don't know how it works 594 00:18:13,439 --> 00:18:15,429 underneath, and I don't see inside AWS, 595 00:18:15,429 --> 00:18:17,475 but I, that's not a Theory I subscribe to. 596 00:18:17,475 --> 00:18:18,945 It just, it doesn't make sense 597 00:18:18,955 --> 00:18:20,055 from a business perspective. 598 00:18:20,055 --> 00:18:21,165 It doesn't make sense from a, 599 00:18:21,275 --> 00:18:22,765 like a technical perspective. 600 00:18:22,815 --> 00:18:23,655 Why would you do that? 601 00:18:23,805 --> 00:18:25,175 They've got way more to lose than they 602 00:18:25,185 --> 00:18:26,625 have to gain by doing something like that. 603 00:18:26,805 --> 00:18:28,175 Corey: That's always been my perspective. 604 00:18:28,175 --> 00:18:29,624 And we know it's expensive to do 605 00:18:29,624 --> 00:18:30,765 it because look at how they priced 606 00:18:30,765 --> 00:18:32,175 Amazon Macy when it launched. 607 00:18:32,175 --> 00:18:33,915 And even after they redid the pricing, 608 00:18:34,034 --> 00:18:35,754 this is something that explicitly looks 609 00:18:35,754 --> 00:18:38,085 through your S3 buckets on your behalf, 610 00:18:38,105 --> 00:18:40,135 looking for sensitive information so 611 00:18:40,135 --> 00:18:41,695 that you can make sure that you're, you 612 00:18:41,695 --> 00:18:43,565 know, where it lives in your environment. 613 00:18:43,565 --> 00:18:43,605 Yeah. 614 00:18:43,865 --> 00:18:45,365 And it is extortionately slow, 615 00:18:45,365 --> 00:18:48,145 incredibly expensive, and not widely 616 00:18:48,145 --> 00:18:49,825 deployed for those two reasons. 617 00:18:50,075 --> 00:18:52,275 I have a really hard time imagining 618 00:18:52,275 --> 00:18:53,944 that if they had this magic thing on the 619 00:18:53,945 --> 00:18:55,125 back, on the back end that would just 620 00:18:55,134 --> 00:18:56,594 tell them what everyone had in every 621 00:18:56,594 --> 00:18:58,224 bucket, that they wouldn't find a more 622 00:18:58,224 --> 00:19:01,025 cost effective way, a more widely adopted 623 00:19:01,025 --> 00:19:03,105 way of being able to perform that task. 624 00:19:03,425 --> 00:19:04,225 I just don't see it. 625 00:19:04,365 --> 00:19:06,004 Daniel: Yeah, look, I just don't 626 00:19:06,634 --> 00:19:08,465 think, I think AWS is a good actor. 627 00:19:08,475 --> 00:19:09,154 Fundamentally. 628 00:19:09,154 --> 00:19:10,644 They're not, they're not a bad actor. 629 00:19:10,814 --> 00:19:12,824 They have so, so much complexity 630 00:19:12,824 --> 00:19:14,854 and not like over 300 services, 631 00:19:14,884 --> 00:19:17,124 like tens of thousands of API calls. 632 00:19:17,124 --> 00:19:18,725 Like at that scale, you end up 633 00:19:19,024 --> 00:19:20,995 seeing weird things happen because 634 00:19:20,995 --> 00:19:22,164 there's just so many things 635 00:19:22,165 --> 00:19:23,394 that could, that could happen. 636 00:19:23,584 --> 00:19:24,954 But fundamentally I've 637 00:19:25,054 --> 00:19:25,764 always found them to. 638 00:19:26,280 --> 00:19:27,620 Uh, be a good actor, try their 639 00:19:27,620 --> 00:19:29,260 best to do security right, 640 00:19:29,310 --> 00:19:30,359 and all of that kind of thing. 641 00:19:30,360 --> 00:19:31,629 Corey: I would agree with that sentiment. 642 00:19:31,649 --> 00:19:34,219 There are, AWS does a lot of things 643 00:19:34,229 --> 00:19:36,800 that I find questionable and weird, 644 00:19:36,939 --> 00:19:39,639 but they don't tend to touch security, 645 00:19:39,639 --> 00:19:41,639 particularly of foundational services. 646 00:19:41,649 --> 00:19:44,449 They, they mean well, and there 647 00:19:44,449 --> 00:19:45,830 are enough people I know who work 648 00:19:45,830 --> 00:19:47,199 there that I think of as canaries, 649 00:19:47,250 --> 00:19:48,949 who would resign on the spot. 650 00:19:49,100 --> 00:19:50,839 On ethical grounds, if nothing 651 00:19:50,839 --> 00:19:52,610 else, if something like that were 652 00:19:52,610 --> 00:19:54,749 to take place, then I'm comfortable, 653 00:19:54,809 --> 00:19:56,239 uh, making that assertion. 654 00:19:56,409 --> 00:19:56,939 I don't know if that's 655 00:19:56,939 --> 00:19:57,699 enough for some people. 656 00:19:57,699 --> 00:19:58,549 I mean, obviously, if you're a 657 00:19:58,579 --> 00:19:59,959 government and like, Well, Cory's 658 00:19:59,999 --> 00:20:01,299 got a good feeling about that. 659 00:20:01,309 --> 00:20:02,669 Does not check your 660 00:20:02,699 --> 00:20:04,379 audit box, nor should it. 661 00:20:04,389 --> 00:20:05,870 Let's be very clear here. 662 00:20:06,129 --> 00:20:07,239 But for my dumb Twitter for 663 00:20:07,239 --> 00:20:08,569 pets startup, yeah, that's good 664 00:20:08,569 --> 00:20:09,829 enough for me in my use case. 665 00:20:10,005 --> 00:20:10,679 Daniel: Yeah. 666 00:20:10,679 --> 00:20:12,304 How would you even check that 667 00:20:12,304 --> 00:20:13,215 assumption if you want them to? 668 00:20:13,564 --> 00:20:14,114 Corey: Exactly. 669 00:20:14,124 --> 00:20:15,425 The way that they've done it before 670 00:20:15,425 --> 00:20:16,554 is, oh, well, we have all these third 671 00:20:16,554 --> 00:20:18,324 party audits that validate the things 672 00:20:18,324 --> 00:20:21,054 that we've said are correct, etc, etc. 673 00:20:21,385 --> 00:20:22,724 I know that there are people 674 00:20:22,724 --> 00:20:24,655 that I've spoken to that I trust. 675 00:20:24,915 --> 00:20:26,284 These are phenomenal technologists, 676 00:20:26,294 --> 00:20:28,364 and they are Supremely confident 677 00:20:28,365 --> 00:20:30,414 that it functions as described. 678 00:20:30,544 --> 00:20:31,955 But I've always viewed, on some 679 00:20:31,955 --> 00:20:33,735 level, you have to trust your cloud 680 00:20:33,735 --> 00:20:35,254 provider or your data should not 681 00:20:35,404 --> 00:20:36,924 live within that cloud provider. 682 00:20:37,154 --> 00:20:38,594 Because there's nothing out there 683 00:20:38,594 --> 00:20:40,534 that says, Oh, when it's Corey's 684 00:20:40,544 --> 00:20:42,334 specific requests, we're going to 685 00:20:42,334 --> 00:20:44,754 send him to an identically performing 686 00:20:44,764 --> 00:20:46,884 set of API endpoints that just 687 00:20:46,955 --> 00:20:48,684 don't do all that pesky encryption 688 00:20:48,695 --> 00:20:49,965 stuff under the hood, and we can 689 00:20:49,965 --> 00:20:52,104 inspect every aspect of what he does. 690 00:20:52,364 --> 00:20:53,734 I don't think that they're doing that. 691 00:20:53,950 --> 00:20:55,790 But there's nothing to my understanding 692 00:20:55,850 --> 00:20:57,340 that would prevent them, on a 693 00:20:57,360 --> 00:20:58,740 technical basis, from doing so. 694 00:20:58,899 --> 00:20:59,850 Daniel: Yeah, look, and you 695 00:20:59,850 --> 00:21:01,169 touch on an interesting point. 696 00:21:01,310 --> 00:21:02,429 There was a great post by 697 00:21:02,450 --> 00:21:04,520 Nick Frechette, uh, recently. 698 00:21:04,760 --> 00:21:07,159 He's been digging into, uh, sort of 699 00:21:07,379 --> 00:21:09,720 non production endpoints, things that, 700 00:21:09,800 --> 00:21:12,160 like, you don't expect to be there, uh, 701 00:21:12,160 --> 00:21:13,890 and where you can send production data. 702 00:21:13,960 --> 00:21:15,150 And so, I would encourage 703 00:21:15,150 --> 00:21:16,470 everyone to read that blog post. 704 00:21:16,650 --> 00:21:18,380 I think, again, that's, that's a case 705 00:21:18,380 --> 00:21:20,530 of Hey, there's so much complexity that 706 00:21:20,810 --> 00:21:22,780 these non production endpoints have 707 00:21:23,220 --> 00:21:25,410 snuck in to the sort of the production 708 00:21:25,410 --> 00:21:27,560 landscape and can very occasionally 709 00:21:27,570 --> 00:21:30,480 be used with production data, but AWS 710 00:21:30,490 --> 00:21:31,889 will fix all of that stuff once they 711 00:21:31,889 --> 00:21:33,269 find out about this kind of thing. 712 00:21:33,480 --> 00:21:34,289 Corey: That's generally the 713 00:21:34,289 --> 00:21:35,240 response that I've gotten. 714 00:21:35,599 --> 00:21:37,780 Since this article, as you say, doesn't 715 00:21:37,810 --> 00:21:39,787 include anything groundbreaking, new 716 00:21:39,787 --> 00:21:42,165 revelations around S3, Have you gotten 717 00:21:42,165 --> 00:21:44,155 any feedback from the S3 team about, 718 00:21:44,205 --> 00:21:46,705 oh, hey, we didn't realize this, or, 719 00:21:46,855 --> 00:21:48,325 or you misunderstood something, which 720 00:21:48,325 --> 00:21:50,164 I, I get a fair bit when I wind up 721 00:21:50,164 --> 00:21:51,805 writing deep technical dives, because 722 00:21:51,975 --> 00:21:53,755 this stuff is complicated and no one has 723 00:21:53,755 --> 00:21:55,165 all the pieces in their head at once. 724 00:21:55,574 --> 00:21:56,544 Have you gotten feedback at all 725 00:21:56,545 --> 00:21:57,675 from them on this, or is this 726 00:21:57,675 --> 00:21:58,925 just one of those things where 727 00:21:59,005 --> 00:22:00,445 we let our work speak for itself? 728 00:22:00,595 --> 00:22:01,795 Daniel: No, look, I always send 729 00:22:01,795 --> 00:22:03,465 them my stuff just to make sure 730 00:22:03,465 --> 00:22:05,475 they, like, I'm, I'm often wrong. 731 00:22:05,705 --> 00:22:08,375 And I, it's 5am and I just got some, I 732 00:22:08,375 --> 00:22:10,455 got an email this morning, And I know 733 00:22:10,455 --> 00:22:12,615 the big, the big thing in there was they 734 00:22:12,615 --> 00:22:15,004 wanted to make sure that people understood 735 00:22:15,215 --> 00:22:17,014 that these weren't vulnerabilities. 736 00:22:17,254 --> 00:22:18,704 And the vast majority of these 737 00:22:18,705 --> 00:22:20,175 things could be protected against 738 00:22:20,205 --> 00:22:21,524 with native configuration. 739 00:22:21,705 --> 00:22:23,555 And I 100 percent agree with them. 740 00:22:23,575 --> 00:22:24,925 These are not vulnerabilities. 741 00:22:24,925 --> 00:22:26,264 And that's why I specifically in the 742 00:22:26,264 --> 00:22:28,245 blog post, I said, I call them quirks 743 00:22:28,245 --> 00:22:29,975 and oddities and just things you 744 00:22:29,975 --> 00:22:32,334 need to know rather than, hey, these 745 00:22:32,334 --> 00:22:33,844 are things that AWS needs to fix. 746 00:22:34,064 --> 00:22:34,874 Corey: Yeah, it's a well written 747 00:22:34,874 --> 00:22:36,884 post and it's very engaging. 748 00:22:37,064 --> 00:22:39,074 But at no point from the point where I 749 00:22:39,074 --> 00:22:40,814 first saw it come across my desk to now, 750 00:22:40,955 --> 00:22:43,004 did I, was I ever under the misconception 751 00:22:43,015 --> 00:22:44,654 that, oh, this is a vulnerability. 752 00:22:44,975 --> 00:22:46,535 This is, I see things that could 753 00:22:46,545 --> 00:22:47,895 lead themselves to customer 754 00:22:47,895 --> 00:22:49,785 side vulnerabilities if the 755 00:22:49,795 --> 00:22:51,905 customer had a misunderstanding 756 00:22:51,965 --> 00:22:53,765 about how something functioned. 757 00:22:54,075 --> 00:22:56,565 Now that is not to say that AWS has 758 00:22:56,565 --> 00:22:58,795 a vulnerability on their side, but 759 00:22:58,795 --> 00:23:01,314 technically the all authenticated users 760 00:23:01,424 --> 00:23:03,664 was not a vulnerability on their side, 761 00:23:03,664 --> 00:23:06,345 but it led to thousands upon thousands 762 00:23:06,355 --> 00:23:08,375 of customer side vulnerabilities 763 00:23:08,575 --> 00:23:10,425 because misconfiguration is one of 764 00:23:10,425 --> 00:23:12,344 the biggest threat vectors in cloud. 765 00:23:12,535 --> 00:23:13,095 Daniel: Exactly. 766 00:23:13,215 --> 00:23:15,345 The way I think about it is, if you've 767 00:23:15,345 --> 00:23:18,025 got an expectation or a mental model 768 00:23:18,095 --> 00:23:20,025 that's one way because of the way that 769 00:23:20,065 --> 00:23:21,555 things are worded or because of your 770 00:23:21,555 --> 00:23:24,875 experiences, often the complexity of AWS 771 00:23:25,145 --> 00:23:27,815 will result in a slight deviation from 772 00:23:27,815 --> 00:23:30,074 that mental model or that expectation. 773 00:23:30,650 --> 00:23:32,570 And so you'll end up making a mistake 774 00:23:32,610 --> 00:23:34,160 that perhaps you otherwise wouldn't. 775 00:23:34,380 --> 00:23:36,230 Um, one of the examples I give in the blog 776 00:23:36,230 --> 00:23:39,470 post about that is object keys inside S3. 777 00:23:39,650 --> 00:23:42,659 Now, object keys look very much 778 00:23:42,660 --> 00:23:44,480 like file names on your file system, 779 00:23:44,650 --> 00:23:45,859 but they're specifically called 780 00:23:45,889 --> 00:23:47,460 keys because they're not files. 781 00:23:47,630 --> 00:23:48,860 But most people will assume they 782 00:23:48,860 --> 00:23:50,189 will function like file names. 783 00:23:50,190 --> 00:23:52,590 And it turns out, that because they're 784 00:23:52,590 --> 00:23:54,470 keys and not file names, you can put any 785 00:23:54,470 --> 00:23:56,110 sort of characters that you want in them. 786 00:23:56,379 --> 00:23:58,470 Slash, slashes, hashes, 787 00:23:58,540 --> 00:23:59,930 percentage signs, etc. 788 00:24:00,089 --> 00:24:02,129 And that's completely fine and okay 789 00:24:02,129 --> 00:24:03,730 to do and very well documented. 790 00:24:03,910 --> 00:24:06,039 But if your application treats 791 00:24:06,039 --> 00:24:08,239 them as if they were file names, it 792 00:24:08,239 --> 00:24:10,385 might end up being Misfunctioning or 793 00:24:10,405 --> 00:24:12,034 introducing a vulnerability itself 794 00:24:12,064 --> 00:24:13,635 because it thinks it's a farming. 795 00:24:13,814 --> 00:24:14,245 Corey: Exactly. 796 00:24:14,274 --> 00:24:16,985 It's the, it's the interpretation of a 797 00:24:17,054 --> 00:24:21,514 very complicated, very explicit set of 798 00:24:21,524 --> 00:24:24,294 things that are designed from the ground 799 00:24:24,304 --> 00:24:26,344 up as very base level service primitives. 800 00:24:26,515 --> 00:24:28,595 That in turn composed together 801 00:24:28,595 --> 00:24:30,085 into something truly incredible. 802 00:24:30,195 --> 00:24:31,855 It's an, a lot of those incredible 803 00:24:31,865 --> 00:24:32,935 things are in fact emergent 804 00:24:32,955 --> 00:24:34,335 properties as best I can tell. 805 00:24:34,355 --> 00:24:35,174 Because I don't think that there 806 00:24:35,174 --> 00:24:36,225 are any people with the perfect 807 00:24:36,225 --> 00:24:37,914 ability to predict the future hanging 808 00:24:37,915 --> 00:24:39,625 out on the S3 team 15 years ago. 809 00:24:39,874 --> 00:24:40,975 This is stuff that happens. 810 00:24:41,004 --> 00:24:43,205 And uh, Mylon Thompson Bukovec gave 811 00:24:43,244 --> 00:24:44,945 a talk at reInvent a few years ago. 812 00:24:45,175 --> 00:24:47,955 Mentioning how after the S3 apocalyptic 813 00:24:47,955 --> 00:24:50,885 event, I think in 2017, they rebuilt 814 00:24:50,985 --> 00:24:54,575 all of S3 as 235 microservices. 815 00:24:54,845 --> 00:24:56,324 And my comment on that immediately 816 00:24:56,324 --> 00:24:58,365 was, this is important for S3. 817 00:24:58,505 --> 00:24:59,934 This is not a how to guide. 818 00:25:00,095 --> 00:25:01,574 They are not Pokemon. 819 00:25:01,615 --> 00:25:03,274 You need not collect them all. 820 00:25:03,365 --> 00:25:04,725 Your five person startup 821 00:25:04,755 --> 00:25:06,175 should absolutely not do this. 822 00:25:06,325 --> 00:25:07,875 Because, yeah, it makes perfect 823 00:25:07,895 --> 00:25:09,695 sense for them to do it the way 824 00:25:09,695 --> 00:25:11,345 that they have at their scale. 825 00:25:11,780 --> 00:25:13,490 Whatever your startup is doing, I 826 00:25:13,490 --> 00:25:15,930 promise it is not S3 levels of scale 827 00:25:15,970 --> 00:25:17,720 and won't be for many, many, many 828 00:25:17,720 --> 00:25:19,060 years and at least seven rewrites. 829 00:25:19,320 --> 00:25:20,310 So you're fine. 830 00:25:20,460 --> 00:25:22,220 Don't, don't view it as an instructional 831 00:25:22,220 --> 00:25:23,850 guide, but that, that glimpse under the 832 00:25:23,850 --> 00:25:25,230 hood that they were able to completely 833 00:25:25,230 --> 00:25:28,589 rewrite all of S3 and customers never knew 834 00:25:28,620 --> 00:25:31,630 because it still supports the same APIs 835 00:25:31,670 --> 00:25:34,610 in bug for bug level of reproducibility 836 00:25:34,860 --> 00:25:35,820 is nothing short of amazing. 837 00:25:35,980 --> 00:25:36,190 Daniel: Yeah. 838 00:25:36,230 --> 00:25:37,220 And that's the beauty of it. 839 00:25:37,260 --> 00:25:38,810 It's there's all that complexity 840 00:25:38,949 --> 00:25:40,479 and it's just really simple to use. 841 00:25:40,510 --> 00:25:40,750 You just. 842 00:25:41,210 --> 00:25:42,310 Send your files there and then 843 00:25:42,310 --> 00:25:43,570 they live there forever unless 844 00:25:43,570 --> 00:25:45,010 you ask them to be deleted. 845 00:25:45,390 --> 00:25:46,060 It's beautiful, I love it. 846 00:25:46,210 --> 00:25:47,250 Corey: Yeah, one of the scarier things 847 00:25:47,250 --> 00:25:48,650 for enterprise is, okay, when I say 848 00:25:48,650 --> 00:25:50,190 delete and you say it's deleted, how 849 00:25:50,220 --> 00:25:52,799 deleted is it at that specific moment? 850 00:25:52,989 --> 00:25:54,840 People always wonder about that one 851 00:25:54,840 --> 00:25:56,560 and they're not wrong to have that 852 00:25:56,590 --> 00:25:58,789 question in their minds because yeah, 853 00:25:58,790 --> 00:26:00,770 there are legal terms, there are legal 854 00:26:00,770 --> 00:26:02,270 definitions in contracts around what 855 00:26:02,270 --> 00:26:04,600 exactly deleted means and Let's not 856 00:26:04,740 --> 00:26:06,800 blunder our way into inadvertently making 857 00:26:06,800 --> 00:26:08,590 representations to our own customers 858 00:26:08,590 --> 00:26:10,440 that turns out aren't strictly true. 859 00:26:10,580 --> 00:26:12,240 Daniel: Yeah, but it's again, it's one 860 00:26:12,240 --> 00:26:14,649 of those things like, doesn't matter if 861 00:26:14,650 --> 00:26:17,219 your object is deleted right now or a 862 00:26:17,230 --> 00:26:19,670 minute from now, uh, across the internet. 863 00:26:20,990 --> 00:26:22,170 Companies and most customers, 864 00:26:22,410 --> 00:26:24,240 there really is no difference. 865 00:26:24,400 --> 00:26:27,080 One of the fun things I found was, uh, 866 00:26:27,120 --> 00:26:28,920 the, uh, this idea of compliance locks. 867 00:26:29,120 --> 00:26:31,130 So if you're a legal team and let's say 868 00:26:31,130 --> 00:26:33,059 someone sent you a subpoena or something 869 00:26:33,059 --> 00:26:35,080 like that and said, Hey, do not delete 870 00:26:35,110 --> 00:26:36,990 these things under any circumstances. 871 00:26:37,140 --> 00:26:39,590 You're able to implement that in an S3 872 00:26:39,590 --> 00:26:41,580 bucket and say, Hey, this, this object 873 00:26:41,730 --> 00:26:43,660 cannot be deleted under any circumstances. 874 00:26:43,660 --> 00:26:45,560 And in fact, the only way that it can be 875 00:26:45,560 --> 00:26:47,695 deleted once, once it's The bucket has, 876 00:26:47,775 --> 00:26:49,505 has that compliance mode enabled and the 877 00:26:49,515 --> 00:26:51,725 object has been set to a compliance lock 878 00:26:51,905 --> 00:26:55,045 is to delete the entire AWS account. 879 00:26:55,185 --> 00:26:56,134 Corey: Exactly. 880 00:26:56,134 --> 00:26:57,585 I've always wondered if there was an 881 00:26:57,585 --> 00:27:00,235 end run around this because, okay, 882 00:27:00,275 --> 00:27:01,785 I break into someone's account. 883 00:27:01,874 --> 00:27:04,624 I now go ahead and waive the compliance 884 00:27:04,625 --> 00:27:08,205 lock, uh, the legal hold, uh, I 885 00:27:08,205 --> 00:27:09,395 forget which version of it it is. 886 00:27:09,635 --> 00:27:11,475 And I can set it for up to a century. 887 00:27:11,645 --> 00:27:13,625 Congratulations in your great 888 00:27:13,625 --> 00:27:15,435 grandchildren will despise your 889 00:27:15,435 --> 00:27:16,744 negligence because they will still have 890 00:27:16,744 --> 00:27:18,564 to pay your AWS bill unless you can 891 00:27:18,565 --> 00:27:19,895 delete everything else in the account. 892 00:27:20,215 --> 00:27:22,545 Is there a mitigation for bad actors? 893 00:27:22,555 --> 00:27:23,975 And I've never gotten a satisfactory 894 00:27:23,975 --> 00:27:24,915 response on that question. 895 00:27:25,124 --> 00:27:26,105 Daniel: So, so two things. 896 00:27:26,105 --> 00:27:29,075 Well, I don't know the answer to that. 897 00:27:29,205 --> 00:27:32,150 I found in the past, for example, When 898 00:27:32,210 --> 00:27:34,700 I've made a KMS key policy that meant 899 00:27:34,700 --> 00:27:36,190 that I could never delete the key. 900 00:27:36,340 --> 00:27:37,290 If I got on an enterprise 901 00:27:37,320 --> 00:27:39,060 support plan, AWS would find a 902 00:27:39,060 --> 00:27:40,850 way around it and help me do it. 903 00:27:41,200 --> 00:27:42,750 So I think it's possible that if you made 904 00:27:42,750 --> 00:27:45,150 a cataclysmic mistake, it would help you. 905 00:27:45,580 --> 00:27:46,600 Delete the objects. 906 00:27:46,770 --> 00:27:49,090 However, the way that object uploads 907 00:27:49,110 --> 00:27:51,270 work means that if the bucket has 908 00:27:51,270 --> 00:27:53,650 compliance mode enabled, the uploader 909 00:27:53,680 --> 00:27:57,220 actually gets to set whether the object 910 00:27:57,260 --> 00:27:59,430 has compliance locking enabled or 911 00:27:59,440 --> 00:28:02,060 not, which again is just a Different 912 00:28:02,060 --> 00:28:03,740 work about how the service works. 913 00:28:03,950 --> 00:28:05,220 Corey: I wish to hell that you 914 00:28:05,220 --> 00:28:06,810 could do that for storage classes. 915 00:28:06,820 --> 00:28:09,420 Everything in this bucket or prefix is 916 00:28:09,420 --> 00:28:11,520 going to be, uh, intelligent tiering, 917 00:28:11,690 --> 00:28:14,630 Go, without having to teach every 918 00:28:14,670 --> 00:28:16,739 single thing that I've got, including 919 00:28:16,739 --> 00:28:19,069 some legacy desktop applications that 920 00:28:19,069 --> 00:28:21,179 are proprietary that I cannot modify. 921 00:28:21,490 --> 00:28:22,950 Yeah, make sure that you put that into 922 00:28:22,950 --> 00:28:25,180 the Intelligent Tiering Storage class. 923 00:28:25,310 --> 00:28:26,310 Instead, I have to go through with 924 00:28:26,310 --> 00:28:27,930 a lifecycle policy, which means that 925 00:28:27,930 --> 00:28:29,660 every object gets written again, 926 00:28:29,840 --> 00:28:31,630 and that counts as a chargeable 927 00:28:31,630 --> 00:28:33,290 fee, which at scale is not nothing. 928 00:28:33,609 --> 00:28:35,200 And then, and only then, it starts 929 00:28:35,229 --> 00:28:37,259 aging into that, which is frustrating. 930 00:28:37,379 --> 00:28:38,680 Daniel: But it's the way it works, and it 931 00:28:38,680 --> 00:28:41,239 matters for some customers, but not most. 932 00:28:41,449 --> 00:28:42,219 Corey: I don't think it would break 933 00:28:42,219 --> 00:28:44,575 anything to be able to say, Okay, now 934 00:28:44,575 --> 00:28:45,915 by default, yeah, everything goes into 935 00:28:45,915 --> 00:28:47,175 standard because the way it currently 936 00:28:47,175 --> 00:28:49,115 works today, but you now have an 937 00:28:49,115 --> 00:28:51,235 option where the bucket can set aside 938 00:28:51,235 --> 00:28:53,055 that anything placed into it winds 939 00:28:53,055 --> 00:28:54,534 up being put into that storage class. 940 00:28:54,705 --> 00:28:55,564 I think that would be 941 00:28:55,595 --> 00:28:56,594 a welcome enhancement. 942 00:28:56,595 --> 00:28:57,714 I don't know that would necessarily 943 00:28:57,725 --> 00:28:59,185 break anything customer side. 944 00:28:59,470 --> 00:29:00,530 It might very well break 945 00:29:00,540 --> 00:29:01,790 things on the AWS side. 946 00:29:01,790 --> 00:29:03,110 I know I've told this to them years 947 00:29:03,110 --> 00:29:04,850 ago and it's, and it just has, no 948 00:29:04,850 --> 00:29:06,290 one was listening is not the reason 949 00:29:06,290 --> 00:29:07,290 that they haven't gone ahead and 950 00:29:07,290 --> 00:29:08,340 implemented something like this. 951 00:29:08,350 --> 00:29:10,200 I'm sure it's complicated, but man, 952 00:29:10,200 --> 00:29:11,410 as a customer, wouldn't that be nice? 953 00:29:11,600 --> 00:29:12,340 Daniel: It would be indeed. 954 00:29:12,549 --> 00:29:13,819 Corey: Uh, I really want to thank 955 00:29:13,820 --> 00:29:15,450 you for taking the time to not just 956 00:29:15,629 --> 00:29:16,850 go ahead and write all this up, 957 00:29:16,859 --> 00:29:18,159 but also to speak with me about it. 958 00:29:18,420 --> 00:29:19,650 If people want to learn more, where's 959 00:29:19,650 --> 00:29:21,040 the best place for them to find you? 960 00:29:21,180 --> 00:29:22,180 Daniel: Uh, thanks by the way. 961 00:29:22,380 --> 00:29:23,310 Uh, yeah, on blog. 962 00:29:24,120 --> 00:29:24,250 daryon. 963 00:29:24,290 --> 00:29:24,640 com. 964 00:29:24,820 --> 00:29:25,780 That's where generally 965 00:29:25,820 --> 00:29:27,080 I do my shit posting. 966 00:29:27,080 --> 00:29:28,920 Luckily, my employer allows me to write 967 00:29:28,920 --> 00:29:30,919 in the style of it, but I like to. 968 00:29:30,919 --> 00:29:32,610 So, uh, it's good fun and there's 969 00:29:32,669 --> 00:29:33,790 a good bit of research on there. 970 00:29:33,990 --> 00:29:34,380 Corey: Excellent. 971 00:29:34,420 --> 00:29:35,330 And we will of course, put links 972 00:29:35,330 --> 00:29:36,660 to all of this in the show notes. 973 00:29:36,940 --> 00:29:38,290 Thank you so much for taking 974 00:29:38,290 --> 00:29:39,150 the time to speak with me. 975 00:29:39,150 --> 00:29:40,110 I really appreciate it. 976 00:29:40,280 --> 00:29:40,880 Daniel: Been a good time. 977 00:29:40,940 --> 00:29:41,400 Thanks, Corey. 978 00:29:41,550 --> 00:29:43,040 Corey: Daniel Zerlak is the Chief 979 00:29:43,040 --> 00:29:44,750 Innovation Officer at Plurium. 980 00:29:45,140 --> 00:29:46,640 I'm Cloud Economist Corey Quinn, 981 00:29:46,659 --> 00:29:48,240 and this is Screaming in the Cloud. 982 00:29:48,540 --> 00:29:50,020 If you enjoyed this podcast, 983 00:29:50,040 --> 00:29:51,649 please leave a five star review on 984 00:29:51,649 --> 00:29:53,160 your podcast platform of choice. 985 00:29:53,169 --> 00:29:54,639 Whereas if you hated this podcast, 986 00:29:54,650 --> 00:29:56,550 please leave a five star review on your 987 00:29:56,560 --> 00:29:58,490 podcast platform of choice, along with 988 00:29:58,490 --> 00:30:00,030 an angry, insulting comment that will 989 00:30:00,030 --> 00:30:02,060 live there forever because some joker 990 00:30:02,060 --> 00:30:03,720 wound up turning on compliance lock.