1 00:00:00,500 --> 00:00:03,860 This is going to be a little bit of a different show, isn't it, Chris? 2 00:00:04,200 --> 00:00:04,640 Why? 3 00:00:05,060 --> 00:00:06,260 Because we're not alone. 4 00:00:06,650 --> 00:00:07,840 There's someone else in the room. 5 00:00:08,469 --> 00:00:09,600 Do I need to call an adult? 6 00:00:10,129 --> 00:00:10,969 Are you not an adult? 7 00:00:11,489 --> 00:00:12,180 Have we met? 8 00:00:12,770 --> 00:00:13,450 No, that's fair. 9 00:00:22,240 --> 00:00:24,990 Hello, alleged human, and welcome to the Chaos Lever Podcast. 10 00:00:25,329 --> 00:00:27,440 My name is Ned, and I'm definitely not a robot. 11 00:00:27,560 --> 00:00:32,250 I'm a real human person, who needs to monitor their dihydrogen oxide intake 12 00:00:32,250 --> 00:00:36,289 and output to make sure it stays within normal tolerances, just like you. 13 00:00:36,750 --> 00:00:38,400 I definitely do not consume the blood of 14 00:00:38,400 --> 00:00:41,040 humans to stabilize my quasi organic internals. 15 00:00:41,390 --> 00:00:44,060 That would be vampirically ridiculous. 16 00:00:44,880 --> 00:00:48,330 With me is Chris, who is also Hi, Chris. 17 00:00:49,090 --> 00:00:51,259 I mean, when you say things are ridiculous, it 18 00:00:51,259 --> 00:00:53,959 always sounds like you don't mean ridiculous. 19 00:00:54,380 --> 00:00:55,199 Maybe. 20 00:00:56,660 --> 00:01:00,670 As I alluded to, we also have a guest joining us, which is very exciting. 21 00:01:00,690 --> 00:01:03,250 Our guest is Doug Midori. 22 00:01:03,615 --> 00:01:05,035 Is that how you say your last name, Doug? 23 00:01:05,345 --> 00:01:05,565 Yup. 24 00:01:05,785 --> 00:01:07,225 Man, I got it on the first try. 25 00:01:07,225 --> 00:01:07,825 That's awesome. 26 00:01:08,265 --> 00:01:12,914 Doug is the Director of Internet Analysis at Kentik, and he has definitely 27 00:01:12,914 --> 00:01:17,314 forgotten more than I have ever learned about BGP, routing, and the internet. 28 00:01:17,515 --> 00:01:18,614 Welcome to the show, Doug. 29 00:01:18,965 --> 00:01:19,584 Hey, glad to be here. 30 00:01:19,584 --> 00:01:20,105 Thanks for having me. 31 00:01:20,605 --> 00:01:21,345 Absolutely. 32 00:01:22,024 --> 00:01:26,154 A few weeks ago, we talked about BGP a bit, and we established 33 00:01:26,164 --> 00:01:28,714 that, well, if you haven't listened to it, probably go and listen 34 00:01:28,714 --> 00:01:32,074 to it, especially if you're not familiar with BGP at all, but it's a 35 00:01:32,095 --> 00:01:36,654 protocol that was initially designed when the internet was a small, 36 00:01:36,675 --> 00:01:41,104 friendly place full of nerds who just wanted to get things connected. 37 00:01:41,445 --> 00:01:44,914 Everyone kinda knew everyone, and asking others not to 38 00:01:44,914 --> 00:01:47,835 be assholes about it just kind of worked at the time. 39 00:01:48,264 --> 00:01:50,805 And that's true of so much of the early internet. 40 00:01:51,250 --> 00:01:56,299 SMTP, HTTP, FTP, and other protocols didn't 41 00:01:56,310 --> 00:01:58,310 have security even as an afterthought. 42 00:01:58,580 --> 00:02:02,610 Essentially, everything was sent in plain text, and trust was just assumed. 43 00:02:03,050 --> 00:02:05,230 And that was the world of the mid 1990s. 44 00:02:06,089 --> 00:02:08,020 30 years later, the internet is a very different 45 00:02:08,020 --> 00:02:11,940 place, and somehow, BGP is Largely the same? 46 00:02:12,329 --> 00:02:14,130 So, let's talk about that. 47 00:02:14,350 --> 00:02:18,480 Doug, so BGP was originally based on building relationships 48 00:02:18,480 --> 00:02:22,100 between neighbors, exchanging network layer reachability 49 00:02:22,110 --> 00:02:25,050 information, and it all had this assumption of trust. 50 00:02:25,500 --> 00:02:27,909 When did that start to become a problem? 51 00:02:29,159 --> 00:02:33,795 So, I mean, it's still Inherently is the same as far as that goes. 52 00:02:34,065 --> 00:02:36,495 I started in this space in the year 2009. 53 00:02:36,515 --> 00:02:38,705 So I've been doing this for about 15 years. 54 00:02:39,135 --> 00:02:43,724 Um, in that time, there's been any number of cases of either like 55 00:02:43,725 --> 00:02:48,904 deliberate routing hijacks in order to disrupt or misdirect traffic or. 56 00:02:49,225 --> 00:02:50,915 More frequently, mistakes. 57 00:02:50,965 --> 00:02:56,525 There's a lot of people will fat finger on a router causing a large internet 58 00:02:56,535 --> 00:03:01,064 outage and a couple of the problems that we deal with in the BGP world. 59 00:03:01,785 --> 00:03:05,745 Okay, so you've sort of, there's two different, I guess, things that 60 00:03:05,745 --> 00:03:08,025 people have to be worried about that you, you brought out there. 61 00:03:08,025 --> 00:03:11,525 One is like malicious activity, trying to mess with 62 00:03:11,665 --> 00:03:14,035 neighbors and, and redirect traffic in some way. 63 00:03:14,325 --> 00:03:17,415 And others just like, man, I had a, it was, it was Friday. 64 00:03:17,464 --> 00:03:18,394 I was trying to get out. 65 00:03:18,495 --> 00:03:22,275 I hit a command and uh oh, I've blown up half the internet. 66 00:03:22,900 --> 00:03:24,340 Yeah, I mean, that was pretty frequent. 67 00:03:24,430 --> 00:03:26,410 I, I would say it was too, a little too frequent 68 00:03:26,470 --> 00:03:28,510 when I was, uh, getting started in this space. 69 00:03:28,510 --> 00:03:32,140 And I would say we've made a lot of progress on that category of building 70 00:03:32,140 --> 00:03:35,890 some bills and suspenders to try to improve that side of the problem. 71 00:03:36,340 --> 00:03:40,300 What we, in the space call the other side is the determined adversary. 72 00:03:40,300 --> 00:03:45,480 So someone who is very an attacker, who's very knowledgeable of what security 73 00:03:45,480 --> 00:03:48,780 mechanisms have been deployed, how they work, what are their weaknesses. 74 00:03:48,870 --> 00:03:53,030 And so the determined adversary is kind of an unsolved problem thus far on BGP.. 75 00:03:54,005 --> 00:03:57,484 Okay, so you mentioned that there are some things have 76 00:03:57,484 --> 00:04:01,065 been put into place to help with the fat fingering problem. 77 00:04:01,195 --> 00:04:02,954 What are some of those controls or ways 78 00:04:02,954 --> 00:04:05,584 that we've tried to fix that half of things? 79 00:04:06,285 --> 00:04:06,875 Yeah, sure. 80 00:04:06,925 --> 00:04:10,685 And when I talk to audiences about this topic, I like to say that, you 81 00:04:10,685 --> 00:04:15,765 know, BGP security or this, this whole topic, it's not a one thing. 82 00:04:15,774 --> 00:04:18,334 This is a constellation of problems that we have 83 00:04:18,695 --> 00:04:20,305 a variety of different things that can go wrong. 84 00:04:20,745 --> 00:04:23,104 It's going to require a few different solutions. 85 00:04:23,175 --> 00:04:26,075 And there's also a spectrum of difficulty of one end or kind of 86 00:04:26,115 --> 00:04:28,645 the bonehead errors that hopefully we could come up with some 87 00:04:28,665 --> 00:04:32,585 automated ways to prevent them from causing disruptions on up to. 88 00:04:32,865 --> 00:04:36,035 Let's turn to the adversary I mentioned a minute ago, but yeah, we 89 00:04:36,035 --> 00:04:38,625 can mention some of the things that people have networks are using. 90 00:04:39,195 --> 00:04:43,164 A lot of it has to do with how do you filter the routes that you accept? 91 00:04:43,175 --> 00:04:45,615 So in your previous episode, I'm sure you kind of went through 92 00:04:45,615 --> 00:04:49,365 this process of this route by rumor on AS will accept routes. 93 00:04:49,685 --> 00:04:53,365 From an adjacent AS to try to learn how to reach other parts of the internet. 94 00:04:53,875 --> 00:04:56,215 So you'd like to have some sort of quality 95 00:04:56,215 --> 00:04:58,065 control over the routes that you accept. 96 00:04:58,485 --> 00:05:02,484 And in that genre of mechanisms, you have the 97 00:05:02,484 --> 00:05:04,495 very coarse thing of like what we call max. 98 00:05:04,915 --> 00:05:05,905 Pref settings. 99 00:05:05,915 --> 00:05:10,225 So I, if I normally get 10 routes from you, I 100 00:05:10,225 --> 00:05:12,545 shouldn't tomorrow suddenly start getting a million. 101 00:05:12,635 --> 00:05:13,694 There's probably a problem. 102 00:05:13,775 --> 00:05:17,325 And if it does, it should maybe kill the session or take some sort of an action. 103 00:05:17,775 --> 00:05:22,035 So that was one of a number of like really simple mechanisms that we, um, 104 00:05:22,674 --> 00:05:25,904 I don't want to take credit for it, but the industry adopted early on. 105 00:05:26,494 --> 00:05:27,964 And then there's a filtering. 106 00:05:27,974 --> 00:05:30,015 There's a couple of different ISPs we use to. 107 00:05:30,470 --> 00:05:32,790 Filter the routes that they receive from their customers, 108 00:05:33,150 --> 00:05:36,670 depending on the complexity, like maybe it's a really easy case. 109 00:05:36,670 --> 00:05:38,080 They know it's just gonna be a couple of ranges. 110 00:05:38,080 --> 00:05:40,150 They can put this into the configuration. 111 00:05:40,209 --> 00:05:43,550 We should only receive this type of route from a customer. 112 00:05:43,620 --> 00:05:48,079 But as you go up the stack in the Internet hierarchy, if you're going 113 00:05:48,079 --> 00:05:52,010 to larger companies, It's going to be very hard for them to know what 114 00:05:52,010 --> 00:05:55,490 are all the possible routes that could come through another, like if 115 00:05:55,510 --> 00:05:58,689 you're a, like a tier one is accepting routes from a tier two, that tier 116 00:05:58,690 --> 00:06:01,109 two could have a lot of different customers, a lot of types of routes. 117 00:06:01,120 --> 00:06:04,159 So then we need to automate a process to build those filters. 118 00:06:04,610 --> 00:06:07,399 And so we use, uh, we call IRR, internet routing 119 00:06:07,399 --> 00:06:10,370 registries to build these, uh, to store information. 120 00:06:11,000 --> 00:06:12,680 Um, so in that we'll say. 121 00:06:12,979 --> 00:06:14,939 There's a couple different mechanisms there, but it'll 122 00:06:15,019 --> 00:06:18,049 essentially try to whitelist what are the routes that 123 00:06:18,229 --> 00:06:21,019 would be acceptable to be received from a customer. 124 00:06:21,540 --> 00:06:25,080 And when you receive something that is outside that list, then reject it. 125 00:06:25,790 --> 00:06:29,510 Problems there is that there's like 30 of those. 126 00:06:29,520 --> 00:06:32,440 There's no single truth that everybody agrees upon. 127 00:06:32,480 --> 00:06:33,330 It can vary. 128 00:06:33,419 --> 00:06:38,429 Some of them have had some security issues, like the registry data itself has 129 00:06:38,430 --> 00:06:43,509 been a target of an attacker who had found a way to put bad information in 130 00:06:43,539 --> 00:06:47,789 that would enable them to announce routes they're not supposed to announce. 131 00:06:48,300 --> 00:06:53,890 And so the IRR area is something we, we use it's widely used, but 132 00:06:53,929 --> 00:06:59,450 we're hoping Try to get past that and use things like RPKI, ROV. 133 00:06:59,610 --> 00:07:03,130 So that's kind of the technology du jour right now. 134 00:07:03,580 --> 00:07:05,380 And let me explain a little about what that is. 135 00:07:05,380 --> 00:07:09,400 So RPKI is a resource public key infrastructure. 136 00:07:10,139 --> 00:07:16,500 So this is a cryptographically secure and enforced platform that ISPs can build 137 00:07:16,500 --> 00:07:21,200 services, use services that are built off of RPKI to perform route filtering. 138 00:07:21,640 --> 00:07:24,920 And so ROV is Route Origin Validation. 139 00:07:25,580 --> 00:07:27,960 One of hopefully, hopefully there'll be more 140 00:07:27,990 --> 00:07:30,430 as the vision, but ROV is the first one. 141 00:07:30,440 --> 00:07:33,650 A lot of times in this space, people say RPKI, they're really referring to 142 00:07:33,770 --> 00:07:37,980 RPKI ROV because that's the one application that people are actually using. 143 00:07:38,469 --> 00:07:40,229 So route origin validation. 144 00:07:40,745 --> 00:07:44,915 The way this works is that an address, or we call it a resource holder, or the 145 00:07:44,915 --> 00:07:48,835 person who owns the address space would, typically they do this through their 146 00:07:49,455 --> 00:07:53,254 RIR, so I'm using a lot of acronyms here, but if you are in North America, 147 00:07:53,275 --> 00:07:57,644 your RIR is Aaron, and you would log into the Aaron portal, you would have the 148 00:07:57,674 --> 00:08:01,485 account login, if you are the owner of the address range, and through there you 149 00:08:01,485 --> 00:08:06,320 can assert, you can build a ROA, Another acronym, a route origin authorization 150 00:08:06,320 --> 00:08:11,510 to say, what is the AS that is allowed to originate this address range? 151 00:08:12,049 --> 00:08:14,869 And then there's an expiration date, a max prefix length, there's a few 152 00:08:14,869 --> 00:08:17,989 other details there, but mostly this is the, what's the correct origin. 153 00:08:18,419 --> 00:08:22,330 Now that information then gets published out to the internet and every 154 00:08:22,330 --> 00:08:26,429 entity in the world that is rejecting RPKI invalid routes will then 155 00:08:26,469 --> 00:08:29,890 use that information to determine When they receive a route, they 156 00:08:29,890 --> 00:08:34,830 would check the AS path and look at the rightmost AS in the AS path 157 00:08:34,880 --> 00:08:39,429 that would be considered the origin and see if that matches the origin 158 00:08:39,429 --> 00:08:44,990 listed in the ROA, the Route Origin Authorization that's stored in RPKI. 159 00:08:45,610 --> 00:08:50,090 So hopefully you can follow all that, but you know, the benefits there are 160 00:08:50,090 --> 00:08:53,950 you've got, it's just, all the information is cryptographically enforced. 161 00:08:53,960 --> 00:08:56,850 You can't force some bad information in the path here. 162 00:08:56,870 --> 00:08:58,779 It is one ground truth for the world. 163 00:08:58,780 --> 00:09:01,060 So we don't have that doubt or uncertainty 164 00:09:01,060 --> 00:09:03,309 of which document are we going off of. 165 00:09:03,839 --> 00:09:07,149 This was discussed for many years and there was advocates and 166 00:09:07,149 --> 00:09:10,709 debate around this, and eventually it finally took hold about. 167 00:09:11,324 --> 00:09:14,354 Four years ago or so, we started to start seeing adoption. 168 00:09:14,405 --> 00:09:18,505 And for a while, the issue was trying to deploy globally. 169 00:09:18,564 --> 00:09:22,125 A security mechanism on the internet is a really hard thing, right? 170 00:09:22,125 --> 00:09:24,134 This is, there's no money in it for anybody. 171 00:09:24,165 --> 00:09:25,584 Everyone doing this is trying to do this 172 00:09:25,585 --> 00:09:27,305 for the benefit of the rest of the internet. 173 00:09:27,844 --> 00:09:29,514 So to get people to do it. 174 00:09:29,845 --> 00:09:31,835 There's two steps that a network has to 175 00:09:31,865 --> 00:09:35,435 perform in order to have deployed RPKI ROV. 176 00:09:35,445 --> 00:09:38,245 So they would create ROAs for their address 177 00:09:38,245 --> 00:09:40,045 space to essentially communicate to the world. 178 00:09:40,045 --> 00:09:41,385 What's the correct origin. 179 00:09:41,785 --> 00:09:44,894 And then they also need to reject RPKI invalid 180 00:09:44,894 --> 00:09:47,164 routes that are coming through their network. 181 00:09:47,655 --> 00:09:51,725 But for a while, we had a chicken or the egg, where why would 182 00:09:51,725 --> 00:09:54,875 anybody bother creating ROAs because no one's rejecting invalids? 183 00:09:54,875 --> 00:09:57,514 And why would anybody reject invalids because no one's creating ROAs? 184 00:09:57,515 --> 00:09:58,755 Well, we've somehow. 185 00:09:59,194 --> 00:10:02,915 Managed to get ourselves past that chicken or the egg phase. 186 00:10:03,114 --> 00:10:06,114 Um, probably the biggest facilitator was when we had tier 187 00:10:06,114 --> 00:10:09,335 ones back in the year 2020, start rejecting invalids. 188 00:10:09,415 --> 00:10:13,384 And I'm talking about, it used to be called Telia, now it's Aurelion, Lumen, 189 00:10:13,454 --> 00:10:18,305 Cogent, GTT, like these really big global, top of the internet telecoms. 190 00:10:18,354 --> 00:10:19,224 They have huge. 191 00:10:19,420 --> 00:10:23,120 Downstream customer cones and cast a wide shadow. 192 00:10:23,120 --> 00:10:25,510 And so when they do something, it has broad effect. 193 00:10:25,600 --> 00:10:30,389 So when in that year around, it's about a year that those networks started 194 00:10:30,390 --> 00:10:34,159 rejecting invalids, and that really started the ball rolling, in my opinion, 195 00:10:34,660 --> 00:10:37,929 and we can see that if you track these things through time, you can see 196 00:10:37,929 --> 00:10:40,980 there's an inflection point around that time where people start creating 197 00:10:41,390 --> 00:10:44,090 ROAs, because I feel like someone's going to actually do something about this. 198 00:10:44,150 --> 00:10:52,050 And on up to just in May this year, we crossed the, um, arbitrary milestone of 199 00:10:52,290 --> 00:10:58,939 getting past 50 percent of the routes in the global IPv4 table now have ROAs 200 00:10:59,159 --> 00:11:03,790 and are essentially eligible for the protection that would RPKI ROV would offer. 201 00:11:04,160 --> 00:11:06,260 IPv6 reached that milestone. 202 00:11:06,734 --> 00:11:11,425 Last year, probably due to the fact that it has less legacy stuff to deal with. 203 00:11:11,444 --> 00:11:14,464 But anyway, so we, we look at this as success 204 00:11:14,545 --> 00:11:16,584 and getting ourselves on the right path. 205 00:11:16,914 --> 00:11:18,794 However, I want to be careful. 206 00:11:18,834 --> 00:11:22,144 The people who are working on this topic, try to choose 207 00:11:22,145 --> 00:11:24,954 our words carefully, not to overstate what RPKI ROV 208 00:11:24,954 --> 00:11:28,275 is going to do for anyone, because it can be defeated. 209 00:11:28,820 --> 00:11:31,880 What it's most successful at is suppressing 210 00:11:32,000 --> 00:11:35,760 routes that are, uh, due to misoriginations. 211 00:11:35,770 --> 00:11:38,830 So someone has incorrectly, whether deliberately or not, originated 212 00:11:38,830 --> 00:11:42,130 address space they're not supposed to, then essentially the system 213 00:11:42,160 --> 00:11:44,749 will just suppress those routes and it'll reject those invalids. 214 00:11:45,860 --> 00:11:48,740 Like I said, it isn't foolproof and we're going to need more 215 00:11:48,770 --> 00:11:52,710 mechanisms to try to go up the spectrum and push the needle 216 00:11:52,710 --> 00:11:55,950 on up to the determined adversary to try to secure that side. 217 00:11:55,950 --> 00:11:58,869 But we had to start somewhere in the space and it's, 218 00:11:58,920 --> 00:12:00,760 as you can imagine, the internet is a big place. 219 00:12:01,090 --> 00:12:03,289 There's a lot of different companies and people working here. 220 00:12:03,289 --> 00:12:06,830 So to pull off a voluntary adoption of a 221 00:12:06,920 --> 00:12:09,140 global technology is a non trivial thing. 222 00:12:09,200 --> 00:12:11,520 So I'll stop there and see if you have any questions. 223 00:12:12,560 --> 00:12:15,920 So one of the biggest things I think if anybody knows anything 224 00:12:15,950 --> 00:12:19,960 about problems with BGP, the things that make the news are like when 225 00:12:20,020 --> 00:12:24,229 countries make a mistake and accidentally transfer 75 percent of the 226 00:12:24,229 --> 00:12:28,339 internet's traffic through Pakistan or through China or accidentally 227 00:12:28,340 --> 00:12:32,500 knock YouTube offline for 25 minutes due to routes that become 228 00:12:32,500 --> 00:12:36,620 completely invalid and are Transmit it out to the entire world. 229 00:12:37,300 --> 00:12:40,680 So these are not necessarily adversarial attacks, right? 230 00:12:40,740 --> 00:12:46,679 But these are people or people that have rights to publish global tables, right? 231 00:12:47,400 --> 00:12:50,849 Does any of the stuff that you've talked about interact there in 232 00:12:51,090 --> 00:12:54,899 any way to help or mitigate with problems that could be propagated 233 00:12:54,899 --> 00:12:59,399 worldwide from what would be considered a valid tier one endpoint? 234 00:13:00,520 --> 00:13:04,740 Yeah, so you brought up the Pakistan YouTube incident, which is probably 235 00:13:04,750 --> 00:13:08,360 maybe one of the more famous BGP incidents that have ever occurred. 236 00:13:08,700 --> 00:13:11,680 It's worth reflecting, it's probably worth a topic on a blog post of 237 00:13:11,680 --> 00:13:15,430 like, if the same thing were to happen today, how would that be different? 238 00:13:15,470 --> 00:13:18,609 Because it would really, it's really not possible what took place. 239 00:13:18,609 --> 00:13:22,490 So just the backstory here was, this occurred I believe 2008. 240 00:13:23,375 --> 00:13:28,125 Where there was a video on YouTube that was deemed anti Islamic. 241 00:13:28,135 --> 00:13:30,194 And so the government of Pakistan gave an 242 00:13:30,205 --> 00:13:32,175 order that they needed to block YouTube. 243 00:13:32,235 --> 00:13:36,135 And so that's came down to PTCL is a state telecom of Pakistan. 244 00:13:36,135 --> 00:13:38,084 They decided they would do this via BGP. 245 00:13:38,145 --> 00:13:41,464 And so they would just attract all of the BGP. 246 00:13:41,494 --> 00:13:45,314 They create a route of YouTube address space, try to attract all the traffic 247 00:13:45,314 --> 00:13:48,765 that was going to YouTube and put it in the bit bucket when it comes. 248 00:13:49,144 --> 00:13:51,114 So that part, it was intentional. 249 00:13:51,164 --> 00:13:53,634 What wasn't intentional was that they announced this. 250 00:13:53,869 --> 00:13:56,319 Accidentally out to one of their international transit 251 00:13:56,319 --> 00:13:58,769 providers who carried it out to the global internet. 252 00:13:59,249 --> 00:14:03,749 And so then because it was a more specific, like a BGP prefers 253 00:14:03,829 --> 00:14:07,870 more specific, longer length routes became very popular. 254 00:14:07,870 --> 00:14:09,989 And about two thirds of the internet for, I don't know, a couple of 255 00:14:09,989 --> 00:14:14,870 hours was believing that they need to go to PTCL in Pakistan for YouTube. 256 00:14:15,390 --> 00:14:17,240 That made YouTube unreachable. 257 00:14:17,310 --> 00:14:19,219 Pakistan also wasn't doing great either. 258 00:14:19,219 --> 00:14:21,489 This was, they've not used to getting that kind of traffic. 259 00:14:22,155 --> 00:14:22,425 Yeah. 260 00:14:22,425 --> 00:14:24,215 So ultimately that was, that was resolved. 261 00:14:24,255 --> 00:14:26,535 I think Google had recently purchased YouTube. 262 00:14:26,555 --> 00:14:29,245 They intervened, got the international carrier to stop carrying the route. 263 00:14:29,795 --> 00:14:33,075 But if we looked at that case again today, I call these 264 00:14:33,175 --> 00:14:35,345 accidental, but also intentional or intentional, but also 265 00:14:35,345 --> 00:14:39,004 accidental because, um, we had this, uh, happen recently. 266 00:14:39,054 --> 00:14:42,465 There was a couple of cases that are worth thinking about where a 267 00:14:42,465 --> 00:14:46,465 similar case in, uh, in Myanmar and let's see, try to get my dates right. 268 00:14:46,465 --> 00:14:48,665 I guess it was a spring of 2021. 269 00:14:48,695 --> 00:14:51,435 There was a military coup in Myanmar and it was a lot of. 270 00:14:52,030 --> 00:14:55,010 Government involvement in suppressing communications. 271 00:14:55,020 --> 00:14:56,550 So there was like a total shutdown. 272 00:14:56,550 --> 00:14:59,250 There were mobile, like nightly shutdowns. 273 00:14:59,250 --> 00:15:00,260 There was all kinds of different things. 274 00:15:00,260 --> 00:15:02,690 Everything, everything we'd ever seen in the digital 275 00:15:02,690 --> 00:15:04,930 rights space was happening over a couple of months. 276 00:15:05,200 --> 00:15:08,800 They even had their own Pakistan YouTube incident where they 277 00:15:08,809 --> 00:15:12,060 had given an order out to the ISPs to block social media. 278 00:15:12,099 --> 00:15:16,394 One of the ISPs in Myanmar decided they would do exactly what. 279 00:15:16,835 --> 00:15:20,595 PTCL had done in 2008, and they would take Twitter address space, 280 00:15:20,625 --> 00:15:23,615 they would basically announce it locally, it was I think their plan, 281 00:15:23,915 --> 00:15:27,645 and just drop that traffic when it comes to announce this out to the 282 00:15:27,645 --> 00:15:31,215 internet, and so around South Asia, there was a Twitter outage where 283 00:15:31,265 --> 00:15:35,795 all this traffic was getting directed to Myanmar, and then fast forward 284 00:15:35,795 --> 00:15:40,975 one year later, almost a year to the day, That incident, the exact 285 00:15:40,975 --> 00:15:44,245 same thing happened to the same address range, same prefix, everything. 286 00:15:44,535 --> 00:15:46,115 And this time it was out of Russia. 287 00:15:46,115 --> 00:15:48,145 So Russia had invaded Ukraine. 288 00:15:48,435 --> 00:15:50,745 There was a backlash in, uh, within Russia. 289 00:15:50,745 --> 00:15:53,295 They started cracking down on independent media and social media. 290 00:15:53,645 --> 00:15:58,655 And we had the same thing as an ISP to created a BGP route to block Twitter. 291 00:15:59,200 --> 00:16:02,720 Accidentally announced it out to a transit provider, but the 292 00:16:02,720 --> 00:16:07,690 difference was that between the spring of 2021 and spring of 2022, 293 00:16:08,079 --> 00:16:12,200 Twitter, now X, had created ROAs for all their address space. 294 00:16:12,510 --> 00:16:16,630 So this route now had a ROA and Routers all over the 295 00:16:16,630 --> 00:16:19,360 world would know when they see the one coming from 296 00:16:19,360 --> 00:16:22,560 Russia, this isn't the right one and they would reject it. 297 00:16:22,610 --> 00:16:25,499 Now that doesn't help the people in Russia, but 298 00:16:25,499 --> 00:16:28,469 at least contains the disruption to that area. 299 00:16:28,470 --> 00:16:29,769 And we just, we're not going to be able to get, 300 00:16:30,319 --> 00:16:31,970 we're not gonna be able to intervene beyond that. 301 00:16:32,815 --> 00:16:35,445 Anyway, so that's a good story of that growth. 302 00:16:35,445 --> 00:16:39,495 So if PDCL today works to announce those YouTube routes, they 303 00:16:39,495 --> 00:16:42,095 would probably go nowhere and they would probably affect no one. 304 00:16:42,285 --> 00:16:46,635 The other thing is that YouTube isn't pulled across trans providers anymore. 305 00:16:46,635 --> 00:16:48,945 It's served through embedded caches in your 306 00:16:48,965 --> 00:16:51,605 ISPs in nearly every country in the world. 307 00:16:51,605 --> 00:16:53,605 There's really very few exceptions to that. 308 00:16:53,945 --> 00:16:57,485 So even if the RPKI wasn't there and the route got out, I think. 309 00:16:57,645 --> 00:17:00,525 Don't know that it would even mess that much, be worthy 310 00:17:00,525 --> 00:17:02,865 of debate of how much impact it would really have. 311 00:17:03,204 --> 00:17:06,885 It certainly would be significantly less than what occurred in 2008, just to 312 00:17:07,135 --> 00:17:10,665 be a, uh, how the internet has evolved, how content is delivered, but within 313 00:17:10,665 --> 00:17:14,524 the routing space, it also kind of can't happen, or at least it'll be limited. 314 00:17:14,554 --> 00:17:18,244 I wouldn't say there's zero impact, but RPKI, 315 00:17:18,245 --> 00:17:20,105 ROV, yeah, like I said, it's good in those cases. 316 00:17:20,155 --> 00:17:21,685 This is where it's strong. 317 00:17:21,724 --> 00:17:26,245 And then, you know, in recent years, Where we've seen a lot of activity 318 00:17:26,245 --> 00:17:30,095 in the determined adversary category is against cryptocurrency services. 319 00:17:30,575 --> 00:17:32,985 So these are great targets for hackers. 320 00:17:32,994 --> 00:17:37,265 Cause if you can crack one of these places and steal the money, it's, you can 321 00:17:37,805 --> 00:17:42,304 have it immediately and launder it and you're gone and there's no recourse. 322 00:17:42,445 --> 00:17:42,905 So. 323 00:17:43,130 --> 00:17:44,730 and make for good targets. 324 00:17:44,740 --> 00:17:47,320 And so we've seen some sophisticated attacks that involve 325 00:17:47,320 --> 00:17:51,879 BGP hijacks, including one that did a hijack against Amazon. 326 00:17:51,960 --> 00:17:54,879 So this is not a mom and pop shop. 327 00:17:54,930 --> 00:17:57,349 This is one of the most well resourced networks 328 00:17:57,349 --> 00:17:59,790 in the world that did all these things. 329 00:17:59,790 --> 00:18:05,030 It defeated RPKI ROV by forging the AS path so that it would be seen as valid. 330 00:18:05,030 --> 00:18:07,480 So the route The bad route would get circulated. 331 00:18:07,480 --> 00:18:11,310 It created a fake entry into, um, ALT db, one of the, 332 00:18:11,310 --> 00:18:15,960 our IRRs that are used for automated creation of filters. 333 00:18:16,140 --> 00:18:18,960 And so there was a lot of lessons learned there, but some 334 00:18:18,960 --> 00:18:21,450 of these cryptocurrency attacks, I wrote up a, a piece, 335 00:18:21,450 --> 00:18:24,420 maybe we can put it in the, the notes for the, the episode. 336 00:18:24,899 --> 00:18:28,460 Just looking at like this is where people in our space are trying to 337 00:18:28,600 --> 00:18:32,169 spend more time thinking and so that's progress where we're less worried 338 00:18:32,169 --> 00:18:35,879 about these fat finger cases kind of covered as far as best as we can 339 00:18:35,879 --> 00:18:39,050 get them and now we need to focus on determined adversaries scenario. 340 00:18:40,514 --> 00:18:44,175 So I read through that article that you're talking about and I found 341 00:18:44,185 --> 00:18:48,024 that one example you gave of the cryptocurrency and the way that they 342 00:18:48,034 --> 00:18:53,455 had used all the infrastructure you would expect to sort of spoof 343 00:18:54,005 --> 00:18:58,845 where the routes were coming from and then make it all look legitimate 344 00:18:58,915 --> 00:19:02,405 so traffic would go there and be none the wiser and then they would 345 00:19:02,405 --> 00:19:05,985 just, you know, steal your bitcoins or whatever and abscond with them. 346 00:19:06,715 --> 00:19:11,360 Are there some new standards or things coming down the 347 00:19:11,370 --> 00:19:15,440 pike to also protect against that sort of scenario? 348 00:19:16,840 --> 00:19:18,780 Yeah, so there's always more. 349 00:19:18,830 --> 00:19:23,720 The next thing that's being pushed or advocated for is something called ASPA. 350 00:19:23,720 --> 00:19:26,019 So that doesn't deal with the cryptocurrency attack 351 00:19:26,019 --> 00:19:29,240 scenario, but autonomous system provider authorization. 352 00:19:29,530 --> 00:19:34,055 So what the way this works is Each network, which is an autonomous system, 353 00:19:34,095 --> 00:19:38,265 is going to, within the system, assert what are its transit providers. 354 00:19:38,865 --> 00:19:42,705 And in the same way that we create AROA, and this will be information that's 355 00:19:42,715 --> 00:19:47,815 stored within the RPKI platform and cryptographically delivered everywhere. 356 00:19:48,135 --> 00:19:51,755 So then by asserting what are your transit providers, this enables other 357 00:19:51,755 --> 00:19:56,055 networks to look at an AS path and detect what we call a value free violation. 358 00:19:56,280 --> 00:19:58,429 And so maybe I'll explain a little bit what that is. 359 00:19:58,790 --> 00:20:02,929 So in BGP, I don't know if you got into this in your 360 00:20:02,929 --> 00:20:06,709 last episode, but there is hierarchy to the whole thing. 361 00:20:06,719 --> 00:20:09,139 So it's not, I think maybe in a textbook, 362 00:20:09,139 --> 00:20:10,829 it might look like it's a little amorphous. 363 00:20:10,830 --> 00:20:12,060 Everybody just kind of connects to everybody 364 00:20:12,060 --> 00:20:14,120 else or, but there is a hierarchy to it. 365 00:20:14,120 --> 00:20:16,110 So for the most part, you have networks 366 00:20:16,129 --> 00:20:18,389 that are buying transit from other networks. 367 00:20:18,419 --> 00:20:21,560 So you get to the top, there's a default free zone or Transit 368 00:20:21,570 --> 00:20:24,159 free zone of the top of the internet is kind of cabal. 369 00:20:24,179 --> 00:20:28,590 There are a dozen or more ISPs that don't buy service from anybody. 370 00:20:28,590 --> 00:20:30,710 They just sell and then they connect to each other. 371 00:20:31,340 --> 00:20:35,030 And so as traffic goes across the internet, it's either, you know, crossing 372 00:20:35,030 --> 00:20:38,999 these transit edges or there's a, the alternative is a peering relationship. 373 00:20:39,000 --> 00:20:41,390 And these are the two classic types of relationships. 374 00:20:42,150 --> 00:20:43,690 What you can't do is. 375 00:20:44,220 --> 00:20:45,760 Your traffic is always going to go over a hill, so you're 376 00:20:45,760 --> 00:20:48,250 going to be going up the transit links until it gets to a 377 00:20:48,250 --> 00:20:51,090 top and then comes down the other side to the destination. 378 00:20:51,410 --> 00:20:53,940 Or maybe you've got a way to cut through it with a peering 379 00:20:53,940 --> 00:20:56,500 relationship, but you can't go down, because if you go down, 380 00:20:56,679 --> 00:20:59,709 when we draw this on diagrams, we say, you know, transit is up. 381 00:20:59,889 --> 00:21:02,039 That's how we draw it, and it's the mental model. 382 00:21:02,130 --> 00:21:05,540 May not translate in the podcast very well, but so then if you're going down, 383 00:21:05,560 --> 00:21:08,830 basically you're drawing a line, you're going from a provider to a customer. 384 00:21:09,235 --> 00:21:11,265 Back up to a provider, then that customer 385 00:21:11,365 --> 00:21:14,255 is paying both sides to send that traffic. 386 00:21:14,255 --> 00:21:15,065 It's all about money. 387 00:21:15,215 --> 00:21:18,355 It's as much about technical stuff as it is around business. 388 00:21:18,394 --> 00:21:22,044 And so when you and I have internet connections at our house, 389 00:21:22,384 --> 00:21:24,604 our phone, for the most part, they're kind of all you can eat. 390 00:21:24,605 --> 00:21:28,565 You pay some sort of flat fee, and unless you do something Crazy, you're 391 00:21:28,565 --> 00:21:33,055 hosting, um, whatever stuff, you don't worry about how much you're using. 392 00:21:33,115 --> 00:21:37,685 But in the wholesale market, it's by bit, so by volume, you pay by volume. 393 00:21:38,145 --> 00:21:40,965 And so then they're trying to either reduce, the providers 394 00:21:40,975 --> 00:21:43,705 are trying to either reduce costs or increase revenue. 395 00:21:44,275 --> 00:21:46,294 And one way they reduce costs is by peering, so 396 00:21:46,294 --> 00:21:48,315 they don't get around other transit providers. 397 00:21:48,784 --> 00:21:52,215 But, um, if you're going, you know, From a provider to a customer, back up to 398 00:21:52,215 --> 00:21:56,215 a provider, then that customer is paying both legs and is receiving no money. 399 00:21:56,215 --> 00:21:59,215 So that's a, that's something you would never want to have happen. 400 00:21:59,635 --> 00:22:03,445 And people go to great lengths to try to avoid that because you're 401 00:22:03,445 --> 00:22:05,825 basically paying twice for something you're not getting paid for. 402 00:22:05,825 --> 00:22:09,605 So that's a valley in this valley free terminology. 403 00:22:09,645 --> 00:22:13,965 And so that does happen, but it's usually a leak, like a mistake. 404 00:22:13,965 --> 00:22:18,075 So as has taken a route from one side and send it to another by mistake. 405 00:22:19,300 --> 00:22:22,230 We covered that a little bit in the last episode because I talked about 406 00:22:22,230 --> 00:22:27,580 what happened with the Allegheny provider where it was using Verizon 407 00:22:27,630 --> 00:22:32,760 and uh, DQE or something were the two providers that it was using. 408 00:22:33,340 --> 00:22:39,600 And it had accidentally leaked routes from Verizon out through DQE 409 00:22:39,600 --> 00:22:43,070 telling everybody, you know, send your traffic through me basically. 410 00:22:43,510 --> 00:22:44,389 And so that would have been a big 411 00:22:45,335 --> 00:22:47,105 Yeah, it was a great example of a valley. 412 00:22:47,135 --> 00:22:51,605 So, uh, in that case, yeah, Allegheny is a customer of both Verizon and DQE. 413 00:22:51,605 --> 00:22:54,515 The routes went from DQE to Allegheny to Verizon. 414 00:22:55,085 --> 00:22:58,875 The vision is that could be picked up and just blocked immediately. 415 00:22:58,955 --> 00:23:02,905 Had, you know, Allegheny asserted in RPI. 416 00:23:03,070 --> 00:23:06,560 Using ASPA, who are its transit providers, people would 417 00:23:06,560 --> 00:23:08,290 look at that and be like, okay, somebody made a mistake. 418 00:23:08,310 --> 00:23:09,280 I won't carry this. 419 00:23:09,830 --> 00:23:12,180 The incident has a few lessons learned. 420 00:23:12,450 --> 00:23:14,790 Obviously, Cloudflare made a lot of big stink 421 00:23:14,800 --> 00:23:17,169 about it because they got affected, rightly so. 422 00:23:17,310 --> 00:23:20,229 And it did prompt a discussion around RPKI ROV. 423 00:23:20,229 --> 00:23:24,010 But what's interesting to me is that RPKI ROV would 424 00:23:24,010 --> 00:23:27,840 have helped in that case, but not because of the origin. 425 00:23:28,005 --> 00:23:32,955 Being changed, the origins for the routes that were leaked were actually intact. 426 00:23:32,965 --> 00:23:33,645 So like CloudFlare's 1. 427 00:23:33,645 --> 00:23:34,205 3. 428 00:23:34,205 --> 00:23:34,225 3. 429 00:23:34,225 --> 00:23:34,244 3. 430 00:23:34,245 --> 00:23:36,984 5, as somebody who works with this stuff, I 431 00:23:36,984 --> 00:23:39,245 have like thousands of these ASNs memorized. 432 00:23:39,795 --> 00:23:43,475 So the, uh, the AS origin, the rightmost AS and any of those route leak paths. 433 00:23:44,045 --> 00:23:44,635 It was correct. 434 00:23:44,635 --> 00:23:48,675 So you, by checking the origin, it wouldn't have filtered the routes, 435 00:23:49,055 --> 00:23:53,415 but because the Bluewoods DQE was using a route optimizer, which 436 00:23:53,495 --> 00:23:56,425 locally creates these more specific routes to try to do traffic 437 00:23:56,425 --> 00:24:00,824 engineering, those more specific routes, then For what really attracted 438 00:24:00,824 --> 00:24:05,045 the traffic, but they also would have been RPKI invalid because 439 00:24:05,074 --> 00:24:08,935 the routes of CloudFlare and I think Akamai was in this as well. 440 00:24:09,005 --> 00:24:12,044 They had ROAs that set a maximum prefix length. 441 00:24:12,154 --> 00:24:17,655 So our prefix, we call a prefix as a BGP route as a address range 442 00:24:17,674 --> 00:24:20,945 and the address range has got, you know, a network portion and a host 443 00:24:20,955 --> 00:24:24,705 portion and then the prefix length then sets what's the network portion. 444 00:24:25,245 --> 00:24:27,965 And then in the, in our Perlansen routing, we just talked 445 00:24:28,054 --> 00:24:31,074 about prefixes, but in that case, those prefixes would have 446 00:24:31,074 --> 00:24:34,335 been invalid due to the max prefix length setting in the ROAs. 447 00:24:34,864 --> 00:24:38,335 So, I also bring that case up because that's probably the 448 00:24:38,335 --> 00:24:42,094 last really big debilitating routing leak that's occurred. 449 00:24:42,094 --> 00:24:44,675 And so that was, I think, 2019. 450 00:24:45,455 --> 00:24:47,065 We're a little more than five years out. 451 00:24:47,480 --> 00:24:51,120 Which I, when I give talks, I was like, there's not an accident that that was 452 00:24:51,150 --> 00:24:54,390 the last, I mean, we may have another one while we're speaking right now, it 453 00:24:54,390 --> 00:24:58,519 could be something disastrous could be happening, but we've gone a long time. 454 00:24:58,550 --> 00:25:00,710 Five years is a long time in internet time. 455 00:25:01,179 --> 00:25:04,339 And so things have gotten better due to a lot of these things. 456 00:25:04,339 --> 00:25:07,040 The filtering we talked about, Max prefix lang. 457 00:25:07,305 --> 00:25:10,875 ASPA really is just getting started, so I don't think we're going to 458 00:25:10,875 --> 00:25:15,764 see benefit from it for a little while, but, um, RPKI ROV is helping. 459 00:25:15,975 --> 00:25:18,424 Yeah, there's just a variety of different, we call it routing 460 00:25:18,424 --> 00:25:21,554 hygiene, just all these, all these different things that riders 461 00:25:21,554 --> 00:25:26,730 do, and there's no, there's a lot of best practices, and Manners is 462 00:25:26,730 --> 00:25:31,980 a organization that's mutually agreed norms for routing security. 463 00:25:32,450 --> 00:25:35,870 It's kind of the industry's advocacy group for enumerating what are 464 00:25:35,870 --> 00:25:39,440 the things that networks need to, steps they should take to improve 465 00:25:39,440 --> 00:25:43,890 routing security, routing hygiene, and they've been very instrumental 466 00:25:43,890 --> 00:25:48,480 in being the go to and being good advocates for routing security. 467 00:25:50,210 --> 00:25:52,740 Manners is sort of like an opt in, right? 468 00:25:52,740 --> 00:25:55,840 You want to be a good citizen, you want to have good manners. 469 00:25:56,170 --> 00:25:57,060 That's correct, yeah. 470 00:25:57,449 --> 00:26:01,949 I know recently the FCC has been trying to push being a little 471 00:26:01,949 --> 00:26:05,860 more stick and less the carrot when it comes to BGP security. 472 00:26:06,130 --> 00:26:10,170 So what is the FCC trying to do and and how is the 473 00:26:10,189 --> 00:26:13,259 industry responding to what they've been doing? 474 00:26:15,670 --> 00:26:16,170 Yeah. 475 00:26:16,170 --> 00:26:22,900 So back in 2022, the FCC, this is a federal communications commission, a 476 00:26:22,900 --> 00:26:26,830 US agency overseeing our telecommunications sector in the United States. 477 00:26:27,410 --> 00:26:30,020 Decide to get involved or get, figure out what, what 478 00:26:30,020 --> 00:26:32,629 leadership does it need to provide in routing security? 479 00:26:32,730 --> 00:26:34,940 Obviously this is something that affects the United States 480 00:26:35,010 --> 00:26:37,260 and there's a national security element to this as well. 481 00:26:38,020 --> 00:26:40,990 They began a process where they were seeking inquiry, asking 482 00:26:41,040 --> 00:26:44,440 industry experts Ask them what they think they should be doing. 483 00:26:44,560 --> 00:26:46,560 And you know, there's been a bunch of events that 484 00:26:46,560 --> 00:26:48,409 they've held and documents they've published. 485 00:26:48,750 --> 00:26:51,049 And then this year within the last couple of months, 486 00:26:51,049 --> 00:26:53,319 they published what are they're seeking comment. 487 00:26:53,460 --> 00:26:56,610 We're really trying to get illicit feedback from the industry and they're 488 00:26:56,610 --> 00:27:02,190 getting Of like what should be the rules they require of us telecoms. 489 00:27:02,230 --> 00:27:06,850 And so what they did was Identify a list of nine telecoms. 490 00:27:07,300 --> 00:27:11,775 I call them bias Oh, what's another acronym is not an acronym 491 00:27:11,775 --> 00:27:14,455 I'd seen before, but there's always, there's always a new 492 00:27:14,455 --> 00:27:19,775 one that these nine telecoms need to, uh, deploy RPKI ROV. 493 00:27:19,775 --> 00:27:23,484 So there's two things that they need to do that they need to create ROAs. 494 00:27:23,725 --> 00:27:26,874 And there's a way we can all measure and see that they provide these. 495 00:27:26,885 --> 00:27:29,784 So BIAS stands for Broadband Internet Access Service. 496 00:27:30,285 --> 00:27:35,665 And the companies are AT& T, Altus, which owns Suddenlink, Cablevision, 497 00:27:35,725 --> 00:27:40,384 Charter, which is also Spectrum, Comcast, Cox, Lumen, which, I don't know 498 00:27:40,384 --> 00:27:43,705 if everybody knows, is both a regional rub in provider and also plays 499 00:27:43,705 --> 00:27:48,764 this role in the global internet, T Mobile, our mobile operator, TDS, 500 00:27:48,795 --> 00:27:52,854 which now owns, or is in the process of owning, US Cellular, Verizon. 501 00:27:52,935 --> 00:27:53,845 Anyway, those are the nine. 502 00:27:54,350 --> 00:27:57,190 The each of these companies needs to create rows for their address space, at 503 00:27:57,190 --> 00:28:01,340 least 90 percent of the routes that they originate and also rejecting ballads. 504 00:28:01,620 --> 00:28:02,899 And so they're seeking comment on this. 505 00:28:02,899 --> 00:28:05,530 And so what I decided I would look at was, all right, well, now 506 00:28:05,530 --> 00:28:08,170 that they named these companies, let's see how they fare today. 507 00:28:08,309 --> 00:28:12,480 And I just ran through the numbers, at least on the row creation side. 508 00:28:12,705 --> 00:28:15,425 It's very tricky for us to remotely figure out to what 509 00:28:15,425 --> 00:28:17,845 extent, if at all, uh, they're rejecting invalids. 510 00:28:17,945 --> 00:28:19,655 Different people have come up with different methodologies. 511 00:28:19,665 --> 00:28:24,604 It's still a kind of open research area, but five out of the nine would do 512 00:28:24,614 --> 00:28:28,605 very well right now where they're probably already at 90 percent of the routes. 513 00:28:28,625 --> 00:28:32,295 And I, I work at a company that does deals in large amounts of net flow. 514 00:28:32,295 --> 00:28:34,264 So we work in the service provider space. 515 00:28:34,264 --> 00:28:38,595 So these are companies that we would Work with, and so we have a lot of, 516 00:28:38,625 --> 00:28:42,925 uh, NetFlow that gets shared to us as part of the service that we provide. 517 00:28:42,955 --> 00:28:45,995 And so we have a, a nice slice of the internet of just 518 00:28:45,995 --> 00:28:49,195 the traffic that's going across it for analysis and study. 519 00:28:49,204 --> 00:28:50,875 This is what I spent a lot of my time doing. 520 00:28:51,355 --> 00:28:54,625 And so I looked at the traffic that we see going to these large 521 00:28:54,635 --> 00:28:58,285 US telecoms and how much of it was going to routes with ROAs. 522 00:28:58,325 --> 00:29:02,955 And for a few of them, like T Mobile and Cox and Comcast. 523 00:29:03,285 --> 00:29:04,125 Charter Spectrum. 524 00:29:04,495 --> 00:29:05,745 It's nearly universal. 525 00:29:05,764 --> 00:29:08,955 Almost every packet going to those networks are going to 526 00:29:08,965 --> 00:29:13,325 routes with ROAs, meaning that they're eligible for protection. 527 00:29:13,395 --> 00:29:16,125 That's one side of the internet transaction. 528 00:29:16,135 --> 00:29:18,135 That's the traffic coming back to the user. 529 00:29:18,495 --> 00:29:20,335 The other side is not really covered in this 530 00:29:20,355 --> 00:29:23,285 proposal, which would be what are they going to? 531 00:29:23,285 --> 00:29:24,335 What are they sending their traffic? 532 00:29:24,585 --> 00:29:29,155 Where were they requesting data from, which could be a bank or Cryptocurrency 533 00:29:29,155 --> 00:29:33,455 service that's under attack or, uh, that side of it, they're not getting 534 00:29:33,455 --> 00:29:36,495 into, although I suspect that may not be where they focus their time. 535 00:29:36,495 --> 00:29:39,015 They're looking at what are the rules that the US telecoms need to follow? 536 00:29:39,655 --> 00:29:39,955 Yeah. 537 00:29:39,955 --> 00:29:42,765 So I went through this and yeah, it's generating a little bit 538 00:29:42,765 --> 00:29:47,035 of discussion here in our community of just, for one, whenever 539 00:29:47,035 --> 00:29:48,735 you do something, I'll just, you can't get it all right. 540 00:29:48,735 --> 00:29:51,915 I try to be explicit about what ASs I was using for the analysis. 541 00:29:52,295 --> 00:29:53,055 I'm getting some feedback. 542 00:29:53,055 --> 00:29:53,805 I missed a couple. 543 00:29:53,815 --> 00:29:54,825 I'm happy to update it. 544 00:29:54,825 --> 00:29:58,755 But some of these companies use dozens or ATT is over a hundred. 545 00:29:58,945 --> 00:30:00,105 ASs that they use. 546 00:30:00,595 --> 00:30:05,905 So I'm trying to manage the complexity of that and still provide some analysis. 547 00:30:05,905 --> 00:30:10,035 But, you know, in this proposal, the Internet Society, which was the 548 00:30:10,035 --> 00:30:13,495 former home of Manners, which you talked about earlier, which since has 549 00:30:13,495 --> 00:30:17,935 moved to the Global Cyber Alliance, those two entities wrote up a joint 550 00:30:18,770 --> 00:30:25,490 Ex parte response to the proposal by the FCC saying, pushing back pretty 551 00:30:25,490 --> 00:30:30,490 strenuously against a federal requirement that telecoms adopt RPROV. 552 00:30:30,980 --> 00:30:36,160 And the points they made were that it's dangerous in security to legislate. 553 00:30:36,190 --> 00:30:40,290 You have to use this particular solution because now everybody 554 00:30:40,290 --> 00:30:44,450 has to do that thing and maybe it's becomes obsolete or things 555 00:30:44,450 --> 00:30:46,910 have changed and now it's a counterproductive and you're stuck. 556 00:30:47,185 --> 00:30:49,415 Complying with this rule, there's a concern there 557 00:30:49,415 --> 00:30:52,455 of just ossifying some kind of a requirement. 558 00:30:52,915 --> 00:30:55,205 And then also, I guess, smaller providers 559 00:30:55,205 --> 00:30:57,015 were kind of excluded from this first pass. 560 00:30:57,035 --> 00:30:58,815 They made the argument that small providers, this would be 561 00:30:58,815 --> 00:31:02,165 a cumbersome or burdensome burdensome thing to comply with. 562 00:31:02,625 --> 00:31:05,915 But, you know, the first point they made was Using the analysis that 563 00:31:05,915 --> 00:31:09,145 myself and this other expert in the space, we've kind of collaborated 564 00:31:09,355 --> 00:31:12,545 quite a bit on this topic, this guy, Job Snyders, who's at Fastly now 565 00:31:12,945 --> 00:31:16,245 as probably the leading voice in routing security has been for awhile. 566 00:31:16,245 --> 00:31:18,415 In fact, a lot of the progress that we've 567 00:31:18,415 --> 00:31:20,735 made is directly attributable to his work. 568 00:31:21,145 --> 00:31:23,674 So he and I worked together quite a bit on this and. 569 00:31:24,095 --> 00:31:28,325 Both the FCC rules and the ex parte response, pushing back 570 00:31:28,325 --> 00:31:31,525 on them, both relied on our analysis that showed that, 571 00:31:31,935 --> 00:31:34,415 you know, we have a lot of ROAs that have been created. 572 00:31:34,445 --> 00:31:37,155 In fact, those ROAs represent the majority. 573 00:31:37,255 --> 00:31:40,295 Now it's a super majority of the traffic that's 574 00:31:40,295 --> 00:31:42,985 exchanged on the internet is going to routes with ROAs. 575 00:31:43,035 --> 00:31:45,375 And then conversely, routes that are deemed 576 00:31:45,385 --> 00:31:47,945 invalid just don't get propagated as much. 577 00:31:47,995 --> 00:31:49,795 In fact, they're suppressed quite a bit. 578 00:31:49,795 --> 00:31:53,795 So the system is working as designed and we've reached a point of. 579 00:31:54,020 --> 00:31:57,410 Adoption where the next network to do these things, to create ROAs 580 00:31:57,430 --> 00:31:59,920 for their address space, to start rejecting invalids would have 581 00:31:59,950 --> 00:32:03,270 immediate benefit because there's been so much adoption thus far. 582 00:32:03,790 --> 00:32:08,900 That work was highlighted in both the FCC document as well as the pushback. 583 00:32:08,900 --> 00:32:12,350 Yeah, their point was, look, the industry has already made 584 00:32:12,350 --> 00:32:14,050 a lot of progress and there was no government mandate. 585 00:32:14,255 --> 00:32:16,965 I mean, that's, uh, it's a pretty good argument because all 586 00:32:16,965 --> 00:32:19,775 this progress I'm describing, there's no government mandate. 587 00:32:19,775 --> 00:32:24,165 It's just simply advocacy work within the communities around the 588 00:32:24,165 --> 00:32:29,045 world and getting it to a point where there's some peer pressure, if 589 00:32:29,045 --> 00:32:32,635 not shame, to motivate networks that aren't doing the stuff to do it. 590 00:32:32,775 --> 00:32:35,195 And if, you know, things change and there's a 591 00:32:35,195 --> 00:32:37,295 better solution, this needs to be abandoned, then. 592 00:32:37,565 --> 00:32:38,235 So be it. 593 00:32:38,235 --> 00:32:41,205 But right now this is it didn't require any government intervention. 594 00:32:41,205 --> 00:32:42,915 And so that's part of their argument. 595 00:32:42,955 --> 00:32:44,785 I think there's a lot of people who certainly 596 00:32:44,785 --> 00:32:46,545 in the industry that would be their take. 597 00:32:46,645 --> 00:32:49,865 We've actually done a pretty good job without any government 598 00:32:49,865 --> 00:32:53,865 rules and folks in our space, even if we are sympathetic to the 599 00:32:53,875 --> 00:32:57,995 issues that are being brought up by the FCC are a little leery 600 00:32:57,995 --> 00:33:01,355 about codifying something and how hard would that be to change? 601 00:33:01,375 --> 00:33:03,485 What are the unintended consequences of that? 602 00:33:03,485 --> 00:33:06,945 I think people Worry and wring our hands a little, rightly 603 00:33:06,945 --> 00:33:10,205 so, but anyway, that's the kind of the latest in that. 604 00:33:10,295 --> 00:33:11,315 And we'll see where this goes. 605 00:33:11,315 --> 00:33:14,155 I think there's a few more days as of the recording. 606 00:33:14,165 --> 00:33:16,615 We're recording this on the July 12th. 607 00:33:16,945 --> 00:33:21,045 I think you can submit, this will probably be published after this deadline, 608 00:33:21,045 --> 00:33:24,725 but on July 17th, I think is the deadline for submitting your pushback. 609 00:33:24,765 --> 00:33:26,305 I'm sure people are writing their opinions. 610 00:33:27,060 --> 00:33:29,160 Yeah, we'll be one day after that for publish. 611 00:33:29,160 --> 00:33:30,730 So you just missed it. 612 00:33:31,890 --> 00:33:32,650 Too bad. 613 00:33:32,830 --> 00:33:36,940 But yeah, no, I echo your concerns because I know anything 614 00:33:36,940 --> 00:33:39,980 that goes into a government regulation tends to ossify. 615 00:33:40,450 --> 00:33:43,580 And so now you're stuck with a very, if it's written 616 00:33:43,580 --> 00:33:45,990 in such a way, that's a very specific technology. 617 00:33:46,020 --> 00:33:49,530 You're just stuck with that until the lawmakers get around to updating it. 618 00:33:49,970 --> 00:33:57,050 We covered a story in our Lightning round this week all around how Japan 619 00:33:57,060 --> 00:34:02,520 has finally gotten rid of using floppy disks in the government agencies. 620 00:34:02,560 --> 00:34:05,980 And the reason they were using floppies is not because people wanted to. 621 00:34:06,210 --> 00:34:08,720 It's because there were very specific regulations 622 00:34:08,720 --> 00:34:11,390 on the books that required them to use floppies. 623 00:34:11,580 --> 00:34:14,870 floppy disks of a specific type and size. 624 00:34:15,160 --> 00:34:17,370 And so they're like, well, that's what we have to do. 625 00:34:18,140 --> 00:34:18,450 Wow. 626 00:34:19,070 --> 00:34:19,520 Yeah. 627 00:34:19,520 --> 00:34:22,950 So then I, then we have an added complication to that whole discussion 628 00:34:22,960 --> 00:34:27,640 is the Supreme court decision that overturned a Chevron deference. 629 00:34:27,850 --> 00:34:29,800 So I am not a lawyer. 630 00:34:29,800 --> 00:34:30,790 I don't know about you guys. 631 00:34:30,800 --> 00:34:34,490 So I, I have a very lay layman's understanding of this, but essentially it 632 00:34:34,490 --> 00:34:38,914 would curtail what a regulatory body can do on its own without a lawyer. 633 00:34:39,095 --> 00:34:42,905 This being explicitly laid out in legislation from Congress. 634 00:34:43,375 --> 00:34:45,935 And that means it's not clear to me again, as a, 635 00:34:45,975 --> 00:34:48,565 as a lay person, can they make these rules now? 636 00:34:48,635 --> 00:34:51,725 Is this considered under something that's presently in legislation? 637 00:34:51,775 --> 00:34:56,275 Because if it's going to require Congress to be involved, then all bets are off. 638 00:34:56,275 --> 00:34:59,655 I think, uh, we're definitely better off just 639 00:34:59,655 --> 00:35:01,875 letting the industry try to take care of itself. 640 00:35:01,915 --> 00:35:05,675 But anyway, directionally, I, uh, I appreciate their concern. 641 00:35:06,085 --> 00:35:06,705 This is the area that. 642 00:35:07,255 --> 00:35:10,045 Is worth trying to improve, but yeah, you just have to 643 00:35:10,045 --> 00:35:12,085 be careful not to create something counterproductive. 644 00:35:13,025 --> 00:35:14,025 Yeah, absolutely. 645 00:35:14,365 --> 00:35:16,015 Well, Doug, we're coming up on time. 646 00:35:16,055 --> 00:35:19,555 Uh, before we say goodbye, where can people find you on the 647 00:35:19,555 --> 00:35:22,545 internet if they want to know more and you can tell us a 648 00:35:22,545 --> 00:35:25,365 little bit about what Kentik does as well as your employer? 649 00:35:26,365 --> 00:35:26,955 Yeah, sure. 650 00:35:27,005 --> 00:35:30,455 So let's see, Kentik is a network observability company. 651 00:35:30,525 --> 00:35:34,995 And so we are best known by the NetFlow analytics products. 652 00:35:34,995 --> 00:35:35,415 We've been. 653 00:35:35,705 --> 00:35:39,245 Building for eight years and we grew up out of the 654 00:35:39,245 --> 00:35:42,595 service provider industry, but we now have, I think the 655 00:35:42,605 --> 00:35:45,335 majority of our customers are what we call enterprise. 656 00:35:45,345 --> 00:35:46,755 So these are just people who aren't. 657 00:35:46,820 --> 00:35:48,380 Telecoms or ISPs. 658 00:35:48,450 --> 00:35:50,570 That's who we've, we're spending a lot of our time with. 659 00:35:51,090 --> 00:35:52,340 So there's the NetFlow thing. 660 00:35:52,340 --> 00:35:54,940 We help companies with, uh, understand how they're 661 00:35:55,140 --> 00:35:56,990 exchanging traffic with their cloud deployments. 662 00:35:57,000 --> 00:36:00,230 So that seems to be a pretty big topic for us these days. 663 00:36:00,270 --> 00:36:01,760 Uh, we see a lot of demand there. 664 00:36:02,090 --> 00:36:05,210 We do synthetics, which is basically performance monitoring and BGB. 665 00:36:05,210 --> 00:36:06,030 So BGB is kind of my area. 666 00:36:06,080 --> 00:36:09,080 And so, uh, we have a BGP monitoring and analysis 667 00:36:09,490 --> 00:36:11,630 capabilities, but yeah, it's a cool company. 668 00:36:12,090 --> 00:36:15,400 And then as far as, uh, reaching out to me, I'm on. 669 00:36:15,845 --> 00:36:16,665 Twitter X. 670 00:36:16,705 --> 00:36:17,545 I'm on LinkedIn. 671 00:36:17,545 --> 00:36:19,265 That's usually the best places to reach me. 672 00:36:19,585 --> 00:36:21,485 Otherwise, maybe we could add a little link 673 00:36:21,525 --> 00:36:24,615 to, uh, I write blog posts for the Kentik blog. 674 00:36:25,045 --> 00:36:27,475 That's usually where I'm publishing stuff out to the world. 675 00:36:28,415 --> 00:36:28,795 Awesome. 676 00:36:28,805 --> 00:36:31,905 Well, we'll include links to all those kinds of things in the show notes. 677 00:36:32,285 --> 00:36:36,175 Doug Midori, thank you so much for being a guest with us on ChaosLiver. 678 00:36:36,195 --> 00:36:39,215 And hey, dear listener, thanks for tuning in. 679 00:36:39,215 --> 00:36:42,465 I guess you found it worthwhile enough if you made it all the way to the end. 680 00:36:42,465 --> 00:36:44,205 So congratulations to you, friend. 681 00:36:44,705 --> 00:36:46,055 You accomplished something today. 682 00:36:46,065 --> 00:36:50,925 Now you can go sit on the couch, fire up some ROAs, and implement RPKI. 683 00:36:50,935 --> 00:36:52,205 You have earned it. 684 00:36:52,595 --> 00:36:55,445 You can find more about the show by visiting our LinkedIn page, 685 00:36:55,445 --> 00:36:59,355 just search Chaos Lever, or you can go to our website, ChaosLever. 686 00:36:59,365 --> 00:37:03,245 com, where you'll find show notes, blog posts, and general tomfoolery. 687 00:37:03,685 --> 00:37:06,595 We'll be back next week to see what fresh hell is upon us. 688 00:37:06,775 --> 00:37:07,575 Ta ta for now.