This is going to be a little bit of a different show, isn't it, Chris?
Why?
Because we're not alone.
There's someone else in the room.
Do I need to call an adult?
Are you not an adult?
Have we met?
No, that's fair.
Hello, alleged human, and welcome to the Chaos Lever Podcast.
My name is Ned, and I'm definitely not a robot.
I'm a real human person, who needs to monitor their dihydrogen oxide intake
and output to make sure it stays within normal tolerances, just like you.
I definitely do not consume the blood of
humans to stabilize my quasi organic internals.
That would be vampirically ridiculous.
With me is Chris, who is also Hi, Chris.
I mean, when you say things are ridiculous, it
always sounds like you don't mean ridiculous.
Maybe.
As I alluded to, we also have a guest joining us, which is very exciting.
Our guest is Doug Midori.
Is that how you say your last name, Doug?
Yup.
Man, I got it on the first try.
That's awesome.
Doug is the Director of Internet Analysis at Kentik, and he has definitely
forgotten more than I have ever learned about BGP, routing, and the internet.
Welcome to the show, Doug.
Hey, glad to be here.
Thanks for having me.
Absolutely.
A few weeks ago, we talked about BGP a bit, and we established
that, well, if you haven't listened to it, probably go and listen
to it, especially if you're not familiar with BGP at all, but it's a
protocol that was initially designed when the internet was a small,
friendly place full of nerds who just wanted to get things connected.
Everyone kinda knew everyone, and asking others not to
be assholes about it just kind of worked at the time.
And that's true of so much of the early internet.
SMTP, HTTP, FTP, and other protocols didn't
have security even as an afterthought.
Essentially, everything was sent in plain text, and trust was just assumed.
And that was the world of the mid 1990s.
30 years later, the internet is a very different
place, and somehow, BGP is Largely the same?
So, let's talk about that.
Doug, so BGP was originally based on building relationships
between neighbors, exchanging network layer reachability
information, and it all had this assumption of trust.
When did that start to become a problem?
So, I mean, it's still Inherently is the same as far as that goes.
I started in this space in the year 2009.
So I've been doing this for about 15 years.
Um, in that time, there's been any number of cases of either like
deliberate routing hijacks in order to disrupt or misdirect traffic or.
More frequently, mistakes.
There's a lot of people will fat finger on a router causing a large internet
outage and a couple of the problems that we deal with in the BGP world.
Okay, so you've sort of, there's two different, I guess, things that
people have to be worried about that you, you brought out there.
One is like malicious activity, trying to mess with
neighbors and, and redirect traffic in some way.
And others just like, man, I had a, it was, it was Friday.
I was trying to get out.
I hit a command and uh oh, I've blown up half the internet.
Yeah, I mean, that was pretty frequent.
I, I would say it was too, a little too frequent
when I was, uh, getting started in this space.
And I would say we've made a lot of progress on that category of building
some bills and suspenders to try to improve that side of the problem.
What we, in the space call the other side is the determined adversary.
So someone who is very an attacker, who's very knowledgeable of what security
mechanisms have been deployed, how they work, what are their weaknesses.
And so the determined adversary is kind of an unsolved problem thus far on BGP..
Okay, so you mentioned that there are some things have
been put into place to help with the fat fingering problem.
What are some of those controls or ways
that we've tried to fix that half of things?
Yeah, sure.
And when I talk to audiences about this topic, I like to say that, you
know, BGP security or this, this whole topic, it's not a one thing.
This is a constellation of problems that we have
a variety of different things that can go wrong.
It's going to require a few different solutions.
And there's also a spectrum of difficulty of one end or kind of
the bonehead errors that hopefully we could come up with some
automated ways to prevent them from causing disruptions on up to.
Let's turn to the adversary I mentioned a minute ago, but yeah, we
can mention some of the things that people have networks are using.
A lot of it has to do with how do you filter the routes that you accept?
So in your previous episode, I'm sure you kind of went through
this process of this route by rumor on AS will accept routes.
From an adjacent AS to try to learn how to reach other parts of the internet.
So you'd like to have some sort of quality
control over the routes that you accept.
And in that genre of mechanisms, you have the
very coarse thing of like what we call max.
Pref settings.
So I, if I normally get 10 routes from you, I
shouldn't tomorrow suddenly start getting a million.
There's probably a problem.
And if it does, it should maybe kill the session or take some sort of an action.
So that was one of a number of like really simple mechanisms that we, um,
I don't want to take credit for it, but the industry adopted early on.
And then there's a filtering.
There's a couple of different ISPs we use to.
Filter the routes that they receive from their customers,
depending on the complexity, like maybe it's a really easy case.
They know it's just gonna be a couple of ranges.
They can put this into the configuration.
We should only receive this type of route from a customer.
But as you go up the stack in the Internet hierarchy, if you're going
to larger companies, It's going to be very hard for them to know what
are all the possible routes that could come through another, like if
you're a, like a tier one is accepting routes from a tier two, that tier
two could have a lot of different customers, a lot of types of routes.
So then we need to automate a process to build those filters.
And so we use, uh, we call IRR, internet routing
registries to build these, uh, to store information.
Um, so in that we'll say.
There's a couple different mechanisms there, but it'll
essentially try to whitelist what are the routes that
would be acceptable to be received from a customer.
And when you receive something that is outside that list, then reject it.
Problems there is that there's like 30 of those.
There's no single truth that everybody agrees upon.
It can vary.
Some of them have had some security issues, like the registry data itself has
been a target of an attacker who had found a way to put bad information in
that would enable them to announce routes they're not supposed to announce.
And so the IRR area is something we, we use it's widely used, but
we're hoping Try to get past that and use things like RPKI, ROV.
So that's kind of the technology du jour right now.
And let me explain a little about what that is.
So RPKI is a resource public key infrastructure.
So this is a cryptographically secure and enforced platform that ISPs can build
services, use services that are built off of RPKI to perform route filtering.
And so ROV is Route Origin Validation.
One of hopefully, hopefully there'll be more
as the vision, but ROV is the first one.
A lot of times in this space, people say RPKI, they're really referring to
RPKI ROV because that's the one application that people are actually using.
So route origin validation.
The way this works is that an address, or we call it a resource holder, or the
person who owns the address space would, typically they do this through their
RIR, so I'm using a lot of acronyms here, but if you are in North America,
your RIR is Aaron, and you would log into the Aaron portal, you would have the
account login, if you are the owner of the address range, and through there you
can assert, you can build a ROA, Another acronym, a route origin authorization
to say, what is the AS that is allowed to originate this address range?
And then there's an expiration date, a max prefix length, there's a few
other details there, but mostly this is the, what's the correct origin.
Now that information then gets published out to the internet and every
entity in the world that is rejecting RPKI invalid routes will then
use that information to determine When they receive a route, they
would check the AS path and look at the rightmost AS in the AS path
that would be considered the origin and see if that matches the origin
listed in the ROA, the Route Origin Authorization that's stored in RPKI.
So hopefully you can follow all that, but you know, the benefits there are
you've got, it's just, all the information is cryptographically enforced.
You can't force some bad information in the path here.
It is one ground truth for the world.
So we don't have that doubt or uncertainty
of which document are we going off of.
This was discussed for many years and there was advocates and
debate around this, and eventually it finally took hold about.
Four years ago or so, we started to start seeing adoption.
And for a while, the issue was trying to deploy globally.
A security mechanism on the internet is a really hard thing, right?
This is, there's no money in it for anybody.
Everyone doing this is trying to do this
for the benefit of the rest of the internet.
So to get people to do it.
There's two steps that a network has to
perform in order to have deployed RPKI ROV.
So they would create ROAs for their address
space to essentially communicate to the world.
What's the correct origin.
And then they also need to reject RPKI invalid
routes that are coming through their network.
But for a while, we had a chicken or the egg, where why would
anybody bother creating ROAs because no one's rejecting invalids?
And why would anybody reject invalids because no one's creating ROAs?
Well, we've somehow.
Managed to get ourselves past that chicken or the egg phase.
Um, probably the biggest facilitator was when we had tier
ones back in the year 2020, start rejecting invalids.
And I'm talking about, it used to be called Telia, now it's Aurelion, Lumen,
Cogent, GTT, like these really big global, top of the internet telecoms.
They have huge.
Downstream customer cones and cast a wide shadow.
And so when they do something, it has broad effect.
So when in that year around, it's about a year that those networks started
rejecting invalids, and that really started the ball rolling, in my opinion,
and we can see that if you track these things through time, you can see
there's an inflection point around that time where people start creating
ROAs, because I feel like someone's going to actually do something about this.
And on up to just in May this year, we crossed the, um, arbitrary milestone of
getting past 50 percent of the routes in the global IPv4 table now have ROAs
and are essentially eligible for the protection that would RPKI ROV would offer.
IPv6 reached that milestone.
Last year, probably due to the fact that it has less legacy stuff to deal with.
But anyway, so we, we look at this as success
and getting ourselves on the right path.
However, I want to be careful.
The people who are working on this topic, try to choose
our words carefully, not to overstate what RPKI ROV
is going to do for anyone, because it can be defeated.
What it's most successful at is suppressing
routes that are, uh, due to misoriginations.
So someone has incorrectly, whether deliberately or not, originated
address space they're not supposed to, then essentially the system
will just suppress those routes and it'll reject those invalids.
Like I said, it isn't foolproof and we're going to need more
mechanisms to try to go up the spectrum and push the needle
on up to the determined adversary to try to secure that side.
But we had to start somewhere in the space and it's,
as you can imagine, the internet is a big place.
There's a lot of different companies and people working here.
So to pull off a voluntary adoption of a
global technology is a non trivial thing.
So I'll stop there and see if you have any questions.
So one of the biggest things I think if anybody knows anything
about problems with BGP, the things that make the news are like when
countries make a mistake and accidentally transfer 75 percent of the
internet's traffic through Pakistan or through China or accidentally
knock YouTube offline for 25 minutes due to routes that become
completely invalid and are Transmit it out to the entire world.
So these are not necessarily adversarial attacks, right?
But these are people or people that have rights to publish global tables, right?
Does any of the stuff that you've talked about interact there in
any way to help or mitigate with problems that could be propagated
worldwide from what would be considered a valid tier one endpoint?
Yeah, so you brought up the Pakistan YouTube incident, which is probably
maybe one of the more famous BGP incidents that have ever occurred.
It's worth reflecting, it's probably worth a topic on a blog post of
like, if the same thing were to happen today, how would that be different?
Because it would really, it's really not possible what took place.
So just the backstory here was, this occurred I believe 2008.
Where there was a video on YouTube that was deemed anti Islamic.
And so the government of Pakistan gave an
order that they needed to block YouTube.
And so that's came down to PTCL is a state telecom of Pakistan.
They decided they would do this via BGP.
And so they would just attract all of the BGP.
They create a route of YouTube address space, try to attract all the traffic
that was going to YouTube and put it in the bit bucket when it comes.
So that part, it was intentional.
What wasn't intentional was that they announced this.
Accidentally out to one of their international transit
providers who carried it out to the global internet.
And so then because it was a more specific, like a BGP prefers
more specific, longer length routes became very popular.
And about two thirds of the internet for, I don't know, a couple of
hours was believing that they need to go to PTCL in Pakistan for YouTube.
That made YouTube unreachable.
Pakistan also wasn't doing great either.
This was, they've not used to getting that kind of traffic.
Yeah.
So ultimately that was, that was resolved.
I think Google had recently purchased YouTube.
They intervened, got the international carrier to stop carrying the route.
But if we looked at that case again today, I call these
accidental, but also intentional or intentional, but also
accidental because, um, we had this, uh, happen recently.
There was a couple of cases that are worth thinking about where a
similar case in, uh, in Myanmar and let's see, try to get my dates right.
I guess it was a spring of 2021.
There was a military coup in Myanmar and it was a lot of.
Government involvement in suppressing communications.
So there was like a total shutdown.
There were mobile, like nightly shutdowns.
There was all kinds of different things.
Everything, everything we'd ever seen in the digital
rights space was happening over a couple of months.
They even had their own Pakistan YouTube incident where they
had given an order out to the ISPs to block social media.
One of the ISPs in Myanmar decided they would do exactly what.
PTCL had done in 2008, and they would take Twitter address space,
they would basically announce it locally, it was I think their plan,
and just drop that traffic when it comes to announce this out to the
internet, and so around South Asia, there was a Twitter outage where
all this traffic was getting directed to Myanmar, and then fast forward
one year later, almost a year to the day, That incident, the exact
same thing happened to the same address range, same prefix, everything.
And this time it was out of Russia.
So Russia had invaded Ukraine.
There was a backlash in, uh, within Russia.
They started cracking down on independent media and social media.
And we had the same thing as an ISP to created a BGP route to block Twitter.
Accidentally announced it out to a transit provider, but the
difference was that between the spring of 2021 and spring of 2022,
Twitter, now X, had created ROAs for all their address space.
So this route now had a ROA and Routers all over the
world would know when they see the one coming from
Russia, this isn't the right one and they would reject it.
Now that doesn't help the people in Russia, but
at least contains the disruption to that area.
And we just, we're not going to be able to get,
we're not gonna be able to intervene beyond that.
Anyway, so that's a good story of that growth.
So if PDCL today works to announce those YouTube routes, they
would probably go nowhere and they would probably affect no one.
The other thing is that YouTube isn't pulled across trans providers anymore.
It's served through embedded caches in your
ISPs in nearly every country in the world.
There's really very few exceptions to that.
So even if the RPKI wasn't there and the route got out, I think.
Don't know that it would even mess that much, be worthy
of debate of how much impact it would really have.
It certainly would be significantly less than what occurred in 2008, just to
be a, uh, how the internet has evolved, how content is delivered, but within
the routing space, it also kind of can't happen, or at least it'll be limited.
I wouldn't say there's zero impact, but RPKI,
ROV, yeah, like I said, it's good in those cases.
This is where it's strong.
And then, you know, in recent years, Where we've seen a lot of activity
in the determined adversary category is against cryptocurrency services.
So these are great targets for hackers.
Cause if you can crack one of these places and steal the money, it's, you can
have it immediately and launder it and you're gone and there's no recourse.
So.
and make for good targets.
And so we've seen some sophisticated attacks that involve
BGP hijacks, including one that did a hijack against Amazon.
So this is not a mom and pop shop.
This is one of the most well resourced networks
in the world that did all these things.
It defeated RPKI ROV by forging the AS path so that it would be seen as valid.
So the route The bad route would get circulated.
It created a fake entry into, um, ALT db, one of the,
our IRRs that are used for automated creation of filters.
And so there was a lot of lessons learned there, but some
of these cryptocurrency attacks, I wrote up a, a piece,
maybe we can put it in the, the notes for the, the episode.
Just looking at like this is where people in our space are trying to
spend more time thinking and so that's progress where we're less worried
about these fat finger cases kind of covered as far as best as we can
get them and now we need to focus on determined adversaries scenario.
So I read through that article that you're talking about and I found
that one example you gave of the cryptocurrency and the way that they
had used all the infrastructure you would expect to sort of spoof
where the routes were coming from and then make it all look legitimate
so traffic would go there and be none the wiser and then they would
just, you know, steal your bitcoins or whatever and abscond with them.
Are there some new standards or things coming down the
pike to also protect against that sort of scenario?
Yeah, so there's always more.
The next thing that's being pushed or advocated for is something called ASPA.
So that doesn't deal with the cryptocurrency attack
scenario, but autonomous system provider authorization.
So what the way this works is Each network, which is an autonomous system,
is going to, within the system, assert what are its transit providers.
And in the same way that we create AROA, and this will be information that's
stored within the RPKI platform and cryptographically delivered everywhere.
So then by asserting what are your transit providers, this enables other
networks to look at an AS path and detect what we call a value free violation.
And so maybe I'll explain a little bit what that is.
So in BGP, I don't know if you got into this in your
last episode, but there is hierarchy to the whole thing.
So it's not, I think maybe in a textbook,
it might look like it's a little amorphous.
Everybody just kind of connects to everybody
else or, but there is a hierarchy to it.
So for the most part, you have networks
that are buying transit from other networks.
So you get to the top, there's a default free zone or Transit
free zone of the top of the internet is kind of cabal.
There are a dozen or more ISPs that don't buy service from anybody.
They just sell and then they connect to each other.
And so as traffic goes across the internet, it's either, you know, crossing
these transit edges or there's a, the alternative is a peering relationship.
And these are the two classic types of relationships.
What you can't do is.
Your traffic is always going to go over a hill, so you're
going to be going up the transit links until it gets to a
top and then comes down the other side to the destination.
Or maybe you've got a way to cut through it with a peering
relationship, but you can't go down, because if you go down,
when we draw this on diagrams, we say, you know, transit is up.
That's how we draw it, and it's the mental model.
May not translate in the podcast very well, but so then if you're going down,
basically you're drawing a line, you're going from a provider to a customer.
Back up to a provider, then that customer
is paying both sides to send that traffic.
It's all about money.
It's as much about technical stuff as it is around business.
And so when you and I have internet connections at our house,
our phone, for the most part, they're kind of all you can eat.
You pay some sort of flat fee, and unless you do something Crazy, you're
hosting, um, whatever stuff, you don't worry about how much you're using.
But in the wholesale market, it's by bit, so by volume, you pay by volume.
And so then they're trying to either reduce, the providers
are trying to either reduce costs or increase revenue.
And one way they reduce costs is by peering, so
they don't get around other transit providers.
But, um, if you're going, you know, From a provider to a customer, back up to
a provider, then that customer is paying both legs and is receiving no money.
So that's a, that's something you would never want to have happen.
And people go to great lengths to try to avoid that because you're
basically paying twice for something you're not getting paid for.
So that's a valley in this valley free terminology.
And so that does happen, but it's usually a leak, like a mistake.
So as has taken a route from one side and send it to another by mistake.
We covered that a little bit in the last episode because I talked about
what happened with the Allegheny provider where it was using Verizon
and uh, DQE or something were the two providers that it was using.
And it had accidentally leaked routes from Verizon out through DQE
telling everybody, you know, send your traffic through me basically.
And so that would have been a big
Yeah, it was a great example of a valley.
So, uh, in that case, yeah, Allegheny is a customer of both Verizon and DQE.
The routes went from DQE to Allegheny to Verizon.
The vision is that could be picked up and just blocked immediately.
Had, you know, Allegheny asserted in RPI.
Using ASPA, who are its transit providers, people would
look at that and be like, okay, somebody made a mistake.
I won't carry this.
The incident has a few lessons learned.
Obviously, Cloudflare made a lot of big stink
about it because they got affected, rightly so.
And it did prompt a discussion around RPKI ROV.
But what's interesting to me is that RPKI ROV would
have helped in that case, but not because of the origin.
Being changed, the origins for the routes that were leaked were actually intact.
So like CloudFlare's 1.
3.
5, as somebody who works with this stuff, I
have like thousands of these ASNs memorized.
So the, uh, the AS origin, the rightmost AS and any of those route leak paths.
It was correct.
So you, by checking the origin, it wouldn't have filtered the routes,
but because the Bluewoods DQE was using a route optimizer, which
locally creates these more specific routes to try to do traffic
engineering, those more specific routes, then For what really attracted
the traffic, but they also would have been RPKI invalid because
the routes of CloudFlare and I think Akamai was in this as well.
They had ROAs that set a maximum prefix length.
So our prefix, we call a prefix as a BGP route as a address range
and the address range has got, you know, a network portion and a host
portion and then the prefix length then sets what's the network portion.
And then in the, in our Perlansen routing, we just talked
about prefixes, but in that case, those prefixes would have
been invalid due to the max prefix length setting in the ROAs.
So, I also bring that case up because that's probably the
last really big debilitating routing leak that's occurred.
And so that was, I think, 2019.
We're a little more than five years out.
Which I, when I give talks, I was like, there's not an accident that that was
the last, I mean, we may have another one while we're speaking right now, it
could be something disastrous could be happening, but we've gone a long time.
Five years is a long time in internet time.
And so things have gotten better due to a lot of these things.
The filtering we talked about, Max prefix lang.
ASPA really is just getting started, so I don't think we're going to
see benefit from it for a little while, but, um, RPKI ROV is helping.
Yeah, there's just a variety of different, we call it routing
hygiene, just all these, all these different things that riders
do, and there's no, there's a lot of best practices, and Manners is
a organization that's mutually agreed norms for routing security.
It's kind of the industry's advocacy group for enumerating what are
the things that networks need to, steps they should take to improve
routing security, routing hygiene, and they've been very instrumental
in being the go to and being good advocates for routing security.
Manners is sort of like an opt in, right?
You want to be a good citizen, you want to have good manners.
That's correct, yeah.
I know recently the FCC has been trying to push being a little
more stick and less the carrot when it comes to BGP security.
So what is the FCC trying to do and and how is the
industry responding to what they've been doing?
Yeah.
So back in 2022, the FCC, this is a federal communications commission, a
US agency overseeing our telecommunications sector in the United States.
Decide to get involved or get, figure out what, what
leadership does it need to provide in routing security?
Obviously this is something that affects the United States
and there's a national security element to this as well.
They began a process where they were seeking inquiry, asking
industry experts Ask them what they think they should be doing.
And you know, there's been a bunch of events that
they've held and documents they've published.
And then this year within the last couple of months,
they published what are they're seeking comment.
We're really trying to get illicit feedback from the industry and they're
getting Of like what should be the rules they require of us telecoms.
And so what they did was Identify a list of nine telecoms.
I call them bias Oh, what's another acronym is not an acronym
I'd seen before, but there's always, there's always a new
one that these nine telecoms need to, uh, deploy RPKI ROV.
So there's two things that they need to do that they need to create ROAs.
And there's a way we can all measure and see that they provide these.
So BIAS stands for Broadband Internet Access Service.
And the companies are AT& T, Altus, which owns Suddenlink, Cablevision,
Charter, which is also Spectrum, Comcast, Cox, Lumen, which, I don't know
if everybody knows, is both a regional rub in provider and also plays
this role in the global internet, T Mobile, our mobile operator, TDS,
which now owns, or is in the process of owning, US Cellular, Verizon.
Anyway, those are the nine.
The each of these companies needs to create rows for their address space, at
least 90 percent of the routes that they originate and also rejecting ballads.
And so they're seeking comment on this.
And so what I decided I would look at was, all right, well, now
that they named these companies, let's see how they fare today.
And I just ran through the numbers, at least on the row creation side.
It's very tricky for us to remotely figure out to what
extent, if at all, uh, they're rejecting invalids.
Different people have come up with different methodologies.
It's still a kind of open research area, but five out of the nine would do
very well right now where they're probably already at 90 percent of the routes.
And I, I work at a company that does deals in large amounts of net flow.
So we work in the service provider space.
So these are companies that we would Work with, and so we have a lot of,
uh, NetFlow that gets shared to us as part of the service that we provide.
And so we have a, a nice slice of the internet of just
the traffic that's going across it for analysis and study.
This is what I spent a lot of my time doing.
And so I looked at the traffic that we see going to these large
US telecoms and how much of it was going to routes with ROAs.
And for a few of them, like T Mobile and Cox and Comcast.
Charter Spectrum.
It's nearly universal.
Almost every packet going to those networks are going to
routes with ROAs, meaning that they're eligible for protection.
That's one side of the internet transaction.
That's the traffic coming back to the user.
The other side is not really covered in this
proposal, which would be what are they going to?
What are they sending their traffic?
Where were they requesting data from, which could be a bank or Cryptocurrency
service that's under attack or, uh, that side of it, they're not getting
into, although I suspect that may not be where they focus their time.
They're looking at what are the rules that the US telecoms need to follow?
Yeah.
So I went through this and yeah, it's generating a little bit
of discussion here in our community of just, for one, whenever
you do something, I'll just, you can't get it all right.
I try to be explicit about what ASs I was using for the analysis.
I'm getting some feedback.
I missed a couple.
I'm happy to update it.
But some of these companies use dozens or ATT is over a hundred.
ASs that they use.
So I'm trying to manage the complexity of that and still provide some analysis.
But, you know, in this proposal, the Internet Society, which was the
former home of Manners, which you talked about earlier, which since has
moved to the Global Cyber Alliance, those two entities wrote up a joint
Ex parte response to the proposal by the FCC saying, pushing back pretty
strenuously against a federal requirement that telecoms adopt RPROV.
And the points they made were that it's dangerous in security to legislate.
You have to use this particular solution because now everybody
has to do that thing and maybe it's becomes obsolete or things
have changed and now it's a counterproductive and you're stuck.
Complying with this rule, there's a concern there
of just ossifying some kind of a requirement.
And then also, I guess, smaller providers
were kind of excluded from this first pass.
They made the argument that small providers, this would be
a cumbersome or burdensome burdensome thing to comply with.
But, you know, the first point they made was Using the analysis that
myself and this other expert in the space, we've kind of collaborated
quite a bit on this topic, this guy, Job Snyders, who's at Fastly now
as probably the leading voice in routing security has been for awhile.
In fact, a lot of the progress that we've
made is directly attributable to his work.
So he and I worked together quite a bit on this and.
Both the FCC rules and the ex parte response, pushing back
on them, both relied on our analysis that showed that,
you know, we have a lot of ROAs that have been created.
In fact, those ROAs represent the majority.
Now it's a super majority of the traffic that's
exchanged on the internet is going to routes with ROAs.
And then conversely, routes that are deemed
invalid just don't get propagated as much.
In fact, they're suppressed quite a bit.
So the system is working as designed and we've reached a point of.
Adoption where the next network to do these things, to create ROAs
for their address space, to start rejecting invalids would have
immediate benefit because there's been so much adoption thus far.
That work was highlighted in both the FCC document as well as the pushback.
Yeah, their point was, look, the industry has already made
a lot of progress and there was no government mandate.
I mean, that's, uh, it's a pretty good argument because all
this progress I'm describing, there's no government mandate.
It's just simply advocacy work within the communities around the
world and getting it to a point where there's some peer pressure, if
not shame, to motivate networks that aren't doing the stuff to do it.
And if, you know, things change and there's a
better solution, this needs to be abandoned, then.
So be it.
But right now this is it didn't require any government intervention.
And so that's part of their argument.
I think there's a lot of people who certainly
in the industry that would be their take.
We've actually done a pretty good job without any government
rules and folks in our space, even if we are sympathetic to the
issues that are being brought up by the FCC are a little leery
about codifying something and how hard would that be to change?
What are the unintended consequences of that?
I think people Worry and wring our hands a little, rightly
so, but anyway, that's the kind of the latest in that.
And we'll see where this goes.
I think there's a few more days as of the recording.
We're recording this on the July 12th.
I think you can submit, this will probably be published after this deadline,
but on July 17th, I think is the deadline for submitting your pushback.
I'm sure people are writing their opinions.
Yeah, we'll be one day after that for publish.
So you just missed it.
Too bad.
But yeah, no, I echo your concerns because I know anything
that goes into a government regulation tends to ossify.
And so now you're stuck with a very, if it's written
in such a way, that's a very specific technology.
You're just stuck with that until the lawmakers get around to updating it.
We covered a story in our Lightning round this week all around how Japan
has finally gotten rid of using floppy disks in the government agencies.
And the reason they were using floppies is not because people wanted to.
It's because there were very specific regulations
on the books that required them to use floppies.
floppy disks of a specific type and size.
And so they're like, well, that's what we have to do.
Wow.
Yeah.
So then I, then we have an added complication to that whole discussion
is the Supreme court decision that overturned a Chevron deference.
So I am not a lawyer.
I don't know about you guys.
So I, I have a very lay layman's understanding of this, but essentially it
would curtail what a regulatory body can do on its own without a lawyer.
This being explicitly laid out in legislation from Congress.
And that means it's not clear to me again, as a,
as a lay person, can they make these rules now?
Is this considered under something that's presently in legislation?
Because if it's going to require Congress to be involved, then all bets are off.
I think, uh, we're definitely better off just
letting the industry try to take care of itself.
But anyway, directionally, I, uh, I appreciate their concern.
This is the area that.
Is worth trying to improve, but yeah, you just have to
be careful not to create something counterproductive.
Yeah, absolutely.
Well, Doug, we're coming up on time.
Uh, before we say goodbye, where can people find you on the
internet if they want to know more and you can tell us a
little bit about what Kentik does as well as your employer?
Yeah, sure.
So let's see, Kentik is a network observability company.
And so we are best known by the NetFlow analytics products.
We've been.
Building for eight years and we grew up out of the
service provider industry, but we now have, I think the
majority of our customers are what we call enterprise.
So these are just people who aren't.
Telecoms or ISPs.
That's who we've, we're spending a lot of our time with.
So there's the NetFlow thing.
We help companies with, uh, understand how they're
exchanging traffic with their cloud deployments.
So that seems to be a pretty big topic for us these days.
Uh, we see a lot of demand there.
We do synthetics, which is basically performance monitoring and BGB.
So BGB is kind of my area.
And so, uh, we have a BGP monitoring and analysis
capabilities, but yeah, it's a cool company.
And then as far as, uh, reaching out to me, I'm on.
Twitter X.
I'm on LinkedIn.
That's usually the best places to reach me.
Otherwise, maybe we could add a little link
to, uh, I write blog posts for the Kentik blog.
That's usually where I'm publishing stuff out to the world.
Awesome.
Well, we'll include links to all those kinds of things in the show notes.
Doug Midori, thank you so much for being a guest with us on ChaosLiver.
And hey, dear listener, thanks for tuning in.
I guess you found it worthwhile enough if you made it all the way to the end.
So congratulations to you, friend.
You accomplished something today.
Now you can go sit on the couch, fire up some ROAs, and implement RPKI.
You have earned it.
You can find more about the show by visiting our LinkedIn page,
just search Chaos Lever, or you can go to our website, ChaosLever.
com, where you'll find show notes, blog posts, and general tomfoolery.
We'll be back next week to see what fresh hell is upon us.
Ta ta for now.