What's very strange about tail scale, uh, and very strange in
the security world in general, is that when you use tail scale to
solve that problem, you accidentally make your system more secure.
And also the easiest thing for all of your engineers and people inside
your company to do becomes the secure thing instead of the insecure thing.
Welcome to Screaming in the Cloud.
I'm Cory Quinn.
It's been a while since I've had Avery Pener run on the show.
Thank you for joining me.
Again, you are still the co-founder and CEO of tail scale,
which at this point is getting pretty darn close to.
You've heard of this company when I bring
it up in almost every conversation I'm in.
That is pretty exciting.
I think.
I can't remember when I was on your show last time, but it was at least a
couple years ago and we've been growing really fast in the last couple years.
This episode is sponsored in part by my day
job Duck.
Bill, do you have a horrifying AWS bill?
That can mean a lot of things.
Predicting what it's going to be, determining what it
should be, negotiating your next long-term contract with
AWS, or just figuring out why it increasingly resembles of.
Phone number, but nobody seems to quite know why that is.
To learn more, visit duck bill hq.com.
Remember, you can't duck the duck bill.
Bill, which my CEO reliably informs me is absolutely not our slogan.
I'm seeing you in more and more places.
I've been using you in my personal environment for many years now,
and the stuff that I set up once upon a time is still working.
You're rolling out new stuff that continues
to add, be additive to this at at at work.
I'm paying you now, which was a big problem I had with you
previously of there's no good way for me to give you money.
Could you maybe fix that?
Good job.
You fix that.
So things are all up and to the right, which is kind of amazing.
It is kind of amazing.
It's amazing how long we can keep doing it.
Although I've been informed that if you keep doubling revenue
at a hundred percent year over year, then in 10 years you'll
be a thousand times bigger, and that might not be realistic.
But it might
at some point you hit population limits.
Uh, last year I gave the opening keynote at Nano 91, and the
whole theme of what I was talking about back then was that
there's been a rising tide in the level of what clouds could take.
From, from folks who are working in on-prem environments,
networking is becoming something of a lost art.
When you find someone who works as a network engineer, they're usually
my age and not new grads who are playing around with these things.
Tail scale is in some ways an answer to some of this, where
you, you're taking things away from the traditional network, uh,
switch and router world and into just make a big flat network.
And then we'll wind up handling this through policy files for access control.
Even recently this year, you folks wound up redoing your policy, uh,
policy, uh, format as far as making it a lot easier to do grants with
access grants, as well as now creating a visual builder, which I've not
yet played with because I haven't found a way to make it work in Vim yet.
One of my fixations as a CEO is I insist that
every change to the policy file get run by me.
Almost nothing else in the whole company runs by me, but I'm like, no,
if you're changing the policy, policy and tax, I wanna see it first.
So we went through a lot of iterations of the ACL grant syntax, uh,
before we finalized it, and I'm really excited about what we came up with.
I realize it's a little strange to be really excited about a
file syntax, but I actually am really excited and I think in a
little understood feature of ACL grants is it's really extensible.
Like you can grant stuff to applications that are provided by people
that are not tail scale, that are running on your tail scale network.
And when you connect to that application over tail scale, it has visibility.
Into the grants that you gave it based on your groups, the tags, the
blah, blah, blah, whatever's going on in your tail, tail net routing.
And it doesn't have to know about, it doesn't have to know what group you're in.
It doesn't have to have its own business logic about what group you're in.
It can just say like, tail scale says this connection
should be allowed to do this thing on this thing.
And you can change all that in a central place.
And so easy example is like Grafana, you can say.
Today, everybody in the production group should have admin access to Grafana.
So when they connect to Grafana, they get admin access.
You don't have to set anything up in Grafana.
If you change your mind later or you change who's in that
group, then next time they connect to Grafana, even if it's
like three seconds later, their permissions are gonna change.
Right?
And that was not possible before we had this ability
to just sort of like pass these things through.
And so it gives you this ability to just build on top of
tail scale and just stop worrying about all that stuff.
You also have a great feature where you can effectively disallow people
from modifying things in the console without going through a whole bunch
of very scary warnings, uh, and mandating effectively a GI ops flow, which
is fantastic, especially combined with the fact that you have test cases
built into your policy files.
Exactly.
Well, it's super fun, right?
Because you, you just said like, you can't use our ACL editor because you like
Vim, and like, that's not actually true because we did this another, another.
Very nerdy thing I'm super excited about is that you can round trip the
JSON of the policy file to the GUI editor and back with no loss of anything.
And it's not just regular JSON, it's our special
weird hug, JSO, that has comments and extra commas.
Uh, which means you can actually have
comments, your js o describing what goes on.
And then when you go to the GUI and then back, the comments don't get lost.
And so you, and also GI Ops can like take this text, store it in GitHub.
Um.
And then push it back when you're done.
And then of course it's not very good to have the
GUI edit it in tail scale if you're using GI Ops.
But you can go to the gui, come up with a rule that you want,
or an edit that you want, and it'll tell you what Text to paste
back into your Git repository to get the results you want.
So it's this very nice flow where like everybody
who likes everything gets to have what they want.
It's weird.
It's just thinking back.
It's been a bunch of small releases, but they add up to almost a, that
completely different product that still does the underlying baseline thing.
It always did, which is flattening the network to make it work
like we all used to think networks did until we knew better.
It's a, it's been a very.
Fun evolution.
I think it was last year you did that partnership with Mulva, where I think
for five bucks a month now I can get, I get access to the Mulva VPN stuff.
It's a couple of clicks of a house and I'm suddenly emerging from anywhere else
I want to be, which is super handy for me and my brother who lives in Brussels.
We are at poe, both have EU and US citizenship, so, but there's an
awful lot of government sites that, oh, you're not physically here.
Clearly you could never wanna access these things for no apparent reason.
Trivial,
easy.
Al
Also my bank, uh, in Canada, whenever I go traveling anywhere
that is not Canada, they're like, oh my God, nobody outside
Canada could possibly need to access a Canadian bank.
And they kick me out.
Uh, but I, you know, I could do that by, what I do is I use a, an exit
node on my Apple TV at home and I just bounce through my Apple tv.
Uh, but I also use Malva for experimenting and stuff.
I did that originally and then the raspberry pie I
sent with my brother to his place wound up dying.
And that is not, he is a government functionary there.
He's not really the, uh, the technical type as far as, Hey,
now log into the Linux console and tell me what you see.
Y no, that's why I went with the Apple tv.
'cause they, you know, they're five times as expensive, at least
as a raspberry pie, but they have five times at least as much
quality control as a raspberry pie in the manufacturing process.
So,
and a warranty service that is comprehensible to humans.
Yeah.
And a gui where you can just tell your brother like, Hey, can you go to
the app store and pig tail scale, uh, as opposed to going to the console.
Yeah.
I, I also like things that are, that have changed or some things have not
changed in tail scale that are still somewhat annoying and I understand why.
Uh, I'd love to be available to connect at two tail nets at the same time.
Now you can be logged in and toggle between them, but yeah, a device
that talks between two networks is generally considered a bridge
and corporate security would like a word if you start doing that.
There are ways now to share nodes between tail nets
that start making that a lot more straightforward.
I would still love, on some level, the ability to set a custom
domain for the tail net domain that I can control the certs for.
I, I get that that is a hard thing to do.
I'm sure some big customer somewhere has it, but Yeah.
Yeah.
It's, it's surprising how it's, well, that particular feature.
It's a little hard to do.
I would say the difficulty of doing it is not actually the thing holding us up.
What's the, the thing holding us up is the
phishing potential when you start doing it.
Uh, 'cause you combine that with tail scale funnel and people register
some arbitrary domain that looks suspiciously like, but is not quite.
google.com and next thing you know, you're hosting phishing sites for Google.
Do.
Right.
If everything ends in blurb, butty blurb ts.net, then you don't have
that problem and it's like remarkable how much trouble that saves us.
So we really want to get to the custom domain thing.
We just need to like very carefully control who gets to
have custom domains and minimize the abuse potential.
One way is like to attach it to not easy.
All the people that pay you.
That was one of the things we've been thinking.
It's actually, it's, I mean, it's, it's a pretty good start.
We should probably do that.
Exchanging money for goods and services.
That's wild.
Yeah.
It's just the, do we need to limit it to only those people?
It's kind of sad to have to do that.
I wish we had a better idea, but like, you know,
nevertheless, uh, yeah, it's definitely on the list.
Similarly with sharing, we've been in the same
state with node sharing since I think like 2021.
And a bunch of internal changes have been going on, uh, architecturally
to finally like enable way more kinds of interesting sharing.
Um, but I really see like there's so much
potential to newer kinds of, uh, node sharing.
I don't think you ever want to be in two tail nets at the same time.
I realize that everybody at first thinks you would
want to do that 'cause it would be really tempting.
But it is this bridge between tail nets and it like really confuses.
Like, as an example, I would like to be in two tail nets at the same time.
I have a personal account with my family stuff on it,
and I have a work account with all my work stuff on it.
And where I'm the CEO, uh, that has access to a bunch of sensitive things.
Right now, if I'm at a computer with my, and maybe your
children should not have access to those same things.
Yeah.
Maybe they shouldn't.
Right?
So if I have a device that my children borrow, right?
An iPad or something like that.
I really should not be logged into that device using my tail scale account.
But if I'm on my corporate device, I really
would like to have access to my private stuff.
'cause why not?
Right?
But if I'm logged into both tail nets at the same time, now I'm
inadvertently creating a bridge between my corporate account.
So the security team should lock me out.
Uh, and my personal account, right?
And the security team incidentally almost locked me
out a few days ago because I wasn't on the MDM yet.
So I like forcibly enrolled into the MDM, which forced me to upgrade my Mac Os.
And there's a bunch of new features in Mac OS that I was missing.
So I guess that's good.
And that yak is getting nicely shaved.
Yep, exactly.
So I, I'm pretty far down this path, but, you know, um, anyway,
what I think people want, and what I want to give people is
the ability to log into each device using exactly one account.
And for you to be able to share many or all, or a good
subset of the nodes from another account into your account.
You are almost taking the GitHub identity model.
Yeah, I guess so.
Yeah, I, I have a GitHub account, but I can, I be
outta different organizations that do different things.
My personal account is also what I use for
work, but you can gate access to things and
yeah, that part makes me nervous.
Like when I log into GitHub, I have access to
all my corporate stuff and my personal stuff.
So if I log into my personal GitHub account.
When I'm not on a work computer, I'm like putting work at risk, which is scary.
So what I think we should do is still have the two accounts, but on my
personal devices, I log into my personal account that doesn't have access
to my corp stuff, but I log in my work computer, I've access to all my corp
stuff, and my corporate user also has outgoing access to my personal stuff.
Yeah.
For the last eight years, I haven't really had anything
personal because my entire life has become work Right around
the time that, uh, shit posting on social media became a job.
Yeah, I guess that makes sense.
So yeah, I mean, yeah, I, I'm really talking
about the experience for other people.
Um, but yes, I mean, I have an Apple tv.
Does the corporation want my Apple TV on the corporate network?
Like not really.
So little things like that, and I think we can do it.
We're getting very close to being able to do it.
We just keep, like doubling in size a lot.
Uh, and so most of the engineering that we do, uh, actually ends
up being just like, Hey, you, you now have like a tail net with
hundreds of thousands of nodes on it with like thousands of nodes
churning per minute because someone is using it in a gigantic.
CICD cluster.
Uh, did you know that's an n squared algorithm?
Uh, did you know that the whole system is gonna crash?
'cause you did that?
I'm like, oh, I didn't know that.
Um, but then we had to fix it.
We learn exciting things, uh, through other people's use cases.
Exactly.
So some of this stuff keeps getting delayed, uh, but
it's gonna be really good when it finally comes out.
Yeah.
And, and you're, you have a great list of customer references that are, that
are doing all sorts of fascinating stuff, some of whom I know reasonably well.
And what I've, I also like the fact that there are options if
tail skill isn't right for people, if you want one that is a
lot more confusing, a lot less capable, and much more expensive.
I mean, AWS has launched VPC Lattice and then they've marketed
it so poorly that people don't know if I'm making that up or not.
Yes, I actually had not heard of them.
Uh, that is maybe embarrassing.
Now, Nope, this is par for the course.
I thought it was great when it came out and then I forgot it existed, and
then it just goes years without being mentioned by anyone until I encounter.
It's like, oh, right, that, that, that exists.
That's kind of neat.
I should look into it and every time I do, I come away with, or
I could just use tail scale and save myself a lot of heartache.
So I do honestly, on some level, your next Go to Market for
enterprise, you'd just be offering people a free month of VPC latice.
We've actually had that a few times and there's like a comparison
and we're like, can we please be like first while you're doing the
comparison and then you can, you know, install the other ones later.
And they do, you know, they're done with tail scale in like 15
minutes and then they go off and they try to install the next one.
But it, but if they try to install the next
one first, they might never get to tail scale.
Right.
'cause they don't finish.
That's the, that's the dark secret of POCs.
Yep.
You've done a fair number of things that are,
it's hard to even describe what tail scale is.
You, you have tail drop, which is effectively
an end-to-end, uh, file sharing option.
Uh, it feels like you are flirting with
becoming almost a service discovery tool.
Uh, one of the, we have enough service niche in the world, but
it feels like this one makes a strong contention for being one.
Well, I have, uh, we're, we're trying out new versions of the
mission statement 'cause previous ones were too complicated.
I will, I will present a preliminary version, uh, that we've been trying out.
Uh, it is a new layer three for every device everywhere.
It's like maybe too simple.
You have to be a network person to know even what I'm talking
about by layer three, I tried like new internet protocol.
Sometimes people are afraid of that 'cause it's not like IPV seven, but it
does the job of what layer three, the internet protocol was supposed to do.
And let me, let me try to explain what that means.
So like way back in the day when I logged into the internet, I could connect
to any device anywhere that was on the internet by using its IP address.
That is, has not been the case for now decades, right?
It's now gotten to the point where, in fact, the only
things I can really connect to by IP address are.
Maybe my wifi router, if I can remember what the IP
address is, and I'm in my house, uh, or cloud providers
who own like most of the public IP space at this point.
And that's like kind of weird.
That defeats a lot of the purpose of the internet.
Another thing that happened is if you have a, even if you
had that connectivity, imagine you had IPV six rolled out
everywhere, which requires a bit of a big imagination.
But let us imagine.
IPV six was everywhere.
If I switch to a different network, like between
wifi and cellular, my IP address changes.
And now the connection breaks.
Uh, and I actually can't find that device unless I use DNS.
Everyone's best friend, DNS, the thing that is not anywhere
in the OSI stack, but is somehow playing some job, making
some of the layers of the OSI stack work together, right?
So now I'm like dynamic DNS, I'll just update it every time my
phone jumps between wifi and cellular, like not likely, right?
And so like the actual interneting part of
the internet stack does not work anymore.
It's not location independent.
And it doesn't make everything in the world addressable to me, right?
It's actually layer two.
It's just a replacement for ethernet addresses
because every time my interface changes.
The address is a different thing.
It might as well be an ethernet port, right?
And it hasn't done this job that's like missing from the stack.
And so tail scale jumps in there and it's a tunnel, but
it's like, hey, it works the way it's supposed to work.
Like obviously the world has changed.
You don't want everyone in the world to be able to access
you, but everyone I want to be able to access me gets an it.
It know it can find out my name, and I get a fixed ip.
And I'll make it work everywhere.
And it doesn't change when my device moves around.
So tail scale, all the stuff you can talk about.
But the thing that it does is it actually pro produces
layer three of the OOSI stack for the first time in decades.
That's nothing short of magical.
Uh.
It's, it's weird in that because this gets highly technical, highly,
quickly, and goes very deep, but it is stupid simple to get set up.
We were just traveling in France, my wife and I, and she
wanted to access something that was only available from home.
Great.
Hand me your iPad a second.
I didn't even bother to have her set up an account.
I just logged it into my tail net so now she can get
access to my shit posting nonsense if she really wants it.
And suddenly it worked when I turned it on as an exit node.
Uh.
I've also found, and this is what really sparked the
idea of having this conversation now, is with now
I have a test Kubernetes cluster that mostly works.
I have your provisioner that auto the operator that
automatically gives access to any service I put on the thing.
It's got some drama when the nodes themselves
are on the tail net and that becomes their magic.
DNS becomes their resolver.
It tries to pass those out to containers and that becomes a little
bit of a, uh, let's patch cord and s to make it not do that.
But once I do, I can spin up arbitrary containers.
Not have to worry about security, which sounds like a wild thing to say,
but the only place that those things are available is on the tail net.
I'm the only person except for my wife's iPad on the tail net,
and even then I could restrict it down further via ACL grants.
Suddenly I'm doing the thing that a lot of people used to do on the open
internet of, oh, I'm not big enough to find, no one will find this weird port.
I've bound it to only there is security.
It's not just pretend security.
Right.
And that's another thing that like, you know, if it's, again,
IPV six, if it had been fully rolled out today, still wouldn't
solve that problem because it was invented 30 years ago and
there's been 30 years of new problems since then, right?
So it's like time for like a thing past IPV six,
if we could move past it ourselves psychologically.
But like there has to be identity, there has to be
security, there has to be a concept of like, which things.
Are allowed to connect to which other things, not just the dream of the late
1990s of like, you know what, if everybody could just talk to everybody,
the whole world would be happier and we'd have world peace and stuff.
And we sort of learned from the internet that like world peace doesn't happen
when everybody can like chase you around and harass you all day, right?
And so you just need that level of security.
But you want the feeling that we had on the small internet
before, you know, most of the really bad people showed up.
I think that's the right path.
It's.
You have, I keep forgetting this because of course in your case you
have to deal with a, especially with a free way to get started here,
you have to deal with a tremendous amount of abuse concerns on this.
But it's, it's also not traffic necessarily passing through you.
One of the smarter things you've done from pure cloud
economics perspective is you're the coordination central
point, but the actual heavy duty traffic is point to point.
Yep.
Exactly.
So tail scale splits in, in network terms.
We call it the control control plane and the data plane, right?
The control plane is decides like how to distribute the keys, how do
you log in, um, who should be allowed to talk to which other people.
And then it like sends those instructions to every device in your tail net.
And then the devices themselves, uh, handle the data plane, which is
sending the data direct whenever possible directly point to point between.
Between themselves.
So it doesn't cost us anything to transport your data.
And it costs us very little to be the
simple coordination point between the notes.
And this is what makes it extremely scalable.
And a lot of this stuff is based on some of
the original concepts of the internet, right?
It's like, look, it should be extremely scalable.
You can't have like one company that is routing all the traffic for
you, such as at and t back in the day with the telephone network, right?
Like you just, you know, it works, but you shouldn't have that.
You should build a system where that doesn't happen.
Uh, and tail scale is very much, uh.
Moving along those lines and it, it, it is kind of magical, especially
because if you get two devices sitting right next to each other on your local
network, they get direct connections to each other on your local network.
Right.
Almost any other thing will try to beam it up to the internet and
back, which is pointless in situations where they're side by side.
And so if you've got a data center or a VPC filled with containers and they
want to talk to each other, it's really silly to send all those things to
the internet and back to say nothing of like the egress fees you'll incur.
It.
What, what's weird to me is also how effective you are at
routing money to other companies, uh, uh, through tail scale.
I use moad, uh, as we've discussed.
I also pay for next DNS because that's where I do most of
my ad blocking, which makes it super handy when I try and
hit something like a link in an email that gets blocked.
Great.
I could special case it.
Why would I do that?
I'll just toggle off tail scale, hit the thing I need to and turn it back on.
I do that multiple times.
Every day you have become, uh, something I use
constantly, but also almost never think about.
Which is the, honestly, the, the Val Howa of infrastructure.
Yep.
Infrastructure is really tricky.
'cause we have, you know, we're trying to balance, uh, word of mouth.
'cause you want everyone to brag about how they use tail scale.
And simultaneously the best infrastructure
is the infrastructure you never think about.
So it reminds me, I forget the name of this, that this trendy workout.
Uh, campaign from like 10 years ago where like the, the
joke was like, how do you know someone's on this trendy
workout campaign is like, they won't stop talking about it.
Uh, so tail scale people, people love their infrastructure so much that they
will not stop talking about it, which is a very strange situation to be in.
Uh, I did not see that coming when we started the company,
but it's more or less what like drives the adoption of tail.
This
episode is sponsored
in part by my day job Duck.
Bill, do you have a horrifying AWS bill?
That can mean a lot of things.
Predicting what it's going to be, determining what it
should be, negotiating your next long-term contract with
AWS, or just figuring out why it increasingly resembles.
Phone number, but nobody seems to quite know why that is.
To learn more, visit duck bill hq.com.
Remember, you can't duck the duck bill.
Bill, which my CEO reliably informs me is absolutely not our slogan.
Yeah, every time I see weird questions on come through on the AWS subreddit,
which I keep a loose eye on, it's like, oh, that sounds like a tail scale usage.
And sure enough, it's always the first comment someone has there.
Have you considered using tail scale for this, like a sensible person, which
Yeah, exactly.
Yeah.
And you mentioned like these partners that we work with and roading
money to them, like tail scale is increasingly, uh, it's a little among.
You know, in the, in the entrepreneur world, you have to be
really careful with this word, but we are increasingly a platform.
And what is a platform?
It's like the base layer of something that people build on top of, right?
And uh, I was talking to our investors the other day and someone said like,
look, the, the advice, uh, or the most important thing to know about building
a platform, and the biggest mistake almost everybody makes is trying to do it.
Uh, and especially doing it too soon, like almost
no company ever actually builds a platform.
And if you are wrong and you go and build one
anyway, you waste a ton of time and energy.
And so we've been a little bit dragged into building a platform.
I've started talking about last year how maybe
someday tail scale's gonna evolve into a platform.
Form.
Uh, and then this year we made a feature that's called the Tail Net's, API.
So a completely automated way to create a new tail net,
add devices to it, and then spin down the tail net, share
it with other people and stuff, just entirely API based.
And so now we have big cloud providers that are
like, you know what, I'm gonna make my InterCloud.
Connections, just use tail scale in the background
and our customers don't even have to know about it.
And I'm gonna do it all using the tail nets A API, right?
So we're kind of like, well, this is way ahead of schedule now we're a platform.
Um, and I don't even,
can you cheat it under the hood to take specific decisions
on the path traffic takes to get from point A to point B?
Yeah, exactly.
I mean, they're basically, um, well the big, the, the problem space that these
people are mostly in is like they're, you know, lower tier cloud providers.
They provide, you know, the biggest thing is usually GPUs, right?
Uh, at better prices than the big cloud providers have.
And then customers like ignore the prices.
They actually have them for rent.
Yeah.
Or more availability, et cetera, right?
Or the right ones at all kinds of things.
Uh, but then the same customers wanna run the rest
of their stuff in a more mature cloud provider.
Now you've got a connection problem between like, kind of
weird GPU Cloud provider and the top tier provider, right?
And so how do you connect between cloud providers?
Well, it's actually hard.
Almost nobody makes a product for that at all.
Um, these cloud providers, they could tell you like,
go use tail scale, but then you have to go figure out a
third product that kind of slows down their marketing.
So they're just like, you know what?
We will provide the service of connecting you to anything.
Uh, don't even worry about it.
Uh, and they just like set up a tail net and suddenly their VPC
on that cloud is actually connected to the VPC on the other cloud
and it's the right path.
But what I have found that is, so I guess.
Compelling about all of this has just been that over the years it
has, it has solved so many weird problems and I continue to watch
the logos on your site continue to expand, uh, to going from small
companies to mid-size companies like, I don't know, Microsoft.
Yeah, Microsoft, uh, recently got added to our logo list.
Uh, there's a bunch of other, you know, there's subsidiaries
of Microsoft, there's a bunch of other big names.
Most of our biggest names are still not actually in our
logo list because we didn't get logo rights for them.
Uh, people often that is always the way that it works.
It's especially true in the security world, uh,
because security people are like, wait, I don't want to
advertise what our infrastructure is using for security.
That's just like painting a sign on our back.
Yeah.
Do you view yourself as a security product tradition?
Well, so I've ex uh, well, I'm, I'm stumbling on this
because the correct answer is sort of, uh, or Yes.
Well, who, whose cost center is our purchase, this contract coming out of?
Sure.
We're a security platform.
I get it.
Go, go where the money is.
I hear you.
Are you an analyst?
No.
Unless you have analyst budget, then?
Yes.
Yeah.
So tail scale.
I think the best term I heard for it is a mesh VPN firewall.
Right.
Um, and the reason for that is most people who end up adopting tail scale,
adopt tail scale 'cause it solves a connectivity problem that they have
right now, and they just, it becomes the easiest way to connect things.
What's very strange about tail scale, uh, and very strange in
the security world in general, is that when you use tail scale to
solve that problem, you accidentally make your system more secure.
And also the easiest thing for all of your engineers
and people inside your company to do becomes.
The secure thing instead of the insecure
thing, and nobody really sees that coming.
Um, but then once it gets there, the security
people are like, wow, how come I'm not the bad guy?
I'm always the bad guy.
I don't wanna be the bad guy.
Uh, we love tail scale.
Uh, most of the time today, tail scale is not adopted
through the security team because the burning problem
is not like blocking people from connecting to things.
The burning problem is usually connecting to things.
Uh, but you get both at the same time.
And that was like from the very beginning.
At tail scale, usually you have to buy like a
connectivity thing, like a router or a VPN and a firewall.
And they're run by different teams and they fight with each other all day.
Honestly, I found that the most people I talk to the most
who are the biggest champions of tail scale are the ones
that are empowered to do the thing that they wanna do.
It's, oh, uh, the policy is because I said so the end, this
doesn't feel like it's something that's gonna be instituted
top down just because it's not painful enough for the user.
We have a new experimental thing that we're working on, uh, that I
think is really gonna appeal to security teams specifically as buyers.
And I wanna run it by you and like hopefully get
feedback from everybody else who's listening.
Um, you can, you can post my email address or my blue sky or whatever you want.
Um, so people get you say that and yet.
You know, I, I get hate mail.
I've, I've received hate mail at this point.
I, as a CI get hate mail from my own employees.
Um, and so, you know, the, the skin gets thicker over time.
Um, but yeah.
So here's the thing.
Uh, ai, I think we've all heard about it.
People are deploying it.
Um, in their companies, uh, and often carelessly, believe it or not, uh,
they don't always think about all the consequences before roll out ai.
Uh, and yet, some many companies, and some of them we've
heard about, uh, more than others, but many companies
have directives from the top down to roll out more ai.
So the CISO is sitting here and it's like, wow.
Everything you guys are doing is horrible.
Uh, and this is a ticking time bomb.
And I can't believe that I have to say yes to this because
my job is not just to block progress in the company.
My job is to like ensure success or ensure security as much as we can.
But if they say yes, it's like there's gonna be a breach.
And if they say no, they're probably gonna
get fired because they're blocking progress.
Right.
I think a solution to this is when you want to, and, oh,
sorry, I forgot another part of the story, which is that.
When you're bringing AI into the company, that's one thing.
The new trend in AI is this MCP protocol model, context protocol that you
can use to connect your favorite AI agent to your favorite data source,
no matter what it might be, or all of your favorite data sources, right?
When you do that, all kinds of terrible and exciting things can happen.
And if you Google around a bit, you can find examples of like someone hooking.
Hooking.
Oh yeah.
The, the attack vector now is quite literally
telling the computer, trust me bro, in those words,
and it's so, so exciting, the kinds of problems you can have, like
some people hooked at GitHub up to this and like the repo that it
looked at contained instructions to the LLM that then convinced it
to take the rest of the data in GitHub and send it to somebody else.
And he is like, wow, that's, that's a super neat attack.
Uh, as a security person, I can appreciate super neat attacks, but also
like, wow, what are you gonna do to defend against this kind of thing?
Right?
And I think the answer is the LLM has gotta be supervised, just like any, uh.
Any person or any weird thing that you put into your network,
you've gotta have auditability control, acls, identity encryption,
uh, all that stuff that you should always have, that you
actually don't have today when you hook an AI ops to stuff.
Right?
The way to do that is to funnel your AI traffic
into a thing that has the ability to audit log.
Um.
And and control and filter and decide what can connect
to which other things and then forward it on through.
Right?
And of course, tail scale is a connectivity and security layer
that makes it easy to build such a thing and deploy such a thing.
But then you have a really interesting other problem.
And I apologize if this is getting like weirdly deep, but I hope
your audience loves weirdly deep things once you've got a proxy.
Once you've got a proxy that is forwarding traffic from, like it's
acting on behalf of Avery, say, on its way to Salesforce, right?
Avery goes into the proxy.
The proxy then wants to go to Salesforce.
The Salesforce says is like, okay, you're a proxy.
You have a like service account.
What did we do?
Do we set up the service account to have, we have global
access to Salesforce, and then the proxy needs to be trusted
to only give Avery the stuff Avery should have access to.
Well, that sounds like a terrible idea, but.
It can't act as Avery by default, because it's not Avery.
It's running as proxy and it had incoming connection
from Avery that doesn't give it rights to Salesforce.
So you have to have this little interchange
to avoid confused deputy that way.
Yeah, exactly.
So you have to have this interesting interchange where Avery makes
a connection to this proxy, and the proxy has the right to exchange
that, that identity for a token that allows it to access Salesforce.
As Avery with a little note on it that says, by the
way, it's Avery's ai, don't give it too much stuff.
So it's like Avery minus minus.
To do that you can use an OAuth protocol that I won't go into, but
it's like there's, you know, originally when the MCP standard came
out 10 months ago, I think, uh, there was like almost literally
at this page intentionally left blank in the security section.
Uh, since then there has been an improvement where they,
they said actually OAuth should be the way you do this.
And then people started implementing that
and now they're at the stage where like.
It tries to o off to like 10 different things and each of those things
leads you to a click through, uh, to granted permission to do some stuff.
So with tail scale, we have this neat feature where like every connection
that happens on the tail scale network has your identity already attached.
You don't have to click through anything.
It's just like inside your tail net, everything knows who you are.
Every request inherently becomes authenticated.
Exactly.
So the trick we did is we wrote this new tool on top of tail
scale called T-S-I-D-P, uh, the tail scale identity provider.
Uh, it's open source, by the way.
You can look at the, uh, GitHub repository and fork can do whatever you want.
It's only a few hundred lines, and what it does is it's a complete OAuth
server, but the user side is just, I already know who you are, right?
So when you try to access a service, the service redirects you to your
IDP, which says, I already know who you are, and then redirects it back.
No click throughs.
But it's controlled by the ACL grant policy we talked about earlier.
'cause it's just a tool on top of tail scale.
We didn't have to modify tail scale to make any of this work.
It decides which kinds of tokens it's willing to exchange
on behalf of this proxy running inside your tail net.
Right, but this proxy, the TS IDP server, can be
accessible over tail scale funnel to the outside world.
So you can even use TS IDP with any service on the
internet that supports custom IDP or custom OIDC.
So you have this really interesting situation where.
From the very beginning, tail scale is like, I'm not gonna be an IDP.
We're not doing usernames and passwords.
Get outta my way.
That's the past.
Let's live in the future.
Use a real IDP.
You should still do that, but you can use that to get into tail scale.
And after that, you can use TS IDP to connect to everything else.
And this MCP thing means your AI can do the same thing.
Right.
And all of it can be zero click because you
can set a policy on your administrator for you.
Company can set a policy on T-S-I-D-P to decide which things can be zero click.
Right?
And if you're worried about sort of privacy, I know a lot
of people who like use Google, uh, log in with Google are
like, ah, Google's tracking me all over the internet now.
'cause I use login with Google every time I log into a service.
They know every service I use.
Now Google only knows that you use tail scale, right?
Because your instance of TS IDP that you ran, that is open
source, is the one doing all the rest of your authentication.
And so you have access to all.
You're the only one that has access to all that private information.
Even we don't.
'cause it's just a tool, right?
Built on top of tail scale.
And so the combination of all that stuff allows you to like
control your AI access, but it also lets you have zero click
authentication to like everything on the internet if you want.
And it also lets you have zero click authentication to
things on your tail net that don't understand tail scale.
All they need to understand is custom OAuth.
So I think Home Assistant is a really popular one.
Grafana is another one, et cetera.
So I apologize for that monologue.
I'm still working on the short version.
No, please.
It's, it, it's a, it's a fascinating approach
because we are definitely in a post network world.
It used to be that once upon a time you had breaches where
I'm gonna go and I'm going to go and take things out of your
system and then send it to a different system somewhere else.
Now you can do all of that just by hitting the same single endpoint.
That's just the AWS Control plane.
And it just a question of what the content of those requests are.
So you effectively have to, I don't think we call it this anymore, but
you need to, uh, man in the middle, everything that is being passed
through for deep packet inspection, which in turn then becomes, if you
can see all the payloads, well, you now have a central point of attack
for that, but people have already accepted you in a security facing role.
I think that.
It is a more novel approach that is likely to get further
than the current security posture, which is putting the
No seriously, bro, be secure in all caps in the system.
Prompt.
Yeah.
Well, exactly.
And the best thing about this MCP proxy
thing, first of all, you can have it right.
We have a little default one.
It's open source.
Again, you can like build your own if you want, and it can run on your
private tail net and it can access stuff that's on your private tail net.
It can be accessed by your favorite LLM that might
or may not be running on your private tail net.
And also it can access things outside your private tail net so there's no people
coming in trying to beat on your MCP server to find the security holes, right?
It's only the content that matters.
And for that, you can have something filtering the content and watching
what's going on to make sure the AI doesn't go wildly off track.
Yeah, I think that's
the, that is the right path.
It's, it's part of a defense in depth approach.
Exactly.
We're aiming for this like, again, convenience, where like the
easiest way to roll out AI in your company is the tail scale
way, and also coincidentally, it's gonna be way more secure.
If we can get that, then I think we'll really, like, we'll be on the.
The two problems I can see, you're gonna have one, you use
the Salesforce, Salesforce example, but everything has to
start supporting this on some level at an application level.
So they need to support OAuth.
They don't need to support any of the rest of the stuff.
And that's what's really neat because everybody who makes an MCP server has
to support OAuth now as part of the standard and like where APIs were kind
of hard to get access to before the trend is that, look, everyone's gonna
be mad at us as a vendor if we don't support OAuth for getting API keys.
Right.
As long as you have that, all of the rest of this magic is
happening behind the scenes, the gateway has to understand all this.
Tis IDP and everything, everybody else just sees to know server.
Yeah.
The, the other challenge that you're gonna have, and this is trivial of
course, is you have to come up with a few, uh, reference implementations of
this that are basically click, click done, and to show folks how it works.
They can modify to their own approach.
But historically, my big problem with.
Uh, early with early stage products is the documentation's not there.
You've gotta basically read the code, come up from first principles,
how you want to tell it to actually do the thing that you do.
A little bit of documentation goes a long way and not for nothing.
Increasingly, that documentation is being written for LLMs
so that they can then explain how to do this to folks.
So there's a, there's a bit of a lead time.
It has to be absorbed into the models before it starts spinning out.
Yep.
Yeah.
The best we have right now, uh, we have a, a YouTube personality
that works for us that runs the tail scale YouTube channel,
Alex, and he's got at least one video about T-S-I-D-P.
Uh, that's from before we added this MCP layer, but it's actually
pretty well done, was like many, many people in their personal
tail nets are already using T-S-I-D-P for their own stuff.
Uh, so I think there's gonna be some, some growth there.
But yeah, we're gonna have to document it.
We're gonna have to do all that work.
This is all pretty, pretty early stage, but we're really, we're interested
in like talking to people who think this is gonna be interesting to
them and like kind of working with them on making the product better
and also integrating into the open source world, uh, because tail scale.
Personal plan is free.
Um, and it's unlimited, essentially, uh, unlimited time.
Lots and lots of devices.
You can do all kinds of stuff with it.
And it would be nice to make people, or have people who are using this
in their home lab already, they can take advantage of this thing as well.
Oh yeah.
I, I do a lot of testing in my home lab for this exact sort of thing.
I, I still haven't gotten quite to a level of comfort where I'm
putting production nodes independently On the tail net, I tend
to use subnet routers, and, and that is for now the way that I
approach it, just because it, it feels like taking anything into
a critical path, past a certain point, has risk attached to it.
That's how we built it and that's how it works for now.
If I were doing it today, I don't know that I would be
as cautious given the conversations I've had since then
with customers who are working with it in that way.
Yep.
There are, there are some very big name customers, some of which I can name
and some of which I can't, that are like all in on, like, we're gonna run
Kubernetes in every single pod, in every single cluster, in every single store.
Um, and turning 'em like crazy 'cause that's what Kubernetes does.
Uh, and they, they seem to be pretty happy.
It means we have to have like pretty high up tam on our control server
Tail scale is designed so that even if the control server went down for.
A while in, in fact, it could go down for hours.
The data plane keeps on working.
So there's only certain things that stop working if
the control plane is like out of touch for a while.
So you have this like, pretty high level of resilience that people
don't expect, and it comes from us not routing your traffic.
Uh, for the most part,
that that is the bridged cross.
And you've, you've hit a point now where
there's enough of a community around tail scale.
That if someone's trying to do something that no one else has really done
before, it is no longer likely that they're doing something correctly.
I, I don't mean to be unkind, but in the early days, I
would, I was talking to your team near constantly with what?
How do I do this thing?
Oh, we hadn't considered that.
Now, whenever I ask any of those questions that come up like, oh, here's a
giant blog post on how to do that, or, here's the GitHub issue where we explain
exactly how you're holding it wrong, and so on and so forth, which is just.
It's, it's a, it's a maturing of the product.
Yep.
Yeah.
We've been putting a lot of work into maturing it.
I think one of the hardest things as CEO, uh, is just con
convincing everybody to not build a. Everything they want.
And just like, let's focus on refining the core.
Let's do everything we can to run this business so
that the core gets better and better and better.
And that's how we're gonna make money.
Not like building tons of stuff on top.
Uh, which I know is a pretty unusual, especially in
the security world, is not the normal way to do it.
The normal way to do it's collect.
Uh, I know it's like collecting Pokemon cards or whatever.
Well, I need a DLP and I need this and I need this
and I need this and I need this and I need this.
And now you can buy it from one vendor and
it's gonna be a collection of like sort of.
Halfheartedly integrated tools, right?
And tail scale is like, look, we're not that.
We have this one thing, it works super well and it's gonna work
with all the other stuff you buy from other people, but it means
we spend all our time just like, you know, writing docs like
those or fixing the bugs that led to the need for docs like those.
It's really neat.
Any, any last words on what we can expect in the somewhat near future?
Anything fun and exciting?
Uh, coming down the pike, which I know is a weird thing
to say about a networking infrastructure tool, and yet.
Um, I think the, the two most interesting
things are happen are that are gonna happen.
One of them is more and more stuff is gonna be buildable on
top of tail scale or include tail scale as an option in it.
So we're starting to see more and more things like, Hey, if
you run my program, it's linked with the tail scale library.
Just paste your offkey here and that thing is just going to work.
Um, a similar one is, I think.
Think, I don't know if we've announced it or not, we're gonna announce it.
If not, this is the announcement.
Um, the workload identity feature that allows, if you're
using tail scale with GitHub actions, for example, to
just like, not even use Offkey because you can set it up.
It's like, oh, this is your account on GitHub.
I believe GitHub when it says it's running under this account.
So now everything just has access to your tail net automatically.
That's super slick way to do it.
You don't have to manage rotating off keys and stuff like that.
And I guess the third one is, uh, for direct connectivity.
You know, life is not always perfect.
Sometimes firewalls are weird.
Um, so we have this new thing called, you
mean there are times where they're not.
Well, some are weirder than others.
So we, we get through almost all the weird firewalls,
but there's some extremely weird firewalls out there.
Uh, we have this new thing called the peer relay also in alpha, but if
you're interested in choir within, uh, should be in beta sometime soon.
But you can have really access if you, if somebody asks, um.
It allows, basically, if you remember the old days of Skype and
Supernodes, it allows you to build supernodes that will route the
traffic in situations where direct connections are not possible.
So you can still get full speed if you put your Supernodes in
the right places, um, including behind a firewall if you want.
So then.
Even when things can't manage to get direct connections, 'cause
your internal firewalls are too weird, if they can connect to the
supernode behind your firewall, you can still avoid the egress traffic.
Uh, so this is something that our biggest customers
with, of course, the weirdest firewalls and the most
firewalls, uh, are gonna benefit from humongously.
I would love to hear the story about which firewalls
that are doing this and how they're configured.
'cause that is such a rare occurrence in the modern era, but,
oh yeah, we actually, we actually sponsored a patch to free BSD to finally
fix this problem, 'cause for a while, free BSD, any free BSD based firewall.
Of course, it's pf.
Yeah, well, uh, it was, it's in, it's intended
as a security feature to be blocking this stuff.
It just turns out when you do the whole like, decision tree, it turned
out that didn't increase security at all and just made everyone's
life miserable and in instead of, uh, of, 'cause it makes it so
secure that to get anything done, people start rolling out UPNP.
Uh, and UPMP is never a good choice.
Uh, security wise, and yet it's the only workaround to this problem.
So we finally convinced them of this.
We sponsored free BSD to like, Hey, can you at least make it a flag?
Why does AHI take up all my CPU core?
Yeah, yeah, yeah.
So we made, uh, now there's a flag and I think the flag is now the default.
Just not be silly, but there are a few other firewall vendors that
are doing the same thing, but I'm hoping we can talk them out of it.
'cause it's actually a relatively simple, it's called a
hard net versus a. Easy net in tail scale terminology,
and they make their hard net hard for like it turns out.
No good reason, and it's avoidable if you change your code just a little bit.
But unfortunately, sometimes it's our competitors making
the firewall so they don't always super eager to do that.
Yeah,
I, I really wanna thank you for taking the time to speak with me.
If people wanna find out more, where should they go?
Uh, well, there's tail scale.com.
We have a blog.
Uh, sometimes I post in the blog.
I also have an account on Blue Sky.
I have a little used account on the system for Blue known as Twitter.
Uh, and I have my own blog on appware.ca,
which has been a recurring presence on the newsletter.
Whenever you put something interesting out there,
and we'll put links to all of this in the show notes.
Thank you so much for taking the time to speak with me.
I appreciate it.
Thank you very much.
It's
always a pleasure.
Avery Penan, CEO, and co-founder of Tail Scale.
I'm cloud economist Cory Quinn, and this is Screaming In the Cloud.
If you've enjoyed this podcast, please, we have a five star review
on your podcast platform of choice, whereas if you've hated this
podcast, please, we have a five star review on your podcast platform
of choice along with an angry comment that isn't going to post properly
because you once again have misconfigured your crappy firewall.