Subscribe
Share
Share
Embed
We're back with an exciting new episode of Cloud Native Compass, and this time we're diving deep into the captivating world of eBPF. Join Laura Santamaria, David Flanagan, and special guest Liz Rice as they unravel the mysteries and explore the incredible potential of this powerful technology!
In this episode, you will learn:
1. The two parts of eBPF: Discover the kernel program and the user space interaction that make up the magic of eBPF.
2. Programming with eBPF: Explore the different options for writing eBPF programs, from bytecode form to higher-level languages like Rust.
3. Compilers and SDKs: Learn which compilers, like clang GCC and the Rust compiler, support eBPF bytecode and how SDKs can make your interaction with eBPF programs easier.
4. The Evolution of Packet Filtering: Trace the history of packet filtering, from its humble beginnings to the powerful and versatile capabilities of eBPF.
5. The Widespread Adoption: Uncover the sudden rise in eBPF's popularity, its impact on observability and performance, and the role it plays in modern networking.
Now, for a fascinating fun fact from the episode: Did you know that eBPF is now considered Turing complete? That's right! With its combination of features, eBPF has surpassed its humble start as a packet filtering tool and has become a full-fledged technology powerhouse.
In this episode, you will learn:
1. The two parts of eBPF: Discover the kernel program and the user space interaction that make up the magic of eBPF.
2. Programming with eBPF: Explore the different options for writing eBPF programs, from bytecode form to higher-level languages like Rust.
3. Compilers and SDKs: Learn which compilers, like clang GCC and the Rust compiler, support eBPF bytecode and how SDKs can make your interaction with eBPF programs easier.
4. The Evolution of Packet Filtering: Trace the history of packet filtering, from its humble beginnings to the powerful and versatile capabilities of eBPF.
5. The Widespread Adoption: Uncover the sudden rise in eBPF's popularity, its impact on observability and performance, and the role it plays in modern networking.
Now, for a fascinating fun fact from the episode: Did you know that eBPF is now considered Turing complete? That's right! With its combination of features, eBPF has surpassed its humble start as a packet filtering tool and has become a full-fledged technology powerhouse.
Creators & Guests
Host
Host
What is Cloud Native Compass?
The podcast that helps you navigate your way through Kubernetes and Cloud Native trends.
Welcome to Cloud Native Compass, a podcast to help you navigate
the vast landscape of the Cloud native ecosystem. We're your
hosts. I'm David Flanagan, a technology magpie that can't stop playing
with new, shiny things. I'm Laura Santamaria, a forever
learner who is constantly breaking production. eBPF is
turing complete and can be written in Rust. So obviously,
I'm sold. That's just two of the things we learned today's
episode. Want to learn more? Contrary to not so popular
belief, eBPF is not some secret Berkeley green project
that's out there to eat your refrigerator, but it is a gateway to
becoming a Linux kernel maintainer. If you're curious about
what eBPF is, why it matters, how to
pronounce it, and how badly you can break your kernel when trying to learn it,
this is the episode for you as we talk with Liz Rice,
the author of Learning eBPF and Chief Open
Source Officer at Isovalent. All right, thank you.
Liz, could you please say hello and tell us a little bit more about you?
Hi. Yeah. So my name is Liz Rice. I am Chief Open Source
Officer at Isovalent, which is the company that originally
created Cilium, and a lot of people will have
heard of Cilium being based on eBPF.
And earlier this year, I published a book about eBPF called Learning
eBPF. I feel like I may have answered more than
one question in one go there. Hey, no, that's fine.
That's fine. Little bit of context. Just us. Perfect. Yeah. Good
context, considering we're about to talk about eBPF and go into a little
bit more detail there, so yeah. Awesome. All right, well, let's
just know not everyone is familiar with
eBPF. So could you give us the TLDR? What do people need to know to
understand the rest of the conversation today. So they don't need to know what
it stands for? It stands for Extended Berkeley Packet
filter, but honestly, forget that it doesn't really help
because it does a lot more than packet filtering now. So we
tend to say the acronym doesn't really mean anything anymore.
What it really is is the ability to run
programs within the kernel, within the operating
system kernel, so we can dynamically change the way that the
kernel behaves by loading these
eBPF programs. And I think when I say that,
I have to make sure that people really know what I mean when
I say the kernel. The kernel is the part of the operating system
that interfaces between our applications and the
hardware that the processor and its
peripherals. So if you are writing to a
file, you're doing anything over the network, writing anything
to screen even allocating memory, the kernel has
to get involved. Your application can't do it directly. It has to
ask the kernel for help. And the kernel is also
coordinating all the different processes that might be running on the machine. And
that means the kernel is involved whenever you're doing anything
interesting, really. So it's a really great
place to write things like observability tools
and security tools. And we can do that with
eBPF, and we also get to customize the way that the kernel
behaves for the things that it takes on, things like the networking
stack. We can modify the way that behaves with eBPF. So
it's really powerful and a really interesting way to
instrument all of your different applications that
are running on that one kernel. Nice.
And here I was hoping you could I was going to say, here I was
hoping you could actually explain why it said Berkeley in the middle of all of
it, but that's a whole more than anything else. The
original packet filter paper
was written by two people whose names I can't quite
remember right now, but they were at Berkeley at the
time that makes precise
paper that it all originates from says, like, Lawrence Berkeley
Lab or however. That'S know, just for
fun for somebody like me, who's like, why does it say
Berkeley? Okay, anyway,
so I'm curious then, did you join
Isovalent because you were really interested in eBPF, or are you now really interested
in eBPF because you've joined Isovalent? Like, what came first?
No, I really got interested in eBPF the first time
I heard of it. I saw Thomas Graff, who is the
CTO of Isovalent, talking about Psyllium and
eBPF at DockerCon back in 2017. And at the time I
thought, well, that's pretty interesting technology. And at that
point, it was really cutting edge in the kernel.
It wasn't available to most people in production. It wasn't available to hardly anybody
in the Linux distributions that they were using back
then. But I thought, this is a really interesting
technology, and I'd kind of kept an eye on it. A
couple of years later, I started working on a project that was sort of
using Ubpf, and I was also, as part of
learning about it myself, I was going out and doing talks.
I find that the best way to make sure you really understand it is to
try and explain something to somebody else.
So, yeah, I'd started doing talks about eBPF as
well, and through that, actually got
invited back to the eBPF Summit in, I guess,
2020, which I surveillant put
on sort of on behalf of the eBPF community.
And there was so much really cool stuff going on
in the world of eBPF, and particularly at I survealent,
it turns out I hadn't realized before that summit just how much
I surveillant had been involved with eBPF right from the get go. So
Daniel Balkman, who's one of the three maintainers of
eBPF in the kernel, was one of the early engineers at
Isovalent. And he's still, you know, we're
so embedded in the way that eBPF
has developed over the years. I really do get
to work with the people who created it and the people who've been
using it and had the vision for building things like Psyllium.
So, yeah, I joined Isovalent because it's just
absolutely full of really cool people doing really fun things with
eBPF. Awesome. Makes sense. So
you started experimenting with eBPF in 2017.
We're now in the latter half of 2023, which just
seems absurd to me now, but over those years, you've seen
the adoption grow, as we all have, especially across the industry and even the
CNCF, with projects like Pixie and Falco and of course Psyllium as
well. Why has that adoption grown
so quickly for a relatively niche? I don't know. Is that a niche technology? I
think it is. Why are people I don't
know. It seems like it's kind of everywhere. Yeah, it's one of those things
that, I guess, expertise in it
is pretty niche, but a lot of people are using it without really even
knowing that they're using it. I mean, there's probably people using
Psyllium who don't realize that it's based on eBPF. Certainly a lot
of people will be using things like TCP Dump and never
really sort of thought about EBPs. And that's fine.
There's so many really powerful tools that have been
built. I think
things that Brendan Greg has
popularized in the kind of
2017 period, he was already out there talking about
how Netflix were using eBPF for
observability purposes, for tracing, for
diagnosing, and then improving performance
issues and really
popularizing the power of
eBPF. The reason
why I think there was that kind of sudden
upturn in adoption is
the fact that the level of
eBPF support in the kernel had reached a point,
I think around the 418 kernel version, around that kind
of time frame, is where you really start getting it's sufficient
eBPF support to do really interesting things. The more
modern your kernel is, the more additional capabilities in
eBPF and probably lots of other areas of the kernel as
well. But there was this real turning point
when I would say particularly when Rel was
probably the last of the distributions to kind of
it's always relatively cautious about upgrading
to new versions of the kernel. And at the point where really all
of the distributions were using a modern enough kernel, that meant
you could just deploy these eBPF based tools in production
regardless of your distro. And I think that really made a huge difference to the
adoption. Yeah. Awesome. So you
mentioned the Berkeley packet filter, and for the people that are not
aware, it's like a networking thing that allows you to do Iptables like
stuff. I'm not trying going to go into it in any more detail than that
because I'll make an absolute mess of it. But it does networking stuff, right? It
blocks packets, it Reroutes packets, it does other stuff. But Evpf has kind
of grown beyond that. Now we're seeing it used for a whole
variety of different technologies like Falco. And
is there like what's the right way to phrase this question? Really? I should have
had it prepared. But why has it extended beyond this? Why does it have these
new capabilities? What is it enabling within the kernel for people? Why is it
interesting to you and to others? I know that's a very broad
question. Yeah. So I think the original idea
of packet filtering was to be able to look at each incoming packet
and make decisions about what to do with that packet.
And I think in the very first place, it was really just am I
interested in sort of seeing this packet? Maybe I want to
filter packets that are going to a particular port so that I can
count them or something like that. So it was making
fairly simple decisions about
what to do with these packet filters. The
extended part involved, I
think, a few different trains of thought. One was the idea that if
you extended this sort of relatively small
instruction set that could be used to examine
packets, if you turn that into something a
bit more kind of like a virtual machine
instruction set, if you look at BPF Bytecode,
it's very reminiscent of machine code.
It's all about registers and loading values into
registers and comparing them and jumping to other instructions.
It's very similar to machine code. So there was this
idea that having a virtual
machine in the kernel could allow you to do all sorts of interesting
things. There was the idea that maybe you could attach these
programs to other points in the kernel, not just to incoming packets,
but you could make decisions or change the
behavior at other points in the kernel. And I
think the last major thing that distinguished extended from
its predecessors is what's called eBPF maps.
And Maps are these data structures that you can access from
within an eBPF program and you can share them between eBPF
programs, and you can also access them using system
calls from user space. So it's a way of
exchanging information between user space and eBPF programs
or between multiple different eBPF programs.
And all those things kind of combined has turned out
to be really powerful to the extent that one of my colleagues
recently did a talk at one of the kind
of Linux kernel developer conferences where he showed that
eBPF is now turing complete, which is
pretty cool. A question just on that, right?
Because the talks I've seen from yourself and others in this
space, when you talk about eBPF, one of the things that's always mentioned is the
fact that the eBPF sandbox the virtual machine can and
shouldn't never crash. Is that
still true even with the ability for eBPF programs to communicate with each
other and with user space programs? Yeah, exactly. So
the reason we're able to make that claim is because of
a thing in the kernel called the eBPF verifier. So as
you load a program into the kernel, it goes through this verification
process, which is really analyzing all the possible paths through the program
and ensuring that, well, first
of all, it will run to completion.
A long time ago, that used to be no loops at all. Now that's
been kind of improved and optimized and you can have loops.
It's checking for things like there's only a limited
set of what's called helper functions that you can make from an eBPF
program and the set that you can call depend on really
the event that triggered it. So if you were being
triggered because a network packet arrived, then you can
call helper functions that are related to looking at that network packet.
But you can't, for example, ask for, well, what's the
user space process associated with this packet because there is no
user space process associated at that time. Whereas
if you were in an eBPF program attached to
a user space program making a system call, then you
absolutely could ask a helper function to give
you the process ID. So the Verifies checking that all this sort of
contextual helper functions are being used
appropriately and that memory access is safe,
that if you're going to dereference a pointer, you have to
explicitly check that it's not nil before. You do so
because dereferencing a nil pointer. If anybody has ever
written a C program, they will have crashed their C program by dereferencing a null
pointer. I guarantee it. Yeah. So the
verify is really just checking that that program is
safe to run. Safe in the sense that it can't crash the machine, that
memory access is safe. Of course it can't
tell the difference between maybe I'm
a legitimate networking packet that's a
legitimate Evpf program that's filtering network packets,
maybe I'm protecting against DDoS attacks, or
maybe I'm a malicious Evpf program and I am
just throwing away packets for fun. The
Verifier can't tell the difference between those two things. But when we talk about
being safe to run in this context, we really mean it's not going to crash
or hang the machine. Okay, cool. And I think worth
pointing face on here. Yeah, I've got my thinking face on.
Sorry. Go ahead. Go ahead, Liz. No, keep going, keep
going. Well, it's the thing that really
distinguishes eBPF from writing a kernel module.
So you always could extend the kernel. Always.
For a very long time you've been able to extend the kernel by writing kernel
modules. But people are pretty reluctant
to install kernel modules because
if there is some kind of bug in it, if it does crash, it's going
to bring down the whole machine and there's no kind of safety
net like what the eBPF Verifier is bringing. Makes
sense. So, out of curiosity, obviously we're talking a
lot more in depth kernel kind of things. And obviously
you referenced if you've ever developed a C program, you know what this is like
in machine code and things like that. Who would you say are probably
the most common user that you encounter or
like the people who are really using this the most. Who
do you think this is the. Most relevant for, I guess, right now. So
lots of people will use eBPF
through tooling that's built on
david mentioned. There's various different projects in the CNCF.
There's all the BPF Trace
and the BCC family of tools that people can use on the command
line to do observability. There's lots of
different tools that have been built on
eBPF as a platform. And I think for the vast majority of people,
that's how they'll really experience it. They'll use things like
Psyllium or Falco
or Pixi and they may be
interested in the fact that it's using eBPF, but they don't actually
have to get involved in the details. Which turns out to be a
really good thing because I'm the sort of person who really wants to
understand how the sausage is made. I want to
kind of get inside and get a feel for how is this really working?
You can learn about eBPF
programming. It's relatively easy to get
things like a hello world or some basic
networking capabilities running in eBPF
programs that you've written yourself. But you do quite
quickly start hitting the point where you're
interacting with kernel data structures. And at that point you kind of need to
understand what those data structures represent and what the effect of you
changing them might have. So it does quite
quickly turn into kernel programming. So I kind of say that on
the one hand, most people just don't need to know anything about the details
of it at all. But if they're interested, it's
relatively accessible for people who are comfortable with
programming to kind of dip your toe in.
And then if you really want to become an expert eBPF
programmer, which I by no means consider myself at
all, that really does start to require kind of
kernel expertise. But fortunately I work with lots of people at
Isovant who have that kernel expertise. So would you
say this is the gateway to actually becoming a Linux
maintainer? Honestly, yes.
I've not done it, but it never crossed my mind that I would ever
make a contribution to the Linux kernel. But now I've kind of in that world,
I sort of start to think if I had another 25 hours in.
The temptation is high.
The temptation is high to go get involved. I understand.
I've asked my one obligatory question, David, now it's your turn.
Like that. I like that discussion, right? Because it's one of those questions
I do a couple of talks that touch on eBPF, right? I don't go deep
on it because I'm not that smart, but I always do the same demos and
it's the Iovisor Bpfcc tools
demo specifically, I show off execsnoop
and open. You know, in the SRE
platform DevOpsy world. It's quite interesting and
important from a security perspective and an automation perspective to be able to show when
certain sensitive files are opened on a disk and eBPF makes that really
simple. And another really cool demo is just by using Execsnoop you
can actually monitor for pseudo and setuid bit one
binaries on the machine. So when people elevate their privileges, you can get
notifications for that kind of stuff too. And the question is always like
how much do I need to know about eBPF to then start doing tools similar
to that. And then you show them the source code and it's like 20 lines
of Python. It's not a lot to do these kind of things. And I think
that's because I feel like people can start to build
eBPF programs using tracy's without going deep into it in the same way
that with containers, we can all run containers on our machine. But you don't really
need to understand what a control group is or a namespace is anymore. And I
feel like eBPF may make that same transition, probably already has made that
same transition. So I'm going to flip that around a little bit and throw the
question to Liz. People do see these demos, they listen to
this and they're like, OK, eBPF is really cool. What are the languages
and the SDKs that they can go and start to work with right away to
experiment with the new tech? Yeah. So you kind of have
to answer that question in two parts. There's the
actual eBPF program itself that's going to run in the kernel. And
then there's the user space part that might interact with it. There are
some occasions where you don't even need a user space part. So for
example, if you're doing networking
functions, sometimes they
don't need any kind of user space interaction because you can just load them into
the kernel and they can do what they need to do. But usually
we're going to have both these parts
for the kernel. The program is going to be
in ultimately it's going to be in eBPF
bytecode form talked about there being these bytecode instructions
that look like machine code. You could
just write the machine code by hand. Apparently there are people who do that,
but for me I would rather
write in a slightly higher language than that. And
the languages are restricted by being
they have to be able to compile down to bytecode.
So the compilers that support it right now
are clang GCC, both of which can compile
C and also the Rust compiler.
I'm not aware of there being other programs that support
BPF bytecode as a target.
So yeah, C or Rust
really become your choice there. There is a little bit of a
caveat in that. There is this project called
Iovisor BCC. David mentioned the tools
and things like Opensnoop and Execsnoop that come from that
project. And BCC gives
you some friendly kind of
macros such that you can write your code in a sort of
hybrid of Python and C. And it
takes care of a few things for you from
your C program. But then there's the user space
side of things and there you have a much wider choice. I
mean, really, you're not restricted
at all, except that you probably want some SDKs that will make system calls
for you and make it easier for you to interact
with the eBPF program through that syscall
interface. There's
a go SDK there's a rust
SDK in fact, there's a couple of Go ones
and there's a C one, which is probably, I would say
today, the most widely used, called libbpf
Psyllium uses Go. We have a Go
eBPF library, but I think a lot of the
projects outside of that are probably directly using
libbpf. Yeah, you said one thing there that I completely disagree
with, and you said you have a choice between Rust and C. That's not
a choice. I knew this was coming. I knew this was coming. I knew it.
Of course. Talk about rust. I'm going to just break in before he gets going
on it. It occurs to me
there was the mention of when
containers came around and things like that, things kind of change.
And as there are more and more languages that people are familiar with that you
can compile down to the eBPF bytecode.
How often do you find people getting into trouble like they used to do
when containers first came around because they didn't quite know what they were doing,
but they kind of got it enough to get away with it. So how often
do you find people getting in trouble and how do you get them out? That's
always my question. How do you troubleshoot this thing? Especially when
this is like kernel level and you can really mess things up
fast? Yeah, I would say that there's probably
two major ways that people get caught out.
One of them is around the kind of the tool chain
and installing things that are compatible with each other,
because every kernel version has different Evpf
support. And then you need maybe your user space libraries
that maybe are or aren't compatible. And different distributions
of Linux might have different
versions of different either Lib, BPF or
Tools, things like the BCC tools. Or
particularly, there's a thing called BPF Tool that you can use for managing
BPF programs and making sure that
you've got a compatible set of
the packages, the source code, the kernel, the tools that
you want can trip you up in numerous
different ways. And the other thing that catches people out,
once they got everything installed and everything seems to be compatible, and they've started
compiling some code, and then they go to load it into the kernel and
they hit verification errors.
And I would say over the last
few years, there's been a lot of improvement in the
kind of information that the Verifier gives you about why it's
objecting to whatever it is it's objecting to. But
there's a blog post somewhere that describes the Verifier as a
fickle beast. I think it is.
All right, so, you know, we've covered a lot about
EBPs so far, and just kind of to understand the
landscape right now, it's heavily used for networking. selium has gone all
in on eBPF, even towards the service mesh angle
now is available and even now have Tetragon going for the security
angle and trying to help people with you know, we've seen like
pexi and Falco to do more security and monitoring automated
observability, all of this
as I don't know if there's an eBPF maturity curve right. But as
people start to do more of it and the skills become more aware or
people are using it more, will eBPF creep into
our day to day application code? Do you see people using eBPF
to write their CMS or their
proprietary applications? I don't know what those use cases are. I don't know if they
exist. But will Evpf become more than what it is today? Which is a bit
of a forward thinking question, but maybe
it's really interesting. To sort of speculate about what you could do and
also what would be useful. But I think one interesting parallel is the
way that networking
capabilities that used to be in user space
have migrated into the kernel TCP
stack. I am old enough to remember when that was more
commonly in user space, you'd use
a TCP library. And now
we just expect the kernel to take care of that. And I think
what eBPF will allow us to do is to gradually
move more of that kind of functionality into the
kernel, but in a way that doesn't require everybody to take
the leap at the same time, because we don't all have to be running the
same Evpf programs. So I think something like
Service Mesh is a really great example where
Psyllium as a CNI networking component
for Kubernetes is in this really great sort of position in the kernel
to be able to pick up network packets and put them where they
need to be and observe them and report on them
and do kind of security related operations on
them. All of which are very much the kind of things that we expect from
a service mesh today. We can't do
everything in the kernel. I mean, theoretically I
think we could, but in practice, think all the kind of layer
seven operations. We're using a user space proxy to do
that. We're using Envoy to do that. I
think over time, I expect that in
some number of years time, all of that code will
be in the kernel. But maybe Kubernetes will be in
the kernel too. Who knows, maybe that's kind of future.
If you fancy rewriting Kubernetes, perhaps we should all do it in
Rust in Evpf.
David's looking like he might actually do that.
Yeah, no, he might. I
am not personally writing any Kubernetes components in Rust, but people
are exploring that these days.
You never know, right? I don't trust that
statement that you're not doing it.
I'm not. I don't have time. I'm too busy talking to you.
All right, well, is there anything else, Liz, you
would like us to throw at you to ask before we wrap this up? Anything
that's just sitting on the tip of your tongue waiting to be said? Sorry. Yeah,
no, there is one thing I would like to mention, which is the upcoming eBPF
Summit. Because if people are interested
in hearing more about what's going on in the kind of eBPF
community, learning more about how it's being used,
seeing some of the kind of interesting directions that people are
going with it, and learning more about the future of eBPF
itself. eBPF Summit. It's online, it's
free, it's a virtual conference. This is going to be, I think,
the fourth time that we've held it. It's on September
the 13th. And yeah, come join in. If you'll go
to Evpf IO, you'll find a link to the summit
there and yeah, it's always been a really
nice kind of community feel event, so I'm really looking forward to
it. Excellent. All right, if you want
to shamelessly plug anything else yourself, Azovalant,
anything else, feel free to mention it now or forever hold your piece.
We'll make sure all the links end up in the show notes as well. I
guess it would be remiss of me, not to mention learning Evpf.
There you go. Available either know, if you want the PDF version,
you can download that from Isovalent.com or you
can order it from your favorite local bookseller or
get it from Amazon if that's your bag
shop local. Exactly. All right,
well, thank you so much for your time. Pleasure. Thank you, Liz. Thanks for joining
us. If you want to keep up with us, consider subscribing to the podcast
on your favorite podcasting app or even go to Cloudnativecompass
FM. And if you want to to talk with someone specific or cover
a specific topic, reach out to us on any social media
platform. Until next time, when exploring the cloud native
landscape. On three, on
3123. Don't
forget your compass. Forget your compass.
Bye.