Long story short, I got involved in open source with lots of great projects,
including the Dart Project and Chromium and Chromium OS and worked on
particularly Chromium OS where I was building OS images and stuff for fun.
But what was interesting was I was very young.
I was 12 years old.
And, you could not have a Google account then legally.
I think it's still the case that if you're under 13,
you're not legally supposed to have a Google account.
Yeah, that's always stopped, kids.
You must be 18 to view this website.
Oh, okay, never mind.
Welcome to Screaming in the Cloud.
I'm Corey Quinn.
I'm joined today by Alex Zenla, who is the founder and CTO at Edera.
Alex, thank you for joining me.
Yeah, thank you for having me.
I'm very excited to chat today.
This episode is sponsored in part by my day job, the Duckbill Group.
Do you have a horrifying AWS bill?
That can mean a lot of things.
Predicting what it's going to be.
Determining what it should be.
Negotiating your next long-term contract with AWS.
Or just figuring out why it increasingly resembles a
phone number, but nobody seems to quite know why that is.
To learn more, visit duckbillgroup.com.
Remember, you can't duck the duck bill, Bill.
And my CEO informs me that is absolutely not our slogan.
Let's start with the, with the basics.
What is an "Edera" and what might it do?
Great question.
So Edera is Italian for "Ivy." the, the, that's where the name comes from,
and at Edera, we build, products that help you secure your infrastructure.
And it's secured by design fashion, but also give you great efficiency and,
allow you to kind of utilize your, your infrastructure more efficiently.
So we, we started with a, product called Protect Kubernetes
that provides virtual machines, even without hardware isolation.
So that you can run virtual mach-containers inside virtual machines,
even with software isolation might have heard something like gVisor.
We basically do what gVisor does, but we do
it more like Kata does than what gVisor does.
Yeah.
So when I take a look at the space, I, I am by no means a security expert.
I'm just noisy.
And while those things can look alike, in reality, I tend to bias
for trusting people who are better at these things than I am.
And when all the majors come out and have been saying for years that
the container level is not a sufficient level of security isolation as
far as boundaries go, I've tended to take that relatively seriously.
So in the workloads that I'm used to, Okay, yeah, you're going to be
running a bunch of Docker containers within a host that you control.
The multi tenancy happens at the hypervisor level in
most cases, or if you're using it somewhere else in AWS's
universe, Firecracker tends to be their expression of this.
Who is the target customer for this?
Yeah, so the target customer is someone who wants to get that isolation
that Firecracker or something like that could provide, Kata containers
being the most famous one, but does not want to pay for the expensive
metal instances on AWS that are required to get nested virtualization.
So the biggest challenge with deploying virtual machine based
container isolation is that hardware requirement of virtualization.
It turns out that interestingly, as you may know, in AWS
history, they utilize this technology called Zen historically.
One of the things that Zen did really well is that it,
did not require hardware virtualization in order to run
the Linux kernel in an isolated environment.
So we took that technology and basically adapted it.
to be able to run containers, securely, without using
hardware isolate or hardware virtualization itself.
So you're running this on hypothetically an EC2 instance running KVM,
then you wind up rolling out Xen on top of that on some level where you
have the DOM0, which is the guest tenant with special hardware privileges.
Hardware, of course, in this context, being a little hand wavy.
Then you have various DOMUs that wind up running on top of that.
You're a wrapper around that, that type of approach.
Yes.
So the way that this came to be, was actually from my
experience in IoT and running containers at the edge, and we
had some problems with the multiple vendors basically needed,
needing to run their application on a single hardware device.
We only had one per place where we were installing these, and so
I was looking for technologies that could do that really well.
And Zen happened to do that extremely well
without the need for hardware virtualization.
The story of why we don't require hardware virtualization
is that we would acquire these industrial IoT devices.
Some of them would have hardware virtualization enabled in the BIOS.
Some would not because they were being shipped directly from the factory.
We could not intercept them to edit the BIOS,
they were just being shipped with our OS image.
And talking end users through modifying BIOS settings is, for
those who have not done it, it's a good way to kill an afternoon.
Yes, exactly.
And the people who install these things are
electrical engineers who do know things like serial.
They know how to run serial lines.
They roughly know what a COM port is and all of these things.
They know a little bit about networking, but BIOS, you can't rely
on them to have a monitor to be able to plug in to these things.
You can't necessarily rely on BIOS to be consistent from week to week, as
these things tend to get upgraded in various batches that get shipped out too.
So it always seems to turn into a bit of a guessing game.
And this does not scale.
It just doesn't.
It does not scale at all.
So what we found is that Zen is perfect for solving this problem.
So you're right.
What we do is, we allow you to basically take your existing, EC2,
machine images, AMIs, and you can basically run our installer.
And it'll boot Xen, it'll boot your entire operating system into Xen,
and then we provide a different CRI implementation for Kubernetes
that allows you to then run your containers inside Xen as DOMUs.
But we have an easy on RAM for that, so you apply just a single
runtime class, and you're able to deploy your containers that way.
Where's the DOMU boundary at that point?
Is that per namespace?
Is it per cluster?
Is it per container?
Good question.
It's, you get to choose.
We call these Domuse Zones, which are, are,
are a term for basically an isolation boundary.
We run a minimal Linux kernel in there, and then a very small Rust-based init.
And, which by the way, the entire platform is written in Rust.
One of the cool things that we did is, reimplemented a lot of Zen and Rust.
So everything that runs in user space is Rust based.
So because we run that piece inside that zone, we can
actually implement multiple container support and provide the
standard kind of namespace based isolation, Linux namespace.
Isolation, which allows you to then choose how you
want to compose your containers on your system.
So, for example, you could discriminate what containers to run where by
saying a tag or an annotation in Kubernetes of a particular customer, like a
customer ID, you could specify it by namespace or by default, we give you the
most secure option, which is running every Kubernetes pod in a separate zone.
What is the overhead look like on this?
Because things like this have been done before back
in the ancient times before Docker was a thing.
I know such a thing did happen for those of my younger listeners.
The challenge with doing that was twofold.
First, the instantiation time was generally non trivial.
And two, it wound up having an awful lot of
overhead to run a bunch of those on a single system.
You would lose some amount of resource to the hypervisor layer.
Yeah, great question.
So, First, I'll plug our research paper into this that is fantastic.
We have a wonderful researcher named Marina Moore
who worked on TUF and other things like that.
She runs Edera Research and we co authored a really
great paper about the performance and the the overhead.
So the, the startup time is roughly 500 milliseconds on, on normal hardware.
So we're able to go from nothing to container in 500 milliseconds.
We think that we can even reduce that further, but this is just kind
of where we're starting as a base, and we're working directly with Zen.
Half-a-second, but I want it now.
Yes, exactly.
It's pretty fast.
In fact, if you look at like how fast Kubernetes can
even schedule a pod, it's kind of minimal in that aspect.
Importantly, you can run our zones and workloads, which
is what we call containers without Kubernetes as well.
So, you can also kind of run it as some, somewhat of a Docker stand in.
In terms of the overhead and, on IO and, CPU and, memory performance
and those kinds of things, we have some really good benchmarks
that show that we are roughly equal to Docker on bare metal.
This is very interesting, and when you look at
how the hypervisor works, it's pretty impressive.
The best way that I like to think about it is that because we
run Xen at that layer, we get ultimate kind of visibility into
CPU scheduling, into memory management, and that kind of thing.
We, I believe our current overhead is like 10 megabytes
per zone, and we think we can do even better there.
We're actively working on projects that, focus on reducing
memory overhead so that you can pack a lot of these things.
On my personal desktop, as an example, which I think has, 64 gigs of RAM,
I've run, 1500 of these independent zones running a very simple Go app.
So it can be very powerful.
That's impressive.
I mean, 64 gigs is almost enough to run, two instances of
Slack at the same time, but that's a separate argument there.
One of the challenges historically with a lot of this has been that dealing
with container isolation has felt like it's a constant game of whack a
mole, where, okay, we fix this issue, and then 20 minutes later, another
one seems to come out, and for enterprise scale, that just doesn't work.
That.
You're always basically on the trailing edge,
regardless of how quickly you can iterate forward.
I like that it sort of sidesteps that entire problem.
Yes, exactly.
I watched one of your podcasts before, and you
had the great comment about the CNCF landscape.
You know, it's very complicated, that, that thing.
I want to get a poster on it of it on my wall.
I have one that was turned into a thousand piece
jigsaw puzzle that my business partner gave me.
We put that together for three days.
My wife and I, it hangs, now glued, behind a frame, in my
home office just because that thing is a freaking nightmare.
One of the best tweets I ever saw, and that was
a picture of it simply captioned "Jesus Christ."
It was, it was, it's horrifying.
It is horrifying, and you know, one of the things that I really
struggled with when, when starting Edera was, "Am I just going
to be another one of those tiles?" And I struggled with that.
I was like, "Should I really be doing this? Should I be
adding to the tools and the things in the CNCF landscape?"
Or, you know, "What am I doing here, really, if I'm just
adding another layer?" And what I determined very quickly
is, you know, someone's got to build a better foundation.
We just got to do it.
I really struggle with our current set of tools and how many layers there are.
There's like four-to-five independent, roughly independent
projects involved with running a container on Kubernetes.
And I think that's just in terms of executables.
We're not even talking about dependencies.
And I, I think that's like kind of scary.
I want there to be a better way to run a container today, and I think we have a
very interesting way of doing that, where you can run, basically everything from
Kubernetes to the container in our platform is, is one single cohesive thing.
I think that's super cool.
Yeah, I just realized this is where Ariadne works.
Yeah, as your distinguished engineer, if, if she says it's good, it's good.
That I, I have no further questions on the security posture of this system.
My God.
Yes.
Ariadne is a great friend.
We started this company together initially.
And then, the former CEO of Chain Guard, Emily, our CEO now joined us.
And, we, we have an amazing team.
I would put our team up against anyone.
It's a tough world out there, and not just for startups, but for people.
And I think, you know, we have such a great
supportive environment and Ariadne just adds to that.
Absolutely.
I do have a, I guess a question I'd like to ask folks who have been.
I guess doing this in a different era than I came
up through, what is your history with technology?
Where did you come from?
Great question.
It's always hard to not go too in depth because
one day I want to write a biography about it.
Yeah.
So I grew up in Alabama.
I was in a small town in Alabama.
Auburn, Alabama.
It is actually Phenix City, Alabama, but that's
even more obscure than, than Auburn, Alabama.
Both of my parents were teachers.
My dad worked in, universities and my mom was
a guidance counselor at an Alabama school.
If you don't know the Alabama public school
system, I did not have a lot of money.
My parents did not make a lot of money.
So I did not have computers really growing up.
I had a family computer, but I, you know, it was a tool.
It wasn't something that I got to spend time with.
So when I was like 11-years-old, well, I guess I should say my father
working at university really got me interested in computers because I
got to go play with all the teacher technology and everything there,
including a VMware cluster that they had running at the university, which
was kind of my first, like, entry into hypervisor stuff and all of that.
Long story short, I got into open source very young
with the Ubuntu 2.0 Project, and I was a packager.
I did not have the money for Windows, so I was using Linux.
I had a hand me down computer that I got from someone from my church.
Like, literally, just like, it had a virus on it, they didn't know what to do.
I got a computer from that.
Long story short, I got involved in open source with, lots of great projects,
including the Dart project and Chromium and Chromium OS and worked on
particularly Chromium OS, where I was building OS images and stuff for fun.
But what was interesting was I was very young.
I was 12-years-old.
And you could not have a Google account then legally.
I think it's still the case that if you're under 13,
you're not legally supposed to have a Google account.
Yeah, that's always stopped, kids.
"You must be 18 to view this website." Oh, okay, never mind.
This episode is sponsored by my own company, The Duckbill Group.
Having trouble with your AWS bill?
Perhaps it's time to renegotiate a contract with them.
Maybe you're just wondering how to predict
what's going on in the wide world of AWS.
Well, that's where The Duckbill Group comes in to help.
Remember, you can't duck the Duckbill bill, which I am reliably
informed by my business partner, is absolutely not our motto.
Yeah, so it did not stop me, that's for sure.
So somehow on a Google group, on Chrome OS related stuff, I posted my
age and a community manager at Google found out this resulted in a call
from a Google lawyer, as well as a community manager to my home phone.
I was freaking out, didn't know what was happening.
For you or like, do you like a legal guardian that we can get on the phone too?
Because if we talk to kids, there's a problem here and a different direction.
Yeah, yeah.
It was my parents, but I think my parents were just like, "What did you do?
Like, what did you do?" and, anyway, the Google lawyer comes
on and says, "You're not allowed to have a Google account.
We have to remove your Google account.
However, we love, you know, your involvement in the community.
We would like to invite you to Google I/O." So in 2012, I got to go to
Google I/O, all expenses paid with my father, and I got a Chromebox,
which ultimately ended up being my dev machine for the next six years.
But how could you use it without a Google account?
Well, so this is a great, a funny question because one of the people
I got to meet at Google I.O. was Bill Richardson, which is the
person who made the bios of, all the Chromebooks in the early days.
He now works on Pixel security, and he, I had some email
exchanges, and he taught me how to get into developer mode.
I was not running Chrome OS on that thing.
I was definitely running just Ubuntu, I think, at the time.
What's interesting is I got involved in, in Dart after that, after
seeing a conference, or conference talk at Google I/O about Dart itself.
If you're not familiar, Dart is way more popular than it was then.
And 2013, it was like nothing.
It was, it was this weird thing that Google, everyone
knew that Google was going to kill off eventually.
They just did not get involved with it.
I got very heavily involved in the community.
And I was on Google Plus at the time.
I had kept so many stories about Google Plus, and I had a Google
Plus circle with every Googler that I knew, and I would regularly
follow and like chat up random Googlers cause I was just a kid.
I didn't know what I was doing.
It's surprising how far that can carry you just by
showing up and talking to people, speaking for a me here.
It is.
So, now we finally get to where my career started.
A random company in Oakland contacted me via email
and was like, "Hey, we use one of your Dart libraries.
Will you come work for us?" And my immediate
reply was, "I'm a kid, but thank you.
Sorry.
I can't." And they replied with, and I, his name was Dennis, the CTO of
DGLogik at the time, an IoT company. And, He goes, "We know how old you are.
We, we already did the math and figured out that you can work here legally."
So I was hired overnight, literally at like 1 a.m. my
time in Alabama to work on IoT technology and Dart, and
from there I started working on Google related stuff.
So they had a project for, various different things.
Google Fiber was one of them, where we were monitoring all of their edge
data centers with IoT technology, and ultimately, I got involved in Google.
And I was at a startup, and then I went to another startup that
was a vendor for them as well, and then ultimately, I ended
up just kind of working on their IoT platforms internally.
And that was where I spent the last 10 years of my career prior to Edera,
and I knew Ariadne from IRCv3 because I worked on IRCv3 stuff in my IRCbots.
I ran, I had owned irc_dart, and there was a Java library called pircbotx,
if I recall, that I was contributing to as well.
So, that was how I met Ariadne was just on the internet
and then her community treehouse as well, and that's how we
kind of reconnected after just knowing each other on IRC.
I met my business partner, Mike Julian, when we
were both, network staff on freenode years ago.
It's fascinating watching even generationally, "Oh, we met on
IRC," continues to be alive and well as far as origin stories go.
Yes.
I mean, so I, I think I got to see the death of IRC even just as a kid,
and by death, I mean, you know, the downfall of freenode, et cetera.
I was in my late thirties, shut up.
Exactly.
So I think, I don't know how old I was when that
happened, but I was on EsperNet because of Minecraft.
All my friends are on EsperNet.
Til this day, my best friend is someone I met on IRC on the Minecraft
Forge Channel on EsperNet, and I think it's just, IRC is a powerful thing.
Discord, it took it, I think it took it over.
I think it'd be fair to say it took it over.
I migrated there in 2015 from IRC and then basically didn't look back.
There's something different about IRC, just the vibe of IRC is
so different, and I think part of it is that, because it's just
text, it feels so, like, slightly impersonal
but personal at the same time, you know?
It forces you to, at least in my experience, it forces you to articulate
your ideas a little bit more clearly if you want to have a good time.
It's, it's sort of a, I guess a breeding ground for writers.
Yes, I would agree with that.
I, so I, one of the people who used to lurk in one of our IRC channels, I
contacted them a few years ago, and they happened to have logs of everything I
said in IRC and my channels over, like, seven years or six years or something.
God, I was a cringy kid, but everyone's cringy as a kid, I think.
But, I did some analysis, and I was like, wow, I wrote a lot of
stuff back then I wrote a lot of words and I think that it really
taught me communication style, particularly because it's in text,
you have to kind of really work hard to get an idea through.
And it's also weird because I was in the Minecraft community there
where you kind of have this weird blend of, like, adults who are
into Minecraft programming and then kids who are, like, just like
12-year-olds who just want to play a Minecraft mod or whatever.
The modding scene in Minecraft is very interesting.
It has a lot of history that is just far too much
to go into, but it was so cool to be a part of it.
Now it's my turn to make you feel old.
My daughters are very much into Minecraft these days, playing with each other.
They are seven and four, and they love playing it on their iPads,
and I'm just waiting until one of the penny drops for at least one
of them to realize, "Hey, we maybe maybe there's more to it than just
consuming as we find. Maybe we could make it do something different."
I'm not forcing it on them because that's child abuse on some level.
"Hey, have you used IRC?" It's like, guys, they'll take my kids away.
No, it's a, but it's very much a, let, meet people where they are.
And I've wondered about that for a long time, because as someone
who started kicking computers around in the '80s and never
really stopped, the road that I walked is very much closed.
If I give career advice to people, it's very boomer-coded.
It's, "Oh, just have a strong handshake," and can't put it on your resume or do
what I did and basically bluff your way through a technical interview at Chapman
University the day that their technical reviewer was out sick and get the job
offer 20 minutes later and cram like absolute hell for the next three months.
And suddenly you're there and doing it.
And then just, yeah, great.
Take that energy and never stop.
You're always going to be learning and cramming
and we're thinking you're not quite far enough.
That advice doesn't scale and it doesn't necessarily
map to the way the world hires technologists today.
So I don't give that advice.
Yeah.
You know, it's fascinating.
I had a time in my life when I had to interview normally
like a, like, you know, like I normally would for a job.
And because I had basically been handed a job by just like
happenstance my whole life, I really struggled with it.
And, you know, what I have always done is focused really hard
on my, on my background and doing things in my personal life.
You know, this doesn't work for everyone, but I am,
I love programming outside of my, outside of work.
I do it for fun.
I have weird, stupid projects.
I have a programming language.
that I wrote in Kotlin called Pork, for example.
A very silly name whose icon is literally a pig from Minecraft.
You know, I just do weird, silly things like that.
And I think it gives me the confidence to say that I have a portfolio
other than my, and other than my, you know, career achievements.
What I say is talking to people really works.
Networking is real.
And when I talk to people about networking.
I am, I am an introvert.
Naturally.
I like to say that I'm an extroverted introvert.
Our CEO, Emily is like my hype woman.
She's a introverted extrovert and I'm an extroverted introvert.
And so we, we work very well together.
She's, she's very good at making me get out there,
like doing this podcast and things like that.
Teach you to swim by throwing you off the end of the pier.
They weren't trying to teach you to swim.
It kind of is like that.
And, you know, it's funny, you know, we work so well
together in that regard because, you know, she, she's
the one that tells me my background is interesting.
Meanwhile, I'm like, Oh, it's boring.
Like I'm just some kid from Alabama, you know?
But the reason why I mentioned that is networking is actually.
Very important, but when people think of networking, I feel like they,
they think about like what I would describe as corporate networking.
Like I'm, I'm going to go to this conference and talk
to some corporate people and that's great, but I, I like
to approach things from a more personal perspective.
I like to get to know people on a personal level and, and, you know,
really, truly be friends with them, not just from a corporate perspective.
And I think that has done wonders for me because,
you know, I'm, I'm dependable in that regard.
People respond well to that.
It is challenging, I think, for Gen Z, my generation, to kind of get used to
the, the more boomer kind of, way of, of job seeking and things like that.
You know, I consider myself lucky in some regards
to be plucked from Alabama out of obscurity.
But, you know, at the same time, I worked hard for it.
So I, It's a balance to kind of figure out what the best way to, what
the best way is to think about it, but yeah, it's, it's a tough topic.
I hear you on this.
I have one last question I want to dive into
a bit to sort of bring this full circle.
As of this recording, it has not been announced.
It will be by the time this airs unless something
horrifying happens to our production embargo system,
but you have just raised a Series A. Tell me about that.
Yeah.
So, our Series A, was, led by, M12 at Microsoft, Microsoft's Venture Fund.
It's very exciting for me.
Microsoft is a very, interesting expert in hypervisors.
And so, you know, it, it's been so cool to see that they understand it.
Deeply and, you know, all of our previous investors invested as well.
And we have some absolutely incredible investors.
It's exciting to be able to do what I'm doing, particularly
as an all female founded company where that typically
is harder to, navigate and, and the funding space.
And I'm so excited to just be doing what I'm doing.
I dreamed of like building a business, not because I wanted to be powerful or
anything, but because I just wanted to do something interesting and I like that
we are in a deeply technical space with people who really understand the space.
And, I'm excited at our future and the cool things
that we get to do with this next phase of our company.
It's, we have such an incredible team.
I, I couldn't, I couldn't even list all of the
absolutely fantastic people we have working here.
We're very, very lucky to have, so many great people.
And, I'm so happy to look forward into the future.
I really want to thank you for taking the time to speak with me today.
If people want to learn more, where's the best place for them to find you?
Yeah, probably Bluesky.
I'm, I want to get a personal website that actually looks good in the future.
Maybe a blog, but, on Bluesky, I'm alex.zenla.io.
That's the best place to interact with me.
And then from there I'm on Discord in various
places as well, but Bluesky is the best.
And I will, of course, include links to that in the show notes.
Thank you so much for taking the time to speak with me today.
I appreciate it.
Yep.
Thank you so much.
It's been great.
Alex Zenla, founder and CTO at Edera.
I'm cloud economist, Corey Quinn, and this is Screaming in the Cloud.
If you've enjoyed this podcast, please leave a five
star review on your podcast platform of choice.
Whereas if you hated this podcast, please leave a five star
review on your podcast platform of choice, along with an angry
comment that I will then edit because that podcast platform
does not understand that containers are not a security boundary.