Oxide and Friends

We've talked about Oxide's Root of Trust--the hardware underpinning the security of the product--but how do we keep a bad actor from spoofing Oxide to load malicious software? Philip Tricca joined Bryan and Adam to describe the key ceremony, the process of creating and storing the secret that plays a critical role in manufacturing. And we paid our respects to the dot matrix printers that made the ultimate sacrifice in the name of security.

In addition to Bryan Cantrill and Adam Leventhal, our special guest was Philip Tricca.

Previously, on Oxide and Friends:
Some of the topics we hit on, in the order that we hit them:
If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Creators and Guests

Host
Adam Leventhal
Host
Bryan Cantrill

What is Oxide and Friends?

Oxide hosts a weekly Discord show where we discuss a wide range of topics: computer history, startups, Oxide hardware bringup, and other topics du jour. These are the recordings in podcast form.
Join us live (usually Mondays at 5pm PT) https://discord.gg/gcQxNHAKCB
Subscribe to our calendar: https://calendar.google.com/calendar/ical/c_318925f4185aa71c4524d0d6127f31058c9e21f29f017d48a0fca6f564969cd0%40group.calendar.google.com/public/basic.ics

Bryan Cantrill:

How are you?

Adam Leventhal:

I'm doing great. I... As as you know, I just got back from watching the the Elizabeth Holmes documentary.

Bryan Cantrill:

You can see everything.

Adam Leventhal:

And I'm just... I'm I'm still processing. Like, if I I feel like the human beach ball spinning.

Bryan Cantrill:

For anybody who has not seen this, we are not gonna give you spoilers right now.

Adam Leventhal:

Right. We actually, you know, I don't know if people know this, but like we we don't talk on Mondays, like, usually. I mean, not that we talk every day or whatever, but, like, I think we particularly don't talk on Mondays because we wanna, like, you know, not not get it all out of our system before the pod. But we just spoke because we needed to get all the spoilers out of our

Bryan Cantrill:

Yeah. And I think we get all the spoilers out of

Bryan Cantrill:

our system.

Bryan Cantrill:

So I think that we can... The non spoiler is she's crazy. Yeah. And the movie's extraordinary. I feel like those are two non spoilers.

Bryan Cantrill:

Yeah.

Adam Leventhal:

Yeah. I came in heights, and it did not disappoint. And it's three hours, which is like... I don't know. It's a long movie, but it was great.

Bryan Cantrill:

I I think... Did I tell you or Steve you cannot have expectations that are too high? I think you told Steve that. And I feel that that is true. I feel like you can't have expectations that are too for this thing.

Adam Leventhal:

Yeah. Yeah. I think that's right.

Bryan Cantrill:

How's my audio, by the way? Is my audio... Am I coming in and out?

Adam Leventhal:

A little in and out. I didn't know if you were, like, reaching for a Diet Coke or what.

Bryan Cantrill:

No. I'm not. Why is this thing? It... Hey.

Bryan Cantrill:

I'm gonna do a complete power cycle. Sorry.

Adam Leventhal:

Good. I'll be right back. Philip, are you in in here or are you gonna be in the litter box?

Philip Tricca:

I think I'm here and Bryan may be restarting me.

Bryan Cantrill:

I'm about to power cycle. Don't get too comfortable.

Adam Leventhal:

Good. I'm not not getting comfortable.

Bryan Cantrill:

I would never never feel comfort. Okay. How do I send... I'd see. There we go.

Bryan Cantrill:

That's... Yeah. Yeah. There we go. That fixed it.

Adam Leventhal:

You can tell you can tell that you sound good. That's good. Can you hear me?

Bryan Cantrill:

I can hear you. Can you

Adam Leventhal:

not hear me? No. No. I can hear you. You sound great.

Bryan Cantrill:

Yeah. No. I sound great, don't I? Yeah. Yeah.

Bryan Cantrill:

There

Bryan Cantrill:

was Don't

Adam Leventhal:

don't die.

Bryan Cantrill:

Don't die. Just... Yeah. Exactly. Yeah.

Bryan Cantrill:

Yeah. I... You know, why is power cycling the cure to all of our ills? Drives me nuts. Anyway...

Bryan Cantrill:

The amazing movie, I thought. I thought amazing movie. And I... It... Everybody...

Bryan Cantrill:

It is only showing on actually a relatively small number of screens. So a bit of a surprise, but I'm sure this is gonna be a nationwide release. So you should absolutely go check it out. Adam, I I I know I'd asked you this, but there was... You had some audible gasps in yours.

Bryan Cantrill:

Definitely auto... Big gasps Yes. In our our screen. Yeah.

Adam Leventhal:

Yes. Yeah. Good. Mean, small theater. I went in Piedmont, small theater, but definitely, like, it was a community event.

Adam Leventhal:

You know, peep... People were gasping. I think saying that there was a wide shot of WeWork that, people found very on the nose.

Bryan Cantrill:

I think... I don't

Adam Leventhal:

think that's spoiling too much.

Bryan Cantrill:

But

Adam Leventhal:

people people were chucking

Bryan Cantrill:

that. That's our local movie theater. That's where we saw it. Yeah. And, showing on all three screens.

Bryan Cantrill:

This is a movie theater that that knows its demographic.

Adam Leventhal:

That's amazing. Because because I had to leave the island for this. Like, normally, wouldn't... On a... Like, on a school day, I wouldn't leave the island.

Adam Leventhal:

But...

Bryan Cantrill:

No. But, they they know... Except they know the demographic. Unbelievable film. Did you...

Bryan Cantrill:

Now I... Because I know you just saw it today, so you probably have not had the chance to go back and relisten at all to our, our March Madness of Fraud episode on Theranos.

Adam Leventhal:

No. I haven't. I... In fact, I had I had sort of forgotten that we had talked about Theranos. But

Bryan Cantrill:

We had no episode on it. This is your suggestion of... I mean, mean, I know I know I I know you've I know you've thrown out many acts of genius, so they blur together for you. But your March Madness a fraud, and like the question is like, who's coming? I mean, and I still feel that like, I mean, Theranos is still the one to beat.

Bryan Cantrill:

I mean, it is still coming in as the as the overwhelming favorite. Yeah. Absolutely. Number one seed. Yeah.

Bryan Cantrill:

And I know it would... But it was really interesting to go re... Because you know, that is from... That's from five years ago now.

Adam Leventhal:

Yeah. Oh, yeah.

Bryan Cantrill:

We've been at this for a minute. And it... But... So it's like before all of the the kind of the AI business, and it's like very much in crypto. And it's just interesting to kind of, listen to that time capsule.

Bryan Cantrill:

Don your authentic 2021 garb as you are listening to our podcast. Because, you know, you and I are both kind of hanging on every episode Yeah, of the

Adam Leventhal:

yeah, the Kararu podcast.

Bryan Cantrill:

The Kararu podcast and Avi Tovanian, the Avi Tovanian episode had just dropped when we recorded that. We were kind of talking about that. And talking about how like there's this... Asking the question, which I think is... I still think is kind of interesting about like how much faking it is...

Bryan Cantrill:

And then faking it is putting it too pejoratively. But like, you kind of have to have this vision for where you want yourself to be. And that is of course contrasted with the reality of where you are. And when you're starting a company, these things are necessarily divergent. And the question is like, how do you represent this divergence with honesty?

Bryan Cantrill:

And there's a segue to our topic here.

Philip Tricca:

Wow.

Bryan Cantrill:

Yeah. Nice. Because this is an example. The topic we're gonna talk about is an example of kind of the opposite, where a lot of the work that we did here was entirely unseen, actually. And I mean...

Bryan Cantrill:

And it's almost by definition unseen. But we... But this was work that we... This was this was much more make it till you fake it kinda work, than than, because this was a lot of real technical work that we had not really talked about publicly. And I was kinda longing for the opportunity to talk about this publicly.

Bryan Cantrill:

So, all the way back from... And Phil... It's it's early twenty twenty three, I think. I wanna say like January, February 2023. Is that right?

Philip Tricca:

Sounds about right, yeah.

Bryan Cantrill:

Okay. So could you... We have now all come to grips with the fact that we have to ship a product. I mean, it it it it would just... Not to say Right.

Bryan Cantrill:

Like everyone everyone knew all along we needed to ship a product. I just wanna make that clear.

Adam Leventhal:

But there's a certain disappointment when you realize it's like not years away, but it... But mere weeks. Yeah.

Bryan Cantrill:

Well, I think it's like, look. I mean, let's just let's just let's just kind of speak directly for a moment. Everyone is kind of assuming that someone else is gonna be the long pole. You know what I mean? You're like, as long as...

Bryan Cantrill:

Oh, there's a supply chain crisis? Like, oh, no. Oh, well. Well, god.

Adam Leventhal:

Schedule chicken. Right?

Bryan Cantrill:

They're like External schedule chicken.

Adam Leventhal:

I'm on time. No. A 100%. I'm on time. My stuff is all gonna be on time.

Adam Leventhal:

Is your stuff on time? Oh, good. Then we're both on time.

Bryan Cantrill:

We're both on time.

Bryan Cantrill:

Meanwhile, a car continues to accelerate towards the wall. And it is someone out... The hardware team veers off at the last second. But the the e... That youth is exactly right.

Bryan Cantrill:

Schedule chicken is such a good way to describe it. And in this game of schedule chicken, sometime in early twenty twenty three, it became clear like, oh, shit. The hardware's gonna work. And we... When this hardware...

Bryan Cantrill:

And I do remember telling people like

Adam Leventhal:

Hardware did not blink.

Bryan Cantrill:

Hardware did not blink. Hardware did not blink. I do remember telling you folks, it's like, when the hardware's ready to go, we are shipping. And it's very important that the hardware be ready to go. It's very important that we not have something that we...

Bryan Cantrill:

And, you know, I think that like through the kind of the fall of twenty twenty two, it... Because we, you know, we had what we called the mass casualty event, which I I apologies for the insensitivity to to actual mass casualty events, but we had the... Where we had... Like we powered on the rack and half the sleds blew out. And you gotta think like that, like, some people in software were like,

Adam Leventhal:

oh, no. Yes. Well, good things my stuff's all on time.

Bryan Cantrill:

Good things my

Bryan Cantrill:

good things my stuff's on time as far as you know. But then it was like, we got a... Oh, it's like, oh, We can't book another compliance run. But then we'll... Actually, we did book another compliance run Oh.

Bryan Cantrill:

January. And it's like, oh, shit. And so it's becoming increasingly clear that like, the hardware is gonna be ready to go. And I do think even though we'd kind of been saying this for a while, I think like, I just remember in all hands where the light really went on for a whole bunch of people that like, schedule checking was over. And you, collective you, have lost.

Bryan Cantrill:

And now it's like, we're shipping this. And Phil, my memory is like, it's somewhere in there that we're like, oh shit, if we are gonna ship this, we've got a root of trust issue that we desperately need to solve before we ship. So could you describe like that issue?

Philip Tricca:

Sure, yeah. So I, you know, this is, I'm sitting here reliving these moments and it was a lot. That realization when, you know, in the fall we had actually started, you know, we've gotten the root of trust to the point where it was actually working to a certain extent, which, you know, was good. But then I realized that I was the only person who was building something that relied on a PKI that did not yet exist.

Bryan Cantrill:

Right. Okay. So what is a PKI? What do you mean by that?

Philip Tricca:

Sure. So this is public infrastructure. It is, I guess maybe the most disappointing thing about computer security and cryptography is that everything seems to boil down to PKIs. Boils down to some assertion made by some entity that holds a private key and all of the security properties of that thing depend on the secrecy of that key.

Bryan Cantrill:

And I mean, a lot of what you developed is this great act of delegation to be like, no, no, like I actually trust this other thing over here. And you... Well, I trust this other thing. Well, I trust this other thing. It's okay.

Bryan Cantrill:

Well, who trusts... Like at some point you get to the point where you get to the... You run out.

Philip Tricca:

Yeah. Delegation is a good word for it. But it is... I mean, in some cases, you know, you don't want it to be blind trust. You wanna actually have a reason to believe that you should trust this thing.

Philip Tricca:

And that, you know, requires trusting the people carrying out the process and maintaining it specifically and a lot. And so, you know, that realization I gotten past the part where it's like, oh, we need to give root of trust and identity, which means we need to be able to communicate its identity into it, some kind of certificate, right? You wanna set these things up the correct way so that you never expose the keying material that's actually in the ROT. So this thing generates its own key internally, has a little protocol for talking to a manufacturing platform so that it can produce a CSR that the manufacturing platform can then send off to some, the PKI process to get that signed and sent back into the actual root of trust. And once, you know, we'd got that working kind of just the realization that like there needed to be a, probably just as much work going into building and maintaining that system that exists outside of the platform as did the actual construction of the root of trust itself.

Philip Tricca:

And that's a tough realization.

Bryan Cantrill:

Tough realization. Yeah, I feel that was a tough realization. And on the one hand, it's like, a surprise intellectually. We all knew in the abstract that this would need to be done. But for whatever reason, of this becoming very concrete was like...

Bryan Cantrill:

Because I think, you know, and speaking for me personally, you kind of have this idea of like, hey, when the hardware is ready, we're we're gonna gonna ship. Then we need to make sure, and we were talking about the kind of the minimum update mechanism update and making sure that we had enough mechanism that we could go update the sleds. And so I was kind of in the mode of thinking that like the software, we are gonna be able to make, improve the software over time. But this is this actually this whole infrastructure that needs to exist, that actually kind of needs to exist before... Software infrastructure that needs to exist before you can ship the hardware.

Bryan Cantrill:

Because otherwise you run the risk of having a secret that is then compromisable. And that would be bad because using the secret... I know you said this in passing, but I just wanna like really emphasize it for folks that in the manufacturing of a sled, we are going to use an oxide secret to encode a device ID. And in that way, the device ID, we know that that you... Even if you were to get the device ID, you don't have a way of getting back to the secret from the root of trust, that's important.

Bryan Cantrill:

But we need that secret to be, if you were to violate that secret, you could spoof Oxide software.

Philip Tricca:

Yeah, this is another one of those situations where I've realized that by not having worked at a startup before Oxide, I had just kind of assumed that it was there. Every company I've worked at, create some time, somewhere along the line, they have to do this thing if it's necessary for part of their product. And that's usually very early on. So I've worked at places like Intel and like their PKI office has been around since probably the eighties. Right.

Philip Tricca:

And suddenly you gotta do it yourself.

Bryan Cantrill:

You gotta do it yourself. Yeah, yeah.

Philip Tricca:

I had built the one thing that depended on this that needed this. So I was looking around at anyone else. I'm like, who else is in this game with me? And of course there's folks that are interested in working on this stuff, but no no other feature actually depended on that other than, you know, the root of trust that we're building.

Adam Leventhal:

Right. I I think it were... It's worth stating that that that secret is also present in that secure compute element. Right? Like that that the hardware root of trust like has security features that also, you know, prevent that from being kind of yanked out of there trivially.

Bryan Cantrill:

Yeah, absolutely. That you can't... That that's right. That it's

Philip Tricca:

stored in the internal flash on the part. So there's no external part. There's no communication off the chip that, you know, would expose this thing. So it's it's... So it's contained within the envelope of the ROT.

Adam Leventhal:

Right. This is not just in some some flash or whatever that people could desolder and and exfiltrate or whatever. It is it is more physically secure than that.

Bryan Cantrill:

Yes. Yep. I mean, we all kind of like ship the like, I mean, we kind of knock on wood and shift uncomfortably in our seat. We say this, but yes, mean, that is the idea.

Philip Tricca:

Mean, they said that Not is the

Adam Leventhal:

like absolutely secure.

Philip Tricca:

Immediately where I go to after that is, I mean, all of the things that we to, that we say are the right thing to do, like never exposing this key outside of the envelope of the ROT. And even specifically, like it's really only exposed to very small subset of the code that runs in the ROT. But like the story around how we create that and the properties of the system as it's moving along are an enormous part of how we convince our customers to trust us. Right. Right.

Philip Tricca:

So we've all lived through the last few years of computing. And I mean, we've seen all sorts of certificate authorities do things that they're not supposed to do with their keys and cause all sorts of firestorms on the internet.

Bryan Cantrill:

Totally. And I think even then, I mean, it's like it's become much more acute in the last couple of years. But... And not like we doubted any of this. But if you were to somehow get ahold of the Oxide signing key, that's extremely bad news for anyone that cares about the software that runs on their Oxide sled, because we can now be impersonated effectively.

Bryan Cantrill:

Yep. So that would be bad. And this is tough too, because it's like, you need to do this again before you ship the first one, this kind of needs to be perfect.

Philip Tricca:

Yeah, I've been perfect is definitely a word. Yes, sorry. So it's impossible to build it in a perfect way, right? You definitely want to get specific properties, right? Like, so for all the people that are participating in this, you don't want any one of them to be able to subvert it without it being obvious to the rest.

Philip Tricca:

So you want to at least require that two parties involved collude order to subvert the system as opposed to just having, you know, one person that could actually, could undermine the whole thing. Yeah, but it doesn't have to be perfect. And in fact, I would argue that it can't be perfect.

Bryan Cantrill:

It can't be perfect.

Philip Tricca:

And more specifically, to your point, this does have to be done in order for us to, had needed to be done in order for us to ship our first product. But the question of like, how you define done is really important here. And if perfection is how you define done, we probably never would have shipped for the first product. And we probably would have had an army of people working

Bryan Cantrill:

on it. Yeah, mean, right. And I think maybe a better way of phrasing it is that it was something we were gonna have to live with for a very long time.

Philip Tricca:

Yeah and our customers are gonna have to live with. The thing we were thinking about was success, right? To plan for success. I've definitely been in situations where I really wasn't expecting to succeed in a certain way. But then suddenly you do and I've had a mentor in the past point this out to me that I was writing proposals at the time and he's like, whatever proposal you write, you've got to actually be able to implement the thing you're talking about if you're gonna actually get grant money to work on these things.

Philip Tricca:

Yeah. And this is, you know, this was about making sure that, you know, if we did take off and things got really busy and we started selling a lot of these things, that the security properties from the first product up through the current state were just as trustworthy as, you know, the ones we were gonna make in the future. So there is really in my mind, this is about making like a minimum bar where it's like, these are the certain things we're going to do a certain way. So we've heard horror stories from manufacturing lines where, you know, a part that's being made like, Oh, I have to certify a key. And so how do you certify a key?

Philip Tricca:

Well, you have to be able to, you you have to be able to calculate a sector over the public part of it. So like, Hey, let's just send the key out or maybe even better, let's create it outside the platform and then inject it into the platform.

Bryan Cantrill:

Oh, and I am just, I am convinced that... I mean, it would be interesting to know if we we kind of had omniscience for a moment about how frequently that happens, but it is not 0% of the time.

Adam Leventhal:

I mean, out of expediency, it's like, look, attached

Bryan Cantrill:

it I think it's an expediency. A 100%. Yeah. I've attached it to an email. It's an email coming from the CEO or it's like sitting in like, no, we're gonna lock the desk drawer that has this.

Bryan Cantrill:

So I don't know what you guys are so upset about.

Philip Tricca:

That was exactly where I was going with this. It's like, I'm sure like the number of startups out there where their PKI was like, you know, an unencrypted key sitting on a thumb drive in someone's desk drawer. Right. Is probably way more than zero.

Bryan Cantrill:

Way more than zero. And I think that this is one of these things where it's like the kind of the lean methodology or you wanna ship like the minimum thing. It's like, well, the minimum, you're gonna live with this decision. The whole idea of that kind of the lean startup thinking is that you have time to iterate and make it better. So you shouldn't spend all this time making it perfect because you wanna learn from your customers to iterate and make it better.

Bryan Cantrill:

All that makes sense. But like, this stuff you're gonna live with for a really, really, really long time. So you don't wanna assume because you're gonna have to go back and re... You really wanna assume you're shipping a product here.

Philip Tricca:

Yeah, yeah. And I mean, it's not impossible to create a new PKI. I mean, we can do that from thin air basically. You're just generating data that becomes an encryption key or assigning key. So, you know, I don't think that, you know, you do have to have something on day one.

Philip Tricca:

It has to hit some kind of minimum bar so that you're not running around trying to fix stuff or trying to explain to people that they bought something that isn't really as good as they thought it was. So we can definitely stand them to PKIs and we will. We've done this as we make new platforms. Our PKIs are specific to the particular hardware platforms that we make. So we do stand those new things up, but the hard part is to finding the minimum bar.

Philip Tricca:

And this is a space where the minimum bar isn't really defined for anyone by anyone. You have to make the choice for yourself. People are really secretive about it. So there's very little information about there about like, what is the actual minimum bar for doing something like this? And I certainly didn't know when I started working on it.

Bryan Cantrill:

Right, so you have this like, oh shit moment where it's like, we gotta build this whole thing, something that we've like depended on. And this is where you do have, I feel like the more established company experience is super helpful in that we have really high expectations for ourselves. But we need to balance that with the need to ship. Fortunately, signing ceremonies, a technical term, do exist elsewhere. Mean, you surely drew some...

Bryan Cantrill:

Did some research. Yeah, yeah. So what are you doing research in terms of like how, because what we need to fundamentally do is we need to generate a secret.

Philip Tricca:

Yep. Yeah, so the only good resources I could find publicly about this stuff was from the groups, specifically Ayanna, that is responsible for effectively shared infrastructure. Like they are very transparent about their processes because they control a set of signing keys that are the root signing keys for DNSSEC infrastructure. And that's something that everyone relies on. So that's in a way kind of a public good.

Philip Tricca:

And they spend a lot of time trying to convince people. And this is again, this exact same thing we're doing with our customers in a way. We're trying to be transparent with them in every regard for the platform, everything from the software, all the way on down to processes that are outside of the system. And AYANA had some exceptional documentation describing their signing ceremonies for the DNS root keys. And I drew from that gladly.

Philip Tricca:

And, you know, anyone from Ianna listens to this, my sincerest appreciation, I would not have been able to do this without that resource.

Bryan Cantrill:

Well, and as you say, it's a public good. So it's really great that that is kind of made available for folks that actually want to do this the right way or do this the best way they possibly can.

Philip Tricca:

Yeah, they record their signing ceremonies. You can go watch them on the internet. You can get the scripts and all their processes are very transparent. And again, I look at that as a user of the larger internet and it, you know, by virtue of knowing their processes, I trust that system more than I would have. And that's exactly the thing that we want to do for our customers.

Philip Tricca:

We want them to look at our processes. And again, this is number one, why we were so transparent and why we were so rigorous from first, from jump is because this is potentially where you either win or lose customers

Bryan Cantrill:

in Yeah, a absolutely. Well, and then the consequences of getting it wrong and needing to go back to your earliest end customers and be like, Oh, by the way, what you're running is not actually secure. And we fixed it later, but it's like, we've got no way of going back and fixing your ROT, sorry. It's like,

Philip Tricca:

we That's don't want an RMA

Bryan Cantrill:

kind of scenario.

Philip Tricca:

And even then like that opens up an entirely new can of worms. Like what does it mean to RMA the root of trust off one of these things and create

Bryan Cantrill:

a new one?

Philip Tricca:

We don't have a process for that.

Bryan Cantrill:

Yeah, that's a please no. Let's just avoid that. And so this is when, I mean, because this was like very involved in like the shipping of the first product, is where Kate Hicks from our operations team got swung in and to help you. I think I know that the idea of a ceremony is a technical term, but I think for the rest of us for whom it was a new term, we definitely found it kind of tickled by the idea of the ceremony. And I guess that you had a very good sense of humor about like, because Kate's like, we gotta get robes for this thing.

Philip Tricca:

Yeah. I think that's Kate and Alan having really good senses of humor. My humor was absolutely at zero. Zero at doing this. Right.

Bryan Cantrill:

Right. I guess you were just too ashen to stop anyone from ordering robes. I was just tired. Exactly right. So Kate orders robes, which we...

Bryan Cantrill:

Okay. So... And then what are the elements of the the kind of in broad strokes we need to... What are what are kind of the steps and the problem that we need to go solve here?

Philip Tricca:

Well, first step is getting robes.

Bryan Cantrill:

Yeah, you check. Okay, good. Exactly. That's right.

Philip Tricca:

Yeah, I mean, so you gotta figure out, the process is the first step in my mind. Like you have some elements in there that are pieces of technology and you're trying to build a story, if you will, to convince people that they can trust the output of the process. And so a lot of what we're doing is trying to, we're not doing this with pencils and paper. There are computers involved and a big reason why we're building what we're building is that I particularly don't find the trust properties, the security properties, a lot of the computer platforms out there very compelling. Right.

Philip Tricca:

And we're not making those types of computers. So we're having to work with commodity hardware and figure out like, you know, again, where is that minimum bar? What needs to happen with the information systems that we're using to do this in order to make them, you know, trustworthy.

Bryan Cantrill:

Okay. This is actually the... This is a very good point you're making because I mean, you, like a lot of people are, you're at Oxide because a part of the mission really appeals to you. Part of the mission that really appeals to you is improving the state of the art of security in our deployed systems. All great.

Bryan Cantrill:

The problem is the... Our existing systems that we might wish to record this infrastructure in is the stuff we're trying to replace. Yep. So we have like a bootstrapping problem of like, you don't wanna like trust the... I mean, so like trusting this to a laptop that has all these known vulnerabilities or what have you.

Bryan Cantrill:

I mean, there are... So there's a real bootstrapping problem here.

Philip Tricca:

Absolutely. Yeah. Yeah. And the, you know, a lot of this is trying to prevent, you know, the root assets here, the real assets you're trying to protect are those secret keys, the signing keys. And so the goal is to expose them as little as possible to as few components in the system as you possibly can.

Philip Tricca:

So you want something like, you know, an HSM, some kind of hardware security module, which is specifically designed for handling keying material and not exposing it to, you know, a general purpose computing platform. And, but even then you want that platform, laptop that's executing this thing and talking to the HSM, you still want that thing to be as trustworthy as you possibly can make it. So similar principles to the way that we build the software that runs on the rack. You want to have a static image. You want, you know, something that does not have a writable file system.

Philip Tricca:

You wanna load it from read only media. You want to know its identity. You wanna be able to know what digest that disk should have. You need to have some mechanism to establish that and to record that as part of the process. And

Bryan Cantrill:

I mean, this is like... These are like IT problems that are load bearing for shipping the rack. I mean, is... I mean, as this thing is kinda like blossoming out, I mean, again, I know I know you knew what kind of the the the scope here, but as the scope is kinda like, oh, right. Yeah.

Bryan Cantrill:

Okay. We need to do... There's a of detail here that has to be attended to.

Philip Tricca:

Yeah. It's a lot of stuff that, you know, folks maybe that work in similar space that I do. We think about this a lot, right? It's a risk analysis. A constant risk analysis.

Philip Tricca:

Looking at this after the fact, I was writing an RFD that describes effectively the properties of the platform that we want for executing these ceremonies. And as I was writing that, I'm realizing, I'm like, this is a lot. It's not trivial. So you buy a laptop, know, one of our biggest concerns in this is wireless technology, right? When you have a wire between two points, stuff can flow between them, electricity data.

Philip Tricca:

When there's a Wi Fi antenna, a Bluetooth radio, where does that stuff go? Right? So we, I mean, we pulled that out of the laptops.

Bryan Cantrill:

Yeah, mean, I'm just amazed that we, I'm sure he was tempted to go like, go buy a K pro or what, go buy in 1980s era, like DOS machine.

Philip Tricca:

Well, and that's the other side of this is that you still have to be able to buy new things. Right. If you were to run on something that isn't made anymore, sure maybe it has the properties that you want, but if it breaks and it will, what do you do? What do do? You can't make a new one.

Bryan Cantrill:

Have to

Philip Tricca:

be able to either make it yourself, which, you know, never would have thought this would have been on the table, but maybe someday we will. I would love to be driving our, not just, you know, our key signing ceremonies, but even our manufacturing processes with systems that we have built that we understand the security problems.

Bryan Cantrill:

Maybe that's the product for the home labbers. Maybe that's how we like solve both problems at once. Maybe we finally deliver something to the home labbers that is like, no, no, actually we did this for our signing ceremonies, I'm glad. But now you can run Oxide in

Adam Leventhal:

the home. Why are you teasing

Bryan Cantrill:

I tease because I love. Which puts a lot of constraints on it.

Philip Tricca:

And they're weird constraints. Not normal constraints. They're the constraints where if you turn to someone and say, Hey, I need to buy a thing with these properties, they look at you like you're crazy and ask you, Why? Why would you ever wanna buy that?

Bryan Cantrill:

Why would you ever... And we bought a dot matrix printer. Yeah, yep. And as we learned, you can actually buy dot matrix printers.

Philip Tricca:

You sure can.

Bryan Cantrill:

They... Which is kind of amazing. And we... So this dot matrix printer is gonna be... Because this dot matrix printer is gonna print out the actual private key material.

Philip Tricca:

No. No. No. So this is again, like one of the, one of these things that we have to do in this process is manage access control to these things. So the keys that are the roots of our PKIs, those are created in a little HSM that's in the, that, you know, is hooked up to the laptop that we use.

Philip Tricca:

Those keys never leave there, but we have authentication values that enable their use. And in order to actually sign something with them, you have to have a secret that is presented to the platform that allows you to perform a signing operation.

Bryan Cantrill:

Got

Philip Tricca:

it. And so we have to deal with that authentication value. And, you know, there's not just, you know, do you keep it as secret as you possibly can, but we also, gotta be able to back up and restore these things. And so we have to have, you know, one individual that has access to that secret need to be able to use it, but we also need to be able to recover. And so, you know, we don't expect these people to actually remember these passwords.

Philip Tricca:

They're not your typical password. We don't reuse them. We generate them randomly. They just are, you know, a random series of bytes. And since the person isn't gonna remember it, you gotta like give them some kind of hint in order to be able to use it.

Philip Tricca:

And, you know, the truth in this one is that these are sensitive values. They're not the signing keys, but they are secrets.

Adam Leventhal:

Okay.

Philip Tricca:

And they do have to leave the confines of that system. Of the computer. And they have to be entrusted to an individual Right. That plays a part either our hardware, like our HSM custodian, the person that is responsible for driving our ceremonies, like physically on the laptop, activities. They don't do much because most of it's automated because we package an entire operating system with very specific code that's in it, knows what it needs to get done.

Philip Tricca:

But, you know, there's still, this has to leave the confines of that and be entrusted to an individual. How you do that is actually, you know, if we were to just do this over like, you know, the wifi antenna that was on the system and send it to someone's phone, not exactly the kind of properties you want. You want it to be very much point to point. You want it to go to a specific person for them to take a physical thing.

Bryan Cantrill:

Take a physical thing. Yeah. Take a physical thing that they can't, that you want them to not be able to tamper with that, right? Or do you wanna have tamper evidence of that?

Philip Tricca:

You still have to, yeah, you have to trust the people that are in the roles to perform the roles. And more importantly, most important thing there is communicating to the individual what their responsibilities are. Right. We don't want them to, like you said, pull these things out and wave them around and show them to all their friends. Right.

Philip Tricca:

So we tell them that and they're, you know, these are my colleagues. I trust them very much. Honestly. So Right. But you know, still you do wanna put some safeguards on that.

Philip Tricca:

And so tamper evident packaging is a very useful tool in that regard.

Bryan Cantrill:

And if someone were to physically steal one of these though, it's not enough, right? You've designed a system where it's actually, you need to actually have several of these to get in that also prevents a single bad actor or what have you.

Philip Tricca:

Yeah, our backup and recovery, we do an M of N key splitting. And then we have M people who receive those M key shares. And in order to do a recovery, order to reconstitute the authentication value, we need to bring together, you know, some number of them, the majority of them.

Bryan Cantrill:

So Adam, if you could take down your Craigslist ad offering to sell yours, because it just doesn't, it's not a buy.

Adam Leventhal:

Nobody even thought about offering to include me in the ceremony. I would be really clear that nobody even thought of me as like a potential backup.

Philip Tricca:

Well, I think we thought of you. Let me just pop out of

Adam Leventhal:

the oven. No, mean Oh, wow. Oh, wow.

Bryan Cantrill:

God, this is like, man, you've got like the killer instinct of my children. I'm just like, wow, I can put bloods in the water. Now is the time. Adam, I was just gonna say that we just had... It did not occur to anybody, but apparently it had...

Bryan Cantrill:

It had been occurred and and explicitly to It the

Adam Leventhal:

did actually occur because I guess somebody was sick and they asked me to come in. And then when I got there, like, no, no, no. Almost anybody else showed up. Don't worry. Like some person we've never met came.

Bryan Cantrill:

Some person never met.

Adam Leventhal:

I said, okay. That's great. Love that for them. Good opportunity. But

Philip Tricca:

the reality of it, in all seriousness is that anyone in this company should be able to perform these tasks. It should be made clear to them what their role and responsibilities are. Jokes aside, I trust anyone and everyone in this company to play pretty much any role in these things.

Bryan Cantrill:

Right. And so we're gonna print this stuff out. Want... Just as like you wanna avoid WiFi, you wanna avoid Bluetooth. This is where the dot matrix printer comes in.

Bryan Cantrill:

Like you wanna actually like have the most rudimentary technology possible that is still attainable.

Philip Tricca:

Yeah, yeah. In retrospect, I think the printer was probably not the right way to go.

Adam Leventhal:

Not secure enough?

Philip Tricca:

No, no, no. It was too much of a pain. It was just too much of a hassle. Yeah. And that's been a big part, like a really big part of, you know, me learning from this process is that, you know, perfection is not the goal.

Bryan Cantrill:

Yeah, The

Philip Tricca:

usability of these things, these things, the mistakes in these processes happen in those gray areas where things aren't as well understood or And as well so, you know, just getting people so that they know what's expected of them, what's going to happen, no surprises, like being able to practice. That was the thing that really hurt us the most in the run up to this thing was the schedule. It was, you know, just getting, we were literally going week by week, like, is it gonna happen this week? Yeah. Okay, do we have all the things in, oh no, we forgot this one.

Philip Tricca:

Okay, next week. And if we did that for, you know, a couple of weeks in a row where we were just shaking out unknown parts of the system, those are the things that scare me. Not nearly as much, you know, the, some of the technology that's hiding in the printer. But the reason we did go with this specific printer is that it was incredibly rudimentary. Taking it apart, it was very easy to identify all of the parts.

Philip Tricca:

It's not one of those small devices that's like, you know, sealed in epoxy or something where you can't like see what's inside of the

Bryan Cantrill:

thing. And it's not one of these like web services in a box that like you can't actually do anything without this thing phone at home.

Philip Tricca:

Right, no Bluetooth, no wifi built into it at all. That was a big part of this for me. But really the thing that we needed was just a writable, like write once media. Right. And we've switched over to using a writable CD ROMs for, you know, and the trade off was that we don't have as much transparency into the CD ROM.

Philip Tricca:

You can't really take apart a CD ROM.

Bryan Cantrill:

That is interesting.

Philip Tricca:

And then put it back together particularly reliably. They have lasers and lenses in there, right? You can't go in there and inspect every part on the boards like we did the printer. And that's a trade off. Like the printer is just a pain.

Philip Tricca:

It's never small enough and it's messy. Like the toner in those things, like

Bryan Cantrill:

As it turns out, don't use dot matrix printers for a reason. Yeah. Kinda suck. The other thing I kinda love about this, I mean, the aspect of this that I love is that of course, like, well, because of what do you do? You use this dot metrics printer.

Bryan Cantrill:

Like, so I guess you now have a free printer to use in the office. Like, oh, contraire. That is not what that... That's not what happens to this printer.

Philip Tricca:

Yeah. Office is the keyword there. You have seen office space.

Bryan Cantrill:

Yes. So the printer dies.

Philip Tricca:

Printer dies a glorious death.

Bryan Cantrill:

Glorious death. Warriors death. Not a like a PS load letter. Because I felt like when I was like, oh, we get to destroy the printer. I'm like, I have like obviously PS load letter and the office space in my head and we're all gonna take it out to a field and just like kick it to death.

Bryan Cantrill:

I'm like, this is gonna be great. And you're like, nope, there's a process like get out the Dremel. It's like a very surgical murder of this dot matrix printer. And that... I mean, I just...

Bryan Cantrill:

I I I appreciate that. It makes... Like, a lot... It's obviously very prudent to no longer use the printer. But I do love this aspect of it because, you know, Hans Christian Andersen has these stories about object tales where he's like, tells the story of like an object that...

Bryan Cantrill:

And I love this idea that like, each object has their own story. This is getting dark. And I just love the dot matrix printer who's like, you know, this is a dot matrix printer.

Adam Leventhal:

Oh, it's me.

Bryan Cantrill:

Like, oh, oh, it's me.

Adam Leventhal:

It's 2023, and they're still choosing me. I'm so happy.

Bryan Cantrill:

They're still choosing me. And, you know, I know they're choosing me for a good reason. Yeah. Important job. It's an important job.

Bryan Cantrill:

And it's like, it is an important job. Oh, it's an important job. It's a very important job. And it just like being opened and like everyone's wearing a robe and it just feels so exciting and we've got the most important job for you. And then I just love the Dremel coming out and the printer being like, wait, what the...

Bryan Cantrill:

What is that? Who's that for? It's like, no. Well, that's that's for you pal. It's like, I am good for it.

Bryan Cantrill:

Like, I don't even know what I printed. I don't even have like, I Yeah,

Philip Tricca:

that's the trust right there.

Bryan Cantrill:

Yeah, we can't know that though.

Philip Tricca:

Which is why we took the thing apart and looked. At all the chips on this thing and it does have writable storage. We tried to fence off, like you can find the data sheet for the memory chip that's on these things. I don't remember how big it was. It was only like 16 megabytes or something.

Philip Tricca:

And you can actually disable, you can hardware disable the writability of that, but the printer wouldn't function when we've done that. You can actually, there's a pin on there that's left You can ground it and then it won't actually write to the flash and that works, but the printer stops working.

Bryan Cantrill:

Right, right. Don't actually need to tell you technically, technically only need to decapitate you, but the problem is that, well, you're

Adam Leventhal:

not your head's connected to a lot of your body.

Bryan Cantrill:

It's connected to a lot of your body

Bryan Cantrill:

as it turns Seriously,

Philip Tricca:

I'm not gonna write that down. It's like, well, I can't really prove that you haven't.

Bryan Cantrill:

So So here it comes.

Philip Tricca:

Here it comes.

Bryan Cantrill:

Yeah. Sad for the dot matrix printer. A moment of glory for a dot matrix printer circa 2023.

Philip Tricca:

Yeah. And I guess the real tragedy there was that the dot matrix printer was just too big to fit in our safe deposit box where we store this thing. Had it been, we could have

Bryan Cantrill:

Okay. So this is an important point that in terms of like where all this apparatus, the laptop, the CD ROM drive ultimately ends up being guarded by a person with a gun in a bank. In an unspecified bank.

Philip Tricca:

Yeah, in an unspecified bank for sure. Yeah. Yeah, I mean, folks have seen, probably a lot of what our office looks like here and the doors open up and they're unattended. And our minimum bar was not having keys in someone's desk. Listen, I will thank you to

Bryan Cantrill:

know that we frequently sweep the leaves out of the office. So I'm not like,

Adam Leventhal:

oh, it would be super easy to steal stuff from the Oxide office on like on So easy.

Philip Tricca:

Gonna drop a pin in chat here.

Bryan Cantrill:

Exactly. Just right here. Yeah, just for myself. Please leave a five star review on Google Maps after you burgle the Oxide office.

Philip Tricca:

The DDR5 is in the vault,

Bryan Cantrill:

so don't... Yeah. Yeah. Don't ideas. Get Wait.

Bryan Cantrill:

Exactly.

Adam Leventhal:

The vault meaning the bathroom.

Bryan Cantrill:

Is that is that what we're calling it now?

Bryan Cantrill:

Okay. That is... The vault... The the bathroom has been very secured. We call it a vault now, Adam.

Adam Leventhal:

Oh, yeah. I do lock it when I

Adam Leventhal:

go in there. That's fair enough.

Bryan Cantrill:

Good. Well, it's all fun and games until there's a heist in the Oxide office trying to get DDR five. The... But the DDR five is secured. Just to just to all all on.

Philip Tricca:

Yeah. This is this is how I get budget for for security stuff, by the way. Just go into the podcast and tell everyone how bad it is and and the money shows up.

Bryan Cantrill:

That's right. Exactly. Just anything, please. That's right. We'll give you whatever budget you want.

Bryan Cantrill:

So we... I I think at this point also, I believe some... We had donned robes. And then I think that that at some point, I think Cliff had pointed out that the robes, the... Which were bargain basement robes.

Bryan Cantrill:

I mean, like...

Philip Tricca:

They were basically plastic bags.

Bryan Cantrill:

Yeah. They were basically plastic bags and the robes themselves are picking up an enormous amount of static and we are gonna zap anything we touch wearing these robes. So it's like the robes at some point during the ceremony actually come off.

Philip Tricca:

Yeah. If you've if you've ever been to Spirit Halloween, you've seen these robes.

Bryan Cantrill:

That's right. That's right. The robes are actually gonna end up doing damage to equipment. So the joke wasn't worth quite that much. Okay.

Bryan Cantrill:

So we do all of this and obviously under an extremely compressed timeframe. And you obviously had a lot of help from folks inside Oxide. Everyone is trying to help.

Philip Tricca:

Yeah, Katie and Kate, sorry, and Rick and Ian, like everyone came together to make this possible.

Bryan Cantrill:

Make it possible. Like, what can we do? And I think it's like also, and Ian Nichols at Oxide has got this disposition too of like, so often in corporate security, the disposition is like, no. Like, what's the question? The answer is no.

Bryan Cantrill:

And we get that, we honor that. I understand where that comes from. There's a lot of bad things people wanna go do and the answer no probably makes sense. In this case, we kind of need to do this.

Philip Tricca:

No, yeah, no was not acceptable.

Bryan Cantrill:

No was actually not an acceptable answer. And we actually need, the only acceptable answer is how do we make really deliberate decisions to minimize risk along the way? And that just required a lot of work and going to a bunch of... And I mean, the way you were phrasing this all along Phil, was like, I want something that I can show to a customer and be proud of and standby. And know that like, okay, this is something that you and I customer can look at together and know that like, yep, guys have taken a lot of really good precautions here to minimize risk.

Philip Tricca:

Yeah, scenario that I played out in my head, you know, is, I'm sure everyone who works at startups does this, this can't just be me, but playing out those scenarios where you're actually successful and great things are happening. And in this space, if great things are happening and someone is buying a lot of these things, you're moments away from having one of like the big four accounting firms show up and audit your processes. Like we're living the notion of supply chain security has gotten a lot of traction and is very visible these days. This is an enormous part of the supply chain.

Bryan Cantrill:

Yeah. And I just felt like from our perspective, like this is a core value of the platform. This is what we wanna go to. It's like, it would be a real shame to have like this very rich infrastructure that was built on a quicksand of like a key that was like locked in a drawer somewhere The or the sink in

Adam Leventhal:

most secure, like, hardware root of trust. I mean, this was the thing that, like, on day one that that was so front and center. So, yeah, then to, like, it in the drawer of your desk at home or whatever, Bryan, yeah, that would be disappointing.

Bryan Cantrill:

That'd be disappointing. And I think also like we knew and it would take us like, again, we shipped in 2023 and it took us several years after that to really connect all the dots from a software perspective and get to where we want. I mean, we were right that like we actually did have, I mean, have time to like evolve the software over time, but this bit we did not have time on. We had to get that right.

Philip Tricca:

Yeah, that a big part of it, right? The software, the RT was definitely not done when we first shipped this thing, but we were putting keys down there and we could update parts of the software that were going to handle the key, but you can't update the key.

Bryan Cantrill:

Yeah. Okay. So we get through the signing ceremony. The way I'm not sure if I'm trying to remember other, anything else that went awry. I think the signing ceremony, I mean, it had been delayed, but we got through the signing ceremony.

Philip Tricca:

Yeah, so before I go too far, I I said something right there about not being able to update the key that's in these things. It is actually, that is kind of the end goal though. The end goal is to be able to rekey these things

Bryan Cantrill:

That

Philip Tricca:

is an enormous amount of work, but we will get there. Like, think I see that as something that we just don't have an option with. We have to have some way to rekey these systems. It is generally accepted in the space that, you you have these kinds of these long lived keys, but the idea is to not require that they be long lived like that. Hopefully I'll around when we get there.

Bryan Cantrill:

But in terms of the parts themselves though, do have only a, there's only a finite amount of re keying you can do, right?

Philip Tricca:

True.

Bryan Cantrill:

So, I mean, once you like you've got, I mean, I think four of them, right?

Philip Tricca:

Yeah. So I guess this gets back into what the keys are used for. So we actually have two PKIs of three that are kind of relevant in that initial pass. One of them is the hardware identity key. The thing that we really think of as, you know, the key that is used by the ROT to make an assertion about a security state.

Philip Tricca:

This is effectively your attestation signer, but there are also, you know, we rely on the verify boot implementation in the chip that we use for the ROT as, you know, a big part of our security infrastructure. And so there are a finite number of key slots in there that you do have to program. That's a write once thing. You do not get to update that according to the part. And when you burn through all of them, you burn through all of them.

Bryan Cantrill:

Yeah. Yeah. And so, and this is the... Because people kind

Philip Tricca:

of ask you, do you

Bryan Cantrill:

guys have secure boot? And it's like, there's a little nugget in the system that does. You have to have something, some route where you know that this payload has been signed by a trusted actor. But that route happens, that it is only in that first bit of the root of trust that actually has that property.

Philip Tricca:

I think that we're trying to get away from those types of architectures for sure. We're seeing this with like things like Calyptra and some of the new ROT thinking that's going on out there, but these aren't, these things are still very new that would allow us to do away with that. But with the part that we have, we rely on that verify boot implementation.

Bryan Cantrill:

Can you describe why we want it? Because we've got much less of it than you would get in a traditional secure boot kind of a thing. Why do we wanna do away with it? What's the problem? Because secure boot sounds great.

Bryan Cantrill:

It's secure boot. Like what's your... I thought you liked security, Phil. What's the problem with secure boot?

Philip Tricca:

It's all the really, really important problem at the time, right? Like the secure boot came from UEFI and they had, you know, this was what 2010 around there when these things were starting to gain real traction, like the late odds. And this is an artifact of the way UEFI works. You can put an add on card into a PC system and your UEFI implementation will pull code out of it and load it into its single address space and run it. And so anyone who is trying to gain persistence and a foothold on one of these platforms would look for persistence in a PCI add on card.

Philip Tricca:

Right. Because then when you rebooted your platform, they took over UEFI. And so the security model, if that's your operational model, you're going to load code from an unknown origin into a single address space. If it's not signed, what else is it? Like you, that's your only option.

Philip Tricca:

And it worked great in those situations, but in, you know, with enough support from the hardware, we don't need that. If we can get a guarantee from the hardware that it is going to make evident the first code that was loaded and executed, we care less about whether or not that's code that came from us. If as long as we can interrogate the platform and see that it is running code that we don't know, then we place as much trust in that as we would any unknown code. Right. And you don't get that from a verified boot implementation like secure You get a guarantee from the platform and if the platform comes up and it's running, then the code that it loaded must have been signed, right?

Philip Tricca:

Right. Yeah. Probably. It can't prove it to you though at runtime. And that's really what we want.

Philip Tricca:

I look at this platform as an opportunity to make it an actual system that is much more transparent and therefore trustworthy than the systems that are out there now. All of the opacity, all of the secrecy in firmware is hiding more problems than it's solving. And if we can get that level of transparency and support from

Bryan Cantrill:

the hardware, we probably won't need to rely on things like verified boot for forever. And this kind of implication of transparency implying trust, which is I think very core to what we believe at Oxide. Mean, we are really trying to give folks transparency into this part of the system that you don't have transparency. Even we are not able to ultimately, we have to take a platform bundle from AMD, for example, that's an encrypted bundle that runs on the hidden cores, non architectural cores on package. Don't have visibility.

Bryan Cantrill:

We'd love to, but we don't have visibility into that. So there are limits to what we can do, but we are really trying to give people maximum transparency into the system.

Philip Tricca:

Absolutely. Yeah. Transparency is a prerequisite for trust in these situations. You have to know what you're trusting. And the trust model, unfortunately with PKI is, you know, if it's a signature from someone you trust, therefore you trust the thing.

Philip Tricca:

And that's why I think of it as being kind of disappointing when it comes down to it is because it always boils down to PKI.

Bryan Cantrill:

It always boils down to PKI. Yeah, that's a good point. I guess that when we pride ourselves on the transparency and yet the transparency ultimately stubs out in this opacity where like, we're not even gonna tell you like which city the bank is in. So it's like, sorry, pal, we're just not gonna be.

Philip Tricca:

But we can be transparent about a lot of the process. We can be transparent about so much of the process. I don't mean to sound like a defeatist when I point that out, but it, yeah, it is definitely, you know, transparency is definitely the root of trust and we want to, you know, we wanna not have to rest and rely on our reputation as a company as a way to win trust from Because number one, we're so new. We don't have a whole lot of history and track record that we can rely upon to say, hey, trust us. Yeah, this stuff is good.

Philip Tricca:

I look at this as we really have something to prove. And it is proving that people can trust us and we can do that technologically. We can do a lot of that with technology, but some of it just requires making visible all of the ceremonies, the scripts so that they could be audited and people can see where we, you know, what happened when we do these things. And a lot of what's in these scripts are us making mistakes. It's us doing a process for the first time, doing something wrong, saying it out loud, like being transparent about it.

Bryan Cantrill:

Yeah, sure.

Philip Tricca:

And you know, you swallow your pride, you realize you didn't do exactly what you had intended to, or you didn't realize exactly what was happening, but you have enough of a process so it's not catastrophic. You know, you just swallow your pride and write it down on the script and say, oops, sorry, I should have done it this way.

Bryan Cantrill:

You

Philip Tricca:

know, anyone who comes along and audits it afterwards, they can look at it and

Bryan Cantrill:

And understand that. Yeah, yeah, That's very interesting. Okay, so we run through the ceremony. This is before we're gonna ship Gimlet. So this is 2023.

Bryan Cantrill:

Yep. And we're saved. We've done this just in time to ship. And everything went perfectly. Everything went perfectly.

Bryan Cantrill:

It's great. And then I think it's like, is it two days after the fact? When is it after the fact?

Philip Tricca:

No, it was immediately.

Bryan Cantrill:

Oh, immediately? Okay.

Philip Tricca:

Yeah. This a huge, huge learning experience for me when it comes to having to operate, I don't know, in meat space, like just physically, And that's where it's doors and corners. It's at where things meet up and that's where things go wrong. And that was the first day that I had ever seen one of the tamper evident packaging we were using.

Bryan Cantrill:

Okay, so we had tamper evident packaging because we described the story earlier. We're gonna give a shards of this to folks of something that's been printed out by our beloved, but now deceased Dot Matrix printer. And we're gonna put that in a tamper evident bag. Yep. But we, I mean, this thing's all coming together so quickly.

Bryan Cantrill:

Right. And it's not like we've got like, well, let's just go to our trusted tamper evident bag supplier, the one that we always use. Because like we're like, this shit, it's like the first time we're trying to figure it out.

Philip Tricca:

Yeah, yeah. And the thing that was really, that was in retrospect was the thing that was really hard for me was that I was trying to explain a process and the responsibilities to different people that were filling different roles. Interesting. For a process that I was, you know, we had written all this down, but it was still very new. Had never gone through this.

Philip Tricca:

So I'm trying to explain, you know, how to do this stuff, how to use these things. These are people that are, you know, taking a sensitive secret on a piece of paper out of a little thing, out of the tray on the printer. You don't want everyone to be able to see them do this. So everyone has to take these things and put them in a tamper evident bag themselves and seal the bag. Now there are no instructions on how to actually use the tamper evident package.

Bryan Cantrill:

Right, does not come with a manual.

Philip Tricca:

It doesn't come with a manual, it just comes in a bag. Right,

Bryan Cantrill:

it's like, listen, it's tamper evident, not foolproof. Doesn't say there's foolproof on here anywhere,

Philip Tricca:

And by the so, you know, we used them to the best that we we knew how. I... The thing that stuck in my head and I was I was trying to balance questioning every motion I was making that day Yeah. Which is exhausting

Bryan Cantrill:

Yeah. Totally.

Philip Tricca:

With a, just trust the process, you wrote it down, have some confidence, just let it happen. And I'd been watching people use these, the tamper evident packaging during the day. And so people were looking at them kind of funny. Like, you know, they're make, they're going to do the thing and then they kind of make that screw face and they're like, what

Adam Leventhal:

the hell?

Philip Tricca:

Okay, well, whatever that should work.

Bryan Cantrill:

I'm just gonna like

Philip Tricca:

And I saw that and it just kind of stuck in my head and I was like, no, no, no, just keep going. We gotta get through this. We gotta get through this. Because there's a hard deadline, right? We keep this in a bank.

Philip Tricca:

It goes into vault, like into their safe deposit box vault, which means we have to be there before they close. Right. And if it doesn't go into the vault, then you're stuck with it overnight. And you got two people sitting there looking at a laptop on each other until the bank opens up

Adam Leventhal:

I the next

Bryan Cantrill:

think, you know, for the reason that had not occurred to me that like, oh, it's close. Like you go back in the morning. You go back in the morning. Oh, we... I would love to go back in the morning.

Bryan Cantrill:

It's it's gonna be a long night, friend. It's gonna be just you and me staring at one another with this revolver here in

Bryan Cantrill:

the mid between us. That's six pack of two. Totally.

Bryan Cantrill:

That's right. So okay, so you got and we can't talk about like the rigor versus urgency trade off or just like, no, we've got like real urgency, like bank closes. Yep. Yeah. Okay.

Philip Tricca:

So there were a lot of ways this could have gone wrong and it wasn't the getting to the bank on time. We did that fine.

Bryan Cantrill:

Right.

Philip Tricca:

The thing that we didn't do fine was the tamper evident packaging. The tamper evident bags had... There was a particular intricacy to the way that you remove the the thing protecting this... The the, you know, the adhesive seal on it. We did it wrong and they were just plastic bags.

Philip Tricca:

I mean, it

Bryan Cantrill:

feels like a tamper evident bag has a bit of a social contract that's been violated here. Don't you think so? I

Adam Leventhal:

mean This is

Philip Tricca:

like a failed sex

Adam Leventhal:

ed class. Like, this is this is like, nope. That was not the banana. I'm sorry. Like, you did it wrong.

Bryan Cantrill:

You did it wrong. That's actually Yeah. You now have a surprise pregnancy.

Philip Tricca:

I've seen people making this face as they were looking at it and I could see them doing that whole like trust the process thing. As I was driving home that night, it really bugged me. Was like, I can't let this go. I'm not gonna sleep tonight. Yeah.

Philip Tricca:

If I don't, if I didn't look at this. I literally just turned my car around and went to the office.

Bryan Cantrill:

Oh yeah, wow.

Philip Tricca:

And I came back to the office, Steve was still here. I grabbed one of the bags, I do the thing. I look to see him like I could just seal and reseal this thing. This is just a plastic bag I'm

Bryan Cantrill:

tampering with it right now. Yeah, am tampering the hell out of this bag. Yeah.

Adam Leventhal:

It's not evident.

Philip Tricca:

Well, was evident to me at the time, but Interesting. It was a problem. And, you know, those are... That's definitely a test of conviction, I guess, at that point.

Bryan Cantrill:

So... Okay. Alright. So now we have... Okay.

Bryan Cantrill:

So just to put the risk into perspective here. So like we have this sharded thing that's out. We've got a bunch of trusted people that have it. You need to have M invented those. But now many of these are just in plastic bags.

Philip Tricca:

Yep, it was just as though they hadn't used them if they just put it into their backpack and take it home.

Bryan Cantrill:

Just to open the backpack and take it home. And I mean, from your perspective, it's like, I mean, it's not like the kind of the concrete risk that like, okay, so the enemy infiltrates the company and begins like one by one going and tampering with these things in a way that's tampering on evident. But I think from your perspective, it's more like, no, no, this is just like, we know this is wrong.

Philip Tricca:

A lot of, like I was saying earlier about this notion of an audit, the idea is for this thing to be able to survive an audit. And if, you know, I cannot imagine trying to make the case to someone who, you know, a serious person who would accept an argument that, oh, well, you know, the password to act to use these keys on the HSM, they were effectively unprotected for twenty four hours. Yeah. I don't really... It's not an argument I'm gonna make.

Bryan Cantrill:

Right. Because even like the like, okay, everyone bring them back to the office. We're gonna reseal them in a... But like, no, it's like, because it's not tamper It's like,

Philip Tricca:

yeah. It's the whole point.

Bryan Cantrill:

It's the whole point. And

Philip Tricca:

yeah, it's a test of conviction at that point. It's a, how serious about this are you? We very much could have done that. And I think, I really don't think in hindsight that it would have been indefensible. I think we could have made the case and it's about whether or not customers and auditors are willing to accept that as a reasonable amount of risk.

Bryan Cantrill:

And

Philip Tricca:

I can't imagine, if someone is really, it is feasible that someone would be willing to accept that as a customer. I don't want any of our customers to have to accept that though.

Bryan Cantrill:

That's right. Yeah, yeah, yeah. I just felt like also it's something where we knew we could do better.

Philip Tricca:

Exactly.

Bryan Cantrill:

And it's like, always assume there are gonna be elements of this that we didn't anticipate. And like, that's always a possibility.

Philip Tricca:

And there certainly were, yeah.

Bryan Cantrill:

Where it's like, what do you mean that dot matrix printer was loaded with well known CVEs and had a secret WiFi. Or as Adam had said in the chat earlier, like Huawei makes printers? Always those things can happen. But boy, when you've got something that like, actually, we've spent so much emotional energy on this. It just feels wrong to like, not be able to do this to the absolute best of our ability.

Philip Tricca:

Yeah, I feel like this was just being serious

Bryan Cantrill:

about it.

Adam Leventhal:

This is

Philip Tricca:

really just about taking it seriously. Totally. You know, Steve and I had the conversation when I came back to the office. He was still here of course, because I'd leave, like I'm back and the CEO's still sitting here banging out work. And I break the news to him and he just, you know, very reasonable looks at me and he's like, do we have to do this again?

Philip Tricca:

And I look at him and I'm gonna say, yes, we we really should. And he's like, do have to? I was like, yes, Steve. Have to.

Adam Leventhal:

Did he ask for a third time?

Philip Tricca:

And he's

Bryan Cantrill:

just like,

Philip Tricca:

yeah, no, that's fine. He's like, all right, cool.

Bryan Cantrill:

We'll do it. We'll do it. No. I think that... Which is...

Bryan Cantrill:

I mean, you wanna, like, pull it out a little bit, but not more than that. You wanna, like, just make... Yeah. You wanna... We were like, okay, we're gonna do it again.

Bryan Cantrill:

So We And we did. So we have... We... So a second dot matrix printer called called to the front.

Philip Tricca:

Two printers.

Bryan Cantrill:

Two two printers. The Tomax joining Zama to make a a a beloved Oxide and Friends reference.

Adam Leventhal:

And death will not be Your death will not be invade. This time, I mean it.

Bryan Cantrill:

Exactly. Well, know, that that that dot matrix person sitting on the shelf was sharing some of the pain of its twin. So it arrives. And we... Which I also think is like, honestly, good...

Bryan Cantrill:

Just it was kinda good health to run this through again. I mean, it was... In many ways, like, a second opportunity to run all of this gave us that. Because I think you were able to go some of the... Because you said like there were mistakes that we made when we were doing the ceremony.

Bryan Cantrill:

It's nice to be

Adam Leventhal:

able to go kind

Bryan Cantrill:

of clean those up as well.

Philip Tricca:

They get better every time we do it. Every time we run a ceremony, every time we write a new script, it gets better. And by better, I mean, it's more smooth, it's more natural. People, it's more evident and obvious the things that should happen at the given times. And also the people that are in these roles are getting more comfortable with doing it.

Bryan Cantrill:

Getting more comfortable with it. Yeah. Did we, we now understand the tamper evident bag. We're now scholars of the tamper evident bag.

Philip Tricca:

Yeah. Tamper evident packaging is so weird. Like, know, you have to believe it has some value if you're using it. Yeah. You can't just say like, there's no reason to use this tamper evident packaging, but like, you know, I'm looking forward to a day when we could buy the expensive stuff.

Bryan Cantrill:

We've really arrived. We've... I don't know. The the $445,000,000 apparently is not enough. So I don't know.

Bryan Cantrill:

We're gonna... We're gonna... We'll get real Tampa Avenue packaging.

Philip Tricca:

Yeah. Like I said, this is this is how I'm getting funding for it.

Bryan Cantrill:

There you go. Exactly. Public shaming. It works very well. And then as we've done...

Bryan Cantrill:

And I think we did the the... We... Recently, r d five eighty five, which you wrote on the Cosmo ceremony where we got updated a bunch of the stuff and folks can can see some of the details in there. Mean, redacted some of the details that aren't like necessary that we think to provide security.

Philip Tricca:

I think in the future we'll be able to be more transparent about this stuff, but it's still coming together and we do have to redact some of those things because, you know, it's weird to think of it, but I I can imagine that we're a pretty appealing target, you know, out in the world.

Bryan Cantrill:

Yeah. We've made the big time. Have you not have you not have you not listened to the podcast? Are you not a podcast listener? We've made the big time now.

Bryan Cantrill:

Been, mean, unsurprisingly, I've just been inundated with people who are convinced we've made the big time and mainly wanna sell us various things. But yes, we've become a bigger target.

Philip Tricca:

Yeah, but 585 is a good example of the process evolving. Like we've, the printer had its shortcomings and wasn't very practical, but still having printed media come out of the thing is useful. And so there's some discussion in there about the selection criteria for a smaller printer that we use that actually fits in the safe deposit box. I mean, and finding one that didn't have any wifi or Bluetooth in it, like there is literally one out there that I could find.

Adam Leventhal:

Wow.

Philip Tricca:

Yeah. So there's the criteria and the reason for going about doing that is interesting and useful. And label printer makes this so much easier.

Adam Leventhal:

Yeah, interesting.

Bryan Cantrill:

So because the printer doesn't need to die. The printer gets to go into the safe... Basically gets to be physically secured. Yes. Yeah, that's great.

Bryan Cantrill:

Interesting. So that's just that is helpful from the mechanics of it

Philip Tricca:

all. Yeah, it lives in tamper evident packaging in a safe box.

Bryan Cantrill:

Yeah, that's great. But this continues to be, I mean, we have evolved it, but I think no regrets about what we did to ship those first.

Philip Tricca:

Other than the emotional scars that came with it, yeah.

Bryan Cantrill:

Yeah, well, I mean, I think you always have these things where you are scrambling a little bit to get to where you need to be. But this is what... And this was a tough one because like this was, I mean, on the one hand urgency and rigor were intention, but we actually needed like healthy doses of both.

Philip Tricca:

Yeah, this one this one was ride or die. This one was was worth it. I'll joke about, you know, uncomfortable it was, but like, it was absolutely worth it. And I have no regrets on the time that I sunk into it. I think it's, it makes our story and our kind of the trust that we're trying to gain and win from our customers, a lot of it rests on this.

Bryan Cantrill:

And

Philip Tricca:

I think that's just the idea is to not have it be the short pole.

Bryan Cantrill:

Yeah, exactly. That's right. Well, I mean, we wanted to talk about this for a long time, but this is specifically prompted because we're talking about people who've had a lot of great questions about the RRT and I had this FAQ Friday, we were talking about the RRT on Friday. Someone And asked a great question of like, how do you prevent someone from impersonating Oxide? And that's like, oh, I'm glad you asked.

Philip Tricca:

Yeah, this is it.

Bryan Cantrill:

A dead dot matrix printer is very glad you asked because they did not die in vain. And then we've done a... There's a bunch of stuff and I think we wanna probably have a future episode to talk about some of the things we've done in that manufacturing flow in terms of making that... Because that manufacturing flow, that we could accept that thing being super choppy for those, but we actually also want to really improve that. We had the kind of the security and the PKI, but there are a whole bunch of things that we've done in the last two, three years to make that a much, much, much better process.

Philip Tricca:

Yeah. Yeah. I really enjoyed working on the manufacturing, the tooling side of the manufacturing stuff. It's really like, it is the foundation of the way that we build trust in these systems. And like, it's just super interesting to work on it.

Philip Tricca:

Like, that's fun, I guess, is really what it just comes down to.

Bryan Cantrill:

That is so great that it is... I mean, the purpose of this company has been to attract people with a warped idea of fun. So that is, I feel we have succeeded in this regard. But no, I mean, it's super interesting and very, very important because as you say, it's like, this is where it all, it's where kind of it all has to happen in that manufacturing flow.

Philip Tricca:

And

Bryan Cantrill:

manufacturing is obviously extraordinarily important. We need to have high throughput and we need to be able to do everything else. So it's like the stakes are high for the kind of the services that we run there and being able to actually get a... And I don't know, did you wanna speak to some of the stuff that we've done in the last couple of years? Because there's been a lot of work from a lot of folks in there.

Philip Tricca:

Oh, there's tons of it. I mean, like you said, when we were first programming the platform identity keys into these systems, the ROT didn't do much. It does a lot more now. We're using this as part of constructing bootstrapping trust between individual nodes in the larger system.

Bryan Cantrill:

So that's see our trust quorum episode, really important stuff there.

Philip Tricca:

Absolutely. Yeah. And it's starting to get some interest from customers. We have some very security minded customers who are really excited about this stuff. We started exposing it up to the virtual machine layer.

Bryan Cantrill:

See RFD605 on that. I'm not sure if you wanna plug RFD605 at all, but that's been a really great one to get out there publicly.

Philip Tricca:

Yeah, for sure. Yeah, so the, I think the way I described this was just not on my bingo card. I had expected to live a nice quiet life of killing printers and building this kind of plumbing. Because these things are, attestation of restructures, in my experience, does not get exposed very far out. But the goal is to be as transparent as possible.

Bryan Cantrill:

That's right.

Philip Tricca:

To give people primitives so that they can build systems where they don't blindly trust these types of things. And this is it. And we had customers that wanted it. And so now we're exposing it up to the virtual machines.

Bryan Cantrill:

Well, and I don't think this is a deep thought, but the, I mean, maybe it's a bit of a surprise about how good these LLMs have become at finding very sophisticated vulnerabilities. And I mean, clearly, it's like part of the reason we started the company and what drew a bunch of us here is really taking the security at this layer of the stack really seriously. And I think one of the things that we couldn't have anticipated, but is that because we've always viewed these vulnerabilities as being really devastating and important and not enough people focusing on them. And also like, I I I just... It's never been in our nature to kind of sensationalize these exploits for purpose...

Bryan Cantrill:

For... I mean, you do get these security companies that like really love to like bang the drum whenever there is a vulnerability showing the reason that they exist. I get it, but I also don't like sensationalizing that stuff. Also, if we sensationalized every UEFI vulnerability, I mean, we would be hoarse There are so many vulnerabilities at this deep point of the stack. But as the LMs are now putting these kind of what were formerly kind of state actor level attacks in potentially much broader hands.

Bryan Cantrill:

So it's like, I feel, I mean, must feel like vindicating, not that you needed it, but like the security at this layer of the stack has become much more important even in the last year.

Philip Tricca:

Yeah, these are weird times. Like these are really interesting times on this front. These are like in a way kind of like nuclear weapons where there are entire industries of incredibly intelligent and experienced people that would spend tons of time hunting for these things. And in a matter of ten minutes, an LLM can take an old code base, flip it over and shake all the change out of it. It is, and this is where things like UEFI really suffer is because they are from, and they were designed in very different times.

Bryan Cantrill:

Very different times.

Philip Tricca:

And the incentive unfortunately is a bit upside down there. Like the incentive for people to change that is very low. And the people that care about it are very few. Now, course, people care about it when it goes wrong. But most of the time, like people are doing the risk analysis that says like, oh, the bios on my system was built twenty five years ago and the architecture of it hasn't changed at all to use any of the security infrastructure or to use most of the security infrastructure that's available to it.

Philip Tricca:

Like that is a perfect storm. Like when put into the hands of pretty much anybody, the ability to do what would have taken five experienced vulnerability, know, kind pen tester types and be able to do it in five minutes with no prior knowledge is terrifying.

Bryan Cantrill:

It's scary.

Philip Tricca:

Yeah.

Bryan Cantrill:

And I think, to the degree that we're now all signing up in the light cone reserves, Adam, to defend the light cone against our robot overlords, this kind of security becomes really important because this is the kind of the physical security of the system. And I mean, I do think... I mean, I think it's honestly, this is a part of this I feel is like extremely healthy that we should be focused on critical infrastructure and the security of critical infrastructure and municipal infrastructure and federal and state local government infrastructure and the infrastructure that we need. I mean, there's like, yeah, it's actually really important. And part of that is folks having infrastructure that they know that they can trust and that this is all right in that kind of bullseye.

Philip Tricca:

Yeah, this goes back to transparency. Like not just that, you know, people that have, you know, it's not that we want to just give people hardware that they know they can trust, but we want to give them a reason to trust that we want to give them some activity that they can do to prove to themselves that it is in a state that we intend it to be in. That's hopefully a game changer. That's the idea. And I mean, even just the ability for these systems for us to update these systems, we've talked a lot about like being able to update our way into the future.

Philip Tricca:

A lot of the things that are difficult to update oftentimes are the things that we say like, okay, well we're just not gonna update that thing. In this new world, I don't think that's gonna last.

Bryan Cantrill:

I think you're right. That's just not an option.

Philip Tricca:

You have to be dynamic. You have to be able to not just to be able to update them, but they have to be able to be updated fairly quickly.

Bryan Cantrill:

Fairly quickly. Yeah. I know media, it's kinda funny because you think, you know, and we've talked a lot about, like, not having a UEFI bias and all of the reasons we did that. And there are, like, many, many, many great reasons to do it. One of the positive consequences of that, one of the many positive consequences is that the lowest level platform neutralization software, update with every software update.

Bryan Cantrill:

We update all the time. And which is, I think in this new world where people are getting a new light, very bright light on these new vulnerabilities. The idea of just like just leaving it alone because it's too scary because your BIOS is telling you not to power off the computer while you update it. Very, like why are you saying that?

Philip Tricca:

The world of don't update your BIOS unless you have a hardware problem or hardware is malfunctioning, that hasn't been the world we live in for a long time. I think a lot of people are realizing that now. Just because anyone can use these tools and the fact of the matter means that that usually ends up meaning that responsible disclosure of processes often don't get followed. Because the people that are using these tools, they don't have that long history of building up that skill set and knowing how to interact with the other companies and stakeholders.

Bryan Cantrill:

Yeah. Well, Phil, are very grateful for your hard work on this. Thank you for it. I mean, you were put right in the kiln in 2023.

Philip Tricca:

Yeah. I got exactly what I was at, what I asked for. Funny how that goes.

Bryan Cantrill:

Yeah. Know exactly. Listen, you care about this stuff. There you go. But it was honestly really terrific and a lot of fun to watch the team.

Bryan Cantrill:

You know, everyone kind of pulled together to to get this thing working and then to go... And then with that done, to go build all of the stuff on top of it in the year since. So I think we're getting there. Thank you for assuming success. That was so dangerous of you to to assume we'd pull it off because we actually are pulling it off, which is really exciting.

Philip Tricca:

Yeah, this is great. This is so much fun.

Bryan Cantrill:

Awesome. All right. Well, Adam, thank you very much. And Adam, we're gonna be doing books in the box at some point. Yeah.

Bryan Cantrill:

Was kinda... So I just... This may be your your one or two week warning books in the box. Because I I think we've we've read a lot of good ones this year. Yeah.

Bryan Cantrill:

But I know That should be fun. That up in the fall. So... And if you haven't, go reread Bad Blood and definitely go watch. You can can see everything because it's really extraordinary.

Bryan Cantrill:

I wanna... Oh, you only need to get... I wanna get... And you can add this to Morris Chang. I wanna get copy to Vanian on here as a as a guest.

Adam Leventhal:

Sure. That'd be great. Let's do that.

Bryan Cantrill:

Let's do that. So that... That's what I'm gonna go. I... And I'm I'm on it.

Bryan Cantrill:

I'm gonna go. Good. Good. Next week. Was gonna go...

Bryan Cantrill:

Maybe. That's right. Next week. I wanna go see reaction to all this stuff. But, anyway, awesome.

Bryan Cantrill:

Phil, thank you again. Adam, thank you. And thanks, everyone. We'll see you next time.