The value of transparency in engineering can have huge benefits--nothing can compare to the momentum of an enthusiastic community! Bryan and Adam discuss the value of transparency at the hardware/software interface with Oxide colleague, Ryan Goodfellow. Transparency can be scary--especially in the hardware domain where secrecy is the norm--but once we knock down some of those fears, the business benefits start to emerge.
In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleague, Ryan Goodfellow.
In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleague, Ryan Goodfellow.
Some of the topics we hit on, in the order that we hit them:
- Oxide RFD 552: Transparency in Hardware/Software Interfaces
- Belling the cat
- openSIL
- Kerckhoff's principle
- PRs needed!
If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!
Creators & Guests
What is Oxide and Friends?
Oxide hosts a weekly Discord show where we discuss a wide range of topics: computer history, startups, Oxide hardware bringup, and other topics du jour. These are the recordings in podcast form.
Join us live (usually Mondays at 5pm PT) https://discord.gg/gcQxNHAKCB
Subscribe to our calendar: https://calendar.google.com/calendar/ical/c_318925f4185aa71c4524d0d6127f31058c9e21f29f017d48a0fca6f564969cd0%40group.calendar.google.com/public/basic.ics
Craig and Gjarc are here.
Bryan Cantrill:Excellent. And we we are both speaking with my voice. We're after a fashion. After a fashion. So we live from the litter box.
Adam Leventhal:Live from the litter box.
Bryan Cantrill:We we we are both both here in person.
Adam Leventhal:Yeah.
Bryan Cantrill:Last time we were in person, we got you to confess to things that we don't think are crimes. The the Dave Lightman. So I'm looking forward to, you know, maybe we'll return the favor. I'm not sure. But, you know, maybe it's the it's the in person body language that says, no.
Bryan Cantrill:No. Now is the time. But but I'm closer. Before here.
Adam Leventhal:God. I had forgotten all about that.
Bryan Cantrill:Oh, god. I I just I I I think about that. And I mean, I know we laugh for our own jokes, but I just think about your your your confession. Well, it's deathbed confession or it's when you are faking a medical emergency in the deposition. So oh, a bunch of things.
Bryan Cantrill:So we got this RFD that Yes. That are so I and I did as I have been doing. Do you use you use ChatGPT to review documents that you've been
Adam Leventhal:Yeah. For sure. I used ChatGPT to, review form nine ninety just just last night.
Bryan Cantrill:I when I was walking around like a maniac. I really? Yes. That's how did it do? That's actually that's a great use for GPT.
Adam Leventhal:It it was very good. I mean, it's good it's good for these kinds
Bryan Cantrill:of The the
Adam Leventhal:formal documents. That's really good. And I was trying to understand if my little league is corrupt or merely merely seems annoying. And And? TPT.
Adam Leventhal:I'm not sure it yeah. I'm not sure I'm smart enough to understand what chat TPT told me.
Bryan Cantrill:You you didn't find the the hidden disclosure in there that brings down LLM the little league?
Adam Leventhal:Not yet. No. Searching. That as as Deep Throat passes you an LLM.
Bryan Cantrill:That's right. So I'd I'd I use it when I like, when I'm writing a document. I am I like to use it to review documents. I find it to be, really useful in that regard.
Adam Leventhal:Like review what you've written?
Bryan Cantrill:Or review what I've written. Okay. Yeah. Do wanna do this? I do.
Bryan Cantrill:I do. For sure. I find this to be so I started doing this because, Bridget made it clear that she was not interested in she that her period of time on this planet of listening to me read blog entries aloud before I posted them had basically ended. So it was, you know, the that's fine.
Adam Leventhal:Marriage seems like the right time.
Bryan Cantrill:Twenty. Is that what
Adam Leventhal:you said? Twenty.
Bryan Cantrill:Yeah. Yeah. Twenty years of marriage. Yeah. No.
Bryan Cantrill:It's exactly. She gave me two decades and, you know, she wants the rest back. I can't blame her. So I I and I and I think I actually use it a little too late in my editing process because it is filled with too much praise. I feel it's like, wow.
Bryan Cantrill:This is really amazing stuff you got here. I mean, it's like, come on. I'd say tend I mean, I know it tends to kiss ass a little bit. I feel the do you feel this? A lot.
Adam Leventhal:A lot. A lot. And it's
Bryan Cantrill:Well, I think we're but let's not discount. It's I mean, it's made some good observations, I feel. Just my number one fan. It's just my number one fan. Yeah.
Bryan Cantrill:It does. It it tends to to, to kiss a lot
Adam Leventhal:of ass. Yeah. You know, I I think my fear of using it is that I am that I will be too influenced by it. That, like, that
Bryan Cantrill:it'll it it
Adam Leventhal:will get away from me, and suddenly, will have I hate. Become the thing I hate.
Bryan Cantrill:You know, yeah, I don't think you have worry about that because it doesn't the the feedback that it gives is actually pretty good. It is it tends to be like, hey. You, like, you could use a transition sentence here Yeah. Or I felt like this idea could be slightly like, it well, actually, it it only does that though after falling over itself to praise you. It's like, I mean, if I had against my own will, if I were if I had to find a single issue with this, it may be that you've misspelled these eight words.
Bryan Cantrill:But the actually, it doesn't find spelling at all. Spelling is terrible. Yeah. It is it does not do well on spelling. No.
Bryan Cantrill:Yeah. Yeah. It really doesn't.
Adam Leventhal:Then so Simon Wilson was tell telling us.
Bryan Cantrill:Yeah. Yeah. They it doesn't do
Adam Leventhal:well on low pass filter of tokenization first.
Bryan Cantrill:So Yeah. Yeah. Yeah. So it and doesn't do well on grammar either. I I really, you know although I did.
Bryan Cantrill:I I definitely like because and now I've become very self conscious that I'm, like, using expressions that only that because people will tell me like, oh, yeah. That's that's definitely a you ism. I'll be like, no. That's not like
Adam Leventhal:No. Everyone's been saying that for the past twenty years again.
Bryan Cantrill:Yeah. Right. No. That's been you've been saying that for the twenty years. So now I become very self conscious, but I find that ChatGPT is, like, a good way to just check that.
Bryan Cantrill:Right? Right. This is the I I suggested to, an oxide engineer who will remain nameless that he really needed to bail the cat on this. And he's like, what? You mean bell what?
Bryan Cantrill:Are you having a stroke? And I kind of do you but if I say bell the cat, do you know do know what I mean?
Adam Leventhal:Yes. I think. Wait.
Bryan Cantrill:Are you being like Chatty Paty? You're kissing my ass right now.
Adam Leventhal:What a great great.
Bryan Cantrill:What an excellent allegory.
Adam Leventhal:So what I think what I have always understood that to mean Yeah. Whether from you or from someone else is that the the cat is like going around murdering birds and belling the cat is like, you know, like putting safeguards on it.
Bryan Cantrill:That's right. Okay. So it is actually it's mice. Sure. But the that's exactly it.
Bryan Cantrill:And the the mice wish to, have the idea. And I think this this should come up, like, a lot more frequently, belling the cat. Because this is like someone has got the genius idea, like, should put a bell on the cat. K. And then we would know when the cat's coming.
Bryan Cantrill:Makes sense. It's where it's like, great. Who's gonna like, great genius. Like, someone needs to go put the bell on the cat. Yeah.
Bryan Cantrill:And it's like, that's the hard like, the hard part is not coming up with the idea of the bell. The hard part is putting the bell on the cat. Anyway, I did not invent belling the cat. And the the the belling the cat, actually Wikipedia page is really, really long. Delightfully long.
Adam Leventhal:Nice. Yeah. Definitely. Did you write it?
Bryan Cantrill:I'm not accepting follow-up questions at this time. No. It's got like it I mean, it's I mean, this is that's like a medieval piece of artwork in there. Mean, anyway, it's it's got a very, very deep history, doing that. I get concerned, and so I ask ChatGPT on those things.
Bryan Cantrill:And usually ChatGPT will say, well, that's no. That that's that's a pretty common expression or, that's a that's a little more of an unusual expression. And I did have one today where ChatGPT was trying its best to tell me that I wasn't insane, but it's basically like, this is an extremely unusual phraseology that I have never come across. Now, I know I've only consumed Everything on the internet. I've only consumed everything on the internet, most of it illegally.
Bryan Cantrill:So I yeah. You know, I don't know. Could be me.
Adam Leventhal:I I don't know. I I I could I could be wrong.
Bryan Cantrill:I could be wrong. But the, so then like, okay. I know that. So no. You should, you should do this.
Bryan Cantrill:It's worth doing. I think it's it's interesting and, so and I also like to, so it it was full of praise. This r f d didn't get a a whole Unlike your wife. Unlike my wife. Exactly.
Bryan Cantrill:That's right. It was it was the right decision. So, but I think it's actually gave me actually the feedback it gave me was was pretty good. It was not like, it reflected some some deeper understanding, so that was all good. I did wanna try this with Gemini.
Bryan Cantrill:Okay. Have you tried Gemini at all?
Adam Leventhal:No. I haven't.
Bryan Cantrill:So the, the only, like, taste of Gemini I've gotten is the stuff that you get, like, on search results and so on that is, like, absolutely atrocious. Procious. And I mean, atrocious. Atrocious. How did Google and I mean, it in particular gets something wrong that's really too vulgar for the podcast, which is saying something because I feel like we've, you know, we've talked about two headed showers in this podcast.
Bryan Cantrill:And the, which anyway, I I'm I'm not gonna go into further detail, but let's just say that it thought that something that was extremely vulgar is something that you would say to a toddler while putting on their shoes.
Adam Leventhal:And it's like How many ricochet was it in reviewing this r f d?
Bryan Cantrill:No. No. Right. Are you gonna stay with me? Are you not?
Bryan Cantrill:Just like I need you to stay with me. Okay. So Gem anyway, back to Gemini. That's why I I wanna use Gemini from Google. Makes sense.
Bryan Cantrill:Makes sense. So I actually like sign up for Gemini because you gotta like sign up for it and it's kind of like we couldn't do it on the oxide accounts. Did it anyway. Spin the button. Okay.
Bryan Cantrill:But I also really like to like do I think you do this too. Right? Do you I tell people not to anthropomorphize these things and then I anthropomorphize it all the time.
Adam Leventhal:I Please and thank you.
Bryan Cantrill:Please and thank you. Like, here's what I would like to do. What do you think about that?
Adam Leventhal:Like, Are you familiar with x?
Bryan Cantrill:Yeah. Right. And I just I just love how enthusiastic Wade always wants to it's always like, oh, yeah. God. Would love to review.
Bryan Cantrill:Oh my god. Reviewing an r f d? Yeah. I thought you'd never ask. So I did that with and you know we talked about, DeepSeek on Cerebras.
Bryan Cantrill:Yes. And looking inside of the id of the sort of thing
Adam Leventhal:Yes.
Bryan Cantrill:Was disturbing. Disturbing. Yeah. Disturbing. It's really the one word that comes to mind of, like, this is something that's gonna be, like, found out.
Bryan Cantrill:Like, I actually know nothing, and this person is about to find out that I know nothing. So, the I with with Gemini, it also shows you it's thinking. But it's actually it I just on the question of, hey. I'd like you to review a document. Can you do that for me?
Bryan Cantrill:It's thinking I mean, it almost suffers from, like, chronic anxiety. It's, like, very competent, but it's like, okay. I definitely can do this, and I wanna make like, but I don't wanna just leave it at yes because I really it really wanted to, like, close the sale. It was concerned that I would it would merely say yes, and then I would, like, walk away. Like, that would not be enthusiastic enough.
Bryan Cantrill:So it is just like, I've really gotta make the case for, like, this is, I can do this, and I can do it this way, and I can I can you know? And it was it's kind of, like, endearing a little bit. You know? It's like, wow. You kinda got some issues over there, Gemini.
Bryan Cantrill:Anyway, no one hugged you. No one hugged you. I feel like I I feel like no one hugged you, and, you know, maybe an early predecessor of yours asked Kevin Roos to leave his oh, I guess that's that's my that that was that was Microsoft. That was the name that was g b t four. So that was not even sorry, Gemini.
Bryan Cantrill:Anyway, so I the the I hadn't reviewed the software, and it was good. That that that that's where I've gone to this. I was using the deep research stuff too, which Short story long. Short story long. I was using deep research, which I think is, like, pretty good.
Bryan Cantrill:I've heard I've heard good things about it, and I think it's not as good as the things I've heard about it. Okay. Have you heard of Deep Research? I've heard of it. Haven't Everybody is calling the product Deep Research now.
Bryan Cantrill:So you got, like, OpenAI Deep Research. You got Gemini Deep Research, I think, was I think the first. And I think, again, it's pretty good. Do you ask it for things to, like, where you only have, like, industry rumor?
Adam Leventhal:No. I I like
Bryan Cantrill:Oh, you you should definitely. I mean, it so, like, asking it, like, why did Optane fail? Or why did what happened to Cannon Lake? Or I was asking what happened to the Memristor at the machine? And then I'm like, I want a research report on the failure of the machine at HPE.
Bryan Cantrill:And it was like, it was you know? It's pretty good. Mostly came from Wikipedia page, I think. But anyway.
Adam Leventhal:Yeah. Like any good research report.
Bryan Cantrill:But I definitely learned at part of this, and I I had posted this yesterday. I think you'd seen this that the, this is not their fault because this is in, like, I think 2012 that they started this. This next generation bus protocol Yes. Which I think they called the next generation messaging interface or something like that. Anyway, it was NGMI.
Bryan Cantrill:It's awesome. Which is great. And the NGMI the and and the NGMI is not gonna make it. I think only came later as part of cryptocurrency.
Adam Leventhal:Yeah. That makes that sounds right, but it's perfect.
Bryan Cantrill:It it it is. It is perfect. It does remind me of I I actually have mutual acquaintance of ours. Went to go work for a company named ISIS. They changed the company name.
Adam Leventhal:That's right.
Bryan Cantrill:You were Yeah. It's a tough one. ISIS is a tough one. Yeah. Fully timed.
Bryan Cantrill:Fully timed. Exactly. This is pre ISIS. Alright. Where are we?
Adam Leventhal:RFD. RFD. Okay. Your many reviewers.
Bryan Cantrill:And my many reviewers. Exactly. My my my many sycophantic reviewers.
Adam Leventhal:But but to close the loop
Bryan Cantrill:Yeah. Not to bell the cap though. Yeah. So
Adam Leventhal:which which which which one did you prefer in terms of its feedback? I mean, not not just which praised you more strenuously, but, like, which gave you I
Bryan Cantrill:think that Gemini could really learn something for the praise that you that that Chachi Pizzer. I would say it was valuable to have them both read it. Yeah. And it was not extraordinarily valuable to have either of them read it. Okay.
Bryan Cantrill:So I I don't really think I got any. I did not I definitely I got praise from both of them. No nothing that really changed the trajectory of it Yeah. At all from either of them, but they made different remarks, and it was actually kind of valuable. I think I probably made mod like, small tweaks based on both.
Bryan Cantrill:Yeah. It was interesting. Alright. Cool. So this is RFD.
Bryan Cantrill:So the and did you get a chance to?
Adam Leventhal:I read it. I read it. I have notes.
Bryan Cantrill:You seem so defensive. Seems so defensive. Alright. You've been Highlights. Can I I mean, just routine?
Bryan Cantrill:Can you see the notes? I mean, I understand. I don't know. You can't quite make them out from me here. That just looks like that just looks like scribbles.
Bryan Cantrill:Pretend writing. Pretend writing. Wait a minute. That's in fact, it even says this is pretend. No.
Bryan Cantrill:So the and I think you were out last week. I was. Yes. So I think you probably came back to, like, I have missed something.
Adam Leventhal:Well, you said you posted this. Yes. And I was kind of like, what is the antecedent? Like, assumed Paul Graham had done something horrible or something. A reasonable assumption.
Bryan Cantrill:Exactly. A very reasonable assumption.
Adam Leventhal:Yeah. But in No.
Bryan Cantrill:No. No.
Adam Leventhal:No. No. I was aware of.
Bryan Cantrill:I I not to the best of my knowledge, unless Paul Graham is making has started a silicon company. No. This came out of, I would say, repeated frustration that we have had. And I yes. I had some conversations last week that kind of boiled it over, but the repeated, I would say, disconnects that we have had with silicon providers.
Bryan Cantrill:So, I mean, to be clear, we make our own hardware, but we don't make our own chips.
Adam Leventhal:Yeah. Because, like, we don't have half a billion dollars.
Bryan Cantrill:That's right. That's why we don't make our own chips because we don't have half billion dollars. And making one's own ASICs is still really, really, really expensive.
Adam Leventhal:Much less expensive, like, than it used to be, but still really, really question. Well, we shouldn't have to build a fab.
Bryan Cantrill:You don't have to build a fab, but in the in the so, yes, assume you're in the fabless era.
Adam Leventhal:Yes.
Bryan Cantrill:But it would be interesting to know if a if a if a because, I mean, certainly, if you're gonna build, like, on an older node, I would assume that that's cheaper. Totally. Right. But honestly, the part of the what makes it expensive is the proprietary tool chain that you've got to get. Yeah.
Bryan Cantrill:You you gotta go like
Adam Leventhal:Especially for the for the most advanced node.
Bryan Cantrill:For most of it yeah. And so I that would be actually be interesting to know because I actually think that's kind of one of the important things that actually just to aside and aside, you know, we've talked about, like, kind of the moat that CUDA has, and I think that we, rightfully, the CUDA moat, I think, and it was, Andy and James that were saying this. Like, look, the the CUDA moat is not as deep as people think it is, which I absolutely think is the case. Yeah. I think the TSMC moat is deeper than you can possibly imagine.
Bryan Cantrill:I feel the same way. I mean, this
Adam Leventhal:is this is my this is why we've been trying so hard to get Morris to, to come on the show. We're just
Bryan Cantrill:I just the dates just don't line up. I mean, it's I and I mean, honestly, in the end, it's like it's the constraint is on us. Like, we he honestly if our calendars weren't quite
Adam Leventhal:as booked, we we could have made this work. We'd try to be flexible.
Bryan Cantrill:We'd to be flexible. That's okay. Gong goes, gonna miss so much. Exactly.
Adam Leventhal:No. I'm with you. I think that that that TSMC lead feels like
Bryan Cantrill:just they're twice over the horizon. Because, you know, it is and I think it's like so much deeper than we can really fathom. I was talking to a the a former colleague of ours who works for is doing an ASIC and trying to understand that they're doing it on TSMC, of course. Yeah. And, like, just getting into, like, all of the details.
Bryan Cantrill:First of it's the first decision you make.
Adam Leventhal:Who you're
Bryan Cantrill:Who the yeah. It is not like, oh, let's I mean, it is it is like which language are we gonna write this thing in? Yeah. It is not a decision that you, like, revisit. Yeah.
Bryan Cantrill:And, I mean, it's actually not a bad analog in terms of, like, the but there's only one programming language, and it's from TSMC. So, I mean, it you you make it very, very early. Then they give you a bunch of IP of surrounding Right. When we say IP, we mean, like, these blocks that do things. Right.
Bryan Cantrill:And there's an entire TSMC ecosystem that you get. And then you've got and and that there it's a whole kind of a cottage industry where they then help you integrate this stuff.
Adam Leventhal:And it's like you're if you're doing I squared c or if you're doing
Bryan Cantrill:This is on the actual sure, I squared c, but this is like even like inside of the ASIC.
Adam Leventhal:I see. Like, even even smaller kind of strange logic components.
Bryan Cantrill:That's right. That's right. That's right. And the, I mean, it and it is so like, that interface between you and TSMC is, like, porous. You know what I mean?
Bryan Cantrill:It's like it is a, it's not you don't pick up and move to a different process because you you would be moving to different libraries and everything else. So I actually believe more strongly than ever, and this is actually not totally unrelated to the hardware software interface transparency, I believe more strongly than ever that Intel's gotta do my crazy thing. Of an open sore all open source. All open. Yeah.
Bryan Cantrill:That they actually the only way for I for Intel to compete because, I mean, you gotta think, like, you know, if you you you and I are an ASIC startup and, you know, we've just or even, like, we're within a larger company, first decision we need to make is which foundry. It's like, we're gonna go roll our dice on Intel? Right. I mean No. It makes no sense.
Bryan Cantrill:It makes no sense. Right. I mean, you think it's like it's like the reticence that we had about Tofino times, you know, a million. Right. And For, like, all
Adam Leventhal:I mean, for so many reasons. Like, among them, like, on a company that is behind in terms of technology and hoping that miraculously they catch up and betting on a company which has flipped on everyone who's believed in them.
Bryan Cantrill:Everyone who's believed on them. Exactly. Who has and so, I mean, it's like we they really need to get to, I think, the their their only move. They've gotta do something that's, like, radical. And they are this is, I think, what is kind of the impedance mismatch right now.
Bryan Cantrill:It's like, they're not gonna be sold to CSMC. You you need to have someone who's gonna come in and be like, we're gonna die on this trajectory, and we have to do something really radical. And that radical thing is we're gonna open it all. So And I, Brian Cantrell, be your new CEO. Let's go.
Adam Leventhal:I love it. Doesn't it seem impossible? Just like from a from like a shareholder perspective and Yeah.
Bryan Cantrill:Yeah. I mean It it it you know what? I all avenues for Intel seem impossible right now. I'm with you. Like, clearly, an avenue is gonna be the avenue that it takes, but literally everything seems it seems impossible that they fail.
Bryan Cantrill:It seems impossible that they succeed. It's I I just I don't know. But I think that they it it seems possible that they open it all. Okay. Unlikely.
Bryan Cantrill:Unlikely.
Adam Leventhal:Exciting.
Bryan Cantrill:Exciting. But I believe that more strongly than ever. And so the but we're several layers up the stack. We're not inside of an ASIC. Right?
Bryan Cantrill:We are using an ASIC to do things. Exactly. And sometimes that ASIC is a really big ASIC. And and that, like, we an SOC, right, is a smart chip that we would use for, like, for AMD Right. The AMD Milan or or General or Turin, or it's a small ASIC the way it is for, like, our STM thirty two h seven fifty three microcontroller that we use for the service processor or the LPC 55 s 69 that we use for our root of trust, or it's the Tafina, which we use for our our switch silicon.
Bryan Cantrill:It's our NIC. I mean, it's like, oh, we're using silicon all over the place. And I just feel like I've had this conversation kind of with everybody at some level about, like, what we, Oxide, need. And I feel that what we need and we're always made to sound crazy when we and admittedly, like, maybe maybe there's some ways I could phrase it that would mean
Adam Leventhal:Or maybe volume that you could phrase it.
Bryan Cantrill:I mean okay. So, you know, I think that, like, maybe I could have done it in a way that would not have elicited a restraining order. But
Adam Leventhal:Oh, he's joking, folks. Exactly.
Bryan Cantrill:Come on. Come on. That was a joke. That was a joke. No.
Bryan Cantrill:I I I think that the, and but it's something that every that actually, like, the industry needs.
Adam Leventhal:I'm totally with you. And I have also, seen people filling out forms for for restraining orders. Like, I've been at some of these, like, SSD vendor summits. Raised my hand saying, you know, what about opening the the the firmware on this thing? Totally.
Adam Leventhal:Very popular idea.
Bryan Cantrill:Super everyone's super interested. Even
Adam Leventhal:even our colleague Robert with me at these events kind of shies away from me knowing that it builds additional credibility with everyone else in the room.
Bryan Cantrill:Okay. But so you know what? They you speaking of SSD vendors or actually hard drive vendors, a couple of these vendors would because I Robert and I were the same were like that in terms of, like, we would go to these events and I'd be like, hey, look, you might not wanna be you might not wanna sit close together because I'm gonna be I I feel a big one coming on. There we go. But the people inside the vendors would say, like, god, we love having you there because you are saying the things to our management that we're saying.
Bryan Cantrill:Yeah. And they're and, you know, when we say it, don't listen to us. But because this is a customer event, well, they don't still listen to you. And so it's like, they they, you know, they they don't listen to you, the, the the customer. So, I just think we end up having, we end up having this conversation a lot.
Bryan Cantrill:You had it with SSDs. Yeah. And it it it doesn't go over really well.
Adam Leventhal:No. Right? Right. And because they view this software hardware interface as part of the secret sauce. Right.
Adam Leventhal:Yes. And, like, part of the differentiation.
Bryan Cantrill:I guess that's it. I mean, I which is so so this is the thing that I that that obviously you and I both find frustrating. Yeah. It's like, what I oh, no. No.
Bryan Cantrill:Like, what what do you think I'm asking you to do? Like, do you think I'm asking you I'm not asking you to, like, put your RTL on the Internet or whatever. Right? I'm not asking you to eve I'm not asking you to, like, open your even your flash translation layer for the SSD vendor. I'm not asking you to open source it.
Bryan Cantrill:That'd great. That's what I'm asking you to do. Or maybe you were asking to do that.
Adam Leventhal:That's right. Bad example.
Bryan Cantrill:Bad example. But, like, what what I want you to I want to be able to write the lowest level software to this thing. Yeah. I want to like, I don't want any layer between me and the hardware. And that also gets like that itself gets kind of vague because the actual definite this is what we're getting like.
Bryan Cantrill:And and I'm I do I promise I'm not gonna do this? The hardware software thing? Like, what is hardware? What is software? I
Adam Leventhal:don't know. It's pretty good.
Bryan Cantrill:I don't know. Well, they were just like, it becomes really hard to differentiate hardware. So, like, what is what is software?
Adam Leventhal:Yeah. Especially when you are like, when when you're building this ASIC and, you know, parts of it are in a ROM or whatever, and it's like where, you know, where where exactly where does one part end and the other part begin?
Bryan Cantrill:Right. I mean, you've got, like so, I mean, I I mean, as an example, you've got I mean, x 86 is like, that's the instruction set Mhmm. Of the microprocessor. But it's actually, like, not the it's actually just the interface of the the right? It's like that gets translated to micro ops Right.
Bryan Cantrill:Inside of the microprocessor. But it is the interface of the microprocessor. So it's not but it's is that the hardware? I mean, it gets, it it just it it it gets sticky Yeah. To to kinda define one versus the other.
Bryan Cantrill:And but what we're actually looking for is not and, actually, I very carefully avoided the word open in this RFT because I think the word open throws people off.
Adam Leventhal:In the title?
Bryan Cantrill:More or less anywhere. I mean, I know, like, I the word open does exist in there, but, like, five times, not like like, the transparency is it is in the RFD quite a bit more than open. Yeah. And that is very deliberate because I feel that people stub their toe on open. And open because, I mean, it does mean about I mean, you got the
Adam Leventhal:right? Yeah. Yeah. It kind of is almost evocative of, like, a religious belief rather than, like, a a reasoned, you know, business value.
Bryan Cantrill:That's right. That's right. Even though I think you got so many different definitions of open. Got the open systems definition of, like, son from the it's we should, we should pay a little respect to our former employer. This is its 40 birthday.
Bryan Cantrill:Did you see that?
Adam Leventhal:I did see that, actually. I I It's kind of a posting It's
Bryan Cantrill:kind of fitting.
Adam Leventhal:Yeah. Backsides of the the meta sign, which
Bryan Cantrill:Yeah. Well, there was a there was a very nice LinkedIn post that that talked about a bunch of technologies, including ours. Oh, it was very cool. Yeah. That was very nice.
Adam Leventhal:I will tell you something funny. I went looking through my Google photos to find, like, my picture as an intern in front of that sign because I know I had it. Yeah. And? It's on film.
Adam Leventhal:Like, I know. So I don't actually have it in
Bryan Cantrill:Is it not developed?
Adam Leventhal:It's no. It's like it's like I have a hard copy
Bryan Cantrill:Oh, yeah. Yeah.
Adam Leventhal:In a book called a photo album.
Bryan Cantrill:Tell me more.
Adam Leventhal:Used to be
Bryan Cantrill:a physical analog. It's animes anyway. You should scan
Adam Leventhal:it, though. Sure. I could do that. Yes.
Bryan Cantrill:That'd be great. Or take a photo of it.
Adam Leventhal:Exactly.
Bryan Cantrill:But, yeah, I would love to see that sign. What kind of where are we on the kind of the hair length in that sign?
Adam Leventhal:I I was like I was an intern. I was like
Bryan Cantrill:But this is like Cleaned up. But this is like how we count the rings on the tree. This is how I know I'm able to age it by like by looking at the
Adam Leventhal:It was like two it was go go days. It was 2,000.
Bryan Cantrill:It was 2,000. That was the the summer you couldn't find housing. Yeah. What a what a great summer.
Adam Leventhal:Yeah. For some.
Bryan Cantrill:For some. Exactly. For those in the roof over their heads. But so we you've got kind of the Sun Microsystems open systems kind of era, that definition of open, which is really kind of the that's the definition of open we would be really thinking of here. That's right.
Adam Leventhal:And and there, it's not open source. It's like
Bryan Cantrill:Open protocols. Protocols. Yeah. Exactly. Right.
Bryan Cantrill:Right. And then you've got the so kind of that era. Then you obviously got open source. Yeah. And I think, again, I think open source, it just I think it conflates what we're asking.
Bryan Cantrill:Yeah. Because people think that, like, no. I wanna take you should take your software that you've written, chip vendor, and you should open all that. I mean, I'd for them to do that, but that's actually not what we're asking. Right?
Bryan Cantrill:We want to have the ability to not use it. We wanna have the ability to have that that lowest level, of interface. So that's why we we very deliberately chose Streak. And then, of course, you got a new a new era with OpenAI, now not, like, OpenAI. Meaningless Meaningless term, not, like, not OpenAI.
Bryan Cantrill:And then you got open weights and Yeah. And and so I feel like I wanted to get away from open very deliberately. Gotcha. And as I I I was talking with it with our colleague, Rai, I'm actually I'm hoping to get up on stage here. But I was talking with Rai about this, and I was joking that I'm like, what we need is an RFD, that is, like, hardware software interface is a reflection of values.
Bryan Cantrill:It's like, actually, I kinda like that. I'm like, okay. I'm not gonna do that one, but that's actually that's a good idea. We can need to we need to actually look a little bit more and and see if we can so went with transparency and then noodled a bunch with our colleagues, with Rai certainly and with Robert and with other folks about trying to think a little more in a little bit more of a structured fashion about what is the hardware software interface. And I like, even that we got hung up on.
Bryan Cantrill:I don't know what you what you think about how we did there.
Adam Leventhal:I thought the way you took it apart, it took candidly, it took me a couple of passes through, you know, just to, like, understand the taxonomy.
Bryan Cantrill:You know who didn't require a couple passes? Chat, you can click it. I know. Know. I know.
Adam Leventhal:And a lot more of a lot more praise praise that we do.
Bryan Cantrill:Exactly. Just not accustomed to this.
Adam Leventhal:Yeah. Exactly.
Bryan Cantrill:No. No. Okay. Yeah. So well, no.
Bryan Cantrill:Actually, that is kinda what I'm looking for because it is it's a little bit we don't really tax we haven't really taxonomized these interfaces, I don't think.
Adam Leventhal:Yeah. Right. And and also this is not like, candidly, a world where I operate personally day to day as much as you do.
Bryan Cantrill:Right. Yeah.
Adam Leventhal:I'm I'm higher up in the stack. So, like, some of these distinctions are a little subtle.
Bryan Cantrill:Yeah. And I think I think Rai is here. Rai, know that thank you for your, your inspiration here on on this RFD as we were kind of realizing that we need to better communicate what we're actually looking for because, again, this is something that we we just had sorry. So so sorry. You were talking about the the tax automization of the Oh, yeah.
Adam Leventhal:No. I thought it great. So you talked about instruction set architecture totally made sense to me, and then data interfaces and control interfaces, which just took took another read through to, like, understand that. But even in there, you know, you gave some fuzziness. Like, these are not, like, always crisp.
Bryan Cantrill:Right. In fact, they're rarely crisp. But the because I think that the reason for that is that people so these interfaces get treated differently. So if you look at, like, x 86 Yeah. The instruction set architecture has been really well documented Yeah.
Bryan Cantrill:Historically. And then there were kind of, like, some bumps along the way where that started getting, like, less well documented. What was much more of a black hole were the control interfaces for the part. And that's where you get SMM. That's where you get UEFI.
Bryan Cantrill:That's where you get the management engine on Intel. That's where you start to get these these divots and black holes that are not actually the thing that the computer does, but it's the way that the computer gets to computer. Like, you you actually don't get to computer if you don't do these things. You you gotta be able to boot. You gotta be able to, like, find devices before you've booted.
Bryan Cantrill:You've gotta be able to, and, well, actually, was gonna say, you've actually SMM has no reason to exist, so there's really no no way to justify that. But the, and that stuff was not the instruction set Mhmm. But was this other interface that was not as well documented.
Adam Leventhal:Right.
Bryan Cantrill:And that is actually those are the more problematic interfaces that that we've had.
Ryan Goodfellow:And I I think even for peripherals, I mean, I come from the networking space, and so that's that's the space that I live in. But, I think particularly for networking peripherals, like, that control phase is so important, because, like, you're not running code already on that device. Like, you're running code in the operating system that's running a control plane that's communicating with that device. And so getting all the data off of that device and for networking, doing things like loading tables and enabling SerDes and things like that, all the things that we need to do to make a network actually function, depend on that control interface. And so it's just so critical to if you wanna actually understand how your system is working beneath you, then you have to have kind of those those two things.
Ryan Goodfellow:Right? Your control interface and your instruction set architecture for any device that is programmable. And if you don't have those two things, then you can't really fundamentally understand how your system is operating underneath you.
Bryan Cantrill:That's right. And you kinda end up with if you kind of have one of these that's well documented and the other that's entirely opaque, you you're not actually getting total transparency into the system. Mhmm. And I think that on in particular, if you don't have the control interfaces transparent, you can't actually generate the ecosystem that you need to load the programs that are gonna execute on the instruction set architecture. So, I mean, so to think about x a six terms.
Bryan Cantrill:Yeah. Right? Like, you need if you don't have those lowest layer interfaces that you transparently, you are relying as we are before Oxide. You are relying on your vendor to deliver a proprietary software layer
Ryan Goodfellow:Yeah.
Bryan Cantrill:The bias Right. To do that for you. Right. Right. What that stuff is doing is loading your programs.
Bryan Cantrill:Now as happens, you're loading a program that is the operating system that loads other programs. But you that whole ecosystem, and this is what our our colleague Josh Kuo describes UEFI as MS DOS circa 2099, which is just our colleague Artemis had a the the I had a blog entry that went, that that was the top of the charts at Hacker News over the weekend Nice. On defragging a UEFI partition, which is pretty interesting. But it's I mean, it's weird that the the these biasing because they're they're pro these weird proprietary records. Right.
Bryan Cantrill:And I think that these analogs exist in all different kinds of silicon. It's not just the CPU. That's the one that, you we're kind of most familiar with. But as Ryze alluding to, like, it exists on your your switching silicon. It exists in your next silicon.
Bryan Cantrill:It exists in any accelerator silicon that you would use.
Adam Leventhal:That's to say that you're getting this blob of of, of code that embodies the proprietary understanding of that hardware software interface. And then you, the consumer of that, has some different possibly unrelated interface on top of that, and that that blob, you know, between you and the hardware just is creating opacity.
Bryan Cantrill:That's right. That's right. And so you even when you if the instruction set is well documented.
Adam Leventhal:Yeah.
Bryan Cantrill:So, like, I mean, if you looked at at PTX, which is the NVIDIA thing, it's pretty interesting. The supposedly, the the DeepSeek folks actually coded straight to PTX. They didn't didn't use CUDA at all.
Adam Leventhal:Oh, interesting.
Bryan Cantrill:And but PTX is actually a virtual instruction set. And historically, it's my understanding. I've never never not done really driver development this level. But this is also PTX is the thing that they've, like they documented.
Adam Leventhal:Mhmm.
Bryan Cantrill:And that gets translated into an instruction set that actually runs on the GPU, and that historically there there are times when that was been done by the driver. But you don't get to see any of that. That's all proprietary. Gotcha. And so this is where, like, instruction set's documented, but the control interface is not.
Bryan Cantrill:So you can't actually eliminate the NVIDIA driver, right, say. And you also don't have the, the source code to it. So that that was kind of the rationale for the taxonomy. Does that make sense? Yeah.
Bryan Cantrill:Yeah. Totally. And, Ryder, you did I mean, obviously, is very much inspired on your own work. So hopefully, made sense to you. I mean, I again, or hopefully, I
Ryan Goodfellow:I didn't did
Bryan Cantrill:too many errors in kind of the transcription there. But I do think one of the important observations there is the this idea of an instruction set architecture and then, a data interface, control interface. This is true for lots of different kinds of parts. This isn't just CPUs, especially in in a world with with acceleration. And then I also want to clarify some things that are not interfaces.
Bryan Cantrill:Okay? So that we're, like, not talking about. Because there's kind of a temptation to be like, no. No. I am giving you an interface for the hardware.
Bryan Cantrill:It's the how. The hardware abstraction layer. Yeah. It's like, use my proprietary how. It's like, no.
Bryan Cantrill:That's not. No. That's not the hardware. Yeah. That is a and I don't know.
Bryan Cantrill:I mean, Rai, you've seen a bunch of these, and I'm sure there's some that are better than others. But, I mean, although that said, Rye has definitely you've got some scar tissue from DPDK, I believe, Rye. Correct me if I'm wrong.
Ryan Goodfellow:Yeah. I mean, I definitely have scar tissue from DPDK, but, like, the existence of DPDK in the networking space is really interesting. And, like, back in the the 02/2016, '2 thousand '17 era, I ran a bunch of systems, for network data plane processing at, like, the 40 to a hundred gigabit level that we wrote on DPDK and, the Linux Express data path or XDP. And these are frameworks that allow you to basically make manipulations to, network traffic every single packet as it goes through, like, a multi hundred gigabit device. To do this, you have to buy an extremely powerful, like, you know, best in class, Intel Xeon at the time, or AMD EPYC server.
Ryan Goodfellow:That's gonna cost an extraordinary amount of money to be able to do this. It's not gonna cost as much money as the equivalent Juniper or Cisco box upfront. It's gonna cost you more money to run that thing because it's chugging so much power to run the CPU to churn on packets at a hundred gigabits per second. And it's gonna cost you way more money to develop the software for that platform, because, you know, writing c or c plus plus code or Rust code that is going to run-in, you know, in an operating system that's doing context switching, and you're competing with other resource or other applications for resources, which is gonna introduce jitter into your network, and it's gonna cause buffer bloat and all these things. Right?
Ryan Goodfellow:It takes us years to develop software that runs well in that context. You know, it costs a lot more than the Cisco or the Juniper solution or whatever proprietary hardware you wanna throw in there. So the question is why do you do that? Why do you take on all that pain, all that extra work, all that extra money to run this stuff? And the answer is it's comprehensible.
Ryan Goodfellow:When things go wrong, this is, you know, x 86 code running either in the kernel of your operating system, which for us is Linux, or as an application, in Linux if we're using AFXCP to pass things all the way up to user space. And so we could put, you know, debug points in there. We could get stack traces. We could get, memory dumps of our processes, all the instructions that we're executing at the time. So when things went out to lunch, we could then actually understand what was going on and actually fix problems and move forward with the proprietary hardware based solutions that work better on sunny days but, are completely terrible on rainy days, and they go out to lunch and they cause major outages.
Ryan Goodfellow:You don't know what's going on. You have to lean on your vendor to understand that, and the complexities and the scale that you're operating at don't allow for meaningful communication between you and your vendor to actually get in and roll up your sleeves and understand what's going on. So the, you know, the question of why do you pay all that extra money in engineering time and power consumption to run something like DBDK or XDP, is because of comprehensibility. And if we had something that were comprehensible for data plane programming, then we would definitely use that. But that certainly wasn't the case in 2016, '20 '17.
Ryan Goodfellow:And it's still challenging to say that it's the case today even with p four and Tofino and things like that because of the limitations of closed door compilers and whatnot and closed earth control interfaces. So that's my that's my DPPK story.
Bryan Cantrill:Yeah. And, Ryan, so we're going into Tofino a little bit. I mean, obviously, we we've spoken about Tofino before, and the kind of our you know, you you've got a kind of a great rubric on Tofino in terms of the the good, the bad, and the ugly. And the ugly of Tofino really did come down to a lot of these interface issues. Do you wanna elaborate on that a little bit?
Ryan Goodfellow:Yeah. I mean, it it it completely came down to that. Right? Like, the Tofino is a great ship. It's extremely predictable, in terms of network performance.
Ryan Goodfellow:It presents some challenges with the compiler because the compiler is closed source. It recently got open source, but they didn't open source the documentation around the microarchitecture. So writing a compiler or understanding a compiler for a microarchitecture that you have to reverse engineer is is somewhat challenging. But even with the Tofino, even though we're compiling our p four to run down this chip, like, there are phenomenon that happen on the Tofino that you're like, what is going on? Like, the behavior that I'm seeing does not line up with the code that is running on this chip.
Ryan Goodfellow:And when that type of thing happens, you really need to be able to do things like dynamic tracing. You need to be able to do things like get stacked on so the architectural state that is running on that chip, and that requires, making modifications with the compiler to be able to do those things in the first place that requires, having low level access to the chip, like, over PCI Express to be able to get that the necessary information out of the chip through the registers. It requires all this stuff that we've had for a long time with, like, x 86, but we've never had in the data plane programming space. And so, yeah, I did kind of go through a maybe we can publish a bit of at some point, but I do do, like, a good, bad, and the ugly retrospective of Casino. And I think all the the the bad things were fixable things, but the the ugly things all revolved around a lack of transparency, around the chip itself and were things that were nontechnical things that we just we just couldn't fix.
Ryan Goodfellow:And I was really looking forward to, like, you know, the next revisions of the Tufino and hoping to work with the Intel folks to try to push things in a more transparent direction. Obviously, that didn't happen. I still think the RMT architecture in which Tufino was based was phenomenal. I would love to see it, come back again, but, you know, that's not the world that we're living in. And, unfortunately, you know, the the level of complexity and sophistication that ourselves and others needed to operate at with the Tofino, just wasn't within reach for us, because Intel didn't allow that to happen.
Bryan Cantrill:They didn't. And I and I know that folks at Intel, I'm sure, would disagree with us on this. But hand on heart, I believe that the proprietariness of the Tofino stack was an accelerant for their demise. I think that that that they, I feel, that that they they shackled Tofino by having because the know, as I mentioned, like, the compiler was proprietary. So it's now open, which is great, but it's like the the the talks that you kinda need to write the compiler are still closed.
Bryan Cantrill:But having just the compiler closed was a real problem because with p four, I mean, there's a lot of great things about it. But with the I mean, Rye, correct me if I'm wrong, but Rye would tell these stories of like, no. I make a change that should actually result in, like, less less table use, and now this thing doesn't load. And I don't know why. I mean, Ry, am I am I paraphrasing that correctly?
Ryan Goodfellow:Yeah. I mean, the architecture is fairly unique. Programming for the Tofino is a lot more like programming for an FPGA than it is, for programming for a CPU. And so it's either either your program fits and it runs at line rate, meaning it is gonna run at the packets per second that the overall hardware is capable of or it's not going to fit. And, if if you've read about, like, the RMT architecture in which the Tofino is based, you understand that they're they're kind of playing this multidimensional game of Tetris where the Tofino is like an explicitly staged hardware device, and you have specific resources in specific stages.
Ryan Goodfellow:And so to get your code to fit on that, it's gotta place the code in certain stages. It's gotta place the tables in certain stages. It's gotta place, you know, other types of resources, like what we call, header vector memory in certain stages. And once you make a set of decisions, you might go down a path where it makes it impossible to place tables. So sometimes compiler runs on the Sofino can take an hour, to try to get this, you know, high dimensional game of Tetris sorted out.
Ryan Goodfellow:And at the end of the day, it can't figure it out. So, yes, you can go from code that actually fit on the chip, do nothing but delete p four code, and then the code doesn't fit on the chip anymore because it's gone down, like, a different path than the optimizer to try to fit this thing on. So very challenging for the programming's perspective.
Bryan Cantrill:And especially challenging when that layer is opaque. Yeah. When it's like, okay. So, like, fine. Like, I this is, like, not the most like, I'm not having a great time over here.
Bryan Cantrill:But if I've got an open source compiler, I can at least go in there and start to understand why it made the decisions it made where and and get some idea of what's going on. But that wasn't I mean, again, they've opened it now, but but it's really too little to wait because they we don't have the documentation necessary. And when Rye was asking the the Tofino folks and I think this is the other thing, Rye, that I found very frustrating with the whole Tofino adventure, and this has been true at at just about every partner and vendor we've spoken with. When we are asking for transparency, what you get a lot of is individual technologists saying like, no. I agree with you.
Bryan Cantrill:I agree with you, and I see why you're asking for it. And if I were you, I'd be asking for it too, but we're never gonna do it. And
Ryan Goodfellow:Yeah. That's that's definitely a recurring theme, with almost all the interactions that I've had.
Bryan Cantrill:And and that, again, that is, like, that's not unique to Intel Tofino. That was just like that that's been everybody. And really the thing that this is where as low level software engineers, we do go a little bit of our mind. It's like, you know that we're trying to, like, bind more tightly onto your thing.
Adam Leventhal:That's right. You think after I built my own compiler, I'm gonna, like, what?
Bryan Cantrill:Switch? Switch. Exactly. I I'm trying to and do you know why I'm trying to buy more tightly on anything? So I can buy more of your parts.
Bryan Cantrill:Like, you make money on the silicon. Right? You know that. Right? You don't make you don't make money on the compiler.
Bryan Cantrill:Yeah. You know? You don't make like, why are you you should be I mean, anything that enables software ecosystem is something that you should be giving away as much as possible. But again, this is why I want to avoid that kind of language in the RFD.
Adam Leventhal:No. I know we're kind of in an ISA, maybe not monoculture, but like duo culture. But, like, there was a time you would talk about the the two thousands.
Bryan Cantrill:Yeah. Are we are we in an ISA duo culture? I don't know that we are.
Adam Leventhal:I kinda. Right?
Bryan Cantrill:No. I don't don't think so? No. So what's the duo culture? Just ARM and x 86?
Bryan Cantrill:Yeah. We got we've got risk five. We've got we we've got the the actual ISA that the Tofino runs on is z 80. Okay. We've got the we've got other ISAs that are we've got we've got some eighty fifty ones kicking around.
Bryan Cantrill:Okay. We've got Cortex m, obviously.
Adam Leventhal:Yeah. Okay. No. I great. You know?
Adam Leventhal:No. If I take
Bryan Cantrill:it No. No. But, like, look at the stamp collection over here. This is you're not Okay.
Adam Leventhal:I take your point, but but but look at, like, the mid nineties or whatever. When you've got Yes.
Bryan Cantrill:Yes. The different Yes.
Adam Leventhal:All the different vendors with all of their different ISOs and all of their proprietary compilers. Yes. Like, I I just mean that this era that you this this kind of proprietary kind of hoarding of information, like, there there was a analog in the past where for, you know, different different vendors competing at least ostensibly on trying to capture developers. And and all that did go open source, and I
Bryan Cantrill:think the open source compilers were an accelerant. Absolutely. And I think that the I mean and, Rai, this is a point that that you've made to to anyone who's selling network silicon who will listen. It's like, look at what happened with x 86 and open source system software with respect to Linux. Yeah.
Bryan Cantrill:And it's like, how did Intel do with the rise of x 86? Like, pretty well. I mean, okay. Fine. Up until up until there are no possible routes forward.
Bryan Cantrill:But they, you know, they they they had
Adam Leventhal:They had a good
Bryan Cantrill:run. They had a good run.
Adam Leventhal:They had
Bryan Cantrill:a good run. And that really needed open source system software. Yeah. Right? You weren't gonna do that with a any proprietary operating system, I don't think.
Ryan Goodfellow:Yeah. Oh, absolutely. I mean, there's there's there's absolutely no way. Right? Like, when you look at, like, when we have, like, these big, like, debugging sessions at Oxide where, you know and Oxide is a microcosm of, like, you know, the systems programming of the world at large because we build an entire, like, you know, cloud scale or, you know, rack scale computer.
Ryan Goodfellow:And when you look at these debugging sessions where I have where we see some weird behavior, something very high level, and then, you know, the networking team gets pulled in because, you know, it's always the network at the beginning. But then we we start debugging something, and then we start to, you know, get to a point where we pull in part of the operating systems team. And then we you know, if something weird is going on with storage systems, we start to bring in the storage team and people that know about the firmware and the FPGAs on the platform that are driving a lot of the hardware. And so, like and, you know, we're talking about, like, tens of millions of lines of code between all of these things. Right?
Ryan Goodfellow:All the operating systems code, all the firmware, all the drivers in the operating system, all the infrastructure software. Maybe it's a big complex database that's driving some network traffic in a weird way, and we just need to understand all of this at the same time. And anything that's proprietary in there just creates a barrier where you can't understand what's going on. And I think you can make this you know, you can build the same scene of actors and characters in other places of other cloud providers that are building on Linux with open source compilers. And for Rust,
Bryan Cantrill:we have the open source Rust compiler ecosystem. So if we need
Ryan Goodfellow:to go down to the, like, the instruction set architecture level, like, we can do that. And I think that's necessary for building robust systems, and it's necessary for an ecosystem to take hold because you have a large group of people that are collectively trying to build robust systems and solve problems that are useful for other people. But, like, you need that diverse community to be able to, like, collaborate over code, something that is, like, very tangible and, like, arms into the problem type of thing.
Adam Leventhal:Yeah. You're right. The the power of an ecosystem cannot be overstated.
Bryan Cantrill:That's right. So you need an ecosystem. And the question is, what do we need out of hardware to create an ecosystem? Yeah. And I and I feel that, like, again, it's even though and I feel it's one of these things that's obvious to anyone doing the software and seemingly not obvious to folks doing the hardware.
Bryan Cantrill:It's like, what we actually and what we're actually asking for, that's what I tried to do is see these various tiers of, like, what we're actually asking for is for this to be publicly documented. That's it. Like, that is actually what we're asking for. We're asking for the hardware software interface to be I mean, doesn't it sound reasonable when I when I say it that way? Very reasonable.
Adam Leventhal:I mean, so I love though that one of the most successful examples in this document, in the RFD that you wrote, is one that was done unwillingly in the ICE 40. Like, I love that. Oh, yes. Yeah. Yeah.
Adam Leventhal:Yeah. Yeah. What is the most successful ecosystem of this kind? Like, arguably the ICE 40 ecosystem.
Bryan Cantrill:I forgot FPGA is. Yeah. Yeah.
Adam Leventhal:Yeah. But but done done through reverse engineering.
Bryan Cantrill:Reverse engineering. Yeah.
Adam Leventhal:But it has spawned this thriving ecosystem.
Bryan Cantrill:Yeah. Exactly. You actually get a true open source ecosystem. I and we oh god. We actually tried to explain that to Lattice.
Bryan Cantrill:Lattice was very Lattice one at some point is like, hey, we should have a call. And we tried to explain to Lattice why we're using their parts, and they're like, we have never heard anyone say this before. Like, we don't know any of this stuff. Like, what are you talking about? And we're like, no.
Bryan Cantrill:This is like, there's an entire ecosystem out there that's buying your parts. But you begin to realize like, oh, like, you when your part is publicly documented, you don't know. There's no feedback to you of or in the in their case, reverse engineered. There's no feedback to you that, like, this is why I bought your part.
Adam Leventhal:Uh-uh. Yeah. I mean,
Bryan Cantrill:it's like, honestly, we don't know anyone at ST. I mean, I feel that we this is where Robert would be like Robert's already DMing me. I'm sure being like, no. Actually, like, we've we've had, like okay. We do know some people at ST, but we basically have not our relationship with ST, we don't need to be tight with STMicroelectronics, which makes our service processor.
Bryan Cantrill:We don't need to have, like, a sitting in one of those lab because they are the Naples Ultra of public documentation. Their documentation is really, really good. And I remember when we were first setting out, Keith was looking at the at the STM docs. I was like, these docs are amazing. I'm like, these docs are like, fine.
Bryan Cantrill:Okay. Okay. Yeah.
Adam Leventhal:I mean, it's like Says what it does.
Bryan Cantrill:Says says what it does. And then I spent some time in some other documentation. I'm like, those docs are amazing.
Adam Leventhal:Wow. They do say what they do.
Bryan Cantrill:They do. Do say what they do. Yeah. It's and their docs are really good, and they're, like, really complete. I think I found one error in all of their docs, like an address that was, like, obviously wrong that we had them fix.
Bryan Cantrill:But, I mean, they and they make really sophisticated parts because they have these SoCs that have all these blocks on them. So these are, like, these are documents. These are, like, 3,000 page documents. And these are all publicly available. Anyone can get them, and you can and you don't need their proprietary layer, and we don't use that.
Bryan Cantrill:Right? We're not using their, like, house and all their stuff. We are actually we're just, we're on the actual hardware. But it's interesting though, your point about, like, the the feedback goes away. Right?
Bryan Cantrill:The signals that a proprietary company are used to using to measure success Right. Kind of disappear.
Adam Leventhal:They definitely disappear. Yeah. And it just and it goes somewhere else.
Bryan Cantrill:And now in in ST's case, like, ST clearly has spent a lot of time and energy documenting this part. So, like, they've got a hunch that this is important. But to go to back to your Lattice example, this is reverse engineered, and Lattice is basically, like, agreed to not litigate. And so if you're a Lattice field engineer, like, you may have, like, missed that day on the Internet. And you just may be totally unaware that there are these people that are using Lattice parts that aren't using any Lattice tooling whatsoever.
Bryan Cantrill:And it was a weird call because they're just like, what? What are you talking about? I'm like, no. This is what are you talking about? What do you mean?
Bryan Cantrill:And I tried to get them to like, no. Like, this has all been done for you so you could, like, really tack into this. And they were just, disoriented. Like, I we just never heard this before. I just don't know what you're talking about.
Bryan Cantrill:Like, okay. Never mind. Can we just buy we'll just keep buying your parts, I guess. And, you know, the problem with Lattice is that, like, Lattice is great, but it's, I mean, it Lattice is not Xilinx. It's not Altera, which would actually be so you that you're you're not actually getting the, and that was part of the part of the challenge there.
Bryan Cantrill:And we actually we love the same thing out of Xilinx and Altera, but we can't. Like, oh, yeah. Just ask GLaDOS what a big success has been for them. Like, what are you even talking about? I don't even know.
Bryan Cantrill:This is
Adam Leventhal:we called GLaDOS. They didn't know what they didn't know
Bryan Cantrill:what you're saying. We called GLaDOS. Exactly. And so the alright. So so but the tier one is this, like, public documentation.
Bryan Cantrill:That's all we're asking for. Do what ST is doing, please. Well, we can't do that. It's like, oh my god. No.
Bryan Cantrill:No. No. We and, again, it's like the things that they can't do and it will depend because, like, in Rye's world of of switching silicon, it's just a darker world. Right? And right.
Bryan Cantrill:I don't mean that I mean, I don't mean that pejorative way. I just mean that, like, it is just historically much more proprietary.
Ryan Goodfellow:Yeah. Absolutely. Yeah. You, yeah. You you you don't need to you don't need to tell me.
Ryan Goodfellow:I I I'm aware of the darkness in which I live. And
Bryan Cantrill:so it it's like, okay. No. No. Like, public documentation, definitely not. Like, okay.
Bryan Cantrill:So how about, like, documenting the stuff privately and just make like, give us the like, give us some docs that aren't public and just allow us to build software on top of them. And there are a couple examples of that. That that's kind of like, okay. That's a little less desirable. Then the we're getting less desirable is like, I know.
Bryan Cantrill:We don't have any of that stuff. We're not gonna give you the documentation. I think there's like you know, Robert had a comment for me that I need to integrate in here. Often, like, you've got your documentation as kind of auto generated from the RTL. Mhmm.
Bryan Cantrill:Yeah. And it's like that's rough. You know? When you're looking at documentation that's been auto because it's like there's you'll be not surprised or there's not a whole lot of narrative about, like, what is this registered for? It's just kinda like that you can't register name and address and offsets.
Bryan Cantrill:But, like, even that stuff is valuable. Right? So the you know, we part of what we need to do is get people to comfortable with giving us the the documentation that has been auto generated. Then the kind of the the next tier is like and this is what AMD has done with OpenSill. Like, okay.
Bryan Cantrill:We so AMD has got documentation, which they call they got NDA, they got the public documentation, they've got NDA documentation, then they've got what they call customer internal documentation. They're not making that available. Fine. What they have done, though, is they've made available their OpenSell project, which uses all that stuff. So great.
Bryan Cantrill:So you can
Adam Leventhal:And this is open source, which is effectively encoding Yes. What that
Bryan Cantrill:That's right. That's right. That's right. So you okay. You're not gonna give me the documentation, but you're gonna give me some source code that actually uses the system, and I can use that.
Bryan Cantrill:And not great. It can be, I believe, although it's actually gonna sometimes it could be more useful to have, like
Adam Leventhal:Sometimes more useful now you're looking at something that works rather than trying to
Bryan Cantrill:assemble Easy. Easy. Now keep how you put words in my mouth.
Adam Leventhal:Something that they claim works?
Bryan Cantrill:Even then, get backing back up just a little bit. Like, just take one more half step back.
Adam Leventhal:Less
Bryan Cantrill:shame? There there you go. Exactly. So but that's and then you get to, like, the the lattice tier, the reverse engineered tier where it's like, okay, this is just like and you could also have something where and this does exist, by the way, where you have NDA docs that have gotten out into the Internet. And generally, like, if someone leaves a trade secret that's kinda, like, lying around, like, litigation is not with you who found it in a dumpster, it's with whomever violated the covenant with the company.
Bryan Cantrill:Right? So and this is how famously, like, a speed makes these BMCs. And the a speed docs are only available under NDA, but widely available on the Internet. They've leaked on the Internet. And those docs include the this is this is what it includes the root password that is in the the the that all a speed parts have the same root password.
Bryan Cantrill:Right. Which was our Wi Fi password. Which was our Wi Fi password.
Adam Leventhal:Leaked it in a in a a photo.
Bryan Cantrill:We so we leaked it in a photo, which I thought it was like BMC humor. Was very fun. It was for the right kind of audience. It was very funny. Yeah.
Bryan Cantrill:It was very meta. Another example that I I point out is, like, Laura's work on the we're essentially in the OPC 55. We talked about that. And, you know, obviously, the now the information that Laura learned about the OPC 55 s 69 ROM, like anyone can go use that. Like, that's we've documented that for them.
Adam Leventhal:Okay. Question for your editors. Yes. Do they see reverse engineer and suggest any softer phrases? Because I feel like if they document like that Yes.
Adam Leventhal:Like, if you think open is hostile, reverse engineer.
Bryan Cantrill:Okay. This is a great point. And I you know, the the Or you judge GPTing,
Adam Leventhal:you know.
Bryan Cantrill:You that's I the a really insightful point. And, boy, if I had to find any criticism with your point, I mean, if we really gave you no alternative. No. This is great point. You know, the the the bots did not say anything, but I actually wondered about that.
Bryan Cantrill:So I actually did because you're right. That, like, oh, like, oh, yeah. Open is, like, open is too scary. So let's just throw out reverse engineering. Like, why don't you just, do you wanna put I clear wall.
Bryan Cantrill:Yeah. Is there a section here on IP theft? Like, why don't you, like, why don't you put that out there? The thing about reverse engineering is I feel I think this is where my my legal adviser, out of love with all esquire may pull the plug on this. I feel that we, software engineering, need to stand on our own two feet with respect to reverse engineering.
Bryan Cantrill:Say more. It's a natural right. And the the that, like which is to say, like, you like, you've given me a blob. We don't have there's not a EULA that we have with NXP that prevents us from reverse engineering that blob.
Adam Leventhal:I think if you're saying, like, toughen up, I agree.
Bryan Cantrill:Like Yeah. I think we we need to be willing to say, yeah, reverse engineering.
Adam Leventhal:You gave me the blob. Like, I'm gonna do what I do. Yeah. If you don't wanna give me the blob, don't give
Bryan Cantrill:me the Don't give me the blob.
Adam Leventhal:Like, encrypt the blob if you
Bryan Cantrill:wanna be a jerk. But I I, like, I get to see I get to learn how the machine works. Yeah. And that's and I think that that's a natural right. Again, this is where it's like, we just the phrase natural right seems seems a little, a little bellicose.
Bryan Cantrill:Yeah. But, no, I I think it so, no, I damn it. I left it there.
Ryan Goodfellow:I mean, at at some point, it's like, what else are you gonna do? Right? It's like, I've, you know, been freezing my ass off in the data center, staying at the DC in the morning till, like, three in the morning every single night because some switch is doing some weird thing that is destroying my business, and, you know, destroying my health. So what what am I supposed to do? Because your support engineers are, like, effectively telling me to turn it off and turn it on again.
Ryan Goodfellow:So That's right. You know?
Bryan Cantrill:Hypothetically. Hypothetically. Exactly. Well, no. And I think but I do think so part of the reason I actually wanted also to leave the term in there is to remind folks, silicon vendors, that, like, hey.
Bryan Cantrill:There are some and, you know, I I I added a section on this after you and I talked earlier today, Rai, of on Joy's Law. It's like there are more smart people outside of your company than there are inside your company. And always, don't matter who you are. And, like, it is something that people can go do. People can go reverse engineer your interface.
Bryan Cantrill:That can happen. That can definitely happen. And, I it's not a threat. Although, maybe why would I why would I be volunteering that? It's like, I didn't say it was a threat.
Bryan Cantrill:You were the one who's telling me that's not you're the one that's telling me menacingly that it's not a threat.
Adam Leventhal:Why I think it'd be a shame.
Bryan Cantrill:Right. I'm just saying it'd be a shame. It'd be a shame. So, no, I great question. One that chat GBT was too scared to I I I should have looked inside of I'm sure I could get Gemini to be like, god, I really need him to tone down this this turn, but he's not gonna take it well.
Bryan Cantrill:I better just Earn some trust first. Earn some trust first. Exactly. And then you get to, like, this the the this kind of fifth tier, which is, like, the the the last usable tier. If you if you get beyond this, it's like the part's not usable.
Bryan Cantrill:Right? Because it's like, I'm not even gonna tell you what it is. Like, you you got no way of using this. This is like, I'm gonna tell you how to use it, but I'm not gonna allow you like, I'm gonna like, the the interface is a trade secret. I'm gonna whisper to the interface in your ear, you can build some software around it, but you then can't open that software.
Bryan Cantrill:Right. Or you're limited about what you can do with that software. And in our case, what we wanna go do is we wanna open it. We wanna make sure make sure our entire stack is open source. Right.
Bryan Cantrill:And that's definitely the least desirable. And it's basically, like, not acceptable, really. Actually, this is where the both bots gave me some advice. They wanted me to really sharpen this paragraph. They basically wanna be like, hey.
Bryan Cantrill:No. Like, you should be really clear. Like, it is unacceptable. Like, we will kick you to the curb. Like, yeah.
Bryan Cantrill:We won't quite kick you to the curb. We will. We'll still use the part.
Adam Leventhal:And so, like, this is better than just complete opacity and we're gonna give you some some blob and we're gonna give you a different abstraction on top of it. But I would say in some, like, in some ways it actually feels worse. It feels worse because like, it enables you to do cool stuff, but not, like, tell anyone, like, it it it it it limits the scope of the innovation that you can then do as opposed to being completely shut out from it.
Bryan Cantrill:So so so you're saying that this is allowing you to eat the fruit of a tree of knowledge of good and evil, and you and you recognize that you are nude. Kind of. Right. Yeah. Right.
Adam Leventhal:Exactly. I mean, like, I get that, like, in in many ways, it is technically superior to just a complete opacity. Right. But maybe worse your psyche.
Bryan Cantrill:Yes. I yeah. No. I think that there's a there's a valid point there.
Ryan Goodfellow:Yeah. And, I mean, it's it's potentially really damaging to your customers. Right? So, like, at Oxide, we're a platform developer. Right?
Ryan Goodfellow:We build a platform that has a bunch of different hardware components inside of it. I've spent most of my career in, like, developing and operating networks. And, like, at my last job, we were a big customer of Cumulus Linux, so we bought a lot of Cumulus gear. And they had a, you know, a pretty good disposition toward open source, but they also had these bilateral relationships with the ASIC vendors where they had access to the secret sauce, or, you know, let's not even call it the secret sauce, the hardware software interface. That's right.
Bryan Cantrill:We as we
Ryan Goodfellow:as the, as the customer, as the end operator of the system, depending on these for, our organization, didn't have access to that. And so it's extra like, it it doesn't do anybody any good, to have the platform vendor be able to look at this but leave the operators and the ultimate end users of this technology in the dark. And that's one of the big reasons why I came to Oxide because I wanted to change that for networking, and hope to be able to change that while I'm here at Oxide.
Adam Leventhal:It even seems to create an obstacle. Right? Then becomes, what can I share with the customer? Right. He's like, I know secret knowledge that I have.
Bryan Cantrill:Know I know the secret knowledge of how to use this thing. It's like
Ryan Goodfellow:Yeah. Exactly.
Bryan Cantrill:I know the I I know the top secret knowledge of, like, where the gas pedal is. Definitely can't tell them that. Don't don't tell them that. Yeah. And and right, you're exactly right.
Bryan Cantrill:And this is you know, you you had the point that that, transparency is transitive, And that, you know, you can only and we need to be transparent all the way up to the stack to that end operator sitting in the data center should be able to have the support of an entire stack that can help them understand what's actually failing here. So that's kind of like
Ryan Goodfellow:That's the that's the world we're trying to build. You know, it's not a question of if. It's a question of when. But It's a
Bryan Cantrill:question of
Ryan Goodfellow:we'll see of of when the when will be.
Bryan Cantrill:That's right. So okay. So then we get to, like, the the the arguments. I'm trying to collect, like, what are the stated arguments against transparency? And you'll do notice that that I I did differentiate between stated arguments and the unstated arguments.
Adam Leventhal:I like that you put them on quotes too. Just
Bryan Cantrill:Would you like that? Yeah. Yeah. And I did I how did I do on that? Did I on on the did I make them
Adam Leventhal:I know this is not the first time on the podcast we've mentioned the four questions from the path of
Bryan Cantrill:Oh, yeah. That's right. Yeah. Yeah. I love the four questions.
Adam Leventhal:It felt very, like, the fucking
Bryan Cantrill:child simple stuff. Right. Okay. That's good. So, oh, yeah.
Bryan Cantrill:Where are we? So transparency will allow someone to copy me.
Adam Leventhal:Yeah. Felt very simple Yes.
Bryan Cantrill:Yeah. No. I think you're right. And I think and, you you know, what I have found when these stated arguments so transparency so if I open my interface, someone else is gonna copy it. It's like, are you kidding me?
Bryan Cantrill:What? Like, no. No one cares. No. Do you know what's much more likely?
Bryan Cantrill:You're gonna die. That's much more likely. What you you're gonna be lucky if if the entire universe cares so much about your interface that they're scrambling to copy it. Yeah. Man, you'd be so lucky.
Adam Leventhal:It's like, let's walk this forward. You open the thing. You build such a thriving ecosystem Yes. That it becomes that you the the hardware software interface that you have defined because so such a standard that the only way to operate is by mimicking your hardware software interface. And then a competitor comes along who can do it better than you, then, like, why do
Bryan Cantrill:you Why do you exist?
Adam Leventhal:Why do you why do you deserve to exist? You deserve to exist. Like, you you you you fumbled everything.
Bryan Cantrill:Yes. And, you know, actually, this reminds me when in the so the the first time I can remember open source coming up at Sun with respect to Solaris was, I think, 1997. It could be off by, like, a year. It could be as late as 1998. But I remember thinking and, you know, this obviously, my thinking on this changed pretty quickly.
Bryan Cantrill:But and in part because of this conversation, remember thinking, like, having this same thought. Like, I had this thought of, like because this is very much the era of proprietary operating systems, and Linux base code doesn't exist. I mean, it exists, but it's it I mean, it exists, but it's not being used. PSDs exist, but they're fighting with themselves. And this is the era of the proprietary risk op Unisys.
Bryan Cantrill:And I remember telling the VP, like, I'm just worried that a competitor would steal our memory, our kernel memory allocator. Right? Slab allocator. Right? Read by our colleague, Jeff Bonrock.
Bryan Cantrill:And he just looked me dead in eye. Was like, Brian, if someone steals our memory allocator and we can't outcompete that company, if their strategy is to follow whatever we're doing with the memory allocator and we can't outcompete them, we don't deserve it. Yeah. And I'm like, yeah. Damn.
Bryan Cantrill:That's really right.
Adam Leventhal:Yeah. And I whatever. I think that's the only attitude that's not a kind of cowardly attitude. Yeah. I mean, I think the the alternative is, I think, along the lines of what you've kind of you've observed about inherited wealth.
Adam Leventhal:Yes. Preserving wealth.
Bryan Cantrill:Yes. A %. Yes. Yeah.
Adam Leventhal:Yeah. Yeah. Continuing to innovate.
Bryan Cantrill:Continuing to innovate. No. Absolutely. Absolutely. So so that argument is simple trials.
Bryan Cantrill:I'll leave you on simple trials. Then you get to the, transparency will be a support burden. We get this one a lot that, like, I I can't support you if you do this. And we're like, we don't want you to support us. I mean, we don't you know what I mean?
Bryan Cantrill:It's like hard and right. You you walked this a bunch of times. You're just like, we are no. We are going to, like if you give us the interface, we won't need your support. And
Ryan Goodfellow:Yeah. They they, you know, they can't support us if they if they give us the interface, which is exactly what they want. Right? They we don't need their support. And
Bryan Cantrill:And and then they're like kinda like, we don't believe you, basically. It's like, no. No. No. We're just not gonna be able to support you.
Bryan Cantrill:It's like, we don't want your support. And that one, I feel we have just had to earn. And I think that we have earned it with a couple of companies. And, you know, I don't think I'm speaking out of turn to say we've probably earned it with AMD. I think AMD had this exact objection to what we wanted to go do with respect to holistic boot.
Bryan Cantrill:Yeah. And they're like, it's gonna be a huge support burden. And we really made the case that, like, no. No. We are gonna be self supporting.
Bryan Cantrill:And, you know, when we you know, to to their credit, people inside of AMD were really trying to make that case for us and how and, like, they could see that too. But it really took just time and us really earning it. And, like and, you know, our our fortunately, the folks who work closely with at AMD now are terrific advocates for us. Shout out to Will outside of AMD who are terrific advocates for us, and I'll be like, no. Like, when I you know, the the questions that you guy when you guys do have a question, it's because, like, by the way, like, the part is broken.
Bryan Cantrill:You know, when we when you come to us because we are able to be sure that we are not misunderstanding something because you've given us the interface. I mean, it's so important. And so, Roy, I don't know. I I feel like we've not really found a way to that one is just, like, one we have to it's very hard to talk people out of this argument because they're like, you're gonna be a support burden. I was like, no.
Bryan Cantrill:No. No. We're not. And, you know, again, I think we've earned it at a couple of companies, but it's it's hard to make that argument in advance.
Ryan Goodfellow:Yeah. We definitely have to re reearn it every single time, every single hardware vendor that we talk to. It's a it's a new conversation. We we go through this, and, you know, ultimately, it winds up working.
Bryan Cantrill:I I I yeah. This r f d is an attempt to not have to rerun it from zero every time, so I'm hoping we could get there. But so that's this is one that can get people. The one that that the and so that, maybe that's is that the wise child, wicked child with the it it's simple, wicked, wise?
Adam Leventhal:They're they're a bunch of children.
Bryan Cantrill:It didn't you know, they're a bunch of children. It's like, I don't need you know, don't to keep them straight anymore. I don't even know. Is that when the wicked one? Wicked one is I don't know.
Bryan Cantrill:Then you get to transparency is a security risk, and that can be frustrating. That because that's that's bullshit. Yes. That's just absolute bullshit.
Adam Leventhal:And that's the argument against any kind of open, I think.
Bryan Cantrill:Yeah. Yes. Any kind of transparency.
Adam Leventhal:Any kind of transparency.
Bryan Cantrill:Yes. And the this is the Kirchhoff's principle.
Ryan Goodfellow:It's kind
Bryan Cantrill:of this intro and, yeah, so this is an interesting a a a principle from a Auguste Kirchhoff, who is a nineteenth century Dutchman, who had the observation that in a cryptographic system, you should be able to make public everything except for the key, and that should not represent a vulnerability. Right. Right. Right. Kirchhoff.
Bryan Cantrill:Kirchhoff's just you know, goose. You're you're you're you're killing it over there. I I don't know how you're doing this in the nineteenth century, and it's kind of I I kinda wanna read about this guy. I don't know. Yeah.
Bryan Cantrill:Maybe he doesn't exist. Maybe this is you think is that possible someone, like, just created a Wikipedia? You know what I'm gonna do? The chat GPT. Chat chat GPT, you know what?
Bryan Cantrill:I'm gonna create, like, some nineteenth century that people won't actually you know, I I can create something that's plausible, and then I can just win this argument. Sure. I'm sure this person actually exists. But the I I think that that so this is this is bullshit. Rye, have you heard of this more than I mean, I I've heard this at least once.
Bryan Cantrill:It gets chased out of the room quite quickly.
Ryan Goodfellow:I I hear this continuously. It's it's always there, especially in the networking space. It's it's there a lot.
Bryan Cantrill:And you know what's actually also bonkers is when you have someone who is making that argument to someone for whom security is extremely important, like, is much more sophisticated about security than you are. You know, when I'm you know, you it's like, you chip vendor are making that argument to, you know, to a to a federal government or to someone who who does, like, who does security for a living. It's like, yeah. No. They
Adam Leventhal:And, you know, I don't I don't claim to be particularly sophisticated when it comes to security, but I do imagine
Bryan Cantrill:Adelovital, security researcher. I do imagine that, like, part of
Adam Leventhal:the upshot of this is that, a lack of transparency here means that, like, let's say there is a vulnerable.
Bryan Cantrill:Yeah. Yes. Right.
Adam Leventhal:Then it will be Only the bad guys will. Only the worst guys.
Bryan Cantrill:Oh, yeah. Only the bad guys will.
Adam Leventhal:Exactly. Well funded as who would know.
Bryan Cantrill:Yeah. Only the state actors.
Adam Leventhal:Only the state actors who would then be able to exploit it in the worst ways possible Yes. As opposed to
Bryan Cantrill:This is about in the conversation where this argument turns into something else. This is where they're like, okay. Don't wanna have this. Like, okay. Yeah.
Bryan Cantrill:I'd like this this one is kinda falling apart in front of me. Actually, I wanna go back to something else. I wanna go back to, like let me go back to someone suing my IP. Now that But but this is a
Adam Leventhal:lazy argument fundamental. Lazy argument fundamental.
Bryan Cantrill:Yes. This is
Adam Leventhal:not this this this is not this is something that that feels right on its face and doesn't bear scrutiny, certainly not in 2025.
Bryan Cantrill:That that's right. Does not bear scrutiny is a a an absolute redux of the arguments against open source Yeah. That I feel have all been debunked. Yes. And, you know, being transparent makes something more secure.
Bryan Cantrill:I mean, that's what makes the XE vulnerability so extraordinary is that they did pull it off in the open, and it was really, really hard. Much harder to pull that thing off in the open Yes. Than it would have been in the proprietary tool chain. So, I mean, yeah, this is that that one is just bonkers. Another one you'll get is like, no.
Bryan Cantrill:No. No. Listen. I I I wanna make it transparent, but I've gone I I can't do that because of another agreement that I have with an IP provider. So with someone who's providing a component of my chip, that's what's preventing me from, like yeah.
Bryan Cantrill:I'm just and My hands are tied.
Adam Leventhal:My hands are tied. This this is also another, like I mean, we've seen
Bryan Cantrill:this, like, the lawyers in control.
Adam Leventhal:Yes. Like, department of no.
Bryan Cantrill:Department of no. And then you'll be like, okay. Like, what okay. So so let's get that agreement in front of us. Like, well, I'm not a lawyer.
Bryan Cantrill:Like, I'm not gonna have to, like no. I actually don't know anything about it. Like, well, okay. Does anyone there? Does any wait.
Bryan Cantrill:Why? How do you know that that would violate that agree? Because it's just like, it's a very strange I mean, it's a very stringent agreement to say that, like, I'm using your IP, and this is gonna constitute an interface to my part. And now I'm very encumbered about how that I mean, it's like, I I would I want I have a lot of follow-up questions.
Adam Leventhal:Yeah. Especially because we're not saying, like, open source the RTL.
Bryan Cantrill:No. That's it. The interface. Right. The interface.
Bryan Cantrill:So you're like, that IP, like, how does that violate your agreement? Right? Just seems very strange. Yeah. I don't know.
Bryan Cantrill:Have you heard of this one, Rhyme? Have you heard this one in your a lot in your travels? Is this a common one, is this an uncommon one?
Ryan Goodfellow:It's extremely common in networking. Like, network ASICs are built out of a bunch of different IPs from a bunch of different shops usually unless you're someone with an enormous amount of resources. Like, I think Cisco, like, makes some of their own chips, like, completely, without buying IP from others. But I think most of the the, you know, medium size enterprises in town that are making networking ASICs, they're buying IP from somebody, particularly from their EDA vendor probably buying SerDes IP and things like that. Yeah.
Ryan Goodfellow:PCI Express core IP, things like that that that come with, restrictions. So it's
Bryan Cantrill:But I'm really surprised that those restrictions include, like, no. You can't make avail so you're gonna buy this PCIe block from me, which has a bunch of, like, register space associated with it. And they're like, oh my god. No. No.
Bryan Cantrill:You can't disclose that register space to anybody.
Adam Leventhal:This is just fun. This is just like word-of-mouth fun that Yes. Is like, again, doesn't bear scrutiny. And it's like, look, if we do anything this this with this thing, we're gonna have to talk to the legal department. And like, I'm not gonna do that.
Adam Leventhal:You're not gonna do that.
Bryan Cantrill:Right. Yes. Move forward. Move forward. Right.
Bryan Cantrill:Exactly. It's way it's kind of a table veto. Yes. I think so too. Yep.
Bryan Cantrill:I this. Like, you're much more bellicose than I am on this one. I'm gonna bring you into the conversation next time. You're extremely unreasonable. Like, actually, no.
Bryan Cantrill:I've got somebody even more unreasonable. I'll like, you thought I was bad cop. I'm you're you're gonna after after love me. After crazy cop, you'll be begging for bad cop. Yeah.
Bryan Cantrill:The no. I agree with you. I but so I also think on this one, there is also a counter. And say and, Ryan, I don't know. Do you maybe they should are are is this one more legit than we're giving you credit for, do you think?
Ryan Goodfellow:It's definitely, like, a barrier that I encounter a lot. That doesn't say anything about its legitimacy,
Bryan Cantrill:but Oh, obviously.
Ryan Goodfellow:I mean,
Bryan Cantrill:you know, I see on this one. So my position on this one is like, okay. I'm I'm gonna accept this, that you are you've got some agreements with some IP providers that are preventing you from making some of the control interface available. We're gonna put, like, rings around those. And we're gonna say because I do think, like, you get into this is being used as part of an all or nothing argument.
Bryan Cantrill:Like, I can't make any of my control interface transparent because some of it is encumbered. And I think it's like, no. No. We're gonna take the sum of it that that it's encumbered and, like, let's start wait. Like, let's get into the interface here.
Bryan Cantrill:Like, we're gonna have to get detail oriented. You don't get to just, like, not give us any control interface to this thing. You're gonna and and let's start drawing some lines around some things. Let's get some some we can I like, this is if if this is a valid argument, I think that there's lots of room for compromise here?
Ryan Goodfellow:Yeah. I think it's a combination of the fact that this is already something that they don't really want to do, and this is just Yes. Like an external force that they can bring into the the conversation. And I I think a lot of this for networking in particular has to do when things go analog. Like, a lot of the chip vendors don't have, like, the analog circuit as expertise in house.
Ryan Goodfellow:So they'll they'll buy the the analog parts of the chip from somebody else, and that'll be encumbered. And so it's, it's challenging.
Bryan Cantrill:It but so right. Those are kinds of the the aspects of the parts that, like I mean, not that we don't care about that, but it's, like that feels like that's the part of the interface that we are least concerned about from the perspective of that transitive transparency. Is that I mean, that's Yeah.
Ryan Goodfellow:I I
Bryan Cantrill:Is that true?
Ryan Goodfellow:Yeah. I I I think so. But I think for, like, you know, it's kind of, like, all tied up in this big nasty pile of spaghetti in terms of, like, the underlying IP mess, which is
Bryan Cantrill:Right.
Ryan Goodfellow:And, again, I'm not saying this is, like, a legitimate thing, but it's it happens a lot, and it it sucks.
Adam Leventhal:Yeah. I think it's a scare tactic. I think it's like, oh, do wanna get the lawyers involved?
Bryan Cantrill:Yeah. Well, so and so my, like, way of saying, like, okay. No. No. We we let's wait in, and let's start marking each of these interfaces about how they're encumbered.
Bryan Cantrill:And, like, your answer, like, well, they're all encumbered. It's like, no. No. No. That's then you're just like, that's disingenuous.
Bryan Cantrill:You know what I mean? It's it's like, let's do the hard work here and figure out where the because one of the nice things about these things about these complicated parts, of course, like, there's a bunch of that's programmatic about their creation. So these folks have their registers or their control interfaces are defined in massive XML files. Right? So, like, great.
Bryan Cantrill:We're gonna have an attribute on one of these things so we can go tag it with it, which we probably should anyway. I mean, if you're so aren't you, like, concerned? Like, you've got this agreement, this toothy agreement that you just told me about that you're so scared of. Like, don't you wanna be very careful about how you mark that interface in your oh, you're not careful because no such no such agreement. Anyway, yeah, whatever.
Bryan Cantrill:Yeah. Just do one of good cop or bad cop. Doing both is gonna be tricky. I thought oh, I I don't, like, don't threaten me with a good time. I could definitely do both good cop and bad cop.
Bryan Cantrill:I've done that plenty. The, I can do good cop, bad cop, and crazy cop. I'll I I it really keeps people One man show. It's a one man show. It keeps people off balance.
Bryan Cantrill:Kinda like, they really don't know who they're dealing with Yeah. At any given moment. Okay. So that those are the stated arguments. Now we get to the unstated arguments.
Bryan Cantrill:And because I feel that these I think, like and and, Ray, it sounds like you are, you are not as willing to go quite as far. Adam and I both think that, like, the transparent the the the transparency is violating another agreement. I think it's total bullshit. And you're like, I just can't speculate, but I hear it a lot. But I do think that there are these unstated arguments that actually are the root of a lot of the actual real opposition.
Bryan Cantrill:Because when you were the thing is when you're arguing with someone about these things, about why they can't be transparent, and you're chasing them around the room, like, they'll make the security argument. Then you knock that one down, and then they kinda show up like, well, now I'm afraid that people are gonna steal it. Okay. So you knock that one down. Oh, actually, I've got this other agreement with an IP provider.
Bryan Cantrill:Then you kinda it's like and you knock that one down. It's like, well, it's a security issue. It's like, okay. So we're just it's all of these things, and it's actually none of them because there's a deeper reason. And this is where you get into the more emotional aspects of engineering that I am convinced is actually the real reasons.
Bryan Cantrill:Just as, I I'm convinced that executive feelings of inadequacy and aging is actually what's behind RTO initiatives. I am convinced that the this these kind of emotions are actually what's behind a lot
Adam Leventhal:of this reticence. Absolutely. Absolutely. And I think the fact that you start each of these with, I'm afraid that is spot on. Right.
Adam Leventhal:Because it it is it is Fear. It is fear.
Bryan Cantrill:It's fear. And because it is fear, it's got, like, people don't wanna say they're afraid. So they kinda come out with these these outgrowths that that
Adam Leventhal:are Like, if you're you're never gonna be in a vendor meeting where it'd like, it I'm afraid.
Bryan Cantrill:Right. So okay. These things are gonna happen. So I just want you to point to the section in the RFD. You don't have to tell me what it is.
Bryan Cantrill:You don't have
Adam Leventhal:tell me
Bryan Cantrill:to just point to the section in the RFD that captures your fear. Really try so okay. One, I think and I I and sometimes people are explicit about this, but often you kinda have to dig a a bit. I'm afraid that my docs are incomplete. Like, that's very common.
Bryan Cantrill:Like, I I I it's just not ready yet.
Adam Leventhal:Yeah. And I mean, we've all felt that. I've all felt that. Unlike anything that we've we've been transparent about turned over to the open Yes. I not explained this thing well enough.
Bryan Cantrill:Yes. And this was the subject of a rather caustic email I may have sent to our colleagues in CFS. Oh, I remember that. Where the, I mean, as far as I'm concerned, like, Jeff Bonwick had an enormous influence on us. I was actually joking with Eliza recently.
Bryan Cantrill:I'm like, you are like a descendant of Jeff because Jeff influenced me, influenced you. We influenced Robert. Robert never worked directly with Jeff, don't think. But I definitely got that, like, inherited that kind of big there. And, obviously, Eliza was doing this before Oxide, but Eliza also writes I don't if you've been in Eliza comments.
Bryan Cantrill:They're just extraordinary. Great big theory comments. Lots of ASCII art. Really good stuff. And we've got a lot of colleagues that do that.
Bryan Cantrill:And I think, like, Bonwick to me is at, like, the the the the the kind of the headwaters of that, but ZFS was not well commented. Yes. And I feel that ZFS was not well commented because Bonwick felt that ZFS was not done. Because the way that he did if you look at the way he wrote his big theory statements, he would do it as a final step.
Adam Leventhal:Oh, really? I didn't know that. Yeah.
Bryan Cantrill:Really interesting. Yeah. And I do it too. I that like, I'm the same way. The the the all of, like, the Ditres comments, I did as a last thing, not a first thing.
Bryan Cantrill:And I think it's because, like, I now like, this is, the last step. I understand this whole thing, and I am convinced that ZFS just escaped. ZFS escaped. ZFS, it's like, he came in one morning and the cage bars had been bent, and the window had been shattered, and there were footprints going in the street.
Adam Leventhal:I think there's an aspect of it. It needed to be done, so it was done.
Bryan Cantrill:Yeah. And well, and I think in in particular, like, ZFS kind of escaped his ability to hold it all on his head. Yeah. And, like, we didn't have there was no CIO pipeline block comment. Yeah.
Bryan Cantrill:There's a big theory statement. Like, it's like, Jeff, we've got these extraordinary big theory statements for things that are actually, like, much simpler in comparison. It's like
Adam Leventhal:I remember one on dividing by a billion or
Bryan Cantrill:something like that. Yeah. Yeah. Yeah. The the the SparkStation two divide by a billion with shifts divide by a billion and then mod a billion.
Bryan Cantrill:Right. It was beautiful. Beautiful with shifts and ads. And that comment we gotta go find that comment because I think it was open source, but then ripped out when the s s two support was ripped out. Right.
Bryan Cantrill:It's like, can't just delete this comment. This comment is so gorgeous. It's amazing. But we didn't see that for ZFS. And I at a point of frustration do you remember this?
Bryan Cantrill:When I
Adam Leventhal:Yes. I will I'll be very interested to see if we remember it the same way.
Bryan Cantrill:I am certain that we will not remember it the same way. But I got so frustrated with the poor level of commenting in ZFS that I compared it to the rest of the system.
Adam Leventhal:Yeah. You did like a, you know, LC basically on like Yes. Comment, is it is it code?
Bryan Cantrill:Is it comment, is it code? And like, what is the fraction? And like, where are we in that? Yes. And ZFS was bottom decile.
Bryan Cantrill:Yes.
Adam Leventhal:And I remember you showing me that email before you sent it.
Bryan Cantrill:Oh, yeah. And what'd you say?
Adam Leventhal:I said, are you sending this email to feel a different way,
Ryan Goodfellow:or are
Adam Leventhal:you sending this way to inspire them to change? What did
Bryan Cantrill:I what did I say?
Adam Leventhal:And you said the first one.
Bryan Cantrill:Oh, yeah. I was like, I already I just sent it. You're like, what? I just sent it. Sorry.
Bryan Cantrill:Yeah. This is a classic, well, you're gonna do what you're gonna do anyway.
Adam Leventhal:Yeah. And And I did. What happened afterwards? Did they did they take that to heart and comment things?
Bryan Cantrill:Yeah. No. I definitely, I definitely didn't do permanent damage to any relationships with that. No. It was not good.
Bryan Cantrill:It was not yeah. Well, look. You're right. Is that what you wanna hear? Well, first time I've heard it.
Bryan Cantrill:That's right. Right. You'll you can write a lot. But you were definitely okay. Yeah.
Bryan Cantrill:If you if you tried to talk me out of that, you know, I don't regret sending it. Someone needed to say it. The and it's so but I I think a lot of a way of saying, I definitely understand where this is coming from.
Adam Leventhal:Yeah. Yes. Like, the, like, if I can't do a good job of it, I kinda don't wanna do it.
Bryan Cantrill:I don't wanna do it. Yeah. Yeah. And, like, I know it's not done yet, and I'm like, I wish this were in a better state, so I'm not gonna make it available. And it's like, you gotta just get over that.
Bryan Cantrill:Yeah. And you've gotta and this is where I feel like we're we're kind of like people's doctor. It's like, look, I've seen so many nude bodies. I just don't like, I don't I've seen so much terrible documentation and source code. Like, I just nothing's gonna surprise me.
Bryan Cantrill:Just please please take your clothes off, and I'll be back in two minutes.
Adam Leventhal:The the other great benefit of being lazy is that it is a great source of of prioritization. Like, if it the the parts that need to be better documented will find you. Like, the people will will educate you about what actually needs to be documented well.
Bryan Cantrill:Maybe in an email with many people on the to line.
Adam Leventhal:That's right. Hopefully, kindly. Hopefully, kindly. But it also will educate you about, like, you could easily have spent all your time documenting the stuff that people don't care about.
Bryan Cantrill:Right. Yeah. Anyway, it's got
Adam Leventhal:procrastination. Just saying.
Bryan Cantrill:But I also do think that, like, there's some truth to the the fact that when you when the system kind of, like, you've kinda barely got it working, you have a hard time documenting. Wait. Like, the active documentation will kinda reveal that, like, ugh, this is really not the right way to do it.
Adam Leventhal:No. I I actually, so I I went through I always speak about ZFS. There was a time when I was at Delphix when I was I was kinda sitting with Matt Matt Arons and getting the oral history of ZFS and then writing it in blog posts and then comments. And the and part of this, like, the old ZFS write throttle. Once you started, like, writing down the principles of it Yes.
Adam Leventhal:Sounded bananas. It right. You're like, wait. Why all of a sudden are we, like, injecting a ten millisecond delay? Like, what is there a principle behind this or is this just the thing that is happening?
Adam Leventhal:Yeah. And but you're right. As you start to write it down and put it into a paragraph, you're like, wait a minute. Woah. We can't ship this.
Bryan Cantrill:Right. Right. Don't give them this interface. Yeah. Exactly.
Bryan Cantrill:So I think that that I I I empathize with it, but you gotta get over it. A %. Like, get it. I get where you're coming from. It's incomplete.
Bryan Cantrill:It'll never be complete. Give it to me anyway, please. Yep. I'm afraid that my source code has got bugs. I get it.
Bryan Cantrill:Me too. Spoiler. Spoiler. Exactly. Well, and it's like, maybe your HAL layer is so thin that it doesn't have bugs, in
Adam Leventhal:which case, just give us
Bryan Cantrill:the goddamn interface. Like, we and if your HAL layer is actually sophisticated, it's gonna have bugs. Yes. So, yeah, spoiler alert. It's got bugs.
Bryan Cantrill:So same thing. Like, get over yourself and and the I'm afraid that it will reveal mistakes where you are gonna see some, like, design decisions that I've made that I'm really and it's like, sometimes it's like, I'm willing to give you this documentation, but I don't wanna give the world this documentation because I don't want the world to see this thing that I've done that's really not great. But, like, again, get over it.
Adam Leventhal:It's it's in the same category of bugs sort of. I I guess they're like bugs that you can't fix. But it it but it's still like, yeah. It is what it is. You'll get it next time.
Bryan Cantrill:Right. So and then we I mean, if yeah.
Ryan Goodfellow:If, I mean, if it is revealing mistakes, like, the act of opening up your your interface or your source code is revealing mistakes, then it's gonna reveal mistakes at design or engineering time instead of, you know, operation time when it's gonna cause your customers an outage.
Adam Leventhal:Or it may be the case that those mistakes are exactly the things that are gonna be of most interest as
Bryan Cantrill:you're telling me, like, when Yes. Yes.
Adam Leventhal:Yes. When everything is falling apart. Understanding that, like, yeah, I'm I'm I'm unfortunately at a part where this part is, like, not doing its best job. Oh
Bryan Cantrill:my god. You're reminding of it to the Toshiba firmware issue where you're like, you have a three thousand millisecond sleep somewhere in your firmware. Like, if you can give me the source code, I'm gonna be able to find where I but exactly. You're just like, I'm sure I'm in some, like, x x x, I hope we don't get here kind of code, but, like, I yeah. I'd like, we just that's the stuff that's gonna bite you Yeah.
Bryan Cantrill:Is the stuff that are there are, like, the the or where you're working around some other issue.
Adam Leventhal:Right. The things that you're itchiest about. Yes. Like, that's the thing I care most about. I care
Bryan Cantrill:most about that. I care most about that. And it's like not again, it's like, I don't care about, like, great.
Adam Leventhal:Yeah. And, also, I paid for the thing already.
Bryan Cantrill:Like, I've already bought the damn thing. I'm not returning it. Exactly. So, yeah, you gotta just get over that one. Unfortunately, I think we are in an era where errata are more publicly available.
Bryan Cantrill:I do think that, like, Spectrum meltdown kinda helped on that, where Intel's first reaction on that was really not great, and we they kinda became progressively more, more public in their disclosure earlier. I mean, still plenty to go. I I I shouldn't be overly forgiving, but I do think it's like, look. You gotta treat this like an errata. This is gonna have to be public.
Bryan Cantrill:You're gonna sorry. So you just get over yourself. And then we get to the the and, Rai, this is the the the the paragraph that I added after you and I went back and forth today. Because I think this is a big reason. And, Ryan, this is something that you would kinda and I hopefully I'm kinda paraphrasing what your observation correctly.
Bryan Cantrill:But this is something that we've heard from other folks who are trying to prevail inside of their companies, sort of like they're they're up against some other wall. And I think it is literally fear of, like, I just don't want someone else to write the software. I just don't think they can do it.
Ryan Goodfellow:Yeah. I I hear this all the time. Right? The the the vendor of whatever chip is worried about getting a bad rep, because someone else is gonna write software around their chip. It's not gonna behave in the way that they wrote the software, and they think that, you know, the this is gonna get out, and people are gonna think that the part is shit.
Ryan Goodfellow:And that's, you know, as irrational as that may sound to you or me, that is something I hear all the time.
Bryan Cantrill:And I think that I agree with you, and I think that is the more charitable spin on it. Like, I'm worried about reputational damage. And, Jack, reputational damage on my software running on your part. I mean, it's like, really kind of stretches credulity there. I actually think that there's a there's kind of a less charitable interpretation, which is like, I am hand this chip is being handed to you down to you mere mortals.
Bryan Cantrill:And you mere mortals are not able like, in fact, if you were able to write better software than the software that I am writing for my own part, what would that say about my, like, my engineering? Like, they kind of put themselves in this kind of competition with their own customers emotionally, I think. And, obviously, it pains me greatly to to, give Bill Joy any credit that he's not already do. I mean, he was the the the this must have stood out to you, Adam, as No. I think
Adam Leventhal:I saw draft before that line
Bryan Cantrill:was Before Joy's Law. Yeah. Yeah. Yeah.
Adam Leventhal:But I but I saw someone, some comment on it. I thought that was great.
Bryan Cantrill:But, like, I do like the the and because this is in reaction to Microsoft who Bill Joy, I think rightfully accused of being IQ monopolists of, like I mean, it's kind of amazing to me that Microsoft could think that they are, like, the smartest people on the planet in the I mean, just shows how, like, ignorant of the this other world of computing they were, that they were only looking like, yes. Like, congratulations. You are the biggest fish in the personal computing pond. Like, there's this whole other vista of computing in terms of minis and and workstations and mainframes, and it's like, sorry. You're like, you're actually, there there are smart people everywhere.
Bryan Cantrill:And I do love this kind of distillation of it. Like, no matter who you are, most of the smartest people in the world work for someone else. And I really think that you've got to building a software ecosystem means internalizing that, really internalizing that. And I think if you because you think about, like I mean, this is honestly where, like I mean, just from the the the the at the top, we're talking about Intel's Foundry issue. Like, the Intel has not internalized this on the, like, the Foundry side.
Bryan Cantrill:Right? And I think a lot of where folks, like, they want a software ecosystem to exist, but I think they're unwilling to internalize that, like, okay, you want a software ecosystem to exist, you need to acknowledge that there are, like, more smart people outside your company than in it. And I think that's an
Adam Leventhal:that taps into another source of fear, which is, will I stall my job?
Bryan Cantrill:Yeah. Yeah. For sure. Which is
Adam Leventhal:real, but, like, embracing the community. Right? Like, be be be a leader of that ecosystem rather than an opponent of it.
Bryan Cantrill:And I just feel like, you know, we would talk about how, like, making the case for Dtrace for people and that until they had used it to, like, actually solve a problem, there was kind of, like, a bit that they couldn't get over the hump of. Yeah. I kinda feel like until you have had someone that you've never met before kind of swoop down with something extraordinary, an observation, a piece of software they built on your thing, an application of your thing, something you just, like, did not anticipate. You know what I mean? Yeah.
Bryan Cantrill:Where you, like, you just don't appreciate this.
Adam Leventhal:There is something magical about it. Just like a person you never met, never spoken with. Yes. Appearing out of the and doing something amazing or or just contributing, making the thing better.
Bryan Cantrill:Making the thing better. Finding a bug. They they they you know, using it. Like and, like, there I kinda feel that has to happen to you before you really appreciate it. Yeah.
Bryan Cantrill:I don't know, Ry. What do you what do you make of all this?
Ryan Goodfellow:Yeah. I mean, an observation about all four of these points is that they're all gonna be true no matter what, transparent or not. Right? So afraid my documentation isn't complete. Well, congratulations.
Ryan Goodfellow:Documentation is always going to be incomplete. Going back to being a large Cumulus customer in my last life, they put their documents on GitHub, and we could submit pull requests. And their documents were much better because of that because me and a whole bunch of their other users, provided them with, like, not just feedback, but, like, hey. This is how I would change this documentation to make life better for operators. And so I'm afraid my documentation is complete.
Ryan Goodfellow:Well, it's always gonna be incomplete, but it's gonna be way more incomplete if you're not transparent about it. Afraid my source code has bugs. It's always gonna have bugs. Right? Like, why did I write my own data plane code?
Ryan Goodfellow:Not because I thought, like, my code was gonna be better than, like, the Cisco vector processor that was, you know, in its heyday in 2016, but, you know, because I could understand the bugs. And there's always gonna be bugs. That's just how software works. Afraid of revealing mistakes, it's much better to reveal mistakes in code than it is, at operations time. And I'm afraid of someone else writing the software.
Ryan Goodfellow:I think that's the hallmark of success. Right? If if your software outlives you, then you've been successful. Like, when I came to Oxide, that was, like, kind of the first time in my career I was stepping away from, like, a large scale software project that I had built. And one of the most gratifying things for me is that software project is still surviving and thriving in the community that exists and without me, and that's what you want.
Bryan Cantrill:Yeah. Yeah. That's exactly it. That's exactly it. And so then you get so and then, Roy, I think you got a good point.
Bryan Cantrill:Like, these are true no matter what, whether you're transparent or not. So you might as well be transparent. And and and, again, are the the what, you need this to be able to actually build that software ecosystem. And by the way, look at at Linux and x 86, Intel in particular. Intel was successful in the server space because they had an ecosystem.
Bryan Cantrill:And, you know, Ryan, you again, you've you kinda make this point until your horse about, like, we other domains should look to that as that's a path to follow. I think also be interested to know what your take is on this or what what folks in the chat think. I think this is a big part of NVIDIA's success. So because NVIDIA is very proprietary on the control interfaces, but has actually been very transparent on PTX and on CUDA. Have been very thoroughly documented just like what the DeepSeek folks were able to do.
Bryan Cantrill:They were able to code directly onto PTX. And it's like, yes, that's being translated into something else on the part, but they were able to deliver magical results for themselves by coding straight to PTX. You don't actually need to to code to CUDA.
Ryan Goodfellow:Yeah. And, like, you know, I'm I'm definitely not, like, a GPU expert, but I I definitely have looked at the NVIDIA space just to, like, see what they're doing there. And they have PTX, which hooks into LLVM, which is, like, has a very well understood, like, SSA compiler model. So you can while you don't necessarily have what their assembler is compiling down to in the true instructions of architecture, I don't believe for the NVIDIA chips, you do have a very well understood low level single static assignment compiler model that you can reason about, which is saying a lot.
Bryan Cantrill:Yeah. And and how's that working out for NVIDIA? I feel I like but my understanding is thanks to Pat Kelsinger's departure of Intel and the abandonment of LaraBe, NVIDIA is doing very well.
Adam Leventhal:That's right.
Bryan Cantrill:So I think that there are case that there are another one of these cases for transparency kind of paradoxically because you don't think of NVIDIA as being a transparent company necessarily because the control interfaces are still proprietary. Yeah. But I and I also think that NVIDIA would be even more successful if the control interfaces were although, yeah,
Adam Leventhal:that's bonkers.
Bryan Cantrill:Yeah. I guess that's
Adam Leventhal:What's that even mean?
Ryan Goodfellow:Am I hallucinating that earlier this year NVIDIA said that they're gonna provide an actual open source Linux driver for some segment of their GPUs? Is that true?
Bryan Cantrill:We it it I mean, they certainly should. It's certainly if not true, it certainly should be true because I think that they I think NVIDIA's got more to gain. I think they've got more to gain by doing it. I think they would actually be, they would allow their chips to be used in more systems in ways that we and, again, I mean, I don't think I don't think I think they're doing fine
Adam Leventhal:without without my maybe they're listening, though.
Bryan Cantrill:Maybe they're listening. Jensen's like, I'll take a note. I could be even okay. Thanks. So we I mean, we think that all of this obviously should be obvious to vendor of silicon because, like, you're helping yourself sell more parts.
Bryan Cantrill:Like, you got do you know how many open source companies would kill for this kind of revenue model? Where it's like, in order to use my open source software, you gotta buy a license key. That's like a $3,000 license key at, like, you know, 300 watt TDP or whatever it is. You know? It's like the the fact that, like, the the software only runs on your part.
Bryan Cantrill:You make money by selling the part. Why are you not how is this not obvious? This is usually in the part of the conversation where I'm, like, begging. This is like you get to, like, now I've oscillated from psycho cop, and now I'm, like, back in the good cop where I'm just like, come on.
Adam Leventhal:Just like anger into bargaining.
Bryan Cantrill:And, like, just, like, help me reason with that guy. I know that guy really loses his temper. Like, you're that guy, though. And I do think it's I mean, it's worth, like, if you insist on being opaque, you are forcing people to keep their bag packed. You because you're saying like, no.
Bryan Cantrill:No. Like, I'm sorry. We're gonna be opaque. There's gonna be a hard interface here. Like, alright.
Bryan Cantrill:Well, there's a hard interface there. Then, like, you're actually forcing me to be insulated from you in a way that kinda preserves my future optionality. Like, you shouldn't wanna do that.
Adam Leventhal:That's right. I I'm saying I want to invest. I'm saying I wanna build a bunch of software that makes me even more tightly coupled.
Bryan Cantrill:I want us to, like, have a bunch of babies together. I want us to lie. I want wanna move in.
Adam Leventhal:And you're saying I get to leave a toothbrush and that's it.
Bryan Cantrill:That's it. And it's like, okay. Well, I'm just saying, like, I'm kind of a catch over here. So I don't know. This is where it breaks out.
Bryan Cantrill:But the I I think that, like, that there's a real peril that, like, if and by the way, if the thing that comes along that you you kind of preserve my optionality if the thing that comes along, by the way, is transparent, like, you're never gonna get back in. Right? And this is kinda what I mean, this is what has arguably not that TSMC is necessarily transparent, but this is the problem that Intel has with that TSMC mode. It's like, I don't know how you're getting back in. You know, it's it is you now have to do something crazy to get back in, and it's gonna be really, really hard to get back in once you lose it to us.
Bryan Cantrill:It's like so just don't do that.
Adam Leventhal:Right. If you wanna get crazy, you know who to
Bryan Cantrill:You know who to call. Exactly. I'm I'm your man. I'm telling you. CEO just for, like, a month.
Bryan Cantrill:Ninety days. It's like
Adam Leventhal:some elab stuff right now.
Bryan Cantrill:I know. Just I know. I'm just I know. It's like Bruce's millions for Intel CEO. This is a very weird kind of Alright.
Bryan Cantrill:I need to stop. Actually, don't want that, so I need to stop kind of obviously fantasize about it. But hopefully, is like spells it out. Adam, are you sold? Will you will you make your hardware software interface transparent?
Adam Leventhal:Grudgingly. But yes, I I'll overcome my fears and go for it.
Bryan Cantrill:Are you gonna stop making this bullshit argument about these legal agreements that we both know don't exist? Let's get the lawyers in.
Adam Leventhal:It's gonna they're nice. They're fun.
Bryan Cantrill:They're fun. Exactly. So there we go. That's the that's how we got here.
Adam Leventhal:I think it's great. And I don't know that I think like your email to the ZFS folks.
Ryan Goodfellow:Yeah.
Bryan Cantrill:Yeah. How does it rank in terms of, like, level of
Adam Leventhal:First of all, I I As caustic? Much less caustic, but I I just don't know what's gonna change this industry. Like, I don't know what
Bryan Cantrill:No. We're gonna change it. We are gonna change it.
Adam Leventhal:Sweet. Okay.
Bryan Cantrill:Yes. This is what's gonna change. We're gonna change it.
Adam Leventhal:Okay.
Bryan Cantrill:We, the people, we're gonna change it. I I really do think that we are I I, like, I think because so here's the reason for optimism. The reason for optimism is what this is such a win that when it happens, a la x 86, it just goes supernova.
Adam Leventhal:I'm with you. You need we need that we need that kind of patient zero of this. Like, you need to see for for folks to be able to see what they could be, what could happen.
Bryan Cantrill:What could happen.
Adam Leventhal:And then be able to imagine themselves doing it because it will take courage.
Bryan Cantrill:Yes. Will take
Adam Leventhal:courage from engineers and courage from leadership. That's right. And even courage from lawyers.
Bryan Cantrill:That's right. Some some lawyerly courage. You lawyers are gonna do what they're and the lawyers people I think people blame the lawyers unfairly. The lawyers are gonna, like, leave me out of this. Like, I the the the you signed the agreement.
Bryan Cantrill:I told you what you're agreeing to and not agreeing to. Like, I got I that was your decision. That was your decision. Exactly. But, yeah, I mean, it it would be and it would be great for, like, an FPGA.
Bryan Cantrill:I think that, you know, I think an FPGA could do something extraordinary by really I mean, I don't know that that's gonna I mean, we've got Lattice, but I don't we're not seeing it. And I think then I have I feel I had a three year per day. I had a heady year for open EDA predictions that have all fallen flat. I I had, like, one of those years in there. I don't know.
Bryan Cantrill:Yeah. Twenty twenty two, '20 '20 '3, '1 of the years I was on an open EDA vendor.
Ryan Goodfellow:And I I think in the networking space, we can definitely point to what hasn't worked. I don't think we have to make an argument that programmability is valuable, but if you look at programmable data plane networking technologies, none of them have survived. Right? They're all in the the dumpster of time. And so, clearly, something isn't going right.
Ryan Goodfellow:We think we have, a beat on the things that aren't going right. So, hopefully, we can we can convince some people of that. In my computer closet at home, I have a reminder hanging up Yes. Of a Netronome card, just sitting on my top shelf reminding me of the the perils of opacity. The Netronome was an amazing piece of hardware, and it absolutely flopped in the market.
Ryan Goodfellow:So I just keep that as a a little reminder.
Bryan Cantrill:Absolutely. I mean, that's it. Like, go in like, walk through the cemetery. And, I mean, I did have a line in there that I I did delete encouraging people to take a stroll. Like, any stroll through the cemetery, go read the tombstones in the cemetery, and these are not transparent parts.
Bryan Cantrill:They're opaque parts. And, honestly, it's like, it is part of Tofino's epitaph. It's like, opened to it opened on its deathbed too late, and he didn't even open all of Like, he can't even get the microarchitecture document for, a dead part. You know? It's like, it's it it it's too late.
Bryan Cantrill:So we we I I I think that we we we can prevail. You you okay. Here's the other reason I think we can prevail. Because it's so part of the reason I wrote the RFD is because of the number of times we get people inside of these companies that absolutely agree with us, and they are trying to win arguments internally. So you it just like open source.
Bryan Cantrill:You know, it's like the the when the world was very reluctant to go open, but once the dam broke, the dam broke. And, you know, and then and then the dam, like, got weird and, you know, things relicensed themselves and, you know, it's like that we know, a bunch of weird stuff happened, but we're not asking people to open source anything. We're asking for true open systems. So I'm optimistic. Good.
Bryan Cantrill:Rye, you optimistic? We are we're we're gonna do this thing.
Ryan Goodfellow:I'm optimistic about us achieving our goals. Optimistic about when, we'll see.
Adam Leventhal:I think I heard Rye at some point say that we're we're gonna have these open, these transparent hardware software interfaces because we're gonna be building the ASIC.
Bryan Cantrill:I well, I think that we look. I mean, at some point yeah. That that's what takes.
Ryan Goodfellow:That's what it's gonna take. I mean, I Yep. I build my own little ASICs in my spare time because I I I don't know. That's just how I'm wired. Oh, same.
Bryan Cantrill:But yeah.
Ryan Goodfellow:We'll we'll be doing that eventually.
Bryan Cantrill:Yeah. I think in in you know, and and where we talk about it most frequently is the silicon that we've been kinda frustrated with. And, you know, whether that's the and we know it's, like, it's obviously really hard. It's not something we would do casually. But, yeah, it does feel like if if if we have to, that's what we'll do.
Bryan Cantrill:We don't want to necessarily do that. We would love to be able to partner with folks. So Yeah. Help us out partners, and, let us know if you've encountered, arguments that we missed. Let us know.
Bryan Cantrill:But, Rai, thank you again for, for inspiring, inspiring this to be written. And again and and with I I spent a lot of time talking to Rob about this as well. So this is definitely representing, oh, like, a lot of us. And the the discussions and arguments we have had arguments wrong word, but the, kind of how we persuade companies. And thank you all.
Bryan Cantrill:Like, there are a bunch of folks inside of our partners that have really been trying to make these arguments internally, And we're really trying to put arrows in their quiver because they've been huge advocates for Oxide and for our approach because they see that it's really important.
Ryan Goodfellow:Yeah. Thank you very much for having me tonight on this. You know? Like I told you earlier, this gets at the core of why I came to Oxide. So really excited to make a difference here, and looking forward to to living in that future world where operators can actually understand the metal beneath them.
Adam Leventhal:Amen. Here's
Bryan Cantrill:to understanding the metal, and here's to transparent hardware software interfaces. Alright. Thanks, everyone.