Oxide and Friends

RFDs--Requests for Discussion--are how we at Oxide discuss... just about everything! Technical design, hardware component selection, changes in process, culture, interview systems, (even) chat--we have RFDs for all of these, over 500 in a bit under 5 years. Bryan and Adam were joined by Oxide colleagues instrumental to RFDs, from their most prolific author to those making them more consumable.

In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleagues, Robert Mustacchi, David Crespo, Ben Leonard, and Augustus Mayo.

Some of the topics we hit on, in the order that we hit them:
If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Creators & Guests

Host
Adam Leventhal
Host
Bryan Cantrill

What is Oxide and Friends?

Oxide hosts a weekly Discord show where we discuss a wide range of topics: computer history, startups, Oxide hardware bringup, and other topics du jour. These are the recordings in podcast form.
Join us live (usually Mondays at 5pm PT) https://discord.gg/gcQxNHAKCB
Subscribe to our calendar: https://sesh.fyi/api/calendar/v2/iMdFbuFRupMwuTiwvXswNU.ics

Ben Leonard:

How does it feel to be doing this from the morning?

Bryan Cantrill:

You know, Ben, great question. The, I gotta say, the sun is pouring into this room at Oxide. I I don't think I quite realized how much absolutely nuclear bomb direct sun exposure this thing gets in the morning. So I'm kind of like hiding a little bit in the shade, but it is glorious. I gotta tell you, I got my morning cup of coffee.

Bryan Cantrill:

I'm I'm I'm really excited for this. How does it feel for us to be doing this in a time that includes you? I'm so sorry that

Ben Leonard:

This is good.

Adam Leventhal:

Can you hear me?

Bryan Cantrill:

Adam from Italy. Yes. Yes. I mean, don't take this the wrong way, Adam. But I think you I think you sound better than you do in Alameda.

Bryan Cantrill:

I mean, I think

Adam Leventhal:

I think so too. I I have a upgraded microphone for for this.

Bryan Cantrill:

I just wanna make clear that I'm not accusing you of using an AI generated photo of Italy. I'm just saying, like, the the photo that you posted is a little on the nose for what for what a generative AI might imagine that Italy would look like. It's still a little too on brand.

Ben Leonard:

If you spent any time in, a European Airbnb regardless of the country you know that they they always have a a large canvas of a black and white scene of London with a colorized red bus so I think that's the first clue this is not actually Italy.

Bryan Cantrill:

You discussed things that are not adding up. I do. I gotta say, I love the the European window dimensions. Hello, the tall narrow windows. It's nice.

Bryan Cantrill:

It's a very European thing. You know, I think it's like, that's how the AI knew to generate in Europe.

Adam Leventhal:

Okay. Yeah. I know. Another very European thing, when I was, going through airport security in Germany, they pulled out this microphone and wanted to have a chat about microphones.

Bryan Cantrill:

Are they are they listeners to the pod? Are they, like, dude,

Adam Leventhal:

now they are.

Bryan Cantrill:

I I I exactly.

Adam Leventhal:

just one

Adam Leventhal:

person at a time.

Bryan Cantrill:

We'd be like even some baseball one.

Ben Leonard:

No. I'm loving this.

Bryan Cantrill:

I have managed to, if I offend our German listeners

Adam Leventhal:

There

Adam Leventhal:

you go.

Bryan Cantrill:

I'm I'm We apologize, Germany? We apologize. Exactly. I I never messaged to our German community from Oxide and Friends. I get no apology videos.

Bryan Cantrill:

I've kinda got I'm trying to bring apology videos back. You know, the I I just feel like, guys, I can't believe we have to have this episode. I'm I never thought I'd I'd be saying this, but Germany, I'm so sorry for what Brian did. I think I'm making it worse. I am really excited for this.

Adam Leventhal:

Me too. This feels like another long time coming kinda episode.

Bryan Cantrill:

Really long time coming. I feel like this is like the episode we were in all along. I can't I I feel that it was, you know, I was I was listening and I'm but I'm actually I'm really glad that we are doing it now and not a couple of years ago, because we have made so many very recent changes that are really, really important. And I also feel that we one of the things that we have done, and I will talk about this a bunch over over this episode, but, we have open sourced the software that we use to do this. And, I think that if anybody's listening to this wondering, like, god, I really like a lot of aspects of the system, but I don't know.

Bryan Cantrill:

I don't I don't wanna just, like, take theirs. It's like, you can definitely take it. We are if the and certainly

Adam Leventhal:

If I may, at risk orienting people, what we are talking about

Bryan Cantrill:

is Okay. Oh, now but they were this again. Okay. You know, it's like we're only a couple minutes in. I was gonna wait for another 10 minutes.

Bryan Cantrill:

But, yeah, for sure. Go ahead. Fine. Orient If

Adam Leventhal:

I blow the big reveal.

Bryan Cantrill:

Blow the big reveal. Alright. Oh, what are we talking about, mister? We have to tell people what we're talking about? Go ahead.

Bryan Cantrill:

Beat the tweet.

Adam Leventhal:

Yeah. Mister, I read the comments on social media.

Bryan Cantrill:

That's fine.

Adam Leventhal:

We are talking about oxide requests for discussion, which is, you know, we were talking about what to name this episode and you stumbled onto, I think the backbone of oxide, and that is just so spot on. It is the mechanism by which we make all kinds of decisions, decisions about the product, about the company, the way we articulate ideas, articulate research. It is really the the durable entity that has, you know, been with Oxide from very, very early on.

Bryan Cantrill:

Very early on. And, I mean, it predates Oxide, really. And we'll kinda get into the history there. But it really is the backbone, as you and I were kinda iterating on titles. So I think you were kinda wanting to me to go bigger on titles.

Bryan Cantrill:

And and you're right. I mean, it's like, this is this is really important. Then you said this in our in our episode on culture idiosyncrasies as well that, like, this is idiosyncratic. And on the what like, it doesn't feel like it should be idiosyncratic, because we do not invent literacy, and we do not invent the importance of writing down one's ideas or reading the ideas of others. But I think a bunch of the things that we've done, borrowing from ideas elsewhere, have added up to something that is idiosyncratic, but I think it shouldn't be, which is the reason that I I I kinda bridle with that, but it is idiosyncratic, and it but regardless of whether it's idiosyncratic or not, it is essential to oxide, and it has been, it really, really, really important.

Bryan Cantrill:

It is really hard to overstate the importance of RFPs, and they're becoming way more important, actually. Yeah. It it Well,

Adam Leventhal:

in in fact, the longer they live, I think it's I think it's what you're getting at. Some of these ones that we wrote ages ages ago have, become really essential to understand where we were. I mean, which is not to diminish the value of the ones that we're we're still writing, but it's really proving its its value as we look back and as as these things age.

Bryan Cantrill:

Yes. And the other thing that has been just extraordinarily important is the ability to make single RFPs public. Mhmm. And that has been that is life changing for us, because it now allows us to, begin to and we've done this over time, but, we we are getting more and more RFPs out there, and allows it for us to be our that kind of public vector as well. Yeah.

Bryan Cantrill:

And so that has been absolutely essential, and that's all built on the the terrific work of of Ben and Augustus and and Dave Crespo who are all with us today. So their work has been, like, it kind of started off as, like, oh, like, this is nice. Like, oh, this is great. Like, this is great. And then, you know, they again, we'll get into the whole history of this, but then added things where it's like, oh, wow.

Bryan Cantrill:

Okay. This is really great. Like, this is this is profoundly great. And now we're just like, okay. This is this is now like load bearing.

Bryan Cantrill:

This is this is, we now cannot envision the system without this. This is extremely important. So, but I'd like to get into the history If we may. Well, if I may.

Adam Leventhal:

If well, it but, so I wanna get in the history, but I I I quick quiz question for you. Because I was looking back at RFT one. RFT one describes the RFT process.

Bryan Cantrill:

Yes.

Adam Leventhal:

Do you do you do you know when that was written? Because, obviously, it was very early in the company's history.

Bryan Cantrill:

Would you wanna practice on on when that was? I will tell you that it is, there's a reason that it's written. I mean, so Jess and I are listed as the coauthors because Steve was not yet working in the garage. So this is gonna be this is going to be in August of, I think, of 2019.

Adam Leventhal:

Well, maybe 1st commit

Bryan Cantrill:

And maybe maybe we retroactively made it after September 9th. The company's founded on September 9th, 2019.

Adam Leventhal:

So it the, the the commit message was October 18, 2019. So maybe we're waiting for a corporation.

Bryan Cantrill:

Yeah. Yeah. Yeah. Well, I think we're waiting for a corporation. There you go.

Adam Leventhal:

But, like, certainly before before money was in the bank or lots of other stuff, but it it was, like, may you've gotta be one of the first things you did.

Bryan Cantrill:

First things we did. Absolutely, the first thing we first thing we did, and it actually felt like the because the it felt like something we could do as well. So there do you know what I mean? Where because we didn't have money in the bank, we were

Adam Leventhal:

Yeah. Totally.

Bryan Cantrill:

Raising around. And there's like, you're kind of limited about how much of this company you can build with 0 additional people and no money. And, you know, what are the kinds of things you can go do? And like, you can do a podcast We did we were recording on the metal during that time. Right?

Bryan Cantrill:

And, but we were, I we were if you look at, like, RFP 2, I think, is principles and values. Right? So that we were, RFP 3 is our hiring process. So we were getting, like, it's not an accident that these are kind of the first things we're laying down in RFT 1 is the is our the RFT process itself. So, yeah.

Bryan Cantrill:

I mean, it was extremely early. Mhmm. And, that's interesting the first of my messages and until October. Maybe I had this kind of image of but you know what? That as you say that, I am feeling cold in that garage, that unheated garage with it's sort of like I think got, like, frosty in Yeah.

Bryan Cantrill:

October, November, December. Like, we really did need to move into a place with heating. Yeah. And so, yeah, maybe it

Adam Leventhal:

was But instead, you got the oxide office, which is barely heated, but that's a that's a subject of a of a different podcast.

Bryan Cantrill:

Okay. You are apparently emphasizing barely in that. I am emphasizing heated. I think it is heated, if barely. No.

Bryan Cantrill:

We we it's it's fine. It's actually today's gonna be warm too. So it's gonna be it's gonna be a it's gonna be a roaster in the office today. So, in terms of the the history here, and Robert, I'm hoping you can help me out because I'm realizing that, like, my recollection of this I'm gonna be interested to, like, compare notes on the recollection of this. So, the the but the but for the history of this, you actually have to go back before oxide, certainly, before Joyant, actually, and have to go back to our days at Sun.

Bryan Cantrill:

Adam, this is kinda where the the the earliest stuff of this gets laid down. When Sun, I think not unusually, had this architectural review process, and in particular, there's a platform software architectural review committee, PSARC, and PSARC would, would would approve architectural changes to the operating system. And it was a, and I I think I'm sure we've mentioned we probably talked about PSARC in the DTrace episode, but, PSARC was this, this body, that fancied itself, a pretty August body. But you would, develop materials for a PSARC case. And, I mean, Adam, I I'm sure you had the same experience where the development of materials for your PSARC case, I thought was very valuable.

Bryan Cantrill:

Where you'd be writing your own materials, or, you know, you would have a case and would iterate on it with, like, Mike and me before you submit it or or whomever. Right? Roger, whatever. That process, I felt to be very valuable. Every

Adam Leventhal:

so interesting to hear you say that because I felt like at the time, I'm not sure I would ever hit the same way. Like, I remember there were certainly parts of it that were valuable and parts of it which felt like a trip to, like, the DMV or, like, Global Entry. Yes.

Adam Leventhal:

Were were

Adam Leventhal:

you guys

Bryan Cantrill:

That's what I'm saying. Okay.

Bryan Cantrill:

No. No. So what I'm saying is it was, like, the the the part of the process that was valuable was the part of the process where you wrote everything down. That was valuable. Right.

Bryan Cantrill:

Right. And I found many issues in my own thing in my own stuff by I mean, small issues, but, like, okay. Oh, right. As I'm writing this down, I'm, like, I realized I hadn't quite thought about this. Mhmm.

Bryan Cantrill:

And I'm gonna get ahead of some kind of just ill conceived discussion about this. Right. So I'm gonna clarify this. Right? And that was all valuable.

Bryan Cantrill:

And then everything from that was not valuable. And in fact, was awful. Yeah. Because you had spent if, you know, if you work on this earnestly, you're like, I've spent a lot of time thinking about this. And, you know, other than, like, it's not impossible that someone would have a suggestion that would be valuable, but that's not the way the piece are generally phrased.

Bryan Cantrill:

Piece are not phrased as suggestions. It was phrased as, like, approval or rejection.

Adam Leventhal:

And It It was very much this body of of, senior engineers, some air quotes around that, but, like, folks who've been doing this for a while and felt like they were the parole board of, like, what could be and what could not be. It was.

Bryan Cantrill:

You know, parole board is a really good way of phrasing it. Like, I don't know. Like, are you a a recidivist? Is this gonna I is this I I just feels like I know you've had a couple character witnesses in here, but I don't know. I just feel Yeah.

Bryan Cantrill:

You. Oh, and there's a there's a lot

Adam Leventhal:

of that too where it was like I don't know. You you would kinda work work these folks a little bit beforehand. You know? Yeah. The politics of it was was onerous.

Bryan Cantrill:

The politics of it was onerous, and, I mean, I I I I I know we turned the cockroach episode into a post cross episode, so I don't wanna turn the RFD episode into a into a episode. But that said, you're like, why would you be saying this unless that is exactly your intent? But I do think it's worth mentioning that and I would say that if you are in an engineering cultural milieu that has this kind of kind of August review boards, a trick that worked exceedingly well was to deliberately have an issue. And this is, by the way, has a deep history in sovereign insurance, the queen's doc. The, but deliberately have an issue that is a small issue that everyone can wrap their brains around that you are you yourself are equivocating on.

Bryan Cantrill:

Like, I just don't know what to call this thing. I don't like, what should be the name for this and so forth? Or you're important to better about the results. Yeah. Exactly.

Bryan Cantrill:

About the results.

Adam Leventhal:

Even if you've already decided, just, like, whether you're equivocating or not, it's something that doesn't matter, but is a, like, a delightful shed for them to discuss the color of.

Bryan Cantrill:

That's right. And this and I I mentioned this is the this is the queen stock, which goes back to a game called Battle Chess. I don't know if you ever played this game, Adam. This is in, like, eighties, early nineties. It was, and the, the, VP at at the the gate the company was making, Battle Chess, was a famous kinda micromanager.

Bryan Cantrill:

So they would deliberately introduce things that they know that that he would be they would basically occupy him, and so he would all and one of the things is is they had the queen in battle chest holding a duck. And he'd be like you know, he's, like, kinda looking over it very thoughtfully, and then, like, you know, the queen queen should not be holding a duck. Like, oh, god. Yes. Thank you.

Bryan Cantrill:

Yes. Thank you. So they this is a trick that works extremely well. We did this, and it worked all too well. But all that is wasted energy.

Bryan Cantrill:

Like, you should not need to like, all of that is is the organizational politicking, and it sucks, and it's not uplifting even when it works extremely well. Okay. When it works extremely well, it's a little uplifting, but not very uplifting, and it's a waste of effort. But the thing that was valuable was writing all that stuff down. And then so, okay.

Bryan Cantrill:

We went to Fishworks where we wrote nothing down, I think.

Robert Mustacchi:

It wrote nothing down.

Adam Leventhal:

Do you remember our first bug database? It was

Bryan Cantrill:

literally a bug thing. Yeah. A text file.

Adam Leventhal:

Single text file under RCS control.

Bryan Cantrill:

Yeah. Where FTCS control, sir.

Adam Leventhal:

Oh, that's right. Excuse me. No.

Bryan Cantrill:

I will thank you not to refer to SCCS as RCS. I will thank you not refer to our defunct source code management system as a different defunct source code management system.

Adam Leventhal:

And every bug had to be summarized on a single line of text, ideally within 80 columns.

Ben Leonard:

Yes. Anyway. Yeah.

Bryan Cantrill:

We did. Do you know what that bug database is from? That's the Detroit's bug database before we did it.

Robert Mustacchi:

That's right.

Bryan Cantrill:

But you remember that?

Adam Leventhal:

Yeah. Yeah. Yeah.

Bryan Cantrill:

It's gonna I

Adam Leventhal:

was gonna hear that dirty laundry.

Bryan Cantrill:

That text file has a has a has a deep pedigree, friend. But we other than writing one line of text for every bug that we fixed, there was basically nothing. We certainly were dealing with these art, and we were kind of a renegade outfit to begin with and talk about, an outfit that should not have been paroled. The and then I went to join, and we kind of fast forward a couple of years, and I was really missing writing things down. And it was happening kind of sporadically.

Bryan Cantrill:

And, Robert, obviously, this is where I would love to call on your memory, because the at some point, and I my memory is that Alex Wilson, our colleague at Joyant, needed he was working on a container naming service for what was then Triton, and was really looking for a vessel in which to write it down. And I think that there'd been a lot of pressure building in general where we actually need to have a way for people to write things down. And it had been enough time since the scarring of Peace Arc, where because I think, actually, one of the things I I kind of present about Peace Arc is that it took me years to really appreciate the value. And, Adam, maybe you're still getting there, because you're, like, didn't you just say this was valuable? Because, like, that's that's not my memory at all.

Bryan Cantrill:

It took me years to appreciate the value and be able to separate it out from the things that were really that could be frankly pretty upsetting as an engineer. And but, you know, once I kinda had not had this for an extended period of time, it's like, yeah, we really need to get right. I I miss writing. I miss writing ideas down. Robert, is that your do I have the

Robert Mustacchi:

Yes. I think that's that's kinda that's kinda right. And I think we had also reached kind of a a software complexity. So when I when we were talking about this, my recollection is, you know, we're kind of going back on what was good about PZARK, what was bad about PZARK. And in particular, it was kind of like the 20 questions part of it.

Robert Mustacchi:

Like, you know, not necessarily did you write every manual page down into a case, Yeah. That was there. But more of the, like, what are we trying to do and why? You know, we had kinda gotten to the point where both Triton and Manta had reached a level of complexity as well, where it's just like, how did these things intersect? Like, what actually happens when you're building this?

Robert Mustacchi:

Like, you know, and that the act to your point, the act of writing it down just forced you to think about it, a little bit ahead of time. But

Bryan Cantrill:

That's

Robert Mustacchi:

Yeah. I I think that that's that's kinda, you know, where we were, and then I think we kinda used Alex as the first willing, you know Yeah. And that's

Bryan Cantrill:

where I'm that's what I was trying to remember is, like, whether I to what degree it was, like, Alex is, like, I need a place to write this. But, yeah, Alex was definitely however it was, Alex was the first RFT.

Robert Mustacchi:

And Yeah. I think it was it was both. I think we kind of both were looking for it and then looking for, you know, a way to kind of lead with a couple examples of what we were looking for. Because it was kind of a hard thing to you know, if you just, like, what is the actual shape of this kind of amorphous document? Right.

Robert Mustacchi:

And so you wanna have, you know, something that kinda describes, like, what's the problem? You know, what are we actually proposing? How does this intersect? You know, especially for kind of, like, a bunch of stuff for us was, you know, then, you know, what is actually gonna be user visible? Like, how do some of these APIs what are some of these APIs?

Adam Leventhal:

What are some

Bryan Cantrill:

of these APIs?

Robert Mustacchi:

What are these sketches of them so we understand how they fit together?

Bryan Cantrill:

That's right. And I think that, you know, and this has happened to me a lot in my career, where, you know, I'm trying to come up with something that I think is, like, wow, this is, like, totally unique and, like, oh, you know, this is very Socratic or a kind of classic, what have you. And then, I mean, at some point, as we were thinking about this, Robert, kind of the light was going on. So what you want is something that's like, I want it to be pretty loose. I wanna encourage people to write things down more than getting them exactly right.

Bryan Cantrill:

And as we're kinda thinking about this, we're like, wait a minute, these are RFCs. This is what this is what the this is how the Internet was built. Like, how this is right in front of us. This is what RFCs and and, you know, in joints joints are a few repo, I I reference RFC 3, and the the IETF has this request for comments. And RFC 3 just absolutely nailed what we were after.

Bryan Cantrill:

And it was kind of like, I've, like, managed to, like, reinvent something that's right in front of me for my entire career, more or less. And I felt a little bit, a little bit silly for not having seen it. But then they're like, okay. So this is like, we need to take this and and extend it. And I just love this note from RFC 3 that the content of a note may be any thought, suggestion, etcetera, lit related to the software or other aspect of the network.

Bryan Cantrill:

Notes are encouraged to be timely rather than polished. Love that line. Philosophical positions without examples or other specific suggestions or implementation techniques without introductory or background, explication, and explicit questions without any attempted answers are all acceptable. The minimum length for a note is 1 sentence. And these standards are lack thereof lack of them are stated explicitly for two reasons.

Bryan Cantrill:

1st, there's a tendency to view a written statement as ipso facto authoritative, and we hope to promote the exchange and discussion of considerably less than authoritative ideas. 2nd, there is a natural hesitancy to publish something unpolished, and we hope to eat this inhibition. That is, those paragraphs are from RFC 3. That is Steve Crocker, April 1969. And, Adam, doesn't that feel like that is just, like, absolutely true today?

Adam Leventhal:

So spot on. In particular, though, like, just encouraging people to get over the hump of not letting the perfect be the enemy of the gut. Just just write the thing down. Doesn't need to be perfect. Better written, better better sent.

Adam Leventhal:

Let other people work on it. Love that sentiment.

Bryan Cantrill:

Totally. And it's like and as a result, like, there is going to be a wide variety of quality here. I mean, there has to be. Right? There's gonna be and there are gonna be r and there are RFCs that are extraordinary, and there are many RFCs that are forgettable, and that that's okay.

Bryan Cantrill:

We're not trying to make something that is kinda and we're in in particular, what I loved about that is, like, this is not something that's kinda up for approval. This is you writing down your own ideas. And and then, Robert, I as we were kinda thinking about RFCs versus I I my recollection is, like, we just wanted to have a different nomenclature, because we felt we'd be just confusing ourselves all the time if we were talking about RFCs. And we were kinda looking for different nomenclature. Is that is that right?

Robert Mustacchi:

Yeah. That that that's my recollection. Especially, like, in the era of, like, chat bots that will, like, link to things. I think the bot already did, like, RFCX to, you know, the actual IETF RFC. So that was going to be doubly confusing.

Robert Mustacchi:

So I think it was I think it was that. And then I also that way we could just distinguish it from what, you know, from the IETF. Yeah. That's mostly why we chose a different name. Right.

Robert Mustacchi:

I don't think there was too much

Bryan Cantrill:

different I don't think there yeah. I don't think it was too much beyond that. I think it was just kind of, like, you know, I think we would call it, like

Robert Mustacchi:

yeah. Yeah. I think I think the sentence you had, the parenthetical you have at the end of that, the joint RFD intro is probably about as accurate as it is. Yeah. Which is

Bryan Cantrill:

we use the term request for discussion and move request for comments to avoid avoid conflation with the IETF construct, and the more formal writing that is converse represents. I RFCs are now kind of drifted from that RFC 3 ethos inside guys and are now quite a bit more formal. So we wanted to be wanted to draw inspiration from our offcies, but be something slightly different, and RFPs were born. And, Robert, I feel it was like I mean, I don't know if if you and I spoke about this. I'm I'm sure we did.

Bryan Cantrill:

But I remember thinking very shortly after after Alex's RFP won, and we start getting a bunch of RFPs. I'm like, oh, man. We waited too long to do this. I felt like this was so should've done this earlier. I, I mean, on the one hand, it's like it's great.

Bryan Cantrill:

Like, okay. This is clearly, like, I'd this is a big step in the right direction, but it's such a big step in the right direction that, felt like we should've done this earlier.

Robert Mustacchi:

Yeah. And I think we had 1 or 2, you know, like, one off documents that had come along the way. Like, I don't remember if we had something as formal for Manta. I think I wrote up a bunch for the fabrics stuff in, like, 2013, 2012.

Bryan Cantrill:

Yeah. In And But they're kinda they

Robert Mustacchi:

were just kinda like one off documents that we would just kinda store and toss on, you know, the, like, local HTTP server for folks to build to see. And then, I guess, eventually, we had Manta. We tossed it in there. But

Bryan Cantrill:

Yeah. And where did those go? I mean, I'm just trying to remember where were those because, I mean, they they must have existed. I just don't know where they went.

Robert Mustacchi:

I think they've been in the sands of time.

Bryan Cantrill:

Yeah. So this was in, I mean, it's kind of embarrassing that, like, this is in you need the first RVs in 2015. And so we did I how did we do engineering for 5 years, Robert? I don't know. I I Very well.

Robert Mustacchi:

Thank you.

Bryan Cantrill:

Fair. So, and then so, Robert, what were our experiences with RFPs at Joynt? Because I felt like there were some things that worked really well, namely, the idea, the, was great and was important. Then there were other things that I don't know. What what were your what was your take on things that worked well and things that did not work well in that kind of that first embodiment of this?

Robert Mustacchi:

Yeah. I think, you know, one of the but definitely a bunch of things we're trying to figure is kinda like what is the content, you know, how do we section it? I was going back and looking at the, like, the prototype we had for, the RFD at at Joyant. And it's literally just like, okay. There's a title, and that's it.

Robert Mustacchi:

And I think sometimes that not even having, like, you know, not that there's too many more sections of

Speaker 5:

what we have at Oxide, but

Robert Mustacchi:

I think having a little bit there helps. And then I think the method of discussion is also something that we're trying to iterate and figure out, you know, what's the right way to do it. Yeah. You know, some of it was just kinda like, there's an issue. Or, you know, some folks discuss in chat.

Robert Mustacchi:

Or sometimes we had an email thread in it. Worked okay ish. I don't know. Yeah.

Bryan Cantrill:

I mean, I

Robert Mustacchi:

kind of stuff like pros and cons there. GitHub GitHub PR today is differently painful because of GitHub more so than anything else.

Bryan Cantrill:

It is painful, but it is much better than it was at Joynt. At Joynt, like, we did not I mean, the I the grand irony of our RFPs at Joynt is the the discussion piece was not it was really hard to discuss them, because we would do this as with GitHub issues, Adam, and it did not work well, because it was very hard to, like, comment on particular pieces. You know what I mean? You'd be like

Ben Leonard:

Yeah.

Bryan Cantrill:

Comment on, like, hey, the word choice or whatever. And then, like, if you walk up to an RFP that is, you know, old, how do you go and, like, you know, this paragraph is no longer correct. Or you just, like, you filing issues on all these things and just, like, the discussion was not great. So that I would I think was a big, needs improvement. But I think the idea Like, we could see that a bunch of Another thing that was actually really important, Robert, is that it was in a repo.

Bryan Cantrill:

I felt like this is a very important, date. Because folks may look at our feeds and be like, wait a minute. Why not just, like, Google Docs or whatever? And I think I'm sure what were you what were you doing at at at Delphix and then Transposit, Adam?

Adam Leventhal:

We were using at at Delphix, we had a sort of similar system. I mean, born of of some similar problems. When I joined Delphix, the problem I saw was that, like, everyone in engineering was very clear on what they thought we should do, and everyone's idea was different. And everyone felt ignored, but ignored without having really raised the issue. So we started a process fairly similar to to let people express those ideas, express, you know, why they thought we should do them, that kind of stuff, and kind of have their day in court, like, as as part of a group discussion.

Adam Leventhal:

So it was it was somewhere between what you're describing at Joyant and a little bit of the PSAR case without some of the rigmarole and, like, council of elders, but, more focused on like on product discussions. But yeah, what we did was mostly in Google Docs, which I think has some downsides in particular what you're alluding to about like the lack of searchability. And there we, like, layered additional indexing on top of it and, like, a meta document to keep track of everything. But then, some benefits too. Like, it's much easier to have multiple authors simultaneously.

Bryan Cantrill:

Yes. Yeah. And I think that what you what the world wants, and we'll get what we're kind of leading up to is you want kind of a Google Doc front end on an that has an ASCII doc back end going to a canonical repository It's kind of like, that's what you want. That doesn't exactly exist.

Robert Mustacchi:

Yeah. Yeah. And I I think some of the other bits here, you know, locating everything in a repo makes it very easy to see what's changed over time. The discoverability was big, and then, you know, we ended up with markdown initially mostly because like, I don't know. It was, you know, it was the the thing to do to.

Robert Mustacchi:

It was easy to see rendered. Trent, you know, Trent had his, Python markdown too, which we had used for a lot of docs and other things, historically.

Bryan Cantrill:

Well, part of the reason I'm chuckling is because we did markdown at Oxide, and we support markdown and Ascii doc. And Yeah. Adam, I think it was you who took me aside at some point. It's been like, hey, you say that RFPs are in markdown or ASCII doc. Yeah.

Bryan Cantrill:

Yeah. Yeah. Exactly. It's like, you should know that only yours are in markdown. Everyone else's are an ASCII doc.

Bryan Cantrill:

So when you say markdown or ASCII, I'm like, I I actually Adam, I I say, I I don't think I'm embedding this conversation. I really appreciated it. Like, I had I had a giant piece of lettuce on my teeth, and only you had I'm all for all Wait for lunch. My oldest and dearest friends were willing to take me aside and tell me to win an ass I was making of myself. I'm like, you mean no?

Bryan Cantrill:

And and I kinda didn't believe you. I'm like, surely other no. Nope. Go ahead. Markdown.

Bryan Cantrill:

Nope. It's just me. And that was a quick re I definitely, like, I am reworking them right now. They are all gonna be at AsciiDoc. I'm so sorry.

Bryan Cantrill:

Yes. But I think it it

Adam Leventhal:

I think it's fair to say that we're weird for our obsession with AsciiDoc. Like, I think lots is that right? I think so.

Robert Mustacchi:

I mean, probably.

Bryan Cantrill:

Given Given how we're

Adam Leventhal:

weird on everything else. Right?

Robert Mustacchi:

Well, I yeah. Part of this evolution, I think, came from a few steps. I mean, once you're in the non WYSIWYG world, so once you're not in, like, you know, a Word or Google Docs, then there's kinda 2 challenges. 1, and this may be my own fault, but, like, it was it is very hard to in the markdown days, particularly in, like, 2012 to 2015, to see a rendered markdown doc per se, locally without pushing it somewhere up. Yep.

Robert Mustacchi:

And that for me was just a, my own mental foible of just, like, wanting to see, like, hey, does this render correctly before I push it?

Bryan Cantrill:

I think mental foible was a little self effacing.

Adam Leventhal:

It is my own

Bryan Cantrill:

it is my own weakness that I would like to be able to, I don't know, see what this thing looks like before I I I I just feels like.

Adam Leventhal:

Oh, feel Robert, this is like how you always wanna test your code before you integrate it.

Speaker 5:

Yeah. It's like tough tough affair.

Robert Mustacchi:

And then I think the the other thing that we kinda saw as we were getting down, and I think the first joint so actually, Alex Wilson who wrote rfd1 was the first one to use ascii doc, in an rfd at joyant.

Bryan Cantrill:

And I

Robert Mustacchi:

think he did it. Yeah. He was the one who found it, and pointed us towards it. And I wanna say it was probably for rfd 77 there. And once you kinda got to you know, I think one of Markdown's strengths was, you know, it was very simple.

Robert Mustacchi:

One of Markdown's weaknesses is, like, if you wanna create a table, there's still

Adam Leventhal:

Oh my god.

Robert Mustacchi:

Probably, like, 20 different ways to create a table. And if you want a table of contents, like, you know Yeah. God forbid that. You're you're you're really on your own. And so the little bit of additional syntactic features that we got out of AsciiDoc was kinda why for complex documents, we want to move there.

Robert Mustacchi:

You know, easy knowing that there's easy ways to link between different sections or other things or kind of a bunch of the other features. And, you know, there's a bunch of features we didn't even use back then that, now. So our my our colleagues like, Ben Nacker is using in the OXQL RFD, so he could easily do callouts or other things. So I I found it to be a, you know, easy to get something basic that looks pretty good. And then if you need to go complex, you had that capability.

Robert Mustacchi:

Whereas with markdown, it was, you know, you were kinda constrained by the markdown renderer, and no 2 renders were gonna do it the same way.

Bryan Cantrill:

Yes. Markdown is really frustrating in a lot of ways, and especially if you wanna do anything sophisticated or you need things like, you know, footnotes and you wanna cross reference and all these things, it's really not good. I had kind of assumed I didn't realize that ask that ask you doc. I said ask you doc, well, obviously, because I was the, like, the last to get the memo on, like, you're the only one doing a markdown. So as far as I was concerned, everyone was doing an AsciiDoc the world over.

Bryan Cantrill:

So I didn't don't think I realized that AsciiDoc and Chris was saying also that he'd not heard AsciiDoc till coming to Oxide. So I guess that's a little more it's it I mean, the irony of AsciiDoc is that the documentation for AsciiDoc is not great. I do find that I and Robert, I can't imagine that, Chat gpt is very helpful on Asquidoc. Like, I wanna be able to render this, and just like because I feel like some of the stuff in Asquidoc is a little, it's been nice to be able to use Chat Sheet Bitty for AscoDoc. Like, I would love to say.

Bryan Cantrill:

I know. I'm just being on brand to get LOMs in every episode, you know, we're gonna

Adam Leventhal:

There we go. Ring that bell. Exactly.

Bryan Cantrill:

The but so I didn't realize that Alex had kinda discovered AsciiDoc. We, and Robert, do you use AsciiDoctor PDF to render locally? How how do you render locally?

Robert Mustacchi:

I just always used AsciiDoctor and just pulled up the HTML file, and just used that. And it was pretty good. Like it had it had default CSS too, which just meant you could like use it and toss a document somewhere and people could see it and it looked fine on both, mobile and, you know, not mobile, which was really nice as someone who doesn't wanna think about CSS. Right.

Bryan Cantrill:

And it so it should be said also, and, Robert, I just now is as good a time as any. So we have, I I created RFT 510 yesterday. So, I we have perhaps there are more than 510 now, so my period wouldn't be we've got over 500 RFTs, at Oxide. Adam, you had, on an earlier episode, had guessed that we must have millions of words. Did you do this, by the way?

Bryan Cantrill:

Have you already done this, what I'm about

Adam Leventhal:

to do? I haven't looked at the numbers for, like, years, but even then

Bryan Cantrill:

Okay.

Adam Leventhal:

It was overwhelming.

Bryan Cantrill:

Yeah. So, we we we do have, there's 1,400,000 words. And I did look at the I I went through everybody who has kinda contributed to RFD and, just looked at the author's field. So didn't didn't do anything more kinda sophisticated than that. And to see in other words, like, giving everyone full credit for any RFD that they're on the authors for.

Bryan Cantrill:

And of the 1,400,000 words across all RFDs, Robert, you have contributed, 3 over 364,000 words to this one before.

Ben Leonard:

Isn't that amazing?

Adam Leventhal:

That is amazing and profoundly unsurprising.

Robert Mustacchi:

Also terrifying.

Bryan Cantrill:

It is. And, and, Robert, it's been so part of the reason, Robert, obviously, you you are, you are our most prolific RFP writer. And, you know, not not that we wanna, you know, just insert our discussion about the parallel of engineering metrics and that it is the the cultural idiosyncrasies episode. But the, you know, we're not trying to measure RFPs by the pound tier, but I did think it was it was interesting and it was revealing. The other thing that Adam was really interesting to me was the, that that was just eye opening.

Bryan Cantrill:

It wasn't hugely surprising, I guess. The, person at Oxide who has contributed with the most number of collaborators on different RFTs, who's collaborated with the most number of people on RFTs. Do you do you know who this might be?

Adam Leventhal:

I'm gonna guess myself.

Bryan Cantrill:

You are absolutely right. You are you it is like you by far. And on the this is one of these things, like, I didn't, you know, I'm not surprised, but I'm also, like, I was actually surprised by how much it stood out. And it makes a lot and actually, it's because you have also, like, really encouraged a bunch of people to write RFTs, or started RFTs that have I mean, it's just like I I thought it was interesting. I mean, it showed one of the your kinda great strengths in terms of cross pollinating in the organization.

Bryan Cantrill:

Well In terms of the clouding, which Some

Adam Leventhal:

of that was learned from what we had done at Delphix, which was that you weren't you were maybe you weren't allowed, but you were certainly highly encouraged to have a co author. Like, we we kinda didn't want solo authors and things. And part of the reason for that was we thought it made for stronger documents. We thought that, you know, even if it was somebody's, you know, deep held idea, the having a co author to walk them through it and convince them of it and get them on board will strengthen the the quality of the document. I'm sure many of my collaborators on those RFTs would also hasten to point out that I like to do, you know, a half ass job and then leave the rest to them.

Adam Leventhal:

But I prefer to think of it as cross pollination as well.

Bryan Cantrill:

Yeah. Exactly. And we're gonna focus on it. But, I thought that would that was ranging. So so, I mean, both of you have, in different aspects, have interacted with the system a lot.

Bryan Cantrill:

And so, you know, Robert, you have written more ASCII doc than any of us. So we the they've written a lot of of of ASCII doc. I would exceed that that the entire collection of Shakespeare is about 800,000 words. So, Robert, you're getting there. What will Shakespeare keep an eye on your rear view mirror?

Bryan Cantrill:

Robert's coming for you.

Adam Leventhal:

So I would also say if you get folks who are, who are wondering, like when someone joins oxide, like, how do you, like, then what? Like, do you just spend the 1st 3 years reading? Good question. I think, and Brian, you probably have the data on this. I feel like we're on escape philosophy.

Adam Leventhal:

Like, I feel like it is impossible for me to read every RFP that is being produced unless I even if I dedicated, like, all my time, all my work time to just reading the new contents of RFPs.

Bryan Cantrill:

The the absolutely. And because we are, you know, very writing intensive culture, we are adding people, people are we've added some very prolific writers, and, yes, that it it is absolutely. And in the end, it's like that shouldn't be the goal. Right? You you don't want to to actually, like, have to read every line of every update.

Bryan Cantrill:

Okay. So this gets us to to because I I I I wanna get to the the the very important bits here that we have, extended where we get to to Ben's work and and the customer's work customer's work. So the so this was, you know, we start Oxide. We know most of us know that ASCIDOC is the right answer. One of us feels that they are still doing it markdown, so we we we solved that problem.

Bryan Cantrill:

That was I think it was in the 1st year, Adam, when you you had we had the intervention, the markdown intervention.

Adam Leventhal:

Well, for what's worth, I had not heard of AsciiDoc before joining Oxide. So I I had to become a convert in a hurry.

Bryan Cantrill:

Right. Okay. So the and this is and I think it's, you know, AskIDOC has been good for us, I I would say, broadly speaking. Totally. Other than being, I guess, more I've been more unique to us than than I realized.

Bryan Cantrill:

So, the one of the things that that we started doing early on, I mean, very early on, actually, the earliest days because we have it's just like we're on the the name of the podcast now, Oxide Friends, comes from actually our initial Friends of Oxide. So when we started the company, we had a bunch of folks who had and Adam, you were one of the initial, like, friends of oxide, right? Where before you worked here, you were a friend of oxide. We had just people that we'd worked with, we'd known that we wanted to, be transparent about what we're building to get their perspective on. So we added them to that GitHub repo, so they could see actually all of the RFPs.

Bryan Cantrill:

So the GitHub repo is a private repo that has all of these ASCII doc all of these documents in it. The important change that we made from Joynt is the way we did discussions, which is it's a little wonky, but it's better than it was at Joynt by a mile, which is we use the fact that GitHub that PRs are actually better than issues for discussions. I mean, the the GitHub PRs have got issues with them for sure, or they got challenges with using it. But we are we use PRs to discuss an RFD, which at least allows people to give kinda line by line feedback. It's nowhere near as good as Google Doc comments, for example, but it's, it could be worse.

Adam Leventhal:

Yeah. It's it's yeah. It it doesn't get in the way, I don't think.

Bryan Cantrill:

I don't think it gets in the way. I think it is, you know, I think that there there there are some challenges, but I think it's broadly again, it was, like, way better than certainly at Joynt, we didn't have discussion that we should have had because it was mechanically so difficult. So kinda solve that problem, I feel.

Adam Leventhal:

On Google Docs, one of the I mean, huge downsides is, like, you can have a great discussion in a comment or whatever, and someone closes their comment and it's, like, gone forever. And I think one of the benefits of of the GitHub history is, like, it's not gone forever. It's all there. Just hard to find.

Bryan Cantrill:

Yeah. I kinda I do like it. I I I wish, obviously, I wish those comments were themselves under kinda Git management, but which would be kind of that'd be asking GitHub to go to be Yeah. One step too useful. But the, I do like the fact that they're there, and they also don't get in the way of reading the doc, which I think is also important.

Bryan Cantrill:

Because Yeah. You know, somebody like, you know, not all those, you know, their comments, like, okay. Yeah. I got to consider this or whatever, and I'm gonna close it. I'm resolving it, but I want you to be able to still see that there's a comment here.

Bryan Cantrill:

Maybe someone else will come along later and have the same comment or what have you. But, yeah. Great. So we and we do that with branches, and then we we put some automation around that where, you know, you could the RFPs have got different states, and they can be in kind of, ideation was an that was a state that we added out if I recall. We wanted something that was kind of, earlier than kind of discussion state, which is, like, I truly do I've got just, like, no idea what I this is a dream I had.

Adam Leventhal:

Yeah. I mean, this actually started I I think the first one was, the out of box experience stuff, which we kept on stumbling into. Oh, yeah. When we first do the installation, we've gotta do this. Or when when we do the install, we gotta remember to bring, you know, the hex screwdriver or whatever.

Adam Leventhal:

And we just wanted somewhere to dump this stuff down. I mean, in part because RFTs are the only durable place we have for this stuff.

Bryan Cantrill:

Right. And that's right. I mean, and RFTs are the way we kind we kind of are maybe you can argue beginning to abuse it a little bit, but the system was flexible enough that it could be extended that way, and adding adding ideation was was was pretty easy. So that that that worked out pretty well. Then we, but so then it was great to be able to take the repository and add someone as a add add their GitHub ID as a collaborator on the repository.

Bryan Cantrill:

And then that became kind of part 1, though, like, it was just great to be able to do that to get people's perspectives on things. But then, secondly, we would use that as part of the hiring process. So part of the hiring process was after we reviewed someone's materials and, like, okay, we're interested in this person. We would add them to the RFTs so they could see all of the RFPs. And they could, and this was a really important step.

Bryan Cantrill:

It was really important for us to be able to show people all of our documentation about how we were building this thing, to give them a much more concrete idea of what we were doing technically. And what I would tell people is when you get into these RFTs, and you're gonna you know, some of some of them are extraordinary, some of them are are are not even correct any longer, some of them are, you know, there's this huge, you know, huge spectrum of quality and so on. But as you read these RFPs, you will begin to to feel that you will wanna work for oxide more or less, but not the same. That's what I would tell people. And I think it was really, really important.

Bryan Cantrill:

And we had, obviously, because we're by the time we're kind of advancing people's after the materials, we talked about this with our on our episode on hiring with Care Guide. But this is all happening after we have decided, like, to to kind of pull someone into conversations, and we want them to kind of understand what we're building at oxide. We had we did have people in there. And I had one person in particular was like, I thought I really wanted to work for oxide. But and I pointed them to, you know, some pretty gnarly RFTs.

Bryan Cantrill:

And it's like, I read some of these RFTs. And like, I am I'm kind of more interested in the idea of working for oxide than I am working for oxide, which I thought was very helpful. Right? Yeah. It's like

Adam Leventhal:

That's that's great. But, I mean, much better for everyone that they figured out before they joined rather than 3 months

Bryan Cantrill:

in. Absolutely. Absolutely. I and Robert, I I I you should take this as praise, but it was absolutely one of your RFTs that actually That, was the clarifying RFD. You're just like, oh, because I think people don't like the amount of detail.

Bryan Cantrill:

You're like, oh, my god. This there is so much detail here. It's like, yes. Yes. There's a lot of detail.

Bryan Cantrill:

It's very, very detailed to actually go, so, the so that was great. The problem that we had, or that I had, was there wasn't a good way to that we had a couple of big problems. 1, there was no way to search RFTs, which was becoming, like, started off, like, not great, and was becoming really, really, really problematic. And then there was no way to share individual RFTs, and other than, like, rendering them as PDFs and sending them, which I did plenty, but it's, like, that's that's not great. And and and this would be a a good time to get to get you and and kind of David and Aaron and and Agostas in terms of, like, this is the kind of the RFD system that you kind of come upon is us kind of, dealing with sticks and stones and and GitHub.

Bryan Cantrill:

And what is the origin of the the RFD site? Because I'm trying to to remember, like, how far did that that date back, first of all? That dates back, I think, a while.

Ben Leonard:

Yeah. There was there was a an initial version that already existed when I joined, and it was just a list of, it was just a list of kind of RFTs that were rendered. I think I I think at that point a lot of people were were looking at RFTs directly on GitHub. I guess, due to the deficiencies of of of the site. And then I don't I don't remember exactly when we kind of relaunched sort of the version that we have now.

Ben Leonard:

We kind of we've built upon it since. But then we kind of we've incrementally added things, like the full text search. That was a that was a huge pain point for me. I I I forgot what kind of command it was. There was some kind of GREP thing, which would take would take minutes

Bryan Cantrill:

to search the It

Ben Leonard:

was a lot of work. It wasn't a great process. And now we have full text search, we're using some some library that, indexes kind of all ARFDs and searches. Which I think is a good, It's, that that's that's kind of super useful. I can to your point earlier, there's a we're sort of at an inflection point where there's so much in there that even full text search, doesn't necessarily get you everything that you need.

Ben Leonard:

I not, Yeah. And Someone's already someone's already mentioned LLM, so I think I think I'm free to do it without without kind of

Bryan Cantrill:

Yeah. You don't feel like you're breaking up. Excuses this.

Ben Leonard:

I'm not he accuses this shill, but I think there's on on the road map, there's there's, I think, there's room for something that kind of summarizes all RFTs and and, at the very least, tells you where to look. And let's say you're joining outside. There there's a there's a there are a bunch of RFTs that everyone should read, but for most people, like, there there's a smaller group which are kind of repurtenant to their work. And I think then we can use something like that to generate reading lists for people that that join, and they can they can kind of they don't feel like they're just losing their mind, like drinking from the fire was trying to read everything to to to get up running.

Bryan Cantrill:

Totally. When I think so a couple of things like the so we added I mean, the inability to search our days was extremely painful. It was also it's made even more painful, because the the great thing about using PRs for discussion is that you get GitHub's features. The downside of that is it means that, like, the actual repo itself, that is to say, like the main branch, only contains RFPs that have been published. They don't have the ones that are in discussion.

Bryan Cantrill:

So you have to like, search every branch. And so I had this thing, RFP graph that you're you're referring to Ben, that was taking minutes. But it was like, it was minutes, but it was actually like delivering penicillin. So it was really, you know, it was like, oh my god. Thank God, we got some way.

Bryan Cantrill:

We were using Lucene grep. And LM grep is what we were using, using Lucene and it was very helpful, but it was also like its performance would vary a lot. It would take minutes. It was really, really painful. And in particular, like, it was so needed.

Bryan Cantrill:

And I remember, like, getting Steve talk set up with RFD GREP because he needed it desperately. Like, we gotta have a better way to do this. So I can't remember when you did the full text search on the RFD site. But that demo Friday was like a religious experience, where I mean, it was like that affected everyone in Oxide to be able, like, oh my god. I could just go there and search all the RFPs, like clean water.

Bryan Cantrill:

Thank god.

Ben Leonard:

Yeah. And it's such an it's such an obvious thing, but but then kind of, afterwards, it felt kind of magic was it magical? There's a few of those moments where I kind of added something and they it just it sort of, I think, transformed at least the way I used the RFD site. Full text search is one of them getting kind of discussion in line on the RFD site. Was it Okay.

Bryan Cantrill:

So describe that because that was I totally agree. So now we did the RFD site, and we got the RFD site, which also, like, looks great, which is very helpful. You are doing so tell me how you're rendering the ASCII doc because that was a a big piece of this. So you gotta go render the ASCII doc.

Speaker 5:

There's there's there's been

Ben Leonard:

a few versions of it, and I think, like, I I kinda have a love hate relationship with ASCII doc. I think it's amazing because it is so versatile, and it is so painful because it's so versatile. You can you you can you can create so much and I know I kind of I'm aware of a small subset of features. The RFD site is the place to check, like, a renderer for AsciiDoc because you will see every version of, in any way which you can use Askinga has been used on the RFD site, and I will get kind of an issue now and again. And I'm just baffled because I'm like, I don't even know it's possible.

Bryan Cantrill:

Yeah. Sorry. I think I was I was definitely a recent version of that where I was definitely, like, using block quotes and attributed quotes and, you know, the

Ben Leonard:

Oh, Brian, that that that is Brian, that's basic. You should see you should you you you should see the stuff people are doing. But I the the thing is, it's it's really rewarding doing this stuff, but there there's a balance. Right? I don't wanna spend all of my time working on, like, an AsciiDoc renderer, and kind of the styling for this thing if if if if it's just kind of used in one place.

Ben Leonard:

And I think one way in which we have worked around that is we use we've embraced AsciDoc. I'm using AsciDoc for everything. So our doc site is built on AsciDoc, our RFD site, and the blog posts on the on the on the marketing site are all AsciDoc. So what we have there is and that in tandem with our design system that colors our typography, all of that stuff, is unified across both, kind of product and internal sites. Which is unusual because usually you have your product design system, and then and then you have some other designers who kind of are off off, like, playing in the sandpit doing, like, the website, doing whatever whatever they want.

Ben Leonard:

I I'm doing both, so I can I I can have a unified system? And what it means is when we when I when I do the styling for the kind of RFD site, it's shared across all these places. So I can justify spending a bit of time on these things because, because because it's so useful.

Bryan Cantrill:

I feel like The Emperor's New Clothes though, it because we do use AsciiDoc everywhere, I've just assumed AsciiDoc is, like, ubiquitous everywhere. It's like I

Ben Leonard:

can't I don't even

Bryan Cantrill:

have a wand. This is yeah. Exactly. Like, I wouldn't have heard of it.

Adam Leventhal:

Brian, this reminds me of when you walked up to a friend of ours who works at Slack. And you're like, hey. So these Element guys must be really nipping at your heels. They're like, I literally have no idea who Element is. Like, what are you saying Element?

Bryan Cantrill:

Element? You you know, formerly known as Riot? Like You know,

Adam Leventhal:

the the chat system that I assume everyone uses. It's like, no. It's just us.

Bryan Cantrill:

Look. I okay. I just assumed because I we because as Ben as you point out that that I'm realizing that part of the reason I thought ASCII doc was ubiquitous is because it is ubiquitous and oxide, but you're it is ubiquitous in oxide because you made it ubiquitous. So you

Ben Leonard:

could Hey. Hey. Don't don't blame me don't blame me for that. No.

Bryan Cantrill:

No. No. No. I think it's great. I I think it's great.

Bryan Cantrill:

I I think it's great. I I think, like, you it's it's actually I think it's terrific. I think it is so much better than markdown. And, so yeah. I think it's great.

Bryan Cantrill:

And it it like blogs and and so on are all in AsciiDoc.

Ben Leonard:

So really briefly on the on the stuff that we've made for it. So there is a the my main issue of AsciiDoc is that there there is no kind of native, like, Rust or JavaScript library for processing it. There is a JavaScript library which is transpiled from Ruby. And so kind of making changes or seeing how it works under the hood is kind of is is is sometimes challenging. And then it it kind of there's a there's a bunch of assumptions on the way that it works.

Ben Leonard:

Like, it it assumes that you're gonna process a whole last document document in one go, top to bottom. Right? Right. Which which which makes sense if you're processing it locally. In in in my case, I kind of worked on a React renderer of AsciiDoc.

Ben Leonard:

Essentially what I do is I take this AsciiDoc tree, and I work through it, and I render each part as React components. If if if I'm if you're not doing that, you have to create, like, a renderer where you're returning a string. So you're kind of you wanna you wanna render an image, you you return a string, and you kinda drop in you you're accessing attributes, and you're kinda swapping stuff in and out, which isn't ideal especially if you wanna do some more interactive things like our our images. It gets kinda complicated because we're we're using signed image signed images that come from GCP. We have you want kind of we want, like, a little light box.

Ben Leonard:

There's some other kind of features where if you want any kind of interactivity, that that model doesn't work. So we wrote a React renderer for that, but then you kind of React doesn't run once, top to bottom, and you have issues again, the the big issue that I spent way too long on was, every time you render a piece of content with a footnote in it, it assumes that it's it's a new footnote. So every time it's re rendering that piece of text to a footnote, your it's it your your number of footnotes is increasing, increasing, increasing at the bottom of the page. So those things are kind of challenging. And we've found ways around them, and I think, like, I have I have a each one of these versions is motivated by a piece of functionality that we want.

Ben Leonard:

So one thing I've been playing with recently is creating this intermediary format for AsciiDoc. So I take I go through the tree, I process it, I turn it into, like, a JSON object that can be passed easily from the server to the client. And that means that you can kind of pre process it on the server, and then you're not shipping, like, this big this, like, 2 megabyte client library just to handle it on the front end. But and the reason why that's important is, let's say we're in the console, and then and this is a big aspiration of Robert's, is if we if we want documentation that exists within the console, it doesn't feel justified to ship this big AsciiDoc JS library in the console, because that kind of is a bit bloated. But I think I'm kinda working on experiments so that we can have dynamic documentation.

Ben Leonard:

Maybe if you're kinda working on, repairs and kind of swapping components, then then we have that without needing to, do, yeah, kind of do all this stuff. So, yeah. It's it's it's interesting. It's a it's a bit of a challenge at times, but, yeah, the versatility is the kind of is the is is the the thing that's great, and the thing that kind of trips me up sometimes.

Bryan Cantrill:

Well, it's just the fact that we also use this for the documentation is so great. And then we've been able to make the documentation site is anyone that's available to the public. If you wanna I mean, we're you can see all of the oxide documentation, which is great. And it, and then we're able to to kind of cross link that with the console. There's just a bunch of things we can do there.

Bryan Cantrill:

So the the other I mean, and it also looks gorgeous. Can you talk about the GitHub integration a little bit? Because that's been a huge, that's been hugely valuable to get the those discussions. That was another demo Friday that I just think just, like, left people, you know, weeping with joy.

Ben Leonard:

Yeah. That The the the

Bryan Cantrill:

the ability to see comments on the RP side.

Ben Leonard:

That was a I think okay. So I think to to to begin, I think there's there's there's something around, like, the accessibility of this stuff. Oxide has always been, like, really kind of engineering driven, and engineers are really familiar with GitHub, and that's really easy for them. But, I think if you want this process to be used outside in kind of, like, kind of sales and operations and and and and design, then, really, you have to make this stuff as accessible as possible. And we're not there and I think like we're kind of working towards it but getting GitHub discussions directly into the RFD site felt like a big step towards that essentially what's nice about the GitHub discussions is you're leaving line comments.

Ben Leonard:

Right? And one great feature of our AsciiDoc is that you can trace you can kind of, given a line in an AsciiDoc document, you can get the, no, on the return in the rent the rent document, you can get the associated line number. So what it meant is we could query the GitHub API, get all of the comments, collect them in a format that can be rendered easily, which is pretty tricky to begin with. And then we go through line by line, and we position them alongside the content directly, if they're still relevant. And then we have this sidebar, which kind of where you can see the kind of full canonical discussion, and you can kind of jump to the the relevant part.

Ben Leonard:

But, yeah, just that little bit of kind of being able to associate the original document with the rendered document, was enough that we could pull, like, from the API people's avatars, and we can kind of have this little this module that people can click on and can and and and view. Yeah. And that I think that my kind of first experiment was this super ugly way of getting, like, a little avatar alongside the text, but that, yeah, that felt kind of a real, kind of a huge improvement.

Bryan Cantrill:

A huge improvement. And this is is a good segue to the the kind of the RFD database that kind of fronts this thing, I think. Because are there I mean, I assume that's hitting the the database that is fronting it and not actually GitHub directly, but I actually don't know the answer to that. Augustus, it would be a good opportunity to get to get your work in here.

Speaker 5:

Yeah. That does actually talk to GitHub directly at the moment.

Bryan Cantrill:

That's a really oh, wow. Yep.

Speaker 5:

Not for hopefully, not for long. Our goal is to model it in the front end, see how people like it.

Bryan Cantrill:

Yeah. And

Speaker 5:

then once we have a good idea, then we port it back down to the back end. And, one of the complicated bits, though, is so, yeah, we have this back end processing bit so that we don't have to go to GitHub to pull data, like, all the static content from the RFPs and the images and all that. What that the problem though is that database, we wanna talk about sort of revisions of RFPs. And this is something we did recently, but we we currently only, like, exposed the currently latest version of RFP. But we have a database of every commit, like, processed.

Speaker 5:

So eventually, like, right, you could go back in time on the RFP site. And then Yeah. Did you ask how do you tie comments into revisions and what's that mean, comments on a specific revision? How do you then, pull them all together? So in a process, but, yeah, as Chris was said, we're building GitHub.

Ben Leonard:

Because that one of one of the reasons to do that is the so the the, the longer the discussion, the longer the longer the the response takes. So I think, there's one RFD something to do with time. And, yeah, why would that have lots of comments? R f d 34.

Bryan Cantrill:

I mean, it's And

Ben Leonard:

it crash it crashes every time. I think because it I I think it basically it basically timed. Oh, no. Oh, wow. This worked.

Ben Leonard:

What a miracle. But, essentially, the the more comments, the longer the longer the response response takes. And so I think there's there's some kind of funkiness in the GitHub API, which is yeah. We we do have we do have something where, essentially, we we serve the regular rendered version. And then, like David might want to talk a little bit about this, but there's a Remix feature.

Ben Leonard:

Remix is the framework that we're using to to do all this, where you can stream, you can stream data from the server later on. So what we do is we give you the the main document first because that's important, and then later on, like, asynchronously, we we give you the comments just so that's not kind of holding up the the

Bryan Cantrill:

the Okay. That it gives the the reason I had assumed that those were coming out of a front of databases is because it was so snappy. So that's why it's so snappy. Because the you're exactly

Ben Leonard:

The the random bit. Yeah. Yeah. Yeah. Yeah.

Ben Leonard:

And then your kind of mileage may vary on on the actual kind of the GitHub the GitHub

Ben Leonard:

portion.

David Crespo:

So when you first load the page, you get the spinner for the, for the comments because it's fetching those

David Crespo:

in the background. So that's why it

David Crespo:

it feels fast. So it's it's kinda fake fast.

Bryan Cantrill:

It is fake. I I get totally faked up by it though. I'm I because I'm not looking at the spinner. I'm generally looking at the rendered content. So that's, definitely worked.

Bryan Cantrill:

It also should be said that it just the rendered content looks gorgeous, and, obviously, a bad business I mean, you've made sure that everything at Oxide looks gorgeous, or or faces your ire. And, it it it just looks it looks great. It also looks great on the phone, which I think is really important, because I think that, I I think folks are reading more and more RFTs on a mobile device. Right? So, because the Augustus, the other reason that the the kind of the the having that front end database has been really, really important is for the the visibility for RFPs.

Bryan Cantrill:

Could you so could you describe maybe some of the problems we had with adding increasing number of folks to this Friends of Oxide group? Because this GitHub group was getting larger and larger and larger. And they all, I think, all had to be, like, outside collaborators with our organization, I think, if I recall correctly.

Speaker 5:

That is absolutely correct. So if you don't know, when you have a private repo and you add an outside collaborator, start paying for it. So you start racking up pretty big GitHub bills around that as well as it's just a hard list to maintain and, like, understanding who still wants to be on that list. Yeah. So and, also, it meant that we have to give full access to every RFP to everybody.

Speaker 5:

We don't have any way to curate that. And we had some workarounds in place, but, like, they're very we have some static config files that list, like, a handful of RFPs and this, like, subset of people get them. We had no real good permissioning system around it. And that was one of the main reasons to rewrite that database and back end processing was we needed to model RFP access around actual, like, permission sets. So being able to grant things all the way down to you have access to a specific RFD or maybe even have access to a specific RFD and the discussion for that RFD, within the RFD site itself.

Bryan Cantrill:

And this is extraordinarily important. I mean, the the this is, I think, you know, I think the full text search was game changing, the rendering was game changing, and I feel that the the fine grain permissions are totally game changing for our updates. Because we need really needed to be we wanted to be well, the reason I think it's so game changing is because it allowed us to much more easily well, one, we can make our individual RFTs public, which was a real problem. When did we first have the ability to do that? I because I So we

Speaker 5:

had this weird I'd have to look at the actual date. We had a weird workaround prior to the permissions where you could technically mark things as public that we did for, like, 1 or 2, but it wasn't until we actually launched the new RFT API which

Bryan Cantrill:

was, I

Speaker 5:

don't know, sometime this past year to actually, like, flag them.

Ben Leonard:

There was a file with an array of RFD.

Speaker 5:

Yep. Pretty much.

Ben Leonard:

And then it was initially, it was semi public. Right? So you had to you could anyone could log in with like any any GitHub login I think. And then I think David really pushed to get, to to kind of have it so anyone could access the public RFTs about logging in which yeah. Was I think another kind of big big improvement.

Speaker 5:

Yeah. That was a that that's a good thing to point out. There's a difference between public as in you could sign in and see stuff versus now you can actually just go to the URL and you can actually just search the things that are public. As in they're actually public, not this fake behind a log in public.

Bryan Cantrill:

And this has been, I think, huge. Because I I feel especially kinda where we are in the past year having brought this product to market. And now there are more and more aspects of this that we wanna share with everybody. We wanna share with all of our customers, and we wanna share with everyone to see, and I feel like there are, you know, it's just kind of amazing that we've, that, you know, we've only been able to do this in the last year, and I because I feel like I do this all the time. We did this obviously, I mean, the the last episode as was talking about an RFP that was made public.

Bryan Cantrill:

Right? So the in terms of, with a cockroach DB and 508, but I feel like it was, really, really important to have that fine grain control and to be able to have, like, groups as well. So you can say, like, okay, this because I think we've got a group for our contract manufacturer, for example, where they want we want them to be able to see everything about a certain number of RFPs that are not public, is extremely helpful.

Speaker 5:

Yeah. That I so I know it's it was less than a year because it was at, last time when we all met up at the office, that's when I demoed the barely working API. That was just on tied together with string. And, honestly so all of these permissions and groups up is so the RFD site was the first test of this, but that's actually now that's been factored out of the RFD site out of the RFD API and now sort of powers a lot of our internal permission application permissions. And now the API RFD API gets gets updates now from the stuff we're doing, throughout the rest of the company.

Bryan Cantrill:

Yeah. That is that is great. And and then so I so that was in before we met up in October, and then I know that I was with, it was out in New York in November talking to someone who had seen the RFP site, and they're like, how can I get a hold of them? Because I think a lot, I mean, I just and I feel like every time we have an RFP, a public RFP that is on Hacker News or is getting discussed, someone is always like, hey, can someone explain to me how this system works, and how can I use it? Because we, there there's a lot of interest in it.

Bryan Cantrill:

Did you wanna talk about a little bit of and they can be open sourcing of the RFD site and of the RFD API?

Ben Leonard:

Yeah. It's a

Speaker 5:

Okay. But

Ben Leonard:

Go ahead.

Speaker 5:

I'll say the RFD a part of the our goal with the RFD API was planned from the beginning to go open source. So part of it is also just sort of detangling from anything that was, like, oxide specific and trying to make it a more abstract, system. The RFD site, I'll let Ben speak to that one. Was, I think, more of a challenge?

Ben Leonard:

The well, I mean, the RFD site is is is kind of fake open source in that. It's all A

Bryan Cantrill:

message to our community.

Ben Leonard:

Well, I mean, we've got the the the the BSL language the the license on it. No. It's You you can't have

Bryan Cantrill:

any RFDE that implies that you're building a computer. Other than that, you can use it as much as you want. Just can't. You know?

Ben Leonard:

Yeah. Even though it's like

Bryan Cantrill:

Right. Too real.

Ben Leonard:

So, be because it it's still it's still the RFD API is much more, is much broader. I think anyone can use that pretty pretty easily. The RFD site, is and it's it's very oxide at the moment. Like, it has all the it has all the styling, all of that. It's kind of direct in in in the repo.

Ben Leonard:

And so, yeah, I think, eventually, it might be nice to have something, a little more,

Bryan Cantrill:

like Yeah. I mean, I but I'm looking to go. Dave, you'll be because the I mean, okay. The CSS is specific to us, but someone could take this and get it working for themselves. Correct?

Bryan Cantrill:

I mean, it feels like they could.

Ben Leonard:

Oh, for sure. Yeah. Yeah. Yeah. Yeah.

Ben Leonard:

For sure.

David Crespo:

Oh, definitely. And wouldn't they love to have our, beautiful styling too?

Bryan Cantrill:

Everyone should be so lucky to have this gorgeous styling.

Ben Leonard:

Just swap the logo. Easy peasy.

Bryan Cantrill:

Yeah. That's right.

Ben Leonard:

Hey, it's trademarked now. Don't go don't touch.

Bryan Cantrill:

Yeah. Exactly. That's oxide TM to you. Java technology, please, TM. But it's the so in terms of, like, being able to take it, we we got all of the elements that someone could do to go, and what what would be kind of some of the ways in which, someone could use this stuff?

Bryan Cantrill:

I mean, how would one get started if you wanted us to to kind of see what it would like be like to use this?

Speaker 5:

Yeah. In terms of the API, it's, like, probably decent place to start. It's also built on top of other oxide stuff like drop shot and progenitor. So there's, like, a built in CLI that you got out of it for free, which is really nice. The but really, you just need, like, a guest info.

Bryan Cantrill:

I mean, Adam, that you oh, you're proud, papa. You and Dave are I I mean, on a percent of RedTrop. I mean, but you just

Adam Leventhal:

just love it when it Yeah. So super proud. I would say that, Augustus is the only, to my knowledge, other consumer of the, per gender generated CLIs, which are pretty gross. So I feel proud and filled with shame.

Bryan Cantrill:

You okay. So, like, so, Augustus, when I'm using the RFD COI, that is a progenitor generated COI?

Speaker 5:

Yep. Absolutely.

Bryan Cantrill:

That's you should you should feel no shame, Adam. Oh, it's

Adam Leventhal:

not You know that I've there's a function in there whose name is x x x. So I I I there is some reason for shame. I mean, I am very proud.

Bryan Cantrill:

Okay. Alright. So feel shame on that. But the, it so I I use that CLI a lot. I mean, that's what I I mean, I make great use of the our PCOI.

Bryan Cantrill:

So, that's auto that's all auto generated. That's amazing.

Speaker 5:

Yeah. So any shame. Right? The end user doesn't see any of that, though. So Oh, perfect.

Ben Leonard:

It's totally

Speaker 5:

Perfect technology. Shame.

Adam Leventhal:

Internal shame. Internal

Bryan Cantrill:

shame.

Speaker 5:

It the only real assumption it makes a couple assumptions maybe around, our like, if you read rfd 1, actually, is what a lot of this is modeled on top of. So it does make some assumptions about how we, like, run RFDs, like, what the states are, what they mean. Right. But in theory, you can point this at, a repo. I've been working on trying to getting a setup guide to be a little bit cleaner.

Speaker 5:

Veggie did, try ran this himself and sent me a whole bunch of really helpful notes on how painful it was. So there is some work to be done there, but it it did it make some assumptions around you're using AsciiDoctor or some of our use cases around, like, mermaid diagrams that require some installation. But realistically, it should be pretty straightforward to run. And what you get out is essentially a nice database or technically, it runs on Postgres. So you have a nice Postgres database of all of your RFD content that you can then really do whatever you want with.

Speaker 5:

We happen to render them as an RFD site then.

David Crespo:

I wanted to ask Augusta something. So something I'm not sure if we've really gone into is, like, how it works in the GitHub repo. So the whole thing is, you know, when you make a PR, creating an rfd, that is a branch with, that's named with the rfd number. And so when the rfd site is scanning the repo for all I mean, when the API is scanning the repo for all the rfds, it's scanning all the branches. You know, you can't just look at main.

David Crespo:

I'm curious how coupled the rfd api is to like that model because I could you know we run into some friction with that even as nice as it is to have you know PRs as the site of discussion you could imagine not wanting to do it that way

Speaker 5:

so how how coupled is it to that? Right. So that's it's fairly coupled to that, right now. Essentially, we have these so if I go into a little bit, the so the processor works on this concept of jobs where we scan a repo or we accept webhooks from GitHub to generate jobs that are going to then look as look at a specific commit and look within, like, that brand that sorry, that directory structure. So it's less coupled to the, the branching model and more coupled to the struct the document structure.

Speaker 5:

So we have a director that has RFPs, then you have a RFP number, and then a RFP document. Outside of that, you could write a separate scanner that, you know, looks at branches and does make some other determination to create a new job that says, okay. Here's where my RFD is located. I need you to go do in some processing on it, which and that processing then pulls that content down into the database. It'll do things like generating PRs if you need them.

Speaker 5:

Like, when a RFT goes into discussion, we generate a PR on the GitHub repo for people to start discussing it. Things like closing them out or updating things like idle states. Yeah.

Bryan Cantrill:

Yeah. And I feel like that automation has made it because the part the branching bit has been great for discussion, but I think across my things, where you're gonna get to is can feel a bit clumsy at times, for the authors of RFTs. And I feel like the automation there has really helped a bunch. And I I think that given all that, I mean, I I could see why someone might wanna do it a different way, but there's a lot of strength to the model we've got. We're just kinda getting GitHub.

Bryan Cantrill:

It it it's using GitHub for its strengths without being while still having a, a a canonical repo that has everything in.

Speaker 5:

Yeah. I think it and especially from, like, the machine side of it, it makes a lot of sense. And I Benjie mentioned this earlier of part of our goal is then how do we as engineers, I think it's fairly comfortable. We use GitHub all the time. But how do we then bring that to everybody and make everybody comfortable with it?

Speaker 5:

And I don't think the plan is to, like, force everyone to yeah. You have to, like, learn GitHub and be super comfortable with it. So understanding how does, like what's it look like for the RFD API to make a new RFD for you if you just give it a title or

Bryan Cantrill:

Right.

Speaker 5:

When you want to edit content.

Ben Leonard:

We we're doing it kind of incrementally, as well, unless the BU is working on something, on a platform like GitHub, is you can, you can use the RFP API package, use GitHub, and you don't even need a front end, at least initially. You don't need to you don't need to worry about that. And then eventually, you can you can kind of add more as as and when. But, yeah, the accessibility thing, I think, is big. Augustus added an end point so that you could, you you can create RFPs from the API.

Ben Leonard:

And so, yeah, I'm kind of working on a way that people can just create a new RFP from the website directly, And then, a a few months ago, I was working on a little experiment where, it was a a live ASCII doc editor, where you could kind of create and share notes, maybe before they're an RFD, and or or or notes that aren't gonna be an RFP at all. And within that, because of the API, I I had an option where you could then turn an you could turn a note into an RFP and it would kind of hit the API and and it would you it would kind of take all of that content and throw it in an RFD. So I think that's what we're working towards, eventually. Some basic authoring. There's a line, though.

Ben Leonard:

It's really tempting to try and recreate GitHub and Google Docs. I think we've yeah. It's it's we've got enough on our plate, but it's tempting. Tempting to do for sure. And Yeah.

Ben Leonard:

And this Yep. I found personally that kind of working on internal tooling is is, really beneficial. Like, we we get a lot of it, especially as we get bigger, and we really benefit when more people can contribute to RFTs, especially peep engineers.

David Crespo:

The fact that we're standardized on all these tools means that it's sort of this, flywheel, you know, because we use, for example, like you said, the the ASCII dock rendering stuff in the dock site. It's sort of the the investments that we make in internal tooling also go into the product itself, which is really, a good way to be.

Bryan Cantrill:

Yes. And I think it gets to so folks in the chat are asking about Notion. I've not actually used Notion. Presley, it sounds like you have. The I I would say that my concern about Notion I would say I've got, like, an Evernote concern with Notion, or any other and this is part of the challenge here, is that, like, using any kind of paid service here is really problematic, because this is so important to us that we it's really hard for us to accept restrictions on how things are used.

Bryan Cantrill:

And we and in particular, like, a hard constraint on the problem is, we have to have a Git repository that has RFD the RFDs in it. Like, that has to be where they live canonically. It they cannot live canonically anywhere else. And so that's extremely important. The I think it's really important, the ASCII doc element of it now that I understand that only oxide is used in the ASCII doc.

Bryan Cantrill:

That becomes really important. Benjie, you mentioned in the chat, the versioning, that is extremely important. And I just think that this is one of these things where we really need to control every aspect of our fate. We would love to, I think, integrate with other, you know, we're not, I I I think, you know, if we could find kinda better ways to do pieces of this, but we really need to control our own fate. And I think that, like, that is that is not unusual for us.

Bryan Cantrill:

I think engineering organizations need to control their own fate in terms of the the actual artifacts they're creating. And this is a really important artifact that we're creating. This is, and it's a as you were saying, Adam, it's like one of our most enduring artifacts, our RFG. So I mean, it's I I mean, it's right up there with the source code, honestly. I mean, it's don't make me pick.

Bryan Cantrill:

Those are they're both very, very important.

Adam Leventhal:

A while ago, I stumbled onto a phrase of, owning your strategic weirdness. And I I would have said, like, oh, RFTs are a thing that we we could have outsourced. But, like, it's Woah. RFTs turned out to be our strategic weirdness. Like, we we we didn't even know what they were going to be.

Adam Leventhal:

Like, we didn't envision them as, like, a way to communicate with the public, for example, as you mentioned that we we did around,

Bryan Cantrill:

well so my I knew that was gonna be a problem. It was a problem at Joint too. I knew we were gonna wanna weigh communicating with the public. I just had much worse ideas of how it would be done.

Adam Leventhal:

Yeah. I mean but I just mean, like, if we had if we had used some off the shelf tool, it might have been great, but then we wouldn't we would have struggled to do some of these other things with it. And well, I I mean, the work that our colleagues who have just been talking about it, like, it's not a small amount of work. It'd also been hugely valuable, and I'm not sure we would have made some different decision even knowing how much work was gonna be involved.

Bryan Cantrill:

Yeah. I know. I don't think so either. In fact, I quite the contrary. I think with all these things, I now cannot imagine doing it's like it's been a lot of work.

Bryan Cantrill:

I mean, not to discount that, but it's also, like, we haven't we're not licensing any proprietary software here. We're not paying for service. In fact, quite the contrary, the work that Augustus was doing was necessary for us to prune our GitHub collaborators because that was costing us money. Like, this actually saved us a bunch of money by so the degree we were using a service, we were using too much of it, and we were I mean, the in fact, actually, Adam, I think you kinda need to look no further than GitHub where we were I mean, just for all the things that Augustus was mentioning about the groups and having that collaborators within the collaborate it just, like, it was really, we were already kind of breaking GitHub, and we've, you know, left GitHub as the kind of the canonical repository for comments, but I think the direction we'd want is to use less of that. Think you could easily envision I mean, god, if I know that that Crespo was defending Ben's time by saying, under no condition will we allow you to add comments on the RFP site.

Bryan Cantrill:

But if you were to add that, you could Transformative. You could see comments going into a Git repo and then us not using GitHub for comments on that.

Adam Leventhal:

Totally amazing.

Bryan Cantrill:

And and then I think that, like, I also think that the strength of us doing this, by the way, is that we've been able to open source it. So, this is now, like, we have generated something that excite and I feel that, like, you know, as we kind of gone through our careers, like, we've added more and more wisdom into the system, And it now reflects a bunch of wisdom that we can all carry forward, even folks that are outside of oxide can can, actually leverage this. One thing I did wanna talk about because it it did come up in the in discussion online, where people were curious about how we use RFPs to resolve conflict. And, that's not exactly their question. I'm kind of I'm rephrasing it a little bit.

Bryan Cantrill:

But, like, how do you deal with conflict in an RFP? How do you deal with where, you know, like, the in the the comment was kinda like, hey, we experimented with it, but it just became too much of a lightning rod. And, and I would say that, like, RFPs do not solve your organizational problems. Right? They don't necessarily solve they don't resolve those things.

Bryan Cantrill:

And, you know, we have had RFTs that become lightning rods, and, it's not good. I mean, it's not I mean, I think it's better than I would say these are lightning rods that are lightning rods anyway. I think as we alluded to on the cultural idiosyncrasies episode, of course, it's like the chat system, which we've already, like, run on the bus several times here. It's like chat is what risk will rip oxide apart. Absolutely.

Bryan Cantrill:

And and this is a good example where, like and I think, Adam, you mentioned this at the time we were talking about it, where where all of these systems have got drawbacks. It's easy for everyone to have an opinion on it. The opinions get, like, kinda tribal, and it's not great. Right? And the in terms of, like, it's not great in that, like, there's not an unequivocal answer.

Bryan Cantrill:

And, you know, the RFD on chat got a little hot. Yeah. Maybe even a lot hot. Maybe maybe I've never that's that's the

Adam Leventhal:

I'll tell you the nice thing about it, like, is you can also tune up for those ones that you know are gonna go nowhere, which is how I've viewed the one on chat. So I've yet to read that those hot comments, but

Bryan Cantrill:

now I'll go check them out. Please don't. Please don't. I I I it is not worth checking out the hot comments on on chat. But the and I thought I think that, like, you're on the one hand, you're you're not gonna resolve that.

Bryan Cantrill:

On the other hand, it is actually helpful to have all the I think it's, like, if you're going to have organizational conflict, let it be in writing. You know, it is very helpful to have everything because you can actually, like it it I think it forces people to be complete about their arguments. I mean, it's obviously, like,

Ben Leonard:

a lot

Adam Leventhal:

of people. Of the lawyers. I mean, it's it's it's great for deposition. It's great for discovery.

Bryan Cantrill:

Think about lawyers. Will somebody please think of the lawyers?

Adam Leventhal:

But, you know, earlier on, Chad had a question about, you know, are these really requests for discussion or are they just things that have already been discussed and decide and very much they are requests for discussion. And they're ideas that often have 1 or a couple of people have seen them, and then we're casting it open to other folks. And I think, Brian, one of the things you mentioned is often that discussion happens in the GitHub comments, but it doesn't always have to happen that way. Like, as things are getting spicy or as things are not resolving, you know, sometimes people have a meeting, sometimes people have it in a live chat. You know, there are other ways of, bringing discussion to determinations.

Adam Leventhal:

Yeah. So it's not and there's not, like, one prescriptive method for doing that or for navigating a conversation or even for knowing the threshold for when, okay, it's it's time to move on. We've published the thing. We just kinda feel it out and and do what feels right for each one.

Bryan Cantrill:

Yeah. And so I do have and I I actually made it, public, this morning or or yesterday on r t 113, which, I I did write to, which is kinda how we actually make determinations. And is I mean, is giving us kinda loose guidance that you're describing. I mean, this is not you you can't be overly prescriptive about how decisions are made kind of obviously. But, you know, really talking about and and kinda differentiating decisions from determinations, talking about some specific examples where, you know, how we made determinations in the past.

Bryan Cantrill:

Because it's like it's on the one hand, there are I would say there there are plenty of RFTs where it's like, no, no, I I wanna throw this out here. I really wanna get someone's discussion. In other cases, like, we've got and especially if you've got an issue where we have to move quickly, with often like, okay, there's been a lot of process that's outside of RFTs. And now actually, the RFT is gonna pull all this together and kinda describe what we've already done a little bit. But it's actually extremely helpful to go do that, where we can actually go through the different things that we've kind of considered, which is very important.

Bryan Cantrill:

So I I think that sometimes the RFP process ends up being a little bit behind the process, but I think oftentimes it's it's a it is. It's an earnest request for discussion. Yeah. The, there's a question about whether an RFP becomes immutable. We do have the published state, which is very valuable.

Bryan Cantrill:

I need to get more of my own things into publish. That RFP 113 is not the published state. Actually, it's funny. So just to to I'd say it's an interesting, like, object lesson. RFP 113 is not in the published state.

Bryan Cantrill:

It's in the discussion state even though it's obviously 113, and we are now at last updated, in December of 2020. It's, like, okay, why is this still in the discussion state? This should clearly be, like, published. And so I'm, like, I'm gonna go mark this as published state. But there's a comment in there, actually, our colleague, Akifasovsky, left a very thoughtful comment on there that I had read, you know, in 2020, but had I actually want to go there's some things in there.

Bryan Cantrill:

I actually wanna modify this RFP a little bit before I publish it based on that single comment. And it's not, like, contentious. It didn't doesn't disagree with anything here, but there are say he's got a perspective that I wanna capture in here a little bit before I moved it into the published state. So that's why that one is still in the discussion. I'm I'm but, yeah.

Bryan Cantrill:

I mean, in terms of, like, when, and I think, you know, Ilyana is also saying that let RFPs can language in discussion, which I think is, like, I think is okay. I think it's, like, I think it's it. I've certainly, I'm guilty of this too, but I, I think when we're moving into the published state, we're saying, like, okay, this is basically, like, we we've kind of accepted this. It doesn't mean that it's immutable. And I think importantly, we can go back and change RFTs that have been published.

Bryan Cantrill:

We don't wanna become captive to our own process in this regard. And I think I think by and large, we haven't been captive to the RFT. I don't think it's been incarcerating, Adam.

Adam Leventhal:

No. I agree. There I think there have been very few RFTs that really are trying to draw a very firm line that we then feel a decent tip. Like, I I think you're you're spot on there that that either we go back and we update it or we create a new RFP and say, you know, this this obsoletes this old RFP or our thinking on this has changed or the facts have changed in these ways, and that's what what changes these determinations. But no, I think I think it's been purely valuable in terms of determination process and then, hugely valuable for for looking back historically and understanding, you know, especially before we change some of these ideas, seeing that full discussion, really embracing that before we decide, to to go some other

Bryan Cantrill:

path. Yeah. And then I think it's also helpful to have the abandoned state. Abandoned it it is and a chat, Arfi, maybe in the abandoned state. The, just to indicate to someone, especially new to the company, like, oh, okay, like, I don't I I don't need to spend a lot of time looking at this one.

Bryan Cantrill:

This was an idea that, and I think, feel that was something, I mean, Robert, I feel that was something that where, that was kind of a needs improvement from the joint era as well, where we ended up with some RFPs that were not what we did. And then like, we need to kind of make clear like, no, this is not we didn't go this way. This was the the the the the thoughtful discussion on or what have you, but you shouldn't view this, as, as kind of canonical because we've abandoned it. I think the other thing I would say is that when we are getting better at is linking RFDs to one another. So you've already seen this and especially, like, in a you're, you know, referring to Dave's RFTs, last week, and, you know, 110 and 53 and kind of the RFTs that they refer to.

Bryan Cantrill:

I think that AsciiDoc makes it, pretty easy to link to those things in kind of footnotes and so on. But I think that being able to navigate that web becomes really important in another My

Adam Leventhal:

one of my one of my favorite efforts that Robert undertook was to build a whole, like, graphical representation of the interlock of about a couple dozen RFDs, it felt like. Oh, yeah. Kind of were orienting you in space. That it it felt like a tough thing to maintain, but, man, was that helpful?

Robert Mustacchi:

Yes. Specifically for the Gimlet, hardware sled.

Bryan Cantrill:

Yeah. It was very And it was kinda like you are here in this RFD, and this is how it relates to the other RFDs. And that was really, really important. And then, I mean, in our our next gen sled, Robert, you wanna talk about how how you because the RFD there we've got a a an RFD that really is the entire architecture of that kind of next gen sled. Do you wanna talk about how we're using RFDs for for Cosmo?

Robert Mustacchi:

Sure. So I mean, I think to kind of talk about how we use RFTs for Cosmo, it helps a little bit to talk about RFTs for Gimlet along the way.

Adam Leventhal:

So,

Robert Mustacchi:

when we were developing Gimlet, I'd say, you know, this was a time, when the first RFTs were written, there's maybe, what, 12 of us, 15 of us? And this is before most of the EE team has started. I mean, this is mid-twenty 20, maybe early. So a lot of it was us trying to separate out the specifics of, you know, what is it what's important for socket SP 3 or, you know, specifics for this particular AMD aspect versus what is it we generally want. And so there's a lot of a lot of things are split up in those early RFTs in part because of that.

Robert Mustacchi:

So, like, you know, we have something about, you know, how does hot plug you know, what does PCIe hot plug look like? And what are our goals and kinda constraints? You know? Right. How do we split up, you know, responsibility between, different subsystems?

Robert Mustacchi:

You know, just edit broad, you know, broad core strokes. What's even the role of power monitoring? When should we do it? When shouldn't we do it? All these things.

Robert Mustacchi:

And so that that kind of gives us a lot of initial foundation there. And then from there, we kind of with Cosmo, you know, a lot of these things that we have, we you know, we're not starting from a a clean slate per se. Like, we have an existing system. There's a lot of stuff we wanna reuse between that, you know. We have a large investment in Hubris and tooling, in different components.

Robert Mustacchi:

You know, we have vendors that we've worked with. So we're not trying to necessarily reinvent the whole wheel there. So, you know, not everything is, like not everything is starting from a clean shade again. So that that kinda helps constrain it. So in there, we kinda in the COSMO RFT, for example, you know, we've broken up that into a bunch of different areas that different folks kinda helped collaborate on.

Robert Mustacchi:

You know, some of it was, Adam and I collaborating on, you know, what is the disk drive interface that we should use? You know, there's U. 2, E3S, E1S, you know, there's E1L, there's 3 different variants of of of of widths and thicknesses of E3S, any longer things, how many lanes do you want? But, like, you know, that's in there. Where it's like the sorry.

Bryan Cantrill:

No. I just you're reminding me of, I think, is it 101, much to do about optics?

Robert Mustacchi:

There's one of those is is is up there. But, but that's that. Whereas, like, we have a very different question, you know, where, like, Nathaniel, who's kinda lurking, I think, on this, you know, was dealing with how do we kind of rethink about how we're doing parts of what the FPGA's role is as we wanna add different features around eSpy and other things. And so, you know, that that's different places where different folks kinda collaborate with, you know, with different, takes on it. And then, you know, we have an appendix, you know, to your point of things that you've abandoned.

Robert Mustacchi:

There's a bunch there's a bunch of early ideas that we kind of ended up not rolling with this time, but we have a note of what they were and then why we didn't because a lot of them weren't you know, they're still good ideas, but just didn't make sense based on certain circumstances. So it's it's helpful there. And I think then the other bit that we've tried to do here is that, you know, to to the folks' points, you know, throughout this is like, hey. How do I ramp up when there's, you know, a 1000000 words, 1,000,000 and a half words of context of this. And I don't know what it is.

Robert Mustacchi:

And it's like, okay. Well, there's you know, if you're just starting, you kind of have a determination section or even a small overview presentation that someone's given. And the specifics of why don't necessarily matter. But if you do but later on, if you come back to this, you do have specifics of why. And that's that's helpful for you don't that's not helpful when you're trying to ramp up.

Robert Mustacchi:

But if you're trying to come back to, like, why did we make this decision? What were the different trade offs? Would we do something different? At least you have some of the context of what were the different things that went into it.

Bryan Cantrill:

Oh, it is extraordinarily helpful. And it was it was, 112 as much to do about optics, Robert. I would we'll we'll need to look to see if there's any and and if there's no NDA but true in there, I wanna make maybe wanna make that one public just because that have you read 112, Adam? Because you're like, I don't know. Optics, how complicated can it be?

Bryan Cantrill:

It's like, oh okay.

Adam Leventhal:

No. I haven't.

Bryan Cantrill:

Oh my god. I mean, this is one of these, and I feel like there are there are many RFPs that are in this category where this thing will take you to absolute crush depth on on optics and how complicated it is. And I I mean, I and then I Robert, you've done this many, many, many times where it is so helpful to have the entirety of the depth of our thinking in an RFD. Because as you say, it's like, you know, you may look at that and then maybe, god, there have been so many RFDs that I read once. I'm like, okay.

Bryan Cantrill:

Yeah. That makes sense. I don't know. Makes sense. And then, like, later, due to, you know, change in circumstances, I'm now reading this very closely.

Bryan Cantrill:

Maybe it's because I'm implementing, or maybe it's because we're about to have a call with, you know, a with a with a partner where this part is misbehaving, or maybe we're having a call with an investor or having a call with a I with a user. When I was like, we're having a conversation where, like, I need to ramp up on everything we know about optics. I need to know in the next, like, 45 minutes. It's like good news. And you can go actually get to the entire depth of our thinking.

Bryan Cantrill:

And I think that this is where the the written artifacts are so important for scaling an organization, because it allows I mean, Robert, with so much of your thinking written down, people can go read your thinking before they they need to, like, ask you a follow-up question. Right? And that is just extraordinarily important. So it's been I mean, you know, you're talking about Adam, how this has become more valuable over time. It has become more valuable because we are able to use them more.

Bryan Cantrill:

We have more people using them to Ben's your your point about making this accessible. We've got more people creating RFTs. We've got more people reading RFTs in terms of the public customers and so on. We and then we we've got this kind of fine grained permissions, so we can get the right RFP to the right person. It's been incredibly important for us.

Bryan Cantrill:

It is the backbone of Oxnard.

Adam Leventhal:

Absolutely.

Bryan Cantrill:

This is, it's been great. Yeah. Not just the discussion, but, boy, this work. All of you have done such terrific work on this stuff, and it's been, really, really essential. Well, thank you.

Bryan Cantrill:

It's all I gotta say. Adam, thank you. Ben, David, Augustus, Robert, thank you for your the the the 300,000 plus report.

Adam Leventhal:

100,000. Wild.

Bryan Cantrill:

Yeah. I'd expect an out loud reading. Adam, thank you for all the collaboration you've done with all of your colleagues on our FDs.

Adam Leventhal:

Oh, much less work than Robert's words.

Bryan Cantrill:

Really great stuff. And again, if you're looking at this thinking, like, boy, I would love to adopt that system, please adopt it before we try to make it so it is adoptable. So, you know, take it lock, stock, and barrel, take the parts that work, and report back. We'd love to hear about it, because I think this is something that a lot of engineering organizations really, really need. On that, thanks, everybody.

Bryan Cantrill:

Thanks, Europe. We'll we'll have to, you know, I actually Adam, I feel like we we'll need to do a European timed episode at a time when you're not in Europe just so we can prove that we're like, oh, yeah. They're remote friendly because the CEO no longer wants to work. You know, he's you know, he doesn't wanna move his family. That's why they're remote friendly.

Bryan Cantrill:

So we'll need to, we'll need to do some European timed episodes. We gotta do something like once a quarter or something like that. Sure. That's the spirit. There we go.

Bryan Cantrill:

Once a quarter, Europe.

Adam Leventhal:

Even you, Germany, and we're sorry again.

Bryan Cantrill:

Exactly. Who could ask for anything more? Alright. Thanks, everybody.