Charity Majors joined Bryan, Adam, and the Oxide Friends to talk about the idea of "innovation tokens"--a fixed budget for, so called, "innovative" projects. When is boring better and when is innovation the safer approach? Is Oxide issuing innovation tokens in some sort of hyper-inflationary cycle!?
In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest Charity Majors.
In addition to Bryan Cantrill and Adam Leventhal, we were joined by special guest Charity Majors.
Some of the topics we hit on, in the order that we hit them:
- Glyph: Against Innovation Tokens
- Charity's Twitter Thread
- OxF: Let That Sink In! (Whither Twitter?) with Charity
- Druid
- Scuba whitepaper
- Oxide RFD 68: Partnership as Shared Values
- Good Strategy Bad Strategy by Richard Rumelt
- Dropshot and Progenitor
- OxF: The Pragmatism of Hubris
- OxF: Helios
If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!
Creators & Guests
What is Oxide and Friends?
Oxide hosts a weekly Discord show where we discuss a wide range of topics: computer history, startups, Oxide hardware bringup, and other topics du jour. These are the recordings in podcast form.
Join us live (usually Mondays at 5pm PT) https://discord.gg/gcQxNHAKCB
Subscribe to our calendar: https://calendar.google.com/calendar/ical/c_318925f4185aa71c4524d0d6127f31058c9e21f29f017d48a0fca6f564969cd0%40group.calendar.google.com/public/basic.ics
Brian.
Bryan Cantrill:Brian is here, and
Adam Leventhal:I see Charity.
Bryan Cantrill:Charity's here.
Adam Leventhal:Come on down.
Charity Majors:I've been invited to speak. Hello.
Bryan Cantrill:Hey, Charity. How are you?
Charity Majors:Hi. I'm doing really good. That was a great blog post. I just finished reading it.
Bryan Cantrill:The the the blog post yeah. And, Adam, I realized that I did you the disservice of not giving you the warning that I gave Charity about did you see this blog post about innovation tokens? Kind of revisiting in the
Adam Leventhal:Like, 5 days ago or something. Right?
Bryan Cantrill:From 5 days ago. Yes. Against innovation tokens. Okay. You saw oh, look at that.
Adam Leventhal:I did. I look at the mentions on the socials.
Bryan Cantrill:That's great. Because I had this kind of fear that I because Charlie, I wasn't sure if you'd seen it or not. And it's a it's a long piece. It's kind of involved. And, so I wanted to give you the the heads up on it.
Bryan Cantrill:But, yeah, I thought it was I thought it was interesting. Well, Jody, thank you very much for for joining us. So Yeah. It it not It might
Charity Majors:be long since we chatted chatted.
Bryan Cantrill:It has been. And, not for the first time, you are a guest on Oxide Friends. Although, the last time, Adam, was a a rather unconventional red phone, Oxide at Friends. This was the last The red phone that I didn't pick up, but Charity did. That's right.
Bryan Cantrill:The the Oxide and Friends Charity. This is after the the the when Elon had descended upon Twitter.
Charity Majors:Oh, yeah.
Bryan Cantrill:We did our Twitter space, and, it's definitely fun to go back and listen to that now. I I think
Charity Majors:3 hours long?
Bryan Cantrill:It was long. At some point, you're like, hey, I've gotta, like, go do other things. Like, I actually sorry. I didn't go out, like, all night for this, so I'm I'm gonna leave. But it was at at 1 point, we were like, well, they are certainly not gonna release all of the supervillains on the Twitter.
Bryan Cantrill:What I did not have on the bingo card is like, actually, they would welcome the supervillains on the Twitter, and some of the supervillains would decide not to come.
Charity Majors:So I know. Right?
Bryan Cantrill:And and I also feel like you do not predict the sheer quantity of porn currently on Twitch.
Charity Majors:Oh my god. Yeah. And I hear that they are taking away the like button and the retweet or something and if you're gonna, like, left swipe or right swipe. Come odd. Got it.
Charity Majors:Oh my god.
Bryan Cantrill:Okay. The other thing is as long sorry. As long as we're here, this, like, the the show possible spam replies thing. Have you seen, like Yeah. When you click on the replies, like, these are all legitimate supplies or replies we're talking about.
Charity Majors:How would they say they can't find the actual spam, but they're like, oh, we don't like these people. It's probably spam.
Bryan Cantrill:Right. These are just like people have chosen not to pay you $8 a month. That's not yeah. I mean, they're as good as spam if they're not paying. Right?
Bryan Cantrill:Apparently. It but the so, Charity, we we got here, speaking of of the Twitter monster. We do not call it x around here, obviously, or, we refuse to to to honor them on that, but the, you had this, this tweet thread, a couple weeks ago, maybe 2 weeks ago, describing kind of the the the history of the the data engine that you all had developed. This I mean, scuba inspired data engine. Is that a fair description?
Bryan Cantrill:And do you wanna, like just because we not to make you do an out loud reading of it of the the tweet thread, certainly.
Charity Majors:Right.
Bryan Cantrill:But but could you just get describe, what you were describing in that thread in terms of the history of its development and some of the intersections that you find yourself. I also love, I have to say I mean, I always love it when people reflect back on things that they did.
Charity Majors:And Yeah. This
Bryan Cantrill:it's it's a very kind of reflective thread in terms of, like, here are the the the strengths of having done this and, you know, kinda looking back in hindsight on this decision. So
Charity Majors:Yeah. Totally. Yeah. It was funny because we accidentally did the right thing. Like, I I you know, coming from very op opulent, I've spent my entire career telling telling people never write a database.
Charity Majors:Never write a database. Never ever ever write a database. And so we really like, this is 2016, and Christine and I wanted to do this very much, like, we're gonna use something if we cannot use anything. And we tried everything out there. The nearest that we came to using something was, this storage thing called Druid.
Charity Majors:But we looked at it and we're like, it's written in Java. We would have to rewrite a third of it just to get the kind of, like, flexible ad hoc. Like, 1 of the things that we love about this is, like, there's no about the thing we built is that there's no schema. If you just wanna start dropping in a new key value pair, do it. And if you stop dropping it in, it will just vanish.
Charity Majors:It'll age out eventually. And that's really, really valuable. Like, having a managed schema in your telemetry is nightmarish. But the solution support it, like, most data most data data data don't support that kind of thing. And we were like, well, crap.
Charity Majors:You know? If we're gonna be rewriting that much of it, you know, we'd rather work in Go, like, anyway. And so we decided to write our own. Also worth noting so there's the Skuva white paper. We had fallen in love with Skuva.
Charity Majors:Like, the the experience of using Skuva was at Facebook was you know, Parse was this massive platform over over a 1000000 mobile apps. And when we were working on it over a decade now, like, we were using microservices before there were the term for microservices. We're doing all this, like, massive, like, parallel stuff with it was just, like, it was something different, something new would break every day because a different app hit the top 10 in iTunes and whole thing would go down because we were using Ruby on Rails. And the experience of trying to figure out from scratch each time what was happening, basically not possible when all you have is metrics and logs because you have to know what you're looking for before you decide what you're looking for. And when we started to bring some datasets into scuba, you know, CUDA butt ugly, like, aggressively hostile to users.
Charity Majors:But it lets you do 1 thing really well, which is slice and dice in near real time on any kind of data, high cardinality, whatever. Yeah. Miracle. And so, like, all of a sudden, figuring out what was, you know, each thing from scratch, like, the time it took us to do that, it dropped like a rock. Like, from hours, days to, like, seconds.
Charity Majors:Like, it wasn't even engineering problem anymore. It was like a support policy just like, ah, something's happening. Click, click, click, click, click. All the trail of breakdown. Ah, this is what's happening.
Charity Majors:Like, it was just, like, mind blowing. So Yeah. Exactly. We so we we we got this Google white paper, and it's really jake. Like, to this day, the way that they replicate is they shell out from c plus plus to do our sync from machine to machine.
Bryan Cantrill:Oh. 0.
Charity Majors:And we're just like
Bryan Cantrill:Oh goodness.
Charity Majors:I know. Right? It's like it's crossed between, like, it isn't it's just reddish quality, like or memcat. It's processed to, like, memcat in the database. So we're like, doesn't look too hard.
Charity Majors:Right? And it has this, like, kind of interesting, really distributed architecture. And we're like, oh, this is cool. This looks like it'll work. But but it was like, it wasn't like writing a database, you know.
Charity Majors:So so we started and and we for the 1st few years, we only had 1 person working on it most of the time. And And
Bryan Cantrill:this is out of Honeycomb. Right? So this is because you're so you want just to give your so your at Parse. Parse is acquired by Facebook. Right?
Bryan Cantrill:You've got you've got
Charity Majors:Right.
Bryan Cantrill:A I mean, I think I was first exposed to your writing. You had a great kinda reflective piece on that whole journey and Yeah. And going from start up to position. You just did not start up. And so but but okay.
Bryan Cantrill:So the I actually kind of and it's kind of an interesting counterweight to the the the trauma of an acquisition and that you were exposed to this technology scuba that you probably Yes. Wouldn't have been exposed to otherwise.
Charity Majors:Yeah. So it's this really interesting sort of, like, germination process of of being a start up using, like, common open source, you know, tools and stuff. But then suddenly having access to the whole Facebook suite of development tools. And it was Yeah. Really fruitful.
Bryan Cantrill:And so you and Scuba hits you, obviously, the right time, the right place, because, you know, this is amazing. You Amazing. Manage to I mean, and I I mean, Adam and I definitely I feel you on the, like, taking a problem that used to take, you know, hours or or weeks Yeah. Or indeterminate and turning in. And I've been because we obviously when you kinda when folks would have that same moment with Etrace, it just it just changes your relationship to a technology when you're like, this is no longer, like, of abstract importance to me.
Bryan Cantrill:Like, I am now like, I'm a I'm a belligerent in making this technology happen
Charity Majors:because this is so a great way of putting it. It changes your relationship with technology. It it's like getting religion. You're just like, oh, there is no way I'm ever going back to building without this. You know?
Charity Majors:Yeah. Because when you have that ability to slice and dice and, you know, interactively zoom in and zoom out. It's like but, I mean, I I think I think a lot about the parallels between, like, when TDD emerged, you know, in early 2000 to to, like and and TDD, I I feel like, you know, we're on the verge of of doing something observability related for, like, ODD. But, like, before TDD, it's like, you wrote your code, you deployed it, and, you know, your users would complain and that's how you found bugs. Right?
Charity Majors:And I feel like
Bryan Cantrill:worst thing TDD, to be clear. Test driven
Charity Majors:Test driven development. And and I feel like TDD is when programming started to turn into software engineering. Right? When when suddenly it's like, you can't it's it's like you're holding yourself to a standard and you're you can tell whether or not your output works or or is regretting or whatever in a way that you really couldn't before that. And but at the time, you know, in the TDD years, like, your architecture was dead simple and almost all of your complexity was bound up in your line of code.
Charity Majors:And I feel like TDD has actually become sort of less and less less than this is a guarantee that your code actually does what it ought to. And you you can't you cannot have any confidence your code is gonna do what it's supposed to do until you are looking at it in production through the lens of your own instrumentation. And I would argue that if all you have is, like, 1 0 observability tools, metrics, logs, and traces. Right? You they're distributed across multiple sources of food.
Charity Majors:You have aggregate, percentiles, averages, and get random exemplars. I don't think you can really do the same sort of, like, analysis of, okay, I put this flag on and off. What happened before and what happened after? Got this bill ID versus that bill ID. What are the differences?
Charity Majors:I think we need those wide raw rows to do sort of
Bryan Cantrill:Yeah.
Charity Majors:Reading time clearing and analysis. And I feel like if if we're successful, like, I think over the next year you're gonna see a bunch of open source and sort of, like, competitors, like, based on ClickHouse, you know, various wise. But, like, it is so transformational to the engineers who so that I I really think of it as, like, there's so much potential for observability 2.0 stuff to open up these waves of transformation, you know. It's almost like it's necessary to kind of complete the DevOps cycle. Now I was thinking about virtualization and how virtualization is what unlocked the the the path to DevOps.
Charity Majors:Right? Because you went from you've got ops that handles, you know, servers and hardware and operating systems and and dependencies and, you know, devs are in their app land. Once you had virtualization, it opened the door to cloud computing and managing your your infrastructure using code. And I feel like, optimistically, we're kind of in the sunset years of DevOps now. We're increasingly those silos don't exist.
Charity Majors:Every engineer writes code. Every engineer is expected to own it and understand it in the market. It's kind of awesome, But I feel like there's a bit of a missing link here when it comes to really widespread adoption of observability tooling. Anyway, bit of a tangent there.
Bryan Cantrill:No. That's I mean, a bunch of got it. A bunch of interesting stuff in there. Certainly agree with you about the the role of virtualization. I think the TDD point is really interesting point.
Bryan Cantrill:I hadn't really kind of viewed it through that lens. But, I mean, Adam, just speaking for our own careers, that's kind of that is the era in which we did divide we didn't call it TDD, but it was development that was driven by our testing. I don't know what the Yeah.
Adam Leventhal:I mean, the that that sort of, testing rigor and integration of testing with the development process, I think, has been the most significant change in my career. I think about, like, how things started, like, at my career in the Slayers Curric group at Sun, to the way we write code now. It I I think that's the most significant difference.
Bryan Cantrill:So a hot take. I it how much of that is fueled by the rise of open source, which is happening at exactly the same time? And in order a really good question. Because in order there's not a QA group in an open source project. And and the the idea of this kind of separate QA or, Charity, you remember and, Adam, I'm sure you remember, like, the Microsoft idea of, like, your like, the separation of powers and, like, your doppelganger who does the QA for your code.
Charity Majors:Oh my god.
Bryan Cantrill:Remember this? Like, am I making this up?
Adam Leventhal:No. No. 100%. No. I was a intern at Microsoft.
Adam Leventhal:I was an SDE, and my buddy and roommate was an SDE in test. And we were told over and over again how we were equally loved.
Bryan Cantrill:Equally loved. Separate separate but equal. It's like separate okay. Is that do you mean a minute? Am I supposed am I supposed to take a part of history on that 1?
Bryan Cantrill:Or, it feels but the
Charity Majors:Yeah. No. You're right.
Bryan Cantrill:That kind of idea went away. Right? And we had this idea of, like, everybody's got responsibility for testing their own code. And it feels like that happens. Certainly, it's concomitant with open source.
Bryan Cantrill:I don't know if it it it how related they are. But, the
Charity Majors:the idea of about that, but I think you're right. I think that, like, it never like, in retrospect I mean, hindsight is always 2020. But in retrospect, it was always kind of absurd for us to think that half after the engineers write the code and the other half would understand the code. Right? I if there's more than anything, it's about feedback loop.
Charity Majors:Right? Feedback loop have to be have to be tight, and you have to have the the response you can have specialization, you can have consultant, you know, all the stuff, but, like, the responsibility for quality to be co located in the brain of the person who is responsible for building.
Bryan Cantrill:Totally. Totally. And I and Simon Wilson's also making the point in the chat that this is how we got to componentization. And certainly and the I mean, Simon, you're mentioning this in the early 2000. It's definitely true in the nineties where all about componentization.
Bryan Cantrill:And it was gonna be the componentization was gonna be via object oriented programming. And it's like, componentization, good idea. Object oriented programming, not the way it's gonna happen. It really did happen via open source. But okay.
Bryan Cantrill:So we get the in terms of your need so when 1 is developing this, a broad broad system in production, Yeah. And I certainly I mean, God, Adam, I definitely agree with your assertion that, like, you don't actually understand something until it's running in production. That's the time when you really need to understand something, and you get so much emergent behavior. You but the kind of data that you want to store in query doesn't really fit well with a traditional database. So I'm I'm curious how you found you you go on this search post Facebook, or we want the thing that we kinda had there in terms of scuba.
Bryan Cantrill:But you and you find Druid. What and but it's not a fit. How if you want me asking you, how did you find because I've always been like, what is the rubric? How do you find technology? Exactly.
Bryan Cantrill:Yeah. Totally. Right? I mean, I feel like, this is where I do do, use chat gpt for this. This is a, 1 of the things that that Simon, I mean, I used try gpt for a lot of things, but 1 of the tricks that Simon do you do you use Simon's trick of tell me 10 more, Adam?
Bryan Cantrill:Yeah.
Adam Leventhal:Yeah. For sure. No. It's been true.
Bryan Cantrill:It's is big on, like, tell me 10 more. And if you ask it for, like, I want technologies that solve this problem, and you the first 10 will be, like, okay. I recognize most of these. But you're like, tell me 10 more. Tell me 10 more.
Bryan Cantrill:Tell me 10 more. And you do that, you know, you start to get into some weird stuff. But, so you're you're finding this thing by just going around, talking to other technologists, what have you.
Charity Majors:Yeah. You know, in the years since then, you know, I've been a few years with CEO and I learned of the, existence about things like Magic Quadrant and analysts and everything, which I have literally never used. Kinda funny. Yeah. No.
Charity Majors:We just well, I knew enough to know that we needed a column and store. Right?
Bryan Cantrill:Yeah. Because you didn't wanna have to deal with
Charity Majors:fucking indexes. You needed everything to be, like, fast and efficient and everything. And your options to call in the store at that time, extremely limited.
Bryan Cantrill:Yes. Very very very limited and in fact, Adam, Adam, do you remember the the database that that I built database? Big error. The, for for analytics at Fishworks. And
Adam Leventhal:Yes. It was, like, very file based.
Bryan Cantrill:Oh, god. II0, yeah. Wanted to ask this question because thing again? What did I call it? III am I am the stubbornly incentivized for you to forget this because this was not me.
Bryan Cantrill:And this was I didn't realize that I wanted a column or I want, like, a a column or database or a time series database kind of before I just and they definitely didn't exist. And what I wrote was not my I'm sorry for the people who
Adam Leventhal:Didn't didn't cause it to exist. That's right.
Bryan Cantrill:Yes. Because I didn't talk to Buzz to exist. That's right.
Adam Leventhal:You know, Brian, that that code could easily still be in production.
Bryan Cantrill:It could be in production. Yeah. It definitely could be in production. And it I mean, it's solved its very narrow problem, like, decently well, but it I just thought the whole time I'm like, this is not this needs to be thought of as kind of a a first class thing. And it's going to be its own separate software.
Bryan Cantrill:And so but you're coming up to this charity now. It's that was in 2, 000 and, like, 6, 2007. You're coming up to this problem, in 2, 000 and you said 16?
Charity Majors:2016. Yeah.
Bryan Cantrill:2016. And and discovering, like, there's not a lot of stuff out here. This is No. No. And then so you go into Druid.
Bryan Cantrill:What was your process? When you saw Druid, did you go through that phase of, like, I've only been on the website, so I'm very optimistic that it solves all my problems. I haven't actually tried 100%.
Charity Majors:Yeah. I mean, every website promises you the world. Everything to me. Fast and efficient and magical and silly.
Adam Leventhal:This thing is saying all the right words. It's got all the right phrases.
Charity Majors:Right. 1 of the other things you realized with Druid was that the overhead of management was just nightmares. I only have to say 1 thing to help you understand, and that is that it used zookeeper.
Bryan Cantrill:Uh-oh. III was just gonna say whatever you're gonna say will apply the same thing for z k, but you actually know what? It's gonna go straight there on what? Sorry. It's old yeah.
Charity Majors:We were like Zookeeper. Zookeeper. Zookeeper. Zookeeper. Zookeeper.
Charity Majors:Zookeeper. I know. He's everywhere for, like, a decade there, and then all of a sudden, he was able to stop. Yeah. But, like, part of this is, like, is kind of relevant to that article we're just talking about because, we realized that the simpler option was, for once in in my life, writing something ourselves in scratch.
Charity Majors:It's also I should point out, I would 100% expected that our company to fail. So this was not like, I was I did not actually have in mind that I was building something for the ages. I'm just like, well, probably gonna fail, but, like, it'll be a fun way to spend a year or 2. We've got a couple $1, 000, 000. Let's just Right.
Charity Majors:Stick around the software.
Bryan Cantrill:Oh my god. That oh, that is such the better disposition to have them. I mean, you're so much more likely to do great work when you're like, no. I was a 100% gonna fail. This is
Adam Leventhal:It's like the VC is asking, why do you keep referring to this as house money?
Charity Majors:I got it. Like, oh,
Bryan Cantrill:did I say that? I thought you know, I thought I was thinking that. Was I saying that? I'm really sorry. Let it roll.
Charity Majors:So So real. So real.
Bryan Cantrill:And and did you try to so you come across Druid, and the website looks great. And Yeah. But you you you wait into it and you're, like, you discovered the z k thing. It sounds like pretty early.
Charity Majors:Yeah.
Bryan Cantrill:And you're like Yeah.
Charity Majors:I'm like, tried using it a bit. And, like, you know, it honestly like, you know, we started with a very small amount of thought. Just like, wow. Really missed scuba. Nothing else.
Charity Majors:Can't imagine going back. I'm just, like, so much less powerful as an engineer. But it really took us, like if you if you look at the blog post we published over like, we we we're running so much on gut instinct. We're just like, you know, we're expecting it to take more data types, like, lots of schemas and, like, it's still right. And it was over the entire 1st year that we started to really work out what matters.
Charity Majors:Like, all the stuff that we we wrote about, like, the the problems with pre aggregation and why you really want, you know, green time aggregation, not right time aggregation. And and we talked about, you know, the wide event stuff and how important it is that it'd be arbitrarily wide. If you don't know, you don't know how important was it not these schemas because anytime you have to predefine upfront what data is gonna be valuable, you're making a promise that your future self will think of both it. Right? Like, it's just, like, all this stuff, you know, we it it felt like fucking just, like, sweating blood trying to figure all this stuff out and articulate what mattered and what didn't.
Charity Majors:And and in retrospect, it looks so easy and obvious. But, like, there was really just, like,
Bryan Cantrill:mhmm. Is this the phase when you're, like, we are writing our own thing? So, like, now we have made a decision to write our own thing.
Charity Majors:We already decided to write our own thing. Yeah. And and so we were like, okay, we need to call and restore. But we don't need to delete, we don't need to update, we just need to do a pen, you know, we need to run our
Bryan Cantrill:Super important. Use. Right? I think when you are kind of doing your own thing, I think that part of the real power of going your own way is that you get to actually have constraints on the problem that other people might not have. And you've got so or eliminate constraints that other people can't eliminate.
Bryan Cantrill:And you've got some big ones you can eliminate here.
Charity Majors:Huge. All of the things and then another 1 that was huge and important was the fact that I think we're the only data storage thing in the world where you can say that fast and close to right is better than perfect. Right. You know? Right.
Charity Majors:We do just span out to to our you know? So our 95th percentile query latency is under a second. It is really fucking fast. And the only way you can do that is by panning out, scanning a ton of columns, merging them, and then returning to the user. And if, you know, 1 of the 1 of the buckets is, like, not returning in a couple of seconds, we just discard it.
Charity Majors:You could get away with that in your in your and we discard it. We make a note of it, you know, not returning blah blah blah or something. But, like, you can't typically get away with that in the database.
Bryan Cantrill:And is that because, like, most the the problems that you're seeking to and the questions you're seeking to answer are effectively latency related. Like, why why are these things taking a long
Charity Majors:It it it won't because we want people to be in a sort of interactive state of flow. A power and a trail of bread crumbs. Very different from the whole I've got my dashboard. I created it 2 years ago. You know?
Charity Majors:Oh. It's like, you know what you don't know. So you're like, what the fuck? What about this? What about this?
Charity Majors:And you're following this trail, you know, so it needs to feel interactive. You can't just, like, have to issue a query and then go go for copy. Right? Like, that's the big problem with, like, lots of log things. It's like, you gotta know what you're looking for in order to find it.
Charity Majors:Yeah. Or it's really fucking slow.
Bryan Cantrill:Yeah. I mean, obvious I mean, Adam, obvious resonance with us in terms of Detroit's that same that that same loop and that ad hoc loop of, like, I don't know what I'm looking for.
Charity Majors:Yeah.
Bryan Cantrill:So I it's really important that the the the faster you can return and answer to me, the the the faster I can know that, like, actually, there's nothing here. That's fine. That's dead end, and I'm down on the other path. But, yeah, that's really interesting.
Charity Majors:Yeah. It's, like, most of the callbacks that we we have, you know, like, this is still obviously divorced in the lineage of the last 40 years of, like, metrics based software or logs based software. And most of the historical analogs that we look to are tools like DTrace. So things where you're you're trying to understand your code in a very interactive manner, not the monitoring sort of, like, mindset.
Bryan Cantrill:Yeah. And I think that they are very different mindsets. And it is I mean, you know, I've always said that the observability is is the capacity to answer questions about our system. And the which is, I mean, sounds great, but you actually the the hard part is formulating the question and this and if you can't get quick answers, you can't formulate follow-up questions quickly. And so much of this is like it's actually not it's it's not gonna be 1 question.
Bryan Cantrill:It's gonna be a whole chain of questions and answers that are gonna get you to what yeah. To to to what's 1 of the
Charity Majors:other things we've baked into the I feel like it's it's entertaining. It's just like it's like you have your back history, but a visual 1. Right? Because still often you're, like, following the plot and then you blocked it. And you're, like, oh, where did I last have it?
Charity Majors:And being able to just kind of, like, go back immediately to, like, okay. It was, like, 2 or 3 clicks ago. So, like, there's the graph. I'm gonna, like, pick it up from there, is is really powerful.
Bryan Cantrill:And so because you're doing your own thing, you've got total control over what choices you make and what trade offs you make there, and you can make that very that that very clear trade off. I mean so not to tip my hand, but the, in terms of because I think we the part of, you know, and I think it was someone else who pointed out that, 1 of the Tailscale folks is like, oh, you know, we talk about this kind of innovation tokens thing, which we I innovation tokens, from the the McKinley blog entry on on choosing boring technology. And I don't think it was meant to be, like, overly reductive. This is not actually, like, you you you do not need to, this is more like kind of thinking about the overhead of innovation. But clearly when you are down this path on a company, it's like, well, good news.
Bryan Cantrill:Like the company's I'm playing with house money here, so I'm actually, you know and I I don't have that kind of anxiety. I mean, you're not thinking, like, boy, do we spend our our kind of innovation budget here or not? Because it's so clear that, like, this is the problem that we need to solve, and we don't have a different way of doing it.
Charity Majors:Yeah. And it's so clear that, like, this is the roaring engine of our success and failure. Like, it's just Yeah. You know, it's like it's not like a graphics library or something. It's it's that we knew that what we needed to build was, like, discontinuous from everything else out there.
Charity Majors:And so, like, that's really where, like, there's there's there's some stuff that would like, a couple years in, Johnny Coleman's actually when we were like, oh, hey. You know, these events could be the crisis too. We could just, like, add some relationships here and and, like, visualize these events over time like a trace. And being, like, being able so, like, there was some yeah. There were just so much that that's there and the ability to, like, kind of own the whole product experience from top to bottom is is just really powerful.
Bryan Cantrill:Okay. So this gets into 1 I mean, Adam, I'm sure you you keyed in on this as well. Your interaction with investors that you described in this Oh, good. And you so there are I I mean, there are a couple in here that are just remarkable. 1, if you were going to succeed, you would have already.
Bryan Cantrill:I love that. Oh, yeah. It's like, oh, this is a very uplifting conversation.
Charity Majors:Less than a year in.
Bryan Cantrill:Yeah. Oh my god. Oh my god. But the the I think it was really interesting because there was this point of view saying that you should have found product market fit first Yeah. And then written the custom storage engine.
Bryan Cantrill:And, it at the time, did you disagree with that? I mean, what was your what was your take on that? I mean, first of all, it's just it's just remarkable arrogance to kind of to to understand what I gotta tell you.
Charity Majors:No. You know, honestly, I feel like that that's the conventional wisdom and I don't argue with it. Like, you're right. Like, 80, 90 percent of the time, that's probably the right thing to do. Get as far as you can with MySQL, and then, like, once you have you know, once you have scale problems or once blah blah blah, then, you know, then invest in all this custom stuff.
Charity Majors:But the problem is that if we had used something off the shelf, we never would have found product market fit because we would have looked and felt just like everything else. And so I think it was just a case of, like, didn't really understand what we were doing or why or infrastructure in general. Yeah.
Adam Leventhal:Infrastructure in general yeah. I mean, just infrastructure in general takes, I don't think that's necessarily a trigger. I think that absolutely. Like, you you can't look for product market fit in the 1st 6 or even 18 months. If there's so much critical stuff you need to build to even find out if it's there.
Charity Majors:Well, I only
Bryan Cantrill:had this, like look. You because you also have to assume success at some level. And Yeah. The like, look. This company is not gonna fail because I spent some time making this framework, that I think we're gonna reuse a lot.
Bryan Cantrill:I mean, companies fail for a lot of reasons. I mean, not impossible, but it's it would be very unlikely. I mean, because, Adam, obviously, reading this, you know, there are lots of, like, oxide analogs. Lots of oxide analogs Yes. In Charity's experience.
Bryan Cantrill:But 1 and I mean, we clearly talk about hubris, not hubris with a capital h charity. So I've heard it's and we talk about lowercase h hubris too. But actually, 1 that came to mind for me was drop shot, Adam. I don't know if you had the same kind of drop shot vibes from
Adam Leventhal:Yeah. I mean, a little bit on drop shot, and then I think I've come to appreciate Progenitor, our, open API, like, SDK generator along these lines a little bit more as well, in terms of
Bryan Cantrill:describe a little bit, like, what progenitor is and kind of why. They they just for for You
Adam Leventhal:know, I think drop shot and progenitor kinda go hand in glove. So, drop shot is our, Rust, HTTP server framework. And say, why did you do that? Well, we looked at a bunch of other Rust server frameworks, and we didn't like them. And they didn't and we didn't like the way that that they were gonna force into us into certain patterns.
Adam Leventhal:And, also, we wanted something that really had open API at the center of it, so we built our own HTTP framework. And and it was like it was and is pretty simple, and it's not it's not for serving every kind of API you could possibly imagine. It's for serving the kinds of ones that we wanted to go build. Charity, there was something you said about not needing to kinda deal with everybody else's wish list or everybody else's customers that that really spoke to me. And that, yes, you are kinda taking on the burden of doing it all yourself, but you're also shedding the burden of needing to combat everybody else's priorities.
Charity Majors:Yeah.
Adam Leventhal:And and then we did that with the sort of optimistic hope that if we admitted OpenAPI documents, we could get SDK generation for free because OpenAPI is a popular thing, we thought. But that was totally false, at least in terms of us be able to get that for free. We built our own, and then there's all kind and that is Progenitor. And there's a bunch of other stuff that we got out of owning that layer of software, like generating our own SDK, generating a test framework for our API. So that that's that kind of thing has play paid surprising dividends.
Adam Leventhal:And I'm sure Charity, that has been the case for, you know, for this stuff as well.
Charity Majors:Yeah. Absolutely. Yeah. Well, the has a lot of cost.
Bryan Cantrill:When and so because you made this point too that, like, part of the reason that, yes, there are now more options available today than there were when you looked at 2016. And, yeah, you might make a different decision today, but actually owning this thing yourself has similar to kind of the drop shot. In general, a general case, like, you I'm now I've got new ways of using this thing that I probably wouldn't have developed otherwise.
Charity Majors:Yeah. Absolutely. Yeah.
Adam Leventhal:If it was somebody else's project and you're trying to beg your way into a PR that really only pertains to you or you're trying to get it to work on the side, yeah, it opens up all kinds of avenues for your own innovation.
Charity Majors:Yeah. And and, like, it's I think I said this in the thread or maybe the different thread, but, like, we want if DuckDuty or QuickHouse has been available, we 1 100% would have used them. But I am so grateful that they didn't and we didn't because, this is very this is a place where my philosophy would have put it in very wrong direction. Our our ability like, for example, a couple years into Honeycomb, we started to realize that, like, the the cost of running all this stuff on SSC all the time, when very little of it gets queried, especially then, it was just, like, unsustainable. Like, we were paying Amazon more than our customers were paying us.
Charity Majors:And so we actually refactored the whole thing. We kind of server list our database, actually. We like, so so when it when when when when events come in through the API, they get dropped into with you. And then each Kafka node has a pair of retrievers that read off of it. So we've got applications, but, you know, not primary and secondary.
Charity Majors:And and they get stored in SSDs briefly, like, maybe for an hour, and then they all get they all get aged out to s 3 segments. And then our query planner, actually, for the most part, runs in a series of Lambda jobs, but does does this massive span out and merge to all of the SSCs and and, and s 3 segments, and then, you know, merges it and returns it to to the user. And this caught this this cut the cost by an order of magnitude or more. And we were really nervous that the latency would, you know, rise, but in fact, it was basically about the same, just certain things are slower and certain things are faster. Interesting.
Bryan Cantrill:And I mean, this is not a change that you would have made had you been No.
Charity Majors:No. No.
Bryan Cantrill:Right. No.
Charity Majors:It it would have been imposs like, It it's gonna be hard to do this in a cost effective way using QuickTowel especially for any kind of multi ton of things. I think it'll be a viable option for, you know, companies that kinda want to run put it on Elk Stack analog, but presumably 2 0 stuff. I think it'll do pretty well for that sort of thing, but, like, I can't imagine running a large multi tenant thing especially in a cost effective way off of QuickDown. And and just the number of of little things that we've been able to do, you know, just like optimize, you know, for compression or or little little things to make, you know, to to add, like, you know, nice automatic sampling and just, like, so many things under the hood that we've been able
Bryan Cantrill:to do just sort of effortlessly. Yeah. That's interesting. So there's a there's a a question in the chat, Ernie. It was kinda funny where, it was asked how much of this is subject to survivor bias?
Bryan Cantrill:And Adam and I answered it completely differently at the same time. So as you say, a fair bit of this is is survivor bias. I say
Adam Leventhal:Probably a fair bit. Yeah.
Bryan Cantrill:But, but Yeah.
Adam Leventhal:So why why do you think not very much?
Bryan Cantrill:I think not very much because I think so often so, you you know, 1 of the things that kind of occurred to me is is it it happened so frequently that the performance problem in a system would not be the part that people had thought a lot about. Yeah. The part that they hadn't thought at all about. That there's almost like a tautology there where it's like, actually, the part that you've thought a lot about, you've thought a lot about. So because you've thought a lot about it, like, you've actually and, I mean, the number of times that I came in, like, we've just been back on this again and again.
Bryan Cantrill:We can't figure out what the performance problem. And, like, well, can we go actually, like, look at the system in production as opposed to, like, the, like, the your your the the way you kinda think of it on the whiteboard. And then you'll get this in production. It's, like, there's, like, URL script that's trying to fork an exec 7, 000 times. It's back end trying to run our sake or whatever.
Bryan Cantrill:Like, that's actually and so I I think that the and so the reason I think, like, I think when you and just like charities approach, like, we tried Druid. We got, you know, we were look it's not like we were, like, we are gonna write our own thing, and we don't care what else is out there. You come to that conclusion because you've looked at other things that are out there.
Charity Majors:Yeah.
Bryan Cantrill:Yeah. I think it would be unusual if you kinda came to that conclusion first, certainly for us with hubris, like, it was the last. We were doing it. That's why we call it Hubris because it was like we're making a joke about ourselves. And Charity, the the debugger for Hubris is is called the humility.
Bryan Cantrill:Of course. The the the irony being, Adam, I the the and maybe this is, you know, I'll just I'll just insult myself if left alone. There is there's much more hubris and humility and much more humility and hubris. It's kind of a grand irony. But the We we
Adam Leventhal:we did hubris after a a lengthy kick of the tires on talk. Do you wanna talk about that at all?
Bryan Cantrill:More than a kick of the tires. Yeah. We were really Yeah.
Adam Leventhal:Yeah. We went pretty deep.
Bryan Cantrill:We went pretty deep and really wanted to make so we I just had this idea of and I think this is the part of, like, the innovation tokens thinking that is important, which is you don't, like, know the things where you're trying to differentiate. And if there's something out there that does solve, like, exactly your problem, you should use it. Yeah. And, you know, you shouldn't do something new for its own sake. And, indeed, if you're trying to do something that is it rip broadly is innovative, then you really need to find those things that are where you can you can kind of use an existing technology, so you're not trying to run the table.
Bryan Cantrill:I mean, Adam, I always think of the of Skunk Works, Lockheed's Skunk Works, and Ben Rich's book, which obviously had a huge Charity, I don't know if you've read this book, but really terrific book on the history of Lockheed Skunk Works. And they when they were developing the u 2, remember they took the avionics from Yeah. In, like, I think the airframe
Adam Leventhal:The flight control. Yeah. They would they would take pieces from all these other things and build this Frankenstein's monster of components. But then on the part that they really wanted to innovate, that's what they would focus on.
Bryan Cantrill:And, Adam, I remember and I don't think I I don't think I'm just retconning ourselves here. I just remember I mean, that book had such an influence on us when we started Fishworks. I remember us going back to that thinking over and over again through Fishworks. Oh, absolutely. As we were thinking build versus buy.
Bryan Cantrill:Like, do we is this where we wanna differentiate, or do we actually wanna just take the thing that's, so I I do and we we definitely had the the predisposition of, like, no. We don't wanna do do our own operating system. No. Of course not. We've already already already trying to do so much, and we're really trying to get it to work.
Bryan Cantrill:And Yeah. The thing that I came to the conclusion of, you know, in a couple of different kind of dimensions simultaneously is the values that this thing had for itself did not match the values that we had. And that and actually, Charlie, my kind of rubric for where we kind of integrate versus build our own is, like, looking for that values match when we're looking at a technology. Yeah. And
Charity Majors:Oh, I love your blog post on on the, the partners versus,
Bryan Cantrill:the Yeah. Right. Yeah. So you're referred to r p 68. Yeah.
Bryan Cantrill:Yeah. Yeah. On on partners versus vendors.
Charity Majors:Yeah.
Bryan Cantrill:And I'm just a big believer in looking for shared values or understanding values. Because, also, I think, like, values mismatch doesn't mean that there's, like, a right versus wrong. It just means that your in your example, for example, I I care about I want it to be fast, and it's okay if it's a little lossy. Well, that's not gonna be the decision that another database would make.
Charity Majors:Correct. Yeah.
Bryan Cantrill:And neither 1 of those is, like it's not that 1 of those absolutely right and absolutely wrong. It's just that 1 of those fits your
Charity Majors:It's your operating system.
Bryan Cantrill:It is
Charity Majors:what are the defaults you're gonna follow? And they're not predictable, you can be, like, in internally, like, the less friction and uncertainty there is because you're just letting you're you're letting people know how you work or how you aspire to work so they can decide whether or not that's who they want to be and how they want to work. And Yes. And it means that people can make decisions autonomously instead of always having, like, check-in with coordinators or managers or something. You know, and I I keep the values as, like, very skeptical of it because just like just to mention the company values and this I would I would because you know, and and it's really been a process over the past several years at Honeycomb of realizing, oh, no.
Charity Majors:They can and often are done very, in the words.
Bryan Cantrill:I I think the importance I think they've gotta be honest. If the values are honest, they are very
Charity Majors:And self aware.
Bryan Cantrill:And self aware. If the if the values are aspirational or if there are things like, you know, be excellent. You're like, I'm not
Charity Majors:be excellent. Yeah. Okay. Thanks. You just eat table stakes.
Charity Majors:Right? And but then there are some things where not, like people do choose different things and they are valid and that's where they become helpful and and valued. As long as, as you say, as long as you're honest and and self aware.
Bryan Cantrill:Yeah. And I think that we you know, 1 of the things that this is why it's always helpful when you are engaging, especially in the open source world and so many things we build on our open source. It is great to, like, have an issue with something and see how that issue is kinda handled by the community. And if that, you know, if that issue is, like, ignored, that kinda tells you 1 thing. If it's shot down, that definitely tells you 1 thing.
Bryan Cantrill:If it's embraced, that tells you another thing. And that signal, I think, can be really important. And, you know, 1 of the things that we just realized is, like, our values and talks values just don't align any non in ways that make them wrong. But, you know, 1 of the things that they cared a lot of, it's a teaching operating system, and so they care a lot about the ability to load programs into this thing. And it's like, we are trying to make a secure operating system.
Bryan Cantrill:We don't wanna load programs at all. We want this thing to be totally, like, statically defined. And it was just gonna be that was gonna be it was just gonna be constant friction. And Yeah. You know, I Charity, I kinda came to this in terms of, like, values as a rubric, not the easy way.
Bryan Cantrill:I would have loved to have come to the easy way. This is after so I because I was a joint, when we did we had this big Node. Js Divarce. And I was so we were, you know, the proponents of Node. Js or the company behind Node.
Bryan Cantrill:Js, and then the whole thing went sideways. And years after it went sideways, the organizer of the Node Summit asked me to give a keynote on Node. And I'm like, that's a terrible idea because I, like, you broke up, man. I don't have a nice do you want me to stay? Like, we broke up.
Bryan Cantrill:Like, I the the how is this gonna be a good he's like, oh, no. That'll be great. That's a great keynote. I'm like, you're warped. But the as I it actually was super it would be interesting if we did this in our personal lives, if you kinda came back to, you know, every relationship that ended and having a good impression 3 years later on what what went wrong, because you're you're just drained of the, like, emotion of the moment.
Bryan Cantrill:Right? You can be
Charity Majors:like, okay.
Bryan Cantrill:What did go wrong? Like, we I thought we did have something, and it, like and yet it ended with, like, throwing pots and pans. So How did we where did we go wrong? And the thing I realized in code is, like, where we went wrong was kind of at the outset where the values that Node. Js had for itself, and the values that we wanted it to have were not the same thing.
Bryan Cantrill:And we were it was just and in particular, Node's values were JavaScript's values, and those values were all around developer growth, making Node more accessible to more developers, and not really focusing as much on what it meant to operate a thing. And we were very interested in what it means to operate a thing, and the the the the cost that you have for the what what does the operability mean? Obviously, you know, Charity, you're obviously cut from the same cloth in that room. Yeah. And so our big disagreements with the node community were all around, we wanna add something to the language or to node that will make it easier for people to use.
Bryan Cantrill:And our perspective was, no. But this makes it, like, way easier to have an issue in production that is totally undebuggable. And we just weren't loggerheads. And it's like, we're not gonna get and so now I'm just very I I just that values it's just really important to me to to to lead with that. And when you begin to get indicators of, like, oh, we what I want is slightly different than what you want.
Bryan Cantrill:I want, you know, super fast queries that may be a little bit lossy and but we're going back and forth in this issue and you're fighting me. And I'm like, okay. You're fighting me because oh, you know, what have you. If you've gotten the Druid path or what have you.
Charity Majors:Yeah. Fascinating. You know, I've often thought that, like, the reason back end, front end is a divide is because those teams and organizations often have such different esteemed values.
Bryan Cantrill:Yeah. Interesting. Okay. So elaborate on that. I totally agree.
Charity Majors:No. It's just like what you're saying. It's like the back end is always like, okay. How are we gonna how is this gonna how are we gonna maintain this over the next 3 years? And what's the performance like?
Charity Majors:And what's the, you know, blah blah blah. How do we instrument the front end's like, you know, what's it gonna look like, and how is the how how are people gonna interact with it, and how are we going how How is this gonna tie into, like, the business stuff and everything? And it's just like they're just very often just speaking, like, right past each other.
Bryan Cantrill:Totally. And and you yeah. It is where you can kinda really
Charity Majors:get And neither is wrong.
Bryan Cantrill:Neither is wrong. Yeah. Right. Neither is wrong. I mean, and and both are kinda try which is I mean, those are always, like, in many ways, the the most emotionally gutting arguments are where no one's wrong because it's like you can't just retreat to, like, well, this person's being an asshole.
Bryan Cantrill:It's like, well, more complicated than that because you're Yeah. You're talking past 1 another. So did did you, have you, you know, as you kind of integrated other technologies, I mean, so it sounds like you are also building this on other things. I mean, you you're building this building retriever on things like Kafka, which,
Charity Majors:No. They're completely separate. I mean, I do think of Kafka as being part of our database, but, like, they're Okay. Yeah. They don't touch except, like, retriever consumes Kafka on disk.
Bryan Cantrill:Okay. So we're so Kafka is totally outside of retriever. Kafka is just
Charity Majors:feeding this 1. Got it.
Bryan Cantrill:Yeah. Interesting.
Charity Majors:Because because the most important SLO that we have is how quickly and reliably can we ingest people's data. And so we just wanna drop it into Kafka as fast as possible. That's another value that I think we don't really think about much because it we just take speed for granted. But, like, god, I remember using, you know, metrics dashboards and and seeing that, like, the data is coming in, like, minutes behind when it's happening. And, like, with Honeycomb, 1 of the things I love most is that as soon if you hit a curl request in 1 window and you switch to Honeycomb window, it's there.
Charity Majors:Like, it's just it feels instantaneous, and that makes such a difference when you're trying to understand something that's happening right now.
Bryan Cantrill:Yeah. That's really interesting. And I I mean, it it definitely reminds, like, in 3 Mile Island. Right? And when they were dealing with that disaster, part of the problem is their alerts were just, like, hours behind.
Charity Majors:Yeah. And
Bryan Cantrill:Exactly. Makes it very hard to reason about the system. And is that I mean, the the low latency that you have, it that comes from, I mean, your own experience, presumably deploying. So I mean, the it's obviously extremely important, and you would choose that over other things if you had to. If you had to make it personal, you had to.
Bryan Cantrill:Yeah.
Charity Majors:Yeah. 1 of 1 of the big choices that we, like, whenever you're dealing with aggregation, you're gonna have this sort of rolling windows that's never gonna be updated and available instantly. But a trade off that you have to make then is sometime if people don't wanna pay to store everything, then you have to sample. Which again isn't better or worse, but it's just something you have to, like, learn to adjust, you know, your mental model of what's happening.
Bryan Cantrill:And so is did you give any thought to open sourcing, Retriever?
Charity Majors:We have, actually. And we we have decided not to thus far for a few reasons. You know, obviously, we need to figure out how to make a business case for it. I I have been a bit as much as I feel like I'm an open source kid at heart, I, I'm also pretty wary of open source in in some ways.
Bryan Cantrill:Why? Does anything ever go wrong? I'm sorry. So is there anything,
Charity Majors:I've heard rumors occasionally.
Bryan Cantrill:Oh, okay.
Charity Majors:But but the other thing is that it's it's non trivial to operate. Like, it's it's very complex. It was not built to be like, drop in, get it up and going, you know, the first 10 minutes experience, you know. It's very complex, and there are a lot of good reasons for that complexity. And again, that's a tax that, like, isn't that much isn't that heavy enough because we have, you know, really good engineers who care about automating and and and, you know, all these things.
Charity Majors:And they only have to learn it once, you know, when they come and but, like, for people to use off the shelf, oh my god. The amount of support we would have to do just
Bryan Cantrill:to, like, help people figure out how to run it. It's absurd. Yeah. Interesting. I mean, and, of course, you can open it without I mean, we maybe I just take the lazy approach by just to open things without, like, sorry.
Bryan Cantrill:We're not I mean, if you can use this, that's great. But this is actually not just my freaking us through. I mean, the Right.
Adam Leventhal:About half of our repos are open just because it's easier for us to build them in CI and GitHub that way.
Bryan Cantrill:Yeah. It it really is. It is really hard to be half open and half closed. I was like It's
Adam Leventhal:a pain in the neck.
Bryan Cantrill:It's a giant pain in the neck. I mean, like, we you you so that I think is more more than anything. But and so in terms of what what when people I mean, so here the the the for whatever it's worth. I mean, I think the thing it might be interesting about it. Certainly, the reason, like, I because hubris is kind of in the same boat of, like, I'm not really expecting anyone else to use Hubris.
Bryan Cantrill:Yeah. The read me more or less tells you, like, hey. That's not what we're doing with this thing right now. We try to be just very clear in the contributing document. Like Mhmm.
Bryan Cantrill:You're welcome to contribute PRs, but, know that, like, we are very guided. We're being very explicit about what what's guiding us. I do think it's helpful, and it has been helpful for us to have folks be able to self select in to aspects of the company, either as a future employee or as a customer by just getting to know that layer of technology, move up to source everything, but that that that's been I Clearly, you're not
Charity Majors:And as a and as a customer, I love it when things are open source even if they're not quarter because then I can read the source and find out what's going on.
Bryan Cantrill:Right. Right. And and then also, like, you can also understand it's very helpful to understand, like, what the the values are of the thing. Right? So you can understand, like, am I trying to push this thing out of its design center?
Bryan Cantrill:And is this thing gonna push back? Totally. So are so has this ex so this experience where, hey, I have the company that I didn't think was gonna succeed.
Charity Majors:Yeah. Oh,
Bryan Cantrill:actually, on that because I I Adam, I worry I do this a little bit too much where I talk about, hey. Hey. We might actually might pull this off at Oxide. They'll be like, what do you mean we can pull this off? Like, sorry.
Bryan Cantrill:I I got a mortgage. And I
Adam Leventhal:I can't hear the premise that we were gonna pull
Bryan Cantrill:it off. You told me. I'm like, no. But we might pull it. I that's good news.
Bryan Cantrill:That's much better news than Yeah.
Adam Leventhal:That's, like, yes. But why do you sound so surprised?
Bryan Cantrill:Yeah. Right. I mean, Cherry, did you go through a phase where you're, like, actually kinda surprised, like, this might work.
Charity Majors:Oh, god. Yeah. Like, every year from 2016 to 2020, I was just, like, well, we've had a great run, but this is definitely the year that's all gonna fall apart. And it's only been, like, since 2020, I've been like, oh, shit. Like, it will now have to be several years before we can totally fail because we have money in the bank.
Charity Majors:And and it's really like, I try not to open my mouth too much about this because nobody really wants to hear it. But, like, I I feel like it's just a psychologically healthy way to to live my life. Like, I I love I love good news, and I hate being disappointed. So I just try not to get too attached to it.
Bryan Cantrill:The the low expectations are the key to happiness. Absolutely. Best. Oh, god. Get it.
Bryan Cantrill:Yeah. Like, instead of trying to make your life happier, try lowering your expectations. Definitely.
Charity Majors:Yes. Exactly. Same vibe.
Bryan Cantrill:I mean, at least experiment with it. You know? Put as much into it anyway. So okay. So you you're kinda transitioning to, like, oh my god.
Bryan Cantrill:We might make it. But, the Yeah.
Charity Majors:And and because of that, I will note, we went from 1 guy, building it to we now have a storage team of of 6 people, 5 software engineers and and 1 SRE type. So we we're like, shit. I guess more than 1 people person should probably know how to Right.
Bryan Cantrill:I was gonna ask. Yeah. Yeah.
Charity Majors:So yeah.
Bryan Cantrill:And then as you've gotten bigger, has that changed I mean, you how do you think about it when someone wants to do something new from a whole cloth? I mean, do you this is kind of the, the the this kind of idea of innovation tokens or, you know, being somewhat realizing that you've got kind of a finite budget for the novel and kind of do you kind of think about that formally?
Charity Majors:I don't know if I would call it formally. What I love about our approach is that we we mostly have product teams. Right? We don't have, like, a backup team and a front end team. We mostly have teams that build products together.
Charity Majors:And the storage team is a fairly recent, it emerged in the last year or 2. It worked out 3 people and now I think I have 6. But but well, there's only 1 or 2 people who really spend almost all their time in receiver. There's a lot of crossover. Like, we just we just ship this thing called relational field that lets you, like, query stand kind of based on the relationship to each other within a trace.
Charity Majors:And and this came from within the storage team who then kind of, like, worked on the product in a bottom up way and then eventually, like, moved in some designers and stuff. And then there are others that start
Bryan Cantrill:Oh, that's great.
Charity Majors:Start from a very, you know, top top of the stack down way where they're they're like, would this be possible in receiver? And so they, like, get together and tunnel with 1 or 2 of the engineers, and they start, like, prototyping. So there's a lot of this sort of, like, you know, crossover and, like, innovation in both directions. So which I I love because honestly, some of the coolest little little features come from the storage folks. We're just like, what if what if we turn this into a product and and it's really neat?
Bryan Cantrill:That's interesting. And then in terms of so really taking a problem focused approach. Like, what is the problem we're trying to solve? Which
Charity Majors:Yes. Yeah.
Bryan Cantrill:It it mean I and I think younger software engineers be forgiven to be like, is there another way that people do it? I
Charity Majors:know. Right?
Bryan Cantrill:Oh, 0, my sweet number child. I'm so sorry. But, yes. Yes. There's another way that people do it.
Charity Majors:Yeah. Yeah.
Bryan Cantrill:Where where the people are focusing on solutions first and problem. Maybe not at all. Maybe second. Maybe.
Charity Majors:Exactly. Yeah. Which, like, I I've gotta say, you know, there's oh, I think I said this in my EMD. There's that whole, like, don't compare your insides to other people's outsides. Like, it took us a while to get here.
Charity Majors:The first couple years, our quote unquote product planning process was me on a whiteboard every quarter with the other engineers kinda hover around. They're like, what would be cool to build this quarter but now I feel like we have we have a company strategy. By the way, anyone out there who hasn't read Good Strategy, Bad Strategy by Richard Rommel, life changing.
Bryan Cantrill:Oh, good book. I read that on your recommendation actually. I I read that. You recommended that a couple years ago and I yeah. It it was good.
Bryan Cantrill:I really enjoyed it. Yeah.
Charity Majors:So good. We have a product strategy and and we have these little hack weeks now and then where engineers can just kinda go hog wild, really really bullshit, which and then we're like, oh my god. I want all these things. But, yeah, we we try to be really disciplined now about, like, approaching it from a problem standpoint, and it's really good. It's really helpful.
Bryan Cantrill:Yeah. That's I'd that's really interesting. And it then so in terms I you know, part of the the way this threat came to my attention is, someone's saying, well, I mean, oxide is, like, clearly, they don't have any innovation token budget. Clearly, it's like
Adam Leventhal:Like, it's a federal federal reserve bank of innovation.
Bryan Cantrill:Like, yeah. We're just, like, printing more innovation tokens over here. We're just, like, hyperinflation for innovation tokens.
Charity Majors:Yeah. Yeah.
Bryan Cantrill:I mean, Oxa and there and there have been times when I There's a
Charity Majors:lot of innovation, like, the more new things you're trying to do, the harder it is and the more important it is to be focused. Because it's it's it's hard to focus. And the more things that you're trying to focus on, the harder it becomes because you're putting your focus, but you're having to focus on all of them. And it's just like, there's a very high degree of difficulty here. And also, the more innovation tokens you are you are trying to say, the more money you better have fucking replaced.
Bryan Cantrill:Yes. Yes. And that is so I think we so we have spent, a decent number of innovation tokens, but we have, I mean, a couple things. 1, we we knew that it was gonna be big out of the chute. So you're just, like, everything is bigger.
Bryan Cantrill:Or you're raising more money out of the chute, you know, your milestones are gonna take longer to hit. I mean, this everything is. And then also, I think that we are we're not there are times I don't remember if you've had this where you're kind of, like, explaining everything we've done, and you do have to sometimes be like, look. We're not actually trying to do everything. Like, we're not trying to write everything from scratch.
Bryan Cantrill:Okay? We are actually, the and, like, I mean, for example, 1 of the things I don't think we've actually even we haven't even talked about it all here is we so we, so Charity got rack rack based machine, RackSco Computer. We've got a power shelf that we that we bought from Muradis, our kind of our partner there, to your kinda RFP 68 partners versus vendors, very much a partner. And but we did our own PowerShell controller for that thing. And I don't know if you were in any of those discussions, but, the and the engineer that was advocating this, Keith Soskie, was is is pretty conservative about about, he would not innovate for its own sake.
Bryan Cantrill:Right? I mean, he is definitely, like, the the like, we are and I'm really sorry. Like, we're gonna do our own PowerShell tutorials. Like, is that wise? Like, we're doing our own this is another board, yet another board that we'd be doing.
Bryan Cantrill:But it's but Keith was totally right. It's actually like a very it it a relatively simple board. I should say very simple board or Ian Sobre, the engineer who designed it. It's gonna be like, oh, it's oh, it's really simple?
Adam Leventhal:Okay. You can decide. You do it.
Bryan Cantrill:I'll write it. Yeah. You do it. I've got to know it's very simple. Not really simple, but it is, it it it is tractable.
Bryan Cantrill:And I think the key thing is and, Charity, I would love your take on this, and I'm here as well. The the reason that was tenable is because we've got certain components that we've used throughout the system. And
Charity Majors:Oh, yeah.
Bryan Cantrill:So that that uses the same service processor, the same root of trust, that uses the same network. So we we are able to get to and I I'm I'm I'm sure it was the same for you with with Retriever. We've got these other elements that you are beginning to reuse inside of of Honeycomb. Is that I am
Charity Majors:absolutely. Absolutely. Like, the more you the familiarity like, cognitive cognitive mode is is, like, 1 of the strongest limits that you have. And the more that you can use things that you're familiar with, that you know how they're gonna sell, you know how they're gonna scale, everybody knows how to use them, you can ask anybody for help on them. It's like a freebie.
Bryan Cantrill:Yeah. Well and that's I think, in
Adam Leventhal:a lot of ways, I think it would have been more costly to not do our own PowerShell controller because it would have been some other thing that was different that, you know, didn't follow these principles that these tools couldn't apply and so forth.
Bryan Cantrill:And and we're hopeful
Charity Majors:that guy's trying to say in this blog post, right, which I I kind of I kind of don't really understand some of his like, the way he's, like he said that people have been, you know, interpreting innovation tokens. Like, I never interpreted it that way, and I don't think other people did either. But it's a great straw man. Right? When you're, like it's it's a very straw.
Charity Majors:But but like boundary tokens. Right? Where he's talking about the consistency of your architecture.
Bryan Cantrill:Well, it's it's such a straw man that he realizes, like, okay. I'm actually I know. I guess things that Dan McKinley didn't then he say, but what if he did say it? What if he did? If he if he had said it, then this would be a good good
Charity Majors:good yes. I will. Yeah. But it's like, you know, I'm sure you're right, dude. I'm sure that there are people out there who have interpreted this in many stupid ways, like, no doubt.
Bryan Cantrill:Well, and so, actually, I did like 1 of his concrete examples where I was kind of, triggered Kron? Half of a colleague, Kron. Yes.
Charity Majors:Yeah.
Bryan Cantrill:Because Kron and and have you ever talked to Dave Pacheco about our adventures with Kron? That sounds like
Adam Leventhal:a great 1.
Bryan Cantrill:Kron. Oh, Kron. Kron has had Kron has damaged humanity a lot. I mean, Charity, am I wrong? I just feel that, like Yeah.
Bryan Cantrill:Kron is 1 of those things if you're like, oh, like, I need to run something at a certain time. Like, I know. There's something that does this. It's Kron. I mean,
Adam Leventhal:you mean the thing that was, like, built to automate, you know, changing over a male spool? Yeah. Right. Right. That's about that.
Adam Leventhal:1972. Right?
Charity Majors:Yes. Yeah.
Bryan Cantrill:Yeah. And that's exactly what I want. I wanna change my old school in 1972. I also don't care if it does it like it may do it 0, once, or many times. I actually don't
Charity Majors:care. Yeah.
Bryan Cantrill:I feel that, like, and Dave's gotta get away of pricing this up. Like, what Kron does is so simple that it ends up exporting all that complexity to anyone using Kron.
Charity Majors:Yeah. Yeah.
Bryan Cantrill:And it's like, the system that uses Kron, you're gonna have so much complexity that you're gonna wonder why the hell am I using Kron. Like, get Kron out of it. Kron is actually, like, do you I mean, Sherry, is that it sounds like you've had some you you got some scar tissue on Kron from Kron.
Charity Majors:Oh my god. Nobody knows how to compose a correct Kron line without peering without peering intently at the documentation. Remember how many stars are supposed to be or what the syntax is for, like, every, yeah, every iteration. Like yeah. Like, that's a great example of something very strong though because yeah.
Charity Majors:Like, the carrying the the carrying capacity of your head to remember things, to model things. Like, I think the reusability of just sort of, like, common building blocks is is so important. The thing about, like, you know yeah. If you're if you're running a Ruby on Rails world, don't just in import the entire Google world just to run a function even if it's quote, unquote better. Like, I think all of that is there there are a lot of different ways to interpret, innovation token, I guess.
Charity Majors:But I think that ultimately, we're talking about frame limits, human limits. Not really talking about the technology parts at all except in so far as they sort of, like, become those.
Bryan Cantrill:Yeah. We yeah. But I was just gonna say that kind of, like, the the the limits of the cognition different language or, you know, it's whatever. I'm just like, I don't understand this part of it. And therefore, like, we're gonna have a virgin baby.
Charity Majors:Attraction is like not having to understand things. Being able to just take it for granted.
Bryan Cantrill:Yes. Right.
Charity Majors:You know what it does, don't need to think too hard about it. That's what lets us scale and move up the stack.
Bryan Cantrill:Absolutely. On to your earlier point about virtualization being an early and very important example of that abstraction that, like, most people really do not need to worry about that abstraction. That abstraction is that that the the virtual machine is pretty well established. You don't need it. We need to think about it, but you don't need to think about it.
Bryan Cantrill:That's right. Of which it's a very successful abstraction, and you don't need to really worry about that's definitely, like, not using an innovation token when you're using virtualization. Yeah. Exactly. Well, it so how Kind of
Charity Majors:the opposite. You'd be using an innovation token if you decided to compile your own kernel.
Bryan Cantrill:Yeah. Yes. Yes. And which again is not, like, I not to be overly productive, but it's
Charity Majors:not new. It's not innovative, but it would, like, cost you.
Bryan Cantrill:Right. Well, that's so that's kind of interesting because I feel that, you know, as you know, I think we talked about this when because we do have our own I mean, we do we got many of our operating systems. We got we got Humorous. We also have our own host operating system. And, you know, which we talked about in terms of Helios.
Bryan Cantrill:And part of the I mean, the the rationale was was many fold for that. And the I I've kind of come to accept, Adam, that, like, familiarity is not a pejorative with respect to that. Like, because I think you you would you were talking about familiarity as a strength there, and I always viewed it as something as Oh, yeah.
Adam Leventhal:I I think you yeah. I I think I I would talk about it incessantly, but you did a great job of ignoring me. But I'm glad you've come around.
Bryan Cantrill:I did. Did I do a great job of ignoring you? It doesn't sound like it. It sounds like I could've done a slightly better job ignoring you. III feel like I I think there's room for improvement in
Adam Leventhal:my expectations. Yeah.
Bryan Cantrill:That's right. But the you know, I so I I think that we but part of the the value for that that I didn't appreciate at the time was when when you are using someone else's technology, you're using someone else's decisions. And it's and so there's I and I said, I do feel that, like, the boring this is a bit that the boring technology doesn't piece doesn't get into that I kinda think the values rubric is a little clearer for. Yeah. It's 1 thing if it's like, alright.
Bryan Cantrill:Cron is not moving, basically. So it is cron is what it is. But there are other things that, like, are moving, and they may move in a way that is not consistent at all. Python 2 may become Python 3, and that may or may not be consistent with where you wanna go. Or they they mean these things make big changes, and they're not always gonna be welcome.
Bryan Cantrill:And you may, you know, and and in the case of, like, 1 of the things that was so, you know, if we had we're using Linux as our host operating system, we would be having our own we'd have to have our own distro, or we would be which is a huge undertaking. That is, like, talk about your innovation tokens. That's like innovate. I mean, you're not, like, you're spending it on toil, not on things that are cheap.
Adam Leventhal:Well, in particular, I mean, with different for a team that had already done that 6 times, you know, than for a team that was unfamiliar with that.
Bryan Cantrill:Yeah. That's true. Although, I got thrown this morning even if you've done it 6 times. I mean, it's like you're trying to I think it's hitting it this pro of, like, you gotta tell which which GDB to use. Which it but it's it's the amount of times that charity said TDD, I was convinced she was saying GDB, and I was so fucking confused.
Bryan Cantrill:I cannot tell you. You're both. Like, we're talking about GDB. And you're like, GDP is, you know, been been critical for first the the, you know, the arrival of software engineering. I'm like, oh my god.
Bryan Cantrill:But it's Yeah.
Adam Leventhal:Like, how do I guess I am that 1? Right?
Bryan Cantrill:How do I guess I
Charity Majors:am that? So I love what you were saying about about it being other people's decisions. I think that's a really clarifying lens because there are a whole lot of technical technical, areas where I would really love to use someone else. Particularly. Somebody who knows it who knows it well.
Charity Majors:Yeah. Somebody who's, like, decided what's best and everything. And then there is my crown jewel. There's our differentiator. The decisions that we need to make will make us different from our competitors and hopefully the best in the world at something.
Charity Majors:And those are the decisions that my team and I need to understand intimately. Yeah. That's what you yeah. Leeway and the latitude, to control them.
Bryan Cantrill:I the totally, Charity. I love you. It's like, you're just like, look. You can hook up to this sewage lateral if you want, but you understand that someone else is making decisions about how that sewage is treated. And you're like, yes.
Bryan Cantrill:Great. Thank you. That's awesome. Oh my god. That's great.
Bryan Cantrill:Someone else is like, I don't have I I do not wanna think about how the sewage is treated. I really don't. And versus other aspects of it. Like, oh, no. No.
Bryan Cantrill:I actually don't feel or I've got really mixed feelings about someone else making decisions for me. Like, I I need to know more details and that that's kind of an interest I hadn't really thought about that as kind of a a more extensive rubric of should you be doing your own thing. Because if you are if you are like, no. I really can't have anyone else making this decision for me. Yeah.
Bryan Cantrill:So you probably should be doing your thing, and it's not the wrong decision. Yeah. How it almost certainly is not the
Charity Majors:Yeah. I think about this a lot in terms of, like, what is infrastructure. Like, infrastructure is software that you are running because you have to in order to run the stuff that you want to. Right? And when it comes to implementation and understanding it, like, you you don't like, you it moves a different pace.
Charity Majors:You do not want to be deploying your infrastructure today. You want to be deploying it at the at the rate of, like, Linux upgrades or whatever, which is not often. And then there's the code that, you know, is your code that is not infrastructure. There's the code that makes you money that you want to be changing constantly, many times a day, and you want to and you want to you need to understand it at a much more intimate level.
Bryan Cantrill:Yeah. Totally. I love that description of infrastructure. Charity, I gotta tell you that that is a great because it is like, if you don't if that is not your view of infrastructure, then you're not consuming infrastructure. You're making it.
Bryan Cantrill:You're 1 of these, like, you're okay. You're fine. You're you're 1 of the very strange group of nerds like us. Like like the collective us that are actually making this stuff, but the the the there there's almost something kinda logical about that. It's like if the infrastructure is the stuff that you view is, like, necessary to deploy the thing you try.
Charity Majors:Yeah. And some companies are infrastructure companies, but even infrastructure companies have infrastructure and then they have their code.
Bryan Cantrill:Yeah. Totally. And what I think is also becoming I mean, I I have you I mean, I I okay. I'm sorry. We're gonna ring the bell here at what however many minutes in at an hour and 10 minutes.
Bryan Cantrill:The AI infrastructure thing, have you followed? The the, because there is there's a lot there there are a lot of new kind of infrastructure problems, and a lot of companies I think they're AI companies that I think are just infrastructure companies. Yeah.
Charity Majors:Yeah.
Bryan Cantrill:I'm not sure how much you've been forced to attack into that by your how much you've been I know it's is minicom.ai not registered to you? That would be very impressive.
Charity Majors:Hilariously, I got a Twitter DM from somebody a couple months ago asking if I wanted to buy it. But
Bryan Cantrill:Oh, did you? Oh, with the what was the, how much the what what was the price? This is I wanna know the same percentage of account.
Charity Majors:Like, I I was like, we could afford about this much, and they're they just stopped responding.
Bryan Cantrill:Yeah. My counter is, like, $1. Also, like, by the way, like, I am Honeycomb. So whatever other I mean, there's not gonna be I'm your only buyer. So I know I know you feel that, like, you can that that you you're can kinda course me.
Charity Majors:We can
Bryan Cantrill:kinda we we are gonna have to,
Charity Majors:stand off.
Bryan Cantrill:Right. Yeah. Domain names are are generally not. The the owner of lockside.com keeps lowering their price, and it's still Oh,
Charity Majors:nice. Nice.
Bryan Cantrill:A little bit. Yeah. It is kinda nice. It's also like I'd be like, no. Just like you're still there's just there's just absolutely no way.
Bryan Cantrill:Stop doing this. You're making it embarrassing to watch. And so 1 question, Bert, is like because I do feel that this piece that this the recent piece against innovation tokens, which is obviously a, I mean, it's a very click baity headline clearly. Yeah. Because but I do this was circulating after your Twitter thread.
Bryan Cantrill:I'm like, are those it feels like it was person inspired to write this by seeing your I don't know.
Charity Majors:My question. I don't know. 1 of the things that I I really 1 of the things I wrote down, because I was just like, I don't like this, was when he said the big problem with modeling operational overhead is an innovation token. Is that even fair concern in selecting an innovative tool is selecting too many tools. I was just like, what?
Charity Majors:That feels like such an arbitrary like, there's no too many tooling tools. Like, the number of tools you have isn't really relevant. I mean, you should have a few
Adam Leventhal:It's like saying you have too many problems.
Charity Majors:I know. It's weird.
Bryan Cantrill:It's like
Adam Leventhal:we should have fewer problems. That's that would simplify things tremendously.
Charity Majors:There you go.
Bryan Cantrill:Let let's go with your approach. Yeah. Yeah. That you that's funny that that now that you mentioned that, that is kind of a strange dimension on it. What does it look like to have too many tools?
Bryan Cantrill:Is that a and what's the and is kind of the argument there that this is a consequence of pulling everything off the shelf all the time, so you are constantly making you're losing coherence effectively. I'm not sure what the argument is.
Charity Majors:Either. I wrote a blog post a few years ago about, Golden Path. And I wrote it because I was traveling around the world and talking to all these companies. For some reason, I kept hearing the same question, like, repeatedly, which is just like, do we let the the all developers choose what they whatever they want, choose the best tool for the job, or do we pick the tools that we support and only let make them use these? And and, like, the answer is clearly yes.
Charity Majors:Right? Like, the answer is when you get to a certain stage of like, you do mostly want to let developers own, you know, the tools they decide to use to do what they need to do. But as you get bigger, you also need to limit, you know, your blast radius and what you can support. And so my recommendation's been just, you know, you define a golden path. This is what we support.
Charity Majors:You know, these are the tools that we we we buy. We have SREs. We have mastery over days. If you want to choose something else, understand that you are on the hook to own it and operate it and and all those things, which I feel like is, like, it's not because the the question is not too many tools. It's like again, the cognitive the cognitive effort and also just the ability of, you know because every engineer is working on a small corner of the large system.
Charity Majors:But then when you're trying to debug or understand the system, you have to try and debug and understand the whole thing. And when you're interested in having engineers move between teams, which I think is such a healthy practice, by the way, making expectation that engineers don't spend more than 2 or 3 years as a tour of duty in any 1 team. So that they're possibly sort of, like, you know, moving around, which helps standardize approaches and, like, you know, distribute knowledge and, you know, all all of these really good things. It it is really hard if every team is like, well, we're going to want a different stack.
Bryan Cantrill:Totally. And, Adam, I it's kind of like, do we are we rushed by fiat at Oxide? I mean, I don't think we are, but we kind of are. I mean
Adam Leventhal:Kind of are, but people know what they're signing up for. And, actually, the both these things remind me of a conversation years ago where I was interviewing for a VP of Ennis job at a a Fintech company. And they said, how do you kinda you know, people are writing stuff in Haskell and in Scala, and in Groovy, and this and that. And how do you kinda keep them doing that? And I just thought, like, that that's such a weird problem to have in that or it's such an unfamiliar problem to me in that.
Adam Leventhal:I think it comes back as you're saying, Brian, to to values where people when you have this shared collection of values, 1 of those is that empathy for how other people are gonna operate. And so you don't need to be the heavy saying, no. You can't do it in Scala. It's that people know, well, why would I do this? Like, yes, it's fun to learn or whatever, but I understand the implications of this choice.
Adam Leventhal:It's just not worth it. Like, the cost benefit isn't there.
Bryan Cantrill:Totally. I
Charity Majors:guess engineers maturing as they grow is, like, understanding how to make these decisions. Like, this is the kind of thing that I would I would expect a senior engineer to be pretty good at making decisions and a staff engineer to be excellent at making decisions.
Bryan Cantrill:Yeah. I would and and to see the kind of, like, the the cost of, like, okay. Yeah. You can do something different here that we're not doing anywhere else. But there's actually like, there needs to be a very, very, very good reason for that.
Bryan Cantrill:Like, it's not
Charity Majors:And I feel like, you know, the the sort of we sort of, like, moved to have every engineer writing code, every engineer owning their code in production. I feel like so much of this naturally shakes out in a good way from having those feedback loops where it's saying people are writing the code and also participating and owning the code.
Bryan Cantrill:Yeah. Because when
Charity Majors:you when you expose them to the consequences of their own assets, they make categorically better decisions.
Bryan Cantrill:I definitely agree with that. I do think, just to grab another of the Internet's many third rails, This is something I'm I always get nervous about when people have a lot of stops. You know, this is 1 of these things where it's like, what does it mean when you see that no one's, you know, had a a stop that has not lasted longer than 2 years in the resume?
Charity Majors:And Yeah.
Bryan Cantrill:And and and my view is, like, it doesn't it's it's it's it prompts questions. There may be very good answers for that. And, like, if, you know, like, I yeah. Sorry. I had, like, a terrible boss here, and and this company went out of business.
Bryan Cantrill:I mean, there may be, like, very good reasons. But I think I get nervous when you've never been at a stop. Yeah. Because how do you had to live with the consequences of your own actions? And
Charity Majors:Over the
Bryan Cantrill:long term. I do. And, Carrie, I love what you're saying and absolutely agree about the versatility. I think there are companies that have that versatility at the tip of a bayonet, and then which is to say, like, no. You have 2 years on this, and then you're off on this other thing.
Bryan Cantrill:Now you're going somewhere else. Mhmm. And the danger of that is, like, well, the decisions I made 2 years ago, were those good decisions or bad decisions?
Charity Majors:Yeah. Yeah.
Bryan Cantrill:And you don't necessarily and I think it's extremely valuable to live with the very long term consequences of your decisions. Positive and negative. The
Charity Majors:Yeah. I I see this also as a way of retaining engineer really good engineers. We wanna do these things at the tip of the bayonet, you know. For the most part, bayonets
Bryan Cantrill:can be For the most part. Yeah.
Charity Majors:Basically called the bayonets. Right. Like, good engineers get restless doing the same thing. And it takes about 2 years in my experience, to really get good at something and have it be kind of, like, automatic. And so there's something somewhere in the more than 2 years, less than 4 years where you're gonna get restless.
Charity Majors:And if you don't have new challenges, you're gonna leave.
Bryan Cantrill:Right.
Charity Majors:And the people who are really comfortable just doing the same thing for 10 years, not typically the ones you're most concerned about maintaining.
Bryan Cantrill:Well, and I also think, like, there are other ways of getting some of these consequences. I mean, this is the nice thing about being an open source. Right? Where you get to, like, you know, it is very helpful to be able to go back to to and, you know, was that the, the right decision? And and then, you know, hopefully feed some of that back and at the very least for some degree of operational empathy.
Bryan Cantrill:I mean, Adam, I feel like, honestly, the mistake that we've made at Oxide is not enough rust. And in in in particular, no. Like, I with like, bash is the root of all evil. You know? I mean, the the bash is such a, a and I I think, you know, even at as just, you know, the chart you're talking about, the the the outside versus the inside.
Bryan Cantrill:It's like, yeah, we've got bash and oxide too. I'm about to tell you we don't, but we're and, you know Well, at
Charity Majors:least we're why do you have to be embarrassed about it?
Bryan Cantrill:Yeah. Yeah. A 100%. And I would say that we are, to an engineer, embarrassed about it. Usually, there are comments in the bash script about about the I mean, it's like, we no 1 is is advocating for it.
Bryan Cantrill:Yeah. It's not it's not like there are a group of bash dead enders.
Adam Leventhal:Well, you you know, there's more there's more c, I think, than we would need ideally at oxide 2. Like, I think that there are some places where we thought this is gonna be just a heck of a lot faster to knock out in c, or there were some significant hurdles for how we're gonna use c in the kernel and that kind of stuff where I don't know.
Bryan Cantrill:Sort of I I
Adam Leventhal:think there were some opportunities to use more Rust, and and we would have been reaping those benefits now.
Bryan Cantrill:Yeah. Otherwise yeah. We didn't rewrite anything. It was that mean We no.
Adam Leventhal:Right. We didn't rewrite anything.
Bryan Cantrill:Well, I also think that that is, like, another kind of variant of this charity is, like, the and I don't think we talk about enough about when 1 makes the decision to rewrite. And because there there's there's a there's kind of it rhymes with this decision about whether to put a person by. Right? Where you're like, I I I'm kind of in I I'm all of these, I I've used this thing, and now I'm appreciating the problems that it solves well. It doesn't solve well, and I think we actually need to rewrite this thing.
Bryan Cantrill:And I think that that's also a real challenge of knowing when to get right. I think it's and, boy, I I there's and I think that that you should not be able you know, I I got some, decent advice from someone who doesn't know which sense decent advice. Not you, Adam. You sense great advice. But, like, why would I volunteer that?
Bryan Cantrill:Right? If I but, no, but I I the the but when we, moved into a new house, if you're moving to a new house, don't do anything radical until you've lived there for a year because
Charity Majors:Yeah.
Bryan Cantrill:The the you just you wanna see you don't know yet. Take an orbit around the sun. Take a lap. And, you know, see you just wanna get to know your like, live there a little bit. And I do think that when you're making a technology decision, I think this is what I liked about, you know, the way you kinda became came to retriever is that, like, your I mean, it helped, I guess, that there was the only thing out there who had that had z k attached to it.
Bryan Cantrill:I guess that is extremely clarifying. But you do wanna try to make it work. You you shouldn't assume that you're gonna be rewriting your own thing. Or or you there there there is time because if if something is not gonna and then you also have to have the guts when it's not working to, do something else. And I remember Adam with going from talk to hubris, that felt like a, like, oh my god.
Bryan Cantrill:Like, we're, you know, really gutsy decision. Of course, in hindsight, we were, like, 15 minutes into the company at that point. Why did That's
Adam Leventhal:right. There's the idea. That's right. It felt like it took forever. It didn't.
Adam Leventhal:And the other thing is 1 of the compelling things for Talk was it was glued to OpenTitan, which we thought was gonna be very compelling, you know, the open hardware root of trust. And I think that has only recently materialized. It's still fiction. I'm not sure where we are with that. But Well,
Bryan Cantrill:it no. But this is a very good point because I think this is how 1 ends up in any number of bad relationships. It's like it's actually we started out because of this other thing, and then that other thing actually went away, and we didn't actually reevaluate the decision.
Charity Majors:That's right. Yeah. Yep. Yeah. And That no.
Charity Majors:When I was at Chargebee, we did we went through we did a rewrite for the entire API from Ruby on Rails to Golang. And it was it was miserable, but it had to be done because Ruby on Rails, no threads, which means you have a fixed pool of Ruby workers, and you have this possibly, you know, multi tenant platform. That means that, you know, any any back end starts to get slow and within seconds, all of the available workers fill up with bread waiting on that back end. And so you go from fine 20% utilization to down for everyone in seconds. And that's just like, we tried to work around and do a bunch of and finally, we're just like, yeah.
Charity Majors:We just gotta bite the bullet and rewrite it and go like and it was miserable for many reasons. But, like, 1 of the worst is that, you know, Ruby and Rails implicit types, go like very, very picky about type. And mobile apps, very very very picky about things. And also don't update all that frequently, especially at the time. And so anytime a type got changed, the fucking app would crash.
Charity Majors:And it was just, like, a lot of a lot. Like, the the the process of going through that rewrite, you know, we've had to build a lot of things on top of Tsukuba to understand just to be able to see what was happening as we slowly wrote rewrote 1 endpoint after another. It was laborious. We made no forward progress in the product for almost 2 years. Of course, a year later, Facebook, got parsed down for good.
Charity Majors:But it had to be done. It had to be done. Right?
Bryan Cantrill:Yeah. Interesting. It's yeah. And I feel the guy that's so so true for for we we should do an entire episode of rewrites because I feel this is so often the rewrites where they they don't like, you're paying down technical debt, and it's all about the promise of how quickly you were gonna be able to go in the future. But meanwhile, we're gonna be like, we're gonna have 2 years of no work.
Bryan Cantrill:Dulled.
Charity Majors:Yeah. You can't you can't keep developing stuff or you're just moving the ball. You're making it longer. Yeah. Yeah.
Bryan Cantrill:Yeah. It is. It it is it is really, really painful. Yeah. I don't know how you this is what it was.
Bryan Cantrill:It's like innovation run on the bank. I'm not sure Yeah. Well, do another episode of that, Adam. I think that's gonna be we'll we'll kick off another Skype soon. Oh, yeah.
Bryan Cantrill:You you gotta run to the fam here and, Charity did despite this is just a problem. Like, when I when the control rods are pulled out, I when when Adam's not here, I mean, I'll just go all night on Twitter, obviously, as we did that that night.
Charity Majors:So Yeah. Well, this is super fun. Thanks so much for having me.
Bryan Cantrill:Oh, this is great. I and thank you so much. I I love the Any time.
Charity Majors:Where the
Bryan Cantrill:throat is sprayed, and, this is a really a timely topic, I think. And this is a this is an evergreen for engineers, and always great to get your perspective. Great to have you back on, talking about something that is not the, Twitter being
Charity Majors:The end of the world. Yeah. The heat up of the universe.
Bryan Cantrill:That's right. Alright. Thanks, Charity. Thanks, Adam. Thanks, everyone.
Bryan Cantrill:Talk to you next time.