Subscribe
Share
Share
Embed
Oxide and Friends Twitter Space: September 6th, 2021
Put the OS back in OSDI
We’ve been holding a Twitter Space weekly on Mondays at 5p for about an hour. Even though it’s not (yet?) a feature of Twitter Spaces, we have been recording them all; here is the recording for our Twitter Space for September 6th, 2021.
In addition to Bryan Cantrill and Adam Leventhal, speakers on September 6th included Dan Cross, Josh Clulow, Tom Lyon, Simeon Miteff, Daniel Maslowski, Matt Campbell and Moritz. (Did we miss your name and/or get it wrong? Drop a PR!)
Some of the topics we hit on, in the order that we hit them:
- Adam’s tweets on recording Twitter Spaces.
- Tweet on recovering a recording!
- [@4:57](https://youtu.be/PVJfqjJJCkg?t=297) Timothy Roscoe’s Keynote
- Screenshots teasing his slides
- Conf video
- Complicated relationship with academia and industry
- [@13:47](https://youtu.be/PVJfqjJJCkg?t=827) What a modern OS does > … accreted and not designed.
> They were not designed, they congealed. - [@17:10](https://youtu.be/PVJfqjJJCkg?t=1030) Rob Pike’s 2000 “Systems Software Research is Irrelevant” paper
- The value of incremental improvements
- [@21:47](https://youtu.be/PVJfqjJJCkg?t=1307) Building on extant working components and interfaces
- Opaque, proprietary hardware
- AMD Platform Security Processor > Artifacts of the OS implementation tend to have outsized impact > on overall system performance
- [@26:27](https://youtu.be/PVJfqjJJCkg?t=1587) Performance is not the only axis of a system
- Security, malleability, convenience, reliability
- [@31:12](https://youtu.be/PVJfqjJJCkg?t=1872) Specialization
- [@37:02](https://youtu.be/PVJfqjJJCkg?t=2222) Open hardware and firmware
- ARM Cortex-M0 > That’s why we land at incrementalism, we ossify at some boundary. > And it’s very hard to change things on either side without moving in lockstep.
Tom: The PC architecture was a great thing, but now the OS vendors have abdicated any knowledge of the hardware. Give us UEFI and we don’t care what happens beneath that.
- Should ARM have UEFI? (or something like it)
- [@45:29](https://youtu.be/PVJfqjJJCkg?t=2729) Developing hardware is still challenging, but has never been easier than today (especially low-speed)
- [@50:58](https://youtu.be/PVJfqjJJCkg?t=3058) Where will new systems development fit in with our existing (working) systems?
- Low-speed is an opportunity area
- RISC-V for peripherals
- [@56:37](https://youtu.be/PVJfqjJJCkg?t=3397) Backwards compatibility seems to be more important than marginal gains:
- Shingled magnetic recording offered <25% density gain at the cost of compatibility
- Optane: gains didn’t justify the cost
- Smart NICs only made sense in hyperscale server fleets > Josh: If you’re going to change the programming model, you have to blow the doors off on at least one axis
- [@1:00:45] Moving management plane to a NIC.
- AWS Nitro implements this with a series of PCIe offload cards.
- [@1:01:22](https://youtu.be/PVJfqjJJCkg?t=3682) Abstraction boundaries not designed for the current circumstances
- [@1:08:57](https://youtu.be/PVJfqjJJCkg?t=4137) Removing things is so hard
- Things change given enough time
- Graham Lee’s essay on legacy and software dependencies …and in the end will be the command line
- If we got something wrong or missed something, please file a PR! Our next Twitter space will likely be on Monday at 5p Pacific Time; stay tuned to our Twitter feeds for details. We’d love to have you join us, as we always love to hear from new speakers!
Creators & Guests
What is Oxide and Friends?
Oxide hosts a weekly Discord show where we discuss a wide range of topics: computer history, startups, Oxide hardware bringup, and other topics du jour. These are the recordings in podcast form.
Join us live (usually Mondays at 5pm PT) https://discord.gg/gcQxNHAKCB
Subscribe to our calendar: https://sesh.fyi/api/calendar/v2/iMdFbuFRupMwuTiwvXswNU.ics
We've been recording all of these, but we've been recording these in and, Adam, take no offense, the jankiest possible way. I mean, Adam actually did, like, had a, whatever, a 9 part tweet thread describing what he was doing to record these things. Because mind you oh, yeah. How how is it?
Speaker 2:Oh, sorry.
Speaker 1:How'd it go? Yeah.
Speaker 2:I'm I'm I'm good. I actually started I was able to start recording on my Bluestacks rig before I was able to get in here properly. So
Speaker 1:I mean, it just sounds like something you use for for Bitcoin mining. I mean, it's proved It's really close. I know. The so we had this, like, this great space last week with with with Matt. And then we had this, we realized that we had actually, or Adam, afterwards, you must have had that horrifying sinking feeling.
Speaker 1:Oh my god. So, yeah, walk us through it. What happened?
Speaker 2:God. So, I mean, the what happened, I I don't know is the the is the short answer.
Speaker 3:That's always fine.
Speaker 1:So so I get off
Speaker 2:the call and, you know, it's gone an hour and a half. It was awesome. And and, Matt, I know you're here. It was terrific. And and if you haven't listened to it, you should go listen to it.
Speaker 2:And it's the it's the one of these that I have shared with my non technical friends. One, because I've been I've been a little embarrassed to to explain to them what our Twitter space is and that I have one and that I that I would do it on a regular basis. But, you know, the the I sort of feel like the the folks for whom this content is designed, and designed is a is a very I've done a lot of work there. I kinda find us, I think, like the the 100 or so, 200 who show up and then like the 400 YouTube views. But last week, I think was extremely approachable and important.
Speaker 2:So I get off that call, and I'm like, sweet. Let's hear the recording. I gotta get it up. And You're open. It is 6 seconds of just angry static.
Speaker 1:Oh, no.
Speaker 2:Shitting me. So I'm I'm I mean so at a moment of, like, okay, QuickTime recording player, where else might you have put these bytes? And there, like, aren't that many places. Right? Because it's like a Mac and it's like an app.
Speaker 2:And it's either gonna, like, dump it all into the trash or maybe some temp file, but there was no big temp file. So anyway
Speaker 1:So well and so Matt and I are DM ing because I, you know, I'm breaking the news to Matt being, like, your, yeah. So the recording, maybe lost at sea here. And Matt's like, I'm sure it's fine. I'm sure it hasn't been. Like, like, I'll I'll help Adam.
Speaker 1:And then he comes back. I don't know. You came back from the company instead of being like, yeah. The recording may be gone. We may, actually.
Speaker 1:But then it had that great ending because we, asked the Internet. And the Internet had 2 delightful answers, each as surprising as the other, I felt. I don't know how you felt. One was Totally. That Twitter records all these, which I knew.
Speaker 1:But recently, they were making it available to you if you downloaded all your personal information, which I also have done because this is actually the only way I've been able to get if you have a long DM conversation where you wanna, you know, record that, the only way to really get this is to download all of your personal information from Twitter. There's no real API. I mean, Twitter, I love you, but how about some APIs or some of the stuff? And the you know, all the APIs are basically turned off, so you have to download all this stuff. Have you done this, Adam?
Speaker 1:Have you downloaded all your personal information Twitter?
Speaker 2:Never.
Speaker 1:It's fascinating. I mean, it's you they they they collect a lot of information when it's very valuable. But you can only do it once every 30 days. The because they want you to prevent you from using this as an API. The recordings are deleted after 30 days, and the recordings may take between 24 and 48 hours, but, quote, may take longer to show up.
Speaker 1:So we're like, okay. What do we do? So we're thinking, like, we're gonna request the recording on Wednesday. And then meanwhile, someone is someone who I if you're here, god bless you, was like, actually, I think I've got a bootleg recording of this. Like, that is great.
Speaker 2:Yeah. And they they they just kinda showed up in my DMs and were, like, yeah. I've got it here. I missed the first 5 minutes. I was like, man, I will take it.
Speaker 1:That is so great. Thank you Yeah. For wearing a wire or I don't know if you guys recording. I don't know. Whoever is recording everything about their lives.
Speaker 1:God bless you.
Speaker 2:Yeah. Yes. Thank you, our police state friends.
Speaker 1:Yes. Absolutely. I think it's great. I would I well, I do feel strong. So I actually do feel strongly about that.
Speaker 1:I'm happy, and maybe this makes me a bit of an outlier. I'm happy to be recorded more or less all of the time because I view the positives of being able to amplify those conversations and people being able to consume them when at a time that is convenient for them as opposed to us is so much more valuable than the, I mean, I don't I feel like I don't have any privacy anyway right now. So, you know. There you go. I'm gonna over share.
Speaker 1:Yeah. Alright. On to the main event. So I think I had seen the have you watched the have you watched this talk, Adam?
Speaker 2:No. So I had not watched the talk. It it been in my queue. And I am so glad that you raised it to the top of the queue queue because it was so amazing. It was awesome.
Speaker 1:It was great. And I yeah. I I think I had seen someone else had tweeted out, like, a screenshot of this thing.
Speaker 2:Yeah. There were some slides that floated around.
Speaker 1:And the slides were juicy. I mean, they were
Speaker 2:Yeah. Yeah. These these looks like these are our people. Like, this is gonna be great. Like, this could be right up our alley.
Speaker 1:Okay. So and then, Adam, I think it we you and I may need to, to confess a, you and I both have had a complicated relationship with graduate school and academia.
Speaker 2:Yeah. I think yours was more complicated than mine, in part because you, you you you poured you kind of poured your, as I was contemplating grad school, you were gonna pour your own feelings of it into, you know, my top ten. And I had to, like, drink it down.
Speaker 1:Wait. Are are you wait a minute. Are you somehow, I you know, I was gonna you and I were gonna hold hands and have an inferiority complex together, but I'm concerned that you are now
Speaker 2:I'm blame thanking you. You're blame thanking
Speaker 1:you. Thanking me. Yeah. You're blanking me, I guess. Yeah.
Speaker 1:The but so I just have, well, think like a lot of people, a complicated relationship with academia. The I in that, I view I viewed myself as getting a PhD as the pinnacle of intellectual achievement. And that that unwinding and realizing that that's actually not what I wanted to go do, and I could actually go into industry and go do interesting work, was complicated, I would say. And I actually you know, and I still believe that I don't have a PhD yet, sadly.
Speaker 2:Wow. It's not Good for you. I mean, a lifelong learner. Well but
Speaker 1:you because both my grandfather and my mother got PhDs very late in life. So I'm I think I've got some, like, genetic mutation.
Speaker 2:Yeah. I hate go for it. I I say, like, let's do this oxide thing for a little while first or or maybe you can do No.
Speaker 1:That's what I'm saying. I'm applying to grad school tonight, Adam. This is what I'm trying to tell you. This is what I'm leading up to. I'm I I Actually, I'm leaving.
Speaker 1:I'm going to grad school. Accepted.
Speaker 2:What he's trying
Speaker 4:to tell you, Adam, is that he's leaving.
Speaker 5:Exactly. Doing good. Good. Good.
Speaker 2:Okay. Well, this is awkward form for that, but I'm I'm I support that decision.
Speaker 1:I just feel like I wanna make sure the recording was on and I could do it. No. No. Of course not. Of course.
Speaker 1:No. No. I the dumbass. Yeah.
Speaker 4:The dumbass is That's not true. I'm just
Speaker 1:Right. The none of it's true. But the what is true is that I think I I I I've always thought highly of academia, and I have wanted academia and industry to work together to help one another out. And I did for academia to get, to be solving problems that were, interesting and relevant to industry, but not doing so in a way that was presumptuous or elitist or and that I would say doesn't really happen. That's that's basically Well
Speaker 2:well, I you know, I think that the movie that dovetails to to my, dalliance with grad school, which was the 2 internships. The 2 most significant internships I had when I was an undergraduate were in the Microsoft Research Graphics Group and then at Sun Microsystems in the kernel team.
Speaker 1:Okay. In the grads group at Microsoft, are you not even allowed in the same elevator with PH I mean, don't you have to, like, take different I I just I imagine that being a very stratified society.
Speaker 2:It was it was not. And in fact, as a as a first person who had just finished his sophomore year of of college, and I gave a presentation, like, my 3rd week of my internship, to a room full of, like, 15 people, 3 of whom have shading algorithms named after them. It was terrifying, like, terrifying and incredibly flat. Like, for and just as as a quick vignette, Jim Blinn of Blinn Phong shading would sit around, at the lunch table with me and explain how he watched the Simpsons with his wife, which which was which took a lot of time because they watched it twice the second time pausing it. So very, very flat.
Speaker 2:And I think the the and and also I'd say that, in that Microsoft Research Graphics Group, there was a lot of that interplay between some of the work going on in games and Xbox and their research. And so, I think that that kumbaya of industry and academia living side by side was what's much more realized.
Speaker 1:Well, and I would say that the of the academic domains, I think graphics has done this well. And of companies, Microsoft has done this well. So it's actually not very surprising that Microsoft graphics is probably, like, was pretty good. Exactly. No.
Speaker 1:Exactly. So I
Speaker 2:I I agree. I I think that's what the, that's another part of the dichotomy for me was, you know, I had I had a lot of interest in graphics in as an undergrad and a lot of interest in systems. And so it was sort of like industry and systems or, or or or academia and graphics. And, obviously, like, I I I showed up at some thinking, oh, I'll do this for a little while and then, and then go to grad school and it and it's been longer than a little while now.
Speaker 1:Am I about to be blamed again? Is this where this
Speaker 2:is No. No. No. No. I I've I've no regrets on that.
Speaker 2:And I think that,
Speaker 1:Can I get that? I mean, just like I just routine can I just get that in writing? I'm just
Speaker 6:gonna need that to be notarized.
Speaker 2:You can't. But just remember, like, I can cut this out of the That's right. Of the recording.
Speaker 1:I am certain that he said that he would but, you know, I now can't find it. I can
Speaker 4:Yeah.
Speaker 2:Yeah. I dreamed it. You have to get the data dump again. That's right. You gotta do it with an you gotta you gotta wait 25 days or whatever until you get it.
Speaker 2:Yeah. So, you know, I I think the part of it was I I saw the kind of work that we were doing at Sun being totally different than the systems work going on in academia. And and just sort of felt like it was disconnected. The academic work was disconnected from, you know, pragmatic real problems.
Speaker 1:Yeah. Which it it it definitely was and is, I feel. I feel systems has it's been not good in systems. And that was what I was kinda waking up to as an undergraduate is that the academic systems that the the more interesting systems work that was being done was being done in industry. And the, I did you did you listened by any chance and only listened not watched because they did not video it?
Speaker 1:The keynote I gave at Usenix ATC. Yes. Yes.
Speaker 2:And and in fact, when, when Muffy was started his talk and said, you know, I can I'm not gonna be a James Mickens, but I am going to, vent my anger and frustration. I was wondering if that was just you or if there's a whole lineage
Speaker 1:of the same frustration. I had the same question. So, Usenix invited me back to give their opening the opening keynote at ATC, but with no vetting of me and no guidance about what they wanted me to speak about. In particular, they're like, oh, wait. You want me what would you like me to speak on?
Speaker 1:Like, you can speak on whatever you want. I'm like, this person has never seen me talk. Like, that is way way too much like
Speaker 2:It's like a a a festivus airing of grievances.
Speaker 1:I well, the I don't think that that's what they realized they were signing to solve for. In particular, what I what I ended up talking about, which I think is getting us to to Timothy's piece, He alludes to a little bit. I don't think we need to necessarily fix it on. But is the problem in academic systems and in particular, the problem with the the fact that we use in computer science. We unfortunately use conferences as the publishing vector and not journals.
Speaker 1:And it leads to lots and lots and lots of perverse incentives. It it does not yield good results. And this is not a controversial opinion. Like, I think everyone kinda knows this, but, nobody knows how to change. And and the origins of this was that computer science was moving too quickly for journals.
Speaker 1:That unlike, you know, biology and chemistry and so on, like, we are moving really fast. So we can't be in a journal. It because by the time that it's in a journal, it's obsolete. And so we need to publish in conferences. And, Adam, have you ever talked to, like, other scientists about this?
Speaker 2:No. Not really.
Speaker 1:Oh, man. It's like explaining Twitter spaces to friends where you're having to repeat it 4 different times, and they're blaming you. They're like, no. You are misunderstanding. I'm like, no.
Speaker 1:I'm not misunderstanding. I know. It's like, no. But what you're saying is bonkers. Like, that you're saying that you publish in the conferences?
Speaker 1:That's not what a conference is. I'm like, yeah. No. I know. I know.
Speaker 1:I know. I know. You know, you're not mad at me. You're you're and so he gets that a little bit, but I but I think the thing that is and he definitely picks on Aussie for and for good reason. But the the thing that he gets to that I thought was way more interesting, obviously, is talking about what the modern operating system does and should look like.
Speaker 1:So so you saw this for the first time. And, Dan, I know I I think, either I turned you onto this or you turned me onto this. I can't I can't I can't lose track, but I know you saw this as well.
Speaker 4:Yeah. Yeah. You you you, pointed me at it. So not to the way around.
Speaker 1:And, with I mean, so what do you think about it? I thought it was, like, great. I thought it was exciting, invigorating, interesting, very forward looking, and it had a it's, with with some spice to it. I like the fact that he was kinda calling some things out in terms of calling out. I mean, I've got some other comments too.
Speaker 1:It's things he gets slightly wrong or things I'd better like to add to. But, what what do you think overall?
Speaker 2:Well, at first, I loved it, And and I thought that one of the most insightful pieces was was that OSD was intended to be about the design and implementation of operating systems. And that he he's not denying that there are operating systems that that of which Linux is a component on, you know, modern systems, but rather that it is accreted and not designed. And I thought that was, was such a great point, to, to con kind of consider the design of the holistic system rather than just viewing parts as static and then working around them, like, you know, like, as though they are, impossible to change.
Speaker 1:Yeah. I I I had the same feeling and his word choice was just beautiful. If you recall, his word choice was and he his timing is very good about the way he says, you know, these were not designed. They congealed. And I'm like
Speaker 2:That's right.
Speaker 1:That's right. Why is congeal not in the rotation for me? I feel that that was you know, game respects game on that one. I just thought that was a great metaphor. I I can it has, like, the grease, the cooling grease that just feels so it just nails it, I feel.
Speaker 1:Because it did. It it it did congeal. It was not you would not design it this way.
Speaker 2:Absolutely. Dan, what'd you think? If you're there, Dan.
Speaker 1:Dan is
Speaker 4:Sorry. I muted myself. Can you hear me now?
Speaker 1:We can.
Speaker 4:Yeah. I totally know how technology works, and therefore, I'm qualified comment on the That's
Speaker 1:right. Exactly. Perfect. Hey. Could you please build me a computer, sir?
Speaker 4:Yeah. Right. Exactly. I thought that the content was absolutely on point. And I I I think, so I shared this, talk with, some members of the UNIX Heritage Society mailing list.
Speaker 4:And some of the some of the feedback I got from those folks was actually quite insightful. One person said, well, okay. What's taking them so long to get there? A lot of other people said, yeah. That's that's great.
Speaker 4:But, you know, like, who cares about the actual operating system? System? You know? Like, what about sort of distributed systems? Like, our our world is very different than it was a while ago.
Speaker 4:No. I I I think that we care. It's Yeah. Right? What I'm saying?
Speaker 4:Like, microservices have to run on some computer somewhere at some point, you know, and being like, you know, Kubernetes is great or whatever the case may be. But, you know, that does not drive the interrupt controller or the Ethernet or, you know, the storage device or or, you know, provide a process abstraction or whatever. So I I thought the content was absolutely on point. It reminded me very much of Ralph Pike's talk from 2,000 where he talked about system software research is irrelevant in the sense of being a a polemic for the systems community. But I I I know you guys have some feels about that.
Speaker 4:I actually cannot hear that.
Speaker 7:Yeah. No. I think the the the,
Speaker 1:yeah. Like, I I can see, Cluell unmuting himself. The the the the polemic, I feel it definitely has I I mean, I liked that it was provocative the way the poem it was. The thing I I view as fundamentally different from Pike's polemic, Pike's polemic felt kind of nilest to me. And, like
Speaker 3:It was exhausting.
Speaker 1:And I the thing I liked about and this is, like, hard to dial up, but I admire it. Roscoe has got, like, indignation, but with a twinkle in his eye and a look to the future. So it feels much more uplifting than the the the the and the Broad Pike's Polemics is like, alright. So let's just all, like, walk in the traffic. And without without, I think, some of the and I think the other frustration, Josh, obviously, love your perspective on this.
Speaker 1:I think the other frustration, the polemic is it felt like it ignored some of the work that had gone on outside of academia. Like, lots of things had actually happened. And I think especially as you as as a technologist gets older, you have to there's going to be a a temptation towards nostalgia and romanticizing one's youth and past and thinking that nothing interesting has been done, or that all the problems have been solved. And I think all those temptations really need to be resisted. They're really, really dangerous.
Speaker 1:And they're because they to young people in the discipline, they don't feel uplifting. They feel like you're just slamming a door in their face. So I I Josh, am I reading too much into the to Rob Pike's No.
Speaker 3:I mean, I think it just it's like a cranky person that's, I don't know, perpetually bumfuzzled that nobody has adopted his specific thing and, Virgo, like, the field is morbid. Like, alright. I guess I'll go back to bed. Like, okay.
Speaker 4:I I I mean, I think that's a little un unfair. So I've met Rob Pike a few times. I wouldn't say that I know him well, but I I I can say that he presents himself in a certain way that would lend to that interpretation, but that's not necessarily what he means. The way advice would have interpreted the the 2,000 polemic was, hey. We're not doing stuff that people are interested in.
Speaker 4:We should we should really think about what we're doing, and we should start doing stuff that people are interested in. But I I can totally see where it comes across as, like, the world doesn't run plan 9, you're all stupid. I'm I'm leaving. You know? And Well,
Speaker 3:but also just also critically, like, it it re it's it appeared to reject the value of incremental improvement, which honestly, like, incrementalism is, like, it's the shoulders we all stand on when we ship things.
Speaker 4:Yeah. I I I think that's true. I I I also think it's a little bit, you know, I mean, like, when we talk about systems research, right, and building whole systems, like, there is a question of, like, do we ever build a a new whole system kinda from whole cloth? Or, you know, in Rob's case or or actually in Moti's case, do we, like, spin the thread? You know, go shear the sheep and wash the stuff and cart it and then spin the thread and and and use that to create the cloth from which we we build the new system.
Speaker 4:And I I I think it's fair to say that for the past, I don't know, 20 or 30 years, the answer very much has been no. And, you know, the industry and academia both have been very much dominated by Linux, specifically, More generally, sort of the UNIX process model and system call model. So, like, the systems we have these days are very much in the UNIX mold, and that's not necessarily a great thing. I mean, I don't think that's bad. I think that's great for shipping systems.
Speaker 4:But I, you know, I I is there a space to ask questions about, hey. Do we wanna change the process model? Hey. What would what would it look like if we did different things with memory? Hey.
Speaker 4:You know, what if
Speaker 1:we wanted to do a
Speaker 4:real systems, a a real single system image again? You know, we don't we don't ask those big questions, and I think that's kinda what Rob was saying. And and to to a lesser extent, I think that that's what Mofi was saying. Although Mofi's focus was almost exclusively on the hardware. He didn't, for example, call out, like, you know, the UNIX system call interface.
Speaker 4:He didn't he didn't, he didn't question at all whether that's a good fit for modern systems. And a lot of folks will claim that, no, it's not. But he didn't he didn't he didn't mention it at all. Just didn't come up.
Speaker 1:Well, so and I which I so I'll tell you that frankly, I kind of appreciate it because I think that the the the problem that we've had is that we because we have settled on an application model that is is functional, whether I mean, that there are problems with it. I think there are there should be room for kinda radical rethinking. But as I think we've also seen over and over again that software takes a long time to build. We've built a lot of software on these accent abstractions. Josh, your point about incrementalism, there are actually are a lot of good reasons to retain these abstractions.
Speaker 1:So we don't get in to the the the kind of the software projects of the nineties where you spend, you know, just tons of resources on something that doesn't end up shipping because you're trying to solve every problem at once. What we what we have done, I feel, is because we have we have accepted some of that application model. We have, at the same time, said, okay. Well, there's something to be done in the operating system. And what I what I loved about about about Timothy Roscoe's keynote is, like, actually no.
Speaker 1:Wait a minute. Hold on. Hold on. Hold on. There's lots of problems to be done because while you've been dismissing the operating system as kind of done, there's all this stuff that's been congealing downstack, to use his language, congealing underneath you that is really germane.
Speaker 1:And you got and I think if anything, I feel that he understates it. He actually doesn't know he's even righter than he thinks he is in that. And I I do think that, like, one of the things that I would say is, like, a major, a major point of clarification is one of the huge problems is the cores that he's talking about are too proprietary right now. That the that the the world he's talking about is not well known in part because for many microprocessors, that's only happening under NDI. And that is and you don't even control the software on there.
Speaker 1:Like, the the the to put it in, you know, stuff that's very relevant, that's it oxide, but you can rephrase this in terms of any CPU. AMD PSP, the platform security processor, the source code to that is not open. You know? We we don't we and we don't have the ability to run our own software on there, and that's a problem. So we we can't see that world, which is part of the reason that the academics don't see it either.
Speaker 2:And that's true for a lot of the components of that broad architecture diagram that that he puts out there that, of which, you know, Linux occupies a tiny corner. But a lot of those pieces are proprietary and being built and designed in isolation. And so while I agree that he has a very hopeful message for academia, I feel less hopeful for the prospects in industry.
Speaker 4:Interesting.
Speaker 5:You know,
Speaker 4:like, an observation that I that I've kinda made through the years is that industry and academia team tend to leapfrog each other on a cadence of once every, like, 10 years or so in terms of who's doing the more interesting work. And I feel like we're we're actually in an extended stretch where industry has really been leading academia in in a pretty serious way now for, like, 20, 25 years. And that really kinda started, I think, with the web. I mean, the web really caught academics just completely flat footed. And then it quickly became, like, the major application of the Internet.
Speaker 4:And once that happened, there were all sorts of funky systems froms that just needed to be solved. And and, like, you know, people pontificating about micro kernels and, you know, do whatever it was that people in academia were talking about in, like, sort of the mid eighties. Like, that stuff just became unimportant, and it was like, whatever. Make you know? And Yeah.
Speaker 1:And I I would also say that Microsoft sucked the air out of the room as well. That the the in the mid nineties, Microsoft has a foregone conclusion. OS resource is effectively dead. It's been solved. Even UNIX is like UNIX is not interesting.
Speaker 1:That and but your to your point, Dan, all the interesting problems now are distributed systems problems circa the early 2000.
Speaker 4:Yeah. Or, I mean, just more generally, it it, like, operating systems are are very interesting because artifacts of the operating system implementation tend to have really outsized impact on overall system performance. Right? It's like, if you don't get buffering right and, you know, the file system or something, like the networking stack, it doesn't matter if it's fast. Or, you know, if if there's a lot of jitter in your process scheduler and you don't get the algorithms correct, then, you know, the hot prod stuff that's in the hot path isn't isn't running often enough.
Speaker 4:And so the, you know, like, tail latency is widely variable to 99 percentile and so on and so forth. Like, those are real problems that where the where the OS has a direct impact, and people in industry really, really care about that stuff. You know? And I think
Speaker 2:So so, Dan, I think that's a super interesting point, and I think that, because performance is one of those things that could be the driver of this holistic take on the system. But but, you know, one of the places that Motee focuses on is security
Speaker 4:Oh, yeah.
Speaker 2:As what the the low hanging fruit.
Speaker 4:And I
Speaker 2:truly agree.
Speaker 1:Yeah.
Speaker 2:Now I I I absolutely agree that there is, like, a 10 x improvement in security, and forgive me for for kind of diving to that nomenclature. But there there's not performance. And, and, you know, I'm gonna go on a slight tangent, but tell me if this if this resonates. You know, I I spent a lot of time in the the mid 2000s, like, 2010s, like, thinking about SSDs and thinking also this is ridiculous that we have this brand this this new I mean, not that new, but we have this different kind of medium that we we're pretending desperately and and and, you know, ex very challengingly pretending to be a spinning disk. Why was it pretending to be a spinning disk?
Speaker 2:Because everything he expected it to be a spinning disk.
Speaker 4:Oh, yeah.
Speaker 2:And I I, you know, I wrote this sort of naive article now in in ACMQ about, you know, we should be building systems. You know, let let's rethink that whole stack. And there are lots of folks that had that op that had that opportunity and and have that incentive. But I think the answer was there wasn't that much of a win there. Maybe you get, like, 10 or 20% better performance or longevity.
Speaker 8:But it
Speaker 3:but it wasn't it wasn't enough. It wasn't 10 times faster, or you didn't get 10 ton 10 times that throughput into the disk by
Speaker 4:That's right. By doing this. That's right.
Speaker 2:But and and I think the the the, you know, it's not time for the epitaph on that idea. But I do wonder, you know, what what I walked away thinking from Malti's talk was, what what pushes that kind of of thoughtful design of the operating system rather than congealing of the operating system? Well yeah. And and what and what pushes past incrementally?
Speaker 4:Adam, like, my question is, like, define 10 x. Right? What what does 10 x performance mean? That doesn't sort of mean that your job is served 10 times faster. That may mean that you can make changes to the operating system 10 times faster.
Speaker 4:So when you find that there is a problem, you can point it on it much quicker.
Speaker 3:You know?
Speaker 6:That that might be where the, some of these, I guess, pathologies actually come from. Like, you know, he mentions your your baseband processor in your mobile phone that just memory maps everything, and, you know, that's major security problem. Whatever. But, you know, I, for instance, have been looking at these, ESP, microprocessors, microcontrollers. I don't know if if folks know these things, but they're these really cheap microcontrollers that are built in Wi Fi.
Speaker 6:And they are super popular because there is another core on the die that runs a a firmware blob that handles all your network stack for you and, you know, embedded programmer that wants to build cheap IoT device has an AT command interface. That blows my mind. And, you know, why? This is a new chip, a new system. It's commercially successful.
Speaker 6:Maybe it's just because, you know, easy always wins.
Speaker 1:Yeah. I think this is an this is an interesting point, and I also Adam, I think kind of the point you're making as well. I think that we over enshrine performance a bit. Performance is not the only axis of a system. And I feel that the operating system and I feel like even I actually went back and read the paper he referred to as 2,000 11 paper.
Speaker 1:And where they try to make the case for the operating system in terms of performance alone. And I feel that if you're trying to make the case for the the the operating system strictly in terms of performance, it it it's kind of a losing argument because and I think that we would also acknowledge that, like, actually, like, memory protection does not make the system faster. We're not doing that to make the system faster. We're doing that to make the system more reliable. And I think, Dan, to your point, that the a system that actually, like, that that stays up, that that allows for software to be be developed more quickly upon it, that allows that is secure.
Speaker 1:I mean, these are there are these other axes of the system that I feel have honestly been kind of ignored, or or rather systems tend to be monomaniacal in their focus on one of these dimensions when in practice, what we actually need to do often is balance them. With performance, we we can't give up any performance. Like, we can't have an operating system that is going to cause you to sacrifice performance. That that is that's obviously also a losing bet. But we need to kinda break some of these false dichotomies, I feel, and deliver we actually need to deliver a balanced system that has these other properties, not just performance.
Speaker 5:Yep.
Speaker 9:So I feel like the current challenges in operating systems will be very much focusing on, well, actual security issues. So, we were just, talking briefly about web platforms. So on the web, we're starting to expose how hardware directly. Right? Like, USB interfaces, for example, or Bluetooth, stuff like that, which can be very, very severe.
Speaker 9:And on the other hand, we are, coming up with new operating systems. I mean, Moti himself acknowledged the, HarmonyOS from Huawei. There is also Fuchsia, and I think that would be something that people should really jump onto these days.
Speaker 1:Yeah. I mean, I and, you know, we talked to you a couple weeks ago, we talked about Hubris, the one that we're working on
Speaker 10:at Oxide. So, yeah, I
Speaker 1:agree with you. The people that there should be more new systems, and those systems are gonna be smaller. And they, I think, they are going to be focused on one piece of the puzzle that he's got. Because I do feel that for the the the other thing that that was that is Merritt's clarification is when he's throwing up up at the architecture and kind of circling, you know, where Linux is running, where that kind of traditional process model is running versus all of these other cores, That is not by area heat or draw. That is by, like, architecture, by blocks.
Speaker 2:Oh, yeah. Yeah. Yeah. Linux is is chewing up the the most dollars or or whatever in terms of the components of the CPU.
Speaker 1:On every components
Speaker 2:of the system. Right.
Speaker 4:That that that map definitely kinda reminded me of the election maps that
Speaker 3:Yes. I mean
Speaker 2:That's right. Land don't
Speaker 4:vote. Right? Like, yeah. Well, land doesn't vote. Right?
Speaker 4:And all those fast tracks of, you know, unpopulated area are like, that that that's great. But, like, yeah, we need to normalize first view cycles and applications and that kind of thing as well.
Speaker 1:Right. And I and I kinda feel that, like, hey. And then maybe this is where where you you know, might might disagree, but I kinda feel like, hey. We should not I don't necessarily think you know, he says that this kind of Sisi Pneuma view of the world
Speaker 5:a fiction.
Speaker 1:It doesn't exist. I'm like, no.
Speaker 7:It exists. It's just
Speaker 1:that there's this other part of the world that also exists and is very important that and that's the part that has congealed. That's the part that I think, you know, we can, you know, whether and I also believe we don't need to make anything coherent necessarily. I mean, we're certainly not gonna have a single system image span, you know, your, you know, your m fours and your a, you know, your 8 30 fives or whatever on this SOC. I don't think. Right?
Speaker 1:I don't No.
Speaker 4:I I I kinda disagree with that on a in a couple different ways. I mean, I think part of the issue is that it it is an illusion. Right? I mean, like, we we we have to acknowledge that. It's like the the the sort of view of the world that we wanna take from the UNIX style operating system is the only thing that exists that matters is this set of, like, you know, big x86 scores where we're actually, like, running processes and we're running the kernel.
Speaker 4:And and nothing else exists on the machine. And it's like, well, of course, that's not
Speaker 1:true. Right?
Speaker 4:Right? I mean, you know, like, hard drives or whatever it is, like, those things have firmware on them these days, and we need to acknowledge that. And, you know, like, moreover, I remember when hybrid drives sort of coming out, you know, these are, like, SSDs compared, like, coupled with spinning rust, and there's, like, caching layer with some DRAM in it. And I remember thinking to myself, how many, like, broken buggy copies the Linux kernel are shipping on your hard drive these
Speaker 1:days. Or or in your DIMM. This is what they were doing. They did for Optane when they were Intel was having still unspecified problems with Optane because they refuse to tell you how it works. So you're it's hard for you to help them kinda diagnose their manufacturing problems.
Speaker 1:But for Optane DIMMs, they would have them fronted by by DRAM, and then you've got firmware in your DIMM that is gonna be syncing your what you thought with d I mean, it's like, no. I don't want firmware in my DIMM.
Speaker 4:Exactly. Like, I wanna be able to reach in and control that stuff. But more to the like, where I think he was really kinda going with that point was, like, look. Hey. You know, because we want to continue to provide this useful fiction to Linux, specifically, and, you know, more generally Windows and other operating they're like, for for all intents and purposes, Windows, Linux, Unix, etcetera, are all effectively the same system architecture as far as this talk was concerned.
Speaker 4:Right?
Speaker 7:Right.
Speaker 4:Meaning, you don't take this sort of, like, view of the world. Yes. They had different process models and threading models and memory management, so on and so forth. But for all intents and purposes, we can lump them into one category of system. And what he's saying is effectively, like, because we want to continue to provide these programming illusions to those systems, the the firmware designers and so forth are gonna continue to build things that, like, assume this ossification of architecture.
Speaker 4:And so those those lines become deeper and yet more entrenched at the same time. It's like the further we go down this path, the harder and harder it is to root those assumptions out of the system. Will we ever end up with a system that's like a single system image that spans, like, Cortex m profile processors and, you know, I'm sure there's an 8051 floating around
Speaker 5:in the modern
Speaker 1:There's always an 8051. Yeah. Yeah. They're they're like spiders. They're never more than, like, 3 feet away from the 8051.
Speaker 4:Yeah. But, I mean, it's like, you know, no. We're not necessarily gonna ship a single system image that includes the blobs that run on the cortex cores and the blobs that run on the 8051 and the blobs that run on the x86 cores. But that's not necessarily what he's talking about. I think what he meant was like, hey.
Speaker 4:Are we gonna have an architecture where we acknowledge the existence of these other things? And we and we architect systems so that maybe it's not the same image that's running, but these things are aware of each other and they're designed to to coordinate and communicate with each other. Because right now, that just doesn't exist.
Speaker 1:That does not exist. No. I totally agree. And this is where you get to, like, one thing I really wish she had called for, but I understand that this is a polemic towards Aussie, not the industry. But we absolutely, positively need open hardware and firmware in order to be able to collectively make progress in this problem.
Speaker 1:And I know that I mean, obviously, this is like talking oxides, but one of the things that's been super frustrated with us for us is the degree to which these things are completely undocumented. We call them hidden cores. I mean, it's like the number of, like you know, you'll be in a meeting with a vendor. You're asking, are there any more hidden cores?
Speaker 5:Cores? Like, we just found out about 2 more hidden cores
Speaker 1:in the course of this, you know, this this documentation dump or this meeting. Like, what
Speaker 4:So you've got signed this letter. Yeah. I know.
Speaker 7:That's true. And it's like we we need the the
Speaker 1:hidden course. And I I do think especially with the end of Moore's Law and more which to me means that you're gonna have more and more programmability in more and more spots. I mean, I think that we are I see a world in which we we don't a lot of these, like, weirdo components that we've got are actually replaced with m zeros sitting on an f it's sitting on soft logic. Right? If you give me an m zero and and soft logic, like, that thing can become, you know, a fan controller.
Speaker 1:That can become and it can become all sorts of things that we need to actually build things. But, like, we need that to be open and transparent. We need to do the software that is actually loaded on that stuff. We can't have this world in which it's totally opaque. Yeah.
Speaker 2:And that's what allows multiple components of the system to be designed and changed together. Because the reason we'd land in incrementalism is we ossify at some boundary. And then everything on one side of the boundary assumes that interface, and everything on the other side assumes that interface as well. And it's very hard to change those things without doing it lockstep. And it's very hard to move in lockstep if it's 2, you know, 2 companies that hate each other.
Speaker 5:Well, yeah.
Speaker 1:This is a very good point, Adam, because I think that the the ossification is kinda coming out of the best intentions of, you know, the you know, and I I I kinda I blame Compact for this, you know, in that blame slash credit, I guess, in in terms of, like, trying to deliberately not vertically integrate a system to decouple software from hardware so you don't have to kind of constantly reinvent it. But then it has this but now you have the congealing as a side effect of that.
Speaker 5:So
Speaker 8:That that that's kinda one of my pet peeves too. The I mean, the p the PC architecture was a great thing, but now the operating system vendors have abdicated any knowledge of the hardware. Just like give us UEFI, and we don't care what happens below that.
Speaker 1:Yeah, Tom. This is a good point because the and then I feel that the and, you know, John Masters is here, so I can kinda pick on him. John feels that felt strongly that the Arm ecosystem needed to add UEFI. And I think a bunch of us are like, god, no. Why would you what?
Speaker 1:No. You've got a beautiful garden. Like, you don't need to have, you know, a strip mine in it.
Speaker 3:You're not you're not already in the hospital. Why would you come
Speaker 1:You're right. Because and it's it's this idea though that Tom is referring to that the the the OS doesn't wanna have to change for my hardware, which is understandable, but it then it leads to the these layers that just feel like they're they're impenetrable.
Speaker 3:Well, but that's
Speaker 8:because that's how Even without even without all these hidden cores, there's hidden crap going on on on the important cores. Right? The system Oh, yeah. Mode and all these firmware calls
Speaker 4:looks looks like.
Speaker 8:Yeah. We we we need something like UEFI, but but the OS guys should continue to own the hardware once the OS is up.
Speaker 7:Tom, what do you mean
Speaker 4:by that? What what do you mean we need something like Eufy? I mean, what is what is that do that we, you know, like like, what what is the function that's actually important there?
Speaker 8:Well, I'm I'm actually I actually like this ability to separate hardware and software. I mean, I think that that's an architectural point. That's a very good thing.
Speaker 5:So is
Speaker 4:it providing for, say, option ROMs and the ability to place things on
Speaker 5:the PCI box and all of that stuff? Or
Speaker 8:Yeah. And and, you know, just it it lets it it it's a it's a good interface for it to allow innovation to happen on either side. Right? But but, for too long, the OS has totally ignored what's happening underneath that level.
Speaker 4:Right.
Speaker 11:If I could, attempt to, defend the, the idea of adding UEFI to Arm. Now, I'm not I'm not an expert on this, but as someone who has dabbled with the single, you know, the the the Arm single board computers, like the Raspberry Pi and the like, it it I I, for 1, don't like every single board computer having its own patched non mainline kernel and its own patched U Boot. So standardizing on hardware interfaces in in the ARM world as it is in the X86 world would seem to help with that situation. Like I say, though, I'm not an expert on that, so it's just my thought on that subject.
Speaker 8:Yeah. Yeah. No. I I mean, in the PC world, you had Intel providing the ammunition, but IBM drove a specific architecture which everyone adopted. And in the ARM world, you didn't have that IBM equivalent.
Speaker 7:Yeah. And and, I mean,
Speaker 1:Matt, what you're identifying is a fundamental tension for sure. I mean, I I think you you don't wanna inflict and and Tom, to your point as well, that that you you you do want to insulate software from hardware, but you don't want those layers to start operating across purposes. I mean, my this is where my view comes to it. And I get to to to, you know, repeat from the songbook, I think that hardware needs to be open. The if the, Tom, you got a lot of white noise behind you there.
Speaker 1:Sorry. No worries. The, but if the if the part of the problem is that everyone's got their own proprietary Glop and that they want they feel that they are either innovating or they're embarrassed. I think that's a
Speaker 3:Well, I'm just so many of those blobs are rubbish. Like, I mean, honestly, if if if I had the manual and the source, and I could fix the fact that every time I tell this server, like, now I want you to Netboot, and 60% of the time it does. Like, it's it's like if you could I the the layering and the industrial structure and and the way these things slot together as as swappable components isn't really
Speaker 8:the problem. It's the
Speaker 3:slots into some of
Speaker 5:these slots that I can fix.
Speaker 9:Yeah. There is no Yeah. Version of some of these.
Speaker 3:There's nothing that slots into some of these slots that I can fix.
Speaker 7:It it I mean, I think
Speaker 4:I think the the idea of these interface layers is good, but that doesn't imply that you then just sort of wholesale adopt the UFI implementation or that UFI is the right solution there. I And there have been things that folks have done in the past and, you know, I like, I know a lot of this audience is gonna know this, but stuff like, I don't know, open boot or whatever. Like, how it addressed many of these problems many years ago or device trees or whatever. I mean, there are there are mechanisms to do this. But I think and, you know, kinda getting back to what Moshe was saying, a lot of it is that we've we've kinda said we've, like, declared by Fiat that, like, okay, Yuki's the solution.
Speaker 4:Okay, UNIX is the solution. We're not gonna look at these problems anymore, there's no more there's nothing left to be done. And I think that that's just categorically not true. Like, there is a lot of work to be done here. And there's a lot of different reasons to do it.
Speaker 4:I mean, performance is 1, obviously. Security is another. Just, you know, velocity and extendibility. I mean, like, you know,
Speaker 7:try working at Linux some point. It's an adventure.
Speaker 4:It's, like, really hard to get stuff done there. That kernel is very dense, very impenetrable. Some of the code is very, very bad. Transact's 1. Some is not.
Speaker 8:When going back to the academic problem, you know, one one problem academics have is they just can't afford the the what it takes to do anything with hardware.
Speaker 1:Yeah.
Speaker 8:And hardware is so integrated that it's nearly impossible anyway.
Speaker 1:So, Tom, that actually and you are hitting on one thing that I loved about this talk that I strongly believe, which is let's build hardware. And he has got at the end, he's like, you know, we it is I mean, it is shockingly easy. Yes. Hardware is hard. But the it is easier than ever before to build your own single port computer.
Speaker 1:The especially low speed.
Speaker 8:Right. Right. Yeah. I I think I made this point on Twitter that it the whole FPGA scene now reminds me of the home computing scene of the seventies.
Speaker 1:So, Tom, I'm so glad you referenced that because that to me was so it it it bears repeating, the and so in terms of the what you said in that tweet was that the what you're seeing around the open EDA scene and KiCad and FPGAs, what's happening with, like, the ICE Forties reminded you of the homebrew computing movement.
Speaker 8:Right. So a lot just a huge amount of enthusiasm and innovation and and people getting by with the crudest of tools.
Speaker 1:And people also helping one another out. I mean, I love FHA Twitter is the best Twitter. I mean, it's it's a very, like, supportive community, I feel. And, you know, people are not yes. There are different technologies, but there's you don't see the rivalries that you see, I think, in the Ospy kind of tribes that we see elsewhere in the stack.
Speaker 1:I also think that, like I mean, how many people would think that and, Tom, as someone who's kinda seen all of this, how many people today think, like, boy, if I could get in a time machine and go back to, like, you know, 19 the late seventies, it's like, well, you know, look around you and look around for the the current analogs for that and and, you know, you get involved and make some hardware. I think of that. I I love that kind of call to action.
Speaker 2:That's right. Especially, like, as we look back at at, you know, the early days of Next and and Windows NT. You do sort of have that moment of, like, wow, what a what a time to be alive. But, like, right now, when you've when you've got folks milling their own PCBs in their basement with, like, a $200 piece of equipment, and and, you know, colleagues of ours who are designing PCBs that show up, like, 2 weeks later Yeah. And and are doing their own sort of, like, you know, toaster oven, you know, pick and place.
Speaker 2:It's it's pretty awesome.
Speaker 1:Is we had a colleague who went from ideation to board out for fabrication in 11 hours. And that's a board that and it gets simple straightforward board, but it's a board that's been super, super useful. And we've able to go a lot faster by having that thing. And that is that's amazing.
Speaker 7:Well, this is this is one of
Speaker 4:the reasons so, like, one thing I wanna say about this, and this, I think, actually, again, sort of bears on what Moti was saying is, systems have become so much more complex. Like, one of the reasons that I'm like, wow, I wish I could have been alive in PDP 11 era is because I could you know, like and I'm not trying to toot my own horn here. What I'm trying to highlight is the simplicity of that system. I could probably fit a good chunk of that into my head. You know?
Speaker 11:There are efforts though to develop, simpler systems, some some of them based on on FPGAs. Have any of you guys heard of the precursor, project
Speaker 1:No. Tell us about it.
Speaker 11:On CrowdSupply. So it's it's it's a, it's an FPGA based, SOC. It's, it's an FPGA based risk 5 SoC with some crypto accelerators in it. It's it's really designed for for, like, security sensitive applications, and and it's in a mobile device form factor. So,
Speaker 4:go ahead. And and that's that's And
Speaker 11:he's developing and and they're developing their own OS for it.
Speaker 6:And then Rust, in fact? Yes. It's called,
Speaker 3:Zeus or something like that?
Speaker 11:Yes. X o u s, Zeus.
Speaker 4:That that stuff is cool, but that is not the same thing.
Speaker 1:U s x o u s y. No. You'll be scoping that out for the rest of your life. Yes. Sorry, Dan.
Speaker 1:I did I was I was still getting knocked back by the spelling of Zeus that they will be,
Speaker 4:No. No. That that that's cool. I mean, but the thing about stuff like that is that that's that's just not the same thing. Right?
Speaker 4:I mean, you know, like, if I wanna build a a a large scale server, that's a massive undertaking even now. You know? And and we see that at work. Right? I mean, you know, like, it's a it's a daunting task.
Speaker 7:Oh, we see. Yeah.
Speaker 1:Yeah. You're you're
Speaker 11:you're right. But I was I was kind of supporting the the the point about for people who wish they could have been there for the Homebrew Computer Club or whatever. I mean, there there there are effort or or for people who wish they could have been there at the time of the PDP 11, there are contemporary efforts to develop Correct. Simpler systems.
Speaker 4:So the the conclusion of the statement that I I was gonna try to make there though is that, you know, like, the PDP 11 was that was the top that was, like, a very high end machine for a time. Yeah. It was a mini computer. It wasn't a mainframe, but still, like, that was a hot box. Right?
Speaker 4:That was that was a very advanced
Speaker 11:Ah, gotcha.
Speaker 4:There there are a lot of systems that I think, you know, like, Arduinos kinda opened up this this wonderful flowering of creativity and and and the hardware hacking side of things, and that's great. But I'm not gonna, like, you know, try to run a large scale organization off of an Arduino. Like, that doesn't make any sense.
Speaker 1:So, Dan, true. But but the the the the flotilla of of CPUs that and I think is only gonna increase in number that surround that server. This stuff is actually pretty germane for. Right?
Speaker 4:Yes. Yes. But, like, when the when the academic folks are talking about building new systems. Right? And that's, you know and, again, kind of going back to the focus of both the talk, it's like, why aren't we building new operating systems?
Speaker 4:It's like, well, what are those operating systems going to do? Right? It's like they are not gonna replace unit. The the and and this is what Rob Pike said 20 years ago. The amount of engineering effort to produce a viable system is enormous these days in a way that it was not when UNIX came around.
Speaker 4:So at the UNIX 50th anniversary Yep. Can you hear me?
Speaker 1:I I can't. No. You're back now. Oh, you're back now. Beautiful.
Speaker 1:God. I thought that was me.
Speaker 5:I was
Speaker 1:convinced that was me.
Speaker 4:No. So what I what I was what I was gonna say and I'll try to keep it brief. But at at ATC in 2019, some folks organized this event, the Living Computer Museum up in Seattle for UNIX 50th anniversary. And I I went up there. It was it was a super cool event at a great time.
Speaker 4:There was a lot of, like, you know, old school Unix people there. And Mary and
Speaker 8:Horton great thing.
Speaker 4:Yeah. That was really cool. Mary and Horton and I, though, we sat down in front of an emulated PDP 7 running version 0 unit, and we tried
Speaker 2:to figure out how to write a
Speaker 4:b program. And eventually, we came with, like, cat or something.
Speaker 7:But it was it was painful. Yeah.
Speaker 4:You know? I mean, like, you you look at this
Speaker 1:and you're like and and it it
Speaker 4:was very interesting because you could you could kinda squint and you're like, oh, yes. Okay. I recognize it. Yes. Here's ED.
Speaker 4:Here's the shell. Like, here's LS. This is this is familiar. But at the same time, you're like, this was super, super primitive. I mean, this was not, you know, like a modern unit system.
Speaker 4:And the amount of work that has gone into building modern unit systems is something that, you know, if I was a grad student trying to do a PhD, I would not embark on something like that.
Speaker 2:It's a it's a great point, Jane, because it feels like with some of these more novel areas where, Mothi observes that there's a lot more AUSDI pro papers like ML, there are contributions that can be made in the span of, PhD thesis. So, you know, I can I can grind on this for 2, 3, 4, 5 years or 10 or however many and make some contribution? Is the same gonna be true here? Because that that's also a, it needs to be a guidepost as as optimistic as Mote is most organizations most institutions, the the work is gonna be done for folks who are motivated in that way.
Speaker 1:So I think that but there is contributions to be made because so much of the stuff that's important is low speed. So I do think that there's this this very important distinction between high speed and low speed and the high speed parts of the system. And, you know, we're currently, you know, dealing with the, you know, the dim placement rules from AMD where we are dealing with, like, 100 of femtoseconds. And, like, that high speed part of the system is always gonna be extremely expensive to engineer. It's gonna be really hard for academic academia to make contributions to the high speed parts of the system.
Speaker 1:But to me, those low speed bits, that flotilla that surrounds it, that's where the problems are all right now. And, like, actually, academia can focus on it. Those actually are meaningful. You actually can pick up one of those things and actually make meaningful progress. Risk 5 was an academic project.
Speaker 1:Right? It'd be born as an academic project. That's meaningful. Like, that is actually are we gonna use risk 5 for a host CPU? Not now.
Speaker 1:I don't know if ever, honestly. I, you know, I I I guess I'm I'm still of the and maybe this just makes me a a POW of the ISO wars, but it makes me I it's hard for me to see anything other than x86, even ARM displacing x86. But I gotta believe that that that it's gonna displace, x86 at some point. Is risk 5 It
Speaker 4:It will. I mean, something will eventually. Maybe not in the next 5 years.
Speaker 2:But Yeah. Forever is a long time bet.
Speaker 1:Forever is a Yeah. Forever is a long time. It just I'll take the under on that.
Speaker 5:Exactly. It it'll it'll be
Speaker 1:both our
Speaker 5:our heads will be
Speaker 9:in jars
Speaker 5:settling a bet.
Speaker 1:I do feel that, like, that is the way it ends with us. Our heads in jars shouting at one another trying to settle old bets in the year 28100. I mean,
Speaker 3:me did the fight blows.
Speaker 1:Me not being able to recall which position that I took. Adam Adam pulling what Persisting. Adam pulling what I have called 11th fall where he he he injects my past self with a position that I can't verify that I held or not, but is now clearly false. I mean, I of course, that's that's my version of that, Adam. That's that's
Speaker 2:That's probably accurate. I what we're calling
Speaker 1:lighting, which I thought was very funny. That's jazz lighting. That's jazz lighting, actually. Displaced in time. But then it gets it'd be hard to see how alright.
Speaker 1:So how do we get, you know, from where we are to a different spot?
Speaker 10:There there's just no defined interface. If you want to change something, you have to conform to all interfaces that we have seen over the last few years, and you are all older than me, so you will have older examples. Of times people tried to make something new. For example, SMR based, device managed SMR drives and use the new technology. But if you want to interface with a usual computer, you have to confirm to the, ossified structures that expect disk with cylinders and blocks and, all these old interfaces.
Speaker 10:And if you want to change that, you have to go against all the other stuff that's around built around these interfaces. Okay. So that's That's really hard.
Speaker 1:Yeah. No.
Speaker 4:Yeah. I don't know. I believe that anymore.
Speaker 1:Yeah. Okay. Yeah. So yeah.
Speaker 8:Well well well, generally, to do anything useful, you you have to be compatible. But with virtualization and emulation, there's a lot of different ways to do that.
Speaker 1:I think that's right. I think you gotta be careful about how you do it. SMR is a really interesting object example. The, Adam, do you wanna give your perspective on having watched the whole history of SMR, I feel?
Speaker 2:I'm dealing with an issue of the recording right now.
Speaker 1:Alright. So I'm so let me I I let me, I'll proxy Adam here. He incorrectly if I get this wrong. But so SMR is single magnetic recording, and it is basically the more so things so there's a really interesting case that you pick up. It's like, why is SMR not ubiquitous?
Speaker 1:And so, the SMR is kind of the observation that, like, hey, if I can actually if I know what if I can delay rights effectively, if I can group rights together and I can shingle them, I can get higher density. And the the problem is that if I the pathologies of getting that wrong are pretty steep. And the I think the problem with SMR is that the win was just not a big enough win. It was like a 20 it was like a 15 to 20% win, which I feel was just like not enough.
Speaker 2:I'm with totally with you. And, you mentioned Optane earlier, and it also didn't have that kind of economic niche to sit in, where it it was it sort of it just didn't quite make sense or or there wasn't enough of an improvement to to justify the cost. In Optane's case, it was the dollar cost. In SMR, it's the complexity cost and and sort of the the long list of, you know, pathologies that come along with it.
Speaker 3:If you're gonna change the programming model, you you do kinda have to blow the doors off on at least one axis, a $1,000,000,000 to develop the software to
Speaker 8:squeeze the next ten $1,000,000,000 to develop the software to squeeze the next 10% out of stuff.
Speaker 1:Although my read was that I feel like sorry. Let me go.
Speaker 6:I feel like that's exactly what happened with SmartNICs. You know, they you know, the business proposition there was, move some stuff off of your general purpose core onto this, you know, NIC. And and then you have more compute. And, it comes at a at a cost of a lot more complexity. And so you need that scale to say, oh, I'm going to put this in a fleet of a 100000 machines.
Speaker 6:So
Speaker 5:having worked on an uploads project where we
Speaker 4:tried to do exactly that, it does add complexity in some dimensions, but not all. And and and that actually, I think, like, disaggregation is probably going to be the future of systems architecture. And, again, that's kind of what Motey was driving at. It's like we're already disaggregated. We just don't know it.
Speaker 4:Right? But, like, if I can move, say, some of my management plane over to a NIC, good. Then I'm not burdening my application processors with, you know, basically system overhead function.
Speaker 5:And that's that's a useful thing.
Speaker 1:It is useful, but but Simeon's also right in that the Extant SmartNIC abstractions as made available to say the non hyperscalers do just introduce a lot more complexity. And I think this I mean, getting back to kind of the the the thrust of the keynote that we we there will be more desegregation. There will be compute in more places. But what we can't do is try to make available the extent abstractions. We need to be comfortable Yeah.
Speaker 1:Taking parts in the abstractions. And Josh, I think I love the way you phrased it about if you're gonna change the programming models, there's gotta be a real big win. So that's models, there's gotta be a real big win. So that's not gonna mean that we're changing the programming model all the way up the stack. And I think that there's gonna be you know, there's a bunch of software that is going to need to continue to run even as we disaggregate, even even as we get more cores in more spots.
Speaker 1:And then I also think the other rider I'd add on that is it's Jesus Christ that's got to be open. When the when it's opaque, it we can't see how we're vulnerable. We can't see that how the system what choices have been made. And so I I feel that all of that stuff needs to be needs to be open open hardware and open software. It's really essential to get the those
Speaker 5:things to to have traction.
Speaker 4:Totally agree with all of that. I mean, like, some of this complexity is derived from the fact that the abstractions just are not in the right places. It's like we we we say this stuff is incredibly complex because it is. Why is it complex? Right?
Speaker 4:This is the finding thing. Like, we don't we don't go to the next level of why in computer science. But if you say, why is this complex? It's like, well, because, you know, the operating system expects that it's the only thing running on the box, and it's running UNIX and you have to talk to it using system calls. But if the operating system understood that, hey, my control plane is actually running over on those cores over there, right, and and I'm aware of them, then now all of a sudden it can it can account for that, and you can simplify that layer.
Speaker 1:Well, it's
Speaker 2:as opposed to having to tell these baroque lies to cover for the the the fact that this, the the the system was designed in one way is now operating in an environment that was never designed for.
Speaker 1:And to me sorry, Maritz. Go ahead.
Speaker 10:But these, Baroque lights as you call them, they are the result of a coordination problem. We don't get the coordination between all interested parties to make something, some new interface, but it's an incremental process. Someone comes up with this shingled magnetic drive and has to fit it in the existing abstraction in the existing interface. If you want to build build a new machine, based on the ideas, of the talks, you will have to provide either everyone your your win has be so big that, everyone or the people using it are interested in investing or you will have to provide some interface like, hypervisor with Virtio based, interfaces so people can use your win with their exact existing technical, not depth, but infrastructure.
Speaker 8:That's a good idea.
Speaker 1:Yeah. So much Yeah. Sorry. Go ahead.
Speaker 8:But this this is why, you know, the Haze command set is still buried in every device known to man.
Speaker 3:Also also, it's fantastic. I mean, come on.
Speaker 1:That's right. Josh will defend it to all comers. Well, I know. I I I think you're right. And I I do and I and, actually, you know, it's funny because the the in some ways, the greatest act of heresy that we're pulling at Oxide is eliminating the AMI bias.
Speaker 1:We are not gonna have an AMI bias. We're gonna have all
Speaker 3:of the onboarding. AST 25100.
Speaker 1:You you know, the AMI bias nuking the AST 25 100 from Orbit is actually that one is a lot less controversial. Like, literally not having a bias, the the micro processor vendors themselves, and I mean both Intel and AMD, honestly, they they act like we are are climbing to the top of a skyscraper with wings made of flax, and we're gonna fly. I mean, they're just like, it won't work.
Speaker 7:It's like, what do you mean?
Speaker 1:Like, you don't know how to boot your own own systems? I mean,
Speaker 7:it it it it's a I mean, they think that we're
Speaker 1:nuts for doing this. And to me, that should not be I mean, that's a real problem that we have got a really thick layer of unseen software that is required to provide this legacy abstraction. Well, just to your point about this coordination problem, I think it's even worse than that because we've plopped these proprietary interests in there that are really operating across purposes, which and that's, like, that's not the challenge we have as an issue.
Speaker 3:The fact that the the buyer's vendor has to make a profit and it is not because they sold the CPU is is a huge, perverse incentive. I mean, right? Like, if if the CPU vendor had to ship a BIOS because that's
Speaker 1:Right? I mean, because, yeah, you would that, Tom, so this is an interesting so, yeah, would you expand on that? Because that's actually an interesting interesting point.
Speaker 8:Well, yeah. I'm I'm not really an expert, but, you know, when when Arm announced they were doing UEFI, I was aghast because every experience I had with UEFI was bad. And and it didn't seem to replace anything about the BIOS. It was all the bad stuff was still there, and you got UEFI.
Speaker 3:There's no pager in the shell, and it kills me every time.
Speaker 4:Every time.
Speaker 8:I just I just wanted
Speaker 3:to get that in there. It's like
Speaker 1:Yep.
Speaker 4:I've tried setting the bot rate to 300, and I can control that.
Speaker 3:You know, not precisely that sequence of things.
Speaker 8:Well and and then UEFI has to clear the screen so that any embedded system where you're trying to figure out what's going on ahead of that, you you can't see it anymore.
Speaker 11:But at least with UEFI, some cloud providers, at least, using UEFI VMs can boot faster because the bootloader doesn't have to bounce stuff through the lower 640 k of RAM. I've read that the other day.
Speaker 8:Yeah. But you you but
Speaker 3:you could also just you could just stick an ELF binary in the memory and jump to it as well. Like, I mean,
Speaker 1:just Yeah.
Speaker 4:Exactly. There are ways around that.
Speaker 1:UEFI is is jazz sliding us all into thinking that it's the only way to boot.
Speaker 4:There there's we we there's
Speaker 8:a Yeah. We need to get Ron Ron Minnick on here. There was a Linux boot.
Speaker 3:There was a be a Beehive port to OS 10 called XHive, and for the longest time they didn't ship firmware at all. They'd, like, set to run-in the guest that they just had effectively, like, a multi boot implementation, that that ran outside the hypervisor. It would just do what you were supposed to do, and put the the bits that were gonna boot into the RAM of the guest in the right place and then arrange for the CPU to look as if it had been through the bootloader already. I mean, that's, like, doesn't actually have to be a ROM in there
Speaker 5:at all.
Speaker 11:Now if we could just provide an ELF binary when creating our our, EC 2 machine images on AWS.
Speaker 4:Well, you you can in many cases. I mean, you know, like, if you're booting Linux, for example, like, you can enter the operating system directly in 64 bit mode. Like, the boot protocol is very well defined in that sense. And I I I happen to know that there was a written in the last couple of years that that did exactly that in the first time that it booted as SOA.
Speaker 10:We as an industry can't rip out stuff. We have stacks, and, the usual metaphor for it is standing on the shoulders of giants, but it's not we are not standing on shoulders of each other as a very high stack, but it's a mesh. You're embedded in a mesh, of all the components you're using and the dependencies of these components and trying to remove any of them. We have the example of the bios. You you have every u e system or almost every, system includes a compatibility layer to BIOS.
Speaker 10:I'm younger than the specification. I'm not sure if if there's any of the buyers. And still, it's it's going to be there for my entire career. And trying to remove remove stuff is so hard.
Speaker 1:We're trying to project.
Speaker 3:In in its in its defense, the the CSM
Speaker 11:On this topic, there is an essay by, Graham Lee called In the End, There Will Be the Command Line. Obviously, a, an allusion to the Neal Stephenson classic. But, Graham Lee compares the situation that we were just talking about here to a car that is started by horses, which in turn are started by a human walking. So, yeah, look it up. And and I'm I'm pretty sure it's called and in the end will be the command line, or something like that.
Speaker 7:Like, that that that
Speaker 4:I think is is very interesting.
Speaker 7:But, like, you know, part of
Speaker 4:the issue with when we talk about about evolving these systems, and I think that Josh actually mentioned a pretty amazing point earlier. It's like if you want to change a computing paradigm, you've got to throw out some access somewhere. And the reality is that computing paradigms are changing all the time. Look at the way that web development is done today compared to, you know, 19 90 1 or whatever. It's it's it's fundamentally different in every dimension.
Speaker 4:Like, some at some point, those things change. Where we're getting stuck, we we make these assumptions that we perfect backward compatibility for all things for
Speaker 1:Yeah. I think, Dan, you're fading a bit there, but I I I I think, actually, in and I'd looking at the time, it's a great discussion. I know we wanna try to keep these to roughly an hour, but I think that that, you know, it's a good note to kinda to to wrap up on that we are you know, on the one hand, we need to be mindful of the abstractions, Josh, to your earliest point about incrementalism. There's there's there's a lot of value in incrementalism. But on the other hand, when these layers become ossified and operate across purposes, It is, it's really problematic, and we've got there's we're living in a great time, Tom.
Speaker 1:I just again, I just love your line about how this this feels like the Homebrew Computing movement in terms of its excitement around open EDA, open firmware, and so on. So I think that there's a lot of interesting and exciting problems to, to be solved. Obviously, it needs to be done carefully, but, boy, there's a lot of room for improvement. And please, god, can it all be open. That's what I think that would be my summary.
Speaker 2:Right on.
Speaker 1:Alright. Hey, thank you so much, everyone. Really appreciate it as always,