Oxide and Friends

Bryan and Adam were joined by members of the Oxide mechanical engineering team to talk the mechanical challenges of building a rack-scale computer, and--in particular--of scaling manufacturing from just a few racks to hundreds. NOTE: Keep an eye on the chapter art for the pictures under discussion.

In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide Colleagues Elliott Donlon, Brooks Willis, Doug Wibben, and Ben Williams.

Some of the topics we hit on, in the order that we hit them:
  • Topic
  • [@M:SS](link into recording) Leventhal's Conundrum
  • PRs needed!
If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!

Creators and Guests

Host
Adam Leventhal
Host
Bryan Cantrill

What is Oxide and Friends?

Oxide hosts a weekly Discord show where we discuss a wide range of topics: computer history, startups, Oxide hardware bringup, and other topics du jour. These are the recordings in podcast form.
Join us live (usually Mondays at 5pm PT) https://discord.gg/gcQxNHAKCB
Subscribe to our calendar: https://calendar.google.com/calendar/ical/c_318925f4185aa71c4524d0d6127f31058c9e21f29f017d48a0fca6f564969cd0%40group.calendar.google.com/public/basic.ics

Bryan Cantrill:

That's Elliot. Elliot is here with me in the litter box.

Adam Leventhal:

Wonderful.

Elliott Donlon:

Hello.

Adam Leventhal:

Hello. Hello.

Brooks Willis:

Either that Bryan's gotten really good at doing impressions. I,

Bryan Cantrill:

you know, I have been polishing. Are you saying are you talking about my Elliot impersonation in particular? Mhmm.

Brooks Willis:

Yeah. Yeah. It's very impressive. I would have never noticed.

Bryan Cantrill:

I think okay. I I I know it's rude to speak of Elliot, the third person here, but I actually feel that, like, Elliot has a voice that I could impersonate Brooks. It's funny. You know, funny you should mention this. I'm like, this is this is something that's been at the you know, Adam, you and I had a CS professor in college.

Bryan Cantrill:

And I'm like, this guy is I think I can impersonate him. I think that there is that there are some I'm going say, Elliot, delightful idiosyncrasies. Just to be we're not on video, so you can't I'm I'm trying to convey Elliot's facial expression right now, which is definitely a

Adam Leventhal:

Oh, you can just forward this directly to HR. It's like not a big deal.

Bryan Cantrill:

Yeah. I think it's taking a wait and see approach. But I I would say that when I I I would like to say that I I broke some new ground, Adam. When I I felt when I started to impersonate him, it was not you, Elliot. I I I haven't yet started to impersonate you, but the Elliot looks so uncomfortable.

Bryan Cantrill:

I I I kind of I I there was a I opened the sluice. It was a it was a a flood

Adam Leventhal:

100%. 100%. And I think in the same way that George Bush first, Dana Carvey's impersonation of him is then what people impersonated.

Bryan Cantrill:

What people impersonate is they I think you're right. I think people impersonate Dana Carvey impersonating George Bush because he he kind of exaggerated all these things. That's where I think the same way when when when folks every man.

Adam Leventhal:

Exactly. When and so when folks are impersonating this professor, really, they owe a a great homage to you.

Bryan Cantrill:

So, Elliot, this is what you have to look forward to. When I really polish this this lurking impersonation of you that now I've given I'm so sorry. Brooks, how how how dare you open this Pandora's box? This is what what what you what do you call it when a but, like, Pandora's box of Discord? I mean, it's like the the golden apple of Discord meets Pandora's box.

Adam Leventhal:

I think that's what was in Pandora's box.

Bryan Cantrill:

I don't Yeah.

Brooks Willis:

That that is that is that is what Pandora's box is, basically.

Bryan Cantrill:

Yeah. Yeah. I don't think it was, like, goodies. I don't know

Adam Leventhal:

if you got that one wrong. It's not like

Doug Wibben:

a shot.

Bryan Cantrill:

Kit from a from a Greek mythology perspective, Pandora's box contained the apple the golden apple of Discord? No. No. It just, had discord in it.

Brooks Willis:

All of Discord.

Adam Leventhal:

Yes. Just in general.

Bryan Cantrill:

Oh, it just was Discord. Yeah. And not Discord. Pretty Spotify. Maybe that too.

Bryan Cantrill:

Yeah. Welcome to the podcast, Mechanical Engineers. Welcome, Elliot.

Adam Leventhal:

It's like finally a box. We're talking about something I can enjoy.

Brooks Willis:

Thank God.

Bryan Cantrill:

It's great to have you all here. And Doug, Ben, great to have you as well. Elliot is provided that I haven't made you entirely self conscious. Do you want do you want to kind of kick us off here about oh, we actually I'm going to back up that heartbeat because I in terms of required listening, so maybe Doug, you can give us some backstory of mechanical engineering at Oxide and then we can use that as a segue to to the modern era to to to Brooks and to Ben and to Elliot here. Because Sure.

Bryan Cantrill:

We've had some great discussions with you on the on the pod about some of the the the terrific work that you've done before you came to Oxide when you're talking then after having come to Oxide.

Doug Wibben:

Yeah. Absolutely. I can go through a chronology here. So like you mentioned, I started out in a contract role here almost six years ago, working through some of the, you know, basic concepts and designs that led to our our MVP, the the minimum viable product here, at which which, we're now looking to take, you know, to the next level. So we've we've got product in the field.

Doug Wibben:

We're looking to, you know, grow our mechanical engineering team and harden our product and, grow our manufacturing volumes. So, yeah, like you mentioned, I've been on a couple of these in the past for, cabling the backplane and the minibar system, and I guess we're we're kinda looking to expand on what we've done in the last year year or two with our expanded team here. So all kinds of fun engineering challenges that that kind of tend to change as we adapt to different situations for customer needs, scaling manufacturing. We we change our hats pretty frequently here, and it's it's always exciting. So, yeah, that's that's kind of where we've been at.

Doug Wibben:

About two and a half years ago, I joined Oxide, you know, proper as a full employee here, as did Brooks. And probably six months ago, we brought on, Ben and Elliot. So we are we are growing and it's exciting times for mechanical engineering at Oxide.

Bryan Cantrill:

It it really is. And it definitely worth going back and listen to some of those previous episodes. And Doug, in particular, when you were I believe it was in the cabling the backplane episode where you were talking about all of the things that that we had done mechanically to to prevent bent pins to really make the and I I gotta say there were so many details that I might I did just did not appreciate. And this is like and this always the truth about like great engineering. Right?

Bryan Cantrill:

It's like you ultimately you got to sit with an engineer to really appreciate all of these kind of finely tuned aspects of the craft. And but but the net result is like, you think it's like, oh, this just works. It's like, well, it just works because it's been very carefully designed. And Doug, it was really great to hear in this kind of previous episode some of the thought that had gone in to to really prevent Adam what you and I have called Adam Leventhal hardware engineer, which is to say which I think what we we used to say involves a running start. Like that's right.

Bryan Cantrill:

That that's that's one of the the key arrows in Adam Leventhal hardware engineers quiver is a running start.

Adam Leventhal:

Absolutely. It's kind of the only one, but yeah.

Bryan Cantrill:

That's right.

Bryan Cantrill:

Right. We also often call this the max power way around here, which is like the wrong way, but faster for those of you who are the your Simpsons Talmudic scholars. Presumably, Max Powerway has got to be on someone's bingo card, but

Adam Leventhal:

There you go.

Bryan Cantrill:

If not, it should be. So maybe we can fast forward a bit, Elliot, to because when you and Ben joined the team, one of the things I love about our mechanical engineering team is that folks are coming from pretty disparate backgrounds. And you were coming from something that was like mechanically pretty interesting making ag machinery and looking at Oxide being like, this is just like a box? Do you guys make like you guys are like Silicon Valley, you're the Silicon Valley episode. Like you make the box.

Bryan Cantrill:

Yes, yes. But it's it's Pandora's box. It's great. Do you want to talk about your kind of a bit of your background and and your thinking about Oxide?

Elliott Donlon:

This is my Elliot impression. I think

Bryan Cantrill:

I've honed it over the last few minutes. Very well played. Yeah.

Elliott Donlon:

But in all seriousness, hello, everyone. First time on Oxide and Friends. Yeah. Like Bryan was saying, I came from directly from a company called Fido. Rest in peace, Fido.

Elliott Donlon:

We were making agricultural machinery, agricultural robots that harvested aquatic plants. So a lot of moving parts, a lot of, like, interesting first principles design that we were doing, and it was, like Bryan was saying, more obviously interesting. But what I've really come to appreciate about Oxide, and we'll talk about it more later, but is some of the similarities to some of the aerospace work I had done previously. Interesting. Where when you're I'm used I'm very comfortable manufacturing in the, like, one to 10 quantity where it really needs to work the first time.

Elliott Donlon:

And I've noticed a lot of similarities between that and manufacturing tens of thousands or soon even hundreds of thousands of parts where it doesn't need to work the first time, but it's really, really helpful if it works the first time because then you don't have a truckload of bad parts, for example, which is a costly mistake in time and money.

Bryan Cantrill:

Yeah. So a different kind of engineering challenge.

Elliott Donlon:

A different kind of engineering challenge, but leads us back to some of the same relationships with prototyping and analysis Interesting. Where the things that you get to care about are are interesting and nuanced, like contact, for example.

Bryan Cantrill:

Okay. So say say more. What do what's do I or jump ahead to contact if the but, yeah, could what do you mean?

Elliott Donlon:

Yeah. What I mean by contact is we like you said earlier, we don't have a lot of moving parts in the rack. We have the fans and the sleds that slide in and out and doors that move, and that's about it. And all of those are not really moving in service. But when you think about the things that matter in the rack, it's all of the electronic pieces talking to each other and all the software talking to each other.

Elliott Donlon:

And those things need to be correctly connected. Yeah. And there was a a problem that we won't probably won't go into in detail here, but it caused us to look at more of how our drives were mated in detail. Yeah. And that problem ended up not being a mechanical problem per se, but caused us to look at exactly how things were touching under a microscope, basically.

Elliott Donlon:

And that led us to some of the works that, I guess we could show now on the shark fin analysis.

Bryan Cantrill:

So you describe what shark fin is a little bit. Well, yeah, let's use this as kind of a because and and we should say that symptomatically, this kind of approached we we came to this because we were having PCIe errors. We were seeing more PCIe errors than we would expect on certain lanes.

Elliott Donlon:

Yeah. And so the the SSDs are connected to what we call shark fins. They're a separate board connected to our motherboard in the Cosmos LED. And exactly how connected those are and exactly how I'll say exactly how connected they are both in what we call the y direction and the z direction actually matters quite a lot. With typical SOLIDWORKS coordinates, y is up and z is forward and back in this case.

Brooks Willis:

And

Elliott Donlon:

what we were looking at is like, okay. How far in do these drives need to be, and how far out can they be before they actually lose their connection or start to cause PCIe errors? And I think what Doug put on the screen here is a look at our original Shark Fin tolerant analysis. And from the uninitiated tolerance analysis is basically the mechanical engineers fancy addition and subtraction with statistics, where you can actually look at the manufacturing tolerances in and tracing them through how all the things are supposed to touch in the mechanical design and basically adding up the distributions of those things and determining, okay, in our our worst case or somewhat near to our worst cases, how connected are these things? And on the other side, how connected are these things?

Elliott Donlon:

So how smushed in or how smushed out are these things? And what we noticed is that we were actually a little bit marginal on some of the stack ups, and that we were going through a lot of different parts, and they were touching in ways that we expected, but were those distributions were a little bit wider than we thought they should be.

Brooks Willis:

Mhmm.

Elliott Donlon:

And the typical thing in the mechanical engineering world is to be, okay. We need these distributions to be tighter. Let's tighten the tolerances. And as we're scaling manufacturing, we really, really do not want to tighten tolerances wherever we don't need to because that imposes additional restrictions on our suppliers that are already doing a great job. We don't wanna make their jobs harder, and, also, we don't wanna increase fallout wherever possible.

Brooks Willis:

Fallout being quite the opposite. We wanna be we wanna be loosening up tolerances wherever we can for exactly that reason of increasing manufacturing yield, like preventing places where we've rejected a part because we had a tight tolerance on it that didn't need to be or was, like, something we hadn't really done a bunch of analysis on. So, you know, very much the opposite of of, hiding tolerances, as Elliot was saying. We wanna be doing very much the opposite of that.

Elliott Donlon:

Exactly. And we took that opportunity here while looking at the shark fin and the the SSTs under a microscope to figure out, like, okay. The way these are touching in that tolerance analysis that Doug dropped into the chat, it's actually going through the shark fin bracket, which is a metal part that connects directly to the the board itself, and then up through a PEM insert, up through a screw, up through another sheet metal part, through a bend, through another PEM insert, through a screw, through the Cosmo chassis, through a whole host of other parts before it actually gets to the SSD and to the other side of that chain where things are supposed to touch. And that is a lot of things. So what we noticed is we could actually flip that stack over.

Elliott Donlon:

I don't know if someone can drop that into the chat, the new stack. But by having that shark fin bracket touch off against the bottom of the shark of the Cosmo chassis instead, that cuts out a couple of bends and a lot of different components from that tolerance stack up. And thus, by simplifying that tolerance stack up, we've actually shrunk the distribution of fit conditions that we could be experiencing. And that makes our thing both more reliable, and we could even loosen the tolerances a little bit and still get at least as good a result.

Bryan Cantrill:

That is really interesting. And I feel like that this is a very common theme where you all discover like we can actually really we can simplify this and by simplifying it, often make it cheaper to manufacture, we make it more reliable to manufacture. Feels like it's got all of these kind of these other benefits that often come along with by by kind of distilling the problem a little bit.

Elliott Donlon:

That's right. And actually, if you look at the sharpened bracket, increasing the length of the tab on the bottom about three quarters of a millimeter, which is a very, very small change, but that actually totally changes what is touching nominally in the way that we're out. And what that enables us to do is we have a bend and a PEM insert in each of those each of those sheet metal pieces, and we can actually remove those features because they're no longer needed.

Bryan Cantrill:

Yeah. That is really interesting.

Brooks Willis:

I wanna jump back to something Brian said a second ago about taking a problem at like, DartPens, I think, are a a perfect example of this. Taking a problem and really dragging it back to first principles and seeing if we can do something weird, different out of the box to solve it. That was actually one of the things that originally drew me into Oxide from, like, an engineering excitement perspective, was just the opportunity to all parts of the product. Because when I joined two and a half years ago or so, we didn't really have a lot of them in the field. It was still very new, very, like, fresh, clean, shiny hardware.

Brooks Willis:

And this this is all still true today. But there was a lot of the the the pitch was basically, hey. We have this thing. We've taken the computer back to first principles, and now we're making it again from scratch completely. And we've you know, in so doing, we fixed a bunch of problems.

Brooks Willis:

We've done a bunch of, like, really fundamental improvements, like Brian's favorite hardware example, the fans. Honestly, fantastic example. There's a lot of things like that, and that kind of continues to be the what's the right word? Not like the MO, like, that that can continues to be the the central theme of a lot of the work we're doing is how can we just fundamentally make this better by looking at it as a blank slate? Like, is what we're doing correct?

Brooks Willis:

Not just how do we take the thing we have today and, like, incrementally improve it.

Bryan Cantrill:

And iterate it. Yeah. I I think that also, I mean, one of the things that has been really fascinating to me about watching the way you all work is the amount of time that you really spend understanding how they're manufactured. I mean, I I mean, you you have to because it is like it's the physicality of the thing, but that to me has been really interesting that like you are I mean, and I guess this shouldn't be a deep thought that like, you know, Adam, in addition to World War two being stressful, mechanical engineering is very important for manufacturing. You can present that to your list.

Bryan Cantrill:

But I just think that like over and over again, I I mean, Brooks, like you had a very concrete example of eliminating a bunch of hems from the from the rack.

Brooks Willis:

Do you

Bryan Cantrill:

want to expand a little bit on that example? Just because this was this was one of these where it's like and I feel like this this often happens with you all where you describe some aspect of the of the the rack or of of the actual sled that I just didn't that I didn't appreciate before and then you start seeing like, oh my god, yeah, there are a lot. It's like you yeah, how do you assemble this? Like, this really is a a real pain right now to assemble or or what have you. Do you want to hit on that a little bit?

Bryan Cantrill:

And yeah, an explanation as Adam saying, I know I'm like, I'm I'm tossing around a pen like I I know what I'm talking about, which I emphatically don't because this was

Brooks Willis:

You do now. Yeah. Absolutely. So the what we're we're referencing here is a couple weeks ago, I did a demo about this change I'll talk about, and pivoted very shortly into the demo realizing that I had not provided any of the background information that the, the audience needed. So we'll start with that this time.

Brooks Willis:

So, PEM, they make any kind of fastener you could possibly want that can be pressed into sheet metal or, like, blocks of metal, that sort of thing, in order to threads at a stud, add, like, a keyhole pattern, add a knob, any kind of three-dimensional, like, fastener or, like, out of sheet metal plane piece you could want to add, PEM makes. The company is is Pen Engineering, there's a 100 different companies that make these. This is the good one that everyone in America uses. They don't really have a serious competitor. So we just call them PEMs.

Brooks Willis:

If someone if you ever hear someone call something a PEM, they mean these. Exact it's it's a thingy. It's a doodad. It's a it's a bit that is not coming out of the original, like, flat sheet of metal that is getting laser cut or stamped or or whatever and then bent up into shape, which is how, you know, every piece of a car is made, every piece of a tractor computer, you know, whatever. Almost everything in the world is made of sheet metal because it's cheap.

Brooks Willis:

It's easy to work with, but it's really bad for putting threads into because it's very thin. So on the previous on the original version of the rack, you might hear some of us call something r rack model zero or r m zero. What we mean by that is the oxide rack as it exists physically today. That is the first version of the rack that was designed by Doug and team and was the intention there was to be really reliable and really solid. And that worked fantastically.

Brooks Willis:

And now that we're trying to go through and make a ton more of these, PEMs take a lot of time to install, and they take a lot of time to deal with and make sure that they're right. And then when you go to paint a part or powder coat a part, you have to cover them over. You can't just, you know, run paint through it. It'll get paint all up in your threads, which is bad. And so right now, the rack has, some some major sheet metal pieces to it.

Brooks Willis:

Obviously, there's, like, there's a base, there's a roof, there's four big vertical columns of sheet metal, each of which don't remember the exact way, but they're very heavy. Like, one person can't reasonably pick one of these things up and manipulate it. So right now, the start to finish fabrication process on I'm just gonna choose, like, the front left vertical. I think we have a picture of this, in one of the in the in the archive. Someone can grab that.

Brooks Willis:

So this this front vertical of the rack is a seven and a half, eight foot tall big piece of metal. And right now, what our sheet metal fabricators are doing is they're cutting this big piece out. They're moving it onto something called a press break, which is a giant set of angled steel bars that will bend it into shape, doing some of the bends, taking it off, bringing it across the factory floor to another piece of machinery, that looks like a giant well, it's like a smallish press, lining up all the hardware, again, on this, like, 100 pound piece of metal being manhandled by two people, pressing in two. I think our most PEMS in a piece is, like, 40 ish. It's a lot.

Brooks Willis:

It's a lot of individual operations. Every single one of those has to be set in a in a tool, aligned to the part, smashed. Set in a tool, aligned to the part, smashed. Times 40 per piece, times four per rack, times however many racks we're making, which is a rapidly growing number.

Bryan Cantrill:

And when your constraint is and we've said this for a little while, like, our problem is how fast can we make them. Like, that becomes a real, real issue because this is someone this is someone who's on a manufacturing line. So this is a person who's doing this.

Brooks Willis:

This is two this is two two full time employees per part, and there's, you know, many of these per part, and it's operations that are happening many times. So any change we can make is scaled times hundreds per rack of of time improvements. So and then they take this part that now has 40 or so pieces of of kit in it back over to the press brakes and do more bends on them, etcetera, etcetera, etcetera. So you can understand how this gets very ungainly, very fast for trying to make these quickly. So there's another kind of thing called punch, extrude, and tap feature accomplishes the same thing.

Brooks Willis:

It's a threat. You can put a screw into it. It's great. It's easy. And then, but instead of having to take the metal all the way off of the machine across the factory, monkey around with it, get them all set up, and then bring it back across the factory every time, It's done in the same kind of machine.

Brooks Willis:

So instead of pressing in a third piece of hardware, it's actually punching a small hole in the metal. And then if, like, two cones I'm gesturing with my hands, which is not helpful for a podcast, but, you know, hardware engineer, what are you gonna It's like two cones that just get smashed together much like if you've seen a video of a car door getting made. There's a big stamp and it goes bang, and then there's a car door there. It's like that, but it's, you know, six millimeters across and not six feet across. So it's much, much less loud, and it forms this cone shape in the metal by just hitting it so hard that it it flows like a liquid.

Brooks Willis:

And then you can tap that hole normally, and then you just have a thread there. And so this operation can happen in seconds as opposed to minutes. And, again, times times hundreds per rack. So it's like a very it's like a very, very simple change of it's still threads. Like, if you're standing four feet away from the park, it looks exactly the same.

Brooks Willis:

The function is exactly the same. There's more, like, upfront pooling and setup costs. So we've kind of

Bryan Cantrill:

gone through Brooks. It's like yeah. Why does why wouldn't one ever use PEM nuts? It is because when you're doing a new product, it would require you to do a bunch of tooling around it. Is that right?

Brooks Willis:

Depending. So some some punch punch and extrude features are pretty standard. When you're making 10 of something, there's different ways in which you're fabricating those parts. It's much more manual, much less automated. You're laser cutting out the profiles, and then someone is hand hand aligning them in a machine, bending them.

Brooks Willis:

And the machine that you're using there can't as easily do these features. But at the volumes we're talking about now, we're on automated turret punch machines, which are basically it it's like a large table with bearings that these things run around on, And it can line up a hole with one of these, like, conical punches and just go bang, slide the part, bang, slide the part, bang, slide the part. Whereas the like, you can do that with an operator, but it's a much less efficient process. And it's like it's not actually faster, the low volumes, if that makes sense.

Bryan Cantrill:

Yeah. Interesting.

Brooks Willis:

You can also get, like, a little more you have a little more design freedom with the PEMS. Like, you can get more threads. You can get different materials. You can you have a little bit more design freedom with them, whereas with the extrude and tap features, you're pretty much at the mercy of your your material thickness. There's only so much physical volume of metal in the space you have to work with, so you can't put, like, an eight millimeter diameter screw in a two millimeter thickness piece of steel.

Brooks Willis:

It just doesn't work. Like, it's impossible. So we haven't gotten rid of

Bryan Cantrill:

of Adam Leventhal Hardware Engineer. You, you you are you paying attention to shaking that one down? Because I know that's the first thing you wanna try. So Right. Yeah.

Bryan Cantrill:

Exactly. Right.

Brooks Willis:

Yeah. No. Can't. It's gonna gonna rip out, but you you try. I've done dumber things in production that worked at small volumes.

Brooks Willis:

Yeah. Oh, we should pull up the picture

Bryan Cantrill:

the so this is a just a concrete example of how we made it faster to manufacture. Then you also, like these pen nuts are are not super, super cheap. I mean, they're they, like, they they add up. I mean, when you have a bunch of them.

Brooks Willis:

Yeah. They're, like, 50 each PEM is, like, 50¢, and I think the charge for at various wildly manufacturer to manufacturer, what country you're in, you know, cycle of the moon, that sort of thing. But my rule of thumb in the back in the back of my head is 50¢ a PEM, 50¢ to put it in. So

Bryan Cantrill:

Yeah. Interesting.

Brooks Willis:

Dollar a thread. The extrude and tap features are, like, free.

Bryan Cantrill:

Right. Yeah. Interesting. And for us, we're really trying to optimize for time there more than anything else at the moment, but the time say savings is really significant.

Brooks Willis:

Yeah. So the picture I just dropped in chat is on the left, everything in the rack except for the pen or both pictures are everything in the rack except for the pens hidden. So you're just seeing the, like This is the ghost image of the rack. Yeah. This is the PEM only view that Elliot put together.

Brooks Willis:

Okay?

Bryan Cantrill:

The PEMs see their universe. They only know one another. This is the, yes, the universe that they are in.

Adam Leventhal:

That's right. The constellation of PEMs.

Brooks Willis:

As you can see how much lighter the image on the right is. And those two big chunks in the middle are sidecar, which we haven't redesigned yet. So that'll all, you know, that will all go away too. But every single little dot that is gone is a dollar off the bomb and, you know, hopefully, only a minute off of production time for a rack, but possibly more depending on the part.

Bryan Cantrill:

A minute per PEM. Obviously, that that that's about the skilled operator that knows what they're doing. Mean, this is the the I assume this is not

Brooks Willis:

Yeah. Yeah. And I'm I'm amortizing the that cost that time cost over, like, taking it off the machine, carrying it across the factory, that sort of thing. Each PEM itself is probably on the order of, like, low double digit seconds. But, again, it adds up, and it's operator time.

Bryan Cantrill:

Yeah. Is there some yeah.

Adam Leventhal:

Does this also increase errors? That is to say, I could imagine, obviously, if you get one of these wrong, it can kind of ruin the whole part. Is is there some compounding error as or or or kind of failure rate or fallout rate as you have more of these?

Brooks Willis:

These being PEMs or these being

Adam Leventhal:

PEMs. PEMs. Pardon me.

Brooks Willis:

Not more so. They're they're incredibly reliable. All all things get more likely to fail as you make more of things. We we talk about, like, quantity one problems, quantity 10 problems, quantity 50 problems, etcetera. I'd say PEMs, assuming your sheet metal house is good at them and isn't, you know, cutting corners, PEMs are a one in, like, thousands problem.

Brooks Willis:

Yeah. You would hope, or, like, one in hundreds problem, and extrude and tap features are similar.

Bryan Cantrill:

Yeah. Interesting.

Brooks Willis:

So from a quality perspective, I think it's probably not too much of a win. It's mostly a cost and operation time perspective. And just, like, hardware simplicity. Like, this makes there's a a lot of cascading things down from here that we're not even talking about of, like, documenting this is easier and, like, communicating this to suppliers is easier because we just put the thread in and we just say, yeah. This is the thread we want.

Brooks Willis:

Please give it to us. And they say, okay. We know how to put that thread in this metal versus, like, specking out the exact PEM, which involves, you know, picking the the thing, the material, making sure that we've designed the right hole into the part, making sure that it's, like, oriented correctly, making sure that we haven't used weird dissimilar metals that are gonna corrode and fall out, that are gonna, like, bind correctly into, like, thickness of sheet metal we're using. None of which is individually hard. It's all on a a very clear datasheet because Penn is a great company.

Brooks Willis:

But it it's just like another layer of engineering thought and validation that has to go into it beyond, yeah, give me an m three thread, please. And they say, okay. Here you go.

Bryan Cantrill:

Yeah. Interesting. And so that that's obviously a huge one. This one was asking if we were concerned about the divergence between the the the rack as we kind of evolve to rack model one and so on. I think a lot of these things are are things that are not necessarily like really I mean, some of them are are certainly visible, but they're not certainly, they're not software visible like a lot of these.

Bryan Cantrill:

These things are are are affect the manufacturability of the rack. It it they affect maybe the serviceability of some elements. I I would love to talk about the the screw elimination. Are we going to talk about that? Or or not?

Bryan Cantrill:

I'm not sure what

Elliott Donlon:

I think that's on the agenda.

Bryan Cantrill:

Is that on the agenda? Yeah. Where's that in the sequencing here? Because I thought that was a and I would just like to emphasize and this will this will come up a a couple of times in this conversation. We have burned down zero data centers.

Bryan Cantrill:

I just wanna emphasize that for I I don't feel I'm not I'm not gonna knock on wood because we're not

Brooks Willis:

I don't know that I would have said that.

Bryan Cantrill:

Which Right. You would not did did you you feel you feel we have burnt down the data center? No.

Brooks Willis:

The attempt to pay.

Bryan Cantrill:

The attempt to pay.

Brooks Willis:

Oh, the the attempt to pay.

Elliott Donlon:

I knocked on the wood.

Bryan Cantrill:

Yeah. That that's good.

Brooks Willis:

Alright. Good. Good. Good.

Bryan Cantrill:

Yeah. I yeah. I'm I'm now realizing that the absolute the the hubris and arrogance of me saying I would not knock on wood, which they I mean, the gods listen for that. They listen for me saying that, and they're like they they they feel that I'm taunting them with their their lack of creativity. So

Brooks Willis:

We opened with Pandora's box, and now we're here. So you know?

Bryan Cantrill:

And now we're here. Exactly.

Brooks Willis:

But to To address the to address the question that came in about variance, I think that is actually a a good question. We normally, like, just with the PEM change, no. It's pretty it's pretty blind to all things. If we were cutting in just that change, we just abrupt apart. Good to go.

Brooks Willis:

Totally fine. With the sum total of changes that we're calling r m one or, like, RAC model one, we don't need to worry about variance between RAC model zero and RAC model one because it's such a large set of changes that we're actually we're doing a compatibility break. Like, there are we will build RAC model ones as a separate, like, not external facing product, but internally from, like, our design cycle, our data management. Like, we're considering it a new product. And so it will be manufactured on its own line as its its own separate thing distinct from rack model zero, which is what we're selling today.

Brooks Willis:

Otherwise, yes, this would be a a intercompatibility. No. We'd have a lot of spreadsheets.

Elliott Donlon:

Yeah. But, I mean, in even in terms of compatibility, we are still using the same sleds. We're using the same sidecar compatible with the fiber patch panel. It's the same height, same doors, etcetera. So all of that, all the customer facing things that we need to have compatible, we've just defined our system boundary of the things that we're changing inside of that.

Bryan Cantrill:

Yeah. And so and then occasionally, I mean, I think that, you know, when we we shipped the the first rack, we knew that the hardware is is much harder to update in the field, to upgrade in the field. The hardware has got to be right. And then the the software is going to have to and then we need this we need the software to be updatable. And so, you know, go go back to our discussion with with Dave Pacheco and team on on the the what we did around software update.

Bryan Cantrill:

Just to give you an excuse, Adam, to to both ring the chime. And then it feels like the Internet always has now an opinion on the chime. Like, they well, they they they like it. They don't like it. And I wonder if one of you included.

Adam Leventhal:

So Yeah. That's good.

Bryan Cantrill:

But we but we did have a mechanical issue where we we wanted to go kind of update it in the field. Doug, do do you wanna describe how we got here with the the the power adventure?

Doug Wibben:

Certainly. Yeah. So this is an instance where where in an opportunity in the field informed us that we needed to make a change to a certain aspect of the design. And we took the opportunity to make other improvements along the way. So as we you know, the changes that we've been discussing here, as we go through those, we have to be fairly, efficient with how we fold in changes as changes not terribly easy, with existing product, with product in process on the manufacturing floor and so on.

Doug Wibben:

So we've got with with any of these changes that we're suggesting, they're they have to be pretty calculated in when and how we make them. So in in this instance, we had a a failure, which drove us to make a change to how we a safety aspect of the design, I'll say. And along with those changes, we found many other changes that we would like to make along with the same change. So are we ready for the photos?

Bryan Cantrill:

Yeah. We're ready. Let's this is mean, with that kind of lead in.

Doug Wibben:

We had an issue in the in the field where there was a cable. So on the back of our cubbies, our cubbies attached to our bus bar. And from the DC bus bar, we have two slits. So they need a cubby or they need a cable, a wide cable essentially to divert power from, from the bus bar to the individual slits. So those look like this.

Doug Wibben:

Alright. So case in point here on my part of lack of engineering rigor. Normally, when you tie off any cable, especially a power cable, you would want two means of protection here. So I don't think think we're necessarily constrained by that in our safety standards for for rack equipment. But from, you know, past history in the medical device industry, you always want two means of protection on your cable.

Bryan Cantrill:

So So what what do mean by, yeah, two means of protection that that goes to mean?

Doug Wibben:

You currently see here in between what would be the conductor on the inside of the jacketing here, you have you have the live wires. You would want a jacket, which you see here, the red, you know, an insulation layer on the on the wire, the red and the black here, and something additional between it and metal as Yeah. You know, various mechanical conditions can cause that insulation to be damaged either at install or at or or, you know, frankly, through shock and vibration in use. You will see a case here. This was actually from our production floor, thankfully, anywhere further.

Doug Wibben:

Spicy photo here. What happens when there's a lapse, you know, a gap in that in that, insulation? So, in this case, we had a very high power cable that contacted some metal. The the metal the the metal cable tie insert there, you can see the square thing nicked the insulation layer. And, yeah, it it blew a pretty large hole in the in the sheet metal.

Bryan Cantrill:

So It was it was Robert that I'm trying to be who

Doug Wibben:

Yes. It was Robert.

Bryan Cantrill:

Yes. Ring the chime for our episode on holistic engineering. Robert really likes to go end to end. And I think as as he did I think, Doug, were you actually physically there? This is again, as you say, this is unfortunately in our own facility.

Doug Wibben:

I was. Yeah.

Bryan Cantrill:

It it was my understanding that it was that there there was a noise associated with this one.

Doug Wibben:

I was a little bit further away, but yes, there was a noise, and I I witnessed the aftermath. So, yes, this is not something we want, certainly. So, again, going along with, you know, finding things as we scale from one to 10 to, you know, a 100 racks, This was not something we saw in our first mini racks because, you know, frankly, you know, with low volumes, it's a lot slower process to build these and you go through a lot to you know, I think more care is probably taken at lower volumes as you have more time to build things. So as we're ramping up production and hiring through things. Yeah.

Adam Leventhal:

Go ahead.

Bryan Cantrill:

And it's also I mean, this is in the cubby. So this is not something I mean, you're saying earlier about, like, there's no moving parts. Like, this is something that you would kind of expect to be pretty inert. Like, there there's you you wouldn't expect something to kind of rub up against this. I mean, you would you you would it could be or or you can see how, like, we wouldn't see something like this, that it would take a while before we we ended up hitting something like

Doug Wibben:

this. Correct.

Elliott Donlon:

Or that it could be good when you packed it up and it goes through a truck and shipping, however you get it from one place to another, it vibrates a bunch and that installation that was formerly together is no longer together.

Bryan Cantrill:

Right. Well, the the that's also really important because I think that the the shop when you are transporting the the the rack and we talked about the, you know, all of the challenges operationally and and engineering the crate and so on. But the shock and like how much shock and vibe do you have in in transmission? Well, it's like you hit a pothole. If you're on a truck and hit a pothole, that is a lot of I mean, I we're talking about this earlier, but what what are the the g forces that you get on a pothole?

Elliott Donlon:

Typically around six g's, you you tend to plan for, nine if you're being conservative. So Quite a few.

Bryan Cantrill:

That's a lot of g's. That so the lots of things can potholes can do. I I mean, God only knows how much damage to computing equipment our our our lovely city of Oakland has done here with the its famous potholes. Or or insert insert your favorite city name. I don't think I think that that's the Oakland has no monopoly on potholes.

Bryan Cantrill:

The so and so Doug, this was as you say, you you kind of took this opportunity to be like, okay, this is obviously bad. But we now we've got an opportunity here to understand and improve many different aspects of this.

Doug Wibben:

Yeah. So, I mean, first, we need an additional layer of protection on that cable, which again, you know, wasn't necessary to see that right away. So we've got, you know, currently, working through production, we we add tape. The long term longer term plan here is just a plastic clamp to go around this. So no more, you know, cables directly on metal.

Doug Wibben:

There's an extra layer of protection there. So, that's that's, you know, the minimum change we need to make here. But since we're making changes, and in you know, with with the opportunity to drive other change and other opportunities that presented themselves as we were going through and reworking these cable these cables, we found the way in which the cubby was attached to the rack left room for improvement. So working with our, you know, our field service team and, you know, our manufacturing folks as we made these changes, again, you kinda listen to what the pain points are of field service. And that's where, you know, we kind of discovered another issue that, again, it certainly was less visible than a big hole in the metal.

Doug Wibben:

But I'll I'll show a picture of the back of the rack here. So you can kinda see in the bottom right here, these cubbies are attached to the back of our rack with two bolts apiece. They in in the center is our bus bar. The kind of on the bottom right there to the to the right, I guess, just above the the sidecar fan into the right, there's a a bolt there that's pretty hard to get to. I put it there.

Doug Wibben:

It's my fault. But you need a very stubby little wrench to get in there, and there's hardly room to turn it. And and then going through and kind of, you know, reworking some of these units, we we've really felt the pain of that both, you know, personally and and with other members of the support team. So the the plan there was, you know, we're making changes to our our cubby you know, our design and manufacturing process. Let's take this opportunity.

Doug Wibben:

Like, let's let's change other things as well to make this better. So that's where, you know, we've we've gone through on our rack model one design, added you know, we've gone through and removed that that bolt that was hidden, you know, behind the other. And this one is much more accessible, accessible for a power tool. And then, again, we can't just make changes like that just off the cuff. This is where, you know, the engineering rigor comes in, and that's where, you know, Elliot helped prove that this was an acceptable change for our rack.

Bryan Cantrill:

And to be clear, because we're going from two bolts to one here. So Yeah. Yeah. Right. So this is we're going to be potentially closer to the wind here, closer to the margins.

Bryan Cantrill:

So we really want to understand what the mechanical consequences were. So, Elliot, how did you model that? What does that look like?

Elliott Donlon:

And Brian has actually railed on me for this for saying that it's not any more complicated than undergraduate statics class.

Bryan Cantrill:

Oh, so they would I first of they're not rail on you. I was I I just wanted I I want to be very clear about that. But what I the what the remark that I did note, what your remark of note was when you were demoing this to the company. You you said recall from your undergraduate statics class. And that was, know, I I think that it was like, well, you know, only a small fraction of the company may have been nodding along at that point.

Bryan Cantrill:

I know that the I mean, the double e's all took statics. The double e's all all took statics. The the rest of us are computer science concentrators, so we didn't have to take statics. That was the, you know but anyway, yes. Sorry.

Bryan Cantrill:

I

Elliott Donlon:

Yeah. But just to Recall

Bryan Cantrill:

from your undergraduate statics class.

Elliott Donlon:

Exactly. Just a relatively simple load analysis. You take the bolt clamping force and you subject it to a certain amount of acceleration in the correct direction, say, acceleration due to a pothole because your rack is on a truck, for example. And you you eventually get an answer that says, oh, well, actually, if we use an m six bolt back here, a single m six bolt, that's okay. But, actually, an m five bolt is not okay.

Elliott Donlon:

And those are the kinds of answers that we we did get in this case and told us that we could actually go down to one bolt relatively safely. And that's, like, another aspect of just general precision machine design is the way I think about it and what I personally find interesting about precision machine design is that it's all about answering the question of how do we make it work the first time, and how do we make it work every time after that? And the analysis that we are justified in doing when we're manufacturing at scale like this, even though it is a box that does not really move, those are still very interesting problems that are present in the oxide rack.

Bryan Cantrill:

Yeah. Absolutely. And and then we did we changed the bolt as well. Didn't we did we make it a did we make it a slightly larger bolt?

Elliott Donlon:

That one has always been m six.

Bryan Cantrill:

That was always been m six. Okay. So that was it was so really just being able to go closer. We actually improving to ourselves that we can actually do this in a way that would that will not collapse when we hit a pothole.

Elliott Donlon:

That's right.

Brooks Willis:

And what you're saying was the the analysis showed that we could not reduce this to an m five. But it's like That is a Right. We cannot do one m five.

Bryan Cantrill:

Right. Do are you are you, like, sweating a little bit when you're doing that analysis, Ollie? I mean, it just feels like this is, like, such higher stakes than software. And that, like, god, I I hope I you because we we we I did we did a house renovation and I did now I am tempted to gods. Gods, I want if you can hear me right now, I'm not actually tempting you.

Bryan Cantrill:

The we did a house renovation and we were going to have a a window in a in a and our structural engineers like you can't have a window there because I've done the the structural analysis and in the event of of a seismic event, like the the moment frame would be I would you what was the language they used? And so like, great, we'll eliminate the window. And then they came back later, like actually you can't have the window there. And I'm like, okay. Like I've done I double checked the math, it's actually the window is fine.

Bryan Cantrill:

You're like, is it fine? Now I like and I it's like a window in a kid's bedroom too and it's like the kid always has like the blind closed on it. I'm like, can you at least open the blind in this if this window is going to result in the in the this house collapsing on itself because I mean, I'm just worried that the updated calculation is wrong. I you know, I I know this is just like this is just part of being an engineer, but just the consequences feel high.

Elliott Donlon:

Yeah. That's the point of review. That's why we have a nice mechanical engineering team that can review those sorts of things, check the math. And then there's always the mechanical engineering fudge factor of factor of safety

Bryan Cantrill:

Yeah. Right.

Elliott Donlon:

Which is basically how sure do you think you are? How much uncertainty is there?

Bryan Cantrill:

Yeah. Interesting.

Elliott Donlon:

In aerospace, the factor of safety is typically around one point o five or 5% more than exactly what you need or 1.1 times yeah.

Bryan Cantrill:

Yeah. Not two x.

Elliott Donlon:

Not two x. No. No. But in agriculture, which is where I was coming from directly, it was more like two is the minimum that you're working with is typically more like six or eight.

Bryan Cantrill:

The really? In agriculture. In agriculture. And presumably because I mean, the safety consequences are pretty grave.

Elliott Donlon:

The safety consequences and there are a lot more uncertainties that you might face in the the use applications.

Bryan Cantrill:

You're right.

Elliott Donlon:

Meaning, people will use the things that you engineer for almost anything.

Bryan Cantrill:

Also different when we. Yeah, sorry Brooks. Go ahead.

Brooks Willis:

No. Go ahead. Go ahead.

Bryan Cantrill:

Well, I was just going to say that on on the the do you know when we had that that small cluster of earthquakes when we were living when we were at the the Fishworks Building? And we were Yes. We all like and we're all like, okay, we get it. We're going to die in this building that was built in 1920. And we were talking to had a contractor doing work at the time.

Bryan Cantrill:

And he's and we we were all confided in them, but we had now like resigned ourselves to die in this building above the Walgreens at at first submission. And he's like, no, no. Actually, this building is great because it was built in 1920. The city had just collapsed and burned within very recent memory. Nobody trusted anyone else's estimates for anything and everybody was doubling everyone else's estimates.

Bryan Cantrill:

So you you are in this this era of unbelievably over engineered buildings in San Francisco. And do you Adam, it sounds like this conversation stuck with you as well. Yeah. Yeah. You could drive a Buick onto this floor and it would not.

Bryan Cantrill:

He's like, the building to worry about and then he points out a building across the street that was brand new. He's like, that's the one to worry about because that one is built right up to the code. Now, I'm not sure. I was still relieved to move out of that building out of Illinois U.

Adam Leventhal:

I just love that this is the same engineer who used, like, decorative two by fours here and there in his HVAC work for

Bryan Cantrill:

us. Yes.

Adam Leventhal:

So

Bryan Cantrill:

yes. Yeah. Engineer may be in air quotes on that one. But the but it just in terms of, like, understanding kind of where those margins are and and engineering around them. And then being obviously very cautious when you are.

Elliott Donlon:

Yeah. But also not overly cautious. Like, we're

Bryan Cantrill:

Yeah. Interesting.

Elliott Donlon:

Going to engine aerospace standards on everything, we would never ship the rack, or we would ship the rack in a couple of years after the rigorous engineering analysis is

Bryan Cantrill:

done. Right.

Elliott Donlon:

So being able to do the correct amount of analysis is also a critical part of this job and also what is interesting about mechanical engineering generally.

Bryan Cantrill:

Yeah. And, Doug, can you talk a little bit about the high pot testing that we did based on this? Because one I did think that I really appreciated about that. We couldn't we saw this problem. We're like, okay.

Bryan Cantrill:

What let's immediately develop a test where we can find this thing before we because we obviously don't want to ship this to any customer. Like, what would we how would we test the manufacture on the manufacturing line?

Doug Wibben:

Yeah. Absolutely. So, I probably won't do this to do justice that, I think Nathaniel's on or somebody else. One of our electrical engineers will do, but I I can speak to it the best that I can. So, basically, to find, you know, issues with, compromised insulation in cables, there's a high potential test that that happens to, I I don't know, run some kind of electrical magic through the cables in the rack to determine where there is a potential gap less than we would like or dielectric strength rather less than we would like between, say, a a a live wire and a piece of sheet metal.

Doug Wibben:

So we we have a tester. Eric Austin developed a tester that plugs into our our rack into the regular sled spot that he hooks up with a that he connects to a a high pot tester. And I wanna say it's 2,000 volts. It might be more than that. We it's tested far above what our rack would normally see, in the field, well above, to to identify issues potential issues with with shorts and installation.

Doug Wibben:

So this was a test that was, you know, previously done on a finished rack. We have since moved it to be done on a, a cabled rack that's does not yet have sleds to find issues earlier. When we find those issues, the, you know, this the cubbies can be pulled out and, you know, analyze to see where the potential lapse in insulation may be.

Bryan Cantrill:

Yeah. That's which is great. I mean, I just I I kinda love that that feeding back into the manufacturing process to like, okay, how how do we make sure we don't? Now that we know this problem can exist, how do we go find it earlier? The and and so then with that issue and then so after we were able to prove to ourselves that like we know we and we one can't use the m fives, got to use the m sixes, but we can use one m six.

Bryan Cantrill:

And then we we know that we don't ever anticipate having to do this in the field again. But if we had to, it would be it would be less work. But importantly, it's also less work to manufacture. It's another one of these things where it's like, now it's faster manufacturer. It's got less parts to it.

Bryan Cantrill:

I mean, it it has all these knock on effects.

Doug Wibben:

Yeah. So exactly. So all the pain points that we saw of having to access that that that bolt and and attach it are are gone now. So they're much easier to attach at at our, you know, manufacturing partners and much easier to service in the field.

Bryan Cantrill:

So and then on manufacturing, so we you know, the thing that you and and Doug, I mean, I love this with the it was eye opening for me when we were doing you know, building a wholly new product. And one of the things that you did is we three d printed a lot of parts which allowed us to really iterate quickly on what we want to go build. But when we when it but now we know what we want to go build and we want to now do it at greater scale like three d printing breaks down a little bit at scale. There are cost issues and other issues. I mean, Brooks, you you came from Formlabs, you did three d printing and had a kind of great insight there.

Bryan Cantrill:

And I know Ben, you were really kind of at the at the the forefront of kind of replacing some of these printed parts from molded parts. Can you kind of go through the calculus there, Brooks, about how we think about three d printing in manufacturing?

Brooks Willis:

Sure. Yeah. I think this is mostly gonna be a a Ben answer, but I can I can tee him up a little bit? Printing is like you said, printing is great for figuring out what you wanna make. It's I'm gonna say cheap, and then I'm gonna say it's expensive.

Brooks Willis:

But it's cheap, it's easy, it's fast. You can just do it. Boom. One. You know, you got the parts in a week, and you're done.

Brooks Willis:

Or you have one in your lab or your house or whatever, and you have parts in an hour. But the part per part cost is massively higher. So one of the things and I'll I'll hand it off to Ben at this point. One of the things Ben has been doing is figuring out everywhere we can possibly injection mold parts instead of printing them or machining them or whatever. So over to Ben.

Ben Williams:

Yeah. Thanks, Brooks. Yeah. I have spent quite a bit of time as after joining Oxide focusing on converting three d printed parts to injection molding. It's really a very big geometry challenge.

Ben Williams:

Injection molding has very different requirements from three d printing.

Bryan Cantrill:

So say more, Ben. What what do you mean?

Ben Williams:

Yeah. So, generally, with injection molding, we're trying to basically have two halves of a mold that we can squish together, and then plastic is injected into the mold, and then the mold pulls apart. So in order to do that, we need to make sure that that the part's kind of, like, all coming off in the same direction. When we three d print a part, we can kinda just have whatever shape we want to an extent, making sure it's still structurally sound, but it it can have a bunch of holes in it all of all sorts of different ways. We can keep the material thicknesses different sizes, but with an injection molded part, you need our wall thicknesses to all be the same thickness, more or less.

Ben Williams:

And then we also have to worry about, what I call, draft angles. So say if you had a a cube with a kinda open face on it and you tried to pull that off of a mold, if it was just a flat wall, it it would be pretty hard to pull it off. Right? You get a lot of drag. But if you draft it, so you just add a few degrees to your walls, now it's a lot easier to to pull that part off.

Ben Williams:

So, generally, the exercise I'm doing is taking Yeah. Parts that we already know work, and they they're three d printed, but they're not gonna work for long because we need to make lots and lots of them, and it's just not very feasible to keep three d printing them. So we need to

Bryan Cantrill:

track And that is mainly a time issue more than a cost issue, I assume. Yeah. Yeah. I understand.

Ben Williams:

Yeah. Yeah. We can buy them. They will be very expensive, but we're mostly focused on making sure we can get get them in time is is the main constraint right now.

Bryan Cantrill:

So what are some of example of parts that that you kinda gone through this process with?

Ben Williams:

For sure. I think I have an image of of one. There we go. So we've got these transceiver blanks. So we've got our switch on the front of the rack that's got ports for a large amount of transceivers.

Ben Williams:

Haven't haven't memorized the number yet. We don't usually populate all of them. Usually, we're just we just have a few, transceivers going into our switch, but we don't wanna just leave an open hole there. 64. Thanks, Brooks.

Ben Williams:

So instead, we're we're putting a three d printed kinda, like, plug in there, and then also the, fiber will go into that plug to kinda protect the fiber and keep it routed where it's supposed to be. So pretty simple function, but there's a lot of complexity there. We need to make sure it mates properly with two different parts, and we have to kind of meet some fiber standards to to make sure we're doing that properly. So there's a three d printed version of that that's working quite well, that I kinda did a comparison on the left there. And then in that picture, there's an injection molded version of it on the right.

Ben Williams:

We've kind of added some details on it to make it, you know, a little more aesthetically interesting. But also, there's just this big challenge of taking kind of a big chunk of of plastic and coring it out in a way that's that's pleasing to look at and is also possible to to pull apart in a mold.

Bryan Cantrill:

Yeah. Interesting. And then is there a are there tools to kind of help you do this in terms of where you need to change the geometry? I mean, is I I feel like this has gotta be a pretty common journey for a part where it starts off three d printing at low quantity and moves to injection molding at a higher quantity. Is that it?

Bryan Cantrill:

Or is this Yep. Yeah. Interesting.

Ben Williams:

Yeah. I mean, generally, we're just using SolidWorks' built in tools to do this.

Bryan Cantrill:

Yeah. Interesting.

Ben Williams:

You can look at, like, draft angles. There's a tool to just kind of visually see. I don't have an image of it, but you would have areas in red that are one direction and areas in green that are another direction. It would kinda tell you just visually real quick, am I am I able to pull this part, in the direction that I want to? And then you can also do things like checking for wall thicknesses, make sure, you know, all your walls are are an even thickness.

Ben Williams:

And, generally, we can work with our molders as well too. They'll they'll provide a level of, kinda rigor that isn't quite available to us just with our, CAD tools, but they'll they'll do some, plastic simulation to kind of show the quality of the mold, and we can take that feedback and kinda iterate a little more before we actually create the mold.

Bryan Cantrill:

How is the mold made? What is the mold made out of?

Ben Williams:

Depends a bit on volume. Most of my career, we've been doing aluminum molds. They're they're cheaper, but they don't last quite as long. We're actually doing a few steel molds in this case because our volumes are are just that high. So it doesn't really change much from my perspective about designing the part, but I I think from the molders perspective, it is different.

Bryan Cantrill:

And is that like is that like milled then? Is that how that is manufactured? Is this like a is that CNC milled? I mean, how is that how is that thing manufactured?

Ben Williams:

Yeah. It's usually CNC'd. In some cases, they'll use wire EDM to do especially difficult features. Right? They're just they just have to make it one time, so it can be a pretty expensive process to make the mold.

Ben Williams:

But, yeah, you can get into some pretty sophisticated, machining technologies there.

Bryan Cantrill:

Yeah. Wild. And then so and then I feel like we've joined the big leagues that we're actually going to steal molds for things. This is the this we've is really arrived, grown out of aluminum molds. And I I mean, these parts, and I don't know if this is an example where you steal molds, but there are some of these parts we're just gonna make we make a lot of them.

Ben Williams:

Yes. Yes. The volumes are we're kind of like talking with our supplier and they're like, that's that's annually? Yeah. Yeah.

Ben Williams:

I guess it is.

Bryan Cantrill:

Yeah. Wow. That is I mean, because I think you know, we we always think about terms like rack volume and the rack volumes are like lower because they've got like but the rack contains many, many, many thousands of parts if you include certainly include all the electronics. So it's like the a small amount of rack volume can actually have a lot of parts, and a lot of rack volume has many, many, many parts.

Brooks Willis:

The highest quantity custom part in a rack is right. Possibly this part we're showing on screen, but it's up to 320 of a part of a specific part inside the rack.

Bryan Cantrill:

And what is the Brooks, what is this part that we're showing on screen?

Brooks Willis:

Ben, you wanna take that one?

Ben Williams:

Sure. Yeah. So this is what we're calling our SSD blank.

Bryan Cantrill:

Oh. We're

Ben Williams:

we're kind of looking at having, SSD, cages that don't actually have an SSC inside of them. They're just a piece of plastic instead. In cases where you might only need a couple SSDs on your server, we still wanna fill in the rest of the server to make sure we're you know, still have the same thermal conditions and EMI conditions and all that. So, yeah, this is a kind of just Yeah. It granular chunk of plastic that's that's doing that function.

Bryan Cantrill:

So you I mean, you you raised an important point because this is not just these these blanks are not just for aesthetics, but we've got these other and the Adam, you can ring the chime again for the oxide of the chamber of history. He's talking about like all for electromagnetic interference and when you know when you get compliance in the rack, it's like you can't just then ship a rack with kind of with all the stuff blank, especially for EMI. And when you say the thermals as well. So this everything has to be engineered.

Ben Williams:

Absolutely.

Brooks Willis:

This is actually one of those changes you do so you don't have to do as much engineering. Like, we could we don't know that you don't need drives in there to pass EMI testing or thermal testing or all that jazz. But we know if you put the drive in there, it passes.

Bryan Cantrill:

Right.

Brooks Willis:

And and we don't wanna redo safety testing and EMI testing because it's time consuming, expensive, and painful in whatever order depending on how you're feeling that day. And so and so making this change and putting this part in there prevents us from having to go back and do all of that engineering to prove or testing to prove that it's still safe, compliant, whatever.

Bryan Cantrill:

Yeah. Yeah. Interesting. And so it'd be and this is our highest quality thing in the rack, especially those that are because I we've got some folks that wanna have different balances of storage versus compute and so on. So they end up with a lot of blanks, potentially.

Brooks Willis:

And I do wanna touch on cost actually on on this part because I think this is one where it does does factor in in a major way. Obviously, this is a lot cheaper than an SSD or orders of magnitude, in fact. But the cost trade off between three d printing and this is where the three d printing is cheap, but expensive comes in versus injection molding. When you're printing, say, this part this part would cost $50 to get printed if you got it out of, equivalent materials to what we're molding them out of, which is fine if you need 10. But if you need 320 per rack, that's actually a lot of money very quickly.

Brooks Willis:

To get the mold for this part is gonna be I think this one's, like, 50 or $60,000, which is on the expensive side for molds. Feel free to correct me if that number is is wrong. At least one of our molds is up. Some of our molds are up in the $5,060,000 dollar range. So, obviously, we really need to know that it's right before we do that because that's expensive.

Brooks Willis:

But now every time we buy one of these blanks, it's, like, 2 or $3, $4 maybe for a larger injection molded part. So there's a lot of upfront tooling cost and then very, very little cost moving forward.

Bryan Cantrill:

Yeah. So this is one of those things where when you get to new levels of scales, were talking about problems that kinda ones versus tens versus hundreds versus thousands. So that that's you get to a crossover point where it definitely makes sense.

Elliott Donlon:

Yeah. And Dan actually did graph that crossover point, and it's it's surprisingly low quantity for some of these parts, actually.

Bryan Cantrill:

Interesting. Yeah. Where it it makes sense. Yeah.

Ben Williams:

One one of those parts is is hilariously fast payoff. I'll I'll share it sometime. It's the when we have a blank for the whole sled. So if we're not completely cop populated with servers, that three d printed part is immensely expensive. So you you Oh, interesting.

Ben Williams:

Extremely quickly. Yeah.

Bryan Cantrill:

And so is that one where you hit the cutoff point earlier, not because the quantity of part, but because of the of the complexity of the print?

Ben Williams:

Yeah. And just the volume of the print, really. Like, prints get very expensive once you talk about larger parts because you kind of have to print the entire volume of that area. So it's a lot of time and it's a lot of plastic. Yeah.

Bryan Cantrill:

And, Doug, you know, I still have correct if I'm wrong, but the very first air shroud you did was three d printed. Yes. The first gimlet. That that that gray shroud. And I I did not realize that you could like, it it I just didn't realize what you could three d print.

Bryan Cantrill:

And it's like, well, this is because it's it's very robust. It does not feel, you know, it's a long way from a home three d printer. Let's put it that way.

Doug Wibben:

Yeah. Those were $600 each. And now at scale, we're molding them for less than 10. So yeah, there's there's the cost difference there.

Bryan Cantrill:

Yeah. And we did Brooks, do wanna describe the the the the because we we have a clear shroud on Cosmo, which is very exciting.

Brooks Willis:

Yeah. Much to Brian's chagrin.

Bryan Cantrill:

You know, no. No. No. I not at all. I I

Brooks Willis:

I You've come you've come around?

Bryan Cantrill:

I've come to love the clear shroud. And and you know what? I have come to love that you love it. That's actually the more important thing. But, no, I I do I I have come to the the clear shroud is is pretty cool.

Bryan Cantrill:

Do you want to describe how we got there?

Brooks Willis:

Yeah. So the on Gimlet, the airflow shrouds do we have a picture of the airflow shroud? We can pull up for context? We have an airflow shroud in the middle of the sled that makes sure all the air goes over the d r five and the or d d r four and the heat sinks and all that jazz and doesn't just pass uselessly over open air because thermal efficiency is good. That's a big injection molded part.

Brooks Willis:

That's one of those $50,000 molds, $10 apart. It's kind of a lot. But it's it's fundamentally like a a two dimensional sheet that has been bent and shaped, lot like sheet metal, but plastic. And so one of the things we looked at pretty hard for Gimlet or for sorry. For Cosmo a little while ago was if we could make this even cheaper than injection molding because we're gonna be making a bunch of these things, and, you know, the cost savings do add up.

Brooks Willis:

And it's it's a kind of silly injection molded part. It's huge. It's very, very wide. So we were looking at a process called thermoforming where you basically you machine out out of wood or steel or aluminum, kind of any any rigid or thermally stable material. You machine out the inside of your shape, whatever that might be.

Brooks Willis:

So if you were gonna mold a, like, just a drinking glass this way, that's a terrible example. If you're gonna mold a plate this way or a bowl, you would machine exactly the piece that fits inside the bowl. And you would flip it upside down, and then you'd get a sheet of plastic, like ABS in this case, really, really hot, not melted, just up to the point where it's sagging, it's soft, it's it's becoming gooey. And you smash it down over this mold, and you pull a vacuum on the whole assembly. And it sucks down, and it forms exactly to that, to that shape.

Brooks Willis:

That this is how, if you get a salad, a salad bar. That's how those are made. That's a thermoform clamshell piece or, like, takeout.

Bryan Cantrill:

Oh, interesting.

Brooks Willis:

Yeah. Okay. That's all thermoformed. Because the again, mold cost something, but then at scale, the parts cost nothing. Like Right.

Brooks Willis:

They're almost free at that scale for that kind of part. It's like penny pennies per part. And the thermoform trout was gonna be, like, a cut like, 3 or $4 per part and, like, $20,000 of mold. So from a cost perspective, it made a lot of sense, and we and in so doing, we're also looking at, oh, could we make this clear? Could we and kind of get it as it's been posted in chat now.

Brooks Willis:

Kinda like the clear Game Boy Advance style, like translucent purple or translucent or just, like, fully clear aesthetic so people can see in yeah. So that people can see into the sled when they're looking at it. Kind of for, like, a marketing perspective of, like, hey. Look. Here's your computer.

Brooks Willis:

You can you can see your computer. Isn't that cool? And because we all grew up with transparent electronics in the nineties, and we just want our purple Game Boy back, basically. So we were we were looking like, can we make a green one? Can we make a clear one?

Brooks Willis:

What are we gonna do? And the fur and the the thermoform parts looked bad. They looked like they were ugly. They did work. They worked fine.

Brooks Willis:

But, oh, buddy, they were not pretty. And if the point was to make this a cosmetic part, that's not the point. So we we would have been saving a lot of money, but, like, severely degrading the aesthetic quality of the product.

Bryan Cantrill:

Yeah.

Brooks Willis:

For something that we're doing in order to make a cool window into the into the technology, it doesn't make sense. Right. But at this point, we'd we'd kind of weaseled the idea of a clear airflow shroud into the oxide consciousness totally accidentally, totally not deliberately, if anyone asks. So when we switched back to injection molding and and did that major update on that part for for Cosmo, we did the first shots of that tool in clear because it's easy to go from clear to opaque and harder to go the other way because of mold texturing that we don't need to get into. But so we got the first shots of that airflow shroud in clear kind of without asking and then just showed up to OXCON with them.

Brooks Willis:

And we're like, look. Isn't this cool? As a as a little internal marketing pitch. And they're still clear. So it worked.

Bryan Cantrill:

They definitely worked. No. They they they look they do they they look pretty good. I I mean, I I love the old shrouds too.

Brooks Willis:

But, you

Bryan Cantrill:

know, I I love all my children.

Adam Leventhal:

But Don't pick your favorites. Exactly.

Bryan Cantrill:

Exactly. But I I those clear shrouds look awfully good. And they definitely look better than the third form of ones, which it's actually a relief to know that you had the same conclusion. Yeah. They it did not.

Bryan Cantrill:

It was not good. Like, these are not good.

Adam Leventhal:

Hey, Brooks. The other day, you were demoing the the manufacturing process that or or some of the changes that you're making in rack model one explicitly to make it easier to to fabricate. So I was wondering if you're gonna talk about that some. And then I was also wondering about the degree to which we collaborate with manufacturers on design. You know, back to Bryan's home renovation example, you know, I I talked to a general contractor, then we did what was called value engineering.

Adam Leventhal:

Like, how do we take these That's right. These these plans and make it so we can actually build it and you can actually pay for it. So I'd love to hear about, you know, the the working with manufacturing to make these fabricatable.

Brooks Willis:

Yeah. I mean, I could talk about that on a house or on a on a rack. But right now, I have carpenters had carpenters in my basement this morning replacing all my beams. So it's a close to close to my hard example. But No.

Brooks Willis:

We do we work with our our supply partners a ton. Many many of the individual specific changes between rock model zero and rock model one came out of a conversation that, I believe, Doug and Ben had at one of our supply at one of our, like, key suppliers with their their engineering team, their their quality managers, their their, like, people from the factory floor who are building the racks, all got in a room and just talked through every single thing they had on rack model zero that was, hey. This is kind of annoying. Like, this is this is hard to fix. This is hard to do.

Brooks Willis:

Like, this process thing here has rough edges. Like, how do we literally or figuratively. Like, the PEMS came out of that conversation. It wasn't it was kind of on our radar, but they were coming in and saying, hey. These PEMS are a huge time and cost problem, and they're really annoying for us.

Brooks Willis:

They're really, like, messing up our our workflow. Can we get rid of them? And we're like, I don't know. That's figure it out. We'll try it.

Brooks Willis:

And then it became, like, one of the major changes.

Bryan Cantrill:

Yeah. Interesting.

Brooks Willis:

And we're constantly engaging in that kind of conversation with, basically, every supplier we work with. Like, we're not and I think this is true at kind of any scale of company, assuming you have good suppliers. Like, Ben's been having Yeah. Tons of back and forth with our molding, our preferred molding vendor or one of our preferred molding vendors about all these injection molded parts he's been doing. They've gotten a lot better through that process.

Brooks Willis:

Like, we can do a pretty good job in a vacuum of looking at the rack and saying, okay. Here's the things we need to change to make it more more more wonderful. But they're the ones really living it. They're the ones who have to touch the stuff every day. So that's one of our most valuable sources of feedback is our suppliers.

Bryan Cantrill:

Well, to and we really want our like, we have got a design that's hard for them to manufacture. We wanna hear about that. We want them to give us that feedback. So that's Brooks, you want talk about this kind of the specific example that Adam's referring to because I think this is a fascinating example of something that is effectively in the product but is not relevant that from like the day the product ships when it got the the by the time the product goes into the crate, this aspect of the product is is no longer needed.

Brooks Willis:

Oh, yeah. Yeah.

Bryan Cantrill:

Yeah. It's very important for manufacturing.

Brooks Willis:

I missed the t up on that one. That's a good thanks for bringing me back to that. So right now I assume you're talking about the welding. So right now, the rack is welded together. And, again, we'll go back to the several massive chunks of metal.

Brooks Willis:

There's a big heavy metal base. There's four big heavy metal verticals. There's a mid shelf. There's a top plate. There's, like, a roof and a bunch of other ancillary stuff.

Brooks Willis:

Right now, our sheet metal supplier has one big, like, endoskeletal contraction that and a bunch of clamps and magnets and stuff, bits and bobs, it's a whole kit, that holds all of those pieces together precisely enough that they can then be welded in place, and that all gets taken off. And that takes somewhere between half a day and a day depending on who's doing it, you know Yeah. Wow. How well it's all going per rack. And, again, if we're talking about making more and more racks and making this process more efficient, a process step that takes four to eight hours of more than one person's time is a really good place to look.

Brooks Willis:

So rack model one has some very simple, modifications, and I don't have any good pictures of this because it's it's not visually interesting, but it's technically very interesting. What we've done is that all of those joints, we've gone in and we've just slapped a couple rivets on there. And this kinda goes back to the engineering analysis end of things. We could completely redesign the rack to not have any welds in it and to be all riveted. But one of the the things about the oxide rack is that it looks very, clean, polished, unified, and you don't really get that with rivets.

Brooks Willis:

It it looks rivets look more tacked together in my experience.

Bryan Cantrill:

So we want to bolted together, you would say?

Brooks Willis:

Exactly. Let's bolted together. And so we we wanna stick to that, like, kind of uni, like, unibody single part. This rack is a thing. It is this is one object, both aesthetically and structurally because it is it's very heavy.

Brooks Willis:

There's a lot of structural analysis that we need to go into changing the welding because it's thing weighs, like, tons of pounds. So what we've done is we've gone and we've added a couple simple rivets at every single interface there. So instead of having to take this rack and, like, jigger it into this whole contraption, they can just line a piece up, bang bang, slap two rivets in, put the next piece on, bang bang, slap two rivets in. And instead of it being several people taking a lot of time to carefully align all of this, that alignment is built into the geometry of the metal itself. And then once those rivets are in, they can just kinda step back and start welding, and they're free to do so.

Brooks Willis:

And then that two is nice because we don't have to go build more fixtures if we wanna build more racks. So you just need another welder. Right. It's much easier to go to I mean, not Harbor Freight, but, like, it's much easier to go to your welding supply store down the street, which they have, don't we all, and just get another welder if you need to add more capacity to your line versus, like, tens of thousands of dollars and and hundreds of hours of engineering time to add another welding station.

Bryan Cantrill:

Well, and I love this because this is a very this is like as things go, this is not a complicated or certainly not a high risk change to the rack. And it and what it ends up but you're you're kinda adding this thing to the rack that then solely for the purpose of it being more readily manufactured and the because it it's the weld ultimately that holds the rack together, not the rivet.

Brooks Willis:

Yeah. The rivets are are barely structural. We we've specifically asked them, like, hey. Are you gonna move the the racks around after you weld them after you rivet them before you weld them? And they were like, no.

Brooks Willis:

No. No. No. We're not gonna do that. Like, okay.

Brooks Willis:

Good. Because it they're not that they're not that strong. Don't do that.

Bryan Cantrill:

Right. This is, not needed for any pothole or for that matter, any movement, really. So yeah.

Brooks Willis:

Exactly. We do not build these in a seismic zone, and that is good. Right.

Bryan Cantrill:

This is like would say this is like a Kia furniture that I've assembled, that seems to be it held on with kind of a a single whatever those little wooden goobers they call. Don't know.

Brooks Willis:

Dallpens. The wooden daulpens.

Bryan Cantrill:

What do they call them? A daul. Oh, daulpens. Yeah. Okay.

Bryan Cantrill:

Yeah.

Brooks Willis:

Yeah. Right. And the but this change too like, we're saying it takes four hours to weld the rack together. Like, this change took maybe twelve hours of engineering time, like, start to finish. Yeah.

Brooks Willis:

Like, started on a Monday morning, and it was done, like, Tuesday afternoon with other stuff happening in the in the meantime. So it's not it's not a complicated change, but it's it's very impactful.

Bryan Cantrill:

It's not a complicated change, but also it's a change that you have to know to make. And you know to make it because we've got this kind of end to end visibility of this thing from from the from the when it is born as sheet metal to all the way to hitting the potholes and all the way into a customer data center.

Elliott Donlon:

Yeah. And there are a lot of people that will say their suppliers are good because they'll do literally anything you ask for. And our suppliers are good because they will do some of that, and they will also ask us for things that'll make their lives easier.

Bryan Cantrill:

Yeah. That sounds great.

Brooks Willis:

And if we ask them to do something too crazy, they'll cock an eyebrow at us and say, no. If you want us to, like, we'll we'll we'll we'll bill you for that. But are you sure?

Bryan Cantrill:

That that means that's well how

Adam Leventhal:

I why I view it through the lens of a general contractor where you want them to say, like, you really don't want to pay for that. Like, that's I hear you asking for that, but I'm gonna do something different.

Bryan Cantrill:

Right. Like, I can technically do that, but you do know. It is not in your economic self interest that I do that exactly. Yeah. This is Wait, I had another contractor with I I had an architect who had a mathematical error and the the the contractor spot is like, I don't understand why we're zigzagging this wall.

Bryan Cantrill:

Like, I can just make this straight. It will be much less concrete and we'll open up square footage in your house. So like, what are we doing here? It like, that was a that was a very that was a very impactful change as it turns out. Open up 300 square feet.

Bryan Cantrill:

That was my my little that's that's where my Cosmo is right now, that little basement lab. Thanks to a contractor that was willing to push back on the architect and get to a better design. Well, is awesome. The I kind of feel like, I mean, we could talk for hours here in terms I think we're gonna we may have to have you all back to do a a second episode here because I feel like there's a lot more we can talk about. But I do want to be mindful.

Bryan Cantrill:

Adam, I know you gotta you you got you got little league to go coach. Is that right?

Adam Leventhal:

Little league. Exactly. Big game.

Bryan Cantrill:

God God. That is so exciting. I the I mean, we are we are we gonna get your your little league team on here for a for an episode? We gotta something it's baseball season. You know?

Bryan Cantrill:

It's exciting.

Adam Leventhal:

Sure. We'll do a live broadcast. I'll do, sound effects for the hits if there are any.

Bryan Cantrill:

You know, that is my that is my dream to do, Adam. I want us to do to do play by play and color commentary for for baseball together. And Hey. Good. I think we should do an Oxide and Friends where you and I, we're we're just gonna be at a ball game doing And stay tuned, everyone.

Bryan Cantrill:

Exactly. Exactly. Make sure you subscribe for that one. Right. This is one of my ideas when Adam is like, are you worried that we have too many subscribers?

Bryan Cantrill:

Is that the problem we're solving for now? Like, we want to There's solutions. Right? It makes sense to me. Let's get rid of those folks.

Bryan Cantrill:

It's been awesome. And Doug Brooks, Elliot Ben, thank you very very much for I think again, we're going have to have you back because we've got a lot more we can talk about mechanically. And there's there is just so much here. I feel like I that must feel the same way. I just did not.

Bryan Cantrill:

I don't look at the rack the same way. I mean, I've I there's so much that about this building this product that looks simple and is just not simple. There's a lot of terrific engineering in it. And thank you all for for all the terrific engineering work that you've done and keeping it safe, you know you know. Safe but not too safe.

Bryan Cantrill:

Right? We we we yeah. You know, we gotta let's not get that safety margin too high.

Brooks Willis:

Thank you for having us on this.

Bryan Cantrill:

This was a

Brooks Willis:

lot of fun.

Bryan Cantrill:

The one we're happy

Brooks Willis:

to come back anytime. We have tons more to talk about. We didn't even touch on any of the the r and d work. This is all all just on the manufacturing end.

Bryan Cantrill:

Oh, just the manufacturing. We should and we should I I I should also I we are expanding the team. So if you are a mechanical engineer with a similar disposition, and yeah, we're not we are not just doing the manufacturing work. We're just that that's been a lot of what we're doing but we've got a lot of exciting work ahead of us as well. Awesome.

Bryan Cantrill:

Alright. Thanks everyone. Adam, go get to that little league team. I'd always send the runner to second. I trust.

Bryan Cantrill:

I I trust you're at that stage where Excellent free base over there. And we will see you all next time.