Bryan and Adam are joined by Doug Wibben and Robert Keith to talk about the mechanical design of the cabled backplane of the Oxide rack that allows for "blind-mated" server sleds--no network and power cables to plug, unplug, and mis-plug! Watch the chapter art for relevant pictures. We've been hosting a live show weekly on Mondays at 5p for about an hour, and recording them all; here is the recording from April 3rd, 2023.
If we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!
Creators & Guests
Host
Adam Leventhal
Host
Bryan Cantrill
What is Oxide and Friends?
Oxide hosts a weekly Discord show where we discuss a wide range of topics: computer history, startups, Oxide hardware bringup, and other topics du jour. These are the recordings in podcast form.
Join us live (usually Mondays at 5pm PT) https://discord.gg/gcQxNHAKCB Subscribe to our calendar: https://sesh.fyi/api/calendar/v2/iMdFbuFRupMwuTiwvXswNU.ics
Speaker 1:
Welcome to some a special guest here. We we've got, Doug Wibben from from Benchmark. Doug, it's great to have you here. Thank you so much for for joining us.
Speaker 2:
Yeah. No problem. Thanks for having me. Appreciate it.
Speaker 1:
So, it it should be I mean, Doug, you've been, with us really from the just about the beginning in terms of our odyssey of building this thing. And the anytime if if folks see the oxide rack and anytime you think of just the gorgeous mechanicals in this rack, it is Doug and his team to to thank for that. They really Doug, you did such a terrific job at I mean, just so many junctures. I was I I don't know we must have been just a little bit bonkers to work with, but really, really appreciate all you've done for us. One thing I was trying to remember is when we were initially talking with you, had we already figured out that we wanted to blind meet the networking, or was that after we had started speaking with you?
Speaker 2:
That was the first 6 3, sorry, 3 months or so, a feasibility study to prove that, it was possible. And then from there, it was the plan of record moving forward.
Speaker 1:
Right. So okay. So we had already we were all but we were already wondering if that was possible by the time we Yes. Because I think that that and that would have been right when we first started talking with you that we because the story here is kind of interesting that we and, Adam, I can't remember if you were on any of those those early calls or not, but we were very much assuming that we were going to have, in terms of the networking of the rack, that we were gonna have a traditional architecture of power and fans in the rear and having networking out the front. And when we were talking to we were kinda surveying the industry and talking to various folks, making various connectors and so on, we were talking to one company in particular, and they were they understood what we wanted to go do.
Speaker 1:
And they were, you know, out of curiosity, why are you not blind meeting the networking? Like, why are you doing this whole thing of the cabling at the front? And there was kind of this moment of, like, I do the hyperscalers. Blindmaiden network. You can do no.
Speaker 1:
No. They don't blindmaiden networking, but that's because they're afraid to blindmaiden networking, which of course is just like catnip to us. It's like, oh, oh, oh, okay. Wait a minute. That's exciting.
Speaker 1:
So when we just were immediately asking the follow-up questions of, like, is it possible to blind mate the networking? Because and by blind mating, we should explain what we mean by that. So this is having the, the networking connection where you can't see it, effectively in the back of the cubby, such that when you slide the sled in, it magically mates with the networking. There is there would be no user visible cabling in this kind of system, which we found really intriguing if it was possible. So, we kinda had a and I think it was at at some time at this point that that kinda Arian had an initial sketch of, based on maybe this is is feasible, Arianne put together a a initial sketch, and I'm gonna drop this photo album, Adam, into the chat if you haven't already.
Speaker 1:
So I have not. Okay. Don't worry. So the, and and it's and someone in the chat asks, are you aware of the open 19 networking? And, yeah, we were aware of open 19, and the open 19 uses this really interesting big connector in the back.
Speaker 1:
And when we talk to people who talk about open 19, this was, like, the thing that was that was most interesting to them about open 19. 19, by the way, not for 2019, but for 19 to track. So we we were aware of that, and we, and and that was that was interesting to us. So this idea of, the and there's Arien's early sketch that that, Adam dropped in. And, boy, if we could actually put together the the the RAC and and blind mate networking, that was terrific, but is it feasible?
Speaker 1:
And, Doug, this is kinda where where your team came in, and, I mean, you you began to inform us that, you know, this is definitely feasible, but here are some of the things that we need to look after when we are designing this thing. So what what were some of the the kind of the problems that you had in mind as we were beginning to design this?
Speaker 2:
Sure. So initially, coming up with a concept that would allow the gimlet to dock into the rack in a manner such that it would align mechanically in x, y, and z with the cabled backplane, pending our vendor selection, that was a difficult task. The oxide rack is primarily sheet metal parts, and sheet metal fabrication is one of the less exact fabrication processes for mechanical items. So in in figuring out how to make that work, we had to go through a lot of mechanical tolerance analysis, to determine how best to approach that.
Speaker 1:
You know, it's occurring me that I'm I'm gonna be getting fears that you had that I never had. I'm actually realizing that I'm gonna come away with this conversation with a whole new set of both appreciation for what what you all have done and, and new fit and fears. So that so you were concerned about, in particular, sheet metal imprecision in terms of getting this thing to to line up properly?
Speaker 2:
Correct. So, you know, between the latching system on the front of the gimlet, which had yet to be designed at the beginning, and the cassette or cartridge at the back of the, cubby. There are something like 15 or 20, mechanical dimensions contributors to the overall tolerance stack, which have to have to be combined and accounted for, to make sure we're making the appropriate connection, at the rear physical connection, at the rear.
Speaker 3:
Doug Doug, I said Gimlet a couple times just for folks who have not listened to the show as, in a detailed fashion over the years. Gimlet is the code name for our server that sits in these cubbies that Brian was alluding to. So, when when you hear Gimlet, that's that's the server that slides in.
Speaker 1:
That's right. And so the concern that I I know and, Doug, we I remember we talked to this early on as well, having having been with backplane systems at various parts of my career, one concern you always have is around bent pins. And then the then and then of course, the the the very worst case of that, the bent pin virus, where you have a something with bent pins that is bending whatever you plug it into. And if you start, start moving that around, you can actually start bending a whole bunch of pins. And Doug, your team did a terrific job in terms of, of engineering, how to prevent someone from bending the pins.
Speaker 1:
Pins. Do you wanna describe how you pulled that off?
Speaker 2:
Sure. So it started with, you know, connector selection. We we assessed a number of, different suppliers and looked at the physical makeup of the connector, and tried to kind of judge what we felt the the capacity for bent pins would be on on any given design. So we moved forward with the design the connector design, which we thought had the lowest potential for bent pins and kind of move forward from there into integrating into the mechanical design. So we use probably more rigid steel mechanical guide pins than we need in aligning the connectors.
Speaker 2:
But, again, given the the concern and and the challenge of correcting, finding and correcting bent pins, we wanted to kind of if we were gonna over design any part of this mechanical design, that would be the one to overdesign. So, yeah, just, I don't know how deep we wanna go from the start.
Speaker 1:
Yeah. Well, well, well, actually, well, I mean, a couple of things that we see there. One, I actually I love your exciting. I god. I mean, we so emphatically agree that, you know, if you're gonna have a system, an aspect of the system that you over engineer, I mean, you gotta figure you're gonna over engineer, under engineer everything.
Speaker 1:
Right? And maybe you're gonna perfectly engineer some things, but basically, you're gonna, solve for problems that you may maybe you will never have. And then, of course, we don't want to, but but not solve for problems we do have. And in terms of over when you describe I mean, because we this has been a very thoroughly solved problem, and boy, is this a good place to thoroughly solve a problem, because the consequences of this are pretty high if you it's pretty expensive, because the connectors are expensive, obviously, the sleds are expensive, and the steel pins are relatively cheap, I I in terms of what this actually added to the bomb cost and so on.
Speaker 2:
Correct. The steel blades are a dollar or 2. So, yeah, in terms of overall bomb cost, it's minimal.
Speaker 1:
And you can I mean, you know, we we, always joke back in the day, we called Adam Adam Leventhal, hardware engineer, for for his various, hardware engineering exploits as a as a software engineer? And, Adam, as I recall, it was, the a key part of Adam Leventhal hardware engineer was the running start.
Speaker 4:
That's right. That's right.
Speaker 3:
And, actually, it started with, with unbending some sheet metal that again, I don't know if you remember this, but an early design for one of the head units of the FishWorks storage product, was such that you couldn't actually insert a PCI card without quite a running start.
Speaker 1:
For this. I do remember this.
Speaker 3:
I mean, I I definitely, Doug's right. Like, that sheet model really gave gave it up quickly. Like, I didn't, you know, didn't really done the heck out of it.
Speaker 1:
And and, Adam, I don't know how much time you spend, moving the slides in and out of again, but it's extremely satisfying with all of It
Speaker 3:
really is. Yeah.
Speaker 1:
The the guide pins. And it doesn't matter. If you take a sled, it really doesn't matter how hard you ram that thing in because you're gonna be caught by this mechanism that Doug and and team designed. Doug, do you wanna speak a little bit to that mechanism? I don't think we'll we'll we'll put a photo of of it.
Speaker 1:
I don't think we've got a photo in the album, but we'll create a photo of of that mechanism. Do you want to just describe a little bit how you did that?
Speaker 2:
Sure. I'll describe it the best I can without a photograph, but the basically, the gimlet, the server latching system on the front, has a rotating handle. And based on, you know, feedback from you, Brian, and and similar experiences from Adam and the like, we wanted to make sure that this running start was something that we had a safeguard for in the mechanical design. So as you're pushing the server sled into the rack, the cubby, it actually bottoms out on a steel, a welded steel pin on the latch before the connectors, you know, less than an inch or so before the, blind mating connectors for power and, the signal in the back of the gimlet are able to connect to the cable backplane in the rack. So if you get, you know, the proverbial running start, you you stop on something before the second stage of insertion, kind of takes place when you lift the handle, then it starts to more gently and with a, kind of decreased mechanical travel, engage and dock the connectors at the rear.
Speaker 2:
So again Yeah. And not completely bulletproof, but we tried to, remove as much of that capacity to occur as possible.
Speaker 1:
Well and I just think I mean, as with so many aspects of the system, but, Doug, this is something that you definitely got from us very quickly is, you know, we've all tried to bring our collective wisdom into the system. And, obviously, you and your team brought a your a lot of your collective wisdom. And I think it it shows when you feel the thing. It's like, okay. Someone has had the same kind of problems that I have had.
Speaker 1:
Someone has had had the person in the lab taking the running start or had to take the running start. So there's a lot of thought has gone into how the this connection is actually made. And could you talk I mean, Doug, I know that there were some of these terms that were very new to me, things like gather, and the way the these things actually the connectors actually mesh. What what are some of the things you've gotta think about as you as you're designing that?
Speaker 2:
Sure. So the the the free end of the the connector on the cable, on the cable backplane in the rack, has a, a steel a pointed steel guide pin on it, and also, you know, used in conjunction with the metal metallic blades that we've got on the floating cartridge. Depending on the angle, of the point of this, pin and the lead in on the receptacle, you've got what the, connector manufacturers publish as gather. I don't remember the exact number for these, X and Max connectors we're using. I believe it was between 23 millimeters of gather.
Speaker 2:
So that means your connector on the back can be misaligned by up to 2 or 3 millimeters, and it will still kind of find its home and align itself for the final kind of docking of the connector. So the connector system itself, the first maybe 1 third of engagement is purely the guide pin mounting or mating with the guide hole on the opposing, nectar pair. And then the last, you know, half or 2 thirds or so is actually the guide, sorry, the the wafers and the blades, and the other electrical interconnect occurring. So when we designed our kind of floating, interconnect system at the back given this is blind mate, we knew fairly early we couldn't rigidly connect, rigidly mount, these cable backplane connectors to the rack to the cubbies or the again, with sheet metal tolerance just stacking up the potential for, you know, misalignment and for things to actually engage was extremely low. So, we designed a floating system so that within the the the actual the the cable cartridge that attaches to the back of the cubby floats mechanically.
Speaker 2:
I believe it's up to 2 millimeters to allow the system to kind of gather itself within this published gather of the connectors themselves. So, I guess in short, our mechanical stack of tolerances had to be below this 2.9, I believe, gather millimeters of gather of the connector in order for us to ensure that we would always make contact.
Speaker 1:
And the and the consequences of getting this wrong, I mean, you obviously have the mechanical consequences in terms of, of things bending or breaking, but then you presumably just have the consequences of just not having enough contact for a for a good signal. Is that is what are some of the consequences of getting this wrong?
Speaker 2:
Correct. So in kind of the, x and y, I'll call it, not the depth of the server, that's where the gather is impactful. So you wanna make sure, vertically and horizontally, the connector is aligned such that you're you're catching it, you're docking it, appropriately. And then basically the, as long as you're within that gather to start, once things start to fully mate together, they align themselves by virtue of the alignment pins and come together in kind of that x and y. The other dimension, I guess I'll call it z that's, you know, basically the depth of the server as you're looking at it from the front.
Speaker 2:
That's a kind of a different metric, in that in that regard. We wanted to make sure the, the connector was fully mated within reason as much as possible. So if these connectors are mismated, basically not plugged all the way in, there is some signal loss, I believe. Right. Yeah.
Speaker 2:
So the the connector, family that we're using has is fairly forgiving in that sense. I believe it's 1 or 2 millimeters allowed to still not have a significant signal loss. However, we wanted to make sure in the connector design, the the connector system design that we always kept this as low as possible. So I think worst case, we could have a half a millimeter, missmate, basically, not fully engaged. And then we've got a a spring system in our connector so that when the connectors crash into each other, the one pair on the rack just floats backwards within the tolerance of the spring.
Speaker 2:
So there's kind of some intentional collision there, by design, which, helps us ensure that we are you know, has made it as closely as possible within that dimension.
Speaker 1:
Yeah. And, you know, someone in the chat said like, it sounds like it's docking a space capsule and it does, I mean, it, it deals like that. You've got a lot of degrees of freedom and you need to be sure that that you've designed for all of these things to not damage one another. And then that's not good enough to merely not damage one another. You actually need to make this really, tight tolerance with respect to to a contact.
Speaker 1:
And just so people know, like, how many connectors are we talking about here? People may be thinking, like, I don't know. How many pins are there? 10? It's like, no.
Speaker 1:
We've got quite a few more.
Speaker 2:
So it's, there are 3 connectors per server, not all of which are used. And each each connector has, 4, let's say, 8 wafers of 4 pairs, differential pairs. So I think we're looking at around a 1000 pairs of IO somewhere in that neighborhood, all of which need to be connected, for the system to work fully work. So Always. Yeah.
Speaker 3:
It's it's I I had no idea the number was that high. Now, Doug, is that is that high? That seems extremely high.
Speaker 1:
Hey. Good. Doug, and you're
Speaker 3:
What if
Speaker 1:
Are we crazy?
Speaker 3:
Mean is, like, what what what is, like, that seems like high for the industry, like a a a bold direction. Is that or or are there are there instances where there are even higher connection counts?
Speaker 2:
That's a great question. I'm not aware of any, but, I I don't know that I have the depth of of experience there.
Speaker 4:
It's pretty dense.
Speaker 1:
Yeah.
Speaker 4:
I mean, x max connectors are you know, if you're looking for a high density blind mateable connector, like, you're not gonna really get other connectors that can do that. I mean, you can get other connectors that are not blind mateable that all have you can can get high density, but for where we're at, like, yeah, this this is pretty dense.
Speaker 1:
It is. And yeah. Because I think it's and I you know, you look at the back of these connectors and, you know, it looks like a hive of of the connection. I mean, there's just a lot, and and then Doug, I assume all, I mean, there's also, you're gonna have a lot of mechanical resistance when you go to to plug that in. And so you also need a way of, like, getting that ins, allowing someone to do it easily, be able to make that that that connection without having to get a running start, without having to to, so there's a there's a lot going on here.
Speaker 2:
There is. So these this connector family, the insertion force is fairly low. I wanna, I don't have a number off the top of my head, but in terms of the overall percent con contribution to the Gimlet, server docking, I think it was less than 50%. The, you know, the mechanical friction of steel on steel is a fair contributor as well. But, the handle has roughly a 3 to 1 mechanical advantage to kind of assist this, to to make the last bit of the docking of the 2 stage docking, a little more friendly for the user, on on the outside end.
Speaker 1:
Yeah. It it it it's yeah. Definitely. And ends up being I mean, I mean, and it's so we obviously need to not just plug and unplug these things all day because these connectors do have a limited lifetime in terms of the number of times you can plug and unplug them, which is a good thing because, otherwise, I might do exactly that. It's so satisfying to plug these things in and and and unplug them.
Speaker 1:
And is yeah. One of the questions folks are asking is like, well, can you just use r j 45 cables? And you really can't at this speed. So this is all, 100 gig, and we actually wanted to be able to go we we one, they actually be able to get not just 28 gig, NRZ, but be able to get the 56 gig PAM4 potentially on the on the backplane. And and maybe this is a good segue to how we actually think of that backplane.
Speaker 1:
And, Robert, do you wanna maybe that intro your favorite design partner in terms of of Samtech?
Speaker 4:
Well, would I ever? Yeah. So, I mean, there's a lot of well, actually, I guess there's not, like, too many different vendors you can go with when you're looking for different types of connectors or interconnect solutions. And particularly when you're going with, like, standardized components, you you kinda get funneled in funneled into maybe, like, 2 or 3 people, really. So but there's, you know, another vendor that does a lot of really specialized stuff, and then a lot of bespoke, they're, like, the specialized but general purpose industry.
Speaker 4:
So, like, the the ASMAX connector, for example, is something that's shared between, like, Amphenol and then Samtech. Amphenol made it, but Samtech uses it. And that's kind of special for what we're doing, but it's pretty general connector. And then they make a bunch of really special stuff that a lot of, say, you would see on FPGA boards, which have, like, mezzanine cards that need, like, 500 pins in this really dense row. And then you come across this company, Samtech, that can, like, let you customize all of these things in all the different ways that you want.
Speaker 4:
And you just build it on their little website, and then they have this little button that says, would you want a sample? And you can click it, and they'll send it to you overnight. And it's amazing. You can do that for anything you want, and they'll just do it. And then you're like, how can these people afford it?
Speaker 4:
And why would they be so kind to me? And then they email you and ask you, hey. I saw you did this in this little, you know, design. Do you need any help? Like, can we can we get our SI team to help you with the model that you need in order to do the breakout on your bull's eye cable that you, like, just randomly ordered and you're might think, yeah.
Speaker 4:
Yeah. Actually, I would like some free help.
Speaker 1:
You know, I would like Isn't
Speaker 4:
that amazing? And they they you know, then you find a vendor, and you're like, you give a shit? Oh, I wanna work with you. And, turns out that they do care, and they will they will send people to your manufacturing plant to help you figure stuff out on the fly. It's pretty great.
Speaker 4:
You know? Like, who are these people? And I've used them before, and there's so many other things that I could talk about that they just, like, totally saved my ass. But the they're amazing. And so we you know, after we're having trouble with another vendor, like, you know, you when think organizations get big and you kinda have difficulties, then they you know, you're not a really big customer, so you kinda run into the problem where maybe they can't make things happen as quickly for you, and they don't wanna devote the time that you know, otherwise, they would.
Speaker 4:
And so I've asked, you know, maybe we should go with Samtech because I've had this experience in the past where they've been so extremely helpful even though, you know, you could be kind of a nobody. You know, we should at least ask. And, so we did. And lo and behold, they, like, deployed in force to help us build this thing that they looked at and said, oh, sure. Yeah.
Speaker 4:
We can do that, and no one else wants to do it. And so, you know, be our pilot. Sure. We'll do everything you that you need. And they helped us build this entire system kind of hand in hand, which was, you know, kind of incredible because they're you know, we're just buying products for them.
Speaker 4:
Right? They were not they are not an investor. They're not, like we we do super well. It looks good for them because we use their stuff, but, like, they have no skin in the game, really. But it didn't matter.
Speaker 4:
They they went well and beyond what they needed to do in order to make us successful. And behold, the backplane integral to the product.
Speaker 1:
Totally. And it's always, you know, it it's a they're a very engineer driven company, and they're they really know how to make engineers happy. And, you know, they with, technical engagement and technical support. But then also, I I mean, I think that for us, you know, and then Doug, I think a part of the reason our relationship with you has been so great is, like, we're looking for fellow engineers who wanna go solve an interesting problem, solve a hard problem, solve solve something that's commercially relevant. And not not everyone has that disposition.
Speaker 1:
You get some folks who are like, why are you doing it this way? It's like, no, because we think this is a better way to do it. And with with Samtech, it was, it's been a it's been a good relationship. And so in particular, and and, RK, you're talking about there, that when we talk about, like, customizing the cable, we did need a custom cable here, because if nothing else, we need to customize the length. And, Doug, I can't remember who coined the term squid cable.
Speaker 1:
I don't know if that would I think it might have been you coined that. I'm not sure who did, but I know we we all latch onto it pretty quickly. Because it does if you look in the photo album, it does look like a squid. And it is, in part because we've got all of these different lengths, because each cubby is a different length away, and this is the kind of thing that would be, you know, when when you're racking stuff on your own, it's a real pain in the butt to get all this stuff right, but we can because we can rack these cubbies together once, we can get all of these cables at exactly the right length. So, Doug, do you want to describe a little bit, like, what the squid cable is and what went into kind of the design of that?
Speaker 2:
Sure. So the the squid cable, I guess, the entire cable backplane is made of 16 +2.18 of these squid cables. 16 plus, just a kind of a single length for the PCIe. Each of these squid cables individually has a, a connection that goes to the sidecar, the network switch, and fans out to 4 of the, 4 smaller connectors for, for each gimlet, each server. So all in all, there are, like I mentioned, 16 cables, in the system and each network switch sidecar needs to talk to each, gimlet server.
Speaker 2:
So in kind of laying out, the kind of the initial 3 d CAD concept, we tried to come up with a way to make these connections without using, 16 unique cable skews, part numbers. So, one of the cat images in in the photo album has kind of a a a view of the the breakdown of of how we split these cables apart. So, basically, left to right are copies on the from viewed from the back, of the rack. And there are 3 kind of unique part numbers or SKUs for the cable squid that we're using. So the thought was we didn't wanna have too many such that if something did happen to get damaged that a, you know, a service personnel would need, to stack, you know, 8 unique cables and bring them and swap them out.
Speaker 2:
So 3 three unique, cable links seem like a an appropriate workable number in the mechanical design. So, there's kind of a short, a medium, and a long. The shortest ones from the sidecar, the network switch connector to the inlet server connector is 1 meter. The medium is 1.25 meters, and the longest is 1.5 meters. So, they're different enough such that when you hold them side by side, you can tell that they're different.
Speaker 2:
But at the same time, there's, you know, enough enough spread there that you could basically cable the whole rack with just 3 unique types. So there there are, a couple of locations where there's a little excess length that we just kind have to hang to the side. But I would say for the most part, we weren't too terribly restricted in choosing, basically 3 kind of unique part number types.
Speaker 1:
That seemed to to have worked out really well. And so as we were putting this thing together, and getting it together, it means that this, I'm trying to recall some of the friction that came when we actually got this thing fully cabled up. But I came together pretty well, but the the I'd know that assembly was definitely a challenge.
Speaker 4:
Do
Speaker 1:
you wanna talk about that at all?
Speaker 2:
Sure. So yeah. Going from, you know, the concept sketch from our end to a a 3 d CAD model, which we did put a fair amount of thought into to try to get a reasonable approximation of what we wanted the lengths of these cables to be for prototype purposes. Again so that we didn't have certainly have a cable too short. For for our first bins given you know our kind of desire for minimal churn.
Speaker 2:
And then from the the 3 d CAD model to physically integrating it into the, prototype assembly, there was a lot of learning opportunity there. So, you you know, everything looks neat and collected in the 3 d CAD model. Obviously, we understand. That's not always going to be the case.
Speaker 1:
It's absolutely right.
Speaker 2:
Be the case. There is a, an image in the chat that's kind of I think I named it spaghetti that's got basically, the prebuild of the cable backplane integration into the rack where there's just all these cables coming off and, you know, to try to discern one from the other was fairly difficult. So that was part of the challenge in, evaluating our early and our next stage prototypes was how to best route these such that they can be repeatedly placed in a consistent manner without damaging them. So, again, with kind of the rat's nest as we put this together for the first time, there were definitely a lot of, points of feedback where we we realized we could do certain things better both in how the cables are bundled and how they are routed and, assembled that led to future design iterations that, that are forthcoming here.
Speaker 1:
And so in terms of the damage, what kind of damage are you concerned about with I mean, it's not on the one hand, it's not fiber optics, so you're not worried about the cables snapping. But on the other hand, like, bend radius is an issue, and you've got and what what are what are some of the the kind of the points of damage you're concerned about?
Speaker 2:
Yeah. So so bend radius is is actually fairly forgiving on this cable type, with within reason. So, these are twin axial differential pair cables. So they it's, you know, Samtech says there's a good bend and a bad bend. Basically, if you bend the cable such that both of the wires inside of the both of the pair wires of the pair, bend the same, it's it's it's a good bend.
Speaker 2:
And if you're bending them kind of against deep themselves, that's a bad bend. So, we we made sure as much as possible in the mechanical design that we were routing these cables with good bends. Now that wasn't completely possible without a few twists and turns. But, to the best of our ability, we routed these cables in the design with the best bends we could. So, you know, the minimum bed radius on the bundle wasn't terrible.
Speaker 2:
Individually where these, cables, these twin axial cables exit the connector, in between where the connector ends and the kind of bulk jacket, the black jacket starts, there has to be a gap basically where just this Teflon coated blue wire is exposed. And this Teflon coating is very thin and easy to mar. So early on in the mechanical design we had a few, metal on Teflon contact points in the design that we had to eliminate and replace with plastic in the DVT build designs. And that that drove a few design changes. And then, more recently, we've, with the crew out, I don't know, a month or so ago, we found some better ways to better kind of bundle and collect, these individual cables so that there was less opportunity for this this rat's nest, the spaghetti that we're that we see in some of these images where, 4 cables from 1 squid just really commingle with 4 cables from another squid.
Speaker 2:
So better bundling techniques, more kind of defined assembly methods, and all of this kind of leads to reduced potential for damage at manufacturing production and certainly I believe, the more consistent and repeatable the location these cables are to each other, the more consistent and repeatable our electoral performance with regard to EMC should be as well.
Speaker 1:
Totally. It's alright. So I wanna I wanna actually go into this that this issue that you all resolved recently because it was, you know, we we've had a couple of issues that have been harrowing. And, you know, one of them certainly was in these early DVT racks. And in fact, the rack that was and RK, you're gonna remind me because I in my recollection was, this is the the it was the rack that was going to go to compliance, and we were seeing pretty disconcertingly high failure of the and these cables won't didn't seem to fail transiently.
Speaker 1:
It seems that they were by the time we because, you know, we're kinda we're very much kinda putting this system together. We're, you know, we're building the system as we're in our DBT builds. And so a a bunch of the validation that ultimately that is gonna be done or is being done, there in Minnesota, Doug, was being done, in in out here in Emeryville. So this thing had already been kind of shipped out and we had discovered, I'm trying to remember you and Arian had discovered that basically that a whole subset of cables needed to be swapped out. And, as I recall, that was a, a late night brain transplant right before compliance.
Speaker 1:
Am I remembering that correctly?
Speaker 4:
Yeah. That was not cool. But but we did it because you gotta. But, you know, you're gonna go to EMC. You want all your sleds to turn on.
Speaker 4:
The, we'll have it or have network or be able to have everything. But, I mean, this is the nervous system of the whole rack. Like, you lose a cable going somewhere, and you've you've lost those sleds. It like and you don't really know why, because at the time, we didn't have the infrastructure to, like, test these fibers, on their own. But so what we had to do is, you know, you identify which sleds don't, you know, come up, and we can put on sidecar, you know, we can actually we can get pretty detailed on, you know, what pairs are not working.
Speaker 4:
And so that's helpful in the you know, and and if you're when you're running at a 100 g, it's actually kind of difficult to tell, you know, exactly why it's not working, because it doesn't necessarily mean there's no continuity. It you know, you could have, like, poor you could have degradation in the cable due to, like, the fact that you're bending it up a bunch, and that would obviously cause a really high speed link just to fail. So you don't know like, you could you could, you know, do a continuity continuity test, and it would be fine, but that actually doesn't tell you why it's, you know, if if it's broken or not or which cable it is. So you have to go into the sidecar and have it been probe, you know, which diff pairs are failing. And then from that, we had to map that back out to our handy, like, cable or pair map that Samtech did, which is, like, this 3 page Excel spreadsheet of, like, how do you get to a from a single diff pair through to where it goes onto what cable and then to what sled to it maps all the way through, which is again, they didn't have to help us through all that, but I know Ariane and Jonathan sat there, I think, for quite some time and got that all figured out because, you know, bless them.
Speaker 4:
And, so we we did that. We figured out which, you know, squids had to come out, so we had to cut the entire backplane out because at the time, you couldn't just pull one bundle out, all at once. You it it was all kinda wrapped together, and you can see in some of the pictures where it looks like this weird nest of stuff that just kinda, like, goes into the center and splits out into a bunch of different directions. And that's the old way it was. And that in in its disorganization, that's kind of where these problems arise.
Speaker 4:
But, you know, you see these zip ties that are, like, cut holding us all together. You have to cut everything out. You so you take the whole thing apart just to replace one piece, which not cool, but so we did that.
Speaker 1:
Well, there there was this moment where so we were here in the office together. It was you, Ariane, and me, and you and Ariane now know exactly what needs to be replaced, but we are running out of racks to actually like, we don't actually have the necessary cabling. And then someone was like, wait a minute. We have what we've been calling the showcase rack, which Doug, this is one of the early, early racks that you all put together that it, has a cable backplane, but, not necessarily much else. So we, and we realized that we should have the cabling from that showcase rack.
Speaker 1:
And, I, as I recall, I mean, the, I think you guys, that's where you ended up getting some of the cabling from, right, for compliance?
Speaker 4:
Yeah. So, you know, we kinda identified that, like, well, this is the same gauge, and it has all of its cables. And we haven't done anything to it, so they should all be okay, you know, assuming we didn't break it the same way we did the other one. Like, well, only one way to find out.
Speaker 1:
So Under the knife.
Speaker 4:
Yeah. I got, you know, got the multi tool out and sat down on bottom of the rack. I think at that point, it was, like, 11 or something PM, and Ari and I were like, well, I guess it's beer time. And then just start cutting the thing up. We're like, okay.
Speaker 4:
Pulling the cables out. Go to the other rack. Like, cut that whole thing down because, you know, you gotta do it twice over. And, while we're at it, we might as well organize the back of that one so that we can shut the doors, and it'll all, you know, look good. And, hopefully, that'll I mean, it would help with how things radiate.
Speaker 4:
The so we cut all that up too, pulled the old ones that we knew were damaged out, put the new ones back in, rewired, like, re zip tied the whole thing, you know, fired it up, and it got.
Speaker 1:
It it just it worked.
Speaker 4:
I mean wanna do that twice.
Speaker 1:
Yeah. Right. Exactly. You know what? This is the the end.
Speaker 1:
You know, the because this is always this challenge of, like, you're trying to do everything all at once. And, it's hard when you it I I'm sure you were very relieved. I mean, I guess it feels that it could have been just as plausible to be like, and now nothing works. But, it was, we were able to get this. So that was the rack that need to go to compliance.
Speaker 1:
And this is, you know, several months ago, we had talked about all that when we regaled from compliance. But so we but we had this issue of why did we have this high dropout rate? And, RK, that's what you went back to Minnesota to work with Doug and Jonathan from Samtech. And, Doug, I gather some member of your members of your team as well. With what can we our hunch and, Doug, I think it was your hunch as well is, like, okay.
Speaker 1:
This is something that we're doing as we are assembling this is is resulting in some damage. Was that the hypothesis going in? Yeah.
Speaker 2:
I believe so. So they, you know, like I mentioned, the the design itself, I I I didn't believe had any issues that were nominally causing these these problems that we're seeing. But in the integration of the assembly, there are plenty of other opportunities to, damage these connectors, these cables, which which believe we believe was the source of the issue here.
Speaker 4:
And it's tough to root cause that kind of stuff too because if you're going through assembly and you have different if if different people trying to do different like, install different components, you get into the realm of random. You know? So it's handling is such a strange way to like, you in order to, like, narrow that down, it gets pretty difficult unless you have a very specific install process. And on these, not really at the time.
Speaker 1:
Well, and when you're when you're when you're building something like this, you're not just designing the thing. You're also designing the way you build the thing. I mean, it's, like, very much part of the development process is how you manufacture and how you assemble it. And, I mean, it's this is, there's a lot that needs to be kind of debugged and understood here, trying to understand where these things are are coming from. So Doug, it seems like the the kind of the first order of business, at least from my perspective, the first order of business was understanding how are these things damaged.
Speaker 1:
And, Semtech was able to turn around pretty quickly and tell us what exactly happened.
Speaker 2:
Yeah. So I I believe they were able to look at our our assembly process and kind of, determine where the issues were coming from. So stepping back, it's, just a little bit. The showcase rack, I I physically built the cable backplane on. I had every intent of building the cable backplane on these other 2 design these other 2 racks as well and became ill.
Speaker 2:
So somebody else had to do it the elephant hour. So we were able to kind of get some operator variability, which is absolutely a part of every manufacturing process. Totally.
Speaker 1:
Yeah. I think it's a positive. Yeah. Exactly.
Speaker 2:
To expose these issues, which, thankfully, we're able to be corrected. But, if there's one thing to gain from it, it's that. So the, yeah so we basically walked through with Samtech on-site the assembly procedure that we had. And I guess it was determined there were a couple of kind of key points within the assembly that needed some attention. So nearest the connectors on the sidecar, the network switches, there was a fairly tight bend where the far sorry, rather a a tight gathering, of these jacketless kind of the exposed pefalon coated differential pairs, where there was some marring happening.
Speaker 2:
So that that's kind of an action for, future consideration both in the design and the assembly procedure to ensure that doesn't happen. And kind of taking a step further here and it was it was both due to kind of Samtech's assessment and and RFK's kind of, mention of the pain of replacing these that that we determined that there were some different different, bundling configurations that we could put forth that would better keep, like, cables together, keep them from kind of going astray, being potentially damaged elsewhere, and, kind of staying together as as a kind of a cohesive bundle. So there was lower potential of damage.
Speaker 1:
And so then can you tell me about the the this thing that you've 3 d printed on-site to help you assemble this. I'd I'd love to understand that. We got a the the slide that's got kind of that that green, the green jig at the bottom. Could you describe that, Doug?
Speaker 2:
Yeah. So there's there's a couple of things there. So, we we tried to kind of quick turn 3 d print in house as many manufacturing aids or tools or fixers as we could. Both to kind of assist in keeping operators hands free and for kind of, error proofing, the potential for, in the assembly. So these these cables are are keyed.
Speaker 2:
They're with small plastic tabs that should prevent you from putting them in upside down but they're not very pronounced. So if, you know, if you're not careful, you can put these, connectors in upside down into this cartridge and I've done it myself a couple of times. Luckily, we're only early on prototypes. The failure is catastrophic. The guide pin of the, the cable completely destroys the connector on the server sled.
Speaker 2:
The the Oh. Signal wafers on the server sled. So that's when we found out early we want to prevent. So, that that's, that was all where again, it was we found it out on some just some rigid copper clad boards that were purely mechanical purposes, nonfunctional. But it it's not just It's
Speaker 3:
just a steel spike effective, like like, hammering a nail into the sensitive connector. Is that effectively what happens?
Speaker 2:
That's effectively that that's exactly what happens. Yes.
Speaker 1:
That must have been an eye opener. I was like, okay.
Speaker 3:
Pull that out. You're like, I think I know what happened here.
Speaker 1:
Yeah.
Speaker 4:
Well, if you're ever in the office, just, like, go go to the back of one of the gimlets
Speaker 1:
that we're not using anymore, and you
Speaker 4:
can just, like, press in the the the tiny, like they're, like, feathered they're feathers of gold little tabs that they, you know, contact with each other. You can just touch them and be like, oh, those are, like, quite light and easy to move. And then just jam a screwdriver into the back of it. That's right. You know?
Speaker 4:
Think think about, you know
Speaker 1:
Yeah. That that was very high consequence. Interesting.
Speaker 2:
3 to 1 mechanical advantage on the lever is is handy for for
Speaker 1:
for click
Speaker 2:
use, but it and you almost don't even notice that you destroyed the thing when when it's incorrect. So yeah. Right. Okay. So you
Speaker 1:
definitely wanna avoid that problem. Yeah.
Speaker 2:
Right. That's a
Speaker 1:
good one. Yeah.
Speaker 2:
So as many fixtures as we can come up with to prevent, incorrect assembly, we we tried to implement for for our build here. And again, to arrange things on a bench in a kind of consistent and repeatable method so that the end isn't hanging off of a table or anything else that could put undue stress on these connector systems, you know we we try to implement as much as possible here. So again these are kind of early prototype fixtures and nests but they'll kind of feed into more long term production type procedures and tooling.
Speaker 1:
Well, and I, and I love the use too of, of 3 d printing. Cause, and Doug, this is something that you, that you did a lot. I mean, there are a bunch of things that you were 3 d printing in house for us. Think about the air shroud around the the m dot 2 connectors that you all designed. I mean, it is and I'm sure actually a bunch of stuff I'm unaware of.
Speaker 1:
But, this is something that is a tool that is clearly been, I imagine transformative from a a mechanical engineer's arsenal.
Speaker 2:
Oh, absolutely. Yeah. So, you know, to be able to come up with a concept and turn it around and have it the next day is huge. I mean, we we had our 3 d printers running almost constantly, to try to, you know, iterate and learn from each one. So this particular green fixture that we used for, the assembly here, Brandon, one of my counterparts here designed that.
Speaker 2:
And, you know, it's one of those things where we designed it in like how it was and we had a new one designed and printed the next day. So it's it's just, yeah, game changing for turnaround time and and kind of cycling of of these types of designs. So the, in the, you know, the end product, there's, you know, a few pieces that are 3 d printed. You know it certainly has its place in in the the product design. But in terms of early prototypes, prototype evaluation, tooling nests, that sort of thing where normally we would have to machine that out of a piece of delrin or something similar.
Speaker 2:
We can learn a lot, from these 3 d printed pieces. And I know the AccuSite team embraces that as well. You've got a few of your members there that have their own printers that make their own tooling as well. And it's it's widespread. It's we use it well to our advantage.
Speaker 1:
Yeah. And I just think it's like, you know, one of the things that we've definitely said before here is that, you know, people like to dismiss hardware as hard. Hardware is hard, but it's also easier than it ever has been. And this ability to take these kind of quick turns from ideation to something in physically in hand that you can iterate on. And then especially for a fixture like this, that's like, this is like a fixture that you're using to assemble the product.
Speaker 1:
So, you know, this is something that the quantities are low enough. I mean, this is presumably something, you know, presumably not gonna 3 d print it, for the what we're gonna use on the line, but you could. I mean, it would be, certainly, great to be able to spend quickly on this. And then so any because you had Samtech on-site with Jonathan was on-site, and, RK, you were there. And this was, it sounds like it was really helpful, I gather, to have everyone kind of there at once physically doing this together because, yielded what sounds like a much cleaner assembly.
Speaker 4:
Yeah. Well, what it was really good to have everybody kinda see the same problem in real time, you know, and then particularly Jonathan initially describing kind of like, oh, these are the this is the kind of behavior and bending that we don't want to see, and then what what what is also acceptable and then kind of how the fibers float inside the Teflon, along with the dielectric. So he was he was clarifying in that he identified a particular type of bin that we we are doing or we're doing, pretty explicitly as it routes outside out of the sidecar, so, like, the center of those photos, you'll see the largest connector where it kind of funnels out into the center, and then it goes up or down. That's a a 90 degree bend with a twist and then a 90 degree bend. And what that'll do is that'll cause, like, an uneven push and pull on the connector side in that little cartridge.
Speaker 4:
And so it's, like, that's a bin that you don't want, or versus, like, you can have, like, the cables if they were allowed to float, they could kind of bend together, in such a way that they wouldn't be pushing and pulling against each other. And so, you know, that that's an okay thing to do. And when we got in there, we kinda looked at how these things were taped up and, you know, how there was a lot of math added by this, sheathing that has that's it's the black sheathing that goes around these cables. You kinda see how they pulled. Like, if you remove the tape and then added the sheathing at a particular distance depending on the size of the cable in order to decrease the mass, how you are able to, you know, bend these things more safely and allow them to float in such a way that they don't, they're not affixed at one end when you try to bend it, which would be affick when it's affixed to that tape that you'll see, like, at the tips of the cables, they it's all bonded together.
Speaker 4:
We got rid of that. You create a and I'm not a mechanical engineer. There is a there is a word for this. The point where it can no longer move and the rest of it moves, and it causes uneven forces on inside and outside the trans. I don't know, but there's something to that.
Speaker 4:
Anyway, we were able to see that problem and run an experiment. We're like, well, let's cut some, the sheathing of the lengths that we want, remove the tape, and allow and see how much mass we can cut down. And, after we all kinda got together and, you know, did the process of painstakingly removing all that tape I mean, oh my god. My fingers. But, yeah, then we got to the one of the pictures where you'll see it's just these kind of 4 tubes that come out instead of this nest, and they funnel very neatly, up and down the sides of the rack, and you there's a serious decrease in amount of mass that we have to deal with, and it makes the assembly and install way easier.
Speaker 4:
There's only, like, 3 or 4 different, cables, I think, total. And, I mean, I mean, this is just visible. You could you could look at it. It's so clear how much better that is, and the install is gonna be repeatable. And you can do it in under 2 hours versus took us probably, like, 3 to do 1 in the office.
Speaker 1:
Right. And and you presume we don't need to be Doug to do it.
Speaker 4:
Right. Doug, you can you can have other people do it, and so you don't have to hurt yourself.
Speaker 1:
Right. I know. No. I'm not gonna say I'm not gonna put I'm throwing my head into the ring or Adam's. No offense, Adam.
Speaker 1:
I just feel like actually, no, Adam. You you're you're much more reliable and mechanical than I
Speaker 3:
am than I am.
Speaker 1:
I I just I we're we're we're not
Speaker 3:
You're right. You're not the I'd probably take some shortcut. It's fair.
Speaker 1:
Not quite at that point, but it's it is really important that you get that. And it and this this cable backplane because, you know, someone asks, like, well, you know, what do you do when this kind of, you know, wears out or, you know, how does this fail on the field? This is really designed to not fail. This is actually designed to be with the the lifetime of the rack. So we wanna be sure that any failure that we that that we might have an assembly that we find on-site, that this is not by the time this leaves, so we've got something that's gonna be really, really, really robust.
Speaker 1:
It's not impossible to replace. It's just and, RFK, you you all made it. If one needed to do a field replacement, it would now be potentially possible. But this is not a field replaceable unit. This is really designed to be, to to go the lifetime of the rack.
Speaker 1:
So it's really important that this thing is actually designed for the the long haul, and this is something that is, like, a we really wanted to invest to get exactly right. We got a that final image there shows the the the rack after you all had recabled it. And, it looks it looks glorious. And so, Doug, this must have been, I mean, satisfying to get that, to get all of this kind of root caused and and figured out. I know, from our case perspective, it was just glorious to get this this all this nailed and to kinda pull together folks from your team and folks from Samtech and their team and and work together to get this thing debugged.
Speaker 2:
Yeah. Absolutely. To have so many different, you know, parties and perspectives from our design engineering group, our manufacturing group, oxide service representation, and, the cable manufacturer, was great. I mean, we we really, I think, hit this on all fronts, and and the result moving forward should be a much more kind of consistent and repeatable, product integration of this cable backplane.
Speaker 1:
That is absolutely awesome. And I think it, you know, just highlights that when you're doing something like this, and something I know we have really emphasized, is finding folks that are that wanna build this with us, who really share our perspective. And, Doug, you and your team at Benchmark certainly do, and the folks at Samtech certainly do, and it's been, you you really see the results here when because when something doesn't, you know, when when something needs to be kind of figured out, you really need everyone together to figure it out as a group. And, this is just a lot of fun to see. And on and and just could not be on something more important in terms of this cable backplane because this thing, again, we live or die on this thing as do our customers.
Speaker 1:
So, well, this has been, this has been great. I know that, it should be said that, you know, Adam is from Connecticut And, you've, you know, normally, we think of Connecticut as being merely New York or Boston, depending on which part of Connecticut you're referring to. But, actually, tonight, Adam, Connecticut stands on its own two feet as your Huskies are in the NCAA finals for basketball. Yes. I I hope you put a lot of money on them being in the finals.
Speaker 3:
I didn't. But usually, I ended it ends in a heartbreak because I always pick them to win, you know, every tournament, and, it only happens occasionally, but it does happen occasionally.
Speaker 1:
So, you know, I actually went with my, my my sister puts together a larger bracket with with the extended family. So, of course, the stakes are extremely high because I actually need to make sure that I beat my sister or or failing that, that one of my children does. I really just, we just can't have my sister winning a broader family pool. Not that I'm competitive at all. And I have a ton I did my all oxide bracket with, with with I I took the schools that I and I this is kinda what I've done every year.
Speaker 1:
The I I let's just say that there's a Texas Tech alum among us that, I think Texas Tech didn't even make a turn of it this year or I would've but, That's okay. Well, Texas Tech did not have the opportunity to disappoint me the way, we we have many Purdue alums here at at Oxide, and, you know, not to add insult to injury, but let's just say go Huskies is what I'm really trying to say.
Speaker 3:
Go Huskies.
Speaker 1:
Right. I and and the by the way, that family pool is being won by my, my brother in law's nephew. This 6 year old has picked Yukon to win. So this this 6 year old is gonna walk away with a I don't know if this is some, like, March Madness child prodigy. I'm not sure what's going on.
Speaker 1:
But Or
Speaker 3:
if it's, like, the octopus that was picking the World Cup winners, either one. It may.
Speaker 1:
I think it may be the the World Cup winning picking octopus. Yeah. I I'm going with the octopus theory, or I like dogs is the other theory that I was going with.
Speaker 3:
I mean, you know That's right. A lot of dog mascots.
Speaker 1:
A lot of dog mascots, and I'm I'm going with that one. But regardless, we wanna make sure that you get to your the Huskies in the finals here. But, Doug, RFK, thank you so much. This has just been so much fun to to to watch, Doug. It's been so great to work with with you and team.
Speaker 1:
And, again, I hope hope folks get the chance to to feel this thing. And when if when you make physical contact with an oxide rack, and you're thinking to yourself that, boy, someone really thought about the mechanical design here. It is Doug. It is you and your team really did that, and, thank you so much. I hope you're you're proud of what you built here because it's it's it's really fun to watch it all come together.
Speaker 2:
Oh, absolutely. It's been great. Yeah. It's been a pleasure working with your team.
Speaker 1:
Awesome. Alright. Well, I think in if if you don't mind, we may have to have you back on a couple of other mechanical topics because we got a lot there are a lot of mechanical bits in in the rack here, and there's a lot that that in your team. At some point, let's talk about the, the the moment arm crisis. I don't know, Adam, if you guys had ordered this like You know what?
Speaker 3:
I this is another one where I've heard mention of the moment arm crisis, but just like, some of these cabling issues, I had to wait for the episode to find out all the detail.
Speaker 1:
Exactly. Well, we could do an entire episode on the sidecar heat sink. That thing of that the monstrous sidecar heat sink.
Speaker 4:
Oh, lord.
Speaker 1:
But with that that will wait till after the the the Huskies win the national championship. So on that, thank you very much everyone, and we'll, we'll see you next time. Thanks again, Doug.
