Ctrl+Alt+Engineer

This week, we discuss an engineering failure which occurred in this week's class.

Creators and Guests

Producer
Michael Crocco
Engineer and Educator. AI in Engineering innovator.

What is Ctrl+Alt+Engineer?

A podcast for mechanical engineers beginning their studies in Design.

Kathy:

Welcome back to the fourth episode of Ctrl Alt Engineer. I'm Kathy Petkoff. I'm joined today by

Michael:

Michael Crocco. Hi, everybody. Is this gonna be a special episode?

Kathy:

This is gonna be a special episode. But before we sort of get into the specialness of what this episode is, I just wanna sort of take us back to sort of give a bit more context to why it became a special episode. So we've talked a lot about what do you do, how do you do things, your design process, and that sort of thing. I wanted to talk this week about getting busy and getting stuff done. So

Michael:

I've got an example of how I would've been getting busy over the last several months. And everybody in in your class, I guess, will have will have seen that. For those of you not in the class, I've been working on a project to provide students with a means of planning out their semester. It's a bit of a project management tool working in the Google Workspace environment. And we did a lot of testing and a lot of development and lot a of pre checking and a lot of focus groups and a lot of that kind of thing.

Michael:

And then we launched to students and unfortunately none of it worked.

Kathy:

But let's go back a step. So what we're gonna do is sort of before we get into that, but that's that's that's kinda like the punch line is that, yeah, we we launched this thing, and it was it could have been really disastrous. It wasn't because we pivoted, and we're gonna talk you through how we did that. But the way we sort of start projects as engineers is that we we do a bunch of okay. What does the problem mean?

Kathy:

What are we actually looking for? That sort of stuff. But it gets to the point where you've you've picked up all these things. You've picked up what does the problem ask me? What am I genuinely looking for?

Kathy:

What do I know? You've drawn a sketch or three or a free body diagram. Please draw a free body diagram. Thought about what kind of formula might fit and that sort of thing. And then you sort of have to get busy.

Kathy:

Yeah. You have to do something. And some days the best thing to do is to start. Am I right?

Michael:

You gotta go to market. Yeah. Like I said, so this is a kind of a big software project given that we had to work within the Google Workspace. Pace. We made that determination early on.

Michael:

We want to do this for several reasons. So it limited our, I guess, what we were working with. So this is a constraint on the system.

Kathy:

We're going

Michael:

to work with Google because everybody's got it. We can't spin up a web server in time, for example, to deploy this in a different way. So we're going to work within those restrictions. Looked at what are the constraints placed on the setup that I did. So I created what's called a service account to distribute all of these sheets to students.

Michael:

Okay, no problem. What are the issues associated with the service account? Does it have restricted rights? These sorts of things. We looked at what were the bounds?

Michael:

I found the explainer document from Google that explained that I could create a thousand API calls every hour. And that was going to be okay because we needed to do basically nineteen eighty eight or thereabouts over about four hours. No problem. And how many scripts could I create in a given timeframe? No problem.

Michael:

We were well within those bounds. So we made a decision and my script took a template file which had scripts attached to it, bound scripts, which are what provide the visibility of the students planning and things like that. It copied for every student, it looked up their details, copied the script, personalized it to that student, and then shared it with that student. And it could do this about one every, I think we got to one every twelve seconds or something like that.

Kathy:

Is not bad.

Michael:

Yeah, it's pretty good. Nine eighty eight students later, we started hearing that nobody's scripts would run. It ended

Kathy:

Oh, did we hear nobody's scripts would run? We doesn't work.

Michael:

Yeah, well, this doesn't work. That's all we got initially. But digging a little bit deeper, it looks like none of the scripts would run. So we launched, right? Before that, we had done testing with you and a number of other academic staff.

Michael:

I had trialed on the order of about 12 sheets, make sure that it was consistent, make sure all the scripts ran, and they did. So we went to market.

Kathy:

And it didn't work.

Michael:

Right, and it didn't work. So what happened? There was an issue with creating projects. So I didn't think of this as individual projects. Right?

Michael:

Here's a terminology difference. And I haven't explained this to you. This is the first time

Kathy:

you've heard. So

Michael:

Google has a definition of a project creation. And every time I was copying the sheet, that was a new project by that one user. What happens is it didn't stop the ability to copy. It did not arrest the process at all. It allowed it to happen, but their countermeasure was that they've red flagged that service account.

Michael:

And so now everything that's created by that service account on a certain date shut down. So even the initial 50 that were under this 50 a day limit that was set, so it's 50 projects a day, even one through number 50 don't work because the flag occurred later and then it flagged those in retrospect.

Kathy:

So it flags the entire

Michael:

Account.

Kathy:

Account as being for that day as being this is this is above and beyond what you're allowed to do with in these bounds. So you came across a restriction sorry,

Michael:

a constraint that you had no idea about. We had some idea that it was there, but it was misinterpreted. Right? So classic engineering problem, right, is misinterpretation of a standard, misinterpretation of specification or a constraint or something like that. So I encountered that.

Michael:

And so, it was an interesting learning for me. I didn't have that interpretation. Now I know what that really means, but we had to run at rate as it's called to really understand. And this, again, relating to mechanical engineering experience, we would develop car prototypes for a long time, and then we would do little production runs and all that kind of stuff and slowly move up to production rate. And then you would do this run at rate trial, at least what we called it, which meant that for a shift you would run at full production rate.

Michael:

You knew you weren't going to keep The design was done, it was ready to go to market. You knew you weren't going to be manufacturing cars that day.

Kathy:

Yeah, Because this is where you pick up all of those little problems and all these little issues. So when the video comes on YouTube, those of you who watch us as, you know, audio visual will notice that I've put an FAA 18 in behind Michael's head, and this is kind of this is just me poking fun at him, which I'm allowed to do. But it brings up even the issue with military vehicles. Like, we would call it an acceptance test plan. So you have to do a test plan of this.

Kathy:

And when the FA 18 was going to be tested, there was a huge running around screaming and yelling because one part of the vehicle and it's an air vehicle, so it's just a vehicle that flies. Sorry. Terminology.

Michael:

It's terminology.

Kathy:

What we found is that the most of the system we were testing fitted in one area of the plane, and one part had to be in the nose, you know, aft, forward. And there's a very different temperature range there. So in the back end of the plane, it's, you know, like it would be even in an f one eleven, but the nose cone runs at 95 degrees Celsius.

Michael:

Right.

Kathy:

So it's super, super hot. And it's like, hang on a second. How do we do that? Why didn't we know about this constraint beforehand? Yeah.

Kathy:

You know, it was the first time we'd heard about it. Yeah. But, again, what we had to do is we had to pivot. And I think this is even what's going on when you're doing a run rate in automotive. It's like, you know you're not gonna sleep, that that shift is not gonna be smooth.

Kathy:

It's not gonna be producing cars or anything. This is fault finding. This is problem finding. And this happens all the time. Have you seen a perfect design ever?

Michael:

No. I don't think so. I mean but if you think of, like it's it's not to discount the process that follows that precedes that, I should say.

Kathy:

Of course not.

Michael:

You're getting 99 however many nines of the issues that away. And essentially, the project that I've launched this week did go off pretty much without a hitch, except for one thing. There were a lot of bugs in that software initially. There were a lot of sort of features that needed to be fixed And we fixed all of those and then went through the thought process and the thought experiment of have we covered everything off?

Kathy:

Because how many scripts did you embed in that particular project itself? Well,

Michael:

I mean, it's four script files, but I mean, the number of functions is dozens.

Kathy:

Yes. So it this is the I I want people, especially those people who are listening who aren't students in the class, to know that this isn't just a Gary, this isn't like you pulled up a a Google Sheet and said, you know, run a few things here and and click a button, and that's the only script that happens. Like, this was a this is a really complicated piece.

Michael:

Yeah. So it's importing information from other sources. It's making inference based on one sort of selections that students are making. It's doing a big visualization layer, which is not very easy in Google Sheets. It's doing another visualization layer.

Michael:

There's just a lot there to make it a usable tool. Essentially, we reproduced a project management tool in Google Sheets with the visualizations that go with it.

Kathy:

And Google Sheets is not very

Michael:

good It's at just not built for that, right?

Kathy:

No, it's

Michael:

not. And so, yeah, we were doing something that we knew was gonna have issues, but we sorted out the issues that we anticipated and then we ran into that. So great learning experience. I know how to get around it. We came up with a countermeasure within about twenty four hours, I guess, start to finish by the time we actually went live with the solution.

Michael:

The solution is not what we wanted to do, where we need a few mouse clicks from the students. And that's important to note that early on in the project, I said, I want no mouse clicks by the students to do the setup. They just get a sheet, they just open it and they just start doing their work. Okay, we've had to relax that. And that's, you know, compromise, design compromise.

Michael:

We got to the point where, look, there's no good way to do this without having to get the student to do some mouse clicks. So we got them to do a 10 step almost. Some of those are checks, but basically a 10 step process. And that was, again, I didn't want to nuke their sheets and redistribute. I could have saved four of those steps if I had nuked their sheets, but we didn't want to do that because we know that it already done work.

Michael:

So again, here we're talking about the compromise. And next semester, when we do the same thing again, there's only going to be five or six steps to do as opposed to the 10 that they had this semester. So look, that's the best of what I can do right now. In the meantime, I'm looking at, can I get a negotiation where we white list with Google that service account to do this specific task? Maybe that's an option.

Michael:

So we might still be able to hit the original target, but we've come up with a countermeasure plan. We had an issue in the market. We've responded, we recovered. We've got a primary design plan to countermeasure it, and we're looking at can we do better than that? And mimics, it doesn't matter what industry you're in, what you're designing, this is the way that engineering ultimately works.

Michael:

And and I've encountered this in in, you know, mechanical industry and and elsewhere, and here I am with software doing virtually the same thing.

Kathy:

Absolutely. And so in in or in defense, we would call it an ECN, an engineering change notice. So because we do notice that these things happen, and we don't sort of go, oh, well, that's you, yeah, that's your lot. You know, you sort of walk away from it. We do offer change and that sort of thing, and it's procedural as well.

Kathy:

So in this situation, we didn't need to have a procedure around it necessarily because, you know, it's it's first off to market effectively. But what you would happen is that if this was a physical product and you were changing that you know, for example, you were changing something that fitted on a a vehicle of any kind, you would you would note it and you would document that that change had happened. You know? And this is why we have recalls, and this is why you know? And it it happens all the time.

Kathy:

You know? If you have a vehicle, you know, for example, I know someone who bought a Ford Ranger, and the recalls on that thing at the moment are

Michael:

It's been a few.

Kathy:

Very, very interesting. Yeah. And there's been a large number of them. Yeah. But this is how people deal with them all the time is that when things go wrong, we fix it.

Michael:

Yeah. So how do we, you know, briefly relate this back to the classroom?

Kathy:

Yeah. So back to the classroom. So if you do have something that you sort of come up against, what do you do? Like, you've gotta sort of think about where can I go from here? This is what we call pivoting.

Kathy:

Ideas are great. This is the other reason why we get you to do some of the things that we do is so that you don't just have the only one or true idea because there isn't one. So we get you to come up and that's why sorry. This is why the previous work is really important because then you've gone through a range of different options so you can pivot more quickly, especially when you're doing a twelve week turnaround.

Michael:

Yeah.

Kathy:

Cause that's really, really short,

Michael:

really Going through the proper design process, really understanding the problem deeply, making models, meaning like the mathematical models or simulation models or prototypes or whatever it is, you're building a depth of knowledge which allows you to pivot and know what needs to be done when the time comes. This is a project that we're discussing that failed earlier this week is something that I built knowledge of over a couple of months, had a really deep understanding of how everything worked. So when it was time to fix problems, I knew where to look. If that had been somebody else's project that they handed to me, we wouldn't have turned around a fix in twenty four hours.

Kathy:

That would have been Yuri's like scrap the project for now, come back to it, Yuri, after you'd then actually done the analysis, and it might've taken you six weeks.

Michael:

Yeah, or if I hadn't thought as deeply as I did about all the different facets, if I hadn't sat with you and some other academics and done a workshop and just built that knowledge around all the moving parts, wouldn't I have been able to pivot even if I had developed it myself. So that deep consideration is really important. Those models and notes and sketches and diagrams and all that kind of stuff, that all is building your knowledge around the product that you're developing. And you should think of it that way. You have to think of it that way.

Michael:

Yeah. The document that's part of the engineering process, part of the design process.

Kathy:

And, like, I keep a notebook. Like, you'll that might be know, for those of you who can watch the video, it might actually be in screen, but I keep a notebook. I actually have two that that run my whole life, and I would die without them. But I make sketches, I make notes, and I I keep track of what I'm doing, and I write down my thoughts, which sounds crazy.

Michael:

It's funny. I I This happened last night. This is not a non sequitur, and this truly did happen last night. I was thinking about something that I had developed about sixteen years ago or seventeen, I couldn't quite place it, I wanted to see what I had done, And I started going through all my notebooks, are terribly ordered at the moment because I've moved and things like that. But I went through like, I don't know what it was, it would have been a dozen notebooks at least in that era.

Michael:

And I found the drawing that I did. I used to draw in an A5 notebook, copy those out to A3 and actually get parts built off them. This is something that I would do just to make things quick, right? Because when I evaluated, was I didn't have a CAD terminal proper. I'd have to go, you know, beg, borrow, or steal somebody's CAD terminal.

Michael:

So I just do it by hand.

Kathy:

Yeah. Was gonna say back in the day, not everybody in

Michael:

the That was only sixteen years ago. Right?

Kathy:

Yeah. Not everyone in the entire organization had access to CAD. Exactly. Which you kinda even I'd forgotten. It's wild.

Michael:

Well, at a 150 to $200,000 a seat. Mean, should everybody have it? Right?

Kathy:

Yeah. A seat.

Michael:

Yeah. Exactly.

Kathy:

Per So Yeah. Per active license. So if you if you didn't need it all the time, you didn't get a seat. You had to borrow a seat.

Michael:

At Honda, used to hand them back and forth between Japan and The United States. So overnight we would hand

Kathy:

the same licenses with an ANSYS license between Australia and Germany when I worked at EDAG.

Michael:

So we're getting off base, but that's okay. That's what we're here for. So I found that old diagram that I'd done and the drawing that I'd issued out of my notebook. And then it was great. It was such a great time for me, honestly.

Michael:

I spent about an hour just flipping through that particular notebook, which was quite rich in that area where I was developing a lot of stuff. Anyway, what's the point? Write your stuff down. Write your notes.

Kathy:

Take notes.

Michael:

Take them seriously, make them neat, make them legible, date them, all that stuff. So whether you're doing a twelve week design one unit or anything else, you know, take that note taking seriously. And if it makes sense to write on paper, as you and I both feel very strongly about, write it on paper.

Kathy:

Yeah. I was gonna say and I find that I find paper is much actually is actually much better Yeah. Just because it's that ability to go backwards and forwards. I find that I lose my electronic notes. So, again, like you, I still have my notebooks.

Michael:

Yeah. Right.

Kathy:

It becomes pride of place. You know? And most of the engineers that I know actually do have these sorts of things in place. So I've said to all of my students, come prepared with pen and paper. I'd prefer pencil, but that's just my own idiom.

Kathy:

Yeah. You're a fountain pen guy.

Michael:

I'm a fountain pen guy. And the other thing I'll note about that same notebook I pulled out, I got given one of those FriXion pens, the pens that are erasable. Those are erasable. They actually use once they rise above a certain heat, and through the first half of the notebook, there was a great big circle of missing, notes everywhere in the middle of the page. And I realized I'd Anyway, sat my coffee on that's why I don't drink coffee anymore.

Kathy:

Boring. So that's our special edition. I think done.

Michael:

I guess we'll you might be listening to we might release two episodes. We we've been releasing episodes about a week behind. We might actually release this in the week that this occurred, hopefully. So you might have two episodes to go through this week.

Kathy:

That might

Michael:

be nice. We'll switch the order. I don't know.

Kathy:

We'll see how

Michael:

we go. It'll be apparent.

Kathy:

It'll be apparent. You'll look. You're not gonna miss out on any content. But that's us. That's Michael and Kathy, Kathy, and and we're we're coming to you from the Allen Finkel Building.

Kathy:

We have an amazing view here out over campus.

Michael:

It might be overexposed. We'll have to see.

Kathy:

We'll see. If you get to see it, that'd be great. Otherwise, I can tell you now we're enjoying just a little bit of sunshine before the end of you know, before autumn really sets in.

Michael:

Seem like summer's over.

Kathy:

Yeah. It does. But thanks so much for joining us, and we look forward to seeing you next week.

Michael:

See you next week, everybody.