Subscribe
Share
Share
Embed
Come join Manufacturing Hub as we kick off an Efficient Engineering Theme!
We'll talk about Agile, Version Control, Documentation, and tools to rapidly deploy solutions.
A special thank you to SIemens for sponsoring this theme and your continued support of the community.
Join Manufacturing Hub at Automate 2023 - Come see us in the Siemens Booth!
Connect with Us
We'll talk about Agile, Version Control, Documentation, and tools to rapidly deploy solutions.
A special thank you to SIemens for sponsoring this theme and your continued support of the community.
Join Manufacturing Hub at Automate 2023 - Come see us in the Siemens Booth!
Connect with Us
#manufacturing #automation #digitaltransformation #engineering
Creators & Guests
What is Manufacturing Hub?
We bring you manufacturing news, insights, discuss opportunities, and cutting edge technologies. Our goal is to inform, educate, and inspire leaders and workers in manufacturing, automation, and related fields.
📍 Hey everyone. Welcome to Manufacturing Hub. Very excited to go kickoff an efficient engineering theme sponsored by Siemens. I'm excited about the number of conversations that we're going to have yeah, over the course of this month and today is going to be no different. We're really lucky to have Matt on and we'll go ahead and introduce Matt in just a moment.
I do want to say if you guys are gonna be at Automate in three weeks, three-ish weeks. Yeah. If you guys are gonna be at Automat in three weeks, Vlad and I. We'll be there in person. I don't know if you heard but Vlad has committed to go leave his office surrounded by all of his PLCs and toys in order to to come out to the world, to to go meet everyone else.
And so we're excited for that. So if you guys are gonna be at automate, go ahead and drop some comments in the chat. We'd love to go see everyone. I know that we are committed to go be in the Siemens booth. For at least one of those days, there is an electric vehicle that allegedly Vlad can go sit in.
And based upon the sounds of it, I think Vlad is gonna try to basically sit in that ev the entirety of the show. And maybe not. He doesn't leave. I don't know. The more we talk about it, the more excited Vlad gets about it. May, maybe the maybe the selfies are sitting next to Vlad or sitting next to Vlad I in this electric vehicle.
So if you guys are gonna go automate, go ahead. Let us know. I think it should be an absolutely fun show. We're gonna have a bunch of good time. Hey, Elliot sh Elliot and Copia are going to be at Automate. Now, if I was really good at this I would like to have booth numbers memorized already.
I do not have booth numbers memorized. We will have to go put that on one of Vlad's spreadsheets as we get into that. But without further ado everyone officially welcome to Manufacturing Hub. I am Dave. This guy down here is Vlad. We are talking all about Efficient Engineering over the course of the month of May.
And today we've got a very special guest, Matt Paulison on here to go ahead and kick us off. Matt, welcome to the show. Thank you for being here. Thanks
for having me.
Thank you so much for taking the time, Matt, I think it's gonna be a great conversation before we dive into the technical topics, which we can certainly cover a lot of.
I wanted to get a little bit more on your background. How did you get started in automation? What it is that you are doing ultimately today, and what was that? Progression
I have a very odd story actually. So I was mechanical engineering in mathematics in to, to pay for school.
Actually. I was working in a pharmacy. I worked in a pharmacy for seven years. In the course of doing that, all the pharmacists were like, Hey, you'd be really good at this. You should go to pharmacy school. So I started over again, started getting all the prerequisites for pre-pharm and all that stuff was applying to pharmacy school when I got offered the job at a wsc, right?
And so I was like do I want to go to school for another six years or do I need to start making money right now and try to start my career? So I made the decision, it was like time maybe to grow up a little bit, and I took the job. And so that was actually 10 years ago, last month. Oh yeah.
So ever since then I've been writing PLC code. So control system code I started off in our tech support group basically. A w c the company I work for is one of the largest Siemens distributors, if not the largest Siemens distributor in the US we have 31 offices, a across kind of the Texas Gulf Coast.
We are an entirely employee-owned company. And where would be considered like a high-tech distributor, right? So we do a lot of package systems and we do a lot of support on PLCs and applications and stuff like that. So I started off doing that, right? And then I went through the entire.
Control system, I guess stack, if we want to call it that. So I know and have a lot of experience with PLCs, hmis networking scada, so every form of Siemens scada, which would be, when CCC Pro, when CCC seven, and even when C O A, which I do quite a bit of as well, and then every flavor of PLC you can think of in instrumentation in mechanical to some degree or the, or another.
So I specialized into that and took on, started taking on a lot more project work, and then I also do a lot of support work for customers when they run into issues that they can't solve that are in intractable ones, they get to throw me out there and I get to be the hero, which is nice.
Here lately though, I've been focusing on. Standardizing and formalizing our process behind control system projects, specifically writing code for PLCs, h hmi, SC A and stuff like that to try to reduce the amount of effort that it takes to do that without sacrificing quality or in most cases, in increasing the quality of the products that we put out.
But
before I, before we dive deeper into that last point, maybe I have two questions maybe on your earlier background. So first of all, the pivot from pharmacy into industrial automation. I want to maybe get your thoughts on how the decision actually went down. Did you have some experience in software?
Was it a conversation
you had? Yeah, obviously. I had three years of school and of engineering school behind me already in a lot of mathematics, right? I also took my actuarial one P, so I have a lot of that stuff. I just couldn't really figure out what I wanted to do, if that makes any sense.
And so I decided it was probably time to just try something and it actually. It's really cool because I fell into something that I actually really love and it's hilarious cuz if I would've seen PLCs and control system stuff when I was doing the engineering school, I probably would've immediately gone towards it.
Yeah. And I guess I personally have the same experience where I didn't know what a PLC was until I got into automation. But I guess maybe to that same point, what was the learning curve like? Were you did you have to take some, kinda, like
a PLC class or rapid? So we spin up people very quickly here.
And actually we're building a lot of infrastructure now to teach people faster. So we have an entire group that's dedicated to getting new engineers from school, training them as rapidly as possible and getting them ready to do work and help our customers out. So when I started, yeah, it was, you get thrown in, I think within two weeks they sent me to, a Siemens training course on programming and got done with that felt, and took the certification course and I was like, oh, this is good.
I feel really good about this. And then immediately got thrown out into a project and learned real quick.
Yeah, no, a absolutely. And I guess to fast forward to today, Matt, just to close the loop on the background side, your, currently your official title is engineering manager, but you work, you implement the strategy, but you're very much a practitioner, right? You still go in the field, you
write code, you Yes.
I wanna paint like a, I write a lot of code and I'm very involved with so all the library creation and stuff is generally me. I have a group under me that I, they do some work with me as well, but generally like the writing of the standards, writing the style guide, creating the libraries, implementing those things.
And then I do a lot of code too. I actually do a lot of project work right now.
No that's really fantastic because a, again, I think there's not enough of, I wanna say innovative thinking in our industry. And I think that all of us have been in situations, and we've discussed this a little bit off stream, where, you know you you have a feeling that it can be done better, because you see it in either an adjacent industry or, you talk to engineers outside of automation and you have ideas that you want to implement, but there's a lot of pushback from the industry telling you you can certainly sit there and work, the standard way.
So I wanted to maybe get your story of how you started thinking a little bit differently and implementing some of the ways, software is doing it, and maybe what was the trigger point for you when it
comes to that. It was really that first big project. I I told you this before and gave you a little bit of feedback on it.
So a project came in a pretty good size on 400 hours, right? That's pretty, pretty good size as far as what we would ever take on. And it was given to a different engineer. It was a conversion from Alan Bradley to Siemens, right? And so they, I guess in naively believed that they could just copy rung p rung, right?
In that this is the quickest way, the shortest distance between two points, right? Is the straight line, which is me copying rung P rung. And so even in the, in, in as fresh as I was like, that's probably not a great idea because if it doesn't work, you're just not gonna have any idea why it doesn't work.
Fast forward to whatever, however many months later when they were out there trying to commission the system and it just wasn't happening, and they were like, Hey, we have to go. We need to go get this done. The customer has a deadline and their customer has a deadline, right? And if we don't get this done, it's gonna be problematic.
It'll damage the relationship with the customer, it'll damage their relationship with their customer, right? So they sent me and another guy out there to go try to help get this done. And again, that's, I had just come back from that training. I was feeling real good, right? Feeling real good about myself.
And I'm like, I know how to do all this. I'm ready, right? Send me out there, put me in the field. And so I go out there and you really quickly learn that it is not that easy, right? In, in, it's like when we talk about this kind of, why are you software projects fail? Idea and maybe fail is too harsh of a word, but.
I'm saying if you go over budget in over time a lot you probably are gonna have problems. So maybe partial failure, like we got it done, but over budget in, on time, but over budget as far as ours are concerned it turns out that writing and reasoning about code is actually really difficult, right?
So you, you have to and there's some thought that people wanna get away from. They're like, we are an expert in this domain, which is using these products and being fluent in the environment to, to configure and program them, right? So that, that's what we're good at as a company. We don't know the domain that the customer's working and not generally, right?
And there's some people that are like, oh, that's the job of an analyst, right? To take their specifications and requirements and distill them in a way that the programmers can go do. The problem with that is, is that, that only flows one direction, right? So the. It, there's no conversational part of it.
What I've learned is regardless, and you would think, or you would hope maybe that you could get away from it, but you have to start learning the application domain as well. So what it turns into is like good programmers are actually kind of knowledge crunchers, right? So they're taking in this, the application domain, they're ingesting that knowledge, they're trying to reason about it, they're trying to model it in their mind, and they're trying to translate it into another language, right?
With PLC code that has its own rules in its own syntax, right? So it turns out that's actually difficult. And then even if you can do all those things structuring and architecting the program in such a way that it's resilient against change is actually a separate skill, right? That you have to train and get good at.
So what happens is you go out there and to answer your question what was the trigger? It was this I go out there, I'm trying to fix things, so I fix one part and I break a part over here, right? Or I fix something. And in, in order to make this change, in support of that change, I must make 10 others, right?
And then the same thing. The guy who's working on the hmi, you're like he can work independently, not really, right? Because he needs some tags on his HMI and he needs to make sure they work. So he's saying, Hey, I need these, can you add them to this DB over here? So I go and add them. I downloaded the plc, and maybe it re initializes stuff that I set points that I was working with, or, and you're like, oh then he goes over there and he like, types it in and he goes, does it show up? And you're like, yes. So it turns out that the right test debug stuff your work climbs geometrically and it takes a lot of time to do what you feel like at the time is very little, especially when you're out there for, 14, 15 hours a day in a hot environment every day for however many days straight to get the thing out the door.
And you get done with something like that. And at the end it's relief cuz you're like, oh I did it. And it's, people are happy we saved the day. And then you get off that, you get back home. And the endorphins wear off a little bit and you go, that this can't be how big projects are done because any, this is not even that in the world of engineering, this is not that big of a project.
And so how would these super projects get done? The shell F L N G or like the Hebron or something. How would any of this stuff get done if this was how it's done? It would be impossible. There's not enough engineering hours in the world to get it done, right? And so once you accept that, once you accept that there must be a better way, and in fact there is one, because the proof of it is these other projects exist and work, right? So once you accept that, then you can start looking for solutions to the problem, right? And it turns out in our arena, or I guess in, in our in control systems, right?
The solutions for those aren't really well known, are well articulated, but they do exist in software engineering. And so that's when I started going down those things and I read 'em, I was reading some of these books, especially about like Robert C. Martin, and I'm like, this sounds exactly like what I'm facing.
Sounds exactly like what I'm facing. And you're like how do I, how did he solve 'em? How did software engineers solve 'em? And you, you start looking at those answers and you're like, I think I can approximate these in PLCs, right? And that's when I went, started going down that direction.
And I did start doing that pretty, pretty early on to some degree or another after after that experience, right? Against the wishes of some people. But I was given some leeway, right? They gave me a long leash because I've learned pretty fast and I'd shown myself to be pretty competent, if that makes sense.
So I was given, I wanna make sure, I guess long, sorry to
interrupt that. Just wanna make sure to capture some of those, I wanna say like parallels, but also nuances, right? Like in PLCs you can't necessarily implement classes, right? But there's ways to create it. Same
inheritance. There's not.
That's right there, there's ways to approximate those things. And I generally do it by, so if we're talking about firewalls against change, so what we're talking about is ultimately is dependency management, right? And so all the design patterns and principles that, that you look up in software engineering are generally related to software architectural dependencies.
So it's dependency management. You're trying to build firewalls against change, across which change are dependencies cannot propagate right in PLCs you can actually do this, like in especially Siemens PLCs, I can do this. So generally what I'll do is I define the interface, right?
And I define the interface to be fairly abstract in order to do this. But I'll share that interface across the blocks that need it. That, that way what's happening is I can have two con, I can have a concrete thing which is, let's say a final control element or something, right? That like that, right?
That's something that is concrete and has to be implemented concretely cuz it actually does work on something real. But its interface can be abstract to the module above it. So what happens is you, instead of the module, depending on a concrete implementation, it depends on the abstraction, right? So they both depend on the abstraction, not a concrete implementation.
And what that does is that really does stop change from propagating up the rest of my code. So if you're writing kind of procedural code where it's this top down thing, what happens is you build this pyramid, right? If any of those bottom pieces of the pyramid change, you see that change up each level, right?
And that's what I was facing in that first project. You're like, man, I changed this and I have to change this, and this. And you're like trying to reason it in your head really quick to see how much time it's gonna take. And it's the same thing with any software project that comes up for PLCs, the customer's Hey, I would like to do this change.
And if you haven't architected it then you're sitting there in your head trying to reason out about how many changes you have to make in support of that change and try to add it up and try to come to some number that makes sense. But at the most, mostly that's a guess, right? And some people, if they have more experience, they're better at guessing right?
As they get older. But what you would really like to do is just make the change 'em one spot and not have to change everything. Yeah. And
I think that PLCs certainly lend themselves a lot better to that type of programming, right? If you look back maybe 20, 30 years, I think it was a lot
more difficult.
They have a lot of tools for that. The user defined data types are one of 'em. Siemens libraries, because you can type inversion things is very powerful. And then, recently, I would say within the last couple of years, they added what they call software units, which really is it.
Is almost like a class. And actually in VA team now they have name spaces, so you can have different name spaces in the plc. So we're getting closer. Like they are, they're adding these features and stuff, and I think they know, ultimately that this is the direction that it's going.
Dave,
What are your thoughts on all of this?
I guess I, I have many thoughts, Matt. Now I super appreciate you, your going and explaining your thought process and how you've come to this. I think that, as I said, off stream, I think that Vlad eye, many of the guests that, that we've had on and many conversations that we've had have basically been how do we go adopt agile or almost any sort of computer software programming skills into what we're doing.
Because at some point, A guy named Matt, or a guy named Dave, or a guy named V Lead doesn't want to go spend 400 hours just go manually to change, I don't know, 50 things across a large plant, but it's still gonna take 10 weeks to go change it because we didn't go think about it upfront up upfront to begin with.
And so I think that it's interesting, like I've come to it, I've come to libraries. I know VLA has, and again, many of the folks that listen to it as I always find that next step, right? So internally, you guys are bought in, we're gonna have libraries, we're gonna have knowledge bases.
We, we are going to architect things in certain ways and use classes and name spaces and all of this in order to go make our and our customer lives better. Yes, it may take a little bit more time on the front end, but the next hundred changes that we're gonna do in the next 20 years is going to make it better, right?
The next 50 times we go roll this out for the customer. Yeah. And
Yeah. Yeah. I'm sorry to interrupt you. Go ahead. Oh,
No. I was gonna say, it's gonna make it better sometimes that's a difficult conversation, right? If we want to go make major changes, sometimes that's a difficult conversation to go have with customers, with the end users of Yep.
It's gonna take a little bit more time and a little bit mo more money up front to do it, but it's going to make everything easier. Have you been successful in going to have that conversation and,
and what did those conversations like? It took some time, and I had to prove that it is better, because you're right.
So the, here's the problem, right? The question you're asking right, is actually a question that's fairly difficult to resolve. So you're, what you're asking is, do you specify first or do you write code first? Now, the, there, there's problems on both sides of that, right? So I can tell you both arguments.
The, one of the problems, one of the biggest problems is that it's actually possible to write code without specifying anything. And so therefore people will do it. So that's, yep, that's one of the problems. The other, on the specification side is, People are like we got specification, but then it changed, right?
And so that, that's true, right? You're, some of it is you're fighting against the limitation of language anyway, so you're trying to interpret what the customer's telling you, right? And then take the requirements that they gave you and make a specification out of them that can then be used to go, write the project.
Now, the problem is language itself has limitations, right? Even though we know what dictionary definitions are and things like that, there's all sorts of, there's connotation in everything else, right? And then there's there's just the fact that the, so I guess what I was gonna say, the, because the requirements can change that doesn't really absolve you from not at least trying, right?
And we all know, right? The most volatile part of any of these projects is scope, right? Scope creep is like the dreaded word. And that's when I talk to people about, Hey, why was this so hard? It's always, that's literally number when I talk to engineers, right? And they're like why is, why was this hard?
Why did it go over whatever? It's always scope creep, right? Scope creeps the number one thing. So m my thinking is, right. And Robert C. Martin also says this essentially, if your design fails because of scope changes, your design was bad. It's one of those things. It sucks, and I had to tell, I had to learn it and swallow my own pride myself. A lot of times when I was writing this stuff, I would write something, I'd be like, this is awesome, right? This is, I feel really good about it. I could tell you I had one that was a it was like a pad site manager, right?
Just, doing alarming for a well pad. And they're like, this is the equipment that's on there and it's not gonna change. Or, I'm like, cool, write it right? And I was like, this is really good. I was really happy with it. And then you become a victim of your own success. They like it so much.
They're like, Hey, we got some other ones we want you to do. Or, and by the way, it's a little bit different, right? And you're like, okay. And so again, naively you do what, when you don't know any better, you save as the project you just did, and you start trying to, tinker with it and modify it to get it to what you want it to be, right?
There's some problems with that though. So the biggest of which is those are now two separate projects. So if I find something wrong in the first one, like systematically wrong, an error, and I correct it, I have to correct it times X copies now, right? There is no mechanism to go and make them consistent again.
So that, that's one of those problems. And then you can't really reuse it. So like your unit of reuse is an entire project, which is silly. So you're dragging a lot of dead weight with it, and then you're wasting time to remove things that were already there. And I get that a lot. People are like, why I do reuse code?
I'm like, but. But do you, so the, whereas if you had built that, if you had built the pieces that you had in that original project into a library, now there is a mechanism to go update all of those. Right Now I have a global library, and in fact, I can use something like TIA openness, or even if you don't want to go to TIA openness, you can use openness descriptor, and I can write a batch file that'll open up every project in that folder.
Open up the global library, update the project library, update the plc. I could do that with one click, right? And it'll go and update every project there in that project directory if I have one. So if they had a hundred projects, the batch file will go and do all of them. You can only do that if you use libraries.
If you don't, you're stuck or I guess there are some other ways around it. If you're using version control, if you use something like Copia, right? Maybe you have a bunch of branches. But again that's very manual. And then if you wanted to rease one, you have to understand, get pretty well to try to rease and try to get the fixes over on this branch and bring them into this other branch.
So there are some other tools to do it, but again, when I was doing this the tools weren't there. So the tools are growing or they're getting better, I should say. Yeah. And
I think that's only gonna continue to to become better and better. And I think that we are to some extent, like reemerging back a little bit into the software industry because we're slowly but surely learning that there have been like better practices that have been used to improve, I wanna say code development.
Yeah.
But yeah we're learning what they learned 40 years ago, which, one, it's a little bit embarrassing, but that's okay. The nice thing about that is though the way has already been paved, so to speak, right? Yes. We don't have to figure these things out ourselves. Somebody has already done it for us.
And that's actually really nice.
And to that point, Matt, like I wanna ask you maybe on the human side, right? And we talked again a little bit about this off stream, but I'm curious, let's say as an automation engineer, I feel like there's not a lot of benchmarking of performance, right?
And so how do you figure out Yeah, if I'm doing like, against like my peers, how do I improve, that time to complete a project is very vaguely
define. So what are your thoughts? Yeah, they don't really, so one the tools aren't in place yet to track those things, right? So people aren't generally doing that.
What they're doing is here's the project. If it makes it under budget and on time, you get a gold star, right? So what they're generally doing and what we're, what we were saying was good beforehand, right? Was does it work? Does it meet the functional specification? If we're being literal, right? And you're like, and they're like, yeah, that's good.
And then you start thinking about, or I start thinking about it, I'm like, meeting the functional spec is the bare minimum. You did, you made it work in the way they said it should work. That's, but there's all these non-functional things like scalability, flexibility, modularity, security, there's all these functional parts that really you want to have, because ultimately, in our old CEO told me this before what we're building generally is sandcastles.
They're ephemeral. The change is inevitable. And that's, you have to come to terms with that. And so what that means is, and what I said before, so I was saying if your program fails or gets unmaintainable because of scope changes are variability, then your design was bad. And I think you have to understand that.
So what you have to design for is flexibility. Ultimately, right? Because even if, even in the best case scenario, you got perfect specifications from your customer, you got it to 'em on time, they're happy with it. Their requirements today may not be their requirements tomorrow. And in a lot of cases we'll get done with something and the customer will realize their requirements didn't meet their needs until they have it in hand using it.
So ultimately I think you have to come to, make peace with that change is gonna happen. And so if you do make peace with that, then you have to start thinking about how am I gonna design stuff in a way that it's resilient against change?
Okay. I guess I would like to jump in on that and I'm sure Vlad has got a thousand questions, but which is like the middle of manufacturing hub.
Bingo. Is Vlad saying, I've got so many questions, Matt. Yeah. But. I want to go back to the, this concept of if you design and build a program in a way that it cannot accept almost infinite amount of change and at
some point places I wouldn't say infinite there within a reason. Okay.
Within reason I guess my question is, whose job should that be to understand what the change looks like? What the scope creep looks like? Is it the programmer writing the code? Is it the end user who is writing the, hopefully pretty good functional spec? Is it a solutions architect who should hopefully come in and have a conversation of where are you, where do you want to go?
Let's go build a program that'll work for you for the next five to 10 years. Who, whose respons responsibility should that
be? You don't necessarily have to think about it that way though so ultimately the way I think about it is I don't design for a certain degree of change, put it that way.
If we're talking about it, what I design for generally, I. Is extremely loose coupling of the modules of my programs to the extent that when I make changes, they are localized in one place and the change doesn't propagate across the rest of the program. So this isn't to say that you can ever write something that's going to be completely resilient to the most, like whenever you write something or whenever you architect something in the first place, right?
There is some assumptions that you make about what's possible, right? Within the design of the application. When somebody comes and asks you something that, that, and this happens to people all the time, right? It depends on how flexible your program is, but at some point, somebody's gonna ask you to change something that violates the original intent of the design, right?
And at that point, there are two choices. You, depending on this depends on how far you are into the design, right? At that point, maybe you, you refactor it in. Make it work, right? And that could take a lot of work if it's something that's already all the way done. So that would be like saying, I'm in my house right now in Houston, and somebody's I would like to put a basement.
You can't really go do that now. House already done. And the other is that you make the change, even though it violates the original intent of the design. And that happens a lot of times. And you convince yourself that I'm only doing it the one time and I'm doing it cuz I'm under pressure and I'll, I'll go back and change it later.
And maybe the first one isn't so bad, but as those violations accumulate, the ability to maintain the project gets lower and eventually it gets to the point where making any change causes too much work and it goes asymptotic. And at that point people are like, oh, we need to rewrite it.
And that does happen. I've seen code that looks like that where it's everyone knows what spaghetti looks like.
Okay. No I appreciate that and I generally agree with what you're saying. I just wanted to dig
into it a little bit more. You're never gonna be perfect, and so I'm very much of the thinking that the enemy of good is perfect, right?
So you're never gonna make something perfect, but we should strive to make things as flexible as possible with the understanding that they're going to change.
Absolutely. And I think within reason, I think it also depends upon what we're programming, right? If we're programming a single plc, we can only do so much on a plc, but if we're looking at a win, C O A or something along those lines you in theory could use that platform to go build anything that you want to go build on it right up into a point, right?
And
that, that's one obviously, but I just mean what you should stay away from doing, let's say, is the worst case scenario, which is I've ridden entirely procedural code with global variables all over the place. With sets and resets of the same global variable in 30 places.
So it becomes hard to localize any error anywhere you should. That's the what you should be away from, right? Even if that works, and this is my point, right? I've seen code like that a lot of times and people are like it works. Yes, but it is impossible to change it, right? And so we can talk about that, right?
What are the symptoms of degradation of the, like dependencies in code? And so again we can lean back on people like Robert C. Martin. And he said it really succinctly there, there's four pieces of that, right? So there's rigidity and I talked about that earlier. How hard is it to make a change to the code?
And how many other changes do I have to make in support of that change, right? So as the code gets more rigid, it takes more and more work to do any change. And that's one of those things from a business perspective, right? When somebody asks you to make a change and it. It begins ballooning in cost.
Maybe they decide not to do it. So what's funny is the software rigidity can lead to business rigidity, right? The business have a good reason to wanna change this machine, but because the cost of doing so in engineering time is so high. They're gonna live with, they're gonna live with what they have, right?
And that a lot of times, right? Where they're like, we just don't touch that anymore. It runs, and if it dies, we'll worry about it thin, right? The other that's related to that would be fragility, right? If I make a change to this piece of the program, how likely is that to break something somewhere else?
And again, that's one of those ones that really scares people and really gets them away from wanting to make changes. If I have something that's running, I don't wanna touch it. If they think that it's fragile, don't make any changes to it, don't touch it, because if you break it, then my machine is down and I'm not making money.
And the other one would be immobility. So I have these pieces and parts in this program. I may wanna use reuse part of them, but in order to do I have to take enormous pieces of the project with me that I don't need for this other project. So what you've written is very immobile.
It's stuck with everything else cuz it's so intertwined in all the other functions of that. And then the last one, which is like the most nebulous would be viscosity. And there's two parts of that. There's like viscosity of the environment, which is when I make change when I want to make a change, how difficult is it to make the change within the environment?
Now thankfully, Siemens is really good about this and they have a really low viscosity environment because the PLCs and the HMI and the SCADA and stuff live in the same software all lives in portal, right? It's a lot more difficult when I have PLC in this software hmi, in this software SCADA in this software, right?
That becomes a lot more arduous. Like I'm export tags and bring 'em over here and make the change here, and then export those tags and get 'em ready to scada right? That's worse. So Siemens has a fairly nice environment to do these things in. And then there's viscosity of the design itself. And that's saying how easy or hard is it to make a change to the program that preserves the original design?
That doesn't violate the design. Because if it's easier to make the change that violates the design than it is to make the change that preserves the intent or the original design, people will make the change that violates it. They'll make the hack change, right? And enough of those hack changes pile up and then you get back to the top three again, if that makes any sense.
And so you don't need to follow up on that and loop back on the human aspect. So I think that we still see a lot of the. I wanna sell like old school ways of programming, right? So if let's I'll give an a concrete example, right? If I'm called to a facility, we have a system, let's say aca SCADA, H M I, PLC that controls, I don't know, like five tanks for example, that have, a pump, a motor in, sure there's some safety circuitry as well.
And they're asking me let's add another tank, right? Like we're adding one more ingredient that's gonna be part of the system. And so obviously the hack approach is gonna be for me to go in there and I'm gonna be copy pasting a bunch of stuff. I'm going to be manually entering in, like one tag at a time.
Hopefully I have a list of, all the ADA's been added and I don't make a single mistake there. But that's, that's a question for maybe the validation step. And so in my mind, obviously you should be modular rising that system, right? Yes. And so I could in theory just copy paste and I think we have the tools, but how do we maybe also not reinforced industry, but help engineers understand that's a better way of doing things versus, just saying we've always done it this way, I'm just gonna be
copy pasting it. It's a complicated question. How do you quantify it's, you quantify that actually. So I've tried this and I've come at it a couple different ways actually to try to, get people along this path.
When I was doing this kind of stuff originally, I would, I figured all this stuff out. I started writing code like this. I started doing a lot more projects and having a lot more happy customers. And then, they would ask for more and more stuff. And so I would try okay, so let me try to pass this knowledge along to people.
And originally what I did was I just, in my mind, I was like, when I started, I want, I would've wanted somebody to just tell me all this stuff, right? Just tell me the best way to do it. And so that's what I did. I wrote it all down. I put it out there and I'm like, this is the best way to do this stuff.
And the retention was not that good. And the reason is because when you don't know the why behind these choices, they look arbitrary, right? And so what I've moved to and what we've moved to as a company is so we believe pretty strongly in problem-based learning, right?
So give them a task and let them try to work their way through it on their own and solve problems. And then on, on my side of things, when I really want to start pushing these ideas to somebody, I actually just have them start, I can give them some base projects and then I start violating their assumptions really quickly, right?
So what I've learned is when you experience the pain of doing it the other way you learn really quickly why you don't do it that way. And it's very, like in your tank example, that's actually the example. I used to do this for a lot of people cuz that's when I was familiar with writing stuff for like tank batteries and things.
I'm like, okay, here's the requirements. I, here's two tanks, they're 20 feet tall and here's the IO that's coming in, and I want you to just turn on a pump. Bang, right? Turn on the pump when it gets to this level and turn it off when it gets to here. And invariably what they'll do is they'll write that code completely procedurally, right?
They'll do what they've been taught most of the time, which is, I'll do the scaling here, I'll do the control here, I'll do the alarming here, right? And then I'm like, okay, good. This is, this works cuz we can test it now. Siemens has a, an add-on call test suite, right? And I can use that to test, and I do use that to test against our style guide.
One to make sure that people are, abiding by this dog guide. And then two, it can do unit testing, right? So you can test their code, make sure it works good. And then, they're like, okay, and then you treat them like a customer, treat them and you're like, Hey, that's really good. I have this other one to do you it's 10 tanks, right?
And by the way, some of 'em are different sizes and then they start thinking about it right in. Takes 'em a lot longer to do because they're doing it all manually, right? So they're like, we'll now have to add, eight more scaling things here, eight more scaling blocks here, and I gotta add eight more control blocks here.
And I have to add eight more alarming blocks here. I gotta go into the HMI and I gotta put all these things on all these bar graphs and I have to manually make the alarms and things like that, right? And they learn really. And then I can go back and I can violate one of their assumptions. Like I let 'em do maybe two or three, or I can let 'em do maybe two or three like that.
And then I'll go, Hey, actually if you have a broken wire, it thinks that the, because they're gonna assume something, right? And I can force a broken wire, and if I force a broken wire and they didn't handle that error right in Siemens and 1200 s, at least you do a broken wire without configuring anything.
It goes to full value 32,767. So their tank would go full, right? And it would run the pump forever. In the real world scenario, it would run the pump dry and burn it up, right? So I'm like, oh, you made a mistake, now you have to go fix it. Across all your, you're one of those things. Every instance, and they're like, and you're like, yeah. And then you say, the tool for this is versioned libraries, right? And that makes it very simple to go, make changes. So I make the change I release a new version or make the changes, release a new version, automatically update, right? But you can only really do that stuff if you start reorganizing and restructuring the program itself.
If I have a tank, why am I, why do I have its code strewn across all over the plc, right? It should really live in one location. And then it makes it very easy instead of dropping in three things and tagging, 20 tags, right? And again, so coming from the pharmacy, doing tons of data entry, I know really well that no matter how good you are, right?
So to see 0.1% error, right? So one in a thousand or something, one of those tags being wrong is gonna hurt you really badly. And because all your code is spread out, there's no way to really localize it, right? So if you wrote the block, to scale your, to do everything for a tank, and you wrote it in one block, if there is a systemic fault, it's in the block, right?
If there's a fault on one of them, it's on the interface of the block. So it becomes way faster to localize error and to make changes and to make updates to all the rest of your code. So it's just better in basically every way, right? I think the, to answer your question, why people don't do it is they don't know that it exists, right?
And mind if I wanna play, they don't ask themselves that question of what would I do different? Are they just accept that's how it is. Does that make sense?
Yeah, no, absolutely. I really like that answer a again, cuz I've witnessed, those projects firsthand. So I cer I can certainly relate to that and I wanna, like maybe play like the other side, right?
So there is yeah, a little bit of an upfront investment to w to set this up and make sure that the infrastructure is right. And ultimately, maybe in a production or a plant environment, you probably need to invest a little bit more into, training your techs or getting them up to speed Yep. On those best practices.
Because I think a lot of the pushback is also my guys don't have the experience or knowledge to be able to deal with those problems. And you need to be able to train them and I think people are usually willing to accept those changes. But I, again I've personally had projects where, there've been ais implemented by a third party and I would be told like, Hey, we need to strip that down to, like bear code because quite simply put, the text, we're not.
Prepared to deal with that. So there is an upfront investment to make things
right. And then there's let's talk about that, I guess a little bit. So we're one, let's address the assumption that it takes more time on the front end work. So what I've found right working and doing this a lot is that upfront time is not time wasted.
It's actually substantially less time than the rework required at the end of the project if you're not doing these things right. And now that I'm tracking time, I can actually see that pretty err readily. So I would say they're saying upfront work and I would say, this is work that you, this is less work than you would do doing rework and all the retesting and stuff that you would have to do if you didn't do it right.
And on top of that, you're getting the gains of being able to use this and reuse it. So you're changing something from like local advantages into global advantages across the organization. So it makes sense to do it that way. Now, on the other point where we're talking about, oh they're afraid or that there's the fear that these more complex blocks, I should say, are not going to be trouble shootable by the maintenance staff.
So we can have a couple conversations about that. So from my point of view and here's the question. I guess we can ask this question. Do you want maintenance staff even in the PLC program? And why? No. And so we're catching the live wire right now, Matt, why did they feel they need to live wire?
Because here's the thing, Siemens has a lot of tools now. So things like prodi and stuff like that, that make it so that I can embed that knowledge inside the block itself, and from the outside on the hmi, they can see what the problem is and I can even link them to like a PDF that tells them the things they can check there without even having to let them go into the PLC program.
So my, the reason people don't do that right now is because the upfront cost of implementing that stuff into the PLC and HMI. Is high, but it's not high if I build it into the block itself. So like I've done that work once I get that advantage over every implementation I use it with in the same thing, like even on the HMI side.
Siemens has a tool called C arc, which automatically generates HMIS off of the POC code. I embed C ARC into my libraries as well. So what happens is I can do something and I just did actually I had a project that was like 400 whatever IO points, basically an entire plant. I did it within like two or three days, right?
And it has 400 screens in, in depth of every single IO point. They can click and make a pop-up screen of any IO point. That's something that's really not doable in any feasible amount of time without these tools. It's I built the PLC program. It was made very modularly with the, software units and things like that.
And software units, by the way, let you contain volatility inside the, so like you can decide the architecture program from a volatility standpoint, things that are likely to change a lot, I can put them in a separate software unit. And what that means is when I make change to that block, it actually doesn't are to that software unit.
It doesn't actually re initialize the other software units. So for working with multiple people or for working with a plant, software units are really good. The reason people don't use them right now is because they enforce the practices I'm talking about rigidly. They're not gonna let you share globals.
They're not gonna let you get tags from one software unit to the other. Unless you define that interface very explicitly. So it, it promotes good standards, but it's a little bit arduous for somebody who's not used to it. For somebody who is, they're very powerful. And then once I had the PLC stuff done, I literally click generate, it generated the entire hmi.
And then I can just move the pieces around to where I want them in. Not only did it generate them, so it created the tags, it generated them, it linked the tags, it linked the popups, it linked the events. That's like, how much work is that across 400 screens? That's not something that's really feasible to do on a normal times frame.
Normally I would quote something of that size like a man month or more probably. And Matt,
I wanted to make a comment cuz I think, maybe your maintenance stake was a bit, how would say take some but look I think like I would actually agree with it, right? Because I think if you look back on software projects, it's the fact that we have to fiddle with software is a symptom rather than it is, I guess because it's poorly implemented like we discussed.
There's somebody didn't put in all the alarms, right? So then you start. Adding like weird alarms, as an example, right? There's obviously a lot of different nuances. Yeah. And then because that alarm now introduces a change that then introduces more changes all over the place, then maintenance is starting to firefight these like little issues that occur on different days Yeah.
At night. So I, I think that, I certainly agree with you that there should be a lot less fiddling with software if it is done right. But if it's not right, then I think that's just the, how to say it, like the practice that we assume to be true. And I think in most factories you will get, a huge log of different changes.
And in worst case and I've seen this there's gonna be somebody from maintenance that comes in on one shift. He makes the tweaks to his standards. Somebody comes in on a different shift, he tweaks either, the software or the set points, the settings, yep. To his kind of standards. And so there's this weird.
How to say, like almost consistent rotation of
culture. It's cultural, it's a cultural problem, is what it really is. And so again, that's why when we're talking about these things, one, one of the first steps, like no matter what, so if you're looking at something like the, if you, are you familiar with the cmm like capability maturity model?
Moving up that, right? So moving from like this ad hoc process where people are just doing things based off problems as they come in the firefighting type stuff. So moving from that to a repeatable defined process, right? So the very first step that you have to take no matter what you're doing, is you have to decide to standardize and formalize some of these things, right?
That's the very first step. And really, I spent a lot of time on the style guide that we have, and people were like, why are you just, why are you so concerned with how people format their tags and how they name things? I'm very concerned about that because when you have multiple people working on the same, Project it becomes extremely important that they speak the same language and the same nomenclature, right?
So we, we didn't really do those things cause we were never forced to originally. If you have only if you give a project to one person and they're the only one doing it there, there's really, it doesn't hurt you if they're not doing any, if they have their own standard that's different from everyone else, it doesn't really hurt you as an organization, right?
If they have their own style of, even if their style is something really obnoxious, like tag underscore one through 30, right? It doesn't really hurt you as an organization until the very end. So like in, in that, we've all seen this at a certain point the only document that corresponds 100% to running code is the code itself.
When that happens, when the code is the basis for maintenance, whenever the person who wrote the code leaves, it becomes very expensive to maintain it, right? Yep. Again, We even on that one-on-one basis, we probably should enforce, standards and style and things like that. But when you try to decide you're like, I'm gonna change how we're doing work and I'm gonna go away from pulling work or from pushing work to people, to having people pull work.
I'm gonna start reducing the batch size of work so I can increase the throughput through the system. When you wanna start answering those questions or when you wanna start doing those things, the first thing you have to do is get people on the same page. And that means adhering to the standards and style guides.
And then I'm again, a big proponent of if I create these things, you have to also validate them because, if you just put 'em out and there's no validation, they don't get done. And again, on, when we're talking about this agile idea in test driven development inside the PLC world which exists by the way, people would tell me, oh, we just do code reviews at the end.
Okay, what's the recourse? If you get to the day before fat, they, the engineer hand you their code, you look over it and it's not what you want. What's the recourse then? None. You tomorrow is the day it has to run so you don't do anything. So instead of that, and instead of having them send me individual pieces of code for me to check over, cuz one even if I had unlimited time to do that, there's still a turnaround time and in the time it takes me to read through it and approve it, I've taken them out of their flow, right?
Because maybe they were doing good work. Now I've, they have to recontextualize and put it all back in their head and, model it back up again so that they can work again. And that's saying that I, what good is my approval on it? I can approve almost anything based off the standard install guide.
Cause I can just look at that. But those things don't really matter that much. It's like the, they do to keep people aligned, but they don't as far as if the code works. So unless I know the project intimately, like what good is my rubber stamp on it? So again, instead I wanna move the quality part of it as close to the work as possible.
And in, in Siemens we could do that with test suite. So I let them run the style guide against their code, the style guide checker. It'll tell them if they're not, if they're missed a comment here or there, right? They go in immediately. So it gives them immediate feedback. They go and fix it, they run it again, it passes.
I have them run against the test suite to do unit test as well so they can run it, see that it passes, and then if it passes the style guide and it passes the unit tests, I can let 'em mark it done, right? And so that way in, they can immediately pull another task and keep going. And actually in inside Siemens now, they just released an add-on that will automatically create the test environment and automatically rig up the block to do the unit test and automatically create a template for the unit test.
So a lot of the work is being done for you already, right? Because that's the pushback, right? I get on test driven development type stuff. They're like we have to do double work cause we have to write the test. And we have, okay. But again the rework is, More time than doing this is what I've found.
And you could be sure, if I make a change, I can run it against the unit tests and make sure that it's not gonna break anything. And that surety is actually really valuable. Absolutely.
One last comment before I let Dave jump in with the Siemens. I guess I, I must admit, Matt, I am familiar with test driven development from the software side, but I've not seen it very much in the automation space.
It's not something that, because it's
like I've never s it's very new. It's very new. So like Siemens just released, I think it was in V 16 that they released this test suite. Okay. And it's still new enough that not many people know it exists. And again, because we are, for better or worse, the industry is documentation light on the code side of things, right?
And I would say rigor light, like not very rigorous on that side of things. They're just not doing it yet. But again it's one of those things that I think if someone were to move in this direction, it's a, I would say if you're doing it the old way and somebody moves to this agile method of doing it where they're doing automated testing they're making very loosely coupled PLC stuff, PLC blocks where they can have people work on it simultaneously, and they're pulling work instead of pushing work and their flow through the system is higher.
They're just gonna take everyone's lunch money is what it turns into. Like the first person who does that is gonna have an enormous competitive advantage against everyone else.
Absolutely. I was gonna say I think that this is, this has been a really interesting series of conversation.
I do wanna continue, but we need to thank some people and Yeah, go ahead. Before we thank Siemens for going and sponsoring this theme again, I do wanna remind everyone listening, and if you guys are listening on podcast forum the first time, Vlad and I will be at Automate coming up May 21st through 24th or thereabouts.
So we're super excited about it. We are gonna go, we're gonna go spend some time in the booth. So we will be at the Siemens booth. I did some looking. It'll be Booth. 36 18 for a good portion of it. If you guys haven't heard, VLA is gonna go sit in an electric vehicle and maybe you'll put a stopwatch on it to see how long you can possibly sit there before things start to start to go numb.
And he gets kicked out by John. We'll also be at the Phoenix Contact booth 62 0 8 for a couple of afternoon segments. And then Copia. We will be there for at least a period of time. And their booth number is easy to remember, which Elliot told me in the chat earlier. Booth 4 44. Speaking of Copia, if you guys wanna listen to more DevOps style conversations.
We had Adam Glu on the podcast all the way back on episode 28. It was a really a. Awesome conversation, which was very much for this show. An introductory to DevOps. So if you guys are gonna be an automate, please feel free to drop a message to Flat. And I there are some more booths.
We, we will go do some Roman around. We've got lots of people to see and say hi to. Speaking of, we do wanna thank Siemens for sponsoring this theme for letting us go hang out in their booth at at automate and for the general sponsorship o of the community. And with that, the potential of automation to eliminate repetitive and low value tasks is quite significant.
The quote unquote engineering tasks are no exception in its report. Jobs lost, jobs gained. What the future of work will mean for job skills and wages. The McKinsey Global Institute found that an eight hour workday being the global norm, the average employee loses up to 60 hours per month to easily automate tasks, which is one third of the time.
Innovative companies like Siemens have been making great strides in topics for interdiscipline interdisciplinary, I can't say that word, interdiscipline. Interdisciplinary collaboration, modularization, automated engineering, openness and validation and testing. If you'd like to unlock efficient engineering workflows, check out the link in the description today.
And again, we wanna thank again, we want to thank Siemens for going and sponsoring all of this. And we look forward to seeing everyone in automate in Detroit, maybe with a few easier words that I'll go change in this ad read as we talk about going forward. But Matt, I guess I want talk about efficient engineering as we get close to to the end of the time, right?
And so I feel like you you have talked about how we can go quickly scale applications. Yeah. I think that we talked about how you can go build all this. You talked a little bit about kind of internal engineering and onboarding. I feel that you guys being a both practitioner, but also a Siemens distributor live in an interesting space, right?
Because you are not only trying to help. End users go build better code, go build better programs. But you're also out there trying to help systems integrators and other practitioners go and understand. Yeah. When you're talking to these group other practitioners I guess a two-part question is one, what is the message that you generally bring to them?
And two, how is that message generally received?
So the messaging is different based off of who I'm talking to, but for integrators, the message is obvious, right? Reduction of engineering time ultimately, right? Increase throughput of work through your system, cuz that makes them money.
So for integrators is very easy. Like these are the tools that you have, things like in introducing them to things like prodi to automatically create the alarms. Things like CVA to automatically generate hmis, right? The building out libraries in making them available to use and how to use them.
All of those tools ultimately come down to making them more money. So it is very easy for them for end users, it really depends. Some end users maintain their own code and some don't. So the ones that maintain their own code, they have their own issues, and some of them are related to what I talked about the copy paste thing before.
So on the updating side, libraries will help them and then on them, for them, a lot of times I would talk about like version control, right? So you mean you, you talked about Copia, right? I use Copia. That's actually what we use for version control. And that's, that becomes very important for some of these customers.
Like I've been the one that had 700 win cc run times. I'm like, like individual run time systems. Yeah. So if you have something like that if that's what your plant looks like, I mean we could ar argue if that's a good implementation or not, but if that's what your plan looks, if that's what your plant looks like, that tool is no longer a nice to have.
It's. I have to have it, it's necessary. Because you need to know what's on you on a machine at any given time. And for us in, for integrators too, these tools are useful as well. So like we talked about breaking the project up into pieces and one of the reasons people don't do that typically right now is because the management layer doesn't exist for it.
And doing it is very cumbersome if you don't have the right tools. So if I'm sharing blocks manually back and forth, like copy paste, or I'm like, they, they archive theirs and give it to me, that's very cumbersome and honors and tedious and error prone on top of that. So you gotta move away from that.
And there are tools now for it, which is nice. We use Copia and I also use Siemens multi-user server. And I found like when I'm doing active development work with multiple people, multi-user is really good. And generally what happens, I get to a certain point with Multiuser where I want to issue a release and I'll export it from Multiuser and then I'll stick it into Copia for version control.
And then on top of that, once that initial one is out there, we bring it into the field, those updates will go into Copia to track them. And actually we use the issue tracking and milestones as well. There. It's like when we're talking about Agile, I've been descriptive for the most part, right?
But we can be, I can be prescriptive on this side of things. It's what do we need to do if this is the direction we want to go the first thing you have to do is just make the work you're doing visible. Cause, because the work we have is really invisible. So what I do and what we have been doing is just put the work on a CanBan board.
So I have, I break the projects down into pieces. I put those pieces on the CanBan board they pull the work from there and it makes it really quickly to see when something gets stuck in the queue. You're like, oh, that's been sitting there. And I, we, I also put work in progress limits as well, right?
So that people can't just pull three different blocks and leave 'em halfway done. I want things getting through the system, so if things get stuck right and I can see you really quickly, if something gets stuck, we can address it as a team rather than leaving them to struggle with it. Which is, ultimately when you have one person doing the work, what happens now is and we're really good cuz we're employee owned, right?
So we're really good at, everyone is willing to help somebody, which is nice, but even though everyone's willing to help, no one is really obligated to do so unless somebody like, oh, above them tells 'em to. So it can feel like for a new engineer that they are on their own right. The this whole thing is theirs and it lives or dies by them. And it's a lot of pressure for people who aren't used to it. And I've seen it burn, I have seen it burn people out. Especially if you go on the first one, you go in what I did was, out there for that amount of time, nothing's working the way you won, the customer's breathing down your neck.
That's a lot of pressure. And some people thrive in that and some people don't. But one of the things that I've, that you have to realize is, the ecosystem isn't very large for us. And, we want to use as many people as possible. So if the thought is, oh, they burnt out, therefore they're not that's the wrong person for this job.
I'm like, no, maybe we should try to make it easier so we can lower the barrier to entry. And that's one of the other things that libraries do. The goal would be right the difference in skills, knowledge, and abilities for a library user versus a library creator.
You wanna maximize that distance or that difference so that they can get all the features and functions in quality out of code, right? But they don't have to be the ones to write it and they don't have to redo the effort as well, cuz that's just, waste essentially. Yeah,
that makes a lot of sense.
Matt, I wanna ask you maybe a lighter question about the industry in general. We talked a lot about us adopting features from Software slash DevOps from maybe 20, 30, 40 years ago. I'm curious, like what's maybe on your wish list or what do you see entering our industry relatively soon that hasn't necessarily made its way to industrial automation?
Are there any things that you're like, Hey I really wish, or you're pushing towards to to having
in the automation space? There are, yeah, there, there's a couple things. So openness is a thing that's been out for a long time but openness, unless you're a really good computer programmer and very knowledgeable about C Sharp is very difficult to use.
So they have some pieces coming out that make some of these things easier. So somatic ax, I think is in beta in Europe. And that's like a, I believe it's based off vs code. It's like a Vs. Code editor for PLCs. I think that's gonna help some, I think there's a tool that Siemens has called the modular application creator, which is like a framework to use openness, building out stuff like that, like in the long run.
So like the goal ultimately for me, if we're talking like long-term goals would be, we spoke at the very beginning where people were saying, if I'm not making executable code, it's, I'm not going anywhere. We can add an addendum to that. If I'm not creating something that can create exec executable code, maybe I'm not going anywhere.
And those things, like what we're saying then versus now, maybe seems like a contradiction, but in some ways it isn't. If you can make tools that will take specification and create code, right? So ultimately what I'm talking about is like meta programming tools and I have some clunky ones that I built myself, right?
But they're very clunky and they're not expandable and they're not, scalable, but, Tools like openness and especially like the modular application creator can really be used to build out a large portion of your projects. Cuz most of these projects, everyone thinks there's everyone wants to say everything they're doing is completely one off, but in reality, they're all built up of some permutation of pieces and parts, right?
It's the same thing, like machines are all built out of some premutation of simple machines, right? So if you can build out the libraries and you may be at the beginning of the project, do a gap analysis to see which, which pieces I don't have. You put those on the board for people to do, right? But then you use the tool to create a lot of the code.
And so if I use that tool to create the PLC code and all my libraries have CVA and pro diag built in, I can generate the rest of it, right? And then maybe the 20% that I need to write that is novel for that particular process. And this would be like the state machines and stuff like that.
Generally the state machines and stuff. I can have those be done by people, but because we've managed all these interfaces well. It becomes really easy to insert them into the rest of the program. So you've taken something that was man months and made it days, right? That's a change that is world ending for some people who are billing massive amounts of time in, but it's really good for some people, right?
Because you could just do a lot of work with few people. And I think that's one of the biggest challenges that we have actually, is getting people into this industry and training them and getting them able to do good work. We spend a lot of time, we spend a lot of time building these training tools to get people up and ready to go, and if we want to grow to a certain extent, you're like what if you, I always ask people when they're writing code, would you do it the same if you had 10 to do, would you do it the same if you had a hundred to do?
Would you do it the same way if you had a thousand to do, right? And those answers can change really rapidly. And it's the same. Yep. Same thing. I think about that from a hiring perspective, would we hire the same, would we train the same if we had 10 people, a hundred people, if we needed a thousand people.
And there's really not any integrators that size and I think part of the reason is because of the way that we do work, it becomes hard to deal with the variation in demand when you're doing work the way we've been doing it. And I do believe that's one of the reasons, like the Big Five or Fang, depending on how you wanna call them, is able to have so many people is because of the way they do work, right?
So we need to shift as an industry probably, right? Especially since it seems like we're onshoring a lot of stuff, right? There's gonna be a lot of work to do. And there's not in, in my opinion, and I think you guys can speak to this, I don't think there's enough automation engineers out there to do the work that's gonna be coming in.
Yes,
I'd say not note it it's probably gonna get a little bit worse before it gets better. For sure. Before I throw in, like there, there's a question we have like in chat from Daniel. I'll say like one thing, like on my wishlist, Madden, maybe you have some input on this as well, but it's having not necessarily a marketplace, but a collection of libraries, right?
Because I think that as automation engineers, and I understand that companies are making it a little bit better internally, but let's say me, Vlad, I go out and I write some kind of a function that works really well for a filling tank. I'm not going to share that. And again, there's, I understand there's some IP and that's a whole other discussion and whether or not, you can patent that code is different.
But I think in software, there's a lot of ways for me to go and look up a best practice or a library on, that component, right? So I don't have to go and reinvent the wheel again and again. And I think there's a lot of really smart people. There's just not a portal. Through which, that's shared.
And I know, like I said, I know there's challenges. I know that there's secrecy, but I think that would be really a great product to see. And I don't think it's,
yeah. So let's address that a little bit there. There's a couple things, right from most of these pieces and parts that I'm talking about, I wouldn't consider them proprietary.
And I actually released the stuff Absolutely. I do as open source anyway, because a lot of these things, like everyone has written a valve blocker, right? That's not, there's nothing new under the sun as far as writing a valve block goes now. Absolutely. And
I agree with you. I agree. I was just saying, there's that, try and put a label on them, so I'm like, I
reluctant.
So some of that, yes. There, there is maybe a little bit of cultural change that needs to happen there around releasing the stuff, open source. And then the bigger one that you addressed is, there's not really a portal right now where you can download those things. You could put them on and I do. So I have a, I can put them on GitHub or whatever and let somebody download them but GitHub doesn't really.
It won't show you the diffs on the binary file. Like portal. Portal. Projects are compressed. Portal projects are binaries. So we weren't really telling you anything, but they could still download them. But you're right I think there exists a need in a demand in the marketplace for an open source repository like GitHub for control system code.
Absolutely. Let me throw this question from Daniel at you, and I know we're coming to time and sorry again. We have so many questions for you, Matt, and certainly appreciate you being here with us. I
have, I'm good today. I'm free. We can go as long as you guys want. So he
wrote you not that many of the constraints we experience in our industry are due to legacy practices that were implemented 10, 15 years ago.
And he says, sorry, what first steps can an organization take to change the current culture and automation programming regarding non-standardized ad hoc programming practices? And I'll tack on a little comment like after that also, because I think you cannot hard pivot, right? Like you've mentioned a lot of really good practices, you cannot just hard pivot.
That's, as of Monday, that's the new standard. So I'm wondering like, what is gonna, if you're talking to somebody like that, what's the transformation that they need to undergo through to implement those best practices?
So it, it's a slow process. The idea would be of, the same as anything else, right?
You address the process that you want to go through or I start with addressing the end goal, right? This is the endstate that I wanna get to. And on my side, the instate I wanna get to is like automatic generation of stuff, right? So large portions. And to get to this agile type of development, DevOps style pulling stuff.
You start thinking, you're like, oh, if that's what I, if that's what I want, right? The first thing I need to do is I need to get everyone speaking the same language. So start writing the style guide and start trying to enforce, right? I need to write a standard, like a programming standards.
So create these things and start at the beginning, start validating them, right? And if you have time that says people have the time, right? I don't have the time. And if you have time, if you're an end user especially, you can start going through the O code and start doing small things like making comments and stuff and things like that, right?
And then the next thing you would have to have, so so we said the standardization formalization, then start writing the libraries. Look at the things that are being done, like kind of the most common pieces and parts that you need. You can start just writing them, right? And yet there is the open PLC library, which is DMC wrote that I actually contributed the C arc portion of it.
So that, that is very useful. And then what other tools would you need if you wanna go down that path, and the next one would be like version control software. You have to, you could pick one. I use Copia. You can use Siemens, multi-user server if you want to. Some people use version dog.
There's a bunch of different choices, right? But you need to start building the infrastructure first, right? The kind of, the unfortunate part about the library conversation is it's like a chicken and an egg thing, right? I want people to start using the libraries, but I don't have any internally to use, right?
And so I'm willing, and AWS C is willing actually to just give those out. And I do all the time. The libraries that I make freely available to all our customers. Integrated integration partners, end users. I give them out. So like we can kickstart that process by giving you access to what I have that we've made is actually very well documented. Because, again, Siemens has made a lot of these tools, I don't know how many people know about them, but there's an add-on called Code to Doc that basically I can write my comments in markdown inside of my library and it will automatically generate the documentation for it.
Very professional looking with pages and pages of data, right? In des descriptions of the interfaces, descriptions of everything. But in order to use that, you have to be following the rules, right? You have to be making your comments. You have comment, everything. You have to be commenting the rungs.
You need to comment the descriptions of the UDTs. You need to comment the descriptions of the blocks. But if you're doing these things right and then you can leverage that code to Doc and on top of that code to Doc will build the, it will build a user. They call 'em a user help file.
It'll build a user help file, which is accessible from portal by pressing shift f1. So they wouldn't, they don't even have to open up the documentation external. They can use your block, drag it on hold, shift F1 and the help menu will come up and tell them exactly how it works. And so I've made a conscious effort to start building that into all of the libraries that re-release as well.
And again, that, that's what I think needs to happen. We can kickstart it, we can give you at least this part of it. We can give you some of these libraries and then you can start moving forward from there. Is there your URL to download that? Not yet. We're gonna, we're working on it.
Generally what happens is, like customers talk to me and they're like, Hey, I have this problem. Gotcha. And I'm like, okay I have this and I'll just send it over to them. Gotcha. Dave.
No I think that all of this is really interesting. Matt, I think you brought up a really good point that Siemens has lots of really interesting tools that more people would use if more people knew about.
Yeah. Which is basically the conversation that VLA and I have with John to tell him every time we talk to him. I will give everyone a bit of a sneak peek. Our goal is to sit down with John and go through a bunch of these tools for some period of time while we are at automate with him. And I think the goal is we're gonna go ahead and stream a bunch of that.
Yeah. So we will go walk through a bunch of these interesting tools. Yeah, we're gonna go walk through a bunch of these interesting tools, so if you guys are interested, please feel free to subscribe and stay up to date on when that is going to happen. Matt, I feel like, I guess my next question as I've pre-warned you, it is always about the future, right?
Yeah. I feel like you gave us a really interesting kind of future around, especially like efficient engineering and scaling industrial applications. So I want to ask a little bit broader, like what do you think the next, 2, 3, 5 years of the manufacturing or industrial industry in general I is
going to look like?
In the US at least we're gonna see a large resurgence of this, right? Stuff is coming back onshore. We've seen all the investment in it. So there's just gonna be a lot more work to do, right? The scope of what we're gonna have to take on is going to be more and more. And I think in the long run we're gonna start seeing conver, especially with tools like Somatic ax, we're gonna start seeing convergence of like we already do see convergence of IT and OT to some extent.
These things are gonna start merging even more. I think that's the ultimate end goal. And then from a, like a large point of view, when you look at Siemens lineup of software suites that they have, think the overarching goal is to start moving. All of the process in parallel. Right now the way it's done is waterfall, right?
So somebody comes up with a design, the mechanical CAD work is done, and somebody is specifying instruments, and at the very end is PLC code, right? That's just how it works. And we know those things go long. We just get less time, we don't get more. I think what you're gonna see is they're gonna start putting those things in parallel, right?
So you're gonna see things like Teamcenter and automatic generation of code and stuff like that's gonna be tied into the engineering side, where all those pieces and parts are going in parallel. And the reason is we're gonna start asking for quicker turnaround times from ideation to implementation.
I think it'll be interesting. I certainly think that we have to get quicker as we go through this. So I'm certainly interested to see the tools and maybe we get to the point where there are less of us needed to go do the physical programming because we've got other better tools, which is,
I don't think there's gonna be less of us.
I think the work is, so I think the work is gonna increase in the number of us doing this isn't gonna increase fast enough, so we're going to have to do more per person, if that makes sense. And the only way to really do that is to augment us with tools. So I see the growth of these tools, Siemens is spending a lot of time and effort making them, and then I think individually, some of these companies, especially some like us are gonna have to take a look and say Hey, we need to start creating tools for this. Some of these like meta coding tools and things like that will compose projects of our libraries. But again, the trust, the first step is always the standardization.
Formalization build the libraries. You gotta build the pieces the Lego blocks, so to speak, before I can build the rest of it, right?
Absolutely. Absolutely. I think that kind of bodes well to the next question is I always like to ask for some career advice and I think you found this career, if we take it all the way back to to the beginning of this conversation a little bit differently.
Yeah. I'm not sure we had any people start in pharmacy or go down the, I think I'm gonna be a pharmacist. Yeah. And then take the hard left. Yeah.
It was a hard left. Yeah, it was.
So we are very happy to to have you here. So if you're talking to someone, especially early career looking at different options and opportunities, what would your best career advice be to get into and to stay into the industrial ecosystem?
So it's hard for me to say, right? Because it depends on really what they want to do. I like doing I like programming, right? I especially like programming of physical systems where I can see what actually happens in the real world. So manufacturing and we do, I do all like upstream, downstream, midstream oil and gas.
I've touched everything cuz like you said earlier, I alluded to, we're in a unique position, right? Cuz we're also a distributor. It means I've been exposed to an amount of projects most people would never get in 10 years, right? So I've had an kind of extraordinarily good opportunity for growth presented to me that's not normal or a little bit abnormal, I think because of our position.
So for individuals, like if they want to get into, application engineering, right? Control systems engineering. Really what I would say is I would tell people to start reading some of these software engineering books. I would actually tell people to, to try out some computer programming languages so they can understand some of these concepts, right?
And then, I don't know, r really, it's just to go get some experience and do it. Find, I get are there find a field. So find a place where you get exposed to as much as fast as possible. That really, that would really be my advice because, and it's not for everyone, right? Because there's some people out there that are like I wanna know what I'm doing day in and day out, right?
If you wanna do what we do and be successful about it, I would say that the person who wants that is the person who's always looking at what they're doing and be like, Hey I want to make it better the next time, or I wanna do something new every day. So if that's you and that's what you wanna do, then I would say look for opportunities to do that in, we're, we are a good place.
And Matt, I guess if I can, Dave, if I can maybe bundle the next question and kind of expand on resources, but I'm curious, like specifically again, if I'm listening to this podcast, or again, viewing the live and I'm trying to figure out how me, as an engineer I'm looking to improve, like what kind of resources, or maybe if I wanna frame it as best ROI in terms of like more software thinking, would you recommend?
And
because I thinks it's, yeah, I've got a bunch. We can go over, I could do more than one. So for sure I would say anything by Robert C. Martin again. So like clean architecture would be the first one that I would say to go look at. And then there's one that's clean code is good too.
That's more about like stylistic choices and how you should do the micro pieces of the code. Not necessarily architectural choices but clean architecture is a good choice because it teaches you how to manage dependencies in a way that are gonna be useful to you. Now again, this is something that, that's hard to do because they're not directly applicable to what we have, right?
So you're gonna have to, you can't just read it and take the examples and then go implement them on the plc cuz the tools they're using don't exist, right? We don't have classes and inheritance and things like that. So you have to understand what the purpose is and then understand how they're solving it to try to approximate it in the plc.
And even if you read the book and you're like, I don't know how to apply that e, eventually you're gonna start in the beginning. Like I said, I read it and I'm like, dude, this sounds exactly like what I've gone through. That'll at least get you some kind of understanding of what, good code looks like.
Absolutely.
I think that is I think that is fantastic. Thank you for for those resources in there, Matt. And last question to you is who should reach out? Who do you guys want to talk to? It's very much your open forum. Are you guys looking for customers? Are you guys looking to hire?
Are you looking to talk to Yes. More people who are interested about this? Tell us how the community can help
you. All of the above. We're always looking to help customers solve their problems, be that end users or integrators, right? We're always looking to partner with our technological partners, like the Siemens and stuff of the world.
That's one of the main things we wanna do. And then of course we are trying to grow like everyone else, right? And that means I'm also looking for engineers who wanna do this type of work. Absolutely. And everyone else Yeah. You're free to, you're free to contact me.
Awesome. Ca can you give us the best website for for people to go check that out, either careers or otherwise awc they're looking to, so you can just awc.
Perfect. Awesome. I appreciate it, Matt. This has been super fantastic. I have three hours of brain refactoring that has to to go on after this just based upon all the things that we've talked
about. Yeah. If y'all wanna go into so if you ever have a podcast that you wanna actually see how to do some of this stuff, I'm very well versed in all things Siemens.
So you were talking about showing off some of these features and stuff. If you ever wanna do that, I can definitely show you how to use basically any of these tools. Awesome. We'll definitely
kick you up on that.
Yes. A absolutely no. Thank you for that, Matt. And thank you everyone for coming to hang out with us.
I will remind everyone if you've made it this far please come join Vlad. I at Automate middle of May. Very excited to to go see a bunch of people live. If you guys are watching live, please hit the thumbs up. Please follow Matt myself. Lad, please follow a manufacturing hub. If you're watching on solo's plc, YouTube, please hit that like button.
That will be that will be very good. And if you guys are listening on podcast form, please write us five stars on Apple and Spotify and all of those places. You can do that. Please hit the follow button. We've got a podcast that comes out virtually every Thursday other than this week when I had to go do a bunch of over re-editing for audio issues.
Daniel is already requesting a part two of this and we'll absolutely have to figure out when we can do that. But again, thank you everyone for coming to hang out with us. We'll see everyone soon. All right. Bye-bye. Thanks guys.
Thank you much.