Show Notes
Hey, everyone. Welcome to The Podcast Registry number 2021.5 CI/CD, I guess, right?
Yep.
There are two Dutch guys, Peter Jaap and me. Yes Sir.
Hello.
We're going to talk about Tech stuff, Magento stuff and Oriented for back end developers, I guess. In this episode, we're going to focus upon CI/CD. Basically, maybe an introduction a little bit about what we understand of CI/CD. What it's should be doing? How it fits into our own businesses and etc.? But Peter Jaap you came up with a topic.
I think, actually, you did.
I did already. So, actually, I have to admit that something else is on my mind. So, you notice Swedish Chef, who says like, a bird, bird, bird, bird. Have you heard about the Finnish Chef, who for, who for, who for. It's changing our agenda quite a bit. The new front ending of internet supported by internet. So, who face is on the rise? But we're not going to talk about that. There was the previous episode. Number four was dedicated toHyvä. Today, talking about CI/CD. So, what is CI?
What is CI? CI/CD is accommodation of "Continuous Integration" and "Continuous Deployment", those two topics are a lot of the times combined, mentioned in one sentence, CI/CD. But if you split them up, they're actually two parts. So you have the Continuous Integration (CI) part and the Continuous Deployment (CD) part. So, let's start with Continuous Integration. What is Continuous Integration? I'm not going to give 100% correct definition, but I'll say what I interpret SCI and I guess that resonates with most people. Because he was integral part in your code flow, where you integrate all the different lines of code, all the different packages, all this other stuff, you combine and have written yourself into one monolith in most cases and you check, mostly whether it integrates with each other. This could be unit tests, although those are more on a unit basis. So, not necessarily integration but that's usually part of this step and Integration Tests is like, End to End Tests.
The Integration Tests of Magento itself as well. So, kick starting the Integration Testing Framework MFTF (The Magento Functional Testing Framework) for Instance has told a part of it.
Cypress IO is used a lot.
Definitely, yeah.
Stuff like that. So, that's the integration part and then you have the continuous deployment part that's where you actually push code to a test environment or review environment and acceptance environment, whatever you want to call it and eventually to a production environment. So, what's this "C"? Because now we've talked about integration and deployment. So, why is it continuous? It's continuous when you trigger these actions, mostly using pipelines, automatically when there's a commit or attack release. So, you automate this stuff, you don't have to do it manually. And then when the pipeline succeeds, then all the checks have passed and you're fairly confident to be able to put your new stuff live. The main benefit of CI/CD is assurance, more assurance than you otherwise would have to deploy stuff. Because you are supposed to have some rigid testing in place, that almost guarantees that nothing will break when you put it live, if you have set it up correctly. This is a Utopia in my opinion, stuff will still break.
Yeah. I think the burden always is that it's quite an effort to put it all up and make sure that it's functioning. So, what would you say about the costs involved to set up everything properly, compared to actually the benefits in general.
In general, it by far outweighs any cost to set it up, especially now since all the systems have become so much easier. I remember when we set up CI/CD for Magento One like, five, six years ago, that was painful. Magento one didn't really Lint itself. They have supported the building stuff like that. The tools we have now weren't as great. Back then as they are now. There were a lot less people working on it back then. So, it was a lot less support, a lot less blogs, and a lot less general knowledge in general about CI/CD. I think when we first set up our deployment pipeline and our review, our testing environment system, which we'll talk about later, probably, I think I spent like, three or four months, full time on building this setting it up.
Yeah. To me like, well, it sounds maybe that you're much more experienced with this whole stuff than I am. However, as soon as I'm using a looking at my own business. While I'm not busy with projects. I have a couple of projects, but those are my own, then I have extensions, I've micro services, I've got a lot of diversity in there. My tool here is never the same. So, it's all sounds pretty impressive. Seeing you talk about all of this stuff that always comes to the natural.
I always feel the other way around that you know more about this stuff than I do. I always say, with your extensions, I always see you're setting up test environments, Unit Tests End to End Tests.
The funny thing is I guess, it's a totally different thing. So, for all my parts, the testing part is much more important, because I'm focused upon extensions that are reusable in numerous environments. Therefore, I need a certain guarantee, easier to apply with. But that's maybe the continuous integration part. So, the continuous integration part is on my end, pretty important. While continuous deployments, I've got a couple of scripts, doing stuff, making sure that everything just works. So, I think, it's also fair to say that it just depends upon your needs and maybe one of the points that I'm always trying to make with regards to testing but it's also true for CI/CD is that if it's overwhelming at first, to get started, then don't try to equal us. Don't try to complete everything at once. But just start with a minor, simple step and see that step actually catch human error, small little mistakes that are caused by this automation process. And if you do that, once you start to see the benefits, and then you can do it another time and another step and so on.
So, what will be the first step in your case?
Yeah, I think actually, the first step is to guarantee that it goes on the hosting or on the local development environment, it's just managed in a correctly. So, one of the things that I still remember of a couple of months ago is that I have created this Microsoft Office for myself, based upon YAML files, and then sooner or later, I created a little typo in that YAML file. Well, there was no any kind of tooling that prevented that eyeball to go live in production. I only found out about it two days later. The Microsoft Surface was basically debts, but hidden away behind all other micro services and a couple of websites and I simply didn't notice it. I should have called that simply by linting YAML. Now, that could be part of the CI/CD part as well. But it could also be just part of your local community.
What is linting for the audience?
Linting, I don't know the perfect definition but it's basically looking at the syntax of a specific language, just to make sure that it's correct. For instance, if we're focused upon XML. Well, XML is all about opening and closing tags. If a certain closing tag is not there, its bad linting step more complex, of course, we have XML schema validators or XML schema definitions exist ease. So, that basically leads into that you could also check for the schema there. Likewise, JSON, if somebody makes the mistake of during a single growth or a double quote into single quotes, stupid little things like that. So for me, YAML would be that it's actually not perfectly lit. But more of that you simply have a PHP parser that is able to read a YAML file and lead into data originating from that YAML file. I think that's not a perfect definition of linting. But at least it's a good step forward.
I described it the same way.
Basically, with linting, you're much more focused upon the syntax. So that stupid little typos don't end up causing bugs that you don't need to analyze. I think that's one of the more peculiar things. While personally believe that unbelievably annoyed with PHP storm lately, because every time when I'm opening up a specific projects, it's automatically suggesting to upgrades PHP environments, so that if I'm running a project based upon PHP 7.3, it's not only supporting PHP 7.4, which sounds great, but then I end up automatically generating all these getters and setters based upon 7.4 syntax, and then I push it into production and then it fails for some kind of reason. That's maybe also a good point. If you're linting, it's not only that you're linting for the PHP version that you have available locally, but you're also linting it in Deployments, specifically for that PHP version that runs in production. I'm not sure if you're personally use it also in a similar way to prevent PHP 8 functionality from ending up in production with 7.1.
Yeah, definitely. We have a PHP linter, specifically for PHP version. We usually put stop files in our project. In the root of our project dot PHP 7.1, dot PHP 7.2, 7.3, 8.0, 8.1, whatever and then, based on that, it will load the correct PHP linter. We'll check for any methods that are deprecated or not available yet.
Yeah. Well, that's basically something that you could do locally, of course, by using tools like, Captain Hook and Grumpy or GrumPHP.
GrumPHP.
Never sure how to pronounce it. I'm still using it in some projects. So, with me. I'm focused upon extensive developments, little experiments, then the total number of repositories that I'm actively maintaining is about 100 or so. I lose track, basically, of where I've configured what, but it seems like you guys really are rolling out grumpy and other trash for every single project.
Yeah. We've been using GrumPHP for about four years now. We basically have rolled our own. Since a few months, we started using the mediaCT Testing suite. So one of our colleague companies here are falling and they it's on GitHub, it's an open source project. If you install that one, it will pull in GrumPHP, has a few checks. It does PHP stands for static analysis of PHP code, PHP mess detector, which is a piece of software to check for messy code in your PHP stack. PHP unit, it will try to find PHP unit tests on from several known locations and London. PHP code sniffer, which basically checks the coding style we adhere to.
Including the Magenta 2 coding style as well. So, that's set for.
I'm not 100% sure.
It's supporting Magenta 2 but it's depending upon your own configuration, whether you want to apply that or not.
It uses the magenta 2 coding standard from mediaCT. mediaCT slash coding stands for Magento 2. This is I'm pretty sure it's a slightly altered one. It pulls in. I'm just checking it out. It pulls it in from Bitbucket slash mediaCT.
One of the things that I always find harder to manage on a more organized scale is actually that with Phpstan and with Phpcs you always can check for things up on a certain level. So that's within the Phpcs that's the security level and with Phpstan that's, I think, also security.
Yeah also. It goes up to 8 or something or 9.
That's always the question. The thing about phpcs, in combination with the magenta 2 coding standards is that the marketplace, the official marketplace, is allowing for any extension that simply just lives up to security level 10. However, the really interesting stuff is more present there with the lower security levels, so that your own code needs to be compliant to more strict standards. I've got a couple of libraries where I'm sticking with Phpstan level four, which was kind of like masochistic to get there.
We also use love for.
Yeah, exactly. Likewise, theoretically, with Magento 2 the coding standards, it could also be just to in tune down to a lower security level, which is basically that it's stricter. That also the harder the punishment if you do something wrong. Cool. Interesting.
If we have level four, which is basically I think it's good average between safety and looseness. Level zero is some basic texts, and its multiple paths level zero. And then you'll get further along. So, basically, level one Checks Form Magic properties, magic methods and getters, unknown methods, level three is Return Types of a fours, death, goat checking, instance of Jack's death, else branches and stuff like that. And then you go away up to seven or eight, which it gets a lot stricter and harder to fix. It's fun to check where project you are, just start with eight, see how many messages you get and then go down.
Exactly. Keep on fixing things until you give up basically.
Exactly. One more thing we do with immediacy testing suite, we actually have our own, which is based on this one, but we add some stuff. And one of the things we always add is a Git Blacklist, that's also an option in in GrumPHP itself. Where you can put in blacklisted words, basically. So, we put in stuff like dump, var_dump, dd, because we do a lot of Laravel as well. So sometimes that sneaks in. It checks for print_r’s without the true argument, it checks for the object manager. Yeah, yes sir. We don't want that in there. We check for we actually have a check for swear words. Luckily, my guys write pretty decent stuff. They never put in any swear words. But sometimes we inherit a project. And then there are all red flags everywhere with some, that's funny. So, that's also a good way to make sure nobody uses the object manager in custom code.
I think the difficulty with all of this would be to configure it to your own needs. So, my personal findings are that sometimes you need to configure a Phpstan in a certain way, with Magento for instance but I'm developing things under Laravel, I'm developing things on Symphony, every time there's a different scenario. Personally, I've always found it overwhelming myself, to just keep track of all the different configuration and the tips and tricks of how I apply them and recently, I found myself actually disabling GrumPHP temporarily because it was just recommending or not allowing me to do any commits. If I was not running the correct PHP version and I was just so annoyed because I needed to fix something.That's kind of like the difficulty that the more you add on top of it, still the more you have to manage. Therefore, my earlier question like, where does this paying off? Well, then I think the bottom line is where it hurts. So, you're doing all of this just to make sure that bugs don't end up on production in short.
Yeah, that's where also CI/CD comes in. We have GrumPHP running locally. It's a GitHub. When you do a Git commit, it'll check these things. If you commit a PHP file, it will only check the PHP linter and the PHP stuff. If you commit a YAML file, it will only run the YAML linter and not all the other stuff. That's great but you can skip it by Passing-N which stands for "no scripts". In that case, you'll just bypass GrumPHP and commit the code anyway. To make sure that code, when it's not as clean as we want it to be, doesn't go into production. We also add these checks into CI/CD pipeline, which cannot be skipped. In that case, our developer can skip it locally, but then it gets onto Gitlab and then the pipeline rule fail. Because there's a swear word in it.
A common question there would be where in that pipeline does the group? I assume that you guys also have Gits strategy, like feature branches and development and no longer Master, but that's called, I don't know, production or something?
Yeah, we renamed it to main. The integration part is in the feature branches. That runs on every feature branch. So, the static type checking the GrumPHP suffer. This tells you about Phpstan, PHP mess detector code sniffer, blacklisted words, linting, pdepend, all that stuff happens in the integration part of the pipeline that runs on all the feature branches. As soon as that gets merged into develop and later on into main. Also, the integration tests for run. And then eventually, when it gets merged into main, also deployment part for run.
One of the interesting things that you just mentioned is like, I'm familiar with tools like a mess detector gold standard Phpstan but then there's also the Phpcpd, sort of copy paste detector but what if your own codes is just looping along perfectly identical to third party extension. I don't know.
How would that happen? Somebody copy pasted code there.
To make this more practical. Did you ever encounter issues where the the pipeline fails because of Phpcpd running into issues?
Yes. But not with third party vendor code. Usually, it's not just with our own code. There's a block here. We needed it somewhere else, let's copy it over. And then refactor later, then forget about refactoring is committed, then Phpcpd complaints like, hey, I see these exact 20 lines here as well. Maybe there's an idea to inherit, or refactor. What's the word, composition?
Composed?
Yeah, that's extracted into a few models of it.
Actually, you're saying that there's a gentle kind, little suggestion to do so. But actually, the whole point is that it's enforced.
Yeah. First, it's a GrumPHP is a locally, it's a suggestion and further down in the actual CI/CD pipeline, when it's in the Git repo then it's enforced.
No. Doesn't that lead into too much refactoring, where actually the refactoring becomes more difficult to understand than the original?
No.
It's also a question that I don't get to answer but as I go along.
I can probably think of a few scenarios where that might be the case. But I think in essence that shouldn't be the case.
For instance, if you create a module in the Magento back ends with an enabled flag, then theoretically, the configuration to XML file, so that's the system dot XML file in the admin HTML folder, would be almost identical with about like, eight lines of codes. So, getting of course also on how strict the copy base directory is tuned.
That's exactly it. You can set copy paste detected to I don't know, free lines and I will complain on every free lines it'll run into but Magento as we all know, is a lot of boilerplate code. It's not that strict. I'm just, I'm just trying to check in the configuration what we have set for Phpcpd but can't find it this quick, but it's from the top of my head, it's about 20 lines, minimum or something.
I'm just also trying to remind me of something. There's always a lot of different tools out there that allow for static analysis. But there's actually this link, or this GitHub repository it is called a Phpmetrics analysis tools, which is just listing up all of those different tools like Phpmd and why everything basically, that we mentioned, but even more like, I'm just looking at for thresh of what used it once in the past and I'm not sure if the project is Gits fan, which is like Phpstan but difference. So, that there's alternatives out there as well. That might be nice to include in our notes as soon as we're publishing these.
I don't know any of those but there has to be multiple of these.
Yeah, exactly and then there's another one that is maybe less important for you guys. But more importantly, on my side as well, Gold Rector. So, Rector, I'm not sure how to define it. But it's kind of like a transformation tool or you could transform things. For instance, I've been only recently working on this, to write my own goals in PHP 7.4 or maybe even PHP 8, share text. But as soon as I'm publishing that code, it automatically transforms, she'll be released as PHP 7.3, just to make it compliant with as many users as possible. I think for a project, that's less relevant, because if a project is running in real life in PHP 7.4, you do want to make sure that the development environment is also 7.4. But from the standpoint of extension development, it's kind of like a cool tool.
You're saying, you're using Rector, not to do upgrades, but to basically do downgrades, it’s like polyfills, it automatically adds in polyfills or something?
No, no, it really just modifies the code itself to become different.
So you write PHP 7.4 code and record will make PHP 7.1 code out of it.
And then automatically branches into gates, or what...
I've only ever used it the other way around to upgrade 7.4 code. I didn't even know we could do downgrades.
Well, it's downgrades and upgrades. In other words, that weird stuff.
Go for it.
In general, yeah.
I didn't know we could go both ways. That's cool.
Apart from that, we began with linting but I just gave a little example for y'all and we talked about PHP but in general, I think linting is just also one of the things you could do locally but also in the in the pipeline. So, CD, it really becomes useful, just to guarantee the minimum level at which the goal should be committed, or should be working. What is that about next?
You were talking about Rector, which is basically if you look at their GitHub page, it says instant upgrade. I thought about instant upgrades of Magento, which is something we've been experimenting with a few weeks. There are two parts of this. It's not just upgrading Magento, but its upgrading composer packages. On GitHub, you have the dependabot, which does automatic upgrades. That doesn't exist for Gitlab, which we use, in a self-hosted environment. But there's an alternative to dependabot called Renovate Bot. We've been implementing Renovate Bot in past few weeks, in our Gitlab instance and that's such a time saver. It's really cool. We make a separate Gitlab project in the group of our clients called Renovate Bot and runs a scheduled pipeline at night, which tries to update all the composer packages it can. We skip the Magento package, because we don't want to automatically upgrade that with Renovate Bot. Renovate Bot tries to update all the other packages within the version constraints you have in your composer JSON. So, every morning, I wake up and then I check my merge requests, because they're automatically assigned by Renovate Bot to the project leads, and in some cases, that's me and I check the change log, it automatically creates change logs with links to the Compare on GitHub. Then I can click on those, scroll through the code, see what changes and if I think it's safe enough, then I'll merge it or I'll check it out locally and test it locally. So, the first time we did that, we got about 40, 50, 60 merge requests per project. That was a lot to get through. But once you've done that initial batch, it's really easy to keep up with it, so you don't have to keep track of which packages are updated in mean time, you stay on a continuous update cycle, you don't do it every once in a while. So, it feels great to automatically update those little packages and have little fixes, processed pretty much every day. Because if there's a small fix, and it's a batch version for any of the PHP composer packages that are upgraded and it's a patch one and I look into the change log in, it's a minor code change for some fix or whatever. I just click merge and the CI/CD does the rest. It's excellent, it's safe to upgrade check sort of it's still compliant for compiles. And if it does it, just push it live. So, that's been a great time saver for us. Also, when you actually get to upgrading Magento, you basically only upgrade Magento and the dependency that it has not all of the packages, you should have been updating along the way, because that's already being done by Renovate Bot.
You mentioned that you don't want to do the upgrade itself with Magento itself. So, that's kind of like common sense in the current state of the ecosystem like everyone else. But why? It's simply because Magento is then sometimes rolling out goals that is known to break stuff. Its sounds a reality that should change.
Yes, that's true. But that's not entirely Magento fault, that's because we do all of the Magento agencies and you, do a large part of our job is customization. So, you customize based on what you have on the latest Magento version. If a new Magento version comes out, and something's changed, like interface, or implementation, which you did a plugin for or whatever, it'll break. There's no way to automatically safely automatically upgrade Magento. There always has to be a manual check for stuff. We don't want to upgrade magenta automatically because there's no way we could safely merge it. So, there's no sense in it automating that.
But that doesn't that count the same for third party extensions as well.
Sometimes yes, if you do a plug in on a third party extension.
You didn't bump into issues there yet.
No. Not yet. We are assigning merge requests to the project lead and project lead usually knows where modifications are being done. So, if they know like, for instance, we have Algolia and some projects, Algolia is sometimes there's major updates in there Magento 2 extension. So, we know to be careful with an Algolia extension because that usually breaks stuff.
It's more on per basis that you know about which packages could easily be managed this way and which not. Personally, I am eager to also implement this because of my other business case being that I want to be loved my extensions to be checked continuously with whatever updates are available. Actually, I would include Magento updates automatically as well, because then I want to know whether the current sources that I shipped with my extension are actually working together with the greater parts of Magento as well. Yeah, that's really cool.
That's really cool. Another thing I just mentioned, we don't do automated Magento upgrades for Renovate Bot. That's true, but we are trying to do automatic Magento upgrades with our own pipeline. That's the Magento auto upgrade pipeline we're developing right now. What that basically does is when we create a new branch called Upgrade-{Magento version}, so let's say, upgrade-2.4.2. Whatthen will happen is a pipeline will be kicked off that does a composer update magento/project-community-edition:2.4.2 --no-update and then run a composer update. This will automatically try to update Magento and if composer runs into dependencies, it can't resolve at that point, the pipeline will fail and we'll know, we will have to update some of the features restraints on the packages or emotional packages or relevant.
To check up on something manually because that the upgrade fails and there's a reason. Yeah, cool.
In that case, we just do a Git checkout of that branch, added the composer JSON and then push it again then the pipeline will again start and we'll try to do a composer update if it succeeds, it will commit the composer log file to that branch. So, we'll have an updated Magento composure package. But at that point, Magento will most likely break something and that's why we use in the pipeline, we use the ampersand patch helper. The magenta to upgrade patch helper. We've talked about this one before, I still think it's underused. I want to point people towards it one more time. But that runs automatically in the pipeline. It will give you an output of all the files that have been changed in the meantime, and which of your customizations, touch those files, and then we'll give you a list. And then you can walk through that list or actually the pipeline creates an issue and assigns it to the project lead with all the things they need to check. When they check that they can update the pipeline again, and then as well run some checks. Eventually we'll be able to move that into development and into main. We'll semi automate the Magento upgrade. We don't have to do a lot of manual stuff.
Cool. That's basically leading into educated upgrades. Magento, with much more confidence. Did you guys already use this for 2.4.2, the latest release?
Yeah. It's just as easy as somebody creates a branch named upgrade to 2.4.2. Click Create Branch and then wait a few minutes and then in our Slack will get a notification whether it failed or succeeded and then we know what to do.
Yeah, exactly. There's also like, we mentioned earlier that the testing could be part of the story as well. Unit Testing, Integration Testing, but also End to End Testing. But then for End to End Testing, you also need to have a domain name, Temporarily Domain name, Temporary Domain name to access the site under. If things would fail, without upgrades, would that instance be still available, somehow? No, not in this automated way.
No, it won't get there. First, we do the all the checks in they will do to build stuff. Once we get to the deploy step. This is a review environment, then we spin up a Kubernetes cluster on Google Cloud Platform, then we use Docker containers to push the Magento site there and then we use the Digitalocean API to create a DNS a record for our development domain, and it will attach it to the IP we get from Google Cloud Platform, then it will generate a random admin user and password. Once that all done, we'll get a Slack notification in the in the clients channel with this information where we can reach debrief your environment.
Basically, this is already touching upon deployment. So, that of course, the CD part that we're slowly leading into. Maybe it would be good to sum up also, that of course, there's Deployer Plus. To my understanding it is right at this moment, the most popular tool to deploy Magento sites. And other people are using still Capistrano. But personally, if you're using Capistrano, my advice would be either you're a Ruby developer that loves Capistrano or you're a PHP developer that doesn't know about the Deployer Plus.
That were we like five years ago, we ran it to Capistrano, implemented it, then later found Deployer. Luckily, Deployer is basically a Capistrano clone. So, we didn't have to change a lot. It's just the runtime changes, but the output and the folder structure stay the same.
Deployer Plus, for those people who don't know, it's just a variation of Deployer(Deployer). Deployer is also kind of like in the in the symphony Laravel, ecosystem pretty large. Well, that there's also a layer. So there's different deploy. But in general, the PHP deployment tool of choice is Deployer, but then on top of it, there's Deployer Plus created originally by Jellow Good.
Jalogut.
Yes, one Alonzo and I'm not sure if you know about pulling a Juan.
Pulling a Juan?
Yeah, that's basically your reference. If you're going out for a wild night after some kind of meet Magento conference, together with somebody named one, and then the whole group is leaving, when one is actually through the toilets. Maybe look for the party, like hours later. That's pulling a Juan.
All right. I need to do that post-Corona.
Exactly, yeah. But anyway, one just created the jelly goods or created Deployer Plus, and I guess he's still maintaining it, then there's not a request anymore. But anyway, that it's a wonderful tool and why? Because you're using it, also because of the file structure of the Deployer Plus.
Actually, we're not using Deployer Plus but that's more of a rendant when we started using Deployer Plus wasn't built yet. We along the way, we created our own Frankenstein Deployer Plus but one made a much more concise and more generic solution of this. So, one of the thing is on my To Do list is to switch to Deployer Plus instead of our Deployer Frankenstein creation.
Was also personally my favorite is actually that there's always a specific needs. In the past, I've customized my own deploy PHP scripts, and a couple of other indications of how things are being organized and simply just looking at the organization, the goals of others, trying to improve my own version. It's just that it's open source. It's based upon deploy PHP and there's reasons why it's more elaborate than just using deploy PHP but I think also it's not for everyone. I don't know like, there's specific choices being made with organizing all of the different files. I forgot a little bit like, what kind of issues I heard about this in the past. But while, my personal take is that deployment in general, but also, therefore, CI/CD is always something that needs to be customized and you don't necessarily need to stick to a tool like Deployer Plus, just like you guys also created something similar in the past as well. That's my main points. It's all gravy, freedom of choice.
Yeah. But as long as there are provisions built into your sidestep the default route, like to buy a plus has.
The rest of the deployment step, basically comes up with compulsory installation, static content deployments. The compile also included there and then actually, the deployments itself. Any comments there, like what is specific in your own CI/CD pipeline or CD pipeline actually?
Compared to Deployer Plus not much, I don't think so. We used to do a Git pull. The Deployer will log into the proxy server and do a Git pull on the Git repository there. But we don't want the Git repository to be there, because we want the files on the production server to be read only. So, we move to push only approach where we create an artifact in the Gitlab pipeline and our sync it to the production server and then extract it there and then make the files read only.
One of the things that needs to be done as well during deployment is, if needed, turn on the maintenance mode, run all of the setup upgrades and turn off automated maintenance mode again. I've been tracking their 60 for a couple of projects, where I simply encountered that the setup upgrade was run every time again and again, as if, with every upgrades of composure, there were still too many modules being upgraded by simply incrementing. Their setup underscore version tag in the module dot XML file, while was actually not needed. What is your current experience there? How often is the setup upgrades actually enforced by the whole upgrade system? While it's actually not really needed, simply because people make mistakes with that set perform, I don't know.
I don't know. I've never checked. When we setup DB status was introduced in magenta 2.2 point something. We implemented that and I've never actually checked whether it does too much a little.
I've been implementing it myself. I've just been noticing that looking better back at the deployment logs, I'm seeing actually that there's a certain downtime, simply because that maintenance mode is needed because setup content-deploy status is just reporting that there's something changed and that actually leads into sometimes well but depending upon your setup but eight seconds or 20 seconds, even of downtime, while the whole goal is of course that we have zero downtime.
I've never noticed that. I've never noticed excessive downtime or unexpected downtime while we weren't expecting it. But we do I'm just looking at our stats, we did over 50,000 pipelines last year, I think of which, I don't know, maybe a quarter of those are deployments. So, we do a ton of these because it's so easy and we're fairly certain everything goes right when we started deployment. So, we do it on a Friday afternoon, what do we get? We do a ton of them. I actually never checked but we've never noticed any excessive downtime.
No, it's only something I started to notice, as soon as I've added to my own deployment scripts, to add a little bit of a timestamp before the maintenance mode is turned on and after the maintenance mode was turned off again. It struck me that basically the 1/3 of the times that I'm actually deploying, the upgrade scripts are taking more than they're needed. So, there's actually an upgrade of the database happening. While I was personally just expecting that I just upgraded a couple of modules and my personal belief in there is also that there's a couple of modules that are using their own set up recurring scripts, so that they're actually doing something pointless with every single upgrade that you do. One of them is something I've already tracked down the Magento New Relic module. So, that's hooking into the recurring face, so that every time that you're running setup upgrades, it's kicking off its own thing for reasons.
We don't use New Relic, so we never, we always accepted. So, we can use it.
Maybe good. Then what else is there actually to consider with CI/CD?
After actual deployment, we run a few sanity checks. That's our latest step. That's part of the process. Some performance testing, we kick off a lighthouse check. To check the lighthouse course, we do site speed delay, to check site speeds, we do a smoke test using Cigar that reads out a Cigar JSON and the project route that will check as some predefined URLs and their status coded returns and our second to last step will be security checks. So, there we do the PHP security track checker from Symphony or that used to be the row for security advisories.
I've been using row for quite some time, but then it got replaced. I keep forgetting about it.
Yeah, it's deprecated. Now it's not the fault local PHP security checker. So, that checks for whether that are known composer packages in your system that are compromised, that have some abilities and we do, wasp check. The zap, wasp zap. That checks for a bunch of rules on server, service security and cross site forgery request checking. All kinds of stuff.
Interesting, maybe sooner or later to just dive into security on its own, which is always like an in depth topic.
We need to get to latch on. If I'm forgetting one step, by the way, the last step in our case is report check. All these tools have an output, obviously and we combine these outputs into a little website generates it and we put that up and get lab pages. So, there's a link, the clients can check or we or developers can check to see all these reports.
Yeah. It's just a large tree of different steps that you need to take just to make it to the old success, so that's also why we're sharing stuff like this. Slowly, we should be ending the podcast. It's always getting late with this guy.
I think we skipped about half of what we wanted to talk about there's so much here.
That simply leaves room for version six as well.
Yeah, in a few episodes, we'll do see CI/CD part 2.
The left overs, but babies still really interesting stuff. Awesome, little listening. We'll be sharing a lot of the different links with the public later on as soon as the podcast is being published, right?
Yep.
Any final comments?
Nope. Live long. Live long and prosper. Feel the CI/CD pipeline.
That's awesome. To the listener, I hope you found it interesting and hope you learn something about it. We're open for debate as well on any of these toolings that we mentioned. It's sharing and caring and let us know your own thoughts as well. But thanks for listening. I hope to see you all back to have us listeners for the next episode on, I don't know what but something interesting.
We'll see.
We'll see.
All right, thanks.
Thanks. We'll see you.
Bye.