duke: Okay folks, today we have a, , huge treat for you.
I was just browsing the, , SuperSecretMVP channels, Where all the MVPs chat and
plan to take over the world, such things.
And I saw a story about somebody who had upgraded their big financial
client to Washington in eight days, using no exotic external
tools, just what's in service now.
he is a friend of the channel already.
, and so we welcome back Mark Rothoff to chat with us again about this amazing
upgrade experience while it's still fresh.
, mark: Thank you for having me.
It's always a pleasure.
Let's have some fun.
duke: for sure, man.
So, set the stage here.
What version were they at?
How big an instance, how complex, what were we staring at down the
barrel before we decided, Hey, let's upgrade this thing in eight days?
mark: Yeah.
, actually, , they were a version behind, so they were actually on, , on
Utah, , and that's all already, , a bit of the reason why we went to, , Yeah,
the upgrade in a week because, , I got some pocket change, , to, make this
change and , to , upgrade in a week.
the instance itself, it's, , one of the largest financial
companies in the Netherlands.
The instance is not that big anymore because I've been busy with.
My database footprint work.
So, , the footprint, , was cut in half.
, so now it's, , only , seven terabytes.
Uh,
duke: terabytes.
mark: it was a bit more, but yeah, then.
I came along, that's my job, I guess.
duke: Uh huh.
mark: a lot of code.
Custom work.
, Also some things really legacy.
It's, , still a lot of workflow, for example, and, , still not
really applying flow designer.
, but yeah, , it's just an instance like.
, we see a lot, I guess.
that's a little bit, background about the instance.
Yeah.
CJ: the thing about your post on the super secret slack channel, . That
really, I think takes something that's already extraordinary.
and then elevates the two, like the next level of like outer space is that
you said that you did it all using only , the free tools in the platform,
mark: That's correct.
Yeah.
, just, , normal upgrade, preview, upgrade, monitor, , also with the cloning, just
the clone definitions, the clone preserves excludes, , clone cleanup sketch.
, ATF is in place, , also some bits are not in place, , so like, uh, we're not, not
using CloudRunner, still we're not using CloudRunner, we really have to, so that's
also one of the things for the evaluation, , which we'll have in a few days.
But yeah, just only out of the box, , free available plugins and capabilities.
, so we also actually used a few of the new Washington bits or actually Vancouver
bits, , like the skipped record rules.
, but we couldn't apply it automatically because the customer is coming from Utah.
We had to run them manually, but, , yeah, we also applied that.
So, uh, really nice to see, , because I guess not that many customers
are still already using it.
, but yeah, things like, , skip record rules can help.
, but yeah, just the out of the box available, , free capabilities,
, yeah, and things like ATF.
So, yeah, , it's all possible.
duke: I think we should break some of those down a bit because there
might be people, some of them might be on this call, who have never
heard of a couple of those things.
So I know that , the clone management tools are awesome.
ATF is.
ATF.
, Cloudrunner.
Tell us what Cloudrunner is.
mark: Ooh.
What is cloud runner?
Basically, if you would, run ATF.
, especially a few releases ago, then you would run ATF, keep your, , laptop
open, keep your screen open, keep it running on the background, but
you, yeah, it was really a attended.
Thing, right?
You really have to be on it.
, and then a few releases ago serves now already introduced something that
you could spin up your own, , Linux or windows server, and, , you could
run the, , the ATF, , from there.
, but now since I don't know, two or three releases, something like that.
I don't know exactly.
, serves now also has.
Cloud runner, , where you can execute two, uh runs at the same time.
I don't know the exact wording, but you can run.
Yeah.
Two, two runs at the same time.
, on something that serves now is actually hosting.
If you want to perform more at the same time.
, then you can buy that, but at least you can, , yeah, run to,
yeah, the two runs in parallel.
, so you don't have to have your laptop still open or whatever.
You would just do that to cloud runner.
It's also pretty easy to set up.
We also explored it before at the customer, , at, , innovation Friday.
And it looked really nice, but yeah, then it comes like we need to pick it up.
It becomes a story, a backlog, et cetera.
And then.
Yeah, then it gets a bit quiet.
So that's still one of the things we do need to apply.
, but yeah, that's one of the things for the evaluation.
duke: So the advantage of cloud runner is that you don't need
to have your laptop open and
mark: yeah, that's one of the advantages of course, but especially
when you're talking about upgrade in a week, , then the timelines are so short.
When are you going to do those tests?
Well, ideally you would start them at six o'clock in the evening or something.
Um, Um, right.
And not at daytime or that someone especially needs to jump on their laptop
at 10 o'clock in the evening or something.
, so yeah, I think it has way more advantages.
And also if you can do two tests at the same time, why can't you?
Shouldn't we split the ATF suite for tests?
, so now the execution takes about eight or nine hours.
Well, maybe we can run two at the same time and it only takes five hours.
, and you don't have to attend your laptop holder all the time.
duke: But it still dumps all the findings and stuff in the same place
as if you had attended it on your lap.
mark: yeah, the eventual results and everything that stays the
same, , there's no difference in that.
So it just makes life easier.
, you can really run it in attended,, at every time.
, it makes you way more flexible.
, , I see good, , potential for that.
Yeah.
CJ: Did you set out to set the record for, , world's fastest, , upgrade of a humongous
company, or did this just happen by like happenstance and you were like, you got
into, and you were like, you know what, I think I can get this done in a week.
mark: Yeah.
, For such a large company, it's quite impressive.
, I did do a upgrades of a week at smaller companies, but then you're talking about
instances of less than one terabyte.
. And then also as a really small user base, but this is quite interesting.
, but yeah, then, it's also a bit like, where did this come from?
And, , I'm now with this company.
For I think one and a half years, I'm working part time
for them, , three days a week.
And, , the first upgrade that I saw there was, yeah, it's
the typical four or five weeks.
, a lot of, , waste, , I'm seeing it in the, yeah, in the time spent
, and, , how the testing goes and with stakeholders and everything.
So already the, upgrades following we, Improved a lot and also the change freeze
was also limited to a few days instead of like one and a half week But still
I was like, come on, we should shorten this we can do this within a week.
Come on and um It's pretty difficult in such a large organization to get things
moving and changing But, um, yeah, this is one of the things, but this is
one of the things we did get moving.
And it, it also was a bit because, , we actually skipped the last upgrade,
, because we actually, , we're planning to upgrade in January to Vancouver,
but then of course we have, , like eight different service now teams.
, it's like 35 or 40, , developers and then one of the teams just says,
yeah, we don't have time to upgrade.
Okay, so we're postponing the whole Vancouver upgrade because one of
the teams thinks that they're busy.
Well, no one is busy.
, maybe some individuals are busy.
But no teams or departments or companies are really busy.
That's for people who cannot organize their own work.
, but yeah, one of the trade offs, but one of the trade offs of that was actually,
well, when we're going to upgrade in a week, so that's, that's a bit how this.
Um,
CJ: yeah, I mean, that is crazy to me and you already touched on it, right?
The typical processes, , 4 to 5 weeks, right?
I typically recommend with my clients about a 4 week
process in my document, right?
And I think I might have done it as quickly as I could.
As three, if I'm following a formal process, right?
Like when I was a customer and I owned the instance, I probably did it
quicker, but I cut a lot of corners,
duke: Yeah, back when it was just ITSM too, right?
And a couple of maybe custom apps.
CJ: Yeah.
Much smaller company too, right?
, and we really just running like change management.
That was the only external facing thing that we cared about.
And so , it was much easier to just flip some switches and nobody really cared.
Like I couldn't imagine.
, upgrading a humongous, , financial organization on top of that and a
week, that's just amazing to me.
So, kudos for even, taking that on.
And especially, it sounds like you already had, like, you've been there a
while, so you have some history, , with these folks in terms of , being not
necessarily being the most receptive to the upgrade process at all.
, . mark: Yeah, it's also because you are dead and already, , for like
one and a half year, you see how it's a dynamics within the company.
What's happening?
Where is all the waste?
, so.
And then you're thinking, we should be able to do this differently.
And, , obviously we did also do some other things like in the background, I
was working on instance scan, so I could run it before a clone, after clone,
before an upgrade, after an upgrade.
So you would see some things.
, obviously we did upgrade one of the instances already a month
ago, , to see, Hey, what are the skipped updates on patch zero?
So we know a bit, what can we expect?
, and we knew, , Oh, one of the things on employee center broke.
So at that moment we could already create a support case , for
something with AI search.
, so there's a little bit of preparation, , but the upgrade
itself, yeah, that's just, yeah.
One week.
It is quite challenging though, because yeah, , you want to clone
instances, you want to upgrade instances, you want to run your ATFs.
If you already look at the.
The timelines for that, the duration, like the clone, , also because
they do, the upgrade on demand.
, so the clone already takes like eight or nine hours.
The upgrade itself also about eight hours.
Then you have ATFs also eight, nine hours.
So the
duke: you're left with four days.
mark: The timelines are getting, pretty heavy or pretty, um, yeah, I don't know,
duke: you said it took like seven days, were the hours you counting?
Is where these things were just chugging along, minding their own business?
, mark: no, it's just really, , in that week, , some of the evenings, nothing was
happening, , but some of the evenings, indeed, the clones, , were just, , going,
the upgrade started at ten, uh, at night on, , on Friday night, so it was,
, finished on Saturday morning, , It's also a bit like just do it, , because if you
talk about customers with this, or even if it's for two weeks upgrade, then there are
so many objections and people jumping into fences and they see all kinds of issues.
So you also just have to do it.
, and also in this case, it's Washington patch one hotfix two.
It's.
Yeah, pretty fresh.
And what you still hear a lot of people saying it's like, Oh, let's wait a
few patches and before it gets stable and the first issues are out, but
that's people mentioning that because they hear other people saying it.
And because people are already saying that for years and years and years,
it's not, ServiceNow is not how it was five years or 10 years ago.
, if now patch one comes out.
We already had patch zero before and already the first hotfix is coming
and the second hotfix is coming and now actually already patch two is out.
So the level is way higher than it used to be.
So you don't need to wait for patch four, five, six or something.
That's also, yeah, that's ancient history.
So people talking about those things, they're a bit stuck in the past.
CJ: Yep.
So, Mark, , you mentioned a lot of people popping out and saying basically no.
And I think one of the, , one of the keys , to the success and being able
to deliver on this, . Is knowing who to talk to, to get a yes.
So talk a little bit about how you cut through,
. All of those no's to find the right person.
To green light this thing and get you that, yes, that you
need it so that you can do it.
, mark: I didn't do so much myself in that area.
I just on the background, I was, , talking with, well, mainly
three, , people within the company.
And, also , , the owner of, , of service now within the company.
, and eventually it.
He also just had so something like we just have to do it.
And also because, , before also we skipped the previous upgrade just because of time
constraints and we're already bad behind.
And, so they also have a feeling like, Hey, we need to do something
and we need to make progress.
And we do want to also save time for the future, because if we can pull this off.
duke: Okay.
mark: was, yeah, I don't know.
It was also just, , just do it.
And, , yeah.
And also the, just the people really owning serves now also really need to
be enthusiastic about this and see the opportunity and the possibility and just.
Just do it and, , and just believe in it.
And, , yeah, one of the big differences, of course, , is that if you also do
this for three, four, five weeks, and it really depends on how many people
are in the company, of course, but then not all the service now teams or all the
service now developers will be dedicated.
Busy with upgrading or dedicated available.
They will just also be busy with doing other stuff.
If you do this upgrade in a week, then way more people are dedicated available.
, so that's also a key factor, to getting this, , to a success
and even that, yeah, yeah,
duke: I would, I do want to make sure that we hit this one point.
Cause I don't think anybody can decide , Hey, we're going to do this in a week
like, and then execute it that way.
I believe just because it's you, that was there, that there
was significant amounts of.
Effort put in beforehand such that you could execute in a week.
Is that true?
mark: yeah, I think that that helps.
Um, and also.
duke: all in one big bite or like a little bit at a time, but I think, I think
there's a lot of customers that don't even have like ATF running in any capacity.
So I wonder if you could talk to the level of effort to get cloning
management ready, ATF, cloud runner, the skipped record rules.
, what was the journey to get to the point where you could do one week upgrades
mark: um,
duke: change freeze?
mark: yeah, well, I think actually we still have a lot to improve.
, if you look at the cloning process or how many manual steps we're, , we're
doing or how long the process, , takes, , but already a lot was in place.
, and also, it helps that.
I'm there.
Yeah.
It helps that you have a certain name in the, space, a certain track record.
, that's true.
, we did, , perform an upgrade a month earlier , on a different instance
to just see how big is it, , what is happening, , what is going on.
I was already working on instance scan, , so that customer also now has
like 800, , scan checks, , in place.
, ATFs.
Yeah, a lot of a TF , is being used.
There is and is in place.
, still during the upgrade in the week, they expanded it massively.
, but there was already a lot in place.
, if you don't have the, cloning and the control, the at don't, , you
don't have ATFs, , or you have, yeah.
A lot of stakeholders that want to do.
every single piece manually, then it becomes really difficult, of course,
then you, yeah, then pulling something like this off will be really difficult.
So yeah, you do have to have some pieces in place and , some of the
teams also do almost all of the stakeholder testing themselves.
So also the number of stakeholders involved is, pretty limited.
, But , if I look at the previous upgrades, which took three, four,
five weeks, then you would just say, Oh, we're going to upgrade and
you can test and whatever, and then it's, , yeah, it's all over the place.
Well, when are people going to test?
And now it's in one week and you just have.
Yeah, the message to the stakeholders was that's the day that you can test.
There's no other day.
That's the day.
Not everyone really likes being put into pressure like that, but it did work.
And also because it did came from, , , yeah, , higher management or yeah, people
a bit higher in the hierarchy in the organization, because I can have a, a
certain name in the space, but if I'm just mentioning something like that, yeah,
and still it won't be adopted, right?
, I'm not, , in the hierarchy of a company, to make such
decisions on my own or whatever.
That's, , that's not going to work.
CJ: Yeah, I want to follow up a little bit on this too.
. Because, , I think , there's , a really good relationship here between A lot of
the things that you do in the service now ecosystem and what enabled you to
actually get through this in a week.
, and I'll and 1 of the reasons that you're such, an amazing service now MVP.
Right?
And so a couple of weeks ago, you made a post on linked in, celebrating.
. And I say kind of celebrating, . Cause you did it in such an understated
way for this accomplishment.
Right.
And, , but you made a post on LinkedIn talking to celebrating, , , that
your, combined service now content had just reached the 1.
25 million view plateau, right?
That's like over a million.
, I can't even.
Put the data, I get, you know, it's hard to even put it in the words, right?
You're combined service.
Now content, . Has reached over 1.
25 million views.
And when I, is that right?
First, congratulations on that.
That's just amazing.
Right.
duke: In another 30 years, we might reach
CJ: Right.
Like CJ and the Duke in 50 years might be at, at the 0.
25.
Right.
duke: Creepers.
mark: No worries.
I'll contribute to that because I watch.
Every episode.
So, uh, yeah, so I'll, keep contributing to that.
CJ: Appreciate that.
. But when you dive into it, , because then you link to a, , to your service
now, a community page, which has like the entirety of your catalog.
. And, , and so in that catalog, I mean, you've got over 325 articles,
blogs, videos, podcasts, et cetera.
And.
, when I looked through that, they, first of all, , your reference, , for me, right?
Like I, , I have your content on bookmark and whenever I, , need
something and I can't quite figure out, I go and check your library to
see if you've already written about it.
Right.
duke: Yeah, crazy.
Um,
CJ: your previous body of work and effort and talk about how, you
mentioned insta scan and the earlier in this conversation, you mentioned
ATF earlier in this conversation.
These are things that you've written about, , extensively that
you share with the community.
And, , and all of that stuff forms, like the underpinnings of being able to
, execute on something like this in a week.
, so I just want to talk about a little bit about what drives you to create
content to this level, man, you are a machine and it's good content.
mark: yeah.
Thanks.
, yeah, what drives me?
, It's just also a lot of fun to create content and, , , and see the feedback
or also like, . Now, what, you're just saying, that's already a good motivation,
, and , to keep on going, , but it's also, I just started creating content,
, because of the surf now ecosystem.
And because yeah, when I started, I also started from the bottom and I
was just searching and, , Oh, we've got the search on community and, Oh,
we had Ricky and all those YouTube.
Things and Oh, , the YouTube videos from, , from the Duke and, , Oh, awesome.
And yeah, why not contribute also myself and let's see if I can do that.
But eventually it was also for me back then I was still working
at the consultancy company now.
I'm doing this freelance, but back then it was also, , you had this
mandatory personal development plan and the company decides what you're
doing on soft skills, hard skills.
So I was also thinking like, Hey, I'm creating this content and actually
I'm working already on my skills.
And I'm taking ownership on my own journey now, , because
I have to dive into subjects.
I need to learn more about it.
, so my technical level is evolving or in different areas.
And that's also fun.
So I don't have that mandatory personal development plan anymore.
I'm doing it myself.
I'm taking ownership and I'm doing it.
Yeah, with fun, , but not your hard skills, also the soft skills, because,
English is not my native language.
Everything I do is in English.
How do I bring a message across?
obviously I do a lot of, , in writing now, but maybe eventually I will just, , switch
to, podcast, who knows, uh, or to videos.
And then also you're working on, on those skills.
And also then it goes more about.
Your presentation and et cetera.
So , it's way more and it's so much fun also because you're taking ownership and
you're controlling , where you're going.
, but eventually it's also just all , the replies and the feedback.
It's, , it's just a lot of fun.
, and yeah, and we are possible to call it a collaborate with, , with
others, , not only MVPs, but, also with the MVPs, , collaborating.
That's just a lot of fun and, , just a lot of motivation
at that also just keeps you going.
So, , yeah, that's just the main reason that, , yeah, so yeah.
CJ: that, that's awesome.
I love that, take ownership of your own journey.
That should be almost be the title of this, episode, cause I could just see
how it applies in so many different ways, . From what we were just talking
to, but , from also , how do you get.
From zero to eight days, , do an upgrade in a huge financial, , company
, and zero to eight days, right?
Like you got to take ownership of that journey.
Right.
And, and then, in part, like that eight days started long ago
when you implemented Insta scan, when you implemented ATF, right?
Like when you made those, , when you exercise those soft skills and made
those relationships internally, right.
And understood like how the business works and understood.
The, who actually had, work that they could put off.
, and, , who was sandbagging a little bit in terms of having
availability to do testing, right?
Like all of those things came into play here.
And all of that to me, can be summed up and to say, take
ownership of your own journey, right?
If you want to get success in a service now ecosystem.
So man, kudos to you on that.
And I love that.
And that we're going to do Duke.
That's the title.
And yeah, I'll give you that one mark for your charge.
, duke: Mark, if you don't mind speaking of owning your own journey here,
cause I'm trying to get some, taking some notes for a customer of mine
who wants to do everything better in service now, including upgrades.
Tell us about why skip record rules are so awesome
mark: Oh, why are they awesome?
duke: Or at least handy.
Yep.
mark: upgrades.
Some customers will have a hundred skipped updates.
Some will have 500, , every release.
And actually it's not only during releases.
You also have skipped records during batches and hot fixes.
, you also should handle those.
I do see that's.
Being ignored at a lot of companies.
, but especially with upgrades, you can have a lot of skips, skip records, , skip
records themselves do take them seriously.
, because some are high priority, some maybe of lower priority,
but actually do pick them up.
, maybe I should immediately give an example.
You can have a priority five, which is a form layout form layout on service
catalog because serves now added some new fields, some new check boxes.
But if you are not taking that over, or if you are not merging that.
you are missing out on that new functionality.
, or it will not land on the form layout.
So most developers won't dive that extensively into the release notes.
So they won't know of that no new functionality.
They won't know that you can do things without skipping, , with,
, with less technical depth.
, you can do it really quick, more quickly.
So you really have to Take those skipped records seriously.
And also those priority fives, but where does skipped record rules come in?
, a good chunk of the skipped records will be handled every release in the
same way, for example, because it's, , notifications, which you on purpose
deactivated, you're not going to activate them, , during the upgrade or merging or
whatever, no, you did that on purpose.
or we have a ton of skipped errors, which you actually cannot handle.
That's also literally in the service now docs and that you should retain them.
, and several system properties, which you absolutely don't want to refer it or
merge, you just always want to keep well for all those situations, you can make.
So that those skipped records are automatically being handled for you.
And especially if you have two or three hundred skipped records, then half of it
can just automatically be handled for you.
, so that's a huge saver.
and you don't have to go through it manually anymore.
So it's less of the repetitive, , Not so attractive work, , less
chance of making mistakes in that.
And so that already helps.
, so that's one of the parts of skip record tools that you can handle those skip
duke: is it kind of like just define your reaction to that skipped update once and
then forever after it's that posture?
mark: yeah, indeed.
Indeed.
You do that once and then
duke: before.
I
mark: No, no, no,
duke: but it's just like.
mark: no, no.
Yeah, well, with Vancouver it got introduced, so
that's actually pretty nice.
, it will be automatically applied, so also when there's a Patch or a
hotfix or with the next upgrade.
So, , Xanadu or Yokohama, , so then it will just automatically , be applied.
There's also another table which keeps which records have been
applied, , which skip the record rules, , have been applied.
So it's also pretty nice.
, but there's more to skip the record rules because you can
also, , for example, at tax.
To skip the record tools.
For example, the customer I was talking about now, yeah.
Eight different serves now team.
So actually all the skip records are divided over the different teams, but
based on , some conditions, you can actually give those skip records a tag.
So one team will already have automatically.
Then skipped records.
Rule.
, skip the records on their team.
And another team will already automatically have 60 skipped
records, , on their team.
, so also with that, , it can help.
So that's actually a pretty good.
Yeah.
Nice enhancement , from the Vancouver release.
Yeah.
, and actually you can start really basic with it, right?
Just use a condition, just a condition builder, click, click, click, and, , and
you're done, but you can make it more complicated by adding a script include
in to the condition, for example.
So you can do it, no code, but you can do it also a bit more.
, it's a good, good addition.
Definitely.
duke: Wow, sounds like a good spot to leave it, huh, Corey?
CJ: yeah, man, this has been an amazing episode, Mark.
Again, take ownership of your own journey.
And leaving us on that nugget about the skip record I think is just right.
A, The sort of thing that you are known for in this ecosystem, right.
It's just, , dropping those jewels on folks that, , in parts of , the
instance and the, , on the platform that, people aren't really looking right.
And , they aren't really hanging out.
So I, I love it.
And I look, man, , this has been , a great call, especially considering, this
was, a bit last minute on top of it.
Right.
mark: Yeah.
Yeah.
duke: on a wall somewhere watching people pass by and just winging
a big fist sized jewel at him.
Ah, mother fu Oh!
What was that?
Oh, it's an emerald.
Okay, I'll take that.
CJ: No, I mean, I think this is going to, I think our, I think, , the
audience is going to love this.
And, before we started recording, , I was telling, you know, so yes to everything.
. And then, so , Mark is living up to it.
. We reached out and he's like, yes, and here we are.
And now we've
mark: Yeah, of course.
CJ: an episode.
Yeah.
So it's
mark: Yeah, I do want to, I do want to mention I did play a role in this
whole upgrade and also with the initial idea of the upgrade in a week, but I
didn't have to execute all of it myself.
Actually, I only did limited things myself during the week.
, so there were, yeah, there was just an involvement of a lot of people.
And especially from, , from , two people, one, from the support, , site and one,
from the owning the search now platform.
And those were also really key, , in this , , whole journey.
I do want to, , mention that.
So it's not like that.
I'm just doing some magic and everything is happening.
It's not that, , it's not all credits to me or something, but, uh,
duke: I guess we're gonna have to take back every good
thing we said about you, Mark.
Oh, shoot.
CJ: Love the modesty man.
Love the modesty
duke: we're at 40 minutes of record.
We gotta cut it off, guys.
Alright, thanks so much, Mark.
CJ: Thanks Mark.
duke: Still
mark: worries, a lot of fun.