Authentic, Authoritative, Unapologetic ServiceNow commentary by Cory "CJ" Wesley and Robert "The Duke" Fedoruk
duke: Okay folks, today we
have a, , huge treat for you.
I was just browsing the, , SuperSecretMVP
channels, Where all the MVPs chat and
plan to take over the world, such things.
And I saw a story about somebody
who had upgraded their big financial
client to Washington in eight
days, using no exotic external
tools, just what's in service now.
he is a friend of the channel already.
, and so we welcome back Mark Rothoff to
chat with us again about this amazing
upgrade experience while it's still fresh.
,
mark: Thank you for having me.
It's always a pleasure.
Let's have some fun.
duke: for sure, man.
So, set the stage here.
What version were they at?
How big an instance, how complex,
what were we staring at down the
barrel before we decided, Hey, let's
upgrade this thing in eight days?
mark: Yeah.
, actually, , they were a version
behind, so they were actually on, , on
Utah, , and that's all already, , a bit
of the reason why we went to, , Yeah,
the upgrade in a week because, , I got
some pocket change, , to, make this
change and , to , upgrade in a week.
the instance itself, it's, , one
of the largest financial
companies in the Netherlands.
The instance is not that big
anymore because I've been busy with.
My database footprint work.
So, , the footprint, , was cut in half.
, so now it's, , only , seven terabytes.
Uh,
duke: terabytes.
mark: it was a bit more, but yeah, then.
I came along, that's my job, I guess.
duke: Uh huh.
mark: a lot of code.
Custom work.
, Also some things really legacy.
It's, , still a lot of workflow,
for example, and, , still not
really applying flow designer.
, but yeah, , it's just an instance like.
, we see a lot, I guess.
that's a little bit,
background about the instance.
Yeah.
CJ: the thing about your post on the
super secret slack channel, . That
really, I think takes something
that's already extraordinary.
and then elevates the two, like the
next level of like outer space is that
you said that you did it all using
only , the free tools in the platform,
mark: That's correct.
Yeah.
, just, , normal upgrade, preview, upgrade,
monitor, , also with the cloning, just
the clone definitions, the clone preserves
excludes, , clone cleanup sketch.
, ATF is in place, , also some bits are not
in place, , so like, uh, we're not, not
using CloudRunner, still we're not using
CloudRunner, we really have to, so that's
also one of the things for the evaluation,
, which we'll have in a few days.
But yeah, just only out of the box, , free
available plugins and capabilities.
, so we also actually used a few of the
new Washington bits or actually Vancouver
bits, , like the skipped record rules.
, but we couldn't apply it automatically
because the customer is coming from Utah.
We had to run them manually,
but, , yeah, we also applied that.
So, uh, really nice to see, , because
I guess not that many customers
are still already using it.
, but yeah, things like, , skip
record rules can help.
, but yeah, just the out of the box
available, , free capabilities,
, yeah, and things like ATF.
So, yeah, , it's all possible.
duke: I think we should break some
of those down a bit because there
might be people, some of them might
be on this call, who have never
heard of a couple of those things.
So I know that , the clone
management tools are awesome.
ATF is.
ATF.
, Cloudrunner.
Tell us what Cloudrunner is.
mark: Ooh.
What is cloud runner?
Basically, if you would, run ATF.
, especially a few releases ago, then
you would run ATF, keep your, , laptop
open, keep your screen open, keep
it running on the background, but
you, yeah, it was really a attended.
Thing, right?
You really have to be on it.
, and then a few releases ago serves
now already introduced something that
you could spin up your own, , Linux
or windows server, and, , you could
run the, , the ATF, , from there.
, but now since I don't know, two or
three releases, something like that.
I don't know exactly.
, serves now also has.
Cloud runner, , where you can execute
two, uh runs at the same time.
I don't know the exact
wording, but you can run.
Yeah.
Two, two runs at the same time.
, on something that serves
now is actually hosting.
If you want to perform
more at the same time.
, then you can buy that, but at
least you can, , yeah, run to,
yeah, the two runs in parallel.
, so you don't have to have your
laptop still open or whatever.
You would just do that to cloud runner.
It's also pretty easy to set up.
We also explored it before at the
customer, , at, , innovation Friday.
And it looked really nice, but yeah,
then it comes like we need to pick it up.
It becomes a story, a backlog, et cetera.
And then.
Yeah, then it gets a bit quiet.
So that's still one of the
things we do need to apply.
, but yeah, that's one of the
things for the evaluation.
duke: So the advantage of cloud
runner is that you don't need
to have your laptop open and
mark: yeah, that's one of the
advantages of course, but especially
when you're talking about upgrade in a
week, , then the timelines are so short.
When are you going to do those tests?
Well, ideally you would start them at
six o'clock in the evening or something.
Um, Um, right.
And not at daytime or that someone
especially needs to jump on their laptop
at 10 o'clock in the evening or something.
, so yeah, I think it has
way more advantages.
And also if you can do two tests
at the same time, why can't you?
Shouldn't we split the
ATF suite for tests?
, so now the execution takes
about eight or nine hours.
Well, maybe we can run two at the same
time and it only takes five hours.
, and you don't have to attend
your laptop holder all the time.
duke: But it still dumps all the
findings and stuff in the same place
as if you had attended it on your lap.
mark: yeah, the eventual results
and everything that stays the
same, , there's no difference in that.
So it just makes life easier.
, you can really run it in
attended,, at every time.
, it makes you way more flexible.
, , I see good, , potential for that.
Yeah.
CJ: Did you set out to set the record for,
, world's fastest, , upgrade of a humongous
company, or did this just happen by like
happenstance and you were like, you got
into, and you were like, you know what,
I think I can get this done in a week.
mark: Yeah.
, For such a large company,
it's quite impressive.
, I did do a upgrades of a week at smaller
companies, but then you're talking about
instances of less than one terabyte.
. And then also as a really small user
base, but this is quite interesting.
, but yeah, then, it's also a bit
like, where did this come from?
And, , I'm now with this company.
For I think one and a half
years, I'm working part time
for them, , three days a week.
And, , the first upgrade that
I saw there was, yeah, it's
the typical four or five weeks.
, a lot of, , waste, , I'm seeing
it in the, yeah, in the time spent
, and, , how the testing goes and
with stakeholders and everything.
So already the, upgrades following we,
Improved a lot and also the change freeze
was also limited to a few days instead
of like one and a half week But still
I was like, come on, we should shorten
this we can do this within a week.
Come on and um It's pretty difficult in
such a large organization to get things
moving and changing But, um, yeah,
this is one of the things, but this is
one of the things we did get moving.
And it, it also was a bit because, , we
actually skipped the last upgrade,
, because we actually, , we're planning
to upgrade in January to Vancouver,
but then of course we have, , like
eight different service now teams.
, it's like 35 or 40, , developers
and then one of the teams just says,
yeah, we don't have time to upgrade.
Okay, so we're postponing the whole
Vancouver upgrade because one of
the teams thinks that they're busy.
Well, no one is busy.
, maybe some individuals are busy.
But no teams or departments
or companies are really busy.
That's for people who cannot
organize their own work.
, but yeah, one of the trade offs, but one
of the trade offs of that was actually,
well, when we're going to upgrade in a
week, so that's, that's a bit how this.
Um,
CJ: yeah, I mean, that is crazy to me
and you already touched on it, right?
The typical processes, , 4
to 5 weeks, right?
I typically recommend with
my clients about a 4 week
process in my document, right?
And I think I might have done
it as quickly as I could.
As three, if I'm following
a formal process, right?
Like when I was a customer and I
owned the instance, I probably did it
quicker, but I cut a lot of corners,
duke: Yeah, back when it
was just ITSM too, right?
And a couple of maybe custom apps.
CJ: Yeah.
Much smaller company too, right?
, and we really just running
like change management.
That was the only external
facing thing that we cared about.
And so , it was much easier to just flip
some switches and nobody really cared.
Like I couldn't imagine.
, upgrading a humongous, , financial
organization on top of that and a
week, that's just amazing to me.
So, kudos for even, taking that on.
And especially, it sounds like you
already had, like, you've been there a
while, so you have some history, , with
these folks in terms of , being not
necessarily being the most receptive
to the upgrade process at all.
, . mark: Yeah, it's also because you
are dead and already, , for like
one and a half year, you see how
it's a dynamics within the company.
What's happening?
Where is all the waste?
, so.
And then you're thinking, we should
be able to do this differently.
And, , obviously we did also do some
other things like in the background, I
was working on instance scan, so I could
run it before a clone, after clone,
before an upgrade, after an upgrade.
So you would see some things.
, obviously we did upgrade one of
the instances already a month
ago, , to see, Hey, what are the
skipped updates on patch zero?
So we know a bit, what can we expect?
, and we knew, , Oh, one of the
things on employee center broke.
So at that moment we could already
create a support case , for
something with AI search.
, so there's a little bit of
preparation, , but the upgrade
itself, yeah, that's just, yeah.
One week.
It is quite challenging though,
because yeah, , you want to clone
instances, you want to upgrade
instances, you want to run your ATFs.
If you already look at the.
The timelines for that, the duration,
like the clone, , also because
they do, the upgrade on demand.
, so the clone already takes
like eight or nine hours.
The upgrade itself also about eight hours.
Then you have ATFs also eight, nine hours.
So the
duke: you're left with four days.
mark: The timelines are getting, pretty
heavy or pretty, um, yeah, I don't know,
duke: you said it took like seven
days, were the hours you counting?
Is where these things were just chugging
along, minding their own business?
, mark: no, it's just really, , in that
week, , some of the evenings, nothing was
happening, , but some of the evenings,
indeed, the clones, , were just, , going,
the upgrade started at ten, uh, at
night on, , on Friday night, so it was,
, finished on Saturday morning, , It's also
a bit like just do it, , because if you
talk about customers with this, or even if
it's for two weeks upgrade, then there are
so many objections and people jumping into
fences and they see all kinds of issues.
So you also just have to do it.
, and also in this case, it's
Washington patch one hotfix two.
It's.
Yeah, pretty fresh.
And what you still hear a lot of people
saying it's like, Oh, let's wait a
few patches and before it gets stable
and the first issues are out, but
that's people mentioning that because
they hear other people saying it.
And because people are already saying
that for years and years and years,
it's not, ServiceNow is not how
it was five years or 10 years ago.
, if now patch one comes out.
We already had patch zero before and
already the first hotfix is coming
and the second hotfix is coming and
now actually already patch two is out.
So the level is way
higher than it used to be.
So you don't need to wait for
patch four, five, six or something.
That's also, yeah, that's ancient history.
So people talking about those things,
they're a bit stuck in the past.
CJ: Yep.
So, Mark, , you mentioned a lot of people
popping out and saying basically no.
And I think one of the, , one of the
keys , to the success and being able
to deliver on this, . Is knowing
who to talk to, to get a yes.
So talk a little bit
about how you cut through,
.
All of those no's to
find the right person.
To green light this thing and
get you that, yes, that you
need it so that you can do it.
, mark: I didn't do so
much myself in that area.
I just on the background, I
was, , talking with, well, mainly
three, , people within the company.
And, also , , the owner of, , of
service now within the company.
, and eventually it.
He also just had so something
like we just have to do it.
And also because, , before also we skipped
the previous upgrade just because of time
constraints and we're already bad behind.
And, so they also have a feeling
like, Hey, we need to do something
and we need to make progress.
And we do want to also save time for the
future, because if we can pull this off.
duke: Okay.
mark: was, yeah, I don't know.
It was also just, , just do it.
And, , yeah.
And also the, just the people really
owning serves now also really need to
be enthusiastic about this and see the
opportunity and the possibility and just.
Just do it and, , and just believe in it.
And, , yeah, one of the big differences,
of course, , is that if you also do
this for three, four, five weeks, and
it really depends on how many people
are in the company, of course, but then
not all the service now teams or all the
service now developers will be dedicated.
Busy with upgrading or
dedicated available.
They will just also be busy
with doing other stuff.
If you do this upgrade in a week, then
way more people are dedicated available.
, so that's also a key factor,
to getting this, , to a success
and even that, yeah, yeah,
duke: I would, I do want to make
sure that we hit this one point.
Cause I don't think anybody can decide
, Hey, we're going to do this in a week
like, and then execute it that way.
I believe just because it's
you, that was there, that there
was significant amounts of.
Effort put in beforehand such
that you could execute in a week.
Is that true?
mark: yeah, I think that that helps.
Um, and also.
duke: all in one big bite or like a
little bit at a time, but I think, I think
there's a lot of customers that don't even
have like ATF running in any capacity.
So I wonder if you could talk to
the level of effort to get cloning
management ready, ATF, cloud
runner, the skipped record rules.
, what was the journey to get to the point
where you could do one week upgrades
mark: um,
duke: change freeze?
mark: yeah, well, I think actually
we still have a lot to improve.
, if you look at the cloning process or
how many manual steps we're, , we're
doing or how long the process, , takes,
, but already a lot was in place.
, and also, it helps that.
I'm there.
Yeah.
It helps that you have a certain name
in the, space, a certain track record.
, that's true.
, we did, , perform an upgrade a month
earlier , on a different instance
to just see how big is it, , what
is happening, , what is going on.
I was already working on instance
scan, , so that customer also now has
like 800, , scan checks, , in place.
, ATFs.
Yeah, a lot of a TF , is being used.
There is and is in place.
, still during the upgrade in the
week, they expanded it massively.
, but there was already a lot in place.
, if you don't have the, cloning and
the control, the at don't, , you
don't have ATFs, , or you have, yeah.
A lot of stakeholders that want to do.
every single piece manually, then it
becomes really difficult, of course,
then you, yeah, then pulling something
like this off will be really difficult.
So yeah, you do have to have some
pieces in place and , some of the
teams also do almost all of the
stakeholder testing themselves.
So also the number of stakeholders
involved is, pretty limited.
, But , if I look at the previous
upgrades, which took three, four,
five weeks, then you would just
say, Oh, we're going to upgrade and
you can test and whatever, and then
it's, , yeah, it's all over the place.
Well, when are people going to test?
And now it's in one
week and you just have.
Yeah, the message to the stakeholders
was that's the day that you can test.
There's no other day.
That's the day.
Not everyone really likes being put into
pressure like that, but it did work.
And also because it did came from,
, , yeah, , higher management or yeah, people
a bit higher in the hierarchy in the
organization, because I can have a, a
certain name in the space, but if I'm
just mentioning something like that, yeah,
and still it won't be adopted, right?
, I'm not, , in the hierarchy
of a company, to make such
decisions on my own or whatever.
That's, , that's not going to work.
CJ: Yeah, I want to follow
up a little bit on this too.
. Because, , I think , there's , a really
good relationship here between A lot of
the things that you do in the service
now ecosystem and what enabled you to
actually get through this in a week.
, and I'll and 1 of the reasons that
you're such, an amazing service now MVP.
Right?
And so a couple of weeks ago, you
made a post on linked in, celebrating.
. And I say kind of celebrating, . Cause
you did it in such an understated
way for this accomplishment.
Right.
And, , but you made a post on LinkedIn
talking to celebrating, , , that
your, combined service now
content had just reached the 1.
25 million view plateau, right?
That's like over a million.
, I can't even.
Put the data, I get, you know, it's
hard to even put it in the words, right?
You're combined service.
Now content, . Has reached over 1.
25 million views.
And when I, is that right?
First, congratulations on that.
That's just amazing.
Right.
duke: In another 30 years, we might reach
CJ: Right.
Like CJ and the Duke in 50
years might be at, at the 0.
25.
Right.
duke: Creepers.
mark: No worries.
I'll contribute to that because I watch.
Every episode.
So, uh, yeah, so I'll,
keep contributing to that.
CJ: Appreciate that.
. But when you dive into it, , because
then you link to a, , to your service
now, a community page, which has
like the entirety of your catalog.
. And, , and so in that catalog, I
mean, you've got over 325 articles,
blogs, videos, podcasts, et cetera.
And.
, when I looked through that, they, first
of all, , your reference, , for me, right?
Like I, , I have your content on
bookmark and whenever I, , need
something and I can't quite figure
out, I go and check your library to
see if you've already written about it.
Right.
duke: Yeah, crazy.
Um,
CJ: your previous body of work and
effort and talk about how, you
mentioned insta scan and the earlier
in this conversation, you mentioned
ATF earlier in this conversation.
These are things that you've
written about, , extensively that
you share with the community.
And, , and all of that stuff forms,
like the underpinnings of being able to
, execute on something like this in a week.
, so I just want to talk about a little
bit about what drives you to create
content to this level, man, you are
a machine and it's good content.
mark: yeah.
Thanks.
, yeah, what drives me?
, It's just also a lot of fun to create
content and, , , and see the feedback
or also like, . Now, what, you're just
saying, that's already a good motivation,
, and , to keep on going, , but it's
also, I just started creating content,
, because of the surf now ecosystem.
And because yeah, when I started, I
also started from the bottom and I
was just searching and, , Oh, we've
got the search on community and, Oh,
we had Ricky and all those YouTube.
Things and Oh, , the YouTube videos
from, , from the Duke and, , Oh, awesome.
And yeah, why not contribute also
myself and let's see if I can do that.
But eventually it was also for
me back then I was still working
at the consultancy company now.
I'm doing this freelance, but back
then it was also, , you had this
mandatory personal development plan
and the company decides what you're
doing on soft skills, hard skills.
So I was also thinking like, Hey, I'm
creating this content and actually
I'm working already on my skills.
And I'm taking ownership on
my own journey now, , because
I have to dive into subjects.
I need to learn more about it.
, so my technical level is
evolving or in different areas.
And that's also fun.
So I don't have that mandatory
personal development plan anymore.
I'm doing it myself.
I'm taking ownership and I'm doing it.
Yeah, with fun, , but not your hard
skills, also the soft skills, because,
English is not my native language.
Everything I do is in English.
How do I bring a message across?
obviously I do a lot of, , in writing now,
but maybe eventually I will just, , switch
to, podcast, who knows, uh, or to videos.
And then also you're
working on, on those skills.
And also then it goes more about.
Your presentation and et cetera.
So , it's way more and it's so much fun
also because you're taking ownership and
you're controlling , where you're going.
, but eventually it's also just
all , the replies and the feedback.
It's, , it's just a lot of fun.
, and yeah, and we are possible to
call it a collaborate with, , with
others, , not only MVPs, but, also
with the MVPs, , collaborating.
That's just a lot of fun
and, , just a lot of motivation
at that also just keeps you going.
So, , yeah, that's just the main
reason that, , yeah, so yeah.
CJ: that, that's awesome.
I love that, take ownership
of your own journey.
That should be almost be the title of
this, episode, cause I could just see
how it applies in so many different
ways, . From what we were just talking
to, but , from also , how do you get.
From zero to eight days, , do an
upgrade in a huge financial, , company
, and zero to eight days, right?
Like you got to take
ownership of that journey.
Right.
And, and then, in part, like
that eight days started long ago
when you implemented Insta scan,
when you implemented ATF, right?
Like when you made those, , when you
exercise those soft skills and made
those relationships internally, right.
And understood like how the
business works and understood.
The, who actually had, work
that they could put off.
, and, , who was sandbagging a
little bit in terms of having
availability to do testing, right?
Like all of those things
came into play here.
And all of that to me, can
be summed up and to say, take
ownership of your own journey, right?
If you want to get success
in a service now ecosystem.
So man, kudos to you on that.
And I love that.
And that we're going to do Duke.
That's the title.
And yeah, I'll give you that
one mark for your charge.
, duke: Mark, if you don't mind speaking
of owning your own journey here,
cause I'm trying to get some, taking
some notes for a customer of mine
who wants to do everything better
in service now, including upgrades.
Tell us about why skip
record rules are so awesome
mark: Oh, why are they awesome?
duke: Or at least handy.
Yep.
mark: upgrades.
Some customers will have
a hundred skipped updates.
Some will have 500, , every release.
And actually it's not
only during releases.
You also have skipped records
during batches and hot fixes.
, you also should handle those.
I do see that's.
Being ignored at a lot of companies.
, but especially with upgrades, you can
have a lot of skips, skip records, , skip
records themselves do take them seriously.
, because some are high priority,
some maybe of lower priority,
but actually do pick them up.
, maybe I should immediately
give an example.
You can have a priority five, which is
a form layout form layout on service
catalog because serves now added some
new fields, some new check boxes.
But if you are not taking that over,
or if you are not merging that.
you are missing out on
that new functionality.
, or it will not land on the form layout.
So most developers won't dive that
extensively into the release notes.
So they won't know of
that no new functionality.
They won't know that you can do
things without skipping, , with,
, with less technical depth.
, you can do it really quick, more quickly.
So you really have to Take
those skipped records seriously.
And also those priority fives, but
where does skipped record rules come in?
, a good chunk of the skipped records
will be handled every release in the
same way, for example, because it's,
, notifications, which you on purpose
deactivated, you're not going to activate
them, , during the upgrade or merging or
whatever, no, you did that on purpose.
or we have a ton of skipped errors,
which you actually cannot handle.
That's also literally in the service now
docs and that you should retain them.
, and several system properties, which
you absolutely don't want to refer it or
merge, you just always want to keep well
for all those situations, you can make.
So that those skipped records are
automatically being handled for you.
And especially if you have two or three
hundred skipped records, then half of it
can just automatically be handled for you.
, so that's a huge saver.
and you don't have to go
through it manually anymore.
So it's less of the repetitive,
, Not so attractive work, , less
chance of making mistakes in that.
And so that already helps.
, so that's one of the parts of skip record
tools that you can handle those skip
duke: is it kind of like just define your
reaction to that skipped update once and
then forever after it's that posture?
mark: yeah, indeed.
Indeed.
You do that once and then
duke: before.
I
mark: No, no, no,
duke: but it's just like.
mark: no, no.
Yeah, well, with Vancouver
it got introduced, so
that's actually pretty nice.
, it will be automatically applied,
so also when there's a Patch or a
hotfix or with the next upgrade.
So, , Xanadu or Yokohama, , so then it
will just automatically , be applied.
There's also another table which
keeps which records have been
applied, , which skip the record
rules, , have been applied.
So it's also pretty nice.
, but there's more to skip the
record rules because you can
also, , for example, at tax.
To skip the record tools.
For example, the customer I
was talking about now, yeah.
Eight different serves now team.
So actually all the skip records are
divided over the different teams, but
based on , some conditions, you can
actually give those skip records a tag.
So one team will already
have automatically.
Then skipped records.
Rule.
, skip the records on their team.
And another team will already
automatically have 60 skipped
records, , on their team.
, so also with that, , it can help.
So that's actually a pretty good.
Yeah.
Nice enhancement , from
the Vancouver release.
Yeah.
, and actually you can start
really basic with it, right?
Just use a condition, just a condition
builder, click, click, click, and, , and
you're done, but you can make it more
complicated by adding a script include
in to the condition, for example.
So you can do it, no code, but
you can do it also a bit more.
, it's a good, good addition.
Definitely.
duke: Wow, sounds like a good
spot to leave it, huh, Corey?
CJ: yeah, man, this has been
an amazing episode, Mark.
Again, take ownership
of your own journey.
And leaving us on that nugget about
the skip record I think is just right.
A, The sort of thing that you are
known for in this ecosystem, right.
It's just, , dropping those jewels
on folks that, , in parts of , the
instance and the, , on the platform
that, people aren't really looking right.
And , they aren't really hanging out.
So I, I love it.
And I look, man, , this has been , a
great call, especially considering, this
was, a bit last minute on top of it.
Right.
mark: Yeah.
Yeah.
Yeah.
duke: on a wall somewhere watching
people pass by and just winging
a big fist sized jewel at him.
Ah, mother fu Oh!
What was that?
Oh, it's an emerald.
Okay, I'll take that.
CJ: No, I mean, I think this is
going to, I think our, I think, , the
audience is going to love this.
And, before we started recording, , I was
telling, you know, so yes to everything.
. And then, so , Mark is living up to it.
. We reached out and he's
like, yes, and here we are.
And now we've
mark: Yeah, of course.
CJ: an episode.
Yeah.
So it's
mark: Yeah, I do want to, I do want
to mention I did play a role in this
whole upgrade and also with the initial
idea of the upgrade in a week, but I
didn't have to execute all of it myself.
Actually, I only did limited
things myself during the week.
, so there were, yeah, there was just
an involvement of a lot of people.
And especially from, , from , two people,
one, from the support, , site and one,
from the owning the search now platform.
And those were also really
key, , in this , , whole journey.
I do want to, , mention that.
So it's not like that.
I'm just doing some magic
and everything is happening.
It's not that, , it's not all
credits to me or something, but, uh,
duke: I guess we're gonna
have to take back every good
thing we said about you, Mark.
Oh, shoot.
CJ: Love the modesty man.
Love the modesty
duke: we're at 40 minutes of record.
We gotta cut it off, guys.
Alright, thanks so much, Mark.
CJ: Thanks Mark.
duke: Still
mark: worries, a lot of fun.