Clinton: Welcome to Testing, Testing 1-2-3, a podcast
brought to you by TestGenius.
Jenny: Welcome everybody.
My name is Jenny Arnez and I'm from TestGenius.
We have today, Mike Callen, VP of Products at TestGenius back with us, as well as Dr.
Clinton Kelly from ioPredict.
If you've had a chance to see our previous session with Dr.
Clinton Kelly, you know that we talked about all things personality testing.
We have Clinton back today to talk about personality testing, specifically in
the 911 dispatch environment, and using CritiCall, our dispatch assessment.
Mike, do you want to say anything about CritiCall before we jump in?
Mike: Yeah, absolutely.
So CritiCall, as most people in the space know, is a skill and ability testing
program that, that leans very heavily on multitasking ability while being
able To receive auditory information and enter it into a computer, which
is one of the most core abilities that is necessary for success on
day one in the dispatch environment.
And I would say the dispatch environment in terms of our conversation has more
than means more than 9 1 1 it's police, fire, it's EMS, it's 9 1 1 utility.
Ambulance, there's all sorts of environments where the job
that's being performed is very much in a stressful situation.
You're under duress and it's a CritiCall incident, meaning that the effects
of making good decisions can be very positive and the effects of making
poor decisions or performing poorly can have incredibly negative effects.
Implications, including even death.
So it's a very important environment to be working in.
CritiCall is a product that is a product of Biddle Consulting Group.
It runs within our TestGenius suite.
And a few years back, we had the great opportunity to work with Clinton
and his firm to create, ground up a.
Dispatcher specific personality test.
And that's what we're really going to be meeting today to talk about.
Jenny: Wonderful.
Clinton, you want to just introduce yourself real briefly for those
who didn't get a chance yet to watch our first session with you?
Clinton: Yeah.
, I'm a Clinton Kelly.
I'm with ioPredict and I have a background in industrial and organizational
psychology and I specifically work on helping organizations to implement
and validate tests for hiring.
And so that's what I've done for almost, I'd say about 20 years now.
It's implementing hiring tests in organizations.
Jenny: Wonderful.
Thank you.
So personality testing and CritiCall.
Why does it matter?
Why is it important?
Clinton: It's, like Mike said, there's, there are multiple things.
There's, there are, can you technically do the job?
Can you work with computer screens?
Can you be taking in information, auditory information and typing at the same time?
And that's the technical, can you do the job?
And then there's the personality side is, are you a good fit for this job?
Do you have the personality type that is conducive to being successful?
And so both of those pieces of the puzzle are important parts when it
comes to making good hiring choices.
And so that's why it matters.
And these are like Mike said, these are stressful jobs, typically
working longer shifts, dealing with situations that aren't always
pleasant, and you need to make sure you have the right fit for this job.
And so that's really important.
Mike: I have had the pleasure of going to many conferences in the
space and there's a conversation that occurs at least once or twice.
Every time I'm at a conference where I'm talking with the center director,
and we're talking the profile of this person is somebody who's been around
the, this particular vertical for 20 plus years and they have a really great.
Grasp and feel of the space.
And what these folks will tell me very often, and I never disagree with
them is that, I've been in this job so long, I can sit down with an applicant.
I can talk to them for five minutes and I can know whether or not they're
going to be successful on the job.
And they don't say that to me in a way where they're arguing with me
saying they don't need to do testing.
Not at all.
What we all understand about that particular situation is that, yeah,
they probably have a really great feel about who would be successful
and who wouldn't be successful.
The problem is that particular process doesn't have a paper trail.
That explains why somebody made an affirmative decision or why
they made a negative decision, and it doesn't leave the applicant
feeling very good about the process.
I remember in years past, I applied for jobs where I thought I was
super highly appropriate for it.
There was a.
bogus selection process.
I didn't get hired and I felt really bad about that.
So one of the things that we want to do on the testing side for CritiCall
and for this, these dispatcher positions is we want to provide a.
process that's holistic and really one that does what these center directors
know that they could do, but creates a paper trail at the same time.
And Jenny, you asked Clinton about why, you want to do the
personality testing in this space.
And one of the things that we look at in testing and selection is you
look at Job performance as like a pie with different pieces in there.
And so you can have tests that measure audio data entry and map reading and
multitasking and those kinds of things.
And each of those pieces of pie are going to represent different
aspects of job performance.
And so the goal in a testing process is to try to get as many
pieces of that Pie filled up with CritiCall performance dimensions.
And some of those are going to be hard skills.
Some of them are going to be, soft skills, situational judgment.
And many of those things are going to be personality, particularly in this
realm, because it's such a difficult job is you want to know as much as you
can know from a job related perspective about the personality of this person.
Clinton said, specifically to be able to find a good job.
a good fit.
And so that's really what our goal is with this particular product right here.
Jenny: So if I was a hiring manager or a sergeant in an agency
that's implementing this CritiCall and using the personality test.
Why it's important to me is that I know that I can hire basically the right person
for the job who's going to stay too.
Is that correct?
Clinton: Yeah.
We've shown with how we built the test and the validation study that
we've done, we've shown that higher scores on this test are more likely
to be rated As higher performers on the job by their supervisors.
And so there, we have actual data that says, Hey, it doesn't get, and there's
no guarantee cause as Mike said, there's a certain amount of, we can't predict
up with a hundred percent accuracy, but we show through our study that
higher scores on the test, they're more likely to be rated as higher
performance by their supervisors.
Mike: Why don't you talk a little bit Clinton about how
this particular test was created.
You, there's a lot of standard personality tests that have the same 110 items.
They get asked, everybody gets asked the same question, but
they're scored or keyed differently.
This test wasn't an off the shelf test.
This was created ground up.
And, when you talk about that relationship between job performance
and the items, talk about how that was established a little bit.
Clinton: Yeah, so what we did is we looked and we analyzed the
job and we did this analysis of the job to see what is important.
We talked with different agencies to see, okay, what are the important
characteristics of individuals who are successful on the job?
And so we custom developed this test from scratch specifically for dispatchers.
And we designed, we built it with the attention to measure
things like adaptability.
With a heinous job, it's important that people can adapt and on the
fly, they get calls and have to figure out how to handle things.
Another one, one of the things we tend to measure is composure.
You have to be able to remain calm when you're on a call.
Another thing is resilience and not being able to let a certain,
be affected one call to the next.
You have to be able to let that one go and move on following procedure and policy.
That's also another thing you, while it's important to have the adaptable,
you also need individuals in these types of roles who are not going
to fly by the seat of their pants, that they know what the policy and
procedure is in certain situations.
And then one of the other ones is confidences.
You have to also be able to take some, I say, be a little bit assertive
and be confident on the calls.
When you're in these situations and then lastly multitasking a
preference for multitasking and that's pretty apparent for anyone
who's been in a dispatching agency.
If you've walked into a room, you just see the multiple screens, the
phones and things that are going on.
There's lots of things done at the same time.
So just individuals who have a preference or have a preference for
multitasking behaviors is another thing we were attempting to measure.
Mike: And you bring up a good point.
When you talk about these, you were talking about which one was it?
Adaptable, sorry, create being creative.
I can't remember exactly which one it was, but the point that you're
making is that there's a, this whole continuum of this particular scale.
And there's the very top end of it and the very low end of it.
And there's actually a sweet spot within each of these areas that, that
you want people to fit into because the highest performers are there.
It's not that you want people who are rated a hundred out of a hundred on all
of these particular areas, rather you want them to fall within a certain range.
How do you determine that?
Clinton: Yeah, so what we did in this case is we actually, we developed a number of
different test items designed to measure those areas that I was just talking about.
And we collected data from a number of variety of different dispatching agencies
throughout the United States and Canada.
And so we asked them, we said, hey, we got their employees, current
dispatchers, and we said, can we get them to take this test?
And then we created a supervisor a performance rating scale.
So in the, what you can see on the screen here, we created a scale where supervisors
rated the individuals who took this test.
They rated their performance on these different areas of being adaptable,
flexible, ambitious, motivated.
And we had them rate those employees.
So we had people throughout the U.
S.
and Canada who are current dispatchers take this test.
We had their supervisors rate them on their job performance.
And then we looked at, we then evaluated the relationship.
Are there patterns in the data that we're seeing that people who are higher or lower
on these questions or on these scales that are more likely to be successful in
the job as rated by their supervisors?
So their supervisors say they're more successful and what patterns
on these different test questions result in better dispatchers.
And sometimes you can get lucky.
What you can do is you can get lucky.
Like I like it is you can throw spaghetti against the wall and maybe it makes
some magical image one time, right?
But can you do it again?
Can you replicate it?
And so what we did here to avoid making sure that we didn't just
capitalize on chance, we didn't just get lucky with these questions.
Like we threw all this stuff in the black box.
And while magically people who preferred to eat, sushi versus
pizza are better dispatchers.
We might, you might find that pattern in data once, but we did here, as you
see on this slide here, see sample one and sample two, we collected
data From all these individuals.
And we randomly split the sample in half.
We randomly said, okay, you're sample one, you're sample two.
We did a random split in sample one.
We said, okay, these questions predicted performance.
And then we use that, we said, do they also predict performance in sample two?
And that cross validation, that helps us to make sure that we didn't
just get lucky the first time.
And you can see these correlations here.
This is, not to get too deep into the data here, but these correlations, when there's
stars next to them, these means they are significant predictors of performance.
They are predicting job performance in both sample one and sample two.
And so we're able to show that we're consistently predicting performance
and didn't just get lucky in the way of the, some unique, just saying
that we just capitalize on chance.
Mike: And one of the things that I think that's unique about this
particular personality test is that you guys wrote all these items ground
up specifically for this space.
So this isn't, one of those personality tests that's talking about, how
much you enjoy long walks on the beach or crowded movie theaters.
It's talking about, real situations that pertain to this environment.
Can you talk a little bit about that?
Why is that important and how does that help?
Clinton: Yeah, it makes it feel a little more like the job.
We talked about that, call that call.
Cause some face validity, it feels a little more job related
to those taking the test.
And so what we have in the way we set up this
test, they are what we call paired like a forced choice personality test
and it's where we have two different statements and we say something like
i'm more likely to And on the left hand side, I might say follow the rules
and the right hand side is change the rules, And so we have that don't you?
Yes, we do have an image of that You That is on right here.
Right here is an example.
This is a silly example here, but let's just say I am more likely
to eat a hamburger or eat a salad.
And so you have the ones on the left or on the right, and people
indicate where they fall in this continuum of their preference.
We set up the questions, the ones on the actual test to be,
so they've look more job related.
For example, something like this, I'm more likely to, on the left hand
side, make quick decisive decisions or make long thought out decisions.
Inherently, neither one of those is bad.
Someone who makes long thought versus quick decisive.
Sometimes there could be benefit to one or these the other, depending on the job.
But in this particular job.
We actually, you need to, you are typically respond, responding quickly.
You need to make quick, decisive decisions.
And so we were looking, are the people who make quick, decisive decisions, are
those who are more likely to be successful in the job as rated by their supervisors?
And so those are the types of things we're looking at with these questions.
We try to write them in ways that are like Mike has said, more related to the job.
Mike: And that's borne out in the data analysis.
How many people did we work with on this?
It was, seems like it was about what?
300, 320 people or so.
Clinton: Yeah, it was over 300.
Yeah.
And so you can see here three in the final sample, we had, I think we had 370 or 380.
In the overall of good data, when I say good data, people that fully
completed the test, we also had supervisor ratings of performance.
So we had some people who completed the test, but we didn't get ratings
of performance from the supervisors.
And some people were the supervisors gave us ratings on how they performed, but
the employee didn't complete the test.
And but of where we had good data, a little over 300 individuals.
And you can see, we randomly split that sample in half.
In each one, we get about 150 in those random splits of the sample.
Mike: Excellent.
I'm going to close my door.
My neighbor's dogs are barking at the most opportune moment.
I appreciate that.
Thank you for that.
Jenny, is there anything that's come to mind right now for you?
Jenny: As I'm looking at this list of performance criteria, and you
talked about designing the test.
Did you do some sort of a job analysis?
Is that what happened?
Clinton: Yeah, so we did do that job analysis and we looked at, and actually
at at Biddle and with TestGenius, they had done this job analysis actually
previously a couple years back.
And we used, we leveraged that job analysis to identify target areas for
potential development of test items.
But then we also had some phone calls with a number of different managers
of dispatching agencies throughout the country, where we asked them to confirm
just again, just as a second, confirm of what we were seeing in the job analysis
results, what are most predictive of success of dispatchers when it comes to
soft skills or the personality kind of different things, different constructs.
And we use that to drive the development of the content.
And one of the things I like to say, we custom developed this, like Mike had said.
But whenever you make a test, not just like on any team, on a sports team,
not all the players are all stars.
We developed these questions, not all of the items worked.
So when we originally developed the test, we had over a hundred questions
that were where we collected data.
Some of those questions did not predict performance.
It didn't matter which side they picked, which side of the
statement they were picking.
He said, Hey, people on either side were equally as effective on the
job according to their supervisors.
So those are questions that got tossed.
They didn't make the final cut because we said it didn't really
matter which side you were on.
These individuals, they didn't seem to predict.
These items didn't predict performance.
And so that's an important thing when you do this development is to
evaluate your item and see which ones are working and which ones aren't.
And those got tossed.
The ones that didn't work, we said, Hey, it didn't work.
We thought we might, but it didn't.
Mike: It's important to be able to know how to use this kind of test.
One of the things that we run into is that people have strengths and weaknesses, even
really high performing people have certain areas of their knowledge, skill, ability,
personality, personal characteristics that would be considered to be very high.
And they might also have some areas that aren't as strong because, no,
no man is a superhero or woman.
When we're looking at the results, we want to be able to take that into account.
And one of the things that I really love about the report is that Clinton, you
and Jason spent a lot of time looking at this overall performance and then certain
subscales that had validity coefficients.
But then to be able to ask probing questions.
So we have a saying in our environment or work environment here, not everybody
is good at everything, but it's important to understand where your
weaknesses lie, maybe more important to not think that your weaknesses
are strengths, but also to be able to
remediate those situations, to be able to understand this
isn't necessarily a strength.
So can you maybe walk us through a sample score report and show us, how is it that
a person who's a hiring manager might use this report, how they might interpret
it and how they might deal with some of these aspects of the scoring and maybe
some followup questions in order to get the best utility out of this report.
Clinton: Yeah, sure.
We can do that.
Let's pull up that report and let's walk through here.
So here is a sample report here.
We'll start to go through this is one I just completed.
You can see it gives a, some information just about the test
taker and their overall test score.
We give three bands of scores and those scores are falling into either
they are, I'm trying to remember off the top of my head, highly
recommended or somewhat recommended.
You'll see at the top right there, I got a highly recommended score.
We also give the total time that the person took on the test.
We give them a 30 minute time limit.
As you can see, I've completed this a number of times, but most
people, it's probably 10 minutes or less that are completing this test.
It doesn't take a long time to complete.
We give them plenty of time because it's not meant to be a speeded test.
In other words, we're not trying to say, Hey, finish this.
You better hurry up.
If you don't go fast, you're not going to finish.
We have some background here about the report and what we want to do is we
mentioned on this background is we want you like to really focus on the driving
factor really should be the overall score.
This overall recommendation, this highly recommended, recommended,
or somewhat recommended.
And the reason for that is because that's considering the most information
that's considering all of the test items that create the overall test.
And what we found from our test, for example, those who are highly
recommended, they have a 77 percent shot of being successful on the job.
And we, what we mean by being successful on the job is they
were rated as above average job performers by their supervisors.
So 77 percent of the people who have a highly recommended score, in our data
collection, they were above average in their job performance ratings.
If you drop down to recommended, 53 percent of those in the recommended were
above average in their job performance.
And if you go down to somewhat recommended, only 31 percent of
those individuals were above average in their job performance ratings.
So as you can see from those numbers, again, no test is perfect, but you
can see there, there is value as you go further down in those scores,
you're are more likely to bring someone on who is not going to be
very as good in their job performance and that the data bears that out.
Mike: I think this is very interesting in terms of having the personality
test coupled with the CritiCall hard skills test because with the
hard skills test there are certain skills and abilities that we're
expecting on day one of the job.
You want people to be able to hear auditory information and
enter it into a CAD system.
Obviously they're going to get trained on the CAD system, but they have to have
the core skills and abilities in place.
They need to be able to multitask under duress.
They need to be able to, read maps and understand, coordinates and such.
So these things are all very important and they're deal killers, right?
If you possess these skills and abilities.
You have the ability to move forward.
If you don't possess these skills and abilities, then you really don't
have any real reason to be continuing in the process because we're, the
centers are expecting people to come into this environment day one.
So if you take that, you say, X percentage of the applicants
have the skills and abilities necessary for success on the job.
Now you take them and you run them just those people,
through the personality test.
Voiceover: And we'll be right back after a word from our sponsor.
Ready to revolutionize your HR strategies?
Head over to TestGenius.com to discover our latest tools and solutions designed
to streamline your hiring processes.
Mike: You're getting a whole nother bite at the apple right there in terms
of predictive success of selecting the right person for the job.
So it's really your odds making you're putting the odds in your favor that
the hiring select or the selection decision that you're making is going to
be advantageous for your organization.
Is that right?
Clinton: Oh, that's absolutely correct.
It's a, you did a great job of explaining in common terms.
We, in every profession, you like to use fancy terms to make
yourself sound smarter than you are.
And we call that incremental variance.
And so then, yeah, and and just what that means essentially is those tests, like you
said, Mike, they tell us different things and that's what we mean by incremental.
It adds on to the information that a technical test tells us.
So technical tests of, typing auditory information, entering
things in that gives us that predicts differently than a personality test.
And that's just the term we call it incremental variance.
And so you're adding unique pieces of information that give you
basically greater predictive power.
Mike: It's interesting because now I'm starting to understand a little bit more
about how this fits in, because I think the next part of this report you're
going to show is the overall score.
And then you have subscales.
And I guess those subscales are some of that predictive variance as well.
Is that correct?
Clinton: Yeah, and that is as well.
So give me some of that predictive variance.
And so we have, like you said, on the second page, we have this
overall recommendation again.
It was on the top of the first page.
We're just going to visually show you the three options here.
And it's highly recommended and you can see here, we give us, we
tell you a little bit about this.
It was keyed using actual employees.
And so we want to let you know Hey, these individuals, this was, this is based on
data from actual people working in these types of roles that we aren't just making
this up and saying, Hey, we think these people are going to be more successful.
This is what the data has shown us.
Now, what we say here now, we introduce these subscales and we have these
subscales and they can provide unique information that then could be useful for
prodding in say, a selection interview.
And so we have one of the subscores that we have is dependability.
And so we give you some information on high scores, what that means for higher
scores on dependability and what that may mean for a lower score on dependability.
And then we give you some potential interview questions that you may want to
follow up with in the interview process.
Thanks.
And like Mike had said, just because someone scores lower on one of these
subscales, let's say I scored really you can see on this one, I'm like
confident assertive, I'm right in the middle, on that 58 on the confident
assertive subscale out of 100.
And I've seen this before where someone says, Ooh, Clinton's only a 58 on
the confident assertive subscale.
I don't know.
We, I don't think we should hire him.
What we need to go back to is again, this overall recommendation
should be the driving decision.
Now, what you can do with this is because maybe I'm moderate on this.
You could say, Hey, maybe you want to ask some confident sort of follow up questions
that we provide here on this page is in as part of the interview questions.
Hey Clinton, tell me about this, discuss this, tell me about a time.
And follow up on those interview questions to see if there's maybe any
sort of flags there or training needs or say, is this going to be a problem
if we bring him in or follow up there?
Mike: That kind of goes back to what we were talking about earlier with the being
aware of strengths and weaknesses, right?
Or not perceiving a weakness as a strength.
If a person, one example that we use in this dispatch space, one
of our tests gives the ability to collect a voice sample.
And if you collect a voice sample from someone, you've
never talked to them before.
They're just an applicant at that point.
And you realize that they talk really softly, there's this, you
want to jump out and you want to say listen, in this role, you need
to be able to speak with authority.
It isn't necessarily problematic that this person speaks really softly.
What's what matters is, are they aware that their natural tendency is to speak
softly, but they have the inherent ability to ramp it up when the time comes.
And I think that's the same with some of these right here and some
of these follow up questions.
Clinton: Now, yeah, and I also like to point out too, often, so those follow
up questions, are people aware of those weaknesses, but also, oftentimes, what
I call our strengths, often become our weaknesses, and what I mean by that is,
because we know it's a strength, so let's say I know I am assertive, and I'm high on
assertiveness that's great, he's going to be really assertive on these phone calls.
Oftentimes it can be than a crutch maybe become too assertive
at times become dominant.
And just because you have a strength also mean you have to be aware of
your strengths, because strengths overused can become weaknesses.
And just knowing these things, so you can also probe on some of these where
they're high scores, because you say, hey, really good, he's really assertive,
but you might want to say, hey, are there times maybe where he's overbearing?
Mike: Overly assertive.
How about that?
That's right.
Yeah.
That's just another really important reason to dissuade somebody from
looking at that number and making a decision, because it's not, it's
a description of an aspect of their personality, but it isn't necessarily
an indication of what you need to do.
It's an indication that you should drill down a little more closely to
be able to make that best decision.
Clinton: Yes, yeah.
And like I said, overall, if you are making a, a go, no go decision, let's
just say you have you know, a hundred applicants and you can't afford,
you just can't interview them all.
And the driving decision there on who you're going to interview, again,
focus on the overall recommendation score and not the subscores when
it comes to say, Hey, we can only interview 20 of these individuals.
Of these 100 let's focus on maybe then the overall recommendation
and you drill down on the subscales in some great interview Yeah,
Mike: that's a good point.
So if you think about let's say you had 100 you drilled down to 20.
You bring those 20 in that's a lot of people to interview.
And if you're only hiring for three or four positions, you've got to
make some really important decisions.
So you need additional information.
You know a lot of times in our case if you have a score, a data entry score,
or some sort of test map reading you know, it's 97 percent and somebody's 96.
There's not a big difference between somebody scoring 96
or 97 on a map reading test.
It's not enough to be able to make some sort of distinguishing decision between,
person one and person two, really, these are the kinds of things that you want
to use to determine of these 20, which are the people that we're going to move
forward to, and whatever our next part of our recruitment is, even if it's hiring.
Clinton: Yes, that's a great point, Mike.
Often tests are not, they're not precise enough to make or, to draw
conclusions from small point differences.
It's like that example, 96 versus 97 on that map.
Now, if someone's a 60 versus a 97, that's a large, far enough away where you can
say, hey, someone struggled a little more.
But when people 70 and this one's a 71.
It's basically, statistically, they're the same.
Mike: If you, if we want to sound really smart, we would say
performance differentiating, right?
Clinton: Yes.
Yes.
Mike: Very cool.
Jenny, anything that you, you can think of that you want to
talk about here in terms of,
Jenny: Just
a comment.
As I look at this report and I just imagine myself in the one who's
doing the hiring and needing to screen out and sit with candidates.
What a great tool this is.
What a time saver to be able to look at this report.
I have questions that I can ask.
And then the other thought I had was, I wonder if I might have a tendency
to try to weight the sub scores.
That one is more important than the other one.
And so I could really, we've already talked about this, but I could just
really see the tendency to, to think that one is more important than the other.
Clinton: Yeah.
Jenny: And
Clinton: people, you have to fight that tendency.
Mike: Yeah, and then go with the questions.
The questions are going to give you the meat.
That's really going to, that's really going to tell you what it is that
you, what you wanted to uncover as you go through that process.
Now, this test has been criterion validated.
And why don't you talk a little bit -when something is criteria and validated that a
lot of people call that the gold standard of validation, because it literally
predicts some sort of job performance that's correlated with the test scores.
But one of the things that's really advantageous about that is that the
Uniform Guidelines does allow this criterion validation of which shoot you
guys wrote a 280 page validation report.
So our clients are able to adopt that validation report as their own, which
will give them all of the defensibility aspects that come with that.
They're able to adopt that to their own environment through a very
simple process that actually takes only a couple of minutes for them
to go and transport that over.
Can you talk about that a little bit?
I think that's very useful to our audience.
Clinton: Yeah, sure.
So most, organizations, they may say that's great.
I would love to do this criterion validation study ourselves, but
we only have five dispatchers, or 10 dispatchers, and that's not
enough to really get the data.
Good news is we did the heavy lifting for you, like Michael says, by having
a multi jurisdictional criterion validation study where we had more
than 300 individuals participate, and we showed the relationship between
test scores and job performance.
All you have to do under the Uniform Guidelines on employee selection
procedures in section 7b There we have a simple process to say is your dispatching
job substantially similar in terms of kind of the major work behaviors to where
we did the original validation study.
And we have a document for you where we have listed out the work behaviors, and
you can see here, major work behaviors, and we have it both for the technical
skills component of the CritiCall test.
So like more of the map reading, audio comprehension, and also for the
personality component, and you need just, all you have to do is have your employees,
your subject matter experts, or your current dispatchers or supervisors of
the position, indicate how similar those are to, to the job at your organization.
And if there's substantial similarity, and this will auto calculate for you,
we'll say, yes, you have enough here to be able to transport the validity.
And so you don't have to do your own kind of criterion validation
study with hundreds of people that you don't even have.
And we've done that for you.
Mike: Now, and this right here is actually from CritiCall, the hard skills test.
So you can see these work behaviors are concrete work behaviors,
reading, comprehending, computer use, data entry, et cetera.
We also have a sheet for the personality test, which targets the work behaviors
that you and Jason targeted in our test development validation study.
So it would be softer skill work behaviors that pertain to the
personality testing aspects of that.
And it looks to me like basically you hit each organization's going to have
a series of subject matter experts, and they're just going to agree or disagree
that these major work behaviors are similar to those that are required in
their environment and from what I've seen in using this transportability
tool over many years with people, the universality of the dispatch environment
is such that everybody goes through this.
And yeah, of course, we have to be able to do this.
And of course, we have to do this.
It's a really high likelihood that the criterion validation that was established
is going to translate or transport over to any other given organization, even if
it's a very small rural organization, or if it's a very large urban organization.
Clinton: That is one of the great things of this job is that it is, like you said,
it is very specialized and that there's, it's very similar across organizations.
Sometimes when I've done some tests, let's say for example, for software
engineer, there, that can be, they can vary widely depending on organization
size or what exactly they're coding or what software they're working with.
And there's a lot of differences, but this dispatcher tends to be, like I said, tends
to be pretty similar, and so that's a good thing is that we usually find people are
saying, yeah, this is accurate and we do this and we can transport that validity.
Mike: Yeah.
Excellent.
And then we can't see all this spreadsheet because there's
just too much stuff on here.
But essentially what happens is you go through and you just, you get in
a room together with your subject matter experts and you say, okay,
number one, one through five, five, meaning it's exactly like my job.
And one, meaning it's not at all.
And everybody shouts off their ratings.
One person enters it into there.
And at the very end of the process, there's a box there.
The bottom that's red and it turns green.
If it's right there, the green signifies that the work behavior behaviors
are considered to be transportable.
Thanks for slicing and dicing this, Jenny.
I really appreciate that.
But it's as easy as that.
Number one, number two, number three, number four, number five.
And then when they get to the end, it turns green and
you can print a copy of that.
You can actually take our validation report that Clinton and Jason authored
and you can attach your transparency portability report, easy for me to say,
to that validation report, and you can then own that validation report for
this position for your organization.
And I would like to say that if someone was to call you up and do
this boots on the ground from end to end at their particular center.
If they were going to, if you were going to show up, you're going
to do a job analysis, you were going to do a test development.
You were going to do criterion validation.
If you're going to do all this stuff on site.
That would be a really expensive process, wouldn't it?
Yeah.
Clinton: Yeah.
So it's at least just a conservative estimate, at least $30, 000.
Mike: That's amazing.
And we'll do it for half that, right?
Clinton.
Clinton: Yeah.
Yeah.
So just half, yeah.
Mike: So that's quite an incredible savings.
And then what, talk a little bit about the defensibility aspect of this.
What we know, we didn't talk a lot about validity.
We can, we're going to actually cover validity on a future podcast.
We didn't really talk a lot about the development of the personality
test in episode one of this
series.
We talked a little bit more about it, if you're interested in it.
When you go through this validation process for this kind of test or any
other kind of test the way I describe it is that it's like a coin with two sides.
And on one side is the utility and the utility speaks to how well does
this selection instrument work to
bring in the people that are going to be successful on the job in some way or shape
or they're not going to steal from you or they're not going to quit immediately.
They're not going to be disruptive or whatever your criteria is.
Then on the other side of that same coin opposite the
utility side is defensibility.
And defensibility is speaking to what in a worst case scenario, if your
selection process gets questioned by some human being with an attorney or some
legal authority how well are you going to fare in that particular instance?
So can you talk a little bit about all the work that we've done and how that
benefits each organization in terms of the utility and defensibility aspects.
Clinton: Yeah and that's a great way of putting those two sides of the coin.
Mike, I always tell with clients, whenever we work with them on validating tests
or implementing hiring tests is that the goals I always say are one A and one B is
making sure you're hiring the best people for the job and then making sure you're
doing so in a legally defensible way.
And that's what you want to do is you want to have both of those.
You should be able to have some legal protection.
So if someone does challenge your hiring process or says, Hey, you didn't
hire me because I am a white male.
Or you didn't hire me because I don't know, some group I'm too old.
You didn't hire, we want to be able to defend it and say, no, we
didn't hire you because, you didn't have the right technical skills.
You couldn't do these things.
You weren't, we showed through our validation efforts we know
that lower scores on this test are less likely to succeed.
And so you were screened out because of that and not because of your age
and not because of some other issue.
And that's what we provide.
And that's what they're the uniform.
We didn't.
I haven't gone into this too much, but the Uniform Guidelines on Employee
Selection Procedures provide a playbook.
And when it comes to personality tests and criterion validation, Section 15B
of the Uniform Guidelines on Employee Selection Procedures provides like a list.
A list of just of things that you need to cover when doing a validation study or
that a test creator vendor should cover if they're developing a test Yeah in
that report and the report that we have we actually at the very beginning of the
report Have a list of all the requirements of Section 15B of the Uniform Guidelines,
and we indicate which page or pages of the report cover those sections of Uniform
Guidelines to show that we're addressing all of the requirements that we're
doing our homework and you're covered.
And so if someone does challenge the process, this is a nice report that then
can be provided to lawyers and people who are going to get in those rooms and
fight those battles to say, Hey, we've done our homework here and we're making
decisions based on good information and not because we just don't like people
from a certain protected group status.
Mike: On the prior episode where we talked generically about personality
testing, I used an example from this 9 1 1 police, fire, EMS space where,
you know, being at the national conferences and having an opportunity
to talk to center directors there's this universal understanding among them.
If they've been around long enough, they can sit down with a human being, for 5
or 10 minutes and talk to them and they can determine whether or not they're
going to be successful on the job.
And I never, I never argue with these folks because I know that they're
right about that, that they have that uncanny ability to be able to sort
out who's going to be successful and who isn't going to be successful.
We would never recommend that they use that as a strategy because
it's fraught with pitfalls, right?
The first one is that you don't have a paper trail that shows why you
selected one person and you didn't select another person and that in
itself can be quite problematic.
And so really when we set out to create CritiCall, what we did without even
really knowing is that we created it to be proxy for that center director who's
able to spend time with an applicant and in a similar way, know whether or
not they're going to be successful on the job, but to do so in a way where
it's completely documented step by step.
Here's the reasons why they should move forward.
Here's the reasons why they shouldn't move forward.
But furthermore to that, what we find, and I would say particularly in this
social media environment or day and age, is that it's so important for
organizations who are hiring to be good citizens, good stewards in their realm.
And so what this critiCall test and even your personality test that's in
the CritiCall test does and does so well is it affords the applicant an
opportunity to find out not only find out for the organization they're applying
for, but find out for themselves.
Am I suited for this job and to do so in a way that is really satisfactory.
In fact, what we see with our testing is that if you're like me and you
can't multitask, you hate the test so much that you pretty much get up
and walk away, don't even finish it.
And what you do is you walk away saying, Oh my gosh, I dodged a
bullet because I would hate that job.
I'm so glad I didn't, get that job so I could work Christmas and
weekends and graveyards and all that stuff that I really also don't like.
So these kinds of well constructed tests really serve both of those masters in
such a way that it leaves the organization feeling very good about the process that
they have in place, as well, it leaves the applicant feeling whether or not they
got hired that they had a fair shake and even maybe more than some other jobs.
If you're not suited for this environment, you really have no disagreement with that.
So do you have any thoughts about that?
Does that resonate with you at all?
Clinton: Yeah, it does.
And I think there's a, I always, a couple of thoughts.
One is I think it's important, like you said, candidates will often self
select out when you have good selection process that are face valid, that
kind of look and feel like the job.
And like you said it's not only our organization, should they be figuring out
who's a good fit for the organization, you as a job applicant should be trying
to figure out, is this a good fit for me?
And that test, it also helps to allow for that.
My, my other thought was, it was like you talked about is we're
providing what I call a standardized, a nice standardized process
for organizations to make hiring decisions.
And while it's not often sexy, people aren't often saying, Ooh, look at
that standardization you have in your process and, really admiring it.
That standardization is actually what helps and saves a lot of organizations
and actually what can build in some of that predictive power, because oftentimes
one of the problems, like you said, Mike, with those individuals who've been in the
job for years and say they can recognize who's going to be the best performer,
is sometimes problems occur when there's a lack of standardization.
If they didn't use tools like this, they all of a sudden they're free
flow and they go in one direction with one applicant, they don't go in that
direction with another job applicant, that lack of standardization can then
where problems can be introduced into the process and a candidate may complain
and say, I didn't get a chance to show what I could do because they asked all
this other candidate these questions that they didn't even ask me, okay.
We're having a standardized process.
We're giving all the candidates a fair playing ground to show their skills
and their abilities and they're fit for this job in a standardized process.
There's a lot of value in that standardization.
Mike: Yeah.
And from the utility aspect as well, that's the defensibility
part, but the standardization gives you more utility in your selection
process, which is equally important.
The other side of that same coin that we talked about earlier.
Excellent.
That's great.
Do you Jenny, did you have anything else that you wanted to talk about before we
end?
Jenny: No, actually.
Yeah, actually, sense we're winding down.
This has been a, an amazing discussion, and I just want to just
say that as somebody who's taken the personality test and the CritiCall
test, it's just not my space to work.
And I can tell you, I have self selected and I'm so grateful for those
who can do such a high stress job.
We definitely need you if you're watching this, we need you and thank
you for the work that you're doing.
So Clinton, as we wrap up, why don't you just briefly tell a couple of
sentences about ioPredict, who you are and how they can reach you.
Clinton: Yeah.
ioPredictict, we do test validation.
So we can, we do test development from scratch, like custom test creation.
And we work actually with Biddle and Mike, and we worked on this CritiCall
test, personality test for dispatchers.
And we partner on other projects as well.
ioPredict.com is our website.
And on there, you can reach out to us through there.
But we also work with organizations when sometimes they've adopted a test
from another vendor where it's created, but they say, Hey, we don't know.
Did they really put this together?
And they reach out to us to do validation studies or to be that other source
to, to review and to implement tests that maybe other people have created.
So we do that as well.
Jenny: Thank you.
And we'll have all of your contact info in the show notes for this episode.
And if you haven't watched our first episode podcast episode with Clinton,
I just want to encourage you to go back and check that out as well as we
talk about personality testing more in general, CritiCall personality, not
CritiCall personality testing specific.
That was a mouthful.
Hey, thank you so much Clinton for just not only for joining us
for this episode, but also you joining us for a previous one.
And thank you for our listeners.
Thank you for our viewers.
Clinton: Thanks
Jenny: for having me.
Thanks for having me.
Yeah, this has been fantastic.
Thank you, Mike.
Mike: Thank you.
Thank you very much, Jenny.
Great job.
And thank you, Clinton, for joining us.
ioPredict is a really important partner of ours.
They, Clinton and Jason, who are the principal consultants there,
worked for us for many years.
They started up their own firm and we have, continued to work
together very closely all along.
And we actually really think of them as a branch of our company.
And now that doesn't mean we'll be paying your taxes for you, but we
certainly think of you in that regard.
And we hold you guys in high esteem.
So it's really great to have you here hanging out with us.
Clinton: Thank you.
Yeah, we feel the same way.
Mike: Thank you.
Jenny: Wonderful.
Thank you everybody.
Mike: Have a great day.
Voiceover: Thanks for tuning in to Testing, Testing 1-2-3 brought to you by
TestGenius and Biddle Consulting Group.
Visit our website at testgenius.com for more information.