Voiceover: Welcome to Testing, Testing 1-2-3 a podcast
brought to you by TestGenius.
Jenny Arnez: Welcome everybody.
Welcome to Testing, Testing 1-2-3.
It's a podcast produced by TestGenius and Biddle Consulting Group.
And I'm Jenny Arnez.
I'm your host and I'm joined today with Mike Callen.
He's our co-host, but also the president of TestGenius.
Mike, you want to say hi?
Mike Callen: Good morning.
Jenny Arnez: Good morning.
Yes.
Or good afternoon.
Good evening.
Depending on when you're depending on wherever
Mike Callen: you are on this round globe.
Jenny Arnez: That's right.
And we're sure glad you joined us today.
So for today's episode, we're going to do things a little bit differently.
We don't have an additional guest joining us today.
What we're going to do is look at a specific question that
a client recently asked Mike.
It is all about some best practices that they could incorporate
in their hiring practices.
And it led to an interesting conversation about, validity and adverse impact.
And so today what I'm going to do is present Mike with a scenario very similar
to what our client offered to him earlier.
And just have a conversation around that.
Does that sound good, Mike?
Mike Callen: Yeah, absolutely.
Jenny Arnez: Alright, so let me present the scenario.
Let's just say Acme County has reached out, and they want to hire, or they
are hiring for Dispatcher I positions.
Now here's some background.
Each department in the county determines their own requirements
for the Dispatcher I position.
All departments are using a pre employment skills test.
And after a phone interview, selected candidates take the pre employment test.
It's only the highest scoring candidates that are selected
for an in person interview.
Acme County has reached out to us and they're concerned that their hiring
practices, including the pre employment skills tests, are resulting in adverse
impact towards protected groups.
Mike, how would you help a client like this?
Mike Callen: Yeah, there's a lot to unpack there.
I, if maybe if we step down through, Each of those bars on this
image right here, we can talk a little about that about each one.
So you have a county that has different departments, all of whom have a job title
called Dispatcher I and then they're determining their own requirements
for these, each of these positions.
Presumably they're slightly different from location to location and so one
of the things that you always want to be concerned with if you're an employer
is the optics in terms of how do things look and feel to the applicant.
And in the case of a county or a private organization, often the applicants
are constituents or customers.
And you want to be a good citizen in your environment and you want to be
doing things that feel good and fair and righteous in terms of their eyes so that
you don't, sully your own reputation.
And if the Dispatcher I position is different in each of those
organizations, it might be better to title them slightly differently.
For instance, it could be Dispatcher One Fire, Dispatcher One Police,
Dispatcher One EMS, or however they're dividing things up.
And that way you get a little bit of disparity from one title to the next.
And I think the reason that I mentioned that is that you have job applicants
who are going to talk to one another.
And if you apply for a position over here and you're tested or you're minimally
tested and you apply for another one over here and you're maximally
tested, they really wear you out.
That's going to feel weird if they're all titled the same thing.
The same job title within different departments could vary and it
could be justifiable to have different recruitment processes.
It probably is justifiable to have different recruitment
processes for each position.
But again, in terms of optics, it might be something to differentiate
between the Dispatcher I position at each of those locations.
So that's definitely one thing to look out for.
Another thing to look out for,
and this comes up very often in these kinds of conversations,
is within the recruitment.
Let's say your recruitment process is different in each of those three or four
departments but your recruitment process within that one particular recruitment at
one department, it's got to be the same.
You can't do different things for different applicants
within the same recruitment.
So you want to be really consistent in that regard.
Jenny Arnez: so this might be a bit of a tangent, but you mentioned
about departments having perhaps changing the job title where
they don't all say Dispatcher I.
I'm wondering how job analysis might relate to that we've had previous
clients or previous guests who've talked about job analysis, just wondering if
you might be able to touch on that.
Just briefly.
Yeah.
Mike Callen: Absolutely.
So a selection plan, which is any part of the recruitment process from beginning
to what should be the job posting all the way through onboarding somebody
should be based upon a job analysis.
This is best HR practices and a job analysis is a really deep dive into,
what the what are the tasks and duties that are performed on the job?
What would be the minimum qualifications?
What would be the desirable qualifications?
What would be the work schedule, the work environment, what personal characteristics
are required in order to perform the job.
So it's really
all of the technical aspects that have to do with any particular position
would be fleshed out in a job analysis and job analyses are really time
consuming, challenging tasks to perform.
And while everybody's supposed to have current job analyses, they very
often don't, and it becomes one of those eating the elephant situations.
How do I do, job analyses when I have 500 different positions within my county or
my organization or company or whatever?
And so the answer for that is, you basically just do it one bite at a time.
It's best in my recommendation, it's best to start with the positions that
are most often recruited for, or the positions that are most often hired
and then reallocated within the organization.
So you have these entry level positions that people come into and then rather
than posting a job externally, they posted internally and then they move
people around within their organization.
So those would be the ones to start with.
And then you keep going, doing the next most high volume positions and until
you get all the way through the process.
And then it's painting the Golden Gate Bridge.
Once you get done, they're really old in three years or when the
position fundamentally changes.
And so you really need to go back and do them again.
But you can use the existing job analyses for the redo, which can
make the process a lot easier.
So job analysis is super important in this particular practice and some
things that come from the job analysis are the job description and the job
description would be a formal paragraph or several paragraphs that talk about
what this person does in this position.
And then from the job description, would be distilled, typically a job posting,
which would be the text that's put into an ad or a posting online that talks
about the job title that's gonna be opening up, some basic requirements,
minimal requirements, some desirable qualifications, and then that little
summary of the job description.
So those, they're a very important part of the selection process.
Jenny Arnez: Okay great.
And actually, I think what I'll do is in our show notes,
I'll post a link to IO predict.
They're one of our partners who actually does job analysis.
And Dr.
Clinton Kelly has been on our show before and talked about that.
Because I think definitely that would be very helpful.
Definitely relates to this scenario here.
As we consider this process that Acme County has for hiring Dispatcher I,
where are the tests in this process?
What are the tests?
Mike Callen: Yeah, that's, it's a really important point
that you're making here, Jenny.
When we're approached by clients or prospects and they talk about, testing
and validation and adverse impact, the focus is always on the literal test.
And the fact of the matter is that there's many other steps of the recruitment.
There are also tests.
I was just talking to Dan Biddle, our CEO, a few minutes ago, and he was
telling me a story about he did some consulting for the Department of Justice.
And he was at a conference where there were about 500 HR practitioners and
the head of the DOJ asked all of the attendees to raise their hands if they
agreed that an interview was a test.
And it was half or less that raised their hand.
And the fact of the matter is an interview is a test.
There are a variety of things called PPTs.
Those are practices, procedures, and tests that occur throughout the
recruitment process in which decision making factors are considered in
this employment selection process.
An interview absolutely positively is one.
People sit down, they talk to another human being or a team of people and
they're rated on their responses and some people are moved forward and
some people aren't moved forward.
And that is part of the selection process and therefore an interview is a test.
But there's other things that are PPTs that also act like tests.
For instance, we talked earlier when you brought up the job analysis, one of the
derivatives of the job analysis is the job posting and the job posting talks about,
hey, here's what you perform on the job.
And if the job posting and its minimum requirements are not valid,
that is, they're not matched exactly to that particular position, then
anyone who's reading the job posting may actually deselect themselves
from the process based upon the information that they've been provided.
And if the information that's provided is incorrect, then you
could have just created adverse impact in the very first step of your
recruitment process by not making an accurate job posting and especially
accurate minimum qualifications.
So that's a test too.
And every little intermediate step along the way that either allows
certain people to continue or
doesn't allow people to continue or rank orders people or weights their scores.
Any of those aspects of the recruitment process that make those kinds of
decisions are also tests and should be scrutinized in the same way that
people and we scrutinize our own tests.
Jenny Arnez: Okay.
So if I'm understanding you correctly, what you're saying is like in our scenario
here, Acme County, they're concerned about
there being adverse impact and certainly they need to look at making sure that
they validated the actual pre employment test, but they actually need to look at
their whole process from job description, job posting, interviewing process,
that phone interview that happens that determines who moves on to the next step
Mike Callen: background checks.
Yeah, that's absolutely correct what you're saying there.
And to that end, people will ask us for, data in terms of pass rates for, certain
protected groups, EEO protected groups.
And we don't have that data because we're not, getting that
EEO data from the applicants.
The employers who are going through the recruitment, they're
getting that information.
They're collecting that information up front and they have a piece of software.
Typically known as an applicant tracking system- an ATS.
And the ATS will actually do analysis at each of these steps to let them
know where they may potentially have problems so they can dig into it and
they can see whether or not there's some adjustments that need to be made.
Now every time
there's adverse impact against a protected group, it doesn't necessarily mean that
you have to do something different.
There are instances where adverse impact occurs and that adverse impact
is simply a part of the process, the part of living in a real world.
And we can talk a little bit later about that as well.
But you always want to stop and you want to look and you want to ask ," Okay.
Is there adverse impact?
Why is that adverse impact occurring?
What could we do differently to avoid that adverse impact?"
Those kinds of things are questions that you ask.
And let me throw out an example because it's a big thing to wrap your head around.
There's a lot of things we're talking about right here.
But the classic example is in a job that requires physical ability.
Let's say, for instance, there's a person who has to load concrete bags onto a
pallet so that it can get shrink wrapped and loaded into a truck and shipped away.
So there's a conveyor belt that brings out these,
these bags.
And so what you might do in terms of a physical ability test to hire
somebody into that position is you might have a conveyor belt.
You might have this person grabbing the bags.
You tell them how you want them stacked.
And you see how many bags they can stack in 5 minutes.
Or can they stack the number of bags that come out in a 5 minute period at
the same rate that would come out later
on the conveyor belt on the job itself.
Now, let's say, for instance, that this particular test is using 60 pound bags.
And so HR calls over to the plant and says, "Hey, we're getting ready
to do this physical ability test.
I want to make sure that.
We're still using 60 pound bags in the plant."
And the supervisor says, "Oh no, I'm so glad you called.
Remember when the risk analysis people said we're going to have
far fewer incidents of injury if we use 40 pound bags instead.
So we switched over to 40 pound bags.
It's we just dodged a bullet because we would have been testing at a level that is
much higher than is required on the job."
And that
a physical ability test is going to almost always result
in adverse impact against women.
Generally speaking, men are stronger than women, and so there will be
a disparity in between the passing rates for men and the passing
rates for women for a physical job, especially one that's really demanding.
And so, there's baked in adverse impact that comes with that kind of position,
but by measuring using the wrong size bag of concrete, you're actually
creating additional adverse impact that otherwise would not have occurred.
And that is an instance that is impossible to defend.
You cannot defend that.
You can defend the adverse impact that occurs between natural differences in
physical strength between men and women, but you cannot defend against the adverse
impact that occurs because you ask them to do more than the job requires.
Jenny Arnez: Okay.
Yeah, that makes sense.
So what you're saying then is that there are times, provided that your
test is valid, that what you require, what your test is requiring of the
candidate is actually a reflection of what they'll be required to do on the job.
When that is in sync, there are times where an adverse impact will occur.
Mike Callen: Yeah, absolutely.
There are going to be times in which it occurs.
It doesn't mean that you don't look for it.
You still look for it because you want to evaluate the process.
If you can measure the same thing in a different way, and it
doesn't have adverse impact, then it's a better way to measure it.
You go that direction.
In our tests, we use something called work sample tests.
So we're actually like the concrete bags.
We're actually simulating the job.
It's very hard to find a testing process that's more valid or
equally valid to actually simulating the job than simulating the job.
And as an employer, if I'm hiring people to load pallets, to shrink
wrap that material and get it onto a truck because I'm selling concrete, I
have the ability to make sure that the pallet loader can load pallets, right?
That's just a fundamental thing.
So whether or not there's adverse impact- it isn't important.
What matters is that the selection process, the test in this case,
is it valid and consistent with business necessity?
If it is, then you are justified in having some adverse impact
that enters into the process.
Jenny Arnez: So you said business necessity.
That sounds like that might be an actual term.
Is there a set of key laws or guidelines that govern pre employment testing?
Mike Callen: And guidelines is the key word.
There's a huge document which can be found at uniformguidelines.com
which is the Uniform Guidelines on Employee Selection Procedures.
And it's a document that was put together, I believe, in the 70's.
It's been revised a couple times and now has a whole host of questions and
answers that comes with it as well.
But it was put together by a really large committee
of practitioners who operate in this space, and it covers things like
job analysis, job descriptions, test validation, also sorts of different
kinds, and it really sets forth the practices and principles that are the
best practices for doing these things.
Now, I don't know that it was ever codified into law.
But the courts have relied on it again and again.
And the DOJ, Department of Justice, the OFCCP, the EEOC, all these different
governing bodies, including other state bodies, have relied upon it as the
standards for doing things correctly.
And so it comes up very often in legal trials and adjudications like
by labor boards and those kinds of things that comes up very often.
So it's a very important document.
Another thing that's really important is the ADA, the
Americans with Disabilities Act.
And we won't really get into that here, because it gets into
a whole other area of essential functions and those kinds of things.
But, yeah, in terms of, dealing with this issue, the Uniform Guidelines is really
probably the main body of literature by which everything should be measured
in terms of doing things correctly.
And that's what we do as well.
In fact, in our software and the TestGenius software, CritiCall and OPAC,
we use a validation wizard, which is based upon section 14c and
15c of the Uniform Guidelines.
Section 14c is the section that deals with content validation.
It's the steps that you need to go through in order to properly content
validate a test, and that's how you determine whether or not it's valid and
consistent with the business necessity.
As well,
15c is the reporting section for the same.
So it talks about what are the things that need to be outputted in a report
that will properly document that this validation has taken place.
And so in our validation wizard, we're running people through a process.
In this case, we would run subject matter experts,
who are performing well on the job for a year or longer or supervisors
of those people are trainers of those people who are abundantly familiar
with the position, we're going to run them through each of the tests that
employer would purport to measure.
And then, at the end of each test will be a survey that will ask them
questions about the job relatedness of that test and that the very
end of each test that would say,
"Jenny Arnez, your score on this test was 96 but given your position and your
knowledge of this job what, in your opinion, would be a minimum qualifying
score for somebody who's entering into this position the first time?"
So you would say, "Hey, I've been doing this for 20 years, and I'm an uber expert.
So 96, it's really good for me but because I directly just felt the difficulty
of that test and I'm knowledgeable about how well skilled enabled someone
should be coming into this position, I would say that if somebody scored
83 or better on this test, then I would consider them to be eligible
to move on from the applicant to the candidate role."
And so the wizard will take in all of the subject matter experts
opinions, including yours and the suggestions for the cutoff scores.
And it will create a validation report that says of this test
battery, let's say it had six tests in there, for this test battery
you should only use these five tests because one of them was determined
to be not valid for this position.
Maybe it was more difficult than the position like the 60 pound bag of concrete
instead of the 40 pound bag of concrete.
And so for that reason, it gets thrown out.
As well, here's four different defensible job related cutoff
scores that you could use.
Your average of your suggested cutoff scores, and then that minus one, two,
and three standard error of measurement, which is a statistical step downward
that one might use to create different significant layers below the high cutoff.
If you don't want to use, let's say 86 was the first one, minus
one SEM might bring you down to 81.
So let's say we didn't have enough people to interview.
We wanna step down to the next appropriate layer.
We might use the cutoff minus one SEM, which would be 81, and that gives us five
people to interview or maybe minus, minus two SEM and it brings this down to 76.
And that might be 10 people to move on to the next step, that, that sort of thing.
Jenny Arnez: Okay, so what you're describing is the validation process,
and in particular, the validation process if someone were to use
TestGenius as their pre employment test.
Mike Callen: Yeah, or if they did it all on their own.
You can do this process manually, and this process is outlined
in the Uniform Guidelines.
It's just that what we've done is we've said, "We value this process so
much that we want to make it as easy as possible for you, our clients.
And so we're creating a wizard that literally just walks them through and
that formats the report and then creates the report and sends it off to people.
Jenny Arnez: Okay.
So let's real briefly, just a high level
outline of the validation processes using the test selection wizard, right?
Mike Callen: we didn't talk about that.
But yeah, you want to start off with the test selection wizard.
Let's show the content validation slide because that's helpful in understanding
the test selection wizard process.
That's okay.
Content validity is like building a bridge across the river.
On one side, you have the test.
So go ahead and go to the next slide
and then on the other side, go ahead and do the next one.
You have the job and it's represented by these duties that
are performed and the bridge is supported in the middle by this peer.
And that is the KSAs.
KSA stands for knowledge, skills and abilities.
So if you look at this process the test requires certain
knowledge, skills and abilities.
The job duties require certain knowledge, skills and abilities.
And so what we're doing is we're connecting the test on one side of
the river to the job on the other side of the river, and the linkage is
the knowledge, skills and abilities.
If we look at the test selection wizard, I don't think we have an image
for it, but basically what we're doing on the test selection wizard is we're
listing duties that might be performed on the job, and so a person is going
to go through, and they're going to check off duties that are performed by
the position that's being recruited.
And as they go through, and they check off those duties that have
linkages to KSA's- knowledge, skills and abilities- which are linked to
specific tests within our system.
And so, you check off 10 duties, and that might create a list of eight
or nine tests that we may suggest that they use for selection purposes
for this particular position.
And so that's the test selection wizard.
The next step that one would go to in order to determine whether or
not the test battery in this case, a battery of individual tests,
is valid and consistent with business necessity is to go through the next
part, which is the validation wizard.
And in that case,
we recommend 7 to 10 subject matter experts, and again, those will be the
incumbents that I described earlier.
People who have been successfully performing on the job one year
or longer or supervisors at that position or trainers for that
position, but people that are abundantly familiar with that job.
And you want to go through and you want to cherry pick really high performers,
but you also want to get a demographic mix that matches that
of your existing workforce and particularly of your recruitment area.
But the idea here is that these people are going to go through those tests.
They're going to take those tests with that wizard turned on, and the wizard
is going to ask them survey questions, cut off score suggestions, and it's
going to result in a validation report
that might be 30 or more pages long that's going to have a table at the end.
It says, here's the test you validated and each of these marked yes, are
ones that you should use on your particular recruitment or could use
in your recruitment as well here's the defensible job related cutoff scores.
So that's those two wizards, how they work, step one, step two.
And then step three is your recruitment itself.
So you take the validated test and you actually employ it for use for
selection with your job applicants.
Jenny Arnez: Okay.
So let's say you have your valid test.
You have candidates taking it.
So do you just assume that the person who scores the highest on the test,
that's your number one applicant?
What's your recommendation there on how to use the scores?
Mike Callen: That's a really great question.
We didn't talk about this, but one of the one of the survey questions that's
asked at the end of each test when you're running your subject matter
experts or your incumbent employees through there is whether or not the
test is performance differentiating.
I'm going to give you an example.
Let's say that you're hiring somebody who has to type.
Dispatcher is a good one.
They're doing short bursts of texts, but they need to be able to type
relatively quickly and they need to be able to type accurately.
So typing with speed and accuracy is an important attribute for that job.
The fastest speed that they may need to type might be 35 words a minute, right?
So the question becomes, does that mean that somebody who types 70
words per minute is twice as good as somebody who types 35 words per minute?
And in this case, the answer is no.
You really only need to do what you need to do and going faster than you
need to go isn't necessarily a benefit.
And so that would be a good example of a test, the results of which are
not performance differentiating.
And so what we would do is we would look at everybody from 35
words per minute and above as being equally suitable for the job.
So there wouldn't be any rank ordering at all.
And that is really
just one of the main reasons why setting the cutoffs is so important.
If you have a defensible job related cutoff score then what it
does is, let's say you just chose the person who could type 70 words
per minute and you hired them.
You've ignored all these other attributes of a human being that may
or may not make them a good employee.
Sure, they can type, but that is not the entirety of any job.
And so by choosing too high, you may have ignored applicants who were equally
qualified in terms of keyboarding, but had so many other things that
they could apply to this work area.
And so that's a really important aspect to keep in mind.
Now there are some that people will go through and say, "Yes, this is
performance differentiating because, people who do scope score higher on
whatever knowledge, skill or ability it is, they may be the best and everybody
who's less than them is less good."
But again, that's just one of the many
reasons why you make a selection decision.
And so again, cutoff scores are really important.
Jenny Arnez: And the validation process helps you to set those cutoff scores.
Mike Callen: It helps you to set those cutoff scores.
And it also will report back to you whether or not something
is performance differentiating.
Which lets you know that once you get to that cutoff score, everybody from
there and above should be considered equally skilled or able for that
particular performance dimension.
And we'll be right back after a word from our sponsor.
Ready to revolutionize your HR strategies?
Head over to TestGenius.
com to discover our latest tools and solutions designed to
streamline your hiring processes.
So I'm going to flip back real quickly add our scenario back on here.
Just so we can summarize where we've what we've touched on Acme County.
They're concerned about adverse impact.
So the first step really is they need to remember that the testing
process or testing happens throughout the whole selection process,
throughout their whole selection plan.
And so they have to pull back the lens, not just focus on the pre
employment skills test, but also to look at their job posting, their
interview practices, and so on.
All the other tests as well.
Jenny Arnez: All the other tests.
And then focusing in on what our specialty is here at test, with TestGenius
is the pre employment skills test.
And then, so then they need to validate that using their subject matter experts,
using the tools that, that they have within TestGenius, assuming that they're
a TestGenius client, using the test selection wizard, the validation wizard,
which then helps them set accurate
- is that the term, accurate cutoff scores?
Mike Callen: Defensible and job related is the important aspect of it
and let me in interrupt one second.
This is our recommended best practice.
Not every client goes through the validation process for every
administration of TestGenius.
We know that.
It's our strong recommendation that they do it because of two things.
We think about the recruitment process.
Think of about it as a coin with two sides.
Okay, on one side, you have defensibility.
So in the instance where somebody questions the hiring practice, is
that hiring practice defensible?
Or if you go to your applicant tracking system and you find that your physical
ability test had adverse impact.
Is that defensible in that particular case?
So being able to answer some sort of challenge to the recruitment
process is really important.
But when you flip that coin over and you look at the other side
from the utility side, or sorry, the defensibility side, utility.
And utility talks about how does this test select people who are going to go on
to be really good employees on the job?
And the beautiful thing about this, and this is the beauty that comes
with the brilliance of the Uniform Guidelines, is because people who were
really experienced in the practices that can both have and result in a
safe recruitment and hire people who do really well on the job are contributing
to this framework that we follow here.
And so that's why it's really important to go through and do the required work
or to do the recommended work in order to be able to end up with a testing
process that's valid and consistent with business necessity and cut off scores
that are defensible and job related.
So that's as far as we can say about that.
If somebody decides that they're not going to go through the validation, that's
their choice, but they've taken on that burden themselves in case the hiring
practice is challenged, for instance.
If it's challenged, then you know they don't have a leg to stand on in
terms of the defensibility, if they go retroactively and do the validation
process and the validation report shows that the test isn't valid and consistent
with business necessity or that they use cut off scores that were too high.
So people are taking that risk upon themselves by skipping
that part of the process.
Jenny Arnez: And how often does the validation process need to happen?
Do we do this every year?
What is that?
How do you determine that?
Mike Callen: So the validation process is good to do, generally
speaking, at about a three year clip.
And that would be the same pace that you would want to try
to review your job analysis.
Now, the difference, or sorry, the the caveat for both of those scenarios would
be unless the job fundamentally changes.
There's been a just a ton of technology that's come into the
dispatch world, for instance, which is to follow our scenario here.
People can text videos and images.
Let's say somebody gives you a license plate number and you type that into
the computer aided dispatch system and they also send you a photograph
of a scene and the photograph has the license plate on there and you can
clearly see that the person reported the wrong license plate number.
So you have to be able to discern the difference between the two.
You have to be able to decide which of those has the right data and then you have
to be able to update it into the system.
Because if you're talking about putting out a BOLO looking for somebody
with this license plate and you use the wrong license plate number,
you know that's really CritiCall.
So that would be a good example of a fundamental change that happened at a
dispatch center that might require you to go back and look at the versions
of these tests that you're using.
Maybe before you were using text only versions, now you want to use versions
that have photographs, images and questions about those images in there.
Jenny Arnez: I think we've done a good job or you've done a great job
on helping Acme County here take a look at their hiring practices and
some action steps that they can take.
And anything else that you want to add to today's conversation?
Mike Callen: We talked about content validity and content validity is
when you're building that bridge from the test to the KSAs, knowledge,
skills, and abilities over to
the job on the other side, and there's another kind of validation, which we
don't do with our wizard, but can be done.
And it's very much valuable.
And that's called criterion validity.
And criterion validity is when you have some sort of aspect of job performance
on the axis over on the left hand side, and then you plot that against test
scores along the bottom access there.
So each of these data points would be some sort of measured
metric of job performance.
Maybe it's supervisor ratings or could be feedback from surveys or, like
tardiness or, all sorts of different things, but each of those data points
would be the intersection between how that particular aspect of job performance
was and what the test score was.
So down in the bottom left, you can see there's two points down there.
So that's people who had very rare, relatively low job performance, and they
also scored relatively low on the test.
And then over at the top you have one blue dot.
That's the hot, the highest scorer who also does very well on the job.
And that black line is a trend line that shows, how did those work together.
So in this case, it's what you would hope for from a test.
The lower scorers have lower job performance.
The higher scorers have higher job performance.
So this kind of study would be taken with existing incumbent employees.
And that would be one way to do it.
Or you could test it against job applicants and then wait A
year and measure how well they were doing on the job and then
go back and correlate it later.
That's a hard thing to do because people don't like to do all that
work and wait a year for the results.
So it's usually done called a concurrent study that's done with incumbent
employees, in which case, you can get the results at that particular point in time.
But what you end up knowing is that people who score higher on the job are going to
have a greater degree of job performance.
And so you can be very confident finding what your cutoff score is
and then interviewing those that are north of the cutoff score.
And you can feel very confident that those people are going to be good performers
on the job, at least in terms of this particular aspect of job performance.
Now with the content validated skill and ability testing, you
don't necessarily find that.
Again, to go back to the person who types 35 words a minute versus the
person who types 70 words a minute, that 70 scorer isn't not going to
be twice the performer that the 35 per minute scorer is going to be.
In many cases, what we're looking for is, do you reach or exceed a threshold?
And that's really all that we want to know.
And that's where you're, content validation approach comes into play.
But in many instances, you can go on and do a criterion validation study
that can be very helpful as well.
Personality tests are always criterion validated.
So basically you're looking at what are the personal aspects of a human
being that would contribute to high performers on the job and what are the
personal aspects of, human beings that contribute to lower performers on the job.
And so when you're testing for personality, you're going to look
for those aspects of personality that match with the higher performers.
And so criterion validation is the only way to go.
Jenny Arnez: So our tests in TestGenius for HR and then also for
our dispatcher test, CritiCall we have the content validation tools but
they are also criterion validated.
Yes.
Yeah.
Can you speak to that a little bit?
Explain.
Mike Callen: Yeah it's always nice to be able to do a criterion validation study.
And one of the beautiful things on the dispatch side is that dispatchers,
call takers, telecommunicators, their jobs are very similar.
There's a universality that exists between little tiny communities
and large federal organizations.
We we re able to do a criterion validation study with several different agencies
who are contributing data to it and then come up with a chart like this one right
here for a variety of different tests.
And then back to the Uniform Guidelines
once again, the Uniform Guidelines in Section 7b talks about how you
can transport criterion validity.
And so what that means is that for criterion validation, they will allow
you to transport a validation study from one organization to your organization
as long as you can establish that the the job performance dimensions
or domains that were measured in that original study are very similar to those
that exist within your organization.
And so it's a very simple survey.
You get in the room with 12 people, you go through those job
performance areas, and then they rate it on a scale of one to five.
One means it doesn't match my environment.
Five means it's exactly like my environment, and then
all the numbers in between.
And so if you get
12 people to agree that, it's 52 and a half or better, roughly three, a rating
of three or better than you're going to be able to transport that validation
study over to your own environment.
And then you can just attach your transport ability sheets to that
validation study, and you can literally claim it as your own.
It's a golden ticket kind of thing.
And I'm really glad that you asked about this because this hits on a
point that we didn't talk about it all.
One of the main reasons that why we have a validation wizard for content validity
is that you cannot transport content validity from one organization to another.
You can't do it.
Employers are responsible for establishing the validity of specific tests within
specific job titles at their organization.
And that's the difference.
That's why we can't do a universal criterion validation study, because
even if you go with administrative assistant, what does that mean?
If you go to a hundred different organizations likely going
to be 100 different jobs with 100 different requirements.
And so you can't really do that for that particular role.
But that's okay.
Because what we've done is we've given you the tools that allow you to
do it in a matter of a couple hours.
And now you have a validation report that says, for Acme County for the position of
Dispatcher I, we've given you to here's our validation report, and it goes page
after page, and it gives them, really a blessing to go through and use that test.
You go to another county, and they want to go ahead and start testing,
they might ask, hey, ACME, send me over the list of tests you're using.
ACME might share those tests with them, but it doesn't mean that they're
valid and consistent with business necessity, nor that the cutoff scores
are job related and defensible.
So that's why you go through the process because, go back to the coin, it's
defensible on one side and has a high degree of utility on the other side.
Jenny Arnez: Excellent.
That was very helpful.
Actually I have the transportability sheet.
Do we want to show that?
Sure.
Mike Callen: Absolutely.
It's a pretty Excel spreadsheet.
It's always fun to show an Excel file with colors in it.
So yeah, basically, we have this for the CritiCall test.
As well we have it for the CritiCall personality test.
And so on the left, you have the work behaviors and orders are too
small for me to be able to see.
But basically, it's going through the different dimensions.
That were measured in the original validation study, and at the top, you
have SME1, SME2, SME as subject matter expert through SME12, and then on the
far right is the rating scale, where 1 is that it doesn't match, and 5 is
that it does, and so basically, with the help of a leader, you go through
and you just number the people and you say, number one, five, number two, five,
number three, four, number four, five.
And you just put the numbers in there.
And then when you get done, it does a calculation down at the bottom.
And I don't know if you have that here, if this is just an image, but it will
just validate that all 12 have filled it in and that the numbers are high enough.
Yeah, there it is down there.
So it's all red now down in the bottom where the orange box is.
When you fill this out, it'll turn to green, and then if it's valid, then
that little red box at the very bottom will turn to green, and that's it.
You print that out, attach it to the validation report again, and you can now
claim that validation report as your own.
On the content validation side, you get your own validation report.
So every validation that you do using the wizard is a custom validation
for content validity for a specific position at a specific location
for use by those people alone.
Jenny Arnez: Oh, that's really cool.
And we will for our clients, if you're listening to this and would like to have a
copy of that, just let reach out to us and we'll make sure that we send that to you.
Mike Callen: Absolutely.
Jenny Arnez: Yeah, so Mike, we've talked about content validity.
We talked about criterion validity.
I've also heard you mentioned face validity, right?
Mike Callen: Face validity.
I'd like to use a physics example here which some people hate physics,
but you have that something called centripetal centrifugal force.
You're in the car and the car turns left and the and you slam over into
the right hand side of the car, and that's called centrifugal force.
And it's really what's known as a pseudo force.
It's not a real force.
It's just that the car turned from underneath you and your
momentum kept going forward.
And so that right hand door ran into you, but it feels like a force.
And face validity is similar to that.
Face validity talks about it.
How does the test feel to the applicant?
Does it feel like a good experience?
And that's another reason why work sample tests are very important.
If you use work sample tests, you're presenting information to people, asking
them to do things that are similar to things that would be done on the job.
But we're doing it in a way where we don't require in prior job knowledge.
So we're asking them generic things, not things that they would need to know
about this particular organization.
Like, how do you use Excel in Acme County, for instance?
That's something that would be trained and we call it WT and LBO- would train
or learned in a brief orientation.
And so you don't want to test on those aspects.
But we test things in a generic format that we can expect anybody with experience
with that particular Excel, for instance, in this case, to be able to know.
And then we translate that to how well they did.
And then then it's validated for the particular job.
So that's how we do that for that part.
And actually we talked about face validity.
So if a test feels like a job and I fail the test, I get a feeling that I probably
wouldn't have done a good job on that job, and I don't feel really bad about that.
Our personality test for dispatchers was created with dispatch types of items, and
so it feels very much like it's asking you questions that if you were a dispatcher.
A lot of standardized personality tests have questions like, do you enjoy long,
quiet strolls on a beach by yourself?
Do you enjoy being in a loud movie theater?
And so it, those things are not very face valid.
They don't make you feel like you just asked me something that
has any applicability to the job.
And so face validity really is not important in terms of a test being valid,
but it goes back to the optics thing that we talked about at the begin with
to begin with, we now in the world realm of social media, everybody's got a voice.
And if I go and I test for a job at Acme company, and it's asking me
about whether I prefer hot dogs or hamburgers, I'm going to talk about that.
It's oh, when you take that test, it's going to be stupid, now all
of a sudden you're branded as, the people with the stupid test.
And so those are the reasons why you want to know about face validity.
And that's another thing that you can really capture, not directly
with our validation wizard.
But you've got a group of people and it's asking them questions like,
is the, here's the knowledge, skill or ability that's being measured.
Does it feel similar enough to the way that the knowledge, skill or ability
is in use on this particular job?
So you do end up getting, that litmus test as to whether or not
there was a face validity or not.
And that can be very helpful just in terms of your reputation.
Jenny Arnez: Oh, for sure.
I've taken our CritiCall test and just the stress that I felt and
then just not having the opportunity to finish one job before the next
call came in was really stressful.
Yeah.
Mike Callen: And it's really differentiating.
It's the thing that says, I started taking that test in 1999 before
it was released and I'm not a multitasker either, and I hate it.
So when I'm talking to people in the dispatch community, it's all about,
Hey, look, I could never do your job and I've never done their job, but
I've done the test and I hate the test, and I don't score well on it and
so it's very effective in terms of it.
In fact, you know what happens with the CritiCall test is people like me who are
not multitaskers will typically get up and walk away from the test rather than
getting hired and get up and getting up and walking away from the job after.
Somebody spent 60 or $70, 000 to train them.
So it's a very useful attribute of the testing process to make people
like you and me uncomfortable and let the real multitaskers get the job.
Jenny Arnez: And we have other tests too, right?
We have a situational judgment tests that actually also is a face valid too.
And.
I found the same experience there.
And I think one of the tests had to do with technical support or
customer service or something.
So all that's to say that TestGenius, and I'm not trying to make a sales statement
here, it's just something I'm really proud of about our software, is that our
tests are face valid You can have the opportunity to make it content valid.
And then also we have tests that have been criterion validated as well.
Mike Callen: Yes.
Yeah.
And those video situational tests that you referred to are phenomenal tests.
They're really great.
And they really hit on performance to mention attributes
that the other tests don't
don't hit on.
Not only does it ask you how you would respond in certain situations, but it
asks you how will that person feel about your response, given your response.
And so there's that it's not an EQ test.
It's not a, um, an EQ test per se that is emotional quotient
which is a different aspect.
But what it does is it requires that somebody has a degree of EQ to be able
to empathize to the person that they would be talking to so that they can
understand how that they were feeling.
And in many jobs, it's a really important aspect, particularly if it's front facing
or, interacting with members of the public is just a really important thing.
And that, and that brings up did we bring up the image with the the test?
The person the testing, the,
Jenny Arnez: oh, we didn't, no.
Mike Callen: Yeah let's show that for a second, because that's
a, that's an important aspect.
We can close on that.
So basically, when you're testing, that would be your job
applicant over to the left there.
And so as you add tests to your test battery or things that you're
talking about in an interview, you're going to get little tiny slivers
of job performance from each one.
And so by having a multifaceted testing strategy, you actually
continue on through this battery, and it might not all be TestGenius.
It might be personality testing.
It might be the Logi-Serve situational judgment testing.
It might be the interview.
It might be all sorts of other aspects, training and experience.
But each one of these things is going to contribute to the picture
of how job ready this person is.
And as you continue on going through this recruitment process, you're going
to learn more and more about them.
And if you pause for just a second right here, what you want to really
remember about this, is that you want to get as many different pieces
of the pie as possible, but it's equally important to make sure that
you're measuring it in the right way.
For instance, audio comprehension.
You could call something audio comprehension, but it's not, if it's
not valid and consistent with business necessity, it's not going to give you
the results that you would expect.
So that's just to something, something to consider and you
can go ahead and fill it out.
You're never going to, you're never going to know everything that you
need to know about a person, but you get to a point where, you've
measured all that you can measure.
And what it does is it helps to mitigate the risk.
I know 80 percent of what I can know about this person from having spent time
with them, talking to them reading their, job application, their experiences,
testing them, looking at those results.
So my, my risk is very low in terms of what I don't know about them.
And, so you can make a hiring decision with a greater degree of confidence when
you have a testing strategy that fits very well with the particular situation.
Jenny Arnez: Yeah, I love this.
This and you've already mentioned this, but this list, it looks like
a dispatcher position, but it could easily be administrative role where you
might have Microsoft Excel on there or
Mike Callen: right.
We're all sorts of
things, telephone answering they're just, spoken ability, English
abilities, foreign language ability, we can measure all of these things.
But it goes back to your test selection wizard.
You're going to go through and you're going to choose the duties that are
appropriate for that job, which are linked to tests that are appropriate for that
job that you then go on and validate.
And then you administer and you have a really good, holistic recruitment
strategy if you follow this process here.
Jenny Arnez: Yeah.
And that's really something to feel good about.
You're creating a process that hires the candidate who's most likely to
be successful on the job, your job, which leads to their fulfillment and
you as an employer having a good fit.
And it's fair and defensible.
That's something to be proud of.
Mike Callen: Yeah, it's fair and defensible and going to
end up with the right people.
One of the things that I don't think people put enough emphasis
on, it's not just about the person.
It's not just about the person that you hire or the person that you don't hire.
It's also about your existing employees.
They're the people who are carrying the burden when you had the job openings.
And so you want to reward these people by bringing somebody into the organization.
That is going to hit the ground running that has the requisite skills and
abilities necessary that can be trained up quickly, and then it's going to
fit with that organization and fit is something that maybe, maybe only a human
being in an interview can determine.
So you want to make sure whoever you're interviewing is somebody who
has the requisite skills and abilities already, because if you find that fit.
And you fall in love with that person and they can't do the job.
You have done a huge disservice to your existing employees.
And they will get so depressed at having gone through that process, waited
for somebody, got them all trained up only to find they can't do the job.
And so the order is very important.
In fact, if you go back to that first Acme slide that you have there.
A lot of people will do this after a phone interview and that third
brown box selected candidates take the preemployment test.
Our recommendation is, ask people on the application.
Everything that you can ask.
Are you willing to work nights?
Are you willing to work?
Evenings, holidays.
Whatever is appropriate for that job.
Ask them on the application.
Then take the people who are qualified applicants and then give them testing.
Don't ever even talk to them because you risk falling in love with the human
being who is not able to do the job.
And in fact, you're using human resources.
And I'm saying that term generically human, the resources of human beings.
You're using them to talk to people, who might not even
take the pre employment test.
So why not save that valuable human time with only the most highly qualified
applicant or candidate pool so that the things that, that software cannot
discern, that a human being can discern get sorted out in the interview.
And that will result in, the, a really great hire with really great benefits.
Jenny Arnez: Yeah, that's, I'm so glad you brought it back to this, Mike.
Thank you for that.
Mike Callen: Many
people do this.
They'll come up to us as shows or on, on the phone and mention it.
And, we have to be careful because, it might be, maybe
that's their civil service laws.
They have to do it that way, so we were a little soft handed
in terms of telling people.
You know what it is that they can do or should do, but the recommendation
is definitely leave the humans out as long as possible and only employ
the human beings time when you know you're working with a really
highly qualified list of candidates.
Applicants apply, candidates are the people that you move forward.
And that's a differentiation, but that's where you're, where you humans
should jump in and start measuring the things that only humans can measure.
Jenny Arnez: I think we're reaching an ending point today.
And Mike, I always learn so much when I talk to you, I've worked with you
and known you for a long time and now you've added another layer on for me.
So I appreciate it.
Mike Callen: I appreciate all your questions.
Your questions are great.
And they really, cause you to start thinking about the process and
think about other aspects of it.
So hopefully the discussions that we've had today will be ones that
will result in, positive step forwards for people who listen to the podcast.
And yeah, it's great.
Really.
Jenny Arnez: I hope so.
Make sure friends to check out the show notes.
We'll have a links to the Uniform Guidelines in there and other information
that we hope that you'll find useful.
So thanks again for joining us today.
Voiceover: Thanks for tuning in to Testing Testing 1 2 3.
Brought to you by TestGenius and Biddle Consulting Group.
Visit our website at testgenius.com for more information.