Overcoming the Fear of Employment Testing: What Exactly is “Testing”?

Presented by: Chris Atkinson, M. S. and Jim

Kuthy Ph.D.

Visit BCGi Online

• If you enjoy this webinar,

– Don’t forget to check out our other training opportunities through the BCGi website.

• BCGi Standard Membership (free)

– Online community

– Monthly webinars on EEO compliance topics

– EEO Insight Journal (e-copy)

• BCGi Platinum (paid) Membership ($299/year)

– Fully interactive online community

– Includes validation/compensation analysis books

– EEO Tools including those needed to conduct AI analyses

– EEO Insight Journal (e-copy and hardcopy)

– Members only webinars and training and much more…

www.BCGinstitute.org

HRCI Credit

• BCGi is an HRCI Preferred Provider

• CE Credits are available for attending this

webinar

• Only those who remain with us for at least 80%

of the webinar will be eligible to receive the

HRCI training completion form for CE

submission

44

About Our Sponsor: BCG

• Assisted hundreds of clients with cases involving Equal Employment

Opportunity (EEO) / Affirmative Action (AA) (both plaintiff and defense)

• Compensation Analyses / Test Development and Validation

• Published: Adverse Impact and Test Validation, 3rd Ed., as a practical

guide for HR professionals

• Editor & Publisher: EEO Insight an industry e-Journal

• Creator and publisher of a variety of productivity

Software/Web Tools:

– OPAC®

(Administrative Skills Testing)

– CritiCall®

(9-1-1 Dispatcher Testing)

– AutoAAP™ (Affirmative Action Software and Services)

– C4™ (Contact Center Employee Testing)

– Encounter™ (Video Situational Judgment Test)

– AutoGOJA®

(Automated Guidelines Oriented Job Analysis®

)

– COMPare: Compensation Analysis in Excel

Contact Information

Jim Kuthy, Ph.D.Principal Consultant

jkuthy@biddle.com

(800) 999-0438 x 239

Chris Atkinson, M.S.Consultant

catkinson@biddle.com

(800) 999-0438 x 120

The Presenters…

• Jim holds Masters and Doctorate Degrees in Industrial/ Organizational Psychology

• Jim has taught Psychology and Business-related courses at the University of Akron and California State University, Sacramento

• Chris holds a Masters Degrees in Industrial/Organizational Psychology

• More than twenty five combined years of experience in the employment selection field

• They have designed and validated a variety of employment tests for many employers

Outline

What is testing?

A Brief Overview of Adverse Impact

A Brief Overview of Test Validation

Commonly Used Employment Tests

Using a “Whole Person” Approach to Testing

Common Testing Myths

What is Employment Testing?

• Assessment Standpoint – A test is a

standardized series of problems or questions

that assess a persons knowledge, skills, abilities,

or other characteristics

• Legal Standpoint - Uniform Guidelines on

Employee Selection Procedures defines a test as any

method used to make an employment decision

U.S Employment Test Usage

• 41% of employers test job applicants in basic literacy and/or math skills.

• 34% of job applicants tested in 2000 lacked sufficient skills for the positions

they sought.

• 68% of employers engage in various forms of job skill testing.

• 29% of employers use one or more forms of psychological measurement or

assessment.

• 20% of employers use cognitive ability tests

• 8% of employers use interest inventories.

• 14% of employers use some form of managerial assessments

• 13% of employers use personality tests 10% of employers use physical

simulations of job tasks

***This information is based on survey data collected in 2001 from 1,627 HR managers who

are members of the American Management Association. The survey accurately reflects the

AMA membership base, and is therefore more reflective of larger organizations than all

employers in the U.S. economy.

Reasons for Testing

1. Testing leads to savings in the decision-making

process

2. The costs of making a wrong decision are high

3. The job requires attributes that are hard to

develop or change

4. Hard-to-get information can be obtained more

easily and efficiently

5. Individuals are treated consistently

6. There are a lot of applicants

A Brief Overview of Adverse Impact

Men

Pass (50)

Men

Fail (50)

Men Passing Rate

(50%)

Women

Pass (35)

Women

Fail (65)

Women Passing

Rate (35%)

Adverse Impact

• 80% Test

• Statistical Significance

A substantially different rate of selection in hiring, promotion, or other

employment decision which works to the disadvantage of members of

a race, sex, or ethnic group

Adverse Impact (i.e., significant difference in rates)

Sufficient Evidence of

Job Relatedness

/Validity

Disparate Impact

Discrimination

According to the 1964/1991 Civil

Rights Act(s)

A Brief Overview of Test Validation

• Traditionally, validation is making sure a selection

practice, procedure, or test (PPT) appropriately measures

what it is designed to measure

• In a legal realm, a selection procedure is valid if it can be

proven by an employer that it is “…job related and

consistent with business necessity”

• It is the inferences regarding the specific uses of a test or

other measurement procedure that are validated, not the

test itself

What is Validity?

What Needs to be Validated?

• Any practice, procedure, or test (PPT) exhibiting

adverse impact

– Written tests

– Interviews

– Physical ability tests

– Resume screens

– Virtually any process that is used for making

employment-related decisions

Validity Overview

• Content: A connection between the important parts of the

job and the test

– Requires demonstrating a connection between the job (using a job

analysis) and the content of the test

• Criterion: A mathematical study that proves the test

predicts job performance:

– It ALWAYS requires: A statistical study and results that are

“statistically significant” (<5% chance)

• Construct: A connection between a test, a trait, and job

performance:

– It ALWAYS requires “empirical evidence” connecting the test to

the trait and the trait to the job (all 3)

Job Duties Operationally

defined KSAs

Other

KSAs

Selection Devices

(e.g., applicationform, tests,

Interviews, BQs)

Content Validation Process

Job Duties

Other

KSAs

Selection Devices

(e.g., applicationform, tests,

Interviews, BQs)

Content Valid!

Operationally defined KSAs

Content Validation Process

20

Criterion-Related Study

0

10

20

30

40

50

60

70

0 20 40 60 80 100

Test Score

Pe

rfo

rma

nc

e M

ea

su

re

Score on some “Criteria” (e.g.,

job performance, days missed

work, etc.)

Score on a

“Test”

Criterion-related Validity

Commonly Used Employment Tests

Commonly Used Employment Tests

Assessment Centers

Personality Testing

Biodata

Cognitive Ability Tests

Job Knowledge Tests

Situational Judgement

Testing

Employment Interviews

Training Education and

Experience (TEE)

Work Sample Tests

Cognitive Ability Tests

• Cognitive ability tests assess abilities involved in

thinking (e.g., reasoning, perception, memory,

verbal and mathematical ability, and problem

solving)

• Examples: “General mental ability”, math,

verbal analogies, reading comprehension, spatial

reasoning

Cognitive Ability Tests

• Validity: High validity for a variety of jobs. The more

complex the job, the higher the validity

• Applicant Reactions: Tests developed for specific jobs

have higher face validity than off-the-shelf tests

• Adverse Impact: High against minorities

• Development Costs: Purchasing off-the shelf tests is

typically less expensive than developing custom tests

• Administration: Easy to administer to many applicants

• Use: Best used for jobs requiring specific cognitive

skills and more complex jobs

Job Knowledge Tests

• Assesses technical or professional expertise in

specific knowledge areas.

• Not appropriate when applicants will be trained

after selection in the critical knowledge areas

needed for the job.

• Examples: Basic accounting principles,

computer programming, and financial

management

Job Knowledge Tests

• Validity: High - Customized job knowledge tests have

been shown to have slightly higher validity than off-the-

shelf tests

• Applicant Reactions: Tend to be favorable

• Adverse Impact: High against minorities

• Development Costs: Frequent updates to the test

content and validation may be needed to keep up with

changes in the job

• Use: Best used for jobs requiring specific job

knowledge on the first day of the job

Situational Judgement Tests

• Situational judgment tests (SJTs) present a description

of a work problem or critical situation related to the job

and asks applicants to identify how they would

respond

• SJTs measure effectiveness in social functioning

dimensions such as conflict management, interpersonal

skills, or problem solving,

• SJT questions and alternatives are typically based on

critical incidents generated by job experts. Scores are

based on subject matter experts' judgments of the best

and worst alternatives.

Encounter Situational Judgment Testing

Situational Judgement Tests

• Validity: High content validity, moderately high

criterion-related validity

• Applicant Reactions: Very positive

• Adverse Impact: Moderate against minorities

• Use: Typically used for managerial positions or

other jobs requiring interpersonal interactions

Employment Interviews

• Interviews are subject to the requirements of the

federal Uniform Guidelines on Employee Selection

Procedures (1978), similar to any other type of test

or selection procedure

• Unstructured Interviews

– No constraints on types of questions asked

– Global evaluation of applicant responses

• Structured Interviews

– Asking exact same questions to all applicants

– Applicant responses evaluated against rating scales

Interview Validity

Least Validity Highest Validity

Unstructured

Single Rater

Generic “one size fits all”

Open Scoring/No Scoring

Structured

Multiple Raters

Job Specific

BARS

Structured Interviews

• Applicant Reactions – Unstructured interviews

are more favorable than structured interviews

• Adverse Impact – Low

• Typically used late in the assessment process as a

final screen

Training, Education & Experience

(TEE) Requirements

• A systematic method used to assess previous

experience, education, and training information

provided by job applicants

• Often collected as a supplement to the application

• This information is evaluated against education and

experience benchmarks to generate scores for

selection purposes blank

Training, Education & Experience (TEE)

Requirements

• Training requirements posted by employers as hiring or

promotion criteria.

– Certifications, credentials, training institution completions,

licensing requirements

• Educational requirements, including:

– Degrees, specific course requirements, etc.

• Experience requirements used as hiring or promotion

criteria, including:

– Closed systems (e.g., time-in-grade requirements)

– Open systems (inside/outside experience counts)

Training, Education & Experience

(TEE) Requirements

• Validity: Highly representative of the job, but

do not relate well to performance on the job.

• Applicant Reactions: Typically more positive

reactions from more experienced applicants

• Adverse Impact: Little to none against women

or minorities

• Uses: Commonly used as a screening device

prior to another selection tool (e.g., structured

interview).

Work Sample Tests

• Work sample testing refers to “hands-on”

performance tests in which a job candidate is

required to perform (or describe) activities/tasks

that are performed on the job

Callinan, M., & Robertson, I. T. (2000). Work sample testing. International Journal of Selection and Assessment, 8(4), 248-260.

Work Sample Test Examples

• Mechanic– Repair a problem on a car

– Use a repair manual

• Clerical– Typing Test

– Proofreading test

• Police Officer– Writing test

– Climbing, running

• Computer Programmer– Create a simple program

– Debug a current program

Adapted from “Selection Assessment Methods,” Pulakos 2005

Work Sample Tests

• Validity: One of the most valid forms of testing

• Adverse Impact: Typically low – depends on the

competencies being assessed

• Applicant Reactions: Favorable – Provides a realistic job

preview

• Development and Administration Costs – Can be high

• Best Used When:

– Measured competencies are critical for successful performance

– Limited number of applicants

– Small number of applicants being tested

Assessment Centers

• Employs multiple assessment methods and

exercises to evaluate a wide range of

competencies

• Applicant performance is usually observed and

evaluated by multiple assessors (i.e., raters)

• Examples: In-basket exercises, leaderless group

discussion

Assessment Centers

• Validity: Moderate to high

• Applicant Reactions: Favorable - Many exercises

serve as a realistic job preview

• Adverse Impact: Low to moderate

• Development Costs: High in terms of time and

money

• Administration Costs: High - Requires many raters

• Use: Particularly effective for assessing higher-level

managerial and leadership competencies as well as sales

and customer service

Personality Testing

• Personality tests elicit information about a person’s motivations,

preferences, interests, emotional make-up and style of interacting with

people and situations

• Self-report inventories typically ask applicants to rate their agreement

level on a various items designed to assess various traits

• Examples: 16PF, OPQ, Hogan Personality Inventory, Big 5

• Specific traits are more predictive for particular jobs

• Personality tests designed to assess psychiatric conditions are

considered medical examinations

Personality Testing

• Validity: Moderate levels of validity for a wide range of

outcome measures in a variety of settings

• Applicant Reactions: Not very favorable, may appear

as invasive

• Adverse Impact: Few, if any differences between men

and women, or ethnicities

• Administration: Easy to administer to large groups as

a screen

• Use: Often used when job performance requires a great

deal of interpersonal interaction or work in team

settings

Biodata

• “Biodata measures (also known as biographical

data, autobiographical information, and life

histories) are instruments assessing an

individual's previous life experiences, usually

with a focus on targeting those previous

experiences that predict future behaviors of

interest.”

Cooper, A. L., & Stokes, S. G. (2003). Biodata. In Hersen, M., Thomas, C. J (Eds.), Comprehensive Handbook of Psychological Assessment, Industrial and Organizational Assessment (Volume 4). Hoboken, NJ: Wiley

The Many Faces of Biodata

• Biodata can tap into many different dimensions

– Cognitive Ability: “What was your academic rank in your

high school graduating class?”

– Personality: “How often did you attend social events during

high school”?

– Vocational Interests: “How often do you read science

related publications”?

– Values: “During high school, how important was it to you to

be regarded as successful?

Cooper, A. L., & Stokes, S. G. (2003). Biodata. In Hersen, M., Thomas, C. J (Eds.), Comprehensive Handbook of Psychological Assessment, Industrial and Organizational Assessment (Volume 4). Hoboken, NJ: Wiley

Biodata

• Validity: Moderate – predictive of other criteria

such as turnover attendance, and safety behavior

• Applicant Reactions: Typically not as

favorable as other assessment methods

• Administration: Can be administered to many

applicants at once

• Adverse Impact: Low against minority groups

Using a “Whole Person” Approach to Testing

As you add more

devices or tests that

measure different

aspects of a person,

you will learn more

about that person

The U.S. Department

of Labor refers to this

as the

“Whole Person”

approach to selection

Observable

Behavior

Performed

During Work

Sample Test

Test of

Attitude,

Personality,

Biodata, Values,

etc.

Written

Knowledge

Test

Interview

Imagine you had the time and resources to conduct

multiple types of selection testing

Test of

Attitude,

Personality,

Biodata Values,

etc.

Written

Knowledge

TestInterview

Research shows work

sample tests are

typically among the

most directly related

to observable job

performance

Observable

Behavior

Performed

During Work

Sample Test

These generally include samples of work performance (e.g., data

entry, video-based tests)

Written

Knowledge

Test

Interview

Adding a test of

attitude,

personality,

values, biodata,

etc. can provide

additional helpful

information

about the job

applicant, but is

typically

somewhat less

directly related to

observable job

performance

Test of

Attitude,

Personality,

Biodata, Values,

etc.

Observable

Behavior

Performed

During Work

Sample Test

Test of

Attitude,

Personality,

Values, Biodata,

etc.

Written

Knowledge

Test

InterviewObservable

Behavior

Performed

During Work

Sample Test

Multiple

Selection

Devices can

measure the

same factors

about an

applicant in

the same way,

the same

factors in

different ways,

and/or

different

factors in the

same way

It is typically

best to try to

measure as

many different

factors about a

person as

possible

during testing

while

balancing

available

resources

Interview Interview

Work

Sample Test

Personality

Test

Knowledge,

Skill, Ability

Test

Personality

Test

Multiple-HurdleCombined Score without Hurdles

Knowledge,

Skill, Ability

Test

Work

Sample Test

Use Test Scores Appropriately

• Combining personality test scores with scores from other

selection devices may result in lower adverse impact

• You can combine scores from various types of tests, but you

should

– Only use scores from those who scored at or above some minimum level (i.e.,

they must “pass” that section of the selection process)

– Weight scores from each selection device appropriately

– Statistically standardize the scores from different tests into a single metric, such as

T-scores or Z-scores, before combining them together

• We suggest that you consider using personality test scores from

attitude/personality tests as a “pass or fail” hurdle

– Giving a personality test earlier in the process can frequently reduce potential

adverse impact during a testing process

• Ranking assumes one applicant is reliably more qualified than the other

• Banding considers the unreliability of the test battery and “ties” applicants

• Pass/fail cutoffs treat all applicants as either “qualified” or “not qualified”

• Weighting/combining test scores can be done using “compensatory” or using cutoff on each test then weighting results

Applicant Score

Tom 100

Stacy 100

Bob 100

Frank 100

Julie 99

Rozanne 99

Mark 98

Luke 98

Henry 97

Paul 97

Peter 96

Rebecca 96

Alyssa 95

Matthew 94

John 93

Annette 93

Ray 92

Thomas 91

Julissa 90

Test Scores Use

Comparison Between “Test Use” Options

Factor Ranking Banding Pass/Fail

CutoffsValidation Requirements High Moderate Low

Adverse Impact High Moderate Low

Defensibility Low High High

Litigation "Red Flag" High Moderate Low

Utility High Moderate Low

Cost Low Moderate High

Applicant Flow Restrictive/

Controllable

Moderate/

Controllable

High

Development Time Low Moderate High

Reliability Requirements High Moderate Low

# Item Requirements High Moderate Low

Common Employment Testing Myths

57

A good test or interview will guarantee the people

you hire will be successful

• Selection devices help you to improve the likelihoodyou are hiring the best person for the job

– They cannot guarantee success (no matter what the sales person says)

• This is why we should try to capture as much as possible about the job candidate using multiple devices, attempting to measure the “whole person”

Myth #1

Myth #2

The federal Uniform Guidelines on Employee Selection Procedures(Section 7A ) say that it is your responsibility, NOT the developers of the selection devices you use, to ensure that they are fair (www.uniformguidelines.com)

– Be proactive…

o Make sure the selection device you use is valid -- Obtain copies of the validity report

o Have the validity report reviewed by professionals– Many Universities have Industrial & Organizational Psychology programs

that may be able to help you

– Ask the right people for assistance

o Stay current in your research– What was good yesterday, may not be good today

The test publisher is responsible for making

sure the test is fair and valid

59

It is better not to test at all because then

you cannot be sued

1. Your chances of obtaining qualified employees without testing is just that… a chance (and an unlikely one at that!)

2. Well constructed and validated tests are used successfully by the best organizations with very few or no problems

3. You are actually more likely to have problems if you do not use appropriately constructed & validated selection devices because

– That allows for discrimination to sneak in

– You have no evidence to back your employment decisionso Without evidence the courts will infer discrimination

Myth #3

60

You should only use tests that have absolutely no

adverse impact

• It is virtually impossible to eliminate adverse impact when attempting to measure some types of skills and abilities

• That being said, minimizing adverse impact is one factor to consider

• However, there are other factors to consider, such as making certain the job candidate can perform at least at a minimally qualified level

• Using only tests with no adverse impact means that you will not be measuring many of the important knowledge, skill, or abilities that are necessary to perform the job

Myth #4

61

It’s Too Expensive to Test• Testing can be expensive, but hiring the wrong person can

be even more expensive– Compute the costs of a “bad” hire

• You can actually reduce employee costs by choosing the most qualified person for the job– Less supervision

– Less training

– Reduced turnover

• You can sometimes reduce testing costs through better planning– Set and enforce valid basic qualifications to minimize the number of applicants to

be tested

– Use Realistic Job Previews to have those who are not truly interested “select” themselves out of the testing process

Myth #5

62

Testing is Too Time Consuming

• How much time are you willing to spend to

find the best people for the job?

• Balance the amount of time it takes to hire

the best person for the job against how long

you would like that person to STAY on the

job

Myth #6

Conclusion

• Employment testing can be a valuable method

to use in your hiring process

• Test validation is required if there is adverse

impact with the use of a test

– Test validation makes good business sense regardless

• Consider many factors in test selection

• Use a “Whole Person” approach to testing

Questions?

800-999-0438