Devel biodata items paullin et al 2006

1

Developing Biodata Inventories:

Cheryl Paullin, Ph.D. (HumRRO)Brian Katz, Ph.D. (PDRI)

Carrie Byrum, Ph.D. (HumRRO)

2006 IPMAAC Conference Presentation

Using Applicants’ Past Behaviors to Predict Job Performance

2 June 26, 2006 -- IPMAAC Conference Presentation

Objectives

1. Enhance understanding of biodata for use in selection testing.

2. Provide hands-on experience writing and reviewing biodata items.


Agenda• Introduction (10 min)• Item Types & Item-Writing Tips (15 min)• Item-Writing & Review Exercise (45 min)• Post-Item Development Activities (15 min)• Wrap-up (5 min)


What is Biodata?“Biodata measures are instruments assessing an

individual’s previous life experience, usually with a focus on targeting those previous experiences that predict future behaviors of interest.” (G. Stokes & L. Cooper)

• Underlying assumptions:– Past behavior is one of the best predictors of future behavior.– We are shaped by our past; it influences the kinds of choices

we make and activities that we pursue in the future.

• Typically self-report format.


Very Brief History• Beginning: Organizations needed to standardize and

score application blank information.

• Expanded application blank to include questions about personal history, including things like marital status, father’s job, parents’ income level, etc.

• “Dustbowl empiricism” in the early to mid 1900’s.– Didn’t care why life experiences predicted future job performance,

just found the ones that did and used them. – Classic: “Have you ever built a model airplane that flew?”– Lack of explanatory power caused this approach to lose favor.


Very Brief History (con’t)• Late 60’s-early 70’s, researchers (e.g., William

Owens & colleagues) began to explore why and how biodata works.– Theoretical basis for using biodata.– Blended approaches (e.g., “rainforest empiricism”

Mael).

• Past two decades, strong resurgence of interest in biodata:– Biodata Handbook (Stokes, Mumford, & Owens, 1994)– Concurrent with resurgence of interest in measuring

non-cognitive competencies (e.g., personality traits).


How Does Biodata Differ From Other Types of Assessment Tools?

• There isn’t a set of biodata “traits” similar to the Big 5 personality dimensions or Holland’s model of vocational interests.

• Rather, a biodata measurement strategy can be used to measure personality traits, interests, cognitive ability (indirectly), physical ability (indirectly), etc.

• No correct-incorrect answers, but some are worth more points than others.

• Minimal subjectivity because biodata items focus on what a person has done or experienced, not on feelings or beliefs.


Why Use Biodata?• It works! Valid predictor of things of interest to

employers.– Organizational Citizenship– Productivity– Turnover

• Items can be written by persons other than job experts, so development requires less time from job experts than some other assessment types.

• Incremental validity and lower adverse impact IF used to measure something other than cognitive ability.


Why Use Biodata (con’t)• Easily administered and scored in a variety of

settings.

• Less vulnerable to faking than some assessment types.– Historical and observable behaviors -- don’t require a

lot of self-insight. – Items are (or at least appear to be) verifiable.– Possible to do unproctored administration (although

practice is somewhat controversial).


Who Should Write Biodata Items?• Persons with a good understanding of:

– Competencies related to the thing you’re trying to predict (e.g., turnover, productivity)

– Job(s) in question (e.g., through job analysis)– Population that will take the test (e.g., external &

internal applicants, general level of education and job experience)

– Laws & rules that govern employee selection (e.g., ADA)

• (Typically) HR Specialists, Psychologists• (Sometimes) Supervisors, Managers


Biodata Item Types:Discrete Option Choices

In the past, which of the following work situations has provided you with the best work environment?a. Coworkers praised your work on a regular basis.b. Laid back working environment.c. Able to take an active part in work-related social activities.d. Supervisor was very watchful and detail oriented.e. Regularly rewarded with financial bonuses for good

performance.f. Other [escape option – maybe necessary, better to avoid if

possible]


Biodata Item Types (con’t):Numeric Continuum Scale

a.Zero timesb.One timec.Two timesd.Three timese.Four or more times

How many times in the past month have you been late to a work meeting?


Biodata Item Types (con’t): Frequency Response Scale

a.Very Oftenb.Oftenc.Sometimesd.Rarelye.Never

How often do you produce your best work products when you are under a lot of stress?


Biodata Item Types (con’t):Agreement Response Scale

a.Strongly Agreeb.Agreec.Neither Agree nor Disagreed.Disagreee.Strongly Disagree

When I am faced with a work activity I find unpleasant, I tend to put it off as long as possible.

NOTE: Many types of response scales (e.g., importance, to what extent)


Other (Less Popular) Biodata Item Types

• Dichotomous– Have you changed jobs more than once in the

last year?

• Choose all options that apply– Which of the following activities have you

participated in during the past year? [Check all that apply]


Empirical Approach to Inventory Development and/or Scoring

• Ask questions about a wide variety of life history experiences.

• Gather responses from a large sample similar to your target sample.

• Figure out which items or response choices are, in fact, related to the thing you’re trying to predict.

• Score only those items or response choices.• Cross-validate the scoring key (i.e., evaluate the validity of

the scoring key in a sample other than the one in which you developed it).


Rational/Construct-Oriented Approach to Inventory Development & Scoring

• Identify and define relevant competencies (e.g., through a job analysis study).

• Write questions that reflect life history experiences relevant to developing and/or exhibiting those competencies.

• Create a scoring key for each question, specifying how you expect persons high and low on the competency to respond.


Item-Writing Exercise: Case Study• The IPMAAC organization has openings for

two temporary positions at this Conference– Administrative Assistant (A.A.)

• Handles phone calls regarding conference questions• Assembles all attendee conference materials• Works independently

– Law Enforcement Officer (Law)• Ensures order is maintained at conference• Arrests and processes unruly conference participants• Works with team of officers and hotel security


Construct-Oriented Approach: Identify Relevant Competencies

XInitiative and MotivationXPhysical FitnessXStress Management XAdapt to ChangeXXRelate to OthersXTeamwork

XCognitive AbilityXOrganizing and Planning

LawA.A.


Mindset for Writing Biodata Items• Start with an understanding of job

requirements and the type of people who will be filling out the biodata inventory.– Maintain focus on job-relevant questions.– Avoid questions that some test-takers won’t be able to

answer.

• Think of people you know who are high or low on the target competencies. Think about and generalize from the kind of experiences they have had or the behaviors you’ve observed when interacting with them.


Biodata Item-Writing Tips• Short, direct statements or questions.

• Ask about observable behaviors, not unobservable attitudes, beliefs, or opinions.

• Focus on things the test-taker could control (e.g., his/her own career choices, not his/her parents).

• Be sensitive to gender, SES, and cultural differences (e.g., don’t assume everyone lived in a two-parent home, grew up in an urban area, and participated in sports).


Biodata Item-Writing Tips (con’t)

• Make sure the items are not invasive or offensive. – Ask about things that have nothing to do with a test-taker’s work

habits or ability to perform the target job (e.g., sexual behavior, religious practices.)

• Write items such that response choice “A” is not always the highest-scoring response.

• Avoid items for which the most socially desirable answer is totally obvious but don’t go to great lengths to make the items highly subtle (which runs the risk of making them non-job relevant and offensive).


Structured Item Review Exercise• Break into two or three small groups.

• In each group, review several items that we prepared in advance (20 minutes).

• Discussion Questions:– For which job is the item relevant?– What competency does the item measure?– Do the response choices make sense? Are they

comprehensive?– Would everyone understand the item?– Would some people be offended by the item?


Item Writing Exercise: Construct-Oriented Approach

• Half of the room focuses on the A.A. job; half focuses on the Law job.

• Assignment (10 minutes):– Write two items to measure “Relate to Others.”– One item should be of the discrete option type (Slide

11) and one should be of the response continuum type (Slides 12-14).

– Indicate how you would score each item.


Discussion Questions

• What did you consider when thinking of the likely test-takers?

• Which type of item – discrete or continuum --was harder to write?

• Which was harder to score?

• Should your question differ depending on the job for which you were writing?


Unstructured Item Review

• Volunteers read one of their items and lets the group critique it.

• Workshop leaders also comment, as needed.


Post-Item Development Activities• Write instructions for test-takers.• Assemble items into an inventory.

– One form vs. multiple forms (usually different item presentation orders, not different items).

– Intermingle items for different competencies.• Pilot test in a sample similar to test-takers.

– Ask for feedback about test instructions and items. – Remove items:

• with little response variability. • that receive more than one negative comment.


Post-Item Development Activities (cont’)

• Choose method of administration:– Paper & pencil vs. computerized/online.– Proctored vs. unproctored.

• Choose scoring method:– Pre-defined key (rational/construct-oriented approach).– Data-driven key (empirical approach).

• Train Test Proctors (if you’re using them).• Think carefully about test security issues!


Questions & Wrap-up

• Thanks for attending!

• Feel free to call any of us with further questions!– Cheryl Paullin (612) 379-3834, ext 301– Brian Katz (703) 812-3057– Carrie Byrum (703) 706-5631

Devel biodata items paullin et al 2006

Business

Transcript of Devel biodata items paullin et al 2006