Post on 07-Apr-2018
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 1/48
Global
SelectionSystemsCase Studies & Lessons LearnedCase Studies & Lessons Learned
©2008 Hogan Assessment Systems
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 2/48
Overview of Session
• Survey of Issues in Global Selection – Jarrett Shalhoop, Hogan Assessment Systems
• Validation of the Shell Recruitment Process – Thi Bui, Royal Dutch Shell
• Global Assessment Development &Implementation
– Kelly Kaminski, Starwood Hotels & Resorts
• Applying Assessments Across Cultures: ACase Study
– Amie Lawrence, Select International
• Q&A Session ©2008 Hogan Assessment Systems
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 3/48
Global Selection Issues
• Perspective – Transporting an assessment outside of the US
and using for selection/assessment
• General Issues
– Language/Translation
– Assessment Norms
– Validation
• Logistical• Psychometric
– Technology
©2008 Hogan Assessment Systems
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 4/48
Language/Translation
• Three approaches to translation (van deVijver & Leung, 1997)
– Application – literal translation
– Adaptation – altering as necessary
– Assembly – developing a parallel assessment
• Back Translation
• Forward Translation
©2008 Hogan Assessment Systems
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 5/48
Assessment Norms
3-Component Model of Error (Meyer & Foster, 2008)
Sample Differences
Translation Differences
Cultural Differences
Absolute Sample Size
Relative Sample Size
Sample Composition
Translation Quality
Lack of Congruous Words
Cultural Relevance
Strength of Item Wording
Response Styles
Reference Group Effects
True Cultural Differences
Session 143 ©2008 Hogan Assessment Systems
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 6/48
Validation
• Logistical – Coordinating research globally a challenge
• Psychometric – Culture as a moderator of relationship to
performance
– Manifestation of characteristics
• Cultural response biases
– Willingness to provide accurate information
©2008 Hogan Assessment Systems
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 7/48
Technology
• Compatibility – Front end
• Web-based assessment
• Light assessment portal
– Back End
• HRIS Integrations• HR-XML Compliance
• Intellectual Property
– Protection of IP
©2008 Hogan Assessment Systems
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 8/48
Human Resources
LEARNING
4/28/2008
Copyright: Shell International Ltd 2005
Global Selection Validation Study Results
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 9/48
About Shell
• Active in over 130 countries
• Employs 108,000 employees world wide
• What we do:
– Exploration & Production
– Downstream
– Gas & Power
– Trading
– Renewables & Hydrogen
– Global Solutions
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 10/48
Global Recruitment Process
• Online application form via www.shell.com/careers
• People who pass screening are invited for interview
• Candidates who are successful at the first interview will be invitedto take part in a formal assessment – a chance for us to assess
their suitability and for the candidates to get a real insight into whatworking life at Shell would be like.
• Graduates can choose one of the three routes to Shell for their
assessment – Shell Recruitment Day, Gourami or AssessedInternship – while experienced professionals take part in astructured interview.
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 11/48
Background and Objectives
• Validation study of the Graduate and Experienced Hire (XP) selection
processes using data from a convenient sample from 2005 and 2006 hireswith readily available performance scores and selection data.
• This study represents the first in a series of ongoing annual validationwork. It is the first global validation study examining the relationship
between selection processes and work performance,1st year IndividualPerformance Factor (IPF) across the Shell Group.
• Systematic efforts to better understand the effectiveness of the selectionprocesses, including; the competence framework, the assessments, the
assessment exercises, and how each of these function in the regions.
• A major objective of this initial study was to better understand how data iscurrently being collected and stored.
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 12/48
Graduate Assessments
• Campus Interviews are functioning differently across the regions
• Evidence supports the effectiveness of SRD
• Assessed Internships are not functioning optimally
• Evidence supports the effectiveness of Gourami assessments
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 13/48
Graduate Competence Framework
• Capacity competence dimension is effective in predicting workperformance however regional differences are present.
• Achievement competence dimension is effective in predicting workperformance.
• Relationship dimension scores are not currently predictive of work
performance.
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 14/48
Graduate Assessment Exercises
• Campus Interview shows a positive but not statistically significant
relationship with work performance.• Campus Interview is more positively correlated with work performance in
the Americas.
• SRD exercises show positive but not significant relationships with workperformance.
• Gourami exercises show positive correlations with work performance.
• Assessed Internship show negative correlations with work performance.
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 15/48
Graduate Assessment Recommendations
• Expand the range of outcome measures
– First year IPF was used as a proxy measure for work performance in this initial study. – This metric does not cover the full range of success outcomes
– Recommend using additional criteria, such as manager ratings
• Improve data collection and archiving
– Automate and integrate data collection as much as possible
– Collect data on applications that do not progress fully through the selection system – totrack flow percentages and pass rates at successive assessment steps
• Improve measures of Relationship competence on Campus Interview and SRD
– The Relationship dimension is a valuable one (as evidenced by the Gourami dimensioncorrelations). Shell Learning will recommend improvements to the Campus Interview andSRD measures.
• Improve predictive validity of Assessed Internship
– Shell Learning will conduct an examination of the training and process factors that may beinhibiting these measures.
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 16/48
XPA Assessments
• Experienced Hire assessment exercises show weak (or slightly negative)
correlations with IPF 1st
year after hire.• Functional scores vary in relation to work performance by exercise.
– Project Discussion functional scores are more indicative to future workperformance than functional scores from the Professional Interview.
– Possible that the Project Discussion is used as the final assessment, with theStructured Interview and Professional Interview acting as precursor orpreliminary hurdles.
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 17/48
XPA Competence Framework
• The XP Competence framework dimensions show weak (and often weak
negative) correlations with work performance.
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 18/48
XPA Recommendations
• 2005 and 2006 XPA was not globally mandated, there may have been a
diversity of practice in implementation. – More rigorous and standardized training and implementation are essential to
improving the performance of these assessments.
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 19/48
Tips & Hints
• Setting client expectations. Timing, Data Collection, Results
• Use first study to understand data collection practices and issues
• Establish a partnership with Recruitment Department to identify dataneeded
• Utilize manager rating forms in addition of other performance indicators
• Have as few focal contact points as possible to collect data (1 per region)
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 20/48
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 21/48
©2008 Starwood Hotels & Resorts Worldwide, Inc.
Assessment Around the World:A Case Study from Starwood Hotels
Kelly A. Kaminski & Monica A. HemingwayStarwood Hotels & Resorts Worldwide, Inc.
23rd Annual SIOP Conference
April 2008, San Francisco
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 22/48
Starwood: A Global Company…
World class hotel and spa brands
890 managed & franchised properties in over 100 countries
Approximately 400 hotels managed by Starwood; we oversee theHR systems
145,000 associates
Conduct business in as many as 40 languages
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 23/48
…With a Global Assessment Program
CallCenter
US, Canada,and Ireland
2 versionslocally norms
Englishonly
Assessment Usage Versions Languages
Sales andMarketing
400 hotelsglobally
Varies by leveland division
9
GuestService
US, UK,China
Local norms 14
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 24/48
Our Approach to Job Analysis
Think globally, act globally: involve experts around the world
Sift through thousands of job titles
Job observations, and SME focus groups
One branded core competency model+ One guest service training program
+ One performance management system
= A common understanding of the jobs
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 25/48
Jobs Vary Across Divisions:
An Example for Sales Managers
Standard Title ComparisonTask
OverlapTask
rComp
OverlapComp
r
Sales Manager NA AP 76.9% .75 95.2% .92
Sales Manager NA EAME 68.8% .86 90.9% .84
Sales Manager NA LA 68.8% .82 95.2% .85
Sales Manager AP EAME 75.0% .84 95.5% .82
Sales Manager AP LA 75.0% .75 90.9% .81
Sales Manager EAME LA 100.0% .82 95.5% .88
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 26/48
Rank Ordering Guest Service Competencies
in North America and Latin America
Importance Ranking (1-25)
North America Latin America
Problem Solving 16 1
Attendance 4 15
Coping with Stress 3 13
Persistence 23 8
Multitasking 8 18
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 27/48
Translation
Translate and back-translate
Involve I/Os who are native speakers of the language
Use in-country reviewers to capture local dialects
Translate before validation Have your validation sample provide feedback about the quality
of the translations
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 28/48
How Many Languages Do You Need?
Our Sales Leaders Said “Just a Few”
English
57%
Bahasa2%
Japanese
3%
Italian
0%German
13%
French5%
Spanish (EU)
1%
Spanish (LA)
9% Chinese10%
N = 281
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 29/48
Getting Technology To Speak Your Language
Starwoodhotels.com English French Spanish
German Chinese JapaneseItalian
ATS Recruiter Site Thai Polish Portuguese
Job Postings Malay Dutch Korean
Assessment Authoring English Spanish (LA) French (CA)
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 30/48
Call Center Candidates in North America
and Ireland Respond Differently
32 of 53 items had statistically significant differences inresponse patterns between Ireland and the US applicants.
How would you describe yourself?Ireland USA
25% Dependable 78%
44% Clever 9%
33% Happy 11%
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 31/48
Implementation and Delivery
Legal requirements and data collection/retention policies vary
Difficulty integrating assessments with e-recruit platforms
Must allow candidates to choose a language
Your norms or mine? Does a candidate test in their homecountry or the country where the job resides?
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 32/48
In Conclusion
Think globally, act globally to avoid problems and get buy-in
Same job title does not mean same job
Same language does not mean same responses
Translate early but scope the need firstGood luck with the technology…
23rd Annual SIOP Conference
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 33/48
Amie Lawrence, Ph.D.
Lance Andrews & Matthew O’Connell, Ph.D.
Applying Assessments Across Cultures: A Case Study
April 10-12, 2008
San Francisco
Our Case Study
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 34/48
y
GLOBAL AUTOMOTIVE MANUFACTURER
• Needed standardized entry-level assessment process in 4 countries
– US, Canada, Mexico, UK
– All locations’ applicants were primarily local
• Web-based assessment
– Personality, SJT, & Cognitive Ability – Administered in group proctored settings
– Part of a multiple-hurdle selection process
Issue #1: Candidate Pr ivacy
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 35/48
y
CANDIDATE PRIVACY
• Other countries have laws limiting the collection and/ortransfer of candidate personal information
W HAT DID W E DO?
• Did not collect demographic data from any candidatesoutside the US
• Obtained approval from all UK candidates regardingtheir personal data
Issue #1: Candidate Pr ivacy
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 36/48
y
LESSONS LEARNED
• Know where candidate data are stored and where they will be transferred
• Involve the client’s legal department in the planning
stages of the project to ensure that these issues areaddressed
Issue #2: Applicant Tracking
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 37/48
pp g
APPLICANT TRACK ING
• Unique Identifier• Database Configuration
W HAT DID W E DO?• Created a unique identifier from telephone number and
month and day of birth
• Reviewed all fields in the system to align with thatcountry
Issue #2: Applicant Tracking
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 38/48
LESSONS LEARNED
• Discuss these issues early in project life-cycle• Verify every field that a candidate completes throughout
the selection process
Issue #3: Assessm ent Content
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 39/48
ASSESSMENT CONTENT
• The American version of the assessment was notappropriate for use in any of the foreign countries
W HAT DID W E DO ?• Translated into Spanish
• Reviewed assessment content for phrasing and spelling
changes• Programmed changes
Issue #3: Assessm ent Content
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 40/48
LESSONS LEARNED
• Changes may be needed even in other English-speakingcountries
• Account for the time and resources needed for this step
Issue #4: Assessment Norm s
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 41/48
ASSESSMENT NORMS
• No local norms available• No concurrent validation study planned
W HAT DID W E DO?
• Delayed report availability
• Conducted Norm Analysis
Issue #4: Assessment Norm s
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 42/48
LESSONS LEARNED
• Local Norms were necessary • Cultural & Regional differences were identified
Issue #4: Assessment Norm s
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 43/48
EVIDENCE FOR LOCAL NORMS
• Web-based Assessment Measurements
– 6 personality scales
– 6 cognitive measures (SJT, IP, Cognitive Ability)
• Assessment Differences by Country
• Assessment Differences by Item Type
Personality Difference Scores
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 44/48
Canada Mexico UK
Teamwork -.13 +.58 -.05
Conscientiousness -.07 +.02 -.44
Locus of Control -.08 +.87 -.65
Positive Attitude -.03 +.05 -.49
Attn to Detail -.11 +1.11 -.46
Safety -.02 +1.31 -.15
Mean Difference Scores as Compar ed to US Norm s
Cognitive Difference Scor es
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 45/48
Canada Mexico UK
IP: Gauge +.43 -.76 -.05
IP: Count +.32 -.57 +.06
SJT: Teamwork +.04 -1.56 -.47
SJT: Safety +.09 -1.25 -.03
Qualitative PS +.15 -1.45 -.44
Quantitative PS +.48 -1.33 +.03
Mean Difference Scores as Compar ed to US Norm s
Sum m ary of Differences
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 46/48
• Applicants in Mexico scored higher than applicants in
the US on all 6 personality measures
• Applicants in Mexico scored lower than applicants in theUS on all cognitive measures with the largest differences
on SJT and traditional cognitive ability measurements
• Applicants in the UK scored lower than applicants in theUS on all 6 personality measures
• Applicants in Canada were most similar to applicants inthe US
Futur e Research Questions
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 47/48
• What is the impact on validity with local v. global norms?
• What is causing the score differences? Are they truescore differences or is it “cultural” variance?
• Are all item types developed in and for US companiestransferable across cultures?
• What are the best processes and procedures for ensuringthat assessments can be confidently applied across
cultures?
About Select Intern ation al
8/4/2019 Global Selection Systems
http://slidepdf.com/reader/full/global-selection-systems 48/48
• Founded in 1993, Select International is a leading
provider of selection and development solutions forGlobal 2000 companies
• From entry-level to executive, organizations identify,
hire and retain top talent using Select International'scutting-edge products, systems and recruitment processoutsourcing services
• Headquartered in Pittsburgh, PA, Select maintainsoffices in Dallas, San Diego, Toronto, London and South Africa