Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL...
Transcript of Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL...
![Page 1: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/1.jpg)
Qualification Standards: Issues and Implications
U.S. Merit Systems Protection Board
Office of Policy and Evaluation
Presentation to the IPMAAC Annual Conference
June 11, 2008
![Page 2: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/2.jpg)
Overview
• Background
– Who we are
– Study scope, questions, and methodology
– The Federal MQ structure
• Issues and hypotheses
• Promising developments
![Page 3: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/3.jpg)
Who We Are: About MSPB
MSPB
FLRA
EEOC
OPM
OSC
DOL
MSPB is one of
several Executive
Branch agencies
responsible for some
aspect of the Federal
civil service.
Adjudicate employee appeals
Monitor the health of the Federal
civil service
• Study and evaluate Federal
agency HR policy and practice
• Report to the President and
Congress
![Page 4: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/4.jpg)
Study Scope and Questions
• Scope: Federal Government, professional and
administrative (PATCO P/A) occupations
• How do agencies define and assess MQ?
• What functions do MQs serve?
• How well are MQs working?
• What changes, if any, should be made?
– Structure and content of MQs
– Implementation of MQs
– Policies related to MQs and ratings
![Page 5: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/5.jpg)
Purpose: Why MQs Matter
• Workforce quality
• Fairness and legal compliance
• Effects on applicants and employees
• Efficiency and effectiveness
– Cost of development and application
– ROI of recruitment and assessment
![Page 6: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/6.jpg)
A Quick Thought Exercise
• What does “qualified” mean to you?
– Able to do the job? With what likelihood?
– “Not unable” to do the job?
• What does “qualified” mean to hiring
managers, policymakers, stakeholders, and
third parties?
• Are these definitions mutually consistent?
![Page 7: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/7.jpg)
Methodology
• Literature review
• Agency questionnaire
– Implementation practices
– Functions, ROI, challenges, issues
– Perspective on possible changes
• Discussion with policymakers
• Text analysis of job announcements
• Idea: simulation
![Page 8: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/8.jpg)
The Federal MQ Structure
• Standards established by the
U.S. Office of Personnel Management (OPM)
• Examining policies grounded in law, defined and
implemented in regulation by OPM
• System designed for delegation and flexibility
• Agencies are responsible for –
– Defining qualifying experience
– Determining required KSAs/competencies
– Assessing applicants
![Page 9: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/9.jpg)
The Federal MQ Structure
• Most standards follow a common pattern
• Standards built around the Federal Government’s
pay system and staffing rules
Component Requirement
Experience • Three years of general experience for lowest entry level.
• One year of specialized experience for all other grade levels.
(“Specialized experience” is experience equivalent to the next lower
grade level that confers necessary competencies.)
Education May substitute for experience at entry- and mid-level.
Positive educational requirement for professional positions only.
Objective Test None for most occupations.
Other Biodata (achievement record) for many occupations.
Reference: OPM Qualification Standards Handbook.
![Page 10: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/10.jpg)
Issues and Hypotheses
• Issues
– Limited research
– Conflicting expectations
– Leveraging technology
– Aligning policy and practice
• Hypotheses
– Risk: inconsistent definition
– Risk: incomplete coverage of competencies
– Risk: inconsistent assessment
![Page 11: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/11.jpg)
Issue: The State of the Art
• Limited research and literature
– Focus on lower-level, “higher-n” jobs
– More attention to selection than screening
– Nature of MQs constrains research
– Problematic from an I/O perspective?
• More information on process than content
• R&D driven more by litigation than utility
![Page 12: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/12.jpg)
Issue: MQs vs. “Cut Scores”
• Research often equates MQs and cut scores
• Federal MQs function like cut scores...
• The functional resemblance is deceptive
Dimension MQ Cut Score
Instrument T&E self-report Objective test
Content Primarily technical General/technical
Scoring Dichotomous
Rater-dependent
Continuous
Rater-independent
Instrument Use Screening Screening/Ranking
![Page 13: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/13.jpg)
Issue: Conflicting Expectations
Flexibility Content validity
Legal: Establish a minimum I/O: Define a minimum
Decentralization “Professionally developed”
“Small n” Traditional job analysis
State of the art High selectivity
![Page 14: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/14.jpg)
Issue: Conflicting Expectations
Openness Applicant reduction
Self-report Reliability and precision
Low administrative burden High-stakes decision
Low cost Defensibility
Low applicant burden Behavioral T&E
Acceptable performance High performance
![Page 15: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/15.jpg)
Issue: Multiple Functions
• Administrative convenience
• Sorting
– “Screen out” those unlikely to succeed
– “Screen in” those likely to succeed
• Communication
• Promote fairness and consistency
• Legal compliance
![Page 16: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/16.jpg)
MQ: High Stakes
• Important to applicants and employees
• Subject to scrutiny and litigation
• Important to agencies and hiring managers
– Selection eligibility or priority contingent on MQ
– MQ may eliminate or preclude selection
of highly qualified applicants
![Page 17: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/17.jpg)
Issue: Leveraging Technology
• Technology provides potential
– Lower cost of application
– New assessment options
– Transparency
• Technology creates challenges
– Higher volume of applicants
– Reliability and integrity
![Page 18: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/18.jpg)
Issue: Aligning Policy and Practice
• Separate evolution of practice and policy
• Challenges
– Shift to knowledge work
– Broadband pay systems
– Technology
– Emphasis on performance and results
– Decentralization
![Page 19: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/19.jpg)
Risk: Inconsistent Definition
• Factors
– Decentralization
– Quality of job analysis
– Differing understandings of “qualified”
• Indicators
– Variation within occupations
– Questionable “mandatory” requirementsExample: Selective factor for a Supervisory IT Specialist
Experience managing a staff supporting an
enterprise data network, server platform,
and IT helpdesk for the Federal Government.
![Page 20: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/20.jpg)
Risk: Incomplete Coverage
• Factors
– Quality of job analysis
– Difficulty of defining minimum level
– Reliance on T&E
• Indicators
– Poorly-defined competencies
– Emphasis on technical competencies
![Page 21: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/21.jpg)
Risk: Incomplete Coverage
One year of specialized experience in the same or similar
work at a level of difficulty and responsibility
equivalent to the next lower pay band that has equipped
you with the necessary knowledge, skills or abilities to
perform successfully the duties of the position.
Creditable specialized experience includes experience
such as a working knowledge of a wide range of security
policies, concepts, principles, and practices to review,
independently analyze, and resolve difficult and complex
security problems; security program interrelationships;
implementation of security policies; and conducting
inspections of security programs.
Example: Experience requirement for a program analyst position
Source: Vacancy announcement for a program analyst position, posted on USAJOBS on May 29, 2008.
![Page 22: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/22.jpg)
Risk: Inconsistent Assessment
• Factors
– Raters and SMEs
– Definition and understanding of requirements
– Self-report information
– Decentralization
– Measurement challenges
• Indicators
– Previous research
– Experiences of employees and job applicants
![Page 23: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/23.jpg)
Risk: Inconsistent Assessment
Source: Responses to 2006 MSPB survey of new upper-level hires.
“Several times I was assessed by HR personnel as
not qualified at a level for a position for
which I was well qualified.
When I inquired why, I was informed that the HR
staff was either new or didn’t understand the
position.”
“I applied for a GS-7 (in my retired military
career field) and came up not qualified; but
then I applied for a GS-12 (same field) and came
up qualified.”
![Page 24: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/24.jpg)
Promising Developments
• Corporate approach
– Workforce analysis
– Task forces
• Cross-occupational approach
• Flexibility plus framework
– Competency models
– Assessment guidance
![Page 25: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/25.jpg)
Promising Developments
• Enhancements to T&E
– Accomplishment record
– Electronic questionnaires
• Beyond T&E: multiple hurdles
– Objective tests
– Structured interviews
– Simulations
• Reengineering: process and policy
![Page 26: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/26.jpg)
Looking Ahead
• Practice: How can we improve MQ?
– Development
– Content
– Assessment
• Perspective:
– What is realistically possible?
– What can we assume about MQ validity?
• Policy: How should MQ be used?
![Page 27: Qualification Standards: Issues and Implicationsannex.ipacweb.org/library/conf/08/tsugawa.pdfDOL MSPB is one of several Executive Branch agencies responsible for some aspect of the](https://reader033.fdocuments.in/reader033/viewer/2022041811/5e57c1cec246fb04fc4d9b69/html5/thumbnails/27.jpg)
For More Information
Contact: [email protected]
On the Web: www.mspb.gov
- Studies
- Newsletters
- ListServ