Topics Today
description
Transcript of Topics Today
![Page 1: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/1.jpg)
AT-SAT:A Test Battery for Screening
Air Traffic Controller Applicants
Gordon WaughLaurie Wise
Human Resources Research Organization (HumRRO)
![Page 2: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/2.jpg)
Topics Today
Background of AT-SAT Overview of AT-SAT tests Air Traffic Scenarios Test Score weighting methods Setting the cut score
![Page 3: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/3.jpg)
Background of AT-SAT
![Page 4: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/4.jpg)
ATCS Selection backgroundATCS Selection background
Previous Selection System Written aptitude test battery Academy screening (9 weeks) On-the-job training
Issues Test utility after 12-15 years use Academy screening cost & fairness
![Page 5: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/5.jpg)
Air Traffic Selection and Training Air Traffic Selection and Training (AT-SAT) Validation Research(AT-SAT) Validation Research
Integrated validation study Job-performance criterion measures 8 new computerized predictor tests (6 hrs) Concurrent validation study on en route controllers
![Page 6: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/6.jpg)
Overview of AT-SAT Tests
![Page 7: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/7.jpg)
Dial ReadingLetter FactoryAT ScenarioAnglesEQApplied MathAnalogiesScan
![Page 8: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/8.jpg)
Dial Reading Test
Temp.
1) 22 2) 24 3) 28 4) 30 5) 45
![Page 9: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/9.jpg)
Order Boxes
Belt A Belt B Belt C Belt D
Quality Control
Conveyor Belt
Order Boxes AreaLoading AreaQuality Control Area
Box Storage Area
Available Zone
Letter Factory Test
![Page 10: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/10.jpg)
Order Boxes
Belt A Belt B Belt C Belt D
Quality Control
A
CD
P
B
![Page 11: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/11.jpg)
Which boxes should be in the loading area in order to correctly place all the letters on the belts?
1. One purple box and one orange box2. One green box, one purple box, and one orange box3. Two purple boxes, one green box, and one orange box4. Two green boxes, one orange box, and one purple box
Letter Factory Test - Situational Awareness Questions
![Page 12: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/12.jpg)
Air Traffic Scenarios Test
![Page 13: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/13.jpg)
Angles Test
This test includes two different types of questions:
The first presents a picture of an angle and asks you the measure of that angle in degrees (From 1 to 360)
What is the measure of this angle?
2) 10°1) 90° 3) 125° 4) 190°
![Page 14: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/14.jpg)
This test includes two different types of questions:
The second provides you with the measure of an angle (from 1 to 360 degrees). It then asks you to select an angle that represents that measure.
Which of the following represents a 10 degree angle?
1)
2)
3)
4)
![Page 15: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/15.jpg)
Experience Questionnaire
1- Definitely True2- Somewhat True3- Neither True nor False4- Somewhat False5- Definitely False
Your emotions have sometimes prevented you from solving a difficult problem.
![Page 16: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/16.jpg)
Applied MathApplied Math
The distance from point A to point B is 560 miles. If the aircraft left point A at 8:00 “Zulu”, and flew at 400 kts, what time would the aircraft cross point B?
A. 8:56 B. 9:02
C. 9:24 D. 10:02
![Page 17: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/17.jpg)
Analogies Test
Water: Liquid Ice: ?
Gas Cube Solid Oxygen Freeze (1) (2) (3) (4) (5)
![Page 18: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/18.jpg)
Visual Analogies:
: :
(1) (2) (3) (4) (5)
![Page 19: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/19.jpg)
V41610
Type the identification numbers contained in the data block with lower line numbers falling beyond the range (360-710):
B12810
P45580
T65120
Y85710
K23250
F75560
C20390
Scan Test
![Page 20: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/20.jpg)
Air Traffic Scenarios Test
![Page 21: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/21.jpg)
Simulation Overview
Description of simulation Design considerations Scoring Instructions and practice
![Page 22: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/22.jpg)
ATST: Description of Simulation
![Page 23: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/23.jpg)
• Display of airspace sector
• Control several planes
• Land at two airports
• Exit at four locations
AT Scenarios in a Nutshell
![Page 24: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/24.jpg)
F2eF2e
F4DF4D
S1S1AA
F3fF3f
F4B
M4C
![Page 25: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/25.jpg)
Plane Icon
M2BExit/Airport B
Altitude Level 2
Speed Medium
Heading
![Page 26: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/26.jpg)
Design Elements Environment: airspace sector with exits, airports,
planes, and controls Events: planes appear, move, and disappear Actions: user clicks controls to control planes. Rules: eight rules related to speed, altitude,
separation, etc.
![Page 27: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/27.jpg)
• Display of air sector:
• four exits
• two airports
• planes
• Plane controls
• Display of time remaining
Environment
![Page 28: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/28.jpg)
• Direction – 8 headings
• Speed - 3 levels
• Altitude - 4 levels
• Accept handoff
Actions: Plane Controls
![Page 29: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/29.jpg)
• New planes appear
• Planes move
• Planes disappear:
• land, exit, or crash
Events
![Page 30: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/30.jpg)
• Land at slowest speed• Land at lowest altitude• Exit at highest altitude• Land/exit at right place• Land in correct direction• Don’t fly over airports• Keep planes separated• Don’t crash!
Eight Simple Rules
![Page 31: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/31.jpg)
ATST: Design
Considerations
![Page 32: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/32.jpg)
• Measure KSAs
• Simulate wide range of job tasks
• Simulate a key job activity
Possible Approaches
![Page 33: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/33.jpg)
• Maximize score variance
• No disadvantage for computer novices
• Short testing time
Design Goals
![Page 34: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/34.jpg)
• Number of planes on the screen at one time
• Initial speed, level, heading, & location
• N of actions per plane
Difficulty Factors: Planes
![Page 35: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/35.jpg)
• Speed of plane movement• N and complexity of rules• N of exits/airports• N of controls• N of control levels• N of things to remember
Difficulty Factors: Other
![Page 36: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/36.jpg)
ATST: Scoring
![Page 37: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/37.jpg)
• Rule violations
• Separation errors
• Crashes
• Elapsed Time
Record the Outcomes
![Page 38: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/38.jpg)
• N of crashes & separation errors
• N of procedural errors
• Percent of successful flights
• Total delay time (handoff & en route)
Á Priori Rational Scales
![Page 39: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/39.jpg)
• Moved “flyovers” based on data• Standardize scores before
combining and weighting• Rescale scores if useful:
• reverse• fix skewness• make scores more sensitive to
differences at higher levels
Adjust Scales and Scores
![Page 40: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/40.jpg)
ATST:Instructions
&Practice
![Page 41: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/41.jpg)
• Will computer experience affect scores?
• If so, can you minimize its effect?
• If computer skill is job relevant, it’s not as serious.
Computer Skill Effects
![Page 42: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/42.jpg)
• Reduces effects of:• learning ability• computer game skills
Simulation Practice
Instructions with examples
Practice a few things in mini-trial
Give feedback and tips
Several mini-trials
![Page 43: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/43.jpg)
• Add practice trials until more trials add little to:
• reliability
• validity
Post-Instruction Practice
![Page 44: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/44.jpg)
• Pilot test with lots of trials
• Compute trial-total r
• Use last one or more trials with high rs
• Trial-criterion rs are even better if available
Which Trials to Score
![Page 45: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/45.jpg)
Score Weighting Methods
![Page 46: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/46.jpg)
Final AT-SAT Validity
Validity, corrected for shrinkage, range restriction, and unreliability in criterion:
.76 with composite criterion
.78 with CBPM
.38 with Ratings
N
1029
1032
1053
![Page 47: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/47.jpg)
Criteria
Reliability .80 - CBPM (test-retest) .71 - Ratings (interrater reliability) .76 - Composite (mean of CBPM & Ratings)
Composite = .6 * CBPM + .4 * Ratings r CBPM J Ratings = .24
![Page 48: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/48.jpg)
Uncorrected Test Validities
.41 Applied Math .38 Analogies .33 Angles .33 Letter Factory .32 Air Traffic Scenarios .27 Time Wall .25 Planes
![Page 49: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/49.jpg)
Uncorrected Test Validities (cont’d)
.24 Dials .23 Memory Retest .19 Memory .19 Scan .17 Experiences Questionnaire .14 Sound
![Page 50: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/50.jpg)
Goals
Find best set of tests for the final battery
Determine scale weights
Measure validity: How well test battery predicts job performance
![Page 51: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/51.jpg)
Decisions to be Made
Which predictors to keep
How to weight predictors
![Page 52: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/52.jpg)
Decision 1: Choosing Predictors
Phase I- Evaluate predictors according to: simple validity incremental validity fairness group differences test administration time
Predictors chosen by group consensus
![Page 53: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/53.jpg)
Decision 1: Choosing Predictors
Phase 2: Optimal weighting algorithm iterated regression negative weights set to zero maximizes R2 while minimizing differences in group means, slopes, and
intercepts
Many runs done while varying the relative importance of R2 and group differences (10 parameters)
![Page 54: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/54.jpg)
Decision 2: How to Weight Predictors
Alternatives Considered regression weights validity weights optimal weights unit weights
Chosen Weighting Method mean of validity and optimal weights
![Page 55: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/55.jpg)
Conclusions: Optimal Weights
An optimal weighting algorithm can balance the many considerations in selecting and weighting predictors
Optimal weighting tends to exclude too many predictors
Optimal weighting used in conjunction with another weighting method can perform very well
![Page 56: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/56.jpg)
Conclusions: Validity Weights
In the current study, validity weights performed very well, especially when combined with optimal weights
Research is needed to compare the shrinkage (due to overfitting and predictor selection) and R2 of validity weights to other weighting methods
![Page 57: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/57.jpg)
Estimating Shrinkage in R
Shrinkage formula has shortcomings Shrinkage formula corrects for overfitting but not predictor
selection Validity weighting - only selection shrinkage Optimal weighting - shrinkage is severe because of extra
parameters
![Page 58: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/58.jpg)
Conclusions: Shrinkage
Research is needed to help estimate shrinkage of R (due to overfitting and predictor selection) using validity weights and optimal weights.
Research needed to compare validity weights, optimal weights, and regression weights in terms of shrunken R2 (due to overfitting and predictor selection)
![Page 59: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/59.jpg)
Setting the Cut Score
![Page 60: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/60.jpg)
Regression Method
Set cut score on criterion Compute corresponding score on predictor based
on regression line Works well when R-squared is high and good
criterion is available
![Page 61: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/61.jpg)
Criterion Cut Point Options
Cut score set at 5% (for example) of current incumbents’ distribution
Set cut score such that mean expected performance of candidates passing is at 60th percentile (for example) of current incumbents
Set score at anchor on ratings scale (e.g., “acceptable.”
![Page 62: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/62.jpg)
Cut Point Considerations
% of applicants passing % of incumbents passing % of each minority passing adverse impact ratio If low pass rate:
need many applicants, or recruiting must target high-quality applicants
![Page 63: Topics Today](https://reader031.fdocuments.in/reader031/viewer/2022013101/568143ab550346895db0340a/html5/thumbnails/63.jpg)
Questions?