National Evaluation of Specialty Selection

Hywel Thomas and Celia TaylorOn behalf of the NESS team: Ian Davison, Steven Field, Harry Gee, Janet Grant, Andy Malins, Laura Pendleton and Elizabeth Spencer

NESS was commissioned and funded by the Policy Research Programme in the Department of Health (Award number 016 0114). The views expressed are not necessarily those of the Department.

Background

• Specialty selection process is one of the hurdles on the way to consultant/GP principal posts

• 2009: 11,417 applicants for 6,580 entry-level posts (competition ratio 1.7 to 1)

• Selection became increasingly politically sensitive following MTAS

• Highlighted need for evolution and evaluation

Aims and scope of NESS

• To evaluate the first round of selection for specialty training in 2009 against four key criteria:– Acceptability– Fairness– Effectiveness: Validity and Reliability– Value for money

• 13 specialties included in the project• Data collection primarily in 5 deaneries• Did not obtain complete data for every specialty/deanery

Data Sources

Data

Sources

Analytical focus

Profile: applicants, candidates &

assessors

Selection processes and methods The Pilots

Organisation Transferability Risks and benefits

Acceptability Reliability Validity Cost-effectiveness

Fairness

Documents

Academic literature

College and specialty documents and websites

Deanery documents and websites

MMC/DH documents and website

Pilot reports

Surveys

Applicants

Candidates

Assessors

Interviews

Candidates

Assessors

Deanery and College representatives

Outcomes

Score data

1

2

3

4

5

6M

ean

scor

e (1

=Str

ongl

y di

sagr

ee; 6

=Str

ongl

y ag

ree)

Candidates (total N=4,027) Assessors (total N=697)

Acceptability: “This selection process was fair”

***

Fairness: Effect of personal characteristics on selection scores

Characteristic Unweighted mean coefficient across specialties

N specialties in which coefficient statistically significant (p<0.05)/5

Male (vs. Female) -0.11 2

Asian (vs. White) -0.31 2

Other Ethnic group (vs. White) -0.35 1

EU trained (vs. UK trained) -0.63 3

Non-EU trained (vs. UK trained) -0.78 3

FY2 post (vs. other post) 0.10 1

Age in years -0.04 4

Postgrad experience in years -0.06 1

R squared 0.29 (range 0.08 to 0.45) N/A

Multiple linear regression analysis by specialtyStandardised scores so comparability across specialtiesN=5 specialties and 1,553 candidates

Effectiveness: Predictive validity of shortlisting scores

Selection Process Uncorrected Corrected

Specialty 1 Deanery F 0.61 0.78

Specialty 2 Deanery A 0.44 0.58

Specialty 5 Deanery B 0.39 0.66

Specialty 5 Deanery E 0.65 0.91

Specialty 6 Deanery A 0.48 0.67

Specialty 7 Deanery A 0.48 N/A

Specialty 7 Deanery E 0.56 N/A

Specialty 9 Deanery A 0.17 N/A

Specialty 9 Deanery B 0.45 0.78

Specialty 9 Deanery E 0.37 0.63

Specialty 10 Deanery F 0.44 0.81

Specialty 13 Deanery F 0.50 0.55

Pearson correlation coefficients: uncorrected and corrected for restriction of range and unreliability of shortlisting scores where possibleN=8 specialties, 13 selection processes and 2,411 candidates

Effectiveness: Reliability • Internal Consistency: Cronbach’s alpha by station

– N=10 specialties, 26 selection processes and 3,505 candidates– Range 0.35 to 0.83– 10/26 (38%) in recommended range 0.7 to 0.9

• Inter-rater reliability: Station-level absolute intra-class correlations– N=4 specialties, 4 selection processes and 395 candidates– Range 0.54 to 0.91– 16/17 (94%) above recommended minimum of 0.7

• Pass-Mark reliability (ignores sub-rules at station-level and only includes candidates attending interview)– N=5 specialties, 7 selection processes and 919 candidates– 12% to 55% of candidates within 1 SEM of appointment cut-off: raises

concerns about fairness– 0% to 20% of candidates within 1 SEM of competency cut-off: raises

concerns about competency

Pass-Mark reliability example

Value for Money• Costing model developed (

http://www.education.bham.ac.uk/research/projects1/dissemination.shtml)• Modified Brogden’s model to estimate cost-benefit:

• Cost-benefit depends on:– Selection process design– Predictive validity– Competition ratio– SD of training performance of candidates– Length of training– Drop-out rate– Number requiring extensions to training– Proportion unsuccessful candidates remaining in NHS

• Cost estimates for ST1 selection: £3.2m for hospital specialties (£800 per post) and £2.4m for GP (£900 per post)

• Cost-benefit estimates - compared to random selection - ranged from £78-97m for hospital specialties and £15-20m for GP

selselranSt

Et

UMxyxy

T

t

St

T

t

SMxyxy

St CEENCZSDRNAZSDRNCB

)](.[]..).(543.0[...11

Summary and implications for selection

• Largest study of specialty selection• Did not obtain complete data – but no evidence of response bias• High acceptability of selection processes by candidates and assessors• Shortlist scores are a good predictor of selection scores• Long-term follow-up is required on predictive validity, particularly to assess

fairness (if scores are predictive then UK-trained candidates will make better trainees but need evidence)

• Inter-rater reliability was good – but potential collusion?• Internal consistency and so pass-mark reliability could be improved: more

stations with 1 assessor?• Only one specialty had a formal standard setting process to identify

competency cut-off• Value for money could only be estimated – but suggests high returns to

investment in selection• Selection has continued to evolve since 2009 e.g. increase in nationally-

coordinated selection processes