Anurag Saxena

Postgraduate Medical Education

University of Saskatchewan, School of Medicine

Next speaker:COMING UP NEXT:

Non-academic behaviors and assessment of personal

characteristics at admissions

FUN FACTS ABOUT OUR NEXT SPEAKER:

● Love to travel

● Old cars

● Cooking all sorts of cuisine

Non-academic behaviors and assessment of personal

characteristics at admissions

Anurag SaxenaProfessor and Associate Dean, PGME

College of Medicine, University of Saskatchewan

Objectives

Reflect

Reflect on the somewhat overlapping construct of cognitive and non-cognitive attributes (personal characteristics) in the context of variables assessed at admissions.

Distinguish

Distinguish the importance of academic from clinical performance measures in predictive and incremental validity studies

Appreciate

Appreciate the need (and locally develop strategies) to balance and (integrate) multiple stakeholder perspectives in selections

Both CAs and NCAs are important for clinical practice

One review is: Patterson et al 2016)

• CAs: (intelligence , learned knowledge)• NCAs: (mindsets, attitudes, personality traits,

learning strategies, socio-emotional skills)• Thought plays a role in both (Borghans et al

2008)• Each influences the development of the other

(Cunha et al 2010)• Malleable – can be variably enhanced

CAs and NCAs are interlinked

• Not mutually exclusive (Eva & Reiter2004)

• (Cleland et al 2012; Paterson et al 2016)

Job analysis-based criteria

Relative roles might be different

Linking selections to outcomes

Variables assessed Cognitive vs. non-cognitive (personal characteristics)Stable vs. malleable

Moderating effect of education and competence development

Outcomes Academic vs. clinical performance

VARIABLES

Achievement & Aptitude

Cognitive & non-cognitive

(personal characteristics)

Stable vs. malleable

RELIABILITY

G theory

Test-retestInter-rater

Internal consistency

VALIDITY

Face

Content

Construct

Criterion: Concurrent, Predictive

Incremental

Political

Linking assessments at admissions to outcomesPurpose: selection / de-selection• More likely to

succeed• Have desired

qualities to learn and practice

Patterson et al 2016; Cleland et al 2012; Koczwara et al 2012; Powis 2020;

Outcome measuresAcademic Performance ≠ Clinical Performance● Dissecting components of overall performance

● Academic performance

● Clinical performance

● Examination performance○ Components of exam performance

● In-training Instructor ratings

● Residents in difficulty aspects○ Professionalism aspects

● Patient satisfaction

● Clinical care outcomes

Linking selections to outcomes is somewhat problematic● Multiple variables assessed at admissions

○ Weighting (compensatory or non-compensatory)

● Paucity of quality and longitudinal studies linking personal characteristics with outcomes○ Especially with clinical performance outcome measures

● Incremental validity studies are rare○ No or negative associations of NCAs with cognitive performance would be convincing!

● Moderating effect of medical education itself

Personal characteristics and outcomes● Predictive validity

○ Mostly favorable■ Lievens 2013: NCAs (by SJTs) predict interpersonal aspects of doctor performance■ Adam et al 2015: NCAs predict clinical performance and professionalism lapses■ Park et al 2016: NCAs predict overall competency outcome in a specialty program■ Patterson & Rowett et al 2016: NCAs ( by SJTs and MMI) predict all end of training

assessments in General practice learners■ Dore et al 2017: CASPer correlation with personal professional components of

MCCQE I and II exams 3-6 years later○ But not by all

■ Mackenzie et al 2017: low effect size and inconsistent overall findings (had measured overall academic achievement as the outcome and exit SJT scores)

SJTs and MMI have been shown to be best for assessing pers chr; online is validated with good Reliability

Our findingsAccepted for AAMC 2021 and RIME supplement of Acad MedSaxena, Desanghere, Dore, Reiter

●Casper introduction ○ 30% reduction in professionalism lapses

■ Less severe issues – using one framework○ Interventions required – less severe: remediation / probation reduced

○ Learner acceptance of interventions improved – no appeals

○ Learning interventions■ Cost savings ■ Time savings

○ No severe adverse outcomes ( suspensions, dismissals) in CASPer cohort (4 years data)

Stakeholder perspectives – may not always be aligned

Education quality, efficiency &

effectiveness, Differentiation;

Reputation

Fair, transparent process; some degree of control to develop what is required

Representation in medicine;

Widening access

Meeting evolving healthcare needs; Quality care for all

Society Society

InstitutionApplicant

Conclusion: Aligning stakeholder interests

Societal representation

Applicants InstitutionsSocietal

healthcare needs

Widening accessGender, ethnicity, SES

Pre-entry activities

Good clinical performance measures; validity (predictive, incremental, political)

Need multi-stakeholder input (patients & society) in blueprinting exercises;

Selection processes: fair, decrease /eliminate disadvantages; Support during learning; studies to determine pathways later on

Thank you!

Questions?

anurag.saxena@usask.ca