Post on 23-Dec-2015
Measuring Post-Measuring Post-Licensure CompetenceLicensure Competence
The Nursing Performance ProfileThe Nursing Performance Profile
Research TeamResearch Team
Janine Hinton RN, Ph.DJanine Hinton RN, Ph.D Mary Mays Ph.DMary Mays Ph.D Debra Hagler RN, Ph.DDebra Hagler RN, Ph.D Pamela Randolph RN, MSPamela Randolph RN, MS Beatrice Kastenbaum RN, MSN, Beatrice Kastenbaum RN, MSN,
CNECNE Ruth Brooks RN, MS, BCRuth Brooks RN, MS, BC Nick DeFalco RN, MSNick DeFalco RN, MS Kathy Miller RN, MSKathy Miller RN, MS Dan Weberg RN, MHIDan Weberg RN, MHI
SupportSupport
Funded by NCSBN CRE GrantFunded by NCSBN CRE Grant Supported by:Supported by:
Scottsdale Community CollegeScottsdale Community College Arizona State UniversityArizona State University Arizona State Board of NursingArizona State Board of Nursing
Statement of the ProblemStatement of the Problem
A valid reliable practice A valid reliable practice assessment is needed to assessment is needed to support intervention on the support intervention on the public’s behalf when the pattern public’s behalf when the pattern of nursing performance results of nursing performance results in or is likely to result in patient in or is likely to result in patient harmharm
Literature ReviewLiterature Review
Medical errors a leading cause Medical errors a leading cause of death of death (IOM, 2000)(IOM, 2000)
Written tests do not directly Written tests do not directly measure performance measure performance (Auerwarakul, (Auerwarakul,
Downing, Jaruratamrong, & Praditsuwan, 2005)Downing, Jaruratamrong, & Praditsuwan, 2005)
Multiple observations of a Multiple observations of a nurse’s performance have nurse’s performance have provided evidence of competent provided evidence of competent practice practice (Williams, Klaman,& McGaghie, 2003)(Williams, Klaman,& McGaghie, 2003)
Literature ReviewLiterature Review
High-fidelity simulation technology High-fidelity simulation technology allows the creation of reproducible allows the creation of reproducible scenarios to evaluate nursing scenarios to evaluate nursing performance performance (Boulet et.al., 2011; Kardong-Edgren, Adamson, & (Boulet et.al., 2011; Kardong-Edgren, Adamson, &
Fitzgerald, 2010)Fitzgerald, 2010)
Nursing and Health care leaders Nursing and Health care leaders have called for performance have called for performance assessments to evaluate assessments to evaluate competence and support competence and support remediation remediation (Benner, Stupen, Leonard, & Day, 2010; IOM, (Benner, Stupen, Leonard, & Day, 2010; IOM,
2011)2011)
Purpose of studyPurpose of study
To develop and evaluate a high-To develop and evaluate a high-stakes simulation testing stakes simulation testing process to measure minimally process to measure minimally safe nursing practice safe nursing practice competence and identify competence and identify remediation needs. remediation needs.
MethodologyMethodology
Needed process to apply Needed process to apply sophisticated measures of validity sophisticated measures of validity and reliabilityand reliability Participants appeared in 3 simulation Participants appeared in 3 simulation
videosvideos 3 subject matter expert rated each video 3 subject matter expert rated each video
on 41 measures of competencyon 41 measures of competency Raters blind to participant ability, Raters blind to participant ability,
experience, order of testingexperience, order of testing Videos presented a range of safe and Videos presented a range of safe and
unsafe performanceunsafe performance Obtained ratio level data suitable for Obtained ratio level data suitable for
parametric, inferential statistical analysisparametric, inferential statistical analysis
Filming Participant Filming Participant DemographicsDemographics Criteria—newly licensed RNs less Criteria—newly licensed RNs less
than 3 years nursing experience than 3 years nursing experience (N=21)(N=21)
Average age=32Average age=32 95% female95% female 58% white, 16% black, 26% hispanic58% white, 16% black, 26% hispanic 79% AD; 21% BSN79% AD; 21% BSN Less than 3 years experience—Less than 3 years experience—
mean experience=1.05 yearsmean experience=1.05 years Majority had some experience with Majority had some experience with
simulation 74% simulation 74%
Rater DemographicsRater Demographics
Criteria--BSN and 3 years Criteria--BSN and 3 years experience and work in a role experience and work in a role that involves evaluating others that involves evaluating others (N=4)(N=4)
Average experience=12.5Average experience=12.5 Age 31-51Age 31-51 White, femaleWhite, female Education: 3 BSN, 1 MSEducation: 3 BSN, 1 MS
Instrument DevelopmentInstrument Development
Developed and established initial Developed and established initial validity/reliability before fundingvalidity/reliability before funding
TERCAP served as the theoretical TERCAP served as the theoretical framework framework (Benner et.al.,2006; Woods & Doan-Johnson, 2003)(Benner et.al.,2006; Woods & Doan-Johnson, 2003)
Survey items on NCSBN’s Clinical Survey items on NCSBN’s Clinical Competency Assessment of Newly Competency Assessment of Newly Licensed Nurses were adapted Licensed Nurses were adapted (NCSBN, (NCSBN,
2007)2007)
Mapped to QSEN competenciesMapped to QSEN competencies
Categories of Items Categories of Items (TERCAP)(TERCAP) Professional ResponsibilityProfessional Responsibility Client AdvocacyClient Advocacy AttentivenessAttentiveness Clinical Reasoning, noticingClinical Reasoning, noticing Clinical Reasoning, understandingClinical Reasoning, understanding CommunicationCommunication PreventionPrevention Procedural CompetencyProcedural Competency DocumentationDocumentation
Example of One Item Example of One Item Category CompetenciesCategory Competencies
PreventionPrevention Infection control Infection control 2 client identifiers2 client identifiers Appropriate positioningAppropriate positioning Safe environmentSafe environment
Scoring—4 possibilitiesScoring—4 possibilities
Performance or action is consistent Performance or action is consistent with standards of practice and free with standards of practice and free from actions that may place the from actions that may place the client at risk for harmclient at risk for harm
Fails to perform or performs in a Fails to perform or performs in a manner that exposes the client to manner that exposes the client to risk for harmrisk for harm
No opportunity to observe in the No opportunity to observe in the scenarioscenario
BlankBlank
Scoring testScoring test
No weighted itemsNo weighted items No pass fail standardNo pass fail standard Description of Nurse’s Description of Nurse’s
performance across 9 performance across 9 categories of competencycategories of competency
Final rating of each item based Final rating of each item based on inter-rater agreement—at on inter-rater agreement—at least 2 of 3 agreeleast 2 of 3 agree
ScenariosScenarios
3 sets of 3 scenarios scripted=93 sets of 3 scenarios scripted=9 Adult acute care, common Adult acute care, common
diagnosesdiagnoses Each scenario had opportunities Each scenario had opportunities
to observe all performance to observe all performance itemsitems
Each sim patient had hospital-Each sim patient had hospital-like chart with information—labs, like chart with information—labs, history, MAR, ordershistory, MAR, orders
Simulation Testing/RatingSimulation Testing/Rating
21 nurse performers= and 63 videos21 nurse performers= and 63 videos Scenario Set 1=5 participantsScenario Set 1=5 participants Scenario Set 2=8 participantsScenario Set 2=8 participants Scenario Set 3=8 participantsScenario Set 3=8 participants
Each video evaluated by 3 ratersEach video evaluated by 3 raters 189 rating instruments189 rating instruments 41 items rated on each instrument41 items rated on each instrument 7,749 ratings7,749 ratings
Analysis ProceduresAnalysis Procedures
Predictive Analysis Software Predictive Analysis Software (v 18.0.3 SPSS (v 18.0.3 SPSS
Inc., Chicago, IL)Inc., Chicago, IL)
Frequency analysis to identify Frequency analysis to identify instrument properties:instrument properties: Used as intended Used as intended Interrater reliabilityInterrater reliability Sensitive to common practice errors Sensitive to common practice errors
(construct validity)(construct validity) Cronbach’s alpha (intercorrelation Cronbach’s alpha (intercorrelation
among items) was used to measure among items) was used to measure internal consistencyinternal consistency
Analysis Procedures ContAnalysis Procedures Cont
ANOVA used toANOVA used to Assess ability of instrument to Assess ability of instrument to
distinguish between experienced distinguish between experienced and inexperienced nursesand inexperienced nurses
Assess potential bias created by Assess potential bias created by administration methodsadministration methods
ResultsResults
Less than 1% of items left blank Less than 1% of items left blank or not observed—or not observed—indicates indicates scenarios comprehensivescenarios comprehensive
Interrater reliability—across all Interrater reliability—across all 41 items at least 2 raters agreed 41 items at least 2 raters agreed on 99.12%on 99.12%
Internal consistency Cronbach’s Internal consistency Cronbach’s alpha=0.91-0.84 on 41 items alpha=0.91-0.84 on 41 items combined and separatecombined and separate
ResultsResults
Construct validity—pass rates Construct validity—pass rates should mirror those in other should mirror those in other studiesstudies Infection control—pass rate 57% Infection control—pass rate 57%
mainly due to lack of hand mainly due to lack of hand hygienehygiene
Documentation—pass rate 29%--Documentation—pass rate 29%--area of frequent concern in area of frequent concern in practicepractice
ResultsResults
Criterion validityCriterion validity 2 groups by nursing experience2 groups by nursing experience
<1 year or<1 year or 1-3 years1-3 years
2 way mixed ANOVA2 way mixed ANOVA Experienced nurses made fewer errors than Experienced nurses made fewer errors than
new nurses (p<0.001)new nurses (p<0.001) Significant in 6 of 9 categoriesSignificant in 6 of 9 categories
AttentivenessAttentiveness Clinical Reasoning (noticing)Clinical Reasoning (noticing) Clinical Reasoning (understanding)Clinical Reasoning (understanding) CommunicationCommunication Procedural CompetencyProcedural Competency DocumentationDocumentation
CategoryInexperienced
NursesExperienced
Nurses
M (S) M (S) p value
Professional Responsibility
-.33 (.58) -.22 (.49)
*
Client Advocacy -.57 (.81) -.25 (.50) *
Attentiveness -.76 (1.00) -.17 (.38) 0.002
Clinical Reasoning - Noticing
-1.19 (1.33) -.47 (.81)
0.01
Clinical Reasoning - Understanding
-1.67 (1.28) -.81 (.95)
0.005
Communication -1.48 (1.44) -.75 (.97) 0.03
Prevention -1.57 (1.50) -1.31 (.82) *
Procedural Competency
-2.76 (2.32) -1.19 (1.37)
0.002
Documentation -3.33 (.73) -2.61 (1.20) 0.02
Comparison of Groups by Comparison of Groups by CategoryCategory
NPP ResultsNPP Results
Inexperienced 0.5 year
1 year experience
Inexperienced 0.5 yr
2 year experience
ResultsResults
Test BiasTest Bias Scenario was not significantScenario was not significant Categories was significant—some Categories was significant—some
competency categories more competency categories more difficultdifficult Communication, prevention, Communication, prevention,
procedural competency and procedural competency and documentation more difficultdocumentation more difficult
ResultsResults
Test Bias continuedTest Bias continued Scenario set—significant only for Scenario set—significant only for
documentation which may be documentation which may be easier on Set 1easier on Set 1
Order of testing and practice effect Order of testing and practice effect not significantnot significant
Location of testing not significantLocation of testing not significant
SummarySummary
Instrument has adequate validity and Instrument has adequate validity and reliabilityreliability
Raters used instrument as instructed Raters used instrument as instructed and in a reproducible mannerand in a reproducible manner
Items were highly interrelatedItems were highly interrelated Sensitive to common errorsSensitive to common errors Inexperienced nurses made more Inexperienced nurses made more
errorserrors Test not biasedTest not biased Plots permit users to visualize Plots permit users to visualize
performanceperformance
ImplicationsImplications
Provides a valid explicit measure of Provides a valid explicit measure of performance that regulatory Boards performance that regulatory Boards could use along with other data to could use along with other data to determine if practice errors are a determine if practice errors are a one-time occurrence or a pattern of one-time occurrence or a pattern of high risk behaviorhigh risk behavior
Potential uses in education and Potential uses in education and practice to assess performance and practice to assess performance and effect of educational interventioneffect of educational intervention
LimitationsLimitations
Volunteer subjects—not random or Volunteer subjects—not random or representativerepresentative
Sample size too small to support Sample size too small to support confirmatory factor analysis of the confirmatory factor analysis of the instruments construct validityinstruments construct validity
Tailored to specific context and Tailored to specific context and purposepurpose
Limitations of simulation—non-verbal Limitations of simulation—non-verbal and skin change cues missing—and skin change cues missing—suspend disbelief suspend disbelief
Future ResearchFuture Research
Funded by NCSBN for Phase IIFunded by NCSBN for Phase II Criterion Validity by comparing RN Criterion Validity by comparing RN
self and supervisor ratingsself and supervisor ratings Compare to education, Compare to education,
certification certification Broader cross section of Broader cross section of
experienced nurses recruitedexperienced nurses recruited
ReferencesReferences
Auewarakul, C., Downing, S. M., Jaturatamrong, U, and Auewarakul, C., Downing, S. M., Jaturatamrong, U, and Praditsuwan, R. (2005). Sources of validity evidence for Praditsuwan, R. (2005). Sources of validity evidence for an internal medicine student evaluation system: An an internal medicine student evaluation system: An evaluative study of assessment methods. evaluative study of assessment methods. Medical Medical Education, 39Education, 39, 276-283., 276-283.
Benner, P., Sutphen, M., Leonard, V., Day, L. (2010). Benner, P., Sutphen, M., Leonard, V., Day, L. (2010). Educating Nurses: A Call for Radical TransformationEducating Nurses: A Call for Radical Transformation. . Stanford, CA: Jossey-Bass.Stanford, CA: Jossey-Bass.
Boulet, J. R., Jeffries, P. R., Hatala, R. A., Korndorffer, J. Boulet, J. R., Jeffries, P. R., Hatala, R. A., Korndorffer, J. R., Feinstein, D. M., & Roche, J. P. (2011). Research R., Feinstein, D. M., & Roche, J. P. (2011). Research regarding methods of assessing learning outcomes. regarding methods of assessing learning outcomes. Simulation in Healthcare, 6Simulation in Healthcare, 6(7), supplement, 48-51.(7), supplement, 48-51.
Institute of Medicine (IOM) (2011). Institute of Medicine (IOM) (2011). The Future of The Future of Nursing: Leading Change, AdvancingNursing: Leading Change, Advancing Health. Health. Washington, DC: National Academies Press. Washington, DC: National Academies Press.
ReferencesReferences
Institute of Medicine. (2000). Institute of Medicine. (2000). To err is human: Building a To err is human: Building a safer system.safer system. Washington, DC: National Academies Washington, DC: National Academies PressPress
Kardong-Edgren, S., Adamson, K. A., & Fitzgerald, C. Kardong-Edgren, S., Adamson, K. A., & Fitzgerald, C. (2010). A review of currently published evaluation (2010). A review of currently published evaluation instruments for human patient simulation. instruments for human patient simulation. Clinical Clinical Simulation in Nursing, 6Simulation in Nursing, 6(1). (1). doi:10.1016/j.ecns.2009.08.004doi:10.1016/j.ecns.2009.08.004
Williams, R. G., Klamen D. A., & McGaghie, W. C. Williams, R. G., Klamen D. A., & McGaghie, W. C. (2003). Cognitive, social and environmental sources (2003). Cognitive, social and environmental sources of bias in clinical performance ratings. of bias in clinical performance ratings. Teaching andTeaching and Learning in Medicine, 15Learning in Medicine, 15(4), 270-292.(4), 270-292.