Clinical Simulation TestingClinical Simulation TestingA Webinar for Program Directors

March 2008National Board for Certification in Occupational Therapy, Inc.

Seminar FormatSeminar Format

Section 1Section 1Background and rationale for using clinical Background and rationale for using clinical simulation testing (CST) for occupational therapy simulation testing (CST) for occupational therapy credentialingcredentialing

Section 2Section 2Format and flow Format and flow Exam development processExam development process

Section 3Section 3Demonstration of a CST itemDemonstration of a CST item

Q & AQ & A

Section 1:Section 1:Background and RationaleBackground and Rationale

What is CST?What is CST?It is a format of assessment designed to It is a format of assessment designed to simulate the types of situations newly simulate the types of situations newly certified occupational therapy practitioners certified occupational therapy practitioners may encounter in practicemay encounter in practice

Background and RationaleBackground and Rationale

The items are designed to measure a The items are designed to measure a candidate’s knowledge and critical reasoning candidate’s knowledge and critical reasoning ability ability sequentiallysequentially across the continuum of across the continuum of care:care:

ScreeningScreeningFormulating conclusions regarding treatment Formulating conclusions regarding treatment needs and prioritiesneeds and prioritiesImplementing interventionsImplementing interventionsAssessing outcomesAssessing outcomes

Background and RationaleBackground and Rationale

How is CST different from Multiple Choice How is CST different from Multiple Choice Testing (MCT) formats?Testing (MCT) formats?

MCT measures breadth of knowledgeMCT measures breadth of knowledgejudgment is made in isolation and the items are judgment is made in isolation and the items are nonnon--sequential in naturesequential in nature

CST complements MCT formats by examining CST complements MCT formats by examining depth of knowledge sequentially where decisions depth of knowledge sequentially where decisions are tied to previous decisions and resultsare tied to previous decisions and results

Why Now?Why Now?

The move to CST came from the need to The move to CST came from the need to measure critical reasoning skills and was measure critical reasoning skills and was driven by:driven by:

NBCOT NBCOT practice analysis studiespractice analysis studiesmissionmission

Revised ACOTE standardsRevised ACOTE standardsCertification industry trendsCertification industry trends

NBCOTNBCOT®®

Practice Analysis StudiesPractice Analysis StudiesCritical foundation for developing a Critical foundation for developing a psychometrically sound and legally defensible psychometrically sound and legally defensible credentialing examinationcredentialing examinationAccreditation agencies require systematic reAccreditation agencies require systematic re--evaluation of content standards to ensure evaluation of content standards to ensure appropriateness of the examination:appropriateness of the examination:

content validcontent validappropriately evaluates what is required to function appropriately evaluates what is required to function as a competent practitioneras a competent practitioner

NBCOTNBCOT®®

Results from the latest NBCOT practice Results from the latest NBCOT practice analysis study:analysis study:

0101 Evaluate the client on an ongoing basis using 0101 Evaluate the client on an ongoing basis using appropriate tools, procedures, and protocols in order appropriate tools, procedures, and protocols in order to determine factors that impact participation in to determine factors that impact participation in occupationoccupation0303 Use critical reasoning to select and implement 0303 Use critical reasoning to select and implement interventions and approaches consistent with interventions and approaches consistent with psychosocial and cognitive abilities, and client needs psychosocial and cognitive abilities, and client needs in order to facilitate outcomes within areas of in order to facilitate outcomes within areas of occupationoccupation

(NBCOT OTR Practice Analysis Study, 2007)(NBCOT OTR Practice Analysis Study, 2007)

NBCOTNBCOT®® MissionMission

Mission “to serve the public interest:”Mission “to serve the public interest:”Assure key stakeholders that certificants have met Assure key stakeholders that certificants have met a minimum practice standard by differentiating a minimum practice standard by differentiating between competent and incompetent practitionersbetween competent and incompetent practitionersJust as in real life, through the sequential nature of Just as in real life, through the sequential nature of the CST, errors may be compounded on top of the CST, errors may be compounded on top of errors resulting in nonproductive assessment and errors resulting in nonproductive assessment and intervention proceduresintervention procedures

Revised ACOTE StandardsRevised ACOTE Standards

Transition to postTransition to post--baccalaureate degree and baccalaureate degree and revised ACOTE standards:revised ACOTE standards:

“The rapidly changing and dynamic nature of “The rapidly changing and dynamic nature of contemporary health and human services delivery contemporary health and human services delivery systems requires the occupational therapist to systems requires the occupational therapist to possess basic skills as a direct care provider, possess basic skills as a direct care provider, consultant, educator, manager, researcher, and consultant, educator, manager, researcher, and advocate for the profession and the consumer.”advocate for the profession and the consumer.”

Preamble to The Standards 2008Preamble to The Standards 2008

ACOTE StandardsACOTE Standards

B5.26 Organize, collect and analyze data in a B5.26 Organize, collect and analyze data in a systematic manner for evaluation of practice systematic manner for evaluation of practice outcomes. Report evaluation results and outcomes. Report evaluation results and modify practice as needed to improve modify practice as needed to improve outcomes.outcomes.

Certification Industry TrendsCertification Industry Trends

Consistent with certification examination Consistent with certification examination practices of other allied health and medical practices of other allied health and medical certification bodies who use defensible certification bodies who use defensible constructs to assess critical clinical constructs to assess critical clinical competenciescompetencies

i.e. Physician Assistants, Counseling i.e. Physician Assistants, Counseling

Historical BackgroundHistorical Background

Branching logic/simulation testing introduced by the Branching logic/simulation testing introduced by the National Board for Medical Examiners in the 1960s National Board for Medical Examiners in the 1960s as an objective, reliable evaluation of clinical as an objective, reliable evaluation of clinical competence in the medical professioncompetence in the medical profession

(Hubbard et al, 1965)(Hubbard et al, 1965)Seminal studies in the medical professions have Seminal studies in the medical professions have concluded there is a statistically significant concluded there is a statistically significant relationship between results obtained by exam relationship between results obtained by exam candidates and those obtained by experienced candidates and those obtained by experienced practitioners taking CST examinationspractitioners taking CST examinations

(Grace et al, 1977)(Grace et al, 1977)

Historical BackgroundHistorical Background

The 1980s saw the emergence of CST testing The 1980s saw the emergence of CST testing formats in the behavioral sciences formats in the behavioral sciences

(Berven & Scofield, 1980)(Berven & Scofield, 1980)

This led to CST being identified as a way to This led to CST being identified as a way to evaluate competence for high stakes evaluate competence for high stakes certification and licensure purposescertification and licensure purposes

(Forker & McDonald, 1996; (Forker & McDonald, 1996; Bersky, Krawczak & Kumar, 1998)Bersky, Krawczak & Kumar, 1998)

ReferencesReferences

Hubbard, J., Levit E., Schumacher, C., & Schnabel, T. (1965). AHubbard, J., Levit E., Schumacher, C., & Schnabel, T. (1965). An n objective evaluation of clinical competence. objective evaluation of clinical competence. The New England The New England Journal of Medicine.Journal of Medicine. 272 (25): 1321272 (25): 1321--1328.1328.

Grace, M., Hanson, J, Fincham, S., Skakun, E & Taylor, W (1977).Grace, M., Hanson, J, Fincham, S., Skakun, E & Taylor, W (1977). A A scoring technique for computerized patient management scoring technique for computerized patient management problems. problems. Medical EducationMedical Education, 11: 335, 11: 335--340.340.

Berven N. & Scofield, M. (1980). Evaluation of clinical problemBerven N. & Scofield, M. (1980). Evaluation of clinical problem--solving skills through standardized casesolving skills through standardized case--management management simulations. simulations. Journal of Counseling PsychologyJournal of Counseling Psychology, 27 (2):199, 27 (2):199--208.208.

Forker, J., & McDonald, M. (1996). Methodologic trends in the Forker, J., & McDonald, M. (1996). Methodologic trends in the healthcare professions: computer adaptive and computer healthcare professions: computer adaptive and computer simulation testing. simulation testing. Nurse EducationNurse Education. 21 (4): 13. 21 (4): 13--4.4.

Bersky A., Krawczak, J., & Kumar, T. (1998). Computerized cliniBersky A., Krawczak, J., & Kumar, T. (1998). Computerized clinical cal simulation testing. A new look for the NCLEXsimulation testing. A new look for the NCLEX--RN RN examination? examination? Nurse EducationNurse Education. 23(1): 20. 23(1): 20--5.5.

NBCOTNBCOT®®

CST Alpha TestingCST Alpha TestingCompleted last quarter of 2007Completed last quarter of 2007N = 50 N = 50

(Beta test N = 250)(Beta test N = 250)

OTR OTR certificantscertificants across geographic and practice across geographic and practice settingssettings0 0 –– 18 months since initial certification18 months since initial certificationEnd of test surveyEnd of test surveyPerformance statistics Performance statistics

NBCOTNBCOT®® CST Alpha TestingCST Alpha Testing

Preliminary Results:Preliminary Results:Over 80% indicated CST as an appropriate style Over 80% indicated CST as an appropriate style of testing for occupational therapy credentialingof testing for occupational therapy credentialingMajority said the format was a fair assessment of Majority said the format was a fair assessment of their occupational therapy knowledgetheir occupational therapy knowledge

Alpha ResultsAlpha Results

“Really replicates what I’m doing everyday “Really replicates what I’m doing everyday in my practice.”in my practice.”“MCT just tests snippets of knowledge, the “MCT just tests snippets of knowledge, the CST goes into more depth CST goes into more depth –– tested my tested my problem solving ability.”problem solving ability.”“… gave me more information to build up a “… gave me more information to build up a picture about the client.”picture about the client.”

Alpha ResultsAlpha Results

“I preferred having questions related to one “I preferred having questions related to one case rather than jumping around from case rather than jumping around from diagnosis to diagnosis like in the MCT diagnosis to diagnosis like in the MCT format.”format.”“If you’ve done 2 good level II fieldwork “If you’ve done 2 good level II fieldwork placements, you really should have no placements, you really should have no problem answering these CST questions.” problem answering these CST questions.”

Alpha ResultsAlpha Results

“Fieldwork is where you put all your “Fieldwork is where you put all your knowledge into practice by using your knowledge into practice by using your clinical judgment and common sense. clinical judgment and common sense. That’s exactly what you have to do when That’s exactly what you have to do when answering a CST item.”answering a CST item.”

Section 2:Section 2:Format and FlowFormat and Flow

2006Advisory Taskforce

BOD approves projectPresentation to program directors

2007CST Development Team

Alpha TestingPresentation & mailing to program directors

2008Beta testing

Program director webinarsStudy tools

Project Timeframe

Timeframe ContinuedTimeframe Continued

A Standard Setting study will be completed in A Standard Setting study will be completed in May:May:

Determine passing standard for the OTR Determine passing standard for the OTR certification examinationcertification examination

Beginning 2009, the OTR examination will be a Beginning 2009, the OTR examination will be a hybrid test consisting of:hybrid test consisting of:

MCT itemsMCT itemsCST itemsCST items

Exam will continue to be ADA compliantExam will continue to be ADA compliant

Exam Development Exam Development ProcessProcess

Practice Analysis StudyPractice Analysis Study

Completed systematically to ensure the Completed systematically to ensure the examination evaluates what is required examination evaluates what is required to function as a competent practitionerto function as a competent practitioner

Practice Analysis StudyPractice Analysis Study

Results from the practice analysis study are Results from the practice analysis study are used at all stages of examination development:used at all stages of examination development:

Creation of test blueprintCreation of test blueprintItem writingItem writingExamination constructionExamination construction

Latest Practice Analysis StudyLatest Practice Analysis Study

Completed in 2007Completed in 2007Initial development and validationInitial development and validationPilot studyPilot studyFull validation surveyFull validation surveyDevelopment of test blueprintsDevelopment of test blueprints

Implemented January 2009Implemented January 2009

2009 OTR Blueprint Specifications2009 OTR Blueprint Specifications

Domain 1Domain 1Gather information regarding factors that influence Gather information regarding factors that influence occupational performanceoccupational performance

Domain 2Domain 2Formulate conclusions regarding the client’s needs and Formulate conclusions regarding the client’s needs and priorities to develop a clientpriorities to develop a client--centered intervention plancentered intervention plan

Domain 3Domain 3Select and implement evidenceSelect and implement evidence--based interventions to support based interventions to support participation in areas of occupation throughout the continuum participation in areas of occupation throughout the continuum of careof care

Domain 4Domain 4Uphold professional standards and responsibilities to promote Uphold professional standards and responsibilities to promote quality in practicequality in practice

MCT Development ProcessMCT Development Process

Written to 2009 blueprint specificationsWritten to 2009 blueprint specificationsValidatedValidated

Subject matter experts (OTR practitioners and Subject matter experts (OTR practitioners and educators)educators)References References Performance statisticsPerformance statistics

Statistics collected to ensure psychometric Statistics collected to ensure psychometric integrity and validityintegrity and validityReviewed and approved by independent testing Reviewed and approved by independent testing agencyagencyComplies with accreditation standardsComplies with accreditation standards

Format of MCT ItemsFormat of MCT Items

Computer deliveredComputer deliveredTutorialTutorialQuestion stemQuestion stem4 response options4 response options

1 correct answer1 correct answer3 3 distractorsdistractors

Example of an MCT ItemExample of an MCT Item

An inpatient who hasAn inpatient who has Guillain BarreGuillain Barre syndrome is syndrome is referred to OT to maximize independence with ADL. referred to OT to maximize independence with ADL. The patient has proximal weakness in upper and lower The patient has proximal weakness in upper and lower extremities. Which method is extremities. Which method is BESTBEST for the OTR to for the OTR to use when assessing the patient’s initial functional use when assessing the patient’s initial functional status?status?

A.A. Ask the nursing staff about the patient’s abilitiesAsk the nursing staff about the patient’s abilitiesB.B. Have the patient complete a lifestyle inventoryHave the patient complete a lifestyle inventoryC.C. Interview family members during visiting hoursInterview family members during visiting hoursD.D. Observe the patient completing ADL tasksObserve the patient completing ADL tasks

Correct answer is…Correct answer is…

D. Observe the patient completing ADL D. Observe the patient completing ADL tasks.tasks.

OObserving the patient’s performance of ADL bserving the patient’s performance of ADL tasks will provide the most accurate and tasks will provide the most accurate and measurable information about the individual’s measurable information about the individual’s functional level.functional level.

Overview of CST Development ProcessOverview of CST Development Process

Written to 2009 OTR blueprint specificationsWritten to 2009 OTR blueprint specificationsValidatedValidated

Subject matter experts (OTR, practitioners and Subject matter experts (OTR, practitioners and educators)educators)References References Performance statisticsPerformance statistics

Statistics collected to ensure psychometric Statistics collected to ensure psychometric integrity and validityintegrity and validityReviewed and approved by independent testing Reviewed and approved by independent testing agencyagencyComplies with accreditation standardsComplies with accreditation standards

CST Item MakeupCST Item Makeup

TutorialTutorialAn opening sceneAn opening sceneSections with section headersSections with section headersA series of decisions/action statementsA series of decisions/action statements

Opening Scene

Section Heading

Format & FlowFormat & Flow

Candidates should select the decisions/actions Candidates should select the decisions/actions that they deem appropriate from the list of that they deem appropriate from the list of options providedoptions providedAs the decision/action statement is selected, As the decision/action statement is selected, a feedback box will appeara feedback box will appearFeedback contains information regarding the Feedback contains information regarding the consequence of selecting that decision and is consequence of selecting that decision and is used to make future selectionsused to make future selections

Feedback Boxes

Format & FlowFormat & FlowThe decisions/action list consists of:The decisions/action list consists of:

Positive responses Positive responses Optimal routeOptimal routePoints awarded if candidate selects these responsesPoints awarded if candidate selects these responses

Neutral responsesNeutral responsesMay be helpful, but not absolutely necessary in solving May be helpful, but not absolutely necessary in solving the problemthe problemPoints neither awarded nor deductedPoints neither awarded nor deducted

Negative responsesNegative responsesHinder or distract from solving the problemHinder or distract from solving the problemMay be harmful to the clientMay be harmful to the clientPoints deducted if candidate selects these optionsPoints deducted if candidate selects these options

Format & FlowFormat & Flow

Candidate will have the ability to scroll back Candidate will have the ability to scroll back through the CST item to view:through the CST item to view:

Opening sceneOpening sceneSection headersSection headersItems they have selectedItems they have selected

Just as in real life, once the decision/action has Just as in real life, once the decision/action has been checked, the candidate will not be able to been checked, the candidate will not be able to dede--select the responseselect the response

SummarySummary--Each CST Will Have:Each CST Will Have:

Opening Scene

Series of Sections

Each section has a section header

List of decisions/actions+ neutral -

As a decision is made, a feedback box willprovide additional information assisting the candidate

to progress through the item

Section 3Section 3

PrimerPrimerDemonstrationDemonstrationQ&AQ&A

PrimerPrimer

CST item is about an OTR working in a school CST item is about an OTR working in a school systemsystemTeacher refers student for OT due to concerns Teacher refers student for OT due to concerns with classroom performance and fine motor with classroom performance and fine motor skillsskillsStudent has an older sibling with autismStudent has an older sibling with autismCST progresses from screening through to CST progresses from screening through to selection of activities to meet IEP goalsselection of activities to meet IEP goals

Primer ContinuedPrimer Continued

Contains 4 sections:Contains 4 sections:Screening Screening

11 potential decisions/actions listed11 potential decisions/actions listed

Assessment Assessment 9 potential decisions/actions listed9 potential decisions/actions listed

Recommendations to IEP teamRecommendations to IEP team11 potential decisions/actions listed11 potential decisions/actions listed

Activities supporting IEP goalsActivities supporting IEP goals7 potential decisions/actions listed7 potential decisions/actions listed

DemonstrationDemonstration

NBCOT® Inc.12 S. Summit Ave, Suite 100

Gaithersburg, MD 21877-4150www.nbcot.org