From Learning Objectives to Outcomes

Marie Gilbert MA, RN

Introduction

Marie Gilbert MA, RN

Simulation Coordinator Fresno State University, California

Discuss core components to a healthcare

simulation.

Discuss the importance of outcomes evaluation

and challenges to traditional assessments.

Discuss the importance of validity, reliability

and feasibility as it relates to assessment.

Discuss types of assessments and their

application in healthcare education.

Learning Objectives

Simulation

Simulation is the imitation or representation of one act or system by another.

Healthcare simulations can be said to have four main purposes;

• education

• assessment

• research

• health system integration in facilitating patient safety

Society for Simulation in Healthcare

The simulation setting with its different, connected parts.

(Adapted from Dieckmann, 2009)

Core Components of Healthcare Simulation

Prebrief Simulation Debrief

Core Components of Healthcare Simulation

PREBRIEF

Core Components of Healthcare Simulation

Simulation

Core Components of Healthcare Simulation

Debrief

Core Components of Healthcare Simulation

Objective-Based Scenario Design

Learner

Objectives and

Outcomes

Assessment

Environment Patient

Flow

Debrief

Healthcare professionals;

•Have complex and multidimensional roles •Work independently •Work in teams •Require specific cognitive, technical and behavioral skills

Learner Population

Healthcare professionals;

Domains of measurable skills range from history taking, physical examination and patient management to areas such as teamwork, cultural competence and professionalism. (Boulet et al., 2011)

Learner Population

Importance of Objectives and Outcomes

Direction for the designer, facilitator and the learner

• They identify what the learner will be able to do at the end of the simulation/course/program

• They identify the knowledge, skills and attitudes needed to be able to do this

• They guide the designer and/or facilitator

Outcomes & Objective

A Learning Outcome relates to the final product or end result

What the student can DO

A Learning Objective relates to the process and content

“Nuts & bolts”

Learning Objectives should map to Learning Outcomes

• Are Essential

• Outcomes express higher level thinking skills that integrate the content and activities

• Can be observed as a behavior, skill, or discrete useable knowledge upon completing the simulation/course/program

• Primary sources of outcomes are core competencies designated by professional bodies, academic or clinical institutes

Outcomes for Simulation Based Learning

Example:

At the end of the nursing program the learner will provide patient-centered care based on a comprehensive and focused health assessment.

Outcomes for Simulation Based Learning

• Are Essential

• Must reflect the intended outcome of the experience

• Specify expected learner behavior

• Include sufficient detail to allow learners to participate in the simulation effectively

Objectives for Simulation Based Learning

Examples

The learner will identify the most relevant data

The learner demonstrates effective communication

The learner recognizes subtle changes in the patients condition

Outcomes for Simulation Based Learning

• At the end of the nursing program the learner will provide patient-centered care based on a comprehensive and focused health assessment.

Outcome

• The learner will identify the most relevant data

• The learner demonstrates effective communication

• The learner recognizes subtle changes in the patients condition

Objective

Objective-based scenario design

http://be-know-do.com/introduction-to-smart-objectives-and-smart-goals/

Guidance on writing Outcomes & Objectives

How do we know we‟ve achieved our learning objectives and learning outcomes?

ASSESSMENT

Objective-based scenario design

• Higher stake

• Used for performance evaluation

• Occurs upon completion of the work and the focus is on the final product

• Certification and licensure

Summative

• Lower stake

• Used to modify teaching and learning

• Occur over time of course or program

• Training/education activities

Formative

Types of Assessment

Practice Review

Audits, Video

Performance Tests

Task demonstration, Simulation, OSCE

Clinical Based Tests

Patient management questions, Essay, Oral

Knowledge Tests

MCQ, Essay, Oral exams

Types of Assessment

Miller‟s Assessment of clinical skills, competence, performance.

Clinical Based Tests

Does

(action)

Shows How

(Performance)

Knows How

(Competence)

Knows

(Knowledge) Knowledge Based Tests

Performance Based Tests

Practice Review

Bloom‟s Taxonomy

Assessment

Kirkpatrick‟s Evaluation Model

(Source from Kirkpatrick, 1989)

Level 1

Reaction

Level 2

Learning

Level 3

Behavior

Level 4

Results

Choosing appropriate assessment methods/tools

Is it valid?

Is it reliable?

Is it feasible?

Assessment - Validity

Are we measuring what we are supposed to be measuring?

Is it an appropriate instrument for the knowledge, skill, or attitude you are testing within the context it is being used?

Assessment - Validity

“Emerging paradigms replace prior distinctions of face, content, and criterion validity with the unitary concept „construct validity‟, the degree to which a score can be interpreted as representing the intended underlying construct.”

(Cook & Beckman, 2006)

Assessment - Reliability

Does the test consistently measure what it is supposed to be measuring?

Types of reliability:

Inter-rater (consistency over raters)

Test-retest (consistency over time)

Assessment - Feasibility

Is the administration of the assessment instrument feasible in terms of time and resources?

Assessment - Feasibility

Considerations

? Time to construct and score ? Ease of interpreting the score/producing

results ? Practical given staffing/organization ? Number of students to be assessed

? Time available for the assessment

? Number of staff available

? Resources/equipment available

Where should you start?

Remember the purpose of the tool is to assess that learning has occurred. It needs to map to your OUTCOMES & OBJECTIVES

Examples of Tools

Kardong-Edgren, S., Adamson, K.A., Fitzgerald, C. (2010).

A review of currently published evaluation instruments for human patient simulation.

Clinical Simulation in Nursing, 6(1), e25-e35. Doi:10.1016/jecns.2009.08.004.

Use of validated tools

It is rarely appropriate to justify the use of a particular scoring tool based solely on previous validation studies (Boulet et al., 2011)

Consider

•The purpose of the assessment

•The administration conditions

•The evidence supporting the tools.

• Self report satisfaction survey

• Outcome measures per semester group

• Specific Learning Objective checklist per group

Where I started

Student Satisfaction

Evaluation of Learning Outcomes

Learning Outcomes S1 S2 S4 S5 Comments

Apply clinical decision

making skills in analyzing

and interpreting complex

data

Met Met Met Met Demonstrated

either directly,

during the

simulation, or

through guided

reflection during

the debriefing

session,

appropriate to

the level of

experience and

expected

knowledge base.

Apply clinical decision

making skills to plan care

Met Met Met Met

Provide care to patients

utilizing principles of

safety

Met Met Met Met

Effectively communicates Met Met Met Met

Reflect on performance

‘in action’ and ‘after

action’

Met Met Met Met

Semester-Specific Learning Objectives

Semester 2 Simulation Learning Objectives

1. Evaluate patient assessment information including vital signs

2. Prioritize and implement physician orders appropriately

3. Recall indications, contraindications, and potential adverse effects of prescribed medication

4. Implement the “5 rights” of medication administration

5. Communicate effectively with patient

6. Document nursing intervention and/or clinical skill performed appropriately

Evaluation of Learning Objectives

Semester 2 Assessment/Evaluation

1 Demonstrated. Initial assessments were generally performed well.

However, most groups were slow to complete a full respiratory

assessment following nebulizer

2 Demonstrated. Some groups were challenged with prioritizing which

meds to administer first. Some groups did not recognize an order

change to increase oxygen from NC to FM. Students were able to

reflect on this during debrief and identified why they thought they were

challenged and what they could do in the future to prioritize more

effectively

3 Demonstrated at an appropriate level for semester 2 students

4 Demonstrated well

5 Demonstrated at a higher level than expected for semester 2 students

6 In most groups documentation was poor. It is unclear why students

performed poorly in this area. It may be because the instructions

during the pre-brief did not explain adequately that documentation was

an expectation during the simulation

Assessment Improvement

• Define low – medium – high performance

• Use of Rubric or rating metric

• Employ a quality assurance/improvement system (Closing the loop)

Assessment Tools

(Brett-Fleegler, 2008)

Assessment Tools

(Malec et al, 2007)

(Gaba, 1998)

Assessment Tools

(Guise et al, 2008).

Assessment Tools

(Lasater, 2007)

Assessment Tools

Objective-Based Scenario Design

Learner

Objectives and Outcomes

Assessment

Environment

Patient

Flow

Debrief

Environment

The environment should be prepared to enhance learning

– Maximize fidelity

– Choose appropriate simulation methodology/equipment

– Room design, props, equipment, patient documentation available to support learning environment

– Actors

– Scripts for patient/actors

Patient

The patient should be created to enhance learning

Must be appropriate to meet the learning outcomes and objectives

Use a template

Example from the California Simulation Alliance

https://www.californiasimulationalliance.org/

Flow

Use a storyboard

Example designed by Marjorie Miller and is used by the CSA

Initial parameters

Identify planned events

Identify triggers

Identify scenario end point

https://www.californiasimulationalliance.org/

Scenario Template

https://www.californiasimulationalliance.org/

Debrief

Debriefing represents facilitated or guided reflection in the cycle of experiential learning (Fanning and Gaba, 2007)

Debrief Process

A 3 Phase model

1.Reactions Phase

2.Understanding Phase

3.Summary

Examples of questions

1. How do you feel about it?

2. Can you explain that further?

3. Was it effective/appropriate?

4. What would you change, if anything, in the future?

5. What do you plan to incorporate into your practice next time?

6. What, if any, obstacles did you encounter?

Summary

• Learning outcomes and objectives are the foundation for your simulation

• Learning outcomes and objectives drive the assessment method, the environment setting, the patient scenario and the flow of the simulation

• The debrief is driven by the learning outcomes and objectives

• Learning outcomes and objectives should be assessed using valid and reliable data

Take Home Message

OUTCOMES & OBJECTIVES ARE ESSENTIAL

FOR AN EFFECTIVE HEALTHCARE SIMULATION

References

Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: the classification of educational goals; Handbook I: Cognitive Domain New York, Longmans, Green, 1956

Boulet, J.R., Jeffries, P.R., Hatala, R.A., Korndorffer, J.R., Frienstein, D.M., and Roache, J.P. (2011) Research Regarding Methods of Assessing Learning Outcomes. Simulation in Healthcare, 6(7): S48-S51

Brett-Fleegler, M., Vinci, R., Weiner, D., Harris, S., Shih, M., & Kleinman, M. (2008). A simulator-based tool that assesses pediatric resident resuscitation competency. Pediatrics, 121(3), e597-603. Dieckmann, P (2009) Using simulation for Education, Training and Research. Lengerich: Pabst cited in Simulation is more than Technology-The Simulation Setting

Cook D.A., & Beckman T.J (2006) Current Concepts in Validity and Reliability for Psychometric Instruments: Theory and Application. The American Journal of Medicine 119, 166.e7-166.e16

Fanning, R.M., Gaba, D.M (2007) The Role of Debriefing in Simulation-Based Learning . Simulation in Healthcare 2(2):115-125

Gaba, DM et al (1998) Assessment of Clinical Performance during Simulated Crises Using Both Technical and Behavioral Ratings. Anesthesiology, 89(1): 8-18.

Guise, J.M., Deering, S.H., Kanki, B.G., Osterweil, P., Li, H., Mori, M., and Lowe, K. (2008) Validation of a Tool to Measure and Promote Clinical Teamwork. Simulation in Healthcare 3(4): 217–223.

Kirkpatrick, D.L (1989) Evaluating Training Programs; The Four Levels. 2nd ed. San Francisco, CA: Berrett-Koehler

Lasater, K (2007) Clinical Judgment Development: Using Simulation to Create an Assessment Rubric, Journal of Nursing Education, 46 (1):496-503

Malec, J.F., Torsher, L.C., Dunn, W.F., Wiegmann, D.A., Arnold, J.J., Brow, D.A; et al. (2007) The Mayo High Performance Teamwork Scale: Reliability and Validity for Evaluating Key Crew Resource, Management Skills, Simulation in Healthcare, (2)1: 4-5.

Miller, G.E (1990) The assessment of clinical skills/competence/performance. Academic Med. 5(9):S63-7

Society for Simulation in Healthcare. http://www.ssih.org/SSIH/ssih/Home/

International Nursing Association for Clinical Simulation and Learning www.inacsl.org

Simulation Innovation Resource Center (SIRC) http://sirc.nln.org

CINHC www.cinhc.org

California Simulation Alliance (CSA) http://www.cinhc.org/programs/simulation/

Bay Area Simulation Collaborative (BASC) www.bayareanrc.org/rsc

Simulation User Network http://simulation.laerdal.com/

www.HealthySimulation.com

http://www.behindthesimcurtain.com

Useful Resources

Thank You

mgilbert@csufresno.edu

Final Slide