Kirkpatrick model of Kirkpatrick model of evaluation to assess public evaluation to assess public

health workforce health workforce preparedness trainingpreparedness trainingAmerican Public Health AssociationAmerican Public Health Association

November 5, 2013November 5, 2013Boston, MABoston, MA

Geraldine Aglipay, BS and Karen Peters, DrPH

Illinois Preparedness and Emergency Response Learning Center

MidAmerica Center for Public Health Practice

University of Illinois at Chicago, School of Public Health

Presenter DisclosuresPresenter Disclosures

(1) The following personal financial relationships with commercial interests relevant to this presentation existed during the past 12 months:

Karen E. Peters

“No relationships to disclose”

Overview: IPERLCOverview: IPERLC

• 1 of 14 Preparedness and Emergency Response Learning Centers (PERLC)

• Funded by CDC’s Office of Public Health Preparedness and Response

• Provides education, training and technical assistance services to enhance capacity/performance of IL public health workforce regarding preparedness and emergency response

Overview: Mini-Grants for Overview: Mini-Grants for Emergency PreparednessEmergency Preparedness

• To increase emergency response capability and capacity in Illinois –To support short-term, targeted

preparedness related activities for which there was a need but no discretionary resources available

–To gauge the efficacy of individual training on organizations, their staff and partners (e.g., fire depts, LTC facilities)

Overview: Mini-Grants for Overview: Mini-Grants for Emergency PreparednessEmergency Preparedness

• Grant awards: Up to $4,500 for 6 month projects (March 1-August 30, 2012)

• Eligibility/Awardees (n=13):–Local health departments [n = 10]

–Primary care clinics (FQHCs or Stand alone) [n=2]

–CBOs (NFPs, Schools, FBOs) [n=1]

Overview: Mini-Grants for Overview: Mini-Grants for Emergency PreparednessEmergency Preparedness

• Project descriptions:–Conduct training activities; Run exercises;

Develop response plans; Conduct needs assessments; Create resources; Build support networks

• Audiences and Partners (at risk):–Disabled persons, rural, seniors, economically

disadvantaged, isolated, infants/children, limited/non-english speaking persons

Overview: Evaluation using Overview: Evaluation using Kirkpatrick ModelKirkpatrick Model

• Adopted by CDC for PERLC evaluation, organized as 4 levels:–Level (1) Trainees' reaction

–Level (2) Knowledge acquired, skills improved, or attitudes changed

–Level (3) Practice behavior changed (field)

–Level (4) Results, or impact (at agency/organizational level)

The IssueThe Issue

• Little literature on the intermediate and long term impacts of preparedness and emergency response training on agency capacity to respond to emergencies

• Focus is generally on individual KSA capacity (knowledge, skills, attitudes)

The Level 3 & 4 GranteesThe Level 3 & 4 Grantees

• IPERLC selected 3 grantee LHDs based on training content and stakeholders (LTCF) for participation in Level 3 & 4:–1 rural, 1 suburban, 1 urban

–All 3 sites completed Level 1 & 2

–Level 3 evaluations (6,12 months) at 2 sites

–Level 4 evaluations (at 12 months) at all 3 sites

Results: Level 3Results: Level 3

• Self report increased strength and confidence in ability to apply training lessons to field experience

• Work policy changes help with work preparedness

• Many semi- or fully retired, so future training delivery/content needs to be considered for their needs [attrition]

Results: Level 3Results: Level 3

• “additional training on policies and procedures needed”;

• “more exercises”;

• “trained people more willing to assist with other activities as they realize that preparedness is about team work”

Results Level 4Results Level 4

• Partnerships have been strengthened [LTCF, special needs populations (4.6/5.0)

• Training outcomes meet organization’s needs/responsibilities for preparedness (4.0/5.0)

Results Level 4Results Level 4

• New resources brought into the partnership

• 2 of 3 report new policies developed

• In absence of minigrant training, not able to accomplish same outcomes

Results Level 4Results Level 4

• ‘Continued relationship building with special needs and LTC facilities…not historically involved with preparedness’

• ‘We grow and expand relationships with partners and staff are more aware of their responsibilities’

• ‘A foundation to prepare for full scale evacuation all LTCF participated and better able/quite confident to respond to emergencies’

DiscussionDiscussion

• Level 3 and 4 labor intensive

• Can do Level 4 (retrospectively) without other levels to see organizational changes but don’t have resources to conduct all 4 Kirkpatrick levels but…

• Requires key informant interviews, document policy reviews not only with primary public health agencies, but their top 1-3 key partners

ConclusionsConclusions

• Evaluation of (preparedness) training using Kirkpatrick methodology provides guidance but is labor intensive to implement

• Instability of funding for preparedness training and personnel changes at local level make comprehensive evaluation difficult

Contact InformationContact Information

• Victoria Wiebel, Assistant Director

• Illinois Preparedness and Emergency Response Learning Center

• vikki@uic.edu

• http://www.midamericacphp.com/iperlc/

Thank you!Thank you!