1Northwest Center for Public Health Practice
CHILD AND FAMILY
DISASTER RESEARCH
TRAINING AND EDUCATION
2Northwest Center for Public Health Practice
Federal Sponsors
NIMH National Institute of Mental Health
NINRNational Institute of Nursing Research
SAMHSA Substance Abuse
and Mental Health Services Administration
3Northwest Center for Public Health Practice
Principal Investigators
Betty Pfefferbaum, MD, JD University of Oklahoma Health Sciences Center
Alan M. Steinberg, PhD University of California, Los Angeles
Robert S. Pynoos, MD, MPHUniversity of California, Los Angeles
John Fairbank, PhDDuke University
4Northwest Center for Public Health Practice
Evaluating Disaster Mental Health Programs
Part II: From Theory to Practice
Clark Johnson, Ph.D.
Adopted / Modified from materials prepared by: Fran Norris Ph.D., Craig Rosen, Ph.D.
Helena Young, Ph.D.National Center for PTSD
5Northwest Center for Public Health Practice
“Disaster research is different from most other fields in that much of the work is motivated by a sense of urgency and concern. Disaster research has both benefited and suffered from this. It has benefited because the cadre of researchers is fluid, and new ideas are accepted and welcomed. It has benefited also because the result has been an impressively diverse database that includes samples from all different regions of the United States[...]. However, disaster research has also suffered from this situation. Scholarship is not always the best because studies often are undertaken under conditions where there simply is not time to absorb a literature that is scattered across a variety of journals and is mixed in quality. Concerns about experimental designs and scientific rigor must often take a back seat to provider beliefs, consumer demands, and clinical necessities. Most of the research is atheoretical and little of it is programmatic. On the basis of this review, we will state our opinion unequivocally that we do not need more research that establishes only that severely exposed disaster victims develop psychological disorders or, worse, that barely exposed disaster victims do not. We need carefully conceived and theory-driven studies of basic process that are longitudinal in design. [...] We need more research that addresses the needs of diverse populations. We need more complex studies of family systems and community-level processes. We need to identify and investigate novel approaches to community intervention, where the intervention itself has been designed to produce collective rather than individual improvements.”
Source :
Norris, Friedman, & Watson. (2002) 60,000 Disaster Victims Speak: Part II. Summary and Implications of the Disaster Mental Health Research.
Psychiatry 65(3), 240-260
We will start the next session in about 10 minutes and will begin with a discussion of the following text
6Northwest Center for Public Health Practice
Disaster research is motivated by a sense of urgency and concern.
It has benefited from this because:• The cadre of researchers is fluid, and new ideas are
accepted and welcomed. • The result has been an impressively diverse database
that includes samples from all different regions of the United States
7Northwest Center for Public Health Practice
Disaster research is motivated by a sense of urgency and concern.
It has suffered from this situation because:• Scholarship is not always the best because studies
often are undertaken under conditions where there simply is not time to absorb a literature that is scattered across a variety of journals and is mixed in quality.
• Concerns about experimental designs and scientific rigor must often take a back seat to provider beliefs, consumer demands, and clinical necessities.
• Most of the research is atheoretical and little of it is programmatic.
8Northwest Center for Public Health Practice
On the basis of this review, we will state our opinion unequivocally that
We do not need • more research that establishes only that “severely exposed
disaster victims” develop psychological disorders or, worse, that “barely exposed disaster victims” do not.
We do need:• carefully conceived and theory-driven studies of basic process
that are longitudinal in design. • more research that addresses the needs of diverse populations. • more complex studies of family systems and community-level
processes. • to identify and investigate novel approaches to community
intervention, where the intervention itself has been designed to produce collective rather than individual improvements.
9Northwest Center for Public Health Practice
Let’s pretend we’re starting today
When disaster strikes the Psychosocial health of the children and families in our community will be adversely impacted.
Our goal is to implement a program that is designed to minimize this impact.
The evaluation of this program must be designed to:• Guide development (Proactive / Clarificative)• Monitor implementation (Interactive)• Summarize outcomes and results (Impact)
How should we proceed?
10Northwest Center for Public Health Practice
Psychological First Aid (PFA)http://www.ncptsd.va.gov/ncmain/ncdocs/manuals/nc_manual_psyfirstaid.html
PFA is an evidence-informed modular approach for assisting people in the immediate aftermath of disaster and terrorism: to reduce initial distress, and to foster short and long-term adaptive functioning.
It is for use by mental health specialists including first responders, incident command systems, primary and emergency health care providers, school crisis response teams, faith-based organizations, disaster relief organizations, Community Emergency Response Teams, Medical Reserve Corps, and the Citizens Corps in diverse settings.
11Northwest Center for Public Health Practice
Ex Ante Evaluation
Overview of key elements• Lessons from the past• Problem analysis and needs assessment• Objective setting• Alternative delivery mechanisms and risk assessment• Added value of this activity• Planning future monitoring and evaluation• Helping to achieve cost-effectiveness
EX ANTE EVALUATION: A practical guide for preparing proposals for expenditure programmes
Available: (http://ec.europa.eu/budget/evaluation/pdf/ex_ante_guide_en.pdf)
12Northwest Center for Public Health Practice
Pubmed “Psychological first aid”
Recent publication history "Psychological First Aid"
0
2
4
6
8
10
2002 2003 2004 2005 2006 2007
Year
Pu
bli
shed
Art
icle
s s
May 28th, 2007
13Northwest Center for Public Health Practice
Pubmed “Psychological first aid”
23 hits General overview
• 1 - Technology• 6 - Disaster Planning / Policy• 7 - How to’s• 1 - Personal Experiences w/ • 8 - N/A
One article that might help us along• Macy, et.al. (2004).
Community-based, acute posttraumatic stress management: A description and evaluation of a psychosocial-intervention continuum. Harvard Review of Psychiatry. 12,217-228.
14Northwest Center for Public Health Practice
Focus on Macy, et. al. (2004). Helps us avoid “re-inventing the wheel”
• Literature – paucity of evidence that CISD is effective (p218) • “Conceptual and practice framework for assessing and intervening with
children, youths, families and their various types of adult caregivers” (p219)• Template for intervention process and practice protocols (p221-222)
• An evaluation study that can be used as a preliminary template (p223)• Stakeholders study• Case records study• Study of training
• Results (p226): Program effective because• It helped communities handle crises• Trained a network of local people to lead or assist with the interventions
• Identifies study’s limitations (p226-7)• No comparison group• No use of standardized / validated instruments• No analysis of quantitative client-outcome data• Long-term effectiveness unclear
15Northwest Center for Public Health Practice
Evaluation Study
The study was conducted over a five-month period, between June and October 2003. The design was essentially that of a case study structured to capture PTSM’s essential elements and to enable an assessment of program effectiveness, specifically through a three-component design:
(1) a study of stakeholders in order to assess their views of the program, its impact on individuals and communities, and its quality
(2) a study of case records of interventions with individuals and community groups experiencing traumatic events in order to assess the breadth and depth of the interventions, the manpower and time required, and the effectiveness of the interventions; and
(3) an assessment of the effectiveness of the training that was designed to create a cadre of people to assist with community interventions.
16Northwest Center for Public Health Practice
Generate an initial intervention plan
A this point the “plan” is – just a rough sketch of your ideas for:
• Training• Process• ??
Documents include:• Training • Program logic• ??
17Northwest Center for Public Health Practice
Ex Ante Evaluation Lessons from the past
Problem analysis and needs assessment• Problem Analysis
• What is the problem to be solved?• What are the main factors and actors involved?
• Needs Assessment• What is the concrete target group• What are the needs and / or interests of this group
Objective setting Alternative delivery mechanisms and risk assessment Added value of this activity Planning future monitoring and evaluation Helping to achieve cost-effectiveness
18Northwest Center for Public Health Practice
Problem analysis
Roadmap• Define the key aspects of the situation to be addressed by the
program• Identify factors that are likely to influence the key problem• Identify the main groups of actors that influence or that are
being influenced by the situation• Analyze the cause-effect relations between the factors
identified and the interests and motivation of the actors• Construct a visual presentation of these relationships
19Northwest Center for Public Health Practice
Needs assessment
Roadmap• Identify the target population and the most important
subgroups within it• Investigate the situation, motivations and interests of
these groups• Make sure that the identified needs actually
correspond with social, economic and environmental objectives of the community
20Northwest Center for Public Health Practice
Ex Ante Evaluation
Lessons from the past Problem analysis and needs assessment
Objective setting• Define general, specific and operational objectives• Define indicators that measure inputs, outputs, results and
impacts Alternative delivery mechanisms and risk assessment Added value of this activity Planning future monitoring and evaluation Helping to achieve cost-effectiveness
21Northwest Center for Public Health Practice
Key questions
To Generate: Ask:General Objectives What goal are we working towards?
Specific Objectives What will have changed when we achieve it?
Operative objectives What will be delivered to achieve the goal?
Progress Indicators How will we know if we are on course
Success criteria How can we judge if the action has been successful
Outcome indicators How do we know if the desired change has been effected?
22Northwest Center for Public Health Practice
- Inputs - Resources Available for Achieving Goals
Some inputs are tangible resources, such as funding, program staff, office space, supplies, and transportation
Others are less concrete, such as the skills of staff and relationships among staff and with local community leaders
Lack of these resources can greatly limit an organization’s ability to deliver services
Given the unexpected nature of disasters, programs often are initiated before all of the necessary inputs are in place, creating challenges for both the program and its evaluation
23Northwest Center for Public Health Practice
-Outputs – The Measurable Units of Products from Program Processes
Evaluations often focus on the outputs of the service delivery process, such as:
• Number of outreach visits concluded• Number of children receiving counseling • Number of people reached in public education• Number of individuals screened and referred for
more extensive treatment
In some cases, evaluations conclude with outputs, which are used as a proxy for outcomes
24Northwest Center for Public Health Practice
Indicators Of Outputs And Outcomes
Indicators are the observable measures or standards used to monitor or evaluate program success or outcome (e.g., number of clients receiving services, changes in consumer self-reported symptoms or behaviors, or changes in community conditions)
It is the job of the evaluator to ensure that these criteria are defensible
For indicators of success to be meaningful, they must exhibit good construct validity (measure what they claim to be measuring)
25Northwest Center for Public Health Practice
-Processes – Activities or Means to Bring About Program Objectives
Such processes might include:
• outreach to affected people in the community• providing classes or community education on
normal responses to trauma• public relations efforts to increase community
awareness of the agency’s services• training secondary helpers in how to provide
reassurance and support to facilitate recovery• providing brief individual or group counseling or
more extensive intervention• arranging treatment referrals for individuals with
more severe mental health needs
26Northwest Center for Public Health Practice
Ready-to-use Data Collection, Data Management, And Reporting Tools
If possible locate and use tools that someone else has developed and validated (!)
27Northwest Center for Public Health Practice
Individual Encounter Log Used to document interactions
with individuals or families lasting 15+ minutes and involving participant disclosure.
Captures encounter characteristics, risk categories, participant characteristics, & referrals.
Completed by the crisis counselor immediately after the encounter is over.
Training considerations: Eliciting personal information through “active listening” without asking directly.
28Northwest Center for Public Health Practice
Participant Survey (1) Used to obtain feedback
about the program. In one selected week each
quarter, all adults receiving individual or group crisis counseling are given a packet containing a cover letter, survey, pen, and stamped return envelope.
Survey provides some data about immediate outcomes of crisis counseling, such as learning about common reactions to disasters, normalization of feelings and help-seeking, and finding ways to take care of one’s self & family.
29Northwest Center for Public Health Practice
Participant Survey (2) Data on disaster experiences
(p. 1) and distress (p. 2) provide information about participant needs.
Distress measure is the SPRINT-E.
Training considerations: The counselor must be convinced that the survey is the recipient’s opportunity to tell the program about community needs and how well program is meeting those needs.
30Northwest Center for Public Health Practice
Provider Survey (1) Used to capture crisis
counselors’ opinions about training, resources, services provided, and overall quality of the CCP.
The provider survey is collected anonymously from crisis counselors and their supervisors at 6 and 12 months post-disaster.
A packet containing a cover letter, survey, and pen is given to each crisis counselor together with a stamped return envelope, addressed to an external evaluator (presently NCPTSD).
31Northwest Center for Public Health Practice
Provider Survey (2) The survey also measures
worker stress (p. 2). Respondent’s identity is
protected by lack of identifying information, return of the survey to an external evaluator, and aggregation of results.
Training considerations: Conveying reasons for, and importance of, the survey; explaining why high response rate matters.
32Northwest Center for Public Health Practice
Resources
For further information about:• Tools• Databases• Evaluation manual
Contact the
SAMHSA’s Disaster Technical Assistance Center (DTAC). (http://mentalhealth.samhsa.gov/dtac/)
• Email -- http://nmhicstore.samhsa.gov/emails/contact.aspx
• Phone – 1-800-308-3515
33Northwest Center for Public Health Practice
Ex Ante Evaluation
Lessons from the past Problem analysis and needs assessment Objective setting Alternative delivery mechanisms and risk assessment Added value of this activity Planning future monitoring and evaluation
• What types of evaluations are needed and when should they be carried out?
• Are the proposed methods of collecting, storing and processing the follow-up data sound?
• Is the monitoring system fully operational already from the outset of the program implementation?
Helping to achieve cost-effectiveness
34Northwest Center for Public Health Practice
Ex Ante Evaluation
Lessons from the past Problem analysis and needs assessment Objective setting Alternative delivery mechanisms and risk assessment Added value of this activity Planning future monitoring and evaluation Helping to achieve cost-effectiveness
35Northwest Center for Public Health Practice
Where are experts and resources
Let your exploration identify the experts and resources that are available
For our hypothetical example we have the following leads:• Fran Norris - [email protected]• Robert Macy – [email protected]• SAMHSA’s Disaster Technical Assistance Center
(DTAC). (http://mentalhealth.samhsa.gov/dtac/)• Email -- http://nmhicstore.samhsa.gov/emails/contact.aspx• Phone – 1-800-308-3515
37Northwest Center for Public Health Practice
Barriers and Challenges to Conducting Program Evaluation
Conducting program evaluation in the aftermath of disasters poses special challenges
38Northwest Center for Public Health Practice
Crisis And Chaos
In the immediate aftermath of disasters, decisions need to be made quickly on the basis on limited information. The prejudice is towards action, not deliberation.
During the crisis, there may be little interest in collecting systematic information on how the program is working. This shortcoming makes it difficult to monitor program progress and provides few data with which to later evaluate program achievements
In this context, evaluation may be viewed as arbitrary and burdensome, imposed by outsiders without a stake in serving survivors
39Northwest Center for Public Health Practice
Evolving, Adapting Services
The nature of the services may evolve over time as the needs of survivors change
Program models often must be adapted to the community, and providers in the field have a sense of “learning as they go”
Evaluations cannot assume that services are being delivered based on a pre-ordained model. It is essential to continually document program services and delivery strategies in order to be able to evaluate what the program is actually doing at different points in time
40Northwest Center for Public Health Practice
Evolving Community Context
Evaluation results are influenced by the community context, which also evolves over time.
• For example, client satisfaction results may be higher during early phases of recovery than during later stages, when disillusionment sets in.
Outreach programs may discover problems that existed prior to the disaster.
• Evaluations must be careful to differentiate new mental health problems from pre-existing problems
41Northwest Center for Public Health Practice
Factors That Boost Capacity For Program Evaluation
How do we establish an evaluation infrastructure that will allow us to maximize learning in future disasters.
42Northwest Center for Public Health Practice
Advance Political Support
Building evaluation capacity in disaster mental health requires creating an evaluation planning component in State Emergency Disaster Preparedness programs
Embedded in this ethos would be a respect for quality management informed by empirical feedback, and the expectation of accountability
A dialogue among key stakeholders involved in post-disaster recovery -- at federal, state, and local levels -- to set evaluation policy is needed to ensure that the evaluation mandate is feasible, relevant to real-world concerns, and not unduly burdensome
43Northwest Center for Public Health Practice
Outcomes – The Societal Benefits While outputs assess “how much” was done,
outcomes focus on “how much good” was done. They are the least well-specified arena in disaster mental health
Outcomes differ over time:
• Immediate outcomes can be observed directly after completing an activity
• Intermediate outcomes derive from immediate outcomes such as alleviation of psychiatric symptoms, reduced substance use, or improved role functioning
• Long-term outcomes are program benefits such as community cohesion or disaster preparedness
Top Related