Post on 26-Dec-2015
Measuring impact with a single case design: Evaluating training on the Wisconsin Indian Child Welfare Act Cindy Parry, Ph.D. & Michelle Graef, Ph.D.
NHSTES 2013
The Logic of Single Subject Designs
The Basics
Repeated measures Individuals are their own controlsBaseline phase
Obtains a profile of variation absent the intervention
Allows identification of systematic patterns indicative of maturational effects, seasonality, history
Provides a basis of comparison for treatment phase
Treatment phase Measurements taken during time the treatment is applied
Phases are compared to make inferences about treatment effect
Overall Requirements
Consistent measures over time Intervention that can be described fully and implemented with fidelity
Systematic introduction of the intervention
Replication (looking for a functional relationship not an isolated incident of change)
Requirements for Dependent Variables
Observable, quantifiable target behavior (dependent variable)
Can be measured repeatedlyCan be measured with a high degree of inter-observer agreement
For training Training could be expected to impact it Impact would be relatively immediate
Examples of CW Training related Dependent variables
Increased identification of children subject to ICWA
Increase in timely/accurate completion of risk and safety assessment tools
Increase in use of SMART objectives in case plans
Increased presence of concurrent plans in case files
Threats to Validity with Repeated Measures Designs
History-another event occurring at the same time as the intervention that could affect the dependent variable
Maturation-normal developmental processes occurring over time that could explain the results
Are others but these are particularly relevant
Stability of Data
Do the data represent a stable pattern or are they unpredictable?
Minimum of 3 separate, consecutive observations required per phase (Tankersley NHSTES 2012)
The more variable the data, the more data points are needed
Are several methods for representing background variability (e.g. putting a confidence interval around a phase mean)
Length of Baseline and Stability
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
T11
T12
T13
T14
T15
0
10
20
30
40
50
60
70
80
Hypothetical Baseline A
T1 T2 T3 T4 T5 T6 T7 T8 T9 T10
T11
T12
T13
T14
T15
0
10
20
30
40
50
60
70
Hypothetical Baseline B
Common Types of Single Subject Designs
Types of Designs: “B Design”
Monitors the dependent variable during treatment
Shows trend but can’t make causal inference
Types of Designs: “AB Design”
Shows change from baseline to treatment
May allow causal inference, but doesn’t control for history
Types of Designs: “ABAB” Design
Allows causal inference but only works where treatment can realistically be withdrawn
Jan. Feb. Mar. Apr. May Jun. Jul. Aug. Sept. Oct. Nov. Dec. Jan. Feb. Mar. Apr.0
10
20
30
40
50
60
70
Percentage of Court Reports Completed on Time
A B A B
Types of Designs: “Multiple Baseline”
Staged start for the intervention for different groups
Allows causal inference where intervention cannot be withdrawn
Project Background
“Best Outcomes for Indian Children”
Tribally-driven collaborative effort between
WDCF, MCWIC, and the 11 Wisconsin Tribes
WICWAlaw
•The Project is focused on the state-wide implementation of the Wisconsin Indian Child Welfare Act (WICWA), which became law in December 2009
• The WICWA is a codification into state law of the Federal Indian Child Welfare Act (ICWA), which became law in 1978.
Goals of project
• SHORT TERM • Train CW agencies on
tribal child welfare practices
• Modify DCF Tribal child welfare approaches
• Incorporate WICWA requirements into court procedures and the legal process
• Update Adm. Rules and program standards to integrate WICWA
• Improve Tribal/State child welfare relationships
• LONG TERM• Strengthen relationships
b/w state, county, adoption agencies, state and tribal courts
• Increase state wide understanding of the history and purposes of the acts in child welfare system
• Increase identification of ICWA eligible children
• Increase formal notice to tribes
• Increase adherence to WICWA placement preferences
State Advisory Board
Variety of disciplines involved in the child welfare continuum
•Recommends policy and practice changes based on stakeholder input
•Three working subcommittees: • Curriculum• Qualified Expert Witness• Active Efforts
Cross-systems integration
• Legislative Branch – Codification • Judicial Branch – State Court Office - Children’s court
Improvement Program– On going judicial training – Revised ICWA Court Forms
• Wisconsin Public Defenders Association • Executive Branch – Department of Children and Families • Specific Programs and internal
Departments
Key Implementation Drivers(NIRN)
Specific drivers to effect system change in this project:
•Leadership
•Training
•Coaching
•Systems Intervention
•Facilitative Administration
•Decision Support Data Systems
National Implementation Research Network (NIRN)
“Pulling multiple levers”
Advisory Board and stakeholder workgroups
New training on WICWA offered to all child welfare staff, supervisors, central office
Specialized legal training for attorneys, briefings for judges
Revised WDCF policies Desk aids for case workersChanges to eWiSACWIS system Revisions to CQI system for review of ICWA cases
Examples of outcomes: ICWA records generated
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July -D
ec. 1
20
200400600800
10001200
162 146 167347
519668 714
963
Number of ICWA Records Cre-ated in eWiSACWIS January 2009 -
December 2012
Examples of outcomes: ICWA identifications
Summary of Project Overview
Training is one of multiple systemic interventions, with overlapping implementation periods
Strong evidence that trainees are satisfied, are learning a lot, and are motivated/plan to use what they’ve learned on the job when they have opportunity
Emerging evidence of improved outcomes in eWiSACWIS data
Can we determine the impact of training over and above that of the new WICWA tab in eWiSACWIS?
Analysis of Single Subject Data: Our findings
Types of Analysis for Single Case Designs
Visual Look for obvious contrast between phases in
Level Trend Overlap Variability
More and larger contrasts are evidence of importance of change
Immediacy is evidence of importance of change
Statistical Apply statistical tests of significance to patterns
Advantages and Disadvantages
Visual Analysis
Simple; e.g. graphs and descriptive statistics
Differences that are obvious are more likely to be meaningful
Low power
Danger of confirmatory bias and over-interpretation of random variation
Low inter-rater reliability
Statistical Analysis More complex; e.g.
regression discontinuity models
Higher power
Less prone to human error and biases
Statistical significance ≠ practical significance
Require a long time series
Must meet assumptions about independent distribution of residuals (autocorrelation)
Combined Graphical and Statistical Analysis
Pros Graphical aids like trend lines and means can aid in interpretation and improve inter-rater agreement about change
Cons Still suffer from risk of over-interpreting random fluctuations in a short time series
Recommendations (Nugent 2010)• Use both mean referenced and and trend
referenced representations of background variability to supplement interpretation
AB DesignVisual Analysis Immediacy of Change
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July -D
ec. 1
20.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Percentage of American Indian Children1 Discharged from Out of Home Placement
Identified as ICWA Children Jan-uary 2009 -December 2012
AB DesignLevel of Change: Visual Analysis Phase Means and Medians
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July -D
ec. 1
20.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Percentage of American Indian Children1 Discharged from Out of Home Placement
Identified as ICWA Children Jan-uary 2009 -December 2012
AB DesignCombined Method:
Background Variability Relative to Mean; 2 SD method
Nourbakhsh and Ottenbacher (1994) Ja
n.-Ju
ne 0
9
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July -D
ec. 1
20.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Percentage of American Indian Children1 Discharged from Out of Home Placement
Identified as ICWA Children Jan-uary 2009 -December 2012
AB DesignCombined Method: Percentage of Data Points Exceeding the Median (PEM)
Ma (2006)Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July -D
ec. 1
20.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Percentage of American Indian Children1 Discharged from Out of Home Placement
Identified as ICWA Children Jan-uary 2009 -December 2012
AB Design: Visual Analysis Trend Based
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July-D
ec. 1
20%
20%
40%
60%
Percentage of American Indian Chil-dren1 Discharged from Out of Home
Placement Identified as ICWA Children January 2009 -December
2012
Observed Predicted
AB Design:
Combined Method Trend Based
WICWA Training Evaluation: What we hoped to see
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July-D
ec. 1
20
20
40
60
80
Average Percentage of Indian Chil-dren Identified as ICWA by Time
Period and Training Group
Group 1 Group 2 Untrained
Combination DesignAB1
AB1B2
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July-D
ec. 1
20.00%
30.00%
60.00%
Untrained
B
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July-D
ec. 1
20.00%
20.00%40.00%60.00%
Training Group 1
A
A
B1 B2
Data Considerations, Lessons Learned, and Recommendations
Data Considerations
Need repeated measures over timeNeed sufficient numbers at each time period
Definitions of the measures need to remain constant
Administrative data sources are both promising and challenging
Offer access to measurements of child and family outcomes over time
Not designed for research Extracts needed for analysis and how they are drawn matters
Getting to the right variables can be like peeling an onion!
The Unit of Analysis: Unique Child
Unique Child Most Recent Completed Episode of Care
Unique Child Earliest Removal in Study Period
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July-D
ec. 1
20
0.1
0.2
0.3
0.4
0.5
0.6
0.7
Average Percentage of Indian Children Identified as ICWA by
Time Period and Training Group
untrainedTrng1 May-Dec11
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July-D
ec. 1
20
0.10.20.30.40.50.60.70.80.9
Average Percentage of Indian Children Identified as ICWA by
Time Period and Training Group
untrained trng1 may-Dec11
The Unit of Analysis: Snapshot (duplicated count)
Jan.
-June
09
July-D
ec. 0
9
Jan.
-June
10
July-D
ec. 1
0
Jan.
-June
11
July -D
ec. 1
1
Jan.
-June
12
July-D
ec. 1
20
0.2
0.4
0.6
Average Percentage of Indian Chil-dren Identified as ICWA by Time
Period and Training Group
Untrained Trng1 May-Dec11
Questions for Discussion
What types of training applications might this work for?
How do we separate the contribution of training from other factors affecting implementation?
Is it even possible, feasible, or necessary to tease out the effects of multiple contemporaneous interventions?
Further Reading
Nugent, William R., (2010). Analyzing Single System Design Data. New York : Oxford University Press
Tankersley, Harjusola-Webb, and Landrum (2012) Using single-subject research to establish the evidence base of special education. Intervention in School and Clinic. 44(2). Pp. 83-90
Exercise
In groups
Discuss a training evaluation situation where single case methods might be appropriate
On your worksheet list Your evaluation question The type of single case design you would use
Data sources Potential issues/pitfalls
Report out