Practice: Evaluate your outcomes to show your program is making a difference

Key Action: Design the most rigorous evaluation possible

TOOL: Decision Tree: Determining Feasibility for Rigorous Evaluation Design

Purpose: This decision tree guides you through a series of questions to help you determine which rigorous evaluation design is most appropriate for your program. Progressing from most to least rigorous designs, the flow chart presents the criteria that must be met for each type of evaluation. The accompanying list of evaluation design definitions provides an explanation of the structure and methods used in each design.

Note: Design 1 or “Experimental” is not the only type of rigorous evaluation design; designs 2 through 5 in the chart, which are all “Quasi-experimental,” are also considered to be rigorous.

Instructions: 1. Review the decision tree and accompanying definitions.

2. Get answers to any questions you have about whether your schools meet the various criteria.

3. Develop an opinion about which evaluation design will be most appropriate.

1

Practice: Evaluate your outcomes to show your program is making a difference

Key Action: Design the most rigorous evaluation possible

Decision Tree: Determining Feasibility for Rigorous Evaluation Design

2

Are students randomly assigned to magnet school(s) by lottery?

Is there an oversubscription, or more students applying than are accepted to the magnet school(s)?

Can you ensure that students who aren’t accepted do not enroll in a school similar to the magnet school(s)?

NO

YES YES YESEXPERIMENTAL DESIGN1

Are consistent (5- year) student achievement data available for magnet and non-magnet schools?

Are consistent (5- year) demographic data available for magnet and non-magnet schools?

Are consistent (5

Have attendance boundaries for non-magnet schools been consistent for the past 5 years?

Have attendance

Are there a large number of schools (e.g., 20 magnet schools and 40 non-magnet comparison schools)?

Have students used the same test for 5 years?

YES YES YES YES

PRE-POST TEST OR POST-TEST ONLY5

INTERRUPTED TIME SERIES3

INTERRUPTED

Does the district contain pre-conversion schools with similar demographics and prior student achievement?

NO

NONO

NO NO NO NO

YES

NO

COMPARISON GROUPS PROPENSITY SCORING TO MATCH SCHOOLS4

Are there multi-year student data in both magnet and non-magnet comparison schools?

Are there multi-year

NO

YES

COMPARISON GROUP PROPENSITY SCORING TO MATCH STUDENTS4

Quasi-Experimental Design2

YES

Practice: Evaluate your outcomes to show your program is making a difference

Key Action: Design the most rigorous evaluation possible

Decision Tree for Magnet Program Rigorous Evaluation Design: Definitions

Refer to these key definitions of various rigorous evaluation designs to help you navigate the decision tree on page 2.

1 Experimental Design

This design is also referred to as a “randomized controlled trial.” Experimental design is possible only when more students apply for the magnet school than can be accommodated. When there is oversubscription, students can be randomly assigned to the magnet school through a lottery system. It is also critical that nonselected students then enroll in school(s) that are not similar to the magnet school to be able to attribute differences in outcomes to differences in the magnet program.

2 Quasi-experimental design

All types of quasi-experimental designs share the characteristic of attempting to control for some unknown quality that may influence an outcome. An important variable is selection bias: whether the very act of choosing to attend a magnet school indicates a difference between a magnet school student and a non-magnet school student, even if both share key characteristics such as demographics and prior achievement. All quasi-experimental designs, then, are efforts to “equalize” as much as possible the two groups.

3 Interrupted Time Series

This design requires student-level data from repeated intervals before and after magnet school attendance. Trends in each magnet student’s outcomes, such as achievement prior to enrollment in a magnet school, are compared to trends after magnet school attendance. Thus, the evaluation avoids the issue of selection bias by comparing each “chooser” with him/herself and it is possible to attribute changes in outcomes to attending the magnet school. Interrupted time series require a fairly long history of scores and sufficient numbers of participants to ensure that the changes are, in fact, associated with enrollment in magnet schools.

4 Comparison Group

Many quasi-experimental designs compare groups of magnet school students to groups of non-magnet school students. The key to this type of design is to ensure as much as possible that students in the comparison group are similar to those in the magnet school. One technique used is propensity score matching, which selects a comparison group using variables that are most likely to have an effect on the outcome, such as demographics and prior test sores, in the case of magnet school achievement. Often, the matching occurs in two stages—first, schools are matched on demographics, prior achievement, and other variables of interest; then students within those schools are matched to form the comparison group.

5 Pre-Post-Test or Post-Test Only

Depending on the data available, these types of comparison group designs may be the most rigorous design possible for your evaluation. Both compare outcomes of magnet students with non-magnet students. In pre-post-test comparison designs, the pre-test indicates differences in outcomes before magnet school attendance and, additionally, matching is used to make sure the differences between the two groups of students is as small as possible. With post-test only design, the matching of students needs to be strong in order to control for differences in post-test results between the magnet students and non-magnet students.

3