Evaluation 101: How to Find out if

Doing is Working

Jessica Dym Bartlett, M.S.W., Ph.D. Mallary I. Swartz, Ph.D.

Elisa Vele-Tabaddor, Ph.D.

Brazelton Touchpoints Center Division of Developmental Medicine

www.brazeltontouchpoints.org 1

Introductions

2

Where to Start

Start with a Good Question

What do you want to know about your program/initiative?

What do you want to know about your services/initiative activities?

3

Types of Evaluation

Needs/Demand Assessment Systematic collection and analysis of information nature of problem and resources

Process/Formative Evaluation Whether programs and activities are operating as planned

Outcome Evaluation Whether programs and activities affect outcomes for participants

4

Participatory Evaluation

Involve participants and programs in evaluation design and implementation Ensure all aspects of the evaluation are made accessible to participants Value process evaluations

View the evaluator role more broadly

5 Patton, MQ (2008). Utilization-focused evaluation: The new century text, fourth edition. Thousand Oaks, CA: Sage. Slide adapted -Tiered Approach 11/1/2012

Developmental Evaluation

Long-term relationships between evaluators and program developers Ask evaluative questions and gather information to provide feedback to program Evaluator is part of collaborative team focused on continuous improvement.

Patton, M.Q. (2008). Utilization-focused evaluation: The new century text, 4th ed. Thousand Oaks, CA: Sage.

6

Evaluation Design

Non-Experimental/Pre-Post One group of program participants

Quasi-Experimental Non-randomly selected comparison group

Experimental

Random assignment to treatment and control groups

7

Sampling

8

Statistical (probability) sampling Random selection of individuals from the population to ensure sample is representative

Non-statistical (judgment) sampling Based on knowledge about population No structured statistical process

Purposeful sampling Selects specific informants best able to answer questions of interest

-

Types of Data

Quantitative vs. Qualitative

Primary vs. Secondary

Cross-sectional vs. Longitudinal

Case-specific vs. Aggregate

9

Common Data Collection Strategies & Measures

Document Review Interviews & focus groups Observations Surveys Standardized Measures Non-Standardized (Home-Grown) Measures

10 -

Ethical Concerns

Informed consent Privacy, confidentiality, anonymity Harm or risk Intervention and advocacy Soundness of research Ownership of data Use of results

11

Adapted from: Miles, M.B. & Huberman, A.M. (1994). Qualitative data analysis: An expanded sourcebook. 2nd ed. Thousand Oaks, CA: Sage Publications.

outside evaluation expert!

12

Small Group Exercise: Creating an Evaluation Plan

13

Large Group Discussion

How was this planning process for you?

What issues came up in your group?

14

Wrap-up

15