Promoting a Culture of Evidence and Use of Data through Program

Evaluation Session Theme 2

Presentation to:

OSEP Project Directors’ ConferenceJuly 21, 2009

Presentation by:

Thomas Fiore, Ph.D.Project DirectorCenter to Improve Project PerformanceWestat

2

Session Theme 2

Evaluation tools and processes can help project directors do the best job with project planning, implementation, and accountability.

3

Presentation Outline

Presentation outline• Keeping a project focused• Logic model as the starting point• Evaluation plan• Evaluation design• Evaluation implementation• CIPP’s role providing general TA• CIPP contact information

4

Keeping a Project Focused

Project purpose is to do a good job of the right thing.• Need data to document the activities (outputs) of a

project.

• Need data to document the overall success or lack of success (outcomes) of a project.

• Most important, need data to determine whether innovations should be scaled up, changed, or abandoned.

• Need a tool to understand which data are important—that tool is a logic model.

5

Logic Model as the Starting Point

A logic model . . .

• Portrays a project’s overall plan;

• Clarifies the relationships among a project’s goals, activities, outputs, and outcomes; and

• Displays the connections between those defining features of a project.

6

Logic Model Structure

Create a coordination hub where the various TA&D centers funded by OSEP and other federal agencies can find and share information and resources, collaborate and problem-solve together.

Provide logistical support for coordination, communication and collaboration maintaining/

expanding workgroups

developing listservs

maintaining and updating an integrated events calendar

Number/types of workgroups created and maintained

Number/types of listservs and listserv participants

Up-to-date events calendar

Goals

Strategies/ Activities

Outputs

Direct Intermediate

Long-term

Outcomes

An effective single point of entry for network resources is implemented/maintained

Beneficial connections exist among Network participants

Successful ways of locating and sharing information and resources among network members and others are implemented/maintained

The capacity of the TA&D Network members to serve clients increases steadily

Inputs• OSEP funding • Project staff• Prior experience• Research-based

policy and practices

External Factors/Context: Other federal initiatives; OSEP policy environment; grantee’s accumulated experience and visibility.

7

Logic Model

• Thus, logic models can be used as a starting point to plan data collection and analysis aimed at measuring project processes and performance.

• Systematically measuring project processes and performance is evaluation.

• A logic model implies a causal relationship that flows from goals to outcomes.

• Evaluation can be viewed as a test of the logic model’s implied hypotheses of this causal relationship.

8

Evaluation Plan

From the logic model, develop a plan for collecting and analyzing data.

• Focus on outputs and outcomes

• Useful for formative [define] or summative purposes [define], or both

9

Evaluation Plan

Develop the specific plan by answering these questions:

• What data are required to demonstrate project effectiveness or to provide information on the overall program effectiveness?

• What are the strategies/activities that should be given priority—that is, which ones should be evaluated because they are important?

• Who are the targeted recipients of interest and in what settings?

10

Evaluation Plan

Develop the specific plan by answering these questions: [more]

• What are the evaluation questions?

• What data collection activities are needed?

• How will the data be analyzed?

• What are the necessary timelines, staff assignments, and cost allocations across years?

• How will this be documented and reported?

11

Evaluation Plan

Logic model leads to evaluation questions:

→ Relevant goals (not necessarily all)

→ Salient strategies/activities related to those goals

→ Outputs associated with the strategies/activities

→ Outcomes (the most consequential ones)

→ Evaluation questions

12

Evaluation Plan

Goal: Create a coordination hub…

Strategies/activities: Provide logistical support…

Outputs: Number/types of workgroups, etc.

Outcomes: Successful ways of locating and sharing information and resources…

Evaluation question: To what extent have activities supporting coordination, communication, and collaboration been effective in enabling TA&D Network members to do their work efficiently and without duplication?

13

Evaluation Design

Evaluation plan leads to an evaluation design:

→ Evaluation questions

→ Measurable outputs or outcomes

→ Methods that capture change

→ Types of data collection

→ Instruments

14

Evaluation Design

Evaluation question: To what extent have activities…been effective in enabling TA&D Network members to do their work efficiently and without duplication?

Measurable outcome: An integrated technology system is implemented and used.

Types of data collection: Record review, survey

Instruments: Record Review and Web Statistics Protocol, Annual TA&D Network Survey

15

Evaluation Design

Evaluation design continues with:

• Instrument development

• Sampling

• Data collection scheduling

16

Evaluation Implementation

Evaluation implementation requires:

• Data Collection

• Analysis

• Reporting

17

Summary

• Sounds complicated, but projects are doing much of this already.

• Best when integrated into the overall implementation of the project.

• Best when formative and summative evaluations are integrated.

• Doesn’t need to be comprehensive—don’t need to measure everything in the most rigorous way to have information that can be useful.

• Overall, evaluation answers the question of great interest to funders and to clients: What good is this doing?

18

CIPP’s Role

CIPP role is two-fold:

• To guide, coordinate, and oversee the summative evaluations of 12 large grant-funded projects selected by OSEP.

• To provide technical assistance to current OSEP grantees in conducting formative and summative evaluations of their projects funded through the following programs: Parent Information Centers, Technical Assistance and Dissemination, Personnel Development, and Technology and Media Services.

19

CIPP’s Role Providing General TA

Vehicles for accessing TA:• CIPP website (www.cippsite.org) provides an

opportunity for staff of OSEP-funded projects to pose questions or ask for assistance.◦ CIPP website contains information and reference

materials on formative and summative evaluation. ◦ Evaluation Briefs will be available that discuss

information requested with some frequency and that can be downloaded from the website.

• Toll-free telephone line (1-888-843-4101) also enables project staff to pose questions or ask for assistance.

20

CIPP Contact Information

Website: www.cippsite.org

Toll-free telephone: 1-888-843-4101