Copyright © 2011

American Institutes

for Research

All rights reserved.

Oregon 21st Century Community

Learning Centers Program Evaluation

Neil Naftzger & Deborah Moroney

October 21, 2011

2

Topics

Provide a summary of our proposed approach to undertaking the evaluation

Evaluation Questions New Data Collection Efforts Leading Indicator Development Timeline

3

The Evaluation Team

• American Institutes for Research Recent merger with Learning Point Associates Demonstrated 21st CCLC and afterschool content

knowledge- Statewide 21st CCLC and afterschool evaluation and

research studies in New Jersey, South Carolina, Texas, Washington and Wisconsin

- Responsible for the development and maintenance of the Profile and Performance Information Collection System (PPICS)

- Support the U.S. Department of Education in monitoring state delivery of 21st CCLC and in the creation of practitioner guides

- Provider of afterschool training and technical assistance based on our Beyond the Bell toolkit and currently serve as the statewide training and technical assistance provider for 21st CCLC in Illinois

4

The Evaluation Team

• Gibson Consulting Group, Inc. Texas-based research and evaluation firm

Working with AIR in completing the statewide evaluation of the 21st CCLC program in Texas

Recently completed 40 two-day site visits of 21st CCLC programs in Texas which included:- Site coordinator interviews- Staff focus groups- Activity observations- Administration of student surveys

5

The Evaluation Team

Leading Indicators Work Group (LIAG)A representative group of local 21CCLC leaders from Oregon.

LIAG ChargeProvide information about how well an individual center

and the state as a whole are doing in implementing programming that is likely to achieve the goals and objectives specified for the program

Inform efforts to establish targets that centers should be striving toward in the implementation of their program

Help inform state staff on what steps need to be taken from a training, technical assistance, and policy development front to support grantees in the achievement of program improvement goals

6

Evaluation Questions: Program Outcomes

• To what extent is there evidence that students participating in services and activities funded by 21st CCLC demonstrated better performance on the outcomes of interest as compared with similar students not participating in the program?

• To what extent is there evidence that students participating in services and activities funded by 21st CCLC more frequently demonstrated better performance on the outcomes of interest?

• To what extent is there evidence of a relationship between center and student characteristics and the likelihood that students demonstrated better performance on desired program outcomes?

7

New Data Collection Activities: PPICS Module

• Modified PPICS to allow for the collection of student identifiable information

• Student-level data will be collected in relation to Regular Attendee participation during the summer of 2010 and 2010-11 school year

• Integrated into the existing Regular Attendee module of PPICS

• Advise selecting the Regular Attendee reporting option in PPICS if you have not already done so

• All PPICS passwords were reset to conform to stricter security parameters and re-entry of username and password information is required when providing student identifiable data

Important Points1. Username and Password form2. PPICS breakout session

8

New Data Collection Activities: Youth Outcomes

• Data will be used to run queries against the state assessment data warehouse to obtain reading and mathematics scores and other relevant outcome data for 21st CCLC participants and non-participating students attending the same schools

• Data will be used to support impact analyses predicated on comparing 21st CCLC program participants with non-participants

• Method of analysis allows us to sort out preexisting differences between students who attend and those who do not

9

New Data Collection Activities:Program Quality

Site Coordinator SurveyFocus on practices, policies, and procedures adopted by 21st

CCLC-funded programs: Collaboration & Partnership Intentionality in activity and session design Linkages to the school day Data on student academic achievement to inform

programming Practices supportive of positive youth development Practices supportive of family engagement

Site VisitsHighlight promising activity delivery practices:

Visit a small number of especially high performing programs

Conduct program observations employing the CLASS observation tool

10

Interrelated Factors of Program Quality

11

Goals of the Leading Indicator System

Provide information about how well an individual center and the state as a whole are doing in implementing programming that is likely to achieve the goals and objectives specified for the program

Help establish a standard of quality that grantees should be striving toward in the implementation of their program

Influence grantee behavior by detailing service delivery expectations and their performance relative to these expectations

Help inform state staff on what steps need to be taken from a training, technical assistance, and policy development front to support grantees in the achievement of program improvement goals

12

We need your help answering these

Key Questions

+ Is this Indicator understandable and interpretable?

+ Does this Indicator convey meaningful information?

+ Will this Indicator support discussions and conversations with 21st CCLC staff?

Other ways you can give us feedbackAttend the afternoon Leading Indicator breakout sessionEmail us your comments

13

LI: Partners associated with the center are actively involved in planning, decision making, evaluating, and supporting the operations of the afterschool program.

LI: Staff from partner organizations are meaningfully involved in the provision of activities at the center.

LI: Staff at the center will be engaged in intentional efforts to collaborate and communicate frequently about ways to improve program quality.

LI: Steps are taken by the center to establish linkages to the school day and use data on student academic achievement to inform programming

Leading Indicators:Collaboration & Partnership

14

LI: Staff at the center are provided with training and/or professional development.

LI: Staff at the center complete one or more self-assessment during the programming period.

LI: Staff at the center are periodically evaluated/assessed during the program period.

Leading Indicators:Staff

15

+Students+

LI: There is evidence of alignment between (a) program objectives relative to supporting youth development, (b) student needs, and (c) program philosophy/model AND frequency/extent to which key opportunities and supports are provided to youth.

LI: There is evidence of alignment between(a) program objectives relative to the academic development of students, (b) student needs, and (c) program philosophy/model AND activities being provided at the center.

LI: Intentionality in activity and session design among staff responsible for the delivery of activities meant to support student growth and development in mathematics and reading/language arts.

Leading Indicators:Intentional Activities

16

+Families+

LI: Steps are taken by the center to reach out and communicate with parents and adult family members of participating students.

LI: There is evidence of alignment between (a) program objectives relative to supporting family literacy and related development, (b) family needs, and(c) program philosophy/ model AND activities being provided at the center.

Leading Indicators:Intentional Activities

17

Leading Indicator Reports

Goal is to embed leading indicator reports into PPICS in the interest of supporting program improvement efforts and mid-year corrections before the programming period ends

Provide a snapshot of center status Needs to be understandable and interpretable Needs to convey meaningful information Needs to support discussions and conversations

with 21st CCLC staff Facilitate an advisory group to guide and support

the leading indicator development process

18

We need your help answering these

Key Questions

+ Is this Indicator understandable and interpretable?

+ Does this Indicator convey meaningful information?

+ Will this Indicator support discussions and conversations with 21st CCLC staff?

Other ways you can give us feedbackAttend the afternoon Leading Indicator breakout sessionEmail us your comments

19

Report Functionality

Goal is to ensure reports can support meaningful comparisons

Against statewide averages Over time By key center characteristics

Grade level Recruitment and retention policies Staffing model Activity model Maturity

May attempt to include recommendations and action planning tools as well

20

Data Collection Timeline

Task Start Date End Date

PPICS Related Activities

APR Data Open 11/15/2011

Student ID Data 10/18/2011 12/15/2011

Site Coordinator Survey

11/21/2011 12/15/2011

Site Visits

Site Visits 2/15/2012 3/15/2012

21

Contact

Oregon Evaluation general email: OR21stcclc@air.org

Neil Naftzger Deborah MoroneyP: 640-649-6616 P: 312-283-7609 nnaftzger@air.org dmoroney@air.org

American Institutes for ResearchGeneral Information: 800-356-2735

www.air.org