After-School Program QualityAfter-School Program QualityHow Do We Know It When We See It?

Theresa Ferrari, Lisa Lauxman, Ina Linville, Deirdre Thompson, Nancy Valentine

CYFAR 2004 Pre-ConferenceMay 11, 2004

Involvement in After-School Programs:An Extension Program Development Model

E x ten s io n M a na g ed

C re a tio n of N ewN o n -P ro fit

B oard

M a na g ed byO th er A gen cy or

O rg an iza tion

C o m m u nity M a na g ed

Im pacting the Q uantity ofQ uality After-School Program s

W orking w ithFam ilies

P ro vid ingC u rr ic ulum

S u pp o rt

S ta rtin g & M ain tain ing4 -H C lu b s inA fter-S ch o ol

S e ttiing

E n ha nc ingE d uc a tio n alE n viro n m e nt

A fter-S ch o olS ta ff De ve lop m e nt

T rain ing

Q u a lity o fE n viro n m e nt

C h ildO u tco m es

E va lua tion

E d uc a tio n andT e c hn ic al A ss is ta n ce

C o m m u n ityP a rtn ersh ips

Im pacting the Q uality ofExisting After-School Program s

C onductingC om m unity N eeds Assessm ent

Adapted from the Extension School-Age Child Care Consortium, 1993

Revised April 2003

Distributed at the 4-H Afterschool National Roll-Out Conference, St. Louis, MO

Welcome!

Our Purpose

Extension Cares Initiative System Update: “The Big Picture”

Dr. Nancy Valentine

Extension CARES…Extension CARES…for America’s for America’s

Children and YouthChildren and Youth

The Big Picture

Overall GoalOverall Goal

• To increase the quality, availability, accessibility, affordability, and sustainability of child care, school-age care (after-school), and teen out-of-school programs

How are We Doing?.......

Did You Know?Did You Know?• Extension has provided 2,530 hours of training

for 22,700 childcare, after-school, and teen out-of-school time staff, families, and community leaders valued at $202,400.*+

• Over three-fourths found the training to be of high quality and would recommend it to others.

*Based on 2,530 training hours reported x 3 hrs. preparation time for each hour taught x average salary calculation of $20.00 per hour. +Based on data from 18 states. 2001 – 2003

Did You Know?Did You Know?

• Over three-fourths of those trained gained knowledge or skills, found the information to be useful, and would use the information.

• Almost 1,400 providers were trained and 7,000 children and youth were impacted by peer- reviewed Extension curricula. (Eight states reporting)

Did You Know?Did You Know?

• An additional 1,200 children and youth were served through the establishment of 90 new programs. (Two states reporting)

• About $5 million was leveraged by Extension to support programs. (Nine states reporting)

Did You Know?Did You Know?

• Almost 5,000 new children and youth were served in 54 new after-school programs that were established and managed by Extension. (Seven states reporting)

• More than 60 4-H clubs were started in after-school programs that served an additional 5,400

children and youth. (Eight states reporting)

ECI Levels of EvaluationECI Levels of Evaluation• Level 1 (single state or county)

– Recording training sessions

• Level 2 (single state or county)– Client satisfaction evaluations– Semi-annual report of outputs, activities,

and outcomes

• Level 3• Level 4• Level 5

Multi-state or county efforts

What Is the ECI Evaluation and What Is the ECI Evaluation and Reporting System?Reporting System?

• A web-based system for reporting information from early care and education, school-age care (after-school), and teen out-of-school programs

• Data entry is easy and fast, and the system performs all the analysis for you.

• The system provides instant professional reports of your results.

Level 1- Recording TrainingsLevel 1- Recording Trainings

A variety of information is collected on the session:– title, date, type of session– type of credit offered – training location, training hours– number of participants, intended target

audience– age groups served by participants in the

session

Level 2- Client Level 2- Client Satisfaction EvaluationsSatisfaction Evaluations

• There are 4 versions of evaluations:

– Provider/Staff– Families– Extension– Community

• You can enter evaluation data on the web and/or scanning services are available.

Semi-Annual ReportSemi-Annual Report• Currently there are 12 questions on which

to report program activities, outputs, and outcomes.

• You may select any or all of the questions on which to report. You are responsible for reporting data for each question.

• A semi-annual report can be generated twice a year from reported data.

What Are People Saying?What Are People Saying?• Federal agencies and organizations are very

interested in the information.

• Gives us critical information to articulate the breadth and depth of our work.

• Helps us position Extension for potential resources.

• Will give states the data needed for the next Plan of Work templates.Cathann Kress, Director, Youth Development, National 4-H

Headquarters, USDA

What Are People Saying?What Are People Saying?

“I love the ECI reporting and evaluation system. I use the data as a tool to improve the quality of the training I deliver.”

Jennifer Miller, County Extension Educator, Ashe County, North Carolina

What Are People Saying?What Are People Saying?

• Flexible reporting and evaluative system that captures and records staff training activities

• Web-based and is fairly easy to master.

Harry Mangle, Extension Educator, University of Connecticut Cooperative Extension

What Are People Saying?What Are People Saying?

• Data to report results to the J.C. Penney Afterschool Fund, John Deere, and other sponsors of 4-H Afterschool

• Critical for securing resources and positioning 4-H in the after-school market

Don Floyd, President and CEO, National 4-H Council

What Are People Saying?What Are People Saying? • Immediate & efficient “up to the minute” aggregated

county data

• State-wide system already in place but easy to merge and use

• Using data for a State 4-H Department Review to show growth and quality

• Program development and improvement and to secure funding

• Excited to be part of a national effort and to see how NC compares to other states

Rosa Andrews, Extension Associate and Coordinator, North Carolina 4-H School-Age Care Programs

What Are People Saying?What Are People Saying? • At State Level:

– Snapshot of programs and audiences reached by county educators

– Report program impacts, identify program needs, looks at strengths, needs, and audience diversity

– Explore other questions to ask

– Very useful with other partners, policymakers, and funders

• At County Level:

– Track programs, audiences, and effectiveness

– Assist with annual performance appraisals

Debbie Richardson, Child Development Asst. Specialist, Oklahoma Cooperative Extension Service, Oklahoma State University

Level 3 & 4 - Program QualityLevel 3 & 4 - Program Quality• Goal: To assess program quality before measuring child/youth outcomes

• National Afterschool Association (a.k.a. NSACA) Standards & Accreditation System

• School Age Care Environmental Rating Scale

(SACERS)

Level 5 – Research DesignLevel 5 – Research Design

• W.T. Grant Foundation proposal submitted

• State and Multi-State Research Projects

• Individual Research

Other ConnectionsOther Connections

• ES-237

• National Outcomes & Indicators by FCS

• Next POW Cycle

Other AlignmentsOther Alignments

• CSREES Strategic Plan

• CSREES Office of Planning and Accountability

• REEIS

• NAE4HA

ScoringScoring• (1) Inadequate

• A rating of 1 is the maximum score given: – If any of the indicators under 1 is

scored yes.

ScoringScoring• A rating of 2 is given:

– If no part of 1 is scored yes

and

– If half or more of the indicators under 3 are scored yes.

ScoringScoring• (3) Minimal and (5) Good

• A rating of 3 or 5 is given:– Only if all of the indicators are met.

• All indicators in 3 must be met before any higher rating may be given for an item.

ScoringScoring• A rating of 4 or 6 is given:

– When all the lower ratings are scored yes

and

– When half or more of the next higher indicators are scored yes.