Supplemental Educational Services Supplemental Educational Services in the State of North Carolina:in the State of North Carolina:

Evaluation Findings and ActivitiesEvaluation Findings and Activities

Steven M. Ross & Martha J AlbergCenter for Research in Educational

PolicyThe University of Memphis

http://www.memphis.edu/crep

Effectiveness (Student Achievement)

ServiceDelivery and Compliance

CustomerSatisfaction

ProviderSurvey

District CoordinatorSurvey

Principal/Site CoordinatorSurvey

Teacher Survey

Parent Survey

AdditionalTests

StateTests

Figure 1. Components of a Comprehensive SES/Evaluation Modeling PlanFigure 1. Components of a Comprehensive SES/Evaluation Modeling Plan

Overall Provider Assessment

SES ParticipationSES Participation

Percentages based upon approved providers and eligible districts.

2006-2007 2007-2008

Active Providers

82% 94%

Districts Served

68% 71%

Students Served

5,297 8,943

SES Survey Response RatesSES Survey Response Rates

Percentages based upon approved providers and eligible districts and schools.

Stakeholder 2006-07 2007-08Providers 36% 87%

District Coordinators

8% 38%

Principals/Site Coordinators

50% 33%

Teachers 49% 26%

Parents 71% 69%

Do providers communicate regularly Do providers communicate regularly with school personnel and parents?with school personnel and parents?

0%10%20%30%40%50%60%70%80%90%

100%

2006-07 2007-08

DistrictCoordinators

Principals/SiteCoordinators

Teachers

Parents

Are providers working with districts, Are providers working with districts, schools, and parents to develop schools, and parents to develop

instructional plans geared to students’ instructional plans geared to students’ needs?needs?

0%10%20%30%40%50%60%70%80%90%

100%

2006-07 2007-08

DistrictCoordinators

Principals/SiteCoordinators

Teachers

Parents

Providers

Are you satisfied with the Are you satisfied with the providers’ services?providers’ services?

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

2006-07 2007-08

DistrictCoordinatorsPrincipals/SiteCoordinatorsTeachers

Parents

Matched SES-Control Matched SES-Control Student DesignStudent Design

• The matched design is more rigorous and yields results that are more valid as compared to other available designs. As such, this design was chosen specifically for NC after great deliberation between CREP and DPI.

• Student-level NCEOG scores from the prior year are gathered for each SES student and matched control student.

• Control students are drawn from a pool of demographically similar students (prior achievement, ethnicity, gender) from the same schools.

• Control students are students who were eligible to receive SES, but were not served during the current year.

Does SES Raise Student Does SES Raise Student Achievement?Achievement?

2006-2007 READING

RESULTSDirectional Effects: % of

23Providers

35%

56%

9%

PositiveNegativeNo Effect

2007-2008 READING

RESULTSDirectional Effects: % of 26Providers

35%

57%

8%

Positive

Negative

No Effect

Does SES Raise Student Does SES Raise Student Achievement?Achievement?

2006-2007 MATHRESULTSDirectional Effects: % of

17Providers

29%

71%

0%

PositiveNegativeNo Effect

2007-2008 MATH RESULTSDirectional Effects: % of 28Providers

46%

50%

4%

PositiveNegative No Effect

ConsiderationsConsiderations

• Provider and teacher communication is vital and has been lacking in both evaluation years.

• Collaboration between all stakeholders is necessary to increase focus on the individual needs of each SES student.

• School personnel and providers could meet to discuss ways to adapt and integrate tutoring services with classroom activities.

• Consistent attendance at tutoring sessions by students is vital for SES to yield results.

ConsiderationsConsiderations

• One can reasonably expect with the limited amount of tutoring (20-40 hours) a child receives, limited impact will be seen on his or her state assessment results.

• The results from these studies can be utilized to require accountability and improvement by providers.

• Policies or criteria should be defined for determining what constitutes satisfactory provider performance.

• A classification system for providers should be clearly defined such as: full status, probation, etc.