21st Centers Program 2016-17 State Evaluation Report › Documents › K-12 › 21st Century... ·...

21st Century Community Learning Centers Program

2016-17 State Evaluation Report

April 2018

COMMONWEALTH OF PENNSYLVANIA DEPARTMENT OF EDUCATION

333 Market Street Harrisburg, PA 17126-0333

www.education.pa.gov

Commonwealth of Pennsylvania

Tom Wolf, Governor

Department of Education Pedro A. Rivera, Secretary

Office of Elementary and Secondary Education

Matthew S. Stem, Deputy Secretary

Bureau of School Support Sherri Smith, Ph.D., Director

Division of Student Services

Carmen M. Medina, Chief The Pennsylvania Department of Education (PDE) does not discriminate in its educational programs, activities, or employment practices, based on race, color, national origin, [sex] gender, sexual orientation, disability, age, religion, ancestry, union membership, gender identity or expression, AIDS or HIV status, or any other legally protected category. Announcement of this policy is in accordance with State Law including the Pennsylvania Human Relations Act and with Federal law, including Title VI and Title VII of the Civil Rights Act of 1964, Title IX of the Education Amendments of 1972, Section 504 of the Rehabilitation Act of 1973, the Age Discrimination in Employment Act of 1967, and the Americans with Disabilities Act of 1990. The following persons have been designated to handle inquiries regarding the Pennsylvania Department of Education’s nondiscrimination policies: For Inquiries Concerning Nondiscrimination in Employment: Pennsylvania Department of Education Equal Employment Opportunity Representative Bureau of Human Resources 333 Market Street, 10th Floor, Harrisburg, PA 17126-0333 Voice Telephone: (717) 787-4417, Fax: (717) 783-9348 For Inquiries Concerning Nondiscrimination in All Other Pennsylvania Department of Education Programs and Activities: Pennsylvania Department of Education School Services Unit Director 333 Market Street, 5th Floor, Harrisburg, PA 17126-0333 Voice Telephone: (717) 783-3750, Fax: (717) 783-6802 If you have any questions about this publication or for additional copies, contact: Pennsylvania Department of Education Voice: (717) 787-8913 Bureau of Teaching and Learning Fax: (717) 783-6617 333 Market Street, 5th Floor TTY: (717) 783-8445 Harrisburg, PA 17126-0333 www.education.pa.gov All Media Requests/Inquiries: Contact the Office of Press & Communications at (717) 783-9802.

Table of Contents

Executive Summary .................................................................................... 1 Program Highlights...................................................................................... 9 Introduction ............................................................................................... 11

Program Description ........................................................................ 11 Reporting Venues ............................................................................ 12 Evaluation Design ............................................................................ 13 How To Use This Report .................................................................. 14

Findings .................................................................................................... 16 Grantee Characteristics ................................................................... 16 Program Implementation ................................................................. 17 Program Participation ..................................................................... 26 Student Outcomes ........................................................................... 28

2016-17 Government Performance and Results Act ................................ 71 Reflections, Implications, and Recommendations for Improvement ......... 73

Pennsylvania 21st Century Community Learning Centers 1 2016-17 State Evaluation Report Originated April 26, 2018

Executive Summary The 21st Century Community Learning Centers program provides federal funding for the establishment of community learning centers that provide academic and enrichment opportunities for children; particularly students who attend high-poverty and low-performing schools, to meet state and local standards in core academic subjects, through a broad array of activities that complement their regular academic programs. In addition to academics, centers are encouraged to offer participants a broad range of other services and programs, such as art, music, recreation activities, character education, career and technical training, drug and violence prevention programming, and technology education. Educational services for families of participating students, such as literacy instruction, computer training, or cultural enrichment, must also be included. The 2016-17 program year included 134 grantees in three funding cycles, each called a cohort: Cohort 6A included 27 grantees,1 Cohort 7 included 62 grantees, and Cohort 8 included 45 grantees. Grantees were mainly schools, districts, or charter schools (43 percent) or community-based/nonprofit organizations (35 percent). This varied somewhat by cohort, with Cohort 7 having a higher concentration of school-based grantees. EVALUATION DESIGN The evaluation of the 2016-17 program year of 21st Century programs in Pennsylvania includes information about the programs operated under the Cohort 6A, Cohort 7, and Cohort 8 funding cycles. The timing of awards dictates what grantees report annually for evaluation. The state evaluation of Pennsylvania’s 21st Century program examined three performance measures focused on students’ positive academic, social, and behavioral changes. Data sources included the new federal 21APR system, Pennsylvania Implementation Survey, PA Operations Spreadsheet, PA De-identified Student Data Spreadsheet, and other data from PDE and the Center for Schools and Communities, which is Pennsylvania’s contractor for 21st Century technical assistance. The Pennsylvania Department of Education (PDE) contracted with the Allegheny Intermediate Unit to conduct a comprehensive external evaluation of the 21st Century Community Learning Centers program to fulfill federal requirements under Title IV, Part B of the Elementary and Secondary Education Act, as amended, Sections 4202 (C) and 4203 (A) and Section H-5 of the 21st Century Community Learning Centers Non-Regulatory Guidance. 1 Cohort 6A grants ended September 30, 2016, so these grantees only operated a small portion of the year.


PROGRAM IMPLEMENTATION Grantees could operate programs during the summer of 2016,2 school year 2016-17, or both. Grantees were required to operate a minimum of 36 school year weeks, for 12-15 hours per week afterschool, unless approved to operate otherwise. Grantees operated programs out of 415 centers. Summer-operating centers (253) operated between 10 and 45 hours per week, with the bulk of these hours occurring during the day on weekdays; five centers operated on weekday evenings and four operated on summer weekends. Center operations averaged 22 hours per week during the summer with the most frequent operations volume being 12 hours per week. Centers operated between three and five days per week. Nearly half of the centers (125) operated five days per week. Centers offered these programs between one and 11 weeks per center; 235 centers (93 percent) operated for six or more weeks. During the school year, programming occurred through 364 centers (88 percent of all centers). School year centers operated between one and six days per week, with an average of four days per week. Of the 364 centers operating during the school year, 356 (98 percent) operated four or more days per week. School year centers operated between 2.75 and 26 total hours per week, with an average of 13 hours per week. The minimum requirement for hours per week during the school year was 12 hours; 362 centers (98 percent) met or exceeded this requirement. Centers operated between three and 42 weeks during the school year 2016-17, with 296 centers (81 percent of school year centers) operating for 36 weeks or more, which was the expected level of implementation for a full year’s program, though this total weeks target was based on a contract year, and not on the standard reporting year. Of the 339 Cohort 7 and Cohort 8 centers – those eligible to operate for the full year – 295 (87 percent) operated for 36 or more weeks during the school year. Based on hours per week and weeks in operation, evaluators estimated that grantees offered a combined total of 34,924 hours or programming during the summer and 161,239 hours during the school year. Program guidance included a list of allowable activities. In the PA Implementation Survey, grantees indicated which program areas they addressed from a list of 15 areas outlined in Pennsylvania’s program guidance. Of the 130 grantees completing the Implementation Survey, the largest percentages indicated they offered math and science education activities, or STEM (97 percent), recreational activities (94 percent), and/or arts and music education activities (92 percent). These were the top three categories in the prior year as well, with the same or nearly the same percentages. 2 Only contracts ending before the end of the program year were permitted to only operate during the school year. Generally, grantees were required to operate during both summer and school year or school year only, depending on their contract.


Grantees indicated in the PA Implementation Survey strategies they used to identify, enroll, and serve students. Grantees could select from a list of strategies or share their own and they could select all strategies that applied to them. The largest portion of grantees (94 percent) used teacher or school recommendation to identify students to enroll, followed by parent referral (80 percent of grantees), among others. In addition to examining implementation and operations of 21st Century programs, the PA Implementation Survey asked grantees to indicate how they collaborated with students’ schools. Grantees collaborated in multiple ways, with 95 percent of grantees selecting ongoing communication with school administrators and/or school day teachers. Many grantees (88 percent) reported that school day teachers also served as program staff, providing a direct link between school and the 21st Century program. In the PA Implementation Survey, grantees were asked to indicate the frequency with which they implemented activities relevant to key content areas within a typical program week. Reading and math activities were most frequently indicated as daily activities; 80 percent of grantees indicated daily reading or literacy activities and 70 percent of grantees indicated they had daily math activities in a typical program week. Science, social studies, and other content areas were implemented less frequently. Behavior and promotion of school attendance, however, were implemented almost as frequently as math activities. Programs were required to serve parents and family members of participating students. In the PA Implementation Survey, grantees could indicate the types of parent or family activities offered from a list of options that included adult English as a Second Language services; adult education opportunities and/or GED classes; career/job training; computer/ technology training; cultural events; family literacy nights; health and wellness; open house events; parent/center staff meetings; parenting skills classes; parent training on how to help their children with schoolwork; parent training on post-secondary options and planning; parent reinforcement of the importance of school and education; parent volunteering at the program; and structured family recreation. Grantees could describe other activity types as well. Grantees could select all activity types that applied to their program for 2016-17. A majority of grantees selected open house activities (88 percent of grantees), followed distantly by family literacy nights (58 percent of grantees) and health, nutrition, fitness, or wellness activities (46 percent of grantees), with other options selected to a lesser extent. When asked to specify participation, 125 grantees shared a count of adult family members of participating students who participated in at least one activity of any type during this program year. Grantee counts ranged from four adults to 1,395, with an average of 139 adults, with 37 grantees having a count that exceeded the average and 88 having a count of less than the average. Five grantees – all Cohort 6A programs operating only a short time – reported having no adult family member participation.


STUDENT OUTCOMES All grantees operating during the school year were required to report reading and math report card grades, cross-year reading and math state assessment results, and teacher-reported behavior and homework outcomes in the federal reporting system. State-level reporting included further detail about the federal student outcomes categories as well as results for school attendance, school behavior and discipline, and credit recovery, for grantees identifying such measures in their applications. Student outcomes reporting was only required for regularly attending students, which means they attended the program for 30 or more days over the course of the program year, which included summer 2016 and school year 2016-17. Grantees reported having a total of 15,542 regular attendees. Students may have outcomes data under any number of the provided outcomes areas and/or data source categories depending on the timing of their participation, grade level, and other reasons. State Reading and Math Assessments Evaluators examined current year (2017) snapshot state assessment results for students having data as well as progress results for students having both prior year (2016) and current year (2017) state assessment data. A total of 92 grantees reported snapshot state assessment results for 2017 for grades 3-8 and 11 (87 percent of grantees that reported having regular attendees in state assessment grades3); 84 of these grantees also reported 2016 state assessment results for examining student improvement status. For 2017 reading state assessments, 8,365 students were included in analysis4 (approximately 76 percent of regular attendees in grades eligible to take the 2017 state assessments). Of these students, 6 percent scored in the advanced performance level, 29 percent scored proficient, 45 percent scored basic, and 20 percent scored in the below basic level. This is an improvement over the prior year (2015-16), where 5 percent scored in the advanced performance level, 27 percent scored proficient, 43 percent scored basic, and 25 percent scored in the below basic level. A total of 5,900 students had two consecutive years of state reading assessment data, which is 67 percent of regular attendees in grades where two consecutive years of state assessment data are most likely (grades 4-8). Of these students, 22 percent made a positive movement of one or more levels from 2016 to 2017, which is an improvement 3 One of these grantees was able to provide individual student assessment results, but was not able to connect it with other data elements because of how the school district provided data. As such, a disaggregation by attendance category was not possible for this program. This grantee’s results are included in the overall state and cohort results but not in any disaggregations by program attendance. 4 One Cohort 6A grantee reported state assessment results for its regular attendees in grade levels that take state assessments. Because of the small number of students and being the only grantee to report results, Cohort 6A results are not broken out in this report; however, Cohort 6A data was included in overall state-level analysis.


over the prior year, in which 19 percent showed gains. Furthermore, 12 percent of students who scored at the basic or below basic levels in 2016 (and had both years of data) improved to proficient or advanced in 2017. More than half of students (61 percent) scored in the same level in 2016 and 2017, 14 percent declined, and 3 percent of students did not need to improve category, meaning they scored at the advanced level on both administrations of the assessment.

For 2017 state math assessments, 92 grantees reported 2017 data (87 percent of grantees reporting students in state assessment grades), and included results for 8,516 students (78 percent of regular attendees in grades eligible to take the state assessments). Of these students, 5 percent scored in the advanced performance level, 15 percent scored proficient, 29 percent scored basic, and 51 percent scored in the below basic level, which is almost exactly the same as the prior year, where 5 percent scored in the advanced performance level, 15 percent scored proficient, 28 percent scored basic, and 52 percent scored in the below basic level.

A total of 5,968 students had two consecutive years of state math assessment data, which is approximately 68 percent of regular attendees in grades likely to have two years of state assessment data (grades 4-8). Of these students, 13 percent made a positive movement of one or more levels from 2016 to 2017, 3 percent scored in the advanced level on both administrations, 68 percent scored in the same performance level in both years, and 17 percent declined, which is similar to the prior year. Furthermore, 5 percent of students who scored at the basic or below basic levels in 2016 (and had data for both years) improved to proficient or advanced in 2017. Reading and Math Report Card Results A total of 12,239 students had reading report card data, which is 80 percent of school year regular attendees for whom outcomes data were reported. A total of 103 grantees (96 percent of applicable grantees) reported usable reading report card grade data. Of students having reading report card grade data that could be analyzed, 29 percent improved from fall to spring. The largest percentage, at 40 percent, showed no change, meaning they earned the same grade for both the first and last grading periods. Results also indicated that 24 percent declined from fall to spring and 7 percent did not need to improve their grade and maintained that grade.

For math report card grades, a total of 12,379 students had math report card, which is 82 percent of school year regular attendees for whom outcomes data were reported. A total of 103 grantees5 (96 percent of applicable grantees) reported usable math report card grade data.

5 One of these grantees was able to provide individual report card grade data, but was not able to connect it with other data elements because of how the school district provided data. As such, a disaggregation by attendance category was not possible. This grantee’s students’ results are included in the overall state and cohort results but not in any disaggregations by program attendance level.


Of students having math report card grade data that could be analyzed, 29 percent improved from fall to spring. The largest percentage, at 38 percent, showed no change, meaning they earned the same grade for both the first and last grading periods. Results also indicated that 26 percent declined from fall to spring and 7 percent did not need to improve their grade and maintained that grade.

21st Century Teacher Survey The 21st Century Teacher Survey included an indicator for teachers to report student change in academics, classroom performance, and behavior. This determination was to be made by the classroom teacher about each regularly attending student participating during the school year based on his/her professional opinion of the student’s classroom performance. Data were available for between 66 and 68 percent of school year regular attendees, depending on the individual indicator. For the academic performance teacher survey indicator, 56 percent of students improved, 17 percent of students included in analysis were reported as not needing to improve, 18 percent showed no change, and 9 percent declined. For the indicator of satisfactory homework completion, 51 percent of students showed improvement, 23 percent of students did not have a need to improve, 17 percent did not change, and 8 percent declined. For the indicator of timely homework completion, 47 percent of students improved, 27 percent did not need to improve, 18 percent showed no change, and 8 percent declined. Regular class attendance results indicated that class attendance was not an area of need for many students. Teachers reported that 45 percent of students did not need to improve their class attendance. Percentages improving (23 percent) or experiencing no change (27 percent) were similar and a small percentage of students declined (5 percent). Results for volunteering in class show the same percentages of students who improved or showed no change (37 percent). Nearly a quarter of students did not need to improve (23 percent). Three percent of students declined. For class participation, 51 percent of students showed improvement, 21 percent did not need to improve, 23 percent showed no change, and 5 percent declined. In the area of class attentiveness, 45 percent of students improved, 25 percent did not need to improve, 21 percent showed no change, and 9 percent declined.

For the 21st Century Teacher Survey indicator concerning class behavior, 35 percent of students improved, followed closely by those not needing to improve in this area with 34 percent. Twenty-two percent of students exhibited no change and 9 percent of students declined.


Of students included in teacher survey results for coming to school motivated to learn, 40 percent improved. Percentages of students showing no change (27 percent) and not needing to improve (26 percent) were similar to each other. Six percent declined.

Approximately a third of students improved in the area of getting along well with other students, while 37 percent did not need to improve. Almost a quarter of students were reported as showing no change (23 percent) and 6 percent declined.

School Behavior/Discipline and Attendance Seventy-two grantees (54 percent of grantees) reported school behavior and discipline results in the de-identified student data spreadsheet submission. Grantees were not obligated to report these results if school behavior and discipline indicators were not part of their application. Student behavior and discipline results in the four general change categories were available for 8,682 students (56 percent of regular attendees). Overall results indicated that more than half of students (59 percent) did not need to improve in the area of school behavior and discipline. The remaining categories showed similar results, 14 percent improved, 13 percent showed no change, and 13 percent declined according to grantee-defined change. A total of 77 grantees (58 percent of grantees) reported school attendance results for 9,457 students, 61 percent of regular attendees, and these results showed equal percentages of students improving and not needing to improve (28 percent), while a third declined and 10 percent showed no change. High School Credit Recovery Thirty-seven grantees reported in the PA Implementation Survey that they had a credit or course recovery component in which students participated during the 2016-17 program year. More than 75 percent of grantees indicated that credit/course recovery was delivered through a blend of face-to-face instructions and computer-based instruction; 14 percent indicated delivery was typically face-to-face, and 8 percent indicated it was typically computer based. A total of 30 grantees reported on high school student credit recovery outcomes for 2016-17. These grantees reported that a total of 1,417 high school students participated in credit recovery activities during this program year, of which 1,043 (74 percent) recovered one or more credits. By grantee, numbers of students recovering credits ranged from one to 200 students recovering one or more credits. Of the 1,043 students recovering one or more credits, 260 were regular attendees (25 percent) and 783 students (75 percent) attended fewer than 30 days. These 1,043 students recovered a total of 1,627 credits during the 2016-17 program year: 466 literacy credits, 490 math credits, and 671 other credits.


Of the 1,043 students recovering one or more credits, 742 recovered credits in one content area reporting category only (literacy, math, other), 246 students recovered credits in two content area categories, and 55 students recovered credits in all three content area categories. CONCLUSION Pennsylvania 21st Century programs provided a variety of academic and enrichment services to students and their families intended to influence student outcomes. In most areas, considerable numbers of students showed improvement in one or more academic and/or behavioral elements, even though percentages may not be high in all areas. Results also indicate considerable needs still exist. Historical presence results may indicate that multiple years of 21st Century participation may contribute to positive outcomes. Evaluators will continue to collect this information to determine the extent to which this may be a trend. Based on evaluation findings, evaluators recommend that grantees collaborate with their local evaluator to examine program findings in order to identify areas of need and strength and make decisions designed to promote continuous program improvement and positive student outcomes. Furthermore, evaluators recommend that the state team examine areas of grantee need in order to design and offer training, professional development, resources, and support designed to increase grantee capacity to implement effective and efficient programs.


Program Highlights In this section, evaluators present several program highlights that showcase program success and progress. Program areas for improvement are addressed in the Reflections, Implications, and Recommendations section at the end of this report.

• Grantees served nearly 40,000 students during the summer 2016 and school year 2016-17 program year. This means that during this year 21st Century programs served approximately 2 percent of Pennsylvania’s public school K-12 population, making it one of the larger supplemental programs in Pennsylvania in terms of student reach.

• Grantees offered an estimated 34,924 hours of summer 2016 programming and 161,239 hours of school year 2016-17 programming.

• More than 90 percent of grantees maintain ongoing communication with school administrators and/or classroom teachers; 88 percent of grantees employ school-day teachers as program staff, providing a direct connection between the school day and the 21st Century program.

• Nearly all grantees (97 percent) reported offering STEM-related6 activities as part of their 2016-17 programs.

• Of students needing to improve and included in analysis, 31 percent improved their reading report card grade and 31 percent improved their math report card grade from fall to spring. Of the 12,008 students who had both reading and math report card grade data that could be analyzed, 11 percent improved their report card grades in both content areas.

• Historical presence analysis indicates that multiple years of 21st Century participation may contribute to positive student outcomes.

• 1,043 students recovered a total of 1,627 credits during the 2016-17 program year: 466 literacy credits, 490 math credits, and 671 other credits.

• The 21st Century Teacher Survey results showed larger percentages of students improving than some of the more objective measures of student achievement. However, classroom teachers may be able to detect small improvements in individual students before they show up on assessments or other measures. These findings may indicate that student improvements may be observed in the future. If students who did not need to improve are excluded from the analysis, eight of the 10 teacher survey indicators showed more than half of regular attendees improving according to 21st Century Teacher Survey data:

o 68 percent of regular attendees with teacher survey data improved their academic performance

o 67 percent of regular attendees with teacher survey data were reported as improving their homework completion to their teacher’s satisfaction

o 64 percent of regular attendees with teacher survey data improved their class participation

6 STEM refers to science, technology, engineering, and mathematics activities.


o 64 percent of regular attendees with teacher survey data improved in the area of timely homework completion

o 60 percent of regular attendees with teacher survey data improved their class attentiveness

o 54 percent of regular attendees with teacher survey data improved their motivation to learn

o 54 percent of regular attendees with teacher survey data improved in the area of getting along with other students

o 53 percent of regular attendees with teacher survey data improved their class behavior


Introduction PROGRAM DESCRIPTION7 The 21st Century Community Learning Centers program provides federal funding for the establishment of community learning centers that offer academic and enrichment opportunities to children, particularly students who attend high-poverty and low-performing schools, to meet state and local standards in core academic subjects through a broad array of activities that can complement their regular academic programs. Literacy and other educational services to the families of participating children must also be provided. The 21st Century Community Learning Centers (21st Century) program is authorized under Title IV, Part B of the Elementary and Secondary Education Act (P.L. 107-110), as amended by the No Child Left Behind Act of 2001. Pennsylvania’s primary goal for its 21st Century program is to assist youth to meet state standards for core academic subjects by providing them with academic and enrichment opportunities. In addition to academics, centers are encouraged to offer participants a broad array of other services and programs during nonschool hours, such as art, music, recreation activities, character education, career and technical training, drug and violence prevention programming, and technology education. Educational services for families of participating students, such as literacy instruction, computer training, or cultural enrichment, must also be included.8 Federal law requires that all 21st Century program sites provide academic enrichment activities and parental involvement activities. Programs are encouraged to use innovative instructional strategies, coordinate academics with local curricula and assessments, and use assessment data to inform instruction and evaluate results. Academics are to involve more than just helping participants with homework and should not just repeat school day activities. Pennsylvania’s 21st Century program encourages active youth and family participation to ensure that both have decision-making roles in the creation, operation, and evaluation of every 21st Century program in Pennsylvania. School and community collaboration is another key in meeting the academic, social, physical, and emotional needs of children and families. Programs are to offer quarterly open house meetings and maintain an open-door policy where adult family members feel welcome and are encouraged to drop in. All activities are to be based on rigorous scientific research and the Pennsylvania Department of Education (PDE) provides “principles of effectiveness” to guide programs

7 Program information and requirements were adapted from 21st Century application and program guidance documentation. 8 The majority of 21st Century activities are to take place during nonschool hours. However, activities for adult family members and pre-kindergarten students may take place during school hours if these times are the most appropriate to these constituents.


in identifying and implementing programs that enhance student learning. Activities must address the needs of local schools and communities and be continuously evaluated at the local level. Grantee Eligibility Federal law mandates, per section 4203 (a)(3), that any public or private organization may apply for funding if it proposes to serve students who primarily attend schools eligible for schoolwide programs under Title I section 1114, or schools that serve a high percentage of students (at least 40 percent) from low-income families and the families of such students. Non-school applicant agencies must collaborate with local education agencies when applying for funds. Participant Eligibility Eligible participants are public and private/nonpublic school students in pre-kindergarten through 12th grade. Programs are to target the ages and grades deemed to be at greatest risk and those students who are academically below proficiency. At-risk behaviors might include poor school performance, poor school attendance, drug or alcohol abuse, criminal activity, or any other indicators judged by the applicant as placing the child at higher risk and greater need. Adult family members of students participating in the community learning center are to be served through educational activities that are appropriate for adults. REPORTING VENUES 21APR 21st Century is a federally-authorized program operating across the nation. One of the requirements of 21st Century grantees is to complete program and outcomes reporting in the federal 21APR system, where “APR” stands for “Annual Performance Report.” The 2016-17 year was the second year that the 21APR system operated. The 21APR system collects information on grantees and their centers, program staffing information, activities, program attendance, student characteristics, and student outcomes based on federal measures. Student outcome measures included state reading and math assessment gains, reading and math report card grades, and teacher survey responses. However, at this time, no data or results entered by grantees are exportable for state use. State Reporting Venues for state reporting changed for the 2015-16 year in response to changes to reporting at the federal level and continued for the 2016-17 program reports. State reporting took three forms: the PA Implementation Survey, the Center Operations


Spreadsheet, and the De-identified Student Data Spreadsheet. State reporting forms provided grantees with a method of reporting information that the 21st Century Request for Proposals included but that is not addressed in the federal annual performance report. The Allegheny Intermediate Unit, the contracted evaluator for Pennsylvania’s 21st Century program, constructed and implemented the state reporting forms. All grantees were required to complete these elements based on how they applied to the programs’ implementation and funding cycles. Evaluators compiled the data from each source for all grantees and analyzed it overall, by cohort, and by grantee. Other Data Sources Additional information was collected about grantees and their programs PDE and the Center for Schools and Communities, PDE’s subcontractor for 21st Century technical assistance. EVALUATION DESIGN The evaluation of the 2016-17 program year of 21st Century programs in Pennsylvania includes information about the programs operated under the Cohort 6A, Cohort 7, and Cohort 8 funding cycles. The 2016-17 program year included 134 grantees: Cohort 6A included 27 grantees, Cohort 7 included 62 grantees, and Cohort 8 included 45 grantees. Cohort 6A grants ended September 30, 2016, making them eligible to operate during summer 2016 through the end of September 2016 only. Cohort 7 and Cohort 8 grantees were eligible to operate the full program year, which included summer 2016 and school year 2016-17. The evaluation of Pennsylvania’s 21st Century program examined three performance measures, within which grantees established their own performance indicators. The measures included:

1. Participants in 21st Century programs will demonstrate educational and social benefits and exhibit positive behavioral changes;

2. Increasing percentages of students regularly participating in the program will meet or exceed state and local academic achievement standards in reading and math; and

3. Students participating in the program will show improvement in the performance measures of school attendance, classroom performance, and reduced disciplinary referrals.

PDE contracted with the Allegheny Intermediate Unit to conduct a comprehensive evaluation of the 21st Century Community Learning Centers program to fulfill federal requirements under Title IV, Part B of the Elementary and Secondary Education Act, as amended, Sections 4202 (C) and 4203 (A) and Section H-5 of the 21st Century Community Learning Centers Non-Regulatory Guidance:


States must conduct a comprehensive evaluation (directly, or through a grant or contract) of the effectiveness of programs and activities provided with 21st Century funds. In their applications to the Department, States are required to describe the performance indicators and performance measures they will use to evaluate local programs. States must also monitor the periodic evaluations of local programs and must disseminate the results of these evaluations to the public.

HOW TO USE THIS REPORT The primary audiences for this report include PDE, technical assistance providers, and Pennsylvania 21st Century grantees, though the results can be useful for other groups. The evaluation of the 2016-17 program year focused on the three performance measures outlined previously. Additionally, grantees provided implementation and contextual data to support and explain program results. Findings and information are provided overall for the state (all grantees combined) and for each cohort as appropriate and available. Throughout this report, the narrative explanation precedes the graphical representation of results. Throughout this report, for ease of reading, percentages have been rounded, which may result in totals not equal to 100 percent. Additionally, in tables or graphs where “0%” appears, the reader should note that these represent values of less than 1 percent expressed as a rounded value. Instances of zero percent where the item truly represents zero instances or individuals have been removed from graphs to make them easier to read. Likewise, where blank cells appear in data tables, the value is zero. Some graphs contained in this report include both the number of instances (in a data table) along with an illustration of the proportional relationship of those figures. This type of graph is typically used when the categories are mutually exclusive and individual category percentages equal 100 percent. Other graphs only include the percentage of instances. This type of graph is typically used when multiple categories can apply to a single item (grantees could select all items that applied). Data tables that include percentages are also used in cases where the percentage is a more accurate representation of the program or the population being examined. The type of illustration included will indicate to the reader the most appropriate way to examine the findings. Some sections provide ranges (minimum and maximum) of results in order to demonstrate the variability of grantee programs and outcomes, as well as an average. An average, or mean, is a measure of central tendency where the result is calculated by adding two or more values together and then dividing the resulting total by the number of values included. It is important for readers to note that not all grantees reported in all areas. In some cases, grantees were not required to report in all areas, as their applications and


program operation dictated the required reporting components. In other areas, grantees may have had no students or programs to which a particular data element applied or they failed to report. The number of grantees reporting in each area is provided to minimize confusion. Care should be taken in making comparisons across cohorts, as each has differing populations, programs, and student counts, and grantees had different approved program applications. Further, some of each cohort’s program requirements were slightly different to accommodate changes in state priorities and federal guidance. This report is not an evaluation of individual grantees, but rather an overall examination of the programs implemented during the 2016-17 program year, which includes summer 2016 and school year 2016-17. Grantees are required to have an external local evaluator that should be providing examination of the individual grantee’s program. Grantees’ local evaluation reports are to be submitted to the state by the end of October each year. This report includes detailed explanations of the program’s implementation and outcomes as addressed throughout the findings section. In addition, this report includes sections that present information contained in findings in the context of the Government Performance and Results Act (GPRA) measures. The report concludes with evaluator reflections, implications, and recommendations for improvement. It is important to remember that because of the nature of 21st Century programs, the students these programs serve, current information collection methods, and other resources available to schools, organizations, communities, and students, it is not possible to attribute student outcomes solely to this program’s efforts. The findings provided within this report should be used to guide program management and assist PDE and the contracted technical assistance team from the Center for Schools and Communities in providing assistance to grantees in order to improve implementation and outcomes.


Findings The program findings shared in this report include information reported by grantees and state-level program staff about the 2016-17 program year, which includes summer 2016 and school year 2016-17. The various reporting venues are explained in the prior section of this report. GRANTEE CHARACTERISTICS The 2016-17 program year included 134 grantees in three funding cycles (cohorts). Grantees were mainly schools, districts, or charter schools (43 percent) or community-based/nonprofit organizations (35 percent). This varied somewhat by cohort, with Cohort 7 having a higher concentration of school-based grantees.

Figure 1.

While the graph above shows grantees’ organization type, this type is only indicative of the organization having fiscal and contractual responsibility for the program. Each grantee operated programming out of one or more centers (locations), which may be a different type than the grantee organization. For example, a community organization may operate its program in school buildings and a school district may operate its program in a community organization’s facility. Each grantee was permitted to operate its program in whatever manner was described in its approved grant application based on the needs of the population to be served. Grantees operated programs out of 415 centers (64 Cohort 6A centers, 226 Cohort 7 centers, 125 Cohort 8 centers). Grantees operated between one and 11 centers per


grantee, with an average of three centers; however, the mode (most frequent value) was one center. PROGRAM IMPLEMENTATION While the concept of 21st Century programs is to provide out-of-school-time programs that offer students supplemental academic and enrichment activities and there are some operational requirements, the 21st Century grant affords grantees a good deal of program design flexibility. Implementation information about student and adult/family activities and staffing were collected in 21APR, but not available for state analysis as they have been in the past. Student demographics were also not available because of the described changes in federal reporting and data availability. Operations Grantees could operate programs during the summer of 2016,9 school year 2016-17, or both. Specific date ranges were not prescribed to allow for the local variance of school year start or end dates. Grantees were required to operate a minimum of 36 school year weeks, for 12-15 hours per week afterschool, unless approved to operate otherwise. Grantees reported operations details in the Center Operations Spreadsheet, which they submitted to the state evaluation team in fall 2017. Grantees operated programs out of 415 centers, as described previously. Summer-operating centers (253) operated between 10 and 45 hours per week, with the bulk of these hours occurring during the day on weekdays; five centers operated on weekday evenings and four operated on summer weekends. Center operations averaged 22 hours per week during the summer with the most frequent operations volume being 12 hours per week. Centers operated between three and five days per week. Nearly half of the centers (125) operated five days per week. Centers offered these programs between one and 11 weeks per center; 235 centers (93 percent) operated for six or more weeks. During the school year, programming occurred through 364 centers (88 percent of all centers). School year centers operated between one and six days per week, with an average of four days per week. Of the 364 centers operating during the school year, 356 (98 percent) operated four or more days per week. School year centers operated between 2.75 and 26 total hours per week, with an average of 13 hours per week. The minimum requirement for hours per week during the school year was 12 hours; 362 centers (98 percent) met or exceeded this requirement. Of the five centers that did not 9 Only contracts ending before the end of the program year were permitted to only operate during the school year. Generally, grantees were required to operate during both summer and school year or school year only, depending on their contract.


meet this requirement, operations ranged from 2.75 hours to 9 hours per week. One was a Cohort 6A program that only operated four weeks before the grant contract ended, during which the grantee operated a SAT/ACT preparation program. The remaining centers were operated by grantees that had other centers that had other centers that did meet the 12 hour requirement. Centers operated between three and 42 weeks during the school year 2016-17, with 296 centers (81 percent of school year centers) operating for 36 weeks or more, which was the expected level of implementation for a full year’s program, though this total weeks target was based on a contract year, and not on the standard reporting year. Of the 339 Cohort 7 and Cohort 8 centers – those eligible to operate for the full year – 295 (87 percent) operated for 36 or more weeks during the school year. Based on hours per week and weeks in operation, evaluators estimated that grantees offered a combined total of 34,924 hours or programming during the summer and 161,239 hours during the school year. Program Design Program guidance included a list of allowable activities. In the PA Implementation Survey, grantees indicated which program areas they addressed from a list of 15 areas outlined in Pennsylvania’s program guidance. Of the 130 grantees completing the Implementation Survey, the largest percentages indicated they offered math and science education activities, or STEM (97 percent), recreational activities (94 percent), and/or arts and music education activities (92 percent). These were the top three categories in the prior year as well, with the same or nearly the same percentages. Categories offered the least included expanded library service hours (24 percent of grantees), counseling programs (28 percent of grantees), and/or programs that provide assistance to students who have been truant, suspended, or expelled to allow them to improve their academic achievement (29 percent of grantees). Grantees indicated in the PA Implementation Survey strategies they used to identify, enroll, and serve students. Grantees could select from a list of strategies or share their own and they could select all strategies that applied to them. The largest portion of grantees (94 percent) used teacher or school recommendation to identify students to enroll, followed by parent referral (80 percent of grantees), among others. Identification and recruitment challenges grantees reported included competition with other programs/activities (52 percent of 130 grantees completing the Implementation Survey), student responsibilities at home or after-school jobs (48 percent of grantees), parent involvement and awareness (45 percent of grantees), and parent commitment to consistent attendance (45 percent of grantees). Fifteen percent of grantees indicated that they did not experience or were not aware of any such challenges. Grantees were able to indicate the strategies or information they used to identify students’ needs. From a list of eight options, a majority of grantees reported using


teacher or school recommendation (95 percent of grantees) followed by observation (85 percent of grantees), and report card grades (84 percent) as the three most-selected options. Five percent of grantees indicated that they did not formally identify individual student needs. Results were similar across cohorts. As a reminder, 21st Century programs are intended to support students in identified high-needs schools. As such, it is likely safe to assume that the students in these schools, on the whole, have needs to be addressed. Grantees may design programming based on whole-school or whole-district information rather than individual student participant information. Grantees reported in the PA Implementation Survey on general strategies they used related to improving student behavior and school attendance, which were included in Pennsylvania’s program guidance, and consequently the annual state reporting, but these elements were not explicitly part of federal reporting beyond teacher feedback. These strategies may represent specific activities the grantee offered or may be infused into activities that are reported in 21APR, but not explicitly addressed in that system. To influence positive student behavior at school (and/or prevention/reduction of disciplinary incidents) and improved school attendance, grantees predominantly indicated using communication with schools, teachers, administrators, or parents; 96 percent or more of grantees selected these options for student behavior and more than 85 percent of grantees selected such options for influencing student attendance at school. Fewer than 10 grantees for each item indicated that no specific strategies were used or that school attendance and discipline were not a focus of their programs. In addition to examining implementation and operations of 21st Century programs, the PA Implementation Survey asked grantees to indicate how they collaborated with students’ schools. Grantees collaborated in multiple ways, with 95 percent of grantees selecting ongoing communication with school administrators and/or school day teachers. Many grantees (88 percent) reported that school day teachers also served as program staff, providing a direct link between school and the 21st Century program. In the PA Implementation Survey, grantees were asked to indicate the frequency with which they implemented activities relevant to key content areas within a typical program week. Reading and math activities were most frequently indicated as daily activities; 80 percent of grantees indicated daily reading or literacy activities and 70 percent of grantees indicated they had daily math activities in a typical program week. Science, social studies, and other content areas were implemented less frequently. Behavior and promotion of school attendance, however, were implemented almost as frequently as math activities.


Figure 2.

Adult Family Member Activities Programs were required to serve parents and family members of participating students. In the PA Implementation Survey, grantees could indicate the types of parent or family activities offered from a list of options that included adult English as a Second Language services; adult education opportunities and/or GED classes; career/job training; computer/ technology training; cultural events; family literacy nights; health and wellness; open house events; parent/center staff meetings; parenting skills classes; parent training on how to help their children with schoolwork; parent training on post-secondary options and planning; parent reinforcement of the importance of school and education; parent volunteering at the program; and structured family recreation. Grantees could describe other activity types as well. Grantees could select all activity types that applied to their program for 2016-17. A majority of grantees selected open house activities (88 percent of grantees), followed distantly by family literacy nights (58 percent of grantees) and health, nutrition, fitness, or wellness activities (46 percent of grantees), with other options selected to a lesser extent. Grantees also reported how they communicate with parents, students, and the community. Grantees most often indicated open house and family nights as methods of sharing information (95 percent of grantees), followed by fliers, promotional materials, or newsletters (92 percent of grantees), and informal feedback and communications (89 percent of grantees), among other formal and informal methods. When asked to specify participation, 125 grantees shared a count of adult family members of participating students who participated in at least one activity of any type during this program year. Grantee counts ranged from four adults to 1,395, with an average of 139 adults, with 37 grantees having a count that exceeded the average and 88 having a count of less than the average. Five grantees – all Cohort 6A programs operating only a short time – reported having no adult family member participation.


In terms of participation in parent education or engagement activities, including such activities as adult ESL, parent education/workshops, computer training, parenting skills, and similar offerings, 99 grantees reported having between two and 848 adult family members participate in such activities, with an average of 61 participants. In terms of participation in family involvement activities, such as open house events, family nights, and similar opportunities, 116 grantees reported participation of between five and 1,164 adult family member participants, with an average of 109 participants. Grantee Provision of Professional Development All 130 grantees completing the Implementation Survey indicated that they offered professional development of some form to staff. This professional development most typically took the form of staff orientations (90 percent of grantees) or health and safety training (76 percent grantees) among other options. Grantee contracts require them to participate in certain professional development opportunities. When asked to indicate how professional development learning, information, and resources were shared with other program staff, email communication was selected most (94 percent of grantees), followed by staff meetings (93 percent), and informal conversations (91 percent) among other methods to a lesser extent. State Provision of Professional Development PDE and the Center for Schools and Communities (PDE’s contractor for 21st Century technical assistance) offered or facilitated grantee access to several professional development opportunities. These opportunities occurred through four venues: the Promising Practices – Proven Strategies Extra Learning Opportunities Conference, which occurs annually in March; the annual 21st Century Grantees Meeting, held immediately before the Extra Learning Opportunities Conference; nine webinars; and three regional trainings, one day each in western, central, and eastern Pennsylvania. Content of the grantee meeting and regional trainings included grant and compliance information as well as implementation strategies. The Extra Learning Opportunities Conference covered various topics related to the implementation of programs during nonschool hours. The Center for Schools and Communities was primarily responsible for state-level training opportunities and submitted a full report about trainings to PDE. As such, only an overview is included here. This summary intends to provide an overview of the scope and reach of state-offered professional development opportunities. The Extra Learning Opportunities Conference, held over two days in Harrisburg in March 2016, was a major professional development opportunity for 21st Century grantees. Although it was open to participants beyond 21st Century programs, the conference focused on out-of-school-time programs, strategies, and resources, making it an especially relevant learning opportunity for grantees. Based on data provided by


the Center for Schools and Communities, which managed the event, 23510 individuals participated in this conference; grantee-specific counts were not available. Based on attendance data provided by the Center for Schools and Communities, 26511 21st Century staff attended the mandatory annual grantee meeting held before the Extra Learning Opportunities Conference. Nine webinars occurred during 2016-17 covering topics relevant to out-of-school-time programs. Members of the state 21st Century team or various experts and contributors from outside the program presented the webinars. These webinar topics, timing, and participation counts are shown in Table 1. It is possible that the actual participant count is higher, as it is known that some groups have multiple people participating from the same location through one registration, but the extent to which this happens is not consistently captured, as participants need to self-report this information. Grantee representation or counts were not available. Each webinar was 60 or 90 minutes long. Table 1: Webinar Details

Topic Presenter(s) Month/Year Participants Applying Game Elements and Activities to Engage Students and Make Learning Fun!

Bloomsburg University

September 2016 54

Brain-Based Strategies to Support Literacy Development in Adolescents

BLaST Intermediate Unit October 2016 33

Finding Each Student’s Sweet Spot: Optimizing Engagement and Learning Begin with the Brain December

2016 38

Promoting Protective Factors with Youth and Families

Center for Schools and Communities January 2017 44

Strengthening Teacher Confidence and Student Achievement by Integrating STEM Across the Curriculum

Virginia Commonwealth

University February 2017 36

Where to Start? A Primer on 21st CCLC Evaluation and Reporting (This webinar was targeted toward novice grantees.)

State Evaluation Team, Allegheny Intermediate Unit

March 2017 31

The Evolving 21st CCLC Evaluation (This webinar was targeted toward experienced grantees.)

State Evaluation Team, Allegheny Intermediate Unit

March 2017 41

Envisioning the Future: Creative Career Exploration in the Afterschool Setting

Dothan Brook School April 2017 40

Who's Doing the Work? Rethinking Scaffolds for Struggling Readers

BLaST Intermediate Unit May 2017 21

10 Count excludes state team staff. 11 Count excludes state team staff.


Regional trainings occurred in Mechanicsburg, Malvern, and Pittsburgh in October 2016. Attendance included 139 individuals for the regional session held in Malvern, 56 for the regional training in Mechanicsburg, and 61 individuals for the Pittsburgh location. Counts of grantees represented were not available. All three locations included a PDE overview of grant requirements in the morning followed by the grantee’s choice of two workshops in the afternoon: “Engagement Matters: Strategies to Deepen Student’s Learning” and “Culturally Responsive Approaches to Afterschool Culture and Climate: Moving from Good Intentions to Positive Impact.” An additional workshop was provided in Malvern: “Beyond Poverty: Brain-Inspired Approaches to Serve Students with Economic Challenges.”

Professional Development and Support Needs Within the PA Implementation Survey, grantees had the opportunity to share or explain their needs or interests for additional training or support. Approximately half of grantees provided a substantive response. These needs and interests are outlined in the following pages in no particular order. This year, few grantees indicated challenges with identification and recruitment, which may be because each of the cohorts had been in operation more than a year. Identification, Recruitment, and Retention

• General recruitment strategies • Successful recruitment of secondary students • How to increase regular attendance for high school students • Innovative recruitment and retention strategies

Operations and Implementation

• Training related to positive student behavior • How to promote parent involvement • Parent programming • Collaborating with other school activities • Connecting with program partners • College and career readiness • STEAM/STEM activity ideas • Trainings held during afternoon/after school hours • Student engagement • Guidance on providing staff development and orientation • Common Core math tutoring/support methodology • Guidance on Title I or Title III collaboration or integration with after school

programming • Strategies for overcoming transportation challenges • Mentoring for struggling programs • Identification of best practice strategies


• Interactive training strategies for managing student behavior in situations related to gender identity, bullying, racism, and cultural sensitivity

• Recruiting, training, and retaining program staff • Science activities • Managing student behavior • Strategies for implementing programming before school, student recruitment, in

particular • Building better relationships with parents • Secondary level literacy remediation • How to conduct parent phone calls • Social/emotional learning, conflict resolution, resilience, and mindfulness • Innovative programs and strategies • Project-based learning • Differentiated instruction in a mixed-grade level environment • Strategies for improving communication with schools • Violence and drug-prevention programs • Sex education • Classroom management • Increasing parent participation in parent activities • How to align activities with the school curriculum • More frequent, local regional trainings • How to develop student voice in enrichment activities • How to manage competition with other school and sports activities • Strategies for working with students who are new arrivals to the United States • More strategies for identifying and serving students’ individual needs

Data and Evaluation

• Suggestions or strategies for making the data collection process from students’ schools easier or more complete

• Continued information on student data collection • Suggestions as to whether or not program staff should advocate for daily access

to school data systems • Effective data collection and organization strategies • Information on PSSA scoring • Data system strategies that eliminate the need for spreadsheets and Google

docs to collect information • Ongoing training for staff in using evaluation/data tools • Managing timely data collection • How to increase the response rate for the teacher survey

Creative and Innovative Strategies In the PA Implementation Survey, grantees had the opportunity to share what they believed were creative or innovative strategies being used in their programs to engage


students and address their needs. Approximately 75 percent of grantees shared one or more strategies. These are listed here, in no particular order.

• Moving enrichment activities to a community recreation center, “which has the feel of a teen club house”

• Various approaches to incorporating STEM/STEAM activities • Using student data to plan small group tutoring • Daily, brief calls to parents with news of students’ positive progress and, as

needed, discussion of needs • Encouraging students toward self-directed achievement of goals • Youth program quality assessment process • STEM courses • Staff attend community events to promote interest in the program • Secured bus transportation for eligible students • Student surveys in order to understand and incorporate their interests into the

program • Thematic literacy curriculum • Alignment of technical instruction with industry needs and real-world applications • Playbooks Reader's Theater STEM kits • Cultural activities • Incorporation of student interests and opportunities to change activities

throughout the year • Layering English language support onto hands-on activities for English language

learners • Career exploration, resume writing, and mock interviews • Parent Cafés for parent input • Individual student conferences and debriefing forms • Collaboration with local libraries and city literacy initiatives • Partnerships for cultural arts and entrepreneurship • Maker spaces, “maker” activities • Coordinating recruitment with the district athletic director • Mailing registration information to all eligible students, posting recruitment flyers

in district neighborhoods • Students staying after school can complete their homework with a certified

school day teachers • Dinner and bussing provided • Response to Intervention and Instruction services to after-school participants not

served during the school day • Drug awareness • Weekly mentoring groups • Student voice/choice is encouraged for greater student investment in the

program • Clubs • Project-based learning • Activity planning based on surveyed student interests


• Adaptive computer literacy program where students can earn computer-based activity incentives

• School day teachers as after school staff • Personalized instruction through open-source materials and direct instruction • Coding • Robotics • Using special event participation as an incentive for participation in after school

program activities • Network of academic and social services • Students built greenhouses, learned about crop production, and prepared food

with crops they grew • Math competition among sites using a computer-based math program • Enrichment activities are selected to complement academic tutoring and

homework assistance • Arts activities • Instruction and activities related to farming • Youth advisory council • Students each have a Chromebook • Free curricular materials from the national parks program • Culminating activities on Fridays • Pairing of older and younger students • A program that teaches the science marketing behind cosmetics • Senior citizen pen pal program • Connecting students to social or community services • High-interest, research-based programs • Hiring staff from the community served

PROGRAM PARTICIPATION Based on data reported to state evaluators, grantees (130 of 134 grantees required to report) served 37,79312 students over the course of the summer 2016 and school year 2016-17 program year, with 15,542 students (41 percent) attending 21st Century programming for 30 or more days and receiving the designation of regular attendee. Pennsylvania public school enrollment, based on third-day enrollment for the 2016-17 academic year, was 1,777,065 students. This means that Pennsylvania’s 21st Century programs served approximately 2 percent of the Pennsylvania public school population. Based on data reported through other mechanisms, it is believed that had the remaining four grantees reported, the total students served count would exceed 38,000 students. 12 Based on grantee-reported data, 64 students were served by two cohorts because they were served through one program/cohort during summer programming prior to the end of Cohort 6A and then were served through Cohort 7 or Cohort 8 programs.


Figure 3.

Cohort results reflected the structure of program contracts, as the Cohort 6A grantees operated in the summer 2016 term and up to September 30, 2016. Of the students served, 58 percent were from Cohort 7 programs, 36 percent were from Cohort 8 programs, and 6 percent were from Cohort 6A programs.13

Figure 4.

13 Includes 64 students who were served by more than one cohort (duplicated). These students make up less than 0.2 percent of students served. These students may be duplicated in served counts throughout this report; however, grantees were advised that their outcomes only be reported under one cohort if the student was a regular attendee.


A majority of students (64 percent) attended only during the school year; 18 percent attended during summer 2016 only and 18 percent attended both summer 2016 and school year 2016-17 terms.

Figure 5.

Participation ranged from four to 1,059 students per grantee, with an average of 291 students per grantee. Of the 130 grantees included in analysis, 51 grantees (39 percent) had a served count that exceeded the average. Nineteen grantees reported having no regular attendees, with all of these being Cohort 6A grantees that offered programs that ended early in the year. For those grantees reporting regular attendees (111), regular attendee counts by grantee ranged from two students to 399, with regular attendee percentages ranging from 1 percent to 100 percent an average of 50 percent of students being regular attendees. STUDENT OUTCOMES Grantees reported on outcomes for regular attendees via the de-identified student data spreadsheet component of annual state reporting. All grantees having regular attendees were required to report student outcomes, which included reading and math report card grade results, teacher survey results, reading and math state assessment results, school attendance, school behavior, and credit recovery, as they applied to the grantee’s program and population served. Also, credit/course recovery outcomes were required for all credit recovery students even if they were not regular attendees. Grantees reported having 15,542 regular attendees, and it is for these individuals that reporting outcomes was required. Students may have outcomes data under any number of the provided outcomes areas and/or data source categories depending on the timing of their participation, grade level, and other reasons. Students may not have data for all areas because they do not apply (i.e. a student may not have state assessment results because they are not in a grade that takes the assessment) or simply because data were not available for them. Results described in this report


include all students having data available that could be analyzed and may not represent all students served by the program. Relevant percentages are provided to describe the portion of students served who had data included. Results shared in the following section are provided overall for all regularly attending students having data and by cohort, by program attendance category (30-59 days, 60-89 days, 90+ days), and by grade or grade band. Academics Results provided in this section address the program performance measure: “Increasing percentages of students regularly participating in the program will meet or exceed state and local academic achievement standards in reading and math.” Some grantees shared that they experienced difficulty related to collecting, managing, or analyzing student academic data. Twenty-six percent of grantees indicated that they experienced no challenges or were not aware of any challenges. The most common challenges included completion of the teacher survey (35 percent of grantees), the state assessment availability timeline (30 percent of grantees), and delays receiving data from students’ home schools/districts (28 percent of grantees), which were the same three most-selected challenges in the prior year. State Reading and Math Assessments Each year, students in certain grades take one of Pennsylvania’s literacy and/or math state assessments (PSSA, PASA, or Keystone Exam). The PSSA is administered to the most students and is given in March or April14 in grades three to eight. Students in 11th grade take the Keystone Exam, which may be administered up to three times per year.15 The PASA is Pennsylvania’s alternative state assessment and is administered in grades three to eight and 11 for students with cognitive disabilities. The Keystone Exam and PASA are aligned to the PSSA and use the same performance levels (below basic, basic, proficient, or advanced). Evaluators examined current year (2017) snapshot state assessment results for students having data as well as progress results for students having both prior year (2016) and current year (2017) state assessment data. Students must have two consecutive years of assessment data in order to calculate change from one year to the next. Students having two consecutive years of data would be (primarily) in fourth through eighth grades. Students who were retained in third grade could also have two consecutive years of state assessment data.

14 Writing and science PSSA data is not included in state or federal 21st Century reporting at this time. 15 Students may begin taking the Keystone Exams as early as eighth grade, but before 11th grade, these assessments may not be used for accountability purposes. If a student scores at the proficient or advanced level on one or more of the Keystone Exams, that score is banked and applied to that student’s 11th grade year.


To determine students’ improvement status, evaluators compared the prior year’s performance level (below basic, basic, proficient, or advanced) to the current year’s performance level. Improving means that the student positively moved from one level to another based on the grade level rubric that applied to the particular assessment, such as moving from below basic to basic, proficient, or advanced; from basic to proficient or advanced; or from proficient to advanced. A total of 92 grantees reported snapshot state assessment results for 2017 for grades 3-8 and 11 (87 percent of grantees that reported having regular attendees in state assessment grades16); 84 of these grantees also reported 2016 state assessment results for examining student improvement status. In the following pages, current year snapshot results for 2017 are provided followed by improvement status results. Students could have had nearly a full year of programming by the time the state assessments were administered in spring 2017. It is important to keep in mind that state assessment results can only be reported for a subset of students, as these assessments are not administered in all grades and two consecutive years of data is required for improvement status to be determined. Also, because only whole performance levels are reported, it takes a good deal of growth before a student can move from one performance level to the next. For 2017 reading state assessments, 8,365 students were included in analysis17 (approximately 76 percent of regular attendees in grades eligible to take the 2017 state assessments). Of these students, 6 percent scored in the advanced performance level, 29 percent scored proficient, 45 percent scored basic, and 20 percent scored in the below basic level. This is an improvement over the prior year (2015-16), where 5 percent scored in the advanced performance level, 27 percent scored proficient, 43 percent scored basic, and 25 percent scored in the below basic level. Results by cohort were similar, despite Cohort 7 having twice the number of students included in analysis as Cohort 8.

16 One of these grantees was able to provide individual student assessment results, but was not able to connect it with other data elements because of how the school district provided data. As such, a disaggregation by attendance category was not possible for this program. This grantee’s results are included in the overall state and cohort results but not in any disaggregations by program attendance. 17 One Cohort 6A grantee reported state assessment results for its regular attendees in grade levels that take state assessments. Because of the small number of students and being the only grantee to report results, Cohort 6A results are not broken out in this report; however, Cohort 6A data was included in overall state-level analysis.


Figure 6.

Results by program attendance do not show much difference, as the percentages scoring in each performance levels differed by two or fewer percentage points, if any, among the three attendance categories.

Figure 7.


By grade level, third grade had the highest percentage of students achieving at the proficient and advanced levels (41 percent); however, this was only two to nine percentage points higher than other grade levels. Eighth grade had the lowest percentage of students achieving at the proficient and advanced levels (31 percent).

Figure 8.

A total of 5,900 students had two consecutive years of state reading assessment data, which is 67 percent of regular attendees in grades where two consecutive years of state assessment data are most likely (grades 4-8). Of these students, 22 percent made a positive movement of one or more levels from 2016 to 2017, which is an improvement over the prior year, in which 19 percent showed gains. Furthermore, 12 percent of students who scored at the basic or below basic levels in 2016 (and had both years of data) improved to proficient or advanced in 2017. More than half of students (61 percent) scored in the same level in 2016 and 2017. While this group includes students who may have improved in the past and those who maintained a proficient level (30 percent of no change students), this group also includes students who scored in the lowest performance level (below basic) on both administrations (22 percent of no change students). Of the students declining (14 percent of students), 99 percent declined by one performance level. This group includes students who declined from advanced but were still proficient (14 percent of students declining) as well as students who declined to or within non-proficient levels. Students who did not need to improve (3 percent of students) scored at the advanced level on both administrations of the assessment. The inclusion of students not needing to improve (in this and other sections) is not necessarily an indication of programs not


targeting students with needs; while a student may not have needed to improve in the area of reading, a student may have a math, behavior, or other need. Results were somewhat consistent across program attendance categories and cohort. However, while Cohort 7 showed nearly equal improvement percentages across program attendance categories, Cohort 8 showed increasing improvement percentages with greater program attendance, where 23 percent improved in the 30 days category, 25 percent improved in the 60 days category, and 27 percent improved in the 90 days category, indicating that for Cohort 8, increased participation may have had a positive influence on outcomes.

Figure 9.

State reading assessment improvement status by grade level, like snapshot results, did not differ dramatically, though there were some greater differences than other categories. Sixth grade showed the highest improvement percentage at 31 percent, while fifth grade had the lowest improvement percentage at 18 percent.


Figure 10.

As in past years, analysis further examined grantee success in contributing to positive outcomes for students based on state reading assessments. To do this, evaluators compared the number of students improving on the state reading assessments to the number of students included in analysis, excluding students who did not need to improve in order to determine the percentage of students improving of students who had room to improve. Of the 84 grantees reporting reading state assessment cross-year comparison data, 59 grantees (70 percent) had results showing 0-25 percent of students improved and 25 grantees (30 percent) had results showing 26-50 percent of students improved. Grantees reported data for four to 318 students. It is relevant to note that 10 of the grantees included in this analysis reported data for fewer than 20 students, so each student’s result contributes more heavily to the grantee’s results than a grantee reporting data for several hundred students. New for the 2016-17, state evaluators collected data on students’ historical presence in 21st Century programs. Grantees had the option to designate, for each regular attendee, whether the student had been involved with that grantee’s 21st Century programs – through the current or a prior cohort – for one, two, three, four, five, or more than five years. This being the first year this element was collected, reporting was not required. Evaluators collected this information in order to determine if ongoing and consistent participation in 21st Century programs may contribute to improved student outcomes, as past evaluation experience suggests that it takes approximately three years of a program to begin seeing results. As 21st Century programs in Pennsylvania are at least three years, it is in these later years that evaluators would expect to begin to see greater differences in results. Certainly, it is possible that a given student could have participated in another grantee’s program and this would not be captured by the current grantee, but it was assumed that this would likely be a small portion of students,


and at present, there is no way to determine the extent to which that was a factor in results, as student identifiers were not required to be reported. Evaluators looked at historical presence both by performance level and improvement status. By performance level, historical presence information was available for 4,468 students, or 53 percent of all regular attendees who had state assessment current year data. Looking at results by performance level score in 2017, percentages scoring proficient or advanced were similar for all regular attendees and between one and three years of participation. At four years of participation, the percentage scoring proficient and advanced increased to 48 percent. At five years, the percentage drops back to 35 percent and for more than five years’ participation, the percentage was 22 percent. These lower percentages in the greatest duration categories may be indicative of students having the greatest needs participating for many years, but it is not possible at this time to determine the extent to which that is true. This may also simply be a factor of smaller numbers of students included in analysis in the greater participation levels. Improved reporting of this element in future years may provide greater accuracy.

Figure 11.

By improvement status, historical presence information was available for 3,049 students, or 52 percent of regular attendees who had two years of state assessment data. Students only participating in the current year had the highest improvement percentage at 23 percent. Years two through four were consistent at 20 or 21 percent, before decreasing at five years (14 percent improved) and more than five years (9 percent improved). However, this may be a factor of very small numbers of students in the greatest participation categories.


Figure 12.

For 2017 state math assessments, 92 grantees reported 2017 data (87 percent of grantees reporting students in state assessment grades), and included results for 8,516 students (78 percent of regular attendees in grades eligible to take the state assessments). Of these students, 5 percent scored in the advanced performance level, 15 percent scored proficient, 29 percent scored basic, and 51 percent scored in the below basic level, which is almost exactly the same as the prior year, where 5 percent scored in the advanced performance level, 15 percent scored proficient, 28 percent scored basic, and 52 percent scored in the below basic level. Like reading, Cohort 8 had a slightly higher percentage of students who scored at advanced or proficient levels (22 percent) compared to Cohort 7 (19 percent), but again, the number of students reported under Cohort 7 was more than double the number reported by Cohort 8.


Figure 13.

By program attendance category, students who attended the program at the highest level (90+ days) were more likely to score in the proficient or advanced performance levels (21 percent); however the difference was only one percentage point. Generally, results were nearly the same, despite different numbers of students in each category.

Figure 14.


By grade level, like reading, third grade had the highest percentage of students scoring in the advanced and proficient levels at 33 percent along with 11th grade with 33 percent. The other grade levels ranged from 13 percent (eighth grade) to 21 percent (fourth grade).

Figure 15.

A total of 5,968 students had two consecutive years of state math assessment data, which is approximately 68 percent of regular attendees in grades likely to have two years of state assessment data (grades 4-8). Of these students, 13 percent made a positive movement of one or more levels from 2016 to 2017, 3 percent scored in the advanced level on both administrations, 68 percent scored in the same performance level in both years, and 17 percent declined, which is similar to the prior year, where 13 percent improved, 66 percent showed no change, 19 percent declined, and 2 percent did not need to improve. Furthermore, 5 percent of students who scored at the basic or below basic levels in 2016 (and had data for both years) improved to proficient or advanced in 2017. The no change category (68 percent) includes students who may have improved in the past and those who maintained a proficient level (11 percent of “no change” students), this group also includes students who scored in the lowest performance level (below basic) on both administrations. Students who scored below basic in both years in math made up 66 percent of the students in the “no change” category. Students declining (17 percent of students) includes students who declined from advanced but were still proficient (13 percent of those students who declined) as well as students who declined to or within nonproficient levels.


Students who did not need to improve (3 percent of students) scored at the advanced level on both administrations of the assessment. The inclusion of students not needing to improve (in this and other sections) is not necessarily an indication of programs not targeting students with needs; while a student may not have needed to improve in the area of reading, a student may have a reading, behavior, or other need. Results were consistent across program attendance categories and cohort, though Cohort 8’s improvement percentage was three percentage points higher than Cohort 7. Students in the 60 and 90 program days attendance category had a slightly higher improvement percentage (14 percent), but the difference was small (11 percent for 30 days. Cohort 8’s 90 days category had the highest improvement percentage at 18 percent, compared to the other levels and Cohort 7 at 9 to 15 percent.

Figure 16.


By grade level, seventh grade had the lowest percentage improving (8 percent), while fifth grade had the highest at 18 percent.

Figure 17.

As in past years, analysis further examined grantee success in contributing to positive outcomes for students based on state reading assessments. To do this, evaluators compared the number of students improving on the state math assessments to the number of students included in analysis, excluding students who did not need to improve in order to determine the percentage of students improving of students who had room to improve. Of the 85 grantees reporting math state assessment improvement data, 76 grantees (89 percent) had results showing 0-25 percent of students improved, eight grantees (9 percent) had results showing 26-50 percent of students improved, and one grantee had results showing 51-75 percent of students improved (of those needing to improve). Grantees had data reported for five to 333 students; nine grantees reported cross-year state assessment results for fewer than 20 grantees. It is important to note that when data are reported for small numbers of students, each student’s result contributes more heavily to the grantee’s overall result. Like reading, evaluators were able to examine math state assessment results by historical presence both by performance level and improvement status. By performance level, historical presence information was available for 4,579 students, or 54 percent of all regular attendees who had state assessment current year data. Looking at results by performance level score in 2017, those with four program years had the highest percentage scoring proficient or advanced (38 percent), followed by five years with 24 percent, compared to 10 to 22 percent for the remaining levels. But


again, these greater years of participation have somewhat smaller numbers of students included, which may be a factor in the results. Also, these lower percentages in the greatest duration categories may be indicative of students having the greatest needs participating for many years, but it is not possible at this time to determine the extent to which that is true.

Figure 18.

By improvement status, historical presence information was available for 3,123 students, or 52 percent of regular attendees who had two years of state assessment data. Students only participating in two years of 21st Century had the highest improvement percentage at 14 percent, with five years being close at 13 percent. The more than five years category had the smallest improvement percentage (6 percent).


Figure 19.

Reading and Math Report Card Results Grantees reported individual student fall and spring reading and math report card grades for regular attendees using the de-identified student data spreadsheet template. In order for a student to be counted as improving, the student had to move half a grade or more from the fall report card grade to the spring report card grade, as defined by federal reporting criteria. For “A-F” scale letter grades, this involves a move within a letter grade,18 for example from a “C-” to a “C+,” or among letter grades, for example “C-” to “B+,” of half a level or more. For numeric scales, this involves a move of five or more percentage points (i.e. 70 percent to 75 percent). For schools using other scales, a student had to go from one level to another for change to be counted. Student academic change was determined based on a comparison of an individual’s fall and spring grade for the same school year; in this case, fall 2015 compared to spring 2016. This methodology was selected to be consistent with prior year’s analysis, which had been based on federal guidance in place at the time. A total of 12,239 students had reading report card data (any grading scale) that could be used for analysis (students had two data points using a scale interpretable by state evaluators), which is 80 percent of school year regular attendees for whom outcomes data were reported. A total of 103 grantees (96 percent of applicable grantees) reported usable reading report card grade data. 18 Some schools do not give half letter grades. In these cases, grantees are instructed to report students’ whole letter grades, meaning it is slightly more difficult for these grantees to show students’ improvement using federal criteria. Data is not available on the extent to which this situation applies.


Of the students having reading report card grade data that could be analyzed, 29 percent improved from fall to spring. The largest percentage, at 40 percent, showed no change, meaning they earned the same grade for both the first and last grading periods. Results also indicated that 24 percent declined from fall to spring and 7 percent did not need to improve their grade and maintained that grade. Results by cohort show very similar percentages of students despite differences in student counts.

Figure 20.

In addition to examining results overall, results were disaggregated by program attendance category. While the differences were small, results show increasing percentages of students improving with each greater program attendance category.

Figure 21.


Analysis by grade band showed relative similarities among the levels with percentages of students improving ranging from 26 percent of students improving at the high school level to 31 percent of students improving in fourth and fifth grades. High school and middle school students were more likely than other grade bands to have a decline in their report card grade.

Figure 22.

Evaluators further examined grantee success in contributing to positive results for students based on reading report card grades by comparing the number of students having reading report card grade data that could be analyzed to the number of students improving for each grantee, excluding those students who did not need to improve. Grantees reported data for four to 376 students. Of the 103 grantees reporting data, 29 (28 percent of reporting grantees) reported results that showed 25 percent or less improved (of those needing to improve), 63 grantees (61 percent of reporting grantees) reported results where 26 to 50 percent of students improved, 10 grantees (10 percent of reporting grantees) reported results where 51 to 75 percent of students improved, and one grantee reported results showing that 76 percent or more of students improved. Historical presence analysis for reading report card grades was also conducted, with 55 percent of regular attendees with report card data also having historical participation information. In looking at the results by years of 21st Century participation, improvement percentages were the same as or within one percentage point for the one to four years categories. For five years participation, the improvement participation increased to 32 percent, and for more than five years, the improvement percentage was 41 percent. This indicates that longer duration of 21st Century participation may have a positive influence on reading report card outcomes.


Figure 23.

For math report card grades, a total of 12,379 students had math report card data (any grading scale) that could be used for analysis (students had two data points using a scale interpretable by state evaluators), which is 82 percent of school year regular attendees for whom outcomes data were reported. A total of 103 grantees19 (96 percent of applicable grantees) reported usable math report card grade data. Of the students having math report card grade data that could be analyzed, 29 percent improved from fall to spring. The largest percentage, at 38 percent, showed no change, meaning they earned the same grade for both the first and last grading periods. Results also indicated that 26 percent declined from fall to spring and 7 percent did not need to improve their grade and maintained that grade. Results by cohort show very similar percentages of students in each outcome category despite differences in student counts.

19 One of these grantees was able to provide individual report card grade data, but was not able to connect it with other data elements because of how the school district provided data. As such, a disaggregation by attendance category was not possible. This grantee’s students’ results are included in the overall state and cohort results but not in any disaggregations by program attendance level.


Figure 24.

In addition to examining results overall, results were disaggregated by program attendance category. It does not appear that greater program attendance has a considerable influence on math report card outcomes, as each category’s results are similar, despite the differing numbers of students included in each category. However, there is a slightly lower percentage of students declining in the 90+ days category (24 percent) compared to those in lesser attendance categories (27 percent of students in the 30 days category and 26 percent of the 60 days category declined), which may indicate a positive influence in preventing students from declining.

Figure 25.


Analysis by grade band showed similar percentages of students improving ranging from 25 percent of students improving at the high school level to 31 percent of students improving at the middle school level.

Figure 26.

Evaluators further examined grantee success in contributing to positive results for students based on math report card grades by comparing the number of students having math report card grade data that could be analyzed to the number of students improving for each grantee, excluding those students who did not need to improve. Of the 103 grantees reporting, 34 grantees had results that showed 0-25 percent of students improved, 62 grantees – the largest group at 60 percent of grantees – had between 26 percent and 50 percent of students improve their math report card grade from fall to spring, six grantees had improvement percentages in the 51 to 75 percent range and on grantee had an improvement percentage in the 76-100 percent range. Historical presence analysis for math report card grades was also conducted, with 54 percent of regular attendees with report card data also having historical participation information. In looking at the results by years of 21st Century participation, improvement percentages were the same as or within one percentage point for the one to three years categories. For four years participation, the improvement participation increased to 32 percent, and for more than five years, the improvement percentage was 41 percent, though the five years participation showed 27 percent improving. This indicates that longer duration of 21st Century participation may have a positive influence on math report card outcomes.


Figure 27.

21st Century Teacher Survey – Academic Performance The 21st Century Teacher Survey included an indicator for teachers to report student change in academics. This determination was to be made by the classroom teacher about each regularly attending student participating during the school year based on his/her professional opinion of the student’s classroom performance. It was recommended that a math or language arts teacher complete the survey, though the content area of the teacher completing the survey was not collected. The survey allowed the respondent teacher to rate the student’s academic performance using a scale that included “did not need to improve,” “significant improvement,” “moderate improvement,” “slight improvement,” “no change,” “slight decline,” “moderate decline,” and “significant decline.” In general, the disaggregated degree of change did not contribute in a notable way to the results as the number of students included in those results was small, so the results from these eight change categories were collapsed to the four change categories used throughout this report: “did not need to improve,” “improved,” “no change,” and “declined.” Data was available for 10,239 students, which is 68 percent of school year regular attendees. Results show that 56 percent of students improved. By degree of improvement, the largest portion showed “slight improvement” (26 percent of all students). Seventeen percent of students included in analysis were reported as not needing to improve, 18 percent showed no change, and 9 percent declined, according to teacher survey results. Of students declining, the majority showed “slight decline” (5 percent of all students). Cohort results in terms of percentages were very similar,


though Cohort 8 had slightly higher improved and declined percentages than Cohort 7, despite differences in student counts. Percentages were consistent across program attendance category, with improvement percentages being either 56 or 57 percent. No change ranged from 17 to 19 percent, and did not need to improve ranged 15 to 19 percent. While there did not appear to be a notable difference in improvement percentages by program attendance category, the percentage of students declining decreased with greater attendance, going from 10 percent at 30 days to 8 percent at 60 days, to 7 percent at 90 days, indicating that increased program attendance may have helped prevent students from declining.

Figure 28.

Analysis by grade band revealed a range of percentages of students improving from 50 percent of students improving at the high school level to 62 percent of students improving in the preschool and first grade band. Older students were more likely to decline than younger students, with decline percentages increasing with each higher grade band. Regularly attending students whose grade level was not reported (less than 1 percent of regular attendees) are not included in grade band results, though they are included in the overall regular attendee results.


Figure 29.

Behavior Results provided in this section address the following program performance measures:

1. Participants in 21st Century programs will demonstrate educational and social benefits and exhibit positive behavioral changes and


21st Century Teacher Survey The 21st Century Teacher Survey included indicators for classroom teachers to report on change in behavior based on his/her professional opinion and experience with each student. The survey provided a scale that included “did not need to improve,” “significant improvement,” “moderate improvement,” “slight improvement,” “no change,” “slight decline,” “moderate decline,” and “significant decline.” In most cases, the disaggregated degree of change did not contribute in a notable way to the results, as the number of students included in those results was small, so from these eight change categories were collapsed into the four change categories used throughout this report: “did not need to improve,” “improved,” “no change,” and “declined.” 21st Century Teacher Survey data for each element includes between 9,947 and 10,268 students of 66 to 68 percent of school year regular attendees. The percentage differs by survey item as some teachers may not have provided a response for all items for all students who were included in grantee-submitted data. Readers should also note that in graphs of results by grade band, any regularly attending student whose grade level


was not specified (less than 1 percent of regular attendees) are not included in grade band results, though they are included in overall results. For the indicator of satisfactory homework completion, 9,973 had survey data and half of these students showed improvement (51 percent), 23 percent of students did not have a need to improve, 17 percent did not change, and 8 percent declined. Of students improving, most showed “slight improvement.” Of those declining, most had a “slight decline.” Cohort results were similar to each other and the overall state results, though Cohort 8 had both a slightly higher improvement percentage and a slightly higher decline percentage than Cohort 7, which showed no change and did not need to improve percentages that were slightly higher than Cohort 8. Attendance category results show little difference, with improvement percentages decreasing slightly with greater program attendance from 53 percent at 30 days to 49 percent at 90 days; however, the percentage of students declining also decreases with greater attendance (from 9 percent to 7 percent) indicating the program may have prevented decline. Also, the 90 days category had a higher did not need to improve percentage (26 percent) compared to the lesser categories at 21 and 23 percent.

Figure 30.

By grade band, younger students were more likely to improve than older ones, with improvement percentages decreasing from 53 percent at the PK-1 level to 47 percent at the high school level. Decline percentages increased for each older category as well. No change was consistent across levels at 17 or 18 percent. Did not need to improve decreased from 25 percent at the PK-1 level to 21 percent at the high school level. However, these results are probably more indicative of the increased liklihood of older students having homework than younger students.


Figure 31.

For the indicator of timely homework completion (9,983 students having data), 47 percent of students improved, 27 percent did not need to improve, 18 percent showed no change, and 8 percent declined. Of those improving, the largest portion showed “slight improvement.” Of those declining, most had a “slight decline.” Like the prior indicator, Cohort 8 had both a slightly higher improvement percentage and a slightly higher decline percentage than Cohort 7, which showed no change and did not need to improve percentages that were slightly higher than Cohort 8. By program attendance, improvement percentages decreased with greater program attendance, going from 49 percent at 30 days to 44 percent in the 90 days category. No change and declined percentages were consistent across attendance categories. Did not need to improve, however, showed a larger percentage of students not needing to improve in the 90 days category (30 percent) compared to the two lesser attendance categories (24 and 26 percent).


Figure 32.

Like satisfactory homework completion, timely homework completion results may reflect the greater chance that older students will have homework. Younger students were more likely than older students to improve, not have a need to improve the timeliness of their homework, and decline according to their teachers’ responses.

Figure 33.


Regular class attendance results were available for 10,268 students and indicated that class attendance was not an area of need for many students. Teachers reported that 45 percent of students did not need to improve their class attendance. Percentages improving (23 percent) or experiencing no change (27 percent) were similar and a small percentage of students declined (5 percent). Of those improving or declining, the degree of their change was most likely “slight” according to their teachers’ survey responses. Cohort 8 results showed a slightly higher improvement percentage (26 percent) than Cohort 7 (21 percent), which had a did not need to improve percentage (49 percent) that was higher than Cohort 8 (39 percent). Results by program attendance showed decreasing improvement percentages and increasing did not need to improve percentages with greater program attendance. While no change and decline percentages remained similar across categories.

Figure 34.

Results by grade band indicate higher percentages of students improving at the high school (28 percent) than the other levels (21-23 percent), while younger students had higher percentages not needing to improve. High school students were also the most likely of the grade bands to experience decline; 9 percent of high school students included in the analysis declined, compared to 3-5 percent in the other grade bands.


Figure 35.

Results for volunteering in class (9,979 students) show the same percentages of students who improved or showed no change (37 percent). Nearly a quarter of students did not need to improve (23 percent). Three percent of students declined. Most students experiencing change had a “slight” change. Cohort 8 had a higher improvement percentage (40 percent) than Cohort 7 (35 percent), which had a slightly higher did not need to improve percentage (25 percent compared to Cohort 8’s 19 percent). Program attendance categories had similar, though slightly decreasing improvement percentages, with 39 percent of students improving in the 30-59 days category compared to 35 percent for the 90+ days category. No change and decline percentages only different by one or two percentage points across attendance categories. Did not need to improve percentages increased with greater attendance, with 20 percent in the 30 days category and 25 percent for the 60 and 90 days categories.


Figure 36.

Results by grade band show similar percentages improving across grade bands (36-38 percent). No change was also similar. High school students were more likely to decline, while younger students were more likely to not have a need to improve in this area.

Figure 37.


For class participation (10,250 students), 51 percent of students showed improvement on the 21st Century Teacher Survey, 21 percent did not need to improve, 23 percent showed no change, and 5 percent declined. Like other areas, the “slight” change category was the largest for improvement and decline. Cohort and attendance category results were very similar to each other and the state, differing by no more than three percentage points, except for a difference of six percentage points in the did not need to improve category, where the percentage of students not needing to improve increased with greater program attendance.

Figure 38.

Improvement percentages for participating in class by grade level were similar (47-52 percent improving), with middle school and PK-1 having the highest improvement percentages. Percentages declining increase for older students, with high school having 10 percent of students declining.


Figure 39.

In the area of class attentiveness (10,028 students), 45 percent of students improved, 25 percent did not need to improve, 21 percent showed no change, and 9 percent declined. Students showing change (improvement or decline) were most likely to experience a “slight” change. Cohort results showed 48 percent improving, compared to 43 percent for Cohort 7, while Cohort 7 had a higher percentage not needing to improve (27 percent) compared to 22 percent for Cohort 8. Percentages showing no change were nearly the same (21-22 percent) and the same for decline (9 percent). By program attendance category, smaller improvement percentages are seen for higher participation (42 percent in the 90 days category, compared to 47 percent at 30 days), while the percentages not needing to improve increased with greater attendance. No change and decline percentages differed only one percentage point by attendance category.


Figure 40.

Results for class attentiveness were very similar across grade bands. Percentages of students improving ranged from 42 percent at the high school level to 46 percent for PK-1 and middle school. Decline percentages ranged 7 to 14 percent, with the 14 percent being at the high school level. Approximately a quarter of students showed no change and another quarter did not need to improve.

Figure 41.


For the 21st Century Teacher Survey indicator concerning class behavior, where 10,256 students were reported, 35 percent of students improved, followed closely by those not needing to improve in this area with 34 percent. Twenty-two percent of students exhibited no change and 9 percent of students declined. Of students experiencing a change, the change was most likely “slight.” Cohort 8 had a slightly higher improvement percentage (38 percent) compared to Cohort 7 (34 percent) and a slightly lower did not need to improve percentage (32 percent compared to Cohort 7’s 35 percent), with no change and decline percentages being nearly the same. Results by program attendance showed decreasing improvement percentages with greater attendance, though the difference between 30 days (38 percent) and 90 days (32 percent) was not large. No change and decline percentages were very close among the categories. Did not need to improve results showed larger percentages of students not needing to improve in the greater attendance category.

Figure 42.

For class behavior by grade band, between a 32 and 35 percent of students did not need to improve. Improvement ranged from 33 percent for second and third grades to 38 percent for pre-K and first grade. No change percentages ranged from 19 to 27 percent and decline percentages ranged 8 to 10 percent, with the 10 percent being from middle school students.


Figure 43.

Of students included in teacher survey results for coming to school motivated to learn (10,016 students), 40 percent improved. Percentages of students showing no change (27 percent) and not needing to improve (26 percent) were similar to each other. Six percent declined. Cohort 8 results showed a higher improvement percentage than Cohort 7 (42 percent compared to 39 percent) and a lower percentage of students not needing to improve. No change and decline percentages were similar to each other and the state. Like other categories, the percentage improving decreased with greater program attendance – 42 percent at 30 days, 40 percent at 60 days, and 38 percent at 90 days – and increasing percentages of students not needing to improve at greater attendance levels, while percentages declining and showing no change were similar.

Figure 44.


Results by grade band for motivation to learn showed decreasing improvement percentages for older students, going from 44 percent of pre-K and first grade students improving to 34 percent of high school students improving. Decline and no change percentages increased for older students.

Figure 45.

Approximately a third of the 9,947 students improved in the area of getting along well with other students, while 37 percent did not need to improve. Almost a quarter of students were reported as showing no change (23 percent) and 6 percent declined. Again, Cohort 8’s improvement percentage (36 percent) was higher than Cohort 7 (33 percent), while Cohort 7’s did not need to improve percentage was higher (39 percent) than Cohort 8 (34 percent). By program attendance, like other areas, improvement percentages decreased and did not need to improve percentages increased with greater program attendance while no change and decline percentages remained consistent. Still, these differences are not dramatic.


Figure 46.

Results by grade band showed improvement percentages ranging from 28 percent at the high school level to 37 percent for pre-K and first grade, though fourth through eighth grades had notable improvement percentages at 35 percent. Positively, decline percentages decreased with age, with high school only having 4 percent reported as showing a decline, compared to 6 percent for the youngest students. Did not need to improve percentages were similar across all categories, ranging from 36 to 38 percent. No change ranged from 19 percent for the youngest students to 31 percent at the high school level.

Figure 47.


School Behavior/Discipline and Attendance Results provided in this section address the program performance measure “Students participating in the program will show improvement in the performance measures of school attendance, classroom performance, and reduced disciplinary referrals.” Seventy-two grantees (54 percent of grantees) reported school behavior and discipline results in the de-identified student data spreadsheet submission. Each grantee established performance indicators in slightly different ways, so grantees were allowed to report results in the general change categories, having freedom to define how change would be calculated. Grantees were not obligated to report these results if school behavior and discipline indicators were not part of their application. Student behavior and discipline results in the four general change categories were available for 8,682 students (56 percent of regular attendees). Overall results indicated that more than half of students (59 percent) did not need to improve in the area of school behavior and discipline. The remaining categories showed similar results, 14 percent improved, 13 percent showed no change, and 13 percent declined according to grantee-defined change. Results by cohort showed Cohort 7 had higher improvement, no change, and decline percentages than Cohort 8, while Cohort 8 had a higher percentage of students not needing to improve. By program attendance category, percentages improving and declining decreased with greater program attendance, while the category for students not needing to improve increased.

Figure 48.


School behavior and discipline results were also examined by grade band. Students in the pre-K and first grade level largely did not need to improve in this area, with 81 percent of students reported in this category. Older students were both more likely to improve and more likely to decline than younger students.

Figure 49.

Historical presence analysis was also conducted for school behavior, with approximately half of students with school behavior data also having historical participation information. This analysis showed increasing improvement percentages with each year, one year through three years to 22 percent improving at three years. The improvement percentage then dropped to 15 percent at four years, only to increase again at five years to 21 percent improving. This may indicate a slight positive program influence on school behavior.


Figure 50.

When evaluators examined grantee results for school behavior and discipline, after excluding students who did not need to improve in this area, 23 grantees had between zero and 25 percent of students improve their behavior at school. Similarly, 24 grantees had improvement percentages in the 26 to 50 percent of students range. Nineteen grantees had between 51 and 75 percent of students improve and four grantees had over 76 percent of students improve (after excluding those who did not need to improve). Grantees reported between four and 392 students, with three grantees reporting data for fewer than 20 students. A total of 77 grantees (58 percent of grantees) reported school attendance results. Each grantee established performance indicators in slightly different ways, so they were allowed to report results in the general change categories, having freedom to define how change would be calculated. Grantees reported school attendance results for 9,457 students, 61 percent of regular attendees, and these results showed equal percentages of students improving and not needing to improve (28 percent), while a third declined and 10 percent showed no change. Cohort results were relatively similar to each other and the state. Program attendance category results were also similar across categories, though students in the highest attendance category were slightly more likely than lesser-attending students to improve and less likely to decline.


Figure 51.

School attendance results by grade band differed slightly, with younger students more likely to improve their school attendance and older students more likely to not need to improve.

Figure 52.


Historical presence analysis was also conducted for school attendance, with 48 percent of students with school behavior data also having historical participation information. This analysis showed similar improvement percentages in years one through three and then slight decreases in year four and beyond.

Figure 53.

When evaluators examined grantee results for school attendance, after excluding students who did not need to improve in this area, nine grantees had between zero and 25 percent of students improve their attendance at school, 51 grantees had improvement percentages in the 26 to 50 percent range, 13 grantees had between 51 and 75 percent of students improve, and two grantees had over 76 percent of students improve (after excluding those who did not need to improve). Grantees reported between two and 342 students, with three grantees reporting data for fewer than 20 students. High School Credit/Course Recovery Thirty-seven grantees reported in the PA Implementation Survey that they had a credit or course recovery component in which students participated during the 2016-17 program year and shared information about this initiative’s implementation, including seven Cohort 6A grantees, 14 Cohort 7 grantees, and 16 Cohort 8 grantees. Not all grantees included credit recovery in their applications for funding. For credit recovery programs that operated during the school year, grantees reported that it typically took students less than a semester to recover a single credit/course. During the summer programs, it typically took students the full term of the summer program to recover one credit/course. More than 75 percent of grantees indicated that credit/course recovery was delivered through a blend of face-to-face instructions and


computer-based instruction; 14 percent indicated delivery was typically face-to-face, and 8 percent indicated it was typically computer based. Sixty-five percent of grantees indicated that students who participated in credit recovery also participated in other 21st Century activities. Of those indicating that students did not typically participate in other 21st Century activities, reasons included other home, family, work, or school obligations; students had so many credits to recover that they did not have the opportunity to participate in other activities, or the program only offered credit recovery activities. A total of 30 grantees (81 percent, six Cohort 6A grantees, 12 Cohort 7 grantees, and 12 Cohort 8) reported on high school student credit recovery outcomes for 2016-17. These grantees reported that a total of 1,417 high school students participated in credit recovery activities during this program year, of which 1,043 (74 percent) recovered one or more credits (141 Cohort 6A, 286 Cohort 7, 356 Cohort 8). By grantee, numbers of students recovering credits ranged from one to 200 students recovering one or more credits. Of the 1,043 students recovering one or more credits, 260 were regular attendees (25 percent) and 783 students (75 percent) attended fewer than 30 days. These 1,043 students recovered a total of 1,627 credits during the 2016-17 program year:

• 466 literacy credits (163 from regular attendees and 303 from non-regular attendees),

• 490 math credits (141 from regular attendees and 349 from non-regular attendees), and

• 671 other credits (147 from regular attendees and 524 from non-regular attendees).

Table 2: Credit Recovery Outcomes

Number of Credits Regular attendees (30+ days) (#)

Non-regular attendees (1-29 days) (#)

Literacy: 1 credit 88 261

2 credits 9 21 3 credits 19 -- Math: 1 credit 111 305

2 credits 12 22 3 credits 2 -- Other: 1 credit 103 373

2 credits 15 60 3 credits 2 9 4 credits 2 1


Of the 1,043 students recovering one or more credits, 742 recovered credits in one content area reporting category only (literacy, math, other), 246 students recovered credits in two content area categories, and 55 students recovered credits in all three content area categories.


2016-17 Government Performance and Results Act The federal 21st Century program established performance objectives as part of the Government Performance and Results Act (GPRA). The indicators addressed here are related to and could be addressed within Pennsylvania’s three performance measures:

1. Participants in 21st Century programs will demonstrate educational and social benefits and exhibit positive behavioral changes;

2. Increasing percentages of students regularly participating in the program will meet or exceed state and local academic achievement standards in reading and math; and


For this report, evaluators pulled GPRA results from the 21APR system, which is the state’s performance based on data that grantees entered in the federal data reporting portal. Evaluators looked specifically at spring term results, as this would be the most complete based on state reporting directives for Pennsylvania grantees. These results may differ from those reported elsewhere in this report, as state evaluators collected and analyzed individual student data submitted by grantees, while grantees reported counts of students by category in the 21APR system. Analysis methods may differ slightly from federal methods, as analysis methods or logic used at the federal level have not been made available to Pennsylvania. As grantee-entered data are not viewable or exportable from 21APR, it is not possible at this time to determine the extent to which grantee-reported counts in 21APR are similar to individual student data grantees submitted to state evaluators. Objective 1: Participants in 21st Century Community Learning Center programs will demonstrate educational and social benefits and exhibit positive behavioral changes. Table 3: GPRA Results Indicator 2016-17 Result 1.1 The percentage of students who improved their math grade from fall to spring. 44% 1.2 The percentage of students who improved English (reading) grade from fall to spring. 43%

1.3 The percentage of students who improved from not proficient to proficient or above in reading on state assessments (elementary). 17%

1.4 The percentage of students who improved from not proficient to proficient or above in math on state assessments (middle and high school). 10%

1.5 The percentage of students who improved homework completion and class participation (teacher-reported). 50%

1.6 The percentage of students who improved behavior (teacher-reported). 37%


Objective 2: 21st Century Community Learning Centers will offer high-quality enrichment opportunities that positively affect student outcomes such as school attendance and academic performance, and result in decreased disciplinary actions or other adverse behaviors. Indicator 2.1: The percentage of 21st Century centers reporting emphasis in at least one core academic area. Indicator 2.2: The percentage 21st Century centers offering enrichment and support activities in other areas. Information for these indicators was not available because of changes in federal reporting. These elements are not explicitly included in 21APR reports. However, based on Implementation Survey data, 97 percent of grantees offered programming related to STEM content and 87 percent offered remedial and enrichment academic activities. Furthermore, grantees indicated the frequency with which they implemented activities in the four core content areas: 80 percent of grantees reported that they implement reading or literacy activities on a daily basis, 70 percent indicated they had math activities on a daily basis, 48 percent indicated they implemented science on a daily basis, and 8 percent indicated they had social studies activities on a daily basis.


Reflections, Implications, and Recommendations for Improvement Overall, evaluation processes and grantee submissions improved over the prior year, which is likely due to prior grantee experience and the new participation worksheet component of state reporting. In following up with grantees, however, evaluators observed that some inconsistencies existed among the reporting venues and conversations with grantees and evaluators revealed some new areas of confusion or difficulty including data incorporation timelines and misunderstanding of terms. Despite this, evaluators believe this year’s data and results may be the most accurate to date, given the additional validity checks put in place. Further, the addition of the historical participation results may begin to provide greater insight into the extent to which 21st Century may be influencing long-term outcomes. The structure of the federal evaluation components and past state evaluations were only set up to examine the current year’s results without regard to prior participation, while it is believed that repeated, consistent participation may more accurately show the program’s impact. Based on evaluation findings and implementation of the state evaluation, evaluators offer recommendations for improvement of Pennsylvania’s 21st Century Community Learning Centers program and its evaluation. CONSIDERATIONS FOR THE STATE TEAM

1. The incongruity of the contract years for 21st Century programs and the federal reporting year not only cause confusion among grantees, but also result in grantees/cohorts operating only a portion of the year when when new grants are brought on or prior cohorts end. This may result in inconclusive findings at the state level for these cohorts. Also, misalignment of the program years and contract years does not allow grantees to maximize their potential for positively influencing student outcomes.

Recommendation: PDE should consider options for more closely aligning the program year and contract years for future funding opportunities in order to maximize program potential. While factors beyond PDE’s control may influence the timing of contracts and programs, where possible, PDE should continue to advocate for a timeline that more closely aligns to the reporting term. A possible solution may be to prohibit grantees from operating active centers after a certain point and use the end portion of the contract year solely for administrative tasks when those final weeks are at the very beginning of a new school year. For example, if a contract year ends September 30th, grantees should be advised against operating after their summer program ends (if they are approved for a summer program). Grantees that operate for just a few days or weeks in the following school year not only adds confusion for the grantee in terms of reporting, but is probably not likely to influence student outcomes and may dilute results


for any possible regular attendees. Grantees should use this time between their summer program and the end of their contract to focus efforts on effectively preparing for summative reporting and evaluation as well as fiscal close-out of their budget.

2. Grantees shared feedback about challenges they experienced during this program year. In several cases, prevalent challenges have been consistent for the past few years. Some of these challenges include parent involvement and commitment to the program, competition with other programs and student responsibilities, and data collection. Grantees also shared interest and need areas for professional development and support.

Recommendation: The state team should continue to collaborate to ensure that professional development through webinars, regional trainings, grantee meetings, and the Extended Learning Opportunities Conference address areas where grantees report challenges and interests. Where possible, additional resources should be considered and developed to support grantees to effectively implement programs that are designed to positively influence change. The state team should continue to use state evaluation findings to identify and address areas of need at the state, cohort, and/or grantee level.

3. Grantees are required to implement a local evaluation that includes contracting with an external local evaluator. Training is provided to grantees relative to local evaluation expectations. However, in collecting local evaluation reports and grantee data, state evaluators noted variance in the content and scope of some of these local evaluations and the degree to which they conform to the expectations set in evaluation training provided to grantees. While grantees are afforded flexibility in how their local evaluation is designed, it appears that some grantees may not be getting maximum benefit from their local evaluation reports as a resource, and some local evaluation reports are missing information that state evaluators believe is important to these reports’ utility as a tool for program improvement as well as a documentation source.

Recommendation: The state team and state evaluators should collaborate to identify a method of efficiently reviewing grantee local reports to note content, strengths, and areas for improvement in order to maximize the value of the local evaluation report as a tool and resource that grantees can use to both document and improve their programs. Strategies for consistently and objectively communicating these attributes to grantees will also need to be identified. State evaluators should review current methods of communicating expectations to grantees and reinforce these expectations during grantee trainings.

4. Several years’ results have been fairly consistent, despite changing grantees.

Recommendation: The state team and state evaluators should collaborate to identify state program priorities or special interest areas in order to: a) plan state-wide training or enrichment opportunities such as webinars, b) use grantee-level results to formally identify grantees that appear to be excelling or struggling so that follow-up can occur,


and c) encourage grantees to examine their local results and take action for program improvement and enrichment. The state team may want to consider a formal process or structure wherein grantees reflect on and respond to their local findings, perhaps using the PA Grantee Report Card that state evaluators prepare for each grantee and share with PDE and technical assistance providers. CONSIDERATIONS FOR GRANTEES

1. Grantees reported that they most often use school, teacher, or parent recommendations to identify and enroll students in programs and similar sources for identifying student needs.

Recommendation: While programs are designed to serve primarily high-needs schools with high-needs students, grantees should consider using more objective sources of data as a primary source for identification, recruitment, and needs assessment, supplementing this with information from teachers, schools, and parents. This data-sourced identification information can be used as baseline information to provide targeted student instruction and evaluate outcomes more accurately.

2. School attendance results indicate notable percentages of students declining. Recommendation: Grantees should review their school attendance outcomes to determine if their results are in line with state results. If grantee results indicate that school attendance is a concern for participating students, grantees should review their program strategies that are intended to influence positive outcomes for school attendance, determine the extent to which these strategies are implemented as designed, and as necessary, implement additional strategies or revise current strategies in order to promote consistent school attendance. If not already a strategy in place, grantees should require and ensure that students attend school on days that they attend the afterschool program.

3. While each outcome area indicates groups of students are improving, these percentages have remained relatively consistent over the past several years, despite changes in grantees and students served. Furthermore, in examining percentages of students improving by grantee, most grantees are showing improvement with less than half of students served, even after excluding those students who did not need to improve.

Recommendation: Grantees should review their local evaluation findings, perhaps with the assistance of their local evaluator, to ensure that they understand what their program results mean. They should then identify areas of strength – and ensure that those areas are continued and possibly expanded and replicated – as well as areas where results are not as positive and identify and implement strategies that are designed to influence positive outcomes for those areas. Grantees should take an active approach to using the wealth of program information and student data available


to them as well as the expertise of their local evaluator in order to make informed decisions about program improvement. Grantees may also want to consider comparing their results to those presented in state evaluation reports for both context and determining areas of local interest that may not currently be examined. Grantees should also consider stronger activities and strategies that may be more likely to contribute to positive student outcomes. Grantees should pay particular attention to students whose results show a decline and those with the most significant needs and provide targeted, intensive strategies designed to support improvement.

4. While this was the first time that historical 21st Century participation was examined, initial findings indicate that outcomes may be positively influenced by multiple years’ participation in the program.

Recommendation: Grantees should reinforce multiple years of participation with students when the grade levels served by the grant permit it. State evaluators will continue to collect this information and reinforce grantee submission. For those students who participated five or more years but are not showing improvement, grantees should examine program strategies to ensure that these students’ specific needs are being addressed.

5. State reporting changes were implemented for 2016-17 based on changes to data availability at the federal level and prior grantee reports and observed challenges. These changes, primarily with how student data was provided, yielded improved accuracy and completeness of both participation and outcomes data. However, in following up with grantees for clarification and error correction, some grantees continue to experience confusion over reporting parameters.

Recommendations: • Grantees should ensure that they and their local evaluator are working from a

consistent and accurate student participation list. Grantees should meet with their local evaluator to ensure both parties understand their roles and expectations for information submission.

• Grantees should review their reports to ensure consistency of information. Completing reports accurately is just as important as completing reports on time. If different parties are completing different reporting components, program staff who are familiar with the overall implementation should review all components to ensure that information is consistent and makes sense.

• Grantees should encourage their local evaluators to participate in state-provided evaluation webinars and training opportunities in order to receive the same information as program staff about report content and expectations.

The evaluation of Pennsylvania’s 21st Century Community Learning Centers programs is intended to provide program results and information that PDE can use to plan for the future and provide technical assistance to grantees. Results are based upon the data available and provided by the program and its grantees.

21st Centers Program 2016-17 State Evaluation Report › Documents › K-12 › 21st Century... ·...

Documents

Transcript of 21st Centers Program 2016-17 State Evaluation Report › Documents › K-12 › 21st Century... ·...