ecp final report

8/14/2019 ecp final report

1/190

Quality Early Learning Key To School Success

A First-Phase 3-Year Program Evaluation Research Report for

Pittsburghs Early Childhood Initiative (ECI)

FINAL REPORT BY:

Stephen J. Bagnato, Ed.D., NCSP,Professor of Pediatrics and PsychologyDirector, Early Childhood Partnerships

Director, SPECS Program Evaluation Research TeamChildrens Hospital of Pittsburgh

The UCLID Center at the University of Pittsburgh

OCTOBER 2002


2/190

Quality Early Learning-Key to School Success

A First-Phase 3-Year Program Evaluation Research Report for

Pittsburghs Early Childhood Initiative (ECI)

FINAL REPORT BY

STEPHEN J. BAGNATO, ED.D., NCSP

JANELL SMITH-JONES, PH.D

GEORGE MCCLOMB, PH.D

JENNETTE COOK-KILROY, M.ED

EARLY CHILDHOOD PARTNERSHIPS

SPECS PROGRAM EVALUATION RESEARCH TEAM

CHILDRENS HOSPITAL OF PITTSBURGH

THE UCLID CENTER AT THE UNIVERSITY OF PITTSBURGH

OCTOBER 2002


3/190

T A B L E O F C O N T E N T S

Acknowledgments

Author Note

Chapter 1:What is the research base on quality early childhood interventions for young children at developmental risk?

Chapter 2:What is Pittsburghs Early Childhood Initiative (ECI)?

Chapter 3:What are the expected success outcomes for ECI?

Chapter 4:How does the SPECS team document the success of ECI?

Chapter 5:What are the research design, questions, and analysis methods for the ECI evaluation?

Chapter 6:How successful was ECI in its first three years?

Chapter 7:Did children benefit from participation in ECI?

Chapter 8:Did families benefit from participation in ECI?

Chapter 9:Did ECI programs achieve quality?

Chapter 10:Was community leadership responsible for the success of ECI programs?

Chapter 11:What are the lessons learned for future ECIs?

Appendix A:Research references

Appendix B:What statistical analyses and results underscore ECI outcomes and conclusions?

Appendix C:Samples of dependent measures in the SPECS evaluation research battery

Appendix D:Sample quality mentoring plan for ECI collaborative program consultation

Appendix E:Letters of correspondence among ECIM, ECI neighborhood leadership councils and SPECS

Appendix F:Organizational capacities of the SPECS Evaluation Team?

Appendix G:Professional profiles for the UCLID faculty on the SPECS evaluation research team?

1

2

3

6

11

14

29

36

42

58

66

73

90

99

104

139

155

159

174

178


4/190

1

A C K N O W L E D G E M E N T S

The SPECS evaluation team is privileged to work with remarkable people in Pittsburghs ECI communities. Providers,teachers, and parents show enormous dedication. Children in the programs inspire with their eagerness and joy in learning. Business, corporate, and foundation leaders have our respect for their vision and their commitment to school readiness and our role in ECI. Perhaps most of all, we are humbled to work with the people associated with the ECI programs in urban neighborhoods, especially the community leaders who have shown such unwavering commitment and creativity. ECI would have been impossible without the unique talents of these partners:

Greater Braddock Early Childhood Network Robert Grom, CEO, Heritage Health Foundation

Wilkinsburg Early Childhood Initiative Leon Haynes, Executive Director

Primary Care Health Services Homewood Early Childhood Initiative Wilford Payne, Executive Director

Sto-Rox Early Learning Network Father Regis J. Ryan, Executive Director

East Liberty Early Childhood Initiative Network Carl Redwood, Program Director, Kingsley Association

SouthSide Early Childhood Initiative Hugh Brannan III, Executive Director, Brashear Association

Highlands Early Childhood Initiative Nancy Kuritzky, Program Director

Hill District Early Childhood Initiative James Henry, Executive Director, Hill House Association

Steel Valley Early Childhood Initiative Penny Dykes, Program Director

Stephen J. Bagnato, Ed.D., Director

George McClomb, Ph.D., Director, Community Strand

THE SPECS TEAMStephen J. Bagnato, Ed.D., DirectorGeorge McClomb, Ph.D., Director, Community Strand

Thanita Adams, MA, MSW, Site LiaisonJennette Cook-Kilroy, M.Ed., SPECS and Program Strand CoordinatorHeidi Feldman, M.D., Ph.D., Health Strand CoordinatorElizabeth Fuchs, BA, CoordinatorKen Jaros, Ph.D., Family Strand CoordinatorMargie Matesa, M.Ed., Site Liaison

Connie Nojeim, BA, Site LiaisonEleana Shair, M.Ed., Data Management CoordinatorJulia Slater, M.Ed., School Transition Strand CoordinatorJanell Smith-Jones, Ph.D., Child Strand CoordinatorKristy Stefero, BA, Research AssistantHoi Suen, Ph.D., Statistics and Research Design, Penn StateCarol Whitacre, BS, Early Literacy Strand Coordinator


5/190

A U T H O R N O T E

Dr. Bagnato and his SPECS Evaluation Research Team were honored to be the recipient of the 2001University of Pittsburgh Chancellors Distinguished Public Service Award for their community-basedprogram evaluation research in the Early Childhood Initiative (ECI) and their leadership in forginginnovative methods of University-Hospital-Community collaboration across Pennsylvania for youngchildren and families at developmental risk in their Early Childhood Partnerships program (ECP).

The SPECS research on ECI is supported by grants from The Vira and Howard Heinz Endowments, Children,Youth, and Families program (A7681) (1997-2004); and The Maternal and Child Health Bureau, Departmentof Health and Human Services, Health Services Resource Administration (5T73MC00036-06), LeadershipEducation in Neurodevelopmental Disabilities, the UCLID Center at the University of Pittsburgh (1995-2005).

Requests for reprints and inquiries about SPECS/ECI or ECP can be directed to: Dr. Stephen J. Bagnato,Ed.D., Professor of Pediatrics & Psychology, Director, Early Childhood Partnerships, Childrens Hospital ofPittsburgh, The UCLID Center at the University of Pittsburgh, 3705 Fifth Avenue, Pittsburgh, PA 15213; email:[email protected].

2


6/190

C H A P T E R 1

WHAT IS THE RESEARCH BASE ON QUALITY EARLY

CHILDHOOD INTERVENTIONS FOR YOUNG

CHILDREN AT DEVELOPMENTAL RISK?

A RESEARCH OVERVIEW

Early childhood educators and researchers have long understood the importance of providing youngchildren with quality early childhood education (Azar, 1999; Guralnick & Bennett, 1987; NICHD EarlyChildcare Research Study 1998; Ramey & Ramey, 1992;1998). Recent findings by the National Institute ofChild Health and Human Development (NICHD) child care study emphasize that child development issignificantly influenced by the quality of child care rather than the amount of time spent in childcareoutside of the home (Azar, 1999). Children in high quality day care programs performed better on tests oflanguage and cognitive skills (Azar, 1999; NICHD Early Childcare Research Study, 1998). In addition, highprogram quality has been an important element in preparing children for the transition to elementaryschool. Many studies show that the quality of childrens child care before they entered school continue toaffect their development at least through kindergarten and perhaps through third grade (Peisner-Feinberg,Burchinal, Clifford, Culkin, Howes, Kagan, Yazwjian, Byler, Rustici, Zelazo, 1999).

Identifying factors related to high quality have been the subject of various early childhood researchstudies (NICHD Early Childhood Care Research Study, 1998; Peisner-Feinberg, et al, 1999). The NICHD studyfound that lower staff ratios, higher levels of caregivers education and training are associated with higherscores of child development (NICHD, Early Childhood Care and Research Study, 1998). Other studies haveemphasized the needs for early literacy activities to be incorporated into early childhood environments to facilitate language development for future school achievement (Anderson, Hiebert, Scott andWilkerson, 1985; Chall, Jacobs, & Baldwin, 1990; Snow and Tabors, 1996). Programs related to basic

cognitive skills (language, and math) as well as childrens behavioral skills in the classroom (thinking/attention skills, sociability, behavior problems and peer relationships) (Peisner- Feinberg, et.al., 1999).Finally, the emotional climate of child care classrooms as well as individual childrens relationships with their teachers are important predictors of childrens outcomes (Peisner-Feinberg, et al, 1999).

Unfortunately, child care programs have many social and economic issues which negatively impact thequality of care delivered to young children (Bryant & Maxwell, 1997; Fujiura & Yamaka, 2000) Issues suchas children in poverty, welfare reform, increased community violence and the increased number ofchildren with mental health, physical health and other special needs within early childhood settingsdelimits the ability of programs to provide high quality care to all young children (Bryant & Maxwell, 1997;Center for the Study of Social Policy, 1991; Fujiura & Yamaka, 2000). Hence, despite increased awarenessof the importance of high quality care, there appears to be considerable variability across child careprograms. Quality standards such as staff/child ratios, classroom size and staff training vary from state tostate (NICHD Early Child Care Research Study, 1998). These variations appear to occur more frequentlywith child care programs serving younger children. Programs serving older children are more likely tomeet common standards of child care (NICHD Early Childhood Research Study, 1998). Children withspecial health/mental health and developmental needs are limited in the access they may have to highquality inclusive programs (Bailey, McWilliams, Buysee,& Welsey, 1998, Welsey & Keyes, 1998)

3


7/190

IMPORTANCE OF INTERAGENCY PARTNERSHIPS

Improving the quality of early care and education requires reforms that extend beyond a single classroomor program to community partnerships and linkages (Buysee, Wesley, Skinner, 1999). Researchers andscholars in all the interdisciplinary fields that emphasize early childhood education advocate for systemchanges that enable agencies, schools, and public and private organizations to pool human and financial

resources and to form innovative partnerships for integrated services. Such partnerships are viewed as the most effective and efficient vehicle to augment their capacity and to integrate their resources to serveall infants and young children including those at developmental risk or with developmental disabilities and their families across the early childhood period (0-8 years).

Some of the most promising of these collaborative ventures in community settings has occurred within the federal Head Start Program and in the school-linked healthcare services and mental health servicesmovement as well as in various states integrated technical assistance networks focusing on youngchildren (Ramey, 1999; Takanishi & DeLeon, 1994; Melaville & Blank, 1997; Bagnato, 1999). Each of theseprogrammatic efforts addressed the specialized needs of children with developmental disabilities orchronic medical and mental health problems.

Despite these few model development efforts, the benefits of the few field-validated University-Hospital-Community partnerships have not been universally realized in the regular early childhood educationsystem. Advocates stress the need for broader initiatives for all young children and families and theprofessionals who support them (Hurd, Lerner, & Barton, 1999). Three areas of need are most prominent:(1) continuing professional development training and ongoing consultation for early childhood teachers,caregivers and administrators; (2) ongoing consultation regarding best practices in early care and edu-cation; and (3) the integration of consultation and services to facilitate the management of young childrenwith challenging behaviors and special medical and educational needs in regular early childhood settings.

QUALITY EARLY LEARNING AND SCHOOL READINESS

The national debate about preventing school failure for young children at developmental risk has renewedinterest in the quality, cost, efficacy, and outcomes of early care and education programs in the UnitedStates (Bryant & Maxwell, 1997; Christian, Morrison, & Bryant, 1998; Clifford, Peisner-Feinberg, Culking,Howes, & Kagan, 1998; Gil & Reynolds, 2000; NICHD, 1999). The accumulated research results of thirtyyears of studies in early childhood intervention indicate clearly that young children at developmental riskfrom impoverished circumstances face progressive declines in their patterns of developmental, behavioral,and learning skills and an early and continuing future of school failure in the absence of structured earlycare and education experiences which can enhance developmental and early school success (Barnett,1995; Bryant & Maxwell; Campbell & Ramey, 1995; Farran, 2000; Marcon, 1999; Schweinhart & Weikart,1997).

Unfortunately, much of the debate about the value of early childhood intervention programs for children ofpoverty surrounds not the issue of quality intervention, itself, but rather the cost of quality (Clifford, et.al.,1998). It is clear, but not universally accepted, that comprehensive early care and education programs arenecessary in order to prevent school failure for children at developmental risk, because the cost of suchintensive programs exceeds the typical cost of daycare.

4


8/190

Much interest and debate surrounds the issue of accountability and its assessment in early childhoodintervention programs (Bagnato, Neisworth, & Munson, 1997; Meisels, Bickel, Nicholson, Xue, & Atkins-Burnett., 2001; Meisels, Burnett, et.al., in press;). Advocates in the fields of early childhood and early inter-vention eschew the tendency to extend downward both the academic standards and traditional testingmethods that pervade school-age practices. It is urgent for the field to conduct research on both assess-

ment and early care and education practices that are developmentally-appropriate and rigorous in docu-menting child progress and the acquisition of precursor skills or early school success.

Finally, the early childhood fields must present evidence-based research on those elements of early careand education practice that best promote positive child outcomes, especially for children at developmental risk and with developmental delays/disabilities (Head Start Bureau, 2000). Two areas of focus are ger-mane to the current study: the impact of ongoing, onsite consultation and mentoring on program qualityimprovements, and the implementation of best practice standards to establish and maintain programquality.

RESEARCH ON EFFECTIVE EARLY CHILDHOOD INTERVENTIONFOR CHILDREN AT DEVELOPMENTAL RISK

Ramey and Ramey (1998) summarized the major experimental studies in the fields of early childhood edu-cation and early intervention since the early 1970s that have resulted in measurable beneficial outcomesfor children at developmental risk. From their analysis, they extracted 7 common elements of effectiveintervention programs that have been associated with initial and long-term positive outcomes for childrenand families. The seven core features are: (1) longitudinal interventions starting in infancy and monitored through functional benchmarks; (2) intensive, comprehensive, and individualized programs and supports:(3) integral parent program participation; (4) high program quality and frequent monitoring; (5) direct childinterventions; (6) community-directed programs and integrated services; and (7) follow-through of childand family supports and program evaluation into the primary grades.

5


9/190

C H A P T E R 2

WHAT IS PITTSBURGHS EARLYCHILDHOOD INITIATIVE (ECI)?

A DESCRIPTIVE OVERVIEW

In 1994, the Heinz Endowments, a nationally renowned Pittsburgh philanthropy organized the business,corporate, agency, and foundation sectors in an ambitious effort to expand quality early care andeducation programs and options for nearly 8,000 unserved children in 80 high-risk neighborhoods. Theoverarching mission of ECI has been to foster preschool and school success for children of poverty whose typical retention and special education placement rates in the early grades have ranged between 18 and40% in Pittsburgh. The Early Childhood Initiative (ECI) has been a unique, collaboratively designed and pri-vately funded joint venture with various Pittsburgh urban neighborhoods to nurture the development ofdiverse early care and education options for young children. Based on the results of both previous localand national research, a consortium of business, community, and foundation leaders designed the goals,approach, and expected outcomes (key performance indicators) of ECI around a business-orientedoperational plan. Despite revisions to the original business and operational plan necessitated by transformations in various state and national policies and unanticipated events (e.g., welfare-to-work;full-day vs. part-day care; infrastructure costs), ECI began enrolling children in September 1998 in diverseprograms in 8 neighborhoods. In 1997, the first ECI participants were 25 children enrolled in the HUDcommunity of Hawkins Village.

The design and implementation of the ECI plan were based on various core efficacy principles summarizedin previous research (Ramey & Ramey, 1998): On-site consultation and mentoring to promote NAEYCprogram quality standards, parent involvement, community-based decision-making, linkages to schoolsand other community resources, child-centered programming, and involvement in an ongoing program

evaluation process to improve quality. In order to receive funding, the leadership councils of communityagencies that operated the ECI programs produced cohesive business and operational plans thatdemonstrated adherence to these core principles to the ECI Oversight Committee, a council of business,foundation, university, community, and corporate representatives who sanctioned each plan. ECI wasdesigned as a six-year longitudinal effort.

6


10/190

The following figure provides a schematic of the major components of the Early Childhood Initiative (ECI) model.

The genesis of ECI was the collaboration among the major public and private sector stakeholder groups in the Pittsburgh region, particularly the Heinz Endowments and the business leaders of the major corpora- tions in the region. With this nexus, an operational and business plan for ECI was generated; counsel wasprovided by the various community leadership groups, including the university, agencies, schools, andlocal neighborhood leaders in the urban sectors.

The basic operational plan that was implemented included a management group for ECI entitled EarlyChildhood Initiative Management with oversight by the stakeholder group. ECIM provided the consultation to community agencies that created their own ECI programs and networks.

In October of 2000 the Early Childhood Initiative was downsized as the original steward withdrew prema- turely and precipitously its commitment to the stakeholder consortium. Heinz-funded Early ChildhoodInitiatives continue to evolve in Pittsburgh, Erie, York and Lancaster. Several programs previously support-ed by ECI have maintained operation on a smaller scale.

OUTCOME BENCHMARKS

ECI is conceived as a local natural experiment whose ultimate objective is to enroll all unserved childrenin high-risk urban neighborhoods into high quality early care and education programs that reflected diverseoptions. Thus, the ECI research is a type of population study in which all children are included in theevaluation and no unserved or untreated group was allowed; each child and program is its own control.

The original ECI plan designed by the business-corporate-foundation consortium established several keyperformance indicators (KPIs) as outcome benchmark categories, culled from the research literature and

7

Pittsburgh's Private & Public Sector Stakeholders & Leaders: Foundations, Businesses, Communities, Universities, Schools, Service Agencies

ECI Intervention ECI EvaluationECIECIM

Consultation

SPECSECI

CommunityPrograms


11/190

which reflected a business approach to monitoring goal-attainment. The KPI categories (specific KPIswere later refined and transformed by the SPECS Evaluation Team) were as follows: Child, family, program,community, and early school success.

COMMUNITY-BASED DECISION-MAKING AND PROGRAM OPTIONS

Various ECI program options were available in ECI neighborhoods based on local preferences establishedby community consensus from the community leadership councils: newly created early care and education centers, previously existing providers, Early Head Start/Head Start centers, early literacy programs,family child care homes, inclusive early childhood and early intervention programs.

COLLABORATIVE CONSULTATION FOR PROGRAM QUALITY

The original ECI operational plan established the policy that program quality as measured by movement toward NAEYC Standards and eventual accreditation would establish the focal intervention yardstickwithin and across all ECI program options. Each community program had some autonomy to establish its

own ECI methodology of intervention beyond this criterion. However, in order to monitor adherence toNAEYC standards and to mentor program administrators and teachers on implementation of best practices, the ECI oversight group instituted a quality control vehicle. The ECIM team consists of local experts fromcounty and state government, professional organizations, agencies, and university groups who providedongoing weekly supervision, monitoring, and mentoring to the caregivers in each ECI program. ECIMstandardized the quality monitoring by using status and feedback data from independent observationalratings on the series of early childhood program quality measures published by Harms and Clifford (EarlyChildhood Environment Rating Scale[ECERS], Infant/Toddler Rating Scale [ITERS], Family Day Care Rating Scale [FDRS]) . Throughout the three-year operation of ECI, all programs received 2 to 4 hours of ECIMconsultation per month. The collaborative consultation process provided by the ECIM team was conceivedfor program evaluation research purposes as the ECI treatment and its impact on child progress,program quality and other benchmarks is being evaluated by the SPECS Evaluation Team. Thus, ECIMconsultation and mentoring is the primary independent variable in the SPECS evaluation research design that is hypothesized to have an impact on child outcome and other dependent variables.

ECIM implemented an ongoing model of onsite consultation and mentoring for program quality, collabo-rative consultation, based on the functional consultation approach which was field-validated in previousspecial education and psychology research. This approach involved a combination of flexible didacticsessions with teachers and caregivers about specific NAEYC standards, collaborative goal-planning toimprove various programmatic dimensions, modeling of best practices by consultants, observations ofcaregiver behaviors, collaborative feedback to caregivers and supervisors, and follow-up observationsand documentation. Such a collaborative approach attempts to operationalize the scaffolding conceptproposed by Vygotsky (1978) that is posed as essential for learners to proceed to the next level of skillacquisition and applied knowledge under the guidance of capable mentors (see sample of theCollaborative Consultation Quality Mentoring model in the Appendix).

8


12/190

CORE ELEMENTS OF THE ECI MODEL

Six overarching programmatic features are common to all ECI program options and distinguish theECI treatment: (a) ongoing consultation to improve program quality; (b) monitoring regarding theimplementation of NAEYC standards, practices, and eventual accreditation; (c) diverse forms of parentparticipation; (d) early care and education routines guided by ongoing child assessments and feedback;

and (e) community-based leadership to marshal creative interagency supports for children and families inECI program settings; (f) longitudinal program evaluation for program quality enhancement, childcurriculum planning, and outcomes research.

WHO ARE THE ECI CHILDREN, FAMILIES AND PROGRAMS?

Children and Families

In the current descriptive and statistical study, developmental outcome data on 1350 urban preschoolchildren and families who participated in the Early Childhood Initiative (ECI) for the longest periods of timewere analyzed. The detailed regression analysis was conducted on n=834 children and families with

complete data sets on all dependent outcome measures. Average length of intervention was 12.3 months(SD = 9.7; range= 7.4 to 22.6 months). Approximately 86% of the children enrolled in ECI programs wereclassified as at-risk using Anna E. Casey (1994) guidelines for distressed communities. Developmentaldelays were identified in 14% of the ECI children based on the Commonwealth of Pennsylvania EarlyIntervention and Special Education Standards (1991) (e.g., 1.5 SD below average or 25% delay of chrono-logical age in one or more domains) and using pre-intervention developmental rates on the DOCS.National incidence rates of developmental delay range from three to eight percent (Fujiura & Yamaki, 2000).Average chronological age of the boys (53%) and girls (47%) enrolled in ECI for this analysis is 3.01 years(SD = .82 yrs, range= .59- 4.00 yrs.). Ethnic mix of children in the urban ECI communities includedAfrican-American (73.4%), Caucasian (24.3%), and Asian and Hispanic (2.3%) representation. Medianyearly family income for the ECI children is $13,024. Reviews of school district web data within the urbanECI communities show the following averages: low income (61%); free or reduced-price lunches (68%); andlimited English proficiency (66%).

Programs, Communities, and Caregivers

Each ECI community leadership council made independent decisions about the type of program optionusing existing and new providers in their local area. Thus, wide diversity in program options is apparentwithin and across the eight ECI neighborhoods of Braddock, Highlands, Hill, Homewood, Steel Valley, SouthSide, Wilkinsburg, Sto-Rox, and East Liberty. The summary averages of the proportion of children partici-pating in each type of early care and education option encompasses Early Head Start and Head Startsettings (36%), inclusive early intervention and early childhood classrooms (34%), family daycare homes(16%), toddler rooms (12%), and infant rooms (2%).Teachers and caregivers in ECI programs have a wide range of professional preparation and experience.Across infant/toddler classrooms, preschool classrooms, and family or group daycare homes, providershave an average of 5.89 years of experience in early care and education (SD= 5.56, range = 0-27 years).Sixty-six percent have 0-5 years experience and 21% have 6-10 years of experience (5%= 11-15 years;

9


13/190

5%=16-20 years; and 3%= 20+ years). Across all program types, providers show the following ceiling lev-els of professional preparation:


14/190

C H A P T E R 3

WHAT ARE THE EXPECTED SUCCESS

OUTCOMES FOR ECI?

KEY PERFORMANCE INDICATORS

As indicated in the original ECI Business Plan, the SPECS Evaluation Team is responsible for revising andrefining the KPIs that are an integral part of appraising the success of ECI. This second revision of the KPIsis based on a further refinement of the benchmarks for success to make the KPIs more objective,measurable and sensitive to the missions of ECI. Moreover, this revision is based on the pilot outcomesfrom our first year of data collection in 1998. Attainment of all ECI evaluation benchmarks in the 6 KeyPerformance Indicator (KPI) categories outlined below will be quantified and monitored by normative (e.g.,both local and national) or criterion-referenced measures using percentiles, standard scores, goal-attainment indices (e.g., percentages), and statistically derived profiles that chart the significance ofindividual patterns of developmental progress over a 3-5 year time period. In addition, external criteriasuch as group consensus measures and extant data from the schools (i.e., kindergarten retention rates;special education placements) will be used as comparative benchmarks to analyze the efficacy of ECI.Finally, we are in the process of collecting randomly selected data from non-ECI programs with similarsocioeconomic status (SES) profiles on all six variables in order to derive a contrast group for compar-ative purposes. The final determinations of success will be based on inter- individual and intra-individualdescriptive and statistical analyses, especially developmental growth curve analyses and associatedregression analyses that identify the interrelationships among child, family, program quality and intensity,community, health/nutrition, early literacy and early school success benchmark clusters.

CHILD BENCHMARKS1. 85% of the ECI children will enter and succeed in Kindergarten without educational support services.

2. After at least 2 years of continuous ECI programming, ECI children will show at least low averageoverall competencies (percentiles of 16-50) in basic developmental skill areas that align with thePennsylvania Revised Academic Standards, including problem-solving, expressive and receptivevocabulary, basic concepts, early literacy (pre-reading, math, writing), socialization, and self-controlbehaviors.

3. As a result of participation in ECI programming, ECI children will show individual patterns of progressin 6 major developmental skill areas that exceed by 50% the expectations projected for them before ECIprogramming began.

4. Those children participating in ECI-funded programming for the longest periods of time will show thegreatest progress and the highest success rate in Kindergarten and first grade.

5. 80% of those ECI children showing evidence of developmental delays (difference of 1 Sd belowaverage) at the beginning of ECI programming will demonstrate average capabilities after, at least,1.5 years of ECI participation.

6. After 3 years of ECI programming, ECI children will demonstrate significant increases in social skillsand self-control behaviors coupled with significant decreases in problem behaviors.

11


15/190

PARENT/FAMILY BENCHMARKS

1. Increased time participating in their childs early care and education program and activities.

2. Increased use of positive parenting skills and discipline in the home.

3. Increases in the number of educational materials and toys used in the home.

4. Increased time spent in weekly educational and enrichment activities with their child both in the homeand in the community.

5. Increases in the number of parents reporting and demonstrating improved personal competence in their play, discipline, teaching, and knowledge of their childs abilities.

6. Decreases reported in the overall amount of family stress reported as a result of participating in ECIprogramming and support services.

7. Increases in the amount of social support reported by parents as a result of participating in ECIprogramming and services

PROGRAM BENCHMARKS

1. Improvements in the physical arrangements of the early care and education setting.

2. Increases in provider knowledge of normal child development expectancies and indicators ofdevelopmental and behavioral problems.

3. Improvements in the availability and use of educational materials and toys.

4. Improvements in the use of family-centered methods to increase and enhance parent participation in the early education program.

5. After 3 years, 85% of ECI programs will consistently use an organized developmentally-appropriatecurriculum of goals and methods to guide teaching, child care, and child progress monitoring.

6. After 3 years of ECIM consultation, 85% of ECI programs will have achieved an overall high qualitystatus ranking on the NAEYC measure of quality benchmarks.

7. After 3 years of ECIM consultation, ECI program administrators will have developed collaborative andworking interagency supports for delivering early intervention, and developmental and behavioralhealthcare services to their children as evidence of a moderate level of program intensity.

12


16/190

COMMUNITY BENCHMARKS

1. For 85% of the ECI children transitioning into kindergarten, direct communications will have beenestablished between the parent, the early childhood provider and the teacher and elementary schoolbuilding principal about the childs strengths and needs and the families priorities for their child.

2. Achievement of a criterion of 75% for the number of neighborhood elementary schools that activelysupport ECI goals across the 80 communities and written procedures for a cooperative transitionprocess between ECI and school entrance.

3. Completion of a collaborative process of social consensus by a focus group of community andpublic school leaders and a written document that rank orders the benchmarks for kindergartenentrance and success.

4. After 5 years of ECI involvement, ECI communities will show a 50% increase in the number of formaland informal interagency or inter-organizational networks that will have developed to support the ECIprograms in the neighborhoods.

5. After 5 years of ECI involvement, 80% of the neighborhood partners will have reached consensus on their individual future goals for ECI expansion and a plan for implementation.

6. Parents who participate actively and for the longest periods of time with ECI will show significantindividual increases in their knowledge and use of formal and informal community resources forsocial support.

7. By the end of the fifth year of ECI, neighborhood consortia advisory boards will have achieved acriterion of 70% accomplishment of the goals on their performance plan submitted to ECIM.

8. By the end of the fifth year of ECI, ECI neighborhood consortia advisory boards will demonstrate amoderately high level on their self-efficacy measures that reveal the strength of neighborhoodconfidence, commitment and resources regarding their ECI programs.

9. After 3 years of ECI program operations, ECI neighborhood consortia will report a criterion level of60% regarding linkages between their program and various formal and informal interagencysupports of their programs.

EARLY SCHOOL SUCCESS BENCHMARKS

1. 85% of ECI children will demonstrate at least low average competencies (16-50%) compared tonational norms on a measure of basic literacy and school skills (BSSI) by the end of theirkindergarten year.

2. 85% of ECI children will reach a criterion of 80% accomplishment of the kindergarten successcurricular competencies on the DOCS upon entrance into kindergarten.

3. Less than 10% of ECI children will be identified as needing special education support services atentry into first grade.

4. Fewer than 15% of ECI children will be retained in Kindergarten compared to historical averagegrade retention rates of approximately 25% for school districts associated with ECI communities.

13


17/190

C H A P T E R 4

HOW DOES THE SPECS TEAM

DOCUMENT THE SUCCESS OF ECI?

After a national competition in 1996, the Heinz Endowments and the Early Childhood Initiative ManagementCouncil composed of business, corporate, foundation, and community members chose an interdisciplinaryresearch team from Childrens Hospital of Pittsburgh and the UCLID Center at the University of Pittsburgh to conduct the longitudinal evaluation of the Early Childhood Initiative (ECI).The SPECS Team (ScalingProgress in Early Childhood Settings) is the independent program evaluation and research group fundedby the Heinz Endowments and other funders to evaluate the outcomes of ECI. SPECS is directed byDr. Stephen J. Bagnato, Ed.D ., a nationally known developmental school psychologist and early childhoodspecialist at both Childrens Hospital and the University of Pittsburgh School of Medicine. Dr. Bagnato isDirector of both the Early Childhood Partnerships program at Childrens Hospital and the Developmental

Psychology Training Core for the UCLID Center (University, Community, Leaders, and Individuals withDisabilities). He is Professor of Pediatrics and Psychology at the University of Pittsburgh School ofMedicine. SPECS has received a continuation grant (in 2001) from Heinz to continue the evaluationresearch of the local and state-wide Heinz Early Childhood Initiatives in Pittsburgh, Erie, York, andLancaster from 2001 to 2004.

Working in collaboration with ECI communities, The SPECS team is one component of a community-basedconsultation, training, and research collaborative (Early Childhood Partnerships) formed by Dr Bagnatowith faculty, staff, and fellows from Childrens Hospital of Pittsburgh and The UCLID Center at the Universityof Pittsburgh, an interdisciplinary leadership training institute funded by the federal Bureau of Maternaland Child Health which specializes in the impact of at-risk circumstances and neurodevelopmentaldisabilities on child, family, program, health, and early school success outcomes, as well as community

needs, resources, and supports.The SPECS Team is conducting a broad-spectrum, longitudinal appraisal of the outcomes and success of theHeinz Early Childhood Initiatives over a five-year period. ECI has implemented a comprehensive model ofintensive consultation to upgrade the quality and scope of early care and education programs, Head Start,and early literacy programs that enrolled 1350 high-risk children and families, in 48 early childhood programs,and 8 communities (25 neighborhoods) throughout Pittsburgh. The central mission of ECI is to prevent schoolfailure for high-risk children and families by enhancing both their experiences in high quality, comprehensiveearly childhood education programs and supportive communities, and their patterns of developmental andbehavioral progress when they enter their kindergarten through third grade school years.Perhaps the primary long-term objective of The Early Childhood Initiative (ECI) is to function as a catalyst

for the formulation of an integrated, seamless, comprehensive, and interagency partnership system of early care and education and support services for all high-risk children and families.

The SPECS research design, evaluation, and analysis strategies enable ECI to monitor the impact of theircomprehensive programmatic methods on five categories of interrelated child, family, program, school,and community outcome benchmarks. SPECS progress benchmarks and measurement strategies match

14


18/190

with the Key Performance Indicators of ECI and with the new Pennsylvania Academic Standards. TheSPECS Team uses assessment techniques that rely on the collection of authentic information in naturalhome, preschool, and community contexts.

SPECS EVALUATION METHODOLOGY

In July of 1997 after a national competition, the SPECS Evaluation Team of Childrens Hospital of Pittsburghand the University, Community, Leaders and Individuals with Disabilities (UCLID) Center at the University ofPittsburgh under the direction of Dr. Bagnato was awarded the multi-year program evaluation researchgrant by the Heinz Endowments to conduct the longitudinal evaluation of the quality, impact, and outcomesof ECI. The SPECS evaluation model and research design were created based on the followingrequirements established by the original ECI business and operational plan:

Assess all children in ECI programs without exclusions for research purposes.

Collect data in the childs natural settings.

Use assessment and evaluation procedures that are compatible with NAEYC standards.

Ensure ongoing collaboration among the teachers, caregivers, and supervisors and the SPECSEvaluation Team in the assessment process.

Establish longitudinal evaluation procedures that provide both formative and summative feedbackin the major KPI areas to caregivers, parents, administrators, and community leadership teams forquality improvement purposes.

Match performance data to the KPIs and the Pennsylvania Academic Standards.

Use data analysis procedures that will determine those combinations of KPI benchmarks thataccount for child progress and early school success.

Based on these parameters, the SPECS Evaluation Team created and implemented a model entitledAuthentic Assessment and Program Evaluation in Early Childhood (Bagnato, 1997, 2000; Bagnato,Neisworth, & Munson, 1997; Neisworth & Bagnato, 2000).

AUTHENTIC ASSESSMENT AND PROGRAM EVALUATION IN EARLY CHILDHOOD

The SPECS research design and evaluation methodology enable the team to monitor the outcomes andimpact of the comprehensive consultation and mentoring model of ECIM and the ECI programming in eachcommunity using the KPIs as benchmarks. The SPECS Evaluation Team uses assessment and evaluationstrategies that rely on the collection of repeated formative and summative authentic data in the naturalhome, preschool, and community contexts. SPECS techniques in the authentic model are innovative infive ways; (1) they match the developmentally-appropriate quality and best practice standards of bothNAEYC (Bredekamp & Copple, 1997), and the Division for Early Childhood of the Council for ExceptionalChildren (Sandall, McLean, & Smith, 2000); (2) they emphasize a check-and-balance system of convergent

15


19/190

and multi-source observations and appraisals of child developmental and behavioral competenciesovertime in everyday routines by teachers and caregivers who know the child best; (3) they do not rely ontable-top testing methods which remove the children from their natural developmental ecology, butrather more appropriate and natural developmental observations and ratings; (4) they sample and feedbackinformation about each childs acquisition of early learning skills within the programs

developmental curriculum that are teachable and predictive of resiliency and early school success; and (5) they offer ongoing feedback to teachers, parents, and community leadership council members aboutchildrens progress and needed enhancements to upgrade program quality and community cohesion.

GENERAL OBJECTIVES AND ORGANIZATION OF THE SPECS RESEARCH

SPECS activities are organized under a two-phase authentic program evaluation research agenda withinfive operational strands that are interrelated: Child, Family, Program, Community, and Early School Success .PHASE 1 (1997-2000)uses a regression design and contrast group studies to analyze the first 3-year resultsand outcomes for ECI. The primary objective is to determine the extent to which participation in qualityearly care and education programs for the longest periods of time results in child success in preschool andschool. The SPECS strands and descriptions follow:

Child Strand... focuses on longitudinal child progress (developmental growth curves) and outcomesfor developmental and behavioral competencies (cognitive, language, social, motor, selfcare, social skills and self-control behaviors); links established with precursors of child-relatedoutcomes in early literacy and early school success strand that align with the PennsylvaniaAcademic Standards (1997); Community StrandConducts qualitative studies on the evolution of more cohesive communityresource networks to support each neighborhoods ECI program. The focus of these studies isindividual profiles of community self-efficacy and workplan performance and, the extent to whichcommunities implement creative plans to link formal and informal interagency resources toenhance their programs for children and families. Program Strand conducts focused research on specific ways that programs increase thequality, scope and intensity of support services for all children, with a focused emphasis on howall types of ECI programs (e.g. infant/toddler, family daycare, centers, early literacy groups, andEarly Head Start/Head Starts) include children with developmental delays and challengingbehaviors into their routines. Family Strandprofiles parent involvement in their childs program, parent preferences for andsatisfaction with their various early care and education program arrangements as these relate toincreasing parenting competencies, decreasing parent stress, and greater family social supports.

Early School Success Strand targets research outcomes studies in two areas: the impact ofquality early learning on the acquisition of early learning competencies at preschool and the sub-sequent achievement and success of children in kindergarten through third grade.

PHASE 2 (2001-2004)involves the continuation of SPECS phase 1 research methods; will use path analysisand structural equation modeling approaches to determine the particular combination of multiple factorsand ECI treatment dosages responsible for child success in the ECI-DP (Demonstration Project forWilkinsburg and Braddock) and Heinz Pennsylvania ECIs in Erie, York, Lancaster, and Central PA.

16


20/190

SPECS EVALUATION RESEARCH DESIGN AND SPECIFIC AUTHENTIC MEASURES

SPECS is based on an authentic assessment and program evaluation conceptual framework and associat-ed methodologies. This unique, non-traditional approach blends norm-based, curriculum-based,observation-based, and qualitative data collected in natural settings by typical caregivers to provide amore representative and real-life portrait of progress and outcomes. SPECS accomplishes the general

purposes of formative (quarterly ongoing quality improvement feedback to providers, parents, and thecommunity) and summative (e.g., yearly progress and impact data regarding child, family, program,community, health benchmarks).

Exhibit 1 characterizes the conceptual and practical limitations of an inauthentic assessment model inearly childhood research and practice (Bronfenbrenner, 1979) which results in strange child behaviorwhen unfamiliar adults and unfamiliar situations and constricted occasions are the contexts for the child.In contrast, authentic assessment and evaluation approaches and methods (see Exhibits 2-4) rely onnatural observations in everyday routines; converge data across settings from multiple caregivers; linkprogram goals, content, and expected outcomes; emphasize intra-individual child progress comparisonsfor formative purposes and both normative comparisons for summative analyses; and align with acceptedprofessional standards for developmentally-appropriate assessment, especially National Association for the Education of Young Children (NAEYC) and the Division for Early Childhood (DEC) of the Council foExceptional Children.

17

Exhibit 1 Exhibit 2

Inauthentic Assessment in Early Childhood

Much of developmental psychology (early childhood assessment) as it now exists is the science of the strange behavior of children with strange adults in strange settings for the briefest possible periods of time.

(Bronfenbrenner, 1979)

Authentic Assessment in Early Childhood

Natural observations in everyday settings and routines vs. contrived testing

Convergent, multisource data from caregivers across settings

Curriculum-based measures linked to program goals, content & benchmarks Intra-individual child progress & Inter-Individual

normative comparisons NAEYC/DEC DAP Assessment Standards

Exhibit 3 Exhibit 4

Developmentally Appropriate Practice: Revised (NAEYC, 1997)Assessment

Reliance on developmental observations Performance on authentic, not contrived activitties Integration of assessment & curriculum Adapting teaching for individual needs Communication with family Evaluation of the program

Developmentally Appropriate Practice: Revised (NAEYC, 1997)Assessment

Content is based on developmental goals Decisions based on multiple sources of data Teacher, parent, & specialist input Child progress based on past performance as

the reference, not group norms Goals across all developmental domains Ongoing observational assessment


21/190

The major features of the SPECS Authentic Evaluation Model underscore the use of a longitudinalmultivariate regression design with cross-sectional studies and the use of Hierarchical Linear Modellingstatistical analysis methods (HLM) or developmental growth curve analyses coupled with the use of amodified-treatment contrast group comparison. The modified-treatment group consists of family daycareand early childhood center programs and children that are not enrolled in the ECI venture and thus do not

receive enhanced ECIM consultation and mentoring for its staff (the ECI treatment). Children enrolled in the contrast group share the same demographics as their ECI counterparts, comprise a group that is stillenrolled in other early care and education programs and are not randomly assigned, and therefore, cannotbe considered a true control group.

The SPECS Conceptual Design Model serves as the underpinning for the longitudinal, repeated measuresdesign using both HLM analyses and contrast group comparisons (Exhibit 5). In this model, ECIMconsultation/mentoring afforded to ECI program administrators and providers is the treatment. ECIM-enhanced programming is hypothesized to exert a direct effect on program quality, particularly teacher-child interactions and other quality dimensions. A direct effect on community participation with theprogram or cohesion is also hypothesized. In contrast, the program (teachers and staff) will exert a directeffect on family participation, child developmental/behavioral progress, learning, and school success.

Families and program staff will implement policies and practices that may affect child health and nutrition.The HLM analysis strategies will use sophisticated regression techniques to determine the interrelation-ships among these hypothesized effects involving child developmental growth curves-child progressand later school success- associated with the impact of ECIM-enhanced programming on increases inprogram quality/intensity, family participation, community cohesion, and health/nutrition status.

18

SPECS Evaluation Conceptual Model

ECIConsultation

CommunityParticipation Health &

NutritionStatus

Learning &School

Competence

Program Quality& Intensity

FamilyParticipation

Exhibit 5


22/190


23/190

SPECS uses a combination of both formative and summative evaluation strategies to document progressand outcomes (Exhibit 10). As will be detailed in the Data Collection section, SPECS Evaluation Teammembers collect assessment data from providers, parents, and community consortia representatives on arepeated measures basis, primarily 2 to 3 times per year. Child status and progress data using specificdevelopmental and behavioral measures are collected quarterly (e.g., September, January, May). After

collection, and trouble-shooting checks on reliability, accuracy, and completeness, the SPECS Child StrandCoordinator with the Information Database Manager generates computerized feedback letters for teachers and parents written from the childs viewpoint and voice (see Exhibit 11). These individualizedletters detail the childs strengths and readiness for the next steps in early learning (i.e., specific goals from the DOCS assessment measure). The teachers and parents then infuse these experiences into the childs typical daily routine at preschool and in the home to enhance opportunities for changes in learning andbehavior. The letters serve a curricular purpose of sensitizing the caregivers to specific developmentalcompetencies that match the expected ECI outcomes. In addition, twice each year the summative resultsare communicated in group meetings to teachers and community consortia board members to provide them with information on apparent changes in the quality of their programs and the changes in children forcontinuous program improvement purposes. The reliability and validity of caregivers ongoing assessments

are ensured in two ways:(1) SPECS team members supervise caregiver assessments during a set week each quarter when

individual modelling occurs and consultation is given. A series of three initial, booster, and folow up training sessions with each teacher occurs until the reach mastery on observation and recording ofskills observed.

(2) Once each year, SPECS specialists conduct random, traditional, but authentic assessments onindividual children as a concurrent validation of teacher/parent assessments (i.e., BattelleDevelopmental Inventory)

20


24/190

DEPENDENT MEASURES, DATA COLLECTION PROTOCOL, AND TIMELINE

SPECS Evaluation strategies are organized currently according to the 5 outcome/benchmark areas orstrands: Child, Family, Program, Community, and Early School Success. The SPECS evaluation researchbattery reflects the convergent and multisource approach described in the authentic assessment and pro-gram evaluation model in the previous section. The following sections illustrate the content of the specif-ic dependent measures that comprise this battery by strand area. The battery below is consistently usedin the core ECI research studies. The following section will provide specific information on the primarymeasures in each strand (see sample measures in Appendix).

Child Strand

The Developmental Observation Checklist System (DOCS; Hresko, etal, 1994) is the primary child develop-ment observation assessment instrument used in the SPECS battery (see Exhibit 13-14). DOCS is a uniquenorm-based measure whose 475 developmental competencies are naturally-occurring child skills (e.g.,recognizes the McDonalds sign; finds the correct toy at the bottom of the toy box; can read simple signs)which are teachable curricular competencies and predictive of school success and sensitive to the effectsof intervention. DOCS samples child skills in the following domains: Cognitive, Language, Social, Motor, and

21

FORMATIVEEVALUATION

SUMMATIVE

AUTHENTIC ASSESSMENT

QUARTERLY COMPUTER FEEDBACK LETTERS: INDIVIDUAL GOAL PLANNING

T1 T10

QUARTERLY PROGRESS REASSESSMENTS & FEEDBACKT1 T10

JanuarySeptember May

Exhibit 10 Exhibit 11 - Developmental Feedback Letter

Exhibit 12

SPECS: Purposes & Goals Conduct authentic evaluation via caregiver Meet NAEYC DAP Standards Link assessment, teaching, progress/program

monitoring & evaluation Implement ecological evaluation scheme Provide formative feedback and ongoing training to

teachers, parents, & community about evaluationmethods and progress

Ensure legacy for providers after ECI

SPECSKEISHA DAVIS

HOW I AM LEARNING AND GROWINGJANUARY 1998 ASSESSMENT

Dear Teacher,I am learning things all the time. Lets go over what Ive beenlearning lately. Well, Im able to do lots and lots of things just likeother kids my age. Here are some things from my last evaluation that Ive been learning and some things I want to learn next.

Some Things Im Learning To Do speak in complete short sentences take turn with toys ask questions using words such as who, what, or where count two to three objects (saying the number as I touch theobject) draw a circle and a cross (0, +) share toys while playing with other children without being asked play a simple board game cooperate while playing with others take turns in a conversation with others name a few letters of the alphabet follow a three-part instruction

Some Things I Want to Do Next repeat from memory a few simple nursery rhyymes, prayers, orsongs correctly give my first and last name when asked use pronouns correctly tell another person when I am happy, sad, afraid, or angry use time words such as morning, afternoon, and nighttime count by rote from one to ten


25/190

Overall. DOCS was nationally (33 states) normed on nearly 1,100 children from birth to 6 years of age.Concurrent, criterion, and predictive validity data are strong (.78-.94) as well as interobserver (parent- teacher= .81) and rating-rerating reliability (.95). Our first 16 months of SPECS/ECI data on 910 childrenreconfirms and provides additional support for the valid use of DOCS with a low SES and high-risk population.For the current study, statistical analyses were conducted on the graduated scoring of the DOCS (e.g., 0= No-

not achieved; 1= Sometimes-getting there or emerging skill; and 2= Yes-fully achieved). Descriptive standardscore data reflect the national norms based only on the total Yes scores in each domain.

The manual for the DOCS shows concurrent validity data and interrater reliability data. Interraterreliability data shows strong overall interrcorrelations between parents and teacher/caregivers (r =. 94).Concurrent validity studies with various traditional measures of language, developmental, and cognitiveskills show moderate to high interrelationships (rs= .35 to .83). Since the DOCS scoring scheme wasmodified for the purpose of the ECI statistical analysis, the SPECS team conducted generalizabilityanalyses on the DOCS to determine the reliability of the composite scores under the modified scoringscheme. Based on the data from a sample of 90 ECI children rated by their caregivers who had been trained in the use of DOCS, the G-coefficient was documented to be .972.

Summary

The DOCS is a comprehensive developmental assessment system composed of 475 items covering the birth to 6 year age range, and normed on nearly 1100 children in 30 states. Developmental competencies on theDOCS are organized into five major functional domains: cognitive, language, social, and motor, and anoverall developmental level. Raw scores and normative (percentiles, and standard scores: M= 100; SD= 15)and age scores are derived in the five domains. The format of the measure recognizes the interactive or transactional nature of development and the interrelatedness of child functioning among the domains(i.e., language and social skills), and allows scoring of skills in more than one domain. Only the DOCS totalraw score was used in the current analysis1.

The DOCS uses an observational strategy of observing and recording (Yes, No) the natural occurrence ofauthentic developmental skills reflective of each domain (i.e., recognizes the McDonalds sign; finds theright toy hidden in the toy box; opens containers to get something inside; identifies the letter B; followsrules in group games). All DOCS items are important and teachable functional skills that align with contentin the developmental curricula of most programs.

SPECS Evaluation Team members have produced (with publisher permission) a computerized version of theDOCS with simplified observational and recording forms that include an additional emerging skill (getting there) scoring category for curricular planning and feedback purposes. Integral to the computerizedDOCS format is a letter written in the childs voice (Hi Mrs. Jones; Im Jimmy; Im 3 years old; next timeI need help doing these things) that summarizes for the teacher and parent the achievements and nextstep skills for the child. The SPECS Team generates such individualized letters for each child for the

teacher and parent each quarter (e.g., September, January, May).

22


26/190

The Preschool and Kindergarten Behavior Scales (PKBS; Merrell, 1994) is an observation rating scale thatemphasizes the focused assessment of social skills and problem behaviors in young children from 3 to 6years (see Exhibit 15). The PKBS stresses social and self-control behaviors that are sensitive to the effectsof intervention and predictive of early school success (e.g., shares toys and other belongings; waits, takes turns; plays with several different children; attempts new tasks before looking for help). PKBS is anorm-based scale whose behaviors are curricular or instructional in content. It was nationally normed on2,855 children and is appropriate for a variety of evaluative and clinical purposes. Validity and reliabilitydata on the PKBS are the strongest of any currently available preschool measure of social skills andbehavior (.81-.98).

23

Exhibit 13 Exhibit 14

Developmental Observation Checklist System (DOCS)

Norms = 1,094 children (0-60 mo.) in 30 states

Norm/curricular hybrid; authentic observations Expects an activity before it happens (0-11 mo.) Repeats own behavior when has effect (0-11 mo.) Talks in own language (0-11 mo.) Responds to come here (12-23 mo.) Understands meaning of 10 words (12-23 mo.) Sits and listens to stories (12-23 mo.) Tries to describe something that happened (2-3 yr.)

Exhibit 15

Preschool and Kindergarten Behavior Scale (PKBS)

Norrns = 2,855 children (3-6 yrs.) Social Skills and Problem Behaviors Sits and listens to stories Shares toys and belongings Attempts new tasks before asking for help Takes turns and plays cooperatively Defies parent, teacher, or caregiver Acts impulsively without thinking

DOCS Sample Items Uses 2-3 word phrases (2-3 yrs.) Tries to say nursery rhymes (2-3 yrs.) Recognizes McDonalds sign (3-4 yrs.) Counts 2-3 objects (3-4 yrs.) Pretend play with common objects (3-4 yrs.) Writes few letters of alphabet (4-5 yrs.) Names 10 printed letters (4-5 yrs.) Knows how many pieces cut in half (4-5 yrs.)


27/190

The Basic School Skills Inventory (BSSI-R; Hammill, 1998) is a norm-based curricular measure of earlylearning and basic competencies that are predictive of school success (see Exhibits 16-19). The BSSI-R iscompleted by teachers based on their observation, knowledge of children, and reviews of the childrenswork performance and portfolios. The scale samples preacademic and academic skills is such areas asreading, math, spoken language, writing, classroom behavior, daily living skills, and social skills. The

BSSI-R was normed nationally on over 1,000 children; its shows reliability and validity data that is adequatefor evaluative purposes (.64-.93).

24


Basic School Skills Inventory (BSSI)

Learning readiness skills for children Authentic teacher observational ratings Ages: 48 to 107 months (Pre-3rd grade) 6 Domains: Spoken Language; Reading; Writing;

Math; Behavior; Daily Living Standard and T-Scores (100/15;50/10) Functional skills/benchmarks for learning Graduated scoring; 0,1,2,3 (mastery) Norms = 757 children; 5 states PRO-ED

BSSI Subscale Samples Spoken Language

Uses complete sentences when talking Listens to and retells a story in sequence Initiates and maintains conversations with others

Reading Recognizes upper/lover case letters Names letters when sounds are spoken Has basic site vocabulary of 5 words


BSSI Subscale Samples Writing

Writes from left to right Writes first name without a model

Writes single letters when asked (b,h,m,t,a,e)

Mathematics Counts objects in set of fewer than 10 Counts aloud from 1-20 Understands concepts of 1st, 2nd, 3rd

BSSI Subscale Samples Classroom Behavior

Makes friends easily

Takes turns Yses teacher feedback to improve learning Can attend to activity for 5 minutes

Daily Living Skills Enters and exits school by self Assumes responsibility for own belongings


28/190

Family Strand

The Parent Behavior Checklist Short Form PBC; Fox, 1995) is a parent-completed report scale designed to determine parenting skills and knowledge and beliefs (20 items) in three core areas: Expectations (childdevelopment); Nurturing (child care, interactions, teaching); and Discipline (behavior management)(see Exhibits 20-23). The PBC was nationally normed on 3,000 parents and includes competencies that

are amenable to parent education and support. Validity and reliability data are moderately strong andsufficient for evaluative purposes.

The Parenting Stress Index- Short Form (PSI; Abidin, 1995) is a parent self-report measure of perceivedparent, child, and family stress (see Exhibit 24). The PSI has been validated in numerous research andclinical studies with disadvantaged and disabled populations. Stress indexes are generated in the areasof Total, Child, Parent, and Life Stress (e.g., my child is so active that I am exhausted; I feel trapped by myresponsibilities as a parent).

25


Parent Behavior Checklist (PBC) Authentic observations of parenting behavior Ages: 12-59 months Norms = 1,,140 children & mothers & fathers;

subsample of 70 with delays 2.9 grade reading level

Parent-child interactions: 3 subscales:Nurturing; Expectations; Discipline Graduated scoring: 4-1 (Always-Never) PRO-ED

PBC Subscale Samples Expectations

My child should be able to use toilet w/o help. My child should be able to ride a tricycle. My child should be old enough to share toys.


PBC Subscale Samples Discipline

I would spank my child in public for bad behavior. I yell at my child for whining. I send my child to bed as punishment.

PBC Subscale Samples Nurturing

I play make-believe with my child. I read to my child at least once a week. I praise my child for learning new things.


29/190

Family Support Scale (FSS; Dunst & Trivette, 1992) offers a survey of parent perceptions regarding the typeand extent of personal and social support that is provided to parents of children with special needs.

Program Strand

The Early Childhood Environment Rating Scale-Revised (ECERS; Harms & Clifford, 1998) and Infant /Toddler Environment Rating Scale, and Family Daycare Environment Rating Scale. The ECERS, ITERS,and FDRS system of program quality scales have a long and distinguished history in early care andeducation. They are the most widely used scales for evaluative, and quality feedback purposes. TheECERS system is criterion-referenced and is supported by extensive field research. Areas sampled by

direct, naturalistic observation include: personal care routines, furnishing and display for children,language-reasoning experiences, fine-gross motor activities, creative activities, social development, andadult needs.Program Quality Profile for Early Childhood Settings (Cook-Kilroy & Bagnato, 1997) offers a graduatedobservation rating scale of status and changes in the quality of various domains of program elementsincluding the responsivity of the learning environment, teacher-child interactions, parent participation,curriculum planning, accommodations for special needs, the physical setting. The PQP was designed forall types of early childhood settings and is based on the NAEYC revised standards.

Community Strand

Community Evaluation Battery: Efficacy Scales, Performance Scale (McClomb, 1997) The CEB is a SPECS-developed evaluation system designed to document the capacity of community consortia board members to organize their resources and efforts to accomplish workplan goals, to accomplish their goals with asense of personal and collective confidence and effectiveness, and to support their ECI programs. Thescale has been field-validated on 28 communities in high-risk neighborhoods using data over 2 timepointswith 68 individuals.

26

1. When my child wants something, my child usually keeps trying to get it.

2. My child is so active that it exhausts me.

3. My child appears disorganized and is easily distracted.

4. Compared to most, my child has more difficulty concentratedand paying attention.

5. My child will often stay occupied with a toy for more than 10 minutes.

6. My child wanders away much more than I expected.

7. My child is much more active than I expected.

8. My child squirms and kicks a great deal when being dressed or bathed.

9. My child can be easily distracted from wanting something.

10. My child rarely does things for me that make me feel good.

11. Most times I feel that my child likes me and wants to be close to me.

Exhibit 24


30/190

Evaluation TimelineThe repeated measures timeline for the SPECS evaluation protocol is summarized in Exhibits 28-30. Allobservation and rating scale assessments in the SPECS evaluation battery are repeated 2-3 times peryear based on the time frame established in each strand. Child assessments recur each September,January, and May. Repeated assessments for each child form a developmental growth curve for indi-

viduals and groups that are analyzed using the Hierarchical Linear Modeling (HLM) and related statistical techniques.

Over the six year longitudinal study, a maximum of 18 time points are projected for children who enteredECI as infants. Based on current data for the first 33 months of ECI operation, children on average areexpected to enroll in ECI at approximately 3 years of age and will be assessed over 6-9 time points.

Assessments in the Family, Program, and Community strands are conducted 1-2 times per year and alignwith the child assessments during those periods. The array of subdomains sampled by SPECS evaluatorsare outlined in the exhibits (31-33).

27


SPECS Evaluation Protocol and Timeline (1997-2002)

MEASURE/DOMAINS TIMELINE Sept Jan May

Child Strand DOCS X X X PKBS/SEEC/TRIAD X X X D-Specs/Infant-Specs X X XParent/Family Strand PBC X X PSI X X FSS X XHealth/Nutrition Strand Extant health records/improvements X X

SPECS Evaluation Protocol and Timeline (1997-2002)

MEASURE/DOMAINS TIMELINE Sept Jan May

Program Strand PQPECS X X Program Intensity X X Consultation-Intensity X XCommunity Process Strand Individual Efficacy Survey X X Organizational Capacity & Performance X X Extant health records/improvements X X

Exhibit 30

SPECS Evaluation Protocol: K-1 Achievements

CHILD DOMAIN 3rd Quarter BSSI-R X Group Achievement Tests X ECI Key Performance Indicators X PA Academics Standards X


31/190

CONFIDENTIALITY OF DATA AND SAFEGUARDS

The SPECS evaluation proposal was originally reviewed and accepted for implementation by the HumanRights Committee (HRC) of Childrens Hospital of Pittsburgh in 1997. Yearly renewals after interim progressreviews of the original HRC proposal have been conducted since 1997.

The original consent form for SPECS was constructed in collaboration with ECIM so that the language ofour hospital-based consent was simple and non-threatening for parents and was compatible with theoverall ECI consent from the steward. The consent form acknowledges that individual child informationwill be shared with parents only and with their teachers with their agreement in the case of the individualcomputer-generated letters from DOCS. To safeguard confidentiality, all child, family, and other SPECSstrand data are numbered by child/family or community or program so that individual identifiers are secure.The R-Client software package and database maintains this security except when parents requestindividually generated child data for advocacy purposes (i..e., to help parents secure child access toearly intervention support services due to suspected delays). All data is aggregated for statistical analy-ses and individual community data is identified only by number that is known only to the InformationDatabase Manager and Principal Investigator (SJB).

28


SPECS Primary Domains: Dependent Variables SPECS Primary Domains: Dependent Variables Program Strand Health and Nutrition Strand C urriculum Planning Immunizations Responsive Learning Primary Care AccessEnvironment Health Insurance

Developing Relationships Diet/Nutritionwith Children & Parents Knowledge Health, Nutr ition, Safety Universal Health Staff & Administration Precautions Program Intensity Anemia & TB Consultation Support & Intensity Screenings Reading/Writing Mathematics Self=Care

Exhibit 33

SPECS Primary Domains: Dependent Variables

Community Process Strand Individual Efficacy: Personal Goals,

Expectations, Perseptions@ ECI Successin Neighborhood Group Organizational Efficacy: Coalition Goals,

Expectations, Perceptions @ ECI Success in Neighborhood Indiviaual Neighborhood Strategies, Capacity, & Goal

Attainment Performance in ECI Implementation Mapping Community Resource Networks

Child Strand Parent/Family Strand Cognition/Basic Concepts Parent Expectations Language Discipline Social-Emotional Nurturing/Teaching

Motor Stress Physical/Sensory Social Support Self-Regulation Child Development Social Skills Child Behavior Problem Behaviors Reading/Writing Mathematics Self=Care


32/190

C H A P T E R 5

WHAT ARE THE RESEARCH DESIGN, QUESTIONS, AND

ANALYSIS METHODS FOR THE ECI EVALUATION?

RESEARCH DESIGN, QUESTIONS, HYPOTHESES AND VARIABLES

As stated previously, the SPECS research design is a longitudinal regression scheme and analysis comple-mented by a constructed comparison group analysis, and a contrast group analysis. The SPECS researchdesign poses questions and hypotheses based on variables that were identified by the original ECIManagement (ECIM) Group and refined by the SPECS team. Two sets of Program Evaluation ResearchBenchmarks serve as the criteria for monitoring impact, quality, outcomes, and efficacy: Key PerformanceIndicators and Research Questions.

RESEARCH QUESTIONSThe following research questions guided the current first-phase analysis of ECI: 1) Are child developmental progress and enhanced developmental trajectories associated with participation in high quality (i.e.,NAEYC standards) early care and education programs? 2) Is the ECIM collaborative consultation and mentoring approach focusing on NAEYC standards associated with enhanced child development outcomes?

First of all, Exhibits 31-33 provide an overview of the research variables that underlie the SPECS/ECIevaluation research in the benchmark outcome categories of Child, Family, Program, Community, and EarlySchool Success, and and various transactional hypotheses. For example, targeted child variables includelanguage and social/self-control behavior skills; family targets include participation in their childs programand parenting behaviors; program quality variables stress curriculum implementation and individualization

of care and education; community variables stress cohesion and group effectiveness in decision-makingand work performance. School transition will target success indicators such as lower retention rates orspecial education rates. As indicated below, The HLM longitudinal analysis will target interrelationshipsamong variables such as hypothesized effects among high program quality, high parent participation,supportive community networks, and greater child success in preschool and at kindergarten entrance.

Related to these variables, the following overarching research questions guide the SPECS evaluation of ECIand the HLM longitudinal analyses.

29


33/190

SPECS Research Questions

Does ECI-enhanced programming result in positive child developmental, behavioral, and early schoolsuccess outcomes that exceed maturational expectations?

Does ECI-enhanced quality mentoring result in significant programmatic improvements in ECIprograms?

If ECI children show positive developmental and behavioral progress patterns that exceedmaturational expectancies and greater school success, what specific programmatic, family, andcommunity dimensions predict these effects?

DATA PROCESSING AND STATISTICAL ANALYSIS

The Early Childhood Partnerships evaluation database (R-Client for Children) developed by Great LakesBehavioral Research Institute, is a windows-based client management information system that includesdata file storage and retrieval, reporting, and file transfer capabilities. The following data are collected andmaintained:

Identifying data, basic demographic data

Evaluation data

Administrative data

Dates related to events

Data are maintained in the database that is housed in a secure (password access) network drive on theChildrens Hospital of Pittsburgh server. The SPECS evaluation team can access information on thedatabase via standard reports or customized reports written by the teams Information Manager. ChildrensHospital of Pittsburgh Computer Information System (CIS) is responsible for daily backup of information on the designated drive. In addition, the SPECS Team Information Manager creates a backup on the localdrive of the SPECS Team computer designated for the database.Data sets are exported in ASCII file format and given to the statistician for statistical analysis. All data setsexported for analysis are stripped of identifying information. A case number is assigned to each clientregistered in the database. All programs and neighborhoods are given codes. All coded data fields havebuilt in verification. All collected data forms are reviewed by SPECS team researchers before data entry.Processed data are also reviewed by the researchers for accuracy. Missing and incomplete data arerecorded in the database.

STATISTICAL ANALYSIS

For the overall analytic design of SPECS, there are basically five categories of potential analyses: 1) Initialpsychometric analyses of assessment procedures through the Generalizability method and other validationprocesses as appropriate; 2) if feasible, initial optimization analyses to identify the most cost-effectivecombination of assessment methods; 3) second though fifth year annual ordinary least square regressions to evaluate the impact of ECI on programs, families, and communities; and the constructed comparisongroup analysis; 4) second through fifth year annual HLM analyses to evaluate the impact of ECI, programs,families, and communities on child outcomes; 5) second through fifth year annual HLM analyses to evalu-ate the impact of ECI, programs, families, and communities on child growth.

30


34/190

1. Generalizability Theory and Analysis

An important potential threat to the validity of any of the outcome measures is the existence of a largeamount of random measurement errors which reduce the validity of the outcomes measures, it canobscure the observation of impact of ECI on program, family, community, and child outcomes; much in thesame way that a great deal of static noise on the radio drowns out the music we wish to hear. Therefore,

it is important to identify the potential sources of measurement errors early on and take steps to minimize them. The Generalizability analyses are designed to identify potential sources of measurement error and to evaluate the reliability and validity of any assessment procedure. This method is different from the morecommon classical reliability analysis methods (e.g., Cronbach alpha, test-retest reliability) in that theclassical methods are appropriate only for standardized objective tests. The Generalizability approachcan accommodate these standardized testing situations as well as other more complex naturalisticobservational assessment procedures, including the multi-faceted, authentic, and observational perform-ance assessments, judgment-based assessment, and convergent assessments used in this project.Through this analytic methods, we seek to identify the potential sources of error and the reliability for eeach of the outcome measures proposed, identify the sources of error and reliability for other potentialalternative measures, and devise convergent evidence of validity for these measures. Through these

analyses, we examine the relative quality of different methods assessing program, family, community, andchild outcome measures.

2. Optimization

In general, assessment is a sampling, probing process. The more performances we sample, the moreraters used, the more items and dimensions we explore and so on, the better the quality (reliability andvalidity) of the assessment outcome measures. A classical expression of this idea for objective testing issomething called a Spearman-Brown prophecy which essentially states that in a standardized objective test, the more items there are in a test, the more reliability the overall score. However, the more items,more raters, and the more performances would also imply the higher cost of assessment in terms of

expense, rater time, child time spent on assessment, development time, and so on. There needs to be abalance somewhere to gain the optimal quality of data at the lowest possible cost. In a complexmultidimensional assessment setting such as the convergent assessment used in this project, reliabilityand validity can be maximized through many different ways: increasing the number of raters, usingcomposite scores only, increasing the number of probes, and so on. Each of these strategies will implydifferent costs and each will lead to a different improvement of the quality of data. If it is feasible togather all the necessary data, a series of optimization analyses (most likely using a method called thebranch-and-bound integer programming method) can be conducted to final the optimal assessment design that will yield outcomes measures with the best possible and validity while keeping cost to a minimum.

3. Ordinary Least Square RegressionsIn an experimental design, the data analysis procedure often follows logically the data collection design.The data analysis procedure is also divided into two aspects: 1) describing some impact or relationshipand 2) ruling out random chance as the explanation for the observed impact or relationship. The formergenerally involves a descriptive statistical analysis and the latter and inferential statistical analysis.

31


35/190

A classical true experimental design is a data collection design and not a data analysis design. In thisdesign, the researcher deliberately manipulates a treatment variable such that some randomly selectedsubjects obtain the treatment while others do not. The most common descriptive data analysis procedurefollowing this data collection design is to compare the mean scores on the outcome measure between those who receive that treatment and those who do not receive that treatment. The most common infer-

ential data analysis procedure for the experimental data collection design is either a t-test or an analysisof variance. That is, if the ECI evaluation project is to use a true experimental data collection design, somerandomly selected children would receive ECI treatment while others do not. We would then most likelyanalyze that data by reporting the means and standard deviations of the experimental and control groupsubjects so that we can conclude which group scores higher than another group. We would also rule outchance by conducting t-tests for those comparisons involving only treatment vs. control groups and byconducting analyses of variance for those comparisons involving several different treatments (e.g., the fourdifferent types of early childhood programs).

The ECI project will not be a true experiment because subjects cannot be randomly assigned children arealready in programs, communities, and families. We may randomly select programs but not children in theprograms to implement treatments. That is, once a program is chosen to receive ECI, ALL children in the

program will receive ECI with no within-program random assignment. In other words, we have a clustersampling situation. Also programs, families, and communities are not randomly chosen to receive or notreceive treatment. Rather, they are self-selected by proposing a particular set of activities. Therefore, wedo not have a classical true experimental data collection design in that there is not control or manipulationof treatment on the part of the researcher and that there is no randomization of programs and subjects.This situation is best described as a quasi-experimental design in that there are comparison groups.Because of the lack of randomization, we are not sure if there are any initial differences in program out-come measures, family outcome measures, community outcome measures, or child outcome measuresbetween ECI and non-ECI groups. The most often used inferential data analytic procedure for this situation is an analysis of covariance, which essentially examines whether we can rule out chanceafter accounting for initial differences statistically . The corresponding descriptive data analysis procedure is tocompare the adjusted means scores of the ECI and the non-ECI groups; after adjusting for initial differ-ences.

Overall, for inferential data analysis, t-test is a special case of analysis of variance and analysis of varianceis a special case of analysis of covariance. Similarly, for descriptive data analysis, comparing two meansis a special case of comparing more than two means and comparing unadjusted means is a special caseof comparing adjusted means.

The inferential data analytic portion of regression is primarily an analysis of variance (which may accom-modate several treatment or predictor variables and several covariates or initial difference variablessimultaneously if desired). In other words, the inferential data analytic portion of regression is simply amore flexible generalization version of what we use (i.e., t-test, ANOVA, ANCOVA, RMANOVA) in classical

true experimental designs or quasi-experimental designs. The descriptive data analytic portion of regres-sion is primarily a prediction equation through which we define how the changes in the predictor variableswill lead to changes in the outcome (called criterion) variable. This again is basically an extension of thedescriptive data analytic procedure we use may describe the outcomes in this fashion: The experimentalgroup has a mean of X1 and the control group has a mean of X2. Alternatively, we can say the same thing

32


36/190

through: If you are in the experimental group, I expect your outcomes scores to be X1 and if you are in the control group, I expect your outcomes score to be X 2. An analysis of variance is no more than anextension if this: If you are in group A, I expect X1 ; in group B, I expect X2 and in group C I expect X3 andso on. Regression is a further extension of this in that we can predict outcomes in the form of an equation that may involve more than 1 treatment or predictor variable. Thus, we would say things like: If your

income is at this level, and you have receive Head Start, and that your parents are highly involved, I expectX1 . On the other hand, if your income is at this other

ecp final report

Documents

Transcript of ecp final report