Issues in Using Assessments in Accountability Systems for ...ectacenter.org/~pdfs/eco/OSEP...

Early Childhood Outcomes CenterEarly Childhood Outcomes Center

Issues in Using Assessments in Accountability Systems for

Young Children with Disabilities

Issues in Using Assessments Issues in Using Assessments in Accountability Systems for in Accountability Systems for

Young Children with Young Children with DisabilitiesDisabilities

Kathy HebbelerECO at SRI International

OPEP Think Tank on AssessmentWashington, DC February 2007


ObjectiveObjective

Provide background information relevant to the discussion, including what the ECO Center has learned about the current state of assessment practices through our work with states


AssumptionsAssumptions

Children 0-5 participating in IDEA programs also will be participating in the required OSEP reporting (approx. 1 million children)Some of these children also may be participating in other assessment systems

OSEP Reporting

Child Care

Head Start

State Preschool

Participation in Multiple

Accountability Systems??


Possible uses of assessment in Possible uses of assessment in EI/ECSEEI/ECSE

Eligibility determination Norm-referenced test % delay

Individual program planningCurriculum-based tools – e.g., LAP, Carolina

Ongoing progress monitoringCurriculum-based tools?

Accountability assessmentE.g., Head Start reporting system


Issue: Variation in how assessments Issue: Variation in how assessments are usedare used

Do not know how many EI/ECSE programs routinely use assessment tools for anything other than eligibilityIn some programs, the only formal assessment is for eligibilityAppears to be much state-to-state variation


Issue: Thinking on assessment has Issue: Thinking on assessment has changed in general early childhood changed in general early childhood communitycommunity

Major changes in last 15 years in how assessment of young children is viewedOld position: Do not test little kidsNew position: Ongoing assessment is part of a high quality early childhood program


What changedWhat changed

New and different tools became available for general EC

Curriculum-based assessments were developed, e.g., Creative Curriculum, Work Sampling, etc. Tools for 3-5 came first; 0-3 tools are coming now

Interesting sidebar: Curriculum-based assessments for programs serving children 0-5 with disabilities have been around for years



The purpose of assessment was redefinedNot about: sorting, labeling, using to deny accessNow about: Getting a rich picture of what children can do and can’t do and using that information to help them acquire new skills

“progress monitoring”



Assessment had always been seen as a process with multiple purposesDistinctions have been made between good and bad uses of assessment with young childrenGood uses are now promotedFor more information: NAEYC web site (Position statement on Curriculum, Assessment and Evaluation)


Interesting IronyInteresting Irony

Even though the disability community had developed many curriculum-based assessment tools, currently [many? some?] programs do not practice ongoing assessmentThe push for ongoing assessment to monitor how a child is doing and plan for instruction/intervention is coming from the general education community


Issue: Use of existing Issue: Use of existing assessments/multiple purposesassessments/multiple purposes

Can assessments already being used by programs for other purposes (whatever they are) be used for accountability purposes?Does this apply to all tools or only some categories of tools?


Issue: Limitations of existing Issue: Limitations of existing assessment toolsassessment tools

“Assessment of young children poses greater challenges than people generally realize….assessment results—in particular, standardized tests, that reflect a given point in time—can easily misrepresent children’s learning…There is widespread dissatisfaction with traditional norm-referenced standardized tests which are based on early 20th century psychological theory.”

National Research Council, 2001


Problem: Nature of the young Problem: Nature of the young childchild

Not well suited to a standardized testing situationPerformance varies from day to day, place to place, person to person Don’t perform well for strangers or on demand Growth is sporadic and uneven


Problem: Response capabilities Problem: Response capabilities of children with disabilitiesof children with disabilities

Same issue as with school-age children: assessment assumes child who can see, hear and understand spoken language, point, etc.Few assessments include accommodationsVery little data on validity of accommodations with young children


Problem: Impact of Problem: Impact of disability/delay on development disability/delay on development

Typically developing children tend to develop in multiple areas simultaneously

Language, cognition, motor skills march forward more or less togetherEven though development has been divided into domains for assessment and research, much of development is intertwined These interconnections present challenges for obtaining a “pure” domain score


Problem: Impact of Problem: Impact of disability/delay on development disability/delay on development

More difficult to accurately portray the development of children developing atypically with available assessments, esp. children with language delays

Do they understand the directions?Is the assessment tapping cognition or language?


Problem: Psychometric properties of Problem: Psychometric properties of existing instrumentsexisting instruments

Some of the most common instruments are being used with limited or no reliability and validity data None have validity or reliability data reporting when used for outcomes and accountability


Response: New forms of Response: New forms of assessmentassessment

Growing recognition that the only way to get a valid picture of what a child can do/does to is look at performance over a variety of settings and people including what the child does spontaneously with familiar adults Can’t base conclusions about child’s capabilities on elicited responses alone “Authentic assessment”


Response: Use multiple Response: Use multiple sources of information (best sources of information (best practice)practice)

“A single test, person, or occasion is not a sufficient source of information. This means that we must gather information from several sources, instruments, settings and occasions to produce the most valid description of the child’s status or progress”---DEC Recommended Practices


Issue: Multiple sources of Issue: Multiple sources of informationinformation

And just how is that information supposed to be put together?


Issue: Validity vs. credibility Issue: Validity vs. credibility dilemma for accountabilitydilemma for accountability

Strangers can’t elicit valid data on young children’s performance capabilities in a testing situation

BUTcan data produced by those who know

the child and whose programs are being evaluated, be credible in an accountability system?


Issue: Validity and reliabilityIssue: Validity and reliability

Are not characteristics of an assessment per se Validity – context dependent on the use of the results

Individual vs. group decisions

“Validity , the degree that an assessment measures what it purports to measure, relates to the use of the test, rather than the test itself.”

Score Reliability, Pg. 113


Issue: Reliability and validityIssue: Reliability and validity

Reliability is a characteristic of a set of scores, not of a test“…reliability refers to the degree of consistency of the information obtained from an information gathering process”“…reliability of the scores provided by an instrument or procedure may fluctuate depending on how, when, and to whom the instrument or procedure is administered..”Joint Committee on Standards for Educational

Evaluation, 1994Quoted in Score Reliability, pg.95


Issue: Validity in an Issue: Validity in an accountability systemaccountability system

Do the assessment results lead to the “right” decisions?

CCSSO framework

How does one measure validity in an accountability system?What should be the measures of the validity of the OSEP data?


Issue: Existing assessments are Issue: Existing assessments are domainsdomains--basedbased

OSEP outcomes are functional and cut across domainsExisting assessments provide scores for domains, not 3 outcomesExisting assessments vary in the extent to which they assess functioning vs. isolated skills


Outcomes Are FunctionalOutcomes Are Functional

Functional outcomes: Refer to things that are meaningful to the child in the context of everyday livingRefer to an integrated series of behaviors or skills that allow the child to achieve the important everyday goals


Functional Outcomes are Functional Outcomes are NOTNOT

A single behavior

The sum of a series of discrete behaviors or splinter skills such as…..

*Knows 10 words *Pincer grasp (picks up a raisin)

*Smiles at mom

*Stacks 3 blocks

*Goes up and down stairs with one foot on each stair


Issue: Variation in provider Issue: Variation in provider knowledge of assessmentknowledge of assessment

(Based on ECO work with states)Some providers are skilled in administering and interpreting multiple assessments, some in one, some rarely use any.Many children served in programs for typically developing children where knowledge and use of assessment is limited. How will providers be trained and supervised?


Issues related to OSEP Issues related to OSEP accountability accountability

States must report data on all children with IFSP/IEPs in the 3 outcomes areas, even if the child has no problems in an area and is not receiving services

E.g., many preschoolers are receiving services only for a communication problem, possibly only an articulation problem, so they are not routinely assessed in other areas (even for eligibility at program entry)

Can a screening tool be used in an accountability system to verify that children are maintaining functioning comparable to same aged peers in those areas?


Issue: Role of families in the Issue: Role of families in the assessment processassessment process

Families provide a unique perspective on the child’s functioningFew assessment tools have procedures for incorporating that perspectiveNeed good tools/procedures for learning about child from the family


Issue: Role of families in the Issue: Role of families in the assessment processassessment process

Programs vary in how much and how they share assessment data with families, especially with regard to communicating developmental ages or extent of a child’s delay.Some providers are “soft pedaling” the assessment resultsProviders need training in sharing results with families


Bottom LineBottom Line

States will be reporting data on outcomesThese data could be critical to federal level (and, in many states, state level) policy decisions related to Part C and Part B Preschool in the future


Bottom LineBottom Line

What needs to happen to make sure assessments make a meaningful contribution to improved outcomes and program improvement?What can be done to insure assessment data used for outcomes measurement are:

MeaningfulValidReliableCredible?

Issues in Using Assessments in Accountability Systems for ...ectacenter.org/~pdfs/eco/OSEP...

Documents

Transcript of Issues in Using Assessments in Accountability Systems for ...ectacenter.org/~pdfs/eco/OSEP...