Illustration of a Validity Argument for Two Alternate

Assessment Approaches

Presentation at the OSEP Project Directors’ Conference

Steve FerraraAmerican Institutes for Research

August 1, 2006

2

Goal

Illustrate planning for the validation process for large-scale assessments using standards-based alternate assessments from two states

Use selected examples from the paper Standards and Assessment Approaches for Students with Disabilities Using a Validity Argument

3

Over-arching concept

Provide evidence to support intended inferences about students

Consideration of assessment design (i.e., tasks, administration conditions, scoring)

Plans for collecting procedural and empirical evidence

Important specific principles and recommendations in the paper

4

Examples from the paper

Massachusetts alternate assessment portfolio (Weiner, 2002)

Oregon performance tasks (Tindal et al., 2003) Mathematics, grades 3-5 Intended inferences

The assessment adequately reflects the domain of knowledge and skills for the construct

The assessment accurately identifies students’ level of proficiency in mathematics

5

Procedural evidence

Test design and development process Quality of the items and tasks Assemblage of items/tasks/evidence into an

assessment Administration and scoring process

6

Empirical evidence

Alignment between the alternate content standards and the assessment items/tasks/evidence (and linkage to grade level/band standards)

Item/task functioning Reliability of scoring and test score interpretations Internal relations among items and tasks Response processes External relations with other measures

7

Target math standards

Massachusetts standards Grades three and four standards that focus on

number sense (seven objectives) and operations (three objectives)

Oregon mathematics standards Numbers, Computation and Operations—Grades

four to five

8

MA: Possible assessment strategies and portfolio products

Addressing access skill(s) (skills embedded in academic instruction)

Alice participates in this activity by assembling money envelopes paired with pictures. Alice works with a classmate who counts the money needed for each item and helps Alice place the correct amount into its corresponding envelope. Alice exchanges these envelopes when making a purchase.

9

Possible portfolio products (cont.)

Teacher note describing the work accomplished by Alice and her classmate

Data collected on Alice’s ability to assemble money envelopes and exchange correct envelopes when making a purchase

Videotape of Alice making a purchase Alice’s choice of money envelopes selected

for her portfolio

10

Oregon item

Standard: Read, write, order, model, and compare whole numbers up to 1,000,000, common fractions, and decimals up to hundredths.

Practice Item 24: Find the missing number in the pattern. 2.6 5.2 ___ 20.8 (A) 7.8 (B) 10.4 (C) 13.0 (D) 15.6

Alternate Assessment Task 11: Order Numbers Present the number cards in this order: 3, 1, 8, 6. Say: Place these numbers in order from smallest to largest.

11

Summary

(MA) Assemble money envelopes with a classmate, make purchases Teacher observations of Alice working, videotape

of making purchases (OR) Order numbers from smallest to largest

12

Test development process

Massachusetts Oregon

Does the actual evidence described in the possible assessment strategy fully reflect this construct?

Have the tasks been adequately developed and assembled into an alternate assessment?

13

Test administration and scoring

Massachusetts Oregon

How well conducted are the test administration and scoring procedures?

Are teachers sufficiently trained in administering and scoring the tests (especially because responses may be scored as partially correct and not just correct or incorrect)?

Does the student independently complete work or is the teacher part of this process? If so, to what extent?

14

Alignment and construct representation

Massachusetts Oregon

Are enough pieces of evidence present to represent the domain and avoid construct under-representation?

Are enough tasks present to represent the domain and avoid construct under-representation?

How closely is the alternate assessment aligned to the state content standards in categorical concurrence, depth of knowledge, range of knowledge, and balance of representation?

15

Rater accuracy and score reliability

Massachusetts Oregon

How accurate is the scoring by trained professional scorers?

How accurate is the scoring by trained teachers?

How consistently and accurately do scores categorize students into performance categories?

Which facets of the assessment process influence scores most (e.g., tasks, raters, administration conditions, occasions)?

16

Conclusion

The same types of validity questions apply for all (alternate) assessment approaches How the questions are posed and the evidence

relevant to those questions may differ Intended inferences, corresponding validity

questions, and evidence: Identify during the conceptualization, design, and

development process Pursue during development and as part of

implementation

17

References

Tindal, G., McDonald, Tedesco, M., Glasgow, A., & Almond, P., Crawford, L., Hollenbeck, K. (2003). Alternate assessments in reading and math: Development and validation for students with significant disabilities. Exceptional Children, 69(4), 481–494.

Wiener, D. (2002). Massachusetts: One state's approach to setting performance levels on the alternate assessment. (Synthesis Report 48). Minneapolis, MN: University of Minnesota, National Center on Educational Outcomes. Retrieved Dec. 8, 2005 from http://education.umn.edu/NCEO/OnlinePubs/Synthesis48.html.