Developmental Disabilities Program Independent Evaluation (DDPIE) Project

Lynn Elinson, Ph.D.

Project Director

Evaluation Standards*

UtilityFeasibilityProprietyAccuracy

*Joint Committee on Standards for Educational Evaluation

Purpose of the Independent Evaluation

Demonstrate impact of DD Network programs on:– Individuals – Families– Service providers– State systems

Provide feedback to ADD to help improve the effectiveness of its programs and policies

Promote accountability to the public

Purpose for UCEDDs

Identification of accomplishments from external organization

Identification of areas that need improvement

DDPIE Project

Independent evaluation 2 phases Phase 1 – development and testing of tools Phase 2 – full-scale evaluation Westat – contracted by ADD to implement

Phase 1

Evaluation

Comparison

Standards Indicators

What do we hope to achieve?What do we hope to achieve?What do we observe

(measurement of indicators)? What do we observe

(measurement of indicators)?

Are there differences/discrepancies? Are there differences/discrepancies?

What is the nature and extent of the differences? What is the nature and extent of the differences?

What action needs to be taken?What action needs to be taken?

Open Systems Model

Structure (Input)

Structure (Input)

ProcessProcessOutput

(Product)Output

(Product)OutcomeOutcome

Effectiveness

Efficiency

Basic Evaluation Approach

Performance-based approach Development of standards and indicators Measurement of indicators to determine level at which

standards are being met Development of measurement matrices that contain standards,

indicators, and performance levels (not developed; limited development; adequate development)

Determination of overall performance at the national level

Evaluation Tools

Measurement matrices

- standards

- indicators (structures, processes, outputs,

outcomes)

- performance levels Data collection instruments

DD ActAdministered by ADD

ProgramsDD CouncilsP&AsUCEDDs

Collaboration

INDEPENDENT EVALUATION

KEY FUNCTIONS

FRAMEWORK OF

INDICATORS

STANDARDS PERFOR-MANCE LEVELS

MEASUREMENT MATRICESWill be developed to organize each program’s key functions, framework of indicators, standards and corresponding indicators, and performance levels for each standard.

DD CouncilsOutreachInforming policymakersUCEDDsTrainingComm. serv. ResearchDisseminationP&AsIndividual advocacyOutreach/public educationCollab.Proj. develop.Proj. implement.

• Structure• Process• Output• Outcome

Standards will be used as benchmarks to evaluate the indicators, as measured.

• Level at which a standard is met.

• Three performance levels for each standard.

Project Tasks and Timing

October, 2005 – December, 2006 Collect and review background information. Establish Advisory Panel, Working Groups, and Validation

Panels. Develop draft performance standards, indicators, and data

collection tools. Work with Validation Panels to finalize matrices for pilot study. Train pilot study staff.

January, 2007 – September, 2007 Conduct pilot study in up to 5 states. Write report to ADD with recommendations.

Advisory Panel

Self-advocates Family members Representatives from 3 programs - UCEDD:

Richard Carroll from Arizona Child/disability advocates Evaluation expert Federal representative (for PAIMI evaluation)

Working Groups

4 Working Groups (UCEDD, P&A, DD Council, Collaboration)

Role: To assist Westat in developing draft measurement matrices that will be reviewed and endorsed by Validation Panels

Process: In-person and telephone meetings; work offline

Criteria for Selection of UCEDD Working Group Members

Director/Associate Director At least one state/jurisdiction has more than

one UCEDD program Rural/urban Geographic distribution Reflects different types of UCEDDS (e.g.,

placement in medical school, school of education, standalone)

Working Group Activities

Description of program Discussion and identification of key functions Identification of structures, processes,

outputs, and outcomes for each key function Discussion of standards for structures,

processes, outputs, and outcomes

What can UCEDDs do?

Stay informed Open process - input welcome Participation in pilot study – random selection

UCEDD Working Group Members

Carl Calkins, Ph.D. Kansas City, MO calkinsc@umkc.edu

Tawara Goode, M.A. Washington, DC tdg2@georgetown.edu

Gloria Krahn, Ph.D. Portland, OR krahng@ohsu.edu

David Mank, Ph.D. Bloomington, IN dmank@indiana.edu

Fred Orelove, Ph.D. Richmond, VA forelove@vcu.edu

Fred Palmer, M.D. Memphis, TN fpalmer@utmem.edu

Lucille Zeph, Ed.D. Orono, ME Lu.Zeph@umit.maine.edu

Westat Contact Information

Lynn Elinson- LynnElinson@westat.com- 412 421-8610

Cynthia Thomas- CynthiaThomas@westat.com- 301 251-4364

Priyanthi Silva - PriyanthiSilva@westat.com- 301 610-5162

Bill Frey- FreyW1@westat.com - 301 610-5198