State Higher Education State Higher Education

Assessment Policies:Assessment Policies: Findings from Case Studies

Thomas E. PerorazioJohn J.K. Cole

The National Center for Postsecondary Improvement Project 5.1

The University of Michigan

Association for Institutional Research42nd Annual Forum

Toronto, Ontario, Canada

Session Goals

• Discuss Policy Process for

Assessment

• Relate Important State

Experiences

• Share Lessons Learned from Cases Relevant to Institutions

The National Center for Postsecondary Improvement

Main Project Web Site

http://www.stanford.edu/group/ncpi/

University of Michigan - Project Five

http://www.umich.edu/~ncpi/

NCPI Project Area 5.1

• Examine State & Regional Policies What Policies Are in Existence? Impact on Institutional Assessment

Practices Impact on Teaching & Learning

• Utilize Policy Process Framework• Analyze Relationships Among Levels

State Government and Regional Accreditation Association Policies of

Assessment for Student Learning

Overview of Project 5.1 Research

• Benchmarking AssessmentDocumented Existing Policies

• SHEAQ SurveyOf SHEEO Administrators

• Literature Review

• Case Studies

Case Study Research

• Examine State Policies in Detail

• Explore Critical Issues with State Officials

• Trace Evolution and Development

• Learn about Policy’s Impact on Institutions

• Infer Lessons about the Policy Process

Purposes

States Selected for Study

• New York (MSACS)

• South Carolina (SACS)

• Washington(NWASC)

• Missouri (NCA/HLC)

• Florida (SACS)

(with Accreditation Region)

Policy Context

• Historical, Political, Social, & Economic Factors

Existing Policies for Accountability, Efficiency, etc.

Political Climate in the State for Higher Education

Previous Political Action RE: Assessment

Governance Structure for Higher Education Relations & Communications between Government,

SHEEO, & Institutions

Budgetary & Financial Issues for Higher Education

Policy Process FrameworkFive Stages

1.      Problem Formation Recognizing the Need for a State-level Assessment Policy

2.      Policy Formulation Development of Proposed Courses of Action

3.      Policy Adoption Development of Support for a Specific Proposal

4.      Policy Implementation Application of the Policy to the Problem

5.      Policy Evaluation Attempt to Determine the Policy’s Effectiveness

Policy ContextHistorical, Political, Social, & Economic Factors

Formation

Formulation

Adoption

Implementation

Evaluation

History

Purpose

Design

Leadership

Links

Outcomes

Inputs

Objectives

Processes

Outcomes

Evaluation

Conclusions RE: Policy Process, Lessons for Policy ActorsUnderstanding Relationships Among Policy Levels

Policy Process Synthesis Case Analysis

Problem Formation & Policy Origination

• SHEEO Seeks to Focus Goals of Institutions on State Priorities Statewide Planning Task Forces Priorities of Quality, Effectiveness, Prestige, &

Efficiency

• Gubernatorial/Legislative Interest in Performance Desires Information on Results/Success Task Force Studies Data Generation & Collection Measures Concern about Public Perceptions of

Prestige/Quality

Problem Formation (2)

• Institutional Actions Initiatives to Engage in Assessment Procedures to Improve Program Quality Programs to Enhance Learning

• Public Opinion Critical for Political Will

• Change in Political Power

Policy Formulation

• Quality Assurance Institutional Data Generation/Collection

Performance Indicators/Assessment Reports

Make Information Publicly Available Increase Information to Policymakers

• Accountability State Planning & Coordination

Institutions Meet State Goals/Targets Centralized Approach to Data Analysis Results Tied to Budgetary Decisions

Options for Consideration

Policy Formulation (2)

• Institutional Improvement Management & Effectiveness

Link Measures to Accreditation Standards

Institutional Self-Evaluation Differentiated by Sector Bring Improvement Through Quality Meet Both State and Institutional

Goals

Policy Adoption

• Legislative Action Mandate Authorize SHEEO & Monitor Work with Institutions for Revision

• SHEEO Authority Originator of Policy Monitor, Collector, & Distributor of Info Mediator between State, Institutions, &

Public

Four General Methods

Policy Adoption (2)

• Task Forces/Blue Ribbon Committees Authority: SHEEO or Legislature Business Leaders, Institutional Presidents Conduct Study/ Make Recommendations

• State/System Planning Process Produces Actionable Objectives Assessment In Service of Plan Goals

Policy Implementation--Mechanisms

• Reporting Institutional Statistics New York, Washington, & Florida

• Performance Funding/Reporting South Carolina Missouri -- FFR Florida

• System Goals & Institutional Improvement Missouri, Washington

• Accountability Florida, South Carolina, Missouri

Policy Implementation (2)

• Decentralized State Sets Broad Parameters for Performance Institutions Develop Effectiveness Plans

Measurement Defined by Institutions Institutions Report Results up to State State Makes Decisions on Aggregated Data

• New York & Washington

Policy Implementation (3)

• Centrally-Guided Prescribed State & Institutional Goals Performance Standards Less Variable Central Data Collection & Analysis Findings Utilized in Budget Decisions

• South Carolina & Missouri

Policy Implementation (4)

• Combination Approach State Expectations & Performance

Guidelines Institutional Variability for Compliance Institutional Activity for Internal

Improvement

• Florida

Data Collection

• Centralized Databases v. Institutional Data Levels of Aggregation

• Instruments/Testing Commonality Associated with Centralization

• Institutional Reporting To SHEEO, Legislature, Public

Data is Reported Up the System

Emphasis on Data & Data Systems

Data Usage for Decision Making

• Rewards-- For Meeting Targets MO & SC

• Incentives-- To Achieve State Goals FL

• Public Knowledge--Consumer

Information WA & NY

Links to Teaching/Learning Improvements

• Making Assessment Institution-centered

• Public Accountability

• Institutions Share Data on Learning

• Revisiting Indicators Regularly

• Close Information Loop

• Focused Goals at Different Inst. Levels

Outcomes of Policy

• Institutional Resistance Disparate Effects “Negative Improvement” Excessive/Burdensome Requirements Indicators Not Useful for Management

• Institutional Cooperation Focus on Improvement Trust between SHEEO & State Partnership to Develop System

Policy Evaluation

• Revisiting &Adjustment MO & SC

• Implementation NY & WA

• Implementation & Evaluation FL

Lessons Learned

• Assessment Must Be Incorporated Into Institutional Management

• Successful Policies Developed in Consultation with Institutions

• Culture of Institutions Can be Changed If The Process Contributes to Mgt

Lessons Learned (2)

• Institutions Must Also Be Willing to Form Working Relationships with State Officials

• Stakeholders at All Levels Must Be Engaged with Assessment

• Involving Too Many Stakeholders in Development Bogs Process Down

Lessons Learned (3)

• Sustained Commitment of Leadership Required

• Political Will for Success Required

• Policy Process Can Be As, If Not More Important Than Its Results

Successful Policies

• Have a Clear & Focused Purpose• Differentiate by Sector/Mission• Emphasize Institutional Improvement• Embrace Simpler, Rather than Complex,

Indicator/Reporting Mechanisms• Incorporate Priorities of Multiple

Stakeholders• Provide Useful Data for Decision Makers

Important Considerations

• Context for Assessment Shapes Process• Policies Result In Improved Data

Systems• Process Forces Articulation of Principles• New Policies May Not Replace Old Ones• Policy Must Be Useful to Instituions