John HayekJillian KinzieGeorge Kuh

James Moran

SHEEO Higher Education Policy Conference

August 13, 2009Denver Colorado

Assessment for Improvement and Accountability:

Examples from the States

Context Global Competitiveness in Degree

Attainment The New Majority and Demographic

Gaps Questionable Levels of Student

Performance In an Environment of Increasing Fiscal

Strain… We Need Higher Levels of Student

Achievement at an Affordable Price

Overview

Purposes of assessment First lessons from NILOA Examples of NSSE use

Advance OrganizersWhat are the most important

SLOs to measure in your state? What is being done to assess

these outcomes?What are the major obstacles or

challenges to SLO assessment?What do you need to further SLO

assessment in your state?

Continuous Improvement

Accountability

Strategic dimensions Purpose Formative (improvement) Summative (judgment)

Orientation Internal External

Motivation Engagement Compliance

Implementation Instrumentation Multiple/triangulation Standardized

Nature of evidence Quantitative and qualitative Quantitative

Reference points Over time, comparative, established goal

Comparative or fixed standard

Communication of results

Multiple internal channels Public communication, media

Use of results Multiple feedback loops Reporting

Two Paradigms of Assessment

Ewell, Peter T. (2007). Assessment and Accountability in America Today: Background and Context. In Assessing and Accounting for Student Learning: Beyond the Spellings Commission. Victor M. H. Borden and Gary R. Pike, Eds. Jossey-Bass: San Francisco.

NOLOA

Far too little is known about assessment practices on campuses around the country

NILOA’s mission is to document SLO assessment work, identify and disseminate best practices, and support institutions in their assessment efforts

www.learningoutcomesassessment.org

NOLOAFUNDERS

Lumina Foundation for EducationCarnegie Corporation of New YorkThe Teagle Foundation

NATIONAL ADVISORY PANEL

NILOA ActivitiesWeb scans: 700 institutionsInterviews & focus groups with key

actors: -- AAC&U -- ACE -- AIR

“Go to” Web site being developed (www.learningoutcomesassessment.org)

Case studiesWhite papers

NILOA White PapersBanta et al.: Authentic assessment

approachesBorden: “Measuring Quality II”Ewell: The improvement and

accountability purposes of assessment

Lenth & Hill: Assessment and state policy

Wellman: Assessment results and resource allocation

NILOA ActivitiesWeb scans: 700 institutionsInterviews & focus groups with key

actors“Go to” Web site being developedCase studiesWhite papersSurvey of institutional assessment

practices

NILOA 2009 Provost SurveyAll accredited, undergraduate

degree-granting 2- and 4-year public, private, and for-profit institutions in the US (n=2809)

53% response rate (n=1518)

Tentative ConclusionsPerhaps more assessment underway than

some acknowledge or wish to believeMore activity at the department/unit level

than institution levelAccreditation is a major force shaping

assessmentMore attention needed to using and

reporting assessment resultsInvolving faculty is a major challengeMore investment likely needed to move

from data to improvement

Genera

l Kno

wledge

Specia

lized

Kno

wledge

Additio

nal A

sses

smen

ts

Externa

l Jud

ges

Nation

al Surv

eys

Loca

l Surv

eys

Rubric

s

Portfol

ios

Studen

t Inter

views

Alumni

Survey

s

Alumni

Interv

iews

Employe

r Surv

eys

Employe

r Inte

rview

s0

20

40

60

80

100

Valid Institutional Samples

Examples from the States

South DakotaTexasTennesseeKentucky Pennsylvania

NSSE State and University System Participation 2000-

2009• California State

Universities• City University of

New York• Concordia

Universities• Connecticut State

Universities• Kentucky Council

on Postsecondary Education

• New Jersey Public Univ.

• North Dakota University System

• Ontario Universities• Penn State System• Pennsylvania

System of Higher Education

• South Dakota Public Universities

• State University of New York

• Tennessee Publics• Texas A&M System

• Texas Six• University System

of Georgia• West Virginia State

System• University Systems

of Hawaii, MaineMaryland, Massachusetts, Missouri, New Hampshire, North Carolina, Texas, Wisconsin

State System Use of NSSE Institutional improvement Research Accountability & public reporting Performance funding Decision-making models   More specifically….

track student engagement overtime, set engagement performance goals, explore engagement patterns by student

characteristics merge NSSE data with other system data to

examine factors associated with student success

Examples: Student Engagement Data in State SystemsAssessment and Research Use data to examine student experience and in predictive models for retention, degree attainment

Institutional Improvement Data for institutional improvement initiatives, and to encourage collaboration among system & campuses to address common challenges

Accountability Monitor and demonstrate quality in undergraduate education

South Dakota Board of Regents Data from 5 NSSE administrations

since 2002 for all public institutions Longitudinal analysis shows upward

trend in student engagement

South Dakota Board of RegentsResearch project findings: Strong links between student

performance and student-faculty interaction

Student effort in-and-outside of class correlated with other Regents Assessments of general education goals, Collegiate Assessment of Academic Proficiency (CAAP)

Academic performance Critical thinking PersistenceSo. Dakota School of Mines & Technology

South Dakota Board of RegentsFindings Influenced Improvements and Policies:Expansion of universities’ research

capacity to foster collaborative projects between faculty and students

Salary competitiveness to retain high quality faculty

University of Texas System

NSSE Use for Accountability detailed in U.T. 2008 System Report: Inform Texas legislature, Board of

Regents, U.T. System Faculty Advisory Council and Student Advisory Council, and general public VSA U.T. System Accountability Report Legislatively mandated Consumer

Satisfaction requirements

University of Texas System

NSSE used for benchmarking:NSSE results considered useful for

benchmarking because items are strongly associated with student success

NSSE Benchmarks provides continuing source of comparisons to gauge U.T. System progress

Where significant differences occur, institutions are encouraged to identify areas for improvement

Student engagement assessment & research

Feedback for internal use Appropriate peer comparisons Focus for improvement Performance funding

Tennessee Higher Education SystemAll TN Public 4yr institutions participated in NSSE as required by TN performance funding program.

Research about the relationship between engagement and satisfaction at TN public institutions showed the importance of: Quality of academic advising Relationship with peers, faculty &

staff Institutional support for success

Tennessee Higher Education System

Interest in NSSE as a process measure… What happens to students

while attending higher education?

Identified State & Institutional Policy Implications:

More research to examine relationship between engagement, satisfaction and degree production (develop longitudinal database, track individual students surveyed)

Institutions encouraged to evaluate conditions of student services and improve advising; etc.

Tennessee Higher Education System

Using student engagement data in

Kentucky to promote student success

John HayekVice President

August 13, 2009

SHEEO conference

8 public universities, 16 community colleges and technical, 20 regionally accredited, nonprofit, independent colleges and universities, and numerous other postsecondary options.

212,000 undergraduate students and 27,000 graduate and professional students in 2007.

49,700 total degrees and certificates awarded; 18,600 bachelor’s degrees in 2007.

600,000 college and university alumni.

$183 million in federal Pell grants, $486 million in federal loans, and over $180 million in state financial aid in 2006.

$4.2 billion in total revenue generated by Kentucky’s public universities and colleges in 2007.

1,869 public postsecondary education buildings with more than 47 million gross square feet.

30,000 full-time faculty and staff employed at Kentucky’s public universities and colleges in 2008.

$327 million in extramural R&D expenditures combined at the University of Kentucky and University of Louisville in 2006.

Kentucky Postsecondary Education Enterprise

28

Net General Funds have been cut $78 million or 7.2% over the past 18 months, while K-12, Medicaid, and Corrections have been exempted.

Budget Update

Source: CPE Comprehensive Database

$1,084

$1,052$1,026

$1,006

$936

Enacted FY08 Revised FY08 After 3% Cut

Enacted FY09 After 2.4% Cut

Revised FY09 After 2% Cut

Revised FY10

Kentucky Public Postsecondary Education Net General Funds ($millions)

FY08 to FY10

$78 M Cut in Net General Funds

$70 MSFSF

Tuition increases (3% for community college, 4% for comprehensive universities, and 5% for research universities), lowest in a decade

Why it Matters

The 50 StatesRelationship between per capita income and college degrees

The single factorwith the greatest powerto explain differences in per capita income between statesis the percentage of college graduates.

Milken Institute, 2002

30

Context Postsecondary Reform Legislation in 1997 - Long-term goals

Public Agenda – Short-term strategic plan updated every four years

Accountability System – Key performance indicators and goal setting process

KY Action NSSE (2001, 2003, 2005, 2007, & 2009)

College-level learning pilot (2002)

CCSSE (2006, 2007, and 2008)

Regular reports to Council on progress

Faculty development meetings

Accreditation and quality enhancement plans

NSSE WorkshopKentucky was one of the first states to participate as a system in NSSE.

Outcomes Increased attention to quantity and quality

Improvement on 25% of benchmark scores between 2001 and 2007

FTFT retention rates improved from 71% (1998) to 74% (2007)

Graduation rates improved from 37% (1998) to 47% (2006)

Bachelor’s degrees increased from 14,600 (1998) to 18,600 (2007)

Key is to keep student engagement and student outcomes data on the radar screen.

Jim MoranVice Chancellor for Academic and

Student Affairs

August 2009

Performance Matters Pennsylvania State System of Higher Education

35

Overview

What is the Pennsylvania State System of Higher Education (PASSHE)?

How did we develop a culture of accountability and performance?

What results have we achieved?

36

PASSHE In BriefFourteen (14) regional public

universities Focus on baccalaureate and masters

programs (doctoral mission at one university)

Rural settingsOver 112,000 students90% undergraduate / 90% Pennsylvania

residentsSystem created in 1983

38

Measuring What Matters

Defining values and goals

Developing meaningful, reliable measures

Data driven decision-making

Discussion

Quantitative MeasuresQualitative MeasuresWhat have we learned?Where are we going?Evolving Use of the NSSE

40

Conclusions

Performance driven by values and goals

Solid data upon which to make decisions is everyone’s business

Accountability for results is shared

What other ways can NSSE and other assessment tools be used to stimulate improvement in state system institutions?

How can NSSE be used to advance state system concerns, or inform policy?

NSSE advises against using results in performance funding. What is an appropriate use?

What other uses and tools are you considering?

For Discussion…