Accountability in

Higher Education Data Driven Schools

A Presentation to the

Educational Approval Board State of Wisconsin

Presentation Outline Institutional Effectiveness

Importance in Higher Education ACCSCT’s Journey Newer thinking – a paradigm shift The ACCSCT outcomes model The Habits of Accountable (data-driven) schools The Pay-off Thoughts about the future

Education In the U.S. U.S. ranks 16th in the World behind most

of Europe in its high school graduation rate at 73%

Of all first-time college students who entered a community college in 1995, only 36% earned a degree within six years.

Completion rates for low-income students and minorities are even lower.

Education in the U.S. Almost 20% of traditional-aged community

college students never complete even 10 credits

Only 6% of the lowest income college entrants obtain a degree at all.

Statistics suggest a need for thinking differently about accountability.

Older view:An effective institution is one that provides resources (faculty, library, services) so learning can occur

Students have responsibility for learning Accreditation/ assessment ensures that appropriate

resources are provided by institutions

No Child Left Behind Shifts the emphasis from teaching to

learning Holds institutions accountable for student

learning Uses standard (although perhaps

imperfect) methodologies to test learning Will influence HEA reauthorization and

Department of Education proposals

Accountability in Post-secondary Education

Discussions regarding institutional effectiveness began before “No Child Left Behind.”

1998 Amendments to the Higher Education Act signaled Congress’ expectation that outcomes measures would be used to determine institutional effectiveness

Early regulation/discussion Focused on “vocational education” Thought of performance outcomes as one

way to measure the effectiveness of institutions in light of mission

Demanded more from accrediting commissions that accredited colleges and schools with clear career objectives (Default rate concerns)

Accrediting Agency Response All institutional accrediting commissions

adopted standards related to measurements of institutional effectiveness

Some institutional accrediting commissions adopted quantitative standards, setting rates of completion, placement, licensure and other outcomes.

Accrediting Commission Response

Accrediting commissions adopted different methodologies for assessing institutional effectiveness Setting specific rates

By program or by institution (aggregate) Changing rates based on averages

By program or by institution (aggregate)

Accrediting Commission Response Some commissions explored alternative

methodologies for assessing quality

HLC of North Central Association: AQUIP SACS adopts similar model and standards for all

institutions

Newer thinking about accountability after 10 years: Workforce Preparation is an important goal of

education Student learning is a shared responsibility Successful student achievement is itself an

important outcome in addition to being a measure of institutional effectiveness (feedback loop)

Higher Education must be accountable to the public for student learning Reflected in H.R. 609 Cost, Accountability and

Transparency Provisions as public policy

Commission on the Future of Higher Education Goals World –class higher-education system that

creates new knowledge, contributes to economic prosperity and global competitiveness and empowers citizens

Accessible to all Americans Affordable high quality instruction Adaptable workplace skills Innovative: adaptable to changing technology and

demographics

ACCSCT’s Journey

ACCSCT Promulgated specific outcomes standards for

completion, placement and licensure in July, 1998 Standards Require “Acceptable” Student Achievement

Completion Placement State licensing examination pass rates

Establish bright line indicators Allow for extenuating circumstances

External factors which have a bearing on student achievement

ACCSCT Completion and Placement Chart

Cohort Reporting Defined Timeframe and Parameters

Individual Tracking Defined Individual Classifications

Starting Point Allowance for Mitigating Factors

Outcomes Data Collection

ACCSCT Outcomes Trends: 1997-2001 Annual Reporting Process Verification Requirements Completion & Placement v. Retention Calculating Student Achievement

Standards

COMPLETION RATES BY REPORTING YEAR

68.9%70%71.6%73%

20.5%19.9%18.8%14%

0

10

20

30

40

50

60

70

80

1998 1999 2000 2001

AVERAGE

STANDARDDEVIATION

85.6%86.0%86.0%85%

15.2%14.0%14.0%15%

0102030405060708090

100

1998 1999 2000 2001

AVERAGE

STANDARDDEVIATION

PLACEMENT RATES BY REPORTING YEAR

Annual Reporting Process Completion and Placement Charts required

annually – for each program Data entry Process & Screening:

Review of each submission for accuracy Additional data requested as required

Supporting Documentation Required Verifiable records of all classifications

Employment, Further Education Schools must keep supporting documentation on file

Verification Required

Completion and placement rates are verified as part of the on-site evaluation process

Random samples of completion and placement data are selected for verification

All submissions of Outcomes reports require supporting documents

Completion and Placement v. Retention Completion and Placement track separate

cohorts of students through graduation Retention measures the extent to which

students remain enrolled in and/or graduate from an institution (snap shot)

Both measures can be used to evaluate successful student achievement (longer programs)

Programs are organized according to a single program commonality - program length

The completion rates for groupings of programs with common lengths are averaged & one standard deviation is calculated.

Raw data & calculations are verified by a third party.

Determining the Student Achievement Standard for Completion

ProgramLength #ofPrograms AvgCompletion<1 4 100%1 83 93%2 63 92%3 95 92%4 69 86%5 94 83%6 134 84%7 257 73%8 272 72%9 245 72%

10 165 70%11 101 66%12 348 66%13 48 66%14 202 61%15 195 61%16 74 65%17 83 58%18 188 58%19 28 58%20 31 61%21 60 65%22 21 56%23 13 41%

Ascending Program Length InMonths

Completion rates

decline as programs lengthen.

Groupings of program length are evident.

Calculating the Student

Achievement Standard

Determining the Student Achievement Standard for Placement

Different program characteristics do not yield any commonality relative to placement.

Data collected over five years did not show that geography, enrollment numbers, program length, or occupational area consistently had any effect on placement.

Some relevancy to economic conditions, extenuating circumstances to be proven by individual institution.

Data and calculations are reviewed by Dr. Morgan Lewis, Statistician, Ohio

State University

STUDENT ACHIEVEMENT STANDARD

Based on 2001 Annual Report Data

Completion Rates Placement Rate

Length in Months

Mean1 Std.

Deviation

Mean Less 1 Standard Deviation

Mean1 Std.

Deviation

Mean Less 1 Standard Deviation

1-688% 13% 75% 85% 15% 70%

7-972% 15% 57%

10-1665% 18% 47%

17 and over 56% 22% 34%

(Based on 3009 programs reported)

School As A Whole

Recruitment

Admissions

Student Progress

Market Needs

Educational Resources

Faculty Qualifications

Program Length

Student Services

Student Outcomes

On-Going Assessment

& Evaluation

Accreditation and Quality Assessment Model

Habits of Accountable Institutions Commitment to strong Leadership

Schools with experienced leaders tend to have better student outcomes (McComis, 2005). But, leadership in our industry requires commitment on the part of owners/investors.

School that establish operations based on best practice, not just because a rule or standard requires it, tend to be more successful. In this way, compliance becomes a natural extension of school operations. For example, the use of PACs because it is a best practice, not because an ACCSCT standard requires it.

Habits of Accountable Institutions: those that care about completion Act at all times in the best interest of students Take attendance React quickly to lack of attendance/progress Take responsibility for student learning and

achievement Listen mightily to student feedback Respond enthusiastically to student needs and

concerns Respond immediately to a change in completion

trends

Habits of Accountable Schools: those that care about placement Involve employers in the development of programs Work feverishly at the school’s reputation in the

community including the workforce community Pursue programmatic accreditations Pursue articulation and consortia agreements with other

institutions Make a strong commitment to continuous improvement Constantly evaluate student achievement data and seek

to improve program content and delivery.

Pay –off: Happier, more successful students

0

2

4

6

8

10

12

14

16

18

20

JAN FEB MAR APR MAY JUN JUL AUG SEP OCT NOV DEC

2004

2005

2006

What have we learned? Verification is critical to any accountability

model Differences in accountability models make

comparable data elusive Diversity in higher education makes one

common model unlikely

What is yet to learn? What is the goal of higher education?

Learning? Workforce preparation? Global conscience? Some combination?

If we know the goal, how will we know when we have attained it? What’s good enough performance? Does striving to be average improve higher education?

How has our thinking changed? Continuous monitoring of program viability a

useful tool for schools and accrediting commissions

Integrating outcomes assessment as part of institutional improvement planning and program advisory committee review is a good practice

Data collection useful for trend analysis and longitudinal studies

The Future . . . Need for further refinement and continuing

study Continued congressional and Department of

Education interest in accountability Increased public disclosure of information on student

achievement Need to ensure accurate disclosure of information by

institutions and accrediting commissions Greater transparency of the accreditation process Greater emphasis on methodology and verification

(perhaps a common approach for similar institutions)

The Future . . . Continuing questions regarding how to judge the relative

quality of education programs (cost/consumer protection issue)

Continuing debate on the limits of federal involvement in determining measures of accountability

Concern over ability of U.S. higher education institutions to compete globally where other countries are investing significantly in research and disclosure of outcomes information

Continuing questions about the cost of higher education and the value added.

Accountability In Higher EducationA Presentation for the Wisconsin Educational Approval

Board

Elise ScanlonAccrediting Commission of Career Schools and Colleges

of Technology2101 Wilson Blvd., Suite 302Arlington, VA 22201703-247-4212escanlon@accsct.org