It Can’t Be Done: Assessing Educational Quality Across

Institutions

Corbin M. Campbell, Assistant Professor

Theresa Cruz Paul, Research Team Member

Higher and Postsecondary Education Program

Teachers College, Columbia University

Advisory Board Members

• Jennifer Glaser• Director of Student Services, Fairfax High School

• Dr. Wendell Hall• Deputy Director• Institute for Higher Education Policy

• Dr. Karen Inkelas • Director, Center for the Advanced Study of Teaching and Learning• University of Virginia

• Dr. Christine Keller• Executive Director, Voluntary System of Accountability; Associate

Vice President for Academic Affairs• Association of Public and Land-grant Universities

• Sharon La Voy• Director of Assessment• University of Maryland

• Dr. Jennifer Lindholm• Special Asst. to the Dean & Accreditation Coordinator• University of California Los Angeles

• Dr. George Mehaffy• Vice President for Academic Leadership and Change• American Assoc. of State Colleges and Universities

• Dr. Jessica Mislevy• Research Social Scientist

• SRI International

• Dr. Daniel Newhart• Senior Researcher & Associate Director• Center for the Study of Student Life• The Ohio State University

• Dr. Anna Neumann• Professor of Higher Education• Teachers College, Columbia University

• Dr. KerryAnn O'Meara• Associate Professor of Higher Education• University of Maryland, College Park

• Dr. Aaron Pallas• Professor of Sociology and Education• Teachers College, Columbia University

• Dr. Stephen Porter• Professor of Higher Education• North Carolina State University

• Dr. Priscilla Wohlstetter• Distinguished Research Professor• Teachers College, Columbia University

Claim

There is currently no comprehensive way for the public and prospective students and families to know about the quality of the education that is happening inside the walls of a college or university—and how that quality compares to the quality at other colleges and universities.

The Black Box of Higher Education

Extensive accountability data is largely unseen by the public:

• Accreditation

• Collegiate Learning Assessment (CLA)

• Course-based learning outcomes

• Course evaluations

Impacts of the Black Box [& Rising Costs]

• Spellings Commission

• Academically Adrift (Arum & Roksa, 2010)

• Numerous articles in top newspapers regarding questions to academic rigor in higher education.

• 12/10/12: “Who will hold colleges accountable?” (Carey, NYT, p. A27)

• “Affront on Higher Education”

How do prospective students, parents, and the public decide which institution has the highest quality education?• US News

• World Rankings

• Other ranking venues: Princeton Review

• Word of Mouth—Family, Friends, High School Guidance Counselors

• Reputation

[Newer, but under used forms: NSSE?, VSA/College Portrait?]

US News & World Report’s Formula

1. Graduation & retention rates-20%

1. Average graduation rates-80%2. Average freshman retention rate-

20%

2. Financial resources-10%1. Average educational expenditure

per student-100%

3. Alumni giving-5%4. Graduation rate

performance–5%5. Peer assessment-25%

6. Student selectivity-15%1. Acceptance rate-10%2. High school ranking-40%3. SAT/ACT scores-50%

7. Faculty resources-20%1. Faculty compensation-35%2. % faculty with top terminal

degrees-15%3. Percent full time faculty-5%4. Student/Faculty ratio-5%5. Class size 1-19 students-30%6. Class size 50 or more-10%

Possible unintended consequences of past and current measures….

Policy: No Child Left Behind (and Collegiate Learning Assessment—CLA)• Teaching to the test• Altering curriculum

Public: US News & World Report• Students and parents use rank instead of fit to select which

college to attend.

Institutions: US News & World Report• Mission creep/Striving• Over-reliance on SAT (GWU, for example)• Numbers Manipulation: Cornell removed non-graduates

from the alumni list

Questions Absent in These Measures…

• What is the level of academic rigor?

• What is the quality of teaching?

• What are the educational practices that an institution employs that affect student learning?

[Maybe a few examples of rankings that use surveys to measure these items, for example, Princeton Review]

Why are these questions absent?

• These data are difficult to obtain!!

• These data are expensive to obtain!!

• Colleges and Universities are protective of academic freedom and are insular with data about the educational core.

Enter NSSE, CLA, VSA/College PortraitThe National Survey of Student Engagement (NSSE): Surveys institutions about their effective educational practices

CLA: Measures student’s critical thinking skills pre and post college via a standardized test.

Voluntary System of Accountability (VSA)/College Portrait: Created to ward off imposed and mandating higher education testing. Aimed at the public transparency of higher education—compiles several data sources: NSSE, CLA, Grad/retention

PROBLEMS:

1) Concerns with validity

2) Relies on a single data collection method

3) Not primarily intended for the public

4) Assumes learning is due to college environment

5) Missing data / limited data

Purpose

This research agenda aims to create alternative, innovative, and comprehensive measures of educational quality across institutions that could contribute to public understanding of college and university quality.

What are the intended consequences of this new educational quality measure?

1) A stronger focus on the educational core of institutions: teaching and academic rigor, and educational experiences

2) Public access to comprehensive data about teaching, academic rigor, and educational experiences in higher education at the institutional level

3) Administrators having an in-depth understanding of how their institution compares to others in terms of teaching quality, academic rigor, and educational experiences

Three Phases:

• Dual-Institution Pilot (Spring 2013)

• One large, public, research extensive institution

• One medium private research extensive institution

• Multi-Institution Peer Benchmarking Pilot (Spring/Fall 2014)

• National Study with Publicly posted Data

(Spring 2016)

Dual-Institution Pilot

1) Student and faculty survey: NSSE FSSE

2) Syllabus analysis

3) Experience sampling method

4) Class observations

5) Analyzing Student Work

6) Course Evaluations*

Academic Rigor, Teaching Quality, Learning Outcomes

Academic Rigor• Based on the cognitive complexity required by students in

the coursework as defined by the revised Bloom’s Taxonomy (Anderson & Krathwohl, 2001)

Teaching Quality

• Based on Anna Neumann’s claims on teaching and learning (2012).

• According to this framework, quality teaching entails: (Part I) Orchestrating an encounter of subject matter ideas (Part II) Connecting student’s learning to prior knowledge (Part III) Supporting students in working through the cognitive and emotional features of encounters between their own long-held understandings and new ones gained during the course.

Essential Learning Outcomes• Based on the American Association of Colleges and

Universities’ (AAC&U) Essential Learning Outcomes (ELO).

• This framework was developed by AAC&U through

engagement with hundreds of institutions, accreditors, and higher education stakeholders (AAC&U, 2004).

• Four Parts:• ELO Part I: Knowledge of Human Cultures and the Physical

and Natural World• ELO Part II: Intellectual and Practical Skills• ELO Part III: Personal and Social Responsibility• ELO Part IV: Integrative and Applied Learning

Where we are now• At this point we have observed 100 classrooms• In the process of collecting 300+ syllabi• Collecting NSSE FSSE data• Discovering that Experience Sampling may not be a

viable option• Collecting student work

Next steps• Preparing to collect data from a second pilot institution• Searching for a multi-site pilot institution that would like

to compare data to defined peer institutions

Questions? Comments?

Corbin M. Campbell, Assistant Professorcampbell2@tc.columbia.edu

Theresa Cruz Paul, Research Team MemberHigher and Postsecondary Education Program

Teachers College, Columbia University