Mapping the Landscape of Student Learning Outcomes Assessment · Most institutions conduct learning...

S T A N I K E N B E R R Y , G E O R G E K U H , S T A C I P R O V E Z I S , N A T A S H A J A N K O W S K I , G L O R I A J E A , J A S O N

G O L D F A R B , A N D J U L I A M A K E L A D E C E M B E R 4 , 2 0 0 9

Mapping the Landscape of Student Learning Outcomes Assessment

Overview of NILOA

  Why NILOA?

  Who We Are?

  Who is funding the Project?

  “Colleges… do so little to measure what students learn between freshman and senior years. So doubt lurks: how much does a college education – the actual teaching and learning that happens on campus – really matter?” David Leonhardt, NYTimes, Sept 27, 2009

  Far too little is known about assessment practices on campuses around the country

  NILOA’s mission is to document SLO assessment work, identify and disseminate best practices, and support institutions in their assessment efforts

Web Scan Questions   How is learning outcomes assessment portrayed on college

and university Web pages?

  What types of tools, assessments, and resources are displayed, providing evidence of learning outcomes activities? Where is the information located?

  What learning outcomes terms are used? How frequently? By whom?

Transparency “Levels” for Web Sites

1.  Mention a feature that could be related to learning outcomes

2.  Mention an assessment type related to learning outcomes

3.  Show assessment results

4.  Show assessment results with comparison data / interpretation

Example

Accredited by _______

Rubric to evaluate capstone course achievement

Scores on a locally-developed proficiency test

NSSE scores provided with nation-wide comparison data and discussion of what that means for the institution

Web Scan Methods and Sample   Stratified, random sample of survey population

based on institutional type and institutional control

  Trained 6 researchers to systematically scan 725 websites

  Data analysis focused on “describing the current landscape” of institutional transparency and Web sites.

17 Learning Outcomes Assessment Types

Accreditation VSA, U-CAN, AQIP, AAUDE, etc.

Standardized Tests Local Tests

General Education Capstone Experiences Portfolios E-Portfolios

Student Achievement Events Student or Alumni Profiles

National Student Surveys Local Student Surveys Alumni Surveys Senior Surveys Other Surveys

Faculty/Staff Development Awards for Assessment

11 Learning Outcomes Search Terms

Core Competency Core Competencies

Institutional Assessment

Learning Objective Learning Objectives

Learning Outcome Learning Outcomes

Outcome Assessment Outcomes Assessment

Student Achievement

Student Learning Projects

Accreditation Study Questions In what ways are regional accreditation commissions

influencing learning outcomes assessment on American college and university campuses?

  How do regional accreditation standards frame the requirements/expectations for learning outcomes assessment?

  How does learning outcomes assessment figure into site visits, team reports, accreditors’ actions and decisions?

  Are regional accreditors providing resources to institutions on strategies for learning outcomes assessment?

Accreditation Study Methods

1. Interviews with key personnel at regional accrediting agencies

2. Document analysis of accreditation standards, annual meeting programs, workshop manuals, etc.

3. Accreditation Web Scan   Reviewed the Web sites of each regional accrediting agency   Examined 118 institutional sites for student learning

outcomes assessment in relation to accreditation information

Themes

 More than you think

 Less than we need

 Accreditation as a driving force

  Improvement as a desired goal

More than you think   Survey Report

  Most institutions have identified a common set of learning outcomes that apply to all students.

  Most institutions use a combination of institution-level and program-level assessment approaches.

 Web Scan Report   Considerable variety exists in the assessment types

portrayed on institution-wide Web pages.   Most institutions select a small number of assessment types

and use them extensively.

0%

20%

40%

60%

80%

100%

Doctoral Master's Baccalaurate Associate's Other All Schools

Schools with Common Learning Outcomes by Carnegie Type

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Institution-Level Assessments of Learning Outcomes for All Institutions

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Program-Level Assessments of Learning Outcomes for All Institutions

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

Per

cen

t of

Web

Sit

es

Variety of Assessment Types on Institutional Web Pages

Number of Assessment Types

0

20

40

60

80

100

120

140

160

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

Num

ber

of In

stitu

tions

Number of Assessment Types Found

0%

10%

20%

30%

40%

50%

60%

70%

Per

cen

t of

Pag

es w

ith

in a

Web

Sit

e Pervasiveness of Assessment Types

Less than we need

 Accreditation Study   Philosophical shift from creating a learning outcome plan to

implementing outcomes assessment.   More follow up visits and reports for accreditation based on

lack of sufficient student learning outcomes assessment activity.

 Web Scan Report   Much assessment activity is visible to those internal to the

higher education community, but not to those external.

Accreditation Study   Northwest

  Their current standards indicate the need for assessment:   “Educational Program Planning and Assessment 2.B.1 The institution’s processes for assessing

its educational programs are clearly defined, encompass all of its offerings, are conducted on a regular basis, and are integrated into the overall planning and evaluation plan.”

  Their new standards talk specifically about having learning outcomes, collecting data on them, and making judgments on what to do with the findings:   “Standard Four: Effectiveness and Improvement: The institution collects data related to

clearly defined indicators of achievement, analyzes those data, and forms evidence-based judgments of achievements of core objectives.

  Western SR and New England revised requirements to emphasize student learning and data collection

Less than we need

 Accreditation Study   Philosophical shift in creating a learning outcome plan to





  Middle States says that 2/3 of institutions have follow-up and the number one reason being assessment.

  NEASC says 80% of institutions are asked for follow up on student learning outcomes assessment

  HLC says that 7 out of 10 institutions are being monitored and the vast majority of the reasons are student learning outcomes assessment.

  SACS last December had 18 out of 32 or 56% follow ups because of student learning outcomes assessment.

  WASC-ACSCU has reviewed 60 institutions in the last two years and only one received a full 10 years without an interim report, and the sole reason why being the concern for sustainability for assessment.

  WASC-ACCJC will not hold institutions completely accountable for student learning outcomes assessment until 2012

Accreditation Letters

  For all institutions, 90% of regional accrediting agency letters posted asked for additional follow-up information on student learning outcomes activity.

  By regional accrediting agency, 4 out of 5 agencies, of the posted letters, ask for additional student learning outcomes activity 100% of the time.

Less than we need

 Accreditation Study   Philosophical shift in creating a learning outcome plan to





0%

10%

20%

30%

40%

50%

60%

70% P

erce

nt

of S

tart

er P

ages

Frequency of Assessment Types Across Starter Pages

Home Admissions Institutional Research Provost Strategic Plan

Institutions are using the VSA to communicate about learning outcomes assessment

  64% of AASCU and APLU member institutions.   Of the 325 VSA College Portraits reviewed…

  68% had information about college experiences or learning outcomes

  28% had information on student learning outcomes, links to assessment, AND testing information

  18% have all the above stated AND test results

  Survey Report   Most institutions conduct learning outcomes assessment on

a shoestring.   Most institutions plan to continue outcomes assessment

work despite budgetary challenges.   Gaining faculty involvement and support remains a major

challenge.

 Occasional Papers   Wellman, J.V.(2010, January). Connecting the dots between

learning and resources.   Swing, R. & Coogan, C. (2010, March). What does it cost to

do assessment right.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0 0-1 1-2 2-5 5 or more

Assessment FTE by Carnegie Type

Doctoral Master's Baccalaurate Associate's Other




challenge.




0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

No effect May Increase May Decrease May Eliminate Unsure

Resources for Assessment in the Future for All Schools




challenge.




0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

What is Required to Improve Learning Outcomes Assessment for All Schools




challenge.




NILOA Sessions at Institutional Meetings

  “NEASC is pushing for outcomes.” President, Liberal Arts College

  “Accreditation visit coming up. This drives what we need to do for assessment.” President, Urban University

  “WASC has influenced our assessment work.” President, Regional Public University

  “SACs QEP was important to our assessment efforts.” President, State Regional Public University

  “HLC came down on us hard over assessment.” President, Small Liberal Arts

Accreditation as a driving force   Survey Report

  The most common use of assessment data is related to accreditation.

 Web Scan Report   Most institutions post that they are accredited on their Web

sites.

 Accreditation Study   Regional accrediting agencies have expectations for

assessment, provide for capacity building, and require follow-up for lack of sufficient assessment practices.

1.00

1.50

2.00

2.50

3.00

3.50

4.00

Assessment Drivers for All Schools

4.00-‐ High Importance 3.00-‐Moderate Importance 2.00-‐Minor Importance 1.00-‐ No Importance




sites.



0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Per

cen

t of

In

stit

uti

ons

Accreditation Posting by Institutional Type




sites.



Accreditation & Learning Outcomes Assessment:   Middle States, New England, Southern, and Western SR:

  Periodic workshops   Evaluator and chair workshops   Sessions during the annual meeting

  Western ACCJC   rubrics on Student Learning Outcomes Assessment with the aim to provide “examples of

behaviors that, if characteristic of an institution, would indicate its stage of implementation of the standards”

  North Central   an Assessment Academy to create an “institution-wide commitment to assessment

of student learning.”   collections of the papers from the annual meetings bounded by subject area with one on

“Assessing and Improving Student Learning.”

Improvement as a desired goal   Survey Report

  Assessment is driven more by accreditation and a commitment to improve than external pressures from government or employers.

  Accreditation Study   Language used in accreditation letters across all regional accrediting

agencies include “creating a culture of evidence” and “using results for improvement.”

  Occasional Paper   Ewell, P. (2009, November). Assessment, Accountability, and

Improvement: Revisiting the Tension. (NILOA Occasional Paper No.1). Urbana, IL: University of Illinois and Indiana University, National Institute of Learning Outcomes Assessment.

1.00

1.50

2.00

2.50

3.00

3.50

4.00

Assessment Drivers

Accreditation Study examples of programs and language that indicate a goal of institutional improvement.

  Southern  Quality Enhancement Program

  North Central  AQIP Program

  Middle States  1- Keep it useful;  2-Tie assessment to goals;  3- Provide direct evidence for student learning;  4- Use multiple measures;  5- Something should be happening everywhere, every year.

Summary ●  More than you think ●  Perhaps more assessment underway than some acknowledge or wish to

believe

●  Less than we need ●  More attention needed to using and reporting assessment results and greater

involvement of faculty ●  More investment likely needed to move from data to improvement

●  Accreditation as a driving force ●  Increased attention to ASSESSING student learning

●  Improvement as a desired goal ●  Increasing institutional commitment to conducting and using assessment

Limitations

  Mapping what is out there, and need to know more about program/department level activity.

  Student learning outcomes assessment is in constant flux and is continuing to be developed. Process unfold while we study.

  We only know what is portrayed, and we do not know use or intent–specifically the Web scan   one time data capture of a fluid and evolving source   constrained by each institutions resources (eg. Technology, staff)

Next Steps

 Greater emphasis on program/departmental

activities.

 Focus on institutional assessment culture.

 Examination of posted results.

D E C E M B E R 4 , 2 0 0 9

Questions or Comments?��

www.learningoutcomesassessment.org

Mapping the Landscape of Student Learning Outcomes Assessment · Most institutions conduct learning...

Documents

Transcript of Mapping the Landscape of Student Learning Outcomes Assessment · Most institutions conduct learning...