Susan Agre-Kippenhan,

Portland State University

Professor, Art Department

Evaluating the Effectiveness of

Service Learning

Why evaluate?Assessment as a continual Service

Learning improvement toolClarification of critical learning

objectives, ability to focus on Service Learning specifics

Yields useful information for programs, faculty, students

Evidence and support claims

SL Outcomes:

Service Learning Claims: Understanding of course contentTheory to practice Understanding of Community Community related skills Civic Engagement Diversity MORE?

Student Learning Focus

Value of authentic work samples

Moving from self-report Moving from less integrated forms of assessment (i.e. observation, focus group)

Continual Improvement

Feedback to actual courses, programs Applicable/actionable assessment Can be course/assignment based Can serve other programmatic goals

How do we know what works? A working model for assessment based on

the premise that there are:

Key questions to ask Logical steps to follow Usable data Transforms student learning and community experiences

1Assessment Cycle Model

4Develop Learning Objectives

Identify Work Samples

Locate Critical Junctures

DetermineAssessment Criteria/Quality

TestValidity /Reliability

DisseminateRevise/Revisit

5

6

2

3

Assessment Cycle Model

4

Develop Learning Objectives

Identify Work Samples

Locate Critical Junctures Determine

Assessment Criteria/Quality

TestValidity /Reliability

DisseminateRevise/Revisit 56

1

23

Assessment Cycle Model

What am I trying to assess?

Action: Define Student Learning Outcome

Student Learning Outcomes in the context of:

Mission Priorities Program Goals

1

What are appropriate work samples?

Action: Identify Student Work Samples

Qualitative vs. Quantitative Examples: Portfolios, Papers (writing samples, essays, reflective papers, editorials, journals…) Performances, Artwork, Lab reports

Assessment Cycle Model

2

When should I assess?

Action: Identify critical junctures

Critical Junctures Entering students Passages Culminating

Assessment Cycle Model

3

What am I looking for in student work?

Action: Define scoring criteria

Rubric Development: Scoring Criteria What indicators will I use to demonstrate desired student learning?

Examples: Bloom's taxonomy

4Assessment Cycle Model

How do I evaluate the range of student work?

Action: Define performance levels

Rubric Development: Performance standards How many levels will my rubric have and how will I define performance at each level?

Specific clear descriptors at each levelQualitative differencesCommon errors

4Assessment Cycle Model

Sample RubricWork Sample

Scoring

Criteria 1

Scoring

Criteria 2

Scoring

Criteria 3

Performance

Level 1-

Low

Performance

Level 2-

Medium

Performance

Level 3-

High

Once I have a rubric how do I know it's valid?

How do I know if the rubric is reliable?

Evaluate the rubric against the learning objectives, learning opportunities etc… Is there clear alignment?

Pilot test with multiple evaluatorsDiscuss how well the rubric worksCompare scores for consistencyDetermine procedures for inconsistent scores

5Assessment Cycle Model

What's next?

Action: Back to the cycle

Revise and revisit - learning outcomes, learning opportunities, assignments, critical junctures, rubrics, program, course offerings/order, curriculum, pedagogy…

6Assessment Cycle Model

SL Case Study University Studies - General Education

4 Year integrated program Interdisciplinary program

Critical Features Clearly defined goals

Communication, Critical thinking, Diversity, Civic Responsibility

Common Pedagogy Required Courses

SL Case Study

Student Learning Outcomes

Action: Define Student Learning OutcomeAppreciation and Understanding of Civic Responsibility

PSU Mission - Let Knowledge Serve the City University Studies Goal - Civic Responsibility

1

SL Case Study

Work samples

Action: Identify Student Work SamplesCommon assignment given in each class

Paper2

Critical Junctures

Action: Identify critical junctures Pilot in Senior Capstone Classes

Senior Capstone required course Culminating group experience for major and general education Application of expertise to a real project in the community

SL Case Study

3

Common Assignment “Please write a 3-5-page reflective analysis

summarizing the ways you have gained experience with the University Studies goal of Civic Responsibility. Use specific examples from your discussions and course readings, or in completing your project.”

SL Case Study

Scoring Criteria and Performance Levels

Action: Define scoring criteria/levels

Civic Responsibility Stages

Based on Bloom's taxonomy

SL Case Study

4

Valid Rubric

How do I know if the rubric is reliable?

PSU Mission - Let Knowledge Serve the City University Studies Goal - Civic Responsibility

Pilot test with multiple evaluators Compare scores for consistency. Discuss how well the rubric works.

SL Case Study

5

Revisiting Process

Suggestions for improvements Revise assignment for clarity Revise rubrics for specificity Clarify pedagogy

Confirmed Program goals addressed in class Basis of assignment is valid Keep assessment in Capstone

SL Case Study

6

Focus on student learning objectives What are appropriate work samples? Are there existing critical junctures? How do we evaluate quality? How can this improve our practice?

SL Assessment in practice

Connect to goals and mission Knowledge of the process that is

transferable Assessment as a tool for partnerships Applicable/actionable assessment

Lessons Learned

For More InformationSusan Agre-KippenhanAgrekis@pdx.edu503-725-8506