Program Assessment Workshop

Mary AllenQatar UniversitySeptember 2011

Workshop participants will be able to: draft/revise learning outcomes develop/analyze curriculum maps develop/refine sustainable, multi-year

assessment plans develop/refine rubrics and calibrate

reviewers analyze assessment results use a variety of strategies to close the loop evaluate the impact of improvement actions

Learning Outcomes for this Workshop

an on-going process designed to monitor and improve student learning

Assessment

Faculty: Develop SLOs Verify curriculum alignment Develop an assessment plan Collect evidence Assess evidence and reach a conclusion Close the loop

Assessment

Standard 3.3.1

SACS Expectation for Assessment

Program goals Cohesive curriculum How students learn Course structure and pedagogy Faculty instructional role Assessment Campus support for learning

Learning-Centered Institutions

Quotations from the Wise and Experienced

Direct vs. indirect assessment Value-added vs. absolute learning outcomes Authentic assessment Formative vs. summative assessment Triangulation

Assessment Vocabulary

If you have absolute outcomes, your assessment plan should emphasize direct, authentic, summative assessment, with triangulation.

Clarify what faculty want students to learn Clarify how each outcome can be assessed

Learning Outcomes

Knowledge Skills Attitudes/Values/Predispositions

Learning Outcomes

CLO PLO ILO

Outcome Levels

Possible Learning Goals

Examples of Outcomes

List of goals and outcomes List of outcomes Typically 6-8 outcomes in all

Goals and Outcomes:Outcomes and Performance Indicators

Bloom’s Taxonomy

1. Active verbs2. Simple language3. Real vs. aspirational4. Aligned with mission5. Avoid compound outcomes6. Outcomes vs. learning processes7. Focus on high-priority learning

Creating Quality Outcomes

Coherence Synthesizing experiences On-going practice of learned skills Opportunities to develop increasing

sophistication and to apply what is learned

The Cohesive Curriculum

I = Introduced D = Developed & Practiced with Feedback M = Demonstrated at the Mastery Level

Appropriate for Graduation

Curriculum Map

Curriculum Map 2 Curriculum Map 3

Curriculum Map Patterns

CLOs that align with relevant PLOs Faculty can provide artifacts for assessment Faculty teach courses consistent with the

map

Entries on the Map Indicate:

Focuses faculty on curriculum cohesion Guides course planning Allows faculty to identify potential sources

of assessment evidence Allows faculty to identify where they might

close the loop

The Curriculum Map:

Except for NCATE-accredited programs

We don’t have to assess every outcome in every student every year!

Who? What? Where? When? Why?

Assessment Plan

Relevant samples Representative samples Reasonably-sized samples

Sampling

Anonymity Confidentiality Privacy Informed consent

Ethical Issues

Find examples of: Direct assessment Indirect assessment Formative assessment Summative assessment Authentic assessment Triangulation

Sample Assessment Plan

PLO When to assess What direct and indirect evidence to collect Who will collect the evidence How evidence will be assessed How decisions will be made

Assessment Plan Template

Valid Reliable Actionable Efficient and cost-effective Engages students Interesting to faculty Triangulation

Properties of Good Assessment

Published tests Locally-developed tests Embedded assessment Portfolios

Direct Assessment Strategies

Surveys Interviews Focus groups

Indirect Assessment Strategies

Holistic Analytic

Rubrics

Rubric Packet AAC&U VALUE Rubrics Specialized Packets

Rubric Examples

Efficiency Defines faculty expectations Well-trained reviewers use the same criteria Criterion-referenced judgments Ratings can be done by multiple people

Rubric Strengths

Assess while grading Assess in a group

Two Common Ways to Apply Rubrics

Columns are used for assessment Faculty can adapt an assessment rubric in

different ways Faculty maintain control over their own

grading

Assessing and Grading Simultaneously

Turn to someone near you and explain how you can grade and assess simultaneously.

Grading may require extra criteria Grading requires more precision Calibrate when doing assessment

Assessment vs. Grading

Speed up grading Clarify expectations to students Reduce student grade complaints Improve the reliability and validity of

assessments and grades Make grading and assessment more

efficient and effective Help faculty create better assignments

Rubrics Can:

Suggestions for Using Rubrics in Courses

Below Expectations Needs Improvement Meets Expectations Exceeds Expectations

Typical Four-Point Rubric Levels

Rubric Category Labels

Adapt an already-existing rubric Analytic method

Creating a Rubric

Consider starting at the extremes Some words I find useful

Drafting the Rubric

One reader/document. Two independent readers/document. Paired readers.

Managing Group Readings

Collect the assessment evidence and remove identifying information.

Develop and pilot test the rubric. Select exemplars of weak, medium, and

strong student work. Consider pre-programming a spreadsheet

so data can be entered and analyzed during the reading and participants can discuss results immediately.

Before Inviting Colleagues:

Correlation Discrepancy Index

Inter-Rater Reliability

Rubric Orientation and Calibration

How good is good enough?

Assessment Standards

Celebrate! Change pedagogy Change curriculum Change student support Change faculty support Change equipment/supplies/space

Closing the Loop

Closing the Loop Example

Annual Assessment Reports

1. Focus on what is important. 2. Don’t try to do too much at once. 3. Take samples.4. Pilot test procedures. 5. Use rubrics. 6. Close the loop.7. If you rely on adjunct faculty, include them

in assessment.8. Keep a written record.

Some Friendly Suggestions