Refining Your Assessment Plan: Session 2

Direct Evidence of Student LearningRyan Smith and Derek HerrmannUniversity Assessment Services (UAS)

IntroductionsRefining Your Assessment Plan: Session 2

Introductions• Ryan Smith, Director, UAS• Derek Herrmann, Coordinator, UAS• Assessment Advisory Council (AAC)• Assessment Academy Team

Introductions• Who you are• In what Department/School you work• Why you are here

Overview of PRAAPRefining Your Assessment Plan: Session 2

Overview of PRAAP• Process for the Review of Academic

Assessment Plans• Two years before Program Review (and

one year before Self-Study)• AAC members and UAS staff• Use an established rubric

Overview of PRAAPElements Undeveloped Developing Established Exemplary

Goals and Learning OutcomesDirect Evidence of Student LearningIndirect Evidence of Student LearningUse of the Results

Direct Evidence of Student Learning

Refining Your Assessment Plan: Session 2

Direct Evidence of Student Learning• Tangible, visible, self-explanatory, and

compelling evidence of exactly what students have and have not learned

Direct Evidence of Student Learning• Performance assessments and

traditional assessments– Traditional assessments

• Multiple-choice tests, essay tests, and oral examinations

• Usually designed only to collect assessment information, not give students a learning opportunity

Direct Evidence of Student Learning• Performance assessments and

traditional assessments– Performance assessments

• Ask students to demonstrate their skills rather than relate what they have learned through traditional tests

• Increasingly popular because they merge learning and assessment

Direct Evidence of Student Learning• Embedded and add-on assessments

– Embedded assessments – program, general education, or institutional assessments that are embedded into course work

– Add-on assessments – beyond course requirements

Direct Evidence of Student Learning• Quantitative and qualitative assessments

– Quantitative assessments – use structured, predetermined response options that can be summarized into meaningful numbers and analyzed statistically

– Qualitative assessment – use flexible, naturalistic methods and are usually analyzed by looking for recurring patterns and themes

Direct Evidence of Student Learning• Effective assessment formats

– Paper/thesis – students develop an argument and support it with information or data they have gathered

– Project – students complete an assignment over a prolonged period of time

– Development of a product – project whose focus is on the development of a tangible product

Direct Evidence of Student Learning• Effective assessment formats (cont’d)

– Performance – students prepare and present a performance of a valued activity

– Exhibition – a project, product, or performance that is presented to judges and defended or debated with them

– Case study/critical incident – students are given a realistic example of an application in their field

Direct Evidence of Student Learning• Effective assessment formats (cont’d)

– Clinical evaluation – students perform a professional service in a real-life setting

– Oral exam – students answer spontaneous questions put to them by experts

– Interview – similar to an oral exam but the forum in which it is carried out may not be as public or involve as many questioners as an oral exam

Direct Evidence of Student Learning• Effective assessment formats (cont’d)

– Comprehensive exam – students complete a time-limited essay test that requires them to organize and present central ideas, facts, and concepts in response to questions

– Portfolio – students gather examples of their work, write about aspects of their learning and achievement, and include their written reflections

Direct Evidence of Student Learning• Effective assessment formats (cont’d)

– Portfolios• Showcase portfolio – document the extent of learning by

featuring the student’s best work• Developmental portfolio – designed to show student

progress and includes evidence of growth by comparing products from early and late stages of the student’s academic career

• Collective portfolio – collections of student work that are created by faculty for assessment purposes

Curriculum Mapping (Part 2)Refining Your Assessment Plan: Session 2

Curriculum Mapping• Analyze learning opportunities by

mapping learning goals against courses

Curriculum MappingCourse Goal 1 Goal 2 Goal 3 Goal 4

111 X X

138 X X

200 X X

231 X X X

331 X X

340 X X

392 X X X X

Curriculum MappingCourse Goal 1 Goal 2 Goal 3 Goal 4

111 Paper/thesis Paper/thesis

138 Paper/thesis Paper/thesis

200 Project Project

231 Performance Performance Performance

331 Exhibition Exhibition

340 Case study/critical incident

Case study/critical incident

392 Clinical evaluation Clinical evaluation Clinical evaluation Clinical evaluation

Curriculum Mapping• Transcript analysis – review a sample of

transcripts of graduating students to learn courses they choose and when they take them

Curriculum Mapping• Syllabus analysis – can help determine

if students have enough assignments and classwork to achieve each of the intended learning goals of the course, including the program, general education, or institutional goals that course is intended to support

RubricsRefining Your Assessment Plan: Session 2

Rubrics• A rubric is a scoring guide: a list or chart

that describes the criteria you and perhaps your colleagues will use to evaluate or grade completed student assignments

Rubrics• Using a rubric to grade student

assignments…– Makes your life easier– Improves student learning

Rubrics• A checklist rubric is

a simple list indicating the presence of the things you’re looking for in a completed assignment

Criteria PresentCriterion 1

Criterion 2

Criterion 3

Criterion 4

Criterion 5

Rubrics• A rating scale rubric

is a checklist with a rating scale added to show the degree to which the things you’re looking for are present in completed assignments

Criteria Not Present Developing Established Advanced

Criterion 1

Criterion 2

Criterion 3

Criterion 4

Criterion 5

Rubrics• Descriptive rubrics

replace the checkboxes of rating scale rubrics with brief descriptions of the performances that merit each possible rating

Criteria Not Present Developing Established Advanced

Criterion 1Description of

the performance that merits this

rating

Description of the performance that merits this

rating

Description of the performance that merits this

rating

Description of the performance that merits this

rating

Criterion 2Description of

the performance that merits this

rating

Description of the performance that merits this

rating

Description of the performance that merits this

rating

Description of the performance that merits this

rating

Criterion 3Description of

the performance that merits this

rating

Description of the performance that merits this

rating

Description of the performance that merits this

rating

Description of the performance that merits this

rating

Criterion 4Description of

the performance that merits this

rating

Description of the performance that merits this

rating

Description of the performance that merits this

rating

Description of the performance that merits this

rating

Criterion 5Description of

the performance that merits this

rating

Description of the performance that merits this

rating

Description of the performance that merits this

rating

Description of the performance that merits this

rating

Rubrics• Creating effective rubrics

– Look for models– List the things you’re looking for– Leave room for the ineffables and the

unexpected

Rubrics• Creating effective rubrics (cont’d)

– Create the rating scale• Create at least three levels• Label each level with names, not just numbers• If you are developing a descriptive rubric, fill in

its boxes– Try out the rubric

Questions, comments, and discussion

Refining Your Assessment Plan: Session 2