Paula O’Neill, MEd, EdDAssociate Dean for Educational Research

Professor, Diagnostic SciencesThe University of Texas Dental Branch at

HoustonPaula.N.Oneill@uth.tmc.edu

1

Rationale and role in using rubrics in dental education

Describe best practices for constructing rubrics for dental education.

A model for evaluating rubrics

2

A Rubric shows how learners will be assessed and/or graded.

A rubric provides a clear guide as to how ‘what learners do’ in a course will be assessed.

3

Scoring tool◦ Specific expectations

Criteria and standards◦ Linked to learning objectives

Consistent assessment criteria

Shared with students◦ Work towards success

Can be either grading or performance rubrics

4

1. You give careful descriptions of assignments, but students are still asking you many questions about the assignment;

2. You have been disappointed by the outcomes of assignments because your students seem to be unaware of your academic expectations;

3. You want students to be able to solve complex problems, yet students fail to understand those expectations;

4. You receive feedback that “your grading practices are not be calibrated with other faculty.”

5

Discussion participation Laboratory work Clinical competencies Group work Oral presentations Essay exams Research papers Student portfolios

6

Reduced grading time Memory of assignments,

competencies, etc at the “front of the mind.”

Provides more specific intentions and expectations

Valuable teaching and learning tool◦ More aware of teaching styles and strategies

Problem solving and critical thinking Classroom dialogue between faculty &

students

7

1. Reflecting

2. Listing criteria

3. Grouping and labeling

4. Application

Other considerations◦ Student involvement?◦ Evaluation of rubric effectiveness

8

What do you want from the students? Why did you create the assignment/test? How does this assignment related to other

content? What skills do students need to have/develop to

“successfully” complete the assignment? What is the exact task assigned? What evidence will students provide to

demonstrate accomplishment? Your highest expectation-Excellence- for

performance is? What is the Poorest fulfillment of the assignment?

9

Specific Learning Objectives

Objectives will vary

10

Organization of Steps 1 and 2

Group similar expectations together in what could become ◦ Dimensions

11

Transfer lists and groups to rubric grid Labels become Dimensions◦ Placement: left column of rubric

Learning tasks/objectives ◦ Become descriptions of highest performance level

Exemplary Competent Developing

Individualpresentations

Spoke clearly, slowly and loudly modulating voice and quality

Was understood but mumbled, spoke too fast or slow, whispered, shouted, not compromised

Mumbled, spoke too softly-loudly, intelligibility was compromised

Group Work Each member given equal opportunity to shine

Presentations followed logically but connections unclear

Little logic evident, connections between presenters unclear

12

1. Task description measuring specific objective

◦ Performance, behavior or quality

2. Scale/range indicating levels of competency, achievement

3. Dimensions of the assignment/task

4. Description of what constitutes each level of performance

KEY COMPONENTS OF DEVELOPMENT

◦ Determine level of student involvement

◦ Pretest/pilot rubric

13

Framed by faculty member◦ Focused upon “performance” expectation◦ Specific task or overall behavior

SCALE Level 1 SCALE Level 2 SCALE Level 3

Dimension 1

Dimension 2

Dimension 3

Dimension 4

14

Description of how well or poorly any expected task has been performed

Start with 3 levelsExcellent Competent Needs Work

Dimension 1

Dimension 2

Dimension 3

Dimension 4

15

Lay out the parts of the rubric

Start with Task Analyses

Provide clarification for the students of how the task for them to accomplish can be◦ Broken into components◦ Which of the components is most important i.e.., facts, research techniques, skill competency◦ Description of how much weight will be given to

each of the components

16

Should not include any description of the QUALITY of the performance◦ “Good, outstanding, poor etc.

Together with good descriptions, the dimension provide detailed feedback on specific parts of an assignment◦ How well or poorly accomplished

17

Description of highest-lowest level of performance

Excellent Competent Needs Work

Knowledge40%/40 points

Thinking30%/30 points

Communication20%/20 points

Presentationskills 10%/10 Points

18

Student’s highest level of performance Three levels of performance more useful

Excellent Competent Needs Work

Knowledge40%/40 points

The case report demonstrates a depth of understanding

Report generally accurate with only minor inaccuracies

Report included little relevant or accurate info about patient

Thinking30%/30 points

The report centered about high level of conceptual ability

Analyticalstructure but analysis not always linked

No analyticalstructure evident

Communication20%/20 points

Case presented effectively & linked to lab reports

Case presented well but questions not all answered

Case as presented confusing in what is communicated

Presentationskills 10%/10 Points

Student speaks clearly, using appropriate terminology

Student speaks clearly, but terminology not always used

Student does not speak clearly nor use correct terminology

19

20

Presentation model — most common◦ Faculty member does all the work, makes decisions◦ Sets expectations, weight, scale, levels and

fulfillmentRubric ConstructionModel

Stage 1:Reflecting

Stage 2: Listing

Stage 3: Groupingand Labeling

Stage 4: Application

1.Presentation

Facultymember

Faculty member

Faculty Member

Faculty member & students who ask questions

21

Actually present to class and encourage feedback for revision purposes◦ Small group activity

Rubric ConstructionModel

Stage 1:Reflecting

Stage 2: Listing

Stage 3: Grouping & Labeling

Stage 4: Application

2. Feedback Model

Professor Professor Professor Professor & Students who edit to clarify

22

Students have high level of flexibility and creatively to develop rubric task expectations

Rubric ConstructionModel

Stage 1:Reflecting

Stage 2: Listing

Stage 3: Grouping & Labeling

Stage 4: Application

3. Pass-the-Hat Model

Professor Professor & Students

Professor & studentswho group student contributions

Professor & Students who create final rubric

23

Provides more control to students who create both dimensions and some of the descriptions

Rubric ConstructionModel

Stage 1:Reflecting

Stage 2: Listing

Stage 3: Grouping & Labeling

Stage 4: Application

4. Post-It Model

Professor Students Professor & students who facilitate grouping

Professor & Students who create final rubric

24

Faculty role limited to setting the assignment, explaining what the finished rubric will resemble, and facilitating the creation by students

Rubric ConstructionModel

Stage 1:Reflecting

Stage 2: Listing

Stage 3: Grouping & Labeling

Stage 4: Application

5.4X4 Model

Professor & students

Students Professor & studentswho facilitate grouping

Students

25

Criteria 1Unacceptable

2Acceptable

3Good/Solid

4Exemplary

Clarity of criteriaCriteria being assessed are unclear, inappropriate and/or have significant overlap

Criteria being assessed can be identified, but are not clearly differentiated or are inappropriate

Criteria being assessed are clear, appropriate and distinct

Each criteria is distinct, clearly denoted and fully appropriate for the assignment(s)/course

Distinction between Levels

Little/no distinction can be made between levels of achievement

Some distinction between levels is evident, but remain unclear

Distinction between levels is apparent

Each level is distinct and progresses in a clear and logical order

Reliability of ScoringCross-scoring among faculty and/or students often results in significant differences

Cross-scoring by faculty and/or students occasionally produces inconsistent results

There is general agreement between different scorers when using the rubric (e.g. differs by less than 5-10% or less than ½ level)

Cross-scoring of assignments using rubric results in consistent agreement among scorers

Clarity of Expectations/

Guidance to Learners

Rubric is not shared with learners

Rubric is shared and provides some idea of the assignment/ expectations

Rubric is used to explicitly introduce an assignment and guide learners

Rubric serves as primary reference point for discussion and guidance for course/assignment(s) as well as evaluation of assignment(s)

Support of Metacognition

(Awareness of Learning)

Learners do not see/know of the rubric

Rubric is shared but no further reference is made to it in the course/ assignment(s)

Rubric is shared and identified as a tool for helping learners to understand what they are learning through the assignment/ in the course

Rubric is regularly referenced and used to help learners identify the skills and knowledge they are developing throughout the course/ assignment(s)

Engagement of Learners in Rubric Development/ Use *

Learners are not engaged in either development or use of the rubrics

Learners offered the rubric and may choose to use it for self assessment

Learners discuss and offer feedback/input into the design of the rubric, and are responsible for use of rubrics in peer and/or self-evaluation

Faculty and learners are jointly responsible for design of rubrics and learners use them in peer and/or self-evaluation

A Rubric for RubricsA Tool for Assessing the Quality and Use of Rubrics in Education

*Considered optional by some educators and a critical component by othersScoring chart: 0 - 10 = needs improvement 11 - 15 = workable 16 – 20 = solid/good 21 - 24= exemplary

© 2002 BBMullinix (Mullinix@tltgroup.org, bbmullinix@gmail.com)web url: http://www.tltgroup.org/resources/Rubrics.htm

Timely feedback Prepare students to use detailed feedback Encourages critical thinking Facilitate communication Helps faculty refine teaching methods Consistently levels the “playing field”

27

Carnegie Mellon. Grading and Performance Rubrics.◦ http://www/cmu.edu/teaching/designteach/teac

h/rubrics.html Fink, DL. Creating Significant Learning Experiences.

2003; San Francisco. Jossey-Bass Publishers. Schreyer Institute for Innovation in Learning

http://www.inov8.psu.edu/toolbox/RubricBasics.pdf

Stevens DD, Levi AJ. Introduction to Rubrics. 2005. Sterling, VA. Stylus Publishing Co.

2002 BBMullinix (Mullinix@tlttltgroup.org)◦ http://www.tltgroup.org/resources/Rubrics.htm

28