INCREASING EQUITY USING CULTURALLY RELEVANT … · Presentation. Inclusive Project. Reflective...

INCREASING EQUITY USING CULTURALLY RELEVANT ASSESSMENT

Karen Singer-Freeman, Director of Academic Planning & Assessment

Christine Robinson, Executive Director

Harriet Hobbs, Director of Assessment Systems & University Accreditation

PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION

Office of Assessment and Accreditation

http://assessment.uncc.edu/

2018 Designee

OVERVIEW

•Learning Outcomes for the Workshop•Approach to analyzing equitability of assessments•Culturally Relevant Assessment

•Theoretical model•Student feedback

•Data dive•Engaging faculty in conversation


WORKSHOP LEARNING OUTCOMES

By the end of the workshop, participants will be prepared to:

1) Identify questions and practices about inclusive assessment and student learning that can be addressed using existing sources of campus data.

2) Identify strengths and weaknesses of assignment types.

3) Understand the importance of disaggregating data by student characteristics.


Equity in Assessment and Data Sources• What questions could we ask about assessments and

equity?• Do different assignments measure similar abilities?• Do assignment formats influence students’ abilities to

demonstrate learning?• Do student learning outcome reports capture important

differences in student success?

• Utilize software systems’ data • Learning Management Systems (LMS)• Assessment Management Systems (AMS)• Student Information Systems (SIS)


STUDENT LEARNING OUTCOME (SLO)

REPORTS


Do student learning outcomes reports capture important differences?

•Student Learning Outcomes Reporting can compress data•Full assignment grade

•5 point ratings on multiple rubric dimensions•Average across rubric grades

•Above or below a criterion cut off

•Using Learning Management Systems it is possible to disaggregate data to see a more complete picture


SLO Report ExampleITCS 2300 Web-Based Application Development-Computer Science• SLO- Students demonstrate ability to create static web pages…

• Assignments - 4-6 webpages using a common theme: • 4 Criteria: Understanding target demographic, Effective use of HTML and JavaScript,

Browser compatibility, and Aesthetics.• Target: 80% will receive average rubric score of 3 (acceptable) or higher• Result: 97% received average rubric score of 3 (acceptable) or higher

• What other questions do we have?

2 3 11

47

139

0

50

100

150

1 2 3 4 5

Number of Students Receiving Each Rubric Rating


Data Disaggregation: Is Achievement Equitable?

0% 0%4%

25%

72%

2% 3%7%

22%

67%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1 2 3 4 5

% Receiving Each Rating Disaggregated by Ethnicity

Non-URM (n = 146) URM (n = 60)

5% of URM Don’t meet but all Non-URM meet7.04 (1), p < .01


Does the difference in achievement matter?

3.76

3.15

3.66

3.01

0

0.5

1

1.5

2

2.5

3

3.5

4

Semester GPA Cummulative GPA

Average Semester and Cumulative GPAURM grades marginally lower than Non-URM, p < .10

Non-URM (n = 146) URM (n = 60)PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION

STUDENT SUCCESS: ASSIGNMENT TYPES

Activity1) Please pick up the paper on your seat and circle the items

that would generally describe this sort of assignment.

2) Compare your assignment and responses with the person next to you

3) Total the number of odd items and the number of even items you selected

4) Discuss – What do you think these questions are trying to measure?


Culturally Relevant Assessment(Singer-Freeman, Hobbs, & Robinson, 2019)

• Aligned with teaching and SLOs

• Clear instructions• Limits effects of prior knowledge and privilege

• Scaffolding• Limits effects of prior knowledge and creates opportunities for early

success

• Inclusive content

• High utility value• Work has meaning beyond the academic context

• Avoids stereotype threat

• Results disaggregated


Predictions Based on Analysis of Assignment Types

Low Risk

• Reflective writing

Moderate Risk

• Open-ended tests

• Inclusive projects

• Writing in discipline

High Risk

• Multiple-choice tests

• Formal Essays

• Homework

High Utility Value Low Utility Value

High Inclusive Content

Reflective Writing Inclusive Projects

LowInclusive Content

Writing in DisciplineFormal Essays,

Tests, HomeworkPLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION

What do students think about different types of assignments?

•16 items assess utility value, inclusive content, & alignment

•Completed for extra credit (n = 162)•Lower level classes in

• Biology (2)• Theater (1)• Writing (2)

•Upper level class in • Public Health

42% 40%35%

0%10%20%30%40%50%

Non-White 1stGeneration

Male

19%10%

46%

24%

0%

10%

20%

30%

40%

50%


The Measure

Utility Value (Chronbach’s Alpha = .90)

• Result in something I will discuss with others.

• Help me understand myself better.

• Improve my understanding.

• Provide me with experience that will be professionally useful.

• Have personal value.

----------------------------------

• Are only completed to earn a grade.

Inclusive Content (Chronbach’s Alpha = .93)

• Allow me to express my learning in my own words.

• Make me feel confident I can succeed.

• Include examples and materials that are familiar to me.

• Measure my true understanding.

• Allow me to relate class materials to my own experiences.

• Include clear instructions.


Assignments RatedPlease indicate the extent to which _______ generally:Formal Papers or Essays

Reflective Writing in which you relate class materials to your own experiences.

Inclusive Projects – Projects that allow you to write in different styles or using familiar content.

Writing in Discipline – Writing assignments in the format used in a career you are interested in pursuing (e.g., lab reports, progress reports, theater reviews).

Oral Presentations

Short Answer Test Questions

Multiple-Choice Test Questions

oNot at all, A little, To a moderate extent, Very muchPLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE

WITHOUT PERMISSION

Student Responses – Utility Value

2.78 2.722.59 2.53 2.44

2.282.12.15

1.912.12 2.17

1.99 1.911.65

1

1.5

2

2.5

3

3.5

4

Writing inDiscipline

OralPresentation

InclusiveProject

ReflectiveWriting

FormalPapers

ShortAnswer Test

Mult. ChoiceTest

Average Rating on 4 point Likert ScaleCritical Value for Utility Value Construct = .11, p < .05

Utility Completed for more than a gradePLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION

Student Responses – Inclusive Content

2.81 2.77 2.77 2.75 2.742.49 2.49

1

1.5

2

2.5

3

3.5

4

Writing inDiscipline

OralPresentation

InclusiveProject

ReflectiveWriting

ShortAnswer Test

FormalPapers

Mult. ChoiceTest

Average Rating on 4 point Likert ScaleCritical Value = .11, p < .05


Student Confirmation of Model

High/ModerateUtility Value

Low /Moderate Utility Value


Reflective WritingOral Presentations

Inclusive Projects


Writing in DisciplineFormal Essays

Tests

Possible Correction Confirmation Addition

PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE

WITHOUT PERMISSION

Some Research Support(Singer-Freeman, Hobbs, & Robinson, 2019)

Theater Appreciation (n = 64)• 69% URM

• Achievement gap • Multiple-choice exams

• No gap • Inclusive projects

Child Development (n = 110)• 44% URM

• Achievement gap • In-class multiple choice quizzes

• No gap • Reflective writing • Online multiple choice quizzes

9073

9483

0

50

100

Inclusive Project Multiple-Choice*

Theater AppreciationURM Non-URM

9077 85

93 85 87

0

50

100

ReflectiveWriting

In-ClassQuizzes*

OnlineQuizzes

Child Development


Experimental Psychology

• 5 semesters

• 135 students • 30% URM

• Achievement Gap• URM students receive

lower grades on exams than on lab reports

• Non-URM students receive similar grades on exams and lab reports

7973

80 76

0102030405060708090

100

Writing in theDiscipline

Open-Ended Exams

Experimental Psychology

URM* Non-URMPLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION

Research Confirmation of Model

High/ModerateUtility Value

Low /Moderate Utility Value


Reflective WritingOral Presentations

Inclusive Projects


Writing in DisciplineFormal Essays

Tests

Possible Correction Confirmation Addition


Discussion

•Do our measures capture utility value and inclusive content?

•Do the predictions in the model and findings make sense given your experience with classroom assessments?

•What else would you want to know about how different groups of students respond to different types of assignments?


BREAK TIME


DISAGGREGATING RESULTS BY ASSESSMENT TYPE


Our Process

•Obtained a list of 88 classes with enrollments of over 50 students and DFW rates of over 30%

•Reviewed 36 classes • Identified 9 that included different forms of assessment and

reported grades through LMS• The majority of classes we rejected only included testing

•Of the 9 classes with differentiated assignments • 100% included Exams and Quizzes

• 2 Formal essays and 1 Writing in the Discipline (Papers)• 2 Projects and 1 Oral presentation (Projects)• 2 Homework problem sets and 3 Informal writing (Homework)


Classes•Pre-Calculus MATH 1103•Organic Chemistry Lab CHEM 2131L•Introduction to Communication Theory COMM 2100•Network Theory ENGR 2112•Principles of Accounting ACCT 2122 (flipped)•Physiological Psychology PSYC 3113•Design and Implementation ITCS 3112•Sociology of Health & Illness SOC 4120•Conservation Biology BIOL 4244


How to disaggregate data•Download grades, assignment descriptions, and syllabi for desired course from LMS• Consider merging data from multiple semesters if enrollments

are low

•Decide whether to include or exclude• Missing assignments• Completion-based grades

•Aggregate grades for equivalent assignments•Convert points to percentages •Merge file with demographic data from Institutional Research

•Compare outcomes using independent groups t-testPLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION

Introduction to the data tables

Partner with someone at your table and pick a dataset

p < .05 p < .10 p < .05 p < .10

Achievement Gaps Reverse Achievement Gaps

Title provides broad introduction to contents

N = Total number of students in the class

(URM, Transfer, or Aid-Based Loans) = Number of students who were members of disaggregated group

Columns labeled with groups contain average percentage scored on each type of assignment

sig. = Whether observed differences were statistically significantPLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION

Assignment Class N (URM)

Non-URM

URM sig N (Aid Loans)

No Loan

Aid Loan

sig N (Trans)

Native

Trans sig

Reading Response 20% Sociology of Health & Illness SOCY 4130

118 (63)

78 73 .05 129 (33)

75 79 ns 130 (81)

82 72 .001 Exams 25% (open book, online) 77 71 .01 72 76 .09 77 71 .01 Paper 35% 82 78 .05 79 83 .09 84 77 .001 Final Grade 77 70 .01 71 78 .001 76 70 .02 Assignment Class N

(URM) Non-URM

URM sig N (Aid Loans)

No Loan

Aid Loan

sig N (Trans)

Native

Trans sig

Quizzes (20%) Physiological Psychology PSYC 3113

51 (18)

87 83 ns 55 (17)

86 84 ns 54 (37)

90 84 .01 Exams (45%) 80 79 ns 79 78 ns 84 77 .04 Oral (15%) 97 94 ns 94 98 ns 97 96 ns Final Grade 82 81 ns 80 80 ns 82 80 ns

Assignment Class N (URM) Non-URM URM sig Quizzes 20%

Network Theory II ENGR 2112

42 (8)

85 84 ns Tests 40% 66 66 ns Final Exam 30% 72 65 ns Homework 10% 74 65 .09 Final Grade 67 65 ns Assignment Class N (URM) Non-URM URM sig Pre-Lab Quiz

Organic Chemistry Lab CHEM 2131L

141 (43) 78 70 ns Lab reports 80 75 ns Final grade 81 76 ns Assignment Class N (URM) Non-URM URM sig Exams (70%)

Introduction to Communication Theory COMM 2100

136(54)

73 72 ns Application Paper (10%) 88 84 .09 Minute Responses (20%) 97 92 .07 Final Grade 72 70 ns

Assignment Class N (URM) Non-URM URM sig Quizzes (7%)

Principles of Accounting ACCT 2122 (Fli d Cl )

50 (18)

89 89 ns Exams (64%) 75 72 ns Shark Tank Group Project (14%) 85 88 .06 Homework (7%) 81 87 ns

Introductory Classes Disaggregated by URM status

Advanced Classes in Sociology and Psychology


Assignment

Class

N (URM)

Non-URM

URM

sig

Quizzes 20%

Network Theory II

ENGR 2112

42 (8)

85

84

ns

Tests 40%

66

66

ns

Final Exam 30%

72

65

ns

Homework 10%

74

65

.09

Final Grade

67

65

ns

Assignment

Class

N (URM)

Non-URM

URM

sig

Pre-Lab Quiz

Organic Chemistry Lab

CHEM 2131L

141 (43)

78

70

ns

Lab reports

80

75

ns

Final grade

81

76

ns

Assignment

Class

N (URM)

Non-URM

URM

sig

Exams (70%)

Introduction to Communication Theory

COMM 2100

136(54)

73

72

ns

Application Paper (10%)

88

84

.09

Minute Responses (20%)

97

92

.07

Final Grade

72

70

ns

Assignment

Class

N (URM)

Non-URM

URM

sig

Quizzes (7%)

Principles of Accounting

ACCT 2122 (Flipped Class)

50 (18)

89

89

ns

Exams (64%)

75

72

ns

Shark Tank Group Project (14%)

85

88

.06

Homework (7%)

81

87

ns

Final Grade

77

75

ns

Assignment

Class

N (URM)

Non-URM

URM

sig

Tests (80%)

Pre-Calculus

MATH 1103

101 (33)

79

78

ns

MyMathLab (20%)

79

80

ns

Final Grade

76

75

ns

Dive into some dataAnswer the following questions:1) For which classes do the final grades tell the same story as

the assignment grades (look at significant and non-significant differences)?

2) For which classes do different assignments result in different patterns of performance than final grades (look at significant and non-significant differences)?

3) Can you identify any particular type of assignment that might be showing evidence of achievement gaps?

4) What else would you want to know?

5) Pick a class from the handout and outline some questions and suggestions you might have for the faculty member who teaches the class.

6) What other observations did you make?PLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE WITHOUT PERMISSION

URM Results

-6

-4

-2

0

2

4

6

8

10

12

Tests Papers Projects Homework Final Grade

Points Separating non-URM from URM students

MATH CHEM COMM ENGR ACCT PSYC ITCS SOC BIOLPLEASE DO NOT COPY, REPRODUCE, OR DISTRIBUTE

WITHOUT PERMISSION

Transfer Student Results

-6

-4

-2

0

2

4

6

8

10

12


Points Separating Native fromTransfer Students


Aid-Based Loan Student Results

-6

-4

-2

0

2

4

6

8

10

12


Points Separating non-Loan from Aid-Based Loan students


Summary

FrequentAchievement Gaps

• Tests• Low stakes or high stakes• Multiple choice or open ended• Open-book and Online

• Low-Stakes Homework, Writing, and Projects

• Formal Writing

Fewer Achievement Gaps

• Writing• Writing in the discipline• Reflective writing• Inclusive projects

• Oral Report• Group Project


CONCLUSIONS &

NEXT STEPS


Next Steps• Disaggregation of responses to different forms of assessment

reveals importance of• Aligning assessments with teaching and SLOs• Attending to context

• Faculty Workshops• Differentiated Assignments/Assessments

• Utility Value and Inclusive Content• Implicit Bias

• Scale project to other courses and programs

• Determine if achievement gaps reflect assessments or competence


DISCUSSION

If achievement gaps exist on your campus, after hearing this talk, what new ways might you use to investigate these gaps?

• Who would you include in the conversation?

• What are approaches you could take to determine whether gaps reflect assessments or competence?

• What would next steps include to address gaps that result from each?


SPECIAL THANKS

UNC Charlotte Academic Affairs

Arna Erega

Students and Faculty


REFERENCESAssociation of American Colleges & Universities (2018). A vision for

equity: results from AAC&U’s project committing to equity and inclusive excellence: Campus-based strategies for student success. Retrieved from https://aacu.org/publications/vision-equity.

Eccles J. S. et al. Expectations, values and academic behaviors. In: Spence JT, editor. Perspective on achievement and achievement motivation. San Francisco: W. H. Freeman; 1983. pp. 75–146.

Harackiewicz et al. (2016). Achievement gaps with a utility-value intervention: Disentangling race and social class. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4853302/

Montenegro, E., & Jankowski, N. A. (2017). Equity and assessment: Moving towards culturally responsive assessment (Occasional Paper No. 29). Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment (NILOA).

Singer-Freeman, K., Hobbs, H., & Robinson, C. (2019). Theoretical matrix of culturally relevant assessment. Assessment Update.


C o n t a c tInformation

Christine Robinson, [email protected] Harriet Hobbs, [email protected]

Karen Singer-Freeman, [email protected] DO NOT COPY, REPRODUCE, OR DISTRIBUTE

WITHOUT PERMISSION

Increasing Equity using culturally Relevant AssessmentOverview Workshop Learning outcomesSlide Number 4Student Learning Outcome (SLO) ReportsDo student learning outcomes reports capture important differences?SLO Report ExampleData Disaggregation: �Is Achievement Equitable?Does the difference in achievement matter?Student Success: ASSIGNMENT TypesActivityCulturally Relevant Assessment�(Singer-Freeman, Hobbs, & Robinson, 2019) Predictions Based on Analysis of �Assignment TypesWhat do students think about �different types of assignments?The MeasureAssignments RatedStudent Responses – Utility ValueStudent Responses – Inclusive ContentStudent Confirmation of ModelSome Research Support�(Singer-Freeman, Hobbs, & Robinson, 2019)Experimental PsychologyResearch Confirmation of ModelDiscussionBreak timeDisaggregating Results by assessment typeOur ProcessClassesHow to disaggregate dataIntroduction to the data tablesSlide Number 30Dive into some dataURM ResultsTransfer Student ResultsAid-Based Loan Student ResultsSummaryConclusions �& �Next StepsNext StepsDiscussionSpecial thanksReferencesContact Information

INCREASING EQUITY USING CULTURALLY RELEVANT … · Presentation. Inclusive Project. Reflective...

Documents

Transcript of INCREASING EQUITY USING CULTURALLY RELEVANT … · Presentation. Inclusive Project. Reflective...