Interpreting Data & “Closing the Loop”
Margaret Kasimatis, PhDVP for Academic Planning & Effectiveness
ARTICULATEMISSION/
GOALS
IDENTIFYSPECIFIC
OUTCOMES
DETERMINEPRACTICES USED
TO ACHIEVEOUTCOMES
GATHEREVIDENCE
REVIEW &INTERPRETRESULTS
RECOMMENDACTIONS
Location in Assessment Cycle
Step #1: Organize Your Data
• Assemble the following in one place:– All data or possible sources of data– List of learning outcomes/research questions– Curriculum, experiences, practices
• Map data sources to outcomes– Outcomes x Measures Map
• Map results to outcomes– Present in “user-friendly” way
LEARNING OUTCOMES
ASSESSMENT MEASURES
Alumni Survey
National Survey of Student Engagement
(NSSE)
Evaluation of Clinic
Presentations
Evaluation of First-Year
Presentations
Rubric Evaluation of Upper-level Coursework
Ability to apply knowledge in math, science, and engineering X X
Ability to design a system, component, or process to meet desired needs X
Ability to identify, formulate, andsolve engineering problems X X X
Ability to communicate effectively X X X X XAbility to use techniques, skills, and modern engineering tools necessary for engineering practice
X X
Ability to function on multidisciplinary teams X X X
Recognition of need for and abilityto engage in lifelong learning X X
Understanding of professional andethical responsibility X X
Broad education necessary tounderstand the impact of engineering solutions in a global & societal context
X X
Mapping Data Sources to Outcomes:Engineering Example
• It’s best to present results one outcome at a time– It’s not recommended to present results primarily
by measure
• Make sure to present the data in a user-friendly way (i.e., in a way that faculty in your department are comfortable with)
Presenting Results
NSSE: Data on Communication Skills
1
2
3
4
Experience contributed to abilityto write clearly and effectively
Experience contributed to abilityto speak clearly and effectively
Mea
n R
esp
on
se (
1-4
scal
e)
HMC Senior Engineers
AITU Seniors
Liberal Arts Seniors
0
10
20
30
40
50
60
To what extent was an emphasis onwriting skills present while you were at
HMC?
To what extent was an emphasis on oralcommuniction skills present while you
were at HMC?
Pe
rce
nt
Ind
ica
tin
g "
Ve
ry M
uc
h"
Biology
Chemistry
Comp Sci
Engineering
Math
Physics
Alumni Survey: Data on Communication Skills Rubric Evaluation of Student Presentations
1
2
3
4
First-Year Presentation Clinic Presentation
DeliveryLanguageOrganizationContent
Rubric Evaluation of Student Work: Writing Style
1
2
3
4
5
Course #1 Course #2
Mea
n R
ub
ric
Sco
re o
n W
riti
ng
Sty
le (
1-5
scal
e)
ACADEMIC CHALLENGE: LMU Jesuit Master’s NSSETotal
How often did you work harder than you thought you could to meet an instructor's standards or expectations?(Percent of students who said ‘Often’ or ‘Very Often)
FY 62% 55% 54% 52%
SR 54% 59% 60% 56%
How much time did you spend preparing for class (e.g. studying, reading, writing, doing homework, rehearsing,
other academic activities)? (Percent of students who reported spending 11 or more hours per week)
FY 65% 68% 49% 56%
SR 40% 58% 52% 54%
How often did you come to class without completing readings or assignments? (Percent of students who said ‘Often’ or ‘Very Often’)
FY 21% 17% 17% 19%
SR 27% 23% 20% 23%
Describe the extent in the current year to which exams challenged you to do your best work. (mean response on a 7-point scale)
FY 5.47 5.56 5.33 5.42
SR 5.33 5.46 5.43 5.40
Step #2: Interpret the Data
• What type of criterion?
• What is “significant?”
• Are the findings reliable/valid?
What type of criterion?
• Norm-referenced– Need appropriate comparison group
• Avoid percentile rankings
– Good for more nebulous findings
• Absolute standard– Usually more appropriate for
performance-based measures
What is significant?
• Important to test when making comparisons– Numbers that look different may not really
be different
• However, just because a difference is significant doesn’t mean it’s important
Are findings reliable/valid?
• Can we believe student self-reports?
• Are standardized measures more valid?
• Single measure vs. multiple measures
Step #3: Make Recommendations
• Start by considering– Where in the curriculum are outcomes
addressed? And at what level?• Refer to (or create) curriculum map, or
• Inventory for particular outcome
– How? What practices/techniques are used?
– Where are the gaps?
– What can be changed?• In the short-term? In the long-term?
Step #3: Make Recommendations
• Possible types of recommendations:– Changes to pedagogy– Changes to curriculum/programming– Allocation of resources
• Keep in mind that you can’t fix everything at once – so start small
• It’s usually better to modify than add
• Be as specific/concrete as possible
PROPOSED ACTION IMPLEMENTATION
Document (with grades) the importance of presentation clarity
Essay questions added to exams
Engineering Curriculum Committee developed writing rubric to be used to evaluate project report
Make common writing style guidelines available to faculty and students
Department selected Hacker, A Writer’s
Reference
Hacker, A Writer’s Reference required in E4
Hold Clinic team members
accountable for peer review
of team reports
Clinic advisors distributed peer review sheets to
Clinic team members. Students completed the
peer review sheets when they reviewed fellow
team members’ portions of Clinic report .
Step #4: Implement & Document Recommend Actions
Example
Example of Assessment Loop
• Step #1: Articulate Goals/Outcomes– Goal: Graduates will possess strong
communication skills• Outcome: Graduates will be able to write clearly and
effectively
• Step #2: Determine practices used to achieve outcome– Survey of required writing experiences in Core
and academic programs
Sem # BIO CHEM CS ENG MATH PHYS H/SS
One 4 papers
Two Report in E4 Tech report in Phys 28
4 papers
Three 1-2 papers in Bio 101
3 lab reports
Lab notebook & paper in Chem 53
Tech report
in Phys 53
Variable
Amount
Of
Writing
Four 4 lab reports
1-2 papers in Bio 108
Essays/paper
in Chem 52
Lab notebook Exp abstracts
Design doc,
Report, &
User guide
in CS 121
Tech report
in Phys 54
Term paper
in Phys 52
Five 1-2 papers in
Bio 109
3 lab reports
3 reports + paper in 109
Exp abstracts
Abstracts in CS 110
Proposal &
mid-year report OR
Six 1-2 papers
in Bio 113
Seminar paper
Project paper
in Chem 114
Essay in
CS 131
Clinic report Tech report in Phys 134
IE Paper IE Paper IE Paper IE Paper IE Paper IE Paper
Seven Proposal
3 Lab reports
Letter of Intro
Prog report
Proposal +
midyear
report
Proposal +
midyear
report
Seminar
Research
PaperEight Thesis Thesis Clinic
reportClinic report Thesis or
Clin. report
Thesis or
Clin. report
Example of Assessment Loop, cont’d
• Step #3: Conduct measurements– Higher Education Research Institute (HERI)
College Student Survey (CSS)– National Survey of Student Engagement– HMC Alumni Survey
• Step #4: Review & interpret results– Faculty committees
• Assessment• Curriculum• Teaching & Learning
– Full faculty - workshop
Example of Assessment Loop, cont’d
Writing SkillsFrequency data from HERI, NSSE & Alumni Survey
0
10
20
30
40
50
60
70
80
"Much stronger writingskills than when entering
college"
"Institution contributedvery much to ability to
write clearly andeffectively"
"To what extent was anemphasis on writing skillspresent while you were at
HMC?" (5)
HMC Seniors
4-Yr Non-Sectarian
1959-1987Cohort
1988-2001CohortHMC
Seniors
Liberal Arts
AITU Seniors
Seniors’ Perceptions of Writing Skills by Major (from NSSE)
1
1.5
2
2.5
3
3.5
4
Biology CS Engineering Mathematics PhysicalSciencesT
o w
hat
ext
ent
did
exp
erie
nce
s at
you
r in
stit
uti
on c
ontr
ibu
te
to y
our
abil
ity
to w
rite
cle
arly
an
d e
ffec
tive
ly?
Alumni Perceptions of Writing Emphasis by Major (alumni graduating in last 10 years)
0
10
20
30
40
50
60
70
Biology Chemistry CS Engineering Math Physics
Wri
tin
g em
ph
asis
was
"ve
ry m
uch
" p
rese
nt
Sem # BIO CHEM CS ENG MATH PHYS H/SS
One 4 papers
Two Report in E4 Tech report in Phys 28
4 papers
Three 1-2 papers in Bio 101
3 lab reports
Lab notebook & paper in Chem 53
Tech report
in Phys 53
Variable
Amount
Of
Writing
Four 4 lab reports
1-2 papers in Bio 108
Essays/paper
in Chem 52
Lab notebook Exp abstracts
Design doc,
Report, &
User guide
in CS 121
Tech report
in Phys 54
Term paper
in Phys 52
Five 1-2 papers in
Bio 109
3 lab reports
3 reports + paper in 109
Exp abstracts
Abstracts in CS 110
Proposal &
mid-year report OR
Six 1-2 papers
in Bio 113
Seminar paper
Project paper
in Chem 114
Essay in
CS 131
Clinic report Tech report in Phys 134
IE Paper IE Paper IE Paper IE Paper IE Paper IE Paper
Seven Proposal
3 Lab reports
Letter of Intro
Prog report
Proposal +
midyear
report
Proposal +
midyear
report
Seminar
Research
PaperEight Thesis Thesis Clinic
reportClinic report Thesis or
Clin. report
Thesis or
Clin. report
• Step #5: Recommend actions– Universal writing handbook– Faculty workshop on writing instruction– Revision to Core Curriculum
Example of Assessment Loop, cont’d
MULTIPLE CURRICULAR OBJECTIVESIN HORIZONTALLY INTEGRATED
CORE CURRICULUM
Content• disciplinary knowledge • discipline-related techniques
Skills
• oral and written communication • critical thinking• teamwork and collaboration• project management• leadership
Context
• relationship of science or technology with contemporary society
• individual and comparative explorations of cultural identity
QUESTIONS?
Top Related