Outcomes Assessment and Assurance of Learning James M. Kraushaar, Ph.D., Associate Professor of MIS...

Outcomes Assessment Outcomes Assessment and Assurance of Learningand Assurance of Learning

Outcomes Assessment Outcomes Assessment and Assurance of Learningand Assurance of Learning

James M. KraushaarJames M. Kraushaar, Ph.D., Associate Professor , Ph.D., Associate Professor of MISof MIS

Nicole ChittendenNicole Chittenden, IT Manager, IT ManagerRobert RohrRobert Rohr, IT Professional Sr., IT Professional Sr.

The School of Business AdministrationThe School of Business Administrationat the University of Vermontat the University of Vermont

Track 4: Technology for Teaching and LearningTrack 4: Technology for Teaching and Learning

© © James Kraushaar et al., 2005.James Kraushaar et al., 2005. This work is the intellectual property of the authors. Permission This work is the intellectual property of the authors. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the authors. To disseminate otherwise or to republish requires written permission from permission of the authors. To disseminate otherwise or to republish requires written permission from the authors.the authors.

Presentation for NERCOMP 2006, Mar. 20-22, Presentation for NERCOMP 2006, Mar. 20-22, Worcester, MAWorcester, MA

© 2005 The School of Business Administration at the University of Vermont

AgendaAgenda ToolsTools ModelModel Data CollectionData Collection Sample - Data AnalysisSample - Data Analysis ConclusionConclusion


School of Business Assessment School of Business Assessment SystemsSystems

Satisfaction:Satisfaction: Educational Educational

Benchmarking (EBI)Benchmarking (EBI) On-line course On-line course

evaluationsevaluations Advisor surveyAdvisor survey AACSB accreditationAACSB accreditation UVM Office of UVM Office of

Institutional Studies Institutional Studies (OIC) year out (OIC) year out surveysurvey

Competency:Competency: Educational Testing Educational Testing

Services (ETS) Services (ETS) Major Field Test in Major Field Test in BusinessBusiness

GradesGrades Faculty Activity Faculty Activity

ReportsReports Curriculum:Curriculum:

Faculty Curriculum Faculty Curriculum Coverage InventoryCoverage Inventory


Curriculum: Product deliveryCurriculum: Product delivery

MissionMission

CurriculumCurriculum

SatisfactionSatisfaction CompetencyCompetency


Are we delivering the academic program Are we delivering the academic program we think we are?we think we are?

Curricular ProgramsCurricular ProgramsCourse Content/DeliveryCourse Content/Delivery

Extracurricular ProgramsExtracurricular ProgramsAcademic AdvisingAcademic Advising

Career & Professional Dev.Career & Professional Dev.Leadership OpportunitiesLeadership Opportunities

OUTCOMES ASSESSMENTOUTCOMES ASSESSMENT

COMPETENCYCOMPETENCYGradesGrades

ETS Major Field TestETS Major Field Test

SATISFACTIONSATISFACTIONCourse EvaluationsCourse Evaluations

EBI Exit SurveyEBI Exit SurveyAdvising SurveyAdvising Survey

GOAL:GOAL:The BestThe Best

Small School ofSmall School ofBusinessBusiness


The Predictable QuestionsThe Predictable Questions

Are our students satisfied with Are our students satisfied with education?education? Course evaluationsCourse evaluations Exit interviewsExit interviews BenchmarkingBenchmarking

Are our students prepared?Are our students prepared? GradesGrades Standardized testsStandardized tests Employer surveysEmployer surveys

MissionMission

CurriculumCurriculumSatisfactionSatisfaction CompetencyCompetency

MissionMission



Less Obvious Less Obvious AnswersAnswers

Curricular choices drive results:Curricular choices drive results: Where should emphasis be placed to Where should emphasis be placed to

address competency shortfalls?address competency shortfalls? Which choices are working? Which choices are working? What changes will have greatest effect?What changes will have greatest effect? Are key concepts reinforced?Are key concepts reinforced? Can curricular focus on mission critical Can curricular focus on mission critical

aims be identified?aims be identified?


The plan… gather The plan… gather Curricular Curricular Coverage Inventory Coverage Inventory

Evaluate aggregate curricular focusEvaluate aggregate curricular focus Quantify the total experience of Quantify the total experience of

programprogram Target specific areas for review Target specific areas for review

and/or adjustmentand/or adjustment MissionMission



Design for alignmentDesign for alignment

Align faculty survey of curricular Align faculty survey of curricular coveragecoverage with mission statementwith mission statement with student competencywith student competency

Can now compare/contrast subjective Can now compare/contrast subjective and objective quality measuresand objective quality measures Student satisfaction vs. competencyStudent satisfaction vs. competency


Curricular Coverage InventoryCurricular Coverage Inventory

Collected from facultyCollected from faculty Info across every courseInfo across every course Three areas:Three areas:

Mission drivenMission driven Competencies drivenCompetencies driven Learning Assessment methodsLearning Assessment methods

Demo Demo


Sample Analysis: Topics bigger Sample Analysis: Topics bigger than any one coursethan any one course

Mission Statement Excerpt:Mission Statement Excerpt:

… … prepares students for … dynamic prepares students for … dynamic globalglobal economy … economy … ethical, social, ethical, social, and environmentaland environmental responsibility… responsibility…


20 27 27

Social Responsibility 2.39 9 8 14 27 25

Environmental/Sustainability Concerns 2.19 5 4

21 28 12

Legal Environment 2.77 16 6 23 19 19

Ethical Environment 2.78 12 10

28 18 13

Global Awareness 2.88 6 23 20 23 11

Use of Information Technology 2.92 13 11

13 5 3

Acquire, Analyze and Interpret Data 4.02 36 27 10 6 4

Critical Thinking 4.16 45 17

9 2 3

Solving Qualitative Problems 4.10 38 28 10 1 6

Defining Qualitative Problems 4.24 42 27

12 21 10

Solving Quantitative Problems 3.41 37 4 9 22 11

Defining Quantitative Problems 3.37 32 8

11 15 27

Interpersonal Skills 2.40 10 7 13 29 24

Ability to work in Teams 2.66 11 19

19 17 8

Leadership Skills 2.10 3 8 13 29 30

Writing Skills 3.31 20 19

19 23 14

Presentation Skills 2.57 7 18 19 10 29

Sense of History 2.81 8 19

Section I - Knowledge & Skill InventoryMission Elements Avg 5 4 3 2 1

Results: Curricular focus on Results: Curricular focus on MissionMission

GlobalGlobal EthicalEthical EnvironEnviron

--mentalmental

SocialSocial


Assessment Tools UsedWritten Examination(s)Written Quiz(zes)Oral Examination(s)Oral Quiz(zes)Individual Presentation(s)Inividual Paper(s)/Project(s)Group Presentation(s)Group Paper(s)/Project(s)Case(s)Problem/Homework Set(s) 4.24

2.770.750.541.220.43

0.0 91.60.06 0.0 1.2 0.0 0.0 94.00.01 0.0 1.2 0.0

24.9 13.43.81 19.4 13.9 12.7 2.4 48.12.20 20.7 13.4 26.4

Section III - Assessment Tools UsedFreq T/F ShA Qnt Ess Oth

Sample Report: How are Sample Report: How are students assessed in an students assessed in an

average courseaverage course


RecapRecap

Talked about curriculum coverage Talked about curriculum coverage data gathering systemdata gathering system

Nicole will discuss external Nicole will discuss external satisfaction and competency satisfaction and competency measurement systemsmeasurement systems


Incorporating Selected Outside Incorporating Selected Outside ToolsTools

Student SatisfactionStudent Satisfaction EBIEBI – – ((http://http://www.webebi.com/wesswww.webebi.com/wess))

Offers “Complete Assessment Solutions”Offers “Complete Assessment Solutions” Focus on annual comparison with Carnegie Focus on annual comparison with Carnegie

class institutionsclass institutions Select 6 - comparisons have limited value Select 6 - comparisons have limited value

given inconsistent participation by peersgiven inconsistent participation by peers

Student CompetencyStudent Competency ETSETS – – ((http://http://www.ets.orgwww.ets.org))

Major field testMajor field test Measures learning outcomes and Measures learning outcomes and

benchmarks against other institutionsbenchmarks against other institutions Required for completion of capstone classRequired for completion of capstone class


EBI – Measure of Student EBI – Measure of Student SatisfactionSatisfaction

Incorporates measures of school Incorporates measures of school initiativesinitiatives

Advising, Instruction in Ethics, Career & Advising, Instruction in Ethics, Career & Professional DevelopmentProfessional Development

Summary format for sharing results Summary format for sharing results with faculty, staff, Board of Advisors, with faculty, staff, Board of Advisors, ProvostProvost

Priority Matrix – identifies items that are highly Priority Matrix – identifies items that are highly correlated with student satisfaction correlated with student satisfaction

Home grown reportsHome grown reports


EBI – Priority Matrix 2005EBI – Priority Matrix 2005


Example: Analyzing Example: Analyzing SatisfactionSatisfaction

0

5

10

15

20

25

30

35

40

# of questions

2001 2002 2003 2004 2005

Year

Per question, how did our students perform vs. All Schools

top 20%

top 50%

bottom 20%

0

5

10

15

20

25

30

35

40

45

# of questions

2004 2005

Year

Trends by Category vs. Carnegie Class

Good

Bad

Ugly


ETS - Student competencyETS - Student competencyWhat has it told us?What has it told us?

% Mean by Indicator

FA04-UVM mean % Correct

FA04-UVM % at or below

SP05-UVM mean % Correct

SP05-UVM % at or below

All B-Schools Mean % Correct

All B-Schools % at or below

1-Accounting 54.0 90% 51.6 90% 44.6 50%2-Economics 52.1 90% 45.8 70% 42.8 50%3-Management 59.7 65% 56.3 40% 57.1 45%4-Quant Bus Analy/MIS 67.8 95% 66.2 90% 56.6 50%5-Finance 43.6 85% 42.8 85% 36.4 50%6-Marketing 55.0 90% 51.1 75% 46.8 50%7-Legal & Social Envir 50.4 50% 48.2 35% 49.8 50%8-International Issues 47.0 65% 42.3 35% 44.3 50%

n= 118 65 80,044Mean 159.5 156.2 152

σ= 13.6 13.5 13.7

Significant Fall/Spring variation. Why?Significant Fall/Spring variation. Why?


Student competency Student competency differences across semesters?differences across semesters?

0

20

40

60

80

100

120

# of students

Fall 2004 Spring 2005

Transfer vs. Non-transfer

Non-transfer

Transfer

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

percent students

0.0-1.0 1.0-2.0 2.0-2.5 2.5-3.0 3.0-3.5 3.5-4.0

GPA Range

GPA distribution

Fall 2004

Spring 2005


Multiple measuresMultiple measures

Moved from individual measurements Moved from individual measurements to composite viewto composite view Neither satisfaction nor competency Neither satisfaction nor competency

tells the whole storytells the whole story Combine them for more complete Combine them for more complete

picturepicture


Curriculum Change: Student Curriculum Change: Student Satisfaction (EBI) vs. Satisfaction (EBI) vs.

Student Competency (ETS)Student Competency (ETS)

Human Human Resource Resource

ManagementManagementLowLow LowLow

AccountingAccounting LowLow HighHigh

Legal and Legal and Social Social

EnvironmentEnvironmentHighHigh LowLow



Case StudyCase Study

Legal environmentLegal environment Courts & legal systemsCourts & legal systems Constitution and businessConstitution and business Administrative lawAdministrative law

Regulatory EnvironmentRegulatory Environment Employment & labor law, Employment & labor law,

Antitrust law, Antitrust law, Consumer protectionConsumer protection Tort lawTort law CrimesCrimes Environmental and Environmental and

international lawinternational law

Business relationshipsBusiness relationships Contract & commercial Contract & commercial

law (including UCC)law (including UCC) Business organizationsBusiness organizations Law of agencyLaw of agency Intellectual propertyIntellectual property

Ethics & Social Ethics & Social responsibilityresponsibility

EthicsEthics Social responsibilitySocial responsibility

10% of ETS questions center around Legal & Social Environment 10% of ETS questions center around Legal & Social Environment Legal & Social Environment Content Includes: Legal & Social Environment Content Includes:


Curricular Coverage Inventory = Curricular Coverage Inventory = Limited Review of Business Limited Review of Business

RelationshipsRelationships

8 8 5Social responsibility 3.14 9 5

Ethics & social responsibility

Ethics 3.11 10 7 3 7 8

5 3 27

Intellectual property 1.54 1 3 1 4 26

Law of agency 1.37 0 0

1 3 31

Business organizations 1.06 0 0 0 2 33

Contracts & commercial law (incl. UCC) 1.14 0 0

3 1 26

Business relationships

Environmental & international law 1.71 3 2

0 2 32

Crimes 1.20 0 1 1 2 31

Tort law 1.14 0 1

2 1 27

Consumer protection 1.46 2 1 2 1 29

Antitrust law 1.66 3 2

Regulatory environment

Employment & labor law 2.29 5 7 1 2 20

2 0 28

Administrative law 1.71 0 5 2 6 22

Constitution & business 1.63 3 2

Legal and Social Environment

Legal environment

Courts & legal systems 2.23 3 8 1 5 185 4 3 2 1Avg


Curricular Coverage Inventory:Curricular Coverage Inventory:Faculty focus within courseFaculty focus within course

Section II: Skills/Topics Mentioned in Syllabus

132 - Legal and Political Environment of Business

Question Value

Legal and Social Environment

Legal environment

Courts & legal systems 3.67

Averyt, William 5.00

Meier, Frederic 2.00

Monte, Peter 4.00

Constitution & business 4.33



Monte, Peter 4.00

Administrative law 2.67



Monte, Peter 3.00

Business relationships

Contracts & commercial law (incl. UCC) 1.33



Monte, Peter 2.00

Business organizations 1.33



Monte, Peter 2.00

Law of agency 1.33



Monte, Peter 2.00

Intellectual property 2.33



Monte, Peter 2.00

Ethics & social responsibility

Ethics 3.33



Monte, Peter 4.00

Social responsibility 3.33



Monte, Peter 5.00(continued, next column…)(continued, next column…)


ConclusionConclusion Based on data (NOT just intuition)Based on data (NOT just intuition) Both satisfaction and competencyBoth satisfaction and competency Better information about delivery of Better information about delivery of

topics across entire programtopics across entire program Better target curricular changesBetter target curricular changes

MissionMission

CurriculumCurriculum



AcknowledgementsAcknowledgements

AACSB Accreditation and Training AACSB Accreditation and Training ProcessProcess

University of Alaska, FairbanksUniversity of Alaska, Fairbanks Western Ontario – Outcomes Western Ontario – Outcomes

Assessment ConferenceAssessment Conference Our staff and facultyOur staff and faculty


QuestionsQuestions


Contact informationContact information web: web: http://http://www.bsad.uvm.eduwww.bsad.uvm.edu/assess/assess James M. Kraushaar, Ph.D.,James M. Kraushaar, Ph.D.,

Associate Professor of MISAssociate Professor of MIS [email protected]@bsad.uvm.edu

Nicole Chittenden, IT ManagerNicole Chittenden, IT Manager [email protected]@uvm.edu

Robert Rohr, IT Professional Sr.Robert Rohr, IT Professional Sr. [email protected]@uvm.edu

Outcomes Assessment and Assurance of Learning James M. Kraushaar, Ph.D., Associate Professor of MIS...

Documents

Transcript of Outcomes Assessment and Assurance of Learning James M. Kraushaar, Ph.D., Associate Professor of MIS...