“Davenport’s Excellence System Assessment Pilot: Measurable Competencies for Improved Teaching and

Learning, Student Certification, and Workplace Success”

Wynalda TLI PresentationAugust, 2014

Dr. Irene Bembenista: Vice Provost for Assessment & Graduate StudiesDr. Wayne Sneath: Program Director Experiential Learning

Mike Carey: Department Chair, Marketing

Audience Question Cards:

Please include e-mail address& questions for follow up

Session Objectives

To explain the purpose and design of the Excellence System Assessment Pilot

To explain the proposal for certification of students in the Excellence System

To present data on the assessment of all nine Excellence System competencies in courses from two years of the pilot

To generate discussion on the implications of this ongoing assessment on improved teaching and learning and workplace success

The DU Excellence System History

The Mission of Davenport University is “to prepare individuals and organizations to excel in the knowledge-driven environment of the 21st Century.”

To that end, the Davenport University Excellence System was created in partnership with employers who helped to define and reinforce the necessity of these competencies.

The Excellence System consists of nine professional competencies necessary for graduates to engage in life long-learning and succeed in their chosen profession.

The Provost’s Charge

Committee Formed in January, 2012

“The committee shall use the DU Excellence System to develop a methodology and a process that will validate students graduating from DU shall have achieved the skills necessary for each of the DU principles of excellence”

Committee Proposal for Excellence System Certification

Summary:

1. Assessment of proficiency in all 9 Excellence System competencies in key Foundations of Excellence courses & provision of feedback via the development of an electronic portfolio system

2. Assessment of proficiency in all 9 Excellence System outcomes in mid- level and capstone-level courses in each program via the portfolio system

3. Determination of minimum cumulative GPA in all assessed courses

What is Proposed as Certification in the Excellence System?

Minimum score at the Mastery Level in each Excellence System Competency via analysis of assignments from capstone and/or

other program courses (score TBD via pilot)

+

Minimum cumulative GPA in all assessed courses in the Foundations + Program Level (GPA TBD)

= Certification

Year I: Goals of the Pilot

1. Determine a baseline of performance in all 9 Excellence System

competencies by examining required assessment assignments from three Foundations of Excellence courses:

COMM 120: Presentation TechniquesRAA: Final Informative or Persuasive SpeechCompetencies Assessed:• Ethical Reasoning and Action• Information & Tech Proficiency• Professional Communication

ENGL 311: Professional WritingRAA: Group ProjectCompetencies Assessed:• Critical & Creative Thinking• Analysis & Problem Solving• Written Communication• Leadership & Teamwork

SOSC 201: Diversity in SocietyRAA: Experiential Learning PaperCompetencies Assessed:• Global & Intercultural Competence • Civic and Social Responsibility

Year 1: Goals of the Pilot

2. Determine performance in all 9 Excellence System competencies at mid-level and capstone-level courses in programs:

Marketing was the pilot program with assessment in:

MKTG 213: Advertising Foundations MKTG 441: Strategic Marketing (Capstone)

3. Create common rubrics for all 9 Excellence System competencies utilizing cross-college faculty teams

4. Report data to the Provost with recommendations by

May 1, 2013, including a recommendation for development of an electronic portfolio system

through which chosen assignments can be deposited and assessed by course-level faculty

Some Important Questions to be Answered via the Pilot

1. What are the appropriate Foundations of Excellence, Foundations of Business, Technology, or Health, and/or Program courses in which to assess the competencies?

2. Is a review of course RAAs via a common rubric the best means of assessment? Should we recommend nationally normed exams or create our own?

3. How can we effectively train all instructors in these courses to perform consistent assessment of competencies and report this data?

4. What are the best means to assess competencies not easily evaluated by a team via an artifact (technology proficiency, teamwork, professional communication?)

5. What role should ongoing assessment teams play for the Excellence System? Are they cross-college or only by Program? Should they be a “quality check” on course-level instructor assessment scores?

Process for Collection & Assessment of RAAs

• Faculty in chosen sections require students to post RAAs or RAAs are downloaded from existing course shells

• A sample of all RAAs submitted were downloaded & identifying information was removed

• 4 person, cross-college faculty assessment teams were formed and trained in inter-rater reliability and sample scoring; teams read samples to calibrate before scoring independently

• Teams assessed 6 of 9 competencies

• Course faculty assessed “Observable” competencies of Information & Technology Proficiency, Leadership & Teamwork, and Professional Communication

E-Portfolio System Vetting Committee

Purposes:

• Research options for an e-portfolio system which would assist with ongoing Excellence System assessment

• Recommend a system with proposed costs and implementation timeline to the Provost as part of the pilot

• Portfolio system should serve multiple purposes:

ES Assessment & Student Formative Feedback

Faculty Rank & Promotion Dossiers Student Career Development Prior Learning Assessment Portfolios

Year One Pilot Results for:

Global & Intercultural Competence

Irene

Global & Intercultural Competence

Graduates understand that working and succeeding in an inclusive, multicultural, and international world involves complex issues present in diverse environments

1. Evaluates and demonstrates culturally-appropriate behavior in diverse domestic and global contexts

2. Interprets issues and events from diverse domestic and global perspectives

3. Demonstrates an understanding of the dynamics of power, privilege and inequality and appropriate interventions/responses to bias in diverse domestic and global contexts

4. Demonstrates awareness of personal culture and biases and how these influence interactions/relationships in diverse domestic and global contexts

5. Demonstrates an ability to interact successfully with people from diverse domestic and global perspectives

6. Understands and examines differing value systems to determine appropriate courses of action and decision making in the diverse work environment and the domestic and global community

-- Introductory Level—SOSC 201’s Experiential Learning Paper

-- Reinforced Level—MKTG 213’s Group Advertising Plan

-- Mastery Level—MKTG 441’s Group Strategic Marketing Plan

RAAs Mapped to GIC

-- Only 32% of RAAs required a 3rd reader, thus 68% agreement

-- AACU’s standard—reliable at 70% or greater when two readers are within one level of scoring agreement on a rubric

-- GIC was one of the highest reliabilities of the 9 Excellence System competencies assessed in Year One

Inter-Rater Reliability for GIC

-- Mean score was 6.7 of 18 rubric points (6 outcomes x 3)

-- 94% of students were proficient in this competency in SOSC 201

-- 27% of these scored above the Introductory Level

Outcome 1 Outcome 2 Outcome 3 Outcome 4 Outcome 5 Outcome 60.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40 1.281.34

1.05

1.30

0.98

0.78

0.540.58

0.62 0.620.70

0.64

Global & Intercultural Competency Results by Rubric Outcome In SOSC 201

MeanStandard Deviation

Scale 0-3

2012-13 Results: GIC in SOSC 201

-- 94% proficiency at Introductory Level does NOT demonstrate the global aspect of this competency

-- GIC Rubric at Introductory Level assesses either domestic OR global competence

-- GIC Rubric at the Mastery Level assesses both domestic AND global competence

94%

6%

% of Students Proficient in Global & Intercultural Competence at Introductory Level in SOSC

201

Proficient Not Proficient

N=110

Implication I

-- Students scored lowest in GIC Competency Outcome 6:

“Understands and examines differing value systems”

Implication II

Aggregate Results ofAll 9 Excellence System

Competencies

Percentage of Pilot Students Proficient vs. Not-Proficient in all Excellence System Competencies:

Year One Pilot Data

Global & In

tercu

ltural

Competency:

SOSC

201 (N=1

10)

Civic &

Socia

l Resp

onsibilit

y: SO

SC 201 (N

=110)

Ethica

l Reaso

ning & Acti

on: COMM 120 (N

= 60)

Critical

& Cre

ative

Thinkin

g: EN

G: 311 (N

= 160)

Analysis

& Pro

blem Solvi

ng: EN

GL 311 (N

= 160)

Lead

ership &

Team

work: EN

GL 311 (N

=234)

Informati

on &Te

chnology

Proficie

ncy: COMM 120 (N

=227)

Writt

en Communication: E

NGL 311 (N

= 160)

Professi

onal Communica

tion: COMM 120 (N

= 285)

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100% 94%

80%

37%

63%

78% 77%81%

49%

91%

6%

20%

63%

37%

22% 23%19%

51%

9%

Excellence System Competency in Foundations of Excellence Courses: COMM 120, SOSC 201, & ENGL 311

% Proficient

% Not Proficient

Total Unique Students Assessed:

N=1076

Mike

Year One Pilot:

Marketing Program Data

Percentage of MKTG441 Students Proficient v. Not Proficient by Excellence System Competency

Critical

& Cre

ative

Thinkin

g N= 2

4

Analysis

& Pro

blem Solvi

ng N= 2

4

Lead

ership &

Team

work N

= 41

Informati

on & Te

chnology

Proficie

ncy N

=41

Writt

en Communication N

= 24

Professi

onal Communica

tion N= 4

10%

10%

20%

30%

40%

50%

60%

70%

80%

90%

29%

13%

44% 46%

63%

22%

71%

87%

56% 54%

37%

78%

% Proficient% Not Proficient

Total Unique Students As-sessed: N=65

Year One Pilot: MKTG Program

Marketing Program Excellence System Certification Analysis for MKTG 441

0%20%40%60%80%

100%

29%

94%

26%

Comparison of Mastery Competency Proficiency, 2.3 or Higher Program GPA, and Combined Mastery Proficiency & Minimum Program GPA

%

N=65

Correlation Analysis ofMKTG Program GPA to RAA Assessment Performance

2 2.5 3 3.5 4 4.50

10

20

30

40

50

60

Series1Linear (Series1)

MKTG Major GPA

Com

pete

ncy

Scor

e To

tal

Irene

Year Two Pilot Planning

&

Results

Conclusion to Year One Pilot

• Average rubric scores compiled and report data was provided to the Provost with recommendations in June, 2013

• Provost and Deans requested a second year of comparative data

• Work began to plan Year Two of the pilot: 2013-14

Planning Year Two of the Excellence System Pilot

• Score a sample of papers from Fall 2013 courses and compare assessment teams’ scores to those already completed by course faculty to determine reliability of instructor scores as a means to do ongoing assessment

• Continue to refine RAA assignments and Excellence System rubrics

• Use data to determine baseline proficiency scores for certification at the Program Level

• Inform curriculum mapping and construction of new RAA assignments

• All PROGRAMS in the colleges charged to identify courses to assess RAAs for all levels of Excellence System proficiency—I, R, and M

2013-14 Pilot: Phase IIPilot Courses: 12 courses, 80+ sections, 1100+ students in

Fall 2013 & Winter 2014

CONTINUED ASSESSMENT OF THE FOLLOWING COURSES:COMM 120: Professional CommunicationENGL 311: Leadership & Teamwork, Written CommunicationSOSC 201: Global & Intercultural Competence, Civic & Social

Responsibility

ADDITIONAL COURSES ADDED FOR ASSESSMENT IN YEAR TWO:HLTH 101: Critical & Creative Thinking, Analysis & Problem SolvingFRSM 100: Critical & Creative Thinking, Analysis & Problem SolvingBUSN 210: Ethical Reasoning & ActionCISP 112: Information & Technology Proficiency

NEW Marketing Program Courses: MKTG 310, 412, 421, 430, 441

Year Two Pilot Results:General

Total Unique Students Assessed:

2014: N=654

Total Unique Students Assessed:

2013: N=1076

Global & In

tercu

ltural

Competence

--I

Civic &

Socia

l Resp

onsibilit

y--I

Ethica

l Reaso

ning & Acti

on (2013=I)

(2014=R

)

Critical

& Cre

ative

Thinkin

g (2013=R

) (2014=I)

Analysis

& Pro

blem Solvi

ng (2013=R

) (2014=I)

Lead

ership &

Team

work--R

Informati

on &Te

chnology

Proficie

ncy*--I

Writt

en Communication--R

Professi

onal Communica

tion--I0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

18.0

20.0

6.7

5.0

3.5

10.3

7.4

9.9

19.0

9.7

13.2

5.44.6 4.9

5.9 6.3

9.911.2 10.3

12.4

2013 Mean Score

2014 Mean Score 2013 v. 2014 Comparison:

Mean Scores by Excellence System Competency

*The number of ITP Rubric Outcomes was reduced from 8 to 5, CSR from 5 to 4, and APS from 6 to 5 in 2014 reflecting different range scores.

I =4-7

2014 I =3-6 2013

I=4-7

2014 R=8-12

2013I=4-7

2014 I=4-72013 R=8-

12

2014 I=4-72013 R=8-

10

R=8-10

2014 I=4-72013 I=6-11

R=10-14

I=4-7

Excellence System Competency Means 2013 v. 2014

Excellence System Proficiency Comparison 2013 v. 2014

Global & In

tercu

ltural

Competence

Civic &

Socia

l Resp

onsibilit

y

Ethica

l Reaso

ning & Acti

on

Critical

& Cre

ative

Thinkin

g

Analysis

& Pro

blem Solvi

ng

Lead

ership &

Team

work

Informati

on &Te

chnology

Proficie

ncy

Writt

en Communication

Professi

onal Communica

tion0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100% 94%

80%

63% 63%

78% 77%81%

49%

91%

6%

20%

37% 37%

22% 23%19%

51%

9%

58%

82%

14%

84% 83%

72%

97%

48%

96%

42%

18%

86%

16% 17%

28%

3%

52%

4%

2013 % Proficient2013 % Not Proficient2014 % Proficient2014 % Not Proficient

2013 v. 2014 Comparison:Percentage of Students Proficient

vs. Not-Proficient by Excellence System Competency

Marketing Program: Year Two Pilot Results

Mike

2014 Marketing Program Courses & Sample Sizes

Total Unique Students Assessed:

2014: N=162

GIC (MKTG 421: N=26)CSR (MKTG 310: N=0)ERA (MKTG 310: N=17)CCT (MKTG 430 N=40)APS (MKTG 412: N=46)LT (MKTG 441: N=33)ITP (MKTG 412: N=46)WC (MKTG 421: N=26)PC (MKTG 441: N=33)

Marketing Program Competency Means 2013 v. 2014

Global & In

tercu

ltural

Competence

--M

Civic &

Socia

l Resp

onsibilit

y*--M

Ethica

l Reaso

ning & Acti

on--M

Critical

& Cre

ative

Thinkin

g--M

Analysis

& Pro

blem Solvi

ng*-M

Lead

ership &

Team

work--M

Informati

on &Te

chnology

Proficie

ncy*M

Writt

en Communication--M

Professi

onal Communica

tion--M

0.0

2.0

4.0

6.0

8.0

10.0

12.0

N/A N/A N/A

8.7 8.8

7.0

11.0

12.0

8.0

5.5

N/A

4.4

7.2

4.9

9.610.0

10.410.0

2013 Mean Score

2014 Mean Score

2013 v. 2014 Comparison:Marketing Student Mean Scores by

Excellence System Competency

*The number of ITP Rubric Outcomes was reduced from 8 to 5, CSR from 5 to 4, and APS from 6 to 5 in 2014 reflect -ing different proficiency range scores.

M =13-18

2014 M =9-12 2013 M=11-15

M=13-18

M=13-18

2014 M=13-18

2013 M=11-15

M=11-15

2014 M=11-152013 M=17-24 M=15-21

M=13-18

Marketing Program Proficiency Comparison 2013 v. 2014

Global & In

tercu

ltural

Competence

Civic &

Socia

l Resp

onsibilit

y

Ethica

l Reaso

ning & Acti

on

Critical

& Cre

ative

Thinkin

g

Analysis

& Pro

blem Solvi

ng

Lead

ership &

Team

work

Informati

on &Te

chnology

Proficie

ncy

Writt

en Communication

Professi

onal Communica

tion0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

N/A N/A N/A

29%

13%

44% 46%

63%

22%

0% 0%

13%

0%

45%39%

15%

30%

100% 100%

87%

100%

55%61%

85%

70%71%

87%

56% 54%

37%

78%

2013 % Proficient2013 % Not Proficient2014 % Proficient2014 % Not Proficient

2013 v. 2014 Comparison:Percentage of Marketing Students Proficient

vs. Not-Proficient by Excellence System Competency

MKTG Program Lessons Learned

Need to Determine the Right:• Course aligned with the Excellence System competency• Assignment (RAA) with the Excellence System competency

Team v. Individual assignments• Rubric language • Training for faculty on scoring using rubrics

Variance in Team v. Faculty scoring in Year Two

Remember:• Not a lot of additional time in reading assignments: most RAAs are read

once for a course grade, but Excellence System rubric scores do NOT impact students’ course grades

• Not all 9 Excellence System competencies are assessed in each course; assessment is spread out across Foundations and Program-Level courses

MKTG Program Next Steps

Davenport University Excellence System Assessment Mapping  

Program: Marketing Program 2014-15

Excellence System Competencies INTRODUCED REINFORCED MASTEREDRAA

ASSIGNMENT

Global & Intercultural Competence SOSC201 ECON200 MKTG 421 Country Notebook

Civic & Social Responsibility SOSC201 BUSN225 MKTG 310 Ethics/CSR Paper

Ethical Reasoning & Action ENVS125 BUSN210 MKTG 310 Ethics/CSR Paper

Critical & Creative Thinking ENGL109 LEGL210 MKTG 430 New Product Development Plan

Analysis & Problem Solving ENGL110 STAT220 MKTG 412 Primary Research Project

Leadership & Teamwork ENGL110 ENGL311 MKTG 441 Strategic Marketing Plan

Information & Technology Proficiency COMM120 ACCT202 MKTG 412 Primary Research Project

Written Communication ENGL109 ENGL311 MKTG 421 Country Notebook

Professional Communication COMM120 ENGL311 MKTG 441 Strategic Plan Presentation

Current Work &

Overall Project Next Steps

Wayne

Curriculum Mapping, CRIRs, and ES Data Collection

Summer 2014 Work

1. Excellence System competencies mapped at I,R, and M levels for all academic programs across the colleges

2. CRIR revisions to identity RAAs to be assessed against ES common rubrics

3. Development of an Excellence System assessment website:

www.davenport.edu/academics/assessment/excellence-system

4. Development of Google Drive score sheet system for 9 Excellence System

competencies to be used by course faculty to report data

Project Next Steps

• Determination on final criteria for Excellence System Certification

• Begin assessment in all sections of all identified Excellence Systems assessment courses across all academic programs in Fall 2014

• E-Portfolio pilot with College of Urban Education cohort (Fall 2014)

• Broader portfolio pilot sample (Winter 2015)

Audience Questions and Comments