1

Student Learning Outcomes Assessment (SLOA) at the Program and Course Levels:Comprehensive Projects That Work Across All Disciplines

AFACCT – January 2008Carroll Community College

T. Bidle, C. Dove, M. Jozik, V. Stein, N. Thorpe

2

Challenges to Student Learning Outcomes Assessment

1. Learning Outcomes vs. Content Objectives

2. Program vs. Course level Outcomes3. Assessment across all sections and

instructors4. In-house vs. benchmarked assessment5. Data collection, analysis, storage6. Closing the loop to improve instruction

3

Levels of Student Learning Outcomes & Assessment

Program

Course

Institutional

Middle States Commission on Higher Education (2003), Student Learning Assessment: Options and Resources, Philadelphia, PA 19104.

4

Program Level SLOA

• Definition of an HCC program – “a sequence of study that results in student achievement of specific learning outcomes”

• Program areas include:- General education- Developmental education- Transfer education- Career education

5

AS in Chemistry Program

General Education 32

Program Requirements

Gen CHM I and II (8)

Organic CHM I and II 8

Calculus 1 4

Principles/ Physics I and II 10

Recommended Electives 10

Total Credits 64

6

Chemistry Program

• Does the two year chemistry transfer program prepare students for successful upper division work?

7

How well does the program transfer to upper division institutions?

HCC Towson Shippensburg Hood Shepherd UMCP

CHM 103 (4)General Chemistry I

CHEM 110 (4)Gen. Chem. I

CHM121 & 125 Chemical Bonding & Lab3

CHEM 101 (4)General Chemistry I

CHEM 207 & 207L (4) General Chemistry & Lab I

CHEM 143 (5)General and Analytical Chemistry6

CHM 104 (4)General Chemistry II

CHEM 111 (4)Gen. Chem. II

CHM122 & 126 Chemical Dynamics & Lab3

CHEM 102 (4)General Chemistry II

CHEM 209 & 209L (4)General Chemistry & Lab II

CHEM 153 (3)General and Inorganic Chemistry6

CHM 203 (4)Organic Chemistry I

CHEM 331 (5)Organic Chem. I1

CHM 221 & 225 (4) Mod. Org. Chem I & Lab

CHEM 209 (4)Organic Chemistry I

CHEM 315 & 315L (4)Organic Chemistry & Lab I5

CHEM 237 (4) Principles of Organic Chemistry I6

CHM 204 (4)Organic Chemistry II

CHEM 332 (5)1

Organic Chem. II1

CHM 222 & 226 (4) Mod. Org. Chem II & Lab

CHEM 210 (4)Organic Chemistry II

CHEM 316 &316L (4)Organic Chemistry & Lab II5

CHEM 247 (4)Principles of Organic Chemistry II6

MAT 203 (4)Calculus I

MATH 211 (3) Calculus for Applications orMATH 273 (4)Calculus I

MAT 211 (4)Calculus I

MATH 201 (4)Calculus I

MAT 207 (4)Calculus I

MATH 140 (4)Calculus I

Add MAT 205 (4)As a recommended elective (Calc II)

MAT 212 (4)Calculus II

MATH 202 (4)Calculus II

MAT 208 (4)Calculus II

MATH 141 (4)Calculus II

PHY 203 (5)Principles of Physics I

PHYS 211 (4)Gen. Physics I (non-calculus based)2

PHY 205 (4)Intermediate Physics I

PHYS 101 (4)4

General Physics IPHYS 201 & 201L (4)College Physics I orGen. Physics I and Lab

Course Transfer EquivalenciesHandout

Incomplete / see handout for complete chart

8

What are Program Outcomes?• Expectations for students when they leave the

classroom and are working in their careers or in subsequent coursework

• Results of accomplishing course learning outcomes which build on each other

• Involve the mastery, integration, and application of content and skills of many courses

• Must be measurable

• More difficult to assess than giving final exam

• “Career” programs may have a certification or licensure exam

9

How are Program Outcomes Assessed?

• Survey students completing all or most of the course sequence in a program

• Assess skills at the course level as the program is in progress

• Maintain portfolios of student work

• Monitor success and persistence of students in the course sequences within a program

• Follow-up with transfer institutions that articulate with many program completers

• Follow-up with employers

• Benchmark with nationally normed exams

10

Chemistry Program Outcomes1. Apply quantitative thinking process and reasoning

skills in upper division academic work

2. Communicate chemical concepts to varied audiences

3. Use mathematical tools essential to success in upper division chemistry courses

4. Design an appropriate response to a research problem

-- appropriate lab equipment-- experimental design (scientific method)

5. Solve problems collaboratively

6. Use software and hardware to analyze experimental data

11

Survey of Chemistry Graduates(chemistry program assessment)

• Survey instrument developed based on Program Outcomes (handout)

• Criteria for selection expanded to include all students who completed 200 level program requirements for degree even if they did not graduate

• 72 surveys were mailed in Fall 2006

• 14 surveys were returned and completed

12

4.1

3.7

4.2

3.9

4.2

3.4

0 1 2 3 4 5

Thinking/Reasoning

Communication

Mathematical Tools

Experimental Design

Collaborative TeamWork

Data Analysis

Chemistry Survey Results(chemistry program assessment)

13

CAAP Scientific Reasoning 2005 Chemistry Students

(Benchmarking)

65.2 66.4

84 86

0

20

40

60

80

100

CHM 101 CHM 104 CHM 204 PHY 202/204

Nat

iona

l Pe

rcen

tile

Mean

n=12 n=24 n=8 n=8

14

Pre-Allied Health “Program”

• Prerequisites from Math and Science Division– Anatomy and Physiology I & II– Math 101– Biology 099– Chemistry 101

15

Pre-Allied Health Program

• Do pre-allied health prerequisites prepare students for success in allied health career programs?

16

Evaluation of Pre-Allied Health Program

Success in BIO 099, A&P I and A&P IIFA 2000 SP 2007

72.9

59.5

45.7

79.2 75.9

0

10

20

30

40

50

60

70

80

90

ABC 099 ABC 103 ABC 104

% s

ucce

ssfu

l at

each

leve

l

With 099Without 099

17

Evaluation of Pre-Allied Health Program

35 34 31

238

127

48

335

207

107

0

50

100

150

200

250

300

350

400

ASN RAD LPN

Started in 099 Directly into 103 Admitted to Program

Students Admitted to Allied Health ProgramsFA 2000 SP 2007

18

Mathematics Courses Completed by A&P Students 2000-2005

Evaluation of Pre-Allied Health Program

40

20

40

51.6

15.1

46.8

58

90.9

0

10

20

30

40

50

60

70

80

90

100

MAT 098 MAT 099 MAT 100 MAT 101

% C

om

ple

ting E

ach

Math

Cours

e

Needed BIO 099 Passed P.T. without BIO 099

19

BIO 099 Course Redesign

• Revised outcomes to include – Reading skills– Problem solving– Math skills

• Increase credits from 2 to 3 hours

• Placement exam revised to reflect outcome changes

20

Pre-Allied Health Program Outcomes

• Successful completion of Anatomy and Physiology I and II

• Successful admission into Allied Health Program

21

HCC Science Course Projects:

Program

Course

Institutional

Middle States Commission on Higher Education (2003), Student Learning Assessment: Options and Resources, Philadelphia, PA 19104.

22

HCC Science Course Projects:

• General Chemistry I and II

• Intro. To College Chemistry

• Biology for Allied Health

• Anatomy & Physiology I and II

23

The Program:Course Matrix To Integrate Outcomes (handout)

Program Outcomes

Course #1

Outcomes

Course #2

Outcomes

Course #3

Outcomes

Course #4

Outcomes

1. Reasoning

skills

2. Communicate

concepts

3. Use math

tools

4. Experimental

design

5. Collaboration

6. Use software for data analysis

24

Assessment of Skills and Content(chemistry course assessment)

• Common Rubrics in Related Courses– Dimensional analysis problems (handout)

– Laboratory reports (handout)

– Chemical Equations (in progress)

• Common Final Exams

– Introductory Chemistry, General Chemistry I and II

25

Definitions

Active Students

= enrollees

Students registered at the

census date (including those that withdrew after the census date)

Completers Active students – withdrawals

Retention Rate

Completers

active students

Success RateStudents earning A, B, C

active students

Completer success rate

Students earning A, B, C

completers

26

Definitions, cont’d

PersistenceProgression through successive

courses in a course sequence

AssessmentMeasurement of student learning of expected learning outcomes

Benchmark Assessment

Aggregate data

from peer institutions or normed national tests

27

General Chemistry I (CHM 103) Retention and Success (Fa 05 Fa 07)

92.5

82.589.288.1

65.474.2

81.362.5

76.9

0

10

20

30

40

50

60

70

80

90

100

Retention Success Completer Success

% s

tud

ents

Spring students Fall students Summer studentsN=40 N=189 N=18

28

American Chemical Society (ACS) Standardized General Chemistry Exams

CHM 103 (Gen Chemistry, I)

35.2 36.2

56.4

43.4

37.642.6

38.241

0

10

20

30

40

50

60

n= 52 n=14

n=5

n=42 n=21n = 7

n = 53

29

CHM 103 (Gen. Chem. I) Fall 05 – Fall 07

A

B

C

D

F

Course Grade as a function of ACS Exam Percentiles: F05 --> F07

0

1

2

3

4

5

0 20 40 60 80 100

ACS Exam Percentiles

Co

urs

e G

rad

e

30

89.1

64.171.9

92.3

76.983.3

0

20

40

60

80

100

Retention Success Completer Success

% s

tud

ents

Spring students Fall studentsN=64 N=26

General Chemistry II (CHM 104) Retention and Success (SP 06 Fa 07)

31

30.927.6

44.5

32.3

39.1

0

5

10

15

20

25

30

35

40

45

50

Spring 2006 Fall 2006 Spring 2007 Fall 2007 ACSBenchmark

American Chemical Society (ACS) Standardized General Chemistry Exam

CHM 104 (Gen Chemistry, II)

n=32 n=8

n=18n =16

32

CHM 104 (Gen. Chem. II) Spring 06 – Fall 07

A

B

C

D

F

Course Grade as a function of ACS Exam Percentiles

0

1

2

3

4

5

0 10 20 30 40 50 60 70 80 90 100

ACS Exam percentiles

Co

urs

e G

rad

e

33

Persistence from General Chemistry I (CHM 103) II (CHM 104)

71

17 15

131

87

64

0

20

40

60

80

100

120

140

Successful 103 Attempted 104 Successful 104

No

. of

Stu

de

nts

66.4%

23.9% 48.9%21.1%

CHM 103 in Spring or SummerCHM 104 completed in Fall

CHM 103 in FallCHM 104 completed the following Spring

34

CHM 101: Introductory College Chemistry

• Assessment started Fall 2007

• Rubric for grading dimensional analysis problems

• Common final exam

35

CHM 101 (Intro. Chem.) Common Final Exam

CHM 101 Fall 2007

0

1

2

3

4

5

0 5 10 15 20 25 30 35 40 45 50

Common Exam Score

Co

urs

e G

rad

e

A

B

C

D

F

36

Anatomy and Physiology I and II

• High Enrollment Course

• Prerequisites– Placement Test – BIO 099 for those who fail the placement test– H.S. Chemistry or Introductory College

Chemistry

37

A&P Project Design Overview

1. Based on the HAPS* learning outcomes

2. Develop a common cumulative final exam to assess student learning at the end of A&P I

3. Administer exam to all students completing A&P I and analyze student and section data

4. Administer national HAPS exam to all A&P II completers

5. Use data to modify course and increase student success

6. Generate Database of students’ exam grades, course grades, success in other prerequisite courses

* Human Anatomy & Physiology Society (HAPS)

38

Using the Student Database

• Identify gaps in student learning

• Modify course components

• Science Learning Center supplements

• Laboratory activities

• Examples of using the data

39

HAPS Exam Scores vs. HCC 1st Semester Exam

40

Student Groups Average + Standard Deviation

Average of 2-yr and 4-yr colleges (n=2430)

53.46 + 11.88

4-yr colleges (n= 964) 57.53 + 12.10

2-yr colleges (n=1466) 50.80 + 10.45

HCC students (n=294) 54.01 + 12.5*

HAPS 2-Semester Comprehensive ExamComparison with Other Colleges

(benchmarking)

*HCC Retention Rate = 71%

41

The Science Team

• Cindy Dove• Veronica Stein• Nancy Thorpe• Judith Peisen• Terri Bidle• Robin Thomas

• Maria Jozik• Richard Campbell• Bernard Murphy• David Karstaedt• James Stemmle• Elaine Ashby

Contact information: Bidlet@hagerstowncc.eduThorpen@hagerstowncc.edu 301-790-2800 extension 268