Pilot Study of the CAAP

Critical Thinking TestApril 27, 2005

Lanette RaymondResearch Associate, Suffolk County Community College

CAAP Test Description

32 item multiple choice testDesigned for use with college studentsAdministered within a single class meeting,Appeared relatively credible in an in-class

administration protocolProvided documentation of reliability and validity

across community college populations

CAAP Subscore Customization

CAAP Critical Thinking test contents closely match the SUNY CT learning objectives

Variance in the way the results are reported ACT developed a customized report for these sub-

scores showing normative comparisons against ACT national community college data.

ACT provided the student data files to SCCC, for further analysis of this data

Administration and Sample

Fall, 2004 Administered in-class to 154 SCCC students in 7 general

education courses Predominately white (77%) Traditional age (60% 20 years old or younger, 25% between

21 and 25 years old) 50% male, 50% female Mostly sophomore status (46%) Fulltime enrollment (85%)

Student Motivation

No motivational tactics were employedCAAP-CT instrument included an item that

addressed students' self-reported motivation levelsOne-third of students (n = 52) did not respond to the

motivation item5 students indicated that they “gave no effort” (n = 1)

or “gave little effort” (n = 4) to the assessment test.

Student Motivation

Lower motivation results in less optimal performance

Less motivated students’ scores are less reliable and less valid Reporting sample is based on data from the 97 students who

reported moderate to best effort The reliability coefficient (calculated with the data from the

original 154 tests) for objective 1 (26 items) is within acceptable range (alpha = .80)

Due to the small number of items (6 items) contributing to objective 2, its reliability coefficient is much lower (alpha = .49).

Tried my bestGave moderateeffort

Gave little effortGave no effortNo Response

Motivation

20

18

16

14

12

10

8

Me

an

+-

1 S

E C

T O

bj 1

# i

tem

s c

orr

ec

t

Tried my bestGave moderateeffort

Gave little effortGave no effortNo Response

Motivation

80

60

40

20

0

Me

an

+-

1 S

E D

ev

elo

p r

eas

on

ed

ag

rum

en

ts

Results

Confirmatory factor analysis substantiates the utility of the CAAP-CT test as a measure of 2 separate but related sets of critical thinking skills based on the 2 GEAR learning objectives

1

CT - 1

1

CT - 2

Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

Item 7

Item 9

Item 10

Item 12

Item 14

Item 15

Item 16

Item 17

Item 18

Item 19

Item 21

Item 22

Item 23

Item 25

Item 26

Item 28

Item 29

Item 30

Item 31

Item 32

Item 1

Item 2

Item 3

Item 4

Item 5

Item 6

2-factor model of Critical Thinking based on

the GEAR Objectives

Results

All of the items loaded well onto their respective factors, with item 1 being only slightly below 1.96 (at 1.81).

The model shows an excellent fit to the data (χ2(463) = 466, p = .46, CFI = .94), providing additional context validity to the assessment.

1 0.68 4.492 0.74 4.873 0.64 4.194 0.70 4.625 0.75 4.916 0.61 4.027 0.53 3.508 0.51 14.899 0.80 5.26

10 0.86 5.6411 0.70 20.6112 0.82 5.4313 0.70 20.6114 0.36 2.3515 0.48 3.1616 0.37 2.4417 0.77 5.0418 0.38 2.4819 0.71 4.6620 0.56 16.6021 0.62 4.1022 0.61 4.0223 0.51 3.3324 0.35 10.3125 0.44 2.8626 0.67 4.4027 0.58 16.9828 0.56 3.6829 0.39 2.5630 0.40 2.6131 0.50 3.2932 0.32 2.09

Learning Objective I Learning Objective 2CAAP Critical Thinking Test

ItemItem

DifficultyItem

WeightItem

DifficultyItem

Weight

** Data for item 20 should be listed under objective 2

Standards

Does not meet standard 59% or less

Approaches standard 60% - 69%

Meets standard 70% - 79%

Exceeds standard 80% or more

CT Learning Obj 1 - Groups

37 38.1 38.1 38.1

24 24.7 24.7 62.9

12 12.4 12.4 75.3

24 24.7 24.7 100.0

97 100.0 100.0

Exceeds standard

Meets standard

Approaches standard

Does not meet standard

Total

ValidFrequency Percent Valid Percent

CumulativePercent

CT Learning Obj 2 - Groups

36 37.1 37.1 37.1

19 19.6 19.6 56.7

9 9.3 9.3 66.0

33 34.0 34.0 100.0

97 100.0 100.0

Exceeds standard

Meets standard

Approaches standard

Does not meet standard

Total

ValidFrequency Percent Valid Percent

CumulativePercent

Standards