1

2014-2019 ETS Proficiency Profile Results

Executive Summary Report Aug 2019

Confidential VUSC IR document. Please use caution in distribution.

2

The ETS Proficiency Profile (formerly MAPP) is used nation-wide to measure and document student proficiency in core academic

skill areas and to assess program effectiveness. VUSC uses the ETS Proficiency Profile to assess CORE/General Education Student

Learning Outcomes, specifically:

- CORE SLO 3: Communication skills: writing, and

- CORE SLO 4: Qualitative and quantitative critical thinking skills

In addition to assessment of core writing, quantitative reasoning, critical thinking, and reading skills, the Proficiency Profile

provides data on student accomplishments in broad areas of student baccalaureate preparation – humanities, social sciences, and

natural sciences. The test is administered to freshmen to objectively evaluate the academic preparedness of incoming students. The

addition of senior profile data, along with authentic coursework assignments, contributes to the assessment of educational

effectiveness of our university’s Core Curriculum and overall baccalaureate preparation.

Since 2009, Vanguard University has administered the ETS Proficiency Profile tests to the TUG entering freshmen in early

Fall and to TUG Seniors (more than 90 semester hours) during either the Fall or Spring semester of their senior year since 2010.

Students are administered the paper and pencil abbreviated (40-minute) test form in a proctored setting. Only results from students

who fully completed the test are used. In this report, total and subscale scores (Critical Thinking, Reading, Writing, Math, Humanities,

Social Sciences, and Natural Sciences) are displayed for each year for both freshmen and senior students. Vanguard freshmen and

senior scores were compared to the averages of their respective comparison groups in national Liberal Arts colleges which consist

of 59 institutions for freshmen and 71 institutions for seniors (please see the appendices for the school list). The comparison was

made using the data from a total of 282 Vanguard freshmen, 25,916 national freshmen, 329 Vanguard seniors, and 21,003 national

seniors who participated in the test from 2014 Fall to 2019 Spring.

Part I presents the comparisons of Vanguard freshmen, national freshmen, Vanguard seniors, and national seniors for total

scores and the seven subscale scores. Vanguard freshmen performed similarly to national freshmen on average; however, Vanguard

seniors performed significantly lower (at the .05 level) than national seniors in Total score, Critical Thinking, Math, and

Humanities (introductory literature, art, music, philosophy, etc.). Part II shows the longitudinal data of Vanguard freshmen for 2011-

2018 and seniors for 2014-2019. It should be noted that Vanguard seniors had performed better than their counterparts until 2014-

2015 but have significantly underperformed their counterparts for the past three years since 2016-17. Also, Vanguard 2018 entering

freshmen showed the lowest performance in total score (431.5) ever. Recent lower performance of Vanguard seniors may be due to

not only lower academic preparedness of the entering cohorts but also decreased academic effectiveness of Vanguard education.

3

Part I. Aggregate Data of Vanguard Freshmen (2014FA-2018FA) and Seniors (2014FA-2019SP) with National

Averages (2014FA-2019SP)

Note: A red box indicates a statistically significant difference at the 5% error rate.

s

436.35 434.7442.81 445.4

400

420

440

460

480

500

VU Freshmen'14-'18

NationalFreshmen

VU Seniors'14-'19

NationalSeniors

Total Score Means

108.91 109 110.49 111.7

100

105

110

115

120

125

130

VU Freshmen'14-'18

NationalFreshmen

VU Seniors'14-'19

NationalSeniors

Critical Thinking Means

115.7114.8

117.3 118.2

100

105

110

115

120

125

130

VU Freshmen'14-'18

NationalFreshmen

VU Seniors'14-'19

NationalSeniors

Reading Means

113.27112.1

115.16 114.5

100

105

110

115

120

125

130

VU Freshmen'14-'18

NationalFreshmen

VU Seniors'14-'19

NationalSeniors

Writing Means

4

111.31 111.5112.57 113.9

100

105

110

115

120

125

130

VU Freshmen'14-'18

NationalFreshmen

VU Seniors'14-'19

NationalSeniors

Math Means

112.24 112.2113.39 114.8

100

105

110

115

120

125

130

VU Freshmen'14-'18

NationalFreshmen

VU Seniors'14-'19

NationalSeniors

Humanities Means

110.94 110.7112.69 113.3

100

105

110

115

120

125

130

VU Freshmen'14-'18

NationalFreshmen

VU Seniors'14-'19

NationalSeniors

Social Sciences Means

113.35112.8

114.44 115.5

100

105

110

115

120

125

130

VU Freshmen'14-'18

NationalFreshmen

VU Seniors'14-'19

NationalSeniors

Natural Sciences Means

5

Part II. Longitudinal Analysis of Freshmen (2011-2018) and Seniors (2014-2019) Data with National Averages

Vanguard freshmen performed similarly to the national norm in

the past, except in 2015 (see Figure 1). VU seniors have performed

similarly to or slightly higher than national seniors in the past until 2014-

2015. However, their performance dropped in 2015-2016 and has been

significantly lower than the national average for the past three years

(see the black box in Figure 2).

The lower performance of Vanguard seniors in the past three

years may be due to the lower academic preparedness of the entering

cohorts during 2013 to 2015 (see the black box in Figure 1) or (and)

lower educational effectiveness of Vanguard. Therefore, average total

score change from freshmen to seniors were compared between the

entering cohorts during 2009 to 2012 and the entering cohorts during

2013 to 2015. Figure 3 shows that the average performance

improvement from freshman to senior for 2013-2015 cohorts (3.95) is

significantly lower than that for 2009-2012 cohorts (9.67) and is also

significantly lower than that for national average (8.4).

438.5 442.83

432.3 432.53

449.42

435.25 438.36

431.5434.7

2011 2012 2013 2014 2015 2016 2017 2018

Total Score (Freshmen)

VU Freshmen National Freshmen (2014-19)

451.46

443.32442.82 439.11

442.09

446.6

2014-15 2015-16 2016-17 2017-18 2018-19

Total Score (Seniors)

VU Senior National Senior (2014-19)

Figure 1

Figure 2

9.67

3.95

8.4

2009-2012cohorts

2013-2015cohorts

National 2010-2015 Cohorts

Avg Total Score Change from Freshman to Senior

Figure 3

6

Similar patterns were observed in the sub-skill scores as in the total score. 2015 incoming VU freshmen outperformed their

national counterparts significantly in all sub-skill areas, and VU seniors scored significantly lower than the national seniors for the

past three years in general, except for writing and mathematics. Senior writing scores are consistently at or slightly above the mean

score for national seniors. Note: A red box indicates a statistically significant difference at the 5% error rate.

109.28110.18

108.05 108.87

112.5

107.97

110.01

107.27109

2011 2012 2013 2014 2015 2016 2017 2018

Critical Thinking (Freshmen)

VU Freshmen National Freshmen (2014-19)

117.43

113.96 113.82

119.03115.81

115.88114.74

114.8

2012 2013 2014 2015 2016 2017 2018

Reading (Freshmen)

VU Freshmen National Freshmen (2014-19)

113.23

111.32110.07

108.83110.63

111.7

2014-15 2015-16 2016-17 2017-18 2018-19

Critical Thinking (Seniors)

VU Senior National Senior (2014-19)

119.85

117.58 117.39 116.28 116.94

118.2

2014-15 2015-16 2016-17 2017-18 2018-19

Reading (Seniors)

VU Senior National Senior (2014-19)

7

115.03

112.07 112.31

115.67 113.66113.83

111.93

112.1 112.1 112.1 112.1 112.1 112.1 112.1

2012 2013 2014 2015 2016 2017 2018

Writing (Freshmen)

VU Freshmen National Freshmen (2014-19)

112.99

111.09 110.44

114.69

110.83 111.2110.62111.5

2012 2013 2014 2015 2016 2017 2018

Mathematics (Freshmen)

VU Freshmen National Freshmen (2014-19)

116.9115.13 115.87

114.25 114.75

114.5 114.5

2014-15 2015-16 2016-17 2017-18 2018-19

Writing (Seniors)

VU Senior National Senior (2014-19)

113.9

112.05 112.37 112.47 112.45

113.9 113.9 113.9 113.9 113.9

2014-15 2015-16 2016-17 2017-18 2018-19

Mathematics (Seniors)

VU Senior National Senior (2014-19)

8

112.85 111.44110.47

115111.59 113.22

111.37

112.2 112.2

2012 2013 2014 2015 2016 2017 2018

Humanities (Freshmen)

VU Freshmen National Freshmen (2014-19)

112.62109.93 110.79

114.08110.95

110.99109.68

110.7 110.7 110.7 110.7 110.7 110.7 110.7

2012 2013 2014 2015 2016 2017 2018

Social Sciences (Freshmen)

VU Freshmen National Freshmen (2014-19)

116.18

113.29 112.67 111.6114.1

114.8

2014-15 2015-16 2016-17 2017-18 2018-19

Humanities (Seniors)

VU Senior National Senior (2014-19)

114.56112.61 112.82 111.8 112.57

113.3 113.3 113.3 113.3 113.3

2014-15 2015-16 2016-17 2017-18 2018-19

Social Sciences (Seniors)

VU Senior National Senior (2014-19)

9

114.46112.11 113.08

116.14112.75 114.32

111.98112.8

2012 2013 2014 2015 2016 2017 2018

Natural Sciences (Freshmen)

VU Freshmen National Freshmen (2014-19)

116.44 115.92

114.34 113.75 113.75

115.5

2014-15 2015-16 2016-17 2017-18 2018-19

Natural Sciences (Seniors)

VU Senior National Senior (2014-19)

10

Part III. Aggregate Mean, Median, and Proficiency Classifications on ETS Proficiency Profile with the National Norms

The following tables (#1 through #4) provide a comparison of Vanguard freshmen and seniors with national liberal arts college

students in terms of mean, median, and the proficiency classifications of Vanguard student learning outcomes: SLO #3

(Communication skills: writing) and SLO #4 (Qualitative and quantitative critical thinking skills). Table #1 shows that the mean and

median scores of VU Freshmen are similar to national norms in all subskill areas. Table #2 shows that the proficiency classification

patterns of incoming Vanguard freshmen are also very similar to those of their national counterparts in terms of the reading, writing,

and math, with the notable exception of higher proficiency rates in ‘Writing Level #1’ (55% vs. 44% national). Table #3 shows that

the means and medians of VU Seniors are similar to those of the national counterparts in general, except in Humanities, where VU

seniors showed lower mean and median (113.39 vs. 114.8 national; 112 vs. 115 national). As Table #4 shows, proficiency

classification patterns of Vanguard seniors are similar to those of their national counterparts. However, they showed lower levels of

proficiency than their national counterparts in both Math Level #2 (20% vs. 30% national) and #3 (4% vs. 8% national).

Table 1. Aggregate Means and Medians on ETS Proficiency Profile for Freshmen.

Possible Range

VU Freshmen 2014-2018 Mean N=282

National Liberal Arts colleges mean, 2014-2018, Freshmen N=25,916

VU Freshmen 2014-2018 50th Percentile N=282

National Liberal Arts colleges 50th Percentile N=25,916

Total Score 400 to 500 436.35 434.7 433 434

Skills Subscores:

Critical Thinking 100 to 130 108.91 109.0 107 108

Reading 100 to 130 115.7 114.8 115 115

11

Writing 100 to 130 113.27 112.1 113 112

Mathematics 100 to 130 111.31 111.5 110 111

Context-Based Subscores:

Humanities 100 to 130 112.24 112.2 111 111

Social Sciences 100 to 130 110.94 110.7 110 110

Natural Sciences

100 to 130 113.35 112.8 113 112

12

Table 2. Summary of Proficiency Classifications of Freshmen

Skill Dimension

Proficiency Classifications, Freshmen Fall 2014- Fall 2018 (N=282)

National Liberal Arts Colleges, Freshmen 2014-2018 (N= 25,916)

Proficient Marginal Not Proficient Proficient Marginal Not Proficient

Reading, Level 1 54% 21% 25% 47% 23% 30%

Reading, Level 2 22% 22% 56% 19% 19% 62%

Critical Thinking 0% 11% 89% 1% 10% 88%

Writing, Level 1 55%

31% 15% 44% 34% 21%

Writing, Level 2 10% 37% 53% 10% 28% 62%

Writing, Level 3 4% 19% 77% 4% 15% 81%

Math, Level 1 37% 34% 28% 41% 29% 30%

Math, Level 2 14% 23% 62% 17% 25% 59%

Math, Level 3 1% 8% 90% 3% 9% 88%

13

Table 3. Aggregate Means and Medians on ETS Proficiency Profile for Seniors.

Possible Range

VU Seniors 2014-2019 Mean N=329

National Liberal Arts colleges mean, 2014-2019, Seniors N=21,003

VU Seniors 2010-2018 50th Percentile N=411

National Liberal Arts colleges 50th Percentile N=21,003

Total Score 400 to 500 442.81 445.4 442 445

Skills Subscores:

Critical Thinking 100 to 130 110.49 111.7 110 110

Reading 100 to 130 117.3 118.2 118 120

Writing 100 to 130 115.16 114.5 115 115

Mathematics 100 to 130 112.57 113.9 113 113

Context-Based Subscores:

Humanities 100 to 130 113.39 114.8 112 115

Social Sciences

100 to 130 112.69 113.3 113 113

Natural Sciences

100 to 130 114.44 115.5 114 116

14

Table 4. Summary of Proficiency Classifications of Seniors

Skill Dimension

Proficiency Classifications, Seniors Fall 2014-2018 (N=329)

National Liberal Arts Colleges, Seniors 2014-2019 (N= 21,003)

Proficient Marginal Not Proficient Proficient Marginal Not Proficient

Reading, Level 1 63% 18% 19% 67% 17% 17%

Reading, Level 2 31% 21% 48% 38% 21% 41%

Critical Thinking 2% 16% 81% 5% 21% 73%

Writing, Level 1 67% 26% 8% 63% 25% 11%

Writing, Level 2 21% 40% 39% 22% 36% 42%

Writing, Level 3 12% 22% 66% 10% 25% 65%

Math, Level 1 49% 27% 24% 56% 23% 21%

Math, Level 2 20% 29% 51% 30% 25% 44%

Math, Level 3 4% 11% 85% 8% 17% 75%

Office IR, Aug 2019 15

Appendix 1. Comparison Group for Freshmen

Office IR, Aug 2019 16

Appendix 2. Comparison Group for Seniors