1

Student Engagement at WSU: What can we Learn From Five years of NSSE Information

Office of Institutional Research

2

NSSE: National Study of Student Engagement

• A national survey of first year students and seniors in 4-year colleges and universities.

• It was designed to measure “engagement” or how students perceive their involvement in the total academic experience

• Engagement is defined by George Kuh as:– What students do -- time and energy devoted to

educationally purposeful activities both in and out of the classroom

– What institutions do -- using effective educational practices to induce students to do the right things

3

NSSE: National Study of Student Engagement

• First full data collection in 2000

• 1300 different 4-year schools have participated

• Averages 600+ schools for the last five years

• Survey of a random sample of freshmen and seniors conducted in spring each year

4

Questions

• Students are asked to respond to questions about their experiences at the university including academics, university services, and various out of classroom activities.

• 85 items with varying response options such as– How often have you… 1 Never to 4 very often– How would you rate… 1 very unfavorable to 7 highly

favorable– How would you evaluate…1 poor to 4 excellent– How much emphasis… 1 very little to 4 very much

5

Seven Principles of Good Practice in Undergraduate Education*

Student-faculty contact Active learning Prompt feedback Time on task High expectations Experiences with

diversity Cooperation among

students

* Chickering & Gamson, 1987; Pascarella & Terenzini, 2005

6

NSSE Indicators ofEffective Educational Practice

Level of Academic Challenge

Active & Collaborative

Learning

Enriching Educational Experiences

Student – Faculty

Interaction

Supportive Campus

Environment

The WSU ExperienceSeveral administrations

Pooling data on several administrations

Providing college level results with sufficient sample size

8

Background Information

NSSE has now been administered at WSU six times (2002, 2004, 2005, 2007, 2008 and 2009)

Although part of two recent KBOR Performance Agreements, use by the academic colleges or other areas has been minimal.

The Foundation of Excellence Project last year did use the results for the first year students.

Academic colleges have expressed concern that the sample size in a single year is insufficient for their use.

9

Thus …

The University Assessment asked us to see if we could legitimately combine several years’ data and then consider differences among the academic colleges.

10

Purposes of Current Analyses

Reviewed data over several years to identify any trends or changes.

Using the data from several survey administrations, disaggregate the WSU NSSE data to individual colleges by major with sample sizes large enough to encourage confidence in results.

Provide information about groups of items that are conceptually and empirically confirmed as related (i.e. benchmarks and scalelets).

Allow colleges and other campus groups the opportunity to use the NSSE as another component of their total outcome assessments.

Limited to seniors since most logical at the college level.

1. Validated item consistency

2. Determined differences, if any, for WSU data over administrations

Step 1: Merging data over multiple survey administrations

12

Item consistency over time:Review using NSSE materials

• Reviewed the detailed report produced by the NSSE staff for item changes over each administration since 2002.– Sufficient number of changes before 2004 that deletion

of 2002 data seemed essential.– One item changed in 2004 but this item not used in our

analysis

• The wording of the survey items and the response options on the surveys are consistent over the years since 2004.

13

Item consistency over time: Review using WSU Data

Compared each item by class and year using Chi-Square or Analysis of Variance.

Used complete responses (e.g. 1-4) and pooled responses (e.g. often/very often) as another means to validate any differences in an item over time.

Removed 2004 and repeated the analyses. No additional meaningful differences noted.

With this large number of cases, occurrence of significant differences by chance alone are highly probable.

Reviewed items for meaningful differences if statistically significant.

14

Item consistency over time: Summary

Differences in items over administrations were minimal.

Fluctuations in scores rather than progressive pattern of changed was noted for some items.

Responses from seniors were remarkably stable NSSE developers indicate that unless deliberate

and systemic university-wide interventions were made, differences should be minimal.

15

Two interesting items

E-mail as a method for communication with faculty Increased from 49% in 2004 to 72% In 2009 as reported by

freshmen. No differences for seniors

University encouraging students to vote quite a bit or very much. Increases were noted in the two presidential election years i.e.

2005 and 2009. In 2005 and 2009, freshmen reported encouragement at 30 %

and 37% respectively but only 18-19% in other years. In 2005 and 2009, seniors reported encouragement at 24% and

27% respectively but only 18-19% in other years.

1. Sample versus total seniors

2. Characteristics of Students for sample versus total seniors

Demographics

17

Comparison of Sample and Total Enrollment by College (N=2173)

College/Division Sample University*

Business 20% 20%

Education 14% 13%

Engineering 13% 11%

Fine Arts 7% 7%

Health Professions 11% 12%

LAS - Humanities 5% 4%

LAS - Math/Science 8% 7%

LAS - Social Science 21% 20%

LAS - Other 2% 2%

*Fall 2008

18

Demographics of Seniors

Sample WSU Comment

>=24 yrs 53% 56% No significant difference among the colleges.

Women 64% 56% Clear gender differences. Women predominate in HP(90%), ED(80%) & LAS SS(79%). Men predominate in EN(77%) & LAS Math/SC(56%).

Minority 17% 18% Range from 10% and 12% in Education and Fine Arts to 21% and 22% in LAS Social Sciences and LAS Math/Science respectively. Differences are not statistically different (p>.001)

International 6% 6% No significant difference among colleges

Full time 73% 63% All colleges are above 70% full time.

Working(Pay) 83% NA Range form 71% n Education to 89% in Business. All colleges above 70%.

Transfer Hours

88% 87% No difference among colleges

A few Individual Items

20

Rating for Academic Advising and Satisfaction Entire Educational Experience

Interrelated items

22

Benchmarks represent Areas of Effective Practice

Level of Academic Challenge

Active & Collaborative

Learning

Enriching Educational Experiences

Student – Faculty

Interaction

Supportive Campus

Environment

23

Scalelets versus Benchmarks

• Scalelets include a small number of items related to specific concept that have empirically validated. (Pike,2002,2006)

• 14 scalelets with 12 as subsections of the benchmarks

• Advantages over benchmarks

– More specific and meaningful concepts– More congruence between items and concept

• Benchmark and scalelet scores are converted to 0-100 scale.

• The mean for the items in each scalelet and benchmark for each student were calculated if he/she answered 3/5 or 60% of items

Benchmark: Academic Challenge

Scalelets: Course Challenge and Writing

Academic Challenge

25

Level of Academic Challenge: Items

1. Preparing for class (studying, reading, writing, doing homework or lab work, etc. related to academic program)

2. Number of assigned textbooks, books, or book-length packs of course readings3. Number of written papers or reports of 20 pages or more4. Number of written papers or reports of between 5 and 19 pages 5. Number of written papers or reports of fewer than 5 pages6. Coursework emphasizes: Analysis of the basic elements of an idea, experience

or theory 7. Coursework emphasizes: Synthesis and organizing of ideas, information, or

experiences into new, more complex interpretations and relationships8. Coursework emphasizes: Making of judgments about the value of information,

arguments, or methods9. Coursework emphasizes: Applying theories or concepts to practical problems or

in new situations 10. Working harder than you thought you could to meet an instructor's standards or

expectations11. Campus environment emphasizes: Spending significant amount of time studying

and on academic work.

26

Level of Academic Challenge: Mean Scores

27

Course challenge

How often…Worked harder than you thought you could to meet an instructor's standards or expectations WORKHARD

How often…Come to class without completing readings or assignments CLUNPREP(reverse)

To what extent have your examinations during the current school year challenged you to do your best work. EXAMS

How many hours per week spent…Preparing for class (studying, reading, writing, doing homework or lab work, analyzing data, rehearsing, and other academic activities) ACADPR01

To what extent does your institution emphasize spending significant amounts of time studying and on academic work

ENVSCHOL

28

Writing

How often have you...Prepared two or more drafts of a paper or assignment before turning it in

REWROPAP

How often have you...Worked on a paper or project that required integrating ideas or information from various sources

INTEGRAT

In current school year…Number of written papers or reports of 20 pages or more WRITEMOR

In current school year…Number of written papers or reports between 5 and 19 pages WRITEMID

In current school year…Number of written papers or reports of fewer than 5 pages WRITESML

29

Higher-Order thinking Skills

Memorizing facts, ideas, or methods from your courses and readings so you can repeat them in pretty much the same form

MEMORIZE(reverse)

Analyzing the basic elements of an idea, experience, or theory, such as examining a particular case or situation in depth and considering its components

ANALYZE

Synthesizing and organizing ideas, information, or experiences into new, more complex interpretations and relationships

SYNTHESZ

Making judgments about the value of information, arguments, or methods, such as examining how others gathered and interpreted data and assessing the soundness of their conclusions

EVALUATE

Applying theories or concepts to practical problems or in new situations

APPLYING

30

Academic Challenge: Related Scalelets

Collaborative Learning

Active Learning

Active and Collaborative Learning

32

Active & Collaborative Learning

1. Asked questions in class or contributed to class discussions

2. Made a class presentation3. Worked with other students on projects during class4. Worked with classmates outside of class to prepare

class assignments5. Tutored or taught other students (paid or voluntary)6. Participated in a community-based project (e.g.,

service learning) as part of a regular course7. Discussed ideas from your readings or classes with

others outside of class (students, family members, co-workers, etc.)

33

Active Learning

How often have you...Asked questions in class or contributed to class discussions

CLQUEST

How often have you..Made a class presentation CLPRESEN

How often have you..Participated in a community-based project (e.g. service learning) as part of a regular course

COMMPROJ

34

Collaborative Learning

How often have you...Worked with other students on projects during class

CLASSGRP

How often have you...Worked with classmates outside of class

OCCGRP

How often have you...Tutored or taught other students (paid or voluntary)

TUTOR

How often have you...Discussed ideas from your readings or classes with others outside of class (students, family members, co-workers, etc.)

OOCIDEAS

35

Active and Collaborative Learning

Course Interaction

Out of Classroom Interaction

Faculty/Staff – Student Interaction

37

Course Interaction

How often have you...Discussed grades or assignments with an instructor

FACGRADE

How often have you..Discussed ideas from your readings or classes with faculty members outside of class

FACIDEAS

How often have you..Received prompt written or oral feedback from faculty on your academic performance

FACFEED

38

Out-of-Class Interaction

How often have you...Talked about career plans with a faculty member or advisor

FACPLANS

How often have you...Worked with faculty members on activities other than coursework (committees, orientation, student life activities, etc.)

FACOTHER

How often have you...Work on a research project with a faculty member outside of course or program requirements

RES

39

Course Interaction and Outside Class Interaction

Varied Experiences

Information Technology

Diversity

Enriching Educational Experiences

41

Varied Experiences(Part I)

Have you or plan to do ... A practicum, internship, field experience, co-op experience, or clinical assignment

INTERN04

Have you, or plan to.. Community service or volunteer work

VOLNTR04

Have you, or plan to.. Participate in a learning community or some other formal program where groups of students take two or more classes together

LRNCOM04

Have you, or plan to.. Foreign language coursework

FORLNG04

Have you or do you plan to.. Study abroad

STDABR04

42

Varied Experiences(Part II)

Have you, or plan to.. Independent study or self-designed major

INDSTD04

Have you, or plan to.. Culminating senior experience (capstone course, senior project or thesis, comprehensive exam, etc.)

SNRX04

How many hours per week...Participating in co-curricular activities (organizations, campus publications, student government, fraternity or sorority, intercollegiate or intramural sports, etc.)

COCURR01

 

To what extent does your institution emphasize… Attending campus events and activities (special speakers, cultural performances, athletic events, etc.)

ENVEVENT

43

Technology

How often have...Used an electronic medium (listserv, chat group, Internet, instant messaging, etc.) to discuss or complete an assignment

ITACADEM

How often have.. Used e-mail to communicate with an instructor

EMAIL

to what extent does your institution emphasize...Using computers in academic work

ENVCOMPT

44

Diversity

How often have you...Had serious conversations with students of a different race or ethnicity than your own

DIVRSTUD

How often have you...Had serious conversations with students who are very different from you in terms of their religious beliefs, political opinions, or personal values

DIFFSTU2

To what extent has your institution encouraged contact among students from different economic, social, and racial or ethnic backgrounds

ENVDIVRS

45

Varied Experiences, Technology, Diversity

Support for Student Success

Interpersonal Environment

Supportive Campus Environment

47

Support for Student Success

To what extent does your institution emphasize… Providing the support you need to help you succeed academically

ENVSUPRT

To what extent does your institution emphasize...Helping you cope with your non-academic responsibilities (work, family, etc.)

ENVNACAD

To what extent does your institution emphasize...Providing the support you need to thrive socially

ENVSOCAL

48

Interpersonal Environment

Relationships with other students ENVSTU

Relationships with faculty members ENVFAC

Relationships with administrative personnel and offices

ENVADM

49

Support for Student Success and Supportive Interpersonal Environment

Practical Skills

General Education

Outcome Measures

51

Gains in Practical Skills

To what extent does your institution emphasize...Using computing and information technology

GNCMPTS

To what extent does your institution emphasize...Analyzing quantitative problems

GNQUANT

To what extent does your institution emphasize...Acquiring job or work-related knowledge and skills

GNWORK

52

Gains in General Education

To what extent does your institution emphasize...Writing clearly and effectively

GNWRITE

To what extent does your institution emphasize...Speaking clearly and effectively

GNSPEAK

To what extent does your institution emphasize...Thinking critically and analytically

GNANALY

To what extent does your institution emphasize...Acquiring a broad general education

GNGENLED

53

Practical Skills and General Education

54

Thank You