Post on 04-Jun-2018
NSSE: More Than Your Average Lake Monster
John Ryan Director, Office of
Institutional Research The University of
Vermont 1/25/13
Presentation Overview
§ An Introduction: NSSE & Student Engagement
§ NSSE Uses and Select Results at UVM
§ Questions & Discussion
§ Contact Information
What is Student Engagement?
§ What students do -- time and energy devoted to educationally purposeful activities
§ What institutions do -- using effective educational practices to induce students to do the right things
§ Educationally effective institutions channel student energy toward the right activities
Time on task (Tyler, 1930s)
Quality of effort (Pace, 1960-70s)
Student involvement (Astin, 1984)
Social, academic integration (Tinto, 1987, 1993)
Good practices in undergraduate education (Chickering & Gamson, 1987)
College impact (Pascarella, 1985)
Student engagement (Kuh, 1991, 2005)
Foundations of Student Engagement
Seven Principles of Good Practice in Undergraduate Education
(Chickering & Gamson, 1987; Pascarella & Terenzini, 2005)
ü Student-faculty contact ü Active learning ü Prompt feedback ü Time on task ü High expectations ü Experiences with diversity ü Cooperation among
students
How College Affects Students (Pascarella & Terenzini, 2005)
Because individual effort and involvement are the critical determinants of impact, institutions should focus on the ways they can shape their academic, interpersonal, and extracurricular offerings to encourage student engagement. – p. 62
Student Success in College: Creating Conditions that Matter
(Kuh, Kinzie, Schuh, Whitt, and Associates, 2005)
Illustrates actual institutional policies, programs, and practices that promote student success. § Comprehensive report of in-depth study
of successful educational practices across institutional types
§ Practical guidance on how to implement effective practice in a variety of contexts
§ Co-sponsored by leading higher education groups (CFAT, Pew Forum on Undergraduate Learning)
NSSE Survey Content
Student Behaviors in College
Institutional Actions And Requirements
Student Reactions to College
Student Background
Information
Student Learning & Development
NSSE Indicators of Effective Educational Practice
Level of Academic Challenge
Active & Collaborative
Learning
Enriching Educational Experiences
Student – Faculty Interaction
Supportive Campus
Environment
Survey Administration § Administered to random
samples of first-year & senior students across U.S.
§ Administration occurs in the spring academic term
§ Paper & Web-based survey
§ Flexible to accommodate consortium questions
§ Multiple follow-ups to increase response rates
How Valid is Self-Reported Data? Research Confirms Self-Reported Data Validity
When Following Conditions Exist… § Requested information is known to respondents
§ Questions are clear and unambiguous
§ Respondents take questions seriously and thoughtfully
§ Answering does not threaten, embarrass, or violate privacy or compel a socially desirable response
The NSSE survey instrument was designed to meet these conditions…but
there is ongoing debate
For more details, see: NSSE Psychometric Portfolio at http://nsse.iub.edu/html/NSSE_Psychometric_Portfolio.cfm
NSSE 2011 Institution Response Rates
§ UVM’s response rate = 33%
§ Slightly more female, ALANA similar compared to
UVM student population; weighted for comparison
reports; participated 2005, 2008, 2011…2014 next
Average Institutional Response Rates
§ 23% for Carnegie Class comparison institutions § 28% for NSSE 2011 Peer Group institutions
§ 26% for Writing Consortium institutions
NSSE 2011 Select Results
§ Many strategies for analyzing the results
§ Comparisons with other universities/groups
§ “Change” over time at UVM (and relative to other schools)
§ First year v. senior respondents
§ Differences among colleges within UVM
§ Compare similar majors across universities
§ Predictive modeling for outcomes (learning, development,
satisfaction, retention, academic performance)
§ Cohort tracking (2008 first year respondents and as seniors
in 2011)
NSSE 2011
UVM Results § Thinking about your overall experience at this
institution, how would you rate the quality of relationships with faculty and administrative personnel and offices (FY)?
§ What percentage of our first year students would say “quite a bit” or “very much” about the extent to which UVM provides the support they need to thrive socially?
Survey says?
Answer:
55% in 2008 and 48% in 2011 § Thinking about your overall experience at this
institution, to what extent does the college provide the support you need to thrive socially (FY)?
Selected Peer Comparison
with UVM Results § In thinking about your undergraduate program as a whole,
including your major, have you done a culminating senior experience (e.g., senior comprehensive exam, capstone course, thesis or project)?
NSSE 2011 UVM Findings
What percentage of our students (in comparison to selected peers) participated in community service or volunteer work?
Class UVM Carnegie Peers
First-Year 29% 39%
Seniors 64% 58%
Survey Says?
§ What percentage of our FY and SR students reported typically spending more than 15 hours per week studying/preparing for class?
NSSE 2011 UVM Findings
What percentage of UVM students spent more than 15 hours per week studying/preparing for class?
Class UVM Carnegie Peers
First-Year 55% 41%
Seniors 42% 42%
Using NSSE at UVM: Student Success and Satisfaction
"SENIORS" 2008 NSSE+Engagement+FactorsImpact+on+Learning++Factor Impact+on+Personal+Development+Factor Quality+Educational+Experience+Rating Likelihood+you+would+return+if+starting+over
Academic(ChallengeActive(Collaborative(LearningStudent(Faculty(InteractionEnriching(Educational(ExperiencesSupportive(Campus(EnvironmentSAT(MathSAT(VerbalFemaleEthnicity( HispanicRankMajor(Category BusinessTransferCampus(residenceMother's(educationFather's(education
2008 NSSE+Engagement+Factors"FIRST+YEARS" Impact+on+Learning++Factor Impact+on+Personal+Development+Factor Quality+Educational+Experience+Rating Likelihood+you+would+return+if+starting+overAcademic(ChallengeActive(Collaborative(LearningStudent(Faculty(InteractionEnriching(Educational(ExperiencesSupportive(Campus(EnvironmentSAT(Critical(ReadingSAT(Critical(WritingSAT(MathFemaleEthnicity(RankMajor(Category Professional,+Social+Science,+OtherTransferCampus(residenceMother's(education Attended+college+or+completed+degreeFather's(education Attended+but+did+not+complete+college Attended+but+did+not+complete+college
Student Success and Satisfaction "SENIORS" 2008 UVM.Engagement.Factors
Impact.on.Learning..Factor Impact.on.Personal.Development.Factor Quality.Educational.Experience.Rating Likelihood.you.would.return.if.starting.overInclusion)and)SupportActive)and)Enriching)LearningHigher8Order)Critical)ThinkingOpenness)to)New)IdeasSAT)MathSAT)VerbalFemaleEthnicity) HispanicRankMajor)Category Arts.and.Humanities,.Business,.ProfessionalTransferCampus)residenceMother's)education Attended.but.did.not.complete.collegeFather's)education Completed.associate.degree Completed.a.bachelor's.degree
2008 UVM.Engagement.Factors"FIRST.YEARS" Impact.on.Learning..Factor Impact.on.Personal.Development.Factor Quality.Educational.Experience.Rating Likelihood.you.would.return.if.starting.overInclusion)and)SupportActive)and)Enriching)LearningHigher8Order)Critical)ThinkingOpenness)to)New)IdeasSAT)Critical)ReadingSAT)Critical)WritingSAT)MathFemaleEthnicity) Caucasian,.HispanicRankMajor)Category Business,.Professional Business,.Professional,.Social.Science,.Other Arts.and.Humanities,.Business,.ProfessionalTransferCampus)residenceMother's)education Attended.college.or.completed.degree Attended.but.did.not.complete.collegeFather's)education Completed.a.bachelor's.degree
Student Persistence
Likelihood of Persistence Fall 2011-Fall 2012 ì ë Grade-Point Supportive
Average Campus Environment
Student Persistence ...and the predictors for the predictors… GPA Supportive Environment -SAT Score -Level of Academic -Active Learning Challenge -Female -Student-Faculty
Interaction -Not transfer student -Fewer hrs./wk. working off campus
Best Practice Benchmarking
Class Mean a Sig bEffect Size c Mean a Sig b
Effect Size c Mean a Sig b
Effect Size c
First-Year 63.0 .00 61.5 * .07 63.0 .00Senior 58.9 * -.10 57.4 -.02 59.1 ** -.11
Note: Each box and whiskers chart plots the 5th (bottom of lower bar), 25th (bottom of box), 50th (middle line), 75th (top of box), and 95th (top of upper bar) percentile scores. The dot shows the benchmark mean. See page 2 for an illustration. See pages 10 and 11 for percentile values.
Supportive Campus Environment (SCE) I temsStudents perform better and are more satisfied at colleges that are committed to their success and cultivate positive working and social relations among different groups on campus.
Mean a
62.957.1
a Weighted by gender and enrollment status (and by institution size for comparison groups). b * p<.05 ** p<.01 ***p<.001 (2-tailed). c Mean difference divided by the pooled standard deviation.
Distributions of Student Benchmark Scores
NSSE 2011 Benchmark ComparisonsUniversity of Vermont
Supportive Campus Environment (SCE)
Mean Comparisons University of Vermont compared with:
UVMWriting
Consortium Carnegie Class NSSE 2011 Peer Group
0
25
50
75
100
UVM WritingConsortium
Carnegie Class NSSE 2011 PeerGroup
First-Year
0
25
50
75
100
UVM WritingConsortium
Carnegie Class NSSE 2011 PeerGroup
Senior
� Campus environment provides the support you need to help you succeed academically � Campus environment helps you cope with your non-academic responsibilities (work, family, etc.) � Campus environment provides the support you need to thrive socially � Quality of relationships with other students � Quality of relationships with faculty members � Quality of relationships with administrative personnel and offices
7
Class Mean a Sig bEffect Size c Mean a Sig b
Effect Size c Mean a Sig b
Effect Size c
First-Year 53.8 * .09 53.3 *** .12 54.5 .03Senior 57.1 .02 56.7 .05 57.5 -.01
NSSE 2011 Benchmark ComparisonsUniversity of Vermont
Level of Academic Challenge (LAC)
Mean Comparisons University of Vermont compared with:
UVMWriting
Consortium Carnegie Class NSSE 2011 Peer GroupMean a
54.957.4
a Weighted by gender and enrollment status (and by institution size for comparison groups). b * p<.05 ** p<.01 ***p<.001 (2-tailed). c Mean difference divided by the pooled standard deviation.
Distributions of Student Benchmark Scores
Note: Each box and whiskers chart plots the 5th (bottom of lower bar), 25th (bottom of box), 50th (middle line), 75th (top of box), and 95th (top of upper bar) percentile scores. The dot shows the benchmark mean. See page 2 for an illustration. See pages 10 and 11 for percentile values.
Level of Academic Challenge (LAC) I temsChallenging intellectual and creative work is central to student learning and collegiate quality. Colleges and universities promote high levels of student achievement by emphasizing the importance of academic effort and setting high expectations for student performance.
0
25
50
75
100
UVM WritingConsortium
Carnegie Class NSSE 2011 PeerGroup
First-Year
0
25
50
75
100
UVM WritingConsortium
Carnegie Class NSSE 2011 PeerGroup
Senior
� Hours spent preparing for class (studying, reading, writing, doing homework or lab work, etc. related to academic program) � Number of assigned textbooks, books, or book-length packs of course readings � Number of written papers or reports of 20 pages or more, between 5 and 19 pages, and fewer than 5 pages � Coursework emphasizes: Analysis of the basic elements of an idea, experience or theory � Coursework emphasizes: Synthesis and organizing of ideas, information, or experiences into new, more complex interpretations and relationships � Coursework emphasizes: Making of judgments about the value of information, arguments, or methods � Coursework emphasizes: Applying theories or concepts to practical problems or in new situations � Working harder than you thought you could to meet an instructor's standards or expectations � Campus environment emphasizes: Spending significant amount of time studying and on academic work
3
Other examples of NSSE at UVM
§ “Strategic Value” § College Key Performance
Indicators § Board of Trustees
Dashboard Indicators § College Portrait/VSA § Student learning outcomes
(writing, diversity, …) § Accreditation evidence
Examples from Other Universities
§ NSSE results are being used across all sectors and types of institutions.
§ Discovering and sharing ways student engagement results are being used is one of NSSE’s most important activities.
§ The following slides illustrate how NSSE data can inform educational policy and practice and provide examples of how specific institutions have used their NSSE results in productive ways.
Areas of Effective
Educational Practice
Areas for Institutional Improvement
Internal Campus Uses § Gauge status of campus
priorities § Examine changes in
student engagement between first and senior years
§ Assess campus progress over time
§ Encourage dialogue about good practice
§ Link with other data to test hypotheses, evaluate programs
§ Improve curricula, instruction, services
Institutional Improvement
Learning Communities 1ST Year
and Senior Experience
Academic Affairs
Learning Assessment
Faculty Development
Academic Advising
Peer Comparison
Student Affairs
Institutional Research
Enrollment Management
External Campus Uses § Assess status vis-à-vis
peers, competitors
§ Identify, develop, market distinctive competencies
§ Encourage collaboration in consortia (e.g., statewide NSSE conference)
§ Provide evidence of accountability for good processes (while awaiting improvement in outcomes)
Public Accountability
Fund Raising
Governing Boards
Prospective Students
Alumni
State Policy
Makers Performance Indicators
Focus on Right Things
Accrediting Bodies
Media
Parents
Level of Academic Challenge UNIVERISTY OF MICHIGAN § Finding: An item of discussion at
a meeting of the Provost’s Faculty Advisory Committee raised the question of students’ writing abilities and how much writing students have to do as U of M undergraduates.
§ Action: NSSE and FSSE results pertaining to academic challenge, i.e. those that focus on the amount of effort involved in class preparation and how much the environment emphasizes studying and academic work, were prepared and shared with faculty.
Active and Collaborative Learning WASHINGTON STATE UNIVERSITY § Finding: Results indicated the
campus was not meeting student expectations for collaborative learning, student-faculty interaction, and enriching educational experiences.
§ Action: Freshman Focus learning communities were created to provide all incoming freshmen the opportunity to engage in an extensive living-learning community system.
Student – Faculty Interaction CALIFORNIA STATE UNIVERSITY -
FRESNO
§ Finding: NSSE results showed that student-faculty interaction was lower than expected.
§ Action: Student success task force identified ways to improve student success. Participated in Building Engagement and Attainment for Minority Students (BEAMS) program to develop Mentoring Institute. Now 200+ faculty members, staff and student mentors have been trained.
Enriching Educational Experiences
SIMONS ROCK COLLEGE OF BARD
§ Finding: NSSE results for out-of-class engagement were not as strong as results on pure academic measures.
§ Action: Student affairs staff members used NSSE results to guide discussions and develop new student activities and community engagement efforts. Have added a new professional position in Student Activities.
Supportive Campus Environment TOWSON UNIVERSITY
§ Finding: Wished to examine and enhance experience of female students.
§ Action: Women’s Center staff developed a narrative about the collegiate experience of women on their campus, strengthened programs that offer leadership opportunities and self-empowerment content and process.
Sharing NSSE Results Campus Wide CLAYTON STATE UNIVERSITY:
Discusses NSSE results at faculty counsel, presidential retreats, student success forums, and in various standing committees.
§ UNIVERSITY OF COLORADO AT COLORADO SPRINGS: Institutional Research Office publishes a series of one- to two-page research briefs on their NSSE Web site, for faculty and staff members to access. Each brief is dedicated to a specific aspect of NSSE, such as “Diversity” or the “Academic Experience.”
Improving Civic Engagement
JAMES MADISON UNIVERSITY § Finding: First-year student
involvement in service-learning programs was lower than desired.
§ Action: Increased the number and quality of class presentations about service-learning; increased the number of contacts with new faculty members regarding the importance of emphasizing service-learning.
Improving General Education WORCESTER POLYTECHNIC
UNIVERSITY § Finding: General education
courses provide foundation essential to numerous learning outcomes. A faculty-appointed committee defined 5 objectives to enhance core education.
§ Action: Have begun developing a new first-year general education curriculum featuring interdisciplinary, inquiry-based seminars, better integration of the disciplines, and broader, more engaging introductions to major areas of study.