Researchers Beware: Researchers Beware: Comparing FSSE with NSSE Comparing FSSE with NSSE

Can Be MessyCan Be Messy

Ed Rugg, DirectorEd Rugg, DirectorCenter for Institutional EffectivenessCenter for Institutional Effectiveness

Kennesaw State UniversityKennesaw State UniversityPresented at the AIR Forum 2005Presented at the AIR Forum 2005

San Diego, California San Diego, California

National Surveys ofNational Surveys of Student Engagement Student Engagement

Student Perspectives (NSSE)Student Perspectives (NSSE) The College Student Report 2004The College Student Report 2004

100 items for response 100 items for response

Faculty Perspectives (FSSE)Faculty Perspectives (FSSE)Faculty Survey of Student Engagement 2004Faculty Survey of Student Engagement 2004

112 items for response112 items for response

Suggested ComparisonsSuggested Comparisons

The 2004 Institutional Report Supported Making The 2004 Institutional Report Supported Making Comparisons of Faculty and Student Responses Comparisons of Faculty and Student Responses on on 5757 Supposedly Similar FSSE and NSSE Items Supposedly Similar FSSE and NSSE Items

FSSE & NSSE: FSSE & NSSE: Kin, But Not TwinsKin, But Not Twins

FSSE is an AppleFSSE is an Apple

NSSE is an NSSE is an Orange Orange

Extra Care is Extra Care is Needed when Needed when Comparing Comparing Apples with Apples with Oranges Oranges

Direct Comparisons of NSSE & Direct Comparisons of NSSE & FSSE Responses Are MessyFSSE Responses Are Messy

““Comparable items” for the two surveys are Comparable items” for the two surveys are rarely worded identically or similarly. rarely worded identically or similarly.

The context or focus for responses is often The context or focus for responses is often not the same across the two surveys.not the same across the two surveys.

Response categories for “comparable Response categories for “comparable items” are often very different for the two items” are often very different for the two surveys.surveys.

Substantive Differences Confound the Results of Most Comparisons

Few Well-Matched PairsFew Well-Matched Pairs

Only 10 of the Only 10 of the 57 “comparable 57 “comparable items” from items” from NSSE and FSSE NSSE and FSSE were nearly were nearly identical in identical in wording—a well-wording—a well-matched pairmatched pair

Well-Matched Pairs in 2004Well-Matched Pairs in 2004

CategoryCategory FSSE ItemsFSSE Items NSSE ItemsNSSE Items

Quality of Student 2, 3, 4 8 a-cQuality of Student 2, 3, 4 8 a-c

RelationshipsRelationships

Institutional 5 a-e, g-h 10 a-gInstitutional 5 a-e, g-h 10 a-g

EnvironmentEnvironment

Informative Direct Comparisons Informative Direct Comparisons for Well-Matched Itemsfor Well-Matched Items

When responding When responding to equivalent to equivalent items in the same items in the same context, faculty context, faculty and students and students agreed more than agreed more than disagreed, with a disagreed, with a few notable few notable exceptions.exceptions.

SeeSee “Well-Matched “Well-Matched Comparisons of Comparisons of NSSE & FSSE NSSE & FSSE Handout” Handout” on on institutional institutional environment and environment and quality of student quality of student relationshipsrelationships

Different Response ContextsDifferent Response Contexts& Focuses& Focuses

NSSE respondents NSSE respondents focus on the first focus on the first year or senior year year or senior year experience as a experience as a wholewhole

FSSE respondents FSSE respondents focus on a selected focus on a selected lower or upper lower or upper division course division course experience experience

Interpret Contextual Differences Interpret Contextual Differences with Cautionwith Caution

For example, when more students than For example, when more students than faculty report that students often have faculty report that students often have conversations with other students of a conversations with other students of a different race or nationality (FSSE 13f different race or nationality (FSSE 13f vs. NSSE 1u), remember that the faculty vs. NSSE 1u), remember that the faculty are referencing in-class experiences are referencing in-class experiences only, while students are reflecting on only, while students are reflecting on their out-of-class as well as in-class their out-of-class as well as in-class experiences as a whole. experiences as a whole.

Different Response Categories Different Response Categories Limit ComparabilityLimit Comparability

For 20 of the 57 For 20 of the 57 “comparable “comparable items,” faculty items,” faculty used different used different response response categories than categories than students usedstudents used

Example of a Messy Item Example of a Messy Item Comparison Involving Different Comparison Involving Different

Response CategoriesResponse Categories

Why compare an instructor’s Why compare an instructor’s rating of the rating of the importanceimportance of study abroad… of study abroad…

with a student’s with a student’s reported plan for completingreported plan for completing a a study abroad experience ?study abroad experience ?

What would any percentage difference meanWhat would any percentage difference mean ? ?

Enriching Educational Experiences(FSSE 1f compared to NSSE 7f)

Example of a Messy Matched Pair Example of a Messy Matched Pair in Wording, Context, & Responsein Wording, Context, & Response

Why compare the faculty’s report of Why compare the faculty’s report of the the percentage of students who percentage of students who frequently askedfrequently asked questions in one selected coursequestions in one selected course……

with the student’s report of with the student’s report of how frequentlyhow frequently he/she he/she asked questions in class during the current asked questions in class during the current school year school year ??

What would any percentage difference mean?What would any percentage difference mean?

Academic & Intellectual Experiences(FSSE 12a compared to NSSE 1a)

Consider Convergent AnalysesConsider Convergent Analyses

When direct When direct comparisons of comparisons of FSSE with NSSE FSSE with NSSE items are messy, items are messy, consider parallel consider parallel descriptive analyses descriptive analyses and inspect the and inspect the results for results for convergence.convergence.

SeeSee “Nuggets from “Nuggets from NSSE & FSSE –NSSE & FSSE –Handouts 1-4”Handouts 1-4” on on student learning student learning outcomes-- extracted outcomes-- extracted from convergent from convergent analyses of NSSE 11 analyses of NSSE 11 & 2 and FSSE 21 & 20& 2 and FSSE 21 & 20

Where Do We Go From Here?Where Do We Go From Here?

Proceed with Proceed with caution caution

Wear bootsWear boots

Get to the Get to the high groundhigh ground

An Opportunity to Improve FSSEAn Opportunity to Improve FSSE

FSSE has great untapped FSSE has great untapped potential to capture valuable potential to capture valuable informed judgments of informed judgments of educational experts (the faculty) educational experts (the faculty) in ways that would enable cross-in ways that would enable cross-validation of the self-reported validation of the self-reported engagement of students in NSSE.engagement of students in NSSE.

Another Key OpportunityAnother Key Opportunity

NSSE and FSSE NSSE and FSSE could be more could be more centrally involved centrally involved in the national in the national conversations on conversations on documenting documenting achievement of achievement of student learning student learning outcomes. outcomes.

Your Turn for Questions & Your Turn for Questions & CommentsComments

PowerPoint and handouts PowerPoint and handouts are available at: are available at:

www.kennesaw.edu/iewww.kennesaw.edu/ie

Thanks!Thanks!