NAFSA PRE-CONFERENCE WORKSHOP #049

RESEARCH FOR INTERNATIONAL EDUCATION: THE NUTS AND BOLTS

IN THREE DIMENSIONS

Washington, DCMonday, May 25, 2009

8:00AM - 5:00PM

Facilitators

Peter Briggs, Director, Office of International Students and Scholars, Michigan State University

Darla Deardorff, Executive Director, Association of International Education Administrators

Kevin Gormley, Senior Program Officer, NSEP

Mick Vande Berg, Vice President for Academic Affairs, CIEE

Workshop Learning GoalsWhen the workshop ends, you will:

• Be able to identify key elements of research theory and design• Be able to identify major developments in the evolution of US

international education research • Understand some of the salient issues facing researchers in three

domains: study abroad, international students and scholars, and the internationalization of higher ed. institutions

• Be aware of a number of major funding possibilities for international education research

• Understand the importance of collaborating in designing and carrying out international education research

• Be able to collaboratively design an international education research project

Workshop Agenda• Introductions & Workshop Overview: Mick• Research Theory and Design: Darla• Research on Study Abroad (Part I): Mick• BREAK (10:00-10:15)• Research on Study Abroad (Part II): Mick• Research on International Students & Services: Peter• Funding Possibilities: Kevin• LUNCH (12:30-1:30)• Designing Group Projects I (table work: all facilitators)• Review of Group Project Status: Mick• BREAK (3:00-3:15)• Designing Group Projects II (table work: all facilitators)• Group Presentations (spokesperson for each group)• Open Discussion, Wrap-Up

The Evolution of Study Abroad & Study Abroad Research

Mick Vande Bergmvandeberg@ciee.org

The Traditional (Teacher-Centered) U.S. Study Abroad Paradigm

• Academic disciplinary learning (including language learning) is primary

• Learning occurs through exposure to the new and different• Learning is “transformational”• “Immersion” as the goal: unmediated learning• The longer the program, the better • Direct enrollment is the best type of program• Students learn more in home stays• Good students at home will be good students abroad• When students don’t learn, it’s their fault

The Newer (Learner-Centered) Paradigm• Academic learning is important; intercultural learning is

fundamental• Learning is developmental • Exposure is a necessary but not sufficient condition for

learning abroad: intervention is necessary• Identifying learning goals—the program’s and students’• Assessment and feedback• Good grades at home are not predictors of learning abroad• When students don’t learn abroad, “the problem” may be in

the program • “Learning through design, not chance”**Savicki, V. (Ed.) (2008). Developing Intercultural Competence and

Transformation. Sterling, VA: Stylus.

Emerging Learner-Centered Paradigm: Contributing Developments

• Assessment movement in U.S. higher education• Soaring U.S. study abroad enrollments: +400%

increase in 22 years • Decades of research & scholarship on teaching &

learning• Coming of age of Intercultural Communication• Growth of research on student learning abroad

Study Abroad Research Growth*

Decade Studies Growth1950s 34 +127%1960s 117 +244%1970s 189 +62%1980s 377 +99%1990s 675 +79%2000-05 c. 500 N/A

(* Bolen, M. (2007). A Guide to Outcomes Assessment in Education Abroad. Carlisle: Forum on Education Abroad, p. 99.)

Some Recent Research Milestones• Frontiers: the Interdisciplinary Journal of SA (1995)• Journal of Studies in International Education (1996)• Freed, B. (Ed.) (1995). Second Language Learning in a Study

Abroad Context. Amsterdam: John Benjamins.• Engle, L. and J. Engle (2003). Study Abroad Levels: Toward a

Classification of Study Abroad Types. Frontiers: The Interdisciplinary Journal of Study Abroad IX: 1-20.

• Vande Berg, M. (Guest Ed.) (2004). Assessing Student Learning Abroad. Frontiers: Vol. X, November.

• Bolen, M. (2007). A Guide to Outcomes Assessment in Education Abroad. Carlisle: Forum on Education Abroad, p. 99.

• Savicki, V. (2008). (Ed.) Developing Intercultural Competence and Transformation. Sterling, VA: Stylus.

• Vande Berg, M.; Connor-Linton, J., & Paige, R. M. (forthcoming: Nov. 2009) The Georgetown Consortium project: Intervening in student learning abroad. Frontiers Vol. XVIII.

Study Abroad Research Evolution

Early Research: 1950s-60s• Primary Domain: Second

Language Learning• Intent: Document that Learning

was Taking Place

Recent Research: 1990s-2000sIncreasingly Sophisticated & Rigorous• From One Learning Domain to Many• Validity & Reliability• Control Groups• Research Partnerships• From Simply Documenting Learning to

Documenting Impact of Variables on Learning

Recent Research: 1990s-2000sMore Complex Research Questions

• Multiple Learning DomainsSecond Language: Four SkillsIntercultural Sensitivity or CompetenceGlobal AwarenessAcademic DisciplinesWorkplace Values and Skills

Recent Research: 1990s-2000sFocus on Environmental Impacts on

Learning• Program Duration• Second Language used in Courses Abroad• On-Site Intercultural Mentoring• Contact with Host Nationals• Housing Types• Home Institution Grading Policy

Recent Research: 1990s-2000sFocus on Learner Characteristics

• Gender• Academic Discipline• Previous Language Study/Competence• Learning Readiness• Student Attitudes: Motivations,

Expectations, Goals

Research Becomes Increasingly Self Reflective • Appearance of Research Journals:

Frontiers (1995)Journal of Studies in International Ed (1996)

• Collections of Articles:Freed, B. (1995). Second Language Learning in a Study Abroad Context. Amsterdam: John Benjamins.Vande Berg, M. (ed.) Assessment of Study Abroad Learning, Special Issue of Frontiers: the Interdisciplinary Journal of Study Abroad. Volume X, Fall 2004.Bolen, M. (2007). A Guide to Outcomes Assessment in Education Abroad. Carlisle: Forum on Education Abroad, p. 99.Savicki, V. (Ed.) (2008). Developing Intercultural Competence and Transformation. Sterling, VA: Stylus.

Three Representative StudiesPaige, R. Michael, Andrew D. Cohen, et al.

“Assessing the Impact of a Strategies Based Curriculum on Language and Culture Learning Abroad,” Frontiers: The Interdisciplinary Journal of Study Abroad, 10 (2004): 253-273.

• Impact of Maximizing Study Abroad on student learning abroad

• Two domains: second language and intercultural learning

• Pre-existing and Specially-developed Instruments

• Quantitative & Qualitative Approaches

Sutton, Richard C., & Donald L. Rubin. “The GLOSSARI Project: Initial Findings from a System-Wide Research

Initiative on Study Abroad Learning Outcomes,”

Frontiers, 10 (2004): 65-82

• Explores Student Acquisition of Content Knowledge & Cognitive Understanding

• Multi-year, six-phase study at multiple GA schools

• Extraordinarily Comprehensive: during past six years, 25,000 students

• Quantitative and Qualitative Approaches

Vande Berg, M.; Connor-Linton, J., & Paige, R. M. (forthcoming: Nov. 2009) The Georgetown Consortium project: Intervening in student learning abroad. Frontiers Vol. XVIII.

Four diverse partner institutions & 61 programs abroad• 1,300 students from 190 home institutions • Pre- and post-testing of learning in three domains:

second language proficiency, intercultural development, disciplinary knowledge

• Relied on & tested Engle & Engle learning variables • Quantitative and qualitative approaches• Outside consultants: methodology and instrument

design, faculty workshops, & data analysis

The Current Debate about U.S. Student Learning Abroad

• View I: Students learn quite effectively abroad when left to their own devices. (“Learning through Chance”*)

• View II: Most Students learn effectively abroad only when we intervene in their learning. (“Learning through Design”*)

* Savicki, V. (Ed.) (2008). Developing Intercultural Competence and Transformation. Sterling, VA: Stylus.

Georgetown Study: Principal Independent Variables*

• Duration of Program• Amount of pre-departure target language study• Language of coursework on site ( a. content courses in

target language; b. target language courses)• Context of academic work (a. location of courses; b.

students-in-course composition)• Type(s) of housing at program• Experiential learning activities• Mentoring, or guided cultural reflection *Engle, L. and J. Engle (2003). Study Abroad Levels: Toward a

Classification of Study Abroad Types. Frontiers: The Interdisciplinary Journal of Study Abroad IX: 1-20.

Georgetown: Additional Independent Variables

• Pre-departure and on-site orientations with cultural component

• Gender• Academic Major• Prior study abroad experience• Prior experience living abroad• Amount of Interaction with host country nationals• Student perception that the new culture is

similar/dissimilar compared to home culture

Testing of Learning Domain I:Second Language Acquisition

• Instrument: Simulated Oral Proficiency Interview (SOPI)*

• Pre- and post-testing of 968 students• Testing of 7 foreign languages• Significant amounts of data for student

learning in French, German, and Spanish*http://www.cal.org/topic/ta/sopi.html

Language Research Finding 1Study Abroad vs. Control Students

Study Abroad Participant (SAP) ratings increased significantly more than Control students’ SOPI ratings.

On average, SAPs’ SOPI scores improved one ACTFL sublevel, from just below Intermediate High to just below Advanced Low.

On average, Controls improved about half a sublevel, from just below Intermediate High to Intermediate High.

Language Research Finding 2Oral Proficiency Gain: Males vs Females

On average, Female SOPI scores improved one ACTFL sublevel, from just below Intermediate High (1.7) to just below Advanced Low (2.0).

However, Male students, on average, improved only about half an ACTFL sublevel, from just below Intermediate High (1.6) to Intermediate High (1.8).

Language Research Finding 3:Housing and Language Gain

SAPs housed either with a) host families or b) other international students made equal oral proficiency gains

SAPs housed with c) other U.S. students or d) host country students gained less than those in housing type a) or b)

Language Research Finding 4SOPI Gains and Time Spent with Host Families

We found a significant relationship between SOPI gains and the percentage of time students spent with a host family (the more time spent with them, the higher the gains)

Testing of Learning Domain II:Intercultural Learning

• Data generated through Pre- and Post-testing, using the Intercultural Development Inventory

• 1,297 students at 61 programs abroad completed pre- and post-IDIs

Copyright, 2007, Mitchell R. Hammer, Ph.D.Modified from the Developmental Model of Intercultural Sensitivity (DMIS), M. Bennett, 1986

Intercultural Development ContinuumDen

ial

Polar

izatio

n

Defen

se/R

ever

sal

Minim

izatio

nAc

cept

ance

Adap

tatio

n

Monocultural Mindset Intercultural Mindset

Intercultural Research Finding 1Intercultural Learning: SAPs vs. Controls

SAPs showed significantly greater increases in IDI scores than Control

students at home.

Intercultural Research Finding 2Gender and Intercultural Learning

Female SAPs showed statistically significant increases in their IDI score.

Male scores, however, actually decreased—males’ IDI scores were lower than the scores of Control students on home campuses.

(See differences in “Change score” column.)

Gender and IDI Gain (SAPs & Controls ; N = 1290)MeanN

IDI-1 SD IDI-2 SD Change score

SAPs Male 384 94.31 14.68 93.81 17.22 -.4919

Female 772 97.19 13.97 100.94 15.29 3.745

Total 1156 96.23 14.27 98.56 16.30 2.33

Controls Male 36 95.12 12.78 95.42 13.72 .3

Female 98 93.62 13.42 93.41 13.16 -.21

Total 134 94.02 13.22 93.95 13.29 -.07

Total 1290 96.00 14.17 98.09 16.07 2.09

Research Finding 3Cultural Mentoring and IDI Gain

Students abroad who received cultural mentoring “often” to “very often” showed the greatest increase in IDI scores.

Research Finding 6Housing Type and IDI Gains

• SAPs who lived with students from the U.S., or with host country students, made significant IDI gains.

• SAPs who lived with a host family or with (non-U.S.) international students did not make significant IDI gains.

Intercultural Finding 6AExposure to Host Family and IDI Gain

The higher the amount of time spent with the host family, the larger the change in SAP IDI score.

Intercultural Research Finding 6BExposure to Other U.S. People and IDI Gain

SAPs who spent the least amount of time with other people from the U.S. had the highest change in IDI score.

SAPs who spent the most time with other people from the U.S. showed the smallest gains in IDI score.

Structured interventions and learning gainsLargely non-intervention: GU Study IDI Gains

• All SAPs: +2.33• Females: +3.745• Males -.4919

Intervention : Two Intercultural Courses IDI GainsAUCP’s French Practicum

• All SAPs (366 SAPs) +12.47• Females:• Males:

Bellarmine/Willamette’s Intercultural Course • All SAPs (37): +8.19• Females (Bellarmine only): +9.16• Males (Bellarmine only): +7.22

Internationalizing Campus/Curriculum:What Role(s) Can SA Play?

Potential learning Outcomes (to be assessed):Increase global competence of faculty, staff, studentsIncrease intercultural development: faculty, staff, studentsIncrease second language ability: faculty, staff, students

Possible Activities (impact of each on learning/development assessed):• Membership on SA committee• Leading SA programs• Evaluating programs, assessing student learning• Developing programs• Pre-departure & Re-entry activities• Mentoring of student research abroad• Leading/participating in reciprocal exchange programs on campus• Partnering with host institution faculty: research, curriculum development

In Closing, Some Research Recommendations. . . .• Seek Partnerships, on and/or off campus• Be clear about why you’re doing this project• Be clear about who your audiences are• Identify clear research questions• Consider how variables influence learning in

different domains (Engle & Engle Program Elements a useful start)

Developing the Research Project: a Protocol[Who is the audience you’re doing the research for?]

[Who will the Principal Investigator & research team be?]

1. A. Either identify a research idea that interests you,OR

1 B. Identify a policy change you want to see implemented.

2. Identify why this research is important: Advocacy for policy change on campus? Program improvement? Assess for accountability? Assess for improvement? To improve academic advising? To maximize student learning? To contribute to knowledge? etc.?

3. Identify research question or questions4. Determine who the subjects will be and how they will be selected5. Identify research methods: Field research? Review/analyze existing data? Surveys?

Interviews? Focus groups? Analyze texts? etc.6. Identify Instrument needs, and identify/discuss development of instruments/data collection

methods7. Identify sequential process for carrying out research, including data collection, analysis

and reporting8. Possible funding source(s)?

Closing Activity: Pairs

• What are the three most important things you learned in this workshop?

• What do you plan to do with what you have learned?

• Who are three people you want to share this information with?