Evaluation of the UF Summer Abroad Program Florida Consortium for International Education David...

1

Evaluation of the UF Summer Abroad Program

Florida Consortium for International Education

David Sammons Dena Roberts James Demery

October 6, 2011

Orlando, Florida

2

Program Improvement◦ Satisfaction with program services and facilities◦ Satisfaction with the experience

Program Impact◦ Changes in attitudes, skills, behaviors

Changes that occurred: while students were abroad because they were abroad

Purposes of the Evaluation

Program LogicInputs

Resources Activities Outputs Immediate Outcome

Long-term Impact

Applicants

Student fees

Annual budget

Staff (10 fulltime and 1 part-time)

Facilities

Faculty support

Information sessions

Recruitment and placement

Faculty and student workshops

Staff training

Students studying abroad

Cross-cultural faculty teaching

Research partnerships

Publications

Presentations

Clear career aspirations

Increased self-awareness

Willing to take appropriate risks

Respect for diversity

Increased intercultural learning

Global citizenship: Behave in ways that sustain and enhance the economic, political, social, environmental systems of the global community

4

How did studying abroad impact the intercultural learning and global citizenship of UF students?

What impact does gender, academic program, class level, ethnicity, parents’ educational attainment, and political affiliation have on intercultural learning and global citizenship?

How can the UF Summer Study Aboard program be improved?

Evaluation Questions

5

Mixed-method

◦ Surveys Study Abroad and Non-study Abroad

◦ Interviews Study Abroad students

◦ Document reviews

Evaluation Methodology

6

Quasi-experimental study◦ Nonequivalent pretest-posttest control group

Matching vs. random assignment

Limitations of the study◦ Self-report data◦ Retrospective Pretest for the ILO survey ◦ Size of the control group (n=50) for ILO survey

limited the minimum detectable effect to .30

Research Design

7

Intercultural Learning Objectives survey (ILO)

◦ 1001 Recipients, 494 Respondents (Study Abroad)

◦ 60 Recipients, 50 Respondents (Control group)

UF Study Abroad Services survey (SAS)

◦ 1001 Recipients, 494 Respondents

◦ *Administered ILO and SAS surveys at same time

Data Collection

8

Student Experience in Research Universities survey (SERU)

◦ 455 of the 494 ILO and SAS respondents

◦ 455 of the 15,000 Non-SA students (Control)

UF Study Abroad Services interview

◦ 15 of the 494 SAS respondents

Data Collection (cont’d)

9

Phase Two (all Study Abroad terms)

◦ Global Competence Awareness Assessment (GCAA)

Data Collection (cont’d)

10

Avoid Type I errors (false positives) and Type II errors (false negatives)

Optimal Design Software (Univ. of Michigan)◦ Alpha level (.05); null hypothesis rejection

◦ Power (.80); to detect meaningful effects

◦ 50 students in control group

*Minimum detectable effect on ILO: .30 std. dev.

Sample Size/Power Analysis

11

T-tests for significant difference in means◦ Paired sample t-tests (pretest vs. posttest)

◦ Independent samples t-tests (SA vs. Control)

ANOVA (with Bonferroni correction) ◦ Where are the significant differences?

Chi square

Type of Analyses

12

Male Female0

10

20

30

40

50

60

70

80

36.00

64

33.00

67

Survey Recipients Survey Respondents

Perc

en

t

Composition of ILO and SAS survey Recipients and Respondents

Afr-Amer

Asian Cauc. Hisp. Other Ukn0

10

20

30

40

50

60

70

8 8

60

17

7

04

7

60

14

7 8

Survey Recipients Survey Respondents

Perc

en

t

Ethnic Composition of ILO and SAS Recipients and Respondents

College Recipients(N=1001)

Respondents(n=494)

Accounting 1% 1%Agriculture & Life Science 17% 8%Business 15% 11%Design, Construct, & Planning 1% 5%Education 2% 2%Engineering 7% 7%Fine Arts 5% 3%Health & Human Performance 5% 4%Journalism & Communication 12% 12%Law <1% 2%Liberal Arts & Sciences 31% 30%Natural Resources & Environ. <1% 1%Nursing <1% <1%Public Health <1% 1%Undecided 2% <1%

ILO and SAS Survey Recipients and Respondents by College

15

8%

24%

13%25

%

7%

12%

2%

9%

Very Liberal (8%)

Liberal (24%)

Slightly Liberal (13%)

Moderate (25%)

Slightly Conservative (7%)

Conservative (12%)

Very Conservative (2%)

Did Not Indicate (9%)

ILO Survey and SAS Survey Respondents by Political Affiliation

16

Freshman6%

Sophomore16%

Junior33%

Senior23%

Graduate17%

Unknown5%

FreshmanSophomoreJuniorSeniorGraduateUnknown

ILO Survey and SAS Survey Respondents by Class Level

17

Educational Attainment Mother(n = 494)

Father (n = 494)

No formal education < 1% < 1%Less than HS diploma 3% 6%High school diploma 18% 17%Assoc./Postsecondary cert. 12% 17%Bachelor’s 34% 24%Post-baccalaureate cert. < 1% < 1%Master’s 15% 17%Professional 5% 8%Doctoral 3% 9%Did not indicate 10% 10%

Parental Education: ILO Survey and SAS Survey Respondents

18

Five Domains

◦ Functional Knowledge

◦ Knowledge of World Geography

◦ Knowledge of Global Interdependence

◦ Knowledge of Interpersonal Accommodation

◦ Knowledge of Cultural Sensitivity

Intercultural Learning Objectives (ILO)

19

Lead discussions and conversations Talk way out of difficult situations Give directions Pacify an angry person Know what’s funny Know advantage of taking risks Locate information in newspaper

Functional Knowledge

20

Locate a bar or restaurant Take public transportation Use public telephone Purchase toothpaste Know norms and taboos Know different ways of expressing ideas Can compare languages and cultures

Functional Knowledge (cont’d)

21

Can name 4 rivers n Europe Can name 3 rivers in Asia Can name 6 countries in Africa Can name 4 countries in South America Can name the 7 continents

Knowledge of World Geography

22

Know how world markets affect own career Can explain US foreign policy Know how foreign manufacturing affects

costs of goods in the US Understand US military situation (N. Korea) Can compare political and cultural freedom

Knowledge of Global Interdependence

23

Know importance of being flexible Know importance of being patient

Knowledge of Interpersonal Accommodation

24

Understand own reactions in different situations

Understand significance of language and cultural differences

Know how settings affect own style of interacting

Knowledge of Cultural Sensitivity

25

Student satisfaction with:

◦ Staff

◦ Services (UF and abroad)

◦ Facilities (UF and abroad)

◦ Housing

◦ Excursions

◦ Host country

Suggestions for improvement

Study Abroad Services (SAS) survey

26

Deeper probe of SAS survey responses

Triangulation (i.e., reliability check)

Study Abroad Services interview

27

Developed by UC, Berkeley Mission of SERU

◦ to help broaden our understanding of the undergraduate experience and to promote a culture of institutional self-improvement

At UF, approximately 20,000 students participate

Student Experience in the Research University survey (SERU)

Results

What are we trying to measure?

Increased linguistic competency

Defined Career aspirations

Increased self-awareness

Willingness to take risks

Global awareness and knowledge

Intercultural competence

Cross-cultural engagement

Respect for diversity

Enhanced communication skills

Personal Growth

Improved learning skills

International Job skills

Personal expectations met

Continued global competency

Glossari

Construct/Domain NMean

(Before)Mean(After)

Mean Gain Std. Dev.Sig.

(2-tailed)

Intercultural Learning

453 3.65 4.02 .36 .415 .000


453 3.43 3.99 .56 .596 .001

World Geography

451 3.85 3.90 .54 .345 .000

Global Interdependence

450 3.67 3.90 .23 .410 .000

Interpersonal Accommodation

434 4.15 4.33 .18 .467 .000

Cultural Sensitivity

451 3.87 4.26 .39 .558 .000

Changes in Intercultural Learning as Measured by the ILO Survey (Study Abroad Participants)


Very liberals (M = 4.27, SD = .491) and conservatives (M = 3.92, SD = .658).

Very liberal (M = 4.27, SD = .491) and moderates (M= 3.93, SD = .551).

Functional Knowledge There a statistically significant gain in

functional knowledge by political affiliation between:


World Geography There a statistically significant gain in world

geography by political affiliation between: Very liberals (M = 4.20, SD = .829) and

slightly liberals (M = 3.66, SD = .846). Very liberals (M = 4.20, SD = .829) and

moderates (M = 3.86, SD = .729). Very liberals (M = 4.20, SD = .829) and

conservatives (M = 3.77, SD = .831).

Global Interdependence There a statistically significant gain in

global interdependence by political affiliation between:

Very liberals (M = 4.14, SD = .669) and moderates (M = 3.75, SD = .754).


There a statistically significant gain in interpersonal accommodation by college between:

College of Liberal Arts and Sciences (M = 4.44, SD = .625) and Design Construction Planning (M = 3.91, SD = .900).

Cultural Sensitivity There a statistically significant gain in cultural

sensitivity by college between: College of Liberal Arts and Science (M = 4.45,

SD = .548) and Engineering (M = 3.96, SD = .848).

There was also a statistically significant gain in cultural sensitivity by political affiliation between:

Very liberals (M = 4.53, SD = .528) and moderates (M = 4.09, SD = .707).


Difference in Intercultural Learning between Study Abroad and Non-Study

Abroad Students

Construct/DomainControl Group

Study Abroad

Mean Diff.

Std. Dev.

Sig.(2-

tailed)

Effect Size

Percentile Gains


3.65 4.02 .750 .415 .000 1.1750th to

88th


3.43 3.99 .967 .596 .000 1.4150th to 92nd

World Geography

3.85 3.90 .631 .345 .000 .7050th to

76th

Global Interdependence

3.67 3.90 .562 .410 .025 .6950th to

75th


4.15 4.33 .250 .467 .000 .2650th to

60th

Cultural Sensitivity

3.87 4.26 .555 .558 .000 .7050th to

76th

SERU

Yes No

Internship under the direction of a faculty member

Other internship (e.g., co-op, clinical assignment)

Any UF study abroad program

A study abroad program with another college or university

Have you completed or are you now participating in any of the following activities?


SERU

When you started at this campus, and Now

Very Poor

Poor Fair GoodVery Good

Excellent

Foreign language skills

Please rate your level of proficiency in the following areas when you started here and now.


SERU

When you started here, and now Very Poor

Poor Fair GoodVery Good

Excellent

Linguistic and cultural competency in at least one language other than my own.

How would you rate your competencies in the flowing areas when you started here and now?


SERU

National Certification ◦ (for some programs)

Increased linguistic competencySAS Survey

How much did you improve your ability to communicate in your host country’s language while abroad?

Increased linguistic competencySAS Survey

Do you plan to continue your language studies at UF or at your home institution?

Career aspirations determinedSERU

What do you plan to do when you graduate?◦ Enroll in graduate or professional school◦ Work full time◦ Be self-employed◦ Study or work abroad◦ Join armed forces◦ Work in an internship or volunteer position◦ Take a year off◦ Do something else◦ I have no idea

Career aspirations determined SERU

What is the highest academic degree or credential you plan to earn, eventually?◦ Bachelor’s degree (BA, BS, etc.)◦ Teaching credential◦ Business mater’s (MBA)◦ Other professional master’s (MEd, MPP, MPH, MFA,

MLIS, MSN, MSW, M.ARCH)◦ Academic master’s (MA, MS, etc.) Law Degree (LLB or

JD)◦ Medical doctorate other than MD (DO, DDS, DVM, etc.)◦ Medical doctorate (MD) Doctorate (PhD, EdD, etc.)◦ Multiple doctoral degrees (MD/PhD)◦ I don’t know yet

Career aspirations determined SAS Survey

Did the study abroad experience influence you to change your major or minor?

Career aspirations determined Interview

Do you have different academic goals after going abroad?

85% of the summer and spring 2011 students said “yes.”

Career aspirations determined Interview

Would you like to work in your host country after graduation?◦ 90% said yes

Career aspirations determined SAS Survey

Has study abroad influenced your career interests?

Skills obtained to function in a professional international atmosphere

SERU

As a University of Florida student, how would you rate your competencies below?


Do you think studying aboard makes students better international job candidates?◦ 100% said “YES”


Do you think you’d like to work in your host country after graduation?◦ 90% said YES◦ No, because they wanted something

different or it was difficult to get jobs in that country without family connections.


SERU

Strongly disagree

DisagreeSomewhat disagree

Somewhat agree

AgreeStrongly agree

I obtained the skills I need to function in the international arena

I obtained information about international opportunities in my academic field

I increased my knowledge about the ethics, culture, politics, society or values of another country

Please rate your level of agreement with the following statements about your experience at UF.

Understanding of a different culture’s current global issues that impact their major field of

study

Never RarelyOccasionally

Somewhat often

OftenVery often

Worked with a faculty member on a project with an international/global theme

Presented a paper at a symposium or conference or participated in a panel on international/global topics

Attended lectures, symposia, workshops or conferences on international/global topics

While attending UF how frequently have you engaged in the following?

Ability to recognize individual and cultural differences, SERU

When you started at this campus, NOW Very Poor

Poor Fair Good Very Good

Excellent

Ability to appreciate, tolerate and understand racial and ethnic diversity

Ability to appreciate the fine arts (e.g., painting, music, drama, dance)

Ability to appreciate cultural and global diversity

Understanding the importance of personal social responsibility

Self awareness and understanding

Ability to recognize individual and

cultural differences SERU

Strongly disagree

Disagree

Somewhat disagree

Somewhat agree

AgreeStrongly agree

I have a deeper understanding of another country’s culture based on activities on campus

I evaluated my own values based on another culture



Ability to recognize individual and cultural differences

SERU

Never RarelyOccasionally

Somewhat often

OftenVery often

Their religious beliefs were very different from my own

Their political opinions were very different from my own

They were an immigrant or from an immigrant family

They were of a different nationality than my own

They were of a different race or ethnicity than my own

Their gender was different

Their sexual orientation was different

They were from a different social class

They had physical or other observable disabilities

They had learning, psychological, or other disabilities that are not readily apparent

How often have you gained a deeper understanding of other perspectives through conversations with fellow students because they differed from you in the following

ways?

Ability to recognize individual and cultural differences Interview

Ability to recognize individual and cultural differences between the host country and USA ◦ Religious freedom ◦ Political freedom◦ public safety◦ Relationships? Dating/courtship rituals?◦ Leisure activities

100% of all of students could speak about some kind of similarities or differences.

Improved learning skills How did the host country’s teaching style(s)

affect your learning? - Summer programs, so there are mostly UF professors, so they were not exposed to new teaching styles

Improved learning skills Did the teaching styles match your

preferred learning style? 50% mixed answers

- No, Yes

Understanding of a different culture’s historical events and

current issuesAppreciate cultural and global

diversity

Never Rarely OccasionallySomewhat often

Often Very often

The United States

Countries outside the US

Global politics and diplomacy

Global climate and environmental issues

International business and economics

Global health issues

During this academic year how frequently have you followed news about the following?

Student expectations met

Did you fulfill your expectations of your program?

Student expectations metWould you recommend this program to other students?

92.2% Yes7.8% No

Student expectations met

How does the cost of studying abroad compare to the benefits?

Personal Growth Would you say this experience made you a

better person or are about the same as before?

91% yes 9% Same

Personal Growth

Not important

Somewhat important

Very important

Be in a position to give something back to my community after finishing my education

Discover what kind of person I really want to be

Explore new ideas

Indicate how important each of the following college goals is to you.

Global awareness and knowledgeSERU

Strongly disagree

Disagree

Somewhat disagree

Somewhat agree

AgreeStrongly agree



So What?

With the collected data, we want to disseminate the information to our stakeholders. ◦ Students, faculty, administrators, staff and

parents Create reports for faculty led programs to

improve their study abroad programs Marketing purposes Justify our budget and increase the value of

a student’s University of Florida degree

Evaluation of the UF Summer Abroad Program Florida Consortium for International Education David...

Documents

Transcript of Evaluation of the UF Summer Abroad Program Florida Consortium for International Education David...