Experiments on Education as a Service

©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011

Experiments on Education as a Service

Hyoyoung KimJames R. Morrison

Department of Industrial and Systems EngineeringKAIST

©2011 – H. Kim, J. Morrison – ICSSSM’11 – June 26, 2011 - 2

Contents

• Motivation• Experiments• Statistical Analysis• Concluding Remarks


Literature Review

Past Research My Research

University Experience O[1,2,4] X

Class experience X O

Kano Classi-fication

Basic Need O[1,4] X

Performance Need O[1,2,4] X

Excitement Need X O

QFD Approach O[1,3] X

SERVQUAL Approach O[3,4] O

Statistical Analysis

Wilcoxon signed-rank test X O

Kruskal-Wallis test X O

Experiment on single course X O[1] Bagchi, U.; , "Delivering student satisfaction in higher education: A QFD approach," Service Systems and Service Management (ICSSSM), 2010 7th International Conference on , vol., no., pp.1-4, 28-30 June 2010, doi: 10.1109/ICSSSM.2010.5530090[2] Mathew Joseph, Beatriz Joseph, (1997) "Service quality in education: a student perspective", Quality Assurance in Education, Vol. 5 Iss: 1, pp.15 – 21[3] Sangeeta Sahney, D.K. Banwet, S. Karunes, (2004) "A SERVQUAL and QFD approach to total quality education: A student perspective", International Journal of Productivity and Per-formance Management, Vol. 53 Iss: 2, pp.143 – 166[4] Kay C. Tan and Sei W. Kek, (2004) “Service Quality in Higher Education Using an enhanced SERVQUAL approach,” Quality in Higher Education, Vol. 10, No. 1, pp. 17-24


Motivation

Course Experi-ence

Satisfaction Evaluation TeachingBetterBetter

Take Home Point: Good service improves the student experience and increases satisfaction But what is good service?

ServiceService

Service Service


Motivation

• Identify how to increase customer satisfaction in Education

• Experiment 1– GOAL: Determine Basic, Performance, and Excitement needs– Kano’s model survey– Result: Identified 4 BN, 11 PN, 20 EN

• Experiment 2– GOAL: Determine which services (among excitement needs) make sta-

tistically significant difference in student satisfaction– SERVQUAL survey– Result:

• Observed drift of service and service memory effect• Numerous services improve/influence student perceptions


Experiment 1 - Spring, 2010

• GOAL: Determine Basic, Performance, and Excitement needs

• KAIST IE 200 Introduction to Operations Research class• Class size: 186 students (from all science and engineering majors)• Conducted Kano’s model survey

– 76 Functional and Dysfunctional questions

– 18 Solution questions

– 15 Overall satisfaction questions

Functional

Dysfunctional



RESULTS:• Basic Need (Must-be):

– Lecture room density is NOT high (large room size with small number of students)

– Lecture notes are uploaded at least one day before class– Lecture room is NOT disturbed by outside noise– Class time is well kept

Must-be

Dissatisfac-tion

Done very well

Done very poorly

Satisfac-tion



RESULTS:• Performance Need (One-dimensional):

– Exams and/or homework are directly related to lectures– You feel you learned a lot at the end of the semester– Lecture room is close to dormitory– Professor is confident about lecture contents– Professor enjoys teaching– Professor prepares the lecture well– Professor delivers the lecture clearly– Lecture is easy to understand– Professor respects you– TAs are kind– Lecture notes are easily understandable– Lecture room is clean

Satisfac-tion

Dissatisfac-tion

Done very wellDone very poorly

One-dimen-sional



RESULTS:• Excitement Need:

– Furniture and classroom equipment are up-to-date– Professor gives candies when answering questions– Professor and TAs provide sufficient and convenient office hours– Professor is humorous– Professor smiles while giving lectures– Professor cares about students– Professor’s appearance is neat– Professor calls students by name– Professor is kind– In-class activities are fun– Course material can be applied in the real world– Lectures make you think

Satisfac-tion

Dissatisfac-tion

Done very well

Done very poorly

Attractive


Experiment 2 - Fall, 2010

• KAIST IE 332 Operations Research II class, 60 students (IE ma-jors)

• SERVQUAL survey to measure expectation, performance, and importance

• GOAL: Determine which excitement needs measurably improve student satisfaction

Topic Service Dura-tion

Chap. 2 Professor is humorous 2 weeks

Chap. 3 Candies or rewards are given when asking or answering questions 2 weeks

Chap. 4 Fun in-class activities 2 weeks

Chap. 5 Nominal service 2 weeks

Chap. 6 Full service 3 weeks

Chap. 7 Real world application + full service 3 weeks


Experiment 2 - Fall, 2010

• Handling missing data: Regression• Data reliability: Cronbach’s alpha (α)

– 0 ≤ α ≤ 1, ↑ α, ↑reliable– Recommended to have a reliability of 0.70 or higher to be used as

data– Data of six surveys: 0.89 ≤ α ≤ 0.94 ⇒ can trust data

• Correlation– Measure of how much each pair of scores relates

• Statistical testing– SAS 9.1– Wilcoxon signed-rank test (paired sample test) – Kruskal-Wallis test

0.00-0.25: little if any0.26-0.49: low

0.50-0.69: moderate0.70-0.89: high

0.90-1.00: very high

non-parametric statistical hypothesis test for the case of two related samples

a non-parametric method for testing equality of population medians among groups


Statistical Analysis <Summary>

ServiceHumor Candy In-class Nominal Full Application

Survey

Humor 4.410 4.188 4.100 4.109 4.351 4.231

Candy 3.897 4.173 3.367 3.047 4.263 3.654

In-class 2.962 3.116 4.250 3.478 4.439 4.1

Application 3.372 3.348 3.550 3.522 3.632 3.712

• Perception 1: Strongly dis-agree2: Disagree3: Neutral4: Agree5; Strongly agree


Statistical Analysis <Humor>


Survey

Humor 4.410 4.188 4.100 4.109 4.351 4.231

Accept (p-value: 0.420 ≥ 0.05) Reject (p-value: 0.017 ≤ 0.05)

Reject (p-value: 0.003 ≤ 0.05)

Accept (p-value: 0.854 ≥ 0.05) Reject (p-value: 0.013 ≤ 0.05) Accept (p-value: 0.939 ≥ 0.05)

• Input– Jokes and funny/silly examples– Short funny music

• Output

– Spillover effect – effect of humor lasted longer than other services.– H0: there is no difference or relationship between the variables of interest– If p-value ≥ 0.05 , H0 is accepted → no difference in measurements– Students still remembered humor and rated humor higher on the next sur-

vey

1: Strongly dis-agree2: Disagree3: Neutral4: Agree5; Strongly agree


Statistical Analysis <Humor>


SurveyWell pre-

pared 4.359 4.116 4.283 4.304 4.456 4.346

Overall Ser-vice 3.949 3.783 3.917 3.891 3.982 3.885

Enjoyment 3.782 3.551 3.717 3.522 3.596 3.673

Kindness 4.385 4.203 4.167 4.153 4.390 4.308

• Outcome

Correlation of 0.6721: Strongly dis-agree2: Disagree3: Neutral4: Agree5; Strongly agree


• Input– Give candy to students when they ask or answer questions

• Output– Perceived candy

– Drift of needs (Excitement need to basic need): 2.26/5 → 3.11/5 for the importance of giving candies

Statistical Analysis <Candy>


Survey

Candy 3.897 4.173 3.367 3.047 4.263 3.654



• Input– Skits– Games– Team activity– Active learning activities

• Output

– Correlation of 0.43324 between in-class activities and confidence

Statistical Analysis <In-class Activity>


Survey

In-class 2.962 3.116 4.250 3.478 4.439 4.1

Time, effort 1: Strongly dis-agree2: Disagree3: Neutral4: Agree5; Strongly agree


Statistical Analysis <Nominal Service>


Survey

Enjoyment 3.782 3.551 3.717 3.522 3.596 3.673

Kindness 4.385 4.203 4.167 4.153 4.390 4.308

• Outcome

Significant changeInsignificant change




Survey

Humor 4.410 4.188 4.100 4.109 4.351 4.231

Candy 3.897 4.173 3.367 3.047 4.263 3.654

In-class 2.962 3.116 4.250 3.478 4.439 4.1

• Input– Candy– Humor– In-class activity– Enthusiasm– Neat appearance

• Output

Statistical Analysis <Full Service>



Statistical Analysis < Full Service >


SurveyWell pre-

pared 4.359 4.116 4.283 4.304 4.456 4.346

Overall Ser-vice 3.949 3.783 3.917 3.891 3.982 3.885

• Outcome– Positive

– Negative


Survey

Lecture 3.936 3.449 3.517 3.522 3.298 3.442

Lecture note 3.769 3.507 3.683 3.609 3.491 3.654

Concentrate 3.910 3.783 3.717 3.717 3.632 3.654

Significant changeInsignificant change


Concluding Remarks

• Students perceived the offering of such services

• We observed needs drift of such services

• We also observed the service memory effect

• Needs that require further improvement: Lecture notes are easy to understand Sufficient and convenient office hours Feel you learned a lot Exams and/or homework are directly related to lectures

• Future Research: Service increases student learning outcome

Experiments on Education as a Service

Documents

Transcript of Experiments on Education as a Service