Nonparametric Survival Analysis of the Loss Rate of ... · applied survival analysis to study a...

349

Journal of Engineering EducationApril 2011, Vol. 100, No. 2, pp. 349–373

© 2011 ASEE. http://www.jee.org

Nonparametric Survival Analysis of the LossRate of Undergraduate Engineering Students

YOUNGKYOUNG MIN, GUILI ZHANGa, RUSSELL A. LONGb, TIMOTHY J. ANDERSONc, AND MATTHEW W. OHLANDb

Korea Foundation for the Advancement of Science and Creativity, East Carolina Universitya, Purdue Universityb, University of Floridac

BACKGROUND

As presented by Willet and Singer (1991), survival analysis can sensitively reveal rich informationabout when students leave their majors. Although survival analysis has been used to investigate stu-dent and faculty retention, it has not been applied to undergraduate engineering student retention.

PURPOSE (HYPOTHESIS)The impact of cohort, gender, ethnicity, and SAT math and verbal scores on the loss rate of undergradu-ate engineering students was investigated to answer the questions: Does the profile of risk of dropout dif-fer among groups with different backgrounds? When are students most likely to leave engineering?Which SAT scores better predict the risk of dropout?

DESIGN/METHOD

Using a large longitudinal database that includes 100,179 engineering students from nine universities andspans 19 years, nonparametric survival analysis was adopted to obtain nonparametric estimates of survivaland associated hazard functions, and complete rank tests for the association of variables.

RESULTS

There are significant differences for early semesters: White or female students tend to leave engineeringearlier than other populations. Engineering students leave engineering during the third semester the most,although students who have an SAT math score less than 550 tend to leave engineering during the sec-ond semester. SAT math score better predicts the risk of dropout than SAT verbal score.

CONCLUSIONS

The results of this study support using survival analysis to better understand factors that determine stu-dent success since student retention is a dynamic problem. Survival analysis allows characteristics such asrisk to be evaluated by semester, giving insight to when interventions might be most effective.

KEYWORDS

longitudinal study, nonparametric survival analysis, retention

INTRODUCTION

Recent reports on ensuring the competitiveness of industry in the United States havecalled for increasing the number and proportion of B.S. engineering graduates (NationalResearch Council, 2007). Improving the retention of undergraduate engineering studentsis an obvious approach to increasing the engineering workforce since these students are al-ready enrolled in engineering and their current retention rates are relatively low. There arealso social and institutional issues that are adversely affected by poor retention of engineer-ing students. There has been considerable research, qualitative and quantitative, directed at

350

Journal of Engineering Education 100 (April 2011) 2

determining the factors that influence retention. To better understand longitudinal trendsand variations between institutions, a database consisting of the student records from nineuniversities over the time span of 1987 to 2005 was developed. Previously, using this data-base we have applied multiple logistic regression to analyze various predictors of gradua-tion (Zhang, Anderson, Ohland, and Thorndyke, 2004) as well as hierarchical linear mod-els to account for the nested structures (Padilla, Zhang, and Anderson, 2005). Multiplelogistic regression techniques are commonly used to probe the influence of various factorson retention, but this approach has limitations. In particular, this method is used to analyzewhether an event (e.g., graduation) has occurred during a given period of time (e.g., six-year period) and thus the information on the timing of the event is not included in theanalysis. For example, the six-year graduation rate for a large public institution comparedto a small private one might be similar, but the graduation pattern between years 4 and 6could be very different, a result not represented in a regression analysis. These models aspreviously adopted do not include time as a variable and thus do not capture informationon the timing of events.

This study applies nonparametric survival analysis to create the risk profiles for studentsdropping out of engineering among groups with different characteristics (cohort, gender,ethnicity, SAT math and SAT verbal scores). This analysis of time to event data is impor-tant in diverse fields, including biology, public health, epidemiology, economics, and engi-neering. This approach asks not only whether the event occurred but also when it occurred.Survival analysis is a class of statistical methods for studying the occurrence and timing ofevents and the methodology has been developed over several decades (Cox, 1972; Kaplanand Meier, 1958). Survival analysis is so named because the method is most often appliedto the study of deaths. Although survival analysis was originally developed to analyze can-cer data, its use was later extended to study a variety of events. The method is known in theengineering literature as reliability or failure time analysis. This investigation allows us toidentify, for example, when undergraduate engineering students are mostly likely to leaveengineering. Thus, this study provides information on appropriate timing for interventionto effectively reduce the number of undergraduate students leaving engineering.

BACKGROUND

Student dropout rates from colleges and universities have long been a concern for edu-cators and the subject of considerable research. Tinto (1975; 2006) treated dropouts fromhigher education as a longitudinal process that is directly related to several factors, includ-ing individual characteristics, interactions within colleges, and institutional characteristics.Tinto’s (1975; 2006) research suggested that the view of retention had changed from oneof reflection of individual attributes, skills, and motivation to understanding the relation-ship between individuals and society, that is, the view of student retention shifted to ac-count for the role of the environment. With respect to engineering education, researchershave examined which factors might be important in predicting loss rates, including cogni-tive factors such as SAT score, high school GPA, and university GPA, as well as non-cog-nitive factors (e.g., academic motivation and institutional integration), individual charac-teristics (e.g., family background, gender and ethnicity) and institutional characteristics(e.g., institutional type, size, and environment) (French, Immekus, and Oakes, 2005;Leslie, McClure, and Oaxaca, 1998; Zhang et al., 2004). Gender, ethnicity and SATscores have been the most important predictive factors influencing not only engineering

college retention rates but also those outside of engineering (Arulampalam, Naylor, andSmith, 2004, 2007; Smith and Naylor, 2001; Zhang et al., 2004). Furthermore, genderand ethnicity are important individual characteristics, while SAT score is a critical factornot only as an individual characteristic but also as a cognitive feature. Tinto (1975, 2006)emphasized the role of the environment of society, which could be related with differentgenerations on the basis of cohort analysis. These previous findings motivated our investi-gation of cohort, gender, ethnicity, and SAT math and SAT verbal scores as major factorsfor loss of engineering majors.

Higher education researchers have long been concerned with the development and ap-plication of methods to adequately assess retention/loss rates, including survival analysis.Morita and colleagues introduced survival analysis as a powerful set of statistical techniquesfor turnover research and other behavior studies with a binary dependent variable and mul-tiple independent variables, any or all of which may be measured over time (Morita, Leeand Mowday, 1989). Singer and Willett (1991) identified the advantages of survival analy-sis for longitudinal studies of duration and the timing of event, and indicated that one ofthe reasons to use survival analysis is that it overcomes the difficulty of handling censoredobservations, i.e., when the time to the event is longer than the time of the study. For thisstudy, survival analysis allows inclusion of information on those students who were not lost(i.e., censored data). For example, information on cohorts who have not been enrolled forthe six year period or those students who are enrolled beyond the six year period is includedin survival analysis. This not only allows estimating the risk of being lost as a function oftime in the program, but also includes the most recent information in the analysis, thusmaking the study more current. In this sense, survival analysis includes the time depen-dence of the data and not just the integral or a fixed period of time (e.g., six-year graduationrate). In addition, the predictors of the event can vary with time and therefore incorporatethe dynamics of the process. Thus, survival analysis allows modeling patterns of occur-rence, comparing these patterns among groups, and building statistical models of the riskof occurrence over time (Singer and Willett, 1991). Willet and Singer (1991) demonstrat-ed how the methods of survival analysis lend themselves naturally to the study of the timingof educational events by drawing examples from the literature on teacher attrition and onstudent dropout and graduation.

Although survival analysis has primarily been applied to the medical field, its advan-tages compared with traditional analytic methods have motivated its use to investigate stu-dent retention or dropout rates in the academic field as illustrated by the following exam-ples. Ronco (1994) applied the methods of survival analysis with repeated events to alongitudinal data set to illustrate the dropout hazard for several groups of college students.Huff and Fang (1999) identified those variables that are most predictive of the risk of med-ical students experiencing academic difficulties and when these students are most suscepti-ble to encountering those difficulties by utilizing survival analysis. Murtaugh, Burns, andSchuster (1999) used survival analysis to model the retention of 8,867 undergraduate stu-dents at Oregon State University between 1991 and 1996. DesJardins and his colleaguesapplied survival analysis to study a variety of time to event problems associated with highereducation, including student dropout and student stopout using student records from mul-tiple sources including the National Center for Education Statistics and the University ofMinnesota (DesJardins, Ahlberg, and McCall, 1999; 2006; Ishitani and DesJardins,2002–2003). These studies have considered traditional factors as well as probed less com-mon ones such as athletic standing, financial aid, high school ranking, and enrollment age.Using survival analysis, Plank and colleagues examined the association between career and

351

100 (April 2011) 2 Journal of Engineering Education

technical education to academic-course-taking ratio and the likelihood of dropping out(Plank, DeLuca, and Estacion, 2005). Tamada and Inman (1997) applied survival analysisto faculty retention data to describe the gender retention difference and found that survivalanalysis is useful for answering questions such as “how to measure retention?” and “how todiscern whether men and women have different survival time?” Visser and Hanslo (2006)investigated methodological issues associated with predictive studies related to selectionand access to higher education and used nonparametric survival analysis to assess the rela-tionship between the Placement Test in English for Educational Purposes and studentdropout. They concluded that the survival analysis approach not only provided valuable in-sight into the attrition and throughput patterns of students and allowed them to answerquestions regarding whether students dropout, but also, through the use of the hazardfunction, it was able to illustrate exactly when the periods of risk were highest.

Survival analysis can be extended to any time to event data. All that is required is lengthof time, where there is a known start date and an end date for the study, and known datefor the event of interest within the study (SAS Institute Inc., 2008). In this study of engi-neering student loss rates, the event of interest is a student leaving engineering at a specificterm. The length of time is the span of the database (1987 to 2005), the known start date isthe matriculation date into the university, and an end date for the study is 2005. It shouldbe noted that in this analysis, the loss of an engineering student occurs when a studentleaves an engineering degree program by either changing major to a non-engineering pro-gram or leaving the institution. Thus it does not necessarily mean that the student droppedout of school as the student may be at the same institution in a non-engineering major or inan engineering program at another institution.

Comparisons of engineering college student loss rates by cohort group were accom-plished by using nonparametric survival analysis and the frequency. The specific group ofcohorts used in this analysis is the four-year academic year time spans 1987–1990,1991–1994, 1995-1998, and 1999-2002, and the final two-year span 2003-2004. Eachgroup of cohorts includes all students that matriculated in the nine institutions as engineer-ing majors during the specified time period. For example, the 1987-1990 group of cohortsincludes students who were enrolled as freshmen engineering students in 1987, 1988,1989, or 1990. There is no specific incident for grouping cohorts. This analysis, however,was used to explore whether there is a changing trend over time.

RESEARCH METHODS

Demographics of the DatasetThe data for this study were taken from MIDFIELD (Multiple-Institution Database

for Investigating Engineering Longitudinal Development). MIDFIELD consists of stu-dent academic records for nine public universities in the Southeastern U.S. (Clemson Uni-versity, Florida A&M University, Florida State University, University of Florida, GeorgiaInstitute of Technology, North Carolina A&T State University, North Carolina StateUniversity, University of North Carolina at Charlotte, and Virginia Polytechnic Instituteand State University). These institutions are diverse by several characteristics, including in-stitutional size, setting, levels of transfer articulation, and matriculation directly to a majorvs. a freshman engineering program. All the institutions are doctoral-granting and state-sponsored schools.

Although MIDFIELD has data for every student that attended or is attending eachMIDFIELD university from the 1987 cohort to 2004 cohort (summer, fall, and spring

352


matriculation), only data for entering freshman that declared engineering as their majorwere extracted; that is, the population of this study is limited to first-time-in-college (na-tive) students matriculating in engineering. In other words, transfer students and studentswho migrated into engineering from another discipline including undeclared majors arenot included. The population of the cohort group 2003 to 2004 is relatively small com-pared with other cohorts because it includes only two cohorts as compared to four cohortsfor the earlier groups. The total sample size for this study is N � 100,179 and Tables 1–4summarize the frequency of selected characteristics. The extracted data include 79,397males and 20,782 females (Table 1), with 6,882 Asian, 75,579 White, 15,241 Minorities,and 2,477 Other students (Table 2).

In this study, the definition of minority mirrors the National Science Foundationguideline of ethnic groups that are significantly underrepresented at advanced levels of en-gineering and science (African Americans, Hispanics, Native Americans, Alaskan Natives,and Native Pacific Islanders). The Other category includes international students and stu-dents who did not specify an ethnicity. Additionally, the students were categorized bySAT Verbal and SAT Math scores as shown in Tables 3 and 4.

Comparing this study to those in which a population is receiving cancer treatment,the participant (student) is going through a treatment (enrollment in an engineeringdegree program or group of programs). Some patients die (student is lost from engi-neering) and others do not experience death (graduate or still enrolled) and thus arecensored. Comparing this approach to engineering failure analysis, a widget (student)undergoes a test (engineering education) and the event examined is widget failure (stu-dent is lost from engineering) while the remaining widgets are shipped or remain in thetest and are considered as censored (student either graduates or is still enrolled). In all

353


TABLE 1Frequency of Gender by Group of Cohorts

TABLE 2Frequency of Gender by Ethnicity

survival analyses presented in this work the terminology of failure and non-failure willbe used since it is most familiar to engineering educators. It is noted, however, that astudent who leaves engineering may graduate and have a successful career in anotherfield or may transfer to another institution and have a successful engineering career, andthus should not necessarily be considered an academic failure, only that the student wasnot retained in the specific institution’s engineering program or specific group of engi-neering colleges as in this work. Thus in this study a non-failure is defined as a studentwho did not leave an engineering major, i.e., non-failure is a student who either gradu-ated with an engineering degree or is still attending school and had not changed majorfrom engineering to any other non-engineering major. Failure is defined as a studentwho left an engineering major, i.e., failure is a student who changed his/her major fromengineering to another discipline or left the university. An implication of these defini-tions for failure and non-failure is that the risk is evaluated only for entering freshmanthat declared engineering as their major in the college(s) of engineering considered inthe analysis. The approach does not examine the risks of events that occur after leaving

354


TABLE 3Frequency of SAT Math Score by Group

TABLE 4Frequency of SAT Verbal Score by Group

355


engineering, which may be positive such as graduation in engineering at another insti-tution. This college(s)-centered focus is particularly relevant to colleges of engineeringas they attempt to manage their enrollments in the context of their budget models. Itdoes not, however, provide information on the outcomes of those students who fail orare currently enrolled. Another important implication of this methodology is that in-formation is extracted for each term based on all students who started that term regard-less of the student outcome for that term. This renders the results more current since itconsiders currently enrolled students not yet graduated and statistically more confidentsince the population includes students who eventually have or may leave engineering.

It is noted that students who changed their majors from engineering may or may not beenrolled or have graduated from the institution. A major change from one engineering de-gree program to another one at the same institution does not constitute failure. Further-more, a student who leaves engineering but returns to engineering (stopout) at the sameinstitution (enrolled or subsequently graduated with an engineering degree) also does notconstitute failure. This study cannot track students who left engineering at one institutionand enrolled in engineering at a different institution within the database. There is the pos-sibility that those students are treated as different students. The number of those students,however, is expected to be very small compared with the population of the database, andthus should not affect the general results and trends revealed by this study. This set of defi-nitions can be considered as a robustness check of the original dropout model since the re-sults obtained under the original definition may depend on how many students re-enrolledat different institutions as engineering majors. A robustness check has been applied in aprevious longitudinal student dropout studies due to the limitation of available information(DesJardins, Ahlberg, and McCall, 1999).

In addition to the overall population of engineering students, there is an interest in un-derstanding the risk of students leaving engineering for several groups as defined by severalpre-existing variables, i.e., cohort group, gender, ethnic group, SAT Math score group,and SAT Verbal score group. Nonparametric survival analyses were conducted for each ofthese groups as well. It is noted that time was represented by enrolled semesters. This is anatural choice for academia because students are tracked by enrolled semesters as opposedto days or months. This time unit, however, requires a conversion factor when students en-roll in the shorter summer terms. Also, some schools followed the quarter system ratherthan semester system for a period of time. To solve this problem, each spring and each fallsemester counted as one semester whereas each summer A/B semester is counted as .33 ofone semester. Each whole summer semester and each quarter are counted as .67 of onesemester.

Statistical MethodologiesSAS software’s PROC LIFETEST function was used to perform the survival analysis

of engineering students in this study. In survival analysis, the outcome of interest is thetime, T, until some event occurs in the time to event data analysis. T is a continuous, non-negative random variable from a homogeneous population. The distribution of T is char-acterized by the survival function, which is the probability of an individual surviving be-yond time t. It is the basic quantity employed to describe time to event and defined as S(t) � P[T�t]. Note that the survival function is a non-increasing function with a value of1 at the origin and 0 as t approaches infinity. If T is a continuous random variable then S(t) isa continuous monotonic decreasing function and the survival function is the complement ofthe cumulative distribution function F(t) � P[T � t], i.e., S(t) � 1�F(t)]. In the context

356


of equipment or manufactured item failure analysis, S(t) is referred to as the reliabilityfunction and F(t) as the failure distribution function. The survival function is the integralof the probability density function f(t). That is

Thus, the following relationship exists:

The expression f(t)�t may be thought of as an approximate probability of the event occur-ring at time t and that f(t) is a non-negative function with the area under f(t) being equal toone (Klein and Moeschberger, 2003).

Since the standard parametric family of a specific distribution (e.g., f(t) or S(t)) is usual-ly not known, it is desirable to estimate the function value in survival data analysis. Such es-timates are called nonparametric, and most commonly the survival function is selected torepresent the estimate (Kaplan and Meier, 1958). A commonly used nonparametric proce-dure is the life-table method (also known as the actuarial method) (Allison, 1995). Thismethod is similar to the Kaplan-Meier product-limit method, but the life-table approachconveniently summarizes data sets with large numbers of observation (Kaplan and Meier,1958; Allison, 1995). The life-table estimates are computed by counting the numbers ofcensored and uncensored observations that fall into each of the time intervals [ti�1, ti), i �1, 2,…, k�1, where t0 � 0 and tk�1 � �. Censoring occurs when an individual does notexperience the event of interest in the time of the study. In this study, a student is regardedas censored if he or she does not leave engineering in each time period (i.e., the student ei-ther graduated or is stilled enrolled in engineering). The survival estimate is calculatedfrom the conditional probabilities of failure in the following manner (Allison, 1995). Forinterval i, let ti be the start time and qi be the conditional probability of failure. The esti-mated probability of surviving to ti or beyond is then

For i � 1 and, hence ti �0, the survival probability is set to 1.0 (Allison, 1995). The es-timate of the hazard function at the midpoint of each interval is calculated as

where for the ith interval, tim is the midpoint, di is the number of events, bi is the width ofthe interval, ni is the number still at risk at the beginning of the interval, and wi is the num-ber of cases withdrawn within the interval (former number of failures or non-failures areexcluded in the current interval).

The log-rank and the Wilcoxon tests were conducted for the rank tests for homogene-ity. In the absence of strata, the log-rank and Wilcoxon statistics are calculated as

where dj is the number of failures at the jth time point, wj is the expected number of failures,and cj � 1 for the log-rank test, and cj � nj, the number at risk of failure at the jth time pointfor the Wilcoxon-Gehan-Breslow test. The test statistic U2/V, where V is the variance ofU, is compared to the �2 distribution with 1 degree of freedom (Thisted, 2005). The log-rank test places more weight on later survival times and conversely the Wilcoxon test placesmore weight on early survival times.

An advantage of survival analysis is that by constructing hazard models of student pop-ulations, the risk of leaving engineering can be compared between particular groups (e.g.,gender or ethnicity) and the risk of this hazard function can be estimated by semester. Themodels and the tests can thus be used to study the relative risk of different groups leavingengineering majors and they lend themselves to addressing a variety of questions such as:

357


TABLE 5Frequency of Leaving Engineering by Cohort Groups

TABLE 6Frequency of Leaving Engineering by Gender

TABLE 7Frequency of Leaving Engineering by Ethnicity

Are students more at risk of leaving engineering during particular stages? Does the profileof risk differ among groups? Which score better predicts the risk of leaving engineering be-tween SAT verbal score and math score?

Censoring occurs when an individual does not experience the event of interest in the time of the study. In this paper, a student is regarded as censored if he or she does notexperience the event of interest (i.e., leaving engineering) in the time period 1987-2005.Another advantage of survival analysis is that the data on those students still in engineeringare incorporated in the analysis, rendering the result more current. Thus cohorts can be in-cluded that have not had six years to graduate within the time period of the study.

RESULTS

Preliminary Descriptive StatisticsTo better understand the overall behavior of selected groups, the frequency (percent-

age) of engineering students leaving engineering was calculated for each variable and the

358


TABLE 9Frequency of Leaving Engineering by SAT Verbal Score Group

TABLE 8Frequency of Leaving Engineering by SAT Math Score Group

results are displayed in Tables 5 to 9. For the groups of cohorts, there were no significantdifferences among the first three groups, but the frequency of failure for the latest group(2003-2004) was reduced by approximately 18% according to Table 5, reflecting a largernumber of students still in engineering but not yet graduated.

The frequency of White failure is the highest and the frequency of Other failure is thelowest. The higher SAT math score the lower the frequency of failure, whereas groupswith SAT verbal scores of � 600, and of 600 to 800 have no significant frequency differ-ences of failure as shown in Tables 8 and 9. The positive relationship between the SATmath score and non-failure is not surprising, but the insensitivity of survival to SAT verbalscore and even a slight decrease when comparing the 700 SATV � 750 and 750 SATV � 800 groups is not obvious. This finding, however, is consistent with our earlierwork (Zhang et al., 2004) and work by Humphreys and Freeland (1992), and suggests thatstudents with high verbal scores are not sufficiently challenged in engineering or are morelikely to develop a preference for a non-engineering major through enrollment in generaleducation courses.

Nonparametric Survival Analysis ResultsThe main focus of this study was to use nonparametric survival analysis to better under-

stand the time-related retention relationships between different cohort groups, gender,ethnicity, and SAT math and verbal score groups. Table 10 lists a variety of life-table sur-vival estimates by interval, including number of failures, number of non-failures, condi-tional probability of failure, failure probability, non-failure probability, and hazard rate for

359


TABLE 10Life-table Survival Estimates for Entire Population (100,179)

the population. The number of non-failures represents the number of censored observa-tions in that interval, and for this study it is the number of students in their last semester ofstudy. For example, the time interval [8, 9) shows these estimates using the accumulatedevents that occurred during the eighth semester. During this semester 617 engineering students changed majors to a non-engineering degree program or left the university. The10,820 non-failures in this interval represent students entering an engineering programduring the last semester of their study either because of graduation or no record exists ofcontinuation. The 133 non-failures in the first interval, representing students entering anengineering program during their only and last semester of study, [0, 1), needs explanation.If the first term on campus was the summer, this interval then includes the students whowere censored in the .33 or .67 semesters. For example the students entering in the 2005summer will be censored since the database does not extend beyond that period. It is notedin Table 10 that intervals [2, 3) and [4, 5) have a number of non-failures (9,543 and 5,971);approximately double that of the succeeding or intervals. This is a result of including thelater cohorts that have only a limited number of semester data. Thus the majority of thestudents in the 2004 cohort will be censored in the second semester and the 2003 cohort inthe fourth semester.

The conditional probability of failure is an estimate of the probability that a student willleave engineering in the semester interval, given that the student made it to the start of theinterval. This estimate is calculated as (number of failed) / (effective sample size) (Allison,1995). For the first interval, the effective sample size is (total sample size � number ofnon-failures / 2). For the last interval, the effective sample size is (number of failures �number of non-failures / 2). For the other intervals, effective sample size is (total samplesize - former number of failures - former number of non-failures - current number of non-failures / 2). The formula to calculate the effective sample size is a fundamental property ofthe life-table method. The method treats any cases censored within an interval as if theywere censored at the midpoint of the interval. This treatment is equivalent to assumingthat the distribution of censoring times is uniform within the interval. Since censored casesare only at risk for half of the interval, they only count for half in figuring the effective sam-ple size and hence the factor of 2. The non-failure probability column is the life-table esti-mate of the survival function, that is, the probability that the event occurs at a time greaterthan or equal to the start time of each interval. For example, the estimated probability thata student will not leave an engineering major until the second semester or later is .9924.The hazard rate listed in the last column of this table gives an estimate of the hazard func-tion at the midpoint of each interval and gives the rate of failure during a particular intervalfor the population that has survived to this interval (semester).

The survival probability, S (ti), is plotted as a function of semester in Figure 1 and repre-sents the probability of surviving until semester ti or later. It is noted that the first semesterdatum is plotted on the abscissas of this figure as .5 and subsequent semesters incrementedby one since the midpoint of the semester is the time at which the ‘failure’ event occurs.The non-failure probability is initially 1 when all students are in the study and stepwise de-creases until it reaches 69.88% at the end of the twelfth semester. This latter numbershould not be confused with the six-year graduation rate. The survival probability for i �12 (i.e., [11, 12)) is the probability of a student surviving through twelve semesters, whichincludes not only engineering graduates but also those still enrolled.

The plot of survival probability estimates shown in Figure 1 is a slowly decreasing func-tion of semester. Figure 2 shows the negative log of survival function for the population,which is useful for comparing two or more groups since the negative log of survival

360


361


FIGURE 1. Survival function for first-time-in-college students matriculating in engineer-

FIGURE 2. Negative log of survival probability for first-time-in-college students matricu-lating in engineering.

function amplifies differences. This plot also provides an empirical check of the appropri-ateness of the underlying exponential model for the survival data. If the exponential modelis appropriate, the curve should be approximately linear (SAS Institute Inc., 2008), and thecurve in Figure 2 is reasonably linear beyond the first two semesters. To identify periods ofparticularly high risk for loss of engineering students, the hazard function is considered. Aplot of the hazard function for the total population is shown in Figure 3. The hazard of en-gineering failure is .76% in the first semester of study indicating that .76% of students arelikely to fail in the first semester of study. This rate increases to 5.68% in the second semes-ter and peaks at 6.74% in the third semester, indicating that engineering college studentsare most likely to fail in their third semester. The hazard then decreases until it reaches1.62% in the eighth semester and then remains relatively constant. It is not surprising thatthe likelihood of loss from engineering is highest in the early semesters. During this periodengineering majors are primarily engaged in the common core math, physics and chem-istry courses and thus often only minimally exposed to and excited about an engineeringcareer. Furthermore, this period often reveals poor preparation of some students for col-lege. The high risk observed in the early terms strongly suggests early intervention strate-gies are essential. This could be more critical given that advising policies (e.g., major trans-fer protocol, grade forgiveness, and probation) might delay the occurrence of the eventrelative to that of related events. Recent trends of introducing engineering early in the cur-riculum, building learning communities, and using active learning should be helpful.

An analysis was performed to examine differences among the cohort groups of thisstudy. The survival function for each cohort group was compared using two tests for ho-mogeneity, the log-rank test, which places more weight on longer survival times, and the

362


FIGURE 3. Hazard function for first-time-in-college students matriculating in engineering

Wilcoxon test, which places more weight on early survival times. The test results for thelog-rank �2(4) � 7.95, p � .0933 and Wilcoxon �2(4) � 19.18, p � .0007 indicate thatthere are no significant differences among the survival rates among the cohort groups forlonger survival times but there are significant differences among the rates for early survivaltimes. There were no significant differences among the data shown in Figure 4 represent-ing the cohort groups from 1987 to 2002. It is noted that the 1999 to 2002 cohort group’snon-failure probabilities by interval are lower than those of other cohort groups, which isdifferent from the result of frequency comparison in Table 5. This illustrates that careshould be taken when comparing simple frequency results to reported retention rates. Fig-ure 5 shows the hazard rate for the 5 different groups of cohort and the hazard rates aresimilar with the behavior of hazard rate for the entire population. The plot for the2003–2004 group shows a more erratic behavior of hazard rate because this group is small-er and more heavily weighted by enrolled students.

The next comparison was made on the basis of gender. The log-rank �2(1) = 247.64, p �0.0001 and Wilcoxon �2(1) � 293.59, p �.0001 tests indicate that there is a significant dif-ference between the survival curves for gender in semesters 3 to 5. Figure 6 shows the nega-tive log of the survival functions for gender, which shows that female students are more like-ly to leave engineering earlier than male students. The frequency of female failure is largerthan that of male by 5% (see Table 6). Other work using MIDFIELD has found a narrowergender gap in persistence (Lord et al., 2008), but a larger gap is consistent with the observa-tion that women have a higher risk of leaving engineering in semesters 2 through 4. In asample such as this one, in which the later cohorts are present in the study for a shorter dura-tion, women are more likely to leave engineering during the study, whereas men who arestill enrolled in engineering at the end of the study period may still leave engineering after

363


FIGURE 4. Negative log of survival probability by cohort group.

364


FIGURE 5. Hazard functions by cohort group.

FIGURE 6. Negative log of survival probability by gender.

365


FIGURE 7. Hazard functions by gender.

FIGURE 8. Negative log of survival probability by ethnicity.

the study period. This is an example of the advantage of using survival analysis in thatknowledge of the time dependence of risk of leaving engineering by gender allowed inter-pretation of the frequency data. The hazard rate in Figure 7 shows that both males and fe-males have the highest hazard rate in the third semester and that the hazard rate of femalesis higher than that of males in semesters 3 to 5. Thus early intervention programs are impor-tant for both groups, but the potential for impact is greater for females. It is noted that thehazard rate for the earlier semesters include the more recently matriculated students, andalong with the cohort analysis suggests that this trend has persisted for a long time.

As another non-cognitive factor based on individual characteristics, ethnicity was ex-amined. The log-rank �2(3) � 270.88, p � .0001 and Wilcoxon �2(3) � 309.32, p �.0001 tests indicate that there are significant differences among the survival curves for eth-nicity as graphically displayed in Figure 8. The order in tendency to leave engineering isWhite students slightly greater than Minority students Asian Other students. In thisstudy, Other students include international students, which demonstrate the highest non-failure probability behavior. Figure 9 shows the hazard rate for each ethnic group. It is seenthat the highest hazard rate is again in the third semester for each ethnic group. It is notedthat the hazard function for the Other category increases in the ninth semester while thatfor Minority students remains high in the ninth semester and beyond. This is possibly re-lated to financial considerations for both international students in those students in theMinority categories as traditional support mechanisms become more limited. Ishitani andDesJardins (2002–2003) examined the timing of dropout over a five-year period and foundthat factors that affect student dropout often have effects that change over time. Their re-sults showed that students who receive financial aid generally have lower dropout ratesthan non-aided students and also dropout rates vary depending on the amount and timingof student financial aid.

Differences between the cognitive factors of verbal and quantitative SAT scores wereevaluated using survival analysis. The student test scores were grouped in bins of 50 pointincrement, except that a single bin was used for scores between 200 and 500. This parti-tioning yielded a reasonably large number of students in each bin. The log-rank �2(6) �1063.95, p � .0001 and Wilcoxon �2(6) � 1106.07, p � .0001 tests indicate that there aresignificant differences among the survival curves for SAT math score groups as shown inFigure 10. As expected the students who have lower math scores tend to leave engineeringmore than the students with higher math scores. The hazard rates shown in Figure 11 forSAT math score illustrate that the value of the maximum hazard rate is ordered inversely asthe math score range. This strong inverse correlation with SAT math score emphasizes theimportance of mathematics preparation for success in engineering and is consistent withour previous finding that the SAT math score is a statistically significant predictor of suc-cess (Zhang et al., 2004). The maximum value of the hazard rate for the two lowest groups(i.e., �550) occurs in the second semester, whereas the students who have scores equal toor more than 550 have the highest hazard rate in the third semester. Since the core or re-medial math courses typically commence in the first semester, it is not surprising that weakmath skills impact the hazard rate earlier and more significantly. It is noted that differencesin the hazard rate diminish with time and are small for the eighth semester and beyond.

Figures 12 and 13 show estimates of the survival and hazard functions, respectively,for SAT verbal scores. The log-rank �2(6) = 79.35, p � .0001 and Wilcoxon �2(6) �77.76, p � .0001 tests indicate that there are also significant differences among thesurvival curves among the SAT verbal score groups. Interestingly, students with verbalscore between 200 and 500 are more likely to be non-failures than students whose

366


367


FIGURE 9. Hazard functions by ethnicity.

FIGURE 10. Negative log of survival probability by SAT math score group.

368


FIGURE 11. Hazard functions by SAT math score group.

FIGURE 12. Negative log of survival probability by SAT verbal score group.

SAT verbal score is between 500 and 600. Compared with survival functions for SATmath score groups, the magnitude of differences among SAT verbal score group isconsiderably smaller than those among SAT math score group. Unlike the math scoreresults, the maximum value of the highest hazard rate for each group occurs in thethird semester. The range of maximum hazard rate for SAT verbal scores between 750and 800, however, is narrower than that for the math scores between 600 and 650, butaverage values are similar. The smaller and not systematic variation between the SATverbal score range indicates that verbal skills do not impact retention in engineering asmuch as math skills. It is interesting that the ranges show larger differences for greaternumber of semesters.

CONCLUSIONS

The nonparametric survival analysis was adopted to analyze the rate at which under-graduate students are lost from engineering as a major. The main questions for this studyare: Does the profile of risk of students leaving engineering differ among cohorts andgroups with different cognitive factors (SAT math and SAT verbal scores) and non-cogni-tive individual characteristics (gender and ethnicity)? When are students most likely toleave engineering as a major? Is SAT score a good predictor of the risk of leaving engineer-ing? The key findings to these and related questions are summarized below:

• A comparison of cohort groups indicates that there are no significant differences forlonger survival times, but there are significant differences for early survival times,whereas for groups based on gender, ethnicity, SAT math and SAT verbal scores,there are significant differences during the entire time.

369


FIGURE 13. Hazard functions by SAT verbal score group.

370


• Females show a higher risk of leaving engineering in semesters 3 to 5 than males,while the risks are similar during other semesters.

• White students tend to leave engineering slightly more than Minority students,which leave engineering more than Asians, which leave engineering more thanOther students. The Minority and Other categories show an increase in hazard rate(i.e., increased risk of leaving engineering) for the ninth semester and beyond, possi-bly related to financial or other pressures.

• Except for groups with SAT math scores greater than 550, engineering college stu-dents have the highest hazard rate during the third semester, which in part may dueto probationary periods offered in earlier periods.

• SAT math score better predicts the risk of ‘failure’ than SAT verbal score. That is,the lower a student’s SAT math score the more likely that student is to leave engi-neering.

• Engineering college students with an SAT verbal score between 200 and 500 areslightly more likely to survive than the student whose SAT verbal is between 500and 600. A plausible explanation is that students with higher SAT verbal scores aremore likely than those with lower SAT verbal scores to switch to disciplines that de-mand good verbal skills. A follow up qualitative study designed to probe deeper intothis interesting phenomenon is warranted.

The results of this study support using survival analysis to better understand factors thataffect student success. Since student retention is a dynamic problem, survival analysis ex-plicitly includes time as a variable. This allows characteristics such as risk of leaving engi-neering to be evaluated by semester, giving insight to when intervention programs mightbe most needed and most effective. Compared with preliminary statistics such as frequen-cy, nonparametric survival analysis is useful in the analysis of time-to-event longitudinaldata, as was demonstrated for analysis of cohort groups in which simple comparison of fre-quency was misleading. The methodology also allows treatment of retention data that lieoutside the range of the typical six-year graduation period, rendering studies more current.Thus information on students who only recently matriculated in engineering could be in-corporated into the survival function for early semesters.

ACKNOWLEDGMENT

The authors would like to acknowledge the support of the NSF ROLE / STEP IIaward 0337629 / 0729596 “Studies using the Multiple-Institution Database for Investi-gating Engineering Longitudinal Development (MIDFIELD).” The technical support ofErica Hughes at the University of Florida is greatly appreciated.

REFERENCES

Allison, P. D. (1995). Survival analysis using SAS: A practical guide. Cary, NC: SAS Institute

Inc.

Arulampalam, W., Naylor, R. A., & Smith, J. P. (2004). Factors affecting the probability of

first year medical student dropout in the UK: A logistic analysis for the intake cohorts of

1980–92. Medical Education, 38(5), 492–503.

Arulampalam, W., Naylor, R. A., & Smith, J. P. (2007). Dropping out of medical school in the

UK: Explaining the changes over ten years. Medical Education, 41(4), 385–394.

Cox, D. R. (1972). Regression models and life-tables. Journal of the Royal Statistical Society, Se-

ries B (Methodological), 34(2), 187–220.

DesJardins, S. L., Ahlberg, D. A., & McCall, B. P. (1999). An event history model of student

departure. Economics of Education Review, 18(3), 375–390.

DesJardins, S. L., Ahlberg, D. A., & McCall, B. P. (2006). The effect of interrupted enroll-

ment on graduation from college: Racial, income, and ability differences. Economics of Edu-

cation Review, 25(6), 575–590.

DesJardins, S. L., & Moye, M. J. (2000). Studying the timing of student departure from College.

Paper presented at the Association for Institutional Research 40th Annual Forum, Cincin-

nati, OH.

French, B. F., Immekus, J. C., & Oakes, W. (2005). An examination of indicators of engineer-

ing students’ success and persistence. Journal of Engineering Education, 94(4), 419–425.

Huff, K. L., & Fang, D. (1999). When are students most at risk of encountering academic dif-

ficulty? A study of the 1992 matriculants to U.S. medical school. Academic Medicals, 74(4),

454–460.

Humphreys, S., & Freeland, R. (1992). Retention in engineering: A study of freshman Cohorts.

University of California at Berkeley. Retrieved from http://www.eecs.berkeley.

edu/~humphrys/retention-eng.pdf

Ishitani, T. T., & DesJardins, S. L. (2002–2003). A longitudinal investigation of dropout from

college in the United States. Journal of College Student Retention, 4(2), 173–201.

Kaplan, E. L., & Meier, P. (1958). Nonparametric estimation from incomplete observations.

Journal of the American Statistical Association, 53(282), 457–481.

Klein, J. P., & Moeschberger, M. (2003). Survival analysis: Techniques for censored and truncated

data, 2nd ed. New York, NY: Springer.

Leslie, L. L., McClure, G.T., & Oaxaca, R. L. (1998). Women and minorities in science and

engineering: A life sequence analysis. The Journal of Higher Education, 69(3), 239–276.

Lord, S. M., Brawner, C. E., Camacho, M. M., Layton, R. A., Long, R. A., Ohland, M. W.,

& Wasburn, M. H. (2008, October). Work in progress — Effect of climate and pedagogy

on persistence of women in engineering programs. Proceedings of the 38th ASEE/IEEE

Frontiers in Education Conference, Saratoga Springs, NY.

Morita, J. G., Lee, T. W., & Mowday, R. T. (1989). Introducing survival analysis to organiza-

tional researchers: A selected application to turnover research. Journal of Applied Psychology,

74(2), 280–292.

Murtaugh, P. A., Burns, L. D., & Schuster, J. (1999). Predicting the retention of university stu-

dents. Research in Higher Education, 40(3), 355–371.

National Research Council. (2007). Rising above the gathering storm: Energizing and employing

America for a brighter economic future. Washington, DC: National Academies Press.

Padilla, M. A., Zhang, G., & Anderson, T. J. (2005, June). Drawing valid inferences from the

nested structure of engineering education data: Application of a hierarchical linear model to

the SUCCEED longitudinal database. Proceedings of the 2005 American Society for Engineer-

ing Education Annual Conference and Exposition, Portland, OR.

371


Plank, S., DeLuca, S., & Estacion, A. (2005). Dropping out of high school and the place of career

and technical education: A survival analysis of surviving high school. National Research Center

for Career and Technical Education, University of Minnesota. Retrieved from

http://136.165.122.102/UserFiles/File/pubs/DroppingOut- Plank.pdf

Public Service Human Resources Management Agency of Canada. (2000). Job mobility and

promotion in the federal public service.

Ronco, S. L. (1994). Meandering ways: Studying student dropout with survival analysis. Pro-

ceedings of the Association for Institutional Research 34th Annual Forum, New Orleans, LA.

Singer, J. D., & Willett, J. B. (1991). Modeling the days of our lives: Using survival analysis

when designing and analyzing longitudinal studies of duration and the timing of events.

Psychological Bulletin, 110(2), 268–290.

Smith, J. P., & Naylor, R. A. (2001). Dropping out of university: A statistical analysis of the

probability of withdrawal for UK university students. Journal of Royal Statistical Society,

164(2), 389–305.

SAS Institute Inc. (2008). SAS/STAT 9.2 User’s Guide: Introduction to Survival Analysis Proce-

dures, Cary, NC: SAS Institute.

Tamada, M., & Inman, C. (1997). Survival analysis of faculty retention data: How long do they

stay? Paper presented at the Association for Institutional Research 37th Annual Forum,

Orlando, FL.

Thisted, R. (2005). STATA Data Analysis and Statistical Software: Why do STATA and SAS differ in

the results that they report for the stratified generalized Wilcoxon test for time-to-event data? Re-

trieved from http://www.stata.com/support/faqs/stat/wilcoxon.html

Tinto, V. (1975). Dropout from higher education: A theoretical synthesis of recent research.

Review of Educational Research, 45(1), 89–125.

Tinto, V. (2006). Research and practice of student retention: What next? Journal of College Stu-

dent Retention, 8(1), 1–19.

Visser, A., & Hanslo, M. (2006). The predictive validity of a university admissions test: A survival

analysis approach. Paper presented at the 32nd Annual Conference of International Associa-

tion for Educational Assessment, Singapore.

Willett, J. B., & Singer, J. D. (1991). From whether to when: New methods for studying stu-

dent dropout and teacher attrition. Review of Educational Research, 61(4), 407–450.

Zhang, G., Anderson, T. J., Ohland, M. W., & Thorndyke, B. R. (2004). Identifying factors

influencing engineering student graduation: A longitudinal and cross-institutional study.

Journal of Engineering Education, 93(4), 313–320.

AUTHORS

Youngkyoung Min is a manager and senior researcher in the Korea Foundation for theAdvancement of Science and Creativity (KOFAC), 509 Yeoksamno, (960-12 Daechi3-Dong,) Gangnam-Gu, Seoul 135-847, Korea; [email protected].

Guili Zhang is an assistant professor of Research Methodology in the Department ofCurriculum and Instruction, East Carolina University, 232 Speight Building, Greenville,NC 27858; [email protected].

372


Russell A. Long is director of Project Assessment in Purdue University’s School of En-gineering Education, 500 Central Drive, Potter Engineering Center, Room 270, WestLafayette, IN 47907; [email protected].

Timothy J. Anderson is a Distinguished Professor in the Department of Chemical En-gineering and director of the Florida Energy Systems Consortium at the University ofFlorida, 311 Weil Hall, P.O. Box 116550, Gainesville, FL 32601; [email protected].

Matthew W. Ohland is associate professor in Purdue University’s School of Engineer-ing Education, 701 West Stadium Avenue, Neil Armstrong Hall of Engineering, WestLafayette, IN 47907-2045; [email protected].

373


Nonparametric Survival Analysis of the Loss Rate of ... · applied survival analysis to study a...

Documents

Transcript of Nonparametric Survival Analysis of the Loss Rate of ... · applied survival analysis to study a...