graduate GPA?

Analyzing admissions metrics as predictors of graduate GPA and whether graduateGPA mediates PhD completion

Mike VerostekDepartment of Physics and Astronomy, University of Rochester and

Department of Physics and Astronomy, Rochester Institute of Technology∗

Casey Miller and Benjamin ZwicklDepartment of Physics and Astronomy, Rochester Institute of Technology

(Dated: July 5, 2021)

An analysis of 1,955 physics graduate students from 19 PhD programs shows that undergraduategrade point average predicts graduate grades and PhD completion more effectively than GRE scores.Students’ undergraduate GPA (UGPA) and GRE Physics (GRE-P) scores are small but statisticallysignificant predictors of graduate course grades, while GRE quantitative and GRE verbal scores arenot. We also find that males and females score equally well in their graduate coursework despite astatistically significant 18 percentile point gap in median GRE-P scores between genders. A coun-terfactual mediation analysis demonstrates that among admission metrics tested only UGPA is asignificant predictor of overall PhD completion, and that UGPA predicts PhD completion indirectlythrough graduate grades. Thus UGPA measures traits linked to graduate course grades, which inturn predict graduate completion. Although GRE-P scores are not significantly associated withPhD completion, our results suggest that any predictive effect they may have are also linked indi-rectly through graduate GPA. Overall our results indicate that among commonly used quantitativeadmissions metrics, UGPA offers the most insight into two important measures of graduate schoolsuccess, while posing fewer concerns for equitable admissions practices.

I. INTRODUCTION

As physics graduate admission committees across thecountry consider eliminating GRE scores from consider-ation when evaluating applicants [1, 2], it is importantto continue examining the GRE’s ability to predict suc-cess in graduate school in order for programs to make in-formed policy choices. Although GRE scores are amongthe numeric metrics that best predict admission into U.S.graduate programs [3, 4], there are significant disparitiesin typical GRE performance between students of differ-ent demographic backgrounds [5]. Combined with thefact that Physics remains one of the least diverse of allthe STEM fields [6], the prospect that GRE tests limitthe ability of certain students to enter graduate schoolhas led researchers to begin questioning the utility ofGRE exam scores in the graduate admissions process incomparison to other quantitative metrics such as under-graduate GPA (UGPA) [1, 7, 8]. Among some of thefindings in this body of work are indications that earninghigh marks on the GRE Physics (GRE-P) test fails tohelp students “stand out” to admissions committees whowould have overlooked them due to an otherwise weakapplication [8], and that typical physics PhD admissionscriteria such as the GRE-P exam fail to predict PhDcompletion despite limiting access to graduate school forunderrepresented groups [1].

Yet overall PhD completion is only one measure of“success” in graduate school. Graduate faculty often cite

∗ [email protected]

high grades, graduation in a reasonable amount of time,and finding a job after graduation as indications of suc-cessful graduate students [9]. It is therefore crucial foradmissions committees understand how these other mea-sures of success are related to common quantitative ad-missions metrics as well. In particular, studying the roleof graduate grade point average (GGPA) is importantfor both historical and practical reasons. Among physicsgraduate students, positive relationships between GRE-Pscores, first-year graduate grades, and cumulative grad-uate grades have traditionally been touted as evidencefor the exam’s utility in evaluating applicants [10, 11].Several other studies [12–15] suggest that a number ofcommon admissions metrics are correlated with GGPAas well. At a practical level, gaining a better understand-ing of which factors best predict graduate grades is valu-able due to the fact that performance in graduate classescan influence whether students will ultimately completea PhD. For instance, programs may institute GPA re-quirements that prevent students from continuing studyif their course grades do not meet certain criteria.

Predictive validity analyses of GRE scores across allSTEM disciplines consistently find that scores on theGRE Quantitative (GRE-Q) and Verbal (GRE-V) testsare more effective predictors of graduate grades than PhDcompletion [12, 13]. For instance, recent studies on PhDadmissions in the biomedical field found that students’GRE-Q and GRE-V scores are poor predictors of PhDcompletion, but are more associated with first-semesterand cumulative graduate school grades [14, 15]. In con-trast, studies cited by the Educational Testing Service(ETS) such as the meta-analysis of GRE predictive va-lidity by Kuncel et al. [11] show a positive correlation

arX

iv:2

104.

0859

1v3

[ph

ysic

s.ed

-ph]

1 J

ul 2

021

2

between GRE subject scores, graduate grades, and PhDcompletion. Kuncel et al. find that GRE subject testsshow larger correlations with GGPA than GRE-Q, GRE-V, or UGPA, which they attribute to the subject-specificknowledge that the GRE subject tests are purported tomeasure. Still, GRE-Q and GRE-V scores, which theauthors presume to be broad measures of cognitive abil-ity, are shown to only moderately correlate with GGPAbut do not significantly correlate with PhD completion.Kuncel et al. also find undergraduate grade point average(UGPA) correlates with GGPA but not completion.

Despite voluminous research on the efficacy of quanti-tative admissions metrics in predicting graduate success,there remains a dearth of studies specifically examiningthese metrics in the context of physics graduate educa-tion. No current study elucidates the relationships be-tween undergraduate grades, GRE scores, and physicsgraduate grades. Moreover, studies such as [1] do notincorporate graduate grades into models of PhD com-pletion despite its theoretical and structural importanceon the road to graduate success. This paper aims to fillthese gaps in the current literature.

The primary goal of this paper is to extend the analy-sis of Miller et al. [1], using the same data set to exam-ine the correlations of common quantitative admissionsstatistics with graduate physics GPA, as well as the rolethat graduate GPA plays in predicting whether a studentcompletes their PhD program. Whereas [1] did not uti-lize information on student graduate course performance,this paper incorporates graduate GPA into several mod-els in order to determine whether commonly used ad-missions metrics predict PhD completion of US studentsdirectly, or indirectly via graduate GPA; a discussion ofthe theoretical motivation for why graduate grades maymediate the relationship between admissions metrics andPhD completion is offered in Section II. Hence, whilethe analysis presented in Miller et al. [1] was primar-ily focused on simply identifying the measures that bestcorrelated with PhD completion, this analysis exploresquestions regarding both how and why those correlationsoccurred.

Exploring whether graduate GPA mediates the rela-tionship between common admissions metrics and PhDcompletion affords us the opportunity to employ statisti-cal methods from the literature on causal inference [16–23]. In doing so we lay out methods of calculating thedirect and indirect effects of common admissions metricson PhD completion, as well as the assumptions neededfor those effects to have a causal interpretation. Thisapproach allows us to gain useful information from thepresent analysis, while careful examination of the as-sumptions required for causal interpretation will helpguide future studies.

Use of statistical methods developed in the causal in-ference literature allows us to build on the findings in [24]by incorporating the ranking of a student’s PhD programalong a mediating pathway to completion rather than asa covariate in regression analysis. We also present models

with various combinations of GRE-P and GRE-Q scoresto show that variance inflation due to collinearity is min-imal, and is therefore not a concern. These analyses areincluded in the Supplemental Material.

We seek to answer two primary research questions inthis paper:

1. How do commonly used admissions metrics and de-mographic factors relate to physics graduate GPA?

2. What role does graduate GPA play in predictingPhD completion, and do quantitative admissionsmetrics predict PhD completion indirectly throughgraduate GPA?

To answer these questions, we begin by exploring therelationships between variables using bivariate correla-tions. We then examine the unique predictive effectsof different admission metrics on graduate GPA usinga multiple linear regression model. These results lay thegroundwork for a mediation analysis, which is used toexamine the role that graduate GPA plays in PhD com-pletion by breaking down effects into direct and indirectcomponents. All of the primary analyses are performedusing data on US physics graduate students, with a re-view of equivalent analyses for international students in-cluded in the Supplemental Materials.

II. BACKGROUND AND MOTIVATION

Before outlining the quantitative methods employed inthis analysis, we briefly describe the student performancemetrics used in this study and the broad individual stu-dent characteristics they help to measure. We discuss theunderlying constructs hypothesized by the GRE Quan-titative, Verbal, and subject tests, as well as undergrad-uate and graduate grades, and several external factorsthat influence these scores. The GRE Analytical Writingtest is not included since it is not used enough in physicsgraduate admissions to warrant investigation. This sec-tion serves as a theoretical motivation for the models ofPhD completion analyzed in this study.

The GRE is a series of standardized tests designedto help admissions committees predict future academicsuccess of students coming from different backgrounds[25, 26]. While the GRE-Q assesses basic concepts ofarithmetic, algebra, geometry, and data analysis, theGRE-V assesses reading comprehension skills and verbaland analytical reasoning skills. These tests are specif-ically constructed to measure “basic developed abilitiesrelevant to performance in graduate studies” [27]. Intheir meta-analysis of GRE predictive validity, Kuncelet al. frame the GRE-Q and GRE-V as most related todeclarative and procedural knowledge and suggest thatthey are best described as measures of general cognitiveability [11]. In contrast, the GRE subject tests “assessacquired knowledge specific to a field of study” [25], indi-cating that the GRE subject tests are ostensibly a direct

3

measure of a student’s knowledge of a particular areaof study. Indeed, admissions committees often interprethigh GRE subject scores as strong evidence of a student’sdiscipline-specific knowledge [9]. Other research suggeststhat higher scores on standardized subject tests couldalso reflect greater student interest in that subject area[28].

The individual characteristics measured by a student’sundergraduate grades include both academic knowledgeand a collection of noncognitive factors [29]. Much re-search exists on the meaning and value of grades, par-ticularly at the K-12 level, and a review [30] of the pastcentury of grading research finds that grades assess a mul-tidimensional construct comprising academic knowledge,engagement (including motivation and interest), and per-sistence. Consistently over the past 100 years only about25% of variance in grades is attributable to academicknowledge as measured by standardized tests [31], withrecent research suggesting that much of the unexplainedvariance is represented by a student’s ability to negotiatethe “social processes” of school [32]. We therefore regardUGPA as broadly measuring student academic achieve-ment across a wide range of subjects in addition to sev-eral aspects of noncognitive traits such as motivation,interest, and work habits. However, we also recognizethe limitations inherent in compressing students’ collegeacademic performance into a single number, includingthe loss of information pertaining to student growth overtime and time to degree completion.

We conceptualize graduate GPA similarly, treating itas a measure of subject-specific academic knowledge aswell as other non-academic characteristics. In additionto the broad research on grades described above, re-search specifically addressing the factors leading to grad-uate success supports this interpretation of GGPA. In-terviews conducted with over 100 graduate school fac-ulty reveal that graduate success, which they define as astudent’s ability to earn high graduate grades and even-tually complete their degree in a timely manner, is largelydependent on noncognitive characteristics [9]. Intervie-wees deemed motivation, work ethic, maturity, and orga-nizational skills as crucial to student success in graduateschool. In a separate review of noncognitive predictorspertaining to graduate success, graduate GPA is specif-ically linked to a variety of personality (e.g. extrover-sion and conscientiousness) and attitudinal factors (e.g.motivation, self-efficacy, and interests) [33]. Indeed, theauthors of the review characterize graduate grades as acomplex composite of many of the cognitive and noncog-nitive factors related to graduate school success.

These conceptions of grades and GRE scores com-pel us to hypothesize that graduate GPA mediates therelationship between common quantitative admissionsmetrics and PhD completion. As a construct measur-ing subject-specific knowledge and several noncognitivecharacteristics, we expect UGPA and GRE-P to moststrongly link to GGPA. Despite the drawbacks of cumu-lative UGPA (such as grade inflation and masking of in-

dividual growth), we expect UGPA to be associated withgraduate course performance since it captures aspects ofboth academic and some non-academic characteristics.We also expect GRE-P scores to be related to gradu-ate grades due to their requirement of specific physicsknowledge. Finally, while we expect GRE-Q may havea small predictive effect on GGPA as a general cognitivemeasure, we do not necessarily expect a similar relation-ship for GRE-V as its content is generally disparate fromphysics curricula.

On both a theoretical and structural level we expectgraduate grades to predict PhD completion. GraduateGPA may offer insight into a student’s mastery of ad-vanced physics concepts as well as their personality andattitudes. All of these contribute to successful physicsPhD completion, but likely vary in importance depend-ing on choice of research area [33]. Structurally, a sat-isfactory performance in graduate courses is implicit onthe path to completing a PhD. For example, GGPA re-quirements can act as thresholds for being allowed tocontinue studying in a PhD program. Poor course per-formance may negatively influence personal factors (e.g.,self-efficacy, identity), limit access to research opportu-nities (e.g., repeating classes, ease of finding a researchlab), or may indicate a lack of preparation for research,all of which could hinder PhD completion. It is also moretemporally proximal to PhD completion than the othermetrics included in the study.

Lastly, we note that although this discussion has fo-cused on students’ individual traits that may predict suc-cess in graduate school, there are undoubtedly a numberof external factors that can influence student attrition.Socioeconomic factors, mental health, family responsi-bilities, work duties, external job prospects, and depart-mental culture are all variables that would play a rolein a comprehensive model of graduate school persistence[9, 34–38]. These uncollected pieces of information mayact as “confounding” variables that can bias results, andwe discuss their influence on the present study in SectionV B.

III. METHOD

A. Data

Student level data for both this study and [1] was re-quested from physics departments that awarded morethan 10 PhD’s per year for students who matriculatedbetween 2000 and 2010, including information on under-graduate GPA (UGPA), GRE-Q, GRE-V, GRE-P, andgraduate GPA (GGPA). Data collected also included thefinal disposition of students (PhD earned or not), startand finish years, and demographic information. GPAdata is analyzed on a 4.0 scale while GRE scores are onthe percentile scale.

We received data from 27 programs (approximately a42% response rate), which spanned a broad range of Na-

4

2.0 2.5 3.0 3.5 4.0

Undergraduate GPA

2.0 2.5 3.0 3.5 4.0

Undergraduate GPA

2.0 2.5 3.0 3.5 4.0

UGGPA

Gender Male Female

US

Non-US

2.0 2.5 3.0 3.5

Graduate GPA

2.0 2.5 3.0 3.5 4.0

Graduate GPA

0 25 50 75 100

GRE Physics Percentile

0 25 50 75 100


50 60 70 80 100

GRE Quantitative Percentile

50 60 70 80 90 100


0 25 50 100

GRE Verbal Percentile

0 25 50 75 100

GRE Verbal Percentile

US

Non-US

US

Non-US

US

Non-US

US

Non-US

Figure 1: Distributions of the quantitative metrics included in the data. “Raincloud plots” show density plots,boxplots, and scatterplots of the data. We see that despite significant score gaps between US male and femaleGRE-P test takers, no such gap exists in subsequent GGPA performance. UGPA distributions for male and femaleapplicants are also similar. Code for generating raincloud plots courtesy of [39]. All figures generated with the Rpackage ggplot2 [40]. Figure themes adapted from [41].

5

US Non-US Total

Male 1638 1164 2802

Female 317 287 604

Total 1955 1451 3406

Table I: Demographic breakdown of the data used inthis analysis. To focus on issues of diversity andinclusion most strongly associated with US applicantsto physics graduate programs, we analyze only the datafrom US graduate students.

tional Research Council (NRC) rankings. The sampleused in [1] consisted of all students in 21 programs forwhich start year was available. Given that the mediantime to degree across physics PhD programs is 6 years,some students who started before 2010 were still activeat the time of data collection in 2016. The probabil-ity of not completing the physics PhD has an exponen-tial time dependence with a time constant of 1.8 years.Thus, students who have been in their programs for threetime constants have only a 5% chance of not completing.These students were thus categorized as completers inthis study.

These data covered 3962 students. Of this subset, twoprograms did not report GGPA data for their students.Hence, the sample for this study excludes these students,thereby reducing the sample size to 3406 students across19 programs. This corresponds to approximately 11%of matriculants to all U.S. physics PhD programs duringthe years studied.

Among the sample of US students, 16% are women(N = 317). Although the authors generally advocate fora nuanced treatment of gender in physics education re-search and recognize the deficits associated with treatinggender as a fixed binary variable [42], the present dataset spans the years 2000 to 2010 during which the datacollected by programs only allowed for the binary op-tion of male/female. Hence, we must treat gender as adichotomous variable in this analysis.

The racial composition of the dataset is 61.6% White,1.3% Black, 2.1% Hispanic, 0.2% Native American, 3.5%Asian, 1.0% multiple or other races, and 30.2% undis-closed. Excluding the cases for which race was unavail-able, the sample is thus roughly representative of annualPhD production in U.S. physics for gender, race, and cit-izenship [43]. We include race as a covariate in each anal-ysis presented; however small N , particularly for Black,Hispanic, Native American, and Asian students, oftenprecludes useful interpretation of the results pertainingto these subsets.

In order to focus on issues of diversity and inclusion as-sociated most strongly with US applicants, we use onlythe subset of data from domestic graduate students. Thisdecision is further motivated by research suggesting thatit is difficult for admission committees to directly com-pare scores earned by US and international students, in-

dicating that separate analyses are appropriate [9]. Usingthe subset of students who are from the US reduces thetotal sample size for the study to N = 1955. A cursoryvisualization of the variables in the data set, as shownin Figure 1, shows that the distributions of scores forUS and Non-US students are markedly different, whichfurther justifies separate analyses of these two studentpopulations.

Examining the distributions of scores in Figure 1, thepresence of non-normality is evident in nearly all of thevariables. Each of the continuous variables in the datasetfail the Shapiro-Wilk test of normality at the α level of.05. However, these tests are often of limited usefulness;in general distributions with skewness |γ1| > 3 or kurtosis|γ2| > 10 likely indicate that they violate any assumptionof normality [44]. For this dataset, the GGPA distribu-tion skewness γ1 = −3.31 and kurtosis γ2 = 20.43, in-dicating severe non-normality. The GRE-Q distributionalso falls into the problematic range (γ1 = −2.11 and

ˆγ2 = 9.33). Ceiling effects are also present, since manystudents earned 4.0 grade point averages or earned themaximum score on the GRE examinations.

The data collection process was limited to gatheringonly cumulative graduate GPA rather than first-yeargraduate grades, which were not recorded by some pro-grams. Thus, depending on whether a student persistedin a program, their graduate GPA may be based on manycourses while others are based on only a few courses. Thedata set is also necessarily subject to range restriction,since data on student performance in graduate school isautomatically limited to include only students who wereaccepted to undertake graduate study. We cannot knowhow students who were not accepted into graduate schoolwould have performed had they been accepted. Range re-striction may act to attenuate the strength of observedeffects in subsequent analyses [45].

Although not used in a majority of this study, webriefly explore the role of the doctoral programs’ NRCranking in PhD completion [46]. Since the NRC onlygives confidence intervals for program rank, we createda ranking for this study by averaging the 5 and 95%confidence bounds for the NRC regression-based rank-ing (NRC-R) and rounded this up to the nearest fiveto protect the confidentiality of participating programs.This led to a ranking range of 5 to 105. We divided theprograms into terciles of approximately equal number ofrecords, and categorized as Tier 1 (highest ranked, NRC-R ≤ 20), Tier 2 (25 ≤ NRC-R ≤ 55), and Tier 3 (NRC-R> 55).

Multiple imputation (predictive mean matching) isused to impute missing UGPA and GRE-P scores. Pre-dictive mean matching is used due to the non-normalityof the data. 160 students do not have data for eitherUGPA or GRE-P, while 400 are missing only UGPA and263 are missing only GRE-P. All multiple imputationis conducted using the mice package in R [47]. 20 im-puted data sets are used for each analysis. For consis-tency, incomplete variables are imputed using the same

6

imputation model used in [1], in which the imputationmodel utilizes all other variables in the data set asidefrom graduate GPA and PhD completion. The modelutilizes GRE-Q, GRE-V, program tier, gender, and race,as well as complete cases of UGPA and GRE-P. Althoughthe imputation approach presented here is theoreticallysound, we also present a comparison of several differentmodels of data imputation in the Supplemental Materi-als.

B. Methods to explore the role of graduate grades

The goal of this section is twofold. First, we seekto gain a cursory look at how graduate GPA is relatedto common admissions metrics. In doing this, we alsowish to determine whether it is reasonable that admis-sions metrics could indirectly predict completion throughgraduate GPA. This section presents a series of analysesmeant to elucidate the relationships between standardadmissions metrics, students’ GGPA, and students’ fi-nal disposition. To make our analysis maximally acces-sible to readers of different statistical backgrounds, wedescribe here in detail the methods used in this section.

Bivariate correlation coefficients provide informationabout the level of association between two variables, andare therefore a useful starting point for analysis. Weconstruct a correlation matrix (see Table II) for all vari-ables in the sample using Pearson correlation coefficients,which are equivalent to the standardized slope coeffi-cients for a linear model predicting y from x. These aregiven by

rxy =

∑(xi − x)(yi − y)√∑

(xi − x)2√∑

(yi − y)2(1)

for any two continuous variables x and y. Calculatingrxy gives us a first glance at the relationships betweenthe continuous variables UGPA, GGPA, GRE-P, GRE-Q, and GRE-V.

When the x variable is treated as dichotomous (e.g.,gender in this data set), Eq. (1) reduces to the point-biserial correlation coefficient rpb,

rpb =y1 − y0σy

√pq, (2)

where y1 and y0 are the means of the continuous y vari-able for the two x groups 1 and 0, q and p are the propor-tions of data belonging to these two groups, and σy is thestandard deviation for the y variable. Like the Pearsoncoefficient, the quantity rpb ranges from -1 to 1 and indi-cates the strength of association between two variables.Conveniently, a significance test for the point-biserial cor-relation is identical to performing an independent t-teston the data [48]. Thus, the point-biserial correlation coef-ficient yields information about whether two group meansare statistically different. For instance, the point-biserialcorrelation tests whether the GGPA of male students are

statistically different from those of female students (wefind that GGPAs are not significantly different by gender,see Table II).

When x and y are both dichotomous, the Pearson co-efficient reduces to the phi coefficient,

φ =

√χ2

n, (3)

where χ2 is the chi-squared statistic for a 2x2 contin-gency table and n is the total number of observations inthe data. The phi coefficient also ranges from -1 to 1 andindicates the strength of association between two binaryvariables. This quantity allows us to examine whether fi-nal disposition is significantly associated with gender (wefind that the association between gender and final dispo-sition just meets the threshold for statistical significance(φ = −0.05 ± 0.04, p = 0.04), likely due to the largesample size of our data, but the very small phi coefficientindicates that the practical strength of this relationshipis negligible [49]).

To characterize how GGPA and other numerical pre-dictors vary across program tier, we conduct several one-way analysis of variance (ANOVA) tests using programtier as the independent variable. ANOVA tests allowus to determine whether there are significant differencesbetween different groups, such as students in differentprogram tiers. These tests produce an F -statistic, whichis interpreted as the ratio of between-group variabilityto within-group variability. Thus, higher values of F in-dicate that between-group variability is large comparedto within-group variability, which is unlikely if the groupmeans all have a similar value.

Lastly we present the results of a multiple regressionanalysis in which we regress GGPA on common admis-sions metrics and demographic factors. Regression allowsus to examine the unique predictive effects of these pre-dictors.

The classical linear regression model is written math-ematically for an outcome variable Y as

Yi = α1Xi1 + α2Xi2 + ...+ αkXik + εi, (4)

where i = 1, ..., n, the number of observations in the data,and k represents the number of predictors in the model.Error terms εi are assumed to be independent and nor-mally distributed with mean 0 and standard deviation σ.α is the vector of regression coefficients that minimizesthe sum of squared errors

Σni=1 = (Yi − αXi)

2 (5)

for the given data. The regression coefficients can beinterpreted as the difference in the outcome variable Y ,on average, when comparing two groups of units thatdiffer by 1 in one predictor X while keeping all the otherpredictors the same.

We report both unstandardized and standardized ver-sions of the regression coefficients. Unstandardized coeffi-

7

M

X Y

C

𝛽𝛽!

𝛾𝛾!𝛽𝛽"

𝛽𝛽#

Independent Variable

(ex. usingactive learning

activities)

Covariate(ex. prior

subject interest)

Mediator(ex. studentengagement)

Dependent Variable

(ex. exam grades)

𝛾𝛾"

Figure 2: A qualitative graphical depiction of amediated relationship between two variables. X, Y andM represent the model’s independent, dependent, andmediating variables while C represents a covariate. Aresearcher who observes a positive relationship betweenusing active learning activities in the classroom and theexam grades of students might posit that a thirdvariable, student engagement, is actually responsible forcausing the observed relationship.

cients are the result of regression analysis using the origi-nal, unscaled variables. Thus, the unstandardized regres-sion coefficients represent the predicted average changein the outcome variable Y when the corresponding pre-dictor X is changed by one unit. This allows for astraightforward interpretation since the variables are notscaled, but does not yield insight into the relative predic-tive strengths of the independent variables since they arescaled differently. Standardized regression coefficients re-sult from regression analyses using continuous variablesthat have been mean-centered and divided by their stan-dard deviation, resulting in variables with variances equalto 1. Thus standardized regression coefficients representthe average number of standard deviations changed in theoutcome variable when a predictor variable is increasedby one standard deviation. By calculating the standard-ized coefficients, we exchange a simple interpretation ofscore change for an interpretation of which variables havethe greatest effect on the dependent variable.

C. Mediation analysis methods

Using mediation analysis we seek to answer the ques-tion of whether graduate GPA mediates the predictiveability of common admissions metrics on PhD comple-tion. Whereas analyses such as logistic regression [1]yield information about whether independent variablessuch as UGPA and GRE-P affect final disposition of agraduate student, they do not offer insight into the ex-planation of why and how UGPA, GRE-P, and other ad-missions metrics affect completion. Mediation analysisis one technique that allows us to probe the underlyingprocess by which some variables influence others [19–23].

Figure 2 graphically depicts a prototypical mediationmodel, where X, Y and M represent the model’s inde-pendent, dependent, and mediating variables. C repre-sents a covariate. As a hypothetical example, let’s saythat previous research has shown a positive relationshipbetween the use of active learning activities in physicsclass and student exam grades. Researchers might hy-pothesize that this relationship is actually due to a thirdmediating variable, student engagement. Using activelearning activities in class may cause students to becomemore engaged with the material, making their subsequentexam grades increase. Engagement is a mediating vari-able in this case. Meanwhile, since students who are al-ready interested in physics could be predisposed to beingmore engaged and performing better on exams, the re-searcher might take students’ prior physics interest intoaccount by including it as a covariate in their analyses.

In this section, we wish to discern whether graduateGPA mediates the relationship between common admis-sions metrics and students’ likelihood of completing grad-uate school. In practice, mediation analysis is done bysimultaneously estimating a set of regression equations[50]. The goal is to partition the total effect of the inde-pendent variable X on the dependent variable Y into twoparts: the direct effect of X on Y and the indirect effectof X on Y through the mediating variable M . For thesimple example given above, the set of regression equa-tions to be solved are:

Yi = β0 + β1Mi + β2Xi + β3Ci + εyi (6)

Mi = γ0 + γ1Xi + γ2Ci + εmi. (7)

Traditional mediation literature [51, 52] defines the di-rect effect of X on Y as the coefficient β2 and the indirecteffect of X on Y as the product of coefficients γ1β1, cor-responding to the products of the path coefficients alongthe mediated path shown in Figure 2. Statistically signifi-cant values of γ1β1 indicate that the relationship betweenX and Y is mediated by M . This method demonstratesthe general intuitive ideas underlying mediation analysis,but is subject to several important limitations. Foremostamong the limitations associated with traditional media-tion analysis is that its applicability to model categoricalvariables (e.g., binary outcomes) and nonlinearities is notwell defined, as these situations preclude the use of sumsand products of coefficients [19, 53, 54]. The difficultiesassociated with binary outcomes are therefore problem-atic for a model predicting final disposition, a binary out-come. Furthermore, traditional mediation models leavethe causal interpretation of their results ambiguous [55].

Recent work in the field of causal inference [16–18] hasformalized and generalized mediation analysis to resolvethese limitations, allowing for categorical outcomes whilealso clarifying that under certain conditions the resultsmay be interpreted causally. In this framework, oftencalled the “potential outcomes” framework, the tradi-tional product-of-coefficients mediation analysis is a spe-cial case for which the mediator and outcome variables

8

are both continuous, while the functional forms of the di-rect and indirect effects for other situations become morecomplicated [50].

For the primary analysis of this paper, we calculatethe direct and indirect effects defined by the potentialoutcomes framework for the case of a continuous inde-pendent variable (UGPA, GRE-P, and GRE-Q), a con-tinuous mediator (GGPA), and a dichotomous outcome(final disposition). The simultaneous regression equa-tions to be calculated are still Eqs. (6) and (7), exceptthe binary Y is replaced with Y ∗, a continuous unob-served latent variable which represents the observed bi-nary variable. Once estimated, the direct and indirecteffects reduce to simple differences in probability of com-pleting a PhD between students across different valuesof the independent variables. Mathematically, the effectsfor a change in the independent variable from a value x0to x1 at a particular value of the control c are given by

IE = Φ[probit(x0, x1)]− Φ[probit(x0, x0)], (8)

DE = Φ[probit(x1, x1)]− Φ[probit(x0, x1)], (9)

where Φ represents the normal cumulative distributionfunction and probit(xa, xb) is given by

probit(xa, xb) = [β0 + β2xa + β3c+

β1(γ0 + γ1xb + γ2c)]/√v(xa), (10)

and v(xa) is

v(xa) = β21σ

2m + 1. (11)

Note that these expressions are all still simply combina-tions of the coefficients from the regression equations (6)and (7). Thus, the potential outcomes framework allowsus to calculate the total predicted change in probabilityof completing a PhD due to an independent variable anddecompose it into that variable’s indirect effect on finaldisposition through GGPA as well as its direct effect (seeFigure 4).

Using this mediation framework can help to give pow-erful insights into nuanced relationships between the vari-ables in an observational study [56]. However, giving atruly causal interpretation to the results of this mediationanalysis requires a set of strong assumptions to be met,and in practice it can be difficult for any observationalstudy fully meet these conditions [57]. Hence, we try toavoid making explicitly causal claims in our discussionof the results. We discuss the assumptions needed for acausal interpretation as well as the robustness of the cur-rent study to violations of those assumptions in SectionV B.

Mediation analyses were conducted using themediation package in R [58]. Checks for the consistencyof results across different computational approacheswere done by performing duplicate mediation analysesin the R package medflex [59] as well as the statisticalsoftware Mplus [60].

2.0

2.5

3.0

3.5

4.0

Tier 3 Tier 2 Tier 1

Tier

Gra

duat

e G

PA

Final Disposition No PhD PhD

2.0

2.5

3.0

3.5

4.0


Tier

Gra

duat

e G

PA

Gender Male Female

0

25

50

75

100


Tier

GR

E Ph

ysic

s Per

cent

ile

Gender Male Female

(a)

(b)

(c)

Figure 3: (a) Graduate GPA by program tier and finaldisposition. Students who do not complete a PhD earnlower graduate grades than students who complete theirprograms (rpb = 0.43, p < .001). Tier 3 students tend toearn slightly lower graduate course grades than Tier 1or 2 students. (b) Graduate GPA by program tier andgender. Male and female students earn similar graduategrades (rpb = −0.01, p = 0.57, equivalent to anon-significant independent t-test), and this trend holdsacross program tier. (c) GRE Physics by program tierand gender. Across all program tiers there is asignificant gap in scores between male and femaleGRE-P test takers (rpb = −0.30, p < .001, equivalent toa statistically significant independent t-test). Scoredistributions trend upward for higher tier programs.

9

Measure (M + SD) UGPA GRE-Q GRE-V GRE-P GGPA Final Disp. Gender

UGPA (3.6 ± 0.3) – (0.25, 0.35) (0.12, 0.22) (0.26, 0.37) (0.24, 0.33) (0.10, 0.20) (-0.11, 0.01)

GRE-Q (83.3 ± 10.4) 0.30 – (0.30, 0.37) (0.47, 0.54) (0.13, 0.22) (0.10, 0.18) (-0.16, -0.07)

GRE-V (76.3 ± 18.7) 0.17 0.33 – (0.23, 0.32) (0.06, 0.15) (0.02, 0.10) (-0.02, 0.07)

GRE-P (52.9 ± 23.2) 0.31 0.51 0.28 – (0.18, 0.27) (0.10, 0.19) (-0.33, -0.24)

GGPA (3.5 ± 0.5) 0.29 0.18 0.10 0.22 – (0.39, 0.46) (-0.06, 0.03)

Final Disp. 0.15 0.15 0.06 0.14 0.43 – (-0.09, -0.01)

Gender -0.05 -0.11 0.03 -0.30 -0.01 -0.05 –

Table II: A matrix showing bivariate correlations between continuous and dichotomous variables used in subsequentanalyses. Correlations are shown in the lower diagonal while confidence intervals for those correlations are shown inthe upper diagonal. For example, the correlation between GGPA and GRE-P is 0.22 and a 95% CI of (0.18, 0.27),indicating a weak correlation. Means and standard deviations are also presented in the first column. GPAs are on a4.0 scale while GRE scores are in terms of percentiles. Correlations are calculated for US students only.

IV. RESULTS

A. Results of exploring the role of graduate grades

Correlations An initial question related to predict-ing a student’s final disposition is whether UGPA, GREscores, and GGPA are reliably correlated with a student’sfinal outcome. We are also interested in the strengthof association between GGPA, UGPA, and GRE scores,as this information yields insight into whether GGPAcould serve as a mediating variable in predicting finaldisposition. Table II contains the bivariate correlations(Pearson, point-biserial, and phi) between each pair ofmeasures for the sample in the lower diagonal. The 95%confidence intervals are reported in the upper diagonal.Confidence intervals that do not include a value of zeroindicate that the correlation is statistically significant.The means and standard deviations of the continuousvariables are presented in the table’s first column (GPAdata is analyzed on a 4.0 scale while GRE scores are onthe percentile scale).

Inspection of Table II reveals that GGPA is the predic-tor most strongly correlated with final disposition (rpb =0.43). This value is statistically significant (p < .001) andpositive, meaning that students with higher GGPA aremore likely to finish their PhD program successfully. Thistrend is visually apparent in Figure 3(a), which showsboxplots of GGPA grouped by PhD completers and non-completers. We also observe that UGPA (rxy = 0.29,p < .001), GRE-Q (rxy = 0.18, p < .001), and GRE-P (rxy = 0.22, p < .001) are all positively correlatedwith GGPA, albeit weakly, meaning that students withhigher scores in these metrics tend to earn higher GG-PAs. Taken together, the observation that higher UGPAand GRE scores positively correlate to GGPA, which inturn correlate with a student’s likelihood of completion,implies that GGPA might play an important role in me-diating the influence of these admissions metrics on PhDcompletion.

The lack of a statistically significant correlation be-tween gender and GGPA indicates that the disparity in

scores on the GRE-P between males and females does notmanifest itself in subsequent GGPA performance. In-deed, there is no statistical difference between averageGGPA for males and females (rpb = −0.01, p = 0.57,equivalent to a non-significant independent t-test), asdemonstrated in Figure 3(b). Yet there exists a sta-tistically significant difference between males and fe-males in GRE-P performance in our data (rpb = −0.30,p < .001, equivalent to a statistically significant indepen-dent t-test). Thus GGPA does not differ between gen-ders despite the known performance gap between malesand females on the GRE-P exam. Furthermore, thephi coefficient measuring the association between genderand PhD completion is negligible despite barely meet-ing the threshold of statistical significance (φ = −0.05,p = 0.04).

Still, the bivariate correlations shown in Table II do notcontrol for possible relationships between the variables ofinterest. For instance, the moderate correlation betweenGRE-Q and GRE-P (rxy = 0.51, p < .001) indicatesthat there may be a spurious relationship between one ofthese variables and GGPA. In addition, we observe lowbut statistically significant correlations between UGPAand GRE-P (rxy = 0.31, p < .001), GRE-Q (rxy = 0.30,p < .001), and GRE-V (rxy = 0.17, p < .001). This isexpected, as UGPA likely contains some information re-garding the specific aspects of students’ aptitudes testedby GRE exams. These results motivate the use of mul-tiple regression analysis later in this section in order todisentangle the unique effects of each independent vari-able on GGPA. That analysis reveals that when we iso-late the unique predictive effects of each variable in theregression model, UGPA and GRE-P remain significantbut weak predictors of GGPA, while GRE-Q does notretain statistical significance.

Similarly, although UGPA (rpb = 0.15, p < .001),GRE-P (rpb = 0.14, p < .001) and GRE-Q (rpb = 0.15,p < .001) are positively correlated with PhD comple-tion, the magnitudes of these correlations are very weakand do not account for other parameters that may beassociated with completion. Multivariate approaches al-

10

lows us to isolate how individual metrics relate to PhDcompletion, which we explore in the mediation analy-sis presented in Section IV B. Consistent with previousstudies of PhD completion using multivariate approaches[1], we find that when accounting for other parameters,only UGPA remains a statistically significant predictorof completion.

Results of one-way independent ANOVA tests showthat the main effect of program tier on GGPA is signif-icant, F (2, 1949) = 26.31, p < .001, which reflects theupward trend in GGPA from Tier 3 to Tier 1 and 2 pro-grams. A Tukey post hoc test reveals that the GGPAwas significantly higher for students at Tier 2 (M = 3.59,SD = 0.40, p < .001) and Tier 1 (M = 3.60, SD = 0.42,p < .001) institutions than those at Tier 3 institutions(M = 3.41, SD = 0.59). There was no statistically sig-nificant difference between the Tier 1 and Tier 2 groups(p = 0.97).

Multiple Regression To disentangle the unique ef-fects of each predictor on GGPA we conduct a multiplelinear regression analysis. Multiple linear regression al-lows us to simultaneously fit many independent variablesto measure each of their relative effects on a single de-pendent variable, GGPA. Analyzing the raw coefficientsfitted by the regression analysis yields insight into thepredicted change in GGPA due to changes in one vari-able while holding all others constant. Standardized co-efficients allow for a comparison of the relative effect sizesof the independent variables.

Our model includes all available GRE scores (GRE-P, GRE-Q, and GRE-V) as well as UGPA in order toexamine the unique predictive effects of each measure.We considered the possibility that including both GRE-P and GRE-Q in the same model would raise collinearityconcerns, but find these concerns unfounded. The bi-variate correlation (rxy = 0.51) between the two is nothigh enough to warrant genuine concern [61, 62]; fur-thermore, the variance inflation factor (VIF) for everyimputed dataset’s regression model was below 1.75, wellbelow the commonly cited threshold of 10. Hence, wedeem the model posed in the study as most appropri-ate to answer the research questions raised in this study(further discussion regarding collinearity concerns in thedata is available in the Supplemental Material).

The results obtained in our analysis are summarizedin Table III. Significant predictive effects at the 95%threshold were found for the numerical metrics UGPA(β = 0.24, t = 8.88, p < .01) and GRE-P (β = 0.15,t = 5.02, p < .01). Students with higher UGPA andGRE-P scores therefore tend to receive higher GGPAs.Among statistically significant predictors, the higheststandardized coefficient is UGPA, which is larger thanthe GRE-P coefficient by approximately 50%.

The regression model predicts that for a 0.10 score in-crease in UGPA, GGPA is expected to increase on av-erage by 0.035 points, holding all other predictors fixed.Meanwhile, a 10 percentile increase in GRE-P score is as-sociated with a 0.031 point increase in GGPA on average,

Multiple Regression Results (*p < .05, **p < .01)

Dependent Variable - GGPA

Independent

Variable

Coefficient

(Standard Error)

Standardized

Coefficient (β)

Intercept 1.92** (0.15) -0.06

UGPA 0.35** (0.04) 0.24**

GRE-P 31 × 10−4** (6 × 10−4) 0.15**

GRE-Q 16 × 10−4 (12 × 10−4) 0.03

GRE-V 3 × 10−4 (6 × 10−4) 0.01

Black -0.11 (0.09) -0.23

Hispanic -0.01 (0.07) -0.02

Native Am. 0.01 (0.23) 0.03

Asian -0.02 (0.06) -0.05

Other -0.24* (0.11) -0.51*

Undisclosed 0.07** (0.02) 0.15**

Gender 0.06* (0.03) 0.13*

N 1955

Adjusted R-Squared 0.11

Table III: Coefficients of a multiple regression analysismodeling graduate GPA as a function of commonquantitative admissions metrics. Reference categoriesare White for race and Male for gender.

again holding other predictors fixed.A significant predictive effects was found for gender

(β = 0.13, t = 2.11, p = .04). The positive β coef-ficient indicates that if a female student and male stu-dent have the same admissions scores, the female stu-dent would earn a higher grades in her graduate classes.We include race as a parameter in the analysis as a cat-egorical variable, though all groups except White hadsmall N (the most students in the dataset identified asWhite, N = 1205, while the fewest identified as NativeAmerican, N = 4). None of the race categories werestatistically significant parameters except “Other.”

Notably the GRE-Q coefficient is not statistically sig-nificant, suggesting that it has little predictive effect ongraduate course performance. Yet in [1], GRE-Q wasseen to link to overall completion. This observation,along with those made above regarding the predictive ef-fects of UGPA and GRE-P on GGPA, motivate the medi-ation analysis in the following section. Some predictors,namely GRE-Q, may link more directly to PhD comple-tion, while others such as UGPA and GRE-P may indi-rectly predict completion through their effect on GGPA,which itself links to completion.

B. Mediation analysis results

Having demonstrated that graduate GPA (GGPA) iscorrelated with several quantitative admissions metricsof interest as well as final PhD completion, we now turn

11

to the results of a mediation analysis to determine the ex-tent to which GGPA mediates the relationship betweenthese variables. As described previously, these resultsyield insight into whether better performance in under-graduate coursework and on GRE examinations increasea student’s likelihood of completing a PhD program di-rectly or indirectly via GGPA. Over the course of per-forming these analyses we explored numerous mediationmodels using different combinations of covariates (e.g.gender and race) to explore their effect on the results.For instance, we probed the effects of including thesevariables as moderators in the analysis to account forvarying predictive effects among different demographics.However, results were consistent regardless of how thesecovariates were included in the model.

We perform a separate mediation analysis for each pre-dictor variable (UGPA, GRE-P, and GRE-Q). For eachanalysis, we begin by calculating the direct and indirecteffects of each metric on PhD completion using Eqs. (8)and (9). We also calculate the total effect, which is thesum of the direct and indirect effects. The total effect iscomparable to the result one would obtain by using lo-gistic regression to predict changes in probability of PhDcompletion as was done in Miller et al. [1].

This procedure requires us to choose both a control andtreatment value for the admissions metrics, since the out-put of the mediation analysis is the predicted differencein probability of PhD completion if a student who earnedthe control score had earned the treatment score instead.

Table IV shows the results of a mediation analysis in

Mediation analysis results

for score changes from 25th to 75th percentiles

Independent

Variable

Direct

Effect

Indirect

Effect

Total

Effect

UGPA

-0.001

(-0.033, 0.034)

p = 0.97

0.060

(0.037, 0.084)

p < 0.01

0.060

(0.030, 0.090)

p < 0.01

GRE-P

-0.006

(-0.047, 0.035)

p = 0.78

0.042

(0.023, 0.063)

p < 0.01

0.037

(-0.002, 0.075)

p = 0.08

GRE-Q

0.024

(-0.009, 0.060)

p = 0.13

0.009

(-0.007, 0.023)

p = 0.26

0.034

(-0.004, 0.068)

p = 0.07

Table IV: Results of mediation analyses that show thepredicted change in probability of PhD completion dueto changes in admissions metrics from their 25thpercentile values to their 75th percentile values amongstudents in the study. The predicted total effect ofUGPA is to increase a student’s overall probability ofPhD completion by 6.0% (p < 0.01). That effect isentirely attributable to the indirect effect of UGPA oncompletion through graduate GPA. The total effectsassociated with GRE-P and GRE-Q are not statisticallysignificant.

which we calculate the predicted change in probability ofPhD completion if a student had earned a 3.90 under-graduate GPA rather than a 3.46 undergraduate GPA.This change corresponds to a shift from the 25th to 75thpercentile of undergraduate GPAs in our data. We ob-serve that the predicted total effect of this change is toincrease the student’s overall probability of PhD com-pletion by 6.0% (p < 0.01). That effect is entirely at-tributable to the indirect effect of UGPA on PhD com-pletion through graduate GPA, since the direct effect isestimated to be -0.1% and is not statistically significant(p = 0.97), while the indirect effect is estimated to be6.0% and is statistically significant (p < 0.01). Mean-while, mediation analysis using GRE-P as the predictorvariable estimates that due to a change from the 25thto 75th percentile of GRE-P scores among students inthis study, the probability of PhD completion increasesby 3.7% (this change corresponds to a shift in GRE-P percentile ranking from 35 to 71 among the overallphysics test-taking population). This result just missesthe threshold of statistical significance (p = 0.08). How-ever, any effect that may be associated with a higherGRE-P score is attributable to the indirect effect of GRE-P on GGPA, which is estimated to be 4.2% and is statisti-cally significant (p < 0.01). For a change in GRE-Q scorefrom the 25th to 75th percentile of scores among studentsin this study (a change in GRE-Q percentile ranking from79 to 91 among the overall physics test-taking popula-tion), the direct (2.4%, p = 0.13) and indirect (0.9%,p = 0.26) predictive effects on PhD completion are notstatistically significant. Their sum, the total effect, esti-mates an increase in PhD completion probability of 3.4%,and like the total effect of GRE-P also just misses thethreshold for statistical significance (p = 0.07).

To examine how the predicted probability of PhD com-pletion changes over a broad range of UGPA and GREscores, we repeat this single mediation analysis for arange of treatment values, as suggested in [22]. We opt tochoose our data’s median values as the baseline controlvalue of each metric, then compute the direct and indi-rect effects for a variety of treatment values with respectto this baseline. The result is a plot of predicted PhDcompletion probability change as each score is varied.Figure 4 graphically summarizes the results of the medi-ation analysis. Figures 4a), c), and e) display the resultsof the three separate analyses predicting PhD comple-tion probability changes relative to the median value ofeach independent variable. Points on each plot representindividual calculations of probability change relative toeach admission metric’s median value. Hence, the me-dian value of the x-axis variable is clearly shown on theplot as the point where the direct (red) and indirect effect(blue) lines intersect, corresponding to a total probabilitychange of 0%.

Percent probability changes due to direct effects ofeach admissions metric on PhD completion are shownin red, while indirect effects on PhD completion trans-mitted through GGPA are shown in blue. Plots of the

12

��

��

��

��

��

��

��

��

−15

−10

−5

0

5

10

65 70 75 80 85 90 95 100


% P

roba

bilit

y C

hang

e R

elat

ive

to M

edia

n Sc

ore

�� Total Effect10% 25% 75% 90%

Predicted changes in PhD completion probability

��

��

��

��

��

��

��

��

��

��

��

��

−15

−10

−5

0

5

10

65 70 75 80 85 90 95 100


% P

roba

bilit

y C

hang

e R

elat

ive

to M

edia

n Sc

ore

�� Direct Effect Indirect Effect10% 25% 75% 90%

��

��

��

��

��

��

��

��

��

��

��

��

−15

−10

−5

0

5

10

0 25 50 75 100


% P

roba

bilit

y C

hang

e R

elat

ive

to M

edia

n Sc

ore


��

��

��

��

��

��

��

��

��

��

��

−15

−10

−5

0

5

10

0 25 50 75 100


% P

roba

bilit

y C

hang

e R

elat

ive

to M

edia

n Sc

ore

�� Total Effect10% 25% 75% 90%

��

��

��

��

��

��

��

��

��

��

��

��

−15

−10

−5

0

5

10

3.0 3.2 3.4 3.6 3.8 4.0

Undergraduate GPA

% P

roba

bilit

y C

hang

e R

elat

ive

to M

edia

n Sc

ore


��

��

��

��

��

��

��

��

��

��

��

−15

−10

−5

0

5

10

3.0 3.2 3.4 3.6 3.8 4.0

Undergraduate GPA

% P

roba

bilit

y C

hang

e R

elat

ive

to M

edia

n Sc

ore

�� Total Effect10% 25% 75% 90%

a) b)

c) d)

e) f)

( (

( (

( (

Figure 4: Mediation analysis results predicting a student’s change in probability of completion, split into direct(red) and indirect via graduate GPA (blue) effects. Points on each plot represent individual calculations ofprobability change relative to each admission metric’s median value. 95% confidence intervals are indicated by theshaded region around each line; hence, if the shaded region contains the line y = 0, the effect is not statisticallysignificant at the α = .05 level. The median value of the x-axis variable is clearly shown on the plot as the pointwhere the direct and indirect effect lines intersect, corresponding to a total probability change of 0%. The totaleffect of UGPA as well as the indirect effects associated with UGPA and GRE-P are statistically significant acrossall magnitudes of score change.

total effect of each variable on PhD completion proba-bility, which is the sum of the direct and indirect effects,are shown in Figures 4b), d), and f). The shaded ribbonsaround the lines representing the best-estimates of prob-ability change show the 95% confidence interval. Hence,if this shaded region contains the line y = 0, the effect isnot statistically significant at the α = .05 level.

In agreement with the results of the example analysisshown in Table IV, the total predictive effect of UGPA onPhD completion is statistically significant while the totaleffects of GRE-P and GRE-Q do not reach the thresholdfor statistical significance, indicated by the error bandsthat encompass the y = 0 line (Figures 4d and f). Still,as suggested by the result in the Total Effect column of

13

Current Study Previous Study (Ref. [1])

Independent

Variable

Total Effect

Value

(Probability Scale)

Predicted Change in

Completion Probability

(25th-75th Percentile)

Ref. [1]

Coefficient

(Log Odds Scale)

Predicted Change in

Completion Probability

(25th-75th Percentile)

UGPA 0.060 (p < 0.01) 6.0% 0.60 (p < 0.01) 3.6%

GRE-P 0.037 (p = 0.08) 3.7% 5 × 10−3 (p = 0.09) 3.1%

GRE-Q 0.034 (p = 0.07) 3.4% 10 × 10−3 (p = 0.04) 2.1%

Table V: Comparison of results from the current study to results presented by Miller et al. [1] in which the authorspredict PhD completion using a logistic regression model. Despite using different statistical methods and omittingprogram tier as a predictor in our model, we observe that the results obtained in this study are qualitativelyconsistent with those reported previously. In both cases, UGPA is the strongest predictor of PhD completion amongadmissions metrics tested.

Table IV for GRE-P (p = 0.08) and GRE-Q (p = 0.07),these effects are close to reaching statistical significance;for reference, each point in Figure 4d) the GRE-P confi-dence ribbon surpasses the line y = 0 by less than 0.5%.Thus these results provide some evidence that scoringmore highly on the GRE-P and GRE-Q are positivelyassociated with higher rates of completion, although theeffects are not statistically significant.

Also consistent with the results of the example analy-sis shown in Table IV the results of the three mediationanalyses indicate that predictive effects of UGPA andGRE-P on a student’s PhD completion are entirely me-diated by GGPA, defined by the fact that the indirecteffect of these admissions metrics are statistically signif-icant across all magnitudes of score change while theirdirect effects are not. These indirect effects are shownin Figure 4 by the blue lines in the UGPA and GRE-Pplots, whose error ribbons do not contain the line y = 0.The interpretation of this result is that UGPA effectivelypredicts a student’s GGPA, which in turn predicts PhDcompletion. Similarly, any increase in PhD completionprobability associated with increases in GRE-P scoresare a result of the indirect effect through graduate GPA,although the total effect is not statistically significant.

With regard to GRE-Q, given the weak relationshipbetween GRE-Q and GGPA revealed by the multiple re-gression analysis in Section IV A, it is unsurprising thatthe indirect effect shown in blue on Figure 4 is nearlyzero. Indeed, as indicated by the earlier results in TableIV any predictive effect from GRE-Q on completion ap-pears to stem from the direct effect of GRE-Q on PhDcompletion, although the direct effect does not achievestatistical significance at the α = 0.05 level.

Lastly, despite using different statistical methods andomitting program tier as a predictor in our model, weobserve that the results obtained in this study are quali-tatively consistent with those reported by Miller et al. [1].Table V shows a comparison of the total effects predictedby the mediation model presented here with the resultsof the logistic regression model presented in [1], again us-ing as an example the predicted changes in probabilityof PhD completion due to shifts from the 25th to 75thpercentile in score for the different admissions metrics.

In both analyses, the predictive effects associated withchanges in UGPA are statistically significant and are thelargest in magnitude among the tested metrics. Effectsassociated with changes in GRE-P scores are not statis-tically significant at the α = 0.05 level in either modelbut are close (p = 0.08 in this study and p = 0.09 in [1]).The only minor inconsistency between the two analysesis revealed in the results of the GRE-Q models. Whereasin [1] the predictive effect of GRE-Q on completion wasbarely significant (p = 0.04), it is not statistically signifi-cant here (p = 0.07). However, as demonstrated in TableV, effects associated with GRE-Q are not strong in eithermodel and both are very close to the α = 0.05 thresholdfor statistical significance, indicating that the two resultsare still approximately consistent.

V. DISCUSSION

A. Interpretation of Results

The results presented in Section IV B give new insightinto how admissions committees may contextualize theuse of quantitative admissions metrics. Clearly, no exist-ing metric provides unassailable evidence that a studentwill complete their graduate program. However, amongthe imperfect quantitative admissions metrics commonlyused by admissions committees, the consistent messagefrom this work and others is that undergraduate GPAoffers the most promising insight into whether physicsgraduate students will earn a PhD. Moreover, there isno significant difference between male and female appli-cants’ UGPAs as there is for the GRE-P, meaning that itsuse in ranking applicants is less likely to skew diversityof admitted students.

As demonstrated by the multiple regression analysesin Section IV A, UGPA is most strongly associated withgraduate course performance among the variables tested.In some ways this is an expected result: UGPA, themetric that directly measures a student’s in-class perfor-mance, is most effective at predicting future in-class per-formance in graduate school. Still, UGPA would seem to

14

vary greatly depending on the student’s particular under-graduate institution while a standardized exam like theGRE-P is consistent across all students. It is possiblethat UGPA may also be signaling socio-emotional skillssuch as achievement orientation and conscientiousness,which are known to predict high levels of performanceboth in and out of the classroom [63–66]. These sortsof socio-emotional skills are also shown in the relevantresearch literature to lack the race, gender, and cultureof origin gaps that are found on many standardized tests[67–69].

While previous work showed that UGPA was an ef-fective predictor of PhD completion, mediation analy-sis demonstrates that relationship is entirely transmit-ted through UGPA’s ability to predict GGPA. Thus, al-though UGPA is the best predictor of a student’s finaldisposition, our analysis indicates that it is not a di-rect measure of PhD completion. Rather, the observedrelationship between undergraduate grades and comple-tion is explained by the intervening variable GGPA. Re-garding the magnitude of the observed effects, we seethat changes in score from the 25th to 75th percentile inUGPA (3.46 to 3.90) are associated with an 6% increasein completion probability. Changes across a broaderrange of UGPAs from the 10th to 90th percentile (3.2to 3.98), the mediation model predicts an 11% increasein PhD completion probability.

Multiple regression and mediation analyses also yieldimproved insight into the information provided by GRE-P. Consistent with prior published work by ETS [10, 11],regression analysis reveals that GRE-P is an effective pre-dictor of graduate grades. However, the effect associatedwith UGPA is approximately 50% larger than the effectassociated with GRE-P. Regarding the relationship be-tween GRE-P scores and PhD completion, any associa-tion between GRE-P performance and PhD completion isentirely mediated by graduate course performance, sim-ilar to UGPA. This particular indirect effect indicatesthat a student who scores more highly on the GRE-P ismore likely to perform better in graduate school courses,which may slightly improve their probability of gradu-ation (although the total effect of GRE-P is not statis-tically significant). However, this indirect effect is stillsmaller than the indirect effect associated with UGPA,as illustrated in Figure 4.

Notably, despite the existence of a large gender gapin GRE-P scores (the median GRE-P percentile for fe-males is 35 and 57 for males), male and female graduatestudents earn nearly indistinguishable graduate grades(Figure 3). Moreover, there is no practical relationshipbetween gender and PhD completion (Table II). We havealso done a preliminary analysis of this same data exam-ining the relation between gender and time to PhD com-pletion, and found that no statistically significant dif-ference exists in the time it takes for male and femalephysics graduate students to complete doctoral degrees.The disparity in GRE-P scores between male and femaletest takers is therefore anomalous, as it does not appear

to be related to differences in ability or level of prepara-tion and is not reflected in subsequent graduate perfor-mance.

Results of multiple regression and mediation analy-ses that show GRE-Q is not strongly associated withincreased graduate course performance are unsurprisinggiven the GRE-Q’s task relevance is lower than subjecttests and undergraduate grades. Indeed, in its Guide tothe Use of Scores[25], ETS describes the GRE-Q as test-ing “high school mathematics and statistics at a levelthat is generally no higher than a second course in alge-bra; it does not include trigonometry, calculus or otherhigher-level mathematics.” As shown in Table IV, theassociation between GRE-Q scores and PhD completionjust misses the α = 0.05 threshold of statistical signif-icance (p = 0.07). Combined with previous studies inwhich this weak association was statistically significant[1], evidence suggests a weak relationship between GRE-Q scores and completion.

Both direct and indirect effects of GRE-Q on comple-tion were not statistically significant and therefore wecannot discern with certainty which small effect is moreimportant. Considering the case of a possible direct ef-fect between GRE-Q and completion, the low task rel-evance makes it unlikely that GRE-Q is a measure ofresearch competence or perseverance. One hypothesis isthat socioeconomic status (SES) could be confoundingthe relationship between GRE-Q, an exam consisting ofhigh school level mathematics questions, and PhD com-pletion. Students with lower SES may have fewer oppor-tunities academically and may perform worse on a stan-dardized exam like the GRE-Q. Indeed, it is estimatedthat roughly 20% of variance in standardized test scorescan be explained by SES [70] In general, previous re-search [71] indicates that SES impacts whether studentspossess the proper resources to support them should fi-nancial, health or other external circumstances make itdifficult to complete. Moreover, doctoral students fromlower social classes are more likely to experience a lowersense of belonging in graduate school, often due to theresidual financial burdens that are not mitigated by grad-uate stipends [72]. Reduced sense of belonging in grad-uate school drives lower interest in pursuing advancedcareers in the field, and ultimately a lower likelihood ofcompleting a PhD.

B. Limitations and future research

1. Assumptions in Causal Mediation Analysis

The causal effect framework laid out in Section III Crequires several assumptions to be satisfied in order foreffect estimates to be properly identified. All of the as-sumptions underlying causal mediation analysis refer to“confounding variables,” which are variables that influ-ence two other variables simultaneously, thereby causinga spurious association between them. In this section we

15

M(e.g.

GGPA)

Y(e.g.PhD)

Treatment-M

ediator

Confounder

Mediator-Outcome

Confounder

Treatment-Outcome Confounder

C4

X(e.g.

UGPA)

Mediator-Outcome Confounder affected by treatment

C3

C1

C2

Figure 5: Mediation diagram showing the relationshipof “confounding variables” C1-4 to the variables ofinterest X, Y , and M . To satisfy the underlyingassumptions of causal mediation analysis, researchersmust account for these confounding variables in theirregression models.

discuss whether it is plausible that our study has satis-fied these assumptions, as well as suggestions for futureresearchers seeking to perform similar analyses.

Although the assumptions prescribed by causal media-tion analysis may be stated in multiple ways [73, 74], wedescribe them in the manner presented by [19], who con-denses them into four requirements. These assumptions,displayed graphically in Figure 5, are that there exists (1)no unmeasured treatment-outcome confounders, (2) nounmeasured mediator-outcome confounders, (3) no un-measured treatment-mediator confounders, and (4) nomediator-outcome confounder affected by the treatment.

The final assumption is equivalent to assuming thatthere is not an alternative mediating variable for whichwe have not accounted [73]. Concerns about this assump-tion are handled using methods for multiple mediators,discussed in detail in the Supplemental Material.

Assumptions 1-3 require us to be sure we have includedall variables that could be confounding the relationshipsbetween X (UGPA, GRE-P or GRE-Q), Y (PhD comple-tion), and M (GGPA) in our regression models. Theseare shown graphically by variables C1, C2, and C3 in Fig-ure 5. Some of the C variables may influence more thanone of X, Y , and M , or may influence each other, socategorization into C1, C2, and C3 is not exclusive. Ac-counting for variables C1 and C3 correspond to assump-tions normally made in observational studies to calculatetotal effects [75], while accounting for variables labeledC2 is important specifically for the estimation of directand indirect effects.

2. Sensitivity Analysis for Violation of Assumptions

Unfortunately, as is the case in any regression analysis,whether these assumptions are met is not testable; how-

ever, there exist several ways to probe the robustness ofour findings under certain violations. In particular, sensi-tivity analyses have been developed [19, 22] to determinehow strong a confounding effect between the mediatorand the outcome would have to be in order to make sta-tistically significant effects become no longer significant.

We perform such sensitivity analyses on each of theimputed data sets used in the single mediator modelspresented in Section IV B to assess their robustness. Wethen report the results of the sensitivity analyses aver-aged across the imputed datasets.

Let R2M and R2

Y represent the proportions of originalvariances explained by the unobserved confounder for themediator GGPA and the outcome PhD completion, re-spectively. The result of a sensitivity analysis is a singlevalue representing the product R2

M × R2Y that identifies

the amount of original variance in the mediator and out-come that the confounder would have to explain in orderto make the observed effect vanish. Hence, the sensitiv-ity analysis results in a family of solutions for which theequation R2

M ×R2Y = constant is satisfied.

Our sensitivity analysis reveals that for a confounderto explain enough variance to make the indirect effect ofUGPA on GGPA vanish, the product R2

M × R2Y would

have to be, on average, 0.072 (standard deviation =0.006). For context, the variables included in the medi-ation models presented here, as a group, are able to ex-plain 11% of variance in the GGPA outcome (R2 = 0.11);for PhD completion, R2 = 0.32.

Comparing these values to those required to satisfythe equation R2

M ×R2Y = 0.072, we see that a mediator-

outcome confounder would have to be a better predic-tor of both GGPA and PhD completion than any quan-tity included in the model so far. For instance, sup-pose a mediator-outcome confounder satisfied the equa-tion R2

M ×R2Y = 0.15× 0.45 = 0.072, sufficient to nullify

the indirect effect of UGPA on completion. The relation-ship between the confounder and GGPA (R2

M = 0.15)would be larger than all other variables combined in themodel. The link between the confounder and PhD com-pletion (R2

Y = 0.45) would be larger than all other vari-ables combined in the model as well.

Since the values of R2M and R2

Y must be so high relativeto the rest of the variables in our model to nullify theindirect effect of UGPA on PhD completion, this resultappears to be fairly robust.

Results of sensitivity analyses on the indirect effect ofGRE-P on PhD completion were similarly robust: for aconfounder to explain enough variance to make the indi-rect of GRE-P on GGPA vanish, the product R2

M × R2Y

on average would have to be 0.069 (standard deviation= 0.001). This is essentially the same as the analyses onthe robustness of the UGPA result, so its interpretationis the same as above.

More details regarding the sensitivity analyses per-formed, including contour plots of R2

M × R2Y products

for which indirect effects of UGPA and GRE-P vanish,are available in the Supplemental Material.

16

3. Conceptualizing possible unmeasured confounders

Thoroughly considering variables that could conceiv-ably act as confounders in Figure 5 would benefit fu-ture researchers studying graduate admissions. Non-quantitative aspects of a student’s admission creden-tials such as letters of recommendation and prior re-search experience may be associated with PhD comple-tion, and could represent several C variables in the dia-gram. For example, prior research experience could actas a mediator-outcome confounder, labeled C2 in Figure5. A student with more research experiences prior to en-tering graduate school may be more likely to completea PhD than a student with fewer research experiences,since they have already become familiar with the expec-tations associated with scientific research and have al-ready demonstrated the motivation to pursue it indepen-dently. More research experience may also translate tobetter graduate course performance, particularly if grad-uate courses are well-aligned with the goal of preparingstudents for future research.

As discussed earlier, information on student socioeco-nomic status could be useful as well. We also see SESas potentially representing several confounding relation-ships. For instance, it could act as a treatment-outcomeconfounder, influencing both GRE scores and PhD com-pletion. It is estimated that roughly 20% of variance instandardized test scores can be explained by SES [70], soSES could be influencing performance on tests such as theGRE-Q and GRE-P. Students with lower SES may havefewer resources to support them should financial, exter-nal circumstances arise that make it difficult for them tocomplete their PhD [35, 71]. Including data on these andother possible confounders would bolster causal claims infuture analyses. Furthermore, our current models onlyexplain a small amount of the overall variance in the out-come variable, and many other factors are surely at play.

Lastly, although graduate course performance andPhD completion represent some aspects of graduateschool success (as evidenced by their inclusion in GREvalidation studies), they are certainly crude metrics. Fu-ture work should explore other outcomes as well, includ-ing success measures such as research productivity, jobattainment, or graduate student satisfaction.

VI. CONCLUSIONS

Using data visualization, regression analyses, and me-diation analyses, we investigated the role that graduateGPA plays on a physics graduate student’s path to PhDcompletion. We aimed to answer two primary researchquestions: 1) How do commonly used admissions met-rics and demographic factors relate to physics graduateGPA?, and 2) What role does graduate GPA play in pre-dicting PhD completion, and does it mediate the influ-ence of these other predictor variables on PhD comple-tion? Broadly, we find that across the dynamic range of

scores in the data, undergraduate GPA was a better pre-dictor of both graduate GPA and final disposition thanGRE scores.

Regarding the first research question of how variousadmissions metrics and demographic factors relate tophysics graduate GPA, we see that significant but weakpredictive effects at the 95% threshold were found forthe numerical metrics undergraduate GPA (β = 0.24,t = 8.88, p < .01) and GRE Physics (β = 0.15, t = 5.02,p < .01); GRE Quantitative and Verbal scores are notsignificantly associated with graduate GPA. The regres-sion model predicts that for a 0.10 score increase in un-dergraduate GPA, a student’s graduate GPA is expectedto increase on average by 0.035 points, holding all otherpredictors fixed. Meanwhile, a 10 percentile increase inGRE Physics score is associated with a 0.031 point in-crease in graduate GPA on average, again holding otherpredictors fixed. For comparison, a change in UGPAfrom the 25th to 75th percentile of scores in our datapredicts a 0.15 point increase in graduate GPA, whereasa change in GRE-P from the 25th to 75th percentile ofscores in our data predicts a 0.11 increase in graduateGPA.

We also observe that the graduate GPAs are not sta-tistically different between males and females. Hence,the statistically significant gap in performance by gen-der on the GRE Physics exam (within our data the me-dian GRE Physics percentile for females is 35 and 57 formales) does not carry over to subsequent graduate courseperformance. The large difference in performance is un-explained, yet is potentially problematic for promotingdiversity in physics graduate school. Multiple regressionanalysis did not reveal race to be a statistically signifi-cant predictor of graduate GPA, but unfortunately it isdifficult to properly interpret relationships between raceand graduate grades due to a small N . Small sample sizeprecludes useful interpretation of the results pertaining toBlack, Hispanic, Native American, and Asian students.

As to the second research question of whether grad-uate GPA mediates the influence of these other predic-tor variables on PhD completion, we find that UGPApredicts PhD completion indirectly through graduategrades. Only UGPA is a statistically significant predictorof overall PhD completion (a change in UGPA from the25th to 75th percentile of scores in our data predicts a 6%increase in PhD completion probability, p < 0.01), andthat effect is entirely attributable to the indirect effect ofUGPA on PhD completion through graduate GPA. Theindirect effect associated with UGPA on PhD completionwas statistically significant across all magnitudes of scorechanges, while the direct effect was not (see Figure 4).Thus UGPA effectively predicts graduate course perfor-mance, which is then associated with degree completion.The association between GRE-P scores and PhD com-pletion is not statistically significant (a change in GRE-P from the 25th to 75th percentile of scores in our datapredicts a 3.7% increase in PhD completion probability,p = 0.08). However, like UGPA, the indirect effect associ-

17

ated with increases in GRE-P score was also statisticallysignificant across all magnitudes of score change, mean-ing that any predictive effect that GRE-P score may haveis therefore also linked indirectly through graduate GPA.

Although these models explain some of the variance instudent outcomes (the variables included in the media-tion models, as a group, explain 11% of variance in grad-uate GPA (R2 = 0.11); for PhD completion, R2 = 0.32),much of the variation lies in factors outside the models,in both unmeasured student characteristics prior to ad-mission and unmeasured aspects of the graduate studentexperience.

No standardized test measures the research and projectmanagement skills it takes to successfully complete amulti-year research project, yet those are the skills thatare so highly valued in PhD graduates. The GRE-P uti-lizes two-minute theoretical physics problems to ascer-tain aspects of students’ physics knowledge, but it ne-glects the broad range of computational and experimen-tal skills used in contemporary physics research. Becauseundergraduate GPA reflects a mix of courses that includetheory, experiment, computation, and in some cases re-search projects, it could be a more useful measure ofresearch-relevant skills. However, our result that UGPAonly indirectly predicts PhD completion seems to indi-cate those research-relevant skills are not a major part ofthe overall UGPA. Identifying a broader set of applicantcharacteristics that predict graduate student outcomes isessential.

By better understanding and improving graduate ed-ucation, we have the opportunity to meet societal goalsof a highly-skilled advanced STEM workforce that re-flects the diversity of our society. While adjusting admis-sions practices may offer some improvements by adjustingwhich students are allowed to undertake graduate study,such efforts do nothing to improve the graduate studentexperience and train graduate students more effectivelyfor STEM careers. The potential for innovation and im-provement within graduate education is large and is thearea deserving substantial increased attention for educa-tion research and programmatic implementation. Whileour study gives admissions committees greater insightinto how and why various quantitative scores link to com-pletion, our discussion of the limitations also points to ar-eas where future researchers can build. We encourage thecontinued study of not only the physics graduate admis-sions process, but also the ongoing experience of studentsin PhD programs, how they are taught, mentored, andsupported through their growth as individuals within alarger scientific community.

ACKNOWLEDGMENTS

The authors are pleased to acknowledge valuable dis-cussions with Nicholas Young, Julie Posselt, and RachelSilvestrini. This work was supported by NSF grants1633275 and 1834516.

[1] C. W. Miller, B. M. Zwickl, J. R. Posselt, R. T. Sil-vestrini, and T. Hodapp, Typical physics Ph.D. admis-sions criteria limit access to underrepresented groups butfail to predict doctoral completion, Science Advances 5,eaat7550 (2019).

[2] GRE requirements & admissions fees for US/CanadianAstronomy & Physics Programs, Available athttps://docs.google.com/spreadsheets/d/

19UhYToXOPZkZ3CM469ru3Uwk4584CmzZyAVVwQJJcyc/

edit#gid=0 (2021).[3] G. Attiyeh and R. Attiyeh, Testing for Bias in Graduate

School Admissions, The Journal of Human Resources 32,524 (1997).

[4] J. R. Posselt, T. E. Hernandez, G. L. Cochran, and C. W.Miller, Metrics first, diversity later? making the short listand getting admitted to physics phd programs, Journalof Women and Minorities in Science and Engineering 25(2019).

[5] C. Miller and K. Stassun, A test that fails, Nature 510,303 (2014).

[6] NSF - Science and Engineering Doctorates: (SEDInteractive), Available at https://www.nsf.gov/

statistics/2016/nsf16300/digest/ (2021).[7] E. M. Levesque, R. Bezanson, and G. R. Tremblay,

Physics GRE Scores of Prize Postdoctoral Fellows in As-tronomy (2015), arXiv:1512.03709 [physics.ed-ph].

[8] N. T. Young and M. D. Caballero, Physics Graduate

Record Exam does not help applicants “stand out”,Physical Review Physics Education Research 17, 010144(2021).

[9] M. Walpole, N. W. Burton, K. Kanyi, and A. Jacken-thal, Selecting successful graduate students: in-depth in-terviews with GRE users, ETS Research Report Series2002, 10.1002/j.2333-8504.2002.tb01875.x (2002).

[10] L. M. Schneider and J. B. Briel, Validity of the GRE:1988-1989 Summary Report, Tech. Rep. (1990).

[11] N. R. Kuncel, S. A. Hezlett, and D. S. Ones, A com-prehensive meta-analysis of the predictive validity of thegraduate record examinations: implications for graduatestudent selection and performance., Psychological bul-letin 127, 162 (2001).

[12] S. L. Petersen, E. S. Erenrich, D. L. Levine, J. Vigoreaux,and K. Gile, Multi-institutional study of GRE scores aspredictors of STEM PhD degree completion: GRE getsa low mark, PloS one 13, e0206570 (2018).

[13] N. R. Kuncel, S. Wee, L. Serafin, and S. A. Hezlett, Thevalidity of the Graduate Record Examination for mas-ter’s and doctoral programs: A meta-analytic investi-gation, Educational and Psychological Measurement 70,340 (2010).

[14] J. D. Hall, A. B. O’Connell, and J. G. Cook, Predictorsof student productivity in biomedical graduate school ap-plications, PLoS One 12, e0169121 (2017).

[15] L. Moneta-Koehler, A. M. Brown, K. A. Petrie, B. J.

18

Evans, and R. Chalkley, The limitations of the GRE inpredicting success in biomedical graduate school, PloSone 12, e0166742 (2017).

[16] J. M. Robins and S. Greenland, Identifiability and Ex-changeability for Direct and Indirect Effects, Epidemi-ology 3, 143 (1992), publisher: Lippincott Williams &Wilkins.

[17] J. Pearl, Causality: Models, Reasoning, and Inference(Cambridge University Press, 2000).

[18] J. Pearl, Direct and indirect effects, in Proceedings ofthe Seventeenth conference on Uncertainty in artificialintelligence, UAI ‘01 (Morgan Kaufmann Publishers Inc.,San Francisco, CA, USA, 2001) pp. 411–420.

[19] T. VanderWeele, Explanation in Causal Inference: Meth-ods for Mediation and Interaction (Oxford UniversityPress, 2015).

[20] A. F. Hayes, Introduction to Mediation, Moderation, andConditional Process Analysis: A Regression-Based Ap-proach (Guilford Press, 2013).

[21] J. Pearl, The causal mediation formula - a guide to theassessment of pathways and mechanisms, Prevention Sci-ence: The Official Journal of the Society for PreventionResearch 13, 426 (2012).

[22] K. Imai, L. Keele, and D. Tingley, A general approachto causal mediation analysis., Psychological Methods 15,309 (2010).

[23] L. Valeri and T. J. Vanderweele, Mediation analysis al-lowing for exposure-mediator interactions and causal in-terpretation: theoretical assumptions and implementa-tion with SAS and SPSS macros, Psychological Methods18, 137 (2013).

[24] C. W. Miller, B. M. Zwickl, J. R. Posselt, R. T. Sil-vestrini, and T. Hodapp, Response to comment on “Typ-ical physics Ph.D. admissions criteria limit access to un-derrepresented groups but fail to predict doctoral com-pletion”, Science Advances 6, 10.1126/sciadv.aba4647(2020).

[25] GRE Guide to the Use of Scores, Available at https:

//www.ets.ord/gre/guide (2021).[26] C. Wendler and B. Bridgeman, The Research Foundation

for the GRE Revised General Test: A Compendium ofStudies (2014).

[27] J. Briel, K. O’Neill, and J. Scheuneman, GRE Techni-cal Manual (Educational Testing Service, Princeton, NJ,1993).

[28] W. W. Willingham, J. M. Pollack, and C. Lewis, Gradesand Test Scores: Accounting for Observed Differences,Journal of Educational Measurement 39, 1 (2002).

[29] A. J. Bowers, What do Teacher Assigned Grades Mea-sure? A one page research summary (2016).

[30] S. M. Brookhart, T. R. Guskey, A. J. Bowers, J. H.McMillan, J. K. Smith, L. F. Smith, M. T. Stevens, andM. E. Welsh, A Century of Grading Research: Meaningand Value in the Most Common Educational Measure,Review of Educational Research 86, 803 (2016).

[31] A. J. Bowers, What’s in a grade? The multidimen-sional nature of what teacher-assigned grades assess inhigh school, Educational Research and Evaluation 17,141 (2011).

[32] A. J. Bowers, Reconsidering grades as data for decisionmaking: more than just academic knowledge, Journal ofEducational Administration 47, 609 (2009).

[33] P. C. Kyllonen, A. M. Walters, and J. C. Kaufman, Therole of noncognitive constructs and other background

variables in graduate education, ETS Research ReportSeries 2011, i (2011).

[34] L. M. Owens, K. Shar, B. M. Zwickl, and C. W. Miller,Student Deficits vs. Sense of Belonging: Exploring Fac-ulty and Student Perspectives on Retention in PhysicsGraduate Programs (2020).

[35] L. M. Owens, B. Zwickl, S. Franklin, and C. Miller, Mis-aligned Visions for Improving Graduate Diversity: Stu-dent Characteristics vs. Systemic/Cultural Factors, inPhysics Education Research Conference 2018, PER Con-ference (Washington, DC, 2018).

[36] S. K. Gardner, Fitting the mold of graduate school: Aqualitative study of socialization in doctoral education,Innovative Higher Education 33, 125 (2008).

[37] B. E. Lovitts, Leaving the ivory tower: The causes andconsequences of departure from doctoral study (Rowman& Littlefield Publishers, 2002).

[38] National Academies of Sciences, Engineering, andMedicine and others, Graduate STEM education for the21st century (National Academies Press, 2018).

[39] M. Allen, D. Poggiali, K. Whitaker, T. R. Marshall, andR. A. Kievit, Raincloud plots: a multi-platform tool forrobust data visualization, Wellcome Open Research 4, 63(2019).

[40] H. Wickham, ggplot2: Elegant Graphics for Data Analy-sis (Springer-Verlag New York, 2016).

[41] K. Desiraju, RPubs - ggplot theme for publication readyfigures, Available at https://rpubs.com/Koundy/71792.

[42] A. L. Traxler, X. C. Cid, J. Blue, and R. Barthelemy,Enriching gender in physics education research: A bi-nary past and a complex future, Physical Review PhysicsEducation Research 12, 020114 (2016).

[43] U.S. Department of Education, Institute of EducationSciences, National Center for Education Statistics, In-tegrated Postsecondary System (IPEDS), Available athttps://nces.ed.gov/ipeds/ (2021).

[44] R. B. Kline, Principles and Practice of Structural Equa-tion Modeling, 4th ed. (Guilford Publications, UnitedStates, 2015).

[45] A. R. Small, Range restriction, admissions criteria,and correlation studies of standardized tests (2017),arXiv:1709.02895 [physics.ed-ph].

[46] National Research Council, A data-based assessment ofresearch doctorate programs in the United States (Na-tional Research Council, 2011).

[47] S. van Buuren and K. Groothuis-Oudshoorn, mice: Mul-tivariate Imputation by Chained Equations in R, Journalof Statistical Software 45, 1 (2011).

[48] A. Field, J. Miles, and Z. Field, Discovering Statistics Us-ing R, 1st ed. (SAGE Publications Ltd, London; Thou-sand Oaks, Calif, 2012).

[49] J. Scott Jones, Learn to Use the Phi Coefficient Measureand Test in R With Data From the Welsh Health Survey(Teaching Dataset) (London, 2021).

[50] B. O. Muthen, L. K. Muthen, and T. Asparouhov, Re-gression and Mediation Analysis Using Mplus (Muthen& Muthen, 2016).

[51] R. M. Baron and D. A. Kenny, The moderator–mediatorvariable distinction in social psychological research: Con-ceptual, strategic, and statistical considerations, Journalof Personality and Social Psychology 51, 1173 (1986).

[52] D. P. MacKinnon, Introduction to Statistical MediationAnalysis (Routledge, 2008).

[53] D. P. MacKinnon, C. M. Lockwood, C. H. Brown,

19

W. Wang, and J. M. Hoffman, The intermediate end-point effect in logistic and probit regression, Clinical Tri-als (London, England) 4, 499 (2007).

[54] R. H. Hoyle, Handbook of Structural Equation Modeling(Guilford Press, 2012).

[55] M. E. Sobel, Identification of Causal Parameters in Ran-domized Studies With Mediating Variables, Journal ofEducational and Behavioral Statistics 33, 230 (2008).

[56] K. Imai, L. Keele, D. Tingley, and T. Yamamoto, Un-packing the Black Box of Causality: Learning aboutCausal Mechanisms from Experimental and Observa-tional Studies, American Political Science Review 105,765 (2011).

[57] B. Muthen, Applications of causally defined direct andindirect effects in mediation analysis using SEM in Mplus(2011).

[58] D. Tingley, T. Yamamoto, K. Hirose, L. Keele, andK. Imai, mediation: R package for Causal MediationAnalysis, Journal of Statistical Software 59, 1 (2014).

[59] J. Steen, T. Loeys, B. Moerkerke, and S. Vansteelandt,medflex: An R package for Flexible Mediation Analysisusing Natural Effect Models, Journal of Statistical Soft-ware 76, 1 (2017).

[60] L. Muthen and B. Muthen, Mplus, Statistical analysiswith latent variables. User’s guide 8.4 (2009).

[61] C. F. Dormann, J. Elith, S. Bacher, C. Buchmann,G. Carl, G. Carre, J. R. G. Marquez, B. Gruber,B. Lafourcade, P. J. Leitao, T. Munkemuller, C. Mc-Clean, P. E. Osborne, B. Reineking, B. Schroder, A. K.Skidmore, D. Zurell, and S. Lautenbach, Collinearity: areview of methods to deal with it and a simulation studyevaluating their performance, Ecography 36, 27 (2013).

[62] K. P. Vatcheva, M. Lee, J. B. McCormick, and M. H.Rahbar, Multicollinearity in Regression Analyses Con-ducted in Epidemiologic Studies, Epidemiology (Sunny-vale, Calif.) 6, 10.4172/2161-1165.1000227 (2016).

[63] F. L. Schmidt and J. E. Hunter, The validity and utilityof selection methods in personnel psychology: Practicaland theoretical implications of 85 years of research find-ings., Psychological bulletin 124, 262 (1998).

[64] M. M. Shultz and S. Zedeck, Admission to law school:New measures, Educational Psychologist 47, 51 (2012).

[65] F. Lievens and P. R. Sackett, The validity of interper-sonal skills assessment via situational judgment tests forpredicting academic success and job performance., Jour-nal of Applied Psychology 97, 460 (2012).

[66] K. Z. Victoroff and R. E. Boyatzis, What is the relation-ship between emotional intelligence and dental studentclinical performance?, Journal of dental education 77,416 (2013).

[67] F. L. Oswald and L. M. Hough, Personality and its as-sessment in organizations: Theoretical and empirical de-velopments. (American Psychological Association, 2011).

[68] A. Feingold, Gender differences in personality: a meta-analysis., Psychological bulletin 116, 429 (1994).

[69] R. Emmerling, R. E. Boyatzis, and R. J. Emmerling,Emotional and social intelligence competencies: crosscultural implications, Cross Cultural Management: AnInternational Journal (2012).

[70] P. R. Sackett, N. R. Kuncel, A. S. Beatty, J. L. Rig-don, W. Shen, and T. B. Kiger, The Role of So-cioeconomic Status in SAT-Grade Relationships andin College Admissions Decisions:, Psychological Science10.1177/0956797612438732 (2012), publisher: SAGEPublicationsSage CA: Los Angeles, CA.

[71] L. Owens, B. Zwickl, S. Franklin, and C. Miller, PhysicsGRE Requirements Create Uneven Playing Field forGraduate Applicants (2020) pp. 382–387.

[72] J. M. Ostrove, A. J. Stewart, and N. L. Curtin, Socialclass and belonging: Implications for graduate students’career aspirations, The Journal of Higher Education 82,748 (2011).

[73] K. Imai and T. Yamamoto, Identification and Sensitiv-ity Analysis for Multiple Causal Mechanisms: RevisitingEvidence from Framing Experiments, Political Analysis21, 141 (2013).

[74] J. Pearl, Interpretation and identification of causal me-diation, Psychological methods 19, 459 (2014).

[75] T. J. VanderWeele, Mediation Analysis: A Practitioner’sGuide, Annual Review of Public Health 37, 17 (2016).

Supplementary Materials:

Analyzing admissions metrics as predictors of graduate GPA and

whether graduate GPA mediates PhD completion

Mike Verostek(1,2), Casey Miller(2) and Benjamin Zwickl(2)

(1)Department of Physics and Astronomy, University of Rochester and

(2)Department of Physics and Astronomy, Rochester Institute of Technology

1

arX

iv:2

104.

0859

1v3

[ph

ysic

s.ed

-ph]

1 J

ul 2

021

I. MULTIPLE MEDIATOR MODEL RESULTS

M2(GGPA)

Y(PhD)

M1(Tier)

X(e.g.

UGPA)

Figure 1: Qualitative diagram displaying the proposed effects in a multiple mediator

model that includes program tier.

The results presented in the main analysis (see Section IV B) utilize models that suppose

the existence of one mediating variable (GGPA). Single mediator models such as this have

been studied extensively [1], and provide a well-established mathematical framework (see

Section III C of the main text) to calculate direct and indirect effects. However, it is

sometimes of interest to consider cases involving multiple mediators.

In Section IV A of the main text we found that students attending institutions categorized

as Tier 3 earned GGPA scores that were statistically lower than students attending Tier 1 or 2

institutions. We suppose that program tier is a proxy measure for institutional resources and

support, meaning that a higher tier institution might possess greater academic resources,

which could improve student graduate performance. Better resources for students could

also directly affect PhD completion, as indicated in [2]. Hence, we also consider a multiple

mediator model corresponding to Figure 1 in which program tier is posited to be a second

mediating variable between admissions metrics and PhD completion.

Methods to disentangle the effect that each individual path in a multiple mediator model

has on the dependent variable has been the subject of much recent research [3, 4]. We follow

the method provided in [5], which extends the counterfactual mediation analysis framework

to deal with particular cases of multiple mediators.

The results of this analysis are direct and indirect effects of each admissions metric on PhD

completion, with the indirect effect broken down into the portion transmitted through GGPA

and the portion transmitted through program tier. Indeed, the output is qualitatively similar

to the single mediator case described previously: direct and indirect effects are calculated,

2

Tier

GGPA

Indirect

Direct

Total

−0.2 0.0 0.2 0.4 0.6 0.8

Log Odds (GRE−P)

Tier

GGPA

Indirect

Direct

Total

−0.2 0.0 0.2 0.4 0.6 0.8

Log Odds (GRE−Q)

Tier

GGPA

Indirect

Direct

Total

−0.2 0.0 0.2 0.4 0.6 0.8

Log Odds (UGGPA)

a) b) c)(( (

Figure 2: Results of a model using multiple mediators to predict PhD completion. Bars

represent the change in log odds associated with a one standard deviation change in the

independent variable. Results are qualitatively the same as the single mediator case, with

graduate GPA. Despite its inclusion, the indirect effects transmitted through program tier

are not statistically significant in any of the multiple mediator models and are minuscule

compared to the effects associated with GGPA.

and effects for which the 95% error bars do not cross zero are statistically significant. These

results are summarized in Figure 2.

The single mediator results presented in Section IV B of the main text are fully consistent

qualitatively with the results of this multiple mediator analysis. Again, only UGPA has a

significant total effect on PhD completion, and that effect is fully transmitted through

GGPA. The indirect effect of GRE-P is significant, meaning that increased GRE-P scores

tend to predict better GGPA scores, which in turn predict higher PhD completion. Still,

this effect lesser in magnitude than the effect of UGPA. Again, none of the predictive effects

of GRE-Q are significant, although the direct predictive effect of GRE-Q on completion

borders closely on statistical significance.

Quantitatively the results of multiple mediation analysis yield effect estimates on the

log odds scale rather than a probability scale. Exponentiation of the log odds converts the

scale to a regular odds ratio. Thus, the numerical estimates for the effects shown in Figure

2 may be interpreted as the change in the log odds ratio (or if exponentiated, the odds)

of PhD completion due to a one standard deviation increase in the independent variable.

For instance, the “Indirect” bar of Figure 2a shows the best estimate for the log odds ratio

for the indirect predictive effect of UGPA on PhD completion is 0.46 (meaning that the

3

odds ratio is exp(0.46)=1.58). Thus, the model predicts that a student who had scored one

standard deviation higher in their UGPA would increase their odds of PhD completion by

58%.

Although this may seem like a large value, recall that the probability of an event is given

by odds/(1+odds), meaning that the larger the probability, the larger the difference in odds.

Models in [2] estimated that probability of PhD completion was typically around 75% for

most U.S. students, so an increase of approximately 50% in odds of completion would result

in a probability increase of about 7%. This estimate falls within the 95% confidence interval

of the results presented in the single mediator model for a one standard deviation increase

in UGPA (approximately an increase in score of 0.3).

Despite its inclusion, the indirect effects transmitted through program tier are not sta-

tistically significant in any of the multiple mediator models and are minuscule compared to

the effects associated with GGPA. For comparison, in the model using UGPA as the inde-

pendent variable the increase in the log odds ratio of PhD completion associated with Tier

is 0.03, while the log odds ratio increase associated with GGPA is 0.43. Thus, the results

of all multiple mediator models analyzed indicate that tier is a far weaker mediator than

GGPA.

II. MULTIPLE REGRESSION AND MULTICOLLINEARITY

To address potential concerns with multicollinearity in our primary regression analysis,

we present here the results of several alternative regression models. Multicollinearity is a

phenomenon in which two or more predictors in a regression model are highly correlated [6].

When multicollinearity is present, the standard errors of the affected variables can become

inflated [7, 8]. Increased standard errors may cause some variables to appear statistically

insignificant when they should be significant.

Diagnosing multicollinearity often consists of examining bivariate correlations between

variables in the regression model and calculating variance inflation factor (VIF) scores. VIF

scores are measures of the extent to which the variance of a regression coefficient is increased

due to collinearity. When two predictors are independent (orthogonal, zero correlation) their

VIF score is 1. As the correlation between the variables increase, so does the VIF score.

Unfortunately the threshold values of metrics for identifying multicollinearity are am-

4

VIF

UGPA 1.16

GRE-P 1.56

GRE-Q 1.47

GRE-V 1.18

Race 1.05

Gender 1.01

Table I: Average Variance Inflation Factors (VIF) for each predictor in a multiple

regression predicting graduate GPA across 20 imputed data sets.

biguous [9]. For instance, some authors suggest that correlations above r = 0.35 [10] are

cause for concern, while some use r = 0.5 as a cutoff [11]. Others assert that r = 0.80 or

r = 0.90 are likely to be problematic [12, 13]. Reviews of articles discussing multicollinearity

thresholds indicate that it is most common for authors to use r = 0.70 to r = 0.80 and VIF

> 10 rule-of-thumb cutoffs for investigating possible multicollinearity [14, 15].

Effects due to multicollinearity and variance inflation are negligible in our analyses. As

noted in the main text, we considered the possibility that it would be inappropriate to

include both GRE-P and GRE-Q in the same model due to collinearity concerns. However,

both the correlation coefficient and VIF scores for these variables fail to meet the most

commonly used thresholds. The bivariate correlation (rxy = 0.51) between the two is not

high enough to warrant genuine concern. Furthermore, the variance inflation factor (VIF)

of both GRE-P and GRE-Q was below 1.75 in each imputed dataset’s regression model, well

below the commonly cited threshold of 10.

Still, we perform several analyses in order to explore the effect of eliminating either GRE-

Q or GRE-P from our analyses. We also consider a model in which we generate a composite

GRE-Average score, which is the simple average of the two exam scores. These are shown

in Table III. The results of the main text’s analysis are shown in Table II for comparison.

Notably, as suggested by the low VIF scores, the standard errors of the GRE-Q and GRE-

P coefficients remain approximately constant regardless of whether one or both variables are

5

included in the regression model. In the model presented in the primary analysis shown in

Table II, the standard errors for GRE-P and GRE-Q are 6×10−4 and 12×10−4, respectively.

When GRE-P is removed from the model, the standard error of GRE-Q decreases slightly

to 11 × 10−4. When GRE-Q is removed, the standard error of GRE-P remains 6 × 10−4.

Across all versions of the multiple regression analyses, UGPA is highly significant and is

the most effective predictor of GGPA among tested variables, evidenced by examining the

standardized regression coefficients in each model. Indeed, even in the case that GRE-P and

GRE-Q are averaged into one score to consolidate their predictive ability we observe that

UGPA is the most effective predictor. This again reinforces the conclusion that undergrad-

uate grades are the best predictors of graduate school course performance.

In the model with GRE-Q removed, the GRE-P coefficient is 0.17, which hardly changes

from the model in which both GRE-Q and GRE-P are included (0.15). Thus the practical

effect on the GRE-P coefficient is negligible regardless of whether the model includes GRE-Q.

Meanwhile, exclusion of the GRE-P variable does indeed cause the GRE-Q coefficient become

a statistically significant predictor. This is unsurprising given the fact that its magnitude

dropped from 0.09 to 0.03 with the inclusion of GRE-P while maintaining approximately

the same standard error.

6



Independent

Variable

Coefficient

(Standard Error)

Standardized

Coefficient

Intercept 1.92** (0.15) -0.06

UGPA 0.35** (0.04) 0.24**

GRE-P 31 × 10−4** (6 × 10−4) 0.15**

GRE-Q 16 × 10−4 (12 × 10−4) 0.03

GRE-V 3 × 10−4 (6 × 10−4) 0.01

Black -0.11 (0.09) -0.23

Hispanic -0.01 (0.07) -0.02

Native Am. 0.01 (0.23) 0.03

Asian -0.02 (0.06) -0.05

Other -0.24* (0.11) -0.51*

Undisclosed 0.07** (0.02) 0.15**

Gender 0.06* (0.03) 0.13*

N 1955


Table II: Coefficients of a multiple regression analysis modeling graduate GPA as a

function of common quantitative admissions metrics. Reference categories are White for

race and Male for gender. These results are presented in the main analysis and are

included here for comparison with alternate regression models.

7


Dependent Variables

Model w/ No GRE-P Model w/ No GRE-Q Model w/ Composite GRE Score

Independent

Variable

Coefficient

(Standard Error)

Standardized

Coefficient

Coefficient

(Standard Error)

Standardized

Coefficient

Coefficient

(Standard Error)

Standardized

Coefficient

Intercept 1.72** (0.15) -0.03 2.00** (0.15) -0.06 1.86** (0.15) -0.06

UGPA 0.39** (0.04) 0.26** 0.36** (0.04) 0.24** 0.35** (0.04) 0.23**

GRE-V 7 × 10−4 (6 × 10−4) 0.03 4 × 10−4 (6 × 10−4) 0.02 2 × 10−4 (6 × 10−4) 0.01

Black -0.13 (0.09) -0.26 -0.11 (0.09) -0.22 -0.10 (0.09) -0.21

Hispanic -0.01 (0.07) -0.03 -0.01 (0.07) -0.03 −26 × 10−4 (0.07) -0.01

Native Am. -0.01 (0.24) -0.02 0.02 (0.24) 0.04 38 × 10−4 (0.24) 0.01

Asian 12 × 10−4 (0.06) 26 × 10−4 -0.02 (0.06) -0.04 -0.02 (0.11) -0.04

Other -0.23* (0.11) -0.49* -0.25* (0.11) -0.51* -0.24* (0.11) -0.50*

Undisclosed 0.06* (0.02) 0.12* 0.07* (0.02) 0.15* 0.07* (0.02) 0.15*

Gender 0.01 (0.03) 0.03 0.06* (0.03) 0.13* 0.06* (0.03) 0.12*

GRE-Q 43 × 10−4** (11 × 10−4) 0.09** – – – –

GRE-P – – 34 × 10−4** (6 × 10−4) 0.17** – –

GRE-Average – – – – 55 × 10−4** (9 × 10−4) 0.17**

N 1955 1955 1955

Adjusted R-Squared 0.09 0.11 0.11

Table III: Coefficients of several multiple regression analysis modeling graduate GPA

using different combinations of GRE scores. Across all versions of the multiple regression

analyses, UGPA is highly significant and is the most effective predictor of GGPA,

evidenced by examining the standardized regression coefficients in each model Reference

categories are White for race and Male for gender.

8

III. MEDIATION WITH CATEGORICAL PREDICTORS

−10

−5

0

5

10

1st 2nd 3rd 4th 5th

Undergraduate GPA Quintile

% P

roba

bilit

y C

hang

e R

elat

ive

to M

iddl

e Q

uint

ile

(Ind

irec

t)

−10

−5

0

5

10

1st 2nd 3rd 4th 5th

GRE−P Quintile

% P

roba

bilit

y C

hang

e R

elat

ive

to M

iddl

e Q

uint

ile

(Ind

irec

t)

−10

−5

0

5

10

1st 2nd 3rd 4th 5th

Undergraduate GPA Quintile

% P

roba

bilit

y C

hang

e R

elat

ive

to M

iddl

e Q

uint

ile

(Dir

ect)

−10

−5

0

5

10

1st 2nd 3rd 4th 5th

GRE−P Quintile

% P

roba

bilit

y C

hang

e R

elat

ive

to M

iddl

e Q

uint

ile

(Dir

ect)

(a)

(c)

(b)

(d)

Figure 3: Plots of the indirect and direct effects for UGPA and GRE-P when treated as

categorical predictors. Results are qualitatively similar to those treating the variables as

continuous. (a) Indirect effect of undergraduate GPA when split into quintiles and treated

as a categorical variable. (b) Indirect effect of GRE-P when split into quintiles and treated

as a categorical variable. (c) Direct effect of undergraduate GPA when split into quintiles

and treated as a categorical variable. (d) Direct effect of GRE-P when split into quintiles

and treated as a categorical variable.

In addition to the analysis presented in the main text in which we treated UGPA, GRE-

P, and GRE-Q as continuous variables, we also performed a mediation analysis in which

we turned these predictors into categorical variables by binning scores into quintiles. This

means that students whose UGPA score fell in the top 20% of our data were classified in

the 5th (top) quintile and students whose scores fell in the bottom 20% were classified as

the 1st (bottom) quintile. We grouped students in this way for their UGPA, GRE-P, and

9

GRE-Q scores separately. Grouping the variables in this way may provide readers with a

more intuitive feeling for how PhD completion probability is predicted to change between

students.

Grouping UGPA scores by quintile leads to five possible classifications: 1st quintile (<

3.38), 2nd quintile (3.38-3.61), 3rd quintile (3.61-3.80), 4th quintile (3.80-3.91), and 5th

quintile (> 3.91). Similarly for GREP the groupings are: 1st quintile (< 30), 2nd quintile

(30-46), 3rd quintile (46-61), 4th quintile (61-74), 5th quintile (> 74). We also explored

treating GRE-Q as a categorical variable by splitting students into quintiles, but the scores

for physicists taking the GRE-Q are so tightly distributed that we do not believe much

intuition is gained from the analysis. In particular, the second quintile of scores ranges

from 89.1-90.9 and the third quintile ranges from 85-89.1, while the fifth quintile includes

all scores less than 77. Since there is such a small score difference between students who

scored in bottom and top quintiles on the GRE-Q, we do not believe it is useful to group

the scores in this way.

Results for treating UGPA and GRE-P as categorical variables are qualitatively similar to

those presented in the main paper. Using the “Medium” group as a reference, the predicted

direct and indirect effects associated with UGPA and GRE-P are shown in 3.

For UGPA, a student who earns a score in the 5th quintile as opposed to the 1st quintile

increases their probability of PhD completion by approximately 10% due to the indirect

effect on GGPA. We had reported in the main paper that the predicted probability change

in completion for a student in the 10th percentile of UGPA scores to the 90th percentile (a

change in score from 3.2 to 3.98) was 11%, which is consistent with the results shown here.

As in the main text, all direct effects of UGPA are not statistically significant.

Meanwhile the indirect effect of GRE-P on completion predicts a probability change of

approximately 6% for a student who earns a top quintile GRE-P score as opposed to a

bottom quintile score. This is consistent with the main analysis, which estimated that due

to the indirect effect of GRE-P on GGPA a change from the 10th to 90th percentile of

GRE-P scores (a change in GRE-P from 21 to 85 among the overall test-taking population)

increased the predicted probability of completion by 7%. Again, as in the main text, all

direct effects of GRE-P are not statistically significant.

10

IV. SENSITIVITY ANALYSIS PLOTS

Undergraduate GPA Sensitivity Analysis

−0.05 −0.04 −0.03

−0.02

−0.01

0 0.01

0.02

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

R~M2

R~ Y2

0.0

0.2

0.4

0.6

0.0 0.2 0.4 0.6 0.8 1.0

GRE Physics Sensitivity Analysis

−0.03 −0.02

−0.01

0

0.01

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

R~M2

R~ Y2

0.0

0.2

0.4

0.6

0.0 0.2 0.4 0.6 0.8 1.0

Figure 4: Contour plots of the total variance of the mediator and outcome variables that

an unobserved confounder would have to explain in order to make the indirect effects of

UGPA and GRE-P vanish. The bold lines represent solutions to the equations

R2M ×R2

Y = 0.072 and R2M ×R2

Y = 0.069, for UGPA and GRE-P respectively. These plots

are generated using a typical imputed dataset and are not pooled across imputations.

Plots are generated in the R package mediation [16].

In the main analysis we summarized the results of sensitivity analyses performed on each

of the imputed data sets used in the single mediator models to assess their robustness. We

let R2M and R2

Y represent the proportions of original variances explained by the unobserved

confounder for the mediator GGPA and the outcome PhD completion, respectively. The

result of a sensitivity analysis is a single value representing the product R2M × R2

Y that

identifies the amount of original variance in the mediator and outcome that the confounder

would have to explain in order to make the observed effect vanish. Hence, the sensitivity

analysis results in a family of solutions for which the equation R2M × R2

Y = constant is

satisfied.

Our sensitivity analysis reveals that for a confounder to explain enough variance to make

the indirect effects of UGPA and GRE-P on GGPA vanish, the product R2M ×R2

Y on average

would have to be 0.072 and 0.069, respectively. The standard deviation of the UGPA

sensitivity analysis results across all imputed datasets was 0.006. This indicates there was

little difference in the outcome of sensitivity analyses regardless of imputation. The standard

11

deviation of the GRE-P sensitivity analysis results across all imputed datasets was 0.001;

again, there was little difference in sensitivity analysis results across imputations.

Contour plots of solutions to the equation R2M × R2

Y = 0.072 and R2M × R2

Y = 0.069, for

UGPA and GRE-P respectively, are shown in Figure 4 by the line labeled 0 and in bold.

Other combinations of higher R2M ×R2

Y values are plotted as well, which demonstrate values

at which the estimated indirect effect would actually become negative.

12

V. ALTERNATIVE IMPUTATION MODEL USING OUTCOME VARIABLES

The primary analysis employed the well-known method of multiple imputation [17, 18]

to handle cases of missing UGPA and GRE-P scores. Multiple imputation handles missing

data by replacing the missing values using a statistical model based on the complete data.

This process is performed several times to generate multiple versions of the complete data

set in order to reflect uncertainty in the missing data.

The imputation model used in Miller et al. [2] predicted students’ missing UGPA and

GRE-P scores using their GRE-Q, GRE-V, gender, race, and program tier (the other inde-

pendent variables in the regression model presented in [2]). GGPA was not included as a

predictor in the imputation model since it was not included as a variable of interest in the

analysis. Thus, including GGPA in the models presented in the current paper presented a

choice regarding our approach to handling missing data: to use the same imputation model

as Miller et al. in order to directly compare results across analyses, or to use a slightly

different imputation model that also included GGPA as a predictor variable. Ultimately we

opted to keep the same imputation approach across papers, but we present here the results

of the paper had we used the alternative imputation model in which GGPA was included.

It may seem counterintuitive to use GGPA, which was used as an outcome variable in

the primary analysis, in a model of data imputation. Employing a model of data imputa-

tion that uses the outcome variable to predict missing values of the independent variable

may seem like a self-fulfilling prophecy, guaranteeing a relationship to exist between them.

However, research suggests that including all variables, including the outcome variable, in

the imputation model in fact tends to produce less biased results [19–21]. This further moti-

vates our decision to include this alternative imputation approach, which demonstrates that

results are qualitatively similar regardless of which imputation model is used.

As in the main paper, we begin by exploring the relationships between variables using

bivariate correlations. We then examine the unique predictive effects of different admission

metrics on graduate GPA using a multiple linear regression model. Mediation analysis is

used to examine the role that graduate GPA plays in PhD completion by breaking down

effects into direct and indirect components. Lastly, we perform a second set of mediation

analyses using multiple mediators in which we compare the relative importance of tier and

graduate GPA as mediators in our model of PhD completion. At each point in the analysis,

13

we provide brief summaries of the important results and point out any differences that arise

from the primary analysis due to the use of a different imputation model.

A. Exploring the Role of GGPA - Alternative imputation model using outcome

variable


UGPA (3.6 ± 0.3) – (0.26, 0.35) (0.13, 0.23) (0.27, 0.37) (0.32, 0.41) (0.14, 0.23) (-0.09, 0.01)

GRE-Q (83.3 ± 10.4) 0.31 – (0.30, 0.37) (0.46, 0.54) (0.13, 0.22) (0.10, 0.19) (-0.16, -0.07)

GRE-V (76.3 ± 18.7) 0.18 0.34 – (0.22, 0.30) (0.06, 0.15) (0.02, 0.10) (-0.02, 0.07)

GRE-P (52.9 ± 23.2) 0.32 0.51 0.26 – (0.21, 0.31) (0.11, 0.20) (-0.33, -0.24)

GGPA (3.5 ± 0.5) 0.37 0.18 0.10 0.26 – (0.39, 0.46) (-0.06, 0.03)

Final Disp. 0.19 0.15 0.06 0.16 0.43 – (-0.09, 0.00)

Gender -0.04 -0.11 0.03 -0.29 -0.02 -0.05 –

Table IV: A matrix showing bivariate correlations between continuous and dichotomous

variables under an alternate imputation model. Correlations are shown in the lower

diagonal while confidence intervals for those correlations are shown in the upper diagonal.

UGPA and GRE-P correlations with GGPA are slightly boosted compared to the

correlations presented in the main text.

Since only aspects of the analysis including UGPA and GRE-P are influenced by the

new imputation model, only correlations involving these two variables are different from the

main paper. For instance, UGPA (rxy = 0.37, p < .001) and GRE-P (rxy = 0.26, p < .001)

are again positively correlated with GGPA, meaning that students with higher scores in

these metrics tend to earn higher GGPA scores. However, as expected due to the use of

GGPA in this imputation model, the correlation between UGPA and GGPA is now higher

(rxy = 0.37 rather than 0.29). The correlation between GGPA and GRE-P is also slightly

higher (rxy = 0.26 rather than 0.22). Thus the boost in correlation is higher for UGPA than

GRE-P, indicating that the primary effect of adding GGPA into the imputation model is to

boost the effects associated with UGPA.

Other results reported in Table IV are identical to the main paper. Again, GGPA is

the predictor most strongly correlated with final disposition (rpb = 0.43). This value is

statistically significant (p < .001), meaning that the mean GGPA score of students who

14

do not complete a PhD are statistically different from those who do complete their PhD.

Specifically, the positive correlation means that students with higher GGPA scores are more

likely to finish their PhD program successfully.



Independent

Variable

Coefficient

(Standard Error)

Standardized

Coefficient

Intercept 1.51** (0.15) -0.08**

UGPA 0.50** (0.04) 0.34**

GRE-P 40 × 10−4** (6 × 10−4) 0.19**

GRE-Q −3 × 10−4 (12 × 10−4) -0.01

GRE-V −2 × 10−6 (6 × 10−4) 0.00

Black -0.10 (0.09) -0.20

Hispanic 0.01 (0.07) 0.01

Native Am. 0.07 (0.24) 0.16

Asian -0.03 (0.07) -0.05

Other -0.24* (0.10) -0.50*

Undisclosed 0.10** (0.02) 0.20**

Gender 0.07* (0.03) 0.15*

N 1955


Table V: Results of a multiple regression for alternative imputation model. As in the

correlation matrix, the coefficients of the UGPA and GRE-P terms are boosted compared

to the main analysis. This effect is stronger for UGPA. Also notable is that the GRE-Q

coefficient is relatively weaker compared to the main analysis.

The results of the multiple regression analysis predicting GGPA shown in Table V are

qualitatively consistent with those presented in the main analysis. Significant unique predic-

tive effects at the 95% threshold were found for the numerical metrics UGPA and GRE-P,

meaning that students with higher UGPA and GRE-P scores tended to received higher

GGPA scores. GRE-Q does not reach the threshold for statistical significance.

The alternative imputation model increased the effects associated with UGPA (β = 0.34,

t = 12.67, p < .01) and GRE-P (β = 0.19, t = 6.18, p < .01). Specifically, the UGPA

standardized coefficient increased from 0.24 to 0.34 and the GRE-P standardized coefficient

increased from 0.15 to 0.19. As with the bivariate correlations, the alternate imputation

15

model boosts the effect of UGPA on GGPA more than it does for GRE-P. Furthermore, the

overall model fit improved under the alternate imputation approach (R2 increased from 0.11

to 0.17).

16

B. Mediation Analysis - Alternative imputation model using outcome variable

Figure 5 graphically summarizes the results of the mediation analysis under the alternate

imputation model. As in the main paper, Figures 5a), c), and e) display the results of

the three separate analyses predicting PhD completion probability changes relative to the

median value of each independent variable. Points on each plot represent individual calcu-

lations of probability change relative to each admission metric’s median value. Hence, the

median value of the x-axis variable is clearly shown on the plot as the point where the direct

(red) and indirect effect (blue) lines intersect, corresponding to a total probability change

of 0%. Percent probability changes due to direct effects of each admissions metric on PhD

completion are shown in red, while indirect effects on PhD completion transmitted through

GGPA are shown in blue. Plots of the total effect of each variable on PhD completion

probability, which is the sum of the direct and indirect effects, are also shown in Figures

5b), d), and f). The shaded ribbons around the lines representing the best-estimates of

probability change show the 95% confidence interval. Hence, if this shaded region contains

the line y = 0, the effect is not statistically significant at the α = .05 level.

For the most part the results presented in Figure 5 are qualitatively consistent with

those in the main analysis. The primary message from this model and the main text is

that among common quantitative admissions metrics, undergraduate GPA offers the most

promising insight into whether physics graduate students will earn a PhD.

Under the alternate imputation model, there is a noticeable increase in both the total

and indirect predictive effects associated with UGPA.

While the total effect of GRE-P just barely missed reaching the threshold for statistical

significance in the main paper, under the alternate imputation model the shaded ribbons

around total effect of GRE-P no longer cross the line y = 0. Hence, the total effect of GRE-P

meets the threshold of statistical significance at the α = 0.05 level in this model, although it

is again very close. The magnitude of the total effect is similar to the main paper (a change

from the 10th to 90th percentile of GRE-P scores among students in this model increases

the predicted probability of completion by approximately 8% in this model as opposed to

7% in the main paper). This is likely due to the slight boost in the magnitude of the

GRE-P’s indirect effect on completion; a change from the 10th to 90th percentile of GRE-P

scores among students in this model increases the predicted probability of completion by

17

approximately 10%, up from 8% in the main analysis.

In contrast to the main analysis, the direct effect of GRE-Q under the alternate imputa-

tion model meets the threshold of statistical significance at the α = 0.05 level; the indirect

predictive effects of GRE-Q on PhD completion remain statistically insignificant.

Their sum, the total effect, also barely meets the threshold for significance. This is

consistent with [2], in which the predictive effect of GRE-Q on completion was significant

but was not strong and in fact bordered on insignificance as well (p = 0.048).

Given the weak relationship between GRE-Q and GGPA revealed by the multiple regres-

sion analysis in Section V A, it is expected that the indirect effect shown in blue on Figure

5 is nearly zero. Indeed, any predictive effect from GRE-Q on completion appears to stem

from the direct effect of GRE-Q on PhD completion.

18

��

��

��

��

��

��

��

��

��

��

−25−20−15−10−50510

60 70 80 90 100


% P

roba

bilit

y C

hang

e R

elat

ive

to M

edia

n Sc

ore

�� Direct Effect Indirect Effect

��

��

��

��

��

��

��

��

��

−25−20−15−10−50510

0 25 50 75 100


% P

roba

bilit

y C

hang

e R

elat

ive

to M

edia

n Sc

ore

�� Total Effect

��

��

��

��

��

��

��

��

��

��

��

��

��

−25−20−15−10−50510

0 25 50 75 100


% P

roba

bilit

y C

hang

e R

elat

ive

to M

edia

n Sc

ore


��

��

��

��

��

��

��

��

−25−20−15−10−50510

60 70 80 90 100


% P

roba

bilit

y C

hang

e R

elat

ive

to M

edia

n Sc

ore

�� Total Effect

��

��

��

��

��

��

��

��

��

��

��

−25−20−15−10−50510

3.0 3.2 3.4 3.6 3.8 4.0

Undergraduate GPA

% P

roba

bilit

y C

hang

e R

elat

ive

to M

edia

n Sc

ore

�� Total Effect

��

��

��

��

��

��

��

��

��

��

��

��

−25−20−15−10−50510

3.0 3.2 3.4 3.6 3.8 4.0

Undergraduate GPA

% P

roba

bilit

y C

hang

e R

elat

ive

to M

edia

n Sc

ore

�� Direct Effect Indirect Effect10% 25% 75% 90% 10% 25% 75% 90%

10% 25% 75% 90% 10% 25% 75% 90%

10% 25% 75%90% 10% 25% 75%90%

(a) (b)

(c) (d)

(e) (f)

Figure 5: Mediation analysis results from an alternate imputation model predicting a

student’s change in probability of completion, split into direct (red) and indirect via

graduate GPA (blue) effects. The indirect effects of UGPA and GRE-P are larger

compared with the main analysis, and again UGPA is clearly a stronger predictor in these

models. Also notable is that the total effect of GRE-P just meets the threshold of

significance under this model. The direct effect of GRE-Q shifts to significant at 95%

threshold as well.

19

C. Multiple Mediators - Alternative imputation model using outcome variable

Tier

GGPA

Indirect

Direct

Total

−0.2 0.0 0.2 0.4 0.6 0.8

Log Odds (GRE−P)

Tier

GGPA

Indirect

Direct

Total

−0.2 0.0 0.2 0.4 0.6 0.8

Log Odds (GRE−Q)

Tier

GGPA

Indirect

Direct

Total

−0.2 0.0 0.2 0.4 0.6 0.8

Log Odds (UGGPA)

(b)(a) (c)

Figure 6: Multiple mediator results under an alternate imputation model. Results are

qualitatively consistent with the single mediator case. As in the main analysis, the indirect

effects transmitted through program tier are not statistically significant in any of the

multiple mediator models and are minuscule compared to the effects associated with

GGPA.

As discussed in the main text, methods to disentangle the effect that individual paths in a

multiple mediator model have on the dependent variable has been the subject of much recent

research [3, 4]. For the alternate imputation model we again follow the method provided in

[5], which extends the counterfactual mediation analysis framework to deal with particular

cases of multiple mediators. The results of this analysis are direct and indirect effects of

each admissions metric on PhD completion, with the indirect effect broken down into the

portion transmitted through GGPA and the portion transmitted through program tier.

The output is again qualitatively similar to the single mediator case described previously:

direct and indirect effects are calculated, and effects for which the 95% error bars do not

cross zero are statistically significant. These results are summarized in Figure 6. The single

mediator results presented in Section V B are fully consistent qualitatively with the results

of this multiple mediator analysis. Again, both UGPA and GRE-P have a significant total

effect on PhD completion, and that effect is fully transmitted through GGPA. However the

effect of UGPA is larger than the effect of GRE-P. The increase in log-odds associated with

a one standard deviation increase in UGPA is 0.60 and GRE-P is 0.35. As in the single

mediator case the direct effect of GRE-Q is just barely significant.

Despite its inclusion, the indirect effects transmitted through program tier are not sta-

20

tistically significant in any of the multiple mediator models and are minuscule compared to

the effects associated with GGPA.

21

VI. NON-US RESULTS

In order to focus on issues of diversity and inclusion associated most strongly with US

applicants, the main paper only used data from the subset of domestic graduate students.

This decision was also motivated by the the markedly different distributions of scores for

US and Non-US students. Here, we repeat the analyses presented in the main paper on the

subset of international graduate students.

To match the primary analysis, we first present the relationships between variables using

bivariate correlations. We then examine the unique predictive effects of different admission

metrics on international students’ graduate GPA using a multiple linear regression model. A

mediation analysis is used to examine the role that graduate GPA plays in PhD completion

by international students by breaking down effects into direct and indirect components.

Lastly, we perform a second set of mediation analyses using multiple mediators in which we

compare the relative importance of tier and graduate GPA as mediators in our model of

PhD completion by international students. At each point in the analysis, we provide brief

summaries of the important results and point out any differences between results for US and

Non-US students.

A. Exploring the Role of GGPA - NonUS

As was the case for US students, GGPA is the predictor most strongly correlated with

final disposition (rpb = 0.26) for international students. This value is statistically significant

(p < .001), meaning that the mean GGPA score of students who do not complete a PhD

are statistically different from those who do complete their PhD. Specifically, the positive

correlation means that students with higher GGPA scores are more likely to finish their PhD

program successfully. This trend is visually apparent in Figure 7(a), which shows boxplots

of GGPA grouped by PhD completers and non-completers.

There is no statistical difference between average GGPA scores for male and female

international students (rpb = 0.00, p = 0.98), as demonstrated in Figure 7(b). However, the

statistically significant difference between males and females in GRE-P performance persists

(rpb = −0.23, p < .001). Thus the gap in GRE-P performance by gender appears in both

the US and NonUS groups, but disappears in subsequent graduate performance metrics.

22


UGPA (3.6 ± 0.4) – (0.05, 0.18) (0.07, 0.20) (0.17, 0.34) (0.05, 0.17) (-0.03, 0.09) (-0.09, 0.02)

GRE-Q (87.2 ± 7.4) 0.12 – (0.25, 0.34) (0.47, 0.56) (0.09, 0.19) (0.00, 0.11) (-0.21, -0.11)

GRE-V (60.5 ± 29.1) 0.14 0.29 – (0.27, 0.37) (0.07, 0.18) (-0.04, 0.06) (-0.29, -0.16)

GRE-P (75.1 ± 22.3) 0.26 0.52 0.32 – (0.15, 0.27) (0.00, 0.11) (-0.29, -0.16)

GGPA (3.7 ± 0.4) 0.11 0.14 0.13 0.21 – (0.21, 0.31) (-0.05, 0.05)

Final Disp. 0.03 0.05 0.01 0.05 0.26 – (-0.08, 0.03)

Gender -0.04 -0.16 -0.06 -0.23 0.00 -0.02 –

Table VI: A matrix showing bivariate correlations between continuous and dichotomous

variables for international students. Correlations are shown in the lower diagonal while

confidence intervals for those correlations are shown in the upper diagonal. GGPA persists

as the best predictor of PhD completion, but none of the other quantitative metrics

(UGPA, GRE-P, GRE-Q, and GRE-V) have a statistically significant correlation with

PhD completion.

In contrast to US students, none of the quantitative metrics (UGPA, GRE-P, GRE-

Q, and GRE-V) have a statistically significant correlation with PhD completion. This

already suggests that the results of subsequent mediation analyses are likely to differ between

the US and NonUS student groups. Also different are the results of one-way independent

ANOVA tests, which show that the main effect of program tier on GGPA is insignificant,

F (2, 1448) = 2.80, p = 0.06. This indicates that GGPA scores of NonUS students do not

differ regardless of the tier of their institution, whereas GGPA scores of US students tended

to be higher as program tier increased.

The results obtained in our multiple regression analysis predicting GGPA are summarized

in Table VII. Significant unique predictive effects at the 95% threshold were found only for

the numerical metric GRE-P (β = 0.17, t = 4.42, p < .01). Hence, international students

with higher GRE-P scores tended to received higher GGPA scores. As in the main text, the

GRE-Q coefficient is not statistically significant (β = 0.04, t = 1.20, p = 0.23) and is less

than a quarter the magnitude of the GRE-P coefficient. Combined with the low correlation

between GRE-Q and PhD completion, this result indicates that GRE-Q scores likely will

not have significant predictive effects in subsequent mediation analyses.

In contrast with the US student group, the UGPA coefficient is statistically insignificant

(β = 0.06, t = 1.69, p = 0.09) and is approximately 33% the size of the GRE-P coefficient.

23

2.0

2.5

3.0

3.5

4.0


Tier

Gra

duat

e G

PAFinal Disposition No PhD PhD

2.0

2.5

3.0

3.5

4.0


Tier

Gra

duat

e G

PA

Gender Male Female

0

25

50

75

100


Tier

GR

E Ph

ysic

s Per

cent

ile

Gender Male Female

(a) (b)

(c)

Figure 7: (a) Graduate GPA by program tier and final disposition. International students

who do not complete a PhD earn lower graduate grades than students who complete their

programs. (b) Graduate GPA by program tier and gender. Male and female international

students earn similar graduate grades, and this trend holds across program tier. (c) GRE

Physics by program tier and gender. As for US students, across all program tiers there is a

significant gap in scores between male and female GRE-P test takers.

This suggests UGPA has comparatively little predictive effect on graduate course perfor-

mance for international students. Similar to the GRE-Q, this result suggests that UGPA

will not not have significant predictive effects in subsequent mediation analyses.

Since [2] did not include the results of a logistic regression model predicting PhD outcome

for international students, we include the results of that analysis here in Table VIII. We

observe that none of the quantitative admissions metrics meet the threshold of statistical

significance at the α = 0.05 level, although GRE-P (β = 0.07, p = 0.49) and GRE-Q

(β = 0.08, p = 0.19) are closest.

24



Independent

Variable

Coefficient

(Standard Error)

Standardized

Coefficient

Intercept 3.03** (0.17) -0.02

UGPA 0.06 (0.03) 0.06

GRE-P 29 × 10−4** (7 × 10−4) 0.17**

GRE-Q 20 × 10−4 (16 × 10−4) 0.04

GRE-V 7 × 10−4 (4 × 10−4) 0.05

Gender 0.05 (0.03) 0.12

N 1451


Table VII: Coefficients of a multiple regression analysis modeling graduate GPA as a

function of common quantitative admissions metrics for international student data.

Reference category for gender is male.

B. Mediation Analysis - NonUS

Figure 8 graphically summarizes the results of the mediation analysis for the subset of

international students. As in previous discussions, Figures 8a), c), and e) display the results

of the three separate analyses predicting PhD completion probability changes relative to the

median value of each independent variable. Points on each plot represent individual calcu-

lations of probability change relative to each admission metric’s median value. Hence, the

median value of the x-axis variable is clearly shown on the plot as the point where the direct

(red) and indirect effect (blue) lines intersect, corresponding to a total probability change

of 0%. Percent probability changes due to direct effects of each admissions metric on PhD

completion are shown in red, while indirect effects on PhD completion transmitted through

GGPA are shown in blue. Plots of the total effect of each variable on PhD completion

probability, which is the sum of the direct and indirect effects, are also shown in Figures

8b), d), and f). The shaded ribbons around the lines representing the best-estimates of

probability change show the 95% confidence interval. Hence, if this shaded region contains

the line y = 0, the effect is not statistically significant at the α = .05 level.

Several of the results presented in Figure 8 for international students are markedly dif-

ferent from the US student population.

25

Logistic Regression Results (*p < .05, **p < .01)

Dependent Variable - Final Disposition

Independent

Variable

Coefficient

(Standard Error)

Standardized

Coefficient

Intercept -0.37 (0.96) 1.19**

UGPA 0.12 (0.19) 0.04

GRE-P 32 × 10−4 (37 × 10−4) 0.07

GRE-Q 0.01 (94 × 10−4) 0.08

GRE-V −13 × 10−4 (23 × 10−4) -0.04

Gender -0.07 (0.16) -0.07

N 1451

Table VIII: Coefficients of a logistic regression analysis modeling PhD completion as a

function of common quantitative admissions metrics for international student data.

Reference category for gender is male. For comparison, 25% of US students (494 of 1955)

in this data set did not earn a PhD while 24% of international students (344 of 1451) in

this data set did not earn a PhD.

Whereas for US students the total effect of UGPA on PhD completion was statistically

significant and almost entirely attributable to the indirect effect of UGPA through GGPA,

the total effect of UGPA on completion is not significant for international students. The

estimated probability change in completion for a student in the 10th percentile of UGPA

scores to the 90th percentile (a change in score from 3.1 to 3.99) is just 3%, compared with

11% for US students.

The confidence ribbon around the estimate of the total effect of GRE-Q for NonUS stu-

dents (Figure 8f) is larger than that for US students, indicating a larger degree of uncertainty

around this estimate. While the total effect of GRE-Q bordered on statistical significance

for the US student subset, the wider confidence interval for the NonUS student estimate

easily includes the line y = 0 and therefore does not provide strong evidence that GRE-Q

effectively predicts NonUS student PhD completion. The direct and indirect effect estimates

of GRE-Q on completion do not meet the threshold of statistical significance either.

The GRE-P mediation analysis results for international students is similar to that of

the US student group. Again, the total effect of GRE-P barely misses the threshold for

statistical significance (Figure 8d). However, any effect of GRE-P on a student’s PhD

26

��

��

��

��

��

��

��

−15

−10

−5

0

5

10

70 80 90 100


% P

roba

bilit

y C

hang

e R

elat

ive

to M

edia

n Sc

ore


��

��

��

��

��

��

−20

−15

−10

−5

0

5

10

0 25 50 75 100


% P

roba

bilit

y C

hang

e R

elat

ive

to M

edia

n Sc

ore

�� Total Effect

��

��

��

��

��

��

��

��

��

��

��

��

−15

−10

−5

0

5

10

0 25 50 75 100


% P

roba

bilit

y C

hang

e R

elat

ive

to M

edia

n Sc

ore


��

��

��

��

��

��

��

−15

−10

−5

0

5

10

70 80 90 100


% P

roba

bilit

y C

hang

e R

elat

ive

to M

edia

n Sc

ore

�� Total Effect

��

−15

−10

−5

0

5

10

3.0 3.2 3.4 3.6 3.8 4.0

Undergraduate GPA

% P

roba

bilit

y C

hang

e R

elat

ive

to M

edia

n Sc

ore

�� Total Effect

��

−15

−10

−5

0

5

10

3.0 3.2 3.4 3.6 3.8 4.0

Undergraduate GPA

% P

roba

bilit

y C

hang

e R

elat

ive

to M

edia

n Sc

ore

�� Direct Effect Indirect Effect10% 25% 75% 90% 10% 25% 75% 90%

10% 25% 75% 90% 10% 25%

10% 25% 75%90% 10% 25% 75%90%

75% 90%

(a) (b)

(c) (d)

(e) (f)

Figure 8: Mediation analysis results using international student data predicting a

student’s change in probability of completion, split into direct (red) and indirect via

graduate GPA (blue) effects. Only the GRE-P mediation analysis results for international

students is similar to that of the US student group.

completion appears to be entirely mediated by GGPA, defined by the fact that the indirect

effect of GRE-P is statistically significant across all magnitudes of score change while its

direct effects are not. The model estimates that due to the indirect effect of GRE-P on

GGPA, a change from the 10th to 90th percentile of GRE-P scores among international

students in this study (a change in GRE-P from 21 to 85 among the overall test-taking

27

population) the predicted probability of completion increases by 8%.

28

C. Multiple Mediators - NonUS

Tier

GGPA

Indirect

Direct

Total

−0.4 −0.2 0.0 0.2 0.4

Log Odds (GRE−P)

Tier

GGPA

Indirect

Direct

Total

−0.4 −0.2 0.0 0.2 0.4

Log Odds (GRE−Q)

Tier

GGPA

Indirect

Direct

Total

−0.4 −0.2 0.0 0.2 0.4

Log Odds (UGGPA)

(b)(a) (c)

Figure 9: Multiple mediator results for NonUS students. As in the main analysis, the

indirect effects transmitted through program tier are not statistically significant in any of

the multiple mediator models and are minuscule compared to the effects associated with

GGPA. Results are mostly consistent with the single mediator case, although in this

GRE-Q model the total indirect effect is slightly larger than the direct effect. However the

large error bars indicate these estimates are highly uncertain.

Methods to disentangle the effect that individual paths in a multiple mediator model

have on the dependent variable has been the subject of much recent research [3, 4]. For

the international student model we again follow the method provided in [5], which extends

the counterfactual mediation analysis framework to deal with particular cases of multiple

mediators.

Again despite its inclusion, the indirect effects transmitted through program tier are not

statistically significant in any of the multiple mediator models and are minuscule compared

to the effects associated with GGPA.

Direct and indirect effects are calculated for the multiple mediator mode; effects for

which the 95% error bars do not cross zero are statistically significant. These results are

summarized in Figure 9. The single mediator results presented in Section VI B are mostly

consistent qualitatively with the results of this multiple mediator analysis. None of the

metrics have a statistically significant total effect, although GRE-P is close.

The primary difference between the single mediator and multiple mediator models is that

in the multiple mediator GRE-Q model, the total indirect effect is slightly larger than the

direct effect, whereas this is reversed in the single mediator case. Still, neither effect is statis-

29

tically significant in either the single or multiple mediator models. The confidence intervals

for the direct effect are large and clearly cross zero in both models, so this discrepancy is

not inconceivable.

[1] T. VanderWeele and S. Vansteelandt, Mediation Analysis with Multiple Mediators, Epidemi-

ologic methods 2, 95 (2014).

[2] C. W. Miller, B. M. Zwickl, J. R. Posselt, R. T. Silvestrini, and T. Hodapp, Typical physics

Ph.D. admissions criteria limit access to underrepresented groups but fail to predict doctoral

completion, Science Advances 5, eaat7550 (2019).

[3] R. M. Daniel, B. L. D. Stavola, S. N. Cousens, and S. Vansteelandt, Causal mediation analysis

with multiple mediators, Biometrics 71, 1 (2015).

[4] K. Imai and T. Yamamoto, Identification and Sensitivity Analysis for Multiple Causal Mech-

anisms: Revisiting Evidence from Framing Experiments, Political Analysis 21, 141 (2013).

[5] J. Steen, T. Loeys, B. Moerkerke, and S. Vansteelandt, Flexible Mediation Analysis With

Multiple Mediators, American Journal of Epidemiology 186, 184 (2017).

[6] J. I. Daoud, Multicollinearity and Regression Analysis, Journal of Physics: Conference Series

949, 012009 (2017).

[7] M. J. McClendon, Multiple Regression and Causal Analysis (Prospect Heights, Ill, 2002).

[8] D. E. Farrar and R. R. Glauber, Multicollinearity in regression analysis: the problem revisited,

The Review of Economic and Statistics , 92 (1967).

[9] C. Mela and P. Kopalle, The impact of collinearity on regression analysis: The asymmetric

effect of negative and positive correlations., Applied Economics 34, 667 (2002).

[10] D. S. Tull and D. I. Hawkins, Marketing Research: Measurement and Method (Prentice Hall,

New York, 1990).

[11] C. Donath, E. Graßel, D. Baier, C. Pfeiffer, S. Bleich, and T. Hillemacher, Predictors of binge

drinking in adolescents: ultimate and distal factors-a representative study, BMC public health

12, 1 (2012).

[12] G. Judge, R. C. Hill, W. E. Griffiths, H. Lutkepol, and T. C. Lee, Introduction to the Theory

and Practice of Econometrics, 2nd ed. (John Wiley and Sons Ltd, New York; Chichester,

1988).

30

[13] A. Field, J. Miles, and Z. Field, Discovering Statistics Using R, 1st ed. (SAGE Publications

Ltd, London;Thousand Oaks, Calif, 2012).

[14] C. F. Dormann, J. Elith, S. Bacher, C. Buchmann, G. Carl, G. Carre, J. R. G. Marquez,

B. Gruber, B. Lafourcade, P. J. Leitao, T. Munkemuller, C. McClean, P. E. Osborne,

B. Reineking, B. Schroder, A. K. Skidmore, D. Zurell, and S. Lautenbach, Collinearity: a

review of methods to deal with it and a simulation study evaluating their performance, Ecog-

raphy 36, 27 (2013).

[15] K. P. Vatcheva, M. Lee, J. B. McCormick, and M. H. Rahbar, Multicollinearity in Re-

gression Analyses Conducted in Epidemiologic Studies, Epidemiology (Sunnyvale, Calif.) 6,

10.4172/2161-1165.1000227 (2016).

[16] D. Tingley, T. Yamamoto, K. Hirose, L. Keele, and K. Imai, mediation: R package for causal

mediation analysis, Journal of Statistical Software 59, 1 (2014).

[17] J. Nissen, R. Donatello, and B. Van Dusen, Missing data and bias in physics education re-

search: A case for using multiple imputation, Physical Review Physics Education Research

15, 020106 (2019).

[18] S. van Buuren, Flexible Imputation of Missing Data, 2nd ed. (CRC Press, Boca Raton, FL,

2018).

[19] K. G. Moons, R. A. Donders, T. Stijnen, and F. E. Harrell, Using the outcome for imputation

of missing predictor values was preferred, Journal of Clinical Epidemiology 59, 1092 (2006).

[20] R. J. A. Little, Regression with missing x’s: A review, Journal of the American Statistical

Association 87, 1227 (1992).

[21] J. R. van Ginkel, M. Linting, R. C. A. Rippe, and A. van der Voort, Rebutting existing

misconceptions about multiple imputation as a method for handling missing data, Journal of

Personality Assessment 102, 297 (2020), https://doi.org/10.1080/00223891.2018.1530680.

31

graduate GPA?

Documents

Transcript of graduate GPA?