DANUBE: Data-driven meta-ANalysis using UnBiasedEmpirical distributions - applied to biological pathway analysis

Supplementary Material

Tin Nguyen1, Cristina Mitrea1, Rebecca Tagett1, and Sorin Draghici1,2,∗

1Department of Computer Science, Wayne State University, Detroit, Michigan, USA.2Department of Obstetrics and Gynecology, Wayne State University, Detroit, Michigan, USA.

2

Contents

1 Introduction 3

2 Description of methods 32.1 Pathway analysis methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.1.1 Gene set enrichment analysis (GSEA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.1.2 Gene set analysis (GSA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.1.3 Pathway analysis with down-weighting overlapping genes (PADOG) . . . . . . . . . . . . . 32.1.4 Signaling pathway impact analysis (SPIA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2.2 Meta-analysis methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

3 Empirical null distributions 53.1 Alzheimer’s disease . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53.2 Acute myeloid leukemia (AML) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

4 Discussion 154.1 Constructing the null distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154.2 Method bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

5 Time complexity 18

6 Meta-analysis results 19

3

1 Introduction

The supplementary material contains additional details and figures which were omitted in the main text due tospace limitations.

2 Description of methods

2.1 Pathway analysis methods

The goal of pathway analysis is to establish whether a given pathway is significantly different in some way betweentwo phenotypes (e.g. control vs. disease samples). The null and alternative hypotheses are formulated differentlyfor each pathway analysis method, mostly due to the fact that each method calculates a different statistic.

2.1.1 Gene set enrichment analysis (GSEA)

The null hypothesis of GSEA [16, 20] is that “the rank ordering of genes in a given comparison is random withregard to the diagnostic categorization of the samples”. The alternative hypothesis is that “the rank ordering ofthe pathway members is associated with the specific diagnostic criteria used to categorize the groups of affectedindividuals” [16].

Denote N as the total number of genes, GSi as the ith gene set, ni as the number of genes in the ithgeneset,(z1, z2, . . . , zni) as the t-statistic of genes in the ith gene set. For gene set GSi, GSEA computes a score S(GSi)which essentially equals to a signed version of the Kolmogorov-Smirnov statistic between the values zj (j ∈ GSi)and their complement. The samples then are permuted many times to build the empirical null distribution of thescore for each gene set. The significance of the ith gene set is determined by the fraction of the distribution thatis more extreme than the observed S(GSi).

2.1.2 Gene set analysis (GSA)

GSA differs from GSEA mainly in two ways: the summary statistic and the re-standardization of gene set scoresbased on row randomization. First, the score of the gene set is the maxmean statistic:

Smax(GSi) = max(

∑z(+)j

ni,

∑z(−)j

ni) (1)

where the (+) and (-) signs identify the positive and negative t-scores, respectively, and ni is the number of genesin the gene set. Second, GSA re-standardizes the gene set scores by taking into account scores from sets formedby random selection of genes. GSA then permutes the samples to compute the significance of the standardizedgene set scores.

2.1.3 Pathway analysis with down-weighting overlapping genes (PADOG)

The null hypothesis of PADOG [22, 23] is that the mean of the (weighted) absolute differences between thephenotypes for the genes on a given pathway is zero. The alternative hypothesis is that this mean is differentfrom zero. An alternative formulation is that the null hypothesis states that no gene on the pathway is a DEG,with the alternative stating that there is at least a gene that is a differentially expressed gene (DEG) on thegiven pathway. This formulation of the null hypothesis belongs to the self-contained category of null hypothesesaccording to [9] and in the second type of null hypotheses according to [24]. The statistic for the gene set GSi isas follows:

S(GSi) =1

ni

ni∑j=1

|T (gj)| · w(gj) (2)

where ni is the number of genes in the gene set, gj (j ∈ [1..ni]) are the genes in the gene sets, T (gj) is themoderate t-score of the gene gj , w(gj) is the weight for gene gj . A gene is weighted less if it appears in moregene sets. The score is then standardized based on row randomization. PADOG then permutes the samples tocompute the significance of the standardized gene set scores.

4

2.1.4 Signaling pathway impact analysis (SPIA)

SPIA [21] performs two simultaneous tests: one is focused on the number of differentially expressed genes (DEGs)that fall on a given pathway, while the other one focuses on the amount of perturbation accumulation observed ona pathway. The first p-value aims to characterize the enrichment of the pathway in DEGs. The null hypothesisfor this test is that the proportion of DEGs on the pathway is less than or equal to the overall proportion ofDEGs. The alternative hypothesis is that the proportion of DEGs on the pathway is higher than the overallproportion of DEGs (one-tail test for enrichment). The second test is concerned with the location, magnitudeand sign of DEGs on the given pathway. The null hypothesis is that the DEGs appear at random positions in thepathway and that they have random differential expression. The alternative hypothesis is that these DEGs arenot randomly distributed on the pathway and their direction of change is somewhat coherent with the direction ofchange of upstream genes and the previously known type of relations between genes. The null distribution of theoverall pathway perturbation accumulation is obtained by randomly permuting the DEG at different locations inthe pathway graph. The two types of evidences captured in the form of p-values (enrichment and topological) arethen combined using Fisher’s method.

2.2 Meta-analysis methods

In this work, we propose two meta-analysis methods: DANUBE and the additive method [6]. We compare theproposed methods with 5 other meta-analysis methods being used in the context of pathway analysis describedin [14, 18]. In total, we compared 7 meta-analysis methods: MetaPath, Fisher’s, Stouffer’s, Z-method, Brown’s,the additive method, and DANUBE. MetaPath [18] is a stand-alone meta-analysis method which does not needan external pathway analysis method. In our manuscript, we use the R package of MetaPath provided in [27].Stouffer’s, Fisher’s, and the additive method are used to combined independent p-values while Brown’s method isused to combine dependent p-values. The “Z-method” code provided in [14] is an extended version of Stouffer’smethod for dependent p-values. The MATLAB code for Fisher’s [10], Stouffer’s [19], Z-method [14], and Brown’smethod [4] is provided by the authors of [14].

Consider m individual null hypotheses H0i (i ∈ [1..m]) of m independent studies. The null hypothesis for theFisher’s method [10] is H0: H0i is true for all i ∈ [1..m]. The alternative hypothesis is HA: H0i is false for at leastone i ∈ [0..m]. Under the null hypothesis, all individual p-values are independently and uniformly distributedbetween zero and one. Fisher’s method uses the log product of the p-values as the test statistic, which followschi-squared distribution under the null:

X = −2

m∑i=1

ln(Pi) ∼ χ22m (3)

The test statistic of Stouffer’s method [19] is the sum of p-values transformed into standard normal variables.Denoting φ as the standard normal cumulative distribution function, pi (i ∈ [1..m]) as the individual p-values thatare independently and uniformly distributed under the the null, then we calculate the z-scores as zi = φ−1(1−pi).These z-scores follow the standard normal distribution. The summary statistic of Stouffer’s method is the sumof individual z-scores, which also follows the standard normal distributions:

Z =

∑mi=1 Zi√m

(4)

The additive method [6] uses the sum of the p-values as the test statistic, which follows the Irwin-Halldistribution [12, 13]. This distribution is used to calculate the combined p-value. In this work, we use CentralLimit Theorem to approximate the combined p-value when the number of p-values is more than 20 [7].

Brown’s method [4] and “Z-method” (described in [14]) are extensions of Fisher’s method and Stouffer’smethod, respectively. These methods utilize expected values and the covariance matrix of p-values. In [14],the covariance matrix and the expected values are estimated from the p-values of all pathways in each dataset.Brown’s method extends Fisher’s method by approximating the distribution of the test statistic using a scaledchi-squared distribution. The scaling parameter and the modified degree of the scaled chi-squared distributionare estimated based on the covariance matrix and expected values mentioned above. Similarly, the Z-methodextends Stouffer’s method to dependent p-values by assuming that the Z-scores follow a multivariate normaldistribution [14].

MetaPath [18] performs meta-analysis at both gene (MAPE G) and pathway levels (MAPE P), and thencombines the results (MAPE I) to give the final p-value and ranking of pathways. MetaPath first calculates the

5

t-statistic for each gene in each study. In MAPE G, these statistics are combined for each gene using maxP [28].The combined statistics are then used to calculate enrichment score for each pathway using a Kolmogorov-Smirnovtest. In MAPE P, the pathway enrichment analysis is done first before meta-analysis. In MAPE I, the p-values ofMAPE G and MAPE P are combined using minP [25]. In our manuscript, we use the R package provided in [27].

3 Empirical null distributions

We use 150 KEGG (version 65) human signaling pathways extracted as graph objects by the R package ROn-toTools1.2.0 (version 1.2.0). The total number of genes in the 150 pathways is 4711. We used expression datarelated to Alzheimer’s disease and acute myeloid leukemia (AML) for our data analysis. The Alzheimer’s datasetsare GSE28146 (hippocampus, 15 cases and 8 controls) and GSE5281 (6 different tissues: entorhinal cortex (EC,9 cases and 12 controls), hippocampus (HIP, 10 cases and 13 controls), medial temporal gyrus (MTG, 16 casesand 11 controls), posterior cingulate (PC, 6 cases and 13 controls), superior frontal gyrus (SFG, 22 cases and 7controls), and primary visual cortex (VCX, 16 cases and 10 controls).

The AML datasets are GSE14924 CD4 (CD4 T cells, 10 cases and 9 controls), GSE14924 CD8 (CD8 T cells,10 cases and 11 controls), GSE17054 (stem cells, 5 cases and 4 controls), GSE12662 (CD34+ cells, promyelocytes,and neutrophils and PR9 cell line, 75 cases and 24 controls), GSE57194 (CD34+ cells, 6 cases and 6 controls),GSE33223 (peripheral blood, bone marrow, 20 cases and 10 controls), GSE42140 (peripheral blood, bone marrow,26 cases and 5 controls), GSE8023 (CD34+ cells, 9 cases and 3 controls), and GSE15061 (bone marrow, 201 casesand 68 controls).

The platform for all of the datasets is the Affymetrix Human Genome U133 Plus 2.0 array. Affymetrix CELfiles containing raw expression data were downloaded from GEO for each dataset and processed using R andBioconductor version 2.13. Quality control was performed using the qc method from the package simpleaffyversion 2.38.0 [15]. Arrays were removed from the analysis if the scale factor was not in the 3-fold range of themean of all arrays. Pre-processing was performed on individual datasets using the threestep function from thepackage affyPLM version 1.38.0 [1–3]. The parameters used for the threestep function are: robust multi-arrayanalysis (RMA) background adjustment, quantile normalization, and median polish summarization.

3.1 Alzheimer’s disease

We generated a total of 40, 000 resampled datasets from 74 control samples of the 7 Alzheimer’s datasets. Werandomly label 37 as control samples and the remaining 37 as disease samples. We repeat this procedure 10, 000times to generate different groups of 37 control and 37 disease samples. In order to understand the effect ofthe group size, we also create 10, 000 datasets consisting of 10 control and 10 disease samples, 10, 000 datasetsconsisting of 10 control and 20 disease samples, and 10, 000 datasets consisting of 20 control and 10 diseasesamples. We then calculate the p-values of the KEGG (version 65) human signaling pathways (extracted as graphobjects by the R package ROntoTools1.2.0 [26]) using the following methods: Gene Set Enrichment Analysis(GSEA) [20], Gene Set Analysis (GSA) [8], Signaling Pathway Impact Analysis (SPIA) [21, 26], and Down-weighting of Overlapping Genes (PADOG) [22]. The distributions of p-values cumulated from all pathways foreach method are shown in Figure S1. The most extreme null distributions (for individual pathways) that arebiased toward zero are displayed in Figures S2, S3, S4, and S5.

3.2 Acute myeloid leukemia (AML)

Similarly, we generated totally 40, 000 resampled datasets from 140 control samples of the 9 AML datasets. Werandomly label 70 as control samples and the remaining 70 as disease samples. We repeat this procedure 10, 000times to generate different groups of 70 control and 70 disease samples. In order to understand the effect ofthe group size, we also create 10, 000 datasets consisting of 10 control and 10 disease samples, 10, 000 datasetsconsisting of 30 control and 50 disease samples, and 10, 000 datasets consisting of 30 control and 10 diseasesamples. The most extreme null distributions that are biased toward zero are displayed in Figures S6, S7, S8,and S9.

6

Distribution of p−values for all pathways (GSEA)

p−values

Den

sity

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

(A)Distribution of p−values for all pathways (GSA)

p−values

Den

sity

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

1.2

(B)

Distribution of p−values for all pathways (SPIA)

p−values

Den

sity

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

(C)Distribution of p−values for all pathways (PADOG)

p−values

Den

sity

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

(D)

Figure S1: Empirical distributions of p-values cumulated from all KEGG signaling pathways of GSEA (panel A), GSA (panel B),SPIA (panel C), and PADOG (panel D) using 74 control samples from 7 Alzheimer’s datasets. The horizontal axes display thep-values while the vertical axes display the p-value densities. The distribution of GSEA is uniform showing that GSEA is notbiased. The other 3 distributions show biases.

7

Cytosolic DNA−sensing pathway (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Autoimmune thyroid disease (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Fanconi anemia pathway (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Staphylococcus aureus infection (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

p53 signaling pathway (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Basal cell carcinoma (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Intestinal immune network for IgA production (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Mineral absorption (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Allograft rejection (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Circadian rhythm (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Parkinson’s disease (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Toll−like receptor signaling pathway (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Small cell lung cancer (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Asthma (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Natural killer cell mediated cytotoxicity (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Complement and coagulation cascades (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Systemic lupus erythematosus (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Colorectal cancer (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

NF−kappa B signaling pathway (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Olfactory transduction (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Ribosome biogenesis in eukaryotes (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Leishmaniasis (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Regulation of autophagy (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Notch signaling pathway (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Figure S2: The top 24 most biased null distributions of GSEA generated from 74 control samples of 7 Alzheimer’s datasets. Thepanels are sorted by the distribution means. The horizontal axes display the p-values while the vertical axes display the p-valuedensities. This figure shows that distributions of the p-values produced by GSEA are reasonably uniform for each of these pathways.

8

Non−small cell lung cancer (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

6

Pancreatic cancer (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

Prostate cancer (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

Focal adhesion (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

4

Endometrial cancer (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

4

Adherens junction (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

4

Renal cell carcinoma (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

Pathways in cancer (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

mTOR signaling pathway (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

ErbB signaling pathway (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Colorectal cancer (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Dorso−ventral axis formation (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Chronic myeloid leukemia (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Neurotrophin signaling pathway (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Glioma (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

GnRH signaling pathway (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Long−term depression (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Insulin signaling pathway (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Dopaminergic synapse (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Fc epsilon RI signaling pathway (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Morphine addiction (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Small cell lung cancer (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Tight junction (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

Melanoma (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

Figure S3: The 24 most extreme null distributions of GSA generated from 74 control samples of 7 Alzheimer’s datasets. The panelsare sorted by the distribution means. The horizontal axes display the p-values while the vertical axes display the p-value densities.The data show that GSA is biased towards generating lower p-values for these specific pathways.

9

Synaptic vesicle cycle (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

67

Amphetamine addiction (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

4

Dorso−ventral axis formation (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

Pathogenic Escherichia coli infection (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

6

ECM−receptor interaction (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

4

Alzheimer’s disease (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

02

46

8

SNARE interactions in vesicular transport (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Prion diseases (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

Vibrio cholerae infection (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Cardiac muscle contraction (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

4

Long−term potentiation (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Gastric acid secretion (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

Retrograde endocannabinoid signaling (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

Thyroid cancer (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

Circadian entrainment (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

GABAergic synapse (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Shigellosis (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

MAPK signaling pathway (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Long−term depression (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

Pertussis (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Axon guidance (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Protein processing in endoplasmic reticulum (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

4

Epithelial cell signaling in Helicobacter pylori infection (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

RNA transport (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

Figure S4: The 24 most extreme null distributions of SPIA generated from 74 control samples of 7 Alzheimer’s datasets. The panelsare sorted by the distribution means. The horizontal axes display the p-values while the vertical axes display the p-value densities.The data show that SPIA is biased towards generating lower p-values for these specific pathways.

10

Focal adhesion (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

02

46

Pathways in cancer (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

Adherens junction (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

4

Prostate cancer (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

Pancreatic cancer (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

Endocytosis (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Tight junction (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Regulation of actin cytoskeleton (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Non−small cell lung cancer (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

ErbB signaling pathway (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Hippo signaling pathway (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Insulin signaling pathway (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Axon guidance (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

MAPK signaling pathway (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Wnt signaling pathway (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Endometrial cancer (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Neurotrophin signaling pathway (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Chronic myeloid leukemia (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Renal cell carcinoma (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Small cell lung cancer (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Morphine addiction (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Colorectal cancer (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

PI3K−Akt signaling pathway (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

Glutamatergic synapse (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Figure S5: The 24 most extreme null distributions of PADOG generated from 74 control samples of 7 Alzheimer’s datasets. Thepanels are sorted by the distribution means. The horizontal axes display the p-values while the vertical axes display the p-valuedensities. The data show that PADOG is biased towards generating lower p-values for these pathways.

11

African trypanosomiasis (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Synaptic vesicle cycle (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Legionellosis (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Maturity onset diabetes of the young (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Pathogenic Escherichia coli infection (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Alcoholism (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

NOD−like receptor signaling pathway (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Systemic lupus erythematosus (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Retrograde endocannabinoid signaling (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Endocrine and other factor−regulated calcium reabsorption (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Endocytosis (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Vasopressin−regulated water reabsorption (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Circadian rhythm (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Sulfur relay system (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Leishmaniasis (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Cardiac muscle contraction (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Dorso−ventral axis formation (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

p53 signaling pathway (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Rheumatoid arthritis (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

ECM−receptor interaction (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Axon guidance (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

HTLV−I infection (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Phagosome (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Bile secretion (GSEA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Figure S6: The top 24 most biased null distributions of GSEA generated from 140 control samples of 9 AML’s datasets. The panelsare sorted by the distribution means. The horizontal axes display the p-values while the vertical axes display the p-value densities.This figure shows that all distributions of the p-values produced by GSEA are uniform.

12

Renal cell carcinoma (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

Endometrial cancer (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

4

ErbB signaling pathway (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

Non−small cell lung cancer (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

Focal adhesion (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

6

Adherens junction (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

Pancreatic cancer (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

4

Prostate cancer (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

4

Pathways in cancer (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

GnRH signaling pathway (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

4

Glioma (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Colorectal cancer (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Dorso−ventral axis formation (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Chronic myeloid leukemia (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Neurotrophin signaling pathway (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

mTOR signaling pathway (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Small cell lung cancer (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

MAPK signaling pathway (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Circadian rhythm (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Long−term depression (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Melanoma (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Bacterial invasion of epithelial cells (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

Insulin signaling pathway (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

VEGF signaling pathway (GSA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

Figure S7: The 24 most extreme null distributions of GSA generated from 140 control samples of 9 AML’s datasets. The panelsare sorted by the distribution means. The horizontal axes display the p-values while the vertical axes display the p-value densities.The data show that GSA is biased towards generating lower p-values for these pathways.

13

Rheumatoid arthritis (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

Transcriptional misregulation in cancer (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

Chemokine signaling pathway (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

6

Antigen processing and presentation (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

67

Cytokine−cytokine receptor interaction (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

4

Neuroactive ligand−receptor interaction (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

NF−kappa B signaling pathway (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

6

Amoebiasis (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Phagosome (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

p53 signaling pathway (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Epstein−Barr virus infection (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Hedgehog signaling pathway (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

Legionellosis (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

4

Leukocyte transendothelial migration (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Retrograde endocannabinoid signaling (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Alzheimer’s disease (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

45

Cardiac muscle contraction (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Cocaine addiction (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Bile secretion (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

1.2

Osteoclast differentiation (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

4

MAPK signaling pathway (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Acute myeloid leukemia (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Taste transduction (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

Tuberculosis (SPIA)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

4

Figure S8: The 24 most extreme null distributions of SPIA generated from 140 control samples of 9 AML’s datasets. The panelsare sorted by the distribution means. The horizontal axes display the p-values while the vertical axes display the p-value densities.The data show that SPIA is biased towards generating lower p-values for these pathways.

14

Pathways in cancer (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

02

46

81

0

Focal adhesion (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

02

46

81

0

Adherens junction (PADOG)

0.0 0.2 0.4 0.6 0.8

01

23

45

ErbB signaling pathway (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

4

Renal cell carcinoma (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

Prostate cancer (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Endocytosis (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

PI3K−Akt signaling pathway (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

4

Pancreatic cancer (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

01

23

MAPK signaling pathway (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Regulation of actin cytoskeleton (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Non−small cell lung cancer (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Small cell lung cancer (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Glioma (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

GnRH signaling pathway (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Endometrial cancer (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Insulin signaling pathway (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Neurotrophin signaling pathway (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Hippo signaling pathway (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Chronic myeloid leukemia (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Wnt signaling pathway (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

2.5

Dopaminergic synapse (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Glutamatergic synapse (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Colorectal cancer (PADOG)

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

Figure S9: The 24 most extreme null distributions of PADOG generated from 140 control samples of 9 AML’s datasets. The panelsare sorted by the distribution means. The horizontal axes display the p-values while the vertical axes display the p-value densities.The data show that PADOG is biased towards generating lower p-values for these pathways.

15

4 Discussion

4.1 Constructing the null distributions

In our work, we put together control samples of different experiments in order to understand the bias (if present)of a given pathway analysis method when random groups of healthy people are compared. In principle, the groupsof healthy controls coming from different experiments may have bias introduced by: normalization techniques,different laboratory protocols, different tissues, etc. These differences are not important because they are uniformlydistributed among the groups. When we pooled all control samples together, we formed a population which shouldbe the reference. This is similar with a situation in which a population can have different sub-groups based onethnicity, gender, race, or living conditions with important differences between subgroups. When one randomlyselects samples (for the 2 random groups to be compared) from the reference population, one expects all bias (e.g.ethnic subgroups) to be represented equally in both random groups. Therefore, there should be no differencebetween these random groups, no matter how many distinct subgroups were present in the original population.Therefore, the p-values of a test for difference between the two randomly selected groups should be uniformlydistributed between zero and one.

In order to demonstrate this, we are including here the results of an experiment in which we pool samplesfrom 3 different populations and then compare the randomly sampled groups from the pooled population (seeFigure S10). Using R, we constructed 3 normal populations with a standard deviation of 1 and means of 1, 5,and 10, respectively (top panel in Figure S10). These model our different control groups coming from differentexperiments. In this example, each has 1, 000 samples. We pooled the three populations together to form asingle “reference” population with 3000 samples (second panel from the top in Figure S10). We then randomlydivided the pooled samples into 2 groups (each group has 1500 samples) and then compared the two groups usingboth a parametric approach (t-test [11, 17]), as well as a non-parametric approach (Mood’s median test [5]).We did this 5,000 times and constructed the distribution of the p-values generated by each test (bottom panelsin Figure S10). Uniform distributions are obtained from both tests even though the pooled populations havedifferences in means as large as 9 standard deviations (the 1st population, leftmost blue, has mean 1 while the3rd population, rightmost, purple, has mean 10 and all have standard deviation of 1). This shows that thedistribution of the p-values is still uniform, even though the reference population is formed by pooling togetherdifferent populations with different means. In other words, neither the fact that the controls come from differentexperiments, nor the different normalization techniques, nor any other specific trait of the data coming from anyone of the experiments should produce a deviation from uniformity in the distribution of the p-values, if thesep-values are produced by an unbiased statistical test.

4.2 Method bias

Panel A in Figure S1 shows the distribution of the p-values produced by GSEA when testing randomly formedgroups of control samples from 7 Alzheimer’s data sets. The distribution of these p-values is almost perfectlyuniform showing both that i) indeed our data correctly models the null and ii) GSEA is an unbiased test. Panel Bin the same Figure S1 shows the distribution of the p-values produced by GSA from exactly the same data. Thisdistribution shows that GSA will produce more false positives than expected by chance. Panels C and D in thesame figure show the distributions of the p-value produced by SPIA and PADOG, respectively. These methods arebiased. Note that all distributions were constructed using the exact same data and exact same random groupingof samples. Therefore, the non-uniformity of the distributions is due to the methods rather than the data.

It is important to note that each pathway analysis method is applied as proposed by its authors. In fact, eachpathway analysis method builds its own null and tests it appropriately. These nulls are constructed includingboth the disease control samples in each data set. Our novel contribution is in the way we correct for the biasintroduced by each method. We capture the bias of each method by repeatedly analyzing the differences betweentwo groups drawn randomly from the same population. The subgroups that are included in this population arenot important since each subgroup will be sampled proportionally and included in each of the two groups testedin each iteration. In order to demonstrate this, we have reconstructed these distributions of the p-values producedby GSEA, GSA, SPIA, and PADOG using all samples, including the disease samples from all experiments. Thedistributions are shown in Figure S11. This figure shows that there is no difference between the distributions ofthe p-values obtained from a reference population that included only the controls from all experiments, versus areference population that includes both controls and diseases from all experiments.

16

Mood's median test

Histogram of p−values for Mood’s median test

p−values

Density

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

Histogram of p−values for t−test

p−values

Density

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

1500samples

p1

p2

...

pi

...

pN=5000

DC DC DC DC

... ...

t-test

randomselection

Histogram of samples from 3 subgroups

Sample values

Den

sity

0 5 10

0.0

0.1

0.2

0.3

0.4

Histogram of the pooled samples

Samples values

Den

sity

0 5 10

0.00

0.04

0.08

0.12

Figure S10: Constructing the reference population by pooling populations with different means does not introduce any bias. Thetop panel shows 1,000 values drawn from each of three distinct normal populations with means 1, 5, and 10, respectively andstandard deviations of 1. The next panel shows the “reference” population of 3000 values. Random samples of 1500 values for“controls” and 1500 values for “disease” are drawn repeatedly from this standard population. A test is performed between thesetwo groups with either a parametric approach (t-test) or a non-parametric one (Mood’s median test). The last two panels showthe distributions of these p-values for both tests. Note that the distribution of the p-values of both tests are very close to uniformeven though the reference population was constructed by mixing samples from three very distinct populations.

17

Distribution of p−values using controls (GSEA)

p−values

Den

sity

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

(A1)Distribution of p−values using all samples (GSEA)

p−values

Den

sity

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

(A2)

Distribution of p−values using controls (GSA)

p−values

Den

sity

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

1.2

(B1)Distribution of p−values using all samples (GSA)

p−values

Den

sity

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

1.2

(B2)

Distribution of p−values using controls (SPIA)

p−values

Density

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

(C1)Distribution of p−values using all samples (SPIA)

p−values

Den

sity

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.5

1.0

1.5

2.0

(C2)

Distribution of p−values using controls (PADOG)

p−values

Den

sity

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

(D1)Distribution of p−values using all samples (PADOG)

p−values

Den

sity

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.2

0.4

0.6

0.8

1.0

(D2)

Figure S11: Comparison between the distribution of the p-values obtained from a reference population constructed using onlycontrol samples (left column) and using all samples (right column) for GSEA (A1-A2), GSA (B1-B2), SPIA (C1-C2), and PADOG(D1-D2). Each of the left panels show the empirical null distribution of p-values generated from 40, 000 simulated datasets usingcontrol samples of Alzheimer’s datasets. Each of the right panels show the empirical null distribution of p-values generated from4, 000 simulated datasets using both control and disease of Alzheimer’s datasets. The figure shows that the bias profiles are almostidentical whether the reference population is constructed from controls alone or both controls and disease samples.

18

5 Time complexity

The data analysis is done on our Linux server X8OBNF, Intel E7-8837 that has 1TB RAM (64 X 16GB DDR3,1067MHz) and multi-core CPU (64 cores, 8 chips, 8 cores/chip, Intel Xeon E7-8837, 2.67GHz). The runningtime of the pathway analysis methods for real datasets is reported in Table S1. Once we have the individualp-values computed, it takes less than a second to combine them using the 5 classical meta-analysis (Stouffer’s,Z-method, Brown’s, Fisher’s method, and the additive method). Therefore, the running time of these 5 meta-analysis methods in conjunction with a pathway analysis method is the time to compute the individual p-values.For example, the time needed for GSA combined with Stouffer’s, Z-method, Brown’s, Fisher’s method, or theadditive method is 7m for Alzheimer’s data and is 13m for AML data. We also report the running time forMetaPath. MetaPath combines multiple studies without an external pathway analysis method and therefore doesnot require an additional explicit meta-analysis step.

The time needed for DANUBE consists of three parts. First, we need to generate the null distributions(Table S2). Second, we need to calculate the p-values for the real datasets (Table S1). Third, we need to calculateand combine the empirical p-values. The third part takes only some seconds and therefore is negligible. Therefore,the running time of DANUBE in conjunction with a pathway analysis method is the time needed to generate theempirical distributions and to calculate the p-values using a pathway analysis method. For example, the timeneeded for DANUBE combined with GSA to analyze Alzheimer’s data is 11h25m + 7m = 11h32m.

Note that we need to create the empirical distribution only once. Once the empirical distribution of thep-values generated by a particular pathway analysis method is calculated, it is subsequently reused for any otheranalyses.

Table S1: Running time of classical meta-analysis methods (Stouffer’s, Z-method, Brown’s, Fisher’s, the additive method combinedwith GSA, GSEA, SPIA, and PADOG) and MetaPath for Alzheimer’s and AML data using a single core (processor). The runningtime is rounded to minutes (m).

Disease Dataset GSA GSEA SPIA PADOG MetaPath

Alzheimer’s

GSE28146 1m 5m 3m 12m

1h13m

GSE5281 EC 1m 5m 3m 11mGSE5281 HIP 1m 5m 3m 11mGSE5281 MTG 1m 6m 3m 12mGSE5281 PC 1m 6m 3m 11mGSE5281 SFG 1m 6m 3m 14mGSE5281 VCX 1m 6m 3m 13m

Meta-analysis 7m 39m 21m 1h24m –

AML

GSE14924 CD4 1m 5m 2m 14m

1h14m

GSE14924 CD8 1m 5m 2m 14mGSE17054 1m 5m 2m 12mGSE12662 2m 5m 5m 15mGSE57194 1m 5m 2m 10mGSE33223 1m 5m 3m 11mGSE42140 1m 6m 3m 11mGSE8023 1m 6m 2m 10mGSE15061 3m 9m 13m 27m

Meta-analysis 13m 51m 34m 2h04m –

Table S2: Running time of each pathway analysis on simulated datasets. The running time is rounded to minutes (m).

Data Groups Number of GSA GSEA SPIA PADOGsimulations Cores Time Cores Time Cores Time Cores Time

Alzheimer’s

10 vs. 10 10,000 60 2h21m 60 20h10m 60 3h36m 20 92h37m10 vs. 20 10,000 60 2h44m 60 18h33m 60 4h04m 20 94h12m20 vs. 10 10,000 60 2h32m 60 18h07m 60 3h58m 20 90h35m37 vs. 37 10,000 60 3h48m 60 19h59m 60 4h11m 20 107h15m

Total 40,000 60 11h25m 60 76h49m 60 15h49m 20 384h39m

AML

10 vs. 10 10,000 60 2h24m 60 17h23m 60 4h11m 40 49h21m30 vs. 50 10,000 60 3h58m 60 19h36m 60 5h03m 40 63h09m50 vs. 30 10,000 60 3h53m 60 19h32m 60 4h42m 40 61h50m70 vs. 70 10,000 60 3h44m 60 19h31m 60 5h23m 40 78h22m

Total 40,000 60 13h59m 60 76h02m 60 19h19m 40 252h42m

19

Rankings of the target pathway "Acute myeloid leukemia"

Ran

king

s

GSE14924_CD4 GSE17054 GSE57194 GSE42140 GSE15061GSE14924_CD8 GSE12662 GSE33223 GSE8023

020

4060

8010

012

0

Datasets

(A)

GSEAGSASPIAPADOG

p−values of the target pathway "Acute myeloid leukemia"

FD

R c

orre

cted

p−

valu

es

GSEAGSASPIAPADOG

GSE14924_CD4 GSE17054 GSE57194 GSE42140 GSE15061GSE14924_CD8 GSE12662 GSE33223 GSE8023

00.

20.

40.

60.

81

Datasets

(B)

Figure S12: Ranks (A) and p-values (B) of the KEGG target pathway, Acute myeloid leukemia, for 9 acute myeloid leukemia (AML)datasets, using the pathway analysis methods: GSEA, GSA, SPIA, and PADOG. The horizonal axes show the 9 AML datasets.The vertical axis in panel (A) shows the rankings of the target pathway for each dataset using the 4 methods. The vertical axis inpanel (B) shows the FDR-corrected p-values of the target pathway. The horizontal red line in (B) shows the threshold 0.01. Therankings and p-values of the target pathway vary greatly across different datasets and methods, so biological interpretation of theresults is difficult.

6 Meta-analysis results

Figure S12 displays the rankings and FDR-corrected p-values of the target pathway Acute myeloid leukemia forthe 9 AML datasets. Table S3 displays the results obtained by combining SPIA p-values using 6 meta-analysismethods for Alzheimer’s disease. Tables S4 displays the results obtained by combining GSEA p-values using 6meta-analysis methods for Alzheimer’s disease. Tables S6 and S7 display those for acute myeloid leukemia (AML)data.

20

Table S3: The 20 top ranked pathways and FDR-corrected p-values obtained by combining the SPIA p-values using 6 meta-analysismethods for Alzheimer’s disease. The target pathway Alzheimer’s disease is significant for all methods. However, the existingmethods report many other significant pathways, several of which are likely to be false positives.

SPIA + Stouffer’s method SPIA + Z-method SPIA + Brown’s method

Pathway pvalue.fdr Pathway pvalue.fdr Pathway pvalue.fdr

1 Parkinson’s disease < 10−4 Parkinson’s disease < 10−4 Parkinson’s disease < 10−4

2 Alzheimer’s disease < 10−4 Alzheimer’s disease < 10−4 Synaptic vesicle cycle < 10−4

3 Synaptic vesicle cycle < 10−4 Synaptic vesicle cycle < 10−4 Alzheimer’s disease < 10−4

4 Huntington’s disease < 10−4 Huntington’s disease < 10−4 Huntington’s disease < 10−4

5 Pathogenic Escherichia coli in-fection

< 10−4 Pathogenic Escherichia coli in-fection

0.0006 Pathogenic Escherichia coli in-fection

< 10−4

6 Cardiac muscle contraction < 10−4 Cardiac muscle contraction 0.0015 Phagosome 0.0003

7 Phagosome < 10−4 Phagosome 0.0034 Cardiac muscle contraction 0.0003

8 Mineral absorption < 10−4 Mineral absorption 0.0232 Mineral absorption 0.0003

9 Vibrio cholerae infection < 10−4 Vibrio cholerae infection 0.0247 Vibrio cholerae infection 0.006410 Endocrine and other factor-

regulated calcium reabsorp-tion

0.0003 Endocrine and other factor-regulated calcium reabsorp-tion

0.0846 Endocrine and other factor-regulated calcium reabsorp-tion

0.0274

11 Epstein-Barr virus infection 0.0006 Epstein-Barr virus infection 0.1054 Long-term potentiation 0.029312 GABAergic synapse 0.0011 GABAergic synapse 0.1281 Retrograde endocannabinoid

signaling0.0293

13 Long-term potentiation 0.0045 Long-term potentiation 0.2162 GABAergic synapse 0.029314 Epithelial cell signaling in He-

licobacter pylori infection0.0052 Epithelial cell signaling in He-

licobacter pylori infection0.2203 RNA transport 0.0324

15 Gap junction 0.0103 Gap junction 0.2786 Glutamatergic synapse 0.040016 RNA transport 0.0138 RNA transport 0.3015 Epstein-Barr virus infection 0.053317 Gastric acid secretion 0.0297 Gastric acid secretion 0.3990 Epithelial cell signaling in He-

licobacter pylori infection0.0536

18 HIF-1 signaling pathway 0.0378 HIF-1 signaling pathway 0.4242 Gap junction 0.104519 Axon guidance 0.0579 Axon guidance 0.4242 Long-term depression 0.143320 Long-term depression 0.0579 Long-term depression 0.4242 Axon guidance 0.1630

SPIA + Fisher’s method SPIA + Additive method SPIA + DANUBE

Pathway pvalue.fdr Pathway pvalue.fdr Pathway pvalue.fdr

1 Parkinson’s disease < 10−4 Parkinson’s disease < 10−4 Parkinson’s disease 0.0051

2 Synaptic vesicle cycle < 10−4 Alzheimer’s disease < 10−4 Alzheimer’s disease 0.0081

3 Alzheimer’s disease < 10−4 Synaptic vesicle cycle 0.0002 Synaptic vesicle cycle 0.0081

4 Huntington’s disease < 10−4 Huntington’s disease 0.0002 Cardiac muscle contraction 0.0081

5 Pathogenic Escherichia coli in-fection

< 10−4 Pathogenic Escherichia coli in-fection

0.0006 Huntington’s disease 0.0081

6 Phagosome < 10−4 Cardiac muscle contraction 0.0006 Pathogenic Escherichia coli in-fection

0.0104

7 Cardiac muscle contraction < 10−4 Phagosome 0.0046 Epstein-Barr virus infection 0.0104

8 Mineral absorption < 10−4 Epstein-Barr virus infection 0.0046 Phagosome 0.0140

9 Vibrio cholerae infection < 10−4 Vibrio cholerae infection 0.0065 Vibrio cholerae infection 0.0374

10 Endocrine and other factor-regulated calcium reabsorp-tion

< 10−4 Endocrine and other factor-regulated calcium reabsorp-tion

0.0328 Endocrine and other factor-regulated calcium reabsorp-tion

0.0584

11 Long-term potentiation 0.0001 GABAergic synapse 0.0459 HIF-1 signaling pathway 0.071212 Retrograde endocannabinoid

signaling0.0001 Lysosome 0.0459 Lysosome 0.0842

13 GABAergic synapse 0.0001 Epithelial cell signaling in He-licobacter pylori infection

0.0459 p53 signaling pathway 0.0842

14 RNA transport 0.0002 Amyotrophic lateral sclerosis(ALS)

0.0543 Epithelial cell signaling in He-licobacter pylori infection

0.1161

15 Glutamatergic synapse 0.0003 Gastric acid secretion 0.0546 Amyotrophic lateral sclerosis(ALS)

0.1161

16 Epstein-Barr virus infection 0.0007 Melanogenesis 0.0568 Bacterial invasion of epithelialcells

0.1390

17 Epithelial cell signaling in He-licobacter pylori infection

0.0007 HIF-1 signaling pathway 0.0594 GABAergic synapse 0.1551

18 Gap junction 0.0035 Endocytosis 0.0678 Gap junction 0.174419 Long-term depression 0.0077 Gap junction 0.0681 Melanogenesis 0.177420 Axon guidance 0.0110 Long-term potentiation 0.0731 Mineral absorption 0.1774

For each meta-analysis method, the 7 p-values for each pathway (one of each of the 7 datasets) were combined into a single p-value.This is done for all of the 150 signaling pathways in KEGG, resulting in 150 combined p-values. The p-values are then adjustedfor multiple comparisons using FDR. The pathways are sorted by the combined p-values, from low to high. The horizontal linesshow the 1% significance threshold. The target pathway Alzheimer’s disease is highlighted in green.

21

Table S4: The 11 top ranked pathways and FDR-corrected p-values obtained by combining the GSEA p-values using 6 meta-analysismethods for Alzheimer’s disease. The additive method and DANUBE yield similar results because GSEA has no bias. These twomethods rank the target pathway Alzheimer’s disease higher than other methods.

GSEA + Stouffer’s method GSEA + Z-method GSEA + Brown’s method

Pathway pvalue.fdr Pathway pvalue.fdr Pathway pvalue.fdr

1 Cardiac muscle contraction < 10−4 Cardiac muscle contraction < 10−4 Cardiac muscle contraction < 10−4

2 Serotonergic synapse < 10−4 Serotonergic synapse < 10−4 Serotonergic synapse < 10−4

3 Dopaminergic synapse < 10−4 Dopaminergic synapse < 10−4 Dopaminergic synapse < 10−4

4 Alzheimer’s disease < 10−4 Alzheimer’s disease < 10−4 Alzheimer’s disease < 10−4

5 Parkinson’s disease < 10−4 Parkinson’s disease < 10−4 Parkinson’s disease < 10−4

6 Amyotrophic lateral sclerosis(ALS)

< 10−4 Amyotrophic lateral sclerosis(ALS)

< 10−4 Amyotrophic lateral sclerosis(ALS)

< 10−4

7 Huntington’s disease < 10−4 Huntington’s disease < 10−4 Huntington’s disease < 10−4

8 Arrhythmogenic right ventric-ular cardiomyopathy (ARVC)

< 10−4 Arrhythmogenic right ventric-ular cardiomyopathy (ARVC)

< 10−4 Arrhythmogenic right ventric-ular cardiomyopathy (ARVC)

< 10−4

9 Ribosome biogenesis in eu-karyotes

0.0021 Ribosome biogenesis in eu-karyotes

0.0559 Ribosome biogenesis in eu-karyotes

0.0516

10 RNA transport 0.0126 RNA transport 0.1491 RNA transport 0.088511 Notch signaling pathway 0.0176 Notch signaling pathway 0.1739 Notch signaling pathway 0.1305

GSEA + Fisher’s method GSEA + Additive method GSEA + DANUBE

Pathway pvalue.fdr Pathway pvalue.fdr Pathway pvalue.fdr

1 Cardiac muscle contraction < 10−4 Cardiac muscle contraction 0.0004 Cardiac muscle contraction 0.0005

2 Serotonergic synapse < 10−4 Huntington’s disease 0.0004 Huntington’s disease 0.0005

3 Dopaminergic synapse < 10−4 Alzheimer’s disease 0.0004 Alzheimer’s disease 0.0005

4 Alzheimer’s disease < 10−4 Parkinson’s disease 0.0005 Parkinson’s disease 0.0006

5 Parkinson’s disease < 10−4 Ribosome biogenesis in eu-karyotes

0.0169 Ribosome biogenesis in eu-karyotes

0.0179

6 Amyotrophic lateral sclerosis(ALS)

< 10−4 Prostate cancer 0.0629 Prostate cancer 0.0614

7 Huntington’s disease < 10−4 Oocyte meiosis 0.0645 Oocyte meiosis 0.0625

8 Arrhythmogenic right ventric-ular cardiomyopathy (ARVC)

< 10−4 PI3K-Akt signaling pathway 0.0892 PI3K-Akt signaling pathway 0.0930

9 Ribosome biogenesis in eu-karyotes

0.0039 Notch signaling pathway 0.0892 Notch signaling pathway 0.0930

10 RNA transport 0.0100 Basal cell carcinoma 0.0892 Basal cell carcinoma 0.093711 Notch signaling pathway 0.0197 Chemokine signaling pathway 0.0907 Chemokine signaling pathway 0.0937

For each meta-analysis method, the 7 p-values for each pathway (one of each of the 7 datasets) were combined into a single p-value.This is done for all of the 150 signaling pathways in KEGG, resulting in 150 combined p-values. The p-values are then adjustedfor multiple comparisons using FDR. The pathways are sorted by the combined p-values, from low to high. The horizontal linesshow the 1% significance threshold. The target pathway Alzheimer’s disease is highlighted in green.

Table S5: MetaPath result for 7 Alzheimer’s datasets. The target pathway Alzheimer’s disease is not significant and is outrankedby 6 other pathways.

MetaPathPathway pvalue.fdr

1 Parkinson’s disease 0.00602 Vasopressin-regulated water reabsorption 0.03073 Synaptic vesicle cycle 0.03604 Vibrio cholerae infection 0.03925 Huntington’s disease 0.06846 Pathogenic Escherichia coli infection 0.06867 Alzheimer’s disease 0.0735

22

Table S6: The 12 top ranked pathways and FDR-corrected p-values obtained by combining the SPIA p-values using 6 meta-analysismethods for acute myeloid leukemia (AML). The target pathway Acute myeloid leukemia is significant for Stouffer’s, Fisher’s, theadditive method, and DANUBE. DANUBE and the additive method have the best ranking.

SPIA + Stouffer’s method SPIA + Z-method SPIA + Brown’s method

Pathway pvalue.fdr Pathway pvalue.fdr Pathway pvalue.fdr

1 Transcriptional misregulationin cancer

< 10−4 Transcriptional misregulationin cancer

< 10−4 Transcriptional misregulationin cancer

< 10−4

2 Cell cycle < 10−4 Cell cycle 0.0121 Cell cycle 0.0002

3 Viral carcinogenesis < 10−4 Viral carcinogenesis 0.0121 RNA transport 0.0035

4 Acute myeloid leukemia < 10−4 Acute myeloid leukemia 0.0317 Viral carcinogenesis 0.0035

5 p53 signaling pathway < 10−4 p53 signaling pathway 0.0364 p53 signaling pathway 0.01046 RNA transport 0.0002 RNA transport 0.0677 Acute myeloid leukemia 0.02117 Colorectal cancer 0.0072 Colorectal cancer 0.2742 Osteoclast differentiation 0.02898 T cell receptor signaling path-

way0.0084 T cell receptor signaling path-

way0.2742 Colorectal cancer 0.0487

9 HTLV-I infection 0.0281 HTLV-I infection 0.4142 HTLV-I infection 0.062010 Epstein-Barr virus infection 0.0294 Epstein-Barr virus infection 0.4142 Cytokine-cytokine receptor in-

teraction0.1123

11 Osteoclast differentiation 0.0432 Osteoclast differentiation 0.4569 Phagosome 0.131112 Phagosome 0.0465 Phagosome 0.4569 T cell receptor signaling path-

way0.1611

SPIA + Fisher’s method SPIA + Additive method SPIA + DANUBE

Pathway pvalue.fdr Pathway pvalue.fdr Pathway pvalue.fdr

1 Transcriptional misregulationin cancer

< 10−4 Transcriptional misregulationin cancer

< 10−4 Transcriptional misregulationin cancer

0.0001

2 Cell cycle < 10−4 Acute myeloid leukemia 0.0004 Acute myeloid leukemia 0.0022

3 Viral carcinogenesis < 10−4 Viral carcinogenesis 0.0004 Viral carcinogenesis 0.0022

4 RNA transport < 10−4 p53 signaling pathway 0.0114 p53 signaling pathway 0.0422

5 p53 signaling pathway < 10−4 T cell receptor signaling path-way

0.0268 T cell receptor signaling path-way

0.0809

6 Acute myeloid leukemia < 10−4 Cell cycle 0.0405 Hippo signaling pathway 0.08097 Osteoclast differentiation 0.0001 Epstein-Barr virus infection 0.0480 Cell cycle 0.08098 Colorectal cancer 0.0005 RNA transport 0.0965 Small cell lung cancer 0.20419 HTLV-I infection 0.0010 Hepatitis B 0.1839 Alcoholism 0.204110 Cytokine-cytokine receptor in-

teraction0.0040 Alcoholism 0.1920 RNA transport 0.2041

11 Phagosome 0.0062 Colorectal cancer 0.2187 Epstein-Barr virus infection 0.204112 T cell receptor signaling path-

way0.0115 HTLV-I infection 0.2187 Pathways in cancer 0.2041

For each meta-analysis method, the 7 p-values for each pathway (one of each of the 7 datasets) were combined into a single p-value.This is done for all of the 150 signaling pathways in KEGG, resulting in 150 combined p-values. The p-values are then adjustedfor multiple comparisons using FDR. The pathways are sorted by the combined p-values, from low to high. The horizontal linesshow the 1% significance threshold. The target pathway Acute myeloid leukemia is highlighted in green.

Table S7: The 10 top ranked pathways and FDR-corrected p-values obtained by combining the GSEA p-values using 6 meta-analysismethods for acute myeloid leukemia (AML). Since GSEA has no bias, the additive method and DANUBE yield similar results. Thetarget pathway Acute myeloid leukemia is not significant for any of the methods.

GSEA + Stouffer’s method GSEA + Z-method GSEA + Brown’s method

Pathway pvalue.fdr Pathway pvalue.fdr Pathway pvalue.fdr

1 Acute myeloid leukemia 0.0998 Acute myeloid leukemia 0.2417 Cocaine addiction 0.45122 Alcoholism 0.0998 Alcoholism 0.2417 Amphetamine addiction 0.45123 Cocaine addiction 0.0998 Cocaine addiction 0.2417 Alcoholism 0.45124 Amphetamine addiction 0.1966 Amphetamine addiction 0.4017 Acute myeloid leukemia 0.46045 Pancreatic secretion 0.3086 Pancreatic secretion 0.5247 Pancreatic secretion 0.55636 ErbB signaling pathway 0.3995 ErbB signaling pathway 0.5247 Allograft rejection 0.55637 Pathways in cancer 0.4281 Pathways in cancer 0.5247 Ribosome biogenesis in eu-

karyotes0.5879

8 Gastric acid secretion 0.4281 Gastric acid secretion 0.5247 Fanconi anemia pathway 0.58799 Gap junction 0.4408 Gap junction 0.5247 ErbB signaling pathway 0.587910 p53 signaling pathway 0.4408 p53 signaling pathway 0.5247 p53 signaling pathway 0.5879

GSEA + Fisher’s method GSEA + Additive method GSEA + DANUBE

Pathway pvalue.fdr Pathway pvalue.fdr Pathway pvalue.fdr

1 Cocaine addiction 0.2454 Acute myeloid leukemia 0.1125 Acute myeloid leukemia 0.10362 Amphetamine addiction 0.2454 Alcoholism 0.1216 Alcoholism 0.12213 Alcoholism 0.2454 Cocaine addiction 0.1216 Cocaine addiction 0.12214 Acute myeloid leukemia 0.2648 Gastric acid secretion 0.3409 Gastric acid secretion 0.33935 Allograft rejection 0.3559 Pancreatic secretion 0.3409 Pancreatic secretion 0.33936 Pancreatic secretion 0.3559 ErbB signaling pathway 0.3409 ErbB signaling pathway 0.33937 Graft-versus-host disease 0.4198 Amphetamine addiction 0.4882 Amphetamine addiction 0.48408 Pathways in cancer 0.4575 Pathways in cancer 0.4935 Pathways in cancer 0.48409 ErbB signaling pathway 0.4575 Gap junction 0.4935 Gap junction 0.484010 Salivary secretion 0.4575 VEGF signaling pathway 0.4935 VEGF signaling pathway 0.4840

For each meta-analysis method, the 7 p-values for each pathway (one of each of the 7 datasets) were combined into a single p-value.This is done for all of the 150 signaling pathways in KEGG, resulting in 150 combined p-values. The p-values are then adjustedfor multiple comparisons using FDR. The pathways are sorted by the combined p-values, from low to high. The target pathwayAcute myeloid leukemia is highlighted in green.

23

Table S8: MetaPath results for 9 acute myeloid leukemia datasets. The target pathway Acute myeloid leukemia is not significantand is outranked by two other pathways.

MetaPathPathway pvalue.fdr

1 Thyroid cancer 0.26802 Circadian rhythm 0.33203 Acute myeloid leukemia 0.40754 Gap junction 0.72285 Staphylococcus aureus infection 0.9966

24

References

[1] Bolstad, B. M. (2004). Low-level analysis of high-density oligonucleotide array data: background, normalization and sum-

marization. Ph.D. thesis, University of California.

[2] Bolstad, B. M., Collin, F., Brettschneider, J., Simpson, K., Cope, L., Irizarry, R., and Speed, T. P. (2005). Quality assessment

of Affymetrix GeneChip data. In Bioinformatics and computational biology solutions using R and Bioconductor , pages 33–47.

Springer, New York.

[3] Brettschneider, J., Collin, F., Bolstad, B. M., and Speed, T. P. (2008). Quality assessment for short oligonucleotide microarray

data. Technometrics, 50(3).

[4] Brown, M. B. (1975). A method for combining non-independent, one-sided tests of significance. Biometrics, pages 987–992.

[5] Conover, W. J. and Conover, W. (1980). Practical nonparametric statistics. Wiley New York, New York.

[6] Edgington, E. S. (1972a). An additive method for combining probability values from independent experiments. The Journal

of Psychology, 80(2), 351–363.

[7] Edgington, E. S. (1972b). A normal curve method for combining probability values from independent experiments. The Journal

of Psychology, 82(1), 85–89.

[8] Efron, B. and Tibshirani, R. (2007). On testing the significance of sets of genes. The Annals of Applied Statistics, 1(1),

107–129.

[9] Emmert-Streib, F. and V. Glazko, G. (2011). Pathway Analysis of Expression Data: Deciphering Functional Building Blocks

of Complex Diseases. PLoS Computational Biology, 7(5), e1002053.

[10] Fisher, R. A. (1925). Statistical methods for research workers. Oliver & Boyd, Edinburgh.

[11] Gosset, W. S. (1908). The Probable Error of a Mean. Biometrika, 6, 1–25.

[12] Hall, P. (1927). The distribution of means for samples of size n drawn from a population in which the variate takes values

between 0 and 1, all such values being equally probable. Biometrika, 19(3-4), 240–244.

[13] Irwin, J. O. (1927). On the frequency distribution of the means of samples from a population having any law of frequency

with finite moments, with special reference to Pearson’s Type II. Biometrika, 19(3-4), 225–239.

[14] Kaever, A., Landesfeind, M., Feussner, K., Morgenstern, B., Feussner, I., and Meinicke, P. (2014). Meta-analysis of pathway

enrichment: combining independent and dependent omics data sets. PloS One, 9(2), e89297.

[15] Miller, C. J. (2013). simpleaffy: Very simple high level analysis of Affymetrix data. R package version 2.38.0.

[16] Mootha, V. K., Lindgren, C. M., Eriksson, K.-F., Subramanian, A., Sihag, S., Lehar, J., Puigserver, P., Carlsson, E., Rid-

derstrale, M., Laurila, E., Houstis, N., Daly, M. J., Patterson, N., Mesirov, J. P., Golub, T. R., Tamayo, P., Spiegelman, B.,

Lander, E. S., Hirschhorn, J. N., Altshuler, D., and Groop, L. C. (2003). PGC-11α-responsive genes involved in oxidative

phosphorylation are coordinately downregulated in human diabetes. Nature Genetics, 34(3), 267–273.

[17] Peaeson, E. and Haetlet, H. (1976). Biometrika tables for statisticians. Biometrika Trust.

[18] Shen, K. and Tseng, G. C. (2010). Meta-analysis for pathway enrichment analysis when combining multiple genomic studies.

Bioinformatics, 26(10), 1316–1323.

[19] Stouffer, S., Suchman, E., DeVinney, L., Star, S., and Williams, RM, J. (1949). The American Soldier: Adjustment during

army life, volume 1. Princeton University Press, Princeton.

[20] Subramanian, A., Tamayo, P., Mootha, V. K., Mukherjee, S., Ebert, B. L., Gillette, M. A., Paulovich, A., Pomeroy, S. L.,

Golub, T. R., Lander, E. S., and Mesirov, J. P. (2005). Gene set enrichment analysis: a knowledge-based approach for interpreting

genome-wide expression profiles. Proceeding of The National Academy of Sciences of the Unites States of America, 102(43),

15545–15550.

[21] Tarca, A. L., Draghici, S., Khatri, P., Hassan, S. S., Mittal, P., Kim, J.-s., Kim, C. J., Kusanovic, J. P., and Romero, R.

(2009). A novel signaling pathway impact analysis. Bioinformatics, 25(1), 75–82.

[22] Tarca, A. L., Draghici, S., Bhatti, G., and Romero, R. (2012). Down-weighting overlapping genes improves gene set analysis.

BMC Bioinformatics, 13(1), 136.

[23] Tarca, A. L., Bhatti, G., and Romero, R. (2013). A comparison of gene set analysis methods in terms of sensitivity, prioritization

and specificity. PloS One, 8(11), e79217.

[24] Tian, L., Greenberg, S. A., Kong, S. W., Altschuler, J., Kohane, I. S., and Park, P. J. (2005). Discovering statistically

significant pathways in expression profiling studies. Proceeding of The National Academy of Sciences of the USA, 102(38),

13544–13549.

25

[25] Tippett, L. H. C. (1931). The methods of statistics. Williams & Norgate, London.

[26] Voichita, C. and Draghici, S. (2013). ROntoTools: R Onto-Tools suite. R package.

[27] Wang, X., Kang, D. D., Shen, K., Song, C., Lu, S., Chang, L.-C., Liao, S. G., Huo, Z., Tang, S., Ding, Y., Kaminski, N., Sibille,

E., Lin, Y., Li, J., and Tseng, G. C. (2012). An R package suite for microarray meta-analysis in quality control, differentially

expressed gene analysis and pathway enrichment detection. Bioinformatics, 28(19), 2534–2536.

[28] Wilkinson, B. (1951). A statistical consideration in psychological research. Psychological Bulletin, 48(2), 156.