Global Explanations of Neural NetworkMapping the Landscape of Predictions

Mark IbrahimCenter for Machine Learning, Capital One

Melissa LouieCenter for Machine Learning, Capital One

Ceena ModarresCenter for Machine Learning, Capital One

John PaisleyColumbia University

ABSTRACTA barrier to the wider adoption of neural networks is their lackof interpretability. While local explanation methods exist for oneprediction, most global attributions still reduce neural network deci-sions to a single set of features. In response, we present an approachfor generating global attributions called GAM, which explains thelandscape of neural network predictions across subpopulations.GAM augments global explanations with the proportion of samplesthat each attribution best explains and specifies which samplesare described by each attribution. Global explanations also havetunable granularity to detect more or fewer subpopulations. Wedemonstrate that GAM’s global explanations 1) yield the knownfeature importances of simulated data, 2) match feature weights ofinterpretable statistical models on real data, and 3) are intuitive topractitioners through user studies. With more transparent predic-tions, GAM can help ensure neural network decisions are generatedfor the right reasons.

CCS CONCEPTS• Computing methodologies → Semantic networks; Super-vised learning by classification.

KEYWORDSneural networks, global interpretability, explainable deep learningACM Reference Format:Mark Ibrahim, Melissa Louie, Ceena Modarres, and John Paisley. 2019. GlobalExplanations of Neural Network: Mapping the Landscape of Predictions.In 2019 AAAI/ACM Conference on AI, Ethics, and Society (AIES’19), Janu-ary 27–28, 2019, Honolulu, HI, USA. ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/3306618.3314230

1 INTRODUCTIONThe present decade has seen the widespread adoption of neuralnetworks across many areas from natural language translation tovisual object recognition [14]. In domains where the stakes are highhowever, adoption has been limited due to a lack of interpretability.The hierarchical non-linear nature of neural networks limits the

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org.AIES ’19, January 27–28, 2019, Honolulu, HI, USA© 2019 Association for Computing Machinery.ACM ISBN 978-1-4503-6324-2/19/01. . . $15.00https://doi.org/10.1145/3306618.3314230

ability to explain predictions. This black-box nature exposes therisk of adverse societal consequences such as bias and prejudice.

Researchers have developed several approaches to explain neuralnetwork predictions through an observation’s saliency or uniqueidentifying factors [10][24][18]. Another set of techniques pro-duce explanations using attributions, which estimate each feature’simportance to a prediction. Some of these techniques are modelagnostic while others attempt to take advantage of a neural net-work’s architecture to generate attributions [17][11] [2]. However,these techniques are local, meaning the explanations are limited toa single prediction.

Global attributions can be a powerful tool for interpretability,since they explain feature importance across a population. Globalexplanations can aid in model diagnostics, uncover opportunitiesfor feature engineering, and expose possible bias. Existing globaltechniques rely on more interpretable surrogate models such asdecision trees or manipulate the input space to assess global pre-dictive power [8] [27] [13]. While these techniques produce aninterpretable set of rules, they can fail to capture the non-linearfeature interactions learned by neural networks. Submodular-PickLIME (SP-LIME) is another technique based on summarizing lo-cal attributions to better reflect learned interactions. AlthoughSP-LIME maximizes coverage of local explanations, it does not de-scribe the proportion or subpopulations best explained by eachglobal attribution [20].

We propose a global attribution method called GAM (Global At-tribution Mapping) that’s able to explain non-linear representationslearned by neural networks across subpopulations. GAM groupssimilar local feature importances to form human-interpretableglobal attributions. Each global attribution best explains a particularsubpopulation, augmenting global explanations with the proportionand specific samples they best explain. In addition, GAM providestunable granularity to capture more or fewer subpopulations forglobal explanations.

2 BACKGROUND: ATTRIBUTIONTECHNIQUES

GAM builds on local attribution techniques. Here we considerthree techniques that are rooted in a strong theoretical foundation,well cited, and relatively easy to implement: Locally InterpretableModel-Agnostic Explanations (LIME), Integrated Gradients, andDeep Learning Important FeaTures (DeepLIFT) [20] [25] [23].

Locally Interpretable Model-Agnostic Explanations (LIME). LIMEproduces interpretable representations of complicated models by

optimizing two metrics: interpretability and local fidelity. In prac-tice, an input is perturbed by sampling in a local neighborhoodto fit a simpler linear model. The now explainable linear model’sweights can be used to interpret a particular model prediction.

There is a global extension of LIME called Submodule pick LIME(SP-LIME). SP-LIME reduces many local attributions into a smallerglobal set by selecting the least redundant set of local attributions.Formally, SP-LIME maximizes coverage defined as the total impor-tance of the features in the selected global set.

Integrated Gradients. Sundararajan et. al developed an approachthat satisfies two proposed axioms for attributions. First, sensitivitystates that for any two inputs that differ in only a single featureand have different predictions, the attribution should be non-zero.Second, implementation invariance states that two networks thathave equal outputs for all possible inputs should also have the sameattributions. The authors develop an approach that takes the pathintegral of the gradient for a neutral reference input to determinethe feature dimension with the greatest activation. An attributionvector for a particular observation (local) is produced as a result.

Deep Learning Important FeaTures (DeepLIFT). Much like Inte-grated Gradients, DeepLIFT seeks to explain the difference in outputbetween an input and a neutral reference. An attribution score iscalculated by constructing a function analogous to a partial deriva-tive that’s used along with the chain rule to trace the change in theoutput layer relative to the input.

3 GAM: GLOBAL ATTRIBUTION MAPPINGWe treat each local attribution as a ranking of features. In contrastto domains such as speech or image recognition where individualinputs are not human-interpretable, here we focus on situationswhere features have well defined semantics [4]. We then group sim-ilar attributions to form global human-interpretable explanationsacross subpopulations.

3.1 Attributions as RankingsAttributions contain more than the content encoded by treatingeach as a vector in Rn : an attribution ranks and weighs each fea-ture’s association with a particular prediction. Each element in anattribution vector answers the question: "how important is thisfeature for a particular prediction?" Furthermore, attributions areconjoined rankings, because every feature appears in every attribu-tion along the same dimension. Therefore to incorporate both therank and weight encoded in an attribution, we treat attributions asweighted conjoined rankings. With this treatment, we summarize thecontent of many local attributions by comparing their similaritiesto form global attributions.

3.2 Rank DistancesWe represent each attribution as a rank vector σ . The unweightedrank of feature i is σ (i) and the weighted rank is σw (i).

First, to ensure distances appropriately reflect similarity amongvectors—as opposed to anomalies in the scale of the original inputs—we normalize each local attribution vector σw by

|σw | 1∑i |σw (i)| ,

whereσw is theweighted attribution vector and is theHadamardproduct.

By transforming the attribution into normalized percentages, weconsider only feature importance, not whether a feature contributespositively or negatively to a particular target. This enables us tomeasure similarity among normalized attributions as weightedrankings of feature importance.

Nextwe consider two options for comparing attributions: Kendall’sTau and Spearman’s Rho squared rank distances. The first optionfor comparing attributions is weighted Kendall’s Tau rank distance,an extension of Kendall’s Tau rank correlation as defined by Leeand Yu 2010 [16].

For a second attribution vector π , we can define the weightedKendall’s Tau distance between π and σ as∑

i<jwiw j I (π (i) − π (j))(σ (i) − σ (j)) < 0, (1)

wherewi = πw (i)σw (i) and I is the indicator function.A feature’s weight, wi , in this context is the product of the

weights in each ranking. Two rankings containing all elementsin the same order (possibly different weights) have zero distanceto imply the same feature importance. Two rankings with at leastone element in a different order have a distance proportional to theout-of-order elements’ weights. When a feature is ranked above orbelow in one ranking compared to the other, it is penalized by theproduct of its weights in the rankings. The total distance betweentwo rankings is then the sum of the penalty for feature appearing ina different order. The more features that appear in a different order,the greater the distance between the rankings. Furthermore, theweighted Kendall’s Tau distance described satisfies symmetry, posi-tive definiteness, and subadditivity (see Section 2 of [15]) meaningit defines a proper distance metric that we can use for clusteringrankings.

A second option for comparing attributions is weighted Spear-man’s Rho squared rank distance, as defined by Shieh et al. [22],

k∑i=1

πw (i)σw (i)(π (i) − σ (i))2. (2)

Weighted Spearman’s Rho squared rank distance as defined com-pares attributions as weighted conjoined rankings much in the sameway as weighted Kendall’s Tau rank distance, but with a runtime ofO(n logn). Weighted Spearman’s Rho squared distance also definesa distance metric [15].

Kendall’s Tau rank distance does have computational and math-ematical limitations. First, Kendall’s Tau has a quadratic runtime,which can be computationally expensive as the number of featureand/or samples grows [12]. Second, Kendall’s Tau tends to producesmaller values compared to Spearman’s Rho, making differencesbetween attributions less pronounced [7].

Therefore, in situations where the number of features and/orsamples is large, Spearman’s Rho squared is a more effective alter-native from a computational perspective and due to its ability tomake differences among rankings more pronounced. Despite theadvantages of Spearman’s Rho squared rank distance, Kendall’s Tauremains an appropriate choice when computational efficiency is lessimportant due to its desirable statistical properties. Kendall’s Tau as

a correlation metric has faster asymptotic convergence with respectto bias and variance [26]. Having defined metrics with which tocompare attributions, we now turn our attention to identifyingsimilar attributions.

3.3 Grouping Similar Local AttributionsClustering algorithms can detect global patterns in a dataset bygrouping similar data points into a cluster. We apply this idea in thecontext of local attribution vectors by transforming many local at-tributions into a few global attributions. We use weighted Kendall’sTau or Spearman’s Rho squared rank distances to measure similar-ity among attributions. We adapt K-medoids to use rank distancerather than Manhattan distance to group similar attributions [19].We then use the resulting medoids to succinctly summarize globalattribution patterns.

The clustering algorithm identifies the K local attributions (medoids)minimizing the pairwise dissimilarity within a cluster relative tothe medoid. We then use the medoid to summarize the patterndetected for each cluster. The algorithm works by initializing Klocal attributions (medoids)m1, ...,mK selected randomly from thefull set of local attributions. The local attributions are reassigneduntil convergence or for a sufficiently large number of iterations,each taking O(n(n − K)2). The choice of medoids at each iterationis updated by

(1) determining cluster membership for each local attribution σvia the nearest medoid,

σ ∈ Ca if rankDist(σ ,ma ) ≤ rankDist(σ ,mj ) ∀j ∈ [1,K],(2) updating each cluster’s medoid by minimizing pairwise dis-

similarity within a cluster,

ma = σl = arдminσl ∈Ca∑

σj ∈CarankDist(σj ,σl ).

The advantage this clustering approach over other algorithmssuch as K-means is that it allows us to compare attributions asconjoined weighted rankings rather than simply vectors in RN .K-means, for example, treats attributions as vectors by minimiz-ing inter-cluster variance in Euclidean space, ignoring the contentencoded in the rank and weights of each feature. Our approach,on the other hand, takes advantage of both the rank and weightinformation in local attributions during clustering. Each clusterspecifies a subpopulation best explained by its medoid.

3.4 Generating Global AttributionsEach of GAM’s global explanations yields the most centrally lo-cated vector of feature importances for a subpopulation. The globalattribution landscape described by GAM is then the collection ofglobal explanations for each subpopulation.

Finally, we form GAM’s global explanations across subpopula-tions by grouping similar normalized attributions.

1em [t] local attributions medoids and corresponding members 1.Normalize the set of local attributions local attribution normalized= abs(attribution) / sum(abs(attribution))

2. Compute pair-wise rank distance matrix distances = [] attri-bution1 in normalizedAttributions attribution2 in normalizedAttri-butions distances += rankDistance(attribution1, attribution2)

3. Cluster Attributions initialMedoids = random.choice(attributions)

x iterations cluster attribution in cluster tempMedoid = attri-bution cost = sum(distance(attribution, tempMedoid)) reassignmedoid to attribution minimizing cost update cluster membershipby assigning to closest medoid

We can gauge the explanatory power of each global attributionby the size of its subpopulation. Since GAM allows us to tracea global explanation to individual samples, we can also examinesummary statistics for each subpopulation. In addition, by adjustingthe cluster sizeK , we can tune the granularity of global explanationsto capture more or fewer subpopulations. As a starting point, K canbe chosen based the Silhouette score, which compares similaritywithin and across clusters [21].

In contrast to global explanations via surrogate models, whichproduce a single global explanation, GAM identifies differences inexplanations among subpopulations. Furthermore, the ability totrace global explanations to individual samples, supplements globalexplanations of techniques such as SP-LIMEwith information aboutsubpopulations.

4 VALIDATING OUR PROPOSED GLOBALATTRIBUTION METHOD

Although there has been significant research in the validation ofclustering algorithms, interpreting clusters involves qualitativeanalysis and domain expertise. Moreover, research in interpretabil-ity inherently faces the challenge of effective and reliable validation[6]. Attributions have no baseline truth. The task of identifying anappropriate validation methodology for a proposed approach is anopen research question. In this paper, we validate the methodologyin three ways.

The first validates GAM against two synthetically generateddatasets with known feature importances. The second compares theglobal attributions generated by GAM against the feature weightsof more interpretable statistical models. The third assesses thepractical value of GAM in real-world settings with user studies. Wealso demonstrate the use of Kendall’s Tau and Spearman’s Rho rankdistances as well as several local attribution techniques.

4.1 Global Attributions Applied to SyntheticDataset

We design a simple synthetic dataset to yield known feature im-portances against which to validate GAM’s global attributions. Thedataset consists of a binary label, class 1 or 2, and two features, Aand B. We aim to generate one group for which only feature A ispredictive and another group for which only feature B is predictive.

For the group where only feature A is predictive, we samplebalanced classes from uniform distributions where

feature A =U [0, 1] if class 1U [1, 2] if class 2

. (3)

To ensure feature B has no predictive power for this group, wedraw from a uniform distribution over [0, 2].

Since values for feature A in [0, 1] mark class 1 and [1, 2] markclass 2 samples, feature A has full predictive power. Feature B onthe other hand is sampled uniformly over the same range for bothclasses, yielding no predictive power.

Conversely, to produce a groupwhere only feature B is predictive,we sample balanced classes where

feature B =U [3, 4] if class 1U [4, 5] if class 2

. (4)

To ensure feature A has no predictive power for this group, wedraw from a uniform distribution over [3, 5].

We then conduct two experiments: 1. using a balanced set wherefeature A is predictive in 50% of cases and 2. using an unbalancedset where feature A is predictive in 75% of cases. We train a singlelayer feed forward neural network on a balanced dataset of 10ksamples. We use a ReLU activation with a four node hidden layerand a sigmoid output. We hold out 2000 test samples and achieve avalidation accuracy of 99%.

In order to generate global attributions for the 2000 balancedtest samples, we first generate local attributions using LIME. Wethen generate global attributions using GAM across several valuesof K . By computing the Silhouette Coefficient score, we find K = 2optimizes the similarity of attributions within the subpopulations.We obtain the corresponding size of the subpopulations and twoexplanation medoids summarizing feature importances (see Table1).

Table 1: Global Attributions for Synthetic Data

BalancedExplanations Subpopulation Size[feature A: 0.93feature B: 0.07

]953[

feature A: 0.05feature B: 0.95

]1047

Table 2: Global Attributions for Unbalanced Synthetic Data

UnbalancedExplanations Subpopulation Size[feature A: 0.94feature B: 0.06

]1436[

feature A: 0.13feature B: 0.87

]564

For the balanced experiment, the global explanations and sub-population proportions match expectations. The first subpopulationcontains nearly half the samples and assigns a high importanceweight to feature A. The second subpopulation contains the remain-ing half of samples and assigns a high importance weight to featureB.

For the unbalanced experiment, the global attributions containtwo subpopulations with the expected proportions (see Table 2).The first assigns a high attribution to feature A and contains 72%of samples. The second assigns a high attribution to feature B andcontains 28% of samples, reflecting the feature importances inherentto the synthetic dataset. See Appendix for discussion on DeepLIFTand Integrated Gradients results.

Overall, the global attributions produced by GAM match theknown feature importance and proportions of both synthetic datasets.

Figure 1: DeepLIFT- Five most important features for eachglobal attribution explanation and the corresponding num-ber of samples in each subpopulation.

4.2 Global Attributions Applied to theMushroom Dataset

Next, we apply GAM to a dataset with a larger number of categor-ical features: The Audubon Society’s mushroom dataset [5]. Thetask is to classify mushroom samples as poisonous or not poisonoususing 23 categorical features such as cap-shape, habitat, odor andso on. Here we highlight the use of gradient-based neural networkattribution techniques (DeepLIFT and Integrated Gradients) to gen-erate local attributions and compare the results against LIME. Wealso choose weighted Spearman’s Rho squared rank distance tocompare attributions for computational efficiency given the largernumber of features and samples.

For the classification task, we train a two hidden-layer feed-foward neural network on 4784 samples (witholding 2392 for vali-dation) based on the architecture proposed in the Kaggle Kernel [9].We one-hot encode the categorical features into a 128-unit inputlayer. The network optimizes binary cross entropy with dropoutusing sigmoid activation functions. We obtain a validation accuracyof approximately 99%. This allows us to produce more accuratelocal attributions by minimizing inaccuracies due to model error.

We apply GAMon three sets of local attributions on 1000 samplesusing DeepLIFT, Integrated Gradients, and LIME [1]. We identifythree global explanations based on each technique. We note thetop features did vary slightly in weight and rank depending on ourchoice of baseline (for DeepLIFT/Integrated Gradients). Here wechoose the baseline yielding a nearly uniform classification output.

The global attributions based on DeepLIFT produce three sub-populations with similar top features (see Figure 1). Odor, habitat,and spore-print color appear among the top five features across allthree subpopulations. Each subpopulation also surfaces distinctionsin the relative feature importances. Spore-print color appears to bethe most important features for 54% of samples, while odor is most

Figure 2: Integrated Gradients - Global Attributions

important for the remaining subpopulation. Gill-size also appearsto be relatively more important for subpopulation A, whereas gill-spacing is more so for subpopulation B. Despite slight variations inrankings, on average 8 out of the top 10 features agree across allthree subpopulations, implying overall concordance in the surfacedtop features (see Appendix for full attributions).

The global attributions based on Integrated Gradients also pro-duce explanations ranking odor and spore-print-color among thetop five features. There appears to also be strong overlap amongother top features such as habitat, population, and stalk-shape (seeFigure 2). Nevertheless, each subpopulation also surfaces nuancesin the top features for each group. For example, population is amongthe top 5 features for 74% of samples, whereas it’s relatively lessimportant for the remaining samples. Across all three subpopula-tions however, on average 7 of the top 10 features agree, once againimplying concordance in the top features.

In addition, we find both gradient-based global attributions over-lap in the top features for global attributions based on LIME. Globalattributions for LIME also rank odor, spore-print-color, population,and gill-color among the top five features. The three subpopula-tions based on LIME produced similar top five features (see Figure3). Despite slight variation in rankings, all three local techniquessurface similar most important features.

Finally, we validate global attributions across all three local tech-niques against the known feature importances of a gradient boost-ing tree classifier. We use the XGBoost classifier implementationand assign feature importances using each feature’s F1 score [3]. Wefind on average 7 out of the top 10 features in the global attributionsacross all three techniques match the top features of XGBoost. Odor,spore-print, population, gill-color, and stalk root appear among themost important features (see Figure 4). Furthermore, the weightsof the top features of XGBoost match closely the weights of globalattributions based on LIME. Although global XGBoost feature im-portances corroborate the most important features surfaced, we

Figure 3: LIME - Global Attributions

Figure 4: Feature importances for XGBoost as measured byeach feature’s F1 score.

qualify these findings by noting that two equally performing mod-els can learn a different set of feature interactions. Nevertheless, inthis case the global attributions’ similarity to known global featureimportances across multiple local techniques suggests the expla-nations generated effectively uncover associations among featuresand the classification task.

We found similar agreement in the top features using GAM onthe Iris dataset with global explanations matching the weights of alogistic classifier (see Appendix: GAM Applied to the Iris Dataset).

4.3 User StudiesWe conduct several user studies on FICO Home Equity Line Creditapplications to assess the practical value of GAM in real-worldsettings. The dataset provides information on credit candidate ap-plications and risk outcome characterized by 90-day payment delin-quencies.

First, we simulated a scenario where a practitioner would useGAM to conduct feature selection. We selected seven predictivefeatures and added in three additional uniformly distributed noisefeatures. We dropped all feature names, trained a neural networkto make credit decisions, produced a GAM model explanation, andpresented the explanation to 55 Amazon Mechanical Turk users.We provided users with GAM explanations in the form of a barchart and asked users to remove the three least important features.In 94% of cases, respondents selected the features with the lowestimportance corresponding to the random noise features.

Second, we polled credit modeling experts to assess whetherGAM’s results are as intuitive as those of the popular SP-LIMEglobal attribution method. We present both explanations to a groupof 55 credit modeling practitioners and asked which explanationallows them to more intuitively improve the model (see Appendix).We found respondents had nearly equal preference among the threecategories: GAM, SP-LIME, and no preference.

Our sample sizes are relatively small, but the results suggest thatGAM has real world practical value in machine learning tasks likefeature selection and produces results that are intuitive to humanpractitioners.

5 CONCLUSIONIn this paper, we develop GAM, a methodology for generatingglobal attributions to supplement existing interpretability tech-niques for neural networks. GAM’s global explanations describethe non-linear representations learned by neural networks. GAMalso provides tunable subpopulation granularity along with the abil-ity to trace global explanations to specific samples. We demonstrateon both real and synthetic datasets that GAM illuminates globalexplanation patterns across learned subpopulations. We validatethat global attributions produced by GAMmatch known feature im-portances and are insightful to humans through user studies. Withexplanations across subpopulations, neural network predictionsare more transparent. A possible next step is to explore how globalinterpretability can help ensure fairness in AI algorithms.

ACKNOWLEDGMENTSThank you to Paul Zeng for his contribution to the implementationand valuable feedback. An implementation for GAM is available athttps://github.com/capitalone/global-attribution-mapping .

REFERENCES[1] Marco Ancona, Enea Ceolini, Cengiz ÃŰztireli, and Markus Gross. 2018. Towards

better understanding of gradient-based attribution methods for Deep NeuralNetworks. In International Conference on Learning Representations. https://openreview.net/forum?id=Sy21R9JAW

[2] Alexander Binder, Grégoire Montavon, Sebastian Bach, Klaus-Robert Müller, andWojciech Samek. 2016. Layer-wise Relevance Propagation for Neural Networkswith Local Renormalization Layers. CoRR abs/1604.00825 (2016). arXiv:1604.00825http://arxiv.org/abs/1604.00825

[3] Tianqi Chen and Carlos Guestrin. 2016. XGBoost: A Scalable Tree BoostingSystem. In Proceedings of the 22Nd ACM SIGKDD International Conference onKnowledge Discovery and Data Mining (KDD ’16). ACM, New York, NY, USA,785–794. https://doi.org/10.1145/2939672.2939785

[4] Anupam Datta, Shayak Sen, and Yair Zick. 2016. Algorithmic transparency viaquantitative input influence: Theory and experiments with learning systems. InSecurity and Privacy (SP), 2016 IEEE Symposium on. IEEE, 598–617.

[5] Dua Dheeru and Efi Karra Taniskidou. 2017. UCI Machine Learning Repository:Mushroom Data Set. https://archive.ics.uci.edu/ml/datasets/mushroom

[6] Finale Doshi-Velez and Been Kim. 2017. Towards a rigorous science of inter-pretable machine learning. arXiv preprint arXiv:1702.08608 (2017).

[7] Gregory A. Fredricks and Roger B. Nelsen. 2007. On the relationship betweenSpearman’s rho and Kendall’s tau for pairs of continuous random variables.Journal of Statistical Planning and Inference 137, 7 (2007), 2143 – 2150. https://doi.org/10.1016/j.jspi.2006.06.045

[8] Nicholas Frosst and Geoffrey Hinton. 2017. Distilling a neural network into asoft decision tree. arXiv preprint arXiv:1711.09784 (2017).

[9] Youngwook Kim. 2018. Kaggle: Modeling on Mushroom Dataset. https://www.kaggle.com/kt66nf/quick-keras-modeling-on-mushroom-dataset/notebook

[10] Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Kristof T. Schutt, Maximil-ian Alber, Sven Dahne, Dumitru Erhan, and Been Kim. 2018. The (Un)reliabilityof saliency methods. (2018).

[11] Pieter-Jan Kindermans, Kristof T. Schutt, Maximilian Alber, Klaus-Robert Muller,Dumitru Erhan, Been Kim, and Sven Dahne. 2018. Learning how to explainneural networks: PatternNet and PatternAttribution. (2018). https://openreview.net/forum?id=Hkn7CBaTW

[12] William R. Knight. 1966. A Computer Method for Calculating Kendall’s Tauwith Ungrouped Data. J. Amer. Statist. Assoc. 61, 314 (1966), 436–439. http://www.jstor.org/stable/2282833

[13] Himabindu Lakkaraju, Stephen H Bach, and Jure Leskovec. 2016. Interpretabledecision sets: A joint framework for description and prediction. In Proceedings ofthe 22nd ACM SIGKDD international conference on knowledge discovery and datamining. ACM, 1675–1684.

[14] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature521, 7553 (2015), 436.

[15] Paul Lee and Philip Yu. 2012. Mixtures of weighted distance-based models forranking data with applications in political studies. 56 (08 2012), 2486–2500.

[16] Paul H. Lee and Philip L. H. Yu. 2010. Distance-based Tree Models for RankingData. Comput. Stat. Data Anal. 54, 6 (June 2010), 1672–1682. https://doi.org/10.1016/j.csda.2010.01.027

[17] Scott Lundberg and Su-In Lee. 2017. A unified approach to interpreting modelpredictions. CoRR abs/1705.07874 (2017). arXiv:1705.07874 http://arxiv.org/abs/1705.07874

[18] Grégoire Montavon, Sebastian Bach, Alexander Binder, Wojciech Samek, andKlaus-Robert Müller. 2015. Explaining NonLinear Classification Decisions withDeep Taylor Decomposition. CoRR abs/1512.02479 (2015). arXiv:1512.02479http://arxiv.org/abs/1512.02479

[19] Hae-Sang Park and Chi-Hyuck Jun. 2009. A Simple and Fast Algorithm forK-medoids Clustering. Expert Syst. Appl. 36, 2 (March 2009), 3336–3341. https://doi.org/10.1016/j.eswa.2008.01.039

[20] Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should ITrust You?": Explaining the Predictions of Any Classifier. CoRR abs/1602.04938(2016). arXiv:1602.04938 http://arxiv.org/abs/1602.04938

[21] Peter J Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation andvalidation of cluster analysis. Journal of computational and applied mathematics20 (1987), 53–65.

[22] Grace S. Shieh, Zhidong Bai, and Wei-Yann Tsai. 2000. RANK TESTS FOR IN-DEPENDENCE âĂŤ WITH A WEIGHTED CONTAMINATION ALTERNATIVE.Statistica Sinica 10, 2 (2000), 577–593. http://www.jstor.org/stable/24306733

[23] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. 2017. Learning Impor-tant Features Through Propagating Activation Differences. CoRR abs/1704.02685(2017). arXiv:1704.02685 http://arxiv.org/abs/1704.02685

[24] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. Deep InsideConvolutional Networks: Visualising Image Classification Models and SaliencyMaps. CoRR abs/1312.6034 (2013). arXiv:1312.6034 http://arxiv.org/abs/1312.6034

[25] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic Attributionfor Deep Networks. CoRR abs/1703.01365 (2017). arXiv:1703.01365 http://arxiv.org/abs/1703.01365

[26] Weichao Xu, Yunhe Hou, Y.S. Hung, and Yuexian Zou. 2013. A comparativeanalysis of Spearman’s rho and Kendall’s tau in normal and contaminated normalmodels. Signal Processing 93, 1 (2013), 261 – 276. https://doi.org/10.1016/j.sigpro.2012.08.005

[27] Chengliang Yang, Anand Rangarajan, and Sanjay Ranka. 2018. Global ModelInterpretation via Recursive Partitioning. arXiv preprint arXiv:1802.04253 (2018).

A RESEARCH METHODSsectionAppendix

B BACKGROUND ON LOCALINTERPRETABILITY TECHNIQUES

B.1 Locally Interpretable Model-AgnosticExplanations (LIME)

LIME produces interpretable representations of complicated modelsby optimizing two metrics: interpretability Ω and local fidelity L.Defined as,

ζ (x) = argminд∈GL(f ,д,πx ) + Ω(д). (5)

In Equation 1, the complexity or comprehensibility Ω of an in-terpretable function д ∈ G (e.g. non-zero coefficients of a linearmodel) is optimized alongside the fidelity L or faithfulness of д inapproximating true function f in local neighborhood πx .

In practice, an input point x is perturbed by random samplingin a local neighborhood and a simpler linear model is fit with thenewly constructed synthetic data set. The method is model agnostic,which means it can be applied to neural networks or any otheruninterpretable model. The now explainable linear model’s weightscan be used to interpret a particular model prediction.

B.2 Integrated GradientsSundararajan et. al developed an approach that satisfies two pro-posed axioms for attributions. First, sensitivity states that for anytwo inputs that differ in only a single feature and have differentpredictions, the attribution should be non-zero. Second, implemen-tation invariance states that two networks that have equal outputsfor all possible inputs should also have the same attributions. Inthis paper, the authors develop an approach that takes the pathintegral of the gradient for a particular point xi and the model’sinference F on the path (α ) between a zero information baseline x ′and the input x .

IGi (x) ::= (xi − x ′i )∫ 1

α=0

∂ f (x ′ + α × (x − x ′))∂xi

dα

The path integral between the baseline and the true input canbe approximated with Riemman sums. An attribution vector for aparticular observation (local) is produced as a result. The authorsfound that 20 to 300 steps can sufficiently approximate the integralwithin 5%. Selecting a baseline vector remains an open researchquestion. A domain-specific heuristic (e.g. a black image in imageclassification) has been recommended for baseline selection.

B.3 Deep Learning Important FeaTures(DeepLIFT)

Much like Integrated Gradients, DeepLIFT seeks to explain thedifference in output ∆t between a ’reference’ or ’baseline’ input t0and the original input t for a neuron output: ∆t = t − t0. For eachinput xi , an attribution score is calculated C∆xi∆t that should sumup to the total change in the output ∆t . DeepLIFT is best defined

by this described summation-to-delta property:n∑i=1

C∆xi∆t = ∆t

Shrikumar et. al then construct a function analogous to a partialderivative and use the chain rule to trace the attribution score fromthe output layer to the original input. This method also requires areference vector and the authors recommend leveraging domainknowledge to select an optimal baseline input.

C INTEGRATED GRADIENTS AND DEEPLIFTANALYSIS FOR SYNTHETIC DATASET

While our local attributions using LIME did vary slightly depend-ing our choice of kernel-width, we find local attributions to beextremely sensitive to our choice of a baseline input. Particularly,the number of points in each cluster varies greatly depending onour choice of baseline for Integrated Gradients. Both feature im-portances and subpopulations varied greatly for DeepLIFT. Unfor-tunately in this context there is no natural choice of baselines. Toselect a baseline for Integrated Gradients, we ran a grid searchacross our range of inputs within +/-.005 to produce a neutralprediction. We obtain two subpopulations using this baseline forIntegrated Gradients. The first contains 610 samples and assigns anattribution of 89% to feature A. The second contains the remaining1390 and assigns a 99% attribution to feature B. We demonstratethat our global attribution technique is local-technique agnosticby generating new attributions using Integrated Gradients andDeepLIFT.

Both feature importances and subpopulations varied greatly forDeepLIFT. To select a baseline for Integrated Gradients, we ran agrid search across our range of inputs within +/-.005 to produce aneutral prediction. We obtain two subpopulation using this base-line for Integrated Gradients. The first contains 610 samples andassigns an attribution of 89% to feature A. The second contains theremaining 1390 and assigns a 99% attribution to feature B.

D GAM APPLIED TO THE IRIS DATASETWe also evaluate the proposed global attribution method on thewell known Iris dataset (Anderson, 1936; Fisher, 1936). The task isto classify Iris flowers into one of three species (setosa, virginica,and versicolor) based on the length and width of each sample’spetals and sepals. We train a 2-hidden layer feed forward networkon 75% of the Iris data and validate on the remaining. We use asoftmax activation function as the final output layer and ReLU acti-vation functions for all other layers. The trained network achievesa validation accuracy of 90%.

As with the synthetic dataset, we generate local attributions us-ing LIME. We did not obtain attributions using Integrated Gradientsor DeepLIFT because that method relies on the use of a baselinevector to determine the attributions, which is not as clearly definedfor the Iris dataset and defining one is beyond the scope of thispaper. We then conduct a silhouette analysis to determine the opti-mal number of clusters for our global attribution method. We findthe peak silhouette score of 0.95 is achieved with three clusters, incontrast to 0.65 for two clusters and 0.78 for four clusters. We thenapply our global attribution method using three subpopulations

Figure 5: Attribution Subpopulations in Kendall Tau’s RankDistance Space. The attributions are represented as nodes ina force-directed graph where the distance between any twonodes is proportional to their Kendall Tau’s rank distance.This allows us to visualize the relative rank distance amonglocal attributions and their cluster medoids (represented asnodes with larger diameter). Subpopulation 1 and subpop-ulation 2 contain attributions associated with predictionsfor the setosa and versicolor flower species. Subpopulation3 contains attributions associated with virginica predictionsand a single versicolor prediction.

and extract three explainations to summarize the global attributionpattern.

To validate our results, we first plot the global attribution clus-ters in a force-directed graph to visualize Kendall Tau’s rank dis-tance among attributions. We find the global attributions formwell-isolated clusters in rank distance space (see Figure 5). Eachcluster groups a set of similar attributions in rank space, which issuccinctly summarized using each cluster’s medoid.

We find each attribution cluster corresponds to a target class.While this structure need not arise in the context of every prob-lem, the attributions here can be explained nicely in terms of theclasses associated with each prediction (a single exception beingone versicolor prediction in subpopulation 3). We did find slightvariation in the assignment of nearby attribution vectors, namelythose in the region between subpopulation 1 and subpopulation3, but the cluster structure appeared stable across many iterationswith distinct clusters.

Table 3: Attribution Subpopulation Explanations

Feature Subpop. A Subpop. B Subpop. Csepal lengthsepal widthpetal lengthpetal width

0.010.610.030.35

0.150.310.370.17

0.110.010.400.48

Each subpopulation’s explanation succinctly summarizes the fea-

ture importances for each species’ predictions (see Table 3). Sepal

and petal width are most important for setosa predictions (subpopu-lation A), whereas sepal width and petal length are most importantfor versicolor predictions (subpopulation B). Petal features turn outto be most important for virginica predictions (subpopulation C).We can then compare these features against the weights of a base-line classifier to assess whether the medoids reasonably capturethe correct set of feature importances.

Table 4: Absolute Weights of Logistic Classifiers

Feature setosa versicolor virginicasepal lengthsepal widthpetal lengthpetal width

0.871.311.661.45

0.041.230.910.83

0.330.171.392.32

We additionally validate our approach by comparing the setof feature importances from our global attributions against theweights learned from a baseline one-vs-rest logistic regression clas-sifier. We find the coefficients from the logistic regression classifierscorrespond directly to the weights we obtain from our global attri-bution method. We find sepal width, petal length, and petal weightsto have the largest coefficients across the three classes, similar towhat we found across all three attribution medoids (see Table 4).Furthermore, we find the largest absolute coefficients across allclasses (except for petal length for setosa) correspond directly tothe feature importances uncovered by our global attributionmethod.For example, the largest weights for virginica predictions are thesame petal features highlighted by the global attribution medoidfor subpopulation C.

It is worth noting that two classifiers may learn a different setof feature interactions. In this case, the feature interactions sur-faced by our global attribution method correspond to the featureweights learned by a one-vs-rest logistic regression classifier. Thisprovides evidence not only that the number of clusters and clusterpatterns are meaningful, but also that the specific feature impor-tances uncovered in each cluster effectively describe global modelattribution.

E MUSHROOM GLOBAL ATTRIBUTIONSGlobal Attributions with explanations for all features across thethree techniques.

E.1 Global Attributions with DeepLIFT

E.2 Global Attributions with IntegratedGradients

E.3 Global Attributions with LIME

F USER STUDIESWe asked 55 MTurk users to identity the top five features based onGAM’s global explanations with two subpopulations on the FICOdataset (see figures below).

In 97% of cases respondents selected the meaningful features forsubpopulation one and 92% for subpopulation two.

We also generated identical attributions for SP-LIME with B=2and asked 55 credit modeling practitioners which explanation theypreferred. In this case, the raw feature names were also displayedwithout a method label (see figures). Below are aggregate anony-mous responses.