Decision Tree Extractionfrom Trained Neural Networks

Darren Dancey and Dave McLean and Zuhair Bandar

Intelligent Systems GroupDepartment of Computing and Mathematics,

Manchester Metropolitan University,Chester Street, Manchester, M1 5GD.

United Kingdom.d.dancey@mmu.ac.uk

Abstract

Artificial Neural Networks (ANNs) have proved both a pop-ular and powerful technique for pattern recognition tasks ina number of problem domains. However, the adoption ofANNs in many areas has been impeded, due to their inabil-ity to explain how they came to their conclusion, or show ina readily comprehendible form the knowledge they have ob-tained.This paper presents an algorithm that addresses these prob-lems. The algorithm achieves this by extracting a DecisionTree, a graphical and easily understood symbolic representa-tion of a decision process, from a trained ANN. The algorithmdoes not make assumptions about the ANN’s architecture ortraining algorithm; therefore, it can be applied to any type ofANN. The algorithm is empirically compared with Quinlan’sC4.5 (a common Decision Tree induction algorithm) usingstandard benchmark datasets. For most of the datasets usedin the evaluation, the new algorithm is shown to extract De-cision Trees that have a higher predictive accuracy than thoseinduced using C4.5 directly.

IntroductionThe two main approaches to machine learning have been Ar-tificial Neural Networks(ANNs) and symbolic learning al-gorithms. ANNs characteristically produce models that arecapable of generalizing to previously unseen data (predic-tion). However, ANN’s do not explicitly reveal the reason-ing behind their decisions. Conversely, symbolic learningmethods, do not generalize as well as ANNs, but presentthe explanation behind their reasoning explicitly. This pa-per presents a method that extracts a symbolic representa-tion from the knowledge embedded within an ANN. There-fore combining the predictive accuracy of an ANN with theadvantage of an explicit explanation provided by a symbolicmodel.

Artificial Neural NetworksThe field of Artificial Neural Networks consists of a largecollection of models and techniques originally inspired bybiological nervous systems such as the human brain. ANNsare based around a number of individual models of neurons

Copyright c© 2004, American Association for Artificial Intelli-gence (www.aaai.org). All rights reserved.

x1

x2

-1

-4 -2 2 4

-1

-0.5

0.5

1

X

X

w1

w2

w3(Bias)

Output

Figure 1: An Artificial Neuron

-1

x1

2x

-4 -2 2 4

-1

-0.5

0.5

1

-4 -2 2 4

-1

-0.5

0.5

1

-4 -2 2 4

-1

-0.5

0.5

1

-4 -2 2 4

-1

-0.5

0.5

1

Output

Figure 2: A two layer Multilayer Perceptron

(figure 1) arranged in a network. These artificial neurons ac-cept a number ofweightedinputs and process these inputsto produce an output. It is the value of these weights thatdetermine the function of the ANN. Using the backprop-agation algorithm (Rumelhart, Hinton, & Williams 1986),Multilayer Perceptrons (MLPs) are able to learn non-linearmappings. It is this type of model that will be used through-out this paper. A typical two layer MLP is shown in figure 2.

Decision TreesDecision Trees are one of the most widely used classifiermodels(Michie, Spiegelhalter, & Taylor 1994). Decision

Outlook

PlayHumidity

NotPlay

Play

Windy

NotPlay

Play

Rain

High

Normal True False

sunny

Overcast

Figure 3: Decision for Quinlan’s Play–Not Play Example

Trees are directed acyclic graphs consisting of nodes andconnections (edges) that illustrate decision rules. Each non-terminal node has a splitting test associated with it, whichsplits the data into mutually exclusive subsets. The terminalnodes called leaves represent a classification. A DecisionTree for the Quinlan’s classic ‘play/not play tennis’ exam-ple(Quinlan 1986) is shown in figure 3.

To make a decision using a Decision Tree start at the rootnode and follow the tree down the branches, according tothe tests for the instance being classified, until a leaf noderepresenting the class is reached. Although Decision Treesare very simple to understand, the method of creating a de-cision tree from examples is a nontrivial task, in fact, it hasbeen shown to be NP complete(Hyafil & Rivest 1976).

Rule Extraction From Multilayer PerceptronsMultilayer Perceptron’s (MLP’s) greatest weakness is theirlack of transparency. Unlike decision trees, which showtheir reasoning explicitly, MLPs hide their knowledge in thecomplex interrelationships of their weights. This means thatalthough MLPs often provide excellent models for predic-tion, they provide no insight into the relationships betweeninput values and output values that the model may havefound(Andrews, Diederich, & Tickle 1995). For example,Rothwell(2002) has created an ANN that can classify a per-sons responses as either deceptive or truthful, using clues intheir nonverbal behaviour (eye moments, shrugs etc) but al-though the ANN has good predictive accuracy it does notreveal the relationships it has found between nonverbal be-haviour and deception.

The aim of rule extraction is to reduce the complexityof an ANN into a more easily understood symbolic form.These rules can then be analyzed for trustworthiness forsafety critical systems or used to provide insights into therelationships found by the ANN.

There have been two main approaches to extractingrules from trained ANNs decompositional and pedagogi-cal(Craven & Shavlik 1994a). The decompositional ap-proach examines the individual weights of the underlyingANN. This approach is typified by the KT algorithm(Fu

1995). The second approach to rule extraction is the ped-agogical approach. This approach is typified by the Trepanalgorithm(Craven & Shavlik 1994b). This approach treatsthe ANN like a ’black box’, and uses a symbolic learningalgorithm to ’learn’ the rules which represent the mappingthe ANN has found.

ExTree

ExTree is an algorithm(figure 5) for extracting DecisionTrees from trained ANNs. ExTree is an example of thepedagogical approach to rule extraction. ExTree usesCraven’s querying and sampling method (Craven & Shavlik1995), but unlike Craven’s Trepan, which uses MofN basedsplits(Murthy 1995), ExTree uses standard splitting tests likeCART and C4.5.

The standard Decision Tree induction algorithms have thelimitation that the selection of the splitting test is based onfewer and fewer instances as the tree grows downwards.Therefore, the splitting tests that are near the bottom of thetree are often poorly chosen because they are based on lessdata. ExTree alleviates this problem by generating new in-stances then querying the ANN (which acts as an oracle)with the newly created instances. ExTree can then select asplitting test based on the newly created instances as well asthe original dataset.

ExTree requires a trained ANN to act as an oracle. In thenext section ExTree is applied to trained MLPs but ExTreecould be as easily applied to other ANN types such as trainedRadial Basis Function networks or even other pattern recog-nition techniques which are opaque. ExTree does not requirethe ANN to use a special training algorithm or architectureonly that it maps the input space to 1 ofK classes. Once atrained ANN is available ExTree proceeds in a similar man-ner to Decision Tree induction algorithms recursively split-ting the tree by finding the best feature to split on.

Split Types

ExTree considers two types of tests: for discrete features Ex-Tree creates a branch for each possible value of the feature,for continuous numeric features a binary split is made withtwo outcomesA ≤ Z andA > Z. The threshold valueZ isdetermined by first sorting the set of instances on the valueof featureA. For a set withm unique values for featureAthere will bem − 1 possible split points that could parti-tion the set into two. ExTree chooses a split point halfwaybetween the bounding values.

Split Selection Measure

To determine which one of the possible splits to use, Ex-Tree uses a modification of Information Gain. InformationGain has a bias towards selecting tests with many outcomes.Quinlan(1999) proposed a modification to Information Gaingiving Information Gain Ratio. Gain Ratio is determinedby dividing the Information Gain by the Information gainedsolely by splitting the data into the number of outcomes re-sulting from the test. The information gained by arbitrarily

Att1 < 50

Class

1

n o y e s

Att2 < 15

Class

2

y e s n o

Node 1

Att3 = 5

Class

1

Class

2

y e s n o

Figure 4: A Decision Tree demonstrating a constraint

splitting a setS into n subsets is given by

split info(X) =n∑

i=1

|Si||S| × log2

( |Si||S|

). (1)

The gain ratio of testX can thus be calculated as

gain ratio(X) =gain(X)

split info(X). (2)

Oracle Querying

As previously stated the advantage of pedagogical ap-proaches such as ExTree is that new instances can be createdand classified by the ANN. ExTree is able to create thesenew instances by maintaining a set of constraints whichflows down the tree with the training instances. These con-straints specify what conditions an instance must have satis-fied to have reached a node as determined by the splittingtests above. For example, new instances created at node1 in figure 4 must satisfy the constraints:Att1 < 50 andAtt2 < 15. Given these constraints, new instances can cre-ated by sampling linearly in the area of input space delim-ited by the constraints. Currently ExTree makes an extra100 extra instances at ever split point but ideally the num-ber of extra instances created would be adjusted to suite thedataset.

PruningExTree only stops growing the tree when the set of instancesreaching a node all belong to the same class or the instancescan not be split any further. For the majority of datasets

Extree( dataset S, constraints Const )BEGINNewInstances :=

Create N new instances constrainedby Const;FOR each instance in NewInstances

Label instance using ANNS := S + NewInstances;IF all S belongs to Ck THEN

label node as leaf Ck

9 RETURNELSE

Find Best Split S∗Split the S into subsets S1..Snaccording

to S*FOR each subset Si

BEGINIF the number of instances in

subset is 0THEN mark node as dominating class

of parentELSE IF node is a mixture of classes

Create new Constraint Consti from Const,ExTree(oracle, Si, Consti)

END

Figure 5: ExTree Algorithm

which contain noise this will lead to overfitting. ExTree usesa form of post-pruning to create smaller trees that shouldgeneralize better and be more comprehendible. Before train-ing, 33% of the training data is set-aside as a validationset. ExTree uses the pruning method of subtree replace-ment. Starting at the leaves and working back towards theroot, each subtree is tested using the validation set to deter-mine whether the replacement would be beneficial. If thetree with the replacement has a lower error then the subtreeis replaced.

Empirical Evaluation of ExtreeExTree was evaluated using benchmark machine learningdatasets from the well known UCI machine learning reposi-tory(Blake & Merz 1998). The predictive performance of atrained MLPs and C4.5 induced Decision Trees were com-pared on number of datasets. Nine datasets from thosewhich the MLP outperformed the C4.5 Decision Tree wererandomly chosen to be used in this evaluation. The num-ber of input features and number of classes for each datasetis given in table 1. The Balance scale dataset is an artifi-cial dataset originally generated to model psychological ex-periments, all the others are real-world datasets originallycollected in their respective fields and then donated to theUCI machine learning repository. The Hepatitis, Diabetes,Housing and Heart datasets consist of only numeric features.The Vote dataset consists of purely discrete data. The Laborand Colic datasets have a mixture of numeric and discretefeatures. The Housing dataset in its original form has a con-tinuous output value, but for these experiments it has been

Dataset num of features num of classesBalance-scale 4 3Colic 8 2Diabetes 24 2Eucalyptus 19 5Heart-statlog 13 2Hepatitis 20 2Housing 14 2Labor 16 2Vote 17 2

Table 1: Number of features and classes for datasets

transformed into a two class discrete problem of predictingwhether the output value is above $20000.

To measure the performance of the algorithm, two stan-dard statistical techniques were used: Stratified ten foldcross-validation(Stone 1974) was used to obtain a reliablemeasure of the predictive accuracy of the algorithm on thedatasets and a Wilcoxon(Wilcoxon 1945) rank sign test wasused to test whether the difference in accuracy between Ex-Tree and C4.5 was statistically significant. In all the ex-periments the same ANN topology was used: a two-layerMLP, with five hidden nodes. All training was done usinggradient descent with momentum to minimize a cross en-tropy (Van Ooyen & Nienhuis 1992) error function. Thehidden nodes used the bipolar activation function. Thenodes in the output layer used the softmax activation func-tion1. Learning rate and momentum were set at 0.01 and0.9 respectively. Performance of the ANNs could possi-bly be improved by optimizing the learning rate, momen-tum and architecture of each ANN to each of the individualdatasets but because the purpose of this paper is to illus-trate the validity of the ExTree approach to rule extractionthis has not been done. To foster generalization, 33% ofthe training set was set aside to be used as an early stop-ping validation set. The input features were normalized tohave a mean of 0, and a standard deviation of 1 for theANN as is normal for MLP training(Demuth & Beale 2002;Haykin 1999). For purposes of comparison, predictive clas-sification accuracy results were obtained for these datasetsusing an implementation of Quinlan’s C4.5 algorithm2. TheC4.5 implementation used the same validation set basedpruning technique as ExTree to ensure that any differencesin predictive accuracy were not due to the pruning techiqueused.

Table 2 shows the results obtained using 10-fold cross-validation. As expected the results confirm that ANNs dooutperform C4.5. C4.5 does not make maximum use ofthe information present in the datasets. ExTree performedslightly better on average than C4.5 did. ExTree produced

1The combination of softmax activation functions and a crossentropy error function has the advantage of allowing a probabilisticinterpretation of the ANNs output(Ripley 1996)(Bishop 1995).

2It should be noted that this was not the ’official’ C4.5 releasedby Quinlan but a C++ work-a-like implementation which sharesmuch of the codebase of the ExTree implementation to ensure afair comparison.

DataSet Neural(CE) C4.52 ExTree100Balance-scale 89.60 77.92 78.60Colic 82.61 81.52 81.79Diabetes 75.91 72.53 76.04Eucalyptus 62.09 60.60 57.20Heart-statlog 83.33 71.11 78.15Hepatitis 83.87 70.32 80.65Housing 87.55 82.41 85.18Labor 90.35 83.33 85.96Vote 96.09 93.79 95.63Mean 84.18 77.69 79.80

Table 2: Percentage of Instances Classified Correctly

more accurate models on 8 of the 9 datasets. A Wilcoxonrank sum test showed that the difference between the C4.5and Extree was significant(p < 0.01). ExTree appeared todo particularly well on numerically dominated datasets withthe largest improvement over C4.5 made on the Heart andHepatitis datasets which consist of purely numeric features.A likely explanation for this improvement is that if the re-gion ofX where the optimal splitting point lies is sparselyrepresented in the dataset then C4.5 will be unlikely to findit, whereas ExTree will have sampled extra points in the re-gion and so will be able to produce a more accurate estimateof the optimal splitting point. There are still large differ-ences between many of results obtained by the ANN andExTree which suggests that there is still much knowledge tobe extracted. The ANN outperformed both ExTree and C4.5by around 10% on the Balance scale dataset. This is almostcertainly due to Decision Trees not being able to representthe mapping required by the balance scale dataset3 This in-dicates that there will be ANNs that ExTree will be unable toextract sufficiently comprehensible rules from because De-cision Trees are simply not powerful enough to represent thefunction that the ANN has learnt.

ConclusionA method for extracting Decision Trees from trained Ar-tificial Neural Networks regardless of the ANNs architec-ture and independent of its learning algorithm has been pre-sented. It was found that the trees produced had better pre-dictive accuracy than trees produced using the C4.5 basedlearning algorithm for eight of the nine datasets. The resultsobtained using the ExTree algorithm indicate that queryingand sampling the ANN to induce a C4.5 like decision treeis a workable approach for a wide range of problem do-mains. The results showed that there were still large dif-ferences between the predictive accuracy of the underlyingANN and ExTree on some datasets. This suggests there isfurther knowledge to be extracted from the ANN. An obvi-ous next step to achieving this would be to modify the num-ber of new instances generated at the nodes (currently 100).

3Because balance scale is an artificial dataset the concept func-tion is actually known: (Feature1× Feature2) is equal, greater thanor less than (Feature2× Feature3)?

Preliminary experiments using an increased number of gen-erated instances on a subset of the datasets used in this paperhave indicated an improvement in predictive accuracy.

The results report in the last section used ANNs had notbeen optimized for the individual datasets. Optimizing theANN topology would most likely increase the accuracy ofthe ANN which would in turn increase the accuracy of theDecision Tree extracted by ExTree.

ReferencesAndrews, R.; Diederich, J.; and Tickle, A. B. 1995. Sur-vey and critique of techniques for extracting rules fromtrained artificial neural networks.Knowledge-Based Sys-tems8(6):373–389.

Bishop, C. M. 1995.Neural Networks for Pattern Recog-nition. Oxford: Oxford University Press.

Blake, C., and Merz, C. 1998. UCI repository of machinelearning databases.University of California, Irvine, Dept.of Information and Computer Sciences.

Craven, M. W., and Shavlik, J. W. 1994a. Using samplingand queries to extract rules from trained neural networks.In Eleventh International Conference of Machine Learn-ing. San Francisco: Morgan Kaufmann.

Craven, M., and Shavlik, J. W. 1994b. Using sampling andqueries to extract rules from trained neural networks. InInternational Conference on Machine Learning, 37–45.

Craven, M., and Shavlik, J. 1995. Extracting tree-structured representations of trained networks. In Touret-zky; S., D.; Mozer; C., M.; Hasselmo; and E., M., eds.,Ad-vances in Neural Information Processing Systems 8. Pro-ceedings of the 1995 Conference., 24–30.

Demuth, H., and Beale, M. 2002.Neural Network Toolbox.Mathsoft.

Fu, L. 1995. Rule learning by searching on adapted nets. InProceedings of the Ninth National Conference on ArtificialIntelligence, 373–389.

Haykin, S. 1999.Neural networks : a comprehensive foun-dation. Upper Saddle River, N.J.: Prentice Hall.

Hyafil, L., and Rivest, R. 1976. Constructing optimal bi-nary decision trees is NP-complete.Information Process-ing Letters5(1):15–17.

Michie, D.; Spiegelhalter, D.; and Taylor, C. 1994.Ma-chine learning, neural and statistical classification. EllisHorwood series in artificial intelligence. New York: EllisHorwood.

Murthy, S. K. 1995. On Growing Better Decision Treesfrom Data. Ph.D. Dissertation, Johns Hopkins University.

Quinlan, J. R. 1986. Induction of decision trees.MachineLearning1(1):81–106.

Quinlan, J. 1999. Simplifying decision trees.InternationalJournal of Human-Computer Studies51(2):497–510.

Ripley, B. D. 1996. Pattern recognition and neural net-works. Cambridge ; New York: Cambridge UniversityPress.

Rothwell, J. 2002. Artificial Neural Networks For Psy-chological Profiling Using Multichannels Of NonverbalBehaviour. Ph.D. Dissertation, Manchester MetropolitanUniversity.Rumelhart, D.; Hinton, G.; and Williams, R. 1986.Learning internal representations by error propagation. InRumelhart, D., and McCleland, J., eds.,Parallel Dis-tributed Processing, volume 1. Cambridge: MIT Press.Stone, M. 1974. Cross-validatory choice and assessmentof statistical predictions.Journal of the Royal StatisticalSociety36:111–147.Van Ooyen, A., and Nienhuis, B. 1992. Improving the con-vergence of the back-propagation algorithm.Neural Net-works5(3):465–71.Wilcoxon, F. 1945. Individual comparisons by rankingmethods.Biometrics1:80–83.