2000,Vol.15,No.2,111–131 DataMiningforFunandProﬁt

Statistical Science2000, Vol. 15, No. 2, 111–131

Data Mining for Fun and ProfitDavid J. Hand, Gordon Blunt, Mark G. Kelly and Niall M. Adams

Abstract. Data mining is defined as the process of seeking interestingor valuable information within large data sets. This presents novel chal-lenges and problems, distinct from those typically arising in the alliedareas of statistics, machine learning, pattern recognition or databasescience. A distinction is drawn between the two data mining activitiesof model building and pattern detection. Even though statisticians arefamiliar with the former, the large data sets involved in data miningmean that novel problems do arise. The second of the activities, pat-tern detection, presents entirely new classes of challenges, some arising,again, as a consequence of the large sizes of the data sets. Data qual-ity is a particularly troublesome issue in data mining applications, andthis is examined. The discussion is illustrated with a variety of realexamples.

Key words and phrases: Data mining, knowledge discovery, large datasets, computers, databases.

1. INTRODUCTION

Data mining is the process of seeking interestingor valuable information within large databases. Thenovelty, and the reason that a new term has beencoined to describe the activity, has its origin pri-marily in the large sizes of modern databases. Thesize means that standard statistical exploratorydata analysis procedures need to be extended, mod-ified, adapted and also supplemented by differentkinds of procedures. The result is that data min-ing is an interdisciplinary subject, representing theconfluence of ideas from statistics, exploratory dataanalysis, machine learning, pattern recognition,database technology, and other disciplines.The size of modern databases is illustrated by the

following examples. Barclaycard, the U.K.’s largestcredit card company, carries out 350 million trans-

David J. Hand is Professor, Department of Math-ematics, Imperial College of Science, Technologyand Medicine, Huxley Building, 180 Queen’s Gate,London SW7 2BZ, U.K. Gordon Blunt is Market-ing Analyst, Marketing Department, Barclaycard,1234 Pavilion Drive, Northampton, NN4 7SG, U.K.Mark G. Kelly is Research Fellow, Departmentof Mathematics, Imperial College, Huxley Build-ing, 180 Queen’s Gate, London SW7 2BZ, U.K.Niall M. Adams is Research Fellow, Departmentof Mathematics, Imperial College, Huxley Building,180 Queen’s Gate, London SW7 2BZ, U.K.

actions a year. However, this is nothing comparedto the American retailer Wal-Mart, which makesover 7 billion transactions a year (Babcock, 1994).More strikingly still, according to Cortes and Preg-ibon (1997), AT&T carries over 70 billion long dis-tance calls annually. Harrison (1993) remarks thatMobil Oil aims to store over 100 terabytes of dataconcerned with oil exploration. Fayyad, Piatetsky-Shapiro, and Smyth (1996) say that the NASAEarth Observing System was projected to gener-ate on the order of 50 gigabytes of data per houraround the turn of the century. The human genomeproject has already collected gigabytes of data.Numbers as large as those above are very much

a consequence of modern electronics, computerand database technology. The data may have beencollected as secondary to some other activity, orto answer a specific question but then retained.Once collected, they clearly represent some kindof resource: it is likely, arguably certain, that suchmountains of data contain valuable or interestinginformation, if only one could identify and extract it.The term “data mining” is not a new one to statis-

ticians. However, rather than the promise implicitin the above (the information is there in the data,and “all we have to do” is extract it), it often car-ries negative connotations because one can alwaysfind apparent structures in data sets. Many of thesestructures will not be real in the sense that theyrepresent aspects of an underlying distribution,but will be attributable to the random aspects of

111

112 D. J. HAND, G. BLUNT, M. G. KELLY, AND N. M. ADAMS

the data generating process. Statisticians, to whominference is a fundamental activity, are acutelyaware of this and have as a central concern thequestion of how to distinguish between the under-lying “systematic” components and the randomcomponents of data. Thus, on observing a smalllocal cluster of data points, one of the statistician’schief concerns may be whether the clustering couldreasonably be attributable to chance or not. In con-trast, data mining practitioners concern themselvesprimarily with identifying potentially interestingor valuable structures in data (i.e., with findingthe “small local cluster of data points” in the firstplace), shifting the responsibility for determining“reality” to the database owner or domain expert.This also allows the expert to characterize struc-tures which, though real, are not of interest (forexample, a data mining exercise may reveal thatalmost all sufferers from prostate cancer in thedatabase are male). We have more to say aboutsuch issues below.Chance is one source of structures in data which

have no matching underlying “reality,” or which arenot valuable or interesting. Another source is datacorruption. One should expect any large data set tobe imperfect. This poses particular challenges forthose concerned with finding interesting, valuableand meaningful structures in large data sets. Itmeans that one may discover structures which arean aspect of systematic variation between objects,but which have arisen because of missing data,distorted data or ambiguity of definition (e.g., afew sufferers from prostate cancer in the illustra-tion mentioned above who are apparently female).Such structures may be of interest, but equally maynot be.We commented above that most statisticians are

concerned with inference of one kind or another.Their aim is to take a sample of data and make astatement, with associated probabilities, about thepopulation from which it came. This may be at alow level—a simple hypothesis test—or at a higherlevel—an overall model. Some data mining tasksare of this kind: in some situations, as we describein Section 2.1, the analysis can be based on a sam-ple from a large database. Others, however, espe-cially pattern detection problems (see below), cannotbe based on a sample. Moreover, in some cases onehas the entire data set available (in the numericalexamples above, all of the Wal-Mart transactionsfor a year will be available, not merely a sample ofthem). Again this can lead to differences from stan-dard statistical approaches.Because data mining is concerned with discover-

ing structure within existing databases, it is seldom

concerned with issues of data collection. In partic-ular, sciences of efficient data collection, such asexperimental design, survey design and question-naire design, are beyond its remit. This is one wayin which data mining differs from statistics. Anotherway is that data mining places greater emphasis onalgorithms than does statistics. That data miningshould be more concerned with algorithms is hardlysurprising. Given the large sizes of the data setsthat may be examined, one must rely very heavilyon automatic data processing of some kind. Thereis no way that one can individually examine a bil-lion data points. Moreover, the algorithms have tobe fast. This has led to an interest in adaptive andsequential estimation methods, in which estimatesare updated data point by data point, so minimiz-ing the number of disc accesses. Machine learn-ing, as the very word “learning” suggests, has his-torically placed emphasis on such approaches. Per-haps because of this, in contrast to most modernstatistics, where the model is central and deriv-ing a model is the goal of statistical analysis (andfrom which other questions can be answered), tomany data mining practitioners the algorithm iscentral. Often they may not even think in terms of amodel-building process at all, instead viewing it as adata driven descriptive exercise, with the algorithmdetermining what sort of description emerges. Hand(1999) has pointed out that the Gifi school of multi-variate statistics also adopts this perspective, so it isnot without precedent, even amongst statisticians.(Gifi 1990, page 34) says: “In this book we adopt thepoint of view that, given some of the most commonMVA [multivariate analysis] questions, it is possibleto start either from the model or from the technique.As we have seen in Section 1.1 classical multivari-ate statistical analysis starts from the model, � � � .In many cases, however, the choice of the model isnot at all obvious, choice of a conventional modelis impossible, and computing optimum proceduresis not feasible. In order to do something reasonablein these cases we start from the other end, witha class of techniques designed to answer the MVAquestions, and postpone the choice of model and ofoptimality criterion.”In the above, we have described the objectives of

data mining as being to find structure in data. Itis useful to distinguish between two types of struc-ture (Hand, 1998a): models and patterns. Both arewidely sought in data mining exercises. The dis-tinction, though sometimes blurred, is neverthelessa useful one. A model is an overall summary of aset of data, or a subset of the data. This is thusthe standard statistical usage. We can speak of aregression model, a Box-Jenkins time series model,

DATA MINING FOR FUN AND PROFIT 113

a dynamic linear model, a conditional independencegraph model, a cluster model and so on. Such struc-tures represent a large-scale summary of a mass ofdata.In contrast, a pattern is a local structure, possi-

bly (though not necessarily) referring to only a rel-atively small number of objects. (“Small” here couldmean 10 or 100,000. It depends on the context and,of course, the size of the overall data set.) Thus apattern might be an anomalously high log-odds ratiofor a small group of objects or a small sequence ofvalues which occurs several times in a time seriestrace (this is precisely the sort of thing that chartistsseek in stock prices). A multiway table of counts canbe analysed via a log-linear model, which will sum-marize its broad features, or one can try to identifyunexpectedly high or low cell counts. This suggeststhat patterns are defined relative to a model. Thisis often the case—outliers are another example—but it is not always so; as we illustrate below, thepattern may simply be defined relative to a broadnotion of continuity, something which is really toogeneral to call a “model.”As with data analysis in general, data mining

is not a once-off activity. That is, presented witha billion point data set, one does not simply mineit and be done with it. Rather, the exercise shouldbe seen as an interactive process involving boththe data miner and the domain expert, as well asthe data.In the next section we examine the sorts of tools

being used and developed for data mining applica-tions. We expand on the distinction between mod-els and patterns, and illustrate with some real datasets. Section 3 discusses some of the challengingproblems which arise as a consequence of the sheersize of the data sets which are becoming available,and Section 4 looks at the vital issue of data quality.Section 5 draws some overall conclusions.

2. DATA MINING TOOLS

In Section 1 we distinguished between models andpatterns. Any kind of statistical model may appearin a data mining application, and we need not dwellon tools for these here, since they will be familiarto statisticians. Examination of recent data miningconference proceedings shows that certain classes oftools are particularly important (or, at least, attractconsiderable attention from those concerned withdeveloping data mining tools). They include toolsfor unsupervised classification (clustering), super-vised classification, more general predictive models(regression), modelling time series to detect trendand other structures, and graphical models.

Even though the objective and basic nature oftools for modelling will be familiar, the algorithmsmay differ from those in standard use in statisticalapplications (recall the comment about sequen-tial estimation methods in Section 1), and oftenthe emphasis is different. For example, recursivepartitioning methods are widely used in data min-ing applications with emphasis placed on theirinterpretability. This perhaps has more in com-mon with machine learning work than statistics.Often, in fact, the data mining work goes furtherin this direction and the aim is not so much toextract entire tree structures as to characterize“local rules” relating variables (e.g., “If A is presentand B is absent, then C has a high probability ofbeing present”).In some contexts, rule extraction, (or association

analysis, as it is sometimes rather unfortunatelycalled) is a key aim. A classic example is marketbasket analysis, so called because of its origins inmining supermarket purchasing data: interest liesin the percent of customers who purchase certaingoods, given that they purchase others. Some sub-tleties were not always recognised by early workersin the field (or, dare we say it, by some more recentworkers), often working in ignorance of the statis-tical content of the analysis. In particular, it willbe obvious to statisticians that a high conditionalprobability P�X�Y = 0� may not be very interest-ing in itself. It may only become interesting if, also,P�X�Y = 1� is low, so that a contrasting behaviorbetween groups is evident. Likewise, the familiarcaution that correlation does not imply causationhas not always been remembered in the enthusiasminspired by the discovery of apparent relationshipsbetween purchases of different goods. The fact thatmost people who buy A also buy B does not meanthat inducing other people to buy A will lead to themalso buying B.The term data visualization is often used to

describe graphical methods in data mining con-texts, where the potentially huge numbers of datapoints can present problems for standard statisticalmethods. As with statistics, dynamic and inter-active methods obviously have considerable novelpotential. This is an active area of research, thoughthus far highly innovatory tools with wide impactseem few and far between. Virtual reality meth-ods for exploring large databases represent oneexciting area of future work. Given the difficulty ofadequately demonstrating dynamic, interactive andvirtual reality methods in the medium of a journalpaper, we will not dwell on them here. Exam-ples of data visualization methods are described in


Cox, Eick, Wills and Brachman (1997), Derthick,Kolojejchick and Roth (1997) and Mihalisin andTimlin (1997).Pattern detection methods will be less familiar

to statisticians than will model building exercises.They represent a relatively new area of work. Muchof it is rather ad hoc, and deep ideas about the beststrategies to pursue in pattern detection appear notto have emerged yet. (Of course, they may neveremerge.) Sections 2.1 and 2.2 give some relativelystraightforward, but we hope interesting, examplesof model building and pattern detection in data min-ing from our own work.

2.1 Models

Figure 1, from Hand (1998b), shows a histogramof the number of weeks of a particular year thatcredit card holders used their cards in supermar-kets. This distribution has various features, whichare evident from the histogram and which we mighttry to model. Most strikingly, there is the rapidlydecaying left-hand mode. This mode is probablyto be expected: it means that many people neveruse their cards in a supermarket; a smaller num-ber, though still relatively large, use them in justone week and so on. However, closer examina-tion shows that the distribution does not decay tozero, but perhaps to some constant level: a similarnumber of customers use their cards 20 weeks asdo 30 or 40 weeks of the year. Furthermore, at theright-hand end of the distribution, there appearsto be a smaller, flatter mode. Having detected thissecond mode, it is not difficult to concoct a ret-rospective explanation. It is likely that there are

Fig. 1. Histogram of number of times credit card owners used their cards in a supermarket in one year.

people who use their cards in a supermarket everyweek, except, perhaps, when they are on holiday orunable to for some other reason. One could build amodel (perhaps a mixture model involving a Pois-son left-hand mode, a reversed Poisson right-handmode, and maybe other components) to summarizethe shape of the distribution in terms of a few con-venient parameters. Whether the features of thismodel (the decaying left-hand mode, the constantmiddle part, the small right-hand mode) are inter-esting or valuable is a matter for the supermarketoperators to determine. We could conduct statis-tical significance tests on aspects of the model,though the numbers involved in this example aresuch that all the aspects above will be highly sig-nificant. Moreover, as noted above, one has to becareful in interpreting such tests; the data maybe all credit card supermarket transactions in theyear in question, not a sample from them. Statisti-cal inferences will then presumably refer to otherdata sets which could have been drawn, with theimplication that one is really concerned with futurepossible transactions.This example illustrates the basics of the mod-

elling approach to data mining. It is very similarto standard statistical modelling. Perhaps onedifference is that one might be more sensitive tothe question of how large (“substantively signifi-cant”) are the features one wants. If the data setinvolved 100 million points, then minute featurescould be detected as highly statistically significant.It is likely, however, that many of these would beso small as to be of no conceivable value. Of course,such issues depend critically on the context. One


Fig. 2. Histogram of sizes of petrol station transactions.

can avoid detecting small features as statisticallysignificant by reducing the size of the data setby sampling. Then standard statistical inferentialissues come into play.For a second modelling example, consider the dis-

tribution shown in Figure 2. This again involvescredit card data, but now is a histogram of theamount spent in each of 40,068 credit card transac-tions in petrol stations over a given period. The cellsin this histogram range from £x�51 to £�x + 1��50,and 196 transactions of value over £100 have beenexcluded so that the distribution of the remain-der is clearer. It is apparent that the distributionhas some striking and, at least to the authors,unexpected features. There are marked peaks atmultiples of £5 and £10, and also at the values £12and £18. Again the numbers of transactions here, aswell as the regularity of the patterns, make it clearthat these structures are not simply chance events:something real is going on. (We note here that thepeaks could legitimately be regarded as patterns.In this example, however, we are concerned withmodelling them, rather than detecting them. In thenext section we give an example where the empha-sis is on detecting and explaining patterns formingsimilar peaks.)Blunt and Hand (1999) have examined this data

set in detail. Closer examination reveals roundingto all integral numbers of pounds, though much lessmarked than is evident in Figure 2. Blunt and Handbuild a mixture model for these data, in which petrolpurchasers are characterized as being one of two

types: those who seek to make purchases of roundedvalues and those who do not. Based on this, theyidentify features which distinguish between the twotypes. For commercial reasons, they stop short ofshowing how this information may be made use ofby the credit card company.Predictive models are important in data min-

ing, perhaps especially in commercial applications,where one is often concerned with the possibilityof manipulating practices so as to increase sales.In one of our studies we knew that customers whoexhibited a certain type of transaction pattern werelikely to respond positively to a marketing initia-tive, while those who did not exhibit the patternwere unlikely to respond positively. We knew that,in the customer base in general, 12.5% of cus-tomers followed the transaction pattern. However,we were able to identify a subgroup of 10% of thecustomer base that had a 43% chance of exhibitingthe transaction pattern. It was then easy to set upan equation including per capita return on the mar-keting initiative and per capita marketing cost, todemonstrate that greater profit would be obtainedby restricting the marketing initiative to the iden-tified subgroup. Here we knew what transactionpattern was to be used as the predictor variable,so we only had to concern ourselves with the mod-elling aspects. In general, we will also be searchingfor patterns that give us this sort of opportunity.This is described in the next section.These examples show that modelling in data min-

ing is closely related to modelling in statistics. There


are differences, however, primarily arising from thesizes of the data sets frequently encountered in datamining. These issues are discussed in Section 3.Perhaps it is not necessary to say that the bor-der between the two disciplines, as far as modellingwork goes, is a fuzzy and shifting boundary, andone that probably depends as much on the individ-ual investigator as the subject matter and objective.

2.2 Patterns

A “pattern” is an unusual structure or relation-ship in the data set. The structure may be sharedby relatively few cases, but nevertheless enough tobe worthy of attention. We should note that the term“pattern” is used in a different way from its use inthe phrase “pattern recognition.” There it refers tothe identification of a particular shape (in an image)or classification of a vector of observations (in sta-tistical pattern recognition).There are different kinds of patterns. They may

be cases that demonstrate apparent departure frommodels, such as outliers. They may be shapes intime series or patterns in event sequences whichoccasionally recur (these are often called episodes:see Mannila, Toivonen and Verkamo, 1997). Ourfirst example illustrates a case in which they aredefined relative to a notion of uniformity of the back-ground data.Adams and Hand (1999) describe a tool for

detecting local groups of points which might beregarded as anomalously similar. If they are notmerely chance aggregations of points, such struc-tures would reflect local peaks of the underlyingdistribution. Multivariate nonparametric densityestimation techniques (e.g., Scott, 1992) can be usedto smooth a data set to detect such peaks, but find-ing their positions after the smooth is difficult inmore than two or three dimensions. To overcome

Fig. 3. Scatterplot (left-hand panel) of chromosome data showing locations (right-hand panel) of locally dense regions.

this difficulty, Adams and Hand (1999) evaluate thesmooth at each of the data points themselves. Thevalue of the estimated density at each data pointis then compared with the estimated value for thepoint’s M neighbors. If it has a larger estimatedvalue than any of its M neighbors, then it is takenas a local peak, and the position flagged as a pat-tern of potential interest. By varying M we varythe meaning of “local.”To illustrate, the left-hand panel of Figure 3

shows a scatterplot of two standardized variablesmeasured on 5520 human chromosomes. The right-hand panel shows the position of the local maximaof estimated probability density, using M = 10.There are clear local maxima around the “center” ofthe plot, which are to be expected. However, thereare other, less predictable and far from obviouspeaks in the tails of the distributions. The local rel-ative overabundance of points at these coordinatesmay merely be chance fluctuations, but we are flag-ging them as worth checking. Of course, the valueof such an exercise is more apparent when morethan two variables are involved.Pattern detection data mining tools are used for

identifying fraud, fault detection, instrument break-down, distinguishing between genuine and spuri-ous alarms, and, of course, also for finding errors indata. Hand (1998b) notes that credit card compa-nies use such methods to trigger an alert when thepattern of card usage deviates from the customer’snormal pattern.We pointed out in Section 2.1 that sampling can

be used in modelling, where the aim is generally tocharacterize the most important (and hence, largescale) features of a data set. However, sampling can-not be used in data mining for patterns. By def-inition, sampling thins out the available data fordetecting the structure, and, since a pattern may be


based on only a small number of cases in the firstplace, reducing this number may remove the patternaltogether. An increasing trend in a customer’s num-ber of credit card transactions may be adequatelymodelled by taking only every tenth transaction,but the fraud patterns mentioned above could wellbe missed by thinning out the data in this way.There are two major problems with pattern detec-

tion methods. One arises from data quality and isdiscussed in Section 4. The other, familiar to statis-ticians, is the fact that many spurious patterns,generated merely by chance fluctuation, will be“discovered.” This is particularly the case if oneis examining many candidates, seeking patternsdefined by small numbers of cases. We discussthis further in Section 3. Whereas much statisticalwork is concerned with characterizing how likelyit is that apparent structure will arise in a dataset given that there is no such structure in theunderlying process, in data mining the emphasis issimply on locating the structure. The responsibilityfor deciding whether the structure has meaning interms of the underlying process (“is real”) is shiftedto a domain expert. For example, perhaps the 100“largest” structures in the database will be referredto the expert, who can decide whether they areinteresting, valuable, or likely merely to be chancefeatures of the random aspect of the data. This isone reason why “data mining” should not be seenas a “once off” activity, but rather as a process, in

Fig. 4. Values of credit card purchases in department stores: £1 cell widths.

which the discovery of apparent structure and theinterpretation of that structure bounce back andforth.Although the aim of the exercise is to find pre-

viously unsuspected patterns, we believe that pat-terns which can be explained (that is, for which aconvincing post hoc rational explanation can be cre-ated) are much more likely to be “real” than thosefor which such an explanation cannot be created.Thus, we suggested an explanation for the right-hand mode in Figure 1 that, at least to us, soundsreasonable. It would be more difficult to find anequally convincing explanation for the mode at 11weeks in that figure. Although a pattern detectiondata mining tool [such as that described by Adamsand Hand (1999), outlined above] could well pick upthis mode also, we would be suspicious of it on thegrounds that we could not explain it.Figure 4 shows a histogram of the values of

credit card purchases in department stores. Thereare clear peaks at some values; notably at valuesending in 0 and, to a rather lesser extent, at valuesending in 5. A pattern detection algorithm, such asthat described above, might pick these up (though,in fact, since this example is one-dimensional forillustrative purposes, this is not really necessary).Since department store purchases seldom providethe opportunity for choosing rounded values, theexplanation provided for the petrol purchases inSection 2.1 will not suffice. Neither will digit pref-erence (see Section 4) since the values are objective


and recorded electronically. Some other explanationis required and a little thought readily provides one.Items in department stores are often priced to 1p

less than a multiple of 5 or 10 pounds. Thus wehave prices of £4�99 and £19�99. If plotted on a his-togram with cell widths of £1, these will be roundedto £5 and £20. Further support for this explana-tion comes from Figure 5, which shows histograms,with cell widths 1p, around some of the peaks fromFigure 4. If one, or sometimes more, purchases withsuch prices were made, one would expect to find pre-cisely these local distributions, tailing off to the left.The structure in Figure 4 can be explained, and

we are confident that (a) the structure is real and(b) our explanation is correct. Whether the infor-mation is valuable or not is a different question.Whether the structure is obvious or not is also a dif-ferent question; perhaps most of the real structuresdiscovered in data mining exercises are obvious inretrospect.Pattern detection is often used in an interactive

way. Thus collaborative filtering describes the pro-cess of extracting conditional probabilities from pur-chasing data used by some retail organizations. (Forexample, the Amazon internet bookstore uses thisstrategy.) The purchases of customers who boughtitem X are examined to see what other items, Y,they tended to buy. Then new customers who buyX are alerted to the fact that they may also find Y

Fig. 5. Histograms of credit card purchases in department stores: 1p cell widths.

interesting. Clearly, rather more than a high con-ditional probability P�Y�X� is needed—some itemsare bought by everyone.The key feature is a high P�Y�X� and a low

P�Y�∼X�, along with, of course, a value of P�X�which is not too small. The exercise here is a fairlyelementary statistical one, although the practicedoes have some interesting features. The data setsare large, of course, and the application (detect apurchase of item X and send a message about Y)occurs in real time, although the processing to findthe conditional and marginal probabilities need not.On the other hand, these probabilities need regularupdating, and this updating must also adapt to thechanging inventory of the supplier.

3. PROBLEMS OF SIZE

It is the size of the data sets encountered in manydata mining applications which provides the poten-tial for discovering and taking advantage of novelstructures. The sheer number of records may serveto conceal features of interest, but these numbersmean that relatively small effects, not easily iden-tifiable or observable with smaller numbers, can beproductively sought. But size is a two-edged sword.As well as providing the opportunity, it also bringswith it problems.One obvious one is that large samples mean that

things which are substantively insignificant may


be highly statistically significant. This is an issuewhich is very much context dependent. For exam-ple, 100,000 records with a common property maymean very high statistical significance. And 100,000records may provide the potential for large extraprofit, if advantage can be taken of whatever com-mon property they have. (In one project with whichwe have been involved, reducing the bad rate on abank loan portfolio of 8 million customers by just0.25%, in which each “bad” means a loss of £1000,would lead to an overall saving of £20 million.) Onthe other hand, if the 100,000 records are from a1 billion item database, then the relative size of theextra profit, may not be worthwhile. It depends onthe context.In searching for interesting patterns, one could

test each candidate pattern for statistical signifi-cance. However, this inevitably means a large num-ber of tests. Carrying out large numbers of testsdoes not really protect us against detection of spu-rious relationships, either because they will alsofalsely reject a large number of null hypotheses (5%of a large number is still a large number, for exam-ple) or because the overall (“experimentwise”) con-trol adopted requires the underlying effect to bevery large to have a nonnegligible chance of beingdetected.Given that formal probability models are of lim-

ited value, a scoring strategy is often adopted. Thisabandons probabilistic interpretations and sim-ply scores each model or pattern according to its“unusualness,” “unexpectedness,” or “interesting-ness.” The quotation marks are intended to signifythat there is some considerable flexibility in defin-ing the measures used for these concepts. (Notethe similarity to projection pursuit, though therethe aim is really modelling rather than patterndetection.)Other issues arising from the size of modern data

sets relate to how to analyse them. For example,the scatterplot is a basic statistical tool that is veryuseful both for probing data and for communicat-ing findings. However, to illustrate what can happenwhen very large data sets are involved, Figure 6(a)shows a scatterplot of a data set of time in employ-ment against the day of application (called “index”in the plots) for around 96,000 bank customers. Thescatterplot does show some structure. For example,there are clear dense horizontal bands. However,it is likely that these bands are an artifact of thecoarseness with which the data are recorded, and donot reflect any underlying reality of interest. In fact,this scatterplot is concealing, rather than revealinginformation, as the contour plot of the same datagiven in Figure 6(b) shows. There are clear modes

for both time in employment and index number,though there appears to be no dependence betweenthe values of these two variables.This example illustrates a key issue for data

mining algorithms: they must be scalable. That is,they must increase in a reasonable way as the sam-ple size (or perhaps as the number of variables)increases. This scalability must relate to the easewith which the methods reveal aspects of structure,and also to computational resources. Algorithms inwhich processing time or required memory increaseexponentially or quadratically with sample sizerapidly become impracticable. This necessary fea-ture of data mining algorithms may mean thatmethods that are optimal from a formal statisticalperspective cannot be used.

4. DATA QUALITY

We believe that data quality is one of the keyissues in data mining. The simple reason is that,as any experienced data analyst will know, it isextremely unusual to find data sets that have noerrors or distortions. Presented with a data set thatappears to be free of errors, a statistician’s suspi-cions may be immediately aroused. One might askwhat happened to the incomplete records, whetherthe recording instruments really never failed ordrifted over time, if all the cases were followed tothe end of the study period, and so on.This being the case, and given that data mining is

concerned with finding structure in data, we mightreasonably expect such exercises often to identifystructures arising solely because of inadequaciesor peculiarities in the data or the data collectionprocess. Our experience shows this expectation tobe entirely justified. Figure 7 shows a histogram ofdiastolic blood pressure for a sample of 10,772 men.A crude analysis would reveal that there is a strongcorrelation between the value of the blood pressureand whether or not it takes odd-numbered values.Indeed, the histogram shows that there are severalother peculiarities with this data set. For exam-ple, there are marked peaks at values ending in 0,and the only odd values which occur in the lowerpart of the distribution are those ending in 5. Weattribute the peaks to digit preference (contrastthis with the explanations for the peaks in thepetrol purchase data and the department store dataabove). Moreover, deeper investigation revealedthat the instrument only recorded even valuesand, when measurement yielded an exceptionallyhigh value of blood pressure, the measurement wasrepeated and the average of the two (even) mea-surements recorded. This can yield odd numbers.


Fig. 6. (a) Scatterplot of 100,000 bank customers, showing their time in employment plotted against their index number. (b) A contourplot corresponding to Figure 6(a).

We believe the values ending in 5 in the lowerpart of the distribution are again due to digit pref-erence. Of course, the necessity for seeking suchexplanations only arises because one has identi-fied unexpected features about the data in the firstplace.We believe it is not farfetched to suggest that most

of the “interesting and unexpected” patterns discov-ered in a data set during the course of a data miningexercise will be attributable to “inadequacies” of thedata.Data distortion and missing values can, of course,

be a serious problem even in relatively small datasets. Figure 8 is produced from a data set of 3884

applicants for loans, with the application form yield-ing scores for 25 variables. The histogram showshow many applicants had no missing values, onemissing value and so on. Only 66 applicants pro-vided complete information, and one applicant had16 of the 25 values missing (history does not recordwhether this applicant was granted a loan). Onlyfive of the variables had no missing values, and twohad over 2000 missing values.Of course, things are complicated by the fact that,

often, whether a value will be recorded for a variableis contingent on the values taken by other variables.If half the subjects are female, questions about tes-ticular cancer remain unanswered. This sort of sit-


Fig. 7. Diastolic blood pressure for 10,772 men.

Fig. 8. Histogram of number of missing values for each of 3884 applicants for personal loans.


uation is illustrated in Figure 9. The data are froma study to develop a screening instrument for osteo-porosis and relate to 1012 elderly women (Cooper, etal., 1991). The vertical axis, V1, shows the patientnumber and the horizontal axis shows the variablenumber (from 1 to 45). A point is printed when-ever there is a missing value. The resolution of thefigure is such that one cannot see individual cases,but it is apparent from the figure that there is struc-ture to the missing values. Variables 4 and 5 areusually either both missing or both not missing. Afew cases have most variables missing. Some vari-ables have many missing values. And so on. Manysuch features are precisely the sorts of things onewould hope a data mining exercise would detect,but they will often be of little interest; it is typicallystatements about the values that are recorded thatone is seeking to make. (On the other hand, miss-ing values can sometimes be useful; for example,in supervised classification problems, sometimes thefact that a value is missing can be predictive.)Missing values are perhaps the most common

kind of data distortion, but there is an infinite num-ber of ways that things can go wrong. Figure 10shows mean weight changes over a four-monthperiod for the control group from a classic study of10,000 children carried out in the 1930s to explorethe effect of free school milk (Leighton and McKin-lay, 1930). The consecutive pairs of points showthe mean weight before the trial and the mean

Fig. 9. A missing value plot of 1012 elderly women.

weight at the end of the four-month trial. The dif-ferent pairs of points relate to different age groups.A priori, one might expect that the curve wouldbe smooth. The manifest irregularities are pre-cisely the sort of thing one would hope a datamining exercise would detect, such structure isunexpected, and certainly interesting. But in thiscase it is of little value. It seems likely that it canbe attributed to the fact that the first measurementof each pair took place in February and the sec-ond in June when the weather was warmer and thechildren would be wearing lighter clothing. (Thisdespite the precautions the experimenters took: “Allof the children were weighed without their bootsor shoes and wearing only their ordinary indoorclothing. The boys were made to turn out the mis-cellaneous collection of articles which is normallyfound in their pockets, and overcoats, mufflers, etc.,were also discarded. Where a child was found tobe wearing three or four jerseys—a not uncom-mon experience—all in excess of one were removed”(Leighton and McKinlay, 1930, page 8).Hand (1998b) distinguishes between two kinds of

data inadequacy. In one kind, individual records aredistorted, while, in the other kind, the overall sam-ple is distorted. If individual records are distorted,one might try to develop methods that allow for thisor correct the distortion. Thus, for example, the EMalgorithm will permit one to find maximum likeli-hood solutions with incomplete records and impu-


Fig. 10. Weight changes for 10,000 children in a trial to study the effect of milk on growth carried out in the 1930s.

tation methods seek to fill in the missing values.Of course, such methods are based on assumptionsand models about why the data are missing. Thelack of transparency which is an inevitable con-comitant of very large data sets means that onemay be tempted to adopt automatic “data correc-tion” procedures. While these may be useful, thereis also the very real risk that they will smooth out orremove the very structures one is hoping to detect.An anomalous tight grouping of a handful of cus-tomers could be regarded as due to data record-ing errors. Clearly the solutions are not straight-forward.The second kind of data inadequacy occurs at

a higher level and is due to distortion in the setof records which is selected for inclusion in thedatabase. Selection bias is an example of this sortof phenomenon, in which whether or not a recordis included in the database depends on the valuesthe variables take. This can cause major difficul-ties. Patients for whom a treatment fails to workmay be more likely to drop out of a study, thus leav-ing those for whom it has worked and hence givinga distorted impression of the treatment’s efficacy.Copas and Li (1997) give an example involving kid-ney dialysis, in which log(hospitalization rate) for anew method of treatment appears to decline withtime, but for which it is possible that the effect isdue to the nonrandom allocation to the new andstandard treatment.

In many data mining problems, in which data col-lection may be a far from straightforward process,it is likely that the samples in the database will beconvenience or opportunity samples: those recordswhich could be easily collected. The implications forinference to the overall population may be unpre-dictable. Furthermore, populations can change overtime. Figure 11 shows weekly averages of fourvariables describing applicants for unsecured per-sonal loans over a four-year period (time-scaledisguised for commercial reasons). Figure 11(a)(a binary indicator of whether or not customerswere aged between 30 and 35) shows very littlechange over time. Figure 11(b) (a binary indica-tor of whether or not the customer had a checkguarantee card) shows a marked downward trend.Figure 11(c) (a binary indicator of whether or notthe loan was for debt consolidation) shows a grad-ual upward trend, with a superimposed seasonalcomponent. Finally, Figure 11(d) (a binary indica-tor of repayment method) shows a clear changehalfway through the observation period. This isthought to be due to a policy change by the bank,requiring customers to pay by a particular method,and is perhaps therefore an example of an unin-teresting structure. (On the other hand, one mightenquire why, if customers are required to pay by aparticular method, there is any irregularity at allin the right-hand half of the figure. Perhaps this isanother issue of data quality.) In such cases, infer-ences made on data collected at one time may have


Fig. 11. Weekly averages of four binary variables. (a) Customers aged between 30 and 35 years, (b) Check guarantee card, (c) Purposeof loan—debt consolidation, (d) Repayment method.

limited applicability later on. Of course, in suchcircumstances one might contemplate building adynamic model.As a final example, consider Figure 12. This arose

during a study of how well different definitions of a“bad” risk could be predicted in a consumer bank-ing operation. The vertical axis shows the Gini coef-ficient, a measure of how well a prediction rule per-forms. The horizontal axis shows different values ofa threshold, varying between 1 and 27, so that dif-ferent definitions of bad risk result. There is a strik-ing pattern in the plot, as in several of the exam-ples above, precisely the sort of pattern one mighthope a data mining exercise would identify. Unfor-tunately, once again, this pattern is an artifact ofproblems with the data. In this case, the bank inquestion had previously worked with a definition inwhich values of the horizontal variable above 5 weretaken to define a bad risk. This variable had beeninadvertently included as a potential predictor inthe analysis, and the looping pattern in the plot isa direct consequence of this.For explanatory purposes, each of the examples

above has focused on single problems. But life is sel-dom so simple. It is far more likely that the sample

will be distorted, and there will be missing values,and those values which are recorded will sometimesbe misrecorded and so on. In one data set on unse-cured personal loans, we found tiny amounts unpaid(e.g., 1p or 2p) leading to a customer being classifiedas a “bad debt,” negative values in the amount owed,12 month loans still active after 24 months (techni-cally not possible under the bank’s rules), outstand-ing balances dropping to zero and then becomingpositive again, balances which were always zero andnumber of months in arrears increasing by morethan a single integer in one month. Once identified,some of these problems can be explained (and, per-haps, corrected). Some, however, defied ready expla-nation.

5. CONCLUSION

The large size of modern data sets means that itwas legitimate to coin a new phrase to denote theactivity of finding structure and pattern in data.Modern statistics has grown from a base of analy-sis of relatively small data sets. Moreover, modernstatistics has placed emphasis on only parts of theentire range of data analytic problems (see, for


Fig. 12. Structure induced in classification rule performance measure due to inclusion of an inappropriate variable.

example, Chambers, 1993 and Bartholomew, 1995)and other disciplines, especially machine learn-ing and database technology, have begun to makeimportant contributions.Because data mining is such a young field, there

remain fundamental open questions, requiringnovel ideas for their solution. Among the mostimportant of these questions are those relating toissues of chance in the context of large numbers ofevents and to issues of data quality.Also because it is a new field, data mining has

developed its own terminology (for example, dic-ing, drilling down, rolling up, and support andconfidence). In many cases, there already existedperfectly good terms for the same exercises indisciplines such as database theory or statistics(conditioning, marginalizing, joint probability, con-ditional probability). In other cases, terms havebeen invented according to the domain in whichthey were developed. “Market basket analysis,”mentioned above, is one such, as is bread dominoes,describing how records can chain together (derivedfrom the fact that shoppers unable to buy the kindof bread they want tend to purchase some similaralternative).One important difference between data mining

and statistics is the emphasis of the former on algo-rithms. In this regard, data mining has more incommon with machine learning than statistics. Wewould go so far as to suggest that the data miningliterature is full of descriptions of algorithms withlittle underlying theory relating them. It is easyto develop new algorithms, but without underlyingtheory it is difficult to see what sort of progress is

being made. It is possible that, as the field matures,so deeper theoretical understanding will lead toproper critical assessment of the algorithms andtheir (comparative) properties. However, it is alsopossible that the ease with which algorithms canbe developed using powerful computers will workagainst that.Data mining has promise, but there are many

difficulties associated with it. It is not to be enteredinto lightly or in ignorance of the obstacles. Per-haps there are similarities to meta-analysis, in thatit is easy to carry out a poor analysis, and veryhard to carry out a good one. One difference isthat the quality of the data mining exercise willsoon be revealed, in terms of whether the struc-tures which have been unearthed are interesting,valuable, surprising or previously unknown. Alsolike meta-analysis, the phrase “data mining” isrich in implication. It almost spells excitement andopportunity. However, given the difficulties we haveoutlined above, one should be wary of getting car-ried away by this. It remains to see precisely whowill benefit from data mining activities, beyondcompanies marketing data mining tools.General overviews of data mining, including dis-

cussions of the relationship between datamining andstatistics, are given in Elder and Pregibon (1996),Fayyad, Piatetsky-Shapiro and Smyth (1996), Gly-mour,Madigan, Pregibon and Smyth (1997), Klosgen(1998), Hand (1998a, b), Hand (1999) and Hand,Mannila and Smyth (2000). The range of currentresearch is showcased in the journal Data Min-ing and Knowledge Discovery and the proceed-ings of the International Conference on Knowledge


Discovery and Data Mining series (e.g., Hecker-man, Mannila, Pregibon and Uthurusamy 1997 andAgrawal, Stolorz and Piatetsky-Shapiro, 1998).

ACKNOWLEDGMENTSWe are grateful to Littlewoods Plc, Abbey

National Plc, Barclaycard and Fergus Daly forpermission to use their data to illustrate some ofthe points made in this paper.

REFERENCES

Adams, N. M. and Hand, D. J. (1999). Mining for unusual pat-terns in data. Working paper, Dept. Mathematics, ImperialCollege, London.

Agrawal, R., Stolorz, P. and Piatetsky-Shapiro, G. (eds.)(1998). Proceedings of the Fourth International Conference onKnowledge Discovery and Data Mining. AAAI Press, MenloPark, CA.

Babcock, C. (1994). Parallel processing mines retail data. Com-puter World. Sept. 6.

Bartholomew, D. J. (1995). What is statistics? J. Roy. Statist.Soc. Ser. A 158 1–20.

Blunt, G. and Hand, D. J. (1999). Credit card petrol purchases:an example of data mining in practice. Working paper, Dept.Mathematics, Imperial College, London.

Chambers, J. M. (1993). Greater or lesser statistics: a choice forfuture research. Statist. Comput. 3 182–184.

Cooper, C., Shah, S., Hand, D. J., Compston, J., Davie, M. andWoolf, A. (1991). Screening for vertebral osteoporosis usingindividual risk factors. Osteoporosis International 2 48–53.

Copas, J. B. and Li, H. G. (1997). Inference for non-random sam-ples. J. Roy. Statist. Soc. Ser. B 59 55–95.

Cortes, C. and Pregibon, D. (1997). Mega-monitoring. Paperpresented at the Univ. Washington/Microsoft SummerResearch Institute on Data Mining, July 6–11.

Cox, K. C., Eick, S. G., Wills, G. J. and Brachman R. J.(1997). Visual data mining: recognizing telephone callingfraud. Data Mining and Knowledge Discovery 1 225–231.

Derthick, M., Kolojejchick, J. and Roth, S. F. (1997). An inter-active visualisation environment for data exploration. InProceeding of the Third International Conference on Knowl-edge Discovery and Data Mining (D. Heckerman, H. Man-nila, D. Pregibon and R. Uthurusamy, eds.) 2–9. AAAI Press,Menlo Park.

Elder, J. IV and Pregibon, D. (1996). A statistical perspectiveon knowledge discovery in databases. In Advances in Knowl-edge Discovery and Data Mining (U. M. Fayyad, G. Piatetsky-Shapiro, P. Smyth and R. Uthurusamy, eds.) 83–113. AAAIPress, Menlo Park, CA.

Fayyad, U. M., Piatetsky-Shapiro, G. and Smyth, P. (1996).From data mining to knowledge discovery: an overview.In Advances in Knowledge Discovery and Data Mining(U. M. Fayyad, G. Piatetsky-Shapiro, P. Smyth and R. Uthu-rusamy, eds.) 1–34. AAAI Press, Menlo Park, CA.

Fayyad, U. M., Piatetsky-Shapiro, G., Smyth, P. and Uthu-rusamy, R. (eds.) (1996). Advances in Knowledge Discoveryand Data Mining. AAAI Press, Menlo Park, CA.

Gifi, A. (1990). Nonlinear Multivariate Analysis. Wiley, Chi-chester.

Glymour, C., Madigan, D., Pregibon, D. and Smyth, P. (1997).Statistical themes and lessons for data mining. Data Miningand Knowledge Discovery 1 11–28.

Hand, D. J. (1998a). Data mining: statistics and more? Amer.Statist. 52 112–118.

Hand, D. J. (1998b). Data mining—reaching beyond statistics.Res. Official Statist. 2 5–17.

Hand, D. J. (1999). Statistics and data mining: intersecting dis-ciplines. SIGKDD Exploration 1 16–19.

Hand, D. J., Mannila, H. and Smyth, P. (2000). Principles ofData Mining. MIT Press.

Harrison, D. (1993). Backing up. Neural Computation 98–104.Heckerman, D., Mannila, H., Pregibon, D. andUthurusamy, R.

(eds.) (1997). Proceedings of the Third International Confer-ence on Knowledge Discovery and Data Mining. AAAI Press,Menlo Park, CA.

Klosgen, W. (1998). Analysing databases with knowledge dis-covery methods. Res. Official Statist. 1 9–35.

Leighton, G. and McKinlay, P. L. (1930). Milk Consumptionand the Growth of School Children. H.M. Stationery Office,Edinburgh.

Mannila, H., Toivonen, H. and Verkamo, A. I. (1997). Discov-ery of frequent episodes in event sequences. Data Miningand Knowledge Discovery 1 259–289.

Mihalisin, T. and Timlin, J. (1997). Fast robust visual data min-ing. In Proceedings of the Third International Conference onKnowledge Discovery and Data Mining (D. Heckerman, H.Mannila, D. Pregibon and R. Uthurusamy, eds.) 231–234.AAAI Press, Menlo Park, CA.

Scott, D. W. (1992).Multivariate Density Estimation. Wiley, NewYork.


CommentWilliam Kahn

The current approach to direct marketing empha-sizes data analysis and, as such, is grossly subop-timal. By approaching direct marketing in a moretraditionally scientific manner, one that involves allaspects of the research process, significant new eco-nomic value can be generated. Specifically, compa-nies will be much more successful if they use mod-ern experimentation than by simply improving theirdata analysis.

1. INTRODUCTION

As an active seeker of both fun and profit, andas a statistician in the financial sector, I welcomethe opportunity to comment on Hand, Blunt, Kellyand Adams. The topic is extremely timely and rele-vant, with tens of millions of dollars spent annuallywithin individual financial firms and with even rel-atively small data mining firms being bought forhundreds of millions of dollars (New York Times,Oct., Nov., 1999).However, while I appreciate the importance of

data mining, in practice the profit it brings hasturned out to be surprisingly limited in many keybusinesses. W. Edwards Deming began his 1943statistics book, The Statistical Adjustment of Datawith “The purpose of collecting data is to provide abasis for action.” It is in this spirit, the spirit of help-ing businesses make better decisions, that I wish todiscuss an extension of Hand et al. beyond analysisof large data sets to the active collection of usefuldata.

2. THE CONTINUOUS LEARNING CYCLE

Before I jump into some of the details of researchdesign, let me present an overview of learningwhich my clients have found useful. The coreobservation is that businesses make decisions.Some decisions, the strategic ones, can never berepeated or replicated, yet have significant impact.Examples include decisions to develop a franchisedistribution model, to demutualize or to expand

William Kahn is Professor, Mitchell Madison Group,West 57th Street, New York, NY 10019 (e-mail:[email protected]).

internationally. These decisions require leadership,commitment and insight into industry structure.Most business decisions, however, are tactical andcan be—and are—repeated. Examples include howmuch to charge, to whom to send mailings andwhich product features to highlight. My main the-sis is that those decisions which are repeated shouldbe tracked and managed so as to be continuouslyimproved.I have broken this continuous learning cycle into

eight basic operations: data archiving, financialanalysis, brainstorming, experimental design, oper-ations, data collection, data analysis and implemen-tation. Hand et al., in focusing on just part of thedata analysis issues, find themselves working withimmense amounts of data which contain very lit-tle information, while trying to reflect on poorlydefined business problems. A systematic approachto the entire statistical problem will greatly improvedata mining’s contribution to business success. Forexample, one of my clients annually mails 340 mil-lion pieces of mail. One of their core business ques-tions was to whom they should remail, that is, maila second advertisement. Unfortunately, in all thehistorical data, every name from a certain sourcehad been remailed and only those names had beenremailed. There was perfect confounding betweenname source and remailing. No amount of dataanalysis on the one gigabyte data set we weregiven would ever allow data-driven estimation ofthe value of remailing. To allieviate this and relatedproblems, we use the following eight steps to helpguide our direct marketing clients.

2.1 Data Archiving

Data warehousing companies have been ex-tremely successful over the past decade in sellingmultiyear, multimillion dollar corporate-wide, inte-grated, data warehouses. In retrospect, a large frac-tion of the companies investing in these projectshave felt that the value generated from the invest-ment was minimal. Terabyte databases are nowcommon. While, as a statistician, I am happy tohave data, the quest for size in these architecturesis so ambitious that the usability and business valueis normally left far behind. Most commonly, thewarehouses support the creation of hundred-page


weekly tallies, which hundreds of managers receiveand no one actually uses to support decisions. Ourexperience is that wisely selected data, for example,fewer than one hundred fields per customer, turnsout to have nearly all of the value and to be eas-ily manageable. We encourage timely use of datawhich is immediately available with additions tothe data driven by documented value and availabletime.

2.2 Financial Analysis

Fundamentally, business decisions are made in anattempt to improve specific business metrics. Thesemetrics become the response variables in predictivemodelling. Most companies, however, grossly sub-optimize their business. It is common, for example,for a credit card company to reward the direct mar-keters for more credit card applications, regardlessof the credit quality of the applicants. The creditevaluation group tries to minimize credit losses, butends up weeding out extremely profitable customerswho use the credit line more extensively than theoriginal limit allowed. The servicing group is mea-sured on minimal cost and not rewarded for effectivecross-sell, and the collections group minimizes res-olution time instead of total future lifetime value.In summary, each stage in the customer life cycle isoptimized, based on local measures, a process whichgreatly suboptimizes the overall business system. Itis crucial to build an integrated financial model ofcustomer activity in order to have a proper view-point of long-term customer value, conditional oneverything known about the customer up to thatpoint.

2.3 Brainstorming

As in any scientific endeavor, a thorough reviewof all possible stimuli must be made before decidingon the key factors which will be studied. A busi-ness unit must decide which factors are under itscontrol and could, therefore, be studied. It is thuscrucial to have a view of everything a businesscould do. This involves explicitly knowing every-thing the company is trying, and has tried, bothrecently and long ago. Furthermore, knowledge ofeverything competitors are doing is obviously key.Understanding of the theoretical and academic lit-erature, as well as of innovative and out-of-the-boxapproaches, will prove useful. In standard directmarketing there are dozens of potentially importantfactors, including the impact of color, paper quality,

reading level, telemarketing, font size, mail class,delivery day, price and free gifts.

2.4 Experimental Design

Given the dozens, if not hundreds, of decisionsdirect marketers must make, it is not suprisingthat many of the decisions are based on luck andfolk-knowledge. However, without the use of formalexperimental design methodologies, it is impossi-ble to learn the impact of all the decisions whichmust be made. Very little of the $250 billion spentevery year in the United States in direct marketingis part of an experiment with complexity beyondcase-control. 22 designs are found on occasion, butthe 22�5−1�, wherein five two-level factors and all 10two-way interactions are studied in 16 packages, adesign common in agriculture and engineering, isvirtually unheard of in this field. I have run five-factor d-optimal designs in direct marketing withtremendous impact. Further, given the ability ofmodern in-line package production, it is now pos-sible to customize every individual offering. I hopeto run a full factorial 2ˆ15 design in the near future.While the interaction effects will likely be ignoredin the analysis, given no incremental cost over the16 offering, highly fractionated, 2 ˆ �15-11� versionof the design, there is no reason not to run full fac-torial.

2.5 Operations

In traditional scientific fields there is a constanttension, usually good-natured, between the theoreti-cians and the experimentalists. One component ofthis banter is the knowledge, by both groups, of theextreme difficulty in making laboratory equipmentactually work or of making field observations reli-ably. In the lab, cables won’t mate, and when theydo, the connections are intermittent. In the field,perhaps the rainy season starts three weeks early,and the only poncho has a hole and drips rain overthe camera. Similarly, in the actual execution of tenmillion piece (and larger) mail drops, much can, anddoes, go wrong. It is hard to ensure the integrityof apparently simple basic operations like coloralignment, address validation and postage meter-ing. When a theoretician asks direct mail opera-tors to handle, say eight separate packages withcorrect randomization, the response is a resound-ing “Impossible!” After much explanation, cajolingand sometimes additional funding, the answer canbe driven to “Well, OK, we’ll try.” However, whilethe eight packages are indeed actually produced,


almost certainly they are mailed on successive days,and not randomly. Making experimentation workrequires great attention to the capability of the oper-ations team. It must be overseen by a principalinvestigator with hourly attention to detail.

2.6 Data Collection

The response to direct solicitation, whether mail,e-mail, telemarketing, in person or web, must becollected reliably and quickly. Initially, the responsemay just be an indicator that the targeted individualwas home. Later response measures would includeexpressed interest in the product, order quantity,net profitability and, finally, up-sell and cross-sellproducing a response equivalent to total customer-value. It is crucial to pick up each of these cus-tomer responses as soon as the data become avail-able. I have seen large business operations ($100million annual revenue) continue to use three-year-old response data in fields which change extremelyrapidly, such as mail-order PCs. When working ina rapidly changing market, collecting timely data iscrucial.

2.7 Data Analysis

Having collected timely response data from adesigned experiment and having built a properfinancial model for the economic value of thechanged customer behavior, we are now in a posi-tion to model the change in customer value as afunction of two sets of variables. The first set arethe factors used in the design, that is, the factors wecan actively control. The second set are the covari-ates which describe each customer that resides inour data archive. Note that we are not interested inwhich package features are best. Nor are we inter-ested in which individuals to mail. Rather, we mustmodel the best possible offer to mail to each indi-vidual customer. The analysis of the design factorsin and of themselves is fairly simple—basic aver-ages normally do pretty well in balanced designs.However, the introduction of the interactions of thedesign factors with several dozen or more covari-ates greatly complicates the analysis. Typically, thecovariates are extremely dirty data, and robust pro-cedures are absolutely required. Hand et al. dis-cuss many of the issues surrounding this phase ofthe research. Let me add from my personal expe-rience that with good quality design data, almostany modelling approach yields significant insights.Adding robust procedures gives significant lift, andthe use of interpretable models, such as recursivetrees, gives good internal saleablity.

2.8 Implementation

We are now in an enviable position. We have a for-mal mathematical model which gives the best pos-sible product to offer to every individual. We alsocan predict the value generated by any alternativeoffer. Thus, we can estimate the value being gen-erated by our optimal assignment process. This isoften a startingly large number; we have routinelylearned how to double the economic return in directmailing. One interpretation of this result is thatbroad application of these procedures will halve theamount of junk mail received by American house-holds, while delivering the same amount of inter-esting and intriguing offers. The targeting proce-dure is simply to mail each customer the offer whichoptimizes the increase in value. Of course, the nulloffer—do nothing—is one of the possibilities. Fur-thermore, because we want to be able to update ourtargeting models continually, we retain some smallproportion of customers (say 1% to 10%) for random(as opposed to optimal) reassignment to new exper-iments.

3. CONCLUSION

The continuous learning cycle discussed will bevery familar to readers of this journal: it is just aspecific articulation of the scientific method. It isstructured in a way that business managers canunderstand and is sufficiently detailed so as tobe implemented relatively directly. While I haveused direct marketing as my running example, theeight steps are easily applied to many other busi-ness activities. For example, I have used this coremethodology in areas including residential mort-gage valuation, automobile collision pure premiumestimation and insurance underwriting.To some degree I have been intrigued by the lack

of use of experimental design in the financial ser-vice sector, despite its obvious tremendous poten-tial impact. But, of course, many other sectors ofour economy also fail to use experimental design,despite the potential value. More interesting, how-ever, is how isolated each of the eight business activ-ities are from each other in most businesses. Thedata warehousing team, by almost universally fix-ing on a large and inflexible data model, locks thebusiness into such a limited mode of operation thatthe data warehouse ends up ignored by the oper-ational teams. Or, more to the current point, thedata analysts work on data which simply does notcontain the information the business actually needs:the impact of every controllable factor on every indi-vidual.


This isolation of activities which should be inte-grated is a generally observed phenomenon. Whilecertainly there are sociological explanations for thebehavior, I suggest that proactive leadership in theintegration of the business processes is the only wayto actually find, and build, the required synergies.

I appreciate the work of Hand et al. in furtherdeveloping our thinking in special areas of dataanalysis and encourage them and all readers of thisjournal to expand our influence, both to increase thesophistication of all the tools used in our researchand to integrate all the core business activities.

Rejoinder:David J. Hand, Gordon Blunt, Mark G. Kelly and Niall M. Adams

We thank Dr. Kahn for his interesting comments.We agree that the profit so far generated by data

mining activities “has turned out to be surprisinglylimited in many key businesses.” We suggest thatone reason for this is the poor quality of much ofthe data, so that the unusual pattern, when it isfound, is more often due to the process of data col-lection than it is to the substantive content to whichit relates. However, we hope that recognition andappreciation of this fact will mean that more careis put on data collection in the future, not necessar-ily in the way that Kahn describes (or perhaps inaddition to this), but simply to ensure better qual-ity data which may be more effectively mined.We like the distinction between strategic and

tactical business decisions. The thesis that tac-tical decisions should be tracked and managedso as to be continuously improved is surely onewith which no one would disagree. Kahn’s anal-ysis of the continuous business learning cycleinto eight operations is certainly one legitimatepartition of the process. Paying careful attentionto these stages, and improving those that canbe improved, will doubtless lead to more effec-tive business decisions. However, as he says, inour paper we were concerned with information-thin, data-rich problems, and it was tools formaking the best of this situation that weaddressed.Surely, the point made under the heading Data

Archiving, that hundreds of managers receivehundred-page weekly tallies, demonstrates theimportance of the model-building aspect of datamining. Business decisions will only be aided ifthose “hundred-page weekly tallies” are reduced tomanageable sizes.The comments about specific business metrics

under the Financial Analysis heading rang bells.We have seen many situations in which the lefthand of a company did not know what the right

hand was doing. Figure 11(d) in our paper mightbe an example of this: the exciting discovery ofa dramatic change in customer behavior half-waythrough the time period loses its interest whenone is told that, at that time, all customers wererequired to repay by a particular method. This isnot the only example we have seen in which changesto the customer population, discovered by a riskassessment group, corresponded to shifts in mar-keting strategy imposed by the marketing depart-ment. This sort of thing is a powerful argumentfor more global perspectives on company manage-ment (and if data mining produces this result, thenthat alone would be a valuable consequence of theexercise).Admittedly, the drive behind the development of

data mining techniques has come from commercialsources (although the discipline is not restrictedto such problems). In view of this, it will comeas no surprise to hear that we entirely endorseKahn’s comment that “it is crucial to build anintegrated financial model of customer activity.”This is precisely the sort of model we illustratedin Hand, McConway and Stanghellini (1997) andStanghellini, McConway and Hand (1999). On theother hand, we feel that Kahn may be a little unre-alistic when he argues, under Brainstorming, that“it is thus crucial to have a view of everything abusiness could do. This involves explicitly knowingeverything the company is trying, and has tried,both recently and long ago. Furthermore, knowl-edge of everything competitors are doing is obvi-ously key.” In general, while we should clearly striveto improve the information on which we base ourdecisions, it will inevitably be at best partial.The comments under Experimental Design and

Implementation also ring true. In various contexts,we have had considerable difficulty convincing busi-ness of the merits of accepting a small sample of

REJOINDER 131

high-risk customers which, while perhaps not prof-itable in themselves, would lead to improved deci-sions overall.The example of population drift in a mail order

PC market is a nice one.We agree with Kahn that there is no doubt that

formal experimental design has tremendous poten-tial for positive impact on the financial services com-munity. The power of such techniques is somethingall statisticians would presumably promote. Ourpaper does not contradict that. We merely argue fora parallel, in-depth investigation of the data thathave been collected, and that will be collected.Dr. Kahn’s comments apply only to particular

kinds of data mining, those where the aim is to

optimize some response function (profit, perhaps).In other situations, however, the aim is simply toexplore the data in a search for interesting struc-tures.

ADDITIONAL REFERENCES

Deming, W. E. (1943). Statistical Adjustment of Data. Wiley, NY.(Republished in 1964 by Dover, New York.)

Hand, D. J., McConway, K. J. and Stanghellini, E. (1997).Graphical models of applicants for credit. IMA J. Math. Appl.Bus. Indust. 8 143–155.

New York Times, 6 October 1999, page 4.New York Times, 18 November 1999, page 4.Stanghellini, E., McConway, K. J. and Hand, D. J. (1999). A

chain graph for applicants for bank credit. J. Roy. Statist.Soc. Ser. C 48 239–251.

2000,Vol.15,No.2,111–131 DataMiningforFunandProﬁt

Documents

Transcript of 2000,Vol.15,No.2,111–131 DataMiningforFunandProﬁt