A review of possible effects of cognitive biases oninterpretation of rule-based machine learning models

Tomas Kliegra,∗, Stepan Bahnıkb, Johannes Furnkranzc

aDepartment of Information and Knowledge Engineering, Faculty of Informatics andStatistics, University of Economics Prague, Czech Republic

E-mail: tomas.kliegr@vse.czbDepartment of Management, Faculty of Business Administration,

University of Economics Prague, Czech RepublicE-mail: stepan.bahnik@vse.cz

c TU Darmstadt, Department of Computer ScienceHochschulstraße 10, D-64289 Darmstadt, Germany

Email: juffi@ke.tu-darmstadt.de

Abstract

This paper investigates to what extent do cognitive biases affect human under-standing of interpretable machine learning models, in particular of rules dis-covered from data. Twenty cognitive biases (illusions, effects) are covered, asare possibly effective debiasing techniques that can be adopted by designersof machine learning algorithms and software. While there seems no universalapproach for eliminating all the identified cognitive biases, it follows from ouranalysis that the effect of most biases can be ameliorated by making rule-basedmodels more concise. Due to lack of previous research, our review transfers gen-eral results obtained in cognitive psychology to the domain of machine learning.It needs to be succeeded by empirical studies specifically aimed at the machinelearning domain.

Keywords: cognitive bias, cognitive illusion, machine learning,interpretability, rule induction

1. Introduction

This paper aims to investigate the effect of cognitive biases on human under-standing of machine learning models, in particular inductively learned rules. Weuse the term cognitive bias as a representative term for various related cogni-tive phenomena (heuristics, effects, illusions and constraints) that demonstrateas seemingly irrational reasoning patterns that are thought to allow humans tomake fast and risk averse decisions. Following the “cognitive biases and heuris-tics” research program started by Tversky and Kahneman in the 1970s over 50

∗Corresponding author

Preprint submitted to Elsevier April 11, 2018

arX

iv:1

804.

0296

9v2

[st

at.M

L]

10

Apr

201

8

cognitive biases have been discovered to date [1]. Their cumulative effect on hu-man reasoning should not be underestimated as already the early work showedthat “cognitive biases seem reliable, systematic, and difficult to eliminate” [2].The effect of some cognitive biases is more pronounced when people do not havewell-articulated preferences [3], which is often the case in explorative machinelearning.

Previous works have analysed the impact of cognitive biases on multipletypes of human behaviour and decision making. A specific example is the semi-nal book “Social cognition” by Kunda [4], which is concerned with their impacton social interaction. Another, more recent work by Serfas [5] is focused on thecontext of capital investment. Closer to the domain of machine learning, in anarticle entitled “Psychology of Prediction” Kahneman and Tversky [6] warnedthat cognitive biases can lead to violations of the Bayes theorem when peoplemake fact-based predictions under uncertainty. These results directly relate toinductively learned rules, since these are associated with measures such as con-fidence and support expressing the (un)certainty of the prediction they make.Despite the early papers [7, 8] showing the importance of study of cognitivephenomena for rule induction and machine learning in general, there has beena paucity of follow-up research. In previous work [9], we have evaluated a selec-tion of cognitive biases in the very specific context of whether minimizing thecomplexity or length of a rule will also lead to increased interpretability, whichis often taken for granted in machine learning research.

In this paper, we attempt to systematically relate cognitive biases to theinterpretation of machine learning results. To that end, we review twenty cog-nitive biases that can distort interpretation of inductively learnt rules. Thereview is intended to help to answer questions such as: Which cognitive bi-ases affect understanding of symbolic machine learning models? What couldhelp as the “debiasing antidote”? We summarize the review by proposing amodel that describes which cognitive biases are triggered when humans assessthe plausibility of inductively learned rules and whether they influence plausi-bility in positive or negative way. The model is based on evidence transposedfrom general empirical studies in psychology to the particular domain of rulelearning. The purpose is to give an indicative holistic view of the problem inthe rule learning domain, and to foster empirical studies specifically aimed atthe machine learning domain.

This paper is organized as follows. Section 2 provides a brief review of relatedwork published at the intersection of rule learning and psychology, defining ruleinduction and cognitive bias on the way. Section 3 motivates our study on theexample of the insensitivity to sample size effect. Section 4 describes the criteriathat we applied to select a subset of cognitive biases into our review. The twentyselected biases are covered in Section 5. The individual cognitive biases havevery disparate effects and causes. Section 6 provides a discussion of our resultsand a concise summary in a form of an illustrative visual model. In Section 7we state the limitations of our review and outline directions for future work.The conclusions summarize the contributions.

2

IF A AND B THEN C

confidence=c and support=s

IF veil is white AND odour is foul THEN mushroom is poisonous

confidence = 90%, support = 5%

Figure 1: Inductively learned rule

2. Background and Related Work

We selected individual rules as learnt by many machine learning algorithmsas the object of our study. Focusing on simple artefacts—individual rules—asopposed to entire models such as rule sets or rule lists allows a deeper, morefocused analysis since a rule is a small self-contained item of knowledge. Makinga small change in one rule, such as adding a new condition, allows to test theeffect of an individual factor that can influence perception of rule plausibility. Inthis section, we will shortly introduce the inductively learnt rule. Then, we willfocus on rule plausibility as a measure of comprehension of rule comprehension.

2.1. Decision Rules in Machine Learning

The type of inductively learned decision rule which we consider is in Fig-ure 1. Following the terminology of Furnkranz et al. [10], A,B,C represent anarbitrary number of literals, i.e., Boolean expressions which are composed of at-tribute name (e.g., veil) and its value (e.g., white). The conjunction of literalson the left side of the rule is called antecedent, the single literal predicted bythe rule is called consequent. Literals in the antecedent are sometimes referredto as conditions throughout the text. While this rule definition is restricted toconjunctive rules, other definitions, e.g., the formal definition given by Slowinskiet al. [11, page 2] also allows for negation and disjunction as connectives.

Rules on the output of rule learning algorithms are most commonly char-acterized by two parameters, confidence and support. The confidence of a ruleis defined as a/(a + b), where a is the number of correctly classified objects,i.e. those matching the rule antecedent as well as the rule consequent, and b isthe number of misclassified objects, i.e. those matching the antecedent, but notthe consequent. The support of a rule is defined either as a/n, where n is thenumber of all objects (relative support), or simply as a (absolute support).

Some rule learning frameworks, in particular association rule learning [12, 13]require the user to set thresholds for minimum confidence and support. Onlyrules with confidence and support values meeting or exceeding these thresholdsare included on the output of rule learning and presented to the user.

2.2. Study of Rules in Cognitive Science

Rules are a commonly embraced model of human reasoning in cognitivescience [14, 15, 16]. They also closely relate to Bayesian inference, another

3

frequently used model of human reasoning. A rule “IF A AND B THEN C” canbe interpreted as a hypothesis corresponding to the logical implication A∧B ⇒C. We can express the plausibility of such hypothesis in terms of Bayesianinference as the conditional probability Pr(C | A,B). This corresponds to theconfidence of the rule, a term used in rule learning, and to strength of evidence,a term used by cognitive scientists [17].1

Given that Pr(C | A,B) is a probability estimate computed on a sample,another relevant piece of information for determining the plausibility of thehypothesis is the robustness of this estimate. This corresponds to the numberof observed instances for which the rule has been observed to be true. The sizeof the sample (typically expressed as ratio) is known as rule support in machinelearning and as weight of the evidence in cognitive science [17].

Rule as object of study in cognitive science. A hypothesis—a rule inductivelylearned from data—is also a very specific form of alternative. Psychologicalresearch specifically on hypothesis testing in rule discovery tasks has been per-formed in cognitive science at least since the 1960’s. The seminal article byWason [21] introduced what is widely referred to as Wason’s 2-4-6 task. Par-ticipants are given the sequence of numbers 2, 4 and 6 and asked to come upwith a rule that generates this sequence. In search for the hypothesized rulethey can ask the experimenter other sequences of numbers, such as 3-5-7 thatare either supposed to conform to the rule or not. The experimenter answersyes or no. While the target rule is simple “ascending sequence”, people find itdifficult to discover this specific rule, because they apply the confirmation bias,a human tendency to focus on evidence confirming the hypothesis at hand [22].

One of the later works in this area is entitled “Strategies of rule discoveryin an inference task” [23]. While the title could suggest that this work is highlyrelevant to our machine learning problem, it is actually a psychological studyof the inference processes (that is “meta-rules” people use in the reasoningprocess), which does not directly relate to the notion of “rules” used in machinelearning as particular pattern in data in a specific domain. Of similar limitedrelevance are follow-up works of Rossi et al. [24], Vallee-Tourangeau and Payton[25].

Another related field are cognitive theories of human decision making, whichstudy how humans combine multiple pieces of evidence, which in our case corre-spond to conditions (literals) in a rule. The contribution of individual conditionsto the overall plausibility of the rule is an important part of our research prob-lem, but there is a paucity of directly applicable research in cognitive science.Most of this research that we identified in our brief survey is based on Bayesianreasoning (studies by Gopnik and Tenenbaum [26], Griffiths et al. [27]), rather

1In the terminology used within the scope of cognitive science [18], confidence correspondsto the strength of the evidence and support to the weight of the evidence. Interestingly, thisproblem was already mentioned by Keynes [19] (according to Camerer and Weber [20]) whodrew attention to the problem of balancing the likelihood of the judgment and the weight ofthe evidence in the assessed likelihood.

4

than rule induction.If we consider an individual rule (hypothesis) as one of several alternatives

between which the user has to decide, we can apply research on human decision-making processes. Most notably, this includes results of the research program oncognitive heuristics and biases started by Amos Tversky and Daniel Kahnemanin the 1970’s. In our work, we draw heavily from this intensely studied area ofhuman cognition.

2.3. Cognitive Bias

According to the Encyclopedia of Human Behavior [28], the term cognitivebias was introduced in the 1970s by Amos Tversky and Daniel Kahneman [17],and is defined as:

Systematic error in judgment and decision-making common to allhuman beings which can be due to cognitive limitations, motiva-tional factors, and/or adaptations to natural environments.

The research on cognitive biases and heuristics is considered as the most impor-tant psychological research done in the past 40 years [28].

The narrow initial definition of cognitive bias as a shortcoming of humanjudgment was criticized by German psychologist Gerd Gigerenzer, who startedin the late 1990s the “Fast and frugal heuristic” program to emphasize ecologicalrationality (validity) of cognitive biases.

As for terminology, the concept of “cognitive bias” includes many cognitivephenomena, multiple of which are not called “biases” but instead heuristics (e.g.Representativeness heuristic), effects (e.g. Mere exposure effect), fallacies (e.g.Conjunction fallacy), illusions (e.g. Illusionary correlation) or otherwise.

Three types of cognitive biases are recognized in the recent authoritativework of Pohl [1]: those relating to thinking, judgment and memory. The Think-ing category covers biases related to thinking processes. These require theperson to apply a certain rule (such as Bayes theorem). Since many peopleare not aware of this rule, they have to apply it intuitively, which can resultin errors. The Judgment category covers biases used by people when they areasked to rate some property of a given object (such as a plausibility of a rule).The Memory category covers biases related to memory, which deal mainly withthe phenomena of various memory errors, such as omission substitution andinterpolation.

Function and validity of cognitive biases. In the introduction, we briefly char-acterized cognitive biases as “seemingly irrational reasoning patterns that arethought to allow humans to make fast and risk averse decisions.” In fact, thefunction of cognitive biases is subject of scientific debate. According to thereview of functional views in Pohl [1], there are three fundamental positionsamong researchers. The first group considers them as dysfunctional errors ofthe system, the second group as faulty by-products of otherwise functional pro-cesses and the third group as adaptive and thus functional responses. According

5

to Pohl [1], most researchers are in the second group, where cognitive biases (il-lusions) are consider to be “built-in errors of the human information-processingsystems”.

In this work, we consider cognitive biases as strategies that evolved to im-prove the fitness and chances of survival of the individual in particular situations.This stand in defense of biases is succinctly expressed by the influential workof Haselton and Nettle [29]: “Both the content and direction of biases can bepredicted theoretically and explained by optimality when viewed through thelong lens of evolutionary theory. Thus, the human mind shows good design, al-though it is design for fitness maximization, not truth preservation.” Accordingto the same paper, empirical evidence shows that cognitive biases are triggeredor their effect strengthened by environmental cues and context [29].

Interpretation of statistical hypotheses and machine learning results is a veryrecent type of cognitive task. We thus assume that when interpreting machinelearning results, the human mind applies many of the heuristics and biasesinappropriately. It also follows that cognitive biases will not demonstrate in allhumans and in all situations equally (or many times even at all).

2.4. Measures of Interpretability, Perceived and Objective Plausibility

We claim that cognitive biases can affect the interpretation of rule-basedmodels. However, how does one measure interpretability? According to ourliterature review, there is no generally accepted measure of interpretability ofmachine learning models. Model size, which was used in several studies, hasrecently been criticized [30, 31, 9].

In our work, we embrace the concept of plausibility to measure interpretabil-ity. Plausibility is defined according to the Oxford Dictionary of US English as“seeming reasonable or probable” and according to the Cambridge dictionaryof UK English as “Seeming likely to be true, or able to be believed”. In the pre-vious, we linked the machine learning’s inductively learned rule to the conceptof “hypothesis” used in cognitive science. There is a body of work in cognitivescience on analyzing the perceived plausibility of hypotheses [32, 33, 34]. Plausi-bility can also be directly elicited for inductively learnt rules [35]. The conceptsof “trust” and “acceptance” are used in connection with measuring comprehen-sion of machine learning models in the influential position paper by Freitas [30].Plausibility is also closely related to the term justifiability, which requires theexpert to assess that the model is in line with existing domain knowledge. In arecent review of interpretability definitions by Bibal and Frenay [36], the termplausibility is not explicitly covered, but justifiability is stated to depend oninterpretability. Martens et al. [37] define justifiability as “intuitively correctand in accordance with domain knowledge”.

We are aware of the fact that if a decision maker finds a rule plausible, itdoes not necessarily mean that the rule is correctly understood, it can be quitethe contrary in many cases. Nevertheless, we believe that the alignment ofthe perceived plausibility with objective, data-driven, plausibility of a hypothesisshould be at the heart of an effort that strives for interpretable machine learning.

6

3. Motivational Example

It is well known in machine learning that chance rules with a deceptively highconfidence can appear in the output of rule learning algorithms [38]. For thisreason, the rule learning process typically outputs both confidence and supportfor the analyst to make an informed choice about merits of each rule.

Example.• IF a film is released in 2006 AND the language of the

film is English THEN Rating is good,

confidence = 80%, support = 10%.

• IF a film is released in 2006 AND the director was John

Smith THEN Rating is good,

confidence = 90%, support = 1%.

In the example listing above, both rules are associated with values of confidenceand support to inform about the strength and weight of evidence for both rules.While the first rule is less strong (80% vs 90% correct), its weight of the evidenceis ten times higher than of the second rule.

According to the insensitivity to sample size effect [17] there is a systematicbias in human thinking that makes humans put higher weight on the strengthof evidence (confidence) than on the weight of evidence (support). It has beenshown that this bias is applicable also to statistically sophisticated psychologists[39] and thus can be applicable to the widening number of professions that areusing rule learning to obtain insights from data.

The second bias that we consider is base rate fallacy, according to whichpeople are unable to correctly process conditional probabilities. The condi-tional probability in our example is the confidence value, which—in the shownexample—is the probability of a good rating on the condition of the film beingreleased in 2006 and in English.

The analysis of relevant literature from cognitive science not only revealsapplicable biases, but also provides in some cases methods for limiting theireffect (debiasing). The standard way used in rule learning software for displayingrule confidence and support metrics is to use ratios, as in our example. Extensiveresearch in psychology has shown that if natural numbers are used instead thenthe number of errors in judgment drops [40, 41]. Reflecting these suggestions,the first rule in our example could be presented as follows:

Example.• IF a film is released in 2006 AND the language of the

film is English THEN Rating is good.

In our data, there are 100 movies which match the

conditions of this rule. Out of these, 80 are predicted

correctly as having good rating.

7

A correct understanding of machine learning models can be difficult even forexperts. In this section, we tried to motivate why addressing cognitive biasescan play an important role in making the results of inductive rule learning moreunderstandable. In the remainder of this paper, both biases involved in ourexample will be revisited in greater depth, along with 18 other biases.

4. Selection Criteria

A number of cognitive biases have been discovered, experimentally studied,and extensively described in the literature. There are at least 51 different biasesfalling into the thinking and judgment categories [42, 1]. As Pohl [1] states in arecent authoritative book on cognitive illusions: “There is a plethora of phenom-ena showing that we deviate in our thinking, judgment and memory from someobjective and arguably correct standard.” In the first phase of our research weselected a subset of biases which will be reviewed. To select applicable biases,we considered those that have some relation to the following properties of in-ductively learned rules: 1. rule length (number of literals in antecedent), 2. ruleinterest measures (especially support and confidence), 3. position (ordering) ofconditions in rule and ordering of rules in the rule list, 4. Specificity and predic-tive power of conditions (correlation with target variable), 5. use of additionallogical connectives (conjunction, disjunction, negation), 6. treatment of missinginformation (inclusion of conditions referring to missing value), and 7. conflictbetween rules in the rule list.

Through selection of appropriate learning heuristics, the rule induction learn-ing algorithm can influence these properties. For example, most heuristics im-plement some form of trade-off between the coverage or support of a rule, andits implication strength or confidence [43, 10].

Inclusion of “overlapping” biases. We do not consider the correlation betweenindividual cognitive biases. For example, it is known that a number of cognitivebiases (such as conjunction fallacy, base rate neglect, insensitivity to sample size,confusion of the inverse) can all be attributed to a more general phenomenoncalled representativeness heuristic [44]. To our knowledge, the correlation be-tween cognitive biases has not yet been systematically studied. In our review,we thus include multiple biases even though they may overlap.

5. Review of Cognitive Biases

In this section, we cover a selection of twenty cognitive biases. For all ofthem, we include a short description, and a paragraph which quantifies the effectof the cognitive bias. We pay particular attention to their potential effect on theinterpretability of rule learning results, which has not been covered in previousworks. For those biases that were categorized by Pohl [1], the name of thecategory (Thinking, Judgment) appears in parenthesis in subsection heading.We have not included any biases categorized into the Memory category.

8

For all cognitive biases we suggest a debiasing technique that could be effec-tive in aligning the perceived plausibility of the rule with its objective plausibil-ity. The suggestions are based on empirical results obtained by psychologists,we indicate when these are our conjectures that are in need of further validation.Most biases have a limited validity scope. In our review we thus put attentionto identifying groups of people who are (not) susceptible to the specific biaswherever this information is available. Also, for selected biases, we report thesuccess rates, i.e., the number of people committing a fallacy corresponding toa specific bias in an experiment.

An overview of the main traits of the reviewed cognitive biases is presentedin Table 1.

5.1. Base-rate Fallacy (Thinking)

The base-rate fallacy indicates that people are unable to correctly processconditional probabilities.

Success rates. In the original experiment reported in Kahneman and Tversky[6] more than 95% of psychology graduate students committed the fallacy.

Implications for rule learning. The application of the base rate fallacy suggeststhat when facing two otherwise identical rules with different values of confidenceand support metrics, an analyst’s preferences will be primarily shaped by theconfidence of the rule.

It follows that by its preference for higher confidence, the base-rate fallacywill generally contribute to a positive correlation between rule length and plau-sibility, since longer rules can better adapt to a particular group in data andthus have a higher confidence than a more general, shorter rules. This is in con-trast to the general bias for simple rules that is implemented by state-of-the-artrule learning algorithms because simple rule tend to be more general, have ahigher support, and are thus statistically more reliable.

Debiasing techniques. Our literature review has surfaced several techniques foraddressing the base-rate fallacy. Gigerenzer and Hoffrage [41] show that repre-sentations in terms of natural frequencies, rather than conditional probabilities,facilitate the computation of cause’s probability. To the authors’ knowledge,confidence is typically presented as ratio in current software systems. The sup-port rule quality metric is sometimes presented as a ratio and sometimes as anatural number. It would foster correct understanding if analysts are consis-tently presented with natural frequencies in addition to ratios.

5.2. Confirmation Bias and Positive Test Strategy (Thinking)

Confirmation bias is the best known and most widely accepted notion ofinferential error of human reasoning [45, p. 552].2 This bias refers to the no-tion that people tend to look for evidence supporting the current hypothesis,

2Cited according to Nickerson [22].

9

ph

enom

enon

imp

lica

tion

sfo

rru

le-l

earn

ing

deb

iasi

ng

tech

niq

ue

bia

ses

that

incr

ease

pla

usi

bilit

yw

ith

rule

len

gth

Availab

ilit

yH

euri

stic

Pre

dic

tive

stre

ngth

of

lite

ral

isb

ase

don

ass

oci

ati

on

bet

wee

nth

elite

rals

inth

eante

ced

ent

an

dth

eco

nse

qu

ent

of

the

rule

Tri

gger

Syst

em2

Aver

agin

gH

euri

stic

Pro

bab

ilit

yof

ante

ced

ent

as

the

aver

age

of

pro

bab

ilit

ies

of

lite

rals

Rem

ind

erof

pro

bab

ilit

yth

eory

Base

-rate

Fallacy

Em

ph

asi

son

con

fid

ence

,n

egle

ctfo

rsu

pp

ort

Exp

ress

con

fid

ence

an

dsu

pp

ort

inn

atu

ral

freq

uen

cies

Con

firm

ati

on

Bia

sR

ule

sco

nfi

rmin

gth

eir

pri

or

hyp

oth

esis

are

“ch

erry

pic

ked

”i)

Exp

lici

tgu

idan

ceto

con

sid

erev

iden

cefo

ran

dagain

sthy-

poth

esis

,ii)

Scr

een

an

aly

sts

for

susc

epti

bilit

yu

sin

gD

efen

seC

on

fid

ence

Sca

lequ

esti

on

nair

eD

isju

nct

ion

Fallacy

Pre

fer

more

spec

ific

lite

rals

over

less

spec

ific

Info

rmon

taxon

om

ical

rela

tion

bet

wee

nlite

rals

,ex

pla

inb

en-

efits

of

hig

her

sup

port

*E

ffec

tof

Diffi

cult

yR

ule

sw

ith

small

sup

port

an

dh

igh

con

fid

ence

are

“over

rate

d”

Filte

rru

les

that

do

not

pass

ast

ati

stic

al

sign

ifica

nce

test

,ex

-p

lain

ben

efits

of

hig

her

sup

port

*In

form

ati

on

Bia

sM

ore

lite

rals

incr

ease

pre

fere

nce

Vis

uali

zevalu

eof

lite

rals

,so

rtby

pre

dic

tive

valu

e*In

sen

siti

vit

yto

Sam

ple

Siz

eA

naly

std

oes

not

realize

the

incr

ease

dre

liab

ilit

yof

con

fid

ence

esti

mate

wit

hin

crea

sin

gvalu

eof

sup

port

Use

sup

port

toco

mp

ute

con

fid

ence

(rel

iab

ilit

y)

inte

rvals

for

the

valu

eof

con

fid

ence

Mer

eE

xp

osu

reE

ffec

tR

epea

ted

exp

osu

reto

lite

ral

resu

lts

inin

crea

sed

pre

fere

nce

Extr

ati

me

or

kn

ow

led

ge

of

lite

rals

*M

isu

nd

erst

an

din

gof

“an

d”

“an

d”

isu

nd

erst

ood

as

dis

jun

ctio

nE

xp

ress

lite

ral

as

pro

posi

tion

rath

erth

an

as

cate

gory

Neg

ati

vit

yB

ias

Neg

ate

dor

neg

ati

ve

lite

rals

are

con

sid

ered

as

more

imp

ort

ant

Avoid

the

use

of

neg

ati

on

*R

ecogn

itio

nH

euri

stic

Rec

ogn

itio

nof

lite

rals

incr

ease

sp

refe

ren

ceE

xtr

ati

me

or

kn

ow

led

ge

of

lite

rals

Rei

tera

tion

Eff

ect

Sam

elite

ral

pre

sent

inm

ult

iple

rule

sin

crea

ses

pre

fere

nce

Pru

nin

galg

ori

thm

s*R

epre

senta

tiven

ess

Heu

rist

icO

ver

esti

mate

the

pro

bab

ilit

yof

lite

ral

rep

rese

nta

tive

of

targ

etU

sen

atu

ral

freq

uen

cies

inst

ead

of

rati

os

Tra

deo

ffC

ontr

ast

Pre

fere

nce

for

lite

ral

isin

flu

ence

dby

oth

erlite

rals

inth

eru

leor

inoth

erru

les

i)P

run

ing

dis

cover

edru

les,

ii)

Info

rmati

on

on

sem

anti

csof

the

lite

ral

an

dco

vari

an

cew

ith

oth

erlite

rals

*U

nit

Bia

sL

iter

als

are

per

ceiv

edto

have

sam

ew

eight

Info

rmon

dis

crim

inato

ryp

ow

erof

lite

rals

*

bia

ses

that

dec

rease

pla

usi

bil

ity

wit

hru

lele

ngth

Am

big

uit

yA

ver

sion

Pre

fer

kn

ow

nlite

ral

over

un

kn

ow

nlite

ral

Tex

tual

des

crip

tion

of

lite

rals

*W

eak

Evid

ence

Eff

ect

Lit

eral

on

lyw

eakly

per

ceiv

edas

pre

dic

tive

of

targ

etd

ecre

ase

sp

lau

sib

ilit

yO

mis

sion

of

wea

kp

red

icto

rsfr

om

ante

ced

ent

effec

tin

dep

end

ent

of

rule

len

gth

Con

fusi

on

of

the

Inver

seC

on

fusi

ng

the

diff

eren

ceb

etw

een

the

con

fid

ence

of

the

rule

Pr(

con

sequ

ent|a

nte

ceden

t)w

ith

Pr(

ante

ced

ent|c

on

sequ

ent)

NA

Pri

macy

Eff

ect

Ru

les

that

are

pre

sente

das

firs

tin

the

rule

mod

elare

more

pre

ferr

edS

ort

rule

sfr

om

stro

nges

tto

wea

kes

t*

Tab

le1:

Su

mm

ary

of

an

aly

sis

of

cogn

itiv

eb

iase

s.*

con

ject

ure

,it

sab

sen

cem

ean

sd

eriv

edfr

om

emp

iric

al

ob

serv

ati

on

.

10

disregarding conflicting evidence. Research suggests that even neutral or un-favourable evidence can be interpreted to support existing beliefs, or, as Tropeet al. [46, p. 115-116] put it, “the same evidence can be constructed and re-constructed in different and even opposite ways, depending on the perceiver’shypothesis.”

A closely related heuristic is the Positive Test Strategy (PTS) proposed byKlayman and Ha [47]. This heuristic suggests that when trying to test a specifichypothesis, people examine cases which they expect to confirm the hypothesisrather than the cases which have the best chance of falsifying it. The differencebetween PTS and confirmation bias is that PTS is applied to test a candidatehypothesis while the true confirmation bias is concerned with hypotheses thatare already established [48, p. 93]. The experimental results of Klayman andHa [47] show that under realistic conditions, PTS can be a very good heuristicfor determining whether a hypothesis is true or false, but it can also lead tosystematic errors if applied to an inappropriate task.

Finally, it should be noted that, according to a review conducted by Klaymanand Ha [47], this heuristic is used as a “general default heuristic” in situationswhere either specific information that identifies some tests as more relevantthan others is absent or when the cognitive demands of the task prevent a morecareful strategy.

Success rates. According to Mynatt et al. [49, p. 404], 70% of the subjects didnot abandon falsified hypotheses in an experiment that simulated a researchenvironment.3 This success rate is particularly relevant for the problem ofcomprehending rule learning results as the simulated research environment isclose to our target domain of analysts interpreting discovered rules.

Implications for rule learning. This bias can have significant impact dependingon the purpose for which the rule learning results are used. If the analyst hadsome prior hypothesis before she obtained the rule learning results, accordingto the confirmation bias she will tend to “cherry pick” rules confirming thisprior hypothesis and disregard rules that contradict it. Given that some rulelearners may output contradicting rules, the analyst can select only the rulesconforming to the hypothesis, disregarding applicable rules with the oppositeconclusion, which could otherwise turn out to be more relevant.

Using evidence gathered using MRI brain scans Westen et al. [50] observeconfirmation bias and explain it by emotions related to the favoured hypothesis.Evidence that challenges such a preferred hypothesis is involuntarily suppressed.The experiments in this study were conducted by presenting information thatchallenged the moral integrity of the politician that the subject favoured. Whileit could be argued that data analysts interpreting the rule learning results arefree of emotional bonds to the problem and can be trained to correctly interpretmachine learning results, they may still be subject to a confirmation bias:

3Result for the experiment performed in a “complex” environment.

11

• Stanovich et al. [51] show that incidence of myside bias, which is closelyrelated to confirmation bias, is surprisingly not related to general intelli-gence. This suggests that even highly intelligent analysts can be affected.

• Some research can even be interpreted as indicating that data analystscan be more susceptible to the myside bias than the general population.An experiment reported by Wolfe and Britt [52] shows that subjects whodefined good arguments as those that can be proved by facts (this stance,we assume, would also apply to many data analysts) were more prone toexhibiting a myside bias.4

Debiasing techniques. Tweney et al. [23] successfully tested a modification ofWason’s 2,4,6 task. In its original setup, participants try to “discover” therule according to which the sequence 2, 4, 6 was created. The correct answer is”ascending sequence of numbers”. In the modification by Tweney et al. [23],participants were asked to search for two rules (“any ascending sequence ofnumbers” and “all other sequences”) instead of one rule (“ascending sequenceof numbers”). Following this, the response format was changed from positiveand negative to whether the rule belongs to the first category “DAX” or thesecond category “MED”, which improved performance in the task. Thus, weconclude that relabeling categories from “positive” and “negative” to somethingmore neutral can possibly help to debias the analysts’ interpretation of the rulelearning result.

Albarracın and Mitchell [53] suggest that the susceptibility to the confirma-tion bias can depend on one’s personality traits. This publication also presentsa diagnostic tool called “defense confidence scale” that can identify individualswho are prone to confirmational strategies.

Wolfe and Britt [52] successfully experimented with providing the subjectswith explicit guidelines for considering evidence both for and against the hypoth-esis. While this research is not directly related to hypothesis testing, providingexplicit guidance combined with modifications of the user interface of the systempresenting the rule learning results could also prove to be an effective debiasingtechnique.

5.3. Conjunction Fallacy and Representativeness Heuristic (Thinking)

Human-perceived plausibility of hypotheses has been extensively studied incognitive science. One of the best-known cognitive phenomena related to ourfocus area of rule plausibility is the Conjuctive fallacy. This fallacy falls intothe research program on cognitive biases and heuristics carried out by AmosTversky and Daniel Kahneman since the 1970s’.

4This tendency is explained as follows: “For people with this belief, facts and support aretreated uncritically. The intended audience is not part of the schema and thus ignored. Moreimportantly, arguments and information that may support another side are not part of theschema and are also ignored.”

12

This heuristic relates to the tendency to make judgments based on similarity,based on rule “like goes with like”, which is typically used to determine whetheran object belongs to a specific category. According to Gilovich and Savitsky[54], the representativeness heuristic can be held accountable for number ofwidely held false and pseudo-scientific beliefs, including those in astrology orgraphology.5 It can also inhibit valid beliefs that do not meet the requirementsof resemblance.

Linda problem. The conjunctive fallacy is in the literature often defined via the“Linda” problem [55, page 299], which was first used to demonstrate it.6 Inthis problem (Figure 2), subjects are asked to compare conditional probabilitiesPr(F,B | L) and Pr(B | L), where B refers to “bank teller”, F to “active infeminist movement” and L to the description of Linda [58].

Linda is 31 years old, single, outspoken, and very bright.

She majored in philosophy. As a student, she was deeply

concerned with issues of discrimination and social justice,

and also participated in anti-nuclear demonstrations.

Which is more probable?

(a) Linda is a bank teller.

(b) Linda is a bank teller and is active in the

feminist movement.

Figure 2: Linda problem

Multiple studies have shown that humans tend to consistently select thesecond, longer hypothesis, which is in conflict with the elementary law of prob-ability: the probability of a conjunction, Pr(A,B), cannot exceed the probabilityof its constituents, Pr(A) and Pr(B) [55]. In other words, it always holds forthe Linda problem that

Pr(F,B | L) ≤ Pr(B | L).

5Gilovich and Savitsky [54] give the following example: astrology relates the resemblanceof the physical appearance of a sign, such as a crab, with personal traits, such as a toughappearance on the outside. For graphology, the following example is given: handwriting tothe left is used to indicate that the person is holding something back.

6Note that the paper [55] also contains a different set of eight answer options for the Lindaproblem on page 297. The two option version on page 299 is prevalently used as a canonicalversion of the Linda problem in subsequent research (cf. the seminal paper of Gigerenzer [56,page 592]), and is referred to by Daniel Kahneman as the “more direct version” of the Lindaproblem [57, page 712].

13

Preference for alternative F&B (option b in Figure 2) is thus always a logicalfallacy. The conjunction fallacy has been shown to hold across multiple settings(hypothetical scenarios, real-life domains), as well as for various kinds of sub-jects (university students, children, experts, as well as statistically sophisticatedindividuals) [59].

Possible causes. According to Tversky and Kahneman [55], the results of theconjunctive fallacy experiments manifest that a conjunction can be more repre-sentative than one of its constituents. The conjunctive fallacy is a symptom of amore general phenomenon, in which people have a tendency to overestimate theprobabilities of representative events and underestimate those of less represen-tative ones. The reason is attributed to the application of the representativenessheuristic [55]. This heuristic provides humans with means for assessing a prob-ability of an uncertain event. It is used to answer questions such as “Whatis the probability that object A belongs to class B? What is the probabilitythat event A originates from process B?” According to the representativenessheuristic, probabilities are evaluated by the degree to which A is representativeof B, that is by the degree to which A resembles B [17].

The representativeness heuristic is not the only explanation for the resultsof the conjunctive fallacy experiments. Hertwig et al. [60] hypothesized that thereason is caused by “a misunderstanding about conjunction”, in other words by adifferent interpretation of “probability” and “and” by the subjects than assumedby the experimenters. The validity of this alternate hypothesis has been subjectto criticism [59], nevertheless the problem of correct understanding of “and”exists and is of particular importance to machine learning. Another proposedhypothesis for explaining the conjunctive fallacy is the averaging heuristic [61](cf. Section 5.8).

Success rates. Tversky and Kahneman [55] report that 85% of the subjectsindicate (b) as the more probable option for the Linda problem, which is definedin Figure 2. It should be noted that the actual proportion may vary, 83% arereported when the experiment was replicated by Hertwig and Gigerenzer [62],and 58% when replicated by [63].

Implications for rule learning. Rules are not composed only of conditions, butalso of an outcome (value of a target variable). A higher number of conditionsgenerally allows the rule to filter a purer set of objects with respect to the valueof the target variable than a smaller number of conditions. This means thatthe conjunctive fallacy does not directly manifest itself when interpreting rulelearning results since it cannot be stated that the selection of a longer ruleis a reasoning error in the rule learning context, even in cases when the setof conditions of the longer rule subsumes the set of conditions of the shorterrule. Nevertheless, application of representativeness heuristic can affect humanperception of rule plausibility, in that rules that are more ”representative” ofthe user’s mental image of the concept may be preferred even in cases whentheir objective discriminatory power may be lower.

14

Debiasing techniques. A number of factors that decrease the ratio of subjectsexhibiting the conjunctive fallacy as an undesired consequence of the represen-tativeness heuristic when its application is not rational have been identified:

• Charness et al. [63] found that the number of committed fallacies is re-duced under a monetary incentive. Such an addition is reported to dropthe fallacy rate to 33%. The observed rate under a monetary incentivebetter hints at smaller importance of this problem for real-life decisions.

• Zizzo et al. [64] found that unless the decision problem is simplified nei-ther monetary incentives nor feedback can ameliorate the fallacy rate. Areduced task complexity is a precondition for monetary incentives andfeedback to be effective.

• Stolarz-Fantino et al. [65] observed that the number of fallacies is reducedbut still strongly present when the subjects receive training in logics.

• Gigerenzer and Goldstein [40] as well as Gigerenzer and Hoffrage [41]show that the number of fallacies can be reduced or even eliminated bypresenting the problems in terms of frequencies rather than probabilities.

5.4. Availability Heuristic (Judgment)

The availability heuristic is a judgmental heuristic in which a person evalu-ates the frequency of classes or the probability of events by the ease with whichrelevant instances come to mind. This heuristic is explained by its discoverers,Tversky and Kahneman [66], as follows: “That associative bonds are strength-ened by repetition is perhaps the oldest law of memory known to man. Theavailability heuristic exploits the inverse form of this law, that is, it uses thestrength of the association as a basis for the judgment of frequency.”

To determine availability, it is sufficient to assess the ease with which in-stances or associations could be brought to mind – it is not necessary to performthe actual operations of retrieval or construction. An illustration of this phe-nomenon by Tversky and Kahneman [66] is: “One may estimate the probabilitythat a politician will lose an election by considering the various ways he maylose support.”

Success rates. Success rates for availability heuristics are very varied and dependgreatly on the experiment setup. Among other factors, they depend on the easeof recall [67]. In one of the original experiments (judgment of word frequency)presented by Tversky and Kahneman [66], the number of wrong judgments was105 out of 152 (70%). The task was to estimate whether letter “R” appearsmore frequently on first or third position in English texts. The reason whymost subjects incorrectly assumed the first position is that it is easier to recallwords starting with R than words with R on the third position.

15

Implications for rule learning. The application of availability heuristic is basedon the perceived association between the literals in the antecedent and theconsequent of the rule. The stronger this perceived association, the higher theperceived confidence of the rule. It is our opinion this heuristic will favourlonger rules, since they have higher chance to contain a literal which the analystperceives as associated with the predicted label.

It is true that the longer rule is also more likely to contain literals notperceived as associated. It can be argued that while the remaining weaklyassociated literals will decrease the preference for the longer rule, this effectcan be attributed to the weak evidence heuristic rather than the availabilityheuristic. However, according to our literature review, the availability heuristiccan only increase the preference level.

Debiasing techniques. Our initial review did not reveal any debiasing strategiesfor the availability heuristic. From a broader perspective, availability is associ-ated with the associative System 1, which can be corrected by the rule-basedSystem 2 [57]. Therefore, inducing conditions known to trigger engagement ofSystem 2 could be effective.

5.5. Effect of Difficulty (Judgment)

When an analyst is supposed to give a preference judgment between twocompeting hypotheses, one of the factors used in the decision making process isthe difficulty of the problem and the corresponding confidence that is related tothe judgment.

Griffin and Tversky [18] developed a model that combines the strength ofevidence with its weight (credibility). Their main research finding is that peo-ple tend to combine strength with weight in suboptimal ways, resulting in thedecision maker being too much or too little confident about the hypothesis athand than would be normatively appropriate given the information available.This discrepancy between the normative confidence and the decision maker’sconfidence is called overconfidence or underconfidence. Research has revealedsystematic patterns in overconfidence and underconfidence:

• If the estimated difference between the two hypotheses is large, it is easyto say which one is better, then there is a pattern of underconfidence.

• As the degree of difficulty rises (the difference between the normative confi-dence of two competing hypotheses is decreasing), there is a strengtheningpattern of overconfidence.

People use the provided data to assess the hypothesis at hand but they insuf-ficiently regard the quality of the data. Griffin and Tversky [18] illustrate thismanifestation of bounded rationality as follows: “If people focus primarily onthe warmth of the recommendation with insufficient regard for the credibility ofthe writer, or the correlation between the predictor and the criterion, they willbe overconfident when they encounter a glowing letter based on casual contact,and they will be underconfident when they encounter a moderately positiveletter from a highly knowledgeable source.”

16

Success rates. Griffin and Tversky [18] used regression to analyze the relationbetween the strength of evidence and weight of evidence. The conclusion wasthat the regression coefficient for strength was larger than the regression coeffi-cient for weight for 30 out of 35 subjects, which was found statistically signif-icant. The median ratio of these coefficients was established to be 2.2 to 1 infavour of strength.

Implications for rule learning. The strongest overconfidence was recorded forproblems where the weight of evidence is low and the strength of evidence ishigh. This directly applies to rules with high value of confidence and low valueof support. These are typically the longer rules. The empirical results relatedto the effect of difficulty therefore suggest that the predictive ability of suchrules will be substantially overrated by analysts. This is particularly interestingbecause rule learning algorithms often suffer from a tendency to unduely preferoverly specific rules that have a high confidence on small parts of the data tomore general rules that have a somewhat lower confidence, a phenomenon alsoknown as overfitting. The above-mentioned results seem to indicate that humanssuffer from a similar problem (albeit for presumably for different reasons), which,e.g., implies that a human-in-the-loop solution may not alleviate this problem.

Debiasing techniques. Similar to fighting overfitting in machine learning, weconjecture that this effect could be ameliorated by filtering out rules that donot pass a statistical significance test from the output and informing the userson the value and meaning of the value of statistical significance.

5.6. Mere Exposure Effect (Judgment)

According to this heuristic (effect), repeated exposure to an object resultsin an increased preference for that object. The mere exposure effect and therecognition heuristic are, according to Pachur et al. [68], two different phenom-ena, because unlike the latter, the mere exposure effect does not “require thatthe object is recognized as having been seen before”.

Success rates. As with other biases, the success rates for the mere exposureeffect are very varied and depend greatly on the experimental setup. Amongother factors, they depend on whether the stimulus the subject is exposed to isexactly the same as in prior exposure or similar to it [69]. Instead of selectingone particular success rate from a specific experiment, we can refer to the well-established finding that when a concrete stimulus is repeatedly exposed, thepreference for that stimulus increases logarithmically as a function of the numberof exposures [70].

Implications for rule learning. Already the initial research of Zajonc [71] in-cluded experimental evidence on the correlation between word frequency andaffective connotation of the word. From this it follows that a longer rule—asmeasured by word length rather than the number of conditions—will have agreater chance of containing a word that the analyst had been strongly exposed

17

to. Moreover, the exposure effects of individual words may possibly add up.This leads to the conclusion that mere exposure effect will increase plausibilityof longer rules.

Debiasing techniques. While our limited literature review did not reveal anydebiasing techniques, we conjecture that similarly to the related recognitionheuristic the knowledge of the criterion variable could ameliorate the mere ex-posure effect: presenting information on the semantics of the literal as well ason its covariance with other literals may suppress the heuristic.

5.7. Ambiguity Aversion

Ambiguity aversion corresponds to the finding that humans tend to preferknown risks over unknown risks. It is not a reasoning error. Consider the follow-ing comparison with the conjunctive fallacy. When a typical subject is explainedthe conjunctive fallacy, they will recognize their reasoning as an “error”, and,as Al-Najjar and Weinstein [72] put it, the subjects “feel embarrassed” for theirirrational choice. This contrasts with the ambiguity aversion, as for exampledemonstrated by the Ellsberg paradox [73], which shows that humans tend tosystematically prefer a bet with a known albeit very small probability of win-ning over a bet with a not precisely known probability of winning, even if itwould in practice mean a near guarantee of winning.

As follows from the research of Camerer and Weber [20], ambiguity aversionis related to the information bias: the demand for information in cases when ithas no effect on decision can be explained by the aversion to ambiguity: peopledislike having missing information.

Success rates. As noted by Camerer and Weber [20], Ellsberg did not performcareful experiments. According to the same paper, follow-up empirical work canbe divided into three categories: replications of Ellsberg’s experiment, determi-nation of psychological causes of ambiguity, and studies of ambiguity in appliedsetting. The most relevant to our work are experiments focusing on the appliedsetting. Curley et al. [74] describe an experiment in a medical domain where20% of subjects avoided ambiguous treatments.

Implications for rule learning. The ambiguity aversion may have profound im-plications for rule learning. The typical data mining task will contain a numberof attributes the analyst has no or very limited knowledge of. The ambiguityaversion will manifest itself in a preference for rules that do not contain am-biguous attributes or literals. Ambiguity aversion may also steer the analyst toshorter rules as these can be expected to have lower chance of containing anambiguous literal.

Debiasing techniques. We conjecture that this bias would be alleviated if textualdescription of the meaning of all the literals is made easily accessible to theanalyst.

18

5.8. Averaging Heuristic

While the representativeness heuristic is the most commonly associatedheuristic with the conjunctive fallacy, the averaging heuristics provides an al-ternate explanation: it suggests that people evaluate the probability of a con-juncted event as the average of probabilities of the component events [61].

Success rates. As reported by Zizzo et al. [64]: “approximately 49% of variancein subjects’ conjunctions could be accounted for by a model that simply averagedthe separate component likelihoods that constituted a particular conjunction.”This high success rate suggests that the averaging heuristic may be an importantsubject of further study within machine learning.

Implications for rule learning. The averaging heuristic can be interpreted to in-crease preference for longer rules. The reason is that longer rules are more likelyto contain literals with low probability. Due to the application of the averagingheuristic the analyst may not fully realise the consequences of the presence ofa low-probability literal for the overall likelihood of the set of conditions in theantecedent of the rule.

Consider the following example: Let us assume that the learning algorithmonly adds independent conditions that have a probability of 0.8, and we comparea 3-condition rule to a 2-condition rule. Averaging would evaluate both rulesequally, because both have an average probability of 0.8. A correct computationof the joint probability, however, shows that the longer rule is considerably lesslikely (0.83 vs. 0.82 because all conditions are assumed to be independent).

Averaging can also affect same-length rules. Fantino et al. [61] derive fromtheir experiments on the averaging heuristic that humans tend to judge “un-likely information [to be] relatively more important than likely information.”Continuing our example, if we compare the above 2-condition rule with anotherrule with two features with more diverse probability values, e.g., one conditionhas 1.0 and the other has 0.6, then averaging would again evaluate both rulesthe same, but in fact the correct interpretation would be that the rule with equalprobabilities is more likely than the other (0.82 > 1.0 × 0.6). In this case, thelow 0.6 probability in the new rule would “knock down” the normative conjointprobability below the one of the rule with two 0.8 conditions.

Debiasing techniques. Experiments presented in [64] showed that prior knowl-edge of probability theory, and a direct reminder of how probabilities are com-bined, are effective tools for decreasing the incidence of conjunctive fallacy,which is the hypothesized consequence of the averaging heuristic.

5.9. Confusion of the Inverse

This effect corresponds to confusing the probability of cause and effect, or,formally, confidence of an implication A → B with its inverse B → A, i.e.,Pr(B | A) is confused with Pr(A | B). This confusion may manifest itselfstrongest in the area of association rule learning, where an attribute can be ofinterest to the analyst both in the antecedent and consequent of a rule.

19

Success rates. In a study referenced from [75] this fallacy was committed by95% of physicians involved.

Implications for rule learning. Obviously, the confusion of the direction of animplication sign has its consequences on the interpretation of a rule. AlreadyMichalski [76] has noted that there are two different kinds of rules, discriminativeand characteristic. Discriminative rules can quickly discriminate an object ofone category from objects of other categories. A simple example is the rule

IF trunk THEN elephant

which states that an animal with a trunk is an elephant. This implicationprovides a simple but effective rule for recognizing elephants among all animals.

Characteristic rules, on the other hand, try to capture all properties that arecommon to the objects of the target class. A rule for characterizing elephantscould be

IF elephant THEN heavy, large, grey, bigEars, tusks, trunk.

Note that here the implication sign is reversed: we list all properties that areimplied by the target class, i.e., by an animal being an elephant. From the pointof understandability, characteristic rules are often preferable to discriminativerules. For example, in a customer profiling application, we might prefer to notonly list a few characteristics that discriminate one customer group from theother, but are interested in all characteristics of each customer group.

Characteristic rules are very much related to formal concept analysis [77, 78].Informally, a concept is defined by its intent (the description of the concept, i.e.,the conditions of its defining rule) and its extent (the instances that are coveredby these conditions). A formal concept is then a concept where the extensionand the intension are Pareto-maximal, i.e., a concept where no conditions canbe added without reducing the number of covered examples. In Michalski’sterminology, a formal concept is both discriminative and characteristic, i.e., arule where the head is equivalent to the body.

The confusion of the inverse thus seems to imply that humans will not clearlydistinguish between these types of rules, and, in particular, tend to interpretan implication as an equivalence. From this, we can infer that characteristicrules, which add all possible conditions even if they do not have additionaldiscriminative power, may be preferable to short discriminative rules.

Debiasing techniques. Edgell et al. [79] studied the influence of the effect oftraining of analysts in probabilistic theory with the conclusion that it is noteffective in addressing the confusion of the inverse fallacy. Our literature reviewdid not reveal any other applicable work.

5.10. Context and Tradeoff Contrast

Tversky and Simonson [3] developed a theory that combines backgroundcontext defined by prior options with local context which is given by the choice

20

problem at hand. The contributions of both types of context are additive. Whileadditivity is considered as not essential for the model, it is included because it“provides a good approximation in many situations and because it permits amore parsimonious representation”. The analyst adjusts the relative weights ofattributes in the light of tradeoffs implied by the background.

The reference application scenario for the tradeoff contrast is that selectionof one of the available alternatives, such as products or job candidates, canbe manipulated by the addition or deletion of alternatives that are otherwiseirrelevant. Tversky and Simonson [3] attribute the tradeoff effect to the factthat “people often do not have a global preference order and, as a result, theyuse the context to identify the most ’attractive’ option.”

Success rates. In one of the experiments described by Tversky and Simonson[3], subjects were asked to choose between two microwave ovens (Panasonicpriced 180 USD and Emerson priced 110 USD), both a third off the regularprice. The number of subjects who chose Emerson was 57% and 43% chosePanasonic. Another group of subjects was presented the same problem with thefollowing manipulation: A more expensive Panasonic valued at 200 USD (10%off the regular price) was added to the list of possible options. The newly addeddevice was described to look as inferior to the other Panasonic, but not to theEmerson device. After this manipulation, only 13% chose the more expensivePanasonic, but the number of subjects choosing the less expensive Panasonicrose from 43% to 60%.

It should be noted that according to Tversky and Simonson [3] if peoplehave well-articulated preferences, the background context has no effect on thedecision.

Implications for rule learning. In rule learning, context manipulation will typ-ically not be deliberate but a systematic result of the algorithmic process. Itwill manifest by presence of redundant rules or attributes within rules on theoutput.

The influence of context can be manifested by preference towards longerrules. The reason is that if a rule contains a literal with unknown predictivepower and multiple other literals with known (positive) predictive power forthe consequent of the rule, these known literals create a context which maymake the analyst believe that also the unknown literal has positive predictivepower. By doing so, the context provided by the longer rule can soften theeffects of ambiguity aversion, which would otherwise have made the analystprefer the shorter rule (cf. Subsection 5.7), and through the information bias(cf. Subsection 5.12) further increase the preference for the longer rule.

An attempt to making contextual attributes explicit was made by Gam-berger and Lavrac [80], who introduced supporting factors as a means for com-plementing the explanation delivered by conventional learned rules. Essentially,supporting factors are additional attributes that are not part of the learned rule,but nevertheless have very different distributions with respect to the classes of

21

the application domain. In line with the results of Kononenko [81], medical ex-perts found that these supporting factors increase the plausibility of the foundrules.

Debiasing techniques. We conjecture that similarly to other effects, the influenceof context can be suppressed by reducing the number of rules the analyst ispresented and removal of irrelevant literals from the remaining rules.

5.11. Disjunction Fallacy

The disjunction fallacy is demonstrated by assessing the probability Pr(X)to be higher than the probability Pr(Z), where Z = X∪Y is a union of event Xwith another event Y . Bar-Hillel and Neter [82] explain the disjunction fallacywith a preference for the narrower possibility over the broader one. In case thenarrower category is unlikely, the broader possibility is preferred.

In experiments reported by Bar-Hillel and Neter [82], X and Z were nestedpairs of categories, such as Brazil and Latin America. Subjects were assignedproblems such as: “Writes letter home describing a country with snowy wildmountains, clean streets, and flower decked porches. Where was the letterwritten?” It follows that since Latin America contains Brazil, the normativeanswer is Latin America. However, Brazil was the most likely answer.

Success rates. The rate of the disjunction fallacy in the experiment presentedby Bar-Hillel and Neter [82] averaged 64%. The authors offer two explanationsfor why this is a lower fallacy rate than for the conjunction fallacy. The first oneis that the disjunction rule is more compelling than the conjunction rule. Thesecond favoured explanation is that the Linda experiments in [55] used highlynon-representative categories (bank teller), while in [82] both levels of categories(Brazil and Latin America) were representative.

Implications for rule learning. In data mining context, it can be the case thatthe feature space is hierarchically ordered. The analyst can thus be confrontedwith rules containing attributes (literals) on multiple levels of granularity. Fol-lowing the disjunction fallacy, the analyst will generally prefer rules containingmore specific attributes, which can result in preference for rules with fewerbacking instances and thus weaker statistical validity.

The disjunction fallacy can be generally expected to bias the analysts to-wards longer rules since these have a higher chance of containing a literal cor-responding to a narrower category.

Debiasing techniques. We conjecture that disjunction fallacy could be alleviatedby making the analysts aware of the taxonomical relation of the individualattributes and educating them on the benefits of larger supporting sample, whichis associated with more general attributes.

22

5.12. Information Bias

Information bias relates to the tendency of people to consider more availableinformation to improve the perceived validity of a statement even if the addi-tional information is not relevant. The typical manifestation of the informationbias is evaluating questions as worth asking even when the answer cannot affectthe hypothesis that will be accepted [83].

Success rates. Baron et al. [83] performed four experiments to show the effectof information bias. For example, in their fourth experiment, subjects wereasked to assess to what degree a medical test is suitable for deciding which ofthe three diseases to treat using a scale from 0 to 100. The test detected achemical “Tutone”, which was with certain given probability associated witheach of the three diseases. This probability was varied across the cases. Therewere ten cases evaluated, the test could normatively help only in two of those(no. 2 and 9)—the correct answer for the remaining eight was thus 0. Forexample, in case no. 1 and 10 the probability of Tutone being associated withall three diseases was equal—the knowledge of Tutone presence had no valuefor distinguishing between the three diseases—and the normative answer was 0.Even for these simple cases, the mean ratings were 21 and 9 instead of 0. Thenormative answer for cases 2 and 9 was equal at 24, while the subjects assigned61 and 75 respectively.

Implications for rule learning. Rules often contain redundant, or nearly redun-dant conditions. By redundant it is meant that the knowledge of the particularpiece of information represented by the additional condition (literal) has no orvery small effect on rule quality. According to information bias, a rule contain-ing additional (redundant) literals may be preferred to a rule not containingthis literal. The information bias clearly steers the analyst towards longer rules.

Debiasing techniques. We conjecture that this bias would be alleviated by avisualization of the information value (e.g. by predictive strength) of individualconditions in the rule.

5.13. Insensitivity to Sample Size

This effect implies that analysts are unable to appreciate the increased re-liability of the confidence estimate with increasing value of support, i.e., theyfail to appreciate that the strength of the connection between antecedent andconsequence of a rule becomes more reliable with an increasing number of ob-servations. Unlike the base-rate fallacy, this effect assumes that the size of thesample is understood: while the base-rate fallacy deals with the more complexcase when people are presented with probabilistic information but are unable tounderstand it correctly, insensitivity to sample size is the related problem thatpeople underestimate the increased benefit of higher robustness of estimatesthat are made on a larger sample.

Another bias to which insensitivity to sample size is connected is the fre-quency illusion, which relates to an overestimation of the base rate of an eventas a result of selective attention and confirmation bias.

23

Success rates. When the insensitivity to sample size effect was introduced byTversky and Kahneman [17], it was supported by experimental results from theso-called hospital problem. In this problem, subjects are asked which hospitalis more likely to record more days in which more than 60 percent of the new-borns are boys. The options are a larger hospital, a smaller hospital or twohospitals with about the same size. The correct expected answer—the smallerhospital—was chosen only by 22% of subjects, the fallacy rate is thus 78%. Theexperimental subjects were 95 undergraduate students.

Implications for rule learning. If confronted with two rules, where one of themhas a slightly higher confidence and the second rule a higher support, this cog-nitive bias states that the analyst will prefer the rule with higher confidence (allother factors equal). As typically rule length trades off coverage and precision—longer rules tend to be more precise but cover fewer examples—this may resultin a preference for longer rules.

Debiasing techniques. In our opinion, one possible approach for mitigation ofthis bias in rule learning research is to use the value of support to computeconfidence (reliability) intervals for the value of confidence. Such confidenceinterval might be better understood than the original “raw” value of support.

5.14. Recognition Heuristic

Pachur et al. [68] define the recognition heuristic as follows: “For two-alternative choice tasks, where one has to decide which of two objects scoreshigher on a criterion, the heuristic can be stated as follows: If one objectis recognized, but not the other, then infer that the recognized object has ahigher value on the criterion.” For example, when asked which of the two citiesChongqing or Hongkong are bigger, subjects from the Western hemisphere tendto prefer the former because it is much better known.

The recognition heuristic can be differentiated from the availability heuris-tic as follows: “To make an inference, one version of the availability heuristicretrieves instances of the target event categories, such as the number of peo-ple one knows who have cancer compared to the number of people who havesuffered from a stroke [84]. The recognition heuristic, by contrast, bases theinference simply on the ability (or lack thereof) to recognize the names of theevent categories.” [68].

Success rates. An experiment performed by Goldstein and Gigerenzer [85] fo-cused on estimating which of two cities in a presented pair is more populated.The estimates were analysed with respect to the recognition of the cities bysubjects. The median proportion of judgments complying to the recognitionheuristic was 93%. It should be noted that the application of this heuristic isin this case ecologically justified since recognition will be related to how manytimes the city appeared in a newspaper report, which in turn is related to thecity size [86].

24

Implications for rule learning. The recognition heuristic can manifest itself bypreference for rules containing a recognized literal or attribute in the antecedentof the rule. Since the odds that a literal will be recognized increase with thelength of the rule, the recognition heuristic generally increases the preferencefor longer rules.

One could argue that for longer rules, the odds of occurrence of an unrec-ognized literal will also increase. The counterargument is the empirical findingthat – under time pressure – people assign a higher value to recognized objectsthan to unrecognized objects. This happens also in situations when recognitionis a poor cue [87].

Debiasing techniques. As to the alleviation of effects of recognition heuristic insituations where it is ecologically unsuitable, Pachur and Hertwig [87] note thatsuspension of the heuristic requires additional time or the direct knowledge ofthe “criterion variable”. This coincides with the intuition that the interpretationof rule learning results by experts should be less prone to recognition heuristic.However, in typical real-world machine learning tasks the data can include a highnumber of attributes that even subject-matter experts are not acquainted within detail. When these recognized – but not understood – attributes are presentin the rule model even the experts are liable to the recognition heuristic. Wetherefore conjecture that the experts can strongly benefit from easily accessibleinformation on the meaning of individual attributes and literals.

5.15. Negativity Bias

According to this bias, negative evidence tends to have a greater effect thanneutral or positive evidence of equal intensity.

Success rates. Extensive experimental evidence for negativity bias was summa-rized by Rozin and Royzman [88] for a range of domains. The most relevant toour focus appears to be the domain of attention and salience. In the experimentsreported by Pratto and John [89], it was investigated whether the valence of aword (desirable or undesirable trait) has effect on the time required to identifythe color in which the word appears on the screen. The result was that thesubjects took 29 ms longer to name the color of an undesirable word than fora desirable word (679 vs 650 ms). As for the number of subjects affected, for9 out of the 11 subjects the mean latency was higher for desirable words. Theauthors explain the fact that the response time was higher for undesirable wordswith the undesirable trait obtaining more attention.

Implications for rule learning. There are two types of effects that we discuss inthe following: 1) effect of a negated literal in the antecedent and 2) effect of anegative class in the consequent.

1. Most rule learning algorithms are capable of generating rules contain-ing negated literals. For example, male gender can be represented asnot(female). According to the negativity bias, the negative formulationof the same information will be given higher weight.

25

2. Considering a binary classification task, when one class is viewed as “pos-itive” and the other class as “negative”, the rule model may contain a mixof rules with the positive and negative class in the consequent. Accord-ing to the negativity bias rules with the negative class in the consequentwill be given higher weight. This bias can also manifest in the multiclasssetting, when one or more classes can be considered as “negative”. Thiseffect can manifest also in the subsequent decision making based on thediscovered and presented rules, because according to the principle of nega-tive potency [88] and prospect theory [90] people are more concerned withthe potential losses than gains.

An interesting discovery applicable to both negation in antecedent and con-sequent shows that negativity is an “attention magnet” [91, 92]. This impliesthat a rule predicting a negative class will obtain more attention than a rule pre-dicting a positive class, which may also apply to appearance of negated literalsin the antecedent. Also, research suggests that negative information is bettermemorized and subsequently recognized [93, 92].

Debiasing techniques. We conjecture that rule learning systems can mitigate theeffects of the negativity bias by avoiding the use of negation: use gender=male

instead of not(gender=female).

5.16. Primacy Effect

Once humans form initial assessment of plausibility (favourability) toward anoption, subsequent evaluations of this option will favour the initial disposition.

Success rates. Bond et al. [94] investigated to what extent changing the orderof information which is presented to a potential buyer affects the propensity tobuy. If the positive information (product description) was presented as first, thenumber of participants indicating they would buy the product was 48%. Whenthe negative information (price) was presented first, this number decreased to22%. Participants were 118 undergraduate students.

Additional experimental evidence was provided by Shteingart et al. [95].

Implications for rule learning. Following the primacy effect the analyst willfavour rules that are presented as first in the rule model. Rule learning al-gorithms, such as CBA [96], are natively capable of taking advantage of theprimacy effect, since they naturally create rule models that contain rules sortedby their strength. Others order rules so that more general rules (i.e., rules thatcover more examples) are presented first. This typically also corresponds to theorder in which rules are learned with the commonly used separate-and-conqueror covering strategy [97]. However, it has been pointed out by Webb [98] thatprepending (adding to the beginning) a new rule to the previously learned rulescan produce simpler concepts. The intuition behind this argument is that thereare often simple rules that would cover many of the positive examples, but alsocover a few negative examples that have to be excluded as exceptions to the rule.

26

Placing the simple general rule near the end of the rule list allows us to handleexceptions with rules that are placed before the general rule and keep the gen-eral rule simple. Experimental results confirmed this hypothesis with respect tothe complexity of the rules, but did not directly evaluate comprehensibility.

Debiasing techniques. A machine learning application can take advantage of theprimacy effect by presenting rules that are considered as most plausible basedon observed data as first in the resulting rule model.

5.17. Reiteration Effect

The reiteration effect describes the phenomenon that repeated statementswill become more believable [99, 68].

Success rates. The experiment performed by Hasher et al. [100] presented sub-jects with general statements and asked them to asses their validity on a 7-pointscale. Part of the statements were false and part were true. The experiment wasconducted in several sessions, where some of the statements repeated in subse-quent sessions. The average validity of repeated true statements rose betweenSession 1 and Session 3 from 4.52 to 4.80, while for non-repeated statements itdropped slightly. Similarly, for false statements, the validity rose from 4.18 to4.67 for repeated statements and dropped for non-repeated statements. In thiscase repeating of false statements increased the subjectively-perceived validityby 11%.

Implications for rule learning. In the rule learning context, “the repeated state-ment which becomes more believable” corresponds to the entire rule or possiblya “sub rule” consisting of the consequent of the rule and a subset of condi-tions in its antecedent. A typical rule learning result contains multiple rulesthat are substantially overlapping. If the analyst is exposed to multiple similarstatements, the reiteration effect will increase the analyst’s belief in the repeat-ing “sub rule”. In the rule learning context the bias behind the reiterationeffect may not be justified. Especially in the area of association rule learning,a very large set of redundant rules—covering the same, or nearly same set ofexamples—is routinely included in the output.

Debiasing techniques. A possible remedy for the reiteration effect can be per-formed already on algorithmic level by ensuring that rule learning output doesnot contain redundant rules. This can be achieved by pruning algorithms [101].We also conjecture that this effect can be alleviated by explaining the redun-dancy on rule learning output to the analyst, for example by clustering rules.

5.18. Misunderstanding of “and”

The misunderstanding of “and” is a phenomenon affecting the syntacticcomprehensibility of the logical connective “and”. As discussed by Hertwiget al. [60], “and” in natural language can express several relationships, including

27

temporal order, causal relationship, and most importantly, can also indicate acollection of sets instead of their intersection.7

Success rates. According to the two experiments reported in Hertwig et al. [60],the conjunction “bank tellers and active feminists” used in the Linda problem(cf. Section 5.3) was found by about half of the subjects as ambiguous—theyexplicitly asked the experimenter how “and” is to be understood. The exper-iment involved determining understanding of “and” based on shading of Venndiagrams. The results indicate that 45 subjects interpreted “and” as intersec-tion and 14 subjects as a union. The fallacy rate is thus 23%. Two thirds ofsubjects were university students and one third of subjects were professionals.

Implications for rule learning. This effect will increase the preference of longerrules for reasons similar to those discussed for the conjunctive fallacy (cf. Sub-section 5.3).

Debiasing techniques. According to Sides et al. [102] “and” ceases to be ambigu-ous when it is used to connect propositions rather than categories. The authorsgive the following example of a sentence which is not prone to misunderstanding:“IBM stock will rise tomorrow and Disney stock will fall tomorrow.” Similarwording of rule learning results may be, despite its verbosity, preferred. Wefurther conjecture that representations that visually express the semantics of“and” such as decision trees may be preferred over rules, which do not providesuch visual guidance.

5.19. Weak Evidence Effect

According to this effect presenting weak evidence in favour of an outcome canactually decrease the probability that a person assigns to it. In an experimentin the area of forensic science reported by Martire et al. [103], it was shown thatparticipants presented with evidence weakly supporting guilt tended to “invert”the evidence, thereby counterintuitively reducing their belief in the guilt of theaccused.

Success rates. Martire et al. [103] performed an experiment in the judicial do-main. When the presented evidence provided by the expert was weak, butpositive, the number of responses incongruent with the evidence provided was62%. When the strength of evidence was moderate or high the correspondingaverage was 13%. The subjects were undergraduate psychology students andAmazon Mechanical Turk workers (altogether over 600 participants).

7As in “He invited friends and colleagues to the party”

28

Implications for rule learning. The weak evidence effect can be directly appliedon rules: the evidence is represented by rule antecedent; the consequent corre-sponds to the outcome. The analyst can intuitively interpret each of the condi-tions in the antecedent as a piece of evidence in favour of the outcome. Typicalof many machine learning problems is the uneven contribution of individualattributes to the prediction. Let us assume that the analyst is aware of theprediction strength of the individual attributes. If the analyst is to choose froma shorter rule containing only the strong predictor and a longer rule contain-ing a strong predictor and a weak (weak enough to trigger this effect) predictor,according to the weak evidence effect the analyst should choose the shorter rule.

Debiasing techniques. Our review did not reveal any debiasing strategies. Thisis related to the fact that the weak evidence effect is a relatively recent discovery.Our conjecture is that this effect can be alleviated by intentional omission ofweak predictors from rules either directly by the rule learner or as part of featureselection.

5.20. Unit Bias

This cognitive bias manifests by humans tending to consider each conditionas a unit of equal weight at the expense of detailed scrutiny of the actual effectof the condition [104].

Success rates. The effect of this bias was evaluated by Geier et al. [104] onthree food items: Tootsie Rolls, pretzels and M&Ms. These food items wereoffered in two sizes/scoops (on different days) and it was observed how this willaffect consumption. For Tootsie Rolls and M&Ms the larger unit size was 4×the smaller one and for pretzels 2× the smaller one. It follows from the figureincluded in [104] that increasing the size of the unit had about 50% effect onthe amount consumed.

Implications for rule learning. From a technical perspective, the number ofconditions (literals) in rules is not important. What matters is the actual dis-criminatory power of the individual conditions, which can vary substantially.However, following the application of unit bias, the number of conditions affectsthe subjective perception of discriminatory power of the antecedent as a whole.Under the assumption that the analyst will favour rule with higher discrimina-tory power, this heuristic will clearly contribute to preference for longer rules,since these contain more literals considered as “units”.

Unlike other modes of communication humans are used to, rules resultingfrom algorithmic analysis of data do not provide clues relating to the importanceof individual conditions, since rules often place conditions of vastly differentimportance side by side, not even maintaining the order from the most importantto the least important. Such computer-generated rules violate conversational

29

rules or “maxims”, because they contain conditions which are not informativeor relevant.8

In summary, the application of the unit bias in the context of rule learn-ing can result in gross errors in interpretation. When domain knowledge onthe meaning of the literals in the rule is absent, the unit bias can manifestparticularly strongly.

Debiasing techniques. We conjecture that informing analysts about the dis-criminatory power of the individual conditions (literals) may alleviate unit bias.Such indicator can possibly be generated automatically by listing the numberof instances in the entire dataset that meet the condition. Second, rule learningalgorithms should ensure that literals are present in the rules in the order ofsignificance, complying to human conversational maxims.

6. A Model for Rule Plausibility and Recommendations for Inter-pretable Rule Learning

Based on literature review and partly on experimental results presented in[35], we propose a graphical model of the plausibility of rules. The model isintended to raise awareness about the effect of cognitive biases on perceptionof rule learning results among the designers of machine learning algorithms andsoftware. It suggests which cognitive biases might be triggered when humansassess plausibility of inductively learned rules and whether they influence plau-sibility. To some extent similar model describing general factors influencingplausibility assessment of a hypothesis was proposed in Gettys et al. [32, 33].In our model, we focus on inductively learnt rule and cognitive biases.

8The relevance maxim is one of four conversation maxims proposed by philosopher PaulGrice, which was brought to relation with the conjunctive fallacy in the work of Gigerenzerand Hoffrage [105] (see also [106]).

30

(a) Contribution of individual literals(b) Aggregation of literal contribu-tions

Figure 3: Proposed model of causality between cognitive biases and human-perceived plausi-bility of inductively learned rules. “+” means increase, “-” decrease of subjectively-perceivedplausibility. Some leaf nodes are associated with hypothesized list of effective biases andheuristics (in bold). Additional explanation of numbered nodes is in [35].

31

The model consists of two decision trees, which are presented in Figure 3.The first tree captures the hypothesized contributions of individual literals inthe antecedent of the rule towards increase or decrease of human perceivedplausibility of the rule. The second tree suggests how the individual literalcontributions might be combined into perception of overall plausibility of therule.

6.1. Categorization of Biases based on Agency

Inspection of the first tree (Figure 3a) hints that the effect of the individualliterals in the rule largely depends on the domain knowledge of the personinspecting the model. In contrast, the way the contribution of the literals isaggregated into a final plausibility score in Figure 3b seems to depend on thegeneral information processing style and preferences of the person. We thuspropose to divide the biases reviewed in this paper into the following two groups:

• Triggered by domain knowledge related to attributes and values in therules. An example is aversion to ambiguous information.

• Generic strategies applied when evaluating alternatives. An example is in-sensitivity to sample size, which implies that rule confidence is consideredas more important than rule support.

While domain knowledge may be difficult to change, systematic errors inreasoning can often be avoided. One example is making the person aware ofthe fact that low rule support influences the reliability of the rule confidenceestimate.

6.2. Implications for Algorithm Design and Visualizations in Machine Learning

This section provides a concise list of considerations that is aimed to raiseawareness among machine learning practitioners regarding availability of mea-sures that could potentially suppress effect of cognitive biases on comprehensionof rule-based models. We expect part of the list to be useful also for other sym-bolic machine learning models, such as decision trees.

1. Remove near-redundant rules and near-redundant literals from rules. Rulemodels often incorporate output that is considered as marginally relevant.This can take form of (near) redundant rules or (near) redundant literalsin the rule. Our analysis shows that these redundancies can induce anumber of biases. For example, a frequently occurring but otherwise notvery important literal can – by the virtue of the mere exposure effect – beperceived as more important than would be appropriate given the data.

2. Represent rule quality measures as frequencies not ratios. Currently, ruleinterest measures such as confidence and support are typically representedas ratios. Extensive research has shown that natural frequencies are betterunderstood.

32

3. Make conjunctions unambiguous. There are several cognitive studies indi-cating “and” is often misunderstood. The results of our experiments alsosupport this conclusion. Machine learning software should thus make surethat the meaning of and in presented rules is clear.

4. Present confidence interval for rule confidence. The tendency of humansto ignore base-rates and sample sizes (which closely relate to rule support)is a well-established fact in cognitive science, results of our experimentson inductively learned rules also provide evidence for this conclusion. Ourproposition is that this effect can be addressed by computing confidence(reliability) intervals for confidence. In this way, the “weight of evidence”will effectively be communicated through confidence.

5. Avoid the use of negated literals as well as positive/negative class labels.It is an established fact in cognitive science that negative informationreceives more attention and is associated with higher weight than positiveinformation. There is research indicating that recasting a yes/no attributeto two “neutral” categories (such as “DAX/MED”) can improve humanunderstanding.

6. Sort rules as well as literals in the rules from strongest to weakest. Peoplehave the tendency to put higher emphasis to information they are exposedto first. By presenting the important information as first, machine learn-ing software can also conform to these human conversational maxims. Theoutput could also visually delimit literals in the rules based on their signif-icance, which would again correspond to humans using various non-verbalclues to convey significance in the spoken word.

7. Provide explanation for literals in rules. Number of biases can be triggeredor strengthened by the lack of domain knowledge of literals in the rules.Some examples include ambiguity aversion or unit bias. Providing theanalyst with easily accessible information on literals in the rules includingtheir predictive power can prove as an effective debiasing technique.

8. Explain difference between negation and absence of a condition. Prior re-sults in cognitive science as well as some experimental results in the rulelearning domain [35] show that absence of a condition can be misinter-preted as negation if the omitted condition is present in in other rules.Consider the following pair of rules

Rule 1: IF bankteller=yes THEN class=A

Rule 2: IF bankteller=yes AND feminist=yes then class=B.

In presence of Rule 2, Rule 1 can be read as

Rule 1’: IF bankteller=yes AND feminist=no THEN class=A.

9. Elicit and respect monotonicity constraints. Research has shown that ifmonotonicity constraints—such as that fuel consumption increases with

33

increasing car weight—are observed, the plausibility of the rule modelincreases.

10. Educate and assess human analysts. One perhaps surprising result relatedto confirmation or myside bias is that its incidence is not related to intel-ligence. Some research even suggests that analysts, who think that goodarguments are those that can be proved by facts, are even more suscepti-ble to myside bias than the general population. There is a psychologicaltest that can reveal the susceptibility of a person to myside bias. Severalstudies have shown that providing explicit guidance and education on for-mal logics, hypothesis testing and critical assessment of information canreduce fallacy rates in some tasks.

7. Limitations and Future Work

Our goal was to validate whether cognitive biases affect interpretation ofmachine learning models and propose remedies if they do. Since this field isuntapped from the machine learning perspective, we tried to approach thisproblem holistically. Our work yielded a number of partial contributions, ratherthan a single profound result. We mapped applicable cognitive biases, identifiedprior works on their suppression and proposed how these could be transferedto machine learning. All the shortcomings of human judgment pertaining tointerpretation of inductively learned rules that we have reviewed are based onempirical cognitive science research. For each cognitive bias, we attemptedto provide a justification how it would relate to machine learning. Due toabsence of applicable prior research, this justification is mostly based on authors’experience in machine learning.

7.1. Incorporating Additional Biases

There are about 24 cognitive biases covered in Cognitive Illusions, the au-thoritative overview of cognitive biases by Pohl [1], and even 51 different biasesare covered by Evans et al. [42]. While doing the initial selection of cognitivebiases to study, we tried to identify those most salient for machine learningresearch matching our criteria. This is the reason why we included the weakevidence effect, which has been discovered only recently and is not yet includedinto the latest edition of Cognitive Illusions. In the end, our review focused ona selection of 20 cognitive biases (effects, illusions). Future work might focuson expanding the review with additional relevant biases, such as labelling andovershadowing effects [1].

7.2. Applicability of Results on Wason’s 2-4-6 Problem

According to our review, the results obtained in cognitive science have onlyrarely been integrated or aligned with research done in machine learning. Asour review also showed, there is a number results in cognitive science relevantfor machine learning. Remarkably, since 1960 there is a consistent line of work

34

done by psychologists on the problem of studying cognitive processes related torule induction, which is centred around the so called Wason’s 2-4-6 problem.

Cognitive science research on rule induction in humans has been so far com-pletely unnoticed in the rule learning subfield of machine learning.9 It wasout of the scope of the objectives of this review to perform analysis of the sig-nificance of results obtained for the Wason’s 2-4-6 problem for rule learning,nevertheless we believe that such investigation could bring interesting insightsfor cognitively-inspired design of rule learning algorithms.

8. Conclusion

To our knowledge, cognitive biases have not yet been discussed in relationto interpretability of machine learning results. We thus initiated this review ofresearch published in cognitive science with the intent to give a psychologicalbasis to changes in inductive rule learning algorithms, and the way their resultsare communicated. Our review identified twenty cognitive biases, heuristics andeffects that can give rise to systematic errors when inductively learned rules areinterpreted.

For most biases and heuristics involved in our study, psychologists have pro-posed “debiasing” measures. Application of prior empirical results obtained incognitive science allowed us to propose several methods that could be effec-tive in suppressing these cognitive phenomena when machine learning modelsare interpreted. While each cognitive bias requires a different “antidote” twonoticeable trends emerged from our analysis.

Our first finding indirectly supports the previous view of interpretability ofmachine learning models, which is that smaller models are better interpretable.However, in our review of literature from cognitive science, we did not identifyresults that would support this view. What our analysis did reveal is a numberof cognitive phenomena that would make longer rules (or generally descriptions)more likely to trigger various cognitive biases than would shorter rules (descrip-tions). An example of such bias is the information bias, i.e., the preference formore information even if it does not help to address the problem at hand. Tosummarize our contribution, we found indirect support in psychology for the“smaller is better” paradigm used in many machine learning algorithms. Whilesmall models may not necessarily be found to be more plausible by humansthan larger models, they provide less opportunities for cognitive biases to betriggered, leading to better, more truthful, comprehension.

Our second observation is that there are two categories of cognitive biases.Those that are associated by individual conditions in rules mostly relate todomain knowledge relating to attributes and values in the rules. An example isaversion to ambiguous information. Second, there are generic objectively validreasoning rules, such as the Bayes theorem. Instead of this rule, a heuristicmay be applied, leading to distortion. An example of such cognitive bias is

9Based on our analysis of cited reference search in Google Scholar for [21].

35

insensitivity to sample size. The choice of debiasing technique depends on thecategory.

Overall, in our review we processed only a fraction of potentially relevantpsychological studies of cognitive biases, however, we were unable to locate asingle study focused on machine learning. Future research should thus focuson empirical evaluation of effects of cognitive biases in the machine learningdomain.

Acknowledgments

TK was supported by long term institutional support of research activitiesby Faculty of Informatics and Statistics, University of Economics, Prague.

References

References

[1] R. Pohl, Cognitive illusions: A handbook on fallacies and biases in think-ing, judgement and memory, Psychology Press, 2017. 2nd ed.

[2] D. Kahneman, A. Tversky, Subjective probability: A judgment of repre-sentativeness, Cognitive psychology 3 (1972) 430–454.

[3] A. Tversky, I. Simonson, Context-dependent preference, Managementscience 39 (1993) 1179–1189.

[4] Z. Kunda, Social cognition: Making sense of people, MIT press, 1999.

[5] S. Serfas, Cognitive biases in the capital investment context, in: CognitiveBiases in the Capital Investment Context, Springer, 2011, pp. 95–189.

[6] D. Kahneman, A. Tversky, On the psychology of prediction, PsychologicalReview 80 (1973) 237 – 251.

[7] R. S. Michalski, On the quasi-minimal solution of the general coveringproblem, in: Proceedings of the V International Symposium on Informa-tion Processing (FCIP 69)(Switching Circuits), Yugoslavia, Bled, 1969,pp. 125–128.

[8] R. S. Michalski, A theory and methodology of inductive learning, in:Machine learning, Springer, 1983, pp. 83–134.

[9] J. Furnkranz, T. Kliegr, H. Paulheim, On cognitive preferences and theinterpretability of rule-based models, CoRR abs/1803.01316 (2018).

[10] J. Furnkranz, D. Gamberger, N. Lavrac, Foundations of Rule Learning,Springer-Verlag, 2012.

36

[11] R. Slowinski, I. Brzezinska, S. Greco, Application of bayesian confirmationmeasures for mining rules from support-confidence pareto-optimal set, Ar-tificial Intelligence and Soft Computing–ICAISC 2006 (2006) 1018–1026.

[12] R. Agrawal, H. Mannila, R. Srikant, H. Toivonen, A. I. Verkamo, Fastdiscovery of association rules, in: U. M. Fayyad, G. Piatetsky-Shapiro,P. Smyth, R. Uthurusamy (Eds.), Advances in Knowledge Discovery andData Mining, AAAI Press, 1995, pp. 307–328.

[13] C. Zhang, S. Zhang, Association Rule Mining: Models and Algorithms,Springer-Verlag, 2002.

[14] E. E. Smith, C. Langston, R. E. Nisbett, The case for rules in reasoning,Cognitive science 16 (1992) 1–40.

[15] R. E. Nisbett, Rules for reasoning, Psychology Press, 1993.

[16] S. Pinker, Words and rules: The ingredients of language, Basic Books,2015.

[17] A. Tversky, D. Kahneman, Judgment under uncertainty: Heuristics andbiases, Science 185 (1974) 1124–1131.

[18] D. Griffin, A. Tversky, The weighing of evidence and the determinants ofconfidence, Cognitive psychology 24 (1992) 411–435.

[19] J. M. Keynes, A Treatise on Probability, Macmillan & Co, 1922.

[20] C. Camerer, M. Weber, Recent developments in modeling preferences:Uncertainty and ambiguity, Journal of Risk and Uncertainty 5 (1992)325–370.

[21] P. C. Wason, On the failure to eliminate hypotheses in a conceptual task,Quarterly journal of experimental psychology 12 (1960) 129–140.

[22] R. S. Nickerson, Confirmation bias: A ubiquitous phenomenon in manyguises, Review of general psychology 2 (1998) 175.

[23] R. D. Tweney, M. E. Doherty, W. J. Worner, D. B. Pliske, C. R. Mynatt,K. A. Gross, D. L. Arkkelin, Strategies of rule discovery in an inferencetask, Quarterly Journal of Experimental Psychology 32 (1980) 109–123.

[24] S. Rossi, J. P. Caverni, V. Girotto, Hypothesis testing in a rule discoveryproblem: When a focused procedure is effective, The Quarterly Journalof Experimental Psychology: Section A 54 (2001) 263–267.

[25] F. Vallee-Tourangeau, T. Payton, Goal-driven hypothesis testing in a rulediscovery task, in: Proceedings of the 30th Annual Conference of theCognitive Science Society, Cognitive Science Society Austin, TX, 2008,pp. 2122–2127.

37

[26] A. Gopnik, J. B. Tenenbaum, Bayesian networks, Bayesian learning andcognitive development, Developmental science 10 (2007) 281–287.

[27] T. L. Griffiths, N. Chater, C. Kemp, A. Perfors, J. B. Tenenbaum, Prob-abilistic models of cognition: Exploring representations and inductive bi-ases, Trends in cognitive sciences 14 (2010) 357–364.

[28] A. Wilke, R. Mata, Cognitive bias, in: V. Ramachandran (Ed.),Encyclopedia of Human Behavior (Second Edition), second edition ed.,Academic Press, San Diego, 2012, pp. 531 – 535. URL: https://www.

sciencedirect.com/science/article/pii/B978012375000600094X.doi:https://doi.org/10.1016/B978-0-12-375000-6.00094-X.

[29] M. G. Haselton, D. Nettle, The paranoid optimist: An integrative evo-lutionary model of cognitive biases, Personality and social psychologyReview 10 (2006) 47–66.

[30] A. A. Freitas, Comprehensible classification models: a position paper,ACM SIGKDD explorations newsletter 15 (2014) 1–10.

[31] J. Stecher, F. Janssen, J. Furnkranz, Shorter rules are better, aren’t they?,in: Proceedings of the 19th International Conference on Discovery Sci-ence (DS-16), Bari, Italy, 2016, pp. 279–294. URL: https://doi.org/10.1007/978-3-319-46307-0_18. doi:10.1007/978-3-319-46307-0_18.

[32] C. F. Gettys, S. D. Fisher, T. Mehle, Hypothesis Generation and Plausi-bility Assessment, Technical Report, Decision Processes Laboratory, Uni-versity of Oklahoma, Norman, 1978. Annual report TR 15-10-78 (ADA060786.

[33] C. F. Gettys, T. Mehle, S. Fisher, Plausibility assessments in hypothesisgeneration, Organizational Behavior and Human Decision Processes 37(1986) 14–33.

[34] J. Anderson, D. Fleming, Analytical procedures decision aids for gener-ating explanations: Current state of theoretical development and impli-cations of their use, Journal of Accounting and Taxation 8 (2016) 51.

[35] T. Kliegr, Effect of Cognitive Biases on Human Understanding of Rule-based Machine Learning, Queen Mary University London, London, UnitedKingdom, 2017. Dissertation Thesis.

[36] A. Bibal, B. Frenay, Interpretability of machine learning models andrepresentations: an introduction, in: Proceedings of the 24th EuropeanSymposium on Artificial Neural Networks (ESANN), 2016, pp. 77–82.

[37] D. Martens, J. Vanthienen, W. Verbeke, B. Baesens, Performance ofclassification models from a user perspective, Decision Support Systems51 (2011) 782–793.

38

[38] P. J. Azevedo, A. M. Jorge, Comparing rule measures for predictive asso-ciation rules, in: Ecml, volume 7, Springer, 2007, pp. 510–517.

[39] A. Tversky, D. Kahneman, Belief in the law of small numbers, Psycho-logical bulletin 76 (1971) 105.

[40] G. Gigerenzer, D. G. Goldstein, Reasoning the fast and frugal way: modelsof bounded rationality, Psychological review 103 (1996) 650.

[41] G. Gigerenzer, U. Hoffrage, How to improve Bayesian reasoning withoutinstruction: frequency formats, Psychological review 102 (1995) 684.

[42] J. S. B. Evans, et al., Hypothetical thinking: Dual processes in reasoningand judgement, volume 3, Psychology Press, 2007.

[43] J. Furnkranz, P. A. Flach, Roc nrule learningtowards a better under-standing of covering algorithms, Machine Learning 58 (2005) 39–77.

[44] M. Ballin, R. Carbini, M. F. Loporcaro, M. Lori, R. Moro, V. Olivieri,M. Scanu, The use of information from experts for agricultural offi-cial statistics, in: European Conference on Quality in Official Statistics(Q2008), 2008.

[45] J. S. B. Evans, Bias in human reasoning: Causes and consequences,Lawrence Erlbaum Associates, Inc, 1989.

[46] Y. Trope, B. Gervey, N. Liberman, Wishful thinking from a pragmatichypothesis-testing perspective, The mythomanias: The nature of decep-tion and self-deception (1997) 105–31.

[47] J. Klayman, Y.-W. Ha, Confirmation, disconfirmation, and informationin hypothesis testing, Psychological review 94 (1987) 211.

[48] R. Pohl, Cognitive illusions: A handbook on fallacies and biases in think-ing, judgement and memory, Psychology Press, 2004.

[49] C. R. Mynatt, M. E. Doherty, R. D. Tweney, Confirmation bias in a simu-lated research environment: An experimental study of scientific inference,The quarterly journal of experimental psychology 29 (1977) 85–95.

[50] D. Westen, P. S. Blagov, K. Harenski, C. Kilts, S. Hamann, Neural basesof motivated reasoning: An fMRI study of emotional constraints on par-tisan political judgment in the 2004 US presidential election, Journal ofcognitive neuroscience 18 (2006) 1947–1958.

[51] K. E. Stanovich, R. F. West, M. E. Toplak, Myside bias, rational thinking,and intelligence, Current Directions in Psychological Science 22 (2013)259–264.

[52] C. R. Wolfe, M. A. Britt, The locus of the myside bias in written argu-mentation, Thinking & Reasoning 14 (2008) 1–27.

39

[53] D. Albarracın, A. L. Mitchell, The role of defensive confidence in prefer-ence for proattitudinal information: How believing that one is strong cansometimes be a defensive weakness, Personality and Social PsychologyBulletin 30 (2004) 1565–1584.

[54] T. Gilovich, K. Savitsky, Like goes with like: The role of representativenessin erroneous and pseudo-scientific beliefs, Cambridge University Press,2002.

[55] A. Tversky, D. Kahneman, Extensional versus intuitive reasoning: theconjunction fallacy in probability judgment, Psychological review 90(1983) 293.

[56] G. Gigerenzer, On narrow norms and vague heuristics: A reply to Kah-neman and Tversky, Psychological Review (1996) 592–596.

[57] D. Kahneman, A perspective on judgment and choice, American Psychol-ogist 58 (2003).

[58] M. Bar-Hillel, Commentary on Wolford, Taylor, and Beck: The conjunc-tion fallacy?, Memory & cognition 19 (1991) 412–414.

[59] K. Tentori, V. Crupi, On the conjunction fallacy and the meaning of and,yet again: A reply to Hertwig, Benz, and Krauss (2008), Cognition 122(2012) 123–134.

[60] R. Hertwig, B. Benz, S. Krauss, The conjunction fallacy and the manymeanings of and, Cognition 108 (2008) 740–753.

[61] E. Fantino, J. Kulik, S. Stolarz-Fantino, W. Wright, The conjunctionfallacy: A test of averaging hypotheses, Psychonomic Bulletin & Review4 (1997) 96–101.

[62] R. Hertwig, G. Gigerenzer, The ”conjunction fallacy” revisited: Howintelligent inferences look like reasoning errors, Journal of BehavioralDecision Making 12 (1999) 275–305.

[63] G. Charness, E. Karni, D. Levin, On the conjunction fallacy in probabil-ity judgment: New experimental evidence regarding Linda, Games andEconomic Behavior 68 (2010) 551 – 556.

[64] D. J. Zizzo, S. Stolarz-Fantino, J. Wen, E. Fantino, A violation of themonotonicity axiom: Experimental evidence on the conjunction fallacy,Journal of Economic Behavior & Organization 41 (2000) 263–276.

[65] S. Stolarz-Fantino, E. Fantino, J. Kulik, The conjunction fallacy: Differen-tial incidence as a function of descriptive frames and educational context,Contemporary Educational Psychology 21 (1996) 208–218.

[66] A. Tversky, D. Kahneman, Availability: A heuristic for judging frequencyand probability, Cognitive psychology 5 (1973) 207–232.

40

[67] N. Schwarz, H. Bless, F. Strack, G. Klumpp, H. Rittenauer-Schatka, A. Si-mons, Ease of retrieval as information: Another look at the availabilityheuristic, Journal of Personality and Social psychology 61 (1991) 195.

[68] T. Pachur, P. M. Todd, G. Gigerenzer, L. Schooler, D. G. Goldstein, Therecognition heuristic: A review of theory and tests, Frontiers in psychology2 (2011) 147.

[69] J. L. Monahan, S. T. Murphy, R. B. Zajonc, Subliminal mere exposure:Specific, general, and diffuse effects, Psychological Science 11 (2000) 462–466.

[70] R. F. Bornstein, Exposure and affect: overview and meta-analysis ofresearch, 1968–1987, Psychological Bulletin 2 (1989) 265–289.

[71] R. B. Zajonc, Attitudinal effects of mere exposure, Journal of personalityand social psychology 9 (1968) 1.

[72] N. I. Al-Najjar, J. Weinstein, The ambiguity aversion literature: a criticalassessment, Economics and Philosophy 25 (2009) 249–284.

[73] D. Ellsberg, Risk, ambiguity, and the Savage axioms, The QuarterlyJournal of Economics 75 (1961) 643–669.

[74] S. P. Curley, S. A. Eraker, J. F. Yates, An investigation of patient’sreactions to therapeutic uncertainty, Medical Decision Making 4 (1984)501–511.

[75] S. Plous, The psychology of judgment and decision making, McGraw-HillBook Company, 1993.

[76] R. S. Michalski, A theory and methodology of inductive learning, ArtificialIntelligence 20 (1983) 111–162.

[77] R. Wille, Restructuring lattice theory: An approach based on hierarchiesof concepts, in: I. Rival (Ed.), Ordered Sets, Reidel, Dordrecht-Boston,1982, pp. 445–470.

[78] B. Ganter, R. Wille, Formal Concept Analysis – Mathematical Founda-tions, Springer, 1999.

[79] S. E. Edgell, J. Harbison, W. P. Neace, I. D. Nahinsky, A. S. Lajoie,What is learned from experience in a probabilistic environment?, Journalof Behavioral Decision Making 17 (2004) 213–229.

[80] D. Gamberger, N. Lavrac, Active subgroup mining: A case study in coro-nary heart disease risk group detection, Artificial Intelligence in Medicine28 (2003) 27–57.

[81] I. Kononenko, Inductive and Bayesian learning in medical diagnosis, Ap-plied Artificial Intelligence 7 (1993) 317–337.

41

[82] M. Bar-Hillel, E. Neter, How alike is it versus how likely is it: A disjunc-tion fallacy in probability judgments, Journal of Personality and SocialPsychology 65 (1993) 1119.

[83] J. Baron, J. Beattie, J. C. Hershey, Heuristics and biases in diagnos-tic reasoning: II. congruence, information, and certainty, OrganizationalBehavior and Human Decision Processes 42 (1988) 88–110.

[84] R. Hertwig, T. Pachur, S. Kurzenhauser, Judgments of risk frequencies:tests of possible cognitive mechanisms, Journal of Experimental Psychol-ogy: Learning, Memory, and Cognition 31 (2005) 621.

[85] D. G. Goldstein, G. Gigerenzer, The recognition heuristic: How igno-rance makes us smart, in: Simple heuristics that make us smart, OxfordUniversity Press, 1999, pp. 37–58.

[86] C. P. Beaman, R. McCloy, P. T. Smith, When does ignorance make ussmart? additional factors guiding heuristic inference, in: Proceedings ofthe Cognitive Science Society, volume 28, 2006.

[87] T. Pachur, R. Hertwig, On the psychology of the recognition heuristic:Retrieval primacy as a key determinant of its use, Journal of ExperimentalPsychology: Learning, Memory, and Cognition 32 (2006) 983.

[88] P. Rozin, E. B. Royzman, Negativity bias, negativity dominance, andcontagion, Personality and social psychology review 5 (2001) 296–320.

[89] F. Pratto, O. P. John, Automatic vigilance: The attention-grabbing powerof negative social information, Social cognition: key readings 250 (2005).

[90] D. Kahneman, A. Tversky, Prospect theory: An analysis of decision underrisk, Econometrica: Journal of the econometric society (1979) 263–291.

[91] S. T. Fiske, Attention and weight in person perception: The impact ofnegative and extreme behavior, Journal of personality and Social Psy-chology 38 (1980) 889.

[92] H. Ohira, W. M. Winton, M. Oyama, Effects of stimulus valence on recog-nition memory and endogenous eyeblinks: Further evidence for positive-negative asymmetry, Personality and Social Psychology Bulletin 24 (1998)986–993.

[93] G. L. Robinson-Riegler, W. M. Winton, The role of conscious recollec-tion in recognition of affective material: Evidence for positive-negativeasymmetry, The Journal of General Psychology 123 (1996) 93–104.

[94] S. D. Bond, K. A. Carlson, M. G. Meloy, J. E. Russo, R. J. Tanner,Information distortion in the evaluation of a single option, OrganizationalBehavior and Human Decision Processes 102 (2007) 240–254.

42

[95] H. Shteingart, T. Neiman, Y. Loewenstein, The role of first impressionin operant learning, Journal of Experimental Psychology: General 142(2013) 476.

[96] B. Liu, W. Hsu, Y. Ma, Integrating classification and association rule min-ing, in: Proceedings of the Fourth International Conference on KnowledgeDiscovery and Data Mining, KDD’98, AAAI Press, 1998, pp. 80–86.

[97] J. Furnkranz, Separate-and-conquer rule learning, Artificial IntelligenceReview 13 (1999) 3–54.

[98] G. I. Webb, Recent progress in learning decision lists by prependinginferred rules, in: Proceedings of the 2nd Singapore International Confer-ence on Intelligent Systems, 1994, pp. B280–B285.

[99] R. Hertwig, G. Gigerenzer, U. Hoffrage, The reiteration effect in hindsightbias, Psychological Review 104 (1997) 194.

[100] L. Hasher, D. Goldstein, T. Toppino, Frequency and the conference ofreferential validity, Journal of Verbal Learning and Verbal Behavior 16(1977) 107–112.

[101] J. Furnkranz, Pruning algorithms for rule learning, Machine Learning 27(1997) 139–172.

[102] A. Sides, D. Osherson, N. Bonini, R. Viale, On the reality of the conjunc-tion fallacy, Memory & Cognition 30 (2002) 191–198.

[103] K. A. Martire, R. I. Kemp, I. Watkins, M. A. Sayle, B. R. Newell, Theexpression and interpretation of uncertain forensic science evidence: ver-bal equivalence, evidence strength, and the weak evidence effect, Law andhuman behavior 37 (2013) 197.

[104] A. B. Geier, P. Rozin, G. Doros, Unit bias a new heuristic that helpsexplain the effect of portion size on food intake, Psychological Science 17(2006) 521–525.

[105] G. Gigerenzer, U. Hoffrage, Overcoming difficulties in Bayesian reasoning:A reply to Lewis and Keren (1999) and Mellers and McGraw (1999).,Psychological Review (1999) 425–430.

[106] G. Mosconi, L. Macchi, The role of pragmatic rules in the conjunctionfallacy, Mind & Society 2 (2001) 31–57.

43