Supervised learning, categorization and signal-detection ... · categorization and signal-detection...
Transcript of Supervised learning, categorization and signal-detection ... · categorization and signal-detection...
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
Supervised learning, categorization andsignal-detection theory
Pantelis P. Analytis
February 28, 2018
1 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
1 Introduction
2 The linear world
3 Instance-based learning
4 Decision trees
5 Signal detection theory
2 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
What is supervised learning?
Classification—categorization problems
Regression—estimation problems
3 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
The lens model: a generic framework
developed by Egon Brunswik in the 30s
Has been adapted and interpreted in different ways bypsychologists.
4 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
The linear lens
Hammond (1955), Todd (1954)
Linear model: y = x0 + x1 · b1 + x2 · b2.... + xn · bn
5 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
Actuarial vs. clinical judgement
First comparison of human and machine judgement byPaul Meehl in 1954.
Comparison on the same prediction problem, but possiblyusing different past training data. Use of cross-validationto evaluate the models.
Linear models or even simpler algorithms consistentlyoutperform humans.
6 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
Paramorphic representation of judgement
Hoffman (1960) used the linear model to develop the firstmethod of preference learning, precursor to conjointanalysis.
7 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
Paramorphic representation of judgement
Hoffman (1960) used the linear model to develop the firstmethod of preference learning, precursor to conjointanalysis.
8 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
Logistic regression: simple but robust
b0 shifts the position of the curve and b1 (or b1 + ...bn)defines its slope.
Logistic regression can deal with any number of variables
Computationally very cheap. It is used heavily in theindustry although there are many better algorithms.
9 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
Instance-based learning—exemplars and prototypes
First appearance in Statistics: Fix and Hodges (1951) andCover and Hart (1967).First appearance of prototypes in Psychology in 1954 in apaper by Fred Attneave. 10 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
K-nearest neighbors
Define a similarity vector:√∑k
i=1 (xi − yi )2
Pros and cons: intuitive and easy to implement butcomputationally very expensive.
Variations of the basic approach: assign weightsdepending on distance, discard some instances.
11 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
K-nearest neighbors
Define a similarity vector:√∑k
i=1 (xi − yi )2
Pros and cons: intuitive and easy to implement butcomputationally very expensive.
Variations of the basic approach: assign weightsdepending on distance, discard some instances.
12 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
Shepard’s Law
Humans and other animals generalize on the basis offeature distance from previously seen items.
Shepard suggested that if there is any law in psychology,that should be it (Science, 1987).
13 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
Discussion: do we have access to our ownalgorithms?
Most of the decisions we make are intuitive. What doesexactly intuition mean?
Hogarth (2001) has dedicated a book on the topic.
14 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
Medical decision trees
Super (1984), Super triage and rapid treatment (STARTtree).
15 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
Bailing out decisions
A model of how British judges decide whether to make apunitive bail (Dahmi, 2003).
16 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
First decision trees
Morgan and Sonquist, 1963, formulated the firstregression tree in Statistics.
Hunt et al. 1966, concept learning system.
17 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
CART, ID3 and C.4.5 algorithm
Friedman (1977), Breiman et al. (1984).
Quinlan (1979,1983,1986,1993).
18 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
Shannon’s entropy and information gain
Entropy =∑n
i=1−pi · log2 pi
G (S ,A) = Entropy(S) −∑
v∈Values(A)|Sv ||S | · Entropy(Sv )
19 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
Pros and cons of decision trees
Pros: Comprehensible and easy to explain, inexpensiveonce trained, work with continuous and categoricalvariables, can capture non-linearities.
Cons: Weaker in estimation tasks, prone to overfittingespecially with small samples (pruning can somewhatcounteract that), can be costly to train.
20 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
Random forests to the rescue
Trained on different samples of data and then aggregatedwith majoritarian voting.
Random forests curb the variance of the algorithm,reducing the overall error.
21 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
Signal detection theory
Origins in Neyman-Pearson hypothesis testing, firstapplications in the use of radars.
Introduced in psychology to study perception andsensation (1954).
22 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory Criterion: the internal response level beyond which a
decision-maker responds yes (radar operator, doctor orsearch engine)
d’ = separation/spread
23 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
Broadly used paradigm in psychophysics.
Also widely employed to define measure of performance ininformation retrieval and recommender systems.
24 / 25
Supervisedlearning,
categorizationand signal-detectiontheory
Pantelis P.Analytis
Introduction
The linearworld
Instance-basedlearning
Decision trees
Signaldetectiontheory
Signal detection theory
Luan et al. (2011) draw connections between the fast andfrugal tree framework and SDT.
25 / 25