MORE CLASSIFIERS

AGENDA Key concepts for all classifiers

Precision vs recall Biased sample sets

Linear classifiers Intro to neural networks

RECAP: DECISION BOUNDARIES With continuous attributes, a decision

boundary is the surface in example space that splits positive from negative examples

x1>=20 x2

x1F

x2>=10

T

F

F

T

x2>=15

T F

T

4

BEYOND ERROR RATES

BEYOND ERROR RATE Predicting security risk

Predicting “low risk” for a terrorist, is far worse than predicting “high risk” for an innocent bystander (but maybe not 5 million of them)

Searching for images Returning irrelevant images is

worse than omitting relevant ones

5

BIASED SAMPLE SETS Often there are orders of magnitude more

negative examples than positive E.g., all images of Kris on Facebook If I classify all images as “not Kris” I’ll have

>99.99% accuracy

Examples of Kris should count much more than non-Kris!

FALSE POSITIVES

7x1

x2

True decision boundary Learned decision boundary

FALSE POSITIVES

8x1

x2

New query

An example incorrectly predicted

to be positive

True decision boundary Learned decision boundary

FALSE NEGATIVES

9x1

x2

New query

An example incorrectly predicted

to be negative

True decision boundary Learned decision boundary

PRECISION VS. RECALL Precision

# of relevant documents retrieved / # of total documents retrieved

Recall # of relevant documents retrieved / # of total

relevant documents Numbers between 0 and 1

10

PRECISION VS. RECALL Precision

# of true positives / (# true positives + # false positives)

Recall # of true positives / (# true positives + # false

negatives) A precise classifier is selective A classifier with high recall is inclusive

11

REDUCING FALSE POSITIVE RATE

12x1

x2

True decision boundary Learned decision boundary

REDUCING FALSE NEGATIVE RATE

13x1

x2

True decision boundary Learned decision boundary

PRECISION-RECALL CURVES

14

Precision

Recall

Measure Precision vs Recall as the decision boundary is tuned

Perfect classifier

Actual performance

PRECISION-RECALL CURVES

15

Precision

Recall

Measure Precision vs Recall as the decision boundary is tuned

Penalize false negatives

Penalize false positives

Equal weight

PRECISION-RECALL CURVES

16

Precision

Recall

Measure Precision vs Recall as the decision boundary is tuned

PRECISION-RECALL CURVES

17

Precision

Recall

Measure Precision vs Recall as the decision boundary is tuned

Better learningperformance

OPTION 1: CLASSIFICATION THRESHOLDS Many learning algorithms (e.g., probabilistic

models, linear models) give real-valued output v(x) that needs thresholding for classification

v(x) > t => positive label given to xv(x) < t => negative label given to x

May want to tune threshold to get fewer false positives or false negatives

18

OPTION 2: WEIGHTED DATASETS Weighted datasets: attach a weight w to

each example to indicate how important it is Instead of counting “# of errors”, count “sum of

weights of errors” Or construct a resampled dataset D’ where each

example is duplicated proportionally to its w As the relative weights of positive vs

negative examples is tuned from 0 to 1, the precision-recall curve is traced out

LINEAR CLASSIFIERS : MOTIVATION Decision tree produces axis-aligned decision

boundaries Can we accurately classify data like this?

x2

x1

PLANE GEOMETRY Any line in 2D can be expressed as the set of

solutions (x,y) to the equation ax+by+c=0 (an implicit surface) ax+by+c > 0 is one side of the line ax+by+c < 0 is the other ax+by+c = 0 is the line itself

y

x

b

a

PLANE GEOMETRY In 3D, a plane can be expressed as the set of

solutions (x,y,z) to the equation ax+by+cz+d=0 ax+by+cz+d > 0 is one side of the plane ax+by+cz+d < 0 is the other side ax+by+cz+d = 0 is the plane itself

a b

c

z

x

y

LINEAR CLASSIFIER In d dimensions,

c0+c1*x1+…+cd*xd =0 is a hyperplane. Idea:

Use c0+c1*x1+…+cd*xd > 0 to denote positive classifications

Use c0+c1*x1+…+cd*xd < 0 to denote negative classifications

24

PERCEPTRON

S gxi

x1

xn

ywi

y = f(x,w) = g(Si=1,…,n wi xi)

+ +

+

++ -

-

--

-x1

x2

w1 x1 + w2 x2 = 0

g(u)

u

25

A SINGLE PERCEPTRON CAN LEARN

S gxi

x1

xn

ywi

A disjunction of boolean literals x1 x2 x3

Majority function

26

A SINGLE PERCEPTRON CAN LEARN

S gxi

x1

xn

ywi

A disjunction of boolean literals x1 x2 x3

Majority functionXOR?

27

PERCEPTRON LEARNING RULE θ θ + x(i)(y(i)-g(θT x(i))) (g outputs either 0 or 1, y is either 0 or 1)

If output is correct, weights are unchanged If g is 0 but y is 1, then the value of g on

attribute i is increased If g is 1 but y is 0, then the value of g on

attribute i is decreased

Converges if data is linearly separable, but oscillates otherwise

28

PERCEPTRON

S gxi

x1

xn

ywi

+ +

+ +

+ -

-

- -

-

?

y = f(x,w) = g(Si=1,…,n wi xi)

g(u)

u

29

UNIT (NEURON)

S gxi

x1

xn

ywi

y = g(Si=1,…,n wi xi)g(u) = 1/[1 + exp(-u)]

30

NEURAL NETWORK Network of interconnected neurons

S gxi

x1

xn

ywi

S gxi

x1

xn

ywi

Acyclic (feed-forward) vs. recurrent networks

31

TWO-LAYER FEED-FORWARD NEURAL NETWORK

Inputs Hiddenlayer

Outputlayer

w1j w2k

32

NETWORKS WITH HIDDEN LAYERS Can represent XORs, other nonlinear

functions Common neuron types:

Soft perceptron (sigmoid), radial basis functions, linear, …

As the number of hidden units increase, so does the network’s capacity to learn functions with more nonlinear features

How to train hidden layers?

33

BACKPROPAGATION (PRINCIPLE) Treat the problem as one of minimizing

errors between the example label and the network output, given the example and network weights as input Error(xi,yi,w) = (yi – f(xi,w))2

Sum this error term over all examples E(w) = Si Error(xi,yi,w) = Si (yi – f(xi,w))2

Minimize errors using an optimization algorithm Stochastic gradient descent is typically used

Gradient direction is orthogonal to the level sets (contours) of E,points in direction of steepest increase

Gradient direction is orthogonal to the level sets (contours) of E,points in direction of steepest increase

Gradient descent: iteratively move in direction

Gradient descent: iteratively move in direction E

Gradient descent: iteratively move in direction E

Gradient descent: iteratively move in direction E

Gradient descent: iteratively move in direction E

Gradient descent: iteratively move in direction

Gradient descent: iteratively move in direction

STOCHASTIC GRADIENT DESCENT For each example (xi,yi), take a gradient

descent step to reduce the error for (xi,yi) only.

43

STOCHASTIC GRADIENT DESCENT Objective function values (measured over all

examples) over time settle into local minimum

Step size must be reduced over time, e.g., O(1/t)

44

NEURAL NETWORKS: PROS AND CONS Pros

Bioinspiration is nifty Can represent a wide variety of decision boundaries Complexity is easily tunable (number of hidden

nodes, topology) Easily extendable to regression tasks

Cons Haven’t gotten close to unlocking the power of the

human (or cat) brain Complex boundaries need lots of data Slow training Mostly lukewarm feelings in mainstream ML

(although the “deep learning” variant is en vogue now)

NEXT CLASS Another guest lecture