Machine Learning

Part I: Classification and Bayesian Learning

Ref: E. Alpaydin, Intro to Machine Learning, MIT 2004

Machine Learning• Machine Leaning is programming computers to optimize a

perf criteria using example data or past experience– Inference from samples

• There is a process that explains the data we observe. But we don’t know the details about how the data are generated.– Internet requests, failure events, etc

• It’s hard to identify (model) the process completely, we could construct a good and useful approximation that detect certain patterns. Such patterns would help us to understand the process and make predictions about the future.

Types of Machine Learning• Supervised learning is to create a function from training data. The

training data consist of pairs of input objects (typically vectors), and desired outputs. – Classification: Given an input, the output is Boolean (yes/no) to predict

a class label of the input object; – Regression: If the label is a numerical value, learn the function f(x)

that best explain the input instance;• Unsupervised learning: manual labels of inputs are not used.

– Clustering: partition a data set into subsets (clusters), so that the data in each subset share some common trait

• Semi-supervised learning: make use of both labeled and unlabeled data for training

• Reinforcement Learning– Learning a policy: A sequence of outputs; No supervised output but

delayed reward– Examples: game playing, robot navigation

Supervised Learning

• Use of Supervised Learning• Classification• Regression• Evaluation Methodology• Bayesian Learning for Classification

Why Supervised Learning?

• Prediction of future cases: Use the rule to predict the output for future inputs

• Knowledge extraction: The rule is easy to understand

• Compression: The rule is simpler than the data it explains

• Outlier detection: Exceptions that are not covered by the rule, e.g., fraud

•5

Classification

•6

• E.g: Credit scoring• Differentiating

between low-risk and high-risk customers from their income and savings

• Rule-based prediction

Discriminant: IF income > θ1 AND savings > θ2 THEN low-risk ELSE high-risk

Learning a Class from Examples• Given a set of examples of

cars, with a label of “family car” or not according to a survey, class learning is to find a description that is shared by all positive examples.

• Use of the class info– Prediction: Is car x a family car?

– Knowledge extraction: What do people expect from a family car?

Training set X

Nt

tt,r 1}{ xX

negative is if 0

positive is if 1

x

xr

2

1

x

xx

Input representation

Attributes: price & engine power

Label of each instance

Hypothesis Class: C

•9

Most specific hypothesis, S

Most general hypothesis, G

2121 power engine AND price eepp

Learning is to find a particular

hypothesis h to approximate C

Hypothesis h and Empirical Error

•10

negative as classifies if 0

positive as classifies if 1)(

x

xx

h

hh

N

t

tt rhhE1

1)|( xX

Error of h:

Model Selection & Generalization• Learning is an ill-posed problem: data is not sufficient to

find a unique solution– Limited number of sample data– Some data might be noise due to imprecision in recording,

labeling, or hidden (latent, unobservable) attributes that affect the label of instances

• The need for inductive bias: assumptions about class structureH – Why rectangle, not circle or irregular shape?

– What’s degree of tightness of fitting?

• Generalization: How well a model performs on new data

Noise and Model ComplexitySimple model is preferred• Easy to use (check)

(lower time complexity)• Easy to train (lower

space complexity)• Easyto explain

(more interpretable)• Easy to generalize (lower

variance )

•12

Noise: any anomaly in the data

which leads it infeasible to reach

a zero-error classification

with a simple hypothesis class

Probably Approximately Correct (PAC) Learning

• How many training examples N should we have, such that with probability at least 1 ‒ δ, h has error at most ε ?

• Each strip is at most ε/4• Pr that we miss a strip 1‒ ε/4• Pr that N instances miss a strip (1 ‒ ε/4)N

• Pr that N instances miss 4 strips 4(1 ‒ ε/4)N

• 4(1 ‒ ε/4)N ≤ δ and (1 ‒ x)≤exp( ‒ x)• 4exp(‒ εN/4) ≤ δ and N ≥ (4/ε)log(4/δ)

•13

2-Class vs K-ClassNt

tt,r 1}{ xX

, if 0

if 1

ijr

jt

it

ti C

C

x

x

, if 0

if 1

ijh

jt

it

ti C

C

x

xx

•14

K-class problem be viewed as K 2-class problem:Train hypotheses hi(x), i =1,...,K:

Regression

• Examples– Price of a used car– Speed of Top500

• x : car attributesy : price

y = g (x | θ)g ( ) model,

θ parameters

•15

•y = wx+w0

Linear regression

Basic Concepts• Interpolation

– Find a function that best fits a training set with no presence of noise

– r = f(x)

• Extrapolation– Predict the output for any x, if x is NOT in the training set

• Regression– Noise factor must be considered– r = f(x) + OR there’re hidden variables we couldn’t

observe: r = f(x, z)

Regression

01 wxwxg

012

2 wxwxwxg

N

t

tt xgrN

gE1

21|X

•17

N

t

tt wxwrN

w,wE1

2

0101

1|X

tt

t

N

ttt

xfr

r

rx 1,X

For a given test set, find g() that minimizes the empirical error

Underfitting vs Overfitting

• Underfitting: Hypothesis (H) less complex than actual model (C)– Using a line to fit data sampled from a 3rd order

polynomial– Accuracy increases with more sample data; may

not enough if the hypothesis is too complex

• Overfitting: H more complex than C– Having more training data helps but only up to a

certain point

Triple Trade-Off Trade-off between three factors :

1. Complexity of the hypothesisH, c (H): capacity of the hypothesis class

2. Training set size, N, 3. Generalization error, E, on new examples

• As NE• As c (H)first Eand then EThe error of an

over-complex hypothesis can be kept in check by increasing the amount of training data, but only up to a point)

Cross-Validation

• To estimate generalization error, we need data unseen during training.

• Three types of data in cross-validation:– Training set (50%)– Validation set (25%)– Test (publication) set (25%)

• Resampling when there is few data

Dimensions of a Supervised Learner: Summary

1. Model g() and parameter

2. Loss function L(): diff between desired output and approximation

3. Optimization procedure:

|xg

t

tt g,rLE || xX

•21

X|min arg

E*

return the argument that minimizes