Artificial IntelligenceArtificial Intelligence

Lecture 15

Introduction to Machine Learning

OverviewOverview

Machine Learning ID3 Decision Tree Algorithm Discussions

Machine LearningMachine Learning

Supervised Learning• Training examples consist of pairs of input

vectors, and desired outputs

Unsupervised Learning• Training examples do not contain hints about

correct outputs• Usually used to identify unusual structures in

data

Inductive Learning BasicsInductive Learning Basics

Inferring a boolean/real-valued function from training examples

A training example is a pair of (x, f(x))• x is the input• f(x) is the output of the function applied to x

Any function that approximates the given set of examples

(a) (b) (c) (d)

Bias: preference for one hypothesis beyond mere consistency

HypothesisHypothesis

Hypothesis SpaceHypothesis Space

A set of all hypotheses consistent with data denoted by H:{H1, H2, …, Hn}

Inductive learning is searching for a good hypothesis in the hypothesis space

Occam’s razor: prefer the simplest hypothesis consistent with data

Inductive Learning Assumption: Any hypothesis found to approximate the target function well over a sufficiently large set of training examples will also approximate the target function well over other unobserved examples.

ID3 Decision Tree Algorithm ID3 Decision Tree Algorithm

Training ExamplesTraining Examples

Decision Tree for Decision Tree for PlayTennisPlayTennis

Outlook Temperature Humidity Wind PlayTennis

Sunny Hot High Weak ?

Overcast Hot High Weak ?

Decision Tree RepresentationDecision Tree Representation

Decision Tree• Each internal node tests an attribute• Each branch takes an attribute value• Each leaf node predict a class label

Disjunction of conjunctions of a set of attribute values

EntropyEntropy

Information GainInformation Gain

ID3 AlgorithmID3 Algorithm

Training ExamplesTraining Examples

Selecting the Best AttributeSelecting the Best Attribute

Hypothesis Space Search by ID3Hypothesis Space Search by ID3

DiscussionsDiscussions

Hypothesis Space Search by ID3Hypothesis Space Search by ID3

Hypothesis space is complete• Target function surely in there …

Outputs a single hypothesis (Which one?)• Cannot determine how many alternatives

No back tracking• Local minima …

Use statistical properties of all training data at each step in search• Robust to noisy data

Inductive Bias in ID3Inductive Bias in ID3

Note the hypothesis space H is the power set of instances X. Unbiased?

Not really.• Preference for short trees, and for those with

high information gain attributes near the root• Bias is a preference for some hypotheses,

rather than a restriction of hypothesis space H, e.g. target concept is not in H.

SummarySummary

Machine Learning ID3 Decision Tree Algorithm Discussions