CS 188: Artificial IntelligenceSpring 2007

Lecture 29: Post-midterm course review

5/8/2007

Srini Narayanan – ICSI and UC Berkeley

Final Exam

8:10 to 11 AM on 5/15/2007 at 50 BIRGE Final prep page up Includes all topics (see page).

Weighted toward post midterm topics.

2 double sided cheat sheets allowed as is a calculator.

Final exam review Thursday 4 PM Soda 306.

Utility-Based Agents

Today

Review of post midterm topics relevant for the final. Reasoning about time

Markov Models HMM forward algorithm, Vitterbi Algorithm.

Classification Naïve Bayes, Perceptron

Reinforcement Learning MDP, Value Iteration, Policy iteration TD-value learning, Q-learning,

Advanced topics Applications to NLP

Questions

What is the basic conditional independence assertion for markov models?

What is a problem with Markov Models for prediction into the future?

What are the basic CI assertions for HMM? How do inference algorithms exploit the CI

assertions Forward Algorithm Viterbi algorithm.

Markov Models

A Markov model is a chain-structured BN Each node is identically distributed (stationarity) Value of X at a given time is called the state As a BN:

Parameters: called transition probabilities or dynamics, specify how the state evolves over time (also, initial probs)

X2X1 X3 X4

Conditional Independence

Basic conditional independence: Past and future independent of the present Each time step only depends on the previous This is called the (first order) Markov property

Note that the chain is just a (growing BN) We can always use generic BN reasoning on it (if we

truncate the chain)

X2X1 X3 X4

Example

From initial state (observation of sun)

From initial state (observation of rain)

P(X1) P(X2) P(X3) P(X)

P(X1) P(X2) P(X3) P(X)

Hidden Markov Models

Markov chains not so useful for most agents Eventually you don’t know anything anymore Need observations to update your beliefs

Hidden Markov models (HMMs) Underlying Markov chain over states S You observe outputs (effects) at each time step As a Bayes’ net:

X5X2

E1

X1 X3 X4

E2 E3 E4 E5

Example

An HMM is Initial distribution: Transitions: Emissions:

Conditional Independence

HMMs have two important independence properties: Markov hidden process, future depends on past via the present Current observation independent of all else given current state

Quiz: does this mean that observations are independent given no evidence? [No, correlated by the hidden state]

X5X2

E1

X1 X3 X4

E2 E3 E4 E5

Forward Algorithm

Can ask the same questions for HMMs as Markov chains Given current belief state, how to update with evidence?

This is called monitoring or filtering

Formally, we want: X5X2

E1

X1 X3 X4

E2 E3 E4 E5

Viterbi Algorithm Question: what is the most likely state sequence given

the observations? Slow answer: enumerate all possibilities Better answer: cached incremental version

X5X2

E1

X1 X3 X4

E2 E3 E4 E5

Classification

Supervised Models Generative Models

Naïve Bayes

Discriminative Models Perceptron

Unsupervised Models K-means Agglomerative Cluster

Parameter estimation

What are the parameters for Naïve Bayes?

What is Maximum Likelihood estimation for NB?

What are the problems with ML estimates?

General Naïve Bayes

A general naive Bayes model:

We only specify how each feature depends on the class Total number of parameters is linear in n

C

E1 EnE2

|C| parameters n x |E| x |C| parameters

|C| x |E|n parameters

Estimation: Smoothing

Problems with maximum likelihood (relative frequency) estimates: If I flip a coin once, and it’s heads, what’s the estimate for

P(heads)? What if I flip 10 times with 8 heads? What if I flip 10M times with 8M heads?

Basic idea: We have some prior expectation about parameters (here, the

probability of heads) Given little evidence, we should skew towards our prior Given a lot of evidence, we should listen to the data

Estimation: Laplace Smoothing

Laplace’s estimate (extended): Pretend you saw every outcome

k extra times

What’s Laplace with k = 0? k is the strength of the prior

Laplace for conditionals: Smooth each condition

independently:

H H T

Types of Supervised classifiers

Generative Models Naïve Bayes

Discriminative Models Perceptron

Questions

What is a binary threshold perceptron? How can we make a multi-class

perceptron? What sorts of patterns can perceptrons

classify correctly

The Binary Perceptron

Inputs are features Each feature has a weight Sum is the activation

If the activation is: Positive, output 1 Negative, output 0

f1

f2

f3

w1

w2

w3

>0?

The Multiclass Perceptron

If we have more than two classes: Have a weight vector for

each class Calculate an activation for

each class

Highest activation wins

Linear Separators

Binary classification can be viewed as the task of separating classes in feature space:

w . x = 0

w . x < 0

w . x > 0

Feature design

Can we design features f1 and f2 to use a perceptron to separate the the two classes?

MDP and Reinforcement Learning

What is an MDP (Basics) ? What is Bellman’s equation and how is it

used in value iteration? What is reinforcement learning

TD-value learning Q learning Exploration vs. exploitation

Markov Decision Processes Markov decision processes (MDPs)

A set of states s S A model T(s,a,s’) = P(s’ | s,a)

Probability that action a in state s leads to s’

A reward function R(s, a, s’) (sometimes just R(s) for leaving a state or R(s’) for entering one)

A start state (or distribution) Maybe a terminal state

MDPs are the simplest case of reinforcement learning In general reinforcement learning, we

don’t know the model or the reward function

Bellman’s Equation for Selecting actions

Definition of utility leads to a simple relationship amongst optimal utility values:

Optimal rewards = maximize over first action and then follow optimal policy

Formally: Bellman’s Equation

That’s my equation!

Elements of RL

Transition model, how action influences states Reward R, immediate value of state-action transition Policy , maps states to actions

Agent

Environment

State Reward Action

Policy

sss 221100 r a2

r a1

r a0 :::

MDPs

Which of the following are true?

A

B

C

D

E

Reinforcement Learning

What’s wrong with the following agents?

Model-Free Learning

Big idea: why bother learning T? Update each time we experience a transition Frequent outcomes will contribute more updates

(over time)

Temporal difference learning (TD) Policy still fixed! Move values toward value of whatever

successor occurs

a

s

s, a

s,a,s’s’

Problems with TD Value Learning

TD value learning is model-free for policy evaluation However, if we want to turn

our value estimates into a policy, we’re sunk:

Idea: Learn state-action pairings (Q-values) directly

Makes action selection model-free too!

a

s

s, a

s,a,s’s’

Q-Learning

Learn Q*(s,a) values Receive a sample (s,a,s’,r) (select a using e-greedy) Consider your old estimate: Consider your new sample estimate:

Nudge the old estimate towards the new sample

Set s = s’ until s is terminal

Applications to NLP

How can generative models play a role in MT, Speech, NLP?

List three kinds of ambiguities often found in language?

NLP applications ofBayes Rules!!

Handwriting recognition P (text | strokes) = P (text) * P (strokes | text)

Spelling correction P (text | typos) = P (text) * P (typos | text)

OCR P (text | image) = P (text) * P (image | text)

MT P (english | french) = P (english) * P (french| english)

Speech recognition P (language | sound) = P (LM) * P (sound | LM)

Ambiguities

Headlines: Iraqi Head Seeks Arms Ban on Nude Dancing on Governor’s Desk Juvenile Court to Try Shooting Defendant Teacher Strikes Idle Kids Stolen Painting Found by Tree Kids Make Nutritious Snacks Local HS Dropouts Cut in Half Hospitals Are Sued by 7 Foot Doctors

Why are these funny?

Learning

I hear and I forget

I see and I remember

I do and I understand

attributed to Confucius 551-479 B.C.

Thanks!

And good luck on the final and for the future!Srini Narayanan

snarayan@icsi.berkeley.edu

Phase II: Update Means

Move each mean to the average of its assigned points:

Also can only decrease total distance… (Why?)

Fun fact: the point y with minimum squared Euclidean distance to a set of points {x} is their mean