Introduction & Information Theory Ling570 Advanced Statistical Methods in NLP January 3, 2012.

Introduction & Information Theory

Ling570Advanced Statistical Methods in NLP

January 3, 2012

RoadmapCourse Overview

Information theory

Course Overview

Course InformationCourse web page:

http://courses.washington.edu/ling572


http://courses.washington.edu/ling572Syllabus:

Schedule and readingsLinks to other readings, slides, links to class recordingsSlides posted before class, but may be revised


http://courses.washington.edu/ling572Syllabus:

Schedule and readingsLinks to other readings, slides, links to class recordingsSlides posted before class, but may be revised

Catalyst tools: GoPost discussion board for class issuesCollectIt Dropbox for homework submission and TA

commentsGradebook for viewing all grades

GoPost Discussion BoardMain venue for course-related questions,

discussionWhat not to post:

Personal, confidential questions; Homework solutions

GoPost Discussion BoardMain venue for course-related questions,

discussionWhat not to post:

Personal, confidential questions; Homework solutionsWhat to post:

Almost anything else course-related Can someone explain…? Is this really supposed to take this long to run?

GoPost Discussion BoardMain venue for course-related questions, discussion

What not to post:Personal, confidential questions; Homework solutions

What to post: Almost anything else course-related

Can someone explain…? Is this really supposed to take this long to run?

Key location for class participationPost questions or answersYour discussion space: Michael & I will not jump in often

GoPostEmily’s 5-minute rule:

If you’ve been stuck on a problem for more than 5 minutes, post to the GoPost!

GoPostEmily’s 5-minute rule:

If you’ve been stuck on a problem for more than 5 minutes, post to the GoPost!

Mechanics:Please use your UW NetID as your user idPlease post early and often !

Don’t wait until the last minuteKeep up with the GoPost – hard to use

retrospectivelyNotifications:

Decide how you want to receive GoPost postings

EmailShould be used only for personal or confidential

issuesGrading issues, extended absences, other problems

General questions/comments go on GoPost

EmailShould be used only for personal or confidential

issuesGrading issues, extended absences, other problems

General questions/comments go on GoPost

Please send email from your UW account Include Ling572 in the subject If you don’t receive a reply in 24 hours (48 on

weekends), please follow-up

Homework SubmissionAll homework should be submitted through CollectIt

Tar cvf hw1.tar hw1_dir

Homework due 11:45 Thursdays

Late homework receives 10%/day penalty (incremental)

Most major programming languages accepted C/C++/C#, Java, Python, Perl, Ruby

If you want to use something else, please check first

Please follow naming, organization guidelines in HW

All programming assignments should run on the CL cluster under Condor

Homework Assignments (Mostly) Implementation tasks designed to get hands-

on understanding of ML approaches

Focus on core concepts, not minute optimizations If gold standard achieves 90.7%, 89.8% is okay

Not scored directly on efficiency, but.. If it’s too slow, hard to debug, test, etc

Not scored on optimal software design either Try to avoid hardcoding, but don’t need complex design

GradingHomework assignments: 80%

Reading assignments: 10%

Class participation: 10%

No midterm or final exams

One homework assignment may be dropped

GradesGrades in Catalyst Gradebook

TA feedback returned through CollectIt

GradesGrades in Catalyst Gradebook

TA feedback returned through CollectIt

Extensions: only for extreme circumstances Illness, family emergencies

Incomplete: only if all work completed up last two weeksUW policy

WorkloadCLMS courses carry a heavy workload

Ling572 is no exception



Estimates (per week):~3 hours: Lecture10-12 hours: Homework assignments

Highly variable, depending on prior programming exp.

1-3 hours: Reading + reading assignments



Estimates (per week): ~3 hours: Lecture 10-12 hours: Homework assignments

Highly variable, depending on prior programming exp. 1-3 hours: Reading + reading assignments

Tracking: GoPost thread for each assignment: please post

Consider automatic time tracker (e.g. ‘hamster’ for linux)

RecordingsAll classes will be recorded

Links to recordings appear in syllabusAvailable to all students, DL and in class



Please remind me to:Record the meeting (look for the red dot)Repeat in-class questions



Please remind me to:Record the meeting (look for the red dot)Repeat in-class questions

Note: Instructor’s screen is projected in classAssume that chat window is always public

Contact InfoGina: Email: [email protected]

Office hour:Fridays: 12:30-1:30 (after Treehouse meeting)Location: Padelford B-201Or by arrangement

Available by Skype or Adobe Connect

mailto:[email protected]

Contact InfoGina: Email: [email protected]

Office hour:Fridays: 12:30-1:30 (after Treehouse meeting)Location: Padelford B-201Or by arrangement

Available by Skype or Adobe Connect

TA: Michael Wayne Goodman: Email: [email protected] hour: Time: TBD, see GoPostLocation: Treehouse




Online OptionPlease check you are registered for correct

sectionCLMS in-class: Section AState-funded: Section BCLMS online: Section C

Online OptionPlease check you are registered for correct

sectionCLMS in-class: Section AState-funded: Section BCLMS online: Section C

Online attendance for in-class studentsNot more than 2 times per term (e.g. missed bus,

ice)

Online OptionPlease check you are registered for correct section

CLMS in-class: Section AState-funded: Section BCLMS online: Section C

Online attendance for in-class studentsNot more than 2 times per term (e.g. missed bus, ice)

Please enter meeting room 5-10 before start of classTry to stay online throughout class

Online TipIf you see:

You are not logged into Connect. The problem is one of the following: the permissions on the resource you are trying to access are incorrectly set.Please contact your instructor/Meeting Host/etc. you do not have a Connect account but need to have

one. For UWEO students: If you have just created your UW NetID or just enrolled

in a course…..

Clear your cache, close and restart your browser

Course Description

Course PrerequisitesProgramming Languages:

Java/C++/Python/Perl/..

Operating Systems: Basic Unix/linux

CS 326 (Data structures) or equivalentLists, trees, queues, stacks, hash tables, …Sorting, searching, dynamic programming,..

Stat 391 (Probability and statistics): random variables, conditional probability, Bayes’ rule, ….

Ling 570 (or similar)

Course Prerequisites Programming Languages:

Java/C++/Python/Perl/..

Operating Systems: Basic Unix/linux

CS 326 (Data structures) or equivalent Lists, trees, queues, stacks, hash tables, … Sorting, searching, dynamic programming,..

Stat 391 (Probability and statistics): random variables, conditional probability, Bayes’ rule, ….

Ling 570 (or similar)

If you haven’t taken Ling570 or Ling472, please email me.

TextbookNo textbook

Online readings

TextbookNo textbook

Online readings

Reference / Background: Jurafsky and Martin, Speech and Language

Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, 2nd edition, 2008Available from UW Bookstore, Amazon, etc

Manning and Schutze, Foundations of Statistical Natural Language ProcessingEarly edition available online through UW library

Course GoalsUnderstand the basis of machine learning

algorithms that achieve state-of-the-art results



Focus on classification and sequence labeling



Focus on classification and sequence labeling

Concentrate on basic concepts of machine learning techniques and application to NLP tasks Not a computational learning theory class

Won’t focus on proofs

Model QuestionsMachine learning algorithms

Decision trees and naïve bayesMaxEnt and Support Vector Machines….

Model QuestionsMachine learning algorithms

Decision trees and naïve bayesMaxEnt and Support Vector Machines….

Key questionsWhat is the model?What assumptions does the model make? How many parameters does the model have?

Model QuestionsTraining: How are the parameters learned?

Decoding: How does the model assign values?

Model QuestionsTraining: How are the parameters learned?

Decoding: How does the model assign values?

Pros and Cons:How does the model handle…

outliers? missing data? noisy data? Is it scalable? How long does it take to train? decode?How much training data is needed? Labeled?

Unlabeled?

Tentative Outline for Ling572

Unit #0 (0.5 weeks): Basics Introduction Information theoryClassification review

Outline for Ling572

Unit #0 (0.5 weeks): Basics Introduction Information TheoryClassification review

Unit #1 (3 weeks): Classic Machine LearningK Nearest NeighborsDecision TreesNaïve BayesPerceptrons (?)

Outline for Ling572Unit #3: (4 weeks): Discriminative Classifiers

Feature SelectionMaximum Entropy ModelsSupport Vectors Machines



Unit #4: (1.5 weeks): Sequence LearningConditional Random FieldsTransformation Based Learning



Unit #4: (1.5 weeks): Sequence LearningConditional Random FieldsTransformation Based Learning

Unit #5: (1 week): Other TopicsSemi-supervised learning,…

Outline for Ling572Topics:

Feature selection approaches

Beam search

Toolkits:Mallet, libSVM

Using binary classifiers for multiclass classification

Early NLPEarly approaches to Natural Language

ProcessingSimilar to classic approaches to Artificial

Intelligence



Intelligence

Reasoning, knowledge-intensive approaches



Intelligence


Largely manually constructed rule-based systems



Intelligence


Largely manually constructed rule-based systems

Typically focused on specific, narrow domains

Early NLP: IssuesRule-based systems:


Too narrow and brittleCouldn’t handle new domains: out of domain -> crash



Hard to maintain and extendLarge manual rule bases incorporate complex

interactionsDon’t scale



Hard to maintain and extendLarge manual rule bases incorporate complex

interactionsDon’t scale

Slow

Reports of the Death of NLP…ALPAC Report: 1966

Automatic Language Processing Advisory Committee



Failed systems efforts, esp. MT, lead to defunding



Failed systems efforts, esp. MT, lead to defunding

Example: (Probably apocryphal)English -> Russian -> English MT“The spirit is willing but the flesh is weak.”“The vodka is good but the meat is rotten.”

…Were Greatly Exaggerated

Today:

Watson wins Jeopardy!

SIRI speaks and understands

Google searches and translates

So What Happened?Statistical approaches and machine learning


Hidden Markov Models boosted speech recognition



Noisy channel model gave statistical MT



Noisy channel model gave statistical MT

Unsupervised topic modeling

Etc

So What Happened?Many stochastic approaches developed 80s-90s

Rise of machine learning accelerated 2000-present

Why?



Why?Large scale data resources

Web dataTraining corpora: Treebank, TimeML, Discourse

treebankWikipedia, etc



Why?Large scale data resources

Web dataTraining corpora: Treebank, TimeML, Discourse treebankWikipedia, etc

Large scale computing resourcesProcessors, storage, memory: local and cloud



Why? Large scale data resources

Web dataTraining corpora: Treebank, TimeML, Discourse treebankWikipedia, etc

Large scale computing resourcesProcessors, storage, memory: local and cloud

Improved learning algorithmsSupervised, semisupervised, unsupervised, structured…

Information Theory

EntropyCan be used a measure of

Match of model to data

How predictive an n-gram model is of next word

Comparison between two models

Difficulty of a speech recognition task

EntropyInformation theoretic measure

Measures information in model

Conceptually, lower bound on # bits to encode


Measures information in model


Entropy: H(X): X is a random var, p: prob fn

)(log)()( 2 xpxpXHXx


Measures information in grammar


Entropy: H(X): X is a random var, p: prob fn

E.g. 8 things: number as code => 3 bits/trans Alt. short code if high prob; longer if lower

Can reduce

)(log)()( 2 xpxpXHXx

Computing EntropyPicking horses (Cover and Thomas)

Send message: identify horse - 1 of 8If all horses equally likely, p(i)


Send message: identify horse - 1 of 8If all horses equally likely, p(i) = 1/8



8

1

38/1log8/1log8/1)(i

bitsXH



Some horses more likely:1: ½; 2: ¼; 3: 1/8; 4: 1/16; 5,6,7,8: 1/64

8

1

38/1log8/1log8/1)(i

bitsXH



Some horses more likely:1: ½; 2: ¼; 3: 1/8; 4: 1/16; 5,6,7,8: 1/64

0, 10, 110, 1110, 111100, 111101, 111110, and 111111.

bitsipipXHi

2)(log)()(8

1

8

1

38/1log8/1log8/1)(i

bitsXH

Entropy of a SequenceBasic sequence

)(log)(1

)(1

1211

1

n

LW

nn WpWpn

WHn n


Entropy of language: infinite lengthsAssume stationary & ergodic

Shannon-Breiman-Mcmillan Theorem

)(log)(1

)(1

1211

1

n

LW

nn WpWpn

WHn n


Entropy of language: infinite lengthsAssume stationary & ergodic

Shannon-Breiman-Mcmillan Theorem

)(log)(1

)(1

1211

1

n

LW

nn WpWpn

WHn n

),...,(log1

lim)(

),...,(log),...,(1

lim)(

1

11

nn

nLW

nn

wwpn

LH

wwpwwpn

LH

Entropy of EnglishShannon’s experiment

Subjects guess strings of letters, count guessesEntropy of guess seq = Entropy of letter seq1.3 bits; Restricted text

Entropy of EnglishShannon’s experiment

Subjects guess strings of letters, count guessesEntropy of guess seq = Entropy of letter seq1.3 bits; Restricted text

Build stochastic model on text & computeBrown computed trigram model on varied corpusCompute (per-char) entropy of model1.75 bits

Cross-EntropyComparing models

Actual distribution unknown pUse simplified model to estimate m

Closer match will have lower cross-entropy

Relative EntropyCommonly known as Kullback-Liebler divergence

Expresses difference between probability distributions



Not a proper distance metric:



Not a proper distance metric: asymmetricKL(p||q) != KL(q||p)

Joint & Conditional Entropy

Joint entropy:

Joint & Conditional Entropy

Joint entropy:

Conditional entropy:

Perplexity and Entropy

Given that

Consider the perplexity equation:

PP(W) = P(W)-1/N =


Given that


PP(W) = P(W)-1/N = = =


Given that


PP(W) = P(W)-1/N = = = 2H(L,P)

Where H is the entropy of the language L

Mutual InformationMeasure of information in common between two

distributions


distributions

Symmetric: I(X;Y) = I(Y;X)


distributions

Symmetric: I(X;Y) = I(Y;X)

I(X;Y) = KL(p(x,y)||p(x)p(y))

Next TimeA little review

Decision TreesApplications of entropy

Introduction & Information Theory Ling570 Advanced Statistical Methods in NLP January 3, 2012.

Documents

Transcript of Introduction & Information Theory Ling570 Advanced Statistical Methods in NLP January 3, 2012.