Sequence Labeling, Contd - Sameer...

30
Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP WINTER 2017 February 2, 2017 Based on slides from Dan Klein, and everyone else he copied from.

Transcript of Sequence Labeling, Contd - Sameer...

Page 1: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

SequenceLabeling,Contd

Prof.SameerSinghCS295:STATISTICALNLP

WINTER2017

February2,2017

BasedonslidesfromDanKlein,andeveryoneelsehecopiedfrom.

Page 2: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

Outline

CS295:STATISTICALNLP(WINTER2017) 2

MarginalInferenceinHMMs:F/Balgorithm

MaximumEntropyMarkovModels

ConditionalRandomFields

NeuralSequenceTagging

Page 3: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

Outline

CS295:STATISTICALNLP(WINTER2017) 3

MarginalInferenceinHMMs:F/Balgorithm

MaximumEntropyMarkovModels

ConditionalRandomFields

NeuralSequenceTagging

Page 4: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

ExpectationMaximization

CS295:STATISTICALNLP(WINTER2017) 4

K-Means

InitializationPickKrandomcentroids

LabelDatafromtheModelClusterallthepoints

UpdatetheModelfromData

Updatecentroids

Page 5: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

LabelDatafromtheModel

CS295:STATISTICALNLP(WINTER2017) 5

Hard-EM

Soft-EM

Page 6: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

DynamicProgramming

CS295:STATISTICALNLP(WINTER2017) 6

Page 7: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

ForwardBackwardAlgorithm

CS295:STATISTICALNLP(WINTER2017) 7

Page 8: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

UpdatingtheModelfromData

CS295:STATISTICALNLP(WINTER2017) 8

Hard-EM

Soft-EM

Page 9: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

Outline

CS295:STATISTICALNLP(WINTER2017) 9

MarginalInferenceinHMMs:F/Balgorithm

MaximumEntropyMarkovModels

ConditionalRandomFields

NeuralSequenceTagging

Page 10: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

NamedEntityRecognition

CS295:STATISTICALNLP(WINTER2017) 10

GeorgeW.BushspokefromtheWhiteHousetoday.

PERPER PEROOOLOCLOCOO

B-PERI-PERI-PEROOOB-LOCI-LOCOO

B-PERI-PERE-PEROOOB-LOCE-LOCOO

Page 11: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

Max.EntropyMarkovModels

CS295:STATISTICALNLP(WINTER2017) 11

Page 12: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

GraphicalModelNotation

CS295:STATISTICALNLP(WINTER2017) 12

S E

HMMs

S E

MEMMs

Page 13: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

AddingFeatures(forPOS)

CS295:STATISTICALNLP(WINTER2017) 13

CurrentWord

WindowWords

Page 14: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

AddingFeatures

CS295:STATISTICALNLP(WINTER2017) 14

S E

Page 15: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

PredictionsUsingMEMMs

CS295:STATISTICALNLP(WINTER2017) 15

Greedy

ViterbiDecoding

Page 16: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

TrainingMEMMs

CS295:STATISTICALNLP(WINTER2017) 16

Page 17: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

Outline

CS295:STATISTICALNLP(WINTER2017) 17

MarginalInferenceinHMMs:F/Balgorithm

MaximumEntropyMarkovModels

ConditionalRandomFields

NeuralSequenceTagging

Page 18: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

LabelBiasProblem

CS295:STATISTICALNLP(WINTER2017) 18

S

O O O

I I I

B B B

E

B B

Page 19: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

ConditionalRandomFields

CS295:STATISTICALNLP(WINTER2017) 19

Page 20: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

GraphicalModelNotation

CS295:STATISTICALNLP(WINTER2017) 20

S E

HMMs/MEMMs

S E

CRFs

Page 21: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

PredictionsUsingCRFs

CS295:STATISTICALNLP(WINTER2017) 21

Page 22: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

LikelihoodTrainingofCRFs

CS295:STATISTICALNLP(WINTER2017) 22

Page 23: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

LikelihoodTrainingofCRFs

CS295:STATISTICALNLP(WINTER2017) 23

Page 24: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

Forward-BackwardAlgorithm

CS295:STATISTICALNLP(WINTER2017) 24

Page 25: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

Outline

CS295:STATISTICALNLP(WINTER2017) 25

MarginalInferenceinHMMs:F/Balgorithm

MaximumEntropyMarkovModels

ConditionalRandomFields

NeuralSequenceTagging

Page 26: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

SimpleNeuralTagger

CS295:STATISTICALNLP(WINTER2017) 26

PRP

I

VB

love

NN

foodStart End

Page 27: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

MEMM-ish NeuralTagger

CS295:STATISTICALNLP(WINTER2017) 27

PRP

I

VB

love

NN

foodStart End

S E

Page 28: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

RecurrentNeuralTagger

CS295:STATISTICALNLP(WINTER2017) 28

PRP

I

VB

love

NN

food

Page 29: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

BidirectionalRNNTagger

CS295:STATISTICALNLP(WINTER2017) 29

PRP

I

VB

love

NN

food

Page 30: Sequence Labeling, Contd - Sameer Singhsameersingh.org/.../wi17/slides/lecture-0202-sequence-labeling-contd… · Sequence Labeling, Contd Prof. Sameer Singh CS 295: STATISTICAL NLP

Upcoming…

CS295:STATISTICALNLP(WINTER2017) 30

• Homework2isdue(~10days):February13,2017• Write-up,data,andcodeforHomework2isup• Askquestionsearly!

Homework

• ProposalisdueonTuesday:February7,2017• Only2pagesProject

• Papersummaries:February17,February28,March14• Only1 pageeachSummaries


Stacks contd.

Stacks contd.

Sequence Labeling (III) Conditional Random Fieldsbrenocon/anlp2018/lectures/...4 • Seq. labeling as log-linear structured prediction 6.2. SEQUENCE LABELING AS STRUCTURE PREDICTION

Sequence Labeling (III) Conditional Random Fieldsbrenocon/anlp2018/lectures/...4 • Seq. labeling as log-linear structured prediction 6.2. SEQUENCE LABELING AS STRUCTURE PREDICTION

ICMLA 2010 Tutorial on Sequence Labeling: Generative and

ICMLA 2010 Tutorial on Sequence Labeling: Generative and

Novel transformer networks for improved sequence labeling ... · are evaluated by the model, hidden Markov models evaluate the full genome sequence. However, due to the limited capacity

Novel transformer networks for improved sequence labeling ... · are evaluated by the model, hidden Markov models evaluate the full genome sequence. However, due to the limited capacity

Multithreading (contd.)

Multithreading (contd.)

CS 6840: Natural Language Processing Sequence Tagging …oucsace.cs.ohio.edu/~razvan/courses/nlp6840/lecture03.pdf[Romeo & Juliet, Nino Rota] 26. Sequence Labeling as Classification

CS 6840: Natural Language Processing Sequence Tagging …oucsace.cs.ohio.edu/~razvan/courses/nlp6840/lecture03.pdf[Romeo & Juliet, Nino Rota] 26. Sequence Labeling as Classification

Eigenvalues - Contd

Eigenvalues - Contd

A Survey on Recent Advances in Sequence Labeling from Deep ...

A Survey on Recent Advances in Sequence Labeling from Deep ...

Sequence Labeling I - IIT Delhi

Sequence Labeling I - IIT Delhi

Linear Methods For Sequence Labeling: Hidden Markov Models vs …ivan-titov.org/.../struct-pred-class-02.pdf · 2017-10-01 · Outline 3 Sequence labeling / segmentation problems:

Linear Methods For Sequence Labeling: Hidden Markov Models vs …ivan-titov.org/.../struct-pred-class-02.pdf · 2017-10-01 · Outline 3 Sequence labeling / segmentation problems:

Middleware (contd.)

Middleware (contd.)

Structured learning for sequence labeling Part 2: Sequence ... · Hidden Markov Models What is a hidden Markov model?I A tool that helps us solve sequence labeling problems Observations

Structured learning for sequence labeling Part 2: Sequence ... · Hidden Markov Models What is a hidden Markov model?I A tool that helps us solve sequence labeling problems Observations

Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data John Lafferty Andrew McCallum Fernando Pereira.

Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data John Lafferty Andrew McCallum Fernando Pereira.

Graphical Models for Sequence Labeling in NLP

Graphical Models for Sequence Labeling in NLP

Zero-shot Sequence Labeling: Transferring ... - Marek Rei · Zero-shot sequence labeling 3 I like to playing the guitar and sing very louder . It was so long time to wait in the theatre

Zero-shot Sequence Labeling: Transferring ... - Marek Rei · Zero-shot sequence labeling 3 I like to playing the guitar and sing very louder . It was so long time to wait in the theatre

Risk Minimization for Zero-shot Sequence Labeling

Risk Minimization for Zero-shot Sequence Labeling

NAT: Noise-Aware Training for Robust Neural Sequence Labeling

NAT: Noise-Aware Training for Robust Neural Sequence Labeling

Properties of Labeling Methods for Determining Shortest ... · 3. SequenceDisciplines As demonstrated in [2,7], the particular choice of sequence discipline employed in labeling procedures

Properties of Labeling Methods for Determining Shortest ... · 3. SequenceDisciplines As demonstrated in [2,7], the particular choice of sequence discipline employed in labeling procedures

Part-of-Speech Tagging & Sequence Labeling Hongning Wang CS@UVa.

Part-of-Speech Tagging & Sequence Labeling Hongning Wang CS@UVa.

Sequence Labeling to Detect Stuttering Events in Read Speechstaff€¦ · Sequence Labeling to Detect Stuttering Events in Read Speech Sadeen Alharbi 1, Madina Hasan 1, Anthony J

Sequence Labeling to Detect Stuttering Events in Read Speechstaff€¦ · Sequence Labeling to Detect Stuttering Events in Read Speech Sadeen Alharbi 1, Madina Hasan 1, Anthony J