UCT Algorithm Circle: Introduction to Natural Language ... · IntroductionHistoryLevels of Language...

Introduction History Levels of Language NLP Approaches N-Grams

UCT Algorithm Circle: Introduction to NaturalLanguage Processing

Roland Elliott

18 May 2011

Roland Elliott Introduction to Natural Language Processing

Outline

1 Introduction

2 History

3 Levels of Language

4 NLP ApproachesSymbolicStatisticalUses

5 N-GramsTheoryApplications

What is Natural Language Processing?

Natural Language Processing (NLP) is a range ofcomputational techniques for analysing and representinghuman-like language.Basically it’s the study of enabling computers to understandhuman langauges such as “English” and “German”.NLP is big

Some of the branches within NLP are:Speech Recognition: The reduction of continuous soundwaves to discrete words.Natural Language Understanding: The movement fromisolated words (either written or determined throughspeech recognition) to ‘meaning’.Natural Language Generation: The generation ofappropriate NL responses to non-domain-specific input.Machine Translation: Translating one Natural Language toanother (e.g. “English” to “Russian”)

Outline

1 Introduction

2 History

History of NLP: Weaver and Booth

First application of computers to Natuaral Language wasMachine Translation (MT).Weaver and Booth, using their code-breaking skills(developed during WWII), started one of the earliest MTprojects in 1946.

History of NLP: Early MT

The core assumption of early MT: All languages have thesame “meaning base” and only differ on the surface interms of vocabulary and word order.Systems developed with this assumption would:

read in a sentence written in the source languagedirectly translate each word in the sentence to the targetlanguage using some sort of dictionaryre-order the words to conform to the word order of thetarget langauge

The results of these systems were very poor. The reasonwas that the core assumption was wrong.What they needed was a better theory of language.

History of NLP: Noam Chomsky

In 1957, Chomsky published a book called SyntacticStructures in which he laid out the foundations for his ideaof transformational generative grammar and which lead tothe field of MT gaining a better insight as to howmainstream linguistics could be used to help.During this period, other areas of application for NLPbegan to emerge such as speech recognition. Thelanguage processing (LP) and speech communities weredivided, with LP dominated by the theoretical perpective ofgenerative grammar (hostile to statistical methods) and thespeech community dominated by statistical informationtheory and hostile towards theoretical linguistics.

History of NLP: ALPAC

Due to the development of the syntactic theory of languageand parsing algorithms people believed that fullyfunctional, high-quality systems could be developed withina matter of years.By high-quality they meant systems that would beindistinguishable from human translators.They were wrong.In 1966, the ALPAC (Automatic Language ProcessingAdvisory Committee of the National Academy of Science -National Research Council) report concluded that MT wasnot immediately acheivable and recommended that it notbe funded.

Theoretical Modelling after ALPAC

Although the was a fair decrease in NLP after the ALPACreport there were still come significant developments.Theoretical work focussed on representing meaning andthe development of computationally tractable solutions thattheories of grammar weren’t able to produce.Chomsky (again) further developed his theory oftransformational generative grammar. But these generativegrammars concentrated too much on syntax and didn’tallow for much in terms of semantics.Other models were developed that were more relevant tothe semantic requirements of the day and lendedthemselves to use in computers.

Prototypes

While theoretical models were being developed, so wereprotoypes of these models.

ELIZA would permute and echo user input to imitate aconversation between a psychologist and a patient.PARRY processed groups of words instead of single words(using synonms if keywords weren’t found) and simulated atheory of paranoia.LUNAR was developed by Woods as an interface to adatabse that held information about rock samples from themoon using his theory of augmented transistion networksand another theory of procedural semantics.

Prototypes

Discourse and Generation

In the late 1970s the focus shifted to semantics issues,dealing with discourseGrosz proposed a theory to divide a discourse into unitsbased on her findings in task-related dialoguesMann and Thompson developed the Rhetorical StructureTheory, which showed a hierarchal structure of discourse.Here we also the significant development in NaturalLanguage Generation:

TEXT is a discourse plannerMUMMBLE is a reponse generator produced descriptionsin the form of short texts, usually paragraphs.

Advent of Statistical Methods

Awareness grew of the fact that isolation of solutions toNLP problems caused a limitation on progress.There was also a push for applications to work on broad,real-world problems.This lead to researchers re-examining the non-symbolic ornon-theoretical models that were abandoned earlier on inNLP.By the end of the 1980s symbolic approaches had beenused to address many problems in NLP and statisticalapproaches were shown to be complementary to symbolicapproaches.

In the 1990s the field starting growing rapidly. This is dueto a number of reasons:

Increasing availability of large amounts of electronic text.Increasing speed and memory in computers.The internet.

Statistical approaches have suceeded in solving manygeneric problems in NLP such as part-of-speech taggingand word sense disambiguation.Ironically, they are now the standard in NLP.

Outline

1 Introduction

2 History

Levels

We can split language up into 7 “levels”:1 Phonology2 Morphology3 Lexical4 Syntactic5 Semantics6 Discourse7 Pragmatic

Phonology

One of the lowest levels of language. Deals withinterpretation of speech within and across words.The 3 rules for phonological analysis:

1 Phonetic rules - sounds within words2 Phonemic rules - variation of sound when words are

spoken together (“sort of”, “should’ve”)3 Prosodic rules - change in stress and tone in sentence (e.g.

emphasis)

NLP problems on this level: Speech Recognition

Phonology

emphasis)

Phonology

emphasis)

Phonology

emphasis)

Phonology

emphasis)

Phonology

emphasis)

Phonology

emphasis)

Morphology

Deals with the components that make up words (calledmorphemes). Morphemes are the smallest part ofmeaning in language.Example: the word preregistration can be split into 3morphemes:

1 A prefix: pre-2 A stem: registra3 A suffix: -tion

Since these morphemes don’t change their meaningacross words, we can break words into their morphemesand determine the approximate meaning of unknownwords.NLP problems on this level: Natural LanguageUnderstanding - Semantics and part-of-speech tagging

Morphology

Lexical

Deals with word-level understanding.Usually makes use of some kind of lexicon (in symbolicapproaches) of varying detail.NLP problems on this level: part-of-speech tagging (we’lllook at this in detail later) and non-ambiguous semanticrepresentation (arguments, types of arugments, etc.)

Lexical

Syntactic

Deals with uncovering meaning from the grammaticalstructure of a sentence.Example: word order contributes to meaningdisambiguation in English.Usually requires some theory of grammar and a parserthat applies this theory.NLP problems on this level: Part-of-speech tagging,Chunking and Chinking

Syntactic

Semantic

Deals with determining of meaning of words based on theircontext in a sentence.Example: Assuming we’ve determined that file is a noun(part-of-speech tagging), we still don’t know what kind offile. A tool to shape one’s fingernails? A folder for storinginformation (in real life or in the computer)? A line ofpeople in a queue?We can use the context of the sentence to determinewhat’s being done to file, for example, to determine whichfile is being spoken of.

Semantic

Discourse

Unlike the syntactic and semantic levels, this level doesn’ttreat paragraphs as a concatenation of independentsentences. Here meaning is determined through theconnections between different sentences.NLP problems on this level: Anaphora resolution (replacingpronouns with the entity they convey) and Discourse/textstructure recognition (determining the purpose ofsentences in a discourse).An example of Discourse/text structure recognition wouldbe placing sentences from a news paper articles into oneof the following categories: Lead, Main Story, PreviousEvents, Evaluation, Attributed Quotes, and Expectation.

Discourse

Pragmatics

Deals with applying world knowledge to text to reduce readmeaning into words that isn’t encoded into them.This requires additional world knowledge andunderstanding, such as understanding of intentions, plansand goals. Some NLP applications use inference modelsand knowledge bases.As an example consider the following sentences:

1 The city councilors refused the demonstrators a permitbecause they feared violence.

2 The city councilors refused the demonstrators a permitbecause they advocated revolution.

Pragmatics

Levels

The 7 levels again:1 Phonology2 Morphology3 Lexical4 Syntactic5 Semantics6 Discourse7 Pragmatic

Outline

1 Introduction

2 History

General NLP approach

One general approach to NLP is the following:1 Read in text either directly from written text or through

speech recoginition.2 Assign lists of meanings to each of the words.3 Use the tools available to reduce the words that have more

than one possible meaning (ambiguity)

How are the various approches different from one another?

Symbolic

Outline

1 Introduction

2 History

Symbolic

Symbolic NLP Approaches

Symbolic approaches are based on analysis of linguisticmodels of lanuage and tend to define rules and uselexicons to perform NLP.The early MT was an exmaple of very basic symbolicprocessing.Their primary source of data is manually defined rules,based on a grammatical theory.Symbolic approaches allow for machine learning throughinference or deduction or other rules.

Symbolic

Statistical

Outline

1 Introduction

2 History

Statistical

Statistical NLP Approaches

Statistical approaches usually have some body ofelectronic text known as a corpus (pl. corpora).From the corpus generalised models of linguistics can beapproximated without the need for linguistic or worldknowledge for disambiguation.In contrast to symbolic approaches, statistical models useobservable data as a primary source.

Statistical

Outline

1 Introduction

2 History

NLP Approaches

The Symbolic and Statistical methods create a spectrumfor other approaches to work in.For example, Connectionist approaches developgeneralised models from extensive examples (statistical)while also combining this with various theories ofrepresentation (symbolic).

NLP Approaches

Sometimes one approach is better for a task than another(statistical methods are better speech recognition). Othertimes the two approaches can be used in harmony(part-of-speech tagging).

Symbolic systems are most useful in Machine Learningsince they usually have some way of inferring knowledge.Statistical systems are most useful in cases where rulesaren’t known for certain but can be determined from corpusdata.

NLP Approaches

Outline

1 Introduction

2 History

What are N-Grams?

They are a tool used in statistical approaches to solvecertain problems that arise in NLP.N-Grams are used to assign a conditional probability topossible next token (e.g. word) in a sequence based onwhat has come so far.It uses the corpus to determine these probabilities.We say that a N-Gram is trained on a corpus when it hascollected this data from the corpus.A 1-gram is also called a Unigram, a 2-gram called aBigram and a 3-gram called a Trigram. We will considermostly Bigrams, but the principles we look at are easilyextendable to higher N-Grams.The N in the N-Gram will determine how many precedingtokens it uses to make its prediction.

What are N-Grams?

Theory

Outline

1 Introduction

2 History

Theory

Example problem: Part-of-speech tagging

Problem Consider the sentence “The President will askCongress to increase grants to states for vocationalrehabilitation”. How to we determine the part-of-speech a wordis depending on the words that preceded it?Part of speech Brown Tag ExampleArticle at at, this, theNoun nn chair, computerVerb vb run, jump, talkAdjective jj red, ambiguousPreposition in through, underSentence-ending punc. . ., ;Modal auxillary md should, may, willInfinitive to to, “om te” in afr.

Theory

Problem Consider the sentence “The President will askCongress to increase grants to states for vocationalrehabilitation”. How to we determine the part-of-speech a wordis depending on the words that preceded it?Part of speech Brown Tag ExampleArticle at at, this, theNoun nn chair, computerVerb vb run, jump, talkAdjective jj red, ambiguousPreposition in through, underSentence-ending punc. . ., ;Modal auxillary md should, may, willInfinitive to to, “om te” in afr.

Theory

Initially we could use morphological and syntatic criterionfor part-of-speech tagging.A morphological criterion is one that looks at the internalstructure of a word. For example, suffixing an adjectivewith -ness usually turns it into a noun. So happy→happiness, rehabilitate→ rehabilitation. So if we encouterthese we can assume they’re nouns.A syntatic criterion is on that looks at the contexts in whicha word can occur. For example we might say thatadjectives most commonly appear directly before nouns.So in the sentence “the red postbox”, if we’ve alreadydetermined that postbox is a noun then we can guess that“red” is being used as an adjective.

Theory

Example problem: Unigram tagging

Unigrams are the simplest N-Grams. For each token in thecorpus they assign the tag that is most likely for that token.Example: frequent is most likely an adjective (tag: jj)because it most often appears as one in the corpus.So for our sentence we might get “The:at President:nnwill:md ask:vb Congress:nn to:to increase:nn grants:nnto:to states:nn for:in vocational:jj rehabilitation:nn”

Theory

Example problem: Bigram tagging

Bigram taggers take into account the word we are lookingat and the word before (2-Grams). Given this context, thetagger assigns the most likely part-of-speech tag to thecurrent word. It does so using a bigram table:

tag ask Congress to increase to grants statesat nntl to to

bd to nns tomd vb vbvb np to vb nns to nnsnp to toto vb vbnn np to nn nns to

Theory

New Problem How do we calculate the data used in theBigram table? Answer Maths!

We want to calculate the probability that a certain word is aspecific part-of-speech given the presceding word. Say forexample, we had the setence “I saw a saw”. Now,assuming we’ve determined that the a preceding thesecond saw is an article (tag:at). Then we want tocalculate the probability that saw is a noun (tag:nn) given itis preceded by an article: P(saw:nn|pn vb at).If we let C(sequence) be the number of times thatsequence appears in the corpus then one way we coulddetermine these probabilities would be:

P(saw : nn|pn, vb,at) =C(pn, vb,at , saw : nn)

C(pn, vb,at)

Theory

C(pn, vb,at)

Theory

C(pn, vb,at)

Theory

C(pn, vb,at)

Theory

When Bigram tagging, we approximate the probability of acertain tag for a word from the information of just theprevious tag:

P(wn : tn|tn−1, tn−2, ...) ≈ P(wn : tn|tn−1)

This assumption, that the probability of a tag only dependson previous tags is called a Markov assumption.How do we calculate these N-Gram probabilities?

Theory

The simplest way to do this is by using the MaximumLikelihood Estimation (MLE).We get the MLE by counting the number of times anN-Gram appears in the corpus and normalise so that it liesbetween 0 and 1.For us (with bigrams) we will calculate C(tn−1,wn : tn) andthen divide it by the total amount of times a bigram startswith tn−1:

P(wn : tn|tn−1) =C(tn−1,wn : tn)∑

t C(tn−1,wn : t)

Note: when calculating these counts we pad the sentenceat the begininng and the end: <s> I saw a saw </s>

Theory

t C(tn−1,wn : t)

Theory

t C(tn−1,wn : t)

Theory

t C(tn−1,wn : t)

Theory

t C(tn−1,wn : t)

Theory

Given our MLE equation we can reduce the bottom term bynoticing that ∑

C(tn−1,wn : t) = C(tn−1,wn)

So we have the following:

P(wn : tn|tn−1) =C(tn−1,wn : tn)

C(tn−1,wn)

Using this approximation we can calculate the data in thebigram table. We needn’t calculate the entire table, though.

Theory

C(tn−1,wn)

Theory

C(tn−1,wn)

Theory

C(tn−1,wn)

Applications

Outline

1 Introduction

2 History

Applications

Applications of N-grams

Some of the things N-Grams can be used for are1 Part-of-speech tagging (as we’ve seen)2 Chunking and Chinking3 Word prediction for Natural Language Generation4 Sequence probabilities for Speech Recognition5 Sequence probabilities for Machine Translation

The last three parts require us to be able to predict wordsthat come after a given sequence. So instead ofP(wn : tn|tn−1, tn − 2, ...) we calculateP(wn|wn−1,wn−2, ...). So if we want to calculate MLEsimilarly to how did before we get:

P(wn|wn−1,wn−2, ...) ≈ P(wn|wn−1) =C(wn−1,wn)

C(wn−1)

Applications

C(wn−1)

Applications

C(wn−1)

Applications

C(wn−1)

Applications

C(wn−1)

Applications

C(wn−1)

Applications

C(wn−1)

Applications

C(wn−1)

Applications

Sequence Probabilities

Once we can predict next word probabilities we can find ajoint probability for an entire sequence, using the ChainRule of Probability :

P(w1, ...,wn) = P(w1)...P(w3|w1,w2)...P(wn|w1,w2, ...,wn−1)

P(wi |w1,w2, ...,wi−1)

Applications

P(wi |w1,w2, ...,wi−1)

Applications

P(wi |w1,w2, ...,wi−1)

Applications

Machine Translation

This is useful when determining missing words in speechrecognition or which of a list of possible translations is thebest:

he briefed to reporters on the chief contents of the statementhe briefed reporters on the chief contents of the statementhe briefed to reporters on the main contents of the statementhe briefed reporters on the main contents of the statement

Here the 4th statement is most likely and is, therefore, thebest translation of the sentence into English.Similarly, we can use this to reduce ambiguity in speechrecognition solutions.

Applications

Machine Translation

Applications

Machine Translation

Applications

Machine Translation

Applications

Machine Translation

UCT Algorithm Circle: Introduction to Natural Language ... · IntroductionHistoryLevels of Language...

Documents

Transcript of UCT Algorithm Circle: Introduction to Natural Language ... · IntroductionHistoryLevels of Language...