Prosody and NLP

Seminar by Nikhil : 06005004Adith : 06005005

Prachur : 06D05011

We have a presentation this Friday ?We have a presentation this Friday ?We have a presentation this Friday ?

Abstract

Speech Processing and Natural Language Processing share a common area of study: Language. However, over time, they have grown to have little in common regarding theoretical models or methods of analysis. NLP takes written text as the starting point for it's analysis,  however, a lot of valuable information is lost in encoding speech as merely text. It is commonly accepted that intonational features of spoken language can greatly aid NLP tasks (like adjective scope resolution).

We explore the foundations of the study of Prosody and observe some approaches that use prosodic cues to aid NLP.

Motivation

• Language is not text driven but speech driven.

• NLP currently has written text as the starting point for it's analysis (primarily due to the abundance of such data).

• A lot of information is lost on ignoring spoken features and just looking at the text.

A Way Out ?

• Utilize spoken features for NLP tasks

• NLP needs all the help it can get.

Dealing at the pragmatics or discourse level is extremely untenable

Prosodic cues carry useful pragmatic information

What is Prosody, exactly?

• Comes from Poetry, prosody refers to the study of poetic meters[1] (rhythms)

• Written text treats words as the basic building blocks of language.

• Spoken language treats syllables as the basic building block.

• Syllables : Consisting of Onset, Nucleus and Coda

• Wikipedia has this to say :  Prosody is the rhythm, stress, and intonation of connected speech (as opposed to smaller elements like syllables or words). 

Prosody may reflect : features of the speaker, emotional state of a speaker, features of the utterance, ironic or sarcastic, emphasis, contrast, and focus

What is Prosody, exactly? (contd.)

Intonation ?• Conveys paralinguistic information, emphasis and

contrast.

• Intonation on a particular word could differentiate between sentence moods.

– You are finISHED (interrogative)– You are FINIshed (imperative)

Image courtesy Google Image Search

• Stress is applied on Content Words in spoken utterances.

    cOntent - Noun. "I really liked their presentation's content.“

contEnt - Verb. "I have done my best. I am content.“

• Stress on a pair of words distinguishes between the syntactic role played by each word in the pair.

• tight<pause>rope : A rope that is held taut.• tight-rope : A circus-act uses this contraption :)

[2]

And Stress!

Courtesy

tom The Dancing Bug

Prosodic cues

 Prosodic functions important for linguistics are – Marking of boundaries (syntactic, semantic or

dialogue units.)

– Relative duration of phonetic segments

– At syllable level : Energy, intensity, duration and intonation of syllable.

We shall see two approaches of using these features in tasks central to NLP.

• Cue Used : Relative duration of phonetic segments

• Aim : To improve the parsing of ambiguous

sentences.

• Method : Augmenting the syntax grammar with a few

non-terminals and rules.

Prosody-Augmented Syntax Grammars[3]

• Concept of “Word Break Indices” used to show prosodic decoupling between neighboring words.

• E.g. - • Andrea 1 moved 1 the 0 bottle 3 under 0 the 0

bridge.

• Andrea 1 moved 3 the 0 bottle 1 under 0 the 0 bridge.

Break indices were generated by analysing the coda that have a pause.Coda is the final sounded consonant of a word E.g. – cup , milk

Grammar Modification

• Original grammar rules like S -> NP VP etc. are changed to S -> NP link1 VP.

• “Link” non-terminals are used for the word-break indices.

• The Parse Trees for the aforementioned example become…

Example 1

S

• Andrea 1 moved 1 the 0 bottle 3 under 0 the 0 bridge.

PP Vmoved NPN

Punder

NP

Nbottle

VP

Dthe

N bridge

NP

Dthe

Andrea

Link1

1

Link2

1 Link4

3

Link3

0

Link5

0

Link6

0

Example 2• Andrea 1 moved 3 the 0 bottle 1 under 0 the 0 bridge.

S

PP

Vmoved

NPN

Punder

NP

Nbottle

VP

Dthe

N bridge

NP

Dthe

Andrea

Link1

1

Link2

3

Link4

1

Link3

0

Link5

0

Link6

0

Results

• The incorporation of prosody resulted in a reduction of about 25% in the number of parses found .

• Parse times increase about 37%.

• Extremely common cases of syntactic ambiguity can be resolved with prosodic information, and that grammars can be modified to take advantage of prosodic information for improved parsing

Using Prosodic Features in Language Models[4]

• The outlined approach uses syllable-based prosodic cues, namely– Duration of the syllable– Average energy (intensity)– The average F0 (fundamental frequency of the

syllable) contour– The slope of the F0 contour (visualised as

intonation-rising or falling/flat)

Recognition of Prosodic Features

Courtesy [4]

Prosody in Language Model

• We want to measure P(wn | wn-1,wn-2,…,F)

• Naively modelled by linear interpolation : – Assumption : prosody features independent of

previous words (not true!!). P(wn | wn-1,wn-2,…,F) = αP(wn | wn-1,wn-2,…) + (1- α)P(wn|F)

• We want something better

Factored Language Model

• Instead of a word W we will deal with a set of word-factors F={f1,f2..fk} (Factors may include the word itself)

• Here, F is chosen as {W, prosodic features}

• The four prosodic features are encoded as binary numbers(s0 to s15).

• These numbers are assigned to each syllable of the word .

• For e.g. the prosodic representation for the word “Actually” can be either ‘s10s12s6’ or ‘s10s15s6’ .

Conclusion

• Prosodic Cues can play an important role as a heuristic for many NLP tasks.

• All is not one way traffic though. POS tagging (since its relatively accurate) is used to aid speech synthesis tasks which conventionally used only prosodic cues[5]

Future Work

• Handling prosodic information is a first step towards integration of Speech Processing and NLP

Courtesy ZITS

References1. Fromkin, Rodman and Hyams, An Introduction to

Language, 7th Ed, Thomson and Wadsworth,20022. En.wikipedia.org : Prosody, Austrian School of

Thought, Intonation3. John Bear and Patti Price (1990), “Prosody,

Syntax and Parsing”, in proceedings of the 28th annual meeting of the ACL

4. Songfang Huang and Steve Renals (2007), “Using Prosodic Features in Language Models for Meetings”, IRTG annual meeting

5. http://speech.iiit.net/~raghavendra/Webpage/ppprts.pdf