1

What is NLG?

Input• Formal representation of some information (linguistic or

non-linguistic)

Output• Single sentences or texts (reports, explanations,

instructions, etc.)

Resources drawn upon• Context of situation• World and domain knowledge• Domain communication knowledge• Linguistic knowledge

2

NLG in Text Summarization

On Monday, GreenChip Solutions made an acquisition offer to BuyOut Inc., a St. Louis-based plastic tree manufacturer that had tremendous success in equipping American households with pink plastic oak trees.

GreenChip offered to acquire the plastic tree

manufacturer BuyOut.

( ... ( ... )

( ... )) Analysis

Generation

3

NLG in Machine Translation

SL-TextSL-Text TL-TextTL-Text

Analysis Generation

Transfer

Interlingua

4

NLG in Dialogue Systems

When does the train leave?

Analysis

Dialoguemanager

Generation“At eleven p.m.,from platform four”

Speech synth.

Speech recog.

5

NLG as a self-contained task

Data/knowledge Base

(...

(...)

(....))Texts

Intermed. Repr. 1 Intermed. Repr. 2

6

NLG Tasks

• Content Determination

• Document Structure Planning

• Sentence (Micro) Planning– Lexicalization– Referring Expression Determination– Aggregation– Syntactic Structure Determination

• Surface Realization

7

Content Determination Strategies

• Data driven strategy

• Document structure driven strategies

– Text plan (schema) driven strategy

– Discourse relation driven strategy

• Combined (data and structure driven) strategy

8

Data Driven Content Determination

Based on:• Formal representation of data

• Context-dependent and domain-specific content selection rules

Strategy:

• Determine what data to communicate in the text according to messages or selection rules

9 Data Driven Content Determination (an Example, Input data)

<station> Stuttgart-Mitte </station>

<substance> ozone </substance><mseries>

<meas>

<time> 06:00 </time>

<value> 20 </value>

</meas>

<meas>

<time> 06:30 </time>

<value> 33 </value>

</mseries>

10 Data Driven Content Determination (an Example, Selection Rules)

IF (hourly average value of substance X > 25)

THEN select substance X for realization

IF (value at x:30) > 1.1(value at x:00)

AND(value at x:00 > 100)

THEN select value at x:00 for realizationAND

select value at x:30 for realization

ELSE select average value of x:00 and x:30

11 Document structure driven content determination

Basic idea:

When generating a text we must ensure that information elements in the text are related to each other so as to

achieve that the text is coherent.

So why not select the content according to a coherent plan or following rhetorical relations that must be recognizable between data elements?

Two common approaches:

1. Text plan (schema) driven strategy

2. Discourse relation driven strategy

12

Schemata

Introduced by K. McKeown (1985)(also known as Generic Structure Potential)

Observations:

• Specific text types often reveal typical structures

• A structure gives rise to a recursive document plan, a schema, which

consists of less complex subschemata or elementary elements.

• A schema ensures the coherence of the text that is built according to it

• Schemata can be compiled in terms of text grammars

13

Schema Example

(NextDayGlobalWeatherForecastSchema:

CloudInfo

PrecipitationInfo

CurrentPrecipitation

PrecipitationProgression

WindInfo

TemperatureInfo

EarlyTemperature

HighestTemperature)

14

Application of Schemata

SubstanceConcentrationSchema:

(CurrentConcentration <cc-sruleset>

ReferenceEvaluation

RefConcentration <rc-sruleset>

CurrentRefCompare <crc-sruleset>

ConcentrIntervalAssociation <cia-sruleset>

CompThreshold <ct-sruleset>

LegalInfo <li-sruleset1>

RegionEvaluationLowestConcentr <lc-sruleset1>HighestConcentr <hc-sruleset1>)

15 Text Structure Driven Content Determination (Example, sel.rules)

<cc-sruleset>

((measure –time-> timepoint);(timepoint –hour-> ?h;?h := (get NOW INPUT))

(measure –substance-> ?s;?s := (get `substance INPUT))

(measure –value-> ?v;?v := (get `value INPUT))

(measure –location-> ?l;?l := (get `location INPUT))

...)

16 Evaluation of Schema Based Content Determination

Pros:• Relatively easy to establish for a well-restricted domain• The selected information elements form a coherent text

plan which adequately reflects the structure of the texts of the domain

• Computationally efficient

Cons:• Domain-specific• Hardly allow for variation

17 Discourse Relation Driven Content Determination

Based on:

• Formal representation of underlying information elements

• Discourse relations between information elements

• Rules for navigation along the discourse relations

• Heuristics for relation sequences for a given text type

Strategy:

• Collect the data or information elements travelling along the

discourse relations and using the heuristics

18

Rhetorical Structure Theory

Introduced by W. Mann & S. Thompson (1987)

Observations:

• Between text elements (sentences, paragraphs, ...) „rhetorical

relations“ hold.

• Rhetorical relations (besides other elements) make the text

coherent.

• Rhetorical relations can be classified with respect to their

function.

• For a specific domain, a sequence of rhetorical relations in a

text can be precompiled.

19

RST-relation example (1)

1. Heavy rain and thunderstorms in North Spain and on the Balearic Islands.

2. In other parts of Spain, still hot, dry weather with temperatures up to 35 degrees Celcius.

CONTRAST

Symmetric (multiple nuclei) Relation:

20

RST-relation example (2)

2. In Cadiz, the thermometer might rise as high as 40 degrees.

1. In other parts of Spain, still hot, dry weather with temperatures up to 35 degrees Celcius.

ELABORATION

Asymmetric (nucleus-satellite) Relation:

21

RST-based Content Determination

• Motto: choosing what to say and deciding how to structure

it cannot be divided

• Text planner by Moore and Paris 1993: Map

communicative goals via linguistic goals to language

• Each time alternative strategies for a (sub-) goal are

considered, new content can be selected

• Example: When the goal is to convince the reader of proposition P,

and the system reckons the reader is unlikely to believe P, check the

knowledge base for evidence supporting P, and verbalize it

22

RST-based Content Determination (2)

• Model of mental states and communicative goals, e.g.:

– (know ?agent (ref ?description))

– (bel ?agent (?predicate ?e1 ?e2))

• Example: plan operator for MOTIVATION from Moore/Paris:

– EFFECT: (MOTIVATION ?act ?goal)

– CONSTRAINTS: (AND (STEP ?act ?goal)

– (GOAL ?hearer ?goal))

– NUCLEUS: (BEL ?hearer (STEP ?act ?goal))

– SATELLITES: NIL

• Moore/Paris text planner works by top-down hierarchical

expansion; alternative: bottom-up planning, e.g. (Marcu 1997)

23 Evaluation of RST-based Content Determination

Pros:• The selected information elements form a coherent text plan

• Flexible production of text plans of variable size

• Allows for explicit reasoning about the reader‘s beliefs

Cons:• Usually, an information element in the data/knowledge base is

involved in discourse relations between several other information elements: constraints for selecting one path must be available

• Formalizing (all) RST relations is difficult

• Needs a sophisticated planning mechanism

• Computationally expensive

24

Sentence (Micro) Planning

Goal:

To map a text plan into a sequence of sentence or phrase plans (with lexical items already determined)

Tasks:• Lexicalization• Referring Expression Determination• Aggregation• Syntactic Structure Determination

25

Lexicalization (1)

Lexicalization is the process of mapping semantic entities onto lexical items.

• Aspects of lexicalization:

- Single open-class words (nouns, verbs, adjectives, adverbs)

- Function words (prepositions) that belong to the subcategorization frames of open-class words

- Discourse markers- Idiosyncratic word combinations (collocations)

26

Lexicalization (2)

Lexicalization is guided by:

• Semantics of the entities to be mapped

• Communicative (textual) constraints of the domain

and previous discourse

• Pragmatics

– Basic-level preferences

– Argumentative intent

– User model: expertise, vocabulary

27

Lexicalization (3): Stylistic Features

• Formality: motion picture - movie - flick

• Euphemism: washroom; ethnic cleansing

• Slant: gentleman - man - jerk

• Archaic: apothecary; albeit

• Floridity: house - habitation

• Abstractness: unemployed - out of work

• Force: big - monstrous

28

Lexicalization Variations, Examples

1. The temperature dropped from 30 on Tuesday to 15 degrees C on Wednesday.

vs.

With 23 degrees C, the temperature on Wednesday was lower than on Tuesday. On Tuesday 30 degrees were measured.

vs.

On Tuesday, the thermometer read 30 degrees C. On Wednesday, it was much cooler.

29

Lexicalization Strategies

• Constrain source entities until only one lexical option is

available

• Match parts of the source structure with parts of lexical

items

• If source items are indexed or labeled with lexical items:

choose one according to constraints that are either explicitly

available or are derived from the context

30 Lexicalization Strategies (Constraining source entities)

(eat

:agent Freddy)IF :agent IS `human´

THEN essen

ELSE fressen

(cause

:causer Freddy

:causee Freddy

:caused: die)

IF :agent = :patient

AND

:agent IS `human´

THEN „commit suicide“

ELSE kill

31 Lexicalization Strategies (Matching parts)

Example from MOOSE (Stede 1999):

pour1

tom1

coolant1

path1 radiator1

“into”

Tom poured coolant into the radiator. Tom schüttete Kühlmittel in den Kühler.

CAUSER

OBJECT

PATH DESTINATION

DIRECTION

32 Lexicalization Strategies (Equating source and lexical entities)

Animal

Water Animal

Fish

Shark

Mammal

Cetacean

Dangerous Fish

Sand Shark

Dolphin

Tiger Shark

FN (Reiter)

33 Lexicalization Strategies (Indexing)

`lecture´ => LECTURE

• Information available in the lexicon:

TALK, PRESENTATION, ...

[to] lecture

give [ART ~]

deliver [ART ~]

attend [ART ~]

follow [ART ~]

prepare [ART ~]

...

• Also (possibly) available: Paraphrasing rules

34

Aggregation

Aggregation is the process of entity grouping at various levels of processing with the goal to avoid redundancy.

Types of aggregation:

• Conceptual aggregation

• Lexical aggregation

• Syntactic aggregation

35

Aggregation (Some Examples, 1)

Conceptual aggregation:1. Heavy rain is expected in Zuffenhausen.

2. Heavy rain is expected in Cannstatt3. Heavy rain is expected in Vaihingen

1.-3. Heavy rain is expected in Metropolitan Stuttgart.

Lexical aggregation:1. From 9 am to 11 am the ozone concentration fell.2. Then the ozone concentration rose.3. Then the ozone concentration fell.4. Then the ozone concentration rose

1.-4. From 9 pm on the ozone concentration varied.

36

Aggregation (Some Examples, 2)

Syntactic aggregation:

• Referential aggregation

1. The employment rate among women fell.

2. The employment rate among men rose.

1.+2. The employment rate among women fell while that among men rose.

• Elision

1. The employment rate among women rose.

2. The employment rate among men rose.

1.+2. The employment rate rose.

37

Aggregation, Rule Examples

(x / process

:agent ?A

...)

AND(x / process

:agent ?B

...)

(x / process

:agent (c /conj:arg (?A ?

B))...)

38

Choice of Referring Expressions

The process of determining how to identify entities known from the extralinguistic context and entities introduced in the previous discourse.

Types of referring expressions:

• Noun Definiteness/Deixis

• Pronominalization

• Elision

• Direct lexical references

• Indirect lexical references

39

Referring Expressions Examples

1. John saw a small boy. The boy was crying.

2. John saw a small boy. He was crying ...3. John saw a small boy. The poor kid was crying

4. The comments are not restricted to classic AI, but are appropriately applied to theoretical linguistics as well.

5. Today‘s lecture is on Agent Technology. The lecturer is a visiting professor from the UCLA.

40

Referring Expressions , Rule Example

IF (X is denotation of a transformation

ANDProp.focus mentioned in last sentence

ANDResultative Noun (RN) available for X)

THEN IF (RN unique)THEN CHOOSE RN

ELSE ...Put the batter into the oven. Remove the cake in two hours.

41

Syntacticization (1)

Syntacticization is the process of choosing the most appropriate syntactic construction for a message.

Options to be chosen from:

• Sequence of sentences vs. Coordination vs. Subordination:

The Black Forest station is located in the woods. At this station, an ozone concentration of 259 g/m3 has been measured.

vs.

At the Black Forest station, which is located in the woods, an ozone concentration of 259 g/m3 has been measured.

42

Syntacticization (2)

• Sentence vs. Nominal Phrase:

Tomorrow, it is cloudy with sunny periods and patchy drizzle ending in the afternoon.

vs.

Tomorrow, clouds with sunny periods and patchy drizzle till the afternoon.

43

Interdependency in Microplanning

Problematic:

• Nearly all microplanning tasks are intertwined with each other, i.e., the realization of one depends on the realization of the other and vice versa.

• Theoretically still unclear which phenomenon belongs to which task.

• Theoretically still not entirely clear whether to treat microplanning as a set of different tasks.

44

Interdependency in Microplanning

1. Today‘s lecture is on Agent Technology. The lecturer is a visiting professor from the UCLA.

2. The topic of today‘s lecture is Agent Technology. It is given by a visiting professor from the UCLA.

3. A visiting professor from the UCLA gives today a lecture on Agent Technology.

4. Today‘s lecture, which is on Agent Technology, is given by a visiting professor from the UCLA.

45

Surface realization (1)

Goal:

To realize a sentence/phrase plan as a sentence/phrase at the surface

Tasks:

• Syntactic realization

• Morphologization

46

Surface Realization Input (1)

(c / creative-material-action:tense past:lex construct-past:passivization-q passivization:actee (h / object

:lex house:multiplicity-q unitary:singularity-q singular:identifiability-q identifiable)

:relations (i / in:range (l / two-d-location

:lex forest:mult…-q unitary:singularity-q singular:ident…-q identifiable)))

47

Surface Realization Input (2)

(construct :tense past:voice passive-subjectival-> (house

:number singular:article def)

-prep.objectival-> (in-objectival-> (forest

:number singular :article def)))

48

Modularization of generation tasks

Content Selection + Text Structuring: Text Planner

Microplanning: Sentence Planner

+ Grammar

Surface Realization: Grammar

OR

Content Selection + Text Structuring: Text Planner

Lexicalization: Lex. Chooser

Syntacticization + Surface Real.: Grammar

49

Modularization of generation tasks (cont.)

OR

Content Selection + Text Structuring: (Text) Planner

Lexicalization: (Text) PlannerSyntacticization: (Text) Planner

Surface Real.: Grammar

OR

All tasks dealt with in one module

50

Architecture Issues

Main Types of NLG-System architectures:

• Pipeline Architecture

• Iterative Architecture

• (Quasi or partially) Parallel Architectures

- Communication of separate modules via a common information

space (e.g. blackboard)

- Incremental providing of information by individual modules or of

the input (interleaved architecture)

- No separate modules (integrated architecture)

51

A Standard Pipeline Architecture

Text planner

Sentence planner

Grammar

52

A Possible Iterative Architecture

Formulator

Grammar

Textplanner

53

A Possible Blackboard Architecture

AdministratorKnowl. Sources

Blackboard(s)

Discourse Str.

Content Deter.

Lexicalization

...

54

Level of Abstraction of Input Structures

The level of abstraction of the input structures is generator-specific and varies extremely from case to case. An input structure may be:

• an export from a data base

• an excerpt from a KL-ONE-like knowledge base

• a Conceptual Graph structure

• a semantic structure

• a syntactic structure

55

Example of a Concrete Input Structure

(c / creative-material-action:tense past:lex construct-past:passivization-q passivization:actee (h / object

:lex house:multiplicity-q unitary:singularity-q singular:identifiability-q identifiable)

:relations (i / in:range (l / two-d-location

:lex forest:mult…-q unitary:singularity-q singular:ident…-q identifiable)))

56

Example of an Abstract Input Structure

<station> Stuttgart-Mitte </station>

<substance> ozone </substance>

<mseries>

<meas>

<time> 06:00 </time>

<value> 20 </value>

</meas>

<meas>

<time> 06:30 </time>

<value> 33 </value>

</mseries>

57

Generation Techniques

Depending on the scale of variation and complexity of texts required, several generation techniques are available:

• Canned text

• Templates

• Full fledged generation

• Combination of the above techniques

58

Templates, Examples

1. Cloudy with sunny periods in <location>. The temperature is expected to rise to <number> degrees C.

2. In <location>, the ozone concentration reached <number> µg/m3.

3. <user-name> was logged in for <duration> hours.

4. The unemployment rate among men for the month of <month> <decreased/increased/remained stable>.

59

Full fledged generation

...

subst. time value

SO2

NO2

19.11.99:18:00

... ... ...

9

7819.11.99:18:00

dimen.

µg/m3

...

station

Berlin

(measure

station: Berlin,

substance:SO2,

time: 19.11.99:18:00,

value:200,

dimension: µg/m3)

On 19.11. at 6pm the SO2 concentration reached 200 µg/m3 in Berlin.

60

Technique of text production

• A paragraph or sentence never changes – its appearance being triggered by specific input data or being obligatory.

Canned Text is appropriate • Only a few variations of sentence and/or phrase structures

are available to communicate a specific information; within a sentence/phrase structure a few arguments may change.

Templates are appropriate• The information to be communicated may vary and the

sentence structures that express depend on the discourse structure progression

Full fledged generation is appropriate

61

62

63

Multimodal Generation

Text presentations and graphical presentations have differing strengths and weaknesses. Their combination can achieve powerful synergies.

However, simply placing textual and graphical information together is no guarantee that one view is supportive of another.If the perspective on the data taken in a graphic and that taken in a text have no relation, then the result is incoherence rather than synergy.

64

Multimodal Generation: Coherence

Multimodal generation is a goal-directed activity, i.e.,

when generating a multimodal document

• the author pursues certain comm. goals

Intentional Structure of the document

• the author chooses an organization of the information that supports its comm. goals

Discourse Structure of the document

65

Intentional Structure of a Document

The Intentional Structure of a document is a hierarchy of Acts that ensure that the goal(s) is/are achieved

1. At each level of the hierarchy, at least one main act must be specified

2. A main act may be supported by subsidiary acts

3. The system must keep track of the beliefs it has and the facts it knows about

66

Discourse Structure of a Document

The Discourse Structure of a document is a hierarchy of coherence relations – as, e.g., specified in RST.

Examples of RST-Relations:

ContrastElaborationMotivationEnablementBackground

67

RST-relation example (1)

1. Heavy rain and thunderstorms in North Spain and on the Balearic Islands.

2. In other parts of Spain, still hot, dry weather with temperatures up to 35 degrees Celcius.

CONTRAST

Symmetric (multiple nuclei) Relation:

68

RST-relation example (2)

2. In Cadiz, the thermometer might rise as high as 40 degrees.

1. In other parts of Spain, still hot, dry weather with temperatures up to 35 degrees Celcius.

ELABORATION

Asymmetric (nucleus-satellite) Relation:

69

Cohesive Links between Doc. Elements

multimodal referring expressions

vs.

crossmode referring expressions

A multimodal referring expression refers to a world object via a combination of at least two media. Each mode conveys some discriminating attributes of the object.

A crossmode referring expression refers to a document part in a different presentation mode.

70

Planning the content and the structure

Communicative Structure + Discourse Structure

Textplanning in (monomodal) Text Generation

Planning mechanisms for multimodal documents can and should be derived from the text planning

mechanisms

!!!RST-like Text Planning!!!

71

RST-based Content Determination (2)

• Model of mental states and communicative goals, e.g.:

– (know ?agent (ref ?description))

– (bel ?agent (?predicate ?e1 ?e2))

• Example: plan operator for MOTIVATION from Moore/Paris:

– EFFECT: (MOTIVATION ?act ?goal)

– CONSTRAINTS: (AND (STEP ?act ?goal)

– (GOAL ?hearer ?goal))

– NUCLEUS: (BEL ?hearer (STEP ?act ?goal))

– SATELLITES: NIL

• Moore/Paris text planner works by top-down hierarchical

expansion; alternative: bottom-up planning, e.g. (Marcu 1997)

72

WIP Planning Strategies

Introduce an object by showing a picture of it:

Header: (Introduce System User ?object Graphics)

Effect: (BMB System user (Isa ?object ?concept)

Applicability Conditions:

(Bel System (Isa ?object ?concept)

Main Acts:

(S-Depict System User ?object ?pic-obj ?picture)

Subsidiary Acts:

(Label System User ?object ?medium)

(Provide-Background System User ?object ?pic-obj ?picture Gr..)

73

WIP Planning Strategies

Provide Background:

Header: (Provide-Background System User ?x ?px ?picture Graphics)

Effect: (BMB System user (Encodes ?px ?x ?picture)

Applicability Conditions:

(And (Bel System (Encodes ?px ?x ?picture))

(Bel System (Perceptually-Access-p User ?x))

(Bel System (Part-of ?x ?z))

Main Acts:

(S-Depict System User ?z ?pz ?picture)

Subsidiary Acts:

(Achieve System (BMB System User (Encodes ?pz ?z ?picture

?medium)

74

WIP Planning Strategies

Establish a coreferential link:

Header: (Establish-coref System User ?r1 ?r2 Graphics)

Effect: (BMB System user (Coref ?r1 ?r2)

Applicability Conditions:

(And (BMB System User (Encodes ?spec1 ?r1))

(BMB System User (Text-Obj ?spec1 ?r1))

(BMB System User (Encodes ?spec2 ?r2))

(BMB System User (Pic-Obj ?spec2 ?r2)))

Main Acts:

(S-Annotate System User ?spec1 ?spec2 ?picture)

75

WIP Planner

1. The user posts the goal to be achieved.

2. The planner identifies the potentially applicable strategies by searching the strategy library for all strategies whose effect field matches the goal.

3. For each strategy found, the conditions are checked.

4. Select one of the applicable strategies (e.g., depending on the preference given to a specific mode).

5. Place the strategy in the corresponding plan node

6. If the strategy has subsidiary act strategies, expand the first; otherwise go to the nearest non-expanded strategy