CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann,...
-
Upload
dorothy-potter -
Category
Documents
-
view
216 -
download
1
Transcript of CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann,...
![Page 1: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/1.jpg)
SUMMARIZATION
![Page 2: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/2.jpg)
CS276BWeb Search and Mining
Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M.
Venkataramani, R. Altman, L. Hirschman, and D. Radev)
Automated Text summarization Tutorial — COLING/ACL’98
![Page 3: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/3.jpg)
3
an exciting challenge...
...put a book on the scanner, turn the dial to ‘2 pages’, and read the result...
...download 1000 documents from the web, send them to the summarizer, and select the best ones by reading the summaries of the clusters...
...forward the Japanese email to the summarizer, select ‘1 par’, and skim the translated summary.
![Page 4: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/4.jpg)
4
Headline news — informing
![Page 5: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/5.jpg)
5
TV-GUIDES — decision making
![Page 6: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/6.jpg)
6
Abstracts of papers — time saving
![Page 7: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/7.jpg)
7
Graphical maps — orienting
![Page 8: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/8.jpg)
8
Textual Directions — planning
![Page 9: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/9.jpg)
9
Cliff notes — Laziness support
![Page 10: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/10.jpg)
10
Real systems — Money making
![Page 11: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/11.jpg)
11
Questions
What kinds of summaries do people want? What are summarizing, abstracting,
gisting,...?
How sophisticated must summ. systems be? Are statistical techniques sufficient? Or do we need symbolic techniques and
deep understanding as well?
What milestones would mark quantum leaps in summarization theory and practice? How do we measure summarization
quality?
![Page 12: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/12.jpg)
What is a Summary?
Informative summary Purpose: replace original document Example: executive summary
Indicative summary Purpose: support decision: do I want to
read original document yes/no? Example: Headline, scientific abstract
![Page 13: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/13.jpg)
Summarization
• Information overload problem• Increasing need for IR and automated
text summarization systems• Summarization: Process of distilling the
most salient information from a source/sources for a particular user and task
![Page 14: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/14.jpg)
Summary : “a reductive transformation of source text to summary text through content condensation by selection and/or generalization on what is important in the source.” (Sparck Jones,1999)
![Page 15: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/15.jpg)
Steps for Summarization
Transform text into an internal representation. Detect important text units. Generate summary
In extracts ▪ no generation▪ information ordering, ▪ anaphora resolution (or avoiding anaphoric structures)
In abstracts ▪ text generation. ▪ Sentence fusion, ▪ paraphrasing, ▪ Natural Language Generation.
![Page 16: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/16.jpg)
16
Summarizing factors Input (Sparck Jones
2007) subject type: domain genre: newspaper articles, editorials, letters, reports... form: regular text structure; free-form source size: single doc; multiple docs (few; many)
Purpose situation: embedded in larger system (MT, IR) or not? audience: focused or general usage: IR, sorting, skimming...
Output completeness: include all aspects, or focus on some? format: paragraph, table, etc. style: informative, indicative, aggregative, critical...
![Page 17: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/17.jpg)
17
ExamplesExercise: summarize the following
texts for the following readers:
text1: Coup Attempt
text2: childrens’ story
reader1: your friend, who knows nothing about South Africa.
reader2: someone who lives in South Africa and knows the political position.
reader3: your 4-year-old niece.reader4: the Library of Congress.
![Page 18: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/18.jpg)
Summarization Techniques
Surface level: Shallow features Term frequency statistics, position in text,
presence of text from the title, cue words/phrases: e.g. “in summary”, “important”
Entity level: Model text entities and their relationship Vocabulary overlap, distance between text
units, co-occurence, syntactic structure, coreference
Discourse level: Model global structure of text Document outlines, narrative stucture
Hybrid
![Page 19: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/19.jpg)
19
‘Genres’ of Summary? Indicative vs. informative
...used for quick categorization vs. content processing.
Extract vs. abstract...lists fragments of text vs. re-phrases content
coherently.
Generic vs. query-oriented...provides author’s view vs. reflects user’s interest.
Background vs. just-the-news...assumes reader’s prior knowledge is poor vs. up-
to-date.
Single-document vs. multi-document source...based on one text vs. fuses together many texts.
![Page 20: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/20.jpg)
20
A Summarization Machine
EXTRACTS
ABSTRACTS
?
MULTIDOCS
Extract Abstract
Indicative
Generic
Background
Query-oriented
Just the news
10%
50%
100%
Very BriefBrief
Long
Headline
Informative
DOC QUERY
CASE FRAMESTEMPLATESCORE CONCEPTSCORE EVENTSRELATIONSHIPSCLAUSE FRAGMENTSINDEX TERMS
![Page 21: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/21.jpg)
21
The Modules of the Summarization Machine
EXTRACTION
INTERPRETATION
EXTRACTS
ABSTRACTS
?
CASE FRAMESTEMPLATESCORE CONCEPTSCORE EVENTSRELATIONSHIPSCLAUSE FRAGMENTSINDEX TERMS
MULTIDOC
EXTRACTS
GENERATION
FILTERING
DOCEXTRACTS
![Page 22: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/22.jpg)
Why Automatic Summarization? Algorithm for reading in many
domains is:1) read summary2)decide whether relevant or not3) if relevant: read whole document
Summary is gate-keeper for large number of documents.
Information overload Often the summary is all that is read.
Human-generated summaries are expensive.
![Page 23: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/23.jpg)
Summary Length (Reuters)
Goldstein et al. 1999
![Page 24: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/24.jpg)
![Page 25: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/25.jpg)
Summarization Algorithms Keyword summaries
Display most significant keywords Easy to do Hard to read, poor representation of content
Sentence extraction Extract key sentences Medium hard Summaries often don’t read well Good representation of content
Natural language understanding / generation Build knowledge representation of text Generate sentences summarizing content Hard to do well
Something between the last two methods?
![Page 26: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/26.jpg)
Sentence Extraction
Represent each sentence as a feature vector
Compute score based on features Select n highest-ranking sentences Present in order in which they occur in text. Postprocessing to make summary more
readable/concise Eliminate redundant sentences Anaphors/pronouns
▪ A woman walks. She smokes. Delete subordinate clauses, parentheticals
![Page 27: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/27.jpg)
Sentence Extraction: Example
Sigir95 paper on summarization by Kupiec, Pedersen, Chen
Trainable sentence extraction
Proposed algorithm is applied to its own description (the paper)
![Page 28: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/28.jpg)
Sentence Extraction: Example
![Page 29: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/29.jpg)
Feature Representation
Fixed-phrase feature Certain phrases indicate summary, e.g. “in summary”
Paragraph feature Paragraph initial/final more likely to be important.
Thematic word feature Repetition is an indicator of importance
Uppercase word feature Uppercase often indicates named entities. (Taylor)
Sentence length cut-off Summary sentence should be > 5 words.
![Page 30: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/30.jpg)
Feature Representation (cont.) Sentence length cut-off
Summary sentences have a minimum length. Fixed-phrase feature
True for sentences with indicator phrase▪ “in summary”, “in conclusion” etc.
Paragraph feature Paragraph initial/medial/final
Thematic word feature Do any of the most frequent content words
occur? Uppercase word feature
Is uppercase thematic word introduced?
![Page 31: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/31.jpg)
Training
Hand-label sentences in training set (good/bad summary sentences)
Train classifier to distinguish good/bad summary sentences
Model used: Naïve Bayes
Can rank sentences according to score and show top n to user.
![Page 32: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/32.jpg)
Evaluation
Compare extracted sentences with sentences in abstracts
![Page 33: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/33.jpg)
Evaluation of features
Baseline (choose first n sentences): 24% Overall performance (42-44%) not very good. However, there is more than one good summary.
![Page 34: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/34.jpg)
Multi-Document (MD) Summarization
Summarize more than one document Harder but benefit is large (can’t scan 100s
of docs) To do well, need to adopt more specific
strategy depending on document set. Other components needed for a production
system, e.g., manual post-editing. DUC: government sponsored bake-off
200 or 400 word summaries Longer → easier
![Page 35: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/35.jpg)
Types of MD Summaries
Single event/person tracked over a long time period Elizabeth Taylor’s bout with pneumonia Give extra weight to character/event May need to include outcome (dates!)
Multiple events of a similar nature Marathon runners and races More broad brush, ignore dates
An issue with related events Gun control Identify key concepts and select sentences
accordingly
![Page 36: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/36.jpg)
Determine MD Summary Type
First, determine which type of summary to generate
Compute all pairwise similarities Very dissimilar articles → multi-event
(marathon) Mostly similar articles
Is most frequent concept named entity? Yes → single event/person (Taylor) No → issue with related events (gun
control)
![Page 37: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/37.jpg)
MultiGen Architecture (Columbia)
![Page 38: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/38.jpg)
Generation
Ordering according to date Intersection
Find concepts that occur repeatedly in a time chunk
Sentence generator
![Page 39: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/39.jpg)
Processing
Selection of good summary sentences
Elimination of redundant sentences Replace anaphors/pronouns with
noun phrases they refer to Need coreference resolution
Delete non-central parts of sentences
![Page 40: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/40.jpg)
Newsblaster (Columbia)
![Page 41: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/41.jpg)
Query-Specific Summarization
So far, we’ve look at generic summaries.
A generic summary makes no assumption about the reader’s interests.
Query-specific summaries are specialized for a single information need, the query.
Summarization is much easier if we have a description of what the user wants.
![Page 42: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/42.jpg)
Genre
Some genres are easy to summarize Newswire stories Inverted pyramid structure The first n sentences are often the best
summary of length n Some genres are hard to summarize
Long documents (novels, the bible) Scientific articles?
Trainable summarizers are genre-specific.
![Page 43: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/43.jpg)
Discussion
Correct parsing of document format is critical. Need to know headings, sequence, etc.
Limits of current technology Some good summaries require natural
language understanding Example: President Bush’s nominees for
ambassadorships▪ Contributors to Bush’s campaign▪ Veteran diplomats▪ Others
![Page 44: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/44.jpg)
44
.Summarization techniques
![Page 45: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/45.jpg)
45
Computational Approach
Top-Down: I know what I want!
User needs: only certain types of info
System needs: particular criteria of interest, used to focus search
Bottom-Up: I’m dead curious:
what’s in the text?
User needs: anything that’s important
System needs: generic importance metrics, used to rate content
![Page 46: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/46.jpg)
46
Review of Methods
Text location: title, position
Cue phrases Word frequencies Internal text cohesion:
word co-occurrences local salience co-reference of names,
objects lexical similarity semantic rep/graph
centrality Discourse structure
centrality
Information extraction templates
Query-driven extraction: query expansion lists co-reference with query
names lexical similarity to
query
Bottom-up methods Top-down methods
![Page 47: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/47.jpg)
47
Query-Driven vs. Text-Driven Focus Top-down: Query-driven focus
Criteria of interest encoded as search specs.
System uses specs to filter or analyze text portions.
Examples: templates with slots with semantic characteristics; termlists of important terms.
Bottom-up: Text-driven focus Generic importance metrics encoded as
strategies. System applies strategies over rep of whole
text. Examples: degree of connectedness in
semantic graphs; frequency of occurrence of tokens.
![Page 48: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/48.jpg)
48
Bottom-Up, using Info. Retrieval IR task: Given a query, find the relevant
document(s) from a large set of documents. Summ-IR task: Given a query, find the
relevant passage(s) from a set of passages (i.e., from one or more documents).
• Questions: 1. IR techniques work on large
volumes of data; can they scale down accurately enough?
2. IR works on words; do abstracts require abstract representations?
xx xxx xxxx x xx xxxx xxx xx xxx xx xxxxx xxxx xx xxx xx x xxx xx xx xxx x xxx xx xxx x xx x xxxx xxxx xxxx xxxx xxxxxx xx xx xxxx x xxxxx x xx xx xxxxx x x xxxxx xxxxxx xxxxxx x xxxxxxxx xx x xxxxxxxxxx xx xx xxxxx xxx xx xxx xxxx xxx xxxx xx xxxxx xxxxx xx xxx xxxxxx xxx
![Page 49: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/49.jpg)
49
Top-Down, using Info. Extraction
IE task: Given a template and a text, find all the information relevant to each slot of the template and fill it in.
Summ-IE task: Given a query, select the best template, fill it in, and generate the contents.• Questions:1. IE works only for very particular
templates; can it scale up?
2. What about information that doesn’t fit into any template—is this a generic limitation of IE?
xx xxx xxxx x xx xxxx xxx xx xxx xx xxxxx xxxx xx xxx xx x xxx xx xx xxx x xxx xx xxx x xx x xxxx xxxx xxxx xxxx xxxx xxxxxx xx xx xxxx x xxxxx x xx xx xxxxx x x xxxxx xxxxxx xxxxxx x xxxxxxxx xx x xxxxxxxxxx xx xx xxxxx xxx xx x xxxx xxxx xxx xxxx xx xxxxx xxxxx xx xxx xxxxxx xxx
Xxxxx: xxxx Xxx: xxxx Xxx: xx xxx Xx: xxxxx xXxx: xx xxx Xx: x xxx xx Xx: xxx x Xxx: xx Xxx: x
![Page 50: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/50.jpg)
50
NLP/IE:• Approach: try to ‘understand’
text—re-represent content using ‘deeper’ notation; then manipulate that.
• Need: rules for text analysis and manipulation, at all levels.
• Strengths: higher quality; supports abstracting.
• Weaknesses: speed; still needs to scale up to robust open-domain summarization.
IR/Statistics:• Approach: operate at lexical
level—use word frequency, collocation counts, etc.
• Need: large amounts of text.
• Strengths: robust; good for query-oriented summaries.
• Weaknesses: lower quality; inability to manipulate information at abstract levels.
Paradigms: NLP/IE vs.
ir/statistics
![Page 51: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/51.jpg)
51
Toward the Final Answer...
Problem: What if neither IR-like nor IE-like methods work?
Solution: semantic analysis of the text
(NLP), using adequate knowledge bases
that support inference (AI).
Mrs. Coolidge: “What did the preacher preach about?”
Coolidge: “Sin.”Mrs. Coolidge: “What did he
say?”Coolidge: “He’s against it.”
– sometimes counting and templates are insufficient,
– and then you need to do inference to understand.
Word counting
Inference
![Page 52: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/52.jpg)
52
The Optimal Solution...
Combine strengths of both paradigms…
...use IE/NLP when you have suitable template(s),
...use IR when you don’t…
…but how exactly to do it?
![Page 53: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/53.jpg)
53
Overview of Extraction Methods Word frequencies throughout the text Position in the text
lead method; optimal position policy title/heading method
Cue phrases in sentences Cohesion: links among words
word co-occurrence coreference lexical chains
Discourse structure of the text Information Extraction: parsing and
analysis
![Page 54: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/54.jpg)
54
Word-frequency-based method (1)
Claim: Important sentences contain words that occur “somewhat” frequently.
Method: Increase sentence score for each frequent word.
Evaluation: Straightforward approach empirically shown to be mostly detrimental in summarization systems.
words
Wordfrequency
The resolving power of words
(Luhn, 59)
![Page 55: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/55.jpg)
Word-frequency-based method
Computes measures of significance
Words: Stemmingdiffer,difference bag of words
Sentences: concentration of high-
score words Cutoff values
established in experiments
SIGNIFICANT WORDS
ALL WORDS
* * * * 1 2 3 4 5 6 7
SENTENCE
SCORE = 42/7 2.3
![Page 56: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/56.jpg)
56
POSition-based method (1)Claim: Important sentences occur
at the beginning (and/or end) of texts.
Lead method: just take first sentence(s)!
Experiments: In 85% of 200 individual paragraphs
the topic sentences occurred in initial position and in 7% in final position (Baxendale, 58).
Only 13% of the paragraphs of contemporary writers start with topic sentences (Donlan, 80).
![Page 57: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/57.jpg)
POSition-based method
Cue method: stigma words (“hardly”, “impossible”) bonus words (“significant”)
Key method: similar to Luhn
Title method: title + headings
Location method: sentences under headings sentences near beginning or end of
document and/or paragraphs (also [Baxendale 58])
![Page 58: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/58.jpg)
58
Optimum Position Policy (OPP) Claim: Important sentences are located at
positions that are genre-dependent; these positions can be determined automatically through training (Lin and Hovy, 97). Corpus: 13000 newspaper articles (ZIFF corpus). Step 1: For each article, determine overlap
between sentences and the index terms for the article.
Step 2: Determine a partial ordering over the locations where sentences containing important words occur: Optimal Position Policy (OPP)
OPP for ZIFF corpus:
(T) > (P2,S1) > (P3,S1) > (P2,S2) > {(P4,S1),(P5,S1),(P3,S2)} >…
(T=title; P=paragraph; S=sentence)OPP for Wall Street Journal: (T)>(P1,S1)>...
![Page 59: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/59.jpg)
59
Title-Based Method (1)
Claim: Words in titles and headings are positively relevant to summarization.
Shown to be statistically valid at 99% level of significance (Edmundson, 68).
Empirically shown to be useful in summarization systems.
![Page 60: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/60.jpg)
60
Cue-Phrase method (1)
Claim 1: Important sentences contain ‘bonus phrases’, such as significantly, In this paper we show, and In conclusion, while non-important sentences contain ‘stigma phrases’ such as hardly and impossible.
Claim 2: These phrases can be detected automatically (Kupiec et al. 95; Teufel and Moens 97).
Method: Add to sentence score if it contains a bonus phrase, penalize if it contains a stigma phrase.
![Page 61: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/61.jpg)
61
Cohesion-based methodsClaim: Important
sentences/paragraphs are the highest connected entities in more or less elaborate semantic structures.
Classes of approaches word co-occurrences; local salience and grammatical relations; co-reference; lexical similarity (WordNet, lexical
chains); combinations of the above.
![Page 62: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/62.jpg)
62
Cohesion: WORD co-occurrence (1)
Apply IR methods at the document level: texts are collections of paragraphs (Salton et al., 94; Mitra et al., 97; Buckley and Cardie, 97): Use a traditional, IR-based, word similarity
measure to determine for each paragraph Pi the set Si of paragraphs that Pi is related to.
Method: determine relatedness score Si for each
paragraph, extract paragraphs with largest Si scores.
P1P2
P3
P4
P5P6
P7
P8
P9
![Page 63: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/63.jpg)
63
Word co-occurrence method (2) Cornell’s Smart-based approach
expand original query compare expanded query against paragraphs select top three paragraphs (max 25% of
original) that are most similar to the original query (SUMMAC,98): 71.9% F-score for relevance judgment
CGI/CMU approach maximize query-relevance while minimizing
redundancy with previous information.(SUMMAC,98): 73.4% F-score for relevance judgment
In the context of query-based summarization
![Page 64: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/64.jpg)
64
Cohesion: Local salience Method
Assumes that important phrasal expressions are given by a combination of grammatical, syntactic, and contextual parameters (Boguraev and Kennedy, 97):
CNTX: 50 iff the expression is in the current discourse segmentSUBJ: 80 iff the expression is a subjectEXST: 70 iff the expression is an existential constructionACC: 50 iff the expression is a direct objectHEAD: 80 iff the expression is not contained in another phraseARG: 50 iff the expression is not contained in an adjunct
![Page 65: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/65.jpg)
65
Cohesion: Lexical chains method (1)
But Mr. Kenny’s move speeded up work on a machine which uses micro-computers to control the rate at which an anaesthetic is pumpedinto the blood of patients undergoing surgery. Such machines are nothing new. But Mr. Kenny’s device uses two personal-computers to achievemuch closer monitoring of the pump feeding the anaesthetic into the patient. Extensive testing of the equipment has sufficiently impressedthe authorities which regulate medical equipment in Britain, and, so far,four other countries, to make this the first such machine to be licensedfor commercial sale to hospitals.
Based on (Morris and Hirst, 91)
![Page 66: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/66.jpg)
66
Lexical chains-based method (2)
Assumes that important sentences are those that are ‘traversed’ by strong chains (Barzilay and Elhadad, 97). Strength(C) = length(C) -
#DistinctOccurrences(C) For each chain, choose the first sentence
that is traversed by the chain and that uses a representative set of concepts from that chain.
LC algorithm Lead-based algorithm[Jing et al., 98] corpus Recall Prec Recall Prec
10% cutoff 67% 61% 82.9% 63.4%
20% cutoff 64% 47% 70.9% 46.9%
![Page 67: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/67.jpg)
67
Cohesion: Coreference method
Build co-reference chains (noun/event identity, part-whole relations) between query and document - In the context of query-based
summarization title and document sentences within document
Important sentences are those traversed by a large number of chains: a preference is imposed on chains (query > title > doc)
Evaluation: 67% F-score for relevance (SUMMAC, 98). (Baldwin and Morton, 98)
![Page 68: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/68.jpg)
68
Cohesion: Connectedness method (1)
Map texts into graphs: The nodes of the graph are the words of
the text. Arcs represent adjacency, grammatical,
co-reference, and lexical similarity-based relations.
Associate importance scores to words (and sentences) by applying the tf.idf metric.
Assume that important words/sentences are those with the highest scores.
(Mani and Bloedorn, 97)
![Page 69: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/69.jpg)
69
Connectedness method (2)
When a query is given, by applying a spreading-activation algorithms, weights can be adjusted; as a results, one can obtain query-sensitive summaries.
Evaluation (Mani and Bloedorn, 97): IR categorization task: close to full-
document categorization results.[Marcu,97] corpus TF-IDF method Spreading activation
10% cutoff F-score 25.2% 32.4%
20% cutoff F-score 35.8% 45.4%
In the context of query-based summarization
![Page 70: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/70.jpg)
70
Claim: The multi-sentence coherence structure of a text can be constructed, and the ‘centrality’ of the textual units in this structure reflects their importance.
Tree-like representation of texts in the style of Rhetorical Structure Theory (Mann and Thompson,88).
Use the discourse representation in order to determine the most important textual units. Attempts: (Ono et al., 94) for Japanese. (Marcu, 97) for English.
Discourse-based method
![Page 71: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/71.jpg)
71
Rhetorical parsing (Marcu,97)
[With its distant orbit {– 50 percent farther from the sun than Earth –} and slim atmospheric blanket,1] [Mars experiences frigid weather conditions.2] [Surface temperatures typically average about –60 degrees Celsius (–76 degrees Fahrenheit) at the equator and can dip to –123 degrees C near the poles.3] [Only the midday sun at tropical latitudes is warm enough to thaw ice on occasion,4] [but any liquid water formed that way would evaporate almost instantly5] [because of the low atmospheric pressure.6]
[Although the atmosphere holds a small amount of water, and water-ice clouds sometimes develop,7] [most Martian weather involves blowing dust or carbon dioxide.8] [Each winter, for example, a blizzard of frozen carbon dioxide rages over one pole, and a few meters of this dry-ice snow accumulate as previously frozen carbon dioxide evaporates from the opposite polar cap.9] [Yet even on the summer pole, {where the sun remains in the sky all day long,} temperatures never warm enough to melt frozen water.10]
![Page 72: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/72.jpg)
72
Rhetorical parsing (2)
Use discourse markers to hypothesize rhetorical relations rhet_rel(CONTRAST, 4, 5) rhet_rel(CONTRAT, 4, 6) rhet_rel(EXAMPLE, 9, [7,8]) rhet_rel(EXAMPLE, 10, [7,8])
Use semantic similarity to hypothesize rhetorical relations if similar(u1,u2) then
rhet_rel(ELABORATION, u2, u1) rhet_rel(BACKGROUND, u1,u2)else
rhet_rel(JOIN, u1, u2)
rhet_rel(JOIN, 3, [1,2]) rhet_rel(ELABORATION, [4,6], [1,2]) Use the hypotheses in order to derive a valid discourse
representation of the original text.
![Page 73: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/73.jpg)
73
Rhetorical parsing (3)
5Evidence
Cause
5 6
4
4 5Contrast
3
3Elaboration
1 2
2BackgroundJustification
2Elaboration
7 8
8Concession
9 10
10Antithesis
8Example
2Elaboration
Summarization = selection of the most important units
2 > 8 > 3, 10 > 1, 4, 5, 7, 9 > 6
![Page 74: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/74.jpg)
Kupiec et al. 95
Extracts of roughly 20% of original text
Feature set: sentence length
▪ |S| > 5 fixed phrases
▪ 26 manually chosen paragraph
▪ sentence position in paragraph
thematic words▪ binary: whether sentence is included in the
set of highest scoring sentences uppercase words
▪ not common acronyms Corpus:
▪ 188 document + summary pairs from scientific journals
![Page 75: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/75.jpg)
Kupiec et al. 95
Uses Bayesian classifier:
• Assuming statistical independence:
k
j j
k
j j
kFP
SsPSsFPFFFSsP
1
121
)(
)()|(),...,|(
),(
)()|,...,(),...,|(
,...21
2121
k
kk FFFP
SsPSsFFFPFFFSsP
![Page 76: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/76.jpg)
Centroid (Radev, 2004)
Centroids consist of words which are central not only to one article in a cluster, but to all the articles.
Hypothesize that sentences that contain the words from the centroid are more indicative of the topic of the cluster.
A centroid is a pseudo-document which consists of words which have Count*IDF scores above a predefined threshold in the documents that constitute the cluster. Count: average number of occurrences of a
word across the entire cluster IDF: computed from a large corpus.
![Page 77: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/77.jpg)
Centroid example
77
![Page 78: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/78.jpg)
78
Topic Signature Method (Hovy and Lin, 98)Claim: Can approximate script
identification at lexical level, using automatically acquired ‘word families’.
Idea: Create topic signatures: each concept is defined by frequency distribution of its related words (concepts):
TS = {topic, signature } = {topic, (t1,w1) (t2,w2) ...}
restaurant -visit waiter + menu + food + eat...
(inverse of query expansion in IR.)
![Page 79: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/79.jpg)
Signature term extraction
likelihood ratio (Dunning 1993) Hypothesis testing method:
79
![Page 80: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/80.jpg)
80
Information extraction Method (1)
Idea: content selection using templates Predefine a template, whose slots specify what is of
interest. Use a canonical IE system to extract from a (set of)
document(s) the relevant information; fill the template. Generate the content of the template as the summary.
Previous IE work: FRUMP (DeJong, 78): ‘sketchy scripts’ of terrorism,
natural disasters, political visits... (Mauldin, 91): templates for conceptual IR. (Rau and Jacobs, 91): templates for business. (McKeown and Radev, 95): templates for news.
![Page 81: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/81.jpg)
81
Information Extraction method (2)
Example template:
MESSAGE:ID TSL-COL-0001SECSOURCE:SOURCE ReutersSECSOURCE:DATE 26 Feb 93
Early afternoonINCIDENT:DATE 26 Feb 93INCIDENT:LOCATION World Trade CenterINCIDENT:TYPE BombingHUM TGT:NUMBER AT LEAST 5
![Page 82: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/82.jpg)
82
Finally: Combining the Evidence
Problem: which extraction methods to believe?
Answer: assume they are independent, and combine their evidence: merge individual sentence scores.
Studies: (Kupiec et al., 95; Aone et al., 97, Teufel and
Moens, 97): Bayes’ Rule. (Mani and Bloedorn,98): SCDF, C4.5, inductive
learning. (Lin and Hovy, 98b): C4.5. (Marcu, 98): rhetorical parsing tuning.
![Page 83: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/83.jpg)
83
And Now, an Example...
![Page 84: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/84.jpg)
84
Example System: SUMMARISTThree stages: (Hovy and Lin, 98)
1. Topic Identification Modules: Positional Importance, Cue Phrases (under construction), Word Counts, Discourse Structure (under construction), ...
2. Topic Interpretation Modules: Concept Counting /Wavefront, Concept Signatures (being extended)
3. Summary Generation Modules (not yet built): Keywords, Template Gen, Sent. Planner & Realizer
SUMMARY = TOPIC ID + INTERPRETATION + GENERATION
![Page 85: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/85.jpg)
85
Internal Format: Preamble
<*docno = AP890417-0167>
<*title = "Former Hostage Accuses Britain of Weakness .">
<*module = PRE|POS|MPH|FRQ|IDF|SIG|CUE|OPP>
<*freq = 544,471,253>
<*tfidf_keywords = france,13.816|holding,9.210|hostage,8.613|iranian,8.342|television,8.342|writer,7.927|release,7.532|negotiate,7.395|germany, ...>
<*signature = #4,0.577|#2,0.455|#6,0.387>
<*sig_keywords = hostage,0.725|hold,0.725|western,0.725|moslem,0.725|iranian,0.725|release,0.725|middle,0.725|kill,0.725|west,0.725|march,0.725|east,0.725|syrian, ...>
<*opp_rule = p:0,1|1,2|2,3|3,4|4,4 s:-,->
<*opp_keywords = kauffmann,4.578|release,3.866|britain,3.811|mccarthy,3.594|hostages,3.406|british,3.150|hostage,2.445|french,2.164|negotiate,2.161| ...>
![Page 86: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/86.jpg)
86
Internal Format: Word-by-WordFormer <pno=1 sno=1 pos=JJ cwd=1 mph=- frq=1 tfidf=0.000
sig=-,-|-,-|-,- cue=0,- opp=-,->
hostage <pno=1 sno=1 pos=NN cwd=0 mph=- frq=6 tfidf=8.613 sig=1,12.169|33,1.370|2,5.791 cue=0,- opp=2.445,0.898>
John-Paul <pno=1 sno=1 pos=NNP cwd=0 mph=- frq=1 tfidf=0.000 sig=-,-|-,-|-,- cue=0,- opp=0.898,0.898>
Kauffmann <pno=1 sno=1 pos=NNP cwd=0 mph=- frq=6 tfidf=0.000 sig=-,-|-,-|-,- cue=0,- opp=4.578,0.898>
on <pno=1 sno=1 pos=IN cwd=1 mph=- frq=4 tfidf=0.000 sig=-,-|-,-|-,- cue=0,- opp=-,->
Monday <pno=1 sno=1 pos=NNP cwd=0 mph=- frq=3 tfidf=0.000 sig=-,-|-,-|-,- cue=0,- opp=2.076,0.898>
urged <pno=1 sno=1 pos=VBD cwd=0 mph=urge frq=1 tfidf=0.000 sig=-,-|-,-|274,0.492 cue=0,- opp=0.898,0.898>
![Page 87: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/87.jpg)
87
Example Output, with Keywords
OPP tf.idfsignature
<QNUM>138</QNUM><DOCNO>AP890417-0167</DOCNO><TITLE>Former Hostage Accuses Britain of Weakness </TITLE><TEXT>Former hostage John-Paul Kauffmann on Monday urged Britain to follow the example set by France and West Germany and negotiate the release of its citizens held captive in Lebanon .Kauffmann said Britain `` has abandoned '' John McCarthy , 32 , a television reporter abducted on his way to Beirut...Keywords:western moslem iranian middle kill march east syrian free anderson group palestinian </TEXT></DOC>
![Page 88: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/88.jpg)
88
Summarization exercise
Write a one-sentence summary for each of the following texts.
![Page 89: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/89.jpg)
89
Flu stopperA new compound is set for human testing (Times)
Running nose. Raging fever. Aching joints. Splitting headache. Are there any poor souls suffering from the flu this winter who haven’t longed for a pill to make it all go away? Relief may be in sight. Researchers at Gilead Sciences, a pharmaceutical company in Foster City, California, reported last week in the Journal of the American Chemical Society that they have discovered a compound that can stop the influenza virus from spreading in animals. Tests on humans are set for later this year.The new compound takes a novel approach to the familiar flu virus. It targets an enzyme,
called neuraminidase, that the virus needs in order to scatter copies of itself throughout thebody. This enzyme acts like a pair of molecular scissors that slices through the protectivemucous linings of the nose and throat. After the virus infects the cells of the respiratorysystem and begins replicating, neuraminidase cuts the newly formed copies free to invadeother cells. By blocking this enzyme, the new compound, dubbed GS 4104, prevents theinfection from spreading.
![Page 90: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/90.jpg)
90
Plant matters How do you regulate an herb? (Scientific American)
If Harlan Page Hubbard were alive, he might be the president of a dietary supplementscompany. In the late 19th century Hubbard sold Lydia E. Pinkham’s Vegetable Compoundfor kidney and sexual problems. The renowned huckster is remembered each year by nationalconsumer and health organizations who confer a “Hubbard” – a statuette clutching a freshlemon – for the “most misleading, unfair and irresponsible advertising of the past 12 months.”
Appropriately enough, one of this year’s winners was a product that Hubbard might havepeddled alongside his Lydia Pinkham elixir. Ginkay, an extract of the herb gingko, received its lemon for advertising and labelling claims that someone ingesting the product will havea better memory. Whereas some studies have shown that gingko improves mental functioningin people with dementia, none has proved that it serves as brain tonic for healthy.
![Page 91: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/91.jpg)
91
Table of contents
1. Motivation.2. Genres and types of summaries.3. Approaches and paradigms.4. Summarization methods (& exercise).
Topic Extraction. Interpretation. Generation.
5. Evaluating summaries.6. The future.
![Page 92: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/92.jpg)
92
From extract to abstract: topic interpretation or concept fusion.
Experiment (Marcu, 98): Got 10 newspaper texts, with human abstracts. Asked 14 judges to extract corresponding
clauses from texts, to cover the same content. Compared word lengths of extracts to
abstracts: extract_length 2.76 abstract_length !!
xx xxx xxxx x xx xxxx xxx xx xxx xx xxxxx xxxx xx xxx xx x xxx xx xx xxx x xxx xx xxx x xx x xxxx xxxx xxxx xxxx xxxx xxxxxx xx xx xxxx x xxxxx x xx xx xxxxx x x xxxxx xxxxxx xxxxxx x xxxxxxxx xx x xxxxxxxxxx xx xx xxxxx xxx xx x xxxx xxxx xxx xxxx xx
Topic Interpretationxxx xx xxx xxxx xxxxx x xxxx x xx xxxxxx xxx xxxx xx x xxxxxx xxxx x xxx x xxxxx xx xxxxx x x xxxxxxxxx xx x xxxxxxxxxx xx xx xxxxx xxx xxxxx xx xxxx x xxxxxxx xxxxx x
![Page 93: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/93.jpg)
93
Some Types of Interpretation Concept generalization:
Sue ate apples, pears, and bananas Sue ate fruit
Meronymy replacement:Both wheels, the pedals, saddle, chain…
the bike Script identification: (Schank and
Abelson, 77)
He sat down, read the menu, ordered, ate, paid, and left He ate at the restaurant
Metonymy:A spokesperson for the US Government
announced that… Washington announced that...
![Page 94: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/94.jpg)
94
General Aspects of Interpretation Interpretation occurs at the conceptual
level...…words alone are polysemous (bat animal and
sports instrument) and combine for meaning (alleged murderer murderer).
For interpretation, you need world knowledge...…the fusion inferences are not in the text!
Little work so far: (Lin, 95; McKeown and Radev, 95; Reimer and Hahn, 97; Hovy and Lin, 98).
![Page 95: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/95.jpg)
95
Template-based operations
Claim: Using IE systems, can aggregate templates by detecting interrelationships.
1. Detect relationships (contradictions, changes of perspective, additions, refinements, agreements, trends, etc.).
2. Modify, delete, aggregate templates using rules (McKeown and Radev, 95):
Given two templates,if (the location of the incident is the same and
the time of the first report is before the time of the second report and
the report sources are different and at least one slot differs in value)then combine the templates using a contradiction
operator.
![Page 96: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/96.jpg)
96
Claim: Can perform concept generalization, using WordNet (Lin, 95).
Find most appropriate summarizing concept:
Concept Generalization: Wavefront
Cash register
Mainframe
Dell Mac IBM
Computer
Calculator
18
65
20
5
0
2
20
PC
1. Count word occurrences in text; score WN concs
2. Propagate scores upward3. R Max{scores} / scores
4. Move downward until no obvious child: R<Rt
5. Output that concept
![Page 97: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/97.jpg)
97
Wavefront Evaluation 200 BusinessWeek articles about
computers: typical length 750 words (1 page). human abstracts, typical length 150
words (1 par). several parameters; many variations
tried. Rt = 0.67; StartDepth = 6; Length =
20%:
Conclusion: need more elaborate taxonomy.
Random Wavefront
Precision 20.30% 33.80%Recall 15.70% 32.00%
![Page 98: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/98.jpg)
98
Inferences in terminological Logic ‘Condensation’ operators (Reimer and Hahn,
97).
1. Parse text, incrementally build a terminological rep.
2. Apply condensation operators to determine the salient concepts, relationships, and properties for each paragraph (employ frequency counting and other heuristics on concepts and relations, not on words).
3. Build a hierarchy of topic descriptions out of salient constructs.
Conclusion: No evaluation.
![Page 99: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/99.jpg)
99
Topic Signatures (1)Claim: Can approximate script
identification at lexical level, using automatically acquired ‘word families’ (Hovy and Lin, 98).
Idea: Create topic signatures: each concept is defined by frequency distribution of its related words (concepts):
signature = {head (c1,f1) (c2,f2) ...}
restaurant waiter + menu + food + eat...
(inverse of query expansion in IR.)
![Page 100: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/100.jpg)
100
Example SignaturesRANKaerospace banking environment telecommunication
1 contract bank epa at&t2 air_force thrift waste network3 aircraft banking environmental fcc4 navy loan water cbs5 army mr. ozone6 space deposit state bell7 missile board incinerator long-distance8 equipment fslic agency telephone9 mcdonnell fed clean telecommunication
10 northrop institution landfill mci11 nasa federal hazardous mr.12 pentagon fdic acid_rain doctrine13 defense volcker standard service14 receive henkel federal news15 boeing banker lake turner16 shuttle khoo garbage station17 airbus asset pollution nbc18 douglas brunei city sprint19 thiokol citicorp law communication20 plane billion site broadcasting21 engine regulator air broadcast22 million national_bankprotection programming23 aerospace greenspan violation television24 corp. financial management abc25 unit vatican reagan rate
![Page 101: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/101.jpg)
101
Topic Signatures (2)Experiment: created 30 signatures
from 30,000 Wall Street Journal texts, 30 categories: Used tf.idf to determine uniqueness in
category. Collected most frequent 300 words per
term.Evaluation: classified 2204 new
texts: Created document signature and
matched against all topic signatures; selected best match.
Results: Precision 69.31%; Recall 75.66% 90%+ for top 1/3 of categories; rest
lower, because less clearly delineated (overlapping signatures).
![Page 102: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/102.jpg)
102
Table of contents
1. Motivation.2. Genres and types of summaries.3. Approaches and paradigms.4. Summarization methods (& exercise).
Topic Extraction. Interpretation. Generation.
5. Evaluating summaries.6. The future.
![Page 103: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/103.jpg)
103
NL Generation for Summaries Level 1: no separate generation
Produce extracts, verbatim from input text.
Level 2: simple sentences Assemble portions of extracted clauses
together.
Level 3: full NLG 1. Sentence Planner: plan sentence content,
sentence length, theme, order of constituents, words chosen... (Hovy and Wanner, 96)
2. Surface Realizer: linearize input grammatically (Elhadad, 92; Knight and Hatzivassiloglou, 95).
![Page 104: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/104.jpg)
104
Full Generation Example
Challenge: Pack content densely!
Example (McKeown and Radev, 95): Traverse templates and assign values to
‘realization switches’ that control local choices such as tense and voice.
Map modified templates into a representation of Functional Descriptions (input representation to Columbia’s NL generation system FUF).
FUF maps Functional Descriptions into English.
![Page 105: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/105.jpg)
105
Generation Example (McKeown and Radev, 95)
NICOSIA, Cyprus (AP) – Two bombs exploded near government ministries in Baghdad, but there was no immediate word of any casualties, Iraqi dissidents reported Friday. There was no independentconfirmation of the claims by the Iraqi National Congress. Iraq’sstate-controlled media have not mentioned any bombings.
Multiple sources and disagreement
Explicit mentioning of “no information”.
![Page 106: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/106.jpg)
106
Table of contents
1. Motivation.2. Genres and types of
summaries.3. Approaches and paradigms.4. Summarization methods (&
exercise).5. Evaluating summaries.6. The future.
![Page 107: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/107.jpg)
107
How can You Evaluate a Summary? When you already have a
summary… ...then you can compare a new one to it:1. choose a granularity (clause; sentence;
paragraph),2. create a similarity measure for that granularity
(word overlap; multi-word overlap, perfect match),
3. measure the similarity of each unit in the new to the most similar unit(s) in the gold standard,
4. measure Recall and Precision.
e.g., (Kupiec et al., 95).
……………..…. but when you don’t?
![Page 108: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/108.jpg)
108
Toward a Theory of EvaluationTwo Measures:
Measuring length: Number of letters? words?
Measuring information: Shannon Game: quantify information
content. Question Game: test reader’s
understanding. Classification Game: compare
classifiability.
Compression Ratio: CR = (length S) / (length T)
Retention Ratio: RR = (info in S) / (info in T)
![Page 109: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/109.jpg)
109
Compare Length and InformationCase 1: just adding
info; no special leverage from summary.
Case 2: ‘fuser’ concept(s) at knee add a lot of information.
Case 3: ‘fuser’ concepts become progressively weaker.
RR
CR
RR
CR
RR
CR
![Page 110: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/110.jpg)
110
Small Evaluation Experiment (Hovy, 98)
Can you recreate what’s in the original? the Shannon Game [Shannon 1947–50]. but often only some of it is really
important. Measure info retention (number of
keystrokes): 3 groups of subjects, each must recreate
text:▪ group 1 sees original text before starting. ▪ group 2 sees summary of original text before
starting. ▪ group 3 sees nothing before starting.
Results (# of keystrokes; two different paragraphs):
Group 1 Group 2 Group 3approx. 10 approx. 150 approx. 1100
![Page 111: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/111.jpg)
111
Q&A Evaluation
Can you focus on the important stuff? The Q&A Game—can be tailored to your interests!
Measure core info. capture by Q&A game: Some people (questioners) see text, must create
questions about most important content. Other people (answerers) see:
1. nothing—but must try to answer questions (baseline),2. then: summary, must answer same questions,3. then: full text, must answer same questions again.
Information retention: % answers correct.
![Page 112: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/112.jpg)
112
How can You Evaluate a Summary?
When you generate a summary… ..…:1. the gold standard summaries (human),2. choose a granularity (clause; sentence;
paragraph),3. create a similarity measure for that
granularity (word overlap; multi-word overlap, perfect match),
4. measure the similarity of each unit in the new to the most similar unit(s) in the gold standard,
5. measure Recall and Precision.
e.g., (Kupiec et al., 95).
![Page 113: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/113.jpg)
Human summaries
Two persons A and B extract sentences from documents as summaries
Their agreement? Kappa value
▪ ≥0.75 Excellent, 0.40 to 0.75 Fair to good, < 0.40 as Poor
▪ 0 to 0.20 Slight, >0.20 to 0.40 Fair, >0.40 to 0.60 Moderate, >0.60 to 0.80 Substantial, >0.80 Almost perfect
113
![Page 114: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/114.jpg)
Kappa
P(A): the observed agreement among raters
P(E): the expectation probability of chance agreement
)(1
)()(
EP
EPAP
![Page 115: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/115.jpg)
B
Yes No
AYes 20 5
No 10 15
115
P(A) = (20 + 15) / 50 = 0.70P(E) = 0.5*0.6+ 0.5* 0.4 = 0.3 + 0.2 = 0.5
)(1
)()(
EP
EPAP
![Page 116: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/116.jpg)
Evaluation Manual
Linguistic Quality(readability)▪ Grammaticality▪ Non-redundancy▪ Referential clarity▪ Focus▪ Structure
Five-point scale (1 very poor, 5 very good)
Pyramid: SCU
Automatic Rouge
▪ ROUGE 2▪ ROUGE SU4
4
3
21
– Responsiveness(content)
– BE(basic element)
![Page 117: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/117.jpg)
Pyramid(1)
The pyramid method is designed to address the observation: summaries from different humans always have partly overlapping content.
The pyramid method includes a manual annotation method to represent Summary Content Units (SCUs) and to quantify the proportion of model summaries that express this content.
All SCUs have a weight representing the number of models they occur in, thus from 1 to maxn, where maxn is the total number of models There are very few SCUs expressed in all models (i.e.,
weight=maxn), and increasingly many SCUs at each lower weight, with the most SCUs at weight=1.
1
2
34
![Page 118: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/118.jpg)
SCU example
118
![Page 119: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/119.jpg)
Pyramid(2)
The approach involves two phases of manual annotation: pyramid construction annotation against the pyramid to
determine which SCUs in the pyramid have been expressed in the peer summary.
The total weight is
![Page 120: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/120.jpg)
![Page 121: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/121.jpg)
ROUGE basics
Rouge(Recall-Oriented Understudy for Gisting Evaluation) Recall-oriented, within-sentence word overlap with model(s)
Models - no theoretical limit to number compared system output to 4 models compared manual summaries to 3 models
Using n-gram Correlate reasonably with human coverage judgements Not address summary discourse characteristics, and suffer
from lack of text cohesion or coherence ROUGE v1.2.1 measures
ROUGE-1,2,3,4: N-gram matching where N = 1,2,3,4 ROUGE-LCS: Longest common substring
![Page 122: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/122.jpg)
ROUGE: Recall-Oriented Understudy for Gisting Evaluation
Rouge – Ngram co-occurrence metrics measuring content overlap
Counts of n-gram overlaps between candidate and model summaries
Total n-grams in summary model
![Page 123: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/123.jpg)
ROUGE
Where Nn represents the set of all n-grams and i is one member from Nn. Xn(i) is the number of times the n-gram i occurred in the summary and Mn(i,j) is the number of times the n-gram i ocurred in the j-th model reference(human) summary. There are totally h human summaries.
1
1
min( ( ), ( , ))( )
( , )n
n
h
n nj i Nn h
nj i N
X i M i jR X
M i j
![Page 124: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/124.jpg)
Example
peer: A B C D E F G A B H1: A B G A D E C D (7) H2: A C E F G A D (6)
How to compute the ROUGE-2 score?
124
![Page 125: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/125.jpg)
125
Table of contents
1. Motivation.2. Genres and types of
summaries.3. Approaches and paradigms.4. Summarization methods .5. Evaluating summaries.6. The future.
![Page 126: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/126.jpg)
In the last decade there has been a surge of interest in automatic summarizing. … There has been some progress, but there is much to do.
Karen Sparck Jones
Karen Sparck Jones, automatic summarising: the state of the art, Information Processing & Management, 43(6): 1449--1481, 2007.
![Page 127: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/127.jpg)
127
The Future (1) — There’s much to do!
Data preparation: Collect large sets of texts with abstracts, all
genres. Build large corpora of <Text, Abstract, Extract>
tuples. Investigate relationships between extracts and
abstracts (using <Extract, Abstract> tuples). Topic Identification:
Develop new identification methods (discourse, etc.).
Develop heuristics for method combination (train heuristics on <Text, Extract> tuples).
![Page 128: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/128.jpg)
128
The Future (2)
Concept Interpretation (Fusion): Investigate types of fusion (semantic, evaluative…). Create large collections of fusion knowledge/rules. Study incorporation of User’s knowledge in
interpretation. Generation:
Develop sentence generation rules (using <Extract, Abstract> pairs).
Evaluation: Develop better automatic evaluation metrics.
![Page 129: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/129.jpg)
129
SUMMAC Q&A EvaluationProcedure (SUMMAC,
98):1. Testers create questions for
each category.2. Systems create summaries,
not knowing questions.3. Humans answer questions
from originals and from summaries.
4. Testers measure answer Recall: how many questions can be answered correctly from the summary?(many other measures as well)
• Results:Large variation by topic,
even within systems...
Normalized Answer Recall
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
1 2 3
Topic number
![Page 130: CS276B Web Search and Mining Lecture 14 Text Mining II (includes slides borrowed from G. Neumann, M. Venkataramani, R. Altman, L. Hirschman, and.](https://reader036.fdocuments.in/reader036/viewer/2022081516/56649de45503460f94adb8f4/html5/thumbnails/130.jpg)
130
Task Evaluation: Text Classification
Can you perform some task faster? example: the Classification Game. measures: time and effectiveness.
TIPSTER/SUMMAC evaluation: February, 1998 (SUMMAC, 98). Two tests: 1. Categorization 2. Ad Hoc (query-sensitive) 2 summaries per system: fixed-length
(10%), best. 16 systems (universities, companies; 3
intern’l).