2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley...
-
date post
20-Dec-2015 -
Category
Documents
-
view
213 -
download
0
Transcript of 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley...
![Page 1: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/1.jpg)
2007.03.13 - SLIDE 1IS 240 – Spring 2007
Prof. Ray Larson University of California, Berkeley
School of InformationTuesday and Thursday 10:30 am - 12:00 pm
Spring 2007http://courses.ischool.berkeley.edu/i240/s07
Principles of Information Retrieval
Lecture 15: IR Components 1
![Page 2: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/2.jpg)
2007.03.13 - SLIDE 2IS 240 – Spring 2007
Overview
• Review– Evaluation
• IR Components– Text processing– Stemming– Mutual Information
![Page 3: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/3.jpg)
2007.03.13 - SLIDE 3IS 240 – Spring 2007
Today
• Components of IR systems– Content Analysis– Stemming
• Statistical Properties of Document collections– Statistical Dependence– Word Associations
![Page 4: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/4.jpg)
2007.03.13 - SLIDE 4IS 240 – Spring 2007
Content Analysis
• Automated Transformation of raw text into a form that represent some aspect(s) of its meaning
• Including, but not limited to:– Automated Thesaurus Generation– Phrase Detection– Categorization– Clustering– Summarization
![Page 5: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/5.jpg)
2007.03.13 - SLIDE 5IS 240 – Spring 2007
Techniques for Content Analysis
• Statistical– Single Document– Full Collection
• Linguistic– Syntactic– Semantic– Pragmatic
• Knowledge-Based (Artificial Intelligence)
• Hybrid (Combinations)
![Page 6: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/6.jpg)
2007.03.13 - SLIDE 6IS 240 – Spring 2007
Text Processing
• Standard Steps:– Recognize document structure
• titles, sections, paragraphs, etc.
– Break into tokens• usually space and punctuation delineated• special issues with Asian languages
– N-grams
– Stemming/morphological analysis– Store in inverted index
![Page 7: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/7.jpg)
2007.03.13 - SLIDE 7
Document Processing Steps
![Page 8: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/8.jpg)
2007.03.13 - SLIDE 8IS 240 – Spring 2007
Stemming and Morphological Analysis
• Goal: “normalize” similar words
• Morphology (“form” of words)– Inflectional Morphology
• E.g,. inflect verb endings and noun number• Never change grammatical class
– dog, dogs– tengo, tienes, tiene, tenemos, tienen
– Derivational Morphology • Derive one word from another, • Often change grammatical class
– build, building; health, healthy
![Page 9: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/9.jpg)
2007.03.13 - SLIDE 9IS 240 – Spring 2007
Statistical Properties of Text
• Token occurrences in text are not uniformly distributed
• They are also not normally distributed
• They do exhibit a Zipf distribution
![Page 10: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/10.jpg)
2007.03.13 - SLIDE 10IS 240 – Spring 2007
Plotting Word Frequency by Rank
• Main idea: count– How many tokens occur 1 time – How many tokens occur 2 times– How many tokens occur 3 times …
• Now rank these according to how of they occur. This is called the rank.
![Page 11: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/11.jpg)
2007.03.13 - SLIDE 11IS 240 – Spring 2007
Plotting Word Frequency by Rank
• Say for a text with 100 tokens• Count
– How many tokens occur 1 time (50)– How many tokens occur 2 times (20) …– How many tokens occur 7 times (10) … – How many tokens occur 12 times (1)– How many tokens occur 14 times (1)
• So things that occur the most often share the highest rank (rank 1).
• Things that occur the fewest times have the lowest rank (rank n).
![Page 12: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/12.jpg)
2007.03.13 - SLIDE 12IS 240 – Spring 2007
Many similar distributions…
• Words in a text collection
• Library book checkout patterns
• Bradford’s and Lotka’s laws.
• Incoming Web Page Requests (Nielsen)
• Outgoing Web Page Requests (Cunha & Crovella)
• Document Size on Web (Cunha & Crovella)
![Page 13: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/13.jpg)
2007.03.13 - SLIDE 13
Zipf Distribution(linear and log scale)
![Page 14: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/14.jpg)
2007.03.13 - SLIDE 14IS 240 – Spring 2007
Zipf Distribution
• The product of the frequency of words (f) and their rank (r) is approximately constant– Rank = order of words’ frequency of occurrence
• Another way to state this is with an approximately correct rule of thumb:– Say the most common term occurs C times– The second most common occurs C/2 times– The third most common occurs C/3 times– …
10/
/1
NC
rCf
![Page 15: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/15.jpg)
2007.03.13 - SLIDE 15IS 240 – Spring 2007
Zipf Distribution
• The Important Points:– a few elements occur very frequently– a medium number of elements have medium
frequency– many elements occur very infrequently
![Page 16: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/16.jpg)
2007.03.13 - SLIDE 16IS 240 – Spring 2007
150 2 enhanc151 2 energi152 2 emphasi153 2 detect154 2 desir155 2 date156 2 critic157 2 content158 2 consider159 2 concern160 2 compon161 2 compar162 2 commerci163 2 clause164 2 aspect165 2 area166 2 aim167 2 affect
Most and Least Frequent Terms
Rank Freq Term1 37 system2 32 knowledg3 24 base4 20 problem5 18 abstract6 15 model7 15 languag8 15 implem9 13 reason10 13 inform11 11 expert12 11 analysi13 10 rule14 10 program15 10 oper16 10 evalu17 10 comput18 10 case19 9 gener20 9 form
![Page 17: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/17.jpg)
2007.03.13 - SLIDE 17IS 240 – Spring 2007
Rank Freq1 37 system2 32 knowledg3 24 base4 20 problem5 18 abstract6 15 model7 15 languag8 15 implem9 13 reason10 13 inform11 11 expert12 11 analysi13 10 rule14 10 program15 10 oper16 10 evalu17 10 comput18 10 case19 9 gener20 9 form
The Corresponding Zipf Curve
![Page 18: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/18.jpg)
2007.03.13 - SLIDE 18IS 240 – Spring 2007
Zoom in on the Knee of the Curve
43 6 approach44 5 work45 5 variabl46 5 theori47 5 specif48 5 softwar49 5 requir50 5 potenti51 5 method52 5 mean53 5 inher54 5 data55 5 commit56 5 applic57 4 tool58 4 technolog59 4 techniqu
![Page 19: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/19.jpg)
2007.03.13 - SLIDE 19IS 240 – Spring 2007
A Standard Collection
8164 the4771 of4005 to2834 a2827 and2802 in1592 The1370 for1326 is1324 s1194 that 973 by
969 on 915 FT 883 Mr 860 was 855 be 849 Pounds 798 TEXT 798 PUB 798 PROFILE 798 PAGE 798 HEADLINE 798 DOCNO
1 ABC 1 ABFT 1 ABOUT 1 ACFT 1 ACI 1 ACQUI 1 ACQUISITIONS 1 ACSIS 1 ADFT 1 ADVISERS 1 AE
Government documents, 157734 tokens, 32259 unique
![Page 20: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/20.jpg)
2007.03.13 - SLIDE 20IS 240 – Spring 2007
Housing Listing Frequency Data
Histogram
0
50
100
150
200
250
300
350
Bin
Freq
uenc
y
Frequency
Bin Frequency1 295
6.72 21612.44 2818.16 723.88 29
29.6 735.32 1041.04 746.76 1452.48 2
58.2 2663.92 969.64 175.36 181.08 0
86.8 292.52 098.24 0
103.96 0109.68 0
115.4 0121.12 1126.84 1132.56 1138.28 0
More 1
6208 tokens, 1318 unique (very small collection)
![Page 21: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/21.jpg)
2007.03.13 - SLIDE 21IS 240 – Spring 2007
Very frequent word stems (Cha-Cha Web Index of berkeley.edu domain)
WORD FREQu 63245ha 65470california 67251m 67903
1998 68662system 69345t 70014about 70923servic 71822work 71958home 72131other 72726research 74264
1997 75323can 76762next 77973your 78489all 79993public 81427us 82551c 83250www 87029wa 92384program 95260
not 100204http 100696d 101034html 103698student 104635univers 105183inform 106463will 109700new 115937have 119428page 128702messag 141542from 147440you 162499edu 167298be 185162publib 189334librari 189347i 190635lib 223851that 227311s 234467berkelei 245406re 272123web 280966archiv 305834
![Page 22: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/22.jpg)
2007.03.13 - SLIDE 22IS 240 – Spring 2007
Words that occur few times (Cha-Cha Web Index)
WORD FREQagendaaugust 1anelectronic 1centerjanuary 1packardequipment 1systemjuly 1systemscs186 1todaymcb 1workshopsfinding 1workshopsthe 1lollini 10+ 1
0 100summary 1
35816 135823 1
01d 135830 135837 1
02-156-10 135844 135851 1
02aframst 1311 1313 1
03agenvchm 1401 1408 1
408 1422 1424 1429 1
04agrcecon 104cklist 105-128-10 1
501 1506 1
05amstud 106anhist 107-149 107-800-80 107anthro 108apst 1
![Page 23: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/23.jpg)
2007.03.13 - SLIDE 23IS 240 – Spring 2007
Resolving Power (van Rijsbergen 79)
The most frequent words are not the most descriptive.
![Page 24: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/24.jpg)
2007.03.13 - SLIDE 24IS 240 – Spring 2007
Other Models
• Poisson distribution
• 2-Poisson Model
• Negative Binomial
• Katz K-mixture– See Church (SIGIR 1995)
![Page 25: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/25.jpg)
2007.03.13 - SLIDE 25IS 240 – Spring 2007
Stemming and Morphological Analysis
• Goal: “normalize” similar words
• Morphology (“form” of words)– Inflectional Morphology
• E.g,. inflect verb endings and noun number• Never change grammatical class
– dog, dogs– tengo, tienes, tiene, tenemos, tienen
– Derivational Morphology • Derive one word from another, • Often change grammatical class
– build, building; health, healthy
![Page 26: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/26.jpg)
2007.03.13 - SLIDE 26IS 240 – Spring 2007
Simple “S” stemming
• IF a word ends in “ies”, but not “eies” or “aies”– THEN “ies” “y”
• IF a word ends in “es”, but not “aes”, “ees”, or “oes”– THEN “es” “e”
• IF a word ends in “s”, but not “us” or “ss”– THEN “s” NULL
Harman, JASIS Jan. 1991
![Page 27: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/27.jpg)
2007.03.13 - SLIDE 27IS 240 – Spring 2007
Stemmer Examples
The SMART stemmer The Porter stemmer The IAGO! stemmer% tstem ateate% tstem applesappl% tstem formulaeformul% tstem appendicesappendix% tstem implementationimple% tstem glassesglass%
% pstemmer ateat% pstemmer applesappl% pstemmer formulaeformula% pstemmer appendicesappendic% pstemmer implementationimplement% pstemmer glassesglass%
% stemate|2eat|2apples|1apple|1formulae|1formula|1appendices|1appendix|1implementation|1implementation|1glasses|1glasses|1%
![Page 28: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/28.jpg)
2007.03.13 - SLIDE 28IS 240 – Spring 2007
Errors Generated by Porter Stemmer (Krovetz 93)
Too Aggressive Too Timidorganization/ organ european/ europe
policy/ police cylinder/ cylindrical
execute/ executive create/ creation
arm/ army search/ searcher
![Page 29: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/29.jpg)
2007.03.13 - SLIDE 29IS 240 – Spring 2007
Automated Methods
• Stemmers:– Very dumb rules work well (for English)– Porter Stemmer: Iteratively remove suffixes– Improvement: pass results through a lexicon
• Newer stemmers are configurable (Snowball)– Demo…
• Powerful multilingual tools exist for morphological analysis– PCKimmo, Xerox Lexical technology– Require a grammar and dictionary– Use “two-level” automata– Wordnet “morpher”
![Page 30: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/30.jpg)
2007.03.13 - SLIDE 30IS 240 – Spring 2007
Wordnet
• Type “wn word” on irony…
• Large exception dictionary:
• Demo
aardwolves aardwolf abaci abacus abacuses abacus abbacies abbacy abhenries abhenry abilities ability abkhaz abkhaz abnormalities abnormality aboideaus aboideau aboideaux aboideau aboiteaus aboiteau aboiteaux aboiteau abos abo abscissae abscissa abscissas abscissa absurdities absurdity…
![Page 31: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/31.jpg)
2007.03.13 - SLIDE 31IS 240 – Spring 2007
Using NLP
• Strzalkowski (in Reader)
Text NLP represDbasesearch
TAGGERNLP: PARSER TERMS
![Page 32: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/32.jpg)
2007.03.13 - SLIDE 32IS 240 – Spring 2007
Using NLP
INPUT SENTENCEThe former Soviet President has been a local hero ever sincea Russian tank invaded Wisconsin.
TAGGED SENTENCEThe/dt former/jj Soviet/jj President/nn has/vbz been/vbn a/dt local/jj hero/nn ever/rb since/in a/dt Russian/jj tank/nn invaded/vbd Wisconsin/np ./per
![Page 33: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/33.jpg)
2007.03.13 - SLIDE 33IS 240 – Spring 2007
Using NLP
TAGGED & STEMMED SENTENCEthe/dt former/jj soviet/jj president/nn have/vbz be/vbn a/dt local/jj hero/nn ever/rb since/in a/dt russian/jj tank/nn invade/vbd wisconsin/np ./per
![Page 34: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/34.jpg)
2007.03.13 - SLIDE 34IS 240 – Spring 2007
Using NLP
PARSED SENTENCE
[assert
[[perf [have]][[verb[BE]]
[subject [np[n PRESIDENT][t_pos THE]
[adj[FORMER]][adj[SOVIET]]]]
[adv EVER]
[sub_ord[SINCE [[verb[INVADE]]
[subject [np [n TANK][t_pos A]
[adj [RUSSIAN]]]]
[object [np [name [WISCONSIN]]]]]]]]]
![Page 35: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/35.jpg)
2007.03.13 - SLIDE 35IS 240 – Spring 2007
Using NLP
EXTRACTED TERMS & WEIGHTS
President 2.623519 soviet 5.416102
President+soviet 11.556747 president+former 14.594883
Hero 7.896426 hero+local 14.314775
Invade 8.435012 tank 6.848128
Tank+invade 17.402237 tank+russian 16.030809
Russian 7.383342 wisconsin 7.785689
![Page 36: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/36.jpg)
2007.03.13 - SLIDE 36IS 240 – Spring 2007
Same Sentence, different sys
Enju ParserROOT ROOT ROOT ROOT -1 ROOT been be VBN VB 5been be VBN VB 5 ARG1 President president NNP NNP 3been be VBN VB 5 ARG2 hero hero NN NN 8a a DT DT 6 ARG1 hero hero NN NN 8a a DT DT 11 ARG1 tank tank NN NN 13local local JJ JJ 7 ARG1 hero hero NN NN 8The the DT DT 0 ARG1 President president NNP NNP 3former former JJ JJ 1 ARG1 President president NNP NNP 3Russian russian JJ JJ 12 ARG1 tank tank NN NN 13Soviet soviet NNP NNP 2 MOD President president NNP NNP 3invaded invade VBD VB 14 ARG1 tank tank NN NN 13invaded invade VBD VB 14 ARG2 Wisconsin wisconsin NNP NNP 15has have VBZ VB 4 ARG1 President president NNP NNP 3has have VBZ VB 4 ARG2 been be VBN VB 5since since IN IN 10 MOD been be VBN VB 5since since IN IN 10 ARG1 invaded invade VBD VB 14ever ever RB RB 9 ARG1 since since IN IN 10
![Page 37: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/37.jpg)
2007.03.13 - SLIDE 37IS 240 – Spring 2007
Other Considerations
• Church (SIGIR 1995) looked at correlations between forms of words in texts
hostages nullhostage 619(a) 479(b)null 648(c) 78223(d)
![Page 38: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/38.jpg)
2007.03.13 - SLIDE 38IS 240 – Spring 2007
Assumptions in IR
• Statistical independence of terms
• Dependence approximations
![Page 39: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/39.jpg)
2007.03.13 - SLIDE 39IS 240 – Spring 2007
Statistical Independence
Two events x and y are statistically independent if the product of their probability of their happening individually equals their probability of happening together.
),()()( yxPyPxP
![Page 40: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/40.jpg)
2007.03.13 - SLIDE 40IS 240 – Spring 2007
Statistical Independence and Dependence
• What are examples of things that are statistically independent?
• What are examples of things that are statistically dependent?
![Page 41: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/41.jpg)
2007.03.13 - SLIDE 41IS 240 – Spring 2007
Statistical Independence vs. Statistical Dependence
• How likely is a red car to drive by given we’ve seen a black one?
• How likely is the word “ambulence” to appear, given that we’ve seen “car accident”?
• Color of cars driving by are independent (although more frequent colors are more likely)
• Words in text are not independent (although again more frequent words are more likely)
![Page 42: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/42.jpg)
2007.03.13 - SLIDE 42IS 240 – Spring 2007
Lexical Associations
• Subjects write first word that comes to mind– doctor/nurse; black/white (Palermo & Jenkins 64)
• Text Corpora yield similar associations• One measure: Mutual Information (Church and
Hanks 89)
• If word occurrences were independent, the numerator and denominator would be equal (if measured across a large collection)
)(),(
),(log),( 2 yPxP
yxPyxI
![Page 43: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/43.jpg)
2007.03.13 - SLIDE 43IS 240 – Spring 2007
Interesting Associations with “Doctor”
I(x,y) f(x,y) f(x) x f(y) y11.3 12 111 Honorary 621 Doctor
11.3 8 1105 Doctors 44 Dentists
10.7 30 1105 Doctors 241 Nurses
9.4 8 1105 Doctors 154 Treating
9.0 6 275 Examined 621 Doctor
8.9 11 1105 Doctors 317 Treat
8.7 25 621 Doctor 1407 Bills
(AP Corpus, N=15 million, Church & Hanks 89)
![Page 44: 2007.03.13 - SLIDE 1IS 240 – Spring 2007 Prof. Ray Larson University of California, Berkeley School of Information Tuesday and Thursday 10:30 am - 12:00.](https://reader035.fdocuments.in/reader035/viewer/2022062714/56649d4d5503460f94a2c37f/html5/thumbnails/44.jpg)
2007.03.13 - SLIDE 44IS 240 – Spring 2007
I(x,y) f(x,y) f(x) x f(y) y0.96 6 621 doctor 73785 with
0.95 41 284690 a 1105 doctors
0.93 12 84716 is 1105 doctors
These associations were likely to happen because the non-doctor words shown here are very commonand therefore likely to co-occur with any noun.
Un-Interesting Associations with “Doctor”