Introduction to Information Retrieval Introduction toInformation RetrievalCS276: Information Retrieval and Web SearchChristopher Manning and Pandu a!a"#ecture $%: #atent Semantic Inde&ingIntroduction to Information Retrieval 'oda!(s topic#atent Semantic Inde&ing'erm)document matrices are ver! large*ut the number of topics that people tal" about is small +in some sense,Clothes- movies- politics- .Can /e represent the term)document space b! a lo/er dimensional latent space0Ch1 $2Introduction to Information Retrieval #inear 3lgebra *ac"groundIntroduction to Information Retrieval 4igenvalues 5 4igenvectorsEigenvectors +for a s6uare mm matri& S,7o/ man! eigenvalues are there at most0only has a non-zero solution if This is a mth order equation inwhich can have at most m distinct solutions (roots of the characteristic polynomial) can be complex even though S is real.eigenvalue (right) eigenvector4&ampleSec1 $21$Introduction to Information Retrieval Matri&)vector multiplicationhas eigenvalues %8- 28- $ /ithcorresponding eigenvectors9n each eigenvector- S acts as a multiple of the identit!matri&: but as a di:erent multiple on each13n! vector +sa! &; , can be vie/ed as a combination ofthe eigenvectors: & ; 2v$ < =v2 < 6v%Sec1 $21$Introduction to Information Retrieval Matri&)vector multiplication'hus a matri&)vector multiplication such as S& +S- & as in the previous slide, can be re/ritten in terms of the eigenvalues>vectors:4ven though & is an arbitrar! vector- the action of S on & is determined b! the eigenvalues>vectors1Sec1 $21$Introduction to Information Retrieval Matri&)vector multiplicationSuggestion: the e:ect of ?small@ eigenvalues is small1If /e ignored the smallest eigenvalue +$,- then instead of/e /ould get'hese vectors are similar +in cosine similarit!- etc1,Sec1 $21$Introduction to Information Retrieval 4igenvalues 5 4igenvectorsAor s!mmetric matrices- eigenvectors for distincteigenvalues are orthogonal3ll eigenvalues of a real s!mmetric matri& are real13ll eigenvalues of a positive semideBnite matri&are non-negativeSec1 $21$Introduction to Information Retrieval Plug in these values and solve for eigenvectors14&ample#et'hen'he eigenvalues are $ and % +nonnegative- real,1 'he eigenvectors are orthogonal +and real,:Real- s!mmetric1Sec1 $21$Introduction to Information Retrieval #etbe a square matri& /ith m linearly independent eigenvectors +a ?non)defective@ matri&,Theorem: 4&ists an eigen decomposition +cf1 matri& diagonaliCation theorem,Columns of U are the eigenvectors of SDiagonal elements of are eigenvalues of 4igen>diagonal DecompositiondiagonalEni6ue for distinct eigen)valuesSec1 $21$Introduction to Information Retrieval Diagonal decomposition: /h!>ho/#et U have the eigenvectors as columns:'hen- SU can be /ritten3nd S=UU1

'hus SU=U- or U1SU=Sec1 $21$Introduction to Information Retrieval Diagonal decomposition ) e&ampleRecall 'he eigenvectorsandform Inverting- /e have'hen- S=UU1 =RecallEEF$ ;$1Sec1 $21$Introduction to Information Retrieval 4&ample continued#et(s divide U +and multipl! U1, b!'hen- S=! "!-1= !T # Wh!0 Sta! tuned .Sec1 $21$Introduction to Information Retrieval Ifis a symmetric matri&:Theorem: 'here e&ists a +uni6ue, eigen decomposition/here ! is orthogonal$!-1= !TColumns of ! are normaliCed eigenvectorsColumns are orthogonal1+ever!thing is real,S!mmetric 4igen DecompositionSec1 $21$Introduction to Information Retrieval 4&ercise4&amine the s!mmetric eigen decomposition- if an!- for each of the follo/ing matrices:Sec1 $21$Introduction to Information Retrieval 'ime outGI came to this class to learn about te&t retrieval and mining- not to have m! linear algebra past dredged up again .*ut if !ou /ant to dredge- Strang(s 3pplied Mathematics is a good place to start1What do these matrices have to do /ith te&t0Recall M term)document matrices . *ut ever!thing so far needs s6uare matrices F so .Introduction to Information Retrieval Similarit! ClusteringWe can compute the similarit! bet/een t/o document vector representations &i and &H b! &i&H'#et I ; J&$ . &

K 'hen II' is a matri& of similaritiesIiH is s!mmetricSo II' ; LML'So /e can decompose this similarit! space into a set of orthonormal basis vectors +given in L, scaled b! the eigenvalues in M'his leads to PC3 +Principal Components 3nal!sis,$7Introduction to Information Retrieval Singular Nalue DecompositionMM M N is Aor an M matri& A of ran" r there e&ists afactoriCation +Singular Nalue Decomposition ; S%&,as follo/s:+ot proven here1,Sec1 $212Introduction to Information Retrieval Singular Nalue Decomposition33' ; LML'33' ; +EON',+EON',' ; +EON',+NOE', ; EO2E' MM M N is 'he columns of U are orthogonal eigenvectors of ''T1'he columns of % are orthogonal eigenvectors of 'T'1Singular values4igenvalues $ . r of ''T are the eigenvalues of 'T'1Sec1 $212Introduction to Information Retrieval Singular Nalue DecompositionIllustration of SND dimensions and sparsenessSec1 $212Introduction to Information Retrieval SND e&ample#et'hus M;%- ;21 Its SND is'!picall!- the singular values arranged in decreasing order1Sec1 $212Introduction to Information Retrieval SND can be used to compute optimal lo(-ran) appro*imations13ppro&imation problem: Aind ') of ran" ) such that3" and I are both mn matrices1'!picall!- /ant " PP r1#o/)ran" 3ppro&imationArobenius normSec1 $21%Introduction to Information Retrieval Solution via SND#o/)ran" 3ppro&imationset smallest r)"singular values to Cerocolumn notation: sum of ran" $ matrices"Sec1 $21%Introduction to Information Retrieval If /e retain onl! " singular values- and set the rest to 8- then /e don(t need the matri& parts in color'hen O is "Q"- E is MQ"- N' is "Q- and 3" is MQ 'his is referred to as the reduced SNDIt is the convenient +space)saving, and usual form for computational applicationsIt(s /hat Matlab gives !ouReduced SND"Sec1 $21%Introduction to Information Retrieval 3ppro&imation error7o/ good +bad, is this appro&imation0It(s the best possible- measured b! the Arobenius norm of the error:/here the i are ordered such that i iterms phrases F precludes simple automatic conversion of topics to latent semantic space1*oolean 6ueries3s usual- freete&t>vector space s!nta& of #SI 6ueries precludes +sa!, ?Aind an! doc having to do /ith the follo/ing R companies@See Dumais for more1Sec1 $21=Introduction to Information Retrieval *ut /h! is this clustering0We(ve tal"ed about docs- 6ueries- retrieval and precision here1What does this have to do /ith clustering0Intuition: Dimension reduction through #SI brings together ?related@ a&es in the vector space1Sec1 $21=Introduction to Information Retrieval Simplistic picture'opic $'opic 2'opic %Introduction to Information Retrieval Some /ild e&trapolation'he ?dimensionalit!@ of a corpus is the number of distinct topics represented in it1More mathematical /ild e&trapolation:if 3 has a ran" " appro&imation of lo/ Arobenius error- then there are no more than " distinct topics in the corpus1Introduction to Information Retrieval #SI has man! other applicationsIn man! settings in pattern recognition and retrieval- /e have a feature)obHect matri&1Aor te&t- the terms are features and the docs are obHects1Could be opinions and users .'his matri& ma! be redundant in dimensionalit!1Can /or" /ith lo/)ran" appro&imation1If entries are missing +e1g1- users( opinions,- can recover if dimensionalit! is lo/1Po/erful general anal!tical techni6ueClose- principled analog to clustering methods1Introduction to Information Retrieval ResourcesIIR $2Scott Deer/ester- Susan Dumais- Seorge Aurnas- 'homas #andauer- Richard 7arshman1$UU81Inde&ing b! latent semantic anal!sis1 [3SIS =$+6,:%U$\=871