Download - Evaluating Heterogeneous Information Access (Position Paper)

Transcript
Page 1: Evaluating Heterogeneous Information Access (Position Paper)

Evalua&ng  Heterogeneous  Informa&on  Access  (Posi&on  Paper)

Ke  Zhou1,  Tetsuya  Sakai2,  Mounia  Lalmas3,    Zhicheng  Dou2  and  Joemon  M.  Jose1  

1University  of  Glasgow  2MicrosoN  Research  Asia  3Yahoo!  Labs  Barcelona

SIGIR  2013  MUBE  workshop

Page 2: Evaluating Heterogeneous Information Access (Position Paper)

IR  Evalua&on

•  System-­‐oriented  Evalua&on  (test  collec&on  +  metrics)  

•  User-­‐oriented  Evalua&on  (interac&ve  user  study)  

•  Current  endeavor  to  incorporate  user  into  system-­‐oriented  metrics  – Time-­‐Biased  Gain  (Smucker,  Clarke.)  – U-­‐measure  (Sakai,  Dou)  – etc.

Page 3: Evaluating Heterogeneous Information Access (Position Paper)

Increasing  Heterogeneous  Nature    on  Search

Page 4: Evaluating Heterogeneous Information Access (Position Paper)

Increasing  Heterogeneous  Nature    on  Search

……  

Page 5: Evaluating Heterogeneous Information Access (Position Paper)

Posi&on

•  Compared  with  tradi&onal  homogeneous  search,  evalua&on  in  the  context  of  heterogeneous  informa&on  is  more  challenging  and  requires  taking  into  account  more  complex  user  behaviors  and  interac4ons.    

Page 6: Evaluating Heterogeneous Information Access (Position Paper)

Challenges

•  Non-­‐linear  Traversal  Browsing    •  Diverse  Search  Tasks    •  Coherence  •  Diversity  •  Personaliza&on    •  etc.

Page 7: Evaluating Heterogeneous Information Access (Position Paper)

Various  Presenta&on  Strategies

Non-­‐linear  Blended Blended

……  

Tabbed

Page 8: Evaluating Heterogeneous Information Access (Position Paper)

User  Browsing  Pa^ern “E”  Browsing  Pa?ern    on  Aggregated  Search  Page

“F”  Browsing  Pa?ern  on  Organic  Search  Page

h^p://searchengineland.com/eye-­‐tracking-­‐on-­‐universal-­‐and-­‐personalized-­‐search-­‐12233

Page 9: Evaluating Heterogeneous Information Access (Position Paper)

Non-­‐linear  Traversal  Browsing  

h^p://searchengineland.com/eye-­‐tracking-­‐on-­‐universal-­‐and-­‐personalized-­‐search-­‐12233

Page 10: Evaluating Heterogeneous Information Access (Position Paper)

Search  Tasks:  Ver&cal  Orienta&on

CIKM’10  (Sushmita  et  al.),  WWW’13  (Zhou  et  al.)  

Page 11: Evaluating Heterogeneous Information Access (Position Paper)

Search  Tasks:  Complexity

SIGIR’12  (Arguello  et  al.)

Page 12: Evaluating Heterogeneous Information Access (Position Paper)

Coherence

Car

Animal

Car

Car

Car

Car

Car

Car

Car

Car

Car

Car

vs.

CIKM’12  (Arguello  et  al.)  

Page 13: Evaluating Heterogeneous Information Access (Position Paper)

Coherence

Car

Animal

Animal

Car

Animal

Car

Car

Car

Animal

Car

Animal

Car

vs.

CIKM’12  (Arguello  et  al.)  

Page 14: Evaluating Heterogeneous Information Access (Position Paper)

Diversity

vs.

Image News+Map+Image SIGIR’12  (Zhou  et  al.)  

Page 15: Evaluating Heterogeneous Information Access (Position Paper)

Personaliza&on

vs.

SIGIRer Average  User

Page 16: Evaluating Heterogeneous Information Access (Position Paper)

Avenues  of  Research

•  Be^er  understanding  of  users  –  Click  models:  WSDM’12  (Chen  et  al.),  SIGIR’13  (Wang  et  al.)  

–  Ver&cal  orienta&on:  CIKM’10  (Sushmita  et  al.),  WWW’13  (Zhou  et  al.)  

–  Task  complexity:  SIGIR’12  (Arguello  et  al.)  –  Task  coherence:  CIKM’12  (Arguello  et  al.)  – Diversity:  SIGIR’12  (Zhou  et  al.)  –  Personaliza&on  – Non-­‐linear  presenta&on  strategies  

Page 17: Evaluating Heterogeneous Information Access (Position Paper)

Avenues  of  Research

•  Be^er  incorpora&on  of  learned  user  behavior  into  evalua&on  metrics  –  follow  SIGIR’13  (Chuklin  et  al)  and  convert  obtained  aggregated  search  click  models  into  system-­‐oriented  evalua&on  metrics.    

– model  addi&onal  features  into  powerful  evalua&on  framework  (e.g.  TBG,  U-­‐measure,  AS-­‐metric).

Page 18: Evaluating Heterogeneous Information Access (Position Paper)

Thank  you!  Ques&ons?

Ke  Zhou,  [email protected]

TREC  2013  FedWeb  track:  h^ps://sites.google.com/site/trecfedweb/