FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is...
Transcript of FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is...
![Page 1: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/1.jpg)
FXPAL at
TRECvid 2007
![Page 2: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/2.jpg)
Collaborative
Exploratory Search
![Page 3: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/3.jpg)
6 November 2007 TRECvid 2007 workshop
“Collaborative” search is overloaded
Real-time awareness
and continual update
context systems
(e.g. Nokia, Imity)
Collaborative
Exploratory Search
(FXPAL)
Chi et al
“Search Trails”
(Xerox PARC)
Web 2.0
Wisdom of Crowds
Collaborative Filtering
Personalization
Explicit Implicit
Asynchro
nous
Synchro
nous
![Page 4: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/4.jpg)
6 November 2007 TRECvid 2007 workshop
“Collaborative” search is overloaded
Collaborative
Exploratory Search
(FXPAL)E
xp
licit
Synchro
nous
Collaborative
Exploratory Search
•Fischlar-DiamondTouch:
Collaborative Video Searching on a Table
(Smeaton et al, 2005)
•Interfaces for Collaborative Exploratory Web Search:
Motivations and Directions for Multi-User Designs
(M. Morris, 2007)
Algorithmically-Mediated Intelligent Interfaces Only
![Page 5: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/5.jpg)
6 November 2007 TRECvid 2007 workshop
Collaborative Exploratory Search
• Synchronous
– Collaborating users use the system at the same time
• Explicitly Shared goals
– Collaborating users share the information need
• Algorithmically-mediated
– System combines users’ inputs in various ways
• Not just keyword pooling
– System generates results based on users’ roles
• Terms, ranked lists, etc.
![Page 6: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/6.jpg)
6 November 2007 TRECvid 2007 workshop
Algorithmic
Collaboration
Logic Unit
Input
Coordinator
User 1
Output
Coordinator
User 2
![Page 7: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/7.jpg)
![Page 8: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/8.jpg)
6 November 2007 TRECvid 2007 workshop
![Page 9: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/9.jpg)
6 November 2007 TRECvid 2007 workshop
![Page 10: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/10.jpg)
6 November 2007 TRECvid 2007 workshop
System overview
MediaMagic RSVPShared Display
Input
Coordinator
Output
Coordinator
Algorithmic
Collaboration
Module
![Page 11: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/11.jpg)
6 November 2007 TRECvid 2007 workshop
![Page 12: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/12.jpg)
6 November 2007 TRECvid 2007 workshop
![Page 13: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/13.jpg)
6 November 2007 TRECvid 2007 workshop
![Page 14: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/14.jpg)
6 November 2007 TRECvid 2007 workshop
![Page 15: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/15.jpg)
6 November 2007 TRECvid 2007 workshop
ProspectorM
iner
![Page 16: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/16.jpg)
6 November 2007 TRECvid 2007 workshop
RSVP Queue Priority
qdocqretrievedqdoc rankNscore,,,
−=
∑ ⋅⋅=
q
qrelqseenqdocdoc wwscorerank,,,
qunseenqseenqseen NNw,,,
/=
qnonrelqrelqrel NNw,,,
/=
Weighted Borda Count fusion
Freshness
Relevance
![Page 17: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/17.jpg)
6 November 2007 TRECvid 2007 workshop
Shared Display
Suggested Query Term
qretrievedqterm TFscore,,
=
∑ ⋅⋅=
q
qrelqseenqtermterm wwscorerank,,,
qunseenqseenqseen NNw,,,
/=
qnonrelqrelqrel NNw,,,
/=
Weighted frequency fusion
Freshness
Relevance
![Page 18: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/18.jpg)
6 November 2007 TRECvid 2007 workshop
Example
![Page 19: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/19.jpg)
6 November 2007 TRECvid 2007 workshop
TRECvid Experiments
• 3 ½ Systems, 4 Users
a. MMA: Single MediaMagic user (full capabilities)
b. MMV: Single MediaMagic user (no text)
c. MMA+V: Post hoc simulated MMA+MMV
combination
– Duplicates (both rel and nonrel) removed
d. COLL: Collaborative search
![Page 20: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/20.jpg)
6 November 2007 TRECvid 2007 workshop
TRECvid Experiments
• Problem: Learning effect?
– All COLL runs done first
– All MMA runs done second
– All MMV runs done third
![Page 21: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/21.jpg)
6 November 2007 TRECvid 2007 workshop
Results: Mean Average Precision
0.0000
0.0500
0.1000
0.1500
0.2000
0.2500
0.3000
0.3500
0.4000
FX
PA
L_C
O15
FX
PA
L_C
O
FX
PA
L_
MM
A
FX
PA
L_C
O11
FX
PA
L_
MM
V
FX
PA
L_C
O07
MA
P
Collaborative search, 7 minutes
Single user, Video only
Single user, text
Collaborative search
![Page 22: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/22.jpg)
6 November 2007 TRECvid 2007 workshop
Additional Metrics
• Examine Recall and Precision separately
• Examine the manually-selected shot set
– What actually happened during the run?
![Page 23: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/23.jpg)
6 November 2007 TRECvid 2007 workshop
Precision
• COLL is:
1.47% relative improvement over MMA
-3.42% relative improvement over MMV
15.4% relative improvement over MMA+V
Legend explaining MMA,etc
FPTP
TP
+
![Page 24: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/24.jpg)
6 November 2007 TRECvid 2007 workshop
Recall
• COLL is:
101.1% relative improvement over MMA
43.3% relative improvement over MMV
-10.7% relative improvement over MMA+V
totalrel
TP
#
![Page 25: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/25.jpg)
6 November 2007 TRECvid 2007 workshop
• COLL outperforms MMA and MMV
• COLL is about the same against MMA+V
– What does this suggest?
– Why bother working collaboratively?
– Let’s examine closer
![Page 26: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/26.jpg)
6 November 2007 TRECvid 2007 workshop
% improvement in precision
-40
-20
0
20
40
60
80
100
120
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
COLL over MMA COLL over MMV
![Page 27: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/27.jpg)
6 November 2007 TRECvid 2007 workshop
% improvement in recall
0
50
100
150
200
250
300
350
400
450
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
COLL over MMA COLL over MMV
![Page 28: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/28.jpg)
6 November 2007 TRECvid 2007 workshop
% improvement COLL over MMA+V Kernel density smoothing
-60
-40
-20
0
20
40
60
80
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Precision
Recall
![Page 29: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/29.jpg)
6 November 2007 TRECvid 2007 workshop
Tentative Conclusion:
Collaborative Search (at least in our current
implementation) offers its best improvements when there are fewer relevant documents to be found
![Page 30: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/30.jpg)
6 November 2007 TRECvid 2007 workshop
Normalizing by Shots Viewed
• Our RSVP system needed another design iteration (missed opportunity)
• Average number of shots viewed:
– MMA: 2,123
– MMV: 2,601
– MMA+V: 4,184
– COLL: 2,614Work smarter not harder?
![Page 31: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/31.jpg)
6 November 2007 TRECvid 2007 workshop
Precision
Precision, with counts normalized by the
number of seen shots, does not change
FPTP
TP
seen
FP
seen
TPseen
TP
+
=
+
##
#
![Page 32: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/32.jpg)
6 November 2007 TRECvid 2007 workshop
Recall
• COLL is:
73.9% relative improvement over MMA
(101.1%)
38.5% relative improvement over MMV
(43.3%)
44.1% relative improvement over MMA+V
(-10.7%)
totalrel
seen
TP
#
#
![Page 33: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/33.jpg)
6 November 2007 TRECvid 2007 workshop
% improvement in recall
-100
0
100
200
300
400
500
600
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
COLL over MMA COLL over MMV
![Page 34: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/34.jpg)
6 November 2007 TRECvid 2007 workshop
COLL over MMAV
-50
0
50
100
150
200
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
COLL over MMAV
% improvement in recallKernel density smoothing
![Page 35: FXPAL at TRECvid 2007 - NIST · 6 November 2007 TRECvid 2007 workshop “Collaborative” search is overloaded Real-time awareness and continual update context systems (e.g. Nokia,](https://reader033.fdocuments.in/reader033/viewer/2022042009/5e713b0566d48f33081624d6/html5/thumbnails/35.jpg)
6 November 2007 TRECvid 2007 workshop
Future Work
• Still like the idea of miner vs. prospector
– But need to give the miner more ability to
“steer”
– And achieve higher throughput
• Also investigate other collaboration roles
• Also investigate types of queries in which different roles work better. Can we know this a priori?