Video Hyperlinking: Pursuing a Moving Target:
Itera9ve Use of Benchmarking of a Task to Understand the Task
MariaEskevich,GarethJ.F.Jones,RobinAly,RoelandOrdelman,BenoitHuet
VideoHyperlinking(LNK)
Chasing History
2009-2010 2011 2012 2013 2014 2015 2016
RichSpeechRetrieval
VideoCLEF2009LinkingTask
SearchandHyperlinking:<search><anchoring><hyperlinking>
TRECVid
Blip10000 BBC(growingdatacollecYon) Blip10000.2
Generalusers(MTurk) MulYmediaprofessionals(archivists,journalists)Cue-guided
users
Datasets
Typeofusersgenera6ngthequeries/anchors
Tasktypesandevalua6onmetrics
Known-itemsearch:MRR
Adhochyperlinking:adaptedMAP,P@10
Adhochyperlinking:
MAiSP
Dataset: Video collec9ons
Benchmark Collec6on Hours Videos Producers
ME'12 bliptv1000 2125 9550 Laymen
ME'13 BBC1 1260 1623 MediaProfessionals
ME'14/TV'15
AXES2 2686 3520 MediaProfessionals
TV'16 bliptv2 2492 11482 GuidedLaymen
Who defines the anchor?
• ME'12:AnchorsarerelevantsegmentsforinformaYonneeds• ME'13:RegularinternetusersinastudyattheBBC• ME'14:Large-scaleseconduserstudyattheBBC• TV'15:AskVideoProfessionals–journalists,archivists,…• TV'16:Usecuesinvideotogettoaspecifictypeofanchor
ME'13 User study
• 28Users-policemen,hairdresser,bouncer,salesmanger,student,self-employed
• Twohoursession: • Browsethearchive• Defineknown-item• Defineanchor
Problems
• Somenonesenseanchors(Beyonce??)• Hyperlinkingconceptwasnotunderstoodbyeveryone• PooranchorsegmentdefiniYon• IncompletedescripYonofdesiredtargets• CorrecYonbyorganizersmethodologicallyproblemaYc
Moving on
• ME’14:• Enlargedstudy• Beleruserinterfaceguidance• Expensive,notmuchbelerthanME’13
• TV’15:• Askmediaprofessionals• InteracYvecoachingtocreateanchors• Mucheffortandavailabilityofprofessionalslimited• MulYmodalityimportantforcommunitybutnotensured
TV’16: Guided and focused chase for mul9modality
LookforsegmentscontainingacombinaYonofverbal-visualinformaYon:• VerballinguisYccues:`cansee',`seeinghere',`thislooks’,`lookslike',`showing',`wanttoshow’• Visualcues:acYonsandobjectscrucialforthisvideoarenotexplicitlynamedormenYoned
Anchors/ParYcipants’results
HIT
RelevanceJudgments/Workers
feedback
Anchors/ParYcipants’results
HIT
RelevanceJudgments
Anchors/ParYcipants’results
HIT
RelevanceJudgments
Itera9ve HIT Improvement
Finding common language with the crowd • ME’12:Watchtwovideosegmentsandsaywhethertheyarerelatedinsomeway.• Yes/Noanswers
• ME’13+14:Yes/Noanswers• WatchtwovideosegmentsandsaywhetherthesecondvideoisrelatedtothefirstoneaccordingtothegivendescripYon.
• TV’15:• Detailsonwhatconnectsordisconnectsgivenvideosegments
• TV’16:twostepprocedure:• Watchthevideosegmentanddescribewhywouldsomeoneshareit• PleasewatchthevideosegmentandpickthedescripYon• Watch2videosegmentsandproviderelevancedescripYondetails
Turning to the Crowd for Insight and Evalua9on 3stageapproach
ManualAnchorsCreaYon
1.AnchorVerificaYon
2.TargetVerng
3.Video-to-VideoRelevanceAnalysis
Video-to-Videoretrievalsystems
MulYmodalAnchors
AutomaYcTargetsCreaYon
Qrel DetaileddescripYonsforrelevance
Adapted MAP
Results
Assess-ments
RawResults
Res1 Res3 Res2
Effective Results Res1 (Rel=1) Res2 (Rel=0)Res3 (Rel=0)
Adapted MAP Undesired Configura9ons
Results
Assess-ments
RawResults Res1 Res3 Res2
Effective Results
Rel=1 Rel=1 Rel=1
Binned Relvance
Bin 1
Time
Bins
Assess-ments
RawResults
Bin2 Bin3
Res1 Res3 Res3
Effective Results Res1 (Rel=1) Res2 (Rel=0)
Bin Size
Tolerance to Non-Relevance
Time
Assess-ments
RawResults
Res1 Res3 Res2
Effective Results Res1 (Rel=1) Res2 (Rel=0)
Tolerance Window
Hands-on take home messages • Beginningoftheyearlycycle:• Gatherfeedbackontask• IsthereaparYcipant?Istheremoney?• ScienYficquesYon
• Tasklifecycle:• Bepreparedfordisaster!• InteractwiththeparYcipants• ClosecollaboraYonbetweentaskorganizers• InteracYonwithbenchmarkorganizersimportant
• Wrapup:• Adjustthetarget• Tasksustainabilityfornextyear
Top Related