1. Why and how? User Intentions and their Relation to Multimedia Applications Mathias Lux 2. Why and how? User Intentions and their Relation to Multimedia Applications Mathias Lux 3. The intention gap [...] Between the users search intention and what he submits as the query, he is caught in the intention gap, due to the difficulty of converting the intention into a search-engine-friendly query. The user intention gap remains a major obstacle to meeting the music information needs of users [...] Src. Zhu, Shenggao, et al. "Bridging the User Intention Gap: an Intelligent and Interactive Multidimensional Music Search Engine." Proceedings of the First International Workshop on Internet-Scale Multimedia Management. ACM, 2014. 4. The intention gap Is it like ... ??? users are not able to query a retrieval system the right way 5. *RTFM* 6. The intention gap Common idea is to improve either the system or the user interface By query re-formulation, query expansion, and query suggestion 7. Example: QBE User has particular information need Need reflected by example image Query is expressed visually 8. PHOG & Flowers 9. ColorLayout & Sunsets 10. EdgeHistogram & Portraits 11. JCD & Portraits 12. ColorLayout & Landscapes 13. PHOG & Birds on the Water 14. We know that ... Some features work better than others Features have different characteristics Some features work out well for some domains, while others dont 15. Which one is right? How to determine the right feature? What are the necessary characteristics? How do I define visual similarity within the domain? What is visual similarity for the user? 16. Approaches ... Show all features, but with a clever UI 17. Approaches ... Educate users? 18. Approaches ... Create a clever system. BUT: How? 19. OK, back again ... We need systems that infer / know what users want. Are there working examples? 20. Amazon? Well, ... we all know the drawbacks of that one. 21. Why is there a different ranking? 22. Users in Context 23. Definition: Context Context is any information that can be used to characterize the situation of an entity. An entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and applications themselves. Ref. G. Abowd et al., Towards a better understanding of context and context-awareness. In Handheld and Ubiquitous Computing, vol. 1707 LNCS, 1999. 24. Definition: Intention noun (1) thing intended; an aim or plan (2) Medicine the healing process of a wound (3) (intentions) Logic conceptions formed by directing the mind towards an object 25. Context vs. Intention? Context is any information that can be used to characterize the situation of an entity. An entity is a person, [] noun (1) thing intended; an aim or plan [] 26. A Users Intention is part of a users context of manageable size (verb & frame) related to the information need in search Examples I want to download a new background for my mobile. I want to share the first step of my kid. I want to see what the Mars surface looks like. 27. User Intentions in the Web Underlying goals of web searches Informational to learn / know something Navigational to go to a specific place (on the web) Transactional to go somewhere to ultimately buy sth. Ref. Broder: A Taxonomy of Web Search. SIGIR Forum, 2002 28. User Intentions in the Web Ref. Rose & Levinson: Understanding user goals in web search, WWW 2004 Broder Survey Broder Log Analysis R&L St udy 1 R&L St udy 2 R&L St udy 3 24,5 20 14,7 11,7 13,5 39 48 60,9 61,3 61,5 36 30 24,3 27 25 Navigat ional Inf ormat ional Transact ional 29. User Intentions in Multimedia Why do people search produce share archive For which type image video audio other 30. Hand-picked Examples Right now there is no all-in-one publication on user intentions in multimedia 31. User Intentions in Image Search Ref. Lux, Kofler & Marques: A classification scheme for user intentions in image search, CHI 2010 beautiful sunset for the background of my mobile old vs. new Starbucks logo my friends new car on flickr.com Latest explore photos 32. Do queries help with the search intention? User information need vs. query formulation in video search. How to support users with video indexing and search methods? Search goal failure is (partially) predictable Based on keywords and Based on natural language Ref. Kofler, Larson & Hanjalic: To Seek, Perchance to Fail: Expressions of User Needs in Internet Video Search, ECIR 2011 33. Asking for the Why? behind the What? Ref. Hanjalic, Kofler & Larson: Intent and its Discontents: The User at the Wheel of the Online Video Search Engine, ACM MM 2012 Hear others singing Learn to sing 34. Online Survey: Why did you watch this video? 270 participants Online questionnaire Why do you watch videos? Which category for which reason? Results in a intention 2 category table Ref. Lagger, Lux & Marques: An Adaptive Video Retrieval System Based On Recent Studies On User Intentions While Watching Videos Online. ACM CIE, online. 35. Helping with clever Uis? Ref. Lagger, Lux & Marques: An Adaptive Video Retrieval System Based On Recent Studies On User Intentions While Watching Videos Online. ACM CIE, online. 36. Who are the Users in a Video Search System? Study on users of YouTube BBC iPlayer Uitzending Gemist Ref. Kemman, Kleppe & Beunders: Who are the users of a video search system? Classifying a heterogeneous group with a profile matrix, WIAMIS 2012 37. Who are the Users in a Video Search System? Ref. Kemman, Kleppe & Beunders: Who are the users of a video search system? Classifying a heterogeneous group with a profile matrix, WIAMIS 2012 38. Why do People make Videos? Study on four main goals: Affection, Function, Sharing & Preservation. Ref. Lux & Huber: Why did you record this video? WIAMIS 2012 39. Correlation of Clusters -Coefficient Showing minimum, actual & maximum Sharing Af f ect ion Funct ion - 0,59 - 0,7 8 - 0,36 - 0,05 - 0,26 - 0,36 0,39 0,84 0,55 - 0,50 - 0,93 0,25 - 0,07 0,46 0,21 - 0,43 - 0,21 0,47 Preservat ion Sharing Af f ect ion 40. Implications? 81% of the instances were assigned to more than one category. Hypothesis: a discrete mapping to exactly one category is insufficient Preservation and Sharing seem to be independent Hypothesis: Personal reflection is not the opposite of publishing (cp. Kindberg) Preservation and Function are neg. correlated Hypothesis: Videos are not archived if they are taken for a functional reason. 41. Intentional Framing Def. The framing of a picture is the Deutungsrahmen (interpretation frame of the image) which image producers have in mind during the creation moment of the picture. 42. Intentional Framing 43. Intentional Framing 44. Intentional Framing and .. 45. Examples for Intentional Framing 46. Intentional Framing Assumption: Similar intentions for taking the pictures will lead to similar framings of the images. Hypothesis: Certain global image features can be used to differentiate framings. Ref. Riegler, Lux, Larson, Kofler (2014) How `how' reflects what's what: Content-based classification exploiting how users frame images, ACM MM 2014 47. Uploader Intent Typology based on text mining, crowd sourcing, literature 1. Explaining 2. Sharing 3. Promoting 4. Communicating 5. Entertaining Kofler et al. (2015) Uploader Intent for Online Video: Typology, Inference and Applications, ACM TOMM, accepted 48. Uploader Intent Kofler et al. (2015) Uploader Intent for Online Video: Typology, Inference and Applications, ACM TOMM, accepted 49. Finding User Intentions & Goals is a hard task . 50. Demand Media The Answer Factory Demand mined from search queries Requests for content put on auction Contractors create content Crowd does quality control see i.e. eHow.com The Answer Factory: Demand Media and the Fast, Disposable, and Profitable as Hell Media Model, http://www.wired.com/magazine/2009/10/ff_demandmedia/ 51. Human Computation 52. Human Computation 53. Human Computation Is crowd-sourcing of any help here? 54. Crowd-Sourcing Its hard to judge intentions of others That makes it error prone a reminder of the beautiful Island were [sic] my father came from o Recall situation o Preserve good feelings o Publish online o Show to family & friends o Support task of mine o Preserve bad feeling 55. turkers Recall situation 2 2 0 1 0 2 Preserve good feeling 2 -2 1 0 0 1 Publish online 2 0 0 0 1 2 Show to family & friends 2 1 2 1 1 0 Support task of mine 0 -2 1 1 1 -2 Preserve bad feeling -2 0 -2 0 -2 -2 a reminder of the beautiful Island were [sic] my father came from o Recall situation o Preserve good feelings o Publish online o Show to family & friends o Support task of mine o Preserve bad feeling 56. Crowd-Sourcing Turkers disagreed with original publishers. But pretests had better inter-rater agreements. Ref. Lux, Taschwer & Marques: A Closer Look at Photographers Intentions: a Test Dataset, CrowdMM WS at ACM MM 2012 Int ent ions Ot her t urkers 0,147 0,232 pret est s 0,57 1 0,510 Ref. Lux, Xhura & Kopper: User Intentions in Digital Photo Production: a Test Data Set, accepted, MMM 2014 57. Where did we go? Intention gap User Intentions Search Production Sharing Crowdsourcing as a Method 58. What did we learn? Its hard for people to judge the intent of others Its hard for people to express their own intent 59. A Word of Caution People can only use the tools that are available to them. 60. What is left? Lots of loose ends & open grounds for research 61. Short Term Goals: Inter-relate CBIR & Intentions We did some work on the relation of BoVW & the degree of intentionality ... New or adapted features Adaptive retrieval systems New approach to ground truth 62. Medium Term Goals: Interdisciplinary Approach Group intentions vs. individual intentions Empirical and qualitative research on user context & intentions 63. Medium Term: Is there an over model? Search Production Sharing Archiving Image Video Audio Other modalities 64. Long term goals Find a way to integrate user context in multimedia information systems through intentions Find out intentions of users with a high degree of accuracy 65. Thanks for listening Mathias Lux Assoc. Prof @ Klagenfurt University mlux@itec.uni-klu.ac.at