Post on 16-Dec-2015
Enterprise Search – Where do we go from here?
Aya Soffer, PhD
DGM, Information and Interaction Technologies
IBM Haifa Research Lab
25/2/08 2IBM Research Lab in Haifa © Aya Soffer
The World of SearchThe World of Search
Content management, groupware, “Intranet search”
E-commerce,
News, public documentation, government,…
Proprietary (owned & stored by
the SE) “local”
Market intelligence, news tracking, job data mining
Private
“inward facing”
Web searchPublic
“outward facing”
Public
(owned and stored by others) “global”Access
Content
25/2/08 3IBM Research Lab in Haifa © Aya Soffer
Enterprise Search
25/2/08 4IBM Research Lab in Haifa © Aya Soffer
User Expectations Shape Products
Global / Web search (public content)Users expect relevant answers to very low-content queries: elections, olympics
Search engines deliver!
Hence reinforcement of user expectations.
The global context is extremely popular Shapes user expectations at the workplace
Shapes enterprise search products
Enterprise search (proprietary content)Users expect a similar interaction style
Yet, users are more sophisticated
25/2/08 5IBM Research Lab in Haifa © Aya Soffer
Enterprise Search — Harder than Web Search
Far fewer resources — but high expectations
Data is not “search friendly”
Must index everything - find everything I can access
Security - but show me only what I am allowed to see
Link-based methods not as effective – not enough linkage
Search is not cheap! - about 5-10 cents/ document/ year – yet ROI is hard to calculate
25/2/08 6IBM Research Lab in Haifa © Aya Soffer
Smaller scaleCorpus is much smaller
Query load is much lower
Less anarchic Central authority
Data formats can be controlled
More potential structure Data is better organized and richer
Can tap into organizational knowledge
No spamAt least not intentional
Enterprise Search — Easier than Web Search
25/2/08 7IBM Research Lab in Haifa © Aya Soffer
Major Players
25/2/08 8IBM Research Lab in Haifa © Aya Soffer
Trends
25/2/08 9IBM Research Lab in Haifa © Aya Soffer
Enterprise Search Trends
Information finding goes beyond simple keyword search in search bar
Text analytics and semantic search allows searching by concepts instead of by keywords Understand content, understand user intent
Collaboration technology is fundamentally changing how people organize and access content
Democratic tagging of data as a bridge across multiple taxonomiesDiscover relationships, find experts, utilize wisdom of the crowds and social networks to find best matches
Information finding isn’t enough – need to provide means to explore search results
New visualizations for search resultsCombined search / browse paradigm will be pervasiveCombination of search and BI BI for the Masses
Search goes mobileAccess information in context from mobile devices
25/2/08 10IBM Research Lab in Haifa © Aya Soffer
Social Search
25/2/08 11IBM Research Lab in Haifa © Aya Soffer
Social Search
With the advent and popularity of Web 2.0, the ability to share and find information is being expanded beyond keyword search mainly by use of tags
While Google's page rank can be viewed as one of the first applications of the Web 2.0 concept of wisdom of crowds, Search has yet to fully harness the power of Web 2.0
Community influenced search is beginning to appear in niche search engines and as Beta’s in major search engines
Wikia Search
Yahoo! MyWeb
Google Co-Op
Eurekster
25/2/08 12IBM Research Lab in Haifa © Aya Soffer
1 billion global users80,000,000 web sitesContent ConsumersContent Providers
User-Generated ContentUser-Generated Content
Published ContentPublished Content
CollaboratorsFacilitators
Collective IntelligenceCollective Intelligence
Web 2.0: The wildly read-write Web 2.0
25/2/08 13IBM Research Lab in Haifa © Aya Soffer
Search Today – Web 2.0 phenomena indirectly influencing search results
Indirect InfluenceIndirect InfluenceCollective IntelligenceCollective Intelligence
User-Generated ContentUser-Generated Content
Published ContentPublished Content
User-Generated MetadataUser-Generated MetadataSocial NetworksSocial Networks
25/2/08 14IBM Research Lab in Haifa © Aya Soffer
Evolution of Search Technologies
Scope
Technology
Business Impact
User
World Wide Web (Html)
• Hyperlink analysis for relevance ranking (good for Web, not as good in enterprise)
• Categorization/ summarization
eCommerce and consumer market
Everyone
Web-based Search (2nd Generation)
Small, closed collections
• String matching• Boolean search• Basic relevance ranking
Vertical business domains (medical, legal)
Trained specialists
InformationRetrieval
(1st Generation)
Structured, semi-structured and unstructured information
• Text analytics with novel linguistic and semantic processing
• Ranking on structure• Faceted navigation
Pervasive use in business processes to realize value from unstructured information
Everyone & applications
Information Discovery
(3rd Generation)
Structured, semi-structured, unstructured information + networks
• Community input • Incorporation of wisdom of crowds
• Tags, tag clouds, social network data
Pervasive use in business processes to realize value from unstructured information
Everyone & applications
Social Information Discovery
(4th Generation)
25/2/08 15IBM Research Lab in Haifa © Aya Soffer
Trend - Mutually Reinforcing Relationship
Data
Metadata
•Author
•Mentioned
•Bookmark
•Reader
•Associated
•TaggedByPeople
tagsfolder label
subject of emailtitle of document
anchor textquery
authordocument owner
email addressusername
codeuser profile
email messagesreceived documentsauthored documents
calendar entrieschat history
25/2/08 16IBM Research Lab in Haifa © Aya Soffer
Web 2.0 Unified Search
The result set includes relevant documents, people, and tags. Ranking is affected by the volume of tagsand comments that are associated with each document.
Tags related to the result set – presented as a tag cloud
People related to the result set –sorted by relevance
25/2/08 17IBM Research Lab in Haifa © Aya Soffer
Guided Navigation Meets Business
Intelligence (BI for the Masses)
25/2/08 18IBM Research Lab in Haifa © Aya Soffer
Multifaceted search or guided navigation
Allows the simultaneous exploration of many aspects of a topic, and the gradual “zooming in” on the information target
Reduces frustration: ensures that only valid choices are presented, so zooming in never yields an empty result set
Solution to the problem of
“few terms too many results, more terms no results”
Supports browsing when the user doesn’t really know what to ask for in a multi-dimensional information space
Multifaceted search is very popular in e-Commerce solutions (Amazon, eBay, buy.com, …), but is also relevant to more traditional text search applications
New applications are emerging every day
25/2/08 19IBM Research Lab in Haifa © Aya Soffer
Multifaceted Search – Example
Categories- {
Featured Dimensions
Categories
Category Counts
QueryCurrent Context
Other Dimensions
Search Results
25/2/08 20IBM Research Lab in Haifa © Aya Soffer
Guided Navigation Meets Business Intelligence
Faceted Navigation LimitationsCurrent faceted navigation interfaces enable drilling down one facet at a time.Counts and context are similarly presented for each facet separately
Business Intelligence today is mainly for structured informationAssumes structure is known in advanceOffline processing Complex report generations
New direction: faceted navigation as a front end for business intelligence applications.
Facets can be defined on a combination of fields including aggregations and simple expressions.
25/2/08 21IBM Research Lab in Haifa © Aya Soffer
MultiFaceted Search with BI - Example
Query
For each author: counts (as before) but also calculated values on result set
Average Rating per Sales Rank
25/2/08 22IBM Research Lab in Haifa © Aya Soffer
Correlating Facets – BI on search results
Top-Ranked books do not
have the highest ranking
Older books have lower sales rank
Clicking on the value will bring
you to the books with high rating and top
sales rank
25/2/08 23IBM Research Lab in Haifa © Aya Soffer
The World of SearchThe World of Search
Content management, groupware, “Intranet search”
E-commerce,
News, public documentation, government,…
Proprietary (owned & stored by
the SE) “local”
Market intelligence, news tracking, job data mining
Private
“inward facing”
Web searchPublic
“outward facing”
Public
(owned and stored by others) “global”Access
Content
25/2/08 24IBM Research Lab in Haifa © Aya Soffer
The World of SearchThe World of Search The boundaries are blurring
Content management, groupware, “Intranet search”
E-commerce,
News, public documentation, government,…
Proprietary (owned & stored by
the SE) “local”
Market intelligence, news tracking, job data mining
Private
“inward facing”
Web searchPublic
“outward facing”
Public
(owned and stored by others) “global”Access
Content