What Innovators Need to Know About IP Protection: A Business-Focused Approach
What Business Innovators Need to Know about Content Analytics
-
Upload
seth-grimes -
Category
Business
-
view
469 -
download
0
description
Transcript of What Business Innovators Need to Know about Content Analytics
Smart ContentWhat Business Innovators Need to Know about Content Analytics
Jeff FriedCTO, [email protected]
Examples from:
Opinions from:
Three Views of Content Analytics
Business Strategist End User Research Scientist
It’s about money, business models, advertising, and
money.
It’s about finding things, having
fun, and getting stuff done.
It’s about fast algorithms,
massive scales, and machine
learning.
Content is Exploding“If you think the information
doesn’t exist you’re not looking hard enough”
6
Traffic, Ads and Information Mash-Upsbecoming a part of emerging ecosystems
cloudplatforms
contentplatforms
adplatforms
services
Rethinking the Data Warehouse
Use of Unstructured Data in Information Analysis Applications
Analyst: Mark Beyer
10
Smart Content is Streaming
Different multimedia applications
Education
Entertainment
Archives
13
Social Video Sharing System
Users create, produce, upload, manage and share video within one system
Music Image Face
Query by example: ContentFusion
Smart Content is Mobile
Location and Form factors
Bing Twitter Maps
Smart Content is Social
Many layers of social media
//twitterviz
Publication Platforms
PublicCommunityPrivate
Publication
Priv
ate
Com
mun
ityPu
blic
Acce
ss
Answers
Web
Twitt
er
Social Graph
Naturally connected community Spam marketing campaign
Spammy communities are highly visible – don’t be part of one!
Permission
Social Search Needs• Relevance
– Filtering the document web
• Social Media Content– Filtering the social web
• Trends / Group Insight– Tapping Community Knowledge
• Answers– Trusted Advisor
Recommendation
• “Java” (coffee, island, or language?)
• “compliance”
• “What should I do in New York?”• Where are my friends now?
• Why did power go out in Palo Alto?• How does adoption work?
• ( on FB update) anybody give their babies baby Benedryl for travel/jet lag? Want to hear from parents whether they have or not and how it went
Enables 1:1 relevance based on user profile
Complexity
Value
3. Social Recommendations
(users to users)
1. Content or “Related item”Recommendations
(items to item)
2. PersonalizedRecommendations
(items to user)
Enables connections between like users
Drives service stickiness
Enables users to ‘browse sideways’ from any item
Recommendations“Personalized” to “Social”
Virtuous CyclesCreate new patterns with
positive reinforcement
Enhanced Modes of Discovery
Simplified Authoring
& Participation
New Value in Combining Data
CONSUMPTION
CONNECTIONS CREATIONS
The Long-Tail of Online Business
70 %30 %
QUERYTRAFFIC
+70% Y/Y
Virtuous Cycles in FindabilityTuned experienceSocial behavior affects relevance
Socially driven feedback loop People and expertise location are the key ‘lens’
Structure drives exploration Aligned with taxonomy and tags
Refinement
Social
Relevance
Text Analytics Isn’t Perfect
Realistic Expectations for Powerful Technology
Analytics! Semantics! Machine Learning!
36
Grab-Bag of Related Technologies• Problem – linguistic variations in concept expression
– Technology: natural language processing (NLP)
• Problem – huge numbers of documents that are the same or versions of the same– Technologies : text mining, text analytics, normalizing & de-duping
• Problem – amount of content exceeds amount of human expertise to analyze & categorize– Technologies : entity extraction, contextual analysis, auto-
categorization
• Problem – understanding trends and relative values expressed in content– Technology : sentiment analysis
• Problem – retrieving & federating contextually related and relevant content– Technologies – All of the above
3810 Entire contents © 2006 Forrester Research, Inc. All rights reserved.
BPM, Service Orchestration, Workflow
Content, Search, Integration, & Composition technologies
Presentation tier
Middle tier
Repositories
Unstructured Information access
Structured Data AccessDynamic Information Applications
Visualization
Portals, AJAX, Mash-ups, RSS, widgets, gadgets
Enterprise Content Management
ERP, CRM
, PIM, PLM, SCM
, HCM
Pro
du
ctiv
ity A
pp
s (m
ail,
IM,
offi
ce to
ols
)
Co
llab
ora
tion
To
ols
EAI, EII, ESB
Business Intelligence
Databases
ETL, Data Cleansing, Data Quality
Identity
MDM, Data Warehouses
File systemsFile filters
Connectors
Taxonomy Text Mining
Desktop Search
Federated Search
Video/audio
Enterprise Search
39
Linguistics, Statistics, & GymnasticsLexicon Base
Language-specific Common Words
Inflection Dictionaries
Part-of-speech Dictionaries
Synonymy Dictionaries
Subject-specific ontologies
Spellcheck dictionaries
Geographical and people’s names
Special terminology lexica
Basic Linguistic Algorithms
Pattern extraction
Stemming / Lemmatization
Part-of-speech Tagging
Language normalization
Vectorization
Applications
Data Cleansing
Categori-zation
Entity Extraction
Suggest Synonyms
Find similar
Stop word elimination
Spell checking
Machine Translation
Relationship Extraction
From Entity Extraction
Acronym
Person Location End of sentence
End of paragraph
Date Base = 2002-03-XX
To Fact Extraction....
Substance
Base=„Gold“
Class=„Element“
Number=79
Symbol=Au
Location
Base=„Qilian“
Country=„China“
Region=„Asia“
Subregion=„East“
„The Red Valley property lies within the Qilian fold beltwhich is host to gold deposits.“
Qilian is location of gold
Extracted Fact: Substances x Locations
Substance
Base=„Gold“
Class=„Element“
Number=79
Symbol=Au
Location=„Qilian“
Location
Base=„Qilian“
Country=„China“
Region=„Asia“
Subregion=„East“
Substance=„Gold“
Indicates a gold location
Intelligent Answers from TextInternal/external text sources Internal/external text sources
ContentFusion
Scale, Simplicity and ExpressivenessAccelerating Content Enrichment / Fusion
LookingGlass
Semantics means what?
Beware of overhype; seek pragmatic use of semantic tech
Solving the Knife problem
Man Allegedly Attacked Wife With KnifeA Tyler man is awaiting arraignment this afternoon afterallegedly attacking his wife with a knife, said Tyler police.The 41-year-old man will face aggravated assault andaggravated robbery charges, said Don Martin, thedepartment's spokesman.Officers took the man in custody near Garden Valley and Loop 323. He ran from his residence after "assaulting his wife with a knife and taking her purseat knifepoint," said information released by Martin. Thewoman refused medical treatment and did not appearto be seriously injured, the statement said.
Excellent Knives!!!
Mere frequency counting of key words can lead to undesired results......understanding relationships between words can reveal the true topic of the document.
Objective: Automatically insert an advertisement that matches the content best.
Actor Director Movie
TV Show
Adventure Comedy Face Image
Actor 0 0.6 1 1 1 1 0.9
Director 0 1 1 1 1 0.3
Movie 0 0.6 1 1 -1
TVShow 0 1 1 -1
Adventure 0 0.14 -1
Comedy 0 -1
FaceImage 0
DocumentAnalytics
SemanticAggregation/
Analytics
DomainKnowledge
48
Cyc Knowledge Base
ThingThing
IntangibleThing
IntangibleThing IndividualIndividual
TemporalThing
TemporalThing
SpatialThing
SpatialThing
PartiallyTangible
Thing
PartiallyTangible
ThingPathsPaths
SetsRelations
SetsRelations
LogicMathLogicMath
HumanArtifactsHumanArtifacts
SocialRelations,
Culture
SocialRelations,
Culture
HumanAnatomy &Physiology
HumanAnatomy &Physiology
EmotionPerception
Belief
EmotionPerception
Belief
HumanBehavior &
Actions
HumanBehavior &
Actions
ProductsDevices
ProductsDevices
ConceptualWorks
ConceptualWorks
VehiclesBuildingsWeapons
VehiclesBuildingsWeapons
Mechanical& Electrical
Devices
Mechanical& Electrical
Devices
SoftwareLiterature
Works of Art
SoftwareLiterature
Works of ArtLanguageLanguage
AgentOrganizations
AgentOrganizations
OrganizationalActions
OrganizationalActions
OrganizationalPlans
OrganizationalPlans
Types ofOrganizations
Types ofOrganizations
HumanOrganizations
HumanOrganizations
NationsGovernmentsGeo-Politics
NationsGovernmentsGeo-Politics
Business, Military
Organizations
Business, Military
Organizations
LawLaw
Business &CommerceBusiness &Commerce
PoliticsWarfarePoliticsWarfare
ProfessionsOccupationsProfessionsOccupations
PurchasingShopping
PurchasingShopping
TravelCommunication
TravelCommunication
Transportation& Logistics
Transportation& Logistics
SocialActivitiesSocial
ActivitiesEveryday
LivingEveryday
Living
SportsRecreation
Entertainment
SportsRecreation
Entertainment
ArtifactsArtifacts
MovementMovement
State ChangeDynamics
State ChangeDynamics
MaterialsParts
Statics
MaterialsParts
Statics
PhysicalAgents
PhysicalAgents
BordersGeometryBorders
Geometry
EventsScriptsEventsScripts
SpatialPaths
SpatialPaths
ActorsActionsActorsActions
PlansGoalsPlansGoals
TimeTime
AgentsAgents
SpaceSpace
PhysicalObjectsPhysicalObjects
HumanBeingsHumanBeings
Organ-izationOrgan-ization
HumanActivitiesHuman
Activities
LivingThingsLivingThings
SocialBehaviorSocial
Behavior
LifeFormsLife
Forms
AnimalsAnimals
PlantsPlants
EcologyEcology
NaturalGeography
NaturalGeography
Earth &Solar System
Earth &Solar System
PoliticalGeography
PoliticalGeography
WeatherWeather
General Knowledge about Various DomainsGeneral Knowledge about Various Domains
Cyc contains:17,000 Predicates
400,000 Concepts5,000,000 Assertions
Represented in:• First Order Logic• Higher Order Logic• Modal Logic• Context Logic• Micro-theories
Specific data, facts, and observationsSpecific data, facts, and observations
Machine Learning Techniques
Create Examples Model
Trainer
„Let the occurrence of the term ‚is host to‘ between a location and a
substance increase the probability that this is a location x substance relation
by 10%, because we have seen it more often in positive than in negative
examples.“
Good enough
?Deploy
yesno
Example: The Semantic Associative Search Method(MMM: The Mathematical Model of Meaning)
A
B
C
A
B
C
|| A || = || B || = || C ||
A
B
C
|| A || > || C || > || B ||
impression words(as a context):light, bright
impression words(as a context):dark, black
A,B,C: image data vectors
semantic space:2,000 dimensional space(presently)
(retrieval candidate image data)
2 2 2
w w w || C || > || B || > || A ||w w w
A: a sunny imageB: a silent imageC: a shady image
semanticsubspace
semanticprojection
semanticprojection
USP: 6,138,116Yasushi Kiyoki, 2009
Context Matters
Information Overload
Relevancy Overload
What’s important to me
right now
Audience-specific search experiences
User context
Inform-ation
context
Application context
Social context
Renee LoEngineeringContoso Consulting”What should I know about implementing ERP?”
Alan BrewerSales ManagerContoso Consulting”What should I know about selling ERP consulting?”
Username & Group Memberships
LocationLanguages
Business UnitDepartment
TeamTime of Day
Preferred SitesSharePoint Audiences
Interests & Current ProjectsContext of Current Task
53
Time is Money
Data = MetadataContent = Connections
57Image by Richard Cyganiak and Anja Jentzsch
Cloud
On- premise
Client
My Data
Enterprise Data
Social netw
orks
Web Data
TextData
People
Media
Apps
Behavior
Ads
ContentFusion
Contextual Matching
PervasiveSearch
INTENT
CONTENT
Smart Content needs Gardeners
From Documents to KnowledgeVa
lue
DocumentSearch
Finds documents containing terms
RelationshipExtraction
Finds relationships within documents
AssertionClustering
Finds assertions and the evidence for them
Profiling
Summarizes different kinds of information
Join
Creates indirect correlations and connections
Knowledge Management Framework
SocialIndividual
History
Event(transaction)
MappingContent management
Standardization
Findability
Common ground (practices, values, belief)
Typologies
Sense-makingCategory busting
Discovery
Coordination
Based on Organizing Knowledge: Taxonomies, Knowledge and Organizational Effectiveness, Patrick Lambe (not exact reproduction)
Culture
Collaboration
Expertise and learning
Information
Communities of Practice
Summary
Content Analytics involves• Gardeners• Context• Virtuous Cycles• Lots of cool, imperfect
technology
Smart Content is• Social• Mobile• Streaming• Exploding