Sensation and Perception - UL-Lafayette Computing Support Services
Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and...
-
Upload
amit-sheth -
Category
Education
-
view
3.613 -
download
0
description
Transcript of Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and...
![Page 1: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/1.jpg)
![Page 2: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/2.jpg)
2
![Page 3: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/3.jpg)
imagine
![Page 4: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/4.jpg)
imagine when
![Page 5: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/5.jpg)
meets
Farm Helper
![Page 6: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/6.jpg)
with this
• Latitude: 38° 57’36” N• Longitude: 95° 15’12” W• Date: 10-9-2007• Time: 1345h
![Page 7: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/7.jpg)
that is sent to
Geocoder
Farm Helper
Services Resource
Sensor Data ResourceStructured Data Resource
Agri DBSoil Survey
Lat-Long
Lawrence, KSWeather
data
Soil InformationPest information …
Weather Resource
LocationDate /Time
Weather Data
![Page 8: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/8.jpg)
and
![Page 9: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/9.jpg)
Six billion brains
![Page 10: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/10.jpg)
imagination today
![Page 11: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/11.jpg)
impacts our experience tomorrow
![Page 12: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/12.jpg)
![Page 13: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/13.jpg)
Computing For Human Experience
ASWC 2008 KeynoteAmit P. Sheth,
Lexis Nexis Eminent Scholar and Director, kno.e.sis center
Knoesis.org
![Page 14: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/14.jpg)
Technology that fits right in
“The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.
Machines that fit the human environment instead of forcing humans to enter theirs will make using a computer as refreshing as a walk in the woods.” Mark Weiser, The Computer for the 21st Century (Ubicomp vision)
“We're crying out for technology that will allow us to combine what we can do on the Internet with what we do in the physical world.” Ian Pearson in Big data: The next Google
14Get citation
http://tarakash.com/guj/tool/hug2.html
![Page 15: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/15.jpg)
(c) 2007 Thomas Gruber
15
But we are not talking about
Intelligent Design
Ubicomp: Mark Wisner and others
Intelligence @ Interface: Gruber –Your life - on-line: search, chat, music, photos, videos, email, multi-touch, mobile phone
![Page 16: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/16.jpg)
What is CHE? Beyond better human interaction
• Focus in the past (eg Ubicomp): How humans interact with the system (computer, Internet)
• Our focus—almost the reverse of the past (and both are needed)
• Computing for Human Experience is about:How computing serves, assists and collaborates with humans to complement and enrich their normal activities – nondestructively and unobtrusively, with minimal explicit concern
or effort on part of humans– anticipatory, knowledgeable, intelligent, ubiquitous– Computing that encompasses semantic, social, service, sensor
and mobile Web
16
![Page 17: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/17.jpg)
Principals of CHE
• Human is the master, system is the slave• Human sees minimal changes to normal behavior and
activity, system is there to serve/assist/support in human’s natural condition– Search, browsing, etc are not primary; HCI is not the focus– Getting the assistance and answers are important, improving
experience is key
• Multimodal and multisensory environment• Integrated and contextual application of (not just access
to) sensor data, databases, collective intelligence, wisdom of the crowd, conceptual models, reasoning
17
![Page 18: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/18.jpg)
Learning from a number of exciting visions
18
Vannevar Bush Trailblazing, Memex (As We May Think) 1945
Mark Weisner Ubicomp (The Computer for 21st Century) 1991
A. Sheth, V. Kashyap, et al Semantic Information Brokering/ Metadata Reference Link 1994 - 2000
Tim Berners-Lee Semantic Web 1999 -
Gordon Bell et al MyLifeBits 2002
Jonathan Rossiter Humanist Computing 2003
Ramesh Jain Experiential Computing 2003
ITU The Internet of Things 2005
Ramesh Jain Event Web 2006-2008
A. Sheth et al Relationship Web (also, Spatio-temporal-thematic analysis, Semantic Sensor Web)
2006-2008
Tom Gruber Intelligence @Interface 2008
![Page 19: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/19.jpg)
Evolution of the Web (and associated computing)
Web of pages - text, manually created links - extensive navigation
2007
1997
Web of databases - dynamically generated pages - web query interfaces
Web of resources - data, service, data, mashups - ubiquitous/mobile computing
Web of people - social networks, user-created casual content - Twine, GeneRIF, Connotea
Web as an oracle / assistant / partner - “ask the Web”: using semantics to leverage text + data + services - Powerset
Sem
antic
Tec
hnol
ogy
Use
d
Computing for Human Experience
![Page 20: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/20.jpg)
Sensing, Observing, Perceptual, Semantic, Social, Experiential
CHE components and enablers
![Page 21: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/21.jpg)
Consider
• that all objects, events and activities in the physical world have a counterpart in the Cyberworld (IoT)
• multi-facted context of real world is captured in the cyberworld (sensor web, citizen sensor)
• each object, event and activity is represented – with semantic annotations (semantic sensor web)
• for a chosen context, with an ability to explicate and associate variety of relationships and events (Relationship Web, EventWeb)
• appropriate reasoning and human/social interaction are available and applied, insights extracted (semantic web, social semantic web, experiential computing)
• answers obtained/ decisions reached/communicated/applied
21
![Page 22: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/22.jpg)
Paradigm shift …
• Where humans act as sensors or observers• Around them is a network of sensors, computing and
communicating with each other– Processing and delivering multi-modal information – Collective Intelligence
• Information-centric to Experience-centric era– Modeling, processing, retrieving event level information
• Use of domain knowledge – ….
• Understanding of casual text
22
![Page 23: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/23.jpg)
Today’s Sensor Network Types
• Inert, fixed sensors
• Carried on moving objects– Vehicles, pedestrians (asthma research)– anonymous data from GPS-enabled
vehicles, toll tags, and cellular signaling
to mark how fast objects are moving – and overlaying that information with location data and maps (traffic.com, Nokia experiment, …)
23Text from http://www.geog.ucsb.edu/~good/presentations/icsc.pdf Images credit – flickr.com, cnet.com
![Page 24: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/24.jpg)
Today’s Network of Sensors
• Are sensing, computing, transmitting
• Are acting in concert – Sharing data – Processing them into meaningful digital representations of the
world
• Researchers using 'sensor webs' to ask new questions or test hypotheses
24
![Page 25: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/25.jpg)
Machine sensing
25
![Page 26: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/26.jpg)
26
Semantic Sensor Web
![Page 27: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/27.jpg)
27
Semantically Annotated O&M
<swe:component name="time"><swe:Time definition="urn:ogc:def:phenomenon:time" uom="urn:ogc:def:unit:date-time">
<sa:swe rdfa:about="?time" rdfa:instanceof="time:Instant"><sa:sml rdfa:property="xs:date-time"/>
</sa:swe></swe:Time>
</swe:component><swe:component name="measured_air_temperature">
<swe:Quantity definition="urn:ogc:def:phenomenon:temperature“ uom="urn:ogc:def:unit:fahrenheit"><sa:swe rdfa:about="?measured_air_temperature“
rdfa:instanceof=“senso:TemperatureObservation"><sa:swe rdfa:property="weather:fahrenheit"/><sa:swe rdfa:rel="senso:occurred_when" resource="?time"/><sa:swe rdfa:rel="senso:observed_by" resource="senso:buckeye_sensor"/>
</sa:sml></swe:Quantity>
</swe:component>
<swe:value name=“weather-data">2008-03-08T05:00:00,29.1
</swe:value>
![Page 28: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/28.jpg)
28
Person
Company
Coordinates
Coordinate System
Time Units
Timezone
SpatialOntology
DomainOntology
TemporalOntology
Mike Botts, "SensorML and Sensor Web Enablement," Earth System Science Center, UAB Huntsville
Semantic Sensor ML – Adding Ontological Metadata
![Page 29: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/29.jpg)
29
Semantic Query
• Semantic Temporal Query
• Model-references from SML to OWL-Time ontology concepts provides the ability to perform semantic temporal queries
• Supported semantic query operators include:– contains: user-specified interval falls wholly within a sensor reading
interval (also called inside)– within: sensor reading interval falls wholly within the user-specified
interval (inverse of contains or inside)– overlaps: user-specified interval overlaps the sensor reading interval
• Example SPARQL query defining the temporal operator ‘within’
![Page 30: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/30.jpg)
Citizen Sensors
• Human beings– 6 billion intelligent sensors– informed observers– rich local knowledge– uplink technology
• broadband Internet• mobile phone
30
Christmas Bird Count
![Page 31: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/31.jpg)
Citizen Science
• Networks of amateur observers– possibly trained, skilled
• Christmas Bird Count http://www.audubon.org/bird/citizen/index.html ,
http://www.audubon.org/bird/cbc/index.html
– thousands of volunteer participants– Protocols
• Project GLOBE– an international network of school children– reporting environmental conditions– central integration and redistribution
31
![Page 32: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/32.jpg)
Citizen Sensor – Humans Actively Engage
• In connecting, searching, processing, stitching together information
• Asks, gets.. Asks again, gets again…
32Images credit – flickr.com
![Page 33: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/33.jpg)
Recent example - #Mumbai
33
![Page 34: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/34.jpg)
34
![Page 35: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/35.jpg)
35
![Page 36: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/36.jpg)
36
![Page 37: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/37.jpg)
37
![Page 39: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/39.jpg)
Twitter In Controversial Spotlight Amid Mumbai Attacks
Posted by Alexander Wolfe, Nov 29, 2008 11:27 AM
• Never before has a crisis unleashed so much raw data -- and so little interpretation -- than what we saw as the deadly terrorist attacks in Mumbai, India unfolded. Amid the real-time video feeds (kudos to CNN International), cellphone pictures, and tweets, we were able to keep abreast of what seemed to be happening, and where it was going down, all the while not really knowing those other key, canonical components of journalistic information gathering -- namely, who or why.
39
![Page 40: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/40.jpg)
• In fairness, no one did. There were so many tentacles to these heinous attacks, and multiple hot spots (the Taj and Oberoi hotels and the Chabad Jewish center, to name the three most prominent), that even the Indian government likely didn't have a handle on things until late in the game. My point here is not criticize, but simply to note that I was struck, as never before, by the ability of data to outstrip information.
40
![Page 41: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/41.jpg)
Alexander Wolfe continues …
• I'd add that Mumbai is likely to be viewed in hindsight as the first instance of the paradigmatic shift in crisis coverage: namely, journalists will henceforth no longer be the first to bring us information. Rather, they will be a conduit for the stream of images and video shot by a mix of amateurs and professionals on scene.
• You've got to add to this the immense influence of Twitter. In the past few days, I've seen a slew of stories pointing out how Twitter was a key source of real-time updates on the attacks, to the point that Indian authorities asked people to stop posting to the microblogging service (for fear that they might be giving away strategic information to the terrorists).
41
![Page 42: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/42.jpg)
Analyzing Citizen Data
42
Semantic Data Store
Citizen-Sensor Stream Aggregation
Term Extraction
Cloud/ Web based Services
Semantic Analysis and Annotation
Model Enrichment
Citizen Sensor Streams
Image Courtesy: Chemical Brothers, Galvanize
Twitter RSS/ATOM feedsBlogs
Twitter: “loud bang near the Taj” [ location:mumbai]
CNN News feed: “terrorist activity reported in Mumbai Hotel” [ location:mumbai]
Location based aggregation
loud bang near the Taj
Typeof:hotel , part of:Hotel_chain
Typeof:noise, relatedto:explosion
Event:explosionLocation:Taj Hotel, MumbaiDueTo: terrorist activity
![Page 43: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/43.jpg)
Online and offline worlds
• Computational abstractions to represent the physical world’s dynamic nature
• Merging online and offline activities– Connecting the physical world naturally with the online world
• What are natural operations on these abstractions?
• How do we detect these abstractions based on other abstractions and multimodal data sources?
43
![Page 44: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/44.jpg)
Enriching Human Experience
• Recognition of objects (IOT) and models of object• Understanding of objects and content• Multimodal interfaces• Multi(level) sensing and perception• From keywords and entities to events and rich sets of
relationships; spatio-temporal-thematic computing• Models (ontologies, folkonomies, taxonomies,
classification, nomenclature)– time, location, sensors, domain
• More powerful reasoning: paths, patterns, subgraphs that connect related things; deductive and abductive reasoning, ….
44
![Page 45: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/45.jpg)
Understanding content … informal text
• I say: “Your music is wicked”
• What I really mean: “Your music is good”
45
![Page 46: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/46.jpg)
Structured text (biomedical literature)
Multimedia Content and Web
data
Web Services
Semantic Metadata: Smile is a TrackLil transliterates to Lilly Allen
Lilly Allen is an Artist
Informal Text (Social Network
chatter)
Your smile rocks Lil
Urban Dictionary
MusicBrainz Taxonomy
Artist: Lilly AllenTrack: Smile
Sentiment expression: Rocks Transliterates to: cool, good
![Page 47: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/47.jpg)
Example: Pulse of a Community
• Imagine millions of such informal opinions– Individual expressions to mass opinions
• “Popular artists” lists from MySpace comments
Lilly Allen
Lady Sovereign
Amy Winehouse
Gorillaz
Coldplay
Placebo
Sting
Kean
Joss Stone
![Page 48: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/48.jpg)
• How do you get comprehensive situational awareness by merging “human sensing” and “machine sensing”?
48
![Page 49: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/49.jpg)
Synthetic but realistic scenario
• an image taken from a raw satellite feed• an image taken by a camera phone with an associated label,
“explosion.” These two images, one from a wide-area sensor and the other from a citizen-sensor, can be correlated with spatial and temporal attributes in order to provide comprehensive situational awareness.
49
![Page 50: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/50.jpg)
Kno.e.sis’ Semantic Sensor Web
50
![Page 51: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/51.jpg)
51
Data
• Raw Phenomenological Data
DATA-TO-KNOWLEDGE ARCHITECTURE
Information
• Entity Metadata
• Feature Metadata
Knowledge
• Object-Event Relations
• Spatiotemporal Associations
• Provenance/Context
Feature Extraction and Entity Detection
Data Storage(Raw Data, XML, RDF)
Semantic Analysis and Query
Sensor Observation
Ontologies• Space Ontology
• Time Ontology
• Domain Ontology
SemanticAnnotation
RHO DELTA SIGMA&=|
![Page 52: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/52.jpg)
On Our Way.. Multimodal interfaces
• We are already seeing efforts toward this larger goal– Video visors - computer image superimposed over the world
around you.
52
![Page 53: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/53.jpg)
Challenges – Multiple Modalities
• Multiple modalities of objects and events
• How do we organize and access multimodal data?
• How do we organize, index, search and aggregate events and multimedia experiences as effectively as modern search engines do for text using keywords?
53
![Page 54: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/54.jpg)
Objects to Events
• If we move from this object mode to an event mode
– A single user action or request or sensory observation could act as a cue for getting all (multi-modal) information associated with an event
– If conditions change, systems could even modify their behavior to suit their changing view of the world
Today text is most prevalent, with increasing but disparate (non-integrated) image and video data, but human experience is event based (at higher levels of abstractions) formed based on multi-sensory, multi-perception (at lower level of abstraction) observations
54
![Page 55: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/55.jpg)
• We are already seeing efforts toward this larger goal– Social connections, interests, locations, alerts, comment– Mobile phone to social compass: LOOPT.com
55Image credit - www.movilae.com
On our way…
![Page 56: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/56.jpg)
On our way…Internet of Things
Internet of Things: “A world where inanimate objects communicate with us and one another over the network via tiny intelligent objects” - Jean Philippe Vasseur, NSSTG Systems
56Image credit - www.forbes.com
![Page 57: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/57.jpg)
Building models…seed word to hierarchy creation using WIKIPEDIA
Seed Query
BWikipedia
Fulltext Concept Search
Wikigraph-Based expansion
Graph Search
Graph Search
Graph Search
Hierarchy Creation
Query: “cognition”
![Page 58: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/58.jpg)
Learn Patterns that indicate Relationships
We know that countries have capitals.Which one is Australia’s?
Query:Australia Sidney
Australia Canberra• in Sydney, New South Wales, Australia• Sydney is the most populous city in Australia
• Canberra, the Australian capital city• Canberra is the capital city of the Commonwealth of
Australia• Canberra, the Australian capital
• in Sydney, New South Wales, Australia• Sydney is the most populous city in Australia
• Canberra, the Australian capital city• Canberra is the capital city of the Commonwealth of
Australia• Canberra, the Australian capital
![Page 59: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/59.jpg)
Experience
Direct Observation of or Participation in
Events as a basis of knowledge
![Page 60: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/60.jpg)
Entities and Events
Events are dynamic.
Event
Name
Duration
Location
AttributesData-streams
ProcessesAdjacent StatesRelated Links
Entity
Name
Attributes
Processes(Services)
Objects and Entities
are static.
Thanks – Ramesh Jain
![Page 61: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/61.jpg)
Strategic Inflection Points
Documents on Web
(Information)
Events on Web(Experience)
1995 2000 2005 2010
Keyword Search
SemanticSearch
ContextualSearch
ImmersiveExperience
Updates and alerts
Ubiquitous Devices
Thanks – Ramesh Jain
![Page 62: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/62.jpg)
Thanks – Ramesh Jain
![Page 63: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/63.jpg)
EventWeb [Jain], RelationshipWeb [Sheth]
Suppose that we create a Web in which• Each node is an event or object • Each node may be connected to other nodes using
– Referential: similar to common links that refer to other related information.
– Spatial and temporal relationships.– Causal: establishing causality among relationships. – Relational: giving similarity or any other relationship. – Semantic or Domain specific:
• Familial • Professional• Genetics,…
63Adapted from a talk by Ramesh Jain
![Page 64: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/64.jpg)
Karthik Gomada
m
Amit Sheth
Attended Google IO
Moscone Center, SFOMay 28-29, 2008
Is advised by
Ph.D Student Researcheris_advised_by
Assistant Professor
Professor
Research Paper
publishes
Journal Conference
Location
published_in published_in
has_location
Image Metadata
Event
Causal
Domain Specific
kno.e.sis
DirectsRelational
Spatio- te
mporal
![Page 65: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/65.jpg)
Structured text (biomedical literature)
Informal Text (Social Network
chatter) Multimedia Content and Web data
Web Services
Metadata Extraction
Patterns / Inference / Reasoning
Domain Models
Meta data / Semantic Annotations
Relationship Web
Search
Integration
Analysis
Discovery
Question Answering
![Page 66: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/66.jpg)
HOW ARE HARRY POTTER AND DAN BROWN RELATED?
Leonardo Da Vinci
The Da Vinci code
The Louvre
Victor Hugo
The Vitruvian man
Santa Maria delle Grazie
Et in Arcadia EgoHoly Blood, Holy Grail
Harry Potter
The Last Supper
Nicolas Poussin
Priory of Sion
The Hunchback of Notre Dame
The Mona Lisa
Nicolas Flammel
painted_by
painted_by
painted_by
painted_by
member_of
member_of
member_of
written_by
mentioned_in
mentioned_in
displayed_at
displayed_at
cryptic_motto_of
displayed_at
mentioned_in
mentioned_in
![Page 67: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/67.jpg)
Challenges – Complex Events
• Formal framework to model complex situations and composite events – Those consisting of interrelated events of varying spatial and
temporal granularity, together with their multimodal experiences
• What computational approaches will help to compute and reason with events and their associated experiences and objects ?
67
![Page 68: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/68.jpg)
However, today
68
Sensors capture and process uni-modal information. Bringing multiple modalities
together is up to an application
Object centric environments – sensors understand objects from data. Events
and not objects lend to holistic views of an experience
Multi-modal information effectively represents
events
Human to machine to human to machine..
Where is the Domain knowledge!?
![Page 69: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/69.jpg)
THE SEMANTICS OF OBSERVATION
• Observation is about capturing (or measuring) phenomena.
• Perception is about explaining the observations.
• When the human mind perceives what it observes– It uses what it already knows in addition to the context
surrounding the observation– Cause-effect relationships play a vital role in how we reach
conclusions
69
![Page 70: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/70.jpg)
ABDUCTION
A formal model of inference which centers on cause-effect relationships and tries to find the best or most plausible explanations (causes) for a set of given observations (effects).
70
![Page 71: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/71.jpg)
FORMS OF REASONING
• Deduction (Prediction)– fact a, rule a => b – INFER b (First-order logic)– Reasoning/inferring from causes to effects
• Abduction (Explanation)– rule a => b, observe b – POSSIBLE EXPLANATION a (different formalizations)– Explaining effects by hypothesizing causes
• Induction (Learning)– observe correlation between a1, b1, ... an, bn– LEARN a -> b– Learning connections/rules from observations
71
![Page 72: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/72.jpg)
Perception as Abduction
• The task of abductive perception is to find a consistent set of perceived objects and events (DELTA), given a background theory (SIGMA) and a set of observations (RHO)
SIGMA && DELTA |= RHO
72
Murray Shanahan, "Perception as Abduction: Turning Sensor Data Into Meaningful Representation"
![Page 73: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/73.jpg)
WHAT WE KNOW + WHAT WE OBSERVE = EUREKA!
• SIGMA & DELTA |= RHO
• Sensor data to knowledge– Semantic annotation (DELTA) of sensor observations (RHO)
using contextual domain knowledge (SIGMA)
73
![Page 74: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/74.jpg)
WHAT WE KNOW + WHAT WE OBSERVE = EUREKA!
• SIGMA & DELTA |= RHO
• Understanding casual text– Meaning of (DELTA) slang sentiment expressions (RHO) using
dictionaries (SIGMA)
74
![Page 75: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/75.jpg)
WHAT WE KNOW + WHAT WE OBSERVE = EUREKA!
• SIGMA & DELTA |= RHO
• Discovering undiscovered knowledge– Connecting (DELTA) seemingly unrelated text (RHO) using
domain knowledge (SIGMA)
75
![Page 76: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/76.jpg)
WHAT WE KNOW + WHAT WE OBSERVE = EUREKA!
• SIGMA & DELTA |= RHO
• Taxonomy Creation– Seed word (RHO) to hierarchy creation (DELTA) using Wikipedia
(SIGMA)
76
![Page 77: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/77.jpg)
REVEALING CONNECTIONS IN BIOMEDICAL TEXT
RHO (observations) – Text from PubMed
SIGMA (what we know) - Background knowledge from UMLS (schema), Mesh (instances)
DELTA (perceived connections) – relationships connecting entities
![Page 78: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/78.jpg)
Observation
(senses)
Observation
(sensors)
Perception
(analysis)
Perception
(cognition)
Communication
(language)
Communication
(services)
People Web(human-centric)
Sensor Web(machine-centric)
![Page 79: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/79.jpg)
Observation
PerceptionCommunication
Enhanced Experience (humans & machines working in
harmony)
Semantics for shared conceptualization and interoperability between machine and human
Ability to share common communication
![Page 80: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/80.jpg)
1. Sensors observe environmental phenomena and nearby vegetation.
• Through abductive reasoning, observation analysis perceives a possible storm as the best explanation hypothesis for observed phenomena.
• Through predictive deductive reasoning, observation analysis determines the effect on the crops, including the potential for the poisoning of the soil from salt carried from the ocean in the wind.
• Through query against a knowledge base of the agriculture domain, observation analysis determines that the best remedy
• for saline soil is to “leach” the soil with excess irrigation water in order to ‘push’ the salts below the crop root zone,
• for sodic soil is to add gypsum before leaching.
Example
2. Observation analysis determines potential situation and effects.
![Page 81: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/81.jpg)
1. Sensors observe environmental phenomena and nearby vegetation.
3. System alerts nearby farmers of situation and possible remedy.
4. Farmer goes outside and looks at the sky and crops.
5. Farmer perceives high-winds and dark rain clouds over the ocean view and agrees with system perception.
6. Farmer calls children and neighbors to help take the necessary precautions to save the vegetables.
Example
2. Observation analysis determines potential situation and effects.
![Page 82: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/82.jpg)
PARADIGM SHIFT
Sensing, Observation, Perception, Semantic, Social Experiential
82
![Page 83: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/83.jpg)
From the Semantic Web Community
• Several key contributing research areas– Operating Systems, networks, sensors, content management
and processing, multimodal data integration, event modelling, high-dimensional data visualization ….
• Semantics and Semantic technologies can play vital role – In the area of processing sensor observations, the Semantic
Web is already making strides– Use of core SW capabilities: knowledge representation, use of
knowledge bases (ontologies, folkonomies, taxonomy, nomenclature), semantic metadata extraction/annotation, exploiting relationships, reasoning
83
![Page 84: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/84.jpg)
EventWeb: Experience Age
FamilySports
Fun
Knowledge
Personal
Finance
Office
I am borrowing from Experiential computing and EventWeb – credit Ramesh Jain
![Page 85: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/85.jpg)
THERE IS MORE HAPPENING AT KNO.E.SIS
http://knoesis.org
85
Thanks: NSF (SemDis, Spatio-temporal-thematic), NIH, AFRL,and also Microsoft Research, HP Research, IBM Research. Seehttp://knoesis.wright.edu/projects/funding/
![Page 86: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/86.jpg)
KNO.E.SIS MEMBERS – A SUBSET
![Page 87: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/87.jpg)
Influential Works
• V Bush, As We May Think, The Atlantic, July 1945. [Memex, trail blazing]• Mark Weiser, The Computer for the Twenty-First Century, Scientific American, Sept 1991, 94-10.
[The original vision paper on ubicomp. Expansive vision albeit technical aspects focused on HCI with networked tabs, pads and boards.]
• V. Kashyap and A. Sheth, Semantics-based information brokering. Third ACM Intl Conf on Information and Knowledge Management (CIKM94), Nov 29 - Dec 02, 1994. ACM, New York, NY. [semantics based query processing (involving multiple ontologies, context, semantic proximity) across a federated information sources across the Web]
• Abowd, Mynatt, Rodden, The Human Experience, Pervasive computing, 2002. [explores Mark Wisner’s original ubicomp vision]
• Jonathan Rossiter , Humanist Computing: Modelling with Words, Concepts, and Behaviours , in Modelling with Words, Springer, 2003, pp. 124-152 [modelling with words, concepts and behaviours defines a hierarchy of methods which extends from the low level data-driven modelling with words to the high level fusion of knowledge in the context of human behaviours ]
• Ramesh Jain, Experiential computing. Commun. ACM 46, 7, Jul. 2003, 48-55. • Amit Sheth, Sanjeev Thacker, and Shuchi Patel, Complex Relationship and Knowledge Discovery
Support in the InfoQuilt System, VLDB Journal, 12 (1), May 2003, 2–27. [complex semantic inter-domain (multi-ontology) relationships including causal relationships to enable human-assisted knowledge discovery and hypothesis testing over Web-accessible heterogeneous data]
87
![Page 88: Computing for Human Experience: Sensors, Perception, Semantics, Social Computing, Web N.0, and beyond](https://reader038.fdocuments.in/reader038/viewer/2022110306/554aef5eb4c905ba058b58a6/html5/thumbnails/88.jpg)
• Ambjörn Naeve: The Human Semantic Web: Shifting from Knowledge Push to Knowledge Pull. Int. J. Semantic Web Inf. Syst. 1(3): 1-30 (2005) [discusses conceptual interface providing human-understandable semantics on top of the ordinary (machine) Semantic Web]
• Ramesh Jain, Toward EventWeb. IEEE Distributed Systems Online 8, 9, Sep. 2007. [a web of temporally related events… informational attributes such as experiential data in the form of audio, images, and video can be associated with the events]
• The Internet of Things, International Telecommunication Union, Nov 2005.
Other Closely Related publications• Amit Sheth and Meena Nagarajan, Semantics empowered Social Computing, IEEE Internet Computing, Jan-Feb
2009.• Amit Sheth, Cory Henson, and Satya Sahoo, "Semantic Sensor Web," IEEE Internet Computing, July/August
2008, p. 78-83. • Amit Sheth and Matthew Perry, “Traveling the Semantic Web through Space, Time and Theme,” IEEE Internet
Computing, 12, (no.2), February/March 2008, pp.81-86. • Amit Sheth and Cartic Ramakrishnan, “Relationship Web: Blazing Semantic Trails between Web Resources,”
IEEE Internet Computing, July–August 2007, pp. 84–88.
88