Pratik Desai Ph.D dissertation defense

55
Department of Electrical Engineering A Semantic Situation Awareness Framework for Indoor Cyber- Physical Systems Dissertation Committee Director Dr. Kuldip Rattan Co- Director Dr. Amit Sheth Dr. Marian Kazimierczuk Dr. Frank Zhang Dr. Guru Subramanyam Ph.D. in Engineering Dissertation Defense Pratikkumar Desai Monday, 4/29/2013

description

Recently, the domain of cyber-physical systems (CPSs) has emerged as a successor to the traditional embedded systems and the wireless sensor networks. The relatively new cyber-physical domain offers tight integration of control, communication and computation components to develop advanced web based application in various heterogeneous domains such as health care, disaster management, automation and environment monitoring. The applications of indoor CPSs include remote patient monitoring, smart home, etc. with focus on situation awareness via event identification from context information. The principal challenges associated with the development of situation awareness applications include uncertainty in contextual data, incomplete domain knowledge, interoperability between interconnected systems and effective utilization of spatial information. This dissertation addresses these challenges by providing a comprehensive situation awareness framework for event comprehension utilizing raw sensor data and spatial information. Semantic web based annotation and mapping techniques are used to provide interoperability. The framework contains contextual situation awareness and location awareness stages towards achieving effective event assessment. The contextual situation awareness stage provides fuzzy abductive reasoning based architecture to transform raw physical sensor data to low-level fuzzy abstraction. These abstractions are used for event assessment with associated degree of certainty. The location awareness stage includes methodologies to hierarchically map indoor objects and define the object-event relationship in ontology, which is further exploited for event discrimination. This dissertation also presents a fusion based indoor positioning algorithm to provide accurate spatial information to assist location awareness. The algorithm uses extensive training of received signal strength (RSS) and time difference of arrival (TDoA) signals to estimate distance and position. The comprehensive framework is evaluated through an implementation of simulated indoor fire in a controlled environment.

Transcript of Pratik Desai Ph.D dissertation defense

Page 1: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

A Semantic Situation Awareness Framework for Indoor Cyber-Physical Systems

Dissertation Committee

Director Dr. Kuldip Rattan

Co-Director Dr. Amit Sheth

Dr. Marian Kazimierczuk

Dr. Frank Zhang

Dr. Guru Subramanyam

Ph.D. in Engineering Dissertation Defense

Pratikkumar DesaiMonday, 4/29/2013

Page 2: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Embedded systemse.g. thermostat

Networked embedded systeme.g. wireless sensor networks

Cyber-physical systeme.g. Intelligent traffic management systems

Page 3: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Cyber-Physical Systems

http://www.cs.binghamton.edu/~tzhu/

Cyber : Computation, communication, and control that are discrete, logical, and switched.

Physical : Natural and human-made systems governed by the laws of physics and operating in continuous time.

Cyber-Physical Systems (CPS) : Systems in which the cyber and physical systems are tightly integrated at all scales and levels

Page 4: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

CPS Examples

Disaster Management

SmartGrid

SmartHome

Remote PatientMonitoring

Traffic Management

Air TrafficControl

Military Drones

Page 5: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Motivation & Challenges(Situation awareness)

Page 6: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Motivation

Mobile sensing platform

Situation: Actual fireat chair

Page 7: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Motivation

Mobile sensing platform

Event : Firefrom temperature and CO2

data

Event : Firefrom temperature and CO2

data

Page 8: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Motivation

Mobile sensing platform

Uncertainty: Sensor datae.g. Due to resoultion, calibration or robustness of sensors

Page 9: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Motivation

Mobile sensing platform

Incomplete domain knowledgee.g. Unknown sources in the environment

Page 10: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Interoperability

Indoor location awareness

Complex system

Page 11: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Context

Environmentalcontext

Location

e.g. temperature,CO2, heart rate

e.g. coordinates

Contextualsituation

awareness

Location awareness

Contextual situation awareness:“is a process of comprehending meaning of environmental context in terms of events or entities”

Location awareness:“is a process of identifying objects from raw spatial information and their relationship with the ongoing events”

Context“is a physical phenomenon, measured using sensors, and product of an event”

Page 12: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Contextual situation awareness + Location awareness

Raw environmental sensor data

Raw spatial information

Entities (High level abstractions)

Object-Entity relationships

Situation

Page 13: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Contextual Situation Awareness

Page 14: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

IntellegO

0

100

C

temperature

CO2

0

1000

ppm

Fire

DryIce

RoomHeater

NormalCondition

Quality-type EntityQuality

LowTemp

HighTemp

LowCO2

HighCO2

• Abductive reasoning• Crisp abstractions

http://wiki.knoesis.org/index.php/Intellego

Domain Knowledge Base

Page 15: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Temperature: 500 C HighTemp

CO2: 1010 ppm HighCO2

Temperature: 500 C HighTemp

CO2: 999 ppm LowCO2

Fire

DryIce

RoomHeater

RoomHeater

Page 16: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Motivation

Mobile sensing platform

Uncertainty: Sensor datae.g. Due to limitation, calibration or robustness of sensors

Incomplete domain knowledgee.g. Unknown sources in the environment

Page 17: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Fuzzy abstractions

 

Membership function

0

1 LowCO2

ppm800

HighCO2

1200

1160 ppm

0.9

0.1

µ

a

Page 18: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Fuzzy abductive reasoning

500 °C

1160 ppm

0

80

120

C

temperature

CO2

0

800

1200

ppm

LowTemp Fire

DryIce

RoomHeater

NormalCondition

Quality-type EntityQuality

HighTemp

LowCO2

HighCO2

𝜇𝐹𝑖𝑟𝑒 (𝑎 )=𝜇 h𝐻𝑖𝑔 𝑇𝑒𝑚𝑝 (𝑎 )∧𝜇 h𝐻𝑖𝑔 𝐶𝑂2(𝑎 )

Page 19: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

𝑖𝑜 :𝑒𝑛𝑡𝑖𝑡𝑦≡ {∃𝑖𝑜 : h𝑖𝑛 𝑒𝑟𝑒𝑠𝐼𝑛 . { h𝐻𝑖𝑔 𝐶𝑂2 }⊔∃𝑖𝑜 : h𝑖𝑛 𝑒𝑟𝑒𝑠𝐼𝑛 . {𝐿𝑜𝑤𝐶𝑂2 } }⊓ {∃𝑖𝑜 : h𝑖𝑛 𝑒𝑟𝑒𝑠𝐼𝑛 . { h𝐻𝑖𝑔 𝑇𝑒𝑚𝑝 }}

≡{𝐹𝑖𝑟𝑒 ,𝑅𝑜𝑜𝑚𝐻𝑒𝑎𝑡𝑒𝑟 }

Page 20: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Evaluation – Contextual Situation Awareness

Reasoning approach Accuracy Precision Recall

Crisp abductive reasoning 86 % 78.57 % 73.33 %

Fuzzy abductive reasoning 94 % 92.85 % 86.66 %

Page 21: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Interoperability

Complex system

Page 22: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Semantic Web• Semantic web:

• Formally define the meaning of information on web.• Provide expressive representation, formal analysis of resources.

• Ontology• Formally represents knowledge as a set of concepts within a domain and the

relationships between pairs of concepts.

• RDF (Resource Description Framework)• Graph-based language for modeling of information.• Allows linking of data through named properties.

http://www-ksl.stanford.edu/kst/what-is-an-ontology.html

Subject ObjectPredicate

HighTempInheresIn

Fire

Page 23: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Raw Sensor Data

SSN annotated

Observations

Low-level fuzzy

abstractions(qualities)

Fuzzy reasoning

High-levelAbstractions(entity)

SSNOntology

Domain Ontology Fuzzy Inference rules ontology

FuzzificationRules

Observation process Perception process

Contextual situation awareness (Semantic modeling)

Page 24: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Indoor Localization

Page 25: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Indoor location awareness

Page 26: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Traditional Indoor Localization Techniques

• Active Badge and Active Bat system.

• RADAR: An In-building RF-based user location and tracking system.

• RFID radar

• Object tracking with multiple cameras• Computer vision based localization

• Wireless Sensor Network

RF

Camera

TDoA

Page 27: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

TDoA (Time Difference of Arrival)

Listener

Beacon

RF Signal

Ultraso

nic Signal

Stage 1: Beacon transmits RF and Us signals together

Stage 2: Listener receives RF signal first at Trf and

starts the clockStage 3: At Tus time US signals is received

by Listener

Stage 3: Distance is calculated using ∆T and speed of signals

Page 28: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

TrilaterationNumber of nodes = 3.

Outlier rejection and Multilateration

Page 29: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

The Proposed Algorithm• Utilizes fusion of RSS (received signal strength) of RF

signal and TDoA data for accurate distance estimation.• The algorithm stages:-

• RSSI data training• Distance estimation• Localization

• Uses TDoA as a primary distance estimation technique.• RSSI data is trained and converted into appropriate

distance measurements.• The proposed algorithm can be used in absence of one or

many TDoA links.

Page 30: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Initial Conditions

• Distances between all beacons are known and fixed

Page 31: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

0 ? ? ?? 0 ? ?? ? 0 ?? ? ? 0

L

B4

B2

B3

B1

0 ? ? ?? 0 ? ?? ? 0 ?

R14 ? ? 0

0 ? ? ?R12 0 ? ?? ? 0 ?? ? ? 0

0 ? ? ?? 0 ? ?

R13 ? 0 ?? ? ? 0

0 ? ? ? R1L T1L

? 0 ? ? ? ?? ? 0 ? ? ?? ? ? 0 ? ?

RSSI LinkTDoA Link

Beacon B1 Transmit Data

Page 32: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

0 R21 ? ?R12 0 ? ?? ? 0 ?? ? ? 0

L

B4

B2

B3

B1

0 ? ? ?R12 0 ? ?? ? 0 ?

R14 R24 ? 0

0 ? ? ?R12 0 ? ?? ? 0 ?? ? ? 0

0 ? ? ?R12 0 ? ?R13 R23 0 ?? ? ? 0

0 ? ? ? R1L T1L

R12 0 ? ? R2L T2L

? ? 0 ? ? ?? ? ? 0 ? ?

RSSI LinkTDoA Link

Beacon B2 Transmit Data

Page 33: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

0 R21 R31 ?R12 0 ? ?R13 R23 0 ?? ? ? 0

L

B4

B2

B3

B1

0 ? ? ?R12 0 ? ?R13 R23 0 ?R14 R24 R34 0

0 ? ? ?R12 0 R32 ?R13 R23 0 ?? ? ? 0

0 ? ? ?R12 0 ? ?R13 R23 0 ?? ? ? 0

0 ? ? ? R1L T1L

R12 0 ? ? R2L T2L

R13 R23 0 ? R3L T3L

? ? ? 0 ? ?

RSSI LinkTDoA Link

Beacon B3 Transmit Data

Page 34: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

0 R21 R31 R41

R12 0 ? ?R13 R23 0 ?R14 R24 R34 0

L

B4

B2

B3

B1

0 ? ? ?R12 0 ? ?R13 R23 0 ?R14 R24 R34 0

0 ? ? ?R12 0 R32 R42

R13 R23 0 ?R14 R24 R34 0

0 ? ? ?R12 0 ? ?R13 R23 0 R43

R14 R24 R34 0

0 ? ? ? R1L T1L

R12 0 ? ? R2L T2L

R13 R23 0 ? R3L T3L

R14 R24 R34 0 R4L T4L

RSSI LinkTDoA Link

Beacon B4 Transmit Data

Page 35: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Evaluation– Proposed Algorithm

Page 36: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Location Awareness

Page 37: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Bed-1

Chair-2

Chair-1

Sofa-1

Fireplace-1

Treadmill-1

Desk-1

Stove-1

Bedroom-1 Bedroom-2

Gym-1

Drawingroom-1

Kitchen-1

Plant-1

IndoorEnvironment

Page 38: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Hierarchical mapping ofthe indoor environment

Page 39: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Chair-1Drawingroom-

1

xsd:float

xsd:float

xsd:float

xsd:float

xsd:float

xsd:float

xsd:string

ChairPOI

POI

Drawingroom

inLo:isLocatedIn

is-a

has individual

is-a

StructuralComponent

has individual

inLo:hasUnit

inLo

:has

Xmax

inLo:hasXmin

inLo:hasYmax

inLo:hasYmininLo:hasZmaxinLo:hasZm

in

inLo:hasPOI

Def

inin

g C

over

age

spac

e (C

hair-

1)

Page 40: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

𝐼𝑑𝑒𝑛𝑡𝑖𝑓𝑖𝑒𝑑𝑃𝑂𝐼

≡ {𝑆𝑜𝑓𝑎−1 , h𝐶 𝑎𝑖𝑟 −1 ,𝐹𝑖𝑟𝑒𝑝𝑙𝑎𝑐𝑒−1 }

≡{ h𝐶 𝑎𝑖𝑟 −1 }

⊓ {𝑆𝑜𝑓𝑎−1 ,𝐶 h𝑎𝑖𝑟 −1 ,𝐹𝑖𝑟𝑒𝑝𝑙𝑎𝑐𝑒−1 }

⊓ {𝑆𝑜𝑓𝑎−1 ,𝑃𝑙𝑎𝑛𝑡−1 ,𝐹𝑖𝑟𝑒𝑝𝑙𝑎𝑐𝑒−1 , h𝐶 𝑎𝑖𝑟 −1 }

Raw location : ( x, y) = (190 cm, 570 cm)

Page 41: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Object-entity relationshipStructural Components Point of Interests Entities

hasIndividual

isLoca

tedIn

hasApplicableEntity

hasApplicableEntity

Page 42: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Evaluation – Location Awareness

Chair-1

Sofa-1

Fireplace-1

Drawingroom-1

Plant-1

Location (a)

Location (b)

(140,560)

(140,640) (430,640)

(430,560)

(60,480) (200,480)

(60,140) (200,140)

(180,110)

(180,10)

(110,340)

(10,340)

(610,360) (760,360)

(610,240) (760,240)

( 190,570 )

( 630,325 )

Mobile-robot route

Page 43: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Reasoning approach

Precision Recall

Crisp abductive reasoning

87.5 % 43.75 %

Fuzzy abductive reasoning

100 % 50 %

Location aided fuzzy abductive reasoning

100 % 88.89 %

Location aided reasoning

Location independent reasoning

Fireplace

Page 44: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Comprehensive

Framework

Page 45: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

EntitiesQualities

Indoor Positioning

System

Environment Sensors

SituationFuzzy

Abductive Reasoning

Fuzzy Abstraction

Rules

Contextual Situation Awareness

Semantic Object

Identifier

Spatial Reasoning

Loca

tion

Aw

aren

essDomain

Knowledge

Comprehensive Framework(System level)

Page 46: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Raw Physical Context

Data

SSN annotated

Observations

Low-level Fuzzy

Abstractions(Qualities)

High-levelAbstractions

(Entities)

OptimizedSituation

Raw Location

Data

Semantic LocationIdentifier

SSNOntology

Domain Ontology Fuzzy Abductive Reasoning Rules

Indoor LocationOntology

Comprehensive Framework(Semantic modeling)

Page 47: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Temp: 150 °CCO2:1120 ppm

Temp: 180 °CCO2:1160 ppm

Location (b): (400,150,30)

Location (a): (200,250,20)

Object coverage area

Mobile robot path

Page 48: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Fire: 0.80Heater: 0.20

Fire: 0.90Heater: 0.10

Location (b): (400,150,30)

Location (a): (200,250,20)

Object coverage area

Mobile robot path

Page 49: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Fire: 0.80Heater: 0.20

Fire: 0.90Heater: 0.10

Location (b): Chair

Location (a): Fireplace

Object coverage area

Mobile robot path

Page 50: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Fire: 0Heater: 0

Fire: 0.90Heater: 0.10

Location (b): Chair

Location (a): Fireplace

Object coverage area

Mobile robot path

Page 51: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Key Contributions• Developed a fusion based indoor localization algorithm to

achieve accurate spatial information of the sensing platform.

• Accurate indoor localization algorithm.• Surveillance and tracking of mobile robots in indoor environments.• Integration of indoor positioning results with virtual world environment.

Related papers:• P. Desai, N. Baine, and K. S. Rattan, “Fusion of RSSI and TDoA Measurements from Wireless Sensor Network

for Robust and Accurate Indoor Localization,” in International Technical Meeting of The Institute of Navigation, 2011, pp. 223–230.

• P. Desai, N. Baine, and K. S. Rattan, “Indoor localization for global information service using acoustic wireless sensor network,” in Proceedings of SPIE, 2011, vol. 8053, no. 1, pp. 805304–805304–10.

• P. Desai and K. S. Rattan, “System Level Approach for Surveillance Using Wireless Sensor Networks and PTZ Camera,” in 2008 IEEE National Aerospace and Electronics Conference, 2008, pp. 353–357.

• P. Desai and K. S. Rattan, “Indoor localization and surveillance using wireless sensor network and Pan/Tilt camera,” in Proceedings of the IEEE 2009 National Aerospace Electronics Conference NAECON, 2009, pp. 1–6.

• An invited journal paper in preparation.

Page 52: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Key Contributions• Introduced fuzzy abstraction and inference technique to

comprehend events via handling the uncertainty in the context information & the ambiguity in the domain knowledge.

• P. Desai, C. Henson, P. Anatharam, and A. Sheth, “SECURE: Semantics Empowered resCUe Environment (Demonstration Paper),” in 4th International Workshop on Semantic Sensor Networks (SSN 2011), 2011, pp. 110–113.

• A journal paper in preparation.

• Developed semantic mapping technique for indoor objects to aid the situational context awareness results via further discriminating not applicable events.

• Developed and deployed a comprehensive situation awareness framework for cyber-physical system.

• A journal paper in preparation.

Page 53: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Future work

• Richer spatio-temporal relation modeling between indoor objects and entities

• Efficient coverage space for the indoor objects

• Accurate indoor localization via smartphones

Page 54: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Acknowledgements

Page 55: Pratik Desai Ph.D dissertation defense

Department ofElectrical Engineering

Questions?