Peter Bajcsy, Ph.D.Research Scientist, ALG NCSAAdjunct Assistant Professor, CS and ECE Departments, UIUC

Understanding Multi-Instrument Measurement Systems

2

Research Focus

Theoretical modeling and experimental understanding of multi-instrument measurement systems that deal with multi-dimensional multi-variate dataand include sensor networks, wireless communication, data acquisition, fusion, spectral & spatial analysis and synthesis components

3

R&D Components

Sensors

Data Fusion

Analyses

Syntheses

Data Acquisition

Building SolutionsSensorsData AcquisitionData FusionAnalysesSyntheses

4

R&D Components: SensorsMeasurement Sensors

Measured physical entities, e.g., displacement, shear, temperature, reflectance, orientation

Positioned as satellite, aerial or ground sensors

Attached to moving or stationary objects

Connected as wireless or wired sensors

Sensors of Interest

Spectral sensors: EO, IR, HS, UV, SAR, X-ray, OCT, MRI, CT

Raster and point sensors“Smart” MEMS sensors

Sensors

Data Fusion

Analyses

Syntheses

Data Acquisition

5

Hardware EquipmentCameras

Hyperspectral (visible & near IR), multispectral (R,G,B, near IR), thermal IR, visible (wireless)

Robot with armOrientation sensors IS300 ProCrossbow MEMS (wireless)Audio (wireless)

6

EXAMPLE:Toward Hazard Aware Spaces

7

Problem Description

Deploy, Sense, Calibrate, Analyze, Trigger,

Confirm, Alarm

8

Problem DescriptionGiven wired and wireless multi-spectral and other sensors,

assess hazards in an indoor environment.Data acquisition:

Wired multi-spectral sensors – thermal infrared, hyperspectral, visible and multi-spectral camerasMicro-Electro-Mechanical Systems (MEMS) with CPU – MICA sensors with tinyOS

Wireless data transmission and receptionSensor registrationSensor calibrationData fusionHazard detection

9

Motivation & ChallengesMotivation:

Proactive surveillanceBandwidth requirement reductionSpatially adaptive indoor hazard detection

Challenges: Sensor selection (ruggedness, weight, capabilities)Power/CPU/Memory constraintsWireless communication limited range, delays and packet losesSensor deploymentOptimal sensor layoutSensor and camera calibrationCamera controlHazard event recognition (image and signal analysis)Sensor based confirmationAlarm propagation

10

Deployable Sensor Network

11

Optimal MICA Sensor LayoutWhat is the most optimal “smart”MEMS system design?

What is the most effective mechanism to receive information from the wireless sensor network?How to synchronize data acquisition from all “smart” MEMS sensors?How to assign time stamps to every sensor reading?Is there an optimal spatial arrangement of sensor networks?What is the impact of other wireless devices on sensor networks?

12

Thermal IR Camera CalibrationThermal IR Image

uncalibrated

calibrated

Color Image

Engineering Units [degrees Celsius]

Goal: Information Content

13

R&D Components: AcquisitionData Acquisition

Sensor SetupsOrientation of sensorsPlacement distribution, e.g., density, of sensorsDistance from objects of interestField of view and spatial resolutionSensor attachment mechanismInterference of multiple sensorsExternal dependencies, e.g., light sources, media propagation

Synchronization of multiple sensorsTriggers and timersTemporal resolution

Wireless data transmissionSpatial coveragePacket lossBandwidth and frequency bandsCommunication protocols

Input DataPhysical entityData type (raster or vector types)Datum precision

Data of Interest

Any combination of miscellaneous data sets combined with image data

Sensors

Data Fusion

Analyses

Syntheses

Data Acquisition

14

EXAMPLE:Robot Control Using Arm Gestures

15

Problem StatementRecognize human gestures according to a gesture lexicon for controlling remotely autonomous vehiclesConstraints:

Environment: Limited line of sight, EM Interference Acoustic noise, Motion (Catapult and shock vibrations, Aircraft deck motion ~5-6° roll), Many emittersOperation Conditions of Yellow Shirts: YS is in all weather conditions, YS is on the deck 14-16 hours, YS is exposed to temperatures –10°F to 140°F

-150

-100

-50

0

50

100

0 2000 4000 6000 8000 10000 12000

Gesture Lexicon Command:Move Ahead

Robot Instructions:Set Velocity X

Sensor Output Robot Action

16

Motivation & ChallengesMotivation:

Allow UAVs to mix seamlessly with the existing traffic on an aircraft carriers deck

Challenges: Sensor selection: active & passiveSensor attachmentData acquisition and measurement repeatabilityAutomatic classification of sensed arm movements into distinct hand signalsReal-time parsing and classificationTranslation of hand gesture sequence into a sequence of autonomous vehicle commands Execution feedback

17

Developed Solution

18

Real-Time Classification Example: Turn to Left

19

R&D Components: FusionData Fusion

Transformation of Measured VariablesAnalytical and Statistical Relationships

Coordinate System RegistrationRaster & Raster/ Vector & Raster/ Vector & VectorGeoreferencing & Medical Atlas Registration & Generic RegistrationAutomation

Spatial Interpolation/Extrapolation of VariablesInterpolation/Extrapolation ModelsBoundary Conditions (constraints)

Temporal Interpolation/Extrapolation of VariablesInterpolation/Extrapolation ModelsStationary & Non-stationary variables

Uncertainty Modeling & FusionPropagation errorsModeling errorsInterpolation/Extrapolation errors

VisualizationData OverlaysUncertainty maps

Fusion of Interest

Any combination of raster, boundary and point data

Sensors

Data Fusion

Analyses

Syntheses

Data Acquisition

20

EXAMPLE:Data Fusion of Raster and Vector Data Using Uncertainty Modeling

21

Problem DescriptionGiven Multiple Advanced Sensors

Measuring Point and Raster Types of Physical Entities, integrate all measurements and compare them with simulations

• Transformation of measured variables• Sensor registration (coordinate system

registration)• Continuous approximation of measured

physical entity • Data fusion• Location dependent

accuracy/confidence prediction• Visualization and comparison of

measured and simulated data

LoadWall

22

Sensor Fusion

Interpolated ImageFrom Vector Data

Raster Image

Image Mask

SelectRasterData

Select InterpolatedData

23

R&D Components: AnalysesData Analyses

Understanding Measurements & Problem Statement & Constraints & Solution Requirements

Underlying physics, e.g., Maxwell EquationsMeasurement constraintsRepeatability of measurementsReliability of measurements

Feature ExtractionFeature SelectionFeature Processing & Data Mining & Pattern Recognition

Supervised or unsupervisedTesting and verification

Output DataPhysical entityResult accuracyResult validation

Data Analyses of Interest

Any application that involves scene understanding, knowledge mining, or optimal decision making

Sensors

Data Fusion

Analyses

Syntheses

Data Acquisition

24

R&D Components: SynthesesData Syntheses

Understanding Phenomenology Data driven prediction & Physics based modelingData variability & phenomenon variability

ModelingModel dimensionalityFeature selectionComputational tradeoffsStatistical & Analytical modeling

Output DataPhysical entityValidation metricsSpatial, temporal and spectral accuracy

Data Syntheses of Interest

Spectral scene modeling using BRDFs

Environmental predictions

Atmospheric forecast

Sensors

Data Fusion

Analyses

Syntheses

Data Acquisition

25

EXAMPLE:Spectral Scene Modeling Using BRDFs

26

Problem Statement

27

Challenges: Geometric & Photometric Properties

specular surfaces (artificial grass)diffuse surfaces (concrete)isotropic surfaces (leather)anisotropic surfaces (straw)surfaces with large height variations (pebbles)surfaces with small height variations (sandpaper)pastel surfaces (cotton)colored surfaces (velvet)natural surfaces (lettuce)man-made surfaces (sponge)

28

Challenges: Outdoor Environment

Wheat Canopy– May 8, 1999, 55° Solar Zenith Angle, (36.77 N; 97.13 W) near Ponca City, Oklahoma

Parallel to the principal plane of the Sun

Perpendicular to the principal plane of the Sun

15 75-75 -15

15

-15

-75

15

75

29

Reflectance Modeling

Reflectance [ ,%]

( ); ( ); ( 2 ); ( );( ); ( 2 ); (2 ); ( );(2 2 )

reflected

incident

i r i r i i r

i r i r i r

dBρ

ρ ω ω ρ ω ω ρ ω π ρ ω ωρ ω ω ρ ω π ρ π ω ρ ω ωρ π π

Φ= Φ

→ →∆ → ∆ →∆ →∆ ∆ → → →∆

→

30

BRDF Definition

Helmholtz Reciprocity Principle( ) ( )

Isotropic Surface( , ; , ) ( , ; , )

r i r r r i

r i i r r r i i r r

f f

f f

ω ω ω ω

θ φ φ θ φ φ θ φ θ φ

→ = →

+ + =

1( , ): ( , ; , ) [ ]( , ) cos

r r rr i i r r

i i i i i

LBRDF f srL d

θ φθ φ θ φθ φ θ ω

−=

1( , ; ): ( , ; , ; ) [ ]( , ; ) cos

r r rr i i r r

i i i i i

LBRDF f srL d

θ φ λθ φ θ φ λθ φ λ θ ω

−=

1( , ; , ): ( , ; , ; , ) [ ]( , ; , ) cos

r r r i rr i i r r i r

i i i i r i i

LBRDF f srL d

θ φ λ λθ φ θ φ λ λθ φ λ λ θ ω

−=Fluorescent surfaces

Wavelength dependent absorbent surfaces

31

Aerial Imagery: Verification

Mask Generated Using Isodata Clustering With 8 Mask Labels

Measured

Synthetic

Mask

32

SummaryUnderstanding of Multi-Instrument Measurement Systems

Cover very broad range of problemsProblems consists of fundamental theoretical issues, experimental issues and applied issuesFrom sensors to knowledge – complex systemsData heterogeneity and dimensionality – processing and understanding challenges

Ubiquitous applicationsSensor-tools (ad-hoc networks, hazard aware spaces)Fusion-tools (Structural engineering, GIS, Medicine)

BioMed-tools (Bio-informatics, Medicine, Biology, Plant Science)Geo-tools (GIS, Remote Sensing, Agriculture, Hydrology, Water Quality Survey, Atmospheric Science, Military)Sim-tools (Military, Structural Engineering, Atmospheric Science, Agriculture)Edu-tools (Education)