Distributed Sensor Exploitation 6.1 Rapid Deployment of Smart Camera Networks Dr. B.S. Manjunath...

Distributed Sensor Exploitation 6.1

Rapid Deployment of Smart Camera Networks

Dr. B.S. ManjunathUniversity of California, Santa [email protected]

Thrust Technical ReviewMarch 2012

Distributed Sensor Exploitation 6.1Rapid Deployment of Smart Camera Networks

SCHEDULE:

TTA: N/ATECH TRANSITION PATH: DYNAMIC WIKI/RESOURCE MANAGER (D&I)

TECHNICAL APPROACH: Geo-calibration using mobile sensors Auto-calibration and network topology discovery

using visual analysis Detecting and tracking objects in a camera network

through fusion of multiple views, despite wireless communication challenges

Analyzing events over non-overlapping camera viewsPERFORMER:University of California at Santa BarbaraB. S. Manjunath, Professor

RESEARCH QUESTIONS:• How can we geo-calibrate a stationary network using mobile sensors and

associated GPS and other information?• What are the issues in discovering the geometric/spatial and temporal

connections and constraints within a camera network with overlapping and non-overlapping views?

• What information needs to be computed at the smart sensor nodes for activity analysis without distributing the raw video?

• When and how to fuse multi-modal information, e.g., mobile, airborne, and fixed cameras, for robust object tracking and activity analysis?

MILITARY RELEVANCE/OPERATIONAL IMPACT: Rapid deployment of a network of smart cameras and UAVs can help

monitor any potential area, especially in high risk and remote sites.

NAVAL S&T FOCUS AREAs ADDRESSED: Asymmetric & irregular warfare

TASKS FY10 FY11 FY12

Camera network setup

Object detection & 2D tracking

Object tracking in 3D

Auto-calibration and topology

Event & behavior detection

Demos on campus testbed

UAV: Provides intelligence from overhead

Rapidly Deployed Smart Camera: Provides ground-based views and

distributed image analysis

2012 Thrust Area Technical Review


3

Principal Events & Activities

FY2010 FY2011 FY2012

O N D J F M A M J J A S O N D J F M A M J J A S O N D J F M A M J J A S

Camera Network Setup

Auto Calibration & Toplology

Geo-calibration

Object Detection and 2D tracking

Object Tracking in 3D

Event/Behavior analysis

Demonstration on DURIP Testbed

Top Level POA&M (Summary Task)

Field deployment Final demonstration



Project Work Breakdown Structure2012 Thrust Area Technical Review


5

Detailed Technical Approach

• Objective– Development of a rapidly deployable network of smart camera

sensors to gather mission-specific information for surveillance and human activity analysis

• Challenges– Minimize human interaction– Wireless communication constraints: limited bandwidth,

unpredictable latency, dropped packets, and time synchronization– Few examples to train the system



6

Events in FY 10

Setup of Outdoor Camera NetworkQuick Chessboard CalibrationPreliminary network topology discovery

Events in FY11

Network topology discovery Automatic calibration without patternsBeyond 2D trackingContinued multi-camera tracking with overlapping and non-overlapping views.Preliminary distributed processing with existing work

Events in FY12

• Geocalibration with mobilesensors• Multi-camera multi-object tracking in a wide-area network• Browsing and search in a distributed network

Project Roadmap



7

• Project Introduction• Overview of the UCSB Campus Camera Network

– Infrastructure funded by a ONR DURIP award– Implementation by students on this project

• Research Accomplishments– Distributed Geocalibration– Distributed tracking and fusion– Topology discovery and network data summarization– Browsing and search in a distributed network

• Summary and Conclusions



1. Camera Placement: Where to place the cameras?– Using prior information, such as GPS tracks, to optimally place

cameras– Novel approach to camera placement based upon “optimal”

reconstruction of tracks and activities with selected observations

2. Calibration and Topology: How are the cameras related?– Necessary for most multi-camera applications to understand how

the cameras are related in a physical space (calibration) or relatively according to their activities (topology)

– Novel methods for calibrating additional cameras in a large calibrated network

– Novel methods for calibrating cameras in a network individually, but in a global coordinate system without entering the scene

– Examined topology methods on our network

8



3. Object Tracking: How to track or search objects?– Tracking or searching objects in a multi-camera network requires

taking advantage of all overlapping and relevant information– Novel method for distributed, online multi-camera tracking using

multiple instance learning and particle filtering with appearance and geometric fusion.

– Novel trajectory searching with graph modeling

4. Event/Behavior Analysis: How to perform activity analysis over a distributed camera network?– The ultimate goal of a wide-area network is to make high level

decisions and recognitions across many cameras– Novel camera network summarization algorithm– Addressing the relatively unexplored problem of activities analysis in

a large, wide-area network with many non-overlapping views

9


Topologydiscovery

Cameraplacement

N+1 cameracalibration

Geo-calibration

Distributedobject tracking

Objectsearching

Networksummarization

Wide-areaactivity analysis

Topic relationships


11

UCSB Camera Network

• Examined challenges through manual network setup• Developing optimal camera placement algorithm based on

GPS trajectories


Project Technical Assesment

New Geo-calibration formulation with mobile sensors Distributed Object Tracking on a Camera Network Video Summarization in a Multi-camera setting Distributed browsing and search

12

Recent Progress


Project Technical Assesment - 1





13


1. Geocalibration using mobile devices


Geo-Calibration

How do you calibrate cameras with non-overlapping views?

15


Geo-Calibration

• Geo-Calibration is the process of finding the projective geometry of a camera in a global coordinate system.

• Traditional, image-based methods are in a local coordinate system and cannot be shared across different calibrations1

• Existing methods for integrating global coordinates involve GPS tagged objects in the scene or aligning with a map2

161. Zhang, “A flexible new technique for camera calibration,” PAMI 20002. Kaminsky et al. “Alignment of 3D Point Clouds to Overhead Images.” W on Internet Vision 2009.


Geo-Calibration Method

• Take pictures of the camera’s scene using smartphones and collect location and orientation metadata– Location sensor: GPS– Orientation sensor: compass + accelerometer or gyroscope

• No objects need to be placed in the scene• Data can be collected rapidly

17


Geo-Calibration Method

• Geo-calibration: Given image(s) from a camera1. Collect, using smartphone, calibration images with GPS and

orientation metadata

2. Calibrate cameras in a local coordinate system using images

3. Transform from local to global GPS coordinates

4. Refine position and orientation estimates

• Two approaches: centralized and distributed

18


Distributed Geo-Calibration

• Assume GPS positions accurate, then average across all stereo/pairwise estimates

• Rotation average using Karcher Mean

• Position average using least squares

19

GPS Points

Two-view estimate

Least squares intersection

d is the geodesic distance in rotational space SO(3)


Simulation Results

• 2 quarter circle rows of cameras every 5 degrees• Middle camera used as fixed camera• Random set of points around origin


Simulation Results

GPS Noise Feature Point Noise

Location Errorfrom Orientation Noise

Orientation Errorfrom Orientation Noise


Real World Results

• Smartphone: Data collected using HTC Evo smartphone• Accurate Sensors: Data collected using CyberQuad UAV

sensors and camcorder• Error Distance compared against GPS/Sensor from data

Dataset Error Distance Batch Distributed

Smartphone Location 2.83 m 4.21 m

Orientation 0.2399 0.0017

Accurate Sensors Location 1.80 m 2.001 m

Orientation 5.1490 0.0002


Smartphone Results

23


Accurate Sensor Results


Geo-calibration MLE

• New formulation as a maximum likelihood• Maximize the orientation and position of the fixed camera

given:– Smartphone orientation position and location– Pairwise relationship based on the image measurements

25


Smartphone Measurements

• Smartphone orientation calculated from compass and gyroscope/accelerometer

• Smartphone position calculated from GPS with GPS error/accuracy

26


Pairwise Measurements

• Find and match features points (Xi, Xq) to find a relative rotation and translation

27

Relative rotation

Relative position andunknown depth


Geo-calibration

• Maximum likelihood of probability of fixed cameras orientation and position (Rq, tq)

• Optimal Estimates

28


Consensus Framework

• This estimate can be solved using a consensus algorithm• In a consensus algorithm, each smartphone has a state of

the fixed camera’s orientation and location. The states are iteratively updated to a consensus state.

29


Geo-calibration Experiments

• Mount a camera over GPS Benchmarks• Collect images and sensor metadata using a smartphone

30


Geo-Calibration Summary

• Technical Report available for initial formulation, Kuo et al. “Map Me: Camera Geo-calibration using mobile devices”

• Next: Experimentation on New Formulation


Project Technical Assesment- 2





32


Topologydiscovery

Cameraplacement


Geo-calibration


Objectsearching



Recap: Topic relationships


Object Tracking and Search

34

In a smart camera network setup,

• Distributed multi-camera object tracking – Given a target, how do the network of cameras with

overlapping views robustly track the target with collaboration despite limited network bandwidth?

• Efficient object searching in a large network– How to search for objects of interest efficiently without

significant visual processing at search time?


Distributed browsing & search

Project Technical Assesment


Problem: • A camera network deployed

over a large area• No live streaming of any video• Local camera nodes have

storage to archive video and limited processing power for simple video analysis

• How a human image analyst at a distance central node interacts with the remote cameras?

36

?

Browsing and Searching


• Envision the following application scenarios:

– A user instantiates the interaction with the network by specifying regions on the image plane (cameras, time intervals) of interest. E.g., “FIND object instances related to region A FROM camera 1 OR region B FROM camera 4 between time 9:30am and 9:35am”

– With the results from the previous scenario, the user could then identify one specific object of interest to initiate further searching for the same or related objects. E.g., “FIND all objects related to the object instance at region C FROM camera 1 at time 9:32:41.3am”.

37

Browsing and Searching


• Try to provide human users with an high-level interface, such as dynamic global scene visualization and aim to detect/track all observed objects across the entire camera network

• To deal with appearance variations across views, much prior work focused on finding the best matching criterion (Javed ICCV’03,Javed CVPR’05, Farenzena CVPR’10, Zhen CVPR’11, Rios-Cabrera CVPR’11.)

• What if we cannot have reliable object detection and tracking? Any possible strategy without pair-wise matching or global trajectory finding?

38

Conventional Approaches


• Instead of trying to find global trajectories for every object visible in the network, model camera observation directly with a graph model

• Act as an intermediate agent between distributed cameras and human image analysts and provide recommendations to the user with a concise and representative set of video snapshots captured by the camera network

• Help the image analysts to browse, search and identify objects of interest by giving a canonical overview of the entire set of visual observations in the network

39

Proposed System


40

Proposed System


• Real time object detection with background modeling and tracking with mean-shift algorithm

• An observation record is generated

for each detected and tracked object

by the camera and sent to the

central node over the network

41

Real-time detection & tracking


• Given a user query, we need to find video frames with following properties: – Centrality, representative ones which are closely related to the query

and many other observations and hence considered important.

– Diversity, covering as many distinct groups as possible

• A graph G(V, W) to model relationship among camera observations and perform unified graph ranking for different queries – Individual camera observations (i.e., frames with detected objects) form the

vertices set V

– Weight matrix W defines the strength of connectivity between camera observations

– G(V, W) built at the central server incrementally as the new records are received in real time from the cameras.

42

Modeling camera observations


• Divide image plane into 8x6 blocks

• Model the time delay for an object to travel between any two blocks across cameras with a Gaussian model with known mean and variance

43

Spatial Temporal Topology


• 11 camera nodes (Cisco wireless-G WVC2300) observing bike path

• Each camera streams video to a dedicated computer to simulate a smart camera node

• Approx. 600 meters in width/length

44

Demonstration – Test bed


45

A browsing example with regions of interest indicated by therectangles in camera C8 and C9

Browsing Demonstration


46

Browsing Demonstration

• Top 10 ranked frames (Decreasing order: left to right, top to down)• A total of 10 distinct objects satisfying the criterion. All of them have been

identified (labeled in yellow). The 8th ranked frame is a “false positive” (it has not passed the queried regions within the specified time interval).


Searching Demonstration

• Results when searching for object P6 (2nd -11th ranked frame)


Topologydiscovery

Cameraplacement


Geo-calibration


Objectsearching



Recap-Topic relationships


49

Forecasted Key Events

UAV: Provides intelligence from

overhead

Rapidly Deployed Smart Camera:

Provides ground-based views and distributed image

analysis

Events in FY 10

Setup of Outdoor Camera NetworkQuick Chessboard CalibrationPreliminary network topology discovery

Events in FY11

Network topology discovery Automatic calibration without patternsBeyond 2D trackingContinued multi-camera tracking with

overlapping and non-overlapping views.Preliminary distributed processing with

existing work

Events in FY12

• Geo-calibration with mobile sensors• Multi-camera multi-object tracking in a

wide-area network• Browsing and search in a distributed

network

Soldier image courtesy of US Marines. Map image courtesy of Google Earth.


Project Technical Risk Assessment

No. Risk Area and Description Risk Rating

Risk Reduction Actions Due Date (Date Completed)(L,M,H)

General

1 Challenging datasets due to wireless communication constraints

L 1) Improve antenna and camera placements

2) Robust algorithms

1) 12/2010

2) 09/2012

Calibration and Topology

2 Lack of distinctive environmental features for autocalibration

M 1) Weak scene calibration

2) Explore mobile cameras

1) 05/2011

2) 05/2011

3 Current topology approach may not scale to large separations

M 1) Explore tracking-based approach

2) Explore mobile sensors

1) 11/2010

2) 6/2012

Object Detection/Tracking

4 Algorithm fails in severe occlusion M Explicit occlusion handling 09/2011

Event/Behavior Analysis

5 Complexity of the algorithms M Distribute the algorithm to multiple computing nodes

03/2012

6 UAV data exploration: FAA and university regulations

H Collect data in remote areas (non-urban)

6/2012

50


51

Summary

• On target with proposed schedule• Setup of campus-wide camera network• Quick chessboard calibration• Camera network topology discovery• Wide-area human mobility patterns• Single camera activity discovery

• Ongoing• Geo-Calibration: automatic camera calibration and camera

placement• Online learning and multi-camera fusion for distributed tracking• Multi-camera activity discovery


52

Students/Acknowledgement

• Zefeng Ni (graduated Fall 2011)– Distributed Tracking, activity analysis

• Thomas Kuo (expected to graduate in 2012)– Calibration/geo calibration on

• Carter De Leo• JieJun Xu (expected to graduate in 2012)• Thanks to ONR/Dr. Martin Kruger for the support and

encouragement!


53

BACK-UP SLIDES


UAV Data Collection

• Extending to aerial imagery– Getting aerial data is challenging.– Not easy to combine with ground based network on campus due to

university insurance policies (expensive) and unknown FAA regulations concerning UAV research

• We are working on setting up a network in a nature preserve area where there is no human traffic– Should be less constrained in obtaining UAV data in such areas– Cons: no human activity in such areas.

• Collect data in remote areas with human actors– Plan to accomplish this before the end of Spring quarter 2012.– Preliminary data already collected.

54

Distributed Sensor Exploitation 6.1 Rapid Deployment of Smart Camera Networks Dr. B.S. Manjunath...

Documents

Transcript of Distributed Sensor Exploitation 6.1 Rapid Deployment of Smart Camera Networks Dr. B.S. Manjunath...