NVL Sensor Fusion Test Bed

Bala Lakshminarayanan, Mike McCullough

NVL Sensor Fusion Test Bed

March 18, 2004


Introduction

• US Army Night Vision & Electronic Sensors Directorate (NVESD)

• Network of acoustic and image sensors– Visible and IR

• Classification of civilian targets


Motivation & Background

• Military targets ~98%– 3 levels of fusion process, acoustic and

seismic data

• Current civilian classification ~80%– Improve accuracy rating– Both civilian targets and personnel

classification– Image data


Objectives

• Generation of data set of image and acoustic data– Development and fusion of moving target ATR

algorithms

• Establish methods to collect data and its “ground truth”


SFTB Setup

SFTB Base Station

Node 3 Acoustic sensor

Node 1 Acoustic sensor Node 2 Acoustic sensor

Node 3 IR sensor


Node 1 IR sensor


Node 2 Visual sensor


Met station & GPS

Commands through DOS Scripts

GPS

GPS

GPS

Wireless Ethernet connection


Sensors

• Indigo Alpha Thermal camera

• Pulnix TMC-7DSP Color camera

• Knowles BL-1994 Microphone


General Test Conditions (1)

• 3 nodes each with hexagonal acoustical array of 7 microphones and imaging sensor– Nodes 1 & 3 have uncooled IR camera– Node 2 has visible color camera

• Nodes gather information simultaneously for 3 minutes

• Acoustic sensor turns on imaging sensors

• MUSIC algorithm for DOA estimation



• Targets moving on gravel & asphalt roads

• Fully exposed– Trees or other vehicles occasionally in the

way

• License plates on the targets are not readable

• Stationary sensors

• Daylight operation (9:30am to 3:30pm)



• Target motion– Constant speed– Stops midway

• Constant acceleration, deceleration• Stops for count of ten

• Each target traverses at 5, 10, 15, 20 mph• Start and stop outside FoV of nodes• Creation of different scenarios


SFTB Operation (1)

• Attended mode– Short term data collection / Demo mode

– Collects 4 types of data

– Surveillance, directed, pan scanning

– SFTB_Base.exe, FullSim.exe

• Data collection mode– Pure data collection

– Collects 4 types of data

– Acquire.exe (Video and acoustic), MetEffects.exe


SFTB Operation (2)

• Collected data– Acoustic .dat – Image .arf– Ground Truth .agt

• Filenames depends on sensor, node, scenario and targets


Numbering System

• SSSN00000_0000

• SSS = camera name– IN1 = Indigo IR camera 1– IN2 = Indigo IR camera 2– PX1 = Polinex Visible camera 1– AC1 = Acoustic number 1

• N = node number (1-3)• 00000 = scenario number• _0000 = number indicating vehicle number (1-7)

– Can have multiple numbers multiple targets


Number System Example

• in1200004_0003– In1 indigo thermal camera #1– 2 node 2– 00004 scenario number 4– _0003 target 3

• ac1300001_0056– ac1 acoustic array #1– 3 node 3– 00001 scenario number 1– _0056 both targets 5 and 6


AGT Format

• Very similar to a class in a high level programming language

• Agt{

PrjSect {…}SenSect {

SenUpd {…}}TgtSec{

TgtUpd {…}}

}


AGT format


PrjSect

• Name = “sftb”

• Scenario = “00001_0001”

• Site = “nvis0306”


SenSect

• Denotes the sensor section of the AGT

• Contains all of the SenUpd

• Name = “in11”– Denotes which sensor being used


SenUpd

• LatLong

• Elevation

• Keyword “Frame #1”

• Keyword “AcousticAz: 0”

• Keyword “Nodeld: 1”

• Azimuth 75.9077301025

• Time 2003 160 16 22 34 587

• Fov 40.0 30.0


TgtSect & TgtUpd

• TgtSect is the sector that contains all the target updates

• TgtUpd– Keyword “Frame: #1”– Time 2003 160 16 22 34 587– Tgt

• Range 48.0• TgtType “MAN”• Aspect 46.0• PixLoc 40 63


ARF Info

• Automatic target recognition working group Raster Format

• Contains header, sub headers, footers, 1 or more images

• Supports multiple frames

• Supports 16 different image types in same file


ARF Info

Rows, cols, version, type, # frames, offset…

Colormap, comments

Image file


ImageJ

• NVL used a plugin for converting image from .arf to other formats– Image processing and analysis in java

• Formats – dicom, pgm, jpg, bmp, tiff, raw…

• Operations – FFT, convolution, fractal box count, morphological…


Acoustic Data

• Raw data in acoustic .dat file

• Contains header information for system time (similar to AGT), node #, longitude/latitude (0’s), and bearing (0’s)

• make_wave.py to convert from .dat to .wav– Changes to specify output file needed within

the python script to make it work properly– Script drops .dat extension and adds .wav


Targets

• Honda CRX (target 1)

• Chevy Cavalier (target 2)


Targets

• Toyota pickup (target 3)

• GMC pickup (target 4)


Targets

• Vehicle (?) (target 5)– Names obtained from AGT files that would eventually

contain a TgtType indicating the target

• Toyota 4Runner (target 6)

• Stake body light truck (target 7)– Dave Rankins


Example Data

• Convert arf files into raw using ImageJ

• Modify raw image into PGM– Switch endianness

• Apply image processing techniques to the image– Very hard to distinguish objects due to having a

dark image


Example Sound Clips

• Normalized data using shareware program

• Target 1 – Node 1– Node 2– Node 3

• Target 6– Node 1– Node 2– Node 3


Example Images (1)

• Sensor placement• FoV of a sensor covers atleast half of total FoV


Example Images (2)


Example Images (3)


Example Images (4)


Future Work

• MatLab acoustical analysis

• Segmentation & shape analysis

• Feature selection & extraction

• Fusion

• Target Recognition algorithms


Questions?

NVL Sensor Fusion Test Bed

Documents

Transcript of NVL Sensor Fusion Test Bed