In the unlikely event that tho author did not $end UMI a cornpieln manuscript and there are missing pages, these will be noted. A b , if unauthoraed copyright

matefial had to be removed. a witi i n d i i the âeleüm.

Oversize materials (e-g.. maps, drawings, charts) am reproduoed by sediorring

the original. ôegi~ing at the upper left-hand corner and continuing from îeft to

nght in equal secfions with small overlaps,

Photographs induded in Vie minal manuscript have bwn reproduced

xerographically in this wpy. H i g M quality 6* x bladc and white phobgmphic

pnnts are availabîe for any photogmphs or illustmüom a ~ ~ e a r i ~ in Ibis ccipy for

an additional charge. Cultact UMI di- do ofder.

ml& H d l Infomtation and Leaming 300 North Zeeô Road, Ann Arbor, MI 4û1ûSlW6 USA

APPLICATION OF SENSOR DATA FUSION TECHNIQUES TO THE LlGHT ARMOURED VEHICLE

RECONNAISSANCE (LAV RECCE)

Lieutenant Dm van Huyssteen, rmc, BmEngm Canadian Forces

Thesis Submitted To: Department of Electrical and Computer Engineering

Royal Military College of Canada Kingston, Ontario

In Partial Fulfillment of the Requirements For the Degree

Master of Engineering May, 1998

O Copyright 1998 by D. van Huyssteen, Kingston, Ontario This thesis may be used within the Depariment of National Defence

but copyright for publication remains the right of the author.

National Library 1+1 of.,, BiMiitheque nationale du Canada

Acquisitions and Acquisitions et Bibiiographic Services senrices bibfiographiques

395 Weliîngîon Street 395. rue Wdlifghm OitawaON K 1 A W OttawaûN K 1 A M can2da Canada

The author has granted a non- exclusive licence aiiowing the National Library of Canada to reproduce, loan, distribute or seil copies of this thesis in microfom, paper or electronic formats.

The author retains ownershîp of the copyright in this thesis. Neither the thesis nor substantiai extracts fkom it may be printed or otherwise reproduced without the author's pemission.

L'auteur a accordé une licence non exclusive permettaut à la Bibliothèque nationale du Canada de reproduire, prêter, distriiuer ou vendre des copies de cette thèse sous la foxme de mîcrofiche/film, de reproduction sur papier ou sur format électronique.

L'auteur conserve la propriété du droit d'auteur qui protège cette thèse. Ni la thèse ni des extraits substantiels de celle-ci ne doivent être imprimés ou autrement reproduits sans son autorisation.

Abstract

The ability to employ dissimiiar sensors to collect information conceming the behavior of a wide varïety of potential threats is critical to the survivability of high priority military assets. To this end. accurately locating and identifjhg enemy targets foms the basis of an effective response. Consequently, research into surveilIance and tracking systems that best utilize available resources is a high priority.

In this thesis, an efficient, robust and effective target tracking architecture is developed that combines active information h m a Moving Target hdicator (MTI) radar and passive information h m a Forward Looking Infrared @LE) imager. The fmal design, a decentralized architecture, is selected out of a fietd of three candidates on the basis of performance in Monte Carlo simulation. The other candidates are examples of sequential and passive decentratid fùsion- The design is intended as a potential enhancernent to the surveittance suite aboard the Canadian Forces' newly-acquired Light Annored Ve hic le Reconnaissance (LAV Recce).

The design effort inc ludes (i) evaluating various data fbsion architectures; (ii) selecting appropriate tracking filters; (iii) choosing maneuver compensation and track initiation schemes; and, (iv) developing a method to extract a target's aagular position fiom analog infrated images. The scope of this thesis is limited to a simulation of the single target case; however, an extension to the multiple target scenarïo is also discussed in ternis of selecting an appropriate data association algorithm.

No tracking system design is complete without a thomugh field test. To judge the effectiveness of the proposed architecture, the LAV Recce was tested in a realistic surveillance exercise that produced a complete set of pre-processed data with which to validate thïs and f h r e prototypes. In the trial, the radar and FLIR sensors were assigned a c o m o a line-of-sight and both covered 200 milliradian horizontal fields of view, The target, a moving vehicle, was observed h m approximately one kitorneter away. An example of the infrared data is presented, and the image processing techniques adopted in this thesis are demonstrated success fully. The complete test results, which include a test of the proposd tracking architecture with both radai. and FLIR data, will follow in a departmental technical report by the author at a later date.

Acknowledgment

f would Iike to acknowledge the many people who made this thesis possible, especialfy:

1. Mr. .J- Cmickshank and members of the PSSTA Section at DREV for providing documentation, tec hnical and financial support;

. . 11. LCol R. Carruthers and the P M 0 LAV for generous organizational support, and for providing a

test vehicle;

. -. 111. LCoI J. Lord at LFTSC who provided initial documentation and various points-of-contact; and,

iv. Dr. M. Farooq for his constant support, patience and m l v e in guiding the development of this thesis to a successfül conclusion-

Table of Contents

Abstract ..................................................................... i.

. . Acknowledgment ............................................................ -ri-

... Vita ........................................................................ III-

... List of Figures .............................. ,. ............. .. .............. -viii-

List of Abbreviations and P ~ c i p l e Symbols ...................................... -x-

....................................................... Chapter 1 introduction -1- ........................................................ 1.1 Pnmary Objective -1-

........................................... 1.2 Tasks Entailed by Primary Objective 1 . ....................................................... 1.3 Secondary Objectives 1 .

............................................................. 1.4 Assumptions -2- ........................................................ 1.5 Perceived Hurdles -2-

.....*..*.............................. .................... 1.6 Motivation ... -2- ......................................................... 1.7 Related Research -3-

1.8 Background ............................................................. 4 ........................................................ 1.9 Technical Context 4

.................................................... 1.10 Thesis Organization -5-

............................. Chapter 2 The Tracking Problem and the Kalman Filter -6- .......................................... 2.1 Discrete Linear Kaiman Filter (DKF) -6-

.............................................. 2.2 Properties of the Kalrnan Filter -10- ........................ 2.2.1 Covariance Properties and Monte Car10 Simulation -10-

................................................ 2.2.2 Innovation Properties -11- .............................................. 2.3 Extended Kalrnan Filter (EKF) -11-

..................................... 2.4 Functional Elements of a Tracking System -13- .................................. 2.4.1 Maneuver Detection And Compensation 13-

2.4.2 TrackInitiation .................................................... -14- ................................................... 2.4.3 Data Association -14-

2.4.3.1 Measuement Oriented Techniques ............................... -1 S- ..................................... 2.4.3.2 Track Oriented Techniques -16-

................................ 2.4.4 Architectures for Kinematic Data Fusion -17- .............................. 2.4.4.1 Decentralized (Sensor level) Fusion -17-

.......................... 2 .4.4.2 Centralized (Measurement Level) Fusion -18- ............................................ 2.4.4.3 Sequential Fusion -19-

............................................. 2.5 Short List of Design Approaches -19-

Chapter 3 Detailed Evaluation of Candidate Architectures .......................... -21 . 3.1 Candidate SF-1 : Sequential Fusion ........................................ -21 .

3.1.1 State and Measurement Models ........................................ -21- 3.1.2 Filter Formulation .................................................. -23- 3.1.3 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5-

3.2 Decentralized Architectures ..............................................~.. -25- 3.2.1 Candidate DF-1: Static Decentralized Fusion ............................. -26-

3.2.1.1 Background ............................................... ..2 6- 3.2.1.2 Application of Static Fusion to the A L I S S ........................ -27- 3 .2.1 -3 Generating Local Pseudoestimates ............................... -28- 3.2.1.4 Rem& ................................................... -30-

. ............................ 3 .2.2 Candidate DF-2: Static Fusion with Feedback -31 ..................... 3.2.3 Candidate DF-3: FuIly Decentralized Dynamic Fusion -32-

3.2.3.1 Timing Considerations ........................................ -33- ................................... 3.2.4 Remarks on Decentralized Approach -35-

......................... 3.3 Replacing Standard EKF with More Efficient Formulation -35- ......................................... 3.4 Data Association and Track Initiation -35-

3.5 Data Association ......................................................... -35- .................................... 3.5.1 Nearest Neighbour Data Association -37-

................. 3.5.2 Application of NN Data Association to Decentralized Fusion -39- ................... 3 S.3 Application of NN Data Association to Sequential Fusion -40-

3.6 Track Initiation ........................................................... -40-

Chapter 4 i n h e d Image Processing .......................................... 42- 4.1 Mapping Between Pixel and Angle Coordiaate Systems ........................... -42- 4.2 Extracting the Target Ceatroid from a Noisy Image ............................... -44-

........................................ 4.3 Interpreting the Analogue Video Signa1 -45- ................. 4.4 Single Frame Estimation: Determining Centroid Mean and Variance -47-

...................................................... 4.4.1 Background - 4 7 - 4.4.2 Application of a Bandpass Filter ...................................... - 4 8 - 4.4.3 Clustering ........................................................ -51-

Chapter 5 Simulation of Design Alternatives .................................... -52- 5.1 Scenario ................................................................- 52- 5.2 Candidate SF- 1 : Sequential Fusion (Case 1 and Case 2) ........................... -54- 5 -3 Candidate DF- 1 : Static Decentralized Fusion ................................... -56- 5.4 Candidate DF-3: Dynamic Decentralized Fusion ................................ -58- 5.5 SelectionofArchitecture ................................................... -61-

Chapter 6 Validation with Real Data ........................................... -62- 6.1 Measures of Performance ................................................... -62- 6.2 Setup and Procedure ....................................................... -63- 6.3 Infrared Data Processing ................................................... -63- 6.4 RadarData ............................................................... 68-

Chapter7 Sumrnary .................................................... -70- 7.1 Objectives Met ........................................................... -70-

................................................. 7.2 Results and Design Choices -70- ................................................ 7.3 Novel Features of this Thesis -71-

................................... 7.4 Recommendation for Continued Investigation -72-

Annex A Project Background ............................................ -73-

Annex B An Altemate Technique for Transfemng Angular .......................... Measurements Between Coordinate Frames -74-

................................ References ,,., . -78-

List of Tables

Table 2-1 The Discrete Linear Kalman Filter (Dm ................................ -8-

Table 2-2 Extended Kahan Filter (EKF) ........................................ -12-

Table 2-3 Short List of Design Appmaches ....................................... -20-

............................................... Table 3-1 Modified-Baheti Filter -36-

Table 3-2 Validation Matrix for Hypothetical Trackiag Scenario ...................... -38-

..................... TabIe 3-3 Assignment Matrix for Hypothetical Trackhg Scenario -39-

List of Figures

Figure 1-1

Figure 2-1

Figure 2-2

Figure 2-3

Figure 3-1

Figure 3-2

Figure 3-3

Figure 3-4

Figure 3-5

Figure 3-6

Figure 3-7

Figure 3-8

Figure 4- 1

Figure 4-2

Figure 4-3

Figure 4-4

Figure 5-1

Figure 5-2

Figure 5-3

Figure 5-4

Figure 5-5

Figure 5-6

Figure 5-7

Figure 5-8

Figure 5-9

Figure 6-1

Figure 6-2

Figure 6-3

Figure 6-4

Figure 6-5

Light Armoured Vehicle (Reconnaissance) Version ........................ -3-

Discrete Linear Kalman Filter ......................................... -9-

DecentraIized (Sensor Level) Fusion .................................... -18-

Centraiized (Meamrement Level) Fusion ................................. 19-

Relationship Betweea State and Measurement Coordinate Systems ............ -22-

Sequential Architecture for A\LRSS .................................... -25-

Image P b Coordinate Systern ....................................... -29-

Mismatch Between Linear Cartesian and image Plane Dynamics .............. -29-

Static Fusion with Feedback .......................................... -32-

Dynamic Decentralized Fusion ........................................ -33-

Time Alignment at Local And Global Levels ............................. -34-

Sample Nearest Neighbour Data Association Scenario ...................... -38-

Target Location in the Image Frame .................................... -43-

Relationship Between Local and Globai Coordinate Frames ................. -44-

Application of a Bandpass Filter ....................................... -48-

Application of a Bandpass Intensity Filter Given Non-Zero Background ........ -50-

Simulated Course Profile for (a) Imager and (b) Radar ...................... -53-

Sequential Fitter Performance (Total Position) ............................ -54-

Sequential FiIter Performance (Y and X Position) .......................... -55-

Sequential Filter Performance (Total Position) ............................ -56-

Static Decentralized Filter Performance ( i and j image Position) .............. -57-

Static Decentralized Filter Performance (Total Position) ..................... -58-

Dynamic Decentralized Filter Performance (Total Position) .................. -59-

Dynamic Decentralized Filter Performance (Total Position) ................. -60-

Dynamic Filter with Range Propagation (Total Position) .................... -61 . .................................................... TargetScenario a

Infrared Video Image Captured by LAV Recce ............................ -64-

Histograrn of Typical Pixel Intensities .................................. -65-

Intensity Bandpass with Overly Wide Threshold .......................... -66-

Intensity Bandpass for PT > 0.95 and P, < 0.05 ............................ -66-

Figure 6-6 Clustered Pixels .................................................... -67- Figure 6-7 Isolated Centroid ................................................... -68-

Figure 6-8 Radar Setup ............-.......................................... -69-

List of Abbreviations and Principle Symbols

Abbreviations

AlLRSS

EE

FOV

IFOV

FLIR

JPDA

LAV Recce

LRSS

ME

MOP

MSE

MSTAR

MTI

NEES

NIS

NN

PDA

RMSE

Advanced LAV Recce Surveillance Suite

Expected Error

Field of View

Instantaneous Field of View

Forward Looking Infrared

Joint Probabilistic Data Association

Ligbt Armored Vehicle Reconnaissance

LAV Recce Surveillance Suite

Measurement Error

Measue of Performance

Mean Square E m r

Man-Portable Surveillance and Target Acquisition Radar System

Moving Target Indicator

Norxnalized Expected E m r Squared

Nonnalized Innovation Squared

Nearest Neighbour

Probabilistic Data Association

Root Mean Square Error

Principle Symbols (And Chapter Where First Used)

Estimated quantity (Chapter 2)

E m r tenn (Chapter 2)

Confidence interval (Chapter 2)

Background extent (Chapter 4)

Binary value of pixel (i j) (Chapter 4)

Pseudoestimate (C hapter 3)

Nonnalized Innovation Squared (MS) (Chapter 2)

Inadiance (Unit flux incident per unit area) (Chapter 4)

Observation matruc (Chapter 2)

Gcayscale value of pixel at coordinate (i j) (Chapter 4)

Vertical pixel coordinate (Chapter 4)

Horizontal pixel coordinate (Chapter 4)

Discrete time index for tirne t, (Chapter 2)

Micates at time t, given information up to and including t, (Cbapter 2)

Status up to and including time km, (Chapter 2)

Radiance (Chapter 4)

Probability of detecthg a target pixel (Chapter 4)

Probability of detecting a background pixel (Chapter 4)

Probability of detection (Chapter 4)

Probability of false alarm (Chapter 4)

Kalrnan Filter covariance matrix (C hapter 2)

Elevation (Chapter 4)

Enor fiinction (Chapter 4)

State noise covariance matrix (Chapter 2)

Measurement noise covariance matrix (Chapter 2)

innovation covariance (Chapter 2)

Variance (Chapter 2)

Continuous time index (Chapter 2)

Transpose operator (C hapter 2)

Target extent (C hapter 4)

Bearing (C hapter 4)

Time difference (C hapter 2)

State noise vector (Chapter 2)

Measurement noise vector (Chapter 2)

State vector (Chapter 2)

Chi square test statistic with n, degrees of fkedom (Chapter 2)

Position in 'x* coordinate (Chapter 2)

Position in 'y' coordinate (Chapter 2)

Position in 'z* coordinate (Chapter 2)

Observation vector (Chapter 2)

Chapter 1 Introduction

The ability to employ dissimiiar sensors to collect information concerning the behavior of a wide variety of

potential threats is critical to the sunrivability of high priority military assets. To this end, accurately

iocating and identi-g enemy targets forms the basis of an effective response- Consequently, research

into surveillance and tracking systems that best utilize available resources is a high priority.

1.1 Primary Objective

The objective of rhis thesis & fo d d o p an eflcient, r o k t and Meciive target tracking

architecture that combines active (range, bearing and elevation) information fiom a Moving Target

Indicator (MTI) radar and passive (bearing and elevation) information fiom a Fonvard Looking

Infrared (FLIR) imager.

The design of the architecture is intended as a potential enhancement to the surveillance suite aboard the

Canadian Forces' newly-acquired Light Armored Vehicle (Reconnaissance version). The "LAV Recce"

is the anny's primary surveillance asset at battk and brigade group levels. Its mandate is to provide

information on the enemy under al1 environmental conditions, across the fidl range of conflict, h m low

intensity to high intensity operations.

1 9 Tasks Entailed by Primsry Objective

Satisvng the prirnary objective entails:

1. Evaluating appropriate h i o n architectures and cboosing one on the basis of robustness.

computational efficiency and effectiveness; . . II. Selecting an appropriate tracking filter (or filters, depending on the architecture);

iii . Choosing maneuver detectiod compensation and tmck initiation schemes;

iv. Deveioping a method to extract a target's angular position h m analog infrared video images;

v. Evaluating the performance of the candidate algorithm via Monte Car10 simulation.

1.3 Secondary Objectives

This thesis constitutes the initial stage of a more comprehensive pmject commissioned by the Defence

Research Establishment Valcartier (DREV). The thesis is limited to a simulation of the single target

scenario while the larger project will incorporate an extension to the multiple target environment (by

adding a data association module), and include a validation of the algorithms with actual surveillance data.

Initial progess toward meeting the secondary objectives is documented in this thesis.

1.4 Assumptions

The architectures presented in this thesis assume the availability of the LAV Recce's Ku band doppler

radar (Surveillance mode) and 8-14 micrometer FLIR (wide FOV setting). For simplicity, the sensors are

assumed to be bore-sighted and stationary- The target profile(s) are constrained to travel at constant

velocity. To avoid unnecessary encumbrances, track initiation is assumed to have k e n completed at the

radar level. For the sake of generality, filter measurement models are developed in spherical coordinates,

even though the LAV Recce's MSTAR doppler radar provides only range and bearing information. The

limiting polar case for al1 equation can be obtained by setting the elevation component to zero. The host

platform for the proposed system Ml1 be referred to as the 'Advanced' LAV Recce Surveillaace Suite

(A\LRSS),

1.5 Perceived Hurdles

The LAV Recce is an operational vehicle, and as such modifications to the existing setup are not trivial.

A particular concem is the fact the sensors' displays are analog. The pmposed system, on the other

hand, requires a digital representation of the plan position indicalor (PPI) display, and digital FLIR image

sequences. Thus. the sensor interfaces would have to be modified to accommodate the proposed tracking

system. However, from a feasibility point-of-view, a radar demoistrator (based on the MSTAR) bas

already been configured at DREV that pmvides digital data- The FLiR images can be converted with a

' framegrab be r' .

1.6 Motivation

The LAV Recce (Figure 1-1) presently has no target tracking facility. While the addition of a simple

radar tracker represents an irnprovement in itself, this thesis demonstrates how an infiared imaging sensor

may be incorporated into the tracking loop to increase the precision of the estimation process. The

improvement relates to the fact that optical devices typically provide much better angular resolution than

radar. Furthexmore, themion of multipk sensors offers potentialiy more robust operational performance

and increased spatial/ temporal coverage compared to single seasor systems. Finally, automating the

overall tracking process will d u c e the operator's workload and d u c e fatigue-relatai errors.

Figure 1-1 Light Armowed Vehicle (Reconnaissance) Version

1.7 Related Research

A group at the Defence Research Establishment Valcartier (DREV) is currently investigating 10 ptential

improvements to the LAV Recce's Surveillance Suite (LRSS). While the proposed tracking algorithm is

envisaged as part of the 2003 mid-life upgrade, the feasibility of the pmject (in tenns of required

hardware/ sensor moditications) bas already been identified by one of the group members Cl]. While this

work was initiated by the author at the Royal Military College, it has since received both financial and

technical support iiom DREV. As mentioned, this thesis constitutes the initial stage of a more

comprehensive project that will be delivered to DREV as a technical report-

1.8 Background

The enhancement of a multi-sensor surveillance platfonn must be preceded by a detailed analysis of the

candidate system, and incIude a description of both its cunent roies and potentia1 capabilities. Initiai

documentation for this project was obtained k m the Land Forces Technical Staff College (LFTSC), the

Project Management Office (PM0 LAV) at National Defence Headquarters (NDHQ), and h m

Defence Research Establishment Valcartier [SI, 131, [4]. After gaining an initial understanding of the

current system's strengths and weaknesses, a meeting was held with two technical authorities h m

DREV to discuss adding a mdimentary target trac king facility b a s 4 on both the radar and infî-ared

sensors. The consensus is that the LAV Recce Surveillance Suite (LRSS) incorporates a set of vety

capable sensors (including a FUR, day carnera, doppler radar, laser range finder and audio input

interface). but is Iimited in its ability to exploit them to hl1 advantage. After identifjhg the general area

for investigation, fiirther inquiry was conducted to ground the development in a practical environment, and

to collect typical surveillance data with which to test the prototype system. A chronology of the

background preparation is attached as Annex A.

1.9 Technical Context

The design of the proposed system incorporates several mathematical methods h m the emerging field of

data fusion. In the present context, data fusion refers to a 'multilevel, multifaceted process dealing with

the detection, association, correlation, estimation and combination of data h m multiple sources to achieve

(i) refined state and identity estimates and (ii) complete and timely assessments of situation and threat'

[5]. A mode1 has been devised by the Unites States' Joint Directorate of Laboratories (JDL) to

categonze potential systems according to this definition (Table 1-1). As the goal of this thesis is to track

targets by fusing kinematic information (range, bearing, elevation) h m active and passive sensors, it

relates to Level 1 Fusion. Two other levels have also been designated [6]; however, they concem more

abstract issues (situational assessrnent and threat analysis) that do not apply to this thesis.

Level 1 - "Inference Level"

1 Objectives:

Functions - Level 1 -

Data alignment

Techniques - Level 1 - l

Coordinate transfonns Augmented rotation mapping l

Correlation Gating techniques Multiple hypothesis tracking (MHT) Joint pmbabilistic data association

(RDA) Nearest neigtibour 0

Kinematic attribute estimation

Recursive estimation (filters) Kaiman, a-& Interactive tIMM)

Batch estimation Maximum likelihood Hybrid melhods

Object identity estimation

Physical models Feature-base techniques

Neural networks Cluster algorithms Pattern recognition

S yutactic models Table 1-1 Description of Level 1 Data Fusion [6]

1.1 0 Thesis Organization

The thesis is organized as follows: Chapter 2 discusses the tracking problem. in particular the Kalman

filter is detailed as it foms the backbone of most modem real-time filtering systems, and because it is

used extensively in data fusion formulations. Chapter 2 also outlines the notion of data association and

introduces standard multisensor filtering architectures. In Chapter 3, a set of three application-specific

architectures are developed as candidates for the AILRSS. These architectures assume the FLiR

provides angular measurements of the target location. A method to generate these angles from FLiR

imagery is presented in Chapter 4. In Chapter 5, the candidate architectures are tested via Monte

Carlo simulation. The purpose of the simulation is to determine which architecture is most suitable to the

AILRSS. Chapter 6 describes a live trial perfomed to record actual data for validating the adopted

prototype (chosen in Chapter 5). The thesis concludes in Chapter 7 with a summary and

recornmendations for fûrther investigation.

Chapter 2

The Tracking Problem and the Kalman Filter

This thesis deals with many aspects of the tracking problem, h m target detection to data association.

Although they differ in fùnction, most of these areas rely in some way on the pmperties of an invaluable

mathematical tool - the unbiased linear minimum mean square e m r (MMSE) estirnator. This chapter

begins by introducing the most prolific MMSE algorithm - the Kalman filter- considered the workhorse

of tracking [7]. The statisticai properties of its output are discussed next, followed by a sketch of non-

linear filtenng in tenns of the extended Kalman filter. The remainder of the chapter concentrates on the

supporting elements of a cornprehensive target t r a c h g system, De facto techniques for maneuver

detection and compensation are outlined, as are standard approaches to architecture and data

association. These issues are revisited in Chapter 3, which expands on the key issues to motivate

appropriate design choices-

2.1 Discrete Linear Kalman Filter (DKF)

Generating an 'optimal' estimate of a system's state, X(k), from a set of noisy observations is a

classical filtering problem, initially champioaed by Gauss in his shidy of celestial orbits [8]. In general, a

state is an n-vector whose elements are determined completely by their previous instances; for

example, the state of a tank negotiating a course on the battlefield can be described in Cartesian

coordinates by

X ( k ) =[m ~ ( 4 dk) ~ ( k ) ml mlr (2- 1

where x(k), y(k) and z(k) represent the tank's position at time tk; *(k), j(k), and i ( k ) its velocity,

and where T denotes the matrix transpose operation. The evolution of a discrete system with linear

dynamics is described by the difference equation:

x(k + 1) = ~ ( k ) x ( k ) + ~ ( k ) ; ~ ( k ) - N ( o , M ~ ) ) k = O,I, ... (2-2)

where X(k+l) is the state at time t,,,; F(k) the n x n transition matrix; and where w(k) is a zero mean

Gaussian noise sequence reptesenting a random disturbance of variance Q(k)zO. in the previous

instance, if the target travels with constant velocity then ~ ( k ) = for AT 5 b, - t, and I3

= 3 x 3 identity rnatrix. Typically, not al1 elements of the system state are observable. Assuming that

the measurements are made in the presence of zero-mean Gaussian noise, then the m x 1 measurement

vector Z(k) is defined by:

~ ( k ) = H(L)X(L) + v(k); v(k) - N(O. ~ ( k ) ) (2-3

where H(k) is the n x m observation m a e and where ik) is the noise sôquence (with variance R(k)

2 O). For example, if it is only possible to record position in X-Y-Z coordinates, then H ( k ) = [I , 41.

The goal of the tracking problem is to estimate the desired state (position in this case) based on the

measurement process and knowledge of the target's dynamics. The 'optimal' solution minimizes the

mean square error (MSE) between the true state X ( k ) and the estimated state k ( k l k ) .

Specifically, it is required that

E [ ( x ( ~ ) - R(k~k)&' (k ) (x (k ) - k(k lk ) ) r ] = min

where S(k) is an arbitrary n x n weighthg matrix. Although Wiener formulated a MMSE algorithm in

the fkequency domain for Iinear the-invariant systems, it was Kalman who finally refomulated Gauss's

earlier resul ts to produce the ticst time domain recursive estimator for linear time varying (LTV)

systems. The equations describing the so-called Kalman filter can be developed according to

Bayesian, maximum Iikelihood or least squares philosophies. These derivations are well documented

[9],[10] and al1 result in an identical algorithm, which is summarized in Table 2-1 [9], accompanied by a

list of pertinent definitions and notation.

System mode1

Measurement mode1

Initialization

Prediction (propagation) equations

Correction (update) equations

Assumptions

Mscrete Linear Kaiman Fitter

k(w) = initial estimate o f state vector P(w) = initial estimate of e m r covariance matrix

k(k lk - 1). P(kk-1) n estimate

k time index (present a priori state vcctor and estimate

instant) error covariance matrix F(k) state transition matrix

H(k) observation rnatrix m k ) , P(kF) K(k) Kalman gain matrix

a posteriori state veetor and estimate (k) vector crror covanance matrix w(k) process noise vcctor

v (k) measurcment noise vector

Table 2-1 The Discrete Linear Kalman Filter (DKF) [9]

While an exhaustive derivation of the Kalman filter equations is outside the scope of this work; a

description of its mechanics is in order. At the h a r t of the filtering process is the construction of the

-8-

gain matrix, K(k). The gain is inversely proportional to the expeaed error of the estimate, P(k[k),

which is an n x n mavix whose ij" element is given by

P,(kl k) = E {(xi(k) - ~ ' ( k l k ) X ~ ' ( k ) - k i ( k l k ) ) ) (2- 15)

where i and j represent one of the n elements of the state vector. For instance, the expected e m r (EE)

in the x direction of the sptem described by Equation 2-5 cornponds to element i = 1, j = 1 of the

covariance matrix, while the expected em>r in the y direction corresponds to P&). Given the

expected error of the propagated (predicted) state estirnate, the pin can then be calculated. The gain

is then used to form the a posteriori state estimate as the optimal combination of the a priori state and

the present measurement (Equation 2-10). The a priori estimate X(k1k-1) is simply a propagation of

the previous update according to the assumed dynamics, and is thus the best estirnate at time k given

measurements until tïme k- 1 -

Figure 2-1 Discrete Linear Kaiman Filter

X(k+l lk) SYSTEM

X(klK)

9-7 z(k) A

> PREDICT FILTER . > P(klk)

. P(k+l lk)

w(k) v(k) A

(2-1 2) (2-13) i ., CORRESPONDING KALMAN FlLTER EQUATIONS ( )

Because it is recursive, the Kalman filter may be mechanized as a ceai-tirne algorithm (Figure 2-1).

Initial values for the state vector and covariance matnx must be assumed to begin the process. While

the most accurate information available should be used to initialize the state, the covariance is usually set

to a conservatively high value to insure gain does not increase t w quickly. High uncertainty resdts in a

Iow gain, which weights the measurements more heavily than the estimates. As the filtering process

continues the covariance decays exponentially toward zero, and the estimate receives increasingly more

weight. Thus an accurate system mode1 is critical. In practice, a 'track maintenance' facility must be

included to manually break the filter loop in the case where the target dynamics are no longer consistent

and the filter has diverged Loosely speakuig, divergence is said to have occurred when the difference

between true and estimated states increases over time.

2.2 Properties of the Kalmrn Filter

2.2.1 Covariance Propefties and Monte Carlo Simuîation

The fact that the Kalman filter calculates the expected estimate error (EE) is a very important property.

Specificatly, it allows the filter's effectiveness to be detennined by conducting a set of independent runs

and calculating the mean square e m r (MSE) between true and estimated states for each sample time.

Specifically. if r = 1 .. R nias are conducted, then the MSE for the estimate of the i element of the state

vector is calculated by

As R tends toward infinity, the MSE will converge on the expected error, P(klk), provided the system's

dynamics have been properly modeled; Le., for the same element of the state vector,

ei(klk) = E { ( x ' ( ~ ) - i i ( k l k ) ) ' }

This technique, called Monte CarIo simulation, is used extensively in this thesis to test the effectiveness of

candidate filters and architectures, Note that the best possible estimate of a linear system by any filter

corresponds to the Cramer Rao Lower Bound, which is equivalent to the expected error, P(wk) [ 1 1 1.

2.2.2 Innovation Properties

The difference between the measwed state and the estimated States over time forms the Gaussian

disûibuted innovation process, defmed by Z(k) , where

Nomalizing the innovation pmess with respect to its variance, S(k), results in a chi-square distributed

quantity with n= degrees of fkedom, where n= is the dimension of the measurement vector. This property

is important since it provides a standard basis for cornparhg the 'proximity' of the estimate to the true

state, which is usehl in detecting maneuvers- Specifkally, the normalized innovation squared (NIS)

sequence (denoted by q,j may be calculated by dividing the square of the latest innovation by its variance;

i.e,

~ ( k ) = ZT(k)S'(k)Z(k) k = 12.3, ... (2- 19)

where

The NIS accounts for the uncertainty in both the present rneasurement and the latest predication. For a

correctly modeled system, the probability that the NIS falls within tolerance X2, is given by

where (1 -a) is the desired confidence, and X\ is the chi-square test statistic for the given dimension

(available from standard mathematical tables 1121). This test also forms the basis for the gating pmcess in

data association, which is freated briefly in tliis thesis in Section 3.5.

2.3 Extended Kalman Filter (EKF)

In Section 2.1 the DKF was fonnulated as the optimum linear estimator. Frequently, however, the state

and measurement equations are non-linear, and are descrikd by the more general equations

X(k + 1) = f ( ~ ( k ) , k)+ w(k) ; w(k) - ~ ( 0 , QW) (2-22)

and

~ ( k ) = h ( ~ ( k ) , k ) + v(k); v(k) - N(O, ~(k)) (2-23)

where f ( ~ ( k ) , k ) and b ( ~ ( k ) . k ) are non-linear functions of the state. To obtain a tractable

solution to the estimation problem, it is essential tbat the non iinearities be expanded by linear

approximation. 'The most common approach is to employ a power senes mund k(&lk - 1). i.e.,

This yields the Extended Kalman Filter (EKF), whose equations are presented in Table 2-2. The EKF

provide a recursive means of computing an approximation to the MMSE estimator of X(k), despite non-

linearities in the system andl or state mode1 [9].

Extended Kalman Filter (EKF)

System Measurements

k(w) = initial estirnate of state vector

P(O(O) = initial estimatc o f error covariance rnatrix

Predic tion

Correction

Defini tions

Table 2-2 Extended Kalman Filter (EKF) [9]

Although the EKF is similar to the linear DKF, it also differs in two cntical ways: First, its precision is

-12-

governed by the number of higher derivative temis kept in f(x(k1.k) and L(x(&). k ) . Second,

its gains cannot be pre cornputeci. As might be imagined, the EKF can be computationally expensive.

Possible remedies to these two independent problems include (i) reformulating the filter as a nonlinear

estimator in the case of large nonlinearities, or more simply, retaining higher derivative ternis in the

power series expansion; and (ii) expanding the measurement and state Jacobians (Eqetions 2-35,36 )

around a pre-set pseudolinear trajectory (which will allow the gain to be precalculated and stored at the

cost of increased bias).

in the present application, computation requirements are helped by the presence of a linear state model;

however, a further reduction in computation can be achieved by employing a slightly suboptimal

reformulation in the Modifieci-Baheti filter (Chapter 3) [13].

2.4 Functional Elements of a Tracking System

Up to this point, Chapter 2 has concentrateci on defining the optimal estimator for the target state under

the assumption that the target has been identified and that it's dynamics are completely specified. in

practice, this cannot be guaranteed. A reaiistic course profile will be time varyhg; thus it becomes

necessary to include a mechanism to not only detect the onset of a maneuver, but also to compensate

for it. A fully functional and robust tracking system must also include several components apart fiom the

filter itself A track initiation scheme is necessary to begin the estimation process and to generate the

initial state, while a data association facility is needed to assign measurements to their origin when

tracking in clutter and to resolve the multiple target case. Finally, the non-trivial issue of choosing an

architecture must be addressed before the design can kgin in eaniest. These issues are now discussed

with the goal of elucidating appropriate strategies for the A\LRSS.

2.4.1 Maneuver Detection And Compensation

In principle, there are many ways to accommodate a maneuvering target profile. Generally, these

techniques may be divided Uito two categones: those that assume a priori knowledge of the maneuver,

and those that do not. in the first case, the interactive Multiple Mode1 (IMM) strategy [14] employs

several filters to follow a target, each according to diffemt set of assumed wget dynamics. Bar

Shalom has akso proposed a variable dimensional filter that switches between constant course and

constant acceleration state models at the onset of a maneuver [15]. To some degree, both of these

techniques require knowledge of the target's bue motion.

However if no a priori kmwledge is available, it is still possible to handle maneuvers by includhg a

noise term in the state mode1 (see Equations 2-5,2-12). This effectively prevents the filter gain fkom

decaying to zero, which would result measurement process receivùig inadequate weight. Altemately, if a

maneuver is detected (by m o n i t o ~ g the innovation process), the filter's covariance matrix, P(ktk), can

be reinitialized to favour the measurements and reacquire the track. Because of their simplicity, the

latter two techniques will be considered the primaty candidates for application to the AILRSS. In the

case of a decentralized architecture (discussed later), separate filters are used to estimate the state at the

local (sensor) level and at the global level. In this case, maneuver detection and compensation would be

carried out at the sensor level; Le., the iocal filters' covariance matrices would be reinitialized.

2.4.2 Track Initiation

Before the kinematic properties of a target can be esthatecl, it is necessary to acquire the track in the

first place. Track initiation can be accomplished as a side effect of data association algorithms which

implicitly assume that each measurement might originate tiom a new target. Otherwise, a separate

mechanism must be incorporated to match the development of a profile to the assumed target dynamics.

Essentially, the process involves predicting the next location of the target based on the previous

measurements. If the predicted and rneasured points coincide, then the track is initiated.

2.4.3 Data Association

When operating in a cluttered andl or multitarget environment, the ability to assign measurements to their

sources is essential. Various techniques exist to accomplisb the socalleci data association; generally

they fa11 within one of two categories: measurement oriented techniques and track oriented techniques.

While the suitabiiity of one method over the other is case dependent, it is generally true that

measurement oriented techniques offer greater pediommce at the pnce of higher computational

requirements and complexity, while track oriented techniques still provide a potentiaiiy robust meam of

attributhg measurements to target tracks yet with significantly less overhead. The latter approach is

more attractive for the present design. A briefdescription of the candidate methodologies helps put hto

context the eventual choice, as detailed in Chapter 3.

2.4.3.1 Measurement 0- Techniques

The predominant feature of measurement oriented data association is iîs ability to defer the ultimate

assignment decision uotil a desired confidence level is met. Practical algorithm are usually a variant of

the so-called Multiple Hypothesis Tracker (MHT) 1161, which makes strictly one-to-one measurementl

target assignment under the assumption that they are exact- Specifically, each new measurement zi E 2,

collected at time k is assigned a set of hypotheses, R(k), such that

T( j ) 3 meas belongs to existing track i = l NT mcas bclongs to new track i = 2

FA s meas is false alarm i = 3

where Tu) represents the jrh track. in hi@ clutterl high target count environments, the MHT becomes

unwieldy over time because new tracks are required for every measurement to account for each

hypothesis. To reduce the computational bwden, various methods have been proposed to elirninate or

'prune' established track files that are probably not correct. The probability of association of each

hypothesis Q(k) given the present measurement set Z(k) E (z,, z2, ... ,zk} is a factor of the probability

of detection, PD, the density of false alarm, BFA , and the density of new targets, 8, [L6],[17].

Assuming the sensor reports the number and location of each target, the probability of association is:

P ( k ) = P ( . ; ( W ( k ) ) (2-3 7)

Apart fiom its potential accuraçy, the MHT offers the obvious advantage of initiating and maintaining

tracks implicitly in the association pmcess; this is not the case with any of the track oriented appmaches.

In practice, the MHT has k e n applied most extensively to highly dedicated tracking systems which

employ a measurement level fiision topology (centralized or sequential).

2m4m3.2 Tnck &iented Techniques

Track oriented techniques emphasize computatiod ef'fïciency over exhaustiveness. Whereas the MHT

considers al1 previous measurements in the assignwnt process (and actually defers making the decision

until enough additional information has been collecteci to guarantee the correct decision), track oriented

techniques use only the most recent observations to make irrevocable assignments for a set number of

targets. Attendant methods to begin the tracking process and maintain the tracks are thus necessary if a

track oriented approach is used; these are described in Chapter 3. Several variations of track oriented

data association procedure exist, ranging h m the mdimentary nearest neighbour (NN) algorïthms to the

significantly more complicated Joint Probability Data Association (JPDA) f r l t e ~ A brïef description of

each is given here.

Nearest Neighbour (NN) Data Association [161. As the name suggests, the nearest neighbour

approach makes associations according to a rule that minimizes the collective distance between al1

measurements and tracks. Only measwements within a precalculated gate are considered for association,

and in the absence of a candidate observation, the track may be either dropped or updated with the

predicted state. The nearest neighbour can assiga each measurement to at most one target, and practical

implernentations require the that the number of associations is maximized Employing this rule leads to

conservatively suboptirnal results; however, the algorithm becomes much more efficient as it reduces

number of potential observation-to-track scenarios. The main weakness of the temporarily and spatialiy

'hard' sequential NN is that it can fom its decisions according to a marginal confidence level, and c m not

accommodate situations requiring that one measurement be shared by multiple targets. Thus, the NN is

not well suited to situations with several closely spaced (and not consistently resoIved) targets. To

alleviate some of its weaknesses, various track-splitting methods have been proposed. These

modifications create altemate tracks which improve the chance of correct assumptions in ambiguous

conditions.

Probabilistic Data Association (PDA). The PDA algorithm improves upon the nearest neighbour

approach in that it is not limited to assign at most one measurement to each target. That is, every

measurement within a track's validation gate will be used for the next update, each weighted by the

reciprocal of their respective distances to the gate centre (i.e, the location of the last a priori estimate.

Thus, the PDA performs averaging over every observation-to-track hypothesis that has roughly the same

likelihood 1181. There is, however, no guarantee that the PDA will yield better results than NN, especially

in high signal-to-noise/ low clutter Environments.

Joint Probabilistic Data Association (JPDA). The PDA or 'all neighbour' strategy is noteworthy as it

allows tracks to share measurements. The JPDA improves upon the PDA in the rnultitarget sceuario by

also letting measurements share targets. Its key is the evaluation of the conditional probabilities of these

two joint association events; specifically, a variable, 0, is used to account for al1 possible target/

measurement associations [ 161, where

for

measurement j on'ginates 9, = P J = I .... mk t = O ..... T

fiPm target r

Assuming that validation gates are not used to eliminate unlikely measurements h m consideration. then

the JPDA is equivalent to a zero scan back MHT, minus track initiation- As such. the joint events are

akin to the MHT hypotheses. The JPDA was developed at the same time as the MHT, and like its rival,

has been applied in various forms to specific problems [19]. While it is versatile the JPDA can also be

quite cumbersome, and for this reason sometimes loses its appeal.

2.4.4 Architectures for Kinematic Data Fusion

The goal of a typical tracking system is to calculate the minimum variance estimate of a target's state.

By adding redundant sensors to a surveillance platforni, it is possible to reduce Mher the estimation

error. However, the mathematical solution to the MMSE problem in the case of multiple sensors may be

achieved by more than one mathematically equivalent formulation, which is important because an efficient,

robust and effective overall system depends on a judicious choice of architecture. To this end, identifiing

a candidate architecture at an early stage of the design is criticai, as the choice bas an impact on the

attendant selection of data association and track initiation mechanism. In the next section, decentralized,

centralized, and sequential architectures are introduced, as they are discussed in 1161, [20], and [21].

2.4.4.1 Decentralized (Sensor kve) Fusion

Decentralized architectures maintain independent tracks at the sensor level. and generate a single master

track at a higher levei 1211 (Figure 2-2)- The local tracks are updated over al1 measurements; however, to

save on bandwidth and central processing reqhments, somethes ouly a subset of the locally filtered

quantities is passed to the global node, There is no restriction on lateral communication in cases where

feedback is required to produce complete local reports h m dimensionally insufficient sensors. Because

of its topology, decentralued architectures are inherently weU served by track onented &ta association

techniques. Its main benefits are robustness and parallelism (if distributed processing is favoured).

Several variations of optimal and suboptimal decentralized architectures have been pmposed in the

Literature [2 11, [22], which typically trade performance for computational efficiency. Given the premium

on fault tolerance, the decentraüzed architecture was identified as the early favorite for application to the

AVRSS, contingent on its ultimate perfomiaricc against a nval seqyentïal stmcture. Two task spec&

decentralized architecture are developed in Chapter 3 and later simulated in Chapter 5.

SENSORS LOCAL

PROCESSORS

GLO6AL A PROCESSOR

MEASUREMENTS LOCAL ESTIMATES

Figure 2-2 Decentralized (Sensor Level) Fusion 1161

2.4.4.2 Centralùed (Measul~nent Level) Fusion

Whereas in decentralized fùsion separate tracks are maintained at each seosor, with centralized

architecture, the cornpiete set of measurements is reported directly to a single processor (Figure 2-3).

Because all observations are available, centralized h i o n d e s weU with MHT, allowing highly accurate

data association aven extensive processing faciMies. However, it is susceptible to track contamination

should just one of the sensors fail; whereas with decentralized fiisioa, the faulty sensor could be pulled

from the system without affecting the other local tracks [21]. Thus, cenaalued fusion ofiers slight

potential advantages in accuracy that might not k m i k d in praftice.

SENSORS

i =2 GLOBAL k (k l k)

PROCESSOR P~(klk)

MEASUREMENTS Xo(k+l Ik) T Figure 2-3 Centralized (Measurement Level) Fusion

2.4.4.3 Sequential Fusion

Sequential fusion is similar to the centralized architecture in that a single filter is used to process aii

incoming measurernents; however, unlike the latter, it does not requires additional logic to align temporarily

non-cornmensurate observations before fhsing them. Sequential fbsion architectures are inherently

appropriate for multisensor systems because they can accommodate a single state mode1 and multiple

measurement models. Specifically, separate gains are rnaintained for each sensor to pnxluce the a

posteriori estimates. Sequential fiision offers optimal performance but is charactecized by the same

weaknesses as centraiked h ion . As it is the de facto approach [23] [24] for fùsing i b d and radar

data, sequential fusion was given çtrong consideration as for use in the A\LRSS.

2.5 Short List of Design Appmaches

This chapter has motivated the tracking and data h i o n problem as an application of estimation theory.

The practical components of a comprehensive tracking system were presented in order to outline the

choices available in designing the A W S . Based on an initial review of options, a short list of

approaches may be constructeci (Table 2-3)-

1 Candidate Iksim Strotegies 1

1 Architecture: 1 SequenfiuI Fusion 1 DecentraIîzed Fusion 1 Track Initialization:

Data Association:

Sequential Method 1

-

Nearest Neighbour

1 Maneuver compensation: 1 Reinitialuution of Covuriunce MarrLr at Senror M e / 1 Table 2-3 Short List of Design Approaches

Emphasizing the efficiency requirement eliminates iMM and MHT for maneuver compensation and &ta

association respectively, while the need for robustness suggests a decentralized architecture. In this

regard, sequentia l initiahation and Nearest Neighbour assignent are particularly attractive strategies, not

only because they are efficient, but because they represents a logical starting point for a preliminary

design. The case for sequential fùsion &as already been motivated.

Chapter 3

Detailed Evaluation of Candidate Architectures

In Cbapter 2, various design requirements were visited at a descriptive levei, resulting in a short list of

candidate strategies for architecture, and a final choice for data association and mck initiation schernes.

In this chapter, each class of architecture is developed as a set of one or more practical algorithms,

starting with sequential fùsion. Data association and track initiation are then addressed. A final choice

for one approach over the others is made later based on their performance in simulation (Chapter 5).

3.1 Candidate SF-1: Sequential Fusion

As discussed in Chapter 2, the sequential filter offers potentially accurate results and intrinsically handles

asynchronous data rates, but has low fault tolerance 12 11.

3.1.1 State and Measurement W s

In this application, the state comprises of the three-dimensional Cartesian positiodvelocity vector

X W =[x(k ) y(k ) ~ ( k ) W) YW) i(k)IT (3- 1

The system mode1 for the constant velocity target profile is thus given by

X(k + 1) = F ( k ) X ( k ) + w(k) ; w(k) - N ( o , P ( ~ ) ) (3-2)

where

and

Note that

AT = [tk -t& O (3-5)

is the generat time di fference between the two most m e n t measurements which amve at random fiom

-21-

either sensor at times t, and &-,.

The measurement mode1 diffets for radar and imager updates (Figure 3-1). For the radar it is given by

zR(k) = hR(x(k) ,k )+vR(k) ; vR(k) - ~ ( 0 , ~ " ( k ) ) (3-6)

where ZR(k) E d3 includes measured range r(k), bearing û(k), and elevation m). These quantities are

related to the state by the non-iinear observation vector

r = range 8 = bearing 41 = elevation

Figure 3-1 Relationship Between State (Cartesian) and Measurement (Sphencal) Coordinate S ystems

The imager measurement

zl(k) = hl (x (k ) . k)+ v'(L); d ( t ) - N(O.R/ ( k ) )

is similar to Equation 3-6. except that range is not observed. Z1(k) E !Jt2 is explicitly described by:

The measurnent noise is assurned to be uncorniateci both between sample and is given b y

~ " ( k ) = ~ [ v ~ ( k ) ( v ' ( k ) ) ' ] = d i a ~ [ . ~ , , 4. +]

R' ( k ) = ~ [ v ' ( t ) { v ' ( k ) } ~ ] = dio.[c$. $1

3.1.2 Filter Formulation

The sequential filter is formulated as an extended Kalman filter; however separate update equations must

be generated to accommodate the differing measunment models. Referrïng to Table 2-2, the Jacobian

H(k) and Kalman gain K(k), are computed differently, depending on whether the rneasurement cornes

fiom the radar or imager.

Radar Update Equations. In the radar case, the conversion between Cartesian and Sphencal

coordinates is accomplished by evaluating

with respect to the a priori Cartesian state estimate, giving

where r, and r2 (A) represent

and

The gain is thus calculated by

~ ' ( k ) = ~ ( k l k - 1 ) ( ~ ' ( k ) ) ~ [ ~ " ( k ) ~ ( k ~ k - I ) { H " ( ~ ) } ~ + ~ " ( k ) ] - '

where X(klk- 1) and P w 1 ) are the a priori state vector and covariance e m r matrices, respectively . Accordingly, the state is updated by

xR(k1k) = K(klk - 1) + K ~ ( L ) [ z ~ ( R ) - ~ ( ~ ) k ( k l k - LI] (3-lf)

andcovarïanceby

P(kl k ) = [I - ~ ~ ( k [ k ) ~ ) ~ " ( k ) ] ~ ( k l k - 1) (3- 1 8)

Imager Update Equations. The equivalent update equations for an imager measurement are developed

in the same approach- We have,

~ ' ( k l k ) = k(k(tlk - 1) + ~ ' ( k ) [ ~ ' ( k ) - ?i(k)k(klk - 1)]

P(k(k) = [I - ~ ' ( k l k ) f f ' ( k ) ] ~ ( k l k - 1)

Radar and Imager Prediction Equatioas. Because the arriva1 time of the next measurement is not

known, the a priori estimates of the state and e m r covariance matrix must be calculated before the

measurement update at t h e & ; i-e,

X ( k l k - 1) = ~ ( k ) k ( k - llk - 1) (3-24)

Although Equations 3-24 and 3-25 are calculated in the same manner as the 'prediction'equations of Table

2-2, the latter form ernphasizes the required physical structure of the present filter, and avoids the

mistaken assumption that AT is constant. Thus, as soon as the system receives a measurement h m

either radar or imager at time k, it calculates the a priori estimate for time (klk-i) imrnediately followed

by the a posteriori estimate for the (klk) index. The filter's timing is depicted in Figure 3-2, with emphasis

on the gain that must be calculated separately depeadiag on whether the m e a m m e n t came from the

radar or imager.

Figure 3-2 Sequential Architecture for A L S S

3.1.3 Remaiks

The sequential fomulation has received favor in case studies of highly dedicated active/ passive aackers.

For instance. Romine and Kamen [24] developed a similar filter which includes additional States to

account for the di fference between the target 's centre of reflection (which the radar observes) and its

geomenic centre of mass (wbich the imager observes). A simulation of the sequential filter detailed in

Section 3.1.2 is performed in Chapter 5 for two cases: (i) equal update rates across sensors, and (ii)

unequal update rates.

3.2 Decentralized Architectures

Decentralized h i o n is more robust than sequential fiision as its maintains separate local tracks. It may

be implemented in optimal and s u b - o p h l configurations. In this thesis, two suboptimai approaches are

considered as well as one optimal design. They are, in order: Static Fusion (DF-l), Static Fusion with

Feedback (DF-2), and Dynamic Decentralized Fusion @F-3).

3.2.1 Candidate DF-1 : &tic Deceirtrslized Fusion

3.2.1 .l Background 121

Static fusion combines the locally produced üwar minimum variance 0 estimates %(kl k) for al1

A A

i = 1 .. p available sensors to pioduces a global estimate, X , (klk) defmed by

The estimation error is minimized according to the performance index

where Mi is an unknown constant weight applied to each local estimate, and Xe) is the tme state. The

result is a h e d estimate that minimizes the square of the estimation emr,

e,(k) = X(k) - ~~k~(k(k) (3-28)

which is assurned to be orthogonal. Obeying this criterion produces an estirnate for the fonn:

which cm be recast into the ' information' form:

where, in general,

is defined as the 'pseudoestimate'. The information fom is advantageous as it can reduce the number of

required matrix inversions.

Static h i o n is the least computationally demanding decentraiized technique because only the most recent

local estimates are used to form the global estimate-, and because information is only pessed in one

direction (local to global level). However, because it does not condition the local estimates with global

information, it is less accurate than the 'dynamic' fùsion detailed in Section 32.3- An improvement c m

also be achieved by feeding back the global information to the local nodes to produce 'static fusion with

feedback', as in Section 3.2.2- Before exploring the latter options, a static formulation is developed,

3.2.1 9 Application of Static Furbn to the AURSS

In the present context, the goal is to produce a complete global estimate (E S3) h m local imager (T) and

radar (R) tracks (E 3t2 and E 3t3 respectively), The h i o n must be performed in spherical coordinates

(denoted by subscnpt S) in order to decouple the cornmon quantities (bearing, elevation) fiom range,

which cannot be estimated by the imager- It is assumed that each sensor bas made available the

following pseudoestimates:

and

To take advantage of the constant velocity assumption, an extended Kalman filter with Cartesian state

mode1 is employed at the radar level. The resulting estimates and P matrix must then be transformed to

Spherical coordinates using Equations 3-7.

Bearing and elevation estimates may be produced at the FUR level using the linear image Plane filter- A

complete point for Fusion can then be f o m d by assigaing an arbitrary range. to augment the imager's

state to S3 (range, bearing, elevation) [20]; Le.,

Because F is arbitrary it should be assigned no confidence (it is only incIuded to produce a dimensionally

compatible point); thus, the correspondhg entries of the inverse covariance matrix for the imager estimate

are set to zero, leading to

[ i y ( k l k ) i t =

The two local estimates c m then be fused in information form. The global quanttities are then gïven by:

d,"(klk) = <îS(klk) +&(klk) (3-37)

and

[ e j G ( W r L = [p:(klk)r' + [ ~ ~ ( k l k ) i ' (3-3 8)

3.2.1.3 Gemrating Local Psmdoedtnates

In the preceding development, it was assuned that appropriate locat pseudoestimates are produced by

both sensors; specifically, that the radar filter provides a three dimensional estimate in Cartesian

coordinates that may be converted to range, beanng and elevation, and that the FLIR filter p d u c e s a

corresponding estimate of bearing and elevation. The production of the radar estimate via the extended

Kalman fiIter bas already been discussed. Two appmaches to the F L R filter are now considered (Case

1 and Case 2).

Case 1 : Linear image Plane F i h . The most efficient strategy is to model the target's centre of m a s as

a single pixel that negotiates a constant velocity course in the i-/' image plane (Figure 3-3). This allows the

DKF to be employed to f d l effecf resulting in a very efficient formulation. The conversion between pixel

coordinates and angular coordinates is achieved by the straightforward method presented in Section 4.1-

Bar Shalom demonstrates good results using a single image plane filter to track a target whose dyoamics

are unluiown [25]; however, in this case the given systern model (constant velocity in Cartesian

coordinates) does not infer h e a r kinematics in the image plane (which implies constant angular rates);

thus, a mismatch between the local filter state models is incorporated (Figure 3-4). Despite this inherent

flaw, the use of the linear image plane filter was not ruled out in case the mismatch pmved to have a

negligible eflect.

j [pixels]

Figure 3-3 Image Plane Coordinate System

B!!l

Target Centoid

Figure 3-4 Mismatch Between Lînear Cartesian Dynamics and Linear Image Plane D ynamics

2: IlodiMd-Polar FP$r m. Second, the Modified-Polar (M-P) filter is considered, which pmvides

an exact mapping between linear Cartesian dynamics and Polar1 dynamics, thus ehinating the mode1

mismatch. The Modified-Polar filter differs h m the standard polar filter in that the inverse of range is

estimated (instead of range); defuiing the state this way (see Y(t) below) decouples the observable

components of the state vector (bearing rate, range rate over range, bearing) from the unobservable

component (range) assuming chat range cannot be measured,

The Modified-Polar (MF) filter refonnulates the constant Cartesian velocity problem using an EKF with a

non-linear state mode1 and iinear measurement modet The state vector, denoted by Y(t) in this case, is

given by

[rad I s]

The non-linear differential equations describing the evolution of the target course (constant veloçity in tbe

Cartesian plane) are given by:

To estimate this system with the EKF requires that Equation 3-40 be integrable in closed fom. While an

advantageous mapping between Cartesian and Polar spaces makes this possible, the resulting filter

includes a highly non linear state Jacobian, which unfortunately, proves too cumbersome to justiQ using

this formulation.

3.2.1.4 Rernarks

A simulation of Candidate DF-1 (Linear image plane filter) is performed in Chapter 5 to gauge the

' The M-P filter is discussed in terrns of generating an estimate for range and bearing. The elevation component could be estimated with a second filter.

effectiveness of the static appmch, in spite of the mode1 mismatch. Static hision using the Modifïed

Polar filter was discounted as an option due to the sigaificant computational requirements. (a fact, using a

dynamic technique to circwnvent the observability issue promises both more accurate results and less

costly implementation. Before the dynamic approach is detailed, another initial candidate is presented:

Static Fusion with Feedback.

3.29 Candidate DF-2: Static Fusion with Feedback [16]

In the very early stages of this project, Static fùsion with feedback was identified as a strong contender

for the final architecture. The scheme is attractive because it genetates an optimal estimate if the global

processor is slewed to the local rate, and demands only a slight increase in computation with respect to

static fusion. Accuracy is increased by feeding back the global track information to the local level, where

it is used to condition sensor level-estimates. The pseudoestimate equations conesponding to the process

as depicted in Figure 3-5 are [2 11:

d, (kfk) = ~-'(k)&(k - Ilk - I ) + H,(k)R;'(k)Z, (k) ( 3 4 1 )

and

where

Provided the global update rate matches the slowest local filter update rate, static fùsion with feedback is

mathematically equivalent to fidl dynamic h i o n [163; however, it saves oa computational requirements for

given state and measurement equations. Unfortunately, closer analysis reveals that this configuration is

impractical given the dimension of the FLIR level estimates. Specifically, as seen in Equation 3-41, the

technique requires a cornmon state model in order to advance the global estimate. The only way to

accommodate this constraint is to employ two modified polar filters, or to recast the dynamic model to

represent a constant angle rate (which is not a practical assumption). As an architecture based on the

MP filter has already been nileci out, the static fiision with feedback is eliminated as a design option prior

to simulation, leaving fidl dynamic firsioa as the remaining decentralized contender.

SENSORS LOCAL

PROCESSORS

- 1 GLOBAL PROCESSOR

LOCAL MEASUREMENTS PSEUDOESTIMATES

Figure 3-5 Static Fusion with Feedback [20]

3.2.3 Candidate DF-3: Fully Oecenfralized Dynamic Fusion

Fuily decentralized fusion provides the optimal LMV estimate for the global mean. Stated othenivise, the

formulation represents the theoretical best option in tenns of performance. The equations for

decentralized fusion may be developed in either covariance or information fom; however. the latter is

more appropriate for practical implementation (see Section 3.3). The global a priori pseudoestimate and

inverse covariance matrix are defined by [ 161: N

d , (k + 11 k) = f-'(k)&(kl k - L) + x [ d l (k + 11 k ) - F - ~ ( ~ ) J , (kl k - l)] irl

and

Inspecting Equation 3-45 reveals not only that the global and local states vectors must comprise the same

quantities, but aiso that the states must be updated by a commoa transition matrix. These requirernents

are circurnvented by employing two similarly configured extended Kalman filters to process both radar

and image data To create a complete estimate in the second case, a range terni is assigned to the imager

angles. Instead of using an arbitxary value with infinite variance (as in the static case), the wist ment

radar measurement is fed into the image fdter, and propagated to the required time for global processing.

SENSORS LOCAL

PROCESSORS

- - - R = Range 8 = Bearing LOCAL C#I = Elevation PSEUDOESTIMATES

(C

U k ) di(klk) I

- - -

Figure 3-6 Dynamic Decentralized Fusion

9 4 1 pl(klk) radar

This method could have also been used in part to accomplish sîatic h i o n with feedback; however, it is

desired to minirnize the amount of feedback in the architecture to avoid the dissemination of corrupt data

should one or both sensors fail, A single unidirectional transfer is required by the present configuration

(radar to imager). On the other han& the design of a static system with feedback would also cequire the

retum of the global estimate to the local fiiters.

*

d4kM

3.2.3.1 Timing ConsWraticm

Until now, the candidate filter formulations have been presented under the assumption that each sensor

provides its estimate to the global prccessor at a unifom rate. in this case, the assumption is untenable.

While the imager updates at IO h m e s per second (a constant), the radar reports only when the target

appears in its line-of sight. Because the radar antenna rotates back and forth between two arcs at a

moderate rate at best, the time between intersections wiU be govemed by a nonlinear hc t ion and almost

certainly be non-uniform. To deal with the unsynchronized measurement schedule, a stight modification to

the fusion architecture is necessary. SpecEcaiiy, Equatious 3 4 5 and 3-46 mua be recast in a more

general fonn. Letting the a posterion time indices (tJW k fepresented simply as (W. the global

pseudoestimates can be fonned by p l :

and

where t, and t, are the indices o f the most recent global and local estimates respectively. It is assumed

that t , 5 t , < t,,+,,i As depicted in Figure 36. the additional ternis in Equatioas 3 4 7 and 3-48

propagate the local estimates generated by the radar (sensor i = 1) and the imager (sensor i = 2) to the

global update tirne. The time-aligneci local estimates are given by

, ( t k t ) = ~ - r ( t , t ) , ( t t ) i = 1 , 2 (349)

and

( t ) = F r ( t k t ) l ( t t ) ' ( t k , t ) : i = 1, 2 (3 -50)

fk- f tk tk+l - y---- f -*-- ----- *--b GLOBAL

Time i tkl : f(k+l) 1 : -

[SI . . . . . . . . radar . . I-)i i =1 . . . .

f(k-1)2 lk2 f(k+1)2 -- e---.-- -- -p - - -0- . . . . imager . - . . . . . . . . . . i = 2 . . . . A

dt(tki 1 hi) dr(k 1 bci) Note: TFLIR = TG = const TRAOAR = variable

Figure 3-7 Time Aligment at Local And Global LeveIs

-34-

Candidate DF-3 is simulated for 2 cases in Cbapter 5: (i) equal imager/ radar update rates and (ii) unequa1

update rates. Having eliminated Static Fusion with Feedback OF-2) eatly h m consideration, DF-3 joins

sequential (SF-1) and static decentralized @F-1) options as one of three possibilities for the AURSS,

3.3 Replacing Stanôarâ EKF with More Efficient Formubtion

In Section 2.2, the extended Kalman filter (EKF) was fonnulated for use in this application as the

minimum variance estimator. While the EKF is essentially optimai, it is not necessariiy computationally

efficient. In this case, slight modifications to its standard form can be made which afford a significant

computational savings with a negIigible sacrifice ofperformance. The drawback to fomulating the state

equation in Cartesian coordinates is the need for a non-linear measurement mode1 (Equation 3-52), with

which to process the sphe~cal observations. However, it is possible to replace the standard measurement

equation by a linear fùnction (Equation 3-53), afier transferring the nonlinearities to the e m r tenn. The

reformulation allows the gain matrix to be calculated in sphencal coordinates; and only later rotated into

Cartesian space for the state update. The resulthg 'Modified-Baheti filter' [13] (Table 3-1) is at least

twice as efficient as the standard EKF [13]. Note that the formulation employs a rotation matrix

(Equation 3-65) to map the emor covariance matrix between sphencal and Cartesian coordinate systems.

This key approximation assumes the offdiagonal terms (range rates / angular rates) are minimal, a valid

assumption given the present target dynarnics.

3.4 Data Association and Track Initiation

The objective of this thesis is to design an efficient, robust and effective tracking system that best exploits

the existing facilities aboard the LAV Recce. Accordingly, al1 subsidiary systems proposed for the

AlLRSS must not only be diable, but also cost effective. The issues of data association and track

initiation are now addressed according to this philosophy.

3,s Data Association

In a multi-target and/ or cluttered environment, a technique is needed to associate al1 incoming

measurements (across ail sensors) to their respective sowces. As outlined in Chapter 2, &ta association

can be accomplished via Multiple Hypothesis Tracking 0, Probabilistid Joint Probabiiistic

System mode1

Measurement mode1

Prediction

Correction

Information representation

Definitions

*(OP) = initial estimate of MP state vcctor

S-' (OP) = initial estimate of inverse spherïcal enor covariance nratnx

Hcn =[f, O,]

R= = F,R=F ,~

c E cartesian quantity s i spherical quantity

Table 3-1 Modi fied-Baheti Filter

Data Association (PD& JPDA), or by Nearest Neighbow (NN). While optimal association is achieved

by MHT, the approach requires an inordinate amount of pmessing and memory cesouces to generate

and maintain its hypotheses. The JPDA represents a more attractive approach, although it also demands

a signifrcant computational overhead, The Nearest Neighbour technique, on the other hand is both

eff~cient, and cm dso be very effective pmvided the target count and false alann rate are not too hi&

In the present case, the NN is a logical choice because: (i) at present the LAV Recce has no data

association facility, wbatsoever, and it is reasonable to approach an upgrade by methodically increasing

the prototype's complexity until the desired performance/ cost ratio is ceacheci; and (ii) there is no

guarantee that a more cornplicated technique will yield significantly better results. Furthemore, because

the radar and the image tracker are developed concurrentiy in this project, the need for a more

complicated technique cannot be established at this point,

3.5.1 Nearest Naighbour ûata Association

Nearest Neighbour (NN) data association makes irreversible 1 : 1 assignmeats of current measurements to

existing tracks by detennining - statistically speaking - the most likely combination of these entities. To

formulate the assignment problem as a practicaf algorithm, sets of vaiidation and association matrices are

used to detemiae fmt the pemissible combinations, then their respective Likelihoods. Let the set of al1

measurements recorded at time k be designated as

E {~,(k),~(k),-..,z.~(k)) (3-7 1)

For a measurement to be considered for association with one of T established targets, it must fall within

a specified distance of the target's predicted location, &k(k - 1). The maximum pemissible distance is

set by a gate G,. according to a chosen confidence criterion. A validation matrix can then be established

to record the possible assignments. Forj = 1 .. Mmeasurements ernanating fiom t = 1 ., Ttargets, the

validation matrix is defined by

~ , ( k ) = [ ~ , ( k ) ] c ~ ~ , j= 1 ,..., M t = l , . - - , ~

w, in Equation 3-72 is a scalar fùnction indicating simply whether measurement j

(3-73)

falls within track gate t.

Once validation (thresholding) bas been penormed, an 'assignment' matrk is made by repiacing the

binary decision with a distance hction that indicaîes how clooely the latest measurements and predictions

coincide. The NIS (Section 2.2.1) provides a convenient measure because it is a n o d i e c i quantity.

Using Equatioa 2-19 to calculaîe the NIS for each potentid measummentl track pair re!suits in the

following ma&:

where j t denotes the aforementioned pairing. As an example. consider the scenario Figure 3-8 [13].

O = validation gate A = measurernent X = propagated track E = normalized distance

Figure 3-8 Samp Ie Nearest Neighbour Data Association Scenario [ 131

According to the preceding development, the validation and assignment matrices are (Tables 3-2 and 3-3):

Table 3-2 Validation Matrix for Hypothetical Tracking Scenario 1131

1 - denotes an inadmissible assignrnent 1

Table 3-3 Assignment Matrix for Hypothetical Tracking Scenario [I3]

Once the association matrix is formed, the measurement-to-track assignment is performed by finding the

pairs (k) such that their cumulative distance x $ ( k ) is minimized. The standard solution r

assumes that a decision is made after every update. The NN is constrained to maximize the total number

of associations to reduce the candidate solution set. In the simple case, the algorithm would fom the

following pairs: (X, , A,), (X, , A ,). Many algorithms have been developed to mechanize NN; for

instance the Munkres algorithm [27] is an appropriate realization.

3.5.2 Application of NN Data Association to Dacentralired Fusion

Should a decentralized architecture be chosen, data association would have to be performed at both the

local (sensor) level and at the global (fùsion) level. Association at the local level concerns the assignment

of measurements to tracks. This problem is straight forward in the case of the radar filter, and consists of

implementing the routine described in the previous section. In case of the imager, association is slightly

more involved because the filter is required to fonn a complete position estimate using angle

measurements from the imaging sensor and a range estimate fiom the radar. Thus, association is first

needed to match the latest imager measurement to an existing radar (range, bearing, elevation) estimate-

Only then can the 'complete' point be associated with previous imager tracks. It is important to keep in

mind that al1 initiations are perfonned by the radar tracker; thus there is no dilemma in processing

extraneous imager detections, even if they originate fiom a new target; i.e., by default, new tracks will not

be initiated if the signal source is seen only by the imager.

Data association at the global level is accomplished by applying the NN method described in Section

2.4.3.2, except that tracks are associated to tracks instead of measurements- To account for the total

uncertainty, the association could be perfomed over the intersection between the prediction gates, one

centered around the radar estirnate, the other around the imager estimate. A more convenient (but

equivalent) approach to account for the total uncertainty is to increase the size of one of the two gates in

proportion to the size of other's covariance. Using tbis strategy, it becomes possible to use the same

algorithm for assoçiating at global and local levek-

3.5.3 Application of NN M a 1i4rtociatiin to Seqmtial Fusion

One advantage of sequential fusion over decentralized fission concems the complexity of the attendant

data association. in the sequential case, it need only be perfonned at one level; Le,, measurement-to

track, and can be accomplished in either complete or incompfete dimensions, depending on whether the

latest measurement came h m the radar or the imager. In either case, the NN algorithm as previously

described would be used.

3.6 Track Initiation

The requirement for an explicit track initiation scheme depends on the attendant choice of data

association- MHT based tracking systems accomplish the task automatically, whereas RDA, PDA, and

NN do not. In general, track initialization is performed either by batch means or sequentially, Batch track

initialization schemes require a complete set of detections to be saved over time in order to make an

optimal decision; however, the present application cannot afford this Iuxury. Sequential methods, on the

other hand, allow decisions to be made according to the last few observations. Of the sequential

algorithms, the 'Logic Based Muititarget Track Initialization 1281 is particulariy suitable.

Logic based multitarget track initialization uses knowledge of the assumed target dynamics and noise

statistics. ï h e test begins by considering al1 m measurements collected at time k. Denote these by set Zi,

i = l..m. For simplicity, Zi is divided in to one of its f =I..n dimensions: Zi' . The test begins by establishing

an acceptance region (Le., a one dimension gate) around each measurement, say a bearing. The size of

the acceptance region is set according to a desired confidence level. The gate's location is then

propagated over the sample p e n d and associated with the next detections (falling within its limits). To

account for al1 possible speed profiles, the gate is increased in size according to the maximum and

minimum possible velocity components of the target along the chosen direction. The process is continued

for a set number of iterations. If a measurement falls within the gate over the prescribed number of

updates. then the previously tentative track will be initiateci.

According to the preceding description, the maximum difference between two measwements recorded at

times L;Cb and t=t,, is 1291,

d: - max[ (+) -&k - 1) -PLAT) ,O] + -[ (+) -&k - 1) -&,,AT ) ,O 1 (3-75)

where AT represents the time between the measurements, and where v,, and v,, represent the

maximum and minimum possible components of the target's velocity. Taking the variance of the

measurement noise into consideration, the normalized statistical difference between the detections is:

Qï -d,$& (k - 1) + R, (k)rld, (3-76)

Assuming the innovation process to be Gaussian (zero-mean), the detections should be associated if, for a

given confidence leveL the standard NIS test statistic is satisfied; i.e. if Du 2 y . where y is a cbi-

squared distributed threshold for a system with n degrees of fieedom. The process is swnmarized by the

following steps in [29]:

Step 1: Starting with a detection fiom scan k-1, establish a Pte using Equations 3-75. For every

measurement from scan k falling within the gate, set up a potential track.

Step 2: For every potential track (consisting of two measurements), perform a straight line extrapolation

to the third sampling instant- The size of the acceptance region for the third scan is determined by the

variance for the estimate in the given dimension (calculated using Equation 3-76)

Step 3: Drop potential tracks whose gates do not see a detection at the third scan. Spilt the track if

more that one measurement is recorded.

Step 4: Continue validation process for desired number of scans. Confkm remaining tracks and

commence tracking in full dimension.

Chapter 4 lnfrared Image Processing

Up to this point is bas been assumed that the LAV Recce Surveillance Suite (LRSS) provides angular

measurernents of target position, characterked by knom noise statistics; however, this is not the case-

The information generated by the Westinghouse MicroFLJR is an analogue representation of the entire

surveillance volume (targets plus background). Consequently, a method must be devised to extract the

required kinematic information. The appropriate mapphg may be devebped given the Iocation of the

target's centre of mass in the image plane, as well as the sensor's horizontal and vertical fields of view

(FOV), and the size of the image screen in pixels. The compIete procedure involves (i) estimating the

target's mean position (centroid) and variance; anâ (ii) transforming this pixel position to an equivalent

angle representation. These requirements will now be detailed, beginning with the second task-

4.1 Mapping Between Pixel and Angk Coordinat8 syriems

Let the target's image location be specified by the followhg parameters (Figure 4-1) 1241:

('w % 1 Vertical and horizontal FOV [radians];

WC, A<pC ) Instantaneous azimuth / elevation of an arbitrary point (i-e., the target centroid)

measwed wxt. the center of the image thme [radians];

{MM, NN} Total vertical and horizontal dimension of image screen Cpixels];

{pi 3 pj) Vertical and horizontal screen position measured w.r.t, to top lefi haad corner,

i = I . ... . MM, j j= 1. ... . NN Ipixels];

{ci , cjI Vertical and horizontal screen position measured w.r.t- screen centre [pixels];

{u,. u d Vertical and horizontal angular extent of each puel [radian / pixel].

j [pixels] --*

n [pixels]

(m. n) : (1.1)

Figure 4-1 Target Location in the Image Frame

The target centroid is typically repocted as the point {pi , pi}, although it is more convenient to work Mth

{ci , ci}. It is possible to map between these coordinates via the following one-to-one transfomi:

c, = 'Iy - pi verticaL location

c, = p, -y horizontal location

nie angular quantities {AW, Aqf may then be determined directly h m {ci , cjJ via

(4-1 ab)

where the pair {u, , h) is detemhed by dividing the total FOV by NN and MM respectively. In this case

a FOV is 200 x 150 milliradians is represented on a 640 x 480 pixel plane. Detenniaing the instmtaneous

field of view (IFOV) in this manner neglects lens aberration; which is a ceasonable assumption provideci

the lens's focal Iength,f, is large compared to the horizontal and vertical dimensions of the colfecting

element. Typically this is the case for a scanning device like the MicmFLIR Note that in this treatment

sub-pixel accuracy is not essential and for that reason the size of the FLIR's blur spot [30] is not

considered (each pixel spans a vertical and horizontal field of view of 0.3 125 radians in each direction),

IC = range of tmget csnlroid & = bang (LOS) BC = bearing (cmtroid) cpr = efevsaiarr (LOS) Qic = devation (csritroid) A = local quantity

w.rL COS

Figure 4-2 Relationship Between Local and Global Coordioate Frames (24 ]

Once the target's angular location has k e n detemiined with respect to the h e center. an equivalent

position. {Oc. qP 1, can be established in ternis of an arbitrary coordinate frame via the augmented rotation

mapping as outlined in Amex B [3 11; i-e.,

rotation

{A@., ~ f ) mappiw

In practice, this operation is required if the global fusion space does not coincide with the sensor's

measurement space (Figure 4-2); specifically, if the sensor's LOS (8, , cp, ) with respect to the arbitrary

h e is non-zero.

4.2 Extracting the Twget Centmid fmm a Noisy Image

Section 4.1 establishes a mapping to relate target image position {pi , pj j to the equivalent angular

representation {BC, qf}. This section is concerned with constructing (pi . pi) in the fmt place. in general.

the problem may be attacked in one of two ways: by cousidering the evolution of the image over tirne

(multiframe image analysis), or by considering the attributes of a single captured image (single thme

image analysis). In both cases, the images are pcovideâ by an analogue to digital converter (i-e-,

framegrabber). Note that the FLIR is a scanning device, and as such the entire search volume will be

scanned in sections by a rosette onto either a singie detector element, or an anay of elements. The h U

search volume is covered at a sufficiently high rate tbat, for practical purposes, a digital rendition of the

entire image over one scan can be assumed to be produced instantaneously. Because the image capture

rate (proposed at 10 thmes per second) is slow compared to the refkesh rate (in the order of 100 fiames

per second), the Nyquist sampling criterion will be satisfied.

Multi-frame image analysis involves either subtracting consecutive fiames tiom each other or correlating

them [32] to reveal movement. The technique is versatile in that it applies equally well to both visual and

infrared imagery. Could the camera be rendered completely motionless, this would be an attractive

strategy - provided that the target were the only moving object. However, in an operational environment

the wind has a tendency to cause the sensor platform to vibrate, creating the illusion of movement in the

search volume; fucthermore, trees and non-stationary objects in the background will appear as possible

targets if their motion is detected.

Conversely, with single fiame image analysis the object is to isolate the target based on the characteristics

of a single image. in the infrared case, an obvious strategy is to exploit the difference between the target

and the background on the basis of their electromagnetic radiation profiles. Specifically, assume that the

captured fiame is recorded as a 256 level grayscale image. The value (level) of each picture element

(pixel) of the MM x NN pixel matrix will be proportional to the irradiance (H) [33] (fkom target and/ or

background) seen by the detector element corresponding to the given pixel.

Based on the preceding discussion and on the precedent set in the literanire, the single h m e method

[22],[25], [34] is chosen for use in the AILRSS.

4.3 lnterpreting the Anaîogue Vidco Signal

Single frame image analysis involves separating the target h m the background by exploiting the

difference in their radiation profiles. To achieve a desired probability of detection (PD) and probability of

false alarrn (P,J, a minimum SNR must be achieved. For an inf?ared imager, the SNR is given by 1331:

where Signal voltage change going h m background to target:

S ystem noise (accounts for Johnson, shot, flicker and signal noise);

Constant terni (accounts for FOV, area of collecter, area of detector, and system

fiequency bandwidth);

h w e r and upper Iimits of infiared bandpass (Westinghouse MicroFLïR operates

between 8 - 14 pm);

Target radiance (power emitted by a unit area of source into a soiid angle);

Background radiance;

Atmospheric transmittance, T, < 1 (accounts for signal loss due to atmospheric

scattering and absorption);

Optical transmittance, to < 1 ( accounts for signal loss due to passage of signal

through optical elements); and where,

'Dee-stai (a normalized measure of a sensor's detectivity as a function of

wavelength).

The terms in Equation 4-4 cm be divided conveniently between those which depend solely on the system,

and those which depend solely on the operating conditions. The latter group (r, N,,, NLB) accounts for

the effects of weather and targetl background radiation profiles, which in turn are determined by the

range of observation, and the difference between target and background temperatures. This classification

is use hl, as it establishes a rapid and simple means of calculating the system's performance envelope for

a required Pm and PD.

A more complete analysis of systern perfonnance is not possible within the context of this project, as most

of the imager's technical specifications are propriety information. However, should the proposed design

be implemented, it would be valuable to establish the Receiver Operating Characteristic (ROC) curves,

which provide an explicit relationship between Pm, PD and SNR.

4.4 Single Frame Estimation [34): m i n i n g Cenüoid üœn and Variance

For the purposes of this investigation, it is assumed that the analogue signal is sampled at a constant rate

( 10 frames pre second) to produce a sequence of 256 level grayscale images. Single fiame estimation

invoIves (i) applying an intensity bandpass to the original sampled images; (ii) quantizing the remahhg

non-zero pixels in each image , aad (iii) clustering the resultant sparse distribution o f binary points into

meaningfb t groups (targets) to eLiminate unlikely candidate clusters.

4.4.1 Background

Because the target is assumed to be an operating vehicle, it's radiant emittance (W) (Wcm-') should be

greater than the that of the background. This information is represented by the value of each pixel in the

MM x NN image fiame in terms of a 'grayscale intensity,' 11, which is proportional to the inadiance (H)

of the source in each pixel's fieid-of-view. In the case where the background inrdiance is negligible, 1, is

given by

where Si is a random variable representing the grayscale inteasity of a pixel within the target extent, T.

and where i and j represent the location of an arbiaary point in vertical and horizontal pixels . The

location of the target's centroid (with respect to the top lefi hand corner of the image h e is given by

the point ( p i pj), where

Mhf NN

CClv(k)j (4-6 ab)

i-i j-1 i-1 j-1

Dropping the dependence on k for notational convenience, and without making any assumptions

conceming the distribution of S,, the variance of the target centmid specitied by Equation 4-6 is given by

where p (4) is the mean value of pixel (i). and w h r e CI$ is its variance-

4.4.2 Application of a Bandpass Riter

In this thesis it is desired to characîerize every pixel as either belonghg to the target (Le., i&T) or aot

belonging to the target (Le., i JET). This binary ciassification (quantizing) is accomplished by applying a

bandpass fiiter to the originai image. As a result, the vaiue of each pixel is given by

where 1, and 1, specifjr lower and upper threshold intensities, which are chosen to guarantee a minimum

probability of detecting the target, p, (Figure 4-4). For instance, assuming the target pixeb are Gaussian

random variables with means as pi and variances di establishes the probability of detecting a pixel as

[34]:

Note that the Gaussian assumption is not necessary but is used hem for illustrative putposes.

Figure 4-3 Application of a Bandpass Filter

-48-

The pmbability of detecting a pixel in the bandpass is chus

where the symbol n: has been chosen to represent probabiiity in lieu ofp to avoid coafiising the symbol

with pixel position, p,. Using Equation 4-10, the overall variance of the centmid location calculated witb

respect to (pi, pi) is given by

while the centroid will be located at

where xij(l-ir,) is the variance of an arbitraxy pixel in the target extent. Although the target pixels will not

be independent and identicaliy distributed (iid), in practice they may be treated as such, Under this

assumption, it follows that for mean p(aj) = z and variance &(pu) = n( 1 -IL),

where N, is the total number of pixels in the target extent and where i, and j, represent the location of a

target pixel in vertical and horizontal directions.

Up to this point, the effect of the background has not been taken into consideration. To accommodate

background, the grayscale intensity of each pixel can be described by

PIXEL INTENSITY P ij = O

Figure 4 4 Application of a Bandpass Intensity Fiiter Given Non-Zero Background

where T represents the target extent (composed of N, pixels) and where B represents the background

extent (composed of Nv pixets). It is assumed that the background pixel intensity distribution will lx

spread out over the bandpass (Figure 4-4). As a result, the intensity filter will not be abie to isolate the

target without also detecting background pixels (Le., noise). Thus, the choice of threshold must take into

account not only the desired probability of detectioa h, but aiso the acceptable probability of false alarm,

p,. These tenns may be computed after identwng the rcquirrd moments of the target and background

distributions (fiom a sample image), and integrating theû respective pmbability density fùnctious between

1, and 1,. The presence o f the background pixels will increase the variance of the centroid estimate.

Assuming both target and background pixels are i.i.d. gives

where q and x,, are the probabilities of detecting target and background pixels respectively, and where i,

and i, are the i and j coordinates of pixels in T and N.

A M e r improvement in the centroid estimation process is achieved by grouping the remaining non-zero

pixels into independent 'clusters' on the basis of proximity. By doing îhis, it becomes possible to eliminate

fiom consideration spatial distributions which are too sparse to represent targets. The clusters are

'grown', starting with an arbitrary pixel, and by linking any candidate which falls within a given proximity

distance, d, to the cluster. This sosalled single linkage algorithm [35] is appropriate as it may be

performed in real-the. As shown in 1321, the proxirnity distance should be chosen such that it falls within

the average distance between neighboring pixels in the target (d ,) and the in background (d .). These

quantities are approximately given by

and

where p, andp,. represent the probability of detecting a target or background pixel, as discussed in the

previous section. Bar Shalom [25] determined by simulation that the 'optimal' proximity distance, dP9,

corresponds to the mean of this interval; i.e.,

Clustering with dpo links most of the target pixels while isolating most background pixels. If the

approximate size of the target (in pixels) can be anticipated, then clusters that are 'too small' may be

eliminated, while the centroid location and variance of the remaining clusters can be calculated by

Equations 4-6 and 4-15.

Chapter 5 Simulation of Design Alternatives

The candidate architectures d i s c d in Chapter 3 were simulated by Monte Carlo trials in order to

establish the most advantageous configuration. The figure of ment throughout is the mot meaa square

error (RMSE). Where appropriate, the expected error (EE) and the measurement e m r (ME) are also

plotted. The EE for a given element of the state vector (i.e., position in the 'x* direction) corresponds to

the square root of the filter's covariance matrix for the appropriate element. The overall expected

position error at each point in time, k, is calculated as

E E , i k ) = Jq,<k> + pz(&) + P,(k) (5-1)

where Pii(k) corresponds the EE (squared) of the ith element of the state vector X(k), i = 1 ..6, given that

X ( k ) = [ d k ) y(k) r (k) i ( k ) Q(k) i (k ) IT (5-2)

The RMS Error is calculated in similar fashion:

R M S E ( ~ ~ ( M ~ ) ) = ~ M S E ( ~ ( ~ ~ I ~ ) ) + MSE(~(&I~))+ MSE(~(~I~)) (5-3)

The measurement e m r corresponds to the standad deviation of the radar measurements. It is included

to indicate the besr performance of the current system, which does not have any estimation capability.

5.1 Scenario

The simulation scenario is based directly on a surveillance recording made by the U V Recce. The

target travels along a straight road between two positions at a velocity of 15.8 meters per second. In

Cartesian (X-Y-Z) coordinates, the start comsponds to (663, 1278,431 meters and the finish to (677.

1 3 30, -20) meters, both measured with respect to the observer location. The radar and FLiR are

coIIocated with overlapping arcs. Figures 5-l(a) and 5-l(b) illustrate the course profile as it appears to

each sensor. The imager reports the centroid position in pixel coordinates, assuming a measurement noise

standard deviation of 10 pixels both in i and j directions. The FOV subtends horizontal and vertical arcs of

200 x 150 milliradians, which corresponds to 640 x 480 pixels in image fiame. Thus, u, and y,, both equal

3.125 x 10J radians per pixel. The radar reports range, be-g and elevation assuming standard

deviations of 24 meters, 0.01047 radians and 0.01047 radians respectively. Al1 noise statistics are realistic,

but do not necessarily correspond to the exact specifications of the LRSS. In order to measure the best

possible performance of each estimation scheme, no state noise is included in each scenario, the initial

conditions correspond to the true state values. With respect to timing, two cases are considered: In Case

1, both the imager and radar provide updates every 0.1 seconds. In Case 2, the imager reports every 0.1

seconds, while the radar updates every 1.0 seconds. The simufation is dormes i over 10 seconds

(simulated time). which corresponds to the maximum period during wbich the target is expected to remain

visible in an operational environment.

Target profile in image piane a , q ,

Figure 5-1 Simulated Course Profile for (a) imager and (b) Radar

5.2 Candidate SF-1: Sequential Furion (Case 1 and Case 2)

Candidate SF-1 corresponds to the sequential filter detailed in Section 3.1.

Case 1: Case 1 establishes the e d y standard for fiiter perfoxmance (Figure 5-2). The RMS error for

total position reaches a minimum value of 5 [ml, a six fold improvement compareci to the present system

(indicated by the radar's ME), ï h e benefit of including the imager angles is dernonstrateci in a separate

plot (Figure 5-3) i l ~ u s t r a ~ g filter performance in X and Y directions separately. The (.) curves indicate

botb the RMS error and the EE for the fUsed estimate in the specified direction. At first giance, it appears

that two separate pIots have been overlain on each axis; however, this is not the case. Because of the

common update schedule, two a posteriori estimates are produced at every sample tirne- The higher of

the two values is calculateci upon receipt of the range observation, which is pmcessed first by defauit.

The estimate is then impmved by pmcessing the imager angle observations 'immediately' thereafter-

- RMS Emr (Fused) 30 - -

Figure 5-2 Sequential Fiiter Performance (Total Position) Case 1 : 50 Runs, Radar T = 0.1 Cs], Imager T = 0.1 [s]

SequentiJ Fnter Performance in Y direction ( 50 Runs, raâT = 0.1 [sl, ImT = 0.1 Es] ) 25

Sequential Filter Performance in X direaion ( 50 Runs. r a d l = 0-1 Isl )

. ExpeaedEnor

*- RMS Emr i , - .

O O 1 2 3 4 5 6 7 8 9 1 O

l ime [s]

Figure 5-3 Sequential Filter Performance (Y and X Position) Case 1 : 50 Ruas. Radar T = 0.1 [SI, Imager T = 0.1 [s]

Processing the imager angles significantly improves the estimate in the early stages of the Y profile. while

the effect in the X direction is less pmnouaced. This phenornenon relates to the course's profile in

spherical coordinates; specificaily, beariag is the predominant factor in the Y direction. whiie range

predominates in the X direction. Thus, the hision offers a potential improvement in estimate accuracy that

is directly related to the orientation of the sensors witb rrspct to the target

Case 2: Decreasing the radar's update rate has a significant impact on filter performance (Figure 54).

Because a range observation is only available every ten simples, the total RMS position emr increases

periodically between radar updates, despite the imager's angular qom. in the extreme case . (at about

2 [s]), the estimates are outpedomed by the radar observations- While the RMSE eventuaily faiis below

15 [ml, the convergence is slow. Thus, although the sequential architecture is suited to the asynchronous

timing problem in theory, in practice it only perfoms well if the dimensionaiiy sufficient observations are

provided at a relatively hi& rate- With this point in mind, the performance of Candidates DF-1 and DF-3

are now discussed.

O O 1 2 3 4 5 6 7 8 9 1 O

Tirne [s]

Figure 5-4 Sequential Filter Performance (Total Position) Case 2: 50 Ruris, RadarT = 1.0 Cs], imager T = 0.1 Cs]

5.3 Candidate DF-1 : Static Decentralized Fusion.

Candidate DF- 1 corresponds to the filter developed in Section 3.2.1. It is the least computationally

demanding of the t h e architectures under consideration; however, it assumes that the target dynamics

will be linear in the image plane, whic h is not necessarily tnre. Only Case 1 is simuiated-

Case 1: The first point of interpst is the performance of the image filter (Figure 5-5). The 'y' axes of

the graphs are! lefi in 'pixel' coordinates, but the conversion to bcsring and eievation h m pixels may be

accomplished via Equatioa 4-2. hitially, the filter converges quickly. However, at the 3 [s] mark, the i

estimate diverges, foUowed by the j esthate at about 7 [s]. These points correspond mughly to the time

at which the target is crossing the imager's optical a i s , resulthg in a large bearing rate and non-Linear

image plane dyuamics. At this point the gain bas also dropped and the estimates cannot be reacquW

Even though the overail fused performance (Figure 5-6) is saIl quite reaso~ble. Candidate DF-1 is

eliminated fiom further consideration based on its unstable performance. in the last graph. the radar's

RMSE and EE are plotied to provide a baseline for judging the effectiveness of the fusion. Note thaî SF-

1 (Case 1) provided ody marpinalIy better results.

Imager Eladon Esümaüon ( 25 Rum. T4.l [SI ) 15, 4

!

Figure 5-5 Static Decentralized Filtet Performance (i and j Image Position) Case 1: 50 Runs, RadarT = 0.1 [s], Imager T = 0.1 [s]

Figure 5-6 Static Decentrdized Filter Pefiormance (Total Position) Case 1: 50 Rus, RadarT=0.1 [s], ImagerT = 0.1 [s]

5.4 Candidate DF-3: Dynamic Decei,tmlized Fusion

Decentraiized fbsion Candidate DF-3 was simuiated for both Cases 1 and 2.

Case 1: In Figure 5-7, DF-3's fused position performance ( - - ) is compared to the radar's EE (-) and

overall (fused) RMSE ( - ). The filter reaches a lower RMSE than SF-1 (about 3.5 Cm] ). This effect

can be attributed the fact h t the imaging sensor employs an aposteriori' range estimate in its input (i.e..

to fonn a dimensionally suficient rneasurement). Candidate DF-3 is now tested in Case 2.

Decentralized Fiiter Perfomance in Position ( 25 Runs. radar T = 0.1 [ç]. imager T = 0.1 [s] )

O 2 4 6 8 1 O Tirne [s]

I I I r

Figure 5-7 Dynamic Decentralized Filter Performance (Total Position) Case 1 : 25 Runs, Radar T = 0.1 [s] , Imager T = 0.1 [s]

-- Measurement Error (Radar) - - - - - - --- - --- - - - E>cpected Error (Radar)

- - - Wected Error (Imager) RMS Error [Fused)

Case 2: A decision for DF-3 over SF-1 c m o t be made on the bais of their results in the Case 1

scenax-io, due to the fact that in practice the radar will not provide updates at 0.1 Cs]. In fact, assuming

that the system operates in Surveillance mode, a simcant period of t h e could elapse between reports (

>l second ). Hence, judging on the basis of Case 2 is more appropriate. Under realistic conditions

Candidate DF-3 performed better than SF-1 (Figure 5-8). Although the fûsd RMSE decays to the sarne

value as in the sequential case, the decentralized filter outperforms a radar tracker ( - - ) by a

considerable margin at al1 times. Thus, based on its performance in Case 1 and Case 2 scenarios, the

decentralized dynamic filter is the best of the three architectures.

-

\ - '-4 3

* \

. \! -

?, \\ 4

'\' ,

-

5z --?&?Y =: - - __;- . - _ -- - .- - - ---

Figure 5-8 Dynamic Decentralized Filter Performance (Total Position) Case 2: 25 Runs, Radar T = 1.0 [s], Imager T = 0.1 [s]

A slight modification to the existing DF-3 algorithm will result in even better performance. In the previous

case (Figure 5-8), only the latest a posteriori range estimate is used at every imager sample instant.

without accounting for the motion of îhe target between radar scaas. However, if the a prion state h m

the radar is propagated iricrementally at the imager's sample rate, then the imager can provide more

accurate updates between radar detections (Figure 5-9). Successful employment of the modified

technique requires accurate image filter hitialization, and because the W s a prion covariance is

initially hi& the imager filter's covariance must k is increased at the begmning of the hision process to

prevent early divergence. This ability to accourit for the motion of the target distinguishes the

decenhalized architecture from the sequentiai architecture in a fiindamental way. in the latter case,

although the new angular measurements are available whenever the FLIR reports, the filter is unable to

propagate the target range before the aext radar detection, and consequently c m not profit from

knowledge of the target dynamics to the same extent-

Decentralized Fiîter Performance ( Radar T = 1 [s] Imager T = 0.1 [s] )

Figure 5-9 Dynamic Filter with Range Propagation (Totai Position) Case 2: 25 Runs, Radar T = 1 .O [s], imager T = 0.1 Cs]

80 - 1 1 1 1

Y---.,

On the basis of the preceding discussion. Architecture DF-3 (with propagation of radar range esthate) is

selected for use in the A\LRSS. Simulation has shown the dynamic decentralized technique to be more

accurate than the sequential füter in both ideal and practical timing senarios. Funherrnore. the technique

is inherently more robust as it incorporates a pair of fdters; thus, if the imager provides degraded

measurements, the radar tracks will remain unaffected.

70 t- i "-

60 n

€ u L

50 W c O :s 40 a Q)

g < 20

10

O

/ . - I - ..

, 4

t .- * .\ -

- l 1. - -. 1 -. .

/ -*. . -- - -- - -

I j

3 3 o r - T - - --- - - - _ _ - - - - - - - ,-;----- --- - *- - - - - -- ;y :>-Y - - -c _ -- - - - - - - -

t 1 I t L

O 2 4 6 8 10 Time [s]

Chapter 6 Validation with Real Data

In Chapter 5. the proposed algorihm were tested via Monte Car10 simulation, resulting in a choice for

Candidate DF-3 . However, the ultimate success of a prototype is determined by its peflormance under

actual operating conditions. For this reason, live trials were staged to collect realistic radar and infiared

surveillance data. Validation consists oÇ(i) processing the raw FLIR irnagery to identify the target

centroid fiom scan to scan and (ii) combining this idonnation with capnued radar data in o d e r to

measure the performance of the üacker over a single run. While ninning the test scenarïo was identified

as a secondary goal (outside the scope of this thesis), g d prelhhary d t s have been achieved with

regards to the image pmcessing. Tbese results are presented now as a confimation of the techniques

discussed in Chapter 4. The second stage of the validation process (see (ii)) is not yet cornplete;

however, it is still appropriate to discuss the testing procedure in terms of the performance measures to be

used. The frnal results will follow later in the year as a technical report released by the author to Defence

Research Establishment Valcartier (DREV).

6.1 Measures of Performance

Various Measures o f Performance (MOP) are available to assesses the effectiveness of a tracking

system. In the present case, we are interested in MOPs that describe behavioral functions [36].

Specifically, we wish to detennine how well the proposed system tracks a target in position and, when

more than one target is present, how well it handles the &ta association problern.

Estirnated position accuracy can be detennined in two ways: by comparing the estimated tmck to the

ground tmth or by forming the NIS (Section 2.1.1). The NIS is usefiil as it relates the measurement

residuals to the total expected e m r covariance, establishing for a desired confidence Level whether or not

a the filter is behaving consistently. The NIS is a coarse test because it assumes the rnie state is

unknown. Given the ground truth, the Nonnalized Estimation E m r Squared (NEES) can be used as a

more precise test- The NEES compares the difference between hue and estimated States by nonnalizing

the estimation error z ( k ) by the estimate e m r covariance over N, nias. Fonnally, the NEES is detined

w here

There is an obvious similarity between the NEES and the MSE used in Monte Carlo simulations.

Because the NEES is chi square distributed, it can be used not only to calculate the estimation error, but

also to determine whether the e m r is consistent with the system. Both the NIS and the NEES are

appropriate MOPs for the validation pmcess-

6.2 Setup and Procedure

The triais, conducted in a nual cornmunity near Ottawa, Oat., consist of singie and multi-target scenarios-

Both were run on a pre-set route (Figure 6-1). The target(s) included a Dodge Dakota and a

Volkswagon Golf, whose radiation profiles are similar to the Forces' LSVW (truck) and Iltis (jeep). The

targets traveled at constant çpeeds between pre-marked 'way points' (WP) along a relatively straight side-

road. They were observed by a LAV Recce positioned roughly a kilometer away. The sensors' Iines of

sight (LOS) were CO-aligne& and theu arcs chosen such that the radar's search volume enveloped the

imager's. The ground m t h was established by recording the grid of each WP using a GPS with an e m r

radius of 10 meters. This information was used initially to align the radar and inffaced sensors. It is also

used by the NEES. The following section outlines the procedure used to capture the radar and infrared

data.

6.3 Infrared Data Proœssing

The infrared imagery was originally recorded in Hi-8 video format. Later, the d o g u e data was

sampled at 15 frames per second and written to compact disk as a series of 640 x 480 pixel bitmaps. The

bitmaps were impoited into MatlabTM for segmentation, quantization and clustering. The proposed

AlLRSS would include a digital image processing facility and would perfom these operations on-line.

During the test, the CRT monitor contrast and brightness were set visualiy to the highest SNR without

distortion. Because the theoretical target and background radiances were unknown, the required

probability distribution fùnctions (for target and background) were be obtained offline by sampling regions

of interest within a test image (Figure 6-2).

Figure 6-1

Figure 6-2

INITIAL INFFMRED IMAGE

256 Level Bitrnap

i&ared Video Image Captwed by LAV Recce

-64-

in Figure 6-2 (captured dwing a trial run) two targets are heading towards each other in the vicinity of the

cross hatch. Apart h m the target statistics, it would be usefil to establish the distribution of the

background in t ems of the sky (top horizontal band), the tree line (middle band) and the foreground

(lowest band). A histogram (Figure 6-3) was constnicted by sarnpling the image in the appropriate area

with a rectaogular window. The target has a sample mean of 104 units and sample variance of 5 uni&

squared. To extract the target location it was necessary to generate a set of thresholds for a given PT and

P,. These quantities were determined by integrating the regions beneath the appropriate target and

background probability distributions for various IL and 1, (In this case the distributions were treated as

Gaussian).

INTENSITY DISTRIBUTION Sky. Treeline. Ground

45 ( I 1 1 1 1 I 1 1

Figure 6-3

TREES mean: 82

GROUND var:7 mean: 70

SKY maan: var: 3

70 80 90 100 110 120 130 Gray Scale Intensity

Histogram of Typical Pixel Intensities

Figure 6 4 illustrates the effect of an overly wide passband 1, E (80-120), resulting in the deteciion of a

large portion of the me-line and sky. AAer tightening the bandpass to Iu ~(92-115) - corresponding to PT

> 0.99, P, < 0.05 - the target centroid was identified more clearly; however, a sparse distribution of

background pixels remained (Figure 6-5).

50

1 O0

150

200

250

300

350

400

450

Figure 6-4

AFTER BANDPASS IH=80 IH=120

Intensity Bandpass with Overly Wide Threshold

AFTER BANDPASS lH=92 l,,=115

100 200 300 400 500 600

Figure 6-5 Intensity Bandpass for PT > 0.95 0.05

To eliminate detections that were W e l y to originate h m the target, the remaining pixels in the search

area were clustered with a cluster distance of 2.8 pixels (Figure 6-6)- As expected, the largest cluster

corresponded to the target vehicle (containitg 13 pixels).

(300. 325)

Figure 4-6

CLUSTERS IN SEARCH WINDOW (200. 425)

Clustered Pixels

Eiiminating al1 clusters with a membership of less than 8 pixels multed in a complete extraction of the

target centroid (Figure 6-7).

ISOLATED CENTROID (pi = 424. pj = 368) $ùels] <=> (theta = 15, phi = 6) [ mrad]

100 200 300 400 500 600 j [pixels]

Figure 6-7 [solated Centroid

6.4 Radar Data

Recording the radar information for each scenario represented a significant hurdle because, like the FLIR

data, it is analog. A two pronged approach was devised to produce the required strings of discrete time-

tagged measurements: First the target's changing location was recorded by an on-board facility. which

cannot be described in any detail. Although in Swveillance mode this information is accurate to within

roughly one tenth of a beamwidth, it lacks temporal significance. To detexmine the tirne of each update, it

was assurned that each detection was made when the antenna's LOS coincided with the target's bearing.

This allowed the temporal association to be made by simply recotding the antenna's orientation with

respect to time - accomplished by fixing a variable resistor to the rotating console, and connecting it in

series to a dc voltage source (Figure 6-8). A strip chart recorder was then c o ~ e c t e d across the

potentiometer to measure the change in voltage as the radar swung through its arcs. Because the anteana

rotates at a fixed rate, the relationship between bearing, f3 , and voltage, V , is linear. Thus, the required

relationship can be detemineci h m the dope (dependence on tirne ornittecl):

Figure 6-8 Radar Senip

Chapter 7 Summary

7.1 Objectives Met

The objective of this thesis was to devdop an eficient, r o b t and Mective targer tracking

architecture that corn bines active in formation fiom a Moving Target Indicator m) radar and

passive infomation&m a Fonvard Looking Infiared (FUR) imager.

Meeting the primary objective entailed (i) evaluating various h i o u architectures; (ii) selecting an

appropriate tracking fiIter, (iiï) choosing maneuver detectiod compensation and track initiation schemes

(iv) developing a method to extract the target's angular position h m analog infiared images; and (v)

evaluating the performance of the candidate algorithms via Monte Carlo simulation.

7.2 Results and Design Choices

Several candidate architectures were initially considered. Of these, three were developed as viable

approaches: A Sequeutid Filter (SF-1) showed initial promise because of its ability to acco1111110date

dimensionally insufficierit measurernent models and process observations as they arrive. A static

decentralized architecture (DF-1) was also considered which employed a Linear image plane filter to

process the optical data. Although it was efficient, the technique was ultimately flawed because a

constant velocity profile in Cartesian Coordinates will not be viewed as a Iinear fimction by the FLlR.

FinalIy, a dynamic approach to decentralized h i o a was developed @F-3) which augments the

insufficient imager measurements with the latest range estimate thus forming a complete point that can be

processed locally and then fùsed at the global node. Al1 three candidates were simuiated, and on the basis

of its superior performance, the third option @F-3) was selected for use in the 'Advanced' LAV Recce

Surveillance Suite (ALRSS). This architecture is inherently more robust than the single filter approach,

and can be adapted advantageousty to realistic timing scenarios.

The Modified Baheti filter was selected as the most appropriate fiIter, which takes advantage of the linear

Cartesian state dynamics to produce significant computational swings. Maneuver compensation was

treated bnefly, and the straight forward strategy of reinitializing the local filters' covariance matrices was

deemed appropnate for this application. Finally, a sequential track initiation scheme was outlined to

complete the initial design.

This thesis is f m and foremost a practical application. In this regard, a method was required to extract

kinematic information h m aoisy FLIR imagery pnor to the fhsion process. A mapping between the

image screen and Spherical Coordinates was presented, which assumes the target centroid bas been

isolated as a mean pixel position with assoçiated variance. The isolation process involves segmentation,

quantization and clustering, as demonstrated on a typical infiared image. These techniques were

success full y demonstrated on actual infiared surveiliance data.

This thesis is limited to a simulation of the single target case; however, an extension to the multiple target

was also discussed. Nearest Neighbour Data Association is favored for the assignment process. Finally,

no design is complete without a field test. To judge the effectiveness of the proposed architecture, the

LAV Recce was tested in a realistic surveillance exercise that will eventually yield a complete set of pre-

processed data with which to validate this and future designs.

7.3 Novel Features of this Thesis

Military Application. The LAV Recce currentiy has m target tracking capability, w hatsoever. This

thesis not only presents a radar tracker, it also incorporates the FLIR to increase the accuracy of the

estimation process.

Fusion Architecture. The tracking system is based on a decentralized &ion architecture, whereas the

de facto approach to combine radar and infrared information is sequential fusion. The decentralized

approach proposed in this thesis is novel in that a common state mode1 cannot be assumed. Specifically, it

is necessary to augment the KIR updates with a prion range estimates to fonn a sufficient point pnor to

sensor-level filtering and global-level fusion. To the author's best Imowledge, this strategy bas not been

suggested in the literature for the current application.

Angle Mapping. Before proceeding with the fusion, locally measured aogular quantities need to be

mapped between coordinate frames. The transfonn is straight forward in the polar case; however, when

the local and global points are located in a sphencal volume, the relationship is not straightfonuard. Using

a geometric rotation mapping typically seen in mbotics[37], an exact transfomi [3 11 is developed (Annex

B) that improves upon a published method that is proven to be at best appcoximate [24].

7.4 Recommendation for Continued Invrrtigiaon

This thesis has attempted to elucidate the enonnous potential of the LAV Recce's Surveillance Suite,

Several other opportunities exist to empioy sensor &ta fbsion to great advantage. Adding the day camera

to the tracking loop, for instance would provide a means of identifjing targets and detecting maneuvers,

Otherwise, a pixel Ievel h i o n between this seasor and the FLR would greatly increase the operator's

ability to perfonn surveiLlance under marginal lighting and atmospheric conditions- However, perhaps the

most challenging extension involves a fiil1 assault on the practical aspects of the &îa association problem.

While the multitarget case has been discussed by many authors, attacked h m various angles, it has yet to

be championed in a simple, robust and effective manner,

Annex A Project Background

Location

LFTSC

RMC

Meaford

DREV

Ottawa

Hosu Eveat

LCol Lord

LCol

Carruthers

Maj Muir

LAV Recce

Tactical

Evaluation

J- Cniickshank

Y. de Villers

LCol

Carruthers

Collected initial documentation on LAV

Race Surveillance Suite (LRSS).

Discussed current DREV mandate to make

immediate improvements to LRSS.

Presented ideas to PM0 UV,

First hand inspection/ demonstration of LRSS

Collected sensor specifications.

Attended LRSS field test.

Solicited feedback on proposeci ideas h m

LRSS operators.

Discussed data fonnat with J, Cmickshank.

- - - --

a Presented current proposal to scientific

authorities in LAV Recce research group.

Discussed hardware requirements.

Collected short sequence of sample image

data

Arranged for more comprehensive copy of

FLR &ta to be recorded on CD for processing

at RMC-

Collection of real data (involves nrnniag a

preplanned scenario and recording data with

LAV Recce sensors-)

Annex B An Alternate Technique for Transferring Angular Measurements Between Coordinate Frames [2 11

The augmenred rotation mapping can be used whenever angular information needs to be mapped

from one coordinate fnune to another (Figure B-1). Define the local and global coordinate frames

according to Figure B-1. To map fiom the local to the global frame, follow the steps in Table B-1.

h = ronge dtoget mtroid & = M g (LOS) 8c = bearing (œntroid) ips = elmation (LOS)

A = local quantity w.r.t. LOS

Figure B-1 Relationship Between Local and Inenial Angles [24]

- -

Step

1

Description Notes

Measure target location {Agc. AqJ E 3t2.

Assign r, = 1 to form {r,, Agc , AQJ E 3t3.

Apply spherical-Cartesian transfonnation to

rom *P = f2px, *P,,, 2pzl

r, is arbitrary and does not affect

W C . A d -

Apply rotation rnap~ing If origins of Frames (1) and (2) are

not collocated, rotation mapping

1 R is replaced wit h

transfonnation mapping [37l

a) Convert ' P = { 'p, . 'p, . 'pz] to sphedcal

coordinates { r,, 8 . cp)-

rc is independent of (8 . Q}.

b) Disregard rc, and extract inertial angles (8 , Q).

Table B-1 Augmented Rotation Mapping [3 11

The augmenteci rotation mapphg bas been pmposed by the author [21] as an exact method to

accomplish the required cmrdinate fhme transformation. It is a suitable replacement for an altemate

technique that has appeared in the recent literahw, which is shown to be at best approximate.

Specifically, it is stated in [ ] that the angular location of a target with respect to an arbitrary inertial

coordinate fiame (8 , cp ) , is equal to the sensor's line-of-sight, (O,, cp,), plus the angular location of the

target (centroid) with respect to the sensor's coordinate fiame, {AOC . Acp, 1; Le., that

O= es+ A 8, @-la)

and

v= q+ A 9, (B-1 b)

where 0 denotes bearing and cp denotes elevatioo. In [ 1, Equations (B-La) and (E3-16) are employed

without restriction; however, the technique does not hold unless <p, = O , and /or Ag, = 0,

Proof

Assume Equations (B-la) and (B-1 b) can be utilized without restriction to transfocm angular quantities

fiom a local coordinate frame, Frame (2 ) , to an inertial coordinate frame, Frame { 1 ) (see Figure B- 1).

The assignment of an arbitrary range, rc, to the inertial mgles, (0 , cp) , establishes a complete point which

can be expressed in Cartesian coordinates. For convenience. let rc = 1, and denote the inertial point by

'P ' , where

Now, it should be possible to generate an equivalent inertial point, say 'P.

by transforming a locally measured point. %

with the rotation mapping presented in [ 1; Le.,

As the local sensor oaly measures angles, 2P E 93' must be fomed by assigning an arbitmy m g e to

(A@, , Acp, ) , as in the previous constniction of ' P r b m {O , cp ). Since axis x, (the x-axis of Frame

(2)) corresponds to the sensor's Lue-oGsight, ;R represents a rotation of the inertial !tame about y, by

-cp, and then about 2, by 8,. By the X-Y-Z fixed angle convention 1281,

Thus, provided Equations @la) and (B-lb) hold without restriction, one would expect

'p., lp W)

However, a close examination of Equation (B-4) reveals that the equality does not hold true in general.

Expanding the x, y, and z components of 'P* and 'P with the aid of standard trigonometric identities yields

a set of three equalities. Starting with the z components. it is cequired that ' p ;= ' p z . where

1 P. r = COS(Q? a. sin(^, )FOS(A <p., )

Comparing Equations (B-sa) and (B-5b) reveals immediately that

' P * ' P

except when AOC = O and/ or cp, = O. Equating the rernaining x and y ternis in the position vectors

confinns this constraint. i

References

Y. de Villers, private conversation with D. van Huyssteen and M. Farooq, Oct., 1997.

A.M Marston, R.A. Schell and J.E. Webb, "Cperational Concept Document Coyote Information Management System," Department of Applied Military Science, Royal Military College of Canada, 3000-7 (LJTSC-l), May 1996.

J. Gagliarducci, "Interface Control Specification for the Radar Extemai Interface," Electronics and Space Corp., St. Louis, Mo., 18 May 1994.

"Equipment Description Light Annoured Vebicle (UV) Reconnaissance (Recce) Mast Mounted Surveiilance System (MMSS) and Remote Mounted Surveillance System (RMSS), C-59-867- 000/MD-00 1" Canadian Department of National Defence, Project Management Office LAV, Jan 1997.

J, Llinas and E- Waltz, Multisensor Data Fmion. Boston, MA: Artech House, 1990.

D. Hall and J. Liinas, "An Introduction to Multisensor Data Fusion," Proceedings of the IEEE, vol. 85, no.1, Jan. 1997, p.7.

Y. Bar-Shalom and X. Li, Multitarget-Multisensor Tracking: Principles and Techniques. 1995, Y .B.S. Publishing, Storrs, CT, 1995.

S.W. Sorenson, "Least-squares estimation: from Gauss to Kaiman," IEEE Specîmm, vol. 7, pp. 63-68, July, 1970.

Applied Optimal Estimation, A. Gelb, ed., M.I.T. Press. Cambridge, Massachusetts, 1974.

A.H. Jazwinski, Stochastic P rocesses and Filtering Theoty, Academic Press, New York, 1970.

J.H. Taylor, "The Cramer-Rao Estimation Error Lower Bound Computation for Deterministic Nonlinear Systems," IEEE Transactions on Automatic Control, Vol. AC-24, No. 2, April 1979.

M.R. Spiegal, Schaunt 's Outline Series: Mathematical Handbook of Formulas and Tables, McGraw-Hill Book Company, New York, 1968.

M. Farooq, A. Rouhi and S.S. Lim, "Analysis of suboptimal filtering algorithms for target tracking," Elsevier Signal Processing, vol. 30,110-2, Jan. 1993.

Y. Bar-shalom, ICC- Chang, "Tracking a Maneuvering Target Using Input Estimation Versus the Inte rac ting Multiple Mode1 Algorithm," IEEE Transactions on Aerospace and Eleclronic Systems, Vol. 2, AES-25 No.2, March, 1989.

Y. Bar-Shalom and K- Binniwaf, "Variable Dimension Filter for Maneuvering Target Tracking," IEEE Transactions on Aerospace and Electronic System, Vol. 2, AES-18 NOS, September, 1982.

S. Bnider, S. Lim, B. Johnson and M. Farooq? "A Study of Multiple Target Tracking in a Cluttered Environment," E.E. Tech Report No. 921 4, Royal Military CoUege of Canada, Kingston, Ont., August, 1992.

C. Chong., et. al., "Distributed Multitarget Multisensor Tracking," in Multitarget-Multisensor Tracking: Advanced Applications, Y- Bar Shalom, cd, Artech House, Boston, 1990.

S.S. Blackrnan and T.J. Broida, "Multiple Sensor Data Fusion in Aerospace Applications," Journal of Robotic Systemr, vol. 7(3), pp. 445-485, 1 WO.

Robert J. Fitzgerald, "Development of Practical PDA Logic for Multitarget Tracking by Microprocessor," in Multitarget-Multiseosot Tracking: Advanced Applications, Y. Bar Shalom, ed., Artech House, Boston, 1990

S. Bnider, M. Farooq, M. Bayoumi, "Robotic Heterogenous Mdti-Sensor Fusion with Spacial and Temporal Alignment," Proceeditigs of the 3(Y"onference on Decision and Control, Dec., 199 1.

S. Bruder and M. Farooq, "Sensor Fusion Techniques and a Hybnd Multi-Sensor Multi-Target Tracker," EE Tech Report no. 90/3, Royal Military College of Canada, Kingston, Ont., Aug. 1992.

H. Hashemipour, S. Roy, and A, Laub, "Decentralized Structures for Parallel Kalman Filtering," IEEE Transactions on Automatic Control, vol. 3 3, No 1, Jan., 1 988.

M. Krieg and D. Gray, "Radar and Optical Tmck Fusion Using Real Data," in Proc First Annual Australian Data Fusion Symposium, Adelaide, South Australia, 1997.

J. Romine and E. Kamen, ''Modelting and Fusion of Radar and Imaging Sensor Data for Target Tracking," Optical Engineering? vol. 35, no. 3, pp. 659-672, Mar., 1996.

E. Oron, A. Kumar, and Y- Bar-Shalom, "Precision Tracking with Segmentation for Imaging Sensors," IEEE Trans. on Aerospace and Efectronic System, July, 1 993, pp. 977-985.

V. Aidala and S. Hammel, "Utilization of Modified Polar Coordinates for Bearings-Only Tracking," IEEE Trans. Automatic Control., Vol., AC-28, No.3, pp. 283-294, 1983.

Burgeois, F. and J-C. LaSalle, " An Extension of the Munkres Algorithm for the Assignment Problem to RectanguIar Matrices," Communications of the ACM, vol. 14, Dec. 197 1, pp. 802-806.

C.B Chang and L.C. Youens, "Measurement Tmckuig for Multiple Sensor Tracking in a Dense Target Environmenf" IEEE Tram- Automatic Conaof, vol. AC-27, December. 1982, pp. 1250- 1252.

Y. Bar-Shalom and T.E. Fortnian, Tracking and Data Association, Academic Press, San Diego, 1988.

B.E.A Saleh and M.C. Teich, Fundamentds of Photonics, John Wiley and Sono, Toronto, 199 1.

[3 11 D. van Huyssteen and M. Famoq, "An altemate technique for transfemng angles between coordinate *es," Opt. Eng. (pending review).

[32] Y. Bar S halom, Multiiensor-MuItitarget Tracking, Advanced Applications, Short Course Notes, UCLA.

[33] R.D. Hudson, I ' a r e d System Engineering, John Wiley and Sons, Toronto, 1969,

[34] A. Kumar, et. ai., "Precision Tracking Based on Segmentation with Optimal Layering for ïmaging Sensors," SPIE Vol, 1954, pp. 244-255.

[3 51 M.R. Anderberg, ed., Cluster Analysis for Applications. Academic Press, 1973.

1361 D-L- Hall, Mathematical Techniques in Muftisensor Data Fusion, Artech House Inc., Boston. I W i .

[37] 1.1. Craig, Introduction to Robotics. Addison-Wesley Publishing Company, New York, 1989, Chap. 2,