Introduction to Classification Methods for Military Munitions...

1

Introduction to ClassificationMethods for Military Munitions

Response Projects

Introduction to ClassificationMethods for Military Munitions

Response Projects

Herb Nelson

2

Introduction 2

Objective of the CourseObjective of the Course

Provide a tutorial on the sensors, methods, and status of the classification of military munitions using geophysical methods

•Advanced processing of data collected with existing commercial instruments

•Promising results from emerging optimized systems

SERDP and ESTCP have supported a number of investigators over the years who have

• developed processing approaches to extract target-specific attributes from data collected by commercial geophysical sensors, and

• demonstrated advanced sensors designed with the munitions response problem in mind.

These research efforts have resulted in an impressive ability to classify the source of geophysical anomalies as “targets-of-interest” or non-hazardous items under simple conditions with the promise of expansion to a wider range of conditions as the newest sensors mature.

This course is intended as a tutorial on these classifications methods. We begin with a brief introduction to some of the terminology and concepts that will be used, introduce the basics of the two primary geophysical instruments used in munitions response, discuss the methods used for classification and illustrate them with two case studies, preview the next generation of EM sensors emerging from the research program, and conclude with a brief summary and presentation of a idealized cost model for classification.

3

Introduction 3

PresentersPresenters

● Dr. Steve Billings (Sky Research)

● Dr. Thomas Bell (SAIC)

● Dr. Dean Keiswetter (SAIC)

The success of this course is due to the hard work of our three primary presenters.

Dr. Steve Billings from Sky Research will introduce the terminology to be used and discuss the concepts of magnetics.

Dr. Tom Bell from SAIC will discuss the basics of EM sensors and later in the course present examples of the capabilities of the emerging EM sensors.

Dr. Dean Keiswetter of SAIC will discuss the methods used for classification and follow with two case studies that illustrate the performance that can be achieved using commercially-available sensors.

4

Introduction 4

The Munitions ProblemThe Munitions Problem

● There are over 3,000 sites suspected of contamination with military munitions

● They comprise 10s of millions of acres

● The current annual cleanup effort is on the order of 1% of the projected total cost

● To make real progress on this problem, we need a better approach

There are a very large number of sites in the US suspected of being contaminated with military munitions but the remediation budget each year represents only about 1% of the multi-billion dollar projected total remediation cost. This leads to remediation projects having planned completion dates late in this century. Given budget realities, the only way to accelerate this effort is to develop methods to accomplish more remediation with the available funding.

5

Introduction 5

05

101520253035404550

SiteAssessment

Survey andMapping

VegetationRemoval

Scrap MetalRemoval

UXO Removal &Disposal

Cos

t -$B

Direct CostIndirect Cost

Munitions Response Cost BreakoutMunitions Response Cost Breakout

Defense Science Board Task Force on UXO

This chart, from the 2003 report of the Defense Science Board Task Force on UXO [http://www.acq.osd.mil/dsb/reports/uxo.pdf], shows us one approach to the savings we seek. On a typical munitions clean-up project, an overwhelming fraction of the money is spent removing non-hazardous items from the site. If a method can be devised to identify these non-hazardous items and remove them with fewer safety precautions or leave them in the ground, this money could be transferred to other projects.

6

Introduction 6

ClassificationClassification

● Classification offers the chance to divide anomalies into those caused by targets‐of‐interest and those caused by other things

● Recognize that current field methods involve implicit discrimination

Mag & Flag – instrument sensitivity setting and human interpretationDigital Geophysics – threshold selection; what is a target?

● Our goal is a principled, data‐based approach to classify targets as “non‐hazardous” or “targets of interest”

As we saw in the last slide, classification (sorting the sources of geophysical anomalies into “targets-of-interest” and non-hazardous items) holds the promise of real cost savings.

Many stakeholders, however, are leery of applying classification methods at their site. It is important that they recognize that classification is being performed implicitly now. In an analog geophysical survey (often termed Mag & Flag), the operator decides on the instrument sensitivity to select and what level of response to call a “hit.” Neither of these choices can be revisited after the survey is complete. Even if digital geophysical mapping techniques are used, the data analyst makes decisions about what to call an anomaly, often on-the-fly and without defined procedures.

What we seek is a principled, data-driven approach to classification. This involves data collection and analysis methods as well development of a process in which all stakeholders can have confidence.

This Page Has Been Intentionally Left Blank

1

Terminology and ConceptsTerminology and Concepts

Stephen Billings

2

Terminology and Concepts

Acronyms and DefinitionsAcronyms and Definitions

● Munitions and Explosives of Concern (MEC)Military Munitions- Unexploded Ordnance ‐ UXO

- Discarded Military Munitions – DMM

Munitions Constituents ‐MC

● Geophysical sensors detect metal (UXO, DMM and other metallic items) and cannot directly detect munitions constituents

2

http://www.epa.gov/fedfac/pdf/MRP_Definitions_12-18-03.pdf

Munitions and Explosives of Concern (MEC). This term, which distinguishes specific categories of military Munitions that may pose unique explosives safety risks, means:(A) Unexploded Ordnance (UXO), as defined in 10 U.S.C. 2710 (e) (9);(B) Discarded military munitions (DMM), as defined in 10 U.S.C. 2710 (e) (2); or(C) Munitions constituents (e.g., TNT, RDX) present in high enough concentrations to pose an

explosive hazard.

Unexploded Ordnance (UXO). Military munitions that:(A) Have been primed, fused, armed, or otherwise prepared for action;(B) Have been fired, dropped, launched, projected, or placed in such a manner as to constitute a

hazard to operations, installations, personnel, or material; and(C) Remains unexploded whether by malfunction, design, or any other cause. (10 U.S.C.

101(e)(5)

Discarded Military Munitions (DMM). Military munitions that have been abandoned without proper disposal or removed from storage in a military magazine or other storage area for the purpose of disposal.

Munitions Constituents (MC). Any materials originating from unexploded ordnance, discarded military munitions, or other military munitions, including explosive and nonexplosive materials, and emission, degradation, or breakdown elements of such ordnance or munitions. (10 U.S.C. 2710 (e) (4))

3


Acronyms and DefinitionsAcronyms and Definitions

● Target‐of‐Interest (TOI): Military munitions, explosive fragments, fuzes, items that give a sensor response indistinguishable from military munitions

● Non‐TOI: Fragments, clutter, cultural items, etc. that are not hazardous

● Parameter Estimation: The extraction or estimation of parameters that represent some useful attributes of the buried object. Also referred to as parameter extraction and geophysical inversion.

4


Digital geophysicsDigital geophysics● Requires a geophysical sensor system (based on either

magnetometry or electromagnetic induction )● A positioning device (e.g. Global Positioning System, GPS)● A computer for digital data acquisition

Magnetometer Electromagnetic sensor

Location device(e.g. GPS)

Geophysicalsensor

Digital dataacquisition

4

The three essentials elements of a digital geophysical mapping system are illustrated on this slide. Note that, in addition to the location device, measurement of sensor orientation can prove useful.

5


Standard processing streamStandard processing stream● The standard processing stream for detection and

classification of munitions using geophysical data

1. Data Collection 2. Parameter Estimation(Target Attributes)

3. Classification

Parameters

Munitions

Non-munitionsdata

5

Schematic showing the standard process flow of a digital geophysical survey. The data are collected and captured by a data logger. After the survey is finished, the data are typically transferred to another computer where initial data processing and then parameter estimation are performed. The parameters are then used to make classification decisions.

6


Collect DataCollect Data

6

An example of this process flow taken from the seafood industry. The task is to automatically sort fish as either salmon or sea bass. The first step is to collect data. In this case, the sensor of choice is a camera that captures a digital photo of the fish to be classified.

7


Extract ParametersExtract Parameters

6

Step 2 is to extract some parameters from the data. One obvious parameter is the length of the fish.

8



6a

Another parameter that might be of value is the width.

9



6b

Or the overall coloration of the fish.

10



6c

Or the number and placement of fins.

11


Classify Based on theExtracted ParametersClassify Based on theExtracted Parameters

7

Using the extracted parameters, we will attempt to classify this fish. Here we see that while salmon are, on average, shorter than sea bass there is significant overlap between the two classes so this is not the best parameter to choose for classification

12



7a

Now let’s try lightness. There is less overlap between the classes using this parameter so we are making progress.

13



7b

Finally, we see that using both lightness and width we have defined a space that allows us to reliably separate salmon from sea-bass.

14


Data and target attributesData and target attributes

● Parameter extraction methods are used to estimate the attributes of a buried object from the measured data

● The buried item is parameterized by a set of magnetic or electromagnetic target attributes that indirectly reflect the physical characteristics of the object

Buried ItemGround-surface

“Data” profilesAnomaly in geophysical datacaused by buried object

8

Schematic of magnetometer data collection. Data are collected along nominally straight pathways at a desired lane spacing. Each black dot shows the location of a magnetic measurement with the magnetic data shown in red. Positive values lie above the measurement plane, with negative values below.

15


Classification vs identificationClassification vs identification

Target of Interest Non‐Targets of Interest

Classification: Distinguish Military Munitions (the Targets of Interest or TOI) from shrapnel, range scrap, cultural debris etc (non‐TOI)Identification: Determine the type of Military Munition

Classification

Iden

tific

atio

n

9

In this context, we seek to classify items as “Targets of Interest” (UXO or DMM) or “Non-Targets of Interest” (all non-hazardous items on the site). There may be value in identifying individual sub-classes (e.g. 81-mm mortars) but that is not our primary objective.

16


Receiver Operating Characteristic (ROC) Curve

Receiver Operating Characteristic (ROC) Curve

Desired Performance100% Detection

No False Positives

Classification 100% Detection

~260 False Positives

Detection Only100% Detection

~430 False Positives

Non-Targets of interest

Pro

porti

on o

f Tar

gets

of i

nter

est

Can’tanalyze

● Used to characterize the performance of a munitions classification strategy

11

We measure the performance of a classification effort using what is known as a Receiver Operating Characteristics (ROC) curve. It is retrospective in that it requires all ground truth. This curve was taken from a study at Camp Sibert in which all contacts were dug for learning purposes.

Basically, it is a plot comparing the number of actual non-TOI objects versus TOI if our prioritized dig list were excavated in order.

Desired Performance – Ideally, we would like perfect classification. If we could achieve this, the curve would rise straight to this location: 100% TOI recovered and 0% non-TOI.

Detection Only – This is the other extreme. Here, we ignore classification altogether and simply excavate all detected targets. In the end, we have 100% of the TOI recovered (approximately 150 munitions), but we also have removed 100% of the non-TOI – which for this site was approximately 600 non-hazardous objects.

Classification – Here, we evaluate the classification results. This is the point at which the demonstrator drew the threshold between high-confidence non-TOI and everything else. As shown here, 100% of the munitions were recovered but only 200 of the non-hazardous non-TOI recovered. This is a huge success.

Can’t Analyze –This portion of the curve is reserved for those anomalies that were thrown into the Can’t Analyze category. Because no useful information regarding the nature of these targets can be extracted from the measured data, they must be treated as potential targets of interest.

17


Signal to Noise Ratio (SNR)Signal to Noise Ratio (SNR)

Smax = 38.3 mV, SNR = 153

Smax = 4.5 mV, SNR = 18

Smax = 0.75 mV, SNR = 3

EM61 (gate 3) rms noise = 0.25 mV

12

Signal to noise ratio (SNR), as the name suggests, is the ratio of the amplitude of the signal relative to the amplitude of the noise. Thus SNR can be varied by changing the characteristics of either the signal or noise. The root-mean-square (rms) noise is 0.25 mV in each of these three panels. The maximum amplitude of the signal varies from 38.3 mV in the top panel to 0.75 mV in the bottom panel. The SNR decreases with decreasing signal amplitude from 153 (signal amplitude 153 times larger than the rms noise) in the top-panel down to 3 in the bottom panel (signal amplitude 3 times larger than the rms noise). Detection of the signal with SNR = 3 would be very difficult, whereas at the higher signal levels detection is easy.

1

Magnetics:Fundamentals and Parameter

Extraction

Magnetics:Fundamentals and Parameter

ExtractionStephen Billings

Magnetics

Magnetic module outlineMagnetic module outline

Parameter extractionConcepts

Real-world examples

Magnetics fundamentalsSensor systems

Data examples and demo

ClassificationUsing the parameters to make

discrimination decisions

2

2

Magnetics

Standard processing streamStandard processing stream● The standard processing stream for detection and

classification of UXO using geophysical data

1. Data Collection 2. Parameter Estimation(Target Attributes)

3. Classification

Parameters

UXO

Non-UXOdata

3

Schematic showing the standard process flow of a digital geophysical survey. The data are collected and captured by a data logger. After the survey is finished, the data are typically transferred to another computer where initial data processing and then parameter estimation are performed. The parameters are then used to make classification decisions.

3

Magnetics

No object Ferrous ordnance

Detection of metal with a magnetometer

Detection of metal with a magnetometer

● Most ordnance contain ferrous metal

● Ferrous metal causes a distortion of the Earth’s magnetic field

4

The Earth has a magnetic field whose direction and magnitude varies across the surface of the Earth. Within most of North America, the magnetic field lies within 10 degrees of true-North and is oriented at about 65 degrees from horizontal. Magnetic “field-lines” are straight when there is no metallic object present. When a ferrous object is introduced, the field lines become distorted and are essentially attracted into the object.

4

Magnetics

Ferrous ordnance

Total-field magnetometersTotal-field magnetometers

● Measure the total magnetic field (generally in the direction of the ambient field)

● In Northern hemisphere, positive lobe to south and negative lobe to the North

Object only

North

Ambient field

+ + + - - -South

5

The distortion of the magnetic field caused by a compact object like a UXO can be approximately modeled as a magnetic dipole (essentially a bar-magnet with a north and south pole on either end). The magnetic field lines leave one end of the object (the south-pole) and wrap around and re-enter the object at the other end (the North pole). Most magnetometers in use today measure the total magnetic field. The Earth’s field is large (around 50,000 nT) compared to the distortions caused by buried metal (typically 1 to 1,000 nT).

By subtracting off the Earth’s field, we see that the total-field “anomaly” caused by a buried object is positive when the field from the object is in the same direction as the earth’s magnetic field and negative when it is in the opposite direction. In the Northern hemisphere this causes a positive lobe to the south and a negative lobe to the north of the buried item.

5

Magnetics

Data collection systemsData collection systemsSingle-sensor G858 Portable quad-sensor array

Cart-based quad-sensor array Towed-array with 8 sensors (MTADS)

6

Examples of magnetometer data collection systems. Each of the systems has one or more magnetic sensors, a positioning system (either Global Positioning System or Robotic Total Station) and a data logger for digital capture of the sensor and position data.

6

Magnetics

What are “data”What are “data”

Ordnance Item

Ground-surface

Sensor locations“Data” profiles

Measurement plane

Burial depth Survey height

7

Schematic of magnetometer data collection. Data are collected along nominally straight pathways at a desired lane spacing. Each black dot shows the location of a magnetic measurement with the magnetic data shown in red. Positive values lie above the measurement plane, with negative values below.

7

Magnetics

Data collectionData collection

8

Screen shot of the data collection animation

8

Magnetics


8a

Screen shot of the data collection animation

9

Magnetics


8b

Screen shot of the data collection animation, showing a gridded image of the data in plan view. The magnetic data are plotted using the color-scheme shown by the color-bar at right. For instance, any regions where the magnetic field is 70 nT are colored red, while regions with -10 nT are shown in blue.

The anomalous field is plotted here; the Earth’s field has been subtracted from the measurements.

10

Magnetics

Real time demonstrationReal time demonstration

Perspectiveview Profile

view

Mapview

Object attributes

Survey attributes

9

Screen-capture of the set-up used for real-time demonstration of magnetic data.

11

Magnetics

Effect of item depthEffect of item depth

20 cm

40 cm

60 cm

80 cm

9a

As the depth of the item increases, the amplitude decreases and the distance between the positive and negative peaks increases. The amplitude decreases as the third-power of distance away from the sensor. The effect of increasing the depth by 10 cm while keeping the sensors the same distance from the ground is the same as increasing the sensor distance by 10 cm without changing the burial depth.

12

Magnetics

Effect of item sizeEffect of item size

50 mm

75 mm

100 mm

125 mm

9b

As the size of the item increases so does the magnitude of the magnetic anomaly that it creates. The amplitude increase is proportional to the diameter of the object. Thus, if the object diameter is doubled, then so is the magnitude of the anomaly.

13

Magnetics

Effect of item orientationEffect of item orientation

Dip 0; Azimuth 0

Dip 65; Azimuth 0

Dip 0; Azimuth 20

Dip 0; Azimuth 40

9c

The magnitude and shape of a magnetic anomaly depends on the orientation of the buried object. The amplitude is largest when the long axis of the object is aligned along the direction of the Earth’s magnetic field and is smallest when oriented perpendicular to that direction.

14

Magnetics

Signal to noise ratioSignal to noise ratio

Depth 0.4 mNoise 0 nT

Depth 0.4Noise 2 nT



9d

Signal to noise ratio (SNR), as the name suggests, is the ratio of the amplitude of the signal relative to the amplitude of the noise. Thus SNR can be varied by changing the characteristics of either the signal or noise. For a given munition at a given depth and orientation, the SNR decreases as the noise measured by the sensor increases. The lower the SNR the lower the detection probability and the harder it becomes to extract parameters that reflect the intrinsic attributes of the buried object. At a fixed noise level, the SNR is also decreased by increasing the depth of the object.

15

Magnetics

Lane spacingLane spacing

25 cm

50 cm

75 cm

100 cm

9e

The distance between adjacent sensor paths (the lane spacing) needs to be small enough to capture the full character of an anomaly. In this example, information on the shape of the anomaly is lost as the lane-spacing is increased from 25 cm to 75 cm.

16

Magnetics

Position errorPosition error

0 cm

2 cm

4 cm

6 cm

9f

Error in the position of the sensor when it takes a magnetic measurement distorts the measured magnetic anomaly.

17

Magnetics

Data collection summaryData collection summary● Item attributes impact the shape, size and amplitude of

the anomalous magnetic field:Depth;Orientation;Size of UXO

● Sensor attributes that effect the quality of the dataSensor noiseLine spacingPositional errorSensor height above ground (and any variation)

10

18

Magnetics

Detection performanceDetection performance● Detection performance is dependent on

Object sizeNoiseData density

4.2” mortar

11

This slide shows how the amplitude (measured as the difference between the positive and negative parts of an anomaly) of a 4.2” mortar varies as the depth below the surface is increased. There is a significant difference between the least favorable (mortar at right-angles to the Earth’s magnetic field) and most favorable (mortar parallel to the Earth’s field) orientation. Also shown are the amplitudes of magnetic anomalies observed over 100 different 4.2” mortars that were seeded at Camp Sibert, Alabama. The magnetic sensor was 30 cm above the ground. An example noise floor of 10 nT is marked on the graph.

19

Magnetics

Detection performanceDetection performance● A smaller 60 mm mortar has a reduced detection depth

4.2” mortar 60 mm mortar

12

The smaller the object, the smaller the anomaly amplitude and hence the shallower the “detection depth”. This is evident in the above plot where the 60 mm anomaly amplitude intersects the notional 10 nT noise line at shallower depths than the 4.2”mortar. At more favorable orientations, the anomaly amplitude remains above the noise floor to greater depths.

20

Magnetics

Target picking processing flow

Target picking processing flow

● Magnetometer data are collected along survey lines

● Geophysicist reviews and processes the profile data“Bad data” are rejected (e.g. out‐of‐range)Filters are applied to suppress diurnal changes in the magnetic field and longer wavelength features due to geology

● Data are generally “gridded” to produce an image of the magnetic data

● Regions of anomalous response are selected as potential metallic targets

13

The above represents the standard processing flow used for digital geophysics.

21

Magnetics

Anomaly identificationTotal-field data from Montana

Anomaly identificationTotal-field data from Montana● “Raw” total‐field data from a 100 m by 100 m area at

Chevallier Ranch Montana

14

Mag

netic

fiel

d (n

T)

Northing (m)

Eastin

g (m)

This slide shows a three-dimensional perspective view of the magnetic field over a 100 m by 100 m area of the Chevallier Ranch site in Montana. Sub-surface metallic objects cause localized distortions in the measured magnetic field. More extensive, largely linear features in the magnetic data are caused by variations in the magnetite content of the underlying geology (trending roughly east-west), or by magnetite transported along drainage channels (the north-south features).

22

Magnetics



● Magnetic data after application of filters

15

Mag

netic

fiel

d (n

T)

Northing (m)

Eastin

g (m)

An appropriately tuned high-pass filter (which passes the shorter spatial scale [higher spatial frequency] target response while removing the longer spatial scale [lower spatial frequency] background interference) can be used to suppress the effect of the longer-wavelength geological features while accentuating the localized anomalies caused by buried metal.

23

Magnetics

Anomaly identificationPlan-view

Anomaly identificationPlan-view

16

This is a plan-view of the same image as the last slide.

24

Magnetics

Anomaly identificationTargets picked

Anomaly identificationTargets picked

17

The locations of potential ordnance items are either selected manually or by automatic target selection methods.

25

Magnetics

Examples of “good” dataExamples of “good” data

High SNR Medium SNR

Medium SNR Low SNR

Easting (m)

Easting (m)Easting (m)

Easting (m)

Nor

thin

g (m

)N

orth

ing

(m)

Nor

thin

g (m

)N

orth

ing

(m)

210 nT

-140 nT

22 nT

-21 nT

4.3 nT

-2.7 nT

8.5 nT

-7.5 nT

18

The four anomalies shown in this slide were obtained by the MTADS magnetometer array at Camp Sibert, AL. Each anomaly has dense data coverage (the black dots) and no obvious distortions caused by positional or other errors in the data. Notice the apparent striping caused by background geology in the image on the lower right.

26

Magnetics

Examples of “poorer” dataExamples of “poorer” data

Data gap, low SNR Data gap

Interference with geology

Geology orsensor lag?

4.2 nT

-1.3 nT

4.2 nT

-1.4 nT

18 nT

-8 nT

14 nT

-10 nT

Easting (m)

Easting (m)

Easting (m)

Easting (m)

Nor

thin

g (m

)

Nor

thin

g (m

)N

orth

ing

(m)

Nor

thin

g (m

)

19

The anomalies shown in this slide were collected by a man-portable magnetometer array at Chevallier Ranch, MT, under more challenging conditions than those at Camp SIbert. The top two anomalies suffer from data gaps caused by variations in the lane-spacing as the operator avoids small bushes at the site (the black dots mark the sensor locations). The bottom two anomalies are distorted either by geology or by positional inconsistencies between adjacent traverses over the anomaly.

27

Magnetics



Real-world examples





20

28

Magnetics

Parameter extractionParameter extraction

● The data that are measured are an indirect indicator of what is buried under the ground

● Inversion or “parameter extraction” is used to estimate the parameters of an underlying model that encapsulates some useful attributes of the buried object

Sensor data: d

Model Parameters: md =g [m]

m=g-1 [d ]

Forward Operator

Inverse Operator

21

The data themselves do not directly tell us if the underlying object is a munition or something non-hazardous like shrapnel, range scrap or cultural debris. The objective of parameter extraction is to estimate the parameters of an underlying model that encapsulates some useful attributes of the buried object. The forward problem involves estimating the magnetic anomaly caused by an object with particular attributes. The parameter extraction, or inverse, operation is more difficult and involves estimating the object attributes from the measured data.

29

Magnetics

Real-time demonstration of parameter extraction


Adjust azimuth“Observed” data Azimuth wrong

22

Screen-shots of a real-time demonstration of parameter extraction. The “observed data” are shown on the left. The data that would be produced by an initial guess at the underlying target attributes are shown in the center. They do not provide a good match to the data. The parameter extraction method (in this case we use physical intuition) adjusts the azimuth so that the orientation of the modeled anomaly now looks correct (the size and shape of the anomaly are still wrong at this stage).

30

Magnetics



Adjust depth“Observed” data Depth wrong

22a

The modeled data in the middle panel produce a smaller anomaly than what was observed. The modeled data agree much better with the observed data after the target is pushed deeper. The anomaly amplitude still doesn’t agree.

31

Magnetics



Size wrong“Observed” data Adjust size

22b

The depth and orientation are correct but the size is wrong. The amplitudes of the observed and modeled data match closely after increasing the size of the target.

32

Magnetics

Demonstration of parameter extraction

Demonstration of parameter extraction

Extracted parameters“Observed” data “Predicted” data

22c

The parameters that provide the best-match to the observed data reflect our best estimate of the target attributes of the underlying object.

33

Magnetics

Parameter extractionParameter extraction

● The demonstration we have just seen described one method of parameter extraction

Search by trial and error with a visual assessment of what modelfits the best

● In practice, highly efficient automated parameter extraction techniques based on non‐linear least squares are used

The objective is to minimize the difference between the actual and predicted data

23

34

Magnetics

The modelThe model

● We model the response of buried items by a dipole (equivalent to a bar‐magnet):

PositionDepthOrientationSize

N

S

DEPTH

PositionORIENTATION

24

A magnetic dipole is used as the underlying model for parameter extraction from magnetic data. The dipole is equivalent to a bar-magnet, whose lateral position, depth, orientation and magnitude need to be estimated.

35

Magnetics

Easting = 0.10 m

Northing = 0.21 m

Depth = 0.46 m

Moment = 0.20 Am2

Azimuth = 31.5o

Dip = -40.3o

Fit quality = 0.98

Parameter extractionParameter extractionDATA MODEL

Residuals

Parameters of interest

25

Example of a dipole model fit to magnetic data collected at Chevallier Ranch, MT. The panel shows the observed data (top left), modeled data (top-right), residuals (which are the difference between observed and modeled data, bottom-left) and extracted parameters (bottom right). While the estimated lateral position and depth are important, they don’t tell us anything substantial about the possible identify of the object. The moment, azimuth and dip, which encapsulate the size and orientation of the underlying dipole, provide information on the likelihood that the underlying object is a munition.

The parameter extraction technique returns an estimate of the “fit quality” which is an indicator of the reliability of the parameter estimates. If the fit-quality is low, then there is considerable uncertainty in the values of the underlying target attributes. In this case the fit-quality is high.

36

Magnetics

Easting = -0.13 m

Northing = 0.16 m

Depth = 0.26 m

Moment = 0.0226 Am2

Azimuth = 37o

Dip = 28.8o

Fit quality = 0.95

Parameter extractionParameter extractionDATA MODEL

Residuals

Parameters of interest

26

Another parameter extraction example from Chevallier Ranch. In this case, the moment is about 1/10th the size of the previous example.

37

Magnetics

Can’t analyzeCan’t analyzeDATA MODEL

Residuals

Easting (m)

Could be 2 anomalies or a mismatch in position on adjacent passes

Easting = 0.20 m

Northing = -0.31 m

Depth = 0.26 m

Moment = 0.05 Am2

Azimuth = 95.5o

Dip = -13o

Fit quality = 0.82

For some anomalies the data are not of sufficient quality to support reliable parameter extraction. We refer to these as “can’t analyze” anomalies

27

Example of an anomaly with low fit quality. In this case, we can’t rely on the extracted parameters and would place the anomaly in a “can’t analyze” category. In the absence of further information, this anomaly would need to be treated as a potential target-of-interest.

38

Magnetics



Real-world examples





28

39

Magnetics


● The objective of a UXO remediation strategy is to produce a prioritized dig‐list with an indication of how many items have to be excavated as potential UXO

● The topic will be covered in detail by Dean Keiswetter in a later module.

● Here we just provide an example of ranking the dig‐sheet based on the size of the moment

29

40

Magnetics

Example: Camp SibertExample: Camp Sibert


Reject as much clutter as possibleWithout leaving any 4.2inch mortars unearthed.

30

The example that follows comes from Camp Sibert, AL where the objective was to recover all 4.2” mortars while leaving as much clutter in the ground as possible.

41

Magnetics

Size versus amplitudeSize versus amplitude● Ranking by amplitude results in significant overlap

between all classes● Overlap is reduced considerably when ranking by size of

objectHistogram based on amplitude Histogram based on size

31

The bar-chart at left provides histograms of the amplitude response (difference between positive and negative lobes of the measure anomaly) from

(1) Shrapnel and cultural debris (or junk)(2) Base-plates(3) Partial rounds; and(4) Intact 4.2 inch mortars

Most of the shrapnel and debris have low anomaly amplitudes. The 4.2” mortars tend to have higher amplitudes, but there is a considerable range in values. To recover all 4.2” mortars would require digging up almost all of the clutter items.The bar-chart at right shows histograms of dipole moments obtained through parameter extraction. The smaller items (shrapnel, base-plates) tend to have small moments and can largely be distinguished from the larger 4.2” mortars which have larger estimated moments. Many clutter items could be left if the ground if the dig-sheet were prioritized based on the size of the moment.

42

Magnetics 32

Summary 1Summary 1

● Ferrous ordnance and non‐ordnance distort the Earth’s magnetic field

● Cesium‐vapor total field magnetometers are used extensively in ordnance detection applications

● Magnetic anomalies depend on the size, shape, orientation and depth of the buried object

● Survey parameters such as sensor height, sensor noise levels, position errors and lane‐spacing effect the quality of the collected magnetic data

Magnetics 33

Summary 2Summary 2

● Parameter extraction routines are used to estimate the attributes (size, orientation, depth) of a detected buried object

● The extracted parameters are used to create a prioritized dig‐list

● Magnetic data are largely immune to sensor orientation, can be rapidly collected and are highly sensitive to the depth of the buried item

● Magnetic data can be adversely affected by geology, only return an approximate estimate of the object’s size and can’t be used to (uniquely) determine the object’s shape.

1

Electromagnetics (EM):Fundamentals and Parameter Extraction

Electromagnetics (EM):Fundamentals and Parameter Extraction

Thomas Bell

EMI

EM Module OutlineEM Module Outline● EM Fundamentals

Basic principles of electromagnetic inductionSensors, EM signals & noise

● ClassificationEstimating target attributes (features) from EM dataFeature‐based classificationData quality requirements & current technology limitations

2

2

EMI

EM FundamentalsEM Fundamentals

3

This sequence of slides shows the fundamental concepts involved in EM measurements, starting with a picture of a typical EM sensor being pulled across a field.

3

EMI 3a


The basic elements of an EM sensor are a transmit coil and a receive coil shown by the loops above the ground surface. A current pulse running through the transmit coil creates the primary EM field, illustrated by the arrows flowing along field lines shown in red. This pulse excites the munitions item under the sensor.

4

EMI 3b


Changes in the primary field set up eddy currents in the object, shown schematically by the green arrows seeming to flow around the buried munitions item.

5

EMI 3c


The eddy currents produce a secondary or induced EM field emanating from the object. This field can be represented by an induced dipole at the object's location. The strength and orientation of the dipole moment are determined by the primary field at the object and physical properties of the object such as its size and shape, as well as its orientation.

6

EMI 3d


The induced field is measured by the receive coil, the output signal being proportional to the rate of change of the EM flux through the receive coil.

7

EMI 3e


There are two basic types of EM sensor: continuous wave (frequency domain), and pulsed wave (time domain). Frequency domain sensors transmit a continuous waveform, while time domain sensors transmit a sequence of EM pulses. The pulsed sensor is the most commonly used configuration because it allows the eddy current response to be measured when the primary field is not changing and is no longer overwhelming the signal due to the induced field. The two plots show typical transmit and receive waveforms for a pulsed EM sensor and identify the three stages of the EM measurement process. (1) The object is magnetized only during the transmit pulse. (2) The eddy currents are excited in the target when the pulse abruptly ends. (3) The EM response is measured during the eddy current decay after the primary field pulse ends. This measured decay contains the information that is used to classify the target.

8

EMI

Basic EM ConceptsBasic EM Concepts

1. The primary field magnetizes the buried object (similar to magnetics)

2. Abrupt change in the primary field excites eddy currents in the object.

3. Eddy currents diffuse throughout the object and decay (basic EM response which applies to all metal objects) 4

Review. A typical EM sensor measures the EM field associated with the decay of eddy currents in metal objects near the sensor. The eddy current decay occurs after the current pulse in the transmitter loop is completed, and hence after the response directly caused by any magnetization of material near the sensor. Unlike magnetometers, EM sensors respond to all types of metal objects, not just ferromagnetic ones.

9

EMI

EM SurveyEM Survey

5

This sequence shows the measured EM response as a sensor travels back and forth along survey lines over an object. The black dots correspond to the measurement positions and the red points represent the measured response at those points. The measured signal corresponds to the induced field at a fixed time during the eddy current decay cycle.

10

EMI

EM SurveyEM Survey

5a

Data are collected along a sequence of survey lines over the object .

11

EMI

EM SurveyEM Survey

5b

The data are mapped to a grid and displayed using a color scheme where the response to the object shows up as an isolated feature or "anomaly" above the background level.

12

EMI

EM vs. MagneticsEM vs. Magnetics

EM Magnetics

● EM eddy current response is less sensitive to interference from geology than magnetic response

● Time decay of eddy current response provides classification information not available with magnetic sensors

6

This slide shows color maps of data from EM and magnetic surveys over the same area. Anomalies due to buried objects show up as red or purple regions in the EM data map, and paired red and blue regions in the magnetic data map. The magnetic data map shows much more background structure than does the EM data map. Because they measure the eddy current response after the magnetization phase, EM sensors are less sensitive to interference from geologic structures than magnetic sensors. The time decay of the eddy current response provides classification information that is not available with magnetic sensors.

13

EMI

EM ClassificationEM Classification● Time decay of eddy current response

Determined by physical properties of object.Varies differently with sensor/object geometry for different objects.

Early time ‐ eddy currents at surface, response set by objectʹs size and shape.

Late time ‐ eddy currents diffused through object, response set by thickness of material.

7

EM classification is based on properties of the eddy current decay which are determined by physical properties of the object being measured with the EM sensor. The graph on the right is a blowup of the eddy current decay illustrated previously. Early in the response the eddy currents are confined near the surface of the object, and the characteristics of the measured signal reflect the size and shape of the object. As time progresses, the eddy currents diffuse into the object until at late times the response is determined primarily by the thickness of the material that the object is made of. A key factor exploited in the classification process is that the induced field due to the eddy current decay varies in a systematic way as the sensor/object geometry is changed.

14

EMI

Processing StreamProcessing Stream● Stages in the EM classification process:

1. Data Collection

2. Signal Attribute Extraction

3. Classification

8

There are three stages in the classification process. They are illustrated schematically in this slide. The first stage is data collection over an object. The picture shows data being collected with a Geonics EM61 metal detector, which is a widely available EM sensor. In the second stage, attributes or features of the EM response that relate to physical properties of the object are extracted from the data by fitting to a physics-based model. Finally, the object is classified by deciding whether the set of attributes is more like those typical of munitions items or those typical of clutter items.

15

EMI

Commercial EM SensorsCommercial EM Sensors

10

There is a wide variety of commercially available EM sensors. A few are illustrated on this slide. Geonics EM61 metal detectors (shown in the upper two pictures) are the most commonly used EM sensors in UXO work. The lower set of pictures shows, from left to right, a Geophex frequency domain sensor, the Geonics EM63 sensor (which measures a larger portion on the eddy current decay cycle than the EM61), and a Minelab sensor which has less sensitivity to magnetic soils than most other EM sensors.

16

EMI

Raw EM DataRaw EM Data

● The Geonics EM61 metal detector is the most commonly used EM sensor for UXO detection

● Signal strength depends on size, shape, orientation and depth of buried object to be detected

EM61 output (channel 2 of 4) as sensor is wheeled over a 3” Stokes mortar buried 80 cm below ground level

11

This slide shows the EM61 sensor and a sample of EM61 data collected as the sensor was pulled over a buried munitions item. The EM61 has four signal channels. The first three sample the eddy current decay measured by the lower rectangular coil at three different times. The fourth channel can sample either the upper coil response in the third time gate or the lower coil response at a fourth time. The peak signal strength in any channel depends in a predictable way on the size, shape and depth of the buried object.

17

EMI

Signal Strength vs. DepthSignal Strength vs. Depth

● Signal strength decays as sixth power of depth (vs. third power for magnetics)

● Peak signal is strongest for UXO if the item is aligned vertically, weakest if it is lying flat

12

The graph on this slide is a typical plot showing measurements of an object's signal strength (peak signal over the object as shown by the blue bar on the EM61 signal trace reproduced from the previous slide) as a function of the depth of the object. The EM signal decays more rapidly with depth than the corresponding magnetic signal (sixth power vs. third power, respectively). The dashed lines on the graph show the bounding curves for maximum and minimum signal strengths corresponding to vertical and horizontal target orientations. The strength of the signal due to an object depends not only on the object's depth, but also on the orientation of the object. This is reflected in the scatter of the measurements at a fixed depth. For horizontal coplanar coils (like the EM61) the peak signal is strongest when the object is oriented with its long axis vertical, and weakest when it is horizontal.

18

EMI

Detection LimitsDetection Limits

● Detectability limited by sensor and background noise

75 mm projectile, 35 cm deep(30° dip)




13

Whether or not a given object will be detected depends not only on how strong a signal it creates, but also on the level of noise in the measurements. This slide shows profiles of noise measured by an EM61 with added signals that would be created by three different munitions types. The root mean square (rms) noise level is 0.25 mV, which is relatively benign. The top plot includes the signal due to a 75 mm projectile buried 35 cm below the ground surface, tilted down 30º from horizontal. Signal plus noise is shown in red, with the noise-only profile overlaid in black. The signal completely dominates the plot. The middle plot is for a 37 mm projectile at the same depth and orientation. Here the signal is about ten times weaker (note the change in scale on the vertical axis), but still much stronger than the noise. At this scale the noise fluctuations are clearly visible. The bottom plot is for a 20 mm projectile at the same depth and orientation. The signal is weaker still (the vertical scale is further magnified in this plot), and is now obscured by the noise fluctuations.

19

EMI

Signal to Noise Ratio (SNR)Signal to Noise Ratio (SNR)

Require SNR (Smax / rms noise) > 5‐6 for reliable detection

Smax = 38.3 mV, SNR = 153

Smax = 4.5 mV, SNR = 18

Smax = 0.75 mV, SNR = 3


14

The signal-to-noise ratio is a simple measure of the relative strengths of the signal and the noise. It is equal to the ratio of the maximum signal strength to the standard deviation of the noise (root mean square or rms noise). This slide repeats the profiles from the previous slide and shows the corresponding SNR values for the different targets. For the 75 mm projectile, the peak signal is 38.3 mV. The rmsnoise is 0.25 mV, so the SNR is 153. The object is clearly detectable. For the 37 mm projectile, the signal is 4.5 mV and the SNR is 18. It is still readily detectable. The 20 mm projectile is not detectable in this noise. Its peak signal is 0.75 mV and its SNR is 3. Typically, we will need a signal to noise ratio greater than 5 or 6 in order for the object to be reliably detected.

20

EMI

Noise SourcesNoise Sources

● Electronic noise in sensor● Ambient electromagnetic noise (natural or man‐made)● Sensor bouncing motion relative to ground & geomagnetic

field● Geologic structure● Metallic debris in ground● Operator‐related noise

Improper/careless operation of sensor (loose cables & connections, metal stuck in wheels, etc.)Metal carried by field personnel

● Processing Artifacts

15

Any fluctuations in the sensor output when there is no target along the path of the sensor represent noise. There are a variety of different sources of noise. The noise can be inherently time-dependent, as in the case of noise from the sensor electronics or atmospheric noise associated with lightning, nearby transmission lines, etc. Or it can arise from the temporal evolution of space/time processes. Noise is introduced when the sensor moves up and down near the ground so that the measured EM soil response fluctuates as the sensor moves along. Similarly, irregular movement of the sensor can introduce noise due to changing flux of the earth's magnetic field through the receive coil. Spatial variations in the geologic response can introduce additional noise as the sensor is moved over the ground. Generally, the largest noise contributions come from metallic debris in the ground, operator-related noise and data artifacts that can be inadvertently introduced during processing. Operator-related noise can result from improper or careless operation of the sensor (loose cables and connections, small metal pieces stuck in the wheels, metal carried by the operator, etc.)

21

EMI

Comparative Noise LevelsComparative Noise Levels● ESTCP noise study at Blossom Point, MD test site

Effects of platform motion (controlled tests, relatively benign site)

Channel 4 = upper coil, gate 316

This slide shows rms EM61 noise levels measured with three different sensor configurations: a wheeled EM61 pushed by the operator, a trailer-mounted EM61 towed behind a vehicle, and a stationary EM61. Each plot shows noise levels for the four EM61 channels measured at different locations at the test site. The moving platforms (left and middle) show significantly higher noise levels than that recorded while the sensor is stationary, and the vehicle-towed sensor shows lower noise levels than the operator-pushed sensor. Recall that the noise profiles used in the previous SNR illustrations were for channel 3 and had an rms level of 0.25 mV. This is comparable to the vehicle-towed noise for channel 3.

22

EMI

Target AttributesTarget Attributes

1. Data Collection


3. Classification

17

Once the EM data have been collected, anomalies possibly due to buried objects are identified and the corresponding data are processed to extract target attributes or features that can be associated with physical attributes of the object. Generally speaking, target attributes are the basic building blocks of the object's EM response. They depend only on what the object is, not where it is or how it is oriented. The blue boxes in the plot in the middle of the slide show the four EM61 measurement time windows, superimposed on the full decay curve for a test object. For single-channel EM61 data, the basic target attributes are a set of three parameters that can be used to calculate the EM61 response in that channel at any location above the object.

23

EMI

Processing EM DataProcessing EM Data● Dipole response model is used to interpret EM data

collected over an objectInduced dipole response (M) to primary field (H0) is expressed as vector sum of responses in the object’s three principal axis directions.Principal axis responses are dipole moment components (Mi)

18

EM data collected over an object are interpreted using a dipole response model similar to the dipole moment representation used with magnetometer data. The figure on the left at the bottom of this slide shows a munitions item with the primary field lines superimposed in red. The eddy current decay response to the primary field can be represented by an induced, time-dependent dipole at the center of the target. The dipole moment is expressed as a vector sum of component responses along each of the object's three principal axes, as shown schematically at the bottom right with the induced field lines superimposed in blue. For the symmetric object shown, the two smaller responses are equal and are designated as M2. The measured signal is the sum of the principal axis responses multiplied by the corresponding primary field components. The principal axis response functions are sometimes called principal axis polarizabilities, and represent how the eddy currents decay when the object is excited in each of the principal directions.

24

EMI

● The set of three principal axis responses (polarizabilities) constitutes the basic EM signature of an object.

Typical UXO items are symmetric, so two of the principal axis responses are the same.

EM SignatureEM Signature

105 mm projectile tractor muffler

Irregular clutter items typically have three different response functions.

19

The set of three principal axis responses constitutes the basic EM signature of an object. This is shown in the two figures at the bottom of the slide. Each figure shows a picture of an object (a 105 mm projectile on the left and a comparably sized muffler on the right), a set of arrows representing the three principal axes of the objects, and a set of response functions color coded by principal axis. All three principal axis responses are shown in each plot, with the one corresponding to the principal axis arrow colored accordingly and the others shown in gray. Two of the principal axis responses for the munitions item are the same because of the object's axial symmetry. All three of the principal axis responses are different for the muffler. This is expected for irregularly shaped clutter items.

25

EMI

Sampling the Full EM Response


20

In order to be able to determine the principal axis responses, the object must be excited and measured over a range of directions. This sequence illustrates how data collected at different locations above an object can accomplish this. In the first slide the object (e.g. the 81 mm mortar shown in the picture) is directly under the sensor, and we measure its response to a vertical field. This is shown by the blue curve in the graph at the right.

26

EMI



20a

When the sensor is moved off to the side, the primary field at the object changes direction and we measure a different response. Now, the original response from above the object is the gray curve, while the new response from off to the side is shown by the new blue curve.

27

EMI



20b

If we take measurements from enough different locations, we can generate a complete set of different decay curves that can then be used to determine the principal axis response functions.

28

EMI

Sensor/Object GeometrySensor/Object Geometry

● An object directly underneath the sensor is excited with a vertical primary field

● If the sensor moves off to the side then the object is excited by a horizontal primary field

● Classification exploits eddy current decays observed from a variety of excitation directions

21

Summarizing, classification exploits eddy currents observed from a variety of excitation directions. An object directly underneath the sensor is excited with a vertical primary field, as shown in the diagram on the left. If the sensor moves off to the side, then the object is excited with a horizontal primary field, as shown in the diagram on the right. By measuring the response from enough different locations we can collect the data needed to calculate the principal axis response functions.

29

EMI

Inversion of EM DataInversion of EM Data

• The EM response must be measured from enough different directions to adequately sample the principal axis responses (e.g. with accurately mapped EM survey data)

• The object’s location, depth & orientation are determined as part of inversion

{ } 1B −⋅μ= )t(CIC)t(V TR0

Data from different sensor/object geometries

Principal axis polarizabilities

Inverse operation

22

The dipole response model is used to extract the principal axis polarizabilities by inverting EM data collected over a buried object.

The dipole response model is used to extract the principal axis response functions by inverting EM data collected over a buried object. Inverting the data simply means using the dipole response model to work backwards from the measured data to the model parameters (the object's location and depth, the orientations of its principal axes, and its principal axis response functions) needed to reproduce the data. The process is illustrated schematically in the flow diagram on this slide. The sheaf of graphs on the left represents the set of induced field decay curves measured at various locations over the object. The equation in the middle represents the inversion process that operates on the data collected over the target. The graph on the right shows the principal axis response functions determined by the inversion process. To begin with, we have only the data collected over the target and the corresponding locations where the measurements were taken. In order for the inversion to produce accurate results, the locations where the measurements were taken must be known very accurately. The location and depth of the object are unknown, as are the orientations of the object's principal axes and the principal axis response functions. These are all determined by inverting the data. The inversion proceeds by systematically varying possible values for the object's location, depth, etc. and using the dipole response model to calculate the expected responses at the measurement locations. As the parameters (location, depth, orientation, and principal axis responses) are systematically varied, the inversion compares the set of calculated responses with the measured responses until the best match is found.

30

EMI


1. Data Collection


3. Classification

23

The final stage is classifying the object as a munition or clutter on the basis of target attributes that have been extracted from the EM data. Depending on the particular EM sensor that is used to collect the data and on the detailed workings of the classifier, the target attributes could be the complete set of principal axis response functions, principal axis responses averaged over some time window, a set of parameter values that describe significant aspects of the basic EM response, etc. The figure used to illustrate classification is a scatter plot of single-channel EM61 principal axis response values for various munitions and clutter items.

31

EMI

Conventional EM TechnologyConventional EM Technology

● EM61 measures eddy current decay averaged over four time windows or gates after primary field cutoff

● Limited decay curve classification capability● Can provide some size/shape/depth information

EM61

24

The EM61 is widely used in munitions detection work. The picture on the right shows an EM61 in the field. The blue boxes in the plot in the middle of the slide show the four EM61 measurement time windows, superimposed on a blowup of the decay curve shown in the plot at the left. Because the EM61 only measures EM response averaged over four time windows or time gates, it has diminished decay curve classification capability. However, it can still provide size and shape information useful for classification.

32

EMI

Size/Shape ClassificationSize/Shape Classification

● Single EM61 channel inversion● The object’s features are related

to the polarizabilities (βs)Size correlates with the net polarizability (sum of βs)Target shape is reflected in the relative sizes of the three βs

● UXO should have one large (β1) and two smaller, equal (β2,3) polarizabilities ‐ clutter often has three distinct βs

25

This slide shows the basic idea of classification using single-channel EM61 data. The data were collected using the handheld version of the EM61 (EM61-HH). Each symbol in the plot on the right corresponds to first time gate EM61 data collected over an object that have been inverted using the dipole response model to determine the object's three principal axis response coefficients. These coefficients are the polarizabilities averaged over the first time window measured by the EM61. They are sometimes referred to as "betas". The sum of the three betas (net polarizability) is a measure of the object's size, and is roughly proportional to the volume of the object. The shape of the object determines the relative magnitudes of the three betas. Munitions items will generally have one large and two smaller, equal betas, while irregular clutter items can have three different betas.The plot on the right shows betas calculated for various munitions and clutter items. The munitions are plotted in blue and the clutter in red. For each set of betas, the dominant (largest) polarizability is plotted along the horizontal axis, and the two secondary polarizabilities are plotted as a vertical line running upwards from the smallest beta to the middle one. There are six sets of betas calculated from different data sets for each munitions item. They all overlay nicely, and all show the property that the secondary (smaller) betas are roughly equal. The magnitude of the polarizaility correlates with munition size, which ranges from 20 mm caliber to 155 mm caliber. The clutter items (range scrap from Aberdeen Proving Ground) scatter throughout the munitions items and show varying degrees of shape irregularity as evidenced by the various lengths of the secondary beta lines.

33

EMI

Processing EM DataProcessing EM Data

EM61

26

With support from ESTCP, procedures for extracting target attributes from EM61 data have been implemented in the Geosoft Oasis montaj geophysical data mapping and analysis software suite. This sequence shows the processing flow. First, EM61 survey data are gridded and mapped, as shown in the first slide.

34

EMI


26a

The operator interactively selects an anomaly from the survey map for analysis. This is shown here by the box around one of the anomalies.

35

EMI


26b

The software performs a model match to the anomaly data and displays the dipole fit alongside the measured anomaly data. In this case, the dipole model does a good job reproducing the data. The target parameters (location, depth, principal axis orientations and polarizabilities) are calculated, along with the fit coherence, which is a measure of how well the model reproduces the data. The fit coherence is equal to the square of the correlation coefficient between model fit and data.

36

EMI

Anomaly Report SheetAnomaly Report Sheet● Single‐channel EM61 inversion● Badlands Bombing Range, Cuny

Table, Target 3Gridded Data Model Fit

Target Attributes

Profiles thru center with fit

27

An anomaly report sheet is prepared for each anomaly that is analyzed. An example from the Badlands Bombing Range is shown on this slide. The report sheet shows side-by-side maps of the gridded data with the anomaly highlighted and the model fit to the anomaly. The locations of the actual measurements are shown as black dots on the anomaly map. Profiles of the four EM61 data channels along the survey line passing closest to the center of the anomaly are plotted at the bottom of the page. The fit results (target parameters and fit quality) are also listed on the sheet.

37

EMI

Survey Data QualitySurvey Data Quality

Good survey quality control is important for classification.

Good survey data

Poor survey data

28

Good survey quality control is essential for reliable classification. This slide compares two different EM61 surveys over the same object. The anomaly map and profiles through the center of the anomaly are shown for each. The survey data illustrated on the top line is noticeably better than that shown on the bottom. In the lower survey, the anomaly shape is distorted, and the profiles are clearly skewed. The dipole inversion relies on subtle features of the anomaly shape, and can end up converging on an erroneous set of target attributes that reproduce the distorted data.

38

EMI

Data RequirementsData Requirements● Inversion of EM data requires accurately mapped survey

data or data collected using a grid template to control positioning

Also requires measurements of sensor orientation and vertical position

● Data density and spatial extent must adequately sample the principal axis polarizabilities

● No overlapping signals from nearby objects● SNR needed for classification > SNR needed to detect

object

29

Because the dipole inversion seeks to faithfully reproduce the shape of the EM anomaly, it is very sensitive to data distortions. It requires accurately mapped survey data or data collected using a grid template to control positioning of the sensor. The orientation and vertical position of the sensor have to be measured or constrained. There cannot be any interference due to signal overlap from nearby objects. Significantly, the signal-to-noise ratio (SNR) required for classification is much higher than that needed to detect an object. Finally, the data density along survey lines, the line spacing, and the spatial extent of the data patch selected for inversion must adequately sample the principal axis polarizabilities.

39

EMI

Model Match Fit QualityModel Match Fit Quality● Fit quality is determined by the mismatch between the

data and dipole model fit to the dataReflects ability of data quality to support inversion and estimation of target attributes for classification

Well resolved anomaly with good dipole fit quality (5% fit error)

EM61 (gate 1) data over 2.75” rocket warhead at Badlands Bombing Range

30

The dipole fit quality is determined by the mismatch between the data and the dipole model fit to the data. It reflects on whether or not the data quality is adequate to support reliable inversion and estimation of target attributes for use in target classification. The mismatch (fit error) is related to the fit coherence introduced a couple of slides back. The fit coherence is equal to one minus the square of the fit error. The plots on this slide show an anomaly due to a 2.75 inch rocket warhead and the dipole fit to the anomaly. This is very good data. The anomaly is well resolved. Several survey lines cross it, and there are a good number of data samples over the anomaly along each line. The dipole fit quality is good, with only a 5% fit error (fit coherence = 0.998).

40

EMI

Parameter Extraction IssuesParameter Extraction Issues

Line spacing does not provide adequate sampling of polarizabilities

Overlapping signals

Weak signal(low SNR)

31

This slide shows some examples of poorer data. The survey line spacing does not provide adequate sampling of the principal axis responses for the anomaly on the left. Only two lines cross the anomaly. The anomaly show in the center is actually a couple of overlapping anomalies. These cannot be sorted out using currently available processing techniques. The anomaly on the right has a signal-to-noise ratio that is too low to support reliable inversion.

41

EMI

SNR RequirementsSNR Requirements● Reliable estimation of target attributes (polarizabilities)

requires very high quality data with dipole fit error less than 5‐10%

● SNR approaching 100 isrequired for classification, compared to ~5 fordetection

32

The plot on this slide shows the results of dipole inversions on channel one data from a relatively high quality EM61 survey on a test field. The dipole fit error for each of the inverted anomalies is plotted as a function of the signal-to-noise ratio (SNR) of the anomaly. In general, the fit error decreases with increasing SNR. Experience has shown that reliable estimation of target attributes (first time gate principal axis response coefficients) requires very high quality data with dipole fit errors less than 5-10%. The plot shows that dipole fits errors in this range are generally not achieved if the SNR is less than about 100. This is much, much larger than the SNR of about 5 or 6 needed to reliably detect the object.

42

EMI

Positioning ErrorsPositioning Errors● Errors in recorded sensor locations corresponding to EM

data can substantially increase dipole fit errorIt is very hard to maintain survey geolocation at the accuracy level required to support reliable classification

33

Errors in the recorded sensor locations corresponding to EM measurements act like noise in the data, and can have a significant impact on the dipole fit error. The basic fit error vs. SNR plot on this slide is reproduced from the previous slide. The solid line shows the relationship between dipole fit error and SNR that would be expected if the recorded sensor positions were correct. The dashed lines show how the relationship is affected by increasing rms positioning errors. Once the errors in the recorded sensor locations get larger than a few cm, the dipole fit errors even for very high SNR targets will be large enough to compromise the accuracy of target attributes extracted from the data.

43

EMI

Inversion FailuresInversion Failures

● For some objects, inversion of the EM data fails to even produce interpretable results (ʺcanʹt analyzeʺ)

Generally due to low SNR, positioning errors and/or interferencefrom signals due to nearby objects.

Results from EM61 survey at Camp Sibert Classification Study

34

For some objects, inversion of the EM data fails to even produce interpretable results. We refer to these cases as "can't analyze". Since we cannot classify them, anomalies that fall into the "can't analyze" category must be treated as indicating possible munitions items. Most anomalies that fall in this category have low SNR, significant positioning errors and/or interference from signals due to nearby objects. The plot is a histogram of the distribution of SNR values for EM61 anomalies analyzed for the ESTCP Camp Sibert Classification Study. The blue line shows the distribution for all of the anomalies, and the grayed area shows the distribution for the subset that were declared "can't analyze". It shows that a significant fraction of the weaker anomalies could not be properly inverted and analyzed.

44

EMI

Technology ImprovementsTechnology Improvements

● Classification performance of conventional technology (e.g. EM61) is limited by two primary factors

The eddy current decay cycle is not fully capturedMulti‐cm positioning errors inherent to field survey work compromise the accuracy of dipole inversion and estimation of target attributes

● New UXO‐specific technologies which avoid these problems are being developed and tested under SERDP and ESTCP

35

Classification performance using data collected with conventional technology such as the EM61 is limited by two primary factors: (1) the eddy current decay cycle is not fully captured, and (2) cm-level positioning errors inherent to field survey work compromise the accuracy of dipole inversion and estimation of target attributes. SERDP and ESTCP are developing and testing new UXO-specific technologies which avoid these problems and should provide significantly improved discrimination performance. These emerging technologies will be described in a later section of this course.

1


Dean Keiswetter

Classification, as used here, is a process that results in a decision. The decision we are trying to make is whether or not the subject anomaly could possibly be caused by a target of interest.In this module, we will introduce basic concepts and approaches used during the classification process. We will briefly touch upon:

1. The attributes that form the basis of the decision,2. Approaches for ranking the anomalies,3. Methods for selecting decision thresholds,4. The ‘product’ of the classification process – a prioritized dig list5. Evaluating performance, and finally6. A model for implementing the classification process.

2

Classification

Classification IntroductionClassification Introduction

● At this stage in the process, we have derived target attributes from the measured data that describe the source for each and every anomaly

● Our task is to use these attributes to identify those anomalies, if any, that cannot possibly be targets‐of‐interest (TOI) at our site

● In other words, we need a principled process that results in a decision ‐‐ is the source of the subject anomaly hazardous or not?

2

In the previous modules, we have learned how to derive attributes from the measured field data that describe the source object. Here, we are interested in using these attributes to make a decision.Our goal is to identify those anomalies, if any, that cannot possibly be caused by targets of interest at the site in question. To the extent that we can uniquely identify the non-TOI (or clutter), we can effect the remediation process and save money, time, and effort.Ideally, the decision process should be transparent, based on quantitative attributes, and principled.

3

Classification

0.00 0.05 0.10 0.15 0.200.01

0.10

1.00

10.00

Attribute 1

Attr

ibut

e 2

UXOHalf ShellMunition DebrisCultural

TOI

Classification ExampleClassification Example

● Assume attributes for a site with a single munitions item

● General Process:1) Visualize attributes2) Obtain labels (e.g., ground

truth information)3) Establish boundaries – this is the classification piece

● It can be this easy…if the features are separable

Remember the goal: identify anomalies that are NOT TOI

TOI ‘space’

Non‐TOI ‘space’

3

Let’s use a simple example to illustrate the process in very general terms. Here we assume that there is only a single target of interest. Geophysical data have been acquired and inverted and we have derived two attributes. It doesn’t matter at this time if we started with magnetometry or EMI data. All that matters is that we have used phenomenological models to derive target attributes.The general process is to (1) first visualize the attributes, (2) obtain labels (viz., ground truth descriptions regarding the physical nature of each source item), and (3) establish boundaries, in parameter space, that segment the TOI from the non-TOI.The third step in the generalized process mentioned above is the classification piece and is the focus of this module.Before diving into details, let’s review our objective in terms of what the classification process will produce.

4

Classification

Classification ProductClassification Product

● Prioritized Dig ListRank based on the probability that the anomaly is non‐TOIDivided into 4 categories

- High‐confidence non‐TOI- Can’t make a decision- High‐confidence TOI- Can’t analyze

Prioritized Dig List

Non

-TO

ITO

I

4

This slide presents a prioritized dig list. This is the product that we are after. It ranks anomalies based upon the probability that the source is a non-TOI and explicitly incorporates uncertainties. It does this by having 4 categories. There is one category for non-TOI, one for TOI, one for ‘Can’t Decide’, and one for ‘Can’t Analyze’.As a geophysical services community, this is where our job ends. Our task is to provide as much information to the stakeholders as possible regarding each and every anomaly. This includes attributes describing the source object and classification results based on information available at the time. The clients and stakeholders, in turn, decide how to act upon the information.

5

Classification

● Visually, we use physical attributesSizeSymmetryShape

How do we classify? How do we classify?

5

Okay – let’s begin by discussing how we might perform the classification process if we could visually see the objects. This is something we can all relate to. We might use size if appropriate. We might use shape if appropriate. We might even use axial symmetry to group the objects.

6

Classification

60mm

● Unfortunately, we cannot visually inspect buried objects● We have to utilize attributes derived from geophysical

data: Size, Symmetry, Shape Decay Rate● EMI & Mag

How do we classify? How do we classify?

Late TimeEddy currents diffused thru target Decay rates determined by thickness of target material

Early TimeEddy currents at surfaceResponse reflects target size and shape

Time (ms)

Pol

ariz

abilit

y

6

Unfortunately, we cannot visually inspect buried objects. We have to use other attributes -- attributes that can be derived from the available data.Earlier, we learned that both magnetic and EMI data can be inverted to obtain model parameters, or attributes as used here, that are intrinsic to the target. Intrinsic means that the attributes depend on the physical nature of the object itself, not on its surrounding (like location or orientation).The principle axis polarization data shown in this example illustrate the information available to us for making the decisions. There are three polarizations, one along the longitudinal axis and two for the transverse axes. They are functions of time and provide information regarding the target’s size and shape, as well as wall thickness.Attributes appropriate for classification can be derived from either magnetic or EMI data.

7

Classification

Which Attributes are Important?Which Attributes are Important?

● Answer: any attribute or set of attributes that are stable and provideseparation between TOI and non‐TOI.

They should be intrinsic to the target (not orientation or location)

● Intrinsic attributes include:Size

- Magnetic Moment- Principal axis polarizations (EMI)

SymmetryShapeDecay rate

● Two basic approaches to making a decision based on the attributesRule‐basedStatistical Classifier

7

Which attributes are important?Answer – any or all of them, as long as they are stable and can be used to differentiate TOI from at least some of the non-TOI.The attributes should relate to the target itself. Orientation with respect to the earth’s field or spatial location, for example, are not good indicators of munitions.The intrinsic attributes that we have access to include

• Size of the target• Symmetry of the target• Shape of the target, and • Decay rate.

As discussed earlier, target size attributes can be derived from either magnetic or EMI data. Target attributes relating to axial symmetry, shape, and EMI decay rate can be derived from EMI data only.Next, let’s discuss two fundamentally different approaches to making the decision based on the selected attributes.

8

Classification

Rule-based DecisionsRule-based Decisions

● Rule‐based classifiers base decisions on formal rules. Uses classical IF (condition) THEN (consequence) logic. For example:

IF the size‐based attribute > 1, THEN TOI.IF the size‐based attribute > 1 AND the decay‐based attribute is > 100, THEN TOI.

● Can be used to bound decision. For example:

IF (regardless of other attributes) the depth > 1m, THEN non‐TOI (viz., not interested in deeply buried objects)IF (regardless of other attributes) the size is less than 0.02, THEN non‐TOI (viz., not interested in small objects).

● Rules can be combined to form decision trees.

8

One approach for making the classification decision is to use a Rule-based method. This method uses classical IF (condition) THEN (consequence) logic.Rules can be used to identify TOI directly.Rules can be used to identify certain non-TOI directly.Rules can be used to bound decisions.Rules can be combined to form decision trees.Rule-based decisions are easy to comprehend when a limited number of attributes are used.

9

Classification

Rule-based DecisionsRule-based Decisions

● Use simple relationships and rules to make a decision { IF (condition) THEN (consequence)}

Remember the Goal: Identify anomalies that are NOT TOI

Target Size‐based Attribute

Decay‐based Attribute

9

Rule-based decisions use simple relationships and rules to make a decision.Here we have some example attributes. The scatter plot shows an attribute related to target-size on the x-axis and an attribute related to the target’s decay rate on the y-axis. Colored symbols are used to show the labels – or ground truth information.In this case, the red plus signs are used for UXO. The remaining colors and symbols identify various types of non-TOI.Our objective is to develop rules that allow us to segment the feature space such that each and every anomaly is ranked as either a TOI or not – and provide some level of confidence or measure of certainty.

10

Classification

Decision ThresholdsBased on Size‐based Attribute

Decision ThresholdsBased on Decay‐basedAttribute

Setting the Threshold fora Rule-based Decision

Setting the Threshold fora Rule-based Decision


Target Size‐based Attribute

Decay‐based Attribute

Decision ThresholdsBased on bothAttributes (aggressive)

Decision ThresholdsBased on bothAttributes (conservative)

10

One approach might be to base the decision solely on the target-size-based attribute. As you can see, given this distribution of attributes, this would result in a good – but perhaps not optimal – result. The blue line separates most of the TOI from the majority of the non-TOI. As shown here, however, a number of the larger fragments [namely the Partials (rounds that have been split open but not fragmented)] would be mis-classified.Another approach might be to base the decision solely on the decay-based attribute (green line). As before, this would have some classification success but is not optimalIf we combine the two rules, we can achieve a much better result. Here (small square), the decision space is aggressively chosen to include all TOI. There isn’t much play allowed for unanticipated variability. Alternatively, (large square) we could select a larger section of the feature space. This later approach provides significantly more buffer to handle TOI variances that are not represented in the training data.

11

Classification

Statistical ClassifiersStatistical Classifiers

● Statistical classifiers are computer algorithms that make use of one or more attributes to make a quantitative decision. They

statistically characterize attributes and create group associationsuse training data to attach labels (viz., ground truth information) to the groupsprovide explicit probabilitiesaccommodate many attributes and data dimensions

11

Statistical classifiers are computer algorithms that consider multiple variables to make a quantitative assessment of the likelihood that a signal corresponds to a target of interest. They statistically characterize the attributes to create group associations. Unlike the rule-based scenario, statistical classifiers partition the feature space automatically based on the nature of the attributes and the training data provided.They do not make yes or no decisions per se. Uncertainties and measures of confidence are inherent to the decision process. In fact, they provide explicit probabilities. Probabilities range from 0 to 1. Simply put, the result from a statistical classifier may be phrased as ‘the probability that this anomaly is caused by a TOI is less than 0.1’Additionally, due to the digital nature of the process, statistical classifiers easily accommodate numerous attributes and associated data dimensions. If presented with many attributes, they can be used to automatically select which attributes provide optimum results.

12

Classification

Statistical ClassifiersStatistical Classifiers

● Classifier performance depends greatly on the characteristics of the data to be classified

● There is no single classifier that works best on all given problems

● There are many statistical classification schemes – a few common classifiers include:

Support Vector Machinesk‐Nearest NeighborsGeneralized Likelihood Ratio Test (a Bayesian approach)Probabilistic Neural Networks

12

Having made the statements on the previous slide, a little caution is warranted…Domain knowledge is certainly required. Without a fair amount of knowledge regarding the specific classifier being used, it is easy to misinterpret and/or overstate the results.

13

Classification

Statistical ClassifiersStatistical ClassifiersRemember the Goal: Identify anomalies that are NOT TOI

Size-based Attribute

Dec

ay-b

ased

Attr

ibut

e

● Use computer algorithms to make a decision Numerically classified based on quantitative attributes and on training data

● Obtain labels● Submit data/labels

to classifier & returnprobabilities

0.00 0.05 0.10 0.15 0.20-1

0

1

2

UXOHalf ShellMunition DebrisCultural

TOI

13

Statistical Classifiers are computer algorithms that statistically characterize the attributes and create group associations.Again, our objective is to segment the feature space such that each and every anomaly is ranked as either a TOI or not – and provide some level of confidence or measure of certainty.Consider these example data. As before, the scatterplot shows an attribute related to target size on the x-axis and an attribute related to the target’s decay rate on the y-axis. Colored symbols are used to show the labels or ground truth information.In this case, black diamonds are used for a single TOI. The remaining colors and symbols identify various types of non-TOI.The next step is submit the attributes and associated labels to the classifier for training. After training, the classifier returns a probability for each anomaly. These are plotted according to the grey scale on the right. The shaded contours identify areas in which the probabilities change rapidly. In this simple case with two attributes and a single item of interest, it is easy to see that the results appear reasonable.

14

Classification

Setting the Threshold fora Statistical Classifier

Setting the Threshold fora Statistical Classifier


Size-based AttributeD

ecay

-bas

ed A

ttrib

ute

0.00 0.05 0.10 0.15 0.20-1

0

1

2Probability Non‐TOI > 0.9

Probability Non‐TOI < 0.1

Classifiers return probabilities

Principle Threshold: 1. Boundary between high

confidence non‐TOI and everything else…

2. Chosen to exclude all TOI3. Adjusted to account for observed

variability3. Adjusted to account for observed

variability UXOHalf ShellMunition DebrisCultural

TOI

14

Given the classifier results, lets move on to setting thresholds.Instead of setting the threshold(s) at a specific attribute level – as was done in the rule-based approach – we can use the probabilities provided by the statistical classifier.The primary threshold that we are after is the boundary between high-confidence non-TOI and everything else. As before, we start by identifying a boundary that includes all of the TOI.The green line shown here identifies the region for which probability of being non-TOI is less than 0.1. The anomalies within the area bounded by this line are unlikely to be to clutter. As you can see, all of the UXO – or the TOI are within this boundary.The next step is to adjust our probability threshold to account for unexpected variability in the TOI distribution.The red line shows the boundary for non-TOI probabilities of 0.9. Any anomaly outside this boundary is very likely to be non-TOI. This is the primary threshold – the boundary between high confidence non-TOI and everything else - that was actually used.

15

Classification

High-confidence & TOI Threshold

High-confidence & TOI Threshold

● The non‐TOI threshold is chosen to exclude all labeled TOI. It is, therefore, driven by the observed variability within the TOI class

Dependent on the data available for trainingIt can and should be updated during excavations (and the unknowns reprioritized based on the new information).

● Unknown targets are classified and ranked based on their relationship to this threshold

15

As shown in these two data examples, the non-TOI threshold was chosen to exclude all TOI and account for potential variability within the TOI class.Because of this, the decision boundary is very dependent on the data available for training and the intra-class variance of the extracted attributes for the TOI.As the dig program proceeds, additional ground truth information will become available. This important information can and should be used to retrain, reclassify, and reprioritize the dig list.

16

Classification

Prioritized Dig ListPrioritized Dig List

● Rankings for Rule‐based classifier are based on the distance, in parameter space, from the TOI boundary

● Rankings for Statistical classifier are based on the probability of belonging to a the non‐TOI class

Prioritized Dig List

Non

-TO

ITO

I

16

Here we see the prioritized dig list again. This is, again, our ultimate product.For Rule-based methods, the anomalies are ranked based on the distance, in parameter space, from the observed TOI boundary.For Statistical classifiers, the anomalies are based on the probability of belonging to the non-TOI class.Next, let’s take a look at one approach for expressing the results.

17

Classification0 100 200 300 400 500 600

0.0

0.2

0.4

0.6

0.8

1.0

Sensor A

Sensor B

0 100 200 300 400 500 6000.0

0.2

0.4

0.6

0.8

1.0

Sensor A

Sensor B

Expressing the ResultsExpressing the Results

Desired Performance100% TOI recovered0 non-TOI removed

Classification 100% TOI recovered

~200 non-TOI removed

Detection Only100% TOI recovered

~600 non-TOI removed

Receiver Operating Characteristic Curve• Retrospective (only exist if all targets removed)

non-TOI

TOI (

norm

aliz

ed)

Can’t Analyze

17

Here we have plotted what is known as a Receiver Operating Characteristics (ROC) curve. It is retrospective in that it requires all ground truth. These curves were taken from a study at Camp Sibert in which all contacts were dug for learning purposes.

Basically, it is a plot comparing the number of actual TOI objects versus non-TOI if our prioritized dig list were excavated in reverse order.

The colors of the points plotted represent the classification categories used for the prioritized dig list. Of specific importance are those in Green, because this identifies anomalies that were classified high-confidence non-TOI.

Desired Performance – Ideally, we would like perfect classification. If we could achieve this, the curve would rise straight to this location: 100% TOI recovered and 0% non-TOI.

Detection Only – this is the other extreme. Here, we ignore classification altogether and simply excavate all detected targets. In the end, we have 100% of the TOI recovered (approximately 150 munitions), but we also have removed 100% of the non-TOI – which for this site was approximately 600 non-hazardous objects.

Classification – Here, we evaluate the classification results. This is the point at which the demonstrator drew the threshold between high-confidence non-TOI and everything else. As shown here, 100% of the munitions were recovered but only 200 of the non-hazardous non-TOI recovered. This is a huge success.

Can’t Analyze –This portion of the curve is reserved for those anomalies that were thrown into the Can’t Analyze category. Because no useful information regarding the nature of these targets can be extracted from the measured data, they must be treated as potential targets of interest.

18

Classification

Practical Model for the Classification Process…

Practical Model for the Classification Process…

● Need prospective rather than retrospective approach…● Real‐world challenges include:

How to proceed with the dig program?- dig all items not classified high‐confidence non‐TOI

How to decide when to stop digging?- accept the analysts’ threshold and stop- increase confidence by

– Seeding the site– Remediate 100% or a number of grids– Excavate a percentage of the non‐TOI class

● Need prospective rather than retrospective approach…● Real‐world challenges include:

How to proceed with the dig program?- dig all items not classified high‐confidence non‐TOI

How to decide when to stop digging?- accept the analysts’ threshold and stop- increase confidence by

– Seeding the site– Remediate 100% or a number of grids– Excavate a percentage of the non‐TOI class

18

The ROC is good for learning, but here we consider a practical model for the classification process.Questions commonly encountered in real-world settings include

1. How to proceed with the dig program2. How do we decide to stop digging? When can we trust the classification?

With regard to the first question – the first step is to recognize that all of the targets not classified as high-confidence non-TOI must be dug.This naturally leads to the second question – when to stop? Here we have some options. The first option is to simply accept the analysts’ threshold and stop.There are, however, a few steps that can be easily implemented to increase confidence along the way. These include

• Seeding the site (recommended)• Digging 100% of a number of the grids• Digging a percentage of the non-TOI class, especially those that cluster in

feature space.

19

Classification

The Classification Process is not static…The Classification Process is not static…

● The initial ranking was trained using labeled data available at the beginning of the project

● Each dig provides additional ground truth

● Classifier should be retrained on a regular basis

● As we gain information, it may be possible to redraw the decision boundaries and exclude more non‐TOI

Target Size Attribute

Decay Attribute

Can

’t an

alyz

e

High confidenceTOI

Can’t Decide High confidenceNon-TOI

0

1

Pro

babi

lity

(TO

I) Initial

19

As discussed before, the classification process is not static.The initial rankings and decision thresholds were based on data and labels available at the time. Often, these can be limited in number and diversity.As ground truth information is revealed, the classification process should be rerun to take into account this new information.The cartoon on the right shows an attribute scatter plot on the top and the corresponding TOI-probability versus classification category on the bottom.

20

Classification

The Classification Process is not static…The Classification Process is not static…

● The initial ranking was trained using labeled data available at the beginning of the project

● Each dig provides additional ground truth

● Classifier should be retrained on a regular basis

● As we gain information, it may be possible to redraw the decision boundaries and exclude more non‐TOI

Can

’t an

alyz

e

High confidenceTOI

Can’t Decide High confidenceNon-TOI

0

1

Pro

babi

lity

(TO

I)

Target Size Attribute

Decay Attribute

Revised

19a

Perhaps with additional data and more ground truth, the classification categories could change as shown here. Basically, the new information has allowed the analyst to establish tighter thresholds and increase the number of high-confidence non-TOI declarations.This is a cartoon, and actual results may trend in the opposite direction for a given site. In other words, the site may become more and more complex as more ground truth is revealed. The point remains, however, that ground truth information acquired over time throughout the digging phase should be incorporated into the classification process.

21

Classification

Summary Thoughts for Real World Sites

Summary Thoughts for Real World Sites

● The goal of classification is a decision● Classification schemes allow us to identify anomalies

that cannot be caused by the site‐specific TOI in a principled manner

● The classification process must be transparent● Attributes should be:

(i) stable, (ii) consistent, and (iii) show separation between TOI and non‐TOI classes

● The value depends on the items defined to be TOI and non‐TOI, the observed variance within the TOI classes, and the risk tolerance of the stakeholders

20

1

Case HistoriesCase Histories

Dean Keiswetter

In this module, we will present two successful classification projects. As we go through the projects, we will revisit many of the topics presented earlier. Specifically, we will discuss the

1. objectives – what are we looking for and what can be left behind safely?2. which data types and methods of collection were used,3. which attributes were used as the decision basis,4. how were the attributes used to make a decision, and5. after digging all anomalies, how would we have performed if our classification

scheme had been implemented.

2

Case Histories

Former Camp Sibert, Gadsden, Alabama

Former Camp Sibert, Gadsden, Alabama

Target of Interest: 4.2-inch mortarsNon-Target of Interest: Native munitions debrisHistorical Use: Training center and airfield for chemical warfareExample of: Physics-based discrimination using magnetic

and electromagnetic survey data2

The first case history concerns work conducted at Former Camp Sibert, located near Gadsden Alabama during 2007. This site was the first of a series of Large Scale Classification Demonstrations being conducted by ESTCP. There are two broad overarching objectives. They are to (1) assess performance of existing and emerging technologies, and (2) investigate the decision making process in cooperation with the regulatory community. The Large Scale Classification program is a multi-year effort involving multiple sites of varying complexityCamp Sibert was selected as the initial site because it has a single target of interest with benign topography and vegetation issues.It was used in the early 1940’s as a training center and airfield for the simulation of chemical air attacks against troops. The site was closed in 1945 and is no longer in active use by the military.The Target of Interest (TOI) at Camp Sibert was a 4.2in mortar.The non-TOI items consist of native munitions debris and cultural clutter.

3

Case Histories


CAMP SIBERT - ObjectiveCAMP SIBERT - ObjectiveReject as much clutter as possibleWithout leaving any 4.2inch mortars unearthed.

3

Here we show photographs of the TOI and non-TOI objects. The 4.2 Inch (107mm) mortar was a US rifled mortar used during the Second World War and the Korean War. It is 4.2 inches in diameter and roughly 1.3ft long. It is a large piece of steel.Examples of the non-TOI at Camp Sibert are also shown here. Representatives of the large, medium and small munitions debris are shown along with some agricultural debris and miscellaneous nails.The objective was to reject as much non-TOI as possible without misclassifying any 4.2inch mortars. Misclassifying a 4.2inch mortar would result in a false negative and represents a classification failure.

4

Case Histories

CAMP SIBERT - DataCAMP SIBERT - Data

EM61‐MK2 Sensor Data

4

Multiple data sets were acquired at Camp Sibert, but for the purposes of this brief we primarily focus on the reconnaissance electromagnetic induction (EMI) and magnetometer data.On the right, we show EM61 MK2 data from Camp Sibert. This is a typical data view. It is a plan map where the colors indicated the magnitude of the EMI measurement at that location. The circles represent anomalies that were selected for analysis and classification by the ESTCP program office. As you can see, the background values are small and consistent across the site. These are clean data. The data were acquired by the Naval Research Laboratory’s (NRL) Multi-sensor Towed Array Detection System (MTADS) EM61 MK2 array. The geophysical data are georeferenced using differential GPS. In addition, the attitude of the sensor is recorded using an auxiliary inertial measurement unit. Data were acquired along survey lines spaced 0.5m apart. Along the survey lines, data were collected every 20cm or so.

5

Case Histories

CAMP SIBERT - DataCAMP SIBERT - Data

Magnetic Sensor Data

4a

Here, we see magnetic data. As discussed earlier, magnetic anomalies have positive and negative lobes. Again, the circles represent targets that were selected for analysis. The data were acquired using the MTADS magnetometer array, shown on the left. Magnetic data were collected along survey lines spaced 0.25m apart. Along the survey lines, data were collected every 15cm or so.

6

Case Histories

CAMP SIBERT – Target SelectionCAMP SIBERT – Target Selection

Depth (m)

0.0 0.5 1.0 1.5 2.0 2.5 3.0

Pea

k S

igna

l (m

V)

1

10

100

1000

10000 most favorable orientationleast favorable orientationtest pit measurementsGPO itemsthreshold

GPO RMS Noise11x depth

EM61‐MK2

14963319 mV, sum of three gatesEM61‐MK2 Cart

14987025 mVEM61‐MK2 Array1499696 nTMagnetometer Array

Seed Targets Detected

Anomalies on Master List

Anomaly Detection ThresholdSensor

Anomaly selection set at 50% of the smallest expected signal amplitude for the 4.2inch mortar at maximum penetration depth (11x diameter or ~1.2m)

The number of total anomalies depends on the sensor and survey combination

All seeded targets were detected

5

Individual anomalies were selected by first determining the smallest expected signal amplitude for the 4.2inch mortar at is least favorable position (viz., maximum depth and horizontal).The target selection threshold was then set to be equal to 50% of this minimum target response. This was done for each sensor independently.First, note that all of the seeded items were detected by each sensor (light yellow).Note also that the number of anomalies is different for each sensor (light blue). Almost 970 anomalies were declared in the magnetometer data compared to only 633 in the EM61 Cart data – a difference of over 330!All of the anomalies were excavated, carefully documented, and archived.

7

Case Histories

CAMP SIBERT – Dig List & Scoring

CAMP SIBERT – Dig List & Scoring

TOI

Non

-TO

I

Anomaly list ranked from highest confidence not‐munitions to highest confidence munitions

number of false positives0 200 400 600

% m

orta

rs c

orre

ctly

iden

tifie

d

0

20

40

60

80

100

high confidence munitionscan't decide - munitions likecan't decide - clutter likehigh confidence not munitions

Threshold

Classification performance evaluated by comparing the number of false alarms versus the percentage mortars correctly identified

Number of non‐TOI

6

For the Sibert Demonstration, we used the Prioritized Dig List and Scoring protocols discussed in the Classification brief.Data analysts from multiple organization used inversion schemes presented earlier to characterize and classify each anomaly.The anomalies were ranked from ‘high confidence non-TOI’ to ‘Can’t make a Decision’ to ‘high confidence TOI’. If reliable attributes could not be extracted from the anomaly data – due perhaps to sensor glitches or data gaps – the anomalies were categorized as ‘Can’t Analyze’. Classification performance is evaluated by plotting the number of non-TOI recovered versus the number of TOI recovered if the targets were excavated in reverse order; that is, the anomalies labeled “can’t analyze” are investigated first, followed by “high-confidence TOI, followed by “can’t decide,” and then “high-confidence non-TOI.”

8

Case Histories

CAMP SIBERT – Magnetic Size Attribute

CAMP SIBERT – Magnetic Size Attribute

● Histogram of the magnetometer‐derived size attribute and analysis results.

Target "Size" from Mag Analysis

Num

ber

0

50

100

150

cluttermortars

number of false positives0 200 400 600

% m

orta

rs c

orre

ctly

iden

tifie

d

0

20

40

60

80

100

high confidence munitionscan't decide - munitions likecan't decide - clutter likehigh confidence not munitions

Threshold

Magnetic Data: All munitions were correctly classified by most vendors40‐70% of the non‐hazardous objects correctly classified

Number of non‐TOI

7

First, let’s look at a target size attributes.Here, we show a histogram of the magnetometer derived target size attribute on the left. For illustration purposes, we show only two classes. Red identifies the 4.2inch mortars, and Green identifies everything else.Clearly, a significant fraction of the clutter is smaller than the 4.2inch mortars so this is a useful attribute. The ROC curve on the right shows classification results using this magnetometer derived size estimate.These are terrific results by any measure. In hindsight, we now know that there were over 700 anomalies caused by non-TOI objects in addition to the roughly 120 TOI in the test set. If classification had not been used, all of the objects would have to be recovered. Using the classification scheme and associated prioritized dig list, we see that only 70 or so non-TOI objects would have been removed in the process of recovering all of the ordnance in addition to the 100 or so anomalies marked “can’t analyze.”

9

Case Histories

CAMP SIBERT – EMI Size Attribute

CAMP SIBERT – EMI Size Attribute

EMI Size-based Feature

Num

ber

0

20

40

60

80

100

120

140

cluttermortars

● Histogram of the EMI‐derived size attribute and comparison of EM61‐MK2 cart and array data analysis results.

EM61‐MK2 Data: All munitions were correctly classified by most vendors40‐50% of the non‐hazardous objects correctly classified

number of false positives0 100 200 300 400 500 600

% m

orta

rs c

orre

ctly

iden

tifie

d

0

20

40

60

80

100

cartarray

Number of non‐TOI

8

Next, let’s look at EMI target size attributes. As before, the histogram on the left clearly shows that a significant number of the non-TOI can be eliminated using only this one attribute.The classification performance of this attribute is also very good. Using this attribute alone, a significant number of the non-TOI, on the order of 40% to 50%, were successfully classified as high-confidence non-TOI without a single false negative.

10

Case Histories

CAMP SIBERT – Target size vs. peak signal

CAMP SIBERT – Target size vs. peak signal

0 200 400 600 8000.0

0.2

0.4

0.6

0.8

1.0

Prioritized by decreasing Signal Strength

Prioritized by Target Size Estimate

Number of non-TOI

Nor

mal

ized

num

ber o

f TO

I165 404

9

Based on the mix of TOI and non-TOI at this sight, it was clear that the size attributes would probably work well – and we have just seen that they did.Here we compare performance results for two prioritization schemes. The first is based on the derived target-size attribute. The second is ordered by decreasing signal strength. Signal strength isn’t entirely unreasonable because, as we’ve seen, signal strength and target size are directly proportional. Depth of burial and target orientation however, also effect the measured signal strength and can confuse the issues.As expected, the list that is prioritized by decreasing signal strength starts out strong. This is seen in the rapidly rising portion of the black curve. It doesn’t finish as strong, however. The last 4.2inch mortar isn’t recovered until 404 non-TOI were needlessly removed.The list prioritized by the EMI-derived target-size attribute, in comparison, doesn’t start strong but does finish strong. Using this attribute for classification purposes, all of the 4.2inch mortars would have been recovered while only extracting 165 non-TOI. This includes the roughly 100 targets categorized as ‘Can’t Analyze’.

11

Case Histories

CAMP SIBERT – Multiple EMI Attributes

CAMP SIBERT – Multiple EMI Attributes

● Combining multiple EMI attributes, namely size and decay rate, improved classification performance for this demonstrator

response amplitude ≈ size-3 -2 -1 0 1 2

deca

y ra

te ≈

wal

l thi

ckne

ss

-3

-2

-1

0

1

2

3

4

5empty hole4.2" mortarpartialbaseplatemunitions debriscultural clutter

number of false positives0 100 200 300 400

% m

orta

rs c

orre

ctly

iden

tifie

d

0

20

40

60

80

100

Full ClassifierSize Only

Number of non‐TOI

10

We have observed good classification performance using a single, simple attribute at this site. Here, we show that the classification performance can be improved by intelligently combining multiple EMI attributes; namely, size and decay rates.

12

Case Histories

CAMP SIBERT – Advanced EM Sensor


● The Berkeley UXO Discriminator (BUD) was deployed over a portion of the entire site and produced a near‐perfect performance.


% m

orta

rs c

orre

ctly

iden

tifie

d

0

20

40

60

80

100

High Confidence Not MunitionHigh Confidence Munition

● Starting from the munitions side, the initial 56 anomalies were munitions.

● The next 6 anomalies were false positives.

● The remaining 203 anomalies true negatives. Number of non‐TOI

11

So far, we have focused on EM and magnetic sensors that are commercially available. In addition to commercially existing sensors, a few emerging sensors were also demonstrated as part of the Camp Sibert Classification Study.Shown here are results for the Berkeley UXO Discriminator (BUD) sensor, which was deployed over a portion of the site. The BUD is a recently developed EMI sensor that consists of multiple coils arranged in a fixed geometry. It will be discussed later in the Emerging Technologies segment.As you can see by the ROC curve, the results are nearly perfect. The classification was based on attributes related to target size and decay rate. First, there were no ‘Can’t Analyze’ or ‘Can’t Decide’ declarations. Of the 62 ‘high confidence TOI’declarations, the initial 56 targets were munitions, and the remaining 6 were false positives. Of the 203 ‘high confidence non-TOI’ declarations, all were true negatives.

13

Case Histories

CAMP SIBERT – Program Conclusions

CAMP SIBERT – Program Conclusions

● Demonstration of successful classification at a simple site● The data were carefully collected and then analyzed

using physics‐based analysis techniques● The target selection criteria were based on the minimum

expected signal strength for the TOI● A number of the data+analysis combinations correctly

classified all munitions● Well over 50% of the detected clutter items were

routinely eliminated with high confidence● The BUD sensor produced near‐perfect results

12

14

Case Histories

Remington WoodsBridgeport & Stratford, Connecticut

Remington WoodsBridgeport & Stratford, Connecticut

Target of Interest: 37mm to 75mm projectilesNon-Target of Interest: Industrial and agricultural clutterHistorical Use: Former munitions testing groundExample of: Classification approach for a site that is

surrounded by residential housing and wooded

`

13

Moving on to the second case history…This site is a former munitions testing ground straddling the towns of Stratford and Bridgeport CT. It is a Brownfield redevelopment site known as Remington Woods.As you can see from the photograph, it is a more challenging environment than Camp Sibert. Generally it is wooded. Where the trees thin, it is vegetated or rocky.The targets of interest at this site include 37mm to 75mm projectiles (105mm are rumored but no real evidence to support the claim has been found).The clutter at the site is generally related to agricultural or industrial activities.

15

Case Histories

● 422‐acre former munitions testing site owned by DuPont

● Safety concerns dictate remote excavation and blast shield containment

● Process is time consuming and costly. Excavation and blast containment costs for a four acre parcel donated for highway improvements was roughly $1,000/target

● Site littered with metal clutter from 100’s of years of use

Remington Woods -Overview

Remington Woods -Overview

14

Remington Woods is owned by DuPont and occupies 422 acres. Dupont and URS Corporation have been systematically clearing the site in phases since 2002.The property is surrounded by residences. Safety concerns dictate remote excavation and blast shield containment. Because of this, the remedial process is slow and costly.

16

Case Histories

Remington Woods - ProcessRemington Woods - Process

Site Clearance● 1: Historical review ● 2: Surface contact removal – thorough sweep of area using Schonstedt to

detect and remove surface contacts to a depth of three inches● 3: EM‐61 Mk II survey to identify contacts

Prioritize contacts based on survey data analysis: (1) possible UXO, (2) uncertain / more info needed, (3) high confidence not UXO

● 4: Cued identificationReacquire category 1 and 2 contacts, collect and process high resolution EM‐61 HH data for target classificationOn basis of EM61‐HH processing, reassign reacquired contacts to category 1, 2 or 3

● 5: Remove final category 1 and 2 targets using remote excavation with blast shield.

15

Because the wooded terrain present problems for georeferencing the geophysical sensors data, a phased approach has been adopted.The geophysical phase of the clearance begins with an EM61 reconnaissance survey to identify anomalies. The EM61 survey used dead-reckoning techniques for spatial registration information. These data undergo a screening process that looks for rapidly decaying signals to identify thin walled items.Following the standard production survey, all anomalies that were not rejected based upon decay rate are resurveyed using an EM61HH sensor using a gridded template.The EM61HH data are characterized and classified using the methods discussed earlier today.The objective is to uniquely identify as much of the anthropic clutter as possible without leaving behind any TOI. At this site, the prioritized dig list allows the stakeholder the option of using a variety of remedial procedures – thereby saving money in aggregate.

17

Case Histories

Remington Woods - Data Collection

Remington Woods - Data Collection

● Identify potential hazards using reconnaissance survey data,

● collect additional data over targets using a template,

● extract attributes & classify

0.25m16

Upper left photograph – This photograph is the EM61 sensor during the reconnaissance survey. The resulting data are used to locate anomalies for further investigation.Upper right photograph – This is a photograph of the EM61HH sensor in operation using a gridded template. In this mode, a wooden template is used to precisely position the sensor at 0.15m intervals.

18

Case Histories

Remington Woods - Classification Approach

Remington Woods - Classification Approach

Classification Approach1. Signal decay rate over four time

gates of EM61 MkII can be used to identify thin‐walled (<1/16 inch) scrap metal and wire

2. High resolution EM61‐HH data grid over contact inverted to estimate target size and shape and match to possible UXO items

Industrial/agricultural clutter

Small to medium caliber projectiles

17

Small to medium sized projectiles are the targets of interest at this site.Small to medium industrial and agricultural clutter represent the vast majority of the non-TOI objects recovered to date. Due to its 100-year long history, a number of horseshoes are recovered. Almost without exception, the non-TOI items are shallowly buried.As alluded to earlier, the classification approach has two steps. First, signal decay rates over the four time gates of the EM61 MKII sensor are used to detect and eliminate thin-walled objects. Next, EM61HH data are acquired using a gridded template over the remaining anomalies. Using a template, the measurements are precisely located with respect to each other but not referenced in any other way. Collecting data using a gridded template allows us to minimize spatial registration problems. The EM61HH data are then analyzed to extract attributes and classify.

19

Case Histories

Remington Woods – Classification Step 1


Screening Sheet metal from Survey Data• EMI signal decays more quickly in sheet metal scrap than in UXO• Target identified as scrap metal if ratio of EM61 Mk II signal in last time gate to signal in first time gate is less than 1/10

Since 2002, the percent rejected varies across the site: average 36% (ranges from 8% to 56%)

Late/Early Decay

Training Data

18

This slide summarizes the screening process applied to the reconnaissance EM61 data. The graph plots the Late to Early Decay rate as measured for various test items of varying wall thickness. Thin-walled objects are characterized by a rapid decay. The ratio of late to early responses, therefore is small. Objects with wall thicknesses of 2mm or more, as is the case with TOI at this site, decay more slowly and therefore generate a larger ratio.Since, 2002, this approach has allowed us to reject an average of 36% of the total anomalies.

20

Case Histories



● Classification based on size and symmetry attributes

● EM61‐HH grid data inverted for polarizability Attributes

Size‐based AttributeSy

mmetry‐based Attribute

Size

Symmetry

Unknown

19

The plot on the right presents the decision space. The plotted values are the principle axis polarizations. On the x-axis, we plot an attribute related to the targets size. On the y-axis, we plot an attribute related to the inverted axial symmetry. Specifically, we use the secondary and tertiary polarizations. A target’s size is reflected in this feature space by where it lies along a diagonal. Axially asymmetry is indicated by the vertical line length.The classification process is based on these attributes. Contrary to Sibert where there was a single TOI, at this site there multiple targets of interest. To derive a classification decision, we essentially evaluate the distance of the attributes for the unknown target to that observed for our training data.

21

Case Histories

Remington Woods –Performance Summary

Remington Woods –Performance Summary

● Active Program since 2002223 acres surveyed to date12,700+ contacts in EM61 reconnaissance survey~4,100 EM61 contacts ruled out (decay‐based)~3,600 EM61HH contacts ruled out (size and symmetry)~3,400 eliminated during cued collection (surface clutter found)

● Net result: ~1,600 contacts not classified as high‐confident non‐TOI● Number of UXO found: 13

(no classification failures)● Client‐estimated cost savings on

excavation and blast containment in 2003 was ~$1M

20

This project has been active since 2002 and has been quite successful.Roughly half of the total acreage has been surveyed and remediated.In total, more than 12,700 anomalies were identified. Of these, ~4,100 were ruled out during the screening process. Approximately 3,600 of the remaining anomalies were ruled out based on the classification analysis of the EM61HH data. Another 3,400 anomalies were eliminated because surface items were discovered during collection of the cued data.The net result – all but 1,600 anomalies were eliminated and do not require expensive intrusive actions.DuPont remediate’s a significant fraction of the eliminated, ‘high confidence non-TOIs’.A total of 13 munitions have been recovered with no classification failures.Cost savings, which are realized by changing the remediation process according to the final classification, are significant. They approached $1M in 2003 - the last time cost savings were estimated by DuPont.

22

Case Histories

Classification in Action…Classification in Action…

“Only 462 of the contacts were possible unexploded ordnance, though, and are being excavated”…

10% of the remaining objects will be excavated “to verify that they are what we say they are”

October 28, 2008

21

This article was recently published in a local Connecticut newspaper. It summarizes the project and illustrates the public nature of this type of program. Inherent in the text is the acceptance of the adopted classification process…

1

Future Sensors: System Design and Classification Implications

Future Sensors: System Design and Classification Implications

Thomas Bell

Future Sensors 2

OutlineOutline

● Electromagnetic Induction (EM) fundamentals● Limitations of current commercial technology● Next generation SERDP/ESTCP technology

UXO‐specific featuresExamples

2

Future Sensors 3


(A) Abrupt change in primary field excites eddy currents in buried object.

(B) Eddy currents diffuse throughout the object and decay.

Details depend on the size, shape and composition of the object.

A typical EM sensor measures the EM field associated with the decay of eddy currents induced in metal objects near the sensor. The eddy current decay occurs after the current pulse in the transmitter loop is completed, and hence after the response directly caused by any magnetization of material near the sensor. Unlike magnetometers, EM sensors respond to all types of metal objects, not just ferromagnetic ones.

3

Future Sensors 4

Commercial EM TechnologyCommercial EM Technology● Widely used for detection surveys. Single axis coil sensor

requires spatially mapped data to determine target parameters for classification.

DGM survey

towed array

grid/template

The most widely used EM sensor for munitions detection is the Geonics EM61. The EM61 is a single-axis coil sensor and so data collected with the EM61 must be spatially mapped before it can be processed to determine target parameters for use in classification. The pictures show the EM61 in various modes of operation: on a gridded template to collect precisely positioned data over a previously located target, in the conventional wheeled mode for digital geophysical mapping (DGM) survey work, and as a vehicle-towed sensor array. The inset plot next to the standard wheeled EM61 shows a sample of survey data. The measurements are indicated by color coded dots. The background level is blue green and the red area shows signals due to a buried piece of metal.

4

Future Sensors 5

Factors Affecting Performance

Factors Affecting Performance

● Limited capability for target classification in survey modeAnalog smoothing distorts signal shapeLimited decay time coverageCentimeter‐level sensor positioning uncertainty degrades target parameter estimates

● Towed arrays have limited target illumination with transmitters operated simultaneously. Reduced data rate otherwise.

● Some success for cued target ID when used to collect static data with grid template

● EM63 (extended decay time coverage) had very good classification performance in cued template mode in Camp Sibert Classification Study

The EM61 has limited capability for target classification when used in the survey mode: smoothing that occurs during data acquisition to improve the signal-to-noise ratio distorts the signal shape, there is limited decay time coverage, and centimeter-level positioning uncertainty degrades target parameter estimates. Towed arrays can improve the positioning accuracy. However, if the different transmitters are pulsed synchronously then the primary fields merge together and do not excite the target from different directions, while if they pulse sequentially then the data rate is reduced. The EM61 has had some success when used for target classification in a cued identification mode, collecting precisely positioned static data on a griddedtemplate placed over the target. The Geonics EM63 is similar to the EM61, but has extended decay time coverage. It had very good classification performance when it was used in a cued template mode in the ESTCP Camp Sibert Classification Study.

5

Future Sensors 6

New EM TechnologyNew EM Technology● New UXO‐specific EM

technologies are being developed and tested under SERDP & ESTCP

● All digital electronics, measuring complete eddy current decay cycle

● Multi‐axis target excitation and observation for complete interrogation of principal axis polarizabilities.

SERDP and ESTCP are developing and testing new munitions-specific technologies which avoid these problems and should provide significantly improved discrimination performance. Several of the new systems are shown in the pictures. They have all digital, programmable electronics and are capable of measuring the complete eddy current decay cycle. They provide multi-axis target excitation and observation for complete interrogation of the principal axis response functions.

6

Future Sensors 7

Classification ProcessClassification Process


3. Classification

1. Data Collection

The new EM technologies provide improvements at each stage in the classification process

There are three stages in the classification process. They are illustrated schematically in this slide. The first stage is data collection over an object, illustrated on the left by a picture of one of the new sensor systems stationed over a target to collect data. In the second stage, attributes or features of the EM response that relate to physical properties of the object are extracted from the data. The center figure shows a principal axis decay curve extracted from data collected over an object, along with a parametric fit to the curve using a physics-based response model. Finally, the object is classified by deciding whether the set of attributes is more like those typical of munitions or those typical of clutter items. The illustration on the right is a scatter plot two of the target attributes extracted from measurements of various mortar fragments (blue symbols) and intact mortars (red symbols).

7

Future Sensors 8

Data CollectionData Collection

In order to observe the complete EM response pattern the object must be excited and measured from all directions. The new technologies accomplish this with multi‐axis coil sensors or single axis coil arrays.

Multi‐axis coil array Single axis planar array

In order to adequately sample the complete EM response pattern the object must be excited and observed from all directions. The new technologies accomplish this with multi-axis coil sensors like that illustrated in the drawing on the left, or with single-axis coil arrays like that shown on the right.

8

Future Sensors 9

Target Attribute EstimationTarget Attribute Estimation

Goal is to excite and measure object from all directions. Then the fundamental response functions (principal axis polarizabilities) can be extracted by inverting the set of EM data using the dipole response model.

{ } 1B −⋅μ= )t(CIC)t(V TR0

If the target is excited and measured from a broad range of angles, then the fundamental response functions (principal axis polarizabilities) can be extracted by using a standard dipole response model to invert the set of EM measurements. The process is illustrated schematically in the flow diagram on this slide. The sheaf of graphs on the left represents the set of measured induced field decay curves. The equation in the middle represents the inversion process that operates on the data collected over the target. The graph on the right shows the principal axis response functions determined by the inversion process.

9

Future Sensors 10


Time decays of the three principal axis polarizabilities are the EM signature of an object and are the basis for classification.

The set of three principal axis responses constitutes the basic EM signature of an object and is the basis for classification. The two figures at the bottom of this slide show the basic idea. Each is a plot of principal axis response curves extracted from data collected over the object shown in the corresponding inset picture: a 105 mm projectile on the left and a comparably sized tractor muffler on the right. At early times the responses are similar, but they evolve differently at later times. Two of the principal axis responses for the munitions item are the same because of the object's axial symmetry. All three of the principal axis responses are different for the muffler. This is expected for irregularly shaped clutter items.

10

Future Sensors 11

New TechnologiesNew Technologies● Multi‐axis coil sytems

Berkeley UXO discriminator (BUD)Metal MapperUSGS ALLTEM

● Single axis coil arraysNRL TEM array

ALLTEM

TEM Array

Metal Mapper

BUD

New EM systems that have undergone testing in ESTCP are illustrated on this slide. The Berkeley UXO Discriminator (BUD), the USGS ALLTEM and the Geometrics Metal Mapper are all multi-axis coil systems, while the NRL time domain EM (TEM) system is a planar array of single axis coils.

11

Future Sensors 12

Berkeley UXO DiscriminatorBerkeley UXO Discriminator● Multiaxis coil system – Operates in survey mode

(detection only) and cued (discrimination) mode. In survey mode, once a target is detected, stop and switch to discrimination mode.

● Excellent performance in Camp Sibert Classification Pilot Program (416 anomalies: 100% PD @ 10.8% PFA)

The Berkeley UXO Discriminator (BUD) is shown in the picture on the left at the bottom of this slide. It is a multi-axis coil system. The coil configuration is shown in the diagram on the right. It can operate in a survey mode for detection only. Once a target is detected, the operator stops and switches over to the discrimination mode to collect multi-axis data for classification. The BUD had excellent performance at the ESTCP Camp Sibert Classification Study. Using data collected at 416 anomalies, the BUD correctly identified all munitions items as munitions, and incorrectly identified only ~10% of the clutter items as munitions.

12

Future Sensors 13

BUD Examples from Camp SibertBUD Examples from Camp Sibert

This slide shows examples of principal axis response functions calculated from BUD data collected at Camp Sibert. The graph on the left shows the three polarizabilitiesfor an intact 4.2 inch mortar (shown in the bottom left picture), while the graph on the right shows the polarizabilities for the large mortar fragment shown in the picture bottom right picture. Both objects were correctly classified based on the calculated polarizabilities.

13

Future Sensors 14

Metal MapperMetal Mapper

● Geometrics, G&G Sciences and Snyder Geoscience● Multi‐axis Tx coils, multiple small 3‐axis Rx● Survey (detection/location) and cued static classification

modes● APG Standardized Test Site demo September ʹ08

This slide shows the Metal Mapper system developed by Geometrics in collaboration with G&G Sciences and Snyder Geoscience. The picture shows the Metal Mapper being towed over the Yuma Proving Ground test field. The diagram on the right shows the multi-axis coil configuration. Like the BUD, the Metal Mappercan operate in either a detection/location survey mode or a static classification mode. It underwent testing at the Aberdeen Test Site in September 2008.

14

Future Sensors 15

MM Examples from YPG CalMM Examples from YPG Cal

This slide shows examples of principal axis response functions calculated from Metal Mapper data collected in the Calibration Area at Yuma Proving Ground. The graph at the upper left shows the three polarizabilities for a steel sphere. All are equal as expected. The upper right and lower left figures are for a 105 mm HEAT round and a 60 mm mortar, respectively. Both show the expected behavior for munitions wherein the two weaker polarizabilities are equal due to the axial symmetry of the objects. The lower right figure is for a fragment from an exploded 155 mm projectile. The three distinct polarizabilities are consistent with the irregular shape of the munitions fragment.

15

Future Sensors 16

ALLTEMALLTEM

● USGS and Colorado School of Mines● Multi‐axis Tx/Rx coils, sampled at ~3 Hz in survey mode● Triangular Tx waveform – includes on‐time induced field

response● 2005 & ‘06 tests at YPG

ALLTEM is a collaborative development of the US Geological Survey (USGS) and the Colorado School of Mines. It is unique among the new SERDP/ESTCP systems in that it uses a continuous triangular waveform, and therefore measures the magnetic response while the primary field is exciting the target in addition to the eddy current response. The ALLTEM is shown behind a tow vehicle in the picture on the left, and in an expanded view on the right, showing the locations of the various transmit and receive coils. It can operate in a survey mode with limited response sampling for use in classification, and has undergone testing at Yuma Proving Ground.

16

Future Sensors 17

ALLTEM Examples from YPGALLTEM Examples from YPG

This slide shows examples of ALLTEM data. The graph in the upper right shows one cycle of the ALLTEM waveform (blue line) along with responses to a MK 118 rockeye (red line) and a 60 mm mortar (green line). The rockeye (bottom picture) is aluminum. Being nonmagnetic, it has a distinctively different response than does the steel 60 mm mortar pictured above the rockeye. The plot on the left is a survey map of the Calibration Area at Yuma Proving Ground that was created using gridded data (at one point in the waveform cycle) collected with the ALLTEM. The system is showing a good, strong response from all of the calibration targets.

17

Future Sensors 18

NRL TEM ArrayNRL TEM Array

● NRL, G&G Sciences, Nova Research and SAIC● 2D array of 25 time domain EMI sensors, decay times

from 0.04 to 25 msec● APG Standardized Test Site demo June ʹ08, > 200

targets/day

The time domain electromagnetic (TEM) array developed by the Naval Research Laboratory (NRL) in collaboration with G&G Sciences, Nova Research and Science Applications International Corporation (SAIC) is shown on this slide. The photograph at the lower left shows the array behind the NRL tow vehicle, and the drawing at the right shows the layout of the array elements. There are 25 transmit/receive coil pairs. The TEM array can measure induced eddy current decay from 0.04 msec to beyond 25 msec after the primary field pulse. It performed very well in a demonstration at the Aberdeen Proving Ground Standardized Test Site in June 2008. The system operates in a cued interrogation mode, collecting data while parked stationary over a target. At Aberdeen it was able to interrogate over 200 targets per day.

18

Future Sensors 19

TEMTADS Examples from APGTEMTADS Examples from APG

This slide shows principal axis responses calculated from TEM array data at the Aberdeen Proving Ground Calibration Area. The plots on the left show polarizabilities for a 60 mm mortar (top) and a 105 mm HEAT round (bottom). As expected, the two secondary polarizabilties are equal for the munitions items. On the right are plots for correspondingly sized clutter items. The response curves for the clutter items are distinctly different than those for the munitions items. Inset pictures on each plot show the targets.

19

Future Sensors 20

Improved EM TechnologiesImproved EM Technologies

● Classification performance of conventional technology (e.g. EM61) is limited by two primary factors

The eddy current decay cycle is not fully capturedMulti‐cm positioning errors inherent to field survey work compromise the accuracy of dipole inversion and estimation of target attributes

● New UXO‐specific technologies which avoid these problems are being developed and tested under SERDP and ESTCP

Results to date are very encouraging

In summary, classification performance of conventional, commercially available EM technology is limited by two primary factors. First, the complete eddy current decay cycle is not captured with the sensors typically used to detect buried munitions, and second, field survey work inevitably introduces centimeter scale or larger positioning errors in the data, which compromises the accuracy of dipole inversion and the estimation of target attributes from the data. New munitions-specific technologies which avoid these problems are being developed and tested under SERDP and ESTCP, and the results to date from testing of these systems are very encouraging.

1

Concluding ThoughtsConcluding Thoughts

Herb Nelson

2

Concluding Thoughts

Objective of the CourseObjective of the Course

Provide a tutorial on the sensors, methods, and status of the classification of military munitions using geophysical methods

•Advanced processing of data collected with existing commercial instruments

•Promising results from emerging optimized systems

2

As we said in the Introduction, SERDP and ESTCP have supported a number of investigators over the years who have

• developed processing approaches to extract target-specific attributes from data collected by commercial geophysical sensors, and

• demonstrated advanced sensors designed with the munitions response problem in mind.

These research efforts have resulted in an impressive ability to classify the source of geophysical anomalies as “targets-of-interest” or non-hazardous items under simple conditions with the promise of expansion to a wider range of conditions as the newest sensors mature.

This course was intended as a tutorial on these classifications methods. We began with a brief introduction to some of the terminology and concepts that were used, introduced the basics of the two primary geophysical instruments used in munitions response, discussed the methods used for classification and illustrated them with two case studies, previewed the next generation of EM sensors emerging from the research program, and are now concluding with a brief summary of the important results and presentation of a idealized cost model for classification.

3

Concluding Thoughts

05

101520253035404550

SiteAssessment

Survey andMapping

VegetationRemoval

Scrap MetalRemoval

UXO Removal &Disposal

Cos

t -$B

Direct CostIndirect Cost

Munitions Response Cost BreakoutMunitions Response Cost Breakout

Defense Science Board Task Force on UXO3

As we saw, this chart, from the 2003 report of the Defense Science Board Task Force on UXO [http://www.acq.osd.mil/dsb/reports/uxo.pdf], shows us the economic driver for these methods. On a typical munitions clean-up project, an overwhelming fraction of the money is spent removing non-hazardous items from the site. If a method can be devised to identify these non-hazardous items and remove them with fewer safety precautions or leave them in the ground, this money could be transferred to other projects.

4

Concluding Thoughts

CAMP SIBERT – Commercial Sensor

CAMP SIBERT – Commercial Sensor

number of false positives0 100 200 300 400 500 600

% m

orta

rs c

orre

ctly

iden

tifie

d

0

20

40

60

80

100

cartarray

EM61‐MK2 Data: All munitions were correctly classified by most vendors40‐50% of the non‐hazardous objects correctly classified using available software

Number of non‐TOI

4

We saw that, on a simple site, data collected with a commercial instrument (EM61-MK2) and analyzed using freely available software (the UX-Analyze module for Oasis montaj) can lead to impressive results. In this demonstration, we decreased the survey line spacing to 0.5 m (rather than the 1 m typically used for detection surveys) but otherwise used industry standard survey practices. The analyst was able to correctly label more than 50% of the anomalies as being high-confidence non-hazardous without missing any of the mortars.

5

Concluding Thoughts




% m

orta

rs c

orre

ctly

iden

tifie

d

0

20

40

60

80

100

High Confidence Not MunitionHigh Confidence Munition

Number of non‐TOI

LBNL BUD: All 56 munitions were correctly classified and 203 of the 209 remaining anomalies were correctly classified as non‐hazardous.

5

The next-generation EMI system (LBNL BUD) did even better. The additional information contained in the signals from this sensor allowed the analyst to achieve nearly perfect results. The classification was based on attributes related to target size and decay rate. First, there were no ‘Can’t Analyze’ or ‘Can’t Decide’declarations. Of the 62 ‘high confidence TOI’ declarations, the first 56 targets on the list were munitions, and the remaining 6 were false positives. Of the 203 ‘high confidence non-TOI’ declarations, all were true negatives.

This performance leads us to expect good classification performance on more difficult sites when the emerging systems can be used.

6

Concluding Thoughts

Cost ConsiderationsCost Considerations

Total Number of Digs

Cum

ulat

ive

Pro

ject

Cos

t ($)

dig allwith full costs

6

Now, let us consider a simple cost model to illustrate the power of classification.

In the traditional approach, there is some fixed cost to collect and analyze the data. This is represented by the star on the cost axis. Included in this cost is site prep, data collection and analysis, dig list preparation, and anomaly reacquisition. Then, there is some cost to excavate each anomaly on the dig list. The cumulative costs grow as anomalies are dug with a slope equal to the per dig cost.

7

Concluding Thoughts


Number of "Wasted" Digs

% m

orta

rs c

orre

ctly

iden

tifie

d

0

20

40

60

80

100

high confidencemunitions

can't decide

high confidencenot munitions

Threshold

6a

We have seen however, that even with standard sensors and advanced analysis we can confidently classify some anomalies as arising from non-hazardous items. The ROC curve presented here shows the anomalies classified into three categories with the threshold set such that all the items on the right are high confidence non-munitions.

8

Concluding Thoughts



Cum

ulat

ive

Pro

ject

Cos

t ($)


costs to deal with munitions

costs to deal with clutter

6b

Looking back at our simple cost model, we see that the all money spent on anomalies to the left of the threshold was money spent to remove munitions and items for which we weren’t able to make a decision. The rest of the spending (to the right of the threshold) went to remove items that were correctly classified as non-hazardous from the site.

9

Concluding Thoughts



Cum

ulat

ive

Pro

ject

Cos

t ($)

dig allwith full costsdig allwith full costs

leave clutterin the ground


leave clutterin the ground

dig clutterwith lower cost

costs to deal with munitions

costs to deal with clutter

6c

What are the implications of using classification?

There is some incremental cost to collect the data required to make classification decisions and analyze the extra data. This is represented by the red star on the cost axes. For this example, I have chosen 50% higher fixed costs. The site prep and mobilization costs are the same but the data collection and analysis costs are higher.

The digs up to the threshold cost the same in both models so the classification line rises with number of digs with the same slope as the line representing the traditional method. But, once we get to our decision threshold, we begin to realize the savings. The site team can either decide to leave all the remaining items in the ground (in which case there is no more spending) or dig those items with fewer safety measures which lowers the cost per dig and lowers the slope of the line.

For any actual site, the savings will depend on the data collection costs, the average dig costs for potentially hazardous items, how the site team decides to handle the items to the right of the threshold, etc.

Introduction to Classification Methods for Military Munitions...

Documents

Transcript of Introduction to Classification Methods for Military Munitions...