FEATURE AUGMENTATION IMPROVES ANOMALOUS CHANGE DETECTION FORHUMAN ACTIVITY IDENTIFICATION IN SYNTHETIC APERTURE RADAR IMAGERY

Hannah J. Murphy? Christopher X. Ren † Matthew T. Calef?†

? New Mexico Institute of Mining and Technology, Socorro, NM, USA†Intelligence and Space Research Division, Los Alamos National Laboratory, Los Alamos, NM, USA

?† Descartes Labs, Inc. Santa Fe, NM, USA

ABSTRACT

Anomalous change detection (ACD) methods separate com-mon, uninteresting changes from rare, significant changes inco-registered images collected at different points in time. Inthis paper we evaluate methods to improve the performanceof ACD in detecting human activity in SAR imagery usingoutdoor music festivals as a target. Our results show that thelow dimensionality of SAR data leads to poor performanceof ACD when compared to simpler methods such as imagedifferencing, but augmenting the dimensionality of our inputfeature space by incorporating local spatial information leadsto enhanced performance.

Index Terms— Anomalous Change Detection, SyntheticAperture Radar, Human Activity

1 IntroductionQuantifying and detecting human activity in remote sensingimagery has important applications in the coordination of hu-manitarian relief in cases of natural disasters or conflict situ-ations and can provide crucial insights into conditions on theground. In recent years, many machine learning applicationshave been developed for the automation of this problem how-ever these applications often require large amounts of labelleddata [1].

In this work we investigate the applicability of an un-supervised method, anomalous change detection (ACD), tothis issue. We also investigate how feature augmentationmethods can improve the performance of ACD on this task.Although ACD methods have been applied to optical [2] andmulti-modal imagery [3], this work seeks to evaluate theireffectiveness for the specific task of identifying human ac-tivity within synethatic aperture radar (SAR) imagery. Themotivation behind this is that SAR imagery is largely unaf-fected by weather and cloud occlusion, eliminating many ofthe pervasive changes which are known to cause variation inoptical data, and making it an appealing platform for changedetection. Furthermore Sentinel-1, the satellite constellation

contact: cren@lanl.gov, LA-UR-18-20166

considered in this work, has a revisit rate of 12 days, provid-ing a large amount of potential data on which to apply changedetection. As a proxy for refugee camps, we consider musicfestivals, in particular Burning Man [4] from 2016, Bonna-roo [5] from 2017, and Coachella [6] from 2016. We suggestthat these outdoor music festivals will have simliar signaturesto refugee camps, and as such serve as useful examples ofoutdoor human activity consisting of temporary buildings andlarge numbers of occupants. We assess the effectiveness ofa number of methods in classifying pixels as changed due tothe introduction of the music festival. Our finding is that aparticular ACD method, hyperbolic ACD (HACD) [2], whenused with an appropriate feature vector, performs as well as,or better than, the other methods we considered.

2 Data

The Sentinel-1 A and B satellites, launched in the spring of2014 and the spring of 2016, collect imagery with a roughlyfifteen meter ground sample distance (GSD). We use theground range detected (GRD) imagery from Sentinel-1, andin particular the VV channel. This channel is the verticallypolarized component of the reflected signal, where the trans-mitted signal was also vertically polarized. The 2016 BurningMan festival took place in the Black Rock Desert of Nevada.The relevant features of the event are that the encampmentwas large at nearly two and a half miles across, as shown inthe left image of the first row of Figure 1. It is significantthat there was very little change other than the introductionof the encampment. Finally, the background is fairly simpleconsisting of a valley floor with very little backscatter, andeast facing mountains with relatively high backscatter.

The 2017 Bonnaroo music festival took place at the GreatStage Park near Manchester Tennessee and is shown in thesecond row of Figure 1 . The festival introduces vehicles,camping, and several large stages, with vehicles were parkedin a less structured way than at Burning Man. Unlike Burn-ing Man, the image pair contains some significant coincidentchanges, in particular changes in backscatter in what appear

arX

iv:1

912.

0353

9v2

[cs

.CV

] 1

5 M

ay 2

020

to be plots of farmland.The 2016 Coachella music festival takes place at the Em-

pire Polo grounds in Indio, California, as shown in the bot-tom row of Figure 1. While the background changes littlebetween the two images, it is quite varied. The scene in-cludes the Coachella music festivals, a remarkable number ofgolf courses, residential areas, farmland and mountains. Wenote that for all three datasets the activity of interest ( truepositives) take place within the red bounding boxes, howeverwe apply our change detection algorithms to the entire scenesshown in Figure 1.

3 Experiments

3.1 Performance evaluationWe use receiver operating characteristic (ROC) curves to as-sess algorithm performance. In this work the sensitivity pa-rameter is the threshold for the degree of anomalousness as-signed to a change at a pixel for that pixel to be marked as partof an anomalous change. We created masks for the region cor-responding to where the music festivals occurred, and usedthese as our ground truth. Ambiguity in the precise bound-aries of the music festivals leads to two masks: the first indi-cating a region that is entirely within the music festival anda second mask that contains the entire music festival – in theabsence of ambiguity the masks are the same. In such am-biguous cases, we present two ROC curves, and the enclosedband suggest the neighborhood of the true ROC curve. OurROC curves are presented as log-log plots, providing a meansfor the reader to assess low false-positive rate behavior whichis the region of interest for ACD algorithms.

3.2 Image DifferencingAn intuitive approach to change detection is to subtract twoco-registered images pixel-by-pixel. However imagery col-lected from optical systems can include pervasive differencesdue to changes in weather or illumination between the twocollections. In this case, image differencing can give signif-icant non-zero values that do not reflect changes of interestBecause the common sources of pervasive changes are largelyabsent in SAR imagery we use image differencing as a base-line for performance. Throughout the rest of the paper, the im-age differencing ACD performance baseline will be depictedas blue curve.

3.3 Hyperbolic ACDA central idea behind HACD is to approximate the joint dis-tribution of pixel values in the two images as Gaussian. Weshall use P12 to denote this distribution, and then P12(X,Y )is the probability density associated with a pixel having valueX in the first image, and having value Y in the second. Weshall use P1 and P2 to denote the marginal distributions ofpixel values for each image. Given these distributions the

anomalousness of a pixel change from value X to value Yis assigned the value

− logP12(X,Y )

P1(X)P2(Y ). (1)

This quantity is high when P12(X,Y ) is small and the prod-uct of P1(X) and P2(Y ) is large. In simple terms, anoma-lousness is high for uncommon changes between commonvalues. By approximating P12 as Gaussian, anomalousnessbecomes quadratic and the coefficients are the covariances.

Applying HACD to the per-pixel intensity values in ourGRD imagery produces the ROC curves shown in Figure 2a.Interestingly, there is not much benefit to HACD over imagedifferencing. We suggest this is due to a dimensionality is-sue in the case of SAR data: many optical platforms providemultiple bands per pixel, thereby increasing the dimension ofthe input space, and providing more opportunities to separateanomalous from pervasive changes.

3.4 Patch-HACDWith this as motivation we consider ways to increase the di-mension of the input space. As a simple first step we considera neighborhood around each pixel as input. We use an eleven-by-eleven patch and form a feature vector for each pixel basedon the intensity values in the patch centered at that pixel, ef-fectively generating new ’channels’ to apply HACD to. Weterm this approach Patch-HACD. In Figure 2b one can seethat, with the exception of the low false-positive regime forBurning Man, this method performs better than image dif-ferencing and HACD, indicating that augmenting the featurespace can indeed improve the performance of HACD for thisapplication.

3.5 Gray-level Co-occurrence Matrix-HACDIt is a reasonable hypothesis that forming a feature vectorfrom a patch, as we did in section 3.4, encodes more informa-tion than is necessary to discriminate pervasive changes fromchanges due to the introduction of the music festival. We thusconsider an alternative and more parsimonious encoding fora patch: our hypothesis is that intensity variations in regionscontaining music festivals are larger than in other regions andas such consider image texture as a method to capture thisdifference. We consider the gray-level co-occurrence matrix(GLCM) [7], a statistical parametrization of texture whichconsiders how often pairs of pixels with a specified spatial re-lationship occur with specific values. The GLCM has beenused effectively to analyse texture in SAR imagery [8].

Here we form the GLCM for each patch and use that as theper-pixel feature vector. The corresponding ROC curves areshown in Figure 2c. We see that GLCM-HACD is compara-ble in performance to Patch-HACD. Performance at BurningMan is markedly better, but performance decreases slightlyfor the Bonnaroo test case. This may indicate that the textureof the festival is closer to those of the pervasive changes in

Fig. 1. The music festivals studied in this work: Burning Man [4], Bonaroo [5] and Coachella [6], with Sentinel-1 VV imagestaken before and during the festivals. Red boxes outline areas where human activity of interest takes place in the images.

the scene for the Bonnaroo test case, however it is importantto note that there are several parameters that one must choosewhen forming a GLCM such as the number of discrete inten-sity levels. As such the results we present here can be consid-ered a lower bound for best performance of GLCM on the testcases we consider, and can potentially be ameliorated throughcareful hyperparameter search.

4 ConclusionsWe report that utilising HACD to detect outdoor human activ-ities in our test cases outperformed image differencing only inthe cases where dimensionality augmentation such as utilisinga local image patch for each pixel or calculating the GLCMfor said image patch was used. This suggests that the combi-nation of dimensionality augmentation and ACD algorithms

can be a powerful tool for detecting human activity in SARimagery, a data modality where it is currently not often used.An interesting extension of this study would be to evaluate theperformance of these methods for detecting similar changesin multi-modal imagery, i.e combining optical and SAR im-agery.

5 References[1] John A Quinn, Marguerite M Nyhan, Celia Navarro,

Davide Coluccia, Lars Bromley, and Miguel Luengo-Oroz, “Humanitarian applications of machine learn-ing with remote-sensing data: review and case study inrefugee settlement mapping,” Philosophical Transactionsof the Royal Society A: Mathematical, Physical and En-gineering Sciences, vol. 376, no. 2128, pp. 20170363,

Fig. 2. Log-log ROC curves for: (a) HACD (b) Patch HACD on an 11 by 11 pixel patch around each pixel (c) GLCM - HACDfor an 11 by 11 pixel patch around each pixel (d) GLCM - RF for an 11 by 11 pixel patch around each pixel for the three testcases

2018.

[2] James Theiler and Simon Perkins, “Proposed frameworkfor anomalous change detection,” in ICML Workshop onMachine Learning Algorithms for Surveillance and EventDetection, 2006, pp. 7–14.

[3] Amanda K. Ziemann, Christopher X. Ren, and JamesTheiler, “Multi-sensor anomalous change detection atscale,” in Algorithms, Technologies, and Applicationsfor Multispectral and Hyperspectral Imagery XXV, Balti-more, United States, May 2019, p. 37, SPIE.

[4] Wikipedia.org, “Burning Man,” 2017.

[5] Wikipedia.org, “Bonnaroo Music Festival,” 2017.

[6] Wikipedia.org, “Coachella Valley Music and Arts Festi-val,” 2017.

[7] Robert M Haralick, “Statistical and structural approachesto texture,” Proceedings of the IEEE, vol. 67, no. 5, pp.786–804, 1979.

[8] L-K Soh and Costas Tsatsoulis, “Texture analysis of SARsea ice imagery using gray level co-occurrence matrices,”IEEE Transactions on geoscience and remote sensing,vol. 37, no. 2, pp. 780–795, 1999.