Energy-Aware Time Change Detection Using Synthetic ...€¦ · Energy Aware Time Change Detection...

Energy-Aware Time Change Detection Using Synthetic Aperture Radar On High-Performance Heterogeneous

Architectures: A DDDAS Approach

Sanjay Ranka (PI)

Sartaj Sahni (Co-PI) • Mark Schmalz (Co-PI)

Anand Ranagarajan (Consultant)

University of Florida • Department of CISE

Gainesville, FL 32611-6120

DDDAS Program • PI Meeting • 06 Sep 2017

Energy Aware Time Change Detection Using Synthetic Aperture Radar On High-Performance Heterogeneous Architectures: A DDDAS Approach

PI: Sanjay Ranka, Ph.D. -- University of Florida Department of CISE

Overview of Presentation

1. Research Team

2. Technical Objectives

3. Technical Approach & Results• Reconstruction

• Energy and Power Reduction

• Video SAR Simulation

• Coherent Change Detection

4. Ongoing and Future Work

5. Discussion

Energy-Aware Time Change Detection in SAR - Ranka (PI)DDDAS PI Meeting - 27 Jan 2016 2

Air Force Relevance:

High-Performance Computing using multi-resolution SAR processing technology.

Surveillance – Efficient reconstruction of SAR imagery from multiple pulse history using “green” multi-resolution approach.

Target Detection / Recognition – Power-efficient multiresolution Change Detection algorithm for reconstructed video SAR, yielding reduced power consumption as a result of multi-resolution processing.



Research Team

Principal Investigator Sanjay Ranka, Ph.D.• Research Interests: High-Performance Computing

Energy-Aware Computing

Big Data Analytics

Co-PI Sartaj Sahni, Ph.D.• Research Interests: High-Performance Computing

Data Structures and Algorithms

Signal & Image Processing

Co-PI Mark Schmalz, Ph.D, O.D.• Research Interests: High-Performance Computing

Signal & Image Processing

Simulation, Error Analysis

DDDAS PI Meeting - 27 Jan 2016 3Energy-Aware Time Change Detection in SAR - Ranka (PI)

Students: Adeesha Wijayasiri,

Xiaohui Huang



Technical Objectives

Develop Energy-Efficient Algorithms for Change Detection in Video Synthetic Aperture Radar (SAR) Imagery

Topics of Investigation

• Heterogeneous Parallel Architectures (CPUs, GPUs, HMPs)

• Adaptive Algorithms for Image Tiling at Multiple Spatial Resolutions

• Adaptive Segmentation of SAR Pulse Dataset(s) at Multiple Resolutions

• Efficient SAR Image Reconstruction at Multiple Spatial Resolutionso STEEP Constraints: Space, Time, Error, Energy Profile, and Power Consumption

• Multiresolution Algorithms for Change Detectiono Effects of Noise and Cluttero Incomplete Data – Packet Drop-out, Channel Interruption (Denied Environments)o Support for Object Detection, Segmentation and Recognitiono Complexity, Efficiency (Time, Space, and Energy or Power Consumption)

DDDAS PI Meeting – 06 Sep 2017 4Energy-Aware Time Change Detection in SAR - Ranka (PI)



Technical Approach – Energy-Aware HPC

5Energy-Aware Time Change Detection in SAR - Ranka (PI)DDDAS PI Meeting – 06 Sep 2017

Overlapping

Communication

Hybrid

Core

Mapping

Dynamic

Voltage

Scaling

Static

Load

Balancing

Manual

Tuning

Multiresolution

Approach

Power/Energy

Performance

Tradeoffs

Man

ual

Tun

ing

Hyb

rid

Co

re Map

s

Overlap

pin

g

Co

mm

un

ication

Dyn

amic V

oltag

e Scalin

g

Mu

ltiresolu

tion

Ap

pro

ach

Perfo

rman

ce /

En

ergy

Tradeo

ff

Static/D

ynam

ic Lo

ad B

alance

Approach

Be

nef

its

Overview of Computational Optimization



Technical Approach – SAR Reconstruction with Backprojection

6Energy-Aware Time Change Detection in SAR - Ranka (PI)

Backprojection on a Single GPU

DDDAS PI Meeting – 06 Sep 2017

Approach #1:Pulse Partitioning

Approach #2:Output Image Partitioning

Backprojection is decomposed (1) along output image dimension, so each processing device rendersa tile using all pulse data, or (2) each processing device renders an image using a subset of the pulses.



Technical Approach - Reconstruction (cont’d)


Change detection accepts twoinput images as well as a windowsize, then generates a differencemap.

DDDAS Approach: Output of

change detector is input to

algorithm that assigns spatial

resolution to image tiles.

Overview of DDDAS Approach



Technical Approach - Reconstruction (cont’d)


DDDAS Resolution and Scheduling1. Master decomposes the

problem into atoms (pulses

rendered onto one tile of

output image).

2. Resolution Controller

determines spatial resolution

for rendering each tile.

3. Master sends each atom to

Multilevel Scheduler that

balances load for hetero-

geneous devices and

maintains locality of access

for efficiency.

DDDAS – Multi-Resolution Architecture




Technical Approach – Multiresolution HPC


Instead of computing 256 tiles at full

resolution, we compute only

6 tiles at full resolution

58 tiles at ¼ resolution

13 tiles at 1/16 resolution

Overall speedup factor = 10X

DDDAS – Multi-Resolution Backprojection Realization



Technical Approach – Multiresolution HPC (cont’d)


DDDAS – Multi-Resolution Backprojection Experimental Platform

ORNL Titan

▪ 18,688 nodes,

▪Hybrid node architecture

▪ AMD Opteron 16-core CPU and one Nvidia Tesla K20 GPU.

▪Memory per node:

32GB CPU + 6GB GPU

▪ Peak performance: 20+ petaflops

▪ 512 service and I/O nodes, and 200 cabinets, 4352 sq. ft. floor space

▪ Cray Gemini 3D torus interconnect



Tuning:• Reduced Register usage by reordering instructions• Global memory reads optimized via Float4 array instead of 4 float arrays• Single array storing 𝑥, 𝑦, 𝑧 coordinates of sensor platform location • L1 cache size increased to 48KB

Technical Approach - Multiresolution HPC (cont’d)


Manual Tuning of Multi-Resolution Backprojection for Single GPUReconstructed

Image Size(pixels)

Number of SAR Pulses

High ResolutionExecution Time (sec)

Medium ResolutionExecution Time (sec)

Low ResolutionExecution Time (sec)

Single Res Multi-Res Single Res Multi-Res Single Res Multi-Res

8192x8192 1000 10.5 4.385 2.7 1.18 0.51 0.28

5000 53.75 22.36 13.36 6.01 2.56 1.37

4096x4096 1000 2.79 1.18 0.72 0.303 0.16 0.073

5000 13.72 5.83 3.52 1.56 0.76 0.36

Execution Time directly propor-tional to image resolution

700 GFlops per GPU





Backprojection on Multiple GPUs


Approach #1:Pulse Partitioning

Approach #2:Output Image Partitioning

➢ Multiresolution images create load imbalance Tile distribution balances computational load - and - Bin packing approaches are employed

Pu

lse

s

Range Bins

Output Image

GPU 1

GPU 2





Techniques for Computing Backprojection on Multiple GPUs


LA (List Assignment) Algorithm▪ Compute lower bound 𝐿𝐵 = 𝑇𝑊/𝑔 where g is the number of GPUs.

▪ Order tiles to form linear list L.

▪ Assign tiles to GPU1 in the order of list L until the first tile that causes the work load in GPU1 to be ≥ 𝐿𝐵.

▪ Then assign tiles to GPU2 beginning with next tile in L , stopping when workload in GPU2 becomes ≥ 𝐿𝐵. … and so forth for remainder of GPUs …

Theorem. When list assignment is used, the ratio of the maximum aggregate workload assigned to any GPU and the maximum aggregate assigned to any GPU in an optimal assignment is at most 1 +

𝑤𝑚𝑎𝑥−1

𝑇𝑜𝑡𝑎𝑙 𝑊𝑜𝑟𝑘𝑔 where 𝑤𝑚𝑎𝑥 is the maximum tile workload.





Algorithms for Computing Backprojection on Multiple GPUs


LPT (Longest Processing Time) Algorithm ▪ Sort the tiles into decreasing order of workload

▪ Tiles are assigned to GPUs in this order

▪ When tile 𝑖 is considered, it is assigned to the GPU that has the least aggregate workload assigned thus far

▪ For g GPUs, LPT performance bound yields 4

3−

1

3𝑔on the ratio of the maximum workload

assigned to any GPU by LPT and the maximum assigned to any GPU in an optimal assignment.

Theorem. LPT generates optimal schedules when job times come from a set𝑤𝑖| 1 ≤ 𝑖 ≤ 𝑘 such that 𝑤𝑖 < 𝑤𝑖+1, 𝑖 < 𝑘 and𝑤𝑗 is an integer multiple of 𝑤𝑖 , whenever 𝑗 > 𝑖.





Comparison of Algorithms for Backprojection on Multiple GPUs


9

5

10

13 14

6

1 2 3 4

7 8

11 12

15 16

5 6

1 2 3 4

7 8

9 10

13 14

11 12

15 16

GPU1 (20)

GPU2 (16)

High resolution tile

Low resolution tile

Naïve Algorithm

LA Algorithm

GPU1 (18)

GPU2 (14)

LPT Algorithm

GPU2 (17)GPU1 (17)

5

1 2 3 4

6

7

1 3 15 4

9 11 13

16

8

2 6 5

10 12 14

7 8

9 10

13 14

11 12

15 16

(14) GPU workload



Technical Approach - Energy-Aware HPC


Scaling with Multiple GPUs


▪ 60% of the tiles are high resolution, 20% are medium resolution, and 20% are low resolution for image size of 32,768 x 32,768 pixels for a SAR dataset of 5,000 pulses

▪ The tiles appear randomly (Naïve algorithm) or in a sorted manner (LA or LPT algorithm)▪ LPT and LA algorithms have comparable performance, and perform better than Naïve



Technical Approach - Energy-Aware HPC (cont’d)


Communication Optimizations for Multiple GPUs


▪ Communication is overlapped with GPU Computation using CUDA streams.▪ When GPU computation time is sufficient, both MPI broadcasting and CPU-to-GPU

communication times are covered by GPU computation time.

Comparison with and without communication

overlapping in broadcasting pulse data and

location data to 256 nodes of Titan machine





Communication Optimization for Multiple GPUs


0

5

10

15

20

25

30

35

16 32 64 128 256 512

Tim

e (

Seco

nd

s)

GPU Nodes

Communication Time

GPU Time

LA LPT Naive

0

50

100

150

200

250

300

16 32 64 128 256 512

Tim

e (

Seco

nd

s)

GPU Nodes

Communication Time

GPU Time

LA LPT Naive

5,000 pulses 40,000 pulses

▪ 60% of the tiles are high resolution, 20% are medium resolution, and 20% are low resolution for image size of 32,768 x 32,768 pixels for a SAR dataset of 5,000 pulses

▪ The tiles appear randomly (Naïve algorithm) or in a sorted manner (LA or LPT algorithm)▪ LPT and LA algorithms have comparable performance, and perform better than Naïve

30 to 40 TFlop on 128 GPUs





Real Machine – Representative of Future Node Architecture

Portion of Computation on CPU

Portion of Computation on GPU

GPU





Theoretical Framework for Performance OptimizationMinimize E(n, f, g, S)Subject to T(n, f, g, S) <= T1 – O(n, f, g, S)

n < Nf ∈ Fg ∈ G

24 (e,g,, 2 12 core machines)

14

4

1344 R

Minimize ECPU(n, f, X) + EGPU(g, S - X)Subject to TCPU(n, f , X) < T1 – Co(n, S)

TGPU(g, S - X) < T1 – Co’(S - X)

n < Nf ∈ Fg ∈ G

24

14

4

(N.F+G).R=340R TCPU(n, f, X) × PCPU(n, f, X) TGPU(g, S-X) × PGPU(g, S-X)

TCPU(n, X)/f × PCPU(n, f) TGPU(S-X) / g × PGPU(g)N.R+N.F+R+G =340+25R

n CPU cores, freq f GPU core, freq g

X S-X

Note: Can have more than 1 GPU

E Energy consumed

n Number of CPUs

f CPU Frequency

g GPU Frequency

S Problem size

T Computation Time

T1 Constrained time

O Communic’n. overhead

N Total available CPUs

F Set of available CPU

frequencies

G Set of available GPU

frequencies

R Cardinality of workload distribution set




21

Modeling Framework for Performance Optimization


Observation 1: CPU Compu-

tation Time is Inversely Pro-

portional to CPU Clock Fre-

quency






Observation 2: GPU Compu-

tation Time is Inversely Pro-

portional to GPU Clock Fre-

quency






Observation 3: CPU

Power Consumption

Does Not Depend

Much on Problem Size






Observation 4: GPU

Power Consumption

Does Not Depend

Much on Problem Size






Observation 5: Equations for CPU

Power Consumption (including

DRAM power) Depend on the

Number of CPU Cores

PCPU(1, 𝑓) = 46.58979 + 0.008488 × 𝑓

PCPU(2, 𝑓) = 43.32728 + 0.009557 × 𝑓

PCPU(4, 𝑓) = 39.42679 + 0.011428 × 𝑓

PCPU(12, 𝑓) = 33.68219 + 0.022718 × 𝑓

PCPU(23, 𝑓) = 74.73546 + 0.043811 × 𝑓

Number of CPU Cores




Pareto Curves for Energy versus Runtime (Aggregated & Actual)


Tota

l En

erg

y (J

)

Actual

Aggregated

484-23-2400-875

256-12-2400-875

36-12-2400-87516-23-2400-875 16-12-2400-81036-12-2400-87516-12-2400-745

484-23-2400-875

256-12-2400-875

36-12-2400-875 16-23-2400-875 16-12-2400-810 36-12-2400-74516-12-2400-745

900

1000

1100

1200

1300

1400

1500

1600

1700

1800

1900

4.5749 4.8379 5.1285 5.1316 5.5082 5.9775 6.1803

4.5489 4.8111 5.0728 5.1113 5.5266 5.9450 5.9980

Aggregated EnergyResults

Number of Tiles in CPU Number of CPU cores

4K x 4K Image5000 pulses

--- Actual Energy ResultsCPU Frequency

GPU Frequency




Modelling Energy versus Runtime (Aggregated & Actual)


Tota

l En

erg

y (J

)

Time (seconds)

4K x 4K Image, 5000 pulses

Aggregated

Actual Tota

l En

erg

y (J

)

Time (seconds)

900

1000

1100

1200

1300

1400

1500

1600

1700

1800

1900

4.5 5 5.5 6 6.5 7900

1100

1300

1500

1700

1900

4.5 5 5.5 6 6.5 7




Modelling Power Consumption vs. Runtime (Aggregated & Actual)


Aggregated

Actual

4K x 4K Image5000 pulses




Energy vs. Runtime - Aggregated Results




Technical Approach (cont’d)

Simulation

• Construct Experimental Pulse Datasets from Digital Elevation Maps• Precisely Controlled Test Data• Controllable Parameters to Facilitate Performance & Error Analysis

Change Detection (CD)

• CD Algorithm Developed at UF Highlights Regions of Interest • Isolation of Regions Containing Moving Vegetation (high frequency variance)

• Construction of Multiresolution Scene Representation Optical Flow• Identification of Future Target Movement Regions (by variance & context) • Statistical ID and Tracking of Targets by Spatiotemporal (ST) Variance




Technical Approach – Simulation

Construct Experimental Pulse Datasets and Video SAR (VSAR)

31

x

y

z

Pulse EmitterReceiver

zsrS

T

Given:

Bidirectional Reflectivity Distribution Function (BRDF)

Emitter Intensity I

Received Intensity Ir = I • BRDF(qi)

Emitter Track S = (s1, s2, …, sP) S = T monostatic SAR

Receiver Track T = (t1, t2, …, tP) S ≠ T bistatic SAR

Number of Pulses P

Pulse Data Resolution (per pulse) NB bins

Resolution NxN pixels of Reconstructed Image defined on X

Frame Rate F

Objective: Construct pulse dataset D having NP

pulses of NB bins each F P NB elements

SimulatedGroundPlane X

Energy-Aware Time Change Detection in SAR - Ranka (PI)DDDAS PI Meeting – 06 Sep 2017



Technical Approach – Simulation (cont’d)

Results: Simulation & Reconstruction Example (urban scene)

1. Construct Digital Map (point cloud) using POVrayTM

2. Construct Experimental Pulse Datasets from Digital Elevation Maps3. Compute VSAR from Pulse Dataset using SAR Reconstruction Algorithm

DDDAS PI Meeting – 06 Sep 2017 32Energy-Aware Time Change Detection in SAR - Ranka (PI)

VSAR Frame Noise = 0, = 0.001 256x256 pixels Noise = 0, = 0.009 256x256 pixels Noise = 0, = 0.09

Packet Dropouts

SensorNoise



Technical Approach (cont’d)


Simulation

• Construct Experimental Pulse Datasets from Digital Elevation Maps• Precisely Controlled Test Data• Controllable Parameters to Facilitate Performance & Error Analysis

Change Detection (CD)

• CD Algorithm Developed at UF Highlights Regions of Interest • Isolation of Regions Containing Moving Vegetation (high frequency variance)

• Construction of Multiresolution Scene Representation Optical Flow• Statistical ID and Tracking of Targets by Spatiotemporal (ST) Variance• Identification of Future Target Movement Regions (by ST variance & context)




Technical Approach – Change Detection


Background


▪ Optical flow and Supervoxel-based Segmentation• 2D gPb-UCM (Arbelaez, 2011) extension to 3D• Optical flow estimation• More accurate and scalable approach

▪ Challenges in 3D volumetric image segmentation• Complex backgrounds• Variations of object size• Imbalance between the number of foreground and

background voxels

▪ Integration with optical flow• Optical flow estimates object motion• Assume different objects move differently



Technical Approach – Change Detection (cont’d)


Contributions of 3D-UCM


Step 1: Image Gradient Features Detection• Developed 3D counterpart of the oriented

gradient operators

Step 2: Globalization• More robust way to compute affinity

matrix

• Reduced order Normalized Cuts

Step 3: Agglomeration• Graph based methods instead of

Oriented Watershed Transform





Application to Change Detection in Airborne/Spaceborne Imagery


▪ Optical flow for moving object detection/tracking

▪ Advance in convolutional neural network

▪ Spatiotemporal machine learning for remote sensing







▪ Optical Flow – Video Application

1. Segmentation

• Background subtraction

• Shot boundary detection

• Motion segmentation

• Object detection and tracking

2. Video stabilization

3. 3D structure estimation

4. Image registration







▪ Deep learning – Modified U-Net

• End-to-end approach

• Train on co-registered image pairs in a segmentation CNN

• Deep convolutional encoder decoder architecture for object-based labelling

• Segment images with small changes in objects or regions

• Predict object-based labels from supervised learning





Preliminary Results of Change Detection on Simulated VSAR


Format: 512x512-pixel SAR video of simulated urban intersection

Noise (additive) = 0 = 0.001 to 0.008

A few False Positives (FPs) – We Are Developing Contextual Methods to Remove FPs







Format: 256x256-pixel SAR video of simulated urban intersection


Original image Optical Flow Approach Detection Result

Format: 512x512-pixel SAR video of simulated 5x5-block urban scene








Format: 512x512-pixel SAR video of simulated 5x5-block urban scene


Detection ResultsPartial TargetsOccluded TargetsFalse Positives

Resolved withContextual Data

Detection Result 512x512 VSAR Segmentation of Target Regions



Accomplishments & Future Work


Accomplishments• Success 1: Green Computing: 1/f power reduction in SAR Image Reconstruction

where f denotes resolution factor• Success 2: Fast, Scalable SAR Image Reconstruction on ORNL TITAN • Success 3: Accurate Models of CPU, GPU Energy and Power Consumption• Success 4: Computationally Efficient VSAR Simulation & Analysis• Success 5: Change Detection Successful on Noisy VSAR imagery

Future Technical Work• Enhance Performance (accuracy, speedup) of Multiresolution Backprojection• Extensive Modeling of Urban Scenes Library of VSAR Test Videos• Integration of Point Cloud Computation from Multiple Images (Transparent Sky LLC)

to Drive UF’s VSAR Simulation Capability• Improvement of Change Detection Algorithm – Multiresolution (faster), Increased

Tolerance of Noise and ClutterDDDAS PI Meeting – 06 Sep 2017



Accomplishments (cont’d)


Publications▪ Monograph

• Schmalz, M.S., W.H. Chapman, S. Ranka, S. Sahni, E. Hayden, U. Majumder, G. Seetharaman (submitted, in revision with added DDDAS chapter) Parallel Hierarchical Reconstruction and Change Detection of Synthetic Aperture Radar Imagery.

▪ Journal Papers

• Wijayasiri, A., T. Banerjee, S. Ranka, S. Sahni and M.S. Schmalz (submitted) “SAR Image Reconstruction in GPU Systems”.

• Seetharaman, G., E.T. Hayden, M.S. Schmalz, W.H. Chapman, S.Ranka, and S. Sahni (in re-submission) “Dynamic multistatic synthetic aperture radar (DMSAR) with image reconstruction algorithms and analysis”.

▪ Conference Paper

• A. Wijayasiri, T. Banerjee, S. Ranka, S. Sahni and M.S. Schmalz. “MultiobjectiveOptimization of SAR reconstruction on hybrid multicore systems”, to appear in Proceedings of HiPC 2017 Conference.




Accomplishments (cont’d)


Coordination / Synergy▪ With Other PIs• Dr. Steve Suddarth (Transparent Sky LLC) and Dr. K. Palaniappan (Univ. Missouri)

✓ Sharing UF’s Computational Optimization Technology✓ Developing Sharing Protocol for UF’s SAR / VSAR Reconstruction & Simulation Technology

▪ AFRL / DoD Efforts – Cross-Fertilization by Current AFRL Project• Computation of 3D Models with Low SWaP Architectures, Subcontract to Transparent Sky

LLC on Navy Phase 1 SBIR, 2017 (PI: M.S. Schmalz)• IPNAC – Intelligent Programming of New Architectures for Computing, Subcontract to

Transparent Sky LLC on DARPA Phase 1 SBIR, 2015 (PI: M.S. Schmalz).• Hierarchical Scalable Multi-Dimensional Indexing for High Performance Video

Search/Retrieval in Image Based Query Systems, US Air Force, 2014-2015 (PI: M.S. Schmalz)

▪ Data Coordination• Data Sharing of 3D Point Clouds among UF, Transparent Sky LLC, and Univ. MIssouri




Discussion


Energy-Aware Time Change Detection Using Synthetic ...€¦ · Energy Aware Time Change Detection...

Documents

Transcript of Energy-Aware Time Change Detection Using Synthetic ...€¦ · Energy Aware Time Change Detection...