Near-shore biogeography and biodiversity patterns around South Africa: setting the scene
Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing
description
Transcript of Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing
![Page 1: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/1.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448© 2014 The MITRE Corporation. All Rights Reserved.
Applications of Spring Integration to
Near-Real Time Radar Data ProcessingG. Wampole – MITRE Corporation
Approved for Public Release; Distribution Unlimited. Case Number 14-2448
![Page 2: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/2.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
NOTICE
This technical data was produced for the U.S. Government
under Contract No. W15P7T-13-C-A802, and is subject to
the Rights in Technical Data-Noncommercial Items clause
at DFARS 252.227-7013 (FEB 2012)
© 2014 The MITRE Corporation. All Rights Reserved.
![Page 3: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/3.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
Your Presenter
• Garrett Wampole– Hometown: Allentown, PA
– Alum: Rochester Inst. Tech.
– Been Doing This: ~15 years
• Currently– Prin. Engineer @ MITRE
– Radars, Battle Management, Remote Sensing, Imagery Exploitation, U-2, JSTARS
• Spare Time– iPhone & Windows Phone
(VoiceNotes anyone?)
– Root for New York Mets
![Page 4: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/4.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
What is MITRE?
• A not-for-profit company that operates FFRDCs(Federally Funded Research and Development Center)
• Presently administers six centers
– National Security Engineering Center (DoD)
– Center for Advanced Aviation System Development (FAA)
– Center for Enterprise Modernization (IRS)
– Homeland Security Systems Eng. and Development Institute (DHS)
– Judiciary Engineering and Modernization Center (Federal Judiciary)
– CMS Alliance to Modernize Healthcare (DHHS)
• Approximately 7000 employees (80.4% engineers)
• Two main headquarters: Bedford, MA and McLean, VA
– ~50 domestic sites, 10 international
• http://www.mitre.org
– Fun Fact: First name registered in the .org TLD (July 1985)
![Page 5: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/5.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
What is Radar?
Image source: NOAA
![Page 6: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/6.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
Simplest Case – Monostatic Radar
Mover
![Page 7: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/7.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
Fusion of Multiple Monostatic Radars
![Page 8: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/8.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
Trade-Offs
• Multistatic
+ Less complicated hardware, smaller sensor footprint
+ Cheaper to Build
– Need multiple sensor locations
– Need coordinated central processing
• Phased Array
+ Everything in-situ, one (bigger) box
+ Potential for greater accuracy
– More complex sensor setup
– Development costs have three commas
![Page 9: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/9.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
It’s a Data Reduction Problem
Mover Reports
Secondary
Processing
![Page 10: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/10.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
Pipeline Processing
> wget -qO - www.cnn.com | grep -i spring | wc -l
1. HTTP GET
2. Grep
3. Count
1. Coordinate Timing
2. Find Movers
3. Track Movers over Time
4. Geolocate and Report
![Page 11: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/11.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
Technical Challenges
• Matching up data from multiple sensors that may arrive at different times
• Coordinating stateful pipeline steps that depend on prior data from each sensor
• Keeping up with near-real-time processing when unforeseen processing hiccups occur
• Supporting multiple types of deployment
– One machine, multiple laptops, servers on different networks
• Support different types of sensor input and product output
– Sensors hardwired or wireless, provide reports as KML or STANAG 4607 to file or network or …
• Provision for a not overly burdensome development environment
• Save intermediate processing products for later analysis
![Page 12: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/12.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
Pipeline Processing Steps
• Service Activator which manages a set of MATLAB objects
maintained in a Matlab Compiler Runtime instance
MCR
service()
onNewData()
blockonProccessingComplete()
![Page 13: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/13.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
Multiple Input/Output Types
• Channel Adapter abstraction is key
– Doesn’t change the rest of the processing pipeline
PbufFileIn/OutboundChannelAdapter
CgmtiFile/UDPOutboundChannelAdapter
PbufTCPIn/OutboundChannelAdapter
PbufUDPIn/OutboundChannelAdapter
PbufUDTIn/OutboundChannelAdapter
ReversePbufFileInboundChannelAdapter
Processing
![Page 14: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/14.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
Development Environment
• Accurately recreating what happened during near-real-
time processing is difficult
– But key for debugging problems and for testing new
algorithms against a representative data set
setRealTimePlaybackEnabled( true )
Recorded
Data
Poller blocked
![Page 15: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/15.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
Flexible Deployments
• One Spring Context / Matlab Runtime combo per JVM
• Use network-based Channel Adapters as IPC
Generate Products Process
Generate Products
Both deployments supported, no recompile
– Pick an XML context file to start
Process
![Page 16: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/16.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
Deployments
Receive Data From Sensors
Coordinate Pre-Processing
Find and Track Movers
Geolocate and Create Products
Do It All At Once
![Page 17: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/17.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
Coordinated Processing
• How to coordinate processing steps that have to occur in proximity, but might be on different processors according to one of many deployments?
– Decorate the service activator with a inter-process cyclic barrier, or “Network Phaser”
Time
arriveAtPhase( 0 )
arriveAtPhase( 0 )
waitForPhase( 1 )
arriveAtPhase( 1 )
blocked
JVM1
JVM2
Phase
Server
Phase
Announcements
![Page 18: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/18.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
Keeping Up
• Fixed-size queue of data inside a message channel
– How to handle out-of-order data over unreliable networks?
Sort according to when sensor produced data
![Page 19: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/19.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
This Stuff Works
http://www.mitre.org/publications/project-stories/bringing-covert-aircraft-out-of-the-shadows
![Page 20: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/20.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448
Thoughts
• Enterprise Integration is not the same as B2BIt can work within near-real-time constraints
Spring Integration == ESB == Slow
• SI’s lightweight implementation enables flexibility by not forcing you into too many paradigms
– It also enables performance
• The Spring IoC container eases management and deployment
– Easy deployment is a driver for quick-response ISR systems
• You guys made this possible, thanks!
Not True
![Page 21: Applications of Enterprise Integration Patterns to Near-Real Time Radar Data Processing](https://reader033.fdocuments.in/reader033/viewer/2022052623/559e84b61a28abb10b8b46c0/html5/thumbnails/21.jpg)
© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448