The Real-TimeMiddleware Experts
Easing Integration of Large-Scale Real-Time Systems with DDS
Rick Warren, Principal [email protected]
© 2010 Real-Time Innovations, Inc. 2
Introduction
Rick Warren– Principal Engineer; with RTI since 2001– Engineering organization; previously Professional Services– Together with CTO, lead RTI’s Object Management Group
(OMG) initiatives– Passionate about systems integration, standards, and usability
This Presentation– Large-scale design patterns for integrating heterogeneous
distributed systems– Part 1: Architecture and requirements– Part 2: Addressing Part 1 with OMG DDS
© 2010 Real-Time Innovations, Inc. 3
RTI: The Global Leader in DDS
70%+ worldwide DDS market share
First with…– DDS API (2004)– DDS-RTPS interoperability protocol
(2007) — based on RTI’s native protocol
Most mature solution– 12+ years of commercial availability– Technology Readiness Level (TRL) 9:
successful mission operations– Diverse range of industries: defense, finance, medical, industrial
control, power generation, communications– 300+ commercial customers, 100+ research projects– 100,000+ licensed copies
Leading OMG standardization– Board of Directors member– Co-chair DDS Special Interest Group– Chair several DDS standard finalization, revision task forces
© 2010 Real-Time Innovations, Inc. 4
Customer Application Highlights
Weapon System
Lockheed Martin Aegis
Radar, weapons, displays, C2
Unmanned Air System
Boeing ScanEagle
Sensors, ground station
Control System
Grand Coulee Dam: largest electricity producer in US
Controls, high availability
Driver Safety
Volkswagen
Vision systems, analysis, driver information systems
Medical Imaging
NMR and MRI
Sensors, RF generators, user interface, control computers
Automated Trading
Automated Trading Desk (ATD, now Citigroup)
Market data feed handlers, pricing engines, algorithmic
trading applications
© 2009 Real-Time Innovations, Inc. 5
What do I mean by “large system”?
Systems of systems– Modular, hierarchical design– Legacy components,
subsystems– Multiple technologies– Global scale
Decoupled subsystem lifecycles– Independent development– Independent deployment and
use– Independent management– Independent revision and
retirement
Multiple communities of interest– Different data interest,
entitlements– Non-uniform levels of trust
LANLAN
DDS #1(Initech)
DDS #1(Initech)
JMS(Dunder Mifflin)
JMS(Dunder Mifflin)
DDS #2(Acme)
DDS #2(Acme)
Legacy(COBOL ‘R’
US)
Legacy(COBOL ‘R’
US)
Satellite Links
© 2009 Real-Time Innovations, Inc. 6
What matters to integrators of large systems?
Governance (including over security)– What information will be exchanged?– Under what conditions will it be exchanged?– Who is allowed to exchange this information?– If these SLAs are violated, can the exchange be prevented?
Can I be notified?– (In the past, what has occurred wrt these SLAs?)
Isolation– When I connect A and B, ensure they don’t break (each other)– When I disconnect A, ensure B doesn’t break– When I connect C, don’t change A or B
Scalability (like in any system,
Fault Tolerance but stakes are higher)
© 2010 Real-Time Innovations, Inc. 7
Part 1: Architecture
(we’ll get to technology later)
© 2009 Real-Time Innovations, Inc. 8
SystemSystem
Component
Component
Component
Class
A Modest Proposition
State
Behavior
Class
Fundamental design principles scale– Abstraction: Provide interface based on relevant concepts– Encapsulation: Hide internal implementation, communication– Composition: Combine existing capabilities into new ones
Subsystem 2
P P
PP P
Subsystem 1
P P
PP P
Schematic of a Composed System
Subsystems may have different network environments
Integration may have different network environment than subsystems themselves
Data may need to be transformed / cleansed as it moves among subsystems
Routing / gateway services will adapt data types / formats / protocols
LAN LANWAN
Router/Gateway
Router/Gateway
Isolation
Additional Governance
Schematic of a Composed System
Wash, rinse, repeat
Subsystem 1
P P
PP P
Subsystem 2
P P
PP P
Router/Gateway
Router/Gateway
© 2010 Real-Time Innovations, Inc. 10
Same data model?Same network env.?
Same lifecycle?
Behavior unaffected?Understandable?
Apples and Oranges
P P
PP P
Subsystem 1 + Subsystem 2
P P
PP P
Subsystem 2
P P
PP P
Subsystem 1
P P
PP P
Router/Gateway
Router/Gateway
© 2009 Real-Time Innovations, Inc. 12
Nothing New Under the Sun
Subsystem Data Space
Subsystem Data Space
Subsystem Data Space
Subsystem Data Space
Subsystem Data Space
Subsystem Data Space
Integration Data Space
Integration Data Space
Router/Gateway
Router/Gateway
Router/Gateway
© 2010 Real-Time Innovations, Inc. 13
Nothing New Under the Sun
U.S. CombatData Space
U.S. CombatData Space
Allied CombatData Space
Allied CombatData Space
U.S. C4IData Space
U.S. C4IData Space
Integration Data Space
Integration Data Space
Router/Gateway
Router/Gateway
Router/Gateway
Defense Industry
© 2010 Real-Time Innovations, Inc. 14
Nothing New Under the Sun
NYC TradingData Space
NYC TradingData Space
London TradingData Space
London TradingData Space
Tokyo TradingData Space
Tokyo TradingData Space
Integration Data Space
Integration Data Space
Router/Gateway
Router/Gateway
Router/Gateway
Financial Services Industry
© 2010 Real-Time Innovations, Inc. 15
Nothing New Under the Sun
GenerationData Space
GenerationData Space
Substation #1Data Space
Substation #1Data Space
Substation #2Data Space
Substation #2Data Space
Integration Data Space
Integration Data Space
Router/Gateway
Router/Gateway
Router/Gateway
Power Industry
© 2010 Real-Time Innovations, Inc. 16
Architecture Recap
Solve big problems like you solve small ones– Break them into
pieces– Define interfaces
between pieces– Implement each piece– Integrate along the
interfaces
Translation
– Define subsystems– Implement them
however you like• May have different
requirements, therefore technology
– Router/gateway defines/enforces interfaces
© 2010 Real-Time Innovations, Inc. 17
Fault Tolerance and Scalability
Fault tolerance: Router/gateway can’t be single point of failure. 3 answers:
1. Persistent data survives system failures2. Segmented/load-balanced configuration limits scope of failures3. Redundancy allows continued service during failure
Scalability: How big can a subsystem be?1. How big a subsystem can you understand, test, and debug?2. How well does infrastructure scale?
© 2010 Real-Time Innovations, Inc. 18
Part 2: DDS
DDS Is Open Architecture
OMG standard for data-centric publish-subscribe communication
Vendor independent– API for portability– Network protocol for interoperability
8+ implementations– Commercial, open-source, and
hybrid licenses
Supports heterogeneous systems– C, C++, C#/.NET, Java, Ada, …– Windows, Linux and other Unix,
embedded, RTOS
Real-Time Publish-Subscribe
Protocol (DDS-RTPS)
Middleware
DDS API
Cross-vendor portability
Cross-vendor interoperability
© 2010 Real-Time Innovations, Inc. 19
Example: Data-Centric Track Data
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL 20
Pu
blish
Su
bscribe
Data Schema
x : float
y : float
id : string (key)
New
45.6
78.9
“AA123”
Update
56.7
89.0
“AA123”
New
65.4
32.1
“DL987”
Dispose
“AA123”
X
Map this into XML; rows + cols
Express content-based filters
Propagate data efficiently
Example: Data-Centric Track Data
© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL 21
Pu
blish
Su
bscribe
Data Schema
x : float
y : float
id : string (key)
Quality of Service
Deadline
Time-Based Filter
History
Once infrastructure understands objects, can attach QoS contracts to them
“Keep only the latest value” or “I need updates at this rate” make no sense unless per-object– Flight AA123 updates shouldn’t overwrite DL987, even if
AA123 is updated more frequently– Update rate for one track shouldn’t change just because
another track appeared
Analysis by U.S. Armed Services
NESI, P1190 rev 3, http://nesipublic.spawar.navy.mil/nesix/Frames/P1190
© 2010 Real-Time Innovations, Inc. 22
Adapted from NSWC-DD OA Documentation
© 2010 Real-Time Innovations, Inc. 23
How Does DDS Stack Up?
Governance– Structural SLA: data type– Behavioral SLA: delivery QoS– Security– Problem
detection/prevention/notification– System monitoring and recording
Isolation– Prevent data “leaks” and side effects– Allow dynamic (dis-|re-)connection– Support independent integration and
evolution
Fault tolerance– Data persistence– Segmented/load-balanced routing– Redundant routing
Scalability– WAN, DIL connectivity– Peer-to-peer communication– Brokered/managed communication
– Built in– Built in– Products available; future standards– Built in
– Products available
– Built in: domains, partitions
– Built in– Built in, esp. w/ DDS-XTypes spec
– Built in– Built in if router uses DDS– Built in if router uses DDS
– Products available; future standards– Highly scalable– Protocol support if necessary
© 2010 Real-Time Innovations, Inc. 24
DDS Scalability: Two Aspects
1. Application data– How does performance fall off as # participants, writers,
readers increase?– Any single writer/reader pair can saturate gig-E:
achieving high aggregate throughput is not main issue– Interesting issue: Fan-out – number of readers per writer
2. Discovery– How many DDS applications can discover one another?– Interesting issue: How many applications need to discover
one another?
Hard limits, if any, very dependent on DDS implementation, machine configuration, network, etc.– Following results based on RTI on modern commodity HW
© 2010 Real-Time Innovations, Inc. 25
1. DDS Scalability: Application Data
DDS-RTPS reliable multicast scales at least: – …to hundreds of readers per writer– …with very little degradation in throughput.– Larger testing facilities welcomed.
< 15% decrease1~900 readers
RTI Data Distribution Service 4.3Red Hat EL 5.0, 32-bit2.4 GHz processorsGigabit EthernetUDP/IPv4Reliable ordered delivery
© 2009 Real-Time Innovations, Inc. 26
2. DDS Scalability: P2P Discovery Scenarios
Single domain, symmetric discovery– Everyone discovers everyone else– Easiest to configure; most challenging wrt scalability– RTI test results: 1,800 participants; 3.2M endpoint matches
Single domain, asymmetric discovery– Each participant only knows of certain others in its domain– Good for partitioning domains in which not everyone talks
Multiple domains– Greatest separation for subsystems that rarely exchange data
• DDS-RTPS maps to different IP ports
1
…
n
© 2009 Real-Time Innovations, Inc. 27
Summary: Large-Scale P2P Scalability
Experimental results: DDS-RTPS allows participant to talk to ≥ 1-2K others in single symmetric domain
Implication: Compose arbitrarily large systems out of subsystems this size or smaller
Large (sub)systems often have requirements for:– Increased governance, e.g. over security boundaries– Data transformation– Protocol mediation– These typically require data brokering anyway
© 2010 Real-Time Innovations, Inc. 28
Geographic Scalability: WAN Transport
Site-to-site comms need DDS across WAN(s)– Including untrusted networks– Including firewall, NAT traversal– Challenge: UDP may not be routable, especially if multicast
Standard DDS-RTPS protocol designed for layeringatop diverse lower-level protocols– RTI supports:
• Shared memory for performance on a single machine• UDP (unicast, multicast) for scalability, determinism w/in LAN• TCP for routing across WAN• (D)TLS for transport-level security• etc.• …simultaneously
© 2010 Real-Time Innovations, Inc. 29
Geographic Scalability: DIL Transport
DDS is also great on disadvantaged networks– DDS QoS provide tunable behavior, performance– DDS-RTPS protocol encapsulates data efficiently w/ CDR
Source:
• SELEX-SI/ Consorzio SESM/ University of Naples
• “Flexible Communication Among DDS Publishers and Subscribers”
• July 2008, Real-Time Systems Workshop, Washington, DC
10x more compact than XML or JSON
© 2010 Real-Time Innovations, Inc. 30
Geographic Scalability: DIL Transport
DDS is also great on disadvantaged networks– DDS QoS provide tunable behavior, performance– DDS-RTPS protocol encapsulates data efficiently w/ CDR
Source:
• SELEX-SI/ Consorzio SESM/ University of Naples
• “Flexible Communication Among DDS Publishers and Subscribers”
• July 2008, Real-Time Systems Workshop, Washington, DC20x faster
RTI Routing Service:Integrating Non-DDS Applications
Adapt to other protocols and interfaces
Adapter SDK with customizable plugins– JMS– Socket– File
Take advantage of RoutingService capabilities– Data transformation and filtering– Data life-cycle management– Cross networks, transports
and platforms– Dynamic discovery
DDS
JMS
Soc
ket LIN
K11
CA
Nbus
Plug-in Adapters
© 2010 Real-Time Innovations, Inc. 31
© 2010 Real-Time Innovations, Inc. 32
Recap
Large systems usually composed of smaller systems– Developed independently– …With different network environments– …And different data interest/entitlements
Not an accident: hierarchical design is a best practice at any scale– Subsystems managed by router/gateway
• Restrict access to internal topics/flows• Mediate internal/external protocols• Transform/cleanse internal/external data types• Impose governance: enforce policy, attach monitoring
– Inter-router integration space is itself a “subsystem” for further hierarchical composition
© 2010 Real-Time Innovations, Inc. 33
Recap
DDS is uniquely qualified to meet these needs– Strong SLAs for governance– Multiple topologies for fault-tolerant subsystem isolation– High-performance, scalable, deterministic interoperability
in various network environments– What other standard technology offers this combination?
(Not JMS. Not CORBA. Not AMQP. Not web services. …)
© 2010 Real-Time Innovations, Inc. 34
Next Steps
Free software downloads– www.rti.com/downloads– Product evaluation:
full RTI Professional Edition• Free trial with comprehensive tutorial• Free licenses for research & academia• Includes:
– Leading DDS implementation– Data recording– System monitoring and debugging– Integrates with JMS, databases,
Microsoft Excel– More
– Shapes demo: no coding required
Videos, webinars, and whitepapers– www.rti.com/resources
Top Related