Easing Integration of Large-Scale Real-Time Systems with DDS

The Real-TimeMiddleware Experts

Easing Integration of Large-Scale Real-Time Systems with DDS

Rick Warren, Principal [email protected]

mailto:[email protected]

© 2010 Real-Time Innovations, Inc. 2

Introduction

Rick Warren– Principal Engineer; with RTI since 2001– Engineering organization; previously Professional Services– Together with CTO, lead RTI’s Object Management Group

(OMG) initiatives– Passionate about systems integration, standards, and usability

This Presentation– Large-scale design patterns for integrating heterogeneous

distributed systems– Part 1: Architecture and requirements– Part 2: Addressing Part 1 with OMG DDS


RTI: The Global Leader in DDS

70%+ worldwide DDS market share

First with…– DDS API (2004)– DDS-RTPS interoperability protocol

(2007) — based on RTI’s native protocol

Most mature solution– 12+ years of commercial availability– Technology Readiness Level (TRL) 9:

successful mission operations– Diverse range of industries: defense, finance, medical, industrial

control, power generation, communications– 300+ commercial customers, 100+ research projects– 100,000+ licensed copies

Leading OMG standardization– Board of Directors member– Co-chair DDS Special Interest Group– Chair several DDS standard finalization, revision task forces


Customer Application Highlights

Weapon System

Lockheed Martin Aegis

Radar, weapons, displays, C2

Unmanned Air System

Boeing ScanEagle

Sensors, ground station

Control System

Grand Coulee Dam: largest electricity producer in US

Controls, high availability

Driver Safety

Volkswagen

Vision systems, analysis, driver information systems

Medical Imaging

NMR and MRI

Sensors, RF generators, user interface, control computers

Automated Trading

Automated Trading Desk (ATD, now Citigroup)

Market data feed handlers, pricing engines, algorithmic

trading applications


What do I mean by “large system”?

Systems of systems– Modular, hierarchical design– Legacy components,

subsystems– Multiple technologies– Global scale

Decoupled subsystem lifecycles– Independent development– Independent deployment and

use– Independent management– Independent revision and

retirement

Multiple communities of interest– Different data interest,

entitlements– Non-uniform levels of trust

LANLAN

DDS #1(Initech)

DDS #1(Initech)

JMS(Dunder Mifflin)

JMS(Dunder Mifflin)

DDS #2(Acme)

DDS #2(Acme)

Legacy(COBOL ‘R’

US)

Legacy(COBOL ‘R’

US)

Satellite Links


What matters to integrators of large systems?

Governance (including over security)– What information will be exchanged?– Under what conditions will it be exchanged?– Who is allowed to exchange this information?– If these SLAs are violated, can the exchange be prevented?

Can I be notified?– (In the past, what has occurred wrt these SLAs?)

Isolation– When I connect A and B, ensure they don’t break (each other)– When I disconnect A, ensure B doesn’t break– When I connect C, don’t change A or B

Scalability (like in any system,

Fault Tolerance but stakes are higher)


Part 1: Architecture

(we’ll get to technology later)


SystemSystem

Component

Component

Component

Class

A Modest Proposition

State

Behavior

Class

Fundamental design principles scale– Abstraction: Provide interface based on relevant concepts– Encapsulation: Hide internal implementation, communication– Composition: Combine existing capabilities into new ones

Subsystem 2

P P

PP P

Subsystem 1

P P

PP P

Schematic of a Composed System

Subsystems may have different network environments

Integration may have different network environment than subsystems themselves

Data may need to be transformed / cleansed as it moves among subsystems

Routing / gateway services will adapt data types / formats / protocols

LAN LANWAN

Router/Gateway

Router/Gateway

Isolation

Additional Governance

Schematic of a Composed System

Wash, rinse, repeat

Subsystem 1

P P

PP P

Subsystem 2

P P

PP P

Router/Gateway

Router/Gateway


Same data model?Same network env.?

Same lifecycle?

Behavior unaffected?Understandable?

Apples and Oranges

P P

PP P

Subsystem 1 + Subsystem 2

P P

PP P

Subsystem 2

P P

PP P

Subsystem 1

P P

PP P

Router/Gateway

Router/Gateway


Nothing New Under the Sun

Subsystem Data Space






Integration Data Space


Router/Gateway

Router/Gateway

Router/Gateway



U.S. CombatData Space

U.S. CombatData Space

Allied CombatData Space

Allied CombatData Space

U.S. C4IData Space

U.S. C4IData Space



Router/Gateway

Router/Gateway

Router/Gateway

Defense Industry



NYC TradingData Space

NYC TradingData Space

London TradingData Space

London TradingData Space

Tokyo TradingData Space

Tokyo TradingData Space



Router/Gateway

Router/Gateway

Router/Gateway

Financial Services Industry



GenerationData Space

GenerationData Space

Substation #1Data Space






Router/Gateway

Router/Gateway

Router/Gateway

Power Industry


Architecture Recap

Solve big problems like you solve small ones– Break them into

pieces– Define interfaces

between pieces– Implement each piece– Integrate along the

interfaces

Translation

– Define subsystems– Implement them

however you like• May have different

requirements, therefore technology

– Router/gateway defines/enforces interfaces


Fault Tolerance and Scalability

Fault tolerance: Router/gateway can’t be single point of failure. 3 answers:

1. Persistent data survives system failures2. Segmented/load-balanced configuration limits scope of failures3. Redundancy allows continued service during failure

Scalability: How big can a subsystem be?1. How big a subsystem can you understand, test, and debug?2. How well does infrastructure scale?


Part 2: DDS

DDS Is Open Architecture

OMG standard for data-centric publish-subscribe communication

Vendor independent– API for portability– Network protocol for interoperability

8+ implementations– Commercial, open-source, and

hybrid licenses

Supports heterogeneous systems– C, C++, C#/.NET, Java, Ada, …– Windows, Linux and other Unix,

embedded, RTOS

Real-Time Publish-Subscribe

Protocol (DDS-RTPS)

Middleware

DDS API

Cross-vendor portability

Cross-vendor interoperability


Example: Data-Centric Track Data

© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL 20

Pu

blish

Su

bscribe

Data Schema

x : float

y : float

id : string (key)

New

45.6

78.9

“AA123”

Update

56.7

89.0

“AA123”

New

65.4

32.1

“DL987”

Dispose

“AA123”

X

Map this into XML; rows + cols

Express content-based filters

Propagate data efficiently

Example: Data-Centric Track Data

© 2009 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL 21

Pu

blish

Su

bscribe

Data Schema

x : float

y : float

id : string (key)

Quality of Service

Deadline

Time-Based Filter

History

Once infrastructure understands objects, can attach QoS contracts to them

“Keep only the latest value” or “I need updates at this rate” make no sense unless per-object– Flight AA123 updates shouldn’t overwrite DL987, even if

AA123 is updated more frequently– Update rate for one track shouldn’t change just because

another track appeared

Analysis by U.S. Armed Services

NESI, P1190 rev 3, http://nesipublic.spawar.navy.mil/nesix/Frames/P1190


Adapted from NSWC-DD OA Documentation

http://nesipublic.spawar.navy.mil/nesix/Frames/P1190


How Does DDS Stack Up?

Governance– Structural SLA: data type– Behavioral SLA: delivery QoS– Security– Problem

detection/prevention/notification– System monitoring and recording

Isolation– Prevent data “leaks” and side effects– Allow dynamic (dis-|re-)connection– Support independent integration and

evolution

Fault tolerance– Data persistence– Segmented/load-balanced routing– Redundant routing

Scalability– WAN, DIL connectivity– Peer-to-peer communication– Brokered/managed communication

– Built in– Built in– Products available; future standards– Built in

– Products available

– Built in: domains, partitions

– Built in– Built in, esp. w/ DDS-XTypes spec

– Built in– Built in if router uses DDS– Built in if router uses DDS

– Products available; future standards– Highly scalable– Protocol support if necessary


DDS Scalability: Two Aspects

1. Application data– How does performance fall off as # participants, writers,

readers increase?– Any single writer/reader pair can saturate gig-E:

achieving high aggregate throughput is not main issue– Interesting issue: Fan-out – number of readers per writer

2. Discovery– How many DDS applications can discover one another?– Interesting issue: How many applications need to discover

one another?

Hard limits, if any, very dependent on DDS implementation, machine configuration, network, etc.– Following results based on RTI on modern commodity HW


1. DDS Scalability: Application Data

DDS-RTPS reliable multicast scales at least: – …to hundreds of readers per writer– …with very little degradation in throughput.– Larger testing facilities welcomed.

< 15% decrease1~900 readers

RTI Data Distribution Service 4.3Red Hat EL 5.0, 32-bit2.4 GHz processorsGigabit EthernetUDP/IPv4Reliable ordered delivery


2. DDS Scalability: P2P Discovery Scenarios

Single domain, symmetric discovery– Everyone discovers everyone else– Easiest to configure; most challenging wrt scalability– RTI test results: 1,800 participants; 3.2M endpoint matches

Single domain, asymmetric discovery– Each participant only knows of certain others in its domain– Good for partitioning domains in which not everyone talks

Multiple domains– Greatest separation for subsystems that rarely exchange data

• DDS-RTPS maps to different IP ports

1

…

n


Summary: Large-Scale P2P Scalability

Experimental results: DDS-RTPS allows participant to talk to ≥ 1-2K others in single symmetric domain

Implication: Compose arbitrarily large systems out of subsystems this size or smaller

Large (sub)systems often have requirements for:– Increased governance, e.g. over security boundaries– Data transformation– Protocol mediation– These typically require data brokering anyway


Geographic Scalability: WAN Transport

Site-to-site comms need DDS across WAN(s)– Including untrusted networks– Including firewall, NAT traversal– Challenge: UDP may not be routable, especially if multicast

Standard DDS-RTPS protocol designed for layeringatop diverse lower-level protocols– RTI supports:

• Shared memory for performance on a single machine• UDP (unicast, multicast) for scalability, determinism w/in LAN• TCP for routing across WAN• (D)TLS for transport-level security• etc.• …simultaneously


Geographic Scalability: DIL Transport

DDS is also great on disadvantaged networks– DDS QoS provide tunable behavior, performance– DDS-RTPS protocol encapsulates data efficiently w/ CDR

Source:

• SELEX-SI/ Consorzio SESM/ University of Naples

• “Flexible Communication Among DDS Publishers and Subscribers”

• July 2008, Real-Time Systems Workshop, Washington, DC

10x more compact than XML or JSON


Geographic Scalability: DIL Transport

DDS is also great on disadvantaged networks– DDS QoS provide tunable behavior, performance– DDS-RTPS protocol encapsulates data efficiently w/ CDR

Source:

• SELEX-SI/ Consorzio SESM/ University of Naples

• “Flexible Communication Among DDS Publishers and Subscribers”

• July 2008, Real-Time Systems Workshop, Washington, DC20x faster

RTI Routing Service:Integrating Non-DDS Applications

Adapt to other protocols and interfaces

Adapter SDK with customizable plugins– JMS– Socket– File

Take advantage of RoutingService capabilities– Data transformation and filtering– Data life-cycle management– Cross networks, transports

and platforms– Dynamic discovery

DDS

JMS

Soc

ket LIN

K11

CA

Nbus

Plug-in Adapters



Recap

Large systems usually composed of smaller systems– Developed independently– …With different network environments– …And different data interest/entitlements

Not an accident: hierarchical design is a best practice at any scale– Subsystems managed by router/gateway

• Restrict access to internal topics/flows• Mediate internal/external protocols• Transform/cleanse internal/external data types• Impose governance: enforce policy, attach monitoring

– Inter-router integration space is itself a “subsystem” for further hierarchical composition


Recap

DDS is uniquely qualified to meet these needs– Strong SLAs for governance– Multiple topologies for fault-tolerant subsystem isolation– High-performance, scalable, deterministic interoperability

in various network environments– What other standard technology offers this combination?

(Not JMS. Not CORBA. Not AMQP. Not web services. …)


Next Steps

Free software downloads– www.rti.com/downloads– Product evaluation:

full RTI Professional Edition• Free trial with comprehensive tutorial• Free licenses for research & academia• Includes:

– Leading DDS implementation– Data recording– System monitoring and debugging– Integrates with JMS, databases,

Microsoft Excel– More

– Shapes demo: no coding required

Videos, webinars, and whitepapers– www.rti.com/resources

http://www.rti.com/downloads

http://www.rti.com/resources

Easing Integration of Large-Scale Real-Time Systems with DDS

Technology

Transcript of Easing Integration of Large-Scale Real-Time Systems with DDS