Engineering Interoperable and Reliable Systems

The Global Leader in DDS

Engineering Interoperable and Reliable Systems Contrasting DDS, JMS, and AMQP

Rick Warren [email protected] Director of Technology Solutions

Introductions

Rick Warren [email protected] !  Director of Technology Solutions

–  Aligning product strategy with customer needs –  Coordinating existing product, accelerated product,

professional services

!  OMG principal –  Participated DDS revision task force –  Authored or co-authored Extensible and Dynamic Types

(DDS-XTypes), C++ PSM, Java PSM, and other specs –  Chaired several specification finalization task forces

!  Former Principal Engineer –  With RTI nearly 10 years –  Experience in R&D and Services

© 2011 Real-Time Innovations, Inc. 2

Most Important Points in this Presentation

Themes: Interoperability, Reliability

!  Flow first from architecture –  What is the information you’re

modeling? –  How does it change over time? –  How to address dynamic

systems? –  How to address system

evolution, integration?

!  Pick the right tool for the job –  Different technologies provide

•  Different degrees of architectural support

•  Different levels of portability, interoperability

–  Biggest mistake: “It’s all just ‘messaging’.”

Agenda 1.  Architecture

–  Levels of interoperability

–  Levels of reliability

–  Data-centricity

2.  Technology Implications –  DDS –  JMS –  AMQP


(Technology Refresher)

!  Java Message Service (JMS) –  Messaging at syntactic level:

no assumed semantics –  Portable API for messaging

in Java –  Specification from Java

Community Process (JCP)

!  Advanced Message Queuing Protocol (AMQP) –  Messaging at syntactic level:

no assumed semantics –  Interoperable protocol for

messaging –  Specification from AMQP

Working Group, industry consortium

!  Data Distribution Service (DDS) –  Application of messaging to

distributed state model –  Portable API for data distribution

in multiple languages (DDS) –  Interoperable protocol for data

distribution (DDS-RTPS) –  Specifications from Object

Management Group (OMG)


Levels of Interoperability and Reliability


Semantic Interpretation of messages e.g. DDS

Syntactic Communication

fragments— “messages”

e.g. JMS, AMQP

Technical Bits and bytes e.g. TCP

Cum

ulative


Data Level Semantic

Agree on what information refers to, how it’s related

Message Level

Syntactic Distinguish one piece of information from another

Byte Level Technical

Exchange raw information, regardless of structure


Interoperability Means:


Data Level Semantic

Order across streams,

eliminating duplicates

“Summarize” data changes for slow / late-joining

consumers

Message Level

Syntactic

Route across multiple

connections Store for later

delivery


Repair dropped bytes over

reliable connection


Reliability Means:


Data Level Semantic

We call these technologies “data-centric.”

Message Level

Syntactic We call these technologies

“message-centric.”



Higher levels implemented on top of lower based on chosen conventions

• Conventions not standardized => no interoperability

•  Implementations of lower layers constrain behavior, performance of upper

Data-Centricity by Example: Calendaring

Alternative Process #1: 1.  Email: “Meeting Monday at 10:00.”

2.  Email: “Meeting moved to Tuesday.”

3.  Email: “Here’s dial-in info for meeting…”

4.  You: “Where do I have to be? When?”

5.  You: (sifting through email…)

9 © 2011 Real-Time Innovations, Inc.

Data-Centricity by Example: Calendaring

Alternative Process #2: 1.  Calendar: (add meeting Monday at 10:00)

2.  Calendar: (move meeting to Tuesday)

3.  Calendar: (add dial-in info)

4.  You: “Where do I have to be? When?”

5.  You: (check calendar)


Data-Centricity by Example: FaceBook

Alternative Process #1: !  Use the FaceBook web site.

Alternative Process #2:

!  The web site doesn’t exist.

!  Deduce your friends’ information based on the notifications FaceBook sends you.


What’s the Difference? State.

!  Things have attributes and characteristics –  The meeting will run 1:00–2:00

in the conference room. –  My friend’s phone number is

555-1234 and he’s currently grooming his cat.

–  The car is blue and is traveling north from Sunnyvale at 65 mph.

!  …whether they exist in the real world, in the computer, or both

!  …whether or not we observe or acknowledge them

“State” (“data”) is a snapshot of those attributes and characteristics.

Best Practice: operate on state directly, not dialogs about state.


Data-Centric =

Describing the world as it is

Implication: State of the world can be maintained by infrastructure, not each app

13

the part of you care about

at a certain point in time

© 2011 Real-Time Innovations, Inc.

Not Data-Centric =

Saying anything else !  “Hey you: go do this.” !  “The thing changed in this way.”

Implication: State must be inferred, reconstructed, managed by each app

(“Message-centricity”: focus on what’s said vs. what is)


Why is it better to just describe the world?

!  Reconstructing the state of the world is hard. –  Must infer based on all previous messages –  Maintaining all these messages is expensive –  Each app makes these inferences => duplicate effort

!  People make mistakes, and systems fail. –  Many copies of state => may be different => bugs

vs. –  Uniform operations on state => fewer bugs

!  The world changes. –  Learning current state must not require replaying

everything ever said to you –  Adding a function to a system must not require

modifying every other function


Reduce Effort

Improve Quality

Future-Proof

Design

Before We Forget: the Definition

A data-centric architecture: 1.  …is based on a data model that is:

–  Appropriately documented— i.e. understandable by humans

–  Formally defined— i.e. understandable by machines

–  Discoverable—i.e. can be found during execution

2.  The data model is independent of any domain-specific functionality / application. –  i.e. made of nouns, not verbs –  Actions are uniform; typically: Create, Read, Update, Delete (CRUD)

3.  The (instantiated) data model is the only authoritative source of state in the system.

16

For example, data structures in XSD

or IDL file.

Calendar Event = •  Start Time •  Duration •  Location •  Organizer


DDS is not messaging.

DDS is data-centric messaging. That is,

!  Messaging as a technique

!  …for distributing, accessing, and modifying stateful data

e.g. DDS complements e.g. database

17

Data at Rest

App App App App App App

Data in Motion


Example: Data-Centric Radar Data as in DDS

© 2011 Real-Time Innovations, Inc. COMPANY CONFIDENTIAL 18

Publish

Subscribe

Data Schema

x : float y : float

id : string (key)

Create

45.6 78.9

“AA123” Update

56.7 89.0

“AA123” Create

65.4 32.1

“DL987” Delete

“AA123”

X

Map this into XML; rows + cols

Express content-based filters

Propagate data efficiently

Example: Data-Centric Radar Data as in DDS


Publish

Subscribe

Data Schema

x : float y : float

id : string (key) Quality of Service

Deadline Time-Based Filter

History

!  Once infrastructure understands objects, can attach QoS contracts to them

!  “Keep only the latest value” or “I need updates at this rate” make no sense unless per-object –  Flight AA123 updates shouldn’t overwrite DL987, even if

AA123 is updated more frequently –  Update rate for one track shouldn’t change just because

another track appeared

Example: Message-Centric Radar Data as in JMS or AMQP


Publish

Subscribe

x=float(45.6) y=float(78.9)

id=“AA123” 0x0000000641410102030042366666429DCCCD

“My app knows this means delete.”

(No Data Schema,

Limited QoS)

Nothing to base filters, xforms on

Error checking dev " integration

Self-describing data is verbose

Message-Level Interoperability

!  JMS, AMQP cover syntax of message exchange –  If we agree on an abstract

destination, I can send you messages

–  Not addressed: •  What these messages

represent •  What they will contain •  How the data in the business

domain changes over time

!  (Interoperability limitations within syntactic level): –  JMS does not specify

network interoperability –  AMQP does not specify API

Needed for interoperability!

–  Especially when integrating independently developed components

Mitigation: implement yourself, idiomatically

–  More work –  Difficult to govern –  Non-standard


Data Level

Message Level

Byte Level

Data-Level Interoperability

!  DDS applies semantics to “messages” –  Topic = group of similar

objects / records –  Instance = particular object,

identified by key –  Sample = state of particular

object at particular point in time –  Like relational DB that changes

over time

!  DDS governs data structure –  At design time and run

time –  Based on discoverable

schemas

!  DDS governs data changes –  Among pubs, subs –  Based on discoverable

QoS

#  Machine-readable definitions amenable to automation


Data Level

Message Level

Byte Level

Topic

Key 1 Value Value

Key 2 Value Value

Key 3 Value Value

Message-Level Reliability

This model is limited:

#  When communication is many-to-many –  How to order messages

across multiple producers?

–  How to eliminate duplicates?

#  When networks are dynamic / disconnected –  How to sync late joiners to

current state?

#  When consumers can’t keep up –  How to summarize / sub-

sample message stream?


Pub

Pub

Pub

Sub

Sub

Sub

JMS, AMQP provide syntactic reliability, 1-to-1. !  You send a message,

I receive it !  Messages have no

particular relationships to each other, outside world

Data Level

Message Level

Byte Level

Data-Level Reliability

DDS provides semantic reliability, many-to-many. !  Some number of sources

publish the changing states of some objects

!  Samples represent snapshots of object state over time

#  Addresses many-to-many: –  Choose: Should updates come

from all sources at once, or one at a time?

–  Choose: If all at once, order by source time or reception time?

–  Duplicates detected and eliminated automatically

#  Addresses dynamic / disconnected networks:

–  Identify number of relevant previous states; late joiners get just these

#  Addresses slow consumers: –  Doesn’t (re)deliver obsolete

previous states –  Sub-sample rapid updates based

on configuration


Pub

Pub

Pub

Sub

Sub

Sub

Topic

Instance

Instance

Instance

Data Level

Message Level

Byte Level

Summary

1.  Architect for interoperability and reliability –  Start with common data model of business

domain •  …Including static and dynamic elements

–  Design interactions based on applying uniform operations (CRUD) on this model

–  Account for enforcement / governance

2.  Select technology based on conformance to architecture wherever possible –  Supports and enforces data model? –  Provides standards-based code portability? –  Provides standards-based component

interoperability?

3.  Mitigate lack of architectural support wherever necessary –  Document and implement data model

governance –  Create portable API wrappers where lacking –  Define protocol usage idioms to increase

interop, reliability levels

Technology Score Card !  DDS

–  Interoperability, reliability founded on system-defined data model •  Discoverable •  Machine-readable •  Automatically enforced

–  Portable API in multiple languages

–  Interoperable protocol

!  JMS –  Interoperability, reliability

defined at message level only: no pre-defined semantics

–  Portable API in Java

!  AMQP –  Interoperability, reliability

defined at message level only: no pre-defined semantics

–  Interoperable protocol


Engineering Interoperable and Reliable Systems

Technology

Transcript of Engineering Interoperable and Reliable Systems