DDS Made Simple

OpenSplice

Angelo CORSARO, Ph.D.Chief Technology Officer OMG DDS Sig Co-Chair

PrismTechangelo.corsaro@prismtech.com

DDS Made Simple

Copyrig

ht 2013, PrismTech – A

ll Rights Reserved.

OpenSplice

What I’ll Cover

☐ Understand DDS and its Data Sharing abstraction

☐ Learn about the benefits that Data Sharing provide over plain Messaging

☐ Learn how much simpler is to build distributed systems with DDS than with messaging technologies

Copyrig

ll Rights Reserved.

OpenSplice

Distributed System Definition

A Distributed System is a collection autonomous software components distributed across a network that coordinate their activities and share system resources to run as a “single system”

N.B. It is useful to remark how this definition does not mention message passing or shared memory, those are implementation details

Copyrig

ll Rights Reserved.

OpenSplice

Distributed Systems Challenges A number of challenges are inherent to the nature of distributed systems:

☐ Transparency

☐ Scalability

☐ Dependability

☐ Performance

☐ Flexibility

Copyrig

ll Rights Reserved.

OpenSplice

Building Distributed Systems

Several abstractions have been proposed to build Distributed Systems:

☐ Distributed Shared Memory

☐ Message Passing/Queueing

☐ Distributed Objects

☐ Tuple Spaces

Copyrig

ll Rights Reserved.

OpenSplice

Building Distributed Systems

Several abstractions have been proposed to build Distributed Systems:

☐ Distributed Shared Memory

☐ Message Passing/Queueing

☐ Distributed Objects

☐ Tuple Spaces

Copyrig

ll Rights Reserved.

OpenSplice

Message Queueing☐ Message Queueing, as

suggested by its name, provide a coordination abstraction based on the exchange of messages between distributed processes

☐ Message delivery is often mediated by brokers

☐ A Message is composed by a header and a body, where the body is opaque (often binary) data

Broker

BodyHeader

Message

Copyrig

ll Rights Reserved.

OpenSplice

Tuple Space☐ Tuple Space provide the

abstraction of a Global Tuple Space that can be used by autonomous applications to coordinate by inserting (out), reading (rd) and removing (in) tuples

☐ The Tuple Space model was initially introduced by Gelernter and Carrero’s Linda and extended over the years

<a,b,c>

<x,y, <j, k>><x,y, <j, k>>

out(<a,b,c>) rd(<a,_,_>)

in(<15,_>)eval(...)

Copyrig

ll Rights Reserved.

OpenSplice

Tuple Spaces vs Messaging☐ Tuple Spaces provides spatial

and temporal decoupling along with content awareness

☐ Tuples remain in the tuple space up to when they are not removed

☐ Tuple Spaces allow application to share information

☐ Overall Tuple Spaces provide a high-level abstraction for building distributed systems

☐ Messaging doesn’t provide content awareness and usually provides limited temporal decoupling

☐ Messages usually disappear once they are delivered to consumers

☐ Messaging allows application to exchange messages. Information has to be reconstructed from messages

☐ Messaging provides a relatively low-level abstraction for building distributed systems

Copyrig

ll Rights Reserved.

OpenSplice

Why is Messaging so Popular?At this point you may ask yourself:

If Tuple Spaces are so cool, why are messaging technologies so popular?

The answer is actually pretty simple:

Tuple Spaces where originally quite inefficient and this lead people to adopt different approaches, such as messaging and distributed objects.

Nowadays, however, there are plenty of technologies deriving from the tuple space concept that have very high performance (often superior than messaging).

Copyrig

ll Rights Reserved.

OpenSplice

Why is Messaging so Popular?

At this point you may ask:

OK, that makes sense. Yet there are still plenty of people using messaging technologies. Why?

The answer is again pretty simple:

The adoption of Tuple Space inspired technologies is growing steadily, but many people are still using messaging since this is what they are familiar with. But enlightenment will eventually hit them ;-)

OpenSplice

DDS Ba

DDSics

Copyrig

ll Rights Reserved.

OpenSplice

Data Distribution Service

☐ Topics: Unit of data sharing

☐ DataWriters: data producers

☐ DataReaders: data consumers

DDS provides a relaxed Tuple Space Abstraction abstraction based on:

DDS Global Data Space

TopicA

TopicBTopicC

TopicD

Data Writer

Data Reader

For Real-Time Systems

Copyrig

ll Rights Reserved.

OpenSplice

Data Distribution Service

☐ DataWriters and DataReaders are automatically and dynamically matched by the DDS Dynamic Discovery

☐ A rich set of QoS allows to control existential, temporal, and spatial properties of data

DDS Global Data Space

TopicA

TopicBTopicC

TopicD

Data Writer

Data Reader

For Real-Time Systems

Copyrig

ll Rights Reserved.

OpenSplice

Fully Distributed Tuple Space☐ DDS Implements a fully

distributed Tuple Space with relaxed consistency.

☐ DDS provides eventual consistency as opposed to strong consistency

☐ In addition, DDS provides a local “take” and a global “dispose” in place of the “in” operation

<a,b,c>

<x,y, <j, k>>

out(<a,b,c>) rd(<a,_,_>)

in(<15,_>)eval(...)

Traditional Tuple Space

DDS Data SpaceLogical Global Data Space

Physical Local Data Space

The local Data Space is a projection of the Logical Global Data Space. This projection reflects the interest of the given

application.

The Global Data Space is an abstraction in DDS built by

the composition of Local Data Spaces.

Copyrig

ll Rights Reserved.

OpenSplice

What DDS Provides☐ Location Transparency => Dynamic Discovery

☐ Anonymity and Temporal Decoupling => Tuple Space

☐ Data Centricity => Tuple Space

☐ Dependability => Fault-Tolerant and Secure

☐ Scalability => Fully distributed architecture easy to scale-out

☐ Portability => OS and Programming Language Independence

☐ Interoperability => Standardized Wire Protocol

Copyrig

ll Rights Reserved.

OpenSplice

DDS Topics☐ A Topic defines a class of

streams ☐ A Topic has associated a user

defined extensible type and a set of QoS policies

☐ The Topic name, type and QoS defines the key functional and non-functional invariants

☐ Topics can be discovered or locally defined

DURABILITY,DEADLINE,PRIORITY,

“TVehicleDynamics”

TopicTypeName

struct VehicleDynamics { long vid; long x; long y; long dx; long dy;};#pragma keylist VehicleDynamics vid

VehicleDynamics

Copyrig

ll Rights Reserved.

OpenSplice

Instances

☐ Each unique key value identifies a unique topic instance

☐ DDS not only demultiplexes “streams” but provides also lifecycle information

☐ A DDS DataWriter can write multiple instances struct Person {

long ssn; String name; String surname; };#pragma keylist Person ssn

<101,Leslie, Lamport >

<010,Edsger, Dijkstra>

<110,Haskell, Curry>

Copyrig

ll Rights Reserved.

OpenSplice

Anatomy of a DDS Application

Copyrig

ll Rights Reserved.

OpenSplice

Running App

☐ During this webcast we’ll be looking at a few DDS aspects through the iShapes applications

☐ The advantage of targeting the iShapes application is that we can actually “see” what our application will do

Copyrig

ll Rights Reserved.

OpenSplice

OpenSplice Mobile

Copyrig

ll Rights Reserved.

OpenSplice

In Action...

Copyrig

ll Rights Reserved.

OpenSplice

iShapes Information Model“Circle”

TopicType

NameQo

ShapeType

IDLstruct ShapeType { string color; long x; long y; long shapesize;};#pragma keylist ShapeType color

“Square”

TopicType

ShapeType

“Triangle”

TopicType

ShapeType

Copyrig

ll Rights Reserved.

OpenSplice

Anatomy of a DDS ApplicationDomain Participant

[ISO C++ API]

auto dp = DomainParticipant(domainId);

Copyrig

ll Rights Reserved.

OpenSplice

Topics

// Create a Topicauto topic = Topic<ShapeType>(dp, “Circle”);

[ISO C++ API]

Copyrig

ll Rights Reserved.

OpenSplice

Topics

// Create a Publisherauto pub = Publisher(dp);

Publisher

// Create a Subscriberauto sub = Subscriber(dp);

Subscriber

[ISO C++ API]

Copyrig

ll Rights Reserved.

OpenSplice

Data Writer

Topics

// Create a DataWriterauto writer = DataWriter<ShapeType>(pub, topic);

// Create a Publisherauto pub = Publisher(dp);

Publisher

// Create a Subscriberauto sub = Subscriber(dp);

Subscriber

Data Reader // Create a DataWriterauto reader = DataReader<ShapeType>(sub, topic);

[ISO C++ API]

Copyrig

ll Rights Reserved.

OpenSplice

Putting it all together 1 int main(int argc, char* argv[]) { 2 try { 3 DomainParticipant dp(0); 4 Topic<ShapeType> topic(dp, "Circle"); 5 Publisher pub(dp); 6 DataWriter<ShapeType> dw(pub, topic); 7 10 for (int i = 0; i < N; ++i) {11 ShapeType bc = {"RED", i, i, 60};12 ShapeType rc = {"BLUE", N-i, N-i, 60};13 dw.write(bc);14 // You can also write with streaming operators!15 dw << rc;16 std::cout << "." << std::flush;17 usleep(sleep_time);18 }19 20 } catch (const dds::core::Exception& e) {21 std::cout << e.what() << std::endl;22 }23 return 0;24 }

Copyrig

ll Rights Reserved.

OpenSplice

Anatomy of a DDS Application

Domain Participant

Data Writer

Topics

// Create a DataWriterDataWriter<ShapeType> writer = pub.createDataWriter<ShapeType>(topic);

DomainParticipant dp = dpf.createParticipant(domainId);

// Create a PublisherPublisher pub = dp.createPublisher();

// Create a TopicTopic<ShapeType> topic = dp.createTopic(“Circle”,ShapeType.class);

Publisher

// Create a SubscriberSubscriber sub = dp.createSubscriber();

Subscriber

Data Reader

[Java 5 API] BootstrapServiceEnvironment env = ServiceEnvironment.createInstance(this.getClass.getClassLoader);DomainParticipantFactory dpf = DomainParticipantFactory.getInstance(env)

// Create a DataWriterDataReader<ShapeType> reader = sub.createDataReader<ShapeType>(topic);

Copyrig

ll Rights Reserved.

OpenSplice

Anatomy of a DDS ApplicationDependency Injection through Implicits

Data Writer

Topics

// Create a DataWriterval writer = DataWriter[ShapeType](topic)

[Scala API]

import dds.config.DefaultEntities._

// Create a Topicval topic = Topic[ShapeType](“Circle”)

Data Reader // Create a DataWriterval reader = DataReader[ShapeType](topic)

OpenSplice

Writing Data

Copyrig

ll Rights Reserved.

OpenSplice

Writing Data in DDS

writer.write(SpapeType(“RED”, 10, 20, 30));

// -- or -- writer << SpapeType(“RED”, 10, 20, 30);

[ISO C++ API]

Copyrig

ll Rights Reserved.

OpenSplice

Writing Data in DDS[Java 5 API]

writer.write(new SpapeType(“RED”, 10, 20, 30));

Copyrig

ll Rights Reserved.

OpenSplice

Writing Data in DDS[Scala API]

writer.write(new ShapeType(“RED”, 10, 20, 30))

// -- Or equivalently:

writer write (new ShapeType(“RED”, 10, 20, 30))

OpenSplice

Reading Data

Copyrig

ll Rights Reserved.

OpenSplice

Reading Data// Read Dataauto data = reader.read();

// -- you can also read with streaming operatorsauto data = reader << read();

// Print the received data on the consolestd::for_each(samples.data().begin(), samples.data.end(), demo::printShape)

[ISO C++ API]

Copyrig

ll Rights Reserved.

OpenSplice

Reading Data// You can read data on your favorite container using iteratorsauto samples = std::vector<Samples<ShapeType>>(MAX_SHAPES);

// Read data into the provided containerreader.read(samples.begin(), MAX_SHAPES);

// Print the received data on the consolestd::for_each(samples.data().begin(), samples.data.end(), demo::printShape)

[ISO C++ API]

Copyrig

ll Rights Reserved.

OpenSplice

Reading Data

// Read Data Sample.Iterator<ShapeType> iterator = reader.read(); while (iterator.hasNext()) { // Assuming ShapeType ovrrides toString

out.println(iterator.next().getData()); }

[Java API]

Copyrig

ll Rights Reserved.

OpenSplice

Reading Data

// Read Data reader.read foreach { s => println(s.data()) }

[Scala API]

OpenSplice

Selecting Samples

OpenSplice

Data Selectors

Copyrig

ll Rights Reserved.

OpenSplice

Cherry Picking in DDS

☐ DDS provides some very flexible mechanisms for selecting the data to be read:☐ Data Content☐ Data Status

☐ These mechanisms are composable

OpenSplice

Content-Based Data Selection

Copyrig

ll Rights Reserved.

OpenSplice

Filters and Queries☐ DDS Filters allow to control what gets

into a DataReader cache

☐ DDS Queries allow to control what gets out of a DataReader cache

☐ Filters are defined by means of ContentFilteredTopics

☐ Queries operate in conjunction with read operations

☐ Filters and Queries are expressed as SQL where clauses

DataReader Cache

DataReader

...... ... ...

Filter

Application

Copyrig

ll Rights Reserved.

OpenSplice

Querystruct VehicleDynamics { long vid; long x; long y; long dx; long dy;};#pragma keylist VehicleStatus vid

// Define the query and the parameters std::vector<std::string> p;p.push_back("100");p.push_back("100");dds::core::Query q("x < %0 AND y < %1", p.begin(), p.end());

auto data = reader .select() .content(q) .read();

[DDS C++ API 2010]

Copyrig

ll Rights Reserved.

OpenSplice

Instances☐ DDS provides a very efficient way of reading data belonging to a

specific Topic Instance

☐ Obviously, one could use queries to match the key’s value, but this is not as efficient as the special purpose instance selector

// C++auto data = reader .select() .instance(handle) .read();

// Scalaval data = reader read(handle)

OpenSplice

State-Based Selection

Copyrig

ll Rights Reserved.

OpenSplice

Sample, Instance, and View State☐ The samples included in the DataReader cache have associated

some meta-information which, among other things, describes the status of the sample and its associated stream/instance

☐ The Sample State (READ, NOT_READ) allows to distinguish between new samples and samples that have already been read

☐ The View State (NEW, NOT_NEW) allows to distinguish a new instance from an existing one

☐ The Intance State (ALIVE, NOT_ALIVE_DISPOSED, NOT_ALIVE_NO_WRITERS) allows to track the life-cycle transitions of the instance to which a sample belongs

Copyrig

ll Rights Reserved.

OpenSplice

State Selector in Action

// Read only new samplesauto data = reader .select() .state(status::DataState::new_data()) .read()

// Read any samples from live instancesauto data = reader .select() .state(status::DataState::any_data()) .read()

C++// Read only new samplesval data = reader read

// Read any samples from live instancesval data = reader read(SampleSelector.AnyData)

Copyrig

ll Rights Reserved.

OpenSplice

Putting all Together

☐ Selectors can be composed in a flexible and expressive manner

C++auto data = reader ! .select() .instance(handle) ! ! .state(status::DataState::new_data()) ! ! .content(q) .read();

OpenSplice

Copyrig

ll Rights Reserved.

OpenSplice

QoS Model☐ QoS-Policies control local and

end-to-end properties of DDS entities

☐ Local properties controlled by QoS are related resource usage

☐ End-to-end properties controlled by QoS are related to temporal and spatial aspects of data distribution

☐ Some QoS-Policies are matched based on a Request vs. Offered Model thus QoS-enforcement

Publisher

DataWriter

writes

DataWriter

Typewrites

Subscriber

DataReaderreads

QoS QoS

QoS matching

......

QoS QoS

Type Matching

DomainParticipant DomainParticipant

QoS QoS

Copyrig

ll Rights Reserved.

OpenSplice

QoS PoliciesQoS Policy Applicability RxO Modifiable

USER_DATATOPIC_DATA

GROUP_DATADURABILITYDURABILITY

SERVICEHISTORY

PRESENTATIONRELIABILITYPARTITION

DESTINATION ORDER

LIFESPAN

DP, DR, DW N Y

ConfigurationT N Y ConfigurationP, S N Y

Configuration

T, DR, DW Y N

Data AvailabilityT, DW N N

Data Availability

T, DR, DW N N

Data Availability

P, S Y N

Data Delivery

T, DR, DW Y N

Data DeliveryP, S N Y Data DeliveryT, DR, DW Y N

Data Delivery

T, DW N Y

Data Delivery

[T: Topic] [DR: DataReader] [DW: DataWriter] [P: Publisher] [S: Subscriber] [DP: Domain Participant]

Copyrig

ll Rights Reserved.

OpenSplice

QoS PoliciesQoS Policy Applicability RxO ModifiableDEADLINELATENCY BUDGET

TRANSPORT PRIORITY

TIME BASED FILTER

OWNERSHIPOWNERSHIP STRENGTHLIVELINESS

T, DR, DW Y Y

Temporal/Importance

Characteristics

T, DR, DW Y YTemporal/

Importance Characteristics

T, DW N YTemporal/

Importance Characteristics

DR N Y

Temporal/Importance

Characteristics

T, DR, DW Y NReplicationDW N Y Replication

T, DR, DW Y N Fault-Detection

[T: Topic] [DR: DataReader] [DW: DataWriter] [P: Publisher] [S: Subscriber] [DP: Domain Participant]

Copyrig

ll Rights Reserved.

OpenSplice

Setting QoS Policies

C++// Setting Partition QoS-Policy on Publisherqos::PublisherQos pubQos;pubQos << policy::Partition("Partition");Publisher pub(dp, pubQoS);

// Setting various QoS-Policy on a Topicqos::TopicQos tqos;tqos << policy::Reliability::Reliable() << policy::Durability::Transient() << policy::History::KeepLast(5);

Topic<VehicleDynamics> topic(dp,"Partition", tqos);

OpenSplice

Ipse Dixit

For non latin speakers: "He, himself, said it"

Copyrig

ll Rights Reserved.

OpenSplice

The “Ipse Dixit” Dejavu☐ As an historical reference“Ipse

Dixit” was used by the Holy Inquisition to push back any argument that would contrast with established knowledge

☐ A famous example was the argument around the Geo-Centric model of the Solar System

☐ What has this to do with Middleware... Let me show you!

Copyrig

ll Rights Reserved.

OpenSplice

Copyrig

ll Rights Reserved.

OpenSplice

What’s Wrong with this Paper?☐ When looking at DDS the

Authors did not refer to the state of the art

☐ DDS is discarded as being too complicated (really?!?!) -- they did not check out the new APIs available since 2010

☐ Beyond that, I think the authors where comparing a bit apples and oranges since DDS provides far more than ZeroMQ!

Copyrig

ll Rights Reserved.

OpenSplice

DDS vs. 0MQ -- What’s Simpler?import dds.config.DefaultEntities._

// Create a DataWriterval writer = DataWriter[ShapeType](topic)

// Create a DataWriterwriter write (new ShapeType(“RED”, 1,2,3))

import zmqimport time

// Create a Contextcontext = zmq.Context()

// Create a Socketpublisher = context.socket (zmq.PUB)

// Bind the socket to a namepublisher.bind ("ipc://nasdaq-feed")

// Send Datapublisher.send (message)

Sending Data

Copyrig

ll Rights Reserved.

OpenSplice

DDS vs. 0MQ -- What’s Simpler?import zmqimport time

// Create a Contextcontext = zmq.Context()

// Create a Socketsubscriber = context.socket (zmq.SUB)

// !!! MANUALLY CONNECT IT !!! What if your // topology changes?!?subscriber.connect ("tcp://192.168.55.112:5556")subscriber.connect ("tcp://192.168.55.201:7721")

// Define subscrptionsubscriber.setsockopt (zmq.SUBSCRIBE, "NASDAQ")

// Receive messagemessage = subscriber.recv()// ... Now decode the msg and do something // with it

Receiving Data

import dds.config.DefaultEntities._

// Create a DataReaderval reader = DataReader[ShapeType](topic)

// Create a DataWriterreader.read() foreach(s => println(s.data))

Copyrig

ll Rights Reserved.

OpenSplice

Concluding Remarks

☐ DDS provides a powerful and feature-rich abstraction for building distributed systems

☐ This technology is widely used in mission and business critical systems and it being swiftly adopted in data-centric/big-data systems

Copyrig

ll Rights Reserved.

OpenSplice

Copyrig

ll Rights Reserved.

OpenSplice

¥@prismtech

¥@acorsaro

¥youtube.com/opensplicetube ¥slideshare.net/angelo.corsaro

¥ crc@prismtech.com¥sales@prismtech.com

¥opensplice.com ¥forums.opensplice.org

¥opensplice.org ¥opensplicedds@prismtech.com

:: Connect with Us ::

DDS Made Simple

Technology

Transcript of DDS Made Simple

ISO Made Simple

Metadata Made Simple

Streaming Made Simple

GRAMMAR MADE SIMPLE

WATERPROOFING MADE SIMPLE

ATO Made Simple

Subrogation Presentation: Made Whole Made Simple

Roleplays made simple

CITES made simple

CE Made Simple

Business Startup Made Simple Business Startup Made Simple

Options Made Simple

SQL Made Simple

Simple Made

Simple Subnetting Made Simple

Fermentation Made Simple

signup made simple

DDS PROVES TO BE Simple & Reliable

Made AIX Simple

Excel Made Simple