Angelo  Corsaro,  PhD  Chief  Technology  Officer  

angelo.corsaro@prismtech.com

Vortex Tutorial Part I

Cop

yrig

ht P

rism

Tech

, 201

4

Vortex enables seamless, ubiquitous, efficient and timely data sharing across mobile, embedded, desktop, cloud and web applications

Vortex is based on the OMG DDS standard

The Vortex PlatformVortex Device

Tools

Integration

Vortex Cloud

MaaS

Cop

yrig

ht P

rism

Tech

, 201

4

Vortex Architecture

Security Model

Cop

yrig

ht P

rism

Tech

, 201

4

Support for transport level security via TLS and DTLS

Support for fine grained access control

Access control plug-in compliant with DDS Security

Vortex Security Model

Arthur Dent

Arthur Dent

Ford Prerfect

Zaphod Beeblebrox

Marvin

Trillian

A(r,w), B(r)

A(r,w), B(r,w), X(r)

*(r,w)

*(r)

A(r,w), B(r,w), C(r,w)

Ford Prerfect

Zaphod Beeblebrox

Trillian

Marvin

A

B

A,BX

*

*

A,B,C

Identity Access RightsSessions are authenticated and communication is encrypted

Only the Topic included as part of the access rights are visible and accessible

Use Cases

Cop

yrig

ht P

rism

Tech

, 201

4

Smart Grids

AnalyticsBig  Data  Store

Cop

yrig

ht P

rism

Tech

, 201

4

E-Health

Hospital A Private Cloud

AnalyticsBig  Data  Store

Hospital B

Cop

yrig

ht P

rism

Tech

, 201

4

Smart City

http://bit.ly/connected-boulevard

Cop

yrig

ht P

rism

Tech

, 201

4

SESAR

PENS

PENS: Pan European Network Service FDP: Flight Data Processor CWP: Controller Working Position

IPv6 network SSM support

FDPFDPFDP

CWP

Tower

DeviceFDPFDPFDP Tower

Device

CWP

FDPFDPFDP Tower

Device

CWP

FDPFDPFDP

CWP

Tower

Device

EU Nation 4

EU Nation 1

EU Nation 2EU Nation 3

Building ChirpIt

Cop

yrig

ht P

rism

Tech

, 201

4

To explore the various features provided by the Vortex platform we will be designing and implementing a micro-blogging platform called ChirpIt. Specifically, we want to support the following features:

ChirpIt users should be able to “chirp”, “re-chirp”, “like” and “dislike” trills as well as get detailed statistics

The ChirpIt platform should provide information on trending topics — identified by hashtags — as well as trending users

Third party services should be able to flexibly access slices of produced trills to perform their own trend analysis

ChirpIt should scale to millions of users

ChirpIt should be based on a Lambda Architecture

ChirpIt Requirements

Cop

yrig

ht P

rism

Tech

, 201

4

ChirpIt Architecture

analytics

data centre

chirp

emotionsstats

batch    layer

chirpit  apps

Cloud Messaging

serving    layer

chirp

emotionsstats

master  dataset

view view view

chirp

emotionsstats

3rd  party  svcs

Speed    layer

The Data Distribution Service

Cop

yrig

ht P

rism

Tech

, 201

4

DDS provides a Global Data Space abstraction that allows applications to autonomously, anonymously, securely and efficiently share data

DDS’ Global Data Space is fully distributed (decentralised), highly efficient and scalable

Data Distribution Service (DDS)

DDS Global Data Space

...

Data Writer

Data Writer

Data Writer

Data Reader

Data Reader

Data Reader

Data Reader

Data Writer

TopicAQoS

TopicBQoS

TopicCQoS

TopicDQoS

Cop

yrig

ht P

rism

Tech

, 201

4DataWriters and DataReaders are automatically and dynamically matched by the DDS Discovery

A rich set of QoS allows to control existential, temporal, and spatial properties of data

Data Distribution Service (DDS)

DDS Global Data Space

...

Data Writer

Data Writer

Data Writer

Data Reader

Data Reader

Data Reader

Data Reader

Data Writer

TopicAQoS

TopicBQoS

TopicCQoS

TopicDQoS

Cop

yrig

ht P

rism

Tech

, 201

4

Elegant and High Level Data Sharing Abstraction

Polyglot and platform independent

• Java, Scala, C, C++, C#, JavaScript, CoffeeScript etc.

• Android, Windows, Linux, VxWorks, etc.

Peer-to-Peer by nature, Brokered when useful

Key Highlights

Cop

yrig

ht P

rism

Tech

, 201

4

Content and Temporal Filtering (both sender and receiver filtering supported)

Queries

20+ QoS to control existential, temporal, and spatial properties of data

Key Highlights

Cop

yrig

ht P

rism

Tech

, 201

4

High Performance and Scalable

• ~30 usec latency*

• 7M msgs/sec node-to-node throughput*

Key Highlights

(*) Performance measured on Linux Host with i7 processor on a 1Gbps Ethernet network

Decomposing DDS

Information Organisation

Cop

yrig

ht P

rism

Tech

, 201

4

DDS data lives within a domain

A domain is identified with a non negative integer, such as 1, 3, 31

The number 0 identifies the default domain

A domain represent an impassable communication plane

Domain

DDS Domain

Cop

yrig

ht P

rism

Tech

, 201

4

Partitions are the mechanism provided by DDS to organise information within a domain

Access to partitions is controlled through QoS Policies

Partitions are defined as strings: - “system:telemetry” - “system:log” - “data:row-2:col-3”

Partitions addressed by name or regular expressions: - ”system:telemetry” - “data:row-2:col-*”

Partitions

Partitions

Information Definition

Cop

yrig

ht P

rism

Tech

, 201

4

A Topic defines a domain-wide information’s class

A Topic is defined by means of a (name, type, qos) tuple, where

• name: identifies the topic within the domain

• type: is the programming language type associated with the topic. Types are extensible and evolvable

• qos: is a collection of policies that express the non-functional properties of this topic, e.g. reliability, persistence, etc.

Topic

TopicTypeName

QoS

...

TopicAQoS

TopicBQoS

TopicCQoS

TopicDQoS

Cop

yrig

ht P

rism

Tech

, 201

4

As explained in the previous slide a topic defines a class/type of information

Topics can be defined as Singleton or can have multiple Instances

Topic Instances are identified by means of the topic key

A Topic Key is identified by a tuple of attributes -- like in databases

Remarks: - A Singleton topic has a single domain-wide instance - A “regular” Topic can have as many instances as the number of different key

values, e.g., if the key is an 8-bit character then the topic can have 256 different instances

Topic and Instances

ChirpIt Data Model

Cop

yrig

ht P

rism

Tech

, 201

4

Chirp Actions      union  ChirpBody  switch  (ChirpActionKind)  {          case  CHIRP_KIND:              string<128>  chirp;          case  RECHIRP_KIND:              string<128>  chirp;              string  user;          case  LIKE_KIND:              string  luser;          case  DISLIKE_KIND:              string  duser;          };  !                  struct  ChirpAction  {              ChirpHeader  header;              ChirpBody      body;          };          #pragma  keylist  ChirpAction  header.id.cid    

     struct  ChirpId  {              string  uid;              string  cid;          };  !        enum  ChirpActionKind  {              CHIRP_KIND,  RECHIRP_KIND,                LIKE_KIND,  DISLIKE_KIND          };                    struct  ChirpHeader  {              ChirpId  id;              Location  location;              unsigned  long  long  timestamp;              ChirpActionKind  kind;          };            

Cop

yrig

ht P

rism

Tech

, 201

4

ChirpIt Statistics

       struct  ChirpStats  {              ChirpId  id;              unsigned  long  rechirps;              unsigned  long  likes;          };          #pragma  keylist  ChirpStats  id.cid  

       struct  UserStats  {              string  userId;              unsigned  long  followers;              unsigned  long  chirps;              unsigned  long  followed;          };          #pragma  keylist  UserStats  userId  

Cop

yrig

ht P

rism

Tech

, 201

4

User Actions

       struct  enum  UserActionKind  {              FOLLOW,  UNFOLLOW          };                    struct  UserAction  {              UserActionKind  kind;              string  fromUser;              string  toUser;          };  

Cop

yrig

ht P

rism

Tech

, 201

4

App Config

       typedef  sequence<string>  Partitions;  !        struct  AppConfig  {              string  userId;              Partitions  ps;              unsigned  short  history;          };              #pragma  keylist  UserConfig  userId  

ChirpIt Information Scoping

Cop

yrig

ht P

rism

Tech

, 201

4

To make the system scale properly we need to ensure that the chirps, rechirps, etc., are distributed only to the interested parties

As we have a single topic representing chirp actions, how can we ensure that information is received only by followers?

DDS Provides two mechanisms:

- Partitions: Allow to create information scopes

- Content Filters: Allow to receive the data that satisfy certains predicates

Scoping Information

Cop

yrig

ht P

rism

Tech

, 201

4

ChirpIt Architecture

ChirpAction

@wolverine

@drxchirp:wolverine

ChirpAction

@drx  follows  @wolverine

Cop

yrig

ht P

rism

Tech

, 201

4

@drx

ChirpAction

ChirpIt Architecture

data centre

ChirpAction

@wolverine

chirp:wolverine

ChirpAction

@magneto

chirp:magneto

@drx  follows  @wolverine  and  @magneto

Cop

yrig

ht P

rism

Tech

, 201

4

In summary partitions are used to scope information

Each user will “join” a partition for each followed party

Example:

- If @drx follows @magneto, @wolverine, @cyclops and @mistique then he will receive ChirpActions from the partitions:

• chirp:@magneto, chirp:@wolverine, chirp:@cyclops and chirp:@mistique

- On the other hand, regardless of the people that follows @wolverine he will always and only write ChirpActions in the partitions

• chirp:@wolverine

Using Partitions to Scope Chirps

Producing Information

Cop

yrig

ht P

rism

Tech

, 201

4

A DataWriter (DW) is a strongly typed entity used to produce samples for one or more instances of a Topic, with a given QoS

Conceptually, the DataWriter QoS should be the same as the Topic QoS or more stringent

However, DDS does enforce a specific relationship between the Topic and DataWriter QoS

DataWriter

DWTypeTopic

QoS

Cop

yrig

ht P

rism

Tech

, 201

4

The DataWriter controls the life-cycle of Topic Instances and allows to:

Define a new topic instance

Write samples for a topic instance

Dispose the topic instance

DataWriter

DWTypeTopic

QoS

Consuming Information

Cop

yrig

ht P

rism

Tech

, 201

4

A DataReader (DR) is a strongly typed entity used to access and/or consume samples for a Topic, with a given QoS

Conceptually, the DataReader QoS should be the same as the Topic QoS or less stringent

However, DDS does enforce a specific relationship between the Topic and DataReader QoS

DataReader

DRTypeTopic

QoS

Putting all Together

Cop

yrig

ht P

rism

Tech

, 201

4

DomainParticipant: Provides access to a data cloud -- called a domain in DDS

Topic: Domain-wide definition of a kind of Information

Publisher/Subscriber: Provide scope to data sharing through the concept of partitions DataReader/DataWriter: Allow to read/write data for a given topic in the partitions their Subscriber/Publisher are associated with.

DDS EntitiesDomain (e.g. Domain 123)

Domain Participant

Topic

Publisher

DataWrter

Subscriber

DataReader

Partition (e.g. “Telemetry”, “Shapes”, )

Topic Instances/Samples

TaTb

Tc

Tx

Ty

T1

T1 T3

ChirpIt Implementation — Chirping —

Cop

yrig

ht P

rism

Tech

, 201

4

ChirpOut App            //  Domain  Participant  defined  by  dependency  injection                            //  Define  the  topic              val  topic  =  Topic[ChirpAction](Config.ChirpActionTopic)              //  Define  the  publisher  and  link-­‐it  up  to  the  “chirp:@user”  partition              val  pubQos  =  PublisherQos()                        .withPolicy(Partition(Config.ChirpRootPartition  +  user))              implicit  val  pub  =  Publisher(pubQos)              //  Define  the  data  writer  and  make  it  reliable  and  for  the  time  being              //  Transient  Local…  Will  refined  this  in  the  Part  II              val  dwQos  =  DataWriterQos().withPolicies(                  Reliability.Reliable,                  Durability.TransientLocal)              //  Create  the  DataWriter              val  dw  =  DataWriter[ChirpAction](pub,  topic,  dwQos)              //  Chirp!!!              val  chirp  =  …              dw.write(chirp)

Cop

yrig

ht P

rism

Tech

, 201

4

ChirpIn            //  Domain  Participant  defined  by  dependency  injection                            //  Define  the  topic              val  topic  =  Topic[ChirpAction](Config.ChirpActionTopic)              //  Define  the  subscribe  and  link-­‐it  up  to  the  partition  associated  to              //  the  user  followed              val  subQos  =  PublisherQos()                        .withPolicy(Partition(followerList.map(Config.ChirpRootPartition  +  _))              implicit  val  sub  =  Subscriber(subQos)              //  Define  the  data  reader  and  make  it  reliable  and  for  the  time  being              //  Transient  Local…  Will  refined  this  in  the  Part  II              val  drQos  =  DataReaderQos().withPolicies(                  Reliability.Reliable,                  Durability.TransientLocal)              //  Create  the  DataWriter              val  dr  =  DataReader[ChirpAction](sub,  topic,  drQos)  

Cop

yrig

ht P

rism

Tech

, 201

4

ChirpIn

//  React  to  new  Chirp  dr  listen  {        case  DataAvailable(_)  =>  {                      //  take  vs.  read  will  become  clear  in  Part  II                      val  samples  =  dr.take()                    //  For  the  time  being  simply  print-­‐out  the  new  chirps                      samples         .filter(s  =>  (s.getData  !=  null))                        .foreach(s  =>  displayChirp)                  }    }

Cop

yrig

ht P

rism

Tech

, 201

4

ChirpOut App// Define the topic — assuming the topic is registered with the server…!topic = new dds.Topic(0, 'Chirp', ‘com.ChirpIt’)!// Define the data writer Qos!dwQos = new dds.DataWriterQos(! dds.Reliability.Reliable, ! dds.Durability.TranslientLocal,! dds.Partition(userPartition))!!// Use an Option monad to maintain the writer!chirpWriter = z_.None!!// Create the writer when the runtime is connected!runtime.onconnect(() ->! dw = new dds.DataWriter(runtime, topic, dwQos)! chirpWriter = z_.Some(dw)!)!!// …!// Write chirps!chirp = …!chirpWriter.map((dw) -> dw.write(chirp))!

Cop

yrig

ht P

rism

Tech

, 201

4

ChirpIn App// Define the topic — assuming the topic is registered with the server…!topic = new dds.Topic(0, 'Chirp', ‘com.ChirpIt’)!// Define the data reader Qos!drQos = new dds.DataWriterQos(! dds.Reliability.Reliable, ! dds.Durability.TranslientLocal,! dds.Partition(followedList))!!// Use an Option monad to maintain the reader!chirpReader = z_.None!!// Create the writer when the runtime is connected!runtime.onconnect(() ->! dr = new dds.DataReader(runtime, topic, dwQos)! // Attach a listener! dr.addListener((chirp) -> diplayChirp(chirp))! chirpReader = z_.Some(dr)!)

Cop

yrig

ht P

rism

Tech

, 201

4

ChirpIt Thus Far…data centre

chirp

emotions

stats

chirp

emotions

stats

chirp

emotions

stats

chirp

emotions

stats

Live Demo

Cop

yrig

ht P

rism

Tech

, 201

4

$ ./chirpOut @wolverin chirp>> @magneto will win!$ ./chirpOut @magneto

@magneto>> @wolverine chirping while asleep… again!

Download the package zip or tar.gz package available at:

- https://dl.dropboxusercontent.com/u/19238968/webcast/2014/10/chirpit.zip

- https://dl.dropboxusercontent.com/u/19238968/webcast/2014/10/chirpit.tar.gz

Extract the package somewhere in your file-system and open a terminal at that location and run the app as shown below

ChirpIt Demo

$ ./chirpIn @magneto @wolverinse @mystique @wolverin>> @magneto will win! @magneto>> @wolverine chirping while asleep… again!

Cop

yrig

ht P

rism

Tech

, 201

4

In Part II we extend our implementation to manage application configuration storage and analytics

To do so, we will need to introduce additional features provided by the DDS and the Vortex Platform, such as Durability and Integration with Big-Data Stores

What’s Next

Cop

yrig

ht P

rism

Tech

, 201

4

In this presentation we have performed the first step toward implementing a micro-blogging platform

The combination of Vortex Device and Vortex Cloud made it very easy to create and deploy our Internet-Scale, multi-device micro-blogging platform

Some of the DDS QoS, such as Durability have already shown how historical data, i.e. chirps made while we were offline — can be provided by the platform

The Part II will illustrate how the complete application fits together and what is the role of Vortex in the Lambda Architecture

Summary

Cop

yrig

ht P

rism

Tech

, 201

4

Online Resources