PrismTech Vortex Tutorial Part 1

Cop

yrig

ht P

rism

Tech

, 201

4

Vortex Architecture

Security Model

Cop

yrig

ht P

rism

Tech

, 201

4

Support for transport level security via TLS and DTLS

Support for fine grained access control

Access control plug-in compliant with DDS Security

Vortex Security Model

Arthur Dent

Ford Prerfect

Zaphod Beeblebrox

Marvin

Trillian

A(r,w), B(r)

A(r,w), B(r,w), X(r)

*(r,w)

*(r)

A(r,w), B(r,w), C(r,w)

Ford Prerfect

Zaphod Beeblebrox

Trillian

Marvin

A

B

A,BX

*

A,B,C

Identity Access RightsSessions are authenticated and communication is encrypted

Only the Topic included as part of the access rights are visible and accessible

Use Cases

Cop

yrig

ht P

rism

Tech

, 201

4

Smart Grids

AnalyticsBig Data Store

Cop

yrig

ht P

rism

Tech

, 201

4

E-Health

Hospital A Private Cloud

AnalyticsBig Data Store

Hospital B

Cop

yrig

ht P

rism

Tech

, 201

4

Smart City

http://bit.ly/connected-boulevard

http://bit.ly/connected-boulevard

Cop

yrig

ht P

rism

Tech

, 201

4

SESAR

PENS

PENS: Pan European Network Service FDP: Flight Data Processor CWP: Controller Working Position

IPv6 network SSM support

FDPFDPFDP

CWP

Tower

DeviceFDPFDPFDP Tower

Device

CWP

FDPFDPFDP Tower

Device

CWP

FDPFDPFDP

CWP

Tower

Device

EU Nation 4

EU Nation 1

EU Nation 2EU Nation 3

Building ChirpIt

Cop

yrig

ht P

rism

Tech

, 201

4

To explore the various features provided by the Vortex platform we will be designing and implementing a micro-blogging platform called ChirpIt. Specifically, we want to support the following features:

ChirpIt users should be able to “chirp”, “re-chirp”, “like” and “dislike” trills as well as get detailed statistics

The ChirpIt platform should provide information on trending topics — identified by hashtags — as well as trending users

Third party services should be able to flexibly access slices of produced trills to perform their own trend analysis

ChirpIt should scale to millions of users

ChirpIt should be based on a Lambda Architecture

ChirpIt Requirements

Cop

yrig

ht P

rism

Tech

, 201

4

ChirpIt Architecture

analytics

data centre

chirp

emotionsstats

batch layer

chirpit apps

Cloud Messaging

serving layer

chirp

emotionsstats

master dataset

view view view

chirp

emotionsstats

3rd party svcs

Speed layer

The Data Distribution Service

Cop

yrig

ht P

rism

Tech

, 201

4

DDS provides a Global Data Space abstraction that allows applications to autonomously, anonymously, securely and efficiently share data

DDS’ Global Data Space is fully distributed (decentralised), highly efficient and scalable

Data Distribution Service (DDS)

DDS Global Data Space

...

Data Writer

Data Reader

Data Writer

TopicAQoS

TopicBQoS

TopicCQoS

TopicDQoS

Cop

yrig

ht P

rism

Tech

, 201

4DataWriters and DataReaders are automatically and dynamically matched by the DDS Discovery

A rich set of QoS allows to control existential, temporal, and spatial properties of data

Data Distribution Service (DDS)

DDS Global Data Space

...

Data Writer

Data Reader

Data Writer

TopicAQoS

TopicBQoS

TopicCQoS

TopicDQoS

Cop

yrig

ht P

rism

Tech

, 201

4

Elegant and High Level Data Sharing Abstraction

Polyglot and platform independent

• Java, Scala, C, C++, C#, JavaScript, CoffeeScript etc.

• Android, Windows, Linux, VxWorks, etc.

Peer-to-Peer by nature, Brokered when useful

Key Highlights

Cop

yrig

ht P

rism

Tech

, 201

4

Content and Temporal Filtering (both sender and receiver filtering supported)

Queries

20+ QoS to control existential, temporal, and spatial properties of data

Key Highlights

Cop

yrig

ht P

rism

Tech

, 201

4

High Performance and Scalable

• ~30 usec latency*

• 7M msgs/sec node-to-node throughput*

Key Highlights

(*) Performance measured on Linux Host with i7 processor on a 1Gbps Ethernet network

Decomposing DDS

Information Organisation

Cop

yrig

ht P

rism

Tech

, 201

4

DDS data lives within a domain

A domain is identified with a non negative integer, such as 1, 3, 31

The number 0 identifies the default domain

A domain represent an impassable communication plane

Domain

DDS Domain

Cop

yrig

ht P

rism

Tech

, 201

4

Partitions are the mechanism provided by DDS to organise information within a domain

Access to partitions is controlled through QoS Policies

Partitions are defined as strings: - “system:telemetry” - “system:log” - “data:row-2:col-3”

Partitions addressed by name or regular expressions: - ”system:telemetry” - “data:row-2:col-*”

Partitions

Information Definition

Cop

yrig

ht P

rism

Tech

, 201

4

A Topic defines a domain-wide information’s class

A Topic is defined by means of a (name, type, qos) tuple, where

• name: identifies the topic within the domain

• type: is the programming language type associated with the topic. Types are extensible and evolvable

• qos: is a collection of policies that express the non-functional properties of this topic, e.g. reliability, persistence, etc.

Topic

TopicTypeName

QoS

...

TopicAQoS

TopicBQoS

TopicCQoS

TopicDQoS

Cop

yrig

ht P

rism

Tech

, 201

4

As explained in the previous slide a topic defines a class/type of information

Topics can be defined as Singleton or can have multiple Instances

Topic Instances are identified by means of the topic key

A Topic Key is identified by a tuple of attributes -- like in databases

Remarks: - A Singleton topic has a single domain-wide instance - A “regular” Topic can have as many instances as the number of different key

values, e.g., if the key is an 8-bit character then the topic can have 256 different instances

Topic and Instances

ChirpIt Data Model

Cop

yrig

ht P

rism

Tech

, 201

4

Chirp Actions union ChirpBody switch (ChirpActionKind) { case CHIRP_KIND: string<128> chirp; case RECHIRP_KIND: string<128> chirp; string user; case LIKE_KIND: string luser; case DISLIKE_KIND: string duser; }; ! struct ChirpAction { ChirpHeader header; ChirpBody body; }; #pragma keylist ChirpAction header.id.cid

struct ChirpId { string uid; string cid; }; ! enum ChirpActionKind { CHIRP_KIND, RECHIRP_KIND, LIKE_KIND, DISLIKE_KIND }; struct ChirpHeader { ChirpId id; Location location; unsigned long long timestamp; ChirpActionKind kind; };

Cop

yrig

ht P

rism

Tech

, 201

4

ChirpIt Statistics

struct ChirpStats { ChirpId id; unsigned long rechirps; unsigned long likes; }; #pragma keylist ChirpStats id.cid

struct UserStats { string userId; unsigned long followers; unsigned long chirps; unsigned long followed; }; #pragma keylist UserStats userId

Cop

yrig

ht P

rism

Tech

, 201

4

User Actions

struct enum UserActionKind { FOLLOW, UNFOLLOW }; struct UserAction { UserActionKind kind; string fromUser; string toUser; };

Cop

yrig

ht P

rism

Tech

, 201

4

App Config

typedef sequence<string> Partitions; ! struct AppConfig { string userId; Partitions ps; unsigned short history; }; #pragma keylist UserConfig userId

ChirpIt Information Scoping

Cop

yrig

ht P

rism

Tech

, 201

4

To make the system scale properly we need to ensure that the chirps, rechirps, etc., are distributed only to the interested parties

As we have a single topic representing chirp actions, how can we ensure that information is received only by followers?

DDS Provides two mechanisms:

- Partitions: Allow to create information scopes

- Content Filters: Allow to receive the data that satisfy certains predicates

Scoping Information

Cop

yrig

ht P

rism

Tech

, 201

4

ChirpIt Architecture

ChirpAction

@wolverine

@drxchirp:wolverine

ChirpAction

@drx follows @wolverine

Cop

yrig

ht P

rism

Tech

, 201

4

@drx

ChirpAction

ChirpIt Architecture

data centre

ChirpAction

@wolverine

chirp:wolverine

ChirpAction

@magneto

chirp:magneto

@drx follows @wolverine and @magneto

Cop

yrig

ht P

rism

Tech

, 201

4

In summary partitions are used to scope information

Each user will “join” a partition for each followed party

Example:

- If @drx follows @magneto, @wolverine, @cyclops and @mistique then he will receive ChirpActions from the partitions:

• chirp:@magneto, chirp:@wolverine, chirp:@cyclops and chirp:@mistique

- On the other hand, regardless of the people that follows @wolverine he will always and only write ChirpActions in the partitions

• chirp:@wolverine

Using Partitions to Scope Chirps

Producing Information

Cop

yrig

ht P

rism

Tech

, 201

4

A DataWriter (DW) is a strongly typed entity used to produce samples for one or more instances of a Topic, with a given QoS

Conceptually, the DataWriter QoS should be the same as the Topic QoS or more stringent

However, DDS does enforce a specific relationship between the Topic and DataWriter QoS

DataWriter

DWTypeTopic

QoS

Cop

yrig

ht P

rism

Tech

, 201

4

The DataWriter controls the life-cycle of Topic Instances and allows to:

Define a new topic instance

Write samples for a topic instance

Dispose the topic instance

DataWriter

DWTypeTopic

QoS

Consuming Information

Cop

yrig

ht P

rism

Tech

, 201

4

A DataReader (DR) is a strongly typed entity used to access and/or consume samples for a Topic, with a given QoS

Conceptually, the DataReader QoS should be the same as the Topic QoS or less stringent

However, DDS does enforce a specific relationship between the Topic and DataReader QoS

DataReader

DRTypeTopic

QoS

Putting all Together

Cop

yrig

ht P

rism

Tech

, 201

4

DomainParticipant: Provides access to a data cloud -- called a domain in DDS

Topic: Domain-wide definition of a kind of Information

Publisher/Subscriber: Provide scope to data sharing through the concept of partitions DataReader/DataWriter: Allow to read/write data for a given topic in the partitions their Subscriber/Publisher are associated with.

DDS EntitiesDomain (e.g. Domain 123)

Domain Participant

Topic

Publisher

DataWrter

Subscriber

DataReader

Partition (e.g. “Telemetry”, “Shapes”, )

Topic Instances/Samples

TaTb

Tc

Tx

Ty

T1

T1 T3

ChirpIt Implementation — Chirping —

Cop

yrig

ht P

rism

Tech

, 201

4

ChirpOut App // Domain Participant defined by dependency injection // Define the topic val topic = Topic[ChirpAction](Config.ChirpActionTopic) // Define the publisher and link-‐it up to the “chirp:@user” partition val pubQos = PublisherQos() .withPolicy(Partition(Config.ChirpRootPartition + user)) implicit val pub = Publisher(pubQos) // Define the data writer and make it reliable and for the time being // Transient Local… Will refined this in the Part II val dwQos = DataWriterQos().withPolicies( Reliability.Reliable, Durability.TransientLocal) // Create the DataWriter val dw = DataWriter[ChirpAction](pub, topic, dwQos) // Chirp!!! val chirp = … dw.write(chirp)

Cop

yrig

ht P

rism

Tech

, 201

4

ChirpIn // Domain Participant defined by dependency injection // Define the topic val topic = Topic[ChirpAction](Config.ChirpActionTopic) // Define the subscribe and link-‐it up to the partition associated to // the user followed val subQos = PublisherQos() .withPolicy(Partition(followerList.map(Config.ChirpRootPartition + _)) implicit val sub = Subscriber(subQos) // Define the data reader and make it reliable and for the time being // Transient Local… Will refined this in the Part II val drQos = DataReaderQos().withPolicies( Reliability.Reliable, Durability.TransientLocal) // Create the DataWriter val dr = DataReader[ChirpAction](sub, topic, drQos)

Cop

yrig

ht P

rism

Tech

, 201

4

ChirpIn

// React to new Chirp dr listen { case DataAvailable(_) => { // take vs. read will become clear in Part II val samples = dr.take() // For the time being simply print-‐out the new chirps samples .filter(s => (s.getData != null)) .foreach(s => displayChirp) } }

Cop

yrig

ht P

rism

Tech

, 201

4

ChirpOut App// Define the topic — assuming the topic is registered with the server…!topic = new dds.Topic(0, 'Chirp', ‘com.ChirpIt’)!// Define the data writer Qos!dwQos = new dds.DataWriterQos(! dds.Reliability.Reliable, ! dds.Durability.TranslientLocal,! dds.Partition(userPartition))!!// Use an Option monad to maintain the writer!chirpWriter = z_.None!!// Create the writer when the runtime is connected!runtime.onconnect(() ->! dw = new dds.DataWriter(runtime, topic, dwQos)! chirpWriter = z_.Some(dw)!)!!// …!// Write chirps!chirp = …!chirpWriter.map((dw) -> dw.write(chirp))!

Cop

yrig

ht P

rism

Tech

, 201

4

ChirpIn App// Define the topic — assuming the topic is registered with the server…!topic = new dds.Topic(0, 'Chirp', ‘com.ChirpIt’)!// Define the data reader Qos!drQos = new dds.DataWriterQos(! dds.Reliability.Reliable, ! dds.Durability.TranslientLocal,! dds.Partition(followedList))!!// Use an Option monad to maintain the reader!chirpReader = z_.None!!// Create the writer when the runtime is connected!runtime.onconnect(() ->! dr = new dds.DataReader(runtime, topic, dwQos)! // Attach a listener! dr.addListener((chirp) -> diplayChirp(chirp))! chirpReader = z_.Some(dr)!)

Cop

yrig

ht P

rism

Tech

, 201

4

ChirpIt Thus Far…data centre

chirp

emotions

stats

chirp

emotions

stats

chirp

emotions

stats

chirp

emotions

stats

Live Demo

Cop

yrig

ht P

rism

Tech

, 201

4

$ ./chirpOut @wolverin chirp>> @magneto will win!$ ./chirpOut @magneto

@magneto>> @wolverine chirping while asleep… again!

Download the package zip or tar.gz package available at:

- https://dl.dropboxusercontent.com/u/19238968/webcast/2014/10/chirpit.zip

- https://dl.dropboxusercontent.com/u/19238968/webcast/2014/10/chirpit.tar.gz

Extract the package somewhere in your file-system and open a terminal at that location and run the app as shown below

ChirpIt Demo

$ ./chirpIn @magneto @wolverinse @mystique @wolverin>> @magneto will win! @magneto>> @wolverine chirping while asleep… again!

https://dl.dropboxusercontent.com/u/19238968/webcast/2014/10/chirpit.zip

https://dl.dropboxusercontent.com/u/19238968/webcast/2014/10/chirpit.tar.gz

Cop

yrig

ht P

rism

Tech

, 201

4

In Part II we extend our implementation to manage application configuration storage and analytics

To do so, we will need to introduce additional features provided by the DDS and the Vortex Platform, such as Durability and Integration with Big-Data Stores

What’s Next

Cop

yrig

ht P

rism

Tech

, 201

4

In this presentation we have performed the first step toward implementing a micro-blogging platform

The combination of Vortex Device and Vortex Cloud made it very easy to create and deploy our Internet-Scale, multi-device micro-blogging platform

Some of the DDS QoS, such as Durability have already shown how historical data, i.e. chirps made while we were offline — can be provided by the platform

The Part II will illustrate how the complete application fits together and what is the role of Vortex in the Lambda Architecture

Summary

Cop

yrig

ht P

rism

Tech

, 201

4

Online Resources

http://www.linkedin.com/pub/dir/Angelo/Corsaro

https://github.com/prismtech

http://www.prismtech.com/