#Pharo Days 2016 Data Formats and Protocols

Data Formats & Protocols

Sven Van Caekenberghe

Pharo Object World

Pharo Object World

• Object Memory + Virtual Machine

• Graphics + Interaction, OS + Libraries

• Tools, IDE

Dream Environment

Pharo

• One simple language used everywhere

• True open source for all parts

• Understandable from high to low level

Island ?

Outside World

• Infinitely larger

• Our world is part of it

• We need to interact with it & affect it

Bridges

Types of Bridges

• Foreign Function Interface

• OS (Sub) Process

• Network Services

FFI

• Link to existing C libraries

• Huge availability, high quality

• Reuse, NIH

FFI

• Complex

• Opaque, not Object Oriented

• Not always that efficient, not always good fit

OS (Sub) Process

• Execute arbitrary programs as sub processes

• Communicate via standard input/output/error

• Huge availability, high quality, reuse, NIH

OS (Sub) Process

• Large overhead

• Opaque, not Object Oriented

• Awkward interface, Not always good fit

Network Services

• Connect to some service

• Send requests and receive responses

• Local, LAN, WAN

Network Services

• Fully implemented in Pharo, Open standards

• Client/server, Distributed, IOT, DB

• Marshalling & communication overhead

Building Bridges

Building Bridges

• What ? - Data Formats

• How ? - Protocols

Data Format

• Given a stream

• Read/parse a stream into objects

• Write objects to a stream

Protocol

• Communication channel

• Network stream

• The conversation itself

Basics

Streams

Types of Streams

• Binary Streams

• Character Streams

Binary Streams

• File Stream

• Socket Stream

Character Streams

• One of the simplest data formats

• ASCII, Latin1, UTF-8, UTF-16, UTF-32

• Fully covered & implemented in Pharo

Character Streams

• Orthogonal concern

• Compose, Wrap

• A solved issue

| array readStream |

array := ByteArray streamContents: [ :out | (ZnCharacterWriteStream on: out encoding: #iso88591) nextPutAll: 'Les élèves français' ].

readStream := array readStream.

(ZnCharacterReadStream on: readStream encoding: #iso8859) upToEnd.

-> 'Les élèves français'

Referencehttp://files.pharo.org/books/enterprise-pharo/book/Zinc-

Encoding-Meta/Zinc-Encoding-Meta.html

http://files.pharo.org/books/enterprise-pharo/book/Zinc-Encoding-Meta/Zinc-Encoding-Meta.html

Standard Protocols

• File Input/Output

• HTTP(S)

• Sub Process Standard Input/Output/Error

Data Formats

Internal Formats

• FUEL (binary)

• STON (textual)

• Very cool, very useful

• Limited to the Pharo World

Example

PVTRecord

• timestamp <DateAndTime>

• id <String>

• position <longitude@latitude> <Float@Float>

• speed <Integer>

Example

• as CSV

• as JSON

• as XML

CSV

• textual

• 1 record per line (any EOL)

• fields separated by delimiter

• fixed number of fields

CSV

• optionally quoted

• untyped strings

• conventional header

id,time,long,lat,speedv1,2016-03-29T15:21:00+00:00,5.3403835,50.918625,37v1,2016-03-29T15:22:00+00:00,5.3397698,50.915192,27v1,2016-03-29T15:23:00+00:00,5.3415751,50.911633,47

Parsing CSV

(FileLocator desktop / 'pvtrecords.csv')readStreamDo: [ :in |

(NeoCSVReader on: in) upToEnd ].


(NeoCSVReader on: in) skipHeader;addField;addFieldConverter: [ :string |

DateAndTime fromString: string ];addFloatField;addFloatField;addIntegerField;upToEnd ].


(NeoCSVReader on: in) skipHeader;recordClass: PVTRecord;addField: #identification:;addField: #timestamp: converter: [ :string |

DateAndTime fromString: string ];addFloatField: #longitude:;addFloatField: #latitude:;addIntegerField: #speed:;upToEnd ].

Generating CSV

(FileLocator desktop / 'pvtrecords2.csv')writeStreamDo: [ :out |

(NeoCSVWriter on: out)writeHeader: #(id time long lat speed);nextPutAll: ( {

PVTRecord one. PVTRecord two. PVTRecord three } collect: [ :each | { each identification. each timestamp. each longitude. each latitude. each speed } ] ) ].

"id","time","long","lat","speed""v1","2016-03-29T15:21:00+00:00","5.3403835","50.918625","37""v1","2016-03-29T15:22:00+00:00","5.3397698","50.915192","27""v1","2016-03-29T15:23:00+00:00","5.3415751","50.911633","47"

(FileLocator desktop / 'pvtrecords2.csv')writeStreamDo: [ :out |

(NeoCSVWriter on: out)writeHeader: #(id time long lat speed);addFields: #(

identification timestamp longitude latitude speed);

nextPutAll: { PVTRecord one. PVTRecord two. PVTRecord three } ].

Referencehttp://files.pharo.org/books/enterprise-pharo/book/

NeoCSV/NeoCSV.html

http://files.pharo.org/books/enterprise-pharo/book/NeoCSV/NeoCSV.html

JSON

• Textual

• Maps & Lists

• Numbers, Strings, Booleans, null

JSON

• Simple, universal, partially self describing

• Maps well to Pharo (Dictionary & Array)

• No class info, no complex graphs

[{ "id":"v1", "time":"2016-03-29T15:21:00+00:00", "long":5.3403835, "lat":50.918625, "speed":37},{ "id":"v1", "time":"2016-03-29T15:22:00+00:00", "long":5.3397698, "lat":50.915192, "speed":27},{ "id":"v1", "time":"2016-03-29T15:23:00+00:00", "long":5.3415751, "lat":50.911633, "speed":47}]

Parsing JSON

(FileLocator desktop / 'pvtrecords.json')readStreamDo: [ :in |

(NeoJSONReader on: in) next ].


STONJSON fromStream: in ].


(NeoJSONReader on: in)for: DateAndTime customDo: [ :mapping |

mapping decoder: [ :string | DateAndTime fromString: string ] ];

for: PVTRecord do: [ :mapping | mapping

mapAccessor: #speed;mapAccessor: #identification to: #id;mapAccessor: #longitude to: #long;mapAccessor: #latitude to: #lat.

(mapping mapAccessor: #timestamp to: #time)valueSchema: DateAndTime ];

nextListAs: PVTRecord ].

Generating JSON

(FileLocator desktop / 'pvtrecords2.json')writeStreamDo: [ :out |

(NeoJSONWriter on: out)nextPut: {

{#id->#v1. #time->'2016-03-29T15:21:00+00:00'. #long->5.3403835. #lat->50.918625. #speed->27 } asDictionary

} ].

[{"speed":27,"long":5.3403835,"lat":50.918625, "time":"2016-03-29T15:21:00+00:00","id":"v1"}]

(FileLocator desktop / 'pvtrecords2.json')writeStreamDo: [ :out |

(NeoJSONWriter on: out)prettyPrint: true;newLine: String lf;for: DateAndTime customDo: [ :mapping |

mapping encoder: [ :dateAndTime | dateAndTime asString ] ];

for: PVTRecord do: [ :mapping | mapping

mapAccessor: #speed;mapAccessor: #identification to: #id;mapAccessor: #longitude to: #long;mapAccessor: #latitude to: #lat.

(mapping mapAccessor: #timestamp to: #time) ];nextPut: {

PVTRecord one. PVTRecord two. PVTRecord three };newline ].

[{

"speed" : 37,"long" : 5.3403835,"lat" : 50.918625,"time" : "2016-03-29T15:21:00+00:00","id" : "v1"

},{


},{


}]

Referencehttp://files.pharo.org/books/enterprise-pharo/book/

NeoJSON/NeoJSON.html

http://files.pharo.org/books/enterprise-pharo/book/NeoJSON/NeoJSON.html

XML

• Textual

• Structured Markup (Tags & Attributes)

• Documents & Data

XML

• Large & Complex

• No inherent typing, Requires add ons

• Verbose

Parsing XML

XMLDOMParser parseFileNamed: (FileLocator desktop / 'pvtrecords.xml') fullName.

((XMLDOMParser parseFileNamed: (FileLocator desktop / 'pvtrecords.xml') fullName)

allElementsNamed: 'pvt')collect: [ :each |

PVTRecord new timestamp: (DateAndTime fromString:

(each contentStringAt: 'time'));identification: (each contentStringAt: 'id');longitude: (each contentStringAt: 'long') asNumber;latitude: (each contentStringAt: 'lat') asNumber;speed: (each contentStringAt: 'speed') asInteger;yourself ]

as: Array.

Generating XML

(FileLocator desktop / 'pvtrecords2.xml')writeStreamDo: [ :out |

| writer |(writer := XMLWriter on: out)

enablePrettyPrinting;lineBreak: String lf;xml.

writer tag: #records with: [{ PVTRecord one. PVTRecord two. PVTRecord three }

do: [ :each | writer tag: #pvt with: [

writer tag: #id with: each identification;tag: #time with: each timestamp asString;tag: #long with: each longitude asString;tag: #lat with: each latitude asString;tag: #speed with: each speed asString ] ] ] ].

<?xml version="1.0"?><records> <pvt> <id>v1</id> <time>2016-03-29T15:21:00+00:00</time> <long>5.3403835</long> <lat>50.918625</lat> <speed>37</speed> </pvt> <pvt> <id>v1</id> <time>2016-03-29T15:22:00+00:00</time> <long>5.3397698</long> <lat>50.915192</lat> <speed>27</speed> </pvt> <pvt> <id>v1</id> <time>2016-03-29T15:23:00+00:00</time> <long>5.3415751</long> <lat>50.911633</lat> <speed>47</speed> </pvt></records>

Protocols

Lingua Franca = HTTP

Web Services

HTTP

• Request / Response

• Transfer Any Entity Type

• URL / URI + Headers

Zinc HTTP Components

Client & Serverpart of standard image

Client Sideone tiny example

ZnClient newurl: 'http://easy.t3-platform.net/rest/geo-ip';queryAt: 'address' put: '81.83.7.35';get.

=> '{"latitude" : 50.8333,"address" : "81.83.7.35","country" : "BE","longitude" : 4.0

}'

ZnClient newsystemPolicy;url: 'http://easy.t3-platform.net/rest/geo-ip';queryAt: 'address' put: '81.83.7.35';accept: ZnMimeType applicationJson;contentReader: [ :entity |

STONJSON fromString: entity contents ];get.

Server Side

• Representational State Transfer (REST)

• Plain Zinc (BYO)

• Zinc-REST, Seaside-REST

• Teapot

2 arbitrary protocol examples

memcached

memcached

• memory caching server

• key/value - LRU

• standard architecture element

memcached

• lots of RAM (GB)

• distributed / shared

• similar to a database

memcached protocol

• simple, text / binary mix

• request / response

• small command set

| client |client := MDBasicClient new.[ client at: 'foo-key' ifAbsentPut: [ 'my-long-query-result' asByteArray ].

] ensure: [ client close ]

MOMMessage Oriented

Middleware

MOM

• infrastructure supporting sending & receiving messages between distributed systems

• heterogeneous & decoupled

• asynchronous

MOM

• exchanges / queues

• client / server

• producer / consumer

• routing / transformation

STOMPStreaming Text Oriented

Messaging Protocol

STOMP

• simple & textual

• wire format / commands

• similar to HTTP, yet different

STAMP

• Implementation of STOMP

• Protocol spec 1.2

• Tested against RabbitMQ

| server |server := self stampClient.[ server open. server subscribeTo: 'factorial'. server runWith: [ :message | | number | message body = 'quit' ifTrue: [ ConnectionClosed signal ]. number := message body asInteger. server sendText: number factorial asString to: message replyTo ] ] fork.

| client request |client := self stampClient.client open.request := client newSendFrameTo: 'factorial'.request text: 42 asString.request replyTo: '/temp-queue/factorial'.client write: request.response := client readMessage.self assert: response body equals: 42 factorial asString.client sendText: 'quit' to: 'factorial'.client close.

There are many other formats & protocols

Finding Formats & Protocols

Finding Formats & Protocols

• The Pharo Catalog

• Spotter

• http://catalog.pharo.org

• Ask on the mailing lists

http://catalog.pharo.org

NotFound ?

Consider doing your own implementation !

The End Q & A

#Pharo Days 2016 Data Formats and Protocols

Software

Transcript of #Pharo Days 2016 Data Formats and Protocols