Maxim Zaks

Beyond JSON - An Introduction to FlatBuffers

Agenda

• About me

• The Why and How of Data Serialisation

• Deep dive into FlatBuffers

RESI News - Nachrichten, die Spaß machen

VS.

Why do we need data serialisation?

• Persisting State

• Machine to Machine Communication

• Representation of Configuration

How can we persist data on mobile?

• Custom binary representation

• Language provided binary serialisation

• Text based representation

• CSV, XML, JSON, YAML

• Embedded SQL or NoSQL DB

• Binary cross platform serialisation library

• FlatBuffers, Protocol Buffers, Cap’n Proto, SBE, Apache Thrift

Important criteria for persisting data

• Size on disk

• Speed of read & write / partial read / memory consumption

• Human readable and writable

• Support of OO language type system

• Data versioning / evolution / migration

VS.

JSON

• Size on disk 👍👎

• Speed of read & write / partial read / memory consumption 👎

• Human readable and writable 👍 👍

• Support of OO language type system 👍👎

• Data versioning / evolution / migration 👍👎

FlatBuffers

• Size on disk 👍👎

• Speed of read & write / partial read / memory consumption 👍 👍

• Human readable and writable 👍👎

• Support of OO language type system 👍 👍

• Data versioning / evolution / migration 👍 👍

JSON FlatBuffers

size on disk 👍👎 👍👎

read & write performance 👎 👍👍

human readable & writable 👍👍 👍👎

typed API 👍👎 👍👍

multi version support 👍👎 👍👍

But what about sending data?

Important criteria for sending data

• Size on disk wire

• Speed of read & write / partial read / memory consumption

• Human readable and writable

• Support of OO language type system

• Data versioning / evolution / migration

Enough theorylet’s do some practice

FlatBuffers IDL

./flatc -n person.fbs

FlatBuffers Schema Editor

FlatBuffers IDL

FlatBuffers IDL

FlatBuffers IDL

FlatBuffers IDL

What does it all mean in practice?

Size on disk / wire

Speed of read, writeand memory allocation

Partial Read

• over 3M cities

• ≈200 MB

• real time search

• in memory

• from file 10x slower

Human readable & writable is a manner of tooling

• ./flatc -b person.fbs person.json

• ./flatc -t person.fbs -- person.bin --raw-binary --strict-json

Support of OO Language type system

Data is backwards & forwards compatible

More information

• https://google.github.io/flatbuffers/

• https://github.com/mzaks/FlatBuffersSchemaEditor

• https://github.com/mzaks/FlatBuffersSwift

• https://twitter.com/iceX33

Questions?

Thank you!