www.techarchday.fiwww.techarchday.fi

Modern Data Platforms

(in the age of microservices)

Dominik Wagenknecht

Senior Technology Architect

Accenture

www.techarchday.fi

About Dominik

Dominik Wagenknecht

Accenture Vienna / Austria

Senior Technology Architect

Emerging Technology

www.techarchday.fi

Accenture Vienna

www.techarchday.fi

SO YOU ADOPTED MICROSERVICES

Chapter 1

www.techarchday.fi

So you adopted microservices

• CI/CD pipeline is pure magic!

• Super-motivated Teams

• Business feels progress

• Great Monitoring

• Data?

www.techarchday.fi

What changed?

Browser

API Gateway

Service Service

Service Service Service

DB DB DB

Stateless

Model

API + Services!

www.techarchday.fi

What hasn’t changed?

Browser

API Gateway

Service Service

Service Service Service

DB DB DBBasically CRUD

Synchronous

www.techarchday.fi

BUSINESS WANTS MORE

Chapter 2

www.techarchday.fi

Business wants more

• More Interaction!

– Mobile Push with content

– Personalized eMail,…

• More Smartness!

– Fraud Detection

– Next-best-action

www.techarchday.fi

Mobile Push “OldSchool”

• We need to de-couple!

– Events need to reach a Queue

– Logic listens (MDB?), then push

• Issues?

– Composes really badly

– Force people to testing Queues + MDB’s

www.techarchday.fi

Data Layer is holding you back

Browser

API Gateway

Service Service

Service Service Service

DB DB DB

ServiceQueue

AppleGoogle

Data flow

async

www.techarchday.fi

MODERN DATA IN S, M & L(+ ONE DARK HORSE)

Chapter 3

www.techarchday.fi

Detour: Every op an event

• Service initiates calls

• Modern DB’s separate

– Ops first go to a (binary) log

– Then data is written into core store async

Application(s)

Database

INSERT INTO...INSERT INTO...COMMIT

(Commit)Log

Data

www.techarchday.fi

Modern Data in Small

• Idea: Don’t do it yourself

• Native

– Firebase, RethinkDB

– PipelineDB (kind of)

• Attached

– Meteor’s LiveQuery

S

www.techarchday.fi

Modern Data in Medium

• Idea: Stay lean, keep control

• Messaging

– Something simple like RabbitMQ

– Kafka if you want (but not more)

• Fill from app (or there’s an easy hook)

• Just process, add leader-election

M

www.techarchday.fi

Modern Data in Large

• Idea: Establish a big data Platform

• Really only two choices

– Hadoop with HBase, Kafka, Spark Streaming / Storm / Samza / Flink, etc…

– Cassandra (mostly used with Spark)

L

www.techarchday.fi

Detour: Hook up to Legacy

Application(s)

Database

INSERT INTO...INSERT INTO...COMMIT

(Commit)Log

Data Log Replicator e.g. Kafka

Async starts here!

www.techarchday.fi

PRAGMATISM

Chapter 4

www.techarchday.fi

Some tips

• Physical != Logical

• Stay small

• Be smart about moving parts

www.techarchday.fi

Dark Horse: In-Memory Grid

• Not “micro”, “stateless” or anything

• Think Cache + Logic + Streams

• Great (OpenSource) choice, in Java-world alone:

– Apache Ignite (GridGain)

– Apache Geode (Incubating, Pivotal Gemfire)

– Hazelcast

– …

www.techarchday.fiwww.techarchday.fi

THANKS