From Overnight to Always On

Enno Runne Reactive Summit 2018-10-24

• Alpakka Tech Lead @ Lightbend

• Stockholm, Sweden

Enno Runne

File content as streams

File-based integration

Disk

Read from file

Disk

Write to file

InputStream is = new BufferedInputStream( new FileInputStream(sourceFilename)); OutputStream os = new BufferedOutputStream( new FileOutputStream(targetFilename));

byte[] buffer = new byte[1024]; int length; while ((length = is.read(buffer)) > 0) { os.write(buffer, 0, length); }

is.close(); os.close();

Copying a file in traditional Java

Let’s call it source and sink

Disk

Read from file

Disk

Write to file

Source<Byte…> Sink<Byte…>

FileIO.fromPath(sourceFile) FileIO.toPath(targetFile)

File source and sink with Akka Streams

Disk

Read from file

Disk

Write to file

Source<ByteString, …> Sink<ByteString, …>

Source<ByteString,…> CompletionStage<IOResult fileSource = FileIO.fromPath(sourceFile); Sink<ByteString,…>CompletionStage<IOResult>>fileSink = FileIO.toPath(targetFile); fileSource .to(fileSink) .run(materializer);

Connect source and sink

Read from file

Write to file

Source<ByteString, CompletionStage<IOResult>> fileSource = FileIO.fromPath(sourceFile); Sink<ByteString, CompletionStage<IOResult>> fileSink = FileIO.toPath(targetFile);

CompletionStage<IOResult> handle = fileSource .to(fileSink) .run(materializer);

Materialized values

Source<ByteString, CompletionStage<IOResult>> fileSource = FileIO.fromPath(sourceFile); Sink<ByteString, CompletionStage<IOResult>> fileSink = FileIO.toPath(targetFile);

CompletionStage<IOResult> handle = fileSource .runWith( fileSink, materializer );

Materialized values

Copying a file with Akka Streams

Read from file

Write to file

CompletionStage<IOResult> handle = FileIO.fromPath(sourceFile) .runWith( FileIO.toPath(targetFile), materializer );

Copying a file with Akka Streams

Read from file

Write to file

CompletionStage<IOResult> handle = FileIO.fromPath(sourceFile) .runWith( FileIO.toPath(targetFile), materializer );

Data flow

Step 1 Step 4Step 2 Step 3

Buffering data• May cure the immediate issue • All buffering is limited

Downstream

Source SinkFlow Flow

Streaming with back-pressure

Dynamic push/pull • Push when downstream is faster • Pull when upstream is faster

Akka Streams

Source

TParts of a Stream

Sink

S

A B

Flow

A B

Stream operators

map

A BmapAsync

A Bvia

Flow

Overnight Batching

Make our file-copy more useful

Disk

Read from file

Disk

Write to file

Detect new file

Detect a new file in the directory

DirectoryChangesSource.create( sourceDir, pollingInterval, maxChangesKept )

Detect new file

.filter(pathChange -> DirectoryChange.Creation == pathChange.second() )

// Pair<Path, DirectoryChange>

.map(Pair::first);

Source<Path, NotUsed> newFileDetector =

We built our own source!

Combine the source with a stream

Disk

Read from file

Disk

Write to file

Detect new file

Combine the source with a stream with a sink

Disk

Read from file

Disk

Write to file

Detect new file

Combine the source with a stream with a sink

newFileDetector .mapAsync(8, p -> {

Path targetFile = targetDir.resolve(p.getFileName()); return createFileToFile(p, targetFile);

}) .runWith(Sink.ignore(), materializer);

Nesting a stream execution within a stream

Outer and inner flows

Disk

Read from file

Disk

Write to file

Detect new file

Reactive Streamsreactive-streams.org

Part of JDK 9 (java.util.concurrent.Flow)

http://www.reactive-streams.org/

Compliant libraries allow full interoperability

Reactive Streams

A standard for asynchronous stream processing with non-blocking back-pressure.

Reactive Integrations

Reactive Integrations

Cross-system back-pressure support is the key thing Reactive Integrations bring to the table.

Overcoming file-based

Overcoming file-based integration

Disk

Read from file ?

From bytes to…

Overcoming file-based integration

Disk

Read from file

Parse bytes

bytes messages

Disk

Read from file

Parse CSV

bytes messages

Parse JSON

Parse XML

Overcoming file-based integration

Parse as CSV with Alpakka

Parse CSV

lines

byteStringSource .via(CsvParsing.lineScanner())

Convert CSV lines to maps

ByteString Collection<ByteString> Map<String, String>

.via(CsvToMap.toMapAsStrings(StandardCharsets.UTF_8));

Flow<ByteString, Map<String, String>, NotUsed> csvBytesToMap =

Flow.of(ByteString.class)

.via(CsvParsing.lineScanner())

.via(CsvToMap.toMapAsStrings( StandardCharsets.UTF_8 ));

Build your own flow

JsonNodeFactory jsonNodeFactory = JsonNodeFactory.instance; FileIO.fromPath(p) .via(csvBytesToMap) .map(data -> { // Using raw Jackson to create JSON objects ObjectNode objectNode = jsonNodeFactory.objectNode(); data.forEach(objectNode::put); return objectNode; })

Use your own flow and apply data mappingFlow<ByteString, Map<String, String>, NotUsed> csvBytesToMap = Flow.of(ByteString.class) .via(CsvParsing.lineScanner()) .via(CsvToMap.toMapAsStrings(StandardCharsets.UTF_8));

Overcoming file-based integration

Disk

Read from file

Parse CSV

bytes JSONMap Map to JSON

What is a stream?

Photo by blucolt - originally posted to Flickr as spearfish creek, CC BY-SA 2.0

50,000 / s 10 / s

1 / month

Reactive File Integration

Disk

Read from file

Destination

Detect new file

Turn into

messages

Other technology

Alpakka is a Reactive Enterprise Integration library for Java and Scala, based on Reactive Streams and Akka.

Alpakka

The short version: “Endpoints for Akka Streams”

Alpakka • Typed message interchange,

compiler tracks compatibility

• Back-pressure as specified by Reactive Streams

• No OSGi support

• Relies on Akka Streams; evolving rapidly, but many integration requirements are not covered, yet

• Moving fast, docs split between Akka and Alpakka sites

My view on Apache Camel vs Alpakka

Apache Camel • Data is wrapped in Exchange instance,

type can only be inspected

• No back-pressure awareness, can connect to Reactive Streams compliant systems

• OSGi support

• Full-featured framework to express integrations

• Comprehensive docs and books

Alpakka connectors for cloud services

Amazon DynamoDB Amazon Kinesis data streams & firehose AWS Lambda Amazon S3 Amazon SNS Amazon SQS

Google Cloud Pub/Sub Google Firebase Cloud Messaging

Azure Storage Queue

Alpakka connectors for data stores

Elasticsearch

Alpakka connectors for messaging

(Eclipse Paho)

AMQP (RabbitMQ)

IronMQ

JMS Java Messaging Service

Apache Kafka

… not as fancy logos, but very well suited for the streaming approach.

Community connectors to Akka Streams

Apache Camel Couchbase Eventuate FS2 Pulsar

… if you know of more, please tell us.

Preparations for Alpakka 1.0 include

• structure all modules the same

• improve chances to stay binary compatible

• ensure good test coverage

Help with this is highly appreciated…

The road to Alpakka 1.0

Code, Issues, Pull Requests @ Github

• https://github.com/akka/alpakka

• https://github.com/akka/alpakka-kafka

Questions, Discussions

• https://discuss.lightbend.com/c/akka

Alpakka community

Thank you!

@ennru enno.runne@lightbend.com

https://www.lightbend.com/alpakka @akkateam

Join the Alpakka community at github.com/akka/alpakka