Portable stateful big data processing in Apache Beam

Kenneth Knowles

Apache Beam PMCSoftware Engineer @ Google

klk@google.com / @KennKnowles https://s.apache.org/ffsf-2017-beam-stateFlink Forward San Francisco 2017

Agenda1. What is Apache Beam?

2. State

3. Timers

4. Example & Little Demo

What isApache Beam?

TL;DR

4(Flink draws it more like this)

20142004 2006 2008 2010 2012 20162005 2007 2009 2013 20152011

MapReduce(paper)

Apache Hadoop

Dataflow Model(paper)

MillWheel(paper)

Heron

ApacheSpark

ApacheStorm

Apache Gearpump

(incubating)Apache

Apex

Apache Flink

Cloud Dataflow

FlumeJava(paper)

Apache Beam

DAGs, DAGs, DAGs

Apache Samza

The Beam Vision

Sum Per Key

6

input.apply(

Sum.integersPerKey())

Java

input | Sum.PerKey()

Python

⋮

Apache Flinklocal, on-prem,

cloud

Apache Sparklocal, on-prem,

cloud

Cloud Dataflow: fully managed

⋮

Apache Apexlocal, on-prem,

cloud

Apache Gearpump

(incubating)

The Beam Vision

KafkaIO

7

input | KakaIO.read()

Python

⋮

class KafkaIO extends

UnboundedSource { … }

Java

Apache Flinklocal, on-prem,

cloud

Apache Sparklocal, on-prem,

cloud

Cloud Dataflow: fully managed

⋮

Apache Apexlocal, on-prem,

cloud

Apache Gearpump

(incubating)

The Beam Model

Pipeline

8

PTransform

PCollection(bounded or unbounded)

The Beam Model

What are you computing? (read, map, reduce)

Where in event time? (event time windowing)

9

When in processing time are results produced? (triggers)

How do refinements relate? (accumulation mode)

What are you computing?

10

ReadParallel connectors to

external systems

ParDoPer element

"Map"

GroupingGroup by key,

Combine per key,"Reduce"

CompositeEncapsulated

subgraph

State

What are you computing?

12

ReadParallel connectors to

external systems

ParDoPer element

"Map"

GroupingGroup by key,

Combine per key,"Reduce"

CompositeEncapsulated

subgraph

State for ParDo

13

ParDo(DoFn)

"Example"

14

Per Key Quantiles

ParDo.of(new DoFn<...>() {

// declare some state

@ProcessElement

public void process(...) {

// update quantiles

// output if needed

}

})

Partitioned

15

Quantiles Quantiles Quantiles

Parallelism!

16

QuantilesDoFn

QuantilesDoFn

QuantilesDoFn

new DoFn<Foo, Quantiles<Foo>>() {

@StateId("quantiles")

private final StateSpec<...>

quantilesState = StateSpecs.combining();

…}

Windowed

17

QuantilesDoFn

QuantilesDoFn

QuantilesDoFn

Window into Fixed windows of one hour

Expected result: Quantiles for each hour

Windowed

18

QuantilesDoFn

QuantilesDoFn

QuantilesDoFn

Window into windows of 30 min sliding by 10 min

Expected result: Quantiles for 30 minutes sliding by 10 min

<k, w>1

<k, w>2

<k, w>3

...

"x" 3 7 15

"y" "fizz" "7" "fizzbuzz"

...

State is per key and window

Bonus: automatically garbage collected when a window expires(vs manual clearing of keyed state)

Windowed

20

QuantilesDoFn

QuantilesDoFn

QuantilesDoFn

Window into Fixed windows of one hour

Expected result: Quantiles for each hour

What about Combine?

21

QuantilesCombiner

Window into Fixed windows of one hour

Expected result: Quantiles for each hour

QuantilesCombiner

QuantilesCombiner

Combine vs State (naive conceptualization)

22

Window hourly

Expected result:Quantiles for each hour

QuantilesCombiner

Window hourly

QuantilesDoFn

Expected result:Quantiles for each hour

Combine vs State (likely execution plan)

Quantiles Combiner

Quantiles Combiner

Quantiles Combiner

Shuffle Accumulators

QuantilesDoFn

Shuffle Elements

associativecommutativesingle-outputenables optimizations(engine is in control)

non-associative*non-commutative*

multi-outputside outputs(user is in control)

Combine vs State

Quantiles Combiner Quantiles

DoFn

output governed by trigger

(data/computation unaware) "output only when there's an interesting change"

(data/computation aware)

Example: Arbitrary indices

D,0

A

Assign indices

B

C

D

E

C,1

A,2

B,3

E,4

non-associative

non-commutative

non-deterministic

and totally fine!

Kinds of state

● Value - just a mutable cell for a value

● Bag - supports "blind writes"

● Combining - has a CombineFn built in; can support blind writes and lazy accumulation

● Set - membership checking

● Map - lookups and partial writes

Timers

Timers for ParDo

28

Stateful ParDo

new DoFn<...>() {

@TimerId("timeout")

private final TimerSpec timeoutTimer =

TimerSpecs.timer(TimeDomain.PROCESSING_TIME);

@OnTimer("timout")

public void timeout(...) {

// access state, set new timers, output

}

…}

Timers in processing time

"call me back in 5 seconds"

output based only on incoming element

output return value of batched RPC

buffer request

batched RPC

On Timer

On Element

Timers in event time

"call me back when the watermark hits the end of the window"

output speculative result immediately

output replacement value only if changed

store speculative result

On Timer

On Element

● Per-key arbitrary numbering

● Output only when result changes

● Tighter "side input" management for slowly changing dimension

● Streaming join-matrix / join-biclique

● Fine-grained combine aggregation and output control

● Per-key "workflows" like user sign up flow w/ expiration

● Low-latency deduplication (let the first through, squash the rest)

More example uses for state & timers

Performance considerations (cross-runner)

32

● Shuffle to colocate keys

● Linear processing of elements for key+window

● Window merging

● Storage of state and timers

● GC of state

Demo

33

SummaryState and Timers in Beam...

● … unlock new uses cases

● … they "just work" with event time windowing

● … are portable across runners (implementation ongoing)

Thank you for listening!This talk:

● Me - @KennKnowles● These Slides - https://s.apache.org/ffsf-2017-beam-state

Go Deeper● Design doc - https://s.apache.org/beam-state● Blog post - https://beam.apache.org/blog/2017/02/13/stateful-processing.html

Join the Beam community:● User discussions - user@beam.apache.org● Development discussions - dev@beam.apache.org● Follow @ApacheBeam on Twitter

You can contribute to Beam + Flink● New types of state● Easy launch of Beam job on Flink-on-YARN● Integration tests at scale● Fit and finish: polish, polish, polish!● … and lots more!

https://beam.apache.org