QCon London 2018 Foundations of streaming SQL · Reconciling streams & tables w/ the Beam Model How...

Foundations of streaming SQLor: how I learned to love stream & table theory

Slides: https://s.apache.org/streaming-sql-qcon-londonTyler AkidauApache Beam PMCSoftware Engineer at Google@takidau

Covering ideas from across the Apache Beam, Apache Calcite, Apache Kafka, and Apache Flink communities, with thoughts and contributions from Julian Hyde, Fabian Hueske, Shaoxuan Wang, Kenn Knowles, Ben Chambers, Reuven Lax, Mingmin Xu, James Xu, Martin Kleppmann, Jay Kreps and many more, not to mention that whole database community thing...

QCon London 2018

Table of Contents

Stream & Table TheoryA BasicsB The Beam Model

Streaming SQLA Time-varying relationsB SQL language extensions

Chapter 7

Chapter 9

01 Stream & Table TheoryTFW you realize everything you do was invented by the database community decades ago...

A BasicsB The Beam Model

Stream & table basics

https://www.confluent.io/blog/making-sense-of-stream-processing/ https://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple/

Special theory of stream & table relativity

streams → tables:

tables → streams:

The aggregation of a stream of updates over time yields a table.

The observation of changes to a table over time yields a stream.

Non-relativistic stream & table definitions

Tables are data at rest.

Streams are data in motion.

01 Stream & Table TheoryTFW you realize everything you do was invented by the database community decades ago...

A BasicsB The Beam Model

The Beam Model

What results are calculated?

Where in event time are results calculated?

When in processing time are results materialized?

How do refinements of results relate?

Reconciling streams & tables w/ the Beam Model

● How does batch processing fit into all of this?

● What is the relationship of streams to bounded and unbounded datasets?

● How do the four what, where, when, how questions map onto a streams/tables world?

ReduceMap

MapReduce

output

MapReduce

output

MapRead

MapWrite

ReduceRead

Reduce

ReduceWrite

MapReduce

MapRead

MapWrite

ReduceRead

Reduce

ReduceWrite

MapReduce

MapRead

MapWrite

ReduceRead

Reduce

ReduceWrite

Map phase

MapRead

MapWrite

Map phase API

void map(K1 key, V1 value, Emit<K2, V2>);

Map phase API

Map phase

MapRead

MapWrite

stream

Map phase API

Map phase

MapRead

MapWrite

stream

Map phase API

void reduce(K2 key, Iterable<V2> value, Emit<V3>);

Map phase

MapRead

MapWrite

stream

MapReduce

MapRead

MapWrite

ReduceRead

Reduce

ReduceWrite

stream

Map phase API

MapReduce

MapRead

MapWrite

ReduceRead

Reduce

ReduceWrite

stream

1. Tables are read into streams.

2. Streams are processed into new streams until a grouping operation is hit.

3. Grouping turns the stream into a table.

4. Repeat steps 1-3 until you run out of operations.

Streams are the in-motion form of data

both bounded and unbounded.

The Beam Model

Example data: individual user scores

What is calculated?

PCollection<KV<Team, Score>> input = IO.read(...) .apply(ParDo.of(new ParseFn()); .apply(Sum.integersPerKey());

What is calculated?

The Beam Model

Windowing divides data into event-time-based finite chunks.

Often required when doing aggregations over unbounded data.

Fixed Sliding1 2 3

Sessions

1 2 3 4

Where in event time?

PCollection<KV<User, Score>> input = IO.read(...) .apply(ParDo.of(new ParseFn()); .apply(Window.into(FixedWindows.of(Duration.standardMinutes(2))) .apply(Sum.integersPerKey());

Where in event time?

The Beam Model

• Triggers control when results are emitted.

• Triggers are often relative to the watermark.

Event Time

~Watermark

When in processing time?

PCollection<KV<User, Score>> input = IO.read(...) .apply(ParDo.of(new ParseFn()); .apply(Window.into(FixedWindows.of(Duration.standardMinutes(2)) .triggering(AtWatermark()) .apply(Sum.integersPerKey());

When in processing time?

The Beam Model: asking the right questions

How do refinements relate?

PCollection<KV<User, Score>> input = IO.read(...) .apply(ParDo.of(new ParseFn()); .apply(Window.into(FixedWindows.of(Duration.standardMinutes(2)) .triggering(AtWatermark().withLateFirings(AtCount(1))) .accumulatingFiredPanes()) .apply(Sum.integersPerKey());

How do refinements relate?

What/Where/When/How Summary

3. Streaming 4. Streaming + Late Data Handling

1. Classic Batch 2. Windowed Batch

General theory of stream & table relativityPipelines : tables + streams + operationsTables : data at restStreams : data in motionOperations : (stream | table) → (stream | table) transformations

● stream → stream: Non-grouping (element-wise) operations Leaves stream data in motion, yielding another stream.

● stream → table: Grouping operations Brings stream data to rest, yielding a table. Windowing adds the dimension of time to grouping.

● table → stream: Ungrouping (triggering) operations Puts table data into motion, yielding a stream. Accumulation dictates the nature of the stream (deltas, values, retractions).

● table → table: (none) Impossible to go from rest and back to rest without being put into motion.

02 Streaming SQLContorting relational algebra for fun and profit

A Time-varying relationsB SQL language extensions

Relational algebra

User Score Time

Julie 7 12:01

Frank 3 12:03

Julie 1 12:03

Julie 4 12:07

Score Time

7 12:01

3 12:03

1 12:03

4 12:07

πScore,Time(UserScores)πUserScoresπ SELECT Score, Time FROM UserScores;-----------------| Score | Time |-----------------| 7 | 12:01 || 3 | 12:03 || 1 | 12:03 || 4 | 12:07 |-----------------

Relational algebra SQLRelation

Relations evolve over time

12:07> SELECT * FROMUserScores;-------------------------| Name | Score | Time |-------------------------| Julie | 7 | 12:01 || Frank | 3 | 12:03 || Julie | 1 | 12:03 || Julie | 4 | 12:07 |-------------------------

12:03> SELECT * FROMUserScores;-------------------------| Name | Score | Time |-------------------------| Julie | 7 | 12:01 || Frank | 3 | 12:03 || Julie | 1 | 12:03 |-------------------------

12:00> SELECT * FROMUserScores;-------------------------| Name | Score | Time |--------------------------------------------------

12:01> SELECT * FROMUserScores;-------------------------| Name | Score | Time |-------------------------| Julie | 7 | 12:01 |-------------------------

Classic SQL vs Streaming SQL

Classic SQL classic relations single point in time:: ::

Streaming SQL time-varying relations every point in time:: ::

Classic SQL vs Streaming SQL

Classic SQL classic relations single point in time:: ::

Streaming SQL time-varying relations every point in time:: ::

Classic relations

12:07> SELECT * FROM UserScores;-----------------------------------------------------------------------------------------------------------------| [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) || ------------------------- | ------------------------- | ------------------------- | ------------------------- || | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | || ------------------------- | ------------------------- | ------------------------- | ------------------------- || | | | | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | || | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | || | | | | | | | | | | | Julie | 1 | 12:03 | | | Julie | 1 | 12:03 | || | | | | | | | | | | | | | | | | Julie | 4 | 12:07 | || ------------------------- | ------------------------- | ------------------------- | ------------------------- |-----------------------------------------------------------------------------------------------------------------

12:07> SELECT * FROMUserScores;-------------------------| Name | Score | Time |-------------------------| Julie | 7 | 12:01 || Frank | 3 | 12:03 || Julie | 1 | 12:03 || Julie | 4 | 12:07 |-------------------------

12:03> SELECT * FROMUserScores;-------------------------| Name | Score | Time |-------------------------| Julie | 7 | 12:01 || Frank | 3 | 12:03 || Julie | 1 | 12:03 |-------------------------

12:00> SELECT * FROMUserScores;-------------------------| Name | Score | Time |--------------------------------------------------

12:01> SELECT * FROMUserScores;-------------------------| Name | Score | Time |-------------------------| Julie | 7 | 12:01 |-------------------------

Time-varying relation

Closure property of relational algebra remains intact with time-varying relations.

Time-varying relations: variations12:07> SELECT * FROM UserScores;-----------------------------------------------------------------------------------------------------------------| [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) || ------------------------- | ------------------------- | ------------------------- | ------------------------- || | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | || ------------------------- | ------------------------- | ------------------------- | ------------------------- || | | | | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | || | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | || | | | | | | | | | | | Julie | 1 | 12:03 | | | Julie | 1 | 12:03 | || | | | | | | | | | | | | | | | | Julie | 4 | 12:07 | || ------------------------- | ------------------------- | ------------------------- | ------------------------- |-----------------------------------------------------------------------------------------------------------------

Time-varying relations: filtering12:07> SELECT * FROM UserScores;-----------------------------------------------------------------------------------------------------------------| [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) || ------------------------- | ------------------------- | ------------------------- | ------------------------- || | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | || ------------------------- | ------------------------- | ------------------------- | ------------------------- || | | | | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | || | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | || | | | | | | | | | | | Julie | 1 | 12:03 | | | Julie | 1 | 12:03 | || | | | | | | | | | | | | | | | | Julie | 4 | 12:07 | || ------------------------- | ------------------------- | ------------------------- | ------------------------- |-----------------------------------------------------------------------------------------------------------------

12:07> SELECT * FROM UserScores WHERE Name = “Julie”;-----------------------------------------------------------------------------------------------------------------| [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) || ------------------------- | ------------------------- | ------------------------- | ------------------------- || | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | || ------------------------- | ------------------------- | ------------------------- | ------------------------- || | | | | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | || | | | | | | | | | | | Julie | 1 | 12:03 | | | Julie | 1 | 12:03 | || | | | | | | | | | | | | | | | | Julie | 4 | 12:07 | || ------------------------- | ------------------------- | ------------------------- | ------------------------- |-----------------------------------------------------------------------------------------------------------------

Time-varying relations: grouping12:07> SELECT * FROM UserScores;-----------------------------------------------------------------------------------------------------------------| [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) || ------------------------- | ------------------------- | ------------------------- | ------------------------- || | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | || ------------------------- | ------------------------- | ------------------------- | ------------------------- || | | | | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | | | Julie | 7 | 12:01 | || | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | || | | | | | | | | | | | Julie | 1 | 12:03 | | | Julie | 1 | 12:03 | || | | | | | | | | | | | | | | | | Julie | 4 | 12:07 | || ------------------------- | ------------------------- | ------------------------- | ------------------------- |-----------------------------------------------------------------------------------------------------------------

12:07> SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name;-----------------------------------------------------------------------------------------------------------------| [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) || ------------------------- | ------------------------- | ------------------------- | ------------------------- || | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | || ------------------------- | ------------------------- | ------------------------- | ------------------------- || | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | || | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | || ------------------------- | ------------------------- | ------------------------- | ------------------------- |-----------------------------------------------------------------------------------------------------------------

How does this relate to streams & tables?

Time-varying relations: tables

12:07> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name;-------------------------| Name | Score | Time |-------------------------| Julie | 12 | 12:07 || Frank | 3 | 12:03 |-------------------------

12:01> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name;-------------------------| Name | Score | Time |-------------------------| Julie | 7 | 12:01 |-------------------------

12:00> SELECT TABLE Name, SUM(Score), MAX(Time) FROM UserScores GROUP BY Name;-------------------------| Name | Score | Time |--------------------------------------------------

Time-varying relations: tables

12:07> SELECT TABLE Name, SUM(Score), MAX(Time) AS OF SYSTEM TIME ‘12:01’ FROM UserScores GROUP BY Name;-------------------------| Name | Score | Time |-------------------------| Julie | 7 | 12:01 |-------------------------

Time-varying relations: tables12:07> SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name;-----------------------------------------------------------------------------------------------------------------| [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) || ------------------------- | ------------------------- | ------------------------- | ------------------------- || | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | || ------------------------- | ------------------------- | ------------------------- | ------------------------- || | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | || | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | || ------------------------- | ------------------------- | ------------------------- | ------------------------- |-----------------------------------------------------------------------------------------------------------------

12:00> SELECT STREAM Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name;-------------------------| Name | Score | Time |-------------------------...

Time-varying relations: streams12:07> SELECT Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name;-----------------------------------------------------------------------------------------------------------------| [-inf, 12:01) | [12:01, 12:03) | [12:03, 12:07) | [12:07, now) || ------------------------- | ------------------------- | ------------------------- | ------------------------- || | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | | | Name | Score | Time | || ------------------------- | ------------------------- | ------------------------- | ------------------------- || | | | | | | Julie | 7 | 12:01 | | | Julie | 8 | 12:03 | | | Julie | 12 | 12:07 | || | | | | | | | | | | | Frank | 3 | 12:03 | | | Frank | 3 | 12:03 | || ------------------------- | ------------------------- | ------------------------- | ------------------------- |-----------------------------------------------------------------------------------------------------------------

12:00> SELECT STREAM Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name;-------------------------| Name | Score | Time |-------------------------| Julie | 7 | 12:01 |...

12:00> SELECT STREAM Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name;-------------------------| Name | Score | Time |-------------------------| Julie | 7 | 12:01 || Frank | 3 | 12:03 || Julie | 8 | 12:03 |...

12:00> SELECT STREAM Name, SUM(Score), MAX(Time) FROM USER_SCORES GROUP BY Name;-------------------------| Name | Score | Time |-------------------------| Julie | 7 | 12:01 || Frank | 3 | 12:03 || Julie | 8 | 12:03 || Julie | 12 | 12:07 |...

How does this relate to streams & tables?

capture a point-in-time snapshot of a time-varying relation.

capture the evolution of a time-varying relation over time.

Tables

Streams

02 Streaming SQLContorting relational algebra for fun and profit

A Time-varying relationsB SQL language extensions

When do you need SQL extensions for streaming?

As a table: As a stream:

SQL extensions rarely needed. SQL extensions sometimes needed.

How is output consumed?

good defaults = often not needed

When do you need SQL extensions for streaming?*

Explicit table / stream selection● SELECT TABLE * from X;● SELECT STREAM * from X;

Timestamps and windowing● Event-time columns● Windowing. E.g.,

SELECT * FROM X GROUP BY SESSION(<COLUMN> INTERVAL '5' MINUTE);

○ Grouping by timestamp○ Complex multi-row transactions

inexpressible in declarative SQL (e.g., session windows)

Sane default table / stream selection● If all inputs are tables, output is a table● If any inputs are streams, output is a stream

Simple triggers● Implicitly defined by characteristics of the sink● Optionally be configured outside of query.● Per-query, e.g.: SELECT * from X EMIT <WHEN>; ● Focused set of use cases:

○ Repeated updates... EMIT AFTER <TIMEDELTA>

○ Completeness... EMIT WHEN WATERMARK PAST <COLUMN>

○ Repeated updates + completeness(e.g., early/on-time/late pattern)... EMIT AFTER <TIMEDELTA> AND WHEN WATERMARK PAST <COLUMN>

* Most of these extensions are theoretical at this point; very few have concrete implementations.

Summary

streams ⇄ tables

streams & tables : Beam Model

time-varying relations

SQL language extensions

Thank you!

In early release nowstreamingsystems.net

These slides: http://s.apache.org/streaming-sql-big-data-spain

Streaming SQL spec (WIP: Apex, Beam, Calcite, Flink): http://s.apache.org/streaming-sql-specStreaming in Calcite (Julian Hyde): https://calcite.apache.org/docs/stream.htmlStreams, joins & temporal tables (Julian Hyde): http://s.apache.org/streams-joins-and-temporal-tables

Streaming 101: http://oreilly.com/ideas/the-world-beyond-batch-streaming-101Streaming 102: http://oreilly.com/ideas/the-world-beyond-batch-streaming-102

Apache Beam: http://beam.apache.orgApache Calcite: http://calcite.apache.orgApache Flink: http://flink.apache.org

Twitter: @takidau

QCon London 2018 Foundations of streaming SQL · Reconciling streams & tables w/ the Beam Model How...

Documents

Transcript of QCon London 2018 Foundations of streaming SQL · Reconciling streams & tables w/ the Beam Model How...

Setup up for your QCON Pro & QCON EX controller section Note:

“Reconciling Love!”

merb - QCon

Domainlang keynote-eric-qcon

Reconciling Streams and Cycles: Avoiding Mixed Metaphors ... · 2 Abstract: The streams metaphor developed by Kingdon from March and Olson’s earlier work has been expanded ... fashion

Dojo (QCon 2007 Slides)

Catalogue 2019 - Home - Quantium Medical...A. qCON sensor new 26 B. qCON sensor 27 C. qCON cable patient new 28 D. qCON cable patient 29 E. EEG simulator 30 4 5 The qCO monitor in

RPC and its Offspring - QCon

Podila QCon SF 2016

ETL is dead; long-live streams - QCon SF is dead; long-live streams Neha Narkhede, ... What role does Kafka play in the new shiny future for data integration? #1: Kafka is the de-facto

QCon Beijing - April 2010

QCon London - Gilt Microservices

Devjam keynote-david-qcon

Edb 100k-trans-qcon-rev1

Qcon SF 2013

Jez Humble QCon London 2011gotocon.com/dl/qcon-london-2011/slides/JezHumble... · 2011. 3. 11. · QCon London 2011 jez@thoughtworks.com @jezhumble #continuousdelivery Thursday, March

Qcon 111122082620-phpapp02

Mobile Transformation - QCon

Netflix web-adrian-qcon

IBM Streams QCon - QCon New York€¦ · Manoj Singh Big Data & Streams Architect manoj.singh101@ibm.com IBM Streams In-Motion Analytics Platform @IBM