Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1....
Transcript of Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1....
![Page 2: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/2.jpg)
About MeNest, Google, Typesafe. (opinions are my own)
Apple, General Magic, Twitter, WebTV, Microsoft, a few other startups
Scala fan
![Page 3: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/3.jpg)
Today1. Intro to sparkle2. Performance & scaling with streams3. Live layer and data architectureBonus: demos, development tips, Q&A
![Page 4: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/4.jpg)
SparkleTool for easily making zooming graphsPlatform for custom visualizations on live data● Built on streams● Big data, low latency
https://github.com/mighdoll/sparkle
![Page 5: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/5.jpg)
Loaders Fast Storage API StreamTransform Display
![Page 6: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/6.jpg)
Loaders
Loading Data● Several inputs, easy to add more
○ Files and directories (.csv / .tsv)○ Kafka / Avro○ HDFS bulk (*) ○ netcat (*)
● Loaders support subscribe/notify
![Page 7: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/7.jpg)
Loaders Fast Storage API StreamTransform Display
![Page 8: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/8.jpg)
Sparkle data modelColumn oriented storeImmutable data model
● Cassandra● RamStoreFast Storage
![Page 9: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/9.jpg)
Loaders Fast Storage API StreamTransform Display
![Page 10: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/10.jpg)
Select and TransformSelect columnsApply standard transforms● Aggregation● Time AggregationExtend with custom transformsFast
Select & Transform
![Page 11: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/11.jpg)
Loaders Fast Storage API StreamTransform Display
![Page 12: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/12.jpg)
Sparkle Protocol● Generic visualization protocol● Designed for (Web)Sockets● HTTP access too (no streaming)● json encoded● Documented
Stream
![Page 13: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/13.jpg)
Loaders Fast Storage API StreamTransform Display
![Page 14: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/14.jpg)
Javascript Client● built on d3● start with declarative javascript● easy to customize / extend● uses server api
○ (or standalone)Display
![Page 15: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/15.jpg)
Simple Demoquick graph of .tsv plot from command line
![Page 16: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/16.jpg)
Simple Demo
![Page 17: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/17.jpg)
Code for basic graph var charts = [ {
title: "90th Percentile Request Time",
groups: [ {
label: "seconds",
axis: sideAxis(),
named: [ { name: "epochs/p90" } ]
} ]
}];
![Page 18: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/18.jpg)
PerformanceAsynchronous Streams of Arrays
![Page 19: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/19.jpg)
Perf Study: Read API
Loaders Fast Storage API StreamTransform Display
![Page 20: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/20.jpg)
Phase I: Optimize Laterclass Event[K, V](key: K, value: V)
def read(): Seq[Event[K, V]]
● Easiest to understand, use, implement
![Page 21: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/21.jpg)
Tip: Perf Approach1. Add measurements directly in the code
○ repeat single flow: end to end, major components○ Async? measure synchronous portions and sum
2. Confirm/detail w/CPU profiling (YourKit)3. Test throughput and latency under load4. Review GC logs (Censum)Graph to review perf numbers
![Page 22: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/22.jpg)
kvE E
v
k
GC / CPU utilization
Ev
k
Ev
k
Ev
kE
v
k
kvE
kvE
kvE
kvE
![Page 23: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/23.jpg)
v v v v v v v v v v v v v v v v v v v v v v
k k k k k k k k k k k k k k k k k k k k k k
DataArray - Saves Heap
DataArray
● Arrays of primitives● High density jvm storage● Cache locality
![Page 24: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/24.jpg)
Phase II: Array Blocksclass DataArray[K: TypeTag, V: TypeTag]
( keys: Array[K], values: Array[V] )
def read(): DataArray[K, V]
![Page 25: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/25.jpg)
Are we done yet?Dense arrays mean less garbageTighter loops, more CPU cache efficient
![Page 26: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/26.jpg)
LatencyFetch Crunch Serve
Start End
![Page 27: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/27.jpg)
Overlap Pipeline Stages?Fetch
Crunch
Serve
Start End
![Page 28: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/28.jpg)
Consider ThroughputFetch Crunch Serve
Fetch Crunch Serve
Fetch Crunch Serve
Fetch Crunch Serve
Fetch Crunch Serve
![Page 29: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/29.jpg)
Throughput: MemoryFetch Crunch Serve
v v v v v v v v v v v v v v v v v v v v v v k k k k k k k k k k k k k k k k k k k k k k
![Page 30: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/30.jpg)
New GenCollection
Generational Hypothesis
CollectableGarbage
Age of Object
![Page 31: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/31.jpg)
Throughput: MemoryFetch Crunch Serve
v v v v v v v v v v v v v v v v v v v v v v k k k k k k k k k k k k k k k k k k k k k k
New GenCollection
![Page 32: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/32.jpg)
Throughput: MemoryFetch Crunch Serve
v v v v v v v v v v v v v v v v v v v v v v k k k k k k k k k k k k k k k k k k k k k k
New GenCollection
k k k k k k k k k k k k k k k k k k k k k kv v v v v v v v v v v v v v v v v v v v v v
v v v v v v v v v v v v v v v v v v v v v v k k k k k k k k k k k k k k k k k k k k k k
v v v v v v v v v v v v v v v v v v v v v v k k k k k k k k k k k k k k k k k k k k k k
v v v v v v v v v v v v v v v v v v v v v v k k k k k k k k k k k k k k k k k k k k k k
![Page 33: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/33.jpg)
New
JVM Heap
Old
Survivor
Survivor
![Page 34: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/34.jpg)
Throughput: MemoryFetch Crunch Serve
v v v v v v v v v v v v v v v v v v v v v v k k k k k k k k k k k k k k k k k k k k k k
New GenCollection
k k k k k k k k k k k k k k k k k k k k k kv v v v v v v v v v v v v v v v v v v v v v
v v v v v v v v v v v v v v v v v v v v v v k k k k k k k k k k k k k k k k k k k k k k
v v v v v v v v v v v v v v v v v v v v v v k k k k k k k k k k k k k k k k k k k k k k
v v v v v v v v v v v v v v v v v v v v v v k k k k k k k k k k k k k k k k k k k k k k
v v v v v v v v v v v v v v v v v v v v v v k k k k k k k k k k k k k k k k k k k k k k
k k k k k k k k k k k k k k k k k k k k k kv v v v v v v v v v v v v v v v v v v v v v
v v v v v v v v v v v v v v v v v v v v v v k k k k k k k k k k k k k k k k k k k k k k
v v v v v v v v v v v v v v v v v v v v v v k k k k k k k k k k k k k k k k k k k k k k
v v v v v v v v v v v v v v v v v v v v v v k k k k k k k k k k k k k k k k k k k k k k
Live Set
![Page 35: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/35.jpg)
Solution: Break Big ArraysFetch Crunch Serve
New GenCollection
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
v v vk k k
Live Set
![Page 36: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/36.jpg)
Phase III: Async Blocksclass DataStream[K: TypeTag, V: TypeTag]
( data: Observable[DataArray[K, V]] )
def read(): DataStream[K, V]
![Page 37: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/37.jpg)
Go Reactive: save GCBe more responsive, timely● Reason enough to go reactive.
Another reason: reduce GC pressure.● Transient working set is key for throughput
![Page 38: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/38.jpg)
More Blocksand more Streams
![Page 39: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/39.jpg)
Blocks for Streaming LayerKafka is already block streaming internally
Encode your data block-wise anyway● Encode/decode is more efficient● Sets the stage for downstream consumers
![Page 40: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/40.jpg)
Blocks for Cassandra Partition-aligned CQL write batches ● 10x write throughput
Store 1K blocks instead of (62) elements● 10x write throughput● 4x read throughput
![Page 41: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/41.jpg)
Stream to Graphing ClientOverlap client processing / communication ● Lowers end to end latency● Display starts sooner● Enables live / progressive updates
![Page 42: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/42.jpg)
StreamRequest
Streams Update Update
StreamControl
Status
Update
Update
Update UpdateUpdate
Client ServerRequest
Request
![Page 43: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/43.jpg)
Async Streams of Arrays
Loaders Fast Storage API StreamTransform Display
![Page 44: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/44.jpg)
Async Streams of Arrays
![Page 45: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/45.jpg)
Architecture
![Page 46: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/46.jpg)
Lambda Architecture?Streams are great for Sparkle'Lambda Architecture' is about using streamsWDYT?
![Page 47: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/47.jpg)
Lambda Architecture?Queries as pure functions that take all data● +1. we're all FP fans here too.
Batch… is too slowSo combine w/streaming, fast but approximate
![Page 48: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/48.jpg)
Lambda Architecture
Streaming
Batch
Servingnew data
ETL
ETL
![Page 49: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/49.jpg)
Lambda solves for latencyProblem: store + computation is batch slowSolution: two pipes. streaming, slow/batch
New Problem: two pipes, two platforms, etc.
Streaming or batch: only 2 choices?
![Page 50: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/50.jpg)
Low Latency Available NowIngest can be live
write 1M items / second (RF=3)Processing can be live
fetch + crunch 1M items < 250 msec5-10x better looks feasible not near IO bound
![Page 51: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/51.jpg)
Live Layer
Introducing: Live Layer
Fast Store Fast compute
High volumneLow latency ingestLow latency fetchTransform quickly
![Page 52: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/52.jpg)
Live with NotificationHigh volumneLow latency ingestLow latency fetchTransform quicklyNotification
Live Layer
Fast computeFast Store
![Page 53: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/53.jpg)
Loaders Fast Storage API StreamTransform Display
Live Layer
(Sparkle has a Live Layer)
![Page 54: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/54.jpg)
Live
Live + Lambda?
StreamingBatch
![Page 55: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/55.jpg)
Live: Enables On DemandGrain aligned - compute live, on request● Low latency response● Fresh data● Trigger as data arrives
![Page 56: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/56.jpg)
Storage GrainExample: time series server17.cpu.idleWith the grain: fast queries, scansWrites against the grain: only 10x slowerReads against the grain: cost grows linearly
![Page 57: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/57.jpg)
Lambda Architecture
Streaming
Batch
Servingnew data
ETL
ETL
![Page 58: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/58.jpg)
Live as Serving Layer
Streaming
Batch
new data
ETL
ETL
Live
![Page 59: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/59.jpg)
Live (vs. Stream Layer)History fully available, not just a windowEfficient calculate views only if needed
Front End to streaming too (serving layer).
Rule of thumb: Per-entity stream can be live
![Page 60: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/60.jpg)
Live + Stream Layer
Stream
Batch
new data
ETL
ETL
Live
![Page 61: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/61.jpg)
API for Live Data: Unifies class TwoPartStream[K,V]
( initial: DataStream,
ongoing: DataStream )
def readWithOngoing()
: TwoPartStream[K,V]
![Page 62: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/62.jpg)
Simplifying ETL
Stream
Batch
new data
ETL
ETL
Live
Extract Transform Load● Format conversion● Grain alignment
![Page 63: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/63.jpg)
Single Pipe + Batch
Streamnew data ETL
Batch
Live
![Page 64: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/64.jpg)
Live (vs. Batch Layer)Flexible parameters, not fixed at batch timeAgile w/o need to bulk reprocessFast responses broaden uses
Rule of thumb: Per-entity batch can now be live+/- One pipe, still two storage+crunch systems
![Page 65: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/65.jpg)
Single Pipe + Batch
StreamETL
Batch
Livenew data
![Page 66: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/66.jpg)
Where to Transform Data?Streaming: ETLLive: fast, with the grainBatch: slow, against the grain
Streaming + Live: fast, against the grain
![Page 67: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/67.jpg)
Single Pipe + Batch
StreamETL
Batch
Livenew data
![Page 68: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/68.jpg)
Data Pipeline of the Future
Live & Batchnew data StreamETL
![Page 69: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/69.jpg)
Scala Console Demo quick graphs from the scala repl
![Page 70: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/70.jpg)
Spark Demoquery against the grainbatch parallel with spark
![Page 71: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/71.jpg)
SparkleTool for easily making zooming graphsPlatform for custom visualizations on live data● Built on streams● Generic visualization protocol ● Live data / big data
https://github.com/mighdoll/sparkle
![Page 73: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/73.jpg)
Tips
![Page 74: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/74.jpg)
Tip: Make tests as REPLMake tests than can be run from the replEncourages simpler syntaxCreates a useful tool
![Page 75: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/75.jpg)
Tip: Always make it betterEvery commit makes the Avoid Forbidden Island Syndrome-> impassible continentsStrive for perfection: clarity, flexibility, efficiency
![Page 76: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/76.jpg)
Scala: Refactoring FTW● Language power: refactoring enabler
○ composition, abstraction, concision, clarity● Types: safety net
○ 'works the first time it compiles' - oft heard, true, fun○ 'works after refactoring' - more important
● Testing: smaller tests, better coverage○ Bulk is drag○ Best in class test libraries
![Page 77: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/77.jpg)
Tip: Go Deep and Make Not just a list of featuresOr a deadline
A little learning is a dangerous thing;Drink deep, or taste not the Pierian spring.
Strive to create a well-made thing.
![Page 78: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/78.jpg)
Challenges
![Page 79: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/79.jpg)
Type recoverystored data has a fixed typeprotocol requests reference databut these types are unknown at compile time
![Page 80: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/80.jpg)
Dynamic type recoveryserialize type tagrecover: match against known typesrecover: match against needed type classes
tryNumeric[T: TypeTag]: Try[Numeric[T]]
![Page 81: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/81.jpg)
Phase IV: DataStreamSpecialization?Stream Fusion?n-arrays?
![Page 82: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/82.jpg)
Scratch
![Page 83: Sparkle - Lightbenddownloads.typesafe.com/website/presentations/Scala... · Scala fan. Today 1. Intro to sparkle 2. Performance & scaling with streams 3. Live layer and data architecture](https://reader034.fdocuments.in/reader034/viewer/2022042408/5f246ee585caf300cc54d830/html5/thumbnails/83.jpg)
serving layer
Lambda Architecture
new data
master dataset
real-time view real-time view
batch view
query
query
speed layer
batch layer
batch view