Post on 16-Oct-2021
Pipes, Arrows, and the UniverseD E C L A R AT I V E T Y P E D S P E C I F I CAT I O N S FO R C O N C U R R E N T A N D C O M P O S A B L E WO R K F L OWS
1
Albert Schimpf
Tools & Toolboxes
Do one thing well
Compose tools together
What about the interface?
2
Universal Interface
3
"Write programs that handle text streams, because that is a universal
interface"
"Expect the output of every program to become the input of
another, as yet unknown, program"
THE BELL SYSTEM TECHNICAL JOURNAL. M. D. MCILROY, ET. AL. “UNIX TIME-SHARING SYSTEM FORWARD”. 1978A QUARTER-CENTURY OF UNIX, PETER H. SALUS, 1994
Pipe = 42
ls -1
wc -l
4
Pipe
Stringp1
Stringp2
5
What's nice?
Separation of business logic and control flow
Composition always works
Good for small chains
Easy (implicit) wiring
6
ShortcomingsConveniently access output previous steps
Graph-structured control flow
Parsing embedded data structures
7
Universal Interface II
"Expect the output of every program to become the input of another, as yet
unknown, program."
Write programs that handle maps1,
because that is a universal interface.
81: HTTPS://WWW.JSON.ORG/JSON-EN.HTML
Workflows COMPOSITION
MAP INTERFACE
9
Workflows COMPOSITION
MAP INTERFACE
10
Map Interface
?
Workflow Systems
Kernmantle ('20), Funflow, Porcupine, iTasks, Luigi, Argo, Oozie, NiFi, Azkaban,
Apache Airflow, AWS Data Pipeline, … (>280) Github
Actions
11HTTPS://GITHUB.COM/COMMON-WORKFLOW-LANGUAGE/COMMON-WORKFLOW-LANGUAGE/WIKI/EXISTING-WORKFLOW-SYSTEMS
WorkflowsCOMPOSITION
MAP INTERFACE
TYPES?
12
Map Interface
1.Chaining tasks 2.Graph of tasks (DAG)
13
Configuration (input, output) of components
Data flow: Map instead of String
p can access any data in the map that accesses p
Transitive implicit dependencies on input possible
ls
wc
p
Configuration
Input Output
14
Configuration (input, output) of components
Data flow: Map instead of String
p can access any data in the map that accesses p
Transitive implicit dependencies on input possible
ls
wc
p
{@x = […]}
{@x = […], @out = 42}
Configuration
Input Output
@x
@x @out
@x@out
15
Graph of Tasks – Semantics
Multiple incoming arrows
Multiple outgoing arrows
Concurrency, isolation??
16
Multiple Outgoing
ArrowsIfThenElse
true
false
λ
λ
17
Isolation:
Dispatched Arrows
Fork
Map
λ
λ
λ
18
Isolation:
Dispatched Arrows
Fork
Map
λ
λ
λ
19
Multiple Incoming
Arrowsλ
20
How to Join?FUTURES!
ForkJoin
?
21
Futures
ForkJoin
join
22
DemoS I M P L E E X A M P L E S & P I P E I N T E R AC T I O N
23
Types!
Atomic values
Lists
Maps
Custom records
Polymorphic types
24
Generated Documentation
25
Design & DemoK E Y WO R D M O N I T O R I N G
26
Summary
The universal string interface is not the only truth
Map interface allows for static type
checking and more
Try it out yourself!
27
Pipes, Arrows, and the UniverseD E C L A R AT I V E T Y P E D S P E C I F I CAT I O N S FO R C O N C U R R E N T A N D C O M P O S A B L E WO R K F L OWS
28
Albert Schimpf
wiki.scraper.server1.link
docs.scraper.server1.link
github.com/scraperflow
hello-world.yml
docker run -v "$PWD":/rt/ --rm albsch/scraper:latest
Install scraper scraper hello-world.yml