ScalaDays Highlights2015

Who?

● BoldRadius Solutions○ boldradius.com○ Typesafe Partner○ Scala, Akka and Play specialists○ Ottawa, Saskatoon, San Francisco, Boston, Chicago, Montreal, New

York, Toronto

● Michael Nash, VP Capabilities● Adam Murray, VP Business

Our Team at ScalaDays

What?

What was ScalaDays all about?● Held in San Francisco March 16th thru 18th

○ About 800 attendees○ Three categories (intermediate, beginner,

advanced), ○ Four tracks, 55 presentations, three keynotes

● Followed by Two Intensive Training days○ All the Typesafe courses offered

Venue: Fort Mason

The main hall...

Keynotes● Martin Odersky, Chief Architect & Co-Founder at Typesafe

○ Scala - where it came from, where it’s going● Danese Cooper, Distinguished Member of Technical Staff -

Open Source at Paypal○ Open Languages

● Dianne March, Director of Engineering Tools at Netflix

○ Technical Leadership from wherever you are

Quote from James Gosling...

HighlightsToo many great sessions to summarize them all, we have to extract a few recurring themes…

○ Distributed Application Design and Development○ Big Data/Fast Data○ Types and Safe Scala○ Performance and Scalability

Distributed Applications

● Life Beyond the Illusion of Present● Reactive Reference Architectures● Akka in Production: Why and How● Easy Scalability with Akka● Scalable task distribution with Scala, Akka

and Mesos● A Guided Tour of a Distributed Application

Performance and Scalability

● Scala Collections and Performance● Shattering Hadoop’s Large-Scala Sort Record with

Spark and Scala● Type-safe off-heap memory for Scala● Akka in Production: Why and How● Easy Scalability with Akka● The JVM Backend and Optimizer in Scala 2.12

Big Data, Fast Data

● Shattering Hadoop’s Large Scala Sort Record with Spark and Scala

● Scala - The Real Spark of Data Science● Apache Spark: A Large Community Project in Scala● S3 at Scala: Async Scala Client with Play Iteratees and

Composable Operations● The Unreasonable Effectiveness of Scala for Big Data● Scalable task distribution with scala, Akka and Mesos

Types and Safer Scala● Keynote: Scala - where it came from, where it’s going● Towards a Safer Scala● Leveraging Scala Macros for Better Validation● Type-level programming in Scala 101● Improving Correctness with Types● Happy Paths: Functional Constructs in the Wild● The Scalactic Way● Delimited dependency-typed monadic checked

exceptions

The Rest...Many excellent talks were outside these categories● Why Scala.js● Reactive Slick for Database Programming● Exercise in machine learning● Functional Natural Language Processing● If I Only Had a Brain...in Scala● Akka HTTP: the Reactive Web Toolkit● many many others

Highlights of Specific Sessions

A quick taster of what was in some of the more popular sessions...

Scala - Where it came from, where it’s going

Where it’s from, where it’s going“Scala is a gateway drug to Haskell” (in actual fact it’s going well beyond Haskell.)

Slides: http://www.slideshare.net/Odersky/scala-days-san-francisco-45917092

Came from a practical combination of OOP and functional programming- Funny story about hipster syntax (..) instead of [..], <..> instead of [..], ??? instead of <..>- Trend in Type Systems- Scala JS is no longer experimental- TASTY: new scala-specific platform- Introduction to DOT- Type Parameters- Better treatment of effects with implicit Functions instead of Monads

TASTYScala faces challenges:

● binary compatibility● having to pick a platform: JDK( 7,8,9,10, ?) or Javascript.

Proposing a scala-specific platform called TASTY, (serialized typed abstract syntax tree),as an intermediate representation before bytecode - carries metadata of types, can be compiled with different versions of JDK, and to Javascript.

Tasty will enable:● instrumentation● optimization● code analysis● refactoring● publish once run anywhere● automated remapping to solve binary compatibility issues.

Explorations:Implicits that compose

Explorations:Hope to find something cooler than Monads to handle effects.● Monads don’t commute● Require Monad transformers for composition● Monad transformers make Martin’s head explode

Toward a Safer ScalaLeif Wickland

http://tinyurl.com/sd15lint

● Scalac enables some error-prone code.○ head of empty List?

● Using Static Analysis to detect errors early● IDE based solutions

○ Inconsistencies○ If not in release build process, doesn’t exist

● Web-based analysis○ outside of compile loop○ relatively immature analysis

Scalac switches

● Scala Style● Wart Remover● Linter● Watch

○ Abide○ Scapegoat○ Supersafe

Life Beyond the Illusion of PresentJonas Bonér:

The idea of the present is an illusion. Everything we see, hear and feel is just an echo from the past. But this illusion has influenced us and the way we view the world in so many ways.

There is no present, all we have is facts derived from the merging of multiple pasts. The truth is closer to Einstein’s physics where everything is relative to one’s perspective. As developers we need to wake up and break free from the perceived reality of living in a single globally consistent present.

The advent of multicore and cloud computing architectures meant that most applications today are distributed systems—multiple cores separated by the memory bus or multiple nodes separated by the network—which puts a harsh end to this illusion.

The only way to design truly scalable and performant systems that can construct a sufficiently consistent view of history—and thereby our local “present”—is by treating time as a first class construct in our programming model and to model the present as facts derived from the merging of multiple concurrent pasts.

How do we deal with failure and communication unreliability in real life?Confirmation and repetition

We can’t force the world into a globally consistent present (CRUD).

Mentioned 2 paradigms/theories: ● CALM (consistency as logical monotonicity)● CRDT (Commutative Replicated Data Type)

CRDT (Commutative Replicated Data Type) eventually consistent data types● minimize contention / coordination in a distributed system.● set, maps, graphs: rich data types.● monotonic merge function: all state change is monotonically increasing, no way

back.

Improving Correctness with Typeshttp://workday.github.io

● Using Wrapper Types (aka Tiny Types) instead of primitives● Never use null, or throw Exceptions● use === org.scalactic.TypeCheckedTripleEquals

○ requires the types of the two values compared to be in a subtype/supertype● Never Use Non-Empty Lists when a list must be populated (org.scalactic.Every)● Use Type Tags (ala Shapeless, Scalaz)● Use Path Dependent Types● Other reading

○ Self recursive types○ Phantom Types○ Shapeless○ Scalactic

Types: Defensive Programming. Fail Fast. Design By Contract.

Function Passing Style Heather Miller:

● A new programming model called function passing designed to overcome many of imperative / weakly-typed issues found in traditional “big data” processing systems.

● Provides a more principled substrate on which to build data-centric distributed systems.

● Pass safe, well-typed serializable functions to immutable distributed data● Based on her work on Pickling and Spores● Uses Spores (Serializable Functions) for a distributed model.● Kind of an Inverse of the Actor Model● Stateless. data is stationary, functions are passed around.● Uses Data Silos accessed through a Silo Ref.

The Unreasonable Effectiveness of Scala for Big Data

Dean Wampler● How Hadoop Works - Map Reduce

○ Problems■ Hard to implement algorithms■ The Hadoop API is horrible

● Scalding○ An improved Hadoop API in Scala○ Problems

■ Still uses a batch mode● Spark

○ An elegant, functional API○ Still in Batch Mode, but with mini batches which

approach real time.

Akka HTTP: The Reactive Web ToolkitRoland Kuhn

● Replaces Spray● Uses Akka Streams

○ Sources emit values to the stream○ Sinks receive values, act on them○ Sources can compose using Zip and Graph shapes

● “The pinball interpreter”○ produce data○ move downstream through transformations○ get to the effect○ go up and ask for more data○ Filters interrupt the flow before getting to the effect, make the pinball go back

upstream.● A live coded demonstration of using Streams and Http● Expected timeline for Streams - 4 weeks● Expected timeline for Http - 8 weeks

The Scalactic WayBill Vennershttp://www.slideshare.net/bvenners/the-scalactic-way

ScalaTest: quality through tests

Scalactic: quality through types

SuperSafe: quality through static analysis

Reactive Slick for Database ProgrammingStefan Zeigerhttp://slick.typesafe.com/talks/scaladays2015sf/Reactive_Slick_for_Database_Programming.pdf

Slick 3.0

● JDBC is inherently blocking (and blocking ties up threads) ● Traditional Model

○ Fully synchronous○ One thread per web request ○ Contention for Connections (getConnection blocks)○ Database back-pressure creates more blocked threads○ Doesn’t scale

New Slick Architecture makes use of a new DataType to provide Asynchronous Database I/O

● based on State, IO and Free Monads.● Returns a Future[R]● Creates a separate ExecutionContext, avoiding blocking of current thread● Works with akka-streams to create back pressure, so DB only gives as much data as client can

process.● For performance purposes it pre-fetches some data to keep client busy while it waits for the next

portion from DB.

sealed trait DBIOAction[+R, +S <: NoStream, -E <: Effect]{

def map[R2](f: R => R2)(implicit executor: ExecutionContext): DBIOAction[R2, NoStream, E]

def flatMap[R2, S2 <: NoStream, E2 <: Effect](f: R => DBIOAction[R2, S2, E2])(implicit

executor: ExecutionContext): DBIOAction[R2, S2, E with E2]

...

}

Easy Scalability with Akka

Easy Scalability with Akka

Michael Nash● Reviewed Akka, CQRS, ES● Introduced Distributed DDD● Identical clustered system with DDDD and Without● Gatling performance tests on both

ConductR● Application manager that empowers ops to

deploy distributed systems● Uses Akka, Play, Aka Streams, Akka

Cluster, FSM, Akka Data Replication● How can we run cluster based apps ensuring

the seed nodes are started first?○ State replicated using Data

Replication● How can we consolidate logging?

○ Using Akka Streams● How can we avoid batching?

○ Use Event Driven Architecture● How can monitor/test multiple nodes?

○ Use the visualizer built into conductor● How can we share state among the nodes?

○ Use Akka Data Replication

Scala.js

Building a data-driven map of the world in 10 minutes….

Whats wrong with Webapps?● Everything String/Map[String, String]● Compiler cannot help you!

○ document.getElementByld("my-id")● Javascript!

○ ["10", "10", "10", "10"].map(parseInt) [10, NaN, 2, 3] // WTF

ScalaJS● lets you write webapp in one language● Swap String-typing for Strong-typing● Type Everything!!

○ val res: Future[Seq[String]] = Ajax[Api].list(inputBox.value).call()● IS NOT JAVASCRIPT

○ scala.js> List("10", "10", "10", "10").map(parseInt) List(10, 10, 10, 10) // Yay!

The BoldRadius Crew Packing up...

The End….

ScalaDays Amsterdam!

● June 8th to 10th● Followed by Training

Q & Amichael.nash@boldradius.comadam.murray@boldradius.com