Real World Akka Actor Recipes JavaOne 2013

Real -World Akka Actor RecipesBjörn Antonsson

@bantonsson

co-authored with

Jamie Allen and Patrik Nordwall

Tuesday, 1 October 13

@bantonsson@bantonsson

Overview

• What is Akka?• What are actors?• Getting into the flow• Getting your message across• Tying it all together

2


What is Akka?



What is Akka?

• Toolkit and runtime for reactive applications• Write applications that are– Concurrent– Distributed– Fault tolerant– Event-driven

4



What is Akka?

• Has multiple tools– Actors– Futures– Dataflow– Remoting– Clustering

5


What are actors?



What are actors?

• Isolated lightweight event-based processes• Share nothing• Communicate through async messages• Each actor has a mailbox (message queue)• Location transparent (distributable)• Supervision-based failure management

7



What is an actor good for?

• An island of sanity in a sea of concurrency• Everything inside the actor is sequential– Processes one message at a time

• Very lightweight– Create millions– Create short lived

• Inherently concurrent

8



Actors compared to Objects

• Think of an Actor as an Object• You can't peek inside it• You don't call methods– You send messages (asynchronously)

• You don't get return values– You receive messages (asynchronously)

• The internal state is thread safe

9


Why should I care?


The world is multicore!


@bantonsson@bantonsson12

Amdahl’s Law



So what's the catch?

• Really no catch• A different programming paradigm• All about tradeoffs– Some things are easier some harder

• Think different

13


Getting into the flow



Getting into the flow

• Why do you need flow control?• How do you control the flow?• Do you really need all messages?

15



Why do you need flow control?

• Function calls are blocking• Message sends are asynchronous• Possible problems– Produce jobs too fast– Many jobs need much CPU and/or Memory– External resources have limits– Unpredictable job patterns

16



Why do you need flow control?

• Free flow of messages can lead to– Blocked from external resource– Actor mailbox backup– Slow system– Out of Memory

17



How do you control the flow?

• Push with rate limiting– A fixed number of jobs per time unit

• Push with acknowledgment– A fixed number of jobs can be in progress– New jobs are started a"er old jobs finish

• Pull– New jobs are pulled as old are completed

18



Push with rate limiting

• A timer sends ticks at fixed intervals• On every tick the master gets new tokens• When there are no tokens, jobs get queued• When there are tokens, start queued jobs

19



Worker

Master [0]

Queue [0]

TimerWorker

1. Work



Worker

Master [0]

Queue [1]

TimerWorker

1. Work 2. Enqueue



Worker

Master [5]

Queue [1]

TimerWorker

1. Work 2. Enqueue

3. Tick



Worker

Master [4]

Queue [0]

TimerWorker

1. Work 2. Enqueue

4. Dequeue

3. Tick



Worker

Master [4]

Queue [0]

TimerWorker

1. Work 2. Enqueue

4. Dequeue

3. Tick 5. Work



Push with acknowledgement

• A fixed number of jobs are started• Wait for ACK before starting more jobs• Jobs that can't be started are queued• When ACK arrives start queued job• To keep workers busy, send more than one

job per worker– Use a high water mark to stop sending and a low

water mark to start sending

25



Worker

Master [0]

Queue [0]

Worker

1. Work



Worker

Master [0]

Queue [1]

Worker

1. Work 2. Enqueue



Worker

Master [1]

Queue [1]

Worker

1. Work 2. Enqueue

3. Ack



Worker

Master [0]

Queue [0]

Worker

1. Work 2. Enqueue

4. Dequeue3. Ack



Worker

Master [0]

Queue [0]

Worker

1. Work 2. Enqueue

4. Dequeue

5. Work3. Ack



Worker

Master [1]

Queue [0]

Worker

1. Work 2. Enqueue

4. Dequeue

5. Work

6. Ack

3. Ack



Pull

• Incoming jobs are queued• Workers ask for jobs• Jobs are handed out when available• Workers don't do active polling• Can lead to lag if jobs are small compared to

cost of getting a new job– Use batching to counteract lag

32



Worker

Master

Queue [0]

Worker

1. Work



Worker

Master

Queue [1]

Worker

1. Work 2. Enqueue



Worker

Master

Queue [1]

Worker

1. Work 2. Enqueue

3. Work?



Worker

Master

Queue [0]

Worker

1. Work 2. Enqueue

4. Dequeue

3. Work?



Worker

Master

Queue [0]

Worker

1. Work 2. Enqueue

4. Dequeue

5. Work

3. Work?



Do you really need all messages?

• Group messages together– Batching

• Discard/Aggregate messages– Scrubbing

38



Batching

• Collect a number of messages before sending/processing them– A predefined number of messages or time

• Useful for things like– Write behind– Database bulk insert/update– Heavyweight operations e.g. GUI rendering

39



Scrubbing

• Discard or aggregate some messages– Predefined number of messages or time

• Useful for things like– Financial market data– Statistics

40


Getting your message across



Getting your message across

• When is a message delivered?• The fallacy of guaranteed delivery• What Akka guarantees• Reliable messaging

42



When is a message delivered?

• Function calls block until done• Message sends return immediately• Which is the right point?– Sent/Received Network?– Enqueued/Dequeued Mailbox?– Processed by Actor?

• Do ACKing at the business level

43



Guaranteed delivery

• From Enterprise Integration Patterns• Messaging system uses built-in store to

persist • ACK everywhere– Producer to sender– Sender to receiver– Receiver to consumer

44



Lots of ACKs. What if I just...

• Use Durable Mailboxes?– When is the message in the mailbox?– No guarantees that it ever got there– Still have to ACK to be certain

45



Lots of ACKs. What if I just...

• Use an External Durable Message Queue– A SPOF/Bottleneck?– When is the message in the message queue?– The queue does ACKing internally– No guarantees that it ever gets out– Still have to ACK to be certain

46



Guaranteed delivery doesn't exist

• Things break– Persistent store crashes– Network fails– Server goes down

• Design for failure and resilience• Do ACKing at the business level

47



What Akka guarantees

• At most once delivery– Message is only delivered once, if at all– The weakest guarantee

• Ordered per actor sender-receiver pair– Actor A sends messages to actor B– If the messages are received by actor B,

it will be in the order as sent by actor A

48



Other delivery guarantees

• At least once– Message will eventually be delivered– Can happen multiple times

• Exactly once– Message will eventually be delivered– Will only happen once

• Have to add these yourself– They involve ACKing ;)

49



Reliable Messaging: At least once

• Send with acknowledge– Keep sending until you get an ACK

• Receive with re-request– When missing a message request it– Needs unique sequence numbers

• Requires– Message store at sender (available/redundant)

50



SenderReceiver

Store



SenderReceiver

Store

1. Message



SenderReceiver

Store

2. Message



SenderReceiver3. Ack

Store

2. Message



SenderReceiver3. Ack

Store

4. Ack

2. Message



Reliable Messaging: At least once

• Use a pipe of actors before ACKing• Keep pipe free of side effects– Same message might come several times

• ACK should be done at the business level

56



Sender

CheckInventory

Authenticate

ReceiverStore



Sender

CheckInventory

Authenticate

ReceiverStore

1. Message



Sender

CheckInventory

Authenticate

Receiver

2. Message

Store



Sender

CheckInventory

Authenticate

Receiver

2. Message

3. Message'

Store



Sender

CheckInventory

Authenticate

Receiver

4. Message''

2. Message

3. Message'

Store



Sender

CheckInventory

Authenticate

Receiver

4. Message''

2. Message

3. Message'

5. Ack

Store



Sender

CheckInventory

Authenticate

Receiver

4. Message''

2. Message

3. Message'

5. Ack

Store

6. Ack



Receive with re-request

• When missing a message request it• Requires– Uniquely identifiable sequence– Message store at sender (available/redundant)

64



SenderReceiver

Store



SenderReceiver

Store

1. Message



SenderReceiver

Store

2. Message



SenderReceiver

Store

3. Message'



SenderReceiver

Store

4. Message'



SenderReceiver5. Request

Store

4. Message'



SenderReceiver

Store

6. Message



SenderReceiver

Store

6. Message

7. Message



SenderReceiver

Store

8. Message'



SenderReceiver

Store

8. Message'

9. Message'


Tying it all together




• Distributed Workers Example– Front Ends receive requests– Master receives work from Front Ends– Workers pull work from Master

• Available as a Typesafe Activator template– Zero configuration setup– Code and tutorial

76



Worker

Master

Worker

Worker

Worker

Frontend

Frontend




• Further Goals– Elastic addition/removal of front end nodes– Elastic addition/removal of workers– Thousands of workers– Jobs should not be lost

78



Master

Worker

Frontend

WorkExecutor



Master

Worker

Frontend

WorkExecutor

RegisterWorker



Master

Worker

Frontend

1. Work

WorkExecutor



Master

Worker

Frontend

1. Work

2. Work

WorkExecutor



Master

Worker

Frontend

1. Work

2. Work

3. Ack

WorkExecutor



Master

Worker

Frontend

1. Work

2. Work

3. Ack

4. OK

WorkExecutor



Master

Worker

Frontend

1. Work

2. Work

3. Ack

4. OK

5. WorkIsReady

WorkExecutor



Master

Worker

Frontend

1. Work

2. Work

3. Ack

4. OK

5. WorkIsReady

6. WorkerRequestWork

WorkExecutor



Master

Worker

Frontend

1. Work

2. Work

3. Ack

4. OK

5. WorkIsReady

6. WorkerRequestWork7. Work

WorkExecutor



Master

Worker

Frontend

1. Work

2. Work

3. Ack

4. OK

5. WorkIsReady

6. WorkerRequestWork7. Work

WorkExecutor

8. Work



Master

Worker

Frontend

WorkExecutor



Master

Worker

Frontend

WorkExecutor

9. WorkComplete



Master

Worker

Frontend

WorkExecutor

9. WorkComplete

10. WorkIsDone



Master

Worker

Frontend

WorkExecutor

9. WorkComplete

10. WorkIsDone

11. Ack



Cluster Technologies/Patterns

• Distributed Pub/Sub Mediator– Publish and Subscribe to message flows

• Cluster Singleton– HA singleton actor instance within the cluster

• Cluster Client– Let other systems connect to the cluster

93



DistributedPubSubMediator94

Frontend Master

Mediator Mediator




Frontend Master

Mediator Mediator

Put




Frontend Master

Mediator Mediator

Send



ClusterSingleton98

ClusterSingletonManager


MasterMaster(Standby)



ClusterSingleton99



MasterMaster(Standby)



ClusterSingleton100



Master Master



ClusterClient & ClusterSingleton101

Master

Mediator

Mediator

Receptionist

Master(Standby)

Receptionist

ClusterClient

ClusterClient

Worker

Worker



Master

Mediator

Mediator

Receptionist

Master(Standby)

Receptionist

ClusterClient

ClusterClient

Worker

Worker

ClusterClient

Worker

Worker

Frontend

Frontend


Resources



Resources: Blog Posts

• Push with rate limiting– Kaspar Fischer

http://letitcrash.com/post/28901663062/throttling-messages-in-akka-2

• Pull– Derek Wyatt

http://letitcrash.com/post/29044669086/balancing-workload-across-nodes-with-akka-2

– Michael Pollmeierhttp://www.michaelpollmeier.com/akka-work-pulling-pattern-to-throttle-work/

104



Resources: Typesafe Activator

• http://typesafe.com/platform/getstarted

105



Resources: Coursera Course

• Principles of Reactive Programming byMartin Odersky, Erik Meijer and Roland Kuhn– Starts 4th of November 2013– 7 weeks– Workload: 5-7 hours a week– Free as in free beer

• https://www.coursera.org/course/reactive

106


Björn Antonsson

[email protected]

@bantonsson


Real World Akka Actor Recipes JavaOne 2013

Technology

Transcript of Real World Akka Actor Recipes JavaOne 2013