Download - Building Resilient Distributed Systems by Using Caching Command and Rollback-Replay

| GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY

#GHC16

2016Building Resilient Distributed Systems by Using Caching Command and Rollback-Replay

Tanuja Phadke

[email protected]


The problem with resiliency in distributed systems

Single node system

Node

Database1

Web container caching

Database2

• All components reside in the same machine.

• It’s not too hard to ensure atomicity.• Either all occur or nothing occurs


The problem with resiliency in distributed systems

node node1

node2

node2node1

node1 node2

• Components are spread out.• Maintaining atomicity and resiliency is a

challenge.• So we strive for eventual consistency.• The change will eventually be propagated

to all the copies of data.


Intuit case study: Login Service

Intuit makes financial software. Many of these products use the Login service for login and fetching users’ bank accounts and transactions securely.


Requirements for the Login Service

• Fast response times• Resilience• Fault-tolerance• Consistency


4-step solution we used to solve the problem

1. Decouple design a. Implement single responsibility principle (SRP)b. Use the command pattern

2. Use circuit breaker framework3. Use reactor to recover4. Use caching (record)


1. Decouple design

• Individual components can be developed independently.• Plug and play components into bigger solution.


1a. Implement single responsibility principle

• Module or class should have responsibility over a single part of the functionality provided by the software, and that responsibility should be entirely encapsulated by the class. All its services should be narrowly aligned with that responsibility.

• Separation of concerns• Each module/method does only one task.


1b. Use command pattern

Invoker

Client

creates

<<interface>> Command

execute()recover()

ConcreteCommand A

ConcreteCommand B

implements

uses

creates

A behavioral design pattern in which an object is used to encapsulate all information needed to perform an action or trigger an event at a later time.


Benefits of the command pattern

• Each command knows how to execute itself.• Each command knows how to react to failures.• Rollback • Retry• Something else


Traditional model with services

Orchestration

Handler

Service A

Service B

create

update

delete

get

create

update

delete

get

GET

PUT


Introduce commands

Orchestration

Handler

Service A

Service B

create

update

delete

get

create

update

delete

get

GET

PUT

Command create, update ...

Commandcreate, update ...


2. Use circuit breaker

• Circuit breaker is used to detect failures, and encapsulates logic to reacting to failure (during maintenance, temporary external system failure or unexpected system difficulties).

• The circuit breaker pattern is a stability patterns applied in a RESTful architecture.

• Several open sources are available (Hystrix is developed by Netflix and is popular open source).


Example of circuit breaker


3. Use reactor

• Gets invoked in case of failure.• We can specify the behavior.• Rollback • Retry• Trigger a back-up • Fallback


Use circuit breaker

Orchestration

Handler

Service A

Service B

create

update

delete

get

create

update

delete

get

GET

PUT

Circuit breaker

Fallback

Short circuitLog error

Error response

CommandCommand


4. Use caching

• Use cache to save the commands so that they can be used for recovery.

• Some popular open source solutions:• Hazelcast• Memcache• Redis


4. Use caching

Orchestration

Handler

Service A

Service B

create

update

delete

get

create

update

delete

get

GET

PUT

Fallback


Error response

Cache Client Cache

Cache Listener

[Reactor]


Full resilient picture

Orchestration

Handler

Service A

Service B

create

update

delete

get

create

update

delete

get

GET

PUT


Service A fails

Orchestration

Handler

Service A

Service B

create

update

delete

get

create

update

delete

get

GET

PUT

Fallback


Error response


Service A is successful and Service B fails

Orchestration

Handler

Service A

Service B

create

update

delete

get

create

update

delete

get

GET

PUT

Fallback


Error response

Cache Client Cache

Cache Listener

[Reactor]

Reactor(recover)Caching dirty


Service A and Service B both succeed

Orchestration

Handler

Service A

Service B

create

update

delete

get

create

update

delete

get

GET

PUT

Fallback


Error response

Cache Client Cache

Cache Listener

[Reactor]

Caching(record)Not Dirty


4-step solution we used to solve the problem

1. Decouple design a. Implement single responsibility principle (SRP)b. Use the command pattern

2. Use caching (record)3. Use circuit breaker framework4. Use reactor to recover


Our story: How did we benefit?

Over 100 user update requests were failing.• They got slow responses.• Resulted in high CPU utilization and cascading failures.

After we implemented this solution, we failed fast and could adhere to the SLAs.


For more info ...

Retry pattern https://msdn.microsoft.com/en-us/library/dn589788.aspx

Command Handlinghttp://www.axonframework.org/docs/2.0/command-handling.html


Thank you

Feedback?

Download at http://bit.ly/ghc16app or search GHC 16 in the app store

Rate and review the session on our mobile app

http://bit.ly/ghc16app