Building Resilient Distributed Systems by Using Caching Command and Rollback-Replay

Click here to load reader

  • date post

    08-Jan-2017
  • Category

    Technology

  • view

    365
  • download

    0

Embed Size (px)

Transcript of Building Resilient Distributed Systems by Using Caching Command and Rollback-Replay

PowerPoint Presentation

Building Resilient Distributed Systems by Using Caching Command and Rollback-Replay

Tanuja [email protected]

#GHC162016

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY The problem with resiliency in distributed systemsSingle node system

NodeDatabase1Web containercachingDatabase2All components reside in the same machine.

Its not too hard to ensure atomicity.Either alloccur ornothingoccurs

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY

The problem with resiliency in distributed systems

nodenode1node2

node2node1

node1node2Components are spread out.Maintaining atomicity and resiliency is a challenge.So we strive for eventual consistency.The change will eventually be propagated to all the copies of data.

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Intuit case study: Login ServiceIntuit makes financial software. Many of these products use the Login service for login and fetching users bank accounts and transactions securely.

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Requirements for the Login ServiceFast response timesResilienceFault-toleranceConsistency

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY

5

4-step solution we used to solve the problemDecouple design Implement single responsibility principle (SRP)Use the command pattern Use circuit breaker frameworkUse reactor to recoverUse caching (record)

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY 1. Decouple designIndividual components can be developed independently.Plug and play components into bigger solution.

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY 1a. Implement single responsibility principleModule or class should have responsibility over a single part of the functionality provided by the software, and that responsibility should be entirely encapsulated by the class. All its services should be narrowly aligned with that responsibility. Separation of concernsEach module/method does only one task.

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY 1b. Use command pattern

InvokerClientcreates Commandexecute()recover()ConcreteCommand AConcreteCommand Bimplementsusescreates A behavioral design pattern in which an object is used to encapsulate all information needed to perform an action or trigger an event at a later time.

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY

9

Benefits of the command pattern Each command knows how to execute itself.Each command knows how to react to failures.Rollback RetrySomething else

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Traditional model with services

OrchestrationHandlerService AService Bcreateupdatedeletegetcreateupdatedeleteget

GETPUT

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Introduce commands

OrchestrationHandlerService AService Bcreateupdatedeletegetcreateupdatedeleteget

GETPUTCommand create, update ...Commandcreate, update ...

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY 2. Use circuit breakerCircuit breaker is used to detect failures, and encapsulates logic to reacting to failure (during maintenance, temporary external system failure or unexpected system difficulties).The circuit breaker pattern is a stability patterns applied in a RESTful architecture.Several open sources are available (Hystrix is developed by Netflix and is popular open source).

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Example of circuit breaker

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY 3. Use reactorGets invoked in case of failure.We can specify the behavior.Rollback RetryTrigger a back-up Fallback

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Use circuit breaker

OrchestrationHandlerService AService Bcreateupdatedeletegetcreateupdatedeleteget

GETPUTCircuit breakerFallbackShort circuitLog errorError responseCommandCommand

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY 4. Use cachingUse cache to save the commands so that they can be used for recovery.Some popular open source solutions:HazelcastMemcacheRedis

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Based on fast reads, writes requirements we can choose chaches. For mem cache read is fastest, max size of value.http://blog.engineering.aol.com/2015/08/28/a-comparative-study-of-distributed-caches/17

4. Use caching

OrchestrationHandlerService AService Bcreateupdatedeletegetcreateupdatedeleteget

GETPUTFallbackShort circuitLog errorError responseCache ClientCacheCache Listener[Reactor]

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Full resilient picture

OrchestrationHandlerService AService Bcreateupdatedeletegetcreateupdatedeleteget

GETPUT

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Service A fails

OrchestrationHandlerService AService Bcreateupdatedeletegetcreateupdatedeleteget

GETPUTFallbackShort circuitLog errorError response

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Service A is successful and Service B fails

OrchestrationHandlerService AService Bcreateupdatedeletegetcreateupdatedeleteget

GETPUTFallbackShort circuitLog errorError responseCache ClientCacheCache Listener[Reactor]Reactor(recover)Caching dirty

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Service A and Service B both succeed

OrchestrationHandlerService AService Bcreateupdatedeletegetcreateupdatedeleteget

GETPUTFallbackShort circuitLog errorError responseCache ClientCacheCache Listener[Reactor]Caching(record)Not Dirty

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY 4-step solution we used to solve the problemDecouple design Implement single responsibility principle (SRP)Use the command pattern Use caching (record)Use circuit breaker frameworkUse reactor to recover

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Our story: How did we benefit?Over 100 user update requests were failing.They got slow responses.Resulted in high CPU utilization and cascading failures. After we implemented this solution, we failed fast and could adhere to the SLAs.

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY For more info ...Retry pattern https://msdn.microsoft.com/en-us/library/dn589788.aspx

Command Handlinghttp://www.axonframework.org/docs/2.0/command-handling.html

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY Rate and review the session on our mobile app

Thank you

Feedback?Download at http://bit.ly/ghc16app or search GHC 16 in the app store

PAGE # | GRACE HOPPER CELEBRATION 2016 | #GHC16PRESENTED BY THE ANITA BORG INSTITUTE AND THE ASSOCIATION FOR COMPUTING MACHINERY