System Reliability and Resilience and stuff. Some things need to be cleared up first.
66
System Reliability and Resilience and stuff
-
Upload
hannah-mosley -
Category
Documents
-
view
227 -
download
0
Transcript of System Reliability and Resilience and stuff. Some things need to be cleared up first.
SystemReliability and
Resilienceand stuff
Some things need to be cleared up first
http://en.wikipedia.org/wiki/Vedette_(cabaret)
tuple
//Initialize customer and invoiceInitialize(customer, invoice);
public void Initialize(Customer customer, Invoice
invoice){
customer.Name = “asdf”;invoice.Date = DateTime.Now;
}
Initialize(customer, invoice);//did something happen to customer// and/or invoice?
customer.Name =
InitNameFrom(customer, invoice);invoice.Date =
InitDateFrom(customer, invoice);
customer.Name =
GetNameFrom(customer, invoice);invoice.Date =
GetDateFrom(customer, invoice);
var results = Initialize(customer,
invoice);
customer.Name = results.Item1;invoice.Date = results.Item2;
public tuple<string, DateTime>Initialize(customer,
invoice){
return new Tuple<string, DateTime>
(“asdf”, DateTime.Now);}
public static bool TryParse(string s, out DateTime result)
or
public static tuple<bool, DateTime?>
TryParse(string s)
tuple• Avoid side effects• Avoid out parameters•multiple values without a specific type
null object
private ILogger _logger;public MyClass(ILogger logger) {
_logger = logger;}
…
if (_logger != null) {_logger.Debug(
“it worked on my machine!”);}
null checks for everyone!
forget one and…
public class NullLogger : ILogger {
public void Debug(string text) {
//do sweet nothing}
}
private ILogger _logger = new NullLogger();
public MyClass(ILogger logger) {_logger = logger;
}
…
_logger.Debug(“it worked on my machine!”);
null object• Can eliminate null checks• Simple to implement
Circuit Breaker
Retry
Your
App
licat
ion Out of Process
Dependency
N times
Out of Process Dependency
N times*
Y clients
= Denial of
Service Attack
Limit the # of retries
N * Ybecomes5 * Y
Y isstill a
problem
Circuit Breaker
State Machine
On :: Off
On Offwhen not healthy
Off Onmanually
Get to softwarebefore we ask you to dance
Healthyor
Unhealthy
Out of Process Dependency
State is independent of requestor
Out of Process Dependency
Your
App
licat
ion Has many
independent external dependencies
Your
App
licat
ion
Can throttle itself
Your
App
licat
ion
Has a wait threshold
Your Application
External Dependency
Circuit Breaker
Threshold = 2Pause = 10msTimeout = 30sState = ClosedRequest
Request
Failure (i.e. HTTP 500)Failure Count = 1Pause 10ms
Request
Failure (i.e. HTTP 500)Failure Count = 2State = Open
OperationFailedException
Threshold = 2Pause = 10msTimeout = 30sState = OpenRequest
30s has not passed
CircuitBreakerOpenException
Request
30s has not passed
CircuitBreakerOpenException
System can try to
become healthyfor 30s
Your Application
External Dependency
Circuit Breaker
Threshold = 2Pause = 10msTimeout = 30sState = ½ OpenRequest
Request
Failure (i.e. HTTP 500)Failure Count = 2State = Open
OperationFailedException
30s has passed
Your Application
External Dependency
Circuit Breaker
Threshold = 2Pause = 10msTimeout = 30sState = ½ OpenRequest
Request
Failure Count = 0State = Closed
Response
30s has passed
Response
Your Application
External Dependency
Circuit Breaker
ClosedOpen
½ Open
½ Open is like a
manual reset
PauseTimeout
Pausebetween calls
in the loop
Timeoutbefore you
can call again
Exceptions
OperationFailed:
AggregateException
CircuitBreakerOpen:
ApplicationException
Don’t Loose Exception Info
Always use InnerException(s)
Threshold = 3State = ClosedRequest
Request
Failure (i.e. HTTP 500)Request
Failure (i.e. HTTP 500)Failure Count = 2
Failure Count = 0State = Closed
Response
Response
Request?Your
ApplicationExternal
DependencyCircuit
Breaker
Failure Count = 1
SegregateDependencies
circuitBreaker(“database”)
circuitBreaker(“weatherservice”)
Dependency type, endpoint svc,
endpoint
Where?
Your
App
licat
ion Out of Process
DependencyCi
rcui
t Bre
aker
Prox
y
Watch forInception
Your
App
licat
ion W
eb ServiceCi
rcui
t Bre
aker
Circ
uit B
reak
er
Prox
y
DatabaseRepo
sitor
y
circuit breaker• retry looping• slow down attempts• good neighbour
¡Muchas gracias!
