Call Me Never
Building apps that don't wake you up
David Dossot
5 million visits a day...
... And counting.
Aiming for five nines...
... And 100x traffic
99.9962( °□°╯ )╯︵ ┻━┻
ReadinessDegradation
RecoveryStrengthening
Load test
Readiness
Quick an dirty
Apache Bench
Richer scenarios
Apache jMeterGatling
Profiling
Readiness
Readiness
Replay access logs
Readiness
Welcome to the Internet
GET /admin.php
GET /catalog/compare/?action=DELETE_FROM_COMPARE_RESULT&IBLOCK_ID=3
Host: -- DROP DATABASE users;
Degradation
Decouple with queues and topics
Degradation
Delay, don't fall down
Degradation
Short circuit blocking-IOs
cache.get(key)
Degradation
Beware of benign code
Recovery
Self heal from transient failures
Recovery
Self heal from transient failures
Recovery
Maintain stable state
Recovery
Replay stable state
Strengthening
Healthy health checks
Strengthening
Healthy health checks
Strengthening
Code defensively and log abundantly
try {
// risky stuff
} catch {
// useful log
}
Strengthening
Failure is opportunity for more tests