2016 05-16 testing_distributed_systems_v1 1
-
Upload
feedzai -
Category
Technology
-
view
125 -
download
1
Transcript of 2016 05-16 testing_distributed_systems_v1 1
![Page 1: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/1.jpg)
Testing Distributed Systems
Presented by Miguel Silva, QA Team Leader
![Page 2: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/2.jpg)
Meet Feedzai
https://feedzai.wistia.com/medias/toz8ny7ep7
![Page 3: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/3.jpg)
Buzzwords
![Page 4: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/4.jpg)
Numbers
99.95 availabilityMax 250ms latency
$2B of transactions processed per day3 deployments in fortune 500 companies
Growing 2 person / week (Hiring!)
![Page 5: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/5.jpg)
A typical deployment
Pulse serverApp engineSQL Databases
ZookeeperRabbitMqCassandra
18 machines
![Page 6: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/6.jpg)
The largest deployment
2 Datacenters 6 environments 150 Servers 13Tb of memory 3000 cores 250Tb of storage
![Page 7: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/7.jpg)
Go beyond unit tests
End to end functionalityFault-toleranceHigh-availabilityMachine / datacenter synchronization
![Page 8: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/8.jpg)
Docker and bash
Hard to maintainHard to expandHard to CIHard to test scriptsHard to script tests
![Page 9: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/9.jpg)
System-tests framework
![Page 10: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/10.jpg)
System-tests framework
![Page 11: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/11.jpg)
System-tests framework
Write system-tests like unit tests
![Page 12: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/12.jpg)
System-tests framework
Write system-tests like unit tests
![Page 13: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/13.jpg)
Feature highlights
Set a multi-container, multi-datacenter test scenario in couple of lines
Easy to assert cluster states, connections, applications, etc.
Make deployment-like scenario system-tests like you do a unit test
![Page 14: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/14.jpg)
Feature highlights
Abstracted container creation, setup, volumes complexity from the tests
Integrated in CI environment (jenkins)
Developers easily implement system-tests
![Page 15: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/15.jpg)
Bonus – production firewalls
All containers are blocked by a firewall
When testing, open only the necessary ports
Ensures functionality with production security
![Page 16: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/16.jpg)
Bumps along the way
Docker demon may have some concurrency problems and some calls fail in some tests
Containers would not be removed
Solved by doing retries on some docker calls
![Page 17: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/17.jpg)
Bumps along the way
Docker-java leaves TCP IP sockets stuck (TCP-WAIT)
No sockets available during a build run
Workaround by enabling socket reuse at kernel level
![Page 18: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/18.jpg)
Bumps along the way
Before 1.9, container IP would change when containers are stopped/etc/hosts updated on every docker run
After 1.9, IPs are static and more predictableContainers always keep their IP
![Page 19: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/19.jpg)
Current status
128Gb RAM24 CPU coresover SSD
315 tests3h build7 parallel tests runningUp to 100 parallel containers running
bottleneck
![Page 20: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/20.jpg)
Next steps
Integrate more images
Docker swarm integration
Distributed test execution
![Page 21: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/21.jpg)
Thank You
![Page 22: 2016 05-16 testing_distributed_systems_v1 1](https://reader035.fdocuments.in/reader035/viewer/2022070518/58e8a5891a28abda4f8b47c5/html5/thumbnails/22.jpg)
Visit www.feedzai.com/wickedsmart
LIVE EVENTS | MEET THE PEOPLE BEHIND THE MACHINE