EuroPython 2014 Devops Risk Mitigation

21
DevOps Risk Mitigation www.immobilienscout24.de Berlin | 23.07.2014 | Schlomo Schapiro Systems Architect, Open Source Evangelist License: http://creativecommons.org/licenses/by-nc-nd/3.0/ Test Driven Infrastructure @schlomoschapiro

description

The (perceived) risk of the DevOps is that too many people get the right to "break" the platform. Test Driven Infrastructure is about adapting proven ideas from our developer colleagues to the development and operations of Infrastructure services like virtualization, OS provisioning, postfix configuration, httpd configuration, ssh tuning, SAN LUN mounting and others. This talk shows how ImmobilienScout24 utilizes more and more test driven development in IT operations to increase quality and to mitigate the risk of opening up the infrastructure developmen to all developers. See https://ep2014.europython.eu/en/schedule/sessions/44/ for conference archive.

Transcript of EuroPython 2014 Devops Risk Mitigation

Page 1: EuroPython 2014 Devops Risk Mitigation

DevOps Risk Mitigationwww.immobilienscout24.de

Berlin | 23.07.2014 | Schlomo SchapiroSystems Architect, Open Source Evangelist

License: http://creativecommons.org/licenses/by-nc-nd/3.0/

Test Driven Infrastructure

@schlomoschapiro

Page 2: EuroPython 2014 Devops Risk Mitigation

ExpensiveFix

CheapFix

Costs Of Finding Bugs In Production

Page 3: EuroPython 2014 Devops Risk Mitigation

PlanDesignBudget

Develop Test

RU

NDEV

PlanDesignBudget

Develop Test RUN OPSTIME

Page 4: EuroPython 2014 Devops Risk Mitigation

PlanDesignBudget

Develop Test

PlanDesignBudget

Develop Test RUN

Proxyconfig

Databaseborked

„Buy Now“button broken

MTA dropall mail

Load BalancerConfiguration

Everything costsonly 0 €

Login possiblewithout password

NFS notavailable

DB Replicationstopped

No adsshown

BrokenCSS / JS

Tomcat won'tstart/stop

Service usernot defined

sudoersinvalid

Page 5: EuroPython 2014 Devops Risk Mitigation

DevOps: Respect & Learning

DEV learn from OPS to think about:

● Resources (CPU, RAM, Disk)● Services (Start, Stop, Status)● Dependencies (Start DB before App)● Logfiles (Rotate, Remove)● Disk Space● Monitoring and Alarming● ...

OPS learn from DEV to think about:

● Incremental Improvement● Infrastructure as Code● Version Control System● Coding (OO, Functions, Libraries …)● Code Quality● Unit & Integration Tests● Test Automation● ...

Page 6: EuroPython 2014 Devops Risk Mitigation

Untested=

Broken

Page 7: EuroPython 2014 Devops Risk Mitigation

Unit TestsTest the smallest possible components in an artifical environment.

System TestsTest the entire application in a real(istic) environment together with other applications.

Page 8: EuroPython 2014 Devops Risk Mitigation

Part of build process

Syntax checks

Scripts

Config Files

Data Files

Unit tests for functions/libs

Run program with test data

Check result

Check program behaviour with wrong/broken test data

Also run on Developer desktop

Quick feedback (~ seconds)

Install on test server

Run tests from outside

HTTP calls

Send emails

Try to login

Run tests from inside

Remote Exec (rsh, ssh …)http://go.schapiro.org/rshpitfall

Service Start, Stop & Status

Modify server to create good & bad test scenarios

Reboot

Unit Tests System Tests

Page 9: EuroPython 2014 Devops Risk Mitigation

UnitTests

Page 10: EuroPython 2014 Devops Risk Mitigation

%prep%setup -q

%installinstall … %{buildroot}/…install … %{buildroot}/…

%files%defattr(-,root,root,-)/...

Page 11: EuroPython 2014 Devops Risk Mitigation

BuildRequires: sudo%buildset -evisudo -c -f sudoers %installinstall -m 0440 sudoers -D \ %{buildroot}/etc/suoders.d/%{name} %files%defattr(-,root,root,-)/etc/suoders.d/%{name}

Page 12: EuroPython 2014 Devops Risk Mitigation

BuildRequires: PyYAML, pylint%buildset -e# syntax checksbash -n my_script.sh# Should be valid python codepylint -E yum-repo-propagate# should be valid YAML filepython -c "↩import yaml ↩yaml.safe_load(open('config.yaml'))↩"...

Page 13: EuroPython 2014 Devops Risk Mitigation

SystemTests

http://impreza-gt-club.ch/V2.0/Tests/WRX08/Koch2.jpg

Page 14: EuroPython 2014 Devops Risk Mitigation

BuildAuto-

mationSourceCode(SVN)

Monitor

Changes

Build Server

Run BuildJob

● Check out source● Run Unit Tests● Create RPM●

Sourc

eCod

eDEVYUMRepo

UploadPROYUMRepo

PropagateRPM

Test Server

Deploy

and

Run

Test

Job

yum

Prod Server

Deploy

to PROD

yum

Page 15: EuroPython 2014 Devops Risk Mitigation

BuildAuto-

mationSourceCode(SVN)

Monitor

Changes

Build Server

Run BuildJob

● Check out source● Run Unit Tests● Create RPM●

Sou

rce

Cod

eDEVYUMRepo

UploadPROYUMRepo

PropagateRPM

Test Server

Deploy

and

Run

Test Job

yum

Prod Server

Deploy

to PROD

yum

Page 16: EuroPython 2014 Devops Risk Mitigation

SAN mount service

Test via rsh

Mock SAN deviceswith losetup

service start, stop mounts/umounts

Error handling

Page 17: EuroPython 2014 Devops Risk Mitigation

HTTP Proxy Configuration

X-Forwarded-For header spoofs source

Check result for ERR_ACCESS_DENIED

Run tests for all function groups!

Test Server

GET http://external.com/X-Forwarded-For: 10.11.12.01

502 Bad Gateway✔GET http://external.com/X-Forwarded-For: 10.34.56.01

403 Forbidden

ERR_ACCESS_DENIED by proxy server

✘✘

Page 18: EuroPython 2014 Devops Risk Mitigation

VM Provisioning & Kickstart Installation

Test via HTTP API

Create broken VMs and check error reports

Create valid VM and install Linux OS

Scrape VM screen via OCR

http://github.com/Immobilienscout24/lab-manager-light

Page 19: EuroPython 2014 Devops Risk Mitigation

34 35 36

2 3

53

87 88 89 90 91 92 93 94 95 96 97

TIME

Continous Live Deployment

Deploy every application when it is ready.Automate the delivery chain from source till production.

Page 20: EuroPython 2014 Devops Risk Mitigation

Low Risk – Lots of Fun

http://go.schapiro.org/slides

Page 21: EuroPython 2014 Devops Risk Mitigation

Kontakt:Immobilien Scout GmbHAndreasstraße 1010243 Berlin

Fon: +49 30 243 01-1229 Email: [email protected]: www.immobilienscout24.de

Thank you very much!Please contact me for further questions and discussions.