Ansible benelux meetup - Amsterdam 27-5-2015
-
Upload
pavel-chunyayev -
Category
Technology
-
view
155 -
download
0
Transcript of Ansible benelux meetup - Amsterdam 27-5-2015
![Page 1: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/1.jpg)
How Ansible helps Backbase
Ansible Benelux meetup
Pavel ChunyayevAmsterdam, 27-5-2015
![Page 2: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/2.jpg)
Who am I
• Come from Ukraine• 11 years in IT• Worked in Ukraine, Estonia and the Netherlands
• Continuous Delivery architect at Levi9 IT Services• Last 6 months - Automation architect at Backbase
![Page 3: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/3.jpg)
Backbase CXP
![Page 4: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/4.jpg)
Backbase Customer Experience Platform• Core services• Content services• Publication services• 3 environments – Editorial, Staging, Live
![Page 5: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/5.jpg)
Different configuration options
• Java version• Application Server• RDBMS• HTTP/HTTPS• Internal configuration options• Optional application features
![Page 6: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/6.jpg)
A lot of things are already automated• There are servers for released CXP version• With different configurations• They can be started/stopped when needed
• Newest version of the application needs to be deployed.• In most cases manually. • For some configurations deployment required repackaging of the application.• Automated through maven
• There is a sandbox environment with the nightly build• Deployed automatically• Far from production setup
![Page 7: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/7.jpg)
Handcrafted servers
• Hard to maintain• Very time/cost sensitive• Setup is not easily reproducible• May be buggy• It should take less time to rebuild a server from the scratch than to log
in and fix/update it.
![Page 8: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/8.jpg)
Solution
![Page 9: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/9.jpg)
Solution diagram
![Page 10: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/10.jpg)
Why Ansible
• Python powered• No master, agentless• Free, open source• Plenty of modules (batteries included)• Great EC2 support• Windows support (kind of)• Parallel, but controllable execution• Quite simple for developer to understand
![Page 11: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/11.jpg)
Why REST service
• Create infrastructure easily• Just send JSON formatted configuration• Service will analyze it and trigger Ansible run
• Service is the single point of contact for any infrastructure requests• Can be integrated into any CI, script, application or other service
![Page 12: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/12.jpg)
Why UI
• Everyone needs to create an environment from time to time• Opening a ticket and then waiting is not an option• In most situations environments are required for a short period of
time.
• Self-service
![Page 13: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/13.jpg)
Demo
• Directory structure• Flow of work• Decision tree
![Page 14: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/14.jpg)
Ansible features we are using
• Handlers• Variables• Jinja2 templates• Facts• Conditions• Playbook includes• Inventory (fake) – hostgroups!• Roles :(
![Page 15: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/15.jpg)
ec2 ec2: key_name: '{{ keypair }}' group_id: '{{ security_group }}' instance_type: '{{ instance }}' image: '{{ image }}' region: '{{ region }}' vpc_subnet_id: '{{ subnet }}' user_data: "{{ item }}" instance_profile_name: 'access-to-s3' instance_tags: origin: "{{ origin }}" environment_name: "{{ environment_name }}" stack_id: "{{ stack_id }}" owner_id: "{{ owner_id }} " role: "{{ item }}" timestamp: "{{ timestamp }}" with_items: server_roles register: ec2
![Page 16: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/16.jpg)
Facts
- name: Set the facts and hostnames hosts: all_hosts connection: ssh gather_facts: True max_fail_percentage: 0 tasks: - name: Gather EC2 facts ec2_facts: - name: Set environment fact set_fact: this_environment="{{ ansible_ec2_user_data }}" - name: Set hostnames hostname: name="{{ environment_name }}-{{ this_environment }}"
![Page 17: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/17.jpg)
route53
route53: command: create zone: backbase.dev private_zone: yes overwrite: yes record: "{{ environment_name }}-{{ item.0 }}.backbase.dev" type: A ttl: 10 value: "{{ item.1.instances[0].private_ip }}" with_together: - server_roles - ec2.results register: r53_result until: r53_result|success retries: 20 delay: "{{ 10 |random }}"
![Page 18: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/18.jpg)
Jinja2 templates
{% block portal_db %}{% endblock %}{% if http_or_https == 'http' %} {% set port = http_port %}{% else %} {% set port = https_port %}{% endif %}
{% if this_environment == "editorial" %}foundation.environment.editorial=true{% else %}foundation.environment.editorial=false{% endif %}foundation.content.proxy.destination={{ http_or_https }}://{{ environment_name }}-{{ this_environment }}.backbase.dev:{{ port }}/contentservices
![Page 19: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/19.jpg)
wait_for
- name: Start WSLC shell: /opt/IBM/Websphere/INIT.websphere start {{ this_environment }}- name: Wait for WSLC to start wait_for: path=“/opt/IBM/Websphere/usr/servers/{{ this_environment }}/logs/console.log” search_regex=“The server {{ this_environment }} is ready to run a smarter planet.” timeout=30"- name: Run the trigger shell: /opt/install/app_start_trigger.sh &> /opt/install/app_start_trigger.log; sleep 2- name: Wait for all apps to start wait_for: path="/opt/IBM/Websphere/usr/servers/{{ this_environment }}/logs/messages.log" search_regex="SRVE0242I: \[portalserver\] \[/portalserver\] \[/WEB-INF/index\.jsp\]: Initialization successful\." timeout=600
![Page 20: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/20.jpg)
Recovering from failure
- name: Download CXP shell: s3cmd get s3://s3_bucket_here/Backbase_Portal_5.6.0-{{ version }}.zip /opt/install/portal-package-5.6.0-{{ version }}.zip --force 2>&1 | tee /opt/install/direct_loader.log register: cxp_download_sleeper until: cxp_download_sleeper.stdout.find("saved as") != -1 retries: 10 delay: "{{ 10 | random }}"
![Page 21: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/21.jpg)
API
• /api/stacks - GET - List stacks available for provisioning• /api/stacks/stack_name - GET - List the stack configuration• /api/environments - GET - List all currently provisioned
environments• /api/stacks/stack_name - POST - Provision specified stack• /api/environments/environment_id - DELETE - Destroy
environment with specified id• /api/environments/all - DELETE - Destroy all environments
![Page 22: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/22.jpg)
Infrastructure life cycle
• Create• Check if the user is valid• Parse the requested configurarion• Generate unique environment name• Trigger Ansible run• Return environment name
• Destroy• Check if requested environment exists• Check if the user can destroy this environment• Delete environment and clean everything up (DNS, ELB, etc.)
![Page 23: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/23.jpg)
REST Service demo
• Create a set of instances• Destroy them
![Page 24: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/24.jpg)
Current UI :)
![Page 25: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/25.jpg)
Ansible testing
• No way to test playbook without applying it• Currently there’s a quick sanity test suite• We do testing every commit for a selected number of stacks
![Page 26: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/26.jpg)
Demo
• Stash• Feature branches• Ansible testing pipeline
![Page 27: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/27.jpg)
Results
• 14-40 minutes to provision and fully configure environments• From 1 stack to 10 stacks automated testing (~25 soon)• We continuously improve to make a robust process
![Page 28: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/28.jpg)
Continuous Deliverywithout Production
![Page 29: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/29.jpg)
Goals for Continuous Delivery
• Create a repeatable and robust process• Treat all configurations identically• Some are more important of course
• Provide feedback as soon as possible • For now – in the morning
• Provide feedback for feature branches• Release tested artifacts more frequently• For now – every iteration
![Page 30: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/30.jpg)
Stages for Continuous Delivery
• Components are built• Unit and integration tests
• Main application is build and packaged• Published to Artifactory and s3
• Testing pipeline is triggered• Environments are created• Sanity tests, API tests, Functional tests, etc. are run• Notification is sent in case of any test failure• Environments are disposed
![Page 31: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/31.jpg)
Continuous Delivery diagramBuild components
PackageProvision Stack1
API Tests E2E Tests
…Dispose
Provision Stack 2API TestsE2E Tests
…Dispose
Provision Stack 3API TestsE2E Tests
…Dispose
……………
Provision Stack 10API TestsE2E Tests
…Dispose
![Page 32: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/32.jpg)
More pipelines…
• Performance tests pipeline• Feature branches• Security tests• Earlier versions of the applications• Bugfix releases• More applications
![Page 33: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/33.jpg)
Achievements
• Huge quality improvements• Numerous bugs were found in the ‘rare’ stacks• Regressions are found during the night
• Minimum cycle time is 1h 30m, maximum – 2h 30m• Dozens environments are created every day• Repeatable process allows to identify instabilities in tests and
configurations
![Page 34: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/34.jpg)
Closing thoughts
![Page 35: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/35.jpg)
Zero downtime deployments
![Page 36: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/36.jpg)
Future
• Asynchronous provisioning• Ansible roles?• Optimization (time)• Pre-baked images• Docker containers• Plugins to help our specific needs
![Page 37: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/37.jpg)
Ansible v2
• Blocks • begin• rescue• always
• Execution strategy – linear vs free (or anything else)• Execution time include evaluation (with*)• Better variable management
![Page 38: Ansible benelux meetup - Amsterdam 27-5-2015](https://reader030.fdocuments.in/reader030/viewer/2022032506/55ca4e86bb61eb65778b45ce/html5/thumbnails/38.jpg)
Key takeaways
• Use Ansible – it’s a great tool :)• Think about immutable infrastructure• Create repeatable and reliable process
for releasing software• Build quality in• Improve continuously
[email protected]@PavelChunyayev
Any questions?