Leveraging Cloud Transformation to Build a DevOps Culture | AWS Public Sector Summit 2016
DevOps - CUNA Councils · DevOps Tools. Build Server. DEV Env. QA Env. UI Automated Testing [Azure]...
Transcript of DevOps - CUNA Councils · DevOps Tools. Build Server. DEV Env. QA Env. UI Automated Testing [Azure]...
Presenters
John TomanSenior Manager, Software
John ClarkSenior Solutions Architect
How Does My Credit Union …• Improve speed to market for my integrated
software, with lower risk?• Make software deployment less expensive, with
less complexity?• Foster better collaboration between IT and my
lines of business?• Respond quicker to feedback from key
stakeholders and customers?
DevOps• Is a culture and a practice• Emphasizes collaboration and
communication– Development– Information Technology– Others
• Automates software delivery and infrastructure changes
Culture• Remove boundaries between
development, QA, IT• Collaborate on every change• Provide continuous feedback
Ingredients• Agile methodologies• Continuous Integration (CI)• Continuous, automated Testing
(CT)• Continuous Delivery (CD)
– Package– Release– Provision– Configure
Automation
Agile’s Role• Keep iterations short
– Small changes– Make it work, make it better
• Limit work in progress– Prioritize
• Respond quickly to feedback
Continuous Integration (CI)• Integrate early, integrate often• Compile and test code on every committed
change• Short feedback loop• Prioritize build stability• Promotes higher quality
Continuous Testing (CT)
• Execute automated tests as part of software delivery pipeline– Build Verification Testing (BVT)
• Exposes application risks quickly
• Utilizes fewer testing resources
Continuous Delivery (CD)• Allows for incremental upgrades into production• Builds on CI and CT• Lots of automation tooling support
– From source control to production• Not the same as Continuous Deployment
– Continuous Delivery may not result in release• DevOps is broader in scope (cultural)
Enabling Technologies• Source Repository• Build / Orchestration Server(s)• Virtual Machine or Container Infrastructure• Test Automation Tooling• Configuration Management Tooling• Deployment Tooling• Cloud Computing
DevOps @ Symitar• Continuous Integration adoption began 2010• Continuous Delivery began 2016• Early focal points:
– Selecting automation tooling– Automating internal deployment– Provisioning
• Systems• Episys (SymXchange)• Databases (SYMs)
– Restoration• Targeting internal production systems (QA, Dev)
Problems We’re Trying to Solve• Manual system configuration
– Deployment– Configuration– Restoration
• Limited self service– Helpdesk ticket required for most issues
• Load Episys service pack or release• Load AIX patch• Restore SYM database• Enabling Episys modules (SymXchange, etc.)• Reload system
Problems We’re Trying to Solve• Longer term
– Long release cycles– Speed to market– Higher efficiency
Tools used for Automation / CD
DevOpsTools
Build Server
DEV Env
QA EnvUI
Automated Testing [Azure]
Build Verification
Test Env
DevOps ToolsOrchestrationConfiguration ManagementSystem Deployment
CD / Automation Tool Selection• Orchestration
– Jenkins
• Configuration Management– Chef
• System Deployment– IBM Network Installation Manager (NIM)
CD / Automation Tools OverviewDeploy
Base image
Post Deploy Configuration
Configuration changes
Auditing
Automated Testing
Chef
Jenkins
NIM
CD / Automation Tools
Orchestration tool
• Initiate jobs• Self service interface• Remote command execution
via SSH• Role based access to GUI
interface• Public Key or password
authentication to end nodes• Server: RHEL
Configuration Management
• Automation tasks:• Application updates• Post deployment
configuration• System auditing• Initiated from Jenkins or
client• GUI Interface• AIX/Windows Client• Initiated via client or via Knife
command line tool• Server: RHEL
System Deployment
• IBM Network Installation Manager
• AIX/Episys image deployment via network
• System restore• System image repository• Initiated from Jenkins• Server: AIX LPAR
Chef Configuration Management• Codification of infrastructure• Provisioning
– Automate system deployment/restoration– Cloud management (Azure, Google Cloud Platform, etc.)
• Ongoing Management– System auditing
• Open Source Cookbooks– Benefit from development community
CD / Automation Solution
AIX 7.1Episys 2016.00
Final
Chef
Jenkins
NIM
SSH keys
Episys Modules
AIX patch
Episys SP
LPAR Orchestration
Post Deployment Configuration
ConfigurationChanges
KSH scripts
expect TCLNetworking
Transaction Test Tools
Add Users
Configuration files
Auditing
SSH keys
Passwords
Automated Testing
Security Patches
System Deployment Process Flow
Create LPAR
• Jenkins execute commands on HMC Server
• Jenkins executes commands on NIM Server to install image
• Trigger job to bootstrap chef client with initial cookbooks
Chef initial run
• Install chef-client• Setup
configurations• Configure
mounts• Enable/Disable
services• Modify packages• Install Episys
Cookbook
• Test Media• Update Episys• Update AIX
Service level• System Backup
Maintenance
• Jenkins job monitoring Utils
• Chef client daemon execution reports unstable nodes
• Chef-client audit mode
• Infrastructure monitoring
Automated Testing Process Flow
• AIX 7.1• Episys 2016.00
LPAR Orchestration
• Episys SP• AIX Patch• Episys Modules• Transaction
testing tools
Post Deployment
Configuration• Users, Network• Permissions• Mounts, Services• Passwords• SSH Keys
Configuration Changes
• KSH Scripts• expect TCL• Coded UI
Automated Testing
Continuous Delivery Solution
Build Server Baseline
2016.01
2016.00
Baseline
2016.01
2016.00BVT
BVT
BVT
Build Verification Test LPARs
2016.00
2016.01
Baseline
Dev QA
Chef
Integration Integration
BVT
Integration
QuestQuest
QuestQuest
UI Automated Testing(Azure)
BVT
BVT
BVT
BVT
Jenkins
QuestQuest
QuestQuest
Symitar Success Story: Release Testing• Before
– Manual refresh of systems between tests– Request to system admins
• Dependent on system admin schedule• Average turnaround time: 4 hours
– Limits on test cases executed due to time constraints• Now
– Self service tool for QA• Completed in 15 minutes• Removed dependency on system admin schedule
– Allowed for 5X release load test cases
Symitar Success Story: Failover Testing• Before
– Manual system setup – very error prone/labor intensive• Now
– Self-service automated system deployments– QA staff can build their own environments using GUI self
service tool• Fully scripted and automated backend• Estimate 8 hours labor saved per environment build (160 hours per
release)
Symitar Success Story: AIX Patching• Before
– All system setup and test cases done manually– AIX patch testing required 3 people / 2 weeks
• Now– Self service interface for QA
• Automation of system setup/restore• Automation of test cases
– Rapid re-run of tests following defect fix– AIX patch testing completed by 1 person
• Included re-run of tests following defect fix
Symitar Success Story: Self Service• Before
– All system deployment performed manually– Request to system admins
• Now– Self service interface– Full automation of system deployment
• Reduced turnaround time• Improved end-user experience
Scenario: Deploy an Episys Patch• Current Process
– Software patches deployed manually– Request to system admin team
• Goals– Fully automated software patch deployment– End-user self service option– No need for system admin intervention– Foundation for automated build testing
Scenario: Build a Test System• Current Process
– Test systems deployed manually– Limited build verification testing
• Critical test cases executed late in development cycle
• Goals– Fully automated test system deployment– Fully automate build verification testing– Execute critical test cases throughout development cycle
Scenario: Integrate a New Feature • Current Process
– New feature integration requires specialized staff and procedures• Slow and lengthy process• SME’s spending time on system setup, not testing
– End result: limited internal deployment and testing• Goals
– Provide self service tool• Fully automated backend
– Simple and rapid deployment– More extensive internal adoption and testing
Takeaway #2• DevOps is culture and practice
– Cultural change needs burn in time– Practice requires tooling
Takeaway #3• Automation is the key driver of
– Efficiency– Quality– Time to market
• Automate all the way to production
Takeaway #5• Utilize feedback loops
– Involve stakeholders, integrators, developers, testers, customers