© Testhouse Ltd. Confidential and Proprietary
DEVOPS MATURITY
© Testhouse Ltd. Confidential and Proprietary
About Testhouse
DevOps a brief overview
Challenges with DevOps
Maturity Assessment
Case Studies
o A software house, specialising in travel, implementing DevOps
o A Large entertainment company mature in DevOps
o A medium office space provider starting DevOps
o A Large Bank planning DevOps
o A Large Financial Services mature in DevOps
AGENDA
KEY FACTS
© Testhouse Ltd. Confidential and Proprietary
Founded in 2000
Headquartered in London
Full Service Centres(UK, UAE, India)
Partnerships with Vendors(Dynamics CRM & ALM)
Quality Partner of many
Leading Organisations
G-Cloud 9 Supplier
RELEVANT EXPERIENCE
VOLKSWAGEN
FINANCIAL SERVICES
Deployments, migrations & upgrades
Release Management
Team Foundation
Server & VSTS
Onsite, onshore and offshore Flexible blended delivery
Managed Services
Compatibility/Mobility SIT/UAT Pre-built scripts
Functional
Testing
Web & Load performance testing
Application Insights deployment
Performance Testing
& APM
Framework design Automation services
Test
Automation
ALM and DevOps assessments
Continuous integration & deployment
Agile coaching
Agile, ALM & DevOps
Consultancy
Services
@testhouseuk
02
03
01
Bringing the three together to make DevOps4Dynamics
VSTS and DevOps
01
Dynamics 365
02Azure
03
The DevOps methodology and VSTS
The on time provisioning of environments for Dev, Test, Performance Test and Production
Dynamics 365, Dynamics CRM & AX
TESTHOUSE FOCUS
© Testhouse Ltd. Confidential and Proprietary© Testhouse Ltd. Confidential and Proprietary
Waterfall Iterative Agile
Disciplined Agile
DevOps
THE EVOLUTION OF DELIVERY PRACTICES
DEVOPS IS ALL ABOUT EXECUTING WITH SPEED!
8
Line-of-business Customer
• So you can continuously improve• application delivery• environment provisioning• application and environment
delivery process
• Getting ideas into production quickly, in small batches• Getting people to use it• Getting their feedback
Continuous Delivery Continuous Feedback
Continuous Innovation
Adapted culture
• KPIs focusing on end-results: the client satisfaction• Encouraging innovation and collaboration• Supportive of failure to accelerate learning
Dev SCM Build Package Deploy Test Prod
© Testhouse Ltd. Confidential and Proprietary
THREE WAYS OF DEVOPSSYSTEMS THINKING
Line-of-business Customer
CI / CD
Dev SCM Build Package Deliver Test Deploy
Getting ideas into production quickly, in small batches, Automating the process
© Testhouse Ltd. Confidential and Proprietary
THREE WAYS OF DEVOPSFEEDBACK LOOPS
Line-of-business Customer
Continuous Innovation
Dev SCM Build Package Deliver Test Deploy
• Getting ideas into production quickly, in small batches• Getting people to use it• Getting their feedback
• Measure and Monitor• Are we building the right product?• Understanding and responding to all customers• Introduce faults to the system
© Testhouse Ltd. Confidential and Proprietary
THREE WAYS OF DEVOPSCULTURE OF EXPERIMENTATION
Line-of-business Customer
Continuous Innovation
Dev SCM Build Package Deliver Test Deploy
• Getting ideas into production quickly, in small batches• Getting people to use it• Getting their feedback •Experiment and reduce MTTR
• Experiment and introduce failure practice makes perfect• Allocate time to experiment• Allow people to take risks• Introduce faults to the system
T H E M I C R O S O F T D E V O P S S T O R Y
DEVOPS BENEFITS
STRONG IT PERFORMANCE IS
A COMPETITIVE ADVANTAGE
DEVOPS PRACTICES
IMPROVE IT PERFORMANCE
DEPLOY CODE 30X FASTER
HAVE 60X FEWER FAILURES
TOP 15 CHALLENGES
© Testhouse Ltd. Confidential and Proprietary
Flexible environment provisioning required by DevOps teams (67%)
There is no coordination of the delivery environment elements (58%)
Governance when applying DevOps (50%) Deployments are largely manual (and error prone) (50%)
There are no or limited DevOps-specific metrics in place (50%)
There is no integrated tools architecture in place (42%)
Insufficient and inconsistent test automation (42%) Test data management (42%)
Testing is not performed early using capabilities such as service virtualisation (33%)
No DevOps vocabulary is in place (33%)
No DevOps vision or strategy defined (33%) Missing or inconsistent mechanisms for getting feedback (33%)
Applying DevOps in a regulated environment (33%) There is a lack of collaboration (and transparency) between business and IT (33%)
There is a lack of collaboration between Dev and Ops (in both directions) (33%)
APPROACH TAKEN – 6 AREAS
© Testhouse Ltd. Confidential and Proprietary
DEV
OP
S
APPROACH TAKEN – 3RA STAGES
Reactionary
Reactionary to the skill, ad hoc behaviours and success through luck
Repeatable
Performs training and regular practice to increase success rate
Reliable
Consistency with success most of the time but still room to improve
Aspirational
Ideal state where success is a given due to the level of reliability
DEV
OP
S
DEVOPS MATURITY
Backlog
Evidence
Flow
Production
Schedule & Team
Technical Debt
Target vs Actual
Actual Target
A SOFTWARE HOUSE, SPECIALISING IN TRAVEL, IMPLEMENTING DEVOPS
© Testhouse Ltd. Confidential and Proprietary
Background A software house providing software solutions to the tourism and travel industry. Currently going through a DevOps implementation.
Challenges Implemented automated build and deploy using in house built solution. The key individual who built this solution is now left. Currently no knowledge of how this was done. We are currently assisting them with the following
• Re-engineering the automated build and deploy / re –develop the automated build and deploy using TFS
• Testing team waiting for the completed build• Building automated regression testing • Do not promote the build but rebuild each time• No automated approval process• All environments on physical machine
Current state In terms of maturity they are early into the DevOps adoption cycle
A LARGE ENTERTAINMENT COMPANY MATURE IN DEVOPS
Background A large entertainment and excitement company who has implemented DevOps and is very mature.
Challenges As part of their continuous improvement programme asked us to review their DevOps processes and make recommendations
• Very mature and the organisation feels that it has benefitted tremendously after the adoption
• They are not using TFS but tools like Jira and Gitlab therefore not a fully integrated solution, now migrating to TFS
• No Portfolio backlog ( portfolio -> program -> team lead ) hence no ability to view at high level
Current state In terms of maturity they are classified as very mature
© Testhouse Ltd. Confidential and Proprietary
A REAL ESTATE INVESTMENT TRUST
Background The company owns and provides office space facilities to businesses.
Challenges Interested in Implementing currently lacks the following• No automated build• No automated testing (functional or performance)• Unit testing for JavaScript but not for .Net code
Current state Currently implementing TFS to provide automated build and deployment and starting Test Automation. Customer is however keen to implement and takes advice seriously.
© Testhouse Ltd. Confidential and Proprietary
A LARGE INVESTMENT BANK
Background A large investment bank who are currently releasing four times a year wants to increase the number of releases to eight times a year.
Challenges Increase speed of delivery with increased quality. Mixture of legacy systems, client server systems and web based applications.
• Develop the automated build and deploy using TFS• Building automated regression testing • Implement automated release management• Use of cloud for environment provision with Docker
Combination of the above can achieve their goal by Oct 2018, with the next step to increase the number of releases to once a month, which is an acceptable standard.
Current state In terms of maturity they are early into the DevOps adoption cycle
© Testhouse Ltd. Confidential and Proprietary
A LARGE FINANCIAL SERVICES COMPANY
Background Traditionally following waterfall methodology, following the appointment of a CIO, implemented DevOps using opensource technology.
Challenges The change was implemented by stealth and quickly. As a consequence face significant issues. In terms of maturity they are early into the DevOps adoption cycle
Current state
© Testhouse Ltd. Confidential and Proprietary
CURRENT STATE - ADVANTAGES
DevOps has been implemented and is reasonably active
Team is receptive to change
Management and team are well aligned
Technically competent team
Collaborative management
© Testhouse Ltd. Confidential and Proprietary
© Testhouse Ltd. Confidential and Proprietary© Testhouse Ltd. Confidential and Proprietary
Application
2 weeks train
TestDevPre
ProdUAT Prod
ISSUES PRIORITISED
© Testhouse Ltd. Confidential and Proprietary
Issue Importance Implementation Priority
Group
Requirements capture volatile High High I
No requirements traceability Medium Medium II
No risk based approach to testing Medium Medium II
Unit tests not automatically generated Low Low III
Scope of unit testing needs to be widened and quality improved High Medium II
Currently no non functional testing implemented Medium Low III
No code scan for security Medium Medium II
No integration testing with API calls carried out Low Low III
No review or testing of SQL statements High Medium II
issues with Fitnesse Fixtures Low Low III
Reference/Model Test Data Set not available Medium High II
The scope of UAT tests are currently too high High High I
Training for staff on nimble toolset is not adequate High High I
Test strategy and guidance High High I
Weak Unit Test Coverage for JavaScript Medium Medium II
Team Communication High High I
FOCUS AREAS- PRIORITIZED
© Testhouse Ltd. Confidential and Proprietary
Importance –
categories rated H, M, L based on the impact it will have on the project.
ASSESSMENT SOLUTIONING IMPLEMENTATION INDUSTRIALISATION
• Test Strategy
• Requirements Volatility• Team Communications
• DevOps Training
• High UAT Scope
•Auto generate unit tests
• Integration API level tests
•Non functional testing
• Review current toolset against
commercial tools ( Observation )
• Research into use of SV ( Observation )
• Environments ( Observation )‐ Adopt Build Process, Build & Validate‐ Refine for Virtual v. Physical
•Model Test Data‐ Adopt Creation Process, Create &
Validate
• Capture & Publish Delivery & ROI
Metrics
Ongoing…
• Continuous Process Review and
Improvement
• Implement items from the research
and review from last stage.
Evaluation
& Scoping
August 2017*
Define Efficient
Test Platform
October 2017*Transformation
December 2017*Continuous
Improvement
March 2018*
*Dates are indicative and will be reviewed after each phase
• Model Test Data
• Data Requirements
• Manipulation
• Reuse
• Tooling
• Requirements Traceability
• Risk Based Testing
• Unit Tests• Improve quality of Junit• Setup Jscript unit tests• Setup SQL unit tests
• Security Code Scan
ROADMAP TO ADDRESS KEY CHALLENGES
Top Related