A Comparative Study of Mobile Internet Deployment Models ...
Deployment Testing in an Internet World
-
Upload
softwarecentral -
Category
Documents
-
view
508 -
download
1
Transcript of Deployment Testing in an Internet World
Deployment Testing in an Internet WorldDeployment Testing in an Internet World
Michael Cookson CSQA, Manager, DTTMary Wood, Senior QA Analyst, DTT
April 14, 2023
Deployment TestingDeployment Testing
AGENDA
- Brief overview of MRO Software
- The Challenge
- Specific Issues
- Creation and Role of Deployment Test Team
- Next Steps
If at any time, there are questions,
please shout them out!
Deployment TestingDeployment Testing
MRO Software, Inc.
- MRO = Maintenance, Repairs and Operations
- Office in London Ontario is the operations center for the hosted production site
- We host a series of Internet applications, allowing for the searching and purchasing of operational materials
- Includes workflow, Purchase Order integration to client distribution systems, real time pricing and availability specific to purchasing organizations, order status and integration to different ERP systems for control of inventory
Deployment TestingDeployment Testing
Mike Cookson started with MRO Software in November 1999 to become the test manager
Mary Wood joined MRO Software in June 1997, and moved through a series of positions before becoming the initial member of the test team in 1998
Initially this new test team was responsible for the functional testing of the product and was part of the development organization
Later, in 2000, the test team was moved to the operations area, but maintained its functional testing focus
This created some conflict, as we will discuss
The ChallengeThe Challenge
Deployment TestingDeployment Testing
The Internet – Client Perspective
- Internet environment demands 7x24 availability
- Clients are increasingly dependant on the availability of the system to support their business processes
- In order to be successful in this marketplace, need to be able to provide the highest availability
- Many contracts will have a Service Level Agreement (SLA) attached with penalties for not hitting targets
Deployment TestingDeployment Testing
The Internet – MRO Software Perspective
- We host applications that our clients access
- Multiple teams within MRO Software provide changes to the hosted environment
- These changes usually arrive independently of one another, but could affect other components
- Many of these changes are attached to potential revenue contracts
- The change needs to be implemented in a timely manner, with delivery commitments already made to clients
- Many clients connections can only be tested in the production environment because they do not have test back end systems and firewall issues
Deployment TestingDeployment Testing
Bottom Line:
Change is needed for revenue;
Change can cause loss of revenue!
Deployment TestingDeployment Testing
The Challenge:
Develop a system to ensure the timely implementation
of change into the hosted production environment,
while at the same time ensuring the change has
minimal negative impact in terms of availability of the
systems and client connections.
Deployment TestingDeployment Testing
What does that mean?
- Most companies perform the deployment testing tasks we are about to describe in one form or another
- The question is whether these tasks need to be formalized or more focus put on them
- This presentation will describe the role of the Deployment Test Team at MRO Software and why we have organized it this way
Deployment TestingDeployment Testing
MRO Software was organized with a traditional development organization and functional testing organization
This organization left some ‘gaps’ in terms of coverage
Organizationally, we were not ready to fill these gaps
- For example:
Who is responsible for coordinating changes being made by multiple teams?
Who is responsible for ensuring the changes are made in the change windows? – 6x24 needed by clients currently and shrinking
Who is ensuring changes integrate with the client systems? – not always possible in the development environment
Who is responsible for ensuring the dependancies for environments are taken into account?
Deployment TestingDeployment Testing
There were many other issues that we were trying to deal with
These issues were summarized into four broad categories
- Change Management
- Implementation Management
- Client Management (at a tactical level)
- Test Environment Management
These issues were beginning to have an impact on the reputation and ability of MRO Software to deliver a quality product to the hosted environment in a reasonable time frame
Deployment TestingDeployment Testing
The Test Team at MRO Software had a functional testing focus, although part of the operations group and so functional and operational issues had to be considered simultaneously at each deploy
This was different from all other parts of the organization where the test team was part of the development organization
The Internet world demanded more than just functional testing, but there was no room in the organization for anything but functional testing
The Customer’s business was directly affected by the success or failure of the code deploy
Deployment TestingDeployment Testing
A number of pressure points were converging
Technical and operational issues were mounting
Organizationally we needed to bridge the gap between functional testing and operational issues
Deployment TestingDeployment Testing
The evolution of the Deployment Testing Team was in direct response to the pressure points
Under the Operations umbrella the Functional Test Team was beginning to take ownership of some of the processes
- For example Change Management and Implementation Management
A completely different mind set and conflicting set of priorities were becoming evident
Operational testing tasks and functional testing tasks had different goals, objectives and accountabilities
Staff was having difficulties switching between these
The SolutionThe Solution
Deployment TestingDeployment Testing
The conclusion was that the functional team had to be split and the responsibilities divided
Functional testing would return to the Development organization
Deployment Test Team was created in the Operations area to assume ownership of the Operational processes
Deployment TestingDeployment Testing
The Mission of the Deployment Test Team is to manage all changes to the production hosted environment to ensure minimal negative impact to our customers.
This will be accomplished through the application of a formal, structured approach to every change
to the hosted environment in order to provide the benefits of the change in a timely manner.
The Deployment Test Team will employ:Risk Assessment Process to identify the appropriate verification and validation procedures;
Change Management Process to co-ordinate varied resources to ensure a smooth introduction into the production environment and ensure that all impacted areas have the necessary information
regarding the change;Manual and automated tools and test plans to ensure the change has been properly implemented.
The goal of the Deployment Test Team is not necessarily to apply perfect change, but rather, to
ensure that every change to the production environment is applied perfectly.
Deployment TestingDeployment Testing
Deployment testing, quite simply, is the actions
taken to ensure a deliverable or change
is properly implemented into a production environment.
The word ‘Testing’ in this case, does not adequately
cover the myriad of tasks necessary to ensure the
changes make it to production successfully.
Deployment TestingDeployment Testing
So what are the items that the DTT are responsible for?
- Implementation Management
- Change Management
- Verification of Deployments
- Ongoing Client communication and coordination
- Maintenance of a mirror test environment
Deployment TestingDeployment Testing
Implementation Planning owned by DTT
- For every change, what is needed to successfully implement?
- What dependencies are there?
- How can we test the plan?
- How can this affect the clients with connections to our software?
- The detail needed in the implementation plan is dependent on the risk of the change
- We have developed a very simple risk model to help us identify low, medium and high risk projects and based on that, develop the implementation plan
- Includes back out planning and verification of the plan
Deployment TestingDeployment Testing
Change Management Process owned by DTT
- What changes are expected?
- What is the time frame for these changes?
- What else is planned for that change window?
- Are all the impacted areas aware of this change?
- Do clients need to be notified – who and when?
- Ensures all change is identified before the date of implementation
- If multiple changes needed the same window, ensures the different teams are involved for cross dependencies
Deployment TestingDeployment Testing
Verification of the change in the production environment is owned by the DTT
- Once a change is implemented, how do we know the change was applied correctly?
- What are the indicators to prove all the changes arrived in production correctly?
- What regression tests should be done to ensure client connectivity and critical functions have not been impacted after the change?
- Deploying the code is only half the battle
- Need to ensure all clients are connecting normally, and the application is also functioning
- If things aren’t working, then need to invoke the back out plan and then verify the back out worked
Deployment TestingDeployment Testing
Ongoing Client Communication and Coordination
- What impact does this change have on the client?
- Is there a risk of disrupting their connections to the software?
- How can we verify these changes?
- What change window do we have based on the client’s business needs?
- Involves continual communication with our clients and with the teams that are working with the clients to ensure their needs are understood
- The operations group owns the relationship with the client at a tactical level, the DTT is one tool used to maintain and enhance this relationship
Deployment TestingDeployment Testing
Maintenance of a Mirror test environment
- To ensure deployments happen appropriately, need to have a practice site
- This mirror site needs to be as close to the live environment as possible
- It is not a place to perform functional testing – by the time the code is applied here, it is functionally correct
- Used to test deploys, backouts, connect to client back ends and perform transactions
Deployment TestingDeployment Testing
What is the DTT not responsible for?
- The functional testing of the change
- DTT assumes the change has been tested and is ready to be put into the production environment
- The actual deployment of the change – done by another operations team
- Work closely with the team performing the deployment to ensure their needs are included in the planning
- The Deployment Test Team is not doing any work that another team is responsible for. We do not redo or recheck previously done work.
Deployment TestingDeployment Testing
Current Status
- Change Process has been defined and implemented
- Risk Assessment has been created and is used for each change
- Checklist of key tests for each client has been created and is used to check each change, based on the change and the risk
- Automation begun on these key tests
- Mirror environment in place for key applications and plans in place to create mirror environments for the others
- Skills inventory completed, and self assessments done.
Deployment TestingDeployment Testing
The splitting of the test team allowed the DTT to focus on the operational issues
Due to the way MRO Software splits the development and operational responsibilities, needed to ensure the operations tasks had the focus required
Development needs to focus on the creation and enhancement of new products
Operations needs to focus on the client relationship on a day to day basis and the availability of the hosted production applications
The DTT team can focus on these tasks without the issues of functional testing and project deadlines
Deployment TestingDeployment Testing
Key Success Factors for the DTT
- Marketing – make sure all areas aware of the role of the DTT
- Ensure all change is properly planned for
- All change implemented within the change window
Need to perform all the tasks necessary, but no more – implies a very solid risk assessment process
Need to automate as much of the verification as possible to save time
- External clients are not negatively impacted by change
All connections available when change is done
We minimize impact to world wide clients for down time due to change
Ensure we will not need to perform an emergency back out of upgrades because something was missed in the deployment
What About You?What About You?
Deployment TestingDeployment Testing
Does your organization need focus on these responsibilities?
If, around the water cooler, you hear these questions, it is possible you want to put some focus on these tasks
- The conversion is still going? That’s over 70 hours!
- Why didn’t the change go in? This project has been in the works for months now!
- It worked in development.
- Why is production running a different version of the database? Of course it won’t work
Deployment TestingDeployment Testing
- Why wasn’t our change tested with the other change? Everything has to go at the same time for this to work.
- Why are these two changes scheduled for the same night? They have to be implemented separately.
- This is an emergency. How do we get our change in there now?!?!
- Our change went in and is working. Of course, it broke everything else, but that is not my problem.
Deployment TestingDeployment Testing
If you are hearing conversations like this, then it is probable that some focus needs to be put on the tasks we have discussed
How you implement will depend on your organizational structure and how responsibilities are divided
The creation of the DTT is working at MRO. At your place, you may need a different structure to work, or maybe, the DTT will work for you as well…
Next Steps @ MRONext Steps @ MRO
Deployment TestingDeployment Testing
Next steps for the DTT
- Need to implement a tool to better track Changes
- Finish the Mirror Environment for all applications
- Continue to build the automated test suite
- Enhance and refine the checklist for verification procedures
- Close the gap between the identified skills needed to be successful and our current skill set and level
- Continue to build the visibility of the DTT at MRO Software
Deployment TestingDeployment Testing
Next Steps for the DTT
- Identify the measures for the Deployment Test Team so we can:
Continually improve the Change Process
Continually improve Implementation Process
Continually improve the Risk Assessment Process
Continually improve our client relationships
Continually improve the effectiveness of our test environments
Deployment TestingDeployment Testing