Wim Demey - Regression Testing in a Migration Project
-
Upload
eurostar-software-testing-conference -
Category
Software
-
view
61 -
download
0
Transcript of Wim Demey - Regression Testing in a Migration Project
Copyright © CTG, Inc.
Organize regression testing in companywide migration project
Challenge or nightmare?
Eurostar 2009 – Wim Demey
Copyright © CTG, Inc.
Agenda
Context Project Challenges Lessons learned Questions
Copyright © CTG, Inc.
Context
Similar project in timeframe of 2 years Customers within same sector Organisation
Business Unit A
Business Unit B
Business Unit D
ITS
Infrastructure ApplicationsDivision A
Business Unit …
Division …
Project
Copyright © CTG, Inc.
Context
Project mode -> service mode
Project mode
Initiate Define Design Build Implement Close
- Project methodology needs to be followed
- A lot of deliverables (project + specific test deliverables)+ Test strategy+ Test requirements matrix+ Test scenarios+ Test reports (intermediate – progress – final)+ Defect management
Project leader
Test coordinator
Test team
Service mode
Business Unit B
Service Delivery Manager
Business Unit …
Business Unit A
SLA based support for application(s)
Maintenance budget
Copyright © CTG, Inc.
Context
Test awareness within organisation Pool of test resources // separate team
Mainly test coordinator profiles Methodology forces projects to use pool/test team
Especially for large projects Being compliant with methodology requires a lot of time Internal/external audits
Difference in focus on testing Infrastructure <-> application development
Separate test environments
Development Validation Production
Copyright © CTG, Inc.
Context
Work station concept -> standard environment By default no admin rights Default software for every user Additional software/rights on demand and dependent of business unit
Application software
Default software
Operating System(SP’s, hotfixes)
Copyright © CTG, Inc.
Agenda
Context Project
Content Test approach
Challenges Lessons learned Questions
Copyright © CTG, Inc.
Project >> Content
Initiate
WHO HAS INITIATED PROJECT?
-> Infrastructure-> Similar project methodology as application development
WHAT IS PROJECT SCOPE?
-> Default software upgrade (e.g. Office2007, IE7, SapGui 7.2)-> Front-end project-> No upgrade of OS
STRUCTURE OF PROJECT TEAM?
-> Infrastructure architect team has lead-> Organized in tracks-> Involvement of test coordinator (build <> define phase)
WHAT IS IMPACT FOR USERS?
-> All business units (company wide)-> All workstation types (12.000 – 3.400)-> Way of migration (upgrade <> re-install)
Define Design Build Implement Close
Copyright © CTG, Inc.
Project >> Content
What is new in Office2007?
Ribbon (with tabs -> menus)
Office button
New and xml based formats-> .docx, xlsx, .accdb -> possibility to save as PDF
Copyright © CTG, Inc.
Project >> Content
What is impact of software upgrades? Huge number of Office related applications
Mainly Access applications Heavy Excel files (VBA, macro,…) New behaviour of Word <> regulations Compatability of third party software with new Office formats
New software versions as prerequisite for ongoing projects Design decisions
Out of the box installation <-> Office configured to needs of BU’s Increase security-level (VBA, macro) in Office and IE
What about applications in project mode?
Copyright © CTG, Inc.
Project >> Test approach
WHO PERFORMS THE TESTS?
-> SDM + team -> Business -> (support by internal test team)
HOW IS TESTING ORGANIZED?
-> Mini test cycle (ad hoc, on preliminary build)-> Major test cycle (3m -> 6 months) (frozen build)
WHICH TEST ENVIRONMENT IS USED?
-> Sandbox idea (=virtual pc)-> SDM is responsible for installation of application + apps/test data
WHAT IS ROLE OF TEST COORDINATOR?
-> Follow up testing by SDM’s -> Follow-up defects/issues-> Ensure tight schedules are followed-> Deliver required test documents
MAIN GOAL REGRESSION TESTING
What works today, still works tomorrow on migrated software
SDM = Service Delivery Manager
Copyright © CTG, Inc.
Project >> Test approach
Overview test environment Sandbox contains clean installation of components
CORP LCORP: XPLV90000xx
- Internet Explorer
- Outlook RepositoryInstall applications
admin
Remote Desktop Connection
Production account
Test account
ValidationProduction
Copyright © CTG, Inc.
Agenda
Context Project
Content Test approach
Challenges How to apply test approach? How to coordinate/motivate SDM organisation? How to gather/report information -> project? Does test environment fit the needs? Can we keep business out of scope?
Lessons learned Questions
Copyright © CTG, Inc.
Challenge 1: Test approach
Late involvement of test coordinatorLate involvement of test coordinator
Very rough estimations of test effortVery rough estimations of test effort
Confusion about what is in the project budgetConfusion about what is in the project budget
No official/central list of applications to testNo official/central list of applications to test
Project methodology / test deliverables not 100% fit for project
Project methodology / test deliverables not 100% fit for project
“Tour of SDMs”
-> Public - individual-> Get overview of portfolio-> Planning-> Roles + responsibilities-> Deliverables-> Support
Methodology adapted
-> Pragmatic but structure is essential-> Risk application test matrix-> Test scenario matrix-> MoSCoW principle
Clear out objections
-> Refine estimations -> Provide enough information-> Escalation to steering committee
Copyright © CTG, Inc.
Challenge 1: Test approach
Risk & Application Test Matrix
Sap GUI 7.0 IE 7.0 Word 2007 Excel 2007 PowerPoint 2007 Outlook 2007 Access 2007
0001 APPLICATION A SDM X N/A X N/A N/A N/A N/A X0002 APPLICATION B SDM X X N/A X X N/A N/A N/A
COMPONENTSID Application SDM
Impact Severity Risk exposure
Priority Overall Status
1 - Low 3 - High 3 Should Not started3 - High 2 - Medium 6 Must Preparation
Link TSM Comment
Copyright © CTG, Inc.
Challenge 2: Test coordination
No dedicated test resources available for this project
No dedicated test resources available for this project
No test cases availableNo test cases available No Office2007 knowledge to adapt application(s)
No Office2007 knowledge to adapt application(s)
Daily activities interfere with test executionDaily activities interfere with test execution
For some applications business has to perform tests
For some applications business has to perform tests
Technical support
-> Analysis of Office2007 issues-> Document solutions-> Assist developers during implementation-> Functional mailbox for business
Pragmatic follow up
-> Planning on weekly basis-> Network driven approach per SDM-> Test scenario template-> Ad hoc meetings-> Simple web based defect form
Copyright © CTG, Inc.
Test scenario matrix
Challenge 2: Test coordination
Copyright © CTG, Inc.
How detailed must info be collected/reported?How detailed must info be collected/reported?
Main focus on planning/milestonesMain focus on planning/milestones
No time to read long reportsNo time to read long reports
Which feedback do we give to SDM/business?Which feedback do we give to SDM/business?
Several documents as input for reportingSeveral documents as input for reporting
Challenge 3: Info gathering & reporting
Dashboard
-> Overview of progress (total, per SDM)-> Overview of defects (type, severity, trends)-> Small VBA code to gather automatically status update-> All information in one sight-> Basis for status meeting with project team
Test news letters
-> Only sent out if relevant & necessary-> Information about general issues & solutions-> Changes in test strategy-> Status about builds-> Keep it as short as possible
Copyright © CTG, Inc.
Challenge 3: Info gathering & reporting
Copyright © CTG, Inc.
Mismatch between theory and practiceMismatch between theory and practice
Test machines needed in production environment
Test machines needed in production environment
Teams create own test environmentTeams create own test environment
Some applications could not be tested remoteSome applications could not be tested remote
Need to travel around with test environmentNeed to travel around with test environment
Security causes a lot of problemsSecurity causes a lot of problems
Physical machines
-> Test accounts-> Separate container in Active Directory-> Availability managed by test coordinator-> Production like install of applications-> Staged by project team-> Pick up & return concept
Security level reviewed
-> Decreased to current level-> Alternative of exception policy doesn’t work-> Digital signing/increased security outscoped to separate project
Challenge 4: Test environment
Copyright © CTG, Inc.
Unknown number of applications developed by business
Unknown number of applications developed by business
Often very critical applicationsOften very critical applications
No support by ITS/helpdeskNo support by ITS/helpdesk
No experience / knowledge to solve issues
No experience / knowledge to solve issues
Problem of resources even higher than SDM teams
Problem of resources even higher than SDM teams
Not always clear what is expectedNot always clear what is expected
Challenge 5: Business
Active involvement
-> Same roadshows as for SDM-> Inventory made + prioritisation-> Critical applications are taken over by SDM-> Technical support-> Test scenario template provided-> Same defect tool + procedure-> Business representatives are SPOC
Copyright © CTG, Inc.
Agenda
Context Project
Content Test approach
Challenges Lessons learned
Some facts & figures Conclusions
Questions
Copyright © CTG, Inc.
Lessons learned >> Facts & figures
Applications 1052 ITS supported applications identified (420)
650 to test (Must, Should, Could) 402 rest (Won’t) -> not impacted by components
All applications tested in major test cycle (3 months) One team had 6 months time due to number of applications (420)
Issues 180 issues reported (both test cycles)
Mainly Excel + IE7 related (70% of all issues) Compatibility issues with third-party software (e.g. SAS/SAP) REMARK: only for ITS supported applications No need to rebuild applications from scratch
Some issues led to delayed migration of users/divisions
Copyright © CTG, Inc.
Lessons learned >> Conclusions
Copyright © CTG, Inc.
Lessons learned >> Conclusions