Tools for Managing QualityHow Microsoft’s Emerging Tool Set
Stacks Up Against Quality Center
Software Quality ForumThursday, November 11, 2010
5:00 – 7:00 pm
BenchmarkQA helps project teams deliver higher quality software through:
Quality Assurance Consulting
Contract & Permanent Staffing
QA/Test Training & Seminars
Local Outsourcing Test Lab
© 2010 BenchmarkQA
Agenda
• Deciphering the MS Marketing• Comparisons
1. Project/requirements management2. Test case management3. Test execution4. Defect Management5. Reporting6. Administration7. Automated Testing
• Conclusions
11/15/2010 © 2010 BenchmarkQA Slide 3
Overview of MS Tools
• 4 primary components
• This is primarily a developer view of the world
11/15/2010 © 2010 BenchmarkQA Slide 4
Reporting Services
Deciphering the MS Marketing
• TFS– Database (think Access application)
• Visual Studio– Developer tool (unit testing)– Automated testing (Coded UI)
• MTM– Microsoft Test Manager
• Reporting– Reporting Services
11/15/2010 © 2010 BenchmarkQA Slide 5
Team Foundation Server (TFS)
• Database repository
• Views from Visual Studio, Reporting Services, IE and MTM
• Query tools
11/15/2010 © 2010 BenchmarkQA Slide 6
Visual Studio
• Developer tool
• Unit testing
• Coded UI testing
• Source code management
11/15/2010 © 2010 BenchmarkQA Slide 7
Microsoft Test Manager (MTM)
• Test case management
• Test execution
• Integrated defect management
11/15/2010 © 2010 BenchmarkQA Slide 8
Reporting
• Reporting Services
• Primarily warehouse based
• Excel
11/15/2010 © 2010 BenchmarkQA Slide 9
© 2010 BenchmarkQA
Comparison 1PROJECT/ REQUIREMENTS MANAGEMENT
TFS Project/Requirements Management
• User Stories (Validation (UAT) tests linked here)
• Tasks (Verification tests linked here)
11/15/2010 © 2010 BenchmarkQA Slide 11
TFS Requirements
• User Stories– High level– More detailed user stories linked to higher levels
• States– Active/Resolved/Closed
• Area Path– Modules or groupings of application functionality– Can be used to sort of create a tree structure of user stories in a report
• Iteration– Can be used to manage releases
11/15/2010 © 2010 BenchmarkQA Slide 12
Task Form
11/15/2010 © 2010 BenchmarkQA Slide 13
Tasks
• Tasks (linked to user stories)– Assigned to person– Estimate of effort– Actual hours– Activity
• Design (Verification tests linked here)• Development• Testing
• Task States– Active– Closed
11/15/2010 © 2010 BenchmarkQA Slide 14
Requirement-Design Links
• Use task activities to deal with design elements
11/15/2010 © 2010 BenchmarkQA Slide 15
Scoring - Project Management
• Scheduling – TFS
• Resource management – TFS
• QC– Can be added to requirements section– Workflow would be challenging– Additional reporting needed (burn down charts, etc.)
11/15/2010 © 2010 BenchmarkQA Slide 16
Scoring – Requirement Management
• Enter requirements – Wash
• Organize requirements – QC
• Link requirements to test cases – Wash
• Link requirements to defects – Wash
11/15/2010 © 2010 BenchmarkQA Slide 17
Comparison 2TEST CASE MANAGEMENT
© 2010 BenchmarkQA
TFS Manual Test Cases
• Similar to QC– Action/step– Expected results
• Parameterized test fields– Defined at test level rather than run-time
• Can import from Excel– Test case importer (CodePlex)– Needs customization
• Organization similar to Test Suite in QC
11/15/2010 © 2010 BenchmarkQA Slide 19
MTM Test Case Management
11/15/2010 © 2010 BenchmarkQA Slide 20
Test Case Entry• Test case entry very similar to QC
• Action• Result• Parameters• Shared steps (call another test in QC)
11/15/2010 Slide 21© 2010 BenchmarkQA
Scoring - Test Case Management
• Enter test cases – QC
• Organize test cases – QC
• Import from Excel – QC
• Link test cases to requirements – Wash
• Link test cases to defects – Wash
• Integration of automated tests – QC
• Unit tests - VS
11/15/2010 © 2010 BenchmarkQA Slide 22
Comparison 3TEST EXECUTION
© 2010 BenchmarkQA
MTM Test Execution Management
• Note configurations
• Progress bar
11/15/2010 © 2010 BenchmarkQA Slide 24
TFS Test Execution
• Very similar to QC
• Filter by environment
• Cannot stop and restart test – Recorded as fail.
• Rearranges your desktop
• History not linked to test case
• List of tests to run
11/15/2010 © 2010 BenchmarkQA Slide 25
Scoring - Test Execution (Manual)
• Organize test runs – Wash
• Execution speed – Wash
• Restart test – QC
• Environment support – MTM
• Execution history - QC
11/15/2010 © 2010 BenchmarkQA Slide 26
Comparison 4DEFECT MANAGEMENT
© 2010 BenchmarkQA
TFS Defect Management
• Remember this is a developer view of the world– Active/Resolved/Closed– Changing workflow is not as easy as QC
• Option to verify defect– Major issue-Defect verification is recorded in test suite defect
was found rather than test suite it was fixed in.– Only reruns if defect created from test run
11/15/2010 © 2010 BenchmarkQA Slide 28
Scoring - Defect Management
• Defect entry – Wash
• Creating defect while running test – Wash
• Managing defects – QC
• Verifying defects - Wash
11/15/2010 © 2010 BenchmarkQA Slide 29
Comparison 5REPORTING
© 2010 BenchmarkQA
TFS Reporting
• Uses a warehouse so data is old
• Reporting services– Much higher skill level and permissions required to create
reports– Good for roll-up/summary reporting– Difficult to nearly impossible to generate accurate requirements
coverage report
11/15/2010 © 2010 BenchmarkQA Slide 31
Scoring - Reporting
• No contest - QC
11/15/2010 © 2010 BenchmarkQA Slide 32
Comparison 6ADMINISTRATION
© 2010 BenchmarkQA
Scoring - Administration
• Project customization – QC
• User administration – Wash
• Defect workflow – QC
• Other workflow – Not easy in either
11/15/2010 © 2010 BenchmarkQA Slide 34
Comparison 7AUTOMATED TESTING
© 2010 BenchmarkQA
Unit Testing
• Integrated part of VS
• Easy for developer to create tests for methods that return values
• MOQ for visual unit test
• Code coverage
11/15/2010 © 2010 BenchmarkQA Slide 36
Coded UI Testing
• Capture/replay
• Manually create object repository
• Higher skill level than QTP to customize tests
• Much more flexible than QTP
• No waiting for support
11/15/2010 © 2010 BenchmarkQA Slide 37
Case Study 1
• QTP 9 – Crashed QTP on login screen
• QTP 10 – QTP hung on login screen
• QTP 11 – QTP recorded, could not replay login screen (some customization would get past this issue)
• VS 2010 – Records and plays back for simple actions. Customization for dealing with grid actions
11/15/2010 © 2010 BenchmarkQA Slide 38
Case Study 2
• QTP 10– Cannot find any objects on screen, not even the menus
• QTP 11 – Can find most standard controls– Cannot see any 3D objects
• VS2010– Sees all objects– View object properties (center, radius, etc.)
11/15/2010 © 2010 BenchmarkQA Slide 39
Scoring - Automated Testing
• No contest – Visual Studio
11/15/2010 © 2010 BenchmarkQA Slide 40
Summary
• Project management – TFS• Requirements – QC (by a hair)• Manual test case management – QC• Manual test execution – QC (by a hair)• Defects – Toss up• Reporting – QC• Administration – QC• Automated testing – VS2010
11/15/2010 © 2010 BenchmarkQA Slide 41
Considerations
• Cost– Initial (Project, QC, VS, QTP)– Maintenance– Conversion– Customization– Productivity improvements
• Skill level of testing staff
• Best of Breed vs. One Solution
• If starting from scratch (i.e. want to get off Excel)
11/15/2010 © 2010 BenchmarkQA Slide 42
And the winner is?
• Microsoft’s Team Foundation Server/Visual Studio 2010
11/15/2010 © 2010 BenchmarkQA Slide 43
How can we help?
• Project engagements – Tool selection and implementation– As needed support
• Staffing– Senior automation resources
• Quality Path™ – Tools and Technology– Broad and deep assessment– Process improvement program
11/15/2010 Slide 44© 2010 BenchmarkQA
Thank You!For additional information on how BenchmarkQA can be of
assistance, please contact:
Warren McLeod952.392.2400
© 2010 BenchmarkQA
Top Related