Tools for Managing Quality (Nov 2010 BenchmarkQA Software Quality Forum)

Tools for Managing QualityHow Microsoft’s Emerging Tool Set

Stacks Up Against Quality Center

Software Quality ForumThursday, November 11, 2010

5:00 – 7:00 pm

BenchmarkQA helps project teams deliver higher quality software through:

Quality Assurance Consulting

Contract & Permanent Staffing

QA/Test Training & Seminars

Local Outsourcing Test Lab

© 2010 BenchmarkQA

Agenda

• Deciphering the MS Marketing• Comparisons

1. Project/requirements management2. Test case management3. Test execution4. Defect Management5. Reporting6. Administration7. Automated Testing

• Conclusions

11/15/2010 © 2010 BenchmarkQA Slide 3

Overview of MS Tools

• 4 primary components

• This is primarily a developer view of the world


Reporting Services

Deciphering the MS Marketing

• TFS– Database (think Access application)

• Visual Studio– Developer tool (unit testing)– Automated testing (Coded UI)

• MTM– Microsoft Test Manager

• Reporting– Reporting Services


Team Foundation Server (TFS)

• Database repository

• Views from Visual Studio, Reporting Services, IE and MTM

• Query tools


Visual Studio

• Developer tool

• Unit testing

• Coded UI testing

• Source code management


Microsoft Test Manager (MTM)

• Test case management

• Test execution

• Integrated defect management


Reporting

• Reporting Services

• Primarily warehouse based

• Excel


© 2010 BenchmarkQA

Comparison 1PROJECT/ REQUIREMENTS MANAGEMENT

TFS Project/Requirements Management

• User Stories (Validation (UAT) tests linked here)

• Tasks (Verification tests linked here)


TFS Requirements

• User Stories– High level– More detailed user stories linked to higher levels

• States– Active/Resolved/Closed

• Area Path– Modules or groupings of application functionality– Can be used to sort of create a tree structure of user stories in a report

• Iteration– Can be used to manage releases


Task Form


Tasks

• Tasks (linked to user stories)– Assigned to person– Estimate of effort– Actual hours– Activity

• Design (Verification tests linked here)• Development• Testing

• Task States– Active– Closed


Requirement-Design Links

• Use task activities to deal with design elements


Scoring - Project Management

• Scheduling – TFS

• Resource management – TFS

• QC– Can be added to requirements section– Workflow would be challenging– Additional reporting needed (burn down charts, etc.)


Scoring – Requirement Management

• Enter requirements – Wash

• Organize requirements – QC

• Link requirements to test cases – Wash

• Link requirements to defects – Wash


Comparison 2TEST CASE MANAGEMENT

© 2010 BenchmarkQA

TFS Manual Test Cases

• Similar to QC– Action/step– Expected results

• Parameterized test fields– Defined at test level rather than run-time

• Can import from Excel– Test case importer (CodePlex)– Needs customization

• Organization similar to Test Suite in QC


MTM Test Case Management


Test Case Entry• Test case entry very similar to QC

• Action• Result• Parameters• Shared steps (call another test in QC)

11/15/2010 Slide 21© 2010 BenchmarkQA

Scoring - Test Case Management

• Enter test cases – QC

• Organize test cases – QC

• Import from Excel – QC

• Link test cases to requirements – Wash

• Link test cases to defects – Wash

• Integration of automated tests – QC

• Unit tests - VS


Comparison 3TEST EXECUTION

© 2010 BenchmarkQA

MTM Test Execution Management

• Note configurations

• Progress bar


TFS Test Execution

• Very similar to QC

• Filter by environment

• Cannot stop and restart test – Recorded as fail.

• Rearranges your desktop

• History not linked to test case

• List of tests to run


Scoring - Test Execution (Manual)

• Organize test runs – Wash

• Execution speed – Wash

• Restart test – QC

• Environment support – MTM

• Execution history - QC


Comparison 4DEFECT MANAGEMENT

© 2010 BenchmarkQA

TFS Defect Management

• Remember this is a developer view of the world– Active/Resolved/Closed– Changing workflow is not as easy as QC

• Option to verify defect– Major issue-Defect verification is recorded in test suite defect

was found rather than test suite it was fixed in.– Only reruns if defect created from test run


Scoring - Defect Management

• Defect entry – Wash

• Creating defect while running test – Wash

• Managing defects – QC

• Verifying defects - Wash


Comparison 5REPORTING

© 2010 BenchmarkQA

TFS Reporting

• Uses a warehouse so data is old

• Reporting services– Much higher skill level and permissions required to create

reports– Good for roll-up/summary reporting– Difficult to nearly impossible to generate accurate requirements

coverage report


Scoring - Reporting

• No contest - QC


Scoring - Administration

• Project customization – QC

• User administration – Wash

• Defect workflow – QC

• Other workflow – Not easy in either


Unit Testing

• Integrated part of VS

• Easy for developer to create tests for methods that return values

• MOQ for visual unit test

• Code coverage


Coded UI Testing

• Capture/replay

• Manually create object repository

• Higher skill level than QTP to customize tests

• Much more flexible than QTP

• No waiting for support


Case Study 1

• QTP 9 – Crashed QTP on login screen

• QTP 10 – QTP hung on login screen

• QTP 11 – QTP recorded, could not replay login screen (some customization would get past this issue)

• VS 2010 – Records and plays back for simple actions. Customization for dealing with grid actions


Case Study 2

• QTP 10– Cannot find any objects on screen, not even the menus

• QTP 11 – Can find most standard controls– Cannot see any 3D objects

• VS2010– Sees all objects– View object properties (center, radius, etc.)


Scoring - Automated Testing

• No contest – Visual Studio


Summary

• Project management – TFS• Requirements – QC (by a hair)• Manual test case management – QC• Manual test execution – QC (by a hair)• Defects – Toss up• Reporting – QC• Administration – QC• Automated testing – VS2010


Considerations

• Cost– Initial (Project, QC, VS, QTP)– Maintenance– Conversion– Customization– Productivity improvements

• Skill level of testing staff

• Best of Breed vs. One Solution

• If starting from scratch (i.e. want to get off Excel)


And the winner is?

• Microsoft’s Team Foundation Server/Visual Studio 2010


How can we help?

• Project engagements – Tool selection and implementation– As needed support

• Staffing– Senior automation resources

• Quality Path™ – Tools and Technology– Broad and deep assessment– Process improvement program

11/15/2010 Slide 44© 2010 BenchmarkQA

Thank You!For additional information on how BenchmarkQA can be of

assistance, please contact:

Warren McLeod952.392.2400

[email protected]

© 2010 BenchmarkQA

Tools for Managing Quality (Nov 2010 BenchmarkQA Software Quality Forum)

Technology

Transcript of Tools for Managing Quality (Nov 2010 BenchmarkQA Software Quality Forum)