October 15, 2004 – 1

WelcomeIPMA and SolutionsIQ

Professional EventTesting, Testing, 1…2…3…

Improving software quality -- one bug at a time

October 15, 2004 – 2

Agenda• Building the Test Framework

– Jan McCollum, SolutionsIQ• Break

• Practical Panel Discussion– Cheryl Hainje – AFRS Product Manager, OFM– Dotti Lane – QA Project Manager, OFM– Tim Vessey – POS Project Manager, LCB– Stein Wang – Quality Assurance Lead, SolutionsIQ

• Break• Testing Templates & Checklists

October 15, 2004 – 3

SolutionsIQ Overview• SolutionsIQ is a full-spectrum IT services

company• 25 years of technology services and solutions • 400+ consultants• Corporate headquarters (Bellevue, WA)• Professional Services (Bellevue, WA)• Oregon Branch Office (Lake Oswego, OR)• 8+ years of serving the State of WA

– DOC, AOC, LCB, DNR, DOL, LNI, & DSHS

October 15, 2004 – 4

SolutionsIQ Expertise• Professional Services Division

– Consulting and Analytical Solutions• Project management• Assessments and feasibility studies• Design and architecture roadmaps

– Development and Test Solutions• Full life cycle development projects• Custom application development• EAI, portals, and business intelligence• Quality assurance and testing solutions

October 15, 2004 – 5

Building the Testing Framework

Jan McCollumManager, Quality Assurance and Testing Solutions

October 15, 2004 – 6

Setting Goals• Knowing WHAT you want is as

important as knowing how to get it

– Defining the vision– Defining the timeline– Gaining acceptance and buy in

October 15, 2004 – 7

Defining the Vision• To define the vision look at what

came before

– What went well– What went badly– What now– Where do you want to go

October 15, 2004 – 8

Testing vs. Quality Assurance• Testing is about finding bugs

• Quality Assurance is about preventing them!

October 15, 2004 – 9

Quality Assurance• Takes time• Is about the overall effort –

including development• Methodologies can be very formal

October 15, 2004 – 10

QA Applied to Testing• Quality assurance principals

applied to the testing effort will produce higher quality work

October 15, 2004 – 11

Establishing a Timeline• The 6 month / 1 year / 3 year plan

– Implement processes and strategies that give the best return on investment

October 15, 2004 – 12

Quality Testing Roadmap• After the goals and objectives are

complete, make them real by publishing the quality testing roadmap

October 15, 2004 – 13

Quality Testing Roadmap• Roadmap should include…

– Test team structure– Communications plans– Test processes– Test procedures

October 15, 2004 – 14

Quality Testing Roadmap• Test scope• Test dependencies and impacts• Automation transition plan• Test deliverables

October 15, 2004 – 15

Gaining Acceptance and Buy In

• Development• Business management• Project management• IT management• Customer/product support

October 15, 2004 – 16

Making it Happen!• Organizational structure• Qualified candidates• Roles and responsibilities

October 15, 2004 – 17

Test Planning• The master test plan: a one-stop

shopping guide for your project

– Contents– Contributing documents– Sign-off procedures

October 15, 2004 – 18

Test Planning• Test matrix and test suites

– Detailed test steps– Pass/Fail results– Tester who performed tests

October 15, 2004 – 19

Test Planning• Test case design – what is a good

test case?

– Accurate – tests what it’s designed to test

– Repeatable, reusable – has a life after this release

– Economical – no unnecessary steps

October 15, 2004 – 20

Test Planning• Test case design

– Traceable to a requirement– Appropriate for test environment,

testers– Self-standing has enough

information for anyone to run

October 15, 2004 – 21

Test Planning• Test case design: How to make

good test cases better

– Setup, environment, data– Steps, actions and expected results– Use active voice in expected results– System displays this, does that– Simple, conversational language

October 15, 2004 – 22

Test Planning• Test case design: Why work to

improve test cases?

– Productivity – less time to write and maintain cases

– Testability – less time to execute them– Scheduling – better reliability in

estimates

October 15, 2004 – 23

Defect (Bug) Management• Deciding upon a tool

– Easy of configuration– Ability to add/change fields– Reporting capabilities

• Integrated solution

October 15, 2004 – 24

Defect (Bug) Management• The bug lifecycle

– Who can create bugs– Who can assign bugs– Who can close bugs

October 15, 2004 – 25

Defect (Bug) Management• The bug triage meeting

– Purpose and who should go• Reporting

– Determining a trend• Bug metrics

– Number of bugs found– Bugs found in production vs. test

cycle

October 15, 2004 – 26

Moving On• Improving the process:

Requirements traceability

– Test cases for each requirement– Requirements matrix– Tracing requirements to defects

October 15, 2004 – 27

Moving On• Improving the process: Risk-based

testing

– You can’t test everything so test what is important

– The risk list and how to use it to drive test strategy

October 15, 2004 – 28

Broadening Your Scope• Build verification testing

– Also called smoke or acceptance tests– Is a subset of the major functional

areas

• Integration testing– Testing the entire system

October 15, 2004 – 29

Broadening Your Scope• Compatibility testing

– How application works with other apps• Configuration testing

– Testing on different configurations• Setup testing

– Testing the installation• Regression testing

– Verify if bug fixes are successful

October 15, 2004 – 30

Broadening Your Scope• Black box testing

• White box testing

• Grey box testing

October 15, 2004 – 31

Improving Quality• Testing metrics – measure your

success

• Bug tracking metrics– Number found– Number found per component– Daily bug find rate

October 15, 2004 – 32

Improving Quality• Test case effectiveness

– Metric: Test case effectiveness; test case effectiveness = bugs found in test/total found * 100

• Test coverage– Metric: Test coverage (absolute) =

tests conducted/total tests * 100

October 15, 2004 – 33

Improving Quality• Test team performance

– Metric: Test process effectiveness: test process effectiveness = bugs fixed/bugs found * 100

– Metric: Planned days vs. actual days in test

October 15, 2004 – 34

Improving Quality• QA and test involvement early!

• Design reviews– Why testers should attend

• Develop and use checklists

• Project closeout meetings– You should have them

October 15, 2004 – 35

Questions?• For additonal information, email

– JMcCollum@SolutionsIQ.com