Quality is about testing early and Quality is about testing early and

testing oftentesting often

Joe Apuzzo, Ngozi Nwana, Sweety Varghese Joe Apuzzo, Ngozi Nwana, Sweety Varghese

Student/Faculty Research DayStudent/Faculty Research Day CSIS Pace University May 6th, 2005CSIS Pace University May 6th, 2005

OverviewOverview IntroductionIntroduction OverviewOverview

RelevanceRelevance Our methodology: Quality AttributesOur methodology: Quality Attributes Main Message to project teamsMain Message to project teams

QA Walkthrough QA Walkthrough Iterative development and testingIterative development and testing Unit Test case creationUnit Test case creation

ConclusionsConclusions Estimate of QA successfulness Estimate of QA successfulness Lessons LearnedLessons Learned RecommendationsRecommendations

SummarySummary Question session Question session

IntroductionIntroduction The CS615/616 is the capstone course for The CS615/616 is the capstone course for

Pace University’s Masters in Computer Pace University’s Masters in Computer Science curriculum. The focus of which is to Science curriculum. The focus of which is to develop soft skills as well as introducing the develop soft skills as well as introducing the student to formal software engineering student to formal software engineering procedures. procedures.

To develop these real world skills, the students To develop these real world skills, the students are broken up into teams, assigned a project are broken up into teams, assigned a project and thus, responsible for delivering it on time.and thus, responsible for delivering it on time.

The customers must accept the provided The customers must accept the provided solution for the corresponding team to achieve solution for the corresponding team to achieve full marks. full marks.

Our team was assigned the duty of being the Our team was assigned the duty of being the Quality Assurance team. Our goal was to assist Quality Assurance team. Our goal was to assist in delivery of high quality projects, through in delivery of high quality projects, through reviews, testing and the establishment of best reviews, testing and the establishment of best practices. practices.

RelevanceRelevance

“ “It is widely accepted that a It is widely accepted that a "test-early, test-often""test-early, test-often" approach approach can greatly assist in the production can greatly assist in the production of quality software … the design of of quality software … the design of software can impact on the ease of software can impact on the ease of testing so testing must be in borne in testing so testing must be in borne in mind from project conception mind from project conception through to completion.”through to completion.”

The Serco consulting group

Team approachTeam approach

Reviewed 2003 2004 QA team’s Reviewed 2003 2004 QA team’s findingsfindings

Researched Quality Assurance Researched Quality Assurance best practicesbest practices

Evaluated development Evaluated development environmentenvironment

Focus on customer satisfaction Focus on customer satisfaction as a measure of software qualityas a measure of software quality

Our methodology: Quality AttributesOur methodology: Quality Attributes

Clarity of user requirementsClarity of user requirements Who are the stakeholders and who are Who are the stakeholders and who are

the users? Match up each requirement the users? Match up each requirement with each one.with each one.

What are the most important, least What are the most important, least important, which have the greatest risks? important, which have the greatest risks?

What are the user stories? What are the user stories? Write them down!Write them down!

Are there Conflicting, Confusing or Are there Conflicting, Confusing or Incomplete user requirements?Incomplete user requirements?

Our methodology: Quality AttributesOur methodology: Quality Attributes

VerifiabilityVerifiability Define how the program knows when to Define how the program knows when to

continue and when to generate an error.continue and when to generate an error. Each component needs to verify that it's Each component needs to verify that it's

input, output and data is correct. When input, output and data is correct. When components fail to do this bugs will be components fail to do this bugs will be harder to locate, since bad data may be harder to locate, since bad data may be passed from module to module before passed from module to module before being detected.being detected.

Use an integrated system log that is Use an integrated system log that is updated with state at each input or updated with state at each input or decision point.decision point.

Our methodology: Quality AttributesOur methodology: Quality Attributes

Modifiability Modifiability ( modularity or reusability)( modularity or reusability)

Document how other developers who Document how other developers who are not familiar with the project should are not familiar with the project should proceed in making code change. proceed in making code change. Explain how to locate components, Explain how to locate components, where to find there interdependences, where to find there interdependences, where special cases are explained, etc.where special cases are explained, etc.

Our methodology: Quality AttributesOur methodology: Quality Attributes

UsabilityUsability Set the expected level of usability and Set the expected level of usability and

document how you will communicate this document how you will communicate this to the stakeholders. to the stakeholders.

define who and how the system will be define who and how the system will be used. used.

What prior knowledge show a user have What prior knowledge show a user have and if they should have specific skills and if they should have specific skills that an average person would not.that an average person would not.

All assumptions should be listed like All assumptions should be listed like "users will be accustomed to MS "users will be accustomed to MS Windows and using a web browser"Windows and using a web browser"

For each customer delivery review For each customer delivery review usability and have sufficient buy in by all usability and have sufficient buy in by all stakeholders.stakeholders.

Our methodology: Quality AttributesOur methodology: Quality Attributes

SecuritySecurity Define what the security requirements Define what the security requirements

are for each user and or stakeholder.are for each user and or stakeholder. Define what a security issue would be, Define what a security issue would be,

and how it would be mitigated.and how it would be mitigated. How will this software protect the users How will this software protect the users

data and from whom?data and from whom? Do the stakeholders agree with the Do the stakeholders agree with the

security statements? security statements?

Main Message to project teamsMain Message to project teams

Test earlyTest early Test oftenTest often Test enoughTest enough Designate (know your role)Designate (know your role) CommunicateCommunicate IterateIterate

SystemTest

Iterative development and testingIterative development and testing

Re-Design

Unit Test

Code

Integration test

Integratefindings

Unit test case creationUnit test case creationData module

Selection A Selection B

Input A1 Input A2

Display

Update server

Save dataPrint

Input B

There are 4 full paths, 3 decision points, 11 sections. All of which should be included in a unit test.

Estimate of QA successfulnessEstimate of QA successfulness

Reviewing designs early helped Reviewing designs early helped each team to clarified design each team to clarified design trade-offs.trade-offs.

Setting dates for when things Setting dates for when things should be delivered helped should be delivered helped focus each team to keeping on focus each team to keeping on task. task.

Problems were found during Problems were found during testing and corrected.testing and corrected.

Lessons learnedLessons learned

Applying Quality Assurance Applying Quality Assurance from the outside of each team from the outside of each team was not as effective as we was not as effective as we thought it would be.thought it would be.

Soft skills were as important as Soft skills were as important as technical skills. technical skills.

Technological barriers still have Technological barriers still have the biggest impact to a project.the biggest impact to a project.

There is not enough time to do There is not enough time to do project work.project work.

RecommendationsRecommendations Decentralize Quality Assurance, Decentralize Quality Assurance,

make one member of each team the make one member of each team the QA focal point.QA focal point.

Mile stone and commitments need to Mile stone and commitments need to be mapped out as early as possible. be mapped out as early as possible.

Start as early as possible with the Start as early as possible with the QA part of new projects, reusing or QA part of new projects, reusing or reworking the existing documents reworking the existing documents from previous Teams. from previous Teams.

SummarySummary There is still work to be done. There is still work to be done. There is opportunity to streamline There is opportunity to streamline

many QA processes. many QA processes. Communication between the teams Communication between the teams

and also between team members can and also between team members can be improved. be improved.

As in other studies, the more frequent As in other studies, the more frequent and thorough unit testing combined and thorough unit testing combined with a frequent and aggressive unit / with a frequent and aggressive unit / system testing will result in a higher system testing will result in a higher quality product that will have the quality product that will have the greatest chance of delivering on time.greatest chance of delivering on time.

Question SessionQuestion Session

Question?Question? Comments?Comments? Suggestions?Suggestions?

Thank you for your timeThank you for your time