Software Quality and Testing

Software Quality and TestingComputer Engineering DepartmentIndus University1UNIT IBackground and Motivation2Software QualityWhat is? : Software Quality is degree to which a system, system component, or process meets specified requirements or meets customer expectationWhy is required ? : Improvement in software quality improves customer satisfactionHow to measure ? : By softwares usability, availability, reliability, maintainability etc.IBM measure quality with CUPRIMDSOHewlett-Packard focus on FURPSHow to improve ? : Adhering to requirements, delivering quickly without bugs

CUPRIMDSO : Capability, Usability, Performance, Reliability, Installability, Maintainability, Documentation, Serviceability and OverallFURPS: Functionality, Usability, Reliability, Performance, Supportability

3Software Quality FactorsISO 9126 Quality Factors:FunctionalityReliabilityUsabilityEfficiencyMaintainabilityPortabilityHP Quality Factors (FURPS):FunctionalityUsabilityReliabilityPerformanceSupportability

MaCalls Quality Factors

CUPRIMDSO : Capability, Usability, Performance, Reliability, Installability, Maintainability, Documentation, Serviceability and OverallFURPS: Functionality, Usability, Reliability, Performance, Supportability

4Ethical Basis for Software QualityQuality is responsibility of everyoneBusiness decision for cost versus qualityCommunication to end user about quality of softwareTesting for applications thoroughly:Functionality: Positive versus Negative testsSecurityPerformance Environmental

Total Quality Management - PrinciplesCustomer FocusProcess ImprovementHunan Side of QualityTotal Quality Management Continuous ImprovementsMetrics, Models, Measurements and AnalysesSoftware Process Models & MethodologiesWaterfall Development ModelPrototyping ApproachSpiral ModelIterative Development Process ModelObject-Oriented Development ProcessCleanroom MethodologyAgile Methodology7Software Process Models Waterfall Development ModelRequirements Gathering and AnalysisArchitectural Design

IntegrationComponent TestSystem Test Functionality, Regression, PerformanceEarly Customer Feedback and Beta Test ProgramsReleaseHLD / I0LLD / I1Code / I2UTWaterfall model is used for very large and complex systems where structured development approach is required

Requirements Gathering and Analysis: Ensuring that requirements are captured as per customer need and not stated as well ensure that Analysis is done to ensure that this requirements can be translated in systems designHLD Develop External functions and interface, Design internal component structure and intra-component interfaces and data structures, Ensure function requirements are satisfied, build structure of component files to system/product structure, LLD Finalize design of components and parts with product or system, complete component test plan, verify all HLD changes are accomplished in detail designCode Design from LLD transform in required code (modules, macros, includes, messages etc), code component test cases, Verify changes in LLD and HLDUnit Test: verify code is complete per HLD-LLD, Ensure logic is correct, data path are correct (test non dependent code), Ensure correct responses, error messages etc are displayedComponent Test: Test external interfaces, test inter-component interface, Test complete component level functionality, multitasking of components, usage of shared resourcesSystem Test: System functionality test, system regression test, system performance test, usability testEarly Customer Programs: Product feedback functions supported, ease to use, quality of documentation etc, Install ability of hardware and software, reliability, performance, system connectivity, customer acceptance

8Software Process Models Waterfall Development Model Model is used when requirements are known by customerModel is used for significantly larger and complex system e.g. implementing HR system company wideCompletion of one cycle leads to final productProduce good quality product due to known requirements and validation is done at each step in development cycleTakes significant amount of time to marketHigher costRequirement modifications will become costly9Software Process Models Prototype ApproachRequirements Gathering and AnalysisQuick DesignBuilding PrototypeRefining Design & PrototypeCustomer Evaluation of PrototypeCustomer SatisfiedFull-Scale Development10Software Process Models Prototype ApproachModel is extremely useful when requirements are not known by customerModel is very useful to know GUI designs to customerShorter time to marketRequirement modification can be implementedIf prototype is not used then effort is wasted May lead to higher cost if several prototypes are build before the final oneQuality may be compromised 11Software Process Models Spiral Model

One development cycle comprise of one spiral: Concept of operation, software requirements, product design, details design, and implementation.12Software Process Models Spiral ModelModified and Improved version of Waterfall ModelRelies heavily on prototyping and risk managementMore flexible then waterfall modelVerification is done at each phaseUsed in many modern systems which involves reuse, object-oriented development, rapid prototyping etcIf prototype is not used then effort is wasted 13Software Process Models Iterative Development Model

14Software Process Models Iterative Development ModelIt is one of the Agile methodology which are extremely important in current development environmentEach Iteration provide additional functionalitiesCustomer are able to see finished product components at very early stage of developmentDepending on feedback from customer functionality can be modified in subsequent iterationCost management is betterLower riskWrong architectural decision in early iteration can cause major roadblocks in subsequent iterationsThere may be duplication of effort in terms of testing15Software Process Models Object Oriented Development ProcessModel the essential system (MVC architecture)Derive candidate-essential classesConstrain essential modelDerive additional classesSynthesis classesDefine interfacesComplete designImplement the solution16Software Process Methodology Cleanroom Methodology

Cleanroom metholody states to avoid dependence on costly defect removal processes by writing code increments right the first time and verifying correctness before testing, Its process model incorporates the statistical quality certification of code increments as they accumulate in the system.17Software Process Methodology Agile MethodologyAlternative to waterfall, or traditional sequential development.Uses incremental, iterative work cadences, known as sprints which are 1 to 4 weeks longAgile team should include a customer representative for efficient face-to-face communicationVery short feedback loop and adaptation cycleQuality focusSCRUM is most common agile methodology18Quality Standards, MethodologiesThe SEIs Capability Maturity Model (CMM & CMMI)The SPR Assessment (Software Productivity Research)The Malcolm Bladrige AssessmentISO 9000 (International Standard Organization)

19Quality Standards, MethodologiesThe SEIs CMMQuality standards specific for software developmentCMM Standard is used to evaluate process maturityFive Maturity Levels: Initial, Repeatable, Defined, Managed and OptimizingInitial (Level 1): Chaotic: Unpredictable cost, schedule and quality performanceRepeatable (Level 2): Intuitive: Cost and quality are not reliable, some control over schedule. Key Process Areas:Requirements ManagementSoftware project planning and oversightSoftware subcontract ManagementSoftware quality assuranceSoftware Configuration Management

20Quality Standards, MethodologiesThe SEIs CMM Cont..Defined (Level 3): Qualitative: Reliable cost and schedule but quality is unpredictable Key Process Areas:Organizational process improvementsOrganizational process definitionTraining ProgramIntegrated Software ManagementIntergroup CoordinationPeer ReviewsManaged (Level 4): Quantitative: reasonable control over quality, cost and schedule. KPA area:Process measurement and analysisQuality managementOptimizing (Level 5): Continuous process improvements and automation. KPA are:Defect preventionTechnology innovationProcess change management

21Quality Standards, MethodologiesThe SPR AssessmentQuality standards specific for software developmentSPR Standard is used to uncover issues with productivity, quality and user satisfaction.It is sometimes used in conjunction with SEIs CMMFive point scale: 1-Excellent, 2-Good, 3-Average, 4-Below Average, 5-PoorHas automated tool (CHECKPOINT) to for collecting a resource planning and quality projection dataStandard collect qualitative productivity and quantity data from each project

22Quality Standards, MethodologiesThe SPR Assessment ContSoftware quality and metrics covered by assessmentQuality and productivity measurementsPretest defect removal experience among programmersTesting defect removal experience among programmersProject quality and reliability targetsPretest defect removal at the project levelProject testing defect removalPost-release defect removalFinding are divided in 5 streams:Findings about projects and software products assessedFindings about the software technologies usedFindings about the software processes usedFindings about the ergonomics and work environment for staffFindings about personnel and training for management and staff

23Quality Standards, MethodologiesThe Malcolm Baldrige AssessmentQuality standards used for all industry (not just software), especially in USA considered as most prestigious award. AKA MBNQA.7 examination criteria: Leadership, Information and Analysis, Strategic quality planning, Human resource utilization, Quality assurance of products and services, Quality results, Customer satisfactionSystem scores based on 3 evaluation dimensions : approach, deployment and results Approach refers to the methods the company is using to achieve purposes addressed in examination item.Deployment refers to the extent to which the approach is appliedResults refers to outcomes and effects of achieving the purpose addresses and applied24Quality Standards, MethodologiesISO 9000Registration standard to show industry that organization is following certain quality standardsRequires audit passing in 20 major elements for software companies to get ISO certification.For software development it collect product metrics as well as process metricsCorrective actions and document control are the area of nonconformance most of the timeVery strong requirements for documentation: ownership, versions, approval, distributionDescribed as say what you do, do what you say, & prove it

20 elements:Management responsibilityQuality systemControl reviewDesign controlDocument controlPurchasingPurchaser-supplied productProduct identification and traceabilityProcess controlInspection and testingInspection, measuring and test equipmentInspection and test statusControl of nonconformance of productCorrective actionHandling, storage, packing and deliveryQuality recordsInternal quality auditTrainingServicingStatistical techniques

25Quality Improvements with MethodologiesCMM, CMMI & ISO Focuses on process improvement and assessmentSPR & Six sigma Focuses on producing zero defect productsMalcolm Baldrige & TQM Management strategies to embed quality awareness in all organizational processThese methods drive quality improvements by process improvements, product improvements and people based improvements

26Information EngineeringDefined as generation, distribution, analysis and use of information in systemsEnables effective communication of information throughout enterpriseHelps improve people skills, procedure and systems and hence the overall qualityTwo types of IE variant:Data Processing driven: Enable IS Departments to develop information systems that satisfied the information needs. CASE tools available today support this DP-drivenBusiness driven : Enable strategic business planning. It was designed for rapid change in the client/server, object-oriented environment of the business-driven 1990s

27Information Engineering ContData Processing driven stages : Information Strategy PlanningOutline Business Area AnalysisDetailed Business Area AnalysisBusiness System DesignTechnical DesignConstructionTransitionBusiness driven stages:Strategy AnalysisStrategic ModelingTactical and Operational ModelingActivity ModelingProcess ModelingCode GenerationSoftware Tools: Information Engineering Facility (IEF) from Texas InstrumentsDP-driven Variant of IEInformation Strategy Planning: The fundamental objective of Information Strategy Planning (ISP) is to develop a plan for implementing business systems to support business needs. The existing systems landscape is compared to the ambitions expressed in the current business plan and a number of development projects for new or enhanced systems are identified.Outline Business Area Analysis: For each development project, business analysts define the business processes and data potentially required in the new system. These are modelled using process decomposition diagrams, process dependency diagrams and entity-relationship models.Detailed Business Area Analysis: The purpose of a DBAA phase is to provide detailed models as a solid basis for system design. Processes are decomposed to elementary business processes and the business logic of the processes is expressed in data actions against the fully normalised data model. In this way, the process and data models are tested against one another before construction.Business System Design: The purpose of a Business System Design project is to specify all aspects of a system that are relevant to its users, in preparation for the technical design, construction, and installation of one or more closely related databases and systems. The elementary processes are designed into procedures that can be executed by users. Unambiguous and consistent specifications with the volume of detail necessary to make planning and technical design decisions are prepared.Technical Design: A Technical Design project prepares an implementation area for construction and installation. The key tasks are structured to produce a system and database that meet the user's acceptance criteria and are technically sound.Construction: The objective of the Construction stage is to produce a system, as defined in the technical specification, on time and within budget. The system should be of an acceptable quality, and contain all necessary operating and user procedures. The task is complete when the acceptance criteria for the business system are met.Transition: Transition is defined as the period during which newly developed procedures gradually replace or are interfaced with existing procedures. The execution of a Transition project obviously demands a thorough understanding of both the system to be installed and the systems to be replaced.

Business-driven Variant of IE for Rapid DeliveryStrategy Analysis: This is a rapid delivery method for senior managers and business unit managers for refinement of existing strategic business plans, or development of new strategic business plans if none exist yet.Strategic Modeling: This uses a facilitated modeling session with senior business managers who review the strategic business plans to develop a strategic model. This is an enterprise data model where many-to-many associations have been decomposed to identify priority business activities and processes identified by management. This uses entity dependency analysis to automatically derive project plans and project maps from the strategic model. It results in reusable processes for rapid delivery into production as integrated databases and reusable systems.Tactical and Operational Modeling: This uses the same approach as for strategic modeling, but focuses on tactical business units - expanding into tactical attribute detail and later operational attribute detail for physical database generation and installation.Activity Modeling: Activity models, based on IDEF0 and activity-based costing, are used to document priority business activities for rapid delivery.Process Modeling: Business Process Modeling Notation (BPMN) is used, supported by modeling tools, to define process model diagrams in BPMN of priority activities for rapid delivery into production.Code Generation: BPMN process model diagrams are used to generate XML-based code in Business Process Execution Language (BPEL) for execution.

28Measuring Customer SatisfactionWhat to measure ? Measure overall satisfactionMeasurement for loyaltyAttribution satisfaction product/feature feedbackMeasure exit/abandonment rateNet promoter score likelihood of referral by customerThings going wrongHow to measure ?Request that your customers fill in a surveyAsk the right questions in surveyFamiliarize yourself with customer expectation and meet them on regular basisDiscover customer satisfaction by recognizing where you are going wrongFind out who has more satisfied customers then you and why29Software Quality EngineeringDefine the skills and knowledge necessary to perform software quality engineering tasksUnderstand the software life cycleDetermine how to evaluate software quality activities and processes and determine whether they meet their intended purposeChoose applicable standards and meeting techniques30Defining Quality RequirementsProduct versus process requirementsFor defining quality requirements all different types of requirements must be defined :Architectural requirementsBusiness requirementsUser (stakeholder) requirementsFunctional (solution) requirementsQuality-of-service (non-functional) requirementsImplementation (transition) requirementsTypes of requirement:Requirements are typically classified into types produced at different stages in a development progression, with the taxonomy depending on the overall model being used. For example, the following scheme was devised by theInternational Institute of Business Analysisin theirBusiness Analysis Body of Knowledge[3](see alsoFURPSandTypes of requirements).Architectural requirements : Architectural requirements explain what has to be done by identifying the necessary systemsstructureand systemsbehavior, i.e.,systems architectureof a system.Business requirements: High-level statements of the goals, objectives, or needs of an organization. They usually describe opportunities that an organization wants to realise or problems that they want to solve. Often stated in abusiness case.User (stakeholder) requirements: Mid-level statements of the needs of a particular stakeholder or group of stakeholders. They usually describe how someone wants to interact with the intended solution. Often acting as a mid-point between the high-level business requirements and more detailed solution requirements.Functional (solution) requirements: Usually detailed statements of capabilities, behavior, and information that the solution will need. Examples include formatting text, calculating a number, modulating a signal. They are also known ascapabilities.Quality-of-service (non-functional) requirements: Usually detailed statements of the conditions under which the solution must remain effective, qualities that the solution must have, or constraints within which it must operate.[4]Examples include: reliability, testability, maintainability, availability. They are also known ascharacteristics,constraintsor theilities.Implementation (transition) requirements: Usually detailed statements of capabilities or behavior required only to enable transition from the current state of the enterprise to the desired future state, but that will thereafter no longer be required. Examples include: recruitment, role changes, education, migration of data from one system to another.31Defining Quality RequirementsCharacteristics of good/quality requirementsUnitary (Cohesive) : Address one and only one thingComplete : Fully stated with no missing informationConsistent : Does not contradict with other requirement and fully consistent with other documentationNon-conjugated (Atomic): Does not contain conjunctions e.g. Password must contain 8 characters long with at least 1 special character from & ( ) *, instead it should be 1) Password must contain 8 characters. 2) Password must have at least 1 special character from & ( ) *Traceable: Must trace back to business needCurrent : Should not be obsolete with passing of timeUnambiguous : It is subject to one and only one interpretationSpecify Importance/Priority : Criticality of the requirementVerifiable : Requirement is testable32Management Issues for Software QualityEach stage of SDLC is prone quality issues e.g.Requirements are ambiguous, not complete etcVersion Control of software is not maintainedSystem architecture cannot handle load of user baseManagement of Quality issue is to capture the quality issues, process them by initiating quality improvement activities and monitoring until resolution.SAP QIM is one of the tool that provides the solution

33Data Quality Assurance & ControlData quality assurance(QA) improves data quality by : Profiling of data to discover inconsistencies and anomaliesPerforming datacleansing activitiesData quality control(QC) is the process of controlling the usage of data with known quality measurements for an application or a processData QA processes provides following information to Data QC :Severity of inconsistencyIncompletenessAccuracyPrecisionMissing / UnknownE.g. if Data QC finds too many inconsistencies in data from QA, it will stop data being sent to process or application needed the same so that incorrect processing / function does not happen

34Benchmarking and CertificationBenchmarking is process of developing requirements and setting goals to measure your performance against that of best-in classRequires research into best practicesGoes beyond industry standardBased on learning from othersSteps for the benchmarking process:1.Planning2.Analysis3.Integration4.Action5.Maturity

Steps for the benchmarking process:1.Planning1.1. Identify what is to be benchmarked1.2. Identify comparative companies1.3. Determine data collection method and collect data2.Analysis2.1. Determine current performance "gap"2.2. Project future performance levels3.Integration3.1. Communicate benchmark findings and gain acceptance3.2. Establish functional goals4.Action4.1. Develop action plans4.2. Implement specific actions and monitor progress4.3. Recalibrate benchmarks5.Maturity5.1. Leadership position attained5.2. Practices fully integrated intoprocess

35Benchmarking and Certification ContCertification is process of confirming that :measurements from your organizations product or process is performing in accordance with benchmarkproduct is functioning and performing as per requirementsprocesses are being followed that was said to be followed

36