SE420 Software Quality Assurancemercury.pr.erau.edu › ~siewerts › se420 › documents ›...
Transcript of SE420 Software Quality Assurancemercury.pr.erau.edu › ~siewerts › se420 › documents ›...
December 3, 2014 Sam Siewert
SE420
Software Quality Assurance
Lecture 12 – Final Review
SQA FIRST HALF
MSQA Chapters 1-6, Notes Through Week 6
Sam Siewert 2
Coverage – As Noted on SyllabusMurali Chapters 1 to 5– Chapter 1 – Quality Gurus, SQA Department or Integrated Process
and Policies (Dijkstra, Demming, Boehm)
– Chapter 2 – Dimensions of Quality (Specification, Design, Development, Conformance)
– Chapter 3 – Black-box, White-box
– Chapter 4 – Role of SQA Department
– Chapter 5 – Verification (Validation), Walkthroughs, Inspections
Validation Definition and Test Scenarios in Chapter 6
In-Class Notes and Assignments– Waterfall, Evolutionary Spiral and Extreme Programming
– Boehm’s V
– Agile Manifesto and Evolutionary Process
– Unit Tests, Drivers, and Black-box / White-box Tools & Methods
– Acceptance Testing
– Integration & Test – Unit/Component, Sub-system, System, System of Systems
Sam Siewert 3
Knowledge and Concepts DayTrue / False, Multiple Choice, Short Answer
Concepts, Terminology, and Understanding of SQA Methods of Validation and Verification
Focus on Boehm’s V – Unit Level and Acceptance
Knowledge of I&T and System Test Concepts Only
Sam Siewert 4
What, How, Why, When, Who
What, Why, When
Day 1 & 2
Day 1
Test TypesAcceptance Tests [Subset of Unit, I&T, System Tests]
– Test as Product Will Be Used
– Out of Box Experience
– Pre-Ship Test
Unit Tests1. Function and Feature
2. Performance Tests [Operations / second]
3. Error Handling and Recovery [Failed API calls, Bad Return Codes]
4. Unit Reliability Stress Tests, Soak Tests [Overnight, Memory Leak, Resource Exhaustion]
5. Corner and Coverage [Initial Conditions, Input Extremes, Pattern]
Regression Tests (Code checkout, Build, Tests at All Levels)– Verify Code Fixes Remove Defect(s)
– Verify that Code Changes Do Not Introduce New Bugs or Anomalies
Integration and System Tests1. Diagnostics [Built-in Self-Test]
2. Ease of Use and Provisioning
3. Interoperability
4. Upgrades
5. Sub-system and System Resource Testing
– Unit Tests Adapted for Sub-System and System
E.g. Features, Performance, Error Handling
E.g. Reliability Stress & Soak Integrated
– Reliability with External Fault Injection – On Interfaces and at Hardware/Software Interface
• E.g. Cable Pull on Disk Drive for HW Fault
• E.g. Data Corruption on Buffer for Function Call Interface
Sam Siewert 5
Theory, Implementation, Methods
Black-box Unit Level
– Test Cases in Test Driver
– Code Under Test in Separate Modules
– Modify Code Under Test as Little as Possible
– Use of Gcov and Lcov
– Exit Criteria
White-box Unit Level
– Basic Blocks
– Disassembly Methods (-S option for GCC)
– DDD Single Step Debugger
– Software in Circuit Tools – Software Debugger, Gcov
– Hardware in Circuit Tools - JTAG
Sam Siewert 6
Familiarity with Test Scenarios
RAID 10, 50– XOR Encoding
– Recovery of Lost Disk Drive (File or Segment) with XOR of Remaining
– Normal Write and Read Operations
– Read with Recovery on Fly [Degraded Mode]
– Write During Degraded Mode
– Lost Segment Rebuild
Encryption and Decryption of Messages– Simple Substitution
– Transposition [Block Cypher]
– Mathematical Hashing Function for Substitution or Transposition
– Combinations of Methods
– Reverse Operation to Decrypt
Image Processing Transformation– Load Image File to Buffer
– Apply Pixel Operations to Buffer [X, Y, Channel]
– Write Buffer to Image File
– Preview Display to User
Sam Siewert 7
Walkthrough vs. Inspection
What is the Difference? [Informal, Formal Gate]
Why are Both Done? [Validation, Verification, Standards]
When is Each Done Ideally? [Phase of Lifecycle]
Who is involved in Each? [Peer Engineers,
Management, SQA]
Where is Each Done? [Office one-on-one,
Meeting/Conference Room with Committee]
Sam Siewert 8
SQA SECOND HALF
Scan MSQA Chapters 7-9, Notes Through Week 14
Sam Siewert 9
Integration and TestIntegrate Software Modules [units] and Hardware Components into Sub-systems
Test Focus on Interfaces [Function, Message, Shared Memory, Hardware], Protocols, and Interoperability of Modules
Sam Siewert 10
Test TypesPositive Tests– Functional Software Interface Tests
Functions calling Functions – API
Message Passing – Local Message Queues, Network, Client-Server
Shared Memory – Synchronization, Buffers
– Hardware Interface TestsDrivers and Device Interfaces
Firmware [ROM Code, Run out of Reset]
Negative Tests– Software Interface Faults
– Hardware Interface Fault Injection
Diagnostics [Built-in Self-Test]
Unit Interoperability– Sub-system Resource Testing – Memory, CPU, I/O, Storage, Power
– Protocols – Message Acknowledgement, Command/Response, Background Commands, Peer-to-Peer, etc.
Sam Siewert 11
Outline for Every Integration Test1. Check out Specific Source Code Test Configuration – CMVC Tools, Git
– Collection of Modules [Units] Tagged by Revision Control
– OR Current
2. Build and Link Modules (*.o) and Libraries (*.a) into Sub-system to Test
3. Load / Install Sub-system Code onto Test Hardware Platform of Known Configuration
– Record key hardware configuration parameters
– E.g. for I/O HW config - lspci, lsusb,
– General config - hwinfo
– Linux OS kernel build config - uname –a
– cat /proc/meminfo
– cat /proc/cpuinfo
4. Run Integrated Test(s) [with Gcov, Lcov, Gprof]
5. Review of Expected Syslogs, Output to Terminal, for Each Feature
6. Review Performance Profiles
7. Track Bugs, Anomalies, and Disposition as Defects
Sam Siewert 12
Negative Tests – Fault Injection
Hardware Layer
– E.g. Cable Pull for Storage [SAS, SATA]
– Cable Pull for Networking [Ethernet]
– Power Loss
– Analog Circuit Methods [Over/Under Voltage, Short, Open, Mis-
route]
Software Layer
– Buffer Corruption [Shared Memory Corrupt, Message Loss,
Corrupt, L from Test Client]
– Network Layer 2, 3, 4 Packet Corruption - Wireshark
– Bad Command Generation [Test Client]
– Bad Pointer [Buffer]
Sam Siewert 13
Fault Injection Methods
(IP Network Layer Forwarding)
Sam Siewert 14
Sockets API
MPEG Program
VLC Stream Service
UDP
IP
IEEE 802.3z - GigE
Cat-6 UTP
Sockets API
MPEG Program
VLC Player
UDP
IP
802.11abgn - Wireless
Radio Transmission
L5
L6
L7
L4
L3
L2
L1
IP Forwarding
[Enable/Disable]
802.3z -
GigE
Cat-6 UTP
802.11abgn -
Wireless
Radio
Transmission
Fault Injection Methods
(Application Layer Message
Forwarding)
Sam Siewert 15
Sockets API
MPEG Program
VLC Stream Service
UDP
IP
IEEE 802.3z - GigE
Cat-6 UTP
Sockets API
MPEG Program
VLC Player
UDP
IP
802.11abgn - Wireless
Radio Transmission
L5
L6
L7
L4
L3
L2
L1
Sockets API
MPEG Program
Store and Forward
[Corrupt, Delay, Drop]
UDP
IP
802.3z -
GigE
Cat-6 UTP
802.11abgn -
Wireless
Radio
Transmission
Root-Cause Analysis
Field Issue - Anomaly, Reported Bug, Data Corruption,
L
– Software Defect?
– Hardware Reliability
– User Error
Reproducibility – Capture Conditions via Logging
– Recreate Scenario in SQA / QA Lab
Trace to Root-Cause– Assert
– Analysis Triggers
– Propose Fixes
– Apply and Regression Test
– Release Maintenance Patch
Sam Siewert 16
Case Study – Mars Pathfinder Story
JPL Mission Flow to Mars, Landing on July 4th, 1997
Pathfinder Rolling Resets on Final Approach to Mars
Capture Orbit
VxWorks RTOS Used
Reproduction of Anomaly on the Ground
Root-Cause Analysis
Proposed Fix
Sam Siewert 17
Bug Open/Close Rates and ReadinessControversy – Bug Counts, Closure and Prediction of Phase Transition Readiness– E.g. Unit to I&T to System Test to Acceptance Test to Shipment
– Can Be inaccurate due to Unsatisfactory Testing or Lack of Criteria
– Guideline for Project Management [Compared to Guessing!]
– Not all Reported Bugs Become Defects [Test Case Errors, Human Error]
Sam Siewert 18
http://www.testandverification.com/DVClub/24_Jan_2011/Greg_Smith.pdf
Test C
ase C
overa
ge
[E.g
. C
ode P
ath
Covera
ge]
Bug C
ounts
[Report
ed,
Not
Verified a
s D
efe
ct]
I&T Summary
Integration of Modules [Units]
Configuration of Sub-systems and Systems
Sub-systems built from Modules [units] and Components
Positive Tests
Negative Tests
Interoperability of Modules and Components
I&T Precedes System Test, Where Sub-systems are L– Fully Integrated
– Configured Similar to Deployment [Perhaps Not Exact – E.g. Spacecraft in Thermal-Vac Testing]
– Stimulated with Tests Replicating Operations
Sam Siewert 19
System TestIntegrate Sub-systems into System Configuration
Test Focus on End-to-End, Black-box Testing and Architecture
Validation, System Requirements Specification
Sam Siewert 20
System Test Types – Extends I&T
Emphasis on Full Configuration End-to-End Testing
Usability and HCI Testing – GUI, CLI, API
Performance Tests – Profiles and Traces
Workload Testing – Normal, Peak
Stress Testing – Accelerated Lifetime
Security Testing
Scalability Testing – Volume [Capacity, Nodes, Size]
Nightly Build and Smoke Test [Regression]
Installation Testing – Out of Box Experience
High Availability Recovery Testing
Mission Critical Fail Safe and FDIR Testing
Compatibility and Interoperability Testing
Sam Siewert 21
Software End-to-End System
Sam Siewert 22
Software Units
Flight Computer [DATA]1. Low Rate Downlink, Uplink Tasks
2. Flight Command/Response [FRTE]
3. Device I/O
Instruments [SXEE, FARUS, SXEE]– Recycled from LASP, Integrated Firmware
Ground Support System [GSFC ACCESS]– Low Rate Downlink/Uplink Interface
– Ground Real-Time Engine
– Selective Telemetry Monitor
– Spacecraft Command Language [Scripts, Rules, Constraints]
– Data I/O Server [Kerberos]
– OODBM Telemetry Database
– Operator GUI
Payload Operations Center [CU-Boulder]– Data I/O Client [Kerberos]
– DATA-CHASER Planning and Scheduling
– Payload Operations GUIs
Remote Science and Engineering [JPL]– Remote Engineering GUIs
Sam Siewert 23
Regression Testing and Test
AutomationRe-test Units After Defects are Fixed
Re-run I&T and System Tests
Automation from Checkout, Build, Test Sets, Verification & Validation of Expected Results
Sam Siewert 24
Regre
ssio
n T
esting
(Test A
uto
mation)
General V&V Automation
Many Tools Exist
DejaGnu – a short
introduction
Go Beyond Batch
Test and Visual
Inspection
Expect / DejaGnu –
Automates V&V
Reports in Addition
to Running Drivers
Sam Siewert 25
Batch is Execution
Automation Only
To Run Drivers in
A Sequence
Test Automation and SEI CMMCoverage Criteria, Bug Open/Close Rate Metrics, Pass/Fail Statistics Provide Quantitative Management [Level 4]
Automation is a Step Toward Optimization – Necessary, but Not Sufficient (Metrics for Feedback Process Improvement)
Sam Siewert 26
Ad Hoc, “Code & Fix” – Driven by
Individual Styles, Un-coordinated,
More of an Art than Practice
Clear Phases with Key
Deliverables – Analysis,
Design, Units, Test Plans
E.g. Boehm’s “V” and
Spiral Model, Applied to
All Projects - Repeatable
Entry/Exit Criteria for
Phases, Metrics Drive
Decisions in Process [Gates]
Feedback Used to
Improve Process with
Automation and Innovation
SEI CMM Levels
http://www.tutorialspoint.com/cmmi/cmmi-maturity-levels.htm
Cyberphysical Systems
Sam Siewert 27
Infamous System Software Defects
Infamous & Famous Field Defects
1. Toyota ABS Software Recall, 2010
2. Windows Genuine Advantage Outage, 2006
3. Mars Climate Orbiter Units Interoperability, Loss, 1999
4. Mars Pathfinder Priority Inversion, Recovered, 1997
5. Ariane-5 501 Cluster Launch Software Failure, 1996
6. Pentium FPU Bug – Software Acceleration Co-Processor Recall, 1994
7. ATT 4ESS Upgrade, Outage, 1990
8. Therac-25 Radiation Therapy Patient Overdosing, 1985-86
9. NORAD False Alarms, User Error for Test Mode, Device Failures, 1979/1980
10. Apollo 11 Guidance Computer Overload, 1969 - Recovered
Sam Siewert 28
http://en.wikipedia.org/wiki/Software_bug
RCA, Patch, Test, Fix!
http://history.nasa.gov/SP-350/ch-11-4.html
Rear Admiral Grace Hopper
Week 14 & 15
Week 14 – Code Walkthrough Practice in Class
Week 15 – Exam #2
Sam Siewert 29