Designing Software For Testability

A forgotten design pattern

Rohit Nayak

Talentica

Agenda Background and Motivation DFT in VLSI and hardware design Issues in Software Testing DFT for Software Development Automation and Testability Some Examples Cons, Review

The Anti-Pattern

If it works, its the developer, if not blame QA!

VLSI/PCB Testing Issues Mass produced Each piece needs to be validated Complexity

Number of subsystems Amount of logic

Access to internal logic Testing costs Cost of recall

Testability: “Effort required to test a product to

ensure that it performs its intended

function”

DFT Principles Controllability Observability

Test Point Insertion Built-In Self Test (BIST) Fault Modelling Fault Simulation Test Pattern Generation

Software Testing“Testability is a design issue and needs to

be addressed with the design of the rest of the system. Projects that defer testing to the later stages of a project will find their programmers unwilling to consider testability changes by the time testers actually roll onto the project and start making suggestions. Testability takes cooperation, appreciation and a team commitment to reliability.”

- Bret Pettichord

Cost Of Bugs

Where Found

Where Introduced

Requirements Architecture Coding System Test Production

Requirements 1x 3x 5-10x 10x 10x-100x

Architecture 1x 10x 15x 25-100x

Coding 1x 10x 10-25x

Software Testability Suitability: clarity of specifications QA has Observability: we can only test what is visible Simplicity: easier design/UI makes testing easier Controllability: better the control, better the coverage

and automation Stability: how often it changes Performance: how fast it works Diagnosability: writing effective bug reports!

Software DFT Patterns BIST

Automation suite w/ oracle

Unit Tests Assertions

Fault Simulation Mock/Stub Invalid params

Controllability Decoupling Bypassing Mock/Stub Set/Reset

Observability Logging Reporting Test Interfaces

Good Automated Tests are

Repeatable Easy to Write Easy to Understand Fast Way Easier with a Testable Software

When/How to Automate? Manager & Team are commited Testers/Dev either experienced or interested

in learning to script Product release cycles managed well Functionality / UI changes under control Early/Incremental approaches work best Build integration, reporting Each bug results in a automated test

Automation As Coding Automation scripts and frameworks are CODE! Use tools and scripting languages Evaluate tools on real problems Follow development processes

Spiral/Agile

Use Source Control same as rest of code These will have bugs as well!

Test “oracle” Expected Result

Generated once from previous run or Manually specified or Legacy system

Time adjustment Results

Database File Inline with code

Test Environment Multiple VMs Automated installs/images Automation tools Bug reporting tools

Unit Tests Each module has independent set Developer written/maintained setup, teardown xUnit, TestNG Runs on build Can run sub-set on install TDD

Diagnosability All environments: Test / Staging / Production

Errors result in visual messages Errors are raised where they occur Errors can be localized Details are sufficient to fix issue Ability to send diag data to dev

Logging Module Log levels (VERBOSE, INFO, WARNING, ERROR, CRITICAL, TIME) Define CRITICAL/ERROR levels well and use them! These should result in urgent notifications Lower levels in Production Time, App, Component, ThreadId, Message

8/2/2010 14:22pm: Notifier: Pop3: 8242: Sending welcome email to userid 334

Delimited columns for import Rotation based on Time, Size

Built-in Self Test Inserting test code/interfaces Set/Reset to bring state to known value Reporting to get current state Assertions about values/state

assert(order.billed==true)

“oracle” based regression

Some Examples Web Application

Software As A Service Browser Only UI Ajax / Dynamic HTML

Banking (Client-Server) Installed Application Desktop Client Server API

Web Application HTML Page Title Id values for important divs/controls Hidden values (non-textual, graphs, tables) Measurable Ajax responses Tools: Selenium, Sahi, Watir, WebTest, curl Logging incl. browser/ip/session cookie Ability to simulate time zone, language

Web Application - 2 Bypassing Captcha Mock TP APIs eg. Facebook, Google, OpenId SSL bypassing Transactions to be voided Multiple runs Always initialise controls

Browser autofill

Desktop Unique identifiers to GUI controls Key Replay tools, SendKeys API test suites bypassing UI Client logs, ability to email Ability to run db queries in scripts

Desktop - 2 Automation Friendly Third Party Controls

Ability to select cell Copy-enable text fields Key shortcuts to Forms UI for control focus,

clicking, navigation Access by value (tree/list controls)

OCR for images/text consoles

Bypassing Credit card payment

Test cards Mock object Dev auto-approval bypass code

Order placement Dummy users, auto-fulfill Dummy vendors (email order) Admin screens move

Dates Never use system date directly Config override

<?xml version="1.0" encoding="utf-8" ?>

<configuration>

<appSettings>

<add key=“SystemDate" value=“$now"\>

</appSettings>

</configuration>

Nant: XMLPoke<xmlpoke file="App.config"

xpath="/configuration/appSettings/add[@key = 'SystemDate']/@value"

value="1-1-2009" />

Sending Emails Email subsystem should be a stub in dev Bypass SMTP: log to file … Email should be sent only to a whitelist Use Gmail Id & Atom API Use Unique Id/Timestamp to distinguish

Mocking / Stubbing Replace module during testing Different kinds:

Sub-system (Email, Print) Webservice (Credit Card, Flight Booking) Hardware device (Biometric, CNC Machine)

Return default values, implement simple/fixed logic

Mocks - 2

Simulate Errors and Exceptions Provide more logging Jmock, Dependency Injection Con:

Code level Needs to change with API

Anti-Testability Viewpoints Security compromise by testability interfaces,

logging

Remove or lock down in prod Extra coding time

Much lower testing costs, better quality Privacy issues in logging

Show only partial data Performance

Almost never an issue in practice!

Review BIST

Automation suite w/ oracle

Unit Tests Assertions

Fault Simulation Mock/Stub Invalid params

Controllability Decoupling Bypassing Mock/Stub Set/Reset

Observability Logging Reporting