SOFTWARE TEST - IDAida.dk/sites/default/files/software_test.pdf · SOFTWARE TEST – ... 2.6 ASP...

SOFTWARE TEST –

fra ”øv!” til ”yes!”

IDA Embedded 07.04.2015

Troels Fedder Jensen, ASE

[email protected]

-2-

Today’s menu

• Introduction

• What and why?

• Unit tests and frameworks

• Dependencies – design for test

• State and interaction-based tests

• Isolation frameworks

• Test quality, test design

• Continuous integration

• (TDD)

-3-

What is software test? Or what should it be?

A systematic and objective verification of software’s state and behavior, based on objective and predefined criteria.

-4-

Software test activities – the V model

System specification

System design

Component design

Integration testing

System testing

Accept testing Requirements

Implement component

Unit test

-5-

Why test software?

Increased product quality

(More) predictable development

Increased customer satisfaction

Ensure easy code refactoring/expansion

Lower overall cost

Drives ”nice” design

Reassurance

Measure of progress

-6-

Why test software?

Barry Boehm, “Equity Keynote Address” March 19, 2007.

-7-

Specification

Design

Implementation

Test

Tests provide feedback Feedback drives improvements

Why test software?

-8-

Lots of tests provide lots of feedback Lots of feedback drives lots of improvements

Specification

Design

Implementation

Test

Why test software?

-9-

Specification

Design

Implementation

Test

Specification

Design

Implementation

Test

Increased product quality

(More) predictable development

Increased customer satisfaction

Reassurement

Eases code refactoring/expansion

Lower overall cost Drives ”nice” design

Instant confirmation

Measure of progress

-10-

Test often

Test enough

Test early

Unit tests &

Unit test frameworks

System specification

System design

Component design

Integration testing

System testing

Accept testing Requirements

Implement component

Unit test

-12-

Definition of a unit and unit test

• A unit test consists of a number of individual test cases for a ”suitably” small, coherent collection of software

• Test cases are usually organized in test fixtures

• Each test case asserts the correct function of a single aspect of the unit

• Test cases should be fast, automated, repeatable, permanent (and a whole bunch of other things)

-13-

Example: The Circle class

// An application class

class Circle

{

private double _radius;

public Coordinate Center { get; set; }

public Circle(double radius)

{

(_radius < 0.0) throw new ArgumentException();

_radius=radius;

}

public double getArea() { return _radius*_radius*PI; }

public double getCircumference() { return 2*PI*_radius; }

}

-14-

A hand-coded unit test of Circle

// Test fixture for the Circle class

public class CircleUnitTest

{

private Circle uut; // The Unit-Under-Test (UUT)

// Test setup

public void setupTest() { uut = new Circle(3.0); }

// Test cases

public bool testArea() { return (uut.getArea() == 3.0*3.0*PI); }

public bool testCircumference() { return (uut.getCircumference() == 2 *PI*3.0); }

...

// Test runner

public bool runTests()

{

setupTests();

bool testResult = true;

// Collect result from test cases

testResult &= testArea();

testResult &= testCircumference();

...

// Report test result

return testResult;

}

}

-15-

Requirements for a UT framework

googletest

2.1 ABAP 2.2 ActionScript / Adobe Flex 2.3 Ada 2.4 AppleScript 2.5 ASCET 2.6 ASP 2.7 BPEL 2.8 C 2.9 C# 2.10 C++ 2.11 Cg 2.12 CFML (ColdFusion) 2.13 Clojure 2.14 Cobol 2.15 Common Lisp 2.16 Curl 2.17 Delphi 2.18 Emacs Lisp 2.19 Erlang 2.20 Fortran 2.21 F# 2.22 Groovy 2.23 Genexus 2.24 Haskell 2.25 Haxe 2.26 HLSL 2.27 ITT IDL 2.28 Internet 2.29 Java 2.30 JavaScript 2.31 Lasso 2.32 LaTeX 2.33 LabVIEW 2.34 LISP 2.35 Logtalk 2.36 Lua 2.37 MATLAB 2.38 .NET programming languages 2.39 Objective-C 2.40 OCaml 2.41 Object Pascal (Free Pascal) 2.42 PegaRULES Process Commander 2.43 Perl 2.44 PHP 2.45 PowerBuilder 2.46 Progress 4GL 2.47 Prolog 2.48 Python 2.49 R programming language 2.50 Racket 2.51 REALbasic 2.52 Rebol 2.53 RPG 2.54 Ruby 2.55 SAS 2.56 Scala 2.57 Scilab 2.58 Scheme 2.59 Shell 2.60 Simulink 2.61 Smalltalk 2.62 SQL and Database Procedural Languages 2.62.1 SQL 2.62.2 MySQL 2.62.3 PL/SQL 2.62.4 IBM DB2 SQL-PL 2.62.5 PostgreSQL 2.62.6 Transact-SQL 2.63 Swift 2.64 SystemVerilog 2.65 TargetLink 2.66 Tcl 2.67 TinyOS/nesC 2.68 TypeScript 2.69 Visual FoxPro 2.70 Visual Basic (VB6.0) 2.71 Visual Lisp 2.72 XML 2.73 XSLT 2.74 Other

https://code.google.com/p/googletest/

-16-

NUnit sample test cases

[TestFixture] public class CircleUnitTests { public Circle _uut; // Unit-under-test [SetUp] public void SetUp() { _uut = new Circle(3.0); } [Test] public void Circle_GetArea_CorrectAreaCalculated() { Assert.That(uut.GetArea(), Is.EqualTo(3.0 * 3.0 * PI); } [Test] public void Circle_GetCircumference_CorrectCircumferenceCalculated() { Assert.That(uut.GetCircumference(), Is.EqualTo(2.0 * 3.0 * PI); } [Test] public void Circle_CtorNegativeRadius_ArgumentExceptionThrown() { Assert.Throws<ArgumentException>(uut = new Circle(-1.0)); } }

Easy addition of new tests

Setup/teardown

Good assertion constructs

Test for exceptions

Test automation

-17-

Demo 1: Unit tests for Calculator Add()+Sub()

• Setting up projects for application and unit tests

• Unit test definitions – Test fixtures (setup, teardown)

– Test cases

• Running the unit tests, checking results – NUnit GUI

– IDE-integrated tool

Dependencies –

designing for test

-19-

Introducing the Drone example

0⁰

90⁰

180⁰

270⁰

-20-

Dependencies

• Objects seldom work alone

– Objects have relations dependencies DroneCtrl needs references to interact with Motor and

Gyro objects DroneCtrl depends on Motor and Gyro

class DroneCtrl { private: list<Motor*> motors; Gyro* gyro; public: DroneCtrl() { for(…) motors.Add(new Motor()); gyro = new Gyro(); } … }

DroneCtrl Motor4

Gyro

+ GetDriftDirection(): uint+ GetDriftSpeed(): uint

+ SetSpeed(s: double)+ GetSpeed(): double

-21-

Dependencies

• Options for testing DroneCtrl: – Wait for Gyro and Motor classes to be implemented?

– Pollute dependencies with #ifdef TEST? (don’t tell me you haven’t!)

– Or…make design a bit (much!) more flexible with a couple of seams

class DroneCtrl { private: list<Motor*> motors; Gyro* gyro; public: DroneCtrl() { for(…) motors.Add(new Motor()); gyro = new Gyro(); } … }

-22-

Dependencies - seams

DroneCtrl IMotor4

Motor

IGyro

Gyro



For the price of two interfaces and a constructor injection, you get… + adherence to OCP + DIP + lower coupling + facilitation of code re-use + control of the dependencies!

-23-

Dependency injection

public class DroneCtrl { private IGyro _gyro; private List<IMotor> _motors; public DroneCtrl(IGyro gyro, List<IMotor> motors) { _gyro = gyro; _motors = motors; } }

// DroneCtrl declaration in test [SetUp] void droneCtrlUTSetup() { IGyro* gyro = new FakeGyro(); List<IMotor> _motors; for(…) motors.Add(new FakeMotor()); DroneCtrl uut(motors, gyro); }

// DroneCtrl declaration for real int main() { IGyro* gyro = new Gyro(); List<IMotor> _motors; for(…) motors.Add(new Motor()); DroneCtrl uut(motors, gyro); }

Dependency injection in constructor – DroneCtrl remains oblivious to concrete type of dependencies

DroneCtrl IMotor4

IGyro

Gyro



MotorFakeGyro FakeMotor

State- and interaction-based tests -

know what you’re testing for!

-25-

State and interaction-based tests

• New functionality to our drone: Power-On Self-Test (POST)

DroneCtrl::DoPOST()

Will succeed iff gyro and all motors pass their POSTs

DroneCtrl IMotor4

IGyro

Gyro

+ DoPOST(): bool+ GetDriftDirection(): uint+ GetDriftSpeed(): uint

+ DoPOST(): bool+ SetSpeed(s: double)+ GetSpeed(): double

FakeGyro MotorFakeMotor

+ DoPOST()+ State {private set}

Idle

Error

[POST OK]

[else]

stm DroneCtrl

InitDoPOST/

-26-


• Some interesting test cases for DroneCtrl: 1. If the gyro and all motors pass POST, DroneCtrl passes POST

2. If the gyro or one of the motors fail their POST, DroneCtrl fails POST

• How to test this? 1. Create IGyro and IMotor fakes that pass (or fail) POST

2. Execute DroneCtrl.DoPOST()

3. Check DroneCtrl.State()

DroneCtrl IMotor4

IGyro

Gyro





Idle

Error

[POST OK]

[else]

stm DroneCtrl

InitDoPOST/

-27-


• This is a state-based test

• State-based tests are characterized by the following: – We are acting on the Unit Under Test (UUT)

– We are asserting that the UUT obtained the desired state after the action

– The fake dependencies are there to provide return values (fakes == stubs)

DroneCtrl IMotor4

IGyro

Gyro





Idle

Error

[POST OK]

[else]

stm DroneCtrl

InitDoPOST/

-28-


• Another interesting test case for DroneCtrl: – That DroneCtrl actually does query the gyro/motors for their POST

results

• How to test this? 1. Create IGyro and IMotor fakes that pass (or fail) POST, and record

that their DoPOST() was called!

2. Run DroneCtrl::DoPOST()

3. Assert on fakes that their DoPOST was called (fakes == mocks)

DroneCtrl IMotor4

IGyro

Gyro





Idle

Error

[POST OK]

[else]

stm DroneCtrl

InitDoPOST/

-29-

Stubs vs. Mocks

• The big difference between state-based and interarction-based tests is in where you make the assertion

• Stubs / state-based test – UUT interacts with stub

– Test asserts that UUT itself assumed correct state

– Stub state-based test

• Mocks / interaction-based test – UUT interacts with mock

– Test asserts that UUT interacted correctly with dependency

– Mock interaction-based test

-30-

Demo 2: State and behaviour tests DroneCtrl::DoPOST()

• DroneCtrl (UUT) and gyro/motor interfaces

• Stubs for state-based tests – Succeeding and failing POST

– Assert on UUT

• Mocks for interactions-based tests – Recording call to DoPOST()

– Assert on mocks

-31-

Using an isolation framework

• Hand-coding fakes takes time

• We can save time by using an isolation framework which helps us “isolate” our unit under test. – C# example: NSubstitute

– C++ example: Google Mock

• The basic idea is the same as before: Create fakes from interface definitions only

-32-

Isolation framework – faking gyro and motors

using NSubstitute; using NUnit.Framework; namespace Drone.Test.Unit { [TestFixture] public class DroneCtrlUnitTest { [Setup] public void setup() { var _gyroSub = Substitute.For<IGyro>(); // Create IGyro fake var _motorSub1 = Substitute.For<IMotor>(); / Create IMotor fake ... } } }

Note: No fake class implementing the interface, no hard-coded return values, no nothin’!

-34-

Demo 4: State and behaviour tests DroneCtrl::Hover() with NSubstitute

• Checking for methods called with correct arguments: When Hover() is called, … – Gyro.GetDriftDirection() + Gyro.GetDriftSpeed() must be called

– On relevant motors, GetSpeed() must be called

– On relevant motors, SetSpeed() must be called with correct arguments

0⁰

Gyro.GetDriftDirection() -> 45 Gyro.GetDriftSpeed() -> 3

SetSpeed( GetSpeed() + 2*3);

SetSpeed( GetSpeed() – 2*3);

-35-

Other neat things in isolation frameworks

• As a stub, … – Throw exceptions

– Fail on n’th call to a method

– Return strings

– Raise events

– …

• As a mock, … – Check that the expected exceptions are thrown

– Check that events are raised

– Check that methods are called a specific number of times

– Check that methods are not called

– …

Test quality

-37-

Test quality - coverage

• Are our tests good enough? When does our test suite contain the necessary and sufficient tests?

• One measure of test quality is code coverage – A quantitative measure of how well our tests cover the application

code

– i.e. measures the quality of our tests, not our application!

• Line coverage (aka statement coverage) most used – not necessarily best, but probably easiest.

int q=4; int* p = 0; if (condition) p = &q; *p = 117;

int array[MAX]; for(int i=0; i<MAX; i++) { array[i] = i++; // ar[i] = i+1…I think }

-38-

• Make the goal 100% coverage, nothing less. – If something cannot be covered, make an informed decision to exclude

it from coverage

• If coverage is 95%, you really know nothing of the remaining 5%

• If 95% is ok, then 90% is also fine … so is 85% … naahhh – let’s make it 80%.

• Oh sh**, the customer is here, 44% is fine!

Two practical experiences

-39-

Test design – boundary value analysis

• Does 100% coverage mean that we’ve tested enough? Or have we maybe tested too much?

• Identifying the “necessary and sufficient” set of test cases can be helped along using boundary value analysis

• Boundary value: A value in the input range for which an output is expected to change in value or validity

• Examples: sign, square root, …or the Drone!

-40-

Drone drift - BVA

1

2 3

4

1,4 + 2,3 -

1 ++ 3 --

1,2 + 3,4 -

2 ++ 4 --

1,4 - 2,3 +

1 -- 3 ++

1,2 - 3,4 +

2 -- 4 ++

Boundary: 22.5

Boundary: 67.5

Boundary: 112.5

Boundary: 157.5 Boundary: 202.5

Boundary: 247.5

Boundary: 292.5

Boundary: 337.5 Boundary: 0, 360

-41-

Test design – boundary value analysis

• Test with input on either side of each boundary value for expected output

-42-

Demo 5: BVA + coverage DroneCtrl::Hover()

• Checking on either side of every boundary value

Continuous integration-

Automate, automate, automate

-44-

Continuous Integration

A software development practice where members of a team integrate their work frequently, usually each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible (…) Martin Fowler, 2006

-45-

The basic setup

Code repository

Build application + ”Green Zone” tests


Fast integration: Build application + ”Green Zone” tests

Nightly: Build application + ”Green Zone” tests + Integration tests + Code inspection + Reporting + …

Build server

Workstation 1

Workstation 2

-46-

Benefits of CI

• No ”integration hell” at the ”end” of a project

• Never more than hours from working (deployable) build

• Rapid feedback

• Several graded builds (continuous, nightly, …)

• Frequent check-ins modular code

• Metrics for code quality never out of date

• Allows thorough (nightly, non-green zone) tests

• Allows tests to be run in a true usage environment

-47-

The setup at ASE



Jenkins server

Student workstation (on VPN)

Git server

Student workstation (on VPN)

-48-

Demo 6: Continuous Integration of Drone

• Drone software updated locally

• Updates committed and pushed

• See CI run automagically

-49-

Details of the Git/Jenkins setup

• Git clone via ssh (R/W) by developers

• Git repo exposed (RO) on web server (symlink)

• Git post-update script ”kicks” Jenkins

• Jenkins clones project via http (RO)

-50-

Take-homes

• Software testing is worthwhile – Money-saver (when after the break-even point)

– A reassuring activity for developers

• Software tests rock from Day 1 – Remove lots of bugs early during development, when it’s cheap

• Use the tools – Unit test frameworks, isolation frameworks, coverage tools, CI tools, …

• Automate, automate, automate

-51-

Thank you!

-52-

Used resources

• Unit test frameworks – C# http://www.nunit.org/

– C++ https://code.google.com/p/googletest/

• Isolation frameworks

– NSubstitute http://nsubstitute.github.io/

– Google Mock https://code.google.com/p/googlemock/

• Continuous integration – Git http://git-scm.com/

– TortoiseGit https://code.google.com/p/tortoisegit/

– Jenkins http://jenkins-ci.org/

http://www.nunit.org/






http://nsubstitute.github.io/

http://nsubstitute.github.io/

https://code.google.com/p/googlemock/

https://code.google.com/p/googlemock/

http://git-scm.com/

http://git-scm.com/

http://git-scm.com/

http://git-scm.com/

https://code.google.com/p/tortoisegit/

https://code.google.com/p/tortoisegit/

http://jenkins-ci.org/




-53-

Other resources

• Source code for examples – C# TBD

– C++ TBD

• Videos (Home-made ”dogma-style” videos, in Danish) – NUnit intro TBD

– Jenkins intro TBD

SOFTWARE TEST - IDAida.dk/sites/default/files/software_test.pdf · SOFTWARE TEST – ... 2.6 ASP...

Documents

Transcript of SOFTWARE TEST - IDAida.dk/sites/default/files/software_test.pdf · SOFTWARE TEST – ... 2.6 ASP...