Testing – the Key to an Efficient and Predictable Software...

12
Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008 (c) AdNovum Informatik AG 1 Testing – the Key to an Efficient and Predictable Software Engineering Process Dr. Tom Sprenger CIO AdNovum Informatik Dipl. Informatik-Ing. ETH [email protected] May, 28 th 2008 @ ETH Zürich 2 AdNovum = Engineering Quality Core Competence High-End Software & Security Engineering Customers Financial services, Banks, Private banks, Swiss Post, Government, and Insurance Key Data Est. 1988 170 employees, 70% graduate engineers Zurich (HQ), Bern, Budapest (Hungary) Strategic Partners Sun, BEA, Swiss Post, HP, IBM, Oracle, SafeNet, Siemens Switzerland, Vasco

Transcript of Testing – the Key to an Efficient and Predictable Software...

Page 1: Testing – the Key to an Efficient and Predictable Software ...archiv.infsec.ethz.ch/education/ss08/Software... · Testing – the Key to an Efficient and Predictable Software Engineering

Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008

(c) AdNovum Informatik AG 1

Testing – the Key to an Efficient and Predictable Software Engineering Process

Dr. Tom Sprenger CIO AdNovum Informatik Dipl. Informatik-Ing. ETH

[email protected]

May, 28th 2008 @ ETH Zürich

2

AdNovum = Engineering Quality

Core Competence   High-End Software & Security Engineering

Customers   Financial services, Banks, Private banks, Swiss Post, Government, and Insurance

Key Data   Est. 1988   170 employees, 70% graduate engineers   Zurich (HQ), Bern, Budapest (Hungary)   Strategic Partners

  Sun, BEA, Swiss Post, HP, IBM, Oracle, SafeNet, Siemens Switzerland, Vasco

Page 2: Testing – the Key to an Efficient and Predictable Software ...archiv.infsec.ethz.ch/education/ss08/Software... · Testing – the Key to an Efficient and Predictable Software Engineering

Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008

(c) AdNovum Informatik AG 2

3

Agenda

  Starting Position

  Motivation

  Challenges in Practice

  Quality Assurance System

  Constructive Measures   SW-Engineering Process   Technology Management   Risk Analysis   Testing Toolbox

  Analytical Measures   Automatic Testing   Continuous Testing (NightlyBuild)

  Q&A

4

Starting Position

Page 3: Testing – the Key to an Efficient and Predictable Software ...archiv.infsec.ethz.ch/education/ss08/Software... · Testing – the Key to an Efficient and Predictable Software Engineering

Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008

(c) AdNovum Informatik AG 3

5

Motivation

  Improving quality of the results while reducing risk

  Improving efficiency while keeping high quality

 Manage distributed development

Near-/Off-shoring

  Provide periodical information points for project management

Controlling

  Provide measurements for assessing improvement efforts

 Well established Testing/QAE is a differentiation in the market

6

Challenges in Practice

  Time/Budget vs. Quality/Testing

  Functionality vs. Quality/Testing

⇒  Testing/QA must be part of the offer and the development process!

  Problem of testing today: “80% of the time is used to build up a suitable test environment; only 20% of the time remains to actually write and execute tests”

⇒ Establish an efficient testing infrastructure and make use of templates and blueprints

  Tests must be 100% repeatable and reproducible

⇒ Prerequisite for efficient testing is a well defined software engineering process

Page 4: Testing – the Key to an Efficient and Predictable Software ...archiv.infsec.ethz.ch/education/ss08/Software... · Testing – the Key to an Efficient and Predictable Software Engineering

Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008

(c) AdNovum Informatik AG 4

7

Quality Assurance System

8

Constructive Measures: SW-Eng Process

The core process 1.  Analysis & design

2.  Development / implementation

3.  Test & integration

4.  Manage & optimize

Each phase defines   Roles & tasks

  Deliverables & checkpoints

Page 5: Testing – the Key to an Efficient and Predictable Software ...archiv.infsec.ethz.ch/education/ss08/Software... · Testing – the Key to an Efficient and Predictable Software Engineering

Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008

(c) AdNovum Informatik AG 5

9

Constructive Measures: SW-Eng Process

What about a Quality Assurance Process?

  There is NO such thing as a QA process   Avoiding the risk of working for the QA process (instead of SW-Eng Process)

  Only process is the Software Engineering Process

 We follow an “Aspect-oriented QA-Approach"   define Q-Aspects along the Software Engineering Process   keep the focus on producing software   enable the developers for QA   QA is an integral part of engineering software

  Each Q-Aspect has one or more Q-Metrics attached   Q-Metrics can be verified automatically

10

Q-Aspekt: Risk Assessment

Q-Aspekt: A&S Sign-off

Q-Aspekt: Code Metrics

Q-Aspekt: Test Coverage

Q-Aspekt: A&S Sign-off

Aspect-Oriented QA-Engineering

Page 6: Testing – the Key to an Efficient and Predictable Software ...archiv.infsec.ethz.ch/education/ss08/Software... · Testing – the Key to an Efficient and Predictable Software Engineering

Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008

(c) AdNovum Informatik AG 6

11

Constructive Measures: Tech. Management

Software tool set   3 levels   Strategic   Tactical   Operative

Central information system   Explicitly show dependencies

  Development from multiple locations

  Expert know-how

  Ability to anticipate

  Cost control

Condition: automation

12

Constructive Measures: Risk Analysis Matrix

Page 7: Testing – the Key to an Efficient and Predictable Software ...archiv.infsec.ethz.ch/education/ss08/Software... · Testing – the Key to an Efficient and Predictable Software Engineering

Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008

(c) AdNovum Informatik AG 7

13

Constructive Measures: Testing Toolbox I

Test Type Tool Description

Source Code Metrics PMD

JDepend

Checks Java code against a defined set of rules to detect common errors.

Generates design quality metrics for each Java package to measure the quality of a design in terms of its extensibility, reusability, and maintainability.

Functional Testing JUnit

Cactus

JTestCase

Tests units of Java code.

Simple test framework for unit testing server-side Java code (Servlets, EJBs, ...).

Enables a systematic separation of test code and test data.

14

Constructive Measures: Testing Toolbox II

Integration Testing HttpUnit

Selenium

Emulates basic browser behavior to test form submission, JavaScript, basic HTTP auth., etc.

Used for Ajax web sites or extensive JavaScript functionality.

Code Coverage Emma Discovers which parts of the code have not been executed during the tests.

Load Testing Sams / Grinder

Proxy Sniffer

Tests your system with heavy load, re-using your integration tests.

Load generator that supports customized load and stress testing of web applications.

Profiling JProbe Detects performance problems and memory leaks in Java code.

Page 8: Testing – the Key to an Efficient and Predictable Software ...archiv.infsec.ethz.ch/education/ss08/Software... · Testing – the Key to an Efficient and Predictable Software Engineering

Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008

(c) AdNovum Informatik AG 8

15

Analytical Measures: Automatic Testing I

  Automatic Testing is your friend!

  Prerequisite

  Homogenous environment over all projects in your company   Environment supports script-based execution (no user interaction

required)   Make consistent use of versioning (source and 3rd-party components) ⇒ These are the minimum requirements for reliable, reproducible builds

  Defined project structure   Defined set of mandatory build targets (e.g. init, build, test, etc.) with   Well-known semantics

  Well-known result/output

16

Analytical Measures: Automatic Testing II

  (Automatic) testing starts with the specification   Risk Analysis (minimize effort, maximize benefits)   Use Cases   Requirements (services, test data, etc.)

  Make use of Testing Toolbox (JUnit, HttpUnit, Selenium, Grinder, ...)

  Distinguish different aspects: functional, performance, load testing   Use clever approach to enable reuse of functional tests for load testing

  Distinguish between test scenario and test cases   test scenario test code   test case # test data

Separate test data from test code and enable reuse of test code

  Define clear and reasonable quality and testing goals!

Page 9: Testing – the Key to an Efficient and Predictable Software ...archiv.infsec.ethz.ch/education/ss08/Software... · Testing – the Key to an Efficient and Predictable Software Engineering

Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008

(c) AdNovum Informatik AG 9

17

Analytical Measures: NightlyBuild I

  Continuous Integration Testing Framework

  Builds all our active projects every 24 hours   Repository checkout of the project head (latest version)   Verification of source code quality (code-style metrics, pattern detection, etc.)

  Complete build of the entire project   Start-up of test environment

  Project deployment   Execution of tests (e.g. source quality, unit, integration, and load tests)   Pull-down of test environment

  Reporting

  Statistics:   Number of Builds: 292   Number of Jobs: > 5000   Number of Platforms: 15   Number of Hosts: 29

18

Analytical Measures: NightlyBuild II

Page 10: Testing – the Key to an Efficient and Predictable Software ...archiv.infsec.ethz.ch/education/ss08/Software... · Testing – the Key to an Efficient and Predictable Software Engineering

Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008

(c) AdNovum Informatik AG 10

19

Analytical Measures: NightlyBuild III

: :

20

Analytical Measures: NightlyBuild IV : :

Page 11: Testing – the Key to an Efficient and Predictable Software ...archiv.infsec.ethz.ch/education/ss08/Software... · Testing – the Key to an Efficient and Predictable Software Engineering

Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008

(c) AdNovum Informatik AG 11

21

Analytical Measures: Application Web

  Display project dependencies as a graph

  Investigate critical project dependencies

  Get information about   Software versions in use   Unused versions   …

  Toolbox information

22

Quality Assurance Engineering Team

  Project Support   Testing opportunities consulting   Setup of testing infrastructure

  OpenSource Tool Productizing   JDepend integration   Emma extensions   HttpUnit & Selenium extensions

  Meta-Software-Engineering   Autom. architecture validation   Autom. risk-based code coverage   AQUARIUS - Quality and Risk Unit

Sensing

  Customer Projects

Page 12: Testing – the Key to an Efficient and Predictable Software ...archiv.infsec.ethz.ch/education/ss08/Software... · Testing – the Key to an Efficient and Predictable Software Engineering

Testing – the Key to an Efficient and Predictable Software Engineering Process 28.05.2008

(c) AdNovum Informatik AG 12

23

Questions?

[email protected]