Automate Your Test Design Process

Get to Market Faster

Ship With Fewer Defects

Testing is one of the main processes impacting time to release and product quality. But do you realize that within testing it is test design that drives both time and quality and that you can improve this process?

Testing has two major parts: test design and test execution. Maybe you have already focused on “test automation”, but that’s only the execution side. Test execution tools won’t find any defects on their own. They are completely dependent on your test design phase.

And that’s why test design matters. It’s the distinction between having an excellent hammer and knowing where to hit.

According to research, improving your testing tools from “high maturity” to “advanced model-based testing” cuts in half the number of remaining defects.1 In a recent interview, significant software bugs were estimated to cost an average of $250,000 in lost revenue and 20 developer hours to fix a piece. Bugs discovered late almost always affect release dates.2 Casper Jones concludes, based on data from 12,500 projects, that on the average, at release, 1/5 of remaining defects lead to major or total failures, and that quality excellence returns more than $15 per every $1

1 Madachy and Boehm: Assessing Quality Processes with ODC COQUALMO. Lecture Notes in Computer Science, Making Globally Distributed Software Development a Success Story, Volume 5007, 2008, Springer Belin / Heidelberg, pp. 198–209

2 A study by Electric Cloud and Osterman Research

spent.3 Add to these Conformiq’s own benchmark results showing design pro-ductivity is consistently improved by a factor of five by switching to advanced test design methods and tools.4

Recognize that testing methods are a business decision too. Direct testing costs in a software-intensive product company are typically above 20% of the total R&D budget.

Inadequate and suboptimal testing delays product releases and causes post-release quality problems taxing R&D, customer support, services, and the entire business unit. Fixing testing at its roots, at the test design phase, can make a difference on your bottom line. What would it mean to your business if, like our customers, you could save 30% on total functional testing costs and find more defects?

See the table on the right for a partial list of typical issues and the improve-ments that can be realized by a better test design process.

3 Casper Jones: Software Quality in 2008: A Survey of the State of the Art

4 See e.g. Patil et al: Model-Based Testing for VoIP phones—an Approach to Increase Test Coverage and Productivity. ISQT STEP AUTO Conference on Software Testing, September 2010

Issue Costs and Risks Improvement Impact

Incorrect tests Lead to wrong test results. They trigger false alarms and delay shipments, or miss defects that should have been captured.

Eliminating incorrect tests reduces time to release and improves defect removal rate.

Missing tests Engineers do not test for all the cases they should because of time pressure and human error. A missing test can lead to a delivered defect.

A systematic test design approach improves defect removal rate and reduces business risk.

Redundant tests

Redundant tests are those that are not actually needed. They slow down test execution time and incur dead-weight maintenance costs for regression tests.

Eliminating redundant tests reduces time to release and testing costs without impacting product quality.

Tests hard to maintain

Tests are costly to maintain when requirements change. Maintenance burden often forces testing departments to delay testing of new features.

Reducing time spent in test maintenance enables new features to be tested earlier, reducing time to market.

Complex testing systems

Automated test execution systems are complex and consist of multiple layers. The testing systems themselves contain defects which lead to incorrect test results.

Simplifying the test execution system reduces business risk and cost.

Testing equipment cost

When tests are run on dedicated hardware, big and ineffective test sets incur high costs and cause bottlenecks when teams have to queue for available hardware.

Reducing test case set sizes enables efficient use of hardware resources, improving time to market and reducing need to invest in hardware.

Personnel cost Test design and maintenance incur heavy personnel costs. Making people more productive in test design reduces testing costs and boosts bottom-line performance.

Design time Calendar time spent in designing and updating tests slows down product release cycle.

Reducing the time required for test design reduces time to market.

Lost traceability

Test engineers do not always maintain test traceability data. This slows down analysis and debugging and breaks compliance with process standards.

Systematic traceability information improves compliance, reduces business risk, and enables faster response to defect findings.

Insufficient documentation

Missing documentation for test cases incurs high maintenance costs and continuity risk when personnel changes.

Well-documented test sets reduce maintenance time and costs and reduce business risk caused by employee churn.

Improving Test Design—An Opportunity and a Necessity

In order to design good tests, every engineer needs to understand how the system being tested should work, and how to verify that it works as it should.

Until now.

In our unique Automated Test Design™ approach, your engineers still need to under-stand how the system should work, but they share this information with their computers, and their computers do the rest of the work. It is test design outsourced—to an algorithm.

Does it work? Yes.

C++ compilers work. Software engineers do not write assembler anymore, having moved to a higher level of productivity. They design programs, and their computers translate them into assembler. But there was a day when everyone wrote assembler and no one trusted compilers.

Microchip design tools work. Microchip engineers do not layout their circuits. They design the logical chips with Verilog and VHDL languages, and their computers translate them into physical layouts. But there was a day when the circuits were laid out by hand.

GPS navigators work. You decide where you want to go, and your navigator computer designs the details of your route. Do you remember the time when these navigators did not exist?

Does Automated Test Design™ work? Yes. Today, its users do not code tests by themselves, having moved to a higher level of productiv-ity. They design models, and their computers translate their models into tests. But there was a day when they did not believe it could work.

In the Conformiq solution, the engineer’s understanding of correct system operation is put into a “model”. In practice it is a computer program that mimics some aspects of the real

system. Both your engineers and their computers can understand it. Conformiq’s software explores the business logic and its corner and exception cases systematically and constructs the necessary tests.

This algorithmic approach eliminates randomly incorrect tests.

There are fewer missing tests, because the algorithm does not acciden-tally miss corner cases.

There are fewer redundant tests because the resulting test sets are optimized by a computer and checked for importance.

Test maintenance is easier because half of the job is automated.

Test execution systems can be simplified, because they are no longer targeted towards human-written tests, but computer-written tests.

Testing equipment costs decrease because the optimized test sets run faster on leaner test execution systems.

Personnel cost can be optimized because the approach delivers a signifi-cant productivity improvement.

Test design time shrinks because the algorithms do the design faster than humans.

Traceability improves as it is now automatically maintained by a computer.

Test documentation is always consistent and up-to-date as it’s generated at the same time as the actual tests.

We offer to transform your test design process so that it enables your products to get to market faster, with fewer defects. Our solution consists of our unique Conformiq Designer™ Automated Test Design™ tool, and services according to your needs.

Why change and why do you need us? Our proven results from customer engagements: 80% cost reduction in test design, leading to 30% cost reduction in functional testing; at the same time, improved test coverage leading to higher defect removal rate and less customer-found defects. It is a significant time, cost and quality advantage that is mostly indepen-dent of your other process improvement initiatives.

Pushing through a change is always a challenge. But when it comes to Automated Test Design™, we know how to make these results happen.

Your deployment would typically start with a business review between us and your product, process and budget owners, resulting in a joint business case proposal. Next, the technical feasibility of the business case is verified in one or several proof-of-concept projects, during which your engineers verify that our solution delivers what we advertise. The actual deployment starts small, typically in collaboration between your and our engineers. Initial training and coaching help to create a pilot team inside your organization. Management focus and good early results help to solidify the initiative and to grow the user base in a controlled fashion. Later an internal user group supports further efforts with our team backing you up when necessary.

The keys to your success? Desire to improve and willingness to change.

Conformiq DesignerModeling documents the functionality of your products and their components.

Models can be reused and modifi ed to support testing across product lifecycle and product families.

Automatic generation of test inputs, outputs, test oracles and verifi cation conditions ensures robust, error-free and rapid generation of tests.

Interactive reviewing of generated test cases helps engineers in optimizing their work.

Extensive options for setting testing goals enable production of test case sets with optimal focus. Multiple test design confi gurations can co-exist and can be compared side by side for maximum productivity.

Conformiq DesignerAutomatically maintained traceability data enables process compliance, reliable reporting and fast problem resolution.

Automatically calculated test case dependencies help to optimize test set execution time.

Graphical mapping of test cases back to original models helps in understanding and reviewing generated tests.

Executable test scripts can be exported in a multitude of formats, including TCL, Perl, Java, Python, C++, XML, Visual Basic, and customer-defi ned formats, ensuring compatibility with existing investments (interactive HTML shown).

Immediately executable test scripts enable fast round-trip validation of tests during a model development cycle.

Automated Test Design

Proof-of-Concept Projects

Together with your domain experts, we demonstrate how our solution works in your context and how it delivers results during your first project with us.

Training

We offer complete training packages to empower your team. Certification examinations are available on multiple skill levels.

Managed Pilot Deployment

We are ready to help you get organized in your initial Conformiq deployments, including selecting projects for best fit, selecting your teams, and orchestrating the first modeling efforts.

Coaching

If you run into a technology or process question during your de-ployment, likely we have heard it before. Our coaching staff helps you to benchmark and optimize your Conformiq deployment.

Pre-Deployment Early Deployment Maturing Deployment

ServicesC2S2Conformiq Customer Success Services

Project Implementation

We can increase your bandwidth in product delivery by having our team be part of your Conformiq Automated Test Design team. We can also take on complete testing project assignments, and by our technology guarantee timely delivery with great quality.

Test Harness Development

We can help you with developing your test harness and test execution environment and to optimize them for fit with computer-generated tests.

End-to-End Solutions

We deliver a complete end-to-end testing solution with ATD and test execution within selected segments together with our partners.

Change Management

Our knowledge and experience are available for your management when needed to help with managing change and skill set evolution.

Executive Consulting

A wider perspective helps you to align your Automated Test Design deployment with your business goals. We can help you in making informed decisions.

Proof-of-Concept Projects

Training

Managed Pilot Deployments

Coaching

Project Implementation

Test Harness Development

End-to-End Solutions

Change Management

Executive Consulting

Maturing Deployment

Take advantage of our deep experience in implementing Automated Test Design at global companies. Ranging from initial proof-of-concept projects to end-to-end solutions and executive consulting, Conformiq Customer Success Services are here to help you. Our own staff and our partners’ certified experts are ready to help and make you successful worldwide.

C2S2Conformiq Customer Success Services

Multicore and Grid Hardware Support

Conformiq Designer provides full support for utilizing multi-core desktop machines as well as network-connected grid clouds1 to maximize test generation speed.

1 Separate product option

Constraint-Based Test Data Generation

Conformiq Designer generates all test input data using a constraint-based data generation algorithm. This eliminates any manual work necessary for preparing test data tables and significantly reduces the risk of incorrect or missing test data.

Powerful Modeling Facilities

Statecharts, Java-based action language, multi-threading, object-orientation and other similar best-in-class language features enable cost-efficient and reusable models.

Support for importing models from leading third-party tools such as IBM Rational, Rhapsody, Sparx Systems Enterprise Architect, HP Quality Center, IBM RequisitePro and DOORS enable companies to continue to embrace the third-party tools that fit their strategy best.

However, no third-party tools are necessary to fully benefit from the Conformiq solution. A free modeling tool, Conformiq Modeler™, is included in our offering.

A Difference that CountsSee Why Conformiq Designer is the Leading Model-Based Automated Test Design Tool

Completely Open Output Formats

Conformiq Designer supports ALL test execution environ-ments that support scripting through providing fully custom-izable output options. A large number of different ready-made output plugins are available without charge.

Based on Eclipse®

Built on top of IBM’s leading IDE environment Eclipse® and available both as a plugin and as a stand-alone applica-tion, Conformiq Designer is an excellent match for companies embracing heterogeneous operating system and development strategies not dependent solely on Microsoft.

Cross-Platform Windows and Linux Support

Conformiq has proudly provided support for the Linux operating system, in addition to Windows variants, since 2002. It is even possible to run the tool’s front-end on a Windows desktop and the actual test generation on a Linux server. This provides maximum opportunities for leveraging existing IT infrastructure and for minimizing operating system costs.

Conformiq Designer™ Licensing

Commercial Licenses

Conformiq’s commercial licensing options enable companies to achieve significant return on investment and short payback times. Availability of both annual, pay-per-use and perpetual licenses as well as the options for named-user, team-wide or corporate-wide licenses maximize your capability to optimize your purchasing decisions to fit your budgetary and process environment.

Academic Licensing

Academic licenses are available free of charge for qualify-ing academic institutions and projects.

Open Model Licensing

Open-source projects and standardization bodies can take advantage of Conformiq’s unique Open Model License, which allows everyone who shares their models with the world (under a qualifying open-source license) to use Conformiq’s tools without charge.

Evaluation Licenses

Long-term evaluation licenses are available upon request. Short-term evaluation licenses are available directly on the company’s website.

Pricing and References

For pricing and reference information, please contact sales@conformiq.com or call one of our global sales offices.

Conformiq Designer Technical Specifications

Operating systems supported

Windows XP, Windows Vista or Windows 7 with latest service packs, 32-bit or 64-bitModern Linux distributions, 32-bit or 64-bit

Installation options Stand-alone application or Eclipse pluginDistributed installation (optional)

Minimum hardware requirements

1 GB available RAM1 CPU100 MB hard disk space

Recommended hardware specifications (or above)

4 GB available RAM1 quad-core CPU or 2 dual-core CPUs100 MB hard disk space

Modeling features supported

Java-based object-oriented action languageUML statecharts (most features)UML class diagrams (on selected tools)

Modeling tools supported

Conformiq ModelerIBM Rational Software ArchitectIBM Telelogic RhapsodyEnterprise Architect(contact Conformiq for information about supported versions)

Requirement and test management tools supported

HP Quality CenterIBM RequisiteProDOORS

Built-in test generation heuristics

System-model driven test generationConstraint-based test data generationRequirements coverageTransition coverageState coverage2-Transition (switch) coverageImplicit consumption coverageBranch coverageAtomic condition coverageBoundary value analysisAll-paths coverageAll-pairs combinatoricsFull combinatoricsCombinatorial test suite optimizationAutomatic test case namingAutomatic test case dependency analysisAutomatic traceability informationAutomatic identification of preambles and postambles

Test scripting languages supported out of the box (separate download may be required)

JavaVisual BasicTCLTTCN-3PerlPythonC/C++HTML (human readable output)XML

Established fi rst in 1998, Conformiq is a leading provider of solutions for automated test design and advanced model-based testing. We are dedicated to improving test design processes

within software-intensive product companies operating in business-, mission- and life-critical industry segments.

Conformiq Designer™ is our fourth-generation test design tool, built upon a decade of advanced basic and applied research as well as testing and test design experience.

Privately held, independent and known for our extraordinarily respon-sive customer service, we are the partner of choice for companies who are ready to step ahead of the curve.

Copyright © Conformiq Inc. and its subsidiaries 2010. All Rights Reserved. All information in this publication is provided for informational purposes only and is subject to change without notice. Conformiq, Conformiq Designer, Conformiq Modeler, Open Model Licensing and Automated Test Design are trademarks of Conformiq Inc. Other trademarks and registered trademarks belong to their respective owners.

About Us