The Total Economic Impact™ Of Compuware Topaz For Total ......Agile practices, including test...

A Forrester Total Economic Impact™

Study Commissioned By Compuware

September 2019

The Total Economic Impact™ Of Compuware Topaz for Total Test

Cost Savings And Business Benefits Enabled By Topaz for Total Test

Table Of Contents Executive Summary 1

Key Findings 1

TEI Framework And Methodology 4

The Topaz for Total Test Customer Journey 5

Interviewed Organization 5

Key Challenges And Opportunities 5

Solution Requirements 6

Key Results 6

Analysis Of Benefits 7

Increased Story Point Production By DevOps Teams 7

Reduced Cost To Remediate Bugs 9

Improved Efficiency Of Automating Unit Test Setup 11

Faster Time-To-Market: Increased Fraud Prevention 12

Faster Time-To-Market: Delivering eCommerce Step-Up Authentication 13

Unquantified Benefits 14

Flexibility 15

Analysis Of Costs 17

Topaz For Total Test Annual Software License Fees 17

Cost Of Employee Time For Topaz for Total Test Planning And Integration 18

Cost Of Employee Time To Maintain Topaz for Total Test 19

Financial Summary 20

Compuware Topaz for Total Test: Overview 21

Appendix A: Total Economic Impact 22

Appendix B: Endnotes 23

Project Director:

Amy Harrison

ABOUT FORRESTER CONSULTING

Forrester Consulting provides independent and objective research-based

consulting to help leaders succeed in their organizations. Ranging in scope from a

short strategy session to custom projects, Forrester’s Consulting services connect

you directly with research analysts who apply expert insight to your specific

business challenges. For more information, visit forrester.com/consulting.

© 2019, Forrester Research, Inc. All rights reserved. Unauthorized reproduction

is strictly prohibited. Information is based on best available resources.

Opinions reflect judgment at the time and are subject to change. Forrester®,

Technographics®, Forrester Wave, RoleView, TechRadar, and Total Economic

Impact are trademarks of Forrester Research, Inc. All other trademarks are the

property of their respective companies. For additional information, go to

forrester.com.

1 | The Total Economic Impact™ Of Compuware Topaz For Total Test

Executive Summary

The mainframe has been maligned for decades, with frequent predictions

of its impending extinction. But in reality, the mainframe is still a viable

technology running highly transactional, business-critical workloads such

as credit card processing and healthcare customer data. In fact, in a

Forrester survey of 425 infrastructure technology decision makers, 82%

said their mainframe usage was staying the same or increasing.1 Forrester

analysts say testing has become a first-class citizen of development

teams. Why? Because if testing cycles don’t decrease, and manual testing

doesn’t get replaced with automated testing, DevOps will never come to

life.

Compuware’s Topaz for Total Test is a solution for DevOps test

automation on the mainframe. It automates both the creation and the

execution of unit, functional, and integration tests, saving developers hours

of manual time. Topaz for Total Test enables developers at all skill levels

to test programs and subprograms almost immediately after updating a

piece of code, bringing projects to market faster with higher quality. In

addition, Topaz speeds up testers and developers in the creation and

automation of functional tests, allowing product owners in Agile sprints to

test that code is meeting business functionality requirements.

Compuware commissioned Forrester Consulting to conduct a Total

Economic Impact™ (TEI) study and examine the potential return on

investment (ROI) enterprises may realize by deploying Topaz for Total

Test. The purpose of this study is to provide readers with a framework to

evaluate the potential financial impact of Topaz for Total Test on their

organizations.

To better understand the benefits, costs, and risks associated with this

investment, Forrester interviewed one customer, an industry-leading bank,

which has been using Topaz for Total Test since its release. The results in

this study only reflect the organization’s use of the unit testing capabilities

of the product. At the time of the study, the bank had not yet implemented

the product’s functional and integration testing capabilities.

The firm uses Topaz for Total Test to extend its DevOps practices to

mainframe unit testing, freeing up time for its mainframe developers to

develop new business functionality, delivering more in less time. For this

case study, Forrester interviewed one team at the bank using Topaz for

Total Test; the analysis is based on that team’s actual results. Prior to

using Topaz for Total Test, the interviewed customer’s testing practices

were highly manual and time-consuming, resulting in the mainframe being

a bottleneck in bringing new functionality to market.

Key Findings

Quantified benefits. The interviewed organization experienced the

following risk-adjusted present value (PV) quantified benefits:

› Increased average story point production by 233%. Prior to

implementing Topaz for Total Test, the organization had no automated

testing for the mainframe. Using Topaz for Total Test for automated unit

testing, four scrum teams of nine developers increased their story point

production from an average of 300 story points per release to an

average of over 1,000 due to the elimination of manual testing.

Key Benefits

Increased story point production by DevOps teams:

$17.8M

Faster time-to-market with business application:

$1.2M

Reduction in bugs released to integrated test environments:

83%


› Reduced bug leakage to production. When teams adopted Topaz for

Total test for automated unit testing, they experienced more than an

83% decrease in bug leakage into integration test environments. The

lead software engineer said: “Our early releases failed because there

were loads of bugs that escaped into our integration test environment.

That code just could not go live. But then by improving the tooling used

by the developers, including Topaz for Total Test, their productivity shot

through the roof.”

› Reduced time to set up unit tests by 80%. Manual setup for unit

testing is time-consuming and monotonous work, and the tests

frequently are not reusable. Topaz for Total Test reduced that burden by

creating test assets that could be stored centrally and reused by future

teams. One development team at the financial services firm reduced its

testing time from 12 to 2 hours per setup (or build) for the 180 tests they

set up annually.

› Faster time-to-market for fraud prevention project: Mainframes

support mission-critical applications that millions of bank customers rely

upon. One example the bank interviewees described where automated

unit testing delivered a direct business impact was a fraud prevention

project. By automating unit testing and testing pieces of code sooner in

the lifecycle, the bank was ultimately able to deliver smaller increments

of software with increased confidence that functionality has been tested

thoroughly. Unit testing allowed the interviewee to bring applications to

market faster, resulting in a direct impact on the business bottom line.

The fraud prevention use case described by the financial services firm

interviewees yielded yearly benefits of nearly $500,000.

› Faster time-to-market for step-up authentication project. The

financial services firm decreased its time-to-market by delivering a

project related to credit card eCommerce step-up authentication five

months faster than previously. Because of this faster time-to-market, the

team was able to launch it live before the holiday buying season.

Unquantified benefits. The interviewed organization experienced the

following benefits, which are not quantified for this study:

› Modernizing the mainframe workforce. A challenge with firms

modernizing their mainframe environments is finding and keeping a staff

with the proper skill set required to support the mainframe. Modern

toolsets address these needs by providing an Eclipse-based integrated

development environment (IDE), which helps in transitioning developers

familiar with Java environments to the mainframe environment. The

ability for entry-level developers to utilize modern tools attracts new

talent in a competitive job market. The interviewees described the

modern interface helping them to attract and hire more than 150 entry-

level developers this year alone.

› Accelerated DevOps practices that power the customer journey.

The interviewed organization has aligned its staff to deliver customer

journeys designed as a cross-functional strategic initiative. Teams

organized into “labs” are responsible for delivering end-to-end customer

journeys. Developers are working alongside testers with a common goal

of delivering functionality to the business, which ultimately produces

delighted customers.

ROI 467%

Benefits PV $21.5 million

NPV $17.7 million

Payback Less than 3 months

During the six-month pilot phase, DevOps teams increased their story point production from a low of 300 to an average of over 1,000.


› Shorter employee ramp time to productivity. Before implementing

DevOps tooling, such as Topaz for Total Test, developer ramp time was

nine months. After deploying tools like Topaz for Total Test, employees

were considered fully productive in four to five months. Not only are new

developers getting up to speed faster, but the IDE enables former

mainframe coders to reenter the space at a faster pace.

› Increased innovation from empowered developers. DevOps tools

like Topaz for Total Test allow more flexibility in developer creativity. With

the burden of manual testing removed and automation in place,

developers can spend their time and intellectual capital on creative

coding versus more mundane tasks that the tool can handle. The

financial services firm reported teams exceeded their goals and used

their renewed energy to fuel development.

Costs. The interviewed organization experienced three categories of costs

that were included in the model. They include the following risk-adjusted

PV costs:

› Topaz for Total Test software license fees. Software licensing for

Topaz for Total Test totaled a PV of $3,282,645 over a three-year period.

› Bank costs for Topaz for Total Test planning and implementation. A

team of developers worked to implement, test, and deploy Topaz for

Total Test over a period of six months. The average interviewee

employee total cost for planning and implementation time for this period

was an initial cost of $194,480.

› Maintenance costs Topaz for Total Test. The time to upgrade and

perform maintenance on the mainframe related to Topaz for Total Test

consisted of one developer’s time for 2 hours per week. In addition,

system admins updated developer PCs, requiring half an hour per month

for the first year. Total costs were a PV of $311,732 over a three-year

analysis.

Forrester’s interview with an existing customer and subsequent financial

analysis found that the interviewed organization experienced benefits of

$21,484,907 over three years versus costs of $3,788,857, adding up to a

net present value (NPV) of $17,696,050 and an ROI of 467%.

Total benefits

PV, $21.5M

Total costs PV,

$3.8M

Initial Year 1 Year 2 Year 3

Financial Summary

Payback period:

<3 months

Increased story point production by DevOps teams, $17,819,274

Reduced cost to remediate bugs,

$169,647

Improved efficiency of automating unit test setup, $1,294,718

Faster time-to-market: Increased fraud prevention, $1,236,378 Faster-time-to-market:

Delivering eCommerce step up authentication, $964,890

three-year total benefits PV

$21.5 million


TEI Framework And Methodology

From the information provided in the interview, Forrester has constructed a

Total Economic Impact™ (TEI) framework for those organizations

considering implementing Compuware Topaz for Total Test.

The objective of the framework is to identify the cost, benefit, flexibility, and

risk factors that affect the investment decision. Forrester took a multistep

approach to evaluate the impact that Topaz for Total Test can have on an

organization:

DUE DILIGENCE Interviewed Compuware stakeholders and Forrester analysts to gather data relative to Topaz for Total Test.

CUSTOMER INTERVIEW Interviewed one organization using Topaz for Total Test to obtain data with respect to costs, benefits, and risks.

FINANCIAL MODEL FRAMEWORK Constructed a financial model representative of the interview using the TEI methodology and risk-adjusted the financial model based on issues and concerns of the interviewed organization.

CASE STUDY Employed four fundamental elements of TEI in modeling Compuware Topaz for Total Test’s impact: benefits, costs, flexibility, and risks. Given the increasing sophistication that enterprises have regarding ROI analyses related to IT investments, Forrester’s TEI methodology serves to provide a complete picture of the total economic impact of purchase decisions. Please see Appendix A for additional information on the TEI methodology.

The TEI methodology

helps companies

demonstrate, justify,

and realize the

tangible value of IT

initiatives to both

senior management

and other key

business

stakeholders.

DISCLOSURES

Readers should be aware of the following:

This study is commissioned by Compuware and delivered by Forrester

Consulting. It is not meant to be used as a competitive analysis.

Forrester makes no assumptions as to the potential ROI that other

organizations will receive. Forrester strongly advises that readers use their own

estimates within the framework provided in the report to determine the

appropriateness of an investment in Compuware Topaz for Total Test.

Compuware reviewed and provided feedback to Forrester, but Forrester

maintains editorial control over the study and its findings and does not accept

changes to the study that contradict Forrester’s findings or obscure the

meaning of the study.

Compuware provided the customer names for the interviews but did not

participate in the interviews.


The Topaz for Total Test Customer Journey

BEFORE AND AFTER THE TOPAZ FOR TOTAL TEST INVESTMENT

Interviewed Organization

For this study, Forrester interviewed two employees from the same

Compuware Topaz for Total Test customer:

› Forrester interviewed the lead software engineer for mainframe

solutions and the DevOps transformation manager.

› The company is one of the largest financial services firms in its region.

› The organization is on a DevOps journey and has over 800 mainframe

developers, 100 of whom are using Topaz for Total Test. The company

has recently transformed its entire organization to orient around the

customer journey, a major reorganization.

› The company uses the entire Topaz suite of Compuware products and

has been using Topaz for Total Test since approximately December of

2017.

Key Challenges And Opportunities

The company is currently going through an extensive technology

transformation. The organization wanted to speed up software delivery

by moving away from a waterfall development approach to embrace

Agile and DevOps, including its mainframe technologies. When talking

about strategic IT budgets, the interviewees described a new way of

operating, focused on prioritizing customer outcomes versus reducing

cost savings for technology investments.

› Value stream mapping identified areas of opportunity. To identify

areas of opportunity for improvement, the financial services firm

participated in a value stream mapping exercise. Value stream

mapping is a method for analyzing the current state and designing a

future state for the series of events that take a product or service from

its ideation through to the end customer. This can be used to visually

demonstrate resource constraints around people, process, and

technology. The organization identified the group responsible for

systems of record as the slowest point in the delivery cycle, and it

identified opportunities for improvement in all three categories. To

address the identified issues, the company ran a session using existing

tooling. While this experiment delivered some change, it was painful.

This began the search for tools to create a more efficient experience

for developers.

› Manual unit testing was slow and error-prone. One of the major

bottlenecks identified was unit testing. The mainframe developers were

responsible for performing their own unit tests and struggled with a

slow, tedious, manual process, often not completing the task.

Unfinished unit testing equaled buggy code.

“Compuware has been a

fantastic partner. At the

beginning, they did free

education onsite. Initially, we

did clinic calls three times a

week where people got trained

on anything. In terms of cost,

we had our own internal cost

for the system programmers to

install software but over and

above that, no external parties

at no additional cost.”

Lead software engineer,

financial services


Solution Requirements

One group within the organization ran a six-month pilot of Topaz for Total

Test and had previously set an aspirational target of a 20% improvement

in developer productivity. During this pilot phase, the group was able to

leapfrog that 20% goal, experiencing a 40% reduction in time for unit test

data management and a more than a 50% decrease in time to perform

code reviews. After initial setup, the process was instantaneous because

it was performed automatically.

Key Results

The interview revealed that key results from the Topaz for Total Test

investment include:

› Enabled DevOps culture change. Adopting DevOps and agile

practices is the direction the financial services firm is headed. The

DevOps transformation manager said, “We had Agile coaches and we

had DevOps experts who came in and opened the team’s eyes to what

was potentially possible.” Teams adopted Topaz for Total Test and were

able to automate their unit testing, previously a set of labor-intensive

tasks, which frees up time for developers to do more exciting work.

The DevOps transformation manager continued, “What they’re doing is

fantastic, and it reflects some of the attitude change as a result of

being able to do things that they’re really proud of.”

› Reenergized workforce. Mainframe developers have long been

subject to a set of regimented technologies and processes. New

Agile/DevOps tools bring increased flexibility in the way DevOps teams

collaborate and build new capabilities. The DevOps transformation

manager said: “Automation should be used to remove some of the

tedious parts of the job. No one wants to sit there writing test cases

when they could be doing more exciting things like writing code. These

tools allow developers to get to the fun stuff over the mundane that the

computer can do for you.” The lead software engineer continued: “It’s

also getting exposure for new technologies. Because they were

energized, they were given new things to learn.”

› More work delivered by the same resources. “The gains really

spiked when developers saw the benefit of getting their unit test batch

fully automated. The team was asked at a minimum to do one quality

engineering story per team per sprint but exceeded that goal,”

described the lead software engineer. The number of story points

delivered increased over 233% using the same fixed resource team.

› Virtually bug-free. The opportunity to test smaller pieces of code

earlier in the development lifecycle returned tremendous quality gains.

Teams were able to reduce bugs found in production by over 83%.

“Early gains have to do with the

culture changes — for

example, removing the hand-

off between dev and test. No

longer is a piece of code to be

tested like a hand grenade

that gets lobbed across the

fence. Because we have

colocated teams, and because

developers and the functional

testers sit together, and we

stopped calling people

functional testers. It became

banned to described

somebody within the team as

a tester. Everybody is a dev

team member.”


financial services


The table above shows the total of all benefits across the areas listed below, as well as present values (PVs) discounted between 5 and 20%. Over three years, the interviewed organization expects risk-adjusted total benefits to be a PV of more than $21.4 million.

Analysis Of Benefits

QUANTIFIED BENEFIT DATA

Increased Story Point Production By DevOps

Teams

The most significant driver of benefits was the ability for the organization

to increase the number of story points produced per release. Story points

are uniquely defined by each team. They are estimates of effort

characterized by the amount of work, complexity, and risk required to

implement a user story. A story point estimate should include everything

involved in getting a product backlog item all the way to done.

Total Benefits

REF. BENEFIT YEAR 1 YEAR 2 YEAR 3 TOTAL PRESENT VALUE

Atr Increased story point production by DevOps teams

$3,200,736 $6,401,472 $12,802,944 $22,405,152 $17,819,274

Btr Reduced cost to remediate bugs

$68,218 $68,218 $68,218 $204,653 $169,647

Ctr Improved efficiency of automating unit test setup

$232,560 $465,120 $930,240 $1,627,920 $1,294,718

Dtr Faster time-to-market: Increased fraud prevention

$497,166 $497,166 $497,166 $1,491,498 $1,236,378

Etr Faster time-to-market: Delivering eCommerce step-up authentication

$387,966 $387,999 $388,031 $1,163,996 $964,890

Total benefits (risk-adjusted) $4,386,645 $7,819,974 $14,686,599 $26,893,219 $21,484,907


Impact risk is the risk that the business or technology needs of the organization may not be met by the investment, resulting in lower overall total benefits. The greater the uncertainty, the wider the potential range of outcomes for benefit estimates.

83% of total benefits

Before implementing Topaz for Total Test, the organization did not

conduct automated testing for the mainframe. As part of its DevOps

journey, the organization wanted to improve the mainframe developer

experience along with the ability to replicate Agile practices found in

nonmainframe environments.

One group within the organization adopted Topaz for Total Test for

automated unit testing. Using Topaz for Total test, four scrum teams of

nine developers were able to increase their story point production by an

average of 233%.

Figure 1 illustrates the number of story points per software release the

teams delivered over an 18-month time period. After piloting a selection

of applications, teams adopted Topaz for Total Test around release 8,

resulting in a spike in story point production.

› In modeling this gain in story point production, Forrester used the

figure $433 per story point reported by the interviewee. This figure is

the cost for the delivery and environments only, excluding

management and overhead costs.

› Forrester used the starting story point production number of 300 in

release 8 as the before-state starting point for the calculation.

› To measure the level of story point production after Topaz for Total Test

usage, Forrester calculated the mean of the story points per release

between releases 8 and 17, an average of 1,000. Because teams form

their own definitions of a story point, the before and after are the key

measure versus the total number.

› In Year 1, both the credit card and mortgage teams were using Topaz

for Total Test. Based on pilots running currently, Forrester confidently

modeled an increase in usage of the testing tool across development

teams to four groups in Year 2 and eight groups in Year 3,

corresponding with the licenses purchased. The benefits will increase

exponentially the more groups embrace this toolset.

› Because Topaz for Total Test is part of a larger Topaz suite from

Compuware and the increases in functionality are enhanced by tools

such as Topaz Workbench and Xpediter, Forrester attributed 55% of

the gains to Topaz for Total Test.

The lead software engineer described: “We measured a consistent

resource pool, and the graphs show the impact of improved tooling,

making the developers more efficient. The teams are now pushing more

work through because there are fewer bugs to remediate.”

An organization’s ability to experience improved story point production

will vary based on:

› Adoption. Organizations must make the commitment to embrace

these tools to see the gains.

› Culture. The ability for development and testing teams to collaborate

to allow for faster resolution to deliver a shippable piece of code.

To account for these risks, Forrester adjusted this benefit downward by

20%, yielding a three-year risk-adjusted total PV of $17,819,274.

83%

three-year benefit PV

$17.8 million


Reduced Cost To Remediate Bugs

Even as the interviewed organization increased story point production, it

was able to do so with reduced bug leakage. Figure 2 represents the

remaining bugs that were leaked to an integration test environment.

From release 8, teams adopted Topaz for Total Test for automated unit

testing and experienced over an 83% decrease in bug leakage. The lead

software engineer said, “After deploying Topaz for Total Test, the

number of bugs per story point, on average, has fallen off the cliff.”

When introducing DevOps into their environments, interviewees

reinforced the importance of organizational culture change as a key

success criterion for achieving high results. The DevOps transformation

manager said: “You don’t want to beat people up if there’s a bug. We’ve

Increased Story Point Production By DevOps Teams: Calculation Table

REF. METRIC CALC. YEAR 1 YEAR 2 YEAR 3

A1 Story point production before Topaz for Total Test

300*$433 per story point

$129,900 $129,900 $129,900

A2 Average story point production after Topaz for Total Test

1,000*$433 per story point

$433,000 $433,000 $433,000

A3 Number of software releases annually 12 12 12 12

A4 Number of teams increasing story point production

2 2 4 8

A5 Percent improvement attributed to Topaz for Total Test

55% 55% 55% 55%

At Increased story point production by DevOps teams

(A2-A1)*A3*A4*A5 $4,000,920 $8,001,840 $16,003,680

Risk adjustment ↓20%

Atr Increased story point production by DevOps teams (risk-adjusted)

$3,200,736 $6,401,472 $12,802,944


got functional test teams sitting next to the developer building the code. If

a bug is discovered in a unit test or functional test, that’s a conversation

between the developer and the tester; there is no naming and shaming.”

For the interviewed organization, the model assumes that:

› The number of story points delivered corresponds to Figure 1, 300

before Topaz for Total Test and 1,000 after.

› Per Figure 2, at release 8, the teams leaked 0.3 bugs to the integration

environment. However, by release 11, the bug leakage rate dropped to

0.05.

› The time it takes to remediate bugs is highly dependent on the size

and complexity of the code. Based on customer and Forrester data,

the model estimates an average of 4 hours of effort to remediate each

bug.





The reduction in cost to remediate bugs will vary with:

› The size and complexity of the bugs.

› The fully loaded compensation of software developers.


5%, yielding a three-year risk-adjusted total PV of $169,647.

Reduced Cost To Remediate Bugs: Calculation Table


B1 Story points per release before Topaz for Total Test

300 300 300 300

B2 Story points per release after Topaz for Total Test

1,000 1,000 1,000 1,000

B3 Bugs per story point before Topaz for Total Test

0.3 0.3 0.3 0.3

B4 Bugs per story point after Topaz for Total Test

0.05 0.05 0.05 0.05

B5 Number of releases 12 12 12 12

B6 Time to remediate bugs 4 hours per bug 4 4 4

B7 Hourly salary (fully burdened) $68 $68 $68 $68

B8 Cost to remediate bugs before TTT B1*B3*B5*B6*B7 $293,760 $293,760 $293,760

B9 Cost to remediate bugs after TTT B2*B4*B5*B6*B7 $163,200 $163,200 $163,200

B10 Percent improvement attributed to Topaz for Total Test

55% 55% 55%

Bt Reduced cost to remediate bugs (B8-B9)*B10 $71,808 $71,808 $71,808


Btr Reduced cost to remediate bugs (risk-adjusted)

$68,218 $68,218 $68,218

“The whole environment the

team operated on improved

over the test period, and

productivity shot through the

roof. And we managed to slay

bugs at the same time.”


financial services



Improved Efficiency Of Automating Unit Test Setup

A primary need for the interviewed organization was to create efficiency

in its batch work environment. Prior to using Topaz for Total Test, the

organization was performing unit test setup manually. Compuware

defines unit testing as “testing performed on the smallest testable parts

of an application, individually and independently scrutinized for proper

operation.” The creation and setup of a unit test is onerous. The test

must be created and documented while the data setup and the program

being tested must be isolated from its environment. This process can

take hours, only to be used once.

The lead software engineer said: “Before Topaz for Total Test, any unit

test was effectively thrown away and had to be repeated because the

developers would do amendments. They would write the unit test,

custom jobs, create custom data sets, prove the change, check the

change and the rest of lifecycle. Unless you give the same developer the

same test all the time, it’s disposable.”

Topaz for Total Test reduced that burden. It can virtualize external

dependencies, which makes the test repeatable by future developers.

Forrester modeled the savings interviewees reported:

› Before Topaz for Total Test, automated unit test preparation and setup

took up to 12 hours.

› Topaz for Total Test reduced the test setup up time to 2 hours, resulting

in an 83% savings.

› One group reported running 180 automated test setups per year.

› In Year 1, two business groups were using Topaz for Total Test.

Forrester modeled an increase in usage to four groups in Year 2 and

eight groups in Year 3.

The efficiencies gained by automated unit test creation will vary with:

› The number of teams using Topaz for Total Test for automated unit test

creation.

› The number of tests being performed per year.

› The fully loaded compensation of mainframe developers.



“The first time run, there is no

saving because the amount of

effort required to set up in TTT

would be the same as the

amount of effort to do the unit

test by hand. But from the

second time you visit that

piece of code, you’re entering

the 83% savings zone

because you don’t have to do

the manual testing again,

because of your reusable test

cases.”


financial services

6%

three-year

$1.3 million



Faster Time-To-Market: Increased Fraud Prevention

Mainframe applications run critical business systems that engage and

influence millions of customers. Topaz for Total Test enables the

business to get these applications to market faster by enabling IT to “shift

left” its testing to test smaller increments of software, earlier in the

process. Smaller bits of code are easier and faster to remediate, saving

time and ultimately dollars.

The bank interviewees described an example to Forrester in which it

brought a business application that prevented fraud to market much

more quickly. The business claimed early benefits were more than

$951,514. The lead software engineer estimated, “We delivered three to

four months earlier than we would have expected if we had followed a

traditional waterfall approach.”

He continued, “Because we’re delivering small increments of software,

we have more confidence what we’re delivering has been tested more

thoroughly, and we keep the blast radius of any change to a minimum.”

The downstream effects of “shifting left” play out directly in the business

bottom line. “The ability to deliver this functionality had a positive impact

on customer experience and a direct impact on bottom-line metrics,” said

the DevOps transformation manager. Because Topaz for Total Test is

part of a larger Topaz suite from Compuware and the increases in

functionality are enhanced by tools such as Topaz Workbench and

Xpediter, Forrester attributed 55% of the gains to Topaz for Total Test.

The gains from bringing an application to market faster will vary by:

› The speed at which an organization can deliver the project.

› The cost of the resources delivering the project

› The functionality and impact of the business application



Improved Efficiency Of Automating Unit Test Setup: Calculation Table


C1 Hours saved by automating test creation (from 12 hrs to 2 hrs)

10 hours 10 10 10

C2 Number of automated tests run in a year 180 180 180 180

C3 Hourly salary (fully burdened) $68 $68 $68 $68

C4 Number of teams using unit test data setup and automation

2 2 4 8

Ct Improved efficiency of unit test setup C1*C2*C3*C4 $244,800 $489,600 $979,200


Ctr Improved efficiency of unit test setup (risk-adjusted)

$232,560 $465,120 $930,240

“The fact that we’ve shifted so

much testing to the left means

that when we do integration

testing, it’s true exploratory

testing, which is new for us.

We are finding things that

regular functional testing

would not have spotted.”


financial services

6%


$1.2 million



Faster Time-To-Market: Delivering eCommerce

Step-Up Authentication

Rising customer expectations force businesses to react — with speed

but without sacrificing quality. Agile practices, including test automation,

allow the DevOps teams to be more responsive to the business. “We

added step-up authentication to our credit and debit card eCommerce

journeys. If you make an eCommerce purchase, there’s a risk engine

that kicks in the sales front end to determine whether to use step-up

authentication. We built that for the debit and credit card portfolios in four

months, which would’ve been a nine-month-plus project before. That was

really fast.”

Compounding the savings, when the business acquired another credit

card company, it was able to replicate the same functionality in a single

release. Previously, this would have been at least a three-month project.

“Creating reusable test assets buys you a lot of time when you come to

do the second pass on the application.”

Forrester modeled the savings interviewees reported:

› Before Topaz for Total Test, the step-up authentication project would

have taken nine months to complete.

› After Topaz for Total Test the time to complete the project took four

months.

› The DevOps team consisted of nine team members, like previously

described development scenarios.





The reduction in software development expense will vary with:

› The number of software development resources used.

› The complexity of the application being tested.

› The fully loaded compensation of software developers.


5%, yielding a three-year risk-adjusted total PV of $964,890.

Faster Time-To-Market: Increased Fraud Prevention: Calculation Table


D1 Reported benefits Source: Finding $951,514 $951,514 $951,514

D2 Percent improvement attributed to Topaz for Total Test

55% 55% 55% 55%

Dt Faster time-to-market: Increased fraud prevention

D1*D2 $523,333 $523,333 $523,333


Dtr Faster time-to-market: Increased fraud prevention (risk-adjusted)

$497,166 $497,166 $497,166

4%


$964,890


Unquantified Benefits

In addition to the benefits outlined above, the interviewed executives

shared other benefits that did not have specific financial implications.

Specifically, the companies benefited in the following ways:

› DevOps practices further power the empowered customer.

Organizations undertaking digital transformation and corresponding

DevOps journeys are forced to evaluate their mainframe strategies.

Forrester analyst Lauren Nelson writes, “New applications built using

modern development practices and relying heavily on automation for

deployment often connect to existing systems. Successful integration

requires that mainframe developers not only insert themselves into

DevOps processes but also leverage the same tools (such as

continuous integration and continuous delivery release tools) as their

more infrastructure-as-code brethren.”2 With this renewed attention to

the mainframe and back-end application modernization investments,

enterprises are looking to extend their DevOps practices to the

mainframe.

Faster Time-To-Market: Delivering eCommerce Step-Up Authentication: Calculation Table


E1 Number of developers on project team 9 9 9 9

E2 Months to complete step-up authentication project before TTT

9 months 9 9 9

E3 Months to complete step-up authentication project after TTT

4 months 4 4 4

E4 Months to add another business unit before TTT

3 months 3 3 3

E5 Months to add another business unit after TTT

1 month 1 1 1

E6 Monthly salary (fully burdened) $11,786 $11,786 $11,787 $11,788

E7 Dollars saved in step-up authentication project

(E1*(E2-E3))*E6 $530,370 $530,415 $530,460

E8 Dollars saved in adding another business unit

E1*(E4-E5)*E6 $212,148 $212,166 $212,184

E9 Percent attributed to Topaz for Total Test 55% 55% 55% 55%

Et Faster time-to-market: Delivering eCommerce step-up authentication

(E7+E8)*E9 $408,385 $408,420 $408,454


Etr Faster time-to-market: Delivering eCommerce step-up authentication (risk-adjusted)

$387,966 $387,999 $388,031

75,000 employees

reorganized to align to a

customer journey.


The interviewed organization underwent a massive change and

reorganized to align 75,000 employees to customer journeys. “We

removed the concept of a separate IT department and business

department; everyone works with the business. It’s part of the whole

cultural change of introducing DevOps.” The DevOps transformation

manager described how the firm created “labs” that own groups of

applications. These labs are responsible for delivering end-to-end

customer journeys. “Within the lab, we have 175 people who can do

not only the mainframe back end, but mobile apps, system of

engagement, system of record, and all the stuff that goes in the

middle.” This results in bringing business value to market faster, with

higher quality.

› Employees hit their stride. A team reported that employees were

able to perform at full capacity in a much shorter timeframe. Before

implementing DevOps tooling, such as Topaz for Total Test, employee

ramp time was nine months. After deploying tools like Topaz for Total

Test, employees were considered fully productive in four to five

months. The lead software engineer said, “We’ve got colleagues who

haven’t coded for years coming back to it and finding it actually a more

exciting place than when they left and quite frankly also finding it a lot

easier to pick up.”

› Automation leads to increased employee innovation. The lead

software engineer said: “When you look at the world of development,

there’s a lot of regimented, ‘Thou shall do this . . . in this order . . . at

this time.’ These tools [Topaz for Total Test] allow more flexibility.

Automation removes some of the mundane parts of the job and allows

you to get to the fun stuff versus the mundane that the computer can

do for you. Writing the code is more interesting than writing test data,

test cases, and unit tests.” By exposing teams to new tooling, they

were exceeding their production goals and eager to learn new skills.

The lead software engineer continued, “They actually were coding on

their own time and because they were energized and given new things

to learn.”

Flexibility

The value of flexibility is clearly unique to each customer, and the

measure of its value varies from organization to organization. There are

multiple scenarios in which a customer might choose to implement

Topaz for Total Test and later realize additional uses and business

opportunities, including:

› Putting the CI in CI/CD. CI/CD stands for continuous integration (CI)

and continuous deployment or continuous delivery (CD), a DevOps

practice groups strive to achieve. The financial services firm performs

settlements for another bank’s transactions, which have complicated

fail structures. The lead software engineer said: “The team has created

a reusable asset for their story in GitHub. At 11 a.m. every day, our

core batch applications unit test themselves. If the DevOps team has a

piece of code that they think is good enough, they check it in, and it’s

automatically unit tested at 11 o’clock in the morning. At 11:05, there

will be a big red splurge on the team dashboard, and they can go fix it.

Because we’ve automated all the unit testing, we’re monitoring our

production fast, quite earlier in our lifecycle. So, we’re moving towards

the CI, a bit of CI/CD.”

“We certainly had a couple of

people that were in danger of

being let go, who are now star

performers because they have

really been energized by the

new ways of working.”


financial services


represents an investment in additional capacity or capability that could be turned into business benefit for a future additional investment. This provides an organization with the "right" or the ability to engage in future initiatives but not the obligation to do so.

› Moving from automated batch only to supporting online systems.

The financial services firm interviewees described their software

development lifecycle and the various teams that support the current

and upcoming software releases. To promote the code past what the

organization calls the quality gateway into the system test

environment, the teams promote the code using Jenkins and require

that it survives the SonarQube check. Today, this is done by another

software provider. When it reaches the system test environment for

their batch changes, that is fully automated by a library of automated

unit tests (created by Topaz for Total Test). The lead software engineer

described: “We want to do something similar for the online system,

which is a mixture of web services and a few green-screen

applications. We are experimenting with Topaz for Total Test to

automate testing of the online system.”

› Millennials, the mainframe, and the modern IDE. A major feature of

the Compuware Topaz suite, including Topaz for Total Test, is the

modern IDE, eliminating the need to work in a green-screen setting. It’s

known that there is a growing shortage of mainframe developers, and

the financial services firm has gone directly at the problem with

creative solutions. Compuware and the bank have created university

partnerships where the vendor supplies the toolset and the bank

provides the training. In addition, they run an apprentice program for

software engineers cross-training developers in mainframe skills.

The DevOps transformation manager said: “We have flung the entire

workforce up in the air, and it’s an exciting time if you want get into IT.

The Topaz IDE, including Total for Test, are Eclipse based. So those

with an interest in software development will probably have

downloaded Eclipse, working on JavaScript or web development.”

As these entry-level developers join the program, their skeptical peers

ribbed them quipping, “Have fun with your PSTs and green screens.”

But the financial services firm interviewees told Forrester they work

alongside a seasoned IT team, using modern graphical user interfaces

(GUIs) and DevOps tools, which in some cases have allowed them to

outperform their Java Stack counterparts.

The apprentice program supported by modern tools provides an ability

to fill roles that might otherwise have been vacant. The business

benefits by an increased output of work, which might have never been

delivered with vacancies in the dev teams, and at potentially lower

starting salaries.

“New developers might think

they’re going to come and

do really sexy digital banking

on a mobile phone but

instead do really sexy stuff

on the mainframe.”


financial services

“The set of tools that they will

have available to them will

be state-of-the-art and will

look and feel like the things

they’ve used before. The

only thing that’s different is

the syntax of the language.

We’re regenerating the

workforce.”


financial services


The table above shows the total of all costs across the areas listed below, as well as present values (PVs) discounted at 5%. Over three years, the interviewed organization expects risk-adjusted total costs to be a PV of nearly $3.8 million.

Implementation risk is the risk that a proposed investment may deviate from the original or expected requirements, resulting in higher costs than anticipated. The greater the uncertainty, the wider the potential range of outcomes for cost estimates.

87% of total costs

Analysis Of Costs

QUANTIFIED COST DATA

Topaz For Total Test Annual Software License Fees

Interviewees paid Compuware software licensing fees for Topaz for Total

Test. The costs portrayed here are for Topaz for Total Test only,

excluding other Compuware products the interviewee has purchased.

The costs for implementation resources by Compuware such as

installation support and training are included as part of the license cost.

There are no additional fees for these services.

Topaz for Total Test licensing costs will vary based on the following:

› The number of developers in the organization.

› Annual, contractual license volume costs.

To account for these risks, Forrester adjusted this cost upward by 10%,

yielding a three-year risk-adjusted total PV of $3,282,645.

Total Costs

REF. COST INITIAL YEAR 1 YEAR 2 YEAR 3 TOTAL PRESENT VALUE

Ftr Topaz for Total Test annual software license fees

$0 $1,320,000 $1,320,000 $1,320,000 $3,960,000 $3,282,645

Gtr Cost of employee time for Topaz for Total Test planning and integration

$194,480 $0 $0 $0 $194,480 $194,480

Htr Cost of employee time to maintain Topaz for Total Test

$0 $184,008 $91,555 $91,555 $367,118 $311,732

Total costs (risk-adjusted) $194,480 $1,504,008 $1,411,555 $1,411,555 $4,521,598 $3,788,857

87%

three-year cost PV

$3.3 million


5% of total costs

Cost Of Employee Time For Topaz for Total Test

Planning And Integration

During the planning and integration phases of the process, the

interviewees described the following:

› One team worked consistently with the tool, but many developers

tested it from different departments.

› Compuware provided ample training and assistance during the

deployment.

› Forrester assumed that there were 10 employees working on this

project for an average of 20 hours per week. Some of the weeks were

more intense, such as travel weeks for training, and others were

lighter, but that 20 hours for 26 months was a fair average.

The costs for the proof-of-concept team will vary based on:

› The size of the team of developers involved in the planning and

deployment of the project.

› The salaries paid based on seniority or geographic location.

› The duration of the proof-of-concept phase.


yielding a three-year risk-adjusted total PV of $194,480.

Topaz for Total Test Annual Software License Fees: Calculation Table

REF. METRIC CALC. INITIAL YEAR 1 YEAR 2 YEAR 3

F1 Topaz for Total Test license fees $3,000 per user 3,000 3,000 3,000

F2 Number of licenses 400 400 400 400

Ft Topaz for Total Test annual software license fees

F1*F2 $0 $1,200,000 $1,200,000 $1,200,000

Risk adjustment ↑10%

Ftr Topaz for Total Test annual software license fees (risk-adjusted)

$0 $1,320,000 $1,320,000 $1,320,000

Cost Of Employee Time For Topaz For Total Test Planning And Integration: Calculation Table


G1 Number of FTEs 10 10

G2 Hourly rate per person $68 $68

G3 Hours working with TTT 10 employees *26 hours

260

Gt Cost of employee time for Topaz for Total Test planning and integration

G1*G2*G3 $176,800 $0 $0 $0


Gtr Cost of employee time for Topaz for Total Test planning and integration (risk-adjusted)

$194,480 $0 $0 $0

5%

three-year cost PV

$194,480


8% of total costs

Cost Of Employee Time To Maintain Topaz for Total

Test

The costs to maintain Topaz for Total Test are inclusive of the

employees who spend time updating the application and deploying it to

the licensed developers.

› The time to upgrade and perform maintenance on the mainframe

consisted of one developer working for 2 hours per month.

› Per month, one PC maintenance developer spent half an hour in Year

1 and a quarter of an hour in subsequent years updating the developer

PCs.

The costs for maintaining Topaz for Total Test will vary based on:

› The size of the organization.

› The salaries associated with the maintenance team.


yielding a three-year risk-adjusted total PV of $311,732.

Cost Of Employee Time To Maintain Topaz for Total Test: Calculation Table


H1 Mainframe maintenance: Time spent for upgrading TTT on Mainframe

Y1: 5 hours per month Y2/Y3: 2 hours per month

5 2 2

H2 PC maintenance: Developers applying maintenance for TTT

0.5 and 0.25 hours per month

0.5 0.25 0.25

H3 PC maintenance: Number of developers

400 400 400 400

H4 Mainframe maintenance: Number of required FTEs

1 1 1 1

H5 Hourly salary (fully burdened) $68 $68 $68 $68

Ht Cost of employee time to maintain Topaz for Total Test

((H1*H4*H5)+ (H2*H3*H5))*12

$0 $167,280 $83,232 $83,232


Htr Cost of employee time to maintain Topaz for Total test (risk-adjusted)

$0 $184,008 $91,555 $91,555

8%

three-year cost PV

$311,732


The financial results calculated in the Benefits and Costs sections can be used to determine the ROI, NPV, and payback period for the interviewed organization’s investment. Forrester assumes a yearly discount rate of 10% for this analysis.

Financial Summary

CONSOLIDATED THREE-YEAR RISK-ADJUSTED METRICS

Cash Flow Chart (Risk-Adjusted)

-$5.0 M

$5.0 M

$10.0 M

$15.0 M

$20.0 M

$25.0 M

Initial Year 1 Year 2 Year 3

Cashflows

Total costs

Total benefits

Cumulative net benefits

These risk-adjusted ROI,

NPV, and payback period

values are determined by

applying risk-adjustment

factors to the unadjusted

results in each Benefit and

Cost section.

Cash Flow Table (Risk-Adjusted)

INITIAL YEAR 1 YEAR 2 YEAR 3 TOTAL PRESENT VALUE

Total costs ($194,480) ($1,504,008) ($1,411,555) ($1,411,555) ($4,521,598) ($3,788,857)

Total benefits $0 $4,386,645 $7,819,974 $14,686,599 $26,893,219 $21,484,907

Net benefits ($194,480) $2,882,637 $6,408,419 $13,275,044 $22,371,620 $17,696,050

ROI 467%

Payback period < 3 months


Compuware Topaz for Total Test: Overview

The following information is provided by Compuware. Forrester has not validated any claims and does not

endorse Compuware or its offerings.

Topaz for Total Test automates the creation of unit, functional and integration tests, saving developers valuable

time that would normally be spent manually creating tests and test assets. The execution of Topaz for Total Test

unit and functional tests can also be included as part of an automated build-test-deploy process via integration

with Jenkins or Xebia Labs XL Release.

› Unit testing enables small portions of a program to be tested, as many times as necessary, to immediately

ensure the program functions as it should prior to performing more complex, time-consuming, and costly tests.

Unit tests work to isolate the program by virtualizing external calls and data access and making assertions on

internal states of the program.

› Functional testing ensures that a program runs successfully on live systems with live data just as it would in

production. Functional tests verify that the external behavior of a program meets business requirements

without any visibility into the code of the program.

› Integration testing verifies that interactions between two or more programs and their data run as expected. Like

functional tests, integration tests run on live systems with live data. Integration testing typically happens after

the individual programs in the integration test have undergone functional testing.

By creating and executing unit, functional, and integration tests of a program or a group of programs, developers

will ensure they have a regression test suite that covers all foreseeable failure scenarios.

More information can be found at https://www.compuware.com/topaz-for-total-test-automation/.

https://www.compuware.com/topaz-for-total-test-automation/


Appendix A: Total Economic Impact

Total Economic Impact is a methodology developed by Forrester

Research that enhances a company’s technology decision-making

processes and assists vendors in communicating the value proposition

of their products and services to clients. The TEI methodology helps

companies demonstrate, justify, and realize the tangible value of IT

initiatives to both senior management and other key business

stakeholders.

Total Economic Impact Approach

Benefits represent the value delivered to the business by the

product. The TEI methodology places equal weight on the

measure of benefits and the measure of costs, allowing for a

full examination of the effect of the technology on the entire

organization.

Costs consider all expenses necessary to deliver the

proposed value, or benefits, of the product. The cost category

within TEI captures incremental costs over the existing

environment for ongoing costs associated with the solution.

Flexibility represents the strategic value that can be

obtained for some future additional investment building on

top of the initial investment already made. Having the ability

to capture that benefit has a PV that can be estimated.

Risks measure the uncertainty of benefit and cost estimates

given: 1) the likelihood that estimates will meet original

projections and 2) the likelihood that estimates will be

tracked over time. TEI risk factors are based on “triangular

distribution.”

The initial investment column contains costs incurred at “time 0” or at the

beginning of Year 1 that are not discounted. All other cash flows are discounted

using the discount rate at the end of the year. PV calculations are calculated for

each total cost and benefit estimate. NPV calculations in the summary tables are

the sum of the initial investment and the discounted cash flows in each year.

Sums and present value calculations of the Total Benefits, Total Costs, and

Cash Flow tables may not exactly add up, as some rounding may occur.

Present value (PV)

The present or current value of (discounted) cost and benefit estimates given at an interest rate (the discount rate). The PV of costs and benefits feed into the total NPV of cash flows.

Net present value (NPV)

The present or current value of (discounted) future net cash flows given an interest rate (the discount rate). A positive project NPV normally indicates that the investment should be made, unless other projects have higher NPVs.

Return on investment (ROI)

A project’s expected return in percentage terms. ROI is calculated by dividing net benefits (benefits less costs) by costs.

Discount rate

The interest rate used in cash flow analysis to take into account the time value of money. Organizations typically use discount rates between 8% and 16%.

Payback period

The breakeven point for an investment. This is the point in time at which net benefits (benefits minus costs) equal initial investment or cost.


Appendix B: Endnotes

1 Source: Forrester Analytics Global Business Technographics® Infrastructure Survey, 2018

2 Source: “Tackling The Unsexy Challenge Of Mainframe Modernization,” Forrester Research, Inc., December 21, 2018.

The Total Economic Impact™ Of Compuware Topaz For Total ......Agile practices, including test...

Documents

Transcript of The Total Economic Impact™ Of Compuware Topaz For Total ......Agile practices, including test...