Managed Testing Services
-
Upload
tony-davis -
Category
Documents
-
view
11 -
download
0
description
Transcript of Managed Testing Services
-
MANAGED TESTING SERVICES
Focus on your core competencies by
test outsourcing
DR DANIEL SIMON
Senior Research Manager
Daniel Simon studied Computer Science and has been working for SQS since
2005. His main responsibilities include conducting research projects for the
Group, evolving and innovating SQS service portfolio, and demonstrating its
thought leadership. His core competencies relate to outsourcing, technical
quality, KPI systems, and quality management. His range of experience covers
all sectors of the IT industry, and he is Programme Chair for the iqnite
conferences 2012 at Vienna and Geneva.
DR FRANK SIMON
Head of SQS Research
Frank Simon studied Computer Science and has been with SQS since 2001.
He is Head of SQS Research, Head of the BITKOM working stream Software
Development Processes and Tools, and a Member of the German Testing
Board. His responsibilities include thinking ahead, innovating SQS service
portfolio, evaluating hypes and trends, conducting research projects, devel-
oping services, and presenting SQS research in lectures.
WHITEPAPER
-
Managed Testing Services Page 2
tion has achieved a reasonable level of maturity
and is demanding changes towards process and
organisational modularisation, standardisation,
automation, and focusing on core business com-
petencies. If the prerequisites are not yet met,
MTS in themselves will be worthwhile to develop
an organisation towards the higher maturity level.
Since MTS are a specic instantiation of the
outsourcing concept, the current status of this
market is analysed and an outlook is given on
expected future developments for MTS: Managed
Testing Services will mainly be delivered by inde-
pendent testing companies, blurring the differ-
ence to the large full-service providers.
Based on these high expectations, this paper de-
scribes the steps towards setting up and operating
MTS, focusing on costs and benets, opportun-
ities and risks in the context of these services.
Since MTS will often be applied to already existing
testing projects, a separate section explains the
transition step which is necessary to apply MTS.
As Adam Smith outlined as early as 1776 in his
famous An Inquiry into the Nature and Causes
of the Wealth of Nations (1), progress and a
qualitative increase in productivity in society
lie in specialisation and industrialism, essen-
tially represented by the concept of division of
labour. In Smiths work, the different approaches
to manufacture pins were investigated and com-
pared, but the same principles apply to the IT in-
dustry today and reect the evolution seen over
the past decades.
This whitepaper outlines the concept of Managed
Testing Services (MTS), an outsourcing technique
for testing-related tasks across one or more pro-
jects delivering one or more applications and
systems, spanning the life cycle of software and
system development and system operation.
The paper argues the idea of IT industrialisation
as the underlying motivation for using MTS, and
shows that MTS are an important step on the indus-
trialisation road. It explains the different param-
eters that need to be taken into account when de-
ciding on IT outsourcing, and gives an overview of
important prerequisites that have to be checked
before MTS benets can be leveraged.
MTS are a logical consequence of IT industrial-
isation in modern IT environments. MTS promise
signicant savings and quality improvements due
to the effects of division of labour and specialisa-
tion, as well as exploiting location-independent
delivery of well-dened pieces of work. Neverthe-
less, MTS are most efcient when an organisa-
This section gives a short overview of the funda-
mental basics of so-called Managed Testing Ser vices
(MTS). Before a clear denition is provided, it will
be shown how both IT and innovations in IT have
evolved over the last decades, explaining that Man-
aged Testing Services are a manifestation of a more
fundamental concept. This overall idea leads to
specialisation (i.e. organisations focus on their core
competencies), division of labour (i.e. outsourcing
in different variations) and (cross-country / cross-
cultural) supply chains (i.e. global delivery).
1 MANAGEMENT SUMMARY
2 THE CONCEPT OF MANAGED TESTING SERVICES
-
Managed Testing Services Page 3
evolve according to the maturity of the busi-
ness partners, offering increased predictability
of costs as well as exibility and improved over-
all resource utilisation. A more detailed view on
these dimensions can be obtained by means of
the global delivery cube explained in the previ-
ous whitepaper.
Finally, Managed Testing Services also can be
directly linked to IT industrialisation since they
help customers focus on their core competency,
which rarely is in the testing area.
2.2 THE TERM MANAGED TESTING
SERVICES
Managed Testing Services are a combination of
managed services and testing activities. Both of
these are rened in terms of the innovation
YkX[(i[[
-
Managed Testing Services Page 4
Figure ': Global delivery cube with its dimensions
BUSINESS PROCESS OUTSOURCING
ONSI
TE
ONSH
ORE
NEAR
SHOR
E
OFFS
HORE
APPLICATION OUTSOURCING
BUSINESS VALUE-BASED
INFRASTRUCTURE OUTSOURCING
TRANSACTION-BASED
OUTTASKING
FIXED PRICEBODY LEASING
COST PLUSDIY
TIME & MATERIAL
-
Managed Testing Services Page 5
with regard to its corresponding specication.
The output will be used to remove errors from
the test asset and to assess the overall comple-
tion of development work. Before the go-live of
systems, test results can be used to evaluate the
systems readiness for use in productive envir-
onments and the risks associated with the go-
live. Testing nowadays is widely recognised as a
mandatory activity in the software development
life cycle and should be closely linked to enter-
prise risk management since testing identies
defects impeding specic risks if rolled out.
In software and systems development, testing
activities can start as early as requirements
elicitation and can span the entire life cycle
down to deployment and maintenance phases
of the software and systems. Recent surveys
conrm that testing activities are deemed most
effective when they are performed by an organi-
sational unit that is independent of the develop-
ment units. ())
There are international standards for testing
(e.g. those provided by the ISTQB), including
capability and qualication schemes. Testing is
a prime candidate for service outsourcing for
customers, not only in order to focus on core
competencies but also to exploit the advantage
of utilising independent resources. Moreover,
many organisations are already used to collabor-
ating with testing teams and will have identied
and set up their testing capacities in a proper
organisational framework. Depending on the
maturity of the organisation, tests are executed
independently from the development tasks and
report to different lines to ensure their inde-
pendence. This set-up is widely used and makes
it possible to integrate a specialised service pro-
vider with reasonable effort.
Managed services can be pinpointed by the fol-
lowing dimensions within the framework of the IT
service innovation cube:
Engagement styles: Managed services can be
applied for infrastructure outsourcing (e.g.
managed services for delivering databases),
application outsourcing (e.g. managed services
for delivering ofce suites), and even business
process outsourcing (see below, Managed Test-
ing Services).
Geographical locations: There is no need
for a particular geographical favourite since
managed services focus on output. However,
in many cases nearshore and / or offshore
resources are utilised to be able to offer
competitive pricing, if legal and regulatory
constraints do not hinder it.
Compensation models: In view of the aims
of competitive pricing and the realisation of
scaling effects, and in order to account for
the investment into the set-up of a managed
service, the overall volume of the services
purchased must be reasonably sized. In most
cases, managed services focus on repetitive
tasks suggesting a transaction-based compen-
sation model. If there is a linkage between the
delivered service and the business value, even
business value-based pricing might be possible.
2.2.2 TESTING ACTIVITIES
In the IT industry, testing refers to the process
consisting of all life cycle activities, both static
and dynamic, concerned with planning, prepara-
tion and evaluation of software products and re-
lated work products to determine that they sat-
isfy specied requirements, to demonstrate that
they are t for purpose and to detect defects. (() Typically, the input to testing is the asset under
test and the corresponding specication of the
asset under test. The output of testing is an
analysis of deviations of the asset under test
-
Managed Testing Services Page 6
Output- or transaction-based compensation
model: This makes it possible to overcome
the time & material-based models and shift
enterprise resource risks to the provider.
Thus, it simplies budget planning and
generates more budget exibility.
Leveraging scaling effects on the MTS pro-
viders side: An MTS provider specialises in
testing and he processes many similar tasks
around testing which permit setting up the
best possible t-for-purpose infrastructure
that is supported by the best-tting tools inte-
grated into an overall smooth toolchain. These
scaling effects allow for very quick testing
cycles (because usually the provider uses an
optimised tool suite for automation) and short
response times (because an MTS provider has
an established and matured process landscape
in place, suggesting dedicated processes for
incident management or reporting).
High quality of service: For the provider of
MTS, testing is a core competency. He has ac-
cess to specialists, best practices, tools and in-
frastructures to deliver high-quality services.
In addition, many service providers are able
to use the advantage of being truly independ-
ent of the development disciplines, as the
provider is not integrated into the customers
organisation (cf. Section 3.3).
Innovation improvement: Using MTS allows
customers to refocus on their core competen-
cies again. Thus, the services free up custom-
er employees capacities, improve motivation,
and generate new innovation potential.
Increased range of the service-oriented
approach: The idea of thinking in service-
oriented structures and organisations is not
limited to testing but can be expanded to HR,
marketing, sales, etc. Using MTS supports this
general approach.
2.2.3 THE DEFINITION OF MANAGED
TESTING SERVICES
In view of the preceding two subsections, a def-
in ition of Managed Testing Services might read
as follows:
Managed Testing Services (MTS) are man-
aged services for testing-related tasks across
one or more projects delivering one or more
applications and systems, spanning the life
cycle of software and system development
and system operation. The resources for test-
ing (staff, testing infrastructure, system under
test) are managed by and under the respon-
sibility of the service provider, to support the
customers business processes. Based on
transaction- or business value-based prices,
the service provider ensures scalability of the
MTS and takes care of the resource manage-
ment according to the utilisation scheme re-
quired to support the tasks.
Denition 1: Managed Testing Services (MTS)
Managed Testing Services according to this def-
inition have the following objectives (cf. (*)): Complexity reduction: Since MTS mean
outsourcing on a high level within the process
chain, they reduce the overall process and
organisation complexity on the customers
side by providing a black-box view of it.
Risk reduction: MTS providers usually take on
a high degree of risk to deliver MTS. This is
underpinned with corresponding Service Level
Agreements (SLAs) to shift testing risks to the
provider altogether.
Competitive pricing: This is typically realised
by a low onshore rate and by moving the
geographical focus of the service mainly to
nearshore and offshore locations.
-
Managed Testing Services Page 7
3 Automation: For each activity within each
process step or even for complete process
steps, the possible level of automation has
to be considered. There will be activities
which require a lot of manual interac-
tion, while others might be automated
completely.
4 Focus on core competencies: At this level,
it can be decided if MTS are possible or not.
The fundamental decision whether it makes
sense to utilise MTS can be taken if and
only if management have a conscious view
on what the core competencies of the busi-
ness are. This decision can only be taken
for identied tasks (modularisation) using
standards (e.g. by the ISTQB) and automa-
tion to a certain degree.
The decision for or against MTS can only be
made if Steps 1 to 3 are successfully completed.
Otherwise, it is not clear (i) what testing means,
(ii) how testing works, and (iii) to what degree
testing can be automated. It is important to
point out that a company on its way to Level
4 can already realise signicant benets along
the way by improving its industrialisation level.
For example, standardisation has values of its
own long before MTS can nally be used. And
these values may be utilised even without tak-
ing the last step of using MTS.
MTS volume: Due to scaling effects and ramp-up
efforts, the degree to which MTS pay off depends
on the size of the testing effort and the number
of applications subject to Managed Testing Ser-
vices. Typically, the benets of MTS are realised
in three- to ve-year engagements and are not
achieved in a single project-based liaison be-
tween customer and provider. Therefore, the cus-
tomer should be prepared to make the strategic
investment with the respective support by senior
management. For MTS, a customer needs to be in
a position to commit to a three- to ve-year plan
(often known as mid-term planning). Otherwise,
the increased risks for the service provider will
2.3 PREREQUISITES AND BENEFITS
MTS have benets for both service providers
and customers. The benets for the service
providers are transferred to the consumer, e. g.
by higher cost efciency and more competi-
tive prices compared to time & material-based
projects.
Even though the monetary and non-monetary
benets are widely recognised, it should be
noted that the exploitation of these benets
comes with prerequisites. If these prerequi-
sites are not met on either the customers or
the service providers side, the risk of false
promises and failure to achieve the objectives
is high. However, a well-dened transition pro-
cess like the one presented in Section 4 will
identify those gaps as early as possible and will
bridge them efciently.
2.3.1 PREREQUISITES
There are three different types of prerequisites
that should be fullled before leveraging MTS
benets:
Industrialised process chain: First of all, the
organisation and processes on the customers
side must be mature enough to support the
division of labour across organisational units.
This typical industrialisation state has at least
four prerequisites:
1 Modularisation: The overall business process
chain must be rened to a level where a
detailed process landscape can be identied.
This process landscape denes activities,
responsibilities, dependencies, and results.
2 Standardisation: Based on the modularisa-
tion, the process landscape as well as the
interfaces must utilise standards. Applying
those standards guarantees a specic level
of quality and makes establishment easier
due to an abundance of existing literature
and knowledge.
-
Managed Testing Services Page 8
tion, the best possible organisational set-up for
running independent tests has been chosen. Both
of these aspects will have a positive impact on
the overall effectiveness of the testing process,
and will ultimately result in a higher quality of
the applications and systems being tested (cf.
Section 3.3).
In many cases, the volume of the MTS is high
enough to justify investments to leverage cross-
project and system synergies in terms of test au-
tomation and systematic regression testing. Not
only does this increase the effectiveness of the
testing service but it also reduces response times
from the testing functions to the development or-
ganisation.
If the MTS provider can make use of offshore
testing in different time zones and the delivery
processes are set up properly, a follow-the-sun
approach of MTS delivery can be achieved which
will dramatically reduce the turnaround times of
testing. This means that the development and the
testing capabilities in a project can be used in a
24-hour schedule by following the time zones. On
the other hand, this approach requires mature
processes and a signicant amount of trust be-
tween the different parties to be successful.
From the perspective of the customer, a signi-
cant risk and management effort concerning the
resources is delegated to the service providers.
Since the latter will have different customers with
different schedules, the scalability of resources in
the project life cycle is no longer an issue for the
customer and will be managed by the service pro-
vider. Therefore, the customers internal resour-
ces can be released from non-business-related
tasks and can focus on his business core activi-
ties. He will still need to contribute and provide in-
put, but this can happen in a more systematically
structured way and thus will result in both higher
quality and less effort.
have to be compensated for by higher prices.
Openness for change: Setting up MTS in an
organisation implies the shift of labour from
internal resources (including body leasing or
outtasking) to external resources having their
own responsibilities. In pure MTS, the external
resources are even no longer visible to the
customers staff. Thus, for most customers
changes will occur that need to be managed
appropriately, and management must be
fully aware of the consequences for internal
resources and address concerns proactively.
Ideally, the customer will already have experi-
ence in the eld of supplier management and
can build upon this experience to collaborate
with the service provider.
2.3.2 BENEFITS FOR CUSTOMERS
Key benets for service consumers can be real-
ised over the run-time of the MTS agreement,
most prominently the cost savings resulting from
transparent pricing models such as output-based
pricing, as well as delegated procurement of test
infrastructure and tooling. If set up properly, cus-
tomers will see an increase in transparency of cost
with regard to their application life cycles. Note
that this does not necessarily imply lower pricing
but prices are made explicit rather than being hid-
den in various budgets. One additional benet is
provided by cost attening: usually, the ramp-up
phase needs more effort than the exe cution of
MTS. However, this peak effort is not directly in-
voiced to the customer, who instead pays a at fee
over the whole life cycle. This is a key enabler for
management as an engineering discipline.
As MTS will be delivered by service providers
having expertise in the eld of quality assurance
and testing, it can be assumed that these service
providers have highly skilled resources and mod-
ern technical environments. If the MTS providers
are independent of the development organisa-
-
Managed Testing Services Page 9
on stafng needs. The signicant sizing of MTS
allows the service provider to set up core teams
that facilitate an internal handover in case of staff
uctuations or peak workload. He also has the op-
tion to build up internal core expert teams which
will be cross-pollinated from testing expertise and
customer / domain expertise. At the same time,
resource demands from different projects (even
across different customers) can be managed on a
more exible and scalable basis.
2.3.3 BENEFITS FOR PROVIDERS
Providing MTS to a customer has a number of
advantages over time & material projects for
the service providers. Typically, MTS are aligned
with an overall strategic sourcing decision of the
service customer, and a strategic partnership is
established for delivering MTS. Strategic partner-
ship decisions usually cover a timeline of three
to ve years commitment and accordingly give
the service provider a security of revenue for his
own planning, along with an enhanced forecast
3.1 OUTSOURCING AND CLOUD COMPUTING
Outsourcing as the overall approach behind MTS
is already a well-known and established concept.
However, in the IT industry, maximum coverage
has not yet been achieved. Typical trends in out-
sourcing and how they have evolved over the last
few years as a percentage of total IT budgets are
given in Figure (, based on a study published by Computer Economics. (+)
3 MARKET CURRENT STATUS AND OUTLOOK
12 %0 % 4 % 8 %
2008
2009
2010 7.1 %
6.1 %
3.8 %
Figure (: Outsourcing ratios as a percentage of total IT budgets over the last three years
These numbers clearly demonstrate the overall
trend of IT industrialisation: a growing number of
companies use third parties to deliver IT services.
And this trend is set to increase over the next
few years. A study conducted by market research
rm PAC expects an outsourcing ratio of about
25 % by 2020 this means that one out of four
euros in the IT market will be spent on outsour-
cing. (,)
-
Managed Testing Services Page 10
Some studies refer 40 % to 50 % of the cost of
software development to testing (.), others are measuring a ratio of 1:1 for development to test
effort (/). In a recent survey conducted on the occasion of the iqnite 2011 conference, 65 % of
the participants reported spending between 20 %
and 40 % of their overall project budget on qual-
ity management and testing activities. ())
However, most of the testing budget is spent on
system integrators which contradicts the best-
practice approach of separating product develop-
ment from system testing. In a PAC study, 78 %
of the overall quality assurance budget was spent
on system integrators and only 22 % was spent
on dedicated and independent testers. ('&) This is expected to change signicantly over the next
few years, as currently 57 % of companies still do
testing on the DIY level and will need to mature in
that respect.
3.3 FULL-SERVICE PROVIDERS VS. TEST
SERVICE SPECIALISTS
In general, Managed Testing Services can be pro-
vided by two different types of providers:
Test service specialists: Specialist providers
typically offer services in the subject mat-
ter testing only. They do not offer forward
development (implementation services), they
do not design software architectures, nor do
they have any particular interest in (or benet
from) any proprietary solution compromising
their independent approach. The advantage
of those players is that they are independent
of all systems under test. According to most
known testing literature, being independent is
one of the most important prerequisites for ef-
cient testing. Usually, test service specialists
full this requirement by being an independent
test organisation (see Denition 3).
Today, outsourcing is strongly associated with the
usage of cloud services and cloud computing in
general. For the following, we will use Gartners
denition of Cloud Computing:
Public cloud computing [is a] style of com-
puting in which scalable and elastic IT-en-
abled capabilities are delivered as a service to
external customers using Internet technolo-
gies. Private cloud computing is dened as
a style of computing in which scalable and
elastic IT-enabled capabilities are delivered
as a service to internal customers using In-
ternet technologies.
Denition 2: Cloud computing (-)
Viewed from the outsourcing perspective, cloud
services are a particular type of outsourcing with
some additional constraints. Nevertheless, cloud
computings part of the outsourcing market will
increase from less than 20 % today to 70 % in
2020. (,) A direct comparison between the def-inition of MTS and the denition of cloud services
reveals a high similarity between MTS and cloud
computing because MTS are scalable and elastic
as well, most of the delivery of MTS is IT-enabled,
and the results can be provided using Internet
technologies.
This similarity suggests that MTS will have a simi-
lar rate of growth and can be delivered by means
of cloud computing.
3.2 TESTING SERVICES
The general overview of the IT market provided
in the previous section can be rened to focus on
MTS. Both academic research and industry experi-
ence report a range of gures on how much test
effort is necessary and reasonable for a project.
direct comparison between the def-
others are
In a recent survey conducted on the
-
Managed Testing Services Page 11
The main advantages of using a full-service pro-
vider are:
Company size: A broader portfolio of services
and tools allows for a bigger market enabling
a bigger company size. Typically, full-service
providers are known globally and have a large
pool of employees at their disposal.
All-inclusive single sourcing: Using a full-
service provider simplies both the process
of selecting third parties and the negotiation
step. Having a contract with a full-service pro-
vider means getting everything through one
contract, with one key-account person it is a
one-stop shop.
Technical knowledge: Introducing an independ-
ent test organisation to a new IT system costs
more time and money than using one of the
service providers developers. The full-service
provider has deep knowledge of the technolo-
gies he develops and provides, which enables
him to start working without having a clear
knowledge-transfer step.
Both independent test organisations and full-ser-
vice providers will try to move into the Managed
Testing Services market (see Section 2.3.3). How-
ever, the market will be further segmented: the
more mature an organisation is (e.g. in terms of
industrialisation), the more important attributes
like provider exchangeability, dependency reduc-
tion, and output-based pricing are going to be.
These considerations assume an explicit testing
awareness on the customers side. If it is given,
the application of MTS is a natural step, i.e. MTS
will successfully be provided by test service spe-
cialists.
Figure ) shows the total revenue generated by in-dependent test organisations in 2010.
An independent test organization is an or-
ganization, person, or company that tests
products, materials, software, etc. according
to agreed requirements. [] They are inde-
pendent because they are not afliated with
the producer nor the user of the item being
tested: no commercial bias is present.
Denition 3: Independent test organisation ('')
The key advantages of using an independent
test organisation are the following:
Independence: Independent people are more
likely to question everything, provide an
objective second opinion, identify implicit
requirements, and objectively report any nd-
ings. Different lines of reporting ensure that
no conicts of interest will interfere.
Scaling effects: Because independent test or-
ganisations are usually pure-play testers, they
test many different projects at the same time,
which allows them to set up highly sophisti-
cated toolchains and utilise a large number of
synergies between different projects.
Resource capabilities: The skills provided by a
test service specialist are clearly focused on
one single topic, i.e. testing. This guarantees
a high, coherent skill level for all employees
and simplies the exchange of resources.
Full-service providers: These providers usu-
ally offer any services relating to IT systems,
starting from collecting requirements, design-
ing system architectures, implementing the
entire system, testing the system, and nally
deploying the system into productive use.
Also administration of the systems and testing
services, including Managed Testing Services,
form part of the comprehensive service port-
folio. In many cases the full-service providers
even market their own tool platforms, comple-
menting and mixing their service ideas with
software license vending.
-
Managed Testing Services Page 12
testing approach to MTS may prove difcult. In
other words, MTS are likely to remain a side prod-
uct for full-service providers, and consequently
only about 5 % of all testing activities provided
by them will be delivered as MTS by 2015. Using
the revenue of the full-service providers listed in
the NelsonHall Report, this amounts to an overall
business size of up to 150 million of revenue for
MTS (5 % of the total market size as listed in ('()).
Consequently, the independent test organisations
will reach the same market share for MTS as the
full-service providers.
Currently, conservative estimates anticipate about
20 % to 25 % of all testing activities provided by
test service specialists to be realised through MTS
by 2015. For the independent test organisations
listed, this amounts to an overall business size of
about 121 million of revenue for MTS (25 % of
the total market size as listed above).
On the other hand, full-service providers usually
blend testing activities and package them into an
all-inclusive bundle. If the client lacks awareness
of what really is involved in testing, testing effort
and testing outputs, motivating him to change the
INDEPENDENT TEST ORGANISATIONS
163
70
40
28
20
16
14
13
13
12
11
10
9
9
8
7
7
7
6
6
5
5
5
0 20 40 60 160
SQS
Applabs*
Tescom
DNV-IT
Tesnet
Plan IT
Thinksoft
Imbus
FHG / IESE
Amsphere
MTP
Acial
ps-testware
Quality House
Maverick
QA Infotech
RTTS
Genilogix
Squerist
Experior
Immune
Assurity
KJR
Figure ) : Revenue of independent test organisations ('()
* In
20
11, A
pp
labs
was
tak
en o
ver
by C
SC
Co
mpa
ny.
m
-
Managed Testing Services Page 13
The Pre-Engagement step covers the strategic
activities a client needs to perform in order to ap-
ply MTS. The MTS Framework Design step covers
the tactical activities within each clients project
to prepare MTS application. And the MTS Opera-
tion step leverages the benets associated with
MTS. Each of these steps is rened in the follow-
ing subsections.
4.1.1 PRE-ENGAGEMENT PHASE
At the very beginning of the transition to MTS,
the customer needs to define and commit to
the vision of industrialising his IT environment
(see Figure +). On the one hand, management need to identify the strategic services and core
competencies they want to operate on their
own behalf, and on the other hand, they need
to specify the supporting non-core services to
be outsourced.
The vision to industrialise IT and make use of ser-
vice providers has to be supported by conscious
planning and budgeting decisions. A success-
ful migration to an outsourcing model demands
proper preparation of the underlying business
case. Before MTS can be deployed into the or-
ganisation, the strategic sourcing decision must
be taken and at least mid-term planning perspec-
tives have to account for handing over a com-
plete process to a service provider. The strategic
decision is a sine qua non because of the sizing
of the engagement. Typically, using a managed
service requires the investment of signicant ef-
forts into organisational maturity, and the corre-
sponding changes have to be communicated and
managed appropriately.
The nal step of the Pre-Engagement phase is to
set up the relationship to potential service pro-
viders and selecting the best supplier. For MTS,
criteria for supplier selection start with match-
ing the three dimensions from the IT innovation
cube (engagement style, geographical location,
and compensation model, cf. Section 2.1) to the
customers needs. Additional parameters to be
considered are the following:
Level of testing expertise
Independence from the development
organisation
Domain knowledge
Size of the provider
Commercial background
Track record and reputation of the service
provider
4.1 TRANSITION OVERVIEW
The MTS process is structured into three phases Pre-Engagement, MTS Framework Design, and
MTS Operation and laid out in Figure *.
Figure *: MTS process overview
1. PRE-ENGAGEMENT2. MTS FRAMEWORK
DESIGN3. MTS
OPERATION
4 MANAGED TESTING SERVICES SET-UP AND OPERATION
On the one hand, management
-
Managed Testing Services Page 14
Heterogeneous views on core business com-
petencies and unclear strategic considerations
with regard to sourcing decisions
Depending on the results from the maturity as-
sessment, different initiatives to industrialise and
professionalise IT will be launched as business
processes are set up, streamlined, and optimised.
Preparing the organisation for MTS increases the
necessity to outline the strategic vision of future
IT set-ups, and makes the corporate strategy vis-
ible to all employees.
Accordingly, the typical set-up for an IT improve-
ment towards industrialisation consists of the
four steps mentioned in Section 2.3.1:
1 Modularise the entire value chain within the
organisation.
2 Standardise the process landscape, the pro-
cesses themselves, and their interfaces.
3 Automate single processes, process interac-
tions, or process chains, wherever possible.
4 Focus on core competencies and delegate per-
ipheral services to external service providers
as managed services.
The benets of preparing for MTS are tangible for
customers when MTS are being used as a driver
for maturing the organisation. As a side effect
of the MTS Framework Design phase, customers
need to reconsider their internal processes and
organisational set-up. The maturity assessment
of processes and organisation can be leveraged
to foster clarity and transparency of core com-
4.1.2 MTS FRAMEWORK DESIGN PHASE
One of the greatest challenges when setting up
MTS lies in the degree of IT industrialisation on
the consumers side. Clearly, without an explicit
organisational set-up and mature processes, the
realisation of benets from division of labour
and the consumption of specic processes /
services from the outside are rarely possible.
Therefore, the rst step in bringing MTS to life is
the assessment of the customers current organ-
isation and its processes to evaluate their tness
for purpose (see Figure ,). The assessment typic-ally unveils a number of necessary improvements
to close the gaps and achieve the adequate level
of maturity. However, these improvements do not
only affect MTS readiness but essentially generate
added value even without using MTS.
In our experience, customers aiming at the use
of MTS (or other managed services) often over-
estimate their readiness for the consumption of
services. Typical unrecognised shortcomings are
the following:
Insufciently structured and dened pro-
cesses, tasks, and supporting IT landscape as
well as undened or unclear ownership of the
process or IT components
Insufcient deployment of process and IT stand-
ards (e.g. non-existent or incomplete process
landscape, high number of variations in business
processes, unclear and undocumented inter-
faces between processes and departments)
High proportion of manually executed pro-
cesses and technological and organisational
ruptures in process chains as well as decien-
cies in process interactions
Figure +: Pre-Engagement activities
VISION TO INDUSTRIALISE IT
STRATEGIC PLANNINGCONSIDERATIONS
(MID-TERM PLANNING)
PROVIDERPRESELECTION
The assessment typic-
-
Managed Testing Services Page 15
4.1.3 MTS OPERATION PHASE
After having set up the general MTS framework
for collaboration between the customer and ser-
vice provider, the testing activities for the indi-
vidual applications have to be transferred from
the customers responsibility to the aegis of the
MTS provider. For greeneld projects, this seems
a straightforward task whereas for existing pro-
jects there is potentially more substantial change
involved. In any case, the readiness for transfer
is ensured in the rst step, where potential gaps
are identied and closed accordingly (see Fig-
ure -). Secondly, the knowledge transfer (both from the development organisation to the MTS
provider, and from the test organisation to the
MTS provider) is initiated, and the rst iteration
of delivery is accomplished jointly. This collabora-
tive delivery is a one-off undertaking designed
(i) to ensure a successful transfer of the testing
activities to the third step (Service Operation) by
passing on comprehensive knowledge to the MTS
provider, and (ii) to sort out any issues that might
arise during stand-alone delivery to the best pos-
sible extent.
petencies and non-core supporting functions. It
is worth mentioning that through the course of
process improvement, the relationship with the
service provider can be established and the trans-
formation can be used to build mutual trust.
Finally, when readiness has been accomplished,
the customer needs to design the framework
agreements with the service provider of choice.
Utilising Managed Testing Services requires meth-
ods to monitor and control service delivery. The
challenge lies in a proper denition of the SLA:
due to the black-box view of MTS, it should focus
on the compensation model rather than on inter-
nal MTS details. In order to link the KPIs to the
compensation model and evaluate them accord-
ing to a reasonable schedule, a sound framework
needs to be dened. From the very beginning, the
denition of KPIs should take into account the
evolution of service level objectives over time to
establish appropriate productivity goals. For moni-
toring and controlling purposes, these KPIs are
utilised for an effective governance and commu-
nication structure and process, so as to be able to
react when the control limits are reached.
Figure -: MTS Operation processes
CONTINUOUS IMPROVEMENT
VERIFICATION OF TRANSFER
READINESS PERAPPLICATION
KNOWLEDGETRANSFER &
COLLABORATIVEDELIVERY
SERVICEOPERATION
SERVICECOMPLETION
Figure ,: MTS Framework Design activities
ASSESSMENT OF CURRENT
ORGANISATIONAND PROCESSES
PROCESS ANDRESOURCETUNING
FRAMEWORK AGREEMENT
WITH PROVIDER
the knowledge transfer (both
-
Managed Testing Services Page 16
laborative Delivery step, it helps determine and
ne-tune the agreed KPIs of the framework agree-
ments and can be used to adjust the SLA. While
Service Operation is repeatedly executed, Con-
tinuous Improvement ensures that the KPIs are
properly monitored and adequate action is taken
to improve and control the delivery. The same
KPIs used to report back on service performance
should also be applied to manage and govern the
MTS Operation phase. Since the benets based
on improvements affect both consumer and MTS
provider, the sharing of the benets needs to be
clearly dened in the SLA.
Out of the four MTS phases, the MTS Operation
phase is the one covering the longest time span
it implies a shift from a project governance struc-
ture towards a programme governance structure.
A stereotypical organisational set-up is shown in
Figure ..
Subsequently, the Service Operation step may be
executed as often as necessary over the lifetime
of the application, until this lifetime comes to a
close and both the application and its MTS are
decommissioned in the Service Completion step.
Since the Service Completion step is the reverse
of the transition step, planning for this step helps
limit dependency on a specic MTS provider. The
Service Completion step takes care of gathering
all information (including internal details like test
scripts, test data, or test procedures) necessary
to deliver MTS for a particular project. This allows
the customer to hand over the collateral informa-
tion to any other MTS provider, or to insource the
testing activities at a later point in time, ensuring
a shorter introduction time.
The sequence of four processes is overarched by
the Continuous Improvement of the MTS Opera-
tion phase. During the Knowledge Transfer & Col-
Figure .: Governance structures for the MTS Operation phase
PROJECT GOVERNANCE MANAGED SERVICE GOVERNANCE
EXECUTIVE STEERING COMMITTEE
SERVICE DELIVERY COMMITTEE
INNOVATION & TECHNOLOGY COMMITTEE
Test FactoryMS Delivery
Management
Applications
TestingServices
Test Team
TestingServices
Test Team
TestingServices
Test TeamFunctionalSub-Projects
TestingServices
Technical Sub-Projects
QualityAssurance
ProjectManagement
-
Managed Testing Services Page 17
proven governance structure consists of the fol-
lowing decision bodies in a strategic and an op-
erational layer:
STRATEGIC LAYER
Executive Steering Committee (ESC) Meet-
ings are held quarterly (or as described in the
framework agreement) as senior management
meetings to monitor overall MTS performance,
to govern MTS from a long-term perspective,
and to de-escalate issues arising from the
operational layer.
Innovation & Technology Committee (ITC)
The ITC meets quarterly to share the latest
industry, innovation and technical trends, in
order to decide on major innovation projects
and jointly agree on business cases and fund-
ing, if required. Its members propose changes
to the agreements to the ESC and also consult
the ESC on strategic decision-making.
OPERATIONAL LAYER
Service Delivery Committee (SDC) The SDC
performs weekly reviews of the current perform -
ance and progress, focusing on the domain,
project and service levels. It decides on prior-
ities, resolves escalation issues as best pos-
sible, and elaborates on how to mitigate risk
and issues within the scope of the services
operation. Issues outside the SDCs responsi-
bility are referred to the ESC.
Operational Status Meetings These meetings
are scheduled weekly and aligned with core
testing processes in order to review progress,
focusing on the project or application level,
and to decide on priorities and work on risks
and issues, seeking for resolution wherever
possible or referring them to the SDC.
Project Governance rests entirely with the Project
Management (which then reports to the executive
level). The Project Management function sup-
ported by the Quality Assurance function per-
forms Testing Services as per project by the same
hierarchical decomposition as the Functional and
Technical Sub-Projects. In the MTS setting, the
Testing Services liaise with the Quality Assurance
function and are managed as per application by
suppliers interfacing with the respective project
teams. The reporting lines to the executive level
are no longer purely bottom-up but complement
the internal reports with an independent view on
projects and applications from the suppliers per-
spective. This typically gives customer manage-
ment deeper insight into the project status and
increases trust in the reports.
As a general principle, the MTS Operation phase
must be supported by an open dialogue between
customer and service provider relating to the
provision of the services throughout the term. In
order to keep up with technological changes over
the run-time of the overall agreement, details of
the agreement may need to be amended, whether
as a result of market trends, new technology,
evolution of new services, or otherwise, and must
be kept under constant review, control, and sub-
ject to proactive search for improvements in the
MTS Operation phase. To facilitate the dialogue,
both partner organisations typically appoint a Re-
lationship Manager at senior management level
to cover all matters concerning the agreements.
The service provider appoints a Service Delivery
Manager as the point of contact for operational
matters affecting the services (e.g. adding new
applications to the portfolio; customer projects
ordering services).
Having dened the roles, a corresponding formal
and structured channel of escalation and com-
munication has to be set up and maintained. The
-
Managed Testing Services Page 18
4.2.1 INEFFECTIVE TESTING
If testing is ineffective, the uncertainty about the
system to be rolled out increases because nobody
knows about possible hidden errors in produc-
tion and potential impacts. Since risks are based
on uncertainties, the overall risk of an IT system
increases as well. This does not necessarily im-
ply that the system fails in production but that
it lacks the necessary transparency and reliable
insights about production readiness. Therefore,
ineffective MTS for a business-critical application
might induce additional risk.
From an MTS perspective as described above, the
following countermeasures are to be taken:
Within the transition step, many prerequisites
for MTS are veried. The transition step will
identify most of the detailed hurdles for ef-
fective testing and eliminate the respective
reasons.
Before MTS enter the operational phase, the
collaborative delivery is an explicit mitigation
activity to ensure in a dry-run that testing can
be implemented as planned.
The MTS process incorporates the denition
of an overall control framework underpinned
by supporting KPIs to make the status of the
testing process transparent to all stakehold-
ers. Therefore, uncertainties about the test
progress should be minimised.
4.2 MTS RISK MANAGEMENT
As with all other business activities, there are
risks associated with MTS. For risk, we use the
following denition, based on the concept of un-
certainty:
Uncertainty is the lack of complete certainty,
that is, the existence of more than one possi-
bility. [] Risk is a state of uncertainty where
some of the possibilities involve a loss, catas-
trophe, or other undesirable outcome.
Denition 4: Uncertainty and risk ('))
So each risk is an uncertainty but not vice versa.
Uncertainties can have both positive or negative
impacts, risks only have negative impacts on busi-
ness. Please note that this understanding of risk
is more focused than the one used in ISO 31.000
since there even positive impacts are classied as
risk.
From a risk management perspective, the focus is
on negative risks because they require mitigation
actions to deal with their impacts.
For a customer, MTS involve two different types
of risks:
Ineffective Testing: Testing does not work
properly, i.e. the SLAs for the MTS or the stra-
tegic objectives to do MTS are not fullled.
Unfullled Collaterals: Testing works but other
collaterals are uncertain. This uncertainty can
cover the economic future of the service pro-
vider or might touch the cultural acceptance of
MTS within an organisation.
Both risks are rened in the following subsec-
tions.
-
Managed Testing Services Page 19
Communication risks: In general, MTS need
more mature communication channels (no
longer gossiping) when communicating
from customer to testing. In addition, a more
mature defect management is necessary since
the organisational boundaries discourage
on-the-oor communication. And last but not
least, cultural and language issues are a major
source of misunderstanding, in particular
if cross-national and offshore locations are
involved. MTS might generate risks in this
eld, but the industrialisation steps, that are
checked on the customers side in advance, as
well as the systematic transition step minimise
them.
Dependency risks: Outsourcing is often associ-
ated with outsourcing responsibilities, i.e. test-
ing capabilities for MTS. The less knowledge is
retained on the customers side the more he
is dependent on MTS providers. Typ ical
examples of a high dependency are price
adjustments without any reason, disburdening
details of the negotiations, or low quality. In all
cases, MTS propose the well-dened Service
Completion step, ensuring that all custom-
ers can step back. Due to the detailed entry
criteria (industrialisation), the customer has a
very clear understanding of the dependency,
possible impact, and alternatives for delivery.
4.2.2 UNFULFILLED COLLATERALS
Outsourcing a well-dened service might stipu-
late uncertainty in some collateral areas. Most
of them are not solely related to MTS but to all
major changes in the context of service delivery.
In a study about risks of outsourcing (not limited
to MTS), a set of most common risks was identi-
ed by survey respondents. ('*) Most of them are valid for MTS as well:
Pricing risks: In many cases, using MTS should
have a positive effect on costs. However, the
specic costs of MTS are not ultimately known
before the rst Service Operation step. The
MTS framework provides general guidance but
only the Collaborative Delivery will determine
the nal price. From a customer perspec-
tive, a mitigation action would be to select an
experienced MTS provider whose rst effort
estimation does not need much adjustment. If
there are price adjustments while the service
is delivered, the Service Completion step en-
sures that it is possible to change the service
provider to a cheaper one.
Planning risks: If some other constraints from
MTS are not met (e.g. missed deadlines, un-
clear planning), this might generate additional
risks as well. The MTS process guarantees to
identify deviations as early as possible. In most
cases, any deviations are followed by contrac-
tual penalties compensating negative effects
by planning risks.
-
Managed Testing Services Page 20
The market for MTS will increase dramatically
over the next few years. Even if both, full-service
providers and test service specialists, will try to
do business in that area, the test service special-
ists will dominate the market in a few years time
due to many advantages in their MTS delivery.
Their process to establish MTS on the consumers
side reects a well-dened current-state analysis
as well as a corresponding transition step for MTS
readiness. Based on a continuous improvement
loop, MTS execution itself can be managed ef-
fectively. A possible closing step at the end can
reect an applications end as well as the demand
to in-house all activities again.
Today, testing is established as an important dis-
cipline for risk management. For maximum ef-
ciency, the testing activities, the methods, the
practices, tools and skills should only be of inter-
est to those who execute testing. From a business
perspective, testing should be seen as a black box
focusing testings output. So there is a great op-
portunity to outsource testing activities at the
rate of output-based pricing: Managed Testing
Services.
Applying MTS is the next step of IT industrialisa-
tion and needs a mature process landscape on
both the consumers side and the producers side.
Even if there is no need to apply MTS as a last
step on the industrialisation roadmap, the road
itself generates a considerable amount of added
value for improving testing efciency and effect-
iveness.
5 CONCLUSION AND OUTLOOK
-
Bibliographical References Page 21
1 Adam Smith. An Inquiry into the Nature and Causes of the Wealth of Na-
tions. Oxford University Press, 2008.
* Dr Wolfgang Balze, Dr Wolfgang Rebel and Peter Schuck. Outsourcing und arbeitsrechtliche Restrukturierung von Unternehmen. Verlagsgruppe Hthig
Jehle Rehm, 2007.
+ Computer Economics. IT Outsourcing Statistics 2010 / 2011: Outsourcing and Offshoring Trends, Cost/Service Level Experiences, and Analysis for 11
Outsourced IT Functions. California, Irvine: Computer Economics, 2010.
, Karsten Leclerque. Outsourcing im Jahr 2020. www.cio.de. [Online] 09/09/2010. [Cited: 19/07/2011.] http://www.cio.de/knowledgecenter/out-
sourcing/2247905/index.html.
- Daryl C. Plummer, David Mitchell Smith, Thomas J. Bittman, David W. Cear -ley, David J. Cappuccio, Donna Scott, Rakesh Kumar and Bruce Robertson.
Five Rening Attributes of Public and Private Cloud Computing. Gartner,
May 2009.
. Praveen Ranjan Srivastava. Estimation of Software Testing Effor t: An intel-ligent approach. Birla Institute of Technology and Science, Pilani, Rajasthan,
India: Computer Science and Information System Group, 2009.
/ Michael A. Cusamano and Richard W. Selby. Microsoft Secrets. New York: Simon and Schuster, 1995.
'& PAC. Compliance and Software Testing. Compliance Magazine. [Online] 04/04/2007. [Cited: 19/07/2011.] http://www.compliancemagazin.de/markt/
studien/pac040407.html.
'' Wikipedia. Independent test organization. Wikipedia, the free encyclopedia. [Online] 27/07/2011. [Cited: 27/07/2011.] http://en.wikipedia.org/wiki/Independ-
ent_test_organization.
2 International Software Testing Qualications Board, Glossary Working Party.
Standard Glossary of Terms Used in Software Testing. Homepage: ISTQB,
December 2007.
) iqnite conference. iqnite interaktiv SMS voting. iqnite germany program. [Online] 01/07/2011. [Cited: 28/07/2011.] http://www.iqnite-conferences.com/
Trial%20Wiki/iqnite_interaktiv_sms-voting.aspx.
Dr Wolfgang Balze, Dr Wolfgang Rebel and Peter Schuck. Outsourcing und
Computer Economics. IT Outsourcing Statistics 2010 / 2011: Outsourcing
Karsten Leclerque. Outsourcing im Jahr 2020. www.cio.de. [Online]
Wikipedia. Independent test organization. Wikipedia, the free encyclopedia.
-
t: An intel-
-
Bibliographical References Page 22
'( NelsonHall Report. Independent Testing Providers vs. System Integrators. Cologne: SQS, April 2010.
'3 Douglas Hubbard. The Failure of Risk Management: Why its broken and how to x it. John Wiley & Sons, 2009.
'* Diamond Management & Technology Consultants. 2006 Global IT Outsour-cing Study. Chicago: Suite 3000 John Hancock Center, 2006.
Cologne: SQS, April 2010.
to x it. John Wiley & Sons, 2009.
cing Study. Chicago: Suite 3000 John Hancock Center, 2006.
-
SQS Software Quality Systems AGPhone: +49 (0)2203 9154-0Stollwerckstrasse 1151149 Cologne / Germanywww.sqs.com
/ColorImageDict > /JPEG2000ColorACSImageDict > /JPEG2000ColorImageDict > /AntiAliasGrayImages false /CropGrayImages true /GrayImageMinResolution 300 /GrayImageMinResolutionPolicy /OK /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth -1 /GrayImageMinDownsampleDepth 2 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict > /GrayImageDict > /JPEG2000GrayACSImageDict > /JPEG2000GrayImageDict > /AntiAliasMonoImages false /CropMonoImages true /MonoImageMinResolution 1200 /MonoImageMinResolutionPolicy /OK /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 1200 /MonoImageDepth -1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict > /AllowPSXObjects false /CheckCompliance [ /None ] /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile () /PDFXOutputConditionIdentifier () /PDFXOutputCondition () /PDFXRegistryName () /PDFXTrapped /False
/CreateJDFFile false /Description > /Namespace [ (Adobe) (Common) (1.0) ] /OtherNamespaces [ > /FormElements false /GenerateStructure false /IncludeBookmarks false /IncludeHyperlinks false /IncludeInteractive false /IncludeLayers false /IncludeProfiles false /MultimediaHandling /UseObjectSettings /Namespace [ (Adobe) (CreativeSuite) (2.0) ] /PDFXOutputIntentProfileSelector /DocumentCMYK /PreserveEditing true /UntaggedCMYKHandling /LeaveUntagged /UntaggedRGBHandling /UseDocumentProfile /UseDocumentBleed false >> ]>> setdistillerparams> setpagedevice