EMPOWERING TECHNOLOGY LEADERS OF TOMORROW thesmarttechie.comOctober 2010

`̀ 25 only

T Ashok, CEOSTAG Software

Guru talk: Sashi Reddi, AppLabs VC Talk: Jagannath Rao, Headstart Ventures

Pursuit

The Smart Tech ie |33|O c t o b e r 2 0 1 0

STAG InClean Softwareof

s o f t w a r e

By Vimali Swamy

COVER STORY

Founded: 2000CEO: T AshokHeadquarters: BangaloreOther Offices: Chennai, San Jose(U.S.), Tokyo (Japan)Employees: About 200Customers: Over 100Website:www.stagsoftware.com

STAG Software at a glance There has been significantprogress in the field ofsoftware testing in re-cent years. However,one finds it hard to pro-

vide a logical means to get closer toperfection.A typical approach to test-ing based on the activity-basedmodel consists of strategizing, plan-ning, designing, automating, execut-ing, and managing. Over the years,the industry has moved from com-pleting these activities in one go in toan agile version consisting of theseactivities done in short incrementsand the field of software testing hasbeen littered with jargons, processmodels, and tools. Yet the notion of‘guarantee’ seems elusive.

Is it possible to guarantee qual-ity? ‘Guarantee’, implies that the de-ployed software will not causebusiness loss and that the means ofvalidation can be proven to be cor-rect. It is generally understood thattesting is a process of uncovering de-fects done via a good mix of tech-niques, tools, and people skills. To

make guarantees, it is imperative thatthe approach to evaluation be sharplygoal-focused. It is to achieve thisquality of guaranteed clean softwarethat STAG Software is striving on.Founded in 2000, STAG Software isa boutique test engineering companyheadquartered in Bangalore.

By goal-focused evaluation, thecompany means that one shouldhave clarity as to what potential de-fects we have to go after. Once thepotential defects are discerned byemploying a scientific approach, itis possible to arrive at an effectivevalidation strategy, a complete setof test cases, better measures ofcleanliness (quality), and appropri-ate tooling.

Having pioneered a scientificapproach to testing – Hypothesisbased testing (HBT), it is attempt-ing to swim against the current bydefying the traditional approach totesting. Today, a decade since itshumble beginnings, HBT is beingslowly accepted as a more result-oriented option by the industry.

T Ashok

The Smart Tech ie |35|O c t o b e r 2 0 1 0The Smart Tech ie |34|O c t o b e r 2 0 1 0

HBT – Hypothesis Based TestingHypothesis Based Testing (HBT) is atest methodology based on solid sci-entific principles, It is like a SherlockHolmes-ian approach to testing - hy-pothesize potential defects and thenscientifically construct the strategy,test cases, measures, and tooling. HBTis a personal test methodology con-sisting of six stages of cleanliness. Itis powered by STEM (STAG Test En-gineering Method), a scientific test en-gineering method consisting of eightpersonal thinking disciplines aided bythirty two core concepts that aid sci-entific enquiry. Evaluation of a soft-ware or system in HBT consists ofsetting up cleanliness criteria, thenidentifying potential defect types, stag-ing them in an optimal order to createa cost-effective staged evaluationmodel, formulating the various typesof test and corresponding techniques, aformal design process to ensure thatthe test cases are sufficient, and thenautomating them as needed.

“The core theme of HBT is hy-pothesizing potential defects and thenscientifically constructing the strategyand test cases, measures, and tooling.It is a goal-centered methodologywherein the goal of software cleanli-ness is set up (i.e. a collection of clean-liness criteria), potential defect typesthat can impede the cleanliness crite-ria identified, and then activities per-formed to ensure purposeful testingthat is indeed effective and efficient,”explains T. Ashok, CEO, STAG Soft-ware. Based on sound engineeringprinciples geared to deliver the prom-ise of guarantee of cleanliness, its corevalue proposition is about hypothesiz-ing potential defects that may be pres-ent in the software and then allowingone to engineer a staged detectionmodel to uncover the defects fasterand cheaper than the other typical testmethodologies.

“The concept of applying HBT onsoftware is similar to visiting the doc-tor where he or she hypothesizes po-

tential problems based on one’s symp-toms, performs diagnostic tests to con-firm the hypothesis, and thenprescribes the treatment regimen,”quips Ashok. “It’s a personal testmethodology that fits any develop-ment methodology and weaves intoany organizational test process. Thebusiness promise of HBT is to slashtest and support costs and acceleratedevelopment.”

HBT has been applied by STAG invarious applications in areas like Mo-bile, Healthcare, ERP, Media, eLearn-ing, and Supply Chain Managementover the last nine years. This has re-sulted in lowered defects escapes (upto 10x lower), increased test coverage(at least 3x), better RoI on automation,lower support costs (by 30 percent)with no increase in effort, time orcosts. HBT has been implemented invarious process models includingAgile. HBT has not only been appliedto validating products, but has beenapplied to validating requirements andarchitecture, quite different from thetypical review or inspection process.HBT has also been applied in assess-ment of test assets to ascertain effec-tiveness and improve them.

One such example is of a companythat provides online banking solution.The company has three major productscatering to over 100 top financial in-stitutions (FI) including the top fiveFIs in the world. It has a successfulproduct line, growing rapidly withmajor releases almost every year, in-corporating new features to cater to thevarious needs of the market place. Asthe code base evolved, the test assetswere also modified to reflect thechanged product. The challenge faced

was that most of the test cases werenot detecting defects and the rate ofuncovering new defects was low.

As the product became huge andthe company decided to re-architect itin order to enable rapid feature addi-tion with low risk. That is when thecompany decided to have a re-look atits test assets and re-architect the sameto increase the test coverage, improvedefect finding ability, and ensure that

the test assets were future-proof. It hadabout 8,000 test cases then.

The company approached STAGto analyze the existing test cases forcompleteness and modifiability andre-architect the same after filling thegaps and to ensure that the future testcases were easily pluggable.ApplyingSTEM, STAG performed a thoroughassessment of the existing test assetsand discovered holes in the same.Using the STEM Test Case Architec-ture (STEM-TCA), it re-engineeredthe test cases by first grouping theminto features, then by levels of testsand segregating into various types oftests, and then finally by separatingthem into positive and negative testcases. During this process of fitmentof existing test cases into the STEM-TCA, STAG uncovered quite a fewholes. These were filled by designing5,000 test cases additionally. Not onlydid the STEM-TCA increase the testcoverage by uncovering the missingtest cases, it also provided a sharpervisibility of the quality as the test caseswere well organized by specific defecttypes. This improved the test coverageby about 250 percent and the technicalmanagement staff was confident aboutthe adequacy of test assets and wasalso convinced about its future up-

gradeability and maintainability.In another example, early imple-

mentation of HBT yielded significantresults for a global chip major. STAGhad to set up an effective validationpractice for their video decoder soft-ware. The challenge that the companyfaced with their complex product thatinvolved both hardware and softwareand later system integration on multi-ple realtime OS on various platformswas that of high defect escapes i.e.post-release field defects.

Ashok and his team spent about amonth understanding their domain andthe associated technologies. Post this,a detailed analysis yielded interestingdata – test cases were primarily con-formance oriented, coverage of testcases was suspect, escaped defectsseemed to propagate from early stages,and finally the process of validationwas loose. Having understood thetypes of defects that were being foundand the post-release defects, they fig-ured out the various types of probabledefects and the various combinatorialaspects that needed to be considered toform a test case. The team then stagedthe validation as consisting of threemajor levels, the first one atAPI level,the next one at a system level, and thelast one made up of a customer-centriclevel that involved using reference ap-plications.

Applying the HBT approach to testdesign, the test cases were developed,yielding about 6,000 test cases at levelone and about 800 at the subsequentlevels. Also, whereas the ratio of +vevs –ve test cases was earlier towardsthe +ve side; after their re-design, theratio shifted to 60:40 percent at thelower and about 85:15 percent at thehigher levels. Moreover, the numberof test cases increased significantly bya factor of 1,000 percent, allowing fora larger and deeper net to catch manymore serious defects. Over the next 9months, the rate and number of defectsdetected increased dramatically, re-sulting in post-release issues reducing

by a jaw-dropping 10x times.Once the test effectiveness prob-

lem was solved, the yield of defects in-creased, the focus shifted tostreamlining the process by setting up

proper gating in the test process andcreating a centralized Web based testrepository, and finally setting up astrong defect analysis system based onOrthogonal Defect Classification(ODC) method. This enabled a strongfeedback system, resulting in shiftingthe defect finding process to earlierstages of SDLC and thereby loweringcycle time. Complementing this,STAG focused on setting up a customtooling framework for automating thisnon-UI based software, resulting in asignificant cycle time reduction – anentire cycle of tests on a platform tookonly less than 15 hours of time.

STEM is the basis for intelligenttesting to deliver compelling businessresults as it enables early detection ofdefects and enhances ability to find de-fects by at least three times, resultingin reduction of cost and time by over30 percent.

“We establish a clear goal andthen perform activities that will indeedget us to the goal. The goal translatesto ‘uncovering potential defects’.

These potential defects are hypothe-sized and all the later activities areabout proving that the hypothesizeddefect(s) do or do not exist,” explainsAshok.

One of the USPs of HBT is its sim-plicity. Since it is a methodology gov-erned by 32 core concepts anddisciplined processes, it is easier tolearn and defies the common notionthat effectiveness in testing dependson the experience of the test staff. It isa general belief that the more experi-enced people in the QA team is thebetter for the quality of work. AtSTAG, teams are not merely assem-bled based on number-of-years-of-experience, but on what keydisciplines are needed to perform therole and therefore the required compe-tency. STAG evaluates the fitness of aperson to a team based on his or herrole and therefore the required compe-tencies to perform the role. Measuringthe increase in a QA’s competencywhile working with HBT, STAG de-rived a competency model – CREAM.

CREAM enables it to evaluate anindividual’s test-related competenciesin terms of test lifecycle activities,technology aspects, and business do-main aspects. Based on this model,

Maintenance testTest reengineering

Coverage enhancementLSPS solution

Product

Organization

People

UT assessment

Tool adoption

DevQ system

IT assessment

QA system

Req validationArchi. validation

SkillingCOMPASSTM

Optimization & reduction

Diagnostics & control

Productivity enhancement

System enhancementSkill enhancement

Quality injection

Quality enhancement

Mobile app validation suiteE-Learning validation suite

Evaluation suites

ERP validation suiteBluetooth validation suitementtt

EBA

E&T

Test services

atiiooooooooooooooooooooooooonnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnttiiiiiiiiiiiiooooooooooonTA

Outsourced testingManaged QA

JumpStart QAAssessment services

Custom toolingFunctional automation

LSPS validation

Corp

Retail

TrainingHypothesis BT

Robust TDPurposeful strategy

Successful automation

Finishing school

STEMTM

The tesing methodology is applied tovalidating products, requirements, andarchitecture, quite different from the typicalreview or inspection process

Boutique solutions from STAG Software

competency gaps are analyzed, appro-priate training plan for the individualis formulated, and the person istrained.Engaging the CustomersAs a method oriented organization, theart of customer engagement is asunique as the company itself. Since theindustry is yet to fully understand thecompany’s approach, STAG first edu-cates its customers about the betterquality and RoI by deploying STEMand HBT.At present, STEM and HBTare primarily deployed by the com-pany’s customers only in certain partsof a project. Ashok has also started tolicense the methodology to companieswho wish to deploy it internally andcustomize it to their requirement.These include software developmentcompanies and other independentservice providers amongst which isone of the top five Japanese system in-tegrators that today deploys STEM asa part of its services.

“We are very serious about theeconomics of testing and its contribu-tion to the bottom-line of our cus-tomers.At every instance, we questionourselves the value being deliveredand how it has impacted our cus-tomer’s business,” says Ashok. Whenengaging with a customer, STAG paysa lot of attention to understanding.HBT takes a scientific approach to theact of understanding the intentions orexpectations by identifying key ele-ments in a requirement or specificationand setting up a rapid personal processpowered by scientific concepts toquickly understand the intentions andidentify the missing information. Thisenables the engineers to come up withintelligent questions leading to adeeper understanding. This results inrapid ramp-up, higher business valuebeing delivered, and leveraging thisunderstanding to build a rich assetbase.

STAG offers unique solutions andservices like test assessments, valida-tion suites, release worthiness assess-

ment, people competency assessment(in testing), test-case re-engineering,custom tooling for test bench, require-ment and architecture validations, inaddition to standard offerings like out-sourced QALab, managed validation,functional test automation, and per-formance assessment.

Developing a Methods CompanySoftware testing, though critical insoftware development, for years hasbeen considered the most jaded andmundane task in software develop-ment life cycle (SDLC). Though Indiahas carved a niche for itself in the test-

ing industry, it is true that most entre-preneurs have shied away from gettinginto the business testing. Industrywatchers attribute two probable rea-sons for this - one, the general mis-conception that testing, unlikesoftware services or products business,is not a lucrative proposition and theother is the feeling that testing wouldfigure at the low end of the market.

Neither of these fears is based onfacts, says Ashok whose passion fortesting resulted in STAG Software.Today, STAG has firmly positioned it-self as a boutique company offeringtesting solutions and services, which

he believes is quite different fromthose of other vendors that werefounded at the same time.

Most testing services players com-pete on the ‘volume game’. Everyproject is won on the basis of the sizeof the team or number of service loca-tions they can offer to their customers.“This is the fundamental differencebetween STAG and other independenttesting vendors,” Ashok points out.

STAG essentially is a ‘methods’company and not a product or servicesone. The core foundation of the com-pany lies in the fact that it developsmethodologies or concepts that can beused in testing irrespective of the do-main or technology.

This ideology of working on test-ing methods was influenced by IBMRational that had worked long years toset up methods for software develop-ment and over the years came up witha standard called UML (Unified Mod-eling Language). Today, UML is thebasic framework around which soft-ware is developed despite what thetechnology may be.

A Masters in computer sciencefrom the Illinois Institute of Technol-ogy,Ashok had always had a knack forresearch oriented approach to prob-lems. It was while leading the world-wide software test analysis group(WW:STAG) at Verifone India, for-merly a division of Hewlett Packard,that he started dabbling with the ideaof finding more effective methods fortesting.

“WW:STAG was a large inde-pendent test organization in Veri-fone/HP responsible for certifyingclient server and electronic commercesoftware. This group built ground up atest group well recognized within andoutside the company. It was the idealplatform for me to experiment on thescientific approach to testing,” saysAshok.

As the software becomes complexday by day, there is also an increase inbugs or software defects that slip into

a software program when it is written.Traditionally, software testing, de-pending on the testing method em-ployed, can be implemented at anytime in the development process.However, most of the test effort occursafter the requirements have been de-fined and the coding process has beencompleted. As such, the methodologyof the test is governed by the softwaredevelopment methodology adopted.“As software services companiesgraduate from plain coding work tofull development (not necessarily de-veloping shrink wrap products) testingbecomes crucial. But unfortunately,many programmers are forced to dominimal testing to beat the last minuteproject deadline,” he says.

It was the lax approach towardstesting and the challenges faced byquality analysts that sparked Ashok’sthought process: could there be amethodology for testing that couldwork as a scanner wherein if one ran apiece of code, it could immediatelyhighlight the bugs. Thus, he came upwith the concept of hypothesis basedtesting.

Today, a decade since its beginning,STAG has successfully worked withover 100 customers and conductedaround 350 cycles of various types oftest viz., functional testing, load/perfor-mance/stress/volume test, reliabil-ity/scalability test, test automation, Betatesting, Installation testing, configura-tion/compatibility testing, L10N test-ing, document testing, API testing, andmore.

Having worked on 170+ projects inthe areas of enterprise applications, em-bedded software, and test automationwith customers spanning industries -

Avionics, BFSI, Consumer Electronics,eLearning, ERP, Healthcare, Logistics,Mobile/Wireless and CDMA, Realty,Semiconductor, Shipping, and popularInternet search engines – STAG trulyproves that its methodologies are do-main independent. The ultimate recog-nition for the company is STEM 2.0being featured in the Unisys Technol-ogy Review, a well known journal inthe Japanese software industry and ac-ademia, and part of the Japanese Na-tionalArchive. “Japan is a research andtechnology oriented industry and it is ahuge honor to be recognized by them,”Ashok smiles.

With its path breaking research ori-ented approach to software testing,STAG has managed to disrupt the way

testing has been practiced over thedecades. “I believe this is quite anachievement but there is still some timebefore the industry as a whole adaptsit,” says Ashok. In order to evangelizethe adoption, STAG itself has taken theinitial steps by offering a full fledgededucation division to train people inSTEM and HBT andmake them indus-try ready. The programs offered fromits Bangalore and Chennai facilitieshave already proved quite effective asa majority of the trainees are being ab-sorbed by the company’s clients them-selves. Also, he plans to publish all theresearch on STEM and HBT and bringout a book soon that could perhaps oneday be an academic reference in soft-ware testing.

Ashok often quips that softwaretesting is like the ‘ugly duckling’ thatnot many prefer to step into, but it isjust a matter of time that it will trans-form into a beautiful swan and hope-fully STAG will be the catalyst.

The Smart Tech ie |37|O c t o b e r 2 0 1 0The Smart Tech ie |36|O c t o b e r 2 0 1 0

Start with establishing a clear goal and thenperform activities that will indeed get you totranslating the goal into ‘uncoveringpotential defects’

“I find that many firms focus on cost per resource but in my experience, this is thewrong metric. The real key to selecting an offshore development/QA partner is pro�ductivity per dollar (or pound) versus cost/resource. To this extent, we have foundSTAG to provide exceptional value.”Marc Zionts, CEO, Ortiva Wireless

“I am very impressed with STAG’s approach to testing. Their quality of test designand completeness of testing as a result of their methodology is evident in every re�lease. The commitment and flexibility shown by the team to accommodate aggres�sive release dates is praise worthy.”Iype Isaac, CTO, Bankbazaar.com

“A true partner to GXS, STAG has worked with almost all the groups and has alsoprovided boutique solutions like Test design services and end�to�end automation ofthe Trading Grid using QTP and Perl.”Ramesh B R, Program Manager, GXS

“STAG has always responded swiftly to our rapidly growing resource needs for test�ing. Their flexibility to ensure business continuity is highly appreciated. The commit�ment and the flexibility shown by their team to meet highly aggressive releasetimelines is commendable. We have always counted on STAG and will continue todo so.”Ramakrishna Nutulapati, Sr. QA Manager, Jamcracker

“STAG focus on scientific way of doing testing with their methodology HBT was re�flected in the effectiveness of test artifacts created by their team. The commitment ofsenior managers in reviewing engagement health periodically and taking appropriatesteps at each stage really helps in getting best ROI for the investment we make inSTAG solutions.” –Raj Khare, Chairman and Founder, Surewaves

SO WHAT DO CLIENTS HAVE TO SAY ABOUT STAG SOFTWARE?