Lasse Löytynoja

Applying Test Automation to GSM Network Element -

Architectural Observations and Case Study

Master’s Thesisin Information Technology

September 5, 2018

University of Jyväskylä

Faculty of Information Technology

Kokkola University Consortium Chydenius

Author: Lasse LöytynojaContact information: lasse.m.loytynoja@jyu.fiPhonenumber: 045 2736112Title: Applying Test Automation to GSM Network Element - Architectural Obser-vations and Case StudyTyön nimi: Testausautomaation soveltaminen GSM-verkkoelementin ohjelmistotes-tauksessa - Arkkitehtuuriset havainnot ja tapaustutkimusProject: Master’s Thesis in Information TechnologyPage count: 64+0Abstract: This thesis examines functional test automation and potential risks thatmay lie in its architecture. The environment of the thesis is a component of theGSM network. The first chapter examines overall testing, test automation, testinglevels and techniques. Next, the structure of the GSM network, its componentsand working principles are presented. Finally, based on literature and the author’sown work experience, the common problems in test automation are presented anddetailed solution are suggested for each one, with case examples.Suomenkielinen tiivistelmä: Tutkielmassa tarkastellaan funktionaalisen testauk-sen automatisointia ja sen arkkitehtuurissa mahdollisesti esiintyviä riskejä. Toim-intaympäristönä on GSM-verkon komponentti. Aluksi tarkastellaan testausta, tes-tausautomaatiota, eri testaustasoja sekä tekniikoita. Seuraavaksi tutustutaan GSM-verkkoon, sen komponentteihin ja toimintaan. Lopuksi esitellään testausautomaa-tiossa mahdollisesti esiintyviä ongelmia kirjallisuuteen ja tekijän omaan työkoke-mukseen perustuen, sekä niiden ratkaisuja yksityiskohtaisin tapausesimerkein.Keywords: GSM, Functional testing, Telecommunications, Test AutomationAvainsanat: GSM, funktionaalinen testaus, matkapuhelinverkot, testausautomaatio

Copyright c© 2018 Lasse Löytynoja

All rights reserved.

Glossary

2G Second generation of wireless telephone technologyA interface The interface between BSS and NSSAbis interface The interface between BTS and BSCAir interface The interface between MS and BTSAter interface The interface between BSC and transcoderBCF Base Control FunctionBTS Base Transceiver StationBSC Base Station ControllerBSS Base Station SubsystemCEPT Conférence des Postes et TelecommunicationsCI Continuous IntegrationDX 200 The digital switching platform of Nokia NetworksEDGE Enhanced Data rates for GSMETSI European Telecommunication Standards InstituteE-GSM Extended GSMGb The interface between BSS and SGSNGPRS General Packet Radio ServiceGSM Global System for Mobile CommunicationsHLR Home Location RegisterIMEI International Mobile Equipment IdentityIMSI International Mobile Subscriber IdentityLA Location AreamcBSC Multicontroller BSCMML Man-Machine LanguageMS Mobile StationMSC Mobile-Services Switching CentreMSISDN Mobile Station International ISDN Number

i

MSRN Mobile Station Roaming NumberNSS Network Switching SubsystemO&M Operations & MaintenanceOSS Operations SubsystemP-GSM Primary GSMPCM Pulse Code ModulationPSTN Public Switched Telephone NetworkR-GSM Railway GSMSGSN Serving Gateway Support NodeSMS Short Message ServiceSSH Secure ShellTDMA Time Division Multiple AccessTestware Software tooling used in testingTMM Testing Maturity ModelTMSI Temporary Mobile Subscriber IdentityV-model A software development processVLR Visitor Location Regiter

ii

Contents

Glossary i

1 Introduction 1

2 Software Testing 32.1 Software testing overview . . . . . . . . . . . . . . . . . . . . . . . . . 3

2.1.1 Software quality . . . . . . . . . . . . . . . . . . . . . . . . . . 32.1.2 Errors and Defects . . . . . . . . . . . . . . . . . . . . . . . . . 52.1.3 Test Activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.1.4 Software Testability and Instrumentation . . . . . . . . . . . . 8

2.2 Software Testing Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.2.1 The V-Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2.2 Acceptance testing . . . . . . . . . . . . . . . . . . . . . . . . . 92.2.3 System Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.2.4 Integration Testing . . . . . . . . . . . . . . . . . . . . . . . . . 112.2.5 Unit Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112.2.6 Other Testing Levels . . . . . . . . . . . . . . . . . . . . . . . . 112.2.7 Regression Testing . . . . . . . . . . . . . . . . . . . . . . . . . 12

2.3 Software Testing Techniques . . . . . . . . . . . . . . . . . . . . . . . . 132.3.1 Black and White Box Testing . . . . . . . . . . . . . . . . . . . 132.3.2 Other testing techniques . . . . . . . . . . . . . . . . . . . . . . 132.3.3 Fault based techniques . . . . . . . . . . . . . . . . . . . . . . . 142.3.4 Usage-based techniques . . . . . . . . . . . . . . . . . . . . . . 152.3.5 Model-based techniques . . . . . . . . . . . . . . . . . . . . . . 152.3.6 Techniques based on nature of application . . . . . . . . . . . 15

2.4 Software Test Automation . . . . . . . . . . . . . . . . . . . . . . . . . 152.4.1 Requirements of the organization . . . . . . . . . . . . . . . . 182.4.2 Continuous Integration . . . . . . . . . . . . . . . . . . . . . . 19

2.5 Software Testing Maturity Model . . . . . . . . . . . . . . . . . . . . . 202.5.1 Maturity levels and goals . . . . . . . . . . . . . . . . . . . . . 21

iii

3 The GSM Network 243.1 History . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.2 Present and future . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.3 Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273.4 Mobile Station . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323.5 Base Station Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

3.5.1 Base Transceiver Station . . . . . . . . . . . . . . . . . . . . . . 333.5.2 Base Station Controller . . . . . . . . . . . . . . . . . . . . . . . 33

3.6 Network Switching Subsystem . . . . . . . . . . . . . . . . . . . . . . 353.7 Operations Subsystem . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

4 Nokia implementation and test process 384.1 DX 200 platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384.2 Base Station Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

4.2.1 Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424.2.2 User interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

4.3 Operations & Maintenance Functional Testing of the BSC Software . 434.4 Holistic Integration Tester . . . . . . . . . . . . . . . . . . . . . . . . . 444.5 Test Automation Framework . . . . . . . . . . . . . . . . . . . . . . . 454.6 Test Automation Process . . . . . . . . . . . . . . . . . . . . . . . . . . 45

4.6.1 Test automation infrastructure . . . . . . . . . . . . . . . . . . 464.6.2 Extending the framework . . . . . . . . . . . . . . . . . . . . . 48

5 The Suggested Structure of Automated Test Cases 495.1 Problems faced with test automation . . . . . . . . . . . . . . . . . . . 49

5.1.1 Test-case portability . . . . . . . . . . . . . . . . . . . . . . . . 495.1.2 Script-to-script dependencies . . . . . . . . . . . . . . . . . . . 515.1.3 Test Setup and Teardown Handling . . . . . . . . . . . . . . . 535.1.4 Test libraries and user interface abstraction . . . . . . . . . . . 565.1.5 Complexity and Execution time of test cases . . . . . . . . . . 58

5.2 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595.3 Supporting findings in another project . . . . . . . . . . . . . . . . . . 60

6 Summary 61

References 62

iv

1 Introduction

The objective of this master’s thesis is to recognize the special features and problemsin test automation of Nokia’s 2G BSC (base station controller) O&M functional test-ing environment. The goal is to suggest solutions to the issues and use them as basisto develop a streamlined, common architecture for automated test cases which canbe used to improve their consistency and maintainability for the testing organiza-tion. The need for this kind of architecture emerged during the author’s own workas a testing engineer of the subject. The task was to introduce the test automation totesting organization and to train and support colleagues in its use. The author cre-ated a basic architecture for the initial ramp-up trainings, but when all requirementswere finally clear, it proved to be insufficient and needed further development. Thisresulted as tedious test code refactoring work, in which all implemented test caseswould use the new model that would provide benefits of transportability and bet-ter maintainability. Without even the support and boundaries of basic architecture,the results would have been a large mass of unmaintainable, non-transferable andfrequently recurring test code. This thesis aims to provide a more complete andmotivated architecture.

The theoretical part is based on literature, some industrial reports of same typeof test automation environments, and some training materials. It describes soft-ware testing and test automation, GSM network structure and operation and finallythe network elements of the base station subsystem. Much of the background alsocomes also from the author’s own working experience on the subject during 2012-2015, the last position being a thesis worker for the Tampere unit. This thesis ismade per request of Nokia Networks.

The main question this thesis attempts to answer is:

• What are the requirements to automate O&M functional testing of the GSM basestation controller for the test automation? Are the automated test cases able to benefitfrom common architecture and what would it look like?

The supporting questions are:

• What are the factors that affect the maintainability of the automated test case?

1

• What are the factors that affect the execution time of the automated test cases?

• What are the factors that affect the the transportability of the automated test case?

The research of software testing and test automation is done mainly by readingacademic and other professional literature. Information about the test automationand the BSS domain at Nokia is obtained through having worked with the productfor several years. The architectural plan is supported by the theoretical backgroundof testing and reviewing academic literature and case studies of similar testing en-vironments.

Chapter 2 defines the software testing and the software quality. We will take lookin errors and defects and basic test activities. Test metrics and software testabilityare also discussed. The chapter also introduces Boehm’s V-model and the basic lev-els of software testing, describes regression testing and its role and costs in softwaredevelopment and explains the different software testing techniques. The latter sub-chapters define test automation and its role in modern software development, andbriefly introduces the software testing maturity model and the associated maturitylevels. Chapter 3 defines the GSM network history from past to present, and the ba-sic structure of the network, with focus on the base station subsystem components.Chapter 4 describes the Nokia implementation of GSM networks, the DX200 plat-form and the structure of the base station controller. The chapter also provides anoverview of the functional O&M testing of the base station controller. Chapter 5 de-fines the issues faced during test automation process and the suggested solutions.Chapter 6 summarizes the thesis.

2

2 Software Testing

This chapter contains the definition of software testing and software quality in gen-eral. The first subchapter briefly reviews verification and validation, as well as er-rors and defects in software and their common causes. Common test activities andthe factors that make software more testable are also discussed. In second subchap-ter, software testing levels are described with support of the V-model. The thirdsubchapter describes software test automation and discusses what it requires fromthe testing organization. It also introduces rudimentary method called continuousintegration. The fourth subchapter discusses the software testing maturity model,which aims to define the level of maturity of a software testing process in an orga-nization.

2.1 Software testing overview

In this subchapter, the software quality and software errors and defects are defined.We’ll also take a look to test activities and software testability and the related instru-mentation.

2.1.1 Software quality

Software quality is an ambiguous term and there is a demand for software testingto become more frequent. The following definition of software quality is made by[7]: "Software quality is achieved by conformance of all requirements regardlessof what characteristics is specified or how requirements are grouped or named".The stakeholder value that the software produces along with price and lead timeand how "fit for use" it is, could also be simplified quality descriptors. In general,software quality refers to desirable characteristics of the product and the level atwhich these are attained, but it also includes the processes and the tools that areused to reach them.

There are many definitions of software testing as well as many misconceptions,resulting in poor testing performance. Some of most common misconceptions are:

3

• Testing is the process of demonstrating that errors are not present.

• The purpose of testing is to show that a program performs its intended func-tions correctly.

• Testing is the process of establishing confidence that a program does what it issupposed to do.

These statements are broad, but also a bit thin. IEEE describes software testingas follows:

• "The process of operating a system or component under specified conditions,observing or recording the results and making an evaluation of some aspect ofthe system or component [20]."

• "Software testing consists of the dynamic verification that a program providesexpected behaviors on a finite set of test cases, suitably selected from the usu-ally infinite execution domain [7]."

Four keywords emerge from the definition of [7], providing some basic insightinto software testing: dynamic, finite, selected and expected. Dynamic verificationmeans that the software or system under test may be complex and nondeterministicand therefore give a different output with the same input, depending on the systemstate. Finite means that a set of test cases must be delimited, as theoretically evensimple programs have so many possible test cases that exhaustive testing could takea long time to complete. Test case selection is affected by the test technique in use,i.e. code-based or specific to the nature of the application. The expertise and in-tuition of the software engineer may also play a big part in the selection process.Defining the exactly expected outcome is not always easy, but it must be done inorder to have rational test results. The outcome must be compared against specifi-cations, requirements or other acceptance criteria.

Software testing is about increasing the quality of the software by using a well-defined process. It is pervasive over the life cycle of the software; starting from earlystages of the requirements process and developed and refined throughout the actualsoftware development, until the maintenance phase of the software.

The importance of software testing is reflected in various past incidents. Accord-ing to James Gleick [14], a small deficiency in the testing phase can have expensive

4

consequences, as with the Ariane 5 rocket. In its maiden voyage, carrying four satel-lites, the software in the guidance system tried to convert 64-bit number in to a 16-bitnumber, which resulted in an overflow error. This caused guidance system to shutdown and the redundant system did not help as it was running same faulty soft-ware. After 39 seconds of flight, the self-destruct mechanism finished the rocket byexploding it. Ultimately the price tag for this 10-year project and equipment wasUS$7 billion. There are numerous other examples in the history of software devel-opment.

2.1.2 Errors and Defects

According to [20], an error in software is when a computed, observed or measuredvalue or condition differs from a true, specified or theoretically correct value or con-dition. Although the terms "error", "fault", "defect" and "failure" may be used inter-changeably in practice [8], Burnstein discusses the differences between the terms.The source of an error is the software developer (this includes software engineers,programmers, analysts and testers), who has made a mistake, misconception ormisunderstanding during the development process. A fault, or a bug or a defect, isa result of an error. It means incorrect behaviour of the software, a situation wherethe functionality of the software fails to adhere to its specification. A failure of soft-ware in turn is "inability of software system or a software component to perform itsrequired functions within specified performance requirements" [20].

According to Whittaker [33] and Burnstein [8], the source of software defectscould be any of these:

• The user executed untested code.

• The statements in the code were executed in a different order to what wouldbe "normal".

• An untested combination of input values is fed to the software.

• The software was executed in a previously untested environment.

In [8], Burnstein classifies defects in four categories. The categories are require-ments and specification defects, design defects, implementation defects and testingdefects.

5

The requirement and specification defects are generally due to incorrect, am-biguous, incomplete or even missing descriptions of functionalities. These kinds ofdefects are often encountered in faulty interactions of separate software features, i.e.saving and subsequent categorization of customer data in a customer relationshipmanagement system, interfaces (i.e. graphical user interfaces) or machine interfaces(i.e. APIs, application programming interfaces).

Design defects are flaws in the design of algorithms, control logic and sequences,data elements, module interface descriptions and external software / hardware /user interface descriptions. If the design description is not detailed enough withpseudocode presentation of major control and data flow structures, many defectsclassified as design defects should instead be classified as implementation defects.

The sources of implementation defects are purely errors in code implementation.An example of an implementation defect would be a comparison between inappro-priate data types or a faulty data type conversion, as described in section 2.1.1. Thesemay be closely related to design defects, if pseudocode is used to describe the de-sign. Moreover misunderstandings or mistakes of software developers, or a lack ofcommunication and education and understanding of the used tools or languagesmay be the sources for this kind of defects.

The actual software product is not the only place where the defects can be found.Testing defects refers to defects that are found in test cases, test plans, test harnessesand test procedures. Defects of this type are i.e. the incorrect or incomplete designof test cases. Some testing levels require the development of additional test har-ness code in order to be able to carry out the testing. This harness code is subjectto similar defects with any other software product and it should also be carefullytested.

2.1.3 Test Activities

Testing activities are parts of the testing process. The exact name and content ofthe activities vary from an organization to another, yet the tasks are usually similardespite the conventions. In [12], the activities are defined as a sequential processwhich is described in Figure 2.1.

The identification phase determines the test conditions, what can be tested andwhat testing technique should be used. The test conditions are also generally pri-oritized in this phase. The test condition refers to an item or an event that can beverified by a test. Testing technique refers to a systematic way of obtaining a test

6

Figure 2.1: The testing activities [12].

result, such as boundary-value analysis. The testing techniques are discussed morein Chapter 2.3.[12]

In the design phase, it is determined how the test is to be conducted. A test casewill specify all steps or actions that are needed to reach the given objective. The testcases produced in this phase contain all prerequisites, input values and expectedoutcomes in a sequential manner.[12]

The build phase consists of building artifacts that are required for test execution.These are test scripts, test data sets and possible input and output definitions if thetooling requires it. The important part, particularly as regards this work work, isthe preparation of the test environment hardware. It can mean the initialization of adatabase, loading a new software build into a system, or configuration of the systemsoftware and hardware to sufficient level for the test at hand.[12]

In the execution phase the software under test is executed using the test cases asguide. This can take place automatically or manually. In manual execution, thoseresponsible for the test execution follow the instructions of the test case and observethe results. In automated execution, the test instructions are executed by a script.The execution can also be a mix of the two, i.e. semi-automatic testing. This methodeither aims to improve efficiency by automating parts of the execution, or the man-ual intervention is mandatory due to changes i.e. to hardware which must be mademanually.[12]

In the comparison phase it is confirmed that the actual outcome of the test iscorrect and conforms to the requirements of the test case. If the result is correct, itis assumed that the test case is passed; if not the test has failed and the changes arethat a defect in software may have been found. Further investigation should takeplace to identify the source of error.[12]

7

2.1.4 Software Testability and Instrumentation

In [19], Huang states that "Normally, a computer program is designed to produceoutputs that are useful only to its users. If test execution of the program producesan incorrect result, it is very significant to the tester in that it attests directly to thepresence of faults in the program, and possibly includes some clues about the natureand whereabouts of the faults. On the other hand, if it produces a correct result,its significance to the tester is rather limited. The only conclusion the tester candraw from a successful test execution is that the program works correctly for thatparticular input. It would be beneficial to the tester if we could make a program toproduce additional information useful for fault-detection and process-managementpurposes."

There are applications for this kind of instrumentation consisting of:

• Test coverage measurement. As described in section 2.3.2, this kind of mea-surement is obtained for example placing a counter variables in branches ofthe control flow and checking the counter values after the test. Non-zero val-ues mean that all paths of control flow would have been traversed at leastonce.

• Test case effectiveness assesment. This means the effectiveness or capabilityof a test to reveal a fault in a program. This can be measured by computinga singularity index, which requires a thorough inspection of every expressionin the program. In other words, effective test cases e.g. verify multiple facetsof an object instead of single one and therefore have better changes of findingfaults.

• Assertion checking. By setting assertions to specific points in the source codeof a program, where the conditions are supposed to always be true in normaloperation, it is certain that there has been a fault in the program.

2.2 Software Testing Levels

This subchapter defines different software testing levels and shows their correspon-dence to the software requirements with the help of the V-model.

8

2.2.1 The V-Model

The V-model 2.2 is a software process model that was originally introduced byBoehm in 1979 [6]. It is most commonly used in more formal environments andwhere embedded software is produced that runs on hardware devices. The V-modelmerely implies a flow, where each phase of the development can be implementedbased on the detailed documentation produced in the previous phase[21]. It may beconsidered an extension to the traditional waterfall process model[3].

The V-model also shows how test activities may and should begin parallel to de-velopment activities The requirements development phase produces user require-ments, from which the acceptance tests are derived. System tests are based on thefunctional requirements and the integration tests on the system architecture. Mostlow-level tests are unit tests which are usually carried out parallel to design andimplementation activities.

By starting the test activities at the very beginning of the project (requirements),defects should be found earlier. For example, [34] suggests that a problem in re-quirements may take 30 minutes to evaluate and fix, but if same issue is found inthe system testing phase, it might take 5 to 17 hours to fix. It is clear that valuableproject hours are saved if the testing process is started early.

2.2.2 Acceptance testing

Regarding [7], acceptance testing is about whether the software system satisfies theacceptance criteria. These criteria are usually the business-level requirements of thesoftware system. In [3], it is also stated that users and other individuals who takepart in acceptance testing should have strong domain knowledge of the softwaresystem. Reflecting on my own experience in software and telecommunications in-dustry, this is true. Domain knowledge is vital with complex, interoperating sys-tems and their acceptance, and functional testing. As stated in [7], "The customer ora customer’s representative thus specifies or directly undertakes activities to checkthat their requirements have been met, or in the case of a consumer product, that theorganization has satisfied the stated requirements for the target market. This testingactivity may or may not involve the developers of the system."

9

Figure 2.2: The V-model

10

2.2.3 System Testing

In [7], system testing is defined as "System testing is concerned with testing the be-havior of an entire system. Effective unit and integration testing will have identifiedmany of the software defects. System testing is usually considered appropriate forassessing the nonfunctional system requirements such as security, speed, accuracy,and reliability. External interfaces to other applications, utilities, hardware devices,or the operating environments are also usually evaluated at this level.". In addition,in [3] Ammann and Offutt state that "This level of testing usually looks for designand specification problems. It is a very expensive place to find lower-level faultsand is usually not done by the programmers, but by a separate testing team."

2.2.4 Integration Testing

Integration testing assesses software with respect to subsystem design. This meansverification that the interfaces between modules communicate correctly. The devel-opment team usually has the responsibility of integration testing [3]. In [7], integra-tion test is defined as "Integration testing is the process of verifying the interactionsamong software components. Classical integration testing strategies, such as top-down and bottom-up, are often used with hierarchically structured software."

2.2.5 Unit Testing

Unit testing is testing done to the "smallest possible testable software component"[8]. Unit therefore refers to a class or a method of a class, which could be testedin isolation from other software. Usually the software developer who implementedthe code is the one who writes and conducts the unit tests [7].

2.2.6 Other Testing Levels

There are several other testing levels; the demand for these levels depends to largeextent on the type of software. Other typical testing levels may consist of:

• "Performance testing verifies that the software meets the specified performancerequirements and assesses performance characteristics - for instance, capacityand response time [7]."

• User interface and usability testing. User interface testing verifies the integrity

11

and functionality of the user interfaces of the software. Usability testing eval-uates how easy it is for users to learn and use the software [7].

• Security testing confirms that the tested software meets given security require-ments. A formal description found in [7] is "Security testing is focused on theverification that the software is protected from external attacks. In particular,security testing verifies the confidentiality, integrity, and availability of the sys-tems and its data. Usually, security testing includes verification against misuseand abuse of the software or system (negative testing)."

2.2.7 Regression Testing

As described by Burnstein in [8], regression testing is not a testing level, but a "retest-ing of software that occurs when changes are made to ensure that the new versionof the software has retained the capabilities of the old version and that no new de-fects have been introduced due to the changes" The name implies the main reasonthe tests are executed: after the changes have been made to the software, be it workneeded to introduce a new feature or fix a defect, there should be no regressions inthe functionality of the existing software. Regression tests usually contain a sub-set of test cases designed for the software, across several testing levels. Regressiontesting is considered important, as error corrections tend to cause more errors tosoftware than the originally implemented code [22].

Regression testing is considered as the most expensive testing of a software prod-uct. Costs of regression tests can be as high as 80% of the total testing expenses. Thehigh cost of regression testing has motivated efforts to reduce it, e.g. by using testset selection and minimization techniques [29] and test automation to reduce thelabor effort involved. While test automation is highly utilized in modern softwareproduction, everything cannot be automated (especially in the context of this the-sis, the world of telecommunications equipment); therefore the prioritization andselection methods are still valid techniques for reducing the regression testing load.More in-depth research on prioritization and selection methods is beyond the scopeof this thesis.

12

2.3 Software Testing Techniques

To detect as many defects in software as possible, a systematic approach in testingis needed. A random selection of test inputs is considered to be the least effectivemethod to achieve this [7][22]. This chapter describes common testing techniquesand their characteristics.

2.3.1 Black and White Box Testing

Black box testing (also known as data-driven and input/output-driven testing) refersto a testing technique where the internals of the program are unknown. In black-boxtesting the test cases rely only on input and output behavior of the software, and theused test data is determined by the software specification. The fault found in blackbox testing can be seen as a symptom of a problem, and the root cause might bemore difficult to identify than in white box testing.

White-box testing (also known as glass-box and logic-driven testing) is basedon assumption that the internal structure of the software is known. The test dataare determined by the design and logic of the software, but should also take thespecification into account. In white-box testing the source of the defect might beeasier to find, as the lines of faulty code are known.

2.3.2 Other testing techniques

The more detailed testing techniques presented in [7] are classified by how the testsare generated: the software engineer’s intuition and experience, the specifications,the code structure, the real or imagined faults to be discovered, predicted usage,models, and the nature of the application.

Ad-hoc and exploratory techniques This category consists of ad-hoc and ex-ploratory testing. Sd-hoc testing relies on the software engineer’s skill, intuitionand experience with similar kinds of programs and is probably the most practicedtechnique. In ad-hoc testing the tester can exercise or improvise some cases andscenarios that are difficult to come up with by means of more formalized methods.Exploratory testing is considered as a subset of ad-hoc testing. In exploratory test-ing there are also no predefined test cases or a test plan. The software engineer’sknowledge and familiarity with the application determine the effectiveness of thetesting. Exploratory testing highlights the concept of "simultaneous learning, test

13

design and test execution".Input domain-based techniques These techniques consist of equivalence parti-

tioning, pairwise testing, boundary value analysis, and random testing. In equiva-lence partitioning, the input domain is divided into subsets (equivalent classes) thatare based on a specific criterion or relation. The selection criterion may i.e. be basedon control flow or a sets of accepted and not accepted input values. Pairwise testingis a combinatorial testing technique, and gathers sets of interesting input value pairsacross the input domain. The boundary-value analysis is based on a rationale thatmost of faults are to be found at the boundaries, the extreme ends of input valueranges. In the random testing the input values are selected randomly from a knownrange of input domains.

Code-based techniques Code-based testing techniques consist of control flow-based criteria and data flow-based criteria. Control flow-based coverage criteriaconsiders all statements, blocks of statements and combinations of these as paths.Exhaustive testing of the paths is not usually possible, but coverage of the visitedstatements, conditions and branches is measured and the percentage is used as ametric for this type of testing. Data flow-based testing uses the control flow as welland annotates the flow graph with variables and their definition, usage and undef-inition. The all definition-use path requires that each variable is defined and goesthrough all control flow possibilities until the definition of variable is used. Thismethod is considered as the strongest, but it might result in large numbers of pathsand all-definitions and all-uses are it’s lesser alternatives.

2.3.3 Fault based techniques

Fault-based techniques use a fault model, i.e. a set of likely or predefined faults.The techniques comprise error guessing and mutation testing. In error guessing, themost anticipated faults are used as the base of test cases. The software engineer’sexpertise and the history of the product/project serve as a setting for guesswork.Mutation-based testing uses an altered version (by a small syntactic change) of thesoftware in parallel with the original version. By executing a test against differingversions, a difference in behavior is expected. It is assumed that small faults arelinked to bigger ones and can be found using this technique. The technique alsorelies on automatic generation of large numbers of differing "mutant" versions.

14

2.3.4 Usage-based techniques

Usage-based techniques consist of operational profile and user observation heuris-tics. In operational profile testing, the software is tested in an as closely reproducedoperating environment as possible. The inputs for the software try to simulate itsactual use, and its reliability is evaluated under these conditions. User observa-tion heuristics use methods such as cognitive walkthroughs, field observations, userquestionnaires and interviews to assess the usability problems of the graphical userinterfaces in controlled conditions.

2.3.5 Model-based techniques

Model based techniques consist of decision tables, finite state machines, formal spec-ifications and workflow models. Finite state machines are described as "by model-ing a program as a finite state machine, tests can be selected in order to cover thestates and transitions." Formal language specifications can automatically be tran-scribed into functional test cases and the corresponding results. TTCN3, which isspecifically used in testing of protocols in telecommunication systems, is an exam-ple of this type of language. In workflow models, activity sequences of humansor software are recognized as workflows. Tests target both typical and alternativeworkflows.

2.3.6 Techniques based on nature of application

Techniques that are based on the nature of the application focus on the software’scharacteristics, such as service-oriented, web software or embedded software. Thesecharacteristics may be used to supplement the previously mentioned techniques orvice versa.

2.4 Software Test Automation

Software testing is a resource- and time-consuming activity, which can use up to80% of the development costs. Test automation aims to reduce the effort needed foradequate testing, or increase testing that can be executed within a limited amountof time. Test automation has become one of central concepts in modern softwaredevelopment.[35][16] Basic building blocks of test automation are test cases, test

15

sets/scenarios and the testware used for test execution and reporting. When atest case is automated, it is turned into a machine-executable script or another ex-ecutable form, which contains definitions of test steps ? usually calls to an inter-face or methods provided by a test automation framework. Test automation can besemi-automatic, meaning scripts that assist in execution of test cases, or fully auto-matic, where test cases contain testing usually follows the continuous integration(CI) paradigm, which is introduced in section 2.4.2.

According to [12], test automation benefits consist of:

• Ability to execute regression tests automatically against new versions of a soft-ware. See paragraph on continuous integration, where this is discussed in de-tail.

• Ability to run more tests in a fixed time. By using automated testing, the timeneeded for testing is reduced. This leads to running increased numbers of testcases and greater confidence in the system.

• Ability to perform tests that would be difficult or impossible to do manually.Some testing, like testing a web-based system in regular intervals with 200concurrent users, would be next to impossible to arrange with test staff. Thistype of testing is a great fit for automation, where machines can be leveragedto simulate user load.

• Better use of resources. Tasks that are repetitive and dull in nature, such asthe constant entering of the same input values, should be automated. For thiskind of tasks, using automation produces greater accuracy and frees humanresources for other tasks.

• Better consistency and repeatability in testing. Automated tests are executedin identical fashion in each test run, which leads to consistent and repeatableresults. The level of consistency gained is hard to reproduce manually. If sametests are executed in a different hardware or system configuration, the resultsmay give valuable insight into differences or possible bottlenecks in the sys-tem.

• Possibility to reuse tests. Compared to manual tests, automated test have thesame design costs, but also additional costs for the automation. Automationpays off in re-using the tests with reduced execution time in consecutive soft-ware releases, and in possible portability between different test setups.

16

• Faster time-to-market. When the initial implementation of automated tests iscomplete, the test execution time will be substantially shorter. This reducesthe time that it takes for a new version of software to enter the market.

• Increased confidence. An extensive set of automated test cases increases theconfidence and trust in a software release.

In the same literature, the negative factors are considered to be:

• Too high overall expectations. Emphasis is put on the tooling instead on theeffort needed in building and maintaining a successful test automation regime.

• Poor testing practices do not improve by using test automation. Practices con-sist of the organization, documentation and quality of the actual tests. If au-tomation is built on poor quality, the result is poor quality delivered faster.

• Expecting that automated test cases find many new defects. Test cases mightfind defects in their initial design and execution phase, but in regression thedefects are usually reflections of source code modifications.

• Assuming that the software does not contain defects if automated test suitedoes not find any. It must be considered that even the tests can contain defectsor otherwise be incomplete.

• Underestimating the maintenance effort needed to keep test cases up to date.Changes in software may require updates in automated test cases.

• Technical and interoperability issues between test automation products. Testautomation tools themselves can contain defects that may severely affect thetest automation regime. Some old closed-source tooling might be irreplaceableand lack support.

The difference should be highlighted between verification by a skilled tester andchecking by an automated test case. Even small-sized software can have many inputcombinations and the tester’s experience is needed to carefully select the test casesthat should find most of the defects in the given time frame allotted for the testingactivities. The tester is able to define whether the outcome of the test case is correctfor the given situation, and possibly react and conform to possible changes in exe-cution. In contrast, test automation tools generally compare a set of test outcomes

17

to expected ones and is blind for what is happening around the checked value. Thiscomparison should be separated from verifying what a skilled tester does.

According to [12], test automation can provide results in two ways. Either it canlimit the effort needed for testing activities, or increase the number of tests that canbe carried out within a limited period of time. Reducing effort is mainly accom-plished by automating mundane manual tasks

Test automation can be applied to almost any testing phase, but some phasesbenefit from it more than others. Test-driven development (TDD), behavior-drivendevelopment (BDD) and acceptance test-driven development (ATDD) are all method-ologies that encourage developers to first write tests for a software feature before im-plementing it. For TDD tests are low-level, such as unit or integration tests. For BDDand ATDD tests are usually more high-level, such as functional or acceptance tests.The main idea is that once the actual feature is properly implemented previouslyfailed tests will be passed and the feature has an automatic runnable regression testin place.

2.4.1 Requirements of the organization

The research article [28] by Persson and Yilmaztürk gathers common pitfalls of testautomation implementations from literature. It reports on the implementation oftest automation at ABB, and the attempt to recognize and avoid these pitfalls. More-over, it evaluates the varying effects of pitfalls on the organizational level. While theprevious section listed the benefits and negative factors of test automation, this arti-cle focuses on some factors of test automation on the organizational level.

• In order to implement test automation, organizational maturity should be at alevel where it can handle process improvements in a structured manner. Theconcepts and terminology of test automation and automated regression testingshould be clear, and communicated effectively throughout the organization.

• Test teams working on test automation need well rounded skill sets in pro-gramming, testing, project management and technical skills like networks anddatabases. The need for competence in these areas may be underestimated. Itis also mentioned that the project knowledge gained during test automationefforts should be secured, and that the staff should not be made up entirely ofexternal resources.

18

• The importance of testware (software tooling used in test automation) reusabil-ity, repeatability, maintainability and modularity is highlighted. Attentionshould be paid to guidelines for test script development, as a lack of guidelinesendangers the reusability, repeatability, and maintainability of the testware.

• The needs for test result reporting should be discussed thoroughly within theorganization and with the stakeholders. It is easy to create reports that containuseless information, and where the useful information is not in the best possi-ble format. The article states that "It was a big gap between the first internalproject proposal, and the final report template". If not automated, actual testreporting creates additional and often unconsidered effort.

2.4.2 Continuous Integration

Continuous integration 2.3 is an integral part of today’s software development, en-forcing the use of TDD/BDD/ATDD development methods (as described in 2.4).The name "continuous integration" originates from a practice in Extreme Program-ming (XP) development process. By developing code using the before-mentionedpractices, it is ensured that the implemented code is covered with a set of tests.These tests act as a "safety net" when a the existing codebase is modified, i.e. for achange in a feature or a refactoring.

Let us consider a case where a developer has been working in a test-driven wayand wants to commit his work into the version-control system. First, it must beensured that the local working copy passes the tests, by running them in his per-sonal development environment. After the local tests are passed, the code can bemerged in to the main branch of the project in version-control system. The "con-tinuous integration" part happens when this commit automatically triggers a build,containing the freshly committed code and running a larger number of tests, thusintegrating the change and ensuring that the commit does not break any other partor functionality of the build.

There are other recommended practices for successful continuous integration.When a build breaks, i.e. a test or tests fail, the most important task is to fix thebuild. This ensures that the main branch always remains healthy, and that no devel-oper deviates too far from it with his local working copy. To achieve this, first, thecommits should not be big and monolithic but preferably smaller and incrementalto reduce problems with integration. Another thing brought up earlier is that in or-

19

Figure 2.3: Continuous integration

der for continuous integration to work, the version control should have a mainlinebranch, and the development should be made to this main branch. Some reasonablebranches for example for bug fixes are acceptable. Third, the build should be auto-mated and the tests should be part of the automated build process. Fourth, everycommit should build the mainline and the build process should be fast, in order toget instant feedback on a possibly broken build[13].

2.5 Software Testing Maturity Model

There are several process models that are used as frameworks in assessing and im-proving the software development process (including the testing process) in organi-zations. In this chapter, we will review a process model that concentrates solely onimproving the testing process in organizations, i.e. the testing maturity model (re-ferred to as TMM from this point). TMM was developed by Burnstein et al in 1999,[8] and it is based on Capability Maturity Model (CMM), which in turn was basedon a study conducted in U.S. Department of Defense. CMM is a staged architecturalmodel, where the architecture prescribes stages that an organization must proceed

20

through in a given order in order to improve their software development process.TMM uses several features from the CMM and is staged in a similar fashion[9]. Thefive stages in TMM are called "maturity levels". Each level has a description that canbe used to assess the current state of the process maturity and the goals that shouldbe pursued in order to reach the next maturity level in an "evolutionary path".

2.5.1 Maturity levels and goals

As stated in overview, there are five maturity levels of the testing process that ac-tually indicate the testing capability of the organization (illustrated in figure 2.4).The first level is being an unmanaged, chaotic ad-hoc type of testing: from there theorganization can evolve step by step to the fifth level, where the test process and in-frastructure are well-defined, managed and measurable, and can then be fine-tunedand optimized.

Each maturity level contains the maturity goals, which in turn contain maturitysub-goals. These subgoals determine the needed accomplishments needed for aparticular level, along with scopes and boundaries. An organization adapts to goalsby committing to the required activities, tasks and responsibilities. In the comingsubsections each of these levels with their respective properties will be reviewed inmore detail.

Level 1 - Initial At maturity level 1 there is no established testing process, andtesting merely resembles debugging. The specification of the software does not existand there is a lack of resources, tools and properly trained staff. The testing activitiesattempt to show that the software works, and the testing and implementation workis combined together. There are no maturity goals at level 1.

Level 2 - Phase definition At maturity level 2 the testing is recognized as a sep-arate phase that follows the implementation phase. Test planning and design maytake place after completion of the implementation, as the completed code is seen asa base to the test activities. Possible defects that occur in the specifications are finallyfound in the testing phase. Basic testing levels and techniques are used.

In level 2, the maturity goals are to develop testing and debugging goals, initiatea testing planning process and institutionalize basic testing techniques and meth-ods.

Level 3 - Integration At maturity level 3, testing is integrated into the softwarelife cycle. Test planning activities start in the requirements phase, not after the im-plementation phase as in level 2. The goals of level 2 have been attained and the

21

Figure 2.4: Life cycle of the five maturity levels of TMM with their respective goals[8].

22

goals of level 3 have been implemented. There is a software test organization thatoverlooks and controls the testing, along with the technical training that focuses ontesting. The test review program is not yet present at this level.

In level 3, the maturity goals are to establish a software test organization, estab-lish a technical training program, integrate the testing into the software life cycle,and control and monitor testing.

Level 4 - Management and Measurement At maturity level 4, measurement andquantification are part of the testing process. Reviews cover the entire softwareprocess and complement the actual testing. Test cases are stored in a database andthe defects are properly logged in another database with severity levels. Software isalso tested for quality attributes, like maintainability, reliability and usability.

In level 4, the maturity goals are to establish an organization-wide review pro-gram, a test measurement program, and software quality evaluation.

Level 5 - Optimization, defect prevention, quality control At maturity level 5,the testing process is solidly defined and managed. The testing process can be opti-mized and further fine tuned. Cost and effectiveness of the testing can be monitored.Defect prevention is practiced along with quality control.

In level 5, the goals are defect prevention, quality control and test process opti-mization.

23

3 The GSM Network

This chapter examines the history of the GSM and addresses the basic standardiza-tion and technology aspects. Furthermore, the base station subsystem componentsand their roles are defined.

3.1 History

Before GSM, there were several different mobile cellular networks. NMT in Scan-dinavia, AMPS in North America and TACS in Europe. France, Germany and Italyhad also implemented their own national systems. This fragmentation of technolo-gies caused high costs of mobile devices and network equipment and made Europe-wide roaming impossible. In the year 1993, Western Europe had population of 300million but only 7,4 million cellular phone users.

As pointed out in a contribution by the Netherlands at the CEPT conference inJune 1982, there was a significant risk that unless a concerted action for a commonEuropean mobile system was started quickly, the 900 MHz band would be takenover by incompatible national systems. That would mean that the change to buildEuropean-wide system in the 20th century would be lost. Initially CEPT proceededwith the harmonization of existing national systems, but soon it was agreed thatinstead of harmonization, there should be more future-proof solutions that wouldbe based on new technology. The Groupe Spécial Mobile (GSM), with membersfrom European mobile network operators, was formed and started the work for thenew mobile cellular network solution.[18]

The work of GSM started in 1982; the intention was that the new system wouldbe available in the early 1990s. Work was based on a strategy and action plan fromCEPT, which set out the following requirements for the system:

• To be able to share the 900 MHz spectrum with the present analog systems

• Support for mobile stations and their free circulation

• Mobile stations should be able to operate in all European countries

24

• High spectrum efficiency

• Telephony as main service, but to offer attractive non-voice services

• Strong security support

As the workload of the GSM group increased, several independent subgroupswere established. A structure for specifications, the system architecture and the con-cept of services were created. The security group and the digital radio transmissiongroup had their concepts trialed and evaluated. The work done from 1984 to 1987resulted in a future-proof basic parameter set for the GSM system. The parameterswere advanced but could be implemented with existing technologies.[18]

The first set of features from the CEPT action plan was selected and specified forGSM Phase 1. It contained the radio subsystem, core network and security archi-tecture along with the Subscriber Identity Module (SIM). It offered telephony, inter-national roaming, call barring and call forwarding services. In 1989 the GSM groupwas transferred to newly established European Telecommunications Standards In-stitute (ETSI). The actual work remained unchanged, but this allowed manufactur-ers to take part in it as ETSI members. At the same time, the UK requested a mobilecellular network that would work in the 1800 MHz band.[18]

GSM Phase 2 started in 1991. It would improve the existing functions from GSMPhase 1 and specify new functions that were omitted in the previous phase. Newcontent in GSM Phase 2 included data service enhancements, supplementary ser-vices, half-rate speech codec and the 1800 MHz system, to be known as GSM1800.Moreover, test specifications for type approval tests of mobile stations were created.It was obvious that the GSM evolution would not stop with Phase 2; confirmation offuture extension possibilities was also important, as well as the compatibility withthe specifications of GSM Phase 1. In 1995, the GSM 2 Phase 2 specifications werefrozen.[18]

In 1993 Nokia contributed to the field of GSM development with a one-page doc-ument titled "GSM in a Future Competitive Environment". This document statedthat by 1994, there would be GSM Phase 1 networks in 40 countries, and that thecurrent cross-phase compatibility mechanisms would prevent the further develop-ment of GSM. The main message was that until next decade and the upcoming thirdgeneration mobile network systems, GSM should remain competitive and offer bet-ter speech quality and improved system capacity, and that the GSM standard shouldtherefore be enhanced by 1997-1998. In a workshop held in Helsinki in 1993, it was

25

realized that GSM Phase 2 should be considered as a platform with true potentialfor evolution.[18]

GSM Phase 2+ fixed the cross-release compatibility issues. It provided new ser-vices and features, i.e. support for ISO Unicode as the second character set in SMS.Adaptive Multi Rate (AMR) codec was put to use to further improve the speechquality. The age of the mobile internet dawned with Wireless Application Protocol(WAP) and the High Speed Circuit Switched Data (HSCSD), which provided a max-imum of 42 kb/s of user data transfer. Higher speeds would be essential, and so theGeneral Packet Radio Service (GPRS) was standardized, requiring large changes inthe GSM radio subsystem and two new core network elements, the Serving GPRSSupport Node (SGSN) and Gateway GPRS Support Node (GGSN). GPRS enableddata transfer speeds up to 100 kb/s shared by multiple users. Even faster data rateswere to be achieved with Enhanced Data rates for GSM Evolution (EDGE). By re-using the GSM radio channel structure and TDMA framing with new modulationand coding techniques, data rates could be as high as 384 kb/s.[18]

The first GSM call in a commercial network was made on July 1, 1991. The net-work was owned by the Finnish operator Radiolinja, which now operates underthe name Elisa. The network was built by Telenokia and Siemens. From there, thenumber of global GSM subscribers grew steadily until the peak of 2010-2013.

In the time period of 1996-2000, the idea for a third generation of cellular net-works was conceived. It was based on GSM core network evolution and a newradio subsystem. To achieve a global working structure, also needed for GSM, theThird Generation Partnership Project (3GPP) was created. 3GPP has existed since1998 and has developed specification releases for 3G and 4G (LTE) systems.[18]

3.2 Present and future

Today, more than 90% of the world’s population is covered by GSM services [15].Based on estimates available in Ericsson’s Traffic Exploration Tool, in 2011 the amountof subscriber data traffic exceeded voice traffic. The growth of data transfer is goingto be significant in the coming years, as shown in figure 3.2. The customer demandfor being "always connected" and being able to stream i.e. high quality music andespecially high-bandwidth video requires the cellular network to be able to delivera high volume of data per user. Clearly, GSM with its data transfer abilities is not themost suitable technology for this kind of task. Estimate of GSM and LTE + WCDMA

26

Figure 3.1: Estimate of global GSM and LTE + WDMA subscribers from 2010 to 2020[10].

are displayed in figure 3.2.The "GSM sunset" is becoming reality, and some major operators shut down their

GSM networks at 2017 [4][30]. The available frequency spectrum is highly regulatedand expensive, and the operators refarm the spectrum to match the demand of mo-bile broadband. Still, there are services like GSM-R and many IoT/M2M solutionslike remote telemetry and security solutions, that use GSM as communication chan-nel, and the impact especially for the IoT/M2M is considerable.

3.3 Technology

The 3GPP specification defines a total of fourteen frequency bands for GSM, rangingfrom 380 MHz to 1900 MHz. GSM bands used in Finland and the rest of Europe areGSM 900 (900 MHz) and DCS 1800 (1800 MHz) whereas the used bands in Northand South America are GSM 850 (850 MHz) and PCS 1900 (1900 MHz). The fre-quency spectrum allocation per band is varied by the used standard. For GSM-900,there are three different standards: P-GSM, E-GSM and R-GSM. The uplink anddownlink frequency spectrums of P-GSM are both 25 MHz wide, containing 124physical channels. Total radio frequency spectrum allocation of P-GSM is 50 MHz.

27

Figure 3.2: Global smartphone traffic, data vs. voice [10].

28

For E-GSM, uplink and downlink frequency spectrums are 35 MHz wide and thetotal spectrum usage is 70 MHz. R-GSM uses 39 MHz uplink and downlink spec-trums, totaling 78 MHz.

Figure 3.3: Frequency channels in GSM 900 (E-GSM) [26].

A GSM band is divided in to frequency channels, and the E-GSM for examplecontains 124 channels (uplink and downlink, figure 3.3). The channel width is 200KHz. The first and last 200 KHz of a band are not used for safety reasons. Thechannel access method that GSM uses on air interface is a combination of frequencydivision multiple access (FDMA), which mean the frequency separation inside aband and time division multiple access (TDMA), which means that traffic is con-tained inside a TDMA frame, which contains timeslots. Timeslots are referenced asphysical channels of the air interface. The FDMA-TDMA structure is shown in Fig-ure 3.4. TDMA timeslot can be a traffic channel, a control channel or both. Thesechannel types are called as logical channels of the GSM.

• Traffic channels A traffic channel can be i.e. a full-rate or a half-rate channel.In the former, the speech data rate is 13 kb/s. The half-rate principle is thatsingle users speech data is carried only in every other TDMA frame, thus thecapacity is doubled. Due to improved codecs the quality of speech does notsuffer significantly from this.

• Control channels Control channels are used when a mobile station (MS) en-

29

ters or leaves the network, in the tracing of MS, initiation, maintenance andending of the calls. High-level separation of control channels are broadcastchannels (BCH) , common control channels (CCCH) and dedicated controlchannels (DCCH). These contain multiple associated channel types for differ-ent purposes, i.e. synchronization, paging and connection management andauthentication.

Figure 3.4: GSM uses a combination of TDMA and FDMA [26].

GSM network is also able to transfer data. The basic data transfer form is the cir-cuit switched (CS data) data, which uses the normal timeslots and has a maximumdata transfer rate of 9,6 kb/s.

General Packet Radio Service (GPRS) provides faster means for data transfer.GPRS and its successor EDGE transfer data as bursts, opposed to the contiguoustime slot reservation of CS data, and therefore enable the simultaneous data trans-fer for multiple users. The data transfer rate per user can be increased by assigningmultiple time slots for the data. Maximum peak data transfer rate with all times-lots assigned to data transfer and by using the most efficient channel coding al-gorithms is 158-171kb/s. GPRS also allows operators to bill based on amount oftransferred data, instead of on connection time. GPRS requires changes in the GSMnetwork, such as packet control unit (PCU) functionality in the BSC and additional

30

network elements service gateway support node (SGSN) and the gateway GPRSsupport node (GGSN).

Enhanced GPRS (EDGE) is based on the GPRS technology and provides fasterdata transfer rates of 384kb/s, but it should be noted that this also varies based onamount of used timeslots. Increased data rates are obtained by using more efficientchannel coding and by using multiple carriers, thus expanding the number of times-lots used. The EDGE is considered as "2,5G", an additional step between the 2nd andthe 3rd evolutions of the mobile network.[2][1][27]

Figure 3.5: Frequency reuse in cellular network [26].

31

3.4 Mobile Station

The MS (mobile station) consists of the physical equipment and the subscriber iden-tity module (SIM) card.

The MS (mobile station)consists of the physical equipment, such as the radiotransceiver, display and digital signal processors, and the SIM card. It provides theair interface to the user in GSM networks. As such, other services are also provided,which include:

• Voice teleservices

• Data bearer services

• The features’ supplementary services

The MS also provides the receptor for short message service (SMS) messages,enabling the user to toggle between the voice and data use. Moreover, the mobilefacilitates access to voice messaging systems. The MS also provides access to thevarious data services available in a GSM network.

The SIM provides personal mobility so that the user can have access to all sub-scribed services irrespective of both the location of the terminal and the use of aspecific terminal. You need to insert the SIM card into another GSM cellular phoneto receive calls at that phone, make calls from that phone, or receive other subscribedservices.

3.5 Base Station Subsystem

The GSM Base Station Subsystem (BSS) consists of following three network ele-ments. Elements and interfaces of BSS are shown in Figure 3.6.

• The Base Station Controller (BSC) is the central network element of the BSSand controls the radio network. This means that the main responsibilities ofthe BSC are connection establishment between the MS and the NSS, mobilitymanagement, statistical raw data collection, as well as Air and A interfacesignalling support.

• The Base Transceiver Station (BTS) is a network element maintaining the Airinterface. It takes care of Air interface signaling, Air interface ciphering, and

32

speech processing. In this context, speech processing refers to all the functionsthe BTS performs in order to guarantee an error-free connection between theMS and BTS.

• The Transcoder (TC) is a BSS element taking care of speech transcoding, i.e. itis capable of converting speech from one digital coding format to another andvice versa [26].

3.5.1 Base Transceiver Station

A base transceiver station is a physical site from where the radio transmission inboth the downlink and uplink direction takes place. The radio resources are thefrequencies allocated to the Base Station. The particular hardware element insidethe BTS responsible for transmitting and receiving these radio frequencies is appro-priately named "transceiver" (TRX).[26] Other equipment belonging to a BTS wouldbe a power supply (/w a battery backup), combiner, antenna cables, possible mastamplifiers and the actual antennas.

A TRX handles the traffic of one radio channel and each radio channel is splitinto eight timeslots (TS). Each channel can have maximum of eight simultaneoususers when full-rate codec is used and 16 if half-rate codec is used. Some of thetimeslots are used for signaling which may reduce the number of speech timeslotsaccordingly.

A cell is an area that is covered by one or more TRX. It can be done as omnidi-rectional (round) coverage, or be more directed, covering only a specific area. Thefrequency reuse in cells is described in Figure 3.5. The MS communicates with BTSvia Air interface and the BTS is connected to BSC via Abis interface.[27]

3.5.2 Base Station Controller

The base Station Controller (BSC) manages the radio resources of its own area. Itknows the available resources, such as free channels in the cells and signal qualityof the ongoing calls. One or more base stations are connected to the base stationcontroller. A location area (LA) consists of groups of cells, and the BSC can controlmultiple location areas; moreover a location area can have one or more controllingBSCs. MS updates its own location area information to controlling BSC when itmoves from one LA to another. When a call comes to a MS residing in certain LA,the BSC sends a request to all cells that belong to that LA. MS gets the request and

33

Figure 3.6: High level view of subsystems [26].

34

the call can be formed. A MS will request a channel, if it is the originating party ofthe call. In either case, the BSC assigns a traffic channel to the MS.

BSC also decides whether a handover should happen. A handover is done byassigning MS to a new radio channel, that might be available in same or anothercell, and the decision may be caused by a MS moving from cell to another or basedon measurement data of signal strength and quality, which are collected from MSand BTS. Multiple types of handover exist and a handover can be done inside a cell,from cell to cell, inside a BSC, between two BSCs and between two MSCs. In thelatter case, MSC also participates in decision-making.

BSC controls the radio interface parameters, which include frequency hoppingsequences of traffic channels, power control, cell location area assignments andchannel timeslot usage setup. BSC is connected to MSC via the A interface andto the Serving Gateway Support Node (SGSN) via the Gb interface.

3.6 Network Switching Subsystem

The Network Switching Subsystem (NSS) has several network elements. It is con-nected to BSS by the A interface and to NMS by the X.25 interface. In the followingsection we take a brief look at the network elements of NSS.

Mobile services Switching Centre

The Mobile services Switching Centre (MSC) is responsible for controlling calls inmobile network. This means tasks such as connecting, maintaining and tearingdown the calls in its own area. It identifies the origin and destination of a call (ei-ther a mobile station or a fixed telephone in both cases), as well as the type of call.A MSC acting as bridge between a mobile network and a fixed network is called aGateway MSC. Usually the registers such as HLR and VLR are physically integratedwith MSC.

Home Location Register

The Home Location Register (HLR) maintains a permanent register of subscribersand stores subscriber and billing information as well as information about addi-tional subscriber services. Physically, it is a database server that stores information.

35

A network requires at least one HLR, that is connected to some MSC through Cinterface. Every subscriber is registered in one and only one HLR register, whichis specified by the operator and the subscription normally ends when subscriberrecord is removed from HLR.

HLR database stores constant information such as MSISDN number, IMSI iden-tity, subscription type and encryption type. It also stores frequently changing infor-mation, like subscribers VLR location and for example call forwarding information.

Visitor Location Register

Visitor Location Register (VLR) is often integrated with MSC, hence the commonname MSC/VLR. It stores subscriber info, which is automatically requested fromHLR when MS moves into the area of the MSC. When MS moves into the area ofanother MSC, the subscriber info is removed from the previous VLR and updatedto new one. A VLR stores the following subscriber info: MSISDN, IMSI, TMSI,MSRN, LA, encryption parameters and other service information. Essentially theVLR stores same subscriber info as the HLR, but with additions such as more spe-cific location info. VLR is attached to MSC via the B-interface.

Authentication Centre

The Authenticaton Centre (AuC) stores secret identity numbers of the subscribers.The identity number is generated when a subscriber joins a network. The identitynumber is compared to MS’s identity number during the call formation and the callis rejected if the numbers do not match. While AuC is not a mandatory part of aGSM network, it is widely used. AuC connects to MSC via the H-interface and theprotocol is left unspecified.

Equipment Identity Register

Each MS has an equipment identity (IMEI), which is stored in the Equipment Iden-tity Register (EIR). It can store identities of i.e. faulty or stolen equipment and blockthe use of certain MS based on that information. The EIR has white, grey and blacklists for equipment identities. On the white list is the equipment, that is acceptedand can be used in the network. On gray list comprises the equipment, that is fol-

36

lowed and may have a temporary permission to network due to type acceptancereasons. Equipment that may not be used in the network is on the blasck list. Evenif equipment is blacklisted, emergency calls can be made. EIR is not a mandatorycomponent of the GSM network. It communicates with MSC via the F-interface.

3.7 Operations Subsystem

The Operations Subsystem (OSS) contains one or more Operations and MaintenanceCenters (OMC). Through the OMC, the operator carries out O&M tasks such as up-grading the software of network elements, modifying the network element param-eters, and supervising the operational statuses of the network elements. Other im-portant tasks carried out via OMC are subscriber information management such asbilling, and the management of mobile stations. OMC therefore provides the opera-tor with a centralized service from where network can be operated and supervised,as well as a source for information about network use and performance throughmeasurement data.

Based on measurements, collected for example from the signaling or transmis-sion channels of the BSC, the operator can i.e. find out why there are dropped callsand whether the fault persists and thus point to a fault in the network planning.Network usage statistics are also provided, and the operator can monitor the loadper element and review whether if there is a need for additional capacity in a certaingeographical area.

OSS itself is not tightly specified, but instead left free for the vendor to imple-ment. OSS is connected to the network through the Q3-interface, which for exampleapplies the frequently used X.25 -protocol for information exchange.

37

4 Nokia implementation and test process

As discussed previously, the GSM is a standard. Telecom equipment vendors fol-low that standard to implement their own sets of network elements. From theseelements, network operators can form a complete network that provides communi-cations services to their subscribers. Standardization should ensure that most of thenetwork elements can work together in same network, regardless of vendor.

This chapter provides additional background for chapter 5 by looking into Nokiaimplementation of the GSM Base Station Controller. It introduces the DX 200 plat-form and the structure of the BSC and it’s user interfaces. Operations & Maintenance(O&M) functional testing is discussed, as well as the related testing subjects. Last,the used test automation framework and the test automation process are defined.

4.1 DX 200 platform

The Nokia implementation of GSM network elements is mostly based on the DX200 switching platform. The development of the DX 200 digital switching platformstarted in the early 1970s. As shown in Figure 4.1, the platform consists of a hard-ware layer, a computing platform and a switching platform. On top of this lies theapplication platform.

Figure 4.1: Layers of DX 200 platform [26].

38

The first commercial installation was a fixed line installation, which was com-pleted in 1982. Network elements using the platform are for example Base StationController (BSC), Mobile-Services Switching Centre (MSC), Visitor Location Regis-ter (VLR), Home Location Register (HLR), Authentication Centre (AC), EquipmentIdentity Register (EIR) and Transcoder (TC).

Multiple early generations of DX 200 based GSM network element hardwarewere built into cabinet structures holding cassettes of plug-in units, which wereconnected together by cabling from the back of the cabinet. The last addition to thisconstruction type was FlexiBSC, which was brought to market in 2010. The multi-controller platform introduced in 2012 differs radically from the old DX hardware,as it uses modern rack-mounted and extensible module-based technology with allIP-based interfaces. Some variants of radio network controller products, such asthe 3G equivalent of BSC, the Radio Network Controller (RNC), are also built onmulticontroller platform. This provides a valuable upgrade path for network oper-ators, as the multicontroller BSC can be converted into multicontroller RNC with asoftware upgrade [24][25].

4.2 Base Station Controller

The functional unit composition of the Base Station Controller (BSC) varies depend-ing on the model. Older cabinet-, cassette- and plug-in unit-based variants rangingback from BSCi to the latest FlexiBSC differ from modern multicontroller architec-ture. The former is built from highly task-specific computer units, whereas the lat-ter uses more general multi-purpose hardware packed into a server-style enclosure.The most common functional units are:

• Operation and Maintenance Unit (OMU) is an interface between the DX 200BSC and a higher-level network management system and/or the user. TheOMU can also be used for local operations and maintenance. The fault indi-cations OMU receives from the BSC produce local alarm printouts to the user,or send the fault indications to the OMC. In the event of a fault the OMUautomatically activates the appropriate recovery and diagnostics procedureswithin the BSC. When a fault occurs on the OMU, the active MCMU takesover its duties.

• Marker and Cellular Management Unit (MCMU) performs the control func-

39

Figure 4.2: A structure of a BSC [26].

40

tions of a switching matrix and the BSC-specific management functions of ra-dio resources. The marker functions of the MCMU control the Group Switch.These control functions include the connection and release of the circuits of theswitching matrix.

The cellular management functions of the MCMU assume responsibility forcells and radio channels that are controlled by the DX 200 BSC. The MCMUreserves and keeps track of the radio resources requested by the MSC or thehandover procedures of the BSC. The MCMU also manages the configurationof the cellular network.

• The duties of Base Station Controller Signaling Unit (BCSU) are highly de-pendent on traffic. It provides the SS7 signaling of the A interface and the Airinterface channel control and LAPD signaling of Abis interface.

• Group Switch (GSWB) is responsible for switching the speech channels. Theoperation is supervised by the MCMU. The DX 200 switching network is fullydigital and non-blocking.

• Exchange Terminal (ET) work as connection points to the A, Abis and Gbinterfaces. The ETs adapt the incoming PCM circuits (encoding and decoding)from and to the Group Switch and synchronize to the system clock. ET can actas an Ethernet interface, providing IP-based connectivity for the previouslymentioned interfaces.

• Message Bus (MB) provides connection between computer units (OMU, MCMU,BCSU).

The redundancy of functional units is an important factor in operational relia-bility of the equipment, which should be highly available. In DX 200 BSC there aretwo primary redundancy types in the functional unit hardware.

• 2N for duplicated units, when there is an active one and a spare one.

• N+1, N+m for N times of units for needed dimensioning of the system and atleast one redundant unit or number of m redundant units, which is propor-tional to N.

41

4.2.1 Software

New software builds can be installed while the BSC is running an old software ver-sion. The switchover to new software package causes short downtime. Besides anew versions of application software, the build also contains software for all plug-in units of the functional units. The software of plug-in units is either updatedautomatically, or initiated via an MML command when the BSC is up and running.

4.2.2 User interfaces

A human user operates the BSC with a command language called Man-MachineLanguage (MML). The MML interface is provided by the OMU and is used over astandard bidirectional text-based telnet protocol. Text-menu based MML allows theuser to carry out Operations & Maintenance (O&M) tasks such as basic configura-tion of the BSC and view the operational state of the functional units (Figure 4.3) andradio network. This is also the interface that is mostly used for functional testing ofthe equipment.

Figure 4.3: Example output of MML command from BSC.

More control over functional units and for example the file system can be ob-

42

tained when the user interacts through service terminal. The user can connect tothe service terminal through an MML command via the OMU or directly through aCOM port located in the OMU or another computer unit.

At first, interfaces between network elements such as A and Abis used PCM asdigital data transfer format. Altough PCM is still supported by hardware vendors,today’s approach is an IP based interface communication.

4.3 Operations & Maintenance Functional Testing of the BSC Soft-

ware

Operations & Maintenance (O&M) basically means the functionality that operatorsuse to put and maintain the network in an operational state. A BSC, as well asmost network elements, provides localized O&M access that is used to commissionthe network elements in use with basic configuration. Localized access means thatsomeone using the O&M provided connectivity must be physically present at thesite where the network element is located, or be able to access a network that allowsneeded connectivity to the element[23].

O&M functional testing is mostly done with real BSC environments. As multiplegenerations of BSCs are supported today, with their respective hardware configura-tion and capacity variants, the total number of needed hardware combinations (BSCand BTS variants combined) to carry out the testing is rather large. When functionaltest activities for a BSC software release are planned, the available environment re-sources are very important. The main factors for test environment planning are theBSC generations, capacity, interface types, and base station types and configura-tions.

The main and most used interface for O&M functional testing of the BSC is theMan Machine Language (MML) connection over Man Machine Interface (MMI). Itis a text-based command interface, which can be accessed over a standard telnetor SSH connection. Another important interfaces are the service terminal connec-tions to the computer units, which also operate over telnet connections. While theMML interface provides the control and observation interface for the base stationcontroller and radio network functionalities, the service terminal connection can beused to observe and control the internal actions of the computer units.

Basic testing activities are usually carried out by giving commands through theseinterfaces and observing the results in the system. Test activities may for example

43

also include intervention to power feeding, network connections, and other exter-nal resources. Real mobile devices are used to verify the connectivity in the testnetwork.

The O&M functional testing of BSC covers a large number of software featurefunctionality. Testing is separated into subareas that are handled by separate testingteams. A few examples of these subareas are:

• The recovery and capacity tests use different hardware, interface and basestation configurations and combinations to verify the functionality after unitswitchovers (from controlling unit to a redundant unit, as described in 4.2),unit restarts and controlled and uncontrolled (i.e. power break) system restarts.Tests also ensure that calls stay connected if for example unit switchover hap-pens due to a faulty unit. These tests may also require large radio networkconfigurations, which accordingly increase the load to the system. Test casesrequiring the maximum or a large radio network are executed using simulatedradio network environment, as it would not be feasible to host and configureall needed BTS hardware for the test setups.

• Radio network functionality and configuration testing verify the functional-ity of different radio network and cell configurations along with the Abis in-terface. The tests may for example contain adding and removing radio net-work elements from the configuration and altering their states and parame-ters. Again, the tests may be executed against real or simulated radio networkhardware.

• The measurements are collected from BSS. They measure characteristics likesignal strengths, dropped calls and handover information from the network.The measurement info is stored in the BSC and can be further analyzed inNMS. The measurement info is further processed to Key Performance Indica-tors (KPI). This test area verifies the functionality of these measurements.

4.4 Holistic Integration Tester

The Holistic Integration Tester (HIT) is perhaps the most widely used tool for testingthe internal services of DX200 platform at Nokia Networks. The tool provides meansto telnet and SSH connections and enables the user to create scripts with a languagethat is a subset of the C programming language. HIT also provides a collection of

44

system functions along with debugging capabilities to assist in script creation. Thescripts themselves are used to execute MML commands and parse the outputs andautomate configuration creation, as well as to assist in test case execution. In termsof telnet connectivity, HIT could be considered as a close relative of an ordinarytelnet client such as Putty.

4.5 Test Automation Framework

The software tools used for automated testing of the BSC were mostly developedby tool teams inside Nokia. The test automation framework that was used in theO&M functional testing of the BSC was also developed internally. The frameworkused the HIT tool and it’s scripting capabilities as foundation. Large part of testautomation development for the BSC was the development of the test automationframework.

The test automation framework consisted of two main parts. The core of theframework contained the general functionality needed for test cases, test sets, re-porting and various file- and MML connection-handling functionalities. The corepart also contained the libraries needed for controlling external devices, such as callgenerators. The second main part was the system libraries, which provided inter-face like functionality for the BSC. System libraries were hardware-specific and alsocontained the MML abstractions described in 5.1.4.

New framework functionality was constantly needed to carry out the automa-tion tasks. A rough estimate of the tool development time would be along the linesof half of the total development time, other half being actual test case development.As stated by Bach [5] and Hendrickson [17], tool development should be consid-ered as a proper software project. Good software development practices such asbug reporting and testing of the automation framework were practised during thedevelopment efforts.

4.6 Test Automation Process

The motivation for the subject of the thesis came from my work experience in theposition of a test engineering at Tieto Finland Oy. I was responsible for implement-ing the automated test cases for the O&M functional testing phase of base stationcontroller software. I was also responsible for the implementation and maintenance

45

of the test automation infrastructure and analysis and for reporting of the test runs.Very strong domain knowledge was needed in the test case implementation andanalysis phases and I benefited from domain support of seasoned test teams, withindividuals even having over decade worth of experience in testing the product.After exploring the [12] by Fewster and Graham, the setting where there is a toolexpert and experts with domain knowledge, much resembled what was presentedin the book.

When the infrastructure was in place and the foundation for test automation waslaid, the test automation tasks were to be distributed to the test teams. This meantthat formerly manual test cases were to be automated by using the test automationframework described in the previous subchapter.

The work environment was fast paced, and being as the subsequent test phasesusually waited execution results from previous phases, the test executions had ratherstrict time frames. Test executions themselves could not be faster than the test en-vironment allowed, and usually tests were run overnight in multiple environmentssimultaneously, and so that the test results were ready for analysis in the morning.Tests needed to be reliable, and to provide enough information for analysis in orderto keep duties on schedule. The typical work-flow in test runs was to load the newsoftware build in the environments, check the need for updates of embedded bootsoftware of the units and check the basic configuration of the BSC, interfaces, andradio network as well as call connectivity. After the environment was verified tobe in a correct starting state, the automated test sets was started. After execution,the test results were analysed and possible failures were discussed with responsibletest team members, whose teams owned the test cases. Re-executions of cases withsufficient monitoring and logging were often needed for the fault report.

4.6.1 Test automation infrastructure

The used continuous integration infrastructure is presented in Figure 4.4 for twotested BSCs. Jenkins is a software that can be used to control and orchestrate CIexecutions and one way to configure it is to have a Jenkins master node that providesa web interface where test executions can be started and reports can be viewed. TheJenkins master node can also "publish" test reports to another master node, wheredifferent views of test results can be composed, e.g. for management purposes.Another Jenkins node type is the slave node. These slaves connect to the masternode and typically contain the test automation framework and the test cases. In this

46

case, the slave nodes also connected to the BSC under test via an MML connectionand other testing tools, such as the call generators and other peripherals. The slavenode to BSC under test relation was 1:1, one slave node per one BSC. Everything ina slave node (environment parameters, test cases and sets and the test framework)were under version control. The development work usually took place on the testengineer’s own environment and the new test cases and features to the test tool wereeventually updated to slave machines via version control.

Figure 4.4: Test environment infrastructure for two BSCs.

47

4.6.2 Extending the framework

The execution steps of course came from carefully planned automated test cases,which up to this point were manually executed. Therefore, the instructions for testcase automation were clear. Problems started first to appear, when these cases wereto be executed in large batches in the test environment BSCs, and then when thecases needed maintenance. Questions such as following were asked:

• What state should system be in for a test case?

• Should a case clean up after itself? What would be the best way?

• This test case leaves a perfect setup for the next one, why can’t I just executethem sequentially?

• What kind of parameters should I supply a case with? How should I namethem?

• How should the test case act if a fault is encountered and the test case isaborted due to that?

So clearly the issues were not in the test case content, but instead in what washappening around the execution and how the whole scheme could work in betterway. This is a very important point to discuss, as if it is neglected, it may quicklydestabilize the batch runs and complicate the maintenance, result analysis or trans-fer of test cases to another environment. These issues and their potential solutionsare discussed in the next chapter.

48

5 The Suggested Structure of Automated Test Cases

The previous chapter briefly discussed problems faced in in test automation. In thischapter, we analyze problems more thoroughly and try to find a solution for eachone. The presented solutions could form an architecture that when followed wouldmake automated test cases more maintainable, faster to analyse and less error-pronein batch executions. Architecture would therefore benefit the whole test automationprocess.

5.1 Problems faced with test automation

Next we will attempt to problematize the issues, provide practical examples to gainbetter understanding of the problems and suggest a detailed solution for each prob-lem.

5.1.1 Test-case portability

It is possible that a test case or a set of test cases are moved from an system under testto another. This may for example be done because of hardware-related issues, loadmanagement of test environments, or moving of the test responsibilities to anotherphysical location (laboratory). In [31], Tervo found limitations in portability to alsobe an issue.

Test cases are not automatically portable. A strategy for parameter managementis needed if portability is to be accomplished. Without a strategy, the migrationfrom one system under test to another may require refactoring efforts. Next we willidentify the parameter classes.

• Test environment-specific parameters are parameters that are extracted froma specific test environment. These are needed to create the required generalsetup of the system under test. Use of these parameters and setup phases areexplained more thoroughly in chapter 5.1.3. Examples are: radio network pa-rameters, unit id values, and interface parameters.

• Test set-specific parameters are parameters that may be used to run a set of

49

Table 5.1: Parameters for a test caseScope of environment Scope of test set / case

Environment parameters e0 e1 e2 e3 e4 e5Test set specific parameters s1 s2 s3 s4Test case specific parameters c0 c1 c3 c4

Actual parameters used in test c0 e1 s1 s2 c1 e5 s3 c3 c4

cases with altered values. These parameters are moved together with a set oftest cases. Example: a BTS id or an interface id.

• Test case-specific parameters are parameters with the scope of a single testcase. These parameters are moved with a single test case. Example: an ex-pected alarm id or message captured from an interface.

Consider a batch of test cases that are implemented with tightly coupled values,such as radio network element identifiers, functional unit identifiers or interfaceparameter values. It is noticed that a batch run of these test cases in the system undertest is taking too long, as the test runs are barely completed with the reserved timewindow. A solution for this would be to move the test cases to a new environmentto even out the load between the environments.

In practice it is not possible to move this kind of case to any other environmentwithout refactoring the case to contain correct parameter values. The same problemis faced, if some part of the configuration is changed in the test environment.

The first logical step to address the issue is to start collecting the required param-eters in a separate file. While this is a straightforward and seemingly good practice,it has problems when there are many parameters and a single file contains parame-ters from all three classes. There is also a high probability of incompatible parameterschemes across developers.

In order to gain control over parameter handling, these things must be defined.

• Which parameters (of all possible parameters in the environment) are needed?

• Classification of environment, test set and test case-specific parameters.

• Test automation tool support for this kind of parameter management.

If and when all this is communicated and agreed in teams doing test automationwork, as early as possible in test automation efforts, portability should be possi-ble. Moreover to make parameter schemes consistent and easy to adopt, two things

50

could be done. First, test tooling could enforce the correct use of parameters. Sec-ond, "seed projects" showing correct parameter use could be developed and sharedamong teams.

5.1.2 Script-to-script dependencies

In order to conduct the testing, the system under test should be brought to the stateof general setup (see Figure 5.2). This setup process is defined in section 5.1.3. Aftertest case execution, the state in system could be suitable for running another testcase immediately. This means that new cases are developed against and dependentof a state that another test case left behind; literally an undefined state, as it is aresult of a combination of preceding test cases. This can be an attractive option atfirst, as it does not require any additional effort.

Problems with this approach will emerge in at least three possible ways. First,when a test case fails at the end of batch run execution, whole set of preceding testcases must be re-executed to get the system to the point where the issue occurs. Thiscan be very slow. Second, the execution order is locked to one possible permutationof a set, that is known to work. The third issue is portability, as only batches of testcases can be moved from one environment to another. Portability is explained indetail in 5.1.1. Persson and Yilmaztürk also remarked on this in [28].

Consider a test set of three test cases, as shown in scenario A in 5.1. Test case 3fails in the batch run, and the test engineer attempts to find out whether this issue isreproducible and should be reported. The execution time for both test-case 1 and 2 is10 minutes and 5 minutes for test case 3, a total of over 20 minutes. After the secondexecution, it is clear that the issue can be reproduced and further investigation isneeded. Some additional monitoring of the system is needed to investigate the rootcause of the issue. The test engineer sets up the monitoring, starts the batch runagain, and collects the needed logs. Approximately 45 minutes in total are spent toprocess the issue to the point where it can be escalated to be fixed.

The issue is eventually fixed and the correction is released in the next softwarebuild. Another run with additional monitoring is required to verify the functional-ity; this needs another 20 minutes.

Now, a total of 65 minutes spent on an issue might not sound much, but thismere example hardly represents a real-life scenario. A batch can contain tens of testcases, and the time spent on similar issue might be multiplied to the extent where itis difficult to remain within the allotted time window of batch runs. Another point

51

Figure 5.1: Script dependency scenarios

52

to consider is that in this example only one issue was found. The situation wouldhave been even more difficult to handle if multiple issues had been found in singlebatch run.

The solution for this would be to have a strict baseline configuration, as de-scribed in Figure 5.2. After the baseline is defined, test cases should follow thescenario B shown in Figure 5.1, where every test case builds general setup and ap-plies the test case specific setup on top of that. After execution of the test case, theteardown follows immediately to revert the system under test back to the baselinestate.

Following this approach, the issues with fixed execution order, slow debuggingof problems and portability are solved. There is certainly a tradeoff of increasedtotal execution time, as the number of setup and teardown routines is increased.This increase may even require additional test environments, but the payback ineasy debugging and flexible organization of test cases in batch runs is worth it.

5.1.3 Test Setup and Teardown Handling

As discussed in the previous subchapter, a test case should be self-contained; itshould apply general setup and test case specific setup before the test case executionand perform a teardown routine afterwards. figure 5.2 defines these stages. Thereasoning for these phases is explained in 5.1.2 and 5.1.1. To be able to discuss thesubject further the three stages of setup are defined.

Figure 5.2: Setup stages of BSC

53

• First is the baseline setup, which in practice contains a functional system con-figuration without a radio network. This is done when a new software buildis taken in use. The baseline setup details should be agreed on in order to beconsistent over test environments.

• In the general setup phase, all recurring setup tasks are performed. These tasksare e.g. radio network creation, alarm handling, and computer log collectionset up. Possible radio network remnants are cleaned from the system (frompossible aborted test case execution).

• Finally, the test case creates any needed configuration in the test case specificsetup phase. This can for example be small alteration to a radio network ele-ment.

The baseline setup is beyond the scope of this work. Figure 5.3 defines the flowof the general setup phase. A test-case execution is triggered either by continuousintegration, or manually by test engineer. The general setup phase is automaticallyincluded in the execution flow.

1. The setup automatically initiates alarm and computer log handling, based onvalues defined in test case-specific parameters.

2. In the radio network setup phase, the first thing to do is to erase possible rem-nants of any old radio network setup. Next, if the test case requires a radionetwork, an attempt to create one will be made. A default network is de-fined, but a test case-specific network may also be supplied as parameter. Atthis point the environment specific parameters are fed to the network creationscripts. If the test case does not require a network, this point is void.

3. Finally, when the general setup is complete, the test case modifies the configu-ration to be suitable for the test. It also verifies that the required configurationexists and is valid before proceeding to the actual test steps.

To accomplish the common setup and teardown phases, first of all tool supportshould exist. Currently the test automation framework does not mandate any re-quired setup or teardown routine, so the routine needs to be implemented. This isalso the place where the parameter handling strategy is put into the action, so thetool should support the parameter scheme discussed in 5.1.1. Note that this is thefoundation on which the test case portability and dependency removal discussed inthe previous chapter is built.

54

Figure 5.3: A flow diagram of the common setup phase.

55

5.1.4 Test libraries and user interface abstraction

Changes in the user interface are common problem of high-level (i.e. functional) testautomation [17]. If interaction with the user interface is tightly coupled, the func-tionality of test cases may break and they may need maintenance efforts to bringthem to a usable state again. When the number of test cases grows, the tightlycoupled code can lead to massive maintenance efforts. Test automation should op-timally provide a fast and effective feedback loop, and the delay caused by mainte-nance does not fit very well into this scheme.

New software build brings a change in MML input or output (i.e. in the userinterface). A basic test step in a test case would be to query the functional state of aninterface. Let us consider that the command syntax required to inquire the state ofthe interface has changed, as well as the command output where the functional statecan be retrieved. As the test case attempts to give the command in the old form, thesystem rejects it and displays an error message. It is now impossible to continue theexecution of the test case.

Fixing the input-related code in the test case will move us forward, which is toparse the output of the command, where the sought interface state lies. Again, as theoutput has also changed, the test case is broken. If the command output is parsedin the test case, it must be fixed to get the test case functional again. Multiplyingthis effort with the number of broken test cases, we are dealing with a considerablemaintenance effort.

A solution to this highly probable scenario would be to have abstraction layers.Optimally the test cases should not communicate directly with the system undertest, but through well-defined interfaces, which the test automation framework pro-vides to the test case. Figure 5.4 introduces two layers of abstractions.

The respective framework used both library level and user interface level ab-stractions, but they were not enforced. The development of new abstractions to theuser interface and to test libraries requires an initial effort, but is paid back gener-ously in maintenance costs, as Hendrickson et al also found out in [17]. Unfortu-nately the need for abstraction may emerge when a lot of work in test automation isalready complete. Latter introduction of abstractions may require much refactoring.Early adoption and strict use of abstraction yields maintainable test cases.

56

Figure 5.4: Test tool abstraction layers against system under test.

57

5.1.5 Complexity and Execution time of test cases

If the goal of test automation project is to automate a certain number of old test casesthat were previously executed manually, there may be a risk that manual cases arenot ideal for automation purposes. Not being ideal means that ease of test case codedebugging, test execution times and result analysis are affected negatively by thelength of test cases. The reason for this is that functional test cases usually bundletoo many things (from an automation viewpoint) into a single test case that thereforegains excessive length. It may be feasible from a manual execution viewpoint, as itrequires less separate documentation and less reporting.

Figure 5.5: Test step relations from test case description to raw log file.

If no attention is paid, test automation efforts may produce automated test cases,which have a very long execution time. The reasoning for keeping test cases shortis the same as discussed with test case dependencies in 5.1.2. Fewster mentionssupporting conclusions for this in [11]. In addition to long execution times, longtest cases may cause difficulties when test case description needs to be compared to

58

the automated test case or when automated test case needs to be compared to theexecution result and raw log file. These relations are described in figure 5.5.

The ideal execution time for an automated test case would be around ten minutesor less. Being conscious about test case execution time makes developers workingwith test automation more productive, as initial implementation and possible de-bugging times are shorter, as are the result analysis times. While requiring an effort,it may be beneficial to rewrite old legacy test cases to better suit automated formbefore implementing the automation.

5.2 Chapter Summary

Previous subsections have presented the problems faced while implementing testautomation. This summary, we’ll recap the suggestions and present a chart of a testcase overall architecture.

• To ensure portability, environment-specific parameters should be recognized,used and named in a consistent manner across developers.

• Test cases should have a consistent setup and teardown functionality, whichsupports the use of the environment-specific parameters.

• Test cases should be self-contained and should not rely on any configurationmade by other cases or scripts.

• External tools should provide uniform interface for test cases to use and avoidthe need for excess setup and teardown code to accomplish a task.

• MML absraction layer should be used where possible to ensure test case andlibrary maintainability.

• The implementation of test cases and libraries benefit from use of the codingconventions and standards, like any other software products.

• Test result analysis can be improved by having a sufficient numbers of logfiles available from every execution. Test case code should be written in sucha manner that test description is easily comparable to it.

• Test case execution time should be kept short. Rule of thumb is ten minutesor less. Result analysis and test case code debugging become more difficult ifexecution time is too long.

59

If the tool used does not mandate the shape of test cases and libraries and thereis no clear pattern to follow for developers, a risk of producing non-portable andunmaintainable test code is present. The technical debt caused by this multipliesthe more cases are implemented. Risk can be suppressed by following a set of pre-defined guidelines such as those presented in this thesis. It is clear that all typesof test cases do not benefit from these guidelines, but particularly the O&M typefunctional test cases clearly do.

5.3 Supporting findings in another project

As I have been recently introduced to another large test automation project, I havefound out that most findings and issues presented in this chapter were clearly iden-tifiable in other project too while the software domain was completely different.Based on these two test automation projects, relatively high level testing like func-tional testing seems to be prone for these kind of issues.

60

6 Summary

This thesis examines the features and problems of a test automation project, in thecontext of GSM network element. Chapter 2 provides a background of software test-ing, software quality and software test automation. Chapter 3 describes the GSMnetwork, the network elements and their interoperation. Chapter 4 introduces theNokia implementation of the GSM, especially the base station controller. TypicalO&M testing areas and the test automation solution are also presented. Chapter 5examines the problems that may occur in test automation project and their respec-tive solutions.

The objective of the thesis was to discover the requirements for test automationin the context of specific mobile network element. Based on those requirements, acommon high-level framework is presented in problem-solution form.

Most of the problems presented in Chapter 5 are presented in some form in theliterature. Some of the solutions are recognized as best practices, such as the pageobject pattern [32], which essentially equals the user interface abstraction presentedin Chapter 5.1.4. This work might serve its purpose by providing insight into com-mon issues in test automation, for anyone starting or maintaining a test automationproject. It seems that although the environment of test automation in this thesis wasnot a typical one and having found supporting findings on another test automationproject, it seems that the pitfalls presented in this thesis could be universal for mostof the test automation out there.

In general, a test automation project should be considered as a software project.A high-level test automation project typically contains large amounts of code, whichhas to be adaptable enough to live in symbiosis with actual software under devel-opment. Without applying the best practices and discipline required by proper soft-ware development, and without having a solid understanding of how a test automa-tion project should be developed, it can take a path with many common pitfalls, andwhich can be very expensive or outright impossible to recover from after the projecthas reached a certain size.

61

References

[1] 3GPP. Feasibility study for evolved gsm/edge radio access network(geran). URL http://www.etsi.org/deliver/etsi_tr/145900_145999/

145912/12.00.00_60/tr_145912v120000p.pdf, referenced 24.6.2015.

[2] 3GPP. Gprs and edge. URL http://www.3gpp.org/technologies/

keywords-acronyms/102-gprs-edge, referenced 24.6.2015.

[3] AMMANN, P., AND OFFUTT, J. Introduction to Software Testing. CambridgeUniversity Press, 2008.

[4] ATT. It’s time to develop a migration plan for m2m, 2012. URL: http://www.business.att.com/enterprise/Family/mobility-services/

machine-to-machine/m2m-applications/cd2migration/page=

addl-info/, referenced 7.4.2015.

[5] BACH, J. Test automation snake oil. In Proceedings of the 14th InternationalConference and Exposition on Testing Computer Software (TCS99) (1999).

[6] BOEHM, B. W. Guidelines for verifying and validating software requirementsand design specifications. Euro IFIP 79 (1979), 711–719.

[7] BORQUE, P., AND FAIRLEY, R. E. D. Guide to the Software Engineering Body ofKnowledge, Version 3.0. IEEE, 2014.

[8] BURNSTEIN, I. Practical Software Testing. Springer-Verlag New York, Inc, 2002.

[9] BURNSTEIN, I., SUWANASSART, T., AND CARLSON, R. Developing a testingmaturity model for software test process evaluation and improvement. Pro-ceedings of International Test Conference 1996 (1996), 581–589.

[10] ERICSSON. Traffic exploration tool. https://www.ericsson.com/TET, refer-enced 7.5.2018.

[11] FEWSTER, M., AND CONSULTANTS, G. Common mistakes in test automation.In Proceedings of Fall Test Automation Conference (2001).

62

[12] FEWSTER, M., AND GRAHAM, D. Software Test Automation - Effective Use of TestExecution Tools. Addison Wesley, 1999.

[13] FOWLER, M. Continuous integration. http://www.martinfowler.com/articles/continuousIntegration.html, referenced 30.3.2015.

[14] GLEICK, J. A bug and a crash, 1996. URL: http://www.around.com/ariane.html/, referenced 27.4.2015.

[15] GSMA. Gsma, 2015. URL: http://www.gsma.com/aboutus/

gsm-technology/gsm, referenced 6.4.2015.

[16] HAIKALA, I., AND MÄRIJÄRVI, J. Ohjelmistotuotanto. Suomen ATK-kustannus,1998.

[17] HENDRICKSON, E. The differences between test automation success and fail-ure. Proceedings of STAR West (1998).

[18] HILLEBRAND, F. The creation of standards for global mobile communica-tion: Gsm and umts standardization from 1982 to 2000. IEEE Wireless Commu-nications October (2013), 24–33.

[19] HUANG, J. Software Error Detection through Testing and Analysis. John Wiley &Sons, Inc., 2009.

[20] IEEE. IEEE Standard Glossary of Software Engineering Terminology. IEEE, 1990.

[21] MATHUR, S., AND MALIK, S. Advancements in the v-model. InternationalJournal of Computer Applications 1, 12 (2010), 29–34.

[22] MYERS, G. J., SANDLER, C., AND BADGETT, T. The Art of Software Testing. JohnWiley & Sons, Inc., 2012.

[23] NOKIA. Gsm bss training material. internal document.

[24] NOKIA. Dx200 hardware, 1999. internal document.

[25] NOKIA. Nokia siemens networks flexi base station controller (flexi bsc), 2009.Datasheet.

[26] OY, N. N. SYSTRA GSM System Training. Nokia Networks Oy, 2000.

63

[27] PENTTINEN, J. GSM-tekniikka: Järjestelmän toiminta ja kehitys kohti UMTS-aikakautta. WSOY, 2002.

[28] PERSSON, C., AND YILMAZTÃ 14 RK, N. Establishment of automated regression

testing at abb: Industrial experience report on ’avoiding the pitfalls’. Proceed-ings of the 19th International Conference on Automated Software Engineering (2004),112–121.

[29] ROTHERMEL, G., UNTCH, R. H., CHU, C., AND HARROLD, M. J. Prioritizingtest cases for regression testing. IEEE Transactions on Software Engineering 27, 10(2001), 929–948.

[30] TELSTRA. It’s time to say goodbye old friend, 2014. URL: http://exchange.telstra.com.au/2014/07/23/its-time-to-say-goodbye-old-friend/,referenced 7.4.2015.

[31] TERVO, B. Standards for test automation. Proceedings of STAR East (2001).

[32] WEBDRIVER.IO. Page object pattern. http://webdriver.io/guide/testrunner/pageobjects.html, referenced 27.3.2018.

[33] WHITTAKER, J. What is software testing? and why is it so hard? IEEE Software17, 1 (2000), 70–79.

[34] WIEGERS, K., AND BEATTY, J. Software requirements. Pearson Education, 2013.

[35] WIKLUND, K., ELDH, S., SUNDMARK, D., AND LUNDQVIST, K. Technical debtin test automation. IEEE Fift International Conference on Software Testing, Verifi-cation and Validation (2012), 887–892.