A Novel Approach to Automated Systems Engineeringon a Multi-Agent Robotics Platform

using Enterprise Configuration Testing Software

Stephen CossellEngineer, ScriptRock Inc.steve@scriptrock.com

Abstract

This paper presents a case study of applyingan enterprise grade systems configuration man-agement platform to a set of unmanned groundvehicles and a ground control station. Muchlike large scale enterprise infrastructure, mod-ern robotics systems are comprised of manydifferent machines communicating over a va-riety of media, let alone the large number ofmodules and applications running on each ma-chine. Each module typically has its own con-figuration settings, with each individual pieceof configuration information being crucial tothe overall working state of the robotic sys-tem. When one configuration item is changedinadvertently, or otherwise, without an opera-tor’s knowledge, a manual and lengthy expedi-tion through a series of configuration files andcommand output is usually used to diagnosethe cause of the problem. This situation is ex-acerbated when the platform is used by a rangeof people for different scenarios on a regular ba-sis. The ScriptRock platform is used by largeenterprise software and infrastructure teams toencode system configuration requirements intoexecutable documentation so the underlyingenvironment their applications run on can bevalidated immediately. The application of theScriptRock platform to a multi-agent roboticsystem has shown improved re-configurationtimes between different use cases as well as asignificantly simplified troubleshooting and di-agnosis process when the system is found notto be in a working state.

1 Introduction

As more robotic systems are entering mainstream oper-ation they are becoming more complex in order to beable to complete more intricate tasks. In addition, these

systems are being given more responsibility in societyand as such must be robust and able to function with-out fault. All modern industrial robotics projects aremore than likely to have extensive systems engineeringprocesses tied into their development life-cycles. Thisprocess typically involves a thorough set of unit andintegration tests to validate each software componentof a system, from a functional point of view. It ishowever difficult to find evidence in published literatureof development and testing practices that validate theunderlying environment and configuration of softwaremodules themselves.

Software configuration management has been a pointof interest for computer science researchers over thelast three decades. In the 1980s, research by [Bersoff,1984] concentrated on identifying and understandingthe problem and promoted the need for configurationmanagement in large enterprise systems. By the1990s, researchers had begun proposing solutions toparticular niche problem areas. Work by [Hall et al.,1997] presented a possible solution to configurationmanagement of wireless local area networks. As partof his job maintaining university workstations, MarkBurgess developed CFEngine[Burgess, 1995], a crossplatform package for automated system configurationmanagement. While CFEngine provides a useful toolfor configuration management, it’s potential usage islimited to developers, as configuration promises must beencoded in scripts. This is in contrast to the ScriptRockplatform, which provides an interface that abstractsunderlying code required to validate in individualconfiguration item[Sharp-Paul, 2012].

In the last decade companies with large dynamiccloud infrastructures such as Netflix [Hoff, 2010], haveapplied custom in-house solutions to test and correctconfiguration state of their own systems and infrastruc-ture via deliberate and regular breaking of configurationitems. This implementation is in line with IBM’s

manifesto release in [Horn, 2001] requesting a push forthe widespread adoption of Autonomic Computing.

In the last six years there has been a push in roboticsresearch circles to adopt a common platform for softwaremodules. Platforms such as aRD [Hirzinger and Bauml,2006], OpenRDK [Quigley et al., 2009] and the RobotOperating System (ROS) [Calisi et al., 2008] attemptto find this common standard, with ROS gaining muchadoption in the global robotics community. While theseplatforms allow rapid prototyping and application ofcommon robotics algorithms, they are still difficult toconfigure correctly and ensure critical dependencies aresatisfied. Although this paper focusses on applyingthe ScriptRock platform to a custom, in-house roboticsystem developed within the mechatronics researchgroup at UNSW, future work will attempt to applya ScriptRock test suite to common ROS modules.The aim is to be able to validate the installation andinclusion of a ROS module by running a ScriptRock testthat is distributed with the module.

This paper uses a multi-node system first presentedin [Whitty et al., 2010] and later from a more softwareand networking perspective in [Guivant et al., 2012]. Al-though these papers describe multi-vehicle systems, asimplified single unmanned ground vehicle (UGV) setupis used in this paper in various common use case sce-narios. As presented in Section 3, these scenarios rangefrom direct on board tele-operation to semi-autonomousoperation, used commonly for research purposes.

1.1 Outline

The application presented in this paper demonstratesa simple and efficient method for defining and manag-ing system configuration information as well as enablingfast troubleshooting when a configuration item, amonghundreds, has been misconfigured. Section 2 begins bygiving a high level background on both the UGV andground control station (GCS) system as well as the highlevel capabilities of the ScriptRock platform. Section 3outlines how the ScriptRock platform has been appliedto the UGV and GCS system. Sections 4 and 5 respec-tively discuss the details and results of an experimentused to gauge the increased benefit of using the Scrip-tRock platform on the UGV and GCS system over anexisting manual method.

2 Platform Background

This section gives a high level overview of both theUGV/GCS and ScriptRock platforms in terms of soft-ware and network system configuration.

2.1 The Unmanned Ground Vehicle andGround Control Station Platform

This section gives a high level software module andnetwork layout outline for the Unmanned GroundVehicle (UGV) and Ground Control Station (GCS)setup discussed in this paper. For a more detaileddescription of the platform as a whole, see [Whittyet al., 2010] and [Guivant et al., 2012]. For a descriptionof specific individual software components, please referto [Robledo et al., 2010], [Guivant, 2008] and [Guivant,2012].

2.2 The ScriptRock Platform

At a high level the ScriptRock platform allows a personto submit system and configuration requirements into anonline cloud based platform. Requirement informationis submitted using templates that abstract away thetechnical knowledge required to actually perform arequirement validation. For example, a requirementmight be to ensure that a particular program is locatedin the correct location on a target machine so thatthe automated start-up scripts can find it properly.Although there are commands that can be run on anymainstream operating system (OS) to check that aparticular file or application exists in a given folder,each command is different on each OS and must bemanually encoded into a test script. The respectiveScriptRock template used for the this type of validationonly requires a full path to be entered with a humanreadable description, with the actual testing processitself abstracted away.

Once a set of requirements have been submitted tothe website they are then downloaded as a zip file con-taining all the requirements translated into executabletests. The zip file contains a shell or batch scriptthat runs the complete set of tests, which generateseither plain text or an HTML report containing theresults of each test run. If a test fails, a remediationstep or set of steps is presented to the tester so theycan correct the state of the system and thereforesubsequently satisfy the test requirement. As softwaremodules get updated, the online tests validating thesemodules can be modified and re-downloaded to reflectthe change in configuration state. Since the UGVsand GCS are used in different scenarios on a regularbasis their underlying configurations change often. Aseparate ScriptRock test project was set up for each ofthese scenarios, so that an particular test script can beused to validate the state of a particular desired scenario.

3 Method

The UGV and GCS system is flexible enough to be con-figured in a number of different ways, but there are threemain scenarios that are used most often and are usu-ally required at short notice. These scenarios are directtele-operation, remote tele-operation and remote point-and-click semi-autonomous operation. Each scenario isoutlined in the following sections, along with the mod-ules and infrastructure required and an overview of thetypes of ScriptRock tests that have been applied to trackconfiguration.

3.1 All Scenarios

Although the modules and devices used in each ofthe following scenarios differ, there are a number ofcommon configuration items that apply to all scenarios.Each scenario relies on a set of dynamic link libraries(DLLs) as well as certain software interpreters to beinstalled and be accessible in the operating system’ssearch path. To check a specific DLL is installed andlocated in a known location, a “file exists” test wascreated within ScriptRock. Modules are started on eachnode using a Python script. As a result, a test wascreated to check that Python was installed on the sys-tem and that the correct version of Python was installed.

Even with the simplest scenario of direct tele-operation, modules are required to communicate dataover a network connection. Direct tele-operationrequires two ethernet connections on board whereasthe other scenarios require multiple hops via wired andwireless connections and over different subnets. Asa result, a set of “ping success” tests were used tocheck that the require nodes could communicate withone another. The underlying network configurations ofthese nodes were also tested by using a combination of“command output” and “file contents” checks. Forexample, the routing table can be checked on Windowsby running the command “route print” and checkingthe output for an expected value. The same can bedone on Unix based platforms using the “route -n”command.

In the remote tele-operation and point-and-click au-tonomy scenarios, certain specific settings were requiredat boot on each of the wireless routers to make surethe UGV’s mesh network settings took precedence overthe system defaults. A custom script was placed in theLinux start-up directory “/etc/init.d” to apply thesesettings. In addition to checking that the required set-tings were actually set using custom “command output”tests running commands like “ifconfig”, an extensiveset of “file contents” tests were also used to validatethat the custom initialisation script itself contained

correct values.

The third major category of tests applicable to allscenarios are the requirements that folder structure,configuration files for custom modules, as well as themodules themselves, existed and were located in thecorrect locations. A number of “directory exists”and “file exists” tests were used to validate thecomplete folder structure was constructed correctly.

It should be noted that although this paper focuseson validating that a system is in the correct state as itchanges between use cases, these tests have also greatlyassisted in rolling out software and network settings toa fresh on board laptop. By running these tests on aregular basis as modules were added, it was immedi-ately apparent that a module or setting was missingwithout needing to actually start the software up andthen manually detect the problem or missing component.

In each of the following sections an infrastructure dia-gram is used to assist in describing each scenario. Figure1 outlines each of the diagram components.

Figure 1: Legend used for infrastructure diagrams.

3.2 Direct Tele-operation

Direct tele-operation involved an operator using an Xbox360 controller to drive the UGV for situations such asOpen Day demonstrations, where direct deterministiccontrol is required for OH&S purposes, as well as simplymoving the robot from one lab to another. The scenariorequires only infrastructure on a single UGV and involvesa minimal set of configuration checks to be created, asoutlined below.

Infrastructure Layout

The infrastructure used in this scenario is containedwithin a single UGV. Figure 2 shows how data flowsbetween hardware and software modules in this scenario.

Tele-operated commands are generated by an Xbox360 controller, which are received by a software moduleon the UGV’s on board laptop. These commands arepushed by the module into the centralised databasesystem.

Laser range finder measurements from the frontand back proximity lasers are also pushed into theirrespective queues in the database. A laser proximitymodule then reads these queues and determines if anobject is too close to allow motion in a particulardirection. This module pushes data to a queue in thedatabase on a regular basis, but modifies a specific flagin each pushed record as to whether it can, or cannotdetect anything in its proximate range.

The command arbiter module reads new tele-operation commands and laser proximity records fromthis centralised database on a regular basis and allowsthe instructions to pass through to the DMC interfacingmodule if the proximity queues in the database have notflagged any objects as being in the immediate vicinity ofthe UGV. The DMC module then takes the final filtereddriving commands from the database and communicatesthem to the on board DMC via the local ethernet con-nection.

Figure 2: The software and networking component lay-out used for direct tele-operation.

Configuration Items

In addition to the general configuration items applica-ble to all scenarios, a number of “file contents” basedchecks were created for critical configuration files of mod-ules required to achieve tele-operation. The most impor-tant checks for this scenario center around sensor inter-facing modules. Each of the three laser range findersattached to the UGV communicates data either via awired ethernet connection or via a RS-422 serial connec-tion converted to USB. Each of these modules encodesthe IP address and port number, or the baud rate andCOM port in a configuration file. Ensuring that thesevalues are not only present, but also set correctly, is animportant step in validating whether the UGV is config-ured correctly.

3.3 Remote Tele-operation

Remote tele-operation is used predominantly for teach-ing purposes during the later stages of undergraduaterobot design and autonomous systems courses. In addi-tion to the on board modules used in the Direct Tele-operation scenario outlined in Section 3.2, this scenariomakes use of a number of machines that replicate dataover both a wireless and wired network connection. Adetailed overview of the infrastructure is given below,followed by the additional configuration items requiredfor this scenario to run.

Infrastructure Layout

In addition to the modules used on board for laserproximity checks and arbitrating driving instructions tothe DMC interfacing module, this scenario relies on anumber of network and centralised database replicationsettings, which are covered in great detail in [Guivantet al., 2012]. This infrastructure is shown in Figure 3.

Data from the UGV is replicated to a teacher’sdesktop computer over a wireless network connection,using a pair of mesh network routers. The environmentthe UGV operates in also makes use of a fixed externallaser range finder, which acts as a local positioningsystem, much like a GPS unit in outdoor environments.This laser data is replicated over an ethernet connectionto the teacher’s desktop computer.

Student machines are connected to the teacher’smachine via a local subnet over ethernet. Students runa local copy of the centralised database, which listensfor broadcasted data from the teacher’s machine. Inturn the student runs custom written software to readdata from their local database, interpret the data andpush driving instructions back into the local database.This local database is then configured to replicatedriving instructions back to the teacher’s machine,which are then in turn replicated back to the UGV’s

database. Once on the UGV, these driving instructionsare handled via the same tele-operation mechanism asin Section 3.2.

Figure 3: The software and networking component lay-out used for remote tele-operation.

Configuration Items

On board, additional “ping success” tests were addedfor each additional node that data needs to be repli-cated to, to ensure the underlying medium is configuredcorrectly. In fact, a test suite of network configurationchecks was created to test the connectivity of each nodeto every other node it is required to communicate with.The database replication functionality is also basedon an internal code, called an icode, that enables datato be received into a correctly configured queue on adestination node. A set of “file contents” checkswere created for each replicated queue on each nodeto ensure the numeric icodes were consistent. Customdatabase queue definition checks were also included foreach node to ensure the internal data structures in thedatabase were configured correctly.

Having a test suite designed for the UGV and teach-ing workstation greatly reduced the set up time requiredto prepare the machines for lab work. In addition, astudent machine test suite allowed students to ensuretheir module set was installed correctly without havingto take up a lab demonstrator’s time. The student ma-chine test suite was also used by lab demonstrators toassist in troubleshooting exotic problems that arose on

students’ machines brought from outside the lab.

3.4 Semi-autonomous Point-and-Click

The semi-autonomous point-and-click scenario is themost complex scenario outlined in this paper. It ispredominantly used for research purposes and involvesa number of modules spread over one or more UGVsand a GCS machine. A typical use case involves fusingsensor data and displaying a virtualised reality on theGCS machine. At a high level, an operator can click aground location on the virtualised reality interface andimmediately see a given UGV travel autonomously tothe selected destination in reality. The point-and-clickfunctionality can be interchanged with other higher levelplanner or voice controlled modules that issue desiredUGV way points and destinations in the same manner.However, simple virtualised interface point-and-click isused here for simplicity.

Infrastructure Layout

This scenario uses the same configuration on the UGVand wireless routers as the Remote Tele-operationscenario discussed in Section 3.3. The main differencelies in the GCS machine used to remotely control theUGV and the missing laser range finder previously usedas a local positioning system. Figure 4 shows a highlevel representation of data flow and modules used inthis scenario.

The GCS database receives replicated data thatincludes raw laser range values from the top mountedrotating laser on the UGV, a pose estimation andDMC wheel rotation and steering positions. This datais then read by both a fusion and laser conversionmodule to produce a three-dimensional point cloudand occupancy grid, which are pushed back into theGCS database. This data is then read by an OpenGLbased visualisation module to display the spatial datain real-time. This interface allows an operator toclick on a location on the occupancy grid, which pushesa desired destination into another queue in the database.

This desired destination is in turn read by a plannermodule, which uses the UGV’s current pose estimation,along with the current occupancy grid, to plan a path tothat location. This path is pushed to another databasequeue on the GCS machine and replicated to the UGV.A module on the UGV reads this path and translatesit into driving instructions. These instructions arehandled in an identical manner to driving instructionsin other scenarios and processed through the arbitrationmodule mentioned in Section 3.2.

Figure 4: The software and networking component lay-out used for point-and-click autonomy.

Configuration Items

Each module requires the correct queue configurationsset so that data passes from one module to the nextcorrectly. Each database queue must also be replicatedbetween database instances correctly and as such arange of configuration items are required to be validatedvia a set of tests.

One example of this involves the point cloud gen-eration module having the correct laser input queueset in its configuration. Since the UGV has threelasers on board, the correct laser queue for point cloudgeneration must be set, or an incorrect point cloud isgenerated. This can have a lead on effect to occupancygrid generation and ultimately path planning throughan environment that does not match reality. A separate“file contents” test was created for each databaseinput and output queue entry on each module.

Another example of an insidious misconfigurationis pushing tele-operated driving instructions, au-tonomously generated driving instructions or laserproximity effecting driving instructions to the wrongqueue. On board, the command arbiter module usesinbuilt flags and a hierarchy system to give precedenceto certain driving instructions over others. This modulerelies on the assumption that a given database queuehas been configured in another module to push recordsfrom the correct location. For example, an emergencymode of tele-operation, that ignores laser proximity

data, has the highest precedence in the commandarbiter program, but is rarely used in practice. If, forexample, the autonomous driving instruction genera-tion module was misconfigured to push data to thisqueue, then all safety protocols will be unintentionallyignored when the UGV is driving in autonomousmode. Again, a series of “file contents” checks wereused to enforce correct queue assignments to all modules.

A final example involves mathematical constantsstored in the planned path to driving instructionsconfiguration file. This module takes a path and currentpose and uses control theory to continually issue drivinginstructions. These calculations rely on correctlyconfigured control constants as well as knowing theUGV’s maximum forward and backward speeds andsteering angle limits. A misconfiguration to one or moreof these values can lead to erratic motion, which candamage the UGV, let alone objects or people in itsoperating environment. A number of tests were alsoincluded to validate each of these constants individually,with certain domain knowledge encoded into the testsrelating to valid ranges of these values for properoperation.

4 Experimental Outline

To test the added troubleshooting benefit of using theScriptRock platform over an otherwise manual process,an experiment consisting of a series of tests was devisedthat cover the three main scenarios outlined in Section3. For each scenario, a set of critical configuration itemswere selected at random and encoded into a chaos scriptas separate tests. When run, the chaos script randomlychooses one configuration item and modifies it into abroken state. This is based on a similar concept usedby the Netflix Chaos Monkey [Hoff, 2010].

Each misconfiguration item encoded into the chaosscript falls into one of the following eight categories:

• a missing DLL file or missing software dependencysuch as the Python runtime being made inaccessible;

• a missing executable file of a software module;

• a missing configuration file for a given software mod-ule;

• an incorrectly set database queue in a module’s con-figuration file for either the queue it reads data fromor pushes data to;

• an icode misalignment between the two databaseinstances replicating data between one another fora particular database queue;

• an incorrect IP address or port number for databasereplication settings;

• an underlying network misconfiguration such as in-correct subnet settings or incorrect IP address as-signment; or

• any module specific configuration file lines such asincorrect baud rates, invalid IP address or COMport for hardware modules and particular constantconfiguration definitions required for module calcu-lations.

A total of 111 misconfigurations were spread acrossthree scenario chaos scripts. Before each test, a scenariois chosen at random and the respective chaos script wasrun on a given node used in that scenario. A workingstate action was then attempted upon the system unsuc-cessfully to prove that the chaos script did in fact break acrucial configuration item. An experienced engineer thatwas involved in the design, development and now mainte-nance of the entire software and networking componentset was given access to the system to attempt to diag-nose and correct the introduced problem and thereforecomplete the given working state action. Use of everyexisting piece of software, testing tool and piece of doc-umentation was allowed in every scenario attempt, butthe engineer was only allowed access to a ScriptRock testsuite for the given scenario on every second attempt. Ascenario attempt is deemed complete if the system is ableto carry out the respective working state action. Theseare defined as:

Direct Tele-operation Use an Xbox 360 controller totele-operate the UGV directly over a distance ofthree metres and return to the starting position,completely demonstrating that it can also steer bothleft and right.

Remote Tele-operation Be able to use a keyboardbased application1 on the GCS machine to tele-operate the UGV remotely over a distance of threemetres, completely demonstrating that it can steerboth left and right while driving in both directions.

Semi-autonomous Point-and-Click Given an openarea that is traversable by the UGV, issue a UGVdesired destination by clicking on a location in a vir-tualised reality interface on the GCS and have theUGV then autonomously travel to that destinationin reality. Then issue another desired destinationback to the starting location. In the process, thepath chosen must demonstrate that the UGV canmove forward and backwards, and be able to steerboth left and right during motion.

1A keyboard tele-operation program was used to simulatea student’s software module.

Comparison of Problem Solving Time

minutes

0 5 10 15 20 25 30

Tes

t No.

0T

est N

o.10

Tes

t No.

20T

est N

o.30

Tes

t No.

40T

est N

o.50

Tes

t No.

60T

est N

o.70

Tes

t No.

80T

est N

o.90

Tes

t No.

100

Tes

t No.

110

Figure 5: Comparison of time take to solve a given sce-nario problem. A blue circle represents an attempt wherethe engineer was allowed to use the ScriptRock platform,while a red cross represents attempts without access tothe ScriptRock platform.

On each scenario attempt, a timer was started whenthe engineer was first allowed access to the system. Thetimer was paused when the engineer believed they hadsolved the problem. A working state action was then at-tempted immediately to determine if the problem had infact been solved. Successful completion of the action leadto the paused time being recorded as the official solvetime of that test. On a unsuccessful action attempt, thetimer was resumed from the previously paused state andthe engineer was again given further access to the systemuntil they again claimed to have solved the problem. Sec-tion 5 outlines the results of this experimental approachapplied multiple times and provides an analysis of theresults.

5 Experimental Results and Analysis

The experiment outlined in Section 4 was run a total of50 times, 25 of which allowed the use of the ScriptRockplatform and the other 25 without. Figure 5 showsthe recorded times of each attempt. The graph clearlydemonstrates that using the ScriptRock platform todiagnose system misconfiguration adds an element ofdeterminism and repeatability to the troubleshootingprocess.

The main observation made from tests allowing theuse of the ScriptRock platform was that the engineer

would immediately initiate the ScriptRock test suitewithout considering the use of any other approaches.The test suites created for each scenario took 1m 25s(± 10s) to execute and upon viewing the resulting testreport2, the engineer subsequently solved the problemwithin a further 10s to 40s.

When not able to use the ScriptRock platform, theengineer appeared to begin the diagnosis process bystarting all software start-up scripts to attempt toobserve which modules appeared to function normallyand which did not. If everything appeared to initialisecorrectly, the engineer would often then attempt todrive the UGV via tele-operation or, in autonomyrelated tests, set a desired destination to drive to. Theengineer also appeared to make extensive use of thequeue activity tab of the centralised database instanceon each node to see if certain critical queues werenot being populated. The database queue activityand attempted motion testing methods then lead theengineer to begin looking through between one and fiveconfiguration files to check configuration items, line byline, until the problem was ultimately solved.

The four quickest test attempts in Figure 5 did notinvolve any use of the ScriptRock platform. These testall fell into the misconfiguration category of moving ormisplacing one of the software module executable files.Due to the engineer beginning each non-ScriptRockdiagnosis attempt by running a start-up script, theoperating system presented a message box when itcould not find a particular executable file to run. Thisprovided the engineer with the ability to solve this typeof problem consistently within one minute.

Direct and remote tele-operation scenario tests wereamong the easier misconfigurations to diagnose. Byattempting to drive the UGV and by observing theactivity of database queues, the engineer could solvethe problem in a number of minutes. If databaseactivity appeared to be normal, then configuration filesspecifically on the UGV were checked to find a solution.

Autonomy related tests were the next most diffi-cult misconfiguration to diagnose as they involved bothchecking low level tele-operation modules on the UGV aswell as sensor fusion and path planning modules on theGCS machine. Some diagnosis attempts were assistedby observing inconsistencies in database queue activityand inter-node database queue replication anomalies.Some tests were also assisted by the fact that the UGV

2The default layout of a results report places the resultsof all failed tests at the top, followed by a complete set of alltest results.

Test Type w/ ScriptRock w/o ScriptRock

Average Time 1m 47s 13m 16sStd. Dev. 11.1s 9m 40.5sTotal Time 44m 38s 5h 31m 38s

Table 1: Statistical summary of experimental test timeswith and without the use of the ScriptRock platform,each over 25 tests.

could be tele-operated but not autonomously driven.This lead the engineer to focus the search predominantlyon the GCS. One of the more successful and uniquetest attempts in this category involved the UGV neverbeing able to steer left. This was eventually diagnosedas a constant being set to zero that represented one oftwo steering angle limits in the DMC interfacing module.

The most difficult group of tests to diagnose involvedsemi-autonomous operation, with the UGV also notresponding to remote or local tele-operation attempts.Here the engineer resorted back to tracking the virtualflow of data from sensors through software modules,database replication and resulting actuator modules.More than half of these type of tests involved theengineer consulting a wiki for information about theusage and configuration of modules.

A statistical summary of the experimental results isgiven in Table 1, which significantly highlights the morethan five-fold improvement in average troubleshootingtime using ScriptRock over existing methods. It shouldbe noted that each test attempt presented in this pa-per involved a single item being misconfigured. Fu-ture experiments are planned to test the troubleshootingtimes of multiple misconfiguration points within a sce-nario. The hypothesis here is that multiple misconfigu-ration items may linearly or exponentially make diagno-sis more difficult for existing manual processes, but onlyconstantly more difficult for ScriptRock enabled tests.

6 Conclusion

The application of the ScriptRock testing platform toa multi-node robotics system has demonstrated an in-creased amount of understandability and determinismin managing the software and networking configuration.Undertaking configuration change, or troubleshootingmisconfigured nodes, has been reduced to time frameswell within five minutes, as opposed to fractions of anhour, using existing manual techniques. As a naturalprogression these tests have also improved the process ofrolling out software and network settings on a new UGV.The progress and ultimate success of a fresh roll outcan be measured accurately through the setup process.Making slight configuration changes for use in differing

scenarios can also be validated without having to resortto the same previously used manual process. Roboticsystems, much like large enterprise grade architectures,have an extensive range of solutions capable of testingwhether an application performs correctly, but little isavailable to test the underlying environment and mod-ule configuration. This paper has demonstrated that thisinnovative application to enterprise systems can also beapplied to complex robotic systems to greatly simplifythe development, maintenance and use of these systems.

Acknowledgments

The author would like to acknowledge, Dr Jose Guivant,Mr Mark Whitty and A/Prof Jayantha Katupitiya fortheir extensive contributions to the design and develop-ment of the UGV and GCS system and providing accessto the platform. Their assistance in the test creationand testing process is also appreciated.

ScriptRock should also be acknowledged for provid-ing a custom educational licence to the engineers andresearchers in Mechatronics at UNSW.

References[Bersoff, 1984] Bersoff, E. H. (1984). Elements of soft-

ware configuration management. IEEE Transactionson Software Engieneering, SE-10(1).

[Burgess, 1995] Burgess, M. (1995). Cfengine: a siteconfiguration engine. USENIX Computing Systems,8(3).

[Calisi et al., 2008] Calisi, D., Censi, A., Iocchi, L., andNardi, D. (2008). Openrdk: A modular framework forrobotic software development. In Intelligent Robotsand Systems, 2008. IROS 2008. IEEE/RSJ Interna-tional Conference on, pages 1872–1877, Nice.

[Guivant, 2012] Guivant, J. (2012). Possum robot.http://www.possumrobot.com.

[Guivant et al., 2012] Guivant, J., Cossell, S., Whitty,M., and Katupitiya, J. (2012). Internet-based oper-ation of autonomous robots: The role of data repli-cation, compression, bandwidth allocation and visual-ization. Journal of Field Robotics, 29(5):793–818.

[Guivant, 2008] Guivant, J. E. (2008). Real time synthe-sis of 3d images based on low cost laser scanner on amoving vehicle. In V Jornadas Argentinas de Robtica(JAR’08).

[Hall et al., 1997] Hall, R. S., Heimbigner, D., van derHoek, A., and Wolf, A. L. (1997). An architecturefor post-development configuration management in awide-area network. In Proceedings of the 17th Interna-tional Conference on Distributed Computing Systems,Baltimore, MD.

[Hirzinger and Bauml, 2006] Hirzinger, G. and Bauml,B. (2006). Agile robot development (ard): A prag-matic approach to robotic software. In IntelligentRobots and Systems, 2006 IEEE/RSJ InternationalConference on, Beijing.

[Hoff, 2010] Hoff, T. (2010). Netflix: Continually testby failing servers with chaos monkey.

[Horn, 2001] Horn, P. (2001). Autonomic computing:Ibms perspective on the state of information technol-ogy.

[Quigley et al., 2009] Quigley, M., Gerkey, B., Conley,K., Faust, J., Foote, T., Leibs, J., Berger, E., Wheeler,R., and Ng, A. (2009). Ros: an open-source robotoperating system.

[Robledo et al., 2010] Robledo, A., Guivant, J., andCossell, S. (2010). Pseudo priority queues for real-time performance on dynamic programming processesapplied to path planning. In Proceedings of the 2010Australasian Conference on Robotics & Automation,Brisbane.

[Sharp-Paul, 2012] Sharp-Paul, A. (2012).Scriptrock - frequently asked questions.https://www.scriptrock.com/faq.

[Whitty et al., 2010] Whitty, M., Cossell, S., Dang,K. S., Guivant, J. E., and Katupitiya, J. (2010).Autonomous navigation using a Real-Time 3D pointcloud. In Proceedings of the 2010 Australasian Con-ference on Robotics & Automation, Brisbane.