Tool for Tool for

Automated Automated

TestingTesting

P. SiveraEuropean Southern Observatory

Garching – January 2007 ESO

Garching, January 2007 Tool for Automated Testing 2

ALMA Project

tat

• tat - Tool for Automated Testing

– run multiple tests with one command– produce reproducible output– automatic setup– compare output against reference file(s)

Garching, January 2007 Tool for Automated Testing 3

ALMA Project

How tat works

• Looks for <mod_name>/test/TestList• Redefine LD_LIBRARY_PATH, PYTHONPATH, PATH

• Executes “make clean all”• Source a tcl script for additional env vars• Executes a prologue script • Executes the tests suite and compare output with

the reference• Executes an epilogue script• Executes “make clean”

Garching, January 2007 Tool for Automated Testing 4

ALMA Project

TestList directives

1. SOURCE <tcl_script_name>

2. PROLOGUE <prog_name> [<arg>]

3. <test_program_name> (1 or more)

4. EPILOGUE <prog_name> [<arg>]

The 3rd directive is the only mandatory one

Garching, January 2007 Tool for Automated Testing 5

ALMA Project

TestList syntax

• Blank lines are allowed

• Comment lines start with “#”

• A test directive can be split on multiple

lines => continuation character is “\”

Garching, January 2007 Tool for Automated Testing 6

ALMA Project

Test case syntax

• <testname>• <testNum> <testName> <proc 1> [<proc 2>..<proc n>]

Ex: 1 mySimpleTest checkCompStatus 2 myComplexTest sendMsg checkCompStatus setParam

makeCrash3 myComplexTest sendMsg “@SLEEP 10”

checkCompStatus “@SLEEP 10” “setParam 3” “@SLEEP 10” makeCrash

Garching, January 2007 Tool for Automated Testing 7

ALMA Project

Cookbook - 1

• getTemplate for MODROOT

• getTemplate for Makefile under test

• getTemplate for script/program

• update Makefile

• write/try the script/program

• prepare TestList

Garching, January 2007 Tool for Automated Testing 8

ALMA Project

Cookbook – 2

• create reference file:$ tat –g testscriptReference file acstHello.sh.ref generated

Reference file generated.

• run the test:$ tat testscriptTEST acstHello.sh PASSED.

PASSED.

Garching, January 2007 Tool for Automated Testing 9

ALMA Project

How to deal with the CDB

• Create directory:test/CDB

• Put your CDB files

• Redefine environment variable: ACS_CDB=`pwd`

(in the SOURCE script)

Garching, January 2007 Tool for Automated Testing 10

ALMA Project

How to run tat

• Run as– tat (make all, source, prologue, test suite,

compare with ref, epilogue, make clean)

• or– tat makeEnv (make all, source, prologue)– tat or tat <testitem>– tat cleanEnv (epilogue, make clean)

Garching, January 2007 Tool for Automated Testing 11

ALMA Project

Some options

• -v to run tat in verbose mode

• -nc (not clean) to run “make all”

and not

“make clean all”

• -f <filename> to use another different testlist besides the TestList

Garching, January 2007 Tool for Automated Testing 12

ALMA Project

Files and log files

• TestList.grep

→ grep –v –f TestList.grep <output>

• TestList.sed

→ sed –f TestList.sed <output>

• test/tatlogs/run$$ → .out, out.orig and .diff(.diff is the unix diff between .ref and .out)

Garching, January 2007 Tool for Automated Testing 13

ALMA Project

Example of SOURCE

• SOURCE needs a tcl script under test dir

• Add a line in TestList.lite:SOURCE <tclscript_name>.tcl

• Write the file:set env(VARIABLE) WHATEVER

Garching, January 2007 Tool for Automated Testing 14

ALMA Project

Example of prologue

#! /bin/bash

acsStart > $ACS_TMP/acsStart.log 2>&1 &

sleep 3

acsStartContainer bilboContainer >$ACS_TMP2/bilbo.log 2>&1 &

sleep 3

See standard prologue:

$ACSROOT/bin/acsutilTATPrologue

Garching, January 2007 Tool for Automated Testing 15

ALMA Project

Example of epilogue

#! /bin/bash

acsStopContainer bilboContainer >bilboStop.log 2>&1

sleep 3

acsStop >acsStop.log 2>&1

sleep 3

See standard epilogue:

$ACSROOT/bin/acsutilTATEpilogue