BERZERQ Testing BERZERQ Testing SystemSystem

2006 Erich Gatejen

SummarySummary

In-house developed QA tool.◦No commercial tools suitable. Most

geared to GUI/Web testing. Data management poor.

It’s not just tests, but tools and data too.

Quality Assurance is more than just banging on a keyboard.◦Tests to run.◦Tools to help.◦Data to use.◦And… information for everyone.

The PiecesThe Pieces

BERZERQ - Main package- Add-On package- BERZERQI- BERZERQT- BERZERQsql- Install Helper

QA TesterThe ToolsData SetsSample DataThe RepositoryOdds and Ends

BERZERQBERZERQ

Complete, installable test package.◦ One-stop-shopping, including test data and test

examples. ◦ Treats the test tools like a polished product.◦ Includes documentation, installation instructions,

and complete source code.◦ Everyone in QA has contributed to the package.

Latest version in public repository.◦ Look for berzerq-RELEASE-*-binary.zip. That is the

main package.Windows and Linux supported in the same

package. ◦ Automated Build2BAT for Linux only.◦ Verified on WinXP, RH9, and Fedora 3 and 4. It

should work on most similar platforms.

BERZERQ BERZERQ (Main Package)(Main Package)

The following pages show a sampling of tools and tests provided.

There is full DOCUMENTATION.◦Look under /doc directory after you

install. Double pointy-clicky the CLICK-ME.html.

There is more. If you are looking for something, ask: egatejen@....

BERZERQBERZERQ (Main Package)(Main Package) TestsTests

Build2BAT◦ This is the complete, automated Smoke Check that is

run for every build.◦ Covers most of the high-level functionality in current

products.◦ The main package provides the testing portion of the

Build2BAT. BERZERQI provides the automated install tools.

You can run all the tests independently too.Convenience scripts under /script run the BAT

or the independent tests. ◦ Or run them in the custom Command Line Interface

(CLI).

Some of the tests in the Build2BAT:◦ Data storage and validation.◦ Data retrieval and validation.◦ GUI login and interaction

Standard configurations. Each screen touched at least once.

◦ Various performance metrics Throughput Startup timing. Et al.

◦ Data management cases◦ And much more….. And it is all documented.

BERZERQ (Main Package) Build2BAT

/script/(version)/ has convenience scripts.Branched by supported test target version.There are large-scale SUITES and shorter

TESTS.The BAT (Smoke Check) SUITE

◦ “run-SUITE-bat.sh” starts the process◦ “report-SUITE-bat.sh” collects results.◦ “run-SUITE-round2.sh and run-SUITE-round3.sh”

complete the set.

Performance SUITE◦ run-SUITE-perf.sh◦ Apples2apples testing across a wide range of data

methods.

BERZERQ (Main Package) Convenience Scripts

Run individual tests◦ Named script, such as “run-TEST-large.sh”◦ Or through the “run-TEST.sh” script.◦ All documented or just run the command for help.

SUITE reports bundled and posted as a web site.◦ Hosted by the test driver machine.◦ Ask QA for the URLs.

Includes a set of pre-configurations that point the tests at the QA labs.◦ But it’s easy to create a configuration that points to any

lab.

BERZERQ (Main Package) Convenience Scripts

XML script system based on the AUTOHIT test system. http://sourceforge.net/projects/autohit/

It was customized for specific test needs. Why this tool?

◦ Needed a general purpose tool, not a GUI tester.◦ Lots of free support. All the pieces are open source or

written by QA team members.◦ It’s FAST, particularly for data test.◦ Easy to extend with java libraries. It took only one day to

add support for testing the SOAP APIs. Is this the tool of the future?

◦ No. While it can do some GUI testing, it’s not good at it.◦ It’ll be the tool of choice for data and throughput tests.

BERZERQ (Main Package) Implementation

Many of these are used in the build the Build2BAT.

Scripted Tests (a sampling)

◦Store/Search/Retrieve Data Lone, looped, or from a LIST (which can be

generated by the other tools, such as Scrubber, or by hand).

ALL external APIs and interfaces supported. Search can work on arbitrary fields. Stress and load testing for all.

◦GUI actions Basic action validation and stress. You can script the actions too! With a simple text

file.◦Data and validation regression and stress.

BERZERQ (Main Package) Sub-tests and building blocks

Scripted Tests (continued)

◦Data management modules Date management for configured ranges from1995

through 2010. Three different management policies Tested against all major data formats.

◦Regular and Compressed (ie Zip) data regression. Including cases for bugs in the backlog.

◦GUI regression, report accuracy, and stress, including the two top data formats. Reconfigurations exist for more than the currently

supported users, so a breaking stress test is easy to run!

BERZERQ (Main Package) Sub-tests and building blocks

Scripted Tests (continued)

◦Negative data cases.◦Interface testing for two custom

solutions for Customer A and Customer B.

◦Backup solution validation (but this test takes a very long time to run).

◦Performance testing Storage for all external interfaces and API GUI integration steps Large documents (through 1.5GB).

BERZERQ (Main Package) Sub-tests and building blocks

Scripted Tests (continued)◦Data manipulation module validation for

the two most commonly used modules◦Data injection for tests not automated in

this package.Scripted Tools

◦Data transformations from common external format to custom format used by two customers.

◦Analyze recognition hit rates.

BERZERQ (Main Package) Sub-tests and building blocks

The following is a sampling. All tools can be found in the /bin directory. Most support Windows and Linux.

Scrubber◦ Converts batches of generic data into useful test data.◦ Remembers key information about the data that can be used

for.◦ Creates formatted lists that can be run as an automated test in

the Test Script system.CrawlNSend

◦ Spiders a directory tree, scrubs data, and applies it to the test system.

◦ Has all the features of the Scrubber and will uniquify all data as it is processed.

◦ Fast and efficient. Have run hundreds of threads on the same machine.

◦ There is a C version of the tool that is useful for very high-load performance testing. But it has less features.

BERZERQ (Main Package) Independent Tools

BERZERQBERZERQ (Main Package)(Main Package) Independent ToolsIndependent Tools

SetBuilder◦ Builds a set of test data from a template.

ReOwner◦ Changes the ownership in a sets of data to test related

features in the product.

Xml2sml◦ Converts configuration exports from various products into

an Excel spreadsheet and back. Alaunch

◦ Launches batch processing for one product.◦ Flexible configuration based testing.

Droprun◦ Uses scripts to generate and deposit files on a foreign system

(using sftp/scp).◦ Created for Data Collector testing, but useful for general purposes.

Reducer◦ A collection of data reduction tools supporting a wide range of

reports.◦ Most of the tools run on the data generated by the other tools. ◦ Written over a reduction framework, so creating new reports for

new data sources is easy—most take only a couple hours.DSC

◦ Generates certain configuration files using parameters. backuptests.sh

◦ Offline backup validation.

BERZERQ (Main Package) Independent Tools

There are a number of canned configurations that automate the testing process for CrawlNSend, droprun, and backuptests.

The configurations cover the following tests out of the box (a sampling, there are more) :◦ Data load tests.◦ 3-day processing load and backend selection test.◦ Backup CPIO absolute validation◦ Data Collector regression and stress, including stress,

functionality, and the backup process.◦ Diverse data validation over all supported formats.◦ Load generation for system negative cases (i.e.,

keeps the system busy while we do Bad Things to it, such as forced outages).

◦ Owner and User account validation

BERZERQ (Main Package) Independent Tools

Additional test data needed by some of the advanced tests.◦Needed for the Regression Suite.◦Storage parameters (FULL3)

It is BIG, so should only be loaded if needed.

BERZERQ Add-On Package

This small package offers some of the tools from the main package.

Configuration helpers (i.e. configuration XML generators).

CrawlnSendData reduction toolsAnd others…

BERZERQT

BERZERQIBERZERQI

A small package of install helpers. ◦ Loaded on the Configuration Server.

Provides the install portion of the Build2BAT◦ Can automatically install new binaries and prepare lab for

fresh Smoke Check.

Does far more than just binaries load.Implemented as a collection of

component scripts◦ You can run the individual scripts to save time, even if you

don’t want to run the complete automated install.

BERZERQsqlBERZERQsql

A small package of database management helpers. ◦Loaded on the database machines.◦Statistics and usage monitoring.◦Processing and configuration status and

monitoring.◦And more…

A collection of bash scriptsPerform useful queries for monitoring

and data collection.

BERZERQERBERZERQER

Complete Berzerq tool set◦Except the test data

TimeWarper◦Shift entire test lab in time◦Useful for date and time sensitive test

case

QA TesterQA Tester

A Windows based testing tool. ◦ Meant to support testing on Windows based interface

products.◦ Two GUIs

General Tool Web Services Tool

Replicates a lot of the functions of BERZERQ ◦ Code, data, and methodologies are shared.◦ Bundled with the BERZERQ package, but not strictly part of

it. Based on the .NET and written C#. It has an easy to use GUI. Functions are added as needed for specific tests.

The ToolsThe Tools

There are several tools that are not bundled with Berzerq or QA Tool.

Most of used for advanced testing and data analysis.

Examples:◦ C based data manager and storage agent used for

precise load and performance testing.◦ Data Perturber for the ultimate in negative data test

cases. ◦ Token generators for guaranteed unique data over a large

data set.◦ Backend processing Stress Tester.

Data SetsData Sets

Good data is as important as good tests for finding bugs.

High quality data has done more for finding bugs than any other QA effort in the last several years.

We now have a wide range of data, most of which free from IP or ownership concerns.◦ In the old days, using customer data was the only

way to find data bugs. However, that meant we didn’t find the bugs until the software went into production.

◦ With a few exceptions, customers do not want us using their data for testing.

Data SetsData SetsSources

◦Email lists and personal data traps◦News groups◦Spamarchive.org◦A few customers (with their permission)◦Hand-crafted for specific cases◦Automatically generated

Usually to include various document attachments.

◦Samples for various integrated 3rd party products

Data SetsData Sets The Process

◦ Scrubbing using to the tools. ◦ Generates SETS. Each message is a file.◦ Most of the tools can uniqify every message in the SET

before it is transmitted or stored. The Output

◦ The data files.◦ Info.txt file with basic information.◦ A list with metadata (which can be used to run tests in

Berzerq and QA Tool).◦ A list of character sets found in all the messages (to

help with internationalization testing).◦ A script that will copy all the message with non-US

character sets to an alternate location. All of this can be done on the fly with

CrawnNSend and other tools, but the output will be different each time. ◦ A Data Set ensures the same data every time. ◦ However, a Data Set can be modified during use by

the various tools too.

Data Sets – Why?Data Sets – Why?Our customers put real data through

our products.We must do the same in testing!

◦ This isn’t theory; this is fact.◦ We have experienced significant downtime because of

customer data issues. Real data means real problems.

◦ Viruses, Worms, and all sorts of Bad Things We run Virus tools in our test environment, because

this is what our customers do.◦ Poorly formatted data, specification violations, and

random junk.◦ Big data, small data, short data, tall data.

Data Sets - BewareData Sets - Beware

DO NOT OPEN CUSTOMER OR REAL WORLD DATA IN MICROSOFT OFFICE!◦Use a text editor. I suggest UltraEdit.◦If you must, do it on an isolated client

that you are willing to rebuild frequently. Ghost images can make this easy.

The content of REAL WORLDdata is not always G-rated.

Sample DataSample DataSupports Product A testingCovers every aspect of configuration

◦ User administration◦ Groups.◦ Data Management.◦ All of it…

Allows us to go from clean, empty database to testing in just a couple hours.

Give us a predicable environment.◦ Though we are constantly adding elements to cover new

tests.The design is documented and available on

request.Two complete configurations that model two

large customers.

The RepositoryThe Repository

QA is an open book.All of our documents and many of

our tests can be found at:\\xxx\yyy\zzz

If it isn’t there, it’s probably because it is too big (particularly the Data Sets). Ask us!

Odds and EndsOdds and Ends

Questions and comments? egatejen@...

We are looking at our future needs. Any suggestions are always welcome.