Post on 21-Jan-2016
description
Load Testing Using Perl
Homer Hummel
Jet Propulsion Laboratory, California Institute of Technology
Homer.Hummel@jpl.nasa.gov
2008-07-25
National Aeronautics & Space Administration
Presentation Overview
‣ UnixLoadGen
‣ Load Test Configuration, RSA Auth and Unix fork
‣ Typical LoadGen Run Scenario
‣ LoadGen directory structure
‣ File formats
‣ Perl scripts
‣ Real-world experiences
‣ General comparison with COTS
‣ References
UnixLoadGen
‣ A general framework for load testing application servers.
‣ Written in Perl.
‣ Uses “fork” feature of Unix.
‣ Uses “RSA Authentication” feature of SSH.
‣ UnixLoadGen distributed on website [TBD]
‣ UnixLoadGen – hereinafter referred to as LoadGen.
Typical Load Test Configuration
Generator 1
Generator 2
Generator X
Control
Workstation
Test Server
Use of RSA Authentication
Generator 1
ControlWorkstatio
n
tester
Public Key
tester
Private KeyPublic Key
Generator 2
Generator X
tester
tester
Public Key
Public Key
Control Workstation and Alternate
Generator 1
ControlWorkstn
tester
Public KeytesterPrivate KeyPublic Key Generator
2
Generator X
tester
tester
Public Key
Public Key
ControlWorkstn
(alt)tester
Private KeyPublic Key
Public Key
Public Key
Public Key
Use of Unix “fork”
Test Server
Generator
startN.pl Y load_script
load_script (1)
load_script (2)
load_script (Y)
Typical LoadGen Run Scenario
Generator 1Test
ServerControl Workstn
Edit scripts & data
push_files.pl
loadgen.pl
process_results.pl
Control file:Gen1 cmd
Gen2 cmd
GenX cmd
load_script (1)load_script (2)load_script (Y)
Generator 2load_script (1)load_script (2)load_script (Y)
Generator Xload_script (1)load_script (2)load_script (Y)
Typical LoadGen Run Scenario
Generator 1Test
ServerControl Workstn
Edit scripts & data
push_files.pl
loadgen.pl
process_results.pl
Control file:Gen1 cmd
Gen2 cmd
GenX cmd
Generator 2
Generator X
Typical LoadGen Run Scenario
Generator 1Test
ServerControl Workstn
Edit scripts & data
push_files.pl
loadgen.pl
process_results.pl
Control file:Gen1 cmd
Gen2 cmd
GenX cmd
scriptsdata files
Generator 2
Generator X
Typical LoadGen Run Scenario
Generator 1Test
ServerControl Workstn
Edit scripts & data
push_files.pl
loadgen.pl
process_results.pl
Control file:Gen1 cmd
Gen2 cmd
GenX cmd
scriptsdata files
Generator 2scriptsdata files
Generator X
Typical LoadGen Run Scenario
Generator 1Test
ServerControl Workstn
Edit scripts & data
push_files.pl
loadgen.pl
process_results.pl
Control file:Gen1 cmd
Gen2 cmd
GenX cmd
scriptsdata files
Generator 2scriptsdata files
Generator Xscriptsdata files
Typical LoadGen Run Scenario
Generator 1Test
ServerControl Workstn
Edit scripts & data
push_files.pl
loadgen.pl
process_results.pl
Control file:Gen1 cmd
Gen2 cmd
GenX cmd
scriptsdata files
Generator 2scriptsdata files
Generator Xscriptsdata files
Typical LoadGen Run Scenario
Generator 1Test
ServerControl Workstn
Edit scripts & data
push_files.pl
loadgen.pl
process_results.pl
Control file:Gen1 cmd
Gen2 cmd
GenX cmd
load_script (1)load_script (2)load_script (Y)
Generator 2
Generator X
Typical LoadGen Run Scenario
Generator 1Test
ServerControl Workstn
Edit scripts & data
push_files.pl
loadgen.pl
process_results.pl
Control file:Gen1 cmd
Gen2 cmd
GenX cmd
load_script (1)load_script (2)load_script (Y)
Generator 2load_script (1)load_script (2)load_script (Y)
Generator X
Typical LoadGen Run Scenario
Generator 1Test
ServerControl Workstn
Edit scripts & data
push_files.pl
loadgen.pl
process_results.pl
Control file:Gen1 cmd
Gen2 cmd
GenX cmd
load_script (1)load_script (2)load_script (Y)
Generator 2load_script (1)load_script (2)load_script (Y)
Generator Xload_script (1)load_script (2)load_script (Y)
Typical LoadGen Run Scenario
Generator 1Test
ServerControl Workstn
Edit scripts & data
push_files.pl
loadgen.pl
process_results.pl
Control file:Gen1 cmd
Gen2 cmd
GenX cmd
raw_results (1)raw_results (2)raw_results (Y)
Generator 2raw_results (1)raw_results (2)raw_results (Y)
Generator Xraw_results (1)raw_results (2)raw_results (Y)
Typical LoadGen Run Scenario
Generator 1Test
ServerControl Workstn
Edit scripts & data
push_files.pl
loadgen.pl
process_results.pl
Control file:Gen1 cmd
Gen2 cmd
GenX cmd
raw_results (1)raw_results (2)raw_results (Y)
Generator 2raw_results (1)raw_results (2)raw_results (Y)
Generator Xraw_results (1)raw_results (2)raw_results (Y)
Typical LoadGen Run Scenario
Generator 1Test
ServerControl Workstn
Edit scripts & data
push_files.pl
loadgen.pl
process_results.pl
Control file:Gen1 cmd
Gen2 cmd
GenX cmd
raw_results (1)raw_results (2)raw_results (Y)
Generator 2raw_results (1)raw_results (2)raw_results (Y)
Generator Xraw_results (1)raw_results (2)raw_results (Y)
Typical LoadGen Run Scenario
Generator 1Test
ServerControl Workstn
Edit scripts & data
push_files.pl
loadgen.pl
process_results.pl
Control file:Gen1 cmd
Gen2 cmd
GenX cmd
Generator 2
Generator X
Summary Test Report
Typical LoadGen Run Scenario
Generator 1Test
ServerControl Workstn
Edit scripts & data
push_files.pl
loadgen.pl
process_results.pl
Control file:Gen1 cmd
Gen2 cmd
GenX cmd
load_script (1)load_script (2)load_script (Y)
Generator 2load_script (1)load_script (2)load_script (Y)
Generator Xload_script (1)load_script (2)load_script (Y)
LoadGen Directory Structure
Home directory of local user account ‘tester’
control (on control workstation only)
datatest_type1 (e.g., dir)test_type2 (e.g., pop3)[etc.]
raw_resultstest_type1test_type2[etc.]
results (on control workstation only)
scripts
Unique Input Data Lists
names000 (file): names001 (file): …names199 (file):
testuser1 testuser51testuser9951
testuser2 testuser52testuser9952
… … …testuser50 testuser100
testuser10000
list_of_names_params_5.list00 (unique param list example):
-names data/dir/names000-names data/dir/names001-names data/dir/names002-names data/dir/names003-names data/dir/names004
list_of_names_params_5.list00 (one file per generator)list_of_names_params_5.list01…list_of_names_params_5.list09
Control File Format(General)
Generator_1 command_stringGenerator_2 command_string…Generator_X command_string
Example: file jabber.ctl:
gen-int-000 'export TERM=vt100; cd /home/tester/jabber/jabsimul/jab_simul; ./jab_simul > /tmp/jabber_output_gen-int-000.txt'gen-int-001 'export TERM=vt100; cd /home/tester/jabber/jabsimul/jab_simul; ./jab_simul > /tmp/jabber_output_gen-int-001.txt‘…gen-int-019 'export TERM=vt100; cd /home/tester/jabber/jabsimul/jab_simul; ./jab_simul > /tmp/jabber_output_gen-int-019.txt'
Control File Format(Y-instances of cmd_str)
Generator_1 startN.pl Y command_stringGenerator_2 startN.pl Y command_string…Generator_X startN.pl Y command_string
Control File Format(y-instances with unique params)
Generator_1 startNupl.pl Y list_of_names_params_5.list00 cmd_stringGenerator_2 startNupl.pl Y list_of_names_params_5.list01 cmd_string…Generator_X startNupl.pl Y list_of_names_params_5.listxx cmd_stringExample: file ldir_auth_mix_edg.ctl:
gen-int-001 scripts/startNupl.pl 10 data/dir/list_of_names_params_10.list00 scripts/lmodattr_lnx.pl -h ldaptest –b dirtest -w xxx -a jplalias -v tester1 -c dirtest -r -iterations 200gen-int-002 scripts/startNupl.pl 10 data/dir/list_of_names_params_10.list01 scripts/lmodattr_lnx.pl -h ldaptest -b dirtest -w xxx -a jplalias -v tester1 -c dirtest -r -iterations 200…gen-int-010 scripts/startNupl.pl 10 data/dir/list_of_names_params_10.list09 scripts/lsgetentry_lnx.pl -h ldaptest -b dirtest -w xxx -p -c uid=celachi -iterations 200
Raw Results File Naming & Format
Filename: test_run_id~servername~generator~log.pidFile contents:start_time,stop_timestart_time,stop_time… (one line for each iteration of load script transaction)
1212105348.215921,1212105349.3077101212105349.307803,1212105351.4219841212105351.422039,1212105353.1075501212105353.107605,1212105356.026773…
Example:File: mix1~ldaptest~gen-int-005~log.12403
Summary Report File Example
Server: ldaptest Load test: lsgetentry Test run: mix1Date/time of beginning of test: 2008-05-29 16:55:50Date/time of end of test: 2008-05-29 17:05:57Test duration: Hours: 0 Minutes: 10 Seconds: 7Values for minimum, average, maximum and standard deviation are in seconds.Min = 0.256 Ave = 2.894 Max = 6.293 Std Dev = 1.354Total number of transactions = 5000
ConcurrentTest Clients Users Iterations Errorsgen-int-003 5 1000 0gen-int-006 5 1000 0gen-int-007 5 1000 0gen-int-012 5 1000 0gen-int-018 5 1000 0
push_files.pl snippets
my $ctrl_c = 0; # initialize $ctrl_c flag$SIG{INT} = sub { $ctrl_c = 1 }; # set the $ctrl_c flag on interrupt
open CONTROL, "<../control/$control_file" or die "Can't open $control_file: $!";while (<CONTROL>) { next if ($_ =~ /^#/); # skip comment lines (# in column 1) ($mach, $remaining) = split /\s+/, $_, 2; push @machines, $mach; push @cmd_strings, $remaining;}close (CONTROL);
push_files.pl snippet
my $localhome = $ENV{HOME};
foreach my $mach (@machines) { my $remotehome = `ssh $mach 'echo \$HOME'`; chomp $remotehome; if ($scripts_flag == 1) { print "Pushing scripts directory...\n"; `scp -r $localhome/scripts $mach:$remotehome`; } if ($data_flag == 1) { print "Pushing data directory...\n"; `scp -r $localhome/data/$test_types[$i] $mach:$remotehome/data`; } print "Push files completed on $mach.\n"; last if ($ctrl_c == 1);}
push_files.pl snippet
my $localhome = $ENV{HOME};
foreach my $mach (@machines) { my $remotehome = `ssh $mach 'echo \$HOME'`; chomp $remotehome; if ($scripts_flag == 1) { print "Pushing scripts directory...\n"; `scp -r $localhome/scripts $mach:$remotehome`; } if ($data_flag == 1) { print "Pushing data directory...\n"; `scp -r $localhome/data/$test_types[$i] $mach:$remotehome/data`; } print "Push files completed on $mach.\n"; last if ($ctrl_c == 1);}
loadgen.pl snippetmy $errors = 0;my @pid;
foreach my $i (0..$#machines) {
FORK: { if ($pid[$i] = fork) { # Parent process, child's pid in the array next
} elsif (defined $pid[$i]) { # Child process print "Starting command string on $machines[$i]\n"; exec "ssh $machines[$i] $cmd_strings[$i] -test_run_id $test_run_id"; die "Exec failed after fork\n"; } elsif ($! =~ /EAGAIN/) { # EAGAIN is the supposedly recoverable fork error sleep 5; redo FORK; } else { die "Can't fork: $!\n"; } } sleep $ramp_up_delay;}
loadgen.pl snippetmy $errors = 0;my @pid;
foreach my $i (0..$#machines) {
FORK: { if ($pid[$i] = fork) { # Parent process, child's pid in the array next
} elsif (defined $pid[$i]) { # Child process print "Starting command string on $machines[$i]\n"; exec "ssh $machines[$i] $cmd_strings[$i] -test_run_id $test_run_id"; die "Exec failed after fork\n"; } elsif ($! =~ /EAGAIN/) { # EAGAIN is the supposedly recoverable fork error sleep 5; redo FORK; } else { die "Can't fork: $!\n"; } } sleep $ramp_up_delay;}
process_results.pl snippet
my $rawdir = "$localhome/raw_results/$test_type";
# get all raw_results files for test_typeif (-e $rawdir) { my @all_raw_results_files = `ls -1 $rawdir`; # -1 (one) option to # 'ls' cmd gives just file names}else { die "No raw results of test type: $test_type\n"; }
# collect relevant raw results file names by test run identifierforeach my $file (@all_raw_results_files) { chomp $file; my @file_name_portions = split /~/, $file; if ($file_name_portions[0] =~ /$test_run_id/) { push @raw_results_files, $file; }}
load_script_template.pl (part 1)
#!/usr/bin/perl
use strict;use warnings;use Getopt::Long;use Sys::Hostname;use Time::HiRes;
my $ctrl_c = 0; # initialize $ctrl_c flag$SIG{INT} = sub { $ctrl_c = 1 }; # set the $ctrl_c flag on interrupt
my $iterations = 3;my $delay_iteration = 0; # delay between iterations in secondsmy $test_run_id; # test run identifier string e.g. run01, aurnn14, or xyzmy $servername; # servername (unix hostname)
my $outdir = "raw_results/";my @start_times = (); # start timestamps in floating point secondsmy @stop_times = (); # stop timestamps in floating point seconds
load_script_template.pl (part 2)
&GetOptions( "iterations:i" => \$iterations, "delay_iteration:i" => \$delay_iteration, # delay 'tween iterations "test_run_id:s" => \$test_run_id, # test run identifier "servername:s" => \$servername, # servername
);
# if test_type subdir of raw_results does not exist, create it.# substitute "test_type" with the real one e.g. "dir"mkdir ($outdir . "test_type") unless -e ($outdir . "test_type");$outdir .= "test_type/";
my $logfile = $outdir . $test_run_id . "~$servername~" . hostname() . "~log.$$";# $logfile example: raw_results/dir/run01~devldap~gen-01~log.3257
load_script_template.pl (part 3)
# main processing loopforeach my $iter (1..$iterations) { push @start_times, Time::HiRes::time; # save start timestamp
# Perl code for client-server transaction goes here
push @stop_times, Time::HiRes::time; # save stop timestamp last if ($ctrl_c eq 1); # exit loop on interrupt sleep $delay_iteration; # delay in seconds before starting next iter}
# output raw timing resultsopen LOG, ">$logfile" or die "Can't create $logfile: $!";for my $i (0..$#start_times) { printf LOG ("%.6f,%.6f\n", $start_times[$i], $stop_times[$i]);}close (LOG);exit;
loadad.pl (part 1)
#!/usr/bin/perl # load Active Directory server use strict;use warnings;use Getopt::Long;use Sys::Hostname;use Time::HiRes;use Authen::Simple::ActiveDirectory; my $ctrl_c = 0; # initialize $ctrl_c flag$SIG{INT} = sub { $ctrl_c = 1 }; # set the $ctrl_c flag on interrupt my $iterations = 3;my $delay_iteration = 0; # delay between iterations in secondsmy $test_run; # test run identifier string e.g. run01, aurnn14, or xyzmy $servername; # servername (unix hostname)my $domain;my $names;
loadad.pl (part 1)
#!/usr/bin/perl # load Active Directory server use strict;use warnings;use Getopt::Long;use Sys::Hostname;use Time::HiRes;use Authen::Simple::ActiveDirectory; my $ctrl_c = 0; # initialize $ctrl_c flag$SIG{INT} = sub { $ctrl_c = 1 }; # set the $ctrl_c flag on interrupt my $iterations = 3;my $delay_iteration = 0; # delay between iterations in secondsmy $test_run; # test run identifier string e.g. run01, aurnn14, or xyzmy $servername; # servername (unix hostname)my $domain;my $names;
loadad.pl (part 2)my $username;my $password = "xxxxxxxx";my $outdir = "raw_results/";my @start_times = (); # start timestamps in floating point secondsmy @stop_times = (); # stop timestamps in floating point seconds &GetOptions( "iterations:i" => \$iterations, "delay_iteration:i" => \$delay_iteration, # delay 'tween iterations "test_run:s" => \$test_run, # test run identifier "servername:s" => \$servername, # servername "names:s" => \$names, # pathname of file of user names "domain:s" => \$domain, # Active Directory domain ); # if test_type subdir does not exist, create it.# substitute "test_type" with the real one e.g. "dir"mkdir ($outdir . "loadad") unless -e ($outdir . "loadad");$outdir .= "loadad/"; my $logfile = $outdir . $test_run . "~$servername~" . hostname() . "~log.$$";# $logfile example: raw_results/dir/run01~devldap~client01~log.3257 open NAMES, "<$names" or die "Can't open $names: $!";
loadad.pl (part 2)my $username;my $password = "xxxxxxxx";my $outdir = "raw_results/";my @start_times = (); # start timestamps in floating point secondsmy @stop_times = (); # stop timestamps in floating point seconds &GetOptions( "iterations:i" => \$iterations, "delay_iteration:i" => \$delay_iteration, # delay 'tween iterations "test_run:s" => \$test_run, # test run identifier "servername:s" => \$servername, # servername "names:s" => \$names, # pathname of file of user names "domain:s" => \$domain, # Active Directory domain ); # if test_type subdir does not exist, create it.# substitute "test_type" with the real one e.g. "dir"mkdir ($outdir . "loadad") unless -e ($outdir . "loadad");$outdir .= "loadad/"; my $logfile = $outdir . $test_run . "~$servername~" . hostname() . "~log.$$";# $logfile example: raw_results/dir/run01~devldap~client01~log.3257 open NAMES, "<$names" or die "Can't open $names: $!";
loadad.pl (part 3)# main processing loopforeach my $iter (1..$iterations) { $username = <NAMES>; # read a line from file of user names if (!($username)) { # if end-of-file seek NAMES,0,0; # point to beginnning of file $username = <NAMES>; # and read the first line } chomp $username; # discard newline push @start_times, Time::HiRes::time; # save start timestamp my $ad = Authen::Simple::ActiveDirectory->new( host => $servername, principal => $domain, ); if ( $ad->authenticate( $username, $password ) ) { # print "Good authentication!\n"; # for debugging single instance } else { print "Bad authentication!\n"; } push @stop_times, Time::HiRes::time; # save stop timestamp last if ($ctrl_c eq 1); # exit loop on interrupt sleep $delay_iteration; # delay in seconds before starting next iter}
loadad.pl (part 3)# main processing loopforeach my $iter (1..$iterations) { $username = <NAMES>; # read a line from file of user names if (!($username)) { # if end-of-file seek NAMES,0,0; # point to beginnning of file $username = <NAMES>; # and read the first line } chomp $username; # discard newline push @start_times, Time::HiRes::time; # save start timestamp my $ad = Authen::Simple::ActiveDirectory->new( host => $servername, principal => $domain, ); if ( $ad->authenticate( $username, $password ) ) { # print "Good authentication!\n"; # for debugging single instance } else { print "Bad authentication!\n"; } push @stop_times, Time::HiRes::time; # save stop timestamp last if ($ctrl_c eq 1); # exit loop on interrupt sleep $delay_iteration; # delay in seconds before starting next iter}
Some utility scripts in the distribution
stop.plpsignal.plcheck_generators.plmakelists.plmakelistoflists.plservermon.plswingbench.pl
Real-world Experience
• Ability to integrate load scripts written in other languages.• World’s first load test of Kerberos authentication servers.• Driver for other simulators.• LoadGen used for LDAP Directory, Active Directory, pop3, imap4,
Remedy, Kerberos and Jabber servers.
General Comparison with COTS
Commercial Tools
LoadGen
Flexibility limited open
Scalability yes, but costs extra
yes
Availability some apps not available
write your own (use CPAN)
Ease of use graphical, click to make load script
write your own load script
Cost $ tens of thousands
open source
Some Relevant References
UnixLoadGen User’s Guide, by Hummel (included in distribution)SSH The Secure Shell by Barrett & Silverman,
publisher O’ReillyLinux System Programming by Love,
publisher O’ReillyProgramming Perl 3rd Ed. by Wall, Christiansen & Orwant,
publisher O’Reilly
Acknowledgements
CSC Management Perl Advisors
Mike Gross Peter Scott
Todd Lucas Patrick Ward
Eric Gerritsen Thomas Berry
Virinder DhillonJPL Management
Kevin Klenk cpan.org sponsors &
Sarala Rajeshuni authors, specifically
Henry Dillard Christian HansenLuke Dahl
Load Testing Using Perl
Homer Hummel
Jet Propulsion Laboratory, California Institute of Technology
Homer.Hummel@jpl.nasa.gov
2008-07-25
National Aeronautics & Space Administration