Post on 22-Apr-2015
description
YAPC::NA 10Pittsburgh, PA
TAP in Depth Michael Peters
Plus Three, LP
TAP
Test Anything Protocol● And we do mean *anything*
● Mainly used for regression testing code● Could be used for systems monitoring,
performance testing, etc● 1987, current version 13● Patient Zero was Perl, but the infection has since
spread to lots of other languages● Ongoing work to make an IETF standard
Definitions
Test● or an "assertion" (ok/not ok)● smallest granule of measurement● did it do what we wanted
Stream● or a "set"● a group of tests● usually output of running a single test file, but is not
necessarily tied to a file or any specific output.
Test Suite● A group of TAP streams
Definitions
Harness● A process which collects a group of TAP streams
Plan● number of tests that are expected to be run in a given
stream
Directive● Changes the meaning of a given test
● TODO/SKIP
Reason● Explains the "why" of a directive
Definitions
Description● What is the test asserting? What functionality is
being tested?
Producer● Anything that generates TAP
Consumer● Anything that processes TAP
● Could just blindly echo it
● Usually parses it
Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed
Document Structure
Line Oriented Protocol● Everything is separated by a new-line (CRLF)
● Not everything fits on 1 line
● Everything starts and ends on it's own line
Encoding● Everything is UTF-8
● Unless the Producer/Consumer agree otherwise
1 Version, 1* plans, 0+ tests,
0* comments, 0* ignored elements
Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed
Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed
Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed
Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed
Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed
Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed
Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed
Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed
Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed
Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed
Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed
Pass vs Fail
Tests● "ok" == pass
● "not ok" == fail
● Yay! binary!
Pass vs Fail
Tests● "ok" == pass
● "not ok" == fail
● If only it were that simple
● Directives!
Normal TODO SKIP
ok pass * pass
not ok pass fail **
* a passing TODO might indicate that a test shouldn't be a TODO anymore
** SKIP means the test wasn't run, so it shouldn't be able to fail
Pass vs Fail
Streams● Can fail if any of the following are true
(1) Any tests in this stream failed
(2) Tests run did not agree with the plan
(3) Other factors● process exit codes (non-zero)● HTTP status code
● Passed only if none of the above are true and
(1) There was at least 1 passing test - or -
(2) There was a "skip all"● 1..0 # SKIP some module not installed
Pass vs Fail
Test Suites● Fails if there are any streams that failed
● Passes if nothing failed and there's at least 1 stream
● Pass or fail at the suite level is boolean
● Tools can show percentages
● But even a 99.999% pass is still a fail
Producing
Perl Modules● Test::More (classic and for good reasons)
Producing
Perl Modules● Test::More (classic and for good reasons)
● Test::Builder
Producing
Perl Modules● Test::More (classic and for good reasons)
● Test::Builder
● Test::Most
Producing
Perl Modules● Test::More (classic and for good reasons)
● Test::Builder
● Test::Most
● Test::Class
Producing
Perl Modules● Test::More (classic and for good reasons)
● Test::Builder
● Test::Most
● Test::Class
● Test::LongString, Test::WWW::Mechanize, Test::Exception, Test::File::Contents, Test::HTML::Content, Test::MockObject, Test::Pod, Test::Warn, Test::WWW::Selenium, Test::NoWarnings, Test::Output, Test::Number::Delta, Test::Perl::Critic
Producing
Other Languages● Javascript →Test.Simple
● C/C++ →libtap, libtap++
● PHP → phpt, PHPUnit, SimpleTest or Apache-Test
● Java →JTap
● Ruby →taptap, Bacon
● Python → PyTAP
● Also Haskell, Erlang, Forth, C#, Shell, Postgres
Consuming
Perl● Test::Harness
● prove
● TAP::Harness
● TAP::Harness::Archive
● GUI Test Runner
● Smolder
Comming Soon to a terminal near you
Nested TAP● Group related blocks of TAP together in the same
stream
● Mimic xUnit style
● Aggregate TAP streams
Comming Soon to a terminal near you
Nested TAP1..3
1..3
ok 1 - guests arrived on time
ok 2 - table was available
ok 3 - order correct
ok 1 - guests served
1..2
ok 1 - chicken is fried
not ok - steak is medium well
not ok 1 - food prepared properly
1..2
ok 1 # SKIP guests did not come back
ok 2 # SKIP guests did not come back
ok 1 # SKIP guests did not come back
Comming Soon to a terminal near you
Metadata
● Everything is crammed into the descriptions or comments
not ok - fooey, Bug #123 # Failed test 'fooey, Bug #123' # at barry.pl line 20. # got: foo bar # expected: bar foo
● If you want to do anything useful with the information in that message you need to parse it. Parsing error messages is evil
Comming Soon to a terminal near you
Metadata● Data should be structured (YAML-ish)
not ok - fooey --- file : barry.pl line : 20 want : foo bar have : bar foo display : > Failed test 'fooey' at barry.pl line 20. want: "foo bar" have: "bar foo" Bug-ID : 123...
Comming Soon to a terminal near you
Metadata● Opens up a whole world of possibilities
● Visual diffs on want/have
● Tagging
● Linking between systems (source code, specs, issue tracking)
● Consumers can show or hide what ever they want or understand.
● Unknown uses (standardization vs growth)
Usage Scenarios
Typical Perl tests
Usage Scenarios
Automated Javascript Testing
* example from Michael Schwern - http://use.perl.org/~schwern/journal/39092
Usage Scenarios
Automated Javascript Testing
Usage Scenarios
Automated Javascript Testing
Thanks!Questions?
TAP in Depth Michael Peters
Plus Three, LP