TAP In Depth

YAPC::NA 10Pittsburgh, PA

TAP in Depth Michael Peters

Plus Three, LP

Test Anything Protocol● And we do mean *anything*

● Mainly used for regression testing code● Could be used for systems monitoring,

performance testing, etc● 1987, current version 13● Patient Zero was Perl, but the infection has since

spread to lots of other languages● Ongoing work to make an IETF standard

Definitions

Test● or an "assertion" (ok/not ok)● smallest granule of measurement● did it do what we wanted

Stream● or a "set"● a group of tests● usually output of running a single test file, but is not

necessarily tied to a file or any specific output.

Test Suite● A group of TAP streams

Definitions

Harness● A process which collects a group of TAP streams

Plan● number of tests that are expected to be run in a given

stream

Directive● Changes the meaning of a given test

● TODO/SKIP

Reason● Explains the "why" of a directive

Definitions

Description● What is the test asserting? What functionality is

being tested?

Producer● Anything that generates TAP

Consumer● Anything that processes TAP

● Could just blindly echo it

● Usually parses it

Document Structure

TAP version 13

ok 1 - Input file opened

not ok 2 - First line of the input valid

ok 3 - Read the rest of the file

# Can't load Digest::MD5. This modules is needed

# to run some optional tests

ok 4 # SKIP MD5 library not installed

not ok 6 - Correct summary # TODO Not written yet

ok 7 - File closed

Document Structure

Line Oriented Protocol● Everything is separated by a new-line (CRLF)

● Not everything fits on 1 line

● Everything starts and ends on it's own line

Encoding● Everything is UTF-8

● Unless the Producer/Consumer agree otherwise

1 Version, 1* plans, 0+ tests,

0* comments, 0* ignored elements

Document Structure

TAP version 13

ok 7 - File closed

Document Structure

TAP version 13

ok 7 - File closed

Document Structure

TAP version 13

ok 7 - File closed

Document Structure

TAP version 13

ok 7 - File closed

Document Structure

TAP version 13

ok 7 - File closed

Document Structure

TAP version 13

ok 7 - File closed

Document Structure

TAP version 13

ok 7 - File closed

Document Structure

TAP version 13

ok 7 - File closed

Document Structure

TAP version 13

ok 7 - File closed

Document Structure

TAP version 13

ok 7 - File closed

Document Structure

TAP version 13

ok 7 - File closed

Pass vs Fail

Tests● "ok" == pass

● "not ok" == fail

● Yay! binary!

Pass vs Fail

Tests● "ok" == pass

● "not ok" == fail

● If only it were that simple

● Directives!

Normal TODO SKIP

ok pass * pass

not ok pass fail **

* a passing TODO might indicate that a test shouldn't be a TODO anymore

** SKIP means the test wasn't run, so it shouldn't be able to fail

Pass vs Fail

Streams● Can fail if any of the following are true

(1) Any tests in this stream failed

(2) Tests run did not agree with the plan

(3) Other factors● process exit codes (non-zero)● HTTP status code

● Passed only if none of the above are true and

(1) There was at least 1 passing test - or -

(2) There was a "skip all"● 1..0 # SKIP some module not installed

Pass vs Fail

Test Suites● Fails if there are any streams that failed

● Passes if nothing failed and there's at least 1 stream

● Pass or fail at the suite level is boolean

● Tools can show percentages

● But even a 99.999% pass is still a fail

Producing

Perl Modules● Test::More (classic and for good reasons)

Producing

● Test::Builder

Producing

● Test::Builder

● Test::Most

Producing

● Test::Builder

● Test::Most

● Test::Class

Producing

● Test::Builder

● Test::Most

● Test::Class

● Test::LongString, Test::WWW::Mechanize, Test::Exception, Test::File::Contents, Test::HTML::Content, Test::MockObject, Test::Pod, Test::Warn, Test::WWW::Selenium, Test::NoWarnings, Test::Output, Test::Number::Delta, Test::Perl::Critic

Producing

Other Languages● Javascript →Test.Simple

● C/C++ →libtap, libtap++

● PHP → phpt, PHPUnit, SimpleTest or Apache-Test

● Java →JTap

● Ruby →taptap, Bacon

● Python → PyTAP

● Also Haskell, Erlang, Forth, C#, Shell, Postgres

Consuming

Perl● Test::Harness

● prove

● TAP::Harness

● TAP::Harness::Archive

● GUI Test Runner

● Smolder

Comming Soon to a terminal near you

Nested TAP● Group related blocks of TAP together in the same

stream

● Mimic xUnit style

● Aggregate TAP streams

Nested TAP1..3

ok 1 - guests arrived on time

ok 2 - table was available

ok 3 - order correct

ok 1 - guests served

ok 1 - chicken is fried

not ok - steak is medium well

not ok 1 - food prepared properly

ok 1 # SKIP guests did not come back

Metadata

● Everything is crammed into the descriptions or comments

not ok - fooey, Bug #123 # Failed test 'fooey, Bug #123' # at barry.pl line 20. # got: foo bar # expected: bar foo

● If you want to do anything useful with the information in that message you need to parse it. Parsing error messages is evil

Metadata● Data should be structured (YAML-ish)

not ok - fooey --- file : barry.pl line : 20 want : foo bar have : bar foo display : > Failed test 'fooey' at barry.pl line 20. want: "foo bar" have: "bar foo" Bug-ID : 123...

Metadata● Opens up a whole world of possibilities

● Visual diffs on want/have

● Tagging

● Linking between systems (source code, specs, issue tracking)

● Consumers can show or hide what ever they want or understand.

● Unknown uses (standardization vs growth)

Usage Scenarios

Typical Perl tests

Usage Scenarios

Automated Javascript Testing

* example from Michael Schwern - http://use.perl.org/~schwern/journal/39092

Usage Scenarios

Thanks!Questions?

TAP in Depth Michael Peters

Plus Three, LP

TAP In Depth

Technology

Transcript of TAP In Depth