Testing

Chris Dolan

Equilibrious LLC

cdolan@cpan.org

April 15, 2008

http://chrisdolan.net/madmongers/testing.html

Why test?

• Nobody likes bugs
• The sooner you find a bug, the easier it is to fix
• Tests build confidence about the code

Bonus: testing forces you to modularize your code

Types of testing

• Compiler, Compiler in strict mode
• Human vs. automated testing
• Unit vs. integration vs. functional testing
• Runtime testing (taint, assert, log)
• Conformity testing
• Targeted bug searches
• Fuzz testing

Advantages of automated testing

Computers are tireless

• Make them test over and over
• Make them run hundreds (millions?) of tests
• Documentation of what you tested

Bonus: a test is an example of intended use

Quality of testing

Code coverage: how much of the software is tested?

Realism: do the tests represent actual use?

Priority: are you testing the important code?

Reward: are you testing the hardest code?

Test::More

• TAP = Test Anything Protocol
• Starts with a plan of how many tests will run
• Prints "ok" if a test passes, prints "not ok" if a test fails
• Supports skip and todo tests

An example

Compute the day of the month for a given MadMongers meeting

• Constant: 3rd Tuesday of the month
• Input: year and month
• Output: day of month

First implementation: use Time::Local to compute the day of week given a date.

(then test, then change implementation and re-test)

Implementation

lib/Calendar/MadMongers.pm

 1: package Calendar::MadMongers;
 2: 
 3: use warnings;
 4: use strict;
 5: use Readonly;
 6: use Time::Local qw(timelocal);
 7: 
 8: # Hard-code 3rd Tuesday
 9: # Sun=0, Mon=1, Tue=2, ...
10: Readonly my $DAY_OF_WEEK => 2;
11: Readonly my $WEEK_OF_MONTH => 3;
12: 

Implementation

13: sub calc_date {
14:    my ($pkg, $year, $month) = @_;
15:    my @date = (0, 0, 19, 1, $month-1, $year-1900);
16:    #           s  m  hh  d  month     year
17:    @date = localtime(timelocal(@date));
18:    while ($date[6] != $DAY_OF_WEEK) {
19:       $date[3]++;
20:       @date = localtime(timelocal(@date));
21:    }
22:    for (my $i = $WEEK_OF_MONTH; $i > 1; $i--) {
23:       $date[3] += 7;
24:    }
25:    return $date[3];
26: }

Test::More

• ok($boolean, $comment);
• is($got, $expected, $comment);
• isnt($got, $expected, $comment);
• like($got, qr/regex/, $comment);
• unlike($got, qr/regex/, $comment);
• is_deeply($got, [1,2,{foo=>'bar'}], $comment);

Example

t/calendar.t

 1: #!/usr/bin/perl -w
 2: 
 3: use strict;
 4: use Calendar::MadMongers;
 5: use Test::More tests => 4;
 6: 
 7: my $pkg = 'Calendar::MadMongers';
 8: is($pkg->calc_date(2008, 4), 15, 'April 2008');
 9: is($pkg->calc_date(2008, 5), 20, 'May 2008');
10: is($pkg->calc_date(2008, 6), 17, 'June 2008');
11: is($pkg->calc_date(2008, 7), 15, 'July 2008');

TAP Output

perl -Ilib t/calendar.t

1..4
ok 1 - April 2008
ok 2 - May 2008
ok 3 - June 2008
ok 4 - July 2008

Test::More style

Optimized for concise, linear test programs. Common usage:

• Emphasize what you're testing, not how you're testing
• If a test is unrecoverable, you explicitly abort
• Split topics into separate .t files

Incorrect test

t/calendar-wrong.t

 1: #!/usr/bin/perl -w
 2: 
 3: use strict;
 4: use Calendar::MadMongers;
 5: use Test::More tests => 4;
 6: 
 7: my $pkg = 'Calendar::MadMongers';
 8: is($pkg->calc_date(2008, 4), 16, 'April 2008');
 9: is($pkg->calc_date(2008, 5), 20, 'May 2008');
10: is($pkg->calc_date(2008, 6), 17, 'June 2008');
11: is($pkg->calc_date(2008, 7), 15, 'July 2008');

Output from failing test

perl -Ilib t/calendar-wrong.t

1..4
not ok 1 - April 2008
#   Failed test 'April 2008'
#   at testing-example/t/calendar-wrong.t line 8.
#          got: '15'
#     expected: '16'
ok 2 - May 2008
ok 3 - June 2008
ok 4 - July 2008
# Looks like you failed 1 test of 4.

Test::Harness

If you have your code set up the right way, Test::Harness makes running tests easier.

1. Put code in lib, e.g. lib/Foo/Bar.pm
2. Put test programs in t, e.g. t/bar.t
3. Run "prove -l" or "prove -lv"

Test::Harness output

prove -l t/calendar.t

t/calendar......ok   
All tests successful.
Files=1, Tests=4,  0 wallclock secs (0.01 usr 0.01 sys)
Result: PASS

Test::Harness verbose output

prove -lv t/calendar.t

t/calendar......
1..4
ok 1 - April 2008
ok 2 - May 2008
ok 3 - June 2008
ok 4 - July 2008
ok
All tests successful.
Files=1, Tests=4,  0 wallclock secs (0.02 usr 0.01 sys)
Result: PASS

Test::Harness failure output

prove -l t/calendar-wrong.t

t/calendar-wrong......1/4 
#   Failed test 'April 2008'
#   at t/calendar-wrong.t line 8.
#          got: '15'
#     expected: '16'
# Looks like you failed 1 test of 4.
t/calendar-wrong...... Dubious, test returned 1
 Failed 1/4 subtests 

Test Summary Report
-------------------
t/calendar-wrong.t (Wstat: 256 Tests: 4 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
Files=1, Tests=4,  0 wallclock secs (0.01 usr 0.01 sys)
Result: FAIL

Refactor Calendar::MadMongers

13: use DateTime;
14: 
15: sub calc_date {
16:    my ($pkg, $year, $month) = @_;
17:    my $date = DateTime->new(year => $year,
18:                             month => $month);
19:    while ($date->day_of_week != $DAY_OF_WEEK) {
20:       $date->add(days => 1);
21:    }
22:    for (my $i = $WEEK_OF_MONTH; $i > 1; $i--) {
23:       $date->add(days => 7);
24:    }
25:    return $date->day_of_month;
26: }

Output

perl -Ilib t/calendar.t

1..4
ok 1 - April 2008
ok 2 - May 2008
ok 3 - June 2008
ok 4 - July 2008

Devel::Cover

perl -MDevel::Cover -I lib t/calendar.t

...
-------------------------- ------ ----- ----- ------ ----
File                         stmt  bran  cond    sub  pod
-------------------------- ------ ----- ----- ------ ----
lib/Calendar/MadMongers.pm  100.0   n/a   n/a  100.0  0.0
t/calendar.t                100.0   n/a   n/a  100.0  n/a
Total                       100.0   n/a   n/a  100.0  0.0
-------------------------- ------ ----- ----- ------ ----

Devel::Cover

cover -report html

Writing HTML output to cover_db/coverage.html ...
done.

Devel::Cover

...

Test::Class

Java, C++ and other languages like "xUnit" testing, popularized by JUnit

• Tests are spit into objects and linear methods instead of linear files
• Methods are encouraged to abort (die) on any failure
• Separate setup and teardown for each method and/or class
• Often test classes mirror production classes

Test::Class example code 1/3

lib/Calendar/MadMongers/Object.pm

 1: package Calendar::MadMongers::Object;
 2: use Carp;
 3: 
 4: sub new {
 5:    my ($class) = @_;
 6:    return bless { events => [] }, $class;
 7: }
 8: 
 9: sub all {
10:    my ($self) = @_;
11:    return @{$self->{events}};
12: }
13: 

Test::Class example code 2/3

lib/Calendar/MadMongers/Object.pm

14: sub add {
15:    my ($self, $what, $when) = @_;
16:    if (!$when || !$when->isa('DateTime')) {
17:       croak 'Expect a DateTime instance';
18:    }
19:    push @{$self->{events}},
20:       { description => $what, when => $when };
21:    return;
22: }
23: 

Test::Class example code 3/3

lib/Calendar/MadMongers/Object.pm

24: sub select {
25:    my ($self, $year, $month) = @_;
26:    if ($month < 1 || $month > 12) {
27:       croak 'Invalid month';
28:    }
29:    return grep {$_->{when}->year == $year &&
30:                 $_->{when}->month == $month}
31:       $self->all;
32: }
33: 
34: 1;

Test::Class example test

t/classes.t

 1: #!/usr/bin/perl -w
 2: use strict;
 3: 
 4: # equivalent to '-I t/lib'
 5: use File::Basename;
 6: use File::Spec;
 7: use lib File::Spec->catdir(
 8:            dirname(__FILE__), 'lib');
 9: 
10: use Calendar::MadMongers::TestObject;
11: Calendar::MadMongers::TestObject->runtests;

Test::Class example test class

t/lib/Calendar/MadMongers/TestObject.pm

 1: package Calendar::MadMongers::TestObject;
 2: 
 3: use base 'Test::Class';
 4: use Test::More;
 5: use Test::Exception;
 6: 
 7: use Calendar::MadMongers::Object;
 8: use DateTime;
 9: 
10: sub every_method : Test(setup) {
11:    shift->{self}
12:        = Calendar::MadMongers::Object->new;
13: }

Test::Class example test class

t/lib/Calendar/MadMongers/TestObject.pm

15: sub empty : Test(1) {
16:    my $self = shift->{self};
17:    is(scalar $self->all, 0, 'empty at start');
18: }
19: 
20: sub add : Test(1) {
21:    my $self = shift->{self};
22:    my $now = DateTime->now();
23:    $self->add('a test', $now);
24:    is_deeply([$self->all],
25:              [{description => 'a test',
26:                when => $now}], 'add');
27: }

Test::Class example test class

t/lib/Calendar/MadMongers/TestObject.pm

29: sub add_error : Test(1) {
30:    my $self = shift->{self};
31:    dies_ok { $self->add(DateTime->now(),
32:                         'a test');  };
33: }
34: 
35: sub select : Test(2) {
36:    my $self = shift->{self};
37:    $self->add('a', DateTime->new(year => 2008,
38:                                  month => 4));
39:    is(+ $self->select(2008, 4), 1, 'April');
40:    is(+ $self->select(2008, 5), 0, 'May');
41: }

Test::Class output

perl -Ilib t/classes.t

1..5
ok 1 - add
ok 2 - add error
ok 3 - empty at start
ok 4 - April
ok 5 - May