cpantestfaq - Frequently Asked Questions Aboout Testing for CPAN $Date: 2002/08/20 12:56:18 $ $Revision: 1.1 $
This FAQ addresses testing CPAN modules and reporting your test results. It points out how simple it is to begin testing, and tries to help you get started. It also offers tips for more advanced or automated testing with such tools as CPAN and CPANPLUS.
The first, and most important thing, to understand about testing for CPAN is that it is very, very easy to get started. There's always more that you can do, but there's very little that you must do in order to get started and be helpful. If you're looking for a way to contribute something to the Perl community, but don't know where to begin, this just might be the place.
Yes, you should.
Install them by hand with
perl Makefile.PL;
make;
make test;
Notice the results shown after make test. Run cpantest to submit the results from your platform. If anything other than PASS, add notes saying what failed and, if you can, why. If you can fix it, fix it and include the fix with your report if it's small. If it's not small, ask the author how to submit it.
Always send a copy of your message to the author. Not every one of them monitors the CPAN-Testers mailing list as closely as you might like.
If you have the CPAN module installed and configured you can do perl -MPCAN -e 'test Some::Module'
If you have CPANPLUS installed, you can run t Some::Module in the default shell. Or you can write your own interface.
CPANPLUS is an excellent tool for interacting with the Comprehensive Perl Archive Network. If you're interested in automating your tests, and don't mind a little setup work and a touch of uncertainty. It has not reached version 1.0 as of this writing (2002-07-09) and the authors do not promise backward-compatibility until version 1.0 is released. Many of the testers you see posting lots of results are not doing all that work by hand.
Use the cpantest script, available from http://testers.cpan.org/. If you just run it as 'cpantest' it will give you usage notes. Generally, you should run it from the directory in which the module unpacked. Unless you ask it not to, cpantest will prompt you for the name and version of the module you're testing, using the name of the current directory as the default answer.
cpantest generates the subject line of the message you'll send, and nearly every automated part of the process is based on that subject line. It's very important that it follow the conventions that cpantest encourages you to use.
The message is sent by deafult to the cpan-testers mailing list. You should always send a copy to the module's author, as well, no matter what result you report.
When reporting results, you choose from one of four grades: PASS, FAIL, UNKNOWN or NA. They are defined as follows:
pass
The build went smoothly and every test included with the package either passed or was skipped.
fail
Some or all tests failed. If you get a fatal error during the build process (perl Makefile.PL or make) you should grade the distribution as ``fail'', too. The only failures that should not be reported with fail are those resulting from unfulfilled dependencies. Some modules require that other modules be installed in order to function. Some modules require that you have certain software on your system - the DBD modules are the clearest example of this. If DBD::MySQL fails because you don't have MySQL installed, that's not the module's fault.
Actually, that depends on how it fails. If it complains during perl Makefile.PL that a dependency is unfulfilled, then the module distribution is behaving properly. You should not report any result. Your system does not meet the requirements that the module distribution laid out, and is therefore not a valid testing platform right now. You may choose to satisfy the dependency by installing whatever the module asked for. If you do that, start the installation over.
If perl Makefile.PL does not complain, but something fails during make test, then you should report fail. If you can tell that it's simply an unfulfilled dependency that was not noticed during perl Makefile.PL, say so in your error report so that the author can adjust the makefile to check next time.
unknown
No tests were supplied in the module distribution. Modules are supposed to be distributed with tests - it is an error (but not a failure) if there are none. This grade is not seen very often, so take a minute to include a note in your result asking the author to please include some tests with the next version. Just in case the author doesn't know what unknown means.
na
The module was specifically designed not to function on the platform you're running. Typically, these are packages that take advantage of a particular operating system, such as those in the WIN32 or Mac namespaces. Unless there's a good reason for it, all modules should work on all platforms.
Do what you'd want someone to do for you. Report the problems with as much detail as the author will need in order to fix - or at least understand - the problem. Most important is to send the output of the command were the failure occurred. Second most important is to pay attention to your e-mail and do your best to reply intelligently if the author has follow-up questions.
This document is copyright 2002, Brian King. Permission is granted to re-distribute this FAQ in its entirety. That's the whole point.
Please send updates, questions or criticism to Brian King <mrbbking@mac.com>, or /msg mrbbking at http://www.perlmonks.org/