Skip to content

Test Suite

Jump to bottom
Chris Dunlap edited this page Mar 8, 2023 · 10 revisions

The MUNGE test suite uses Sharness for higher-level tests of the executables, and libtap for lower-level tests of the C internals. Sharness tests reside in the tests/ directory, whereas libtap tests are located in the directories under src/.

Both Sharness and libtap are TAP Producers, outputting test results according to the Test Anything Protocol. This TAP output can be fed into a TAP Consumer for further processing.

Running the tests

  • Run the tests by using the check target of the Makefile. Specify options by appending shell variables to the make command.
    make check root=/tmp/munge-test-$$ verbose=t

  • Run the tests by using the prove command. See the prove(1) manpage or Perl docs for details.
    prove tests/*.t

  • Run an individual test directly. Specify options by prepending shell variables to the command, or appending command-line options.
    tests/0010-basic.t --root=/tmp/munge-test-$$ --verbose

A note for package maintainers

Package maintainers adding the test suite to their build will probably want to use a command similar to the following:
make check root=/tmp/munge-test-$$ verbose=t VERBOSE=t

  • Specifying the -j / --jobs= option to make will run the test suite faster but interleave its output.

  • Specifying root is commonly needed to overcome permission problems with the build environment.

  • Appending "$$" for shell pid expansion may need to be escaped (e.g., using "$$$$" in makefiles).

  • verbose is needed to capture stdout and stderr from the test commands. It must be set to "t".

  • VERBOSE is needed to display the output from failed tests.

  • verbose and VERBOSE are different variables that should both be specified. They are essential for troubleshooting failures.

  • TEST_LONG is not recommended here.

Variables of note

  • chain_lint is a sharness variable for checking whether each test properly "&&-chains" all commands in order to ensure a failure of any command in the chain will cause the test to fail. If chain_lint=t, sharness will perform this check in addition to running the tests. This is primarily intended for development.

  • debug is a sharness variable primarily intended for development. If debug=t, execution of test_debug() commands will be performed, and removal of "trash" directories (for storing all temporary data during a test) will be prevented in order to allow their contents to be inspected afterwards.

  • root is a sharness variable for specifying a directory to be prepended to the "trash" directories. A trash directory is created for each test script to store temporary data, and that directory becomes the HOME directory for the given test. If this variable is not specified, these directories are created in the current working directory. Each directory is removed after its corresponding test script finishes unless a test fails or debug=t is specified.

  • TEST_LONG is a sharness variable for running tests marked as being "expensive". If this variable is not set, sharness tests requiring sudo (expensive in privilege) or valgrind (expensive in memory and time) will be skipped. This is primarily intended for development.

  • verbose is a sharness variable for making test output more verbose. By default, standard output and standard error streams are discarded, and only a result of "ok" or "not ok" is reported. But if verbose=t, the test commands being run and their resulting output will be logged to the corresponding .log files as well as test-suite.log.

  • VERBOSE is an automake test harness variable. If set, the output from failed tests collected in test-suite.log will be displayed after all tests have completed.

Tests utilizing sudo

  • Sharness tests checking the behavior of executables running as root use sudo to elevate privileges. They require sudo-1.7.0 or later. These tests will be skipped unless the TEST_LONG variable is set and sudo is configured to run commands without prompting for a password (see the NOPASSWD: tag in sudoers(5)).

Tests utilizing valgrind

  • Sharness tests using valgrind to check for memory errors require valgrind-3.9.0 (2013-10-31) or later. These tests will be skipped unless the TEST_LONG variable is set and valgrind is installed.

Troubleshooting

  • If tests fail, try running the test suite with verbose=t and examining tests/test-suite.log afterwards for more information. Search for lines beginning with FAIL:.

  • If your current working directory (or one of its parent directories) has overly-permissive permissions, you may see a multitude of errors noting various files and directories are insecure. Try moving the trash directories to another filesystem such as /tmp (which should have its sticky-bit set) by running the test suite with root=/tmp/munge-test-$$.

  • If your current working directory is inside an NFS mount, you may encounter race conditions with NFS locks; for example, rm: cannot remove '/path/to/some/.nfs000000000123456789abcdef': Device or resource busy.* Try moving the trash directories to a non-NFS local filesystem by running the test suite with root=/tmp/munge-test-$$.

Clone this wiki locally