testcode.py

testcode.py - a command-line interface to testcode.

Synopsis

testcode.py [options] [action1 [action2…]]

Description

Run a set of actions on a set of tests.

Requires two configuration files, jobconfig and userconfig. See testcode documentation for further details.

testcode.py provides a command-line interface to testcode, a simple framework for comparing output from (principally numeric) programs to previous output to reveal regression errors or miscompilation.

Actions

‘’run’’ is th default action.

compare

compare set of test outputs from a previous testcode run against the benchmark outputs.

diff

diff set of test outputs from a previous testcode run against the benchmark outputs.

make-benchmarks

create a new set of benchmarks and update the userconfig file with the new benchmark id. Also runs the ‘run’ action unless the ‘compare’ action is also given.

recheck

compare set of test outputs from a previous testcode run against benchmark outputs and rerun any failed tests.

run

run a set of tests and compare against the benchmark outputs.

tidy

Remove files from previous testcode runs from the test directories.

Options

-h, --help	show this help message and exit
-b BENCHMARK, --benchmark=BENCHMARK
	Set the file ID of the benchmark files. If BENCHMARK is in the format t:ID, then the test files with the corresponding ID are used. This allows two sets of tests to be compared. Default: specified in the [user] section of the userconfig file.
-c CATEGORY, --category=CATEGORY
	Select the category/group of tests. Can be specified multiple times. Wildcards or parent directories can be used to select multiple directories by their path. Default: use the _default_ category if run is an action unless make-benchmarks is an action. All other cases use the _all_ category by default. The _default_ category contains all tests unless otherwise set in the jobconfig file.
-e EXECUTABLE, --executable=EXECUTABLE
	Set the executable(s) to be used to run the tests. Can be a path or name of an option in the userconfig file, in which case all test programs are set to use that value, or in the format program_name=value, which affects only the specified program. Only relevant to the run action. Default: exe variable set for each program listed in the userconfig file.
-i, --insert	Insert the new benchmark into the existing list of benchmarks in userconfig rather than overwriting it. Only relevant to the make-benchmarks action. Default: False.
--jobconfig=JOBCONFIG
	Set path to the job configuration file. Default: jobconfig.
--job-option=JOB_OPTION
	Override/add setting to jobconfig. Takes three arguments. Format: section_name option_name value. Default: none.
--older-than=OLDER_THAN
	Set the age (in days) of files to remove. Only relevant to the tidy action. Default: 14 days.
-p NPROCS, --processors=NPROCS
	Set the number of processors to run each test on. Only relevant to the run action. Default: run tests as serial jobs.
-q, --quiet	Print only minimal output. Default: False.
-s QUEUE_SYSTEM, --submit=QUEUE_SYSTEM
	Submit tests to a queueing system of the specified type. Only PBS system is currently implemented. Only relevant to the run action. Default: none.
-t TEST_ID, --test-id=TEST_ID
	Set the file ID of the test outputs. If TEST_ID is in the format b:ID, then the benchmark files with the corresponding ID are used. This allows two sets of benchmarks to be compared. Default: unique filename based upon date if running tests and most recent test_id if comparing tests.
--total-processors=TOT_NPROCS
	Set the total number of processors to use to run as many tests as possible at the same time. Relevant only to the run option. Default: run all tests concurrently run if –submit is used; run tests sequentially otherwise.
--userconfig=USERCONFIG
	Set path to the user configuration file. Default: userconfig.
--user-option=USER_OPTION
	Override/add setting to userconfig. Takes three arguments. Format: section_name option_name value. Default: none.
-v, --verbose	Increase verbosity of output. Can be specified up to two times. The default behaviour is to print out the test and its status. (See the –quiet option to suppress even this.) Specify -v or –verbose once to show (if relevant) which data values caused warnings or failures. Specify -v or –verbose twice to see all (external) commands run and all data extracted from running the tests. Using the maximum verbosity level is highly recommended for debugging.

Exit status

1 if one or more tests fail (run and compare actions only) and 0 otherwise.

License

Modified BSD License. See LICENSE in the source code for more details.

Bugs

Contact James Spencer (j.spencer@imperial.ac.uk) regarding bug reports, suggestions for improvements or code contributions.