[Python-ideas] Move some regrtest or test.support features into unittest?
victor.stinner at gmail.com
Wed Sep 13 09:42:56 EDT 2017
tl; dr How can we extend unittest module to plug new checks
before/after running tests?
The CPython project has a big test suite in the Lib/test/ directory.
While all tests are written with the unittest module and the
unittest.TestCase class, tests are not run directly by unittest, but
run by "regrtest" (for "regression test") which is a test runner doing
more checks (and more).
I would like to see if and how we can integrate/move some regrtest
features into the unittest module. Example of regrtest features:
* skip a test if it allocates too much memory, command line argument
to specify how many memory a test is allowed to allocate (ex:
--memlimit=2G for 2 GB of memory)
* concept of "resource" like "network" (connect to external network
servers, to the Internet), "cpu" (CPU intensive tests), etc. Tests are
skipped by default and enabled by the -u command line option (ex: "-u
* track memory leaks: check the reference counter, check the number of
allocated memory blocks, check the number of open file descriptors.
* detect if the test spawned a thread or process and the
thread/process is still running at the test exit
* --timeout: watchdog killing the test if the run time exceed the
timeout in seconds (use faulthandler.dump_traceback_later)
* multiprocessing: run tests in subprocesses, in parallel
* redirect stdout/stderr to pipes (StringIO objects), ignore them on
success, or dump them to stdout/stderr on test failure
* --slowest: top 10 of the slowest tests
* --randomize: randomize test order
* --match, --matchfile, -x: filter tests
* --forever: run the test in a loop until it fails (or is interrupted by CTRL+c)
* --list-tests / --list-cases: list test files / test methods
* --fail-env-changed: mark tests as failed if a test altered the environment
* detect if a "global variable" of the standard library was modified
but not restored by the test:
resources = ('sys.argv', 'cwd', 'sys.stdin', 'sys.stdout', 'sys.stderr',
'os.environ', 'sys.path', 'sys.path_hooks', '__import__',
'logging._handlers', 'logging._handlerList', 'sys.gettrace',
'files', 'locale', 'warnings.showwarning',
* test.bisect: bisection to identify the failing method, used to track
memory leaks or identify a test leaking a resource (ex: create a file
but don't remove it)
* ... : regrtest has many many features
My question is also connected to test.support
(Lib/test/support/__init__.py): a big module containing a lot of
helper functions to write tests and to detect bugs in tests. For
example, @reap_children decorator emits a warnig if the test leaks a
child process (and reads its exit status to prevent zombie process).
I started to duplicate code in many files of Lib/test/test_*.py to
check if tests "leak running threads" ("dangling threads"). Example
self._threads = test.support.threading_setup()
I would like to get this test "for free" directly from the regular
unittest.TestCase class, but I don't know how to extend the unittest
module for that?
More information about the Python-ideas