Mailman 3 Move some regrtest or test.support features into unittest? - Python-ideas

13 Sep 2017

      Hi,

tl; dr How can we extend unittest module to plug new checks
before/after running tests?

The CPython project has a big test suite in the Lib/test/ directory.
While all tests are written with the unittest module and the
unittest.TestCase class, tests are not run directly by unittest, but
run by "regrtest" (for "regression test") which is a test runner doing
more checks (and more).

I would like to see if and how we can integrate/move some regrtest
features into the unittest module. Example of regrtest features:

* skip a test if it allocates too much memory, command line argument
to specify how many memory a test is allowed to allocate (ex:
--memlimit=2G for 2 GB of memory)

* concept of "resource" like "network" (connect to external network
servers, to the Internet), "cpu" (CPU intensive tests), etc. Tests are
skipped by default and enabled by the -u command line option (ex: "-u
cpu).

* track memory leaks: check the reference counter, check the number of
allocated memory blocks, check the number of open file descriptors.

* detect if the test spawned a thread or process and the
thread/process is still running at the test exit

* --timeout: watchdog killing the test if the run time exceed the
timeout in seconds (use faulthandler.dump_traceback_later)

* multiprocessing: run tests in subprocesses, in parallel

* redirect stdout/stderr to pipes (StringIO objects), ignore them on
success, or dump them to stdout/stderr on test failure

* --slowest: top 10 of the slowest tests

* --randomize: randomize test order

* --match, --matchfile, -x: filter tests

* --forever: run the test in a loop until it fails (or is interrupted by CTRL+c)

* --list-tests / --list-cases: list test files / test methods

* --fail-env-changed: mark tests as failed if a test altered the environment

* detect if a "global variable" of the standard library was modified
but not restored by the test:

    resources = ('sys.argv', 'cwd', 'sys.stdin', 'sys.stdout', 'sys.stderr',
                 'os.environ', 'sys.path', 'sys.path_hooks', '__import__',
                 'warnings.filters', 'asyncore.socket_map',
                 'logging._handlers', 'logging._handlerList', 'sys.gettrace',
                 'sys.warnoptions',
                 'multiprocessing.process._dangling', 'threading._dangling',
                 'sysconfig._CONFIG_VARS', 'sysconfig._INSTALL_SCHEMES',
                 'files', 'locale', 'warnings.showwarning',
                 'shutil_archive_formats', 'shutil_unpack_formats',
                )

* test.bisect: bisection to identify the failing method, used to track
memory leaks or identify a test leaking a resource (ex: create a file
but don't remove it)

* ... : regrtest has many many features

My question is also connected to test.support
(Lib/test/support/__init__.py): a big module containing a lot of
helper functions to write tests and to detect bugs in tests. For
example, @reap_children decorator emits a warnig if the test leaks a
child process (and reads its exit status to prevent zombie process).

I started to duplicate code in many files of Lib/test/test_*.py to
check if tests "leak running threads" ("dangling threads"). Example
from Lib/test/test_theading.py:

class BaseTestCase(unittest.TestCase):
    def setUp(self):
        self._threads = test.support.threading_setup()

    def tearDown(self):
        test.support.threading_cleanup(*self._threads)
        test.support.reap_children()

I would like to get this test "for free" directly from the regular
unittest.TestCase class, but I don't know how to extend the unittest
module for that?

Victor

Move some regrtest or test.support features into unittest?

Victor Stinner

Antoine Pitrou

Michel Desmoulin

Victor Stinner

Antoine Pitrou

Serhiy Storchaka

Victor Stinner

tags

participants (4)