Hi,

tl; dr How can we extend unittest module to plug new checks before/after running tests?

The CPython project has a big test suite in the Lib/test/ directory. While all tests are written with the unittest module and the unittest.TestCase class, tests are not run directly by unittest, but run by "regrtest" (for "regression test") which is a test runner doing more checks (and more).

I would like to see if and how we can integrate/move some regrtest features into the unittest module. Example of regrtest features:

• skip a test if it allocates too much memory, command line argument to specify how many memory a test is allowed to allocate (ex: --memlimit=2G for 2 GB of memory)

• concept of "resource" like "network" (connect to external network servers, to the Internet), "cpu" (CPU intensive tests), etc. Tests are skipped by default and enabled by the -u command line option (ex: "-u cpu).

• track memory leaks: check the reference counter, check the number of allocated memory blocks, check the number of open file descriptors.

• detect if the test spawned a thread or process and the thread/process is still running at the test exit

• --timeout: watchdog killing the test if the run time exceed the timeout in seconds (use faulthandler.dump_traceback_later)

• multiprocessing: run tests in subprocesses, in parallel

• redirect stdout/stderr to pipes (StringIO objects), ignore them on success, or dump them to stdout/stderr on test failure

• --slowest: top 10 of the slowest tests

• --randomize: randomize test order

• --match, --matchfile, -x: filter tests

• --forever: run the test in a loop until it fails (or is interrupted by CTRL+c)

• --list-tests / --list-cases: list test files / test methods

• --fail-env-changed: mark tests as failed if a test altered the environment

• detect if a "global variable" of the standard library was modified but not restored by the test:

  resources = ('sys.argv', 'cwd', 'sys.stdin', 'sys.stdout', 'sys.stderr',

             'os.environ', 'sys.path', 'sys.path_hooks', '__import__',
             'warnings.filters', 'asyncore.socket_map',
             'logging._handlers', 'logging._handlerList', 'sys.gettrace',
             'sys.warnoptions',
             'multiprocessing.process._dangling', 'threading._dangling',
             'sysconfig._CONFIG_VARS', 'sysconfig._INSTALL_SCHEMES',
             'files', 'locale', 'warnings.showwarning',
             'shutil_archive_formats', 'shutil_unpack_formats',
            )
  

• test.bisect: bisection to identify the failing method, used to track memory leaks or identify a test leaking a resource (ex: create a file but don't remove it)

• ... : regrtest has many many features

My question is also connected to test.support (Lib/test/support/__init__.py): a big module containing a lot of helper functions to write tests and to detect bugs in tests. For example, @reap_children decorator emits a warnig if the test leaks a child process (and reads its exit status to prevent zombie process).

I started to duplicate code in many files of Lib/test/test_*.py to check if tests "leak running threads" ("dangling threads"). Example from Lib/test/test_theading.py:

class BaseTestCase(unittest.TestCase): def setUp(self): self._threads = test.support.threading_setup()

def tearDown(self):
    test.support.threading_cleanup(*self._threads)
    test.support.reap_children()

I would like to get this test "for free" directly from the regular unittest.TestCase class, but I don't know how to extend the unittest module for that?

Victor