Move some regrtest or test.support features into unittest?
Hi, tl; dr How can we extend unittest module to plug new checks before/after running tests? The CPython project has a big test suite in the Lib/test/ directory. While all tests are written with the unittest module and the unittest.TestCase class, tests are not run directly by unittest, but run by "regrtest" (for "regression test") which is a test runner doing more checks (and more). I would like to see if and how we can integrate/move some regrtest features into the unittest module. Example of regrtest features: * skip a test if it allocates too much memory, command line argument to specify how many memory a test is allowed to allocate (ex: --memlimit=2G for 2 GB of memory) * concept of "resource" like "network" (connect to external network servers, to the Internet), "cpu" (CPU intensive tests), etc. Tests are skipped by default and enabled by the -u command line option (ex: "-u cpu). * track memory leaks: check the reference counter, check the number of allocated memory blocks, check the number of open file descriptors. * detect if the test spawned a thread or process and the thread/process is still running at the test exit * --timeout: watchdog killing the test if the run time exceed the timeout in seconds (use faulthandler.dump_traceback_later) * multiprocessing: run tests in subprocesses, in parallel * redirect stdout/stderr to pipes (StringIO objects), ignore them on success, or dump them to stdout/stderr on test failure * --slowest: top 10 of the slowest tests * --randomize: randomize test order * --match, --matchfile, -x: filter tests * --forever: run the test in a loop until it fails (or is interrupted by CTRL+c) * --list-tests / --list-cases: list test files / test methods * --fail-env-changed: mark tests as failed if a test altered the environment * detect if a "global variable" of the standard library was modified but not restored by the test: resources = ('sys.argv', 'cwd', 'sys.stdin', 'sys.stdout', 'sys.stderr', 'os.environ', 'sys.path', 'sys.path_hooks', '__import__', 'warnings.filters', 'asyncore.socket_map', 'logging._handlers', 'logging._handlerList', 'sys.gettrace', 'sys.warnoptions', 'multiprocessing.process._dangling', 'threading._dangling', 'sysconfig._CONFIG_VARS', 'sysconfig._INSTALL_SCHEMES', 'files', 'locale', 'warnings.showwarning', 'shutil_archive_formats', 'shutil_unpack_formats', ) * test.bisect: bisection to identify the failing method, used to track memory leaks or identify a test leaking a resource (ex: create a file but don't remove it) * ... : regrtest has many many features My question is also connected to test.support (Lib/test/support/__init__.py): a big module containing a lot of helper functions to write tests and to detect bugs in tests. For example, @reap_children decorator emits a warnig if the test leaks a child process (and reads its exit status to prevent zombie process). I started to duplicate code in many files of Lib/test/test_*.py to check if tests "leak running threads" ("dangling threads"). Example from Lib/test/test_theading.py: class BaseTestCase(unittest.TestCase): def setUp(self): self._threads = test.support.threading_setup() def tearDown(self): test.support.threading_cleanup(*self._threads) test.support.reap_children() I would like to get this test "for free" directly from the regular unittest.TestCase class, but I don't know how to extend the unittest module for that? Victor
On Wed, 13 Sep 2017 15:42:56 +0200 Victor Stinner <victor.stinner@gmail.com> wrote:
That would be suitable for a plugin if unittest had a plugin architecture, but not as a core functionality IMO.
Good as a core functionality IMO.
* track memory leaks: check the reference counter, check the number of allocated memory blocks, check the number of open file descriptors.
Good for a plugin IMO.
* detect if the test spawned a thread or process and the thread/process is still running at the test exit
Good for a plugin IMO.
* --timeout: watchdog killing the test if the run time exceed the timeout in seconds (use faulthandler.dump_traceback_later)
Good for a plugin IMO.
* multiprocessing: run tests in subprocesses, in parallel
Good as a core functionality IMO.
* redirect stdout/stderr to pipes (StringIO objects), ignore them on success, or dump them to stdout/stderr on test failure
Good for a plugin IMO.
* --slowest: top 10 of the slowest tests
Good for a plugin IMO.
* --randomize: randomize test order
Will be tricky to mix with setupClass.
* --match, --matchfile, -x: filter tests
Good as a core functionality IMO.
* --forever: run the test in a loop until it fails (or is interrupted by CTRL+c)
Good for a plugin IMO.
* --list-tests / --list-cases: list test files / test methods
Good as a core functionality IMO.
* --fail-env-changed: mark tests as failed if a test altered the environment
Good for a plugin IMO.
* detect if a "global variable" of the standard library was modified but not restored by the test:
Good for a plugin IMO.
Good as a core functionality IMO.
Instead of creating tons of distinct base TestCase classes, you can just provide helper functions / methods calling addCleanup(). Regards Antoine.
Antoine:
My hidden question is more why unittest doesn't already have a plugin system, whereas there are tons on projects based on unittest extending its features like pytest, nose, testtools, etc. Longer list: https://wiki.python.org/moin/PythonTestingToolsTaxonomy pytest has a plugin system. I didn't use it, but I know that there is a pytest-faulthandler to enable my faulthandler module, and this extension seems tp be used in the wild: https://pypi.python.org/pypi/pytest-faulthandler I found a list of pytest extensions: https://docs.pytest.org/en/latest/plugins.html It seems like the lack of plugin architecture prevented enhancements. Example: "unittest: display time used by each test case" https://bugs.python.org/issue4080 Michael Foord's comment (July, 2010): "Even if it is added to the core it should be in the form of an extension (plugin) so please don't update the patch until this is in place." Victor
13.09.17 16:42, Victor Stinner пише:
Instead of just making checks before running some tests it would be worth to limit the memory usage hard (by setting ulimit or analogs on other plathforms). The purpose of this option is preventing swapping which makes tests just hanging for hours.
The problem is what include in "all". The set of resources is application specific. Some of resources used in CPython tests make sense only for single test.
* --timeout: watchdog killing the test if the run time exceed the timeout in seconds (use faulthandler.dump_traceback_later)
This feature looks functionally similar to limiting memory usage.
* --match, --matchfile, -x: filter tests
The discovery feature of unittest looks similar.
* ... : regrtest has many many features
Many of them contain a bunch of engineering tricks and evolve quickly. Regrtest now is not a regrtest two years ago, and I'm sure that two years later it will differ too much from the current. Unittest should be more stable.
On Wed, 13 Sep 2017 15:42:56 +0200 Victor Stinner <victor.stinner@gmail.com> wrote:
That would be suitable for a plugin if unittest had a plugin architecture, but not as a core functionality IMO.
Good as a core functionality IMO.
* track memory leaks: check the reference counter, check the number of allocated memory blocks, check the number of open file descriptors.
Good for a plugin IMO.
* detect if the test spawned a thread or process and the thread/process is still running at the test exit
Good for a plugin IMO.
* --timeout: watchdog killing the test if the run time exceed the timeout in seconds (use faulthandler.dump_traceback_later)
Good for a plugin IMO.
* multiprocessing: run tests in subprocesses, in parallel
Good as a core functionality IMO.
* redirect stdout/stderr to pipes (StringIO objects), ignore them on success, or dump them to stdout/stderr on test failure
Good for a plugin IMO.
* --slowest: top 10 of the slowest tests
Good for a plugin IMO.
* --randomize: randomize test order
Will be tricky to mix with setupClass.
* --match, --matchfile, -x: filter tests
Good as a core functionality IMO.
* --forever: run the test in a loop until it fails (or is interrupted by CTRL+c)
Good for a plugin IMO.
* --list-tests / --list-cases: list test files / test methods
Good as a core functionality IMO.
* --fail-env-changed: mark tests as failed if a test altered the environment
Good for a plugin IMO.
* detect if a "global variable" of the standard library was modified but not restored by the test:
Good for a plugin IMO.
Good as a core functionality IMO.
Instead of creating tons of distinct base TestCase classes, you can just provide helper functions / methods calling addCleanup(). Regards Antoine.
Antoine:
My hidden question is more why unittest doesn't already have a plugin system, whereas there are tons on projects based on unittest extending its features like pytest, nose, testtools, etc. Longer list: https://wiki.python.org/moin/PythonTestingToolsTaxonomy pytest has a plugin system. I didn't use it, but I know that there is a pytest-faulthandler to enable my faulthandler module, and this extension seems tp be used in the wild: https://pypi.python.org/pypi/pytest-faulthandler I found a list of pytest extensions: https://docs.pytest.org/en/latest/plugins.html It seems like the lack of plugin architecture prevented enhancements. Example: "unittest: display time used by each test case" https://bugs.python.org/issue4080 Michael Foord's comment (July, 2010): "Even if it is added to the core it should be in the form of an extension (plugin) so please don't update the patch until this is in place." Victor
13.09.17 16:42, Victor Stinner пише:
Instead of just making checks before running some tests it would be worth to limit the memory usage hard (by setting ulimit or analogs on other plathforms). The purpose of this option is preventing swapping which makes tests just hanging for hours.
The problem is what include in "all". The set of resources is application specific. Some of resources used in CPython tests make sense only for single test.
* --timeout: watchdog killing the test if the run time exceed the timeout in seconds (use faulthandler.dump_traceback_later)
This feature looks functionally similar to limiting memory usage.
* --match, --matchfile, -x: filter tests
The discovery feature of unittest looks similar.
* ... : regrtest has many many features
Many of them contain a bunch of engineering tricks and evolve quickly. Regrtest now is not a regrtest two years ago, and I'm sure that two years later it will differ too much from the current. Unittest should be more stable.
participants (4)
-
Antoine Pitrou -
Michel Desmoulin -
Serhiy Storchaka -
Victor Stinner