Hi,
tl; dr How can we extend unittest module to plug new checks before/after running tests?
The CPython project has a big test suite in the Lib/test/ directory. While all tests are written with the unittest module and the unittest.TestCase class, tests are not run directly by unittest, but run by "regrtest" (for "regression test") which is a test runner doing more checks (and more).
I would like to see if and how we can integrate/move some regrtest features into the unittest module. Example of regrtest features:
skip a test if it allocates too much memory, command line argument to specify how many memory a test is allowed to allocate (ex: --memlimit=2G for 2 GB of memory)
concept of "resource" like "network" (connect to external network servers, to the Internet), "cpu" (CPU intensive tests), etc. Tests are skipped by default and enabled by the -u command line option (ex: "-u cpu).
track memory leaks: check the reference counter, check the number of allocated memory blocks, check the number of open file descriptors.
detect if the test spawned a thread or process and the thread/process is still running at the test exit
--timeout: watchdog killing the test if the run time exceed the timeout in seconds (use faulthandler.dump_traceback_later)
multiprocessing: run tests in subprocesses, in parallel
redirect stdout/stderr to pipes (StringIO objects), ignore them on success, or dump them to stdout/stderr on test failure
--slowest: top 10 of the slowest tests
--randomize: randomize test order
--match, --matchfile, -x: filter tests
--forever: run the test in a loop until it fails (or is interrupted by CTRL+c)
--list-tests / --list-cases: list test files / test methods
--fail-env-changed: mark tests as failed if a test altered the environment
detect if a "global variable" of the standard library was modified but not restored by the test:
resources = ('sys.argv', 'cwd', 'sys.stdin', 'sys.stdout', 'sys.stderr',
'os.environ', 'sys.path', 'sys.path_hooks', '__import__',
'warnings.filters', 'asyncore.socket_map',
'logging._handlers', 'logging._handlerList', 'sys.gettrace',
'sys.warnoptions',
'multiprocessing.process._dangling', 'threading._dangling',
'sysconfig._CONFIG_VARS', 'sysconfig._INSTALL_SCHEMES',
'files', 'locale', 'warnings.showwarning',
'shutil_archive_formats', 'shutil_unpack_formats',
)
test.bisect: bisection to identify the failing method, used to track memory leaks or identify a test leaking a resource (ex: create a file but don't remove it)
... : regrtest has many many features
My question is also connected to test.support (Lib/test/support/__init__.py): a big module containing a lot of helper functions to write tests and to detect bugs in tests. For example, @reap_children decorator emits a warnig if the test leaks a child process (and reads its exit status to prevent zombie process).
I started to duplicate code in many files of Lib/test/test_*.py to check if tests "leak running threads" ("dangling threads"). Example from Lib/test/test_theading.py:
class BaseTestCase(unittest.TestCase): def setUp(self): self._threads = test.support.threading_setup()
def tearDown(self):
test.support.threading_cleanup(*self._threads)
test.support.reap_children()
I would like to get this test "for free" directly from the regular unittest.TestCase class, but I don't know how to extend the unittest module for that?
Victor
On Wed, 13 Sep 2017 15:42:56 +0200 Victor Stinner victor.stinner@gmail.com wrote:
I would like to see if and how we can integrate/move some regrtest features into the unittest module. Example of regrtest features:
That would be suitable for a plugin if unittest had a plugin architecture, but not as a core functionality IMO.
Good as a core functionality IMO.
Good for a plugin IMO.
Good for a plugin IMO.
Good for a plugin IMO.
Good as a core functionality IMO.
Good for a plugin IMO.
Good for a plugin IMO.
Will be tricky to mix with setupClass.
Good as a core functionality IMO.
Good for a plugin IMO.
Good as a core functionality IMO.
Good for a plugin IMO.
Good for a plugin IMO.
Good as a core functionality IMO.
I started to duplicate code in many files of Lib/test/test_*.py to check if tests "leak running threads" ("dangling threads"). Example from Lib/test/test_theading.py:
class BaseTestCase(unittest.TestCase): def setUp(self): self._threads = test.support.threading_setup()
def tearDown(self):
test.support.threading_cleanup(*self._threads)
test.support.reap_children()
I would like to get this test "for free" directly from the regular unittest.TestCase class, but I don't know how to extend the unittest module for that?
Instead of creating tons of distinct base TestCase classes, you can just provide helper functions / methods calling addCleanup().
Regards
Antoine.
Can you elaborate on _why_ you think something is good for core/a plugin ? Cause right now it's impossible to know what's the logic behind.
Le 17/09/2017 à 22:31, Antoine Pitrou a écrit :
On Wed, 13 Sep 2017 15:42:56 +0200 Victor Stinner victor.stinner@gmail.com wrote:
I would like to see if and how we can integrate/move some regrtest features into the unittest module. Example of regrtest features:
That would be suitable for a plugin if unittest had a plugin architecture, but not as a core functionality IMO.
Good as a core functionality IMO.
Good for a plugin IMO.
Good for a plugin IMO.
Good for a plugin IMO.
Good as a core functionality IMO.
Good for a plugin IMO.
Good for a plugin IMO.
Will be tricky to mix with setupClass.
Good as a core functionality IMO.
Good for a plugin IMO.
Good as a core functionality IMO.
Good for a plugin IMO.
Good for a plugin IMO.
Good as a core functionality IMO.
I started to duplicate code in many files of Lib/test/test_*.py to check if tests "leak running threads" ("dangling threads"). Example from Lib/test/test_theading.py:
class BaseTestCase(unittest.TestCase): def setUp(self): self._threads = test.support.threading_setup()
def tearDown(self):
test.support.threading_cleanup(*self._threads)
test.support.reap_children()
I would like to get this test "for free" directly from the regular unittest.TestCase class, but I don't know how to extend the unittest module for that?
Instead of creating tons of distinct base TestCase classes, you can just provide helper functions / methods calling addCleanup().
Regards
Antoine.
Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/
Antoine:
That would be suitable for a plugin if unittest had a plugin architecture, but not as a core functionality IMO.
My hidden question is more why unittest doesn't already have a plugin system, whereas there are tons on projects based on unittest extending its features like pytest, nose, testtools, etc. Longer list: https://wiki.python.org/moin/PythonTestingToolsTaxonomy
pytest has a plugin system. I didn't use it, but I know that there is a pytest-faulthandler to enable my faulthandler module, and this extension seems tp be used in the wild: https://pypi.python.org/pypi/pytest-faulthandler
I found a list of pytest extensions: https://docs.pytest.org/en/latest/plugins.html
It seems like the lack of plugin architecture prevented enhancements.
Example: "unittest: display time used by each test case" https://bugs.python.org/issue4080
Michael Foord's comment (July, 2010): "Even if it is added to the core it should be in the form of an extension (plugin) so please don't update the patch until this is in place."
Victor
On Mon, 18 Sep 2017 12:16:35 +0200 Victor Stinner victor.stinner@gmail.com wrote:
Antoine:
That would be suitable for a plugin if unittest had a plugin architecture, but not as a core functionality IMO.
My hidden question is more why unittest doesn't already have a plugin system, whereas there are tons on projects based on unittest extending its features like pytest, nose, testtools, etc.
Michael Foord or Robert Collins may be able to answer that question :-) Though I suspect the answer has mainly to do with lack of time and the hurdles of backwards compatibility.
Regards
Antoine.
13.09.17 16:42, Victor Stinner пише:
Instead of just making checks before running some tests it would be worth to limit the memory usage hard (by setting ulimit or analogs on other plathforms). The purpose of this option is preventing swapping which makes tests just hanging for hours.
The problem is what include in "all". The set of resources is application specific. Some of resources used in CPython tests make sense only for single test.
This feature looks functionally similar to limiting memory usage.
The discovery feature of unittest looks similar.
Many of them contain a bunch of engineering tricks and evolve quickly. Regrtest now is not a regrtest two years ago, and I'm sure that two years later it will differ too much from the current. Unittest should be more stable.
This feature looks functionally similar to limiting memory usage.
Hum, I don't think so. Limiting the memory usage doesn't catch deadlocks for example.
Victor