[Python-ideas] Move some regrtest or test.support features into unittest?

[Michel Desmoulin]

Can you elaborate on _why_ you think something is good for core/a plugin
? Cause right now it's impossible to know what's the logic behind.

Le 17/09/2017 à 22:31, Antoine Pitrou a écrit :
> On Wed, 13 Sep 2017 15:42:56 +0200
> Victor Stinner <victor.stinner at gmail.com>
> wrote:
>> I would like to see if and how we can integrate/move some regrtest
>> features into the unittest module. Example of regrtest features:
>>
>> * skip a test if it allocates too much memory, command line argument
>> to specify how many memory a test is allowed to allocate (ex:
>> --memlimit=2G for 2 GB of memory)
> 
> That would be suitable for a plugin if unittest had a plugin
> architecture, but not as a core functionality IMO.
> 
>> * concept of "resource" like "network" (connect to external network
>> servers, to the Internet), "cpu" (CPU intensive tests), etc. Tests are
>> skipped by default and enabled by the -u command line option (ex: "-u
>> cpu).
> 
> Good as a core functionality IMO.
> 
>> * track memory leaks: check the reference counter, check the number of
>> allocated memory blocks, check the number of open file descriptors.
> 
> Good for a plugin IMO.
> 
>> * detect if the test spawned a thread or process and the
>> thread/process is still running at the test exit
> 
> Good for a plugin IMO.
> 
>> * --timeout: watchdog killing the test if the run time exceed the
>> timeout in seconds (use faulthandler.dump_traceback_later)
> 
> Good for a plugin IMO.
> 
>> * multiprocessing: run tests in subprocesses, in parallel
> 
> Good as a core functionality IMO.
> 
>> * redirect stdout/stderr to pipes (StringIO objects), ignore them on
>> success, or dump them to stdout/stderr on test failure
> 
> Good for a plugin IMO.
> 
>> * --slowest: top 10 of the slowest tests
> 
> Good for a plugin IMO.
> 
>> * --randomize: randomize test order
> 
> Will be tricky to mix with setupClass.
> 
>> * --match, --matchfile, -x: filter tests
> 
> Good as a core functionality IMO.
> 
>> * --forever: run the test in a loop until it fails (or is interrupted by CTRL+c)
> 
> Good for a plugin IMO.
> 
>> * --list-tests / --list-cases: list test files / test methods
> 
> Good as a core functionality IMO.
> 
>> * --fail-env-changed: mark tests as failed if a test altered the environment
> 
> Good for a plugin IMO.
> 
>> * detect if a "global variable" of the standard library was modified
>> but not restored by the test:
> 
> Good for a plugin IMO.
> 
>> * test.bisect: bisection to identify the failing method, used to track
>> memory leaks or identify a test leaking a resource (ex: create a file
>> but don't remove it)
> 
> Good as a core functionality IMO.
> 
>> I started to duplicate code in many files of Lib/test/test_*.py to
>> check if tests "leak running threads" ("dangling threads"). Example
>> from Lib/test/test_theading.py:
>>
>> class BaseTestCase(unittest.TestCase):
>>     def setUp(self):
>>         self._threads = test.support.threading_setup()
>>
>>     def tearDown(self):
>>         test.support.threading_cleanup(*self._threads)
>>         test.support.reap_children()
>>
>> I would like to get this test "for free" directly from the regular
>> unittest.TestCase class, but I don't know how to extend the unittest
>> module for that?
> 
> Instead of creating tons of distinct base TestCase classes, you can just
> provide helper functions / methods calling addCleanup().
> 
> Regards
> 
> Antoine.
> 
> 
> _______________________________________________
> Python-ideas mailing list
> Python-ideas at python.org
> https://mail.python.org/mailman/listinfo/python-ideas
> Code of Conduct: http://python.org/psf/codeofconduct/
>