Tests now start with only 131 imported modules, instead of 233
Hi,
tl;dr Tests of the Python Test Suite now start with 131 imported modules, instead of 233. For example, asyncio, logging, multiprocessing and warnings are no longer imported by default.
--
The Python test suite is run by a custom test runner called "libregrtest" (test.libregrtest). It can detect reference leaks, write the output as JUnit XML, run tests in parallel with multiple processes, etc. Tests are written with test.support which is a collection of helper functions.
Over the years, test.support got more and more functions, and libregrtest got more and more features. The problem is that they import more and more modules. For example, "import test.support" imports 173 modules in Python 3.8!
$ python3.8
import sys, sys before=set(sys.modules); import test.support; after=set(sys.modules) len(after - before)
173
In these imports, you can find some "heavy" modules like asyncio and multiprocessing. Moreover, some modules have "side effects" on import. For example, "import logging" registers an "at fork" callback.
In April 2020, I worked with Hai Shi on Python 3.10 in bpo-40275 to reduce the number of test.support imports from 171 to 25! test.support was splitted into multiple sub-modules:
- bytecode_helper - hashlib_helper - import_helper - logging_helper - os_helper - script_helper - socket_helper - threading_helper - warnings_helper
I continued the work in bpo-41718 to reduce the number of libregrtest imports, since libregrtest also imported many modules.
A dummy test which does nothing in the master branch now only has 131 modules in sys.modules, whereas in Python 3.9 it has 233 (+102) modules! For example, asyncio, logging, multiprocessing and warnings are no longer imported by default.
"import test.support" is now faster. master branch compared to Python 3.8 using ./python -c 'import test.support' command and Python built in release mode:
[py38] 85.7 ms +- 6.8 ms -> [master] 32.4 ms +- 1.3 ms: 2.64x faster
More realistic command running a single test method: ./python -m test test_os -m test_access
[py38] 136 ms +- 5 ms -> [master] 97.8 ms +- 2.4 ms: 1.39x faster
The only annoying point is that splitting test.support into sub-modules was not backported to Python 3.8 and 3.9, and so it will make backports a little bit more annoying. Sorry about that!
--
If you know that two modules should be tested together, please write a dedicated test for that ;-)
Victor
I didn't quite get what the big effect is. Saving 30 milliseconds?
Also, how is the now-split-off funcionality to be invoked? Does it require two or more imports now, or it's imported on demand when one invokes an appropriate test.support entry?
On 23.03.2021 4:29, Victor Stinner wrote:
Hi,
tl;dr Tests of the Python Test Suite now start with 131 imported modules, instead of 233. For example, asyncio, logging, multiprocessing and warnings are no longer imported by default.
--
The Python test suite is run by a custom test runner called "libregrtest" (test.libregrtest). It can detect reference leaks, write the output as JUnit XML, run tests in parallel with multiple processes, etc. Tests are written with test.support which is a collection of helper functions.
Over the years, test.support got more and more functions, and libregrtest got more and more features. The problem is that they import more and more modules. For example, "import test.support" imports 173 modules in Python 3.8!
$ python3.8
import sys, sys before=set(sys.modules); import test.support; after=set(sys.modules) len(after - before)
173
In these imports, you can find some "heavy" modules like asyncio and multiprocessing. Moreover, some modules have "side effects" on import. For example, "import logging" registers an "at fork" callback.
In April 2020, I worked with Hai Shi on Python 3.10 in bpo-40275 to reduce the number of test.support imports from 171 to 25! test.support was splitted into multiple sub-modules:
- bytecode_helper
- hashlib_helper
- import_helper
- logging_helper
- os_helper
- script_helper
- socket_helper
- threading_helper
- warnings_helper
I continued the work in bpo-41718 to reduce the number of libregrtest imports, since libregrtest also imported many modules.
A dummy test which does nothing in the master branch now only has 131 modules in sys.modules, whereas in Python 3.9 it has 233 (+102) modules! For example, asyncio, logging, multiprocessing and warnings are no longer imported by default.
"import test.support" is now faster. master branch compared to Python 3.8 using ./python -c 'import test.support' command and Python built in release mode:
[py38] 85.7 ms +- 6.8 ms -> [master] 32.4 ms +- 1.3 ms: 2.64x faster
More realistic command running a single test method: ./python -m test test_os -m test_access
[py38] 136 ms +- 5 ms -> [master] 97.8 ms +- 2.4 ms: 1.39x faster
The only annoying point is that splitting test.support into sub-modules was not backported to Python 3.8 and 3.9, and so it will make backports a little bit more annoying. Sorry about that!
--
If you know that two modules should be tested together, please write a dedicated test for that ;-)
Victor
Hi Ivan,
On Tue, Mar 23, 2021 at 9:04 AM Ivan Pozdeev via Python-Dev python-dev@python.org wrote:
I didn't quite get what the big effect is. Saving 30 milliseconds?
I started to dig into this issue while debugging a random crash on AIX (bpo-40091). A test_threading test using fork randomly crashed on AIX. I discovered that the crash was triggered by the logging module. It surprised me since threading and test_threading don't use the logging module.
"import test.support" not only loads code but also has side effects because it runs code. Some examples:
* "import logging" calls atexit.register() and os.register_at_fork(). * "import random" calls os.register_at_fork(). * "import readline" sets SIGWINCH signal handler.
In general, it's perfectly fine in tests to import modules which have side effects. But I'm talking about the very specific case of the Python test suite which tests low-level Python internals, any side effect causes multiple annoying issues and makes debugging way harder.
I'm working for years on making the Python test suite more reliable. These side effects indirectly make tests less reliable and less reproducible.
IMO test_threading must not test the logging module. If someone wants to test the relationship between logging, threading and fork, a new dedicated test should be written in test_logging. In this case, there are already fork tests in test_logging ;-)
By the way, I'm also working on fixing these random crashes ;-) For example, I partially fixed the AIX crash at fork in a generic way by adding new _at_fork_reinit() methods to locks: see also the underlying _PyThread_at_fork_reinit() function.
Also, how is the now-split-off funcionality to be invoked? Does it require two or more imports now, or it's imported on demand when one invokes an appropriate test.support entry?
For example, TESTFN should now be get from test.support.os_helper.
Some tests use "from test.support.os_helper import TESTFN", other tests prefer "from test.support import os_helper" and then use "os_helper.TESTFN".
All tests in Lib/test/ have been updated last year, you don't have to do anything.
By the way, the test module should not be used outside Python, it's clearly specified in its documentatioin ;-) On Fedora, the "test" module is not installed by default, but is in a separated python3-test package. (Another reason is to reduce the disk space.)
I hope that it gives you a better idea why these changes were made ;-)
Victor
On Tue, Mar 23, 2021 at 9:04 AM Ivan Pozdeev via Python-Dev python-dev@python.org wrote:
Also, how is the now-split-off funcionality to be invoked? Does it require two or more imports now, or it's imported on demand when one invokes an appropriate test.support entry?
By the way, splitting test.support into sub-modules is also a way to work around the lack of lazy import in Python. It would be nice to have this feature, but so far, no consensus on the lazy import specification was reached.
One issue is that some use cases need the import side effects to be applied as soon as the import is done ... It goes against the principle of lazy import :-( I don't recall the details.
Victor
participants (2)
-
Ivan Pozdeev -
Victor Stinner