Nick Coghlan schrieb am 21.07.2017 um 08:23:
I'll also note that in these cases where the import overhead is proportionally significant for always-imported modules, we may want to look at the benefits of freezing them (if they otherwise remain as pure Python modules), or compiling them as builtin modules (if we switch them over to Cython), in addition to looking at ways to make the modules themselves faster.
Just for the sake of it, I gave the Cython compilation a try. I had to apply the attached hack to Lib/typing.py to get the test passing, because it uses frame call offsets in some places and Cython functions do not create frames when being called (they only create them for exception traces). I also had to disable the import of "abc" in the Cython generated module to remove the circular self dependency at startup when the "abc" module is compiled. That shouldn't have an impact on the runtime performance, though. Note that this is otherwise using the unmodified Python code, as provided in the current modules, constructing and using normal Python classes for everything, no extension types etc. Only two stdlib Python modules were compiled into shared libraries, and not statically linked into the CPython core. I used the "python_startup" benchmark in the "performance" project to measure the overall startup times of a clean non-debug non-pgo build of CPython 3.7 (rev d0969d6) against the same build with a compiled typing.py and abc.py. To compile these modules, I used the following command (plus the attached patch) $ cythonize -3 -X binding=True -i Lib/typing.py Lib/abc.py I modified the startup benchmark to run "python -c 'import typing'" etc. instead of just executing "pass". - stock CPython starting up and running "pass": Mean +- std dev: 14.7 ms +- 0.3 ms - stock CPython starting up and running "import abc": Mean +- std dev: 14.8 ms +- 0.3 ms - with compiled abc.py: Mean +- std dev: 14.9 ms +- 0.3 ms - stock CPython starting up and running "import typing": Mean +- std dev: 34.6 ms +- 1.0 ms - with compiled abc.py Mean +- std dev: 34.4 ms +- 0.6 ms - with compiled typing.py: Mean +- std dev: 33.5 ms +- 0.7 ms - with both compiled: Mean +- std dev: 33.1 ms +- 0.4 ms That's only a 4% improvement in the overall startup time on my machine, and about a 7% faster overall runtime of "import typing" compared to "pass". Note also that compiling abc.py leads to a slightly *increased* startup time in the "import abc" case, which might be due to the larger file size of the abc.so file compared to the abc.pyc file. This is amortised by the decreased runtime in the "import typing" case (I guess). I then ran the test suites for both modules in lack of a better post-startup runtime benchmark. The improvement for abc.py is in the order of 1-2%, but test_typing.py has many more tests and wins about 13% overall: - stock CPython executing essentially "runner.run(deepcopy(suite))" in "test_typing.py" (the deepcopy() takes about 6 ms): Mean +- std dev: 68.6 ms +- 0.8 ms - compiled abc.py and typing.py: Mean +- std dev: 60.7 ms +- 0.7 ms One more thing to note: the compiled modules are quite large. I get these file sizes: 8658 Lib/abc.py 7525 Lib/__pycache__/abc.cpython-37.pyc 369930 Lib/abc.c 122048 Lib/abc.cpython-37m-x86_64-linux-gnu.so 80290 Lib/typing.py 73921 Lib/__pycache__/typing.cpython-37.pyc 2951893 Lib/typing.c 1182632 Lib/typing.cpython-37m-x86_64-linux-gnu.so The .so files are about 16x as large as the .pyc files. The typing.so file weighs in with about 40% of the size of the stripped python binary: 2889136 python As it stands, the gain is probably not worth the increase in library file size, which also translates to a higher bottom line for the memory consumption. At least not for these two modules. Manually optimising the files would likely also reduce the .so file size in addition to giving better speedups, though, because the generated code would become less generic. Stefan