On 21 July 2017 at 05:38, Cesare Di Mauro <cesare.di.mauro@gmail.com> wrote:
2017-07-20 19:23 GMT+02:00 Victor Stinner <victor.stinner@gmail.com>:
2017-07-20 19:09 GMT+02:00 Cesare Di Mauro <cesare.di.mauro@gmail.com>:
I assume that Python loads compiled (.pyc and/or .pyo) from the stdlib. That's something that also influences the startup time (compiling source vs loading pre-compiled modules).
My benchmark was "python3 -m perf command -- python3 -c pass": I don't explicitly remove .pyc files, I expect that Python uses prebuilt .pyc files from __pycache__.
Victor
OK, that should be the best case.
An idea to improve the situation might be to find an alternative structure for .pyc/pyo files, which allows to (partially) "parallelize" their loading (not execution, of course), or at least speed-up the process. Maybe a GSoC project for some student, if no core dev has time to investigate it.
Unmarshalling the code object from disk generally isn't the slow part - it's the module level execution that takes time. Using the typing module as an example, a full reload cycle takes almost 10 milliseconds: $ python3 -m perf timeit -s "import typing; from importlib import reload" "reload(typing)" ..................... Mean +- std dev: 9.89 ms +- 0.46 ms (Don't try timing "import typing" directly - the sys.modules cache amortises the cost down to being measured in nanoseconds, since you're effectively just measuring the speed of a dict lookup) We can separately measure the cost of unmarshalling the code object: $ python3 -m perf timeit -s "import typing; from marshal import loads; from importlib.util import cache_from_source; cache = cache_from_source(typing.__file__); data = open(cache, 'rb').read()[12:]" "loads(data)" ..................... Mean +- std dev: 286 us +- 4 us Finding the module spec: $ python3 -m perf timeit -s "from importlib.util import find_spec" "find_spec('typing')" ..................... Mean +- std dev: 69.2 us +- 2.3 us And actually running the module's code (this includes unmarshalling the code object, but *not* calculating the import spec): $ python3 -m perf timeit -s "import typing; loader_exec = typing.__spec__.loader.exec_module" "loader_exec(typing)" ..................... Mean +- std dev: 9.68 ms +- 0.43 ms Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia