On 21 July 2017 at 05:38, Cesare Di Mauro <cesare.di.mauro@gmail.com> wrote:

2017-07-20 19:23 GMT+02:00 Victor Stinner <victor.stinner@gmail.com>:
2017-07-20 19:09 GMT+02:00 Cesare Di Mauro <cesare.di.mauro@gmail.com>:
> I assume that Python loads compiled (.pyc and/or .pyo) from the stdlib. That's something that also influences the startup time (compiling source vs loading pre-compiled modules).

My benchmark was "python3 -m perf command -- python3 -c pass": I don't
explicitly remove .pyc files, I expect that Python uses prebuilt .pyc
files from __pycache__.


OK, that should be the best case.

An idea to improve the situation might be to find an alternative structure for .pyc/pyo files, which allows to (partially) "parallelize" their loading (not execution, of course), or at least speed-up the process. Maybe a GSoC project for some student, if no core dev has time to investigate it.

Unmarshalling the code object from disk generally isn't the slow part - it's the module level execution that takes time.

Using the typing module as an example, a full reload cycle takes almost 10 milliseconds:

$ python3 -m perf timeit -s "import typing; from importlib import reload" "reload(typing)"
Mean +- std dev: 9.89 ms +- 0.46 ms

(Don't try timing "import typing" directly - the sys.modules cache amortises the cost down to being measured in nanoseconds, since you're effectively just measuring the speed of a dict lookup)

We can separately measure the cost of unmarshalling the code object:

$ python3 -m perf timeit -s "import typing; from marshal import loads; from importlib.util import cache_from_source; cache = cache_from_source(typing.__file__); data = open(cache, 'rb').read()[12:]" "loads(data)"
Mean +- std dev: 286 us +- 4 us

Finding the module spec:

$ python3 -m perf timeit -s "from importlib.util import find_spec" "find_spec('typing')"
Mean +- std dev: 69.2 us +- 2.3 us

And actually running the module's code (this includes unmarshalling the code object, but *not* calculating the import spec):

$ python3 -m perf timeit -s "import typing; loader_exec = typing.__spec__.loader.exec_module" "loader_exec(typing)"
Mean +- std dev: 9.68 ms +- 0.43 ms


Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia