[Python-Dev] Python startup time

[Nick Coghlan]

On 21 July 2017 at 05:38, Cesare Di Mauro <cesare.di.mauro at gmail.com> wrote:

>
>
> 2017-07-20 19:23 GMT+02:00 Victor Stinner <victor.stinner at gmail.com>:
>
>> 2017-07-20 19:09 GMT+02:00 Cesare Di Mauro <cesare.di.mauro at gmail.com>:
>> > I assume that Python loads compiled (.pyc and/or .pyo) from the stdlib.
>> That's something that also influences the startup time (compiling source vs
>> loading pre-compiled modules).
>>
>> My benchmark was "python3 -m perf command -- python3 -c pass": I don't
>> explicitly remove .pyc files, I expect that Python uses prebuilt .pyc
>> files from __pycache__.
>>
>> Victor
>>
>
> OK, that should be the best case.
>
> An idea to improve the situation might be to find an alternative structure
> for .pyc/pyo files, which allows to (partially) "parallelize" their loading
> (not execution, of course), or at least speed-up the process. Maybe a GSoC
> project for some student, if no core dev has time to investigate it.
>

Unmarshalling the code object from disk generally isn't the slow part -
it's the module level execution that takes time.

Using the typing module as an example, a full reload cycle takes almost 10
milliseconds:

$ python3 -m perf timeit -s "import typing; from importlib import reload"
"reload(typing)"
.....................
Mean +- std dev: 9.89 ms +- 0.46 ms

(Don't try timing "import typing" directly - the sys.modules cache
amortises the cost down to being measured in nanoseconds, since you're
effectively just measuring the speed of a dict lookup)

We can separately measure the cost of unmarshalling the code object:

$ python3 -m perf timeit -s "import typing; from marshal import loads; from
importlib.util import cache_from_source; cache =
cache_from_source(typing.__file__); data = open(cache, 'rb').read()[12:]"
"loads(data)"
.....................
Mean +- std dev: 286 us +- 4 us

Finding the module spec:

$ python3 -m perf timeit -s "from importlib.util import find_spec"
"find_spec('typing')"
.....................
Mean +- std dev: 69.2 us +- 2.3 us

And actually running the module's code (this includes unmarshalling the
code object, but *not* calculating the import spec):

$ python3 -m perf timeit -s "import typing; loader_exec =
typing.__spec__.loader.exec_module" "loader_exec(typing)"
.....................
Mean +- std dev: 9.68 ms +- 0.43 ms

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170721/ce1d38ef/attachment.html>