[Python-Dev] Python startup time

[Barry Warsaw]

On May 2, 2018, at 09:42, Gregory Szorc <gregory.szorc at gmail.com> wrote:

> As for things Python could do to make things better, one idea is for "package bundles." Instead of using .py, .pyc, .so, etc files as separate files on the filesystem, allow Python packages to be distributed as standalone "archive" files.

Of course, .so files have to be extracted to the file system, because we have to live with dlopen()’s API.  In our first release of shiv, we had a loader that did exactly that for just .so files.  We ended up just doing .pyz file unpacking unconditionally, ignoring zip-safe, mostly because too many packages still use __file__, which doesn’t work in a zipapp.

I’ll plug shiv and importlib.resources (and the standalone importlib_resources) again here. :)

> If you go this route, please don't require the use of zlib for file compression, as zlib is painfully slow compared to alternatives like lz4 and zstandard.

shiv works in a similar manner to pex, although it’s a completely new implementation that doesn’t suffer from huge sys.paths or the use of pkg_resources.  shiv + importlib.resources saves us 25-50% of warm cache startup time.  That makes things better but still not ideal.  Ultimately though that means we don’t suffer from the slowness of zlib since we don’t count cold cache times (i.e. before the initial pyz unpacking operation).

Cheers,
-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: Message signed with OpenPGP
URL: <http://mail.python.org/pipermail/python-dev/attachments/20180502/33835037/attachment.sig>