[Python-Dev] PEP 441 - Improving Python ZIP Application Support

[Barry Warsaw]

On Feb 15, 2015, at 05:21 PM, Thomas Wouters wrote:

>FWIW, we have a similar tool to 'pyzaa' at Google, although a lot older (it
>has a pure-python implementation of zipimport, because it had to work with
>Python 2.2 back in the day) that *does* support extension modules and other
>shared libraries (by extracting them to disk on program startup.) We're
>also working to replace that by loading the extension modules directly from
>the archive (using dlopen_with_offset as described by ppluzhnikov in
>https://sourceware.org/bugzilla/show_bug.cgi?id=11767, although it should
>also be possible using dlopen_ehdr/dlopen_phdr), which requires that
>extension modules are stored uncompressed (simple) and page-aligned
>(harder, as the zipfile.ZipFile class doesn't directly support
>page-aligning anything, but it turns out it's easy to hack around by
>overriding _writecheck :P) And yes, we really are in a position to modify
>glibc to make our life distributing Python applications easier, internally;
>the old code involves shell and Python bootstrap code that tries very hard
>to avoid race conditions and security problems, which is a pain to maintain.

So, let's add this to zipimport so we can remove an unfortunate limitation and
avoid everyone having to reinvent the wheel!

Oh, and the dlopen extension sound pretty cool.  Can't wait until Google gets
that into glibc so us mere mortals can use it.  Seems reasonable to add this
to zipimport too, with whatever feature guards are necessary.  You'll want to
give your future self an opportunity to trot out the keys to the time
machine. :)

>It might be nice to consider those use-cases in pyzapp as well, especially
>once the glibc feature is released. It requires some fairly big changes to
>zipimport (I ended up rewriting the whole thing) but we can easily
>opensource the code, if there's any interest at all.

+1

>I'd be happy to discuss this in more detail at PyCon, at which time we should
>be deploying code internally using all of this.

+1

Cheers,
-Barry