[Distutils] Using Wheel with zipimport

[Donald Stufft]

On Jan 29, 2014, at 10:00 AM, Vinay Sajip <vinay_sajip at yahoo.co.uk> wrote:

> 
> --------------------------------------------
> On Wed, 29/1/14, Donald Stufft <donald at stufft.io> wrote:
> 
>> It’s hard to pin down because the failure modes of zipped
>> eggs are nebulous themselves.
> 
>> For instance take pip. I just recently redid the get-pip.py
>> installer to use a zip file (not a Wheel or Egg) that contained pip
>> and add that zip file straight to sys.path instead of unzipping it.
>> One of the failure modes was that it was suddenly unable to validate
>> TLS certificates. The reason why, was because it bundles it’s
>> own cacerts.pem which it passes to the ssl library to validate the
>> connection. In this case it’s impossible as far as I can tell to use
>> pkgutil.get_data directly because the ssl library does not directly
>> support pkgutil.get_data nor does it support passing the certificates
>> as a string or other in memory.
> 
>> That’s really the biggest problem with strictly pure python Zipped Eggs.
> 
> If that's all, I'm not overly worried as long as there's a mountability flag on
> wheels. I looked at running pip from a zip a while ago, the cacerts thing
> was easy to diagnose and fix - a five-or-ten-minute job. (I'm not trying to
> come across as a show-off.) Here's the code I added as a fix in
> pip/locations.py (no doubt it could be improved):
> 
>    def _check_for_zip():
>        import zipimport
>        global default_cert_path
>        try:
>            if isinstance(__loader__, zipimport.zipimporter):
>                # running from a zip file. Save data to a handy location.
>                data = __loader__.get_data(default_cert_path)
>                destdir = os.path.expanduser('~/.pip')
>                if not os.path.isdir(destdir):
>                    os.mkdir(destdir)
>                default_cert_path = os.path.join(destdir, 'cacert.pem')
>                with open(default_cert_path, 'wb') as f:
>                    f.write(data)
>        except NameError:
>            # __loader__ not defined, so not in zip
>            pass
> 
>    _check_for_zip()
>    del _check_for_zip
> 
> Of course, pip wasn't originally designed to run from zip, so one would
> expect to find these kinds of issues. OTOH, if importable zips were more
> commonplace, then presumably people would think about these kinds of
> issues more when writing their code. Of course, in the above case, using
> pkgutil wouldn't have worked, and there are bound to be other similar cases,
> but I still don't see any real reason to fear the consequences of importable
> wheels.
> 
> Regards,
> 
> Vinay Sajip

FWIW I'm not particularly that upset about this feature itself. I think it's a
bad idea and I think the way it locks the format into a particular method is a
poor trade off and that there is possibly a better form for this feature, but
if I had engaged the process argued my side, and then lost I could deal with
that.

What I'm upset about is that I can find no support that the *PEP* text (not the
intentions of the PEP) contains explicit support for this feature (as in, it
exists because we want it to work and we think people should use it) and little
to no evidence that the fact that this was (apparently) a design goal was
properly communicated (which includes multiple statements by various people
including
Daniel that it wasn't really designed for this, though Daniel is apparently recanting
that meaning). According to PEP1 a PEP must "be a clear and complete description
of the proposed enhancement". 

I believe that what was proposed for discussion, if it was indeed intended that
Wheels should be generally importable by adding them to sys path, was not clear
about the fact that this was part of the proposal (and in fact contains text
that reads contrary to that intent), and because the proposal and the resulting
discussion was unclear, that people, such as myself, were not given the chance
to discuss this particular aspect of the proposal.

I feel that as BDFL delegate Nick is abusing the fact that he ultimately
accepted the PEP to circumvent the lack of a clear proposal of a particular
design goal (albeit likely unwittingly) to include this part of the proposal
without discussion. *This* is what I have a problem with. The PEP process
exists to gather everyone's input on a proposal and that ability was not
afforded to people who do not believe that this particular feature should be
supported.

This is why I started this thread (which has someone gone off the rails trying
to discuss whether or not this is a good feature or not which is immaterial to
the actual issue), because I believe that this change should be reverted, and
if this is something that Nick, Daniel, Vinay, or anyone else feels is
important then it should be proposed as part of Wheel 1.1 and everyone should
be given a fair chance to discuss it before it is accepted.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140129/d56e160e/attachment.sig>