[Distutils] Using Wheel with zipimport

[Donald Stufft]

On Jan 28, 2014, at 11:54 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> 
> On 29 Jan 2014 11:19, "Daniel Holth" <dholth at gmail.com> wrote:
> >
> > On Tue, Jan 28, 2014 at 7:18 PM, Donald Stufft <donald at stufft.io> wrote:
> > >
> > > On Jan 28, 2014, at 6:38 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > >
> > > I think you're reading too much into that comment. Putting a wheel file
> > > directly on sys.path is no different from putting any other zipfile directly
> > > on sys.path - whether or not it will work depends on the context, but it's a
> > > useful capability if used responsibly (as we do in the ensurepip
> > > implementation).
> > >
> > > The key problems with eggs in relation to this were:
> > > - easy_install preferring to install as eggs by default
> > > - setuptools install a global site hook that added every installed egg to
> > > sys.path for every application run in that Python installation
> > >
> > > Neither of those applies to wheels - pip always unpacks them when
> > > installing, and if you want to add one to sys.path you have to do it
> > > manually, it doesn't happen automatically.
> > >
> > > All the new note in the PEP is clarifying is that it *isn't* an accident
> > > that the wheel format is zipimport compatible for pure Python wheels, we
> > > deliberately designed it that way (hence the "Root-is-purelib" setting,
> > > rather than requiring separate purelib and platlib subdirectories).
> > >
> > > Cheers,
> > > Nick.
> > >
> > >
> > > Regardless if it was or wasn't an accident, I believe it was a mistake.
> > > Supporting it officially at all means that we have limitations on what we
> > > can
> > > do to make Wheel a better format. I had hopes that Wheel could be made more
> > > generic than it currently is, but because of the fact that directly adding
> > > them to sys.path is supported that makes it much much more awkward to do so.
> >
> > Hey, as long as they are zipfiles that don't totally scramble the
> > layout of the internal Python code you can add them to sys.path. Did
> > you know you can even add subpaths of a zipfile to sys.path? </mind
> > blown>
> 
> Yep, the only requirement is "there will be a non empty subset of valid wheel files that can be used directly with zipimport".
> 
> Wheels that include C extensions, or otherwise need to be unpacked to disk in order to work correctly aren't in that subset, and running directly from a wheel does prevent bytecode caching, but those are both OK - "supported when practical" isn't the same thing as "recommended".
> 
> Cheers,
> Nick.
> 
> >
> > I'm opposed to making wheel generic as in "package PostgreSQL itself"
> > generic. There are other ways to do that.
> 


And yet on another read through of PEP427 the first item in “Comparison to Egg”
is "Wheel is an installation format; egg is importable.”. The only other mention of
importing any of them in that PEP is the section you just added saying that Wheels
are designed to be importable. The original text of the PEP does not provide any
evidence that Wheels were meant to be importable and instead makes it a point
to call that a difference from Eggs.

Further more there is a comment from Daniel on python-dev [1] which states that

    “Jim Fulton is right that weird failures are a characteristic of zipped 
     eggs, so one of the #1 requests for setuptools is how to prohibit 
     zipping from ever happening. This is an important reason why wheel is 
     billed as an installation format -- fewer users with pitchforks. It's 
     very cool that it works though. Debugging is slightly easier than it 
    was in the old days because pdb can now read the source code from the 
    zip.”

Which further shows that at the time it was “cool that it worked” but that the
“weird failures” being a reason that Wheel was an installation format.

I believe that adding this “feature” to the PEP ex post facto is a bad idea. Had the
PEP contained anything to indicate that Wheels were intended to be importable 
as part of the design philosophy I would have spoken out about it then instead it’s
been added after it was accepted with no discussion that I can see.

If I missed where this discussion happened during the PEP please direct me to it
because I’ve just spent 15 minutes googling attempting to find any information on it,
and all I’ve been able to find is Vinay’s experiment with Wheel.mount(), You mentioning
the possibility of using Wheels in the sense of Multi Version Installs (which ends up
talking about a Wheel inspired directory layout instead), and the thread I mentioned
above. Not that it’s entirely relevant because really regardless of if it was discussed
if it wasn’t encoded in the PEP than people following along by watching the PEPs
had no ability to step in to speak for or against it.

[1] https://groups.google.com/d/msg/dev-python/BS28mF7mb6c/o8jOo1NcousJ

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140129/4c0e5a69/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140129/4c0e5a69/attachment-0001.sig>