[Distutils] Beyond wheels 1.0: helping downstream, FHS and more

[Daniel Holth]

On Mon, Apr 13, 2015 at 10:54 AM, David Cournapeau <cournape at gmail.com> wrote:
>
>
> On Mon, Apr 13, 2015 at 10:44 AM, Donald Stufft <donald at stufft.io> wrote:
>>
>>
>> On Apr 13, 2015, at 10:39 AM, David Cournapeau <cournape at gmail.com> wrote:
>>
>> Hi there,
>>
>> During pycon, Nick mentioned there was interest in updating the wheel
>> format to support downstream distributions. Nick mentioned Linux
>> distributions, but I would like to express interest for other kind of
>> downstream distributors like Anaconda from Continuum or Canopy from
>> Enthought (disclaimer: I work for Enthought).
>>
>> Right now, wheels have the following limitations for us:
>>
>> 1. lack of post/pre install/removing
>> 2. more fine-grained installation scheme
>> 3. lack of clarify on which tags vendors should use for custom wheels:
>> some packages we provide would not be installable on "normal" python, and it
>> would be nice to have a scheme to avoid confusion there as well.
>>
>> At least 1. and 2. are of interest not just for us.
>>
>> Regarding 2., it looks like anything in the <wheel_name>.data/data
>> directory will be placed as is in sys.prefix by pip. This is how distutils
>> scheme is defined ATM, but I am not sure whether that's by design or
>> accident ?
>>
>> I would suggest to use something close to autotools, with some tweaks to
>> work well on windows. I implemented something like this in my project bento
>> (https://github.com/cournape/Bento/blob/master/bento/core/platforms/sysconfig.py),
>> but we could of course tweak that.
>>
>> For 1., I believe it was a conscious decision not to include them in wheel
>> 1.0 ? Would it make sense to start a discussion to add it to wheel ?
>>
>> I will be at the pycon sprints until wednesday evening, so that we can
>> flesh some concrete proposal first, if there is enough interest.
>>
>> As a background: at Enthought, we have been using eggs to distribute
>> binaries of python packages and other packages (e.g. C libraries, compiled
>> binaries, etc...) for a very long time. We had our own extensions to the egg
>> format to support this, but I want to get out of eggs so as to make our own
>> software more compatible with where the community is going. I would also
>> like to avoid making ad-hoc extensions to wheels for our own purposes.
>>
>>
>> To my knowledge, (1) was purposely punted until a later revision of Wheel
>> just to make it easier to land the “basic” wheel.
>
>
> Great. Was there any proposal made to support it at all ? Or should I just
> work from scratch there ?
>
>>
>>
>> I think (2) is a reasonable thing as long as we can map it sanely on all
>> platforms.
>
>
> Yes. We support all platforms at Enthought, and Windows is important for us
> !
>
>>
>> I’m not sure what (3) means exactly. What is a “normal” Python, do you
>> modify Python in a way that breaks the ABI but which isn’t reflected in the
>> standard ABI tag?
>
>
> It could be multiple things. The most obvious one is that generally.
> cross-platforms python distributions will try to be "relocatable" (i.e. the
> whole installation can be moved and still work). This means they require
> python itself to be built a special way. Strictly speaking, it is not an ABI
> issue, but the result is the same though: you can't use libraries from
> anaconda or canopy on top of a normal python
>
> More generally, we could be modifying python in a way that is not forward
> compatible with upstream python: a binary that works on our python may not
> work on the python from python.org (though the opposite is true). It would
> be nice if one could make sure pip will not try to install those eggs when
> installed on top of a python that does not advertise itself as "compatible"

We need a hook to alter pip's list of compatible tags
(pip.pep425tags), and to alter the default tags used by bdist_wheel
when creating wheels. One sensible proposal for "special" wheels is to
just use a truncated hash of the platform description (a random hex
string) in place of the wheel platform tag.