[Distutils] Building wheels - project metadata, and specifying compatibility

[Daniel Holth]

On Wed, Mar 20, 2013 at 9:11 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 20 March 2013 12:44, Daniel Holth <dholth at gmail.com> wrote:
>> On the other hand Python 3.4 knows it is compatible with "py33" and
>> will pick up that wheel too.
>>
>> It is designed this way to provide a (small) distinction between the
>> safe default and intentional cross-Python-compatible publishing.
>
> Good point. I keep forgetting it does that. (I still maintain that
> behaviour is very non-intuitive, but I'm willing to accept that unless
> someone else pipes up,. it's probably just me :-))
>
>>> In addition to a means for projects to specify this themselves, tools
>>> (bdist_wheel, pip wheel) should probably have a means to override the
>>> default at the command line, as it will be some time before projects
>>> specify this information, even once it is standard. There's always the
>>> option to rename the generated file, but that feels like a hack...
>>
>> I need to do a "wheel retag" tool instead of a simple "rename" because
>> now the WHEEL metadata is supposed to contain all the information in
>> the filename through the Tag and Build keys. This lets us effectively
>> sign the filename.
>
> Again good point. If I get some free time, I might take a stab at that
> if you'd like...
>
>>> Where C extensions are involved, there are other questions. Mostly,
>>> compiled code is implementation, architecture, and minor version
>>> specific, so there's little to do here. The stable ABI is relevant,
>>> but I have no real experience of using it to know how that would work.
>>> There is also the case of projects with C accelerators - it would be
>>> good to be able to easily build both the accelerated version and a
>>> fallback pure-python wheel. I don't believe this is easy as things
>>> stand - distutils uses a compiler if it's present, so forcing a
>>> pure-python build when you have a compiler is harder work than it
>>> needs to be when building binary distributions.
>>
>> This is an open problem, for example in pypy they might be C
>> decelerators. There should be a better way to have optional or
>> conditional C extensions.
>
> Agreed. These are definitely hard issues, and a proper solution won't
> be quickly achieved. What we have now is a good 80% solution, but
> let's keep the remaining 20% in mind.
>
>>> Comments? Should the default in bdist_wheel and pip wheel be changed
>>> or should it remain "as safe as possible" (practicality vs purity)? If
>>> the latter, should override flags be added, or is renaming the wheel
>>> in the absence of project metadata the recommended approach? And does
>>> anyone have any experience of how this might all work with C
>>> extensions?
>>
>> I would like to see the setup.cfg metadata used by bdist_wheel
>> expanded and standardized. The command line override would also be
>> good. Does anyone have the stomach to put some of that into distutils
>> or setuptools itself?
>
> Agreed. My question would be, should this be exposed anywhere in the
> project metadata? (For example, for other tools that use distlib to
> build wheels and need to know programatically what tags to use).

I think setup.cfg counts as far as build metadata is concerned.

> By the way, one point I dislike with the bdist_wheel solution is that
> it explicitly strips #-comments from the end of the universal= line. I
> can see why you want to be able to use end-of-line comments, but it's
> not part of the standard configparser format, and you don't support
> ';' style comments (which could confuse people).

That wasn't really intentional and it probably doesn't need to do
that. It's piggybacking on top of the distutils config parsing system
which may do what's needed already.