[Distutils] Building wheels - project metadata, and specifying compatibility

[Daniel Holth]

On Wed, Mar 20, 2013 at 8:27 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> When building wheels, it is necessary to know details of the
> compatibility requirements of the code. The most common case is for
> pure Python code, where the code could in theory be valid for a single
> Python version, but in reality is more likely to be valid either for
> all Pythons, or sometimes for just Python 2 or Python 3 (where
> separate code bases or 2to3 are involved). The wheel project supports
> a "universal" flag in setup.cfg, which sets the compatibility flags to
> 'py2.py3', but that is only one case.
>
> Ultimately, we need a means (probably in metadata) for (pure Python)
> projects to specify any of the following:
>
> 1. The built code works on any version of Python (that the project supports)
> 2. The built code is specific to the major version of Python that it
> was built with
> 3. The built code is only usable for the precise Python version it was
> built with
>
> The default is currently (3), but this is arguably the least common
> case. Nearly all code will support at least (2) and more and more is
> supporting (1).
>
> Note that this is separate from the question of what versions the
> project supports. It's about how the code is written. Specifically,
> there's no point in marking code that uses new features in Python 3.3
> as .py33 - it's still .py3 as it will work with Python 3.4. The fact
> that it won't work on Python 3.2 is just because the project doesn't
> support Python 3.2. Installing a .py3 wheel into Python 3.2 is no
> different from installing a sdist there. So overspecifying the wheel
> compatibility so that a sdist gets picked up for earlier versions
> isn't helpful.

On the other hand Python 3.4 knows it is compatible with "py33" and
will pick up that wheel too.

It is designed this way to provide a (small) distinction between the
safe default and intentional cross-Python-compatible publishing.

> In addition to a means for projects to specify this themselves, tools
> (bdist_wheel, pip wheel) should probably have a means to override the
> default at the command line, as it will be some time before projects
> specify this information, even once it is standard. There's always the
> option to rename the generated file, but that feels like a hack...

I need to do a "wheel retag" tool instead of a simple "rename" because
now the WHEEL metadata is supposed to contain all the information in
the filename through the Tag and Build keys. This lets us effectively
sign the filename.

> Where C extensions are involved, there are other questions. Mostly,
> compiled code is implementation, architecture, and minor version
> specific, so there's little to do here. The stable ABI is relevant,
> but I have no real experience of using it to know how that would work.
> There is also the case of projects with C accelerators - it would be
> good to be able to easily build both the accelerated version and a
> fallback pure-python wheel. I don't believe this is easy as things
> stand - distutils uses a compiler if it's present, so forcing a
> pure-python build when you have a compiler is harder work than it
> needs to be when building binary distributions.

This is an open problem, for example in pypy they might be C
decelerators. There should be a better way to have optional or
conditional C extensions.

> Comments? Should the default in bdist_wheel and pip wheel be changed
> or should it remain "as safe as possible" (practicality vs purity)? If
> the latter, should override flags be added, or is renaming the wheel
> in the absence of project metadata the recommended approach? And does
> anyone have any experience of how this might all work with C
> extensions?

I would like to see the setup.cfg metadata used by bdist_wheel
expanded and standardized. The command line override would also be
good. Does anyone have the stomach to put some of that into distutils
or setuptools itself?

Daniel Holth