[Python-ideas] bdist naming scheme (compatibility tags) PEP

[Paul Moore]

On 26 September 2012 13:31, Daniel Holth <dholth at gmail.com> wrote:
>> I think that this is fine, but the PEP needs to be explicit. If it's a
>> user option, the PEP should say "installers should allow the user to
>> specify the list of compatibility tags, and the default should be
>> XXX". If it's static, the PEP should say what it is.
>>
>> Having different installers make different, incompatible assumptions,
>> is unpleasant. At present, of course, the only 2 real contenders are
>> the reference wheel implementation and pip. Others like
>> distutils2/packaging may follow.
>
> It might be easier to explain by defining a static list for each
> version of Python and then say "and you can add previous versions to
> the ordered set". Then for CPython 3.3, ignoring abi3, with pymalloc
> giving the cp33m suffix, you could have only
>
> cp33-cp33m-win32
> cp33-none-win32
> cp33-none-any
> py33-none-any
> py3-none-any
>
> implementation - preferred abi - plat
> implementation - none - plat
> implementation - none - any
> python major minor - none - any
> python major - none - any
>
> The rule for generating the last version's tags ignoring abi3 is that
> you only keep the none-any tags:
>
> cp32-none-any
> py32-none-any
> py3-none-any
>
> appending the lists without duplicates you get
>
> cp33-cp33m-win32
> cp33-none-win32
> cp33-none-any
> py33-none-any
> py3-none-any
> cp32-none-any
> py32-none-any
>
> I'm not sure what to do with abi3 or whether to use the cp3 (major
> only) implementation tag.

Win32 is not a good example here. As far as I know (I've experimented
and read docs, but haven't analyzed the code), there is never a
declared ABI on Win32. In fact, Windows is pretty much trivially
simple:

cpXY-none-win32 (for distributions with C extensions)
pyXY-none-any (for pure-Python distributions)

In fact, those two are the only values the bdist_wheel format can
generate. Actually, for non-Windows, it's just as simple - bdist_wheel
can only generate

cpXY-ABI-PLAT (for distributions with C extensions)
pyXY-none-any (for pure-Python distributions)

ABI is the preferred ABI (the part of SOABI after the '-' from the
Python used to build) and PLAT is the platform. (So essentially,
Windows follows the standard pattern, but with an ABI of "none").

Eggs and wininst installers, if they used this convention, would be
the same. As would bdist_msi, as far as I know. So the question is,
what use case is there for anything more complicated than this? The
only possibilities I can see are:

1. The stable ABI. At the moment, I don't know how well that's
supported - I don't think the build tools detect whether code only
uses the stable ABI, so they assume the full ABI. Users could claim to
use the stable ABI by manual renaming. But without an agreed and
documented convention for the stable ABI, they can't do that, so I
think it's premature to worry about that case. It's easy enough to add
if needed (see below - it's just another ABI for installers to allow)

2. UCS2 vs UCS4. This is dead for Python 3.3+, so not worth
complicating the model for.

3. In theory, if a tool could create "fat" archives containing code
for multiple platforms/ABIs, then that might need something more
complex. But in the absence of any such tool, I'd call YAGNI on this.

4. Pure-python code which works on multiple versions of Python. This
is a real case, and needs to be considered. Code that is (presumed)
valid on all Python versions within the current major version can be
manually retagged as pyX. And code that supports Python 2 and 3 can be
retagged as py2.py3. To allow forward compatibility, installers should
allow the user to install pyXZ code on Python version X.Y when Z<Y.
But this should be a user option (possibly off by default) and an
exact match should always be preferred.

I'm not aware of any other cases that might matter here. The other
implementations may well add further use cases - for example, PyPy can
load (some) cPython code, I believe. But without details, let's wait
to hear from them rather than speculating.

The cases above where I suggest manual retagging may benefit from a UI
in the build tools to automatically change the tags, but that's a
quality of implementation issue. At a pinch, renaming a wheel file
would work fine (as long as you didn't lie by doing so!)

That covers the side of the proposal relating to how binary
distributions declare what they were built for.

As regards how installers should check whether packages are
compatible, it seems to me that the rules can be reasonably simple.

1. The installer maintains a spec of what tagsets the current Python
will support - that would be the exact implementation/version ("cpXY"
or similar), a list of supported ABIs in preference order, and the
current platform. The PEP should document how to get the list of
supported ABIs, for completeness.
2. An exact match wins every time. Where there are multiple ABIs, the
best match is based on the preference order supplied.
3. Non-exact matches can only occur for pure-Python packages (as
platform-specific ones declare an exact version/abi/platform as noted
above). Here, we ignore ABI and platform (they will always be
none-any) and work down the list pyXY, pyX, pyXZ (Z<Y, this only if
user allows it) in that order. Where a package declares multi-tags
(py2.py3 is the only likely case) break ties by taking the package
that specifies the fewest tags.

That should be it.

For me, the above summary (or something similar) needs to be in the
PEP, to provide a proper background if nothing else. What do people
think?

Paul.