[Distutils] Plans for binary wheels, and PyPi and OS-X

[Paul Moore]

On 1 November 2013 15:48, Chris Barker <chris.barker at noaa.gov> wrote:
> On Fri, Nov 1, 2013 at 6:59 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> The key point here is the granularity of the PEP 425 tags used by wheel.
>>
>> The risk is that a wheel created on another system might declare (via
>> its filename) that it is compatible with your system, and then not be,
>> causing segfaults or similar when used.
>
> indeed, which is why it _might_ be a good idea to include an extra python
> build flag or something: "python.org", "homebrew", "macports". However, it's
> probably the case that those aren't really the issues that are going to
> cause problems -- at least ones that aren't already handled by the OS-X
> version flags --- i.e if a package is built for 10.6+, then it should have
> the same system libs as a python built for 10.6+.

The first thing to address is what wheel currently *does*. Basically,
a (binary, let's ignore pure Python here) wheel is tagged as follows:

Python version - cpXY
Platform - distutils.util.get_platform(), which is "darwin" for OSX,
if I'm reading the code correctly.
ABI - essentially sysconfig.get_config_vars().get('SOABI') if that exists

The only one here which may vary based on build is ABI. So the
question is, do different Python builds on OSX have different SOABI
values?

If not, then does a simple extension with no external dependencies
work when built with one Python and used with a different one?
   If it does, then there's probably no OSX issue[1].
   If it doesn't, then there *is* an issue, and some code needs to
change for wheels to be tagged correctly. Probably by changing the
SOABI values, or by special coding in the ABI tag generation in
bdist_wheel and wherever else it needs to be determined.

If there are different SOABI values, then the above question about a
simple extension should be asked for any SOABI value used by more than
one Python build (if any).

The issue is different on Windows (whether for technical or social
reasons is debatable but irrelevant) because there is *no* ABI tag
used. The platform tag catches cygwin and 32-bit vs 64-bit
differences, and the Python version catches the C runtime
compatibility (because any version of Python is only supported with
one particular CRT - that's embedded in distutils in a way that means
that you'd need to change the code to alter it, which is why it's
arguably a technical issue rather than a social one).

The point marked [1] above is where I ignore the question of "external
libraries" :-)

I'm honestly not sure what is meant by "external libraries" here. Does
it mean which version of the C runtime is used? If so, then that
should probably affect the ABI, but I don't know enough to know if
that's right or not. If the libc version isn't part of the ABI, then I
think this is the issue that caused people to back off on wheels for
non-Windows platforms. I'd like to see a clear statement on whether
building an extension with one libc version and running it with a
different one is supported (I suspect "no") and whether it is
likely/possible (on Windows, "not possible for all practical
purposes", on Linux I suspect "happens all the time" and I have no
idea for OSX).

If by "external libraries" you mean something other than libc (for
example, something like BLAS libraries for scipy) then my view is that
this is not something taht wheel compatibility tags is intended to
solve. The project should document what external libraries need to be
present and any limitations on how they were built that may apply. If
the user disregards these, that's their problem. If projects want to
distribute one wheel for BLAS-compiled-with-homebrew, and one for
BLAS-compiled-with-macports, then I'd put the onus on the project to
come up with a solution - I think it's a specialised enough case that
the wheel spec shouldn't be worrying about it yet, if ever.

I hope this is of some use - maybe it'll help the people with access
to OSX to have a clearer idea of what needs checking.

Paul