[Distutils] Plans for binary wheels, and PyPi and OS-X

[Ronald Oussoren]

On 31 Oct, 2013, at 15:26, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 31 October 2013 23:38, Ronald Oussoren <ronaldoussoren at mac.com> wrote:
>> 
>> On 21 Oct, 2013, at 20:52, Donald Stufft <donald at stufft.io> wrote:
>> 
>>> 
>>> On Oct 21, 2013, at 1:02 PM, Chris Barker <chris.barker at noaa.gov> wrote:
>>> 
>>>> On Fri, Oct 18, 2013 at 6:22 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>>>> -- it would be very useful if folks could easily
>>>>>> get binary wheels for OS-X
>>>>> 
>>>>> We do plan to support it, but the pip devs uncovered a hole in the current
>>>>> wheel spec that means it generates the same filename on *nix systems for
>>>>> wheels that need to have different names for the download side of things to
>>>>> work properly
>>>> 
>>>> THanks -- but really? don't OS-X wheels get:
>>>> 
>>>> macosx_10_6_intel
>>>> 
>>>> or some such tacked on? Where does that go wrong?
>>> 
>>> Homebrew, Mac Ports, Fink. That would work OK if nobody ever installed things
>>> that the system didn't provide.
>> 
>> I don't understand this, what would break? Binary wheels that reference shared libraries that aren't included in the wheel (or a default system install) won't work, but that's also true on Windows.
> 
> The main difference I see is that on Windows, the ABI for the
> *CPython* shared library is significantly less likely to vary across
> machines.
> 
> By contrast, it's relatively easy to change the ABI on *nix systems,
> as it depends on what you pass as configure options.
> 
> Will a C extension built with Homebrew Python work with the Python Mac
> OS X installer from python.org?

As long as they use the same "ABI Tag" from PEP 425 extensions build with
one should work with the other. 

> Probably, but given how painful ABI
> mismatches with shared libraries can be to debug, "probably" doesn't
> cut it until someone has the chance to thoroughly review the potential
> for problems.

I'd have to make time for a thorough review but be sure, but AFAIK this is 
only a little more complicated on OSX than on (say) Linux and the additonal 
complication is already captured by the platform tag mentioned by Chris.  

That is, the CPython ABI is affected by:

* The usual configure flags (--with-pydebug, --enable-unicode, ...)
   
  These issues also exists on Windows, and the "ABI Tag" from PEP 425 means
  that wheels for different sets of configure flags can be recognized from
  their filename.

* The OSX deployment target (the minimal OSX release you want to support)

  This is already recorded by the distutils platform tag 

* The processor architectures supported

  This is also recorded in the distutils platform tag.

* Possibly the compiler

  This is pretty unlikely, as I (and others) already load extensions build 
  with clang in CPython binaries built with GCC (and v.v.), but the low-level 
  compiler support library might affect the ABI when using a compiler other
  than GCC or clang. And I'd expect that two-level namespaces on OSX make that
  question moot except for code that's explicitly built to disable that feature.

  And this does definitely affect CPython on Windows unless you use the limited ABI.
  The only reason this doesn't cause problems with binary installers on PyPI
  right now is that most users use the binary installers from www.python.org and hence
  are all using the same compiler version.

The deployment target can affect the unix level API a little, IIRC around 10.4 
or 10.5 the APIs on the POSIX level were adjusted a little to more fully comply 
with the UNIX specification and because of this the affected named resolve to 
different symbols based on the deployment target (more or less simular to symbol 
versioning in glibc). AFAIK that shouldn't affect the CPython ABI, but that would 
need to be carefully checked to be sure and I don't have time to do so right now.

To provide more detail on the disutils platform tag, on my machine:

>>> import distutils.util
>>> distutils.util.get_platform()
'macosx-10.8-intel'

This means the deployment target is OSX 10.8, with suppport for the 'intel' set 
of archictures (i386 and x86_64). This records the two variables that are most likely
to cause problems.

I'd expected more problems with binary eggs that are dynamicly linked to libraries
that are provided in the wheel because OSX's linker add an absolute path to the library
to be loaded unless you're careful.

Ronald