Wheel already supports compound tags. Just like py2.py3-none-any a tag py3-none-x86.ppc (with real platform names) would work. Does that make sense for Mac? On May 31, 2013 6:30 PM, "Chris Barker - NOAA Federal" < chris.barker@noaa.gov> wrote:

HI Folks,

A few of us over on the pythonmac list have been hashing out how best to support binary packages on OS-X. The binary wheel option seems very promising.

However, there is one Mac-specific issue that does not seem to be addressed:

On OS-X, binaries can be "universal" -- what this means is that they have more than one binary architecture embedded in a single file; the system selects the right one at run time. Both executables and dynamic libraries can be universal. In theory, an universal binary can currently contain up to 4 architectures:

32+64 bit PPC and 32+64bit x86

In practice, 64 bit PPC was never broadly used, and 32bit PPC is fading fast. Nevertheless, both 32 and 64 Intel systems are pretty common, and who knows what there may be in the future (you never know with Apple...)

The binary builds of Python served up on the python.org web site have supported universal builds for years -- currently there are two options: 32bit PPC+x86 or 32+64bit x86. It's not a single build, as it turns out it's hard to support both up to date x86_64 and PPC with the same compiler. In these builds, the configuration is set up so that distutils will build universal extensions that match the architectures included in the builds. This makes it pretty easy to build binary packages, and/or full app distributions (py2app) that are universal as well.

setuptools' easy_install supports binary eggs, but always choked on universal builds -- it didn't know how to identify what matched the system. It would be really nice if future tools could identify and install the correct binary wheels for universal builds.

I've taken a look at PEP 427, and see this:

platform tag E.g. 'linux_x86_64', 'any'.

That looks to me like there is a built-in assumption that a wheel is either specific to a single architecture, or any -- so no way to specify that it may have multiple architectures. So I propose that platform tag allow a list of some sort:

['macosx-10_6_i386', 'macosx_10_6-x86_64']

or, I suppose to keep it simple, a single string:

'macosx_10_6_i386+macosx_10_6_x86_64' or 'macosx_10_6_i386+x86_64'

If so, then a binary wheel could be built (and discovered) that supported multiple architectures.

Then how would an installer ( pip? ) identify the right one? This is a bit tricky. PEP 425 states:

"The platform tag is simply distutils.util.get_platform() with all hyphens - and periods . replaced with underscore"

On my system right now, I have 32bitPPC + 32bit i386 universal python, running on an intel system:

In [2]: distutils.util.get_platform() Out[2]: 'macosx-10.3-i386'

So we may want to patch get_platform() to support universal builds - or add anther function or something -- you may sometimes what to know what you are running right now, and other times want to know how the current python is built. (note that while distutils can build universal binaries, it's not very smart about it -- all is does is grab the compiler and linker flags from the Makefile, which include multiple "-arch" flags.

Even if there is an easy way to query the system, that leaves the question of what to do. If a user wants to intall a package into a universal python, will the installer only accept a binary wheel with all the supported architectures? or one with only the currently running one?

Anyway, it would be nice to be able to get this stuff to "just work" with universal binaries on the Mac.

Thoughts, ideas?

-Chris

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

On Fri, May 31, 2013 at 3:38 PM, Daniel Holth <dholth@gmail.com> wrote:

Wheel already supports compound tags. Just like py2.py3-none-any a tag

Is this from PEP 425? It's still a little unclear to me exactly how those tags are to be used, but, yes, this seems to be the right direction.

py3-none-x86.ppc (with real platform names) would work. Does that make sense for Mac?

hmm, for one, I"m confused about the "Python Tag" vs. the "ABI Tag" vs. the "Platform Tag" -- I guess these all get mushed together in the filename? but a few issues anyway: it looks like the platform tag has the OS and a architecture in one, so would a universal bild on the mac be: macosx_10_3_i386.ppc or macosx_10_3_i386.macosx_10_3_ppc I'm thinking the latter, though it gets pretty wordy: A potential full name: my_package-3_2-cp27-cp27-macosx_10_3_i386.macosx_10_3_ppc.whl (Im a little confused about python version tag vs. ABI tag, in the case of a compiled extension...) But a potential confusion -- in this case, having two platform tags indicates that _both_ are present, rather than one thing that works with either. Maybe that would always be the case with platform. So if that's how the wheel is specified, we still need the other side of the process: Installing: A "universal" macpython binary needs to know that it is universal (distutils.util.get_platform() returns only the currently running version). Once that is know, the installer needs to make some choice: If a matching universal binary is found -- use it If a non-universal binary is found that matches the currently running python -- then what? install it with a warning? raise an exception? -Chris

On May 31, 2013 6:30 PM, "Chris Barker - NOAA Federal" <chris.barker@noaa.gov> wrote:

...

HI Folks,

A few of us over on the pythonmac list have been hashing out how best to support binary packages on OS-X. The binary wheel option seems very promising.

However, there is one Mac-specific issue that does not seem to be addressed:

On OS-X, binaries can be "universal" -- what this means is that they have more than one binary architecture embedded in a single file; the system selects the right one at run time. Both executables and dynamic libraries can be universal. In theory, an universal binary can currently contain up to 4 architectures:

32+64 bit PPC and 32+64bit x86

In practice, 64 bit PPC was never broadly used, and 32bit PPC is fading fast. Nevertheless, both 32 and 64 Intel systems are pretty common, and who knows what there may be in the future (you never know with Apple...)

The binary builds of Python served up on the python.org web site have supported universal builds for years -- currently there are two options: 32bit PPC+x86 or 32+64bit x86. It's not a single build, as it turns out it's hard to support both up to date x86_64 and PPC with the same compiler. In these builds, the configuration is set up so that distutils will build universal extensions that match the architectures included in the builds. This makes it pretty easy to build binary packages, and/or full app distributions (py2app) that are universal as well.

setuptools' easy_install supports binary eggs, but always choked on universal builds -- it didn't know how to identify what matched the system. It would be really nice if future tools could identify and install the correct binary wheels for universal builds.

I've taken a look at PEP 427, and see this:

platform tag E.g. 'linux_x86_64', 'any'.

That looks to me like there is a built-in assumption that a wheel is either specific to a single architecture, or any -- so no way to specify that it may have multiple architectures. So I propose that platform tag allow a list of some sort:

['macosx-10_6_i386', 'macosx_10_6-x86_64']

or, I suppose to keep it simple, a single string:

'macosx_10_6_i386+macosx_10_6_x86_64' or 'macosx_10_6_i386+x86_64'

If so, then a binary wheel could be built (and discovered) that supported multiple architectures.

Then how would an installer ( pip? ) identify the right one? This is a bit tricky. PEP 425 states:

"The platform tag is simply distutils.util.get_platform() with all hyphens - and periods . replaced with underscore"

On my system right now, I have 32bitPPC + 32bit i386 universal python, running on an intel system:

In [2]: distutils.util.get_platform() Out[2]: 'macosx-10.3-i386'

So we may want to patch get_platform() to support universal builds - or add anther function or something -- you may sometimes what to know what you are running right now, and other times want to know how the current python is built. (note that while distutils can build universal binaries, it's not very smart about it -- all is does is grab the compiler and linker flags from the Makefile, which include multiple "-arch" flags.

Even if there is an easy way to query the system, that leaves the question of what to do. If a user wants to intall a package into a universal python, will the installer only accept a binary wheel with all the supported architectures? or one with only the currently running one?

Anyway, it would be nice to be able to get this stuff to "just work" with universal binaries on the Mac.

Thoughts, ideas?

-Chris

--

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig

-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Mon, Jun 3, 2013 at 12:25 PM, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:

On Fri, May 31, 2013 at 3:38 PM, Daniel Holth <dholth@gmail.com> wrote:

...

Wheel already supports compound tags. Just like py2.py3-none-any a tag

Is this from PEP 425? It's still a little unclear to me exactly how those tags are to be used, but, yes, this seems to be the right direction.

...

py3-none-x86.ppc (with real platform names) would work. Does that make sense for Mac?

hmm, for one, I"m confused about the "Python Tag" vs. the "ABI Tag" vs. the "Platform Tag" -- I guess these all get mushed together in the filename?

but a few issues anyway: it looks like the platform tag has the OS and a architecture in one, so would a universal bild on the mac be:

macosx_10_3_i386.ppc or macosx_10_3_i386.macosx_10_3_ppc

I'm thinking the latter, though it gets pretty wordy:

A potential full name:

my_package-3_2-cp27-cp27-macosx_10_3_i386.macosx_10_3_ppc.whl

(Im a little confused about python version tag vs. ABI tag, in the case of a compiled extension...)

But a potential confusion -- in this case, having two platform tags indicates that _both_ are present, rather than one thing that works with either. Maybe that would always be the case with platform.

So if that's how the wheel is specified, we still need the other side of the process:

Installing:

A "universal" macpython binary needs to know that it is universal (distutils.util.get_platform() returns only the currently running version). Once that is know, the installer needs to make some choice:

If a matching universal binary is found -- use it If a non-universal binary is found that matches the currently running python -- then what? install it with a warning? raise an exception?

-Chris

The Python Tag is the Python implementation or language version. It lets you indicate compatibility with only CPython or Jython or PyPy even if you do not have compiled extensions: cp27, py27, py2, py3 The ABI tag is the Python ABI. The difference is that it will indicate "d"ebug "m"alloc, "u"nicode: none (also means "we don't know"), cp33dm, abi3 The platform tag is the OS+architecture. They get joined together with a dash '-'.join(python, abi, platform) For compatibility matching the tags are expanded to the Cartesian product of the string.split('.') of each of the python, abi, and platform tags. This is mostly only useful when only one part of the tag is compound. http://www.python.org/dev/peps/pep-0425/#compressed-tag-sets There used to be a rule about preferring packages with the fewest number of tags in case of a tie but I can't seem to find it now. It's probably not very important. If this doesn't work then it might be more useful to invent a tag like "macosx_10_3_universal" for some value of universal. Daniel Holth

On Tue, Jun 4, 2013 at 1:23 AM, Ronald Oussoren <ronaldoussoren@mac.com> wrote:

This isn't really a problem, distutils uses labels for the set of supported architectures as the architecture label in binary distributions (e.g. pyobjc_core-2.5-py3.4-macosx-10.8-intel.egg for an egg that supports the 'intel' set of architectures (x86_64 and i386), see _osx_support.get_platform_osx in the CPython repository for the labels that are currently used.

This is not ideal, but works good enough for now. In the long run this should likely be replaced by the compressed tags sets from PEP 425 (at the cost of a much longer file names).

I think now may be the "long run". But in any case, as we were discussing, as in theory there could be lots of possible combinations, in practice there are only two in the wild (plus the non-universal ones...), and probably only one we really need to keep supporting, so simply have a canonical name for the one or two is fine.

A much more interesting question is which binary wheels should be considered when installing a package,

indeed.

there a two clear ways to select binaries and both have there usecases. Let's say you're running on OSX 10.8 with a CPU that supports x86_64 and a Python installation with support for x86_64, i386 and ppc and compiled with support for OSX 10.4 or later ("MACOSX_DEPLOYMENT_TARGET=10.4"). When installing a package you can pick either:

1) a binary wheel that supports this machine (a wheel compiled with deployment target 10.8 with only x86_64 support)

can you link a newer deployment target-built lib with an older deployment-target executable? I guess so...

2) a binary wheel that supports the same systems as the Python installations

The former is good enough when you only want to run code on your machine, the latter can be needed when you want to deploy the installed binaries to another machine (for example because you are using py2app to create an application bundle that will be installed elsewhere).

I would say that we should clearly _prefer_ a wheel that fully supports the pyton the installer is used with. But I'm a bit ambivalent about what to do if there is no such wheel found, but there is one that supports the currently running architecture. I"m inclined to say don't install it: - We build the universal binaries to support people that don't want to know or care about such details. - We want to support universal binary wheels for those folks. - The primary reason to build binary wheels at all is to support non-native architectures -- it's a whole lot easier to simply build for your current machine, and homebrew and macports, and ??? support that already. All that being said, I sometimes think that we should simply have a 32 bit and 64 bit binary out there, and forget all this universal stuff! It works OK for Windows....

...

On my system right now, I have 32bitPPC + 32bit i386 universal python, running on an intel system:

In [2]: distutils.util.get_platform() Out[2]: 'macosx-10.3-i386'

On my machine this returns the correct value: 'macosx-10.8-intel' (binary supports 10.8 or later and the i386 and x86_64 architectures). I do know this used to return wrong values on some version of OSX when using the Apple installation of Python because of the way they build Python.

This isn't an Apple build -- I'm pretty sure it's the one from Python.org...though looking now it's only 2.7.2 -- I guess I haven't upgraded in a while! now with 2.7.5: In [3]: distutils.util.get_platform() Out[3]: 'macosx-10.3-fat' I guess "fat" means 32bit ppc + intel, deployment target 10.3 ? So we may want to update distutils.util.get_platform to do a PEP 425 name, but we've got something that could work now. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Tue, Jun 4, 2013 at 10:06 AM, Ronald Oussoren <ronaldoussoren@mac.com> wrote:

...

...

This is not ideal, but works good enough for now. In the long run this should likely be replaced by the compressed tags sets from PEP 425 (at the cost of a much longer file names).

I think now may be the "long run".

Maybe, at least for 3.4 (assuming that wheel support gets in the stdlib by then).

yup.

I'd currently prefer to only install binaries that exactly match the python installation, that's the most conservative solution, requires no changes to installation tools and is easier to reason about.

we've got a consensus of two now -- good enough for me ;-)

...

All that being said, I sometimes think that we should simply have a 32 bit and 64 bit binary out there, and forget all this universal stuff! It works OK for Windows....

For some definition of OK ;-), the registry can be fun when you're running 32 binaries on a 64-bit windows. A clear advantage of universal binaries is that you can have one set of binaries and don't have to pick which architecture gets the most obvious installation directory (e.g. /lib vs. /lib64 on a lot of Linux systems).

yeah -- I do like them, when they work...

...

So we may want to update distutils.util.get_platform to do a PEP 425 name, but we've got something that could work now.

That can wait until the new packaging stuff (wheels, metadata 2.0, distlib, ...) gets merged, and even then it might be nicer to keep 'fat' and 'intel' as short names.

it would be best for any change to get merged around the same time as the packaging stuff -- so there is only one versin out there. But I agree, the short names are nice -- if the only place anyone would get the names is distutils, then we're fine. This is all closer that I thought was -- nice! -Chris

Ronald

-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov