PEP425 - Wheel Binary Package Compatibility
Hi there, Following up on https://bitbucket.org/pypa/wheel/issue/124/glibc-incompatibility. How should we deal with incompatibility of dynamically linked libraries? Doing "pip wheel numpy" on a Linux 64bit machine results in , which is linked dynamically against the GLIBC version installed on the build machine. So should the wheel be shipped with GLIBC, or the GLIBC version be specified in the wheel name (means to build a new wheel for each GLIBC version?). Also, maybe this is not the only library it is dynamically linked against? Why does that work on MacOS, btw? Are all library versions fixed there for one version of OSX? Thanks for putting some light into this issue. Best regards, Matthias
On 28 October 2014 10:13, Matthias Hafner <hafner87@gmail.com> wrote: (in reverse order, easiest to hardest :-))
Why does that work on MacOS, btw? Are all library versions fixed there for one version of OSX?
I believe (I'm not 100% sure as I'm a Windows user not an OSX user) that PyPI only supports binaries compatible with the official python.org binaries of Python. So library version decisions are moot. This means that homebrew or "the other ones" (sorry :-)) users can't avail themselves of PyPI-hosted binaries yet, until the wider library compatibility issues are resolved.
So should the wheel be shipped with GLIBC, or the GLIBC version be specified in the wheel name (means to build a new wheel for each GLIBC version?). Also, maybe this is not the only library it is dynamically linked against?
Whatever solution is devised will need to be encapsulated in a set of compatibility tags (see PEP 425) that capture what it means to be "compatible". The wheels can then be uploaded with the correct tags, and pip and other tools will automatically select the right one (or report an error if no suitable binary exists and source is not available).
How should we deal with incompatibility of dynamically linked libraries? Doing "pip wheel numpy" on a Linux 64bit machine results in , which is linked dynamically against the GLIBC version installed on the build machine.
This is the big question. On Windows, we only really have to deal with 32-bit or 64-bit binaries, using the C runtime that the python.org binaries use. So the architecture is sufficient and the ABI flag is unused. On OSX, as I say, PyPI has in effect artificially set up a similar situation, by not considering anything other than python.org build compatible binaries. On Linux, nobody has really looked at the problem (beyond acknowledging that it exists). We need a group of Linux experts with an understanding of how all the various aspects of distribution, kernel, libc, etc, work together to form a "compatible environment" for binaries. As Linux has a strong culture of only building from source or using distro packages, there doesn't seem to me (as an outsider) to be much momentum around solving this. Sorry I can't offer any more help or clarification than this. Hopefully someone from the Linux side of things can offer some views on how this can be moved forwards. Paul
2014-10-28 15:04 GMT+01:00 Paul Moore <p.f.moore@gmail.com>:
On 28 October 2014 10:13, Matthias Hafner <hafner87@gmail.com> wrote:
(in reverse order, easiest to hardest :-))
Why does that work on MacOS, btw? Are all library versions fixed there for one version of OSX?
I believe (I'm not 100% sure as I'm a Windows user not an OSX user) that PyPI only supports binaries compatible with the official python.org binaries of Python. So library version decisions are moot. This means that homebrew or "the other ones" (sorry :-)) users can't avail themselves of PyPI-hosted binaries yet, until the wider library compatibility issues are resolved.
Homebrew-installed Python and the system Python that comes by default on OSX are binary compatible with wheels generated with Python from the official OSX installer from python.org. In current versions of pip you "just" have to rename the wheel filenames to get them picked up from PyPI on those platforms. This explains the strange names of the OSX wheels for numpy for instance: numpy-1.9.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl numpy-1.9.0-cp33-cp33m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl numpy-1.9.0-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.whl https://pypi.python.org/pypi/numpy macosx_10_9_x86_64 is the platform tag used by Homebrew-installed Python while macosx_10_6_intel is the platform tag of the python.org Python. Off-course if your python extension link to non-system libraries, you have to embed them in the wheel, for instance using: https://pypi.python.org/pypi/delocate -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel
Here is a test matrix that checks binary compatibility of OSX wheels for many projects of the SciPy stack: https://github.com/MacPython/scipy-stack-osx-testing -- Olivier
2014-10-28 16:22 GMT+01:00 Olivier Grisel <olivier.grisel@ensta.org>:
Here is a test matrix that checks binary compatibility of OSX wheels for many projects of the SciPy stack:
And here are the results: https://travis-ci.org/MacPython/scipy-stack-osx-testing -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel
In article <CAFvE7K4U7L4+8n3U72eHER4fa8m-fCANv0q75BhM1AaLfmA2nw@mail.gmail.com>, Olivier Grisel <olivier.grisel@ensta.org> wrote:
Homebrew-installed Python and the system Python that comes by default on OSX are binary compatible with wheels generated with Python from the official OSX installer from python.org.
In current versions of pip you "just" have to rename the wheel filenames to get them picked up from PyPI on those platforms.
This explains the strange names of the OSX wheels for numpy for instance:
numpy-1.9.0-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.w hl numpy-1.9.0-cp33-cp33m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64. whl numpy-1.9.0-cp34-cp34m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64. whl
https://pypi.python.org/pypi/numpy
macosx_10_9_x86_64 is the platform tag used by Homebrew-installed Python while macosx_10_6_intel is the platform tag of the python.org Python.
To expand on this, the platform tag is generated by Python's Distutils and wheels use and expand on the pattern established by it. For OS X, the "10_x" part is based on the minimum OS X deployment target of the Python build, normally set by the compiler -mmacosx-version-min argument or the MACOSX_DEPLOYMENT_TARGET env variable at configure time (these are standard options supported by the Apple-supplied buildtool chains). That value is saved and used by Distutils in subsequent extension module builds, although you can override it to a higher (not lower) value for a particular build. The arch values in the tag also come from the configured universal architectures. Any Python configured the same way (wrt MACOSX_DEPLOYMENT_TARGET and archs) will supply the same platform tag. So there's nothing special about Homebrew or MacPorts or python.org or the Apple-supplied system Pythons; it is assumed that any identically configured Python will provide a compatible ABI for C extension modules. We also assume that Apple will provide backward compatibility on newer releases of OS X, such that, for example, an extension module compiled for 10.6 will run just fine on later OS X releases *and* will run with a Python configured for a later OS X release, assuming there is a non-zero intersection of universal archs between the interpreter and the extension module. In practice, those assumptions seems to have worked fairly well on OS X, starting from the beginning of universal arch support and at least up to now although they may break down in the future. It's still not perfect. For example, the platform string doesn't take into account the Python 2 --enable-unicode=ucs2/ucs4 build option; but ucs4 builds aren't widely used on OS X and Python 3 no longer has this problem. (For C++ extension modules, there also may be issues with Apple's change in default c++ libraries in OS X 10.9.) On Linux systems, it can be dangerous to make similar simplifying assumptions, with many more variables in play, much less reflected in the platform string. -- Ned Deily, nad@acm.org
On Tue, Oct 28, 2014 at 10:13 AM, Matthias Hafner <hafner87@gmail.com> wrote:
Hi there,
Following up on https://bitbucket.org/pypa/wheel/issue/124/glibc-incompatibility.
How should we deal with incompatibility of dynamically linked libraries? Doing "pip wheel numpy" on a Linux 64bit machine results in , which is linked dynamically against the GLIBC version installed on the build machine.
Wheel by itself simply cannot deal with this, you need something more. Practically speaking, there is no such a thing as ABI on Linux: even if you somehow managed to deal with glibc, you would then need to deal with fortran ABI, then with C++ ABI, etc... Dealing with this at the python level is simply intractable. That needs to be handled at the distribution level, or within a "sumo distribution" like Anaconda from Continuum or Canopy from Enthought (disclaimer, I work for Enthought), which makes the problem more tractable by picking up an ABI and stick with it for the full set of packages. In theory, you can try to emulate this by building on the oldest version of each compiled library, and hope that your target platforms are ABI compatible with this, but that's hard on Linux. David
So should the wheel be shipped with GLIBC, or the GLIBC version be specified in the wheel name (means to build a new wheel for each GLIBC version?). Also, maybe this is not the only library it is dynamically linked against?
Why does that work on MacOS, btw? Are all library versions fixed there for one version of OSX?
Yes. Same for windows. The native toolchains there are fairly easy to configure for targetting a specific minimum version. David
Thanks for putting some light into this issue.
Best regards, Matthias
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 29 Oct 2014 01:02, "David Cournapeau" <cournape@gmail.com> wrote:
Practically speaking, there is no such a thing as ABI on Linux: even if
you somehow managed to deal with glibc, you would then need to deal with fortran ABI, then with C++ ABI, etc... Dealing with this at the python level is simply intractable. That needs to be handled at the distribution level, or within a "sumo distribution" like Anaconda from Continuum or Canopy from Enthought (disclaimer, I work for Enthought), which makes the problem more tractable by picking up an ABI and stick with it for the full set of packages. This is the kind of direction we're also exploring for Fedora & EPEL: setting up a distro specific devpi instance so we can automatically publish distro compatible wheel files, as well as separating the distro level licencing and preliminary security review step from the step of repackaging in a language independent format. (I need to set up a curated PyPI mirror for work anyway, and since I work at Red Hat, and the Fedora community are open to the idea, we're working on it upstream rather than inside the firewall) This initial experiment with Python is a pilot - if we can get it to work in a way that seems sustainable, we'll look to offer it as a supported service in the Fedora 22 time frame, as well as opening things up for folks to pitch other language specific stacks. Cheers, Nick.
This is the kind of direction we're also exploring for Fedora & EPEL: setting up a distro specific devpi instance so we can automatically publish distro compatible wheel files, as well as separating the distro level licencing and preliminary security review step from the step of repackaging in a language independent format. (I need to set up a curated PyPI mirror for work anyway, and since I work at Red Hat, and the Fedora community are open to the idea, we're working on it upstream rather than inside the firewall)
Btw, I've been doing this at my company, that is maintaining separate wheel indexes per platform we need (in our case, cent5 and cent6). One problem here is that it's possible for pip download caches to get "corrupted" with the wrong distributions (i.e. the incompatible ones), since the distributions are not distinguishable by name Would Fedora really consider releasing a public service where there fedora release is not represented in the distribution name? or would you be wanting to get more tagging spec'd out in Wheel 2.0 first? P.S. In looking at the PEP425/427 specs again, I'm slightly confused about the "build tag". It's barely covered, but I can imagine using that possibly to distinguish dists, assuming pip support was added, alhough it's supposed to start with a number, which doesn't seem to fit the use case we'd want.
On Oct 29, 2014, at 2:36 PM, Marcus Smith <qwcode@gmail.com> wrote:
This is the kind of direction we're also exploring for Fedora & EPEL: setting up a distro specific devpi instance so we can automatically publish distro compatible wheel files, as well as separating the distro level licencing and preliminary security review step from the step of repackaging in a language independent format. (I need to set up a curated PyPI mirror for work anyway, and since I work at Red Hat, and the Fedora community are open to the idea, we're working on it upstream rather than inside the firewall)
Btw, I've been doing this at my company, that is maintaining separate wheel indexes per platform we need (in our case, cent5 and cent6). One problem here is that it's possible for pip download caches to get "corrupted" with the wrong distributions (i.e. the incompatible ones), since the distributions are not distinguishable by name Would Fedora really consider releasing a public service where there fedora release is not represented in the distribution name? or would you be wanting to get more tagging spec'd out in Wheel 2.0 first?
P.S. In looking at the PEP425/427 specs again, I'm slightly confused about the "build tag". It's barely covered, but I can imagine using that possibly to distinguish dists, assuming pip support was added, alhough it's supposed to start with a number, which doesn't seem to fit the use case we'd want.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
It’s a build number, it’s used incase you need to rebuild a Wheel using the same source files. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
I've always thought people might like to have a custom platform tag or otherwise customize the supported list in a configuration file loaded by pep425tags. Then you could create wheels called somewheel-4.0-py33-none-30caa8a209d6.whl (choose your own random string). Only a pip with a matching config would pay any attention to those wheels. That way we could help people manage their internal deployments without falsely advertising that we've solved the binary incompatibility feature baked into every Linux distribution. On Wed, Oct 29, 2014 at 3:52 PM, Donald Stufft <donald@stufft.io> wrote:
On Oct 29, 2014, at 2:36 PM, Marcus Smith <qwcode@gmail.com> wrote:
This is the kind of direction we're also exploring for Fedora & EPEL: setting up a distro specific devpi instance so we can automatically publish distro compatible wheel files, as well as separating the distro level licencing and preliminary security review step from the step of repackaging in a language independent format. (I need to set up a curated PyPI mirror for work anyway, and since I work at Red Hat, and the Fedora community are open to the idea, we're working on it upstream rather than inside the firewall)
Btw, I've been doing this at my company, that is maintaining separate wheel indexes per platform we need (in our case, cent5 and cent6). One problem here is that it's possible for pip download caches to get "corrupted" with the wrong distributions (i.e. the incompatible ones), since the distributions are not distinguishable by name Would Fedora really consider releasing a public service where there fedora release is not represented in the distribution name? or would you be wanting to get more tagging spec'd out in Wheel 2.0 first?
P.S. In looking at the PEP425/427 specs again, I'm slightly confused about the "build tag". It's barely covered, but I can imagine using that possibly to distinguish dists, assuming pip support was added, alhough it's supposed to start with a number, which doesn't seem to fit the use case we'd want.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
It’s a build number, it’s used incase you need to rebuild a Wheel using the same source files.
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
yes, I'm partial to a solution like this prior to wheel 2.0 (that I imagine would support additional/custom tags) On Wed, Oct 29, 2014 at 1:46 PM, Daniel Holth <dholth@gmail.com> wrote:
I've always thought people might like to have a custom platform tag or otherwise customize the supported list in a configuration file loaded by pep425tags. Then you could create wheels called somewheel-4.0-py33-none-30caa8a209d6.whl (choose your own random string). Only a pip with a matching config would pay any attention to those wheels. That way we could help people manage their internal deployments without falsely advertising that we've solved the binary incompatibility feature baked into every Linux distribution.
On Wed, Oct 29, 2014 at 3:52 PM, Donald Stufft <donald@stufft.io> wrote:
On Oct 29, 2014, at 2:36 PM, Marcus Smith <qwcode@gmail.com> wrote:
This is the kind of direction we're also exploring for Fedora & EPEL: setting up a distro specific devpi instance so we can automatically
distro compatible wheel files, as well as separating the distro level licencing and preliminary security review step from the step of repackaging in a language independent format. (I need to set up a curated PyPI mirror for work anyway, and since I work at Red Hat, and the Fedora community are open to the idea, we're working on it upstream rather than inside the firewall)
Btw, I've been doing this at my company, that is maintaining separate wheel indexes per platform we need (in our case, cent5 and cent6). One problem here is that it's possible for pip download caches to get "corrupted" with the wrong distributions (i.e. the incompatible ones), since the distributions are not distinguishable by name Would Fedora really consider releasing a public service where there fedora release is not represented in the distribution name? or would you be wanting to get more tagging spec'd out in Wheel 2.0 first?
P.S. In looking at the PEP425/427 specs again, I'm slightly confused about the "build tag". It's barely covered, but I can imagine using that
to distinguish dists, assuming pip support was added, alhough it's supposed to start with a number, which doesn't seem to fit the use case we'd want.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
It’s a build number, it’s used incase you need to rebuild a Wheel using
publish possibly the
same source files.
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 30 Oct 2014 07:20, "Marcus Smith" <qwcode@gmail.com> wrote:
yes, I'm partial to a solution like this prior to wheel 2.0 (that I
imagine would support additional/custom tags) +1 for being able to add additional custom platform tags in the file naming convention from me as well. As Marcus noted earlier, even if you set up distro specific indexes currently, there's nothing built into the tooling to keep you from trying to install (e.g.) a Fedora 21 wheel on RHEL or CentOS 5 (which is highly unlikely to work, given that the core ABIs in RHEL/CentOS 5 are 7 or 8 years old at this point). We'd be highly unlikely to flip the switch from "experimental service, use at your own risk" to "fully supported Fedora feature" while that's still the case. With arbitrary platform tags, we could inject that into the wheel filenames as part of the build process, and then again when invoking pip. That opens things up for us to figure out how to best flag compatibility on the distro side, without committing to a specific approach upstream (not yet, anyway). An alternative approach would be to add an "additional wheel suffix" setting for pip that allowed us to have names with endings like ".fc21.whl" or ".el7.whl" recognised as valid wheel files. I'm not that worried about the exact details though - the main feature I'd like is the ability to create wheel files that pip will ignore by default, but will accept if I specifically tell it what to look for. Cheers, Nick.
On Wed, Oct 29, 2014 at 1:46 PM, Daniel Holth <dholth@gmail.com> wrote:
I've always thought people might like to have a custom platform tag or otherwise customize the supported list in a configuration file loaded by pep425tags. Then you could create wheels called somewheel-4.0-py33-none-30caa8a209d6.whl (choose your own random string). Only a pip with a matching config would pay any attention to those wheels. That way we could help people manage their internal deployments without falsely advertising that we've solved the binary incompatibility feature baked into every Linux distribution.
On Wed, Oct 29, 2014 at 3:52 PM, Donald Stufft <donald@stufft.io> wrote:
On Oct 29, 2014, at 2:36 PM, Marcus Smith <qwcode@gmail.com> wrote:
This is the kind of direction we're also exploring for Fedora & EPEL: setting up a distro specific devpi instance so we can automatically
distro compatible wheel files, as well as separating the distro level licencing and preliminary security review step from the step of repackaging in a language independent format. (I need to set up a curated PyPI mirror for work anyway, and since I work at Red Hat, and the Fedora community are open to the idea, we're working on it upstream rather than inside the firewall)
Btw, I've been doing this at my company, that is maintaining separate wheel indexes per platform we need (in our case, cent5 and cent6). One problem here is that it's possible for pip download caches to get "corrupted" with the wrong distributions (i.e. the incompatible ones), since the distributions are not distinguishable by name Would Fedora really consider releasing a public service where there fedora release is not represented in the distribution name? or would you be wanting to get more tagging spec'd out in Wheel 2.0 first?
P.S. In looking at the PEP425/427 specs again, I'm slightly confused about the "build tag". It's barely covered, but I can imagine using that
publish possibly
to distinguish dists, assuming pip support was added, alhough it's supposed to start with a number, which doesn't seem to fit the use case we'd want.
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
It’s a build number, it’s used incase you need to rebuild a Wheel using the same source files.
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Oct 29, 2014, at 7:57 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 30 Oct 2014 07:20, "Marcus Smith" <qwcode@gmail.com <mailto:qwcode@gmail.com>> wrote:
yes, I'm partial to a solution like this prior to wheel 2.0 (that I imagine would support additional/custom tags)
+1 for being able to add additional custom platform tags in the file naming convention from me as well. As Marcus noted earlier, even if you set up distro specific indexes currently, there's nothing built into the tooling to keep you from trying to install (e.g.) a Fedora 21 wheel on RHEL or CentOS 5 (which is highly unlikely to work, given that the core ABIs in RHEL/CentOS 5 are 7 or 8 years old at this point).
We'd be highly unlikely to flip the switch from "experimental service, use at your own risk" to "fully supported Fedora feature" while that's still the case.
With arbitrary platform tags, we could inject that into the wheel filenames as part of the build process, and then again when invoking pip.
That opens things up for us to figure out how to best flag compatibility on the distro side, without committing to a specific approach upstream (not yet, anyway).
An alternative approach would be to add an "additional wheel suffix" setting for pip that allowed us to have names with endings like ".fc21.whl" or ".el7.whl" recognised as valid wheel files.
I'm not that worried about the exact details though - the main feature I'd like is the ability to create wheel files that pip will ignore by default, but will accept if I specifically tell it what to look for.
Do we plan for this to be allowed to upload to PyPI proper? These custom tags? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
It could get quite messy, without consensus. I figured custom tags would be a more local thing. On 30 October 2014 11:17, Donald Stufft <donald@stufft.io> wrote:
On Oct 29, 2014, at 7:57 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 30 Oct 2014 07:20, "Marcus Smith" <qwcode@gmail.com> wrote:
yes, I'm partial to a solution like this prior to wheel 2.0 (that I
imagine would support additional/custom tags)
+1 for being able to add additional custom platform tags in the file naming convention from me as well. As Marcus noted earlier, even if you set up distro specific indexes currently, there's nothing built into the tooling to keep you from trying to install (e.g.) a Fedora 21 wheel on RHEL or CentOS 5 (which is highly unlikely to work, given that the core ABIs in RHEL/CentOS 5 are 7 or 8 years old at this point).
We'd be highly unlikely to flip the switch from "experimental service, use at your own risk" to "fully supported Fedora feature" while that's still the case.
With arbitrary platform tags, we could inject that into the wheel filenames as part of the build process, and then again when invoking pip.
That opens things up for us to figure out how to best flag compatibility on the distro side, without committing to a specific approach upstream (not yet, anyway).
An alternative approach would be to add an "additional wheel suffix" setting for pip that allowed us to have names with endings like ".fc21.whl" or ".el7.whl" recognised as valid wheel files.
I'm not that worried about the exact details though - the main feature I'd like is the ability to create wheel files that pip will ignore by default, but will accept if I specifically tell it what to look for.
Do we plan for this to be allowed to upload to PyPI proper? These custom tags?
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 30 October 2014 10:29, Richard Jones <richard@python.org> wrote:
It could get quite messy, without consensus. I figured custom tags would be a more local thing.
Yep, that's what I would suggest - keep it off PyPI for now, but provide some way to actually try the concept in real environments. That way future standardisation can be based on experience rather than trying to guess potential use cases in advance. Cheers, Nick.
On 30 October 2014 11:17, Donald Stufft <donald@stufft.io> wrote:
On Oct 29, 2014, at 7:57 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 30 Oct 2014 07:20, "Marcus Smith" <qwcode@gmail.com> wrote:
yes, I'm partial to a solution like this prior to wheel 2.0 (that I imagine would support additional/custom tags)
+1 for being able to add additional custom platform tags in the file naming convention from me as well. As Marcus noted earlier, even if you set up distro specific indexes currently, there's nothing built into the tooling to keep you from trying to install (e.g.) a Fedora 21 wheel on RHEL or CentOS 5 (which is highly unlikely to work, given that the core ABIs in RHEL/CentOS 5 are 7 or 8 years old at this point).
We'd be highly unlikely to flip the switch from "experimental service, use at your own risk" to "fully supported Fedora feature" while that's still the case.
With arbitrary platform tags, we could inject that into the wheel filenames as part of the build process, and then again when invoking pip.
That opens things up for us to figure out how to best flag compatibility on the distro side, without committing to a specific approach upstream (not yet, anyway).
An alternative approach would be to add an "additional wheel suffix" setting for pip that allowed us to have names with endings like ".fc21.whl" or ".el7.whl" recognised as valid wheel files.
I'm not that worried about the exact details though - the main feature I'd like is the ability to create wheel files that pip will ignore by default, but will accept if I specifically tell it what to look for.
Do we plan for this to be allowed to upload to PyPI proper? These custom tags?
--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Wed, Oct 29, 2014 at 8:17 PM, Donald Stufft <donald@stufft.io> wrote:
On Oct 29, 2014, at 7:57 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 30 Oct 2014 07:20, "Marcus Smith" <qwcode@gmail.com> wrote:
yes, I'm partial to a solution like this prior to wheel 2.0 (that I imagine would support additional/custom tags)
+1 for being able to add additional custom platform tags in the file naming convention from me as well. As Marcus noted earlier, even if you set up distro specific indexes currently, there's nothing built into the tooling to keep you from trying to install (e.g.) a Fedora 21 wheel on RHEL or CentOS 5 (which is highly unlikely to work, given that the core ABIs in RHEL/CentOS 5 are 7 or 8 years old at this point).
We'd be highly unlikely to flip the switch from "experimental service, use at your own risk" to "fully supported Fedora feature" while that's still the case.
With arbitrary platform tags, we could inject that into the wheel filenames as part of the build process, and then again when invoking pip.
That opens things up for us to figure out how to best flag compatibility on the distro side, without committing to a specific approach upstream (not yet, anyway).
An alternative approach would be to add an "additional wheel suffix" setting for pip that allowed us to have names with endings like ".fc21.whl" or ".el7.whl" recognised as valid wheel files.
I'm not that worried about the exact details though - the main feature I'd like is the ability to create wheel files that pip will ignore by default, but will accept if I specifically tell it what to look for.
Do we plan for this to be allowed to upload to PyPI proper? These custom tags?
No
On 28.10.2014 11:13, Matthias Hafner wrote:
Hi there,
Following up on https://bitbucket.org/pypa/wheel/issue/124/glibc-incompatibility.
How should we deal with incompatibility of dynamically linked libraries? Doing "pip wheel numpy" on a Linux 64bit machine results in , which is linked dynamically against the GLIBC version installed on the build machine.
So should the wheel be shipped with GLIBC, or the GLIBC version be specified in the wheel name (means to build a new wheel for each GLIBC version?). Also, maybe this is not the only library it is dynamically linked against?
Why does that work on MacOS, btw? Are all library versions fixed there for one version of OSX?
Thanks for putting some light into this issue.
Since OSes typically ship with older libc versions, your best bet is to build the package on a OS release that's a few years older than the latest version, e.g. for Ubuntu you'd use 12.04 or even 10.04 instead of 14.04. On Linux, you can check the min required GLIBC version by looking at the nm output of the binaries. They will have a "@@GLIBC_x.x.x" modifier attached to the symbols loaded from the GLIBC. The platform module has a helper function which does this for you: https://docs.python.org/2.7/library/platform.html#platform.libc_ver That way you can avoid many incompatibilities with libc versions. This works on Mac OS X and other Unix-based systems as well. You may run into library version info problems, though: http://stackoverflow.com/questions/137773/what-does-the-no-version-informati... A way around this is to build on systems that don't yet include this version information. The alternative is static linking, but this is often not possible or desired. On Windows you have to use the libc versions that were used to compile Python itself, so things are easier. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Oct 28 2014)
Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
2014-10-24: Released eGenix pyOpenSSL 0.13.5 ... http://egenix.com/go63 ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/
participants (11)
-
Daniel Holth -
David Cournapeau -
Donald Stufft -
M.-A. Lemburg -
Marcus Smith -
Matthias Hafner -
Ned Deily -
Nick Coghlan -
Olivier Grisel -
Paul Moore -
Richard Jones