Plans for binary wheels, and PyPi and OS-X
Someone on another list indicated that pip installing binary wheels from PyPi will ONLY work for Windows. Is that the case? I think it's desperately needed for OS-X as well. Linux is so diverse that I can't imagine it being useful, but OS-X has only so many versions, and the python.org OS-X binaries are very clear in their requirements -- it would be very useful if folks could easily get binary wheels for OS-X -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Oct 18, 2013, at 12:58 PM, Chris Barker <chris.barker@noaa.gov> wrote:
Someone on another list indicated that pip installing binary wheels from PyPi will ONLY work for Windows.
Is that the case? I think it's desperately needed for OS-X as well.
For the time being yes, it'll change in the future.
Linux is so diverse that I can't imagine it being useful, but OS-X has only so many versions, and the python.org OS-X binaries are very clear in their requirements -- it would be very useful if folks could easily get binary wheels for OS-X
-Chris
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 19 Oct 2013 04:59, "Chris Barker" <chris.barker@noaa.gov> wrote:
Someone on another list indicated that pip installing binary wheels from PyPi will ONLY work for Windows.
Is that the case? I think it's desperately needed for OS-X as well.
Linux is so diverse that I can't imagine it being useful, but OS-X has only so many versions, and the python.org OS-X binaries are very clear in their requirements -- it would be very useful if folks could easily get binary wheels for OS-X
We do plan to support it, but the pip devs uncovered a hole in the current wheel spec that means it generates the same filename on *nix systems for wheels that need to have different names for the download side of things to work properly - hence the current Windows-only restriction. Once ensurepip has landed in Python 3.4 and pip 1.5 is released, we should be able to get back to updating the various metadata specs, with the aim of getting cross-platform wheel support in pip 1.6 :) Cheers, Nick.
-Chris
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Fri, Oct 18, 2013 at 6:22 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- it would be very useful if folks could easily get binary wheels for OS-X
We do plan to support it, but the pip devs uncovered a hole in the current wheel spec that means it generates the same filename on *nix systems for wheels that need to have different names for the download side of things to work properly
THanks -- but really? don't OS-X wheels get: macosx_10_6_intel or some such tacked on? Where does that go wrong?
Once ensurepip has landed in Python 3.4 and pip 1.5 is released, we should be able to get back to updating the various metadata specs, with the aim of getting cross-platform wheel support in pip 1.6 :)
Getting there... thanks, -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Oct 21, 2013, at 1:02 PM, Chris Barker <chris.barker@noaa.gov> wrote:
On Fri, Oct 18, 2013 at 6:22 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- it would be very useful if folks could easily get binary wheels for OS-X
We do plan to support it, but the pip devs uncovered a hole in the current wheel spec that means it generates the same filename on *nix systems for wheels that need to have different names for the download side of things to work properly
THanks -- but really? don't OS-X wheels get:
macosx_10_6_intel
or some such tacked on? Where does that go wrong?
Homebrew, Mac Ports, Fink. That would work OK if nobody ever installed things that the system didn't provide.
Once ensurepip has landed in Python 3.4 and pip 1.5 is released, we should be able to get back to updating the various metadata specs, with the aim of getting cross-platform wheel support in pip 1.6 :)
Getting there... thanks,
-Chris
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Mon, Oct 21, 2013 at 11:52 AM, Donald Stufft <donald@stufft.io> wrote:
Thanks -- but really? don't OS-X wheels get:
macosx_10_6_intel
or some such tacked on? Where does that go wrong?
Homebrew, Mac Ports, Fink. That would work OK if nobody ever installed things that the system didn't provide.
OK -- yes, that will NEVER work. It's worse than the Linux situation. But then, it's the same everywhere -- if someone builds a binary wheel for Windows that depends on some non-standard dll, or is built against a weirdly custom-built Python, it won't work either. It's been more or less a consensus in the python-mac community that we seek to provide binaries for the Python.org pythons, and that they shouldn't depend on non-standard external libs -- just like on Windows. Major hard-to-build packages have been doing this for years: wxPython numpy, scipy, matplotlib But these installers often don't work with virtualenv, and can't be discovered or installed pip or easy_install. So I think it would be VERY useful if we established this standard for PyPi and binary wheels. macports, fink, and homebrew have been doing their own thing for ages, and can continue to do so -- they HAVE package management built in, just like the linux distros. If they want to do wheels, they will need to make sure that the neccesary info is in the platform-tag. On my python.org build: 'macosx-10.6-i386' so they should patch their python to return something like: 'macosx-macports-10.6-i386' or just: 'macports-10.6-i386' and probably a macports version, rather than "10.6". However, the _point_ of macports, homebrew, etc, is that your stuff is all custom compiled for your system the way you have configured it -- so binary wheels really don't make sense. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 23 Oct 2013 05:42, "Chris Barker" <chris.barker@noaa.gov> wrote:
On Mon, Oct 21, 2013 at 11:52 AM, Donald Stufft <donald@stufft.io> wrote:
Thanks -- but really? don't OS-X wheels get:
macosx_10_6_intel
or some such tacked on? Where does that go wrong?
Homebrew, Mac Ports, Fink. That would work OK if nobody ever installed
things
that the system didn't provide.
OK -- yes, that will NEVER work. It's worse than the Linux situation.
But then, it's the same everywhere -- if someone builds a binary wheel for Windows that depends on some non-standard dll, or is built against a weirdly custom-built Python, it won't work either.
It's been more or less a consensus in the python-mac community that we seek to provide binaries for the Python.org pythons, and that they shouldn't depend on non-standard external libs -- just like on Windows. Major hard-to-build packages have been doing this for years:
wxPython numpy, scipy, matplotlib
But these installers often don't work with virtualenv, and can't be discovered or installed pip or easy_install.
So I think it would be VERY useful if we established this standard for PyPi and binary wheels.
macports, fink, and homebrew have been doing their own thing for ages, and can continue to do so -- they HAVE package management built in, just like the linux distros. If they want to do wheels, they will need to make sure that the neccesary info is in the platform-tag. On my python.org build:
'macosx-10.6-i386'
so they should patch their python to return something like:
'macosx-macports-10.6-i386'
or just:
'macports-10.6-i386'
and probably a macports version, rather than "10.6".
However, the _point_ of macports, homebrew, etc, is that your stuff is all custom compiled for your system the way you have configured it -- so binary wheels really don't make sense.
PEP 453 has had most of my attention lately, but my tentative thought has been to introduce a relatively freeform "variant" field to the wheel spec. Windows and Mac OS X would then default to an empty variant, while other *nix systems would require a nominated variant That would then cover things like SSE builds on Windows, alternative sources on Mac OS X, etc. However, worrying about that is a fair way down the todo list until ensurepip and the CPython doc updates for PEP 453 are done, along with everything else that has a 3.4b1 deadline :) Cheers, Nick.
-Chris
--
Christopher Barker, Ph.D. Oceanographer
Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception
Chris.Barker@noaa.gov _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Tue, Oct 22, 2013 at 1:19 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
PEP 453 has had most of my attention lately, but my tentative thought has been to introduce a relatively freeform "variant" field to the wheel spec.
Windows and Mac OS X would then default to an empty variant, while other *nix systems would require a nominated variant
That would then cover things like SSE builds on Windows, alternative sources on Mac OS X, etc.
sounds good.
However, worrying about that is a fair way down the todo list until ensurepip and the CPython doc updates for PEP 453 are done, along with everything else that has a 3.4b1 deadline :)
Fair enough, but enabling binary wheels for OS-X (the pyton.org build, NOT macports and friends) would be great, and shouldn't take much work, yes? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 21 Oct, 2013, at 20:52, Donald Stufft <donald@stufft.io> wrote:
On Oct 21, 2013, at 1:02 PM, Chris Barker <chris.barker@noaa.gov> wrote:
On Fri, Oct 18, 2013 at 6:22 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- it would be very useful if folks could easily get binary wheels for OS-X
We do plan to support it, but the pip devs uncovered a hole in the current wheel spec that means it generates the same filename on *nix systems for wheels that need to have different names for the download side of things to work properly
THanks -- but really? don't OS-X wheels get:
macosx_10_6_intel
or some such tacked on? Where does that go wrong?
Homebrew, Mac Ports, Fink. That would work OK if nobody ever installed things that the system didn't provide.
I don't understand this, what would break? Binary wheels that reference shared libraries that aren't included in the wheel (or a default system install) won't work, but that's also true on Windows. What makes OSX more fun[1] than Linux is including shared libraries in the binary archive, unless you are careful with linking you'll end up with binaries that can only be installed in 1 filesystem location (that is, don't work in virtualenvs). Ronald [1] for a fairly twisted definition of fun ;-)
On 31 October 2013 23:38, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 21 Oct, 2013, at 20:52, Donald Stufft <donald@stufft.io> wrote:
On Oct 21, 2013, at 1:02 PM, Chris Barker <chris.barker@noaa.gov> wrote:
On Fri, Oct 18, 2013 at 6:22 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- it would be very useful if folks could easily get binary wheels for OS-X
We do plan to support it, but the pip devs uncovered a hole in the current wheel spec that means it generates the same filename on *nix systems for wheels that need to have different names for the download side of things to work properly
THanks -- but really? don't OS-X wheels get:
macosx_10_6_intel
or some such tacked on? Where does that go wrong?
Homebrew, Mac Ports, Fink. That would work OK if nobody ever installed things that the system didn't provide.
I don't understand this, what would break? Binary wheels that reference shared libraries that aren't included in the wheel (or a default system install) won't work, but that's also true on Windows.
The main difference I see is that on Windows, the ABI for the *CPython* shared library is significantly less likely to vary across machines. By contrast, it's relatively easy to change the ABI on *nix systems, as it depends on what you pass as configure options. Will a C extension built with Homebrew Python work with the Python Mac OS X installer from python.org? Probably, but given how painful ABI mismatches with shared libraries can be to debug, "probably" doesn't cut it until someone has the chance to thoroughly review the potential for problems. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 31 Oct, 2013, at 15:26, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 31 October 2013 23:38, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 21 Oct, 2013, at 20:52, Donald Stufft <donald@stufft.io> wrote:
On Oct 21, 2013, at 1:02 PM, Chris Barker <chris.barker@noaa.gov> wrote:
On Fri, Oct 18, 2013 at 6:22 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- it would be very useful if folks could easily get binary wheels for OS-X
We do plan to support it, but the pip devs uncovered a hole in the current wheel spec that means it generates the same filename on *nix systems for wheels that need to have different names for the download side of things to work properly
THanks -- but really? don't OS-X wheels get:
macosx_10_6_intel
or some such tacked on? Where does that go wrong?
Homebrew, Mac Ports, Fink. That would work OK if nobody ever installed things that the system didn't provide.
I don't understand this, what would break? Binary wheels that reference shared libraries that aren't included in the wheel (or a default system install) won't work, but that's also true on Windows.
The main difference I see is that on Windows, the ABI for the *CPython* shared library is significantly less likely to vary across machines.
By contrast, it's relatively easy to change the ABI on *nix systems, as it depends on what you pass as configure options.
Will a C extension built with Homebrew Python work with the Python Mac OS X installer from python.org?
As long as they use the same "ABI Tag" from PEP 425 extensions build with one should work with the other.
Probably, but given how painful ABI mismatches with shared libraries can be to debug, "probably" doesn't cut it until someone has the chance to thoroughly review the potential for problems.
I'd have to make time for a thorough review but be sure, but AFAIK this is only a little more complicated on OSX than on (say) Linux and the additonal complication is already captured by the platform tag mentioned by Chris. That is, the CPython ABI is affected by: * The usual configure flags (--with-pydebug, --enable-unicode, ...) These issues also exists on Windows, and the "ABI Tag" from PEP 425 means that wheels for different sets of configure flags can be recognized from their filename. * The OSX deployment target (the minimal OSX release you want to support) This is already recorded by the distutils platform tag * The processor architectures supported This is also recorded in the distutils platform tag. * Possibly the compiler This is pretty unlikely, as I (and others) already load extensions build with clang in CPython binaries built with GCC (and v.v.), but the low-level compiler support library might affect the ABI when using a compiler other than GCC or clang. And I'd expect that two-level namespaces on OSX make that question moot except for code that's explicitly built to disable that feature. And this does definitely affect CPython on Windows unless you use the limited ABI. The only reason this doesn't cause problems with binary installers on PyPI right now is that most users use the binary installers from www.python.org and hence are all using the same compiler version. The deployment target can affect the unix level API a little, IIRC around 10.4 or 10.5 the APIs on the POSIX level were adjusted a little to more fully comply with the UNIX specification and because of this the affected named resolve to different symbols based on the deployment target (more or less simular to symbol versioning in glibc). AFAIK that shouldn't affect the CPython ABI, but that would need to be carefully checked to be sure and I don't have time to do so right now. To provide more detail on the disutils platform tag, on my machine:
import distutils.util distutils.util.get_platform() 'macosx-10.8-intel'
This means the deployment target is OSX 10.8, with suppport for the 'intel' set of archictures (i386 and x86_64). This records the two variables that are most likely to cause problems. I'd expected more problems with binary eggs that are dynamicly linked to libraries that are provided in the wheel because OSX's linker add an absolute path to the library to be loaded unless you're careful. Ronald
On Thu, Oct 31, 2013 at 10:26 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 31 October 2013 23:38, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 21 Oct, 2013, at 20:52, Donald Stufft <donald@stufft.io> wrote:
On Oct 21, 2013, at 1:02 PM, Chris Barker <chris.barker@noaa.gov> wrote:
On Fri, Oct 18, 2013 at 6:22 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
-- it would be very useful if folks could easily get binary wheels for OS-X
We do plan to support it, but the pip devs uncovered a hole in the current wheel spec that means it generates the same filename on *nix systems for wheels that need to have different names for the download side of things to work properly
THanks -- but really? don't OS-X wheels get:
macosx_10_6_intel
or some such tacked on? Where does that go wrong?
Homebrew, Mac Ports, Fink. That would work OK if nobody ever installed things that the system didn't provide.
I don't understand this, what would break? Binary wheels that reference shared libraries that aren't included in the wheel (or a default system install) won't work, but that's also true on Windows.
The main difference I see is that on Windows, the ABI for the *CPython* shared library is significantly less likely to vary across machines.
By contrast, it's relatively easy to change the ABI on *nix systems, as it depends on what you pass as configure options.
I'm sure you could build your own broken Windows Python, but who bothers? IMO it pretty much boils down to the fact that on Windows you are probably using the python.org version of Python and not linking with random shared libraries from C:\Program Files\, but on Linux you are probably using one of a few dozen distro x distro-release Pythons AND your extension probably dynamically links to some other things your specific distro provides AND maybe you are depending on some versioned symbols in glibc oh the horror. On OS X I would hope you are uploading only wheels built with python.org-Python but maybe you aren't, everyone has their preferences.
Will a C extension built with Homebrew Python work with the Python Mac OS X installer from python.org? Probably, but given how painful ABI mismatches with shared libraries can be to debug, "probably" doesn't cut it until someone has the chance to thoroughly review the potential for problems.
On 31 Oct, 2013, at 17:49, Daniel Holth <dholth@gmail.com> wrote:
On Thu, Oct 31, 2013 at 10:26 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 31 October 2013 23:38, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
On 21 Oct, 2013, at 20:52, Donald Stufft <donald@stufft.io> wrote:
On Oct 21, 2013, at 1:02 PM, Chris Barker <chris.barker@noaa.gov> wrote:
On Fri, Oct 18, 2013 at 6:22 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
> -- it would be very useful if folks could easily > get binary wheels for OS-X
We do plan to support it, but the pip devs uncovered a hole in the current wheel spec that means it generates the same filename on *nix systems for wheels that need to have different names for the download side of things to work properly
THanks -- but really? don't OS-X wheels get:
macosx_10_6_intel
or some such tacked on? Where does that go wrong?
Homebrew, Mac Ports, Fink. That would work OK if nobody ever installed things that the system didn't provide.
I don't understand this, what would break? Binary wheels that reference shared libraries that aren't included in the wheel (or a default system install) won't work, but that's also true on Windows.
The main difference I see is that on Windows, the ABI for the *CPython* shared library is significantly less likely to vary across machines.
By contrast, it's relatively easy to change the ABI on *nix systems, as it depends on what you pass as configure options.
On OS X I would hope you are uploading only wheels built with python.org-Python but maybe you aren't, everyone has their preferences.
And that shouldn't be a problem, the various builds of Python on OSX should be compatible if the have the same platform tag in the wheel filename. Well, barring idiocy like installing a completely different libc, that might cause problems because (at least for 2.7) some libc types leak into the CPython ABI. But anyway, I'd love to use binary wheels in the future to distribute prebuilt binaries for some of my projects and that's why I reacted on a message that mentioned that binary installs aren't supported yet. If I don't understand what the issue is I can't try to help finding I solution :-) Is it "just" a matter of researching if the various build options on OSX really lead to binaries with the same ABI, or is more work needed? Ronald
Will a C extension built with Homebrew Python work with the Python Mac OS X installer from python.org? Probably, but given how painful ABI mismatches with shared libraries can be to debug, "probably" doesn't cut it until someone has the chance to thoroughly review the potential for problems.
On Oct 31, 2013, at 1:15 PM, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
Is it "just" a matter of researching if the various build options on OSX really lead to binaries with the same ABI, or is more work needed?
Basically it’s this: I was told a wheel built on Ubuntu probably won’t work on Linux, so I shut off Linux Wheels, at the same time I asked about Windows and OSX wheels, the answers I got from people were they would generally just work on Windows, and nobody gave me a straight answer on OSX. If you build a Wheel with Homebrew Python will it work on the official OSX installers? What if I have a library installed from Homebrew? Essentially trying to figure out how likely it is that with the existing tags a wheel is going to work if we select based on them. Once we know what, if any, problems exist for the various platforms then we can come up with fixes for them. This just hasn’t been high priority for me because of PEP453 work. To be honest the same problems likely exists on Windows, it’s just less likely and the benefits of prebuilt binaries greater. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/31/2013 02:24 PM, Donald Stufft wrote:
To be honest the same problems likely exists on Windows, it?s just less likely and the benefits of prebuilt binaries greater.
For all platforms *except* Windows, wheels are essentially caches -- there is no real reason to distribute them via PyPI at all, because OSx and Linux develpoers will have tools to build them from sdists. Pip *does* need to allow installing them on those platforms, but probably only via explicit paths / URLS (rather than finding them on PyPI). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlJywmwACgkQ+gerLs4ltQ5P/ACfXcMJj4dmnlNJEccZ8gxi8FLR GrQAoLxxgeVnKRvuoscR6GuabwGxfsnF =gZW1 -----END PGP SIGNATURE-----
On Oct 31, 2013, at 4:49 PM, Tres Seaver <tseaver@palladion.com> wrote:
Pip *does* need to allow installing them on those platforms, but probably only via explicit paths / URLS (rather than finding them on PyPI).
You can install them just fine on any platform, the only restrictions are PyPI won’t let you upload them, and pip won’t try to install them from pypi.python.org. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/31/2013 04:55 PM, Donald Stufft wrote:
On Oct 31, 2013, at 4:49 PM, Tres Seaver <tseaver@palladion.com> wrote:
Pip *does* need to allow installing them on those platforms, but probably only via explicit paths / URLS (rather than finding them on PyPI).
You can install them just fine on any platform, the only restrictions are PyPI won?t let you upload them, and pip won?t try to install them from pypi.python.org.
Excellent -- sorry I misunderstood. AFAICT, you could leave that restriction in place for ever for Linux. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlJy10kACgkQ+gerLs4ltQ5C0QCePWr4hX6zTff9hyCO9+w5s0Ac R8sAn11NPAr3d+SCpricLcdLOhtk/nmp =V/rv -----END PGP SIGNATURE-----
On 1 Nov 2013 06:50, "Tres Seaver" <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 10/31/2013 02:24 PM, Donald Stufft wrote:
To be honest the same problems likely exists on Windows, it?s just less likely and the benefits of prebuilt binaries greater.
For all platforms *except* Windows, wheels are essentially caches -- there is no real reason to distribute them via PyPI at all, because OSx and Linux develpoers will have tools to build them from sdists.
They're caches that can speed things up a *lot*, though, and for new users "can build C extensions" is a higher bar than "can run Python and pip". Given PEP 453, it's probably worth allowing wheels on Mac OS X in pip 1.5, then we can tackle the thornier general *nix problem in the pip 1.6 time frame (which should also improve external dependency handling on Windows and Mac OS X as well). Cheers, Nick.
Pip *does* need to allow installing them on those platforms, but probably only via explicit paths / URLS (rather than finding them on PyPI).
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlJywmwACgkQ+gerLs4ltQ5P/ACfXcMJj4dmnlNJEccZ8gxi8FLR GrQAoLxxgeVnKRvuoscR6GuabwGxfsnF =gZW1 -----END PGP SIGNATURE-----
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/31/2013 5:34 PM, Nick Coghlan wrote:
On 1 Nov 2013 06:50, "Tres Seaver" <tseaver@palladion.com <mailto:tseaver@palladion.com>> wrote:
On 10/31/2013 02:24 PM, Donald Stufft wrote:
To be honest the same problems likely exists on Windows, it?s just less likely and the benefits of prebuilt binaries greater.
For all platforms *except* Windows, wheels are essentially caches -- there is no real reason to distribute them via PyPI at all, because OSx and Linux develpoers will have tools to build them from sdists.
They're caches that can speed things up a *lot*, though, and for new users "can build C extensions" is a higher bar than "can run Python and pip".
I don't think it's true that they're caches on all Linux boxes. I have many production machines that do not have compilers on them, and do not have the development headers, etc. installed. They're no better off than a Windows machine! Sure, I could install all of this, but: - - I'd rather not increase the attack surface by installing lots of software - - Some of these are small VM instances, and I don't have room to install a full build environment Where I can, right now I typically install the tools I need to install, do the installation from source, and then remove the tools. It's a pain. I'd much rather have binary Linux wheels. I'm more than happy to build them myself on a dedicated build machine. Eric. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (Cygwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJSctEsAAoJENxauZFcKtNxSYYIAJW3BArMWX+Z669AYe/lpoj9 WMggmdfr6u6BXrOVdujHJpXG43U9UhcqJTYLOitNHyO9XKVq/ODDfbV7SN9LTlHI atkEf87xxjuaBI5nBiS0q/ozKMmU4AR82aHYQ2wcxlLry/CSt9zgBBM/vt4I0so2 UXeUa5CgkXGGsG5nYvk9zOAQmMknUSESZrE/WNsgjImk9oBBySs9Tvz+hUF407Y2 sRFuWVZCoDjFwWg8/e98YI6sxuqx7fFhdzn5uoXq9wQ27xg9FjKQlbAWPYcBVK1M ZAQj0SgSMY4paKQ9QsOdQchieJjRZ21C7ue8WGaca/IeyY280RTA0ztuRVMFz58= =nmb3 -----END PGP SIGNATURE-----
On Oct 31, 2013, at 5:52 PM, Eric V. Smith <eric@trueblade.com> wrote:
I'm more than happy to build them myself on a dedicated build machine.
This works! You just can’t upload/install from PyPI. The restriction is *only* centered around PyPI. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/31/2013 6:05 PM, Donald Stufft wrote:
On Oct 31, 2013, at 5:52 PM, Eric V. Smith <eric@trueblade.com> wrote:
I'm more than happy to build them myself on a dedicated build machine.
This works! You just can’t upload/install from PyPI.
The restriction is *only* centered around PyPI.
Ah, I misunderstood. Cool, and thanks for all of the work! Eric. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.13 (Cygwin) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJSctWHAAoJENxauZFcKtNxzXEH/0XhiAoMeS8WODLrKSREkDFi trvaIFU/AaC0fIidkjqSmSk/axaD8mp8yPCjo/PVeq+ZM7Mi1pmZmGmU7k6KkpeS xdEYCpMBg72yYFzo/fyvIQGgVm3Y/mMaIVc3fBbzfEk08vMbNUVFwgg0Jtz8OOfh F2oqTLxqLYhaLQ36hpWuQpJYHDNEEa0fDMiomzKhvzoFEu4e5XhgMpi2DrkYQWbM q7mmLOhJ76SVzODx+fx+jvAu8xQ++MGNxcQ90O4m1pP+RPHnw5DAXVE6jJ1Ko8jB S5zKKvInFeLB5eQwxgQ1S3JMLLQOD02y2pI+w3T7fbhUBtiKVctL02dXxOUw6/Q= =1dem -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/31/2013 05:52 PM, Eric V. Smith wrote:
I'm more than happy to build them myself on a dedicated build machine.
Right -- that makes them back into caches. ;) You can safely deploy them because you know the architecture / libc / etc. against which you built them. They are highly unlikely to be useful to anyone *else*, however, except by accident. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlJy1vwACgkQ+gerLs4ltQ7HqQCgs3toEwtRSn5q+USRXdOWDB+C dusAn0fSfRyUmRSvqGQ3IUh1Kl6Jh36X =OCEp -----END PGP SIGNATURE-----
On Thu, Oct 31, 2013 at 2:34 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
For all platforms *except* Windows, wheels are essentially caches -- there is no real reason to distribute them via PyPI at all, because OSx and Linux develpoers will have tools to build them from sdists.
That's not at all true -- it IS true of homebrew, etc users, but not the least bit true of the genreral Mac user: * Installing XCode is free, but not default, and less than trivial, and even less than trivial to get right to build python extensions. * Many packages require third party compiled libs -- even harder to do on the Mac -- some are a downright pain in the &^%*&. What if an OSX user wants to install numpy/scipy? How easy is it to do this
from source (I really don't know)?
A serious pain in the %^&$ -- numpy is pretty easy, but scipy is a nightmare, requiring Fortran, etc. The community has addressed with with "scientific python distributions": Anaconda, Canopy, Python(x,y), etc. But is sure would be nice to have binary wheels on PyPi And the cache thing is really nice, actually.
Given PEP 453, it's probably worth allowing wheels on Mac OS X in pip 1.5, then we can tackle the thornier general *nix problem in the pip 1.6 time frame (which should also improve external dependency handling on Windows and Mac OS X as well).
Sounds great! -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Oct 31, 2013 8:50 PM, "Tres Seaver" <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 10/31/2013 02:24 PM, Donald Stufft wrote:
To be honest the same problems likely exists on Windows, it?s just less likely and the benefits of prebuilt binaries greater.
For all platforms *except* Windows, wheels are essentially caches -- there is no real reason to distribute them via PyPI at all, because OSx and Linux develpoers will have tools to build them from sdists.
What if an OSX user wants to install numpy/scipy? How easy is it to do this from source (I really don't know)? Building the necessary BLAS/LAPACK libraries isn't easy on any platform. It's just easier on a linux distro when the package manager can do it for you. Oscar
On Oct 31, 2013, at 07:24 PM, Donald Stufft <donald@stufft.io> wrote: On Oct 31, 2013, at 1:15 PM, Ronald Oussoren <ronaldoussoren@mac.com> wrote: Is it "just" a matter of researching if the various build options on OSX really lead to binaries with the same ABI, or is more work needed? Basically it’s this: I was told a wheel built on Ubuntu probably won’t work on Linux, so I shut off Linux Wheels, at the same time I asked about Windows and OSX wheels, the answers I got from people were they would generally just work on Windows, and nobody gave me a straight answer on OSX. I can't promise anything, but I'll try to provide some documentation on what does and does not work on OSX. From what I've seen the problem with Ubuntu wheels not working on other Linux's is due to changes in the glibc ABI (that is, code compiled with glibc X+1 might not work with glibc X), and that shouldn't be a problem on OSX. If you build a Wheel with Homebrew Python will it work on the official OSX installers? What if I have a library installed from Homebrew? Essentially trying to figure out how likely it is that with the existing tags a wheel is going to work if we select based on them. Again, someone (and if no-one beats me to it, I) will have to document and test the various scenarios. The only real problem I expect is with extensions linking to shared libraries that aren't in the normal OSX install and aren't provided in the wheel. That is, a wheel for lxml that's linked with the Homebrew version of libxml2. That will only work on systems with Homebrew that have libxml2 installed. Once we know what, if any, problems exist for the various platforms then we can come up with fixes for them. This just hasn’t been high priority for me because of PEP453 work. To be honest the same problems likely exists on Windows, it’s just less likely and the benefits of prebuilt binaries greater. I'm not sure about the benefits of prebuilt binaries. Apparently building universal binaries on OSX is hard for a lot of users, although I've never had problems with that. But I've written lots of cross-platform C code when cross-platform Unix code was more than just supporting osx, linux and freebsd and am therefore not a good example of a Python users just wants to build an extension that works with his Python code :-) Ronald P.S. sorry if this message is somewhat garbled, I'm sending this from iCloud webmail and that doesn't always work properly with mailing lists ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 1 November 2013 13:41, Ronald Oussoren <ronaldoussoren@mac.com> wrote:
If you build a Wheel with Homebrew Python will it work on the official OSX installers? What if I have a library installed from Homebrew? Essentially trying to figure out how likely it is that with the existing tags a wheel is going to work if we select based on them.
Again, someone (and if no-one beats me to it, I) will have to document and test the various scenarios.
The key point here is the granularity of the PEP 425 tags used by wheel. The risk is that a wheel created on another system might declare (via its filename) that it is compatible with your system, and then not be, causing segfaults or similar when used. On Windows, thanks to the history of bdist_wininst, and the small number of people who build their own *anything* on Windows, there is really only one ABI/C Library/whatever to worry about and that is the python.org one. (Actually, there are two - 32 and 64 bit). If all builds on OSX are compatible at the ABI/CLib/architecture level then there should be no problem. Equally, if incompatible builds result in wheels with different compatibility tags, also no problem. It's only if 2 incompatible environments use the same tags that there could be an issue. I don't believe that linking with external libraries should be a problem here - if wheel X depends on library Y, then either it should include it or it should simply document "Needs library Y installed". It's not a *compatibility* issue as such. But maybe the Unix concept of executables linking to libraries in hard-coded paths might be an issue here, I don't know enough about how that works to say. Hope this clarifies things a bit. Paul
On Fri, Nov 1, 2013 at 6:59 AM, Paul Moore <p.f.moore@gmail.com> wrote:
The key point here is the granularity of the PEP 425 tags used by wheel.
The risk is that a wheel created on another system might declare (via its filename) that it is compatible with your system, and then not be, causing segfaults or similar when used.
indeed, which is why it _might_ be a good idea to include an extra python build flag or something: "python.org", "homebrew", "macports". However, it's probably the case that those aren't really the issues that are going to cause problems -- at least ones that aren't already handled by the OS-X version flags --- i.e if a package is built for 10.6+, then it should have the same system libs as a python built for 10.6+. Practically speaking, the issues I've run into are: * Packages built for a newer OS-X won't work on an older one -- but that should be handled by the OS-X version handling that exists. * "universal" binaries -- packages built for 32 bit aren't going to work with a 64 bit python, and a universal python can be both 32 and 64 bit (and PPC, but I think those days are gone...) -- but this _should_ be handled by the platform flag: IIRC, "intel" means 32+64 bit Intel. Though I'm not sure what homebrew or macports python report. But distutils generally does the right thing with self-contained C code. * External dependencies: This is the BIG ONE: it's the hardest to get right, and the hardest to check for. Third party libs must: - Be built to match the python, including SDK and architecture (including universal) - Be included somehow -- ideally statically linked, but I'm thinking that they could be included as part of another dependent package (I think that's how Anocanda does it). The trick with dynamic linking on OS-X is that that standard way to install and link a lib has the path to the lib hard-coded in -- so you can't move it without re-writing the headers. This can be done on install, but I don't hink we want pip to have to deal with that! You _can_ install and link libs with relative paths, which I think is what Anaconda is doing, but I haven't figured out how yet, and it's certainly not a no-brainer. So I don't think there is any way to get around the fact that you need to be careful to build a binary wheel that will work on the systems you are targeting -- but this is no different than the situation we've had for years with building binary installers for the Mac. But those dont work with pip, or virtualenv, or...
On Windows, thanks to the history of bdist_wininst, and the small number of people who build their own *anything* on Windows, there is really only one ABI/C Library/whatever to worry about and that is the python.org one. (Actually, there are two - 32 and 64 bit).
Well, technically, the situation is very similar -- it's hard to build a Windows binary (at least if it has external dependencies), so that it will just work. Socially, the situation is different -- there are a (relatively) small number of people building their own stuff. On the Mac, however, you have homebrew and macports, and ??, so lots of people building their own stuff. But those aren't the people we need to support with binaries! Is anyone expecting a binary built for Windows to work with a cygwin python? Is anyone expecting that they can build a binary on Windows with cygwin and give it out? That's what we're talking about here with the Mac. thanks to the history of bdist_wininst .. The mac has a history of bdist_mpkg as well, not as widely used, and a bit neglected lately, but it's there. And there is a history of folks providing binary installers for the python.org mac. But it would be really nice if we could go to wheels, and use pypi to distribute them. It really is the same as Windows -- anyone putting a binary on PyPi has an obligation to built is so it will work with the python.org python -- and it's not inherently any harder to do that than on Windows -- the only difference is that it may be easier to do it badly -- by simply running bdist_wheel without thinking about it (i.e with homebrew, or macports and whatever shared libs happen to be linked to). But again, that's a social problem -- we need to have an understanding about what is required to put a binary wheel up on pypi. Also, where we _could_ have a way to identify python.org, vs homebrew, vs. macports as different "platfroms", that's not going to help the hard problem, which is making sure third party libs are built and included properly.
If all builds on OSX are compatible at the ABI/CLib/architecture level then there should be no problem. Equally, if incompatible builds result in wheels with different compatibility tags, also no problem. It's only if 2 incompatible environments use the same tags that there could be an issue.
yeah -- but the third party libs are the bigger issue anyway...
I don't believe that linking with external libraries should be a problem here - if wheel X depends on library Y, then either it should include it or it should simply document "Needs library Y installed". It's not a *compatibility* issue as such.
no -- but it's the hard problem -- the POINT of providing binaries is for people that don't know how to built it themselves to have something they can use. There is no point at all in providing a binary python package that depends on a particular freetype, or libpng, or whatever -- if you can build those, you can build the package. (or have homebrew, etc. do it for you). But maybe the Unix concept
of executables linking to libraries in hard-coded paths might be an issue here, I don't know enough about how that works to say.
well, that can be gotten around, but it's not really different -- if you have a binary linked to a non-system shared lib, that shared lib needs to be properly installed. Properly may be differently defined than on other systems, but it still needs to be done. NOTE: somewhere in this thread a saw anote about binary eggs not working well with OS-X. AFAIK, the issue there is that setuptools did not understand "universal" packages -- i.e. if you happened to be running ppc, then it would look for a ppc egg, and not know a "universal" egg was what it needed. IN fact, it would often install it correctly, but then not know it had, and try to build it from source anyway. But I think pip and wheel have got this right now -- but we won't know 'till we try! -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 1 November 2013 15:48, Chris Barker <chris.barker@noaa.gov> wrote:
On Fri, Nov 1, 2013 at 6:59 AM, Paul Moore <p.f.moore@gmail.com> wrote:
The key point here is the granularity of the PEP 425 tags used by wheel.
The risk is that a wheel created on another system might declare (via its filename) that it is compatible with your system, and then not be, causing segfaults or similar when used.
indeed, which is why it _might_ be a good idea to include an extra python build flag or something: "python.org", "homebrew", "macports". However, it's probably the case that those aren't really the issues that are going to cause problems -- at least ones that aren't already handled by the OS-X version flags --- i.e if a package is built for 10.6+, then it should have the same system libs as a python built for 10.6+.
The first thing to address is what wheel currently *does*. Basically, a (binary, let's ignore pure Python here) wheel is tagged as follows: Python version - cpXY Platform - distutils.util.get_platform(), which is "darwin" for OSX, if I'm reading the code correctly. ABI - essentially sysconfig.get_config_vars().get('SOABI') if that exists The only one here which may vary based on build is ABI. So the question is, do different Python builds on OSX have different SOABI values? If not, then does a simple extension with no external dependencies work when built with one Python and used with a different one? If it does, then there's probably no OSX issue[1]. If it doesn't, then there *is* an issue, and some code needs to change for wheels to be tagged correctly. Probably by changing the SOABI values, or by special coding in the ABI tag generation in bdist_wheel and wherever else it needs to be determined. If there are different SOABI values, then the above question about a simple extension should be asked for any SOABI value used by more than one Python build (if any). The issue is different on Windows (whether for technical or social reasons is debatable but irrelevant) because there is *no* ABI tag used. The platform tag catches cygwin and 32-bit vs 64-bit differences, and the Python version catches the C runtime compatibility (because any version of Python is only supported with one particular CRT - that's embedded in distutils in a way that means that you'd need to change the code to alter it, which is why it's arguably a technical issue rather than a social one). The point marked [1] above is where I ignore the question of "external libraries" :-) I'm honestly not sure what is meant by "external libraries" here. Does it mean which version of the C runtime is used? If so, then that should probably affect the ABI, but I don't know enough to know if that's right or not. If the libc version isn't part of the ABI, then I think this is the issue that caused people to back off on wheels for non-Windows platforms. I'd like to see a clear statement on whether building an extension with one libc version and running it with a different one is supported (I suspect "no") and whether it is likely/possible (on Windows, "not possible for all practical purposes", on Linux I suspect "happens all the time" and I have no idea for OSX). If by "external libraries" you mean something other than libc (for example, something like BLAS libraries for scipy) then my view is that this is not something taht wheel compatibility tags is intended to solve. The project should document what external libraries need to be present and any limitations on how they were built that may apply. If the user disregards these, that's their problem. If projects want to distribute one wheel for BLAS-compiled-with-homebrew, and one for BLAS-compiled-with-macports, then I'd put the onus on the project to come up with a solution - I think it's a specialised enough case that the wheel spec shouldn't be worrying about it yet, if ever. I hope this is of some use - maybe it'll help the people with access to OSX to have a clearer idea of what needs checking. Paul
Paul Moore wrote:
Python version - cpXY Platform - distutils.util.get_platform(), which is "darwin" for OSX, if I'm reading the code correctly. ABI - essentially sysconfig.get_config_vars().get('SOABI') if that exists
Does the MAXOSX_DEPLOYMENT_TARGET figure into this anywhere? Just knowing that the platform is "darwin" isn't really enough. -- Greg
On Nov 1, 2013, at 6:35 PM, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Paul Moore wrote:
Python version - cpXY Platform - distutils.util.get_platform(), which is "darwin" for OSX, if I'm reading the code correctly. ABI - essentially sysconfig.get_config_vars().get('SOABI') if that exists
Does the MAXOSX_DEPLOYMENT_TARGET figure into this anywhere? Just knowing that the platform is "darwin" isn't really enough.
-- Greg _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
python -c "import distutils; print(distutils.util.get_platform().replace('.', '_').replace('-', '_'))" macosx_10_8_x86_64 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 1 November 2013 22:36, Donald Stufft <donald@stufft.io> wrote:
python -c "import distutils; print(distutils.util.get_platform().replace('.', '_').replace('-', '_'))" macosx_10_8_x86_64
OK, cool. That means that binary wheels will be specific to the OSX version and the word size. Even more specific than Windows. So that just leaves SOABI. Do homebrew, macports etc actually use different libc versions, or is there a systemwide libc that everything uses which only changes when the OSX version changes? Paul
On Nov 1, 2013, at 6:49 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 1 November 2013 22:36, Donald Stufft <donald@stufft.io> wrote:
python -c "import distutils; print(distutils.util.get_platform().replace('.', '_').replace('-', '_'))" macosx_10_8_x86_64
OK, cool. That means that binary wheels will be specific to the OSX version and the word size. Even more specific than Windows.
So that just leaves SOABI. Do homebrew, macports etc actually use different libc versions, or is there a systemwide libc that everything uses which only changes when the OSX version changes?
Paul
How can I get the SOABI? ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Nov 1, 2013, at 6:57 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 1 November 2013 22:51, Donald Stufft <donald@stufft.io> wrote:
How can I get the SOABI?
Sorry, didn't I include that before?
sysconfig.get_config_var('SOABI')
should do it. Paul
Hmm, python -c "import sysconfig; print(sysconfig.get_config_var('SOABI’))" None ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Looks like wheels are not being created with an ABI tag at all. Here’s a Wheel I made to test things: https://testpypi.python.org/pypi/PyNaCl On Nov 1, 2013, at 6:58 PM, Donald Stufft <donald@stufft.io> wrote:
On Nov 1, 2013, at 6:57 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 1 November 2013 22:51, Donald Stufft <donald@stufft.io> wrote:
How can I get the SOABI?
Sorry, didn't I include that before?
sysconfig.get_config_var('SOABI')
should do it. Paul
Hmm,
python -c "import sysconfig; print(sysconfig.get_config_var('SOABI’))" None
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
In article <5B1E4FF5-1107-4E85-B0AC-F29461CEE4C2@stufft.io>, Donald Stufft <donald@stufft.io> wrote:
Looks like wheels are not being created with an ABI tag at all.
IIRC, distutils.util.get_platform() was not modified to reflect the SOABI when that feature was introduced in Python 3.2 (?). Perhaps it should have been or still can be for 3.4. -- Ned Deily, nad@acm.org
On Nov 1, 2013, at 8:39 PM, Ned Deily <nad@acm.org> wrote:
In article <5B1E4FF5-1107-4E85-B0AC-F29461CEE4C2@stufft.io>, Donald Stufft <donald@stufft.io> wrote:
Looks like wheels are not being created with an ABI tag at all.
IIRC, distutils.util.get_platform() was not modified to reflect the SOABI when that feature was introduced in Python 3.2 (?). Perhaps it should have been or still can be for 3.4.
Oh that is a 3.x feature? That would explain why my 2.7 built Wheel doesn’t have it then :) ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Donald Stufft wrote:
python -c "import distutils; print(distutils.util.get_platform().replace('.', '_').replace('-', '_'))" macosx_10_8_x86_64
Hmm, this just appears to reflect the version of MacOSX that the Python running distutils was built on, or is running on (not sure which). This is not quite the same thing as MACOSX_DEPLOYMENT_TARGET, which is the minimum version of MacOSX that is needed to run a piece of code. Distutils seems to assume they're the same, but if you're building a binary wheel for distribution, it makes sense to set MACOSX_DEPLOYMENT_TARGET as low as possible. Will there be a mechanism to get the actual MacOSX version needed into the metadata, rather than the one you happen to be building on? -- Greg
On 1 November 2013 23:10, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Will there be a mechanism to get the actual MacOSX version needed into the metadata, rather than the one you happen to be building on?
There can be anything - the question here is really whether anything is *needed*, or is what we have sufficient. If it's sufficient, we can lift the restriction on uploading OSX wheels and we're done. If it's not, we need to keep the restriction until the wheel code is updated to provide sufficiently flexible tags. We can take as long as we need to define and implement those. (When I say "we" here, as a Windows user I really mean "you", of course :-)) Paul
On 2 Nov 2013 09:15, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 1 November 2013 23:10, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Will there be a mechanism to get the actual MacOSX version needed into the metadata, rather than the one you happen to be building on?
There can be anything - the question here is really whether anything is *needed*, or is what we have sufficient.
If it's sufficient, we can lift the restriction on uploading OSX wheels and we're done. If it's not, we need to keep the restriction until the wheel code is updated to provide sufficiently flexible tags. We can take as long as we need to define and implement those. (When I say "we" here, as a Windows user I really mean "you", of course :-))
I spoke to Donald about this on IRC yesterday, and I take the following view: * the key relevant points about users on Windows and Mac OS X are that most (perhaps only many on Mac OS X) tutorials and introductory courses will direct them to the binary installers on python.org, and such users are highly unlikely to have a C compiler installed, so their current "out of the box" user experience with pip is that it doesn't work for even the simplest of C extensions. * by contrast, in other *nix environments (including cygwin on Windows and homebrew etc on Mac OS X), using the system/environment Python is far more common, and a C compiler is far more likely to be available * accordingly, the following defaults make sense for pip 1.5: - allow wheel files from the index server for Windows and Mac OS X - allow local wheel files everywhere * the following options should also be available: - force use of wheel files from the index server (useful for private index servers) - prevent use of wheel files from the index server (useful to force local builds on Windows and Mac OS X) - prevent use of wheel files (useful to force a local rebuild, overwriting the wheel cache) It would be good to enable the use of index server wheel files everywhere by default, but we need to add some kind of "environment" or "variant" marker to the wheel naming scheme before we can do that (i.e. wheel 1.1, which isn't planned until the 1.6 time frame). There are also problems with how the abi and platform tags are set and interpreted that need to be resolved before we can expand wheel sharing through PyPI beyond the environments defined by the python.org binary installers. Cheers, Nick.
Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Nov 1, 2013, at 8:45 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 2 Nov 2013 09:15, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 1 November 2013 23:10, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Will there be a mechanism to get the actual MacOSX version needed into the metadata, rather than the one you happen to be building on?
There can be anything - the question here is really whether anything is *needed*, or is what we have sufficient.
If it's sufficient, we can lift the restriction on uploading OSX wheels and we're done. If it's not, we need to keep the restriction until the wheel code is updated to provide sufficiently flexible tags. We can take as long as we need to define and implement those. (When I say "we" here, as a Windows user I really mean "you", of course :-))
I spoke to Donald about this on IRC yesterday, and I take the following view:
* the key relevant points about users on Windows and Mac OS X are that most (perhaps only many on Mac OS X) tutorials and introductory courses will direct them to the binary installers on python.org, and such users are highly unlikely to have a C compiler installed, so their current "out of the box" user experience with pip is that it doesn't work for even the simplest of C extensions.
* by contrast, in other *nix environments (including cygwin on Windows and homebrew etc on Mac OS X), using the system/environment Python is far more common, and a C compiler is far more likely to be available
* accordingly, the following defaults make sense for pip 1.5: - allow wheel files from the index server for Windows and Mac OS X - allow local wheel files everywhere
Just to be clear about things, pip already supports any wheel from any source *except* for pypi.python.org.
* the following options should also be available: - force use of wheel files from the index server (useful for private index servers)
At the exclusion of sdists? Not sure I see a point?
- prevent use of wheel files from the index server (useful to force local builds on Windows and Mac OS X)
I don’t think this needs to be a special option, the solution for the below should work fine here.
- prevent use of wheel files (useful to force a local rebuild, overwriting the wheel cache)
I do think we’ll need a —no-use-wheel
It would be good to enable the use of index server wheel files everywhere by default, but we need to add some kind of "environment" or "variant" marker to the wheel naming scheme before we can do that (i.e. wheel 1.1, which isn't planned until the 1.6 time frame).
There are also problems with how the abi and platform tags are set and interpreted that need to be resolved before we can expand wheel sharing through PyPI beyond the environments defined by the python.org binary installers.
Cheers, Nick.
Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 2 Nov 2013 11:10, "Donald Stufft" <donald@stufft.io> wrote:
On Nov 1, 2013, at 8:45 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 2 Nov 2013 09:15, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 1 November 2013 23:10, Greg Ewing <greg.ewing@canterbury.ac.nz>
Will there be a mechanism to get the actual MacOSX version needed into the metadata, rather than the one you happen to be building on?
There can be anything - the question here is really whether anything is *needed*, or is what we have sufficient.
If it's sufficient, we can lift the restriction on uploading OSX wheels and we're done. If it's not, we need to keep the restriction until the wheel code is updated to provide sufficiently flexible tags. We can take as long as we need to define and implement those. (When I say "we" here, as a Windows user I really mean "you", of course :-))
I spoke to Donald about this on IRC yesterday, and I take the following view:
* the key relevant points about users on Windows and Mac OS X are that most (perhaps only many on Mac OS X) tutorials and introductory courses will direct them to the binary installers on python.org, and such users are highly unlikely to have a C compiler installed, so their current "out of
* by contrast, in other *nix environments (including cygwin on Windows
and homebrew etc on Mac OS X), using the system/environment Python is far more common, and a C compiler is far more likely to be available
* accordingly, the following defaults make sense for pip 1.5: - allow wheel files from the index server for Windows and Mac OS X - allow local wheel files everywhere
Just to be clear about things, pip already supports any wheel from any
wrote: the box" user experience with pip is that it doesn't work for even the simplest of C extensions. source *except* for pypi.python.org.
* the following options should also be available: - force use of wheel files from the index server (useful for private
index servers)
At the exclusion of sdists? Not sure I see a point?
I didn't realise wheel files were already permitted by default for private index servers. With that clarification, I agree there's no need for this option.
- prevent use of wheel files from the index server (useful to force
local builds on Windows and Mac OS X)
I don’t think this needs to be a special option, the solution for the below should work fine here.
- prevent use of wheel files (useful to force a local rebuild,
overwriting the wheel cache)
I do think we’ll need a —no-use-wheel
It would be good to enable the use of index server wheel files everywhere by default, but we need to add some kind of "environment" or "variant" marker to the wheel naming scheme before we can do that (i.e. wheel 1.1, which isn't planned until the 1.6 time frame).
There are also problems with how the abi and platform tags are set and interpreted that need to be resolved before we can expand wheel sharing
If someone adds a new dependency, they should be able to easily say "build anything that I don't already have a local wheel file for from source". By contrast, I'd expect "--no-use-wheel" to rebuild everything (even stuff with already cached wheels). Cheers, Nick. through PyPI beyond the environments defined by the python.org binary installers.
Cheers, Nick.
Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Nov 1, 2013, at 9:56 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 2 Nov 2013 11:10, "Donald Stufft" <donald@stufft.io> wrote:
On Nov 1, 2013, at 8:45 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 2 Nov 2013 09:15, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 1 November 2013 23:10, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Will there be a mechanism to get the actual MacOSX version needed into the metadata, rather than the one you happen to be building on?
There can be anything - the question here is really whether anything is *needed*, or is what we have sufficient.
If it's sufficient, we can lift the restriction on uploading OSX wheels and we're done. If it's not, we need to keep the restriction until the wheel code is updated to provide sufficiently flexible tags. We can take as long as we need to define and implement those. (When I say "we" here, as a Windows user I really mean "you", of course :-))
I spoke to Donald about this on IRC yesterday, and I take the following view:
* the key relevant points about users on Windows and Mac OS X are that most (perhaps only many on Mac OS X) tutorials and introductory courses will direct them to the binary installers on python.org, and such users are highly unlikely to have a C compiler installed, so their current "out of the box" user experience with pip is that it doesn't work for even the simplest of C extensions.
* by contrast, in other *nix environments (including cygwin on Windows and homebrew etc on Mac OS X), using the system/environment Python is far more common, and a C compiler is far more likely to be available
* accordingly, the following defaults make sense for pip 1.5: - allow wheel files from the index server for Windows and Mac OS X - allow local wheel files everywhere
Just to be clear about things, pip already supports any wheel from any source *except* for pypi.python.org.
* the following options should also be available: - force use of wheel files from the index server (useful for private index servers)
At the exclusion of sdists? Not sure I see a point?
I didn't realise wheel files were already permitted by default for private index servers. With that clarification, I agree there's no need for this option.
- prevent use of wheel files from the index server (useful to force local builds on Windows and Mac OS X)
I don’t think this needs to be a special option, the solution for the below should work fine here.
If someone adds a new dependency, they should be able to easily say "build anything that I don't already have a local wheel file for from source". By contrast, I'd expect "--no-use-wheel" to rebuild everything (even stuff with already cached wheels).
The “local wheel” path isn’t the greatest right now, but to do this # Assumes you’ve already populated the wheelhouse with pip wheel pip install —find-links ~/.pip/wheelhouse whatever Wheels take precedence over sdists.
Cheers, Nick.
- prevent use of wheel files (useful to force a local rebuild, overwriting the wheel cache)
I do think we’ll need a —no-use-wheel
It would be good to enable the use of index server wheel files everywhere by default, but we need to add some kind of "environment" or "variant" marker to the wheel naming scheme before we can do that (i.e. wheel 1.1, which isn't planned until the 1.6 time frame).
There are also problems with how the abi and platform tags are set and interpreted that need to be resolved before we can expand wheel sharing through PyPI beyond the environments defined by the python.org binary installers.
Cheers, Nick.
Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 2 Nov 2013 11:59, "Donald Stufft" <donald@stufft.io> wrote:
On Nov 1, 2013, at 9:56 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 2 Nov 2013 11:10, "Donald Stufft" <donald@stufft.io> wrote:
On Nov 1, 2013, at 8:45 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 2 Nov 2013 09:15, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 1 November 2013 23:10, Greg Ewing <greg.ewing@canterbury.ac.nz>
Will there be a mechanism to get the actual MacOSX version needed into the metadata, rather than the one you happen to be building on?
There can be anything - the question here is really whether anything is *needed*, or is what we have sufficient.
If it's sufficient, we can lift the restriction on uploading OSX wheels and we're done. If it's not, we need to keep the restriction until the wheel code is updated to provide sufficiently flexible tags. We can take as long as we need to define and implement those. (When I say "we" here, as a Windows user I really mean "you", of course :-))
I spoke to Donald about this on IRC yesterday, and I take the following view:
* the key relevant points about users on Windows and Mac OS X are
* by contrast, in other *nix environments (including cygwin on
Windows and homebrew etc on Mac OS X), using the system/environment Python is far more common, and a C compiler is far more likely to be available
* accordingly, the following defaults make sense for pip 1.5: - allow wheel files from the index server for Windows and Mac OS X - allow local wheel files everywhere
Just to be clear about things, pip already supports any wheel from any
* the following options should also be available: - force use of wheel files from the index server (useful for private
index servers)
At the exclusion of sdists? Not sure I see a point?
I didn't realise wheel files were already permitted by default for
- prevent use of wheel files from the index server (useful to force
local builds on Windows and Mac OS X)
I don’t think this needs to be a special option, the solution for the
below should work fine here.
If someone adds a new dependency, they should be able to easily say "build anything that I don't already have a local wheel file for from
wrote: that most (perhaps only many on Mac OS X) tutorials and introductory courses will direct them to the binary installers on python.org, and such users are highly unlikely to have a C compiler installed, so their current "out of the box" user experience with pip is that it doesn't work for even the simplest of C extensions. source *except* for pypi.python.org. private index servers. With that clarification, I agree there's no need for this option. source". By contrast, I'd expect "--no-use-wheel" to rebuild everything (even stuff with already cached wheels).
The “local wheel” path isn’t the greatest right now, but to do this
# Assumes you’ve already populated the wheelhouse with pip wheel pip install —find-links ~/.pip/wheelhouse whatever
Wheels take precedence over sdists.
Cheers, Nick.
- prevent use of wheel files (useful to force a local rebuild,
overwriting the wheel cache)
I do think we’ll need a —no-use-wheel
It would be good to enable the use of index server wheel files
everywhere by default, but we need to add some kind of "environment" or "variant" marker to the wheel naming scheme before we can do that (i.e. wheel 1.1, which isn't planned until the 1.6 time frame).
There are also problems with how the abi and platform tags are set
and interpreted that need to be resolved before we can expand wheel sharing
Cool - sounds like allowing wheels by default on both Windows and Mac OS X and offering a --no-use-wheel option are the only changes needed then. Cheers, Nick. through PyPI beyond the environments defined by the python.org binary installers.
Cheers, Nick.
Paul _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Wheels take precedence over sdists.
also, for those that don't know, pip also prioritizes wheels over other wheels (using the same PEP425 module that the wheel project uses, which has a sort order). In short, plat-specific over python-specific, over general wheels. not a very likely scenario, but the logic is there.
The “local wheel” path isn’t the greatest right now, but to do this
# Assumes you’ve already populated the wheelhouse with pip wheel pip install —find-links ~/.pip/wheelhouse whatever
fwiw, I found one of the discussions we had ~year ago about "wheel caching". https://github.com/pypa/pip/pull/684 at the time, we weren't even sure if wheel was going to get merged, so something like "pip wheel" (that was outside of the main install pipeline) seemed like the first step. but now that things are moving along, an automatic wheel cache sounds awesome.
If someone adds a new dependency, they should be able to easily say "build anything that I don't already have a local wheel file for from source".
"pip wheel -r requirements.txt" will blindly rebuild wheels for all your dependencies, regardless of it being in your wheel dir already. it's been on the list for about 9 months now to make this better : ) https://github.com/pypa/pip/issues/855
On Fri, Nov 1, 2013 at 5:45 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
* the key relevant points about users on Windows and Mac OS X are that most (perhaps only many on Mac OS X) tutorials and introductory courses will direct them to the binary installers on python.org, and such users are highly unlikely to have a C compiler installed, so their current "out of the box" user experience with pip is that it doesn't work for even the simplest of C extensions.
Thank you for being so articulate about that -- I"ve been unsuccesfully trying to say this this whole thread .... Note also that it's not just what tutorials say, it's what the _should_ say. WE really wouldn't want to say to new users: """ Want to learn to program in Python? First, install a compiler, which, by the way is a multi-GB download from Apple, that you have to register as a developer to get..... """ Though I"ll also add that binaries for the python.org builds also support users that may have compiler, but not the expertise to build third-party libs, and build re-distributable binaries for older OS versions, etc.
* by contrast, in other *nix environments (including cygwin on Windows and homebrew etc on Mac OS X), using the system/environment Python is far more common, and a C compiler is far more likely to be available
indeed, required, for homebrew and macports (and cygwin?)
* accordingly, the following defaults make sense for pip 1.5: - allow wheel files from the index server for Windows and Mac OS X - allow local wheel files everywhere
sounds good. -- and have a stated policy )or at least recommendation) that binary wheels for OS-X be built for the python.org pythons.
* the following options should also be available:
- force use of wheel files from the index server (useful for private index servers) - prevent use of wheel files from the index server (useful to force local builds on Windows and Mac OS X) - prevent use of wheel files (useful to force a local rebuild, overwriting the wheel cache)
sounds good. One question: should pip be able to install a incompatible binary wheel directly without even a warning? It does now, but I don't think it should. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
One question: should pip be able to install a incompatible binary wheel directly without even a warning? It does now, but I don't think it should.
pip does confirm the wheel file is compatible with your platform/python (based on the file tags), when downloading from indexes and links. BUT, just noticed, when installing directly from a specific file, it seems it's not checking, so I'll look into that and log an issue. Marcus
On Nov 1, 2013, at 11:25 PM, Marcus Smith <qwcode@gmail.com> wrote:
pip does confirm the wheel file is compatible with your platform/python (based on the file tags), when downloading from indexes and links. BUT, just noticed, when installing directly from a specific file, it seems it's not checking, so I'll look into that and log an issue
Great, thanks!
In article <527434D6.9020603@canterbury.ac.nz>, Greg Ewing <greg.ewing@canterbury.ac.nz> wrote:
Donald Stufft wrote:
python -c "import distutils; print(distutils.util.get_platform().replace('.', '_').replace('-', '_'))" macosx_10_8_x86_64
Hmm, this just appears to reflect the version of MacOSX that the Python running distutils was built on, or is running on (not sure which).
This is not quite the same thing as MACOSX_DEPLOYMENT_TARGET, which is the minimum version of MacOSX that is needed to run a piece of code.
Distutils seems to assume they're the same, but if you're building a binary wheel for distribution, it makes sense to set MACOSX_DEPLOYMENT_TARGET as low as possible.
Will there be a mechanism to get the actual MacOSX version needed into the metadata, rather than the one you happen to be building on?
Assuming that the wheel build is using distutils.util.get_platform(), that number *is* the value of MACOSX_DEPLOYMENT_TARGET that was used when Python was built unless it is overridden during execution of Python by setting a MACOSX_DEPLOYMENT_TARGET env variable. For example, with the python.org 64-bit installer used on OS X 10.8: $ /usr/local/bin/python2.7 -c 'import distutils.util;print(distutils.util.get_platform())' macosx-10.6-intel It works on 10.6, 10.7, 10.8, and 10.9. With the Apple supplied system Python 2.7 on OS X 10.8: $ /usr/bin/python2.7 -c 'import distutils.util;print(distutils.util.get_platform())' macosx-10.8-intel -- Ned Deily, nad@acm.org
Chris Barker wrote:
anyone putting a [MacOSX] binary on PyPi has an obligation to built is so it will work with the python.org <http://python.org> python -- and it's not inherently any harder to do that than on Windows
Is there some way that a wheel could be checked to make sure it doesn't depend on any libraries in non-standard locations, etc? If not, could enough information be added to the metadata to make it possible? -- Greg
On Thu, Oct 31, 2013 at 9:49 AM, Daniel Holth <dholth@gmail.com> wrote:
I'm sure you could build your own broken Windows Python, but who bothers?
As long as we are clear that we are talking about a social difference here, not a technical one... IMO it pretty much boils down to the fact that on Windows you
are probably using the python.org version of Python and not linking with random shared libraries from C:\Program Files\, but on Linux you are probably using one of a few dozen distro x distro-release Pythons AND your extension probably dynamically links to some other things your specific distro provides AND maybe you are depending on some versioned symbols in glibc oh the horror.
On OS X I would hope you are uploading only wheels built with python.org-Python but maybe you aren't, everyone has their preferences.
yes, they do -- but what is the target audience here? yes, a lot of folks use macports, homebrew etc, fewer, but I'm sure some, build their own Python from scratch -- but these are NOT the people that we want binary wheels for -- they don't want them anyway. The folks that we want to provide binary wheels for are NOT going to be building their own esoteric python, really, they are not. The MacPython community has a long standing tradition of building binaries (if at all) that are compatible with the python.org builds (and secondarily with the Apple-supplied Python) -- that is what I'd like to see supported by PyPi -- just like Windows.... Sure, someone could upload some oddly-built binary wheel to PyPi -- then it would not work for most users, and they would get complaints and hopefully fix it -- just like uploading a package with any other kind of bug in it. It is kind of a pain to build a truly portable binary package (when it depends on third-party compiled libs), but there is a small but committed group of folks doing that already -- let's make it easier to get stuff out there.
Will a C extension built with Homebrew Python work with the Python Mac
OS X installer from python.org? Probably, but given how painful ABI mismatches with shared libraries can be to debug, "probably" doesn't cut it until someone has the chance to thoroughly review the potential for problems.
I disagree: 1) I don't care if homebrew built extensions work with other pythons -- you want to build with homebrew, create a homebrew recipe. -- there should be a policy about how binary packages posted on PyPi should be built. 2) We're never going to find out what the problems are until we give it a try. Fundamentally, I disagree with the premise here: "If we can't guarantee that anything anyone uploads will work for everyone, we shouldn't allow it" -- that's an unattainable goal. If we do want a more fool-proof approach, then the name auto-generated by wheel should include something that means "python.org-build" only if built with the python.org build. And I suppose we could try to put check code in there to make sure that extensions aren't linked to outside libs. Actually, that would be a handy utility to have, even if it didn't enforce anything. (and by the way, it's rea;lly easy to build a binary for Windows that's linked to an external dll also -- we expect package builders to be careful with that...) I was told a wheel built on Ubuntu probably won’t work on Linux, so I shut
off Linux Wheels, at the same time I asked about Windows and OSX wheels, the answers I got from people were they would generally just work on Windows, and nobody gave me a straight answer on OSX.
Sorry we weren't out there answering! Linux is a different story -- not only are there a lot of variations out there, but there also is no obvious "standard" one could point to that we'd expect folks to build binary wheels for. OS-X has (to much) variety though it is less than Linux, and more to the point, there is a standard Python out there -- the python.org builds. And there is a tradition of building binaries for that standard. AFAIK, it is pretty much the ONLY build of Python that package maintainers support with binaries (if they support anything).
If you build a Wheel with Homebrew Python will it work on the official OSX installers? What if I have a library installed from Homebrew?
probably not, but again, I don't care -- that's not what binary wheels on Python would be for. And more to the point -- this is a policy question -- don't upload a binary wheel to pypi that depends on homebrew (or anything else that Apple didn't provide) Essentially trying to figure out how likely it is that with the existing
tags a wheel is going to work if we select based on them.-Chris
One thing I'm not clear on -- if you do : pip install something will pip preferentially select a binary wheel (if enabled on pypi?) -- that may be an issue as folks will surely try to pip install stuff with homebrew, macports, etc. pythons (though the wheels are more likely to work in that direction. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Oct 31, 2013, at 7:25 PM, Chris Barker <chris.barker@noaa.gov> wrote:
On Thu, Oct 31, 2013 at 9:49 AM, Daniel Holth <dholth@gmail.com> wrote: I'm sure you could build your own broken Windows Python, but who bothers?
As long as we are clear that we are talking about a social difference here, not a technical one...
IMO it pretty much boils down to the fact that on Windows you are probably using the python.org version of Python and not linking with random shared libraries from C:\Program Files\, but on Linux you are probably using one of a few dozen distro x distro-release Pythons AND your extension probably dynamically links to some other things your specific distro provides AND maybe you are depending on some versioned symbols in glibc oh the horror.
On OS X I would hope you are uploading only wheels built with python.org-Python but maybe you aren't, everyone has their preferences.
yes, they do -- but what is the target audience here? yes, a lot of folks use macports, homebrew etc, fewer, but I'm sure some, build their own Python from scratch -- but these are NOT the people that we want binary wheels for -- they don't want them anyway.
The folks that we want to provide binary wheels for are NOT going to be building their own esoteric python, really, they are not.
The MacPython community has a long standing tradition of building binaries (if at all) that are compatible with the python.org builds (and secondarily with the Apple-supplied Python) -- that is what I'd like to see supported by PyPi -- just like Windows....
Sure, someone could upload some oddly-built binary wheel to PyPi -- then it would not work for most users, and they would get complaints and hopefully fix it -- just like uploading a package with any other kind of bug in it.
It is kind of a pain to build a truly portable binary package (when it depends on third-party compiled libs), but there is a small but committed group of folks doing that already -- let's make it easier to get stuff out there.
Will a C extension built with Homebrew Python work with the Python Mac OS X installer from python.org? Probably, but given how painful ABI mismatches with shared libraries can be to debug, "probably" doesn't cut it until someone has the chance to thoroughly review the potential for problems.
I disagree:
1) I don't care if homebrew built extensions work with other pythons -- you want to build with homebrew, create a homebrew recipe. -- there should be a policy about how binary packages posted on PyPi should be built.
Well it was more of we didn’t know, so we played it safe. I don’t personally have a Python.org Installer Python (mine come from Homebrew) and I know others use the System Python. My knowledge of compiled stuff is pretty slim tbh :)
2) We're never going to find out what the problems are until we give it a try.
Fundamentally, I disagree with the premise here: "If we can't guarantee that anything anyone uploads will work for everyone, we shouldn't allow it" -- that's an unattainable goal.
It was less about guarantee and more about “Can we be reasonably sure this is going to work for most people”.
If we do want a more fool-proof approach, then the name auto-generated by wheel should include something that means "python.org-build" only if built with the python.org build.
And I suppose we could try to put check code in there to make sure that extensions aren't linked to outside libs. Actually, that would be a handy utility to have, even if it didn't enforce anything. (and by the way, it's rea;lly easy to build a binary for Windows that's linked to an external dll also -- we expect package builders to be careful with that...)
I was told a wheel built on Ubuntu probably won’t work on Linux, so I shut off Linux Wheels, at the same time I asked about Windows and OSX wheels, the answers I got from people were they would generally just work on Windows, and nobody gave me a straight answer on OSX.
Sorry we weren't out there answering!
It’s not your fault! We realized this right before the pip release and sort of quickly added it after a quick check to see if anyone could tell us one way or another.
Linux is a different story -- not only are there a lot of variations out there, but there also is no obvious "standard" one could point to that we'd expect folks to build binary wheels for.
OS-X has (to much) variety though it is less than Linux, and more to the point, there is a standard Python out there -- the python.org builds. And there is a tradition of building binaries for that standard. AFAIK, it is pretty much the ONLY build of Python that package maintainers support with binaries (if they support anything).
If you build a Wheel with Homebrew Python will it work on the official OSX installers? What if I have a library installed from Homebrew?
probably not, but again, I don't care -- that's not what binary wheels on Python would be for. And more to the point -- this is a policy question -- don't upload a binary wheel to pypi that depends on homebrew (or anything else that Apple didn't provide)
Essentially trying to figure out how likely it is that with the existing tags a wheel is going to work if we select based on them.-Chris
One thing I'm not clear on -- if you do :
pip install something
will pip preferentially select a binary wheel (if enabled on pypi?) -- that may be an issue as folks will surely try to pip install stuff with homebrew, macports, etc. pythons (though the wheels are more likely to work in that direction.
Right this second it would require opting in via —use-wheel to get wheel installs from a simple ``pip install something`` but there’s discussion around making wheels enabled by default.
-Chris
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 19 Oct, 2013, at 3:22, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 19 Oct 2013 04:59, "Chris Barker" <chris.barker@noaa.gov> wrote:
Someone on another list indicated that pip installing binary wheels from PyPi will ONLY work for Windows.
Is that the case? I think it's desperately needed for OS-X as well.
Linux is so diverse that I can't imagine it being useful, but OS-X has only so many versions, and the python.org OS-X binaries are very clear in their requirements -- it would be very useful if folks could easily get binary wheels for OS-X
We do plan to support it, but the pip devs uncovered a hole in the current wheel spec that means it generates the same filename on *nix systems for wheels that need to have different names for the download side of things to work properly - hence the current Windows-only restriction.
Is that hole documented somewhere? Ronald
On Sat, Oct 19, 2013 at 3:22 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Once ensurepip has landed in Python 3.4 and pip 1.5 is released, we should be able to get back to updating the various metadata specs, with the aim of getting cross-platform wheel support in pip 1.6 :)
If I understand right, we must waiting Python 3.5 (~2016) to have out-of-the-box wheel packages on Linux. Really? -- Sebastien Douche <sdouche@gmail.com> Twitter: @sdouche / G+: +sdouche
On Oct 31, 2013, at 7:14 PM, Sebastien Douche <sdouche@gmail.com> wrote:
On Sat, Oct 19, 2013 at 3:22 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Once ensurepip has landed in Python 3.4 and pip 1.5 is released, we should be able to get back to updating the various metadata specs, with the aim of getting cross-platform wheel support in pip 1.6 :)
If I understand right, we must waiting Python 3.5 (~2016) to have out-of-the-box wheel packages on Linux. Really?
No, Python 3.4.x will ship whatever the latest version of pip is, so if 1.6 is out when 3.4.1 ships it’ll ship with pip 1.6. You can also always pip install —upgrade pip before then. There’s only so many hours in a day and so many people willing to get involved some things have to be delayed.
-- Sebastien Douche <sdouche@gmail.com> Twitter: @sdouche / G+: +sdouche _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Fri, Nov 1, 2013 at 12:17 AM, Donald Stufft <donald@stufft.io> wrote:
No, Python 3.4.x will ship whatever the latest version of pip is, so if 1.6 is out when 3.4.1 ships it’ll ship with pip 1.6.
Good to know. Thanks Donald. -- Sebastien Douche <sdouche@gmail.com> Twitter: @sdouche / G+: +sdouche
On 19.10.13 03:22, Nick Coghlan wrote:
On 19 Oct 2013 04:59, "Chris Barker" <chris.barker@noaa.gov <mailto:chris.barker@noaa.gov>> wrote:
Someone on another list indicated that pip installing binary wheels from PyPi will ONLY work for Windows.
Is that the case? I think it's desperately needed for OS-X as well.
Linux is so diverse that I can't imagine it being useful, but OS-X has only so many versions, and the python.org <http://python.org> OS-X
binaries are very clear
in their requirements -- it would be very useful if folks could easily get binary wheels for OS-X
We do plan to support it, but the pip devs uncovered a hole in the current wheel spec that means it generates the same filename on *nix systems for wheels that need to have different names for the download side of things to work properly - hence the current Windows-only restriction.
Once ensurepip has landed in Python 3.4 and pip 1.5 is released, we should be able to get back to updating the various metadata specs, with the aim of getting cross-platform wheel support in pip 1.6 :)
Ok, but then wheel should be explicit about that and not pretend to work on OS X. I tried that a week ago on a PySide install on OS X, which compiled very long, and crashed at the end. If wheel does not support bdist_wheel on a platform, it should refuse to install at all, IMHO. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
On Fri, Nov 1, 2013 at 6:39 AM, Christian Tismer <tismer@stackless.com> wrote:
On 19.10.13 03:22, Nick Coghlan wrote:
On 19 Oct 2013 04:59, "Chris Barker" <chris.barker@noaa.gov> wrote:
Someone on another list indicated that pip installing binary wheels from PyPi will ONLY work for Windows.
Is that the case? I think it's desperately needed for OS-X as well.
Linux is so diverse that I can't imagine it being useful, but OS-X has only so many versions, and the python.org OS-X binaries are very clear in their requirements -- it would be very useful if folks could easily get binary wheels for OS-X
We do plan to support it, but the pip devs uncovered a hole in the current wheel spec that means it generates the same filename on *nix systems for wheels that need to have different names for the download side of things to work properly - hence the current Windows-only restriction.
Once ensurepip has landed in Python 3.4 and pip 1.5 is released, we should be able to get back to updating the various metadata specs, with the aim of getting cross-platform wheel support in pip 1.6 :)
Ok, but then wheel should be explicit about that and not pretend to work on OS X. I tried that a week ago on a PySide install on OS X, which compiled very long, and crashed at the end. If wheel does not support bdist_wheel on a platform, it should refuse to install at all, IMHO.
cheers - chris
A reminder that *uploading non-Windows binary wheels to pypi* is the only thing that is restricted. That is because it was tried with eggs and found to be frustrating. bdist_wheel is supposed to work on all platforms.
On 01.11.13 13:34, Daniel Holth wrote:
On Fri, Nov 1, 2013 at 6:39 AM, Christian Tismer <tismer@stackless.com> wrote:
On 19.10.13 03:22, Nick Coghlan wrote:
On 19 Oct 2013 04:59, "Chris Barker" <chris.barker@noaa.gov> wrote:
Someone on another list indicated that pip installing binary wheels from PyPi will ONLY work for Windows.
Is that the case? I think it's desperately needed for OS-X as well.
Linux is so diverse that I can't imagine it being useful, but OS-X has only so many versions, and the python.org OS-X binaries are very clear in their requirements -- it would be very useful if folks could easily get binary wheels for OS-X We do plan to support it, but the pip devs uncovered a hole in the current wheel spec that means it generates the same filename on *nix systems for wheels that need to have different names for the download side of things to work properly - hence the current Windows-only restriction.
Once ensurepip has landed in Python 3.4 and pip 1.5 is released, we should be able to get back to updating the various metadata specs, with the aim of getting cross-platform wheel support in pip 1.6 :)
Ok, but then wheel should be explicit about that and not pretend to work on OS X. I tried that a week ago on a PySide install on OS X, which compiled very long, and crashed at the end. If wheel does not support bdist_wheel on a platform, it should refuse to install at all, IMHO.
cheers - chris A reminder that *uploading non-Windows binary wheels to pypi* is the only thing that is restricted. That is because it was tried with eggs and found to be frustrating.
bdist_wheel is supposed to work on all platforms.
Ah, sorry, then I have hit a real bug and should report that ;-) -- Christian Tismer :^) <mailto:tismer@stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
participants (15)
-
Chris Barker -
Chris Barker - NOAA Federal
-
Christian Tismer -
Daniel Holth -
Donald Stufft -
Eric V. Smith -
Greg Ewing -
Marcus Smith -
Ned Deily -
Nick Coghlan -
Oscar Benjamin -
Paul Moore -
Ronald Oussoren -
Sebastien Douche -
Tres Seaver