Linux binary wheels?
Hi folks. I have a pair of ideas about Linux binary wheels, which are currently (I heard) unsupported. It seems like it should be possible to support Linux binary wheels using one or both of these technologies: * https://build.opensuse.org/ is a service that builds packages for a variety of Linuxes * Docker could be used to automate the building of wheels for a handful of Linuxes with minimal dependencies. It seems like if you get Debian/Ubuntu/Mint, Fedora/CentOS, openSUSE and perhaps one or two others, that would cover almost all Linuxes and Linux users. I'm up to my hears in commitments already, but I sincerely someone will grab onto one or both of these possibilities and run with them. Thanks for reading.
How does this proposal differ from manylinux2010? https://github.com/pypa/manylinux/blob/master/README.rst#example PEP 513: manylinux1 https://www.python.org/dev/peps/pep-0513/ PEP 571: The manylinux2010 Platform Tag (latest, as of 2019) https://www.python.org/dev/peps/pep-0571/ On Monday, August 19, 2019, Dan Stromberg <drsalists@gmail.com> wrote:
Hi folks. I have a pair of ideas about Linux binary wheels, which are currently (I heard) unsupported. It seems like it should be possible to support Linux binary wheels using one or both of these technologies: * https://build.opensuse.org/ is a service that builds packages for a variety of Linuxes * Docker could be used to automate the building of wheels for a handful of Linuxes with minimal dependencies. It seems like if you get Debian/Ubuntu/Mint, Fedora/CentOS, openSUSE and perhaps one or two others, that would cover almost all Linuxes and Linux users.
I'm up to my hears in commitments already, but I sincerely someone will grab onto one or both of these possibilities and run with them. Thanks for reading.
It sounds pretty similar, but I'm still getting: $ twine upload --verbose --repository-url https://test.pypi.org/legacy/ dist/* below cmd output started 2019 Wed Aug 21 07:05:31 AM PDT Enter your username: dstromberg Enter your password: Uploading distributions to https://test.pypi.org/legacy/ Uploading treap-1.39-cp36-cp36m-linux_x86_64.whl 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 427k/427k [00:01<00:00, 306kB/s] Content received from server: <html> <head> <title>400 Binary wheel 'treap-1.39-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'.</title> </head> <body> <h1>400 Binary wheel 'treap-1.39-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'.</h1> The server could not comply with the request since it is either malformed or otherwise incorrect.<br/><br/> Binary wheel 'treap-1.39-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'. </body> </html> HTTPError: 400 Client Error: Binary wheel 'treap-1.39-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'. for url: https://test.pypi.org/legacy/ If a better way is available now, the error message probably should point the user at it. ? Thanks! On Mon, Aug 19, 2019 at 6:12 PM Wes Turner <wes.turner@gmail.com> wrote:
How does this proposal differ from manylinux2010?
https://github.com/pypa/manylinux/blob/master/README.rst#example
PEP 513: manylinux1 https://www.python.org/dev/peps/pep-0513/
PEP 571: The manylinux2010 Platform Tag (latest, as of 2019) https://www.python.org/dev/peps/pep-0571/
On Monday, August 19, 2019, Dan Stromberg <drsalists@gmail.com> wrote:
Hi folks. I have a pair of ideas about Linux binary wheels, which are currently (I heard) unsupported. It seems like it should be possible to support Linux binary wheels using one or both of these technologies: * https://build.opensuse.org/ is a service that builds packages for a variety of Linuxes * Docker could be used to automate the building of wheels for a handful of Linuxes with minimal dependencies. It seems like if you get Debian/Ubuntu/Mint, Fedora/CentOS, openSUSE and perhaps one or two others, that would cover almost all Linuxes and Linux users.
I'm up to my hears in commitments already, but I sincerely someone will grab onto one or both of these possibilities and run with them. Thanks for reading.
Hi, Yes, here you are trying to upload a generic Linux wheel. PyPI refuses, because there's no way Pip can tell if the generic Linux wheel will work for any particular Linux distribution. For example, you could build a LInux wheel on Debian 10, but someone on Redat 6 could try to install it - chances are that the installed package will be badly broken. This is the reason for the Manylinux standard - it gives Pip a standard to check against, to see whether the Manylinux wheel will work on the installing platform. Cheers, Matthew On Wed, Aug 21, 2019 at 3:09 PM Dan Stromberg <drsalists@gmail.com> wrote:
It sounds pretty similar, but I'm still getting: $ twine upload --verbose --repository-url https://test.pypi.org/legacy/ dist/* below cmd output started 2019 Wed Aug 21 07:05:31 AM PDT Enter your username: dstromberg Enter your password: Uploading distributions to https://test.pypi.org/legacy/ Uploading treap-1.39-cp36-cp36m-linux_x86_64.whl 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 427k/427k [00:01<00:00, 306kB/s] Content received from server: <html> <head> <title>400 Binary wheel 'treap-1.39-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'.</title> </head> <body> <h1>400 Binary wheel 'treap-1.39-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'.</h1> The server could not comply with the request since it is either malformed or otherwise incorrect.<br/><br/> Binary wheel 'treap-1.39-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'.
</body> </html> HTTPError: 400 Client Error: Binary wheel 'treap-1.39-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'. for url: https://test.pypi.org/legacy/
If a better way is available now, the error message probably should point the user at it.
?
Thanks!
On Mon, Aug 19, 2019 at 6:12 PM Wes Turner <wes.turner@gmail.com> wrote:
How does this proposal differ from manylinux2010?
https://github.com/pypa/manylinux/blob/master/README.rst#example
PEP 513: manylinux1 https://www.python.org/dev/peps/pep-0513/
PEP 571: The manylinux2010 Platform Tag (latest, as of 2019) https://www.python.org/dev/peps/pep-0571/
On Monday, August 19, 2019, Dan Stromberg <drsalists@gmail.com> wrote:
Hi folks. I have a pair of ideas about Linux binary wheels, which are currently (I heard) unsupported. It seems like it should be possible to support Linux binary wheels using one or both of these technologies: * https://build.opensuse.org/ is a service that builds packages for a variety of Linuxes * Docker could be used to automate the building of wheels for a handful of Linuxes with minimal dependencies. It seems like if you get Debian/Ubuntu/Mint, Fedora/CentOS, openSUSE and perhaps one or two others, that would cover almost all Linuxes and Linux users.
I'm up to my hears in commitments already, but I sincerely someone will grab onto one or both of these possibilities and run with them. Thanks for reading.
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/RSJ57...
You probably want to try 'repairing' the wheel with auditwheel to make a manylinux wheel from it: https://pypi.org/project/auditwheel/ On Wed, Aug 21, 2019, at 4:18 PM, Matthew Brett wrote:
Hi,
Yes, here you are trying to upload a generic Linux wheel. PyPI refuses, because there's no way Pip can tell if the generic Linux wheel will work for any particular Linux distribution. For example, you could build a LInux wheel on Debian 10, but someone on Redat 6 could try to install it - chances are that the installed package will be badly broken. This is the reason for the Manylinux standard - it gives Pip a standard to check against, to see whether the Manylinux wheel will work on the installing platform.
Cheers,
Matthew
On Wed, Aug 21, 2019 at 3:09 PM Dan Stromberg <drsalists@gmail.com> wrote:
It sounds pretty similar, but I'm still getting: $ twine upload --verbose --repository-url https://test.pypi.org/legacy/ dist/* below cmd output started 2019 Wed Aug 21 07:05:31 AM PDT Enter your username: dstromberg Enter your password: Uploading distributions to https://test.pypi.org/legacy/ Uploading treap-1.39-cp36-cp36m-linux_x86_64.whl 100%|██████████████████████████████████████████████████████████████████████████████████████████████| 427k/427k [00:01<00:00, 306kB/s] Content received from server: <html> <head> <title>400 Binary wheel 'treap-1.39-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'.</title> </head> <body> <h1>400 Binary wheel 'treap-1.39-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'.</h1> The server could not comply with the request since it is either malformed or otherwise incorrect.<br/><br/> Binary wheel 'treap-1.39-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'.
</body> </html> HTTPError: 400 Client Error: Binary wheel 'treap-1.39-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'. for url: https://test.pypi.org/legacy/
If a better way is available now, the error message probably should point the user at it.
?
Thanks!
On Mon, Aug 19, 2019 at 6:12 PM Wes Turner <wes.turner@gmail.com> wrote:
How does this proposal differ from manylinux2010?
https://github.com/pypa/manylinux/blob/master/README.rst#example
PEP 513: manylinux1 https://www.python.org/dev/peps/pep-0513/
PEP 571: The manylinux2010 Platform Tag (latest, as of 2019) https://www.python.org/dev/peps/pep-0571/
On Monday, August 19, 2019, Dan Stromberg <drsalists@gmail.com> wrote:
Hi folks. I have a pair of ideas about Linux binary wheels, which are currently (I heard) unsupported. It seems like it should be possible to support Linux binary wheels using one or both of these technologies: * https://build.opensuse.org/ is a service that builds packages for a variety of Linuxes * Docker could be used to automate the building of wheels for a handful of Linuxes with minimal dependencies. It seems like if you get Debian/Ubuntu/Mint, Fedora/CentOS, openSUSE and perhaps one or two others, that would cover almost all Linuxes and Linux users.
I'm up to my hears in commitments already, but I sincerely someone will grab onto one or both of these possibilities and run with them. Thanks for reading.
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at
https://mail.python.org/archives/list/distutils-sig@python.org/message/RSJ57... -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/AJHTJ...
On Thu, 22 Aug 2019 at 00:10, Dan Stromberg <drsalists@gmail.com> wrote:
HTTPError: 400 Client Error: Binary wheel 'treap-1.39-cp36-cp36m-linux_x86_64.whl' has an unsupported platform tag 'linux_x86_64'. for url: https://test.pypi.org/legacy/
If a better way is available now, the error message probably should point the user at it.
That's a fair point: Warehouse issue filed at https://github.com/pypa/warehouse/issues/6545 There's unfortunately a related packaging.python.org issue that would probably need to be addressed first, since there isn't currently a solid end user facing page that a custom error message could link to. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hi, On Tue, Aug 20, 2019 at 1:23 AM Dan Stromberg <drsalists@gmail.com> wrote:
Hi folks. I have a pair of ideas about Linux binary wheels, which are currently (I heard) unsupported.
Your information is wrong, I'm afraid - Manylinux wheels have been standard, and widely used, for several years now. Unless you meant wheels for non-Intel platforms, in which case, please do say more about you need. See the links that Wes posted for more details, or try pip installing Numpy and Scipy in a new virtualenv, and see what happens. Cheers, Matthew
On Tue, Aug 20, 2019, 06:05 Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
See the links that Wes posted for more details, or try pip installing
Numpy and Scipy in a new virtualenv, and see what happens.
Probably needs to: pip install --only-binary :all: numpy scipy Else it'll just build from source?
Cheers,
Matthew --
-Brian
Hi, On Tue, Aug 20, 2019 at 11:10 AM Brian Skinn <brian.skinn@gmail.com> wrote:
On Tue, Aug 20, 2019, 06:05 Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
See the links that Wes posted for more details, or try pip installing Numpy and Scipy in a new virtualenv, and see what happens.
Probably needs to:
pip install --only-binary :all: numpy scipy
Pip doesn't build from source by default - here's the output from my machine just now: $ pip install numpy scipy Collecting numpy Downloading https://files.pythonhosted.org/packages/4a/2e/cf0a2fea6d97604a0e058e804b50d5... (14.9MB) |████████████████████████████████| 15.0MB 38.9MB/s Collecting scipy Downloading https://files.pythonhosted.org/packages/c5/40/f73bc951b060ba143f2092a3a2dd75... (27.6MB) |████████████████████████████████| 27.6MB 35.8MB/s Installing collected packages: numpy, scipy Successfully installed numpy-1.17.0 scipy-1.3.1 Cheers, Mattthew
On Tue, Aug 20, 2019 at 6:14 AM Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Tue, Aug 20, 2019 at 11:10 AM Brian Skinn <brian.skinn@gmail.com> wrote:
On Tue, Aug 20, 2019, 06:05 Matthew Brett <matthew.brett@gmail.com>
wrote:
Hi,
See the links that Wes posted for more details, or try pip installing Numpy and Scipy in a new virtualenv, and see what happens.
Probably needs to:
pip install --only-binary :all: numpy scipy
Pip doesn't build from source by default - here's the output from my machine just now:
$ pip install numpy scipy Collecting numpy Downloading https://files.pythonhosted.org/packages/4a/2e/cf0a2fea6d97604a0e058e804b50d5... (14.9MB) |████████████████████████████████| 15.0MB 38.9MB/s Collecting scipy Downloading https://files.pythonhosted.org/packages/c5/40/f73bc951b060ba143f2092a3a2dd75... (27.6MB) |████████████████████████████████| 27.6MB 35.8MB/s Installing collected packages: numpy, scipy Successfully installed numpy-1.17.0 scipy-1.3.1
Hard to argue with concrete data -- my bad! I wonder if there's an OS dependence here, though -- I'm almost certain I've had to use `--only-binary` in the past, to avoid pip on my Windows machines trying to download and build sdists, even when wheels were available.
Cheers,
Mattthew
On Tue, 20 Aug 2019 at 14:50, Brian Skinn <brian.skinn@gmail.com> wrote:
I wonder if there's an OS dependence here, though -- I'm almost certain I've had to use `--only-binary` in the past, to avoid pip on my Windows machines trying to download and build sdists, even when wheels were available.
Pip prefers newer versions over older ones. If there's a newer version with no binaries, pip will use the source for that version and try to build, even if there are older binaries. `--only-binary` would address that (although there is also a `--prefer-binary` flag in newer versions of pip, which is better suited to that situation). Paul
On Tue, Aug 20, 2019, at 3:50 PM, Brian Skinn wrote:
I wonder if there's an OS dependence here, though -- I'm almost certain I've had to use `--only-binary` in the past, to avoid pip on my Windows machines trying to download and build sdists, even when wheels were available.
Possibly you were trying to install something where the latest version didn't have a compatible wheel you could use, but an older version did. I think pip will prefer a newer version even if that means building from source. Thomas
On Tue, Aug 20, 2019, at 5:05 AM Matthew Brett <matthew.brett@gmail.com> wrote:
... Unless you meant wheels for non-Intel platforms, in which case, please do say more about you need.
Minor tangent: I've seen some people use https://www.piwheels.org/ for Raspberry Pi (ARM 6/7), but could the ARM binaries be uploaded to PyPI? I think I'm conflating the wheel building spec (is manylinux amd64 specific, or as long as the libraries are on any architecture?), toolchains, environment (sounds like Piwheels provides a platform to build them on), and package hosting (can PyPI host arbitrary archs?) in that one sentence.
On 20 Aug 2019, at 23:47, Nick Timkovich <prometheus235@gmail.com> wrote:
On Tue, Aug 20, 2019, at 5:05 AM Matthew Brett <matthew.brett@gmail.com> wrote:
... Unless you meant wheels for non-Intel platforms, in which case, please do say more about you need.
Minor tangent: I've seen some people use https://www.piwheels.org/ for Raspberry Pi (ARM 6/7), but could the ARM binaries be uploaded to PyPI?
I think I'm conflating the wheel building spec (is manylinux amd64 specific, or as long as the libraries are on any architecture?), toolchains, environment (sounds like Piwheels provides a platform to build them on), and package hosting (can PyPI host arbitrary archs?) in that one sentence.
This issue may be of relevant: https://github.com/pypa/warehouse/issues/3668 And there are even more layers to this problem. Wheels on piwheels are currently maintained by RPi folks; if they are going into PyPI, either package maintainers need to take over uploading (and even building) them, or PyPI needs a way to allow (qualified) people to upload stuffs for packages they don’t own. And maintainers might decide that ARM is not their supported platform anyway, and get us back to where we started.
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/OXSUW...
FWIW, conda supports the e.g. armv7l aarch32 and armv8 aarch64 / "ARM64" platforms. Third-party-built packages are the norm there; where there are channels like conda-forge and rpi. What does it mean to sign a CI build from a given unsigned git tag? "Build conda packages for ARM" https://github.com/conda-forge/conda-forge.github.io/issues/269 Raspbian is built for armv7. Raspberry Pi 2, 3, and 4 are armv8 CPUs (with hw SIMD, AES, SHA-1, SHA-256), so they support armv7l (aarch32) and armv8 (aarch64) Raspberry Pi Zero W have armv6 CPUs. Raspberry Pi 4 have max 4GB of RAM, so that may be a reason for the community to support aarch64. On Tuesday, August 20, 2019, Tzu-ping Chung <uranusjr@gmail.com> wrote:
On 20 Aug 2019, at 23:47, Nick Timkovich <prometheus235@gmail.com> wrote:
On Tue, Aug 20, 2019, at 5:05 AM Matthew Brett <matthew.brett@gmail.com> wrote:
... Unless you meant wheels for non-Intel platforms, in which case, please do say more about you need.
Minor tangent: I've seen some people use https://www.piwheels.org/ for Raspberry Pi (ARM 6/7), but could the ARM binaries be uploaded to PyPI?
I think I'm conflating the wheel building spec (is manylinux amd64 specific, or as long as the libraries are on any architecture?), toolchains, environment (sounds like Piwheels provides a platform to build them on), and package hosting (can PyPI host arbitrary archs?) in that one sentence.
This issue may be of relevant: https://github.com/ pypa/warehouse/issues/3668
And there are even more layers to this problem. Wheels on piwheels are currently maintained by RPi folks; if they are going into PyPI, either package maintainers need to take over uploading (and even building) them, or PyPI needs a way to allow (qualified) people to upload stuffs for packages they don’t own. And maintainers might decide that ARM is not their supported platform anyway, and get us back to where we started.
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@ python.org/message/OXSUW73EO5DTUO34EFURN3KHCDAKNS4Z/
The nice thing with the new macaroons is that theoretically I can provide someone my key to upload packages on my behalf for a singular PyPI project. This way I could allow a third-party service to backfill binary wheels for other platforms once I've released a version to PyPI. Bert
On Aug 20, 2019, at 12:25, Tzu-ping Chung <uranusjr@gmail.com> wrote:
On 20 Aug 2019, at 23:47, Nick Timkovich <prometheus235@gmail.com <mailto:prometheus235@gmail.com>> wrote:
On Tue, Aug 20, 2019, at 5:05 AM Matthew Brett <matthew.brett@gmail.com <mailto:matthew.brett@gmail.com>> wrote: ... Unless you meant wheels for non-Intel platforms, in which case, please do say more about you need.
Minor tangent: I've seen some people use https://www.piwheels.org/ <https://www.piwheels.org/> for Raspberry Pi (ARM 6/7), but could the ARM binaries be uploaded to PyPI?
I think I'm conflating the wheel building spec (is manylinux amd64 specific, or as long as the libraries are on any architecture?), toolchains, environment (sounds like Piwheels provides a platform to build them on), and package hosting (can PyPI host arbitrary archs?) in that one sentence.
This issue may be of relevant: https://github.com/pypa/warehouse/issues/3668 <https://github.com/pypa/warehouse/issues/3668>
And there are even more layers to this problem. Wheels on piwheels are currently maintained by RPi folks; if they are going into PyPI, either package maintainers need to take over uploading (and even building) them, or PyPI needs a way to allow (qualified) people to upload stuffs for packages they don’t own. And maintainers might decide that ARM is not their supported platform anyway, and get us back to where we started.
-- Distutils-SIG mailing list -- distutils-sig@python.org <mailto:distutils-sig@python.org> To unsubscribe send an email to distutils-sig-leave@python.org <mailto:distutils-sig-leave@python.org> https://mail.python.org/mailman3/lists/distutils-sig.python.org/ <https://mail.python.org/mailman3/lists/distutils-sig.python.org/> Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/OXSUW... <https://mail.python.org/archives/list/distutils-sig@python.org/message/OXSUW73EO5DTUO34EFURN3KHCDAKNS4Z/> -- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/HYGOQ...
participants (10)
-
Bert JW Regeer -
Brian Skinn -
Dan Stromberg -
Matthew Brett -
Nick Coghlan -
Nick Timkovich -
Paul Moore -
Thomas Kluyver -
Tzu-ping Chung -
Wes Turner