Mailman 3 heads-up on a plot to bring linux wheels to pypi - Distutils-SIG

heads-up on a plot to bring linux wheels to pypi

Nathaniel Smith

14 Jan 2016 14 Jan '16

11:25 a.m.

Hi all, Just wanted give distutils-sig a heads-up that there seems to be some momentum gathering around a plot to bring linux wheels to pypi: https://github.com/manylinux/manylinux Basically the idea is to define a standard baseline linux environment (available libraries + version numbers + ABI of these), give it a name (for now, "manylinux"), and then provide tools to build wheels against this environment, teach pip how to recognize that the system it's on conforms to the environment, and then convince pypi to start accepting wheels with this as a platform tag :-). And if we carefully define the baseline environment based on the experiences of the popular scientific python distros (Continuum's Anaconda + Enthought's Canopy) we can be highly confident that essentially all desktop + server linux distros will be compatible with these wheels. This strategy is orthogonal to the more ambitious efforts to define an interface between wheels and the platform package manager; they can proceed in parallel and potentially complement each other. The goal here is just to get linux up to feature parity with windows and osx. Status: - we have a draft policy - there's a cool tool for scanning wheels and checking whether they conform to the policy: https://github.com/manylinux/auditwheel - there's a draft docker image to make it easy to build such wheels (https://github.com/manylinux/manylinux/pull/2) To do: - bikeshed the name ("manylinux" was picked after about 2 minutes of discussion so as to get started. maybe "genericlinux" would be better? I dunno.) - build some test wheels - write a proper PEP - convince pip and pypi maintainers that this is a good idea ;-) -n -- Nathaniel J. Smith -- http://vorpus.org

Show replies by date

Nick Coghlan

14 Jan 14 Jan

3:42 p.m.

On 14 January 2016 at 15:55, Nathaniel Smith wrote:

...

Hi all,

Just wanted give distutils-sig a heads-up that there seems to be some momentum gathering around a plot to bring linux wheels to pypi:

https://github.com/manylinux/manylinux

Basically the idea is to define a standard baseline linux environment (available libraries + version numbers + ABI of these), give it a name (for now, "manylinux"), and then provide tools to build wheels against this environment, teach pip how to recognize that the system it's on conforms to the environment, and then convince pypi to start accepting wheels with this as a platform tag :-). And if we carefully define the baseline environment based on the experiences of the popular scientific python distros (Continuum's Anaconda + Enthought's Canopy) we can be highly confident that essentially all desktop + server linux distros will be compatible with these wheels.

Very nice!

...

This strategy is orthogonal to the more ambitious efforts to define an interface between wheels and the platform package manager; they can proceed in parallel and potentially complement each other. The goal here is just to get linux up to feature parity with windows and osx.

That sounds like a good strategy to me - improved automation of upstream -> downstream package conversions is a nice feature to have, but it's a redistributor oriented one, moreso than an end user focused one. For end users, a pragmatic approach to defining a baseline "manylinux" ABI based on the experience of cross-distro redistributors will provide more benefit, sooner.

...

Status: - we have a draft policy - there's a cool tool for scanning wheels and checking whether they conform to the policy: https://github.com/manylinux/auditwheel - there's a draft docker image to make it easy to build such wheels (https://github.com/manylinux/manylinux/pull/2)

To do: - bikeshed the name ("manylinux" was picked after about 2 minutes of discussion so as to get started. maybe "genericlinux" would be better? I dunno.)

Since you've already set up the GitHub group etc as "manylinux", I think it makes sense to just call it "good enough". "manylinux" is an accurate description (as *many* Linux distributions provide the ABI subset you're targeting, but not all of them), and "genericlinux" has potential to cause confusion with "linux-generic" kernel packages.

...

- build some test wheels - write a proper PEP - convince pip and pypi maintainers that this is a good idea ;-)

While I've historically advocated against the idea of defining our own "Linux platform ABI" subset, the fact that Enthought and Continuum are successfully distributing pre-built binaries through the simple "use CentOS 5.11" approach seems promising. In terms of non-scientific packages, the main group I'd suggest getting in touch with is pycryptography, as we'll probably want to baseline a more recent version of OpenSSL than the one in CentOS 5.11. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nick Coghlan

3:49 p.m.

On 14 January 2016 at 20:12, Nick Coghlan wrote:

...

On 14 January 2016 at 15:55, Nathaniel Smith wrote:

...
- build some test wheels - write a proper PEP - convince pip and pypi maintainers that this is a good idea ;-)

While I've historically advocated against the idea of defining our own "Linux platform ABI" subset, the fact that Enthought and Continuum are successfully distributing pre-built binaries through the simple "use CentOS 5.11" approach seems promising.

In terms of non-scientific packages, the main group I'd suggest getting in touch with is pycryptography, as we'll probably want to baseline a more recent version of OpenSSL than the one in CentOS 5.11.

Ah, looking at https://github.com/manylinux/auditwheel, I see anything linking to OpenSSL would fail the platform audit, at least for the current draft policy. That also seems like a potentially reasonable approach (although it could lead to complaints about "Why doesn't project <X> offer a wheel file?") Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Nathaniel Smith

4:43 p.m.

On Thu, Jan 14, 2016 at 2:19 AM, Nick Coghlan wrote:

...

On 14 January 2016 at 20:12, Nick Coghlan wrote:

...
On 14 January 2016 at 15:55, Nathaniel Smith wrote:

...
- build some test wheels - write a proper PEP - convince pip and pypi maintainers that this is a good idea ;-)

While I've historically advocated against the idea of defining our own "Linux platform ABI" subset, the fact that Enthought and Continuum are successfully distributing pre-built binaries through the simple "use CentOS 5.11" approach seems promising.

In terms of non-scientific packages, the main group I'd suggest getting in touch with is pycryptography, as we'll probably want to baseline a more recent version of OpenSSL than the one in CentOS 5.11.

Ah, looking at https://github.com/manylinux/auditwheel, I see anything linking to OpenSSL would fail the platform audit, at least for the current draft policy. That also seems like a potentially reasonable approach (although it could lead to complaints about "Why doesn't project <X> offer a wheel file?")

Yeah, it's very unfortunate, that's exactly the library that you'd like to be able to depend on, so we checked specifically. But it turns out that openssl has broken ABI over the relevant time-period, so there's no choice, you can't rely on the system version and have to statically link :-(. Though I guess this is no worse than than if you want to distribute a wheel that needs openssl on Windows. -n -- Nathaniel J. Smith -- http://vorpus.org

Alexander Walters

4:56 p.m.

On 1/14/2016 06:13, Nathaniel Smith wrote:

...

...
...
On 14 January 2016 at 15:55, Nathaniel Smith wrote:

...
- build some test wheels - write a proper PEP - convince pip and pypi maintainers that this is a good idea ;-) While I've historically advocated against the idea of defining our own "Linux platform ABI" subset, the fact that Enthought and Continuum are successfully distributing pre-built binaries through the simple "use CentOS 5.11" approach seems promising.

In terms of non-scientific packages, the main group I'd suggest getting in touch with is pycryptography, as we'll probably want to baseline a more recent version of OpenSSL than the one in CentOS 5.11. Ah, looking at https://github.com/manylinux/auditwheel, I see anything

On 14 January 2016 at 20:12, Nick Coghlan wrote: linking to OpenSSL would fail the platform audit, at least for the current draft policy. That also seems like a potentially reasonable approach (although it could lead to complaints about "Why doesn't project <X> offer a wheel file?") Yeah, it's very unfortunate, that's exactly the library that you'd

On Thu, Jan 14, 2016 at 2:19 AM, Nick Coghlan wrote: like to be able to depend on, so we checked specifically. But it turns out that openssl has broken ABI over the relevant time-period, so there's no choice, you can't rely on the system version and have to statically link :-(.

Though I guess this is no worse than than if you want to distribute a wheel that needs openssl on Windows.

-n

Theoretically you can statically link openssl on windows, or otherwise include it. An odd benefit (in this case, only) of an operating system that includes nothing is a culture of vendoring everything.

Glyph Lefkowitz

5:27 p.m.

...

On Jan 14, 2016, at 2:12 AM, Nick Coghlan wrote:

In terms of non-scientific packages, the main group I'd suggest getting in touch with is pycryptography, as we'll probably want to baseline a more recent version of OpenSSL than the one in CentOS 5.11.

1. It's "cryptography" (or "PyCA's Cryptography", or "cryptography.io"), not "pycryptography". This is an important distinction because "PyCrypto" is the crappy, old thing you should not use, and "cryptography" is the new hotness. 2. On every other platform where they distribute wheels, the Cryptography developers have statically linked both OpenSSL and libffi; I was tangentially involved in the effort to do this on OS X, and in the process of debugging that, I learned that the Linux toolchain is fairly similar. I would imagine that they'd want to statically link OpenSSL the same way, for the same reasons, on Linux. Cryptography does regular releases to bundle in newer OpenSSLs, generally more often than the underlying platforms do. (Since Cryptography does not directly export OpenSSL's API as such, it's easier to do multi-verison compatibility with Python than with C.) In fact I am going to go out on a limb and say that I think Cryptography could be ready to go with this in a few weeks if PyPI just started allowing Linux wheels. We've discussed using ancient-CentOS containers for building static binaries the same way PyPy does. The potentially tricky part is just building the static new versions of OpenSSL from scratch on old systems, I think... -glyph

Nathaniel Smith

5:31 p.m.

On Thu, Jan 14, 2016 at 3:57 AM, Glyph Lefkowitz wrote: [...]

...

In fact I am going to go out on a limb and say that I think Cryptography could be ready to go with this in a few weeks if PyPI just started allowing Linux wheels. We've discussed using ancient-CentOS containers for building static binaries the same way PyPy does. The potentially tricky part is just building the static new versions of OpenSSL from scratch on old systems, I think...

https://phusion.github.io/holy-build-box/ https://github.com/phusion/holy-build-box/blob/master/image/build.sh#L308 -n -- Nathaniel J. Smith -- http://vorpus.org

Nate Coraor

15 Jan 15 Jan

3:17 a.m.

On Thu, Jan 14, 2016 at 12:55 AM, Nathaniel Smith wrote:

...

Hi all,

Just wanted give distutils-sig a heads-up that there seems to be some momentum gathering around a plot to bring linux wheels to pypi:

https://github.com/manylinux/manylinux

Basically the idea is to define a standard baseline linux environment (available libraries + version numbers + ABI of these), give it a name (for now, "manylinux"), and then provide tools to build wheels against this environment, teach pip how to recognize that the system it's on conforms to the environment, and then convince pypi to start accepting wheels with this as a platform tag :-). And if we carefully define the baseline environment based on the experiences of the popular scientific python distros (Continuum's Anaconda + Enthought's Canopy) we can be highly confident that essentially all desktop + server linux distros will be compatible with these wheels.

Hi Nathaniel, This is exciting news. As you probably know, I've been working in this space as well. In fact, the January 2016 release of Galaxy[1] will be the first to use wheels to distribute or framework dependencies, and they're installed via pip. We also have a Docker-based build system that automates the process of building these wheels[2]. Unfortunately, this necessitates our own modified versions of pip[3] and wheel[4] for messing with the platform tags. My previous discussion here on distutils-sig on the subject stalled on the discussion of these environments, but it sounds like you're making progress where that stopped. If there is momentum to manylinux and it will gain the favor of pip/PyPI maintainers, I would be happy to transfer my effort there. I've written a linux distro detection library that may be of some use if it's not possible to base all environments off Centos 5. I, and the Galaxy project as a whole, are very motivated to make official Linux wheels a reality.

...

This strategy is orthogonal to the more ambitious efforts to define an interface between wheels and the platform package manager; they can proceed in parallel and potentially complement each other. The goal here is just to get linux up to feature parity with windows and osx.

Status: - we have a draft policy - there's a cool tool for scanning wheels and checking whether they conform to the policy: https://github.com/manylinux/auditwheel - there's a draft docker image to make it easy to build such wheels (https://github.com/manylinux/manylinux/pull/2)

To do: - bikeshed the name ("manylinux" was picked after about 2 minutes of discussion so as to get started. maybe "genericlinux" would be better? I dunno.)

FWIW, as an illumos proponent, this same work as is being done for Linux will transfer over to other "distro" OSes like illumos.

...

- build some test wheels - write a proper PEP - convince pip and pypi maintainers that this is a good idea ;-)

-n

-- Nathaniel J. Smith -- http://vorpus.org _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig

Not directly related, but Galaxy is moving toward Conda as its tool dependency distribution system, so we are likely to converge efforts in other places as well. --nate [1] https://github.com/galaxyproject/galaxy/ [2] https://github.com/galaxyproject/starforge/ [3] https://github.com/natefoo/pip/tree/linux-wheels [4] https://bitbucket.org/natefoo/wheel

3024

Age (days ago)

3024

Last active (days ago)

List overview

Download

7 comments

5 participants

participants (5)

Alexander Walters
Glyph Lefkowitz
Nate Coraor
Nathaniel Smith
Nick Coghlan