[Distutils] heads-up on a plot to bring linux wheels to pypi

[Nick Coghlan]

On 14 January 2016 at 15:55, Nathaniel Smith <njs at pobox.com> wrote:
> Hi all,
>
> Just wanted give distutils-sig a heads-up that there seems to be some
> momentum gathering around a plot to bring linux wheels to pypi:
>
>   https://github.com/manylinux/manylinux
>
> Basically the idea is to define a standard baseline linux environment
> (available libraries + version numbers + ABI of these), give it a name
> (for now, "manylinux"), and then provide tools to build wheels against
> this environment, teach pip how to recognize that the system it's on
> conforms to the environment, and then convince pypi to start accepting
> wheels with this as a platform tag :-). And if we carefully define the
> baseline environment based on the experiences of the popular
> scientific python distros (Continuum's Anaconda + Enthought's Canopy)
> we can be highly confident that essentially all desktop + server linux
> distros will be compatible with these wheels.

Very nice!

> This strategy is orthogonal to the more ambitious efforts to define an
> interface between wheels and the platform package manager; they can
> proceed in parallel and potentially complement each other. The goal
> here is just to get linux up to feature parity with windows and osx.

That sounds like a good strategy to me - improved automation of
upstream -> downstream package conversions is a nice feature to have,
but it's a redistributor oriented one, moreso than an end user focused
one.

For end users, a pragmatic approach to defining a baseline "manylinux"
ABI based on the experience of cross-distro redistributors will
provide more benefit, sooner.

> Status:
> - we have a draft policy
> - there's a cool tool for scanning wheels and checking whether they
> conform to the policy: https://github.com/manylinux/auditwheel
> - there's a draft docker image to make it easy to build such wheels
> (https://github.com/manylinux/manylinux/pull/2)
>
> To do:
> - bikeshed the name ("manylinux" was picked after about 2 minutes of
> discussion so as to get started. maybe "genericlinux" would be better?
> I dunno.)

Since you've already set up the GitHub group etc as "manylinux", I
think it makes sense to just call it "good enough". "manylinux" is an
accurate description (as *many* Linux distributions provide the ABI
subset you're targeting, but not all of them), and "genericlinux" has
potential to cause confusion with "linux-generic" kernel packages.

> - build some test wheels
> - write a proper PEP
> - convince pip and pypi maintainers that this is a good idea ;-)

While I've historically advocated against the idea of defining our own
"Linux platform ABI" subset, the fact that Enthought and Continuum are
successfully distributing pre-built binaries through the simple "use
CentOS 5.11" approach seems promising.

In terms of non-scientific packages, the main group I'd suggest
getting in touch with is pycryptography, as we'll probably want to
baseline a more recent version of OpenSSL than the one in CentOS 5.11.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia