[Distutils] heads-up on a plot to bring linux wheels to pypi

[Nate Coraor]

On Thu, Jan 14, 2016 at 12:55 AM, Nathaniel Smith <njs at pobox.com> wrote:

> Hi all,
>
> Just wanted give distutils-sig a heads-up that there seems to be some
> momentum gathering around a plot to bring linux wheels to pypi:
>
>   https://github.com/manylinux/manylinux
>
> Basically the idea is to define a standard baseline linux environment
> (available libraries + version numbers + ABI of these), give it a name
> (for now, "manylinux"), and then provide tools to build wheels against
> this environment, teach pip how to recognize that the system it's on
> conforms to the environment, and then convince pypi to start accepting
> wheels with this as a platform tag :-). And if we carefully define the
> baseline environment based on the experiences of the popular
> scientific python distros (Continuum's Anaconda + Enthought's Canopy)
> we can be highly confident that essentially all desktop + server linux
> distros will be compatible with these wheels.
>

Hi Nathaniel,

This is exciting news. As you probably know, I've been working in this
space as well. In fact, the January 2016 release of Galaxy[1] will be the
first to use wheels to distribute or framework dependencies, and they're
installed via pip. We also have a Docker-based build system that automates
the process of building these wheels[2].

Unfortunately, this necessitates our own modified versions of pip[3] and
wheel[4] for messing with the platform tags. My previous discussion here on
distutils-sig on the subject stalled on the discussion of these
environments, but it sounds like you're making progress where that stopped.

If there is momentum to manylinux and it will gain the favor of pip/PyPI
maintainers, I would be happy to transfer my effort there. I've written a
linux distro detection library that may be of some use if it's not possible
to base all environments off Centos 5. I, and the Galaxy project as a
whole, are very motivated to make official Linux wheels a reality.


> This strategy is orthogonal to the more ambitious efforts to define an
> interface between wheels and the platform package manager; they can
> proceed in parallel and potentially complement each other. The goal
> here is just to get linux up to feature parity with windows and osx.
>
> Status:
> - we have a draft policy
> - there's a cool tool for scanning wheels and checking whether they
> conform to the policy: https://github.com/manylinux/auditwheel
> - there's a draft docker image to make it easy to build such wheels
> (https://github.com/manylinux/manylinux/pull/2)
>
> To do:
> - bikeshed the name ("manylinux" was picked after about 2 minutes of
> discussion so as to get started. maybe "genericlinux" would be better?
> I dunno.)
>

FWIW, as an illumos proponent, this same work as is being done for Linux
will transfer over to other "distro" OSes like illumos.


> - build some test wheels
> - write a proper PEP
> - convince pip and pypi maintainers that this is a good idea ;-)
>
> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>


Not directly related, but Galaxy is moving toward Conda as its tool
dependency distribution system, so we are likely to converge efforts in
other places as well.

--nate

[1] https://github.com/galaxyproject/galaxy/
[2] https://github.com/galaxyproject/starforge/
[3] https://github.com/natefoo/pip/tree/linux-wheels
[4] https://bitbucket.org/natefoo/wheel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160114/74a2cbb9/attachment.html>