[Distutils] Working toward Linux wheel support

[Nate Coraor]

Hi all,

Thanks for the lively debate - I sent the message to start the thread and
then had a week's vacation - and I appreciate the discussion that took
place in the interim. I've encountered all of the problems discussed here,
especially the dependencies both with Python and other attempts at package
management in distributed heterogeneous systems. For Galaxy's controlled
ecosystem we deal with this using static linking (e.g. our psycopg2 egg is
statically linked to a version of libpq5 built for the egg), but this is
not an ideal solution for a variety of reasons that I doubt I need to
explain.

As for the Python side of things, I certainly agree with the point raised
by Leonardo that an ENOENT is probably easier for most to debug than a
missing Python.h. For what it's worth, some libraries like PyYAML have a
partial solution for this: If libyaml.so.X is not found at runtime, it
defaults to a pure Python implementation. This is not ideal, for sure, nor
will it be possible for all packages, and it depends on the package author
to implement a pure Python version, but it does avoid an outright runtime
failure.

I hope - and I think that Nick is advocating for this - that incremental
improvements can be made, rather than what's been the case so far:
identifying the myriad of problems and the shortcomings of the packaging
format(s), only to stall on making progress towards a solution.

As to the comments regarding our needs being met today with a wheelhouse,
while this is partially true (e.g. we've got our own PyPI up at
https://wheels.galaxyproject.org), we still need to settle on an
overspecified tag standard and fix SOABI support in Python 2.x in order to
avoid having to ship a modified wheel/pip with Galaxy.

Is there any specific direction the Distutils-SIG would like me to take to
continue this work?

Thanks,
--nate

On Mon, Jul 27, 2015 at 10:19 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 21 July 2015 at 04:37, Paul Moore <p.f.moore at gmail.com> wrote:
> > On 20 July 2015 at 18:37, Chris Barker <chris.barker at noaa.gov> wrote:
> >> sure -- but isn't that use-case already supported by wheel -- define
> your
> >> own wheelhouse that has the ABI you know you need, and point pip to it.
> >
> > I presume the issue is wanting to have a single shared wheelhouse for
> > a (presumably limited) number of platforms. So being able to specify a
> > (completely arbitrary) local platform name at build and install time
> > sounds like a viable option.
>
> While supporting multiple distros in a single repo is indeed one use
> case (and the one that needs to be solved to allow distribution via
> PyPI), the problem I'm interested in isn't the "success case" where a
> precompiled Linux wheel stays nicely confined to the specific
> environment it was built to target, but rather the failure mode where
> a file "escapes".
>
> Currently, there's nothing in a built Linux wheel file to indicate its
> *intended* target environment, which makes debugging ABI mismatches
> incredibly difficult. By contrast, if the wheel filename says "Fedora
> 22" and you're trying to run it on "Ubuntu 14.04" and getting a
> segfault, you have a pretty good hint as to the likely cause of your
> problem.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150727/a0ab62d8/attachment.html>