[Wheel-builders] libpythonX.Y.so.1

[Robert T. McGibbon]

Hey guys,

Sorry I'm out of the loop. Nathaniel correctly diagnosed the problem as
thesis stuff. Quick responses below:

> 1. With SOABI tag support now available for Python 2.X, would a PR adding
UCS-2 builds of Python to the docker image be accepted?

+1 from me. I think there's an issue open for this. You just need to decide
something rational for the directory layout of which Python goes where and
what the symlinks will be.

I think this will also require getting the UCS-2 detection code properly
removed from auditwheel (https://github.com/pypa/auditwheel/pull/17). Sorry
I haven't done that yet.

> 2. Is anyone working on the pip side of the changes necessary to install
manylinux1 wheels? If not, I'll do this.

IIRC, there is an open PR to pip (https://github.com/pypa/pip/pull/3497). I
think adding your voice there might help move it along. I think that PR is
ready to merge, but just lacking the last necessary push from a pip
maintainer to merge it.


> Also, I wonder if we should vendor auditwheel into wheel itself so you
can directly build manylinux1 wheels.

IMO it's too early for that. auditwheel is not mature, and besides you
really need the docker env. On that note, any help getting auditwheel in
better shape -- especially UX-wise if you have any free cycles -- would be
totally awesome.


On Wed, Mar 2, 2016 at 9:23 AM, Nate Coraor <nate at bx.psu.edu> wrote:

> On Wed, Mar 2, 2016 at 11:05 AM, Nate Coraor <nate at bx.psu.edu> wrote:
>
>> On Tue, Mar 1, 2016 at 4:58 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>>> On Tue, Mar 1, 2016 at 11:18 AM, Nate Coraor <nate at bx.psu.edu> wrote:
>>> > So I am working on building my first set of manylinux1 wheels. The
>>> first
>>> > candidate is psycopg2. I was able to generate the wheel which brings
>>> in all
>>> > the necessary libs:
>>> [...]
>>> > However, attempting to use this wheel on Ubuntu 14.04 results in a
>>> failure
>>> > to load libpython2.7.so.1:
>>> >
>>> >   ImportError: libpython2.7.so.1.0: cannot open shared object file: No
>>> such
>>> > file or directory
>>> >
>>> > PEP 513 addresses libpythonX.Y.so.1 but only to say that it does not
>>> need to
>>> > be linked, nor should it be. Is my understanding correct, then, that I
>>> would
>>> > need to fix psycopg2 myself to *not* link to libpython (if this works -
>>> > won't there be unresolved symbols at link time?)? Is there a more
>>> general
>>> > solution to this problem?
>>>
>>> Right, you need to not link to libpython. This does work -- it turns
>>> out that the linker doesn't check for unresolved symbols unless you
>>> specifically ask it to (via -Wl,--no-undefined or similar), I guess
>>> because the people who write linkers consider what we're doing here to
>>> be a valid use case :-).
>>>
>>> Are you using the docker images to build these? I thought we fixed it
>>> to not link to libpython by default, but if not then we should... (If
>>> your python is built with --disable-shared, which is the default when
>>> building from source, then by default any wheels you build with that
>>> python will not link to libpython. Unfortunately CentOS's python is
>>> built with --enable-shared. This is one of the reasons that the docker
>>> images build their own python instead of using the one shipped by
>>> CentOS.)
>>>
>>
>> I'm using the docker image and the purpose-built Python, yeah. It's a
>> container that I started a few weeks ago though, so I can pull a new
>> version of the image and see if it's fixed. The Python in my version of the
>> image definitely has a shared libpython in the purpose-built Python build:
>>
>> libpython2.7.so.1.0 => /opt/2.7.11/lib/libpython2.7.so.1.0
>> (0x00007fab9bdfd000)
>>
>
> Yeah, sure enough, this is just due to an out-of-date image. The one I was
> using only had the shared libpython, the latest one only has the static. A
> psycopg2 wheel built in the new image is working.
>
> Two further questions now that this works for me:
>
> 1. With SOABI tag support now available for Python 2.X, would a PR adding
> UCS-2 builds of Python to the docker image be accepted?
>
> 2. Is anyone working on the pip side of the changes necessary to install
> manylinux1 wheels? If not, I'll do this.
>
> Also, I wonder if we should vendor auditwheel into wheel itself so you can
> directly build manylinux1 wheels.
>
>
>>
>>>
>>> Also, I guess Robert is a bit distracted by thesis-ing, but I think he
>>> was at least planning to make auditwheel check for and reject wheels
>>> that link to libpython (or possibly repair them -- I think patchelf
>>> can do that). Perhaps he'll speak up about that, or I'm sure he'd
>>> welcome a pull request :-). I guess that's the more-general solution
>>> you're looking for.
>>>
>>
>> A good idea, but if the docker image isn't supposed to produce wheels
>> that link libpython, that's also a good general solution for me. =)
>>
>>
>>>
>>> > Related, some of these are concerning to me. Although CentOS 5 receives
>>> > updates, so its OpenSSL should be secure and bug-free, this is still
>>> OpenSSL
>>> > 0.9.8e. And bundling SELinux and Kerberos/GSSAPI libs also makes me a
>>> bit
>>> > worried.
>>>
>>> I agree, it is worrying! But there's not much that can be done about
>>> it: if you want to make a single wheel that works across lots of linux
>>> distributions, and you need openssl, then you need to ship openssl,
>>> and that means that you need to take responsibility for keeping an eye
>>> out for security releases and respin your wheels when they happen.
>>>
>>> I guess the psycopg2 developers are already doing this for their
>>> Windows wheels, so perhaps they'll be willing to take on that
>>> responsibility for Linux wheels too? Or there might be things we can
>>> do to mitigate this somewhat: e.g., one can imagine the different
>>> folks who need to ship openssl getting together and building some
>>> shared infrastructure, so that they can share the burden -- like have
>>> one set of build scripts to update openssl and rebuild all their
>>> packages. Or once the native-libraries-in-wheels effort gets off the
>>> ground (sorry, I've been meaning to send out an email to kick that off
>>> but have been distracted by other things), then they'll be able to
>>> share a single openssl wheel and then that'll also centralize the
>>> update process. But fundamentally, the unavoidable fact is that if
>>> someone's not willing to take on responsibility for this somehow, then
>>> probably it's a bad idea for that person to ship manylinux wheels (or
>>> windows wheels, for that matter).
>>>
>>
>> Sure, this makes sense to me. I wonder if it would be a good idea to have
>> something like Conda in the docker image that would make installing
>> up-to-date versions of these sorts of dependencies trivial.
>>
>> Thanks,
>> --nate
>>
>> P.S., the archives for this list aren't showing up on mail.python.org,
>> could you take a look in the list admin settings and see if they're enabled?
>>
>>
>>
>>>
>>> -n
>>>
>>> --
>>> Nathaniel J. Smith -- https://vorpus.org
>>>
>>
>>
>
> _______________________________________________
> Wheel-builders mailing list
> Wheel-builders at python.org
> https://mail.python.org/mailman/listinfo/wheel-builders
>
>


-- 
-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20160302/78cbe03f/attachment-0001.html>