[Wheel-builders] Error from numpy wheel - any thoughts

[Nathaniel Smith]

On Tue, Mar 29, 2016 at 10:50 AM, Matthew Brett <matthew.brett at gmail.com> wrote:
> On Tue, Mar 29, 2016 at 4:29 AM, Olivier Grisel
> <olivier.grisel at ensta.org> wrote:
>>> You can replicate fairly simply with:
>>>
>>> pip install -f https://d9a97980b71d47cde94d-aae005c4999d7244ac63632f8b80e089.ssl.cf2.rackcdn.com
>>> numpy
>>> cd numpy/f2py/tests/src/kind
>>> f2py -c foo.f90 -m foo
>>> python -c 'import numpy; import foo'
>>
>> I can replicate. I can also force the use of the system fortran with
>> LD_PRELOAD (on ubuntu 14.04):
>>
>> LD_PRELOAD='/usr/lib/x86_64-linux-gnu/libgfortran.so.3' python -c
>> 'import numpy; import foo'
>
> This one is obviously a show-stopper - I believe that it means that
> any module compiled against gfortran (and presumably any shipped lib
> that numpy loads) will break after numpy has been imported.
>
> I don't understand the linux link rules well enough to know how to
> fix.  I guess we could rename the libraries that we ship / vendor into
> libs? (numpy_libgfortran.so etc).

Yeah, I'm not at all sure what's going on here and whether it's
intentional or a bug in glibc, and we should figure out what's
actually happening and why for our own understanding, but I'm pretty
confident that this is going to be either the solution or the
workaround :-).

I guess the general approach is that auditwheel should assign a unique
name to every vendored library. Some possible naming strategies:
  <wheel package name>-libgfortran.so
  <truncated sha-256 of libgfortran.so>-libgfortran.so
  <random uuid>-libgfortran.so

I guess the truncated-sha256 strategy might be best because if there
are lots of manylinux wheels all vendoring the same copy of
libgfortran, they will (or at least might) end up sharing it in memory
instead of mmap'ing multiple different copies of the identical .so
file, which is good? And this is certainly safe if the different
copies of libgfortran all have the same sha256.

And then auditwheel should use patchelf --replace-needed to modify the
DT_NEEDED in the various extension modules to point to the new name,
at the same time it's changing the RPATH. (--replace-needed doesn't
seem to be mentioned in most of the patchelf docs I can find, but it
is described here: https://github.com/NixOS/patchelf)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org