[Numpy-discussion] Defining a base linux-64 environment [was: Should I use pip install numpy in linux?]

[Robert McGibbon]

I started working on a tool for checking linux wheels for "manylinux"
compatibility, and fixing them up if possible, based on the same ideas as
Matthew Brett's delocate <https://github.com/matthew-brett/delocate> for OS
X. Current WIP code, if anyone wants to help / throw penuts, is here:
https://github.com/rmcgibbo/deloc8.

It's currently fairly modest and can only list non-whitelisted external
shared library dependencies, and verify that sufficiently old versioned
symbols for glibc and its ilk are used.

-Robert

On Sun, Jan 10, 2016 at 1:19 AM, Robert McGibbon <rmcgibbo at gmail.com> wrote:

> Hi all,
>
> I followed Nathaniel's advice and restricted the search down to the
> packages included in the Anaconda release (as opposed to all of the
> packages in their repositories), and fixed some technical issues with the
> way I was doing the analysis.
>
> The new list is much smaller. Here are the shared libraries that the
> components of Anaconda require that the system provides on Linux 64:
>
> libpanelw.so.5, libncursesw.so.5, libgcc_s.so.1, libstdc++.so.6,
> libm.so.6, libdl.so.2, librt.so.1, libcrypt.so.1, libc.so.6, libnsl.so.1,
> libutil.so.1, libpthread.so.0, libX11.so.6, libXext.so.6,
> libgobject-2.0.so.0, libgthread-2.0.so.0, libglib-2.0.so.0,
> libXrender.so.1, libICE.so.6, libSM.so.6, libGL.so.1.
>
> Many of these libraries are required simply for the interpreter. The
> remaining ones that aren't required by the interpreter are, but are
> required by some other package in Anaconda are:
>
> libgcc_s.so.1, libstdc++.so.6, libXext.so.6, libSM.so.6,
> libgthread-2.0.so.0, libgobject-2.0.so.0, libglib-2.0.so.0, libICE.so.6,
> libXrender.so.1, and libGL.so.1.
>
> Most of these are parts of X11 required by Qt (
> http://doc.qt.io/qt-5/linux-requirements.html).
>
> -Robert
>
>
>
> On Sat, Jan 9, 2016 at 4:42 PM, Robert McGibbon <rmcgibbo at gmail.com>
> wrote:
>
>> > Maybe a better approach would be to look at what libraries are used on
>> by an up-to-date default Anaconda install (on the assumption that this
>> is the best tested configuration)
>>
>> That's not a bad idea. I also have a couple other ideas about how to
>> filter
>> this based on using debian popularity-contests and the package graph. I
>> will report back when I have more info.
>>
>> -Robert
>>
>> On Sat, Jan 9, 2016 at 3:04 PM, Nathaniel Smith <njs at pobox.com> wrote:
>>
>>> On Sat, Jan 9, 2016 at 3:52 AM, Robert McGibbon <rmcgibbo at gmail.com>
>>> wrote:
>>> > Hi all,
>>> >
>>> > I went ahead and tried to collect a list of all of the libraries that
>>> could
>>> > be considered to constitute the "base" system for linux-64. The
>>> strategy I
>>> > used was to leverage off the work done by the folks at Continuum by
>>> > searching through their pre-compiled binaries from
>>> > https://repo.continuum.io/pkgs/free/linux-64/ to find shared
>>> libraries that
>>> > were dependened on (according to ldd)  that were not accounted for by
>>> the
>>> > declared dependencies that each package made known to the conda package
>>> > manager.
>>> >
>>> > The full list of these system libraries, sorted in from
>>> > most-commonly-depend-on to rarest, is below. There are 158 of them.
>>> [...]
>>> > So it's not perfect. But it might be a useful starting place.
>>>
>>> Unfortunately, yeah, it looks like there's a lot of false positives in
>>> here :-(. For example your list contains liblzma and libsqlite, but
>>> both of these are shipped as dependencies of python itself. So
>>> probably someone just forgot to declare the dependency explicitly, but
>>> got away with it because the libraries were pulled in anyway.
>>>
>>> Maybe a better approach would be to look at what libraries are used on
>>> by an up-to-date default Anaconda install (on the assumption that this
>>> is the best tested configuration), and then erase from the list all
>>> libraries that are shipped by this configuration (ignoring declared
>>> dependencies since those seem to be unreliable)? It's better to be
>>> conservative here, since the end goal is to come up with a list of
>>> external libraries that we're confident have actually been tested for
>>> compatibility by lots and lots of different users.
>>>
>>> -n
>>>
>>> --
>>> Nathaniel J. Smith -- http://vorpus.org
>>> _______________________________________________
>>> NumPy-Discussion mailing list
>>> NumPy-Discussion at scipy.org
>>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160111/29671833/attachment.html>