[Numpy-discussion] Defining a base linux-64 environment [was: Should I use pip install numpy in linux?]

[Robert McGibbon]

Hi all,

I went ahead and tried to collect a list of all of the libraries that could
be considered to constitute the "base" system for linux-64. The strategy I
used was to leverage off the work done by the folks at Continuum by
searching through their pre-compiled binaries from
https://repo.continuum.io/pkgs/free/linux-64/ to find shared libraries that
were dependened on (according to ldd)  that were not accounted for by the
declared dependencies that each package made known to the conda package
manager.

The full list of these system libraries, sorted in from
most-commonly-depend-on to rarest, is below. There are 158 of them.

['linux-vdso.so.1', 'libc.so.6', 'libpthread.so.0', 'libm.so.6',
'libdl.so.2', 'libutil.so.1', 'libgcc_s.so.1', 'libstdc++.so.6',
'libexpat.so.1', 'librt.so.1', 'libpng12.so.0', 'libcrypt.so.1',
'libffi.so.6', 'libresolv.so.2', 'libkeyutils.so.1', 'libcom_err.so.2',
'libp11-kit.so.0', 'libkrb5.so.26', 'libheimntlm.so.0', 'libtasn1.so.6',
'libheimbase.so.1', 'libgssapi.so.3', 'libroken.so.18', 'libhcrypto.so.4',
'libhogweed.so.4', 'libnettle.so.6', 'libhx509.so.5', 'libwind.so.0',
'libgnutls-deb0.so.28', 'libasn1.so.8', 'libgmp.so.10', 'libsasl2.so.2',
'libidn.so.11', 'librtmp.so.1', 'liblber-2.4.so.2', 'libldap_r-2.4.so.2',
'libXdmcp.so.6', 'libX11.so.6', 'libXau.so.6', 'libxcb.so.1',
'libgssapi_krb5.so.2', 'libkrb5.so.3', 'libk5crypto.so.3',
'libkrb5support.so.0', 'libicudata.so.55', 'libicuuc.so.55',
'libhdf5_serial.so.10', 'libcurl-gnutls.so.4', 'libhdf5_serial_hl.so.10',
'libtinfo.so.5', 'libgcrypt.so.20', 'libgpg-error.so.0', 'libnsl.so.1',
'libXext.so.6', 'libncursesw.so.5', 'libpanelw.so.5', 'libXrender.so.1',
'libjbig.so.0', 'libpcre.so.3', 'libglib-2.0.so.0',
'libnvidia-tls.so.352.41', 'libnvidia-glcore.so.352.41', 'libGL.so.1',
'libuuid.so.1', 'libSM.so.6', 'libICE.so.6', 'libgobject-2.0.so.0',
'libgfortran.so.1', 'liblzma.so.5', 'libXt.so.6', 'libgmodule-2.0.so.0',
'libXi.so.6', 'libgstpbutils-1.0.so.0', 'liborc-0.4.so.0',
'libgstreamer-1.0.so.0', 'libgsttag-1.0.so.0', 'libgstvideo-1.0.so.0',
'libxslt.so.1', 'libaudio.so.2', 'libjpeg.so.8', 'libgstaudio-1.0.so.0',
'libgstbase-1.0.so.0', 'libgstapp-1.0.so.0', 'libz.so.1',
'libgthread-2.0.so.0', 'libfreetype.so.6', 'libfontconfig.so.1',
'libdbus-1.so.3', 'libsystemd.so.0', 'libltdl.so.7', 'libGLU.so.1',
'libsqlite3.so.0', 'libpgm-5.1.so.0', 'libgomp.so.1', 'libxcb-render.so.0',
'libxcb-shm.so.0', 'libncurses.so.5', 'libxml2.so.2', 'libXss.so.1',
'libXft.so.2', 'libtk.so', 'libtcl.so', 'libasound.so.2',
'libharfbuzz.so.0', 'libpixman-1.so.0', 'libgio-2.0.so.0',
'libXinerama.so.1', 'libselinux.so.1', 'libXcomposite.so.1',
'libthai.so.0', 'libXdamage.so.1', 'libgdk-x11-2.0.so.0',
'libpangoft2-1.0.so.0', 'libcairo.so.2', 'libpangocairo-1.0.so.0',
'libdatrie.so.1', 'libatk-1.0.so.0', 'libXcursor.so.1', 'libXfixes.so.3',
'libgraphite2.so.3', 'libgdk_pixbuf-2.0.so.0', 'libgtk-x11-2.0.so.0',
'libquadmath.so.0', 'libpango-1.0.so.0', 'libXrandr.so.2',
'libgfortran.so.3', 'libjson-c.so.2', 'libshiboken-python2.7.so.1.1',
'libogg.so.0', 'libvorbis.so.0', 'libatlas.so.3', 'libcurl.so.4',
'libhdf5.so.9', 'libodbcinst.so.1', 'libpcap.so.0.9', 'libnetcdf.so.7',
'libblas.so.3', 'libpulse.so.0', 'libcaca.so.0', 'libgstreamer-0.10.so.0',
'libXxf86vm.so.1', 'libhdf5_hl.so.9', 'libpulse-simple.so.0',
'libasyncns.so.0', 'libwrap.so.0', 'libvorbisenc.so.2', 'libmagic.so.1',
'libssl.so.1.0.0', 'libFLAC.so.8', 'libSDL-1.2.so.0', 'libsndfile.so.1',
'libslang.so.2', 'libglapi.so.0', 'libaio.so.1',
'libgstinterfaces-0.10.so.0', 'libpulsecommon-6.0.so', 'libjpeg.so.62',
'libcrypto.so.1.0.0']


This list actually contains a fair number of false positives, so it would
need to be pruned manually. If you stare at it a little while, you might
see some libraries in there that you recognize that shouldn't be part of
the base system, like libatlas.so.3.

This gist https://gist.github.com/rmcgibbo/a13e7623c38ec54fcc93 contains
some more detailed data -- for each of libraries in the list above, it
gives a list of names of the packages that depend on this library. For
example, for libatlas.so.3, the there is only a single package which
depends on it, ["scikit-learn-0.11-np16py27_ce0"]. So, probably a bug.

"libgfortran.so.1" is also in the list. It's depended on by
["cvxopt-1.1.6-py27_0", "cvxopt-1.1.7-py27_0", "cvxopt-1.1.7-py34_0",
"cvxopt-1.1.7-py35_0", "numpy-1.5.1-py27_1", "numpy-1.5.1-py27_3",
"numpy-1.5.1-py27_4", "numpy-1.5.1-py27_ce0", "numpy-1.6.2-py27_1",
"numpy-1.6.2-py27_3", "numpy-1.6.2-py27_4", "numpy-1.6.2-py27_ce0",
"numpy-1.7.0-py27_0", "numpy-1.7.0b2-py27_ce0", "numpy-1.7.0rc1-py27_0",
"numpy-1.7.1-py27_0", "numpy-1.7.1-py27_2", "numpy-1.8.0-py27_0",
"numpy-1.8.1-py27_0", "numpy-1.8.1-py34_0", "numpy-1.8.2-py27_0",
"numpy-1.8.2-py34_0", "numpy-1.9.0-py27_0", "numpy-1.9.0-py34_0",
"numpy-1.9.1-py27_0", "numpy-1.9.1-py34_0", "numpy-1.9.2-py27_0",
"numpy-1.9.2-py34_0"].

Note that this list of numpy versions doesn't include the latest ones --
all of the numpy-1.10 binaries made by Continuum pick up libgfortan from a
conda package and don't depend on it being provided by the system. Also,
the final '_0' or '_1' segment of many of these package names is the build
number, which is to make a new release of the same release of a package,
usually because of a packaging problem. So many of these packages were
probably built incorrectly and superseded by new builds with a higher build
number.

So it's not perfect. But it might be a useful starting place.

-Robert
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20160109/616bf487/attachment.html>