[Distutils] draft PEP: manylinux1

Nathaniel Smith njs at pobox.com
Thu Jan 21 17:38:36 EST 2016

On Jan 21, 2016 12:08 PM, "Donald Stufft" <donald at stufft.io> wrote:
>> On Jan 21, 2016, at 1:43 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> On Jan 21, 2016 9:32 AM, "Donald Stufft" <donald at stufft.io> wrote:
>> >
>> >
>> > > On Jan 20, 2016, at 10:55 PM, Nathaniel Smith <njs at pobox.com> wrote:
>> > >
>> > > The permitted external shared libraries are: ::
>> > >
>> > >    libpanelw.so.5
>> > >    libncursesw.so.5
>> > >    libgcc_s.so.1
>> > >    libstdc++.so.6
>> > >    libm.so.6
>> > >    libdl.so.2
>> > >    librt.so.1
>> > >    libcrypt.so.1
>> > >    libc.so.6
>> > >    libnsl.so.1
>> > >    libutil.so.1
>> > >    libpthread.so.0
>> > >    libX11.so.6
>> > >    libXext.so.6
>> > >    libXrender.so.1
>> > >    libICE.so.6
>> > >    libSM.so.6
>> > >    libGL.so.1
>> > >    libgobject-2.0.so.0
>> > >    libgthread-2.0.so.0
>> > >    libglib-2.0.so.0
>> >
>> >
>> > Forgive my, probably stupid, question… but why these libraries? Are
>> > libraries unable to be reasonably statically linked like glibc is?
>> It's not a stupid question at all :-).
>> What we want is a list of libraries that are present, and compatible,
and relevant, across the Linux distributions that most people use in
>> Unfortunately, there isn't really any formal specification or published
data on this -- the only way to make such a list is to make a guess, start
distributing software to lots of people based on your guess, wait for bug
reports to come in, edit your list to avoid the problematic libraries, add
some more libraries to your list to cover the new packages you've added to
your distribution and where you think the risk is a worthwhile trade off
versus just static linking, then repeat until you stop getting bug reports.
>> This is expensive and time consuming and requires data we don't have, so
we delegated: the list in the PEP is exactly the set of libraries that
Continuum's Anaconda distribution links to and a subset of the libraries
that Enthought's Canopy links to [1]. They've both been going through the
loop above for years, with lots of packages, and lots of lots of downloads,
so we're piggybacking off their effort.
>> Here's the list of packages in Anaconda:
>> http://docs.continuum.io/anaconda/pkg-docs
> I guess my underlying question is, if we’re considering static linking
(or shipping the .so dll style) to be good enough for everything not on
this list, why are these specific packages on the list? Why are we not
selecting the absolute bare minimum packages that you *cannot* reasonably
static link or ship the .so?

So, there are tradeoffs here of course. Some libraries are more or less
difficult to statically link, and vendoring libraries does have costs in
terms of maintenance, download sizes, memory usage, and so forth... it's
complicated and varies from library to library. I suspect that even the
people who have been dealing with these problems full time for years can't
necessarily remember all the details of what happened that made them take
particular decisions. (Both Anaconda and Canopy have made the decision to
vendor libz, even though you'd think that libz would be a prime candidate
for getting from the package manager, being both ubiquitous and
historically notable for all the problems that vendoring it has caused in
the past. Why do they do this? I don't know, and I haven't shipped millions
of Linux binary packages so my guesses probably aren't worth that much
:-).) The list probably isn't perfect, but we do know that it will work,
which is an advantage that other lists don't have.

The other thing is the Mason-Dixon principle: we have to draw this
<censored> line somewhere ;-).

You're suggesting more conservative, Marc-Andre and Matthais want it to be
more liberal, and of course any self respecting Linux geek has a guess
about exactly which libraries should be included (myself included). This
bikeshed won't paint itself! So rather than open the door to endless petty
debate and fiddling, just sticking with a known good list is pretty

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160121/f600254b/attachment.html>

More information about the Distutils-SIG mailing list