[Distutils] draft PEP: manylinux1

Fri Jan 22 04:33:11 EST 2016

On 21.01.2016 20:05, Matthew Brett wrote:
> Hi,
> 
> On Thu, Jan 21, 2016 at 2:05 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>> On 21.01.2016 10:31, Nick Coghlan wrote:
>>> On 21 January 2016 at 19:03, M.-A. Lemburg <mal at egenix.com> wrote:
>>>> By using the version based approach, we'd not run into this
>>>> problem and gain a lot more.
>>>
>>> I think it's better to start with a small core that we *know* works,
>>> then expand later, rather than trying to make the first iteration too
>>> wide. The "manylinux1" tag itself is versioned (hence the "1" at the
>>> end), so "manylinux2" may simply have *more* libraries defined, rather
>>> than newer ones.
>>
>> My argument is that the file based approach taken by the PEP
>> is too limiting to actually make things work for a large
>> set of Python packages.
>>
>> It will basically only work for packages that do not interface
>> to other external libraries (except for the few cases listed in
>> the PEP, e.g. X11, GL, which aren't always installed or
>> available either).
>>
>> IMO, testing the versions of a set of libraries is a safer
>> approach. It's perfectly fine to have a few dependencies
>> not work in a module because an optional system package is not
>> installed, e.g. say a package comes with UIs written in
>> Qt and one in GTK.
> 
> Please forgive my slowness, but I don't understand exactly what you
> mean.  Can you give a specific example?
> 
> Say my package depends on libpng.
> 
> Call the machine I'm installing on the client machine.
> 
> Are you saying that, when I build a wheel, I should specify to the
> wheel what versions of libpng I can tolerate on the the client
> machine, and if if the client does have a compatible version, then pip
> should raise an error, perhaps with a useful message about how to get
> libpng?
> 
> If you do mean that, how do you want the PEP changed?

I already posted a change proposal earlier on in the thread.
I'll repeat it here (with a minor enhancements):

"""
The ``manylinux1`` policy
=========================

For these reasons, we define a set of library versions that
are supported by a wide range of Linux distributions. We therefore
pick library versions which have been around for at least 5 years.

When using these external libraries, Python wheels should
only depend on library versions listed in the section below.

Python wheels are free to depend on additional libraries
not included in this set, however, care should be taken
that these additional libraries do not depend on later
versions of the listed libraries, e.g. OpenSSL libraries
compiled against the C library versions listed below.

The ``manylinux1`` policy thus encompasses a standard for which
versions of these external shared libraries a wheel may depend on,
and the maximum depended-upon symbol versions therein.

Future versions of the manylinux policy may include more
libraries, or move on to later versions.

The permitted external shared libraries versions for
``manylinux1``are: ::

    libgcc_s.so.1
    libstdc++.so.6
    ... only include the basic set of libs, no GUI or curses ...

"""

The idea is to not pin down the set of usable external libraries,
but instead pin down a set of versions for the most important
libraries wheels will depend on and then let the wheels use
other external libraries as necessary without version checks.

In more details:

If you want a wheel to run on many Linux distributions, you have
to make sure that the basic lib C and a few other utility
libraries are available and compatible with the ones you used
to build the wheel.

This can be addressed by defining a set of important libraries
and corresponding versions. You do not have to limit the overall
set of usable libraries for this, since less commonly used
libraries will usually have to be installed separately anyway.

For example, if a package needs a specific version of libpng,
the package author can document this and the user can then make
sure to install that particular version.

The PEP should only be concerned with the basic set of
libraries you typically need for a wheel, not any of the
less common ones. The X11 libs for example do not have to
be version pinned for the manylinux tag, since they are not
essential for the vast majority of Python packages (and here
I'm talking about the thousands of packages on PyPI, not the
few hundred mentioned earlier in the thread, which are
covered by Anaconda and Canopy).

By defining "manylinux1" in such a way you get:

 * broad compatibility of wheel files on Linux

 * full flexibility of wheels interfacing or wrapping
   to other external libraries not covered in the PEP

 * no lock-out of package authors who would like to push
   wheel files for their packages to PyPI, but happen to
   use libraries not in the predefined list of the original
   draft PEP

I left out the other details I mentioned (symbol versioning
and dealing with different architectures) to focus on
pinning libraries vs. pinning versions for now. Later on,
we'd have to apply a similar strategy to other platforms
as well, e.g. *BSD, AIX, Solaris, etc.

Since we're targeting Linux, it may be helpful to base
the list of libraries and versions on the Linux Standard
Base (LSB), since one of the main ideas behind the LSB is
binary compatibility between distributions:

http://refspecs.linuxfoundation.org/lsb.shtml

If people on this list still can't see the benefit of
just pinning down versions of specific library files
for "manylinux1" over limiting the set of allowed libraries
and forcing people to embed any other libs in the wheel files,
I guess I'll just have to write up a competing or additional
tag PEP to enable all package authors to publish wheels for
Linux on PyPI, but this won't happen until February.

Cheers,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Jan 22 2016)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/