[Distutils] draft PEP: manylinux1

Chris Barker chris.barker at noaa.gov
Sat Jan 23 21:19:12 EST 2016


I'll try to stop being emotional here :-)

2016-01-22 3:47 GMT+01:00 Chris Barker - NOAA Federal <chris.barker at noaa.gov
> >:
> >
> >  I'm skeptical because I
> > tried to to that for years for OS-X and it was just too much to do. And
> the
> > infrastructure was there.

> My point is that once we have clearly defined best-practices for
> packaging and convenient tools to build the packages automatically and
> test that they work as expected (e.g. free hosted CI that support
> running an old centos-based docker container), I am rather confident
> that the community will do the work.

OK -- here is the emotional part -- I worked for years to try to get
support to:

"clearly defined best-practices for packaging and convenient tools to build
the packages automatically"

Primarily for OS-X. I got zero support -- nada -- nothing. Really. A
handful of people did their own thing to support the community, but no
cooperation of standards -- each package built was done with a lot of hand
work and each in its own way. So when i found the conda community working
on common tools and methods, it was very refreshing.

But OK -- maybe times have changed.

By the way, I'm in the middle of some build hell with conda -- it's doing
some really weird stuff with finding shared libs provided by other conda
packages -- so maybe I'm open to looking at wheels again :-)

But my concern is not the base libs -- that will get Linux on par with
Windows and OS-X. My concern is with third party libs, what's the plan

1)  each package that needs a third partly lib statically links it in.
2)  each package that needs a third partly lib provides it, linked with a
realtive path (IIUC, that's how most Windows packages are done.
3) We establish some standard for providing binary libs as wheels, so that
other packages can depend on them and link to them.

1) is a pain int he %^# with gcc and linux, which really likes to
dynamically link

2) seems to have been made pretty easy with auditwheel -- nice!

3) seems like the "proper" way to go. somehow making everyone figure out
how to build and shop those deps, and then bloating the wheels and
installations and binaries feels wrong to me.

We've been using the example of, say, libpng, in this discussion --that's a
good example, because it's pretty commonly used, but not part of all base
distributions. but it's also pretty easy to build and pretty small.

So let's look at a couple examples I've dealt with:

The netcdf / hdf5 stack -- there is a pynetcd4 package, which requires the
c netcdf lib, which requires libhdf5, which requires libcurl, zlib, (anda
few others, I think) non-trivial to build and ship. Also, there are at
least two other commonly used python packages I know of that use hdf5:
pytables and h5py.

So it would be really nice if all these python packages didn't need to
solve the build issues and ship all these libs themselves, resulting in
possibly incompatible version all loaded into python all at once. Oh, and
as i happens, I've got my obscure python package that uses a bunch of C
code that also needs libnetcdf....

Then there is the OpenGIS stack: there is the geos lib and proj4 lib, that
most others things are built on. There is the GDAL/OGR lib, that requires
those, plus (optionally), a lot of other ugly libs (this is ugly to build,
believe me). And GDAL comes with its own python bindings, but there is also
shapely, that wraps geos, and pyproj4 that wraps proj4, and fiona that
wraps OGR, and .... A big interconnected web. [oh, fiona and shapely also
required numpy at the binary level...)

I can only imagine that the image processing or video or audio processing
communities have similar piles of interconnected packages. (I know I've
tried, unsuccessfully, to get FFMPEG to work...)

So all this requires, I think, (3) to get anywhere -- is the community
ready to support such an effort? And this going to requires some more
tooling, too.

Somewhere on this thread, someone suggested there may be a videolinuxapi,
or some such. Perhaps the better way to to have a core base (such as
manylinux), and then a handful of binary lib collections as a single wheel:


hmm -- kind of liking that idea, actually.



Christopher Barker, Ph.D.

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20160123/8fbdf7f7/attachment.html>

More information about the Distutils-SIG mailing list