<div dir="ltr"><div>OK,<br><br></div>I'll try to stop being emotional here :-)<br><div><div><br><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="">2016-01-22 3:47 GMT+01:00 Chris Barker - NOAA Federal <<a href="mailto:chris.barker@noaa.gov">chris.barker@noaa.gov</a>>:<br>
><br>
> I'm skeptical because I<br>
> tried to to that for years for OS-X and it was just too much to do. And the<br>
> infrastructure was there.<br>
</span></blockquote><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">My point is that once we have clearly defined best-practices for<br>
packaging and convenient tools to build the packages automatically and<br>
test that they work as expected (e.g. free hosted CI that support<br>
running an old centos-based docker container), I am rather confident<br>
that the community will do the work.<br></blockquote><div><br></div><div>OK -- here is the emotional part -- I worked for years to try to get support to: <br><br>"clearly defined best-practices for packaging and convenient tools to build the packages automatically"<br><br></div><div>Primarily for OS-X. I got zero support -- nada -- nothing. Really. A handful of people did their own thing to support the community, but no cooperation of standards -- each package built was done with a lot of hand work and each in its own way. So when i found the conda community working on common tools and methods, it was very refreshing.<br><br></div><div>But OK -- maybe times have changed.<br><br></div><div>By the way, I'm in the middle of some build hell with conda -- it's doing some really weird stuff with finding shared libs provided by other conda packages -- so maybe I'm open to looking at wheels again :-)<br><br></div><div>But my concern is not the base libs -- that will get Linux on par with Windows and OS-X. My concern is with third party libs, what's the plan there?<br><br></div><div>1) each package that needs a third partly lib statically links it in.<br clear="all"></div></div>2) each package that needs a third partly lib provides it, linked with a realtive path (IIUC, that's how most Windows packages are done.<br></div><div class="gmail_extra">3) We establish some standard for providing binary libs as wheels, so that other packages can depend on them and link to them.<br><br></div><div class="gmail_extra">1) is a pain int he %^# with gcc and linux, which really likes to dynamically link<br><br></div><div class="gmail_extra">2) seems to have been made pretty easy with auditwheel -- nice!<br><br></div><div class="gmail_extra">3) seems like the "proper" way to go. somehow making everyone figure out how to build and shop those deps, and then bloating the wheels and installations and binaries feels wrong to me.<br><br></div><div class="gmail_extra">We've been using the example of, say, libpng, in this discussion --that's a good example, because it's pretty commonly used, but not part of all base distributions. but it's also pretty easy to build and pretty small.<br><br></div><div class="gmail_extra">So let's look at a couple examples I've dealt with:<br><br></div><div class="gmail_extra">The netcdf / hdf5 stack -- there is a pynetcd4 package, which requires the c netcdf lib, which requires libhdf5, which requires libcurl, zlib, (anda few others, I think) non-trivial to build and ship. Also, there are at least two other commonly used python packages I know of that use hdf5: pytables and h5py.<br></div><div class="gmail_extra"><br></div><div class="gmail_extra">So it would be really nice if all these python packages didn't need to solve the build issues and ship all these libs themselves, resulting in possibly incompatible version all loaded into python all at once. Oh, and as i happens, I've got my obscure python package that uses a bunch of C code that also needs libnetcdf....<br><br></div><div class="gmail_extra">Then there is the OpenGIS stack: there is the geos lib and proj4 lib, that most others things are built on. There is the GDAL/OGR lib, that requires those, plus (optionally), a lot of other ugly libs (this is ugly to build, believe me). And GDAL comes with its own python bindings, but there is also shapely, that wraps geos, and pyproj4 that wraps proj4, and fiona that wraps OGR, and .... A big interconnected web. [oh, fiona and shapely also required numpy at the binary level...)<br><br></div><div class="gmail_extra">I can only imagine that the image processing or video or audio processing communities have similar piles of interconnected packages. (I know I've tried, unsuccessfully, to get FFMPEG to work...)<br><br></div><div class="gmail_extra">So all this requires, I think, (3) to get anywhere -- is the community ready to support such an effort? And this going to requires some more tooling, too.<br><br></div><div class="gmail_extra">Somewhere on this thread, someone suggested there may be a videolinuxapi, or some such. Perhaps the better way to to have a core base (such as manylinux), and then a handful of binary lib collections as a single wheel:<br><br></div><div class="gmail_extra">osgeowheel<br></div><div class="gmail_extra">hdf-wheel<br></div><div class="gmail_extra">audio-wheel <br></div><div class="gmail_extra">image-wheel<br><br></div><div class="gmail_extra">hmm -- kind of liking that idea, actually.<br><br></div><div class="gmail_extra">-CHB<br></div><div class="gmail_extra"><br></div><div class="gmail_extra">-- <br><div class="gmail_signature"><br>Christopher Barker, Ph.D.<br>Oceanographer<br><br>Emergency Response Division<br>NOAA/NOS/OR&R (206) 526-6959 voice<br>7600 Sand Point Way NE (206) 526-6329 fax<br>Seattle, WA 98115 (206) 526-6317 main reception<br><br><a href="mailto:Chris.Barker@noaa.gov" target="_blank">Chris.Barker@noaa.gov</a></div>
</div></div></div></div>