[Distutils] Current Python packaging status (from my point of view)

Matthew Brett matthew.brett at gmail.com
Wed Nov 2 20:02:52 EDT 2016


Hi,

On Wed, Nov 2, 2016 at 4:08 PM, Chris Barker <chris.barker at noaa.gov> wrote:
> Hey Matthew,
>
>> > [1] There seems to be some animosity among pip supporters and conda
>> > supports, or at least a perception that there is.
>>
>> I don't know whether there is animosity, but there is certainly
>> tension.  Speaking personally, I care a lot about having the option to
>> prefer pip.
>
>
> indeed -- and you have made herculean efforts to make that possible :-)
>
>>
>> There are others in the scientific community who feel we
>> should standardize on conda.   I think this is the cause of Chris'
>> frustration.  If we could all use conda instead of pip, then this
>> would make it easier for him in terms of packaging, because he would
>> not have to support pip  (Chris please correct me if I'm wrong).
>
>
> yup -- nor would you :-)
>
> My personal frustration comes with my history -- I've been in this community
> for over 15 years -- and I spent a lot of effort back in the day to make
> packages available for the Mac (before pypi, pip, before wheels, ...). And I
> found I was constantly re-figuring out how to build the dependent libraries
> needed, and statically linking everything, etc... It sucked. A lot of work,
> and not at all fun (for me, anyway -- maybe some folks enjoy that sort of
> thing).
>
> So I started a project to share that effort, and to build a bit of
> infrastructure. I also started looking into how to handle dependent libs
> with pip+wheel -- I got no support whatsoever for that. I got frustrated
> 'cause it was too hard, and I also felt like I was fighting the tools. I did
> not get far.
>
> Mathew ended up picking up that effort and really making it work, and had
> gotten all the core SciPY stuff out there as binary wheels -- really great
> stuff.
>
> But then I discovered conda -- and while I was resistant at first, I found
> that it was a much nicer environment to do what I needed to do. It started
> not because of anything specific about conda, but because someone had
> already built a bunch of the stuff I needed -- nice!
>
> But the breaking point was when I needed to build a package of my own: py_gd
> -- it required libgd, which required libpng, libjpeg, libtiff --- some of
> those come out of the box on a Mac, it's all available from homebrew, and
> all pretty common on Linux -- but I have users that are on Windows, or on a
> Mac and can't grok homebrew or macports. And, frankly, neither do all of my
> development team.

Anaconda has an overwhelming advantage on Windows, in that Continuum
can bear the licensing liabilities enforced by the Intel Fortran
compiler, and we can not.  We therefore have no license-compatible
Fortran compiler for Python 3.5 - so we can't build scipy, and that's
a full stop for the scipy stack.   I'm sure you know, but the only
practical open-source option is mingw-w64, that does not work with the
Microsoft runtime used by Python 3.5 [1].  It appears the only person
capable of making mingw-w64 compatible with this runtime is Ray
Donnelly, who now works for Continuum, and we haven't yet succeeded in
finding the 15K USD or so to pay Continuum for his time.

> But not pyHDF, netCDF5, gdal, shapely, ... (to name a few that I need to
> work with). And these are ugly: which means very hard for end-users to
> build, and very hard for people to package up into wheels (is it even
> possible?)

I'd be surprised if these packages were very hard to build OSX and
Linux wheels for.  We're already building hard packages like Pillow
and Matplotlib and h5py and pytables.  If you mean netCDF4 - there are
already OSX and Windows wheels for that [2].

Cheers,

Matthew

[1] https://mingwpy.github.io/issues.html#the-vs-14-2015-runtime
[2] https://pypi.python.org/pypi/netCDF4


More information about the Distutils-SIG mailing list