[Distutils] Current Python packaging status (from my point of view)

Chris Barker chris.barker at noaa.gov
Tue Nov 1 13:01:42 EDT 2016

On Tue, Nov 1, 2016 at 2:50 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> > I wrote some lines, but I deleted my thoughts about the topic
> "Automating wheel creation", since
> > I am a afraid it could raise bad mood in this list again. That's not my
> goal.
> I currently see 3 main ways that could eventually happen:
> - the PSF sorts out its general sustainability concerns to the point
> where it believes it can credibly maintain such a service on the
> community's behalf

That would be great.

> - the conda-forge folks branch out into offering wheel building as
> well (so it becomes a matter of "publish your Python projects for the
> user level conda platform, get platform independent Python wheels as
> well")

I doubt that's going to happen, but conda-forge is an open source project
-- someone else could certainly make a friendly fork and make a wheel-forge.

- someone builds such a service independently of the current PyPI
> infrastructure team, and convinces package publishers to start using
> it

and starting with the nice tools conda-forge has built would be a good way
to get going.

There is an odd thing here, though -- conda-forge works because of two

1) The gitHub workflow is helpful for developing and vetting recipes

2) gitHub already has arrangements with CI services -- so it's easy to
leverage them to do the building.

But in theory, the PSF could negotiate an arrangement with the CI services
to make it easy to call them from PyPi.

and (1) is really necessary because of two things -- the folks making teh
recipes generally are not the package maintainers. and, critical here --
there are a lot of issues with consistent dependency management -- which
his what conda can help with, and wheels just don't.

ANother key point about all of this:

Nick missed the key point about conda. He mentioned that conda allows you
to manage the python run-time itself, which is in deed a nice feature, but
getting a python run-time as never been the hard part (maybe on Linux if
you want a different one than your system supplies).

And it's not about needing a compiler -- wheels solve that -- and getting
the system compiler is not all that hard either -- on all three major OSs.

It's about the non-python dependencies -- all the various C libs, etc that
your python packages might need. Or, for that matter, about non-python
command line utilities, and who knows what else. Sure, this is a bigger
issue in scientific computing --  which is why conda was born there -- but
it's an issue in many other realms as well. For many years, a Web-dev I
worked with avoided using PIL because it was too much a pain to install --
that's pretty easy these days (thanks Pillow!) but it's not long before a
serious web service might need SOMETHING that's not pure python...

So -- an auto wheel building system would be nice, but until it can handle
non-python dependencies, it's going to hit a wall....



Christopher Barker, Ph.D.

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20161101/22155fd2/attachment-0001.html>

More information about the Distutils-SIG mailing list