[Distutils] Maintaining a curated set of Python packages

Chris Barker chris.barker at noaa.gov
Wed Dec 14 12:41:44 EST 2016

As pointed out by others, there are external groups doing "curating".
conda-forge is one such project, so I'll comment from that perspective:

It's difficult because the definition of compatibility is highly dependent
> the consumer's environment.  For example, C extension compatibility will
> depend on the version of libraries available on the platform versions you
> care
> about.

Indeed -- which is why Anaconda and conda-forge are built on conda rather
than pip -- it is designed to handle these issues.

However with the many linux effort, and some efforts to kludge C libs into
binary wheels, pypi may just be able to handle more of these issues -- so
curating may have it's advantages.

As it happens, I recently proposed a version system for conda-forge (much
like Anaconda) where you would select a given conda-forge version, and ALL
the packages in that version would be expected to work together. the idea
is that you don't want one package dependent on libjpeg6 while another
depends on libjpeg7, for instance.

But the community has decided that rather than try to version the whole
system, we instead rely on robust "pinning" -- i.e. version specific
dependencies -- package A depends on libjpeg7, not just "libjpeg". Then
there are tools that go through all the packages and check for incompatible
pinnings, and update an build new versions where required (or at least ping
the maintainers of a package that un update is needed)

Ubuntu has an elaborate automated system for testing some dimension of
> compatibility issues between packages, not just Python packages.  Debian
> has
> the same system but isn't gated on the results.

This brings up the larger issue -- PyPi is inherently different than these
efforts -- PyPi has always been about each package author maintaining their
own package -- Linux distros and conda-forge, and ??? all have a small set
of core contributions that do the package maintenance. This is a large
effort, and wold be insanely hard with the massive amount of stuff on

In fact, I think the kinda-sort curation that comes from individual
communities is working remarkably well:

the scipy community
the django community



Christopher Barker, Ph.D.

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20161214/d9cd438f/attachment.html>

More information about the Distutils-SIG mailing list