[Distutils] Maintaining a curated set of Python packages

Barry Warsaw barry at python.org
Thu Dec 8 10:10:24 EST 2016

On Dec 01, 2016, at 10:45 AM, Freddy Rietdijk wrote:

>Having a compatible set of packages is not only interesting for developers,
>but also for downstream distributions. All distributions try to find a set
>of packages that are working together and release them. This is a lot of
>work, and I think it would be in everyone's benefit if we try to solve this
>issue together.

It's an interesting but difficult problem at the level of PyPI.

It's difficult because the definition of compatibility is highly dependent on
the consumer's environment.  For example, C extension compatibility will
depend on the version of libraries available on the platform versions you care
about.  There are also dependents on Python libraries that you can't capture
only in PyPI, e.g. some distro-only package may depend on a particular Python
package API.  PyPI can't test any of this.  And some distros (e.g. Ubuntu) may
have multiple dimensions of consumability, e.g. the classic apt packages
vs. snaps, primary devel archive vs. stable distro versions vs. backports, etc.

Ubuntu has an elaborate automated system for testing some dimension of
compatibility issues between packages, not just Python packages.  Debian has
the same system but isn't gated on the results.  Individual distro packages
can include a set of tests that are run against the built version of the
package (as opposed to tests run at package build time).  When a new version
of that package is uploaded it ends up in a "proposed" pocket that is
generally not installed by users.  The proposed new version has its tests run,
and any package that depends on that package also has its tests run.  Only
when all those tests pass is the package automatically promoted to the release
pocket, and thus is installable by the general user population.  This does a
great job within the context of the distro of raising the quality of the
archive because obvious incompatibilities (i.e. those for which tests exist)
block such promotion.

(FWIW, the difference is that Debian doesn't block promotion of packages
failing their automated tests, so 1) it provides less value to Debian; 2) we
end up inheriting and seeing these problems in Ubuntu and so have to expend
downstream effort to fix the failures.)

All of this happens automatically within the context of the distro, on
multiple architectures, so it provides a lot of value, but I'm not sure how
useful it would be higher up the food chain, since those contexts will be
different enough to cause both false positives and false negatives.  And it
does often take quite a bit of focused engineering effort to monitor packages
which don't promote (something we want to automate), to actually fix the
problems wherever is most appropriate (and as far upstream as possible), and
to create meaningful tests of compatibility in the first place (except for
default tests such as installability).

Still, there may be value in inter-Python package compatibility tests, but
it'll take serious engineering effort (i.e. $ and time), ongoing maintenance,
ongoing effort to fix problems, and tooling to gate installability of failing
packages (with overrides for downstreams which don't care or already expend
such effort).

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 801 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20161208/bfeb1e21/attachment.sig>

More information about the Distutils-SIG mailing list