[Distutils] Maintaining a curated set of Python packages

Nick Coghlan ncoghlan at gmail.com
Thu Dec 8 22:15:01 EST 2016


Putting the conclusion first, I do see value in better publicising
"Recommended libraries" based on some automated criteria like:

- recommended in the standard library documentation
- available via 1 or more cross-platform commercial Python redistributors
- available via 1 or more Linux distro vendors
- available via 1 or more web service development platforms

That would be a potentially valuable service for folks new to the
world of open source that are feeling somewhat overwhelmed by the
sheer number of alternatives now available to them.

However, I also think that would better fit in with the aims of an
open source component tracking community like libraries.io than it
does a publisher-centric community like distutils-sig.

The further comments below are just a bit more background on why I
feel the integration testing aspect of the suggestion isn't likely to
be particularly beneficial :)

On 9 December 2016 at 01:10, Barry Warsaw <barry at python.org> wrote:
> Still, there may be value in inter-Python package compatibility tests, but
> it'll take serious engineering effort (i.e. $ and time), ongoing maintenance,
> ongoing effort to fix problems, and tooling to gate installability of failing
> packages (with overrides for downstreams which don't care or already expend
> such effort).

I think this is really the main issue, as both desktop and server
environments are moving towards the integrated platform + isolated
applications approach popularised by mobile devices.

That means we end up with two very different variants of automated
integration testing:

- the application focused kind offered by the likes of requires.io and
pyup.io (i.e. monitor for dependency updates, submit PRs to trigger
app level CI)
- the platform focused kind employed by distro vendors (testing all
the platform components work together, including the app isolation
features)

The first kind makes sense if you're building something that runs *on*
platforms (Docker containers, Snappy or FlatPak apps, web services,
mobile apps, etc).

The second kind inevitably ends up intertwined with the component
review and release engineering systems of the particular platform, so
it becomes really hard to collaborate cross-platform outside the
context of specific projects like OpenStack that provide clear
definitions for "What components do we collectively depend on that we
need to test together?" and "What does 'working' mean in the context
of this project?".

Accordingly, for an initiative like this to be successful, it would
need to put some thought up front into the questions of:

1. Who are the intended beneficiaries of the proposal?
2. What problem does it address that will prompt them to contribute
time and/or money to solving it?
3. What do we expect people to be able to *stop doing* if the project
proves successful?

For platform providers, a generic "stdlib++" project wouldn't really
reduce the amount of integration testing we'd need to do ourselves (we
already don't test arbitrary combinations of dependencies, just the
ones we provide at any given point in time).

For application and service developers, the approach of pinning
dependencies to specific versions and treating updates like any other
source code change already works well in most cases.

That leaves library and framework developers, who currently tend to
adopt the policy of "for each version of Python that we support, we
test against the latest versions of our dependencies that were
available at the time we ran the test", leaving testing against older
versions to platform providers. If there's a key related framework
that also provides LTS versions (e.g. Django), then some folks may add
that to their test matrix as well.

In that context, "Only breaks backwards compatibility for compelling
reasons" becomes a useful long term survival trait for libraries and
frameworks, as gratuitous breakages are likely to lead to people
migrating away from particularly unreliable dependencies.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


More information about the Distutils-SIG mailing list