[Distutils] Comments on PEP 426

[Nick Coghlan]

On 4 September 2013 16:44, M.-A. Lemburg <mal at egenix.com> wrote:
> We need to get rid off hacks like setuptools if we ever
> want to see light at the end of the packaging tunnel.

Yes, the whole point of formally defining the requirements for the
setup.py CLI (along with a hook system for metadata 2.0 to make
providing setup.py itself completely optional) is to make it so that
projects don't need to use distutils *or* setuptools. They just need
to provide a setup.py that follows the spec or the appropriate static
metadata to tell automated build tools how to run Python code from the
source distribution to do the build. With the assumption that pip (or
an equivalent tool) will be available to retrieve dependencies,
developers can just declare the appropriate dependency metadata for
their chosen build system (using install_requires in current
setuptools defined metadata, or build_requires in metadata 2.0), and
the build system will sort it out based on the dependency
declarations.

However, the lack of a viable *transition plan* is one of the big
things that crippled the ultimate effectiveness of the distutils2
effort. The two most popular build systems for Python at this point
are vanilla distutils and setuptools. setuptools is the one that
people that actually need the dependency management and entry point
features are using. Our feelings about its implementation details are
completely irrelevant: those projects *have* to be able to migrate
easily, or the transition to metadata 2.0 isn't going to work.

Fortunately, Jason and PJE are both on-board with the current metadata
2.0 efforts, and metadata 2.0 has been redesigned so it can be shipped
in parallel with metadata 1.1 and the supporting setuptools-defined
metadata formats. This means it is likely that setuptools will gain
the ability to emit those formats once they're standardised.

So, the projects that explicitly use setuptools shouldn't be a problem.

That then leaves the projects that use plain distutils. How do we get
*them* into the shiny metadata 2.0 future, given that we have five
legacy versions of distutils in widespread use that we need to get to
support the new metadata 2.0 standards, when those standards *haven't
even been defined yet*, and definitely won't be defined in time for
Python 3.4 feature freeze in November (adding a *sixth* legacy version
to support).

Note that the distutils freeze has *already been lifted*. The problem
that originally led to the freeze was a change to an internal API
breaking setuptools in a CPython *maintenance* release. PJE has since
indicated that he would have accepted that breakage as requiring a
setuptools upgrade in a *feature* release - it was the fact it
happened in a maintenance release that he had a problem with. So, no
distutils freeze, but still a problem with supporting new metadata
standards in previously released versions of Python. Once that was
clarified, the freeze was lifted, but there was also the realisation
around that time that any proposed changes to the packaging ecosystem
are irrelevant if they aren't supported at least as far back as Python
2.6 (so they work with the system Python on
RHEL/CentOS/ScientificLinux and other long term support Linux versions
of similar vintage).

Given where we are now, the *ideal* solution would be for Python 3.4
distutils to be brought up to feature parity with current setuptools
and a "backports.distutils" module published on PyPI.

When the metadata 2.0 standards are defined, support would be added to
the Python 3.5 distutils, and published immediately on PyPI as
backports.distutils.

One potential problem with that approach is that distutils in trunk is
currently Python 3 only, while a backport would need to be 2/3
compatible. That's likely more feasible now than it was when
packaging/distutils2 were being worked on, but the initial set of
changes to make it 2/3 compatible will probably still involve some
pain.

However, a more significant problem is that the whole idea is based on
pure vapourware. That ideal "it's just like setuptools, but with a
more elegant implementation that could be used to replace distutils in
Python 3.4" library *doesn't exist*, and I have no desire to wait
around in the (likely vain) hope of somebody stepping up to create it.

Instead, I think the far more pragmatic option at this point is to
just tell people "your setup.py must run correctly with setuptools
imported in that process. If it doesn't, it is your setup.py is
broken, not the build tool that imported setuptools, or setuptools
itself for monkey-patching distutils". Most projects don't need to do
anything special to handle that situation, and for the few that do
"'setuptools' in sys.modules" is a reliable way to check for its
presence. This approach is a useful hack until somebody actually
*produces* a "setuptools-lite" which does everything important that
setuptools does without monkey-patching distutils, or until we have
successfully made the transition to a metadata 2.0 ecosystem where
projects feel comfortable declaring a dependency on a custom build
system and assuming that end users will be able to handle it (whether
directly or by downloading a wheel file).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia