[Distutils] Extracting distutils into setuptools

[Nick Coghlan]

On 1 October 2017 at 10:59, Donald Stufft <donald at stufft.io> wrote:
>
>
> On Sep 30, 2017, at 3:52 PM, xoviat <xoviat at gmail.com> wrote:
>
> I don't think CPython needs to bundle all of its build-time dependencies.
> That principle doesn't really apply to other Python programs nor most other
> programs in general. AFAIK, CPython already has a build-time dependency on
> another, external, Python, so it wouldn't be too much to require the
> external Python to have setuptools installed with something like
> pyproject.toml (other programming languages usually bootstrap themselves
> with previous versions of the language along with some associated build
> tools).
>
>
> As far as I can tell, CPython does *not* have a build time dependency on
> having Python available. I just spun up a bare alpine linux container and
> compiled CPython `master` branch on it. As far as I can tell the only Python
> that exists in this container is the one I just compiled.
>
> That means that in order for CPython to depend on distutils to build as you
> indicate, it would also need to start depending on an existing version of
> Python being available. I don’t think that’s a great idea. I think Python
> should not depend on Python to build.

CPython's self-bootstrapping is a bit complicated :)

>From a regular checkout, the way it works is:

1. We build CPython itself, including all frozen and builtin modules,
since the source code for the frozen modules is checked in, as is the
code generated by Argument Clinic
2. We use distutils & that just built CPython to build a bunch of
optional extension modules

For working on the compiler itself, we also have an extra step where
we build CPython with no import system, use that to refreeze importlib
with the modified compiler, then rebuild CPython with the recompiled
import system.

Breaking the code to run the code generators can thus break your
build, although that should be less common now that we require
execution of the code generators to be requested explicitly (and rely
on CI to check when they haven't been run when they should).

However, there's *nothing* in that arrangement which says we can't use
a virtual environment to do the optional extension module builds, and
plenty of good arguments to say that we *should* be using a virtual
environment for those.

And if we move them into a virtual environment, then *those builds*
will still have access to distutils, regardless of whether we're
picking it up from Lib/distutils (which is what we currently do), or
from an installed setuptools (which is what I'd propose we switch to
for 3.8+).

As far as PEP 4 goes, I think the constraint that places on us is that
*after installation of CPython*, then "import distutils" has to
continue working, just as it does today.

I *don't* think it places any requirement on us to ensure that
"Lib/distutils" is a thing that exists in the CPython source repo -
I'd consider it entirely acceptable for us to install it into
site-packages with `pip` (PEP 453 style), and have it be missing by
default from things like the embeddable Windows distribution.

As far as the intended pay-off goes, this is entirely aimed at reduce
the level of work for the current setuptools maintainers, as they have
a complex bug triage process that requires them to do something like
this:

1. Figure out if something is a distutils bug or a setuptools bug
2. If it's a distutils bug, figure out if setuptools can work around it
3. Regardless of whether setuptools can work around it or not, file a
bug against distutils on b.p.o
4. Decide which branches to fix the distutils bug on
5. If setuptools can't work around it, tell people they'll have to
upgrade CPython to fix their problem

setuptools forking distutils wholesale simplifies that whole triage &
process to:

1. It doesn't matter if the problem is a distutils bug or a setuptools
bug, fix it in the setuptools code base
2. Ship a new release of setuptools to make the fix available to users

However, if setuptools *does* fork distutils wholesale, then the
already low level of contributions to distutils maintenance are likely
to drop to essentially zero, at which point the question becomes "How
can we switch *CPython itself* to using and providing the
independently upgradable setuptools fork instead of the Lib/distutils
version?"

And there, PEP 453 has already established a relevant precedent: we
can bundle setuptools as a wheel file, with a helper module to request
installation of that wheel file's contents into the current
environment.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia