On 1 October 2017 at 10:59, Donald Stufft <donald@stufft.io> wrote:
On Sep 30, 2017, at 3:52 PM, xoviat <xoviat@gmail.com> wrote:
I don't think CPython needs to bundle all of its build-time dependencies. That principle doesn't really apply to other Python programs nor most other programs in general. AFAIK, CPython already has a build-time dependency on another, external, Python, so it wouldn't be too much to require the external Python to have setuptools installed with something like pyproject.toml (other programming languages usually bootstrap themselves with previous versions of the language along with some associated build tools).
As far as I can tell, CPython does *not* have a build time dependency on having Python available. I just spun up a bare alpine linux container and compiled CPython `master` branch on it. As far as I can tell the only Python that exists in this container is the one I just compiled.
That means that in order for CPython to depend on distutils to build as you indicate, it would also need to start depending on an existing version of Python being available. I don’t think that’s a great idea. I think Python should not depend on Python to build.
CPython's self-bootstrapping is a bit complicated :)
From a regular checkout, the way it works is:
1. We build CPython itself, including all frozen and builtin modules, since the source code for the frozen modules is checked in, as is the code generated by Argument Clinic 2. We use distutils & that just built CPython to build a bunch of optional extension modules For working on the compiler itself, we also have an extra step where we build CPython with no import system, use that to refreeze importlib with the modified compiler, then rebuild CPython with the recompiled import system. Breaking the code to run the code generators can thus break your build, although that should be less common now that we require execution of the code generators to be requested explicitly (and rely on CI to check when they haven't been run when they should). However, there's *nothing* in that arrangement which says we can't use a virtual environment to do the optional extension module builds, and plenty of good arguments to say that we *should* be using a virtual environment for those. And if we move them into a virtual environment, then *those builds* will still have access to distutils, regardless of whether we're picking it up from Lib/distutils (which is what we currently do), or from an installed setuptools (which is what I'd propose we switch to for 3.8+). As far as PEP 4 goes, I think the constraint that places on us is that *after installation of CPython*, then "import distutils" has to continue working, just as it does today. I *don't* think it places any requirement on us to ensure that "Lib/distutils" is a thing that exists in the CPython source repo - I'd consider it entirely acceptable for us to install it into site-packages with `pip` (PEP 453 style), and have it be missing by default from things like the embeddable Windows distribution. As far as the intended pay-off goes, this is entirely aimed at reduce the level of work for the current setuptools maintainers, as they have a complex bug triage process that requires them to do something like this: 1. Figure out if something is a distutils bug or a setuptools bug 2. If it's a distutils bug, figure out if setuptools can work around it 3. Regardless of whether setuptools can work around it or not, file a bug against distutils on b.p.o 4. Decide which branches to fix the distutils bug on 5. If setuptools can't work around it, tell people they'll have to upgrade CPython to fix their problem setuptools forking distutils wholesale simplifies that whole triage & process to: 1. It doesn't matter if the problem is a distutils bug or a setuptools bug, fix it in the setuptools code base 2. Ship a new release of setuptools to make the fix available to users However, if setuptools *does* fork distutils wholesale, then the already low level of contributions to distutils maintenance are likely to drop to essentially zero, at which point the question becomes "How can we switch *CPython itself* to using and providing the independently upgradable setuptools fork instead of the Lib/distutils version?" And there, PEP 453 has already established a relevant precedent: we can bundle setuptools as a wheel file, with a helper module to request installation of that wheel file's contents into the current environment. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia