[Distutils] The future of invoking pip

[Paul Moore]

On 6 November 2015 at 14:56, Tim Golden <mail at timgolden.me.uk> wrote:
>> One option would be to add a "py -which" mode that just does the
>> configuration lookup and then prints the path to the real python
>> executable. This would add the cost of one subprocess startup per pip,
>> but not 6?
>
> FWIW this will do that:
>
>   py -c "import sys; print(sys.executable)"

Indeed. We don't want to require modifications to py.exe, as that
would imply some sort of "Python 3.6 only" requirement (there are
backport options, as the launcher is largely independent from Python
itself, but that's yet more obstacles for end users).

Don't take the 6 processes too literally, but just to explain:

    pip launcher
        - runs Python. Either directly (= 2 processes so far) or via
py (= 3 processes so far)
        - checks what Python it's running (just code to read
sys.executable in pip)
        - processes the -p flag
            - runs py -c "import sys; print(sys.executable)" to work
out what version the -p flag is requesting
              (unless -p requires the full pathname of an executable,
which is lousy UI)
              That's 2 more processes run, albeit only for a short time
            - Finds out it needs a different exe, launches py for the
correct Python (add 2 more processes to the stack, we're at 5 now).
            - The subordinate pip runs sys.executable to execute
setup.py (1 more process) which in turn launches the C compiler
              (that's a stack of 7 processes)

It's quite likely that this can be trimmed, and normal cases won't be
nearly as bad. And while creating new processes on Windows is
distinctly worse than on Linux, it's not *that* bad.

But when wheels were introduced, installing via wheels rather than via
setup.py caused *substantial* time savings when installing pure Python
wheels, simply from removing process creation overheads (creating a
virtualenv, which installed setuptools and pip at the time, i.e., 2
wheels instead of 2 sdists, went from something like 30 seconds down
to a couple of seconds, if I recall).

I'm not wanting to make a big issue out of this, but it needs to be
considered...
Paul