[Distutils] PEP DRAFT - Inclusion of pip bootstrap in Python installation

[Daniel Holth]

On Tue, Mar 19, 2013 at 3:11 PM, Donald Stufft <donald at stufft.io> wrote:
>
> On Mar 19, 2013, at 2:04 PM, Richard Jones <richard at python.org> wrote:
>
>> Hi all,
>>
>> I present for your deliberation a draft PEP for the inclusion of a pip
>> bootstrap program in Python 3.4. Discussion of this PEP should remain
>> here on the distutils SIG.
>>
>> The PEP is revision controlled in my bitbucket account
>> https://bitbucket.org/r1chardj0n3s/pypi-pep (this is also where I'm
>> intending to develop the implementation.)
>>
>>
>>    Richard
>>
>> PEP: XXXX
>> Title: Inclusion of pip bootstrap in Python installation
>> Version:
>> Last-Modified:
>> Author: Richard Jones <richard at python.org>
>> BDFL-Delegate:  Nick Coghlan <ncoghlan at gmail.com>
>> Discussions-To: <distutils-sig at python.org>
>> Status: Draft
>> Type: Standards Track
>> Created: 18-Mar-2013
>> Python-Version: 3.4
>> Post-History: 19-Mar-2013
>>
>> Abstract
>> ========
>>
>> This PEP proposes the inclusion of a pip boostrap executable in the Python
>> installation to simplify the use of 3rd-party modules by Python users.
>>
>> This PEP does not propose to include the pip implementation in the Python
>> standard library. Nor does it propose to implement any package management or
>> installation mechanisms beyond those provided by PEPs 470 ("The Wheel Binary
>> Package Format 1.0") and TODO distlib PEP.
>>
>>
>> Rationale
>> =========
>>
>> Currently the user story for installing 3rd-party Python modules is
>> not as simple as it could be. It requires that all 3rd-party modules inform
>> the user of how to install the installer, typically via a link to the
>> installer. That link may be out of date or the steps required to perform the
>> install of the installer may be enough of a roadblock to prevent the user from
>> further progress.
>>
>> Large Python projects which emphasise a low barrier to entry have shied away
>> from depending on third party packages because of the introduction of this
>> potential stumbling block for new users.
>>
>> With the inclusion of the package installer command in the standard Python
>> installation the barrier to installing additional software is considerably
>> reduced. It is hoped that this will therefore increase the likelihood that
>> Python projects will reuse third party software.
>>
>> It is also hoped that this is reduces the number of proposals to include
>> more and more software in the Python standard library, and therefore that
>> more popular Python software is more easily upgradeable beyond requiring
>> Python installation upgrades.
>>
>>
>> Proposal
>> ========
>>
>> Python install includes an executable called "pip" that attempts to import pip
>> machinery. If it can then the pip command proceeds as normal. If it cannot it
>> will bootstrap pip by downloading the pip implementation wheel file.
>> Once installed, the pip command proceeds as normal.
>>
>> A boostrap is used in the place of a the full pip code so that we don't have
>> to bundle pip and also the install tool is upgradeable outside of the regular
>> Python upgrade timeframe and processes.
>>
>> To avoid issues with sudo we will have the bootstrap default to installing the
>> pip implementation to the per-user site-packages directory defined in PEP 370
>> and implemented in Python 2.6/3.0. Since we avoid installing to the system
>> Python we also avoid conflicting with any other packaging system (on Linux
>> systems, for example.) If the user is inside a virtual environment (TODO PEP
>> ref) then the pip implementation will be installed into that virtual
>> environment.
>>
>> The bootstrapping process will proceed as follows:
>>
>> 1. The user system has Python (3.4+) installed. In the "scripts" directory of
>>   the Python installation there is the bootstrap script called "pip".
>> 2. The user will invoke a pip command, typically "pip install <package>", for
>>   example "pip install Django".
>> 3. The boostrap script will attempt to import the pip implementation. If this
>>   succeeds, the pip command is processed normally.
>> 4. On failing to import the pip implementation the bootstrap notifies the user
>>   that it is "upgrading pip" and contacts PyPI to obtain the latest download
>>   wheel file (see PEP 427.)
>> 5. Upon downloading the file it is installed using the distlib installation
>>   machinery for wheel packages. Upon completing the installation the user
>>   is notified that "pip has been upgraded." TODO how is it verified?
>> 6. The pip tool may now import the pip implementation and continues to process
>>   the requested user command normally.
>>
>> Users may be running in an environment which cannot access the public Internet
>> and are relying solely on a local package repository. They would use the "-i"
>> (Base URL of Python Package Index) argument to the "pip install" command. This
>> use case will be handled by:
>>
>> 1. Recognising the command-line arguments that specify alternative or additional
>>   locations to discover packages and attempting to download the package
>>   from those locations.
>> 2. If the package is not found there then we attempt to donwload it using
>>   the standard "https://pypi.python.org/pypi/simple/pip" index.
>> 3. If that also fails, for any reason, we indicate to the user the operation
>>   we were attempting, the reason for failure (if we know it) and display
>>   further instructions for downloading and installing the file manually.
>>
>> Manual installation of the pip implementation will be supported through the
>> manual download of the wheel file and "pip install <downloaded wheel file>".
>>
>> This installation will not perform standard pip installation steps of saving the
>> file to a cache directory or updating any local database of installed files.
>>
>> The download of the pip implementation install file should be performed
>> securely. The transport from pypi.python.org will be done over HTTPS but the CA
>> certificate check will most likely not be performed. Therefore we will
>> utilise the embedded signature support in the wheel format to validate the
>> downloaded file.
>
> Major concern is how will this deal with key revocation? If the key used to sign the pip wheels gets compromised what's the path for this tool to revoke the key?

The wheel scheme skips revocation to simplify the implementation. You
would be hard pressed to argue that it's not better than the current
pypi security model ;-)

A proper revocation model would look like TUF, a simple one would
consist of grabbing the author keys over HTTPS at time of use. The
scheme is flipped from the revocation model: require an up-to-date
assertion that the key is current in order to trust the key, instead
of trusting a key until a revocation happens.

> On the side of the HTTPS I've been looking into this recently as far as Python goes. If openssl is correctly configured (this is the case on Linux, and any Python compiled against the OSX OpenSSL) then `urllib.request.urlopen('https://pypi.python.org/', cadefault=True) will do the right thing. This gets sticker on cases where openssl _isn't_ configured with a default set of certificates (Windows i'm assuming, Homebrew on OSX for sure) this will cause a certificate error. It's possible a workable solution can be designed via SSL.