PEP DRAFT - Inclusion of pip bootstrap in Python installation
Hi all, I present for your deliberation a draft PEP for the inclusion of a pip bootstrap program in Python 3.4. Discussion of this PEP should remain here on the distutils SIG. The PEP is revision controlled in my bitbucket account https://bitbucket.org/r1chardj0n3s/pypi-pep (this is also where I'm intending to develop the implementation.) Richard PEP: XXXX Title: Inclusion of pip bootstrap in Python installation Version: Last-Modified: Author: Richard Jones <richard@python.org> BDFL-Delegate: Nick Coghlan <ncoghlan@gmail.com> Discussions-To: <distutils-sig@python.org> Status: Draft Type: Standards Track Created: 18-Mar-2013 Python-Version: 3.4 Post-History: 19-Mar-2013 Abstract ======== This PEP proposes the inclusion of a pip boostrap executable in the Python installation to simplify the use of 3rd-party modules by Python users. This PEP does not propose to include the pip implementation in the Python standard library. Nor does it propose to implement any package management or installation mechanisms beyond those provided by PEPs 470 ("The Wheel Binary Package Format 1.0") and TODO distlib PEP. Rationale ========= Currently the user story for installing 3rd-party Python modules is not as simple as it could be. It requires that all 3rd-party modules inform the user of how to install the installer, typically via a link to the installer. That link may be out of date or the steps required to perform the install of the installer may be enough of a roadblock to prevent the user from further progress. Large Python projects which emphasise a low barrier to entry have shied away from depending on third party packages because of the introduction of this potential stumbling block for new users. With the inclusion of the package installer command in the standard Python installation the barrier to installing additional software is considerably reduced. It is hoped that this will therefore increase the likelihood that Python projects will reuse third party software. It is also hoped that this is reduces the number of proposals to include more and more software in the Python standard library, and therefore that more popular Python software is more easily upgradeable beyond requiring Python installation upgrades. Proposal ======== Python install includes an executable called "pip" that attempts to import pip machinery. If it can then the pip command proceeds as normal. If it cannot it will bootstrap pip by downloading the pip implementation wheel file. Once installed, the pip command proceeds as normal. A boostrap is used in the place of a the full pip code so that we don't have to bundle pip and also the install tool is upgradeable outside of the regular Python upgrade timeframe and processes. To avoid issues with sudo we will have the bootstrap default to installing the pip implementation to the per-user site-packages directory defined in PEP 370 and implemented in Python 2.6/3.0. Since we avoid installing to the system Python we also avoid conflicting with any other packaging system (on Linux systems, for example.) If the user is inside a virtual environment (TODO PEP ref) then the pip implementation will be installed into that virtual environment. The bootstrapping process will proceed as follows: 1. The user system has Python (3.4+) installed. In the "scripts" directory of the Python installation there is the bootstrap script called "pip". 2. The user will invoke a pip command, typically "pip install <package>", for example "pip install Django". 3. The boostrap script will attempt to import the pip implementation. If this succeeds, the pip command is processed normally. 4. On failing to import the pip implementation the bootstrap notifies the user that it is "upgrading pip" and contacts PyPI to obtain the latest download wheel file (see PEP 427.) 5. Upon downloading the file it is installed using the distlib installation machinery for wheel packages. Upon completing the installation the user is notified that "pip has been upgraded." TODO how is it verified? 6. The pip tool may now import the pip implementation and continues to process the requested user command normally. Users may be running in an environment which cannot access the public Internet and are relying solely on a local package repository. They would use the "-i" (Base URL of Python Package Index) argument to the "pip install" command. This use case will be handled by: 1. Recognising the command-line arguments that specify alternative or additional locations to discover packages and attempting to download the package from those locations. 2. If the package is not found there then we attempt to donwload it using the standard "https://pypi.python.org/pypi/simple/pip" index. 3. If that also fails, for any reason, we indicate to the user the operation we were attempting, the reason for failure (if we know it) and display further instructions for downloading and installing the file manually. Manual installation of the pip implementation will be supported through the manual download of the wheel file and "pip install <downloaded wheel file>". This installation will not perform standard pip installation steps of saving the file to a cache directory or updating any local database of installed files. The download of the pip implementation install file should be performed securely. The transport from pypi.python.org will be done over HTTPS but the CA certificate check will most likely not be performed. Therefore we will utilise the embedded signature support in the wheel format to validate the downloaded file. Beyond those arguments controlling index location and download options, the "pip" boostrap command may support further standard pip options for verbosity, quietness and logging. The "--no-install" option to the "pip" command will not affect the bootstrapping process. An additional new Python package will be proposed, "pypublish", which will be a tool for publishing packages to PyPI. It would replace the current "python setup.py register" and "python setup.py upload" distutils commands. Again because of the measured Python release cycle and extensive existing Python installations these commands are difficult to bugfix and extend. Additionally it is desired that the "register" and "upload" commands be able to be performed over HTTPS with certificate validation. Since shipping CA certificate keychains with Python is not really feasible (updating the keychain is quite difficult to manage) it is desirable that those commands, and the accompanying keychain, be made installable and upgradeable outside of Python itself. Implementation ============== TBD Risks ===== The Fedora variant of Linux has had a separate program called "pip" (a Perl package installer) available for install for some time. The current Python "pip" program is installed as "pip-python". It is hoped that the Fedora community will resolve this issue by renaming the Perl installer. Currently pip depends upon setuptools functionality. It is intended that before Python 3.4 is shipped that the required functionlity will be present in Python's standard library as the distlib module, and that pip would be modified to use that functionality when present. TODO PEP reference for distlib References ========== None, so far, beyond the PEPs. Acknowledgments =============== Nick Coghlan for his thoughts on the proposal and dealing with the Red Hat issue. Jannis Leidel and Carl Meyer for their thoughts. Copyright ========= This document has been placed in the public domain.
On Tue, Mar 19, 2013 at 2:04 PM, Richard Jones <richard@python.org> wrote:
The Fedora variant of Linux has had a separate program called "pip" (a Perl package installer) available for install for some time. The current Python "pip" program is installed as "pip-python". It is hoped that the Fedora community will resolve this issue by renaming the Perl installer.
A modest suggestion: renaming pip to "pypi" (Python Package Installer) will address this and other issues, especially if the 'pypi' command grows register/publish functions as well. Yes, it puts pip in a privileged position, but really it's just going to be acknowledging the status quo. As soon as pip can handle multi-version installs, binaries, and plugin scenarios as well as easy_install can, there will be no reason to keep easy_install around or bother upgrading it to do TUF or whatever else comes down the pike. And I'm not aware of any other competition (buildout isn't really aimed at the same space), so I don't think there's any reason not to just bless "pip" as *the* "pypi" tool. (I suppose there is a small possibility of confusion between the tool and the site, but then again, if you look at e.g. PHP's pear, the command line tools and repository have the same name. And Perl has a "cpan shell" for accessing CPAN, etc. I don't recall anybody in those communities being confused by those distinctions.)
On Tue, Mar 19, 2013 at 11:16 AM, PJ Eby <pje@telecommunity.com> wrote:
On Tue, Mar 19, 2013 at 2:04 PM, Richard Jones <richard@python.org> wrote:
The Fedora variant of Linux has had a separate program called "pip" (a Perl package installer) available for install for some time. The current Python "pip" program is installed as "pip-python". It is hoped that the Fedora community will resolve this issue by renaming the Perl installer.
A modest suggestion: renaming pip to "pypi" (Python Package Installer) will address this and other issues, especially if the 'pypi' command grows register/publish functions as well.
Unfortunately, this would just make the confusion with pypy worse, as well as put the community through yet another name change. Persisting with the "pip" name seems to be the best of the available options (the only wrinkle is that Perl tool sitting in the Fedora repos, but as far as we can tell that's just an old package that even Perl people don't use)
Yes, it puts pip in a privileged position, but really it's just going to be acknowledging the status quo. As soon as pip can handle multi-version installs, binaries, and plugin scenarios as well as easy_install can, there will be no reason to keep easy_install around or bother upgrading it to do TUF or whatever else comes down the pike. And I'm not aware of any other competition (buildout isn't really aimed at the same space), so I don't think there's any reason not to just bless "pip" as *the* "pypi" tool.
Yep, that's where all this is going (except we'll be keeping the pip name). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 19 March 2013 11:16, PJ Eby <pje@telecommunity.com> wrote:
On Tue, Mar 19, 2013 at 2:04 PM, Richard Jones <richard@python.org> wrote:
The Fedora variant of Linux has had a separate program called "pip" (a Perl package installer) available for install for some time. The current Python "pip" program is installed as "pip-python". It is hoped that the Fedora community will resolve this issue by renaming the Perl installer.
A modest suggestion: renaming pip to "pypi" (Python Package Installer) will address this and other issues, especially if the 'pypi' command grows register/publish functions as well.
We did discuss using a name other than pip - the main reason being that Fedora Linux has the Perl "pip" tool. We are hoping that will be resolved in our favour. There's just too much momentum in the community behind the "pip" name. If it does not work out then we may have to revisit the issue. If we do then it'd probably be "pyinstall" or something equally elegant <0.5 wink>
(I suppose there is a small possibility of confusion between the tool and the site, but then again, if you look at e.g. PHP's pear, the command line tools and repository have the same name. And Perl has a "cpan shell" for accessing CPAN, etc. I don't recall anybody in those communities being confused by those distinctions.)
Interestingly when I asked my Perl-using friends about "pip" the Perl tool, they indicated that these days they use "cpanm" by preference. Well, first they said "pip - isn't that the Python installer?" :-) Richard
On Tue, Mar 19, 2013 at 2:04 PM, Richard Jones <richard@python.org> wrote:
Hi all,
I present for your deliberation a draft PEP for the inclusion of a pip bootstrap program in Python 3.4. Discussion of this PEP should remain here on the distutils SIG.
The PEP is revision controlled in my bitbucket account https://bitbucket.org/r1chardj0n3s/pypi-pep (this is also where I'm intending to develop the implementation.)
Richard
PEP: XXXX Title: Inclusion of pip bootstrap in Python installation Version: Last-Modified: Author: Richard Jones <richard@python.org> BDFL-Delegate: Nick Coghlan <ncoghlan@gmail.com> Discussions-To: <distutils-sig@python.org> Status: Draft Type: Standards Track Created: 18-Mar-2013 Python-Version: 3.4 Post-History: 19-Mar-2013
Abstract ========
This PEP proposes the inclusion of a pip boostrap executable in the Python installation to simplify the use of 3rd-party modules by Python users.
This PEP does not propose to include the pip implementation in the Python standard library. Nor does it propose to implement any package management or installation mechanisms beyond those provided by PEPs 470 ("The Wheel Binary Package Format 1.0") and TODO distlib PEP.
Rationale =========
Currently the user story for installing 3rd-party Python modules is not as simple as it could be. It requires that all 3rd-party modules inform the user of how to install the installer, typically via a link to the installer. That link may be out of date or the steps required to perform the install of the installer may be enough of a roadblock to prevent the user from further progress.
Large Python projects which emphasise a low barrier to entry have shied away from depending on third party packages because of the introduction of this potential stumbling block for new users.
With the inclusion of the package installer command in the standard Python installation the barrier to installing additional software is considerably reduced. It is hoped that this will therefore increase the likelihood that Python projects will reuse third party software.
It is also hoped that this is reduces the number of proposals to include more and more software in the Python standard library, and therefore that more popular Python software is more easily upgradeable beyond requiring Python installation upgrades.
Proposal ========
Python install includes an executable called "pip" that attempts to import pip machinery. If it can then the pip command proceeds as normal. If it cannot it will bootstrap pip by downloading the pip implementation wheel file. Once installed, the pip command proceeds as normal.
A boostrap is used in the place of a the full pip code so that we don't have to bundle pip and also the install tool is upgradeable outside of the regular Python upgrade timeframe and processes.
To avoid issues with sudo we will have the bootstrap default to installing the pip implementation to the per-user site-packages directory defined in PEP 370 and implemented in Python 2.6/3.0. Since we avoid installing to the system Python we also avoid conflicting with any other packaging system (on Linux systems, for example.) If the user is inside a virtual environment (TODO PEP ref) then the pip implementation will be installed into that virtual environment.
The bootstrapping process will proceed as follows:
1. The user system has Python (3.4+) installed. In the "scripts" directory of the Python installation there is the bootstrap script called "pip". 2. The user will invoke a pip command, typically "pip install <package>", for example "pip install Django". 3. The boostrap script will attempt to import the pip implementation. If this succeeds, the pip command is processed normally. 4. On failing to import the pip implementation the bootstrap notifies the user that it is "upgrading pip" and contacts PyPI to obtain the latest download wheel file (see PEP 427.) 5. Upon downloading the file it is installed using the distlib installation machinery for wheel packages. Upon completing the installation the user is notified that "pip has been upgraded." TODO how is it verified? 6. The pip tool may now import the pip implementation and continues to process the requested user command normally.
Users may be running in an environment which cannot access the public Internet and are relying solely on a local package repository. They would use the "-i" (Base URL of Python Package Index) argument to the "pip install" command. This use case will be handled by:
1. Recognising the command-line arguments that specify alternative or additional locations to discover packages and attempting to download the package from those locations. 2. If the package is not found there then we attempt to donwload it using the standard "https://pypi.python.org/pypi/simple/pip" index. 3. If that also fails, for any reason, we indicate to the user the operation we were attempting, the reason for failure (if we know it) and display further instructions for downloading and installing the file manually.
Manual installation of the pip implementation will be supported through the manual download of the wheel file and "pip install <downloaded wheel file>".
This installation will not perform standard pip installation steps of saving the file to a cache directory or updating any local database of installed files.
The download of the pip implementation install file should be performed securely. The transport from pypi.python.org will be done over HTTPS but the CA certificate check will most likely not be performed. Therefore we will utilise the embedded signature support in the wheel format to validate the downloaded file.
Beyond those arguments controlling index location and download options, the "pip" boostrap command may support further standard pip options for verbosity, quietness and logging.
The "--no-install" option to the "pip" command will not affect the bootstrapping process.
An additional new Python package will be proposed, "pypublish", which will be a tool for publishing packages to PyPI. It would replace the current "python setup.py register" and "python setup.py upload" distutils commands. Again because of the measured Python release cycle and extensive existing Python installations these commands are difficult to bugfix and extend. Additionally it is desired that the "register" and "upload" commands be able to be performed over HTTPS with certificate validation. Since shipping CA certificate keychains with Python is not really feasible (updating the keychain is quite difficult to manage) it is desirable that those commands, and the accompanying keychain, be made installable and upgradeable outside of Python itself.
Implementation ==============
TBD
Risks =====
The Fedora variant of Linux has had a separate program called "pip" (a Perl package installer) available for install for some time. The current Python "pip" program is installed as "pip-python". It is hoped that the Fedora community will resolve this issue by renaming the Perl installer.
Currently pip depends upon setuptools functionality. It is intended that before Python 3.4 is shipped that the required functionlity will be present in Python's standard library as the distlib module, and that pip would be modified to use that functionality when present. TODO PEP reference for distlib
References ==========
None, so far, beyond the PEPs.
Acknowledgments ===============
Nick Coghlan for his thoughts on the proposal and dealing with the Red Hat issue.
Jannis Leidel and Carl Meyer for their thoughts.
Copyright =========
This document has been placed in the public domain. _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
++1 on including a bootstrap wheel-only installer for getting pip -1 on giving it the same name -1 on scripts that start with "py". "python -m something"? Wheel's builtin signature checking is both controversial and awesome. Its most important properties are that it is key centric (signatures ONLY mean the signer possessed the private signing key and do not by themselves assert any other hard-to-verify information like e-mail, timestamps etc.); keys are short and referenced literally; it has a simple ~500 line pure-Python implementation including the crypto math with optional C speedups. The reference uses a highly convenient elliptic curve scheme called Ed25519 developed by respected cryptographers Bernstein et al. I would be OK with trusting the cheeseshop to make all decisions needed for "get me the newest version of pip" to simplify the bootstrap installer and to trust that version based only on the integrity of the SSL connection. Would it be feasible/helpful to include a short root CA list "the 3 that pypi is allowed to buy certificates from" for this purpose? It's also possible to use the downloaded archive to initiate a more complete install. Wheel's own installer can do this but it may be too circular for some: "python wheel-1.0.0-py2.py3-none-any.whl/wheel install wheel-1.0.0-py2.py3-none-any.whl". With some form of distlib in the standard library this might not be needed. It would be a weekend project to include a vendorized version of pkg_resources with pip. This version of pip would be able to do everything pip does now, except install from sdist. The hardest part would be complaining in all the right places when setuptools wasn't installed. We will want something to replace setup.py register / upload / ... and pip definitely isn't the place for it. I'm not too worried about this since you will be able to download it once we have the bootstrap install feature in place.
On Mar 19, 2013, at 2:04 PM, Richard Jones <richard@python.org> wrote:
Hi all,
I present for your deliberation a draft PEP for the inclusion of a pip bootstrap program in Python 3.4. Discussion of this PEP should remain here on the distutils SIG.
The PEP is revision controlled in my bitbucket account https://bitbucket.org/r1chardj0n3s/pypi-pep (this is also where I'm intending to develop the implementation.)
Richard
PEP: XXXX Title: Inclusion of pip bootstrap in Python installation Version: Last-Modified: Author: Richard Jones <richard@python.org> BDFL-Delegate: Nick Coghlan <ncoghlan@gmail.com> Discussions-To: <distutils-sig@python.org> Status: Draft Type: Standards Track Created: 18-Mar-2013 Python-Version: 3.4 Post-History: 19-Mar-2013
Abstract ========
This PEP proposes the inclusion of a pip boostrap executable in the Python installation to simplify the use of 3rd-party modules by Python users.
This PEP does not propose to include the pip implementation in the Python standard library. Nor does it propose to implement any package management or installation mechanisms beyond those provided by PEPs 470 ("The Wheel Binary Package Format 1.0") and TODO distlib PEP.
Rationale =========
Currently the user story for installing 3rd-party Python modules is not as simple as it could be. It requires that all 3rd-party modules inform the user of how to install the installer, typically via a link to the installer. That link may be out of date or the steps required to perform the install of the installer may be enough of a roadblock to prevent the user from further progress.
Large Python projects which emphasise a low barrier to entry have shied away from depending on third party packages because of the introduction of this potential stumbling block for new users.
With the inclusion of the package installer command in the standard Python installation the barrier to installing additional software is considerably reduced. It is hoped that this will therefore increase the likelihood that Python projects will reuse third party software.
It is also hoped that this is reduces the number of proposals to include more and more software in the Python standard library, and therefore that more popular Python software is more easily upgradeable beyond requiring Python installation upgrades.
Proposal ========
Python install includes an executable called "pip" that attempts to import pip machinery. If it can then the pip command proceeds as normal. If it cannot it will bootstrap pip by downloading the pip implementation wheel file. Once installed, the pip command proceeds as normal.
A boostrap is used in the place of a the full pip code so that we don't have to bundle pip and also the install tool is upgradeable outside of the regular Python upgrade timeframe and processes.
To avoid issues with sudo we will have the bootstrap default to installing the pip implementation to the per-user site-packages directory defined in PEP 370 and implemented in Python 2.6/3.0. Since we avoid installing to the system Python we also avoid conflicting with any other packaging system (on Linux systems, for example.) If the user is inside a virtual environment (TODO PEP ref) then the pip implementation will be installed into that virtual environment.
The bootstrapping process will proceed as follows:
1. The user system has Python (3.4+) installed. In the "scripts" directory of the Python installation there is the bootstrap script called "pip". 2. The user will invoke a pip command, typically "pip install <package>", for example "pip install Django". 3. The boostrap script will attempt to import the pip implementation. If this succeeds, the pip command is processed normally. 4. On failing to import the pip implementation the bootstrap notifies the user that it is "upgrading pip" and contacts PyPI to obtain the latest download wheel file (see PEP 427.) 5. Upon downloading the file it is installed using the distlib installation machinery for wheel packages. Upon completing the installation the user is notified that "pip has been upgraded." TODO how is it verified? 6. The pip tool may now import the pip implementation and continues to process the requested user command normally.
Users may be running in an environment which cannot access the public Internet and are relying solely on a local package repository. They would use the "-i" (Base URL of Python Package Index) argument to the "pip install" command. This use case will be handled by:
1. Recognising the command-line arguments that specify alternative or additional locations to discover packages and attempting to download the package from those locations. 2. If the package is not found there then we attempt to donwload it using the standard "https://pypi.python.org/pypi/simple/pip" index. 3. If that also fails, for any reason, we indicate to the user the operation we were attempting, the reason for failure (if we know it) and display further instructions for downloading and installing the file manually.
Manual installation of the pip implementation will be supported through the manual download of the wheel file and "pip install <downloaded wheel file>".
This installation will not perform standard pip installation steps of saving the file to a cache directory or updating any local database of installed files.
The download of the pip implementation install file should be performed securely. The transport from pypi.python.org will be done over HTTPS but the CA certificate check will most likely not be performed. Therefore we will utilise the embedded signature support in the wheel format to validate the downloaded file.
Major concern is how will this deal with key revocation? If the key used to sign the pip wheels gets compromised what's the path for this tool to revoke the key? On the side of the HTTPS I've been looking into this recently as far as Python goes. If openssl is correctly configured (this is the case on Linux, and any Python compiled against the OSX OpenSSL) then `urllib.request.urlopen('https://pypi.python.org/', cadefault=True) will do the right thing. This gets sticker on cases where openssl _isn't_ configured with a default set of certificates (Windows i'm assuming, Homebrew on OSX for sure) this will cause a certificate error. It's possible a workable solution can be designed via SSL.
Beyond those arguments controlling index location and download options, the "pip" boostrap command may support further standard pip options for verbosity, quietness and logging.
The "--no-install" option to the "pip" command will not affect the bootstrapping process.
An additional new Python package will be proposed, "pypublish", which will be a tool for publishing packages to PyPI. It would replace the current "python setup.py register" and "python setup.py upload" distutils commands. Again because of the measured Python release cycle and extensive existing Python installations these commands are difficult to bugfix and extend. Additionally it is desired that the "register" and "upload" commands be able to be performed over HTTPS with certificate validation. Since shipping CA certificate keychains with Python is not really feasible (updating the keychain is quite difficult to manage) it is desirable that those commands, and the accompanying keychain, be made installable and upgradeable outside of Python itself.
Implementation ==============
TBD
Risks =====
The Fedora variant of Linux has had a separate program called "pip" (a Perl package installer) available for install for some time. The current Python "pip" program is installed as "pip-python". It is hoped that the Fedora community will resolve this issue by renaming the Perl installer.
Currently pip depends upon setuptools functionality. It is intended that before Python 3.4 is shipped that the required functionlity will be present in Python's standard library as the distlib module, and that pip would be modified to use that functionality when present. TODO PEP reference for distlib
References ==========
None, so far, beyond the PEPs.
Acknowledgments ===============
Nick Coghlan for his thoughts on the proposal and dealing with the Red Hat issue.
Jannis Leidel and Carl Meyer for their thoughts.
Copyright =========
This document has been placed in the public domain. _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Tue, Mar 19, 2013 at 3:11 PM, Donald Stufft <donald@stufft.io> wrote:
On Mar 19, 2013, at 2:04 PM, Richard Jones <richard@python.org> wrote:
Hi all,
I present for your deliberation a draft PEP for the inclusion of a pip bootstrap program in Python 3.4. Discussion of this PEP should remain here on the distutils SIG.
The PEP is revision controlled in my bitbucket account https://bitbucket.org/r1chardj0n3s/pypi-pep (this is also where I'm intending to develop the implementation.)
Richard
PEP: XXXX Title: Inclusion of pip bootstrap in Python installation Version: Last-Modified: Author: Richard Jones <richard@python.org> BDFL-Delegate: Nick Coghlan <ncoghlan@gmail.com> Discussions-To: <distutils-sig@python.org> Status: Draft Type: Standards Track Created: 18-Mar-2013 Python-Version: 3.4 Post-History: 19-Mar-2013
Abstract ========
This PEP proposes the inclusion of a pip boostrap executable in the Python installation to simplify the use of 3rd-party modules by Python users.
This PEP does not propose to include the pip implementation in the Python standard library. Nor does it propose to implement any package management or installation mechanisms beyond those provided by PEPs 470 ("The Wheel Binary Package Format 1.0") and TODO distlib PEP.
Rationale =========
Currently the user story for installing 3rd-party Python modules is not as simple as it could be. It requires that all 3rd-party modules inform the user of how to install the installer, typically via a link to the installer. That link may be out of date or the steps required to perform the install of the installer may be enough of a roadblock to prevent the user from further progress.
Large Python projects which emphasise a low barrier to entry have shied away from depending on third party packages because of the introduction of this potential stumbling block for new users.
With the inclusion of the package installer command in the standard Python installation the barrier to installing additional software is considerably reduced. It is hoped that this will therefore increase the likelihood that Python projects will reuse third party software.
It is also hoped that this is reduces the number of proposals to include more and more software in the Python standard library, and therefore that more popular Python software is more easily upgradeable beyond requiring Python installation upgrades.
Proposal ========
Python install includes an executable called "pip" that attempts to import pip machinery. If it can then the pip command proceeds as normal. If it cannot it will bootstrap pip by downloading the pip implementation wheel file. Once installed, the pip command proceeds as normal.
A boostrap is used in the place of a the full pip code so that we don't have to bundle pip and also the install tool is upgradeable outside of the regular Python upgrade timeframe and processes.
To avoid issues with sudo we will have the bootstrap default to installing the pip implementation to the per-user site-packages directory defined in PEP 370 and implemented in Python 2.6/3.0. Since we avoid installing to the system Python we also avoid conflicting with any other packaging system (on Linux systems, for example.) If the user is inside a virtual environment (TODO PEP ref) then the pip implementation will be installed into that virtual environment.
The bootstrapping process will proceed as follows:
1. The user system has Python (3.4+) installed. In the "scripts" directory of the Python installation there is the bootstrap script called "pip". 2. The user will invoke a pip command, typically "pip install <package>", for example "pip install Django". 3. The boostrap script will attempt to import the pip implementation. If this succeeds, the pip command is processed normally. 4. On failing to import the pip implementation the bootstrap notifies the user that it is "upgrading pip" and contacts PyPI to obtain the latest download wheel file (see PEP 427.) 5. Upon downloading the file it is installed using the distlib installation machinery for wheel packages. Upon completing the installation the user is notified that "pip has been upgraded." TODO how is it verified? 6. The pip tool may now import the pip implementation and continues to process the requested user command normally.
Users may be running in an environment which cannot access the public Internet and are relying solely on a local package repository. They would use the "-i" (Base URL of Python Package Index) argument to the "pip install" command. This use case will be handled by:
1. Recognising the command-line arguments that specify alternative or additional locations to discover packages and attempting to download the package from those locations. 2. If the package is not found there then we attempt to donwload it using the standard "https://pypi.python.org/pypi/simple/pip" index. 3. If that also fails, for any reason, we indicate to the user the operation we were attempting, the reason for failure (if we know it) and display further instructions for downloading and installing the file manually.
Manual installation of the pip implementation will be supported through the manual download of the wheel file and "pip install <downloaded wheel file>".
This installation will not perform standard pip installation steps of saving the file to a cache directory or updating any local database of installed files.
The download of the pip implementation install file should be performed securely. The transport from pypi.python.org will be done over HTTPS but the CA certificate check will most likely not be performed. Therefore we will utilise the embedded signature support in the wheel format to validate the downloaded file.
Major concern is how will this deal with key revocation? If the key used to sign the pip wheels gets compromised what's the path for this tool to revoke the key?
The wheel scheme skips revocation to simplify the implementation. You would be hard pressed to argue that it's not better than the current pypi security model ;-) A proper revocation model would look like TUF, a simple one would consist of grabbing the author keys over HTTPS at time of use. The scheme is flipped from the revocation model: require an up-to-date assertion that the key is current in order to trust the key, instead of trusting a key until a revocation happens.
On the side of the HTTPS I've been looking into this recently as far as Python goes. If openssl is correctly configured (this is the case on Linux, and any Python compiled against the OSX OpenSSL) then `urllib.request.urlopen('https://pypi.python.org/', cadefault=True) will do the right thing. This gets sticker on cases where openssl _isn't_ configured with a default set of certificates (Windows i'm assuming, Homebrew on OSX for sure) this will cause a certificate error. It's possible a workable solution can be designed via SSL.
On Tue, Mar 19, 2013 at 3:37 PM, Daniel Holth <dholth@gmail.com> wrote:
On Tue, Mar 19, 2013 at 3:11 PM, Donald Stufft <donald@stufft.io> wrote:
On Mar 19, 2013, at 2:04 PM, Richard Jones <richard@python.org> wrote:
Hi all,
I present for your deliberation a draft PEP for the inclusion of a pip bootstrap program in Python 3.4. Discussion of this PEP should remain here on the distutils SIG.
The PEP is revision controlled in my bitbucket account https://bitbucket.org/r1chardj0n3s/pypi-pep (this is also where I'm intending to develop the implementation.)
Richard
PEP: XXXX Title: Inclusion of pip bootstrap in Python installation Version: Last-Modified: Author: Richard Jones <richard@python.org> BDFL-Delegate: Nick Coghlan <ncoghlan@gmail.com> Discussions-To: <distutils-sig@python.org> Status: Draft Type: Standards Track Created: 18-Mar-2013 Python-Version: 3.4 Post-History: 19-Mar-2013
Abstract ========
This PEP proposes the inclusion of a pip boostrap executable in the Python installation to simplify the use of 3rd-party modules by Python users.
This PEP does not propose to include the pip implementation in the Python standard library. Nor does it propose to implement any package management or installation mechanisms beyond those provided by PEPs 470 ("The Wheel Binary Package Format 1.0") and TODO distlib PEP.
Rationale =========
Currently the user story for installing 3rd-party Python modules is not as simple as it could be. It requires that all 3rd-party modules inform the user of how to install the installer, typically via a link to the installer. That link may be out of date or the steps required to perform the install of the installer may be enough of a roadblock to prevent the user from further progress.
Large Python projects which emphasise a low barrier to entry have shied away from depending on third party packages because of the introduction of this potential stumbling block for new users.
With the inclusion of the package installer command in the standard Python installation the barrier to installing additional software is considerably reduced. It is hoped that this will therefore increase the likelihood that Python projects will reuse third party software.
It is also hoped that this is reduces the number of proposals to include more and more software in the Python standard library, and therefore that more popular Python software is more easily upgradeable beyond requiring Python installation upgrades.
Proposal ========
Python install includes an executable called "pip" that attempts to import pip machinery. If it can then the pip command proceeds as normal. If it cannot it will bootstrap pip by downloading the pip implementation wheel file. Once installed, the pip command proceeds as normal.
A boostrap is used in the place of a the full pip code so that we don't have to bundle pip and also the install tool is upgradeable outside of the regular Python upgrade timeframe and processes.
To avoid issues with sudo we will have the bootstrap default to installing the pip implementation to the per-user site-packages directory defined in PEP 370 and implemented in Python 2.6/3.0. Since we avoid installing to the system Python we also avoid conflicting with any other packaging system (on Linux systems, for example.) If the user is inside a virtual environment (TODO PEP ref) then the pip implementation will be installed into that virtual environment.
The bootstrapping process will proceed as follows:
1. The user system has Python (3.4+) installed. In the "scripts" directory of the Python installation there is the bootstrap script called "pip". 2. The user will invoke a pip command, typically "pip install <package>", for example "pip install Django". 3. The boostrap script will attempt to import the pip implementation. If this succeeds, the pip command is processed normally. 4. On failing to import the pip implementation the bootstrap notifies the user that it is "upgrading pip" and contacts PyPI to obtain the latest download wheel file (see PEP 427.) 5. Upon downloading the file it is installed using the distlib installation machinery for wheel packages. Upon completing the installation the user is notified that "pip has been upgraded." TODO how is it verified? 6. The pip tool may now import the pip implementation and continues to process the requested user command normally.
Users may be running in an environment which cannot access the public Internet and are relying solely on a local package repository. They would use the "-i" (Base URL of Python Package Index) argument to the "pip install" command. This use case will be handled by:
1. Recognising the command-line arguments that specify alternative or additional locations to discover packages and attempting to download the package from those locations. 2. If the package is not found there then we attempt to donwload it using the standard "https://pypi.python.org/pypi/simple/pip" index. 3. If that also fails, for any reason, we indicate to the user the operation we were attempting, the reason for failure (if we know it) and display further instructions for downloading and installing the file manually.
Manual installation of the pip implementation will be supported through the manual download of the wheel file and "pip install <downloaded wheel file>".
This installation will not perform standard pip installation steps of saving the file to a cache directory or updating any local database of installed files.
The download of the pip implementation install file should be performed securely. The transport from pypi.python.org will be done over HTTPS but the CA certificate check will most likely not be performed. Therefore we will utilise the embedded signature support in the wheel format to validate the downloaded file.
Major concern is how will this deal with key revocation? If the key used to sign the pip wheels gets compromised what's the path for this tool to revoke the key?
The wheel scheme skips revocation to simplify the implementation. You would be hard pressed to argue that it's not better than the current pypi security model ;-)
A proper revocation model would look like TUF, a simple one would consist of grabbing the author keys over HTTPS at time of use. The scheme is flipped from the revocation model: require an up-to-date assertion that the key is current in order to trust the key, instead of trusting a key until a revocation happens.
On the side of the HTTPS I've been looking into this recently as far as Python goes. If openssl is correctly configured (this is the case on Linux, and any Python compiled against the OSX OpenSSL) then `urllib.request.urlopen('https://pypi.python.org/', cadefault=True) will do the right thing. This gets sticker on cases where openssl _isn't_ configured with a default set of certificates (Windows i'm assuming, Homebrew on OSX for sure) this will cause a certificate error. It's possible a workable solution can be designed via SSL.
It would also be incredibly easy to require n signatures on the wheel instead of just 1. The signature format used, JWS-JS, already holds a list of signatures. This would make it much harder to compromise the system by pwning a single pip developer's machine and may help to put your mind at ease about revocation.
participants (5)
-
Daniel Holth -
Donald Stufft -
Nick Coghlan -
PJ Eby -
Richard Jones