PEP453 - Explicit bootstrapping of pip in Python installations
Abstract ======== This PEP proposes the inclusion of a method for explicitly bootstrapping `pip`_ as the default package manager for Python. It also proposes that the distributions of Python available via Python.org will automatically run this explicit bootstrapping method and a recommendation to third party redistributors of Python to also provide pip by default (in a way reasonable for their distributions). This PEP does *not* propose the inclusion of pip itself in the standard library. Proposal ======== This PEP proposes the inclusion of a ``getpip`` bootstrapping module in Python 3.4, as well as in the upcoming maintenance releases of Python 2.7 and Python 3.3. Rationale ========= Installing a third party package into a freshly installed Python requires first installing the package manager. This requires users ahead of time to know what the package manager is, where to get them from, and how to install them. The effect of this is that these external projects are required to either blindly assume the user already has the package manager installed, needs to duplicate the instructions and tell their users how to install the package manager, or completely forgo the use of dependencies to ease installation concerns for their users. All of the available options have their own drawbacks. If a project simply assumes a user already has the tooling then they get a confusing error message when the installation command doesn't work. Some operating may ease this pain by providing a global hook that looks for commands that don't exist and suggest an OS package they can install to make the command work. If a project chooses to duplicate the installation instructions and tell their users how to install the package manager before telling them how to install their own project then whenever these instructions need updates they need updating by every project that has duplicated them. This will inevitably not happen in every case leaving many different instructions on how to install it many of them broken or less than optimal. These additional instructions might also confuse users who try to install the package manager a second time thinking that it's part of the instructions of installing the project. The problem of stale instructions can be alleviated by referencing `pip's own bootstrapping instructions http://www.pip-installer.org/en/latest/installing.html`__, but the user experience involved still isn't good (especially on Windows, where downloading and running a Python script with the default OS configuration is significantly more painful than downloading and running a binary executable or installer). The projects that have decided to forgo dependencies all together are forced to either duplicate the efforts of other projects by inventing their own solutions to problems or are required to simply include the other projects in their own source trees. Both of these options present their own problems either in duplicating maintenance work across the ecosystem or potentially leaving users vulnerable to security issues because the included code or duplicated efforts are not automatically updated when upstream releases a new version. By providing the package manager by default it will be easier for users trying to install these third party packages as well as easier for the people distributing them as they no longer need to pick the lesser evil. This will become more important in the future as the Wheel_ package format does not have a built in "installer" in the form of ``setup.py`` so users wishing to install a Wheel package will need an installer even in the simple case. Reducing the burden of actually installing a third party package should also decrease the pressure to add every useful module to the standard library. This will allow additions to the standard library to focus more on why Python should have a particular tool out of the box instead of needing to use the difficulty in installing a package as justification for inclusion. Explicit Bootstrapping ====================== An additional module called ``getpip`` will be added to the standard library whose purpose is to install pip and any of its dependencies into the appropriate location (most commonly site-packages). It will expose a single callable named ``bootstrap()`` as well as offer direct execution via ``python -m getpip``. Options for installing it such as index server, installation location (``--user``, ``--root``, etc) will also be available to enable different installation schemes. It is believed that users will want the most recent versions available to be installed so that they can take advantage of the new advances in packaging. Since any particular version of Python has a much longer staying power than a version of pip in order to satisfy a user's desire to have the most recent version the bootstrap will contact PyPI, find the latest version, download it, and then install it. This process is security sensitive, difficult to get right, and evolves along with the rest of packaging. Instead of attempting to maintain a "mini pip" for the sole purpose of installing pip the ``getpip`` module will, as an implementation detail, include a private copy of pip which will be used to discover and install pip from PyPI. It is important to stress that this private copy of pip is *only* an implementation detail and it should *not* be relied on or assumed to exist. Not all users will have network access to PyPI whenever they run the bootstrap. In order to ensure that these users will still be able to bootstrap pip the bootstrap will fallback to simply installing the included copy of pip. This presents a balance between giving users the latest version of pip, saving them from needing to immediately upgrade pip after bootstrapping it, and allowing the bootstrap to work offline in situations where users might already have packages downloaded that they wish to install. Updating the Bundled pip ------------------------ In order to keep up with evolutions in packaging as well as providing users who are using the offline installation method with as recent version as possible the ``getpip`` module should be updates to the latest versions of everything it bootstraps. During the preparation for any release of Python, a script, provided as part of this PEP, should be run to update the bundled packages to the latest versions. This means that maintenance releases of the CPython installers will include an updated version of the ``getpip`` bootstrap module. Pre-installation ================ During the installation of Python from Python.org ``python -m getpip`` should be executed. Leaving people using the Windows or OSX installers with a working copy of pip once the installation has completed. The exact method of this is left up to the maintainers of the installers however if the bootstrapping is optional it should be opt out rather than opt in. The Windows and OSX installers distributed by Python.org will automatically attempt to run ``python -m getpip`` by default however the ``make install`` and ``make altinstall`` commands of the source distribution will not. Keeping the pip bootstrapping as a separate step for make based installations should minimize the changes CPython redistributors need to make to their build processes. Avoiding the layer of indirection through make for the getpip invocation also ensures those installing from a custom source build can easily force an offline installation of pip, install it from a private index server, or skip installing pip entirely. Python Virtual Environments =========================== Python 3.3 included a standard library approach to virtual Python environments through the ``venv`` module. Since it's release it has become clear that very few users have been willing to use this feature in part due to the lack of an installer present by default inside of the virtual environment. They have instead opted to continue using the ``virtualenv`` package which *does* include pip installed by default. To make the ``venv`` more useful to users it will be modified to issue the pip bootstrap by default inside of the new environment while creating it. This will allow people the same convenience inside of the virtual environment as this PEP provides outside of it as well as bringing the ``venv`` module closer to feature parity with the external ``virtualenv`` package making it a more suitable replacement. Recommendations for Downstream Distributors =========================================== A common source of Python installations are through downstream distributors such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX package managers [#homebrew]_, or python specific tools [#conda]_. In order to provide a consistent, user friendly experience to all users of Python regardless of how they attained Python this PEP recommends and asks that downstream distributors: * Ensure that whenever Python is installed pip is also installed. * This may take the form of separate with dependencies on each either so that installing the python package installs the pip package and installing the pip package installs the Python package. * Do not remove the bundled copy of pip. * This is required for offline installation of pip into a virtual environment. * This is similar to the existing ``virtualenv`` package for which many downstream distributors have already made exception to the common "debundling" policy. * This does mean that if ``pip`` needs to be updated due to a security issue, so does the bundled version in the ``getpip`` bootstrap module * Migrate build systems to utilize `pip`_ and `Wheel`_ instead of directly using ``setup.py``. * This will ensure that downstream packages can utilize the new formats which will not have a ``setup.py`` easier. * Ensure that all features of this PEP continue to work with any modifications made. * Online installation of the latest version of pip into a global or virtual python environment using ``python -m getpip``. * Offline installation of the bundled version of pip into a global or virtual python environment using ``python -m getpip``. * ``pip install --upgrade pip`` in a global installation should not affect any already created virtual environments. * ``pip install --upgrade pip`` in a virtual environment should not affect the global installation. Policies & Governance ===================== The maintainers of the bundled software and the CPython core team will work together in order to address the needs of both. The bundled software will still remain external to CPython and this PEP does not include CPython subsuming the responsibilities or decisions of the bundled software. This PEP aims to decrease the burden on end users wanting to use third party packages and the decisions inside it are pragmatic ones that represent the trust that the Python community has placed in the authors and maintainers of the bundled software. Backwards Compatibility ----------------------- The public API of the ``getpip`` module itself will fall under the typical backwards compatibility policy of Python for its standard library. The externally developed software that this PEP bundles does not. Security Releases ----------------- Any security update that affects the ``getpip`` module will be shared prior to release with the PSRT. The PSRT will then decide if the issue inside warrants a security release of Python. Appendix: Rejected Proposals ============================ Implicit Bootstrap ------------------ `PEP439`_, the predecessor for this PEP, proposes it's own solution. Its solution involves shipping a fake ``pip`` command that when executed would implicitly bootstrap and install pip if it does not already exist. This has been rejected because it is too "magical". It hides from the end user when exactly the pip command will be installed or that it is being installed at all. It also does not provide any recommendations or considerations towards downstream packagers who wish to manage the globally installed pip through the mechanisms typical for their system. Including pip In the Standard Library ------------------------------------- Similar to this PEP is the proposal of just including pip in the standard library. This would ensure that Python always includes pip and fixes all of the end user facing problems with not having pip present by default. This has been rejected because we've learned through the inclusion and history of ``distutils`` in the standard library that losing the ability to update the packaging tools independently can leave the tooling in a state of constant limbo. Making it unable to ever reasonably evolve in a timeframe that actually affects users as any new features will not be available to the general population for *years*. Allowing the packaging tools to progress separately from the Python release and adoption schedules allows the improvements to be used by *all* members of the Python community and not just those able to live on the bleeding edge of Python releases. .. _Wheel: http://www.python.org/dev/peps/pep-0427/ .. _pip: http://www.pip-installer.org .. _setuptools: https://pypi.python.org/pypi/setuptools .. _PEP439: http://www.python.org/dev/peps/pep-0439/ References ========== .. [#ubuntu] `Ubuntu http://www.ubuntu.com/` .. [#debian] `Debian http://www.debian.org` .. [#fedora] `Fedora https://fedoraproject.org/` .. [#homebrew] `Homebrew http://brew.sh/` .. [#conda] `Conda http://www.continuum.io/blog/conda` ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Aug 30, 2013, at 9:44 AM, Donald Stufft
[snip]
Available online at http://www.python.org/dev/peps/pep-0453/ ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Fri, Aug 30, 2013 at 9:44 AM, Donald Stufft
[SNIP] Pre-installation ================
During the installation of Python from Python.org ``python -m getpip`` should be executed. Leaving people using the Windows or OSX installers with a working copy of pip once the installation has completed.
"should be executed, leaving".
The exact method of this is left up to the maintainers of the installers however if the bootstrapping is optional it should be opt out rather than opt in.
"installers, however" "opt-in", "opt-out"
The Windows and OSX installers distributed by Python.org will automatically attempt to run ``python -m getpip`` by default however the ``make install`` and ``make altinstall`` commands of the source distribution will not.
Is the plan to leave getpip entirely out of the source distribution or to have it checked into hg.python.org/cpython?
Keeping the pip bootstrapping as a separate step for make based installations should minimize the changes CPython redistributors need to make to their build processes. Avoiding the layer of indirection through make for the getpip invocation also ensures those installing from a custom source build can easily force an offline installation of pip, install it from a private index server, or skip installing pip entirely.
Python Virtual Environments ===========================
Python 3.3 included a standard library approach to virtual Python environments through the ``venv`` module. Since it's release it has become clear that very few users have been willing to use this feature in part due to the lack of an installer present by default inside of the virtual environment. They have instead opted to continue using the ``virtualenv`` package which *does* include pip installed by default.
To make the ``venv`` more useful to users it will be modified to issue the pip bootstrap by default inside of the new environment while creating it. This will allow people the same convenience inside of the virtual environment as this PEP provides outside of it as well as bringing the ``venv`` module closer to feature parity with the external ``virtualenv`` package making it a more suitable replacement.
What about a --without-pip option?
Recommendations for Downstream Distributors ===========================================
A common source of Python installations are through downstream distributors such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX package managers [#homebrew]_, or python specific tools [#conda]_. In order to provide a consistent, user friendly experience to all users of Python regardless of how they attained Python this PEP recommends and asks that downstream distributors:
* Ensure that whenever Python is installed pip is also installed.
* This may take the form of separate with dependencies on each either so that installing the python package installs the pip package and installing the pip package installs the Python package.
* Do not remove the bundled copy of pip.
* This is required for offline installation of pip into a virtual environment. * This is similar to the existing ``virtualenv`` package for which many downstream distributors have already made exception to the common "debundling" policy. * This does mean that if ``pip`` needs to be updated due to a security issue, so does the bundled version in the ``getpip`` bootstrap module
* Migrate build systems to utilize `pip`_ and `Wheel`_ instead of directly using ``setup.py``.
* This will ensure that downstream packages can utilize the new formats which will not have a ``setup.py`` easier.
* Ensure that all features of this PEP continue to work with any modifications made.
* Online installation of the latest version of pip into a global or virtual python environment using ``python -m getpip``. * Offline installation of the bundled version of pip into a global or virtual python environment using ``python -m getpip``. * ``pip install --upgrade pip`` in a global installation should not affect any already created virtual environments. * ``pip install --upgrade pip`` in a virtual environment should not affect the global installation.
Policies & Governance =====================
The maintainers of the bundled software and the CPython core team will work together in order to address the needs of both. The bundled software will still remain external to CPython and this PEP does not include CPython subsuming the responsibilities or decisions of the bundled software. This PEP aims to decrease the burden on end users wanting to use third party packages and the decisions inside it are pragmatic ones that represent the trust that the Python community has placed in the authors and maintainers of the bundled software.
This should specify if it is ever expected to be kept in the CPython repo (and thus distributed in the source tarball) or if it will simply be bundled in the installers (and thus never included with the source). -Brett
On Aug 30, 2013, at 11:39 AM, Brett Cannon
What about a --without-pip option?
Added this and the grammar mistakes you noted (I suck at this english thing~).
This should specify if it is ever expected to be kept in the CPython repo (and thus distributed in the source tarball) or if it will simply be bundled in the installers (and thus never included with the source).
Nick may say differently (he's much more "in tune" with what files are owned by which processes) but I'd expect that the ``getpip`` module will be included as a normal part of the standard library and with it a tarball (or a Wheel) containing the sources of the bundled software. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
This should specify if it is ever expected to be kept in the CPython repo (and thus distributed in the source tarball) or if it will simply be bundled in the installers (and thus never included with the source).
Nick may say differently (he's much more "in tune" with what files are owned by which processes) but I'd expect that the ``getpip`` module will be included as a normal part of the standard library and with it a tarball (or a Wheel) containing the sources of the bundled software.
will getpip also replace the current role of "get-pip.py" (available in pip's master branch), i.e. will getpip be available for download and be mentioned in the pip install instructions for python's that don't have it?
On Aug 30, 2013, at 12:32 PM, Marcus Smith
This should specify if it is ever expected to be kept in the CPython repo (and thus distributed in the source tarball) or if it will simply be bundled in the installers (and thus never included with the source).
Nick may say differently (he's much more "in tune" with what files are owned by which processes) but I'd expect that the ``getpip`` module will be included as a normal part of the standard library and with it a tarball (or a Wheel) containing the sources of the bundled software.
will getpip also replace the current role of "get-pip.py" (available in pip's master branch), i.e. will getpip be available for download and be mentioned in the pip install instructions for python's that don't have it?
Pip will probably need to maintain get-pip.py for any Python version it supports that Doesn't have the getpip module. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 30 August 2013 14:44, Donald Stufft
The Windows and OSX installers distributed by Python.org will automatically attempt to run ``python -m getpip`` by default however the ``make install`` and ``make altinstall`` commands of the source distribution will not.
Presumably the uninstaller components of these installers should similarly uninstall pip before uninstalling Python. Would something like a "python -m getpip uninstall" command be worthwhile to support this? (That's probably a question for the authors of the installers to answer, as I don't know if such a command would be needed - I suspect that the Windows MSI installer just records what files it installs and uninstalls them, so it may not be needed there). But otherwise +1 on this PEP. Paul
On 31 Aug 2013 02:02, "Paul Moore"
On 30 August 2013 14:44, Donald Stufft
wrote: The Windows and OSX installers distributed by Python.org will
attempt to run ``python -m getpip`` by default however the ``make install`` and ``make altinstall`` commands of the source distribution will not.
Presumably the uninstaller components of these installers should similarly uninstall pip before uninstalling Python. Would something like a "python -m getpip uninstall" command be worthwhile to support this? (That's
automatically probably a question for the authors of the installers to answer, as I don't know if such a command would be needed - I suspect that the Windows MSI installer just records what files it installs and uninstalls them, so it may not be needed there).
But otherwise +1 on this PEP.
Yeah, a command to uninstall pip and everything it installed would be desirable. I wouldn't consider it a *blocker* for inclusion (since anyone bootstrapping pip manually already has this problem), but it would be a nice one to solve (and having it on getpip neatly avoids the issues with pip uninstalling itself on Windows). For the other questions people asked: * yes, pyvenv should gain a "--without-pip" option * the only change to the existing get-pip.py bootstrap script should be to check for "getpip" and use it if available, and otherwise continue on with the legacy bootstrap mechanism. * I agree with Donald that getpip should be an ordinary standard library Python module * To allow releases to be recreated exactly from the source tarball, I agree with Donald that we need to actually include the relevant wheel files inside the CPython source tree. Retrieving them automatically at build time would be nice, but that unfortunately creates ugly version reproducibility issues, and also requires that the build process be patched before it can run on a trusted build server with no network access. * If we have to bundle setuptools as well, so be it. However, we should document that it is *not* guaranteed to be available, so projects that need it should declare the appropriate dependencies and mention "pip install setuptools" in the appropriate places to ensure it is available * a "--wheel-only" option for both getpip and pyvenv might be interesting, if pip gets to a point where it only needs setuptools for source builds and not installing from wheels. However, for now, I think the PEP should just assume setuptools will be bundled and installed as a pip dependency. I'm OK with that - Guido was amenable to adding setuptools to the standard library back in 2.5 as a case of practicality beating purity. (In retrospect, that would have just meant setuptools suffered from the same "can't readily be updated in existing Python versions" problem as distutils, so it's probably a net positive that PJE eventually decided not to continue with the idea) Cheers, Nick.
Paul
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
There's no mention of setuptools. I guess the handling of that dependency (or not handling it up front, if pip is refactored such that setuptools just becomes a dependency when building) is considered an implementation detail of get-pip?
On Aug 30, 2013, at 12:04 PM, Marcus Smith
There's no mention of setuptools. I guess the handling of that dependency (or not handling it up front, if pip is refactored such that setuptools just becomes a dependency when building) is considered an implementation detail of get-pip?
It's not mentioned by name but instead it's said: An additional module called ``getpip`` will be added to the standard library whose purpose is to install pip and any of its dependencies into the appropriate location (most commonly site-packages). Setuptools would be the "and any of its dependencies". I want to make setuptools not depend on setuptools but I added that as an escape clause incase that proves to be hard to actually do. The PEP itself specifies only pip because I'm considering setuptools an implementation detail of pip (one that can be replaced eventually or removed from it's "special" status). ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
ok, got it. so, theoretically if the pip/setuptools relationship stays
what it is at the moment, get-pip would end up including a bundled
setuptools to fulfill the "Not all users will have network access to PyPI"
On Fri, Aug 30, 2013 at 9:07 AM, Donald Stufft
On Aug 30, 2013, at 12:04 PM, Marcus Smith
wrote: There's no mention of setuptools. I guess the handling of that dependency (or not handling it up front, if pip is refactored such that setuptools just becomes a dependency when building) is considered an implementation detail of get-pip?
It's not mentioned by name but instead it's said:
An additional module called ``getpip`` will be added to the standard library whose purpose is to install pip and any of its dependencies into the appropriate location (most commonly site-packages).
Setuptools would be the "and any of its dependencies". I want to make setuptools not depend on setuptools but I added that as an escape clause incase that proves to be hard to actually do. The PEP itself specifies only pip because I'm considering setuptools an implementation detail of pip (one that can be replaced eventually or removed from it's "special" status).
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Aug 30, 2013, at 12:12 PM, Marcus Smith
ok, got it. so, theoretically if the pip/setuptools relationship stays what it is at the moment, get-pip would end up including a bundled setuptools to fulfill the "Not all users will have network access to PyPI"
Yes. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Donald Stufft
It also proposes that the distributions of Python available via Python.org will automatically run this explicit bootstrapping method and a recommendation to third party redistributors of Python to also provide pip by default (in a way reasonable for their distributions).
Before getpip executes code it just downloaded from the 'net, how is it validated? Would getpip contain the public keys of select maintainers to verify the download? regards, Anders
On 3 September 2013 22:33, Anders J. Munch
Donald Stufft
It also proposes that the distributions of Python available via Python.org will automatically run this explicit bootstrapping method and a recommendation to third party redistributors of Python to also provide pip by default (in a way reasonable for their distributions).
Before getpip executes code it just downloaded from the 'net, how is it validated? Would getpip contain the public keys of select maintainers to verify the download?
It would be trusting the integrity of PyPI for the software itself, and the CA system to know that it's actually talking to PyPI. Far from ideal, but we don't have a viable end-to-end signing system yet (mostly due to the associated key management and update/revocation problems). Given that the trust model for the installer itself is usually "I downloaded it from python.org", the risk isn't actually increased all that much. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sep 3, 2013, at 8:47 AM, Nick Coghlan
On 3 September 2013 22:33, Anders J. Munch
wrote: Donald Stufft
It also proposes that the distributions of Python available via Python.org will automatically run this explicit bootstrapping method and a recommendation to third party redistributors of Python to also provide pip by default (in a way reasonable for their distributions).
Before getpip executes code it just downloaded from the 'net, how is it validated? Would getpip contain the public keys of select maintainers to verify the download?
It would be trusting the integrity of PyPI for the software itself, and the CA system to know that it's actually talking to PyPI. Far from ideal, but we don't have a viable end-to-end signing system yet (mostly due to the associated key management and update/revocation problems).
Given that the trust model for the installer itself is usually "I downloaded it from python.org", the risk isn't actually increased all that much.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On top of that it would gain improvements as pip itself gains improvements in this area. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Nick Coghlan:
It would be trusting the integrity of PyPI for the software itself, and the CA system to know that it's actually talking to PyPI. Far from ideal, but we don't have a viable end-to-end signing system yet (mostly due to the associated key management and update/revocation problems).
So retrieving pip is over https and the cert is validated? That's a satisfactory answer, certainly.
Given that the trust model for the installer itself is usually "I downloaded it from python.org", the risk isn't actually increased all that much.
I'd worry about any increase in risk. If the target becomes big enough, malware may start targeting Python auto-install mechanisms, even if it doesn't today. The python.org installers are PGP signed, by the way. Maybe you meant the installers retrievable through PyPI? regards, Anders
On 3 Sep 2013 23:14, "Anders J. Munch"
Nick Coghlan:
It would be trusting the integrity of PyPI for the software itself, and the CA system to know that it's actually talking to PyPI. Far from ideal, but we don't have a viable end-to-end signing system yet (mostly due to the associated key management and update/revocation problems).
So retrieving pip is over https and the cert is validated? That's a satisfactory answer, certainly.
Given that the trust model for the installer itself is usually "I downloaded it from python.org", the risk isn't actually increased all that much.
I'd worry about any increase in risk. If the target becomes big enough, malware may start targeting Python auto-install mechanisms, even if it doesn't today. The python.org installers are PGP signed, by the way. Maybe you meant the installers retrievable through PyPI?
Those too, but I meant I don't know of anyone that checks the signatures of the Windows installers before running them. Certainly beginners don't, since "setting up GPG is painful on Windows" is one of the reasons relying on it for PyPI is a problem. Sure, it can be done in *theory*, but in practice... :P For me, the bar is currently set at "more secure than it used to be" (a baseline which is fortunately higher than it used to be now both pip and easy_install do SSL cert verification, but still disturbingly low in other ways). Cheers, Nick.
regards, Anders
We have integrated PyCrypto into TUF and are planning to distribute
binaries for it along with TUF so that TUF will work smoothly on Windows,
Linux, Mac, etc.
We will have a demo that shows TUF integration into pip later this week.
It will have a bunch of example tests you can run that show how pip can be
hacked (some of which will work even if GPG signature verification was
implemented), but that TUF blocks.
More to come!
Justin
P.S. Should we make the unofficial TUF motto "more secure than it used to
be"? :)
On Tue, Sep 3, 2013 at 7:29 PM, Nick Coghlan
On 3 Sep 2013 23:14, "Anders J. Munch"
wrote: Nick Coghlan:
It would be trusting the integrity of PyPI for the software itself, and the CA system to know that it's actually talking to PyPI. Far from ideal, but we don't have a viable end-to-end signing system yet (mostly due to the associated key management and update/revocation problems).
So retrieving pip is over https and the cert is validated? That's a satisfactory answer, certainly.
Given that the trust model for the installer itself is usually "I downloaded it from python.org", the risk isn't actually increased all that much.
I'd worry about any increase in risk. If the target becomes big enough, malware may start targeting Python auto-install mechanisms, even if it doesn't today. The python.org installers are PGP signed, by the way. Maybe you meant the installers retrievable through PyPI?
Those too, but I meant I don't know of anyone that checks the signatures of the Windows installers before running them. Certainly beginners don't, since "setting up GPG is painful on Windows" is one of the reasons relying on it for PyPI is a problem. Sure, it can be done in *theory*, but in practice... :P
For me, the bar is currently set at "more secure than it used to be" (a baseline which is fortunately higher than it used to be now both pip and easy_install do SSL cert verification, but still disturbingly low in other ways).
Cheers, Nick.
regards, Anders
_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On 4 September 2013 09:55, Justin Cappos
We have integrated PyCrypto into TUF and are planning to distribute binaries for it along with TUF so that TUF will work smoothly on Windows, Linux, Mac, etc.
We will have a demo that shows TUF integration into pip later this week. It will have a bunch of example tests you can run that show how pip can be hacked (some of which will work even if GPG signature verification was implemented), but that TUF blocks.
Great to hear! (uh, well, sort of... this whole "everything is broken, but we have ways to possibly make it less broken in the future" thing really needs some more appropriate adjectives...)
More to come! Justin P.S. Should we make the unofficial TUF motto "more secure than it used to be"? :)
Heh, it's certainly one of the best ways I know to remind myself there needs to be an appropriate balance between security, backwards compatibility and usability :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (7)
-
Anders J. Munch
-
Brett Cannon
-
Donald Stufft
-
Justin Cappos
-
Marcus Smith
-
Nick Coghlan
-
Paul Moore