[Python-Dev] PEP 453 updated with improved rationale
Nick Coghlan
ncoghlan at gmail.com
Sun Sep 29 08:45:24 CEST 2013
PEP 453 has been updated in response to the previous discussion thread.
Updated version is online at http://www.python.org/dev/peps/pep-0453/
Relevant diffs can be found at:
http://hg.python.org/peps/log/bdd50f92ee73/pep-0453.txt
The bulk of the updates can be seen in the following two diffs:
http://hg.python.org/peps/rev/58f33b8eb68d
http://hg.python.org/peps/rev/438e3fa95c88
The underlying technical proposal itself hasn't actually changed from
the previous iteration: we still want to make "ensurepip" a public
module in all of Python 2.7, 3.3 and 3.4, and invoke it from the
binary installers in those versions.
However, the rationale sections of the PEP, both the general rationale
for making the proposal at all and the specific rationale for the
backporting exemption, have been updated significantly, and the PEP
now includes more detail on various parts of the proposal (especially
proposed changes to the documentation. The two backporting
compromises discussed in the previous thread (making this a
binary-installer-only feature and using a different module name for
the backported version) are covered explicitly as rejected proposals.
The two major additions to the rationale are a longer discourse on the
differences between python-dev and distutils-sig in terms of what it
means to "support" a version of Python and how this PEP is seen as a
major enabling change for the future evolution of the Python packaging
ecosystem, as well as much deeper coverage of the origins of the "no
new features in maintenance releases" rule and how this case differs
from the incidents that caused problems in the 2.2 series, and how it
serves to *weaken* the case for any future exemptions rather than
strengthen it.
Regards,
Nick.
================================
PEP: 453
Title: Explicit bootstrapping of pip in Python installations
Version: $Revision$
Last-Modified: $Date$
Author: Donald Stufft <donald at stufft.io>,
Nick Coghlan <ncoghlan at gmail.com>
BDFL-Delegate: Martin von Löwis
Status: Draft
Type: Process
Content-Type: text/x-rst
Created: 10-Aug-2013
Post-History: 30-Aug-2013, 15-Sep-2013, 18-Sep-2013, 19-Sep-2013,
23-Sep-2013, 29-Sep-2013
Abstract
========
This PEP proposes that the
`Installing Python Modules <http://docs.python.org/3/install>`__ guide be
updated to officially recommend the use of ``pip`` as the default
installer for Python packages, and that appropriate technical changes be
made in Python 2.7, 3.3 and 3.4 to provide ``pip`` by default in support
of that recommendation.
Rationale
=========
There are two related, but distinct rationales for the proposal in this
PEP. The first relates to the experience of new users, while the second
relates to better enabling the evolution of the broader Python packaging
ecosystem.
Improving the new user experience
---------------------------------
Currently, on systems without a platform package manager and repository,
installing a third-party Python package into a freshly installed Python
requires first identifying an appropriate package manager and then
installing it.
Even on systems that *do* have a platform package manager, it is unlikely to
include every package that is available on the Python Package Index, and
even when a desired third-party package is available, the correct name in
the platform package manager may not be clear.
This means that, to work effectively with the Python Package Index
ecosystem, users must know which package manager to install, where to get
it, and how to install it. The effect of this is that third-party Python
projects are currently required to choose from a variety of undesirable
alternatives:
* Assume the user already has a suitable cross-platform package manager
installed.
* Duplicate the instructions and tell their users how to install the
package manager.
* Completely forgo the use of dependencies to ease installation concerns
for their users.
All of these available options have significant drawbacks.
If a project simply assumes a user already has the tooling then beginning
users may get a confusing error message when the installation command
doesn't work. Some operating systems may ease this pain by providing a
global hook that looks for commands that don't exist and suggest an OS
package they can install to make the command work, but that only works
on systems with platform package managers that include a package that
provides the relevant cross-platform installer command (such as many major
Linux distributions). No such assistance is available for Windows and
Mac OS X users, or more conservative Linux distributions. The challenges
of dealing with this problem for beginners (who are often also completely
new to programming, the use of command line tools and editing system
environment variables) are a regular feature of feedback the core Python
developers receive from professional educators and others introducing new
users to Python.
If a project chooses to duplicate the installation instructions and tell
their users how to install the package manager before telling them how to
install their own project then whenever these instructions need updates
they need updating by every project that has duplicated them. This is
particular problematic when there are multiple competing installation
tools available, and different projects recommend different tools.
This specific problem can be partially alleviated by strongly promoting
``pip`` as the default installer and recommending that other projects
reference `pip's own bootstrapping instructions
<http://www.pip-installer.org/en/latest/installing.html>`__ rather than
duplicating them. However the user experience created by this approach
still isn't good (especially on Windows, where downloading and running
the ``get-pip.py`` bootstrap script with the default OS configuration is
significantly more painful than downloading and running a binary executable
or installer). The situation becomes even more complicated when multiple
Python versions are involved (for example, parallel installations of
Python 2 and Python 3), since that makes it harder to create and maintain
good platform specific ``pip`` installers independently of the CPython
installers.
The projects that have decided to forgo dependencies altogether are forced
to either duplicate the efforts of other projects by inventing their own
solutions to problems or are required to simply include the other projects
in their own source trees. Both of these options present their own problems
either in duplicating maintenance work across the ecosystem or potentially
leaving users vulnerable to security issues because the included code or
duplicated efforts are not automatically updated when upstream releases a new
version.
By officially recommending and providing by default a specific cross-platform
package manager it will be easier for users trying to install these
third-party packages as well as easier for the people distributing them as
they should now be able to safely assume that most users will have the
appropriate installation tools available (or access to clear instructions on
how to obtain them). This is expected to become more important in the future
as the Wheel_ package format (deliberately) does not have a built in
"installer" in the form of ``setup.py`` so users wishing to install
from a wheel file will want an installer even in the simplest cases.
Reducing the burden of actually installing a third-party package should
also decrease the pressure to add every useful module to the standard
library. This will allow additions to the standard library to focus more
on why Python should have a particular tool out of the box, and why it
is reasonable for that package to adopt the standard library's 18-24 month
feature release cycle, instead of using the general difficulty of installing
third-party packages as justification for inclusion.
Providing a standard installation system also helps with bootstrapping
alternate build and installer systems, such as ``setuptools``,
``zc.buildout`` and the ``hashdist``/``conda`` combination that is aimed
specifically at the scientific community. So long as
``pip install <tool>`` works, then a standard Python-specific installer
provides a reasonably secure, cross platform mechanism to get access to
these utilities.
Enabling the evolution of the broader Python packaging ecosystem
----------------------------------------------------------------
As no new packaging standard can achieve widespread adoption without a
transition strategy that covers the versions of Python that are in
widespread *current* use (rather than merely future versions, like most
language features), the change proposed in this PEP is considered a
necessary step in the evolution of the Python packaging ecosystem
The broader community has embraced the Python Package Index as a mechanism
for distributing and installing Python software, but the different concerns
of language evolution and secure software distribution mean that a faster
feature release cycle that encompasses older versions is needed to properly
support the latter.
In addition, the core CPython development team have the luxury of
dropping support for earlier Python versions well before the rest of the
community, as downstream commercial redistributors pick up the task of
providing support for those versions to users that still need it, while
many third party libraries maintain compatibility with those versions as
long as they remain in widespread use.
This means that the current ``setup.py install`` based model for package
installation poses serious difficulties for the development and adoption
of new packaging standards, as, depending on how a project writes their
``setup.py`` file, the installation command (along with other operations)
may end up invoking the standard library's ``distutils`` package.
As an indicator of how this may cause problems for the broader ecosystem,
consider that the feature set of ``distutils`` in Python 2.6 was frozen
in June 2008 (with the release of Python 2.6b1), while the feature set of
``distutils`` in Python 2.7 was frozen in April 2010 (with the release of
Python 2.7b1).
By contrast, using a separate installer application like ``pip`` (which
ensures that even ``setup.py`` files that invoke ``distutils`` directly
still support the new packaging standards) makes it possible to support
new packaging standards in older versions of Python, just by upgrading
``pip`` (which receives new feature releases roughly every 6 months). The
situation on older versions of Python is further improved by making it
easier for end users to install and upgrade newer build systems like
``setuptools`` or improved PyPI upload utilities like ``twine``.
It is not coincidental that this proposed model of using a separate installer
program with more metadata heavy and less active distribution formats matches
that used by most operating systems (including Windows since the introduction
of the installer service and the MSI file format), as well as many other
language specific installers.
For Python 2.6, this compatibility issue is largely limited to various
enterprise Linux distributions (and their downstream derivatives). These
distributions often have even slower update cycles than CPython, so they
offer full support for versions of Python that are considered "security
fix only" versions upstream (and sometimes may even be to the point where
the core development team no longer support them at all - you can still get
commercial support for Python 2.3 if you really need it!).
In practice, the fact that tools like ``wget`` and ``curl`` are readily
available on Linux systems, that most users of Python on Linux are
already familiar with the command line, and that most Linux distributions
ship with a default configuration that makes running Python scripts easy,
means that the existing ``pip`` bootstrapping instructions for any \*nix
system are already quite straightforward. Even if ``pip`` isn't provided by
the system package manager, then using ``wget`` or ``curl`` to retrieve the
bootstrap script from www.pip-installer.org and then running it is just a
couple of shell commands that can easily be copied and pasted as necessary.
Accordingly, for any version of Python on any \*nix system, the need to
bootstrap ``pip`` in older versions isn't considered a major barrier to
adoption of new packaging standards, since it's just one more small
speedbump encountered by users of these long term stable releases. For
\*nix systems, this PEP's formal endorsement of ``pip`` as the preferred
default packaging tool is seen as more important than the underlying
technical details involved in making ``pip`` available by default, since
it shifts the nature of the conversation between the developers of ``pip``
and downstream repackagers of both ``pip`` and CPython.
For Python 2.7, on the other hand, the compatibility issue for adopting new
metadata standards is far more widespread, as it affects the python.org
binary installers for Windows and Mac OS X, as well as even relatively
fast moving \*nix platforms.
Firstly, and unlike Python 2.6, Python 2.7 is still a fully supported
upstream version, and will remain so until the release of Python 2.7.9
(currently scheduled for May 2015), at which time it is expected to enter
the usual "security fix only" mode. That means there are at least another
19 months where Python 2.7 is a deployment target for Python applications
that enjoys full upstream support. Even after the core development team
switches 2.7 to security release only mode in 2015, Python 2.7 will likely
remain a commercially supported legacy target out beyond 2020.
While Python 3 already presents a compelling alternative over Python 2 for
*new* Python applications and deployments without an existing investment
in Python 2 and without a dependency on specific Python 2 only third party
modules (a set which is getting ever smaller over time), it is going to take
longer to create compelling business cases to update existing Python 2.7
based infrastructure to Python 3, especially in situations where the culture
of automated testing is weak (or nonexistent), making it difficult to
effectively use the available migration utilities.
It is quite likely that it is this difference in perspective regarding what
it means for a version of Python to be "supported" which lies at the heart
of the long history of conflicts between the developers of Python packaging
tools and the core development team for CPython. A key goal of this PEP is
thus to better enable the two groups to collaborate more effectively, by
using the ``ensurepip`` module as the technical bridge between the two
distinct software lifecycles and deployment models.
Why pip?
--------
``pip`` has been chosen as the preferred default installer, as it is an
already popular tool that addresses several design and user experience
issues with its predecessor ``easy_install`` (these issues can't readily
be fixed in ``easy_install`` itself due to backwards compatibility
concerns). ``pip`` is also well suited to working within the bounds of
a single Python runtime installation (including associated virtual
environments), which is a desirable feature for a tool bundled with CPython.
Other tools like ``zc.buildout`` and ``conda`` are more ambitious in their
aims (and hence substantially better than ``pip`` at handling external
binary dependencies), so it makes sense for the Python ecosystem to treat
them more like platform package managers to interoperate with rather than
as the default cross-platform installation tool. This relationship is
similar to that between ``pip`` and platform package management systems
like ``apt`` and ``yum`` (which are also designed to handle arbitrary
binary dependencies).
Proposal Overview
=================
This PEP proposes that the
`Installing Python Modules <http://docs.python.org/3/install>`__ guide be
updated to officially recommend the use of ``pip`` as the default
installer for Python packages, rather than the current approach of
recommending the direct invocation of the ``setup.py install`` ``distutils``
command.
However, to avoid recommending a tool that CPython does not provide, it is
further proposed that the `pip`_ package manager be made available by
default when installing CPython and when creating virtual environments
using the standard library's ``venv`` module via the ``pyvenv`` command line
utility).
To support that end, this PEP proposes the inclusion of an ``ensurepip``
bootstrapping module in Python 3.4 (along with the upcoming maintenance
releases of Python 2.7 and 3.3), as well as changes to the way Python
installed scripts are handled on Windows (for Python 3.4 only).
To clearly demarcate development responsibilities, and to avoid
inadvertently downgrading ``pip`` when updating CPython, the proposed
mechanism to achieve this is to include an explicit `pip`_ bootstrapping
mechanism in the standard library that is invoked automatically by the
CPython installers provided on python.org.
To ensure the smoothest possible experience for new users of Python, this
PEP also proposes that the ``ensurepip`` module and the option to install
``pip`` when installing CPython be backported to Python 2.7 and 3.3. It
does *not* propose backporting any changes to ``pyvenv`` (in Python 3.3)
or to Windows script handling (in either version).
Finally, the PEP also strongly recommends that CPython redistributors and
other Python implementations ensure that ``pip`` is available by default, or
at the very least, explicitly document the fact that it is not included.
This PEP does *not* propose making pip (or any dependencies) directly
available as part of the standard library. Instead, pip will be a
bundled application provided along with CPython for the convenience
of Python users, but subject to its own development life cycle and able
to be upgraded independently of the core interpreter and standard library.
Explicit bootstrapping mechanism
================================
An additional module called ``ensurepip`` will be added to the standard
library whose purpose is to install pip and any of its dependencies into the
appropriate location (most commonly site-packages). It will expose a
callable named ``bootstrap()`` as well as offer direct execution via
``python -m ensurepip``.
The bootstrap will *not* contact PyPI, but instead rely on a private copy
of pip stored inside the standard library. Accordingly, only options
related to the installation location will be supported (``--user``,
``--root``, etc).
It is considered desirable that users be strongly encouraged to use the
latest available version of ``pip``, in order to take advantage of the
ongoing efforts to improve the security of the PyPI based ecosystem, as
well as benefiting from the efforts to improve the speed, reliability and
flexibility of that ecosystem.
In order to satisfy this goal of providing the most recent version of
``pip`` by default, the private copy of ``pip`` will be updated in CPython
maintenance releases, which should align well with the 6-month cycle used
for new ``pip`` releases.
Security considerations
-----------------------
The design in this PEP has been deliberately chosen to avoid making any
significant changes to the trust model of the CPython installers for end
users that do not subsequently make use of ``pip``.
The installers will contain all the components of a fully functioning
version of Python, including the ``pip`` installer. The installation
process will *not* require network access, and will *not* rely on
trusting the security of the network connection established between
``pip`` and the Python package index.
Only users that choose to use ``pip`` directly will need to pay
attention to any PyPI related security considerations.
Reliability considerations
--------------------------
By including the bootstrap as part of the standard library (rather than
solely as a feature of the binary installers), the correct operation of
the bootstrap command can be easily tested using the existing CPython
buildbot infrastructure rather than adding significantly to the testing
burden for the installers themselves.
Implementation strategy
-----------------------
To ensure there is no need for network access when installing Python or
creating virtual environments, the ``ensurepip`` module will, as an
implementation detail, include a complete private copy of pip and its
dependencies which will be used to extract pip and install it into the target
environment. It is important to stress that this private copy of pip is
*only* an implementation detail and it should *not* be relied on or
assumed to exist beyond the public capabilities exposed through the
``ensurepip`` module (and indirectly through ``venv``).
There is not yet a reference ``ensurepip`` implementation. The existing
``get-pip.py`` bootstrap script demonstrates an earlier variation of the
general concept, but the standard library version would take advantage of
the improved distribution capabilities offered by the CPython installers
to include private copies of ``pip`` and ``setuptools`` as wheel files
(rather than as embedded base64 encoded data), and would not try to
contact PyPI (instead installing directly from the private wheel files.
Rather than including separate code to handle the bootstrapping, the
``ensurepip`` module will manipulate ``sys.path`` appropriately to allow
the wheel files to be used to install themselves, either into the current
Python installation or into a virtual environment (as determined by the
options passed to the bootstrap command).
It is proposed that the implementation be carried out in five separate
steps (all steps after the first are independent of each other and can be
carried out in any order):
* the first step would add the ``ensurepip`` module and the private copies
of the most recently released versions of pip and setuptools, and update
the "Installing Python Modules" documentation. This change
would be applied to Python 2.7, 3.3, and 3.4.
* the Windows installer would be updated to offer the new ``pip``
installation option for Python 2.7.6, 3.3.3 and 3.4.0.
* the Mac OS X installer would be updated to offer the new ``pip``
installation option for Python 2.7.6, 3.3.3 and 3.4.0.
* the ``venv`` module and ``pyvenv`` command would be updated to make use
of ``ensurepip`` in Python 3.4+
* the PATH handling and ``sysconfig`` directory layout on Windows would be
updated for Python 3.4+
Proposed CLI
------------
The proposed CLI is based on a subset of the existing ``pip install``
options::
Usage:
python -m ensurepip [options]
General Options:
-h, --help Show help.
-v, --verbose Give more output. Option is additive, and
can be used up to 3 times.
-V, --version Show the pip version that would be extracted and exit.
-q, --quiet Give less output.
Installation Options:
-U, --upgrade Upgrade pip and dependencies, even if
already installed
--user Install using the user scheme.
--root <dir> Install everything relative to this
alternate root directory.
In most cases, end users won't need to use this CLI directly, as ``pip``
should have been installed automatically when installing Python or when
creating a virtual environment. However, it is formally documented as a
public interface to support at least these known use cases:
* Windows and Mac OS X installations where the "Install pip" option was
*not* chosen during installation
* any installation where the user previously ran "pip uninstall pip"
* virtual environments created with ``pyvenv`` in Python 3.3
Users that want to retrieve the latest version from PyPI, or otherwise
need more flexibility, can then invoke the extracted ``pip`` appropriately.
Proposed module API
-------------------
The proposed ``ensurepip`` module API consists of the following two
functions::
def version():
"""
Returns a string specifying the bundled version of pip.
"""
def bootstrap(root=None, upgrade=False, user=False, verbosity=0):
"""
Bootstrap pip into the current Python installation (or the given root
directory).
"""
Invocation from the CPython installers
--------------------------------------
The CPython Windows and Mac OS X installers will each gain a new option:
* Install pip (the default Python package management utility)?
This option will be checked by default.
If the option is checked, then the installer will invoke the following
command with the just installed Python::
python -m ensurepip --upgrade
This ensures that, by default, installing or updating CPython will ensure
that the installed version of pip is at least as recent as the one included
with that version of CPython. If a newer version of pip has already been
installed then ``python -m ensurepip --upgrade`` will simply return without
doing anything.
Installing from source
----------------------
While the prebuilt binary installers will be updated to run
``python -m ensurepip`` by default, no such change will be made to the
``make install`` and ``make altinstall`` commands of the source
distribution.
``ensurepip`` itself (including the private copy of ``pip`` and its
dependencies) will still be installed normally (as it is a regular
part of the standard library), only the implicit installation of pip and
its dependencies will be skipped.
Keeping the pip bootstrapping as a separate step for ``make``-based
installations should minimize the changes CPython redistributors need to
make to their build processes. Avoiding the layer of indirection through
``make`` for the ``ensurepip`` invocation avoids any challenges
associated with determining where to install the extracted ``pip``.
Changes to virtual environments
-------------------------------
Python 3.3 included a standard library approach to virtual Python environments
through the ``venv`` module. Since its release it has become clear that very
few users have been willing to use this feature directly, in part due to the
lack of an installer present by default inside of the virtual environment.
They have instead opted to continue using the ``virtualenv`` package which
*does* include pip installed by default.
To make the ``venv`` more useful to users it will be modified to issue the
pip bootstrap by default inside of the new environment while creating it. This
will allow people the same convenience inside of the virtual environment as
this PEP provides outside of it as well as bringing the ``venv`` module closer
to feature parity with the external ``virtualenv`` package, making it a more
suitable replacement.
To handle cases where a user does not wish to have pip bootstrapped into
their virtual environment a ``--without-pip`` option will be
added.
The ``venv.EnvBuilder`` and ``venv.create`` APIs will be updated to accept
one new parameter: ``with_pip`` (defaulting to ``False``).
The new default for the module API is chosen for backwards compatibility
with the current behaviour (as it is assumed that most invocation of the
``venv`` module happens through third part tools that likely will not
want ``pip`` installed without explicitly requesting it), while the
default for the command line interface is chosen to try to ensure ``pip``
is available in most virtual environments without additional action on the
part of the end user.
This particular change will be made only for Python 3.4 and later versions.
The third-party ``virtualenv`` project will still be needed to obtain a
consistent cross-version experience in Python 3.3 and 2.7.
Documentation
-------------
The "Installing Python Modules" section of the standard library
documentation in Python 2.7, 3.3 and 3.4 will be updated to recommend
the use of the bootstrapped ``pip`` installer. It will give a brief
description of the most common commands and options, but delegate
to the externally maintained ``pip`` documentation for the full details.
In Python 3.4, the ``pyvenv`` and ``venv`` documentation will also be
updated to reference the revised module installation guide.
In Python 2.7 and 3.3, the documentation will make clear that the feature
was added in a maintenance release and users may need to upgrade in order
to take advantage of it. Specifically, it is proposed to include the
following warning as a note in the documentation for the ``ensurepip``
module in these versions (adjust version numbers for 3.3 as appropriate):
This is an optional module, which may not be available in all
installations of Python 2.7. It is provided solely to simplify the
process of bootstrapping ``pip`` onto end user's systems. If it is
not available, please investigate the following alternatives:
* This module was first added in Python 2.7.6. If using an earlier
maintenance release, it will not be available. If upgrading to
a more recent maintenance release is not an option, consider
the alternative bootstrapping mechanisms below.
* Some platforms provide alternative mechanisms to obtain ``pip``.
In such cases, the platform documentation should provide
appropriate details.
* If upgrading to the latest maintenance release is not feasible, and
no platform specific instructions are provided, then refer to the
upstream `pip bootstrapping instructions
<http://www.pip-installer.org/en/latest/installing.html>`__.
The existing content of the module installation guide will be retained in
all versions, but under a new "Invoking distutils directly" subsection.
Bundling CA certificates with CPython
-------------------------------------
The ``ensurepip`` implementation will include the ``pip`` CA bundle along
with the rest of ``pip``. This means CPython effectively includes
a CA bundle that is used solely by ``pip`` after it has been extracted.
This is considered preferable to relying solely on the system
certificate stores, as it ensures that ``pip`` will behave the same
across all supported versions of Python, even those prior to Python 3.4
that cannot access the system certificate store on Windows.
Automatic installation of setuptools
------------------------------------
``pip`` currently depends on ``setuptools`` to handle metadata generation
during the build process, along with some other features. While work is
ongoing to reduce or eliminate this dependency, it is not clear if that
work will be complete for pip 1.5 (which is the version likely to be current
when Python 3.4.0 is released).
This PEP proposes that, if pip still requires it as a dependency,
``ensurepip`` will include a private copy of ``setuptools`` (in addition
to the private copy of ``ensurepip``). ``python -m ensurepip`` will then
install the private copy in addition to installing ``pip`` itself.
However, this behavior is officially considered an implementation
detail. Other projects which explicitly require ``setuptools`` must still
provide an appropriate dependency declaration, rather than assuming
``setuptools`` will always be installed alongside ``pip``.
Once pip is able to run ``pip install --upgrade pip`` without needing
``setuptools`` installed first, then the private copy of ``setuptools``
will be removed from ``ensurepip`` in subsequent CPython releases.
As long as setuptools is needed, it will be a completely unmodified copy of
the latest upstream setuptools release, including the ``easy_install``
script if the upstream setuptools continues to include it. The installation
of ``easy_install`` along with ``pip`` isn't considered desirable, but
installing a broken setuptools would be worse. This problem will
naturally resolve itself once the ``pip`` developers have managed to
eliminate their dependency on ``setuptools`` and the private copy of
``setuptools`` can be removed entirely from CPython.
Updating the private copy of pip
--------------------------------
In order to keep up with evolutions in packaging as well as providing users
with as recent version a possible the ``ensurepip`` module will be
regularly updated to the latest versions of everything it bootstraps.
After each new ``pip`` release, and again during the preparation for any
release of Python (including feature releases), a script, provided as part
of the implementation for this PEP, will be run to ensure the private
copies stored in the CPython source repository have been updated to the
latest versions.
Updating the ensurepip module API and CLI
-----------------------------------------
Like ``venv`` and ``pyvenv``, the ``ensurepip`` module API and CLI
will be governed by the normal rules for the standard library: no
new features are permitted in maintenance releases.
However, the embedded components may be updated as noted above, so
the extracted ``pip`` may offer additional functionality in maintenance
releases.
Feature addition in maintenance releases
========================================
Adding a new module to the standard library in Python 2.7, and 3.3
maintenance releases breaks the usual policy of "no new features in
maintenance releases". The rationale for doing so in this case is slightly
different for each of the two versions.
Rationale for this policy on maintenance releases
-------------------------------------------------
Python's strict "no new features in maintenance releases" was instituted
following the introduction of a number of new features over the course of
the Python 2.2.x series.
Firstly, the ``True`` and ``False`` builtins were added in Python 2.2.1 (at
the time, they were merely aliases for the values ``1`` and ``0``,
in Python 2.3 they became instances of the new ``bool`` type and in Python
3.0 they became true constants recognised by the compiler).
Python 2.2.2 then made the situation worse by adding a new ``chars``
parameter to the ``lstrip`` and ``rstrip`` string methods, along with an
entirely new ``zfill`` method. The corresponding changes in the
``string`` module were not incorporated until Python 2.2.3.
The reason introducing new features in maintenance releases like this is
problematic is that, except in the cases where code fails due to a bug in
CPython, developers expect to be able to identify the supported Python
versions for a library or application solely through the first two
components of the version number.
The introduction of new builtins and string methods in Python 2.2.1 and
2.2.2 resulted in many developers claiming Python 2.2 compatibility for
code that did not in fact run on the original Python 2.2. In effect,
Python 2.2.2 became the minimum usable version, since there was a
relatively high chance of code breaking when run on 2.2 (or even 2.2.1).
Scope of this proposal
----------------------
By contrast with the changes that caused such problems during the 2.2.x
series, this PEP is merely proposing the addition of a new standard
library module, rather than adding new builtins or changing the interface
of a builtin type.
The categorical difference between these kinds of changes has already been
recognised in the Python 3.3 Language Moratorium (PEP 3003), where the
addition of new builtins was disallowed outright and changes to builtin
types required an explicit exemption. By contrast, adding new modules was
explicitly permitted, even while the moratorium was in place.
Furthermore, the proposed ``ensurepip`` module is only a means to the end of
getting ``pip`` installed on the system. While "upgrade to the latest
CPython maintenance release" will become the *recommended* approach to
obtaining ``pip`` for users of Python 2.7 and 3.3 on Windows and Mac OS X
systems, all of the existing ``pip`` bootstrapping mechanisms will still
work in cases where upgrading Python isn't a practical alternative.
As described in the documentation update proposal, the ``ensurepip``
documentation in older releases will include the text explaining how to
obtain ``pip`` if updating to the latest maintenance release isn't an
option, or if the module has been removed by a redistributor. This contrasts
significantly with the changes made during the Python 2.2 series, where they
were normal additions with no alternatives except to update to a sufficiently
recent version of Python if a library or application depended on them.
Potential consequences of permitting this exemption
---------------------------------------------------
The concern has been expressed that approving an exemption to the "no new
features in maintenance releases" policy in this case will open the
flood gates to requests for more such exemptions in the future. It is the
perspective of the PEP authors that the specific nature of this proposal
should itself serve to allay those fears.
Firstly, as a proposal to add a new module to the standard library, granting
an exemption in this case sets no precedent for the more restricted
categories identified in the PEP 3003 language moratorium.
Secondly, this exemption is requested for a new module that *makes it easy
to download other modules from PyPI*. If this PEP is accepted, then it can
be reasonably assumed that modules on PyPI are only a ``pip install`` away
for most users, with only those users that depend on standard library
inclusion to make it through corporate compliance reviews still affected
(and, for many such reviews, inclusion in a future version of the standard
library will be enough for a backported version to be considered
acceptable for use).
Making ``pip`` readily available in all versions still under normal
maintenance thus means that accepting this PEP should have the effect of
*weakening* the case for any further exemptions to the policy, rather
than strengthening it.
Rationale for permitting the exemption in Python 2.7
----------------------------------------------------
While Python 3 adoption is proceeding nicely, it remains the case that many
new users of Python are introduced to Python 2.7 first. This may be
because their instructors have yet to migrate their courses to Python 3, or
because they work in an environment where Python 2 is still the preferred
version, or simply because the frequently adopted approach of writing
libraries in the common Python 2/3 subset means there are (as of
September 2013) still more Python 2 only libraries than there are Python 3
only libraries.
Since one of the primary aims of this PEP is to aid new Python users, it is
contrary to its spirit to target *only* Python 3.4, when so many users in
at least the next 12-18 months (where Python 2.7 is still fully supported
by the core development team) are still going to be introduced to Python
2 before being introduced to Python 3.
Users first installing Python 2.7 on Windows and Mac OS X following
acceptance and release of this PEP won't even need to look up how to
bootstrap ``pip``, since it will already be provided with the CPython
installer. For those that already have Python installed, but are just
beginning to explore the PyPI ecosystem, the bootstrapping instructions
can be simplified to "just install the latest maintenance release of
CPython".
Making ``pip`` readily available also serves to ease the migration path
from Python 2 to Python 3, as a number of the standard library additions
in Python 3 are also available from PyPI for Python 2. Lowering the barrier
to adoption for these backports makes it easier for current Python 2 users
to selectively adopt backporting Python 3 versions, reducing the number of
updates needed in any eventual Python 3 migration.
Finally, this PEP solves a serious problem for the ``distutils-sig``
community, as it means we will finally have a standard mechanism decoupled
from the standard library's development lifecycle that we can reasonably
assume to be present on end user's systems (or at least readily
available) that allows us to support new packaging standards in older
versions of Python. A tentative, half-hearted endorsement from the
CPython core development team that tries to hide the existence of the
pip boostrapping support from end users is unlikely to provide quite the
same benefits.
Rationale for permitting the exemption in Python 3.3
----------------------------------------------------
The rationale for permitting the exemption in Python 3.3 is admittedly
not as strong as it is for Python 2.7, as instructors currently using
Python 3.3 are quite likely to upgrade to Python 3.4 shortly after it is
released.
In the case of Python 3.3, the rationale is primarily a consistency
argument, as it permits the recommended ``pip`` bootstrapping instructions
for both 2.7 and 3.3 to be to upgrade to the latest maintenance version of
CPython. While existing bootstrapping mechanisms will still be supported,
the cases where they are needed should be reduced significantly.
Adding the ``ensurepip`` module in Python 3.3 also makes the Python 3.3
version of the ``pyvenv`` utility far more useful (even without the
integration proposed for Python 3.4), as it allows users to execute
``python -m ensurepip`` to bootstrap ``pip`` after activating an
existing or newly created virtual environment.
Uninstallation
==============
No changes are proposed to the CPython uninstallation process by this PEP.
The bootstrapped pip will be installed the same way as any other pip
installed packages, and will be handled in the same way as any other
post-install additions to the Python environment.
At least on Windows, that means the bootstrapped files will be
left behind after uninstallation, since those files won't be associated
with the Python MSI installer.
While the case can be made for the CPython installers clearing out these
directories automatically, changing that behaviour is considered outside
the scope of this PEP.
Script Execution on Windows
===========================
While the Windows installer was updated in Python 3.3 to optionally
make ``python`` available on the PATH, no such change was made to
include the Scripts directory. Independently of this PEP, a proposal has
also been made to rename the ``Tools\Scripts`` subdirectory to ``bin`` in
order to improve consistency with the typical script installation directory
names on \*nix systems.
Accordingly, in addition to adding the option to extract and install ``pip``
during installation, this PEP proposes that the Windows installer (and
``sysconfig``) in Python 3.4 and later be updated to:
- install scripts to PythonXY\bin rather than PythonXY\Tools\Scripts
- add PythonXY\bin to the Windows PATH (in addition to PythonXY) when the
PATH modification option is enabled during installation
For Python 2.7 and 3.3, it is proposed that the only change be the one to
bootstrap ``pip`` by default.
This means that, for Python 3.3, the most reliable way to invoke pip
globally on Windows (without tinkering manually with PATH) will actually be
``py -m pip`` (or ``py -3 -m pip`` to select the Python 3 version if both
Python 2 and 3 are installed) rather than simply calling ``pip``.
For Python 2.7 and 3.2, the most reliable mechanism will be to install the
standalone Python launcher for Windows and then use ``py -m pip`` as noted
above.
Adding the scripts directory to the system PATH would mean that ``pip``
works reliably in the "only one Python installation on the system PATH"
case, with ``py -m pip``, ``pipX``, or ``pipX.Y`` needed only to select a
non-default version in the parallel installation case (and outside a virtual
environment). This change should also make the ``pyvenv`` command substantially
easier to invoke on Windows, along with all scripts installed by ``pip``,
``easy_install`` and similar tools.
While the script invocations on recent versions of Python will run through
the Python launcher for Windows, this shouldn't cause any issues, as long
as the Python files in the Scripts directory correctly specify a Python version
in their shebang line or have an adjacent Windows executable (as
``easy_install`` and ``pip`` do).
Recommendations for Downstream Distributors
===========================================
A common source of Python installations are through downstream distributors
such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX
package managers [#homebrew]_ [#macports]_ [#fink]_, and commercial Python
redistributors [#ContinuumIO]_ [#ActiveState]_ [#Enthought]_. In order to
provide a consistent, user-friendly experience to all users of Python
regardless of how they obtained Python this PEP recommends and asks that
downstream distributors:
* Ensure that whenever Python is installed ``pip`` is either installed or is
otherwise made readily available to end users.
* For redistributors using binary installers, this may take the form of
optionally executing the ``ensurepip`` bootstrap during installation,
similar to the CPython installers.
* For redistributors using package management systems, it may take the
form of separate packages with dependencies on each other so that
installing the Python package installs the pip package and installing
the pip package installs the Python package.
* Another reasonable way to implement this is to package pip separately but
ensure that there is some sort of global hook that will recommend
installing the separate pip package when a user executes ``pip`` without
it being installed. Systems that choose this option should ensure that
the ``ensurepip`` module still installs pip directly when invoked inside
a virtual environment, but may modify the module in the system Python
installation to redirect to the platform provided mechanism when
installing ``pip`` globally.
* Even if pip is made available globally by other means, do not remove the
``ensurepip`` module in Python 3.3 or later.
* In Python 3.3, ``ensurepip`` will be the recommended way of bootstrapping
pip in virtual environments created through the ``venv`` module and the
associated ``pyvenv`` command line tool.
* Starting with Python 3.4, ``ensurepip`` will be required for automatic
installation of pip into virtual environments by the ``venv`` module.
* This is similar to the existing ``virtualenv`` package for which many
downstream distributors have already made exception to the common
"debundling" policy.
* This does mean that if ``pip`` needs to be updated due to a security
issue, so does the private copy in the ``ensurepip`` bootstrap module
* However, altering the private copy of pip to remove the embedded
CA certificate bundle and rely on the system CA bundle instead is a
reasonable change.
* If ``pip`` is made available globally by other means in Python 2.7, then
it is acceptable (although not desirable) to disable the ``ensurepip``
module (as the third party ``virtualenv`` distribution is needed to
create virtual environments in Python 2.7 and ``virtualenv`` already
ensures ``pip`` is installed into the virtual environments it creates).
Redistributors that take this course should ensure an appropriate error
message is displayed if users attempt to import ``ensurepip``, rather
than simply removing it entirely.
* Ensure that all features of this PEP continue to work with any modifications
made to the redistributed version of Python.
* Checking the version of pip that will be bootstrapped using
``python -m ensurepip --version`` or ``ensurepip.version()``.
* Installation of pip into a global or virtual python environment using
``python -m ensurepip`` or ``ensurepip.bootstrap()``.
* ``pip install --upgrade pip`` in a global installation should not affect
any already created virtual environments (but is permitted to affect
future virtual environments, even though it will not do so when using
the standard implementation of ``ensurepip``).
* ``pip install --upgrade pip`` in a virtual environment should not affect
the global installation.
* Migrate build systems to utilize `pip`_ and `Wheel`_ wherever feasible
and avoid directly invoking ``setup.py``.
* This will help ensure a smoother and more timely migration to improved
metadata formats as the Python packaging ecosystem continues to evolve.
In the event that a Python redistributor chooses *not* to follow these
recommendations, we request that they explicitly document this fact and
provide their users with suitable guidance on translating upstream ``pip``
based installation instructions into something appropriate for the platform.
Other Python implementations are also encouraged to follow these guidelines
where applicable.
Policies & Governance
=====================
The maintainers of the bootstrapped software and the CPython core team will
work together in order to address the needs of both. The bootstrapped
software will still remain external to CPython and this PEP does not
include CPython subsuming the development responsibilities or design
decisions of the bootstrapped software. This PEP aims to decrease the
burden on end users wanting to use third-party packages and the
decisions inside it are pragmatic ones that represent the trust that the
Python community has already placed in the Python Packaging Authority as
the authors and maintainers of ``pip``, ``setuptools``, PyPI, ``virtualenv``
and other related projects.
Backwards Compatibility
-----------------------
The public API and CLI of the ``ensurepip`` module itself will fall under
the typical backwards compatibility policy of Python for its standard
library. The externally developed software that this PEP bundles does not.
Most importantly, this means that the bootstrapped version of pip may gain
new features in CPython maintenance releases, and pip continues to operate on
its own 6 month release cycle rather than CPython's 18-24 month cycle.
Security Releases
-----------------
Any security update that affects the ``ensurepip`` module will be shared
prior to release with the Python Security Response Team
(security at python.org). The PSRT will then decide if the reported issue
warrants a security release of CPython with an updated private copy of
``pip``.
Licensing
---------
``pip`` is currently licensed as 1 Clause BSD, and it contains code taken
from other projects. Additionally this PEP will include setuptools until
such time as pip no longer requires it. The licenses for these appear in
the table below.
================= ============
Project License
================= ============
requests Apache 2.0
six 1 Clause BSD
html5lib 1 Clause BSD
distlib PSF
colorama 3 Clause BSD
Mozilla CA Bundle LGPL
setuptools PSF
================= ============
All of these licenses should be compatible with the PSF license. Additionally
it is unclear if a CA Bundle is copyrightable material and thus if it needs
or can be licensed at all.
Appendix: Rejected Proposals
============================
Include pip *only* inside the installers in Python 2.7, and 3.3
---------------------------------------------------------------
An alternative to making an exception to the "no new features" policy in
Python 2.7 and 3.3 would be to simply bundle pip with the installer and not
modify the source tree at all. The motivation behind this modification is
that adding a new feature in a maintenance release is a risky proposition
and that doing it in this way doesn't violate that policy.
This has been rejected because:
* It's dubious to declare the binary installers beyond the scope of the
"no new features in maintenance releases" policy. If the rationale for
adding this feature to the standard library in a maintenance release isn't
considered adequate, then it isn't clear why moving that complexity to the
binary installers should change the verdict.
* Attempting to hide the existence of the bootstrap module from end users
makes it more difficult to write updated package installation documentation
for Python 2.7 and 3.3
* For 3.3 users that choose to use ``pyvenv`` rather than ``virtualenv``,
an explicit ``python -m ensurepip`` will be needed to bootstrap ``pip``
into virtual environments. This can only be documented clearly if the
module is public
* Making the bootstrap an installer only feature in Python 2.7 and 3.3
guarantees the introduction of cross-platform inconsistencies, whereas
the proposal in this PEP more strongly encourages redistributors to
offer a more consistent user experience.
* Making the bootstrap an installer only feature in Python 2.7 and 3.3
would make it difficult to re-use the bootstrap implementation from 3.4.
* Making the bootstrap an installer only feature prevents the buildbots
from being able to run automatic tests against it, which would make
ensuring that this feature remains working a much more difficult task.
Use a different module name in Python 2.7, and 3.3
--------------------------------------------------
Naming the module ``_ensurepip`` in Python 2.7 and 3.3 was considered as
another means of skirting the "no new features in maintenance releases"
policy. However, similar to the proposal to only include the new
feature in the installers rather than the standard library, this feels like
relying on a technicality to nominally "comply" with the policy, while still
breaking it in spirit.
It is the considered opinion of the PEP authors that attempting to hide
the addition of the ``ensurepip`` module in earlier versions will only
serve to increase confusion rather than to reduce it, so the proposal
remains to be up front about the fact that the policy is being broken in
this case, and clearly documenting the rationale for doing so in this PEP.
As noted in the section describing the proposed documentation updates,
having ``ensurepip`` as a public module in these earlier versions also
provides a convenient home for the fallback bootstrapping instructions in
those cases where it *isn't* available.
Automatically contacting PyPI when bootstrapping pip
----------------------------------------------------
Earlier versions of this PEP called the bootstrapping module ``getpip`` and
defaulted to downloading and installing ``pip`` from PyPI, with the private
copy used only as a fallback option or when explicitly requested.
This resulted in several complex edge cases, along with difficulties in
defining a clean API and CLI for the bootstrap module. It also significantly
altered the default trust model for the binary installers published on
python.org, as end users would need to explicitly *opt-out* of trusting
the security of the PyPI ecosystem (rather than opting in to it by
explicitly invoking ``pip`` following installation).
As a result, the PEP was simplified to the current design, where the
bootstrapping *always* uses the private copy of ``pip``. Contacting PyPI
is now always an explicit separate step, with direct access to the full
pip interface.
Implicit bootstrap
------------------
`PEP439`_, the predecessor for this PEP, proposes its own solution. Its
solution involves shipping a fake ``pip`` command that when executed would
implicitly bootstrap and install pip if it does not already exist. This has
been rejected because it is too "magical". It hides from the end user when
exactly the pip command will be installed or that it is being installed at
all. It also does not provide any recommendations or considerations towards
downstream packagers who wish to manage the globally installed pip through
the mechanisms typical for their system.
The implicit bootstrap mechanism also ran into possible permissions issues,
if a user inadvertently attempted to bootstrap pip without write access to
the appropriate installation directories.
Including pip directly in the standard library
----------------------------------------------
Similar to this PEP is the proposal of just including pip in the standard
library. This would ensure that Python always includes pip and fixes all of the
end user facing problems with not having pip present by default. This has been
rejected because we've learned, through the inclusion and history of
``distutils`` in the standard library, that losing the ability to update the
packaging tools independently can leave the tooling in a state of constant
limbo. Making it unable to ever reasonably evolve in a time frame that actually
affects users as any new features will not be available to the general
population for *years*.
Allowing the packaging tools to progress separately from the Python release
and adoption schedules allows the improvements to be used by *all* members
of the Python community and not just those able to live on the bleeding edge
of Python releases.
There have also been issues in the past with the "dual maintenance" problem
if a project continues to be maintained externally while *also* having a
fork maintained in the standard library. Since external maintenance of
``pip`` will always be needed to support earlier Python versions, the
proposed bootstrapping mechanism will becoming the explicit responsibility
of the CPython core developers (assisted by the pip developers), while
pip issues reported to the CPython tracker will be migrated to the pip
issue tracker. There will no doubt still be some user confusion over which
tracker to use, but hopefully less than has been seen historically when
including complete public copies of third-party projects in the standard
library.
The approach described in this PEP also avoids some technical issues
related to handling CPython maintenance updates when pip has been
independently updated to a more recent version. The proposed pip-based
bootstrapping mechanism handles that automatically, since pip and the
system installer never get into a fight about who owns the pip
installation (it is always managed through pip, either directly, or
indirectly via the ``ensurepip`` bootstrap module).
Finally, the separate bootstrapping step means it is also easy to avoid
installing ``pip`` at all if end users so desire. This is often the case
if integrators are using system packages to handle installation of
components written in multiple languages using a common set of tools.
Defaulting to --user installation
---------------------------------
Some consideration was given to bootstrapping pip into the per-user
site-packages directory by default. However, this behavior would be
surprising (as it differs from the default behavior of pip itself)
and is also not currently considered reliable (there are some edge cases
which are not handled correctly when pip is installed into the user
site-packages directory rather than the system site-packages).
.. _Wheel: http://www.python.org/dev/peps/pep-0427/
.. _pip: http://www.pip-installer.org
.. _setuptools: https://pypi.python.org/pypi/setuptools
.. _PEP439: http://www.python.org/dev/peps/pep-0439/
References
==========
.. [1] Discussion thread 1 (distutils-sig)
(https://mail.python.org/pipermail/distutils-sig/2013-August/022529.html)
.. [2] Discussion thread 2 (distutils-sig)
(https://mail.python.org/pipermail/distutils-sig/2013-September/022702.html)
.. [3] Discussion thread 3 (python-dev)
(https://mail.python.org/pipermail/python-dev/2013-September/128723.html)
.. [4] Discussion thread 4 (python-dev)
(https://mail.python.org/pipermail/python-dev/2013-September/128780.html)
.. [5] Discussion thread 5 (python-dev)
(https://mail.python.org/pipermail/python-dev/2013-September/128894.html)
.. [#ubuntu] `Ubuntu <http://www.ubuntu.com/>`
.. [#debian] `Debian <http://www.debian.org>`
.. [#fedora] `Fedora <https://fedoraproject.org/>`
.. [#homebrew] `Homebrew <http://brew.sh/>`
.. [#macports] `MacPorts <http://macports.org>`
.. [#fink] `Fink <http://finkproject.org>`
.. [#ContinuumIO] `Anaconda <https://store.continuum.io/cshop/anaconda/>`
.. [#ActiveState] `ActivePython <http://www.activestate.com/activepython>`
.. [#Enthought] `Enthought Canopy <https://www.enthought.com/products/canopy/>`
Copyright
=========
This document has been placed in the public domain.
--
Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia
More information about the Python-Dev
mailing list