While playing around with packaging using Distutils2-py3, I created a package
called PyIdGen https://pypi.python.org/pypi/PyIdGen and added README.txt file.
I have seen that some packages have their documentation embedded into the
packages' corresponding package pages @ pypi.python.org.
For example, the 'py-postgresql 1.1.0' module @
'https://pypi.python.org/pypi/py-postgresql/1.1.0' has its documentation on its
A few questions:
1. What format should I use in my README.txt file for my package's content to be
displayed on its package page?
2. Do I have to use a different extension for the README?
Recently, we tested how pip would respond, without and with TUF,
to attacks on PyPI:
TUF now uses the portable PyCrypto cryptography library, though we are
watching cryptography-dev with great interest. In our internal tests,
pip-with-TUF works on Microsoft Windows 7-8 32/64 bit,
Apple OS X (10.7-10.8), and Debian/Ubuntu GNU/Linux 32/64 bit.
We also have integration tests where we show TUF protecting against other
kinds of attacks:
Previously, we demonstrated that we could efficiently secure PyPI with
*** We need your guidance here! ***
Our next step is to integrate TUF with the PyPI server itself to see how
everything would work in production. This would allow us, amongst other
things, to build better package-signing tools for developers, and make
continuous release of packages as smooth as possible.
Before we go any further, though, we would like your thoughts on the
matter. Should we modify the PyPI server ourselves? Or should we
wait for Warehouse instead? We want to work together with the DistUtils
SIG community on all of this, and would appreciate any feedback and
thoughts you have for us. What would you like to see from us?
The TUF Team
I'd like to discuss and possibly implement a feature of PyPI that would
facilitate in quickly discovering which of a given project's
dependencies need to be updated because of new security-related fixes.
A little background is in order.
Zato has currently 80+ buildout dependencies
and I'm betting at least 80 more will be added with time. I'm in a camp
that has absolutely no problems with as many dependencies as it's needed
if it saves time and means less wheels being reinvented.
Once a dependency is in, it's pinned to a concrete version and that
version is updated only in a couple of situations
1. Stability fixes that are to do with the functionality currently in
use by Zato
2. A dependency offers a new functionality I'd like to make use of
3. A version disappears from PyPI
4. A security fix is available
For the last point, it would be really convenient if authors were
offered a 'contains security fixes' kind of checkboxes somewhere in the GUI.
This would be displayed in a couple of places
- On the project's PyPI page, for instance here -
https://pypi.python.org/pypi/redis - there could be a 'This version
contains security fixes' box right below the download button
- Would be added to the Recent Updates feed
- There would be a new feed at /pypi?%3Aaction=security_rss that would
list only these recent uploads that have the flag set
As far as the underlying database goes, this would be a single boolean
column in the 'releases' table.
Such a feature would allow for quickly reacting to any security changes
without chasing dozens of mailing lists, Twitter, RSS, asking authors
to be notified when they change something etc.
Naturally, nothing would force people to actually use it but authors who
treat their own work seriously would hopefully find it an interesting
addition as well.
I'm familiarizing myself with https://bitbucket.org/pypa/pypi right now
but I'd like to ask you if such a feature would be accepted at all if I
implemented it. Also, it's not a priority one so if someone beats me to
it, it's all good with me.
cheers and take care,
ESB, SOA and cloud integrations in Python
I have always assumed that pip install . and python setup.py install . from
a same root directory of a package should produce the same results.
Pip is build tool, isn't it? Expected behaviour in my head is:
- pip make a nice package (even from the actual dir)
- extract it in a build dir according to env settings
- cd to build, setenv
- and then do a python setup.py install
Thus I expected naively that pip install . and python setup.py install
should have the same results.
Where is my mistake ?
I have a reproducible case when the dependency handling differs if you use
pip install -r requirements.txt or when you use python setup.py install
I am trying to sort if the problem is environmental (OS related with bad
env), me being stupid (happens a lot), or software related.
I can reproduce it, however I cannot pinpoint what is the matter.
Is it something obvious I ignore? Is there a way to trouble shoot that
easily? Is there documentation that can help me understand easily the
differences between both?
I also have a very annoying bug that occurs when you install dependencies:
from time to time calling pip install in a virtualenv does not : package
that are installed failed to be seen by the module or by a requirement
needing previously installed pacakage.
My work around is to have a script that install every requirements one by
one sleep a second, sleep 1 second and then install the module.
This happens both on linux and windows.
(looks like https://github.com/pypa/pip/issues/1183)
It looks like a cache needing to be flushed and like a race condition.
In creating a source distribution, I have found a disparity between the
behaviors of distutils and setuptools with respect to package_data. As of
python Issue 2279: http://bugs.python.org/issue2279, entries listed in
package_data are used when building an sdist. I have verified that this
works in a simple example when setup() is imported from distutils.core.
However, if I import setup() from setuptools, it does not pull in the data
as listed in package_data (and presumedly data_files).
I see that the documented behavior for setuptools is to do some searching
to see what files are in source control, but I think that is broken too
(maybe an issue with the new SVN 1.7?), as the files I was trying to
include are under version control.
In any case, this deviation in behavior is very confusing as I am often
lead to believe that setuptools builds upon distutils, so I am often using
the distutils documentation as they are fairly complete. I would suggest
incorporating both distutils' and setuptools' behavior in this regard.
P.S. - on a related note, for a package "foo", foo.egg-info directory is
created with a SOURCES.txt file. I have found that under certain
situations, it seems that a successful install would result in a fully
listed SOURCES.txt, and then subsequent calls to sdist seems to use that
information to build seemingly correct archives. A co-working
double-checking a deployment process I made did an sdist and created a
source distribution without the package_data when he did a fresh checkout,
but whenever I did it from my development branch, the source distribution
worked fine. I haven't figured out exactly how this came about, but it
seems to be tied to the SOURCES.txt file.
In creating the next draft of PEP 453, I noticed an odd quirk of the
proposed "pyvenv --no-download" option: it bootstraps the version of
pip *that was provided with Python*, rather than the version currently
installed in the base Python installation.
That seems incredibly strange to me, and I expect it will confuse
users as well. "I'm using Python 3.4 and have upgraded pip to 1.6, but
my virtual environments are only getting pip 1.5 when I use the
'--no-download' option. HALP!".
Rather than doing any complicated trickery to make it make sense, I'm
considering two comparatively simple changes:
1. Change the proposed "--no-download" option for getpip and pyvenv to
2. Change the getpip API to pass all other options through to pip
install, rather than only defining a supported subset
That way the behaviour with the flag set makes slightly more intuitive
sense (since that flag will be documented specifically as forcing
installation from the getpip module's internal copy of pip)
If users want to use the more advanced features of pip when
bootstrapping their virtual environments (like installing from a
prebuilt wheel rather than the stdlib internal copy or from PyPI),
then the recommended approach will be to use the ``--without-pip``
option to ``pyvenv`` itself, and then run ``getpip`` directly after
activating the virtual environment.
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
Not sure whether this is interesting for anyone, but since I saw some
threads about bootstrapping pip and setuptools, I though I might
throw in a tool which does this.
For a while now, we've been making eGenix PyRun available, a Python
run time that fits into a single file on Unix:
It's a great virtualenv replacement and doesn't rely on the system
provided Python installation.
Now, to make installation of PyRun even easier, we added an install
script called install-pyrun:
This script downloads the correct PyRun for the platform and
then goes on to bootstrap pip and setuptools fully automatically.
It does this by fetching the most recent versions of these tools
straight from PyPI, without relying on Python (or PyRun).
The script is a simple bash script and uses curl or wget for
the fetch operation, so you get certificate checking, HTTPS,
etc. for free.
Professional Python Services directly from the Source (#1, Sep 18 2013)
>>> Python Projects, Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
2013-09-11: Released eGenix PyRun 1.3.0 ... http://egenix.com/go49
2013-09-20: PyCon UK 2013, Coventry, UK ... 2 days to go
2013-09-28: PyDDF Sprint ... 10 days to go
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
Since the bootstrapping PEP isn't an external standards PEP, I've
asked Martin von Löwis to act as BDFL-Delegate and posted the third
draft to python-dev rather than here.
If there's anyone here that would like to comment on the latest
version, but doesn't participate in python-dev, feel free to reply
here and I'll take the feedback into account :)
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
So there've been a number of updates to PEP453, so i'm posting it here again for more discussion:
Viewable online at: http://www.python.org/dev/peps/pep-0453/
This PEP proposes the inclusion of a method for explicitly bootstrapping
`pip`_ as the default package manager for Python. It also proposes that the
distributions of Python available via Python.org will automatically run this
explicit bootstrapping method and a recommendation to third party
redistributors of Python to also provide pip by default (in a way reasonable
for their distributions).
This PEP proposes the inclusion of a ``getpip`` bootstrapping module in
Python 3.4, as well as in the next maintenance releases of Python 3.3 and
This PEP does *not* propose making pip (or any dependencies) part of the
standard library. Instead, pip will be a bundled application provided
along with CPython for the convenience of Python users, but subject to
its own development life cycle and able to be upgraded independently of
the core interpreter and standard library.
Installing a third party package into a freshly installed Python requires
first installing the package manager. This requires users ahead of time to
know what the package manager is, where to get them from, and how to install
them. The effect of this is that these external projects are required to
either blindly assume the user already has the package manager installed,
needs to duplicate the instructions and tell their users how to install
the package manager, or completely forgo the use of dependencies to ease
installation concerns for their users.
All of the available options have their own drawbacks.
If a project simply assumes a user already has the tooling then they get a
confusing error message when the installation command doesn't work. Some
operating may ease this pain by providing a global hook that looks for commands
that don't exist and suggest an OS package they can install to make the command
If a project chooses to duplicate the installation instructions and tell their
users how to install the package manager before telling them how to install
their own project then whenever these instructions need updates they need
updating by every project that has duplicated them. This will inevitably not
happen in every case leaving many different instructions on how to install it
many of them broken or less than optimal. These additional instructions might
also confuse users who try to install the package manager a second time
thinking that it's part of the instructions of installing the project.
The problem of stale instructions can be alleviated by referencing `pip's
own bootstrapping instructions
<http://www.pip-installer.org/en/latest/installing.html>`__, but the user
experience involved still isn't good (especially on Windows, where
downloading and running a Python script with the default OS configuration is
significantly more painful than downloading and running a binary executable
or installer). The situation becomes even more complicated when multiple
Python versions are involved (for example, parallel installations of Python
2 and Python 3).
The projects that have decided to forgo dependencies altogether are forced
to either duplicate the efforts of other projects by inventing their own
solutions to problems or are required to simply include the other projects
in their own source trees. Both of these options present their own problems
either in duplicating maintenance work across the ecosystem or potentially
leaving users vulnerable to security issues because the included code or
duplicated efforts are not automatically updated when upstream releases a new
By providing the package manager by default it will be easier for users trying
to install these third party packages as well as easier for the people
distributing them as they no longer need to pick the lesser evil. This will
become more important in the future as the Wheel_ package format does not have
a built in "installer" in the form of ``setup.py`` so users wishing to install
a Wheel package will need an installer even in the simple case.
Reducing the burden of actually installing a third party package should also
decrease the pressure to add every useful module to the standard library. This
will allow additions to the standard library to focus more on why Python should
have a particular tool out of the box instead of needing to use the difficulty
in installing a package as justification for inclusion.
Providing a standard installation system also helps with bootstrapping
alternate build and installer systems, such as ``setuptools``,
``zc.buildout`` and the ``hashdist``/``conda`` combination that is aimed
specifically at the scientific community. So long as
``pip install <tool>`` works, then a standard Python-specific installer
provides a reasonably secure, cross platform mechanism to get access to
An additional module called ``getpip`` will be added to the standard library
whose purpose is to install pip and any of its dependencies into the
appropriate location (most commonly site-packages). It will expose a single
callable named ``bootstrap()`` as well as offer direct execution via
``python -m getpip``. Options for installing it such as index server,
installation location (``--user``, ``--root``, etc) will also be available
to enable different installation schemes.
It is believed that users will want the most recent versions available to be
installed so that they can take advantage of the new advances in packaging.
Since any particular version of Python has a much longer staying power than
a version of pip in order to satisfy a user's desire to have the most recent
version the bootstrap will contact PyPI, find the latest version, download it,
and then install it. This process is security sensitive, difficult to get
right, and evolves along with the rest of packaging.
Instead of attempting to maintain a "mini pip" for the sole purpose of
installing pip the ``getpip`` module will, as an implementation detail,
include a private copy of pip and its dependencies which will be used to
discover and install pip from PyPI. It is important to stress that this
private copy of pip is *only* an implementation detail and it should *not*
be relied on or assumed to exist.
Not all users will have network access to PyPI whenever they run the
bootstrap. In order to ensure that these users will still be able to
bootstrap pip the bootstrap will fallback to simply installing the included
copy of pip. The pip ``--no-download`` command line option will be supported
to force installation of the bundled version, without even attempting to
This presents a balance between giving users the latest version of pip,
saving them from needing to immediately upgrade pip after bootstrapping it,
and allowing the bootstrap to work offline in situations where users might
already have packages downloaded that they wish to install.
The proposed CLI is based on a subset of the existing ``pip install``
python -m getpip [options]
--no-download Install the bundled version, don't attempt to download
-i, --index-url <url> Base URL of Python Package Index (default https://pypi.python.org/simple/).
--proxy <proxy> Specify a proxy in the form [user:passwd@]proxy.server:port.
--timeout <sec> Set the socket timeout (default 15 seconds).
--cert <path> Path to alternate CA bundle.
-U, --upgrade Upgrade pip and dependencies, even if already installed
--user Install using the user scheme.
--root <dir> Install everything relative to this alternate root directory.
Additional options (such as verbosity and logging options) may also
Automatic installation of setuptools
``pip`` currently depends on ``setuptools`` to handle metadata generation
during the build process, along with some other features. While work is
ongoing to reduce or eliminate this dependency, it is not clear if that
work will be complete for pip 1.5 (which is the version likely to be
bundled with Python 3.4.0).
This PEP proposes that, if pip still requires it, ``setuptools`` will be
bundled along with pip itself, and thus installed when running
``python -m getpip``.
However, this behaviour will be officially declared an implementation
detail. Other projects which explicitly require setuptools should still
provide an appropriate dependency declaration, rather than assuming
``setuptools`` will always be installed alongside ``pip``.
Updating the bundled pip
In order to keep up with evolutions in packaging as well as providing users
who are using the offline installation method with as recent version as
possible the ``getpip`` module should be updated to the latest versions of
everything it bootstraps. After each new pip release, and again during the
preparation for any release of Python, a script, provided as part of this
PEP, should be run to ensure the bundled packages have been updated to the
This means that maintenance releases of the CPython installers will include
an updated version of the ``getpip`` bootstrap module.
Feature Addition in Maintenance Releases
Adding a new module to the standard library in Python 2.7 and 3.3
maintenance releases breaks the usual policy of "no new features in
It is being proposed in this case as the bootstrapping problem greatly
affects the experience of new users, especially on Python 2 where many
Python 3 standard library improvements are available as backports on PyPI,
but are not included in the Python 2 standard library.
By updating Python 2.7, 3.3 and 3.4 to easily bootstrap the PyPI ecosystem,
this should aid the vast majority of Python users, rather than only those
with the freedom to adopt Python 3.4 as soon as it is released.
This is also a matter of starting as we mean to continue: similar to IDLE
(see PEP 434), ``getpip`` will be permanently exempted from the "no new
features in maintenance releases" restriction, as it will include (and
rely on) upgraded versions of ``pip`` even in maintenance releases.
During the installation of Python from Python.org ``python -m getpip`` should
be executed, leaving people using the Windows or OSX installers with a working
copy of pip once the installation has completed. The exact method of this is
left up to the maintainers of the installers, however if the bootstrapping is
optional it should be opt-out rather than opt-in.
The Windows and OSX installers distributed by Python.org will automatically
attempt to run ``python -m getpip`` by default however the ``make install``
and ``make altinstall`` commands of the source distribution will not. Note
that ``getpip`` itself will still be installed normally (as it is a regular
part of the standard library), only the installation of pip and its
dependencies will be skipped.
Keeping the pip bootstrapping as a separate step for make based
installations should minimize the changes CPython redistributors need to
make to their build processes. Avoiding the layer of indirection through
``make`` for the getpip invocation also ensures those installing from a custom
source build can easily force an offline installation of pip, install it
from a private index server, or skip installing pip entirely.
Open Question: Uninstallation
No changes are currently proposed to the uninstallation process. The
bootstrapped pip will be installed the same way as any other pip
installed packages, and will be handled in the same way as any other
post-install additions to the Python environment.
At least on Windows, that means the bootstrapped files will be
left behind after uninstallation, since those files won't be associated
with the Python MSI installer.
Perhaps the installer needs to be updated to clobber everything in
site-packages and the Scripts directory, but I would prefer not to make
this PEP conditional on that change.
Open Question: Script Execution on Windows
While the Windows installer was updated in Python 3.3 to make ``python``
available on the PATH, no such change was made to include the scripts
directory. This PEP proposes that this be changed to also add the scripts
Without this change, the most reliable way to invoke pip on Windows (without
tinkering with paths) is actually be ``py -m pip`` (or ``py -3 -m pip``
if both Python 2 and 3 are installed) rather than simply calling ``pip``.
Adding the scripts directory to the system PATH would mean that ``pip``
works reliably in the "only one Python installation" case, with
``py -m pip`` needed only for the parallel installation case.
Python Virtual Environments
Python 3.3 included a standard library approach to virtual Python environments
through the ``venv`` module. Since it's release it has become clear that very
few users have been willing to use this feature in part due to the lack of
an installer present by default inside of the virtual environment. They have
instead opted to continue using the ``virtualenv`` package which *does* include
pip installed by default.
To make the ``venv`` more useful to users it will be modified to issue the
pip bootstrap by default inside of the new environment while creating it. This
will allow people the same convenience inside of the virtual environment as
this PEP provides outside of it as well as bringing the ``venv`` module closer
to feature parity with the external ``virtualenv`` package making it a more
suitable replacement. To handles cases where a user does not wish to have pip
bootstrapped into their virtual environment a ``--without-pip`` option will be
added. The ``--no-download`` option will also be supported, to force the
use of the bundled ``pip`` rather than retrieving the latest version from
Bundling CA Certificates with CPython
The reference ``getpip`` implementation includes the ``pip`` CA
bundle along with the rest of pip. This means CPython effectively includes
a CA bundle that is used solely for ``getpip``.
This is considered desirable, as it ensures that ``pip`` will behave the
same across all supported versions of Python, even those prior to Python
3.4 that cannot access the system certificate store on Windows.
Recommendations for Downstream Distributors
A common source of Python installations are through downstream distributors
such as the various Linux Distributions [#ubuntu]_ [#debian]_ [#fedora]_, OSX
package managers [#homebrew]_, or python specific tools [#conda]_. In order to
provide a consistent, user friendly experience to all users of Python
regardless of how they attained Python this PEP recommends and asks that
* Ensure that whenever Python is installed pip is also installed.
* This may take the form of separate packages with dependencies on each
other so that installing the Python package installs the pip package
and installing the pip package installs the Python package.
* Do not remove the bundled copy of pip.
* This is required for offline installation of pip into a virtual environment.
* This is similar to the existing ``virtualenv`` package for which many
downstream distributors have already made exception to the common
* This does mean that if ``pip`` needs to be updated due to a security
issue, so does the bundled version in the ``getpip`` bootstrap module
* However, altering the bundled version of pip to remove the embedded
CA certificate bundle and rely the system CA bundle instead is a
* Migrate build systems to utilize `pip`_ and `Wheel`_ instead of directly
* This will ensure that downstream packages can more easily utilize the
new metadata formats which may not have a ``setup.py``.
* Ensure that all features of this PEP continue to work with any modifications
* Online installation of the latest version of pip into a global or virtual
python environment using ``python -m getpip``.
* Offline installation of the bundled version of pip into a global or virtual
python environment using ``python -m getpip``.
* ``pip install --upgrade pip`` in a global installation should not affect
any already created virtual environments.
* ``pip install --upgrade pip`` in a virtual environment should not affect
the global installation.
Policies & Governance
The maintainers of the bundled software and the CPython core team will work
together in order to address the needs of both. The bundled software will still
remain external to CPython and this PEP does not include CPython subsuming the
responsibilities or decisions of the bundled software. This PEP aims to
decrease the burden on end users wanting to use third party packages and the
decisions inside it are pragmatic ones that represent the trust that the
Python community has placed in the authors and maintainers of the bundled
The public API of the ``getpip`` module itself will fall under the typical
backwards compatibility policy of Python for its standard library. The
externally developed software that this PEP bundles does not.
Most importantly, this means that the bundled version of pip may gain new
features in CPython maintenance releases, and pip continues to operate on
its own 6 month release cycle rather than CPython's 18-24 month cycle.
Any security update that affects the ``getpip`` module will be shared prior to
release with the PSRT. The PSRT will then decide if the issue inside warrants
a security release of Python.
Appendix: Rejected Proposals
`PEP439`_, the predecessor for this PEP, proposes its own solution. Its
solution involves shipping a fake ``pip`` command that when executed would
implicitly bootstrap and install pip if it does not already exist. This has
been rejected because it is too "magical". It hides from the end user when
exactly the pip command will be installed or that it is being installed at all.
It also does not provide any recommendations or considerations towards
downstream packagers who wish to manage the globally installed pip through the
mechanisms typical for their system.
Including pip In the Standard Library
Similar to this PEP is the proposal of just including pip in the standard
library. This would ensure that Python always includes pip and fixes all of the
end user facing problems with not having pip present by default. This has been
rejected because we've learned through the inclusion and history of
``distutils`` in the standard library that losing the ability to update the
packaging tools independently can leave the tooling in a state of constant
limbo. Making it unable to ever reasonably evolve in a timeframe that actually
affects users as any new features will not be available to the general
population for *years*.
Allowing the packaging tools to progress separately from the Python release
and adoption schedules allows the improvements to be used by *all* members
of the Python community and not just those able to live on the bleeding edge
of Python releases.
.. _Wheel: http://www.python.org/dev/peps/pep-0427/
.. _pip: http://www.pip-installer.org
.. _setuptools: https://pypi.python.org/pypi/setuptools
.. _PEP439: http://www.python.org/dev/peps/pep-0439/
.. [#ubuntu] `Ubuntu <http://www.ubuntu.com/>`
.. [#debian] `Debian <http://www.debian.org>`
.. [#fedora] `Fedora <https://fedoraproject.org/>`
.. [#homebrew] `Homebrew <http://brew.sh/>`
.. [#conda] `Conda <http://www.continuum.io/blog/conda>`
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA