I just joined up after the various discussions at PyCon and wanted to say hi. (If you were also there and want to put a face/voice to the name, I did the Visual Studio demo at one of the lightning talks.)
The main reason I want to get involved is the openly acknowledged lack of Windows expertise that's available. I work at Microsoft and part of my job description is to contribute code/testing/time/documentation/help/etc. to CPython. (I can also do testing/time/help for other projects, but copyrightable artifacts are more complicated and, for now, not okay with our lawyers.)
I expect I'll mainly be lurking until I can be useful, which is why I wanted to start with this post. I'm pretty good with Windows, and I have direct access to all the experts and internal mailing lists. So just shout out when something comes up and I'll be happy to clarify or research an answer.
If I run (on an Ubuntu machine) the command:
sudo pip install --upgrade distribute
Everything goes well and the package installs.
But if I run the same with Python3 with:
sudo pip-3.2 install --upgrade distribute
I get this:
"File "setuptools/dist.py", line 103
except ValueError, e:
SyntaxError: invalid syntax"
I think the problem is that with someone recently forgot to put the
parentheses (i.e. "except ValueError, e:" should be "except (ValueError,
e):") and therefore this does not work anymore with python3.
It should be easy to fix. Furthermore, this problem seems to be widespread
within the package.
pkg_resources.requires() is our only current solution for parallel
installation of incompatible versions. This can be made to work and is
a lot better than the nothing we had before it was created, but also
has quite a few issues (and it can be a nightmare to debug when it
Based on the exchanges with Mark McLoughlin the other week, and
chatting to Matthias Klose here at the PyCon US sprints, I think I
have a design that will let us support parallel installs in a way that
builds on existing standards, while behaving more consistently in edge
cases and without making sys.path ridiculously long even in systems
with large numbers of potentially incompatible dependencies.
The core of this proposal is to create an updated version of the
installation database format that defines semantics for *.pth files
inside .dist-info directories.
Specifically, whereas *.pth files directly in site-packages are
processed automatically when Python starts up, those inside dist-info
directories would be processed only when explicitly requested
(probably through a new distlib API). The processing of the *.pth file
would insert it into the path immediately before the path entry
containing the .dist-info directory (this is to avoid an issue with
the pkg_resources insert-at-the-front-of-sys.path behaviour where
system packages can end up shadowing those from a local source
checkout, without running into the issue with
append-to-the-end-of-sys.path where a specifically requested version
is shadowed by a globally installed version)
To use CherryPy2 and CherryPy3 on Fedora as an example, what this
would allow is for CherryPy3 to be installed normally (i.e. directly
in site-packages), while CherryPy2 would be installed as a split
install, with the .dist-info going into site-packages and the actual
package going somewhere else (more on that below). A cherrypy2.pth
file inside the dist-info directory would reference the external
location where cherrypy 2.x can be found.
To use this at runtime, you would do something like:
The other part of this question is how to avoid the potential
explosion of one sys.path entry per dependency. The first part of that
is that for cases where there is no incompatible version installed,
there won't be a *.pth file, and hence no extra sys.path entry (the
module/package will just be installed directly into site-packages as
The second part has to do with a possible way to organise the
versioned installs: group them by the initial fragment of the version
number according to semantic versioning. For example, define a
"versioned-packages" directory that sits adjacent to "site-packages".
When doing the parallel install of CherryPy2 the actual *code* would
be installed into "versioned-packages/2/", with the cherrypy2.pth file
pointing to that directory. For 0.x releases, there would be a
directory per minor version, while for higher releases, there would
only be a directory per major version.
The nice thing though is that Python wouldn't actually care about the
actual layout of the installed versions, so long as the *.pth files in
the dist-info directories described the mapping correctly.
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
Jason Coombs (head of the Distribute project) and I are working on
merging the bulk of the improvements distribute made into the
setuptools code base. He has volunteered to take over maintenance of
setuptools, and I welcome his assistance. I appreciate the
contributions made by the distribute maintainers over the years, and
am glad to have Jason's help in getting those contributions into
setuptools as well. Continuing to keep the code bases separate isn't
helping anybody, and as setuptools moves once again into active
development to deal with the upcoming shifts in the Python-wide
packaging infrastructure (the new PEPs, formats, SSL, TUF, etc.), it
makes sense to combine efforts.
Aside from the problems experienced by people with one package that
are fixed in the other, the biggest difficulties with the fork right
now are faced by the maintainers of setuptools-driven projects like
pip, virtualenv, and buildout, who have to either take sides in a
conflict, or spend additional time and effort testing and integrating
with both setuptools and distribute. We'd like to end that pain and
simplify matters for end users by bringing distribute enhancements to
setuptools and phasing out the distribute fork as soon as is
In the short term, our goal is to consolidate the projects to prevent
duplication, wasted effort, and incompatibility, so that we can start
moving forward. This merge will allow us to combine resources and
teams, so that we may focus on a stable but actively-maintained
toolset. In the longer term, the goal is for setuptools as a concept
to become obsolete. For the first time, the Python packaging world
has gotten to a point where there are PEPs *and implementations* for
key parts of the packaging infrastructure that offer the potential to
get rid of setuptools entirely. (Vinay Sajip's work on distlib,
Daniel Holth's work on the "wheel" format, and Nick Coghlan's taking
up the reins of the packaging PEPs and providing a clear vision for a
new way of doing things -- these are just a few of the developments in
"Obsolete", however, doesn't mean unmaintained or undeveloped. In
fact, for the "new way of doing things" to succeed, setuptools will
need a lot of new features -- some small, some large -- to provide a
At the moment, the merge is not yet complete. We are working on a
common repository where the two projects' history has been spliced
together, and are cleaning up the branch heads to facilitate
re-merging them. We'd hoped to have this done by PyCon, but there
have been a host of personal, health, and community issues consuming
much of our available work time. But we decided to go ahead and make
an announcement *now*, because with the big shifts taking place in the
packaging world, there are people who need to know about the upcoming
merge in order to make the best decisions about their own projects
(e.g. pip, buildout, etc.) and to better support their own users.
Thank you once again to all the distribute contributors, for the many
fine improvements you've made to the setuptools package over the
years, and I hope that you'll continue to make them in the future.
(Especially as I begin to phase myself out of an active role in the
I now want to turn the floor over to Jason, who's put together a
Roadmap/FAQ for what's going to be happening with the project going
forward. We'll then both be here in the thread to address any
questions or concerns you might have.
When building wheels, it is necessary to know details of the
compatibility requirements of the code. The most common case is for
pure Python code, where the code could in theory be valid for a single
Python version, but in reality is more likely to be valid either for
all Pythons, or sometimes for just Python 2 or Python 3 (where
separate code bases or 2to3 are involved). The wheel project supports
a "universal" flag in setup.cfg, which sets the compatibility flags to
'py2.py3', but that is only one case.
Ultimately, we need a means (probably in metadata) for (pure Python)
projects to specify any of the following:
1. The built code works on any version of Python (that the project supports)
2. The built code is specific to the major version of Python that it
was built with
3. The built code is only usable for the precise Python version it was
The default is currently (3), but this is arguably the least common
case. Nearly all code will support at least (2) and more and more is
Note that this is separate from the question of what versions the
project supports. It's about how the code is written. Specifically,
there's no point in marking code that uses new features in Python 3.3
as .py33 - it's still .py3 as it will work with Python 3.4. The fact
that it won't work on Python 3.2 is just because the project doesn't
support Python 3.2. Installing a .py3 wheel into Python 3.2 is no
different from installing a sdist there. So overspecifying the wheel
compatibility so that a sdist gets picked up for earlier versions
In addition to a means for projects to specify this themselves, tools
(bdist_wheel, pip wheel) should probably have a means to override the
default at the command line, as it will be some time before projects
specify this information, even once it is standard. There's always the
option to rename the generated file, but that feels like a hack...
Where C extensions are involved, there are other questions. Mostly,
compiled code is implementation, architecture, and minor version
specific, so there's little to do here. The stable ABI is relevant,
but I have no real experience of using it to know how that would work.
There is also the case of projects with C accelerators - it would be
good to be able to easily build both the accelerated version and a
fallback pure-python wheel. I don't believe this is easy as things
stand - distutils uses a compiler if it's present, so forcing a
pure-python build when you have a compiler is harder work than it
needs to be when building binary distributions.
Comments? Should the default in bdist_wheel and pip wheel be changed
or should it remain "as safe as possible" (practicality vs purity)? If
the latter, should override flags be added, or is renaming the wheel
in the absence of project metadata the recommended approach? And does
anyone have any experience of how this might all work with C
Earlier today we merged the existing wheel branch into mainline pip.
This adds opt-in wheel install support (built into pip, "pip install
--use-wheel ...") and the convenient "pip wheel ..." command for
creating the wheels you need.
"pip wheel ..." uses the wheel reference implementation ("pip install
wheel") to compile a dependency tree as .whl archives. Used together
with "pip install --use-wheel ..." it provides a powerful way to speed
up repeated installs and reap other good packaging benefits. I've been
using this code in production for months and it works well.
I am now a pip maintainer. We are committed to offering excellent
wheel support in pip including a good way to produce and consume the
format. In the future we will likely refactor the code to offer the
same features with more distlib and less setuptools but this change
will be mostly transparent to the end user. Once everyone is
comfortable with the format we will move towards installing wheels by
default when they are available instead of requiring the --use-wheel
Enjoy! Please share your experiences with the new feature. The most
common issue is that you must install distribute >= 0.6.34 for
everything to work well.
I present for your deliberation a draft PEP for the inclusion of a pip
bootstrap program in Python 3.4. Discussion of this PEP should remain
here on the distutils SIG.
The PEP is revision controlled in my bitbucket account
https://bitbucket.org/r1chardj0n3s/pypi-pep (this is also where I'm
intending to develop the implementation.)
Title: Inclusion of pip bootstrap in Python installation
Author: Richard Jones <richard(a)python.org>
BDFL-Delegate: Nick Coghlan <ncoghlan(a)gmail.com>
Type: Standards Track
This PEP proposes the inclusion of a pip boostrap executable in the Python
installation to simplify the use of 3rd-party modules by Python users.
This PEP does not propose to include the pip implementation in the Python
standard library. Nor does it propose to implement any package management or
installation mechanisms beyond those provided by PEPs 470 ("The Wheel Binary
Package Format 1.0") and TODO distlib PEP.
Currently the user story for installing 3rd-party Python modules is
not as simple as it could be. It requires that all 3rd-party modules inform
the user of how to install the installer, typically via a link to the
installer. That link may be out of date or the steps required to perform the
install of the installer may be enough of a roadblock to prevent the user from
Large Python projects which emphasise a low barrier to entry have shied away
from depending on third party packages because of the introduction of this
potential stumbling block for new users.
With the inclusion of the package installer command in the standard Python
installation the barrier to installing additional software is considerably
reduced. It is hoped that this will therefore increase the likelihood that
Python projects will reuse third party software.
It is also hoped that this is reduces the number of proposals to include
more and more software in the Python standard library, and therefore that
more popular Python software is more easily upgradeable beyond requiring
Python installation upgrades.
Python install includes an executable called "pip" that attempts to import pip
machinery. If it can then the pip command proceeds as normal. If it cannot it
will bootstrap pip by downloading the pip implementation wheel file.
Once installed, the pip command proceeds as normal.
A boostrap is used in the place of a the full pip code so that we don't have
to bundle pip and also the install tool is upgradeable outside of the regular
Python upgrade timeframe and processes.
To avoid issues with sudo we will have the bootstrap default to installing the
pip implementation to the per-user site-packages directory defined in PEP 370
and implemented in Python 2.6/3.0. Since we avoid installing to the system
Python we also avoid conflicting with any other packaging system (on Linux
systems, for example.) If the user is inside a virtual environment (TODO PEP
ref) then the pip implementation will be installed into that virtual
The bootstrapping process will proceed as follows:
1. The user system has Python (3.4+) installed. In the "scripts" directory of
the Python installation there is the bootstrap script called "pip".
2. The user will invoke a pip command, typically "pip install <package>", for
example "pip install Django".
3. The boostrap script will attempt to import the pip implementation. If this
succeeds, the pip command is processed normally.
4. On failing to import the pip implementation the bootstrap notifies the user
that it is "upgrading pip" and contacts PyPI to obtain the latest download
wheel file (see PEP 427.)
5. Upon downloading the file it is installed using the distlib installation
machinery for wheel packages. Upon completing the installation the user
is notified that "pip has been upgraded." TODO how is it verified?
6. The pip tool may now import the pip implementation and continues to process
the requested user command normally.
Users may be running in an environment which cannot access the public Internet
and are relying solely on a local package repository. They would use the "-i"
(Base URL of Python Package Index) argument to the "pip install" command. This
use case will be handled by:
1. Recognising the command-line arguments that specify alternative or additional
locations to discover packages and attempting to download the package
from those locations.
2. If the package is not found there then we attempt to donwload it using
the standard "https://pypi.python.org/pypi/simple/pip" index.
3. If that also fails, for any reason, we indicate to the user the operation
we were attempting, the reason for failure (if we know it) and display
further instructions for downloading and installing the file manually.
Manual installation of the pip implementation will be supported through the
manual download of the wheel file and "pip install <downloaded wheel file>".
This installation will not perform standard pip installation steps of saving the
file to a cache directory or updating any local database of installed files.
The download of the pip implementation install file should be performed
securely. The transport from pypi.python.org will be done over HTTPS but the CA
certificate check will most likely not be performed. Therefore we will
utilise the embedded signature support in the wheel format to validate the
Beyond those arguments controlling index location and download options, the
"pip" boostrap command may support further standard pip options for verbosity,
quietness and logging.
The "--no-install" option to the "pip" command will not affect the bootstrapping
An additional new Python package will be proposed, "pypublish", which will be a
tool for publishing packages to PyPI. It would replace the current "python
setup.py register" and "python setup.py upload" distutils commands. Again
because of the measured Python release cycle and extensive existing Python
installations these commands are difficult to bugfix and extend. Additionally
it is desired that the "register" and "upload" commands be able to be performed
over HTTPS with certificate validation. Since shipping CA certificate keychains
with Python is not really feasible (updating the keychain is quite difficult to
manage) it is desirable that those commands, and the accompanying keychain, be
made installable and upgradeable outside of Python itself.
The Fedora variant of Linux has had a separate program called "pip" (a Perl
package installer) available for install for some time. The current Python "pip"
program is installed as "pip-python". It is hoped that the Fedora community will
resolve this issue by renaming the Perl installer.
Currently pip depends upon setuptools functionality. It is intended that before
Python 3.4 is shipped that the required functionlity will be present in Python's
standard library as the distlib module, and that pip would be modified to use
that functionality when present. TODO PEP reference for distlib
None, so far, beyond the PEPs.
Nick Coghlan for his thoughts on the proposal and dealing with the Red Hat
Jannis Leidel and Carl Meyer for their thoughts.
This document has been placed in the public domain.
As PJE mentioned in his e-mail, he and I have been working on a merge of the
code lines of Setuptools and Distribute. I'm excited about this transition
and I hope you are too.
In this message, I will provide some answers based on questions that he and
I encountered in our discussions and subsequent merge activity. If you have
further questions, please direct them to both of us and we intend to answer
promptly and also update the FAQ at the wiki
- Jason R. Coombs
Where does the merge occur?
The merge is occurring between the heads of the default branch of Distribute
and the setuptools-0.6 branch of Setuptools. The Setuptools SVN repo has
been converted to a Mercurial repo hosted on Bitbucket. The work is still
underway, so the exact changesets included may change, although the
anticipated merge targets are Setuptools at 0.6c12 and Distribute at 0.6.35.
What happens to other branches?
Distribute 0.7 was abandoned long ago and won't be included in the resulting
code tree, but may be retained for posterity in the original repo.
Setuptools default branch (also 0.7 development) may also be abandoned or
may be incorporated into the new merged line if desirable (and as resources
What history is lost/changed?
As setuptools was not on Mercurial when the fork occurred and as Distribute
did not include the full setuptools history (prior to the creation of the
setuptools-0.6 branch), the two source trees were not compatible. In order
to most effectively communicate the code history, the Distribute code was
grafted onto the (originally private) setuptools Mercurial repo. Although
this grafting maintained the full code history with names, dates, and
changes, it did lose the original hashes of those changes. Therefore,
references to changes by hash (including tags) are lost.
Additionally, any heads that were not actively merged into the Distribute
0.6.35 release were also omitted. As a result, the changesets included in
the merge repo are those from the original setuptools repo and all
changesets ancestral to the Distribute 0.6.35 release.
What features will be in the merged code base?
In general, all "features" added in distribute will be included in
setuptools. Where there exist conflicts or undesirable features, we will be
explicit about what these limitations are. Changes that are
backward-incompatible from setuptools 0.6 to distribute will likely be
removed, and these also will be well documented.
Bootstrapping scripts (ez_setup/distribute_setup) and docs, as with
distribute, will be maintained in the repository and built as part of the
release process. Documentation and bootstrapping scripts will be hosted at
python.org, as they are with distribute now. Documentation at telecommunity
will be updated to refer or redirect to the new, merged docs.
On the whole, the merged setuptools should be largely compatible with the
latest releases of both setuptools and distribute and will be an easy
transition for users of either library.
Who is invited to contribute? Who is excluded?
While we've worked privately to initiate this merge due to the potential
sensitivity of the topic, no one is excluded from this effort. We invite
all members of the community, especially those most familiar with Python
packaging and its challenges to join us in the effort.
We have lots of ideas for how we'd like to improve the codebase, release
process, everything. Like distribute, the post-merge setuptools will have
its source hosted on bitbucket. (So if you're currently a distribute
contributor, about the only thing that's going to change is the URL of the
repository you follow.) Also like distribute, it'll support Python 3, and
hopefully we'll soon merge Vinay Sajip's patches to make it run on Python 3
without needing 2to3 to be run on the code first.
Why Setuptools and not Distribute or another name?
We do understand that this announcement might be unsettling for some. The
setuptools name has been subjected to a lot of deprecation in recent years,
so the idea that it will now be the preferred name instead of distribute
might be somewhat difficult or disorienting for some. We considered use of
another name (Distribute or an entirely new name), but that would serve to
only complicate matters further. Instead, our goal is to simplify the
packaging landscape but without losing any hard-won advancements. We hope
that the people who worked to spread the first message will be equally
enthusiastic about spreading the new one, and we especially look forward to
seeing the new posters and slogans celebrating the new setuptools.
What is the timeframe of release?
There are no hard timeframes for any of this effort, although progress is
underway and a draft merge is underway and being tested privately. As an
unfunded volunteer effort, our time to put in on it is limited, and we've
both had some recent health and other challenges that have made working on
this difficult, which in part explains why we haven't met our original
deadline of a completed merge before PyCon.
What version number can I expect for the new release?
The new release will roughly follow the previous trend for setuptools and
release the new release as 0.7. This number is somewhat arbitrary, but we
wanted something other than 0.6 to distinguish it from its ancestor forks
but not 1.0 to avoid putting too much emphasis on the release itself and to
focus on merging the functionality. In the future, the project will likely
adopt a versioning scheme similar to semver to convey semantic meaning about
the release in the version number.
I've been lurking on the list for a while, but have been pretty quiet so
I was watching the live stream of the PyCon packaging panel today and the
pytjon-meta-packaging resource hub idea was mentioned, which I hadn't heard
of before, and was spurred to action.
So I created a skeleton Sphinx project in a fork of this project here:
Sadly, bitbucket won't let me do a pull request for some reason. If anyone
knows why that is, feel free to let me know.
(FYI I think it's because the main repo has no commits in it, which makes
the "Create a pull request not load properly.)
At any rate, somebody somewhere should feel free pull it into the main
repo, so we can get things moving on that front. I'm happy to help out
where I can.
As folks may be aware, I am moderating a panel called "Directions in
Packaging" on the Saturday afternoon at PyCon US.
Before that though, I am also organising what I am calling a
"Packaging & Distribution Mini-Summit" as an open space on the Friday
night (we have one of the larger open space rooms reserved, so we
should have a fair bit of space if a decent crowd turns up).
An overview of what I'm hoping we can achieve at the session is at
(that page should be editable by anyone that has registered for PyCon
We're certainly not going to resolve all our disagreements and come up
with a grand plan to "fix Python packaging" in a couple of hours on a
Friday night. My hopes are far more modest: that we can get an idea of
the different things people are worried about and trying to resolve,
and start pondering ways we can work towards an cooperative ecosystem
of interoperable tools, rather than the "one tool to rule them all"
model of distutils.
I fully expect that discussions on the Friday night will continue as
hallway track discussions during the conference, development efforts
at the sprints, and, of course, requests for feedback on the mailing
lists (since there will likely be quite a few interested people that
won't be present at PyCon US). This is not a new conversation - it is
one that has been going on for years, and while some improvements have
been made, we still have a long way to go.
For those that are able to make it, I look forward to meeting you in
person in March :)
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia