Has anyone bothered to write a meta path hook that checks all the
top_level.txt on a sys.path full of eggs, and uses
imp.load_module(name, short list of paths) to find the module instead
of the default load_module(name, full sys.path)?
I've contacted gocept.com who defered me to zoneedit.com, who I also
sent a mail to. Since there is also an issue with the site and I don't
know who administers it I've come to spam you all with this
Currently the A record for the naked buildout.org domain is missing.
http://buildout.org/bootstrap.py (tested through www.buildout.org) is
also missing. This causes buildout's bootstrap.py to fail checking for
a new version.
Several pages from the buildout.org site are also missing, possibly
due to a bad server migration?
I’m putting up a last-minute proposal for a panel about directions for
the packaging ecosystem at the next PyCon. For that I would need a list
of panelists. I think it would be interesting to have developers (say
from distribute, buildout, pip, wheel) as well as users from
subcommunities (packaging people tend to be web developers, but their
experience doesn’t match the needs of the scipy community for example).
The threads of the panel would be something like:
- working on a better distutils vs. rethinking the whole thing;
- what to put in the stdlib (distlib vs. distutils2);
- decide if we revise the accepted PEPs to address the known problems or
start new ones
- devise a roadmap for new PEPs.
Who wants to volunteer?
I would be a panelist too, so I need a moderator. Nick, would you like
to take that role?
This is a timely discussion for me because I need to get back to packaging
some tools for RHEL/Centos that we have as dependencies. The tools are
available on pypi or from a git repository. They seem to be available in
older versions for Fedora 14 but not in EPEL or other sites for
One issue I encountered when previously working on the packaging for
Fedora was that pypi packages do not necessarily follow the Linux
Filesystem Hierarchy Standard. In fact, the packages may be implicitly
organized for something like Windows where documentation goes in the same
directory as code. I found I needed to use pypi2rpm bdist_rpm2 and modify
the spec file to get things into the right places.
BTW, the tools right now use python 2.7.
So, please add the FHS issue to your list.
I've also found that everything seems to be in a state of flux right now,
with setuptools, distribute, distutils, distutils2, and
pypi2rpm/bdist_rpm2 all out there and bumping into each other. When I
tried to do a yum update to my Fedora 14 system, I ran into trouble
because of conflicts among these packages. I hope this all settles down
On Thu, September 13, 2012 8:00 am, Toshio Kuratomi <a.badger(a)gmail.com>
> Message: 1
> Date: Wed, 12 Sep 2012 15:14:07 -0700
> From: Toshio Kuratomi <a.badger(a)gmail.com>
> To: Fedora Python SIG <python-devel(a)lists.fedoraproject.org>
> Subject: Re: Upstream packaging feedback
> On Wed, Sep 12, 2012 at 01:41:22PM +1000, Nick Coghlan wrote:
>> I'll likely be helping to guide updates to the Python packaging format
>> standards over the coming months. While they won't hit the standard
>> library until 3.4, there will likely be third party tool support in
>> earlier versions (since the whole point of the exercise is to eliminate
>> the current implementation coupling to distutils and setuptools in
>> favour of better defined metadata standards for communication between
>> multiple tools).
>> The first step will be reviewing the status quo and then creating a
>> plausible road map (as well as describing current efforts for various
>> aspects). I've started on that here:
>> One thing I would *love* to be able to enable is adding support for
>> automatic mapping of PyPI distribution names (similar to what already
>> exists for Perl and CPAN) where (for example), a developer could just
>> write "Requires: python(south)" instead of having to figure out manually
>> the name of the appropriate RPM package in Fedora.
>> I believe that the new metadata fields defined in PEP 345 and PEP 426
>> should be enough to support that when generating a SPEC file from the
>> PyPI metadata.
> Hey Nick! Thanks for working on this whole thing. I haven't been looking
> at things in the past year but was active on it a few years ago if you
> to bounce any ideas around, get some idea of what's been discussed i nthe
> past by whom, or talk about what Fedora-specifically is currently
> found troublesome in the past.
> Mapping of pypi distribution names was something that I looked into a
> bit with some Canonical people at PyCon two years ago. IIRC, it was part
> trying to map distribution packages with each other and with pypi in order
> to figure out the state of python3 porting. Barry Warsaw still works
> and might know more about what happened to it -- Allison Randal is who I
> working with but I got the impression that she isn't working on that
> so I don't know if she'll still have code around or not.
> There are numerous caveats to trying to do this, none of them
> insurmountable. I believe we were trying to compare versions as well as
> package names which was even tougher. Some things I can remember off the
> top of my head:
> * Multiple names for a package
> * pypi usually has the same name as the setup.py name field which is
> encoded in the egg file name and metadata
> * The name of the module that is imported
> * One upstream package having multiple downstream names -- for instance,
> Debian has setuptools and pkg_resources as two separate binary packages.
> However, they're provided by the setuptools pypi entry.
> * A module that is provided by multiple upstreams: For instance,
> is provided by setuptools and distribute. pexpect is provided by
> and pexpect-u.
> * Some packages aren't present on pypi. Many library bindings
> provided by the libraries are this way, for instance libselinux-python.
> * Some libraries have conflicting names upstream (mock was one example in
> the past. ming still is one: http://www.libming.org/ and
> * The naming for packages in the distribution isn't always simple. In
> Fedora, for instance, we have several styles and not all of them are
> mutually exclusive:
> * python-foo is the common case for libraries (python-docutils)
> * foo-python are the majority (but not all) of libraries which are
> bindings to C libraries (selinux-python)
> * Fedora allows modules that have py as a prefix to not add python- to
> their name. (pygtk2)
> * Applications may provide libraries that other packages need but since
> they're "primarily applications" they may simply bear the name of the
> application. (bzr)
> * Case sensitivity can get you. There are some maintainers that prefer
> lowercase the package names and others who prefer to do exactly what
> upstream did.
> * dashes, periods, and other punctuation can also get you. Sometimes
> those are translated into dashes, other times they're left alone, and
> other times they're omitted.
> * Even today, not all packages provide egg-info. For instance, a useful
> python module might consist of a single .py file so upstream might not
> provide a setup.py for it. We install it by copying the .py file to
> * We'll need to differentiate between things provided for python2 and
> (and python26 in EPEL)
> All that said.... you can probably sidestep some of these issues by having
> python packages contain explicit virtual Provides. These might be
> added or automatically generated by a tool like pypi2rpm with the
> maintainer editting them afterwards to make sure they didn't hit any of
> above cornercases.
> There's some prior work done by other people:
> * http://www.rpm.org/ticket/154
> * http://lists.fedoraproject.org/pipermail/packaging/2008-June/004715.html
> (Despite my having written that email, the hard parts were all dmalcolm
> I've been chatting to Chris McDonough a bit as well, and one
> potentially useful thing would be to clearly document the
> setuptools/distribute metadata precisely as it is generated today.
> Currently these formats are entirely implicit in the implementation of
> the code that reads and writes them, as far as I can tell anyway. The
> distribute docs seem to do a decent job of explaining setup.py and the
> various setuptools specific arguments, but *not* what the file formats
> will look like inside the metadata directory when installed.
> The main advantages of this would be to make it clear:
> 1. What can setuptools metadata describe, that v1.2 of the official
> metadata standard cannot?
> 2. Does v1.3 allow everything that setuptools can currently describe
> (either directly, or as an extension)?
> 3. Does v1.3 allow some things to be expressed more clearly than they
> can be with setuptools?
The big new feature with Metadata 1.2 and up that has no
representation in setuptools is the environment markers specification,
which for common cases like 'require x when Python version is y'
allows the generated requirements file to be the same no matter which
version of Python was used to produce the dist:
Requires-Dist: argparse; python_version < '2.7'
would be a common example. The only thing I don't like about it is
that I cannot remember whether to use _ or . to separate the words.
Metadata 1.3 restores extras from setuptools by adding 'extra' as a
variable in environment markers.
Nothing prevents an installer from generating requires.txt from
Metadata 1.3, but you can't go the other way due to the environment
entry_points.txt is the plugin mechanism. I am leaving this unchanged
in my implementation; I'm not motivated to put them into PKG-INFO and
there's no benefit. Each section is a kind of entry point, and each
key = module.name:callable
Split on the colon :, import the first part, return the second part.
console_scripts is the most widely used by far, but thousands of
packages provide other kinds of entry points. I wish pypi was smart
enough to index this file.
wininst2wheel = wheel.wininst2wheel:main
egg2wheel = wheel.egg2wheel:main
wheel = wheel.__main__:main
bdist_wheel = wheel.bdist_wheel:bdist_wheel
top_level.txt is just a list of top-level packages defined by the
dist, one per line. Is this still used?
namespace_packages.txt - same format as top_level.txt. Is used.
SOURCES.txt - list of source files, one per line. Not used.
not-zip-safe: present if package should not be zipped up. empty.
dependency_links.txt: url's of the package's dependencies. Speak up if
you use this; it is very surprising, and has a much better replacement
with pip --index options and requirements files.
Provides works the same way in setuptools, it is in PKG-INFO.
I intend to write up the wheel binary package format version 1.0 as a
Python PEP. The draft is at
It is quite similar to the spec as it has existed since July. I think
the best explanation is how it is installed:
Installing a wheel ‘distribution-1.0.py32.none.any.whl’
Check that installer is compatible with Wheel-Version. Warn if
minor version is greater, abort if major version is greater.
If Root-Is-Purelib == ‘true’, unpack archive into purelib
Else unpack archive into platlib (site-packages).
Unpacked archive includes distribution-1.0.dist-info/ and (if
there is data) distribution-1.0.data/
Move each subtree of distribution-1.0.data/ onto its
destination path. Each subdirectory of distribution-1.0.data/ is a key
into sysconfig.get_paths(), such as
If applicable, update scripts starting with “#!python” to
point to the correct interpreter.
Update distribution-1.0.dist.info/RECORD with the installed paths.
Remove empty distribution-1.0.data directory.
I am implementing requirements-in-setup.cfg for wheel because there is
no place to put environment markers in setup.py
The Requires-Dist: lines in PKG-INFO use ; to separate environment
markers from the dep name. Is setup.cfg supposed to use -- ? Why?
requires-dist = keyring; python_version == '2.6'
requires-dist = keyring -- python_version == '2.6'
Do I correctly recall an earlier syntax that used longer section names
as a place to store environment markers?