I've been aware that the distutils sig has been simmerring away, but
until recently it has not been directly relevant to what I do.
I like the look of the proposed api, but have one question. Will this
support an installed system that has multiple versions of the same
package installed simultaneously? If not, then this would seem to be a
significant limitation, especially when dependencies between packages
Assuming it does, then how will this be achieved? I am presently
managing this with a messy arrangement of symlinks. A package is
installed with its version number in it's name, and a separate
directory is created for an application with links from the
unversioned package name to the versioned one. Then I just set the
pythonpath to this directory.
A sample of what the directory looks like is shown below.
I'm sure there is a better solution that this, and I'm not sure that
this would work under windows anyway (does windows have symlinks?).
So, has this SIG considered such versioning issues yet?
Tim Docker timd(a)macquarie.com.au
Quantative Applications Division
qad16:qad $ ls -l lib/python/
drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1
-r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so
drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1
drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1
dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8
drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1
dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1
dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3
drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7
dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1
qad16:qad $ ls -l lib/python/1.1
lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so
lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so
lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so
lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so
lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so
lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so
lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1
lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8
lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1
lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1
lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so
lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so
lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3
lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7
lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1
I am trying to install distribute-0.6.25 in a windows 7 machine. I have
python 2.7.3 in 32 bits, although the machine is 64 bits. I installed
the 32 bit python version because I want to install ipython and there
is no 64 bit builds of the Windows installer for ipython.
In any case, upon installing disuitils with
python setup.py install
I get the following error
No such file or directory
Any help will be greatly appeciated. Thanks,
Manuel López Mariscal
Depto. de Oceanografía Física/CICESE
In a fit of madness caused by another 30 seconds-long PyPI search I
decided to investigate the code, in the hopes of perhaps finding
something simple that would alleviate the extremely long search time.
I discovered what appears to be a function that makes 6 SQL queries
for each term provided by the search user, in turn those queries
expand to what appear to be %SUBSTRING% table scans across the
releases table, which appears to contain upwards of half a gigabyte of
Now the root cause has been located, what to do about it? I looked at
hacking on the code, but it seems webui.py is already too massive for
its own good, and in any case PostgreSQL's options for efficient
search are quite limited. It might be all-round good if the size of
that module started to drop..
I wrote a crawler to pull a reasonable facsimile of the releases table
on to my machine via the XML-RPC API, then arranged for Xapian to
index only the newest releases for each package. The resulting
full-text index weighs in at a very reasonable 334mb, and searches
complete almost immediately, even on my lowly Intel Atom colocated
I wrote a quick hack Flask app around it, which you can see here:
The indexer takes as input a database produced by the crawler, which
is smart enough to know how to use PyPI's exposed 'changelog' serial
numbers. Basically it is quite trivial and efficient to run this setup
in an incremental indexing mode.
As you can see from the results, even my lowly colo is trouncing what
is currently on PyPI, and so my thoughts tend toward making an
arrangement like this more permanent.
The crawler code weighs in at 150 lines, the indexer a meagre 113
lines, and the Flask example app is 74 lines. Implementing an exact
replica of PyPI's existing scoring function is already partially done
at indexing time, and the rest is quite easy to complete (mostly
Updating the Flask example to provide an XML-RPC API (or similar),
then *initially augmenting* the old search facility seems like a good
start, with a view to removing the old feature entirely. Integrating
indexing directly would be pointless, the PyPI code really doesn't
need anything more added to it until it gets at least reorganized a
So for the cost of 334mb of disk, a cron job, and a lowly VPS with
even just 700MB RAM, PyPI's search pains might be solved permanently.
Naturally I'm writing this mail because it bothers me enough to
volunteer help. :)
Prototype code is here: https://bitbucket.org/dmw/pypi-search (note:
relies on a pre-alpha quality DB library I'm hacking on)
The latest pip supports HTTPS URLs and certificate checks
(according to the change log).
Will there be a release of distribute that implements the
same changes ?
The current 0.6.36 still defaults to the HTTP PyPI address
and doesn't do certificate checks.
Professional Python Services directly from the Source (#1, Apr 25 2013)
>>> Python Projects, Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
2013-04-17: Released eGenix mx Base 3.2.6 ... http://egenix.com/go43
::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
I've released version 0.1.1 of distil, downloadable from . It's based
on distlib 0.1.2. The other changes are as follows:
* Added "distil init" to support creating an initial version of
package.json metadata, which can then be added to during development.
* Added "distil link" to support "editable" installations, similar to
pip install -e local_dir.
* Take into account pre-confirmation (-y) during uninstallation when dists
that are no longer needed are found. These are now removed automatically
when -y is specified.
* Fixed error in setting up SSL certificate verification, and adjusted PyPI
URLs to be https:// where specified as http:// in metadata. Successful
SSL verification is now logged.
* Added --local-dists option to allow local wheels and sdists to be used
* Fixed a bug in the handling of local archives (e.g. those returned
through a configured DirectoryLocator). Local archives shouldn’t be
deleted after unpacking.
* Added --python-tags argument to distil package when building wheels to
configure the tags for the built wheel.
* Added --no-unpack option to distil download.
* Fixed problem with rollback on error caused by not recording SHARED and
* Fixed bug in writing entry points (EXPORTS) file.
* Use of 2to3 now defaults to True when run under 3.x.
* Fixed bug when run in venvs which caused e.g. dist-packages to be used
instead of site-packages.
* Improved error message when run with Python 2.5 (not supported, but this
is now clear from the error message).
Your feedback will be gratefully received!
I've just released version 0.1.2 of distlib on PyPI. The changes are as follows:
Added BaseConfigurator backport for 2.6.
Return RECORD path from write_installed_files (or None if dry_run).
Explicitly return None from write_shared_locations if dry run.
Added missing condition in todict().
Add variants and clobber flag for generation of foo/fooX/foo-X.Y.
Added .exe manifests for Windows.
Regularised recording of written files.
Tidyups, most suggested by Donald Stufft: Made key functions private,
removed _Common class, removed checking for huge version numbers, made
UnsupportedVersionError a ValueError.
Replaced absolute import with relative.
Handle None return from write_shared_locations correctly.
Fixed bug in Mounter for extension modules not in sub-packages.
Made dylib-cache Python version-specific.
Numerous documentation updates, not detailed further here.
Numerous test refinements, not detailed further here.
Corrected setup.py to ensure that sysconfig.cfg is included. The
omission of sysconfig.cfg from the source distribution caused
failures on 2.6.
Please try it out!
New submission from Ian Wienand:
I noticed the following issue installing the keyring package 
Downloading keyring-1.2.2.zip (79Kb): 79Kb downloaded
Running setup.py egg_info for package keyring
zip_safe flag not set; analyzing archive contents...
Traceback (most recent call last):
File "<string>", line 14, in <module>
File "/root/build/keyring/setup.py", line 114, in <module>
File "/usr/lib64/python2.6/distutils/core.py", line 152, in setup
File "/usr/lib64/python2.6/distutils/dist.py", line 975, in run_commands
File "/usr/lib64/python2.6/distutils/dist.py", line 995, in run_command
File "<string>", line 12, in replacement_run
File "/usr/lib/python2.6/site-packages/setuptools/command/egg_info.py", line 254, in find_sources
File "/usr/lib/python2.6/site-packages/setuptools/command/egg_info.py", line 308, in run
File "/usr/lib/python2.6/site-packages/setuptools/command/egg_info.py", line 335, in add_defaults
rcfiles = list(walk_revctrl())
File "/usr/lib/python2.6/site-packages/setuptools/command/sdist.py", line 46, in walk_revctrl
for item in ep.load()(dirname):
File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1948, in load
entry = __import__(self.module_name, globals(),globals(), ['__name__'])
ImportError: No module named hgtools.plugins
Complete output from command python setup.py egg_info:
zip_safe flag not set; analyzing archive contents...
The dependency chain works out to be
keyring -> pytest-runner -> hgtools
hgtools is a "setup_requires" dependency for pytest-runner. This
means the following code is run:
if attrs and 'setup_requires' in attrs:
def fetch_build_eggs(self, requires):
"""Resolve pre-setup requirements"""
from pkg_resources import working_set, parse_requirements
for dist in working_set.resolve(
That goes fine, hgtools gets installed into whatever /tmp directory, is added to sys.path and pytest-runner is happy.
However, the temporary hgtools has registered some entry points with its distribution object in working_set. It then disappears but doesn't remove itself. Later on in sdist.py we have
"""Find all files under revision control"""
for ep in pkg_resources.iter_entry_points('setuptools.file_finders'):
for item in ep.load()(dirname):
pkg_resources.iter_entry_points walks all the distribution objects in working_set, finds hgtools' entry points for this "setuptools.file_finders" stuff, tries to call it and a backtrace ensues because that temporary directory with hgtools has now gone.
This also explains why it works when run the second time around; pytest-runner is already there, so we don't need to look at its setup_requires and the problematic distribution doesn't get added.
Somehow or other, hgtools needs to get removed from the working_set in dist.py before it is removed?
title: setup_requires dependency removal causes installation failure
Setuptools tracker <setuptools(a)bugs.python.org>
I notice that distlib appears to support chaining of comparisons, but
the PEP does not. ISTM that there is no sane use case for chaining,
given the nature of comparisons that are allowed. (That is, if you
perform one equality or contains comparison between two values, then
you know how the next comparison would turn out.)
If chained comparisons are disallowed in the PEP, then distlib
probably shouldn't be allowing them either, to prevent interop issues
w/tools that only support what's in the PEP.
On Thu, Apr 25, 2013 at 4:58 PM, Daniel Holth <dholth(a)gmail.com> wrote:
> I would prefer to see PEP 390 withdrawn and I think this has been
> suggested before. The metadata is already sourced from different files
> depending on your build system.
> The .ini format and our parsers for it are really awful. I always
> resented having to learn it in order to use distutils/setuptools when
> every other language (RFC822, Python, JSON) is both better and already
Everyone here has an excellent user stories. I am stunned that there is not
process to help PEP authors collect them? Original user stories are much
better to learn and draft new standards than when reworked into PEP
> FWIW bdist_wheel does something half-PEP-390 inspired with setup.cfg:
> provides-extra =
> requires-dist =
> distribute (>= 0.6.34)
> argparse; python_version == '2.6'
> keyring; extra == 'signatures'
> dirspec; sys.platform != 'win32' and extra == 'signatures'
> ed25519ll; extra == 'faster-signatures'
> license-file = LICENSE.txt
Why reinvent own format when there is already YAML with indented sections?
Yes, I have to see examples to write in this format, but it is readable and
familiar across a broad range of products. It is better than JSON at least,
because it doesn't require to wrap anything in quotes.