I've just checked in support for revision control plugins in
setuptools. This means that folks using any of the various revision
control systems out there other than CVS and SVN can now write a plugin for
setuptools to support automatically finding data and source files when
building packages and their source distributions.
Please see setuptools.txt in the source for documentation, and let me know
if you have any questions or problems.
I am working on getting setup tools into fink. If anyone has time, please
take a look at the setup file and let me know what you think.
Once setuptools is installed, I am having trouble with SQLAlchemy which I am
using as my test case to see if setuptools is installed correctly. Clearly
I am doing something wrong. Can someone please set me straight? I am
working with the 10.4-transitional tree within fink.
This makes me think setup tools is ok since it is in the default
Python 2.4.2 (#1, Feb 24 2006, 17:05:21)
[GCC 4.0.1 (Apple Computer, Inc. build 5247)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from ez_setup import use_setuptools
But then the SQLAlchemy build tries to pull down setuptools, which I don't
want it to do:
CC=gcc-3.3 /sw/bin/python2.4 setup.py build
This script requires setuptools version 0.6a5 to run (even to display
help). I will attempt to download it for you (from
you may need to enable firewall access for this script first.
I will start the download in 15 seconds.
At 09:23 AM 3/24/2006 -0500, Mars wrote:
>The new egg file definiton is used to open the egg in the install directory.
>However, the egg that the install script finds there is a copy of the
>orginal temporary egg file, and the old file offsets do not match the new
>definition. Thus a ZipImportError is raised.
It sounds like easy_install needs to clear the zipimport directory cache,
or at least remove any entry corresponding to the file it's installing, in
the case where it's overwriting an existing file.
As for the SOURCES.txt problem, it seems to be a quirk of the way the
distutils FileList class works, such that if a file doesn't exist before
SOURCES.txt is built, it doesn't end up listed in SOURCES.txt. So, I'll
have to add a workaround for that as well.
Thanks for taking the time to follow up on this.
At 05:32 PM 3/25/2006 -0800, Grig Gheorghiu wrote:
>1. Is there any chance you can release a new setuptools version that
>includes the fix for the fetch breakage?
I expect to release that version within a week. Nothing stops you,
however, from using intermediate snapshots. Run this command to create an
egg in the current directory that's a snapshot of the current SVN version:
easy_install -zmaxd. setuptools==dev
You can then distribute this egg with your package if needed, by passing
the correct version info to the ez_setup() function in ez_setup.py (so it
uses your copy of the snapshot.
>2. Another thing that seems to be broken, and that I was depending on
>in cheesecake, is installing a package to a non-default location via
>the --home option.
>I'm not sure when this started to break in terms of setuptools, but it
>used to work at least a couple of months ago. Now I get something
>similar to this when I try to run setuptools-based setup.py scripts
>[ggheo@concord twill-0.8.4]$ python setup.py install
>Checking .pth file support in /tmp/install_twill//lib/python/
>error: can't create or remove files in install directory
>The following error occurred while trying to add or remove files in the
> [Errno 2] No such file or directory:
>The installation directory you specified (via --install-dir, --prefix,
>the distutils default setting) was:
>This directory does not currently exist. Please create it and try
>choose a different installation directory (using the -d or
>Is there a better/less-prone-to-breakage way to test whether a package
>can be installed to an alternate location, other than using --home? I
>want this to work with distutils-based versions of setup.py too.
If you use --root instead of --home, it will work nicely with both,
assuming you use the current SVN version of setuptools. When setuptools'
"install" command sees the --root option, it doesn't try to verify that the
installation location is valid. This should be ideal for your purposes,
although you may have to fiddle with some of the other options a
little. --root also tells setuptools to use its "legacy installation"
format, which will more closely mimic what the distutils would do in the
> From: pje(a)telecommunity.com (Phillip J. Eby)
> Date: 3/17/2006 10:07 PM (Eastern Standard Time)
> To: mfogels(a)interactdirect.com (Maris Fogels)
> Cc: distutils-sig(a)python.org
> Subject: Re: easy_install: bugs with pathing and multi-version
> At 10:11 PM 3/16/2006 +0000, Maris Fogels wrote:
> >zipimport.ZipImportError: bad local file header in
> >If I re-run the 'easy_install -m .' command it works properly.
> >Everything in the installed .egg file appears to work correctly, so I
> >could not figure out why a ZipImportError was being raised by a
> >working .egg.
> Me either. The only time I've seen anything like that happen is with
> Python 2.3 on a 64-bit platform. If you can figure out how to reproduce
> it, please let me know.
Thankfully the problem is trivial for me to reproduce, and I was able
to run the script through pdb. I believe I have isolated the cause of
the problem, but I have not had the time to devise a solution.
The problem starts with a normal installation, in a clean directory,
using the command 'easy_install .'. There is a custom setup.cfg file
there that specifies the install-directory. During installation a
temporary egg file is created, along with the projectname.egg-info
directory. The temporary egg is copied into the installation
directory and renamed.
I noted that the newly created SOURCES.txt manifest on my system
does not contain an entry for itself. But the [incorrect] manifest file is
properly wrapped into the egg.
To uninstall, we run the command 'easy_install -m .'. This creates a
new egg file, along with a new manifest. This new manifest does
contain a reference to itself. This changes the egg file size and the
contained zip-file offsets. At this point we have two different egg files
and two different egg file definitions, with different zip-file offsets.
The new egg file definiton is used to open the egg in the install directory.
However, the egg that the install script finds there is a copy of the
orginal temporary egg file, and the old file offsets do not match the new
definition. Thus a ZipImportError is raised.
I was under the impression that a fix to the class
setuptools.command.egg_info.manifest_maker was in order, but the code
already ensures that the manifest contains the needed self-reference.
A run through the debugger should either verify this or pick up the
At 05:26 PM 3/20/2006 -0500, Kevin Dangoor wrote:
>On 3/20/06, Phillip J. Eby <pje(a)telecommunity.com> wrote:
> > An entry point won't really help you much here, because it would have to
> > know how to perform this notification. Presumably, it would either have to
> > talk to a socket or modify a file somewhere.
>The egg that has that entry point would have to know how to find its
>configuration file to know what to notify, yes.
configuration file*s* - you might have multiple copies of something. And
Python packages don't really (i.e. shouldn't) own configuration files of
> > However, if you're going to have to monitor for file changes anyway, then
> > you might as well just monitor sys.path directories for new .egg files, or
> > monitor changes to easy-install.pth files.
>My thought was that no file monitoring would be required. easy_install
>would keep track of the eggs it installed and then pass that list
>along to everything that defines that entry point.
Yeah, except that your approach effectively requires building an egg for
every *instance* of every app on the box, and also requires you to have a
privileged process running an individual user's code.
I know that you probably have a rather more restricted deployment scenario
in mind, I'm just pointing out that the approach you were proposing falls
apart as soon as multiple instances or users with different privileges are
involved. (Heck, even my SIGHUP approach has some issues with that, since
a high-privilege installer could then be tricked into signalling some other
> > The only other thing I can think of that would work is if you had server
> > processes write their process IDs some place, and then the hypothetical
> > entry point could send them SIGHUPs or something. But monitoring
> > file/directory changes is a bit more cross-platform, doesn't require adding
> > any entry point code, is probably part of existing reload code, and doesn't
> > require a pidfile arena, with all the associated permissions and cleanup
> > headaches thereof.
>The entry point is probably a premature optimization. File
>modification time lookups are pretty cheap. And, if it's only watching
>easy-install.pth, that would be dirt cheap.
Yeah, and this latter exchange has convinced me that it's the only simple
way to avoid privilege escalation attacks. That is, it doesn't require
low/medium-privilege server processes to be able to write stuff (like entry
point code, pids, or configuration files) that gets read or executed (or
otherwise relied upon) by the high-privilege installation process.
Seems it would be nice to have the easy_install bootstrap installer create
easy_installX.Y links for each version X.Y of Python. As it is, you have
to fiddle with shell backticks, or type the full path to easy_install if
At 04:02 PM 3/20/2006 -0500, Kevin Dangoor wrote:
>Someone just prodded me about automatic application reloading, which
>is always a hairy topic. The autoreload code that many people run in
>development mode forks to run the actual application in a child
>process, and everything in sys.modules is watched for a change. If one
>of the files change, the child exits and the parent process fires off
>a new one. It's ugly, but it's reliable and is generally fine for
>For TurboGears apps, I'm of the opinion that the right way to deploy
>an app to production is to turn it into an egg and install it. I had
>the (not entirely fleshed out) thought that it might be interesting if
>there was an entry_point that got called when new eggs were installed.
>This would make it possible to notify a running server that it needs
>to be restarted.
An entry point won't really help you much here, because it would have to
know how to perform this notification. Presumably, it would either have to
talk to a socket or modify a file somewhere.
However, if you're going to have to monitor for file changes anyway, then
you might as well just monitor sys.path directories for new .egg files, or
monitor changes to easy-install.pth files.
The only other thing I can think of that would work is if you had server
processes write their process IDs some place, and then the hypothetical
entry point could send them SIGHUPs or something. But monitoring
file/directory changes is a bit more cross-platform, doesn't require adding
any entry point code, is probably part of existing reload code, and doesn't
require a pidfile arena, with all the associated permissions and cleanup
Someone just prodded me about automatic application reloading, which
is always a hairy topic. The autoreload code that many people run in
development mode forks to run the actual application in a child
process, and everything in sys.modules is watched for a change. If one
of the files change, the child exits and the parent process fires off
a new one. It's ugly, but it's reliable and is generally fine for
For TurboGears apps, I'm of the opinion that the right way to deploy
an app to production is to turn it into an egg and install it. I had
the (not entirely fleshed out) thought that it might be interesting if
there was an entry_point that got called when new eggs were installed.
This would make it possible to notify a running server that it needs
to be restarted.
Author of the Zesty News RSS newsreader
At 10:11 PM 3/16/2006 +0000, Maris Fogels wrote:
>All of our applications are installed in a flat structure; an
>application directory is one big file-soup. The applications are so
>large that we do not have the budget to correct them, so I must
>emulate this flat structure in the installation target directory. The
>emulations requires the use of a setup.cfg and the distutils '$base'
># example setup.cfg
Note that easy_install only has the equivalent of an install_lib and
install_scripts; it doesn't install headers or any data other than package
data embedded in eggs.
>1. I unpacked the project sources in ~/src/myproject-1.0. When I tried to
>use easy_install from the ~/src directory it ignores the install-base
>setting specified in setup.cfg:
By design, easy_install only loads its own settings from ./setup.cfg. Any
setup scripts it runs will read their own setup.cfg files, but since
easy_install only runs "setup.py bdist_egg" on its targets,
installation-related settings have no effect.
This is intentional, since it's not good practice for a distributed
setup.cfg to customize the install location. If easy_install honored those
settings in the general case, it could result in quite a bit of chaos.
>The command works if I cd into the project directory itself, or if I
>specify the install directory with '-d'
Yes, those are the two ways you can do that.
>2. After successfully installing I attempted to uninstall the project
>as per the instructions on the EasyInstall wiki page, by changing the
>installed files to use a multi-install. It produces the following error:
>$ cd ~/src/myproject-1.0
>$ easy_install -m .
I suggest using "easy_install -m myproject", as this does not need to run a
setup script. Instead, it will simply detect that 'myproject' is already
installed and remove it from sys.path.
>zipimport.ZipImportError: bad local file header in
>If I re-run the 'easy_install -m .' command it works properly.
>Everything in the installed .egg file appears to work correctly, so I
>could not figure out why a ZipImportError was being raised by a
Me either. The only time I've seen anything like that happen is with
Python 2.3 on a 64-bit platform. If you can figure out how to reproduce
it, please let me know.
>3. This may be a problem with distutils or setuptools, depending on
>how the install command is dispatched. I thought it would be nice to put the
>install record in the same directory I installed the other files in,
>so I appended the following to setup.cfg:
>Alas, $base is not expanded for the 'record' configuration option.
I think this feature can be added by simply inlcuding 'record' in the list
of variables that are expanded in line 125 of
setuptools/command/easy_install.py. If you have a chance to try that,
please let me know if it works for you, and then I'll add it to the