At 11:11 PM 3/12/2006 +0200, Ville Vainio wrote:
>I can't remember whether this has been discussed previously, but I'd
>like to see the ability to quickly install simple python modules
>globally, with the usage of
If you do this:
EasyInstall will treat the .py file as being an egg named MyProject, with a
version number of 0.1.
This should also work with file: URLs, and can also be used as links in a
download page or --find-links option.
EasyInstall automatically generates a setup.py to go with the .py file, and
runs it to create the egg which it then installs.
>Additionally, it would be groovy to be able to specify the entry
>points in .py files, a'la
># setuptools_entrypoint = myscript:main
> """ will be called by 'myscript' script."""
That won't work, but it's possible I could have it look for an assignment
__setuptools_entry_points__ = """
foo = mymodule:main
On the other hand, this seems like a path to diminishing returns. You
might as well start using a setup.py if you need entry points.
I can't remember whether this has been discussed previously, but I'd
like to see the ability to quickly install simple python modules
globally, with the usage of
Additionally, it would be groovy to be able to specify the entry
points in .py files, a'la
# setuptools_entrypoint = myscript:main
""" will be called by 'myscript' script."""
Ville Vainio - vivainio.googlepages.comvainio.blogspot.com - g[mail | talk]='vivainio'
So, coming back to the idea of a working environment, an isolated and
more-or-less self-contained environment for holding installed packages.
Sorry if this is a little scattered. I'm just summarizing my thoughts
and the open issues I see, in no particular order (because I'm not sure
what order to approach these things in).
I'm assuming such an environment will be encapsulated in a single
directory, looking something like:
The conf/ directory doesn't really relate to much of this specifically,
but in many situations it would be useful. Depending on the situation,
other subdirectories may exist.
Each of the scripts in bin/ should know what their working environment
is. This is slightly tricky, depending on what that means. If it is a
totally isolated environment -- no site-packages on sys.path -- then I
feel like the script wrappers have to be shell scripts, to invoke Python
with -S (which is hard to do portably on the #! line). I don't know the
details of doing the same thing on Windows, but I assume it is possible.
The actual directory location should be portable -- all paths should
be relative, and you should be able to move the directory around.
lib/python2.4/ is for packages. I'm almost inclined to say that
--single-version-externally-managed makes sense on some level, with a
record kept in some standard place (lib/python2.4/install-record.txt?)
-- but I'm basically indifferent. I at least don't see a need for
multiple simultaneous versions in this setup, and multiple versions do
lead to confusion. Version metadata is still nice, of course.
src/ is for checkouts, where each package is installed with setup.py
develop. These are naturally single-version, which is part of why I
like the idea of only using single-version setups. I'm a little unsure
of how src/ should be layed out. In practice I want all "my" packages
to be installed in src/ as checkouts, either from tags or the trunk (or
a branch or whatever). So I'm not sure if I should name the
subdirectories after the package, or maybe even the package plus a tag
One of the things SwitchTower (now "Cappucino", I think) does in Rails
land is it makes a dated checkout, then activates that checkout (it does
it with a symlink, we'd do it with setup.py develop). It then rolls
back by switching to an existing checkout. Of course svn switch + svn
up does this in place, and with less checkout trash laying around, even
if rollbacks aren't as fast as a result. So, I'm thinking just
There's an installation issue there -- it would be nice if I could say
"these are the packages I want to install as editable" and easy_install
would pick those up (maybe detecting based on what package index the
package was found in) and install them in src/ as editable.
sys.path would contain /usr/lib/python2.4, optionally
/usr/lib/python2.4/site-packages, and env/lib/python2.4/, and all the
similar directories. Unfortunately figuring out what "similar"
directories there are is hard. sys.path on my machine now has 63
entries normally and 12 with python -S. I guess I'd really like to
start with 12 and build up, instead of 63 and try to strip them down.
Installation as a whole is an open issue. Putting in env/setup.cfg with
the setting specific to that working environment works to a degree --
easy_install will pick it up if invoked from there. But that doesn't
work with setup.py develop, or setup.py install, or some other
scenarios. The system distutils.cfg doesn't really work, because the
only expansion it knows how to do is of user directories, so there's
little way to pass interesting information in (like a "this is my
setup.cfg" environmental variable or something). Maybe with PYTHONPATH
to indicate the working environment, and a distutils monkeypatch put
into lib/python2.4/distutils/__init__.py? I played around with putting
the path setup in sitecustomize, but that runs after site.py, and
doesn't run at all if python -S is used, so it seems like it brings in
too much before it can remove stuff.
Another option is a completely new python interpreter bound to the
environment. Basically the virtual-python.py option
In this model using env/bin/python indicate the proper environment,
and you'd have local installs of *everything* including easy_install.
This fixes so many problems without crazy hacks that it strongly appeals
to me, especially if we can make it somewhat lighter. I get this
/usr/lib/python2.4.zip on my path, that doesn't usually exist (does it
ever get created by default); if we could create that somehow on demand
and use such a big-bundle-zip, that seems lighter and faster and nicer.
If we just put .pyc files in it, and those .pyc files refer back to
the actual module source (in /usr/lib/python2.4/), then tracebacks
should also still work, right? No actual symlinks either, so it should
work on Windows. I'm not entirely sure where I'm going with this, though.
Sorry for the length. I've been stewing on this without a lot of
progress since PyCon so I thought I'd just throw out my current
thoughts. Maybe what I really want to do is hack on virtual-python.py
some more and see where that gets me.
Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org
I wrote a script last night that implements some of the ideas of a
working environment, similar in goals to virtual-python.py, but it
doesn't actually work like that.
When you run "python working-env.py env" you get:
activate (something you source in your shell to activate this
You activate the environment by adding env/lib/python2.4 to your
PYTHONPATH, and maybe adding env/bin/ to your PATH.
The custom site.py accomplishes most of the work, I guess. It's just
like the normal site.py, except it doesn't add site-packages (I should
probably make an option so you can set it up either way).
distutils/__init__.py is Ronald Oussoren's suggestion for monkeypatching
distutils. It both manages to get the working environment's
distutils.cfg to be used (comes for free, since distutils.dist looks
alongside distutils.__file__), and it substitutes __WORKING__ for the
actual working directory.
The setuptools monkeypatch changes how scripts are generated. But I
don't like how it works. It changes scripts to look more like:
import sys, os
join, dirname = os.path.join, os.path.dirname
lib_dir = join(dirname(dirname(__file__)), 'lib', 'python%s.%s' %
... normal stuff ...
Monkeypatching distutils is fine, because it's not going anywhere, but
monkeypatching setuptools is not going to be so reliable. But I guess
we just need to figure out exactly what we want to do, then move those
changes to setuptools. Also, distutils should probably be monkeypatched
directly in some fashion, so that distutils install also installs
scripts with this header.
A goal I have -- that is pretty much accomplished so far -- is that the
working environment be movable, and all paths be relative. But
setuptools creates some absolute directory names in .pth files, I believe.
Ian Bicking | ianb(a)colorstudy.com | http://blog.ianbicking.org
Just FYI, if you want to make double-clicking .egg files automatically
install them with EasyInstall, you can do it by putting the following
entries in the Windows registry (Python 2.3 example shown):
Windows Registry Editor Version 5.00
@="Install Egg for Python 2.3"
[HKEY_CLASSES_ROOT\Python.Egg\shell\Install Egg for Python 2.3]
[HKEY_CLASSES_ROOT\Python.Egg\shell\Install Egg for Python 2.3\command]
@="c:\\windows\\command\\cmd.exe /D /K C:\\Python23\\Scripts\\easy_install
I wasn't able to get Firefox to directly install straight from a download,
but saving to disk and then double-clicking worked. It might be
interesting to add a script to setuptools that would make these entries for
you, automatically adjusting paths where necessary. :)
Note that the '/K' part of the cmd.exe line ensures that a command shell
window is left open for confirmation when easy_install finishes. That way,
you can see any errors, warnings, etc.
Thanks to Tiago Cogumbreiro, by the way, for inadvertently giving me the
idea that something like this might be possible without writing any new
Okay, so I've been looking over the various remaining issues that people
have encountered with setuptools installation, and there's a kind of theme
emerging: conflict management and resolution.
First, let me give some of the problems. First off, the conflict detection
system at installation time isn't very bright. It'll happily report
conflicts with system-installed packages, even ones that are installed
using --single-version-externally-managed. It doesn't pay attention to
what order stuff is installed in on sys.path, either; it just says, "hey,
this *might* conflict" and kicks up a fuss. The "develop" command
currently doesn't check for conflicts at all. And finally, the
--delete-conflicting flag will happily attempt to delete system-installed
packages, even if they're .egg-info ones. Ouch.
And those are just the problems with install-time conflict
detection. Runtime conflict resolution's not that great, either. Once an
egg gets in the working set (e.g., due to being the default or
system-installed version of something), it never comes out, meaning that
you can't really override the default version. There's a special trick
that's used to bypass this issue for scripts, that for all practical
purposes just throws out sys.path and starts over if there's a conflict,
but it's a little drastic and in any case can't be generalized.
Finally, Guido has proposed allowing setuptools to get into the stdlib for
Python 2.5, but because it's a moving target, we need a way (ala PyXML) to
allow users to safely install newer versions of setuptools in such a way as
to replace the stdlib-supplied version.
After a fair amount of head-scratching, I think I have some ideas that can
fix all of these problems, but I want to air them out here before starting
implementation, to see if anybody can find any holes in the plan or
improvements to it. The runtime part is pretty easy and less likely to
have any controversial impacts, so I'll save that part for last, and deal
with the installation conflict issues first.
The original reason for having installation conflict detection was that
eggs added using .pth files normally ended up at the tail end of sys.path,
after any site-packages or other locally-installed packages. In effect,
eggs were always the lowest man on the totem pole, so you *had* to get rid
of any other installed version in order for it to work.
But in setuptools 0.6a6, I added logic to pkg_resources that automatically
placed activated eggs *before* their containing directory in
sys.path. Thus, an egg in site-packages would be placed (at runtime)
*before* the site-packages directory in sys.path, thereby overriding any
"legacy" or "system-installed" packages there. This is also true of any
other directory on sys.path: if it contains an egg, and you activate the
egg, the egg will be placed just before that directory in sys.path.
So in theory, conflict detection shouldn't have been needed since
0.6a6. In practice, however, people need to be able to use eggs *without*
importing pkg_resources and munging sys.path, so this is only a partial fix.
However, it does point the way to a full fix. If we could get .pth files
to do essentially the same trick, we'd be all set. Better still, if we
could do it in such a way that the added entries were before the stdlib on
sys.path, then we would at the same time accomplish the other need:
upgrading a stdlib-provided version of setuptools!
At PyCon, I had a few hallway conversations that touched on this issue, and
as I mentioned then, the simplest possible hack is something like adding
this line to a .pth file (e.g. setuptools.pth):
import sys; sys.path.insert(0,"/path/to/setuptools.egg")
But this isn't very helpful in the case where you might have more than one
sys.path directory with an easy-install.pth file in it. Why? Because the
later the directory is on sys.path, the *earlier* on sys.path the egg will
be inserted. Thus, the eggs end up in reverse order, with eggs installed
in site-packages would take precedence over even those on PYTHONPATH!
A Half-baked Solution
So, I've been racking my brain for a solution to this. Mostly, I was
thinking in terms of tracking what egg version is selected, but today I
realized there is a simpler solution. We just need to ensure that the
insertion point *moves forward* each time we insert an egg, e.g.:
import sys; sys.__egginsert=p=getattr(sys,'__egginsert',-1)+1; \
I've used a \ continuation for clarity here, but of course the real .pth
file couldn't do this; the whole thing has to be on one line.
I've considered a few other ways to do this, like inserting a marker value
directly into sys.path:
import sys; '@@@' not in sys.path and sys.path.insert(0,'@@@'); \
But this has the problem of needing to choose a suitable marker value that
won't be a valid import string, and doesn't introduce security problems,
etc. So I came up with the __egginsert approach as a way to fix that.
The main (potential) problems I see remaining with __egginsert are:
1. Change in semantics of current install (can't be helped, but needs
2. If somebody else is crazy enough to do something similar, hilarity may
ensue as hacks in different .pth files use insert points that don't go
where they think. (Not that likely; few people have even heard of .pth
files, and in any case the insertions will retain a consistent order
*within* each set, putting an upper limit on the possible chaos.)
3. EasyInstall will have to be able to *read* these lines as well as write
them, and parsing one correctly looks a bit daunting to do.
4. The 'site' module has to 'exec' lines beginning with 'import', so
there's a compile-and-exec delay imposed by each line, thus increasing the
incremental startup cost for each installed egg.
A More Complete Solution
So here's a new idea, made up fresh while writing this:
import sys; sys.__oldpath=sys.path[:]; sys.path[:]=
... normal .pth lines go here ...
import sys; new=sys.path[:]; sys.path[:]=sys.__oldpath; \
p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; \
sys.__egginsert = p+len(new)
(Again, the \ continuations won't be in the real file.)
Anyway, what this does is temporarily nuke sys.path, then let the normal
.pth processing logic run until we get to the end of the file, where we
restore it and slap in the list of directories that were added by the
Or, perhaps more simply:
import sys; sys.__plen = len(sys.path)
... normal .pth lines go here ...
import sys; new=sys.path[sys.__plen:]; \
del sys.path[sys.__plen:]; \
p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; \
sys.__egginsert = p+len(new)
This doesn't wipe out sys.path during the interim processing, although I
suppose it is really just as complex as the previous version. Somehow, it
just feels "safer" not to ditch sys.path during the processing.
Anyway, either of these will keep the number of 'exec's capped at a maximum
of 2 lines, *and* they'll make easy_install's parsing job easy, because it
can just ignore the "import" lines when reading, and just regenerate them
This change would also thus affect only easy_install'ed eggs; it would have
no effect on normal .pth processing. But I would have to slightly alter
the existing 'site.py' hack that setuptools installs in PYTHONPATH
directories, so that it resets the sys.__egginsert position to zero before
processing the .pth files on PYTHONPATH, and then restores its original
value afterward. (That way, eggs installed on PYTHONPATH will come before
those installed in site-packages, but any subsequent addsitedir()
operations will add to the right spot, more or less.)
That does suggest that there's some fragility involved in this process, if
you use addsitedir() at runtime. This is probably the biggest issue with
I did a bit of googling, however, and was only able to find 3 or 4 projects
that do any kind of addsitedir stuff, including the usual suspects such as
Twisted and Zope. :) I didn't find anything that looked like it would be
negatively affected, although I admit I didn't study the surrounding code
too deeply. Mostly, they were doing things that were basically the same as
what we're trying to do here: i.e., support .pth files in PYTHONPATH and in
some cases, put them first on sys.path.
The only place where confusing results are likely, is the case where
somebody calls addsitedir() *after* the pkg_resources module is imported,
thus causing the working_set to be out of sync with sys.path. Of course,
this is *already* a possible problem if you munge sys.path after importing
pkg_resources, which suggests that some additional conflict detection there
would also be helpful.
(Side rant: this is one of those times where the singleton nature of
certain Python features is really really annoying. If only one could
instantiate interpreters in standard Python!)
Anyway, sys.path manipulation is already fraught with numerous dangers, so
I won't dwell on that issue for now. Perhaps in 0.7 or 0.8 I'll revamp the
WorkingSet class so it can deal with dynamic manipulation of an
Proposed Installation Strategy
To summarize my proposal for handling installation "conflicts":
* EasyInstall will write special .pth files with a header and trailer
"import" hack. It will ignore "import" lines when reading them.
* I'll change the existing site.py hack to accomodate the out-of-order
insertion, and make EasyInstall update any existing hacked site.py files.
* EasyInstall will stop checking for (and deleting) installation conflicts,
because there will *no longer be such a thing*. The options that control
conflict detection will remain, but will have no function except to issue
And to summarize the effects of these changes:
* Eggs installed by EasyInstall will have sys.path precedence over
everything else, including the standard library, script directory,
etc. They will, however, have a precedence order amongst themselves that
reflects the sequence in which the .pth files were loaded.
* Invoking site.addsitedir() after pkg_resources is imported may produce
slightly weirder results than it already does. :)
* EasyInstall will be able to upgrade stdlib packages, not just via
PYTHONPATH installs, but also via site-packages installs.
And the issues that remain open are:
* Should --single-version-externally-managed installations check for
conflicts? Of course, users of the option are theoretically more
experienced users/packagers, and are also accustomed to the distutils'
default (and thoroughly unsafe) behavior, so perhaps they should be left to
their own devices.
* Should pkg_resources resort to more extreme measures to track the runtime
value of sys.path? We'll perhaps look more at this a bit more closely
below, in the section on runtime conflict management.
Does anybody see any issues I haven't covered? Any ideas for improvements?
The solution to runtime conflicts was a little easier to figure out,
although now that I've figured out the installation conflicts side, I think
the runtime side is actually going to be a lot tougher to implement
correctly, as there are a lot more moving parts.
But the idea is relatively simple to explain: when trying to resolve
dependencies, handle VersionConflict errors by:
1. Removing the conflicting package from the working set, if and only if
it's safe to do so.
2. Retry the operation, and repeat.
If a package isn't safe to remove, reraise the conflict error after
restoring any previously-removed packages, so that the working set is
rolled back to its original state.
At this level, it's all pretty simple. The tricky parts are in the details:
1. This isn't really compatible with the existing API, for anything but the
require() operation. There's no sane way to change the resolve() API to
work with this, because resolve() isn't supposed to have side effects, and
it only returns a list of distributions to *add*, not ones to *remove*.
2. Defining "safe to remove" is complex. As a first approximation, it's
safe to remove a distribution from the working set if:
a) it's the only distribution with that file/directory name (Think of
.egg-info dirs all sitting in site-packages; you can't remove one of those
babies without removing *all* system-installed eggs in site-packages, since
they all share a single sys.path entry!)
b) it hasn't had anything imported from it yet
c) no namespace packages it participates in have been imported
Those latter two conditions are interesting, because they basically
mean you have to go through every sys.modules entry looking for names
starting with anything in the egg's top_level list, and then check those
modules' __file__ and __path__ values to see if anything starts with the
egg's location. This is likely to be slow, but that's not really a big
deal; this is a fallback for resolving a runtime version conflict, after
all. But it's somewhat tricky to test well, due to the singletons (e.g.
sys.modules) that are involved.
3. Working sets don't actually allow you remove anything currently, and I'm
not sure how to actually implement that feature!
So, after reviewing this, I think I'm going to avoid touching this during
the remainder of the 0.6 development cycle, because it seems too likely to
introduce instability. Also, requirement 2c above isn't really practical
before 0.7, because of the current eager loading of namespace packages. It
would basically rule out conflict resolution for eggs that currently
participate in namespace packages.
Last, but not least, if I put it off till 0.7, I can go whole hog on
refactoring the WorkingSet class, to also support better sys.path tracking
at runtime. And if some corners of the API have to get tweaked a bit to
make that work in 0.7, I can probably live with it.
Any other thoughts or suggestions?
At 04:37 PM 3/7/2006 -0500, Kevin Dangoor wrote:
>On 3/7/06, Ben Bangert <ben(a)groovie.org> wrote:
> > Indeed, I just noticed that setup.py was importing version from the
> > package, and the location it was importing, then imported other stuff
> > that had the dependencies. Thanks for the pointer on what was going
> > wrong.
>This appears to be the general solution to that particular problem. In
>or something along those lines.
That only works if 'release.py' doesn't import any dependencies. :)
That is, the issue doesn't have anything to do with imports per se, only
imports that cause dependencies to come into play.
I think I had this problem earlier, and it went away, but now its
back for some reason. Several users have reported it to me as well.
Apparently the sandbox testing runs, and tries to import a package
that wasn't installed yet. Is there anyway to make it actually
install dependencies *before* doing its sandbox stuff?
The traceback looks like this:
[ben-bangerts-power-mac-g4:~] ben% sudo easy_install -U Pylons==dev
Searching for Pylons==dev
Best match: Pylons dev
Doing subversion checkout from http://pylons.groovie.org/svn/Pylons/
trunk to /tmp/easy_install-8oXRPG/trunk
Running setup.py -q bdist_egg --dist-dir /tmp/easy_install-8oXRPG/
Traceback (most recent call last):
File "/usr/local/bin/easy_install", line 6, in ?
command/easy_install.py", line 1506, in main
command/easy_install.py", line 1495, in with_ei_usage
command/easy_install.py", line 1509, in <lambda>
python2.4/distutils/core.py", line 149, in setup
python2.4/distutils/dist.py", line 946, in run_commands
python2.4/distutils/dist.py", line 966, in run_command
command/easy_install.py", line 211, in run
self.easy_install(spec, not self.no_deps)
command/easy_install.py", line 446, in easy_install
return self.install_item(spec, dist.location, tmpdir, deps)
command/easy_install.py", line 461, in install_item
dists = self.install_eggs(spec, download, tmpdir)
command/easy_install.py", line 655, in install_eggs
return self.build_and_install(setup_script, setup_base)
command/easy_install.py", line 930, in build_and_install
self.run_setup(setup_script, setup_base, args)
command/easy_install.py", line 919, in run_setup
sandbox.py", line 26, in run_setup
sandbox.py", line 63, in run
sandbox.py", line 29, in <lambda>
File "/Users/ben/Programming/Python/pudge/setup.py", line 5, in ?
File "/tmp/easy_install-8oXRPG/trunk/pylons/__init__.py", line 6,
File "/tmp/easy_install-8oXRPG/trunk/pylons/decorators.py", line
4, in ?
ImportError: No module named simplejson
I was having a bit of trouble installing python module packages on Gentoo
after following the "Administrator Installation" section of the documentation.
I kept getting packages installed to /var/tmp/homedir/python2.4/ or something
like that. I couldn't figure out what was going on until I came accross a
gentoo bug that someone filed and then closed citing installation of
setuptools as the culprit. Thats when I realised that I added the following
to my distutils.cfg:
install_lib = ~/lib/python2.3
# This next line is optional but often quite useful; it directs EasyInstall
# and the distutils to install scripts in the user's "bin" directory. For
# Mac OS X framework Python builds, you should use /usr/local/bin instead,
# because neither ~/bin nor the default script installation location are on
# the system PATH.
install_scripts = ~/bin
Which would cause all distutils based installations to do that.
Further up in the documenation it talks about using the [easy_install] section
of distutils.cfg to set some preferences. I changed the above to:
install_dir = ~/lib/python2.3
scripts_dir = ~/bin
I would recommend changing the documenation to avoid confusion since most
people probably want to leave their old distutils operation alone and add
these settings for easy_install.