Working from Phillip's last email, here's my proposed list of steps for
creating a development environment; some of these could be automated,
but lets presume the automation does the same thing this manual process
Application name: MyApp
# We don't care where this directory is, except that we put all our
# in-development checkouts here...
# or an svn repository /MyApp/trunk, and check out to MyApp/
echo "Name: MyApp
Version: 0.0" > MyApp.egg-info/PKG-INFO
Cheetah" > MyApp.egg-info/depends.txt
# But I'm going to be developing SQLObject, but not pysqlite and
svn co http://svn.colorstudy.com/trunk/SQLObject ../SQLObject
# Well, this isn't right, because SQLObject doesn't import
# setuptools... I guess I have to run package/easy_install somehow?
python setup.py bdist_egg
# What do I run here to get the dependencies that aren't already
easy_install -d app-packages (?)
# But then I try to install pysqlite, but there's some weird conflict,
# and I wish I was just using the OS package... so what do I do to
# satisfy the dependency? Preferably without writing to
# site-packages/, where pysqlite is installed?
# and where are my scripts at this point? What should MyApp/setup.py
# look like? What should MyApp/ look like? Should it have a package
# underneath it?
Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org
Hi -- I'm getting ready to go out of town, so I haven't been able to
track everything that's gone on in 0.4 (or 0.5 now?), and I might not be
able to follow up on this conversation very thoroughly, but anyway...
I feel like we need several suggested ways to use easy_install and eggs,
given different cases. This is probably a documentation task; I'm more
interested in how I *should* use this, rather than how I *can*. I guess
that's the same as best practices. In turn I'd like to echo these best
practices in any documentation or examples that use easy_install/eggs.
Anyway, here's some cases:
* I am developing an application. I don't have a distutil script for
* I am simultaneously developing distutils-based libraries. I want to
edit these during my development process.
* I also have some distutils libraries that I'm not developing, but I
need to install.
* I also have some libraries laying around that I already installed
(before easy_install existed). And some that are from OS packages.
* I have other Python stuff on my computer that I don't want to mess up
because of my ongoing development.
So, given this, what commands do I run? Where do I put files? What if
I don't have permission to put files in global locations
(site-packages)? How do I clean up after myself later?
Later, this becomes...
* I have developed a web application. Maybe it also doesn't have a
distutils script...? It could, though currently I don't develop one for
my web applications. Also, I sometimes make hot fixes, especially when
the application is deployed but not yet live.
* How and where should libraries be installed? How should application
dependencies be expressed?
* Some libraries are internal, and so aren't available from a public
location. Maybe on the web with HTTP auth, though I'm more inclined to
simply keep them in a branch in the private repository. Or fetch over scp.
* Should I change my require()s to use a specific version of the
libraries, so that I don't accidentally upgrade (/break) the application
when a later application is installed? How do I manage that process?
* Later a library might be broken and I need to fix it. Is there
anything I should do on installation so I can later track who uses what
library? Also so I can collect unused versions of libraries.
* I'd like to distribute some data that doesn't have much (or maybe any)
(a bunch of templates and images), or something like that. Can I
facilitate that with easy_install?
* I have library A, and library B. Library B optionally provides a
"plugin" to library A, but both are usable in isolation. Library B
needs to inject stuff into library A -- i.e., at runtime some code in
library B enhances library A. How do I make this work? How do I make
library A aware of library B?
Other People's Code
* Someone wrote some code I'd like to use. But it's poorly packaged --
maybe no setup.py, or maybe a bad one. For instance, I've decided that
zpt.sf.net's setup.py is just broken -- you can't use extra_path, no
package, and provide an __init__.py all at once. I'd like to write my
own setup.py, but use that package. And it's on SF, so I'd like to use
easy_install to download the package.
Those are some of the things I'd like to do now -- easy_install doesn't
have to magically make all of them work wonderfully; if I have to do
things by hand, keep separate records, write custom code, or whatever,
that's fine; I just want to know what I should be doing right now for
each of these cases. Also, I'm interested in conventions we can define
so that we all start doing the same thing.
Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org
I've been getting the following when trying to use resource_filename
when my package is in a zipped egg (since somewhere around 0.3a2 or so):
Traceback (most recent call last):
File "/usr/local/bin/somescript.py", line 4, in ?
File "setuptools/pkg_resources.py", line 110, in run_main
File "setuptools/pkg_resources.py", line 599, in run_script
exec script_code in namespace, namespace
File "/usr/local/bin/somescript.py", line 7, in ?
File "egg/somepackage/somemodule.py", line 3, in ?
File "setuptools/pkg_resources.py", line 388, in resource_filename
File "setuptools/pkg_resources.py", line 728, in
return self._extract_resource(manager, resource_name)
File "setuptools/pkg_resources.py", line 745, in _extract_resource
zip_stat = self.zipinfo[os.path.join(*self.prefix+parts)]
Here's a test egg that illustrates the problem:
Install that and then run the included "somescript.py". You should
see the traceback above.
The problem seems to be with loading resources relative to a module
in a package. The egg contains a package "somepackage" and a module
within that package "somemodule". somemodule calls ``resource_filename
(__name__, 'someresource.txt')``. The ZipProvider seems to handle
this case badly.
Attached is a patch against 0.4a3 that fixes this particular case but
I'm not confident it won't cause problems elsewhere. I'll comment
each hunk here with an explanation of what I think may be happening.
@@ -573,7 +573,7 @@
- return self._listdir(self._fn(self.egg_info,resource_name))
+ return self._listdir(self._fn(self.module_path, resource_name))
Here we see NullProvider.resource_listdir using [path]/EGG_INFO
instead of [path]. This seems like a simple typo to me as there are
resource_XXX and metadata_XXX methods, the first based on the module
path and the second set based on the EGG_INFO dir.
@@ -703,7 +703,7 @@
def __init__(self, module):
self.zipinfo = zipimport._zip_directory_cache
- self.zip_pre = self.loader.archive+os.sep
+ self.zip_pre = self.module_path + os.sep
def _short_name(self, path):
This is where I was a bit shaky:
loader_archive = /path/to/the-egg.egg
module_path = /path/to/the-egg.egg/package
Shouldn't resource loading be relative to the package directory and
not the root of the egg?
@@ -738,7 +738,7 @@
def _extract_resource(self, manager, resource_name):
- return self._extract_dir(resource_name)
+ return self._extract_directory(manager, resource_name)
parts = resource_name.split('/')
zip_path = os.path.join(self.module_path, *parts)
I believe this was a simple typo / refactoring-miss. This isn't
breaking the test case but is causing problems in another package
that is using resource_filename to get a directory.
From "[Distutils] A plan for scripts (in EasyInstall)"
On Mon Jun 6 16:44:19 CEST 2005, Phillip J. Eby wrote:
> The 'run_main' function would do several things:
> * Clear everything but __name__ from the __main__ namespace
> * exec the script in __main__, using something like:
> maindict['__file__'] = pseudo_filename code =
> compile(script_source, pseudo_filename, "exec") exec code in
> maindict, maindict
It seems that bullet one is happening but not bullet two? I have
scripts that are attempting to use __file__ but failing with:
NameError: name '__file__' is not defined
The script looks like it would work properly if it was given a pseudo
filename but this has me thinking about what the best way to detect
development environments in scripts will look like in an eggified
environment. My wrapper scripts generally look something like the
following to detect whether the script is being run from a
development location or a deployed location::
from os.path import dirname, abspath, join, exists
devel_dir = dirname(dirname(abspath(__file__)))
if exists(join(devel_dir, 'support')):
sys.path.insert(1, join(devel_dir, 'support'))
from package.module import main
Assuming a directory layout of:
[devel-dir]/bin (scripts in here)
At first I thought I should switch from using path operations on
__file__ to using `pkg_resources.resource_isdir` and
`resource_filename` but that doesn't make any sense - if the script
is running from within a deployed egg, I'm not using it from a
development environment and the resource_*** functions don't make
sense in __main__ context anyway. So my current thinking is that the
existing idiom should remain and that a pseudo filename shouldn't
pose any problems in the scenarios I'm dealing with:
* Deployed egg: don't tamper with sys.path
* Deployed site-packages: don't tamper with sys.path
* Development environment: insert development paths
At any rate, I'm guessing that __file__ appears in a good percentage
of scripts in the wild. I've switched mine to use sys.argv in
place of __file__ but I'll see if I can get a patch together for
getting a psuedo __file__ into the exec dict (but don't wait up - I'm
not familiar with the code and figuring out how to get the psuedo
filename will take me infinite orders of magnitude longer than you ;)
Ran into a minor issue with resource_filename:
Traceback (most recent call last):
File "pkg_resources.py", line 388, in resource_filename
File "pkg_resources.py", line 728, in get_resource_filename
File "pkg_resources.py", line 740, in _extract_resource
File "pkg_resources.py", line 569, in resource_isdir
NameError: global name 'resource_name' is not defined
RCS file: /cvsroot/python/python/nondist/sandbox/setuptools/
retrieving revision 1.29
diff -u -r1.29 pkg_resources.py
--- pkg_resources.py 12 Jun 2005 21:47:33 -0000 1.29
+++ pkg_resources.py 13 Jun 2005 23:26:44 -0000
@@ -566,7 +566,7 @@
- return self._isdir(self._fn(self.module_path, resource_name))
+ return self._isdir(self._fn(self.module_path, name))
return self.egg_info and self._isdir(self._fn
Just a heads-up for any early adopters... I've released EasyInstall 0.4a2,
with support for installing scripts, and a new 'ez_setup.py' script/module
that you can use to install setuptools. Plus, you can include it in a
package that needs setuptools, and it will automatically download
setuptools for you, for use during your installation procedure. (It won't
actually *install* setuptools on the user's system in that mode, however;
I'll tackle that in a later release. Also, see the script's docstrings for
instructions on integrating it with your setup script.)
If you've been embedding easy_install facilities in another program or
setup script, be aware that the embedding interface changed dramatically in
this release due to the distutils integration, but it should be
comparatively stable from here on out.
Also, the installation procedure for easy_install has changed; be sure to
consult the updated documentation at:
Please help me test this new version; it is a *major* upgrade and
refactoring compared to the last release. I tried a variety of scenarios,
but I don't yet have any automated regression tests for many of its features.
As I move towards implementing features for EasyInsall like having
distutils-style configuration files, and refactoring the 'main()' function
for reusability and extension by other packages, I began to notice
something: I was basically reinventing a distutils Command object.
Sure, if I subclass Command, I'm stuck with a crummy 'log', and an ancient
ancestor of 'optparse' (i.e. 'fancy_getopt'), but I get config file parsing
for free, I get a known structure for arranging things, and I can trivially
get access to the work done by other distutils commands, like finding out
where the 'install' command would've installed libraries and scripts.
Not only that, but if I wanted to implement the commands other people have
been asking for, like to list installed packages, uninstall packages,
search PyPI, etc., I could just add them to the same harness. What's more,
people could add third-party commands using the --command-packages option
under 2.4 (and I could perhaps backport support into setuptools so that it
would work with 2.3 as well, at least for EasyInstall).
My main concern with this approach is lengthening the command line, because
now you'd have to specify a command, resulting in things like this:
python -m easy_install get SQLObject
The current name seems to clash with the idea of having different commands,
too. It almost seems like it should be more like:
python -m package get SQLObject
python -m package list
python -m package delete mechanize
python -m package check FooBar[someopt,otheropt]
python -m package download somepkg
python -m package extract otherpkg
In addition to providing a bit more flexibility, this would let us break up
the somewhat monolithic structure of the current installer code, so that
you could use individual pieces as shown, like extracting a package to a
directory without building or installing it. Not only would that make
these commands available to the user, but it would also make them available
to packages that use setuptools in their setup.py, such that a package
could e.g. download and extract the source of some C library that it
depends on, if it's not installed on the user's system. The 'download'
command could delegate to 'download_url', 'download_file', and
'download_req' subcommands, corresponding to the current
features. 'extract' would delegate to 'download', and so on. There'd
probably be a 'setup' command, too, to run a setup script under sandboxing,
So what are the downsides to this, apart from needing to turn lots of
little methods into entire Command subclasses? Well, I imagine it makes it
harder for somebody to build-in the commands to non-distutils programs,
unless there's an easy way to create and run the commands, like if I added
a class method like 'invoke(**args)' to a base class, so that you could say
stuff like 'package.download_url.invoke(url="whatever")' in order to call
'em from inside the program. I've already added something similar to the
'bdist_egg' comamnd called 'call_command(name,**args)', so that you can
more easily invoke other commands as subroutines.
Features like those could well be useful for the distutils as well, so
perhaps 'setuptools.Command' should include them. Right now, setuptools
exports the same Command class as the distutils, but I could easily change
that, so that Commands can be easily created and invoked from non-distutils
The other downside that needs mitigation is the logging crud. Distutils is
spattered with write() calls to sys.stderr and sys.stdout, and it
inconsistently uses its own 'log.' calls (e.g., it has messages beginning
with the word "Warning" that nonetheless use 'log.info()' as the
level!). It has stuff that depends on a distutils-specific verbosity
level, and so on.
All of this can and should be fixed in Python 2.5, but EasyInstall needs it
now, so setuptools will probably do some monkeypatching here and there to
get 'distutils.log' rerouted to 'logging.getLogger("distutils")', and
perhaps hack things in such a way that the distutils verbose/quiet stuff is
only used to change the log level, and and restores the level when commands
are finished. I'm not entirely sure how feasible this is. Some of the
sys.stderr writes are also not monkeypatchable without replacing giant
routines, and there are close to 70 print statements spread across 16
modules. Even just going through to verify which ones are debug prints is
a PITA. However, if someone were trying to integrate these commands into
some type of GUI application, it might cause occasional problems unless
stdout and stderr were replaced or redirected somehow.
Still, the bulk of the logging issues can probably be safely handled in a
couple of monkeypatches to Command, CCompiler, and text_file.TextFile; most
of the rest occur under pretty obscure circumstances. So, I think an 80%
solution is quite doable.
I don't really want to tangle with the fancy_getopt vs. optparse
distinction, though. Or any of the other places where a utility module in
the distutils has been left behind by a newfangled version in the standard
I mean, on one level it's always tempting to try to write "Distutils 2",
but at this point there's so much accumulated knowledge in distutils that I
don't think it's reasonable to expect a replacement to ever be doable in a
single rewrite. Refactoring the existing stuff is the only way to go.
Anyway, now I'm digressing. Thoughts, anyone?
Here's my semi-final plan for doing scripts with EasyInstall; please let me
know if you see any issues with it:
* Make bdist_egg stick the scripts under an EGG-INFO/scripts/ subdirectory
of the egg, by abusing the install_scripts command. (Note: this does *not*
mean they will be copied to 'PackageName.egg-info', only that they will be
included in the built egg file. Currently, subdirectories of .egg-info are
not included in the built egg, so this is completely separate.)
* Add 'metadata_isdir' and 'metadata_listdir' APIs to pkg_resources, to
allow inspecting the contents of EGG-INFO/scripts
* Add these options to the easy_install command:
--exclude-scripts, -x Don't install scripts
--scripts-to=DIR, -t DIR Install scripts to DIR
The --scripts-to option would default to being the *same as the
installation directory*, unless you're installing to site-packages, in
which case they would go to the Python default location for installing scripts.
Why the installation directory? Because if you run the scripts from there,
the egg(s) will be in a directory on sys.path, meaning that 'require()'
will then work. In essence, I'm assuming that the normal use case for
specifying an install-dir is to create an "application directory" filled
with the eggs needed to run a specific application. For example, on a
Unix-like system you might be installing to your ~/bin.
The downside to this assumption is that since the scripts are in the same
place, they might become importable when that's not intended. So if you're
installing to a personal ~/lib/python, you will probably want to use -x or
-t to override.
Anyway, if you are installing scripts, easy_install will just do what the
distutils does now to install them in their specified locations. This
basically means giving them executable permissions and munging the #! line
if applicable. On a multi-version install (-m or -i), it seems like we
should also add a line like:
from pkg_resources import require; require("thispackage==version"); del
But, as I've pointed out before, it's a tricky modification, as it would
need to be inserted just before the first *executable* line of code in the
script, which is often not the first line. Docstrings and __future__
statements both have to be skipped, or else the script could be broken by
the modification. Further, even a successful modification is going to
change the script's line numbering, which could have tech support
implications. So, I'm somewhat reluctant to do this without a way to turn
it off (other than by skipping scripts). There also needs to be a way to
verify that the script is in fact a Python script! (Presumably by checking
for a .py/.pyw extension or a #! line containing "python".)
Also, adding such a 'require()' line might be more restrictive than
necessary; the script might include its own require() already!
So, here's an alternative possibility. Let's suppose that the script *is*
a Python script. What if we ran it from inside the egg? We could write
out the script as a stub loader, looking something like this:
#!python <-- copied from original script, if present
The 'run_main' function would do several things:
* require() the appropriate package(s)
* Clear everything but __name__ from the __main__ namespace
* Load the script file into memory, and "poke" it into linecache.cache,
so that tracebacks from the script execution will still show its source
code, with correct line numbers
* exec the script in __main__, using something like:
maindict['__file__'] = pseudo_filename
code = compile(script_source, pseudo_filename, "exec")
exec code in maindict, maindict
Hm. execfile() could also be used if the script file actually exists, in
which case we could also skip the seeding of linecache. Probably we can
add a run_script() method to the IMetadataProvider interface, so that
different egg formats can handle this appropriately.
Now that we've come this far, it becomes clear that these "scripts" are
nothing more than bootstraps -- which means that in a future version I can
imagine allowing more user-friendly installation options, like .exe files
on Windows, "applications" on the Mac, and extension-stripping everywhere
else. However, it may be that the choice of script installation policy is
largely a matter of vehement personal preference, so there should probably
be a way to configure that. It could also include a way to define custom
installation policies in Python modules, and a way to select a particular
policy at runtime, e.g.:
easy_install --script-policy=mymodule.foo_policy ...
At this point, however, easy_install options will have gotten complex
enough to warrant configuration files for standard settings. We could
probably hijack the existing distutils configuration scheme for that,
though, treating 'easy_install' as if it were a distutils command.
Whew. I think that about covers it. Thoughts, anyone?
Il server non è riuscito a recapitare il messaggio perchè è infetto da virus.
Leggi le righe successive per il dettaglio del problema.
The message has been infected virus that you send to vigilanza(a)defecondo.com
delivery failure: the message has been infected virus.
--- Below this line is a report of the anti-virus engine.
Scan started: Mon Jun 6 15:13:07 2005
35866391/your_document.zip: Worm.SomeFool.P FOUND
-- summary --
Known viruses: 35438
Engine version: 0.85
Scanned directories: 1
Scanned files: 2
Infected files: 1
Data scanned: 0.03 MB
Time: 3.062 sec (0 m 3 s)
--- Below this line is a copy of the message.
Received: from unknown (HELO defecondo.com) ([184.108.40.206])
by [192.168.188.97] with ESMTP
for <vigilanza(a)defecondo.com>; Mon, 06 Jun 2005 15:13:06 +0200
Subject: Stolen document
Date: Mon, 6 Jun 2005 15:20:32 +0200