Mailman 3 June 2005 - Distutils-SIG

EasyInstall: development environment
by Ian Bicking 16 Jun '05

16 Jun '05

Working from Phillip's last email, here's my proposed list of steps for creating a development environment; some of these could be automated, but lets presume the automation does the same thing this manual process does... Application name: MyApp # We don't care where this directory is, except that we put all our # in-development checkouts here... cd ~/devel-directory mkdir MyApp # or an svn repository /MyApp/trunk, and check out to MyApp/ cd MyApp mkdir MyApp.egg-info echo "Name: MyApp Version: 0.0" > MyApp.egg-info/PKG-INFO echo "SQLObject>=0.6.1 pysqlite2 Cheetah" > MyApp.egg-info/depends.txt # But I'm going to be developing SQLObject, but not pysqlite and # Cheetah... svn co http://svn.colorstudy.com/trunk/SQLObject ../SQLObject cd ../SQLObject # Well, this isn't right, because SQLObject doesn't import # setuptools... I guess I have to run package/easy_install somehow? python setup.py bdist_egg cd .. export PYTHONPATH=`pwd` cd MyApp mkdir app-packages # What do I run here to get the dependencies that aren't already # fulfilled? easy_install -d app-packages (?) # But then I try to install pysqlite, but there's some weird conflict, # and I wish I was just using the OS package... so what do I do to # satisfy the dependency? Preferably without writing to # site-packages/, where pysqlite is installed? # and where are my scripts at this point? What should MyApp/setup.py # look like? What should MyApp/ look like? Should it have a package # underneath it? -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

2 1

EasyInstall: suggested usage patterns
by Ian Bicking 16 Jun '05

16 Jun '05

Hi -- I'm getting ready to go out of town, so I haven't been able to track everything that's gone on in 0.4 (or 0.5 now?), and I might not be able to follow up on this conversation very thoroughly, but anyway... I feel like we need several suggested ways to use easy_install and eggs, given different cases. This is probably a documentation task; I'm more interested in how I *should* use this, rather than how I *can*. I guess that's the same as best practices. In turn I'd like to echo these best practices in any documentation or examples that use easy_install/eggs. Anyway, here's some cases: Development * I am developing an application. I don't have a distutil script for the application. * I am simultaneously developing distutils-based libraries. I want to edit these during my development process. * I also have some distutils libraries that I'm not developing, but I need to install. * I also have some libraries laying around that I already installed (before easy_install existed). And some that are from OS packages. * I have other Python stuff on my computer that I don't want to mess up because of my ongoing development. So, given this, what commands do I run? Where do I put files? What if I don't have permission to put files in global locations (site-packages)? How do I clean up after myself later? Later, this becomes... Deployment * I have developed a web application. Maybe it also doesn't have a distutils script...? It could, though currently I don't develop one for my web applications. Also, I sometimes make hot fixes, especially when the application is deployed but not yet live. * How and where should libraries be installed? How should application dependencies be expressed? * Some libraries are internal, and so aren't available from a public location. Maybe on the web with HTTP auth, though I'm more inclined to simply keep them in a branch in the private repository. Or fetch over scp. * Should I change my require()s to use a specific version of the libraries, so that I don't accidentally upgrade (/break) the application when a later application is installed? How do I manage that process? * Later a library might be broken and I need to fix it. Is there anything I should do on installation so I can later track who uses what library? Also so I can collect unused versions of libraries. Non-code * I'd like to distribute some data that doesn't have much (or maybe any) code. This might be a Javascript library, or a skin for the application (a bunch of templates and images), or something like that. Can I facilitate that with easy_install? Enabling plugins * I have library A, and library B. Library B optionally provides a "plugin" to library A, but both are usable in isolation. Library B needs to inject stuff into library A -- i.e., at runtime some code in library B enhances library A. How do I make this work? How do I make library A aware of library B? Other People's Code * Someone wrote some code I'd like to use. But it's poorly packaged -- maybe no setup.py, or maybe a bad one. For instance, I've decided that zpt.sf.net's setup.py is just broken -- you can't use extra_path, no package, and provide an __init__.py all at once. I'd like to write my own setup.py, but use that package. And it's on SF, so I'd like to use easy_install to download the package. Those are some of the things I'd like to do now -- easy_install doesn't have to magically make all of them work wonderfully; if I have to do things by hand, keep separate records, write custom code, or whatever, that's fine; I just want to know what I should be doing right now for each of these cases. Also, I'm interested in conventions we can define so that we all start doing the same thing. -- Ian Bicking / ianb(a)colorstudy.com / http://blog.ianbicking.org

2 3

PATCH: setuptools-0.4a3 resource loading from egg problem
by Ryan Tomayko 14 Jun '05

14 Jun '05

I've been getting the following when trying to use resource_filename when my package is in a zipped egg (since somewhere around 0.3a2 or so): Traceback (most recent call last): File "/usr/local/bin/somescript.py", line 4, in ? pkg_resources.run_main('somepackage==1.0', 'somescript.py') File "setuptools/pkg_resources.py", line 110, in run_main require(dist_spec)[0].metadata.run_script(script_name, __main__.__dict__) File "setuptools/pkg_resources.py", line 599, in run_script exec script_code in namespace, namespace File "/usr/local/bin/somescript.py", line 7, in ? File "egg/somepackage/somemodule.py", line 3, in ? File "setuptools/pkg_resources.py", line 388, in resource_filename return get_provider(package_name).get_resource_filename( File "setuptools/pkg_resources.py", line 728, in get_resource_filename return self._extract_resource(manager, resource_name) File "setuptools/pkg_resources.py", line 745, in _extract_resource zip_stat = self.zipinfo[os.path.join(*self.prefix+parts)] KeyError: 'somepackage/someresource.txt' Here's a test egg that illustrates the problem: <http://lesscode.org/eggs/somepackage-1.0-py2.4.egg> Install that and then run the included "somescript.py". You should see the traceback above. The problem seems to be with loading resources relative to a module in a package. The egg contains a package "somepackage" and a module within that package "somemodule". somemodule calls ``resource_filename (__name__, 'someresource.txt')``. The ZipProvider seems to handle this case badly. Attached is a patch against 0.4a3 that fixes this particular case but I'm not confident it won't cause problems elsewhere. I'll comment each hunk here with an explanation of what I think may be happening. @@ -573,7 +573,7 @@ def resource_listdir(self,resource_name): - return self._listdir(self._fn(self.egg_info,resource_name)) + return self._listdir(self._fn(self.module_path, resource_name)) def metadata_listdir(self,name): if self.egg_info: Here we see NullProvider.resource_listdir using [path]/EGG_INFO instead of [path]. This seems like a simple typo to me as there are resource_XXX and metadata_XXX methods, the first based on the module path and the second set based on the EGG_INFO dir. @@ -703,7 +703,7 @@ def __init__(self, module): DefaultProvider.__init__(self,module) self.zipinfo = zipimport._zip_directory_cache [self.loader.archive] - self.zip_pre = self.loader.archive+os.sep + self.zip_pre = self.module_path + os.sep def _short_name(self, path): if path.startswith(self.zip_pre): This is where I was a bit shaky: loader_archive = /path/to/the-egg.egg module_path = /path/to/the-egg.egg/package Shouldn't resource loading be relative to the package directory and not the root of the egg? @@ -738,7 +738,7 @@ def _extract_resource(self, manager, resource_name): if self.resource_isdir(resource_name): - return self._extract_dir(resource_name) + return self._extract_directory(manager, resource_name) parts = resource_name.split('/') zip_path = os.path.join(self.module_path, *parts) I believe this was a simple typo / refactoring-miss. This isn't breaking the test case but is causing problems in another package that is using resource_filename to get a directory. Thanks, Ryan Tomayko rtomayko(a)gmail.com http://naeblis.cx/rtomayko/

2 5

setuptools-0.4a2: Eggs, scripts, and __file__
by Ryan Tomayko 14 Jun '05

14 Jun '05

From "[Distutils] A plan for scripts (in EasyInstall)" <http://mail.python.org/pipermail/distutils-sig/2005-June/004594.html> On Mon Jun 6 16:44:19 CEST 2005, Phillip J. Eby wrote: > The 'run_main' function would do several things: <snip> > > * Clear everything but __name__ from the __main__ namespace <snip> > * exec the script in __main__, using something like: > > maindict['__file__'] = pseudo_filename code = > compile(script_source, pseudo_filename, "exec") exec code in > maindict, maindict It seems that bullet one is happening but not bullet two? I have scripts that are attempting to use __file__ but failing with: NameError: name '__file__' is not defined The script looks like it would work properly if it was given a pseudo filename but this has me thinking about what the best way to detect development environments in scripts will look like in an eggified environment. My wrapper scripts generally look something like the following to detect whether the script is being run from a development location or a deployed location:: import sys from os.path import dirname, abspath, join, exists devel_dir = dirname(dirname(abspath(__file__))) if exists(join(devel_dir, 'support')): sys.path.insert(1, join(devel_dir, 'support')) sys.path.insert(1, devel_dir) from package.module import main main(sys.argv) Assuming a directory layout of: [devel-dir]/bin (scripts in here) [devel-dir]/support (packages/modules) [devel-dir]/[package-name] At first I thought I should switch from using path operations on __file__ to using `pkg_resources.resource_isdir` and `resource_filename` but that doesn't make any sense - if the script is running from within a deployed egg, I'm not using it from a development environment and the resource_*** functions don't make sense in __main__ context anyway. So my current thinking is that the existing idiom should remain and that a pseudo filename shouldn't pose any problems in the scenarios I'm dealing with: * Deployed egg: don't tamper with sys.path * Deployed site-packages: don't tamper with sys.path * Development environment: insert development paths At any rate, I'm guessing that __file__ appears in a good percentage of scripts in the wild. I've switched mine to use sys.argv[0] in place of __file__ but I'll see if I can get a patch together for getting a psuedo __file__ into the exec dict (but don't wait up - I'm not familiar with the code and figuring out how to get the psuedo filename will take me infinite orders of magnitude longer than you ;) Ryan Tomayko rtomayko(a)gmail.com http://naeblis.cx/rtomayko/

2 6

PATCH: setuptools-0.4a2 resource_filename prob
by Ryan Tomayko 14 Jun '05

14 Jun '05

Ran into a minor issue with resource_filename: Traceback (most recent call last): File "pkg_resources.py", line 388, in resource_filename File "pkg_resources.py", line 728, in get_resource_filename File "pkg_resources.py", line 740, in _extract_resource File "pkg_resources.py", line 569, in resource_isdir NameError: global name 'resource_name' is not defined One liner: Index: pkg_resources.py =================================================================== RCS file: /cvsroot/python/python/nondist/sandbox/setuptools/ pkg_resources.py,v retrieving revision 1.29 diff -u -r1.29 pkg_resources.py --- pkg_resources.py 12 Jun 2005 21:47:33 -0000 1.29 +++ pkg_resources.py 13 Jun 2005 23:26:44 -0000 @@ -566,7 +566,7 @@ return yield_lines(self.get_metadata(name)) def resource_isdir(self,name): - return self._isdir(self._fn(self.module_path, resource_name)) + return self._isdir(self._fn(self.module_path, name)) def metadata_isdir(self,name): return self.egg_info and self._isdir(self._fn (self.egg_info,name)) Ryan Tomayko rtomayko(a)gmail.com http://naeblis.cx/rtomayko/

2 2

FYI: EasyInstall 0.4a2 with scripts and 'ez_setup' bootstrapper
by Phillip J. Eby 13 Jun '05

13 Jun '05

Just a heads-up for any early adopters... I've released EasyInstall 0.4a2, with support for installing scripts, and a new 'ez_setup.py' script/module that you can use to install setuptools. Plus, you can include it in a package that needs setuptools, and it will automatically download setuptools for you, for use during your installation procedure. (It won't actually *install* setuptools on the user's system in that mode, however; I'll tackle that in a later release. Also, see the script's docstrings for instructions on integrating it with your setup script.) If you've been embedding easy_install facilities in another program or setup script, be aware that the embedding interface changed dramatically in this release due to the distutils integration, but it should be comparatively stable from here on out. Also, the installation procedure for easy_install has changed; be sure to consult the updated documentation at: http://peak.telecommunity.com/DevCenter/EasyInstall Please help me test this new version; it is a *major* upgrade and refactoring compared to the last release. I tried a variety of scenarios, but I don't yet have any automated regression tests for many of its features.

2 2

EasyInstall --> distutils command + setuptools enhancements?
by Phillip J. Eby 12 Jun '05

12 Jun '05

As I move towards implementing features for EasyInsall like having distutils-style configuration files, and refactoring the 'main()' function for reusability and extension by other packages, I began to notice something: I was basically reinventing a distutils Command object. Sure, if I subclass Command, I'm stuck with a crummy 'log', and an ancient ancestor of 'optparse' (i.e. 'fancy_getopt'), but I get config file parsing for free, I get a known structure for arranging things, and I can trivially get access to the work done by other distutils commands, like finding out where the 'install' command would've installed libraries and scripts. Not only that, but if I wanted to implement the commands other people have been asking for, like to list installed packages, uninstall packages, search PyPI, etc., I could just add them to the same harness. What's more, people could add third-party commands using the --command-packages option under 2.4 (and I could perhaps backport support into setuptools so that it would work with 2.3 as well, at least for EasyInstall). My main concern with this approach is lengthening the command line, because now you'd have to specify a command, resulting in things like this: python -m easy_install get SQLObject The current name seems to clash with the idea of having different commands, too. It almost seems like it should be more like: python -m package get SQLObject python -m package list python -m package delete mechanize python -m package check FooBar[someopt,otheropt] python -m package download somepkg python -m package extract otherpkg In addition to providing a bit more flexibility, this would let us break up the somewhat monolithic structure of the current installer code, so that you could use individual pieces as shown, like extracting a package to a directory without building or installing it. Not only would that make these commands available to the user, but it would also make them available to packages that use setuptools in their setup.py, such that a package could e.g. download and extract the source of some C library that it depends on, if it's not installed on the user's system. The 'download' command could delegate to 'download_url', 'download_file', and 'download_req' subcommands, corresponding to the current features. 'extract' would delegate to 'download', and so on. There'd probably be a 'setup' command, too, to run a setup script under sandboxing, etc. So what are the downsides to this, apart from needing to turn lots of little methods into entire Command subclasses? Well, I imagine it makes it harder for somebody to build-in the commands to non-distutils programs, unless there's an easy way to create and run the commands, like if I added a class method like 'invoke(**args)' to a base class, so that you could say stuff like 'package.download_url.invoke(url="whatever")' in order to call 'em from inside the program. I've already added something similar to the 'bdist_egg' comamnd called 'call_command(name,**args)', so that you can more easily invoke other commands as subroutines. Features like those could well be useful for the distutils as well, so perhaps 'setuptools.Command' should include them. Right now, setuptools exports the same Command class as the distutils, but I could easily change that, so that Commands can be easily created and invoked from non-distutils code. The other downside that needs mitigation is the logging crud. Distutils is spattered with write() calls to sys.stderr and sys.stdout, and it inconsistently uses its own 'log.' calls (e.g., it has messages beginning with the word "Warning" that nonetheless use 'log.info()' as the level!). It has stuff that depends on a distutils-specific verbosity level, and so on. All of this can and should be fixed in Python 2.5, but EasyInstall needs it now, so setuptools will probably do some monkeypatching here and there to get 'distutils.log' rerouted to 'logging.getLogger("distutils")', and perhaps hack things in such a way that the distutils verbose/quiet stuff is only used to change the log level, and and restores the level when commands are finished. I'm not entirely sure how feasible this is. Some of the sys.stderr writes are also not monkeypatchable without replacing giant routines, and there are close to 70 print statements spread across 16 modules. Even just going through to verify which ones are debug prints is a PITA. However, if someone were trying to integrate these commands into some type of GUI application, it might cause occasional problems unless stdout and stderr were replaced or redirected somehow. Still, the bulk of the logging issues can probably be safely handled in a couple of monkeypatches to Command, CCompiler, and text_file.TextFile; most of the rest occur under pretty obscure circumstances. So, I think an 80% solution is quite doable. I don't really want to tangle with the fancy_getopt vs. optparse distinction, though. Or any of the other places where a utility module in the distutils has been left behind by a newfangled version in the standard library. :( I mean, on one level it's always tempting to try to write "Distutils 2", but at this point there's so much accumulated knowledge in distutils that I don't think it's reasonable to expect a replacement to ever be doable in a single rewrite. Refactoring the existing stuff is the only way to go. Anyway, now I'm digressing. Thoughts, anyone?

2 9

EasyInstall and build-time dependencies
by Ronald Oussoren 10 Jun '05

10 Jun '05

Are there any plans to teach EasyInstall about build time dependencies? And example of this would be the dependency of PIL on libjpeg and libtiff. Ronald

2 2

A plan for scripts (in EasyInstall)
by Phillip J. Eby 06 Jun '05

06 Jun '05

Here's my semi-final plan for doing scripts with EasyInstall; please let me know if you see any issues with it: * Make bdist_egg stick the scripts under an EGG-INFO/scripts/ subdirectory of the egg, by abusing the install_scripts command. (Note: this does *not* mean they will be copied to 'PackageName.egg-info', only that they will be included in the built egg file. Currently, subdirectories of .egg-info are not included in the built egg, so this is completely separate.) * Add 'metadata_isdir' and 'metadata_listdir' APIs to pkg_resources, to allow inspecting the contents of EGG-INFO/scripts * Add these options to the easy_install command: --exclude-scripts, -x Don't install scripts --scripts-to=DIR, -t DIR Install scripts to DIR The --scripts-to option would default to being the *same as the installation directory*, unless you're installing to site-packages, in which case they would go to the Python default location for installing scripts. Why the installation directory? Because if you run the scripts from there, the egg(s) will be in a directory on sys.path, meaning that 'require()' will then work. In essence, I'm assuming that the normal use case for specifying an install-dir is to create an "application directory" filled with the eggs needed to run a specific application. For example, on a Unix-like system you might be installing to your ~/bin. The downside to this assumption is that since the scripts are in the same place, they might become importable when that's not intended. So if you're installing to a personal ~/lib/python, you will probably want to use -x or -t to override. Anyway, if you are installing scripts, easy_install will just do what the distutils does now to install them in their specified locations. This basically means giving them executable permissions and munging the #! line if applicable. On a multi-version install (-m or -i), it seems like we should also add a line like: from pkg_resources import require; require("thispackage==version"); del require But, as I've pointed out before, it's a tricky modification, as it would need to be inserted just before the first *executable* line of code in the script, which is often not the first line. Docstrings and __future__ statements both have to be skipped, or else the script could be broken by the modification. Further, even a successful modification is going to change the script's line numbering, which could have tech support implications. So, I'm somewhat reluctant to do this without a way to turn it off (other than by skipping scripts). There also needs to be a way to verify that the script is in fact a Python script! (Presumably by checking for a .py/.pyw extension or a #! line containing "python".) Also, adding such a 'require()' line might be more restrictive than necessary; the script might include its own require() already! So, here's an alternative possibility. Let's suppose that the script *is* a Python script. What if we ran it from inside the egg? We could write out the script as a stub loader, looking something like this: #!python <-- copied from original script, if present import pkg_resources pkg_resources.run_main("scriptname", "EggName") The 'run_main' function would do several things: * require() the appropriate package(s) * Clear everything but __name__ from the __main__ namespace * Load the script file into memory, and "poke" it into linecache.cache, so that tracebacks from the script execution will still show its source code, with correct line numbers * exec the script in __main__, using something like: maindict['__file__'] = pseudo_filename code = compile(script_source, pseudo_filename, "exec") exec code in maindict, maindict Hm. execfile() could also be used if the script file actually exists, in which case we could also skip the seeding of linecache. Probably we can add a run_script() method to the IMetadataProvider interface, so that different egg formats can handle this appropriately. Now that we've come this far, it becomes clear that these "scripts" are nothing more than bootstraps -- which means that in a future version I can imagine allowing more user-friendly installation options, like .exe files on Windows, "applications" on the Mac, and extension-stripping everywhere else. However, it may be that the choice of script installation policy is largely a matter of vehement personal preference, so there should probably be a way to configure that. It could also include a way to define custom installation policies in Python modules, and a way to select a particular policy at runtime, e.g.: easy_install --script-policy=mymodule.foo_policy ... At this point, however, easy_install options will have gotten complex enough to warrant configuration files for standard settings. We could probably hijack the existing distutils configuration scheme for that, though, treating 'easy_install' as if it were a distutils command. Whew. I think that about covers it. Thoughts, anyone?

2 2

Allarme Virus
by postmaster＠defecondo.com 06 Jun '05

06 Jun '05

Salve. Il server non è riuscito a recapitare il messaggio perchè è infetto da virus. Leggi le righe successive per il dettaglio del problema. The message has been infected virus that you send to vigilanza(a)defecondo.com delivery failure: the message has been infected virus. --- Below this line is a report of the anti-virus engine. -------------------------------------- Scan started: Mon Jun 6 15:13:07 2005 35866391/your_document.zip: Worm.SomeFool.P FOUND 35866391/your_document.zip: Removed -- summary -- Known viruses: 35438 Engine version: 0.85 Scanned directories: 1 Scanned files: 2 Infected files: 1 Data scanned: 0.03 MB Time: 3.062 sec (0 m 3 s) --- Below this line is a copy of the message. Received: from unknown (HELO defecondo.com) ([62.10.78.206]) (envelope-sender <distutils-sig(a)python.org>) by [192.168.188.97] with ESMTP for <vigilanza(a)defecondo.com>; Mon, 06 Jun 2005 15:13:06 +0200 From: distutils-sig(a)python.org To: vigilanza(a)defecondo.com Subject: Stolen document Date: Mon, 6 Jun 2005 15:20:32 +0200 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_0016----=_NextPart_000_0016" X-Priority: 3 X-MSMail-Priority: Normal

1 0