Mailman 3 March 2006 - Distutils-SIG

Revision control plugin support for setuptools
by Phillip J. Eby March 29, 2006

March 29, 2006

I've just checked in support for revision control plugins in setuptools. This means that folks using any of the various revision control systems out there other than CVS and SVN can now write a plugin for setuptools to support automatically finding data and source files when building packages and their source distributions. Please see setuptools.txt in the source for documentation, and let me know if you have any questions or problems.

1 0

setuptools in fink
by Kurt Schwehr March 29, 2006

March 29, 2006

Hi All, I am working on getting setup tools into fink. If anyone has time, please take a look at the setup file and let me know what you think. http://schwehr.org/software/fink/setuptools-py.info Once setuptools is installed, I am having trouble with SQLAlchemy which I am using as my test case to see if setuptools is installed correctly. Clearly I am doing something wrong. Can someone please set me straight? I am working with the 10.4-transitional tree within fink. Thanks! -kurt http://schwehr.org/software/fink/sqlalchemy-py.info This makes me think setup tools is ok since it is in the default site-packages: & /sw/bin/python2.4 Python 2.4.2 (#1, Feb 24 2006, 17:05:21) [GCC 4.0.1 (Apple Computer, Inc. build 5247)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from ez_setup import use_setuptools But then the SQLAlchemy build tries to pull down setuptools, which I don't want it to do: CC=gcc-3.3 /sw/bin/python2.4 setup.py build --------------------------------------------------------------------------- This script requires setuptools version 0.6a5 to run (even to display help). I will attempt to download it for you (from http://cheeseshop.python.org/packages/2.4/s/setuptools/), but you may need to enable firewall access for this script first. I will start the download in 15 seconds. ---------------------------------------------------------------------------

1 0

Re: [Distutils] easy_install: bugs with pathing and multi-version
by Phillip J. Eby March 28, 2006

March 28, 2006

At 09:23 AM 3/24/2006 -0500, Mars wrote: >The new egg file definiton is used to open the egg in the install directory. >However, the egg that the install script finds there is a copy of the >orginal temporary egg file, and the old file offsets do not match the new >definition. Thus a ZipImportError is raised. It sounds like easy_install needs to clear the zipimport directory cache, or at least remove any entry corresponding to the file it's installing, in the case where it's overwriting an existing file. As for the SOURCES.txt problem, it seems to be a quirk of the way the distutils FileList class works, such that if a file doesn't exist before SOURCES.txt is built, it doesn't end up listed in SOURCES.txt. So, I'll have to add a workaround for that as well. Thanks for taking the time to follow up on this.

2 2

Re: [Distutils] Breakage of fetch in setuptools-0.6a10-py2.4.egg
by Phillip J. Eby March 25, 2006

March 25, 2006

At 05:32 PM 3/25/2006 -0800, Grig Gheorghiu wrote: >Hi, Philip > >2 questions: > >1. Is there any chance you can release a new setuptools version that >includes the fix for the fetch breakage? I expect to release that version within a week. Nothing stops you, however, from using intermediate snapshots. Run this command to create an egg in the current directory that's a snapshot of the current SVN version: easy_install -zmaxd. setuptools==dev You can then distribute this egg with your package if needed, by passing the correct version info to the ez_setup() function in ez_setup.py (so it uses your copy of the snapshot. >2. Another thing that seems to be broken, and that I was depending on >in cheesecake, is installing a package to a non-default location via >the --home option. > >I'm not sure when this started to break in terms of setuptools, but it >used to work at least a couple of months ago. Now I get something >similar to this when I try to run setuptools-based setup.py scripts >with --home: > >================ >[ggheo@concord twill-0.8.4]$ python setup.py install >--home=/tmp/install_twill/ >running install >Checking .pth file support in /tmp/install_twill//lib/python/ >error: can't create or remove files in install directory > >The following error occurred while trying to add or remove files in the >installation directory: > > [Errno 2] No such file or directory: >'/tmp/install_twill//lib/python/test-easy-install-12889.pth' > >The installation directory you specified (via --install-dir, --prefix, >or >the distutils default setting) was: > > /tmp/install_twill//lib/python/ > >This directory does not currently exist. Please create it and try >again, or >choose a different installation directory (using the -d or >--install-dir >option). >================ > >Is there a better/less-prone-to-breakage way to test whether a package >can be installed to an alternate location, other than using --home? I >want this to work with distutils-based versions of setup.py too. If you use --root instead of --home, it will work nicely with both, assuming you use the current SVN version of setuptools. When setuptools' "install" command sees the --root option, it doesn't try to verify that the installation location is valid. This should be ideal for your purposes, although you may have to fiddle with some of the other options a little. --root also tells setuptools to use its "legacy installation" format, which will more closely mimic what the distutils would do in the same circumstances.

1 0

Re: [Distutils] easy_install: bugs with pathing and multi-version
by Mars March 24, 2006

March 24, 2006

> From: pje(a)telecommunity.com (Phillip J. Eby) > Date: 3/17/2006 10:07 PM (Eastern Standard Time) > To: mfogels(a)interactdirect.com (Maris Fogels) > Cc: distutils-sig(a)python.org > Subject: Re: easy_install: bugs with pathing and multi-version > > At 10:11 PM 3/16/2006 +0000, Maris Fogels wrote: > >zipimport.ZipImportError: bad local file header in > >/data/myproject/myproject-1.0-py2.3.egg > > > >If I re-run the 'easy_install -m .' command it works properly. > >Everything in the installed .egg file appears to work correctly, so I > >could not figure out why a ZipImportError was being raised by a > >working .egg. > > Me either. The only time I've seen anything like that happen is with > Python 2.3 on a 64-bit platform. If you can figure out how to reproduce > it, please let me know. > Thankfully the problem is trivial for me to reproduce, and I was able to run the script through pdb. I believe I have isolated the cause of the problem, but I have not had the time to devise a solution. The problem starts with a normal installation, in a clean directory, using the command 'easy_install .'. There is a custom setup.cfg file there that specifies the install-directory. During installation a temporary egg file is created, along with the projectname.egg-info directory. The temporary egg is copied into the installation directory and renamed. I noted that the newly created SOURCES.txt manifest on my system does not contain an entry for itself. But the [incorrect] manifest file is properly wrapped into the egg. To uninstall, we run the command 'easy_install -m .'. This creates a new egg file, along with a new manifest. This new manifest does contain a reference to itself. This changes the egg file size and the contained zip-file offsets. At this point we have two different egg files and two different egg file definitions, with different zip-file offsets. The new egg file definiton is used to open the egg in the install directory. However, the egg that the install script finds there is a copy of the orginal temporary egg file, and the old file offsets do not match the new definition. Thus a ZipImportError is raised. I was under the impression that a fix to the class setuptools.command.egg_info.manifest_maker was in order, but the code already ensures that the manifest contains the needed self-reference. A run through the debugger should either verify this or pick up the bug. Maris

1 0

Re: [Distutils] entry point for new installations?
by Phillip J. Eby March 20, 2006

March 20, 2006

At 05:26 PM 3/20/2006 -0500, Kevin Dangoor wrote: >On 3/20/06, Phillip J. Eby <pje(a)telecommunity.com> wrote: > > An entry point won't really help you much here, because it would have to > > know how to perform this notification. Presumably, it would either have to > > talk to a socket or modify a file somewhere. > >The egg that has that entry point would have to know how to find its >configuration file to know what to notify, yes. configuration file*s* - you might have multiple copies of something. And Python packages don't really (i.e. shouldn't) own configuration files of their own. > > However, if you're going to have to monitor for file changes anyway, then > > you might as well just monitor sys.path directories for new .egg files, or > > monitor changes to easy-install.pth files. > >My thought was that no file monitoring would be required. easy_install >would keep track of the eggs it installed and then pass that list >along to everything that defines that entry point. Yeah, except that your approach effectively requires building an egg for every *instance* of every app on the box, and also requires you to have a privileged process running an individual user's code. I know that you probably have a rather more restricted deployment scenario in mind, I'm just pointing out that the approach you were proposing falls apart as soon as multiple instances or users with different privileges are involved. (Heck, even my SIGHUP approach has some issues with that, since a high-privilege installer could then be tricked into signalling some other process.) > > The only other thing I can think of that would work is if you had server > > processes write their process IDs some place, and then the hypothetical > > entry point could send them SIGHUPs or something. But monitoring > > file/directory changes is a bit more cross-platform, doesn't require adding > > any entry point code, is probably part of existing reload code, and doesn't > > require a pidfile arena, with all the associated permissions and cleanup > > headaches thereof. > >The entry point is probably a premature optimization. File >modification time lookups are pretty cheap. And, if it's only watching >easy-install.pth, that would be dirt cheap. Yeah, and this latter exchange has convinced me that it's the only simple way to avoid privilege escalation attacks. That is, it doesn't require low/medium-privilege server processes to be able to write stuff (like entry point code, pids, or configuration files) that gets read or executed (or otherwise relied upon) by the high-privilege installation process.

2 1

easy_install2.3, easy_install2.4 etc.?
by John J Lee March 20, 2006

March 20, 2006

Seems it would be nice to have the easy_install bootstrap installer create easy_installX.Y links for each version X.Y of Python. As it is, you have to fiddle with shell backticks, or type the full path to easy_install if on Windows. John

2 1

Re: [Distutils] entry point for new installations?
by Phillip J. Eby March 20, 2006

March 20, 2006

At 04:02 PM 3/20/2006 -0500, Kevin Dangoor wrote: >Someone just prodded me about automatic application reloading, which >is always a hairy topic. The autoreload code that many people run in >development mode forks to run the actual application in a child >process, and everything in sys.modules is watched for a change. If one >of the files change, the child exits and the parent process fires off >a new one. It's ugly, but it's reliable and is generally fine for >development. > >For TurboGears apps, I'm of the opinion that the right way to deploy >an app to production is to turn it into an egg and install it. I had >the (not entirely fleshed out) thought that it might be interesting if >there was an entry_point that got called when new eggs were installed. >This would make it possible to notify a running server that it needs >to be restarted. > >Thoughts? An entry point won't really help you much here, because it would have to know how to perform this notification. Presumably, it would either have to talk to a socket or modify a file somewhere. However, if you're going to have to monitor for file changes anyway, then you might as well just monitor sys.path directories for new .egg files, or monitor changes to easy-install.pth files. The only other thing I can think of that would work is if you had server processes write their process IDs some place, and then the hypothetical entry point could send them SIGHUPs or something. But monitoring file/directory changes is a bit more cross-platform, doesn't require adding any entry point code, is probably part of existing reload code, and doesn't require a pidfile arena, with all the associated permissions and cleanup headaches thereof.

2 1

entry point for new installations?
by Kevin Dangoor March 20, 2006

March 20, 2006

Someone just prodded me about automatic application reloading, which is always a hairy topic. The autoreload code that many people run in development mode forks to run the actual application in a child process, and everything in sys.modules is watched for a change. If one of the files change, the child exits and the parent process fires off a new one. It's ugly, but it's reliable and is generally fine for development. For TurboGears apps, I'm of the opinion that the right way to deploy an app to production is to turn it into an egg and install it. I had the (not entirely fleshed out) thought that it might be interesting if there was an entry_point that got called when new eggs were installed. This would make it possible to notify a running server that it needs to be restarted. Thoughts? Kevin -- Kevin Dangoor Author of the Zesty News RSS newsreader email: kid(a)blazingthings.com company: http://www.BlazingThings.com blog: http://www.BlueSkyOnMars.com

1 0

Re: [Distutils] easy_install: bugs with pathing and multi-version
by Phillip J. Eby March 17, 2006

March 17, 2006

At 10:11 PM 3/16/2006 +0000, Maris Fogels wrote: >All of our applications are installed in a flat structure; an >application directory is one big file-soup. The applications are so >large that we do not have the budget to correct them, so I must >emulate this flat structure in the installation target directory. The >emulations requires the use of a setup.cfg and the distutils '$base' >varible: > ># example setup.cfg >[install] >install-base=/data/myproject >install-lib=$base >install-scripts=$base >install-headers=$base >install-data=$base Note that easy_install only has the equivalent of an install_lib and install_scripts; it doesn't install headers or any data other than package data embedded in eggs. >1. I unpacked the project sources in ~/src/myproject-1.0. When I tried to >use easy_install from the ~/src directory it ignores the install-base >directory >setting specified in setup.cfg: By design, easy_install only loads its own settings from ./setup.cfg. Any setup scripts it runs will read their own setup.cfg files, but since easy_install only runs "setup.py bdist_egg" on its targets, installation-related settings have no effect. This is intentional, since it's not good practice for a distributed setup.cfg to customize the install location. If easy_install honored those settings in the general case, it could result in quite a bit of chaos. >The command works if I cd into the project directory itself, or if I >specify the install directory with '-d' Yes, those are the two ways you can do that. >2. After successfully installing I attempted to uninstall the project >as per the instructions on the EasyInstall wiki page, by changing the >installed files to use a multi-install. It produces the following error: > >$ cd ~/src/myproject-1.0 >$ easy_install -m . I suggest using "easy_install -m myproject", as this does not need to run a setup script. Instead, it will simply detect that 'myproject' is already installed and remove it from sys.path. >zipimport.ZipImportError: bad local file header in >/data/myproject/myproject-1.0-py2.3.egg > >If I re-run the 'easy_install -m .' command it works properly. >Everything in the installed .egg file appears to work correctly, so I >could not figure out why a ZipImportError was being raised by a >working .egg. Me either. The only time I've seen anything like that happen is with Python 2.3 on a 64-bit platform. If you can figure out how to reproduce it, please let me know. >3. This may be a problem with distutils or setuptools, depending on >how the install command is dispatched. I thought it would be nice to put the >install record in the same directory I installed the other files in, >so I appended the following to setup.cfg: > >record=$base/uninstall.dat > >Alas, $base is not expanded for the 'record' configuration option. I think this feature can be added by simply inlcuding 'record' in the list of variables that are expanded in line 125 of setuptools/command/easy_install.py. If you have a chance to try that, please let me know if it works for you, and then I'll add it to the official version.

1 0