Mailman 3 October 2006 - Distutils-SIG

Building and using a custom python with buildout
by Michael Twomey Oct. 5, 2006

Oct. 5, 2006

Hi, I've been experimenting with buildout, so far so good. One thing I'd like to do is build and use a python interpreter from source during the buildout process. I want to make my buildouts as self contained as possible, since I'm dealing with machines which might have python 2.3 installed but I want to use 2.4 or 2.5 for my app. Is this possible with buildout? I've built python but I can't figure out how to get buildout to use it: [buildout] develop = zc.recipe.cmmi parts = python mypython [python] recipe = zc.recipe.cmmi url = http://.../Python-2.5.tgz [mypython] recipe = zc.recipe.egg interpreter = mypython eggs = someegg I'm guessing something along the lines of a python var in the buildout section might work: python = ${buildout:parts-directory}/python/bin/python (this doesn't work). python = ${python:?}/bin/python (can't figure out a variable which works). cheers, Michael

1 0

Re: [Distutils] More Windows questions
by Phillip J. Eby Oct. 4, 2006

Oct. 4, 2006

At 09:31 AM 10/4/2006 +0100, Paul Moore wrote: >On 10/4/06, Phillip J. Eby <pje(a)telecommunity.com> wrote: >>Okay, I've looked at the registry a bit, and it seems like I can register >>App Paths either under HKLM (for the whole machine) or under HKCU (for the >>current user). However, both seem to have issues. >[...] >>2. Register HKCU "App Paths" entries for all console scripts, pointing to a >>second '-conio-.exe' wrapper for the same '-script.py', thus allowing >>console scripts to be run from the "Start/Run" command line. > >As I said in a previous email, keep away from App Paths, and in >particular, don't register an executable under any name but its own. > >There's very little documentation that I could find on what App Paths >should do, so all you've got to go on is experiment and looking at >what others do. No-one that I have seen ever registers an exe under a >different name. Actually, I've seen *two* on my own PC; one was a game, the other the Adobe SVG viewer. >And some programs (4NT, for example) expect the App >Path name to match the exe name (and give a broken user experience if >that isn't true). Broken how?

2 1

Re: [Distutils] Draft of new setuptools installation instructions
by Phillip J. Eby Oct. 4, 2006

Oct. 4, 2006

At 09:23 AM 10/4/2006 +0100, Paul Moore wrote: >On 10/4/06, Phillip J. Eby <pje(a)telecommunity.com> wrote: >>At 11:12 PM 10/3/2006 +0100, Paul Moore wrote: >> >Also, the automatic pause would annoy me. How would you make it *not* >> >happen if I ran the command from a console window I already had open? >> >>The idea would be to register a separate "easy_install-win.exe" under >>"easy_install" in App Paths. This version would pause after termination, >>while the normal version would not. Since console programs invoked via the >>App Paths mechanism *always* get their own new window, this should work out >>nicely. > >Sadly, not true. The replacement console program 4NT (which I use >pretty much exclusively) uses App Paths to locate executables. So if >you (in effect) alias easy_install to easy_install-win, you'll break >it for 4NT users. Ah crap. I wonder if there's any way to tell whether you're the only process on a particular console? That would fix this, I guess. There's a GetConsoleProcessList() API, but it doesn't even work on Win2K as far as I can tell; a minimum of XP is required. :( Further Googling shows that many other people with more Windows savvy than me have tried going this path and failed -- apparently the only way to find out who your parent process is is an undocumented function in ntdll. :( >Why not just have the second executable with a simple to type name >(easy_installw, say) and suggest people type *that* in the Run window? >Keep it simple! That doesn't solve the problem for other Python console scripts; It'd be ideal if I could solve this in the .exe launcher itself.

2 1

Python 2.4.3
by Radwan, Neveen Oct. 4, 2006

Oct. 4, 2006

Hello, Can you let me know if there is a way to install this application silently on a system (ie. No dialog boxes) - are there any command line switches I can use? Thanks! Neveen Radwan Neveen Radwan Client Engineering California Engineering Computing (CAEC) - Santa Clara SC11-3-301 408.765.0446 - Office 408.765.7183 - Fax

2 1

Re: [Distutils] Draft of new setuptools installation instructions
by Phillip J. Eby Oct. 4, 2006

Oct. 4, 2006

At 06:48 AM 10/4/2006 -0400, Jim Fulton wrote: >Phillip J. Eby wrote: >>I'm working on new installation instructions for setuptools 0.6c4, which >>will no longer use ez_setup.py for end-user manual installation of >>setuptools (as opposed to bundled installation with another package). > >This seems to me to be a pretty huge change for a c4 release. >Why not get 0.6 final out and do this for 0.7? Because I want 0.6 to be very stable; 0.7 may be a long time coming. The idea is to start making setuptools into *infrastructure*: something you just install like Python, not something that you need an automatic updater for. :) >>I'm looking for any feedback people might have, on the installation >>process itself or on the directions. Some particular areas of >>question/concern: >>* Should ez_setup perhaps launch a web browser pointing to these >>instructions, instead of trying to download setuptools itself? (This >>would bypass most of those annoying proxy problems right off the bat, at >>least when just installing setuptools.) > >Would it be possible, instead to provide a proxy option to >easy_install/ez_setup >(and the underlying setptools apis)? The HTTP_PROXY or http_proxy environment variable controls proxying for Python programs, including easy_install and ez_setup.py. What doesn't work is if they require NTLM authentication, which is not built into urllib. As long as the information is available in the Windows registry or environment, and NTLM auth isn't needed, it should work. But when NTLM auth is required, easy_install isn't going to work unless you use a proxy program like the one at http://ntlmaps.sf.net/ >>* The new install procedure bypasses firewall issues for installing >>setuptools itself, but doesn't do anything about the issue for packages >>that have dependencies. Is there anything else that can be done about that? > >Yes, provide a proxy option, ideally one that can be read from >a configuration file. That's probably not a bad idea, since it would be more discoverable than the environment variable and registry stuff. However, I'll have to tone up my urllib-fu in order to do that, as I have no idea how to set the proxy information from *code* (as opposed to the environment). It also won't help with ez_setup.py, which doesn't have any configuration file abilities or command-line option parsing of its own, although I could perhaps add the necessary arguments to the ez_setup API, since it's going away as a command-line tool anyway.

1 0

Re: [Distutils] Draft of new setuptools installation instructions
by Phillip J. Eby Oct. 4, 2006

Oct. 4, 2006

At 11:12 PM 10/3/2006 +0100, Paul Moore wrote: >Also, the automatic pause would annoy me. How would you make it *not* >happen if I ran the command from a console window I already had open? The idea would be to register a separate "easy_install-win.exe" under "easy_install" in App Paths. This version would pause after termination, while the normal version would not. Since console programs invoked via the App Paths mechanism *always* get their own new window, this should work out nicely.

3 2

Re: [Distutils] Transitioning a Python Library on a Shared Network Drive to eggs/setuptools
by Phillip J. Eby Oct. 4, 2006

Oct. 4, 2006

At 02:08 PM 10/4/2006 -0400, Alexander Michael wrote: >In the past I've managed a shared library of Python packages by using >distutils to install them in secondary Library and Scripts directories on >a shared network drive. This worked fine, even in our multi-platform >environment. With advent of eggs, however, the secondary Library directory >must be a formal "Site Directory" and not just on sys.path. The extra >delay caused by the layer of network has caused simply getting --help for >a simple script to take almost three seconds when it previously only took >a tenth of a second. Some scripts that use many packages installed as eggs >on the network drive can take as many as 8 seconds just to display the >help message. > >I would like to install architecture independent Python packages in a >single shared location so that everyone using that location is >automatically upgraded. The in-house packages are modified about five >times a day on average. I would like to take advantage of setuptools >versioning (thus using the pkg_resources mechanisms) so deprecated >portions of the system can be kept intact in some frozen state of >development without having to include the version number in the package >name explicitly ( i.e. mymod, mymod2, .., mymod42). > >What is the recommended way of using eggs in such an environment? I'm not sure I understand your question. If you want to avoid the overhead due to network latency, you'd have to put the packages on a local drive. If you want to avoid the additional network round-trips being caused by setuptools looking for packages, you'll need to do away with the eggs (e.g. by installing everything using a package manager like RPM or bdist_wininst, etc.). I don't think there is any obvious way of accomplishing what you want without some way to "notice" that a newer version of something is available, yet without using the network. That seems to be a contradiction in terms. The closest thing I know of to what you're doing here is using "setup.py develop" on local revision-control checkouts of shared packages, but that requires that somebody explicitly update changed packages, or at least periodically run a script to do so. If I were in a situation like yours, I would arrange a revision control setup that allows all the subproject trees to be checked out under a common root, and a script to update each tree and rerun "setup.py develop" if any changes occurred, then leave it to the devs to decide when they want to sync. They could also *not* run "develop" (or not sync) packages they didn't want to. I find it hard to imagine using a networked filesystem to import Python code, however, even without eggs being involved, although I've heard rumors that Google does this. If you have to have a networked filesystem, however, I think you'll have to do without versioning, because it adds too many additional network round trips. The only thing I can think of that could work around this would be some sort of client-side caching of egg contents, so that startups can happen faster.

1 0

Transitioning a Python Library on a Shared Network Drive to eggs/setuptools
by Alexander Michael Oct. 4, 2006

Oct. 4, 2006

In the past I've managed a shared library of Python packages by using distutils to install them in secondary Library and Scripts directories on a shared network drive. This worked fine, even in our multi-platform environment. With advent of eggs, however, the secondary Library directory must be a formal "Site Directory" and not just on sys.path. The extra delay caused by the layer of network has caused simply getting --help for a simple script to take almost three seconds when it previously only took a tenth of a second. Some scripts that use many packages installed as eggs on the network drive can take as many as 8 seconds just to display the help message. I would like to install architecture independent Python packages in a single shared location so that everyone using that location is automatically upgraded. The in-house packages are modified about five times a day on average. I would like to take advantage of setuptools versioning (thus using the pkg_resources mechanisms) so deprecated portions of the system can be kept intact in some frozen state of development without having to include the version number in the package name explicitly (i.e. mymod, mymod2, .., mymod42). What is the recommended way of using eggs in such an environment? Your thoughts and advice are appreciated, Alex

1 0

Re: [Distutils] Draft of new setuptools installation instructions
by Phillip J. Eby Oct. 4, 2006

Oct. 4, 2006

At 10:43 PM 10/3/2006 +0100, Paul Moore wrote: >On 10/3/06, Phillip J. Eby <pje(a)telecommunity.com> wrote: >>Should we make more effort to create a usable command-line experience on >>Windows? Perhaps use a post-install script to register easy_install so it >>works from "Start/Run", with an automatic pause to prevent the opened >>console window from closing when it finishes running? Should we include >>optional registration for the ``.egg`` extension so downloading or >>double-clicking an egg installs it? > >IMHO, no. Your particular suggestions would annoy me immensely, Are you sure? Two points here, that I think you're not noticing: 1. Registering an "App Paths" entry for easy_install would simply make it *possible* to type "easy_install" in the Start/Run box and have it work. It wouldn't force you to use it, and seems quite unlikely to conflict with anything else. (It could also be made optional, as per point #2.) 2. I said *optional* registration of the ``.egg`` extension; the idea would be that you'd have a dialog come up at the end of the .win32.exe run, to ask you if you wanted to add it. So, unless you're saying that having the *options* to do these things would annoy you, I don't understand your statement. >>* The new install procedure bypasses firewall issues for installing >>setuptools itself, but doesn't do anything about the issue for packages >>that have dependencies. Is there anything else that can be done about that? > >I've not worked with enough such packages to know the answer to this >without asking, but is there an incantation which says "check >dependencies, if all are present install, otherwise list the missing >dependencies and stop". If that's available, users can manually >download what they need. That is minimal, but effective - in my view, >simplicity is a virtue here. Well, keep in mind that such a thing would only work for *one* level of dependencies; if your dependencies have dependencies you'll go through that repeatedly. But I could perhaps add a --no-deps option to "install" that would do the trick. easy_install already has a --no-deps option, but it currently doesn't *display* the dependency list; it probably should. It also doesn't stop the installation process. One minor issue: dependency checking currently takes place *after* the primary egg is installed, and changing that is too big a refactoring to happen in the remaining time until the 0.6 release. But it's definitely something to think about for 0.7.

3 2

Re: [Distutils] Draft of new setuptools installation instructions
by Jim Fulton Oct. 4, 2006

Oct. 4, 2006

Jim Fulton wrote: ... > This will probably break buildout's bootstrapping script, which uses > ez_setup. I was too hasty in making this remark. The buildout bootstrap script, http://svn.zope.org/zc.buildout/trunk/bootstrap/bootstrap.py?view=markup Uses the use_setup function from ez_setup.py and, as this will still be supported, there is reason to hope that this won't be broken. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

1 0