Mailman 3 August 2000 - Distutils-SIG

Re: Opening up backdoors to 'setup()'
by Greg Ward 30 Aug '00

30 Aug '00

[quoth Amos] > So distutils.core.setup(script_name=file, script_args=commands) is the > same as distutils.core.run_setup(file, commands)? No. Passing 'script_name' to 'setup()' is just a "fool yourself" thing; it doesn't actually change the script being run. It's mainly there for completism's sake; if you can override sys.argv[1:], you ought to be able to override sys.argv[0]. Here's a summary of what I'm thinking right now: setup(var=value, ...) - the traditional and usual way to invoke the Distutils - parses sys.argv (unless you override it) and carries out all actions implied by the command line and config files - currently returns nothing -- not much point, because it's all about side-effect, and the intended use is from a setup script, which is run by an installer who doesn't give a whit about the Distribution instance run_setup(file, script_args=sys.argv[1:], stop_after="run") - execfile's the setup script named by 'file' - this causes 'setup()' to be called in the normal way, so that a Distribution instance is created in the normal way, etc. - but some strings are pulled behind the scenes so that we don't necessarily run to completion - and something -- probably the Distribution instance -- is returned, which the caller can than query to discover the state of the distribution (eg. metadata, contents of config files, etc.) > If so, is there any need for a run_setup function? I still think so. The differences are subtle but important. Greg -- Greg Ward - Unix geek gward(a)python.net http://starship.python.net/~gward/ "He's dead, Jim. You get his tricorder and I'll grab his wallet."

3 2

Re: bdist_pkgtool attempt
by Greg Ward 30 Aug '00

30 Aug '00

[quoth Mark Alexander] > SD-UX is Software Distributor for HP-UX. I have the same problem > with the 'd'. I'm not attached to the name, so _hpux or _depot > (that's what an hp package is called) or even just _hp. I'm just > not positive that HP is the only one that uses this format. Then "bdist_sdux" is the right name. There are already approximately 17,000 known installation tools and/or schemes named "depot"; why confused matters? [my loopy idea] > Ahh, gotcha. Idea: don't use Distutils to do the install, but do take > advantage of it for the install-time scripts. Or rather, take advantage > of Python and assume that the Distutils are installed on the target > machine. Then you can do > > from distutils.sysconfig import get_python_inc > > libdir = get_python_lib(plat_specific=1) > ... > > instead of that nasty shell trickery-pokery. [Mark responds skeptically] > Hmm, so I would then invoke distutils 'install' and force it to > that directory? Then I'd either have to force the file inventory > somehow Or just use the Distutils' handy tools for copying files around; see distutils.dir_util.copy_tree(). That's what the "install" command uses. Most of the distutils.*_util modules are intended to make writing shell-style code in Python relatively simple. (Then will someone please explain to me why the Distutils is 6,300+ lines of code... sigh... another great idea shot down by cold, hard reality.) > I tend to favor letting the native tool do as much as possible, > though. Definitely. If "native tools" means including shell code in your installer rather than Python code, so be it. It *is* icky to have the "prefix + lib/pythonX.Y/site-packages" knowledge in 3 places, though... 2 (sysconfig.py and install.py) is bad enough, but just because I slipped up doesn't mean we should let things get worse. > My assumption is that if you're building binaries, your doing it > for people who don't care about distutils or anything else about > how the binary was created. They just want it to work like > everything else they do. Umm, if they want everything to "just work", what on earth are they still using Solaris for? (Note the total absence of a smiley here: I'm dead serious! I used to think Linux was "just another Unix", so adjusting to Solaris at work wouldn't be so bad. I guess I didn't stop to think about what an enormous difference it makes to have "less" installed out-of-the-box... never mind Emacs, GCC, KDE, Gnome, Gtk, Qt, etc. etc.) > This assumption is too painfull for me. When I assumed my current > position, I took over dozens of archaic machines where the disks > where sliced horribly. Of course, I HAD to have python everywhere, > so I built packages that relocated (and linked to /usr/bin and > /usr/local/bin). Because of that experience, I've become a > fanatic advocate of relocatable packages. Ooh, that reminds me: need to check in to the relocatability of Distutils-generated RPMs. This is not a concern for "dumb" built distributions (they *are* dumb, after all), but I see no good reason why our RPMs (and other "smart" distributions) shouldn't at least be relocatable at user preference. (Note that I'm drawing a distinction between "relocatable" -- user specifies the installation prefix -- and "smart enough to figure out where Python is installed, and put itself there". I still don't consider the latter essential for RPMs, but I do for Windows installers. Go figure. Anyone else care either way?) Greg -- Greg Ward - Unix nerd gward(a)python.net http://starship.python.net/~gward/ I have a VISION! It's a RANCID double-FISHWICH on an ENRICHED BUN!!

3 2

re: Opening up backdoors to 'setup()'
by Amos Latteier 29 Aug '00

29 Aug '00

Greg Ward: > On 28 August 2000, Amos Latteier said: > > I've thought about this a little. It seems to me that the basic desire > > is to be able to run setup from python, not the command line. I think we > > could accommodate this with something like this > > > > distutils.core.run_setup(file, commands) > > Hmmm, that does look cleaner for your needs than setting a global mode > variable. (Of course, it might end up setting a global mode variable > behind the scenes, but never mind that. If I break 'setup()' up into > its constituent parts, or farm more of it off to the Distribution class, > that shouldn't be necessary.) > > However, since I just went ahead and implemented 'script_name' and > 'script_args' arguments to 'setup()', this is not needed for Paul's > case. So distutils.core.setup(script_name=file, script_args=commands) is the same as distutils.core.run_setup(file, commands)? If so, is there any need for a run_setup function? > > In my case I'll probably write a new command called something like > > 'metadata' that doesn't do anything but return meta-data about the > > distribution. > > We've been down this road before! The outcome was the > DistributionMetadata class, defined in dist.py. Once you have a > Distribution object, you can query its metadata using > 'get_description()', 'get_fullname()', etc. ... > I think I would spell this: > > dist = distutils.core.run_setup('foo-1.0/setup.py', stop_after="init") > > and then you can do this: > > print "%s (version %s), provided by %s" % > (dist.get_name(), dist.get_version(), dist.get_contact()) > > Seem reasonable? Yes! I don't care about the exact spelling, I just want access to the Distribution object from Python. However... to me run_setup seems different enough from setup to be a different function. Also stop_after=init seems fairly close to a command to me. But I promise to stop quibbling ;-) Whatever you decide is great. Thanks! -Amos -- Amos Latteier mailto:amos@digicool.com Digital Creations http://www.digicool.com

1 0

bdist_pkgtool attempt
by Mark W. Alexander 28 Aug '00

28 Aug '00

Thanks to Harry Henry Gebels bdist_rpm module, I hacked out a bdist_pkgtool module to create binary packages for those of us stuck with Solaris. The module is attached. As a first pass, it takes somewhat of a sledgehammer approach, but it does work for at least a simple pure-python module. I have not tested pre/post-install scripts, and I doubt if they'd work anyway. I did however, include a "request" script which searches for the python installation directory before the package is installed and automatically relocates the module to the site-packages directory on the installation target. The same module package can be installed on machines with python installed in different locations and it will figure out the correct place to go. Let me know of any problems. I'm off to do HP-UX depot format now..... Mark Alexander mwa(a)gate.net

2 4

script_name and script_args
by Greg Ward 28 Aug '00

28 Aug '00

Hi all -- I've just tweaked the Distribution class and 'setup()' function a bit so that you can pass 'script_name' (sys.argv[0]) and 'script_args' (sys.argv[1:]) in as keyword args to 'setup()'. Obviously this would never be used in the "canonical" case of a setup script run by the user from the command-line, but it's a handy back door for whatever trickery Paul is doing in PyFort. (Auto-generated setup scripts, indeed! Well, as long as m4 isn't involved...) Now we just have to deal with Amos' dist-bot needs, and that'll be that for Distutils 0.9.2. Marching ever onwards... Greg -- Greg Ward - Linux weenie gward(a)python.net http://starship.python.net/~gward/ A day without sunshine is like night.

1 0

Opening up backdoors to 'setup()'
by Greg Ward 28 Aug '00

28 Aug '00

Hi all -- in the last couple of weeks, two people (Amos Latteier & Paul Dubois, to be specific) have asked about better programmatic access to the 'setup()' function. The two cases are sort of opposite but closely related: * Amos wants a "dist-bot" that pulls Python module distributions off the net, unpacks them, and extracts their meta-data. Running "python setup.py --name --version ..." through a pipe is an option, but it would be much nicer to import (or execfile) setup.py and query it programmatically. * Paul wants to run a setup script with specific arguments from Python code. Again, he could just spawn "python setup.py ...", or hack sys.argv and call 'setup()' directly. But those are both kind of gross. In both cases, what's needed is a bit of "backdoor" access that skirts around the usual convention: setup script encodes useful information in keyword arguments, which are then passed into the guts of the Distutils by calling 'setup()'. I think Paul's backdoor is easiest: add a special keyword argument, 'argv', to 'setup()', which overrides sys.argv. Or a possible variation on this: call it 'args', and have it override just sys.argv[1:]. Subtle distinction, but important. I'm not sure offhand which is better, but I think it would be excessive to support both. For Amos, I can see a couple ways to do it. I think the cleanest is a global variable that tells 'setup()', when to stop. (It can't be another keyword arg, because you're just execfile'ing any old setup script: you can't get in there and add arguments to the 'setup()' call.) The default, of course, would be to do everything: create the Distribution instance, parse the config files and command line, and run all commands (modulo help options, of course). But you could stop after initialization (Distribution object with just the info from the setup script), after config-file parsing, or after command-line parsing. Of course, there would be a function to set the global variable, so you might do this: distutils.core.stop_after('init') execfile("foo-1.0/setup.py") But that's only half the story: you really want to get your hands on the Distribution instance that 'setup()' created. The obvious answer is another global variable, and another function to read it. Yuck. Or rather, getting yucky; not bad enough to out-and-out refuse it, but I'd like to know if anyone has a better idea before I go and implement this. So: any better ideas out there? Does anyone think these backdoors are useful enough to slip in, or should we make Paul and Amos live with whatever gross hacks they're using currently? (I hate gross hacks.) Greg -- Greg Ward - programmer-at-big gward(a)python.net http://starship.python.net/~gward/ ... I don't like FRANK SINATRA or his CHILDREN.

3 3

New bdist_wininst
by Thomas Heller 28 Aug '00

28 Aug '00

(From the readme file:) This is a new release of the distutils windows installer. The known bug (bogus error message when an empty file is extracted) is fixed. Other changes: - The target-compile and target-optimize flags of bdist_wininst are gone. It is no longer possible to compile the python files during installation. - The zlib module is no longer required or used by bdist_wininst. - I moved the decompression/extraction code into a separate file (extract.c). - The installer stub is now compressed by UPX (see http://upx.tsx.org/). This reduces the size of the exe (and thus the overhead of the final installer program) from 40 kB to 16 kB. - The installer displays a more uptodate user wizard-like user interface, also containing a graphic: Just's Python Powered logo. (I could not convince myself to use one of the BeOpen logos). - The installation progress bar now moves correctly. Greg, the zip-file contains complete files, not diffs. When you check it in, note that archive.h, extract.c, install.c, install.rc, README.txt, resource.h, wininst.dsp, wininst.dsw and bdist_wininst.py are text-files in DOS format. PythonPowered.bmp and wininst.exe are binary files and should be checked in with -kb. (I'm not sure if wininst.exe should be checked in at all) Thomas Heller

2 3

Was I asleep? (Extension instance message)
by Mark W. Alexander 25 Aug '00

25 Aug '00

Distutils 0.9.1 Making setup scripts for all my old modules, I see: warning: build_ext: old-style (ext_name, build_info) tuple found in ext_modules for extension '(module_name)'-- please convert to Extension instance Say what? I don't remember this on the list, and I can't find any docs, so I decipher distutils extension.py and make something like: myExt=Extension('module_name', ['path/source.c'], libraries=['mylib']) and then I get: Traceback (most recent call last): File "setup.py", line 64, in ? ext_modules=prpwdExtension File "/opt/python-1.6b1/lib/python1.6/distutils/core.py", line 112, in setup dist.run_commands () File "/opt/python-1.6b1/lib/python1.6/distutils/dist.py", line 776, in run_commands self.run_command (cmd) File "/opt/python-1.6b1/lib/python1.6/distutils/dist.py", line 797, in run_command cmd_obj.run () File "/opt/python-1.6b1/lib/python1.6/distutils/command/build.py", line 116, in run if self.distribution.has_ext_modules(): File "/opt/python-1.6b1/lib/python1.6/distutils/dist.py", line 807, in has_ext_modules return self.ext_modules and len (self.ext_modules) > 0 AttributeError: 'Extension' instance has no attribute '__len__' I went back to the "old-style" way since it works, but I'd like to buy a clue on how to properly use an Extension instance. Anyone have a link to a doc or a list message somewhere? Thanks, Mark Alexander mwa(a)gate.net

4 7

New code snapshot
by Greg Ward 25 Aug '00

25 Aug '00

Hi all -- I've just created a new code snapshot; this fixes a rather embarassing bug, and also includes Thomas' latest version of the Windows installer code. Windows folks -- please try this snapshot to build PyXML and/or NumPy under Windows! The bug affects any module distribution using "old-style" extension descriptions, ie. the big ugly list-of-dicts-of-tuples instead of Extension instances, and only under Windows. And of course, give bdist_wininst a good exercise. This will become Distutils 0.9.2 if I don't hear any loud squawks. The URL for snapshots is back to python.org: http://www.python.org/sigs/distutils-sig/download.html#snapshot Enjoy! Greg -- Greg Ward - Unix geek gward(a)python.net http://starship.python.net/~gward/ Animals can be driven crazy by placing too many in too small a pen. Homo sapiens is the only animal that voluntarily does this to himself.

1 0

Need help with bdist_wininst bugs
by Greg Ward 24 Aug '00

24 Aug '00

Hi all -- there are a couple of small bugs in the bdist_wininst command that need fixing. Thomas Heller is busy with personal stuff, and I am a fumbling babe in the woods when it comes to hacking on Windows C++ code. So we need help! Anyone care to tackle one of these? The bugs I know about are: * two particular Python version numbers -- "1.5" and "1.6" are hard- coded into the C source file (misc/install.c) for the installer; this list should at least be expanded to include "2.0", and preferably a less yucky solution than hard-coding version numbers can be found * problems (can't remember the details) when including zero-byte files in the archive Part of the solution to the latter is probably a warning in sdist.py that squawks if you have any zero-byte files in your manifest -- it's good practice to always include at least a comment or docstring in __init__.py, and I can't think of any other reason to include empty files. It's known to cause problems on Windows, so should probably just be avoided. Seem reasonable? Thomas (or others), what are the other known problems with bdist_wininst? Thanks -- Greg -- Greg Ward - Linux geek gward(a)python.net http://starship.python.net/~gward/ Speak softly and carry a +6 two-handed sword.

3 3