Mailman 3 September 2000 - Distutils-SIG

New snapshot needs testing
by Greg Ward 18 Sep '00

18 Sep '00

Hi all -- I've just put out a new code snapshot that needs some widespread testing. (More so than usual, I mean.) This is mainly because I tweaked the sysconfig module to parse Python's makefile lazily rather than eagerly, and to store what it finds in a dictionary rather than global variables. This facility of sysconfig was only used in a few places throughout the Distutils, and I'm pretty sure I got all of them -- as usual, It Works For Me (TM). But I just have my one Linux box at home; at least now I test with Python 1.6 as well as 1.5.2. (And I have that enormous RPM of 2.0b1 ready to install one of these days...) So please, download tonight's snapshot and give it a good workout -- build stuff, lots of stuff, on lots of platforms. Please! Oh, a neat feature added today: I devised a brilliant and entirely original new method for describing extensions to build. I call it "the Setup file". This amazingly innovative mechanism lets you describe the extension name, source files, include paths, macros to define/undefine, and a few other goodies in a simple, compact, familiar (to Unix geeks at least) notation. For example, you could describe the "_tkinter" extension as follows: _tkinter _tkinter.c tkappinit.c -DWITH_APPINIT -UFOO \ -I/usr/include \ -I/usr/X11R6/include \ -DWITH_TIX -ltix4.1.8.0 \ -L/usr/lib \ -ltk8.0 -ltcl8.0 \ -L/usr/X11R6/lib \ -lX11 Then, a setup script to build this extension is quite simple: from distutils.core import setup from distutils.extension import read_setup_file exts = read_setup_file("Setup.tk") setup(name = "Tkinter", ext_modules = exts) Hmmm... on second thought, this syntax seems vaguely familiar... where *have* I seen it before? ;-) Greg P.S. yes, it does support comments just like in Modules/Setup! -- Greg Ward gward(a)python.net http://starship.python.net/~gward/

2 2

setup.py sdist hangs on windows with 1.6 and 2.0b1
by Thomas Heller 18 Sep '00

18 Sep '00

This may not be directly related to distutils, it may also be a bug in 1.6 and 2.0b1 re implementation. 'setup.py sdist' with the current distutils CVS version hangs while parsing MANIFEST.in, executing the re.sub command in these lines in text_file.py: # collapse internal whitespace (*after* joining lines!) if self.collapse_ws: line = re.sub (r'(\S)\s+(\S)', r'\1 \2', line) Has anyone else noticed this, or is smething wrong on my side? Thomas

2 1

define_macros ignored?
by Jeremy Hylton 18 Sep '00

18 Sep '00

I am working on a setup script for Tkinter, which requires a -DWITH_APPINIT flag at compile time. The docs suggest that I should add the following definition to my Extension constructor call: define_macros=[('WITH_APPINIT', None),], I did this, but it had no effect. It looks like there is no where that the define_macros attribute of the Extension object is actually used. I assumed this is a bug and tried to figure out where the macros definitions ought to be processed. I'm not familiar with the code, so I may have put it in the wrong place. A patch is included below, which allows _tkinter to build. Jeremy *** build_ext.py Tue Sep 12 20:58:48 2000 --- /home/jeremy/src/python/dist/src/Lib/distutils/command/build_ext.py Thu Sep 7 11:09:18 2000 *************** *** 401,407 **** # any sensible compiler will give precedence to later # command line args. Hence we combine them in order: extra_args = ext.extra_compile_args or [] - macros = ext.define_macros or [] # XXX and if we support CFLAGS, why not CC (compiler # executable), CPPFLAGS (pre-processor options), and LDFLAGS --- 401,406 ---- *************** *** 410,419 **** if os.environ.has_key('CFLAGS'): extra_args.extend(string.split(os.environ['CFLAGS'])) ! objects = self.compiler.compile (sources, output_dir=self.build_temp, ! macros=macros, include_dirs=ext.include_dirs, debug=self.debug, extra_postargs=extra_args) --- 409,418 ---- if os.environ.has_key('CFLAGS'): extra_args.extend(string.split(os.environ['CFLAGS'])) ! objects = self.compiler.compile (sources, output_dir=self.build_temp, ! #macros=macros, include_dirs=ext.include_dirs, debug=self.debug, extra_postargs=extra_args)

2 1

Minor issues
by Bastian Kleineidam 16 Sep '00

16 Sep '00

1) Why is the --formats list comma-separated and the --include-dirs and --library-dirs lists not? They are separated by os.pathsep. Well, at least its documented :) 2) The os.pathsep separator is not documented in the help text: > python setup.py build_ext --help ... --library-dirs (-L) directories to search for external C libraries This should be: ... --library-dirs (-L) colon-separated list of directories to search for external C libraries or whatever os.pathsep is on the current platform. 3) bdist_wininst does not seem to install data files and scripts. If you execute "strings" with the .exe, you will only see Python module files. And I did not found a file with the install log. The installer just copied the files in the Python dir. Bastian

3 4

Re: Distutils under Mac OS
by Greg Ward 16 Sep '00

16 Sep '00

[cc'ing the SIG again, for the same reason] On 13 September 2000, Corran Webster said: > At the moment to install you have to use the command line interface - > you can't just double click on setup.py. What Greg is saying is that > one way to avoid this on the mac side is to have some version of the > bdist command which builds an intelligent installer of some sort (for > example, it may bundle everything up into a VISE installer) which > then puts everything in the right place. That only helps installers -- see my previous post for a possible way to help all Distutils users (developers and packagers too). > I can see two slightly simpler ways of dealing with this. The > easiest to implement is to have an applet which you can just drop > setup.py files on and it handles everything from there (easy to do > thanks to the "run_setup" function - I have a 5 line applet which > does this now). This "Install package" applet could be added to the > standard mac distribution of Python much like the "Build applet" and > "Build application" applets are now. Hmm, neat idea. You could even make your "Install package" applet smart enough to unpack a Distutils-style archive and dig up setup.py there -- just download a source release off the net, drop it onto the applet, and away-we-go. How's *that* for ease of use? ;-) > A slightly more sophisticated approach would be to have a bdist-type > command which creates an installer applet for the package, and then > bundles everything up in a zip/tar/stuffit file. All the end users > would just unpack the archive and run the applet and it would take > care of everything automatically. That might be more sophisticated, but it sounds inferior to the previous option in usability terms: the user has to unpack the archive and "do something" with what he finds there. > An even more sophisticated approach would bundle everything up using > zip/tar/stuffit and then store the data inside a python script > somehow (a really big string?). This script would know how to unpack > and install the archived data. This script would then be turned into > an applet (or even a standalone application) and distributed - > essentially making a pure python poor man's version of VISE. This is essentially what Thomas Heller's bdist_wininst does, except the wrapper is a C program -- really a self-expanding ZIP file with a bit of GUI to make it look pretty. Despite the fact that I have to carry around C source and a Windows .exe in the Distutils CVS and source distribution, this strikes me as more elegant than making a big Python script with a little bit of code and a lot of encoded ZIP/StuffIt/whatever data. > The most sophisticated would be to package everything up in a VISE > Installer. Jack would know far more than I about how > possible/practical that would be. The original idea for Windows was to generate WISE installers. (Any relation to VISE?) The impetus for this has dropped off with the success of the bdist_wininst command; it sounds like not many module distributions need the sophistication of a full-blown commercial installer. I suspect the same would hold on Mac OS. However, when it comes time to deal with full-blown Python applications, we may have to reexamine the issue. Again, I'll deal with header file installation in a separate post. Greg -- Greg Ward gward(a)python.net http://starship.python.net/~gward/

2 1

Re: Distutils on Mac OS
by Greg Ward 15 Sep '00

15 Sep '00

On 14 September 2000, Jack Jansen said: > is there a reason that simply starting setup.py doesn't do an install? > That may be the easiets way to go...

1 0

Re: Distutils documentation Mac mods
by Greg Ward 15 Sep '00

15 Sep '00

On 13 September 2000, Jack Jansen said: > Greg, > I've cc-d Corran on this reply, it's a bit silly to keep two conversations > going with similar content. Don't worry, we'll keep our esoteric Mac-talk to > ourselves:-) And I'm bringing the whole SIG in on this -- surely you two aren't the only Mac hackers in the world! [I opined:] > I like the model we've used for Windows installers: developers and > packagers have to put up with a command line interface -- ie. open a > "DOS box" and run "python setup.py bdist_wininst" -- but then end users > have a nice little Windows-friendly executable installer. They're not > even aware of such a thing as the Distutils, which is how it should be > for end-users. [Jack replies] > Ok, that's fine. From reading the document I got the impression that the > various command-line switches to install (in the end-user situation) would > actually be necessary relatively often, but you're saying that they're rare, > and that the end user will normally just do "python setup.py". Right? Not quite. You're right that the options to install will not be needed very often, especially on single-user systems with no file permissions. But the normal case is "python setup.py install" -- you CANNOT use the Distutils without providing at least one argument to the setup script. The rationale for this reveals my deep-seated Unix geek prejudice: how am I supposed to find out what this program does? Well, I run it without arguments and see what it says. As long as it's not a filter (remember trying to learn how to use grep before you understood what a filter was?), you're fine. Well-behaved command-line programs give you a usage summary when run with bad args or no args, and any Distutils setup script automatically plays by those rules. This is bad on Mac OS, where providing *any* command-line args is a pain. However, I have a cunning plan... When a setup script is run under Mac OS, instead of printing a usage summary to stdout and exiting, throw up a dialog with the user's options -- ie. what commands they can run. You can get a barebones list of commands from distutils.command.__all__, and a list of all commands (including non-standard commands provided by the current setup script) by poking around in the Distribution object. See 'print_commands()' in dist.py for an example, and 'print_command_list()' for how to get the description of each command in turn. (The command module has to be imported, which is why running "setup.py --help-commands" is slow and chunky: it has to import an entire module for each line of help text!) However, the barebones list is probably sub-optimal; you'd want to shuffle it around a bit so "install", "build", "clean", "sdist", and "bdist" are up at the top. Then the sub-commands can come below, grouped as they already are (which is a function of the ordering of distutils.command.__all__). This reshuffling is purely a usability issue; 90% of users will just run "install" 90% of the time, so it should be first, and checked by default. Eg. here's a rough sketch of the dialog I have in mind: +-------------------------------------------------------------+ | (x) install install everything from build directory | | ( ) build build everything needed to install | | ( ) clean clean up output of 'build' command | | ( ) sdist create a source distribution ... | | ( ) bdist create a built (binary) distribution | | ----------------------------------------------------------- | | ( ) build_py ... | | ( ) build_ext ... | | ( ) build_clib ... | | ( ) build_scripts ... | | ( ) install_lib ... | | ( ) install_headers | | ( ) ... | | ( ) bdist_dumb | | ( ) ... | | | | [ Ok ] [ Cancel ] | +-------------------------------------------------------------+ (Memo to myself: if this ordering makes more sense in a GUI dialog, it makes more sense in the output of --help-commands. Hmmm.) I *think* the commands should be radiobuttons, even though you can run multiple commands in one go, because of the need to associate command-line options with each command. Hmmm, maybe not. If you pick multiple commands *and* choose to "Supply additional command-line arguments" (this would probably have to be another checkbox, off by default), then you could get a dialog with space for multiple argument lists: Arguments for: build_clib ____________________________ install_headers ____________________________ if the user selected to run "build_clib" and "install_headers". You might want to use distutils.util.split_quoted() to parse those argument strings into words, in case -- gasp! -- people use directories with spaces in them. (Which will have to be quoted -- ugh! I can just see Joe Mac User saying, "Whaddya mean, the standard shell quoting convention?!??!") Of course, if the user doesn't check the "Supply additional command-line arguments" dialog, then the requested command(s) immediately run. stdout should be preserved and presented to the user in a big scrolling window -- yeah it's ugly, yeah it'll look like someone took a big ol' Unix program and ported to Mac OS as quickly and cheaply as possible. But that's what (will have) happened! So whaddya think? How outrageously horrible and Unix-oid is that? I figure it's not too bad as long as the user doesn't want to supply any extra args, and I don't think that'll be necessary for most end-user installation situations. > Mac installers don't have to worry about HKEY_USER and HKEY_MACHINE and the > difference between OS version X and OS version Y and such: there's always a > single place to put things, and if that somehow changes "the old way" > continues to work. Usually through Apple-supplied magic, sometimes through > VISE-supplied magic. What, you mean you have a *sensible* operating system? I thought they were banned by the secret Microsoft World Domination Act of 1994 (passed by the hidden One World Government, of course). (*wink*... I think.) The issue of where to install header files affects other platforms as well, so I think I'll bring that up in a new thread on the SIG. Greg -- Greg Ward gward(a)python.net http://starship.python.net/~gward/

1 0

BeOS library location?
by Greg Ward 15 Sep '00

15 Sep '00

Hi all -- just noticed the BeOS-specific code in sysconfig._init_posix(): g['LDSHARED'] = ("%s -L%s/lib -lpython%s" % (linkerscript, PREFIX, sys.version[0:3])) Should that be EXEC_PREFIX? I dunno in one would ever do a prefix != exec-prefix installation on BeOS, but it seems like good practice to respect the "exec-prefix for binaries" convention in any event. Greg -- Greg Ward gward(a)python.net http://starship.python.net/~gward/

1 0

Path caching when running multiple commands
by A.M. Kuchling 14 Sep '00

14 Sep '00

I've written a script to apply a command across a bunch of setup.py scripts in a subdirectory, and found a problem: distutils.dir_utils._path_cache stores a list of directories that have already been created. The directory name is just stored as 'dist', not a full path name, so if two subdirectories need to create a 'dist' directory, the second one thinks it's already been created, and dies. Fix: storing full path names in the cache seems reasonable. (The setup.py script is included below, since it may be instructive; feel free to use it as an example.) --amk # Top-level script for running a command-line over all the subpackages import sys, os, glob from distutils.core import run_setup files = os.listdir('.') dirs = filter(os.path.isdir, files) packages = [] for dir in dirs: if os.path.exists( os.path.join( dir, 'setup.py') ): packages.append( dir ) top_dir = os.getcwd() for package_dir in packages: print 'Running', package_dir os.chdir( package_dir ) dist = run_setup('setup.py', script_args=sys.argv[1:], stop_after="run") os.chdir( top_dir )

1 0

bdist_wininst and data files
by Bastian Kleineidam 13 Sep '00

13 Sep '00

Hello, I just discovered the "bug" which prevents bdist_wininst from installing data files and scripts. bdist_wininst.py:109: root_dir = install.install_lib I noticed this comment in the source: # XXX hack! Our archive MUST be relative to sys.prefix # XXX What about .install_data, .install_scripts, ...? So what about data and scripts? Why dont they get in the archive? Bastian

2 1

Distutils-SIG September 2000