Mailman 3 May 2000 - Distutils-SIG

Where to install non-code files
by Greg Ward 29 May '00

29 May '00

Another one for the combined distutils/python-dev braintrust; apologies to those of you on both lists, but this is yet another distutils issue that treads on python-dev territory. The problem is this: some module distributions need to install files other than code (modules, extensions, and scripts). One example close to home is the Distutils; it has a "system config file" and will soon have a stub executable for creating Windows installers. On Windows and Mac OS, clearly these should go somewhere under sys.prefix: this is the directory for all things Python, including third-party module distributions. If Brian Hooper distributes a module "foo" that requires a data file containing character encoding data (yes, this is based on a true story), then the module belongs in (eg.) C:\Python and the data file in (?) C:\Python\Data. (Maybe C:\Python\Data\foo, but that's a minor wrinkle.) Any disagreement so far? Anyways, what's bugging me is where to put these files on Unix. <prefix>/lib/python1.x is *almost* the home for all things Python, but not quite. (Let's ignore platform-specific files for now: they don't count as "miscellaneous data files", which is what I'm mainly concerned with.) Currently, misc. data files are put in <prefix>/share, and the Distutil's config file is searched for in the directory of the distutils package -- ie. site-packages/distutils under 1.5.2 (or ~/lib/python/distutils if that's where you installed it, or ./distutils if you're running from the source directory, etc.). I'm not thrilled with either of these. My inclination is to nominate a directory under <prefix>lib/python1.x for these sort of files: not sure if I want to call it "etc" or "share" or "data" or what, but it would be treading in Python-space. It would break the ability to have a standard library package called "etc" or "share" or "data" or whatever, but dammit it's convenient. Better ideas? Greg -- Greg Ward - "always the quiet one" gward(a)python.net http://starship.python.net/~gward/ I have many CHARTS and DIAGRAMS..

7 10

py_compile and CR in source files
by Greg Ward 27 May '00

27 May '00

Just made an unpleasant discovery: if a Python source file has CR-LF line-endings, you can import it just fine under Unix. But attempting to 'py_compile.compile()' it fails with a SyntaxError at the first line-ending. Arrghh! This means that Distutils will either have to check/convert line-endings at build-time (hey, finally, a good excuse for the "build_py" command), or implicitly compile modules by importing them (instead of using 'py_compile.compile()'). Perhaps I should "build" modules by line-at-a-time copying -- currently it copies them in 16k chunks, which would make it hard to fix line endings. Hmmm. Greg

5 6

Current to-do list
by Greg Ward 27 May '00

27 May '00

I've started maintaining my Distutils to-do list again. In the absence of a proper bug database or issue tracker, this'll have to do. Please add any items I've missed; feel free to claim items for yourself if you can submit a patch. If you're already working on a patch, let me know so I can add you. If you see your name in a "delegated to" comment but you are unwilling/unable to do that task, let me know and I'll take you off. Here's the list... GENERAL ------- * I think fancy_getopt needs to get fancier to properly support the -I, -D, -l, -L, etc. options of "build_ext": need to be able to accumulate multiple options in a list, should be able to split a string on a given delimiter, probably want to specify if repetitions of the same option will accumulate or replace, etc. * do the above options even work at all? seem to recall hearing reports of dismal failure, but I never looked into it [knowing how FancyGetopt works, there's no way these options can work: damn] * think about how to support distribution-specific "configure" commands -- eg. how can they push option values (include/library directories, that sort of thing) onto the "build_ext" command? * should/can the 'finalize_options()' methods be made more robust, so you can sensibly call them multiple times? this might be all that's necessary to enable option-pushing... HELP ---- * interface for "help about some option" -- eg. --help-compiler to build_ext would list available compilers; --help-formats to "sdist" or "bdist" would list available distribution formats; ... [delegated to Rene Liebscher] DOCS ---- * write write write * standard blurb for README files BUILD/INSTALL ------------- EXTENSION BUILDING ------------------ * review and integrate cygwin patch from Rene Liebscher DISTRIBUTIONS ------------- * review/tweak bdist_rpm code from Harry Gebel (minor style issues, fix hardcoded directories) * figure out why bdist_rpm doesn't work with RPM 2.x, and make it work if possible EXTERNAL DEPENDENCIES --------------------- * ??? do we need to implement autoconf in Python (shudder) ??? * if not, how to handle the case of "if Tk present, build with it" in PIL, or "define HAVE_STRFTIME if we have strftime()" in mxDateTime"? -- Greg Ward - maladjusted computer geek gward(a)python.net http://starship.python.net/~gward/ "Passionate hatred can give meaning and purpose to an empty life." -- Eric Hoffer

1 0

Installing config files
by Andrew M. Kuchling 27 May '00

27 May '00

I'm working on packaging the remote microscope software using Distutils, because it'll make my life easier and be a good test case for installing an application. One issue that's just come up is configuration files and upgrading. If someone installed a previous version of the software, you don't want to just blow away their painstakingly customized configuration. Strawman proposal: do this the way RPM does it, marking configuration files as special and making a backup copy. You can list config filenames in the RPM spec file, and on installation the existing version of the file is copied to <name>.rpmsave before the new version of the file is written. The distutils version of this would be: * Add another argument to setup(): config_files = ['spam.cfg', ...] * Add build_config and install_config commands. install_config checks before overwriting an existing config file, and renames the file by adding .backup (suggestions for the extension?). Because of its installation database, RPM can check if the file hasn't been modified from the originally installed version and doesn't bother backing it up if you haven't changed the file. That doesn't seem possible for Distutils, since there's no such database, so it'll just back up all the config files. What do you all think of the idea? -- A.M. Kuchling http://starship.python.net/crew/amk/ "Python? Oh, I've heard of that. I have a friend at the NSA who uses it." -- Overhead at a meeting, quoted in c.l.p on 3 Dec 1998

3 2

New command: "install_headers"
by Greg Ward 27 May '00

27 May '00

Hi all -- just checked in a new command, "install_headers", for installing C/C++ headers. This makes the NumPy setup script a lot smaller, and should work similar wonders for CXX and any other module distribution that needs to install header files. (Commonly needed by extensions whose C code is used by other extensions.) On Unix, header files are installed to $prefix/include/python1.x/$distname> -- eg. for NumPy, $prefix/include/python1.x/Numerical. On Windows, they go in $prefix\Include\$distname, and on Mac OS to $prefix:Include:$distname. Please, give the latest CVS code a whirl -- I'd especially like to hear if anyone else has setup scripts that install header files, and find out if "install_headers" worked for you. Greg -- Greg Ward - just another /P(erl|ython)/ hacker gward(a)python.net http://starship.python.net/~gward/ MTV -- get off the air! -- Dead Kennedys

1 0

Terminology question
by Greg Ward 26 May '00

26 May '00

A question of terminology: frequently in the Distutils docs I need to refer to the package-that-is-not-a-package, ie. the "root" or "empty" package. I can't decide if I prefer "root package", "empty package" or what. ("Empty" just means the *name* is empty, so it's probably not a very good thing to say "empty package" -- but "package with no name" or "unnamed package" aren't much better.) Is there some accepted convention that I have missed? Here's the definition I've just written for the "Distribution Python Modules" manual: \item[root package] the ``package'' that modules not in a package live in. The vast majority of the standard library is in the root package, as are many small, standalone third-party modules that don't belong to a larger module collection. (The root package isn't really a package, since it doesn't have an \file{\_\_init\_\_.py} file. But we have to call it something.) Confusing enough? I thought so... Greg -- Greg Ward - Unix nerd gward(a)python.net http://starship.python.net/~gward/ Beware of altruism. It is based on self-deception, the root of all evil.

4 4

Re: [Distutils] handling complex link/library specifications
by andrew.macintyre＠aba.gov.au 26 May '00

26 May '00

The reason that using "cc" worked and "ld" didn't was that the Sun compiler was silently using a different library directory (/opt/SUNWspro/SC4.2/lib) to what I thought was the correct library directory (/opt/SUNWspro/lib) for ld to use. The latter is just a set of symlinks to the former, but doesn't include links for the static libraries :-(. Once I fed the former to ld, the -B toggles had the desired result (all 4 of those libraries don't get installed on the target system, so static linking them removes a headache). I was then able to use 'extra_link_args' to handle those static libs and the -R option. ------------------------------------------------------------- Andrew MacIntyre \ andrew.macintyre(a)aba.gov.au Planning & Licensing Branch \ Tel: +61 2 6256 2812 Australian Broadcasting Authority \ Fax: +61 2 6253 3277 ---------- Original Text ---------- From: Greg Ward <gward(a)python.net>, on 26/05/00 12:17: On 25 May 2000, Andrew MacIntyre said: > <ever hopeful> > There wouldn't be a convenient way to override the linker command ("ld -G" on > Solaris, from the Python Makefile) would there? > </ever hopeful> I have taken to editing the Makefile installed by Python because of this problem. I also made a patch for the configure script that would fix this for Python 1.6 (use "$(CC) -g" on Solaris if cc=gcc); the only feedback I've had is "it can't make things *worse*". Hardly encouraging, that. But no, there's no way to override what's in Python's Makefile: Distutils takes it at face-value. This should probably be fixed, especially now that we have config files (hooray!). BTW, you mentioned in your followup that you figured out a workaround: care to share it with the rest of us? Greg -- Greg Ward - Linux nerd gward(a)python.net http://starship.python.net/~gward/ Cheops' Law: Nothing *ever* gets built on schedule or within budget. _______________________________________________ Distutils-SIG maillist - Distutils-SIG(a)python.org http://www.python.org/mailman/listinfo/distutils-sig

1 0

Re: [Distutils] handling complex link/library specifications
by andrew.macintyre＠aba.gov.au 26 May '00

26 May '00

'extra_link_args' looks just what I'm after, especially for the "-R<path>" option. Platform portability isn't important to me (yet!) with this particular project. The static/dynamic bits appear to have wrinkles at the compiler/linker level:- when used with "cc" as the linker it works, but it doesn't work with "ld" :-( At least things work without those toggles, albeit with administrative side-effects. <ever hopeful> There wouldn't be a convenient way to override the linker command ("ld -G" on Solaris, from the Python Makefile) would there? </ever hopeful> I have to say that Distutils, from my experience with it so far, has been easy to work with. Apart from my questions here, all my problems with this project have been with the commercial library I'm wrapping. Many thanks for the fine work! ------------------------------------------------------------- Andrew MacIntyre \ andrew.macintyre(a)aba.gov.au Planning & Licensing Branch \ Tel: +61 2 6256 2812 Australian Broadcasting Authority \ Fax: +61 2 6253 3277 ---------- Original Text ---------- From: Greg Ward <gward(a)python.net>, on 25/05/00 12:33: On 24 May 2000, Andrew MacIntyre said: > The recommended library spec (for Solaris, which is what I'm working with at > the moment) looks something like this: > > -l<lib> ... -l<lib> -Bstatic -l<lib>... -l<lib> -Bdynamic -l<lib>...-l<lib> > -R<path> Wow, same question twice in one day. Time to work on the docs again. You should be able to do this with an 'extra_link_args' entry in the "build info" dictionary that describes your extension. *rummage rummage rummage* Ahh, here's the example I sent to someone else not 'alf an 'our ago (I knew I should have cc'd the sig): ext_modules = [('_imaging', { 'sources': ['_imaging.c', 'decode.c', ... , 'path.c'] 'include_dirs': ['libImaging'], 'library_dirs': ['libImaging'], 'libraries': ['Imaging'], 'extra_link_args': ['-lX11', '-lXext', '-lXshm'], 'extra_compile_args': ['-Dfoo=bar', '-Ublah'] } )] 'extra_compile_args' is only in the current CVS code, but 'extra_link_args' has been around for a while. BTW, the requirement in the above example was to worm output from a configure script and from gtk-config into the compiler/linker commands. Obviously portability is thrown out the window, but in return you have a working setup script. > (BTW http://www.python.org/sigs/distutils-sig/doc/dist/source-dist.html has an > HTML problem, at least with IE 3.02) Hmm, based on my (indirect) experience with IE 3, I'd be more inclined to say *it's* the one with an HTML problem... *smirk* Greg -- Greg Ward - Unix geek gward(a)python.net http://starship.python.net/~gward/ I wonder if I ought to tell them about my PREVIOUS LIFE as a COMPLETE STRANGER? _______________________________________________ Distutils-SIG maillist - Distutils-SIG(a)python.org http://www.python.org/mailman/listinfo/distutils-sig

2 1

Thinking about a Windows Installer
by Thomas Heller 26 May '00

26 May '00

Hi, I'm currently thinking about a bdist_winexe command for distutils. (Is there a better name?) This command will create a single executable file similar to those created with Wise or Installshield installers (except that you don't need to have these _commercial_ programs installed). This exe-File consists of a C-program embedding python, appended is a zip-compatible archive of the modules to distribute, the setup.py script, as well as an ini-file "setup.ini" or something like that which contains the distribution meta-data in a format easily readable by the C-program. This exe-File, when run, simply unpacks the archive into a temporary directory, displays a nice GUI, runs the setup.py script, and cleans up the temp-dir. (The design is inspired by previous work: Gordons installer, squeeze and whatever) Comments? Thomas Heller

2 5

Re: [Distutils] handling complex link/library specifications
by andrew.macintyre＠aba.gov.au 25 May '00

25 May '00

scratch the <ever hopeful>...</ever hopeful> bit, I've found an incantation (library directory) that allows the 'extra_link_args' option to do all I need to do. Many thanks! ------------------------------------------------------------- Andrew MacIntyre \ andrew.macintyre(a)aba.gov.au Planning & Licensing Branch \ Tel: +61 2 6256 2812 Australian Broadcasting Authority \ Fax: +61 2 6253 3277 ---------- Original Text ---------- From: Andrew MacIntyre <andrew.macintyre(a)aba.gov.au>, on 25/05/00 14:08: 'extra_link_args' looks just what I'm after, especially for the "-R<path>" option. Platform portability isn't important to me (yet!) with this particular project. The static/dynamic bits appear to have wrinkles at the compiler/linker level:- when used with "cc" as the linker it works, but it doesn't work with "ld" :-( At least things work without those toggles, albeit with administrative side-effects. <ever hopeful> There wouldn't be a convenient way to override the linker command ("ld -G" on Solaris, from the Python Makefile) would there? </ever hopeful> I have to say that Distutils, from my experience with it so far, has been easy to work with. Apart from my questions here, all my problems with this project have been with the commercial library I'm wrapping. Many thanks for the fine work! ------------------------------------------------------------- Andrew MacIntyre \ andrew.macintyre(a)aba.gov.au Planning & Licensing Branch \ Tel: +61 2 6256 2812 Australian Broadcasting Authority \ Fax: +61 2 6253 3277 ---------- Original Text ---------- From: Greg Ward <gward(a)python.net>, on 25/05/00 12:33: On 24 May 2000, Andrew MacIntyre said: > The recommended library spec (for Solaris, which is what I'm working with at > the moment) looks something like this: > > -l<lib> ... -l<lib> -Bstatic -l<lib>... -l<lib> -Bdynamic -l<lib>...-l<lib> > -R<path> Wow, same question twice in one day. Time to work on the docs again. You should be able to do this with an 'extra_link_args' entry in the "build info" dictionary that describes your extension. *rummage rummage rummage* Ahh, here's the example I sent to someone else not 'alf an 'our ago (I knew I should have cc'd the sig): ext_modules = [('_imaging', { 'sources': ['_imaging.c', 'decode.c', ... , 'path.c'] 'include_dirs': ['libImaging'], 'library_dirs': ['libImaging'], 'libraries': ['Imaging'], 'extra_link_args': ['-lX11', '-lXext', '-lXshm'], 'extra_compile_args': ['-Dfoo=bar', '-Ublah'] } )] 'extra_compile_args' is only in the current CVS code, but 'extra_link_args' has been around for a while. BTW, the requirement in the above example was to worm output from a configure script and from gtk-config into the compiler/linker commands. Obviously portability is thrown out the window, but in return you have a working setup script. > (BTW http://www.python.org/sigs/distutils-sig/doc/dist/source-dist.html has an > HTML problem, at least with IE 3.02) Hmm, based on my (indirect) experience with IE 3, I'd be more inclined to say *it's* the one with an HTML problem... *smirk* Greg -- Greg Ward - Unix geek gward(a)python.net http://starship.python.net/~gward/ I wonder if I ought to tell them about my PREVIOUS LIFE as a COMPLETE STRANGER? _______________________________________________ Distutils-SIG maillist - Distutils-SIG(a)python.org http://www.python.org/mailman/listinfo/distutils-sig _______________________________________________ Distutils-SIG maillist - Distutils-SIG(a)python.org http://www.python.org/mailman/listinfo/distutils-sig

1 0