Mailman 3 February 2000 - Distutils-SIG

Stylistic comments
by Greg Ward 11 Feb '00

11 Feb '00

Hi all (especially Robin and Thomas!) -- I was just looking through the new registry-grovelling code in msvccompiler.py, making some stylistic changes, when I noticed the following in '_find_exe()' (comments added by me): try: for p in string.split(os.environ['Path'],';'): fn=os.path.join(os.path.abspath(p),exe) if os.path.isfile(fn): return fn # XXX BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD BAD !!!!!!!!!!!!!!!! except: # XXX WHAT'S BEING CAUGHT HERE?!?!? pass What is this 'except' clause supposed to catch? Other changes: I have renamed '_devstudio_versions()' to 'get_devstudio_versions()' because it's more descriptive and I don't see a big need for the leading underscore; likewise '_msvc_get_paths()' is now 'get_msvc_paths()'. I've changed the 'get_devstudio_versions()' docstring to: """Get list of devstudio versions from the Windows registry. Return a list of strings (???) containing version numbers; the list will be empty if we were unable to access the registry (eg. couldn't import a registry-access module) or the appropriate registry keys weren't found. (XXX is this correct???)""" Input, please? I'd like a similarly explicit and useful docstring for 'get_msvc_paths()', but don't understand it well enough to write one: could someone supply me one please? Oh yeah, I'm mystified by the purpose for '_find_SET()' and '_do_SET()'. Could I get docstrings for those please? Better parameter names (full words, not single characters!) might help. Also, the real reason I went off on this tangent was because we get an even *worse* traceback than before now on machines without MSVC++ (eg. my Linux box being told to use an MSVCCompiler object, which used to work just fine in dry-run mode). In particular, it blows up because 'get_devstudio_versions()' returns None, but '_find_exe()' expects a sequence. My inclination is to fix 'get_devstudio_versions()' to always return a list, empty if the imports failed. Ditto for 'get_msvc_paths()' for consistency. Anyone see a problem with this? (I.e., are we ever going to need to distinguish "couldn't find anything in the registry" from "couldn't access the registry at all"?) Thanks -- Greg -- Greg Ward - Linux geek gward(a)python.net http://starship.python.net/~gward/ "He's dead, Jim. You get his tricorder and I'll grab his wallet."

3 4

Package-meta-info-ala-ConfigParser patch
by Michael Muller 11 Feb '00

11 Feb '00

Finally managed to scrape together enough spare time to complete this... This is my original meta-info patch with the pkginfo module altered to read and write ConfigParser style files. It was more difficult to do than I expected, largely because it needs to deal with some complex data structures - I think I might as well have done a complete parser from scratch. However, I must admit, the RFC-822 style is even more readable than the original python data structures. Enjoy. ============================================================================= michaelMuller = mmuller(a)enduden.com | http://www.cloud9.net/~proteus ----------------------------------------------------------------------------- Government, like dress, is the badge of lost innocence; the palaces of kings are built on the ruins of the bowers of paradise. - Thomas Paine =============================================================================

1 0

patch for msvccompiler.py and build_ext.py, debug flag
by Thomas Heller 10 Feb '00

10 Feb '00

This patch against the current CVS version implements the --debug flag. Also I've included the def-file kludge again, because I still think if a def-file is present, it should be used, but I would still like to hear opinions from more windows people here. Other notes: - The python15.lib is no longer set in msvccompiler - We can still argue about the debug-flags to be used I've tested this (with and without def-files, with and without the debug flag) on numpy-15.2. It builds nicely in normal and debug mode, and testwin.bat completes successfully. Thomas Heller PS: I'm still new to diff. If I've done somthing wrong, please complain! diff -cr c:\distutils\distutils/command/build_ext.py distutils/command/build_ext.py *** c:\distutils\distutils/command/build_ext.py Wed Feb 9 09:10:32 2000 --- distutils/command/build_ext.py Wed Feb 9 11:54:04 2000 *************** *** 241,249 **** libraries = build_info.get ('libraries') library_dirs = build_info.get ('library_dirs') extra_args = build_info.get ('extra_link_args') or [] if self.compiler.compiler_type == 'msvc': ! mod_name = string.split (extension_name, '.')[-1] ! extra_args.append ('/export:init%s' % mod_name) ext_filename = self.extension_filename \ (extension_name, self.package) --- 241,263 ---- libraries = build_info.get ('libraries') library_dirs = build_info.get ('library_dirs') extra_args = build_info.get ('extra_link_args') or [] + if self.compiler.compiler_type == 'msvc': ! def_file = build_info.get ('def_file') ! if def_file is None: ! source_dir = os.path.dirname (sources[0]) ! ext_base = (string.split (extension_name, '.'))[-1] ! def_file = os.path.join (source_dir, "%s.def" % ext_base) ! if not os.path.exists (def_file): ! ## self.warn ("file '%s' not found: " % def_file + ! ## "might have problems building DLL") ! def_file = None ! ! if def_file is not None: ! extra_args.append ('/DEF:' + def_file) ! else: ! modname = string.split (extension_name, '.')[-1] ! extra_args.append('/export:init%s'%modname) ext_filename = self.extension_filename \ (extension_name, self.package) diff -cr c:\distutils\distutils/msvccompiler.py distutils/msvccompiler.py *** c:\distutils\distutils/msvccompiler.py Wed Feb 9 09:10:32 2000 --- distutils/msvccompiler.py Wed Feb 9 11:47:16 2000 *************** *** 114,123 **** CCompiler.__init__ (self, verbose, dry_run, force) - # XXX This is a nasty dependency to add on something otherwise - # pretty clean. move it to build_ext under an nt specific part. - # shared libraries need to link against python15.lib - self.add_library ( "python" + sys.version[0] + sys.version[2] ) self.add_library_dir( os.path.join( sys.exec_prefix, 'libs' ) ) self.cc = _find_exe("cl.exe") --- 114,119 ---- *************** *** 133,140 **** --- 129,142 ---- os.environ['path'] = string.join(path,';') self.preprocess_options = None self.compile_options = [ '/nologo', '/Ox', '/MD' ] + self.compile_options_debug = [ + '/nologo', '/Od', '/MDd', '/Z7', '/D_DEBUG' + ] self.ldflags_shared = ['/DLL', '/nologo', '/INCREMENTAL:NO'] + self.ldflags_shared_debug = [ + '/DLL', '/nologo', '/INCREMENTAL:no', '/pdb:None', '/DEBUG' + ] self.ldflags_static = [ '/nologo'] *************** *** 176,181 **** --- 178,188 ---- base_pp_opts.append('/c') + if debug: + compile_options = self.compile_options_debug + else: + compile_options = self.compile_options + for srcFile in sources: base,ext = os.path.splitext(srcFile) objFile = base + ".obj" *************** *** 188,198 **** inputOpt = fileOpt + srcFile outputOpt = "/Fo" + objFile ! cc_args = self.compile_options + \ base_pp_opts + \ [outputOpt, inputOpt] ! if debug: ! pass # XXX what goes here? if extra_preargs: cc_args[:0] = extra_preargs if extra_postargs: --- 195,204 ---- inputOpt = fileOpt + srcFile outputOpt = "/Fo" + objFile ! cc_args = compile_options + \ base_pp_opts + \ [outputOpt, inputOpt] ! if extra_preargs: cc_args[:0] = extra_preargs if extra_postargs: *************** *** 250,257 **** # XXX should we sanity check the library name? (eg. no # slashes) ! self.link_shared_object (objects, ! self.shared_library_name(output_libname)) def link_shared_object (self, objects, --- 256,270 ---- # XXX should we sanity check the library name? (eg. no # slashes) ! self.link_shared_object (objects=objects, ! output_libname=\ ! self.shared_library_name(output_libname), ! output_dir=output_dir, ! libraries=libraries, ! library_dirs=library_dirs, ! debug=debug, ! extra_preargs=extra_preargs, ! extra_postargs=extra_postargs) def link_shared_object (self, objects, *************** *** 259,264 **** --- 272,278 ---- output_dir=None, libraries=None, library_dirs=None, + debug=0, extra_preargs=None, extra_postargs=None): """Link a bunch of stuff together to create a shared object *************** *** 273,282 **** self.library_dirs + library_dirs, self.libraries + libraries) - ld_args = self.ldflags_shared + lib_opts + \ - objects + ['/OUT:' + output_filename] if debug: ! pass # XXX what goes here? if extra_preargs: ld_args[:0] = extra_preargs if extra_postargs: --- 287,304 ---- self.library_dirs + library_dirs, self.libraries + libraries) if debug: ! ldflags = self.ldflags_shared_debug ! basename, ext = os.path.splitext (output_filename) ! #XXX not sure this belongs here ! # extensions in debug_mode are named 'module_d.pyd' ! output_filename = basename + '_d' + ext ! else: ! ldflags = self.ldflags_shared ! ! ld_args = ldflags + lib_opts + \ ! objects + ['/OUT:' + output_filename] ! if extra_preargs: ld_args[:0] = extra_preargs if extra_postargs:

2 1

Compiling / Installing extensions
by Thomas Heller 09 Feb '00

09 Feb '00

It seems, with Robins latest patch compiling extensions under windows goes smoothly. Some points are still nonstandard: 1. Usually extensions are named xxx.pyd, not xxx.dll Should this be changed? 2. install copies all files built: not only the dll but also .exp, .lib, everthing which is output by the linker. Is this also the case on unix? Or is my setup.py wrong? 3. It would really be nice if there were a possibility to compile debug versions. Thomas Heller

5 27

Release mechanics and other bureaucracy
by Greg Ward 09 Feb '00

09 Feb '00

Hi all -- those of you who were at IPC8 (or follow python-dev) are probably aware of the suddenly accelerated release schedule for Python 1.6. The major consequence of this for me (and this SIG) is that Distutils must be in full working order *very* soon now, or I will have a big Dutch guy named Guido out for my code (yipes!). Consequently, I have to spend as much time as possible writing a) code, b) developer documentation, c) user documentation, and d) test suites. You'll note that doing releases doesn't fall anywhere on that list. This means that no matter how many bugs are fixed, or however many new features are added, there probably won't be another official Distutils release until 1.0, which will be the run-up to inclusion in Python 1.6. (Paul Dubois tried to convince me -- if I understood properly -- that I should call that version Distutils 1.6 instead of 0.2. It strikes me that both are equally silly, so I've decided to split the difference and call it Distutils 1.0.) I expect that Python 1.6 will ship with Distutils 1.0.x, where x is the number of tries it takes before we get it right. Now, I do have a very simple release mechanism that I've been using since before Distutil 0.1 was released: code snapshots. This is done with a little P**l script that throws all the current code into a tarball and puts it on the Distutils-SIG web page. No documentation is included, no files are cvs-tagged, I don't even ensure that everything is checked in. This was mainly intended as a convenience for people who don't have CVS but want to stay on top of things. So, two questions for the audience: would you like to see versions 0.2, 0.3 etc. that gradually incorporate the new features. (It is *very* unlikely that any amount of howling will sway me on this; it took me one full day to release 0.1.3, what with checkins and doc updates and all, and I'm not eager to throw away many more days to do that. I'm just curious to hear how many people will be put off by this.) And, more importantly, would you like to see periodic code snapshots -- or are those of you who actually have the code and work on it just as happy using CVS? Please reply to me personally (gward(a)python.net); no need to clutter the sig with meta-issues. Greg -- Greg Ward - all-purpose geek gward(a)python.net http://starship.python.net/~gward/ Know thyself. If you need help, call the CIA.

1 0

Building PIL with Distutils
by Greg Ward 07 Feb '00

07 Feb '00

Hi all -- I've just finished hacking in the feature that allows Distutils to build a standalone C library before building any extension modules. This model is used by PIL, and I've used it in the past for a Perl extension, so I'm sure it's a useful thing to have. To demonstrate, I've just released a new code snapshot: http://www.python.org/sigs/distutils-sig/distutils-20000204.tar.gz If you want to try it out, you'll need * the 2000-02-04 Distutils code snapshot installed on your system * the PIL source distribution * a C compiler It's entirely possible that this will work under Windows, but I expect it will take some messing around. The first hurdle is to generate libImaging/ImConfig.h; under Unix, this is a snap: cd libImaging ; ./configure Fredrik, do you perchance have a version of ImConfig.h that you use to build PIL on Windows? Next, you have to (ugh) edit the setup script. I hate this, but there's going to have to be a heckuva lot of code written before it's no longer required (can you say "Autoconf in Python"?). You need to a) specify which optional libraries (from jpeglib, zlib, tcl, and tk) you have installed, and b) where they are installed. This is the bit that, in principle, ought to work under Windows as long as you have the .h and .lib files and know where they are. Then just run the setup script. If you like doing things in small chunks: python setup.py build_lib python setup.py build_ext Or, if you prefer medium sized chunks: python setup.py build And if you really enjoy biting off a lot at once: python setup.py install Please let me know how this works for you -- it worked great for me under Linux, with the four optional libraries all in /usr/{lib,include}. When I tried it on Solaris, I had to hack up the setup script to have the big editable section at the top ("Oh yeah, not every Unix system has all of these libraries installed..."). Now it works for me on Solaris. As usual, I'd love to hear if anyone gets it to work under Windows. Greg PS. I've attached the example PIL setup script in case you're curious, but not curious enough to download the Distutils snapshot. Just to be sure you know: this WILL NOT WORK with Distutils 0.1.3! It needs the 2000-02-04 snapshot.

2 2

And a proposed new manifest syntax
by Greg Ward 07 Feb '00

07 Feb '00

...picking up where I left earlier today, here's my proposed new syntax for the MANIFEST.in file. Note that the semantics are the same as the existing MANIFEST, but it should be a) easier to read, b) easier to write, and c) easier to understand the results (thanks to the division of labour between MANIFEST.in and MANIFEST -- see my earlier post). The requirements for the MANIFEST.in syntax is that the developer must be able to include or exclude files by - exact path (including wildcards -- "foo/bar.py" is just a special case of "foo/*.py") - wildcard applied recursively throughout a directory subtree And of course the MANIFEST.in file must be clear, concise, and expressive. The existing syntax is already concise and expressive, but pssbly a lttl too cncse. In my proposed syntax, there are four types of lines in the MANIFEST.in file: <path> include exactly the specified file(s) (path can be a complete directory specification ending with a wildcard, eg. "foo.py", "bar/README.txt", "qux/*.c") exclude <path> exclude exactly the specified file(s) recursive <path> split <path> into <dirname> and <basename>; recursively include all files under <dirname> that match <basename> recursive-exclude <path> you figure it out ;-) Possible points of contention (not "open issues" because I've pretty much made up my mind -- but I am willing to consider the existence of unenlightened heathens who dare to disagree ;-): * no "include" keyword -- clarity/concision tradeoff * no wildcards on directories, ie. "foo/bar/*.py" is OK, but not "foo/*/bar.py" A subtlety that I don't think I've mentioned very widely: like in the setup script, paths would be delimited with a forward slash rather than using the local convention. (Distutils explicitly does "string.split (..., '/')" in many places -- this is a feature, as the resulting list is then put back together with 'os.path.join()' before being used in the real world.) The purpose is the same for both the setup script and MANIFEST.in -- portability. As an example, here's how the MANIFEST file for Distutils 0.1.3 would look with the new syntax: *.txt examples/*.txt examples/*.py exclude distutils/command/build_lib.py So -- opinions? I'll probably implement the new 'dist' command in the next few days, so get back to me soon if you think I'm doing it wrong. Greg

1 0

Proposed new manifest scheme
by Greg Ward 06 Feb '00

06 Feb '00

Hi all -- right up at the top of my list of things to do for the Distutils is "fix the dist command". The main problems are that the syntax for the MANIFEST file (where you specify the files to go in a source distribution) is a bit goofy, and there's poor feedback about which files will actually be included. Part 1 of the fix is to rearrange how the "dist" command works. I've worked out a solution that appeals to me; give this a read and see if it sounds right to you. Step 1 (optional): Module developer creates MANIFEST.in file. This is a concise-but- readable specification of the files to be included in the source distribution with wildcards, directory recursion, and exclusion. (Syntax to be worked out in a future post; it should have the same features as the current syntax, but be more readable.) Step 2 (optional): Developer runs "dist" command with "--manifest-only" option. Distutils parses MANIFEST.in and spits out MANIFEST, containing a complete, explicit list of every file to be included in the source distribution. If MANIFEST.in doesn't exist -- ie. the developer skipped step 1 -- Distutils creates a MANIFEST using the "default fileset", mostly the pure Python modules and extension module source files mentioned in setup.py. Possibly: print a warning for every file in the source tree that *won't* be included in the source distribution; or write this information to another file (with a "files were excluded: see ..." warning). Step 3: Developer runs "dist" command. If MANIFEST doesn't exist or is out-of-date (ie. developer skipped step 2, or edited MANIFEST.in since running step 2), it is regenerated. Distutils then creates a source distibution (tarball, zipfile, whatever) containing exactly the files listed in MANIFEST. Possibly *this* is the step where the warning about excluded files should be generated. Open issues: * Obviously we should regenerate MANIFEST whenever MANIFEST.in is updated. (I.e. MANIFEST is only auto-generated, never edited -- unless you like playing with fire.) But what about when the filesystem changes? If I add or remove files or directories, MANIFEST should be regenerated. I seem to recall that detecting this in a portable way is pretty much impossible, especially in the presence of network filesystems (NFS, SMB). Two possible solutions: regenerate MANIFEST every time the "dist" command is run, or add a "--force-manifest" option. I prefer the latter because of the (presumed) expense of walking the directory tree. * Should MANIFEST be included with the source distribution? What about MANIFEST.in? The only reason I can see for including MANIFEST is that it would enable trivial integrity checking; it buys you zero security-wise, but at least ensures the download wasn't truncated. I think this is a bogus argument; a truncated ZIP or .tar.gz file simply won't unpack without errors, so I don't see a big need for checking for the presence of all files. A less trivial integrity check could be done by adding MD5 signatures (or something) to MANIFEST, but that would all-but-require regenerating the damn thing every time the "dist" command is run. Including MANIFEST.in is more defensible; it's one of the "source" files the developer uses to maintain the distribution, and third parties should be able to regenerate the author's source distribution if needed. (Just following the letter of the free software licences: forking should be an option, but hopefully not a commonly used one!) * When should the developer be warned about files in his development tree that weren't picked up by the MANIFEST-generating scan? If we do it when the MANIFEST is generated (step 2 above), that's when the developer is most likely to be watching -- why would you bother doing an explicit separate MANIFEST generation if you're not going to watch it closely? However, doing it as late as possible -- ie. when the MANIFEST is read and the source distribution is generated -- maximizes the chance of spotting any late additions not caught in MANIFEST.in's net. At the very least, this will remind the developer to re-run "dist --force-manifest", and it might well remind him to edit his MANIFEST.in to catch the late additions. * Speaking of warning about files not included: there should be a way to say, "Don't warn me about excluding X". X could be *~, *.bak, *.o, or whatever (depending on your platform, editor, compiler, development style, ...). One possibility: any file explicitly excluded by MANIFEST.in would be exempt from "not included" warnings. This would require you to explicitly exclude *~, *.o, etc. -- normally they would not be caught in MANIFEST.in's web at all, so no need to exclude them. Alternately, we could add a bit more syntax to MANIFEST.in that says "Don't necessarily include or exclude this file, but don't warn me if it happens to be excluded". I don't see any need for this, and it could be confusing. Thoughts? * Should the "default fileset" be included even when you have a MANIFEST.in? This seems obvious to me, but others (hi Fred!) have disagreed in the past. Since you can always exclude files from the default set (a feature of the present syntax that will be included in any future syntax), I see no need to make the great utility of a default fileset disappear just because you need to distribute files *not* mentioned in setup.py. Coming soon: proposed new syntax for the MANIFEST.in file. Greg

1 0

patch to unixccompiler.py
by Joe Van Andel 06 Feb '00

06 Feb '00

I've encountered a situation where I'd like to be able to over-ride the default compiler used by distutils. I have an extension which requires a newer version of gcc than I used to build python. The following patch uses the value of the CC environment variable to override the default. I'd like to see this in the next version of distutils, if possible: % diff -c unixccompiler.orig unixccompiler.py *** unixccompiler.orig Sun Dec 12 09:57:47 1999 --- unixccompiler.py Tue Feb 1 13:58:39 2000 *************** *** 22,27 **** --- 22,32 ---- from copy import copy from sysconfig import \ CC, CCSHARED, CFLAGS, OPT, LDSHARED, LDFLAGS, RANLIB, AR, SO + import os + tcc = os.environ.get('CC') + if tcc: + CC=tcc + from ccompiler import CCompiler, gen_preprocess_options, gen_lib_options from util import move_file, newer_pairwise, newer_group -- Joe VanAndel National Center for Atmospheric Research http://www.atd.ucar.edu/~vanandel/ Internet: vanandel(a)ucar.edu

3 6

Proposal: Registry access for Python on Windows
by Thomas Heller 05 Feb '00

05 Feb '00

I propose to include functions to access the windows registry into the python 1.6 core. I have thrown together some code which I will post hopefully tomorrow, but I would like to hear some comments before. ---------------------------------------------------------------------- winreg - registry access module Constants: HKEY_CLASSES_ROOT, HKEY_LOCAL_MACHINE, HKEY_CURRENT_USER, HKEY_USERS - bases of registry KEY_READ, KEY_WRITE, KEY_ALL_ACCESS - access rights REG_SZ, REG_DWORD, REG_BINARY - typecodes for values Exception: error - raised when a function fails. Will usually contain a windows error code and a textual description. Functions: OpenKey (base, subkey, [, rights=KEY_READ]) -> integer Opens an existing key with the specified access rights and returns an integer handle. The handle must be closed by a call to CloseKey(). CreateKey (base, subkey [, sam=KEY_ALL_ACCESS]) -> integer Creates a new subkey or opens an exsiting one and returns an integer handle. base must be one of the HKEY_ constants or an integer handle. The handle must be closed by a call to CloseKey(). CloseKey (handle) Closes a key handle. EnumValues (handle, subkey) -> sequence Returns a sequence containing name, value, typecode triples for each existing value. EnumKeys (handle, subkey) -> sequence Returns a sequence containing the names of all subkeys. QueryValue (handle, subkey) -> tuple Returns a tuple containing name, value and typecode. SetValue (handle, name, typecode, value) Sets the value of a name to value. value must be a string for REG_SZ or REG_BINARY, an integer for REG_DWORD. DeleteValue (handle, valuename) Deletes the value. DeleteKey (handle, name [,recursive=0) Deletes the named key if no subkeys exist. If recursive is given as a true value, subkeys are recursively deleted. This is done with Reg* calls, NOT with SHDeleteKey and SHDeleteEmptyKey functions, so should work under any flavor of NT and Win9x. Note: To use the SetValue and Delete* functions, the key must have been opened with KEY_WRITE_ACCESS or KEY_ALL_ACCESS. ---------------------------------------------------------------------- Open Questions: Is the recursive-flag for DeleteKey() too dangerous? Should I switch from an integer handle to a full blown python-object, which would call CloseKey() automatically in the destructor? A higher level interface would be nice, but this should probably implemented on top of this module in python. Comments appreciated! Thomas Heller ION-TOF GmbH

5 10