From groma@nucleus.szbk.u-szeged.hu Wed Jan 12 11:15:17 2000 From: groma@nucleus.szbk.u-szeged.hu (Geza Groma) Date: Wed, 12 Jan 2000 12:15:17 +0100 Subject: [Distutils] Possible bug in Distutils-0.1.2 on Windows Message-ID: <387C6245.70D08168@nucleus.szbk.u-szeged.hu> Trying to install the new Numerical package by Distutils-0.1.2 on Windows NT with MSVC 6.0, I got the following error: D:\Archives\Python\Numerical-14>python setup.py build running build running build_py not copying Lib\ArrayPrinter.py (output up-to-date) not copying Lib\FFT.py (output up-to-date) not copying Lib\LinearAlgebra.py (output up-to-date) not copying Lib\Matrix.py (output up-to-date) not copying Lib\MLab.py (output up-to-date) not copying Lib\Numeric.py (output up-to-date) not copying Lib\Precision.py (output up-to-date) not copying Lib\RandomArray.py (output up-to-date) not copying Lib\UserArray.py (output up-to-date) running build_ext C:\Program Files\Microsoft Visual Studio\VC98\bin\cl.exe /nologo -Ic:\Program Fi les\Python\include\python1.5 -Ic:\Program Files\Python\include -IInclude /c /FoS rc/_numpymodule.obj /TcSrc/_numpymodule.c Command line warning D4024 : unrecognized source file type 'Files\Python\include \python1.5', object file assumed Command line warning D4027 : source file 'Files\Python\include\python1.5' ignore d Command line warning D4024 : unrecognized source file type 'Files\Python\include ', object file assumed Command line warning D4027 : source file 'Files\Python\include' ignored _numpymodule.c Src/_numpymodule.c(1) : fatal error C1083: Cannot open include file: 'Python.h': No such file or directory Traceback (innermost last): File "setup.py", line 29, in ? ext_modules = [('_numpy', File "c:\Program Files\Python\distutils\core.py", line 96, in setup dist.run_commands () File "c:\Program Files\Python\distutils\core.py", line 442, in run_commands self.run_command (cmd) File "c:\Program Files\Python\distutils\core.py", line 491, in run_command cmd_obj.run () File "c:\Program Files\Python\distutils\command\build.py", line 52, in run self.run_peer ('build_ext') File "c:\Program Files\Python\distutils\core.py", line 802, in run_peer self.distribution.run_command (command) File "c:\Program Files\Python\distutils\core.py", line 491, in run_command cmd_obj.run () File "c:\Program Files\Python\distutils\command\build_ext.py", line 146, in ru n self.build_extensions (self.extensions) File "c:\Program Files\Python\distutils\command\build_ext.py", line 209, in bu ild_extensions include_dirs=include_dirs) File "c:\Program Files\Python\distutils\msvccompiler.py", line 103, in compile self.spawn ([self.cc] + cc_args) File "C:\Program Files\Python\distutils\ccompiler.py", line 421, in spawn spawn (cmd, verbose=self.verbose, dry_run=self.dry_run) File "c:\Program Files\Python\distutils\spawn.py", line 37, in spawn _spawn_nt (cmd, search_path, verbose, dry_run) File "c:\Program Files\Python\distutils\spawn.py", line 68, in _spawn_nt raise DistutilsExecError("command failed: %d" % rc) distutils.errors.DistutilsExecError: command failed: 2 I guess the problem is that cl.exe does not accept a directory name including spaces in its argument list unless in quotation marks. In the above example -Ic:\Program Files\Python\include\python1.5 should be written as -I"c:\Program Files\Python\include\python1.5" I am not a member of this list, so please, cc all responses to my private e-mail address. Thanks, Geza Groma From thomas.heller@ion-tof.com Wed Jan 12 15:57:16 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Wed, 12 Jan 2000 16:57:16 +0100 Subject: [Distutils] Comments on Distutils-0.1.2 on Windows Message-ID: <005d01bf5d15$b39efa20$4500a8c0@thomasnotebook> Well, I finally got a chance to test distutils 0.1.2 under Windows. 1. There is no standard zip-utility under windows, but zip.exe from info-zip works right out of the box. I've fixed the source to use James C. Ahlstrom's zipfile.py when zip.exe is not available (Should this always be used?) class Dist (Command): ... def make_zipfile (self, base_dir): # This assumes the Unix 'zip' utility -- it could be easily recast # to use pkzip (or whatever the command-line zip creation utility # on Redmond's archaic CP/M knockoff is nowadays), but I'll let # someone who can actually test it do that. try: self.spawn (["zip", "-r", base_dir + ".zip", base_dir]) except OSError: import zipfile z = zipfile.ZipFile (base_dir + ".zip", "wb", compression=zipfile.ZIP_DEFLATED) def visit (z, dirname, names): for name in names: path = os.path.join (dirname, name) if os.path.isfile (path): z.write (path, path) os.path.walk (base_dir, visit, z) z.close() 2. Libraries for VC++ are named python15.lib, not libpython15.lib. I had to fix this in msvccompiler.py: class MSVCCompiler (CCompiler): ... def library_filename (self, libname): """Return the static library filename corresponding to the specified library name.""" # return "lib%s%s" %( libname, self._static_lib_ext ) return "%s%s" %( libname, self._static_lib_ext ) def shared_library_filename (self, libname): """Return the shared library filename corresponding to the specified library name.""" # return "lib%s%s" %( libname, self._shared_lib_ext ) return "%s%s" %( libname, self._shared_lib_ext ) 3. os.link does not exist under windows, use shutil.copyfile for that: class Dist (Command): def make_release_tree (self, base_dir, files): ... self.announce ("making hard links in %s..." % base_dir) for file in files: dest = os.path.join (base_dir, file) if not os.path.exists (dest): try: self.execute (os.link, (file, dest), "linking %s -> %s" % (file, dest)) except AttributeError: import shutil self.execute (shutil.copyfile, (file, dest), "copying %s -> %s" % (file, dest)) Regards Thomas Heller ION-TOF GmbH From thomas.heller@ion-tof.com Thu Jan 13 14:34:23 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Thu, 13 Jan 2000 15:34:23 +0100 Subject: [Distutils] More (Windows) comments on distutils-0.1.2 Message-ID: <001a01bf5dd3$49e80bd0$4500a8c0@thomasnotebook> 1. Python extensions for Windows MUST be linked with the /MD switch to use the multithreaded runtime library. Optimizations are also enabled by the following line: > class MSVCCompiler (CCompiler) : > def __init__ (self, > [...] > self.compile_options = [ '/nologo', '/Ox', '/MD', '/GD' ] 2. In build_ext.py I find the following code: > def build_extensions (self, extensions): > if self.compiler.compiler_type == 'msvc': > def_file = build_info.get ('def_file') > if def_file is None: > source_dir = os.path.dirname (sources[0]) > ext_base = (string.split (extension_name, '.'))[-1] > def_file = os.path.join (source_dir, "%s.def" % ext_base) > if not os.path.exists (def_file): > self.warn ("file '%s' not found: " % def_file + > "might have problems building DLL") > def_file = None > > if def_file is not None: > extra_args.append ('/DEF:' + def_file) Doesn't this belong into msvccompiler.py? We do not need any stinkin' DEF-files, if we use the link options /EXPORT and /BASE. The following code builds the required arguments: > if def_file is not None: > extra_args.append ('/DEF:' + def_file) > else: > # if no def file is found, export the init function > # and calculate a random base address > extra_args.append ('/EXPORT:init' + extension_name) > import whrandom > base = whrandom.randint (0x1000, 0x8000) * 0x10000 > extra_args.append ('/BASE:0x%x' % base) Regards, Thomas Heller ION-TOF GmbH From dubois1@llnl.gov Thu Jan 13 17:31:55 2000 From: dubois1@llnl.gov (Paul F. Dubois) Date: Thu, 13 Jan 2000 09:31:55 -0800 Subject: [Distutils] wish list Message-ID: <00011309392900.32087@almanac> Here's my wish list after some use of Distutils-0.1.2 1. The object files should go in some subdirectory of build so that I can= clean up by just blowing away that directory. 2. Sometimes I eliminate a .py file but there is a copy in build that sti= ll gets installed. =20 3. If you just type python setup.py the information you get back is not helpful, since you don't know what the commands are. I think it should l= ist all the legal commands. From mmuller@enduden.com Fri Jan 14 01:06:43 2000 From: mmuller@enduden.com (Michael Muller) Date: Thu, 13 Jan 2000 20:06:43 -0500 Subject: [Distutils] wish list References: <00011309392900.32087@almanac> Message-ID: <387E76A3.7E0653C8@enduden.com> "Paul F. Dubois" wrote: > > Here's my wish list after some use of Distutils-0.1.2 > > 1. The object files should go in some subdirectory of build so that I can clean > up by just blowing away that directory. I'd rather see a "clean" command - delete all generated files. ============================================================================= michaelMuller = mmuller@enduden.com | http://www.cloud9.net/~proteus ----------------------------------------------------------------------------- Those who do not understand Unix are condemned to reinvent it, poorly. -- Henry Spencer ============================================================================= From mhammond@skippinet.com.au Mon Jan 17 03:58:13 2000 From: mhammond@skippinet.com.au (Mark Hammond) Date: Sun, 16 Jan 2000 19:58:13 -0800 Subject: [Distutils] wish list In-Reply-To: <387E76A3.7E0653C8@enduden.com> Message-ID: Michael wrote: > "Paul F. Dubois" wrote: > > > > Here's my wish list after some use of Distutils-0.1.2 > > > > 1. The object files should go in some subdirectory of build so > that I can clean > > up by just blowing away that directory. > > I'd rather see a "clean" command - delete all generated files. Id rather see both! I hate having object files with the sources, but more to the point, a seperate directory better supports debug and release builds - which is very important on Windows, and presumably quite important everywhere else... Mark. From robin@jessikat.demon.co.uk Mon Jan 17 11:35:55 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Mon, 17 Jan 2000 11:35:55 +0000 Subject: [Distutils] bugs in MSVCCompiler ? Message-ID: <6qBjnFAb6vg4Ewt2@jessikat.demon.co.uk> I've just found what appears to be a bug in 0.1.2 in the MSVCCompiler class. at line 267 the library_filename functions should look like def library_filename (self, libname): """Return the static library filename corresponding to the specified library name.""" return "%s%s" %( libname, self._static_lib_ext ) def shared_library_filename (self, libname): """Return the shared library filename corresponding to the specified library name.""" return "%s%s" %( libname, self._shared_lib_ext ) the original format strings were "lib%s%s" leading to errors such as trying to use libpython15.lib during the build. -- Robin Becker From gward@cnri.reston.va.us Mon Jan 17 16:47:57 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 17 Jan 2000 11:47:57 -0500 Subject: [Distutils] Disposition of C extensions and packages In-Reply-To: <199912190243.VAA03844@207-172-57-180.s180.tnt2.ann.va.dialup.rcn.com>; from amk1@erols.com on Sat, Dec 18, 1999 at 09:43:19PM -0500 References: <199912190243.VAA03844@207-172-57-180.s180.tnt2.ann.va.dialup.rcn.com> Message-ID: <20000117114757.A9577@cnri.reston.va.us> On 18 December 1999, A.M. Kuchling said: > [Crossposted to xml-sig, distutils-sig] > > I'm working on getting the XML-SIG's CVS tree to install using the > current version of the Distutils. Right now there are two C > extensions, sgmlop.so and pyexpat.so, and they're installed under > xml/parsers/ . It's hard to handle this case using the distutils code > as it stands, because it expects to put extensions into a > build/platlib/ directory, from which they'll be installed into > site-packages. Well, I missed this thread first time around because I was on holiday. Andrew went to such amazing lengths to do something that's actually quite easy that I thought it would be worth following up, even if I'm a month late. In short, putting Python extension modules into a package is really, really, *really* easy. Here's what *was* in Andrew's setup.py for the XML package (edited for readability): ext_modules = [('sgmlop', { 'sources' : ['extensions/sgmlop.c'] }), ('pyexpat', { 'define': [('XML_NS', None)], 'include_dirs': [ 'extensions/expat/xmltok', 'extensions/expat/xmlparse' ], 'sources' : [...] } ) ] Note that the names of the two extensions are 'sgmlop' and 'pyexpat'. In regards to where the .so files wind up, *the name is the most important thing* (and usually all you need). All we had to do to get rid of Andrew's extended command classes (and other kludgery) was set the names of the two extension modules appropriately, so the XML setup script now has this: ext_modules = [('xml.parsers.sgmlop', { 'sources' : ['extensions/sgmlop.c'] }), ('xml.parsers.pyexpat', { 'define': [('XML_NS', None)], 'include_dirs': [ 'extensions/expat/xmltok', 'extensions/expat/xmlparse' ], 'sources' : [...] } ) ] i.e. the names of the extensions had to change... and nothing else. (Well, we commented out the unnecessarily extended command classes.) I assumed that large module distributions with many extensions will probably put them in the same package. Thus, we could have done the change this way as well: ext_package = 'xml.parsers' ext_modules = [('sgmlop', { 'sources' : ['extensions/sgmlop.c'] }), ('pyexpat', { 'define': [('XML_NS', None)], 'include_dirs': [ 'extensions/expat/xmltok', 'extensions/expat/xmlparse' ], 'sources' : [...] } ) ] And if you only have a common base package for many extensions, you should still be able to specify that base package with 'ext_package': ext_package = 'xml.parsers' ext_modules = [('pkg1.sgmlop', { 'sources' : ['extensions/sgmlop.c'] }), ('pkg2.pyexpat', { 'define': [('XML_NS', None)], 'include_dirs': [ 'extensions/expat/xmltok', 'extensions/expat/xmlparse' ], 'sources' : [...] } ) ] If everything goes as planned, that should result in two extensions called 'xml.parsers.pkg1.sgmlop' and 'xml.parsers.pkg2.pyexpat'. Note that my assertions about these last two examples are based solely on reading the code and a dim recollection of what I had in mind when I wrote it -- i.e. I haven't tested them. YMMV, but please let me know if it does. Greg From gward@cnri.reston.va.us Mon Jan 17 16:58:08 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 17 Jan 2000 11:58:08 -0500 Subject: [Distutils] distutils and JPython In-Reply-To: <38647663.4686478@pipmail.dknet.dk>; from bckfnn@pipmail.dknet.dk on Sat, Dec 25, 1999 at 07:46:51AM +0000 References: <38647663.4686478@pipmail.dknet.dk> Message-ID: <20000117115807.C9577@cnri.reston.va.us> On 25 December 1999, Finn Bock said: > Generally I'm looking at distutils to see if it can help me simplify > the installation of some of the JPython modules I have created. Finn -- thanks for all the suggestions. I have skimmed them, but not come close to absorbing them. JPython support is definitely a medium-term goal for the Distutils, but it certainly wasn't planned for 0.1. I also wasn't planning on it for 0.2, but I am open to persuasion and/or threats. ;-) > These are normal installation issues. I am a bit unsure on how much I > can reasonably expect distutils to deal with this. From my very > limited experience with distutils, I think distutils is exactly the > right place to put some of the logic that deals with JVM and OS > dependencies as described above. I totally agree. The JVM is just another sort of "platform", and should probably be dealt with on the same level as "posix" or "nt". I hadn't thought previously about the differences between JVMs; I had been naively thinking that we would add "java" support to the existing "posix" and "nt" (and someday, "mac") support. Guess it won't be that simple... sigh... > a) Documentation buglet. In the USAGE file for "dist", -f is described > as an alias for --formats. -f does not work. D'oh! Well, it *used* to be -- "-f" is now an alias for the global command-line option "--force", which was snuck in somewhere around 0.1.1. Fixed. > b) How about allowing for some alias names for the README file. Some > of us lazy double-clicking windows types prefer names like README.txt. Oh, all right. It's also nicer for HTTP servers that way. Damn and blast this cross-platform targeting. Why can't everyone just be happy using Unix? ;-) > c) Why are "packages" and "py_module" mutually exclusive. One of my > products happens to have a toplevel module which contains "from > pck.Main import *". I think that I find the current behavior limiting. Mainly laziness -- I had a sneaking suspicion that allowing both of them at the same time might have subtle problems, so I sidestepped the whole issue. Make a good case for where they're both needed, and I'll reconsider. (I agree that it's limiting, but *how* limiting?) Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gward@cnri.reston.va.us Mon Jan 17 17:15:58 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 17 Jan 2000 12:15:58 -0500 Subject: [Distutils] Package Meta-information Patch In-Reply-To: <386966F0.AA74F1DF@cloud9.net>; from proteus@cloud9.net on Tue, Dec 28, 1999 at 08:42:08PM -0500 References: <386966F0.AA74F1DF@cloud9.net> Message-ID: <20000117121558.D9577@cnri.reston.va.us> On 28 December 1999, Michael Muller said: > I've enclosed a patch which addresses a particular concern of mine: package > meta-information. At the end of the install command, it creates a package > information file in /_pkginfo named after the package > (it also creates the "_pkginfo" directory, if necessary). The file contains > python variable definitions for the package name, version number, list of > files installed, dependencies, and compatible versions (although the latter > two are always empty at this time). Hmmm... interesting idea. I mean, we've known all along that some sort of "package metainfo database" (metadatabase? ugh) is going to be needed for exactly the reasons you listed (uninstall, dependeny analysis, and system cataloging). I have not spent a lot of time thinking about it, but I think I was stuck in a "database must be one big file" rut, with all the attendant problems of performance, concurrent access, etc. About as far as I got was thinking "text files suck for size and performance, and DB or dbm files might not be portable enough". But I think I like your approach -- at least part of it. Specifically, I think I like the notion of spreading the "metainfo database" across many files in many directories. To find information about all module distributions installed, you troll sys.path, looking for a "_pkginfo" subdirectory in each entry, and then look at the files installed there. At least, that's the understanding I get from reading your message and a cursory scan of the patch -- am I right? This pretty much solves the practical side of "what to do about concurrent access" -- in practice, it's not going to happen much, so don't get too worried about it. It doesn't sound very good for performance, unless all you want is a list of packages installed -- that should be pretty fast (you can get everything you need from a succession of os.listdir() calls). What I'm a little leery about is using Python code as a data format. It's attractive because we all know the syntax and don't have to write a parser. But using a general-purpose language for *such* a specific, tightly-targeted task seems ... I dunno ... overkill-ish. And I wonder if there are security holes lurking in the concept of using code for system catalog data. Does anyone else share my reservations (which are vague, ill-defined, and more superstitious than anything else)? Conversely, does anyone think that Python code is absolutely the right way to store module distribution metadata? Thanks again for the patch -- I think it should find its way into Distutils 0.2 after the SIG has thrashed through some of the issues it raises. Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From fdrake@acm.org Mon Jan 17 18:40:58 2000 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Mon, 17 Jan 2000 13:40:58 -0500 (EST) Subject: [Distutils] Package Meta-information Patch In-Reply-To: <20000117121558.D9577@cnri.reston.va.us> References: <386966F0.AA74F1DF@cloud9.net> <20000117121558.D9577@cnri.reston.va.us> Message-ID: <14467.25146.323892.829680@weyr.cnri.reston.va.us> Greg Ward writes: > About as far as I got was thinking "text files suck for size and > performance, and DB or dbm files might not be portable enough". They're not *bad* for size any more than page alignment in a dbm database. ;-) > This pretty much solves the practical side of "what to do about > concurrent access" -- in practice, it's not going to happen much, so > don't get too worried about it. It doesn't sound very good for > performance, unless all you want is a list of packages installed -- that > should be pretty fast (you can get everything you need from a succession > of os.listdir() calls). Tools that need faster access during an operation can build temporary databases to accelerate operation as needed, so I don't see any problems here. I wouldn't expect that to be needed too often. > What I'm a little leery about is using Python code as a data format. > It's attractive because we all know the syntax and don't have to write a > parser. But using a general-purpose language for *such* a specific, > tightly-targeted task seems ... I dunno ... overkill-ish. And I wonder > if there are security holes lurking in the concept of using code for > system catalog data. > > Does anyone else share my reservations (which are vague, ill-defined, Yes. This stuff should not require any exec or eval. It might be reasonable to use something like the .ini format; this can be handled using ConfigParser. This way we still don't need to write a parser. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From thomas.heller@tascon-gmbh.de Mon Jan 17 19:02:08 2000 From: thomas.heller@tascon-gmbh.de (Thomas Heller) Date: Mon, 17 Jan 2000 20:02:08 +0100 Subject: [Distutils] Comments on Distutils-0.1.2 on Windows References: <005d01bf5d15$b39efa20$4500a8c0@thomasnotebook> <20000117125637.E9577@cnri.reston.va.us> Message-ID: <011301bf611d$5a8f7480$4500a8c0@thomasnotebook> > On 12 January 2000, Thomas Heller said: > > 1. There is no standard zip-utility under windows, > > but zip.exe from info-zip works right out of the box. > > Oh good! I've changed the comment in the source now. > > > I've fixed the source to use James C. Ahlstrom's zipfile.py > > when zip.exe is not available (Should this always be used?) > > Do you mean should zipfile.py be the default rather than the standalone > zip utility? Not yet -- the latter is far more widespread than the > former! In Python 1.6, maybe. Maybe you could include zipfile.py into the distutils package? > > Anyways, I've incorporated your support for zipfile.py into the dist > command, but I don't have that module installed to try it out. The code > is slightly different from yours, so would you mind trying out the > attached version of dist.py? Well, you didn't expect that I would really test it ;-) ? This code try: self.spawn (["zip", "-r", base_dir + ".zip", base_dir]) except DistutilsExecError: try: import zipfile except ImportError: raise DistutilsExecError, \ fails for two simple reasons: - You are not doing 'from distutils.errors import *' in dist.py - self.spawn() raises an OSError, not an DistutilsExecError. Otherwise your code looks fine. (Except that my patch for using shutils.copyfile is missing (if os.link fails). > > I'll get to the bugs you found in MSVCCompiler shortly. Thanks! > Thank you for distutils! Thomas From gward@cnri.reston.va.us Mon Jan 17 20:10:30 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 17 Jan 2000 15:10:30 -0500 Subject: [Distutils] Linux Journal Python Supplement Message-ID: <20000117151029.B9755@cnri.reston.va.us> Hi all -- the folks at Linux Journal asked me to post this to the sig. Free magazine, woo hoo! ----- Forwarded message from Rebecca Cassity ----- [...] The May 2000 issue of Linux Journal is going to be accompanied by a special edition Python Supplement! Python users can sign up for their free copy at http://www.linuxjournal.com/python/. ----- End forwarded message ----- -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gward@cnri.reston.va.us Mon Jan 17 20:31:32 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 17 Jan 2000 15:31:32 -0500 Subject: [Distutils] bugs in MSVCCompiler ? In-Reply-To: <6qBjnFAb6vg4Ewt2@jessikat.demon.co.uk>; from robin@jessikat.demon.co.uk on Mon, Jan 17, 2000 at 11:35:55AM +0000 References: <6qBjnFAb6vg4Ewt2@jessikat.demon.co.uk> Message-ID: <20000117153131.B9793@cnri.reston.va.us> On 17 January 2000, Robin Becker said: > I've just found what appears to be a bug in 0.1.2 in the MSVCCompiler > class. > > at line 267 the library_filename functions should look like > > def library_filename (self, libname): > """Return the static library filename corresponding to the > specified library name.""" > return "%s%s" %( libname, self._static_lib_ext ) > > def shared_library_filename (self, libname): > """Return the shared library filename corresponding to the > specified library name.""" > return "%s%s" %( libname, self._shared_lib_ext ) > > > the original format strings were "lib%s%s" leading to errors such as > trying to use libpython15.lib during the build. Yep, someone else already caught that -- fixed now. Greg From gward@cnri.reston.va.us Mon Jan 17 20:39:51 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 17 Jan 2000 15:39:51 -0500 Subject: [Distutils] More (Windows) comments on distutils-0.1.2 In-Reply-To: <001a01bf5dd3$49e80bd0$4500a8c0@thomasnotebook>; from thomas.heller@ion-tof.com on Thu, Jan 13, 2000 at 03:34:23PM +0100 References: <001a01bf5dd3$49e80bd0$4500a8c0@thomasnotebook> Message-ID: <20000117153950.C9793@cnri.reston.va.us> On 13 January 2000, Thomas Heller said: > 1. Python extensions for Windows MUST be linked with > the /MD switch to use the multithreaded runtime > library. Optimizations are also enabled by the following > line: > > > class MSVCCompiler (CCompiler) : > > def __init__ (self, > > [...] > > self.compile_options = [ '/nologo', '/Ox', '/MD', '/GD' ] I haven't seen any disagreement with this, so I'll take your word for it. I'd feel more comfortable if another Python/Windows extension developer would speak up and say, "Yep, that's the Right Thing to do". (Mark? Guido?) So what are the flags here? I assume /Ox is for optimization; you said /MD selects the multithreaded RT library. What is /GD for? > 2. In build_ext.py I find the following code: > [...code to deal with .def files elided...] > Doesn't this belong into msvccompiler.py? If anywhere at all. I really, really dislike this whole .def file thing, and I don't like them cluttering up the build options dictionary or the build_ext command class. But I don't know how else to do it, because I know about as much about Python on Windows as I do about skateboarding on Pluto. > We do not need any stinkin' DEF-files, if we use the link options > /EXPORT and /BASE. The following code builds the required arguments: Hallelujah! I've been waiting for someone to say this. Ever since I found out what .def files are, I've been wondering why the heck they are needed. > > if def_file is not None: > > extra_args.append ('/DEF:' + def_file) > > else: > > # if no def file is found, export the init function > > # and calculate a random base address > > extra_args.append ('/EXPORT:init' + extension_name) > > import whrandom > > base = whrandom.randint (0x1000, 0x8000) * 0x10000 > > extra_args.append ('/BASE:0x%x' % base) But this looks weird: a random base address? What the heck is going on here? Again, could another Python-on-Windows expert speak up? Nothing personal, Thomas, but I don't want to take the word of just one person on this code, and I'd like to know what the heck the random base address is all about. Thanks for the code -- now let's hear from some other voices as to whether this is the Right Thing to add to Distutils to compile extensions with MSVC. Anyone? Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gward@cnri.reston.va.us Mon Jan 17 21:09:40 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 17 Jan 2000 16:09:40 -0500 Subject: [Distutils] wish list In-Reply-To: <00011309392900.32087@almanac>; from dubois1@llnl.gov on Thu, Jan 13, 2000 at 09:31:55AM -0800 References: <00011309392900.32087@almanac> Message-ID: <20000117160939.D9793@cnri.reston.va.us> On 13 January 2000, Paul F. Dubois said: > Here's my wish list after some use of Distutils-0.1.2 > > 1. The object files should go in some subdirectory of build so that I > can clean up by just blowing away that directory. Agreed. It's not done that way because I thought it would be cute to compile a bunch of source files together (gcc -c foo.c bar.c baz.c instead of 3 compiler invocations). I remember the ability to do this with Borland make under MS-DOS was a big win about 8 years ago, but it has a couple of annoying side-effects: poor feedback (compiler sits there churning for several minutes without uttering a peep) and inability to control where the .o files go (without doing a chdir, which I want to avoid). (Actually, it can and does move them out of the current directory when done. This doesn't work too well if the user kills the build, though.) Oh, another thing: you've given a good reason for dropping the .o's in 'build/'. But there's an almost equally good reason for *not* doing so: 'build/' is supposed to be trivially installable, and putting temporary files there would break that. Damn. I guess a 'clean' command is the right way to do it. It would need to know where to find .o files (s/.c/.o/ on all the sources listed in 'ext_modules' should do the trick... well, OK, s/.[cC](pp|xx)?/.o/ to handle C++ source files. You get the idea. And it could just blow away 'build/' with a single 'rmtree()' call. One nice thing about having a 'clean' command is that you could tell it how clean you want to be, e.g. only clean up .o files, only clean up the build directory, etc. > 2. Sometimes I eliminate a .py file but there is a copy in build that still > gets installed. Glib answer: that's why we need a 'clean' command! Real answer: hmmm. Does that mean we need to scan the build directory, see if there's anything there that doesn't exist in the source directory, and warn about it? That's the only real solution I can think of. > 3. If you just type python setup.py the information you get back is not > helpful, since you don't know what the commands are. I think it should list > all the legal commands. That should be doable. Guess it means I'd need to add a canonical list of the standard commands -- I s'pose distutils/command/__init__.py would be the way to do that. I just realized something nice: the cmdclass dictionary (which is how you supply custom command classes, whether they're extended versions of standard commands or completely new commands) will give the list of all non-standard commands available in the current run. Nice! Minor problem: there needs to be a standard way for a command class to advertise what it does, so that this mythical list of commands can include a blurb about each one. This should be doable with a class attribute, but it adds one more little thing to the interface you must follow to implement a command class. This doesn't bother me especially. Slightly bigger problem: many commands are clearly subordinate to each other (eg. 'build_py' and 'build_ext' "belong to" 'build'), but this is not formally noted anywhere. Would the mythical list of commands be helped if it could convey this information? Here are some possible formats for the command list to take. First, no mention of the implicit "hierarchy of commands": Available commands: build - build all modules and extensions build_py - build (ie. prepare for installation) all pure Python modules build_ext - build (compile and link) all C/C++ extension modules install - install all modules and extensions install_py - install all pure Python modules install_ext - install all extension modules Here's what it might look like if we leave it up to the command classes' descriptive blurb to describe the run-time relationships between commands in the text: Available commands: build - build all modules and extensions (runs 'build_py' and 'build_ext') build_py - build (ie. prepare for installation) all pure Python modules build_ext - build (compile and link) all C/C++ extension modules install - install all modules and extensions (runs 'build', 'install_py', and 'install_ext') install_py - install all pure Python modules install_ext - install all extension modules And here's what it might look like if the code "knows" about the run-time command hierarchy: Available commands: build - build all modules and extensions build_py - build (ie. prepare for installation) all pure Python modules build_ext - build (compile and link) all C/C++ extension modules install - install all modules and extensions install_py - install all pure Python modules install_ext- install all extension modules Any preferences? Greg From thomas.heller@tascon-gmbh.de Mon Jan 17 21:17:02 2000 From: thomas.heller@tascon-gmbh.de (Thomas Heller) Date: Mon, 17 Jan 2000 22:17:02 +0100 Subject: [Distutils] More (Windows) comments on distutils-0.1.2 References: <001a01bf5dd3$49e80bd0$4500a8c0@thomasnotebook> <20000117153950.C9793@cnri.reston.va.us> Message-ID: <003701bf6130$33404e00$4500a8c0@thomasnotebook> > On 13 January 2000, Thomas Heller said: > > 1. Python extensions for Windows MUST be linked with > > the /MD switch to use the multithreaded runtime > > library. Optimizations are also enabled by the following > > line: > > > > > class MSVCCompiler (CCompiler) : > > > def __init__ (self, > > > [...] > > > self.compile_options = [ '/nologo', '/Ox', '/MD', '/GD' ] > > I haven't seen any disagreement with this, so I'll take your word for > it. I'd feel more comfortable if another Python/Windows extension > developer would speak up and say, "Yep, that's the Right Thing to do". > (Mark? Guido?) > > So what are the flags here? I assume /Ox is for optimization; you said > /MD selects the multithreaded RT library. What is /GD for? > /GD means 'optimize for DLLs'. Not really sure what it does ;-> Oh, shit: Does nothing. MS> /GD (Optimize for Windows DLL) MS> This optimization option is for future use. > > > 2. In build_ext.py I find the following code: > > > [...code to deal with .def files elided...] > > > Doesn't this belong into msvccompiler.py? > > If anywhere at all. I really, really dislike this whole .def file > thing, and I don't like them cluttering up the build options dictionary > or the build_ext command class. But I don't know how else to do it, > because I know about as much about Python on Windows as I do about > skateboarding on Pluto. > > > We do not need any stinkin' DEF-files, if we use the link options > > /EXPORT and /BASE. The following code builds the required arguments: > > Hallelujah! I've been waiting for someone to say this. Ever since I > found out what .def files are, I've been wondering why the heck they are > needed. DEF files define (at least) the exported functions in a DLL. Python extensions usually only export the init function, so this can easily be defined in a linker switch. > > > > if def_file is not None: > > > extra_args.append ('/DEF:' + def_file) > > > else: > > > # if no def file is found, export the init function > > > # and calculate a random base address > > > extra_args.append ('/EXPORT:init' + extension_name) > > > import whrandom > > > base = whrandom.randint (0x1000, 0x8000) * 0x10000 > > > extra_args.append ('/BASE:0x%x' % base) > > But this looks weird: a random base address? What the heck is going on On windows DLLs are linked to a certain start address. The linker uses a default of 0x11000000, if not otherwise specified. The windows loader, when trying to load the DLL, must relocate the DLL to a different address if a conflict occurrs (which is slow). Unless we have a central registry of base adresses for python extensions, we normally cannot avoid these conflicts. MS says: #Because there is no way to know what base addresses might be chosen by other #in-process components your users might employ, the best practice is to choose #an address at random from the indicated range, and round it up to the next # multiple of 64K. #If your company produces many in-process components, you may wish #to randomly calculate the base address of the first, and then arrange #the others above or below the first, thus guaranteeing at least #that your company's components will not have memory conflicts. http://msdn.microsoft.com/library/devprods/vs6/vbasic/vbcon98/vbconsettingba seaddressesforolecontrolprojects.htm > here? Again, could another Python-on-Windows expert speak up? Nothing > personal, Thomas, but I don't want to take the word of just one person > on this code, and I'd like to know what the heck the random base address > is all about. It is a try. At least, if you still support .def files, the module distributor can select: random base address, or using one from a def file. Further info, other possibilities to supply a base address: http://msdn.microsoft.com/library/devprods/vs6/visualc/vccore/_core_.2f.base .htm > > Thanks for the code -- now let's hear from some other voices as to > whether this is the Right Thing to add to Distutils to compile > extensions with MSVC. Anyone? > > Greg > -- > Greg Ward - software developer gward@cnri.reston.va.us > Corporation for National Research Initiatives > 1895 Preston White Drive voice: +1-703-620-8990 > Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG@python.org > http://www.python.org/mailman/listinfo/distutils-sig From gward@cnri.reston.va.us Mon Jan 17 22:02:29 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 17 Jan 2000 17:02:29 -0500 Subject: [Distutils] Pre-0.1.3 snapshot available Message-ID: <20000117170228.A10072@cnri.reston.va.us> Hi all -- I've finally caught up with the last couple weeks' traffic on the sig, and fixed many of the bugs reported. There's a new code snapshot available: http://www.python.org/sigs/distutils-sig/distutils-20000117.tar.gz Please give it a whirl, especially if you're using Windows. This mostly includes Windows portability fixes. It does *not* include Finn Bock's fixes for JPython; I haven't had time to look closely at those. They might have to wait for version 0.2. There *might* be a 0.1.3 release before IPC8, but I really should work on my slides for the conference first... so please give this snapshot a try. Just download it and run the setup script as usual -- the main difference between the snapshot and a real release is no doc updates (hence README, USAGE, and CHANGES are not included in the snapshot). Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From robin@jessikat.demon.co.uk Tue Jan 18 00:08:37 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Tue, 18 Jan 2000 00:08:37 +0000 Subject: [Distutils] More (Windows) comments on distutils-0.1.2 In-Reply-To: <20000117153950.C9793@cnri.reston.va.us> References: <001a01bf5dd3$49e80bd0$4500a8c0@thomasnotebook> <20000117153950.C9793@cnri.reston.va.us> Message-ID: <8fmyaEAF86g4Ew$y@jessikat.demon.co.uk> In article <20000117153950.C9793@cnri.reston.va.us>, Greg Ward writes >On 13 January 2000, Thomas Heller said: >> 1. Python extensions for Windows MUST be linked with >> the /MD switch to use the multithreaded runtime >> library. Optimizations are also enabled by the following >> line: >> >> > class MSVCCompiler (CCompiler) : >> > def __init__ (self, >> > [...] >> > self.compile_options = [ '/nologo', '/Ox', '/MD', '/GD' ] > >I haven't seen any disagreement with this, so I'll take your word for >it. I'd feel more comfortable if another Python/Windows extension >developer would speak up and say, "Yep, that's the Right Thing to do". >(Mark? Guido?) /GD is for a DLL specific optimisation > >So what are the flags here? I assume /Ox is for optimization; you said >/MD selects the multithreaded RT library. What is /GD for? > > >> 2. In build_ext.py I find the following code: >> >[...code to deal with .def files elided...] > >> Doesn't this belong into msvccompiler.py? > >If anywhere at all. I really, really dislike this whole .def file >thing, and I don't like them cluttering up the build options dictionary >or the build_ext command class. But I don't know how else to do it, >because I know about as much about Python on Windows as I do about >skateboarding on Pluto. > >> We do not need any stinkin' DEF-files, if we use the link options >> /EXPORT and /BASE. The following code builds the required arguments: > >Hallelujah! I've been waiting for someone to say this. Ever since I >found out what .def files are, I've been wondering why the heck they are >needed. .def files are for exporting things which aren't declared exported in the code. They're not needed if you use the MSVC specific declarators. > >> > if def_file is not None: >> > extra_args.append ('/DEF:' + def_file) >> > else: >> > # if no def file is found, export the init function >> > # and calculate a random base address >> > extra_args.append ('/EXPORT:init' + extension_name) >> > import whrandom >> > base = whrandom.randint (0x1000, 0x8000) * 0x10000 >> > extra_args.append ('/BASE:0x%x' % base) > >But this looks weird: a random base address? What the heck is going on >here? Again, could another Python-on-Windows expert speak up? Nothing >personal, Thomas, but I don't want to take the word of just one person >on this code, and I'd like to know what the heck the random base address >is all about. > Please don't bother to use random addresses. If you're lucky your build may make a very small difference. Ideally under windows all the dlls needed for a given application should have non intersecting default address ranges. See the file PC/dll_base.txt in the source distribution. However, this is only a minor optimisation and it seems that Guido has abandoned it in favour of simplicity. In a distribution where all the dlls are fixed it might be worth rebasing all of the dlls. The random hack may work, but is not guaranteed to avoid base clashes. >Thanks for the code -- now let's hear from some other voices as to >whether this is the Right Thing to add to Distutils to compile >extensions with MSVC. Anyone? > > Greg -- Robin Becker From mmuller@enduden.com Tue Jan 18 02:26:50 2000 From: mmuller@enduden.com (Michael Muller) Date: Mon, 17 Jan 2000 21:26:50 -0500 Subject: [Distutils] Package Meta-information Patch References: <386966F0.AA74F1DF@cloud9.net> <20000117121558.D9577@cnri.reston.va.us> Message-ID: <3883CF6A.3D3F43B2@enduden.com> Greg Ward wrote: [snip] > many files in many directories. To find information about all > module distributions installed, you troll sys.path, looking for a > "_pkginfo" subdirectory in each entry, and then look at the files > installed there. At least, that's the understanding I get from reading > your message and a cursory scan of the patch -- am I right? Actually, I was kind of stuck on the problem of what to do about multiple locations in sys.path, but now that you mention it, this sounds like a good approach :-). > What I'm a little leery about is using Python code as a data format. > It's attractive because we all know the syntax and don't have to write a > parser. But using a general-purpose language for *such* a specific, > tightly-targeted task seems ... I dunno ... overkill-ish. And I wonder > if there are security holes lurking in the concept of using code for > system catalog data. I'm a little leery about this also - that's partly why I isolated this code within the pkginfo module. You should really be able to use whatever meta-info repository that you want, providing that only _one_ is used on a particular system. The reason I went with the Python code approach is that it was extremely easy to write and debug (it's always easier to see what's going on when the data itself is readable), and because it was the only format that I was _absolutely certain_ would be available on every platform that would be using distutils. Also, there's the fact that with Python code, the interpreter is written in C - it will run significantly faster than any code that could be written in Python. ============================================================================= michaelMuller = mmuller@enduden.com | http://www.cloud9.net/~proteus ----------------------------------------------------------------------------- Mantra for the 60's: Tune in, turn on, drop out Mantra for the 90's: Turn on, jack in, jerk off ============================================================================= From thomas.heller@ion-tof.com Tue Jan 18 10:02:55 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue, 18 Jan 2000 11:02:55 +0100 Subject: [Distutils] Pre-0.1.3 snapshot available References: <20000117170228.A10072@cnri.reston.va.us> Message-ID: <000d01bf619b$31551d30$4500a8c0@thomasnotebook> > Hi all -- > > I've finally caught up with the last couple weeks' traffic on the sig, > and fixed many of the bugs reported. There's a new code snapshot > available: > > http://www.python.org/sigs/distutils-sig/distutils-20000117.tar.gz > > Please give it a whirl, especially if you're using Windows. > > This mostly includes Windows portability fixes. It does *not* include > Finn Bock's fixes for JPython; I haven't had time to look closely at > those. They might have to wait for version 0.2. > > There *might* be a 0.1.3 release before IPC8, but I really should work > on my slides for the conference first... so please give this snapshot a > try. Just download it and run the setup script as usual -- the main > difference between the snapshot and a real release is no doc updates > (hence README, USAGE, and CHANGES are not included in the snapshot). 1. Syntax errors in (unused?) commands/build_lib.py (look, you will see them) 2. _spawn_nt() still raises an OSError when the command is not found, see the traceback I sent you yesterday. Fix follows: if not dry_run: # spawn for NT requires a full path to the .exe try: rc = os.spawnv (os.P_WAIT, executable, cmd) except OSError, detail: raise DistutilsExecError(detail) if rc != 0: raise DistutilsExecError("command failed: %d" % rc) 3. There is still the indentation error in dist.py: make_zipfile (line 475) Otherwise it works fine. Thanks! Thomas From gstein@lyra.org Tue Jan 18 12:20:02 2000 From: gstein@lyra.org (Greg Stein) Date: Tue, 18 Jan 2000 04:20:02 -0800 (PST) Subject: [Distutils] Package Meta-information Patch In-Reply-To: <14467.25146.323892.829680@weyr.cnri.reston.va.us> Message-ID: On Mon, 17 Jan 2000, Fred L. Drake, Jr. wrote: > Greg Ward writes: >... > > What I'm a little leery about is using Python code as a data format. > > It's attractive because we all know the syntax and don't have to write a > > parser. But using a general-purpose language for *such* a specific, > > tightly-targeted task seems ... I dunno ... overkill-ish. And I wonder > > if there are security holes lurking in the concept of using code for > > system catalog data. > > > > Does anyone else share my reservations (which are vague, ill-defined, > > Yes. This stuff should not require any exec or eval. It might be > reasonable to use something like the .ini format; this can be handled > using ConfigParser. This way we still don't need to write a parser. I second Fred here... use a format compatible with ConfigParser. Simple, clean, and easily handled. DBM or central databases are probably a bit bogus. What's the speed for? If you want to *locate* the _pkginfo files, then just append a pathname to a central file. Let the tool go and see if it is still there. Or the tool can be invoked with "do a filetree walk -- the log file may be out of sync." KISS :-) Cheers, -g -- Greg Stein, http://www.lyra.org/ From gward@cnri.reston.va.us Tue Jan 18 13:38:27 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Tue, 18 Jan 2000 08:38:27 -0500 Subject: [Distutils] Pre-0.1.3 snapshot available In-Reply-To: <000d01bf619b$31551d30$4500a8c0@thomasnotebook>; from thomas.heller@ion-tof.com on Tue, Jan 18, 2000 at 11:02:55AM +0100 References: <20000117170228.A10072@cnri.reston.va.us> <000d01bf619b$31551d30$4500a8c0@thomasnotebook> Message-ID: <20000118083826.A10383@cnri.reston.va.us> On 18 January 2000, Thomas Heller said: > 1. Syntax errors in (unused?) commands/build_lib.py (look, you will see > them) Oops, unfinished code. That's for 0.2, not 0.1.3... > 2. _spawn_nt() still raises an OSError when the command is not found, > see the traceback I sent you yesterday. Fix follows: [...] > 3. There is still the indentation error in dist.py: make_zipfile (line 475) D'ohh! Sorry... I forgot to sync from my working directory to python.org. You downloaded yesterday's premature code snapshot, the one that didn't have these two fixes. Please try again -- it's still called distutils-20000117.tar.gz, but it's slightly newer. wiping egg off face... Greg From fdrake@acm.org Tue Jan 18 18:03:23 2000 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Tue, 18 Jan 2000 13:03:23 -0500 (EST) Subject: [Distutils] Package Meta-information Patch In-Reply-To: <3883CF6A.3D3F43B2@enduden.com> References: <386966F0.AA74F1DF@cloud9.net> <20000117121558.D9577@cnri.reston.va.us> <3883CF6A.3D3F43B2@enduden.com> Message-ID: <14468.43755.927408.811728@weyr.cnri.reston.va.us> Michael Muller writes: > itself is readable), and because it was the only format that I was > _absolutely certain_ would be available on every platform that > would be using distutils. No, it's fair to assume that the entire standard library is available. That makes available ConfigParser, rfc822, shlib, and xmllib. Writing a parser *is* an option, because it can be incorporated into distutils, but I don't see any need to do so with so many already-debugged modules available. > Also, there's the fact that with Python code, the interpreter is > written in C - it will run significantly faster than any code that > could be written in Python. And the first malevolent packager to add a "while 1" loop to his package metadata breaks the whole system. If performance is really an issue, C and Java implementations can be built as needed. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From mmuller@enduden.com Tue Jan 18 23:53:23 2000 From: mmuller@enduden.com (Michael Muller) Date: Tue, 18 Jan 2000 18:53:23 -0500 Subject: [Distutils] Package Meta-information Patch References: <386966F0.AA74F1DF@cloud9.net> <20000117121558.D9577@cnri.reston.va.us> <3883CF6A.3D3F43B2@enduden.com> <14468.43755.927408.811728@weyr.cnri.reston.va.us> Message-ID: <3884FCF3.69B798C@enduden.com> "Fred L. Drake, Jr." wrote: > No, it's fair to assume that the entire standard library is > available. That makes available ConfigParser, rfc822, shlib, and > xmllib. Writing a parser *is* an option, because it can be > incorporated into distutils, but I don't see any need to do so with so > many already-debugged modules available. You are correct (with the exception of "shlib" - which I seem to be unable to import; new module?). However, none of these was as easy to use as this: execfile(os.path.join(default_location, name), globals) Writing the file was slightly more difficult, it took a whopping 9 lines of code. > And the first malevolent packager to add a "while 1" loop to his > package metadata breaks the whole system. > If performance is really an issue, C and Java implementations can be > built as needed. Maybe I'm missing something here, but I fail to see how this possibility is any more threatening than that of a malevolent packager installing viral code on your system. In the system that I have submitted, the packager does not directly provide the meta-database file (sorry :-). The manipulation of these files is performed through the pkginfo module, which can encapsulate any kind of information repository that you like. Nonetheless, as I said, I'm not a particularly strong proponent of using python code to store this information, myself. If everybody thinks that this approach sucks, that's fine. I'm much more concerned with the content and the interface to this system. ============================================================================= michaelMuller = mmuller@enduden.com | http://www.cloud9.net/~proteus ----------------------------------------------------------------------------- Society in every state is a blessing, but government even in its best state is but a necessary evil; in its worst state an intolerable one... - Thomas Paine ============================================================================= From fdrake@acm.org Tue Jan 18 23:37:32 2000 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Tue, 18 Jan 2000 18:37:32 -0500 (EST) Subject: [Distutils] Package Meta-information Patch In-Reply-To: <3884FCF3.69B798C@enduden.com> References: <386966F0.AA74F1DF@cloud9.net> <20000117121558.D9577@cnri.reston.va.us> <3883CF6A.3D3F43B2@enduden.com> <14468.43755.927408.811728@weyr.cnri.reston.va.us> <3884FCF3.69B798C@enduden.com> Message-ID: <14468.63804.615523.719527@weyr.cnri.reston.va.us> Michael Muller writes: > You are correct (with the exception of "shlib" - which I seem to be > unable to import; new module?). However, none of these was as easy > to use as this: shlib was in 1.5.2, I think, but it is fairly new. Perhaps the ease of use indicates that the interface to modules like ConfigParser needs to be enhanced. You're right: it *should* be easy! > Maybe I'm missing something here, but I fail to see how this possibility is > any more threatening than that of a malevolent packager installing viral > code on your system. Even if buggy code is installed, the package manager itself should remain usable so that it can be removed easily. > In the system that I have submitted, the packager does not directly provide > the meta-database file (sorry :-). The manipulation of these files is > performed through the pkginfo module, which can encapsulate any kind of > information repository that you like. That does help; my misunderstanding. But I still think a non-code syntax is preferable. Something like an .ini file is very readable and is already familiar to sys-admin types; Python syntax is almost there, but not quite. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From gstein@lyra.org Wed Jan 19 00:19:51 2000 From: gstein@lyra.org (Greg Stein) Date: Tue, 18 Jan 2000 16:19:51 -0800 (PST) Subject: [Distutils] Package Meta-information Patch In-Reply-To: <3884FCF3.69B798C@enduden.com> Message-ID: On Tue, 18 Jan 2000, Michael Muller wrote: >... > You are correct (with the exception of "shlib" - which I seem to be unable to > import; new module?). However, none of these was as easy to use as this: > > execfile(os.path.join(default_location, name), globals) I use: parser = ConfigParser.ConfigParser() parser.read(os.path.join(default_location, name) The biggest difference between the ConfigParser and the execfile() approach is that the latter can easily create complicated structures. For the ConfigParser, you may need to do something like: files = map(string.strip, string.split(parser.get('info', 'files'), ',')) For each "structured" entry. > Writing the file was slightly more difficult, it took a whopping 9 lines of > code. Writing a .ini might be a bit longer, but it mostly depends on the input structures and the number of output sections. Cheers, -g -- Greg Stein, http://www.lyra.org/ From fdrake@acm.org Wed Jan 19 13:46:58 2000 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Wed, 19 Jan 2000 08:46:58 -0500 (EST) Subject: [Distutils] Package Meta-information Patch In-Reply-To: <14468.63804.615523.719527@weyr.cnri.reston.va.us> References: <386966F0.AA74F1DF@cloud9.net> <20000117121558.D9577@cnri.reston.va.us> <3883CF6A.3D3F43B2@enduden.com> <14468.43755.927408.811728@weyr.cnri.reston.va.us> <3884FCF3.69B798C@enduden.com> <14468.63804.615523.719527@weyr.cnri.reston.va.us> Message-ID: <14469.49234.4629.892641@weyr.cnri.reston.va.us> Fred L. Drake, Jr. writes: > shlib was in 1.5.2, I think, but it is fairly new. I checked up on this; I was thinking of shlex, which was new in 1.5.2. It was contributed by Eric Raymond, and provides support for parsing shell-like syntaxes. -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From mmuller@enduden.com Fri Jan 21 01:31:07 2000 From: mmuller@enduden.com (Michael Muller) Date: Thu, 20 Jan 2000 20:31:07 -0500 Subject: [Distutils] Package Meta-information Patch References: Message-ID: <3887B6DB.AD23EA8@enduden.com> Greg Stein wrote: > The biggest difference between the ConfigParser and the execfile() > approach is that the latter can easily create complicated structures. For > the ConfigParser, you may need to do something like: > > files = map(string.strip, > string.split(parser.get('info', 'files'), ',')) > > For each "structured" entry. This has been one of my main objections to using ConfigParser on the occasions when I've needed such a thing. However, as you've indicated, it is easily overcome. If everyone is good with using ConfigParser, I'd be happy to submit another patch. ============================================================================= michaelMuller = mmuller@enduden.com | http://www.cloud9.net/~proteus ----------------------------------------------------------------------------- Mantra for the 60's: Tune in, turn on, drop out Mantra for the 90's: Turn on, jack in, jerk off ============================================================================= From strang@NMR.MGH.Harvard.EDU Tue Jan 25 00:56:56 2000 From: strang@NMR.MGH.Harvard.EDU (Gary Strangman) Date: Mon, 24 Jan 2000 19:56:56 -0500 (EST) Subject: [Distutils] WinNT and distutils Message-ID: Hello all, I'm brand new to distutils, but wanted to install the newest numpy on my WinNT machine to try out some recent array-casting enhancements. Downloading distutils-20000117.tar.gz and installing with "python setup.py install" seemed to go fine. Then I downloaded Numerical-15.2.tar.gz from sourceforge and got the following ... C:\temp\Numerical-15.2> python setup.py build install running build running build_py not copying Lib\ArrayPrinter.py (output up-to-date) <=== 2nd run of above, not copying Lib\FFT.py (output up-to-date) <=== 1st run did copy not copying Lib\LinearAlgebra.py (output up-to-date) not copying Lib\MLab.py (output up-to-date) not copying Lib\Matrix.py (output up-to-date) not copying Lib\Numeric.py (output up-to-date) not copying Lib\Precision.py (output up-to-date) not copying Lib\RandomArray.py (output up-to-date) not copying Lib\UserArray.py (output up-to-date) running build_ext cl.exe /nologo /Ox /MD -Id:\program files\python\include\python1.5 -Id:\program files\python\include -IInclude /c /FoSrc/_numpymodule.obj /TcSrc/_numpymodule.c Traceback (innermost last): File "setup.py", line 58, in ? ext_modules = [('_numpy', File "d:\program files\python\distutils\core.py", line 96, in setup dist.run_commands () File "d:\program files\python\distutils\core.py", line 442, in run_commands self.run_command (cmd) File "d:\program files\python\distutils\core.py", line 491, in run_command cmd_obj.run () File "d:\program files\python\distutils\command\build.py", line 52, in run self.run_peer ('build_ext') File "d:\program files\python\distutils\core.py", line 802, in run_peer self.distribution.run_command (command) File "d:\program files\python\distutils\core.py", line 491, in run_command cmd_obj.run () File "d:\program files\python\distutils\command\build_ext.py", line 146, in run self.build_extensions (self.extensions) File "d:\program files\python\distutils\command\build_ext.py", line 209, in build_extensions include_dirs=include_dirs) File "D:\Program Files\Python\distutils\msvccompiler.py", line 103, in compile self.spawn ([self.cc] + cc_args) File "D:\Program Files\Python\distutils\ccompiler.py", line 431, in spawn spawn (cmd, verbose=self.verbose, dry_run=self.dry_run) File "D:\Program Files\Python\distutils\spawn.py", line 37, in spawn _spawn_nt (cmd, search_path, verbose, dry_run) File "D:\Program Files\Python\distutils\spawn.py", line 70, in _spawn_nt raise DistutilsExecError, \ distutils.errors.DistutilsExecError: command 'cl.exe' failed: No such file or directory C:\temp\Numerical-15.2> Is this a known problem? In the distutils test/ directory, I was able to successfully run every program except test_fgo.py and test_getopt.py (which gave ImportErrors on distutils.exceptions) and test_spawn.py (which crashed when looking for /bin/ls). Any assistance would be most gratefully accepted (please copy any replies to my email address). Gary From thomas.heller@ion-tof.com Tue Jan 25 07:33:42 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Tue, 25 Jan 2000 08:33:42 +0100 Subject: [Distutils] WinNT and distutils References: Message-ID: <035e01bf6706$816cf750$4500a8c0@thomasnotebook> [...] > running build_ext > cl.exe /nologo /Ox /MD -Id:\program files\python\include\python1.5 > -Id:\program files\python\include -IInclude /c /FoSrc/_numpymodule.obj /TcSrc/_numpymodule.c > Traceback (innermost last): > File "setup.py", line 58, in ? > ext_modules = [('_numpy', > File "d:\program files\python\distutils\core.py", line 96, in setup > dist.run_commands () > File "d:\program files\python\distutils\core.py", line 442, in run_commands > self.run_command (cmd) > File "d:\program files\python\distutils\core.py", line 491, in run_command > cmd_obj.run () > File "d:\program files\python\distutils\command\build.py", line 52, in run > self.run_peer ('build_ext') > File "d:\program files\python\distutils\core.py", line 802, in run_peer > self.distribution.run_command (command) > File "d:\program files\python\distutils\core.py", line 491, in run_command > cmd_obj.run () > File "d:\program files\python\distutils\command\build_ext.py", line 146, in run > self.build_extensions (self.extensions) > File "d:\program files\python\distutils\command\build_ext.py", line 209, in build_extensions > include_dirs=include_dirs) > File "D:\Program Files\Python\distutils\msvccompiler.py", line 103, in compile > self.spawn ([self.cc] + cc_args) > File "D:\Program Files\Python\distutils\ccompiler.py", line 431, in spawn > spawn (cmd, verbose=self.verbose, dry_run=self.dry_run) > File "D:\Program Files\Python\distutils\spawn.py", line 37, in spawn > _spawn_nt (cmd, search_path, verbose, dry_run) > File "D:\Program Files\Python\distutils\spawn.py", line 70, in _spawn_nt > raise DistutilsExecError, \ > distutils.errors.DistutilsExecError: command 'cl.exe' failed: No such file or directory > C:\temp\Numerical-15.2> > > Is this a known problem? In the distutils test/ directory, I was able to > successfully run every program except test_fgo.py and test_getopt.py > (which gave ImportErrors on distutils.exceptions) and test_spawn.py (which > crashed when looking for /bin/ls). > You have to make sure that the environment is set up for VC. Run the file vcvars32.bat in the Visual C bin directory. Thomas Heller From mal@lemburg.com Tue Jan 25 08:38:57 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 25 Jan 2000 09:38:57 +0100 Subject: [Distutils] WinNT and distutils References: <035e01bf6706$816cf750$4500a8c0@thomasnotebook> Message-ID: <388D6121.A4D31685@lemburg.com> Thomas Heller wrote: > > [...] > > running build_ext > > [errors running MSVC] > > > You have to make sure that the environment is set up for VC. Run the file > vcvars32.bat > in the Visual C bin directory. Perhaps the msvccompiler.py class should take care of this too... it would sure help to avoid a distutils FAQ. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From robin@jessikat.demon.co.uk Tue Jan 25 10:53:13 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Tue, 25 Jan 2000 10:53:13 +0000 Subject: [Distutils] WinNT and distutils In-Reply-To: <388D6121.A4D31685@lemburg.com> References: <035e01bf6706$816cf750$4500a8c0@thomasnotebook> <388D6121.A4D31685@lemburg.com> Message-ID: In article <388D6121.A4D31685@lemburg.com>, M.-A. Lemburg writes >Thomas Heller wrote: >> >> [...] >> > running build_ext >> > [errors running MSVC] >> > >> You have to make sure that the environment is set up for VC. Run the file >> vcvars32.bat >> in the Visual C bin directory. > >Perhaps the msvccompiler.py class should take care of this too... >it would sure help to avoid a distutils FAQ. > Surely msvccompiler can tell if a particular file (say cl.exe) isn't in any of the well known places. Then it can just issue the correct commands with the path attached or tell the user it can't find cl.exe/link.exe and then ask for vcvars32.bat to be run first. I assume we're not going to search all known drives for the correct exe's. I guess we're not allowed to look in the registry unless Mark Hammond's things are around. -- Robin Becker From mal@lemburg.com Tue Jan 25 11:56:31 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 25 Jan 2000 12:56:31 +0100 Subject: [Distutils] distutils and mxDateTime Message-ID: <388D8F6F.7FA07303@lemburg.com> I've just tried out the latest distutils version with mxDateTime 1.3.0. Some comments: · The setup.py file does print a help screen, but does not tell the user which commands are available. I think this should be done to help the overloaded sysadmin ;-) · I found that using a file layout like this: setup.py DateTime/ does not work... how do I have to fix setup.py to work in this situation (setup.py being outside the package dir) ? · Tracebacks are nice for debugging, but not so nice for the end user. I'd suggest to only print tracebacks when setup.py is run in -d mode and otherwise only output a single line explaining the error condition (and via some dictionary the possible solution to the problem). · For many of my packages the user will have to edit a Setup file and change paths and/or compile time defines. How can this be done using setup.py ? I think I'd like this procedure to either be semi-automatic (a setdefaults.py module would scan the system and set the switches) or interactive with the user entering paths and answering to questions like: Do you want to setup the subpackage mx.ODBC.Sybase ? [y/n] which then control the install flow later on. I guess this can only be done by creating the setup.py:setup() call parameters on-the-fly... Apart from that: Great work, Greg ! -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From thomas.heller@ion-tof.com Wed Jan 26 07:59:59 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Wed, 26 Jan 2000 08:59:59 +0100 Subject: [Distutils] WinNT and distutils References: <035e01bf6706$816cf750$4500a8c0@thomasnotebook><388D6121.A4D31685@lemburg.com> Message-ID: <00d301bf67d3$59180510$4500a8c0@thomasnotebook> > In article <388D6121.A4D31685@lemburg.com>, M.-A. Lemburg > writes > >Thomas Heller wrote: > >> > >> [...] > >> > running build_ext > >> > [errors running MSVC] > >> > > >> You have to make sure that the environment is set up for VC. Run the file > >> vcvars32.bat > >> in the Visual C bin directory. > > > >Perhaps the msvccompiler.py class should take care of this too... > >it would sure help to avoid a distutils FAQ. > > > > Surely msvccompiler can tell if a particular file (say cl.exe) isn't in > any of the well known places. Then it can just issue the correct > commands with the path attached or tell the user it can't find > cl.exe/link.exe and then ask for vcvars32.bat to be run first. I assume > we're not going to search all known drives for the correct exe's. > > > I guess we're not allowed to look in the registry unless Mark Hammond's > things are around. > -- Python should *really* have a standard module allowing (at least) to read the registry. Thomas Heller From stephen.walton@csun.edu Wed Jan 26 23:08:40 2000 From: stephen.walton@csun.edu (Stephen Walton) Date: Wed, 26 Jan 2000 15:08:40 -0800 (PST) Subject: [Distutils] distutils on hpux Message-ID: Hi, I just joined the list but am not sure I should have :-). I really just want to get distutils working on our HP/UX 10.20 host so I can install NumPy. Using the 0.1.2 release, all the tests fail, basically because "from sysconfig import AR" fails. Where should that come from? It appears it should be in the config.h set up when I built Python, but I can't find it. The 2000-1-17 snapshot doesn't install. Here is the error (wrapped to make mailers happy): File "/opt/hppd/lib/python1.5/site-packages/distutils/command/build_lib.py", line 26 self. = None ^ SyntaxError: invalid syntax -- Stephen Walton, Professor, Dept. of Physics & Astronomy, Cal State Northridge stephen.walton@csun.edu From cjensen@bioeng.ucsd.edu Fri Jan 28 00:34:30 2000 From: cjensen@bioeng.ucsd.edu (Curtis Jensen) Date: Thu, 27 Jan 2000 16:34:30 -0800 Subject: [Distutils] Complling on IRIX OS Message-ID: <3890E416.D97B745A@be-research.ucsd.edu> To install on an IRIX OS, sometimes we need to compile with the -o32 flag. I can't seem to find how to do this with distutils. Any ideas? -- Curtis Jensen cjensen@be-research.ucsd.edu http://www-bioeng.ucsd.edu/~cjensen/ FAX (425) 740-1451 From cjensen@bioeng.ucsd.edu Fri Jan 28 00:39:51 2000 From: cjensen@bioeng.ucsd.edu (Curtis Jensen) Date: Thu, 27 Jan 2000 16:39:51 -0800 Subject: [Distutils] stable, working system Message-ID: <3890E557.EB95A1ED@be-research.ucsd.edu> Does anyone find it strange that the new versions of Numeric require Distutils? However the README says: Do not expect perfection: if you're trying out this release, it should be because you want to help debug and develop, not because you need a stable, working system immediately. Something as widely used as Numeric requires a module that is in it's beta version? Is this to force bug reports? -- Curtis Jensen cjensen@be-research.ucsd.edu http://www-bioeng.ucsd.edu/~cjensen/ FAX (425) 740-1451 From gward@cnri.reston.va.us Fri Jan 28 04:48:51 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Thu, 27 Jan 2000 23:48:51 -0500 Subject: [Distutils] WinNT and distutils In-Reply-To: ; from robin@jessikat.demon.co.uk on Tue, Jan 25, 2000 at 10:53:13AM +0000 References: <035e01bf6706$816cf750$4500a8c0@thomasnotebook> <388D6121.A4D31685@lemburg.com> Message-ID: <20000127234851.B12656@cnri.reston.va.us> On 25 January 2000, Robin Becker said: > >Perhaps the msvccompiler.py class should take care of this too... > >it would sure help to avoid a distutils FAQ. I'll happily accept patches! (I'd prefer not to be the one to translate a Microsoft-written batch file into Python: there're reasons I abandoned DOS almost 6 years ago, and batch files were one of them...) > Surely msvccompiler can tell if a particular file (say cl.exe) isn't in > any of the well known places. Depends what you mean by "well-known places". If you mean os.environ['PATH'], then the spawn module can (and should) do a bit of work ahead of time to see if the expected command is really there. Also, somebody somewhere should probably catch that exception and die without a traceback. Not sure whose responsibility that is; I guess it belongs in the compiler class (MSVCCompiler in this case). If you mean encoding some knowledge about where MSVC++ is "typically" installed, then that might belong in the MSVCCompiler class. I'll need a good explanation and broad consensus from the Windows experts in the crowd before I'll put that kind of code in, though. As for dreaming about searching the registry: as long as it's not available under Python 1.5.*, then I don't want Distutils to depend on it. Greg From gward@cnri.reston.va.us Fri Jan 28 05:04:43 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Fri, 28 Jan 2000 00:04:43 -0500 Subject: [Distutils] distutils and mxDateTime In-Reply-To: <388D8F6F.7FA07303@lemburg.com>; from mal@lemburg.com on Tue, Jan 25, 2000 at 12:56:31PM +0100 References: <388D8F6F.7FA07303@lemburg.com> Message-ID: <20000128000442.C12656@cnri.reston.va.us> On 25 January 2000, M.-A. Lemburg said: > · The setup.py file does print a help screen, but does not > tell the user which commands are available. I think this > should be done to help the overloaded sysadmin ;-) You're not the first to ask. I'll see if I can squeeze that into version 0.1.3 (this weekend perhaps?). > · I found that using a file layout like this: > > setup.py > DateTime/ > > does not work... how do I have to fix setup.py to work > in this situation (setup.py being outside the package dir) ? I assume you're using the mxdatetime_setup.py included in the Distutils distribution, in which case you have the following two options: packages = ['DateTime', 'DateTime.Examples', 'DateTime.mxDateTime'], package_dir = {'DateTime': ''}, Short answer: delete 'package_dir'. Long answer: by putting the DateTime package in the DateTime directory, you are doing the "preferred thing" and thus don't have to tell Distutils where to find your .py source files. The version of mxDateTime I based the setup script on had the DateTime package in the distribution root (the current directory, or ''), so I had to tell Distutils to look there for it -- that's what the 'package_dir' option is for. > · Tracebacks are nice for debugging, but not so nice for > the end user. I'd suggest to only print tracebacks when > setup.py is run in -d mode and otherwise only output > a single line explaining the error condition (and via > some dictionary the possible solution to the problem). Hey, as long as version < 1.0, you're in debugging mode whether you like it or not. I'm not sure that a "debugging mode" flag is the answer. If the exception is due to an error in the code (Distutils code *or* a custom/extended command class), that's a bug and there should always be a traceback. If the exception is due to user error, there should never be a traceback and the presence of one is a bug. I know there are many exceptions that should be turned into a simple one-line "augh! user error, die" termination, but still show tracebacks. Nevertheless, you can keep me honest by reporting these as bugs when you see them. > · For many of my packages the user will have to edit a > Setup file and change paths and/or compile time defines. > How can this be done using setup.py ? It can't (yet), short of forcing users the edit the setup script. (Oh yeah, now I remember why I haven't been pressing you to Distutil-ize mxDateTime...) I think the right thing to do is have config files; same idea as the old-style Setup file, but more structured. > I think I'd like > this procedure to either be semi-automatic (a setdefaults.py > module would scan the system and set the switches) or > interactive with the user entering paths and answering > to questions like: > > Do you want to setup the subpackage mx.ODBC.Sybase ? [y/n] At the risk of sounding dogmatic, my initial reaction to interactive installation is "Absolutely out of the question!". Serial Q&A is not a user interface model I wish to perpetrate or inflict on anyone. > I guess > this can only be done by creating the setup.py:setup() > call parameters on-the-fly... Hmmm, you've been drinking from the cup of PyFort (which generates a setup script based on a Fortan-ish description of Fortran subroutines that will be glued into Python via generated C code... *shudder*). Greg From gward@cnri.reston.va.us Fri Jan 28 05:29:46 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Fri, 28 Jan 2000 00:29:46 -0500 Subject: [Distutils] To-do for Distutils 0.2 (and beyond) Message-ID: <20000128002946.D12656@cnri.reston.va.us> Hi all -- [cc'd to python-dev because this explicitly addresses features to be included in Python 1.6: please keep replies on the distutils-sig!] recent developments in the planned release schedule for Python 1.6 (Guido doesn't want to wait for everything planned, but get something out the door in the next couple of months) are lighting a fire under me to get a seriously usable version of Distutils ready *really* soon now. This will be Distutils 0.2; I anticipate that 0.2.x will be included in Python 1.6, where x is the number of attempts it takes me to get something reasonably bug-free out the door. Those if you who were at the Python Conference this past week will have seen bits and pieces of my "laundry list" of desired features that should be added to the Distutils at some point in the future. Given the shortened schedule, it looks like it's necessary to do some pruning and concentrate on the essentials to get something in Python 1.6. So, here is my current laundry list, sorted into a couple of categories: essential for 0.2 (Python 1.6) ----------------- * fix the "dist" command (two-phase manifest, better feedback) (steal ideas and hopefully code from Gordon Macmillan's installer) * fix the "install" command (have the right interface and do the right thing for installating to non-standard or personal directories) * fix some bad nomenclature in the command classes (most likely 'options' -> 'user_options', 'set_default_options()' -> 'initialize_options()', and 'set_final_options()' -> ??? (maybe 'finalize_options()'?)) * build C libraries (for PIL, and other similar distributions) * documentation (two small manuals that should become standard Python manuals: "Installing Python Modules" and "Developing and Distributing Python Modules") * add a bdist command; should at least be able to generate dumb archives of built distributions (eg. a tarball that you unpack from /usr/local, or maybe /usr/local/lib/python1.5/site-packages) desirable for 0.2 ----------------- * "what's installed" database * dependency checking ("is version 1.3 of foo installed?") * don't automatically clobber an existing installation -- confirm, or require a "force" option, or something * command to install C headers (assuming more extensions than NumPy need to do this) put off to 0.3 (Python 1.7?) ---------------------------- * JPython support (most importantly, know where to install .py modules when run from JPython and be able to build Java extensions for JPython) * build static Python binary (for shared-library-challenged OSs) * SWIG support (mainly, know how to run it before building the C extension it generates) * PyFort support (ditto) * Mac support * allow overlapping multi-architecture installations (Michel Sanner's pet peeve and Guido's nightmare ;-) (this would *not* be the default; it would have to be explicitly chosen by brave/cheap/foolhardy installers) * support for archive sites (Parnassus) to pull out meta-info Anyone feel strongly about moving any of these items around, or discarding any entirely, or adding anything? Please let me know by email; I'll summarize to the list. Thanks -- Greg From gward@cnri.reston.va.us Fri Jan 28 05:34:22 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Fri, 28 Jan 2000 00:34:22 -0500 Subject: [Distutils] stable, working system In-Reply-To: <3890E557.EB95A1ED@be-research.ucsd.edu>; from cjensen@bioeng.ucsd.edu on Thu, Jan 27, 2000 at 04:39:51PM -0800 References: <3890E557.EB95A1ED@be-research.ucsd.edu> Message-ID: <20000128003421.E12656@cnri.reston.va.us> On 27 January 2000, Curtis Jensen said: > Do not expect perfection: if you're trying out this > release, it should be because you want to help debug and develop, not > because you need a stable, working system immediately. > > Something as widely used as Numeric requires a module that is in it's > beta version? Is this to force bug reports? Calling Distutils "beta" is quite generous. I would say that it's "under development", i.e. pre-alpha. Whatever that means. Paul's decision could be seen as a bit, ummm, controversial. If it smokes out bugs in Distutils -- as it is already doing -- that's definitely a long-term good thing. It does put the poor folks just trying to use Numeric into a bit of a guinea pig situation, though. 'bout all I can say is that I will do my damndest to fix the bugs you folks do find. (Of course, I'm momentarily stumped by the two most recent bug reports, including yours. However, it's after midnight on the last day of the Python Conference, and my brain is quite thoroughly fried. Hopefully I'll be better able to think about compiler options on HP-UX in the morning...) Greg From thomas.heller@ion-tof.com Fri Jan 28 08:42:47 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 28 Jan 2000 09:42:47 +0100 Subject: [Distutils] WinNT and distutils References: <035e01bf6706$816cf750$4500a8c0@thomasnotebook> <388D6121.A4D31685@lemburg.com> <20000127234851.B12656@cnri.reston.va.us> Message-ID: <003a01bf696b$a7673800$4500a8c0@thomasnotebook> > > >Perhaps the msvccompiler.py class should take care of this too... > > >it would sure help to avoid a distutils FAQ. > > I'll happily accept patches! (I'd prefer not to be the one to translate > a Microsoft-written batch file into Python: there're reasons I abandoned > DOS almost 6 years ago, and batch files were one of them...) > > > Surely msvccompiler can tell if a particular file (say cl.exe) isn't in > > any of the well known places. > > Depends what you mean by "well-known places". If you mean > os.environ['PATH'], then the spawn module can (and should) do a bit of > work ahead of time to see if the expected command is really there. On NT, there are three kinds of "well-known places": 1. on the PATH 2. in the default directory which MS suggests when installing MSVC 3. in the real directory where MSVC is installed, assumuming the default has not been accepted. Only 1 is easy. Points 2 and 3 can only be retrieved by searching the registry, because even the default directory depends on the Windows Version: (english) c:\Program Files\... (german) c:\Programme\... Note that not only you have to know where to find cl.exe, you also have to set the INCLUDE and LIB directories! We could assume that a developer who has installed MSVC would know how to set up the environment, but I guess this is usually not the case, because most developers I know of simply fire up DevStudio, they never use the command line. > As for dreaming about searching the registry: as long as it's not > available under Python 1.5.*, then I don't want Distutils to depend on > it. The conclusion is: You can not do too much under NT when you cannot look into the registry. (That reminds me of the 'good old times' when every program added its own directory to the PATH ;-) > > Greg > Thomas Heller From mwh21@cam.ac.uk Fri Jan 28 08:58:52 2000 From: mwh21@cam.ac.uk (Michael Hudson) Date: 28 Jan 2000 08:58:52 +0000 Subject: [Distutils] WinNT and distutils In-Reply-To: "Thomas Heller"'s message of "Fri, 28 Jan 2000 09:42:47 +0100" References: <035e01bf6706$816cf750$4500a8c0@thomasnotebook> <388D6121.A4D31685@lemburg.com> <20000127234851.B12656@cnri.reston.va.us> <003a01bf696b$a7673800$4500a8c0@thomasnotebook> Message-ID: "Thomas Heller" writes: > The conclusion is: > You can not do too much under NT when you cannot look into the registry. > (That reminds me of the 'good old times' when every program added its own > directory > to the PATH ;-) How hard are registry files to parse? They're just text aren't they? (not on windows, can't tell). Would a simple parser written in Python be even vaguely possible? Cheers, M. From gstein@lyra.org Fri Jan 28 09:34:17 2000 From: gstein@lyra.org (Greg Stein) Date: Fri, 28 Jan 2000 01:34:17 -0800 (PST) Subject: [Distutils] WinNT and distutils In-Reply-To: Message-ID: On 28 Jan 2000, Michael Hudson wrote: >... > How hard are registry files to parse? They're just text aren't they? > (not on windows, can't tell). Would a simple parser written in Python > be even vaguely possible? Registry files are binary files. You're thinking of a .reg file which can be passed to regsvr32. .reg files have syntax similar to .ini files. In short: no, you cannot parse them. You must have access to the Win32 registry functions. There has been discussion about putting them into Python 1.6. Cheers, -g -- Greg Stein, http://www.lyra.org/ From robin@jessikat.demon.co.uk Fri Jan 28 12:58:53 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Fri, 28 Jan 2000 12:58:53 +0000 Subject: [Distutils] WinNT and distutils In-Reply-To: References: Message-ID: In article , Greg Stein writes >On 28 Jan 2000, Michael Hudson wrote: >>... >> How hard are registry files to parse? They're just text aren't they? >> (not on windows, can't tell). Would a simple parser written in Python >> be even vaguely possible? > >Registry files are binary files. You're thinking of a .reg file which can >be passed to regsvr32. .reg files have syntax similar to .ini files. > >In short: no, you cannot parse them. You must have access to the Win32 >registry functions. > >There has been discussion about putting them into Python 1.6. > >Cheers, >-g > I offer this as a way to go if win32 is installed. import string def msvc_get_paths(path, vNum='6.0', platform='x86'): "Get a devstudio path (include, library or bin)" try: import win32api import win32con except ImportError: return None L = [] if path=='bin': path = 'Path' path = string.upper(path + ' Dirs') for base in (win32con.HKEY_CLASSES_ROOT,win32con.HKEY_LOCAL_MACHINE, win32con.HKEY_CURRENT_USER,win32con.HKEY_USERS): try: k = 'Software\\Microsoft\\Devstudio\\%s\\Build System\\Components\\Platforms\\Win32 (%s)\\Directories' \ % (vNum,platform) k = win32api.RegOpenKeyEx(base,k) i = 0 while 1: try: (p,v,t) = win32api.RegEnumValue(k,i) if string.upper(p)==path: V = string.split(v,';') for v in V: if v=='' or v in L: continue L.append(v) break i = i + 1 except win32api.error: break except win32api.error: pass return L -- Robin Becker From mal@lemburg.com Fri Jan 28 09:56:13 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 28 Jan 2000 10:56:13 +0100 Subject: [Distutils] WinNT and distutils References: <035e01bf6706$816cf750$4500a8c0@thomasnotebook> <388D6121.A4D31685@lemburg.com> <20000127234851.B12656@cnri.reston.va.us> Message-ID: <389167BD.C4FE7D4D@lemburg.com> Greg Ward wrote: > > > >Perhaps the msvccompiler.py class should take care of this too... > > >it would sure help to avoid a distutils FAQ. > > I'll happily accept patches! (I'd prefer not to be the one to translate > a Microsoft-written batch file into Python: there're reasons I abandoned > DOS almost 6 years ago, and batch files were one of them...) What I had in mind was that distutils should execute vcvars32.bat prior to calling calling the compiler. Should be simple enough, I guess, although I'm not sure how you have implemented OS environment inheritance... could be that you have to execute vcvars32.bat prior to *every* call to MSVC. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From mal@lemburg.com Fri Jan 28 10:21:10 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 28 Jan 2000 11:21:10 +0100 Subject: [Distutils] distutils and mxDateTime References: <388D8F6F.7FA07303@lemburg.com> <20000128000442.C12656@cnri.reston.va.us> Message-ID: <38916D96.E85998A9@lemburg.com> Greg Ward wrote: > > On 25 January 2000, M.-A. Lemburg said: > > · The setup.py file does print a help screen, but does not > > tell the user which commands are available. I think this > > should be done to help the overloaded sysadmin ;-) > > You're not the first to ask. I'll see if I can squeeze that into > version 0.1.3 (this weekend perhaps?). Cool. > > · I found that using a file layout like this: > > > > setup.py > > DateTime/ > > > > does not work... how do I have to fix setup.py to work > > in this situation (setup.py being outside the package dir) ? > > I assume you're using the mxdatetime_setup.py included in the Distutils > distribution, in which case you have the following two options: > > packages = ['DateTime', 'DateTime.Examples', 'DateTime.mxDateTime'], > package_dir = {'DateTime': ''}, > > Short answer: delete 'package_dir'. > > Long answer: by putting the DateTime package in the DateTime directory, > you are doing the "preferred thing" and thus don't have to tell > Distutils where to find your .py source files. The version of > mxDateTime I based the setup script on had the DateTime package in the > distribution root (the current directory, or ''), so I had to tell > Distutils to look there for it -- that's what the 'package_dir' option > is for. Ah, ok. > > · Tracebacks are nice for debugging, but not so nice for > > the end user. I'd suggest to only print tracebacks when > > setup.py is run in -d mode and otherwise only output > > a single line explaining the error condition (and via > > some dictionary the possible solution to the problem). > > > Hey, as long as version < 1.0, you're in debugging mode whether you like > it or not. > > > I'm not sure that a "debugging mode" flag is the answer. If the > exception is due to an error in the code (Distutils code *or* a > custom/extended command class), that's a bug and there should always be > a traceback. If the exception is due to user error, there should never > be a traceback and the presence of one is a bug. I know there are many > exceptions that should be turned into a simple one-line "augh! user > error, die" termination, but still show tracebacks. Nevertheless, you > can keep me honest by reporting these as bugs when you see them. Well, if it's a bug, the user can run setup.py in -d mode and then extract the traceback. It is just that user errors producing those deeply nested tracebacks will keep users from even thinking that they made a mistake ;-) > > · For many of my packages the user will have to edit a > > Setup file and change paths and/or compile time defines. > > How can this be done using setup.py ? > > It can't (yet), short of forcing users the edit the setup script. (Oh > yeah, now I remember why I haven't been pressing you to Distutil-ize > mxDateTime...) I think the right thing to do is have config files; same > idea as the old-style Setup file, but more structured. You mean a setup.py file containing Python code plus a config.ini file for the user adjustments. Sounds fine to me... In that case I'd like an option on setup.py which allows me to preedit the config.ini file via a Python module, something like "setup.py --boot" which then calls a function defined in setup.py to preprocess config.ini. This function could also do the interactive part I mentioned below, BTW. > > I think I'd like > > this procedure to either be semi-automatic (a setdefaults.py > > module would scan the system and set the switches) or > > interactive with the user entering paths and answering > > to questions like: > > > > Do you want to setup the subpackage mx.ODBC.Sybase ? [y/n] > > At the risk of sounding dogmatic, my initial reaction to interactive > installation is "Absolutely out of the question!". Serial Q&A is not a > user interface model I wish to perpetrate or inflict on anyone. Doesn't matter, I want it anyway ;-) Note that it doesn't have to be tty based: a nice colorful installer could take care of this part. Seriously, given the above setup.py + config.ini file configuration the interactive parts could be coded in the boot function. Would be nice if there were an option --interactive, -i which enables a switch to do interactive setup usable by the boot function. > > I guess > > this can only be done by creating the setup.py:setup() > > call parameters on-the-fly... > > Hmmm, you've been drinking from the cup of PyFort (which generates a > setup script based on a Fortan-ish description of Fortran subroutines > that will be glued into Python via generated C code... *shudder*). Provided we find a good intergration of the config.ini data with the setup() call parameters, this may not be as bad as expected. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From thomas.heller@ion-tof.com Fri Jan 28 13:28:13 2000 From: thomas.heller@ion-tof.com (Thomas Heller) Date: Fri, 28 Jan 2000 14:28:13 +0100 Subject: [Distutils] WinNT and distutils References: <035e01bf6706$816cf750$4500a8c0@thomasnotebook> <388D6121.A4D31685@lemburg.com> <20000127234851.B12656@cnri.reston.va.us> <389167BD.C4FE7D4D@lemburg.com> Message-ID: <015c01bf6993$8aca3580$4500a8c0@thomasnotebook> > Greg Ward wrote: > > > > > >Perhaps the msvccompiler.py class should take care of this too... > > > >it would sure help to avoid a distutils FAQ. > > > > I'll happily accept patches! (I'd prefer not to be the one to translate > > a Microsoft-written batch file into Python: there're reasons I abandoned > > DOS almost 6 years ago, and batch files were one of them...) > > What I had in mind was that distutils should execute vcvars32.bat > prior to calling calling the compiler. Should be simple enough, > I guess, although I'm not sure how you have implemented OS > environment inheritance... could be that you have to execute > vcvars32.bat prior to *every* call to MSVC. Does not work, because you do not know where to find it. It is *not* on the path, it is in the VC/bin directory. Maybe even that it is optional (I do not remember). Thomas Heller From mwh21@cam.ac.uk Fri Jan 28 14:39:09 2000 From: mwh21@cam.ac.uk (Michael Hudson) Date: 28 Jan 2000 14:39:09 +0000 Subject: [Distutils] WinNT and distutils In-Reply-To: Greg Stein's message of "Fri, 28 Jan 2000 01:34:17 -0800 (PST)" References: Message-ID: Greg Stein writes: > On 28 Jan 2000, Michael Hudson wrote: > >... > > How hard are registry files to parse? They're just text aren't they? > > (not on windows, can't tell). Would a simple parser written in Python > > be even vaguely possible? > > Registry files are binary files. You're thinking of a .reg file which can > be passed to regsvr32. .reg files have syntax similar to .ini files. Yup, you're right. > In short: no, you cannot parse them. You must have access to the Win32 > registry functions. Presumably Microsoft change the binary format every six months for fun, too... > There has been discussion about putting them into Python 1.6. That would strike me as quite a good idea. It's hard to see how distutils can function *really* well on Windows without them. Cheers, Michael From gstein@lyra.org Fri Jan 28 14:45:46 2000 From: gstein@lyra.org (Greg Stein) Date: Fri, 28 Jan 2000 06:45:46 -0800 (PST) Subject: [Distutils] WinNT and distutils In-Reply-To: Message-ID: On 28 Jan 2000, Michael Hudson wrote: > Greg Stein writes: >... > > Registry files are binary files. You're thinking of a .reg file which can > > be passed to regsvr32. .reg files have syntax similar to .ini files. > > Yup, you're right. > > > In short: no, you cannot parse them. You must have access to the Win32 > > registry functions. > > Presumably Microsoft change the binary format every six months for > fun, too... Heh. Guess Microsoft isn't the only advocate of FUD. > > There has been discussion about putting them into Python 1.6. > > That would strike me as quite a good idea. It's hard to see how > distutils can function *really* well on Windows without them. I definitely agree. However, Greg Ward has this requirement in his head: distutils must work on Python 1.5. Therefore, you may have some problems here. Go fight him about it, though. I'll stay on the sidelines :-) Cheers, -g -- Greg Stein, http://www.lyra.org/ From mal@lemburg.com Fri Jan 28 13:54:46 2000 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 28 Jan 2000 14:54:46 +0100 Subject: [Distutils] WinNT and distutils References: <035e01bf6706$816cf750$4500a8c0@thomasnotebook> <388D6121.A4D31685@lemburg.com> <20000127234851.B12656@cnri.reston.va.us> <389167BD.C4FE7D4D@lemburg.com> <015c01bf6993$8aca3580$4500a8c0@thomasnotebook> Message-ID: <38919FA6.E517F54C@lemburg.com> Thomas Heller wrote: > > > Greg Ward wrote: > > > > > > > >Perhaps the msvccompiler.py class should take care of this too... > > > > >it would sure help to avoid a distutils FAQ. > > > > > > I'll happily accept patches! (I'd prefer not to be the one to translate > > > a Microsoft-written batch file into Python: there're reasons I abandoned > > > DOS almost 6 years ago, and batch files were one of them...) > > > > What I had in mind was that distutils should execute vcvars32.bat > > prior to calling calling the compiler. Should be simple enough, > > I guess, although I'm not sure how you have implemented OS > > environment inheritance... could be that you have to execute > > vcvars32.bat prior to *every* call to MSVC. > > Does not work, because you do not know where to find it. > It is *not* on the path, it is in the VC/bin directory. > Maybe even that it is optional (I do not remember). I have that directory on my PATH... I don't remember whether I put it there or the MSVC installation did, though. Even if it cannot be found: trying to run vcvars32.bat prior to executing cl.exe should be ok in any case... -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ From fdrake@acm.org Fri Jan 28 21:05:33 2000 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 28 Jan 2000 16:05:33 -0500 (EST) Subject: [Distutils] WinNT and distutils In-Reply-To: References: Message-ID: <14482.1181.29982.349164@weyr.cnri.reston.va.us> Greg Stein writes: > However, Greg Ward has this requirement in his head: distutils must work > on Python 1.5. Therefore, you may have some problems here. As Guido pointed out at IPC8, distutils is most valuable if it doesn't need to be included with the packages that use it and doesn't need to be installed separately. So it's really not "there" until 1.6 anyway. Drop support for 1.5.2 and we're in good shape. Got that, Greg (Ward)? ;) -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From cjensen@bioeng.ucsd.edu Fri Jan 28 22:29:15 2000 From: cjensen@bioeng.ucsd.edu (Curtis Jensen) Date: Fri, 28 Jan 2000 14:29:15 -0800 Subject: [Distutils] Numeric Install over old existing Numeric Message-ID: <3892183B.61479EF7@be-research.ucsd.edu> [SGI Octane; IRIX 6.5.2] I tried installing Numeric 14 over an older version of Numeric (Not sure which one). When copying the files to /usr/local/Python/lib/python1.5/site-packages/Numeric/ if it was trying to copy a file with the same name as the older version, the setup script gave me an error something to the affect of "Operation not allowed" (Sorry I don't have the actual error syntax). However the file was copied over the old one, and if I ran the setup script again then it would proceed to the next file. I also removed all the files with the same name, just in case the files were just touched and not actualy copied. I got it to work. Just so you know that there is a problem with an install over an existing version. -- Curtis Jensen cjensen@be-research.ucsd.edu http://www-bioeng.ucsd.edu/~cjensen/ FAX (425) 740-1451 From gward@cnri.reston.va.us Sat Jan 29 00:02:17 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Fri, 28 Jan 2000 19:02:17 -0500 Subject: [Distutils] WinNT and distutils In-Reply-To: <14482.1181.29982.349164@weyr.cnri.reston.va.us>; from fdrake@acm.org on Fri, Jan 28, 2000 at 04:05:33PM -0500 References: <14482.1181.29982.349164@weyr.cnri.reston.va.us> Message-ID: <20000128190216.B12965@cnri.reston.va.us> On 28 January 2000, Fred L. Drake, Jr. said: > As Guido pointed out at IPC8, distutils is most valuable if it > doesn't need to be included with the packages that use it and doesn't > need to be installed separately. So it's really not "there" until 1.6 > anyway. Drop support for 1.5.2 and we're in good shape. > Got that, Greg (Ward)? ;) Yeah, I've been (slowly, quietly) moving towards that position anyways. Using all sorts of 1.6 features would make it a bit hard to develop, test, and debug the Distutils -- I guess at some point I'll have to break down and start running the CVS version of Python instead of the standard Red Hat RPM. ;-) What 1.6 features are really, absolutely necessary, though? So far I see only one: * registry access on Windows Others? Oh yeah, will the registry-access code that Robin Becker posted work under 1.5.2 with Mark's extensions installed as well as under 1.6 with built-in registry support? Greg From gstein@lyra.org Sat Jan 29 00:08:07 2000 From: gstein@lyra.org (Greg Stein) Date: Fri, 28 Jan 2000 16:08:07 -0800 (PST) Subject: [Distutils] WinNT and distutils In-Reply-To: <20000128190216.B12965@cnri.reston.va.us> Message-ID: On Fri, 28 Jan 2000, Greg Ward wrote: >... > What 1.6 features are really, absolutely necessary, though? So far I > see only one: > > * registry access on Windows > > Others? Some people have suggested that distutils does some of its install work via the new import stuff. I'm not sure what that entails (I don't understand why distutils would need the custom importing/packaging that imputil provides). Also, I don't have feedback from Guido yet on what needs to happen to imputil before its incorporation into 1.6. So I'd count this as a "thought" rather than a requirement. > Oh yeah, will the registry-access code that Robin Becker posted work > under 1.5.2 with Mark's extensions installed as well as under 1.6 with > built-in registry support? It won't work until 1.6 because we don't have the 1.6 design yet :-) But, there is no reason it shouldn't (with some creative try/except around some imports). Note that win32api isn't going into 1.6 -- either a new module from scratch, or a small extraction of win32api. In any case, it certainly won't (can't) be called win32api. Cheers, -g -- Greg Stein, http://www.lyra.org/ From guido@CNRI.Reston.VA.US Sat Jan 29 02:51:18 2000 From: guido@CNRI.Reston.VA.US (Guido van Rossum) Date: Fri, 28 Jan 2000 21:51:18 -0500 Subject: [Distutils] WinNT and distutils In-Reply-To: Your message of "Fri, 28 Jan 2000 19:02:17 EST." <20000128190216.B12965@cnri.reston.va.us> References: <14482.1181.29982.349164@weyr.cnri.reston.va.us> <20000128190216.B12965@cnri.reston.va.us> Message-ID: <200001290251.VAA04695@eric.cnri.reston.va.us> > Oh yeah, will the registry-access code that Robin Becker posted work > under 1.5.2 with Mark's extensions installed as well as under 1.6 with > built-in registry support? Where do you get the impression that 1.6 will have Windows registry access built in? I didn't even know that was on the wish list. (If it is, maybe someone can mail me code?) --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Sat Jan 29 03:56:03 2000 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Fri, 28 Jan 2000 22:56:03 -0500 (EST) Subject: [Distutils] WinNT and distutils In-Reply-To: <20000128190216.B12965@cnri.reston.va.us> References: <14482.1181.29982.349164@weyr.cnri.reston.va.us> <20000128190216.B12965@cnri.reston.va.us> Message-ID: <14482.25811.673319.173880@weyr.cnri.reston.va.us> Greg Ward writes: > What 1.6 features are really, absolutely necessary, though? So far I > see only one: > > * registry access on Windows If that's the only dependency we need, that's all that we need. I don't think we need to introduce dependencies artificially. If it works with 1.5.2 on Unix & 1.6 on Windows, I'd be quite happy. With the proposed timeline for 1.6, that seems quite acceptable. > Oh yeah, will the registry-access code that Robin Becker posted work > under 1.5.2 with Mark's extensions installed as well as under 1.6 with > built-in registry support? This I don't know. ;) -Fred -- Fred L. Drake, Jr. Corporation for National Research Initiatives From gstein@lyra.org Sat Jan 29 10:57:05 2000 From: gstein@lyra.org (Greg Stein) Date: Sat, 29 Jan 2000 02:57:05 -0800 (PST) Subject: [Distutils] WinNT and distutils In-Reply-To: <200001290251.VAA04695@eric.cnri.reston.va.us> Message-ID: On Fri, 28 Jan 2000, Guido van Rossum wrote: > > Oh yeah, will the registry-access code that Robin Becker posted work > > under 1.5.2 with Mark's extensions installed as well as under 1.6 with > > built-in registry support? > > Where do you get the impression that 1.6 will have Windows registry > access built in? I didn't even know that was on the wish list. (If > it is, maybe someone can mail me code?) It will appear in time, or it won't. :-) [ with the caveat of not knowing when "in time" is :-) ] Thomas Heller has offered to write such an extension. Cheers, -g -- Greg Stein, http://www.lyra.org/ From dubois1@llnl.gov Sat Jan 29 15:43:40 2000 From: dubois1@llnl.gov (Paul F. Dubois) Date: Sat, 29 Jan 2000 07:43:40 -0800 Subject: [Distutils] stable, working system In-Reply-To: <20000128003421.E12656@cnri.reston.va.us> Message-ID: Greg wrote: Paul's decision could be seen as a bit, ummm, controversial. -- Consider the alternative: maintaining some sort of similar utility myself. I think calling this utility "pre-alpha" is nonsense. I used it because it works. I've used it on several things. Yes, it doesn't do everything for everybody. But people had similar trouble with various platforms with the old system too. From dascher@mindspring.com Sat Jan 29 22:51:00 2000 From: dascher@mindspring.com (David Ascher) Date: Sat, 29 Jan 2000 14:51:00 -0800 Subject: [Distutils] WinNT and distutils References: <035e01bf6706$816cf750$4500a8c0@thomasnotebook> <388D6121.A4D31685@lemburg.com> <20000127234851.B12656@cnri.reston.va.us> <003a01bf696b$a7673800$4500a8c0@thomasnotebook> Message-ID: <004501bf6aac$2ca39360$fec3aec7@ski.org> From: Michael Hudson > "Thomas Heller" writes: > > > The conclusion is: > > You can not do too much under NT when you cannot look into the registry. > > (That reminds me of the 'good old times' when every program added its own > > directory > > to the PATH ;-) > > How hard are registry files to parse? They're just text aren't they? > (not on windows, can't tell). Would a simple parser written in Python > be even vaguely possible? No. Registry files are not text files. We're talking about the windows registry here. Registry access functions should be put in the 1.6 tree. I'll talk to Mark about figuring out which subset is needed for common things. If I were Greg, I'd write code to see if the functions were available, use them if they were, and give it the best shot if they're not. Lowest-common denominator leaves everyone running CP/M. --david From dalke@acm.org Sat Jan 29 04:22:57 2000 From: dalke@acm.org (Andrew Dalke) Date: Fri, 28 Jan 2000 21:22:57 -0700 Subject: [Distutils] Complling on IRIX OS Message-ID: <040301bf6ab6$bd6b9640$0005fea9@adalke> Curtis Jensen said: > To install on an IRIX OS, sometimes we need to > compile with the -o32 flag. There are two problems here: 1) Before 1.5.2 (or 1.5.1?), Python for IRIX didn't store the SGI_ABI setting. Thus, the compiler uses the default settings. With 1.5.2, the Makefile.pre.in contains the ABI value. 2) The default ABI changed with IRIX 6.5 from -o32 to -n32. If your Python install is from 1.5.1 days, then compilations will fail because of the mismatched ABIs. In theory, setting SGI_ABI environment variable to -o32 should work, but I found it best to modify Makefile.pre.in to force the setting on the CFLAGS and LDFLAGS lines. If my guess as to what's wrong with your set up is right, then it isn't a distutils problem; it's a Python problem which has since been fixed, and you can use my workaround until you do a newer install. Andrew Dalke dalke@acm.org From gward@cnri.reston.va.us Sun Jan 30 15:04:57 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Sun, 30 Jan 2000 10:04:57 -0500 Subject: [Distutils] WinNT and distutils In-Reply-To: <200001290251.VAA04695@eric.cnri.reston.va.us>; from guido@cnri.reston.va.us on Fri, Jan 28, 2000 at 09:51:18PM -0500 References: <14482.1181.29982.349164@weyr.cnri.reston.va.us> <20000128190216.B12965@cnri.reston.va.us> <200001290251.VAA04695@eric.cnri.reston.va.us> Message-ID: <20000130100456.A15896@cnri.reston.va.us> On 28 January 2000, Guido van Rossum said: > Where do you get the impression that 1.6 will have Windows registry > access built in? I didn't even know that was on the wish list. (If > it is, maybe someone can mail me code?) Distant memories of one of the Windows guys rambling on python-dev several months ago, I think. I was sort of assuming that rambling had actually gone somewhere. (Maybe now it will!) Greg From gward@cnri.reston.va.us Sun Jan 30 20:45:07 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Sun, 30 Jan 2000 15:45:07 -0500 Subject: [Distutils] IPC8 paper and slides on Distutils home page Message-ID: <20000130154506.A16964@cnri.reston.va.us> Hi all -- I've put the paper I presented at IPC8 last week, along with the slides I used for my talk, on the Distutils web page at http://www.python.org/sigs/distutils-sig/ In particular you'll find the HTML version of the paper at http://www.python.org/sigs/distutils-sig/ipc8_paper.html PostScript and PDF versions of the paper are also there. The slides are only available in Applix Presents and PowerPoint format (exported by Applix). I have no idea how good or bad the PowerPoint slides are; I'll try to take a look at them when I'm at work Monday. If anyone knows how to get decent HTML slids out of Applix Presents, let me know... Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gward@cnri.reston.va.us Sun Jan 30 20:49:51 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Sun, 30 Jan 2000 15:49:51 -0500 Subject: [Distutils] ANNOUNCE: Distutils 0.1.3 available Message-ID: <20000130154951.A16974@cnri.reston.va.us> ...title says it all. Here's the change log entry: * "dist" command works on Windows now * better error handling in a few places * fixed some bugs in the MSVC++ interface class * "install_py" and "install_ext" commands now run their corresponding build commands, in case they are invoked standalone by the user * added the "--help-commands" option to get a list of all commands * renamed the text files to README.txt, USAGE.txt, and CHANGES.txt (for convenience to Windows users) * started adding code to allow building static C libraries and standalone executables (not used yet) Get yours today from the distutils home page: http://www.python.org/sigs/distutils-sig/ Works for me to build and install Numerical Python 15.2 under Linux and Solaris. All other uses of the software are strictly forbidden and not supported. Ha ha! just kidding. I just haven't *tested* anything else. Bottom line: this fixes most of the easy bugs. If you had suggestions for me at the conference last week, don't go looking in this release for them -- sorry. If you have problems with it, post to the sig. I'm especially interested in hearing how it works on platforms other than Linux and Solaris, since I only have access to those two for testing. Right, now on to hacking on release 0.2... Greg -- Greg Ward - software developer gward@cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913 From gward@cnri.reston.va.us Sun Jan 30 21:19:31 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Sun, 30 Jan 2000 16:19:31 -0500 Subject: [Distutils] Name game for 0.2 Message-ID: <20000130161930.A16990@cnri.reston.va.us> Hi all -- the easiest change I want to make in Distutils 0.2 is to fix some bad nomenclature in the command classes. This will break a small amount of existing code, in particular any setup script that extends the system by defining new command classes or extending existing ones. AFAIK, this means just Numerical Python. I'm perfectly happy to break code this early in the game, but *we must get it right* this time. Here's the background: every command class has four key elements. No, wait, five -- I just added one today. OK, every command class has *five* key elements: options - class attribute, lists command-line-settable options description - class attribute, a one-line description for the "--help-commands" option set_default_options() - method, initializes all command options (whether they can be set on the command-line or not) set_final_options() - method, decide on final values for all command options so we can go ahead and run the command with minimal logic run() - method, run the command The bad news is I'm only happy with two of these names ('description' and 'run()'). The goods news I *think* I know what I want to rename the other three to. 'options' should be called 'user_options' for several reasons. First, it only describes a subset of the command's options, i.e. there are more options dealt with in 'set_default_options()' and 'set_final_options()' than are mentioned in the 'options' attribute. The attribute just lists options that can be set on the command-line -- but I don't want to restrict myself to *just* the command-line; someday we'll have config files too, and users will be able to set these options in config files to boot. The key is that *users* -- most importantly installers, but in general anyone who runs the setup script -- can set the options listed in the 'options' attribute. Hence 'user_options'. Easy decision, eh? The second-easiest one is 'set_default_options()'. The purpose of this method is to assign initial values to *all* of the command's options, not just the user-settable ones listed in the attribute soon to be called 'user_options'. A typical 'set_default_methods()' looks like this (from the standard 'build' command): def set_default_options (self): self.build_base = 'build' self.build_lib = None self.build_platlib = None Pretty simple: in fact, it looks a lot like a typical Python constructor. The only reason this isn't done in a constructor is that Command, the base class for all Distutils commands, provides a perfectly good constructor that happens to call 'set_default_options()' to do command-specific initialization. So I think the name for this method should, pretty obviously, be 'initialize_options()'. Nice "action-verb" form there, much better than the current lame form. Finally, 'set_final_options()'. I've never much liked this name; it was thought up in a rush when I suddenly realized the need for the method and had to throw it in quickly to get things working. In addition to being the hardest to think of a good name for, this one is the hardest to explain: it exists to decide on final option values for any options that were not set by the user. To clarify, here's how things typically go for a given command in a Distutils run: * create command object - call 'set_default_options()' to initialize option values [ * parse config files - set various command options from what's found in the config files ] [vapourware] * parse command line - set various command options from what's found on the command-line [overriding config-file option values, when that's possible] * call 'set_final_options()' * call 'run()' The idea is that any option that's still None -- ie. wasn't touched by the command-line, config files, or whatever -- by the time we reach 'set_final_options()' will be decided by the code in 'set_final_options()'. For example, here's the 'build' command's implementation of that method: def set_final_options (self): # 'build_lib' and 'build_platlib' just default to 'lib' and # 'platlib' under the base build directory if self.build_lib is None: self.build_lib = os.path.join (self.build_base, 'lib') if self.build_platlib is None: self.build_platlib = os.path.join (self.build_base, 'platlib') For the 'build' command, 'build_base' is really only used to derive the two "important" options, build_lib and build_platlib. If the user supplies one of those important options (eg. on the command line), then we don't override their request -- we only compute a value if build_lib or build_platlib are None. So, some silly names for this method are: decide_final_option_values() settle_on_values_for_any_options_not_set_by_the_user() and some not-so-silly, but not much better names, are: decide_options() finalize_options() finish_options() Anyone have strong feelings about any of these? I'm wary of "finalize" because the Java people for some reason decided that "finalize" means "clean up before destruction" (well, OK, I'm not sure if they're to blame for this odd piece of jargon -- but it is established jargon, like it or not). The basic idea behind the method is to settle on a set of option values that will drive how we run the command: once the option values are decided, running the command should be relatively straightforward. So this really is a preparatory step, but it's the *final* preparatory step. If this gives you any great ideas for what to call the bloody thing, please let me know, because I'm stumped. Thanks -- Greg From gward@cnri.reston.va.us Mon Jan 31 14:34:29 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 31 Jan 2000 09:34:29 -0500 Subject: [Distutils] WinNT and distutils In-Reply-To: ; from gstein@lyra.org on Fri, Jan 28, 2000 at 04:08:07PM -0800 References: <20000128190216.B12965@cnri.reston.va.us> Message-ID: <20000131093428.A17133@cnri.reston.va.us> On 28 January 2000, Greg Stein said: > Some people have suggested that distutils does some of its install work > via the new import stuff. > > I'm not sure what that entails (I don't understand why distutils would > need the custom importing/packaging that imputil provides). I agree with you: I don't understand why distutils would want to depend on imputils. It seems desirable for distutils to be able to generate archives (or whatever) that can only be imported from using imputils, but I don't see a need to take it farther than that. One can speculate about a world where Python module distributions are distributed as archives with a __run__.py (JPython convention brought to my attention by Barry) that is the setup script. But I think Python itself would have to grok the archive and its __run__.py, and anyways that all seems rather far out at this point. > It won't work until 1.6 because we don't have the 1.6 design yet :-) > > But, there is no reason it shouldn't (with some creative try/except around > some imports). Note that win32api isn't going into 1.6 -- either a new > module from scratch, or a small extraction of win32api. In any case, it > certainly won't (can't) be called win32api. Ahh, OK, I wasn't really clear on that. I had a vague idea that some sort of registry access would be in 1.6, I gather that Mark's win32api provides registry access, and thus I assumed that something called win32api would be in 1.6. So much for blithe assumptions. So it seems to me as if something like the following will be needed: try: import win32api, win32con get MSVC++ registry keys from it except ImportError: do the same thing with whatever the official Python 1.6 registry access mechanism is except ImportError: just assume cl.exe is on the path if we got the MSVC++ registry keys: use Robin Becker's code to figure out where cl.exe is supposed to be Ugh. This is starting to look like writing an autoconf script. At least it's not in Bourne shell... >shudder<... I guess for now I should stick with the code Robin provided, since there is no "official Python 1.6 registry access mechanism" yet. Greg From gward@cnri.reston.va.us Mon Jan 31 14:36:39 2000 From: gward@cnri.reston.va.us (Greg Ward) Date: Mon, 31 Jan 2000 09:36:39 -0500 Subject: [Distutils] WinNT and distutils In-Reply-To: <14482.25811.673319.173880@weyr.cnri.reston.va.us>; from fdrake@acm.org on Fri, Jan 28, 2000 at 10:56:03PM -0500 References: <14482.1181.29982.349164@weyr.cnri.reston.va.us> <20000128190216.B12965@cnri.reston.va.us> <14482.25811.673319.173880@weyr.cnri.reston.va.us> Message-ID: <20000131093638.B17133@cnri.reston.va.us> On 28 January 2000, Fred L. Drake, Jr. said: > If that's the only dependency we need, that's all that we need. I > don't think we need to introduce dependencies artificially. If it > works with 1.5.2 on Unix & 1.6 on Windows, I'd be quite happy. With > the proposed timeline for 1.6, that seems quite acceptable. Don't forget, Distutils seems to work fine on Windows as long as cl.exe is in your PATH. I agree that it would be better to be more ironclad (ie. get the VC++ bin directory from the registry if possible), but for now it seems to be muddling along (modulo stupid bugs in my code). Greg From robin@jessikat.demon.co.uk Mon Jan 31 15:13:08 2000 From: robin@jessikat.demon.co.uk (Robin Becker) Date: Mon, 31 Jan 2000 15:13:08 +0000 Subject: [Distutils] WinNT and distutils In-Reply-To: <20000131093428.A17133@cnri.reston.va.us> References: <20000128190216.B12965@cnri.reston.va.us> <20000131093428.A17133@cnri.reston.va.us> Message-ID: In article <20000131093428.A17133@cnri.reston.va.us>, Greg Ward writes .... >Ahh, OK, I wasn't really clear on that. I had a vague idea that some >sort of registry access would be in 1.6, I gather that Mark's win32api >provides registry access, and thus I assumed that something called >win32api would be in 1.6. So much for blithe assumptions. > >So it seems to me as if something like the following will be needed: > > try: > import win32api, win32con > get MSVC++ registry keys from it > except ImportError: > do the same thing with whatever the official Python 1.6 registry > access mechanism is > except ImportError: > just assume cl.exe is on the path > > if we got the MSVC++ registry keys: > use Robin Becker's code to figure out where cl.exe is supposed to be > >Ugh. This is starting to look like writing an autoconf script. At >least it's not in Bourne shell... >shudder<... I guess for now I should >stick with the code Robin provided, since there is no "official Python >1.6 registry access mechanism" yet. > ... it seems that not all of the directories obtained from the keys are actually rock solid eg they are part of the devstudio customisables. I'm fairly sure that you could add all of the Path dirs to the path set Include=all the directories in Include Dirs (; sep) set Lib=all in the lib dirs (; sep) and then cl.exe etc are supposed to work. I guess that other explicit includes and libs get looked at first so there shouldn't be any harm in adding the extras. -- Robin Becker From cjensen@bioeng.ucsd.edu Mon Jan 31 19:24:31 2000 From: cjensen@bioeng.ucsd.edu (Curtis Jensen) Date: Mon, 31 Jan 2000 11:24:31 -0800 Subject: [Distutils] Complling on IRIX OS References: <040301bf6ab6$bd6b9640$0005fea9@adalke> Message-ID: <3895E16F.8D8A01B1@be-research.ucsd.edu> Andrew Dalke wrote: > > Curtis Jensen said: > > To install on an IRIX OS, sometimes we need to > > compile with the -o32 flag. > > There are two problems here: > 1) Before 1.5.2 (or 1.5.1?), Python for IRIX > didn't store the SGI_ABI setting. Thus, the > compiler uses the default settings. With 1.5.2, > the Makefile.pre.in contains the ABI value. > > 2) The default ABI changed with IRIX 6.5 from > -o32 to -n32. If your Python install is from > 1.5.1 days, then compilations will fail because > of the mismatched ABIs. > > In theory, setting SGI_ABI environment variable > to -o32 should work, but I found it best to modify > Makefile.pre.in to force the setting on the CFLAGS > and LDFLAGS lines. > > If my guess as to what's wrong with your set up > is right, then it isn't a distutils problem; it's > a Python problem which has since been fixed, and > you can use my workaround until you do a newer > install. > > Andrew Dalke > dalke@acm.org the command "python -c "import sys;print sys.version"" reveals: "1.5.2 (#1, Sep 20 1999, 15:11:12) [C]" So, we are using Python 1.5.2. -- Curtis Jensen cjensen@be-research.ucsd.edu http://www-bioeng.ucsd.edu/~cjensen/ FAX (425) 740-1451 From wijngaar@nas.nasa.gov Mon Jan 31 20:10:12 2000 From: wijngaar@nas.nasa.gov (Rob F. Van Der Wijngaart) Date: Mon, 31 Jan 2000 12:10:12 -0800 (PST) Subject: [Distutils] installing distutils Message-ID: <200001312010.MAA22173@octane28.nas.nasa.gov> Dear Distutils-sig, I would like to install Python's distutils, but do not have write permission in the Python installation directory on my system. The README file in the distutils distribution does not tell me how to install elsewhere, and I am too much of a novice at Python to figure out what to change otherwise. Can you tell me how to install distutils in a user directory and use it from there? Thanks. Regards, Rob F. Van der Wijngaart NASA Ames Research Center From cjensen@bioeng.ucsd.edu Mon Jan 31 20:29:17 2000 From: cjensen@bioeng.ucsd.edu (Curtis Jensen) Date: Mon, 31 Jan 2000 12:29:17 -0800 Subject: [Distutils] installing distutils References: <200001312010.MAA22173@octane28.nas.nasa.gov> Message-ID: <3895F09D.710839ED@be-research.ucsd.edu> "Rob F. Van Der Wijngaart" wrote: > > Dear Distutils-sig, > > I would like to install Python's distutils, but do not have > write permission in the Python installation directory on my > system. The README file in the distutils distribution does > not tell me how to install elsewhere, and I am too much of > a novice at Python to figure out what to change otherwise. > Can you tell me how to install distutils in a user directory > and use it from there? Thanks. > > Regards, Rob F. Van der Wijngaart > NASA Ames Research Center > > _______________________________________________ > Distutils-SIG maillist - Distutils-SIG@python.org > http://www.python.org/mailman/listinfo/distutils-sig View the USAGE file. Near the end it tells how. It'll be a command similar to: setup.py -v install --prefix=/home/greg --exec-prefix=/home/greg If /home/greg was where you wanted to install to. -- Curtis Jensen cjensen@be-research.ucsd.edu http://www-bioeng.ucsd.edu/~cjensen/ FAX (425) 740-1451 From mhammond@skippinet.com.au Mon Jan 31 23:25:01 2000 From: mhammond@skippinet.com.au (Mark Hammond) Date: Mon, 31 Jan 2000 15:25:01 -0800 Subject: [Distutils] WinNT and distutils In-Reply-To: Message-ID: > Surely msvccompiler can tell if a particular file (say cl.exe) isn't in > any of the well known places. Then it can just issue the correct > commands with the path attached or tell the user it can't find > cl.exe/link.exe and then ask for vcvars32.bat to be run first. I assume > we're not going to search all known drives for the correct exe's. I agree. IMO, we could take the following approach: * Look in C:\Program Files\The default place\... * Look in \Program Files\The default place\... * Try and import win32api, catching the import error And if that fails, give the user a nice polite message. This is likely to catch 95% of the cases, and still make it obvious what the user should do in the small percentage left... However, it is worth noting that running "vcvars32.bat" wont work if run via a seperate "system()" call - the environment variables set will not persist. The only decent solution I can see is to create a temporary batch file, that first calls "vcvars32.bat" before the builds. Depending on how distutils does its thing, it may just be simpler to detect the condition, and politely ask the user to run the batch file themselves... Mark.