Mailman 3 September 2000 - Distutils-SIG

Re: [Distutils] test command for setup.py
by Berthold Höllmann 26 Sep '00

26 Sep '00

"Thomas Heller" <thomas.heller(a)ion-tof.com> writes: > I like the simplicity of your test command, > however I would like (and have partly implmented) > the following changes: > - don't use so many options, the build directories could > be retrieved from the 'build' command I would, if I could figure out how. )-: > - wrap 'try' blocks around running the tests I think, if a test failed, the script should abort. If you want it to continue, the try blocks should go into the test files themselves. > - output a summary of failed and passed tests > I will probably post my script later. This again should go into the test implementations, or into a test framework. I think these are only good for the "enduser" testing. But I would like to handle some "developer" testing also by the distutils test command. Berthold -- email: hoel(a)GermanLloyd.org ) tel. : +49 (40) 3 61 49 - 73 74 ( C[_] These opinions might be mine, but never those of my employer.

2 1

Appears bdist_rpm completely broken
by Uche Ogbuji 25 Sep '00

25 Sep '00

Maybe I'm missing something, but I don't see how bdist_rpm works for anyone. I've tried 0.9.2, the 9/12 and 9/16 snapshots but I have a problem with the generated pycs. First the symptom. I build 4Suite using distutils. The sdist, bdist and bdist_win packages work just fine, but when I move the generated RPM to another machine and try to use 4Suite: [uogbuji@dynip-23 uogbuji]$ python Python 1.5.2 (#1, Jul 12 2000, 10:39:14) [GCC 2.96 20000702 (experimental)] on linux-i386 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> import sys >>> sys.path ['', '/usr/lib/python1.5/', '/usr/lib/python1.5/plat-linux-i386', '/usr/lib/python1.5/lib-tk', '/usr/lib/python1.5/lib-dynload', '/usr/lib/python1.5/site-packages'] >>> import xml.xslt Traceback (innermost last): File "<stdin>", line 1, in ? File "/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages/xml/xslt/__init__.py", line 3, in ? File "/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages/xml/xpath/__init__.py", line 3, in ? ImportError: cannot import name XPathParserBase >>> The problem is the phantom "/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages"... [uogbuji@dynip-23 uogbuji]$ ls /var/tmp [uogbuji@dynip-23 uogbuji]$ Further investigation turns up the culprit: The generated pyc files from the RPM have the "/var/tmp/..." hard-coded in to their imports. So am I doing something wrong, or is distutils broken for RPM build? I'm not on the distutils list so ccs to me would be appreciated. Otherwise I can check the archives. Thanks. -- Uche Ogbuji Principal Consultant uche.ogbuji(a)fourthought.com +1 303 583 9900 x 101 Fourthought, Inc. http://Fourthought.com 4735 East Walnut St, Ste. C, Boulder, CO 80301-2537, USA Software-engineering, knowledge-management, XML, CORBA, Linux, Python

4 8

Patch to have distutils generate PYC *and* PYO files
by M.-A. Lemburg 24 Sep '00

24 Sep '00

Attached you find a patch against the current CVS tree which allows distutils to include PYC as well as PYO files. I haven't dug deep into the option system yet, but it would be nice if there also were an option which lets the packager set the optimization level she want to use for the PYO files. Currently level 1 is used. Level 2 would also eliminate doc-strings which can sometimes be useful to reduce package size or to hide interface information. -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/ --- CVS-Python/Lib/distutils/util.py Mon Sep 18 10:31:17 2000 +++ Python+Unicode/Lib/distutils/util.py Wed Sep 20 13:14:27 2000 @@ -5,11 +5,11 @@ one of the other *util.py modules.""" # created 1999/03/08, Greg Ward __revision__ = "$Id: util.py,v 1.44 2000/09/15 01:16:14 gward Exp $" -import sys, os, string, re, shutil +import sys, os, string, re, shutil, py_compile from distutils.errors import * from distutils.spawn import spawn def get_platform (): @@ -265,5 +265,39 @@ def execute (func, args, msg=None, verbo # And do it, as long as we're not in dry-run mode if not dry_run: apply(func, args) # execute() + + +def compile(filename, optimize=None): + + """ Compile the Python script filename. + + If optimize is not given, the current optimization setting is + used. Otherwise, optimize must be an integer stating the value + of the optimization level to use: + + 0 - no optimization (generates .pyc file) + 1 - normal optimization (generates .pyo file, same as -O) + 2 - optimize + remove doc-strings (generates .pyo file, -OO) + + """ + if (optimize is None) or \ + (not optimize and __debug__) or \ + (optimize and not __debug__): + return py_compile.compile(filename) + + # Otherwise will have to run a system command + python = sys.executable + filename = os.path.abspath(filename) + if optimize: + options = '-' + 'O' * optimize + else: + options = '' + command = "import py_compile; py_compile.compile('%s')" % filename + rc = os.system('%s %s -c "%s"' % (python, options, command)) + if rc != 0: + # py_compile currently does not pass back errors, so this + # is not really executed... + raise SyntaxError,'could not compile "%s"' % filename + --- CVS-Python/Lib/distutils/command/install_lib.py Mon Sep 18 10:31:17 2000 +++ Python+Unicode/Lib/distutils/command/install_lib.py Wed Sep 20 13:19:11 2000 @@ -61,27 +61,24 @@ class install_lib (Command): else: self.warn("'%s' does not exist -- no Python modules to install" % self.build_dir) return - # (Optionally) compile .py to .pyc - # XXX hey! we can't control whether we optimize or not; that's up - # to the invocation of the current Python interpreter (at least - # according to the py_compile docs). That sucks. - + # (Optionally) compile .py to .pyc and .pyo if self.compile: - from py_compile import compile + from distutils.util import compile for f in outfiles: # only compile the file if it is actually a .py file if f[-3:] == '.py': - out_fn = f + (__debug__ and "c" or "o") - compile_msg = "byte-compiling %s to %s" % \ - (f, os.path.basename (out_fn)) - skip_msg = "skipping byte-compilation of %s" % f - self.make_file (f, out_fn, compile, (f,), - compile_msg, skip_msg) + for optimize in (0, 1): + out_fn = f + (optimize and "o" or "c") + compile_msg = "byte-compiling %s to %s" % \ + (f, os.path.basename (out_fn)) + skip_msg = "skipping byte-compilation of %s" % f + self.make_file (f, out_fn, compile, (f, optimize), + compile_msg, skip_msg) # run () def _mutate_outputs (self, has_any, build_cmd, cmd_option, output_dir): @@ -102,12 +99,12 @@ class install_lib (Command): # _mutate_outputs () def _bytecode_filenames (self, py_filenames): bytecode_files = [] for py_file in py_filenames: - bytecode = py_file + (__debug__ and "c" or "o") - bytecode_files.append(bytecode) + bytecode_files.append(py_file + "c") + bytecode_files.append(py_file + "o") return bytecode_files def get_outputs (self): """Return the list of files that would be installed if this command

4 5

How to add include dirs for compiler
by Dragons Spam Account 22 Sep '00

22 Sep '00

How do I inform distutils of addition include directories to find headers in when compiling c source in a python module. This is outside the package. I'm a user not the author. My scenerio: I installed one module that provides a header file. Rather than inflict my additions on all the users of the system I'm installing locally to my own home directory with --home=~ This pops the libraries and such into ~/lib/python but drops the header file into ~/include python. For the sake of python finding ~/lib/python, PYTHONPATH suffices. Now I attempt to install a second seperate module that relies on the first module. Regardless of whether I install this second module to the system or just to my home directory it needs to find that header and I can't seem to find the interface through which to provide distutils with additional include directories to use when compiling source. Which in this case would be ~/include/python. The whole scenerio works fine if I'm installing everything to the system python directories but currently this is not an option for these modules. Any help?

2 1

BisonGen, 4Suite, and Distutils
by Greg Ward 22 Sep '00

22 Sep '00

Uche, Mike -- I'm digging into BisonGen and 4Suite now. I managed to install BisonGen (from the sdist) without problems, but there's a dependency you forgot to document: $ /usr/local/bin/python Python 1.5.2 (#5, Sep 8 2000, 22:21:43) [GCC egcs-2.91.66 19990314/Linux (egcs- on linux2 Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam >>> import BisonGen.BisonGen Traceback (innermost last): File "<stdin>", line 1, in ? File "/usr/local/python/lib/python1.5/site-packages/BisonGen/BisonGen.py", line 80, in ? import BGenReader File "/usr/local/python/lib/python1.5/site-packages/BisonGen/BGenReader.py", line 24, in ? from xml.sax import saxlib, saxexts ImportError: No module named xml.sax >>> Luckily, I also have PyXML's source dist. sitting around, so I can install it. Still, you should probably mention this in your README. Another thing: setting "verbose = 0" in the setup script is the moral equivalent of setting ".SILENT" in a Makefile -- and IMHO it's Considered Harmful. Building software *should* be noisy, because so many things can go wrong. You're free to run the setup script with -q, but if people are used to seeing noisy builds, then one particular build being quiet is unsettling. (It unsettled me, that's for sure!) Yet another: requiring SWIG to be installed in order to build your software is the moral equivalent of requiring bison and flex on a non-GNU-powered Unix box -- ie. it's a bother, it's a pain, and it's completely unnecessary since the software that I now have to go find, download, build, and install is used *only* to build this particular package! (I'm speaking here from the narrow, just-want-to-get-the- silly-thing-working perspective of someone who needs to install 4Suite in order to get work done. I'm a little embarassed to admit that I don't even have SWIG installed on my home machine, but it does let me discover lapses in other people's source dists. ;-) Now, if SWIG is turning 20k .i files into 400k .c files, that's a good reason for not distributing the .c files. And I'm starting to get a grasp of the enormity of 4Suite; this is a boatload-and-a-half of a code, so I can see why you might want to cut down on the amount of stuff distributed. Anyways, looks like I won't be able to debug your bdist_rpm problem this morning; have to go get SWIG, etc. etc. Grumble. BTW, congratulations on finding -- and making use of -- every known "backdoor" entry into the Distutils. At least it looks like the 4Suite setup.py uses every sneaky trick that I can remember slipping in to the code at some point. Very gratifying to find that all those tricks are finally useful to someone! Also, it looks like 4Suite's setup.py is so complex it will be a good shaker-out of Distutils bugs. Thanks (I think)! Greg -- Greg Ward gward(a)python.net http://starship.python.net/~gward/

2 1

Re: [Distutils] test command for setup.py
by Berthold Höllmann 22 Sep '00

22 Sep '00

>>>>> "Greg" == Greg Ward <gward(a)python.net> writes: Greg> On 21 September 2000, Berthold Höllmann said: >> I think it is easier for the developer to have code and tests >> together. Greg> Not if the tests are really exhaustive! When I write good Greg> tests (or at least what I think are good tests), the test Greg> script is roughly the size of the module being tested. I Greg> don't want my "if __name__ == '__main__'" block to be as big Greg> as the rest of the module. I think the "test" command should Greg> support "in-situ" testing, but it's *not* enough for serious Greg> tests. >> One can leave out the tests from any distribution by using a >> Manifest file. Greg> Huh? That loses half the point of writing tests, which is to Greg> assure people building and installing your modules that they Greg> probably work as advertised on their platform. My Idea here was, to distinguish between test to enshure correct compilation on the "customers" computer and tests to help the developer my the Manifest file, but to keep all tests with your code in one (CVS?)repository. >> But for the developer it is helpful to have a set of tests that >> one can run after changes have been made to the sources to >> enshure that "little" changes have not unexpected side effects. Greg> Yes! That's the other half of the reason for writing tests. Greg> (You can define "half" here as 40-60, 20-80, whatever.) >> ... Greg> Oooh, yes, in a previous life I struggled (briefly) with Greg> automated testing of numerical algorithms. Not fun. But Greg> asking end-users to examine the test output and make sure Greg> that the difference between 1.0234324 and 1.0234821 is less Greg> than the machine epsilon is surely even less fun. It begins with the difference between 1.00000E+00 and 1.0000E+000 (where is it?). >> Also the underlaying Fortran code makes different decisions on >> different platforms. That makes automatic testing nearly >> impossible, but I want at least a constistant way to generate >> the needed output, and I want to test everything before I >> install it. Greg> Here's my extreme position: Greg> If a test isn't fully automated -- i.e., if it takes more Greg> than epsilon human intelligence to determine if the test Greg> passed or failed -- then it's not a proper test, it's just a Greg> demo. Accepted for "functionality" tests, to be executed by the "customer", but I don't want to distinguish these from heavier tests, to be executed by the developer. Greg> ... >> All this can be done with my posted, extremly lightweight >> test.py extension for the _command_ subdirectory. The test >> scripts can written to implement/use any testframework you >> like. Maybe a Greg> OK, lightweight is good. (Something I occasionally need to Greg> be reminded of...) I'll definitely take a look at it. Thanks Greetings Berthold -- bhoel(a)starship.python.net / http://starship.python.net/crew/bhoel/ It is unlawful to use this email address for unsolicited ads (USC Title 47 Sec.227). I will assess a US$500 charge for reviewing and deleting each unsolicited ad.

1 0

Extension Modules
by Mike Olson 22 Sep '00

22 Sep '00

Hello, We are working towards converting 4Suite over to use distutils. We are almost there, but have run into a problem. We needed to implement our own parser to generate some files using a script we call BisonGen. This generates both C and py files. Then we hand the C off to build_ext to get compilied. The .so (or pyd) is installed fine. However, the additional py files generated are not copied into the installation directory. Is there a simple way of doing this? Looking through the code, build_ext only uses Extension.name to generate a list of outputs. I've tried a couple of approaches, but all seem to break someother portion of the script (usually bdist --format=rpm). We have the same problem with shadow files generated via swig..... Help is appreciated. Mike -- Mike Olson Principal Consultant mike.olson(a)fourthought.com (303)583-9900 x 102 Fourthought, Inc. http://Fourthought.com Software-engineering, knowledge-management, XML, CORBA, Linux, Python

2 1

Bug in parsing of .cfg files
by M.-A. Lemburg 22 Sep '00

22 Sep '00

The following .cfg file produces an error: [sdist] manifest = MANIFEST no-defaults = 1 keep-tree = 1 formats=bztar,gztar,tar,zip,ztar ... error: error in setup.cfg: command 'sdist' has no such option 'keep-tree' Looks as if the config file parser doesn't like options with embedded hyphens (or maybe it's just me not knowing how to pass the options ?). -- Marc-Andre Lemburg ______________________________________________________________________ Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

5 11

Latest snapshot
by Greg Ward 21 Sep '00

21 Sep '00

Hi all -- I've just released another code snapshot. The main change is that Windows installers created on non-Windows platforms should now include scripts and data correctly. This involved some subtle rearrangement of the "install" command; as usual, It Works For Me (TM), but please try it out and see if it works for you. Also, some patches for Mac OS support have snuck in over the last couple of days. Download it from the usual place: http://www.python.org/sigs/distutils-sig/download.html#snapshot Greg -- Greg Ward gward(a)python.net http://starship.python.net/~gward/

1 0

Re: [Distutils] test command for setup.py
by Berthold Höllmann 21 Sep '00

21 Sep '00

>>>>> "Greg" == Greg Ward <gward(a)python.net> writes: Greg> On 18 September 2000, Berthold Höllmann said: >> I'm thinking about providing a framework for pre-install >> testing for distutils enabled Python packages. I think about a >> new "test" command, which inserts the lib build path into >> PYTHONPATH and runs specified tests. Is anyone else working on >> something like this, and what are your opinions and >> requirements on this? Greg> I've been thinking about it from Day 1, personally. Here are Greg> some of my opinions/requirements: Greg> * serious tests don't belong in the module being tested, Greg> because test code tends to be at least as long as the code Greg> being tested I think it is easier for the developer to have code and tests together. One can leave out the tests from any distribution by using a Manifest file. But for the developer it is helpful to have a set of tests that one can run after changes have been made to the sources to enshure that "little" changes have not unexpected side effects. Greg> ... Greg> * the requirements on what a test script may output should Greg> be fairly strict. My favourite is loosely based on Perl's Greg> testing convention, which is a great success largely because Greg> the standard-way-to-build- Perl-modules includes a "make Greg> test" step, and it's always immediately clear if everything Greg> passed. My proposed convention is this: each test script Greg> outputs a line saying how many tests are expected, then a Greg> series of lines that start with either "ok" or "not ok". If Greg> there are any "not ok" lines, the test script fails; if the Greg> number of "ok" lines != the number of expected tests, the Greg> test script fails. Each "ok" or "not ok" may be followed by Greg> ":" and an arbitrary string, which makes it a lot easier to Greg> track down problems. (The Perl convention is "ok 1", "ok 2", Greg> ... "ok N", which makes it really awkward to track down Greg> problems.) Any lines not starting with "ok" or "not ok" are Greg> ignored. Thats not always an option. I'm writing modules using numerical opreations. These require test results to be looked over very carefully to decide, whether the differences are significant or not. Also the underlaying Fortran code makes different decisions on different platforms. That makes automatic testing nearly impossible, but I want at least a constistant way to generate the needed output, and I want to test everything before I install it. Greg> ... Greg> * we might also want to accomodate a Python-style test Greg> suite, where you compare "known good" output of a script Greg> with the current output. I'm not a fan of this methodology, Greg> but sometimes it's easier, and it may have some currency in Greg> the Python community outside of Lib/test. It should be the Greg> developer's choice what style of test suite he wants to Greg> write. All this can be done with my posted, extremly lightweight test.py extension for the _command_ subdirectory. The test scripts can written to implement/use any testframework you like. Maybe a >python setup.py test command is only a convenience command but it gives you a starting point when maintaining code you wrote ages ago, or somebody else wrote. Greetings Berthold -- bhoel(a)starship.python.net / http://starship.python.net/crew/bhoel/ It is unlawful to use this email address for unsolicited ads (USC Title 47 Sec.227). I will assess a US$500 charge for reviewing and deleting each unsolicited ad.

2 1