"Thomas Heller" <thomas.heller(a)ion-tof.com> writes:
> I like the simplicity of your test command,
> however I would like (and have partly implmented)
> the following changes:
> - don't use so many options, the build directories could
> be retrieved from the 'build' command
I would, if I could figure out how. )-:
> - wrap 'try' blocks around running the tests
I think, if a test failed, the script should abort. If you want it to
continue, the try blocks should go into the test files themselves.
> - output a summary of failed and passed tests
> I will probably post my script later.
This again should go into the test implementations, or into a test
framework. I think these are only good for the "enduser" testing. But
I would like to handle some "developer" testing also by the distutils
test command.
Berthold
--
email: hoel(a)GermanLloyd.org
) tel. : +49 (40) 3 61 49 - 73 74
(
C[_] These opinions might be mine, but never those of my employer.
Maybe I'm missing something, but I don't see how bdist_rpm works for
anyone. I've tried 0.9.2, the 9/12 and 9/16 snapshots but I have a
problem with the generated pycs.
First the symptom. I build 4Suite using distutils. The sdist, bdist
and bdist_win packages work just fine, but when I move the generated RPM
to another machine and try to use 4Suite:
[uogbuji@dynip-23 uogbuji]$ python
Python 1.5.2 (#1, Jul 12 2000, 10:39:14) [GCC 2.96 20000702
(experimental)] on linux-i386
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> import sys
>>> sys.path
['', '/usr/lib/python1.5/', '/usr/lib/python1.5/plat-linux-i386',
'/usr/lib/python1.5/lib-tk', '/usr/lib/python1.5/lib-dynload',
'/usr/lib/python1.5/site-packages']
>>> import xml.xslt
Traceback (innermost last):
File "<stdin>", line 1, in ?
File
"/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages/xml/xslt/__init__.py",
line 3, in ?
File
"/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages/xml/xpath/__init__.py",
line 3, in ?
ImportError: cannot import name XPathParserBase
>>>
The problem is the phantom
"/var/tmp/4Suite-buildroot/usr/lib/python1.5/site-packages"...
[uogbuji@dynip-23 uogbuji]$ ls /var/tmp
[uogbuji@dynip-23 uogbuji]$
Further investigation turns up the culprit: The generated pyc files
from the RPM have the "/var/tmp/..." hard-coded in to their imports.
So am I doing something wrong, or is distutils broken for RPM build?
I'm not on the distutils list so ccs to me would be appreciated.
Otherwise I can check the archives.
Thanks.
--
Uche Ogbuji Principal Consultant
uche.ogbuji(a)fourthought.com +1 303 583 9900 x 101
Fourthought, Inc. http://Fourthought.com
4735 East Walnut St, Ste. C, Boulder, CO 80301-2537, USA
Software-engineering, knowledge-management, XML, CORBA, Linux, Python
Attached you find a patch against the current CVS tree which allows
distutils to include PYC as well as PYO files.
I haven't dug deep into the option system yet, but it would be
nice if there also were an option which lets the packager
set the optimization level she want to use for the PYO files.
Currently level 1 is used. Level 2 would also eliminate doc-strings
which can sometimes be useful to reduce package size or to hide
interface information.
--
Marc-Andre Lemburg
______________________________________________________________________
Business: http://www.lemburg.com/
Python Pages: http://www.lemburg.com/python/
--- CVS-Python/Lib/distutils/util.py Mon Sep 18 10:31:17 2000
+++ Python+Unicode/Lib/distutils/util.py Wed Sep 20 13:14:27 2000
@@ -5,11 +5,11 @@ one of the other *util.py modules."""
# created 1999/03/08, Greg Ward
__revision__ = "$Id: util.py,v 1.44 2000/09/15 01:16:14 gward Exp $"
-import sys, os, string, re, shutil
+import sys, os, string, re, shutil, py_compile
from distutils.errors import *
from distutils.spawn import spawn
def get_platform ():
@@ -265,5 +265,39 @@ def execute (func, args, msg=None, verbo
# And do it, as long as we're not in dry-run mode
if not dry_run:
apply(func, args)
# execute()
+
+
+def compile(filename, optimize=None):
+
+ """ Compile the Python script filename.
+
+ If optimize is not given, the current optimization setting is
+ used. Otherwise, optimize must be an integer stating the value
+ of the optimization level to use:
+
+ 0 - no optimization (generates .pyc file)
+ 1 - normal optimization (generates .pyo file, same as -O)
+ 2 - optimize + remove doc-strings (generates .pyo file, -OO)
+
+ """
+ if (optimize is None) or \
+ (not optimize and __debug__) or \
+ (optimize and not __debug__):
+ return py_compile.compile(filename)
+
+ # Otherwise will have to run a system command
+ python = sys.executable
+ filename = os.path.abspath(filename)
+ if optimize:
+ options = '-' + 'O' * optimize
+ else:
+ options = ''
+ command = "import py_compile; py_compile.compile('%s')" % filename
+ rc = os.system('%s %s -c "%s"' % (python, options, command))
+ if rc != 0:
+ # py_compile currently does not pass back errors, so this
+ # is not really executed...
+ raise SyntaxError,'could not compile "%s"' % filename
+
--- CVS-Python/Lib/distutils/command/install_lib.py Mon Sep 18 10:31:17 2000
+++ Python+Unicode/Lib/distutils/command/install_lib.py Wed Sep 20 13:19:11 2000
@@ -61,27 +61,24 @@ class install_lib (Command):
else:
self.warn("'%s' does not exist -- no Python modules to install" %
self.build_dir)
return
- # (Optionally) compile .py to .pyc
- # XXX hey! we can't control whether we optimize or not; that's up
- # to the invocation of the current Python interpreter (at least
- # according to the py_compile docs). That sucks.
-
+ # (Optionally) compile .py to .pyc and .pyo
if self.compile:
- from py_compile import compile
+ from distutils.util import compile
for f in outfiles:
# only compile the file if it is actually a .py file
if f[-3:] == '.py':
- out_fn = f + (__debug__ and "c" or "o")
- compile_msg = "byte-compiling %s to %s" % \
- (f, os.path.basename (out_fn))
- skip_msg = "skipping byte-compilation of %s" % f
- self.make_file (f, out_fn, compile, (f,),
- compile_msg, skip_msg)
+ for optimize in (0, 1):
+ out_fn = f + (optimize and "o" or "c")
+ compile_msg = "byte-compiling %s to %s" % \
+ (f, os.path.basename (out_fn))
+ skip_msg = "skipping byte-compilation of %s" % f
+ self.make_file (f, out_fn, compile, (f, optimize),
+ compile_msg, skip_msg)
# run ()
def _mutate_outputs (self, has_any, build_cmd, cmd_option, output_dir):
@@ -102,12 +99,12 @@ class install_lib (Command):
# _mutate_outputs ()
def _bytecode_filenames (self, py_filenames):
bytecode_files = []
for py_file in py_filenames:
- bytecode = py_file + (__debug__ and "c" or "o")
- bytecode_files.append(bytecode)
+ bytecode_files.append(py_file + "c")
+ bytecode_files.append(py_file + "o")
return bytecode_files
def get_outputs (self):
"""Return the list of files that would be installed if this command
How do I inform distutils of addition include directories to find headers in
when compiling c source in a python module.
This is outside the package. I'm a user not the author.
My scenerio:
I installed one module that provides a header file.
Rather than inflict my additions on all the users of the system
I'm installing locally to my own home directory with --home=~
This pops the libraries and such into ~/lib/python but drops the header
file into ~/include python.
For the sake of python finding ~/lib/python, PYTHONPATH suffices.
Now I attempt to install a second seperate module that relies
on the first module. Regardless of whether I install this second
module to the system or just to my home directory it needs to
find that header and I can't seem to find the interface through
which to provide distutils with additional include directories
to use when compiling source. Which in this case would be ~/include/python.
The whole scenerio works fine if I'm installing everything to
the system python directories but currently this is not an option
for these modules.
Any help?
Uche, Mike --
I'm digging into BisonGen and 4Suite now. I managed to install BisonGen
(from the sdist) without problems, but there's a dependency you forgot
to document:
$ /usr/local/bin/python
Python 1.5.2 (#5, Sep 8 2000, 22:21:43) [GCC egcs-2.91.66 19990314/Linux (egcs- on linux2
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> import BisonGen.BisonGen
Traceback (innermost last):
File "<stdin>", line 1, in ?
File "/usr/local/python/lib/python1.5/site-packages/BisonGen/BisonGen.py", line 80, in ?
import BGenReader
File "/usr/local/python/lib/python1.5/site-packages/BisonGen/BGenReader.py", line 24, in ?
from xml.sax import saxlib, saxexts
ImportError: No module named xml.sax
>>>
Luckily, I also have PyXML's source dist. sitting around, so I can
install it. Still, you should probably mention this in your README.
Another thing: setting "verbose = 0" in the setup script is the moral
equivalent of setting ".SILENT" in a Makefile -- and IMHO it's
Considered Harmful. Building software *should* be noisy, because so
many things can go wrong. You're free to run the setup script with -q,
but if people are used to seeing noisy builds, then one particular build
being quiet is unsettling. (It unsettled me, that's for sure!)
Yet another: requiring SWIG to be installed in order to build your
software is the moral equivalent of requiring bison and flex on a
non-GNU-powered Unix box -- ie. it's a bother, it's a pain, and it's
completely unnecessary since the software that I now have to go find,
download, build, and install is used *only* to build this particular
package! (I'm speaking here from the narrow, just-want-to-get-the-
silly-thing-working perspective of someone who needs to install 4Suite
in order to get work done. I'm a little embarassed to admit that I
don't even have SWIG installed on my home machine, but it does let me
discover lapses in other people's source dists. ;-)
Now, if SWIG is turning 20k .i files into 400k .c files, that's a good
reason for not distributing the .c files. And I'm starting to get a
grasp of the enormity of 4Suite; this is a boatload-and-a-half of a
code, so I can see why you might want to cut down on the amount of stuff
distributed.
Anyways, looks like I won't be able to debug your bdist_rpm problem this
morning; have to go get SWIG, etc. etc. Grumble.
BTW, congratulations on finding -- and making use of -- every known
"backdoor" entry into the Distutils. At least it looks like the 4Suite
setup.py uses every sneaky trick that I can remember slipping in to the
code at some point. Very gratifying to find that all those tricks are
finally useful to someone! Also, it looks like 4Suite's setup.py is so
complex it will be a good shaker-out of Distutils bugs. Thanks (I
think)!
Greg
--
Greg Ward gward(a)python.net
http://starship.python.net/~gward/
>>>>> "Greg" == Greg Ward <gward(a)python.net> writes:
Greg> On 21 September 2000, Berthold Höllmann said:
>> I think it is easier for the developer to have code and tests
>> together.
Greg> Not if the tests are really exhaustive! When I write good
Greg> tests (or at least what I think are good tests), the test
Greg> script is roughly the size of the module being tested. I
Greg> don't want my "if __name__ == '__main__'" block to be as big
Greg> as the rest of the module. I think the "test" command should
Greg> support "in-situ" testing, but it's *not* enough for serious
Greg> tests.
>> One can leave out the tests from any distribution by using a
>> Manifest file.
Greg> Huh? That loses half the point of writing tests, which is to
Greg> assure people building and installing your modules that they
Greg> probably work as advertised on their platform.
My Idea here was, to distinguish between test to enshure correct
compilation on the "customers" computer and tests to help the
developer my the Manifest file, but to keep all tests with your code
in one (CVS?)repository.
>> But for the developer it is helpful to have a set of tests that
>> one can run after changes have been made to the sources to
>> enshure that "little" changes have not unexpected side effects.
Greg> Yes! That's the other half of the reason for writing tests.
Greg> (You can define "half" here as 40-60, 20-80, whatever.)
>> ...
Greg> Oooh, yes, in a previous life I struggled (briefly) with
Greg> automated testing of numerical algorithms. Not fun. But
Greg> asking end-users to examine the test output and make sure
Greg> that the difference between 1.0234324 and 1.0234821 is less
Greg> than the machine epsilon is surely even less fun.
It begins with the difference between 1.00000E+00 and 1.0000E+000
(where is it?).
>> Also the underlaying Fortran code makes different decisions on
>> different platforms. That makes automatic testing nearly
>> impossible, but I want at least a constistant way to generate
>> the needed output, and I want to test everything before I
>> install it.
Greg> Here's my extreme position:
Greg> If a test isn't fully automated -- i.e., if it takes more
Greg> than epsilon human intelligence to determine if the test
Greg> passed or failed -- then it's not a proper test, it's just a
Greg> demo.
Accepted for "functionality" tests, to be executed by the "customer",
but I don't want to distinguish these from heavier tests, to be
executed by the developer.
Greg> ...
>> All this can be done with my posted, extremly lightweight
>> test.py extension for the _command_ subdirectory. The test
>> scripts can written to implement/use any testframework you
>> like. Maybe a
Greg> OK, lightweight is good. (Something I occasionally need to
Greg> be reminded of...) I'll definitely take a look at it.
Thanks
Greetings
Berthold
--
bhoel(a)starship.python.net / http://starship.python.net/crew/bhoel/
It is unlawful to use this email address for unsolicited ads
(USC Title 47 Sec.227). I will assess a US$500 charge for
reviewing and deleting each unsolicited ad.
Hello,
We are working towards converting 4Suite over to use distutils. We are
almost there, but have run into a problem. We needed to implement our
own parser to generate some files using a script we call BisonGen. This
generates both C and py files. Then we hand the C off to build_ext to
get compilied. The .so (or pyd) is installed fine. However, the
additional py files generated are not copied into the installation
directory. Is there a simple way of doing this? Looking through the
code, build_ext only uses Extension.name to generate a list of outputs.
I've tried a couple of approaches, but all seem to break someother
portion of the script (usually bdist --format=rpm). We have the same
problem with shadow files generated via swig.....
Help is appreciated.
Mike
--
Mike Olson Principal Consultant
mike.olson(a)fourthought.com (303)583-9900 x 102
Fourthought, Inc. http://Fourthought.com
Software-engineering, knowledge-management, XML, CORBA, Linux, Python
The following .cfg file produces an error:
[sdist]
manifest = MANIFEST
no-defaults = 1
keep-tree = 1
formats=bztar,gztar,tar,zip,ztar
...
error: error in setup.cfg: command 'sdist' has no such option 'keep-tree'
Looks as if the config file parser doesn't like options with
embedded hyphens (or maybe it's just me not knowing how to
pass the options ?).
--
Marc-Andre Lemburg
______________________________________________________________________
Business: http://www.lemburg.com/
Python Pages: http://www.lemburg.com/python/
Hi all --
I've just released another code snapshot. The main change is that
Windows installers created on non-Windows platforms should now include
scripts and data correctly. This involved some subtle rearrangement of
the "install" command; as usual, It Works For Me (TM), but please try it
out and see if it works for you.
Also, some patches for Mac OS support have snuck in over the last couple
of days. Download it from the usual place:
http://www.python.org/sigs/distutils-sig/download.html#snapshot
Greg
--
Greg Ward gward(a)python.net
http://starship.python.net/~gward/
>>>>> "Greg" == Greg Ward <gward(a)python.net> writes:
Greg> On 18 September 2000, Berthold Höllmann said:
>> I'm thinking about providing a framework for pre-install
>> testing for distutils enabled Python packages. I think about a
>> new "test" command, which inserts the lib build path into
>> PYTHONPATH and runs specified tests. Is anyone else working on
>> something like this, and what are your opinions and
>> requirements on this?
Greg> I've been thinking about it from Day 1, personally. Here are
Greg> some of my opinions/requirements:
Greg> * serious tests don't belong in the module being tested,
Greg> because test code tends to be at least as long as the code
Greg> being tested
I think it is easier for the developer to have code and tests
together. One can leave out the tests from any distribution by using a
Manifest file. But for the developer it is helpful to have a set of
tests that one can run after changes have been made to the sources to
enshure that "little" changes have not unexpected side effects.
Greg> ...
Greg> * the requirements on what a test script may output should
Greg> be fairly strict. My favourite is loosely based on Perl's
Greg> testing convention, which is a great success largely because
Greg> the standard-way-to-build- Perl-modules includes a "make
Greg> test" step, and it's always immediately clear if everything
Greg> passed. My proposed convention is this: each test script
Greg> outputs a line saying how many tests are expected, then a
Greg> series of lines that start with either "ok" or "not ok". If
Greg> there are any "not ok" lines, the test script fails; if the
Greg> number of "ok" lines != the number of expected tests, the
Greg> test script fails. Each "ok" or "not ok" may be followed by
Greg> ":" and an arbitrary string, which makes it a lot easier to
Greg> track down problems. (The Perl convention is "ok 1", "ok 2",
Greg> ... "ok N", which makes it really awkward to track down
Greg> problems.) Any lines not starting with "ok" or "not ok" are
Greg> ignored.
Thats not always an option. I'm writing modules using numerical
opreations. These require test results to be looked over very
carefully to decide, whether the differences are significant or not.
Also the underlaying Fortran code makes different decisions on
different platforms. That makes automatic testing nearly impossible,
but I want at least a constistant way to generate the needed output,
and I want to test everything before I install it.
Greg> ...
Greg> * we might also want to accomodate a Python-style test
Greg> suite, where you compare "known good" output of a script
Greg> with the current output. I'm not a fan of this methodology,
Greg> but sometimes it's easier, and it may have some currency in
Greg> the Python community outside of Lib/test. It should be the
Greg> developer's choice what style of test suite he wants to
Greg> write.
All this can be done with my posted, extremly lightweight test.py
extension for the _command_ subdirectory. The test scripts can written
to implement/use any testframework you like. Maybe a
>python setup.py test
command is only a convenience command but it gives you a starting
point when maintaining code you wrote ages ago, or somebody else
wrote.
Greetings
Berthold
--
bhoel(a)starship.python.net / http://starship.python.net/crew/bhoel/
It is unlawful to use this email address for unsolicited ads
(USC Title 47 Sec.227). I will assess a US$500 charge for
reviewing and deleting each unsolicited ad.
