[Distutils] test command for setup.py
Tue Sep 19 21:28:12 2000
On 18 September 2000, Berthold Höllmann said:
> I'm thinking about providing a framework for pre-install testing for
> distutils enabled Python packages. I think about a new "test" command,
> which inserts the lib build path into PYTHONPATH and runs specified
> tests. Is anyone else working on something like this, and what are
> your opinions and requirements on this?
I've been thinking about it from Day 1, personally. Here are some of my
* serious tests don't belong in the module being tested, because test
code tends to be at least as long as the code being tested
* however, ensuring that each module imports (and possibly runs as
a script) is not a bad idea. Running modules as scripts should
be an option that the developer can turn off.
* the convention I've always leaned towards is that test/test*.py
is your test suite; run each of those scripts in turn and analyze
the results to determine if the entire module distribution
* the requirements on what a test script may output should be fairly
strict. My favourite is loosely based on Perl's testing convention,
which is a great success largely because the standard-way-to-build-
Perl-modules includes a "make test" step, and it's always
immediately clear if everything passed. My proposed convention is
this: each test script outputs a line saying how many tests are
expected, then a series of lines that start with either "ok" or "not
ok". If there are any "not ok" lines, the test script fails; if the
number of "ok" lines != the number of expected tests, the test
script fails. Each "ok" or "not ok" may be followed by ":" and
an arbitrary string, which makes it a lot easier to track down
problems. (The Perl convention is "ok 1", "ok 2", ... "ok N", which
makes it really awkward to track down problems.) Any lines not
starting with "ok" or "not ok" are ignored.
* there should be a nice framework that makes it easy to write good
tests and play with the above rules. Such a framework exists in
prototype form and has been used quite successfully for several
months internally where I work; it suffers from code bloat and
creeping featuritis but is extremely useful.
* but developers should be allowed to go their own way and ignore
the "official" testing framework: some things don't fit into
* we might also want to accomodate a Python-style test suite, where
you compare "known good" output of a script with the current output.
I'm not a fan of this methodology, but sometimes it's easier, and it
may have some currency in the Python community outside of Lib/test.
It should be the developer's choice what style of test suite he
wants to write.
Greg Ward firstname.lastname@example.org