Hello,

I'm thinking about providing a framework for pre-install testing for distutils enabled Python packages. I think about a new "test" command, which inserts the lib build path into PYTHONPATH and runs specified tests. Is anyone else working on something like this, and what are your opinions and requirements on this?

A good idea! Related: IMHO, distutils needs a test suite for itself! Thomas

On 18 September 2000, Berthold Höllmann said:

I'm thinking about providing a framework for pre-install testing for distutils enabled Python packages. I think about a new "test" command, which inserts the lib build path into PYTHONPATH and runs specified tests. Is anyone else working on something like this, and what are your opinions and requirements on this?

I've been thinking about it from Day 1, personally. Here are some of my opinions/requirements: * serious tests don't belong in the module being tested, because test code tends to be at least as long as the code being tested * however, ensuring that each module imports (and possibly runs as a script) is not a bad idea. Running modules as scripts should be an option that the developer can turn off. * the convention I've always leaned towards is that test/test*.py is your test suite; run each of those scripts in turn and analyze the results to determine if the entire module distribution passes * the requirements on what a test script may output should be fairly strict. My favourite is loosely based on Perl's testing convention, which is a great success largely because the standard-way-to-build- Perl-modules includes a "make test" step, and it's always immediately clear if everything passed. My proposed convention is this: each test script outputs a line saying how many tests are expected, then a series of lines that start with either "ok" or "not ok". If there are any "not ok" lines, the test script fails; if the number of "ok" lines != the number of expected tests, the test script fails. Each "ok" or "not ok" may be followed by ":" and an arbitrary string, which makes it a lot easier to track down problems. (The Perl convention is "ok 1", "ok 2", ... "ok N", which makes it really awkward to track down problems.) Any lines not starting with "ok" or "not ok" are ignored. * there should be a nice framework that makes it easy to write good tests and play with the above rules. Such a framework exists in prototype form and has been used quite successfully for several months internally where I work; it suffers from code bloat and creeping featuritis but is extremely useful. * but developers should be allowed to go their own way and ignore the "official" testing framework: some things don't fit into frameworks! * we might also want to accomodate a Python-style test suite, where you compare "known good" output of a script with the current output. I'm not a fan of this methodology, but sometimes it's easier, and it may have some currency in the Python community outside of Lib/test. It should be the developer's choice what style of test suite he wants to write. Greg -- Greg Ward gward@python.net http://starship.python.net/~gward/