[Distutils] test command for setup.py

[Greg Ward]

On 18 September 2000, Berthold Höllmann said:
> I'm thinking about providing a framework for pre-install testing for
> distutils enabled Python packages. I think about a new "test" command,
> which inserts the lib build path into PYTHONPATH and runs specified
> tests. Is anyone else working on something like this, and what are
> your opinions and requirements on this?

I've been thinking about it from Day 1, personally.  Here are some of my 
opinions/requirements:

  * serious tests don't belong in the module being tested, because test
    code tends to be at least as long as the code being tested

  * however, ensuring that each module imports (and possibly runs as
    a script) is not a bad idea.  Running modules as scripts should
    be an option that the developer can turn off.

  * the convention I've always leaned towards is that test/test*.py
    is your test suite; run each of those scripts in turn and analyze
    the results to determine if the entire module distribution
    passes

  * the requirements on what a test script may output should be fairly
    strict.  My favourite is loosely based on Perl's testing convention,
    which is a great success largely because the standard-way-to-build-
    Perl-modules includes a "make test" step, and it's always
    immediately clear if everything passed.  My proposed convention is
    this: each test script outputs a line saying how many tests are
    expected, then a series of lines that start with either "ok" or "not
    ok".  If there are any "not ok" lines, the test script fails; if the
    number of "ok" lines != the number of expected tests, the test
    script fails.  Each "ok" or "not ok" may be followed by ":" and
    an arbitrary string, which makes it a lot easier to track down
    problems.  (The Perl convention is "ok 1", "ok 2", ... "ok N", which
    makes it really awkward to track down problems.)  Any lines not
    starting with "ok" or "not ok" are ignored.

  * there should be a nice framework that makes it easy to write good
    tests and play with the above rules.  Such a framework exists in
    prototype form and has been used quite successfully for several
    months internally where I work; it suffers from code bloat and
    creeping featuritis but is extremely useful.

  * but developers should be allowed to go their own way and ignore
    the "official" testing framework: some things don't fit into
    frameworks!

  * we might also want to accomodate a Python-style test suite, where
    you compare "known good" output of a script with the current output.
    I'm not a fan of this methodology, but sometimes it's easier, and it
    may have some currency in the Python community outside of Lib/test.
    It should be the developer's choice what style of test suite he
    wants to write.

        Greg
-- 
Greg Ward                                      gward@python.net
http://starship.python.net/~gward/