[Python-Dev] Fire, may I present accellerant. Accellerant, this is fire.
Fri, 28 Jul 2000 23:19:59 -0700
In general I try to keep my head down. I recently needed some
support for regression testing as part of a stealth Python
introduction-cum-advocacy effort. I wound up using Steve Purcell's
pyunit testing framework, and thought it was a shame that it wasn't
part of the standard distribution.
To that end ...
Title: Guido-blessed unit/regression test framework
Target of Opportunity: firstname.lastname@example.org (Mitchell Morris)
This proto-PEP advocates that there should be, in the standard
library, a unit/regression test framework. This Guido-blessed
framework would provide test semantics both useful and
standardized. In addition to being a useful structure in
it's own right, it would provide an obvious direction for
development projects to prepare and use their own regression
Why Should I Care?
Software developers have known for quite some time that the
majority of our time is spent identifying and correcting
errors. Various suggestions for reducing this burden have
been explored and most of them have been rejected. One thing
that seems to provide some benefit is writing test plans
and running them against the software product. This doesn't
necessarily eliminate bugs, but it does allow us to find some
of them and insure they don't reoccur as we correct later ones.
One variation of this process is to extend the test plan
as other defects are identified and corrected. Part of the
correction would be updating the test suite to include a
test that would have identified and exercised the defect. In
this way the test plan grows to verify that correct behavior
This kind of test plan also allows very aggressive refactoring,
in that the developer can make sweeping changes to module
internals with great confidence that things will either (a)
not break elsewhere, or (b) break loudly when the unit tests
fail. Given the (relative) success of test plans and Python's
general friendliness to rapid development and refactoring,
there should be a standard tool in all distributions to
encourage this dynamic.
But ... Smalltalk and Java are Evil!
Kent Beck has recently generated a lot of press by writing
about his development methodology which he calls "Extreme
Programming". Aside from the unfortunate connotations
to sports which would still be completely anonymous without
ESPN2, there are a number of useful tools emerging from the
camp. One of the prime tenets is that developers write their
classes and their unit tests at the same time. According to
XP, these unit tests have to run at 100% for the code to be
acceptable for integration.
The design of the testing framework generated by XP is fairly
simple: individual tests check the correct functionality of a
single quantum of testable behavior, usually public behavior.
These individual tests can be aggregated into suites, which
are also runnable. Suites can be aggregated as well. Tools
are provided to execute these tests and suites in both text
and GUI formats.
Kent Beck and Erich Gamma wrote a paper called "Test
Infected" to demonstrate how the code and tests are
developed in tandem. This paper, written as a script for a
pair programming session, shows how the various pieces of the
JUnit implementation work together. (JUnit is the XP testing
tool for Java.)
We've Already Got One, Thanks.
The current distribution has a regression testing framework,
in Lib/test/regrtest.py, which addresses some of these same
issues. The obvious question is why not use the tool which is
already there? The current tool presents several issues that
make this a less-than-exemplary solution: it doesn't allow for
graphical manipulation, it separates test and expected result
into separate files stored in separate locations, and it tries
to generate the tests as well as run them. The objections to
regrtest.py then boil down to the it's philosophy.
We've Got More Than We Can Stand, Thanks. Really.
Also for our consideration is doctest.py, contributed by
the inestimable Tim Peters. doctest.py presents a simple
straightforward means to embed the individual tests into the
production code itself as commentary. This suggests that the
separation of test and result presents a cognitive dissonance
in other people besides just me. doctest.py doesn't provide
any mechanism, however, for aggregating several modules
into a single test run, nor does it provide a GUI. Once
again, additional effort could obviously provide the missing
functionality. On the other hand, doctest.py isn't currently
part of the standard distribution.
What's All This Then?
The point of this proposal is Steve Purcell's pyunit
package. This is a Python implementation of XP's xUnit
specification. It provides a means for running individual
tests, aggregating individual tests into larger suites,
running those tests in both text and GUI modes, and keeps the
tests and expected results together in a single source module.
You Have A Point?
Yes: pyunit should be part of the standard library. pyunit
documentation should be included in the distribution. New
development should be encouraged to use pyunit. We should
all rub pyunit all over our bodies as often as we can.
Even if it isn't pyunit, Python should have a documented
standard component for unit and regression testing, and we
should encourage new development to use it, and we should
demand that core development include it.
1 -- XP Wiki
2 -- Beck, Kent and Gamma, Erich, "Test Infected"
3 -- pyunit project at Source Forge