[Python-Dev] Fire, may I present accellerant. Accellerant, this is fire.

[Mitchell Morris]

In general I try to keep my head down. I recently needed some 
support for regression testing as part of a stealth Python 
introduction-cum-advocacy effort. I wound up using Steve Purcell's 
pyunit testing framework, and thought it was a shame that it wasn't 
part of the standard distribution.

To that end ...

PEP: eric-the-half-a-pep
Title: Guido-blessed unit/regression test framework
Version: 0
Target of Opportunity: mmorris@mindspring.com (Mitchell Morris)
Python-Version: 2.0
Status: pre-pre-pre-Draft




Introduction

	This proto-PEP advocates that there should be, in the standard
	library, a unit/regression test framework. This Guido-blessed
	framework would provide test semantics both useful and
	standardized. In addition to being a useful structure in
	it's own right, it would provide an obvious direction for
	development projects to prepare and use their own regression
	test policies.




Why Should I Care?

	Software developers have known for quite some time that the
	majority of our time is spent identifying and correcting
	errors. Various suggestions for reducing this burden have
	been explored and most of them have been rejected. One thing
	that seems to provide some benefit is writing test plans
	and running them against the software product. This doesn't
	necessarily eliminate bugs, but it does allow us to find some
	of them and insure they don't reoccur as we correct later ones.

	One variation of this process is to extend the test plan
	as other defects are identified and corrected. Part of the
	correction would be updating the test suite to include a
	test that would have identified and exercised the defect. In
	this way the test plan grows to verify that correct behavior
	increases monotonically.

	This kind of test plan also allows very aggressive refactoring,
	in that the developer can make sweeping changes to module
	internals with great confidence that things will either (a)
	not break elsewhere, or (b) break loudly when the unit tests
	fail. Given the (relative) success of test plans and Python's
	general friendliness to rapid development and refactoring,
	there should be a standard tool in all distributions to
	encourage this dynamic.




But ... Smalltalk and Java are Evil!

	Kent Beck has recently generated a lot of press by writing
	about his development methodology which he calls "Extreme
	Programming"[1].  Aside from the unfortunate connotations
	to sports which would still be completely anonymous without
	ESPN2, there are a number of useful tools emerging from the 
XP
	camp. One of the prime tenets is that developers write their
	classes and their unit tests at the same time.	According to
	XP, these unit tests have to run at 100% for the code to be
	acceptable for integration.

	The design of the testing framework generated by XP is fairly
	simple: individual tests check the correct functionality of a
	single quantum of testable behavior, usually public behavior.
	These individual tests can be aggregated into suites, which
	are also runnable. Suites can be aggregated as well. Tools
	are provided to execute these tests and suites in both text
	and GUI formats.

	Kent Beck and Erich Gamma wrote a paper called "Test
	Infected"[2] to demonstrate how the code and tests are
	developed in tandem. This paper, written as a script for a
	pair programming session, shows how the various pieces of the
	JUnit implementation work together. (JUnit is the XP testing
	tool for Java.)




We've Already Got One, Thanks.

	The current distribution has a regression testing framework,
	in Lib/test/regrtest.py, which addresses some of these same
	issues.  The obvious question is why not use the tool which is
	already there?	The current tool presents several issues that
	make this a less-than-exemplary solution: it doesn't allow for
	graphical manipulation, it separates test and expected result
	into separate files stored in separate locations, and it tries
	to generate the tests as well as run them. The objections to
	regrtest.py then boil down to the it's philosophy.




We've Got More Than We Can Stand, Thanks. Really.

	Also for our consideration is doctest.py, contributed by
	the inestimable Tim Peters. doctest.py presents a simple
	straightforward means to embed the individual tests into the
	production code itself as commentary. This suggests that the
	separation of test and result presents a cognitive dissonance
	in other people besides just me. doctest.py doesn't provide
	any mechanism, however, for aggregating several modules
	into a single test run, nor does it provide a GUI. Once
	again, additional effort could obviously provide the missing
	functionality. On the other hand, doctest.py isn't currently
	part of the standard distribution.




What's All This Then?

	The point of this proposal is Steve Purcell's pyunit
	package[3]. This is a Python implementation of XP's xUnit
	specification. It provides a means for running individual
	tests, aggregating individual tests into larger suites,
	running those tests in both text and GUI modes, and keeps the
	tests and expected results together in a single source module.




You Have A Point?

	Yes: pyunit should be part of the standard library. pyunit
	documentation should be included in the distribution. New
	development should be encouraged to use pyunit. We should
	all rub pyunit all over our bodies as often as we can.

	Even if it isn't pyunit, Python should have a documented
	standard component for unit and regression testing, and we
	should encourage new development to use it, and we should
	demand that core development include it.




References

1 -- XP Wiki
	<URL:http://c2.com/cgi/wiki?ExtremeProgramming>

	see also:
	<URL:http://www.xprogramming.com/>

2 -- Beck, Kent and Gamma, Erich, "Test Infected"
	<URL:http://members.pingnet.ch/gamma/junit.htm>

3 -- pyunit project at Source Forge
	<URL:http://pyunit.sourceforge.net/>

pyromanically y'rs
+Mitchell