[Python-Dev] unittest: shortDescription, _TextTestResult and other issues

[Michael Foord]

Hello all,

I've been looking at outstanding unittest issues as part of my 
preparation for my PyCon talk. There are a couple of changes (minor) I'd 
like to make that I thought I ought to run past Python-Dev first. If I 
don't get any responses then I'll just do it, so you have been warned. :-)

The great google merge into unittest happened at PyCon last year [1]. 
This included a change to TestCase.shortDescription() so that it would 
*always* include the test name, whereas previously it would return the 
test docstring or None.

The problem this change solved was that tests with a docstring would not 
have their name (test class and method name) reported during the test 
run. Unfortunately the change broke part of twisted test running. 
Reported as issue 7588:

http://bugs.python.org/issue7588

It seems to me that the same effect (always reporting test name) can be 
achieved in _TextTestResult.getDescription(). I propose to revert the 
change to TestCase.shortDescription() (which has both a horrible name 
and a horrible implementation and should probably be renamed 
getDocstring so that what it does is obvious but never mind) and put the 
change into _TextTestResult.

It annoys me that _TextTestResult is private, as you will almost 
certainly want to use it or subclass it when implementing custom test 
systems. I am going to rename it TextTestResult, alias the old name and 
document the old name as being deprecated.

Another issue that I would like to address, but there are various 
possible approaches, is issue 7559: http://bugs.python.org/issue7559
Currently loadTestsFromName catches ImportError and rethrows as 
AttributeError. This is horrible (it obscures the original error) but 
there are backwards compatibility issues with fixing it. There are three 
possible approaches:

1) Leave it (the default)
2) Only throw an AttributeError if the import fails due to the name 
being invalid (the module not existing) otherwise allow the error 
through. (A minor but less serious change in behavior).
3) A new method that turns failures into pseudo-tests that fail with the 
original error when run. Possibly deprecating loadTestsFromName

I favour option 3, but can't think of a good replacement name. :-)

Comments welcomed.

Despite deprecating (in the documentation - no actual deprecations 
warnings I believe) a lot of the duplicate ways of doing things (assert* 
favoured over fail* and assertEqual over assertEquals) we didn't include 
deprecating assert_ in favour of assertTrue. I would like to add that to 
the documentation. After 3.2 is out I would like to clean up the 
documentation, removing mention of the deprecated methods from the 
*main* documentation into a separate 'deprecated methods' section. They 
currently make the documentation very untidy. The unittest page should 
probably be split into several pages anyway and needs improving.

Other outstanding minor issues:

Allow dotted names for test discovery
http://bugs.python.org/issue7780 - I intend to implement this as 
described in the last comment

A 'check_order' optional argument (defaulting to True) for 
assertSequenceEqual
http://bugs.python.org/issue7832 - needs patch

The breaking of __unittest caused by splitting unittesst into a package 
needs fixing. The fix needs to work when Python is run without frames 
support (IronPython).
http://bugs.python.org/issue7815 - needs patch

Allow a __unittest (or similar) decorator for user implemented assert 
functions
http://bugs.python.org/issue1705520 - needs patch

Allow modules to define test_suite callable.
http://bugs.python.org/issue7501 - I propose to close as rejected. Use 
load_tests instead.

Display time taken by individual tests when in verbose mode.
http://bugs.python.org/issue4080 - anyone any opinions?

Allow automatic formatting of arguments in assert* failure messages.
http://bugs.python.org/issue6966 - I propose to close as rejected

removeTest() method on TestSuite
http://bugs.python.org/issue1778410 - anyone any opinions?

expect methods (delayed fail)
http://bugs.python.org/issue3615 - any opinions? Personally I think that 
the TestCase API is big enough already


All the best,

Michael Foord

[1] Mostly in revision 7-837. 
http://svn.python.org/view?view=rev&revision=70837




-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/blog

READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.