[Python-Dev] Guidance regarding tests for the standard lib

[Terry Reedy]

On 8/13/2013 7:51 PM, Steven D'Aprano wrote:

> http://bugs.python.org/issue18606

Tests at end of statistics.patch.

> and I'm about to submit a patch containing my updated code and tests,
> but I've run into a problem with testing. My existing tests use
> unittest, and follow the basic boilerplate documented here:
>
> http://docs.python.org/3/library/test.html

+def test_main():
+    # run_unittest()
+    unittest.main()
+
+
+if __name__ == "__main__":
+    test_main()

This is faulty, as explained below. It is not the boilerplate in
http://docs.python.org/3/library/test.html#writing-unit-tests-for-the-test-package

which is correct. Compress to just

if __name__ == "__main__":
    unittest_main()

> To avoid all doubt, the tests pass if I call them like this:
> ./python Lib/test/test_statistics.py

That is a faulty test as explained below.

Patch applies cleanly on Win7, 32bit build. And indeed,
F:\Python\dev\py34\PCbuild>python_d ../Lib/test/test_statistics.py
works. (the progess dots will have to go before being applied ;-).

The above should be equivalent to
...> python -m test.test_statistics.
but it is not.

F:\Python\dev\py34\PCbuild>python_d -m test.test_statistics
Traceback (most recent call last):
   File "F:\Python\dev\py34\lib\runpy.py", line 160, in _run_module_as_main
     "__main__", fname, loader, pkg_name)
   File "F:\Python\dev\py34\lib\runpy.py", line 73, in _run_code
     exec(code, run_globals)
   File "F:\Python\dev\py34\lib\test\test_statistics.py", line 16, in 
<module>
     from test_statistics_approx import NumericTestCase
ImportError: No module named 'test_statistics_approx'

Your test *depends* on Lib/test being the current directory, at the 
beginning of the path. It should not. Your import must be
    from test.test_statistics_appox import NumericTestCase

With this fixed, the above command works.

Once the test runs from anywhere with unittest, run with regrtest,

...> python -m test test_statistics

Initially, this gives a traceback with the same last three lines. With 
the revision, I get

   File "F:\Python\dev\py34\lib\unittest\loader.py", line 113, in 
loadTestsFromName
     parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'test_statistics'

This is related to the first fault I mentioned above. If regrtest finds 
the old style 'test_main', it uses the regrtest loader. If it does not, 
it uses the unittest loader. This is documented in the code ;-). With 
the proper 2-line boilerplate as a second change, the second line now works.

> but raise errors when I call them like this:
> ./python -m test -j3

LOL. In one jump, you changed the current directory, the test runner, 
the number of test files run, and the mode of testing. Try just one 
change at a time.

When you run the suite, you run test_statistics_approx.py, including the 
test case imported into test_statistics. (Is it run twice?).

Tested as part of the suite gives a bizarre message I do not pretend to 
understand.
[290/379/3] test_statistics_approx
Usage: regrtest.py [options]

regrtest.py: error: no such option: --slaveargs
test test_statistics_approx crashed -- Traceback (most recent call last):
   File "F:\Python\dev\py34\lib\optparse.py", line 1391, in parse_args
     stop = self._process_args(largs, rargs, values)
   File "F:\Python\dev\py34\lib\optparse.py", line 1431, in _process_args
     self._process_long_opt(rargs, values)
   File "F:\Python\dev\py34\lib\optparse.py", line 1484, in 
_process_long_opt
     opt = self._match_long_opt(opt)
   File "F:\Python\dev\py34\lib\optparse.py", line 1469, in _match_long_opt
     return _match_abbrev(opt, self._long_opt)
   File "F:\Python\dev\py34\lib\optparse.py", line 1674, in _match_abbrev
     raise BadOptionError(s)
optparse.BadOptionError: no such option: --slaveargs

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
   File "F:\Python\dev\py34\lib\test\regrtest.py", line 1305, in 
runtest_inner
     test_runner()
   File "F:\Python\dev\py34\lib\test\test_statistics_approx.py", line 
597, in test_main
     unittest.main()
   File "F:\Python\dev\py34\lib\unittest\main.py", line 124, in __init__
     self.parseArgs(argv)
   File "F:\Python\dev\py34\lib\unittest\main.py", line 148, in parseArgs
     options, args = parser.parse_args(argv[1:])
   File "F:\Python\dev\py34\lib\optparse.py", line 1393, in parse_args
     self.error(str(err))
   File "F:\Python\dev\py34\lib\optparse.py", line 1573, in error
     self.exit(2, "%s: error: %s\n" % (self.get_prog_name(), msg))
   File "F:\Python\dev\py34\lib\optparse.py", line 1563, in exit
     sys.exit(status)
SystemExit: 2

So back up and test it by itself first. It passes under unittest, but 
needs the revised boilerplate for regrtest. The two together work fine. 
When I rerun the suite, poof!, the mysterious tracback is gone and both 
pass.

The following appears after the 2nd. '''
..
----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK
'''
Perhaps this is from the doctest. If so, it probably should be surpressed.

One last thing: test order. Buildbots randomize the order of running 
files, so if your tests modify the environment in a way that affects 
another test, it might cause occasional failures.

-- 
Terry Jan Reedy