Recently, Guido van Rossum <guido@python.org> said:
You said that when you run the test standalone the type names are different. *How* do you run it standalone? If I run it like this:
./python Lib/test/test_descrtut.py
it works fine (and -v works here too). But if I run it like this:
./python >>> import test.test_descrtut # this does not run the tests! >>> test.test_descrtut.test_main() # this does!
Indeed, I run it the second way (no command line on the Mac, remember:-). The fun thing is that if I run it completely standalone (by dragging test_descrtut.py to the interpreter, the test can handle this situation) it works fine! So, there must be something wrong with either the test framework (or maybe this re-import trick?) that doesn't work as expected on the Mac. Maybe test_regrtest can be thought which tests use unittest or doctest, and not try to be helpful in those cases? For now I've just put a note in the readme file for MacPython that this test is expected to fail, but I'd like to fix it eventually, of course. I have another gripe with the new unittest stuff (this is the first time I've seen the doctest thing, never knew it was there!), and that's that most of the test failures are difficult to interpret. Whereas the old-style tests simply "print math.sin(math.pi)" and tell you they expected the output to be 0 but got -1 in stead the unittest-based tests often don't give that information. Most of the unittest-based tests use only failUnless/assert_ in stead of the higher-level functions, i.e. test_time uses things like self.assert_(time.ctime(self.t) == time.asctime(time.localtime(self.t))) where the test results would be a lot easier to interpret if they used self.assertEqual(time.ctime(self.t), time.asctime(time.localtime(self.t)), "ctime(T) != asctime(localtime(T)") -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | see http://www.xs4all.nl/~tank/spg-l/sigaction.htm
[Jack Jansen]
... The fun thing is that if I run it completely standalone (by dragging test_descrtut.py to the interpreter, the test can handle this situation) it works fine! So, there must be something wrong with either the test framework (or maybe this re-import trick?) that doesn't work as expected on the Mac.
It's that the name of a module depends on how it's first imported, and that's got nothing to do with Macs or the framework. It does have to do with the exact way in which you run the tests; apparently nobody else runs tests that way, or nobody else runs tests at all <wink>. BTW, the "re-import trick" is neither a trick nor a re-import: doctest takes a module object as input, and importing a module is the *obvious* way to get a module object. The module may or may not have been imported before, but even it was it wasn't imported before *by* the doctested module; so it's no more of a re-import than two modules both deciding to import, say, math or sys. Unfortunately, unlike as for math or sys, the name of a module in a package is fuzzy.
Maybe test_regrtest can be thought which tests use unittest or doctest, and not try to be helpful in those cases?
Sure -- but it's not in my plans.
For now I've just put a note in the readme file for MacPython that this test is expected to fail, but I'd like to fix it eventually, of course.
Or you can include the hacked version I just checked in (forcing "test." prefix gibberish to show up in the module name no matter how it's run, via Guido's suggested trick).
I have another gripe with the new unittest stuff (this is the first time I've seen the doctest thing, never knew it was there!), and that's that most of the test failures are difficult to interpret. Whereas the old-style tests simply "print math.sin(math.pi)" and tell you they expected the output to be 0 but got -1 in stead the unittest-based tests often don't give that information. Most of the unittest-based tests use only failUnless/assert_ in stead of the higher-level functions, i.e. test_time uses things like self.assert_(time.ctime(self.t) == time.asctime(time.localtime(self.t))) where the test results would be a lot easier to interpret if they used self.assertEqual(time.ctime(self.t), time.asctime(time.localtime(self.t)), "ctime(T) != asctime(localtime(T)")
The same gripe is just as true of many of the straight regrtests, using test_support.verify() lazily, as in verify(D.goo(1) == (D, 1)) verify(str(c1).find('C instance at ') >= 0) verify(d == {}) verify(lines == ["A\n", "B\n", "C", "D\n", "E\n", "F"]) verify(len(m) == 2*PAGESIZE) verify(str(e) == 'Truncated input file') verify(re.split("(?::*)", ":a:b::c") == ['', 'a', 'b', 'c']) and on & on & on. There are over 900 (static) calls of verify(), and it's rare to see the optional "explanation" argument. Many of those in turn were originally lazy uses of the assert stmt (also skipping the "reason" clause). This is human nature <0.1 wink>, and it played a large role in doctest's design. For example, a doctest for the last one there could look like: """
import re re.split("(?::*)", ":a:b::c") ['', 'a', 'b', 'c'] """
and if it fails you get a message showing you the example, the expected output, and the output you actually got. Like ***************************************************************** Failure in example: re.split("(?::*)", ":a:b::c") from line #2 of dc Expected: ['', 'a', 'b', ':c'] Got: ['', 'a', 'b', 'c'] ***************************************************************** Less typing and better error reports. Multiline "expected" and "got" stuff work just as slick, which regrtest doesn't handle well (it compares output at a low per-.write() level, not at line granularity; that's why when a straight regrtest fails, you'll sometimes see a string of "expected" characters that come out of the middle of one of the expected-output lines).
You said that when you run the test standalone the type names are different. *How* do you run it standalone? If I run it like this:
./python Lib/test/test_descrtut.py
it works fine (and -v works here too). But if I run it like this:
./python >>> import test.test_descrtut # this does not run the tests! >>> test.test_descrtut.test_main() # this does!
Indeed, I run it the second way (no command line on the Mac, remember:-).
The fun thing is that if I run it completely standalone (by dragging test_descrtut.py to the interpreter, the test can handle this situation) it works fine! So, there must be something wrong with either the test framework (or maybe this re-import trick?) that doesn't work as expected on the Mac.
To find out, try this: >>> import sys >>> sys.argv = ['regrtest.py', '-v', 'test_decrtut'] >>> import test.regrtest # this does not run any tests >>> test.regrtest.main() This should run just the one test, under regrtest, in verbose mode, so without regrtest comparing and interpreting the output of doctest.
Maybe test_regrtest can be thought which tests use unittest or doctest, and not try to be helpful in those cases?
That's up to Tim; sounds like a good idea to me but requires that regrtest knows which tests use doctest.
For now I've just put a note in the readme file for MacPython that this test is expected to fail, but I'd like to fix it eventually, of course.
Of course.
I have another gripe with the new unittest stuff (this is the first time I've seen the doctest thing, never knew it was there!), and that's that most of the test failures are difficult to interpret. Whereas the old-style tests simply "print math.sin(math.pi)" and tell you they expected the output to be 0 but got -1 in stead the unittest-based tests often don't give that information. Most of the unittest-based tests use only failUnless/assert_ in stead of the higher-level functions, i.e. test_time uses things like self.assert_(time.ctime(self.t) == time.asctime(time.localtime(self.t))) where the test results would be a lot easier to interpret if they used self.assertEqual(time.ctime(self.t), time.asctime(time.localtime(self.t)), "ctime(T) != asctime(localtime(T)")
Absolutely! Doctest is actually better in this respect (when not thwarted by regrtest) because it does this automatically, but we should definitely try to use self.assertEqual(x, y) rather than self.assert(x == y)! (If you find any examples, please fix them or at least report them. --Guido van Rossum (home page: http://www.python.org/~guido/)
[Jack]
Maybe test_regrtest can be thought which tests use unittest or doctest, and not try to be helpful in those cases?
[Guido]
That's up to Tim; sounds like a good idea to me but requires that regrtest knows which tests use doctest.
OK, done and checked in. For example, after planting two errors in test_generators.py now:
python ../lib/test/regrtest.py test_cookie test_generators test_descrtut test_cookie test_generators
Failure in example: print list(g2()) from line #80 of test_generators.__test__.tut Expected: [24] Got: [42] ***************************************************************** Failure in example: list(zrange(5)) from line #128 of test_generators.__test__.tut Expected: [0, 1, 3, 4] Got: [0, 1, 2, 3, 4] ***************************************************************** 1 items had failures: 2 of 29 in test_generators.__test__.tut ***Test Failed*** 2 failures. test test_generators failed -- 2 of 136 doctests failed test_descrtut 2 tests OK. 1 test failed: test_generators
participants (3)
-
Guido van Rossum -
Jack Jansen -
Tim Peters