test_descrtut failing
Test_descrtut is driving me up the wall (or actually, the whole test suite is:-). If I run it from regrtest it fails with the completely useless message test_descrtut The actual stdout doesn't match the expected stdout. This much did match (between asterisk lines): ********************************************************************** test_descrtut ********************************************************************** Then ... We expected (repr): '' But instead we got: '*****************************************************************\n' test test_descrtut failed -- Writing: '*****************************************************************\n', expected: '' If I run it standalone it doesn't work because all the typenames are different. This is a general problem, I've seen the same on unix. In general, I find the new test framework not much of an improvement. Whereas with old tests you would get some indication as to what was failing, currently it is at best a not very helpful "We expected '' But in stead we got 'TestFailed'", and at worst something like the above. -- Jack Jansen | ++++ stop the execution of Mumia Abu-Jamal ++++ Jack.Jansen@oratrix.com | ++++ if you agree copy these lines to your sig ++++ www.cwi.nl/~jack | ++++ see http://www.xs4all.nl/~tank/ ++++
Test_descrtut is driving me up the wall (or actually, the whole test suite is:-).
If I run it from regrtest it fails with the completely useless message test_descrtut The actual stdout doesn't match the expected stdout. This much did match (between asterisk lines): ********************************************************************** test_descrtut ********************************************************************** Then ... We expected (repr): '' But instead we got: '*****************************************************************\n' test test_descrtut failed -- Writing: '*****************************************************************\n', expected: ''
If I run it standalone it doesn't work because all the typenames are different. This is a general problem, I've seen the same on unix.
In general, I find the new test framework not much of an improvement. Whereas with old tests you would get some indication as to what was failing, currently it is at best a not very helpful "We expected '' But in stead we got 'TestFailed'", and at worst something like the above.
The problem seems to be that both doctest (which is just *one* of the
new test frameworks -- the other one is unittest) and regrtest (the
regression test suite driver) are trying to be friendly when they
encounter a failed test. One of the tests fails, and doctest prints
its failure message, which looks something like this:
*****************************************************************
Failure in example: C().foo()
from line #23 of test_descrtut.__test__.tut8
Expected: zcalled A.foo()
Got: called A.foo()
*****************************************************************
1 items had failures:
1 of 6 in test_descrtut.__test__.tut8
***Test Failed*** 1 failures.
But regrtest, which is comparing the output to the contents of the
file Lib/test/output/test_descrtut, doesn't expect that kind of
output, so *it* attempts to print a report of a failure, which looks
something like this:
The actual stdout doesn't match the expected stdout.
This much did match (between asterisk lines):
**********************************************************************
test_descrtut
**********************************************************************
Then ...
We expected (repr): ''
But instead we got: 'Blargh'
test test_descrtut failed -- Writing: 'Blargh', expected: ''
Except that if a doctest test fails, the unexpected output is not
'Blargh' but a line of asterisks followed by a newline. The unexected
output is repeated twice in the regrtest output.
The trick to finding out what's wrong with the test is to run it as
follows:
./python Lib/test/regrtest.py -v test_descrtut
This runs the test in "verbose mode", and in that mode regrtest
doesn't do its output comparison trick -- it just shows you all the
test's output. Now, doctest also scans sys.argv for a -v (which I
think is wrong because it's not the main program, but that's how it
works), and this causes doctest to be much more verbose -- which may
or may not be helpful but shouldn't be a problem unless you are
without a scrollbar.
You said that when you run the test standalone the type names are
different. *How* do you run it standalone? If I run it like this:
./python Lib/test/test_descrtut.py
it works fine (and -v works here too). But if I run it like this:
./python
>>> import test.test_descrtut # this does not run the tests!
>>> test.test_descrtut.test_main() # this does!
then indeed doctset complains that instead of
[Jack Jansen]
Test_descrtut is driving me up the wall (or actually, the whole test suite is:-). ... If I run it standalone it doesn't work because all the typenames are different. This is a general problem, I've seen the same on unix.
What does "run it standalone" mean to you, exactly? Here are 4 possible meanings that work fine for me (running it alone with or without regrtest, and from the test directory or from my build directory; and I haven't been able to dream up another meaning!): C:\Code\python\PCbuild>python ../lib/test/test_descrtut.py C:\Code\python\PCbuild>python ../lib/test/regrtest.py test_descrtut.py test_descrtut 1 test OK. C:\Code\python\Lib\test>..\..\pcbuild\python test_descrtut.py C:\Code\python\Lib\test>..\..\pcbuild\python regrtest.py test_descrtut.py test_descrtut 1 test OK.
In general, I find the new test framework not much of an improvement.
The problem is more that we don't have a new framework: we've squashed additional frameworks *under* a catch-all old framework, and the latter doesn't really know how to deal with the former. unittest and doctest both print their failure reports to stdout, but stdout is what regrtest is looking at, and the latter complains if anything shows up on stdout that it doesn't expect. [Guido]
... But regrtest, which is comparing the output to the contents of the file Lib/test/output/test_descrtut,
Which latter doesn't exist -- that's another twist in the tale.
The actual stdout doesn't match the expected stdout. This much did match (between asterisk lines): ********************************************************************** test_descrtut **********************************************************************
When an expected-output file doesn't exist, regrtest "pretends" that it does exist, and consists of a single line containing the name of the test. That's why it says "This much did match" despite that output/test_descrtut doesn't actually exist.
... The trick to finding out what's wrong with the test is to run it as follows:
./python Lib/test/regrtest.py -v test_descrtut
That works well for unnittest-based tests too. For a doctest-based test, ./python Lib/test/test_descrtut.py -v can simplify life by getting regrtest out of the picture.
This runs the test in "verbose mode", and in that mode regrtest doesn't do its output comparison trick -- it just shows you all the test's output. Now, doctest also scans sys.argv for a -v (which I think is wrong because it's not the main program, but that's how it works),
Not really, that's how the doctest-based test_descrtut.py was *written*, so that the alternative line above works to get regrtest out of the equation entirely (when desired), i.e. so that you can run it exactly the same way you'd run a native doctested module (== one that never heard of regrtest). It didn't have to be coded that way. An alternative is shown by test_difflib.py, here in its entirety: from test_support import verbose import doctest, difflib doctest.testmod(difflib, verbose=verbose) There doctest is explicitly told which verbose mode to use, and in that case doctest doesn't look at sys.argv. This makes its behavior easier to understand *in the context* of regrtest, but harder to understand as a doctest (which normally are run as "main programs").
and this causes doctest to be much more verbose -- which may or may not be helpful but shouldn't be a problem unless you are without a scrollbar.
If you're running on Win9x, you are indeed without a scrollbar (50 lines max), and that's part of why doctest never writes to stderr: even under a 50-line Win95 shell, you can pipe its output to 'more' and not lose anything (and Win9x shells don't support stderr redirection at all).
You said that when you run the test standalone the type names are different. *How* do you run it standalone? If I run it like this:
./python Lib/test/test_descrtut.py
it works fine (and -v works here too). But if I run it like this:
./python >>> import test.test_descrtut # this does not run the tests! >>> test.test_descrtut.test_main() # this does!
then indeed doctset complains that instead of
it saw
(and other failures, all caused by the incorporation of the module name (__name__) in type names when they are printed).
Jack, is that really what you do? I'm finding that hard to believe.
It is possible to rewrite the tests to avoid the dependency on the module name: change all occurrences of test_descrtut to test.test_descrtut, including the three places in the test_main() function. But since it's not my test, I'll leave that to Tim.
I'd rather delete the test. But since I doubt Jack is running the test in the devious way shown above, I'll hold off until he surprises me <wink>.
... This still doesn't solve your real problem (that test_descrtut fails), but the regrtest.py -v trick should make it possible for you to find out painlessly. :-)
For this particular test, ./python Lib/test/test_descrtut.py is the easiest way. Then doctest will show all and only the tests that fail, and you don't even have to think about what regrtest may or may not be doing to the output.
participants (3)
-
Guido van Rossum -
Jack Jansen -
Tim Peters