[Python-Dev] test_descrtut failing

[Tim Peters]

[Jack Jansen]
>> Test_descrtut is driving me up the wall (or actually, the whole test
>> suite is:-).
>> ...
>> If I run it standalone it doesn't work because all the typenames are
>> different. This is a general problem, I've seen the same on unix.

What does "run it standalone" mean to you, exactly?  Here are 4 possible
meanings that work fine for me (running it alone with or without regrtest,
and from the test directory or from my build directory; and I haven't been
able to dream up another meaning!):

C:\Code\python\PCbuild>python ../lib/test/test_descrtut.py

C:\Code\python\PCbuild>python ../lib/test/regrtest.py test_descrtut.py
test_descrtut
1 test OK.

C:\Code\python\Lib\test>..\..\pcbuild\python test_descrtut.py

C:\Code\python\Lib\test>..\..\pcbuild\python regrtest.py test_descrtut.py
test_descrtut
1 test OK.


>> In general, I find the new test framework not much of an
>> improvement.

The problem is more that we don't have a new framework:  we've squashed
additional frameworks *under* a catch-all old framework, and the latter
doesn't really know how to deal with the former.  unittest and doctest both
print their failure reports to stdout, but stdout is what regrtest is
looking at, and the latter complains if anything shows up on stdout that it
doesn't expect.

[Guido]
> ...
> But regrtest, which is comparing the output to the contents of the
> file Lib/test/output/test_descrtut,

Which latter doesn't exist -- that's another twist in the tale.

>    The actual stdout doesn't match the expected stdout.
>    This much did match (between asterisk lines):
>    **********************************************************************
>    test_descrtut
>    **********************************************************************

When an expected-output file doesn't exist, regrtest "pretends" that it does
exist, and consists of a single line containing the name of the test.
That's why it says "This much did match" despite that output/test_descrtut
doesn't actually exist.

> ...
> The trick to finding out what's wrong with the test is to run it as
> follows:
>
>     ./python Lib/test/regrtest.py -v test_descrtut

That works well for unnittest-based tests too.  For a doctest-based test,

      ./python Lib/test/test_descrtut.py -v

can simplify life by getting regrtest out of the picture.

> This runs the test in "verbose mode", and in that mode regrtest
> doesn't do its output comparison trick -- it just shows you all the
> test's output.  Now, doctest also scans sys.argv for a -v (which I
> think is wrong because it's not the main program, but that's how it
> works),

Not really, that's how the doctest-based test_descrtut.py was *written*, so
that the alternative line above works to get regrtest out of the equation
entirely (when desired), i.e. so that you can run it exactly the same way
you'd run a native doctested module (== one that never heard of regrtest).
It didn't have to be coded that way.  An alternative is shown by
test_difflib.py, here in its entirety:

from test_support import verbose
import doctest, difflib
doctest.testmod(difflib, verbose=verbose)

There doctest is explicitly told which verbose mode to use, and in that case
doctest doesn't look at sys.argv.  This makes its behavior easier to
understand *in the context* of regrtest, but harder to understand as a
doctest (which normally are run as "main programs").

> and this causes doctest to be much more verbose -- which may
> or may not be helpful but shouldn't be a problem unless you are
> without a scrollbar.

If you're running on Win9x, you are indeed without a scrollbar (50 lines
max), and that's part of why doctest never writes to stderr:  even under a
50-line Win95 shell, you can pipe its output to 'more' and not lose anything
(and Win9x shells don't support stderr redirection at all).

> You said that when you run the test standalone the type names are
> different.  *How* do you run it standalone?  If I run it like this:
>
>     ./python Lib/test/test_descrtut.py
>
> it works fine (and -v works here too).  But if I run it like this:
>
>     ./python
>     >>> import test.test_descrtut	# this does not run the tests!
>     >>> test.test_descrtut.test_main()	# this does!
>
> then indeed doctset complains that instead of
>
>     <type 'test_descrtut.defaultdict'>
>
> it saw
>
>     <type 'test.test_descrtut.defaultdict'>
>
> (and other failures, all caused by the incorporation of the module
> name (__name__) in type names when they are printed).

Jack, is that really what you do?  I'm finding that hard to believe.

> It is possible to rewrite the tests to avoid the dependency
> on the module name: change all occurrences of test_descrtut to
> test.test_descrtut, including the three places in the test_main()
> function.  But since it's not my test, I'll leave that to Tim.

I'd rather delete the test.  But since I doubt Jack is running the test in
the devious way shown above, I'll hold off until he surprises me <wink>.

> ...
> This still doesn't solve your real problem (that test_descrtut fails),
> but the regrtest.py -v trick should make it possible for you to find
> out painlessly. :-)

For this particular test,

      ./python  Lib/test/test_descrtut.py

is the easiest way.  Then doctest will show all and only the tests that
fail, and you don't even have to think about what regrtest may or may not be
doing to the output.