bug in regrtest

[Andrew Dalke]

Hello,

  We use a modified version of regrtest (Lib/test/regrtest.py)
for internal regression tests.  I just tracked down a bug in
our version which seems to be present in the original one.


  The regression run is compared to golden output from a
known, correct file.  This is done in "runtest" using an instance
of the Compare class, which compares each output data with
the corresponding data from the standard.  If they are different,
an exception is raised.

  The problem comes when the regression run is finished while
golden data is still available.  There is a `close' method in the
Compare class which can check for this sort of error and raise
an exception, but it is never called.

  The relevant code from regrtest.runtest looks like (with some
editing for clarity):

     if generate:
         cfp = open(outputfile, "w")
     elif verbose:
         cfp = sys.stdout
     else:
         cfp = Compare(outputfile)
     try:
        save_stdout = sys.stdout
        try:
            if cfp:
                sys.stdout = cfp
                print test              # Output file starts with test name
            __import__(test, globals(), locals(), [])
        finally:
            sys.stdout = save_stdout
     except ....
        return 0
     else:
        return 1


It looks like the easiest, correct fix is to add the following:

            if cfp and not generate and not cfp:
               cfp.close()

just after the __import__ statement.

I tested against the Python 1.5.1 distribution, which has a
slightly older version of regrtest than 1.5.2, although the latest
CVS version still has the flaw.  There are two tests which now fail
the regression tests, test_re and test_select.

I looked more closely at test_select, since it is smaller than
test_re.  The output/test_select file contains output which
agrees with the output of
  python regrtest.py -v test_select.py

*However*, if you look at test_select.py, the proper output will
only be generated when the verbose flag is set, because of constructs
like:

>                 if verbose:
>                         print 'timeout =', tout

In normal tests, test_select.py should generate no output.  This
can be verified by using -g to generate a new output/test_select
file, which will contain only the test name.

So this output difference was masked by the Compare problem
mentioned above.

Looking at the CVS logs, it has been that way for a long time,
1996/12/11.

						Andrew Dalke
						dalke at bioreason.com