Python 2.2.2 regrtest.py Module Produces Unpredictable Results

[Michael Hudson]

Merton Campbell Crockett <mcc at TO.GD-ES.COM> writes:

> There seems to be a problem with Lib/test/regrtest.py that was included in
> the Python 2.2.2 distribution.

There's certainly something up with your build of Python.

> After downloading Python 2.2.2, I modified configure to skip tests of
> modules that are not supported under BSD/OS 4.x and 5.x.  I ran the standard
> regression test using "gmake test".  Lib/test/regrtest.py reported that the
> following modules crashed.
> 
> 	test_long_future.py(c)
> 	test_math.py(c)

Can you run regrtest verbosely?  Just knowing a test failed isn't a
whole heap o' use.

> Setting the TESTOPTS environmental variable, I tested each of the modules
> individually.  Lib/test/regrtest.py reported that each test completed
> successfully in both passes.

Hmm.  Something odd going on, here.

> This seemed a little strange.  So I changed TESTOPS to conduct the tests in
> a random order.  This time both modules passed but two other modules failed.
> 
> At first, it appeared that the tests that failed were whatever ended up at 
> positions 86 and 91 in the list.  These positions in the list correspond to
> the positions in the default order of execution of the above modules.
> 
> Several more random tests showed this not to be the case.  Conducting the
> regression tests in random order results in random tests failing with tests
> failing in pass 1 being successful in pass 2.

Sometimes one of the tests screws up the environment in the process of
failing sufficiently that all tests that run after it are at risk.
Not knowing your system well, I'd nominate test_signal as being one of
the more likely ones...

> How does one know that Python is built correctly when the regression tests
> produce different results from run to run and pass to pass?

That's easy: it's not built correctly.  It's possible running tests in
a random order is bust generally, but unlikely.

> Is the random test feature one of those "Doctor.  Doctor.  It hurts when I
> do this." scenarios?

Not really.  Are you confident of your RAM?  Have you tried building
without optimizations?

Cheers,
M.

-- 
  But since I'm not trying to impress  anybody in The Software Big
  Top, I'd rather walk the wire using a big pole, a safety harness,
  a net, and with the wire not more than 3 feet off the ground.
                                   -- Grant Griffin, comp.lang.python