[IronPython] Misleading test runner output /all VS /test:name_of_test

Dino Viehland dinov at microsoft.com
Fri Feb 4 19:11:08 CET 2011

Daniel wrote:
> Please, help me wrap my head around this issue. Wanted to run unittests to see
> what areas need to be addressed.
> This is the tail of
> ...
> Languages\IronPython\IronPython\cpy test_univnewlines_cpy
>                     PASSED    6.7633868
> Languages\IronPython\IronPython\cpy test_weakref_cpy
>                     FAILED    14.8568497
> Languages\IronPython\IronPython\cpy test_zipfile_cpy
>                     DISABLED  0
> Languages\IronPython\IronPython\cpy test_zipfile64_cpy
>                     DISABLED  0
> Languages\IronPython\IronPython\modules modules_io_related_ipy
>                     PASSED    15.1468663
> Failed test summary:
> test_pyc_ipy
> modules_system_related_ipy
> test_subprocess_cpy
> test_weakref_cpy
> Total time: 1743.824741 seconds
> Immediately after that ran the "failing" test_subprocess_cpy individually and
> got a passing grade:
> C:\work\ironlang>Test\TestRunner\TestRunner\bin\Debug\TestRunner.exe
> Test\IronPython.tests /test:test_subprocess_cpy /verbose
> Languages\IronPython\IronPython\cpy test_subprocess_cpy
>                     PASSED    343.3856406
> Total time: 343.428643 seconds
> I guess we can expect some variation in outcome there, due to random load
> effects on pipe throughput between subprocess and calling parent code, and I
> will check for that. Besides the randomness of conditions, are there variations
> in settings between /all and /test:name_of_test that would cause different test
> outcomes?

The only difference between running a single test and running multiple tests is 
that we do attempt to run tests in parallel when we run multiple tests (by default 
we run 6 at a time which can be configured w/ the /threads: option).  

Test_subprocess is marked as being parallel safe so this can run w/ other 
processes.  If it's failing only when run in parallel then it should be marked as
being not-parallel safe and then it'll run at the end of the test run after all of
the parallel safe tests have completed.

More information about the Ironpython-users mailing list