[Pytest-commit] Issue #485: xdist: --boxed test crash throws away STDOUT and STDERR even if they have data (hpk42/pytest)

Ethan John issues-reply at bitbucket.org
Tue Mar 18 20:52:31 CET 2014


New issue 485: xdist: --boxed test crash throws away STDOUT and STDERR even if they have data
https://bitbucket.org/hpk42/pytest/issue/485/xdist-boxed-test-crash-throws-away-stdout

Ethan John:

I'm using pytest 2.5.2 and xdist 1.10 under CentOS 6.2.

We use py.test --boxed for test isolation and resiliency to underlying crashes.

We use Cython extensively and various underlying libraries written in C, and we were running into situations where py.test was reporting that processes crashed (usually with signal 6, ABORT due to assertion failure) and not leaving anything else behind with which to track down the problem.

After fixing our core file generation permissions and discovering where the processes were hanging, it was clear in reproductions outside of py.test that the processes were generating output that our test reports were not seeing.

Some digging into xdist's code shows the reason: If a test function crashes, the result returned from plugin.py:forked_run_report():ff.waitfinish() does not have a retval (because it was never returned), and therefore the report_process_crash() method executes and simply reports the result.signal.

This is fine in the default case where py.test is run with --capture fd, because result.err and result.out will be empty (I also think this is incorrect, but it's nontrivial to fix given my reading of the source).

However, in the event that a user runs with --capture sys, then result.out and result.err do indeed have useful information in them, but this data is thrown away.

This is trivial to reproduce with a simple C file that just asserts:
{noformat}
#include <assert.h>
int main() {
  assert(0);
}
{noformat}

And a degenerate test file that can be executed after compiling the above:
{noformat}
#!/usr/bin/env crpython

import os
import unittest2 as unittest

class TestAbrt(unittest.TestCase):
    def testit(self):
        os.execvp('./a.out', ['a.out'])

if __name__ == "__main__":
    unittest.main()
{noformat}

What I would expect:
# A process crash with --boxed should not be considered a failure! It should be a test error and relevant information like stdout and stderr should be communicated to the user.
#* Ideally, test results should not be "all or nothing" in the world of --boxed even if --capture is specified as "fd". It should be possible to read input from child tests even in the event of failure, at least in the UNIX world. Maybe --boxed should override the default --capture behavior in order to capture as much as possible? Without xdist the --capture flag seems useful, but with it, more advanced behavior is possible.
# If a crash continues to be classified as a failure, then any status information collected by the test report should be included in the failure report. Assuming that result.err and result.out are empty seems incorrect.

Thanks!




More information about the pytest-commit mailing list